2024-04-12T02:53:34.305 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-12T02:53:34.305 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596 -- /home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596/orig.config.yaml 2024-04-12T02:53:34.341 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-12T02:53:34.506 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596 branch: reef-release description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} email: null first_in_suite: false job_id: '7652596' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: rhel os_version: '8.6' overrides: admin_socket: branch: reef-release ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - enabled_features: - resharding endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 rgw-multisite-tests: args: - tests.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: reef-release sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 6358 sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 sleep_before_teardown: 0 subset: 111/120000 suite: rgw suite_branch: reef-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 targets: smithi043.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM360nfrc8+vzqLBmt/bUgq/Qji/8Ta1ZqA1yis6gch2B7vP9ozECqHJvYXqFyNKiYsii47X9221GsM42/v4qhM= smithi149.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN4+Al/t3FeS8VxGyA79xzsyQJKFWB6bq35ATLnGIa+oeWgp3QpA7yvw/te4TGZwz36fc5R+dr7d8WywxqnZzLc= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-11_20:30:42 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-12T02:53:34.506 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa; will attempt to use it 2024-04-12T02:53:34.507 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks 2024-04-12T02:53:34.507 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-12T02:53:34.508 INFO:teuthology.task.internal:Checking packages... 2024-04-12T02:53:34.532 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'd540ebaca6b131a1dd560e7f69e024b133bbaa42' 2024-04-12T02:53:34.532 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T02:53:34.532 INFO:teuthology.packaging:ref: None 2024-04-12T02:53:34.532 INFO:teuthology.packaging:tag: None 2024-04-12T02:53:34.532 INFO:teuthology.packaging:branch: reef-release 2024-04-12T02:53:34.532 INFO:teuthology.packaging:sha1: d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T02:53:34.532 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=reef-release 2024-04-12T02:53:34.692 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1187.gd540ebac 2024-04-12T02:53:34.693 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-12T02:53:34.700 INFO:teuthology.task.internal:no buildpackages task found 2024-04-12T02:53:34.700 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-12T02:53:34.740 INFO:teuthology.task.internal:Saving configuration 2024-04-12T02:53:34.755 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-12T02:53:34.765 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T02:53:34.797 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 02:48:34.056169', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM360nfrc8+vzqLBmt/bUgq/Qji/8Ta1ZqA1yis6gch2B7vP9ozECqHJvYXqFyNKiYsii47X9221GsM42/v4qhM='} 2024-04-12T02:53:34.831 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 02:48:34.054259', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN4+Al/t3FeS8VxGyA79xzsyQJKFWB6bq35ATLnGIa+oeWgp3QpA7yvw/te4TGZwz36fc5R+dr7d8WywxqnZzLc='} 2024-04-12T02:53:34.831 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-12T02:53:34.837 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2024-04-12T02:53:34.838 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2024-04-12T02:53:34.838 INFO:teuthology.run_tasks:Running task console_log... 2024-04-12T02:53:34.929 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0dbe620e50>, signals=[15]) 2024-04-12T02:53:34.929 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-12T02:53:34.936 INFO:teuthology.task.internal:Opening connections... 2024-04-12T02:53:34.936 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi043.front.sepia.ceph.com 2024-04-12T02:53:34.938 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T02:53:35.041 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi149.front.sepia.ceph.com 2024-04-12T02:53:35.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T02:53:35.122 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-12T02:53:35.128 DEBUG:teuthology.orchestra.run.smithi043:> uname -m 2024-04-12T02:53:35.150 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64 2024-04-12T02:53:35.150 DEBUG:teuthology.orchestra.run.smithi043:> cat /etc/os-release 2024-04-12T02:53:35.211 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="Red Hat Enterprise Linux" 2024-04-12T02:53:35.211 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="8.6 (Ootpa)" 2024-04-12T02:53:35.211 INFO:teuthology.orchestra.run.smithi043.stdout:ID="rhel" 2024-04-12T02:53:35.211 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE="fedora" 2024-04-12T02:53:35.211 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="8.6" 2024-04-12T02:53:35.211 INFO:teuthology.orchestra.run.smithi043.stdout:PLATFORM_ID="platform:el8" 2024-04-12T02:53:35.211 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-12T02:53:35.211 INFO:teuthology.orchestra.run.smithi043.stdout:ANSI_COLOR="0;31" 2024-04-12T02:53:35.211 INFO:teuthology.orchestra.run.smithi043.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-12T02:53:35.212 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://www.redhat.com/" 2024-04-12T02:53:35.212 INFO:teuthology.orchestra.run.smithi043.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-12T02:53:35.212 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-12T02:53:35.212 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T02:53:35.212 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-12T02:53:35.212 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-12T02:53:35.212 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-12T02:53:35.212 INFO:teuthology.orchestra.run.smithi043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-12T02:53:35.213 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2024-04-12T02:53:35.245 DEBUG:teuthology.orchestra.run.smithi149:> uname -m 2024-04-12T02:53:35.265 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64 2024-04-12T02:53:35.266 DEBUG:teuthology.orchestra.run.smithi149:> cat /etc/os-release 2024-04-12T02:53:35.324 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="Red Hat Enterprise Linux" 2024-04-12T02:53:35.324 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="8.6 (Ootpa)" 2024-04-12T02:53:35.324 INFO:teuthology.orchestra.run.smithi149.stdout:ID="rhel" 2024-04-12T02:53:35.324 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="fedora" 2024-04-12T02:53:35.324 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="8.6" 2024-04-12T02:53:35.324 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el8" 2024-04-12T02:53:35.324 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-12T02:53:35.325 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31" 2024-04-12T02:53:35.325 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-12T02:53:35.325 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://www.redhat.com/" 2024-04-12T02:53:35.325 INFO:teuthology.orchestra.run.smithi149.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-12T02:53:35.325 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-12T02:53:35.325 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T02:53:35.325 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-12T02:53:35.325 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-12T02:53:35.325 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-12T02:53:35.325 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-12T02:53:35.325 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2024-04-12T02:53:35.361 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-12T02:53:35.370 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-12T02:53:35.389 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-12T02:53:35.389 DEBUG:teuthology.orchestra.run.smithi043:> test '!' -e /home/ubuntu/cephtest 2024-04-12T02:53:35.392 DEBUG:teuthology.orchestra.run.smithi149:> test '!' -e /home/ubuntu/cephtest 2024-04-12T02:53:35.411 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-12T02:53:35.417 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-12T02:53:35.417 DEBUG:teuthology.orchestra.run.smithi043:> test -z $(ls -A /var/lib/ceph) 2024-04-12T02:53:35.452 DEBUG:teuthology.orchestra.run.smithi149:> test -z $(ls -A /var/lib/ceph) 2024-04-12T02:53:35.483 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-12T02:53:35.572 INFO:teuthology.run_tasks:Running task kernel... 2024-04-12T02:53:35.587 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-12T02:53:35.587 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-12T02:53:35.587 DEBUG:teuthology.orchestra.run.smithi043:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-12T02:53:35.587 DEBUG:teuthology.orchestra.run.smithi149:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-12T02:53:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T02:53:35.607 DEBUG:teuthology.orchestra.run.smithi149:> uname -r 2024-04-12T02:53:35.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T02:53:35.608 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2024-04-12T02:53:35.666 INFO:teuthology.orchestra.run.smithi149.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-12T02:53:35.667 INFO:teuthology.task.kernel:Running kernel on smithi149: 4.18.0-513.9.1.el8_9.x86_64 2024-04-12T02:53:35.667 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum install -y kernel 2024-04-12T02:53:35.670 INFO:teuthology.orchestra.run.smithi043.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-12T02:53:35.670 INFO:teuthology.task.kernel:Running kernel on smithi043: 4.18.0-513.9.1.el8_9.x86_64 2024-04-12T02:53:35.670 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum install -y kernel 2024-04-12T02:53:37.219 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T02:53:37.220 INFO:teuthology.orchestra.run.smithi043.stdout:Unable to read consumer identity 2024-04-12T02:53:37.279 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T02:53:37.279 INFO:teuthology.orchestra.run.smithi149.stdout:Unable to read consumer identity 2024-04-12T02:53:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T02:53:37.329 INFO:teuthology.orchestra.run.smithi043.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-12T02:53:37.329 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T02:53:37.414 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T02:53:37.414 INFO:teuthology.orchestra.run.smithi149.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-12T02:53:37.414 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T02:53:39.059 INFO:teuthology.orchestra.run.smithi043.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2024-04-12T02:53:39.207 INFO:teuthology.orchestra.run.smithi149.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-04-12T02:53:39.835 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 22 MB/s | 16 MB 00:00 2024-04-12T02:53:40.408 INFO:teuthology.orchestra.run.smithi149.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-04-12T02:53:43.860 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 426 kB/s | 24 kB 00:00 2024-04-12T02:53:44.040 INFO:teuthology.orchestra.run.smithi149.stdout:lab-extras 413 kB/s | 24 kB 00:00 2024-04-12T02:53:44.663 INFO:teuthology.orchestra.run.smithi043.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-12T02:53:44.663 INFO:teuthology.orchestra.run.smithi043.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-12T02:53:44.751 INFO:teuthology.orchestra.run.smithi149.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-12T02:53:44.752 INFO:teuthology.orchestra.run.smithi149.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-12T02:53:44.763 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T02:53:44.765 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-12T02:53:44.765 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T02:53:44.816 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T02:53:44.817 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-04-12T02:53:44.818 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T02:53:44.911 DEBUG:teuthology.orchestra.run.smithi043:> echo no | sudo yum reinstall kernel || true 2024-04-12T02:53:44.970 DEBUG:teuthology.orchestra.run.smithi149:> echo no | sudo yum reinstall kernel || true 2024-04-12T02:53:45.300 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T02:53:45.300 INFO:teuthology.orchestra.run.smithi043.stdout:Unable to read consumer identity 2024-04-12T02:53:45.307 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T02:53:45.307 INFO:teuthology.orchestra.run.smithi149.stdout:Unable to read consumer identity 2024-04-12T02:53:45.357 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T02:53:45.357 INFO:teuthology.orchestra.run.smithi043.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-12T02:53:45.357 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T02:53:45.364 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T02:53:45.365 INFO:teuthology.orchestra.run.smithi149.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-12T02:53:45.365 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T02:53:45.686 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:00:02 ago on Fri 12 Apr 2024 02:53:43 AM UTC. 2024-04-12T02:53:45.717 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:02 ago on Fri 12 Apr 2024 02:53:43 AM UTC. 2024-04-12T02:53:45.774 INFO:teuthology.orchestra.run.smithi149.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-12T02:53:45.774 INFO:teuthology.orchestra.run.smithi149.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-12T02:53:45.811 INFO:teuthology.orchestra.run.smithi043.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-12T02:53:45.814 INFO:teuthology.orchestra.run.smithi043.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-12T02:53:45.817 INFO:teuthology.orchestra.run.smithi149.stderr:Error: No packages marked for reinstall. 2024-04-12T02:53:45.866 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum reinstall -y kernel || true 2024-04-12T02:53:45.888 INFO:teuthology.orchestra.run.smithi043.stderr:Error: No packages marked for reinstall. 2024-04-12T02:53:45.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum reinstall -y kernel || true 2024-04-12T02:53:46.203 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T02:53:46.203 INFO:teuthology.orchestra.run.smithi149.stdout:Unable to read consumer identity 2024-04-12T02:53:46.256 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T02:53:46.256 INFO:teuthology.orchestra.run.smithi149.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-12T02:53:46.256 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T02:53:46.330 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T02:53:46.330 INFO:teuthology.orchestra.run.smithi043.stdout:Unable to read consumer identity 2024-04-12T02:53:46.390 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T02:53:46.390 INFO:teuthology.orchestra.run.smithi043.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-12T02:53:46.390 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T02:53:46.567 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:00:03 ago on Fri 12 Apr 2024 02:53:43 AM UTC. 2024-04-12T02:53:46.629 INFO:teuthology.orchestra.run.smithi149.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-12T02:53:46.629 INFO:teuthology.orchestra.run.smithi149.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-12T02:53:46.662 INFO:teuthology.orchestra.run.smithi149.stderr:Error: No packages marked for reinstall. 2024-04-12T02:53:46.715 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q kernel | sort -rV | head -n 1 2024-04-12T02:53:46.742 INFO:teuthology.orchestra.run.smithi043.stdout:Last metadata expiration check: 0:00:03 ago on Fri 12 Apr 2024 02:53:43 AM UTC. 2024-04-12T02:53:46.811 INFO:teuthology.orchestra.run.smithi043.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-12T02:53:46.811 INFO:teuthology.orchestra.run.smithi043.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-12T02:53:46.845 INFO:teuthology.orchestra.run.smithi043.stderr:Error: No packages marked for reinstall. 2024-04-12T02:53:46.885 INFO:teuthology.orchestra.run.smithi149.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-12T02:53:46.886 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-12T02:53:46.886 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-12T02:53:46.886 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2024-04-12T02:53:46.886 DEBUG:teuthology.orchestra.run.smithi149:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-12T02:53:46.905 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q kernel | sort -rV | head -n 1 2024-04-12T02:53:46.924 INFO:teuthology.orchestra.run.smithi149.stdout:ttyS1 2024-04-12T02:53:46.940 DEBUG:teuthology.parallel:result is None 2024-04-12T02:53:47.076 INFO:teuthology.orchestra.run.smithi043.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-12T02:53:47.076 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-12T02:53:47.076 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-12T02:53:47.076 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2024-04-12T02:53:47.076 DEBUG:teuthology.orchestra.run.smithi043:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-12T02:53:47.114 INFO:teuthology.orchestra.run.smithi043.stdout:ttyS1 2024-04-12T02:53:47.130 DEBUG:teuthology.parallel:result is None 2024-04-12T02:53:47.130 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-12T02:53:47.137 INFO:teuthology.task.internal:Creating test directory... 2024-04-12T02:53:47.137 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-12T02:53:47.172 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-12T02:53:47.190 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-12T02:53:47.196 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-12T02:53:47.222 INFO:teuthology.task.internal:Creating archive directory... 2024-04-12T02:53:47.222 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-12T02:53:47.233 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-12T02:53:47.272 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-12T02:53:47.279 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-12T02:53:47.279 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-12T02:53:47.299 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-12T02:53:47.334 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T02:53:47.347 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T02:53:47.348 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T02:53:47.358 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T02:53:47.359 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-12T02:53:47.366 INFO:teuthology.task.internal:Configuring sudo... 2024-04-12T02:53:47.366 DEBUG:teuthology.orchestra.run.smithi043:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-12T02:53:47.392 DEBUG:teuthology.orchestra.run.smithi149:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-12T02:53:47.426 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-12T02:53:47.435 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-12T02:53:47.435 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-12T02:53:47.468 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-12T02:53:47.487 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T02:53:47.584 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T02:53:47.632 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T02:53:47.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T02:53:47.769 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T02:53:47.769 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-12T02:53:47.844 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T02:53:47.895 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T02:53:47.943 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T02:53:48.035 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T02:53:48.069 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T02:53:48.069 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-12T02:53:48.136 DEBUG:teuthology.orchestra.run.smithi043:> sudo service rsyslog restart 2024-04-12T02:53:48.138 DEBUG:teuthology.orchestra.run.smithi149:> sudo service rsyslog restart 2024-04-12T02:53:48.199 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T02:53:48.225 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T02:53:48.612 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-12T02:53:48.623 INFO:teuthology.task.internal:Starting timer... 2024-04-12T02:53:48.623 INFO:teuthology.run_tasks:Running task pcp... 2024-04-12T02:53:48.650 INFO:teuthology.run_tasks:Running task selinux... 2024-04-12T02:53:48.682 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-12T02:53:48.682 DEBUG:teuthology.orchestra.run.smithi043:> sudo service auditd rotate 2024-04-12T02:53:48.749 INFO:teuthology.orchestra.run.smithi043.stdout:Rotating logs: 2024-04-12T02:53:48.751 DEBUG:teuthology.orchestra.run.smithi149:> sudo service auditd rotate 2024-04-12T02:53:48.799 INFO:teuthology.orchestra.run.smithi149.stdout:Rotating logs: 2024-04-12T02:53:48.801 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-12T02:53:48.801 DEBUG:teuthology.orchestra.run.smithi043:> /usr/sbin/getenforce 2024-04-12T02:53:48.825 INFO:teuthology.orchestra.run.smithi043.stdout:Permissive 2024-04-12T02:53:48.825 DEBUG:teuthology.orchestra.run.smithi149:> /usr/sbin/getenforce 2024-04-12T02:53:48.878 INFO:teuthology.orchestra.run.smithi149.stdout:Permissive 2024-04-12T02:53:48.878 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi043.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi149.front.sepia.ceph.com': 'permissive'} 2024-04-12T02:53:48.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T02:53:48.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T02:53:48.912 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T02:53:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T02:53:48.940 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-12T02:53:48.940 DEBUG:teuthology.orchestra.run.smithi043:> sudo /usr/sbin/setenforce permissive 2024-04-12T02:53:48.983 DEBUG:teuthology.orchestra.run.smithi149:> sudo /usr/sbin/setenforce permissive 2024-04-12T02:53:49.010 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-12T02:53:49.021 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-12T02:53:49.036 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-12T02:53:49.037 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi043.front.sepia.ceph.com,smithi149.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-12T03:02:45.646 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi043.front.sepia.ceph.com'), Remote(name='ubuntu@smithi149.front.sepia.ceph.com')] 2024-04-12T03:02:45.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-04-12T03:02:45.648 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T03:02:45.774 DEBUG:teuthology.orchestra.run.smithi043:> true 2024-04-12T03:02:45.848 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-04-12T03:02:45.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-04-12T03:02:45.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T03:02:45.931 DEBUG:teuthology.orchestra.run.smithi149:> true 2024-04-12T03:02:45.999 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-04-12T03:02:46.000 INFO:teuthology.run_tasks:Running task clock... 2024-04-12T03:02:46.009 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-12T03:02:46.010 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-12T03:02:46.010 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T03:02:46.012 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-12T03:02:46.012 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T03:02:46.051 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-12T03:02:46.069 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-12T03:02:46.071 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-12T03:02:46.086 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-12T03:02:46.149 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: ntpd: command not found 2024-04-12T03:02:46.160 INFO:teuthology.orchestra.run.smithi149.stderr:sudo: ntpd: command not found 2024-04-12T03:02:46.163 INFO:teuthology.orchestra.run.smithi043.stdout:506 Cannot talk to daemon 2024-04-12T03:02:46.171 INFO:teuthology.orchestra.run.smithi149.stdout:506 Cannot talk to daemon 2024-04-12T03:02:46.181 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-12T03:02:46.186 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-12T03:02:46.198 INFO:teuthology.orchestra.run.smithi043.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-12T03:02:46.200 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-12T03:02:46.276 INFO:teuthology.orchestra.run.smithi043.stderr:bash: ntpq: command not found 2024-04-12T03:02:46.281 INFO:teuthology.orchestra.run.smithi149.stderr:bash: ntpq: command not found 2024-04-12T03:02:46.282 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T03:02:46.282 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-04-12T03:02:46.282 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T03:02:46.282 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T03:02:46.283 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T03:02:46.283 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T03:02:46.285 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T03:02:46.286 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2024-04-12T03:02:46.286 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T03:02:46.286 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T03:02:46.286 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T03:02:46.286 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T03:02:46.287 INFO:teuthology.run_tasks:Running task install... 2024-04-12T03:02:46.295 DEBUG:teuthology.task.install:project ceph 2024-04-12T03:02:46.295 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}} 2024-04-12T03:02:46.295 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'} 2024-04-12T03:02:46.295 INFO:teuthology.task.install:Using flavor: default 2024-04-12T03:02:46.301 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-12T03:02:46.301 INFO:teuthology.task.install:extra packages: [] 2024-04-12T03:02:46.301 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'tag': None, 'wait_for_package': False} 2024-04-12T03:02:46.302 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T03:02:46.304 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'tag': None, 'wait_for_package': False} 2024-04-12T03:02:46.304 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T03:02:46.466 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/ 2024-04-12T03:02:46.467 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1187.gd540ebac 2024-04-12T03:02:46.479 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/ 2024-04-12T03:02:46.479 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1187.gd540ebac 2024-04-12T03:02:46.604 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-12T03:02:46.605 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:02:46.605 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-12T03:02:46.633 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-12T03:02:46.633 DEBUG:teuthology.orchestra.run.smithi149:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d540ebaca6b131a1dd560e7f69e024b133bbaa42/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-12T03:02:46.645 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/d540ebaca6b131a1dd560e7f69e024b133bbaa42/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-12T03:02:46.646 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:02:46.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-12T03:02:46.682 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-12T03:02:46.682 DEBUG:teuthology.orchestra.run.smithi043:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d540ebaca6b131a1dd560e7f69e024b133bbaa42/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-12T03:02:46.702 DEBUG:teuthology.orchestra.run.smithi149:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-12T03:02:46.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-12T03:02:46.779 DEBUG:teuthology.orchestra.run.smithi149:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-12T03:02:46.846 INFO:teuthology.orchestra.run.smithi149.stdout:check_obsoletes = 1 2024-04-12T03:02:46.847 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum clean all 2024-04-12T03:02:46.849 DEBUG:teuthology.orchestra.run.smithi043:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-12T03:02:46.882 INFO:teuthology.orchestra.run.smithi043.stdout:check_obsoletes = 1 2024-04-12T03:02:46.883 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum clean all 2024-04-12T03:02:47.232 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:02:47.318 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:02:48.426 INFO:teuthology.orchestra.run.smithi043.stdout:61 files removed 2024-04-12T03:02:48.437 INFO:teuthology.orchestra.run.smithi149.stdout:61 files removed 2024-04-12T03:02:48.481 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-radosgw 2024-04-12T03:02:48.497 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-radosgw 2024-04-12T03:02:48.832 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:02:48.866 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:02:50.722 INFO:teuthology.orchestra.run.smithi043.stdout:ceph packages for x86_64 202 kB/s | 80 kB 00:00 2024-04-12T03:02:50.747 INFO:teuthology.orchestra.run.smithi149.stdout:ceph packages for x86_64 172 kB/s | 80 kB 00:00 2024-04-12T03:02:51.022 INFO:teuthology.orchestra.run.smithi043.stdout:ceph noarch packages 44 kB/s | 11 kB 00:00 2024-04-12T03:02:51.074 INFO:teuthology.orchestra.run.smithi149.stdout:ceph noarch packages 39 kB/s | 11 kB 00:00 2024-04-12T03:02:51.341 INFO:teuthology.orchestra.run.smithi149.stdout:ceph source packages 6.2 kB/s | 1.5 kB 00:00 2024-04-12T03:02:51.397 INFO:teuthology.orchestra.run.smithi043.stdout:ceph source packages 4.3 kB/s | 1.5 kB 00:00 2024-04-12T03:02:51.551 INFO:teuthology.orchestra.run.smithi149.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-04-12T03:02:51.573 INFO:teuthology.orchestra.run.smithi043.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-04-12T03:02:52.484 INFO:teuthology.orchestra.run.smithi149.stdout:Extra Packages for Enterprise Linux 18 MB/s | 16 MB 00:00 2024-04-12T03:02:53.240 INFO:teuthology.orchestra.run.smithi043.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 16 MB 00:01 2024-04-12T03:02:56.118 INFO:teuthology.orchestra.run.smithi149.stdout:lab-extras 383 kB/s | 24 kB 00:00 2024-04-12T03:02:56.961 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 11 MB/s | 8.8 MB 00:00 2024-04-12T03:02:57.349 INFO:teuthology.orchestra.run.smithi043.stdout:lab-extras 404 kB/s | 24 kB 00:00 2024-04-12T03:02:58.126 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 13 MB/s | 8.8 MB 00:00 2024-04-12T03:02:59.779 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 52 MB/s | 67 MB 00:01 2024-04-12T03:03:01.485 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 41 MB/s | 67 MB 00:01 2024-04-12T03:03:10.738 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 46 MB/s | 60 MB 00:01 2024-04-12T03:03:13.794 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 44 MB/s | 60 MB 00:01 2024-04-12T03:03:26.715 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:03:26.717 INFO:teuthology.orchestra.run.smithi149.stdout:==================================================================================================== 2024-04-12T03:03:26.717 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-04-12T03:03:26.717 INFO:teuthology.orchestra.run.smithi149.stdout:==================================================================================================== 2024-04-12T03:03:26.718 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-12T03:03:26.718 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-radosgw x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 13 M 2024-04-12T03:03:26.718 INFO:teuthology.orchestra.run.smithi149.stdout:Upgrading: 2024-04-12T03:03:26.718 INFO:teuthology.orchestra.run.smithi149.stdout: librados2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.7 M 2024-04-12T03:03:26.718 INFO:teuthology.orchestra.run.smithi149.stdout: librbd1 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.6 M 2024-04-12T03:03:26.718 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-04-12T03:03:26.718 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-base x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 5.5 M 2024-04-12T03:03:26.718 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-common x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 22 M 2024-04-12T03:03:26.718 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-selinux x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 24 k 2024-04-12T03:03:26.718 INFO:teuthology.orchestra.run.smithi149.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-12T03:03:26.719 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 780 k 2024-04-12T03:03:26.719 INFO:teuthology.orchestra.run.smithi149.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-12T03:03:26.719 INFO:teuthology.orchestra.run.smithi149.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-12T03:03:26.719 INFO:teuthology.orchestra.run.smithi149.stdout: libradosstriper1 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 474 k 2024-04-12T03:03:26.719 INFO:teuthology.orchestra.run.smithi149.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-12T03:03:26.719 INFO:teuthology.orchestra.run.smithi149.stdout: librgw2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 6.6 M 2024-04-12T03:03:26.719 INFO:teuthology.orchestra.run.smithi149.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-12T03:03:26.719 INFO:teuthology.orchestra.run.smithi149.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-12T03:03:26.719 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-argparse x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 45 k 2024-04-12T03:03:26.719 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-common x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 129 k 2024-04-12T03:03:26.719 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cephfs x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 197 k 2024-04-12T03:03:26.720 INFO:teuthology.orchestra.run.smithi149.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-12T03:03:26.720 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rados x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 372 k 2024-04-12T03:03:26.720 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rbd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 364 k 2024-04-12T03:03:26.720 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rgw x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 107 k 2024-04-12T03:03:26.720 INFO:teuthology.orchestra.run.smithi149.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-12T03:03:26.720 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:03:26.720 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-12T03:03:26.720 INFO:teuthology.orchestra.run.smithi149.stdout:==================================================================================================== 2024-04-12T03:03:26.720 INFO:teuthology.orchestra.run.smithi149.stdout:Install 21 Packages 2024-04-12T03:03:26.720 INFO:teuthology.orchestra.run.smithi149.stdout:Upgrade 2 Packages 2024-04-12T03:03:26.720 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:03:26.721 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 59 M 2024-04-12T03:03:26.721 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-12T03:03:27.755 INFO:teuthology.orchestra.run.smithi149.stdout:(1/23): ceph-radosgw-18.2.2-1187.gd540ebac.el8. 15 MB/s | 13 MB 00:00 2024-04-12T03:03:27.813 INFO:teuthology.orchestra.run.smithi149.stdout:(2/23): ceph-selinux-18.2.2-1187.gd540ebac.el8. 418 kB/s | 24 kB 00:00 2024-04-12T03:03:27.905 INFO:teuthology.orchestra.run.smithi149.stdout:(3/23): libcephfs2-18.2.2-1187.gd540ebac.el8.x8 8.3 MB/s | 780 kB 00:00 2024-04-12T03:03:27.996 INFO:teuthology.orchestra.run.smithi149.stdout:(4/23): libradosstriper1-18.2.2-1187.gd540ebac. 5.1 MB/s | 474 kB 00:00 2024-04-12T03:03:28.431 INFO:teuthology.orchestra.run.smithi149.stdout:(5/23): librgw2-18.2.2-1187.gd540ebac.el8.x86_6 15 MB/s | 6.6 MB 00:00 2024-04-12T03:03:28.506 INFO:teuthology.orchestra.run.smithi149.stdout:(6/23): ceph-base-18.2.2-1187.gd540ebac.el8.x86 3.4 MB/s | 5.5 MB 00:01 2024-04-12T03:03:28.554 INFO:teuthology.orchestra.run.smithi149.stdout:(7/23): python3-ceph-argparse-18.2.2-1187.gd540 363 kB/s | 45 kB 00:00 2024-04-12T03:03:28.590 INFO:teuthology.orchestra.run.smithi149.stdout:(8/23): python3-ceph-common-18.2.2-1187.gd540eb 1.5 MB/s | 129 kB 00:00 2024-04-12T03:03:28.615 INFO:teuthology.orchestra.run.smithi149.stdout:(9/23): python3-cephfs-18.2.2-1187.gd540ebac.el 3.2 MB/s | 197 kB 00:00 2024-04-12T03:03:28.674 INFO:teuthology.orchestra.run.smithi149.stdout:(10/23): python3-rados-18.2.2-1187.gd540ebac.el 4.3 MB/s | 372 kB 00:00 2024-04-12T03:03:28.699 INFO:teuthology.orchestra.run.smithi149.stdout:(11/23): python3-rbd-18.2.2-1187.gd540ebac.el8. 4.3 MB/s | 364 kB 00:00 2024-04-12T03:03:28.741 INFO:teuthology.orchestra.run.smithi149.stdout:(12/23): python3-rgw-18.2.2-1187.gd540ebac.el8. 1.6 MB/s | 107 kB 00:00 2024-04-12T03:03:28.891 INFO:teuthology.orchestra.run.smithi149.stdout:(13/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2024-04-12T03:03:28.908 INFO:teuthology.orchestra.run.smithi149.stdout:(14/23): liboath-2.6.2-3.el8.x86_64.rpm 352 kB/s | 59 kB 00:00 2024-04-12T03:03:28.925 INFO:teuthology.orchestra.run.smithi149.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.2 MB/s | 75 kB 00:00 2024-04-12T03:03:29.051 INFO:teuthology.orchestra.run.smithi149.stdout:(16/23): thrift-0.13.0-2.el8.x86_64.rpm 12 MB/s | 1.7 MB 00:00 2024-04-12T03:03:29.123 INFO:teuthology.orchestra.run.smithi149.stdout:(17/23): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 238 kB/s | 47 kB 00:00 2024-04-12T03:03:29.268 INFO:teuthology.orchestra.run.smithi149.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.8 MB/s | 259 kB 00:00 2024-04-12T03:03:29.293 INFO:teuthology.orchestra.run.smithi149.stdout:(19/23): python3-prettytable-0.7.2-14.el8.noarc 181 kB/s | 44 kB 00:00 2024-04-12T03:03:29.560 INFO:teuthology.orchestra.run.smithi149.stdout:(20/23): ceph-common-18.2.2-1187.gd540ebac.el8. 8.0 MB/s | 22 MB 00:02 2024-04-12T03:03:29.602 INFO:teuthology.orchestra.run.smithi149.stdout:(21/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.0 MB/s | 354 kB 00:00 2024-04-12T03:03:29.827 INFO:teuthology.orchestra.run.smithi149.stdout:(22/23): librbd1-18.2.2-1187.gd540ebac.el8.x86_ 13 MB/s | 3.6 MB 00:00 2024-04-12T03:03:29.927 INFO:teuthology.orchestra.run.smithi149.stdout:(23/23): librados2-18.2.2-1187.gd540ebac.el8.x8 5.8 MB/s | 3.7 MB 00:00 2024-04-12T03:03:29.931 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:03:29.931 INFO:teuthology.orchestra.run.smithi149.stdout:Total 18 MB/s | 59 MB 00:03 2024-04-12T03:03:29.977 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-12T03:03:30.022 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-12T03:03:30.022 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-12T03:03:30.355 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-12T03:03:30.364 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-12T03:03:31.374 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:03:31.377 INFO:teuthology.orchestra.run.smithi043.stdout:==================================================================================================== 2024-04-12T03:03:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-12T03:03:31.378 INFO:teuthology.orchestra.run.smithi043.stdout:==================================================================================================== 2024-04-12T03:03:31.378 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-12T03:03:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 13 M 2024-04-12T03:03:31.378 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrading: 2024-04-12T03:03:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: librados2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.7 M 2024-04-12T03:03:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.6 M 2024-04-12T03:03:31.378 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-12T03:03:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 5.5 M 2024-04-12T03:03:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 22 M 2024-04-12T03:03:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 24 k 2024-04-12T03:03:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-12T03:03:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 780 k 2024-04-12T03:03:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-12T03:03:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-12T03:03:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 474 k 2024-04-12T03:03:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-12T03:03:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2 x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 6.6 M 2024-04-12T03:03:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-12T03:03:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-12T03:03:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 45 k 2024-04-12T03:03:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 129 k 2024-04-12T03:03:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 197 k 2024-04-12T03:03:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-12T03:03:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 372 k 2024-04-12T03:03:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 364 k 2024-04-12T03:03:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 107 k 2024-04-12T03:03:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-12T03:03:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:03:31.380 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-12T03:03:31.380 INFO:teuthology.orchestra.run.smithi043.stdout:==================================================================================================== 2024-04-12T03:03:31.380 INFO:teuthology.orchestra.run.smithi043.stdout:Install 21 Packages 2024-04-12T03:03:31.380 INFO:teuthology.orchestra.run.smithi043.stdout:Upgrade 2 Packages 2024-04-12T03:03:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:03:31.380 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 59 M 2024-04-12T03:03:31.380 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-12T03:03:31.655 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-12T03:03:31.852 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-04-12T03:03:32.010 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-12T03:03:32.421 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-12T03:03:32.435 INFO:teuthology.orchestra.run.smithi043.stdout:(1/23): ceph-base-18.2.2-1187.gd540ebac.el8.x86 8.2 MB/s | 5.5 MB 00:00 2024-04-12T03:03:32.485 INFO:teuthology.orchestra.run.smithi043.stdout:(2/23): ceph-selinux-18.2.2-1187.gd540ebac.el8. 491 kB/s | 24 kB 00:00 2024-04-12T03:03:32.619 INFO:teuthology.orchestra.run.smithi043.stdout:(3/23): libcephfs2-18.2.2-1187.gd540ebac.el8.x8 5.7 MB/s | 780 kB 00:00 2024-04-12T03:03:32.635 INFO:teuthology.orchestra.run.smithi149.stdout: Upgrading : librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-12T03:03:32.702 INFO:teuthology.orchestra.run.smithi043.stdout:(4/23): libradosstriper1-18.2.2-1187.gd540ebac. 5.6 MB/s | 474 kB 00:00 2024-04-12T03:03:32.718 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-12T03:03:32.924 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 4/25 2024-04-12T03:03:33.175 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2024-04-12T03:03:33.361 INFO:teuthology.orchestra.run.smithi043.stdout:(5/23): librgw2-18.2.2-1187.gd540ebac.el8.x86_6 10 MB/s | 6.6 MB 00:00 2024-04-12T03:03:33.393 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 6/25 2024-04-12T03:03:33.436 INFO:teuthology.orchestra.run.smithi043.stdout:(6/23): python3-ceph-argparse-18.2.2-1187.gd540 596 kB/s | 45 kB 00:00 2024-04-12T03:03:33.603 INFO:teuthology.orchestra.run.smithi043.stdout:(7/23): ceph-radosgw-18.2.2-1187.gd540ebac.el8. 7.1 MB/s | 13 MB 00:01 2024-04-12T03:03:33.629 INFO:teuthology.orchestra.run.smithi043.stdout:(8/23): python3-ceph-common-18.2.2-1187.gd540eb 672 kB/s | 129 kB 00:00 2024-04-12T03:03:33.662 INFO:teuthology.orchestra.run.smithi043.stdout:(9/23): python3-cephfs-18.2.2-1187.gd540ebac.el 3.3 MB/s | 197 kB 00:00 2024-04-12T03:03:33.696 INFO:teuthology.orchestra.run.smithi043.stdout:(10/23): python3-rados-18.2.2-1187.gd540ebac.el 5.4 MB/s | 372 kB 00:00 2024-04-12T03:03:33.721 INFO:teuthology.orchestra.run.smithi043.stdout:(11/23): python3-rbd-18.2.2-1187.gd540ebac.el8. 6.1 MB/s | 364 kB 00:00 2024-04-12T03:03:33.790 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 6/25 2024-04-12T03:03:33.930 INFO:teuthology.orchestra.run.smithi043.stdout:(12/23): ceph-common-18.2.2-1187.gd540ebac.el8. 10 MB/s | 22 MB 00:02 2024-04-12T03:03:33.955 INFO:teuthology.orchestra.run.smithi043.stdout:(13/23): python3-rgw-18.2.2-1187.gd540ebac.el8. 413 kB/s | 107 kB 00:00 2024-04-12T03:03:34.019 INFO:teuthology.orchestra.run.smithi149.stdout: Upgrading : librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-12T03:03:34.080 INFO:teuthology.orchestra.run.smithi043.stdout:(14/23): liboath-2.6.2-3.el8.x86_64.rpm 392 kB/s | 59 kB 00:00 2024-04-12T03:03:34.110 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-12T03:03:34.114 INFO:teuthology.orchestra.run.smithi043.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 474 kB/s | 75 kB 00:00 2024-04-12T03:03:34.164 INFO:teuthology.orchestra.run.smithi043.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 692 kB/s | 306 kB 00:00 2024-04-12T03:03:34.294 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-12T03:03:34.322 INFO:teuthology.orchestra.run.smithi043.stdout:(17/23): thrift-0.13.0-2.el8.x86_64.rpm 7.2 MB/s | 1.7 MB 00:00 2024-04-12T03:03:34.339 INFO:teuthology.orchestra.run.smithi043.stdout:(18/23): python3-prettytable-0.7.2-14.el8.noarc 251 kB/s | 44 kB 00:00 2024-04-12T03:03:34.359 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-12T03:03:34.365 INFO:teuthology.orchestra.run.smithi043.stdout:(19/23): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 188 kB/s | 47 kB 00:00 2024-04-12T03:03:34.532 INFO:teuthology.orchestra.run.smithi043.stdout:(20/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2024-04-12T03:03:34.558 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 9/25 2024-04-12T03:03:34.565 INFO:teuthology.orchestra.run.smithi043.stdout:(21/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.6 MB/s | 354 kB 00:00 2024-04-12T03:03:34.615 INFO:teuthology.orchestra.run.smithi043.stdout:(22/23): librados2-18.2.2-1187.gd540ebac.el8.x8 15 MB/s | 3.7 MB 00:00 2024-04-12T03:03:34.696 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8. 10/25 2024-04-12T03:03:34.841 INFO:teuthology.orchestra.run.smithi043.stdout:(23/23): librbd1-18.2.2-1187.gd540ebac.el8.x86_ 12 MB/s | 3.6 MB 00:00 2024-04-12T03:03:34.843 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:03:34.843 INFO:teuthology.orchestra.run.smithi043.stdout:Total 17 MB/s | 59 MB 00:03 2024-04-12T03:03:34.895 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-12T03:03:34.937 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 11/25 2024-04-12T03:03:34.946 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-12T03:03:34.946 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-12T03:03:35.219 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 12/25 2024-04-12T03:03:35.315 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-12T03:03:35.325 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-12T03:03:35.418 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 13/25 2024-04-12T03:03:35.491 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 13/25 2024-04-12T03:03:35.620 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2024-04-12T03:03:35.845 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2024-04-12T03:03:36.616 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-12T03:03:36.668 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2024-04-12T03:03:36.804 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-04-12T03:03:36.860 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 17/25 2024-04-12T03:03:36.939 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 17/25 2024-04-12T03:03:36.990 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-12T03:03:37.178 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 18/25 2024-04-12T03:03:37.318 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x8 19/25 2024-04-12T03:03:37.438 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-12T03:03:37.665 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-12T03:03:37.770 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-12T03:03:37.964 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 4/25 2024-04-12T03:03:38.212 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2024-04-12T03:03:38.434 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 6/25 2024-04-12T03:03:38.866 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 6/25 2024-04-12T03:03:39.090 INFO:teuthology.orchestra.run.smithi043.stdout: Upgrading : librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-12T03:03:39.194 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-12T03:03:39.348 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-12T03:03:39.424 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2024-04-12T03:03:39.589 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 9/25 2024-04-12T03:03:39.729 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8. 10/25 2024-04-12T03:03:39.963 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 11/25 2024-04-12T03:03:40.204 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 12/25 2024-04-12T03:03:40.431 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 13/25 2024-04-12T03:03:40.523 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 13/25 2024-04-12T03:03:40.667 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2024-04-12T03:03:40.870 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2024-04-12T03:03:41.108 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-12T03:03:41.318 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-12T03:03:41.727 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2024-04-12T03:03:41.908 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-12T03:03:41.940 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 17/25 2024-04-12T03:03:42.020 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 17/25 2024-04-12T03:03:42.180 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 21/25 2024-04-12T03:03:42.285 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 18/25 2024-04-12T03:03:42.295 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 21/25 2024-04-12T03:03:42.457 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x8 19/25 2024-04-12T03:03:42.463 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-12T03:03:46.542 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-12T03:03:46.764 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-12T03:03:47.424 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 20/25 2024-04-12T03:03:47.687 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 21/25 2024-04-12T03:03:47.786 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 21/25 2024-04-12T03:03:47.975 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-12T03:04:01.828 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-12T03:04:01.828 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /sys 2024-04-12T03:04:01.828 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /proc 2024-04-12T03:04:01.828 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /dev 2024-04-12T03:04:01.828 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /run 2024-04-12T03:04:01.828 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /mnt 2024-04-12T03:04:01.828 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /var/tmp 2024-04-12T03:04:01.828 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /home 2024-04-12T03:04:01.829 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /tmp 2024-04-12T03:04:01.829 INFO:teuthology.orchestra.run.smithi149.stdout:skipping the directory /dev 2024-04-12T03:04:01.829 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:04:03.185 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 23/25 2024-04-12T03:04:03.207 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 23/25 2024-04-12T03:04:03.207 INFO:teuthology.orchestra.run.smithi149.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-12T03:04:03.458 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-12T03:04:03.458 INFO:teuthology.orchestra.run.smithi149.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T03:04:04.191 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T03:04:04.191 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 1/25 2024-04-12T03:04:04.191 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2/25 2024-04-12T03:04:04.191 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-12T03:04:04.192 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 4/25 2024-04-12T03:04:04.192 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 5/25 2024-04-12T03:04:04.192 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 6/25 2024-04-12T03:04:04.192 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-12T03:04:04.192 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8. 8/25 2024-04-12T03:04:04.192 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x8 9/25 2024-04-12T03:04:04.192 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 10/25 2024-04-12T03:04:04.192 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 11/25 2024-04-12T03:04:04.192 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 12/25 2024-04-12T03:04:04.192 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 13/25 2024-04-12T03:04:04.192 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2024-04-12T03:04:04.192 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2024-04-12T03:04:04.192 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-12T03:04:04.193 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2024-04-12T03:04:04.193 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 18/25 2024-04-12T03:04:04.193 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2024-04-12T03:04:04.193 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2024-04-12T03:04:04.193 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2024-04-12T03:04:04.193 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-12T03:04:04.193 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-12T03:04:04.193 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 24/25 2024-04-12T03:04:10.040 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-12T03:04:10.040 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /sys 2024-04-12T03:04:10.040 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /proc 2024-04-12T03:04:10.040 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /dev 2024-04-12T03:04:10.040 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /run 2024-04-12T03:04:10.041 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /mnt 2024-04-12T03:04:10.041 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /var/tmp 2024-04-12T03:04:10.041 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /home 2024-04-12T03:04:10.041 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /tmp 2024-04-12T03:04:10.041 INFO:teuthology.orchestra.run.smithi043.stdout:skipping the directory /dev 2024-04-12T03:04:10.041 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:04:11.500 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 23/25 2024-04-12T03:04:11.527 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 23/25 2024-04-12T03:04:11.527 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-12T03:04:11.780 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-12T03:04:11.780 INFO:teuthology.orchestra.run.smithi043.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T03:04:12.551 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T03:04:12.552 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 1/25 2024-04-12T03:04:12.552 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2/25 2024-04-12T03:04:12.552 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 3/25 2024-04-12T03:04:12.552 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 4/25 2024-04-12T03:04:12.552 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 5/25 2024-04-12T03:04:12.552 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_6 6/25 2024-04-12T03:04:12.552 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 7/25 2024-04-12T03:04:12.552 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8. 8/25 2024-04-12T03:04:12.552 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x8 9/25 2024-04-12T03:04:12.553 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 10/25 2024-04-12T03:04:12.553 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 11/25 2024-04-12T03:04:12.553 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 12/25 2024-04-12T03:04:12.553 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 13/25 2024-04-12T03:04:12.553 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2024-04-12T03:04:12.553 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2024-04-12T03:04:12.553 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-12T03:04:12.553 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2024-04-12T03:04:12.553 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 18/25 2024-04-12T03:04:12.553 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2024-04-12T03:04:12.553 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2024-04-12T03:04:12.553 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2024-04-12T03:04:12.553 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 22/25 2024-04-12T03:04:12.554 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-12T03:04:12.554 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 24/25 2024-04-12T03:04:13.355 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T03:04:13.355 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2024-04-12T03:04:13.930 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:04:13.930 INFO:teuthology.orchestra.run.smithi149.stdout:Upgraded: 2024-04-12T03:04:13.930 INFO:teuthology.orchestra.run.smithi149.stdout: librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:13.930 INFO:teuthology.orchestra.run.smithi149.stdout: librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:13.930 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-12T03:04:13.931 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:13.931 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:13.931 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:13.931 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:13.931 INFO:teuthology.orchestra.run.smithi149.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-12T03:04:13.931 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:13.932 INFO:teuthology.orchestra.run.smithi149.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-12T03:04:13.932 INFO:teuthology.orchestra.run.smithi149.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-12T03:04:13.932 INFO:teuthology.orchestra.run.smithi149.stdout: libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:13.932 INFO:teuthology.orchestra.run.smithi149.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-12T03:04:13.932 INFO:teuthology.orchestra.run.smithi149.stdout: librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:13.932 INFO:teuthology.orchestra.run.smithi149.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-12T03:04:13.933 INFO:teuthology.orchestra.run.smithi149.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-12T03:04:13.933 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:13.933 INFO:teuthology.orchestra.run.smithi149.stdout: python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:13.933 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:13.933 INFO:teuthology.orchestra.run.smithi149.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-12T03:04:13.934 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:13.934 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:13.934 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:13.934 INFO:teuthology.orchestra.run.smithi149.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-12T03:04:13.934 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:04:13.934 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:04:14.138 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-test 2024-04-12T03:04:14.495 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:04:16.368 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 32 kB/s | 4.1 kB 00:00 2024-04-12T03:04:16.758 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 26 kB/s | 4.5 kB 00:00 2024-04-12T03:04:19.120 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:04:19.121 INFO:teuthology.orchestra.run.smithi149.stdout:============================================================================================ 2024-04-12T03:04:19.122 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-04-12T03:04:19.122 INFO:teuthology.orchestra.run.smithi149.stdout:============================================================================================ 2024-04-12T03:04:19.122 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-12T03:04:19.122 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-test x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 47 M 2024-04-12T03:04:19.122 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-04-12T03:04:19.122 INFO:teuthology.orchestra.run.smithi149.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-12T03:04:19.122 INFO:teuthology.orchestra.run.smithi149.stdout: libcephsqlite x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 142 k 2024-04-12T03:04:19.122 INFO:teuthology.orchestra.run.smithi149.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-12T03:04:19.122 INFO:teuthology.orchestra.run.smithi149.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-12T03:04:19.122 INFO:teuthology.orchestra.run.smithi149.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-12T03:04:19.122 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:04:19.122 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-12T03:04:19.123 INFO:teuthology.orchestra.run.smithi149.stdout:============================================================================================ 2024-04-12T03:04:19.123 INFO:teuthology.orchestra.run.smithi149.stdout:Install 6 Packages 2024-04-12T03:04:19.123 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:04:19.123 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 48 M 2024-04-12T03:04:19.123 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 213 M 2024-04-12T03:04:19.123 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-12T03:04:19.157 INFO:teuthology.orchestra.run.smithi149.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.0 MB/s | 70 kB 00:00 2024-04-12T03:04:19.316 INFO:teuthology.orchestra.run.smithi149.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.1 MB/s | 323 kB 00:00 2024-04-12T03:04:19.341 INFO:teuthology.orchestra.run.smithi149.stdout:(3/6): libcephsqlite-18.2.2-1187.gd540ebac.el8. 653 kB/s | 142 kB 00:00 2024-04-12T03:04:19.450 INFO:teuthology.orchestra.run.smithi149.stdout:(4/6): jq-1.6-7.el8.x86_64.rpm 1.5 MB/s | 202 kB 00:00 2024-04-12T03:04:19.544 INFO:teuthology.orchestra.run.smithi149.stdout:(5/6): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 928 kB/s | 188 kB 00:00 2024-04-12T03:04:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-12T03:04:21.310 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2024-04-12T03:04:21.583 INFO:teuthology.orchestra.run.smithi149.stdout:(6/6): ceph-test-18.2.2-1187.gd540ebac.el8.x86_ 19 MB/s | 47 MB 00:02 2024-04-12T03:04:21.585 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:04:21.586 INFO:teuthology.orchestra.run.smithi149.stdout:Total 20 MB/s | 48 MB 00:02 2024-04-12T03:04:21.625 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-12T03:04:21.646 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-12T03:04:21.646 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-12T03:04:21.920 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-12T03:04:21.928 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-12T03:04:21.934 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:04:21.934 INFO:teuthology.orchestra.run.smithi043.stdout:Upgraded: 2024-04-12T03:04:21.934 INFO:teuthology.orchestra.run.smithi043.stdout: librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:21.934 INFO:teuthology.orchestra.run.smithi043.stdout: librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:21.934 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-12T03:04:21.934 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:21.934 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:21.934 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-radosgw-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:21.934 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-selinux-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:21.934 INFO:teuthology.orchestra.run.smithi043.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-12T03:04:21.935 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:21.935 INFO:teuthology.orchestra.run.smithi043.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-12T03:04:21.935 INFO:teuthology.orchestra.run.smithi043.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-12T03:04:21.935 INFO:teuthology.orchestra.run.smithi043.stdout: libradosstriper1-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:21.935 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-12T03:04:21.935 INFO:teuthology.orchestra.run.smithi043.stdout: librgw2-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:21.935 INFO:teuthology.orchestra.run.smithi043.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-12T03:04:21.935 INFO:teuthology.orchestra.run.smithi043.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-12T03:04:21.935 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:21.935 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:21.935 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:21.935 INFO:teuthology.orchestra.run.smithi043.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-12T03:04:21.935 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:21.935 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:21.936 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:21.936 INFO:teuthology.orchestra.run.smithi043.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-12T03:04:21.936 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:04:21.936 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:04:22.160 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-test 2024-04-12T03:04:22.431 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-12T03:04:22.566 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:04:22.596 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-12T03:04:22.678 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-12T03:04:22.914 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : jq-1.6-7.el8.x86_64 2/6 2024-04-12T03:04:23.145 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2024-04-12T03:04:23.366 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2024-04-12T03:04:23.569 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 5/6 2024-04-12T03:04:24.500 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 35 kB/s | 4.1 kB 00:00 2024-04-12T03:04:24.841 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2024-04-12T03:04:27.546 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:04:27.547 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================================ 2024-04-12T03:04:27.547 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-12T03:04:27.547 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================================ 2024-04-12T03:04:27.547 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-12T03:04:27.547 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 47 M 2024-04-12T03:04:27.547 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-12T03:04:27.547 INFO:teuthology.orchestra.run.smithi043.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-12T03:04:27.547 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 142 k 2024-04-12T03:04:27.547 INFO:teuthology.orchestra.run.smithi043.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-12T03:04:27.548 INFO:teuthology.orchestra.run.smithi043.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-12T03:04:27.548 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-12T03:04:27.548 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:04:27.548 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-12T03:04:27.548 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================================ 2024-04-12T03:04:27.548 INFO:teuthology.orchestra.run.smithi043.stdout:Install 6 Packages 2024-04-12T03:04:27.548 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:04:27.549 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 48 M 2024-04-12T03:04:27.549 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 213 M 2024-04-12T03:04:27.549 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-12T03:04:27.583 INFO:teuthology.orchestra.run.smithi043.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.0 MB/s | 70 kB 00:00 2024-04-12T03:04:27.818 INFO:teuthology.orchestra.run.smithi043.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.4 MB/s | 323 kB 00:00 2024-04-12T03:04:27.843 INFO:teuthology.orchestra.run.smithi043.stdout:(3/6): libcephsqlite-18.2.2-1187.gd540ebac.el8. 486 kB/s | 142 kB 00:00 2024-04-12T03:04:27.962 INFO:teuthology.orchestra.run.smithi043.stdout:(4/6): jq-1.6-7.el8.x86_64.rpm 1.4 MB/s | 202 kB 00:00 2024-04-12T03:04:27.988 INFO:teuthology.orchestra.run.smithi043.stdout:(5/6): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-04-12T03:04:28.078 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 5/6 2024-04-12T03:04:28.339 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 6/6 2024-04-12T03:04:28.641 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 6/6 2024-04-12T03:04:28.642 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 1/6 2024-04-12T03:04:28.642 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 2/6 2024-04-12T03:04:28.642 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2024-04-12T03:04:28.642 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2024-04-12T03:04:28.642 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : jq-1.6-7.el8.x86_64 5/6 2024-04-12T03:04:29.809 INFO:teuthology.orchestra.run.smithi043.stdout:(6/6): ceph-test-18.2.2-1187.gd540ebac.el8.x86_ 21 MB/s | 47 MB 00:02 2024-04-12T03:04:29.811 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:04:29.811 INFO:teuthology.orchestra.run.smithi043.stdout:Total 21 MB/s | 48 MB 00:02 2024-04-12T03:04:29.855 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-12T03:04:29.878 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-12T03:04:29.878 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-12T03:04:30.175 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-12T03:04:30.185 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-12T03:04:30.688 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-12T03:04:30.832 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-12T03:04:30.923 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 1/6 2024-04-12T03:04:31.133 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : jq-1.6-7.el8.x86_64 2/6 2024-04-12T03:04:31.355 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2024-04-12T03:04:31.543 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2024-04-12T03:04:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 5/6 2024-04-12T03:04:36.690 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 5/6 2024-04-12T03:04:36.918 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 6/6 2024-04-12T03:04:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 6/6 2024-04-12T03:04:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 1/6 2024-04-12T03:04:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 2/6 2024-04-12T03:04:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2024-04-12T03:04:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2024-04-12T03:04:37.222 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : jq-1.6-7.el8.x86_64 5/6 2024-04-12T03:04:39.849 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 6/6 2024-04-12T03:04:39.849 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2024-04-12T03:04:40.420 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:04:40.420 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-12T03:04:40.420 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:40.421 INFO:teuthology.orchestra.run.smithi149.stdout: jq-1.6-7.el8.x86_64 2024-04-12T03:04:40.421 INFO:teuthology.orchestra.run.smithi149.stdout: libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:40.421 INFO:teuthology.orchestra.run.smithi149.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-12T03:04:40.421 INFO:teuthology.orchestra.run.smithi149.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-12T03:04:40.421 INFO:teuthology.orchestra.run.smithi149.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-12T03:04:40.421 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:04:40.421 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:04:40.598 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph 2024-04-12T03:04:40.940 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:04:42.872 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-12T03:04:43.201 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2024-04-12T03:04:45.578 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:04:45.582 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================= 2024-04-12T03:04:45.582 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-04-12T03:04:45.582 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================= 2024-04-12T03:04:45.582 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-12T03:04:45.582 INFO:teuthology.orchestra.run.smithi149.stdout: ceph x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 6.4 k 2024-04-12T03:04:45.582 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-04-12T03:04:45.582 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mds x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 2.3 M 2024-04-12T03:04:45.583 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 1.6 M 2024-04-12T03:04:45.583 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 245 k 2024-04-12T03:04:45.583 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mon x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 4.8 M 2024-04-12T03:04:45.583 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-osd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 18 M 2024-04-12T03:04:45.583 INFO:teuthology.orchestra.run.smithi149.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-12T03:04:45.583 INFO:teuthology.orchestra.run.smithi149.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-12T03:04:45.583 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-12T03:04:45.583 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-12T03:04:45.583 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-12T03:04:45.583 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-12T03:04:45.583 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-12T03:04:45.583 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-12T03:04:45.583 INFO:teuthology.orchestra.run.smithi149.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-12T03:04:45.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-12T03:04:45.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-12T03:04:45.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-12T03:04:45.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-12T03:04:45.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-12T03:04:45.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-12T03:04:45.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-12T03:04:45.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-12T03:04:45.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-12T03:04:45.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-12T03:04:45.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-12T03:04:45.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-12T03:04:45.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-12T03:04:45.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-12T03:04:45.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-12T03:04:45.585 INFO:teuthology.orchestra.run.smithi149.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-12T03:04:45.585 INFO:teuthology.orchestra.run.smithi149.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-12T03:04:45.585 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:04:45.585 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-12T03:04:45.585 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================= 2024-04-12T03:04:45.585 INFO:teuthology.orchestra.run.smithi149.stdout:Install 32 Packages 2024-04-12T03:04:45.585 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:04:45.586 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 31 M 2024-04-12T03:04:45.586 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 124 M 2024-04-12T03:04:45.586 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-12T03:04:46.044 INFO:teuthology.orchestra.run.smithi149.stdout:(1/32): ceph-18.2.2-1187.gd540ebac.el8.x86_64.r 52 kB/s | 6.4 kB 00:00 2024-04-12T03:04:46.253 INFO:teuthology.orchestra.run.smithi149.stdout:(2/32): ceph-mgr-18.2.2-1187.gd540ebac.el8.x86_ 4.7 MB/s | 1.6 MB 00:00 2024-04-12T03:04:46.311 INFO:teuthology.orchestra.run.smithi149.stdout:(3/32): ceph-mds-18.2.2-1187.gd540ebac.el8.x86_ 5.8 MB/s | 2.3 MB 00:00 2024-04-12T03:04:46.386 INFO:teuthology.orchestra.run.smithi149.stdout:(4/32): ceph-mgr-modules-core-18.2.2-1187.gd540 3.2 MB/s | 245 kB 00:00 2024-04-12T03:04:46.695 INFO:teuthology.orchestra.run.smithi149.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 141 kB/s | 44 kB 00:00 2024-04-12T03:04:46.937 INFO:teuthology.orchestra.run.smithi149.stdout:(6/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 766 kB/s | 185 kB 00:00 2024-04-12T03:04:47.037 INFO:teuthology.orchestra.run.smithi149.stdout:(7/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.7 MB/s | 173 kB 00:00 2024-04-12T03:04:47.146 INFO:teuthology.orchestra.run.smithi149.stdout:(8/32): ceph-mon-18.2.2-1187.gd540ebac.el8.x86_ 4.4 MB/s | 4.8 MB 00:01 2024-04-12T03:04:47.338 INFO:teuthology.orchestra.run.smithi149.stdout:(9/32): ceph-osd-18.2.2-1187.gd540ebac.el8.x86_ 17 MB/s | 18 MB 00:01 2024-04-12T03:04:47.388 INFO:teuthology.orchestra.run.smithi149.stdout:(10/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.1 MB/s | 384 kB 00:00 2024-04-12T03:04:47.430 INFO:teuthology.orchestra.run.smithi149.stdout:(11/32): python3-jaraco-6.2-6.el8.noarch.rpm 39 kB/s | 11 kB 00:00 2024-04-12T03:04:47.480 INFO:teuthology.orchestra.run.smithi149.stdout:(12/32): python3-logutils-0.3.5-11.el8.noarch.r 531 kB/s | 49 kB 00:00 2024-04-12T03:04:47.572 INFO:teuthology.orchestra.run.smithi149.stdout:(13/32): python3-jaraco-functools-2.0-4.el8.noa 75 kB/s | 18 kB 00:00 2024-04-12T03:04:47.597 INFO:teuthology.orchestra.run.smithi149.stdout:(14/32): python3-more-itertools-7.2.0-3.el8.noa 351 kB/s | 59 kB 00:00 2024-04-12T03:04:47.623 INFO:teuthology.orchestra.run.smithi149.stdout:(15/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.9 MB/s | 283 kB 00:00 2024-04-12T03:04:47.690 INFO:teuthology.orchestra.run.smithi149.stdout:(16/32): python3-portend-2.6-1.el8.noarch.rpm 133 kB/s | 16 kB 00:00 2024-04-12T03:04:47.715 INFO:teuthology.orchestra.run.smithi149.stdout:(17/32): python3-simplegeneric-0.8.1-17.el8.noa 160 kB/s | 19 kB 00:00 2024-04-12T03:04:47.740 INFO:teuthology.orchestra.run.smithi149.stdout:(18/32): python3-singledispatch-3.4.0.3-18.el8. 204 kB/s | 24 kB 00:00 2024-04-12T03:04:47.782 INFO:teuthology.orchestra.run.smithi149.stdout:(19/32): python3-tempora-1.14.1-5.el8.noarch.rp 317 kB/s | 29 kB 00:00 2024-04-12T03:04:47.818 INFO:teuthology.orchestra.run.smithi149.stdout:(20/32): python3-trustme-0.6.0-4.el8.noarch.rpm 274 kB/s | 27 kB 00:00 2024-04-12T03:04:47.849 INFO:teuthology.orchestra.run.smithi149.stdout:(21/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.3 MB/s | 255 kB 00:00 2024-04-12T03:04:47.967 INFO:teuthology.orchestra.run.smithi149.stdout:(22/32): python3-zc-lockfile-2.0-2.el8.noarch.r 196 kB/s | 23 kB 00:00 2024-04-12T03:04:48.025 INFO:teuthology.orchestra.run.smithi149.stdout:(23/32): python3-webtest-2.0.33-1.el8.noarch.rp 408 kB/s | 85 kB 00:00 2024-04-12T03:04:48.125 INFO:teuthology.orchestra.run.smithi149.stdout:(24/32): python3-pycparser-2.14-14.el8.noarch.r 688 kB/s | 109 kB 00:00 2024-04-12T03:04:48.184 INFO:teuthology.orchestra.run.smithi149.stdout:(25/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 625 kB/s | 251 kB 00:00 2024-04-12T03:04:48.243 INFO:teuthology.orchestra.run.smithi149.stdout:(26/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.1 MB/s | 238 kB 00:00 2024-04-12T03:04:48.293 INFO:teuthology.orchestra.run.smithi149.stdout:(27/32): python3-cryptography-3.2.1-7.el8_9.x86 3.3 MB/s | 559 kB 00:00 2024-04-12T03:04:48.352 INFO:teuthology.orchestra.run.smithi149.stdout:(28/32): python3-markupsafe-0.23-19.el8.x86_64. 377 kB/s | 39 kB 00:00 2024-04-12T03:04:48.390 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 6/6 2024-04-12T03:04:48.390 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2024-04-12T03:04:48.402 INFO:teuthology.orchestra.run.smithi149.stdout:(29/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.1 MB/s | 457 kB 00:00 2024-04-12T03:04:48.428 INFO:teuthology.orchestra.run.smithi149.stdout:(30/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 772 kB/s | 103 kB 00:00 2024-04-12T03:04:48.519 INFO:teuthology.orchestra.run.smithi149.stdout:(31/32): python3-pytz-2017.2-11.el8.noarch.rpm 464 kB/s | 54 kB 00:00 2024-04-12T03:04:48.544 INFO:teuthology.orchestra.run.smithi149.stdout:(32/32): python3-mako-1.0.6-14.el8.noarch.rpm 817 kB/s | 157 kB 00:00 2024-04-12T03:04:48.545 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:04:48.546 INFO:teuthology.orchestra.run.smithi149.stdout:Total 10 MB/s | 31 MB 00:02 2024-04-12T03:04:48.640 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-12T03:04:48.677 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-12T03:04:48.677 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-12T03:04:48.905 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-12T03:04:48.921 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-12T03:04:48.990 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:04:48.990 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-12T03:04:48.990 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-test-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:48.990 INFO:teuthology.orchestra.run.smithi043.stdout: jq-1.6-7.el8.x86_64 2024-04-12T03:04:48.990 INFO:teuthology.orchestra.run.smithi043.stdout: libcephsqlite-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:04:48.991 INFO:teuthology.orchestra.run.smithi043.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-12T03:04:48.991 INFO:teuthology.orchestra.run.smithi043.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-12T03:04:48.991 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-12T03:04:48.991 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:04:48.991 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:04:49.186 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph 2024-04-12T03:04:49.576 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:04:50.078 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-12T03:04:50.329 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-12T03:04:50.570 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-12T03:04:50.718 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 3/32 2024-04-12T03:04:50.951 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2024-04-12T03:04:51.329 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 5/32 2024-04-12T03:04:51.586 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/32 2024-04-12T03:04:51.639 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 24 kB/s | 4.1 kB 00:00 2024-04-12T03:04:51.817 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2024-04-12T03:04:52.014 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 32 kB/s | 4.5 kB 00:00 2024-04-12T03:04:52.173 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 8/32 2024-04-12T03:04:52.484 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 9/32 2024-04-12T03:04:52.681 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2024-04-12T03:04:52.849 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2024-04-12T03:04:53.122 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2024-04-12T03:04:53.368 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2024-04-12T03:04:53.605 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2024-04-12T03:04:53.761 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2024-04-12T03:04:53.984 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2024-04-12T03:04:54.168 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2024-04-12T03:04:54.321 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2024-04-12T03:04:54.562 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2024-04-12T03:04:54.785 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2024-04-12T03:04:54.822 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:04:54.826 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================================= 2024-04-12T03:04:54.826 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-12T03:04:54.826 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================================= 2024-04-12T03:04:54.826 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-12T03:04:54.826 INFO:teuthology.orchestra.run.smithi043.stdout: ceph x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 6.4 k 2024-04-12T03:04:54.826 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-12T03:04:54.826 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 2.3 M 2024-04-12T03:04:54.827 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 1.6 M 2024-04-12T03:04:54.827 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 245 k 2024-04-12T03:04:54.827 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 4.8 M 2024-04-12T03:04:54.827 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 18 M 2024-04-12T03:04:54.827 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-12T03:04:54.827 INFO:teuthology.orchestra.run.smithi043.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-12T03:04:54.827 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-12T03:04:54.827 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-12T03:04:54.827 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-12T03:04:54.827 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-12T03:04:54.827 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-12T03:04:54.827 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-12T03:04:54.827 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-12T03:04:54.828 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-12T03:04:54.828 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-12T03:04:54.828 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-12T03:04:54.828 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-12T03:04:54.828 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-12T03:04:54.828 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-12T03:04:54.828 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-12T03:04:54.828 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-12T03:04:54.828 INFO:teuthology.orchestra.run.smithi043.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-12T03:04:54.828 INFO:teuthology.orchestra.run.smithi043.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-12T03:04:54.828 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-12T03:04:54.828 INFO:teuthology.orchestra.run.smithi043.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-12T03:04:54.829 INFO:teuthology.orchestra.run.smithi043.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-12T03:04:54.829 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-12T03:04:54.829 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-12T03:04:54.829 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-12T03:04:54.829 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-12T03:04:54.829 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:04:54.829 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-12T03:04:54.829 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================================= 2024-04-12T03:04:54.829 INFO:teuthology.orchestra.run.smithi043.stdout:Install 32 Packages 2024-04-12T03:04:54.829 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:04:54.830 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 31 M 2024-04-12T03:04:54.830 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 124 M 2024-04-12T03:04:54.830 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-12T03:04:55.003 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-12T03:04:55.087 INFO:teuthology.orchestra.run.smithi043.stdout:(1/32): ceph-18.2.2-1187.gd540ebac.el8.x86_64.r 50 kB/s | 6.4 kB 00:00 2024-04-12T03:04:55.229 INFO:teuthology.orchestra.run.smithi043.stdout:(2/32): ceph-mgr-18.2.2-1187.gd540ebac.el8.x86_ 5.8 MB/s | 1.6 MB 00:00 2024-04-12T03:04:55.262 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-12T03:04:55.511 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-12T03:04:55.721 INFO:teuthology.orchestra.run.smithi043.stdout:(3/32): ceph-mon-18.2.2-1187.gd540ebac.el8.x86_ 7.6 MB/s | 4.8 MB 00:00 2024-04-12T03:04:55.756 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2024-04-12T03:04:55.821 INFO:teuthology.orchestra.run.smithi043.stdout:(4/32): ceph-mgr-modules-core-18.2.2-1187.gd540 2.4 MB/s | 245 kB 00:00 2024-04-12T03:04:55.930 INFO:teuthology.orchestra.run.smithi043.stdout:(5/32): ceph-mds-18.2.2-1187.gd540ebac.el8.x86_ 2.3 MB/s | 2.3 MB 00:00 2024-04-12T03:04:55.980 INFO:teuthology.orchestra.run.smithi043.stdout:(6/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 275 kB/s | 44 kB 00:00 2024-04-12T03:04:56.067 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2024-04-12T03:04:56.105 INFO:teuthology.orchestra.run.smithi043.stdout:(7/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.4 MB/s | 173 kB 00:00 2024-04-12T03:04:56.297 INFO:teuthology.orchestra.run.smithi043.stdout:(8/32): ceph-osd-18.2.2-1187.gd540ebac.el8.x86_ 17 MB/s | 18 MB 00:01 2024-04-12T03:04:56.314 INFO:teuthology.orchestra.run.smithi043.stdout:(9/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 482 kB/s | 185 kB 00:00 2024-04-12T03:04:56.347 INFO:teuthology.orchestra.run.smithi043.stdout:(10/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.6 MB/s | 384 kB 00:00 2024-04-12T03:04:56.373 INFO:teuthology.orchestra.run.smithi043.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 301 kB/s | 18 kB 00:00 2024-04-12T03:04:56.406 INFO:teuthology.orchestra.run.smithi043.stdout:(12/32): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2024-04-12T03:04:56.432 INFO:teuthology.orchestra.run.smithi043.stdout:(13/32): python3-logutils-0.3.5-11.el8.noarch.r 582 kB/s | 49 kB 00:00 2024-04-12T03:04:56.438 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-12T03:04:56.457 INFO:teuthology.orchestra.run.smithi043.stdout:(14/32): python3-more-itertools-7.2.0-3.el8.noa 700 kB/s | 59 kB 00:00 2024-04-12T03:04:56.524 INFO:teuthology.orchestra.run.smithi043.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 170 kB/s | 16 kB 00:00 2024-04-12T03:04:56.549 INFO:teuthology.orchestra.run.smithi043.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 203 kB/s | 19 kB 00:00 2024-04-12T03:04:56.591 INFO:teuthology.orchestra.run.smithi043.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 359 kB/s | 24 kB 00:00 2024-04-12T03:04:56.616 INFO:teuthology.orchestra.run.smithi043.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 437 kB/s | 29 kB 00:00 2024-04-12T03:04:56.650 INFO:teuthology.orchestra.run.smithi043.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 470 kB/s | 27 kB 00:00 2024-04-12T03:04:56.675 INFO:teuthology.orchestra.run.smithi043.stdout:(20/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.0 MB/s | 283 kB 00:00 2024-04-12T03:04:56.709 INFO:teuthology.orchestra.run.smithi043.stdout:(21/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2024-04-12T03:04:56.742 INFO:teuthology.orchestra.run.smithi043.stdout:(22/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.7 MB/s | 251 kB 00:00 2024-04-12T03:04:56.768 INFO:teuthology.orchestra.run.smithi043.stdout:(23/32): python3-webtest-2.0.33-1.el8.noarch.rp 925 kB/s | 85 kB 00:00 2024-04-12T03:04:56.793 INFO:teuthology.orchestra.run.smithi043.stdout:(24/32): python3-zc-lockfile-2.0-2.el8.noarch.r 274 kB/s | 23 kB 00:00 2024-04-12T03:04:56.815 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8. 27/32 2024-04-12T03:04:56.968 INFO:teuthology.orchestra.run.smithi043.stdout:(25/32): python3-pycparser-2.14-14.el8.noarch.r 484 kB/s | 109 kB 00:00 2024-04-12T03:04:57.010 INFO:teuthology.orchestra.run.smithi043.stdout:(26/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 983 kB/s | 238 kB 00:00 2024-04-12T03:04:57.027 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 28/32 2024-04-12T03:04:57.035 INFO:teuthology.orchestra.run.smithi043.stdout:(27/32): python3-cryptography-3.2.1-7.el8_9.x86 2.4 MB/s | 559 kB 00:00 2024-04-12T03:04:57.086 INFO:teuthology.orchestra.run.smithi043.stdout:(28/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.8 MB/s | 457 kB 00:00 2024-04-12T03:04:57.111 INFO:teuthology.orchestra.run.smithi043.stdout:(29/32): python3-markupsafe-0.23-19.el8.x86_64. 385 kB/s | 39 kB 00:00 2024-04-12T03:04:57.137 INFO:teuthology.orchestra.run.smithi043.stdout:(30/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 1.0 MB/s | 103 kB 00:00 2024-04-12T03:04:57.197 INFO:teuthology.orchestra.run.smithi043.stdout:(31/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.4 MB/s | 157 kB 00:00 2024-04-12T03:04:57.245 INFO:teuthology.orchestra.run.smithi043.stdout:(32/32): python3-pytz-2017.2-11.el8.noarch.rpm 406 kB/s | 54 kB 00:00 2024-04-12T03:04:57.247 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:04:57.247 INFO:teuthology.orchestra.run.smithi043.stdout:Total 13 MB/s | 31 MB 00:02 2024-04-12T03:04:57.354 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-12T03:04:57.396 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-12T03:04:57.396 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-12T03:04:57.650 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-12T03:04:57.660 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-12T03:04:58.673 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 28/32 2024-04-12T03:04:58.780 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-12T03:04:59.009 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-12T03:04:59.220 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-12T03:04:59.277 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 29/32 2024-04-12T03:04:59.359 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 3/32 2024-04-12T03:04:59.592 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2024-04-12T03:04:59.760 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 29/32 2024-04-12T03:04:59.898 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 5/32 2024-04-12T03:04:59.981 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 30/32 2024-04-12T03:05:00.163 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/32 2024-04-12T03:05:00.196 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 30/32 2024-04-12T03:05:00.363 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2024-04-12T03:05:00.397 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 31/32 2024-04-12T03:05:00.416 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 31/32 2024-04-12T03:05:00.554 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 32/32 2024-04-12T03:05:00.783 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 8/32 2024-04-12T03:05:00.973 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 32/32 2024-04-12T03:05:00.974 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 1/32 2024-04-12T03:05:00.974 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 2/32 2024-04-12T03:05:00.974 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 3/32 2024-04-12T03:05:00.974 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 4/32 2024-04-12T03:05:00.974 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 5/32 2024-04-12T03:05:00.974 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8. 6/32 2024-04-12T03:05:00.974 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2024-04-12T03:05:00.974 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2024-04-12T03:05:00.974 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2024-04-12T03:05:00.974 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2024-04-12T03:05:00.974 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2024-04-12T03:05:00.974 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2024-04-12T03:05:00.975 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2024-04-12T03:05:00.975 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2024-04-12T03:05:00.975 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2024-04-12T03:05:00.975 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2024-04-12T03:05:00.975 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2024-04-12T03:05:00.975 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2024-04-12T03:05:00.975 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2024-04-12T03:05:00.975 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2024-04-12T03:05:00.975 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2024-04-12T03:05:00.975 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2024-04-12T03:05:00.975 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2024-04-12T03:05:00.975 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2024-04-12T03:05:00.975 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2024-04-12T03:05:00.975 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 26/32 2024-04-12T03:05:00.976 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 27/32 2024-04-12T03:05:00.976 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2024-04-12T03:05:00.976 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2024-04-12T03:05:00.976 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 30/32 2024-04-12T03:05:00.976 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 31/32 2024-04-12T03:05:01.135 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 9/32 2024-04-12T03:05:01.295 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2024-04-12T03:05:01.455 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2024-04-12T03:05:01.747 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2024-04-12T03:05:01.965 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2024-04-12T03:05:02.177 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2024-04-12T03:05:02.349 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2024-04-12T03:05:02.583 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2024-04-12T03:05:02.764 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2024-04-12T03:05:02.925 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2024-04-12T03:05:03.146 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2024-04-12T03:05:03.340 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2024-04-12T03:05:03.525 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-12T03:05:03.764 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-12T03:05:04.072 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-12T03:05:04.307 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2024-04-12T03:05:04.608 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2024-04-12T03:05:04.972 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-12T03:05:05.336 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8. 27/32 2024-04-12T03:05:05.533 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 28/32 2024-04-12T03:05:07.346 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 28/32 2024-04-12T03:05:07.707 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 29/32 2024-04-12T03:05:08.240 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 29/32 2024-04-12T03:05:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 30/32 2024-04-12T03:05:08.648 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 30/32 2024-04-12T03:05:08.826 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 31/32 2024-04-12T03:05:08.849 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 31/32 2024-04-12T03:05:08.909 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 32/32 2024-04-12T03:05:08.909 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2024-04-12T03:05:08.959 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 32/32 2024-04-12T03:05:09.395 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 32/32 2024-04-12T03:05:09.395 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 1/32 2024-04-12T03:05:09.395 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 2/32 2024-04-12T03:05:09.395 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 3/32 2024-04-12T03:05:09.395 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 4/32 2024-04-12T03:05:09.396 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 5/32 2024-04-12T03:05:09.396 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8. 6/32 2024-04-12T03:05:09.396 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2024-04-12T03:05:09.396 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2024-04-12T03:05:09.396 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2024-04-12T03:05:09.396 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2024-04-12T03:05:09.396 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2024-04-12T03:05:09.396 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2024-04-12T03:05:09.396 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2024-04-12T03:05:09.396 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2024-04-12T03:05:09.396 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2024-04-12T03:05:09.396 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2024-04-12T03:05:09.396 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2024-04-12T03:05:09.397 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2024-04-12T03:05:09.397 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2024-04-12T03:05:09.397 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2024-04-12T03:05:09.397 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2024-04-12T03:05:09.397 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2024-04-12T03:05:09.397 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2024-04-12T03:05:09.397 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2024-04-12T03:05:09.397 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2024-04-12T03:05:09.397 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 26/32 2024-04-12T03:05:09.397 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 27/32 2024-04-12T03:05:09.397 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2024-04-12T03:05:09.397 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2024-04-12T03:05:09.397 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 30/32 2024-04-12T03:05:09.398 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 31/32 2024-04-12T03:05:09.468 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:05:09.468 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-12T03:05:09.468 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:05:09.468 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:05:09.468 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:05:09.468 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:05:09.468 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:05:09.468 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:05:09.468 INFO:teuthology.orchestra.run.smithi149.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-12T03:05:09.468 INFO:teuthology.orchestra.run.smithi149.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-12T03:05:09.468 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-12T03:05:09.469 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-12T03:05:09.469 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-12T03:05:09.469 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-12T03:05:09.469 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-12T03:05:09.469 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-12T03:05:09.469 INFO:teuthology.orchestra.run.smithi149.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-12T03:05:09.469 INFO:teuthology.orchestra.run.smithi149.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-12T03:05:09.469 INFO:teuthology.orchestra.run.smithi149.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-12T03:05:09.469 INFO:teuthology.orchestra.run.smithi149.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-12T03:05:09.469 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-12T03:05:09.469 INFO:teuthology.orchestra.run.smithi149.stdout: python3-portend-2.6-1.el8.noarch 2024-04-12T03:05:09.470 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-12T03:05:09.470 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-12T03:05:09.470 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-12T03:05:09.470 INFO:teuthology.orchestra.run.smithi149.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-12T03:05:09.470 INFO:teuthology.orchestra.run.smithi149.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-12T03:05:09.470 INFO:teuthology.orchestra.run.smithi149.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-12T03:05:09.470 INFO:teuthology.orchestra.run.smithi149.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-12T03:05:09.470 INFO:teuthology.orchestra.run.smithi149.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-12T03:05:09.470 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-12T03:05:09.470 INFO:teuthology.orchestra.run.smithi149.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-12T03:05:09.470 INFO:teuthology.orchestra.run.smithi149.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-12T03:05:09.470 INFO:teuthology.orchestra.run.smithi149.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-12T03:05:09.470 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:05:09.470 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:05:09.650 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-base 2024-04-12T03:05:10.003 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:05:12.115 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-12T03:05:12.463 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-12T03:05:14.637 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:05:14.828 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:05:14.829 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-04-12T03:05:14.829 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:05:14.994 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install cephadm 2024-04-12T03:05:15.343 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:05:17.279 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-12T03:05:17.639 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-04-12T03:05:19.588 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 32/32 2024-04-12T03:05:19.588 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2024-04-12T03:05:20.010 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:05:20.011 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:05:20.011 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-04-12T03:05:20.011 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:05:20.011 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-12T03:05:20.011 INFO:teuthology.orchestra.run.smithi149.stdout: cephadm noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 170 k 2024-04-12T03:05:20.011 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:05:20.011 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-12T03:05:20.012 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:05:20.012 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2024-04-12T03:05:20.012 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:05:20.012 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 170 k 2024-04-12T03:05:20.012 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 692 k 2024-04-12T03:05:20.012 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-12T03:05:20.189 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:05:20.189 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-12T03:05:20.189 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:05:20.189 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mds-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:05:20.189 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:05:20.189 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:05:20.190 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mon-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:05:20.190 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-osd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:05:20.190 INFO:teuthology.orchestra.run.smithi043.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-12T03:05:20.190 INFO:teuthology.orchestra.run.smithi043.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-12T03:05:20.190 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-12T03:05:20.190 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-12T03:05:20.190 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-12T03:05:20.190 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-12T03:05:20.190 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-12T03:05:20.190 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-12T03:05:20.190 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-12T03:05:20.190 INFO:teuthology.orchestra.run.smithi043.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-12T03:05:20.190 INFO:teuthology.orchestra.run.smithi043.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-12T03:05:20.191 INFO:teuthology.orchestra.run.smithi043.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-12T03:05:20.191 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-12T03:05:20.191 INFO:teuthology.orchestra.run.smithi043.stdout: python3-portend-2.6-1.el8.noarch 2024-04-12T03:05:20.191 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-12T03:05:20.191 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-12T03:05:20.191 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-12T03:05:20.191 INFO:teuthology.orchestra.run.smithi043.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-12T03:05:20.191 INFO:teuthology.orchestra.run.smithi043.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-12T03:05:20.191 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-12T03:05:20.191 INFO:teuthology.orchestra.run.smithi043.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-12T03:05:20.191 INFO:teuthology.orchestra.run.smithi043.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-12T03:05:20.191 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-12T03:05:20.191 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-12T03:05:20.191 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-12T03:05:20.192 INFO:teuthology.orchestra.run.smithi043.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-12T03:05:20.192 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:05:20.192 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:05:20.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-base 2024-04-12T03:05:20.584 INFO:teuthology.orchestra.run.smithi149.stdout:cephadm-18.2.2-1187.gd540ebac.el8.noarch.rpm 298 kB/s | 170 kB 00:00 2024-04-12T03:05:20.584 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:05:20.585 INFO:teuthology.orchestra.run.smithi149.stdout:Total 297 kB/s | 170 kB 00:00 2024-04-12T03:05:20.585 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-12T03:05:20.593 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-12T03:05:20.593 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-12T03:05:20.596 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-12T03:05:20.604 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-12T03:05:20.648 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-12T03:05:20.805 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:05:22.511 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-12T03:05:22.672 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-12T03:05:22.702 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-12T03:05:22.917 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-12T03:05:22.960 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 84 kB/s | 4.5 kB 00:00 2024-04-12T03:05:25.556 INFO:teuthology.orchestra.run.smithi043.stdout:Package ceph-base-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:05:25.762 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:05:25.763 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-12T03:05:25.763 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:05:25.933 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install cephadm 2024-04-12T03:05:26.332 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:05:28.342 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-12T03:05:28.738 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-12T03:05:31.490 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:05:31.491 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:05:31.491 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-12T03:05:31.491 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:05:31.491 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-12T03:05:31.491 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 170 k 2024-04-12T03:05:31.491 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:05:31.491 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-12T03:05:31.491 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:05:31.491 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-04-12T03:05:31.492 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:05:31.492 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 170 k 2024-04-12T03:05:31.492 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 692 k 2024-04-12T03:05:31.492 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-12T03:05:31.737 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-12T03:05:31.737 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2024-04-12T03:05:31.799 INFO:teuthology.orchestra.run.smithi043.stdout:cephadm-18.2.2-1187.gd540ebac.el8.noarch.rpm 556 kB/s | 170 kB 00:00 2024-04-12T03:05:31.799 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:05:31.799 INFO:teuthology.orchestra.run.smithi043.stdout:Total 553 kB/s | 170 kB 00:00 2024-04-12T03:05:31.800 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-12T03:05:31.810 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-12T03:05:31.810 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-12T03:05:31.813 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-12T03:05:31.823 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-12T03:05:31.867 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-12T03:05:32.297 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:05:32.297 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-12T03:05:32.297 INFO:teuthology.orchestra.run.smithi149.stdout: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:05:32.297 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:05:32.297 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:05:32.489 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-immutable-object-cache 2024-04-12T03:05:32.855 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:05:34.131 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-12T03:05:34.291 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-12T03:05:34.528 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-12T03:05:34.723 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-12T03:05:35.088 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-04-12T03:05:37.505 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:05:37.505 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:05:37.505 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repo Size 2024-04-12T03:05:37.505 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:05:37.505 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-12T03:05:37.506 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 165 k 2024-04-12T03:05:37.506 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:05:37.506 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-12T03:05:37.506 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:05:37.506 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2024-04-12T03:05:37.506 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:05:37.507 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 165 k 2024-04-12T03:05:37.507 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 503 k 2024-04-12T03:05:37.507 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-12T03:05:37.909 INFO:teuthology.orchestra.run.smithi149.stdout:ceph-immutable-object-cache-18.2.2-1187.gd540eb 411 kB/s | 165 kB 00:00 2024-04-12T03:05:37.910 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:05:37.910 INFO:teuthology.orchestra.run.smithi149.stdout:Total 410 kB/s | 165 kB 00:00 2024-04-12T03:05:37.910 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-12T03:05:37.922 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-12T03:05:37.922 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-12T03:05:37.976 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-12T03:05:37.985 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-12T03:05:38.115 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-12T03:05:38.300 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-12T03:05:38.709 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-12T03:05:43.372 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/1 2024-04-12T03:05:43.372 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2024-04-12T03:05:43.994 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:05:43.995 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-12T03:05:43.995 INFO:teuthology.orchestra.run.smithi043.stdout: cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:05:43.995 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:05:43.995 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:05:44.184 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-immutable-object-cache 2024-04-12T03:05:44.601 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:05:46.672 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-12T03:05:47.011 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 40 kB/s | 4.5 kB 00:00 2024-04-12T03:05:49.751 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:05:49.752 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:05:49.752 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2024-04-12T03:05:49.753 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:05:49.753 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-12T03:05:49.753 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 165 k 2024-04-12T03:05:49.753 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:05:49.753 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-12T03:05:49.753 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:05:49.753 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-04-12T03:05:49.753 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:05:49.754 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 165 k 2024-04-12T03:05:49.754 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 503 k 2024-04-12T03:05:49.754 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-12T03:05:50.231 INFO:teuthology.orchestra.run.smithi043.stdout:ceph-immutable-object-cache-18.2.2-1187.gd540eb 347 kB/s | 165 kB 00:00 2024-04-12T03:05:50.232 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:05:50.232 INFO:teuthology.orchestra.run.smithi043.stdout:Total 345 kB/s | 165 kB 00:00 2024-04-12T03:05:50.233 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-12T03:05:50.247 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-12T03:05:50.247 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-12T03:05:50.313 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-12T03:05:50.323 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-12T03:05:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-12T03:05:50.628 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-12T03:05:50.710 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-12T03:05:50.710 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2024-04-12T03:05:51.067 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-12T03:05:51.274 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:05:51.274 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-12T03:05:51.274 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:05:51.274 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:05:51.274 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:05:51.457 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-mgr 2024-04-12T03:05:51.811 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:05:53.489 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-12T03:05:53.803 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 42 kB/s | 4.5 kB 00:00 2024-04-12T03:05:55.974 INFO:teuthology.orchestra.run.smithi149.stdout:Package ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:05:56.162 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:05:56.164 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-04-12T03:05:56.164 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:05:56.432 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-mgr-dashboard 2024-04-12T03:05:56.781 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:05:58.682 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 34 kB/s | 4.1 kB 00:00 2024-04-12T03:05:59.112 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 23 kB/s | 4.5 kB 00:00 2024-04-12T03:06:01.493 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:06:01.494 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:06:01.494 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-04-12T03:06:01.494 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:06:01.494 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-12T03:06:01.494 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 3.5 M 2024-04-12T03:06:01.494 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-04-12T03:06:01.494 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 24 k 2024-04-12T03:06:01.494 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 15 k 2024-04-12T03:06:01.494 INFO:teuthology.orchestra.run.smithi149.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-12T03:06:01.494 INFO:teuthology.orchestra.run.smithi149.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-12T03:06:01.494 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:06:01.495 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-12T03:06:01.495 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:06:01.495 INFO:teuthology.orchestra.run.smithi149.stdout:Install 5 Packages 2024-04-12T03:06:01.495 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:06:01.495 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 3.8 M 2024-04-12T03:06:01.495 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 87 M 2024-04-12T03:06:01.496 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-12T03:06:02.113 INFO:teuthology.orchestra.run.smithi149.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1187.gd54 105 kB/s | 24 kB 00:00 2024-04-12T03:06:02.129 INFO:teuthology.orchestra.run.smithi149.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1187.gd540 59 kB/s | 15 kB 00:00 2024-04-12T03:06:02.230 INFO:teuthology.orchestra.run.smithi149.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 288 kB/s | 34 kB 00:00 2024-04-12T03:06:02.305 INFO:teuthology.orchestra.run.smithi149.stdout:(4/5): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2024-04-12T03:06:02.364 INFO:teuthology.orchestra.run.smithi149.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1187.gd540ebac 7.3 MB/s | 3.5 MB 00:00 2024-04-12T03:06:02.365 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:06:02.366 INFO:teuthology.orchestra.run.smithi149.stdout:Total 4.4 MB/s | 3.8 MB 00:00 2024-04-12T03:06:02.366 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-12T03:06:02.374 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-12T03:06:02.375 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-12T03:06:02.448 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-12T03:06:02.456 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-12T03:06:02.717 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-12T03:06:02.976 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-12T03:06:03.209 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-12T03:06:03.343 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.n 3/5 2024-04-12T03:06:04.544 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8. 4/5 2024-04-12T03:06:04.766 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 5/5 2024-04-12T03:06:04.826 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac. 1/1 2024-04-12T03:06:04.826 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2024-04-12T03:06:05.069 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 5/5 2024-04-12T03:06:05.069 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8. 1/5 2024-04-12T03:06:05.069 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 2/5 2024-04-12T03:06:05.069 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.n 3/5 2024-04-12T03:06:05.069 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-12T03:06:05.446 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:06:05.446 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-12T03:06:05.446 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-immutable-object-cache-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:06:05.446 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:06:05.447 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:06:05.811 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr 2024-04-12T03:06:06.213 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:06:08.121 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 34 kB/s | 4.1 kB 00:00 2024-04-12T03:06:08.494 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 32 kB/s | 4.5 kB 00:00 2024-04-12T03:06:11.024 INFO:teuthology.orchestra.run.smithi043.stdout:Package ceph-mgr-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:06:11.238 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:06:11.239 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-12T03:06:11.239 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:06:11.394 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr-dashboard 2024-04-12T03:06:11.795 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:06:13.776 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-12T03:06:14.152 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 30 kB/s | 4.5 kB 00:00 2024-04-12T03:06:15.844 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-12T03:06:15.844 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2024-04-12T03:06:16.410 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:06:16.411 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-12T03:06:16.411 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:06:16.411 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:06:16.411 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:06:16.411 INFO:teuthology.orchestra.run.smithi149.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-12T03:06:16.411 INFO:teuthology.orchestra.run.smithi149.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-12T03:06:16.411 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:06:16.411 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:06:16.606 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-12T03:06:16.896 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:06:16.897 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:06:16.897 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-12T03:06:16.897 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:06:16.897 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-12T03:06:16.897 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 3.5 M 2024-04-12T03:06:16.897 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-12T03:06:16.897 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 24 k 2024-04-12T03:06:16.898 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 15 k 2024-04-12T03:06:16.898 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-12T03:06:16.898 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-12T03:06:16.898 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:06:16.898 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-12T03:06:16.898 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:06:16.898 INFO:teuthology.orchestra.run.smithi043.stdout:Install 5 Packages 2024-04-12T03:06:16.898 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:06:16.899 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 3.8 M 2024-04-12T03:06:16.899 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 87 M 2024-04-12T03:06:16.899 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-12T03:06:16.959 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:06:17.201 INFO:teuthology.orchestra.run.smithi043.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1187.gd54 165 kB/s | 24 kB 00:00 2024-04-12T03:06:17.309 INFO:teuthology.orchestra.run.smithi043.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1187.gd540 58 kB/s | 15 kB 00:00 2024-04-12T03:06:17.476 INFO:teuthology.orchestra.run.smithi043.stdout:(3/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 122 kB/s | 34 kB 00:00 2024-04-12T03:06:17.643 INFO:teuthology.orchestra.run.smithi043.stdout:(4/5): ceph-mgr-dashboard-18.2.2-1187.gd540ebac 6.0 MB/s | 3.5 MB 00:00 2024-04-12T03:06:17.777 INFO:teuthology.orchestra.run.smithi043.stdout:(5/5): python3-routes-2.4.1-12.el8.noarch.rpm 421 kB/s | 196 kB 00:00 2024-04-12T03:06:17.778 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:06:17.779 INFO:teuthology.orchestra.run.smithi043.stdout:Total 4.3 MB/s | 3.8 MB 00:00 2024-04-12T03:06:17.779 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-12T03:06:17.789 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-12T03:06:17.789 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-12T03:06:17.871 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-12T03:06:17.881 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-12T03:06:18.117 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-12T03:06:18.335 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2024-04-12T03:06:18.548 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2024-04-12T03:06:18.698 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.n 3/5 2024-04-12T03:06:18.875 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-12T03:06:19.219 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-04-12T03:06:19.965 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8. 4/5 2024-04-12T03:06:20.178 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 5/5 2024-04-12T03:06:20.487 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 5/5 2024-04-12T03:06:20.487 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8. 1/5 2024-04-12T03:06:20.487 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarc 2/5 2024-04-12T03:06:20.487 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.n 3/5 2024-04-12T03:06:20.487 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2024-04-12T03:06:21.582 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:06:21.583 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================================= 2024-04-12T03:06:21.583 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-04-12T03:06:21.583 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================================= 2024-04-12T03:06:21.583 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-12T03:06:21.583 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 7.4 M 2024-04-12T03:06:21.584 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-04-12T03:06:21.584 INFO:teuthology.orchestra.run.smithi149.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-12T03:06:21.584 INFO:teuthology.orchestra.run.smithi149.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-12T03:06:21.584 INFO:teuthology.orchestra.run.smithi149.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-12T03:06:21.584 INFO:teuthology.orchestra.run.smithi149.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-12T03:06:21.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-12T03:06:21.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-12T03:06:21.584 INFO:teuthology.orchestra.run.smithi149.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-12T03:06:21.584 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:06:21.584 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-12T03:06:21.584 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================================= 2024-04-12T03:06:21.584 INFO:teuthology.orchestra.run.smithi149.stdout:Install 8 Packages 2024-04-12T03:06:21.585 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:06:21.585 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 36 M 2024-04-12T03:06:21.585 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 207 M 2024-04-12T03:06:21.585 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-12T03:06:21.703 INFO:teuthology.orchestra.run.smithi149.stdout:(1/8): libgfortran-8.5.0-20.el8.x86_64.rpm 5.4 MB/s | 645 kB 00:00 2024-04-12T03:06:21.728 INFO:teuthology.orchestra.run.smithi149.stdout:(2/8): libquadmath-8.5.0-20.el8.x86_64.rpm 1.2 MB/s | 172 kB 00:00 2024-04-12T03:06:21.870 INFO:teuthology.orchestra.run.smithi149.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 22 MB/s | 3.7 MB 00:00 2024-04-12T03:06:21.895 INFO:teuthology.orchestra.run.smithi149.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.3 MB/s | 225 kB 00:00 2024-04-12T03:06:22.154 INFO:teuthology.orchestra.run.smithi149.stdout:(5/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 18 MB/s | 5.2 MB 00:00 2024-04-12T03:06:22.355 INFO:teuthology.orchestra.run.smithi149.stdout:(6/8): openblas-0.3.15-6.el8.x86_64.rpm 11 MB/s | 5.0 MB 00:00 2024-04-12T03:06:22.647 INFO:teuthology.orchestra.run.smithi149.stdout:(7/8): python3-scipy-1.0.0-21.module+el8.9.0+20 28 MB/s | 14 MB 00:00 2024-04-12T03:06:22.747 INFO:teuthology.orchestra.run.smithi149.stdout:(8/8): ceph-mgr-diskprediction-local-18.2.2-118 6.4 MB/s | 7.4 MB 00:01 2024-04-12T03:06:22.748 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:06:22.748 INFO:teuthology.orchestra.run.smithi149.stdout:Total 31 MB/s | 36 MB 00:01 2024-04-12T03:06:22.995 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-12T03:06:23.013 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-12T03:06:23.014 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-12T03:06:23.328 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-12T03:06:23.337 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-12T03:06:23.933 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-12T03:06:24.100 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-12T03:06:24.226 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-12T03:06:24.409 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-12T03:06:25.022 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-12T03:06:25.192 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-12T03:06:25.778 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-12T03:06:25.942 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2024-04-12T03:06:26.655 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2024-04-12T03:06:26.962 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-12T03:06:27.192 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-12T03:06:28.942 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-12T03:06:30.219 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2024-04-12T03:06:30.466 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 8/8 2024-04-12T03:06:30.807 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 8/8 2024-04-12T03:06:30.807 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 1/8 2024-04-12T03:06:30.807 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-12T03:06:30.807 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 3/8 2024-04-12T03:06:30.807 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-12T03:06:30.808 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-12T03:06:30.808 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 6/8 2024-04-12T03:06:30.808 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 7/8 2024-04-12T03:06:31.170 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2024-04-12T03:06:31.170 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2024-04-12T03:06:31.794 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:06:31.795 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-12T03:06:31.795 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-grafana-dashboards-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:06:31.795 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:06:31.795 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-prometheus-alerts-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:06:31.795 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-12T03:06:31.795 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-12T03:06:31.795 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:06:31.795 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:06:31.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-12T03:06:32.401 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:06:34.527 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-12T03:06:34.882 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 46 kB/s | 4.5 kB 00:00 2024-04-12T03:06:37.563 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:06:37.565 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================================================= 2024-04-12T03:06:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-12T03:06:37.565 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================================================= 2024-04-12T03:06:37.565 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-12T03:06:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 7.4 M 2024-04-12T03:06:37.565 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-12T03:06:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-12T03:06:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-12T03:06:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-12T03:06:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-12T03:06:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-12T03:06:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-12T03:06:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-12T03:06:37.566 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:06:37.566 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-12T03:06:37.566 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================================================= 2024-04-12T03:06:37.566 INFO:teuthology.orchestra.run.smithi043.stdout:Install 8 Packages 2024-04-12T03:06:37.566 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:06:37.566 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 36 M 2024-04-12T03:06:37.566 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 207 M 2024-04-12T03:06:37.567 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-12T03:06:37.725 INFO:teuthology.orchestra.run.smithi043.stdout:(1/8): libquadmath-8.5.0-20.el8.x86_64.rpm 1.1 MB/s | 172 kB 00:00 2024-04-12T03:06:37.775 INFO:teuthology.orchestra.run.smithi043.stdout:(2/8): libgfortran-8.5.0-20.el8.x86_64.rpm 3.0 MB/s | 645 kB 00:00 2024-04-12T03:06:37.867 INFO:teuthology.orchestra.run.smithi043.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-04-12T03:06:38.076 INFO:teuthology.orchestra.run.smithi043.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2024-04-12T03:06:38.184 INFO:teuthology.orchestra.run.smithi043.stdout:(5/8): openblas-threads-0.3.15-6.el8.x86_64.rpm 16 MB/s | 5.2 MB 00:00 2024-04-12T03:06:38.326 INFO:teuthology.orchestra.run.smithi043.stdout:(6/8): openblas-0.3.15-6.el8.x86_64.rpm 20 MB/s | 5.0 MB 00:00 2024-04-12T03:06:38.601 INFO:teuthology.orchestra.run.smithi043.stdout:(7/8): python3-scipy-1.0.0-21.module+el8.9.0+20 33 MB/s | 14 MB 00:00 2024-04-12T03:06:38.701 INFO:teuthology.orchestra.run.smithi043.stdout:(8/8): ceph-mgr-diskprediction-local-18.2.2-118 6.5 MB/s | 7.4 MB 00:01 2024-04-12T03:06:38.703 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:06:38.703 INFO:teuthology.orchestra.run.smithi043.stdout:Total 32 MB/s | 36 MB 00:01 2024-04-12T03:06:38.972 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-12T03:06:38.992 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-12T03:06:38.992 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-12T03:06:39.338 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-12T03:06:39.348 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-12T03:06:39.931 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-12T03:06:40.095 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-12T03:06:40.241 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2024-04-12T03:06:40.411 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-12T03:06:41.106 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-12T03:06:41.278 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-12T03:06:41.950 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 3/8 2024-04-12T03:06:42.120 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : openblas-0.3.15-6.el8.x86_64 4/8 2024-04-12T03:06:42.890 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 4/8 2024-04-12T03:06:43.194 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-12T03:06:43.428 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-12T03:06:43.711 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 8/8 2024-04-12T03:06:43.711 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2024-04-12T03:06:44.288 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:06:44.288 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-12T03:06:44.288 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:06:44.288 INFO:teuthology.orchestra.run.smithi149.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-12T03:06:44.288 INFO:teuthology.orchestra.run.smithi149.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-12T03:06:44.288 INFO:teuthology.orchestra.run.smithi149.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-12T03:06:44.288 INFO:teuthology.orchestra.run.smithi149.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-12T03:06:44.288 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-12T03:06:44.288 INFO:teuthology.orchestra.run.smithi149.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-12T03:06:44.288 INFO:teuthology.orchestra.run.smithi149.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-12T03:06:44.288 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:06:44.289 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:06:44.458 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-mgr-rook 2024-04-12T03:06:44.810 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:06:45.397 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-12T03:06:46.754 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 7/8 2024-04-12T03:06:46.764 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-12T03:06:46.977 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 8/8 2024-04-12T03:06:47.106 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-12T03:06:47.308 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 8/8 2024-04-12T03:06:47.308 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540eba 1/8 2024-04-12T03:06:47.308 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 2/8 2024-04-12T03:06:47.308 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 3/8 2024-04-12T03:06:47.308 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-12T03:06:47.309 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-12T03:06:47.309 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 6/8 2024-04-12T03:06:47.309 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 7/8 2024-04-12T03:06:49.603 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:06:49.605 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================== 2024-04-12T03:06:49.605 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-04-12T03:06:49.605 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================== 2024-04-12T03:06:49.605 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-12T03:06:49.605 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-rook noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 49 k 2024-04-12T03:06:49.605 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-04-12T03:06:49.605 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-12T03:06:49.606 INFO:teuthology.orchestra.run.smithi149.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-12T03:06:49.606 INFO:teuthology.orchestra.run.smithi149.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-12T03:06:49.606 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-12T03:06:49.606 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-12T03:06:49.606 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-12T03:06:49.606 INFO:teuthology.orchestra.run.smithi149.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-12T03:06:49.606 INFO:teuthology.orchestra.run.smithi149.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-12T03:06:49.606 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-12T03:06:49.606 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-12T03:06:49.606 INFO:teuthology.orchestra.run.smithi149.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-12T03:06:49.606 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-12T03:06:49.606 INFO:teuthology.orchestra.run.smithi149.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-12T03:06:49.606 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:06:49.606 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-12T03:06:49.606 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================== 2024-04-12T03:06:49.607 INFO:teuthology.orchestra.run.smithi149.stdout:Install 14 Packages 2024-04-12T03:06:49.607 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:06:49.607 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 2.0 M 2024-04-12T03:06:49.607 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 25 M 2024-04-12T03:06:49.607 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-12T03:06:49.870 INFO:teuthology.orchestra.run.smithi149.stdout:(1/14): python3-certifi-2018.10.15-7.el8.noarch 129 kB/s | 16 kB 00:00 2024-04-12T03:06:49.896 INFO:teuthology.orchestra.run.smithi149.stdout:(2/14): python3-cachetools-3.1.1-4.el8.noarch.r 217 kB/s | 33 kB 00:00 2024-04-12T03:06:49.921 INFO:teuthology.orchestra.run.smithi149.stdout:(3/14): ceph-mgr-rook-18.2.2-1187.gd540ebac.el8 277 kB/s | 49 kB 00:00 2024-04-12T03:06:49.971 INFO:teuthology.orchestra.run.smithi149.stdout:(4/14): python3-google-auth-1.1.1-10.el8.noarch 809 kB/s | 81 kB 00:00 2024-04-12T03:06:50.063 INFO:teuthology.orchestra.run.smithi149.stdout:(5/14): python3-websocket-client-0.56.0-5.el8.n 668 kB/s | 61 kB 00:00 2024-04-12T03:06:50.123 INFO:teuthology.orchestra.run.smithi149.stdout:(6/14): python3-rsa-4.9-2.el8.noarch.rpm 318 kB/s | 64 kB 00:00 2024-04-12T03:06:50.205 INFO:teuthology.orchestra.run.smithi149.stdout:(7/14): python3-kubernetes-11.0.0-6.el8.noarch. 3.8 MB/s | 1.2 MB 00:00 2024-04-12T03:06:50.289 INFO:teuthology.orchestra.run.smithi149.stdout:(8/14): python3-oauthlib-2.1.0-1.el8.noarch.rpm 970 kB/s | 155 kB 00:00 2024-04-12T03:06:50.314 INFO:teuthology.orchestra.run.smithi149.stdout:(9/14): python3-jwt-1.6.1-2.el8.noarch.rpm 170 kB/s | 43 kB 00:00 2024-04-12T03:06:50.348 INFO:teuthology.orchestra.run.smithi149.stdout:(10/14): python3-requests-oauthlib-1.0.0-1.el8. 300 kB/s | 43 kB 00:00 2024-04-12T03:06:50.457 INFO:teuthology.orchestra.run.smithi149.stdout:(11/14): python3-jsonpointer-1.10-11.el8.noarch 120 kB/s | 20 kB 00:00 2024-04-12T03:06:50.490 INFO:teuthology.orchestra.run.smithi149.stdout:(12/14): python3-jsonpatch-1.21-2.el8.noarch.rp 187 kB/s | 27 kB 00:00 2024-04-12T03:06:50.515 INFO:teuthology.orchestra.run.smithi149.stdout:(13/14): python3-pyasn1-modules-0.3.7-6.el8.noa 548 kB/s | 110 kB 00:00 2024-04-12T03:06:50.607 INFO:teuthology.orchestra.run.smithi149.stdout:(14/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 840 kB/s | 126 kB 00:00 2024-04-12T03:06:50.608 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:06:50.608 INFO:teuthology.orchestra.run.smithi149.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-04-12T03:06:50.683 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-12T03:06:50.699 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-12T03:06:50.699 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-12T03:06:50.783 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-12T03:06:50.791 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-12T03:06:51.323 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-12T03:06:51.572 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-04-12T03:06:51.799 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/14 2024-04-12T03:06:51.986 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/14 2024-04-12T03:06:52.203 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/14 2024-04-12T03:06:52.389 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/14 2024-04-12T03:06:52.646 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 6/14 2024-04-12T03:06:52.936 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 7/14 2024-04-12T03:06:53.184 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/14 2024-04-12T03:06:53.412 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 9/14 2024-04-12T03:06:54.012 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-04-12T03:06:54.296 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 11/14 2024-04-12T03:06:55.403 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 12/14 2024-04-12T03:06:55.632 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 13/14 2024-04-12T03:06:55.820 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 14/14 2024-04-12T03:06:56.141 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 14/14 2024-04-12T03:06:56.141 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 1/14 2024-04-12T03:06:56.141 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/14 2024-04-12T03:06:56.141 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/14 2024-04-12T03:06:56.141 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/14 2024-04-12T03:06:56.141 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/14 2024-04-12T03:06:56.141 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/14 2024-04-12T03:06:56.141 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/14 2024-04-12T03:06:56.141 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 8/14 2024-04-12T03:06:56.141 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/14 2024-04-12T03:06:56.141 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 10/14 2024-04-12T03:06:56.142 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/14 2024-04-12T03:06:56.142 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/14 2024-04-12T03:06:56.142 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 13/14 2024-04-12T03:07:02.340 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525 8/8 2024-04-12T03:07:02.341 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2024-04-12T03:07:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:07:02.955 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-12T03:07:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:07:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-12T03:07:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-12T03:07:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-12T03:07:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-12T03:07:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-12T03:07:02.956 INFO:teuthology.orchestra.run.smithi043.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-12T03:07:02.956 INFO:teuthology.orchestra.run.smithi043.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-12T03:07:02.956 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:07:02.956 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:07:03.148 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr-rook 2024-04-12T03:07:03.550 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:07:05.338 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-12T03:07:05.747 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 25 kB/s | 4.5 kB 00:00 2024-04-12T03:07:08.494 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 14/14 2024-04-12T03:07:08.495 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2024-04-12T03:07:08.555 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:07:08.557 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================================== 2024-04-12T03:07:08.558 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-12T03:07:08.558 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================================== 2024-04-12T03:07:08.558 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-12T03:07:08.558 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 49 k 2024-04-12T03:07:08.558 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-12T03:07:08.558 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-12T03:07:08.558 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-12T03:07:08.558 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-12T03:07:08.558 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-12T03:07:08.558 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-12T03:07:08.558 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-12T03:07:08.558 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-12T03:07:08.558 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-12T03:07:08.559 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-12T03:07:08.559 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-12T03:07:08.559 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-12T03:07:08.559 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-12T03:07:08.559 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-12T03:07:08.559 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:07:08.559 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-12T03:07:08.559 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================================== 2024-04-12T03:07:08.559 INFO:teuthology.orchestra.run.smithi043.stdout:Install 14 Packages 2024-04-12T03:07:08.559 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:07:08.560 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 2.0 M 2024-04-12T03:07:08.560 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 25 M 2024-04-12T03:07:08.560 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-12T03:07:08.799 INFO:teuthology.orchestra.run.smithi043.stdout:(1/14): python3-cachetools-3.1.1-4.el8.noarch.r 457 kB/s | 33 kB 00:00 2024-04-12T03:07:08.816 INFO:teuthology.orchestra.run.smithi043.stdout:(2/14): python3-certifi-2018.10.15-7.el8.noarch 183 kB/s | 16 kB 00:00 2024-04-12T03:07:08.858 INFO:teuthology.orchestra.run.smithi043.stdout:(3/14): python3-google-auth-1.1.1-10.el8.noarch 1.5 MB/s | 81 kB 00:00 2024-04-12T03:07:08.891 INFO:teuthology.orchestra.run.smithi043.stdout:(4/14): python3-rsa-4.9-2.el8.noarch.rpm 1.9 MB/s | 64 kB 00:00 2024-04-12T03:07:08.933 INFO:teuthology.orchestra.run.smithi043.stdout:(5/14): python3-kubernetes-11.0.0-6.el8.noarch. 9.9 MB/s | 1.2 MB 00:00 2024-04-12T03:07:08.988 INFO:teuthology.orchestra.run.smithi043.stdout:(6/14): python3-websocket-client-0.56.0-5.el8.n 639 kB/s | 61 kB 00:00 2024-04-12T03:07:09.102 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:07:09.102 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-12T03:07:09.102 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:07:09.103 INFO:teuthology.orchestra.run.smithi149.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-12T03:07:09.103 INFO:teuthology.orchestra.run.smithi149.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-12T03:07:09.103 INFO:teuthology.orchestra.run.smithi149.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-12T03:07:09.103 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-12T03:07:09.103 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-12T03:07:09.103 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-12T03:07:09.103 INFO:teuthology.orchestra.run.smithi149.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-12T03:07:09.103 INFO:teuthology.orchestra.run.smithi149.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-12T03:07:09.103 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-12T03:07:09.103 INFO:teuthology.orchestra.run.smithi149.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-12T03:07:09.103 INFO:teuthology.orchestra.run.smithi149.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-12T03:07:09.103 INFO:teuthology.orchestra.run.smithi149.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-12T03:07:09.103 INFO:teuthology.orchestra.run.smithi149.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-12T03:07:09.103 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:07:09.103 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:07:09.109 INFO:teuthology.orchestra.run.smithi043.stdout:(7/14): python3-jwt-1.6.1-2.el8.noarch.rpm 244 kB/s | 43 kB 00:00 2024-04-12T03:07:09.134 INFO:teuthology.orchestra.run.smithi043.stdout:(8/14): ceph-mgr-rook-18.2.2-1187.gd540ebac.el8 121 kB/s | 49 kB 00:00 2024-04-12T03:07:09.168 INFO:teuthology.orchestra.run.smithi043.stdout:(9/14): python3-oauthlib-2.1.0-1.el8.noarch.rpm 862 kB/s | 155 kB 00:00 2024-04-12T03:07:09.210 INFO:teuthology.orchestra.run.smithi043.stdout:(10/14): python3-requests-oauthlib-1.0.0-1.el8. 424 kB/s | 43 kB 00:00 2024-04-12T03:07:09.285 INFO:teuthology.orchestra.run.smithi043.stdout:(11/14): python3-pyasn1-modules-0.3.7-6.el8.noa 1.0 MB/s | 110 kB 00:00 2024-04-12T03:07:09.306 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-mgr-cephadm 2024-04-12T03:07:09.311 INFO:teuthology.orchestra.run.smithi043.stdout:(12/14): python3-jsonpatch-1.21-2.el8.noarch.rp 264 kB/s | 27 kB 00:00 2024-04-12T03:07:09.344 INFO:teuthology.orchestra.run.smithi043.stdout:(13/14): python3-jsonpointer-1.10-11.el8.noarch 95 kB/s | 20 kB 00:00 2024-04-12T03:07:09.411 INFO:teuthology.orchestra.run.smithi043.stdout:(14/14): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2024-04-12T03:07:09.412 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:07:09.412 INFO:teuthology.orchestra.run.smithi043.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2024-04-12T03:07:09.497 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-12T03:07:09.514 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-12T03:07:09.515 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-12T03:07:09.609 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-12T03:07:09.619 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-12T03:07:09.663 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:07:10.169 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-12T03:07:10.379 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/14 2024-04-12T03:07:10.617 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/14 2024-04-12T03:07:10.807 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/14 2024-04-12T03:07:10.965 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/14 2024-04-12T03:07:11.137 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/14 2024-04-12T03:07:11.407 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 6/14 2024-04-12T03:07:11.519 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 kB/s | 4.1 kB 00:00 2024-04-12T03:07:11.683 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 7/14 2024-04-12T03:07:11.861 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 kB/s | 4.5 kB 00:00 2024-04-12T03:07:11.897 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/14 2024-04-12T03:07:12.132 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 9/14 2024-04-12T03:07:12.315 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 10/14 2024-04-12T03:07:12.543 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 11/14 2024-04-12T03:07:13.751 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 12/14 2024-04-12T03:07:14.356 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:07:14.357 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================================== 2024-04-12T03:07:14.357 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-04-12T03:07:14.357 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================================== 2024-04-12T03:07:14.357 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-12T03:07:14.358 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 138 k 2024-04-12T03:07:14.358 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-04-12T03:07:14.358 INFO:teuthology.orchestra.run.smithi149.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-12T03:07:14.358 INFO:teuthology.orchestra.run.smithi149.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-12T03:07:14.358 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-12T03:07:14.358 INFO:teuthology.orchestra.run.smithi149.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-12T03:07:14.358 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:07:14.358 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-12T03:07:14.358 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================================== 2024-04-12T03:07:14.358 INFO:teuthology.orchestra.run.smithi149.stdout:Install 5 Packages 2024-04-12T03:07:14.358 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:07:14.359 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 6.0 M 2024-04-12T03:07:14.359 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 26 M 2024-04-12T03:07:14.359 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-12T03:07:14.911 INFO:teuthology.orchestra.run.smithi149.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 121 kB/s | 60 kB 00:00 2024-04-12T03:07:15.028 INFO:teuthology.orchestra.run.smithi149.stdout:(2/5): ceph-mgr-cephadm-18.2.2-1187.gd540ebac.e 225 kB/s | 138 kB 00:00 2024-04-12T03:07:15.053 INFO:teuthology.orchestra.run.smithi149.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 745 kB/s | 476 kB 00:00 2024-04-12T03:07:15.147 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 13/14 2024-04-12T03:07:15.187 INFO:teuthology.orchestra.run.smithi149.stdout:(4/5): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2024-04-12T03:07:15.241 INFO:teuthology.orchestra.run.smithi149.stdout:(5/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.5 MB/s | 538 kB 00:00 2024-04-12T03:07:15.242 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:07:15.243 INFO:teuthology.orchestra.run.smithi149.stdout:Total 6.8 MB/s | 6.0 MB 00:00 2024-04-12T03:07:15.297 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-12T03:07:15.306 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 14/14 2024-04-12T03:07:15.309 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-12T03:07:15.309 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-12T03:07:15.400 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-12T03:07:15.409 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-12T03:07:15.621 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 14/14 2024-04-12T03:07:15.621 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 1/14 2024-04-12T03:07:15.621 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/14 2024-04-12T03:07:15.621 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/14 2024-04-12T03:07:15.622 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/14 2024-04-12T03:07:15.622 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/14 2024-04-12T03:07:15.622 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/14 2024-04-12T03:07:15.622 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/14 2024-04-12T03:07:15.622 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 8/14 2024-04-12T03:07:15.622 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/14 2024-04-12T03:07:15.622 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 10/14 2024-04-12T03:07:15.622 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/14 2024-04-12T03:07:15.622 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/14 2024-04-12T03:07:15.622 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 13/14 2024-04-12T03:07:16.401 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-12T03:07:16.789 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2024-04-12T03:07:17.044 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2024-04-12T03:07:17.382 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-12T03:07:17.684 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-12T03:07:17.850 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 5/5 2024-04-12T03:07:18.153 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 5/5 2024-04-12T03:07:18.153 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/5 2024-04-12T03:07:18.153 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-12T03:07:18.153 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-12T03:07:18.153 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 4/5 2024-04-12T03:07:24.326 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 14/14 2024-04-12T03:07:24.327 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2024-04-12T03:07:24.971 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:07:24.971 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-12T03:07:24.971 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-rook-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:07:24.971 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-12T03:07:24.971 INFO:teuthology.orchestra.run.smithi043.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-12T03:07:24.971 INFO:teuthology.orchestra.run.smithi043.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-12T03:07:24.971 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-12T03:07:24.971 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-12T03:07:24.972 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-12T03:07:24.972 INFO:teuthology.orchestra.run.smithi043.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-12T03:07:24.972 INFO:teuthology.orchestra.run.smithi043.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-12T03:07:24.972 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-12T03:07:24.972 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-12T03:07:24.972 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-12T03:07:24.972 INFO:teuthology.orchestra.run.smithi043.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-12T03:07:24.972 INFO:teuthology.orchestra.run.smithi043.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-12T03:07:24.972 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:07:24.972 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:07:25.175 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-mgr-cephadm 2024-04-12T03:07:25.583 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:07:27.495 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-12T03:07:27.856 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 kB/s | 4.5 kB 00:00 2024-04-12T03:07:29.773 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 5/5 2024-04-12T03:07:29.773 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2024-04-12T03:07:30.336 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:07:30.336 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-12T03:07:30.336 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:07:30.336 INFO:teuthology.orchestra.run.smithi149.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-12T03:07:30.336 INFO:teuthology.orchestra.run.smithi149.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-12T03:07:30.336 INFO:teuthology.orchestra.run.smithi149.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-12T03:07:30.336 INFO:teuthology.orchestra.run.smithi149.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-12T03:07:30.337 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:07:30.337 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:07:30.524 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-fuse 2024-04-12T03:07:30.643 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:07:30.644 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================================== 2024-04-12T03:07:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-12T03:07:30.645 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================================== 2024-04-12T03:07:30.645 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-12T03:07:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 138 k 2024-04-12T03:07:30.645 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-12T03:07:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-12T03:07:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-12T03:07:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-12T03:07:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-12T03:07:30.645 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:07:30.645 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-12T03:07:30.645 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================================== 2024-04-12T03:07:30.646 INFO:teuthology.orchestra.run.smithi043.stdout:Install 5 Packages 2024-04-12T03:07:30.646 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:07:30.646 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 6.0 M 2024-04-12T03:07:30.646 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 26 M 2024-04-12T03:07:30.646 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-12T03:07:30.884 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:07:31.048 INFO:teuthology.orchestra.run.smithi043.stdout:(1/5): ceph-mgr-cephadm-18.2.2-1187.gd540ebac.e 424 kB/s | 138 kB 00:00 2024-04-12T03:07:31.131 INFO:teuthology.orchestra.run.smithi043.stdout:(2/5): python3-natsort-7.1.1-2.el8.noarch.rpm 147 kB/s | 60 kB 00:00 2024-04-12T03:07:31.315 INFO:teuthology.orchestra.run.smithi043.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 18 MB/s | 4.8 MB 00:00 2024-04-12T03:07:31.349 INFO:teuthology.orchestra.run.smithi043.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.4 MB/s | 538 kB 00:00 2024-04-12T03:07:31.457 INFO:teuthology.orchestra.run.smithi043.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 647 kB/s | 476 kB 00:00 2024-04-12T03:07:31.460 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:07:31.460 INFO:teuthology.orchestra.run.smithi043.stdout:Total 7.3 MB/s | 6.0 MB 00:00 2024-04-12T03:07:31.522 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-12T03:07:31.536 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-12T03:07:31.536 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-12T03:07:31.636 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-12T03:07:31.646 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-12T03:07:32.705 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-12T03:07:32.785 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 46 kB/s | 4.1 kB 00:00 2024-04-12T03:07:33.093 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2024-04-12T03:07:33.119 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 40 kB/s | 4.5 kB 00:00 2024-04-12T03:07:33.334 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2024-04-12T03:07:33.696 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-12T03:07:33.977 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-12T03:07:34.145 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 5/5 2024-04-12T03:07:34.455 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 5/5 2024-04-12T03:07:34.455 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 1/5 2024-04-12T03:07:34.455 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-12T03:07:34.455 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2024-04-12T03:07:34.455 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 4/5 2024-04-12T03:07:35.523 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:07:35.524 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:07:35.524 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-04-12T03:07:35.524 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:07:35.524 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-12T03:07:35.524 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-fuse 2024-04-12T03:07:35.524 INFO:teuthology.orchestra.run.smithi149.stdout: x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 859 k 2024-04-12T03:07:35.524 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-04-12T03:07:35.524 INFO:teuthology.orchestra.run.smithi149.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-12T03:07:35.524 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:07:35.524 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-12T03:07:35.525 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:07:35.525 INFO:teuthology.orchestra.run.smithi149.stdout:Install 2 Packages 2024-04-12T03:07:35.525 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:07:35.525 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 942 k 2024-04-12T03:07:35.525 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 3.0 M 2024-04-12T03:07:35.525 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-12T03:07:35.697 INFO:teuthology.orchestra.run.smithi149.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 486 kB/s | 83 kB 00:00 2024-04-12T03:07:36.205 INFO:teuthology.orchestra.run.smithi149.stdout:(2/2): ceph-fuse-18.2.2-1187.gd540ebac.el8.x86_ 1.2 MB/s | 859 kB 00:00 2024-04-12T03:07:36.206 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:07:36.206 INFO:teuthology.orchestra.run.smithi149.stdout:Total 1.4 MB/s | 942 kB 00:00 2024-04-12T03:07:36.219 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-12T03:07:36.241 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-12T03:07:36.241 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-12T03:07:36.329 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-12T03:07:36.337 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-12T03:07:36.505 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-12T03:07:36.755 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-04-12T03:07:36.913 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2/2 2024-04-12T03:07:37.299 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2/2 2024-04-12T03:07:37.299 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/2 2024-04-12T03:07:45.143 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 5/5 2024-04-12T03:07:45.144 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2024-04-12T03:07:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:07:45.778 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-12T03:07:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-cephadm-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:07:45.778 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-12T03:07:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-12T03:07:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-12T03:07:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-12T03:07:45.779 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:07:45.779 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:07:45.959 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-fuse 2024-04-12T03:07:46.372 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:07:48.412 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 24 kB/s | 4.1 kB 00:00 2024-04-12T03:07:48.772 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-04-12T03:07:49.599 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-04-12T03:07:49.599 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2024-04-12T03:07:50.160 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:07:50.160 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-12T03:07:50.160 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-04-12T03:07:50.160 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:07:50.160 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:07:50.344 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install ceph-volume 2024-04-12T03:07:50.714 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:07:51.537 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:07:51.538 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:07:51.538 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-12T03:07:51.538 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:07:51.538 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-12T03:07:51.538 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse 2024-04-12T03:07:51.538 INFO:teuthology.orchestra.run.smithi043.stdout: x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 859 k 2024-04-12T03:07:51.539 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-12T03:07:51.539 INFO:teuthology.orchestra.run.smithi043.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-12T03:07:51.539 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:07:51.539 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-12T03:07:51.539 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:07:51.539 INFO:teuthology.orchestra.run.smithi043.stdout:Install 2 Packages 2024-04-12T03:07:51.539 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:07:51.539 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 942 k 2024-04-12T03:07:51.540 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 3.0 M 2024-04-12T03:07:51.540 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-12T03:07:51.693 INFO:teuthology.orchestra.run.smithi043.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 544 kB/s | 83 kB 00:00 2024-04-12T03:07:51.993 INFO:teuthology.orchestra.run.smithi043.stdout:(2/2): ceph-fuse-18.2.2-1187.gd540ebac.el8.x86_ 1.9 MB/s | 859 kB 00:00 2024-04-12T03:07:51.994 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:07:51.994 INFO:teuthology.orchestra.run.smithi043.stdout:Total 2.0 MB/s | 942 kB 00:00 2024-04-12T03:07:52.009 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-12T03:07:52.035 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-12T03:07:52.035 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-12T03:07:52.135 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-12T03:07:52.145 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-12T03:07:52.328 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-12T03:07:52.524 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-12T03:07:52.618 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-04-12T03:07:52.784 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2/2 2024-04-12T03:07:52.819 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 50 kB/s | 4.5 kB 00:00 2024-04-12T03:07:53.216 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2/2 2024-04-12T03:07:53.216 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/2 2024-04-12T03:07:55.212 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:07:55.213 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================== 2024-04-12T03:07:55.213 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repository Size 2024-04-12T03:07:55.213 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================== 2024-04-12T03:07:55.213 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-12T03:07:55.213 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-volume noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 263 k 2024-04-12T03:07:55.213 INFO:teuthology.orchestra.run.smithi149.stdout:Installing dependencies: 2024-04-12T03:07:55.213 INFO:teuthology.orchestra.run.smithi149.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-12T03:07:55.213 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:07:55.213 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-12T03:07:55.213 INFO:teuthology.orchestra.run.smithi149.stdout:======================================================================================================== 2024-04-12T03:07:55.214 INFO:teuthology.orchestra.run.smithi149.stdout:Install 2 Packages 2024-04-12T03:07:55.214 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:07:55.214 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 311 k 2024-04-12T03:07:55.214 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 1.4 M 2024-04-12T03:07:55.214 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-12T03:07:55.394 INFO:teuthology.orchestra.run.smithi149.stdout:(1/2): python3-packaging-16.8-9.el8.noarch.rpm 269 kB/s | 48 kB 00:00 2024-04-12T03:07:55.419 INFO:teuthology.orchestra.run.smithi149.stdout:(2/2): ceph-volume-18.2.2-1187.gd540ebac.el8.no 1.3 MB/s | 263 kB 00:00 2024-04-12T03:07:55.420 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:07:55.420 INFO:teuthology.orchestra.run.smithi149.stdout:Total 1.5 MB/s | 311 kB 00:00 2024-04-12T03:07:55.432 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-12T03:07:55.444 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-12T03:07:55.444 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-12T03:07:55.483 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-12T03:07:55.491 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-12T03:07:55.614 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-12T03:07:55.951 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-04-12T03:07:56.253 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2/2 2024-04-12T03:07:56.661 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2/2 2024-04-12T03:07:56.661 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 1/2 2024-04-12T03:08:01.650 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-04-12T03:08:01.650 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2024-04-12T03:08:02.341 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:08:02.342 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-12T03:08:02.342 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-04-12T03:08:02.342 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:08:02.342 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:08:02.835 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install ceph-volume 2024-04-12T03:08:03.249 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:08:05.309 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-04-12T03:08:05.309 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2024-04-12T03:08:05.362 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 23 kB/s | 4.1 kB 00:00 2024-04-12T03:08:05.700 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 48 kB/s | 4.5 kB 00:00 2024-04-12T03:08:05.870 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:08:05.870 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-12T03:08:05.870 INFO:teuthology.orchestra.run.smithi149.stdout: ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:08:05.870 INFO:teuthology.orchestra.run.smithi149.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-12T03:08:05.870 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:08:05.871 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:08:06.044 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install librados-devel 2024-04-12T03:08:06.407 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:08:08.188 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-12T03:08:08.480 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:08:08.481 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================================== 2024-04-12T03:08:08.481 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repository Size 2024-04-12T03:08:08.481 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================================== 2024-04-12T03:08:08.481 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-12T03:08:08.481 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume noarch 2:18.2.2-1187.gd540ebac.el8 ceph-noarch 263 k 2024-04-12T03:08:08.482 INFO:teuthology.orchestra.run.smithi043.stdout:Installing dependencies: 2024-04-12T03:08:08.482 INFO:teuthology.orchestra.run.smithi043.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-04-12T03:08:08.482 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:08:08.482 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-12T03:08:08.482 INFO:teuthology.orchestra.run.smithi043.stdout:======================================================================================================== 2024-04-12T03:08:08.482 INFO:teuthology.orchestra.run.smithi043.stdout:Install 2 Packages 2024-04-12T03:08:08.482 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:08:08.483 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 311 k 2024-04-12T03:08:08.483 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 1.4 M 2024-04-12T03:08:08.483 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-12T03:08:08.556 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 32 kB/s | 4.5 kB 00:00 2024-04-12T03:08:08.607 INFO:teuthology.orchestra.run.smithi043.stdout:(1/2): python3-packaging-16.8-9.el8.noarch.rpm 390 kB/s | 48 kB 00:00 2024-04-12T03:08:08.707 INFO:teuthology.orchestra.run.smithi043.stdout:(2/2): ceph-volume-18.2.2-1187.gd540ebac.el8.no 1.1 MB/s | 263 kB 00:00 2024-04-12T03:08:08.711 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:08:08.711 INFO:teuthology.orchestra.run.smithi043.stdout:Total 1.3 MB/s | 311 kB 00:00 2024-04-12T03:08:08.724 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-12T03:08:08.739 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-12T03:08:08.739 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-12T03:08:08.791 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-12T03:08:08.795 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-12T03:08:08.932 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-12T03:08:09.238 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : python3-packaging-16.8-9.el8.noarch 1/2 2024-04-12T03:08:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2/2 2024-04-12T03:08:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2/2 2024-04-12T03:08:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 1/2 2024-04-12T03:08:10.960 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:08:10.960 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:08:10.961 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repo Size 2024-04-12T03:08:10.961 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:08:10.961 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-12T03:08:10.961 INFO:teuthology.orchestra.run.smithi149.stdout: librados-devel x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 139 k 2024-04-12T03:08:10.961 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:08:10.961 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-12T03:08:10.961 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:08:10.961 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2024-04-12T03:08:10.961 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:08:10.961 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 139 k 2024-04-12T03:08:10.961 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 513 k 2024-04-12T03:08:10.961 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-12T03:08:11.142 INFO:teuthology.orchestra.run.smithi149.stdout:librados-devel-18.2.2-1187.gd540ebac.el8.x86_64 769 kB/s | 139 kB 00:00 2024-04-12T03:08:11.143 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:08:11.143 INFO:teuthology.orchestra.run.smithi149.stdout:Total 764 kB/s | 139 kB 00:00 2024-04-12T03:08:11.143 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-12T03:08:11.149 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-12T03:08:11.149 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-12T03:08:11.204 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-12T03:08:11.213 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-12T03:08:11.351 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-12T03:08:11.566 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:08:11.842 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:08:23.728 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 2/2 2024-04-12T03:08:23.728 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2024-04-12T03:08:24.359 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:08:24.359 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-12T03:08:24.359 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-volume-2:18.2.2-1187.gd540ebac.el8.noarch 2024-04-12T03:08:24.359 INFO:teuthology.orchestra.run.smithi043.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-12T03:08:24.359 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:08:24.360 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:08:24.546 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install librados-devel 2024-04-12T03:08:24.963 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:08:25.433 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:08:25.433 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2024-04-12T03:08:26.012 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:08:26.012 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-12T03:08:26.012 INFO:teuthology.orchestra.run.smithi149.stdout: librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:08:26.013 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:08:26.013 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:08:26.189 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install libcephfs2 2024-04-12T03:08:26.552 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:08:26.777 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-12T03:08:27.099 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 52 kB/s | 4.5 kB 00:00 2024-04-12T03:08:28.214 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 37 kB/s | 4.1 kB 00:00 2024-04-12T03:08:28.581 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-04-12T03:08:29.863 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:08:29.863 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:08:29.863 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2024-04-12T03:08:29.863 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:08:29.863 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-12T03:08:29.864 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 139 k 2024-04-12T03:08:29.864 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:08:29.864 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-12T03:08:29.864 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:08:29.864 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-04-12T03:08:29.864 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:08:29.864 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 139 k 2024-04-12T03:08:29.865 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 513 k 2024-04-12T03:08:29.865 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-12T03:08:30.162 INFO:teuthology.orchestra.run.smithi043.stdout:librados-devel-18.2.2-1187.gd540ebac.el8.x86_64 468 kB/s | 139 kB 00:00 2024-04-12T03:08:30.162 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:08:30.162 INFO:teuthology.orchestra.run.smithi043.stdout:Total 466 kB/s | 139 kB 00:00 2024-04-12T03:08:30.163 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-12T03:08:30.170 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-12T03:08:30.170 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-12T03:08:30.242 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-12T03:08:30.251 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-12T03:08:30.389 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-12T03:08:30.602 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:08:30.898 INFO:teuthology.orchestra.run.smithi149.stdout:Package libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:08:30.917 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:08:31.094 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:08:31.095 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-04-12T03:08:31.096 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:08:31.256 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install libcephfs-devel 2024-04-12T03:08:31.621 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:08:33.465 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 44 kB/s | 4.1 kB 00:00 2024-04-12T03:08:33.750 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 58 kB/s | 4.5 kB 00:00 2024-04-12T03:08:36.235 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:08:36.236 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:08:36.236 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repo Size 2024-04-12T03:08:36.236 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:08:36.236 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-12T03:08:36.236 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs-devel x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 31 k 2024-04-12T03:08:36.236 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:08:36.236 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-12T03:08:36.236 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:08:36.236 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2024-04-12T03:08:36.236 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:08:36.237 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 31 k 2024-04-12T03:08:36.237 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 139 k 2024-04-12T03:08:36.237 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-12T03:08:36.463 INFO:teuthology.orchestra.run.smithi149.stdout:libcephfs-devel-18.2.2-1187.gd540ebac.el8.x86_6 137 kB/s | 31 kB 00:00 2024-04-12T03:08:36.464 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:08:36.464 INFO:teuthology.orchestra.run.smithi149.stdout:Total 136 kB/s | 31 kB 00:00 2024-04-12T03:08:36.464 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-12T03:08:36.468 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-12T03:08:36.468 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-12T03:08:36.481 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-12T03:08:36.490 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-12T03:08:36.551 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-12T03:08:36.771 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:08:37.016 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:08:38.953 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:08:38.953 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2024-04-12T03:08:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:08:39.588 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-12T03:08:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: librados-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:08:39.588 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:08:39.588 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:08:39.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install libcephfs2 2024-04-12T03:08:40.191 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:08:42.327 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-12T03:08:42.705 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 32 kB/s | 4.5 kB 00:00 2024-04-12T03:08:44.058 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:08:44.059 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2024-04-12T03:08:44.628 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:08:44.628 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-12T03:08:44.629 INFO:teuthology.orchestra.run.smithi149.stdout: libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:08:44.629 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:08:44.629 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:08:44.808 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install librados2 2024-04-12T03:08:45.170 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:08:45.247 INFO:teuthology.orchestra.run.smithi043.stdout:Package libcephfs2-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:08:45.461 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:08:45.463 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-12T03:08:45.463 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:08:45.632 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install libcephfs-devel 2024-04-12T03:08:46.046 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:08:47.166 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-12T03:08:47.493 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 41 kB/s | 4.5 kB 00:00 2024-04-12T03:08:47.776 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 44 kB/s | 4.1 kB 00:00 2024-04-12T03:08:48.098 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 46 kB/s | 4.5 kB 00:00 2024-04-12T03:08:49.671 INFO:teuthology.orchestra.run.smithi149.stdout:Package librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:08:49.870 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:08:49.871 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-04-12T03:08:49.871 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:08:50.020 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install librbd1 2024-04-12T03:08:50.380 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:08:50.863 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:08:50.864 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:08:50.864 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2024-04-12T03:08:50.864 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:08:50.864 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-12T03:08:50.864 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 31 k 2024-04-12T03:08:50.865 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:08:50.865 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-12T03:08:50.865 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:08:50.865 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-04-12T03:08:50.865 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:08:50.865 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 31 k 2024-04-12T03:08:50.866 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 139 k 2024-04-12T03:08:50.866 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-12T03:08:51.258 INFO:teuthology.orchestra.run.smithi043.stdout:libcephfs-devel-18.2.2-1187.gd540ebac.el8.x86_6 79 kB/s | 31 kB 00:00 2024-04-12T03:08:51.258 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:08:51.258 INFO:teuthology.orchestra.run.smithi043.stdout:Total 79 kB/s | 31 kB 00:00 2024-04-12T03:08:51.259 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-12T03:08:51.264 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-12T03:08:51.264 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-12T03:08:51.281 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-12T03:08:51.291 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-12T03:08:51.353 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-12T03:08:51.549 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:08:51.840 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:08:52.204 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-12T03:08:52.562 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-12T03:08:54.743 INFO:teuthology.orchestra.run.smithi149.stdout:Package librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:08:54.933 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:08:54.934 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-04-12T03:08:54.934 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:08:55.098 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install python3-rados 2024-04-12T03:08:55.458 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:08:57.135 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 40 kB/s | 4.1 kB 00:00 2024-04-12T03:08:57.441 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 46 kB/s | 4.5 kB 00:00 2024-04-12T03:08:59.681 INFO:teuthology.orchestra.run.smithi149.stdout:Package python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:08:59.873 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:08:59.877 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-04-12T03:08:59.878 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:09:00.025 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install python3-rgw 2024-04-12T03:09:00.384 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:09:02.275 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 35 kB/s | 4.1 kB 00:00 2024-04-12T03:09:02.642 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-04-12T03:09:03.530 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:09:03.531 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2024-04-12T03:09:04.142 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:09:04.143 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-12T03:09:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-devel-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:09:04.143 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:09:04.143 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:09:04.330 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install librados2 2024-04-12T03:09:04.742 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:09:04.907 INFO:teuthology.orchestra.run.smithi149.stdout:Package python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:09:05.098 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:09:05.100 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-04-12T03:09:05.100 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:09:05.259 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install python3-cephfs 2024-04-12T03:09:05.620 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:09:06.718 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 30 kB/s | 4.1 kB 00:00 2024-04-12T03:09:07.052 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 49 kB/s | 4.5 kB 00:00 2024-04-12T03:09:07.385 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 kB/s | 4.1 kB 00:00 2024-04-12T03:09:07.767 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 26 kB/s | 4.5 kB 00:00 2024-04-12T03:09:09.634 INFO:teuthology.orchestra.run.smithi043.stdout:Package librados2-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:09:09.852 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:09:09.853 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-12T03:09:09.853 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:09:09.973 INFO:teuthology.orchestra.run.smithi149.stdout:Package python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:09:10.020 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install librbd1 2024-04-12T03:09:10.164 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:09:10.165 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-04-12T03:09:10.165 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:09:10.324 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install python3-rbd 2024-04-12T03:09:10.431 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:09:10.687 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:09:12.300 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 33 kB/s | 4.1 kB 00:00 2024-04-12T03:09:12.593 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 33 kB/s | 4.1 kB 00:00 2024-04-12T03:09:12.681 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-04-12T03:09:12.970 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 27 kB/s | 4.5 kB 00:00 2024-04-12T03:09:15.159 INFO:teuthology.orchestra.run.smithi149.stdout:Package python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:09:15.237 INFO:teuthology.orchestra.run.smithi043.stdout:Package librbd1-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:09:15.353 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:09:15.354 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-04-12T03:09:15.354 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:09:15.459 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:09:15.461 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-12T03:09:15.461 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:09:15.502 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install rbd-fuse 2024-04-12T03:09:15.642 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install python3-rados 2024-04-12T03:09:15.863 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:09:16.055 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:09:17.722 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 34 kB/s | 4.1 kB 00:00 2024-04-12T03:09:18.053 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-04-12T03:09:18.090 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-04-12T03:09:18.396 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 kB/s | 4.5 kB 00:00 2024-04-12T03:09:20.549 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:09:20.550 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:09:20.550 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-04-12T03:09:20.550 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:09:20.550 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-12T03:09:20.550 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-fuse x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 95 k 2024-04-12T03:09:20.550 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:09:20.550 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-12T03:09:20.550 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:09:20.550 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2024-04-12T03:09:20.551 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:09:20.551 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 95 k 2024-04-12T03:09:20.551 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 280 k 2024-04-12T03:09:20.551 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-12T03:09:20.927 INFO:teuthology.orchestra.run.smithi043.stdout:Package python3-rados-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:09:21.027 INFO:teuthology.orchestra.run.smithi149.stdout:rbd-fuse-18.2.2-1187.gd540ebac.el8.x86_64.rpm 200 kB/s | 95 kB 00:00 2024-04-12T03:09:21.028 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:09:21.028 INFO:teuthology.orchestra.run.smithi149.stdout:Total 199 kB/s | 95 kB 00:00 2024-04-12T03:09:21.028 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-12T03:09:21.038 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-12T03:09:21.038 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-12T03:09:21.093 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-12T03:09:21.103 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-12T03:09:21.143 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:09:21.144 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-12T03:09:21.144 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:09:21.271 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-12T03:09:21.416 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install python3-rgw 2024-04-12T03:09:21.471 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:09:21.753 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:09:21.827 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:09:23.656 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-12T03:09:24.044 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 29 kB/s | 4.5 kB 00:00 2024-04-12T03:09:26.576 INFO:teuthology.orchestra.run.smithi043.stdout:Package python3-rgw-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:09:26.798 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:09:26.800 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-12T03:09:26.800 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:09:26.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install python3-cephfs 2024-04-12T03:09:27.368 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:09:29.421 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-12T03:09:29.772 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 kB/s | 4.5 kB 00:00 2024-04-12T03:09:31.157 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:09:31.157 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2024-04-12T03:09:31.776 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:09:31.776 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-12T03:09:31.776 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:09:31.776 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:09:31.776 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:09:31.939 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install rbd-mirror 2024-04-12T03:09:32.298 INFO:teuthology.orchestra.run.smithi043.stdout:Package python3-cephfs-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:09:32.301 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:09:32.513 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:09:32.514 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-12T03:09:32.514 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:09:32.675 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install python3-rbd 2024-04-12T03:09:33.086 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:09:34.080 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 41 kB/s | 4.1 kB 00:00 2024-04-12T03:09:34.432 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-04-12T03:09:34.981 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-04-12T03:09:35.387 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-04-12T03:09:36.817 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:09:36.817 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:09:36.817 INFO:teuthology.orchestra.run.smithi149.stdout: Package Arch Version Repo Size 2024-04-12T03:09:36.817 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:09:36.817 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-12T03:09:36.818 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-mirror x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.5 M 2024-04-12T03:09:36.818 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:09:36.818 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-12T03:09:36.818 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:09:36.818 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2024-04-12T03:09:36.818 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:09:36.818 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 3.5 M 2024-04-12T03:09:36.819 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 16 M 2024-04-12T03:09:36.819 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-12T03:09:37.999 INFO:teuthology.orchestra.run.smithi043.stdout:Package python3-rbd-2:18.2.2-1187.gd540ebac.el8.x86_64 is already installed. 2024-04-12T03:09:38.214 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:09:38.215 INFO:teuthology.orchestra.run.smithi043.stdout:Nothing to do. 2024-04-12T03:09:38.215 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:09:38.389 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install rbd-fuse 2024-04-12T03:09:38.571 INFO:teuthology.orchestra.run.smithi149.stdout:rbd-mirror-18.2.2-1187.gd540ebac.el8.x86_64.rpm 2.0 MB/s | 3.5 MB 00:01 2024-04-12T03:09:38.572 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:09:38.572 INFO:teuthology.orchestra.run.smithi149.stdout:Total 2.0 MB/s | 3.5 MB 00:01 2024-04-12T03:09:38.573 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-12T03:09:38.586 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-12T03:09:38.587 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-12T03:09:38.659 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-12T03:09:38.668 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-12T03:09:38.800 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:09:39.118 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-12T03:09:39.262 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:09:39.679 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:09:40.620 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 35 kB/s | 4.1 kB 00:00 2024-04-12T03:09:40.950 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 48 kB/s | 4.5 kB 00:00 2024-04-12T03:09:43.638 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:09:43.639 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:09:43.639 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2024-04-12T03:09:43.639 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:09:43.639 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-12T03:09:43.639 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 95 k 2024-04-12T03:09:43.639 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:09:43.639 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-12T03:09:43.639 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:09:43.639 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-04-12T03:09:43.639 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:09:43.640 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 95 k 2024-04-12T03:09:43.640 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 280 k 2024-04-12T03:09:43.640 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-12T03:09:43.923 INFO:teuthology.orchestra.run.smithi043.stdout:rbd-fuse-18.2.2-1187.gd540ebac.el8.x86_64.rpm 336 kB/s | 95 kB 00:00 2024-04-12T03:09:43.923 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:09:43.923 INFO:teuthology.orchestra.run.smithi043.stdout:Total 335 kB/s | 95 kB 00:00 2024-04-12T03:09:43.924 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-12T03:09:43.938 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-12T03:09:43.938 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-12T03:09:44.008 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-12T03:09:44.018 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-12T03:09:44.197 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-12T03:09:44.390 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:09:44.692 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:09:47.083 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:09:47.083 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2024-04-12T03:09:47.650 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:09:47.650 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-12T03:09:47.650 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:09:47.650 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:09:47.650 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:09:47.827 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum -y install rbd-nbd 2024-04-12T03:09:48.192 INFO:teuthology.orchestra.run.smithi149.stdout:Updating Subscription Management repositories. 2024-04-12T03:09:50.192 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-12T03:09:50.562 INFO:teuthology.orchestra.run.smithi149.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-04-12T03:09:52.726 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:09:52.727 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2024-04-12T03:09:52.944 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-12T03:09:52.944 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:09:52.944 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-04-12T03:09:52.944 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:09:52.944 INFO:teuthology.orchestra.run.smithi149.stdout:Installing: 2024-04-12T03:09:52.945 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-nbd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 181 k 2024-04-12T03:09:52.945 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:09:52.945 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-12T03:09:52.945 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-12T03:09:52.945 INFO:teuthology.orchestra.run.smithi149.stdout:Install 1 Package 2024-04-12T03:09:52.945 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:09:52.945 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 181 k 2024-04-12T03:09:52.945 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 572 k 2024-04-12T03:09:52.945 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-12T03:09:53.145 INFO:teuthology.orchestra.run.smithi149.stdout:rbd-nbd-18.2.2-1187.gd540ebac.el8.x86_64.rpm 911 kB/s | 181 kB 00:00 2024-04-12T03:09:53.145 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:09:53.146 INFO:teuthology.orchestra.run.smithi149.stdout:Total 906 kB/s | 181 kB 00:00 2024-04-12T03:09:53.146 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-12T03:09:53.156 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-12T03:09:53.156 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-12T03:09:53.213 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-12T03:09:53.222 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-12T03:09:53.356 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-12T03:09:53.364 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:09:53.364 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-12T03:09:53.364 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-fuse-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:09:53.364 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:09:53.364 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:09:53.541 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install rbd-mirror 2024-04-12T03:09:53.558 INFO:teuthology.orchestra.run.smithi149.stdout: Installing : rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:09:53.836 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:09:53.962 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:09:55.934 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 26 kB/s | 4.1 kB 00:00 2024-04-12T03:09:56.261 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 49 kB/s | 4.5 kB 00:00 2024-04-12T03:09:59.050 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:09:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:09:59.051 INFO:teuthology.orchestra.run.smithi043.stdout: Package Arch Version Repo Size 2024-04-12T03:09:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:09:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-12T03:09:59.051 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 3.5 M 2024-04-12T03:09:59.051 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:09:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-12T03:09:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:09:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-04-12T03:09:59.051 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:09:59.051 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 3.5 M 2024-04-12T03:09:59.052 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 16 M 2024-04-12T03:09:59.052 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-12T03:09:59.410 INFO:teuthology.orchestra.run.smithi043.stdout:rbd-mirror-18.2.2-1187.gd540ebac.el8.x86_64.rpm 9.6 MB/s | 3.5 MB 00:00 2024-04-12T03:09:59.411 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:09:59.411 INFO:teuthology.orchestra.run.smithi043.stdout:Total 9.6 MB/s | 3.5 MB 00:00 2024-04-12T03:09:59.411 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-12T03:09:59.427 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-12T03:09:59.427 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-12T03:09:59.508 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-12T03:09:59.518 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-12T03:10:00.008 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-12T03:10:00.172 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:10:00.623 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:10:02.363 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:10:02.363 INFO:teuthology.orchestra.run.smithi149.stdout:Installed products updated. 2024-04-12T03:10:02.966 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:02.966 INFO:teuthology.orchestra.run.smithi149.stdout:Installed: 2024-04-12T03:10:02.966 INFO:teuthology.orchestra.run.smithi149.stdout: rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:10:02.966 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:02.966 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-12T03:10:03.153 DEBUG:teuthology.parallel:result is None 2024-04-12T03:10:10.483 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:10:10.483 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2024-04-12T03:10:11.262 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:11.263 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-12T03:10:11.263 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-mirror-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:10:11.263 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:11.263 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:10:11.431 DEBUG:teuthology.orchestra.run.smithi043:> sudo yum -y install rbd-nbd 2024-04-12T03:10:11.892 INFO:teuthology.orchestra.run.smithi043.stdout:Updating Subscription Management repositories. 2024-04-12T03:10:13.800 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-04-12T03:10:14.206 INFO:teuthology.orchestra.run.smithi043.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 26 kB/s | 4.5 kB 00:00 2024-04-12T03:10:16.885 INFO:teuthology.orchestra.run.smithi043.stdout:Dependencies resolved. 2024-04-12T03:10:16.885 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:10:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: Package Architecture Version Repository Size 2024-04-12T03:10:16.885 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:10:16.885 INFO:teuthology.orchestra.run.smithi043.stdout:Installing: 2024-04-12T03:10:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd x86_64 2:18.2.2-1187.gd540ebac.el8 ceph 181 k 2024-04-12T03:10:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:16.885 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction Summary 2024-04-12T03:10:16.886 INFO:teuthology.orchestra.run.smithi043.stdout:================================================================================ 2024-04-12T03:10:16.886 INFO:teuthology.orchestra.run.smithi043.stdout:Install 1 Package 2024-04-12T03:10:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:16.886 INFO:teuthology.orchestra.run.smithi043.stdout:Total download size: 181 k 2024-04-12T03:10:16.886 INFO:teuthology.orchestra.run.smithi043.stdout:Installed size: 572 k 2024-04-12T03:10:16.886 INFO:teuthology.orchestra.run.smithi043.stdout:Downloading Packages: 2024-04-12T03:10:17.080 INFO:teuthology.orchestra.run.smithi043.stdout:rbd-nbd-18.2.2-1187.gd540ebac.el8.x86_64.rpm 939 kB/s | 181 kB 00:00 2024-04-12T03:10:17.081 INFO:teuthology.orchestra.run.smithi043.stdout:-------------------------------------------------------------------------------- 2024-04-12T03:10:17.081 INFO:teuthology.orchestra.run.smithi043.stdout:Total 932 kB/s | 181 kB 00:00 2024-04-12T03:10:17.081 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction check 2024-04-12T03:10:17.095 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction check succeeded. 2024-04-12T03:10:17.095 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction test 2024-04-12T03:10:17.163 INFO:teuthology.orchestra.run.smithi043.stdout:Transaction test succeeded. 2024-04-12T03:10:17.173 INFO:teuthology.orchestra.run.smithi043.stdout:Running transaction 2024-04-12T03:10:17.316 INFO:teuthology.orchestra.run.smithi043.stdout: Preparing : 1/1 2024-04-12T03:10:17.485 INFO:teuthology.orchestra.run.smithi043.stdout: Installing : rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:10:17.800 INFO:teuthology.orchestra.run.smithi043.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:10:28.568 INFO:teuthology.orchestra.run.smithi043.stdout: Verifying : rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 1/1 2024-04-12T03:10:28.568 INFO:teuthology.orchestra.run.smithi043.stdout:Installed products updated. 2024-04-12T03:10:29.203 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:29.203 INFO:teuthology.orchestra.run.smithi043.stdout:Installed: 2024-04-12T03:10:29.203 INFO:teuthology.orchestra.run.smithi043.stdout: rbd-nbd-2:18.2.2-1187.gd540ebac.el8.x86_64 2024-04-12T03:10:29.203 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:29.203 INFO:teuthology.orchestra.run.smithi043.stdout:Complete! 2024-04-12T03:10:29.393 DEBUG:teuthology.parallel:result is None 2024-04-12T03:10:29.393 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T03:10:29.549 DEBUG:teuthology.orchestra.run.smithi043:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-12T03:10:29.577 INFO:teuthology.orchestra.run.smithi043.stdout:18.2.2-1187.gd540ebac.el8 2024-04-12T03:10:29.578 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1187.gd540ebac.el8 2024-04-12T03:10:29.578 INFO:teuthology.task.install:The correct ceph version 18.2.2-1187.gd540ebac is installed. 2024-04-12T03:10:29.579 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d540ebaca6b131a1dd560e7f69e024b133bbaa42 2024-04-12T03:10:29.743 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-12T03:10:29.769 INFO:teuthology.orchestra.run.smithi149.stdout:18.2.2-1187.gd540ebac.el8 2024-04-12T03:10:29.770 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1187.gd540ebac.el8 2024-04-12T03:10:29.770 INFO:teuthology.task.install:The correct ceph version 18.2.2-1187.gd540ebac is installed. 2024-04-12T03:10:29.772 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-12T03:10:29.772 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:29.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-12T03:10:29.808 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:10:29.808 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-12T03:10:29.841 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-12T03:10:29.841 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:29.842 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/daemon-helper 2024-04-12T03:10:29.883 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-12T03:10:29.958 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:10:29.958 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/daemon-helper 2024-04-12T03:10:29.987 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-12T03:10:30.054 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-12T03:10:30.055 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:30.055 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-12T03:10:30.090 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-12T03:10:30.183 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:10:30.183 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-12T03:10:30.215 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-12T03:10:30.290 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-12T03:10:30.291 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:30.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/stdin-killer 2024-04-12T03:10:30.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-12T03:10:30.418 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:10:30.419 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/usr/bin/stdin-killer 2024-04-12T03:10:30.452 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-12T03:10:30.527 INFO:teuthology.run_tasks:Running task ceph... 2024-04-12T03:10:30.645 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-12T03:10:30.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 777 /var/log/ceph 2024-04-12T03:10:30.648 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 777 /var/log/ceph 2024-04-12T03:10:30.681 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-12T03:10:30.681 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-12T03:10:30.725 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-12T03:10:30.757 INFO:tasks.ceph:Creating extra log directories... 2024-04-12T03:10:30.757 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-12T03:10:30.801 DEBUG:teuthology.orchestra.run.smithi149:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-12T03:10:30.845 INFO:tasks.ceph:Creating ceph cluster c1... 2024-04-12T03:10:30.845 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-12T03:10:30.845 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596', 'branch': 'reef-release', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7652596', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'reef-release'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'reef-release', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 6358, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'reef-release', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'targets': {'smithi043.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM360nfrc8+vzqLBmt/bUgq/Qji/8Ta1ZqA1yis6gch2B7vP9ozECqHJvYXqFyNKiYsii47X9221GsM42/v4qhM=', 'smithi149.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN4+Al/t3FeS8VxGyA79xzsyQJKFWB6bq35ATLnGIa+oeWgp3QpA7yvw/te4TGZwz36fc5R+dr7d8WywxqnZzLc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-11_20:30:42', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-12T03:10:30.846 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-04-12T03:10:30.884 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-04-12T03:10:30.903 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m0777 -- /var/run/ceph 2024-04-12T03:10:30.950 DEBUG:teuthology.orchestra.run.smithi149:> sudo install -d -m0777 -- /var/run/ceph 2024-04-12T03:10:30.983 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:30.984 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/scratch_devs of=/dev/stdout 2024-04-12T03:10:31.046 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-12T03:10:31.047 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_1 2024-04-12T03:10:31.109 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-12T03:10:31.109 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T03:10:31.109 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 6h/6d Inode: 178814 Links: 1 2024-04-12T03:10:31.109 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T03:10:31.109 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T03:10:31.109 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-04-12 03:10:00.356390273 +0000 2024-04-12T03:10:31.109 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-04-12 03:01:57.074885205 +0000 2024-04-12T03:10:31.109 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-04-12 03:01:57.074885205 +0000 2024-04-12T03:10:31.109 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2024-04-12T03:10:31.110 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-12T03:10:31.184 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-04-12T03:10:31.185 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-04-12T03:10:31.185 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000139029 s, 3.7 MB/s 2024-04-12T03:10:31.186 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-12T03:10:31.250 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_2 2024-04-12T03:10:31.312 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-12T03:10:31.312 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T03:10:31.313 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 6h/6d Inode: 173727 Links: 1 2024-04-12T03:10:31.313 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T03:10:31.313 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T03:10:31.313 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-04-12 03:10:00.356390273 +0000 2024-04-12T03:10:31.313 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-04-12 03:01:57.440880454 +0000 2024-04-12T03:10:31.313 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-04-12 03:01:57.440880454 +0000 2024-04-12T03:10:31.313 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2024-04-12T03:10:31.313 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-12T03:10:31.385 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-04-12T03:10:31.385 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-04-12T03:10:31.386 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000118065 s, 4.3 MB/s 2024-04-12T03:10:31.387 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-12T03:10:31.451 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_3 2024-04-12T03:10:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-12T03:10:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T03:10:31.513 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 6h/6d Inode: 178926 Links: 1 2024-04-12T03:10:31.513 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T03:10:31.513 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T03:10:31.513 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-04-12 03:10:00.356390273 +0000 2024-04-12T03:10:31.513 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-04-12 03:01:57.949873848 +0000 2024-04-12T03:10:31.513 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-04-12 03:01:57.949873848 +0000 2024-04-12T03:10:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2024-04-12T03:10:31.514 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-12T03:10:31.586 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-04-12T03:10:31.586 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-04-12T03:10:31.587 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000109139 s, 4.7 MB/s 2024-04-12T03:10:31.588 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-12T03:10:31.652 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_4 2024-04-12T03:10:31.714 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-12T03:10:31.715 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T03:10:31.715 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 6h/6d Inode: 180567 Links: 1 2024-04-12T03:10:31.715 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T03:10:31.715 INFO:teuthology.orchestra.run.smithi043.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T03:10:31.715 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-04-12 03:10:00.356390273 +0000 2024-04-12T03:10:31.715 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-04-12 03:01:58.323868994 +0000 2024-04-12T03:10:31.715 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-04-12 03:01:58.323868994 +0000 2024-04-12T03:10:31.715 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2024-04-12T03:10:31.715 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-12T03:10:31.788 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-04-12T03:10:31.788 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-04-12T03:10:31.788 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000114141 s, 4.5 MB/s 2024-04-12T03:10:31.789 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-12T03:10:31.853 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-12T03:10:31.853 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi043.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-12T03:10:31.853 INFO:tasks.ceph:Generating config... 2024-04-12T03:10:31.854 INFO:tasks.ceph:[client] debug rgw = 20 2024-04-12T03:10:31.854 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-04-12T03:10:31.854 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-04-12T03:10:31.855 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-04-12T03:10:31.855 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-12T03:10:31.855 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-04-12T03:10:31.855 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-04-12T03:10:31.855 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-04-12T03:10:31.855 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-04-12T03:10:31.855 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-04-12T03:10:31.855 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-04-12T03:10:31.855 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-04-12T03:10:31.855 INFO:tasks.ceph:[client] setgroup = ceph 2024-04-12T03:10:31.855 INFO:tasks.ceph:[client] setuser = ceph 2024-04-12T03:10:31.855 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-04-12T03:10:31.855 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-12T03:10:31.855 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-12T03:10:31.856 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-12T03:10:31.856 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-12T03:10:31.856 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-12T03:10:31.856 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-12T03:10:31.856 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-12T03:10:31.856 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-04-12T03:10:31.856 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-04-12T03:10:31.856 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-04-12T03:10:31.856 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-04-12T03:10:31.856 INFO:tasks.ceph:Setting up c1.mon.a... 2024-04-12T03:10:31.856 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2024-04-12T03:10:31.953 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/c1.keyring 2024-04-12T03:10:31.957 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2024-04-12T03:10:32.011 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-12T03:10:32.045 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.43')] 2024-04-12T03:10:32.045 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.43'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-04-12T03:10:32.046 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:32.046 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-12T03:10:32.111 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.43 --print /home/ubuntu/cephtest/c1.monmap 2024-04-12T03:10:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:10:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:10:32.210 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2024-04-12T03:10:32.210 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool: generated fsid f012a16c-f03c-46fb-892c-37001de8aa41 2024-04-12T03:10:32.210 INFO:teuthology.orchestra.run.smithi043.stdout:setting min_mon_release = pacific 2024-04-12T03:10:32.211 INFO:teuthology.orchestra.run.smithi043.stdout:epoch 0 2024-04-12T03:10:32.211 INFO:teuthology.orchestra.run.smithi043.stdout:fsid f012a16c-f03c-46fb-892c-37001de8aa41 2024-04-12T03:10:32.211 INFO:teuthology.orchestra.run.smithi043.stdout:last_changed 2024-04-12T03:10:32.209398+0000 2024-04-12T03:10:32.211 INFO:teuthology.orchestra.run.smithi043.stdout:created 2024-04-12T03:10:32.209398+0000 2024-04-12T03:10:32.211 INFO:teuthology.orchestra.run.smithi043.stdout:min_mon_release 16 (pacific) 2024-04-12T03:10:32.211 INFO:teuthology.orchestra.run.smithi043.stdout:election_strategy: 1 2024-04-12T03:10:32.211 INFO:teuthology.orchestra.run.smithi043.stdout:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-12T03:10:32.211 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2024-04-12T03:10:32.212 DEBUG:teuthology.orchestra.run.smithi043:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-12T03:10:32.234 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID f012a16c-f03c-46fb-892c-37001de8aa41... 2024-04-12T03:10:32.235 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-04-12T03:10:32.278 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-04-12T03:10:32.334 INFO:teuthology.orchestra.run.smithi149.stdout:[global] 2024-04-12T03:10:32.334 INFO:teuthology.orchestra.run.smithi149.stdout: chdir = "" 2024-04-12T03:10:32.334 INFO:teuthology.orchestra.run.smithi149.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-12T03:10:32.334 INFO:teuthology.orchestra.run.smithi149.stdout: auth supported = cephx 2024-04-12T03:10:32.334 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.334 INFO:teuthology.orchestra.run.smithi149.stdout: filestore xattr use omap = true 2024-04-12T03:10:32.334 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.335 INFO:teuthology.orchestra.run.smithi149.stdout: mon clock drift allowed = 1.000 2024-04-12T03:10:32.335 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.335 INFO:teuthology.orchestra.run.smithi149.stdout: osd crush chooseleaf type = 0 2024-04-12T03:10:32.335 INFO:teuthology.orchestra.run.smithi149.stdout: auth debug = true 2024-04-12T03:10:32.335 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.335 INFO:teuthology.orchestra.run.smithi149.stdout: ms die on old message = true 2024-04-12T03:10:32.335 INFO:teuthology.orchestra.run.smithi149.stdout: ms die on bug = true 2024-04-12T03:10:32.335 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.335 INFO:teuthology.orchestra.run.smithi149.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-12T03:10:32.335 INFO:teuthology.orchestra.run.smithi149.stdout: mon pg warn max object skew = 0 2024-04-12T03:10:32.335 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.335 INFO:teuthology.orchestra.run.smithi149.stdout: # disable pg_autoscaler by default for new pools 2024-04-12T03:10:32.335 INFO:teuthology.orchestra.run.smithi149.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-12T03:10:32.335 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.336 INFO:teuthology.orchestra.run.smithi149.stdout: osd pool default size = 2 2024-04-12T03:10:32.336 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.336 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd allow primary affinity = true 2024-04-12T03:10:32.336 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd allow pg remap = true 2024-04-12T03:10:32.336 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on legacy crush tunables = false 2024-04-12T03:10:32.336 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on crush straw calc version zero = false 2024-04-12T03:10:32.336 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on no sortbitwise = false 2024-04-12T03:10:32.336 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on osd down out interval zero = false 2024-04-12T03:10:32.336 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on too few osds = false 2024-04-12T03:10:32.336 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-12T03:10:32.336 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-12T03:10:32.336 INFO:teuthology.orchestra.run.smithi149.stdout: mon_allow_pool_size_one = true 2024-04-12T03:10:32.336 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.337 INFO:teuthology.orchestra.run.smithi149.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-12T03:10:32.337 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.337 INFO:teuthology.orchestra.run.smithi149.stdout: osd default data pool replay window = 5 2024-04-12T03:10:32.337 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.337 INFO:teuthology.orchestra.run.smithi149.stdout: mon allow pool delete = true 2024-04-12T03:10:32.337 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.338 INFO:teuthology.orchestra.run.smithi149.stdout: mon cluster log file level = debug 2024-04-12T03:10:32.338 INFO:teuthology.orchestra.run.smithi149.stdout: debug asserts on shutdown = true 2024-04-12T03:10:32.338 INFO:teuthology.orchestra.run.smithi149.stdout: mon health detail to clog = false 2024-04-12T03:10:32.338 INFO:teuthology.orchestra.run.smithi149.stdout: mon host = 172.21.15.43 2024-04-12T03:10:32.338 INFO:teuthology.orchestra.run.smithi149.stdout: fsid = f012a16c-f03c-46fb-892c-37001de8aa41 2024-04-12T03:10:32.338 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.338 INFO:teuthology.orchestra.run.smithi149.stdout:[osd] 2024-04-12T03:10:32.338 INFO:teuthology.orchestra.run.smithi149.stdout: osd journal size = 100 2024-04-12T03:10:32.338 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.338 INFO:teuthology.orchestra.run.smithi149.stdout: osd scrub load threshold = 5.0 2024-04-12T03:10:32.338 INFO:teuthology.orchestra.run.smithi149.stdout: osd scrub max interval = 600 2024-04-12T03:10:32.338 INFO:teuthology.orchestra.run.smithi149.stdout: osd mclock profile = high_recovery_ops 2024-04-12T03:10:32.338 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.338 INFO:teuthology.orchestra.run.smithi149.stdout: osd recover clone overlap = true 2024-04-12T03:10:32.339 INFO:teuthology.orchestra.run.smithi149.stdout: osd recovery max chunk = 1048576 2024-04-12T03:10:32.339 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.339 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug shutdown = true 2024-04-12T03:10:32.339 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug op order = true 2024-04-12T03:10:32.339 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug verify stray on activate = true 2024-04-12T03:10:32.339 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.339 INFO:teuthology.orchestra.run.smithi149.stdout: osd open classes on start = true 2024-04-12T03:10:32.339 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug pg log writeout = true 2024-04-12T03:10:32.339 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.339 INFO:teuthology.orchestra.run.smithi149.stdout: osd deep scrub update digest min age = 30 2024-04-12T03:10:32.339 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.339 INFO:teuthology.orchestra.run.smithi149.stdout: osd map max advance = 10 2024-04-12T03:10:32.339 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.339 INFO:teuthology.orchestra.run.smithi149.stdout: journal zero on create = true 2024-04-12T03:10:32.340 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.340 INFO:teuthology.orchestra.run.smithi149.stdout: filestore ondisk finisher threads = 3 2024-04-12T03:10:32.340 INFO:teuthology.orchestra.run.smithi149.stdout: filestore apply finisher threads = 3 2024-04-12T03:10:32.340 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.340 INFO:teuthology.orchestra.run.smithi149.stdout: bdev debug aio = true 2024-04-12T03:10:32.340 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug misdirected ops = true 2024-04-12T03:10:32.340 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2024-04-12T03:10:32.340 INFO:teuthology.orchestra.run.smithi149.stdout: debug osd = 20 2024-04-12T03:10:32.340 INFO:teuthology.orchestra.run.smithi149.stdout: rocksdb delete range threshold = 0 2024-04-12T03:10:32.340 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.340 INFO:teuthology.orchestra.run.smithi149.stdout:[mgr] 2024-04-12T03:10:32.340 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2024-04-12T03:10:32.340 INFO:teuthology.orchestra.run.smithi149.stdout: debug mgr = 20 2024-04-12T03:10:32.340 INFO:teuthology.orchestra.run.smithi149.stdout: debug mon = 20 2024-04-12T03:10:32.341 INFO:teuthology.orchestra.run.smithi149.stdout: debug auth = 20 2024-04-12T03:10:32.341 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min pgs per osd = 4 2024-04-12T03:10:32.341 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min bytes per osd = 10 2024-04-12T03:10:32.341 INFO:teuthology.orchestra.run.smithi149.stdout: mgr/telemetry/nag = false 2024-04-12T03:10:32.341 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.341 INFO:teuthology.orchestra.run.smithi149.stdout:[mon] 2024-04-12T03:10:32.341 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2024-04-12T03:10:32.341 INFO:teuthology.orchestra.run.smithi149.stdout: debug mon = 20 2024-04-12T03:10:32.341 INFO:teuthology.orchestra.run.smithi149.stdout: debug paxos = 20 2024-04-12T03:10:32.341 INFO:teuthology.orchestra.run.smithi149.stdout: debug auth = 20 2024-04-12T03:10:32.341 INFO:teuthology.orchestra.run.smithi149.stdout: mon data avail warn = 5 2024-04-12T03:10:32.341 INFO:teuthology.orchestra.run.smithi149.stdout: mon mgr mkfs grace = 240 2024-04-12T03:10:32.341 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min pgs per osd = 4 2024-04-12T03:10:32.341 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd reporter subtree level = osd 2024-04-12T03:10:32.342 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd prime pg temp = true 2024-04-12T03:10:32.342 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min bytes per osd = 10 2024-04-12T03:10:32.342 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.342 INFO:teuthology.orchestra.run.smithi149.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-12T03:10:32.342 INFO:teuthology.orchestra.run.smithi149.stdout: auth mon ticket ttl = 660 # 11m 2024-04-12T03:10:32.342 INFO:teuthology.orchestra.run.smithi149.stdout: auth service ticket ttl = 240 # 4m 2024-04-12T03:10:32.342 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.342 INFO:teuthology.orchestra.run.smithi149.stdout: # don't complain about insecure global_id in the test suite 2024-04-12T03:10:32.342 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-12T03:10:32.342 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-12T03:10:32.342 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.342 INFO:teuthology.orchestra.run.smithi149.stdout: # 1m isn't quite enough 2024-04-12T03:10:32.342 INFO:teuthology.orchestra.run.smithi149.stdout: mon_down_mkfs_grace = 2m 2024-04-12T03:10:32.342 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.342 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_filestore_osds = false 2024-04-12T03:10:32.343 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:32.343 INFO:teuthology.orchestra.run.smithi149.stdout:[client] 2024-04-12T03:10:32.343 INFO:teuthology.orchestra.run.smithi149.stdout: rgw cache enabled = true 2024-04-12T03:10:32.343 INFO:teuthology.orchestra.run.smithi149.stdout: rgw enable ops log = true 2024-04-12T03:10:32.343 INFO:teuthology.orchestra.run.smithi149.stdout: rgw enable usage log = true 2024-04-12T03:10:32.343 INFO:teuthology.orchestra.run.smithi149.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-12T03:10:32.343 INFO:teuthology.orchestra.run.smithi149.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-12T03:10:32.343 INFO:teuthology.orchestra.run.smithi149.stdout: debug rgw = 20 2024-04-12T03:10:32.343 INFO:teuthology.orchestra.run.smithi149.stdout: debug rgw sync = 20 2024-04-12T03:10:32.343 INFO:teuthology.orchestra.run.smithi149.stdout: rgw crypt require ssl = False 2024-04-12T03:10:32.343 INFO:teuthology.orchestra.run.smithi149.stdout: rgw crypt s3 kms backend = testing 2024-04-12T03:10:32.343 INFO:teuthology.orchestra.run.smithi149.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-12T03:10:32.344 INFO:teuthology.orchestra.run.smithi149.stdout: rgw curl low speed time = 300 2024-04-12T03:10:32.344 INFO:teuthology.orchestra.run.smithi149.stdout: rgw data log num shards = 4 2024-04-12T03:10:32.344 INFO:teuthology.orchestra.run.smithi149.stdout: rgw md log max shards = 4 2024-04-12T03:10:32.344 INFO:teuthology.orchestra.run.smithi149.stdout: rgw sync data inject err probability = 0.1 2024-04-12T03:10:32.344 INFO:teuthology.orchestra.run.smithi149.stdout: rgw sync log trim interval = 0 2024-04-12T03:10:32.344 INFO:teuthology.orchestra.run.smithi149.stdout: rgw sync meta inject err probability = 0.1 2024-04-12T03:10:32.344 INFO:teuthology.orchestra.run.smithi149.stdout: rgw sync obj etag verify = True 2024-04-12T03:10:32.344 INFO:teuthology.orchestra.run.smithi149.stdout: setgroup = ceph 2024-04-12T03:10:32.344 INFO:teuthology.orchestra.run.smithi149.stdout: setuser = ceph 2024-04-12T03:10:32.344 INFO:teuthology.orchestra.run.smithi149.stdout:[mon.a] 2024-04-12T03:10:32.344 INFO:teuthology.orchestra.run.smithi149.stdout:[client.0] 2024-04-12T03:10:32.344 INFO:teuthology.orchestra.run.smithi149.stdout: rgw data notify interval msec = 0 2024-04-12T03:10:32.345 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.0] 2024-04-12T03:10:32.345 INFO:teuthology.orchestra.run.smithi149.stdout: osd_max_omap_entries_per_request = 10 2024-04-12T03:10:32.345 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.1] 2024-04-12T03:10:32.345 INFO:teuthology.orchestra.run.smithi149.stdout: osd_max_omap_entries_per_request = 1000 2024-04-12T03:10:32.345 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.2] 2024-04-12T03:10:32.345 INFO:teuthology.orchestra.run.smithi149.stdout: osd_max_omap_entries_per_request = 10000 2024-04-12T03:10:32.345 INFO:teuthology.orchestra.run.smithi043.stdout:[global] 2024-04-12T03:10:32.345 INFO:teuthology.orchestra.run.smithi043.stdout: chdir = "" 2024-04-12T03:10:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-12T03:10:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: auth supported = cephx 2024-04-12T03:10:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: filestore xattr use omap = true 2024-04-12T03:10:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: mon clock drift allowed = 1.000 2024-04-12T03:10:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: osd crush chooseleaf type = 0 2024-04-12T03:10:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: auth debug = true 2024-04-12T03:10:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: ms die on old message = true 2024-04-12T03:10:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: ms die on bug = true 2024-04-12T03:10:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.346 INFO:teuthology.orchestra.run.smithi043.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-12T03:10:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: mon pg warn max object skew = 0 2024-04-12T03:10:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: # disable pg_autoscaler by default for new pools 2024-04-12T03:10:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-12T03:10:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default size = 2 2024-04-12T03:10:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd allow primary affinity = true 2024-04-12T03:10:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd allow pg remap = true 2024-04-12T03:10:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on legacy crush tunables = false 2024-04-12T03:10:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on crush straw calc version zero = false 2024-04-12T03:10:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on no sortbitwise = false 2024-04-12T03:10:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on osd down out interval zero = false 2024-04-12T03:10:32.347 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on too few osds = false 2024-04-12T03:10:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-12T03:10:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-12T03:10:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: mon_allow_pool_size_one = true 2024-04-12T03:10:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-12T03:10:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: osd default data pool replay window = 5 2024-04-12T03:10:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: mon allow pool delete = true 2024-04-12T03:10:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: mon cluster log file level = debug 2024-04-12T03:10:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: debug asserts on shutdown = true 2024-04-12T03:10:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: mon health detail to clog = false 2024-04-12T03:10:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: mon host = 172.21.15.43 2024-04-12T03:10:32.348 INFO:teuthology.orchestra.run.smithi043.stdout: fsid = f012a16c-f03c-46fb-892c-37001de8aa41 2024-04-12T03:10:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.349 INFO:teuthology.orchestra.run.smithi043.stdout:[osd] 2024-04-12T03:10:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: osd journal size = 100 2024-04-12T03:10:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: osd scrub load threshold = 5.0 2024-04-12T03:10:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: osd scrub max interval = 600 2024-04-12T03:10:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: osd mclock profile = high_recovery_ops 2024-04-12T03:10:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: osd recover clone overlap = true 2024-04-12T03:10:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: osd recovery max chunk = 1048576 2024-04-12T03:10:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug shutdown = true 2024-04-12T03:10:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug op order = true 2024-04-12T03:10:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug verify stray on activate = true 2024-04-12T03:10:32.349 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: osd open classes on start = true 2024-04-12T03:10:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug pg log writeout = true 2024-04-12T03:10:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: osd deep scrub update digest min age = 30 2024-04-12T03:10:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: osd map max advance = 10 2024-04-12T03:10:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: journal zero on create = true 2024-04-12T03:10:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: filestore ondisk finisher threads = 3 2024-04-12T03:10:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: filestore apply finisher threads = 3 2024-04-12T03:10:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: bdev debug aio = true 2024-04-12T03:10:32.350 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug misdirected ops = true 2024-04-12T03:10:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-04-12T03:10:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: debug osd = 20 2024-04-12T03:10:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: rocksdb delete range threshold = 0 2024-04-12T03:10:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.351 INFO:teuthology.orchestra.run.smithi043.stdout:[mgr] 2024-04-12T03:10:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-04-12T03:10:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: debug mgr = 20 2024-04-12T03:10:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: debug mon = 20 2024-04-12T03:10:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: debug auth = 20 2024-04-12T03:10:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min pgs per osd = 4 2024-04-12T03:10:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min bytes per osd = 10 2024-04-12T03:10:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: mgr/telemetry/nag = false 2024-04-12T03:10:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.351 INFO:teuthology.orchestra.run.smithi043.stdout:[mon] 2024-04-12T03:10:32.351 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-04-12T03:10:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: debug mon = 20 2024-04-12T03:10:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: debug paxos = 20 2024-04-12T03:10:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: debug auth = 20 2024-04-12T03:10:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: mon data avail warn = 5 2024-04-12T03:10:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: mon mgr mkfs grace = 240 2024-04-12T03:10:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min pgs per osd = 4 2024-04-12T03:10:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd reporter subtree level = osd 2024-04-12T03:10:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd prime pg temp = true 2024-04-12T03:10:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min bytes per osd = 10 2024-04-12T03:10:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-12T03:10:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: auth mon ticket ttl = 660 # 11m 2024-04-12T03:10:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: auth service ticket ttl = 240 # 4m 2024-04-12T03:10:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.352 INFO:teuthology.orchestra.run.smithi043.stdout: # don't complain about insecure global_id in the test suite 2024-04-12T03:10:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-12T03:10:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-12T03:10:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: # 1m isn't quite enough 2024-04-12T03:10:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: mon_down_mkfs_grace = 2m 2024-04-12T03:10:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_filestore_osds = false 2024-04-12T03:10:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:32.353 INFO:teuthology.orchestra.run.smithi043.stdout:[client] 2024-04-12T03:10:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: rgw cache enabled = true 2024-04-12T03:10:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: rgw enable ops log = true 2024-04-12T03:10:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: rgw enable usage log = true 2024-04-12T03:10:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-12T03:10:32.353 INFO:teuthology.orchestra.run.smithi043.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-12T03:10:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: debug rgw = 20 2024-04-12T03:10:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: debug rgw sync = 20 2024-04-12T03:10:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: rgw crypt require ssl = False 2024-04-12T03:10:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: rgw crypt s3 kms backend = testing 2024-04-12T03:10:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-12T03:10:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: rgw curl low speed time = 300 2024-04-12T03:10:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: rgw data log num shards = 4 2024-04-12T03:10:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: rgw md log max shards = 4 2024-04-12T03:10:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: rgw sync data inject err probability = 0.1 2024-04-12T03:10:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: rgw sync log trim interval = 0 2024-04-12T03:10:32.354 INFO:teuthology.orchestra.run.smithi043.stdout: rgw sync meta inject err probability = 0.1 2024-04-12T03:10:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: rgw sync obj etag verify = True 2024-04-12T03:10:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: setgroup = ceph 2024-04-12T03:10:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: setuser = ceph 2024-04-12T03:10:32.355 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.a] 2024-04-12T03:10:32.355 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-04-12T03:10:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: rgw data notify interval msec = 0 2024-04-12T03:10:32.355 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.0] 2024-04-12T03:10:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: osd_max_omap_entries_per_request = 10 2024-04-12T03:10:32.355 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.1] 2024-04-12T03:10:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: osd_max_omap_entries_per_request = 1000 2024-04-12T03:10:32.355 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.2] 2024-04-12T03:10:32.355 INFO:teuthology.orchestra.run.smithi043.stdout: osd_max_omap_entries_per_request = 10000 2024-04-12T03:10:32.357 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2024-04-12T03:10:32.357 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2024-04-12T03:10:32.470 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-12T03:10:32.470 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:32.470 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2024-04-12T03:10:32.491 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:32.491 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2024-04-12T03:10:32.554 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi043.front.sepia.ceph.com 2024-04-12T03:10:32.555 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:32.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/c1.keyring 2024-04-12T03:10:32.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-12T03:10:32.642 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:32.643 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-04-12T03:10:32.706 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi149.front.sepia.ceph.com 2024-04-12T03:10:32.706 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:10:32.706 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/c1.keyring 2024-04-12T03:10:32.706 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-04-12T03:10:32.751 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:10:32.751 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-04-12T03:10:32.813 INFO:tasks.ceph:Setting up mon nodes... 2024-04-12T03:10:32.814 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-12T03:10:32.814 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2024-04-12T03:10:32.882 INFO:teuthology.orchestra.run.smithi043.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2024-04-12T03:10:32.885 INFO:tasks.ceph:Setting up mds nodes... 2024-04-12T03:10:32.885 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-12T03:10:32.885 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2024-04-12T03:10:32.940 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/c1.client.0.keyring 2024-04-12T03:10:32.955 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2024-04-12T03:10:33.007 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/c1.client.1.keyring 2024-04-12T03:10:33.023 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-12T03:10:33.023 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi043.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-12T03:10:33.023 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2024-04-12T03:10:33.056 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-12T03:10:33.057 INFO:tasks.ceph:role: c1.osd.0 2024-04-12T03:10:33.057 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi043.front.sepia.ceph.com 2024-04-12T03:10:33.057 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-12T03:10:33.134 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-12T03:10:33.134 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-12T03:10:33.134 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-12T03:10:33.134 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-12T03:10:33.134 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-12T03:10:33.134 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2024-04-12T03:10:33.134 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-12T03:10:33.134 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-12T03:10:33.135 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-12T03:10:33.135 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-12T03:10:33.157 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2024-04-12T03:10:33.159 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2024-04-12T03:10:33.159 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2024-04-12T03:10:33.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2024-04-12T03:10:33.288 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2024-04-12T03:10:33.366 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-12T03:10:33.366 INFO:tasks.ceph:role: c1.osd.1 2024-04-12T03:10:33.367 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi043.front.sepia.ceph.com 2024-04-12T03:10:33.367 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-12T03:10:33.443 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-12T03:10:33.443 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-12T03:10:33.443 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-12T03:10:33.443 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-12T03:10:33.444 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-12T03:10:33.444 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2024-04-12T03:10:33.444 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-12T03:10:33.444 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-12T03:10:33.444 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-12T03:10:33.444 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-12T03:10:33.466 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2024-04-12T03:10:33.468 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2024-04-12T03:10:33.469 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2024-04-12T03:10:33.557 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2024-04-12T03:10:33.595 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2024-04-12T03:10:33.669 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-12T03:10:33.669 INFO:tasks.ceph:role: c1.osd.2 2024-04-12T03:10:33.669 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi043.front.sepia.ceph.com 2024-04-12T03:10:33.670 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-12T03:10:33.744 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-12T03:10:33.744 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-12T03:10:33.744 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-12T03:10:33.744 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-12T03:10:33.744 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-12T03:10:33.744 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2024-04-12T03:10:33.744 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-12T03:10:33.744 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-12T03:10:33.744 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-12T03:10:33.745 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-12T03:10:33.767 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2024-04-12T03:10:33.769 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2024-04-12T03:10:33.769 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2024-04-12T03:10:33.857 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2024-04-12T03:10:33.895 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-12T03:10:33.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:33.993+0000 7f6c9648a7c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2024-04-12T03:10:33.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:33.993+0000 7f6c9648a7c0 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2024-04-12T03:10:33.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:33.993+0000 7f6c9648a7c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-12T03:10:33.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:33.993+0000 7f6c9648a7c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-12T03:10:33.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:33.993+0000 7f6c9648a7c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-04-12T03:10:33.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:33.994+0000 7f6c9648a7c0 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2024-04-12T03:10:36.076 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2024-04-12T03:10:36.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-12T03:10:36.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:36.208+0000 7fb7f8cbc7c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2024-04-12T03:10:36.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:36.208+0000 7fb7f8cbc7c0 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2024-04-12T03:10:36.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:36.208+0000 7fb7f8cbc7c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-12T03:10:36.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:36.208+0000 7fb7f8cbc7c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-12T03:10:36.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:36.208+0000 7fb7f8cbc7c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-04-12T03:10:36.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:36.209+0000 7fb7f8cbc7c0 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2024-04-12T03:10:38.292 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2024-04-12T03:10:38.327 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2024-04-12T03:10:38.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:38.424+0000 7f27d77257c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2024-04-12T03:10:38.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:38.424+0000 7f27d77257c0 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2024-04-12T03:10:38.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:38.424+0000 7f27d77257c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-12T03:10:38.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:38.424+0000 7f27d77257c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-12T03:10:38.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:38.424+0000 7f27d77257c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-04-12T03:10:38.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:10:38.425+0000 7f27d77257c0 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2024-04-12T03:10:40.500 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2024-04-12T03:10:40.535 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-12T03:10:40.535 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:40.535 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2024-04-12T03:10:40.609 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:40.609 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2024-04-12T03:10:40.683 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:40.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2024-04-12T03:10:40.758 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:40.759 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2024-04-12T03:10:40.833 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:40.834 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2024-04-12T03:10:40.895 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:40.895 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2024-04-12T03:10:40.958 INFO:tasks.ceph:Adding keys to all mons... 2024-04-12T03:10:40.958 DEBUG:teuthology.orchestra.run.smithi043:> sudo tee -a /etc/ceph/c1.keyring 2024-04-12T03:10:41.031 INFO:teuthology.orchestra.run.smithi043.stdout:[mgr.x] 2024-04-12T03:10:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQAophhmocSNNBAAnD2774+atVIDD4wTYso2Qw== 2024-04-12T03:10:41.031 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.0] 2024-04-12T03:10:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQApphhm7dpGOxAALyIJ4hiMQnALLRkcgk/M1Q== 2024-04-12T03:10:41.031 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.1] 2024-04-12T03:10:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQAsphhmRwCCDBAAXsH2dLsjqhZQks9qLb5dFw== 2024-04-12T03:10:41.031 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.2] 2024-04-12T03:10:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQAuphhmCHpeGRAAiVeQMikU3Ka4taQdPbnusw== 2024-04-12T03:10:41.032 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-04-12T03:10:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQAophhmpPn0NxAAwz83emJm3+CHsO00TdbO5Q== 2024-04-12T03:10:41.032 INFO:teuthology.orchestra.run.smithi043.stdout:[client.1] 2024-04-12T03:10:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQApphhmW7xnABAAucKdUgfxgf1v0abL1WgT2w== 2024-04-12T03:10:41.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-12T03:10:41.139 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-12T03:10:41.202 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-12T03:10:41.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-12T03:10:41.323 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-12T03:10:41.387 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-12T03:10:41.450 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-12T03:10:41.450 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2024-04-12T03:10:41.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2024-04-12T03:10:41.589 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2024-04-12T03:10:41.621 DEBUG:teuthology.orchestra.run.smithi043:> rm -- /home/ubuntu/cephtest/c1.monmap 2024-04-12T03:10:41.682 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2024-04-12T03:10:41.683 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2024-04-12T03:10:41.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2024-04-12T03:10:41.725 INFO:tasks.ceph.c1.mon.a:Started 2024-04-12T03:10:41.725 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2024-04-12T03:10:41.725 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2024-04-12T03:10:41.726 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2024-04-12T03:10:41.728 INFO:tasks.ceph.c1.mgr.x:Started 2024-04-12T03:10:41.728 INFO:tasks.ceph:Setting crush tunables to default 2024-04-12T03:10:41.728 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster c1 osd crush tunables default 2024-04-12T03:10:41.966 INFO:teuthology.orchestra.run.smithi043.stderr:adjusted tunables profile to default 2024-04-12T03:10:41.979 INFO:tasks.ceph:check_enable_crimson: False 2024-04-12T03:10:41.979 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2024-04-12T03:10:41.980 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:41.980 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2024-04-12T03:10:42.009 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:42.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2024-04-12T03:10:42.080 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:42.080 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2024-04-12T03:10:42.153 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster c1 osd new e9b49cf3-23ca-4cdd-b14b-93c1dea271e3 0 2024-04-12T03:10:42.428 INFO:teuthology.orchestra.run.smithi043.stdout:0 2024-04-12T03:10:42.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster c1 osd new 55a0ea3e-a92a-4ae6-be81-8f9e2e233428 1 2024-04-12T03:10:42.500 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:42.499+0000 7f7cd4adb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T03:10:42.593 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:42.592+0000 7f7cd4adb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T03:10:42.677 INFO:teuthology.orchestra.run.smithi043.stdout:1 2024-04-12T03:10:42.677 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:42.676+0000 7f7cd4adb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T03:10:42.691 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster c1 osd new a66fa46f-ae95-4bab-a586-751faedf2fa0 2 2024-04-12T03:10:42.764 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:42.762+0000 7f7cd4adb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T03:10:42.929 INFO:teuthology.orchestra.run.smithi043.stdout:2 2024-04-12T03:10:42.943 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2024-04-12T03:10:42.943 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2024-04-12T03:10:42.945 INFO:tasks.ceph.c1.osd.0:Started 2024-04-12T03:10:42.945 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2024-04-12T03:10:42.945 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2024-04-12T03:10:42.947 INFO:tasks.ceph.c1.osd.1:Started 2024-04-12T03:10:42.948 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2024-04-12T03:10:42.948 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2024-04-12T03:10:42.950 INFO:tasks.ceph.c1.osd.2:Started 2024-04-12T03:10:42.950 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-12T03:10:42.979 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:42.978+0000 7f7cd4adb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T03:10:43.182 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:43.182 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":5,"fsid":"f012a16c-f03c-46fb-892c-37001de8aa41","created":"2024-04-12T03:10:41.861308+0000","modified":"2024-04-12T03:10:42.926712+0000","last_up_change":"0.000000","last_in_change":"2024-04-12T03:10:42.926712+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e9b49cf3-23ca-4cdd-b14b-93c1dea271e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"55a0ea3e-a92a-4ae6-be81-8f9e2e233428","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a66fa46f-ae95-4bab-a586-751faedf2fa0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T03:10:43.200 INFO:tasks.ceph.ceph_manager.c1:[] 2024-04-12T03:10:43.200 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-12T03:10:43.207 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:43.205+0000 7f7cd4adb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T03:10:43.334 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:43.332+0000 7f7cd4adb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T03:10:43.440 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:43.438+0000 7f7cd4adb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T03:10:43.503 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-04-12T03:10:43.526 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:43.524+0000 7f7cd4adb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T03:10:43.560 INFO:tasks.ceph.c1.osd.0.smithi043.stderr:2024-04-12T03:10:43.558+0000 7f5b5cc997c0 -1 Falling back to public interface 2024-04-12T03:10:43.565 INFO:tasks.ceph.c1.osd.1.smithi043.stderr:2024-04-12T03:10:43.564+0000 7ff4295ed7c0 -1 Falling back to public interface 2024-04-12T03:10:43.597 INFO:tasks.ceph.c1.osd.2.smithi043.stderr:2024-04-12T03:10:43.595+0000 7f00c0cf17c0 -1 Falling back to public interface 2024-04-12T03:10:43.746 INFO:teuthology.misc.health.smithi043.stdout: 2024-04-12T03:10:43.747 INFO:teuthology.misc.health.smithi043.stdout:{"epoch":5,"fsid":"f012a16c-f03c-46fb-892c-37001de8aa41","created":"2024-04-12T03:10:41.861308+0000","modified":"2024-04-12T03:10:42.926712+0000","last_up_change":"0.000000","last_in_change":"2024-04-12T03:10:42.926712+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e9b49cf3-23ca-4cdd-b14b-93c1dea271e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"55a0ea3e-a92a-4ae6-be81-8f9e2e233428","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a66fa46f-ae95-4bab-a586-751faedf2fa0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T03:10:43.771 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-12T03:10:43.797 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:43.795+0000 7f7cd4adb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T03:10:43.878 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:43.876+0000 7f7cd4adb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T03:10:44.299 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:44.298+0000 7f7cd4adb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T03:10:44.665 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:44.664+0000 7f7cd4adb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T03:10:44.846 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:44.844+0000 7f7cd4adb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-12T03:10:44.935 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:44.933+0000 7f7cd4adb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T03:10:45.726 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:45.725+0000 7f7cd4adb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T03:10:45.820 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:45.818+0000 7f7cd4adb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T03:10:45.970 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:45.968+0000 7f7cd4adb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T03:10:46.299 INFO:tasks.ceph.c1.osd.0.smithi043.stderr:2024-04-12T03:10:46.297+0000 7f5b5cc997c0 -1 osd.0 0 log_to_monitors true 2024-04-12T03:10:46.331 INFO:tasks.ceph.c1.osd.1.smithi043.stderr:2024-04-12T03:10:46.329+0000 7ff4295ed7c0 -1 osd.1 0 log_to_monitors true 2024-04-12T03:10:46.352 INFO:tasks.ceph.c1.osd.2.smithi043.stderr:2024-04-12T03:10:46.351+0000 7f00c0cf17c0 -1 osd.2 0 log_to_monitors true 2024-04-12T03:10:46.618 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:46.617+0000 7f7cd4adb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T03:10:46.809 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:46.808+0000 7f7cd4adb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T03:10:46.997 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:46.995+0000 7f7cd4adb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T03:10:47.874 INFO:tasks.ceph.c1.osd.0.smithi043.stderr:2024-04-12T03:10:47.872+0000 7f5b51051700 -1 osd.0 0 waiting for initial osdmap 2024-04-12T03:10:47.889 INFO:tasks.ceph.c1.osd.0.smithi043.stderr:2024-04-12T03:10:47.888+0000 7f5b48c2b700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T03:10:47.923 INFO:tasks.ceph.c1.osd.1.smithi043.stderr:2024-04-12T03:10:47.921+0000 7ff41d9a5700 -1 osd.1 0 waiting for initial osdmap 2024-04-12T03:10:47.932 INFO:tasks.ceph.c1.osd.1.smithi043.stderr:2024-04-12T03:10:47.931+0000 7ff41557f700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T03:10:48.038 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:48.036+0000 7f7cd4adb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T03:10:48.305 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:48.304+0000 7f7cd4adb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T03:10:48.397 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:48.395+0000 7f7cd4adb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T03:10:48.725 INFO:tasks.ceph.c1.osd.2.smithi043.stderr:2024-04-12T03:10:48.723+0000 7f00b50a9700 -1 osd.2 0 waiting for initial osdmap 2024-04-12T03:10:48.740 INFO:tasks.ceph.c1.osd.2.smithi043.stderr:2024-04-12T03:10:48.738+0000 7f00acc83700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T03:10:48.889 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:48.888+0000 7f7cd4adb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T03:10:49.901 INFO:tasks.ceph.c1.mgr.x.smithi043.stderr:2024-04-12T03:10:49.899+0000 7f7c8fd55700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-12T03:10:50.073 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-04-12T03:10:50.432 INFO:teuthology.misc.health.smithi043.stdout: 2024-04-12T03:10:50.432 INFO:teuthology.misc.health.smithi043.stdout:{"epoch":9,"fsid":"f012a16c-f03c-46fb-892c-37001de8aa41","created":"2024-04-12T03:10:41.861308+0000","modified":"2024-04-12T03:10:49.725306+0000","last_up_change":"2024-04-12T03:10:49.725306+0000","last_in_change":"2024-04-12T03:10:42.926712+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e9b49cf3-23ca-4cdd-b14b-93c1dea271e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":40791},{"type":"v1","addr":"172.21.15.43:6801","nonce":40791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":40791},{"type":"v1","addr":"172.21.15.43:6803","nonce":40791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":40791},{"type":"v1","addr":"172.21.15.43:6807","nonce":40791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":40791},{"type":"v1","addr":"172.21.15.43:6805","nonce":40791}]},"public_addr":"172.21.15.43:6801/40791","cluster_addr":"172.21.15.43:6803/40791","heartbeat_back_addr":"172.21.15.43:6807/40791","heartbeat_front_addr":"172.21.15.43:6805/40791","state":["exists","up"]},{"osd":1,"uuid":"55a0ea3e-a92a-4ae6-be81-8f9e2e233428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":40792},{"type":"v1","addr":"172.21.15.43:6809","nonce":40792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":40792},{"type":"v1","addr":"172.21.15.43:6811","nonce":40792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":40792},{"type":"v1","addr":"172.21.15.43:6815","nonce":40792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":40792},{"type":"v1","addr":"172.21.15.43:6813","nonce":40792}]},"public_addr":"172.21.15.43:6809/40792","cluster_addr":"172.21.15.43:6811/40792","heartbeat_back_addr":"172.21.15.43:6815/40792","heartbeat_front_addr":"172.21.15.43:6813/40792","state":["exists","up"]},{"osd":2,"uuid":"a66fa46f-ae95-4bab-a586-751faedf2fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":40817},{"type":"v1","addr":"172.21.15.43:6817","nonce":40817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":40817},{"type":"v1","addr":"172.21.15.43:6819","nonce":40817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":40817},{"type":"v1","addr":"172.21.15.43:6823","nonce":40817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":40817},{"type":"v1","addr":"172.21.15.43:6821","nonce":40817}]},"public_addr":"172.21.15.43:6817/40817","cluster_addr":"172.21.15.43:6819/40817","heartbeat_back_addr":"172.21.15.43:6823/40817","heartbeat_front_addr":"172.21.15.43:6821/40817","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T03:10:47.293310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T03:10:47.295855+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T03:10:50.447 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-12T03:10:50.447 INFO:tasks.ceph:Creating RBD pool 2024-04-12T03:10:50.447 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster c1 osd pool create rbd 8 2024-04-12T03:10:50.909 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'rbd' created 2024-04-12T03:10:50.923 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-12T03:10:51.918 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-12T03:10:51.933 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2024-04-12T03:10:51.933 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2024-04-12T03:10:51.933 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2024-04-12T03:10:51.933 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2024-04-12T03:10:51.934 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-12T03:10:52.369 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:52.383 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":4,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6824","nonce":40412},{"type":"v1","addr":"172.21.15.43:6825","nonce":40412}]},"active_addr":"172.21.15.43:6825/40412","active_change":"2024-04-12T03:10:48.890389+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":176964569}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":2290309617}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.43:0","nonce":703866945}]}]} 2024-04-12T03:10:52.656 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2024-04-12T03:10:52.657 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2024-04-12T03:10:52.657 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-12T03:10:53.026 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:53.026 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":12,"fsid":"f012a16c-f03c-46fb-892c-37001de8aa41","created":"2024-04-12T03:10:41.861308+0000","modified":"2024-04-12T03:10:52.907421+0000","last_up_change":"2024-04-12T03:10:49.725306+0000","last_in_change":"2024-04-12T03:10:42.926712+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-12T03:10:50.851116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-12T03:10:50.951086+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e9b49cf3-23ca-4cdd-b14b-93c1dea271e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":40791},{"type":"v1","addr":"172.21.15.43:6801","nonce":40791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":40791},{"type":"v1","addr":"172.21.15.43:6803","nonce":40791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":40791},{"type":"v1","addr":"172.21.15.43:6807","nonce":40791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":40791},{"type":"v1","addr":"172.21.15.43:6805","nonce":40791}]},"public_addr":"172.21.15.43:6801/40791","cluster_addr":"172.21.15.43:6803/40791","heartbeat_back_addr":"172.21.15.43:6807/40791","heartbeat_front_addr":"172.21.15.43:6805/40791","state":["exists","up"]},{"osd":1,"uuid":"55a0ea3e-a92a-4ae6-be81-8f9e2e233428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":40792},{"type":"v1","addr":"172.21.15.43:6809","nonce":40792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":40792},{"type":"v1","addr":"172.21.15.43:6811","nonce":40792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":40792},{"type":"v1","addr":"172.21.15.43:6815","nonce":40792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":40792},{"type":"v1","addr":"172.21.15.43:6813","nonce":40792}]},"public_addr":"172.21.15.43:6809/40792","cluster_addr":"172.21.15.43:6811/40792","heartbeat_back_addr":"172.21.15.43:6815/40792","heartbeat_front_addr":"172.21.15.43:6813/40792","state":["exists","up"]},{"osd":2,"uuid":"a66fa46f-ae95-4bab-a586-751faedf2fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":40817},{"type":"v1","addr":"172.21.15.43:6817","nonce":40817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":40817},{"type":"v1","addr":"172.21.15.43:6819","nonce":40817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":40817},{"type":"v1","addr":"172.21.15.43:6823","nonce":40817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":40817},{"type":"v1","addr":"172.21.15.43:6821","nonce":40817}]},"public_addr":"172.21.15.43:6817/40817","cluster_addr":"172.21.15.43:6819/40817","heartbeat_back_addr":"172.21.15.43:6823/40817","heartbeat_front_addr":"172.21.15.43:6821/40817","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T03:10:47.293310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T03:10:47.295855+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T03:10:47.347710+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T03:10:53.042 INFO:tasks.ceph.ceph_manager.c1:all up! 2024-04-12T03:10:53.042 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-04-12T03:10:53.426 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:53.426 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":12,"fsid":"f012a16c-f03c-46fb-892c-37001de8aa41","created":"2024-04-12T03:10:41.861308+0000","modified":"2024-04-12T03:10:52.907421+0000","last_up_change":"2024-04-12T03:10:49.725306+0000","last_in_change":"2024-04-12T03:10:42.926712+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-12T03:10:50.851116+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-12T03:10:50.951086+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e9b49cf3-23ca-4cdd-b14b-93c1dea271e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":40791},{"type":"v1","addr":"172.21.15.43:6801","nonce":40791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6802","nonce":40791},{"type":"v1","addr":"172.21.15.43:6803","nonce":40791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6806","nonce":40791},{"type":"v1","addr":"172.21.15.43:6807","nonce":40791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6804","nonce":40791},{"type":"v1","addr":"172.21.15.43:6805","nonce":40791}]},"public_addr":"172.21.15.43:6801/40791","cluster_addr":"172.21.15.43:6803/40791","heartbeat_back_addr":"172.21.15.43:6807/40791","heartbeat_front_addr":"172.21.15.43:6805/40791","state":["exists","up"]},{"osd":1,"uuid":"55a0ea3e-a92a-4ae6-be81-8f9e2e233428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6808","nonce":40792},{"type":"v1","addr":"172.21.15.43:6809","nonce":40792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":40792},{"type":"v1","addr":"172.21.15.43:6811","nonce":40792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6814","nonce":40792},{"type":"v1","addr":"172.21.15.43:6815","nonce":40792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":40792},{"type":"v1","addr":"172.21.15.43:6813","nonce":40792}]},"public_addr":"172.21.15.43:6809/40792","cluster_addr":"172.21.15.43:6811/40792","heartbeat_back_addr":"172.21.15.43:6815/40792","heartbeat_front_addr":"172.21.15.43:6813/40792","state":["exists","up"]},{"osd":2,"uuid":"a66fa46f-ae95-4bab-a586-751faedf2fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":40817},{"type":"v1","addr":"172.21.15.43:6817","nonce":40817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":40817},{"type":"v1","addr":"172.21.15.43:6819","nonce":40817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":40817},{"type":"v1","addr":"172.21.15.43:6823","nonce":40817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6820","nonce":40817},{"type":"v1","addr":"172.21.15.43:6821","nonce":40817}]},"public_addr":"172.21.15.43:6817/40817","cluster_addr":"172.21.15.43:6819/40817","heartbeat_back_addr":"172.21.15.43:6823/40817","heartbeat_front_addr":"172.21.15.43:6821/40817","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T03:10:47.293310+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T03:10:47.295855+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T03:10:47.347710+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T03:10:53.442 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2024-04-12T03:10:53.443 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2024-04-12T03:10:53.443 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2024-04-12T03:10:53.640 INFO:teuthology.orchestra.run.smithi043.stdout:34359738370 2024-04-12T03:10:53.640 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-04-12T03:10:53.645 INFO:teuthology.orchestra.run.smithi043.stdout:38654705666 2024-04-12T03:10:53.646 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-04-12T03:10:53.735 INFO:teuthology.orchestra.run.smithi043.stdout:34359738370 2024-04-12T03:10:53.735 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-04-12T03:10:54.031 INFO:teuthology.orchestra.run.smithi043.stdout:34359738369 2024-04-12T03:10:54.048 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.0 2024-04-12T03:10:54.197 INFO:teuthology.orchestra.run.smithi043.stdout:34359738369 2024-04-12T03:10:54.204 INFO:teuthology.orchestra.run.smithi043.stdout:38654705665 2024-04-12T03:10:54.216 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738369 for osd.1 2024-04-12T03:10:54.223 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705666 got 38654705665 for osd.2 2024-04-12T03:10:55.049 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-04-12T03:10:55.216 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-04-12T03:10:55.224 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-04-12T03:10:55.434 INFO:teuthology.orchestra.run.smithi043.stdout:34359738370 2024-04-12T03:10:55.450 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.0 2024-04-12T03:10:55.450 DEBUG:teuthology.parallel:result is None 2024-04-12T03:10:55.588 INFO:teuthology.orchestra.run.smithi043.stdout:34359738370 2024-04-12T03:10:55.604 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738370 got 34359738370 for osd.1 2024-04-12T03:10:55.604 DEBUG:teuthology.parallel:result is None 2024-04-12T03:10:55.632 INFO:teuthology.orchestra.run.smithi043.stdout:38654705666 2024-04-12T03:10:55.648 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705666 got 38654705666 for osd.2 2024-04-12T03:10:55.648 DEBUG:teuthology.parallel:result is None 2024-04-12T03:10:55.648 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-04-12T03:10:55.648 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-12T03:10:56.048 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:56.050 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-12T03:10:56.064 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-12T03:10:54.896242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81360,"kb_used_data":300,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314491440,"statfs":{"total":322122547200,"available":322039234560,"internally_reserved":0,"allocated":307200,"data_stored":77910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.942433"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:10:53.336472+0000","last_change":"2024-04-12T03:10:51.916455+0000","last_active":"2024-04-12T03:10:53.336472+0000","last_peered":"2024-04-12T03:10:53.336472+0000","last_clean":"2024-04-12T03:10:53.336472+0000","last_became_active":"2024-04-12T03:10:51.916206+0000","last_became_peered":"2024-04-12T03:10:51.916206+0000","last_unstale":"2024-04-12T03:10:53.336472+0000","last_undegraded":"2024-04-12T03:10:53.336472+0000","last_fullsized":"2024-04-12T03:10:53.336472+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_clean_scrub_stamp":"2024-04-12T03:10:50.901216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T06:25:22.787379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:10:53.336410+0000","last_change":"2024-04-12T03:10:51.918035+0000","last_active":"2024-04-12T03:10:53.336410+0000","last_peered":"2024-04-12T03:10:53.336410+0000","last_clean":"2024-04-12T03:10:53.336410+0000","last_became_active":"2024-04-12T03:10:51.917774+0000","last_became_peered":"2024-04-12T03:10:51.917774+0000","last_unstale":"2024-04-12T03:10:53.336410+0000","last_undegraded":"2024-04-12T03:10:53.336410+0000","last_fullsized":"2024-04-12T03:10:53.336410+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_clean_scrub_stamp":"2024-04-12T03:10:50.901216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T12:07:05.110341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:10:52.910248+0000","last_change":"2024-04-12T03:10:51.915684+0000","last_active":"2024-04-12T03:10:52.910248+0000","last_peered":"2024-04-12T03:10:52.910248+0000","last_clean":"2024-04-12T03:10:52.910248+0000","last_became_active":"2024-04-12T03:10:51.914737+0000","last_became_peered":"2024-04-12T03:10:51.914737+0000","last_unstale":"2024-04-12T03:10:52.910248+0000","last_undegraded":"2024-04-12T03:10:52.910248+0000","last_fullsized":"2024-04-12T03:10:52.910248+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_clean_scrub_stamp":"2024-04-12T03:10:50.901216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T13:22:17.096914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:10:53.336591+0000","last_change":"2024-04-12T03:10:51.914346+0000","last_active":"2024-04-12T03:10:53.336591+0000","last_peered":"2024-04-12T03:10:53.336591+0000","last_clean":"2024-04-12T03:10:53.336591+0000","last_became_active":"2024-04-12T03:10:51.914080+0000","last_became_peered":"2024-04-12T03:10:51.914080+0000","last_unstale":"2024-04-12T03:10:53.336591+0000","last_undegraded":"2024-04-12T03:10:53.336591+0000","last_fullsized":"2024-04-12T03:10:53.336591+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_clean_scrub_stamp":"2024-04-12T03:10:50.901216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T11:09:58.001630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T03:10:51.917394+0000","last_change":"2024-04-12T03:10:51.915158+0000","last_active":"2024-04-12T03:10:51.917394+0000","last_peered":"2024-04-12T03:10:51.917394+0000","last_clean":"2024-04-12T03:10:51.917394+0000","last_became_active":"2024-04-12T03:10:51.914867+0000","last_became_peered":"2024-04-12T03:10:51.914867+0000","last_unstale":"2024-04-12T03:10:51.917394+0000","last_undegraded":"2024-04-12T03:10:51.917394+0000","last_fullsized":"2024-04-12T03:10:51.917394+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_clean_scrub_stamp":"2024-04-12T03:10:50.901216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T04:21:47.406716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"12'85","reported_seq":121,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:10:53.603531+0000","last_change":"2024-04-12T03:10:53.337352+0000","last_active":"2024-04-12T03:10:53.603531+0000","last_peered":"2024-04-12T03:10:53.603531+0000","last_clean":"2024-04-12T03:10:53.603531+0000","last_became_active":"2024-04-12T03:10:53.337082+0000","last_became_peered":"2024-04-12T03:10:53.337082+0000","last_unstale":"2024-04-12T03:10:53.603531+0000","last_undegraded":"2024-04-12T03:10:53.603531+0000","last_fullsized":"2024-04-12T03:10:53.603531+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:51.902733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:51.902733+0000","last_clean_scrub_stamp":"2024-04-12T03:10:51.902733+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T08:52:45.356976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:10:53.336377+0000","last_change":"2024-04-12T03:10:51.914136+0000","last_active":"2024-04-12T03:10:53.336377+0000","last_peered":"2024-04-12T03:10:53.336377+0000","last_clean":"2024-04-12T03:10:53.336377+0000","last_became_active":"2024-04-12T03:10:51.913836+0000","last_became_peered":"2024-04-12T03:10:51.913836+0000","last_unstale":"2024-04-12T03:10:53.336377+0000","last_undegraded":"2024-04-12T03:10:53.336377+0000","last_fullsized":"2024-04-12T03:10:53.336377+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_clean_scrub_stamp":"2024-04-12T03:10:50.901216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T03:33:29.239836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:10:53.336378+0000","last_change":"2024-04-12T03:10:51.914033+0000","last_active":"2024-04-12T03:10:53.336378+0000","last_peered":"2024-04-12T03:10:53.336378+0000","last_clean":"2024-04-12T03:10:53.336378+0000","last_became_active":"2024-04-12T03:10:51.913728+0000","last_became_peered":"2024-04-12T03:10:51.913728+0000","last_unstale":"2024-04-12T03:10:53.336378+0000","last_undegraded":"2024-04-12T03:10:53.336378+0000","last_fullsized":"2024-04-12T03:10:53.336378+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_clean_scrub_stamp":"2024-04-12T03:10:50.901216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T14:38:52.785648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:10:52.910254+0000","last_change":"2024-04-12T03:10:51.915698+0000","last_active":"2024-04-12T03:10:52.910254+0000","last_peered":"2024-04-12T03:10:52.910254+0000","last_clean":"2024-04-12T03:10:52.910254+0000","last_became_active":"2024-04-12T03:10:51.914745+0000","last_became_peered":"2024-04-12T03:10:51.914745+0000","last_unstale":"2024-04-12T03:10:52.910254+0000","last_undegraded":"2024-04-12T03:10:52.910254+0000","last_fullsized":"2024-04-12T03:10:52.910254+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_clean_scrub_stamp":"2024-04-12T03:10:50.901216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T13:42:42.705033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27128,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830472,"statfs":{"total":107374182400,"available":107346403328,"internally_reserved":0,"allocated":110592,"data_stored":28882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27116,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830484,"statfs":{"total":107374182400,"available":107346415616,"internally_reserved":0,"allocated":98304,"data_stored":24514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27116,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830484,"statfs":{"total":107374182400,"available":107346415616,"internally_reserved":0,"allocated":98304,"data_stored":24514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T03:10:56.065 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-04-12T03:10:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:56.426 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-12T03:10:56.439 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-12T03:10:54.896242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81360,"kb_used_data":300,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314491440,"statfs":{"total":322122547200,"available":322039234560,"internally_reserved":0,"allocated":307200,"data_stored":77910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.942433"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:10:53.336472+0000","last_change":"2024-04-12T03:10:51.916455+0000","last_active":"2024-04-12T03:10:53.336472+0000","last_peered":"2024-04-12T03:10:53.336472+0000","last_clean":"2024-04-12T03:10:53.336472+0000","last_became_active":"2024-04-12T03:10:51.916206+0000","last_became_peered":"2024-04-12T03:10:51.916206+0000","last_unstale":"2024-04-12T03:10:53.336472+0000","last_undegraded":"2024-04-12T03:10:53.336472+0000","last_fullsized":"2024-04-12T03:10:53.336472+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_clean_scrub_stamp":"2024-04-12T03:10:50.901216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T06:25:22.787379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:10:53.336410+0000","last_change":"2024-04-12T03:10:51.918035+0000","last_active":"2024-04-12T03:10:53.336410+0000","last_peered":"2024-04-12T03:10:53.336410+0000","last_clean":"2024-04-12T03:10:53.336410+0000","last_became_active":"2024-04-12T03:10:51.917774+0000","last_became_peered":"2024-04-12T03:10:51.917774+0000","last_unstale":"2024-04-12T03:10:53.336410+0000","last_undegraded":"2024-04-12T03:10:53.336410+0000","last_fullsized":"2024-04-12T03:10:53.336410+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_clean_scrub_stamp":"2024-04-12T03:10:50.901216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T12:07:05.110341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:10:52.910248+0000","last_change":"2024-04-12T03:10:51.915684+0000","last_active":"2024-04-12T03:10:52.910248+0000","last_peered":"2024-04-12T03:10:52.910248+0000","last_clean":"2024-04-12T03:10:52.910248+0000","last_became_active":"2024-04-12T03:10:51.914737+0000","last_became_peered":"2024-04-12T03:10:51.914737+0000","last_unstale":"2024-04-12T03:10:52.910248+0000","last_undegraded":"2024-04-12T03:10:52.910248+0000","last_fullsized":"2024-04-12T03:10:52.910248+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_clean_scrub_stamp":"2024-04-12T03:10:50.901216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T13:22:17.096914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:10:53.336591+0000","last_change":"2024-04-12T03:10:51.914346+0000","last_active":"2024-04-12T03:10:53.336591+0000","last_peered":"2024-04-12T03:10:53.336591+0000","last_clean":"2024-04-12T03:10:53.336591+0000","last_became_active":"2024-04-12T03:10:51.914080+0000","last_became_peered":"2024-04-12T03:10:51.914080+0000","last_unstale":"2024-04-12T03:10:53.336591+0000","last_undegraded":"2024-04-12T03:10:53.336591+0000","last_fullsized":"2024-04-12T03:10:53.336591+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_clean_scrub_stamp":"2024-04-12T03:10:50.901216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T11:09:58.001630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T03:10:51.917394+0000","last_change":"2024-04-12T03:10:51.915158+0000","last_active":"2024-04-12T03:10:51.917394+0000","last_peered":"2024-04-12T03:10:51.917394+0000","last_clean":"2024-04-12T03:10:51.917394+0000","last_became_active":"2024-04-12T03:10:51.914867+0000","last_became_peered":"2024-04-12T03:10:51.914867+0000","last_unstale":"2024-04-12T03:10:51.917394+0000","last_undegraded":"2024-04-12T03:10:51.917394+0000","last_fullsized":"2024-04-12T03:10:51.917394+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_clean_scrub_stamp":"2024-04-12T03:10:50.901216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T04:21:47.406716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"12'85","reported_seq":121,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:10:53.603531+0000","last_change":"2024-04-12T03:10:53.337352+0000","last_active":"2024-04-12T03:10:53.603531+0000","last_peered":"2024-04-12T03:10:53.603531+0000","last_clean":"2024-04-12T03:10:53.603531+0000","last_became_active":"2024-04-12T03:10:53.337082+0000","last_became_peered":"2024-04-12T03:10:53.337082+0000","last_unstale":"2024-04-12T03:10:53.603531+0000","last_undegraded":"2024-04-12T03:10:53.603531+0000","last_fullsized":"2024-04-12T03:10:53.603531+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:51.902733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:51.902733+0000","last_clean_scrub_stamp":"2024-04-12T03:10:51.902733+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T08:52:45.356976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:10:53.336377+0000","last_change":"2024-04-12T03:10:51.914136+0000","last_active":"2024-04-12T03:10:53.336377+0000","last_peered":"2024-04-12T03:10:53.336377+0000","last_clean":"2024-04-12T03:10:53.336377+0000","last_became_active":"2024-04-12T03:10:51.913836+0000","last_became_peered":"2024-04-12T03:10:51.913836+0000","last_unstale":"2024-04-12T03:10:53.336377+0000","last_undegraded":"2024-04-12T03:10:53.336377+0000","last_fullsized":"2024-04-12T03:10:53.336377+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_clean_scrub_stamp":"2024-04-12T03:10:50.901216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T03:33:29.239836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:10:53.336378+0000","last_change":"2024-04-12T03:10:51.914033+0000","last_active":"2024-04-12T03:10:53.336378+0000","last_peered":"2024-04-12T03:10:53.336378+0000","last_clean":"2024-04-12T03:10:53.336378+0000","last_became_active":"2024-04-12T03:10:51.913728+0000","last_became_peered":"2024-04-12T03:10:51.913728+0000","last_unstale":"2024-04-12T03:10:53.336378+0000","last_undegraded":"2024-04-12T03:10:53.336378+0000","last_fullsized":"2024-04-12T03:10:53.336378+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_clean_scrub_stamp":"2024-04-12T03:10:50.901216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T14:38:52.785648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:10:52.910254+0000","last_change":"2024-04-12T03:10:51.915698+0000","last_active":"2024-04-12T03:10:52.910254+0000","last_peered":"2024-04-12T03:10:52.910254+0000","last_clean":"2024-04-12T03:10:52.910254+0000","last_became_active":"2024-04-12T03:10:51.914745+0000","last_became_peered":"2024-04-12T03:10:51.914745+0000","last_unstale":"2024-04-12T03:10:52.910254+0000","last_undegraded":"2024-04-12T03:10:52.910254+0000","last_fullsized":"2024-04-12T03:10:52.910254+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:10:50.901216+0000","last_clean_scrub_stamp":"2024-04-12T03:10:50.901216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T13:42:42.705033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27128,"kb_used_data":108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830472,"statfs":{"total":107374182400,"available":107346403328,"internally_reserved":0,"allocated":110592,"data_stored":28882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]}]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27116,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830484,"statfs":{"total":107374182400,"available":107346415616,"internally_reserved":0,"allocated":98304,"data_stored":24514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27116,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830484,"statfs":{"total":107374182400,"available":107346415616,"internally_reserved":0,"allocated":98304,"data_stored":24514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T03:10:56.440 INFO:tasks.ceph.ceph_manager.c1:clean! 2024-04-12T03:10:56.440 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2024-04-12T03:10:56.440 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2024-04-12T03:10:56.440 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2024-04-12T03:10:56.847 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:56.847 INFO:teuthology.orchestra.run.smithi043.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-12T03:10:56.864 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2024-04-12T03:10:56.864 INFO:teuthology.run_tasks:Running task ceph... 2024-04-12T03:10:56.876 INFO:tasks.ceph:Creating ceph cluster c2... 2024-04-12T03:10:56.876 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-12T03:10:56.876 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596', 'branch': 'reef-release', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '7652596', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'reef-release'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'reef-release', 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 6358, 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'reef-release', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'targets': {'smithi043.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM360nfrc8+vzqLBmt/bUgq/Qji/8Ta1ZqA1yis6gch2B7vP9ozECqHJvYXqFyNKiYsii47X9221GsM42/v4qhM=', 'smithi149.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN4+Al/t3FeS8VxGyA79xzsyQJKFWB6bq35ATLnGIa+oeWgp3QpA7yvw/te4TGZwz36fc5R+dr7d8WywxqnZzLc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'd540ebaca6b131a1dd560e7f69e024b133bbaa42', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-11_20:30:42', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-12T03:10:56.877 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-04-12T03:10:56.879 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-04-12T03:10:56.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m0777 -- /var/run/ceph 2024-04-12T03:10:56.938 DEBUG:teuthology.orchestra.run.smithi149:> sudo install -d -m0777 -- /var/run/ceph 2024-04-12T03:10:56.986 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:10:56.986 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/scratch_devs of=/dev/stdout 2024-04-12T03:10:57.005 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-12T03:10:57.005 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_1 2024-04-12T03:10:57.063 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-12T03:10:57.064 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T03:10:57.064 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 6h/6d Inode: 171912 Links: 1 2024-04-12T03:10:57.064 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T03:10:57.064 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T03:10:57.064 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-04-12 03:09:39.436231198 +0000 2024-04-12T03:10:57.064 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-04-12 03:01:25.373701131 +0000 2024-04-12T03:10:57.064 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-04-12 03:01:25.373701131 +0000 2024-04-12T03:10:57.064 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2024-04-12T03:10:57.065 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-12T03:10:57.131 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-04-12T03:10:57.131 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-04-12T03:10:57.131 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000268061 s, 1.9 MB/s 2024-04-12T03:10:57.132 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-12T03:10:57.192 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_2 2024-04-12T03:10:57.249 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-12T03:10:57.250 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T03:10:57.250 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 6h/6d Inode: 179639 Links: 1 2024-04-12T03:10:57.250 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T03:10:57.250 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T03:10:57.250 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-04-12 03:09:39.436231198 +0000 2024-04-12T03:10:57.250 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-04-12 03:01:25.716694231 +0000 2024-04-12T03:10:57.250 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-04-12 03:01:25.716694231 +0000 2024-04-12T03:10:57.250 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2024-04-12T03:10:57.250 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-12T03:10:57.316 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-04-12T03:10:57.317 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-04-12T03:10:57.317 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000240222 s, 2.1 MB/s 2024-04-12T03:10:57.319 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-12T03:10:57.389 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_3 2024-04-12T03:10:57.457 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-12T03:10:57.457 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T03:10:57.457 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 6h/6d Inode: 178547 Links: 1 2024-04-12T03:10:57.457 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T03:10:57.457 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T03:10:57.457 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-04-12 03:09:39.436231198 +0000 2024-04-12T03:10:57.457 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-04-12 03:01:26.069687130 +0000 2024-04-12T03:10:57.458 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-04-12 03:01:26.069687130 +0000 2024-04-12T03:10:57.458 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2024-04-12T03:10:57.458 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-12T03:10:57.528 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-04-12T03:10:57.528 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-04-12T03:10:57.528 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000204665 s, 2.5 MB/s 2024-04-12T03:10:57.531 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-12T03:10:57.596 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_4 2024-04-12T03:10:57.660 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-12T03:10:57.661 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T03:10:57.661 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 6h/6d Inode: 180743 Links: 1 2024-04-12T03:10:57.661 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T03:10:57.661 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T03:10:57.661 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-04-12 03:09:39.436231198 +0000 2024-04-12T03:10:57.661 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-04-12 03:01:26.420680070 +0000 2024-04-12T03:10:57.661 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-04-12 03:01:26.420680070 +0000 2024-04-12T03:10:57.661 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: - 2024-04-12T03:10:57.662 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-12T03:10:57.736 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-04-12T03:10:57.736 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-04-12T03:10:57.736 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000224962 s, 2.3 MB/s 2024-04-12T03:10:57.737 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-12T03:10:57.800 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-12T03:10:57.800 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi149.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-12T03:10:57.800 INFO:tasks.ceph:Generating config... 2024-04-12T03:10:57.801 INFO:tasks.ceph:[client] debug rgw = 20 2024-04-12T03:10:57.801 INFO:tasks.ceph:[client] debug rgw sync = 20 2024-04-12T03:10:57.801 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-04-12T03:10:57.801 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-04-12T03:10:57.801 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-12T03:10:57.801 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-04-12T03:10:57.801 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-04-12T03:10:57.801 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-04-12T03:10:57.801 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2024-04-12T03:10:57.801 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-04-12T03:10:57.802 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-04-12T03:10:57.802 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-04-12T03:10:57.802 INFO:tasks.ceph:[client] setgroup = ceph 2024-04-12T03:10:57.802 INFO:tasks.ceph:[client] setuser = ceph 2024-04-12T03:10:57.802 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-04-12T03:10:57.802 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-12T03:10:57.802 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-12T03:10:57.802 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-12T03:10:57.802 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-12T03:10:57.802 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-12T03:10:57.802 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-12T03:10:57.802 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-12T03:10:57.802 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-04-12T03:10:57.802 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-04-12T03:10:57.802 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-04-12T03:10:57.803 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-04-12T03:10:57.803 INFO:tasks.ceph:Setting up c2.mon.a... 2024-04-12T03:10:57.803 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2024-04-12T03:10:57.902 INFO:teuthology.orchestra.run.smithi149.stdout:creating /etc/ceph/c2.keyring 2024-04-12T03:10:57.907 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2024-04-12T03:10:57.952 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-12T03:10:57.979 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.149')] 2024-04-12T03:10:57.979 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.149'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-04-12T03:10:57.980 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:10:57.980 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-12T03:10:58.048 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.149 --print /home/ubuntu/cephtest/c2.monmap 2024-04-12T03:10:58.134 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:10:58.134 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:10:58.136 INFO:teuthology.orchestra.run.smithi149.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2024-04-12T03:10:58.136 INFO:teuthology.orchestra.run.smithi149.stdout:monmaptool: generated fsid fe2a5e2e-a277-4813-952c-9c1e63e6f1cc 2024-04-12T03:10:58.136 INFO:teuthology.orchestra.run.smithi149.stdout:setting min_mon_release = pacific 2024-04-12T03:10:58.136 INFO:teuthology.orchestra.run.smithi149.stdout:epoch 0 2024-04-12T03:10:58.136 INFO:teuthology.orchestra.run.smithi149.stdout:fsid fe2a5e2e-a277-4813-952c-9c1e63e6f1cc 2024-04-12T03:10:58.136 INFO:teuthology.orchestra.run.smithi149.stdout:last_changed 2024-04-12T03:10:58.133887+0000 2024-04-12T03:10:58.136 INFO:teuthology.orchestra.run.smithi149.stdout:created 2024-04-12T03:10:58.133887+0000 2024-04-12T03:10:58.136 INFO:teuthology.orchestra.run.smithi149.stdout:min_mon_release 16 (pacific) 2024-04-12T03:10:58.136 INFO:teuthology.orchestra.run.smithi149.stdout:election_strategy: 1 2024-04-12T03:10:58.136 INFO:teuthology.orchestra.run.smithi149.stdout:0: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.a 2024-04-12T03:10:58.137 INFO:teuthology.orchestra.run.smithi149.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2024-04-12T03:10:58.137 DEBUG:teuthology.orchestra.run.smithi149:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-12T03:10:58.193 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID fe2a5e2e-a277-4813-952c-9c1e63e6f1cc... 2024-04-12T03:10:58.195 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-04-12T03:10:58.198 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-04-12T03:10:58.255 INFO:teuthology.orchestra.run.smithi043.stdout:[global] 2024-04-12T03:10:58.255 INFO:teuthology.orchestra.run.smithi043.stdout: chdir = "" 2024-04-12T03:10:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-12T03:10:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: auth supported = cephx 2024-04-12T03:10:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: filestore xattr use omap = true 2024-04-12T03:10:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: mon clock drift allowed = 1.000 2024-04-12T03:10:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: osd crush chooseleaf type = 0 2024-04-12T03:10:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: auth debug = true 2024-04-12T03:10:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.256 INFO:teuthology.orchestra.run.smithi043.stdout: ms die on old message = true 2024-04-12T03:10:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: ms die on bug = true 2024-04-12T03:10:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-12T03:10:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: mon pg warn max object skew = 0 2024-04-12T03:10:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: # disable pg_autoscaler by default for new pools 2024-04-12T03:10:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-12T03:10:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default size = 2 2024-04-12T03:10:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd allow primary affinity = true 2024-04-12T03:10:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd allow pg remap = true 2024-04-12T03:10:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on legacy crush tunables = false 2024-04-12T03:10:58.257 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on crush straw calc version zero = false 2024-04-12T03:10:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on no sortbitwise = false 2024-04-12T03:10:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on osd down out interval zero = false 2024-04-12T03:10:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on too few osds = false 2024-04-12T03:10:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-12T03:10:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-12T03:10:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: mon_allow_pool_size_one = true 2024-04-12T03:10:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-12T03:10:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: osd default data pool replay window = 5 2024-04-12T03:10:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: mon allow pool delete = true 2024-04-12T03:10:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.258 INFO:teuthology.orchestra.run.smithi043.stdout: mon cluster log file level = debug 2024-04-12T03:10:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: debug asserts on shutdown = true 2024-04-12T03:10:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: mon health detail to clog = false 2024-04-12T03:10:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: mon host = 172.21.15.149 2024-04-12T03:10:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: fsid = fe2a5e2e-a277-4813-952c-9c1e63e6f1cc 2024-04-12T03:10:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.259 INFO:teuthology.orchestra.run.smithi043.stdout:[osd] 2024-04-12T03:10:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: osd journal size = 100 2024-04-12T03:10:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: osd scrub load threshold = 5.0 2024-04-12T03:10:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: osd scrub max interval = 600 2024-04-12T03:10:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: osd mclock profile = high_recovery_ops 2024-04-12T03:10:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: osd recover clone overlap = true 2024-04-12T03:10:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: osd recovery max chunk = 1048576 2024-04-12T03:10:58.259 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug shutdown = true 2024-04-12T03:10:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug op order = true 2024-04-12T03:10:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug verify stray on activate = true 2024-04-12T03:10:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: osd open classes on start = true 2024-04-12T03:10:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug pg log writeout = true 2024-04-12T03:10:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: osd deep scrub update digest min age = 30 2024-04-12T03:10:58.260 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: osd map max advance = 10 2024-04-12T03:10:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: journal zero on create = true 2024-04-12T03:10:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: filestore ondisk finisher threads = 3 2024-04-12T03:10:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: filestore apply finisher threads = 3 2024-04-12T03:10:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: bdev debug aio = true 2024-04-12T03:10:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug misdirected ops = true 2024-04-12T03:10:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-04-12T03:10:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: debug osd = 20 2024-04-12T03:10:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: rocksdb delete range threshold = 0 2024-04-12T03:10:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.261 INFO:teuthology.orchestra.run.smithi043.stdout:[mgr] 2024-04-12T03:10:58.261 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-04-12T03:10:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: debug mgr = 20 2024-04-12T03:10:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: debug mon = 20 2024-04-12T03:10:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: debug auth = 20 2024-04-12T03:10:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min pgs per osd = 4 2024-04-12T03:10:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min bytes per osd = 10 2024-04-12T03:10:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: mgr/telemetry/nag = false 2024-04-12T03:10:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.262 INFO:teuthology.orchestra.run.smithi043.stdout:[mon] 2024-04-12T03:10:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-04-12T03:10:58.262 INFO:teuthology.orchestra.run.smithi043.stdout: debug mon = 20 2024-04-12T03:10:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: debug paxos = 20 2024-04-12T03:10:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: debug auth = 20 2024-04-12T03:10:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: mon data avail warn = 5 2024-04-12T03:10:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: mon mgr mkfs grace = 240 2024-04-12T03:10:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min pgs per osd = 4 2024-04-12T03:10:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd reporter subtree level = osd 2024-04-12T03:10:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd prime pg temp = true 2024-04-12T03:10:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min bytes per osd = 10 2024-04-12T03:10:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-12T03:10:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: auth mon ticket ttl = 660 # 11m 2024-04-12T03:10:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: auth service ticket ttl = 240 # 4m 2024-04-12T03:10:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: # don't complain about insecure global_id in the test suite 2024-04-12T03:10:58.263 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-12T03:10:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-12T03:10:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: # 1m isn't quite enough 2024-04-12T03:10:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: mon_down_mkfs_grace = 2m 2024-04-12T03:10:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_filestore_osds = false 2024-04-12T03:10:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:10:58.264 INFO:teuthology.orchestra.run.smithi043.stdout:[client] 2024-04-12T03:10:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: rgw cache enabled = true 2024-04-12T03:10:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: rgw enable ops log = true 2024-04-12T03:10:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: rgw enable usage log = true 2024-04-12T03:10:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-12T03:10:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-12T03:10:58.264 INFO:teuthology.orchestra.run.smithi043.stdout: debug rgw = 20 2024-04-12T03:10:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: debug rgw sync = 20 2024-04-12T03:10:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: rgw crypt require ssl = False 2024-04-12T03:10:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: rgw crypt s3 kms backend = testing 2024-04-12T03:10:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-12T03:10:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: rgw curl low speed time = 300 2024-04-12T03:10:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: rgw data log num shards = 4 2024-04-12T03:10:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: rgw md log max shards = 4 2024-04-12T03:10:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: rgw sync data inject err probability = 0.1 2024-04-12T03:10:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: rgw sync log trim interval = 0 2024-04-12T03:10:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: rgw sync meta inject err probability = 0.1 2024-04-12T03:10:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: rgw sync obj etag verify = True 2024-04-12T03:10:58.265 INFO:teuthology.orchestra.run.smithi043.stdout: setgroup = ceph 2024-04-12T03:10:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: setuser = ceph 2024-04-12T03:10:58.266 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.a] 2024-04-12T03:10:58.266 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-04-12T03:10:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: rgw data notify interval msec = 0 2024-04-12T03:10:58.266 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.0] 2024-04-12T03:10:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: osd_max_omap_entries_per_request = 10 2024-04-12T03:10:58.266 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.1] 2024-04-12T03:10:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: osd_max_omap_entries_per_request = 1000 2024-04-12T03:10:58.266 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.2] 2024-04-12T03:10:58.266 INFO:teuthology.orchestra.run.smithi043.stdout: osd_max_omap_entries_per_request = 10000 2024-04-12T03:10:58.281 INFO:teuthology.orchestra.run.smithi149.stdout:[global] 2024-04-12T03:10:58.281 INFO:teuthology.orchestra.run.smithi149.stdout: chdir = "" 2024-04-12T03:10:58.281 INFO:teuthology.orchestra.run.smithi149.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-12T03:10:58.281 INFO:teuthology.orchestra.run.smithi149.stdout: auth supported = cephx 2024-04-12T03:10:58.281 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.281 INFO:teuthology.orchestra.run.smithi149.stdout: filestore xattr use omap = true 2024-04-12T03:10:58.282 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.282 INFO:teuthology.orchestra.run.smithi149.stdout: mon clock drift allowed = 1.000 2024-04-12T03:10:58.282 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.282 INFO:teuthology.orchestra.run.smithi149.stdout: osd crush chooseleaf type = 0 2024-04-12T03:10:58.282 INFO:teuthology.orchestra.run.smithi149.stdout: auth debug = true 2024-04-12T03:10:58.282 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.282 INFO:teuthology.orchestra.run.smithi149.stdout: ms die on old message = true 2024-04-12T03:10:58.282 INFO:teuthology.orchestra.run.smithi149.stdout: ms die on bug = true 2024-04-12T03:10:58.282 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.282 INFO:teuthology.orchestra.run.smithi149.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-12T03:10:58.282 INFO:teuthology.orchestra.run.smithi149.stdout: mon pg warn max object skew = 0 2024-04-12T03:10:58.282 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.283 INFO:teuthology.orchestra.run.smithi149.stdout: # disable pg_autoscaler by default for new pools 2024-04-12T03:10:58.283 INFO:teuthology.orchestra.run.smithi149.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-12T03:10:58.283 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.283 INFO:teuthology.orchestra.run.smithi149.stdout: osd pool default size = 2 2024-04-12T03:10:58.283 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.283 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd allow primary affinity = true 2024-04-12T03:10:58.283 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd allow pg remap = true 2024-04-12T03:10:58.283 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on legacy crush tunables = false 2024-04-12T03:10:58.283 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on crush straw calc version zero = false 2024-04-12T03:10:58.283 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on no sortbitwise = false 2024-04-12T03:10:58.283 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on osd down out interval zero = false 2024-04-12T03:10:58.284 INFO:teuthology.orchestra.run.smithi149.stdout: mon warn on too few osds = false 2024-04-12T03:10:58.284 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-12T03:10:58.284 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-12T03:10:58.284 INFO:teuthology.orchestra.run.smithi149.stdout: mon_allow_pool_size_one = true 2024-04-12T03:10:58.284 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.284 INFO:teuthology.orchestra.run.smithi149.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-12T03:10:58.284 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.284 INFO:teuthology.orchestra.run.smithi149.stdout: osd default data pool replay window = 5 2024-04-12T03:10:58.284 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.284 INFO:teuthology.orchestra.run.smithi149.stdout: mon allow pool delete = true 2024-04-12T03:10:58.284 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.285 INFO:teuthology.orchestra.run.smithi149.stdout: mon cluster log file level = debug 2024-04-12T03:10:58.285 INFO:teuthology.orchestra.run.smithi149.stdout: debug asserts on shutdown = true 2024-04-12T03:10:58.285 INFO:teuthology.orchestra.run.smithi149.stdout: mon health detail to clog = false 2024-04-12T03:10:58.285 INFO:teuthology.orchestra.run.smithi149.stdout: mon host = 172.21.15.149 2024-04-12T03:10:58.285 INFO:teuthology.orchestra.run.smithi149.stdout: fsid = fe2a5e2e-a277-4813-952c-9c1e63e6f1cc 2024-04-12T03:10:58.285 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.285 INFO:teuthology.orchestra.run.smithi149.stdout:[osd] 2024-04-12T03:10:58.285 INFO:teuthology.orchestra.run.smithi149.stdout: osd journal size = 100 2024-04-12T03:10:58.285 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.285 INFO:teuthology.orchestra.run.smithi149.stdout: osd scrub load threshold = 5.0 2024-04-12T03:10:58.285 INFO:teuthology.orchestra.run.smithi149.stdout: osd scrub max interval = 600 2024-04-12T03:10:58.285 INFO:teuthology.orchestra.run.smithi149.stdout: osd mclock profile = high_recovery_ops 2024-04-12T03:10:58.286 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.286 INFO:teuthology.orchestra.run.smithi149.stdout: osd recover clone overlap = true 2024-04-12T03:10:58.286 INFO:teuthology.orchestra.run.smithi149.stdout: osd recovery max chunk = 1048576 2024-04-12T03:10:58.286 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.286 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug shutdown = true 2024-04-12T03:10:58.286 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug op order = true 2024-04-12T03:10:58.286 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug verify stray on activate = true 2024-04-12T03:10:58.286 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.286 INFO:teuthology.orchestra.run.smithi149.stdout: osd open classes on start = true 2024-04-12T03:10:58.286 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug pg log writeout = true 2024-04-12T03:10:58.286 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.286 INFO:teuthology.orchestra.run.smithi149.stdout: osd deep scrub update digest min age = 30 2024-04-12T03:10:58.287 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.287 INFO:teuthology.orchestra.run.smithi149.stdout: osd map max advance = 10 2024-04-12T03:10:58.287 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.287 INFO:teuthology.orchestra.run.smithi149.stdout: journal zero on create = true 2024-04-12T03:10:58.287 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.287 INFO:teuthology.orchestra.run.smithi149.stdout: filestore ondisk finisher threads = 3 2024-04-12T03:10:58.287 INFO:teuthology.orchestra.run.smithi149.stdout: filestore apply finisher threads = 3 2024-04-12T03:10:58.287 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.287 INFO:teuthology.orchestra.run.smithi149.stdout: bdev debug aio = true 2024-04-12T03:10:58.287 INFO:teuthology.orchestra.run.smithi149.stdout: osd debug misdirected ops = true 2024-04-12T03:10:58.287 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2024-04-12T03:10:58.287 INFO:teuthology.orchestra.run.smithi149.stdout: debug osd = 20 2024-04-12T03:10:58.287 INFO:teuthology.orchestra.run.smithi149.stdout: rocksdb delete range threshold = 0 2024-04-12T03:10:58.287 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.288 INFO:teuthology.orchestra.run.smithi149.stdout:[mgr] 2024-04-12T03:10:58.288 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2024-04-12T03:10:58.288 INFO:teuthology.orchestra.run.smithi149.stdout: debug mgr = 20 2024-04-12T03:10:58.288 INFO:teuthology.orchestra.run.smithi149.stdout: debug mon = 20 2024-04-12T03:10:58.288 INFO:teuthology.orchestra.run.smithi149.stdout: debug auth = 20 2024-04-12T03:10:58.288 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min pgs per osd = 4 2024-04-12T03:10:58.288 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min bytes per osd = 10 2024-04-12T03:10:58.288 INFO:teuthology.orchestra.run.smithi149.stdout: mgr/telemetry/nag = false 2024-04-12T03:10:58.288 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.288 INFO:teuthology.orchestra.run.smithi149.stdout:[mon] 2024-04-12T03:10:58.288 INFO:teuthology.orchestra.run.smithi149.stdout: debug ms = 1 2024-04-12T03:10:58.288 INFO:teuthology.orchestra.run.smithi149.stdout: debug mon = 20 2024-04-12T03:10:58.288 INFO:teuthology.orchestra.run.smithi149.stdout: debug paxos = 20 2024-04-12T03:10:58.289 INFO:teuthology.orchestra.run.smithi149.stdout: debug auth = 20 2024-04-12T03:10:58.289 INFO:teuthology.orchestra.run.smithi149.stdout: mon data avail warn = 5 2024-04-12T03:10:58.289 INFO:teuthology.orchestra.run.smithi149.stdout: mon mgr mkfs grace = 240 2024-04-12T03:10:58.289 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min pgs per osd = 4 2024-04-12T03:10:58.289 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd reporter subtree level = osd 2024-04-12T03:10:58.289 INFO:teuthology.orchestra.run.smithi149.stdout: mon osd prime pg temp = true 2024-04-12T03:10:58.289 INFO:teuthology.orchestra.run.smithi149.stdout: mon reweight min bytes per osd = 10 2024-04-12T03:10:58.289 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.289 INFO:teuthology.orchestra.run.smithi149.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-12T03:10:58.289 INFO:teuthology.orchestra.run.smithi149.stdout: auth mon ticket ttl = 660 # 11m 2024-04-12T03:10:58.289 INFO:teuthology.orchestra.run.smithi149.stdout: auth service ticket ttl = 240 # 4m 2024-04-12T03:10:58.289 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.289 INFO:teuthology.orchestra.run.smithi149.stdout: # don't complain about insecure global_id in the test suite 2024-04-12T03:10:58.290 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-12T03:10:58.290 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-12T03:10:58.290 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.290 INFO:teuthology.orchestra.run.smithi149.stdout: # 1m isn't quite enough 2024-04-12T03:10:58.290 INFO:teuthology.orchestra.run.smithi149.stdout: mon_down_mkfs_grace = 2m 2024-04-12T03:10:58.290 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.290 INFO:teuthology.orchestra.run.smithi149.stdout: mon_warn_on_filestore_osds = false 2024-04-12T03:10:58.290 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:10:58.290 INFO:teuthology.orchestra.run.smithi149.stdout:[client] 2024-04-12T03:10:58.290 INFO:teuthology.orchestra.run.smithi149.stdout: rgw cache enabled = true 2024-04-12T03:10:58.290 INFO:teuthology.orchestra.run.smithi149.stdout: rgw enable ops log = true 2024-04-12T03:10:58.290 INFO:teuthology.orchestra.run.smithi149.stdout: rgw enable usage log = true 2024-04-12T03:10:58.291 INFO:teuthology.orchestra.run.smithi149.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-12T03:10:58.291 INFO:teuthology.orchestra.run.smithi149.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-12T03:10:58.291 INFO:teuthology.orchestra.run.smithi149.stdout: debug rgw = 20 2024-04-12T03:10:58.291 INFO:teuthology.orchestra.run.smithi149.stdout: debug rgw sync = 20 2024-04-12T03:10:58.291 INFO:teuthology.orchestra.run.smithi149.stdout: rgw crypt require ssl = False 2024-04-12T03:10:58.291 INFO:teuthology.orchestra.run.smithi149.stdout: rgw crypt s3 kms backend = testing 2024-04-12T03:10:58.291 INFO:teuthology.orchestra.run.smithi149.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-04-12T03:10:58.291 INFO:teuthology.orchestra.run.smithi149.stdout: rgw curl low speed time = 300 2024-04-12T03:10:58.291 INFO:teuthology.orchestra.run.smithi149.stdout: rgw data log num shards = 4 2024-04-12T03:10:58.291 INFO:teuthology.orchestra.run.smithi149.stdout: rgw md log max shards = 4 2024-04-12T03:10:58.291 INFO:teuthology.orchestra.run.smithi149.stdout: rgw sync data inject err probability = 0.1 2024-04-12T03:10:58.291 INFO:teuthology.orchestra.run.smithi149.stdout: rgw sync log trim interval = 0 2024-04-12T03:10:58.291 INFO:teuthology.orchestra.run.smithi149.stdout: rgw sync meta inject err probability = 0.1 2024-04-12T03:10:58.292 INFO:teuthology.orchestra.run.smithi149.stdout: rgw sync obj etag verify = True 2024-04-12T03:10:58.292 INFO:teuthology.orchestra.run.smithi149.stdout: setgroup = ceph 2024-04-12T03:10:58.292 INFO:teuthology.orchestra.run.smithi149.stdout: setuser = ceph 2024-04-12T03:10:58.292 INFO:teuthology.orchestra.run.smithi149.stdout:[mon.a] 2024-04-12T03:10:58.292 INFO:teuthology.orchestra.run.smithi149.stdout:[client.0] 2024-04-12T03:10:58.292 INFO:teuthology.orchestra.run.smithi149.stdout: rgw data notify interval msec = 0 2024-04-12T03:10:58.292 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.0] 2024-04-12T03:10:58.292 INFO:teuthology.orchestra.run.smithi149.stdout: osd_max_omap_entries_per_request = 10 2024-04-12T03:10:58.292 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.1] 2024-04-12T03:10:58.292 INFO:teuthology.orchestra.run.smithi149.stdout: osd_max_omap_entries_per_request = 1000 2024-04-12T03:10:58.292 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.2] 2024-04-12T03:10:58.292 INFO:teuthology.orchestra.run.smithi149.stdout: osd_max_omap_entries_per_request = 10000 2024-04-12T03:10:58.295 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2024-04-12T03:10:58.295 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2024-04-12T03:10:58.410 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-12T03:10:58.411 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:10:58.411 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2024-04-12T03:10:58.428 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:10:58.428 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2024-04-12T03:10:58.488 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi043.front.sepia.ceph.com 2024-04-12T03:10:58.488 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:58.488 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/c2.keyring 2024-04-12T03:10:58.488 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-12T03:10:58.538 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-12T03:10:58.538 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-04-12T03:10:58.604 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi149.front.sepia.ceph.com 2024-04-12T03:10:58.604 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:10:58.604 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/c2.keyring 2024-04-12T03:10:58.604 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-04-12T03:10:58.641 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:10:58.641 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-04-12T03:10:58.708 INFO:tasks.ceph:Setting up mon nodes... 2024-04-12T03:10:58.708 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-12T03:10:58.708 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2024-04-12T03:10:58.806 INFO:teuthology.orchestra.run.smithi149.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2024-04-12T03:10:58.808 INFO:tasks.ceph:Setting up mds nodes... 2024-04-12T03:10:58.808 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-12T03:10:58.809 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2024-04-12T03:10:58.852 INFO:teuthology.orchestra.run.smithi149.stdout:creating /etc/ceph/c2.client.0.keyring 2024-04-12T03:10:58.866 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2024-04-12T03:10:58.910 INFO:teuthology.orchestra.run.smithi149.stdout:creating /etc/ceph/c2.client.1.keyring 2024-04-12T03:10:58.923 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-12T03:10:58.923 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi043.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi149.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-12T03:10:58.926 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2024-04-12T03:10:58.998 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-12T03:10:58.999 INFO:tasks.ceph:role: c2.osd.0 2024-04-12T03:10:58.999 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi149.front.sepia.ceph.com 2024-04-12T03:10:58.999 DEBUG:teuthology.orchestra.run.smithi149:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-12T03:10:59.080 INFO:teuthology.orchestra.run.smithi149.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-12T03:10:59.080 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-12T03:10:59.080 INFO:teuthology.orchestra.run.smithi149.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-12T03:10:59.081 INFO:teuthology.orchestra.run.smithi149.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-12T03:10:59.081 INFO:teuthology.orchestra.run.smithi149.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-12T03:10:59.081 INFO:teuthology.orchestra.run.smithi149.stdout: = sunit=0 swidth=0 blks 2024-04-12T03:10:59.081 INFO:teuthology.orchestra.run.smithi149.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-12T03:10:59.081 INFO:teuthology.orchestra.run.smithi149.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-12T03:10:59.081 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-12T03:10:59.081 INFO:teuthology.orchestra.run.smithi149.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-12T03:10:59.512 INFO:teuthology.orchestra.run.smithi149.stdout:Discarding blocks...Done. 2024-04-12T03:10:59.513 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi149.front.sepia.ceph.com -o noatime 2024-04-12T03:10:59.513 DEBUG:teuthology.orchestra.run.smithi149:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2024-04-12T03:10:59.692 DEBUG:teuthology.orchestra.run.smithi149:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2024-04-12T03:10:59.728 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2024-04-12T03:10:59.801 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-12T03:10:59.801 INFO:tasks.ceph:role: c2.osd.1 2024-04-12T03:10:59.801 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi149.front.sepia.ceph.com 2024-04-12T03:10:59.801 DEBUG:teuthology.orchestra.run.smithi149:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-12T03:10:59.877 INFO:teuthology.orchestra.run.smithi149.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-12T03:10:59.877 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-12T03:10:59.877 INFO:teuthology.orchestra.run.smithi149.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-12T03:10:59.877 INFO:teuthology.orchestra.run.smithi149.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-12T03:10:59.878 INFO:teuthology.orchestra.run.smithi149.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-12T03:10:59.878 INFO:teuthology.orchestra.run.smithi149.stdout: = sunit=0 swidth=0 blks 2024-04-12T03:10:59.878 INFO:teuthology.orchestra.run.smithi149.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-12T03:10:59.878 INFO:teuthology.orchestra.run.smithi149.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-12T03:10:59.878 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-12T03:10:59.878 INFO:teuthology.orchestra.run.smithi149.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-12T03:11:00.297 INFO:teuthology.orchestra.run.smithi149.stdout:Discarding blocks...Done. 2024-04-12T03:11:00.299 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi149.front.sepia.ceph.com -o noatime 2024-04-12T03:11:00.299 DEBUG:teuthology.orchestra.run.smithi149:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2024-04-12T03:11:00.411 DEBUG:teuthology.orchestra.run.smithi149:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2024-04-12T03:11:00.443 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2024-04-12T03:11:00.511 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-12T03:11:00.511 INFO:tasks.ceph:role: c2.osd.2 2024-04-12T03:11:00.511 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi149.front.sepia.ceph.com 2024-04-12T03:11:00.511 DEBUG:teuthology.orchestra.run.smithi149:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-12T03:11:00.587 INFO:teuthology.orchestra.run.smithi149.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-12T03:11:00.588 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-12T03:11:00.588 INFO:teuthology.orchestra.run.smithi149.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-12T03:11:00.588 INFO:teuthology.orchestra.run.smithi149.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-12T03:11:00.588 INFO:teuthology.orchestra.run.smithi149.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-12T03:11:00.588 INFO:teuthology.orchestra.run.smithi149.stdout: = sunit=0 swidth=0 blks 2024-04-12T03:11:00.588 INFO:teuthology.orchestra.run.smithi149.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-12T03:11:00.588 INFO:teuthology.orchestra.run.smithi149.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-12T03:11:00.588 INFO:teuthology.orchestra.run.smithi149.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-12T03:11:00.588 INFO:teuthology.orchestra.run.smithi149.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-12T03:11:01.034 INFO:teuthology.orchestra.run.smithi149.stdout:Discarding blocks...Done. 2024-04-12T03:11:01.036 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi149.front.sepia.ceph.com -o noatime 2024-04-12T03:11:01.036 DEBUG:teuthology.orchestra.run.smithi149:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2024-04-12T03:11:01.187 DEBUG:teuthology.orchestra.run.smithi149:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2024-04-12T03:11:01.220 DEBUG:teuthology.orchestra.run.smithi149:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-12T03:11:01.307 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:01.305+0000 7f70387e27c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2024-04-12T03:11:01.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:01.307+0000 7f70387e27c0 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2024-04-12T03:11:01.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:01.307+0000 7f70387e27c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-12T03:11:01.309 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:01.307+0000 7f70387e27c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-12T03:11:01.310 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:01.307+0000 7f70387e27c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-04-12T03:11:01.317 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:01.315+0000 7f70387e27c0 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2024-04-12T03:11:03.395 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2024-04-12T03:11:03.425 DEBUG:teuthology.orchestra.run.smithi149:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-12T03:11:03.515 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:03.513+0000 7fec599117c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2024-04-12T03:11:03.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:03.513+0000 7fec599117c0 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2024-04-12T03:11:03.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:03.513+0000 7fec599117c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-12T03:11:03.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:03.513+0000 7fec599117c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-12T03:11:03.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:03.513+0000 7fec599117c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-04-12T03:11:03.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:03.514+0000 7fec599117c0 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2024-04-12T03:11:05.592 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2024-04-12T03:11:05.623 DEBUG:teuthology.orchestra.run.smithi149:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2024-04-12T03:11:05.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:05.714+0000 7f9311fb27c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2024-04-12T03:11:05.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:05.714+0000 7f9311fb27c0 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2024-04-12T03:11:05.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:05.714+0000 7f9311fb27c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-12T03:11:05.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:05.714+0000 7f9311fb27c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-12T03:11:05.716 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:05.714+0000 7f9311fb27c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-04-12T03:11:05.717 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:05.715+0000 7f9311fb27c0 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2024-04-12T03:11:07.808 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2024-04-12T03:11:07.838 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-12T03:11:07.838 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:11:07.839 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2024-04-12T03:11:07.907 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:11:07.908 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2024-04-12T03:11:07.976 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:11:07.976 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2024-04-12T03:11:08.044 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:11:08.044 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2024-04-12T03:11:08.111 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:11:08.111 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2024-04-12T03:11:08.169 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:11:08.169 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2024-04-12T03:11:08.225 INFO:tasks.ceph:Adding keys to all mons... 2024-04-12T03:11:08.226 DEBUG:teuthology.orchestra.run.smithi149:> sudo tee -a /etc/ceph/c2.keyring 2024-04-12T03:11:08.291 INFO:teuthology.orchestra.run.smithi149.stdout:[mgr.x] 2024-04-12T03:11:08.291 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQBCphhm4PwDMBAA6H9qpUDOsbz2u04CKHtTkQ== 2024-04-12T03:11:08.292 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.0] 2024-04-12T03:11:08.292 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQBFphhmUKJLEhAAmh0EZ+iQhOy6aXZJVkSv+Q== 2024-04-12T03:11:08.292 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.1] 2024-04-12T03:11:08.292 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQBHphhmWn2tHhAAddtF9DgEYIBDekZTMT6LMg== 2024-04-12T03:11:08.292 INFO:teuthology.orchestra.run.smithi149.stdout:[osd.2] 2024-04-12T03:11:08.292 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQBJphhmHV2lKhAASZjVsriYAjqB49LlrlogQw== 2024-04-12T03:11:08.292 INFO:teuthology.orchestra.run.smithi149.stdout:[client.0] 2024-04-12T03:11:08.292 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQBCphhm2SPPMhAAaIr7ZuWYdmg/qVnVOtaRrg== 2024-04-12T03:11:08.292 INFO:teuthology.orchestra.run.smithi149.stdout:[client.1] 2024-04-12T03:11:08.292 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQBCphhmhwQxNhAA2kHQD6lWwmXtt4+HNJ0jNQ== 2024-04-12T03:11:08.293 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-12T03:11:08.386 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-12T03:11:08.440 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-12T03:11:08.493 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-12T03:11:08.547 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-12T03:11:08.604 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-12T03:11:08.661 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-12T03:11:08.661 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2024-04-12T03:11:08.687 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2024-04-12T03:11:08.787 DEBUG:teuthology.orchestra.run.smithi149:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2024-04-12T03:11:08.813 DEBUG:teuthology.orchestra.run.smithi149:> rm -- /home/ubuntu/cephtest/c2.monmap 2024-04-12T03:11:08.870 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2024-04-12T03:11:08.871 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2024-04-12T03:11:08.871 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2024-04-12T03:11:08.913 INFO:tasks.ceph.c2.mon.a:Started 2024-04-12T03:11:08.914 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2024-04-12T03:11:08.914 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2024-04-12T03:11:08.914 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2024-04-12T03:11:08.916 INFO:tasks.ceph.c2.mgr.x:Started 2024-04-12T03:11:08.916 INFO:tasks.ceph:Setting crush tunables to default 2024-04-12T03:11:08.916 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster c2 osd crush tunables default 2024-04-12T03:11:09.141 INFO:teuthology.orchestra.run.smithi149.stderr:adjusted tunables profile to default 2024-04-12T03:11:09.153 INFO:tasks.ceph:check_enable_crimson: False 2024-04-12T03:11:09.153 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2024-04-12T03:11:09.154 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:11:09.154 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2024-04-12T03:11:09.180 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:11:09.181 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2024-04-12T03:11:09.248 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-12T03:11:09.249 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2024-04-12T03:11:09.317 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster c2 osd new e69be0fa-5c27-4187-83ba-0db366836d6b 0 2024-04-12T03:11:09.584 INFO:teuthology.orchestra.run.smithi149.stdout:0 2024-04-12T03:11:09.597 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster c2 osd new 8826dbf6-d1c8-4614-9b33-887a9cc8f7a5 1 2024-04-12T03:11:09.645 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:09.643+0000 7f7819270200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T03:11:09.729 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:09.727+0000 7f7819270200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T03:11:09.805 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:09.803+0000 7f7819270200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T03:11:09.813 INFO:teuthology.orchestra.run.smithi149.stdout:1 2024-04-12T03:11:09.826 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster c2 osd new 8969b929-490a-44f0-9464-dbcd1fdd2caa 2 2024-04-12T03:11:09.881 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:09.879+0000 7f7819270200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T03:11:10.044 INFO:teuthology.orchestra.run.smithi149.stdout:2 2024-04-12T03:11:10.056 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2024-04-12T03:11:10.057 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2024-04-12T03:11:10.058 INFO:tasks.ceph.c2.osd.0:Started 2024-04-12T03:11:10.059 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2024-04-12T03:11:10.059 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2024-04-12T03:11:10.060 INFO:tasks.ceph.c2.osd.1:Started 2024-04-12T03:11:10.061 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2024-04-12T03:11:10.061 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2024-04-12T03:11:10.063 INFO:tasks.ceph.c2.osd.2:Started 2024-04-12T03:11:10.063 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-12T03:11:10.066 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:10.065+0000 7f7819270200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T03:11:10.262 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:10.260+0000 7f7819270200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T03:11:10.274 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:11:10.274 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":5,"fsid":"fe2a5e2e-a277-4813-952c-9c1e63e6f1cc","created":"2024-04-12T03:11:09.044013+0000","modified":"2024-04-12T03:11:10.040395+0000","last_up_change":"0.000000","last_in_change":"2024-04-12T03:11:10.040395+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e69be0fa-5c27-4187-83ba-0db366836d6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8826dbf6-d1c8-4614-9b33-887a9cc8f7a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8969b929-490a-44f0-9464-dbcd1fdd2caa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T03:11:10.287 INFO:tasks.ceph.ceph_manager.c2:[] 2024-04-12T03:11:10.288 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-12T03:11:10.377 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:10.375+0000 7f7819270200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T03:11:10.472 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:10.470+0000 7f7819270200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T03:11:10.548 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:10.547+0000 7f7819270200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T03:11:10.590 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-04-12T03:11:10.669 INFO:tasks.ceph.c2.osd.0.smithi149.stderr:2024-04-12T03:11:10.668+0000 7f2d88f777c0 -1 Falling back to public interface 2024-04-12T03:11:10.694 INFO:tasks.ceph.c2.osd.2.smithi149.stderr:2024-04-12T03:11:10.693+0000 7fd64f1857c0 -1 Falling back to public interface 2024-04-12T03:11:10.698 INFO:tasks.ceph.c2.osd.1.smithi149.stderr:2024-04-12T03:11:10.697+0000 7f7585ad57c0 -1 Falling back to public interface 2024-04-12T03:11:10.804 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:10.803+0000 7f7819270200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T03:11:10.820 INFO:teuthology.misc.health.smithi149.stdout: 2024-04-12T03:11:10.820 INFO:teuthology.misc.health.smithi149.stdout:{"epoch":5,"fsid":"fe2a5e2e-a277-4813-952c-9c1e63e6f1cc","created":"2024-04-12T03:11:09.044013+0000","modified":"2024-04-12T03:11:10.040395+0000","last_up_change":"0.000000","last_in_change":"2024-04-12T03:11:10.040395+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e69be0fa-5c27-4187-83ba-0db366836d6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8826dbf6-d1c8-4614-9b33-887a9cc8f7a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8969b929-490a-44f0-9464-dbcd1fdd2caa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T03:11:10.832 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-12T03:11:10.878 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:10.877+0000 7f7819270200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T03:11:11.262 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:11.260+0000 7f7819270200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T03:11:11.588 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:11.586+0000 7f7819270200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T03:11:11.747 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:11.746+0000 7f7819270200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-12T03:11:11.828 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:11.827+0000 7f7819270200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T03:11:12.533 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:12.532+0000 7f7819270200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T03:11:12.618 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:12.617+0000 7f7819270200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T03:11:12.748 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:12.747+0000 7f7819270200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T03:11:13.170 INFO:tasks.ceph.c2.osd.1.smithi149.stderr:2024-04-12T03:11:13.169+0000 7f7585ad57c0 -1 osd.1 0 log_to_monitors true 2024-04-12T03:11:13.310 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:13.309+0000 7f7819270200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T03:11:13.397 INFO:tasks.ceph.c2.osd.0.smithi149.stderr:2024-04-12T03:11:13.396+0000 7f2d88f777c0 -1 osd.0 0 log_to_monitors true 2024-04-12T03:11:13.420 INFO:tasks.ceph.c2.osd.2.smithi149.stderr:2024-04-12T03:11:13.419+0000 7fd64f1857c0 -1 osd.2 0 log_to_monitors true 2024-04-12T03:11:13.488 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:13.487+0000 7f7819270200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T03:11:13.653 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:13.652+0000 7f7819270200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T03:11:14.512 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:14.510+0000 7f7819270200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T03:11:14.741 INFO:tasks.ceph.c2.osd.1.smithi149.stderr:2024-04-12T03:11:14.740+0000 7f7579e8d700 -1 osd.1 0 waiting for initial osdmap 2024-04-12T03:11:14.765 INFO:tasks.ceph.c2.osd.1.smithi149.stderr:2024-04-12T03:11:14.764+0000 7f7571a67700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T03:11:14.771 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:14.770+0000 7f7819270200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T03:11:14.847 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:14.845+0000 7f7819270200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T03:11:15.232 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:15.231+0000 7f7819270200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T03:11:16.079 INFO:tasks.ceph.c2.osd.0.smithi149.stderr:2024-04-12T03:11:16.078+0000 7f2d7d32f700 -1 osd.0 0 waiting for initial osdmap 2024-04-12T03:11:16.132 INFO:tasks.ceph.c2.osd.0.smithi149.stderr:2024-04-12T03:11:16.131+0000 7f2d74f09700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T03:11:16.322 INFO:tasks.ceph.c2.mgr.x.smithi149.stderr:2024-04-12T03:11:16.321+0000 7f77d44ea700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-12T03:11:16.602 INFO:tasks.ceph.c2.osd.2.smithi149.stderr:2024-04-12T03:11:16.601+0000 7fd64353d700 -1 osd.2 0 waiting for initial osdmap 2024-04-12T03:11:16.617 INFO:tasks.ceph.c2.osd.2.smithi149.stderr:2024-04-12T03:11:16.616+0000 7fd63b117700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T03:11:17.134 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-04-12T03:11:17.458 INFO:teuthology.misc.health.smithi149.stdout: 2024-04-12T03:11:17.459 INFO:teuthology.misc.health.smithi149.stdout:{"epoch":9,"fsid":"fe2a5e2e-a277-4813-952c-9c1e63e6f1cc","created":"2024-04-12T03:11:09.044013+0000","modified":"2024-04-12T03:11:17.079631+0000","last_up_change":"2024-04-12T03:11:17.079631+0000","last_in_change":"2024-04-12T03:11:10.040395+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e69be0fa-5c27-4187-83ba-0db366836d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":40829},{"type":"v1","addr":"172.21.15.149:6801","nonce":40829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":40829},{"type":"v1","addr":"172.21.15.149:6803","nonce":40829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":40829},{"type":"v1","addr":"172.21.15.149:6807","nonce":40829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":40829},{"type":"v1","addr":"172.21.15.149:6805","nonce":40829}]},"public_addr":"172.21.15.149:6801/40829","cluster_addr":"172.21.15.149:6803/40829","heartbeat_back_addr":"172.21.15.149:6807/40829","heartbeat_front_addr":"172.21.15.149:6805/40829","state":["exists","up"]},{"osd":1,"uuid":"8826dbf6-d1c8-4614-9b33-887a9cc8f7a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":40840},{"type":"v1","addr":"172.21.15.149:6817","nonce":40840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":40840},{"type":"v1","addr":"172.21.15.149:6819","nonce":40840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":40840},{"type":"v1","addr":"172.21.15.149:6823","nonce":40840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":40840},{"type":"v1","addr":"172.21.15.149:6821","nonce":40840}]},"public_addr":"172.21.15.149:6817/40840","cluster_addr":"172.21.15.149:6819/40840","heartbeat_back_addr":"172.21.15.149:6823/40840","heartbeat_front_addr":"172.21.15.149:6821/40840","state":["exists","up"]},{"osd":2,"uuid":"8969b929-490a-44f0-9464-dbcd1fdd2caa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":40831},{"type":"v1","addr":"172.21.15.149:6809","nonce":40831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":40831},{"type":"v1","addr":"172.21.15.149:6811","nonce":40831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":40831},{"type":"v1","addr":"172.21.15.149:6815","nonce":40831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":40831},{"type":"v1","addr":"172.21.15.149:6813","nonce":40831}]},"public_addr":"172.21.15.149:6809/40831","cluster_addr":"172.21.15.149:6811/40831","heartbeat_back_addr":"172.21.15.149:6815/40831","heartbeat_front_addr":"172.21.15.149:6813/40831","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T03:11:14.142479+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T03:11:17.471 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-12T03:11:17.471 INFO:tasks.ceph:Creating RBD pool 2024-04-12T03:11:17.472 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster c2 osd pool create rbd 8 2024-04-12T03:11:18.329 INFO:teuthology.orchestra.run.smithi149.stderr:pool 'rbd' created 2024-04-12T03:11:18.342 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-12T03:11:19.335 INFO:teuthology.orchestra.run.smithi149.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-12T03:11:19.351 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2024-04-12T03:11:19.351 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2024-04-12T03:11:19.352 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2024-04-12T03:11:19.352 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2024-04-12T03:11:19.352 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-12T03:11:19.780 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:11:19.795 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":40450},{"type":"v1","addr":"172.21.15.149:6825","nonce":40450}]},"active_addr":"172.21.15.149:6825/40450","active_change":"2024-04-12T03:11:15.261157+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.149:0","nonce":1742184604}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.149:0","nonce":87571170}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.149:0","nonce":3631567729}]}]} 2024-04-12T03:11:19.798 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2024-04-12T03:11:19.798 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2024-04-12T03:11:19.799 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-12T03:11:20.125 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:11:20.126 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":11,"fsid":"fe2a5e2e-a277-4813-952c-9c1e63e6f1cc","created":"2024-04-12T03:11:09.044013+0000","modified":"2024-04-12T03:11:19.322416+0000","last_up_change":"2024-04-12T03:11:17.079631+0000","last_in_change":"2024-04-12T03:11:10.040395+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T03:11:17.360055+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-12T03:11:17.812861+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e69be0fa-5c27-4187-83ba-0db366836d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":40829},{"type":"v1","addr":"172.21.15.149:6801","nonce":40829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":40829},{"type":"v1","addr":"172.21.15.149:6803","nonce":40829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":40829},{"type":"v1","addr":"172.21.15.149:6807","nonce":40829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":40829},{"type":"v1","addr":"172.21.15.149:6805","nonce":40829}]},"public_addr":"172.21.15.149:6801/40829","cluster_addr":"172.21.15.149:6803/40829","heartbeat_back_addr":"172.21.15.149:6807/40829","heartbeat_front_addr":"172.21.15.149:6805/40829","state":["exists","up"]},{"osd":1,"uuid":"8826dbf6-d1c8-4614-9b33-887a9cc8f7a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":40840},{"type":"v1","addr":"172.21.15.149:6817","nonce":40840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":40840},{"type":"v1","addr":"172.21.15.149:6819","nonce":40840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":40840},{"type":"v1","addr":"172.21.15.149:6823","nonce":40840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":40840},{"type":"v1","addr":"172.21.15.149:6821","nonce":40840}]},"public_addr":"172.21.15.149:6817/40840","cluster_addr":"172.21.15.149:6819/40840","heartbeat_back_addr":"172.21.15.149:6823/40840","heartbeat_front_addr":"172.21.15.149:6821/40840","state":["exists","up"]},{"osd":2,"uuid":"8969b929-490a-44f0-9464-dbcd1fdd2caa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":40831},{"type":"v1","addr":"172.21.15.149:6809","nonce":40831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":40831},{"type":"v1","addr":"172.21.15.149:6811","nonce":40831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":40831},{"type":"v1","addr":"172.21.15.149:6815","nonce":40831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":40831},{"type":"v1","addr":"172.21.15.149:6813","nonce":40831}]},"public_addr":"172.21.15.149:6809/40831","cluster_addr":"172.21.15.149:6811/40831","heartbeat_back_addr":"172.21.15.149:6815/40831","heartbeat_front_addr":"172.21.15.149:6813/40831","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T03:11:14.369277+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T03:11:14.142479+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T03:11:14.460097+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T03:11:20.139 INFO:tasks.ceph.ceph_manager.c2:all up! 2024-04-12T03:11:20.140 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-04-12T03:11:20.467 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:11:20.468 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":12,"fsid":"fe2a5e2e-a277-4813-952c-9c1e63e6f1cc","created":"2024-04-12T03:11:09.044013+0000","modified":"2024-04-12T03:11:20.326026+0000","last_up_change":"2024-04-12T03:11:17.079631+0000","last_in_change":"2024-04-12T03:11:10.040395+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T03:11:17.360055+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-12T03:11:17.812861+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e69be0fa-5c27-4187-83ba-0db366836d6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":40829},{"type":"v1","addr":"172.21.15.149:6801","nonce":40829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":40829},{"type":"v1","addr":"172.21.15.149:6803","nonce":40829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":40829},{"type":"v1","addr":"172.21.15.149:6807","nonce":40829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":40829},{"type":"v1","addr":"172.21.15.149:6805","nonce":40829}]},"public_addr":"172.21.15.149:6801/40829","cluster_addr":"172.21.15.149:6803/40829","heartbeat_back_addr":"172.21.15.149:6807/40829","heartbeat_front_addr":"172.21.15.149:6805/40829","state":["exists","up"]},{"osd":1,"uuid":"8826dbf6-d1c8-4614-9b33-887a9cc8f7a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":40840},{"type":"v1","addr":"172.21.15.149:6817","nonce":40840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":40840},{"type":"v1","addr":"172.21.15.149:6819","nonce":40840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":40840},{"type":"v1","addr":"172.21.15.149:6823","nonce":40840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":40840},{"type":"v1","addr":"172.21.15.149:6821","nonce":40840}]},"public_addr":"172.21.15.149:6817/40840","cluster_addr":"172.21.15.149:6819/40840","heartbeat_back_addr":"172.21.15.149:6823/40840","heartbeat_front_addr":"172.21.15.149:6821/40840","state":["exists","up"]},{"osd":2,"uuid":"8969b929-490a-44f0-9464-dbcd1fdd2caa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":40831},{"type":"v1","addr":"172.21.15.149:6809","nonce":40831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":40831},{"type":"v1","addr":"172.21.15.149:6811","nonce":40831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":40831},{"type":"v1","addr":"172.21.15.149:6815","nonce":40831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":40831},{"type":"v1","addr":"172.21.15.149:6813","nonce":40831}]},"public_addr":"172.21.15.149:6809/40831","cluster_addr":"172.21.15.149:6811/40831","heartbeat_back_addr":"172.21.15.149:6815/40831","heartbeat_front_addr":"172.21.15.149:6813/40831","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T03:11:14.369277+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T03:11:14.142479+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T03:11:14.460097+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T03:11:20.482 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2024-04-12T03:11:20.482 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2024-04-12T03:11:20.483 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2024-04-12T03:11:20.649 INFO:teuthology.orchestra.run.smithi149.stdout:38654705666 2024-04-12T03:11:20.650 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-04-12T03:11:20.657 INFO:teuthology.orchestra.run.smithi149.stdout:34359738370 2024-04-12T03:11:20.657 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-04-12T03:11:20.658 INFO:teuthology.orchestra.run.smithi149.stdout:38654705666 2024-04-12T03:11:20.658 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-04-12T03:11:20.994 INFO:teuthology.orchestra.run.smithi149.stdout:38654705665 2024-04-12T03:11:21.009 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705665 for osd.0 2024-04-12T03:11:21.040 INFO:teuthology.orchestra.run.smithi149.stdout:38654705665 2024-04-12T03:11:21.047 INFO:teuthology.orchestra.run.smithi149.stdout:34359738369 2024-04-12T03:11:21.055 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705665 for osd.2 2024-04-12T03:11:21.062 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738369 for osd.1 2024-04-12T03:11:22.010 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-04-12T03:11:22.057 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-04-12T03:11:22.062 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-04-12T03:11:22.380 INFO:teuthology.orchestra.run.smithi149.stdout:38654705666 2024-04-12T03:11:22.395 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705666 for osd.0 2024-04-12T03:11:22.395 DEBUG:teuthology.parallel:result is None 2024-04-12T03:11:22.425 INFO:teuthology.orchestra.run.smithi149.stdout:38654705666 2024-04-12T03:11:22.440 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705666 got 38654705666 for osd.2 2024-04-12T03:11:22.440 DEBUG:teuthology.parallel:result is None 2024-04-12T03:11:22.469 INFO:teuthology.orchestra.run.smithi149.stdout:34359738370 2024-04-12T03:11:22.483 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738370 got 34359738370 for osd.1 2024-04-12T03:11:22.484 DEBUG:teuthology.parallel:result is None 2024-04-12T03:11:22.484 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-04-12T03:11:22.484 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-04-12T03:11:22.845 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:11:22.847 INFO:teuthology.orchestra.run.smithi149.stderr:dumped all 2024-04-12T03:11:22.859 INFO:teuthology.orchestra.run.smithi149.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-12T03:11:21.319961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82880,"kb_used_data":1452,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489920,"statfs":{"total":322122547200,"available":322037678080,"internally_reserved":0,"allocated":1486848,"data_stored":1258271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4776,"internal_metadata":82963800},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T03:11:19.335307+0000","last_change":"2024-04-12T03:11:19.331945+0000","last_active":"2024-04-12T03:11:19.335307+0000","last_peered":"2024-04-12T03:11:19.335307+0000","last_clean":"2024-04-12T03:11:19.335307+0000","last_became_active":"2024-04-12T03:11:19.331658+0000","last_became_peered":"2024-04-12T03:11:19.331658+0000","last_unstale":"2024-04-12T03:11:19.335307+0000","last_undegraded":"2024-04-12T03:11:19.335307+0000","last_fullsized":"2024-04-12T03:11:19.335307+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T09:14:08.360980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T03:11:19.335267+0000","last_change":"2024-04-12T03:11:19.331909+0000","last_active":"2024-04-12T03:11:19.335267+0000","last_peered":"2024-04-12T03:11:19.335267+0000","last_clean":"2024-04-12T03:11:19.335267+0000","last_became_active":"2024-04-12T03:11:19.331608+0000","last_became_peered":"2024-04-12T03:11:19.331608+0000","last_unstale":"2024-04-12T03:11:19.335267+0000","last_undegraded":"2024-04-12T03:11:19.335267+0000","last_fullsized":"2024-04-12T03:11:19.335267+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T13:17:47.731218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T03:11:19.335292+0000","last_change":"2024-04-12T03:11:19.331980+0000","last_active":"2024-04-12T03:11:19.335292+0000","last_peered":"2024-04-12T03:11:19.335292+0000","last_clean":"2024-04-12T03:11:19.335292+0000","last_became_active":"2024-04-12T03:11:19.331724+0000","last_became_peered":"2024-04-12T03:11:19.331724+0000","last_unstale":"2024-04-12T03:11:19.335292+0000","last_undegraded":"2024-04-12T03:11:19.335292+0000","last_fullsized":"2024-04-12T03:11:19.335292+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T09:54:31.192076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:11:20.331729+0000","last_change":"2024-04-12T03:11:19.338878+0000","last_active":"2024-04-12T03:11:20.331729+0000","last_peered":"2024-04-12T03:11:20.331729+0000","last_clean":"2024-04-12T03:11:20.331729+0000","last_became_active":"2024-04-12T03:11:19.338594+0000","last_became_peered":"2024-04-12T03:11:19.338594+0000","last_unstale":"2024-04-12T03:11:20.331729+0000","last_undegraded":"2024-04-12T03:11:20.331729+0000","last_fullsized":"2024-04-12T03:11:20.331729+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T08:45:09.385700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T03:11:19.794897+0000","last_change":"2024-04-12T03:11:19.332206+0000","last_active":"2024-04-12T03:11:19.794897+0000","last_peered":"2024-04-12T03:11:19.794897+0000","last_clean":"2024-04-12T03:11:19.794897+0000","last_became_active":"2024-04-12T03:11:19.332006+0000","last_became_peered":"2024-04-12T03:11:19.332006+0000","last_unstale":"2024-04-12T03:11:19.794897+0000","last_undegraded":"2024-04-12T03:11:19.794897+0000","last_fullsized":"2024-04-12T03:11:19.794897+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T12:24:22.400845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T03:11:19.343551+0000","last_change":"2024-04-12T03:11:19.339235+0000","last_active":"2024-04-12T03:11:19.343551+0000","last_peered":"2024-04-12T03:11:19.343551+0000","last_clean":"2024-04-12T03:11:19.343551+0000","last_became_active":"2024-04-12T03:11:19.338920+0000","last_became_peered":"2024-04-12T03:11:19.338920+0000","last_unstale":"2024-04-12T03:11:19.343551+0000","last_undegraded":"2024-04-12T03:11:19.343551+0000","last_fullsized":"2024-04-12T03:11:19.343551+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T13:42:34.738563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:11:20.331756+0000","last_change":"2024-04-12T03:11:19.338897+0000","last_active":"2024-04-12T03:11:20.331756+0000","last_peered":"2024-04-12T03:11:20.331756+0000","last_clean":"2024-04-12T03:11:20.331756+0000","last_became_active":"2024-04-12T03:11:19.338629+0000","last_became_peered":"2024-04-12T03:11:19.338629+0000","last_unstale":"2024-04-12T03:11:20.331756+0000","last_undegraded":"2024-04-12T03:11:20.331756+0000","last_fullsized":"2024-04-12T03:11:20.331756+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T07:49:19.721248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T03:11:19.336308+0000","last_change":"2024-04-12T03:11:19.333707+0000","last_active":"2024-04-12T03:11:19.336308+0000","last_peered":"2024-04-12T03:11:19.336308+0000","last_clean":"2024-04-12T03:11:19.336308+0000","last_became_active":"2024-04-12T03:11:19.333501+0000","last_became_peered":"2024-04-12T03:11:19.333501+0000","last_unstale":"2024-04-12T03:11:19.336308+0000","last_undegraded":"2024-04-12T03:11:19.336308+0000","last_fullsized":"2024-04-12T03:11:19.336308+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T09:39:01.838336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T03:11:19.335305+0000","last_change":"2024-04-12T03:11:19.331950+0000","last_active":"2024-04-12T03:11:19.335305+0000","last_peered":"2024-04-12T03:11:19.335305+0000","last_clean":"2024-04-12T03:11:19.335305+0000","last_became_active":"2024-04-12T03:11:19.331656+0000","last_became_peered":"2024-04-12T03:11:19.331656+0000","last_unstale":"2024-04-12T03:11:19.335305+0000","last_undegraded":"2024-04-12T03:11:19.335305+0000","last_fullsized":"2024-04-12T03:11:19.335305+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T03:40:35.527216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27120,"kb_used_data":100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830480,"statfs":{"total":107374182400,"available":107346411520,"internally_reserved":0,"allocated":102400,"data_stored":27703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27880,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829720,"statfs":{"total":107374182400,"available":107345633280,"internally_reserved":0,"allocated":692224,"data_stored":615284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27880,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829720,"statfs":{"total":107374182400,"available":107345633280,"internally_reserved":0,"allocated":692224,"data_stored":615284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T03:11:22.860 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-04-12T03:11:23.180 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:11:23.182 INFO:teuthology.orchestra.run.smithi149.stderr:dumped all 2024-04-12T03:11:23.194 INFO:teuthology.orchestra.run.smithi149.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-12T03:11:21.319961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82880,"kb_used_data":1452,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":314489920,"statfs":{"total":322122547200,"available":322037678080,"internally_reserved":0,"allocated":1486848,"data_stored":1258271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4776,"internal_metadata":82963800},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T03:11:19.335307+0000","last_change":"2024-04-12T03:11:19.331945+0000","last_active":"2024-04-12T03:11:19.335307+0000","last_peered":"2024-04-12T03:11:19.335307+0000","last_clean":"2024-04-12T03:11:19.335307+0000","last_became_active":"2024-04-12T03:11:19.331658+0000","last_became_peered":"2024-04-12T03:11:19.331658+0000","last_unstale":"2024-04-12T03:11:19.335307+0000","last_undegraded":"2024-04-12T03:11:19.335307+0000","last_fullsized":"2024-04-12T03:11:19.335307+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T09:14:08.360980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T03:11:19.335267+0000","last_change":"2024-04-12T03:11:19.331909+0000","last_active":"2024-04-12T03:11:19.335267+0000","last_peered":"2024-04-12T03:11:19.335267+0000","last_clean":"2024-04-12T03:11:19.335267+0000","last_became_active":"2024-04-12T03:11:19.331608+0000","last_became_peered":"2024-04-12T03:11:19.331608+0000","last_unstale":"2024-04-12T03:11:19.335267+0000","last_undegraded":"2024-04-12T03:11:19.335267+0000","last_fullsized":"2024-04-12T03:11:19.335267+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T13:17:47.731218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T03:11:19.335292+0000","last_change":"2024-04-12T03:11:19.331980+0000","last_active":"2024-04-12T03:11:19.335292+0000","last_peered":"2024-04-12T03:11:19.335292+0000","last_clean":"2024-04-12T03:11:19.335292+0000","last_became_active":"2024-04-12T03:11:19.331724+0000","last_became_peered":"2024-04-12T03:11:19.331724+0000","last_unstale":"2024-04-12T03:11:19.335292+0000","last_undegraded":"2024-04-12T03:11:19.335292+0000","last_fullsized":"2024-04-12T03:11:19.335292+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T09:54:31.192076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:11:20.331729+0000","last_change":"2024-04-12T03:11:19.338878+0000","last_active":"2024-04-12T03:11:20.331729+0000","last_peered":"2024-04-12T03:11:20.331729+0000","last_clean":"2024-04-12T03:11:20.331729+0000","last_became_active":"2024-04-12T03:11:19.338594+0000","last_became_peered":"2024-04-12T03:11:19.338594+0000","last_unstale":"2024-04-12T03:11:20.331729+0000","last_undegraded":"2024-04-12T03:11:20.331729+0000","last_fullsized":"2024-04-12T03:11:20.331729+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T08:45:09.385700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T03:11:19.794897+0000","last_change":"2024-04-12T03:11:19.332206+0000","last_active":"2024-04-12T03:11:19.794897+0000","last_peered":"2024-04-12T03:11:19.794897+0000","last_clean":"2024-04-12T03:11:19.794897+0000","last_became_active":"2024-04-12T03:11:19.332006+0000","last_became_peered":"2024-04-12T03:11:19.332006+0000","last_unstale":"2024-04-12T03:11:19.794897+0000","last_undegraded":"2024-04-12T03:11:19.794897+0000","last_fullsized":"2024-04-12T03:11:19.794897+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T12:24:22.400845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T03:11:19.343551+0000","last_change":"2024-04-12T03:11:19.339235+0000","last_active":"2024-04-12T03:11:19.343551+0000","last_peered":"2024-04-12T03:11:19.343551+0000","last_clean":"2024-04-12T03:11:19.343551+0000","last_became_active":"2024-04-12T03:11:19.338920+0000","last_became_peered":"2024-04-12T03:11:19.338920+0000","last_unstale":"2024-04-12T03:11:19.343551+0000","last_undegraded":"2024-04-12T03:11:19.343551+0000","last_fullsized":"2024-04-12T03:11:19.343551+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T13:42:34.738563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-12T03:11:20.331756+0000","last_change":"2024-04-12T03:11:19.338897+0000","last_active":"2024-04-12T03:11:20.331756+0000","last_peered":"2024-04-12T03:11:20.331756+0000","last_clean":"2024-04-12T03:11:20.331756+0000","last_became_active":"2024-04-12T03:11:19.338629+0000","last_became_peered":"2024-04-12T03:11:19.338629+0000","last_unstale":"2024-04-12T03:11:20.331756+0000","last_undegraded":"2024-04-12T03:11:20.331756+0000","last_fullsized":"2024-04-12T03:11:20.331756+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T07:49:19.721248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T03:11:19.336308+0000","last_change":"2024-04-12T03:11:19.333707+0000","last_active":"2024-04-12T03:11:19.336308+0000","last_peered":"2024-04-12T03:11:19.336308+0000","last_clean":"2024-04-12T03:11:19.336308+0000","last_became_active":"2024-04-12T03:11:19.333501+0000","last_became_peered":"2024-04-12T03:11:19.333501+0000","last_unstale":"2024-04-12T03:11:19.336308+0000","last_undegraded":"2024-04-12T03:11:19.336308+0000","last_fullsized":"2024-04-12T03:11:19.336308+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T09:39:01.838336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-12T03:11:19.335305+0000","last_change":"2024-04-12T03:11:19.331950+0000","last_active":"2024-04-12T03:11:19.335305+0000","last_peered":"2024-04-12T03:11:19.335305+0000","last_clean":"2024-04-12T03:11:19.335305+0000","last_became_active":"2024-04-12T03:11:19.331656+0000","last_became_peered":"2024-04-12T03:11:19.331656+0000","last_unstale":"2024-04-12T03:11:19.335305+0000","last_undegraded":"2024-04-12T03:11:19.335305+0000","last_fullsized":"2024-04-12T03:11:19.335305+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T03:11:18.317998+0000","last_clean_scrub_stamp":"2024-04-12T03:11:18.317998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T03:40:35.527216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27120,"kb_used_data":100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104830480,"statfs":{"total":107374182400,"available":107346411520,"internally_reserved":0,"allocated":102400,"data_stored":27703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27880,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829720,"statfs":{"total":107374182400,"available":107345633280,"internally_reserved":0,"allocated":692224,"data_stored":615284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27880,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":104829720,"statfs":{"total":107374182400,"available":107345633280,"internally_reserved":0,"allocated":692224,"data_stored":615284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T03:11:23.195 INFO:tasks.ceph.ceph_manager.c2:clean! 2024-04-12T03:11:23.195 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2024-04-12T03:11:23.195 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2024-04-12T03:11:23.195 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2024-04-12T03:11:23.563 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-12T03:11:23.563 INFO:teuthology.orchestra.run.smithi149.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-12T03:11:23.577 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2024-04-12T03:11:23.577 INFO:teuthology.run_tasks:Running task rgw... 2024-04-12T03:11:23.592 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2024-04-12T03:11:23.592 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2024-04-12T03:11:23.593 INFO:tasks.rgw:Creating data pools 2024-04-12T03:11:23.593 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2024-04-12T03:11:23.593 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-04-12T03:11:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'default.rgw.buckets.data' created 2024-04-12T03:11:24.076 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-04-12T03:11:25.070 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-12T03:11:25.092 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-04-12T03:11:26.073 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'default.rgw.buckets.index' created 2024-04-12T03:11:26.087 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-04-12T03:11:27.097 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-12T03:11:27.111 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2024-04-12T03:11:27.111 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-04-12T03:11:27.479 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'default.rgw.buckets.data' already exists 2024-04-12T03:11:27.492 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-04-12T03:11:28.067 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-12T03:11:28.081 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-04-12T03:11:28.451 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'default.rgw.buckets.index' already exists 2024-04-12T03:11:28.464 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-04-12T03:11:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-12T03:11:29.087 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2024-04-12T03:11:29.087 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-04-12T03:11:30.356 INFO:teuthology.orchestra.run.smithi149.stderr:pool 'default.rgw.buckets.data' created 2024-04-12T03:11:30.372 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-04-12T03:11:31.375 INFO:teuthology.orchestra.run.smithi149.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-12T03:11:31.387 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-04-12T03:11:32.360 INFO:teuthology.orchestra.run.smithi149.stderr:pool 'default.rgw.buckets.index' created 2024-04-12T03:11:32.377 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-04-12T03:11:33.375 INFO:teuthology.orchestra.run.smithi149.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-12T03:11:33.395 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2024-04-12T03:11:33.395 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-04-12T03:11:33.724 INFO:teuthology.orchestra.run.smithi149.stderr:pool 'default.rgw.buckets.data' already exists 2024-04-12T03:11:33.736 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-04-12T03:11:34.364 INFO:teuthology.orchestra.run.smithi149.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-12T03:11:34.376 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-04-12T03:11:34.706 INFO:teuthology.orchestra.run.smithi149.stderr:pool 'default.rgw.buckets.index' already exists 2024-04-12T03:11:34.717 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-04-12T03:11:35.369 INFO:teuthology.orchestra.run.smithi149.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-12T03:11:35.381 DEBUG:tasks.rgw:Pools created 2024-04-12T03:11:35.381 INFO:tasks.rgw:Configuring compression type = random 2024-04-12T03:11:35.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2024-04-12T03:11:35.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2024-04-12T03:11:35.381 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2024-04-12T03:11:35.462 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:35.463 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:35.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:35.473+0000 7f4d7e4aa780 20 rados->read ofs=0 len=0 2024-04-12T03:11:36.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.983+0000 7f4d7e4aa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:36.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.983+0000 7f4d7e4aa780 20 realm 2024-04-12T03:11:36.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.983+0000 7f4d7e4aa780 20 rados->read ofs=0 len=0 2024-04-12T03:11:36.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.983+0000 7f4d7e4aa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:36.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.983+0000 7f4d7e4aa780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-12T03:11:36.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.983+0000 7f4d7e4aa780 20 rados->read ofs=0 len=0 2024-04-12T03:11:36.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.984+0000 7f4d7e4aa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:36.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.984+0000 7f4d7e4aa780 20 rados->read ofs=0 len=0 2024-04-12T03:11:36.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.985+0000 7f4d7e4aa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:36.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.985+0000 7f4d7e4aa780 20 rados->read ofs=0 len=0 2024-04-12T03:11:36.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.985+0000 7f4d7e4aa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:36.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.985+0000 7f4d7e4aa780 20 rados->read ofs=0 len=0 2024-04-12T03:11:36.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.986+0000 7f4d7e4aa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:36.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.986+0000 7f4d7e4aa780 10 Creating default zonegroup 2024-04-12T03:11:36.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.987+0000 7f4d7e4aa780 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-04-12T03:11:36.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.997+0000 7f4d7e4aa780 10 WARNING: driver->list_zones() returned r=-2 2024-04-12T03:11:36.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.997+0000 7f4d7e4aa780 20 rados->read ofs=0 len=0 2024-04-12T03:11:36.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:36.998+0000 7f4d7e4aa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:37.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.000+0000 7f4d7e4aa780 20 rados->read ofs=0 len=0 2024-04-12T03:11:37.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.000+0000 7f4d7e4aa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:37.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.000+0000 7f4d7e4aa780 10 could not read realm id: (2) No such file or directory 2024-04-12T03:11:37.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.000+0000 7f4d7e4aa780 10 WARNING: failed to set zone as default, r=-22 2024-04-12T03:11:37.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.000+0000 7f4d7e4aa780 20 rados->read ofs=0 len=0 2024-04-12T03:11:37.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.000+0000 7f4d7e4aa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:37.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.003+0000 7f4d7e4aa780 20 rados->read ofs=0 len=0 2024-04-12T03:11:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.003+0000 7f4d7e4aa780 20 rados_obj.operate() r=0 bl.length=874 2024-04-12T03:11:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.003+0000 7f4d7e4aa780 20 rados->read ofs=0 len=0 2024-04-12T03:11:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.003+0000 7f4d7e4aa780 20 rados_obj.operate() r=0 bl.length=420 2024-04-12T03:11:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.004+0000 7f4d7e4aa780 10 Using default name default 2024-04-12T03:11:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.004+0000 7f4d7e4aa780 20 rados->read ofs=0 len=0 2024-04-12T03:11:37.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.004+0000 7f4d7e4aa780 20 rados_obj.operate() r=0 bl.length=46 2024-04-12T03:11:37.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.004+0000 7f4d7e4aa780 20 rados->read ofs=0 len=0 2024-04-12T03:11:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.004+0000 7f4d7e4aa780 20 rados_obj.operate() r=0 bl.length=874 2024-04-12T03:11:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.004+0000 7f4d7e4aa780 20 zone default found 2024-04-12T03:11:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.004+0000 7f4d7e4aa780 4 Realm: () 2024-04-12T03:11:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.004+0000 7f4d7e4aa780 4 ZoneGroup: default (d5efc6c4-0fd1-416f-924c-f9f4ca6967cf) 2024-04-12T03:11:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.004+0000 7f4d7e4aa780 4 Zone: default (093a9360-a958-4011-aea6-1d1412b1233d) 2024-04-12T03:11:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.004+0000 7f4d7e4aa780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-12T03:11:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.004+0000 7f4d7e4aa780 20 zonegroup default 2024-04-12T03:11:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.004+0000 7f4d7e4aa780 20 rados->read ofs=0 len=0 2024-04-12T03:11:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.004+0000 7f4d7e4aa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:37.004+0000 7f4d7e4aa780 20 rados->read ofs=0 len=0 2024-04-12T03:11:38.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:38.990+0000 7f4d7e4aa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:38.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:38.990+0000 7f4d7e4aa780 20 rados->read ofs=0 len=0 2024-04-12T03:11:38.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:38.991+0000 7f4d7e4aa780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:38.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:38.991+0000 7f4d7e4aa780 20 started sync module instance, tier type = 2024-04-12T03:11:38.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:38.991+0000 7f4d7e4aa780 20 started zone id=093a9360-a958-4011-aea6-1d1412b1233d (name=default) with tier type = 2024-04-12T03:11:41.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.015+0000 7f4d7e4aa780 20 add_watcher() i=0 2024-04-12T03:11:41.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.015+0000 7f4d7e4aa780 20 add_watcher() i=1 2024-04-12T03:11:41.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.015+0000 7f4d7e4aa780 20 add_watcher() i=2 2024-04-12T03:11:41.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.015+0000 7f4d7e4aa780 20 add_watcher() i=3 2024-04-12T03:11:41.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.015+0000 7f4d7e4aa780 20 add_watcher() i=4 2024-04-12T03:11:41.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.015+0000 7f4d7e4aa780 20 add_watcher() i=5 2024-04-12T03:11:41.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.015+0000 7f4d7e4aa780 20 add_watcher() i=6 2024-04-12T03:11:41.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.016+0000 7f4d7e4aa780 20 add_watcher() i=7 2024-04-12T03:11:41.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.016+0000 7f4d7e4aa780 2 all 8 watchers are set, enabling cache 2024-04-12T03:11:41.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.016+0000 7f4d7e4aa780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-04-12T03:11:41.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.016+0000 7f4d7e4aa780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-04-12T03:11:41.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.016+0000 7f4d7e4aa780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T03:11:41.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.017+0000 7f4d7e4aa780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-04-12T03:11:41.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.017+0000 7f4d7e4aa780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-04-12T03:11:41.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.017+0000 7f4d7e4aa780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T03:11:41.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.017+0000 7f4d7e4aa780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-04-12T03:11:41.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.017+0000 7f4d7e4aa780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-04-12T03:11:41.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.017+0000 7f4d7e4aa780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T03:11:41.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.017+0000 7f4d7e4aa780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-04-12T03:11:41.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.017+0000 7f4d7e4aa780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-04-12T03:11:41.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.017+0000 7f4d7e4aa780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T03:11:41.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.017+0000 7f4d7e4aa780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-04-12T03:11:41.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.017+0000 7f4d7e4aa780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-04-12T03:11:41.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.019+0000 7f4d7e4aa780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T03:11:41.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.021+0000 7f4d63425700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-12T03:11:41.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.021+0000 7f4d7e4aa780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-12T03:11:41.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:41.021+0000 7f4d7e4aa780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-12T03:11:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:43.992+0000 7f4d7e4aa780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-04-12T03:11:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:43.992+0000 7f4d7e4aa780 5 note: GC not initialized 2024-04-12T03:11:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:43.993+0000 7f4d5ec1c700 20 reqs_thread_entry: start 2024-04-12T03:11:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:43.994+0000 7f4d7e4aa780 20 init_complete bucket index max shards: 11 2024-04-12T03:11:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:43.994+0000 7f4d7e4aa780 10 rgw notify: Started notification manager with: 1 workers 2024-04-12T03:11:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:43.994+0000 7f4d5cc18700 20 rgw notify: INFO: next queues processing will happen at: Fri Apr 12 03:12:14 2024 2024-04-12T03:11:43.995 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-12T03:11:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:43.994+0000 7f4d5c3f6700 20 reqs_thread_entry: start 2024-04-12T03:11:44.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.015+0000 7f4d63425700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-12T03:11:44.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.017+0000 7f4d7e4aa780 20 remove_watcher() i=0 2024-04-12T03:11:44.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.017+0000 7f4d7e4aa780 2 removed watcher, disabling cache 2024-04-12T03:11:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.018+0000 7f4d7e4aa780 20 remove_watcher() i=1 2024-04-12T03:11:44.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.019+0000 7f4d7e4aa780 20 remove_watcher() i=2 2024-04-12T03:11:44.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.020+0000 7f4d7e4aa780 20 remove_watcher() i=3 2024-04-12T03:11:44.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.022+0000 7f4d7e4aa780 20 remove_watcher() i=4 2024-04-12T03:11:44.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.023+0000 7f4d7e4aa780 20 remove_watcher() i=5 2024-04-12T03:11:44.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.024+0000 7f4d7e4aa780 20 remove_watcher() i=6 2024-04-12T03:11:44.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.025+0000 7f4d7e4aa780 20 remove_watcher() i=7 2024-04-12T03:11:44.028 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-04-12T03:11:44.031 DEBUG:tasks.util.rgw: json result: [] 2024-04-12T03:11:44.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-12T03:11:44.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-12T03:11:44.031 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-12T03:11:44.089 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:44.090 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:44.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.100+0000 7fd56a116780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-12T03:11:44.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.100+0000 7fd56a116780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-12T03:11:44.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.105+0000 7fd54b88a700 20 reqs_thread_entry: start 2024-04-12T03:11:44.113 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:11:44.113 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "093a9360-a958-4011-aea6-1d1412b1233d", 2024-04-12T03:11:44.113 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default", 2024-04-12T03:11:44.113 INFO:teuthology.orchestra.run.smithi043.stdout: "domain_root": "default.rgw.meta:root", 2024-04-12T03:11:44.113 INFO:teuthology.orchestra.run.smithi043.stdout: "control_pool": "default.rgw.control", 2024-04-12T03:11:44.113 INFO:teuthology.orchestra.run.smithi043.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-12T03:11:44.113 INFO:teuthology.orchestra.run.smithi043.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-12T03:11:44.113 INFO:teuthology.orchestra.run.smithi043.stdout: "log_pool": "default.rgw.log", 2024-04-12T03:11:44.113 INFO:teuthology.orchestra.run.smithi043.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-12T03:11:44.113 INFO:teuthology.orchestra.run.smithi043.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-12T03:11:44.113 INFO:teuthology.orchestra.run.smithi043.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-12T03:11:44.113 INFO:teuthology.orchestra.run.smithi043.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-12T03:11:44.114 INFO:teuthology.orchestra.run.smithi043.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-12T03:11:44.114 INFO:teuthology.orchestra.run.smithi043.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-12T03:11:44.114 INFO:teuthology.orchestra.run.smithi043.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-12T03:11:44.114 INFO:teuthology.orchestra.run.smithi043.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-12T03:11:44.114 INFO:teuthology.orchestra.run.smithi043.stdout: "otp_pool": "default.rgw.otp", 2024-04-12T03:11:44.114 INFO:teuthology.orchestra.run.smithi043.stdout: "system_key": { 2024-04-12T03:11:44.114 INFO:teuthology.orchestra.run.smithi043.stdout: "access_key": "", 2024-04-12T03:11:44.114 INFO:teuthology.orchestra.run.smithi043.stdout: "secret_key": "" 2024-04-12T03:11:44.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:11:44.114 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_pools": [ 2024-04-12T03:11:44.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:11:44.114 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "default-placement", 2024-04-12T03:11:44.114 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:11:44.115 INFO:teuthology.orchestra.run.smithi043.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-12T03:11:44.115 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": { 2024-04-12T03:11:44.115 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD": { 2024-04-12T03:11:44.115 INFO:teuthology.orchestra.run.smithi043.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-12T03:11:44.115 INFO:teuthology.orchestra.run.smithi043.stdout: "compression_type": "random" 2024-04-12T03:11:44.115 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:11:44.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:11:44.115 INFO:teuthology.orchestra.run.smithi043.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-12T03:11:44.115 INFO:teuthology.orchestra.run.smithi043.stdout: "index_type": 0, 2024-04-12T03:11:44.115 INFO:teuthology.orchestra.run.smithi043.stdout: "inline_data": true 2024-04-12T03:11:44.115 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:11:44.115 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:11:44.115 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:11:44.115 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "", 2024-04-12T03:11:44.115 INFO:teuthology.orchestra.run.smithi043.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-12T03:11:44.116 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:11:44.116 DEBUG:tasks.util.rgw: json result: {'id': '093a9360-a958-4011-aea6-1d1412b1233d', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-12T03:11:44.117 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2024-04-12T03:11:44.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2024-04-12T03:11:44.117 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2024-04-12T03:11:44.171 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:44.171 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:44.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.180+0000 7f010bcf8780 20 rados->read ofs=0 len=0 2024-04-12T03:11:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.181+0000 7f010bcf8780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.181+0000 7f010bcf8780 20 realm 2024-04-12T03:11:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.181+0000 7f010bcf8780 20 rados->read ofs=0 len=0 2024-04-12T03:11:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.181+0000 7f010bcf8780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.181+0000 7f010bcf8780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-12T03:11:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.181+0000 7f010bcf8780 20 rados->read ofs=0 len=0 2024-04-12T03:11:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.181+0000 7f010bcf8780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.181+0000 7f010bcf8780 20 rados->read ofs=0 len=0 2024-04-12T03:11:44.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.182+0000 7f010bcf8780 20 rados_obj.operate() r=0 bl.length=46 2024-04-12T03:11:44.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.182+0000 7f010bcf8780 20 rados->read ofs=0 len=0 2024-04-12T03:11:44.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.182+0000 7f010bcf8780 20 rados_obj.operate() r=0 bl.length=890 2024-04-12T03:11:44.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.182+0000 7f010bcf8780 20 searching for the correct realm 2024-04-12T03:11:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 20 RGWRados::pool_iterate: got zonegroup_info.d5efc6c4-0fd1-416f-924c-f9f4ca6967cf 2024-04-12T03:11:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 20 RGWRados::pool_iterate: got zone_info.093a9360-a958-4011-aea6-1d1412b1233d 2024-04-12T03:11:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-12T03:11:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-12T03:11:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 20 rados->read ofs=0 len=0 2024-04-12T03:11:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 20 rados->read ofs=0 len=0 2024-04-12T03:11:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 20 rados_obj.operate() r=0 bl.length=46 2024-04-12T03:11:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 20 rados->read ofs=0 len=0 2024-04-12T03:11:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 20 rados_obj.operate() r=0 bl.length=420 2024-04-12T03:11:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 20 zone default found 2024-04-12T03:11:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 4 Realm: () 2024-04-12T03:11:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 4 ZoneGroup: default (d5efc6c4-0fd1-416f-924c-f9f4ca6967cf) 2024-04-12T03:11:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 4 Zone: default (093a9360-a958-4011-aea6-1d1412b1233d) 2024-04-12T03:11:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-12T03:11:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 20 zonegroup default 2024-04-12T03:11:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.191+0000 7f010bcf8780 20 rados->read ofs=0 len=0 2024-04-12T03:11:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.192+0000 7f010bcf8780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.192+0000 7f010bcf8780 20 rados->read ofs=0 len=0 2024-04-12T03:11:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.192+0000 7f010bcf8780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.192+0000 7f010bcf8780 20 rados->read ofs=0 len=0 2024-04-12T03:11:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.192+0000 7f010bcf8780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.192+0000 7f010bcf8780 20 started sync module instance, tier type = 2024-04-12T03:11:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.192+0000 7f010bcf8780 20 started zone id=093a9360-a958-4011-aea6-1d1412b1233d (name=default) with tier type = 2024-04-12T03:11:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.201+0000 7f010bcf8780 20 add_watcher() i=0 2024-04-12T03:11:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.201+0000 7f010bcf8780 20 add_watcher() i=1 2024-04-12T03:11:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.201+0000 7f010bcf8780 20 add_watcher() i=2 2024-04-12T03:11:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.201+0000 7f010bcf8780 20 add_watcher() i=3 2024-04-12T03:11:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.201+0000 7f010bcf8780 20 add_watcher() i=4 2024-04-12T03:11:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.201+0000 7f010bcf8780 20 add_watcher() i=5 2024-04-12T03:11:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.201+0000 7f010bcf8780 20 add_watcher() i=6 2024-04-12T03:11:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.202+0000 7f010bcf8780 20 add_watcher() i=7 2024-04-12T03:11:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.202+0000 7f010bcf8780 2 all 8 watchers are set, enabling cache 2024-04-12T03:11:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.203+0000 7f00f0c73700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-12T03:11:44.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.204+0000 7f010bcf8780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-12T03:11:44.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.204+0000 7f010bcf8780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-12T03:11:44.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.208+0000 7f010bcf8780 5 note: GC not initialized 2024-04-12T03:11:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.209+0000 7f00ec46a700 20 reqs_thread_entry: start 2024-04-12T03:11:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.209+0000 7f010bcf8780 20 init_complete bucket index max shards: 11 2024-04-12T03:11:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.209+0000 7f010bcf8780 10 rgw notify: Started notification manager with: 1 workers 2024-04-12T03:11:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.209+0000 7f00ea466700 20 rgw notify: INFO: next queues processing will happen at: Fri Apr 12 03:12:14 2024 2024-04-12T03:11:44.210 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-12T03:11:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.209+0000 7f00e9c44700 20 reqs_thread_entry: start 2024-04-12T03:11:44.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.218+0000 7f00f0c73700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-12T03:11:44.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.220+0000 7f010bcf8780 20 remove_watcher() i=0 2024-04-12T03:11:44.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.220+0000 7f010bcf8780 2 removed watcher, disabling cache 2024-04-12T03:11:44.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.221+0000 7f010bcf8780 20 remove_watcher() i=1 2024-04-12T03:11:44.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.222+0000 7f010bcf8780 20 remove_watcher() i=2 2024-04-12T03:11:44.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.223+0000 7f010bcf8780 20 remove_watcher() i=3 2024-04-12T03:11:44.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.224+0000 7f010bcf8780 20 remove_watcher() i=4 2024-04-12T03:11:44.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.225+0000 7f010bcf8780 20 remove_watcher() i=5 2024-04-12T03:11:44.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.226+0000 7f010bcf8780 20 remove_watcher() i=6 2024-04-12T03:11:44.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.227+0000 7f010bcf8780 20 remove_watcher() i=7 2024-04-12T03:11:44.231 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-04-12T03:11:44.233 DEBUG:tasks.util.rgw: json result: [] 2024-04-12T03:11:44.233 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-12T03:11:44.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-12T03:11:44.234 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-12T03:11:44.291 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:44.291 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:44.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.300+0000 7f256cd4d780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-12T03:11:44.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.300+0000 7f256cd4d780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-12T03:11:44.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:44.304+0000 7f254e4c1700 20 reqs_thread_entry: start 2024-04-12T03:11:44.312 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:11:44.313 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "093a9360-a958-4011-aea6-1d1412b1233d", 2024-04-12T03:11:44.313 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default", 2024-04-12T03:11:44.313 INFO:teuthology.orchestra.run.smithi043.stdout: "domain_root": "default.rgw.meta:root", 2024-04-12T03:11:44.313 INFO:teuthology.orchestra.run.smithi043.stdout: "control_pool": "default.rgw.control", 2024-04-12T03:11:44.313 INFO:teuthology.orchestra.run.smithi043.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-12T03:11:44.313 INFO:teuthology.orchestra.run.smithi043.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-12T03:11:44.313 INFO:teuthology.orchestra.run.smithi043.stdout: "log_pool": "default.rgw.log", 2024-04-12T03:11:44.314 INFO:teuthology.orchestra.run.smithi043.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-12T03:11:44.314 INFO:teuthology.orchestra.run.smithi043.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-12T03:11:44.314 INFO:teuthology.orchestra.run.smithi043.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-12T03:11:44.314 INFO:teuthology.orchestra.run.smithi043.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-12T03:11:44.314 INFO:teuthology.orchestra.run.smithi043.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-12T03:11:44.314 INFO:teuthology.orchestra.run.smithi043.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-12T03:11:44.314 INFO:teuthology.orchestra.run.smithi043.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-12T03:11:44.315 INFO:teuthology.orchestra.run.smithi043.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-12T03:11:44.315 INFO:teuthology.orchestra.run.smithi043.stdout: "otp_pool": "default.rgw.otp", 2024-04-12T03:11:44.315 INFO:teuthology.orchestra.run.smithi043.stdout: "system_key": { 2024-04-12T03:11:44.315 INFO:teuthology.orchestra.run.smithi043.stdout: "access_key": "", 2024-04-12T03:11:44.315 INFO:teuthology.orchestra.run.smithi043.stdout: "secret_key": "" 2024-04-12T03:11:44.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:11:44.315 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_pools": [ 2024-04-12T03:11:44.315 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:11:44.316 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "default-placement", 2024-04-12T03:11:44.316 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:11:44.316 INFO:teuthology.orchestra.run.smithi043.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-12T03:11:44.316 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": { 2024-04-12T03:11:44.316 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD": { 2024-04-12T03:11:44.316 INFO:teuthology.orchestra.run.smithi043.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-12T03:11:44.316 INFO:teuthology.orchestra.run.smithi043.stdout: "compression_type": "random" 2024-04-12T03:11:44.316 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:11:44.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:11:44.316 INFO:teuthology.orchestra.run.smithi043.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-12T03:11:44.317 INFO:teuthology.orchestra.run.smithi043.stdout: "index_type": 0, 2024-04-12T03:11:44.317 INFO:teuthology.orchestra.run.smithi043.stdout: "inline_data": true 2024-04-12T03:11:44.317 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:11:44.317 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:11:44.317 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:11:44.317 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "", 2024-04-12T03:11:44.317 INFO:teuthology.orchestra.run.smithi043.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-12T03:11:44.317 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:11:44.319 DEBUG:tasks.util.rgw: json result: {'id': '093a9360-a958-4011-aea6-1d1412b1233d', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-12T03:11:44.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2024-04-12T03:11:44.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2024-04-12T03:11:44.319 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2024-04-12T03:11:44.408 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:44.409 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:44.419 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:44.417+0000 7fad0103d780 20 rados->read ofs=0 len=0 2024-04-12T03:11:46.391 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.389+0000 7fad0103d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:46.392 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.389+0000 7fad0103d780 20 realm 2024-04-12T03:11:46.392 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.389+0000 7fad0103d780 20 rados->read ofs=0 len=0 2024-04-12T03:11:46.393 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.390+0000 7fad0103d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:46.393 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.390+0000 7fad0103d780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-12T03:11:46.393 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.390+0000 7fad0103d780 20 rados->read ofs=0 len=0 2024-04-12T03:11:46.393 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.390+0000 7fad0103d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:46.393 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.390+0000 7fad0103d780 20 rados->read ofs=0 len=0 2024-04-12T03:11:46.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.401+0000 7fad0103d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:46.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.401+0000 7fad0103d780 20 rados->read ofs=0 len=0 2024-04-12T03:11:46.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.401+0000 7fad0103d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:46.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.401+0000 7fad0103d780 20 rados->read ofs=0 len=0 2024-04-12T03:11:46.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.401+0000 7fad0103d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:46.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.401+0000 7fad0103d780 10 Creating default zonegroup 2024-04-12T03:11:46.404 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.402+0000 7fad0103d780 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-04-12T03:11:46.412 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.410+0000 7fad0103d780 10 WARNING: driver->list_zones() returned r=-2 2024-04-12T03:11:46.412 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.410+0000 7fad0103d780 20 rados->read ofs=0 len=0 2024-04-12T03:11:46.413 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.411+0000 7fad0103d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:46.417 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.415+0000 7fad0103d780 20 rados->read ofs=0 len=0 2024-04-12T03:11:46.417 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.415+0000 7fad0103d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:46.417 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.415+0000 7fad0103d780 10 could not read realm id: (2) No such file or directory 2024-04-12T03:11:46.417 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.415+0000 7fad0103d780 10 WARNING: failed to set zone as default, r=-22 2024-04-12T03:11:46.417 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.415+0000 7fad0103d780 20 rados->read ofs=0 len=0 2024-04-12T03:11:46.417 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.415+0000 7fad0103d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:46.422 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.420+0000 7fad0103d780 20 rados->read ofs=0 len=0 2024-04-12T03:11:46.422 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.420+0000 7fad0103d780 20 rados_obj.operate() r=0 bl.length=874 2024-04-12T03:11:46.422 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.420+0000 7fad0103d780 20 rados->read ofs=0 len=0 2024-04-12T03:11:46.424 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.421+0000 7fad0103d780 20 rados_obj.operate() r=0 bl.length=420 2024-04-12T03:11:46.424 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.421+0000 7fad0103d780 10 Using default name default 2024-04-12T03:11:46.424 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.421+0000 7fad0103d780 20 rados->read ofs=0 len=0 2024-04-12T03:11:46.424 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.421+0000 7fad0103d780 20 rados_obj.operate() r=0 bl.length=46 2024-04-12T03:11:46.424 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.421+0000 7fad0103d780 20 rados->read ofs=0 len=0 2024-04-12T03:11:46.424 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.422+0000 7fad0103d780 20 rados_obj.operate() r=0 bl.length=874 2024-04-12T03:11:46.424 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.422+0000 7fad0103d780 20 zone default found 2024-04-12T03:11:46.424 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.422+0000 7fad0103d780 4 Realm: () 2024-04-12T03:11:46.424 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.422+0000 7fad0103d780 4 ZoneGroup: default (409690a3-b0c9-4903-86f9-c1a77bbc9b71) 2024-04-12T03:11:46.424 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.422+0000 7fad0103d780 4 Zone: default (c853a4c6-a4dd-44a1-abf1-3d809a4f082e) 2024-04-12T03:11:46.425 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.422+0000 7fad0103d780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-12T03:11:46.425 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.422+0000 7fad0103d780 20 zonegroup default 2024-04-12T03:11:46.425 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.422+0000 7fad0103d780 20 rados->read ofs=0 len=0 2024-04-12T03:11:46.425 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.422+0000 7fad0103d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:46.425 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:46.422+0000 7fad0103d780 20 rados->read ofs=0 len=0 2024-04-12T03:11:48.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:48.400+0000 7fad0103d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:48.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:48.400+0000 7fad0103d780 20 rados->read ofs=0 len=0 2024-04-12T03:11:48.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:48.401+0000 7fad0103d780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:48.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:48.401+0000 7fad0103d780 20 started sync module instance, tier type = 2024-04-12T03:11:48.403 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:48.401+0000 7fad0103d780 20 started zone id=c853a4c6-a4dd-44a1-abf1-3d809a4f082e (name=default) with tier type = 2024-04-12T03:11:50.444 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.442+0000 7fad0103d780 20 add_watcher() i=0 2024-04-12T03:11:50.444 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.442+0000 7fad0103d780 20 add_watcher() i=1 2024-04-12T03:11:50.444 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.442+0000 7fad0103d780 20 add_watcher() i=2 2024-04-12T03:11:50.445 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.442+0000 7fad0103d780 20 add_watcher() i=3 2024-04-12T03:11:50.445 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.442+0000 7fad0103d780 20 add_watcher() i=4 2024-04-12T03:11:50.445 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.442+0000 7fad0103d780 20 add_watcher() i=5 2024-04-12T03:11:50.445 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.442+0000 7fad0103d780 20 add_watcher() i=6 2024-04-12T03:11:50.447 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.444+0000 7fad0103d780 20 add_watcher() i=7 2024-04-12T03:11:50.447 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.444+0000 7fad0103d780 2 all 8 watchers are set, enabling cache 2024-04-12T03:11:50.447 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.444+0000 7fad0103d780 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-04-12T03:11:50.447 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.445+0000 7fad0103d780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-04-12T03:11:50.447 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.445+0000 7fad0103d780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T03:11:50.447 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.445+0000 7fad0103d780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-04-12T03:11:50.447 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.445+0000 7fad0103d780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-04-12T03:11:50.447 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.445+0000 7fad0103d780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T03:11:50.447 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.445+0000 7fad0103d780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-04-12T03:11:50.447 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.445+0000 7fad0103d780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-04-12T03:11:50.448 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.445+0000 7fad0103d780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T03:11:50.448 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.445+0000 7fad0103d780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-04-12T03:11:50.448 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.445+0000 7fad0103d780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-04-12T03:11:50.448 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.445+0000 7fad0103d780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T03:11:50.448 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.445+0000 7fad0103d780 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-04-12T03:11:50.449 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.445+0000 7fad0103d780 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2024-04-12T03:11:50.449 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.447+0000 7fad0103d780 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2024-04-12T03:11:50.454 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.452+0000 7face5fb8700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-12T03:11:50.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.453+0000 7fad0103d780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-12T03:11:50.455 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:50.453+0000 7fad0103d780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-12T03:11:53.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.407+0000 7fad0103d780 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-04-12T03:11:53.409 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.407+0000 7fad0103d780 5 note: GC not initialized 2024-04-12T03:11:53.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.409+0000 7face17af700 20 reqs_thread_entry: start 2024-04-12T03:11:53.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.409+0000 7fad0103d780 20 init_complete bucket index max shards: 11 2024-04-12T03:11:53.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.409+0000 7fad0103d780 10 rgw notify: Started notification manager with: 1 workers 2024-04-12T03:11:53.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.409+0000 7facdf7ab700 20 rgw notify: INFO: next queues processing will happen at: Fri Apr 12 03:12:23 2024 2024-04-12T03:11:53.411 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-04-12T03:11:53.411 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.409+0000 7facdef89700 20 reqs_thread_entry: start 2024-04-12T03:11:53.434 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.432+0000 7face5fb8700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-12T03:11:53.436 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.434+0000 7fad0103d780 20 remove_watcher() i=0 2024-04-12T03:11:53.436 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.434+0000 7fad0103d780 2 removed watcher, disabling cache 2024-04-12T03:11:53.437 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.436+0000 7fad0103d780 20 remove_watcher() i=1 2024-04-12T03:11:53.438 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.437+0000 7fad0103d780 20 remove_watcher() i=2 2024-04-12T03:11:53.439 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.438+0000 7fad0103d780 20 remove_watcher() i=3 2024-04-12T03:11:53.440 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.438+0000 7fad0103d780 20 remove_watcher() i=4 2024-04-12T03:11:53.441 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.439+0000 7fad0103d780 20 remove_watcher() i=5 2024-04-12T03:11:53.442 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.440+0000 7fad0103d780 20 remove_watcher() i=6 2024-04-12T03:11:53.443 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.441+0000 7fad0103d780 20 remove_watcher() i=7 2024-04-12T03:11:53.446 INFO:teuthology.orchestra.run.smithi149.stdout:[] 2024-04-12T03:11:53.448 DEBUG:tasks.util.rgw: json result: [] 2024-04-12T03:11:53.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-12T03:11:53.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-12T03:11:53.448 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-12T03:11:53.499 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:53.500 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:53.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.506+0000 7f8b1215e780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-12T03:11:53.508 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.506+0000 7f8b1215e780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-12T03:11:53.511 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.510+0000 7f8af38d2700 20 reqs_thread_entry: start 2024-04-12T03:11:53.518 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-04-12T03:11:53.518 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "c853a4c6-a4dd-44a1-abf1-3d809a4f082e", 2024-04-12T03:11:53.518 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "default", 2024-04-12T03:11:53.518 INFO:teuthology.orchestra.run.smithi149.stdout: "domain_root": "default.rgw.meta:root", 2024-04-12T03:11:53.518 INFO:teuthology.orchestra.run.smithi149.stdout: "control_pool": "default.rgw.control", 2024-04-12T03:11:53.518 INFO:teuthology.orchestra.run.smithi149.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-12T03:11:53.519 INFO:teuthology.orchestra.run.smithi149.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-12T03:11:53.519 INFO:teuthology.orchestra.run.smithi149.stdout: "log_pool": "default.rgw.log", 2024-04-12T03:11:53.519 INFO:teuthology.orchestra.run.smithi149.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-12T03:11:53.519 INFO:teuthology.orchestra.run.smithi149.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-12T03:11:53.519 INFO:teuthology.orchestra.run.smithi149.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-12T03:11:53.519 INFO:teuthology.orchestra.run.smithi149.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-12T03:11:53.519 INFO:teuthology.orchestra.run.smithi149.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-12T03:11:53.519 INFO:teuthology.orchestra.run.smithi149.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-12T03:11:53.519 INFO:teuthology.orchestra.run.smithi149.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-12T03:11:53.519 INFO:teuthology.orchestra.run.smithi149.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-12T03:11:53.519 INFO:teuthology.orchestra.run.smithi149.stdout: "otp_pool": "default.rgw.otp", 2024-04-12T03:11:53.519 INFO:teuthology.orchestra.run.smithi149.stdout: "system_key": { 2024-04-12T03:11:53.519 INFO:teuthology.orchestra.run.smithi149.stdout: "access_key": "", 2024-04-12T03:11:53.519 INFO:teuthology.orchestra.run.smithi149.stdout: "secret_key": "" 2024-04-12T03:11:53.520 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:11:53.520 INFO:teuthology.orchestra.run.smithi149.stdout: "placement_pools": [ 2024-04-12T03:11:53.520 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:11:53.520 INFO:teuthology.orchestra.run.smithi149.stdout: "key": "default-placement", 2024-04-12T03:11:53.520 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:11:53.520 INFO:teuthology.orchestra.run.smithi149.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-12T03:11:53.520 INFO:teuthology.orchestra.run.smithi149.stdout: "storage_classes": { 2024-04-12T03:11:53.520 INFO:teuthology.orchestra.run.smithi149.stdout: "STANDARD": { 2024-04-12T03:11:53.520 INFO:teuthology.orchestra.run.smithi149.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-12T03:11:53.520 INFO:teuthology.orchestra.run.smithi149.stdout: "compression_type": "random" 2024-04-12T03:11:53.520 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:11:53.520 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:11:53.520 INFO:teuthology.orchestra.run.smithi149.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-12T03:11:53.521 INFO:teuthology.orchestra.run.smithi149.stdout: "index_type": 0, 2024-04-12T03:11:53.521 INFO:teuthology.orchestra.run.smithi149.stdout: "inline_data": true 2024-04-12T03:11:53.521 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:11:53.521 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:11:53.521 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:11:53.521 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "", 2024-04-12T03:11:53.521 INFO:teuthology.orchestra.run.smithi149.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-12T03:11:53.521 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-04-12T03:11:53.522 DEBUG:tasks.util.rgw: json result: {'id': 'c853a4c6-a4dd-44a1-abf1-3d809a4f082e', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-12T03:11:53.522 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2024-04-12T03:11:53.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2024-04-12T03:11:53.523 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2024-04-12T03:11:53.565 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:53.566 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:53.574 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.572+0000 7f0ea6459780 20 rados->read ofs=0 len=0 2024-04-12T03:11:53.576 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.573+0000 7f0ea6459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:53.576 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.573+0000 7f0ea6459780 20 realm 2024-04-12T03:11:53.576 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.573+0000 7f0ea6459780 20 rados->read ofs=0 len=0 2024-04-12T03:11:53.576 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.574+0000 7f0ea6459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:53.576 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.574+0000 7f0ea6459780 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-12T03:11:53.576 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.574+0000 7f0ea6459780 20 rados->read ofs=0 len=0 2024-04-12T03:11:53.576 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.574+0000 7f0ea6459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:53.576 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.574+0000 7f0ea6459780 20 rados->read ofs=0 len=0 2024-04-12T03:11:53.576 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.574+0000 7f0ea6459780 20 rados_obj.operate() r=0 bl.length=46 2024-04-12T03:11:53.576 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.574+0000 7f0ea6459780 20 rados->read ofs=0 len=0 2024-04-12T03:11:53.577 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.575+0000 7f0ea6459780 20 rados_obj.operate() r=0 bl.length=890 2024-04-12T03:11:53.577 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.575+0000 7f0ea6459780 20 searching for the correct realm 2024-04-12T03:11:53.585 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 20 RGWRados::pool_iterate: got zonegroup_info.409690a3-b0c9-4903-86f9-c1a77bbc9b71 2024-04-12T03:11:53.585 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 20 RGWRados::pool_iterate: got zone_info.c853a4c6-a4dd-44a1-abf1-3d809a4f082e 2024-04-12T03:11:53.585 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 20 RGWRados::pool_iterate: got zone_names.default 2024-04-12T03:11:53.585 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-12T03:11:53.585 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 20 rados->read ofs=0 len=0 2024-04-12T03:11:53.585 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:53.585 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 20 rados->read ofs=0 len=0 2024-04-12T03:11:53.585 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 20 rados_obj.operate() r=0 bl.length=46 2024-04-12T03:11:53.586 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 20 rados->read ofs=0 len=0 2024-04-12T03:11:53.586 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 20 rados_obj.operate() r=0 bl.length=420 2024-04-12T03:11:53.586 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 20 zone default found 2024-04-12T03:11:53.586 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 4 Realm: () 2024-04-12T03:11:53.586 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 4 ZoneGroup: default (409690a3-b0c9-4903-86f9-c1a77bbc9b71) 2024-04-12T03:11:53.586 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 4 Zone: default (c853a4c6-a4dd-44a1-abf1-3d809a4f082e) 2024-04-12T03:11:53.586 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-12T03:11:53.586 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 20 zonegroup default 2024-04-12T03:11:53.586 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.582+0000 7f0ea6459780 20 rados->read ofs=0 len=0 2024-04-12T03:11:53.586 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.583+0000 7f0ea6459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:53.586 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.583+0000 7f0ea6459780 20 rados->read ofs=0 len=0 2024-04-12T03:11:53.586 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.583+0000 7f0ea6459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:53.586 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.583+0000 7f0ea6459780 20 rados->read ofs=0 len=0 2024-04-12T03:11:53.587 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.583+0000 7f0ea6459780 20 rados_obj.operate() r=-2 bl.length=0 2024-04-12T03:11:53.587 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.583+0000 7f0ea6459780 20 started sync module instance, tier type = 2024-04-12T03:11:53.587 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.583+0000 7f0ea6459780 20 started zone id=c853a4c6-a4dd-44a1-abf1-3d809a4f082e (name=default) with tier type = 2024-04-12T03:11:53.595 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.593+0000 7f0ea6459780 20 add_watcher() i=0 2024-04-12T03:11:53.595 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.593+0000 7f0ea6459780 20 add_watcher() i=1 2024-04-12T03:11:53.595 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.593+0000 7f0ea6459780 20 add_watcher() i=2 2024-04-12T03:11:53.595 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.593+0000 7f0ea6459780 20 add_watcher() i=3 2024-04-12T03:11:53.595 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.593+0000 7f0ea6459780 20 add_watcher() i=4 2024-04-12T03:11:53.595 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.593+0000 7f0ea6459780 20 add_watcher() i=5 2024-04-12T03:11:53.595 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.593+0000 7f0ea6459780 20 add_watcher() i=6 2024-04-12T03:11:53.597 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.596+0000 7f0ea6459780 20 add_watcher() i=7 2024-04-12T03:11:53.597 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.596+0000 7f0ea6459780 2 all 8 watchers are set, enabling cache 2024-04-12T03:11:53.600 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.599+0000 7f0e8b3d4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-12T03:11:53.601 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.599+0000 7f0ea6459780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-12T03:11:53.601 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.599+0000 7f0ea6459780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-12T03:11:53.604 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.602+0000 7f0ea6459780 5 note: GC not initialized 2024-04-12T03:11:53.605 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.603+0000 7f0e86bcb700 20 reqs_thread_entry: start 2024-04-12T03:11:53.606 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.603+0000 7f0ea6459780 20 init_complete bucket index max shards: 11 2024-04-12T03:11:53.606 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.603+0000 7f0ea6459780 10 rgw notify: Started notification manager with: 1 workers 2024-04-12T03:11:53.606 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.603+0000 7f0e84bc7700 20 rgw notify: INFO: next queues processing will happen at: Fri Apr 12 03:12:23 2024 2024-04-12T03:11:53.606 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-04-12T03:11:53.606 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.604+0000 7f0e843a5700 20 reqs_thread_entry: start 2024-04-12T03:11:53.612 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.611+0000 7f0e8b3d4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-12T03:11:53.616 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.614+0000 7f0ea6459780 20 remove_watcher() i=0 2024-04-12T03:11:53.616 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.614+0000 7f0ea6459780 2 removed watcher, disabling cache 2024-04-12T03:11:53.621 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.619+0000 7f0ea6459780 20 remove_watcher() i=1 2024-04-12T03:11:53.622 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.620+0000 7f0ea6459780 20 remove_watcher() i=2 2024-04-12T03:11:53.625 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.623+0000 7f0ea6459780 20 remove_watcher() i=3 2024-04-12T03:11:53.628 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.626+0000 7f0ea6459780 20 remove_watcher() i=4 2024-04-12T03:11:53.630 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.629+0000 7f0ea6459780 20 remove_watcher() i=5 2024-04-12T03:11:53.631 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.629+0000 7f0ea6459780 20 remove_watcher() i=6 2024-04-12T03:11:53.633 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.631+0000 7f0ea6459780 20 remove_watcher() i=7 2024-04-12T03:11:53.635 INFO:teuthology.orchestra.run.smithi149.stdout:[] 2024-04-12T03:11:53.637 DEBUG:tasks.util.rgw: json result: [] 2024-04-12T03:11:53.638 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-12T03:11:53.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-04-12T03:11:53.638 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-04-12T03:11:53.726 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:53.726 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:53.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.733+0000 7fdec6835780 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-12T03:11:53.735 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.733+0000 7fdec6835780 20 check_secure_mon_conn(): mode 1 is insecure 2024-04-12T03:11:53.739 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:11:53.737+0000 7fdea7fa9700 20 reqs_thread_entry: start 2024-04-12T03:11:53.745 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-04-12T03:11:53.745 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "c853a4c6-a4dd-44a1-abf1-3d809a4f082e", 2024-04-12T03:11:53.745 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "default", 2024-04-12T03:11:53.745 INFO:teuthology.orchestra.run.smithi149.stdout: "domain_root": "default.rgw.meta:root", 2024-04-12T03:11:53.745 INFO:teuthology.orchestra.run.smithi149.stdout: "control_pool": "default.rgw.control", 2024-04-12T03:11:53.746 INFO:teuthology.orchestra.run.smithi149.stdout: "gc_pool": "default.rgw.log:gc", 2024-04-12T03:11:53.746 INFO:teuthology.orchestra.run.smithi149.stdout: "lc_pool": "default.rgw.log:lc", 2024-04-12T03:11:53.746 INFO:teuthology.orchestra.run.smithi149.stdout: "log_pool": "default.rgw.log", 2024-04-12T03:11:53.746 INFO:teuthology.orchestra.run.smithi149.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-04-12T03:11:53.746 INFO:teuthology.orchestra.run.smithi149.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-04-12T03:11:53.746 INFO:teuthology.orchestra.run.smithi149.stdout: "roles_pool": "default.rgw.meta:roles", 2024-04-12T03:11:53.746 INFO:teuthology.orchestra.run.smithi149.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-04-12T03:11:53.746 INFO:teuthology.orchestra.run.smithi149.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-04-12T03:11:53.746 INFO:teuthology.orchestra.run.smithi149.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-04-12T03:11:53.746 INFO:teuthology.orchestra.run.smithi149.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-04-12T03:11:53.747 INFO:teuthology.orchestra.run.smithi149.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-04-12T03:11:53.747 INFO:teuthology.orchestra.run.smithi149.stdout: "otp_pool": "default.rgw.otp", 2024-04-12T03:11:53.747 INFO:teuthology.orchestra.run.smithi149.stdout: "system_key": { 2024-04-12T03:11:53.747 INFO:teuthology.orchestra.run.smithi149.stdout: "access_key": "", 2024-04-12T03:11:53.747 INFO:teuthology.orchestra.run.smithi149.stdout: "secret_key": "" 2024-04-12T03:11:53.747 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:11:53.747 INFO:teuthology.orchestra.run.smithi149.stdout: "placement_pools": [ 2024-04-12T03:11:53.747 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:11:53.747 INFO:teuthology.orchestra.run.smithi149.stdout: "key": "default-placement", 2024-04-12T03:11:53.747 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:11:53.747 INFO:teuthology.orchestra.run.smithi149.stdout: "index_pool": "default.rgw.buckets.index", 2024-04-12T03:11:53.747 INFO:teuthology.orchestra.run.smithi149.stdout: "storage_classes": { 2024-04-12T03:11:53.748 INFO:teuthology.orchestra.run.smithi149.stdout: "STANDARD": { 2024-04-12T03:11:53.748 INFO:teuthology.orchestra.run.smithi149.stdout: "data_pool": "default.rgw.buckets.data", 2024-04-12T03:11:53.748 INFO:teuthology.orchestra.run.smithi149.stdout: "compression_type": "random" 2024-04-12T03:11:53.748 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:11:53.748 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:11:53.748 INFO:teuthology.orchestra.run.smithi149.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-04-12T03:11:53.748 INFO:teuthology.orchestra.run.smithi149.stdout: "index_type": 0, 2024-04-12T03:11:53.748 INFO:teuthology.orchestra.run.smithi149.stdout: "inline_data": true 2024-04-12T03:11:53.748 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:11:53.748 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:11:53.748 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:11:53.749 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "", 2024-04-12T03:11:53.749 INFO:teuthology.orchestra.run.smithi149.stdout: "notif_pool": "default.rgw.log:notif" 2024-04-12T03:11:53.749 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-04-12T03:11:53.750 DEBUG:tasks.util.rgw: json result: {'id': 'c853a4c6-a4dd-44a1-abf1-3d809a4f082e', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-04-12T03:11:53.750 INFO:tasks.rgw:Starting rgw... 2024-04-12T03:11:53.750 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2024-04-12T03:11:53.750 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-12T03:11:53.751 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-04-12T03:11:53.751 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-04-12T03:11:53.754 INFO:tasks.rgw.c1.client.0:Started 2024-04-12T03:11:53.754 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2024-04-12T03:11:53.754 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-12T03:11:53.754 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T03:11:53.755 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T03:11:53.757 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T03:11:53.757 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2024-04-12T03:11:53.757 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-12T03:11:53.757 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-12T03:11:53.757 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-12T03:11:53.760 INFO:tasks.rgw.c2.client.0:Started 2024-04-12T03:11:53.760 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2024-04-12T03:11:53.760 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-12T03:11:53.760 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-04-12T03:11:53.760 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-04-12T03:11:53.762 INFO:tasks.rgw.c2.client.1:Started 2024-04-12T03:11:53.762 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi043.front.sepia.ceph.com:8000/ 2024-04-12T03:11:53.762 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8000/ 2024-04-12T03:11:53.801 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:11:53.801 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:11:53.801 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi043.front.sepia.ceph.com port 8000: Connection refused 2024-04-12T03:11:53.802 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T03:11:54.803 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8000/ 2024-04-12T03:11:54.830 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:11:54.830 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:11:54.833 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-12T03:11:54.833 INFO:teuthology.orchestra.run.smithi043.stdout:anonymous 2024-04-12T03:11:54.833 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi043.front.sepia.ceph.com:8001/ 2024-04-12T03:11:54.833 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001/ 2024-04-12T03:11:54.899 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:11:54.899 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:11:54.900 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-12T03:11:54.901 INFO:teuthology.orchestra.run.smithi043.stdout:anonymous 2024-04-12T03:11:54.901 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi149.front.sepia.ceph.com:8000/ 2024-04-12T03:11:54.901 DEBUG:teuthology.orchestra.run.smithi149:> curl http://smithi149.front.sepia.ceph.com:8000/ 2024-04-12T03:11:54.937 INFO:teuthology.orchestra.run.smithi149.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:11:54.938 INFO:teuthology.orchestra.run.smithi149.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:11:54.938 INFO:teuthology.orchestra.run.smithi149.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T03:11:54.939 INFO:teuthology.orchestra.run.smithi149.stdout:anonymous 2024-04-12T03:11:54.939 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi149.front.sepia.ceph.com:8001/ 2024-04-12T03:11:54.940 DEBUG:teuthology.orchestra.run.smithi149:> curl http://smithi149.front.sepia.ceph.com:8001/ 2024-04-12T03:11:55.000 INFO:teuthology.orchestra.run.smithi149.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:11:55.000 INFO:teuthology.orchestra.run.smithi149.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:11:55.001 INFO:teuthology.orchestra.run.smithi149.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T03:11:55.002 INFO:teuthology.orchestra.run.smithi149.stdout:anonymous 2024-04-12T03:11:55.002 INFO:teuthology.run_tasks:Running task rgw-multisite... 2024-04-12T03:11:55.137 INFO:tasks.rgw_multisite:creating realm.. 2024-04-12T03:11:55.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:11:55.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:11:55.138 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:11:55.193 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:55.193 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:55.222 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:11:55.222 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:11:55.222 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "test-realm", 2024-04-12T03:11:55.222 INFO:teuthology.orchestra.run.smithi043.stdout: "current_period": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T03:11:55.222 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1 2024-04-12T03:11:55.223 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:11:55.224 INFO:tasks.rgw_multisite:creating master zonegroup.. 2024-04-12T03:11:55.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:11:55.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:11:55.225 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup a --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:11:55.278 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:55.278 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:55.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:55.293+0000 7f1b19eb4780 -1 old format 2024-04-12T03:11:55.306 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:11:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T03:11:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T03:11:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:11:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T03:11:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:11:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:11:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:11:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:11:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:11:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "", 2024-04-12T03:11:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [], 2024-04-12T03:11:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:11:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:11:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:11:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:11:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [] 2024-04-12T03:11:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:11:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:11:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:11:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:11:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:11:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:11:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:11:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:11:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:11:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:11:55.308 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:11:55.309 INFO:tasks.rgw_multisite:creating master zone.. 2024-04-12T03:11:55.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--endpoints', 'http://smithi043.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:11:55.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--endpoints', 'http://smithi043.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:11:55.309 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --default --master --access-key 6ESEI28JZW8UW9BY --secret oVIj7wCADBnecpdBF65ZyUvXEbg14PDu --endpoints http://smithi043.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:11:55.409 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:55.409 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:55.426 INFO:teuthology.orchestra.run.smithi043.stderr:NOTICE: set zone's realm_id=a55da930-a452-44e6-b1f3-2c34e822e09c 2024-04-12T03:11:55.443 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:11:55.443 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:11:55.443 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T03:11:55.444 INFO:teuthology.orchestra.run.smithi043.stdout: "domain_root": "a1.rgw.meta:root", 2024-04-12T03:11:55.444 INFO:teuthology.orchestra.run.smithi043.stdout: "control_pool": "a1.rgw.control", 2024-04-12T03:11:55.444 INFO:teuthology.orchestra.run.smithi043.stdout: "gc_pool": "a1.rgw.log:gc", 2024-04-12T03:11:55.444 INFO:teuthology.orchestra.run.smithi043.stdout: "lc_pool": "a1.rgw.log:lc", 2024-04-12T03:11:55.444 INFO:teuthology.orchestra.run.smithi043.stdout: "log_pool": "a1.rgw.log", 2024-04-12T03:11:55.444 INFO:teuthology.orchestra.run.smithi043.stdout: "intent_log_pool": "a1.rgw.log:intent", 2024-04-12T03:11:55.444 INFO:teuthology.orchestra.run.smithi043.stdout: "usage_log_pool": "a1.rgw.log:usage", 2024-04-12T03:11:55.444 INFO:teuthology.orchestra.run.smithi043.stdout: "roles_pool": "a1.rgw.meta:roles", 2024-04-12T03:11:55.444 INFO:teuthology.orchestra.run.smithi043.stdout: "reshard_pool": "a1.rgw.log:reshard", 2024-04-12T03:11:55.444 INFO:teuthology.orchestra.run.smithi043.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2024-04-12T03:11:55.444 INFO:teuthology.orchestra.run.smithi043.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2024-04-12T03:11:55.444 INFO:teuthology.orchestra.run.smithi043.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2024-04-12T03:11:55.444 INFO:teuthology.orchestra.run.smithi043.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2024-04-12T03:11:55.444 INFO:teuthology.orchestra.run.smithi043.stdout: "otp_pool": "a1.rgw.otp", 2024-04-12T03:11:55.445 INFO:teuthology.orchestra.run.smithi043.stdout: "system_key": { 2024-04-12T03:11:55.445 INFO:teuthology.orchestra.run.smithi043.stdout: "access_key": "6ESEI28JZW8UW9BY", 2024-04-12T03:11:55.445 INFO:teuthology.orchestra.run.smithi043.stdout: "secret_key": "oVIj7wCADBnecpdBF65ZyUvXEbg14PDu" 2024-04-12T03:11:55.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:11:55.445 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_pools": [ 2024-04-12T03:11:55.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:11:55.445 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "default-placement", 2024-04-12T03:11:55.445 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:11:55.445 INFO:teuthology.orchestra.run.smithi043.stdout: "index_pool": "a1.rgw.buckets.index", 2024-04-12T03:11:55.445 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": { 2024-04-12T03:11:55.445 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD": { 2024-04-12T03:11:55.445 INFO:teuthology.orchestra.run.smithi043.stdout: "data_pool": "a1.rgw.buckets.data" 2024-04-12T03:11:55.445 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:11:55.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:11:55.446 INFO:teuthology.orchestra.run.smithi043.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2024-04-12T03:11:55.446 INFO:teuthology.orchestra.run.smithi043.stdout: "index_type": 0, 2024-04-12T03:11:55.446 INFO:teuthology.orchestra.run.smithi043.stdout: "inline_data": true 2024-04-12T03:11:55.446 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:11:55.446 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:11:55.446 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:11:55.446 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:11:55.446 INFO:teuthology.orchestra.run.smithi043.stdout: "notif_pool": "a1.rgw.log:notif" 2024-04-12T03:11:55.446 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:11:55.447 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph osd pool create a1.rgw.buckets.data 64 64 --cluster c1 2024-04-12T03:11:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'a1.rgw.buckets.data' created 2024-04-12T03:11:56.059 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph osd pool application enable a1.rgw.buckets.data rgw --cluster c1 2024-04-12T03:11:57.068 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rgw' on pool 'a1.rgw.buckets.data' 2024-04-12T03:11:57.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:11:57.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:11:57.089 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:11:57.146 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:57.146 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:57.167 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:11:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:11:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T03:11:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "domain_root": "a1.rgw.meta:root", 2024-04-12T03:11:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "control_pool": "a1.rgw.control", 2024-04-12T03:11:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "gc_pool": "a1.rgw.log:gc", 2024-04-12T03:11:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "lc_pool": "a1.rgw.log:lc", 2024-04-12T03:11:57.167 INFO:teuthology.orchestra.run.smithi043.stdout: "log_pool": "a1.rgw.log", 2024-04-12T03:11:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "intent_log_pool": "a1.rgw.log:intent", 2024-04-12T03:11:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "usage_log_pool": "a1.rgw.log:usage", 2024-04-12T03:11:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "roles_pool": "a1.rgw.meta:roles", 2024-04-12T03:11:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "reshard_pool": "a1.rgw.log:reshard", 2024-04-12T03:11:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2024-04-12T03:11:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2024-04-12T03:11:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2024-04-12T03:11:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2024-04-12T03:11:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "otp_pool": "a1.rgw.otp", 2024-04-12T03:11:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "system_key": { 2024-04-12T03:11:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "access_key": "6ESEI28JZW8UW9BY", 2024-04-12T03:11:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "secret_key": "oVIj7wCADBnecpdBF65ZyUvXEbg14PDu" 2024-04-12T03:11:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:11:57.168 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_pools": [ 2024-04-12T03:11:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:11:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "default-placement", 2024-04-12T03:11:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:11:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: "index_pool": "a1.rgw.buckets.index", 2024-04-12T03:11:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": { 2024-04-12T03:11:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD": { 2024-04-12T03:11:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: "data_pool": "a1.rgw.buckets.data", 2024-04-12T03:11:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: "compression_type": "random" 2024-04-12T03:11:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:11:57.169 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:11:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2024-04-12T03:11:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "index_type": 0, 2024-04-12T03:11:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "inline_data": true 2024-04-12T03:11:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:11:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:11:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:11:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:11:57.170 INFO:teuthology.orchestra.run.smithi043.stdout: "notif_pool": "a1.rgw.log:notif" 2024-04-12T03:11:57.170 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:11:57.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:11:57.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:11:57.172 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:11:57.225 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:11:57.225 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:11:57.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:11:57.237+0000 7f84a1a03780 0 period (ce930f76-1fa9-4790-8e77-21dfefa411cf does not have zone 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61 configured 2024-04-12T03:12:04.129 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:12:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:12:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-12T03:12:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T03:12:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T03:12:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T03:12:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:12:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T03:12:04.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T03:12:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T03:12:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:12:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T03:12:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:12:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:12:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:12:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:12:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:12:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:12:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:12:04.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:12:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T03:12:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:12:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:12:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:12:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:12:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": false, 2024-04-12T03:12:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:12:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:12:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:12:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:12:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:12:04.131 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:12:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:12:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:12:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:12:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:12:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:12:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:12:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:12:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:12:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:12:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:12:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:12:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:04.132 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:12:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:12:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:12:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:12:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:12:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:12:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:12:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:12:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T03:12:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:12:04.133 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T03:12:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:12:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T03:12:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:12:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T03:12:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T03:12:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:12:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:12:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:12:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:12:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:12:04.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T03:12:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:12:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:12:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:12:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:12:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:12:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T03:12:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:12:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:12:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:12:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:12:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:12:04.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T03:12:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:12:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:12:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:12:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:12:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:12:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T03:12:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:12:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:12:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:12:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:12:04.136 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:12:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T03:12:04.137 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:04.137 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:12:04.138 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-04-12T03:12:04.138 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T03:12:04.139 INFO:tasks.rgw.c1.client.0.smithi043.stdout:2024-04-12T03:12:04.138+0000 7fa7dd1de700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 45806) UID: 0 2024-04-12T03:12:04.139 INFO:tasks.rgw.c1.client.0.smithi043.stdout:2024-04-12T03:12:04.138+0000 7fa7e23b37c0 -1 shutting down 2024-04-12T03:12:04.204 INFO:tasks.rgw.c1.client.0:Stopped 2024-04-12T03:12:04.204 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-04-12T03:12:04.204 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-04-12T03:12:04.247 INFO:tasks.rgw.c1.client.0:Started 2024-04-12T03:12:04.247 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8000 2024-04-12T03:12:04.272 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:12:04.272 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:12:04.273 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi043.front.sepia.ceph.com port 8000: Connection refused 2024-04-12T03:12:04.273 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T03:12:05.274 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8000 2024-04-12T03:12:05.301 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:12:05.301 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:12:05.302 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-12T03:12:05.303 INFO:teuthology.orchestra.run.smithi043.stdout:anonymous 2024-04-12T03:12:05.303 INFO:tasks.rgw_multisite:creating admin user.. 2024-04-12T03:12:05.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:05.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:05.304 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key 6ESEI28JZW8UW9BY --secret oVIj7wCADBnecpdBF65ZyUvXEbg14PDu --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:12:05.411 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:05.411 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:05.448 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:12:05.449 INFO:teuthology.orchestra.run.smithi043.stdout: "user_id": "realm-admin", 2024-04-12T03:12:05.449 INFO:teuthology.orchestra.run.smithi043.stdout: "display_name": "Realm Admin", 2024-04-12T03:12:05.449 INFO:teuthology.orchestra.run.smithi043.stdout: "email": "", 2024-04-12T03:12:05.449 INFO:teuthology.orchestra.run.smithi043.stdout: "suspended": 0, 2024-04-12T03:12:05.449 INFO:teuthology.orchestra.run.smithi043.stdout: "max_buckets": 1000, 2024-04-12T03:12:05.449 INFO:teuthology.orchestra.run.smithi043.stdout: "subusers": [], 2024-04-12T03:12:05.449 INFO:teuthology.orchestra.run.smithi043.stdout: "keys": [ 2024-04-12T03:12:05.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:05.449 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "realm-admin", 2024-04-12T03:12:05.449 INFO:teuthology.orchestra.run.smithi043.stdout: "access_key": "6ESEI28JZW8UW9BY", 2024-04-12T03:12:05.449 INFO:teuthology.orchestra.run.smithi043.stdout: "secret_key": "oVIj7wCADBnecpdBF65ZyUvXEbg14PDu" 2024-04-12T03:12:05.449 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:05.450 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:12:05.450 INFO:teuthology.orchestra.run.smithi043.stdout: "swift_keys": [], 2024-04-12T03:12:05.450 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": [], 2024-04-12T03:12:05.450 INFO:teuthology.orchestra.run.smithi043.stdout: "op_mask": "read, write, delete", 2024-04-12T03:12:05.450 INFO:teuthology.orchestra.run.smithi043.stdout: "system": true, 2024-04-12T03:12:05.450 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "", 2024-04-12T03:12:05.450 INFO:teuthology.orchestra.run.smithi043.stdout: "default_storage_class": "", 2024-04-12T03:12:05.450 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_tags": [], 2024-04-12T03:12:05.450 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T03:12:05.450 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:12:05.450 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:12:05.450 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:12:05.451 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:12:05.451 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:12:05.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:05.451 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T03:12:05.451 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:12:05.451 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:12:05.451 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:12:05.451 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:12:05.451 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:12:05.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:05.451 INFO:teuthology.orchestra.run.smithi043.stdout: "temp_url_keys": [], 2024-04-12T03:12:05.451 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "rgw", 2024-04-12T03:12:05.451 INFO:teuthology.orchestra.run.smithi043.stdout: "mfa_ids": [] 2024-04-12T03:12:05.451 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:12:05.452 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:12:05.462 INFO:tasks.rgw_multisite:creating zone.. 2024-04-12T03:12:05.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--endpoints', 'http://smithi043.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:05.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--endpoints', 'http://smithi043.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:05.463 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --access-key 6ESEI28JZW8UW9BY --secret oVIj7wCADBnecpdBF65ZyUvXEbg14PDu --endpoints http://smithi043.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:12:05.522 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:05.522 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:05.540 INFO:teuthology.orchestra.run.smithi043.stderr:NOTICE: set zone's realm_id=a55da930-a452-44e6-b1f3-2c34e822e09c 2024-04-12T03:12:05.556 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:12:05.556 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:12:05.556 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T03:12:05.556 INFO:teuthology.orchestra.run.smithi043.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-12T03:12:05.557 INFO:teuthology.orchestra.run.smithi043.stdout: "control_pool": "a2.rgw.control", 2024-04-12T03:12:05.557 INFO:teuthology.orchestra.run.smithi043.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-12T03:12:05.557 INFO:teuthology.orchestra.run.smithi043.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-12T03:12:05.557 INFO:teuthology.orchestra.run.smithi043.stdout: "log_pool": "a2.rgw.log", 2024-04-12T03:12:05.557 INFO:teuthology.orchestra.run.smithi043.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-12T03:12:05.557 INFO:teuthology.orchestra.run.smithi043.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-12T03:12:05.557 INFO:teuthology.orchestra.run.smithi043.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-12T03:12:05.557 INFO:teuthology.orchestra.run.smithi043.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-12T03:12:05.557 INFO:teuthology.orchestra.run.smithi043.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-12T03:12:05.557 INFO:teuthology.orchestra.run.smithi043.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-12T03:12:05.557 INFO:teuthology.orchestra.run.smithi043.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-12T03:12:05.557 INFO:teuthology.orchestra.run.smithi043.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-12T03:12:05.557 INFO:teuthology.orchestra.run.smithi043.stdout: "otp_pool": "a2.rgw.otp", 2024-04-12T03:12:05.557 INFO:teuthology.orchestra.run.smithi043.stdout: "system_key": { 2024-04-12T03:12:05.558 INFO:teuthology.orchestra.run.smithi043.stdout: "access_key": "6ESEI28JZW8UW9BY", 2024-04-12T03:12:05.558 INFO:teuthology.orchestra.run.smithi043.stdout: "secret_key": "oVIj7wCADBnecpdBF65ZyUvXEbg14PDu" 2024-04-12T03:12:05.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:05.558 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_pools": [ 2024-04-12T03:12:05.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:05.558 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "default-placement", 2024-04-12T03:12:05.558 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:12:05.558 INFO:teuthology.orchestra.run.smithi043.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-12T03:12:05.558 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": { 2024-04-12T03:12:05.558 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD": { 2024-04-12T03:12:05.558 INFO:teuthology.orchestra.run.smithi043.stdout: "data_pool": "a2.rgw.buckets.data" 2024-04-12T03:12:05.558 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:05.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:05.559 INFO:teuthology.orchestra.run.smithi043.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-12T03:12:05.559 INFO:teuthology.orchestra.run.smithi043.stdout: "index_type": 0, 2024-04-12T03:12:05.559 INFO:teuthology.orchestra.run.smithi043.stdout: "inline_data": true 2024-04-12T03:12:05.559 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:05.559 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:05.559 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:12:05.559 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:05.559 INFO:teuthology.orchestra.run.smithi043.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-12T03:12:05.559 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:12:05.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph osd pool create a2.rgw.buckets.data 64 64 --cluster c1 2024-04-12T03:12:06.097 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'a2.rgw.buckets.data' created 2024-04-12T03:12:06.110 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph osd pool application enable a2.rgw.buckets.data rgw --cluster c1 2024-04-12T03:12:07.092 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rgw' on pool 'a2.rgw.buckets.data' 2024-04-12T03:12:07.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:07.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:07.115 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:12:07.181 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:07.181 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:07.203 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:12:07.203 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:12:07.204 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T03:12:07.204 INFO:teuthology.orchestra.run.smithi043.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-12T03:12:07.204 INFO:teuthology.orchestra.run.smithi043.stdout: "control_pool": "a2.rgw.control", 2024-04-12T03:12:07.204 INFO:teuthology.orchestra.run.smithi043.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-12T03:12:07.204 INFO:teuthology.orchestra.run.smithi043.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-12T03:12:07.204 INFO:teuthology.orchestra.run.smithi043.stdout: "log_pool": "a2.rgw.log", 2024-04-12T03:12:07.204 INFO:teuthology.orchestra.run.smithi043.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-12T03:12:07.204 INFO:teuthology.orchestra.run.smithi043.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-12T03:12:07.204 INFO:teuthology.orchestra.run.smithi043.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-12T03:12:07.204 INFO:teuthology.orchestra.run.smithi043.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-12T03:12:07.204 INFO:teuthology.orchestra.run.smithi043.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-12T03:12:07.204 INFO:teuthology.orchestra.run.smithi043.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-12T03:12:07.204 INFO:teuthology.orchestra.run.smithi043.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-12T03:12:07.204 INFO:teuthology.orchestra.run.smithi043.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-12T03:12:07.205 INFO:teuthology.orchestra.run.smithi043.stdout: "otp_pool": "a2.rgw.otp", 2024-04-12T03:12:07.205 INFO:teuthology.orchestra.run.smithi043.stdout: "system_key": { 2024-04-12T03:12:07.205 INFO:teuthology.orchestra.run.smithi043.stdout: "access_key": "6ESEI28JZW8UW9BY", 2024-04-12T03:12:07.205 INFO:teuthology.orchestra.run.smithi043.stdout: "secret_key": "oVIj7wCADBnecpdBF65ZyUvXEbg14PDu" 2024-04-12T03:12:07.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:07.205 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_pools": [ 2024-04-12T03:12:07.205 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:07.206 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "default-placement", 2024-04-12T03:12:07.206 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:12:07.206 INFO:teuthology.orchestra.run.smithi043.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-12T03:12:07.206 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": { 2024-04-12T03:12:07.206 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD": { 2024-04-12T03:12:07.206 INFO:teuthology.orchestra.run.smithi043.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-12T03:12:07.207 INFO:teuthology.orchestra.run.smithi043.stdout: "compression_type": "random" 2024-04-12T03:12:07.207 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:07.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:07.207 INFO:teuthology.orchestra.run.smithi043.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-12T03:12:07.207 INFO:teuthology.orchestra.run.smithi043.stdout: "index_type": 0, 2024-04-12T03:12:07.207 INFO:teuthology.orchestra.run.smithi043.stdout: "inline_data": true 2024-04-12T03:12:07.207 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:07.207 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:07.207 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:12:07.208 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:07.208 INFO:teuthology.orchestra.run.smithi043.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-12T03:12:07.208 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:12:07.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:07.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:07.210 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:12:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:07.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:12:07.275+0000 7f67308dd780 0 period (b69a664f-970b-4d72-be0d-0071b991ee5c does not have zone 1ba046e0-d0ae-437d-b741-28b4299423c2 configured 2024-04-12T03:12:14.156 INFO:teuthology.orchestra.run.smithi043.stderr:Sending period to new master zone 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61 2024-04-12T03:12:14.179 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:12:14.179 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:12:14.179 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2, 2024-04-12T03:12:14.179 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T03:12:14.179 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T03:12:14.180 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T03:12:14.180 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:12:14.180 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T03:12:14.180 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:14.180 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T03:12:14.180 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T03:12:14.180 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:12:14.180 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T03:12:14.180 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:12:14.180 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:12:14.180 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:12:14.180 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:12:14.180 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:12:14.180 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:12:14.181 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:12:14.181 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:14.181 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:12:14.181 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T03:12:14.181 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:12:14.181 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T03:12:14.181 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:12:14.181 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:12:14.181 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:12:14.181 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:12:14.181 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:12:14.182 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:12:14.182 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:12:14.182 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:12:14.182 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:12:14.182 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:12:14.182 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:12:14.182 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:12:14.182 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:12:14.182 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:14.182 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:14.182 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:12:14.182 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T03:12:14.182 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:12:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:12:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:12:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:12:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:12:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:12:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:12:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:12:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:12:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:12:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:12:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:12:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:12:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:12:14.183 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:12:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:12:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:12:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:12:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:12:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:12:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:12:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:12:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:12:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:12:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:14.184 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:12:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:12:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:12:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:12:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:12:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:12:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T03:12:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:12:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T03:12:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:14.185 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:12:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T03:12:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:12:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T03:12:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:12:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T03:12:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T03:12:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:12:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:12:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:12:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:12:14.186 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:12:14.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:14.187 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T03:12:14.187 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:12:14.187 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:12:14.187 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:12:14.187 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:12:14.187 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:12:14.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:14.187 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T03:12:14.187 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:12:14.187 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:12:14.188 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:12:14.188 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:12:14.188 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:12:14.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:14.188 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T03:12:14.188 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:12:14.188 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:12:14.188 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:12:14.188 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:12:14.190 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:12:14.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:14.190 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T03:12:14.190 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:12:14.190 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:12:14.190 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:12:14.190 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:12:14.190 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:12:14.191 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:14.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:14.191 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:14.191 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T03:12:14.191 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:14.191 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:12:14.193 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T03:12:14.193 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T03:12:14.194 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:12:14.192+0000 7fc7ca7cb700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 45811) UID: 0 2024-04-12T03:12:14.194 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:12:14.192+0000 7fc7cf9a07c0 -1 shutting down 2024-04-12T03:12:15.931 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T03:12:20.294 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T03:12:20.294 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T03:12:20.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T03:12:20.298 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T03:12:20.298 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T03:12:20.323 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:12:20.323 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:12:20.324 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T03:12:20.325 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi043.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T03:12:21.325 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T03:12:21.352 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:12:21.352 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:12:21.353 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-12T03:12:21.354 INFO:teuthology.orchestra.run.smithi043.stdout:anonymous 2024-04-12T03:12:21.355 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-04-12T03:12:21.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi043.front.sepia.ceph.com:8000', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:21.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi043.front.sepia.ceph.com:8000', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:21.356 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi043.front.sepia.ceph.com:8000 --access-key 6ESEI28JZW8UW9BY --secret oVIj7wCADBnecpdBF65ZyUvXEbg14PDu --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T03:12:21.415 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:21.415 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:21.453 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-04-12T03:12:21.453 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:21.454 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "test-realm", 2024-04-12T03:12:21.454 INFO:teuthology.orchestra.run.smithi149.stdout: "current_period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:12:21.454 INFO:teuthology.orchestra.run.smithi149.stdout: "epoch": 2 2024-04-12T03:12:21.454 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-04-12T03:12:21.455 INFO:tasks.rgw_multisite:creating zonegroup.. 2024-04-12T03:12:21.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:21.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:21.455 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zonegroup set --rgw-zonegroup b --rgw-realm test-realm --default --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T03:12:21.502 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:21.502 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:21.516 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:12:21.514+0000 7f016b77c780 -1 old format 2024-04-12T03:12:21.523 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-04-12T03:12:21.523 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "b", 2024-04-12T03:12:21.523 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "b", 2024-04-12T03:12:21.523 INFO:teuthology.orchestra.run.smithi149.stdout: "api_name": "", 2024-04-12T03:12:21.523 INFO:teuthology.orchestra.run.smithi149.stdout: "is_master": false, 2024-04-12T03:12:21.523 INFO:teuthology.orchestra.run.smithi149.stdout: "endpoints": [ 2024-04-12T03:12:21.523 INFO:teuthology.orchestra.run.smithi149.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:12:21.524 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:21.524 INFO:teuthology.orchestra.run.smithi149.stdout: "hostnames": [], 2024-04-12T03:12:21.524 INFO:teuthology.orchestra.run.smithi149.stdout: "hostnames_s3website": [], 2024-04-12T03:12:21.524 INFO:teuthology.orchestra.run.smithi149.stdout: "master_zone": "", 2024-04-12T03:12:21.524 INFO:teuthology.orchestra.run.smithi149.stdout: "zones": [], 2024-04-12T03:12:21.524 INFO:teuthology.orchestra.run.smithi149.stdout: "placement_targets": [ 2024-04-12T03:12:21.524 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:21.524 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "default-placement", 2024-04-12T03:12:21.524 INFO:teuthology.orchestra.run.smithi149.stdout: "tags": [], 2024-04-12T03:12:21.524 INFO:teuthology.orchestra.run.smithi149.stdout: "storage_classes": [] 2024-04-12T03:12:21.524 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:21.524 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:21.524 INFO:teuthology.orchestra.run.smithi149.stdout: "default_placement": "default-placement", 2024-04-12T03:12:21.525 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:21.525 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_policy": { 2024-04-12T03:12:21.525 INFO:teuthology.orchestra.run.smithi149.stdout: "groups": [] 2024-04-12T03:12:21.525 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:21.525 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled_features": [ 2024-04-12T03:12:21.525 INFO:teuthology.orchestra.run.smithi149.stdout: "resharding" 2024-04-12T03:12:21.525 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:21.525 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-04-12T03:12:21.526 INFO:tasks.rgw_multisite:creating zone.. 2024-04-12T03:12:21.526 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--endpoints', 'http://smithi149.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:21.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--endpoints', 'http://smithi149.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:21.526 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --default --master --access-key 6ESEI28JZW8UW9BY --secret oVIj7wCADBnecpdBF65ZyUvXEbg14PDu --endpoints http://smithi149.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T03:12:21.619 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:21.619 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:21.636 INFO:teuthology.orchestra.run.smithi149.stderr:NOTICE: set zone's realm_id=a55da930-a452-44e6-b1f3-2c34e822e09c 2024-04-12T03:12:21.636 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T03:12:21.651 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-04-12T03:12:21.651 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:12:21.651 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "b1", 2024-04-12T03:12:21.651 INFO:teuthology.orchestra.run.smithi149.stdout: "domain_root": "b1.rgw.meta:root", 2024-04-12T03:12:21.651 INFO:teuthology.orchestra.run.smithi149.stdout: "control_pool": "b1.rgw.control", 2024-04-12T03:12:21.651 INFO:teuthology.orchestra.run.smithi149.stdout: "gc_pool": "b1.rgw.log:gc", 2024-04-12T03:12:21.651 INFO:teuthology.orchestra.run.smithi149.stdout: "lc_pool": "b1.rgw.log:lc", 2024-04-12T03:12:21.651 INFO:teuthology.orchestra.run.smithi149.stdout: "log_pool": "b1.rgw.log", 2024-04-12T03:12:21.651 INFO:teuthology.orchestra.run.smithi149.stdout: "intent_log_pool": "b1.rgw.log:intent", 2024-04-12T03:12:21.652 INFO:teuthology.orchestra.run.smithi149.stdout: "usage_log_pool": "b1.rgw.log:usage", 2024-04-12T03:12:21.652 INFO:teuthology.orchestra.run.smithi149.stdout: "roles_pool": "b1.rgw.meta:roles", 2024-04-12T03:12:21.652 INFO:teuthology.orchestra.run.smithi149.stdout: "reshard_pool": "b1.rgw.log:reshard", 2024-04-12T03:12:21.652 INFO:teuthology.orchestra.run.smithi149.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2024-04-12T03:12:21.652 INFO:teuthology.orchestra.run.smithi149.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2024-04-12T03:12:21.652 INFO:teuthology.orchestra.run.smithi149.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2024-04-12T03:12:21.652 INFO:teuthology.orchestra.run.smithi149.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2024-04-12T03:12:21.652 INFO:teuthology.orchestra.run.smithi149.stdout: "otp_pool": "b1.rgw.otp", 2024-04-12T03:12:21.652 INFO:teuthology.orchestra.run.smithi149.stdout: "system_key": { 2024-04-12T03:12:21.652 INFO:teuthology.orchestra.run.smithi149.stdout: "access_key": "6ESEI28JZW8UW9BY", 2024-04-12T03:12:21.652 INFO:teuthology.orchestra.run.smithi149.stdout: "secret_key": "oVIj7wCADBnecpdBF65ZyUvXEbg14PDu" 2024-04-12T03:12:21.653 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:21.653 INFO:teuthology.orchestra.run.smithi149.stdout: "placement_pools": [ 2024-04-12T03:12:21.653 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:21.653 INFO:teuthology.orchestra.run.smithi149.stdout: "key": "default-placement", 2024-04-12T03:12:21.653 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:12:21.653 INFO:teuthology.orchestra.run.smithi149.stdout: "index_pool": "b1.rgw.buckets.index", 2024-04-12T03:12:21.653 INFO:teuthology.orchestra.run.smithi149.stdout: "storage_classes": { 2024-04-12T03:12:21.653 INFO:teuthology.orchestra.run.smithi149.stdout: "STANDARD": { 2024-04-12T03:12:21.653 INFO:teuthology.orchestra.run.smithi149.stdout: "data_pool": "b1.rgw.buckets.data" 2024-04-12T03:12:21.653 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:21.653 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:21.653 INFO:teuthology.orchestra.run.smithi149.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2024-04-12T03:12:21.653 INFO:teuthology.orchestra.run.smithi149.stdout: "index_type": 0, 2024-04-12T03:12:21.653 INFO:teuthology.orchestra.run.smithi149.stdout: "inline_data": true 2024-04-12T03:12:21.654 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:21.654 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:21.654 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:21.654 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:21.654 INFO:teuthology.orchestra.run.smithi149.stdout: "notif_pool": "b1.rgw.log:notif" 2024-04-12T03:12:21.654 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-04-12T03:12:21.655 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph osd pool create b1.rgw.buckets.data 64 64 --cluster c2 2024-04-12T03:12:22.459 INFO:teuthology.orchestra.run.smithi149.stderr:pool 'b1.rgw.buckets.data' created 2024-04-12T03:12:22.478 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph osd pool application enable b1.rgw.buckets.data rgw --cluster c2 2024-04-12T03:12:23.495 INFO:teuthology.orchestra.run.smithi149.stderr:enabled application 'rgw' on pool 'b1.rgw.buckets.data' 2024-04-12T03:12:23.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:23.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:23.510 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T03:12:23.563 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:23.563 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:23.581 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-04-12T03:12:23.582 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:12:23.582 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "b1", 2024-04-12T03:12:23.582 INFO:teuthology.orchestra.run.smithi149.stdout: "domain_root": "b1.rgw.meta:root", 2024-04-12T03:12:23.582 INFO:teuthology.orchestra.run.smithi149.stdout: "control_pool": "b1.rgw.control", 2024-04-12T03:12:23.582 INFO:teuthology.orchestra.run.smithi149.stdout: "gc_pool": "b1.rgw.log:gc", 2024-04-12T03:12:23.582 INFO:teuthology.orchestra.run.smithi149.stdout: "lc_pool": "b1.rgw.log:lc", 2024-04-12T03:12:23.582 INFO:teuthology.orchestra.run.smithi149.stdout: "log_pool": "b1.rgw.log", 2024-04-12T03:12:23.582 INFO:teuthology.orchestra.run.smithi149.stdout: "intent_log_pool": "b1.rgw.log:intent", 2024-04-12T03:12:23.582 INFO:teuthology.orchestra.run.smithi149.stdout: "usage_log_pool": "b1.rgw.log:usage", 2024-04-12T03:12:23.582 INFO:teuthology.orchestra.run.smithi149.stdout: "roles_pool": "b1.rgw.meta:roles", 2024-04-12T03:12:23.582 INFO:teuthology.orchestra.run.smithi149.stdout: "reshard_pool": "b1.rgw.log:reshard", 2024-04-12T03:12:23.583 INFO:teuthology.orchestra.run.smithi149.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2024-04-12T03:12:23.583 INFO:teuthology.orchestra.run.smithi149.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2024-04-12T03:12:23.583 INFO:teuthology.orchestra.run.smithi149.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2024-04-12T03:12:23.583 INFO:teuthology.orchestra.run.smithi149.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2024-04-12T03:12:23.583 INFO:teuthology.orchestra.run.smithi149.stdout: "otp_pool": "b1.rgw.otp", 2024-04-12T03:12:23.583 INFO:teuthology.orchestra.run.smithi149.stdout: "system_key": { 2024-04-12T03:12:23.583 INFO:teuthology.orchestra.run.smithi149.stdout: "access_key": "6ESEI28JZW8UW9BY", 2024-04-12T03:12:23.583 INFO:teuthology.orchestra.run.smithi149.stdout: "secret_key": "oVIj7wCADBnecpdBF65ZyUvXEbg14PDu" 2024-04-12T03:12:23.583 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:23.583 INFO:teuthology.orchestra.run.smithi149.stdout: "placement_pools": [ 2024-04-12T03:12:23.583 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:23.583 INFO:teuthology.orchestra.run.smithi149.stdout: "key": "default-placement", 2024-04-12T03:12:23.583 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:12:23.583 INFO:teuthology.orchestra.run.smithi149.stdout: "index_pool": "b1.rgw.buckets.index", 2024-04-12T03:12:23.584 INFO:teuthology.orchestra.run.smithi149.stdout: "storage_classes": { 2024-04-12T03:12:23.584 INFO:teuthology.orchestra.run.smithi149.stdout: "STANDARD": { 2024-04-12T03:12:23.584 INFO:teuthology.orchestra.run.smithi149.stdout: "data_pool": "b1.rgw.buckets.data", 2024-04-12T03:12:23.584 INFO:teuthology.orchestra.run.smithi149.stdout: "compression_type": "random" 2024-04-12T03:12:23.584 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:23.584 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:23.584 INFO:teuthology.orchestra.run.smithi149.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2024-04-12T03:12:23.584 INFO:teuthology.orchestra.run.smithi149.stdout: "index_type": 0, 2024-04-12T03:12:23.584 INFO:teuthology.orchestra.run.smithi149.stdout: "inline_data": true 2024-04-12T03:12:23.584 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:23.584 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:23.584 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:23.584 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:23.584 INFO:teuthology.orchestra.run.smithi149.stdout: "notif_pool": "b1.rgw.log:notif" 2024-04-12T03:12:23.585 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-04-12T03:12:23.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:23.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:23.586 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T03:12:23.633 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:23.633 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:23.645 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:12:23.643+0000 7fded94ac780 0 period (b69a664f-970b-4d72-be0d-0071b991ee5c does not have zone 1873e84b-44dc-48e3-babd-40afdce2764a configured 2024-04-12T03:12:30.522 INFO:teuthology.orchestra.run.smithi149.stderr:Sending period to new master zone 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61 2024-04-12T03:12:30.562 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-04-12T03:12:30.563 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:12:30.563 INFO:teuthology.orchestra.run.smithi149.stdout: "epoch": 3, 2024-04-12T03:12:30.563 INFO:teuthology.orchestra.run.smithi149.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T03:12:30.563 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_status": [], 2024-04-12T03:12:30.563 INFO:teuthology.orchestra.run.smithi149.stdout: "period_map": { 2024-04-12T03:12:30.563 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:12:30.563 INFO:teuthology.orchestra.run.smithi149.stdout: "zonegroups": [ 2024-04-12T03:12:30.563 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:30.563 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "a", 2024-04-12T03:12:30.563 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "a", 2024-04-12T03:12:30.563 INFO:teuthology.orchestra.run.smithi149.stdout: "api_name": "", 2024-04-12T03:12:30.563 INFO:teuthology.orchestra.run.smithi149.stdout: "is_master": true, 2024-04-12T03:12:30.564 INFO:teuthology.orchestra.run.smithi149.stdout: "endpoints": [ 2024-04-12T03:12:30.564 INFO:teuthology.orchestra.run.smithi149.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:12:30.564 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:30.564 INFO:teuthology.orchestra.run.smithi149.stdout: "hostnames": [], 2024-04-12T03:12:30.564 INFO:teuthology.orchestra.run.smithi149.stdout: "hostnames_s3website": [], 2024-04-12T03:12:30.564 INFO:teuthology.orchestra.run.smithi149.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:12:30.564 INFO:teuthology.orchestra.run.smithi149.stdout: "zones": [ 2024-04-12T03:12:30.564 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:30.564 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:12:30.564 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "a2", 2024-04-12T03:12:30.564 INFO:teuthology.orchestra.run.smithi149.stdout: "endpoints": [ 2024-04-12T03:12:30.564 INFO:teuthology.orchestra.run.smithi149.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T03:12:30.564 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:30.564 INFO:teuthology.orchestra.run.smithi149.stdout: "log_meta": false, 2024-04-12T03:12:30.565 INFO:teuthology.orchestra.run.smithi149.stdout: "log_data": true, 2024-04-12T03:12:30.565 INFO:teuthology.orchestra.run.smithi149.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:12:30.565 INFO:teuthology.orchestra.run.smithi149.stdout: "read_only": false, 2024-04-12T03:12:30.565 INFO:teuthology.orchestra.run.smithi149.stdout: "tier_type": "", 2024-04-12T03:12:30.565 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_from_all": true, 2024-04-12T03:12:30.565 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_from": [], 2024-04-12T03:12:30.565 INFO:teuthology.orchestra.run.smithi149.stdout: "redirect_zone": "", 2024-04-12T03:12:30.565 INFO:teuthology.orchestra.run.smithi149.stdout: "supported_features": [ 2024-04-12T03:12:30.565 INFO:teuthology.orchestra.run.smithi149.stdout: "compress-encrypted", 2024-04-12T03:12:30.565 INFO:teuthology.orchestra.run.smithi149.stdout: "resharding" 2024-04-12T03:12:30.565 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:30.565 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:30.565 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:30.565 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:12:30.566 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "a1", 2024-04-12T03:12:30.566 INFO:teuthology.orchestra.run.smithi149.stdout: "endpoints": [ 2024-04-12T03:12:30.566 INFO:teuthology.orchestra.run.smithi149.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:12:30.566 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:30.566 INFO:teuthology.orchestra.run.smithi149.stdout: "log_meta": false, 2024-04-12T03:12:30.566 INFO:teuthology.orchestra.run.smithi149.stdout: "log_data": true, 2024-04-12T03:12:30.566 INFO:teuthology.orchestra.run.smithi149.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:12:30.566 INFO:teuthology.orchestra.run.smithi149.stdout: "read_only": false, 2024-04-12T03:12:30.566 INFO:teuthology.orchestra.run.smithi149.stdout: "tier_type": "", 2024-04-12T03:12:30.566 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_from_all": true, 2024-04-12T03:12:30.566 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_from": [], 2024-04-12T03:12:30.566 INFO:teuthology.orchestra.run.smithi149.stdout: "redirect_zone": "", 2024-04-12T03:12:30.567 INFO:teuthology.orchestra.run.smithi149.stdout: "supported_features": [ 2024-04-12T03:12:30.567 INFO:teuthology.orchestra.run.smithi149.stdout: "compress-encrypted", 2024-04-12T03:12:30.567 INFO:teuthology.orchestra.run.smithi149.stdout: "resharding" 2024-04-12T03:12:30.567 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:30.567 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:30.567 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:30.567 INFO:teuthology.orchestra.run.smithi149.stdout: "placement_targets": [ 2024-04-12T03:12:30.567 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:30.567 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "default-placement", 2024-04-12T03:12:30.567 INFO:teuthology.orchestra.run.smithi149.stdout: "tags": [], 2024-04-12T03:12:30.567 INFO:teuthology.orchestra.run.smithi149.stdout: "storage_classes": [ 2024-04-12T03:12:30.567 INFO:teuthology.orchestra.run.smithi149.stdout: "STANDARD" 2024-04-12T03:12:30.567 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:30.567 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:30.568 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:30.568 INFO:teuthology.orchestra.run.smithi149.stdout: "default_placement": "default-placement", 2024-04-12T03:12:30.568 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:30.568 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_policy": { 2024-04-12T03:12:30.568 INFO:teuthology.orchestra.run.smithi149.stdout: "groups": [] 2024-04-12T03:12:30.568 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:30.568 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled_features": [ 2024-04-12T03:12:30.568 INFO:teuthology.orchestra.run.smithi149.stdout: "resharding" 2024-04-12T03:12:30.569 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:30.569 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:30.569 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:30.569 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "b", 2024-04-12T03:12:30.569 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "b", 2024-04-12T03:12:30.569 INFO:teuthology.orchestra.run.smithi149.stdout: "api_name": "", 2024-04-12T03:12:30.570 INFO:teuthology.orchestra.run.smithi149.stdout: "is_master": false, 2024-04-12T03:12:30.570 INFO:teuthology.orchestra.run.smithi149.stdout: "endpoints": [ 2024-04-12T03:12:30.570 INFO:teuthology.orchestra.run.smithi149.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:12:30.570 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:30.570 INFO:teuthology.orchestra.run.smithi149.stdout: "hostnames": [], 2024-04-12T03:12:30.571 INFO:teuthology.orchestra.run.smithi149.stdout: "hostnames_s3website": [], 2024-04-12T03:12:30.571 INFO:teuthology.orchestra.run.smithi149.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:12:30.571 INFO:teuthology.orchestra.run.smithi149.stdout: "zones": [ 2024-04-12T03:12:30.571 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:30.572 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:12:30.572 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "b1", 2024-04-12T03:12:30.572 INFO:teuthology.orchestra.run.smithi149.stdout: "endpoints": [ 2024-04-12T03:12:30.572 INFO:teuthology.orchestra.run.smithi149.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:12:30.573 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:30.573 INFO:teuthology.orchestra.run.smithi149.stdout: "log_meta": false, 2024-04-12T03:12:30.573 INFO:teuthology.orchestra.run.smithi149.stdout: "log_data": false, 2024-04-12T03:12:30.573 INFO:teuthology.orchestra.run.smithi149.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:12:30.573 INFO:teuthology.orchestra.run.smithi149.stdout: "read_only": false, 2024-04-12T03:12:30.573 INFO:teuthology.orchestra.run.smithi149.stdout: "tier_type": "", 2024-04-12T03:12:30.574 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_from_all": true, 2024-04-12T03:12:30.574 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_from": [], 2024-04-12T03:12:30.574 INFO:teuthology.orchestra.run.smithi149.stdout: "redirect_zone": "", 2024-04-12T03:12:30.574 INFO:teuthology.orchestra.run.smithi149.stdout: "supported_features": [ 2024-04-12T03:12:30.574 INFO:teuthology.orchestra.run.smithi149.stdout: "compress-encrypted", 2024-04-12T03:12:30.579 INFO:teuthology.orchestra.run.smithi149.stdout: "resharding" 2024-04-12T03:12:30.579 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:30.579 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:30.579 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:30.579 INFO:teuthology.orchestra.run.smithi149.stdout: "placement_targets": [ 2024-04-12T03:12:30.579 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:30.579 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "default-placement", 2024-04-12T03:12:30.579 INFO:teuthology.orchestra.run.smithi149.stdout: "tags": [], 2024-04-12T03:12:30.579 INFO:teuthology.orchestra.run.smithi149.stdout: "storage_classes": [ 2024-04-12T03:12:30.579 INFO:teuthology.orchestra.run.smithi149.stdout: "STANDARD" 2024-04-12T03:12:30.580 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:30.580 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:30.580 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:30.580 INFO:teuthology.orchestra.run.smithi149.stdout: "default_placement": "default-placement", 2024-04-12T03:12:30.580 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:30.580 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_policy": { 2024-04-12T03:12:30.580 INFO:teuthology.orchestra.run.smithi149.stdout: "groups": [] 2024-04-12T03:12:30.580 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:30.580 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled_features": [ 2024-04-12T03:12:30.580 INFO:teuthology.orchestra.run.smithi149.stdout: "resharding" 2024-04-12T03:12:30.581 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:30.581 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:30.581 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:30.581 INFO:teuthology.orchestra.run.smithi149.stdout: "short_zone_ids": [ 2024-04-12T03:12:30.581 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:30.581 INFO:teuthology.orchestra.run.smithi149.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:12:30.581 INFO:teuthology.orchestra.run.smithi149.stdout: "val": 755891344 2024-04-12T03:12:30.581 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:30.581 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:30.582 INFO:teuthology.orchestra.run.smithi149.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:12:30.582 INFO:teuthology.orchestra.run.smithi149.stdout: "val": 4153048242 2024-04-12T03:12:30.582 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:30.582 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:30.582 INFO:teuthology.orchestra.run.smithi149.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:12:30.582 INFO:teuthology.orchestra.run.smithi149.stdout: "val": 400045239 2024-04-12T03:12:30.582 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:30.582 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:30.582 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:30.582 INFO:teuthology.orchestra.run.smithi149.stdout: "master_zonegroup": "a", 2024-04-12T03:12:30.583 INFO:teuthology.orchestra.run.smithi149.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:12:30.583 INFO:teuthology.orchestra.run.smithi149.stdout: "period_config": { 2024-04-12T03:12:30.583 INFO:teuthology.orchestra.run.smithi149.stdout: "bucket_quota": { 2024-04-12T03:12:30.583 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled": false, 2024-04-12T03:12:30.583 INFO:teuthology.orchestra.run.smithi149.stdout: "check_on_raw": false, 2024-04-12T03:12:30.583 INFO:teuthology.orchestra.run.smithi149.stdout: "max_size": -1, 2024-04-12T03:12:30.583 INFO:teuthology.orchestra.run.smithi149.stdout: "max_size_kb": 0, 2024-04-12T03:12:30.583 INFO:teuthology.orchestra.run.smithi149.stdout: "max_objects": -1 2024-04-12T03:12:30.583 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:30.584 INFO:teuthology.orchestra.run.smithi149.stdout: "user_quota": { 2024-04-12T03:12:30.584 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled": false, 2024-04-12T03:12:30.584 INFO:teuthology.orchestra.run.smithi149.stdout: "check_on_raw": false, 2024-04-12T03:12:30.584 INFO:teuthology.orchestra.run.smithi149.stdout: "max_size": -1, 2024-04-12T03:12:30.584 INFO:teuthology.orchestra.run.smithi149.stdout: "max_size_kb": 0, 2024-04-12T03:12:30.584 INFO:teuthology.orchestra.run.smithi149.stdout: "max_objects": -1 2024-04-12T03:12:30.584 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:30.585 INFO:teuthology.orchestra.run.smithi149.stdout: "user_ratelimit": { 2024-04-12T03:12:30.585 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_ops": 0, 2024-04-12T03:12:30.585 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_ops": 0, 2024-04-12T03:12:30.585 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_bytes": 0, 2024-04-12T03:12:30.585 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_bytes": 0, 2024-04-12T03:12:30.585 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled": false 2024-04-12T03:12:30.585 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:30.585 INFO:teuthology.orchestra.run.smithi149.stdout: "bucket_ratelimit": { 2024-04-12T03:12:30.586 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_ops": 0, 2024-04-12T03:12:30.586 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_ops": 0, 2024-04-12T03:12:30.586 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_bytes": 0, 2024-04-12T03:12:30.586 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_bytes": 0, 2024-04-12T03:12:30.586 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled": false 2024-04-12T03:12:30.586 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:30.586 INFO:teuthology.orchestra.run.smithi149.stdout: "anonymous_ratelimit": { 2024-04-12T03:12:30.586 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_ops": 0, 2024-04-12T03:12:30.587 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_ops": 0, 2024-04-12T03:12:30.587 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_bytes": 0, 2024-04-12T03:12:30.587 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_bytes": 0, 2024-04-12T03:12:30.587 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled": false 2024-04-12T03:12:30.587 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:30.587 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:30.587 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:30.587 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_name": "test-realm", 2024-04-12T03:12:30.587 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:12:30.587 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-04-12T03:12:30.589 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-12T03:12:30.589 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T03:12:30.590 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T03:12:30.588+0000 7f719ff5e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 45994) UID: 0 2024-04-12T03:12:30.590 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T03:12:30.588+0000 7f71a51337c0 -1 shutting down 2024-04-12T03:12:30.661 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-12T03:12:30.662 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-04-12T03:12:30.663 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-04-12T03:12:30.704 INFO:tasks.rgw.c2.client.0:Started 2024-04-12T03:12:30.705 DEBUG:teuthology.orchestra.run.smithi149:> curl http://smithi149.front.sepia.ceph.com:8000 2024-04-12T03:12:30.725 INFO:teuthology.orchestra.run.smithi149.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:12:30.726 INFO:teuthology.orchestra.run.smithi149.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:12:30.726 INFO:teuthology.orchestra.run.smithi149.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi149.front.sepia.ceph.com port 8000: Connection refused 2024-04-12T03:12:30.726 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T03:12:31.728 DEBUG:teuthology.orchestra.run.smithi149:> curl http://smithi149.front.sepia.ceph.com:8000 2024-04-12T03:12:31.750 INFO:teuthology.orchestra.run.smithi149.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:12:31.750 INFO:teuthology.orchestra.run.smithi149.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:12:31.751 INFO:teuthology.orchestra.run.smithi149.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T03:12:31.751 INFO:teuthology.orchestra.run.smithi149.stdout:anonymous 2024-04-12T03:12:31.752 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-04-12T03:12:31.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi043.front.sepia.ceph.com:8000', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--url', 'http://smithi043.front.sepia.ceph.com:8000', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:31.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi043.front.sepia.ceph.com:8000', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--url', 'http://smithi043.front.sepia.ceph.com:8000', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:31.752 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi043.front.sepia.ceph.com:8000 --access-key 6ESEI28JZW8UW9BY --secret oVIj7wCADBnecpdBF65ZyUvXEbg14PDu --url http://smithi043.front.sepia.ceph.com:8000 --access-key 6ESEI28JZW8UW9BY --secret oVIj7wCADBnecpdBF65ZyUvXEbg14PDu --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T03:12:31.848 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:31.848 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:31.893 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-04-12T03:12:31.893 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:31.893 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "test-realm", 2024-04-12T03:12:31.893 INFO:teuthology.orchestra.run.smithi149.stdout: "current_period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:12:31.893 INFO:teuthology.orchestra.run.smithi149.stdout: "epoch": 2 2024-04-12T03:12:31.893 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-04-12T03:12:31.894 INFO:tasks.rgw_multisite:creating zone.. 2024-04-12T03:12:31.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--endpoints', 'http://smithi149.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:31.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--endpoints', 'http://smithi149.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:31.895 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --access-key 6ESEI28JZW8UW9BY --secret oVIj7wCADBnecpdBF65ZyUvXEbg14PDu --endpoints http://smithi149.front.sepia.ceph.com:8001 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T03:12:31.947 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:31.947 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:31.961 INFO:teuthology.orchestra.run.smithi149.stderr:NOTICE: set zone's realm_id=a55da930-a452-44e6-b1f3-2c34e822e09c 2024-04-12T03:12:31.976 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-04-12T03:12:31.976 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:12:31.976 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "b2", 2024-04-12T03:12:31.976 INFO:teuthology.orchestra.run.smithi149.stdout: "domain_root": "b2.rgw.meta:root", 2024-04-12T03:12:31.976 INFO:teuthology.orchestra.run.smithi149.stdout: "control_pool": "b2.rgw.control", 2024-04-12T03:12:31.976 INFO:teuthology.orchestra.run.smithi149.stdout: "gc_pool": "b2.rgw.log:gc", 2024-04-12T03:12:31.976 INFO:teuthology.orchestra.run.smithi149.stdout: "lc_pool": "b2.rgw.log:lc", 2024-04-12T03:12:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "log_pool": "b2.rgw.log", 2024-04-12T03:12:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "intent_log_pool": "b2.rgw.log:intent", 2024-04-12T03:12:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "usage_log_pool": "b2.rgw.log:usage", 2024-04-12T03:12:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "roles_pool": "b2.rgw.meta:roles", 2024-04-12T03:12:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "reshard_pool": "b2.rgw.log:reshard", 2024-04-12T03:12:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2024-04-12T03:12:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2024-04-12T03:12:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2024-04-12T03:12:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2024-04-12T03:12:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "otp_pool": "b2.rgw.otp", 2024-04-12T03:12:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "system_key": { 2024-04-12T03:12:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "access_key": "6ESEI28JZW8UW9BY", 2024-04-12T03:12:31.977 INFO:teuthology.orchestra.run.smithi149.stdout: "secret_key": "oVIj7wCADBnecpdBF65ZyUvXEbg14PDu" 2024-04-12T03:12:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: "placement_pools": [ 2024-04-12T03:12:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: "key": "default-placement", 2024-04-12T03:12:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:12:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: "index_pool": "b2.rgw.buckets.index", 2024-04-12T03:12:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: "storage_classes": { 2024-04-12T03:12:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: "STANDARD": { 2024-04-12T03:12:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: "data_pool": "b2.rgw.buckets.data" 2024-04-12T03:12:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2024-04-12T03:12:31.978 INFO:teuthology.orchestra.run.smithi149.stdout: "index_type": 0, 2024-04-12T03:12:31.979 INFO:teuthology.orchestra.run.smithi149.stdout: "inline_data": true 2024-04-12T03:12:31.979 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:31.979 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:31.979 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:31.979 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:31.979 INFO:teuthology.orchestra.run.smithi149.stdout: "notif_pool": "b2.rgw.log:notif" 2024-04-12T03:12:31.979 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-04-12T03:12:31.980 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph osd pool create b2.rgw.buckets.data 64 64 --cluster c2 2024-04-12T03:12:32.503 INFO:teuthology.orchestra.run.smithi149.stderr:pool 'b2.rgw.buckets.data' created 2024-04-12T03:12:32.520 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph osd pool application enable b2.rgw.buckets.data rgw --cluster c2 2024-04-12T03:12:33.531 INFO:teuthology.orchestra.run.smithi149.stderr:enabled application 'rgw' on pool 'b2.rgw.buckets.data' 2024-04-12T03:12:33.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:33.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:33.547 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T03:12:33.600 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:33.601 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:33.620 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-04-12T03:12:33.620 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:12:33.620 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "b2", 2024-04-12T03:12:33.620 INFO:teuthology.orchestra.run.smithi149.stdout: "domain_root": "b2.rgw.meta:root", 2024-04-12T03:12:33.620 INFO:teuthology.orchestra.run.smithi149.stdout: "control_pool": "b2.rgw.control", 2024-04-12T03:12:33.620 INFO:teuthology.orchestra.run.smithi149.stdout: "gc_pool": "b2.rgw.log:gc", 2024-04-12T03:12:33.620 INFO:teuthology.orchestra.run.smithi149.stdout: "lc_pool": "b2.rgw.log:lc", 2024-04-12T03:12:33.620 INFO:teuthology.orchestra.run.smithi149.stdout: "log_pool": "b2.rgw.log", 2024-04-12T03:12:33.620 INFO:teuthology.orchestra.run.smithi149.stdout: "intent_log_pool": "b2.rgw.log:intent", 2024-04-12T03:12:33.620 INFO:teuthology.orchestra.run.smithi149.stdout: "usage_log_pool": "b2.rgw.log:usage", 2024-04-12T03:12:33.620 INFO:teuthology.orchestra.run.smithi149.stdout: "roles_pool": "b2.rgw.meta:roles", 2024-04-12T03:12:33.621 INFO:teuthology.orchestra.run.smithi149.stdout: "reshard_pool": "b2.rgw.log:reshard", 2024-04-12T03:12:33.621 INFO:teuthology.orchestra.run.smithi149.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2024-04-12T03:12:33.621 INFO:teuthology.orchestra.run.smithi149.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2024-04-12T03:12:33.621 INFO:teuthology.orchestra.run.smithi149.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2024-04-12T03:12:33.621 INFO:teuthology.orchestra.run.smithi149.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2024-04-12T03:12:33.621 INFO:teuthology.orchestra.run.smithi149.stdout: "otp_pool": "b2.rgw.otp", 2024-04-12T03:12:33.621 INFO:teuthology.orchestra.run.smithi149.stdout: "system_key": { 2024-04-12T03:12:33.621 INFO:teuthology.orchestra.run.smithi149.stdout: "access_key": "6ESEI28JZW8UW9BY", 2024-04-12T03:12:33.621 INFO:teuthology.orchestra.run.smithi149.stdout: "secret_key": "oVIj7wCADBnecpdBF65ZyUvXEbg14PDu" 2024-04-12T03:12:33.622 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:33.622 INFO:teuthology.orchestra.run.smithi149.stdout: "placement_pools": [ 2024-04-12T03:12:33.622 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:33.622 INFO:teuthology.orchestra.run.smithi149.stdout: "key": "default-placement", 2024-04-12T03:12:33.622 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:12:33.622 INFO:teuthology.orchestra.run.smithi149.stdout: "index_pool": "b2.rgw.buckets.index", 2024-04-12T03:12:33.622 INFO:teuthology.orchestra.run.smithi149.stdout: "storage_classes": { 2024-04-12T03:12:33.622 INFO:teuthology.orchestra.run.smithi149.stdout: "STANDARD": { 2024-04-12T03:12:33.622 INFO:teuthology.orchestra.run.smithi149.stdout: "data_pool": "b2.rgw.buckets.data", 2024-04-12T03:12:33.622 INFO:teuthology.orchestra.run.smithi149.stdout: "compression_type": "random" 2024-04-12T03:12:33.622 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:33.622 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:33.622 INFO:teuthology.orchestra.run.smithi149.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2024-04-12T03:12:33.623 INFO:teuthology.orchestra.run.smithi149.stdout: "index_type": 0, 2024-04-12T03:12:33.623 INFO:teuthology.orchestra.run.smithi149.stdout: "inline_data": true 2024-04-12T03:12:33.623 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:33.623 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:33.623 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:33.623 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:33.623 INFO:teuthology.orchestra.run.smithi149.stdout: "notif_pool": "b2.rgw.log:notif" 2024-04-12T03:12:33.623 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-04-12T03:12:33.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:33.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:33.624 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-04-12T03:12:33.671 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:33.672 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:33.683 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:12:33.682+0000 7f34c7a8e780 0 period (b69a664f-970b-4d72-be0d-0071b991ee5c does not have zone 334587ae-93af-45b4-96b1-ade7b1773c00 configured 2024-04-12T03:12:40.582 INFO:teuthology.orchestra.run.smithi149.stderr:Sending period to new master zone 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61 2024-04-12T03:12:40.625 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-04-12T03:12:40.625 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:12:40.625 INFO:teuthology.orchestra.run.smithi149.stdout: "epoch": 4, 2024-04-12T03:12:40.625 INFO:teuthology.orchestra.run.smithi149.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T03:12:40.625 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_status": [], 2024-04-12T03:12:40.625 INFO:teuthology.orchestra.run.smithi149.stdout: "period_map": { 2024-04-12T03:12:40.625 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:12:40.625 INFO:teuthology.orchestra.run.smithi149.stdout: "zonegroups": [ 2024-04-12T03:12:40.625 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:40.625 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "a", 2024-04-12T03:12:40.625 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "a", 2024-04-12T03:12:40.626 INFO:teuthology.orchestra.run.smithi149.stdout: "api_name": "", 2024-04-12T03:12:40.626 INFO:teuthology.orchestra.run.smithi149.stdout: "is_master": true, 2024-04-12T03:12:40.626 INFO:teuthology.orchestra.run.smithi149.stdout: "endpoints": [ 2024-04-12T03:12:40.626 INFO:teuthology.orchestra.run.smithi149.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:12:40.626 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:40.626 INFO:teuthology.orchestra.run.smithi149.stdout: "hostnames": [], 2024-04-12T03:12:40.626 INFO:teuthology.orchestra.run.smithi149.stdout: "hostnames_s3website": [], 2024-04-12T03:12:40.626 INFO:teuthology.orchestra.run.smithi149.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:12:40.626 INFO:teuthology.orchestra.run.smithi149.stdout: "zones": [ 2024-04-12T03:12:40.626 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:40.626 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:12:40.626 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "a2", 2024-04-12T03:12:40.626 INFO:teuthology.orchestra.run.smithi149.stdout: "endpoints": [ 2024-04-12T03:12:40.626 INFO:teuthology.orchestra.run.smithi149.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T03:12:40.627 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:40.627 INFO:teuthology.orchestra.run.smithi149.stdout: "log_meta": false, 2024-04-12T03:12:40.627 INFO:teuthology.orchestra.run.smithi149.stdout: "log_data": true, 2024-04-12T03:12:40.627 INFO:teuthology.orchestra.run.smithi149.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:12:40.627 INFO:teuthology.orchestra.run.smithi149.stdout: "read_only": false, 2024-04-12T03:12:40.627 INFO:teuthology.orchestra.run.smithi149.stdout: "tier_type": "", 2024-04-12T03:12:40.627 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_from_all": true, 2024-04-12T03:12:40.627 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_from": [], 2024-04-12T03:12:40.627 INFO:teuthology.orchestra.run.smithi149.stdout: "redirect_zone": "", 2024-04-12T03:12:40.627 INFO:teuthology.orchestra.run.smithi149.stdout: "supported_features": [ 2024-04-12T03:12:40.627 INFO:teuthology.orchestra.run.smithi149.stdout: "compress-encrypted", 2024-04-12T03:12:40.627 INFO:teuthology.orchestra.run.smithi149.stdout: "resharding" 2024-04-12T03:12:40.627 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:40.627 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:40.628 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:40.628 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:12:40.628 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "a1", 2024-04-12T03:12:40.628 INFO:teuthology.orchestra.run.smithi149.stdout: "endpoints": [ 2024-04-12T03:12:40.628 INFO:teuthology.orchestra.run.smithi149.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:12:40.628 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:40.628 INFO:teuthology.orchestra.run.smithi149.stdout: "log_meta": false, 2024-04-12T03:12:40.628 INFO:teuthology.orchestra.run.smithi149.stdout: "log_data": true, 2024-04-12T03:12:40.628 INFO:teuthology.orchestra.run.smithi149.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:12:40.628 INFO:teuthology.orchestra.run.smithi149.stdout: "read_only": false, 2024-04-12T03:12:40.628 INFO:teuthology.orchestra.run.smithi149.stdout: "tier_type": "", 2024-04-12T03:12:40.628 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_from_all": true, 2024-04-12T03:12:40.628 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_from": [], 2024-04-12T03:12:40.628 INFO:teuthology.orchestra.run.smithi149.stdout: "redirect_zone": "", 2024-04-12T03:12:40.629 INFO:teuthology.orchestra.run.smithi149.stdout: "supported_features": [ 2024-04-12T03:12:40.629 INFO:teuthology.orchestra.run.smithi149.stdout: "compress-encrypted", 2024-04-12T03:12:40.629 INFO:teuthology.orchestra.run.smithi149.stdout: "resharding" 2024-04-12T03:12:40.629 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:40.629 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:40.629 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:40.629 INFO:teuthology.orchestra.run.smithi149.stdout: "placement_targets": [ 2024-04-12T03:12:40.629 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:40.629 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "default-placement", 2024-04-12T03:12:40.629 INFO:teuthology.orchestra.run.smithi149.stdout: "tags": [], 2024-04-12T03:12:40.629 INFO:teuthology.orchestra.run.smithi149.stdout: "storage_classes": [ 2024-04-12T03:12:40.629 INFO:teuthology.orchestra.run.smithi149.stdout: "STANDARD" 2024-04-12T03:12:40.629 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:40.629 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:40.630 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:40.630 INFO:teuthology.orchestra.run.smithi149.stdout: "default_placement": "default-placement", 2024-04-12T03:12:40.630 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:40.630 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_policy": { 2024-04-12T03:12:40.630 INFO:teuthology.orchestra.run.smithi149.stdout: "groups": [] 2024-04-12T03:12:40.630 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:40.630 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled_features": [ 2024-04-12T03:12:40.630 INFO:teuthology.orchestra.run.smithi149.stdout: "resharding" 2024-04-12T03:12:40.630 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:40.630 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:40.630 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:40.630 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "b", 2024-04-12T03:12:40.631 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "b", 2024-04-12T03:12:40.631 INFO:teuthology.orchestra.run.smithi149.stdout: "api_name": "", 2024-04-12T03:12:40.631 INFO:teuthology.orchestra.run.smithi149.stdout: "is_master": false, 2024-04-12T03:12:40.631 INFO:teuthology.orchestra.run.smithi149.stdout: "endpoints": [ 2024-04-12T03:12:40.631 INFO:teuthology.orchestra.run.smithi149.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:12:40.631 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:40.631 INFO:teuthology.orchestra.run.smithi149.stdout: "hostnames": [], 2024-04-12T03:12:40.631 INFO:teuthology.orchestra.run.smithi149.stdout: "hostnames_s3website": [], 2024-04-12T03:12:40.631 INFO:teuthology.orchestra.run.smithi149.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:12:40.631 INFO:teuthology.orchestra.run.smithi149.stdout: "zones": [ 2024-04-12T03:12:40.631 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:40.631 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:12:40.631 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "b1", 2024-04-12T03:12:40.631 INFO:teuthology.orchestra.run.smithi149.stdout: "endpoints": [ 2024-04-12T03:12:40.632 INFO:teuthology.orchestra.run.smithi149.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:12:40.632 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:40.632 INFO:teuthology.orchestra.run.smithi149.stdout: "log_meta": false, 2024-04-12T03:12:40.632 INFO:teuthology.orchestra.run.smithi149.stdout: "log_data": true, 2024-04-12T03:12:40.632 INFO:teuthology.orchestra.run.smithi149.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:12:40.632 INFO:teuthology.orchestra.run.smithi149.stdout: "read_only": false, 2024-04-12T03:12:40.632 INFO:teuthology.orchestra.run.smithi149.stdout: "tier_type": "", 2024-04-12T03:12:40.632 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_from_all": true, 2024-04-12T03:12:40.632 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_from": [], 2024-04-12T03:12:40.632 INFO:teuthology.orchestra.run.smithi149.stdout: "redirect_zone": "", 2024-04-12T03:12:40.632 INFO:teuthology.orchestra.run.smithi149.stdout: "supported_features": [ 2024-04-12T03:12:40.632 INFO:teuthology.orchestra.run.smithi149.stdout: "compress-encrypted", 2024-04-12T03:12:40.641 INFO:teuthology.orchestra.run.smithi149.stdout: "resharding" 2024-04-12T03:12:40.641 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:40.641 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:40.641 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:40.642 INFO:teuthology.orchestra.run.smithi149.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:12:40.642 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "b2", 2024-04-12T03:12:40.642 INFO:teuthology.orchestra.run.smithi149.stdout: "endpoints": [ 2024-04-12T03:12:40.642 INFO:teuthology.orchestra.run.smithi149.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T03:12:40.642 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:40.642 INFO:teuthology.orchestra.run.smithi149.stdout: "log_meta": false, 2024-04-12T03:12:40.642 INFO:teuthology.orchestra.run.smithi149.stdout: "log_data": true, 2024-04-12T03:12:40.642 INFO:teuthology.orchestra.run.smithi149.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:12:40.642 INFO:teuthology.orchestra.run.smithi149.stdout: "read_only": false, 2024-04-12T03:12:40.642 INFO:teuthology.orchestra.run.smithi149.stdout: "tier_type": "", 2024-04-12T03:12:40.642 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_from_all": true, 2024-04-12T03:12:40.642 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_from": [], 2024-04-12T03:12:40.642 INFO:teuthology.orchestra.run.smithi149.stdout: "redirect_zone": "", 2024-04-12T03:12:40.642 INFO:teuthology.orchestra.run.smithi149.stdout: "supported_features": [ 2024-04-12T03:12:40.643 INFO:teuthology.orchestra.run.smithi149.stdout: "compress-encrypted", 2024-04-12T03:12:40.643 INFO:teuthology.orchestra.run.smithi149.stdout: "resharding" 2024-04-12T03:12:40.643 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:40.643 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:40.643 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:40.643 INFO:teuthology.orchestra.run.smithi149.stdout: "placement_targets": [ 2024-04-12T03:12:40.643 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:40.643 INFO:teuthology.orchestra.run.smithi149.stdout: "name": "default-placement", 2024-04-12T03:12:40.643 INFO:teuthology.orchestra.run.smithi149.stdout: "tags": [], 2024-04-12T03:12:40.643 INFO:teuthology.orchestra.run.smithi149.stdout: "storage_classes": [ 2024-04-12T03:12:40.643 INFO:teuthology.orchestra.run.smithi149.stdout: "STANDARD" 2024-04-12T03:12:40.643 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:40.643 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:40.643 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:40.644 INFO:teuthology.orchestra.run.smithi149.stdout: "default_placement": "default-placement", 2024-04-12T03:12:40.644 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:40.644 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_policy": { 2024-04-12T03:12:40.644 INFO:teuthology.orchestra.run.smithi149.stdout: "groups": [] 2024-04-12T03:12:40.644 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:40.644 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled_features": [ 2024-04-12T03:12:40.644 INFO:teuthology.orchestra.run.smithi149.stdout: "resharding" 2024-04-12T03:12:40.644 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:40.644 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:40.644 INFO:teuthology.orchestra.run.smithi149.stdout: ], 2024-04-12T03:12:40.644 INFO:teuthology.orchestra.run.smithi149.stdout: "short_zone_ids": [ 2024-04-12T03:12:40.644 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:40.644 INFO:teuthology.orchestra.run.smithi149.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:12:40.644 INFO:teuthology.orchestra.run.smithi149.stdout: "val": 755891344 2024-04-12T03:12:40.645 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:40.645 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:40.645 INFO:teuthology.orchestra.run.smithi149.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:12:40.645 INFO:teuthology.orchestra.run.smithi149.stdout: "val": 4153048242 2024-04-12T03:12:40.645 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:40.645 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:40.645 INFO:teuthology.orchestra.run.smithi149.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:12:40.645 INFO:teuthology.orchestra.run.smithi149.stdout: "val": 2639555996 2024-04-12T03:12:40.645 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:40.645 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:12:40.645 INFO:teuthology.orchestra.run.smithi149.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:12:40.645 INFO:teuthology.orchestra.run.smithi149.stdout: "val": 400045239 2024-04-12T03:12:40.645 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:40.645 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:12:40.645 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:40.646 INFO:teuthology.orchestra.run.smithi149.stdout: "master_zonegroup": "a", 2024-04-12T03:12:40.646 INFO:teuthology.orchestra.run.smithi149.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:12:40.646 INFO:teuthology.orchestra.run.smithi149.stdout: "period_config": { 2024-04-12T03:12:40.646 INFO:teuthology.orchestra.run.smithi149.stdout: "bucket_quota": { 2024-04-12T03:12:40.646 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled": false, 2024-04-12T03:12:40.646 INFO:teuthology.orchestra.run.smithi149.stdout: "check_on_raw": false, 2024-04-12T03:12:40.646 INFO:teuthology.orchestra.run.smithi149.stdout: "max_size": -1, 2024-04-12T03:12:40.646 INFO:teuthology.orchestra.run.smithi149.stdout: "max_size_kb": 0, 2024-04-12T03:12:40.646 INFO:teuthology.orchestra.run.smithi149.stdout: "max_objects": -1 2024-04-12T03:12:40.646 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:40.646 INFO:teuthology.orchestra.run.smithi149.stdout: "user_quota": { 2024-04-12T03:12:40.646 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled": false, 2024-04-12T03:12:40.646 INFO:teuthology.orchestra.run.smithi149.stdout: "check_on_raw": false, 2024-04-12T03:12:40.647 INFO:teuthology.orchestra.run.smithi149.stdout: "max_size": -1, 2024-04-12T03:12:40.647 INFO:teuthology.orchestra.run.smithi149.stdout: "max_size_kb": 0, 2024-04-12T03:12:40.647 INFO:teuthology.orchestra.run.smithi149.stdout: "max_objects": -1 2024-04-12T03:12:40.647 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:40.647 INFO:teuthology.orchestra.run.smithi149.stdout: "user_ratelimit": { 2024-04-12T03:12:40.647 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_ops": 0, 2024-04-12T03:12:40.647 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_ops": 0, 2024-04-12T03:12:40.647 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_bytes": 0, 2024-04-12T03:12:40.647 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_bytes": 0, 2024-04-12T03:12:40.647 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled": false 2024-04-12T03:12:40.647 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:40.647 INFO:teuthology.orchestra.run.smithi149.stdout: "bucket_ratelimit": { 2024-04-12T03:12:40.647 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_ops": 0, 2024-04-12T03:12:40.647 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_ops": 0, 2024-04-12T03:12:40.647 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_bytes": 0, 2024-04-12T03:12:40.648 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_bytes": 0, 2024-04-12T03:12:40.648 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled": false 2024-04-12T03:12:40.648 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:40.648 INFO:teuthology.orchestra.run.smithi149.stdout: "anonymous_ratelimit": { 2024-04-12T03:12:40.648 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_ops": 0, 2024-04-12T03:12:40.648 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_ops": 0, 2024-04-12T03:12:40.648 INFO:teuthology.orchestra.run.smithi149.stdout: "max_read_bytes": 0, 2024-04-12T03:12:40.648 INFO:teuthology.orchestra.run.smithi149.stdout: "max_write_bytes": 0, 2024-04-12T03:12:40.648 INFO:teuthology.orchestra.run.smithi149.stdout: "enabled": false 2024-04-12T03:12:40.648 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:12:40.648 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:12:40.649 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:12:40.649 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_name": "test-realm", 2024-04-12T03:12:40.649 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:12:40.649 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-04-12T03:12:40.650 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2024-04-12T03:12:40.650 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T03:12:40.651 INFO:tasks.rgw.c2.client.1.smithi149.stdout:2024-04-12T03:12:40.649+0000 7f75763a2700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 45996) UID: 0 2024-04-12T03:12:40.651 INFO:tasks.rgw.c2.client.1.smithi149.stdout:2024-04-12T03:12:40.649+0000 7f757b5777c0 -1 shutting down 2024-04-12T03:12:44.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2024-04-12T03:12:46.752 INFO:tasks.rgw.c2.client.1:Stopped 2024-04-12T03:12:46.752 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-04-12T03:12:46.752 DEBUG:teuthology.orchestra.run.smithi149:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-04-12T03:12:46.755 INFO:tasks.rgw.c2.client.1:Started 2024-04-12T03:12:46.755 DEBUG:teuthology.orchestra.run.smithi149:> curl http://smithi149.front.sepia.ceph.com:8001 2024-04-12T03:12:46.780 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T03:12:46.780 INFO:teuthology.orchestra.run.smithi149.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:12:46.781 INFO:teuthology.orchestra.run.smithi149.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:12:46.781 INFO:teuthology.orchestra.run.smithi149.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi149.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T03:12:47.781 DEBUG:teuthology.orchestra.run.smithi149:> curl http://smithi149.front.sepia.ceph.com:8001 2024-04-12T03:12:47.804 INFO:teuthology.orchestra.run.smithi149.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:12:47.804 INFO:teuthology.orchestra.run.smithi149.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:12:47.804 INFO:teuthology.orchestra.run.smithi149.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T03:12:47.805 INFO:teuthology.orchestra.run.smithi149.stdout:anonymous 2024-04-12T03:12:47.805 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2024-04-12T03:12:47.806 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2024-04-12T03:12:47.817 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi 2024-04-12T03:12:47.850 INFO:tasks.rgw_multisite_tests:creating test user.. 2024-04-12T03:12:47.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:47.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:12:47.851 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:12:47.918 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:47.918 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:47.960 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:12:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "user_id": "rgw-multisite-test-user", 2024-04-12T03:12:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "display_name": "Multisite Test User", 2024-04-12T03:12:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "email": "", 2024-04-12T03:12:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "suspended": 0, 2024-04-12T03:12:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "max_buckets": 1000, 2024-04-12T03:12:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "subusers": [], 2024-04-12T03:12:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "keys": [ 2024-04-12T03:12:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "rgw-multisite-test-user", 2024-04-12T03:12:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "access_key": "KUXF4P9H15V1YOGZBQF6", 2024-04-12T03:12:47.960 INFO:teuthology.orchestra.run.smithi043.stdout: "secret_key": "hw4K7Q12aVU5MnIq1OTajIW68j2tPD9fwMSrWdk4" 2024-04-12T03:12:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:12:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: "swift_keys": [], 2024-04-12T03:12:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: "caps": [ 2024-04-12T03:12:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "roles", 2024-04-12T03:12:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: "perm": "*" 2024-04-12T03:12:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:12:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: "op_mask": "read, write, delete", 2024-04-12T03:12:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "", 2024-04-12T03:12:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: "default_storage_class": "", 2024-04-12T03:12:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_tags": [], 2024-04-12T03:12:47.961 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T03:12:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:12:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:12:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:12:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:12:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:12:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T03:12:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:12:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:12:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:12:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:12:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:12:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:47.962 INFO:teuthology.orchestra.run.smithi043.stdout: "temp_url_keys": [], 2024-04-12T03:12:47.963 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "rgw", 2024-04-12T03:12:47.963 INFO:teuthology.orchestra.run.smithi043.stdout: "mfa_ids": [] 2024-04-12T03:12:47.963 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:12:47.963 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:12:47.975 INFO:rgw_multi.tests:meta checkpoint 2024-04-12T03:12:47.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:47.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:47.975 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:12:48.028 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:48.028 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:48.060 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:12:48.073 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:12:48.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:48.074 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:12:48.074 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:12:48.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:48.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:48.074 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:12:48.074 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:12:48.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:48.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:48.074 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:12:48.074 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:12:48.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:48.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:48.075 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:12:48.075 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:12:48.075 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:48.075 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:12:48.076 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:12:48.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:48.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:48.076 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:12:48.130 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:48.130 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:48.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:12:48.161+0000 7efc9085a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:48.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:12:48.162+0000 7efc9085a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:48.176 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:12:48.176 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:12:48.176 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:12:48.176 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:12:48.176 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:12:48.176 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:12:48.177 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:48.177 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:48.177 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:12:48.177 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:48.177 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:12:48.177 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:12:48.177 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:12:48.177 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:12:48.177 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:12:48.177 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:12:48.177 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:12:48.177 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:12:48.178 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:48.178 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:48.178 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:48.178 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:48.178 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:12:48.178 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:12:48.178 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:12:48.178 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:12:48.178 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:12:48.178 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:12:48.178 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:12:48.178 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:12:48.178 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:48.179 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:48.179 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:48.179 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:48.179 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:12:48.179 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:12:48.179 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:12:48.179 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:12:48.179 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:12:48.179 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:12:48.179 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:12:48.179 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:12:48.179 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:48.179 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:48.179 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:48.179 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:48.180 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:12:48.180 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:12:48.180 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:12:48.180 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:12:48.180 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:12:48.180 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:12:48.180 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:12:48.180 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:12:48.180 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:48.180 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:48.180 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:48.180 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:12:48.180 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:48.180 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:12:48.180 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:12:48.181 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:12:48.181 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:48.181 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:12:48Z" 2024-04-12T03:12:48.181 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:12:48.182 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:12:49.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2024-04-12T03:12:53.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:53.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:53.183 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:12:53.242 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:53.242 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:53.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:12:53.274+0000 7f4594e1f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:53.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:12:53.276+0000 7f4594e1f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:53.290 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:12:53.290 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:12:53.290 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:12:53.290 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:12:53.290 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:12:53.290 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:12:53.290 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:53.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:53.291 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:12:53.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:53.291 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:12:53.291 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:12:53.291 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:12:53.291 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:12:53.291 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:12:53.291 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:12:53.291 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:12:53.291 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:12:53.291 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:53.291 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:53.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:53.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:53.292 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:12:53.292 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:12:53.292 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:12:53.292 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:12:53.292 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:12:53.292 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:12:53.292 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:12:53.292 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:12:53.292 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:53.292 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:53.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:53.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:53.292 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:12:53.293 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:12:53.293 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:12:53.293 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:12:53.293 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:12:53.293 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:12:53.293 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:12:53.293 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:12:53.293 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:53.293 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:53.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:53.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:53.293 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:12:53.293 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:12:53.293 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:12:53.294 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:12:53.294 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:12:53.294 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:12:53.294 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:12:53.294 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:12:53.294 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:53.294 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:53.294 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:53.294 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:12:53.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:53.294 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:12:53.294 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:12:53.294 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:12:53.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:53.295 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:12:53Z" 2024-04-12T03:12:53.295 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:12:53.296 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:12:58.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:58.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:12:58.297 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:12:58.356 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:12:58.356 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:12:58.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:12:58.390+0000 7fc8ed865780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:58.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:12:58.391+0000 7fc8ed865780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:12:58.404 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:12:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:12:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:12:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:12:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:12:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:12:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:12:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:12:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:12:58.405 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:12:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:12:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:12:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:12:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:12:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:12:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:12:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:12:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:12:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:12:58.406 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:12:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:12:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:12:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:12:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:12:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:12:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:12:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:12:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:12:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:12:58.407 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:12:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:12:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:12:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:12:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:12:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:12:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:12:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:12:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:12:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:12:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:12:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:12:58.408 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:12:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:12:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:12:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:12:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:12:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:12:58.409 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:12:58Z" 2024-04-12T03:12:58.409 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:12:58.410 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:13:03.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:03.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:03.411 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:03.467 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:03.467 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:03.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:03.499+0000 7fa99feca780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:03.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:03.500+0000 7fa99feca780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:03.515 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:13:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:13:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:13:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:13:03.515 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:13:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:13:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:13:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:13:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:13:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:03.516 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:13:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:03.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:13:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:03.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:13:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:13:03.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:13:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:13:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:13:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:03.520 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:13:03Z" 2024-04-12T03:13:03.520 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:13:03.521 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:13:08.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:08.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:08.522 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:08.582 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:08.582 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:08.614+0000 7f86d646a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:08.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:08.615+0000 7f86d646a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:08.630 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:13:08.630 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:13:08.630 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:13:08.630 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:13:08.630 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:13:08.630 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:13:08.631 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:08.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:08.631 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:13:08.631 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:08.631 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:13:08.631 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:08.631 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:08.631 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:08.631 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:08.631 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:13:08.631 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:08.631 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:08.631 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:08.632 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:08.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:08.632 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:08.632 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:13:08.632 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:08.632 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:08.632 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:08.632 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:08.632 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:08.632 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:08.632 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:08.632 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:08.632 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:08.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:08.633 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:08.633 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:13:08.633 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:08.633 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:08.633 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:08.633 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:08.633 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:08.633 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:08.633 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:08.634 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:08.634 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:08.634 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:08.634 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:08.634 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:13:08.634 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:08.634 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:08.634 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:08.634 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:08.634 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:08.634 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:08.635 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:08.635 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:08.635 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:08.635 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:08.635 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:13:08.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:08.635 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:13:08.635 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:13:08.635 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:13:08.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:08.635 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:13:08Z" 2024-04-12T03:13:08.635 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:13:08.637 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:13:13.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:13.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:13.642 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:13.703 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:13.703 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:13.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:13.735+0000 7fd5df32e780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:13.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:13.736+0000 7fd5df32e780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:13.751 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:13:13.751 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:13:13.751 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:13:13.751 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:13:13.751 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:13:13.751 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:13:13.751 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:13.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:13.752 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:13:13.752 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:13.752 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:13:13.752 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:13.752 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:13.752 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:13.752 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:13.752 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:13:13.752 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:13.752 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:13.752 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:13.752 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:13.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:13.752 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:13.753 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:13:13.753 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:13.753 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:13.753 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:13.753 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:13.753 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:13.753 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:13.753 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:13.753 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:13.753 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:13.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:13.753 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:13.753 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:13:13.754 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:13.754 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:13.754 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:13.754 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:13.754 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:13.754 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:13.754 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:13.754 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:13.754 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:13.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:13.754 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:13.754 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:13:13.754 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:13.754 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:13.755 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:13.755 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:13.755 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:13.755 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:13.755 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:13.755 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:13.755 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:13.755 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:13.755 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:13:13.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:13.755 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:13:13.755 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:13:13.755 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:13:13.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:13.755 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:13:13Z" 2024-04-12T03:13:13.756 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:13:13.756 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:13:18.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:18.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:18.759 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:18.818 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:18.818 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:18.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:18.850+0000 7f8a20aad780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:18.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:18.851+0000 7f8a20aad780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:18.865 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:13:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:13:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:13:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:13:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:13:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:13:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:13:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:13:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:13:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:13:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:18.868 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:18.868 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:18.868 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:18.868 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:18.868 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:18.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:18.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:18.868 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:13:18.868 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:18.868 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:18.868 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:18.868 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:18.868 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:18.868 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:18.869 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:18.869 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:18.869 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:18.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:18.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:18.869 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:13:18.869 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:18.869 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:18.869 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:18.869 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:18.869 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:18.869 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:18.869 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:18.869 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:18.870 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:18.870 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:18.870 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:13:18.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:18.870 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:13:18.870 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:13:18.870 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:13:18.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:18.870 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:13:18Z" 2024-04-12T03:13:18.870 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:13:18.871 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:13:23.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:23.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:23.873 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:23.933 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:23.933 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:23.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:23.965+0000 7fa4ca736780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:23.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:23.966+0000 7fa4ca736780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:23.981 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:13:23.981 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:13:23.981 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:13:23.981 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:13:23.981 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:13:23.981 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:13:23.981 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:23.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:23.981 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:13:23.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:23.982 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:13:23.982 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:23.982 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:23.982 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:23.982 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:23.982 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:13:23.982 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:23.982 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:23.982 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:23.982 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:23.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:23.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:23.982 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:13:23.983 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:23.983 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:23.983 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:23.983 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:23.983 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:23.983 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:23.983 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:23.983 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:23.983 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:23.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:23.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:23.983 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:13:23.983 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:23.983 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:23.984 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:23.984 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:23.984 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:23.984 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:23.984 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:23.984 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:23.984 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:23.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:23.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:23.984 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:13:23.984 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:23.984 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:23.984 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:23.984 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:23.985 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:23.985 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:23.985 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:23.985 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:23.985 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:23.985 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:23.985 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:13:23.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:23.985 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:13:23.985 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:13:23.985 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:13:23.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:23.986 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:13:23Z" 2024-04-12T03:13:23.986 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:13:23.987 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:13:28.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:28.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:28.988 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:29.047 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:29.047 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:29.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:29.080+0000 7f9543cbe780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:29.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:29.081+0000 7f9543cbe780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:29.096 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:13:29.097 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:13:29.097 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:13:29.097 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:13:29.097 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:13:29.097 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:13:29.097 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:29.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:29.097 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:13:29.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:29.097 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:13:29.097 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:29.098 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:29.098 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:29.098 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:29.098 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:13:29.098 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:29.098 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:29.098 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:29.098 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:29.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:29.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:29.098 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:13:29.098 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:29.098 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:29.099 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:29.099 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:29.099 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:29.099 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:29.099 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:29.099 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:29.099 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:29.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:29.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:29.099 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:13:29.099 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:29.099 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:29.099 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:29.100 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:29.100 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:29.100 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:29.100 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:29.100 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:29.100 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:29.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:29.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:29.100 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:13:29.100 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:29.100 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:29.100 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:29.100 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:29.100 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:29.101 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:29.101 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:29.101 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:29.101 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:29.101 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:29.101 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:13:29.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:29.101 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:13:29.101 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:13:29.101 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:13:29.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:29.101 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:13:29Z" 2024-04-12T03:13:29.101 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:13:29.102 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:13:34.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:34.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:34.103 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:34.164 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:34.164 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:34.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:34.197+0000 7fd2f66e7780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:34.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:34.198+0000 7fd2f66e7780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:34.212 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:13:34.212 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:13:34.213 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:13:34.213 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:13:34.213 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:13:34.213 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:13:34.213 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:34.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:34.213 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:13:34.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:34.213 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:13:34.213 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:34.213 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:34.213 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:34.213 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:34.213 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:13:34.214 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:34.214 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:34.214 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:34.214 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:34.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:34.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:34.214 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:13:34.214 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:34.214 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:34.214 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:34.214 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:34.214 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:34.214 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:34.215 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:34.215 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:34.215 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:34.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:34.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:34.216 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:13:34.216 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:34.216 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:34.216 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:34.216 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:34.216 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:34.216 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:34.216 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:34.216 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:34.216 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:34.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:34.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:13:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:13:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:13:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:13:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:13:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:13:34Z" 2024-04-12T03:13:34.218 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:13:34.219 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:13:39.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:39.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:39.221 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:39.280 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:39.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:39.312+0000 7f391c9f8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:39.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:39.314+0000 7f391c9f8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:39.328 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:13:39.328 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:13:39.328 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:13:39.328 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:13:39.328 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:13:39.328 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:13:39.328 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:39.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:39.329 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:13:39.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:39.329 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:13:39.329 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:39.329 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:39.329 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:39.329 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:39.329 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:13:39.329 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:39.329 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:39.329 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:39.329 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:39.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:39.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:39.330 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:13:39.330 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:39.330 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:39.330 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:39.330 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:39.330 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:39.330 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:39.330 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:39.330 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:39.330 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:39.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:39.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:39.331 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:13:39.331 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:39.331 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:39.331 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:39.331 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:39.331 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:39.331 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:39.331 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:39.331 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:39.331 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:39.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:39.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:39.331 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:13:39.332 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:39.332 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:39.332 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:39.332 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:39.332 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:39.332 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:39.332 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:39.332 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:39.332 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:39.332 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:39.332 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:13:39.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:39.332 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:13:39.332 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:13:39.333 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:13:39.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:39.333 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:13:39Z" 2024-04-12T03:13:39.333 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:13:39.334 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:13:44.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:44.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:44.335 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:44.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:44.427+0000 7fae8d5c9780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:44.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:44.428+0000 7fae8d5c9780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:44.443 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:13:44.443 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:13:44.443 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:13:44.443 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:13:44.443 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:13:44.443 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:13:44.444 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:44.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:44.444 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:13:44.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:44.444 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:13:44.444 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:44.444 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:44.444 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:44.444 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:44.444 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:13:44.444 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:44.444 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:44.444 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:44.445 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:44.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:44.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:44.445 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:13:44.445 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:44.445 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:44.445 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:44.445 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:44.445 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:44.445 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:44.445 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:44.445 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:44.445 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:44.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:44.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:44.446 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:13:44.446 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:44.446 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:44.446 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:44.446 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:44.446 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:44.446 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:44.446 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:44.446 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:44.446 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:44.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:44.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:44.447 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:13:44.447 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:44.447 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:44.447 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:44.447 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:44.447 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:44.447 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:44.447 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:44.447 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:44.447 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:44.447 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:44.447 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:13:44.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:44.447 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:13:44.448 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:13:44.448 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:13:44.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:44.448 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:13:44Z" 2024-04-12T03:13:44.448 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:13:44.449 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:13:49.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:49.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:49.450 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:49.510 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:49.510 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:49.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:49.542+0000 7f757d945780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:49.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:49.543+0000 7f757d945780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:49.557 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:13:49.558 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:13:49.558 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:13:49.558 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:13:49.558 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:13:49.558 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:13:49.558 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:49.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:49.558 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:13:49.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:49.558 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:13:49.558 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:49.558 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:49.558 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:49.559 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:49.559 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:13:49.559 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:49.559 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:49.559 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:49.559 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:49.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:49.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:49.559 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:13:49.559 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:49.559 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:49.559 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:49.559 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:49.559 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:49.560 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:49.560 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:49.560 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:49.560 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:49.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:49.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:49.560 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:13:49.560 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:49.560 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:49.560 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:49.560 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:49.560 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:49.560 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:49.560 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:49.560 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:49.561 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:49.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:49.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:49.561 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:13:49.561 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:49.561 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:49.561 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:49.561 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:49.561 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:49.561 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:49.561 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:49.561 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:49.561 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:49.561 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:49.562 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:13:49.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:49.562 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:13:49.562 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:13:49.562 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:13:49.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:49.562 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:13:49Z" 2024-04-12T03:13:49.562 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:13:49.563 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:13:54.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:54.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:54.564 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:54.625 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:54.625 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:54.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:54.659+0000 7fe3c2119780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:54.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:54.660+0000 7fe3c2119780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:13:54.675 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:54.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:13:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:13:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:13:54.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:54.680 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:13:54Z" 2024-04-12T03:13:54.680 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:13:54.681 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:13:59.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:59.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:13:59.682 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:13:59.741 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:13:59.742 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:13:59.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:59.773+0000 7f9bab67c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:59.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:13:59.774+0000 7f9bab67c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:13:59.788 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:13:59.789 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:13:59.789 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:13:59.789 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:13:59.789 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:13:59.789 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:13:59.789 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:59.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:59.789 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:13:59.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:59.790 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:13:59.790 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:59.790 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:59.790 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:59.790 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:59.790 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:13:59.790 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:59.790 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:59.790 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:59.790 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:59.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:59.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:59.791 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:13:59.791 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:59.791 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:59.791 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:59.791 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:59.791 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:59.791 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:59.791 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:59.791 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:59.791 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:59.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:59.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:59.791 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:13:59.791 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:59.792 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:59.792 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:59.792 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:59.792 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:59.792 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:59.792 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:59.792 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:59.792 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:59.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:59.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:13:59.792 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:13:59.792 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:13:59.792 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:13:59.792 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:13:59.793 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:13:59.793 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:13:59.793 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:13:59.793 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:13:59.793 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:13:59.793 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:59.793 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:13:59.793 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:13:59.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:59.793 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:13:59.793 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:13:59.794 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:13:59.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:13:59.794 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:13:59Z" 2024-04-12T03:13:59.795 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:13:59.796 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:14:04.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:04.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:04.798 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:04.858 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:04.858 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:04.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:04.890+0000 7fc6aaaf0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:04.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:04.891+0000 7fc6aaaf0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:04.906 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:14:04.906 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:14:04.906 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:14:04.906 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:14:04.906 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:14:04.906 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:04.906 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:04.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:04.906 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:14:04.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:04.906 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:14:04.907 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:04.907 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:04.907 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:04.907 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:04.907 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:14:04.907 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:04.907 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:04.907 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:04.907 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:04.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:04.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:04.907 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:14:04.907 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:04.908 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:04.908 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:04.908 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:04.908 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:04.908 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:04.908 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:04.908 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:04.908 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:04.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:04.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:04.908 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:14:04.908 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:04.908 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:04.908 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:04.909 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:04.909 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:04.909 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:04.909 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:04.909 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:04.909 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:04.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:04.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:04.909 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:14:04.909 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:04.909 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:04.909 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:04.909 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:04.909 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:04.910 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:04.910 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:04.910 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:04.910 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:04.910 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:04.910 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:14:04.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:04.910 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:14:04.910 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:14:04.910 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:14:04.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:04.910 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:14:04Z" 2024-04-12T03:14:04.911 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:14:04.911 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:14:09.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:09.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:09.913 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:09.972 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:09.972 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:10.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:10.008+0000 7f33e8412780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:10.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:10.009+0000 7f33e8412780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:10.022 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:14:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:14:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:14:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:14:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:14:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:14:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:14:10.023 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:14:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:14:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:10.024 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:14:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:10.025 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:14:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:10.026 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:14:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:14:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:14:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:14:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:10.027 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:14:10Z" 2024-04-12T03:14:10.027 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:14:10.028 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:14:15.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:15.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:15.031 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:15.093 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:15.093 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:15.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:15.133+0000 7f88d53df780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:15.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:15.134+0000 7f88d53df780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:15.148 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:14:15.149 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:14:15.149 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:14:15.149 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:14:15.149 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:14:15.149 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:15.149 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:15.149 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:15.149 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:14:15.149 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:15.149 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:14:15.149 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:15.150 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:15.150 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:15.150 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:15.150 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:14:15.150 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:15.150 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:15.150 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:15.150 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:15.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:15.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:15.150 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:14:15.150 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:15.151 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:15.151 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:15.151 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:15.151 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:15.151 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:15.151 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:15.151 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:15.151 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:15.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:15.151 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:15.151 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:14:15.151 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:15.151 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:15.151 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:15.151 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:15.152 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:15.152 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:15.152 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:15.152 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:15.152 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:15.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:15.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:15.152 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:14:15.152 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:15.152 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:15.152 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:15.152 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:15.152 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:15.152 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:15.152 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:15.153 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:15.153 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:15.153 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:15.153 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:14:15.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:15.153 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:14:15.153 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:14:15.153 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:14:15.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:15.153 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:14:15Z" 2024-04-12T03:14:15.153 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:14:15.154 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:14:20.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:20.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:20.156 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:20.216 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:20.216 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:20.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:20.248+0000 7f1d83d13780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:20.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:20.249+0000 7f1d83d13780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:20.264 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:14:20.264 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:14:20.264 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:14:20.264 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:14:20.264 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:14:20.264 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:20.264 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:20.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:20.264 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:14:20.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:20.265 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:14:20.265 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:20.265 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:20.265 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:20.265 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:20.265 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:14:20.265 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:20.265 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:20.265 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:20.265 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:20.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:20.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:20.265 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:14:20.265 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:20.265 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:20.266 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:20.266 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:20.266 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:20.266 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:20.266 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:20.266 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:20.266 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:20.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:20.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:20.266 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:14:20.266 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:20.266 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:20.266 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:20.266 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:20.267 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:20.267 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:20.267 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:20.267 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:20.267 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:20.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:20.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:20.267 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:14:20.267 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:20.267 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:20.267 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:20.267 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:20.267 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:20.268 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:20.268 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:20.268 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:20.268 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:20.268 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:20.268 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:14:20.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:20.268 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:14:20.268 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:14:20.268 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:14:20.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:20.268 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:14:20Z" 2024-04-12T03:14:20.268 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:14:20.269 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:14:25.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:25.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:25.270 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:25.331 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:25.331 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:25.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:25.364+0000 7fe586587780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:25.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:25.365+0000 7fe586587780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:25.380 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:14:25.380 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:14:25.380 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:14:25.380 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:14:25.380 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:14:25.381 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:25.381 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:25.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:25.381 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:14:25.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:25.381 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:14:25.381 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:25.381 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:25.381 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:25.382 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:25.382 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:14:25.382 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:25.382 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:25.382 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:25.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:25.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:25.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:25.383 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:14:25.383 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:25.383 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:25.383 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:25.383 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:25.383 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:25.383 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:25.383 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:25.384 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:25.384 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:25.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:25.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:25.384 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:14:25.384 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:25.384 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:25.384 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:25.385 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:25.385 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:25.385 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:25.385 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:25.385 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:25.385 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:25.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:25.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:25.386 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:14:25.386 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:25.386 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:25.386 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:25.386 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:25.386 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:25.387 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:25.387 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:25.387 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:25.387 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:25.387 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:25.387 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:14:25.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:25.388 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:14:25.388 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:14:25.388 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:14:25.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:25.388 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:14:25Z" 2024-04-12T03:14:25.388 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:14:25.389 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:14:30.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:30.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:30.391 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:30.450 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:30.450 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:30.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:30.481+0000 7efe46112780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:30.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:30.483+0000 7efe46112780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:30.497 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:14:30.497 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:14:30.497 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:14:30.497 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:14:30.497 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:14:30.498 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:30.498 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:30.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:30.498 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:14:30.498 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:30.498 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:14:30.498 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:30.498 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:30.498 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:30.498 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:30.498 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:14:30.498 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:30.498 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:30.499 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:30.499 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:30.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:30.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:30.499 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:14:30.499 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:30.499 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:30.499 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:30.499 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:30.499 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:30.499 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:30.499 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:30.499 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:30.499 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:30.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:30.500 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:30.500 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:14:30.500 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:30.500 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:30.500 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:30.500 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:30.500 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:30.500 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:30.500 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:30.500 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:30.500 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:30.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:30.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:30.501 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:14:30.501 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:30.501 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:30.501 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:30.501 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:30.501 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:30.501 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:30.501 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:30.501 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:30.502 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:30.502 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:30.502 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:14:30.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:30.502 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:14:30.502 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:14:30.502 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:14:30.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:30.502 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:14:30Z" 2024-04-12T03:14:30.502 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:14:30.503 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:14:35.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:35.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:35.505 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:35.565 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:35.565 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:35.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:35.596+0000 7f0ab7a31780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:35.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:35.597+0000 7f0ab7a31780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:35.612 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:14:35.612 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:14:35.613 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:14:35.613 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:14:35.613 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:14:35.613 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:35.613 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:35.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:35.614 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:14:35.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:35.614 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:14:35.614 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:35.614 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:35.615 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:35.615 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:35.615 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:14:35.615 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:35.615 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:35.616 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:35.616 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:35.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:35.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:35.616 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:14:35.616 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:35.617 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:35.617 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:35.617 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:35.617 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:35.617 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:35.618 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:35.618 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:35.618 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:35.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:35.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:35.618 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:14:35.619 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:35.619 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:35.619 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:35.619 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:35.619 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:35.619 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:35.620 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:35.620 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:35.620 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:35.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:35.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:35.621 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:14:35.621 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:35.621 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:35.621 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:35.621 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:35.621 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:35.622 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:35.622 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:35.622 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:35.622 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:35.622 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:35.622 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:14:35.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:35.623 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:14:35.623 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:14:35.623 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:14:35.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:35.623 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:14:35Z" 2024-04-12T03:14:35.624 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:14:35.625 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:14:40.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:40.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:40.627 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:40.687 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:40.688 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:40.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:40.720+0000 7f2894434780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:40.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:40.721+0000 7f2894434780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:40.736 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:14:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:14:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:14:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:14:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:14:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:14:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:14:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:14:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:14:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:14:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:14:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:14:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:14:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:14:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:14:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:14:40Z" 2024-04-12T03:14:40.740 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:14:40.741 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1712891567.956902_97.1 target= 2024-04-12T03:14:45.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:45.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:45.742 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:45.802 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:45.803 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:45.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:45.836+0000 7f6d29b65780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:45.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:45.838+0000 7f6d29b65780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:45.853 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:14:45.853 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:14:45.853 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:14:45.853 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:14:45.853 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:14:45.853 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:45.853 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:45.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:45.853 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:14:45.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:45.854 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:14:45.854 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:45.854 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:45.854 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:45.854 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:45.854 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:14:45.854 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:45.854 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:14:45.854 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:45.854 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:45.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:45.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:45.855 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:14:45.855 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:45.855 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:45.855 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:45.855 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:45.855 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:45.855 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:45.855 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:45.855 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:45.855 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:45.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:45.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:45.856 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:14:45.856 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:45.856 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:45.856 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:45.856 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:45.856 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:45.856 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:45.856 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:45.856 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:45.856 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:45.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:45.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:45.856 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:14:45.857 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:45.857 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:45.857 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:45.857 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:45.857 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:45.857 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:45.857 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:45.857 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:45.857 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:45.857 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:45.857 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:14:45.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:45.857 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:14:45.857 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:14:45.858 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:14:45.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:45.858 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:14:45Z" 2024-04-12T03:14:45.858 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:14:45.859 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:14:45.859 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2024-04-12T03:14:45.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:45.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:45.859 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:45.917 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:45.917 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:45.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:14:45.951+0000 7ff94aa24780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:45.952 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:14:45.952+0000 7ff94aa24780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:45.964 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-04-12T03:14:45.965 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_status": { 2024-04-12T03:14:45.965 INFO:teuthology.orchestra.run.smithi149.stdout: "info": { 2024-04-12T03:14:45.965 INFO:teuthology.orchestra.run.smithi149.stdout: "status": "sync", 2024-04-12T03:14:45.966 INFO:teuthology.orchestra.run.smithi149.stdout: "num_shards": 4, 2024-04-12T03:14:45.966 INFO:teuthology.orchestra.run.smithi149.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:45.966 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:45.966 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:45.966 INFO:teuthology.orchestra.run.smithi149.stdout: "markers": [ 2024-04-12T03:14:45.966 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:14:45.966 INFO:teuthology.orchestra.run.smithi149.stdout: "key": 0, 2024-04-12T03:14:45.966 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:14:45.966 INFO:teuthology.orchestra.run.smithi149.stdout: "state": 1, 2024-04-12T03:14:45.966 INFO:teuthology.orchestra.run.smithi149.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:45.966 INFO:teuthology.orchestra.run.smithi149.stdout: "next_step_marker": "", 2024-04-12T03:14:45.967 INFO:teuthology.orchestra.run.smithi149.stdout: "total_entries": 1, 2024-04-12T03:14:45.967 INFO:teuthology.orchestra.run.smithi149.stdout: "pos": 0, 2024-04-12T03:14:45.967 INFO:teuthology.orchestra.run.smithi149.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:14:45.967 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:45.967 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:45.967 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:45.967 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:14:45.968 INFO:teuthology.orchestra.run.smithi149.stdout: "key": 1, 2024-04-12T03:14:45.968 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:14:45.968 INFO:teuthology.orchestra.run.smithi149.stdout: "state": 1, 2024-04-12T03:14:45.968 INFO:teuthology.orchestra.run.smithi149.stdout: "marker": "", 2024-04-12T03:14:45.968 INFO:teuthology.orchestra.run.smithi149.stdout: "next_step_marker": "", 2024-04-12T03:14:45.968 INFO:teuthology.orchestra.run.smithi149.stdout: "total_entries": 0, 2024-04-12T03:14:45.968 INFO:teuthology.orchestra.run.smithi149.stdout: "pos": 0, 2024-04-12T03:14:45.968 INFO:teuthology.orchestra.run.smithi149.stdout: "timestamp": "0.000000", 2024-04-12T03:14:45.968 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:45.968 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:45.968 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:45.968 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:14:45.968 INFO:teuthology.orchestra.run.smithi149.stdout: "key": 2, 2024-04-12T03:14:45.969 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:14:45.969 INFO:teuthology.orchestra.run.smithi149.stdout: "state": 1, 2024-04-12T03:14:45.969 INFO:teuthology.orchestra.run.smithi149.stdout: "marker": "", 2024-04-12T03:14:45.969 INFO:teuthology.orchestra.run.smithi149.stdout: "next_step_marker": "", 2024-04-12T03:14:45.969 INFO:teuthology.orchestra.run.smithi149.stdout: "total_entries": 0, 2024-04-12T03:14:45.969 INFO:teuthology.orchestra.run.smithi149.stdout: "pos": 0, 2024-04-12T03:14:45.969 INFO:teuthology.orchestra.run.smithi149.stdout: "timestamp": "0.000000", 2024-04-12T03:14:45.969 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:45.969 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:45.970 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:45.970 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:14:45.970 INFO:teuthology.orchestra.run.smithi149.stdout: "key": 3, 2024-04-12T03:14:45.970 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:14:45.970 INFO:teuthology.orchestra.run.smithi149.stdout: "state": 1, 2024-04-12T03:14:45.970 INFO:teuthology.orchestra.run.smithi149.stdout: "marker": "", 2024-04-12T03:14:45.970 INFO:teuthology.orchestra.run.smithi149.stdout: "next_step_marker": "", 2024-04-12T03:14:45.970 INFO:teuthology.orchestra.run.smithi149.stdout: "total_entries": 0, 2024-04-12T03:14:45.970 INFO:teuthology.orchestra.run.smithi149.stdout: "pos": 0, 2024-04-12T03:14:45.970 INFO:teuthology.orchestra.run.smithi149.stdout: "timestamp": "0.000000", 2024-04-12T03:14:45.970 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:45.970 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:45.970 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:45.971 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:14:45.971 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:45.971 INFO:teuthology.orchestra.run.smithi149.stdout: "full_sync": { 2024-04-12T03:14:45.971 INFO:teuthology.orchestra.run.smithi149.stdout: "total": 1, 2024-04-12T03:14:45.971 INFO:teuthology.orchestra.run.smithi149.stdout: "complete": 1 2024-04-12T03:14:45.971 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:45.971 INFO:teuthology.orchestra.run.smithi149.stdout: "current_time": "2024-04-12T03:14:45Z" 2024-04-12T03:14:45.971 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-04-12T03:14:45.972 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2024-04-12T03:14:45.972 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2024-04-12T03:14:45.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:45.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:45.973 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:46.022 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:46.023 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:46.050 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:14:46.050+0000 7fe0738b1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:46.051 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:14:46.051+0000 7fe0738b1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:46.062 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-04-12T03:14:46.062 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_status": { 2024-04-12T03:14:46.062 INFO:teuthology.orchestra.run.smithi149.stdout: "info": { 2024-04-12T03:14:46.063 INFO:teuthology.orchestra.run.smithi149.stdout: "status": "sync", 2024-04-12T03:14:46.063 INFO:teuthology.orchestra.run.smithi149.stdout: "num_shards": 4, 2024-04-12T03:14:46.063 INFO:teuthology.orchestra.run.smithi149.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:46.063 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:46.063 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.063 INFO:teuthology.orchestra.run.smithi149.stdout: "markers": [ 2024-04-12T03:14:46.063 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:14:46.063 INFO:teuthology.orchestra.run.smithi149.stdout: "key": 0, 2024-04-12T03:14:46.063 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:14:46.063 INFO:teuthology.orchestra.run.smithi149.stdout: "state": 1, 2024-04-12T03:14:46.063 INFO:teuthology.orchestra.run.smithi149.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:46.063 INFO:teuthology.orchestra.run.smithi149.stdout: "next_step_marker": "", 2024-04-12T03:14:46.063 INFO:teuthology.orchestra.run.smithi149.stdout: "total_entries": 1, 2024-04-12T03:14:46.064 INFO:teuthology.orchestra.run.smithi149.stdout: "pos": 0, 2024-04-12T03:14:46.064 INFO:teuthology.orchestra.run.smithi149.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:14:46.064 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:46.064 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:46.064 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.064 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:14:46.064 INFO:teuthology.orchestra.run.smithi149.stdout: "key": 1, 2024-04-12T03:14:46.064 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:14:46.064 INFO:teuthology.orchestra.run.smithi149.stdout: "state": 1, 2024-04-12T03:14:46.064 INFO:teuthology.orchestra.run.smithi149.stdout: "marker": "", 2024-04-12T03:14:46.065 INFO:teuthology.orchestra.run.smithi149.stdout: "next_step_marker": "", 2024-04-12T03:14:46.065 INFO:teuthology.orchestra.run.smithi149.stdout: "total_entries": 0, 2024-04-12T03:14:46.065 INFO:teuthology.orchestra.run.smithi149.stdout: "pos": 0, 2024-04-12T03:14:46.065 INFO:teuthology.orchestra.run.smithi149.stdout: "timestamp": "0.000000", 2024-04-12T03:14:46.065 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:46.065 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:46.065 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.065 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:14:46.065 INFO:teuthology.orchestra.run.smithi149.stdout: "key": 2, 2024-04-12T03:14:46.065 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:14:46.065 INFO:teuthology.orchestra.run.smithi149.stdout: "state": 1, 2024-04-12T03:14:46.065 INFO:teuthology.orchestra.run.smithi149.stdout: "marker": "", 2024-04-12T03:14:46.065 INFO:teuthology.orchestra.run.smithi149.stdout: "next_step_marker": "", 2024-04-12T03:14:46.066 INFO:teuthology.orchestra.run.smithi149.stdout: "total_entries": 0, 2024-04-12T03:14:46.066 INFO:teuthology.orchestra.run.smithi149.stdout: "pos": 0, 2024-04-12T03:14:46.066 INFO:teuthology.orchestra.run.smithi149.stdout: "timestamp": "0.000000", 2024-04-12T03:14:46.066 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:46.066 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:46.066 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.066 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:14:46.066 INFO:teuthology.orchestra.run.smithi149.stdout: "key": 3, 2024-04-12T03:14:46.066 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:14:46.066 INFO:teuthology.orchestra.run.smithi149.stdout: "state": 1, 2024-04-12T03:14:46.066 INFO:teuthology.orchestra.run.smithi149.stdout: "marker": "", 2024-04-12T03:14:46.066 INFO:teuthology.orchestra.run.smithi149.stdout: "next_step_marker": "", 2024-04-12T03:14:46.066 INFO:teuthology.orchestra.run.smithi149.stdout: "total_entries": 0, 2024-04-12T03:14:46.067 INFO:teuthology.orchestra.run.smithi149.stdout: "pos": 0, 2024-04-12T03:14:46.067 INFO:teuthology.orchestra.run.smithi149.stdout: "timestamp": "0.000000", 2024-04-12T03:14:46.067 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:46.067 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:46.067 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:46.067 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:14:46.067 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.067 INFO:teuthology.orchestra.run.smithi149.stdout: "full_sync": { 2024-04-12T03:14:46.067 INFO:teuthology.orchestra.run.smithi149.stdout: "total": 1, 2024-04-12T03:14:46.067 INFO:teuthology.orchestra.run.smithi149.stdout: "complete": 1 2024-04-12T03:14:46.067 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.067 INFO:teuthology.orchestra.run.smithi149.stdout: "current_time": "2024-04-12T03:14:46Z" 2024-04-12T03:14:46.067 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-04-12T03:14:46.068 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2024-04-12T03:14:46.068 INFO:rgw_multi.tests:meta checkpoint 2024-04-12T03:14:46.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:46.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:46.069 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:46.127 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:46.127 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:46.162 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:14:46.176 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:14:46.176 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:46.176 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:46.176 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:14:46.176 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:46.177 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:46.177 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:46.177 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:14:46.177 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:46.177 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:46.177 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:46.177 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:14:46.177 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:46.177 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:46.177 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:46.178 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:14:46.178 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:46.178 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:14:46.179 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:14:46.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:46.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:46.179 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:46.235 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:46.235 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:46.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:46.266+0000 7f39cef24780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:46.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:46.268+0000 7f39cef24780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:46.282 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:14:46.282 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:14:46.282 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:14:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:14:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:14:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:14:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:14:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:46.283 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:14:46.284 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:46.284 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:14:46.284 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:46.284 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:46.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:46.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:46.284 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:14:46.284 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:46.284 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:46.284 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:46.284 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:46.284 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:46.284 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:46.285 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:46.285 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:46.285 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:46.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:46.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:46.285 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:14:46.285 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:46.285 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:46.285 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:46.285 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:46.285 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:46.285 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:46.285 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:46.285 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:46.286 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:46.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:46.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:46.286 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:14:46.286 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:46.286 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:46.286 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:46.286 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:46.286 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:46.286 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:46.286 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:46.286 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:46.286 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:46.287 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:46.287 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:14:46.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:46.287 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:14:46.287 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:14:46.287 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:14:46.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:46.287 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:14:46Z" 2024-04-12T03:14:46.287 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:14:46.288 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:14:46.288 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2024-04-12T03:14:46.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:46.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:46.289 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:46.337 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:46.337 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:46.367 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:14:46.367+0000 7fc9ce649780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:46.368 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:14:46.368+0000 7fc9ce649780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:46.379 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-04-12T03:14:46.379 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_status": { 2024-04-12T03:14:46.379 INFO:teuthology.orchestra.run.smithi149.stdout: "info": { 2024-04-12T03:14:46.379 INFO:teuthology.orchestra.run.smithi149.stdout: "status": "sync", 2024-04-12T03:14:46.380 INFO:teuthology.orchestra.run.smithi149.stdout: "num_shards": 4, 2024-04-12T03:14:46.380 INFO:teuthology.orchestra.run.smithi149.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:46.380 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:46.380 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.380 INFO:teuthology.orchestra.run.smithi149.stdout: "markers": [ 2024-04-12T03:14:46.380 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:14:46.380 INFO:teuthology.orchestra.run.smithi149.stdout: "key": 0, 2024-04-12T03:14:46.380 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:14:46.380 INFO:teuthology.orchestra.run.smithi149.stdout: "state": 1, 2024-04-12T03:14:46.380 INFO:teuthology.orchestra.run.smithi149.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:46.380 INFO:teuthology.orchestra.run.smithi149.stdout: "next_step_marker": "", 2024-04-12T03:14:46.380 INFO:teuthology.orchestra.run.smithi149.stdout: "total_entries": 1, 2024-04-12T03:14:46.380 INFO:teuthology.orchestra.run.smithi149.stdout: "pos": 0, 2024-04-12T03:14:46.380 INFO:teuthology.orchestra.run.smithi149.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:14:46.381 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:46.381 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:46.381 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.381 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:14:46.381 INFO:teuthology.orchestra.run.smithi149.stdout: "key": 1, 2024-04-12T03:14:46.381 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:14:46.381 INFO:teuthology.orchestra.run.smithi149.stdout: "state": 1, 2024-04-12T03:14:46.381 INFO:teuthology.orchestra.run.smithi149.stdout: "marker": "", 2024-04-12T03:14:46.381 INFO:teuthology.orchestra.run.smithi149.stdout: "next_step_marker": "", 2024-04-12T03:14:46.381 INFO:teuthology.orchestra.run.smithi149.stdout: "total_entries": 0, 2024-04-12T03:14:46.381 INFO:teuthology.orchestra.run.smithi149.stdout: "pos": 0, 2024-04-12T03:14:46.381 INFO:teuthology.orchestra.run.smithi149.stdout: "timestamp": "0.000000", 2024-04-12T03:14:46.381 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:46.381 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:46.382 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.382 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:14:46.382 INFO:teuthology.orchestra.run.smithi149.stdout: "key": 2, 2024-04-12T03:14:46.382 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:14:46.382 INFO:teuthology.orchestra.run.smithi149.stdout: "state": 1, 2024-04-12T03:14:46.382 INFO:teuthology.orchestra.run.smithi149.stdout: "marker": "", 2024-04-12T03:14:46.382 INFO:teuthology.orchestra.run.smithi149.stdout: "next_step_marker": "", 2024-04-12T03:14:46.382 INFO:teuthology.orchestra.run.smithi149.stdout: "total_entries": 0, 2024-04-12T03:14:46.382 INFO:teuthology.orchestra.run.smithi149.stdout: "pos": 0, 2024-04-12T03:14:46.382 INFO:teuthology.orchestra.run.smithi149.stdout: "timestamp": "0.000000", 2024-04-12T03:14:46.382 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:46.382 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:46.382 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.383 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:14:46.383 INFO:teuthology.orchestra.run.smithi149.stdout: "key": 3, 2024-04-12T03:14:46.383 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:14:46.383 INFO:teuthology.orchestra.run.smithi149.stdout: "state": 1, 2024-04-12T03:14:46.383 INFO:teuthology.orchestra.run.smithi149.stdout: "marker": "", 2024-04-12T03:14:46.383 INFO:teuthology.orchestra.run.smithi149.stdout: "next_step_marker": "", 2024-04-12T03:14:46.383 INFO:teuthology.orchestra.run.smithi149.stdout: "total_entries": 0, 2024-04-12T03:14:46.383 INFO:teuthology.orchestra.run.smithi149.stdout: "pos": 0, 2024-04-12T03:14:46.383 INFO:teuthology.orchestra.run.smithi149.stdout: "timestamp": "0.000000", 2024-04-12T03:14:46.383 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:46.383 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:46.383 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:46.383 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:14:46.383 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.384 INFO:teuthology.orchestra.run.smithi149.stdout: "full_sync": { 2024-04-12T03:14:46.384 INFO:teuthology.orchestra.run.smithi149.stdout: "total": 1, 2024-04-12T03:14:46.384 INFO:teuthology.orchestra.run.smithi149.stdout: "complete": 1 2024-04-12T03:14:46.384 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.384 INFO:teuthology.orchestra.run.smithi149.stdout: "current_time": "2024-04-12T03:14:46Z" 2024-04-12T03:14:46.384 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-04-12T03:14:46.385 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2024-04-12T03:14:46.385 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2024-04-12T03:14:46.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:46.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:46.385 DEBUG:teuthology.orchestra.run.smithi149:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:46.435 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:46.435 INFO:teuthology.orchestra.run.smithi149.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:46.462 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:14:46.462+0000 7f1eb0ad8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:46.463 INFO:teuthology.orchestra.run.smithi149.stderr:2024-04-12T03:14:46.462+0000 7f1eb0ad8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:46.474 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-04-12T03:14:46.474 INFO:teuthology.orchestra.run.smithi149.stdout: "sync_status": { 2024-04-12T03:14:46.474 INFO:teuthology.orchestra.run.smithi149.stdout: "info": { 2024-04-12T03:14:46.474 INFO:teuthology.orchestra.run.smithi149.stdout: "status": "sync", 2024-04-12T03:14:46.474 INFO:teuthology.orchestra.run.smithi149.stdout: "num_shards": 4, 2024-04-12T03:14:46.474 INFO:teuthology.orchestra.run.smithi149.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:46.474 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:46.475 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.475 INFO:teuthology.orchestra.run.smithi149.stdout: "markers": [ 2024-04-12T03:14:46.475 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:14:46.475 INFO:teuthology.orchestra.run.smithi149.stdout: "key": 0, 2024-04-12T03:14:46.475 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:14:46.475 INFO:teuthology.orchestra.run.smithi149.stdout: "state": 1, 2024-04-12T03:14:46.475 INFO:teuthology.orchestra.run.smithi149.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:46.475 INFO:teuthology.orchestra.run.smithi149.stdout: "next_step_marker": "", 2024-04-12T03:14:46.475 INFO:teuthology.orchestra.run.smithi149.stdout: "total_entries": 1, 2024-04-12T03:14:46.475 INFO:teuthology.orchestra.run.smithi149.stdout: "pos": 0, 2024-04-12T03:14:46.475 INFO:teuthology.orchestra.run.smithi149.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:14:46.475 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:46.475 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:46.476 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.476 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:14:46.476 INFO:teuthology.orchestra.run.smithi149.stdout: "key": 1, 2024-04-12T03:14:46.476 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:14:46.476 INFO:teuthology.orchestra.run.smithi149.stdout: "state": 1, 2024-04-12T03:14:46.476 INFO:teuthology.orchestra.run.smithi149.stdout: "marker": "", 2024-04-12T03:14:46.476 INFO:teuthology.orchestra.run.smithi149.stdout: "next_step_marker": "", 2024-04-12T03:14:46.476 INFO:teuthology.orchestra.run.smithi149.stdout: "total_entries": 0, 2024-04-12T03:14:46.476 INFO:teuthology.orchestra.run.smithi149.stdout: "pos": 0, 2024-04-12T03:14:46.476 INFO:teuthology.orchestra.run.smithi149.stdout: "timestamp": "0.000000", 2024-04-12T03:14:46.476 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:46.476 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:46.476 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.476 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:14:46.477 INFO:teuthology.orchestra.run.smithi149.stdout: "key": 2, 2024-04-12T03:14:46.477 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:14:46.477 INFO:teuthology.orchestra.run.smithi149.stdout: "state": 1, 2024-04-12T03:14:46.477 INFO:teuthology.orchestra.run.smithi149.stdout: "marker": "", 2024-04-12T03:14:46.477 INFO:teuthology.orchestra.run.smithi149.stdout: "next_step_marker": "", 2024-04-12T03:14:46.477 INFO:teuthology.orchestra.run.smithi149.stdout: "total_entries": 0, 2024-04-12T03:14:46.477 INFO:teuthology.orchestra.run.smithi149.stdout: "pos": 0, 2024-04-12T03:14:46.477 INFO:teuthology.orchestra.run.smithi149.stdout: "timestamp": "0.000000", 2024-04-12T03:14:46.477 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:46.477 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:46.477 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.477 INFO:teuthology.orchestra.run.smithi149.stdout: { 2024-04-12T03:14:46.477 INFO:teuthology.orchestra.run.smithi149.stdout: "key": 3, 2024-04-12T03:14:46.477 INFO:teuthology.orchestra.run.smithi149.stdout: "val": { 2024-04-12T03:14:46.478 INFO:teuthology.orchestra.run.smithi149.stdout: "state": 1, 2024-04-12T03:14:46.478 INFO:teuthology.orchestra.run.smithi149.stdout: "marker": "", 2024-04-12T03:14:46.478 INFO:teuthology.orchestra.run.smithi149.stdout: "next_step_marker": "", 2024-04-12T03:14:46.478 INFO:teuthology.orchestra.run.smithi149.stdout: "total_entries": 0, 2024-04-12T03:14:46.478 INFO:teuthology.orchestra.run.smithi149.stdout: "pos": 0, 2024-04-12T03:14:46.478 INFO:teuthology.orchestra.run.smithi149.stdout: "timestamp": "0.000000", 2024-04-12T03:14:46.478 INFO:teuthology.orchestra.run.smithi149.stdout: "realm_epoch": 2 2024-04-12T03:14:46.478 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:46.478 INFO:teuthology.orchestra.run.smithi149.stdout: } 2024-04-12T03:14:46.478 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-12T03:14:46.479 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.479 INFO:teuthology.orchestra.run.smithi149.stdout: "full_sync": { 2024-04-12T03:14:46.479 INFO:teuthology.orchestra.run.smithi149.stdout: "total": 1, 2024-04-12T03:14:46.479 INFO:teuthology.orchestra.run.smithi149.stdout: "complete": 1 2024-04-12T03:14:46.479 INFO:teuthology.orchestra.run.smithi149.stdout: }, 2024-04-12T03:14:46.479 INFO:teuthology.orchestra.run.smithi149.stdout: "current_time": "2024-04-12T03:14:46Z" 2024-04-12T03:14:46.479 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-04-12T03:14:46.480 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2024-04-12T03:14:46.480 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2024-04-12T03:14:46.551 DEBUG:boto:Using access key provided by client. 2024-04-12T03:14:46.551 DEBUG:boto:Using secret key provided by client. 2024-04-12T03:14:46.551 DEBUG:boto:Using access key provided by client. 2024-04-12T03:14:46.551 DEBUG:boto:Using secret key provided by client. 2024-04-12T03:14:46.552 DEBUG:boto:Using access key provided by client. 2024-04-12T03:14:46.552 DEBUG:boto:Using secret key provided by client. 2024-04-12T03:14:46.552 DEBUG:boto:Using access key provided by client. 2024-04-12T03:14:46.553 DEBUG:boto:Using secret key provided by client. 2024-04-12T03:14:46.553 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-1 2024-04-12T03:14:46.553 DEBUG:boto:path=/dkyges-1/ 2024-04-12T03:14:46.553 DEBUG:boto:auth_path=/dkyges-1/ 2024-04-12T03:14:46.553 DEBUG:boto:Method: PUT 2024-04-12T03:14:46.553 DEBUG:boto:Path: /dkyges-1/ 2024-04-12T03:14:46.553 DEBUG:boto:Data: 2024-04-12T03:14:46.553 DEBUG:boto:Headers: {} 2024-04-12T03:14:46.553 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:14:46.553 DEBUG:boto:Port: 8000 2024-04-12T03:14:46.554 DEBUG:boto:Params: {} 2024-04-12T03:14:46.554 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:14:46.554 DEBUG:boto:Token: None 2024-04-12T03:14:46.554 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:14:46 GMT /dkyges-1/ 2024-04-12T03:14:46.554 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YBMGwqmNcmc3iUKoEkVSy4cXNkk= 2024-04-12T03:14:46.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YBMGwqmNcmc3iUKoEkVSy4cXNkk=', 'Content-Length': '0'} 2024-04-12T03:14:49.113 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000335f368b926fded0-006618a726-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:49.113 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-2 2024-04-12T03:14:49.113 DEBUG:boto:path=/dkyges-2/ 2024-04-12T03:14:49.113 DEBUG:boto:auth_path=/dkyges-2/ 2024-04-12T03:14:49.113 DEBUG:boto:Method: PUT 2024-04-12T03:14:49.114 DEBUG:boto:Path: /dkyges-2/ 2024-04-12T03:14:49.114 DEBUG:boto:Data: 2024-04-12T03:14:49.114 DEBUG:boto:Headers: {} 2024-04-12T03:14:49.114 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:14:49.114 DEBUG:boto:Port: 8001 2024-04-12T03:14:49.114 DEBUG:boto:Params: {} 2024-04-12T03:14:49.114 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:14:49.114 DEBUG:boto:Token: None 2024-04-12T03:14:49.114 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:14:49 GMT /dkyges-2/ 2024-04-12T03:14:49.114 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A3Gq8wfae65x4T7kQk7eKgWuT28= 2024-04-12T03:14:49.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A3Gq8wfae65x4T7kQk7eKgWuT28=', 'Content-Length': '0'} 2024-04-12T03:14:52.121 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b4f1287cb57071ec-006618a729-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:52.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:52.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:52.121 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:52.176 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:52.176 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:52.211 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:14:52.225 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:14:52.225 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:52.225 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:14:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:14:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891689.140245_69.1", 2024-04-12T03:14:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:14:49.140245Z" 2024-04-12T03:14:52.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:52.227 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:52.227 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:52.227 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:14:52.227 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:52.227 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:14:52.228 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891689.140245_69.1', 3: ''} 2024-04-12T03:14:52.228 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:14:52.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:52.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:52.229 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:52.283 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:52.284 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:52.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:52.313+0000 7ff48bebb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:52.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:52.314+0000 7ff48bebb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:52.329 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:14:52.329 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:14:52.329 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:14:52.330 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:14:52.330 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:14:52.330 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:52.330 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:52.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:52.331 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:14:52.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:52.331 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:14:52.331 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:52.331 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:14:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:14:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:52.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:14:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:52.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:14:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891689.100262_62.1", 2024-04-12T03:14:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:14:49.100262Z", 2024-04-12T03:14:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:52.334 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:14:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:52.335 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:14:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:14:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:14:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:14:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:52.336 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:14:52Z" 2024-04-12T03:14:52.336 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:14:52.337 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891689.100262_62.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:14:49.100262Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:52Z" } 2024-04-12T03:14:52.338 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891689.100262_62.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:14:49.100262Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-12T03:14:52.338 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891689.140245_69.1', 3: ''} 2024-04-12T03:14:52.338 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891689.100262_62.1', 3: ''} 2024-04-12T03:14:52.338 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712891689.140245_69.1 target=1_1712891689.100262_62.1 2024-04-12T03:14:57.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:57.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:57.341 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:57.401 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:57.401 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:57.434+0000 7f61f93f9780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:57.436+0000 7f61f93f9780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:57.450 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:14:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:14:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:14:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:14:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:14:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:14:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:14:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:14:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:14:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:14:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:14:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891689.140245_69.1", 2024-04-12T03:14:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:14:49.140245Z", 2024-04-12T03:14:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:14:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:14:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:14:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:14:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:14:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:14:57Z" 2024-04-12T03:14:57.457 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:14:57.458 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891689.140245_69.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:14:49.140245Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:57Z" } 2024-04-12T03:14:57.458 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891689.140245_69.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:14:49.140245Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-12T03:14:57.458 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891689.140245_69.1', 3: ''} 2024-04-12T03:14:57.459 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891689.140245_69.1', 3: ''} 2024-04-12T03:14:57.459 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:14:57.459 DEBUG:boto:path=/dkyges-1/ 2024-04-12T03:14:57.459 DEBUG:boto:auth_path=/dkyges-1/ 2024-04-12T03:14:57.459 DEBUG:boto:Method: HEAD 2024-04-12T03:14:57.459 DEBUG:boto:Path: /dkyges-1/ 2024-04-12T03:14:57.459 DEBUG:boto:Data: 2024-04-12T03:14:57.459 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.459 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:14:57.459 DEBUG:boto:Port: 8000 2024-04-12T03:14:57.460 DEBUG:boto:Params: {} 2024-04-12T03:14:57.460 DEBUG:boto:Token: None 2024-04-12T03:14:57.460 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:57 GMT /dkyges-1/ 2024-04-12T03:14:57.460 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:naDnz/H4YYKtIDoRe2MzolcX1CM= 2024-04-12T03:14:57.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:naDnz/H4YYKtIDoRe2MzolcX1CM=', 'Content-Length': '0'} 2024-04-12T03:14:57.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000264031c5e807e31-006618a731-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.463 DEBUG:boto:path=/dkyges-2/ 2024-04-12T03:14:57.463 DEBUG:boto:auth_path=/dkyges-2/ 2024-04-12T03:14:57.463 DEBUG:boto:Method: HEAD 2024-04-12T03:14:57.463 DEBUG:boto:Path: /dkyges-2/ 2024-04-12T03:14:57.463 DEBUG:boto:Data: 2024-04-12T03:14:57.463 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.463 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:14:57.463 DEBUG:boto:Port: 8000 2024-04-12T03:14:57.463 DEBUG:boto:Params: {} 2024-04-12T03:14:57.463 DEBUG:boto:Token: None 2024-04-12T03:14:57.464 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:57 GMT /dkyges-2/ 2024-04-12T03:14:57.464 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ae4bY6cjBQwmFAmqxaE4oljITeE= 2024-04-12T03:14:57.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ae4bY6cjBQwmFAmqxaE4oljITeE=', 'Content-Length': '0'} 2024-04-12T03:14:57.466 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff816f4c176229d5-006618a731-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.466 DEBUG:boto:path=/dkyges-1/ 2024-04-12T03:14:57.466 DEBUG:boto:auth_path=/dkyges-1/ 2024-04-12T03:14:57.466 DEBUG:boto:Method: HEAD 2024-04-12T03:14:57.466 DEBUG:boto:Path: /dkyges-1/ 2024-04-12T03:14:57.466 DEBUG:boto:Data: 2024-04-12T03:14:57.466 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.467 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:14:57.467 DEBUG:boto:Port: 8001 2024-04-12T03:14:57.467 DEBUG:boto:Params: {} 2024-04-12T03:14:57.467 DEBUG:boto:Token: None 2024-04-12T03:14:57.467 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:57 GMT /dkyges-1/ 2024-04-12T03:14:57.467 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:naDnz/H4YYKtIDoRe2MzolcX1CM= 2024-04-12T03:14:57.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:naDnz/H4YYKtIDoRe2MzolcX1CM=', 'Content-Length': '0'} 2024-04-12T03:14:57.469 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a93f3ff9ca07e569-006618a731-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.469 DEBUG:boto:path=/dkyges-2/ 2024-04-12T03:14:57.469 DEBUG:boto:auth_path=/dkyges-2/ 2024-04-12T03:14:57.470 DEBUG:boto:Method: HEAD 2024-04-12T03:14:57.470 DEBUG:boto:Path: /dkyges-2/ 2024-04-12T03:14:57.470 DEBUG:boto:Data: 2024-04-12T03:14:57.470 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.470 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:14:57.470 DEBUG:boto:Port: 8001 2024-04-12T03:14:57.470 DEBUG:boto:Params: {} 2024-04-12T03:14:57.470 DEBUG:boto:Token: None 2024-04-12T03:14:57.470 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:57 GMT /dkyges-2/ 2024-04-12T03:14:57.471 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ae4bY6cjBQwmFAmqxaE4oljITeE= 2024-04-12T03:14:57.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ae4bY6cjBQwmFAmqxaE4oljITeE=', 'Content-Length': '0'} 2024-04-12T03:14:57.473 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef77a586f24db5ca-006618a731-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.473 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2024-04-12T03:14:57.474 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-3 2024-04-12T03:14:57.474 DEBUG:boto:path=/dkyges-3/ 2024-04-12T03:14:57.475 DEBUG:boto:auth_path=/dkyges-3/ 2024-04-12T03:14:57.475 DEBUG:boto:Method: PUT 2024-04-12T03:14:57.475 DEBUG:boto:Path: /dkyges-3/ 2024-04-12T03:14:57.475 DEBUG:boto:Data: 2024-04-12T03:14:57.475 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.475 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:14:57.475 DEBUG:boto:Port: 8000 2024-04-12T03:14:57.475 DEBUG:boto:Params: {} 2024-04-12T03:14:57.476 DEBUG:boto:Token: None 2024-04-12T03:14:57.476 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:14:57 GMT /dkyges-3/ 2024-04-12T03:14:57.476 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VFTtuEL2bPOz7XnJR3dXXVB2uL8= 2024-04-12T03:14:57.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VFTtuEL2bPOz7XnJR3dXXVB2uL8=', 'Content-Length': '0'} 2024-04-12T03:14:57.490 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002711e3e072c86bc4-006618a731-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.490 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-4 2024-04-12T03:14:57.490 DEBUG:boto:path=/dkyges-4/ 2024-04-12T03:14:57.490 DEBUG:boto:auth_path=/dkyges-4/ 2024-04-12T03:14:57.490 DEBUG:boto:Method: PUT 2024-04-12T03:14:57.491 DEBUG:boto:Path: /dkyges-4/ 2024-04-12T03:14:57.491 DEBUG:boto:Data: 2024-04-12T03:14:57.491 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.491 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:14:57.491 DEBUG:boto:Port: 8001 2024-04-12T03:14:57.491 DEBUG:boto:Params: {} 2024-04-12T03:14:57.491 DEBUG:boto:Token: None 2024-04-12T03:14:57.492 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:14:57 GMT /dkyges-4/ 2024-04-12T03:14:57.492 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:F2BHbx2YXg+oE8S/c5GTGtVl7J0= 2024-04-12T03:14:57.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:F2BHbx2YXg+oE8S/c5GTGtVl7J0=', 'Content-Length': '0'} 2024-04-12T03:14:57.515 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000013c3cc93e84934b4-006618a731-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:57.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:57.516 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:57.572 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:57.572 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:57.606 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:14:57.623 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:14:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:14:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:14:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891697.505052_77.1", 2024-04-12T03:14:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:14:57.505052Z" 2024-04-12T03:14:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:14:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:57.625 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:14:57.626 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891697.505052_77.1', 3: ''} 2024-04-12T03:14:57.626 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:14:57.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:57.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:57.626 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:57.682 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:57.682 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:57.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:57.713+0000 7f6130d8e780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:57.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:57.714+0000 7f6130d8e780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:57.728 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:14:57.729 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:14:57.729 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:14:57.729 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:14:57.729 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:14:57.729 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:57.730 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:57.730 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.730 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:14:57.730 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:57.730 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:14:57.731 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:57.731 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:57.731 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:57.731 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:57.731 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:14:57.732 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:57.732 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:14:57.732 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:57.732 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:57.732 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.733 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:57.733 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:14:57.733 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:57.733 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:57.733 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:57.734 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:57.734 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:57.734 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:57.734 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:57.735 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:57.735 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:57.735 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.735 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:57.735 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:14:57.736 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:57.736 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:57.736 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891697.505052_77.1", 2024-04-12T03:14:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:57.737 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:14:57.505052Z", 2024-04-12T03:14:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:57.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:14:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:57.739 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:57.740 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:57.740 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:14:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:14:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:14:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:14:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:14:57Z" 2024-04-12T03:14:57.744 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:14:57.746 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891697.505052_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:14:57.505052Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:57Z" } 2024-04-12T03:14:57.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891697.505052_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:14:57.505052Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-12T03:14:57.747 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891697.505052_77.1', 3: ''} 2024-04-12T03:14:57.747 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891697.505052_77.1', 3: ''} 2024-04-12T03:14:57.747 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:14:57.747 DEBUG:boto:path=/dkyges-3/ 2024-04-12T03:14:57.747 DEBUG:boto:auth_path=/dkyges-3/ 2024-04-12T03:14:57.748 DEBUG:boto:Method: HEAD 2024-04-12T03:14:57.748 DEBUG:boto:Path: /dkyges-3/ 2024-04-12T03:14:57.748 DEBUG:boto:Data: 2024-04-12T03:14:57.748 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.748 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:14:57.748 DEBUG:boto:Port: 8000 2024-04-12T03:14:57.748 DEBUG:boto:Params: {} 2024-04-12T03:14:57.749 DEBUG:boto:Token: None 2024-04-12T03:14:57.749 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:57 GMT /dkyges-3/ 2024-04-12T03:14:57.749 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ovy590VKQOjBfJKza565L5akc0U= 2024-04-12T03:14:57.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ovy590VKQOjBfJKza565L5akc0U=', 'Content-Length': '0'} 2024-04-12T03:14:57.751 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6c21a9927bb0586-006618a731-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.752 DEBUG:boto:path=/dkyges-4/ 2024-04-12T03:14:57.752 DEBUG:boto:auth_path=/dkyges-4/ 2024-04-12T03:14:57.752 DEBUG:boto:Method: HEAD 2024-04-12T03:14:57.752 DEBUG:boto:Path: /dkyges-4/ 2024-04-12T03:14:57.752 DEBUG:boto:Data: 2024-04-12T03:14:57.752 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.753 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:14:57.753 DEBUG:boto:Port: 8000 2024-04-12T03:14:57.753 DEBUG:boto:Params: {} 2024-04-12T03:14:57.753 DEBUG:boto:Token: None 2024-04-12T03:14:57.753 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:57 GMT /dkyges-4/ 2024-04-12T03:14:57.753 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FQNaUp6GARJfyWIiP9tlX3jZCco= 2024-04-12T03:14:57.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FQNaUp6GARJfyWIiP9tlX3jZCco=', 'Content-Length': '0'} 2024-04-12T03:14:57.756 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e2f97cdd4721539-006618a731-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.756 DEBUG:boto:path=/dkyges-3/ 2024-04-12T03:14:57.756 DEBUG:boto:auth_path=/dkyges-3/ 2024-04-12T03:14:57.756 DEBUG:boto:Method: HEAD 2024-04-12T03:14:57.757 DEBUG:boto:Path: /dkyges-3/ 2024-04-12T03:14:57.757 DEBUG:boto:Data: 2024-04-12T03:14:57.757 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.757 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:14:57.757 DEBUG:boto:Port: 8001 2024-04-12T03:14:57.757 DEBUG:boto:Params: {} 2024-04-12T03:14:57.758 DEBUG:boto:Token: None 2024-04-12T03:14:57.758 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:57 GMT /dkyges-3/ 2024-04-12T03:14:57.758 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ovy590VKQOjBfJKza565L5akc0U= 2024-04-12T03:14:57.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ovy590VKQOjBfJKza565L5akc0U=', 'Content-Length': '0'} 2024-04-12T03:14:57.761 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e644c15d3ed63a36-006618a731-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.761 DEBUG:boto:path=/dkyges-4/ 2024-04-12T03:14:57.761 DEBUG:boto:auth_path=/dkyges-4/ 2024-04-12T03:14:57.762 DEBUG:boto:Method: HEAD 2024-04-12T03:14:57.762 DEBUG:boto:Path: /dkyges-4/ 2024-04-12T03:14:57.762 DEBUG:boto:Data: 2024-04-12T03:14:57.762 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.762 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:14:57.762 DEBUG:boto:Port: 8001 2024-04-12T03:14:57.763 DEBUG:boto:Params: {} 2024-04-12T03:14:57.763 DEBUG:boto:Token: None 2024-04-12T03:14:57.763 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:57 GMT /dkyges-4/ 2024-04-12T03:14:57.764 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FQNaUp6GARJfyWIiP9tlX3jZCco= 2024-04-12T03:14:57.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FQNaUp6GARJfyWIiP9tlX3jZCco=', 'Content-Length': '0'} 2024-04-12T03:14:57.766 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e011ac2bd05f2fb-006618a731-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.767 DEBUG:boto:path=/dkyges-3/ 2024-04-12T03:14:57.767 DEBUG:boto:auth_path=/dkyges-3/ 2024-04-12T03:14:57.767 DEBUG:boto:Method: PUT 2024-04-12T03:14:57.768 DEBUG:boto:Path: /dkyges-3/ 2024-04-12T03:14:57.768 DEBUG:boto:Data: 2024-04-12T03:14:57.768 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.768 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:14:57.768 DEBUG:boto:Port: 8000 2024-04-12T03:14:57.769 DEBUG:boto:Params: {} 2024-04-12T03:14:57.769 DEBUG:boto:Token: None 2024-04-12T03:14:57.769 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:14:57 GMT /dkyges-3/ 2024-04-12T03:14:57.770 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VFTtuEL2bPOz7XnJR3dXXVB2uL8= 2024-04-12T03:14:57.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VFTtuEL2bPOz7XnJR3dXXVB2uL8=', 'Content-Length': '0'} 2024-04-12T03:14:57.773 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003f0afb4a57c486ff-006618a731-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.774 DEBUG:boto:path=/dkyges-4/ 2024-04-12T03:14:57.774 DEBUG:boto:auth_path=/dkyges-4/ 2024-04-12T03:14:57.774 DEBUG:boto:Method: PUT 2024-04-12T03:14:57.774 DEBUG:boto:Path: /dkyges-4/ 2024-04-12T03:14:57.775 DEBUG:boto:Data: 2024-04-12T03:14:57.775 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.775 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:14:57.776 DEBUG:boto:Port: 8000 2024-04-12T03:14:57.776 DEBUG:boto:Params: {} 2024-04-12T03:14:57.776 DEBUG:boto:Token: None 2024-04-12T03:14:57.777 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:14:57 GMT /dkyges-4/ 2024-04-12T03:14:57.777 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:F2BHbx2YXg+oE8S/c5GTGtVl7J0= 2024-04-12T03:14:57.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:F2BHbx2YXg+oE8S/c5GTGtVl7J0=', 'Content-Length': '0'} 2024-04-12T03:14:57.780 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000205939c202599c8e-006618a731-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.781 DEBUG:boto:path=/dkyges-3/ 2024-04-12T03:14:57.781 DEBUG:boto:auth_path=/dkyges-3/ 2024-04-12T03:14:57.781 DEBUG:boto:Method: PUT 2024-04-12T03:14:57.782 DEBUG:boto:Path: /dkyges-3/ 2024-04-12T03:14:57.782 DEBUG:boto:Data: 2024-04-12T03:14:57.782 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.782 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:14:57.782 DEBUG:boto:Port: 8001 2024-04-12T03:14:57.783 DEBUG:boto:Params: {} 2024-04-12T03:14:57.783 DEBUG:boto:Token: None 2024-04-12T03:14:57.783 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:14:57 GMT /dkyges-3/ 2024-04-12T03:14:57.783 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VFTtuEL2bPOz7XnJR3dXXVB2uL8= 2024-04-12T03:14:57.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VFTtuEL2bPOz7XnJR3dXXVB2uL8=', 'Content-Length': '0'} 2024-04-12T03:14:57.789 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000186faae7b4bc79c7-006618a731-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.789 DEBUG:boto:path=/dkyges-4/ 2024-04-12T03:14:57.789 DEBUG:boto:auth_path=/dkyges-4/ 2024-04-12T03:14:57.790 DEBUG:boto:Method: PUT 2024-04-12T03:14:57.790 DEBUG:boto:Path: /dkyges-4/ 2024-04-12T03:14:57.790 DEBUG:boto:Data: 2024-04-12T03:14:57.790 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.790 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:14:57.790 DEBUG:boto:Port: 8001 2024-04-12T03:14:57.790 DEBUG:boto:Params: {} 2024-04-12T03:14:57.790 DEBUG:boto:Token: None 2024-04-12T03:14:57.790 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:14:57 GMT /dkyges-4/ 2024-04-12T03:14:57.791 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:F2BHbx2YXg+oE8S/c5GTGtVl7J0= 2024-04-12T03:14:57.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:F2BHbx2YXg+oE8S/c5GTGtVl7J0=', 'Content-Length': '0'} 2024-04-12T03:14:57.836 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000055694fb5943c8ad-006618a731-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.837 DEBUG:boto:path=/dkyges-3/ 2024-04-12T03:14:57.837 DEBUG:boto:auth_path=/dkyges-3/ 2024-04-12T03:14:57.837 DEBUG:boto:Method: HEAD 2024-04-12T03:14:57.837 DEBUG:boto:Path: /dkyges-3/ 2024-04-12T03:14:57.837 DEBUG:boto:Data: 2024-04-12T03:14:57.837 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.838 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:14:57.838 DEBUG:boto:Port: 8000 2024-04-12T03:14:57.838 DEBUG:boto:Params: {} 2024-04-12T03:14:57.838 DEBUG:boto:Token: None 2024-04-12T03:14:57.838 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:57 GMT /dkyges-3/ 2024-04-12T03:14:57.838 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ovy590VKQOjBfJKza565L5akc0U= 2024-04-12T03:14:57.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ovy590VKQOjBfJKza565L5akc0U=', 'Content-Length': '0'} 2024-04-12T03:14:57.841 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4c9eaf0559a65c6-006618a731-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.841 DEBUG:boto:path=/dkyges-4/ 2024-04-12T03:14:57.841 DEBUG:boto:auth_path=/dkyges-4/ 2024-04-12T03:14:57.841 DEBUG:boto:Method: HEAD 2024-04-12T03:14:57.841 DEBUG:boto:Path: /dkyges-4/ 2024-04-12T03:14:57.842 DEBUG:boto:Data: 2024-04-12T03:14:57.842 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.842 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:14:57.842 DEBUG:boto:Port: 8000 2024-04-12T03:14:57.842 DEBUG:boto:Params: {} 2024-04-12T03:14:57.842 DEBUG:boto:Token: None 2024-04-12T03:14:57.842 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:57 GMT /dkyges-4/ 2024-04-12T03:14:57.842 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FQNaUp6GARJfyWIiP9tlX3jZCco= 2024-04-12T03:14:57.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FQNaUp6GARJfyWIiP9tlX3jZCco=', 'Content-Length': '0'} 2024-04-12T03:14:57.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052c0924a7687c483-006618a731-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.845 DEBUG:boto:path=/dkyges-3/ 2024-04-12T03:14:57.845 DEBUG:boto:auth_path=/dkyges-3/ 2024-04-12T03:14:57.845 DEBUG:boto:Method: HEAD 2024-04-12T03:14:57.845 DEBUG:boto:Path: /dkyges-3/ 2024-04-12T03:14:57.845 DEBUG:boto:Data: 2024-04-12T03:14:57.845 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.845 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:14:57.845 DEBUG:boto:Port: 8001 2024-04-12T03:14:57.846 DEBUG:boto:Params: {} 2024-04-12T03:14:57.846 DEBUG:boto:Token: None 2024-04-12T03:14:57.846 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:57 GMT /dkyges-3/ 2024-04-12T03:14:57.846 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ovy590VKQOjBfJKza565L5akc0U= 2024-04-12T03:14:57.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ovy590VKQOjBfJKza565L5akc0U=', 'Content-Length': '0'} 2024-04-12T03:14:57.848 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2bcfe19a0abe885-006618a731-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.849 DEBUG:boto:path=/dkyges-4/ 2024-04-12T03:14:57.849 DEBUG:boto:auth_path=/dkyges-4/ 2024-04-12T03:14:57.849 DEBUG:boto:Method: HEAD 2024-04-12T03:14:57.849 DEBUG:boto:Path: /dkyges-4/ 2024-04-12T03:14:57.850 DEBUG:boto:Data: 2024-04-12T03:14:57.850 DEBUG:boto:Headers: {} 2024-04-12T03:14:57.850 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:14:57.851 DEBUG:boto:Port: 8001 2024-04-12T03:14:57.851 DEBUG:boto:Params: {} 2024-04-12T03:14:57.851 DEBUG:boto:Token: None 2024-04-12T03:14:57.851 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:57 GMT /dkyges-4/ 2024-04-12T03:14:57.851 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FQNaUp6GARJfyWIiP9tlX3jZCco= 2024-04-12T03:14:57.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FQNaUp6GARJfyWIiP9tlX3jZCco=', 'Content-Length': '0'} 2024-04-12T03:14:57.853 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbc7582d09c1f5ee-006618a731-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:57.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:57.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:57.854 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:57.915 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:57.915 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:57.950 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:14:57.963 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:14:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:14:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:14:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891697.505052_77.1", 2024-04-12T03:14:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:14:57.505052Z" 2024-04-12T03:14:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:14:57.965 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:57.965 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:14:57.966 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891697.505052_77.1', 3: ''} 2024-04-12T03:14:57.967 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:14:57.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:57.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:57.967 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:58.027 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:58.027 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:58.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:58.054+0000 7f9022394780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:58.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:58.055+0000 7f9022394780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:58.070 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:14:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:14:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:14:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:14:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:14:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.070 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:14:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:14:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:14:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:14:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:14:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:58.072 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:14:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891697.505052_77.1", 2024-04-12T03:14:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:14:57.505052Z", 2024-04-12T03:14:58.073 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:14:58.074 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:58.075 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:14:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:14:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:14:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:14:58.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.077 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:14:58Z" 2024-04-12T03:14:58.077 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:14:58.078 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891697.505052_77.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:14:57.505052Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:58Z" } 2024-04-12T03:14:58.078 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891697.505052_77.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:14:57.505052Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-12T03:14:58.078 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891697.505052_77.1', 3: ''} 2024-04-12T03:14:58.078 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891697.505052_77.1', 3: ''} 2024-04-12T03:14:58.078 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:14:58.078 DEBUG:boto:path=/dkyges-3/ 2024-04-12T03:14:58.078 DEBUG:boto:auth_path=/dkyges-3/ 2024-04-12T03:14:58.079 DEBUG:boto:Method: HEAD 2024-04-12T03:14:58.079 DEBUG:boto:Path: /dkyges-3/ 2024-04-12T03:14:58.079 DEBUG:boto:Data: 2024-04-12T03:14:58.079 DEBUG:boto:Headers: {} 2024-04-12T03:14:58.079 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:14:58.079 DEBUG:boto:Port: 8000 2024-04-12T03:14:58.079 DEBUG:boto:Params: {} 2024-04-12T03:14:58.079 DEBUG:boto:Token: None 2024-04-12T03:14:58.079 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:58 GMT /dkyges-3/ 2024-04-12T03:14:58.080 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zfoJ5PbsyU7I/tdkiOnOSvAAXA0= 2024-04-12T03:14:58.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zfoJ5PbsyU7I/tdkiOnOSvAAXA0=', 'Content-Length': '0'} 2024-04-12T03:14:58.082 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038ecbaa117ef1339-006618a732-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:58.082 DEBUG:boto:path=/dkyges-4/ 2024-04-12T03:14:58.082 DEBUG:boto:auth_path=/dkyges-4/ 2024-04-12T03:14:58.082 DEBUG:boto:Method: HEAD 2024-04-12T03:14:58.083 DEBUG:boto:Path: /dkyges-4/ 2024-04-12T03:14:58.083 DEBUG:boto:Data: 2024-04-12T03:14:58.083 DEBUG:boto:Headers: {} 2024-04-12T03:14:58.083 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:14:58.083 DEBUG:boto:Port: 8000 2024-04-12T03:14:58.083 DEBUG:boto:Params: {} 2024-04-12T03:14:58.083 DEBUG:boto:Token: None 2024-04-12T03:14:58.083 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:58 GMT /dkyges-4/ 2024-04-12T03:14:58.083 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l3rj7wa3lMheL+Zu6SoRBkesjtU= 2024-04-12T03:14:58.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l3rj7wa3lMheL+Zu6SoRBkesjtU=', 'Content-Length': '0'} 2024-04-12T03:14:58.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8b3060c1fdb41d2-006618a732-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:58.086 DEBUG:boto:path=/dkyges-3/ 2024-04-12T03:14:58.086 DEBUG:boto:auth_path=/dkyges-3/ 2024-04-12T03:14:58.086 DEBUG:boto:Method: HEAD 2024-04-12T03:14:58.086 DEBUG:boto:Path: /dkyges-3/ 2024-04-12T03:14:58.086 DEBUG:boto:Data: 2024-04-12T03:14:58.086 DEBUG:boto:Headers: {} 2024-04-12T03:14:58.086 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:14:58.087 DEBUG:boto:Port: 8001 2024-04-12T03:14:58.087 DEBUG:boto:Params: {} 2024-04-12T03:14:58.087 DEBUG:boto:Token: None 2024-04-12T03:14:58.087 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:58 GMT /dkyges-3/ 2024-04-12T03:14:58.087 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zfoJ5PbsyU7I/tdkiOnOSvAAXA0= 2024-04-12T03:14:58.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zfoJ5PbsyU7I/tdkiOnOSvAAXA0=', 'Content-Length': '0'} 2024-04-12T03:14:58.089 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d65167f69bbe9d13-006618a732-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:58.090 DEBUG:boto:path=/dkyges-4/ 2024-04-12T03:14:58.090 DEBUG:boto:auth_path=/dkyges-4/ 2024-04-12T03:14:58.090 DEBUG:boto:Method: HEAD 2024-04-12T03:14:58.090 DEBUG:boto:Path: /dkyges-4/ 2024-04-12T03:14:58.090 DEBUG:boto:Data: 2024-04-12T03:14:58.090 DEBUG:boto:Headers: {} 2024-04-12T03:14:58.091 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:14:58.091 DEBUG:boto:Port: 8001 2024-04-12T03:14:58.091 DEBUG:boto:Params: {} 2024-04-12T03:14:58.091 DEBUG:boto:Token: None 2024-04-12T03:14:58.091 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:58 GMT /dkyges-4/ 2024-04-12T03:14:58.091 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l3rj7wa3lMheL+Zu6SoRBkesjtU= 2024-04-12T03:14:58.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l3rj7wa3lMheL+Zu6SoRBkesjtU=', 'Content-Length': '0'} 2024-04-12T03:14:58.093 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027e0567f531ca9e8-006618a732-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:58.093 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2024-04-12T03:14:58.094 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-5 2024-04-12T03:14:58.094 DEBUG:boto:path=/dkyges-5/ 2024-04-12T03:14:58.094 DEBUG:boto:auth_path=/dkyges-5/ 2024-04-12T03:14:58.094 DEBUG:boto:Method: PUT 2024-04-12T03:14:58.094 DEBUG:boto:Path: /dkyges-5/ 2024-04-12T03:14:58.094 DEBUG:boto:Data: 2024-04-12T03:14:58.094 DEBUG:boto:Headers: {} 2024-04-12T03:14:58.094 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:14:58.094 DEBUG:boto:Port: 8000 2024-04-12T03:14:58.095 DEBUG:boto:Params: {} 2024-04-12T03:14:58.095 DEBUG:boto:Token: None 2024-04-12T03:14:58.095 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:14:58 GMT /dkyges-5/ 2024-04-12T03:14:58.095 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qIaE0PEiRDGvv7udUDmFVJmneds= 2024-04-12T03:14:58.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qIaE0PEiRDGvv7udUDmFVJmneds=', 'Content-Length': '0'} 2024-04-12T03:14:58.109 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a0b33cb12be0d030-006618a732-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:58.109 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-6 2024-04-12T03:14:58.109 DEBUG:boto:path=/dkyges-6/ 2024-04-12T03:14:58.109 DEBUG:boto:auth_path=/dkyges-6/ 2024-04-12T03:14:58.109 DEBUG:boto:Method: PUT 2024-04-12T03:14:58.109 DEBUG:boto:Path: /dkyges-6/ 2024-04-12T03:14:58.110 DEBUG:boto:Data: 2024-04-12T03:14:58.110 DEBUG:boto:Headers: {} 2024-04-12T03:14:58.110 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:14:58.110 DEBUG:boto:Port: 8001 2024-04-12T03:14:58.110 DEBUG:boto:Params: {} 2024-04-12T03:14:58.110 DEBUG:boto:Token: None 2024-04-12T03:14:58.110 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:14:58 GMT /dkyges-6/ 2024-04-12T03:14:58.110 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SAry6wAZNrh3GQNlofgAaZDdzh8= 2024-04-12T03:14:58.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SAry6wAZNrh3GQNlofgAaZDdzh8=', 'Content-Length': '0'} 2024-04-12T03:14:58.134 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000647be86cf3c2a5d2-006618a732-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:58.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:58.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:58.135 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:58.189 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:58.189 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:58.228 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:14:58.242 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:14:58.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.242 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:58.242 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:14:58.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:14:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891698.124356_85.1", 2024-04-12T03:14:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:14:58.124356Z" 2024-04-12T03:14:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.244 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:58.244 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:14:58.244 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.244 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:14:58.245 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891698.124356_85.1', 3: ''} 2024-04-12T03:14:58.245 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:14:58.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:58.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:58.246 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:58.302 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:58.302 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:58.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:58.331+0000 7f1a4052a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:58.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:58.332+0000 7f1a4052a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:58.346 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:14:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:14:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:14:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:14:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:14:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:14:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:14:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:14:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:14:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:14:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:14:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891698.124356_85.1", 2024-04-12T03:14:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:14:58.124356Z", 2024-04-12T03:14:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:14:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:14:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:14:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:14:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:14:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:14:58Z" 2024-04-12T03:14:58.353 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:14:58.353 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891698.124356_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:14:58.124356Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:58Z" } 2024-04-12T03:14:58.354 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891698.124356_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:14:58.124356Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-12T03:14:58.354 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891698.124356_85.1', 3: ''} 2024-04-12T03:14:58.354 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891698.124356_85.1', 3: ''} 2024-04-12T03:14:58.354 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:14:58.354 DEBUG:boto:path=/dkyges-5/ 2024-04-12T03:14:58.354 DEBUG:boto:auth_path=/dkyges-5/ 2024-04-12T03:14:58.354 DEBUG:boto:Method: HEAD 2024-04-12T03:14:58.355 DEBUG:boto:Path: /dkyges-5/ 2024-04-12T03:14:58.355 DEBUG:boto:Data: 2024-04-12T03:14:58.355 DEBUG:boto:Headers: {} 2024-04-12T03:14:58.355 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:14:58.355 DEBUG:boto:Port: 8000 2024-04-12T03:14:58.355 DEBUG:boto:Params: {} 2024-04-12T03:14:58.355 DEBUG:boto:Token: None 2024-04-12T03:14:58.355 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:58 GMT /dkyges-5/ 2024-04-12T03:14:58.355 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VKCyoylx9FBbsDibkk1YAQnB2BE= 2024-04-12T03:14:58.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VKCyoylx9FBbsDibkk1YAQnB2BE=', 'Content-Length': '0'} 2024-04-12T03:14:58.358 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006aed6e2a68034db5-006618a732-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:58.358 DEBUG:boto:path=/dkyges-6/ 2024-04-12T03:14:58.358 DEBUG:boto:auth_path=/dkyges-6/ 2024-04-12T03:14:58.358 DEBUG:boto:Method: HEAD 2024-04-12T03:14:58.358 DEBUG:boto:Path: /dkyges-6/ 2024-04-12T03:14:58.359 DEBUG:boto:Data: 2024-04-12T03:14:58.359 DEBUG:boto:Headers: {} 2024-04-12T03:14:58.359 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:14:58.359 DEBUG:boto:Port: 8000 2024-04-12T03:14:58.359 DEBUG:boto:Params: {} 2024-04-12T03:14:58.359 DEBUG:boto:Token: None 2024-04-12T03:14:58.359 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:58 GMT /dkyges-6/ 2024-04-12T03:14:58.359 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r1+O3jStC9dc5KSk3cSvveWyjmg= 2024-04-12T03:14:58.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r1+O3jStC9dc5KSk3cSvveWyjmg=', 'Content-Length': '0'} 2024-04-12T03:14:58.361 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006ddf54710ac8437-006618a732-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:58.362 DEBUG:boto:path=/dkyges-5/ 2024-04-12T03:14:58.362 DEBUG:boto:auth_path=/dkyges-5/ 2024-04-12T03:14:58.362 DEBUG:boto:Method: HEAD 2024-04-12T03:14:58.362 DEBUG:boto:Path: /dkyges-5/ 2024-04-12T03:14:58.362 DEBUG:boto:Data: 2024-04-12T03:14:58.362 DEBUG:boto:Headers: {} 2024-04-12T03:14:58.362 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:14:58.362 DEBUG:boto:Port: 8001 2024-04-12T03:14:58.362 DEBUG:boto:Params: {} 2024-04-12T03:14:58.363 DEBUG:boto:Token: None 2024-04-12T03:14:58.363 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:58 GMT /dkyges-5/ 2024-04-12T03:14:58.363 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VKCyoylx9FBbsDibkk1YAQnB2BE= 2024-04-12T03:14:58.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VKCyoylx9FBbsDibkk1YAQnB2BE=', 'Content-Length': '0'} 2024-04-12T03:14:58.365 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005178db73653ce40d-006618a732-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:58.365 DEBUG:boto:path=/dkyges-6/ 2024-04-12T03:14:58.365 DEBUG:boto:auth_path=/dkyges-6/ 2024-04-12T03:14:58.365 DEBUG:boto:Method: HEAD 2024-04-12T03:14:58.365 DEBUG:boto:Path: /dkyges-6/ 2024-04-12T03:14:58.366 DEBUG:boto:Data: 2024-04-12T03:14:58.366 DEBUG:boto:Headers: {} 2024-04-12T03:14:58.366 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:14:58.366 DEBUG:boto:Port: 8001 2024-04-12T03:14:58.366 DEBUG:boto:Params: {} 2024-04-12T03:14:58.366 DEBUG:boto:Token: None 2024-04-12T03:14:58.366 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:14:58 GMT /dkyges-6/ 2024-04-12T03:14:58.366 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r1+O3jStC9dc5KSk3cSvveWyjmg= 2024-04-12T03:14:58.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r1+O3jStC9dc5KSk3cSvveWyjmg=', 'Content-Length': '0'} 2024-04-12T03:14:58.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed22cfcc5c697bfd-006618a732-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:14:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:58.369 DEBUG:boto:path=/dkyges-5/ 2024-04-12T03:14:58.369 DEBUG:boto:auth_path=/dkyges-5/ 2024-04-12T03:14:58.369 DEBUG:boto:Method: DELETE 2024-04-12T03:14:58.369 DEBUG:boto:Path: /dkyges-5/ 2024-04-12T03:14:58.369 DEBUG:boto:Data: 2024-04-12T03:14:58.369 DEBUG:boto:Headers: {} 2024-04-12T03:14:58.369 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:14:58.369 DEBUG:boto:Port: 8000 2024-04-12T03:14:58.369 DEBUG:boto:Params: {} 2024-04-12T03:14:58.370 DEBUG:boto:Token: None 2024-04-12T03:14:58.370 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:14:58 GMT /dkyges-5/ 2024-04-12T03:14:58.370 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QhYLFg+mejB+fT4QX339r4fkfbs= 2024-04-12T03:14:58.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QhYLFg+mejB+fT4QX339r4fkfbs=', 'Content-Length': '0'} 2024-04-12T03:14:58.401 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003aaa718fad624319-006618a732-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:14:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:58.401 DEBUG:boto:path=/dkyges-6/ 2024-04-12T03:14:58.401 DEBUG:boto:auth_path=/dkyges-6/ 2024-04-12T03:14:58.401 DEBUG:boto:Method: DELETE 2024-04-12T03:14:58.402 DEBUG:boto:Path: /dkyges-6/ 2024-04-12T03:14:58.402 DEBUG:boto:Data: 2024-04-12T03:14:58.402 DEBUG:boto:Headers: {} 2024-04-12T03:14:58.402 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:14:58.402 DEBUG:boto:Port: 8001 2024-04-12T03:14:58.402 DEBUG:boto:Params: {} 2024-04-12T03:14:58.402 DEBUG:boto:Token: None 2024-04-12T03:14:58.402 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:14:58 GMT /dkyges-6/ 2024-04-12T03:14:58.403 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hw4Z35eGg+lVOrT6poqOhrKpteM= 2024-04-12T03:14:58.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:14:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hw4Z35eGg+lVOrT6poqOhrKpteM=', 'Content-Length': '0'} 2024-04-12T03:14:58.470 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a7c9a781b104693a-006618a732-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:14:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:14:58.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:58.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:58.470 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:58.525 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:58.526 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:58.559 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:14:58.572 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:14:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:14:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.573 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:14:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891698.437986_93.1", 2024-04-12T03:14:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:14:58.437986Z" 2024-04-12T03:14:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:14:58.574 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.574 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:14:58.575 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891698.437986_93.1', 3: ''} 2024-04-12T03:14:58.576 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:14:58.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:58.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:14:58.576 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:14:58.631 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:14:58.631 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:14:58.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:58.664+0000 7f8fe4ace780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:58.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:14:58.665+0000 7f8fe4ace780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:14:58.680 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:14:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:14:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:14:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:14:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:14:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:14:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:14:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:14:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:14:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:14:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:14:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:14:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:14:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891698.423155_90.1", 2024-04-12T03:14:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:14:58.423155Z", 2024-04-12T03:14:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:14:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:14:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:14:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:14:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:14:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:14:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:14:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:14:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:14:58.685 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:14:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:14:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:14:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:14:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:14:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:14:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:14:58.686 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:14:58Z" 2024-04-12T03:14:58.687 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:14:58.688 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891698.423155_90.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:14:58.423155Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:14:58Z" } 2024-04-12T03:14:58.688 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891698.423155_90.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:14:58.423155Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-12T03:14:58.688 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891698.437986_93.1', 3: ''} 2024-04-12T03:14:58.688 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891698.423155_90.1', 3: ''} 2024-04-12T03:14:58.688 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1712891698.437986_93.1 target=1_1712891698.423155_90.1 2024-04-12T03:15:03.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:03.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:03.690 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:15:03.751 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:03.751 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:03.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:15:03.783+0000 7efda647f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:03.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:15:03.784+0000 7efda647f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:03.799 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:15:03.800 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:15:03.800 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:15:03.800 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:15:03.800 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:15:03.800 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:15:03.800 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:15:03.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:15:03.801 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:15:03.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:15:03.801 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:15:03.801 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:15:03.801 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:15:03.801 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:15:03.801 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:15:03.801 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:15:03.801 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:15:03.802 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:15:03.802 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:15:03.802 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:15:03.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:15:03.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:15:03.802 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:15:03.802 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:15:03.802 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:15:03.802 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:15:03.802 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:15:03.803 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:15:03.803 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:15:03.803 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:15:03.803 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:15:03.803 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:15:03.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:15:03.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:15:03.803 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:15:03.803 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:15:03.803 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:15:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891698.437986_93.1", 2024-04-12T03:15:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:15:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:15:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:15:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:14:58.437986Z", 2024-04-12T03:15:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:15:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:15:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:15:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:15:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:15:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:15:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:15:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:15:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:15:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:15:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:15:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:15:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:15:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:15:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:15:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:15:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:15:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:15:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:15:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:15:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:15:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:15:03Z" 2024-04-12T03:15:03.809 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:15:03.811 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891698.437986_93.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:14:58.437986Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:15:03Z" } 2024-04-12T03:15:03.811 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891698.437986_93.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:14:58.437986Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-12T03:15:03.811 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891698.437986_93.1', 3: ''} 2024-04-12T03:15:03.811 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891698.437986_93.1', 3: ''} 2024-04-12T03:15:03.811 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:15:03.811 DEBUG:boto:path=/dkyges-5/ 2024-04-12T03:15:03.811 DEBUG:boto:auth_path=/dkyges-5/ 2024-04-12T03:15:03.812 DEBUG:boto:Method: HEAD 2024-04-12T03:15:03.812 DEBUG:boto:Path: /dkyges-5/ 2024-04-12T03:15:03.812 DEBUG:boto:Data: 2024-04-12T03:15:03.812 DEBUG:boto:Headers: {} 2024-04-12T03:15:03.812 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:15:03.812 DEBUG:boto:Port: 8000 2024-04-12T03:15:03.812 DEBUG:boto:Params: {} 2024-04-12T03:15:03.812 DEBUG:boto:Token: None 2024-04-12T03:15:03.812 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:15:03 GMT /dkyges-5/ 2024-04-12T03:15:03.813 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:p+eye6Gj7UY+eQJzHU63OZKYkj0= 2024-04-12T03:15:03.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p+eye6Gj7UY+eQJzHU63OZKYkj0=', 'Content-Length': '0'} 2024-04-12T03:15:03.814 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000de09a85d1ff5f7c5-006618a737-110f-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:03.814 DEBUG:boto:path=/dkyges-6/ 2024-04-12T03:15:03.814 DEBUG:boto:auth_path=/dkyges-6/ 2024-04-12T03:15:03.815 DEBUG:boto:Method: HEAD 2024-04-12T03:15:03.815 DEBUG:boto:Path: /dkyges-6/ 2024-04-12T03:15:03.815 DEBUG:boto:Data: 2024-04-12T03:15:03.815 DEBUG:boto:Headers: {} 2024-04-12T03:15:03.815 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:15:03.815 DEBUG:boto:Port: 8000 2024-04-12T03:15:03.815 DEBUG:boto:Params: {} 2024-04-12T03:15:03.815 DEBUG:boto:Token: None 2024-04-12T03:15:03.815 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:15:03 GMT /dkyges-6/ 2024-04-12T03:15:03.815 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7G25+cw0kgOWXmyqRjf3kpEOKa0= 2024-04-12T03:15:03.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7G25+cw0kgOWXmyqRjf3kpEOKa0=', 'Content-Length': '0'} 2024-04-12T03:15:03.816 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx000008b75508858217b8c-006618a737-110f-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:03.817 DEBUG:boto:path=/dkyges-5/ 2024-04-12T03:15:03.817 DEBUG:boto:auth_path=/dkyges-5/ 2024-04-12T03:15:03.817 DEBUG:boto:Method: HEAD 2024-04-12T03:15:03.817 DEBUG:boto:Path: /dkyges-5/ 2024-04-12T03:15:03.817 DEBUG:boto:Data: 2024-04-12T03:15:03.817 DEBUG:boto:Headers: {} 2024-04-12T03:15:03.817 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:15:03.817 DEBUG:boto:Port: 8001 2024-04-12T03:15:03.817 DEBUG:boto:Params: {} 2024-04-12T03:15:03.817 DEBUG:boto:Token: None 2024-04-12T03:15:03.818 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:15:03 GMT /dkyges-5/ 2024-04-12T03:15:03.818 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:p+eye6Gj7UY+eQJzHU63OZKYkj0= 2024-04-12T03:15:03.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p+eye6Gj7UY+eQJzHU63OZKYkj0=', 'Content-Length': '0'} 2024-04-12T03:15:03.819 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000563c09df26695682-006618a737-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:03.819 DEBUG:boto:path=/dkyges-6/ 2024-04-12T03:15:03.819 DEBUG:boto:auth_path=/dkyges-6/ 2024-04-12T03:15:03.820 DEBUG:boto:Method: HEAD 2024-04-12T03:15:03.820 DEBUG:boto:Path: /dkyges-6/ 2024-04-12T03:15:03.820 DEBUG:boto:Data: 2024-04-12T03:15:03.820 DEBUG:boto:Headers: {} 2024-04-12T03:15:03.820 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:15:03.820 DEBUG:boto:Port: 8001 2024-04-12T03:15:03.820 DEBUG:boto:Params: {} 2024-04-12T03:15:03.820 DEBUG:boto:Token: None 2024-04-12T03:15:03.820 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:15:03 GMT /dkyges-6/ 2024-04-12T03:15:03.820 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7G25+cw0kgOWXmyqRjf3kpEOKa0= 2024-04-12T03:15:03.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7G25+cw0kgOWXmyqRjf3kpEOKa0=', 'Content-Length': '0'} 2024-04-12T03:15:03.822 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000e0c9f80854518305-006618a737-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:03.822 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2024-04-12T03:15:03.823 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-7 2024-04-12T03:15:03.823 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:15:03.823 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:15:03.823 DEBUG:boto:Method: PUT 2024-04-12T03:15:03.823 DEBUG:boto:Path: /dkyges-7/ 2024-04-12T03:15:03.823 DEBUG:boto:Data: 2024-04-12T03:15:03.823 DEBUG:boto:Headers: {} 2024-04-12T03:15:03.823 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:15:03.823 DEBUG:boto:Port: 8000 2024-04-12T03:15:03.823 DEBUG:boto:Params: {} 2024-04-12T03:15:03.824 DEBUG:boto:Token: None 2024-04-12T03:15:03.824 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:15:03 GMT /dkyges-7/ 2024-04-12T03:15:03.824 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vxEr3TIhijq5O67IvPoLi2c4rVU= 2024-04-12T03:15:03.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vxEr3TIhijq5O67IvPoLi2c4rVU=', 'Content-Length': '0'} 2024-04-12T03:15:03.838 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000acb997c76653b487-006618a737-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:03.838 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-8 2024-04-12T03:15:03.839 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:15:03.839 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:15:03.839 DEBUG:boto:Method: PUT 2024-04-12T03:15:03.839 DEBUG:boto:Path: /dkyges-8/ 2024-04-12T03:15:03.839 DEBUG:boto:Data: 2024-04-12T03:15:03.839 DEBUG:boto:Headers: {} 2024-04-12T03:15:03.839 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:15:03.839 DEBUG:boto:Port: 8001 2024-04-12T03:15:03.839 DEBUG:boto:Params: {} 2024-04-12T03:15:03.839 DEBUG:boto:Token: None 2024-04-12T03:15:03.840 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:15:03 GMT /dkyges-8/ 2024-04-12T03:15:03.840 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WWXRnoHxpyAiTLSlt/ITbToyPWo= 2024-04-12T03:15:03.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WWXRnoHxpyAiTLSlt/ITbToyPWo=', 'Content-Length': '0'} 2024-04-12T03:15:03.864 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e8e04964166723e9-006618a737-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:03.864 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:15:03.865 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:15:03.865 DEBUG:boto:Method: HEAD 2024-04-12T03:15:03.865 DEBUG:boto:Path: /dkyges-7/ 2024-04-12T03:15:03.865 DEBUG:boto:Data: 2024-04-12T03:15:03.865 DEBUG:boto:Headers: {} 2024-04-12T03:15:03.865 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:15:03.865 DEBUG:boto:Port: 8000 2024-04-12T03:15:03.865 DEBUG:boto:Params: {} 2024-04-12T03:15:03.865 DEBUG:boto:Token: None 2024-04-12T03:15:03.865 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:15:03 GMT /dkyges-7/ 2024-04-12T03:15:03.865 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8FagovkJnKR4AWq2jl/DyoADA4o= 2024-04-12T03:15:03.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8FagovkJnKR4AWq2jl/DyoADA4o=', 'Content-Length': '0'} 2024-04-12T03:15:03.868 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b6e86c0dcea115a-006618a737-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:03.868 DEBUG:boto:path=/dkyges-7/myobj 2024-04-12T03:15:03.868 DEBUG:boto:auth_path=/dkyges-7/myobj 2024-04-12T03:15:03.868 DEBUG:boto:Method: PUT 2024-04-12T03:15:03.868 DEBUG:boto:Path: /dkyges-7/myobj 2024-04-12T03:15:03.869 DEBUG:boto:Data: 2024-04-12T03:15:03.869 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:15:03.869 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:15:03.869 DEBUG:boto:Port: 8000 2024-04-12T03:15:03.869 DEBUG:boto:Params: {} 2024-04-12T03:15:03.869 DEBUG:boto:Token: None 2024-04-12T03:15:03.869 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:15:03 GMT /dkyges-7/myobj 2024-04-12T03:15:03.869 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SBoVj6OE8Qj0D4hsRt3GHBfRXb0= 2024-04-12T03:15:03.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:15:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SBoVj6OE8Qj0D4hsRt3GHBfRXb0='} 2024-04-12T03:15:03.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb0619abaeea4357-006618a737-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:03.881 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:15:03.881 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:15:03.881 DEBUG:boto:Method: HEAD 2024-04-12T03:15:03.881 DEBUG:boto:Path: /dkyges-7/ 2024-04-12T03:15:03.881 DEBUG:boto:Data: 2024-04-12T03:15:03.881 DEBUG:boto:Headers: {} 2024-04-12T03:15:03.881 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:15:03.881 DEBUG:boto:Port: 8000 2024-04-12T03:15:03.881 DEBUG:boto:Params: {} 2024-04-12T03:15:03.882 DEBUG:boto:Token: None 2024-04-12T03:15:03.882 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:15:03 GMT /dkyges-7/ 2024-04-12T03:15:03.882 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8FagovkJnKR4AWq2jl/DyoADA4o= 2024-04-12T03:15:03.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8FagovkJnKR4AWq2jl/DyoADA4o=', 'Content-Length': '0'} 2024-04-12T03:15:03.884 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e14750d59cfe576d-006618a737-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:03.884 DEBUG:boto:path=/dkyges-7/_myobj 2024-04-12T03:15:03.885 DEBUG:boto:auth_path=/dkyges-7/_myobj 2024-04-12T03:15:03.885 DEBUG:boto:Method: PUT 2024-04-12T03:15:03.885 DEBUG:boto:Path: /dkyges-7/_myobj 2024-04-12T03:15:03.885 DEBUG:boto:Data: 2024-04-12T03:15:03.885 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:15:03.885 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:15:03.885 DEBUG:boto:Port: 8000 2024-04-12T03:15:03.885 DEBUG:boto:Params: {} 2024-04-12T03:15:03.885 DEBUG:boto:Token: None 2024-04-12T03:15:03.885 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:15:03 GMT /dkyges-7/_myobj 2024-04-12T03:15:03.886 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:79RBimn8BLTXJSuhMZOnGfaDHtU= 2024-04-12T03:15:03.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:15:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:79RBimn8BLTXJSuhMZOnGfaDHtU='} 2024-04-12T03:15:03.930 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3c510442e4d3b6c-006618a737-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:03.931 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:15:03.931 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:15:03.931 DEBUG:boto:Method: HEAD 2024-04-12T03:15:03.931 DEBUG:boto:Path: /dkyges-7/ 2024-04-12T03:15:03.931 DEBUG:boto:Data: 2024-04-12T03:15:03.931 DEBUG:boto:Headers: {} 2024-04-12T03:15:03.931 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:15:03.931 DEBUG:boto:Port: 8000 2024-04-12T03:15:03.931 DEBUG:boto:Params: {} 2024-04-12T03:15:03.932 DEBUG:boto:Token: None 2024-04-12T03:15:03.932 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:15:03 GMT /dkyges-7/ 2024-04-12T03:15:03.932 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8FagovkJnKR4AWq2jl/DyoADA4o= 2024-04-12T03:15:03.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8FagovkJnKR4AWq2jl/DyoADA4o=', 'Content-Length': '0'} 2024-04-12T03:15:03.934 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ccaba542ff4901d-006618a737-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:03.935 DEBUG:boto:path=/dkyges-7/%3A 2024-04-12T03:15:03.935 DEBUG:boto:auth_path=/dkyges-7/%3A 2024-04-12T03:15:03.935 DEBUG:boto:Method: PUT 2024-04-12T03:15:03.935 DEBUG:boto:Path: /dkyges-7/%3A 2024-04-12T03:15:03.935 DEBUG:boto:Data: 2024-04-12T03:15:03.935 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:15:03.935 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:15:03.935 DEBUG:boto:Port: 8000 2024-04-12T03:15:03.935 DEBUG:boto:Params: {} 2024-04-12T03:15:03.935 DEBUG:boto:Token: None 2024-04-12T03:15:03.936 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:15:03 GMT /dkyges-7/%3A 2024-04-12T03:15:03.936 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q4/ivOyfpm6xD6Rlg85He1J1z84= 2024-04-12T03:15:03.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:15:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q4/ivOyfpm6xD6Rlg85He1J1z84='} 2024-04-12T03:15:03.945 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef4c6451436e7ba7-006618a737-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:03.945 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:15:03.945 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:15:03.946 DEBUG:boto:Method: HEAD 2024-04-12T03:15:03.946 DEBUG:boto:Path: /dkyges-7/ 2024-04-12T03:15:03.946 DEBUG:boto:Data: 2024-04-12T03:15:03.946 DEBUG:boto:Headers: {} 2024-04-12T03:15:03.946 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:15:03.946 DEBUG:boto:Port: 8000 2024-04-12T03:15:03.946 DEBUG:boto:Params: {} 2024-04-12T03:15:03.946 DEBUG:boto:Token: None 2024-04-12T03:15:03.946 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:15:03 GMT /dkyges-7/ 2024-04-12T03:15:03.946 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8FagovkJnKR4AWq2jl/DyoADA4o= 2024-04-12T03:15:03.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8FagovkJnKR4AWq2jl/DyoADA4o=', 'Content-Length': '0'} 2024-04-12T03:15:03.949 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ddcebaf6e771e467-006618a737-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:03.949 DEBUG:boto:path=/dkyges-7/%26 2024-04-12T03:15:03.949 DEBUG:boto:auth_path=/dkyges-7/%26 2024-04-12T03:15:03.949 DEBUG:boto:Method: PUT 2024-04-12T03:15:03.949 DEBUG:boto:Path: /dkyges-7/%26 2024-04-12T03:15:03.949 DEBUG:boto:Data: 2024-04-12T03:15:03.949 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:15:03.949 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:15:03.950 DEBUG:boto:Port: 8000 2024-04-12T03:15:03.950 DEBUG:boto:Params: {} 2024-04-12T03:15:03.950 DEBUG:boto:Token: None 2024-04-12T03:15:03.950 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:15:03 GMT /dkyges-7/%26 2024-04-12T03:15:03.950 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:57zIcXejNsPEetmvjjhgiecvo2c= 2024-04-12T03:15:03.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:15:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:57zIcXejNsPEetmvjjhgiecvo2c='} 2024-04-12T03:15:03.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083ffada4fdc5f3a2-006618a737-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:03.995 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:15:03.995 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:15:03.995 DEBUG:boto:Method: HEAD 2024-04-12T03:15:03.995 DEBUG:boto:Path: /dkyges-8/ 2024-04-12T03:15:03.995 DEBUG:boto:Data: 2024-04-12T03:15:03.995 DEBUG:boto:Headers: {} 2024-04-12T03:15:03.995 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:15:03.995 DEBUG:boto:Port: 8001 2024-04-12T03:15:03.995 DEBUG:boto:Params: {} 2024-04-12T03:15:03.995 DEBUG:boto:Token: None 2024-04-12T03:15:03.996 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:15:03 GMT /dkyges-8/ 2024-04-12T03:15:03.996 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3Zg5mdCK1I/BaejPJzYMMpTg9wA= 2024-04-12T03:15:03.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3Zg5mdCK1I/BaejPJzYMMpTg9wA=', 'Content-Length': '0'} 2024-04-12T03:15:03.998 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007b467960b27fce9-006618a737-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:03.998 DEBUG:boto:path=/dkyges-8/myobj 2024-04-12T03:15:03.999 DEBUG:boto:auth_path=/dkyges-8/myobj 2024-04-12T03:15:03.999 DEBUG:boto:Method: PUT 2024-04-12T03:15:03.999 DEBUG:boto:Path: /dkyges-8/myobj 2024-04-12T03:15:03.999 DEBUG:boto:Data: 2024-04-12T03:15:03.999 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:15:03.999 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:15:03.999 DEBUG:boto:Port: 8001 2024-04-12T03:15:03.999 DEBUG:boto:Params: {} 2024-04-12T03:15:03.999 DEBUG:boto:Token: None 2024-04-12T03:15:03.999 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:15:03 GMT /dkyges-8/myobj 2024-04-12T03:15:04.000 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d3LxT3J3CPpB7PAwV7W6vuSvYJk= 2024-04-12T03:15:04.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:15:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d3LxT3J3CPpB7PAwV7W6vuSvYJk='} 2024-04-12T03:15:04.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000924027cd1aebc9d0-006618a738-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:04.043 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:15:04.043 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:15:04.043 DEBUG:boto:Method: HEAD 2024-04-12T03:15:04.043 DEBUG:boto:Path: /dkyges-8/ 2024-04-12T03:15:04.043 DEBUG:boto:Data: 2024-04-12T03:15:04.043 DEBUG:boto:Headers: {} 2024-04-12T03:15:04.043 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:15:04.043 DEBUG:boto:Port: 8001 2024-04-12T03:15:04.043 DEBUG:boto:Params: {} 2024-04-12T03:15:04.043 DEBUG:boto:Token: None 2024-04-12T03:15:04.044 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:15:04 GMT /dkyges-8/ 2024-04-12T03:15:04.044 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IWdGOvtsmNzrbuV43NzIqjBsJ3U= 2024-04-12T03:15:04.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IWdGOvtsmNzrbuV43NzIqjBsJ3U=', 'Content-Length': '0'} 2024-04-12T03:15:04.046 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085ee63e7340a6ab1-006618a738-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:04.046 DEBUG:boto:path=/dkyges-8/_myobj 2024-04-12T03:15:04.046 DEBUG:boto:auth_path=/dkyges-8/_myobj 2024-04-12T03:15:04.046 DEBUG:boto:Method: PUT 2024-04-12T03:15:04.047 DEBUG:boto:Path: /dkyges-8/_myobj 2024-04-12T03:15:04.047 DEBUG:boto:Data: 2024-04-12T03:15:04.047 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:15:04.047 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:15:04.047 DEBUG:boto:Port: 8001 2024-04-12T03:15:04.047 DEBUG:boto:Params: {} 2024-04-12T03:15:04.047 DEBUG:boto:Token: None 2024-04-12T03:15:04.047 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:15:04 GMT /dkyges-8/_myobj 2024-04-12T03:15:04.047 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KkOMqHdgXfmBwrHZKszD9080AkQ= 2024-04-12T03:15:04.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:15:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KkOMqHdgXfmBwrHZKszD9080AkQ='} 2024-04-12T03:15:04.057 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce54d5828dcae35a-006618a738-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:04.057 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:15:04.057 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:15:04.057 DEBUG:boto:Method: HEAD 2024-04-12T03:15:04.058 DEBUG:boto:Path: /dkyges-8/ 2024-04-12T03:15:04.058 DEBUG:boto:Data: 2024-04-12T03:15:04.058 DEBUG:boto:Headers: {} 2024-04-12T03:15:04.058 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:15:04.058 DEBUG:boto:Port: 8001 2024-04-12T03:15:04.058 DEBUG:boto:Params: {} 2024-04-12T03:15:04.058 DEBUG:boto:Token: None 2024-04-12T03:15:04.058 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:15:04 GMT /dkyges-8/ 2024-04-12T03:15:04.058 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IWdGOvtsmNzrbuV43NzIqjBsJ3U= 2024-04-12T03:15:04.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IWdGOvtsmNzrbuV43NzIqjBsJ3U=', 'Content-Length': '0'} 2024-04-12T03:15:04.061 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055d8b46334f46dea-006618a738-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:04.061 DEBUG:boto:path=/dkyges-8/%3A 2024-04-12T03:15:04.061 DEBUG:boto:auth_path=/dkyges-8/%3A 2024-04-12T03:15:04.061 DEBUG:boto:Method: PUT 2024-04-12T03:15:04.061 DEBUG:boto:Path: /dkyges-8/%3A 2024-04-12T03:15:04.061 DEBUG:boto:Data: 2024-04-12T03:15:04.061 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:15:04.061 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:15:04.061 DEBUG:boto:Port: 8001 2024-04-12T03:15:04.062 DEBUG:boto:Params: {} 2024-04-12T03:15:04.062 DEBUG:boto:Token: None 2024-04-12T03:15:04.062 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:15:04 GMT /dkyges-8/%3A 2024-04-12T03:15:04.062 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H4fwzMSDLCTUKzOE25/1D4qwOqI= 2024-04-12T03:15:04.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:15:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H4fwzMSDLCTUKzOE25/1D4qwOqI='} 2024-04-12T03:15:04.075 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000765776a6f70182e7-006618a738-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:04.075 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:15:04.075 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:15:04.075 DEBUG:boto:Method: HEAD 2024-04-12T03:15:04.075 DEBUG:boto:Path: /dkyges-8/ 2024-04-12T03:15:04.075 DEBUG:boto:Data: 2024-04-12T03:15:04.075 DEBUG:boto:Headers: {} 2024-04-12T03:15:04.076 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:15:04.076 DEBUG:boto:Port: 8001 2024-04-12T03:15:04.076 DEBUG:boto:Params: {} 2024-04-12T03:15:04.076 DEBUG:boto:Token: None 2024-04-12T03:15:04.076 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:15:04 GMT /dkyges-8/ 2024-04-12T03:15:04.076 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IWdGOvtsmNzrbuV43NzIqjBsJ3U= 2024-04-12T03:15:04.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:15:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IWdGOvtsmNzrbuV43NzIqjBsJ3U=', 'Content-Length': '0'} 2024-04-12T03:15:04.078 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009caee47d497cb109-006618a738-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:04.079 DEBUG:boto:path=/dkyges-8/%26 2024-04-12T03:15:04.079 DEBUG:boto:auth_path=/dkyges-8/%26 2024-04-12T03:15:04.079 DEBUG:boto:Method: PUT 2024-04-12T03:15:04.079 DEBUG:boto:Path: /dkyges-8/%26 2024-04-12T03:15:04.079 DEBUG:boto:Data: 2024-04-12T03:15:04.079 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:15:04.079 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:15:04.079 DEBUG:boto:Port: 8001 2024-04-12T03:15:04.079 DEBUG:boto:Params: {} 2024-04-12T03:15:04.079 DEBUG:boto:Token: None 2024-04-12T03:15:04.080 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:15:04 GMT /dkyges-8/%26 2024-04-12T03:15:04.080 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WtPLCBxjtmBo7/7N5rdeMGKZBB8= 2024-04-12T03:15:04.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:15:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WtPLCBxjtmBo7/7N5rdeMGKZBB8='} 2024-04-12T03:15:04.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008346279b588e15ff-006618a738-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:15:04.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:04.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:04.127 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:15:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:04.185 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:04.219 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:15:04.234 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:15:04.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:15:04.234 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:15:04.234 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:15:04.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:15:04.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:15:04.235 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:15:04.235 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:15:04.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:15:04.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:15:04.236 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891703.853772_101.1", 2024-04-12T03:15:04.236 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:15:03.853772Z" 2024-04-12T03:15:04.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:15:04.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:15:04.237 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:15:04.237 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:15:04.237 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:15:04.237 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:15:04.239 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891703.853772_101.1', 3: ''} 2024-04-12T03:15:04.239 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:15:04.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:04.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:15:04.240 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:15:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:04.295 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:04.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:15:04.326+0000 7fa21eacf780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:04.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:15:04.327+0000 7fa21eacf780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:15:04.342 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:15:04.342 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:15:04.342 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:15:04.342 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:15:04.342 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:15:04.343 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:15:04.343 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:15:04.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:15:04.343 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:15:04.343 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:15:04.343 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:15:04.343 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:15:04.343 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:15:04.343 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:15:04.343 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:15:04.343 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:15:04.344 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:15:04.344 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:15:04.344 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:15:04.344 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:15:04.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:15:04.344 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:15:04.344 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:15:04.344 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:15:04.344 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:15:04.344 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:15:04.344 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:15:04.345 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:15:04.345 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:15:04.345 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:15:04.345 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:15:04.345 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:15:04.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:15:04.345 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:15:04.345 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:15:04.345 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:15:04.346 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:15:04.346 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891703.853772_101.1", 2024-04-12T03:15:04.346 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:15:04.346 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:15:04.346 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:15:04.346 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:15:03.853772Z", 2024-04-12T03:15:04.346 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:15:04.346 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:15:04.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:15:04.346 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:15:04.347 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:15:04.347 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:15:04.347 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:15:04.347 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:15:04.347 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:15:04.347 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:15:04.347 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:15:04.347 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:15:04.347 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:15:04.347 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:15:04.347 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:15:04.348 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:15:04.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:15:04.348 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:15:04.348 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:15:04.348 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:15:04.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:15:04.348 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:15:04Z" 2024-04-12T03:15:04.348 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:15:04.349 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891703.853772_101.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:15:03.853772Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:15:04Z" } 2024-04-12T03:15:04.349 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891703.853772_101.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:15:03.853772Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2024-04-12T03:15:04.349 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891703.853772_101.1', 3: ''} 2024-04-12T03:15:04.349 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891703.853772_101.1', 3: ''} 2024-04-12T03:15:04.350 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:15:04.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:15:04.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:15:04.350 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-7 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:15:04.412 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:15:04.412 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:15:04.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:15:04.449+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:15:09.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:15:09.450+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:15:14.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:15:14.450+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:15:19.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:15:19.451+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:15:24.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:15:24.452+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:15:29.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:15:29.452+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:15:34.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:15:34.452+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:15:39.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:15:39.453+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:15:44.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:15:44.454+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:15:49.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:15:49.454+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:15:54.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:15:54.455+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:15:59.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:15:59.456+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:16:04.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:16:04.456+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:16:09.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:16:09.457+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:16:14.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:16:14.457+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:16:19.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:16:19.458+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:16:24.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:16:24.459+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:16:29.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:16:29.459+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:16:34.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:16:34.460+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:16:39.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:16:39.461+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:16:44.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:16:44.461+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:16:49.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:16:49.461+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:16:54.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:16:54.462+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:16:59.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:16:59.463+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:17:04.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:17:04.464+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:17:09.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:17:09.464+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:17:14.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:17:14.465+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:17:19.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:17:19.466+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:17:24.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:17:24.466+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:17:29.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:17:29.467+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:17:34.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:17:34.467+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:17:39.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:17:39.468+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:17:44.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:17:44.469+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:17:49.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:17:49.471+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:17:54.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:17:54.470+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:17:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:17:59.471+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:18:04.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:04.471+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:18:09.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:09.472+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:18:14.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:14.473+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:18:19.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:19.473+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:18:24.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:24.474+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:18:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:29.475+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:18:34.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:34.475+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:18:39.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:39.476+0000 7fb167551780 1 waiting to reach incremental sync.. 2024-04-12T03:18:44.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:44.478+0000 7fb167551780 1 bucket sync caught up with source: 2024-04-12T03:18:44.479 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.4.6, , 00000000001.4.6, , , , 00000000001.7.6, 00000000001.3.6, , ] 2024-04-12T03:18:44.480 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.4.6, , 00000000001.4.6, , , , 00000000001.7.6, 00000000001.3.6, , ] 2024-04-12T03:18:44.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:44.478+0000 7fb167551780 0 bucket checkpoint complete 2024-04-12T03:18:44.496 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-7 zones={a2, a1} 2024-04-12T03:18:44.496 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.496 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.497 DEBUG:boto:Method: HEAD 2024-04-12T03:18:44.497 DEBUG:boto:Path: /dkyges-7/ 2024-04-12T03:18:44.497 DEBUG:boto:Data: 2024-04-12T03:18:44.497 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.497 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.497 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.497 DEBUG:boto:Params: {} 2024-04-12T03:18:44.498 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:18:44.498 DEBUG:boto:Token: None 2024-04-12T03:18:44.498 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/ 2024-04-12T03:18:44.498 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QJfCIRNcpNbnCO/YsqN3jtR2YXE= 2024-04-12T03:18:44.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QJfCIRNcpNbnCO/YsqN3jtR2YXE=', 'Content-Length': '0'} 2024-04-12T03:18:44.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bb174b35003f471-006618a814-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.512 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.512 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.512 DEBUG:boto:Method: HEAD 2024-04-12T03:18:44.512 DEBUG:boto:Path: /dkyges-7/ 2024-04-12T03:18:44.513 DEBUG:boto:Data: 2024-04-12T03:18:44.513 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.513 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.513 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.513 DEBUG:boto:Params: {} 2024-04-12T03:18:44.513 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:18:44.513 DEBUG:boto:Token: None 2024-04-12T03:18:44.513 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/ 2024-04-12T03:18:44.514 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QJfCIRNcpNbnCO/YsqN3jtR2YXE= 2024-04-12T03:18:44.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QJfCIRNcpNbnCO/YsqN3jtR2YXE=', 'Content-Length': '0'} 2024-04-12T03:18:44.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044a92445667517f7-006618a814-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.526 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:18:44.526 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.526 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.526 DEBUG:boto:path=/dkyges-7/?versions&max-keys=999 2024-04-12T03:18:44.527 DEBUG:boto:auth_path=/dkyges-7/?versions&max-keys=999 2024-04-12T03:18:44.527 DEBUG:boto:Method: GET 2024-04-12T03:18:44.527 DEBUG:boto:Path: /dkyges-7/?versions&max-keys=999 2024-04-12T03:18:44.527 DEBUG:boto:Data: 2024-04-12T03:18:44.527 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.527 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.527 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.527 DEBUG:boto:Params: {} 2024-04-12T03:18:44.527 DEBUG:boto:Token: None 2024-04-12T03:18:44.528 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/?versions 2024-04-12T03:18:44.528 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nhswdsfhN+s5jRe9vbV6DCojSIg= 2024-04-12T03:18:44.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nhswdsfhN+s5jRe9vbV6DCojSIg=', 'Content-Length': '0'} 2024-04-12T03:18:44.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d8a9bd177cfac32-006618a814-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.530 DEBUG:boto:b'dkyges-7999false&nulltrue2024-04-12T03:15:03.952Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-12T03:15:03.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-12T03:15:03.887Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-12T03:15:03.872Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:44.533 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-12T03:18:44.533 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-12T03:18:44.533 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-12T03:18:44.533 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-12T03:18:44.533 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:18:44.533 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.534 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.534 DEBUG:boto:path=/dkyges-7/?versions&max-keys=999 2024-04-12T03:18:44.534 DEBUG:boto:auth_path=/dkyges-7/?versions&max-keys=999 2024-04-12T03:18:44.534 DEBUG:boto:Method: GET 2024-04-12T03:18:44.534 DEBUG:boto:Path: /dkyges-7/?versions&max-keys=999 2024-04-12T03:18:44.534 DEBUG:boto:Data: 2024-04-12T03:18:44.534 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.534 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.534 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.534 DEBUG:boto:Params: {} 2024-04-12T03:18:44.535 DEBUG:boto:Token: None 2024-04-12T03:18:44.535 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/?versions 2024-04-12T03:18:44.535 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nhswdsfhN+s5jRe9vbV6DCojSIg= 2024-04-12T03:18:44.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nhswdsfhN+s5jRe9vbV6DCojSIg=', 'Content-Length': '0'} 2024-04-12T03:18:44.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a7231fef4983d13-006618a814-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.538 DEBUG:boto:b'dkyges-7999false&nulltrue2024-04-12T03:15:03.952Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-12T03:15:03.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-12T03:15:03.887Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-12T03:15:03.872Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:44.538 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-12T03:18:44.538 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-12T03:18:44.538 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-12T03:18:44.538 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-12T03:18:44.539 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.539 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.539 DEBUG:boto:path=/dkyges-7/?versions&max-keys=999 2024-04-12T03:18:44.539 DEBUG:boto:auth_path=/dkyges-7/?versions&max-keys=999 2024-04-12T03:18:44.539 DEBUG:boto:Method: GET 2024-04-12T03:18:44.539 DEBUG:boto:Path: /dkyges-7/?versions&max-keys=999 2024-04-12T03:18:44.539 DEBUG:boto:Data: 2024-04-12T03:18:44.539 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.539 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.539 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.539 DEBUG:boto:Params: {} 2024-04-12T03:18:44.540 DEBUG:boto:Token: None 2024-04-12T03:18:44.540 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/?versions 2024-04-12T03:18:44.540 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nhswdsfhN+s5jRe9vbV6DCojSIg= 2024-04-12T03:18:44.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nhswdsfhN+s5jRe9vbV6DCojSIg=', 'Content-Length': '0'} 2024-04-12T03:18:44.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5a392e03575247e-006618a814-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.543 DEBUG:boto:b'dkyges-7999false&nulltrue2024-04-12T03:15:03.952Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-12T03:15:03.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-12T03:15:03.887Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-12T03:15:03.872Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:44.543 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.543 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.543 DEBUG:boto:path=/dkyges-7/?versions&max-keys=999 2024-04-12T03:18:44.544 DEBUG:boto:auth_path=/dkyges-7/?versions&max-keys=999 2024-04-12T03:18:44.544 DEBUG:boto:Method: GET 2024-04-12T03:18:44.544 DEBUG:boto:Path: /dkyges-7/?versions&max-keys=999 2024-04-12T03:18:44.544 DEBUG:boto:Data: 2024-04-12T03:18:44.544 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.544 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.544 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.544 DEBUG:boto:Params: {} 2024-04-12T03:18:44.544 DEBUG:boto:Token: None 2024-04-12T03:18:44.545 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/?versions 2024-04-12T03:18:44.545 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nhswdsfhN+s5jRe9vbV6DCojSIg= 2024-04-12T03:18:44.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nhswdsfhN+s5jRe9vbV6DCojSIg=', 'Content-Length': '0'} 2024-04-12T03:18:44.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019dad11e72436a7c-006618a814-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.547 DEBUG:boto:b'dkyges-7999false&nulltrue2024-04-12T03:15:03.952Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-12T03:15:03.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-12T03:15:03.887Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-12T03:15:03.872Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:44.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-12T03:18:44.548 DEBUG:boto:path=/dkyges-7/%26 2024-04-12T03:18:44.548 DEBUG:boto:auth_path=/dkyges-7/%26 2024-04-12T03:18:44.548 DEBUG:boto:path=/dkyges-7/%26?versionId=null 2024-04-12T03:18:44.548 DEBUG:boto:auth_path=/dkyges-7/%26?versionId=null 2024-04-12T03:18:44.549 DEBUG:boto:Method: GET 2024-04-12T03:18:44.549 DEBUG:boto:Path: /dkyges-7/%26?versionId=null 2024-04-12T03:18:44.549 DEBUG:boto:Data: 2024-04-12T03:18:44.549 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.549 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.549 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.549 DEBUG:boto:Params: {} 2024-04-12T03:18:44.549 DEBUG:boto:Token: None 2024-04-12T03:18:44.549 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%26?versionId=null 2024-04-12T03:18:44.549 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JD0tqS7jiea4M2CtT5UleHLxg98= 2024-04-12T03:18:44.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JD0tqS7jiea4M2CtT5UleHLxg98=', 'Content-Length': '0'} 2024-04-12T03:18:44.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009cf2a1cd86f15a6e-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.598 DEBUG:boto:path=/dkyges-7/%26 2024-04-12T03:18:44.599 DEBUG:boto:auth_path=/dkyges-7/%26 2024-04-12T03:18:44.599 DEBUG:boto:path=/dkyges-7/%26?versionId=null 2024-04-12T03:18:44.599 DEBUG:boto:auth_path=/dkyges-7/%26?versionId=null 2024-04-12T03:18:44.599 DEBUG:boto:Method: GET 2024-04-12T03:18:44.599 DEBUG:boto:Path: /dkyges-7/%26?versionId=null 2024-04-12T03:18:44.599 DEBUG:boto:Data: 2024-04-12T03:18:44.599 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.599 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.599 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.599 DEBUG:boto:Params: {} 2024-04-12T03:18:44.600 DEBUG:boto:Token: None 2024-04-12T03:18:44.600 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%26?versionId=null 2024-04-12T03:18:44.600 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JD0tqS7jiea4M2CtT5UleHLxg98= 2024-04-12T03:18:44.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JD0tqS7jiea4M2CtT5UleHLxg98=', 'Content-Length': '0'} 2024-04-12T03:18:44.602 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e5ab16b0f0260e5-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.603 DEBUG:boto:path=/dkyges-7/%26 2024-04-12T03:18:44.603 DEBUG:boto:auth_path=/dkyges-7/%26 2024-04-12T03:18:44.603 DEBUG:boto:path=/dkyges-7/%26?versionId=null 2024-04-12T03:18:44.603 DEBUG:boto:auth_path=/dkyges-7/%26?versionId=null 2024-04-12T03:18:44.603 DEBUG:boto:Method: HEAD 2024-04-12T03:18:44.603 DEBUG:boto:Path: /dkyges-7/%26?versionId=null 2024-04-12T03:18:44.603 DEBUG:boto:Data: 2024-04-12T03:18:44.603 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.603 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.604 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.604 DEBUG:boto:Params: {} 2024-04-12T03:18:44.604 DEBUG:boto:Token: None 2024-04-12T03:18:44.604 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%26?versionId=null 2024-04-12T03:18:44.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6F8Fs86T7tgqHLqQ0TcvrbQzy08= 2024-04-12T03:18:44.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6F8Fs86T7tgqHLqQ0TcvrbQzy08=', 'Content-Length': '0'} 2024-04-12T03:18:44.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f3b37ae749e1e47-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.607 DEBUG:boto:path=/dkyges-7/%26 2024-04-12T03:18:44.607 DEBUG:boto:auth_path=/dkyges-7/%26 2024-04-12T03:18:44.607 DEBUG:boto:path=/dkyges-7/%26?versionId=null 2024-04-12T03:18:44.607 DEBUG:boto:auth_path=/dkyges-7/%26?versionId=null 2024-04-12T03:18:44.607 DEBUG:boto:Method: HEAD 2024-04-12T03:18:44.607 DEBUG:boto:Path: /dkyges-7/%26?versionId=null 2024-04-12T03:18:44.607 DEBUG:boto:Data: 2024-04-12T03:18:44.607 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.607 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.607 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.607 DEBUG:boto:Params: {} 2024-04-12T03:18:44.608 DEBUG:boto:Token: None 2024-04-12T03:18:44.608 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%26?versionId=null 2024-04-12T03:18:44.608 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6F8Fs86T7tgqHLqQ0TcvrbQzy08= 2024-04-12T03:18:44.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6F8Fs86T7tgqHLqQ0TcvrbQzy08=', 'Content-Length': '0'} 2024-04-12T03:18:44.609 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006616a53f991cf3d0-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-12T03:18:44.610 DEBUG:boto:path=/dkyges-7/%26 2024-04-12T03:18:44.610 DEBUG:boto:auth_path=/dkyges-7/%26 2024-04-12T03:18:44.610 DEBUG:boto:path=/dkyges-7/%26?versionId=null 2024-04-12T03:18:44.610 DEBUG:boto:auth_path=/dkyges-7/%26?versionId=null 2024-04-12T03:18:44.610 DEBUG:boto:Method: GET 2024-04-12T03:18:44.610 DEBUG:boto:Path: /dkyges-7/%26?versionId=null 2024-04-12T03:18:44.610 DEBUG:boto:Data: 2024-04-12T03:18:44.610 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.611 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.611 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.611 DEBUG:boto:Params: {} 2024-04-12T03:18:44.611 DEBUG:boto:Token: None 2024-04-12T03:18:44.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%26?versionId=null 2024-04-12T03:18:44.611 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JD0tqS7jiea4M2CtT5UleHLxg98= 2024-04-12T03:18:44.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JD0tqS7jiea4M2CtT5UleHLxg98=', 'Content-Length': '0'} 2024-04-12T03:18:44.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062c524c8eed0e902-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.613 DEBUG:boto:path=/dkyges-7/%26 2024-04-12T03:18:44.613 DEBUG:boto:auth_path=/dkyges-7/%26 2024-04-12T03:18:44.613 DEBUG:boto:path=/dkyges-7/%26?versionId=null 2024-04-12T03:18:44.613 DEBUG:boto:auth_path=/dkyges-7/%26?versionId=null 2024-04-12T03:18:44.613 DEBUG:boto:Method: GET 2024-04-12T03:18:44.614 DEBUG:boto:Path: /dkyges-7/%26?versionId=null 2024-04-12T03:18:44.614 DEBUG:boto:Data: 2024-04-12T03:18:44.614 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.614 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.614 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.614 DEBUG:boto:Params: {} 2024-04-12T03:18:44.614 DEBUG:boto:Token: None 2024-04-12T03:18:44.614 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%26?versionId=null 2024-04-12T03:18:44.614 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JD0tqS7jiea4M2CtT5UleHLxg98= 2024-04-12T03:18:44.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JD0tqS7jiea4M2CtT5UleHLxg98=', 'Content-Length': '0'} 2024-04-12T03:18:44.616 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ae4dd94838b366ca-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.616 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.617 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.617 DEBUG:boto:path=/dkyges-7/?prefix=%26 2024-04-12T03:18:44.617 DEBUG:boto:auth_path=/dkyges-7/?prefix=%26 2024-04-12T03:18:44.617 DEBUG:boto:Method: GET 2024-04-12T03:18:44.617 DEBUG:boto:Path: /dkyges-7/?prefix=%26 2024-04-12T03:18:44.617 DEBUG:boto:Data: 2024-04-12T03:18:44.617 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.617 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.617 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.617 DEBUG:boto:Params: {} 2024-04-12T03:18:44.617 DEBUG:boto:Token: None 2024-04-12T03:18:44.618 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/ 2024-04-12T03:18:44.618 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY= 2024-04-12T03:18:44.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY=', 'Content-Length': '0'} 2024-04-12T03:18:44.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020b9a34df43a81c3-006618a814-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.620 DEBUG:boto:b'dkyges-7&1000false&2024-04-12T03:15:03.952Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:44.621 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.621 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.621 DEBUG:boto:path=/dkyges-7/?prefix=%26 2024-04-12T03:18:44.621 DEBUG:boto:auth_path=/dkyges-7/?prefix=%26 2024-04-12T03:18:44.621 DEBUG:boto:Method: GET 2024-04-12T03:18:44.621 DEBUG:boto:Path: /dkyges-7/?prefix=%26 2024-04-12T03:18:44.621 DEBUG:boto:Data: 2024-04-12T03:18:44.621 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.621 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.621 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.622 DEBUG:boto:Params: {} 2024-04-12T03:18:44.622 DEBUG:boto:Token: None 2024-04-12T03:18:44.622 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/ 2024-04-12T03:18:44.622 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY= 2024-04-12T03:18:44.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY=', 'Content-Length': '0'} 2024-04-12T03:18:44.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9423b379175e378-006618a814-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.624 DEBUG:boto:b'dkyges-7&1000false&2024-04-12T03:15:03.952Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:44.625 DEBUG:boto:path=/dkyges-7/%26 2024-04-12T03:18:44.625 DEBUG:boto:auth_path=/dkyges-7/%26 2024-04-12T03:18:44.625 DEBUG:boto:Method: HEAD 2024-04-12T03:18:44.625 DEBUG:boto:Path: /dkyges-7/%26 2024-04-12T03:18:44.625 DEBUG:boto:Data: 2024-04-12T03:18:44.625 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.625 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.625 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.626 DEBUG:boto:Params: {} 2024-04-12T03:18:44.626 DEBUG:boto:Token: None 2024-04-12T03:18:44.626 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%26 2024-04-12T03:18:44.626 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LomGU5ab74l79whOGLSJBvHCTEE= 2024-04-12T03:18:44.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LomGU5ab74l79whOGLSJBvHCTEE=', 'Content-Length': '0'} 2024-04-12T03:18:44.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014dd5494ca43091f-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.628 DEBUG:boto:path=/dkyges-7/%26 2024-04-12T03:18:44.628 DEBUG:boto:auth_path=/dkyges-7/%26 2024-04-12T03:18:44.628 DEBUG:boto:Method: HEAD 2024-04-12T03:18:44.629 DEBUG:boto:Path: /dkyges-7/%26 2024-04-12T03:18:44.629 DEBUG:boto:Data: 2024-04-12T03:18:44.629 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.629 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.629 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.629 DEBUG:boto:Params: {} 2024-04-12T03:18:44.629 DEBUG:boto:Token: None 2024-04-12T03:18:44.629 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%26 2024-04-12T03:18:44.629 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LomGU5ab74l79whOGLSJBvHCTEE= 2024-04-12T03:18:44.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LomGU5ab74l79whOGLSJBvHCTEE=', 'Content-Length': '0'} 2024-04-12T03:18:44.634 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c59d5985244e4aa2-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-12T03:18:44.634 DEBUG:boto:path=/dkyges-7/%26 2024-04-12T03:18:44.634 DEBUG:boto:auth_path=/dkyges-7/%26 2024-04-12T03:18:44.634 DEBUG:boto:Method: GET 2024-04-12T03:18:44.635 DEBUG:boto:Path: /dkyges-7/%26 2024-04-12T03:18:44.635 DEBUG:boto:Data: 2024-04-12T03:18:44.635 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.635 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.635 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.635 DEBUG:boto:Params: {} 2024-04-12T03:18:44.635 DEBUG:boto:Token: None 2024-04-12T03:18:44.635 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%26 2024-04-12T03:18:44.635 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kkgb1gQLIAIJWf2jzhCMyxIIhrE= 2024-04-12T03:18:44.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kkgb1gQLIAIJWf2jzhCMyxIIhrE=', 'Content-Length': '0'} 2024-04-12T03:18:44.637 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e367406f91e75d8-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.638 DEBUG:boto:path=/dkyges-7/%26 2024-04-12T03:18:44.638 DEBUG:boto:auth_path=/dkyges-7/%26 2024-04-12T03:18:44.638 DEBUG:boto:Method: GET 2024-04-12T03:18:44.638 DEBUG:boto:Path: /dkyges-7/%26 2024-04-12T03:18:44.638 DEBUG:boto:Data: 2024-04-12T03:18:44.638 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.638 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.638 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.638 DEBUG:boto:Params: {} 2024-04-12T03:18:44.639 DEBUG:boto:Token: None 2024-04-12T03:18:44.639 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%26 2024-04-12T03:18:44.639 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kkgb1gQLIAIJWf2jzhCMyxIIhrE= 2024-04-12T03:18:44.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kkgb1gQLIAIJWf2jzhCMyxIIhrE=', 'Content-Length': '0'} 2024-04-12T03:18:44.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd09a7b4959fef02-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.641 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.641 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.641 DEBUG:boto:path=/dkyges-7/?prefix=%26 2024-04-12T03:18:44.641 DEBUG:boto:auth_path=/dkyges-7/?prefix=%26 2024-04-12T03:18:44.641 DEBUG:boto:Method: GET 2024-04-12T03:18:44.641 DEBUG:boto:Path: /dkyges-7/?prefix=%26 2024-04-12T03:18:44.641 DEBUG:boto:Data: 2024-04-12T03:18:44.641 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.642 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.642 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.642 DEBUG:boto:Params: {} 2024-04-12T03:18:44.642 DEBUG:boto:Token: None 2024-04-12T03:18:44.642 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/ 2024-04-12T03:18:44.642 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY= 2024-04-12T03:18:44.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY=', 'Content-Length': '0'} 2024-04-12T03:18:44.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f633c4c3a8aa366-006618a814-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.645 DEBUG:boto:b'dkyges-7&1000false&2024-04-12T03:15:03.952Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:44.645 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.645 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.645 DEBUG:boto:path=/dkyges-7/?prefix=%26 2024-04-12T03:18:44.646 DEBUG:boto:auth_path=/dkyges-7/?prefix=%26 2024-04-12T03:18:44.646 DEBUG:boto:Method: GET 2024-04-12T03:18:44.646 DEBUG:boto:Path: /dkyges-7/?prefix=%26 2024-04-12T03:18:44.646 DEBUG:boto:Data: 2024-04-12T03:18:44.646 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.646 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.646 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.646 DEBUG:boto:Params: {} 2024-04-12T03:18:44.646 DEBUG:boto:Token: None 2024-04-12T03:18:44.646 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/ 2024-04-12T03:18:44.646 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY= 2024-04-12T03:18:44.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY=', 'Content-Length': '0'} 2024-04-12T03:18:44.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9f9b4b309a8a0b1-006618a814-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.649 DEBUG:boto:b'dkyges-7&1000false&2024-04-12T03:15:03.952Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:44.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-12T03:18:44.649 DEBUG:boto:path=/dkyges-7/%3A 2024-04-12T03:18:44.649 DEBUG:boto:auth_path=/dkyges-7/%3A 2024-04-12T03:18:44.650 DEBUG:boto:path=/dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.650 DEBUG:boto:auth_path=/dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.650 DEBUG:boto:Method: GET 2024-04-12T03:18:44.650 DEBUG:boto:Path: /dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.650 DEBUG:boto:Data: 2024-04-12T03:18:44.650 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.650 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.650 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.650 DEBUG:boto:Params: {} 2024-04-12T03:18:44.650 DEBUG:boto:Token: None 2024-04-12T03:18:44.651 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.651 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KVfpkEFDqkK42ccDZDiuh0g22to= 2024-04-12T03:18:44.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KVfpkEFDqkK42ccDZDiuh0g22to=', 'Content-Length': '0'} 2024-04-12T03:18:44.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbe8aa33205cff72-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.653 DEBUG:boto:path=/dkyges-7/%3A 2024-04-12T03:18:44.653 DEBUG:boto:auth_path=/dkyges-7/%3A 2024-04-12T03:18:44.653 DEBUG:boto:path=/dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.653 DEBUG:boto:auth_path=/dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.653 DEBUG:boto:Method: GET 2024-04-12T03:18:44.653 DEBUG:boto:Path: /dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.653 DEBUG:boto:Data: 2024-04-12T03:18:44.653 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.654 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.654 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.654 DEBUG:boto:Params: {} 2024-04-12T03:18:44.654 DEBUG:boto:Token: None 2024-04-12T03:18:44.654 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.654 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KVfpkEFDqkK42ccDZDiuh0g22to= 2024-04-12T03:18:44.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KVfpkEFDqkK42ccDZDiuh0g22to=', 'Content-Length': '0'} 2024-04-12T03:18:44.656 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e516d8d86994f38e-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.656 DEBUG:boto:path=/dkyges-7/%3A 2024-04-12T03:18:44.657 DEBUG:boto:auth_path=/dkyges-7/%3A 2024-04-12T03:18:44.657 DEBUG:boto:path=/dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.657 DEBUG:boto:auth_path=/dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.657 DEBUG:boto:Method: HEAD 2024-04-12T03:18:44.657 DEBUG:boto:Path: /dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.657 DEBUG:boto:Data: 2024-04-12T03:18:44.657 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.657 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.657 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.657 DEBUG:boto:Params: {} 2024-04-12T03:18:44.658 DEBUG:boto:Token: None 2024-04-12T03:18:44.658 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.658 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2K0FWyV4/ocaqDLlsbTGEE0SIIs= 2024-04-12T03:18:44.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2K0FWyV4/ocaqDLlsbTGEE0SIIs=', 'Content-Length': '0'} 2024-04-12T03:18:44.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005442a170f3e0dab0-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.660 DEBUG:boto:path=/dkyges-7/%3A 2024-04-12T03:18:44.660 DEBUG:boto:auth_path=/dkyges-7/%3A 2024-04-12T03:18:44.660 DEBUG:boto:path=/dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.660 DEBUG:boto:auth_path=/dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.660 DEBUG:boto:Method: HEAD 2024-04-12T03:18:44.660 DEBUG:boto:Path: /dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.660 DEBUG:boto:Data: 2024-04-12T03:18:44.661 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.661 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.661 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.661 DEBUG:boto:Params: {} 2024-04-12T03:18:44.661 DEBUG:boto:Token: None 2024-04-12T03:18:44.661 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.661 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2K0FWyV4/ocaqDLlsbTGEE0SIIs= 2024-04-12T03:18:44.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2K0FWyV4/ocaqDLlsbTGEE0SIIs=', 'Content-Length': '0'} 2024-04-12T03:18:44.663 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b37e039b5e64566-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-12T03:18:44.663 DEBUG:boto:path=/dkyges-7/%3A 2024-04-12T03:18:44.663 DEBUG:boto:auth_path=/dkyges-7/%3A 2024-04-12T03:18:44.664 DEBUG:boto:path=/dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.664 DEBUG:boto:auth_path=/dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.664 DEBUG:boto:Method: GET 2024-04-12T03:18:44.664 DEBUG:boto:Path: /dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.664 DEBUG:boto:Data: 2024-04-12T03:18:44.664 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.664 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.664 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.664 DEBUG:boto:Params: {} 2024-04-12T03:18:44.664 DEBUG:boto:Token: None 2024-04-12T03:18:44.664 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.665 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KVfpkEFDqkK42ccDZDiuh0g22to= 2024-04-12T03:18:44.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KVfpkEFDqkK42ccDZDiuh0g22to=', 'Content-Length': '0'} 2024-04-12T03:18:44.666 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7af50cbaf083683-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.667 DEBUG:boto:path=/dkyges-7/%3A 2024-04-12T03:18:44.667 DEBUG:boto:auth_path=/dkyges-7/%3A 2024-04-12T03:18:44.667 DEBUG:boto:path=/dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.667 DEBUG:boto:auth_path=/dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.667 DEBUG:boto:Method: GET 2024-04-12T03:18:44.667 DEBUG:boto:Path: /dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.667 DEBUG:boto:Data: 2024-04-12T03:18:44.668 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.668 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.668 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.668 DEBUG:boto:Params: {} 2024-04-12T03:18:44.668 DEBUG:boto:Token: None 2024-04-12T03:18:44.668 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%3A?versionId=null 2024-04-12T03:18:44.668 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KVfpkEFDqkK42ccDZDiuh0g22to= 2024-04-12T03:18:44.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KVfpkEFDqkK42ccDZDiuh0g22to=', 'Content-Length': '0'} 2024-04-12T03:18:44.670 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f714f0e1237b4c8-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.714 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.715 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.715 DEBUG:boto:path=/dkyges-7/?prefix=%3A 2024-04-12T03:18:44.715 DEBUG:boto:auth_path=/dkyges-7/?prefix=%3A 2024-04-12T03:18:44.715 DEBUG:boto:Method: GET 2024-04-12T03:18:44.715 DEBUG:boto:Path: /dkyges-7/?prefix=%3A 2024-04-12T03:18:44.715 DEBUG:boto:Data: 2024-04-12T03:18:44.715 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.715 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.715 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.715 DEBUG:boto:Params: {} 2024-04-12T03:18:44.715 DEBUG:boto:Token: None 2024-04-12T03:18:44.716 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/ 2024-04-12T03:18:44.716 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY= 2024-04-12T03:18:44.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY=', 'Content-Length': '0'} 2024-04-12T03:18:44.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f0cd1d7091756e3-006618a814-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.719 DEBUG:boto:b'dkyges-7:1000false:2024-04-12T03:15:03.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:44.719 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.719 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.719 DEBUG:boto:path=/dkyges-7/?prefix=%3A 2024-04-12T03:18:44.719 DEBUG:boto:auth_path=/dkyges-7/?prefix=%3A 2024-04-12T03:18:44.719 DEBUG:boto:Method: GET 2024-04-12T03:18:44.719 DEBUG:boto:Path: /dkyges-7/?prefix=%3A 2024-04-12T03:18:44.720 DEBUG:boto:Data: 2024-04-12T03:18:44.720 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.720 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.720 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.720 DEBUG:boto:Params: {} 2024-04-12T03:18:44.720 DEBUG:boto:Token: None 2024-04-12T03:18:44.720 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/ 2024-04-12T03:18:44.720 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY= 2024-04-12T03:18:44.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY=', 'Content-Length': '0'} 2024-04-12T03:18:44.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a77d6dd51978778f-006618a814-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.723 DEBUG:boto:b'dkyges-7:1000false:2024-04-12T03:15:03.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:44.723 DEBUG:boto:path=/dkyges-7/%3A 2024-04-12T03:18:44.723 DEBUG:boto:auth_path=/dkyges-7/%3A 2024-04-12T03:18:44.723 DEBUG:boto:Method: HEAD 2024-04-12T03:18:44.723 DEBUG:boto:Path: /dkyges-7/%3A 2024-04-12T03:18:44.723 DEBUG:boto:Data: 2024-04-12T03:18:44.724 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.724 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.724 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.724 DEBUG:boto:Params: {} 2024-04-12T03:18:44.724 DEBUG:boto:Token: None 2024-04-12T03:18:44.724 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%3A 2024-04-12T03:18:44.724 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xYIoguIfqmwbTCf1qsQmU5lFsok= 2024-04-12T03:18:44.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xYIoguIfqmwbTCf1qsQmU5lFsok=', 'Content-Length': '0'} 2024-04-12T03:18:44.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099b2e45e7894c803-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.726 DEBUG:boto:path=/dkyges-7/%3A 2024-04-12T03:18:44.726 DEBUG:boto:auth_path=/dkyges-7/%3A 2024-04-12T03:18:44.726 DEBUG:boto:Method: HEAD 2024-04-12T03:18:44.727 DEBUG:boto:Path: /dkyges-7/%3A 2024-04-12T03:18:44.727 DEBUG:boto:Data: 2024-04-12T03:18:44.727 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.727 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.727 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.727 DEBUG:boto:Params: {} 2024-04-12T03:18:44.727 DEBUG:boto:Token: None 2024-04-12T03:18:44.727 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%3A 2024-04-12T03:18:44.727 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xYIoguIfqmwbTCf1qsQmU5lFsok= 2024-04-12T03:18:44.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xYIoguIfqmwbTCf1qsQmU5lFsok=', 'Content-Length': '0'} 2024-04-12T03:18:44.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e93f6c9ce0ed7dc5-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-12T03:18:44.730 DEBUG:boto:path=/dkyges-7/%3A 2024-04-12T03:18:44.730 DEBUG:boto:auth_path=/dkyges-7/%3A 2024-04-12T03:18:44.730 DEBUG:boto:Method: GET 2024-04-12T03:18:44.730 DEBUG:boto:Path: /dkyges-7/%3A 2024-04-12T03:18:44.730 DEBUG:boto:Data: 2024-04-12T03:18:44.730 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.730 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.730 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.731 DEBUG:boto:Params: {} 2024-04-12T03:18:44.731 DEBUG:boto:Token: None 2024-04-12T03:18:44.731 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%3A 2024-04-12T03:18:44.731 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LwtDvjeo8tvfu7cGh3+r1V64EhE= 2024-04-12T03:18:44.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LwtDvjeo8tvfu7cGh3+r1V64EhE=', 'Content-Length': '0'} 2024-04-12T03:18:44.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000506caddd5c63f39e-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.778 DEBUG:boto:path=/dkyges-7/%3A 2024-04-12T03:18:44.779 DEBUG:boto:auth_path=/dkyges-7/%3A 2024-04-12T03:18:44.779 DEBUG:boto:Method: GET 2024-04-12T03:18:44.779 DEBUG:boto:Path: /dkyges-7/%3A 2024-04-12T03:18:44.779 DEBUG:boto:Data: 2024-04-12T03:18:44.779 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.779 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.779 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.779 DEBUG:boto:Params: {} 2024-04-12T03:18:44.780 DEBUG:boto:Token: None 2024-04-12T03:18:44.780 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/%3A 2024-04-12T03:18:44.780 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LwtDvjeo8tvfu7cGh3+r1V64EhE= 2024-04-12T03:18:44.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LwtDvjeo8tvfu7cGh3+r1V64EhE=', 'Content-Length': '0'} 2024-04-12T03:18:44.782 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abdb87ba03ec273c-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.782 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.783 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.783 DEBUG:boto:path=/dkyges-7/?prefix=%3A 2024-04-12T03:18:44.783 DEBUG:boto:auth_path=/dkyges-7/?prefix=%3A 2024-04-12T03:18:44.783 DEBUG:boto:Method: GET 2024-04-12T03:18:44.783 DEBUG:boto:Path: /dkyges-7/?prefix=%3A 2024-04-12T03:18:44.783 DEBUG:boto:Data: 2024-04-12T03:18:44.783 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.783 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.783 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.784 DEBUG:boto:Params: {} 2024-04-12T03:18:44.784 DEBUG:boto:Token: None 2024-04-12T03:18:44.784 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/ 2024-04-12T03:18:44.784 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY= 2024-04-12T03:18:44.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY=', 'Content-Length': '0'} 2024-04-12T03:18:44.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061950cb07a95a8db-006618a814-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.787 DEBUG:boto:b'dkyges-7:1000false:2024-04-12T03:15:03.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:44.787 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.787 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.787 DEBUG:boto:path=/dkyges-7/?prefix=%3A 2024-04-12T03:18:44.788 DEBUG:boto:auth_path=/dkyges-7/?prefix=%3A 2024-04-12T03:18:44.788 DEBUG:boto:Method: GET 2024-04-12T03:18:44.788 DEBUG:boto:Path: /dkyges-7/?prefix=%3A 2024-04-12T03:18:44.788 DEBUG:boto:Data: 2024-04-12T03:18:44.788 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.788 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.788 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.788 DEBUG:boto:Params: {} 2024-04-12T03:18:44.788 DEBUG:boto:Token: None 2024-04-12T03:18:44.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/ 2024-04-12T03:18:44.789 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY= 2024-04-12T03:18:44.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY=', 'Content-Length': '0'} 2024-04-12T03:18:44.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c1c1698ffb0384d-006618a814-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.791 DEBUG:boto:b'dkyges-7:1000false:2024-04-12T03:15:03.937Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:44.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-12T03:18:44.792 DEBUG:boto:path=/dkyges-7/_myobj 2024-04-12T03:18:44.792 DEBUG:boto:auth_path=/dkyges-7/_myobj 2024-04-12T03:18:44.792 DEBUG:boto:path=/dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.792 DEBUG:boto:auth_path=/dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.792 DEBUG:boto:Method: GET 2024-04-12T03:18:44.792 DEBUG:boto:Path: /dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.792 DEBUG:boto:Data: 2024-04-12T03:18:44.792 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.793 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.793 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.793 DEBUG:boto:Params: {} 2024-04-12T03:18:44.793 DEBUG:boto:Token: None 2024-04-12T03:18:44.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.793 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HXdBWwCiT0ttEd6MJGEBFwpOfFg= 2024-04-12T03:18:44.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HXdBWwCiT0ttEd6MJGEBFwpOfFg=', 'Content-Length': '0'} 2024-04-12T03:18:44.795 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ab6ad0cd35c1a38-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.795 DEBUG:boto:path=/dkyges-7/_myobj 2024-04-12T03:18:44.796 DEBUG:boto:auth_path=/dkyges-7/_myobj 2024-04-12T03:18:44.796 DEBUG:boto:path=/dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.796 DEBUG:boto:auth_path=/dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.796 DEBUG:boto:Method: GET 2024-04-12T03:18:44.796 DEBUG:boto:Path: /dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.796 DEBUG:boto:Data: 2024-04-12T03:18:44.796 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.796 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.796 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.796 DEBUG:boto:Params: {} 2024-04-12T03:18:44.796 DEBUG:boto:Token: None 2024-04-12T03:18:44.797 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.797 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HXdBWwCiT0ttEd6MJGEBFwpOfFg= 2024-04-12T03:18:44.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HXdBWwCiT0ttEd6MJGEBFwpOfFg=', 'Content-Length': '0'} 2024-04-12T03:18:44.798 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d82c4f38a95c7fb4-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.842 DEBUG:boto:path=/dkyges-7/_myobj 2024-04-12T03:18:44.842 DEBUG:boto:auth_path=/dkyges-7/_myobj 2024-04-12T03:18:44.843 DEBUG:boto:path=/dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.843 DEBUG:boto:auth_path=/dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.843 DEBUG:boto:Method: HEAD 2024-04-12T03:18:44.843 DEBUG:boto:Path: /dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.843 DEBUG:boto:Data: 2024-04-12T03:18:44.843 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.843 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.843 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.843 DEBUG:boto:Params: {} 2024-04-12T03:18:44.843 DEBUG:boto:Token: None 2024-04-12T03:18:44.844 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.844 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7qVqOMgWCSXbePil+MFsGp6+zBc= 2024-04-12T03:18:44.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7qVqOMgWCSXbePil+MFsGp6+zBc=', 'Content-Length': '0'} 2024-04-12T03:18:44.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000013e787bc3b63080e-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.846 DEBUG:boto:path=/dkyges-7/_myobj 2024-04-12T03:18:44.846 DEBUG:boto:auth_path=/dkyges-7/_myobj 2024-04-12T03:18:44.846 DEBUG:boto:path=/dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.846 DEBUG:boto:auth_path=/dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.846 DEBUG:boto:Method: HEAD 2024-04-12T03:18:44.846 DEBUG:boto:Path: /dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.846 DEBUG:boto:Data: 2024-04-12T03:18:44.846 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.846 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.847 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.847 DEBUG:boto:Params: {} 2024-04-12T03:18:44.847 DEBUG:boto:Token: None 2024-04-12T03:18:44.847 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.847 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7qVqOMgWCSXbePil+MFsGp6+zBc= 2024-04-12T03:18:44.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7qVqOMgWCSXbePil+MFsGp6+zBc=', 'Content-Length': '0'} 2024-04-12T03:18:44.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051dbfbfc3b8a670d-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-12T03:18:44.849 DEBUG:boto:path=/dkyges-7/_myobj 2024-04-12T03:18:44.849 DEBUG:boto:auth_path=/dkyges-7/_myobj 2024-04-12T03:18:44.849 DEBUG:boto:path=/dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.850 DEBUG:boto:auth_path=/dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.850 DEBUG:boto:Method: GET 2024-04-12T03:18:44.850 DEBUG:boto:Path: /dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.850 DEBUG:boto:Data: 2024-04-12T03:18:44.850 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.850 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.850 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.850 DEBUG:boto:Params: {} 2024-04-12T03:18:44.850 DEBUG:boto:Token: None 2024-04-12T03:18:44.850 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.851 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HXdBWwCiT0ttEd6MJGEBFwpOfFg= 2024-04-12T03:18:44.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HXdBWwCiT0ttEd6MJGEBFwpOfFg=', 'Content-Length': '0'} 2024-04-12T03:18:44.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e58d7e0dc273ce0-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.853 DEBUG:boto:path=/dkyges-7/_myobj 2024-04-12T03:18:44.853 DEBUG:boto:auth_path=/dkyges-7/_myobj 2024-04-12T03:18:44.853 DEBUG:boto:path=/dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.853 DEBUG:boto:auth_path=/dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.853 DEBUG:boto:Method: GET 2024-04-12T03:18:44.853 DEBUG:boto:Path: /dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.853 DEBUG:boto:Data: 2024-04-12T03:18:44.853 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.853 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.853 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.853 DEBUG:boto:Params: {} 2024-04-12T03:18:44.854 DEBUG:boto:Token: None 2024-04-12T03:18:44.854 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/_myobj?versionId=null 2024-04-12T03:18:44.854 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HXdBWwCiT0ttEd6MJGEBFwpOfFg= 2024-04-12T03:18:44.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HXdBWwCiT0ttEd6MJGEBFwpOfFg=', 'Content-Length': '0'} 2024-04-12T03:18:44.856 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083f3c865e40630ba-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.856 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.856 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.856 DEBUG:boto:path=/dkyges-7/?prefix=_myobj 2024-04-12T03:18:44.856 DEBUG:boto:auth_path=/dkyges-7/?prefix=_myobj 2024-04-12T03:18:44.856 DEBUG:boto:Method: GET 2024-04-12T03:18:44.857 DEBUG:boto:Path: /dkyges-7/?prefix=_myobj 2024-04-12T03:18:44.857 DEBUG:boto:Data: 2024-04-12T03:18:44.857 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.857 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.857 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.857 DEBUG:boto:Params: {} 2024-04-12T03:18:44.857 DEBUG:boto:Token: None 2024-04-12T03:18:44.857 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/ 2024-04-12T03:18:44.857 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY= 2024-04-12T03:18:44.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY=', 'Content-Length': '0'} 2024-04-12T03:18:44.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000806de2395a88dec0-006618a814-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.860 DEBUG:boto:b'dkyges-7_myobj1000false_myobj2024-04-12T03:15:03.887Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:44.860 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.861 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.861 DEBUG:boto:path=/dkyges-7/?prefix=_myobj 2024-04-12T03:18:44.861 DEBUG:boto:auth_path=/dkyges-7/?prefix=_myobj 2024-04-12T03:18:44.861 DEBUG:boto:Method: GET 2024-04-12T03:18:44.861 DEBUG:boto:Path: /dkyges-7/?prefix=_myobj 2024-04-12T03:18:44.861 DEBUG:boto:Data: 2024-04-12T03:18:44.861 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.861 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.861 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.861 DEBUG:boto:Params: {} 2024-04-12T03:18:44.861 DEBUG:boto:Token: None 2024-04-12T03:18:44.862 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/ 2024-04-12T03:18:44.862 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY= 2024-04-12T03:18:44.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY=', 'Content-Length': '0'} 2024-04-12T03:18:44.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8af235561b8b806-006618a814-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.864 DEBUG:boto:b'dkyges-7_myobj1000false_myobj2024-04-12T03:15:03.887Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:44.865 DEBUG:boto:path=/dkyges-7/_myobj 2024-04-12T03:18:44.865 DEBUG:boto:auth_path=/dkyges-7/_myobj 2024-04-12T03:18:44.865 DEBUG:boto:Method: HEAD 2024-04-12T03:18:44.865 DEBUG:boto:Path: /dkyges-7/_myobj 2024-04-12T03:18:44.865 DEBUG:boto:Data: 2024-04-12T03:18:44.865 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.865 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.865 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.865 DEBUG:boto:Params: {} 2024-04-12T03:18:44.866 DEBUG:boto:Token: None 2024-04-12T03:18:44.866 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/_myobj 2024-04-12T03:18:44.866 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:owPBfDMgMT+vqdJp3ioNIQFNBHo= 2024-04-12T03:18:44.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:owPBfDMgMT+vqdJp3ioNIQFNBHo=', 'Content-Length': '0'} 2024-04-12T03:18:44.868 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000879069b895da4f6d-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.868 DEBUG:boto:path=/dkyges-7/_myobj 2024-04-12T03:18:44.868 DEBUG:boto:auth_path=/dkyges-7/_myobj 2024-04-12T03:18:44.869 DEBUG:boto:Method: HEAD 2024-04-12T03:18:44.869 DEBUG:boto:Path: /dkyges-7/_myobj 2024-04-12T03:18:44.869 DEBUG:boto:Data: 2024-04-12T03:18:44.869 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.869 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.869 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.869 DEBUG:boto:Params: {} 2024-04-12T03:18:44.869 DEBUG:boto:Token: None 2024-04-12T03:18:44.869 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/_myobj 2024-04-12T03:18:44.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:owPBfDMgMT+vqdJp3ioNIQFNBHo= 2024-04-12T03:18:44.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:owPBfDMgMT+vqdJp3ioNIQFNBHo=', 'Content-Length': '0'} 2024-04-12T03:18:44.871 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c953a86c816193dc-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-12T03:18:44.872 DEBUG:boto:path=/dkyges-7/_myobj 2024-04-12T03:18:44.872 DEBUG:boto:auth_path=/dkyges-7/_myobj 2024-04-12T03:18:44.872 DEBUG:boto:Method: GET 2024-04-12T03:18:44.872 DEBUG:boto:Path: /dkyges-7/_myobj 2024-04-12T03:18:44.872 DEBUG:boto:Data: 2024-04-12T03:18:44.872 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.872 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.872 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.872 DEBUG:boto:Params: {} 2024-04-12T03:18:44.873 DEBUG:boto:Token: None 2024-04-12T03:18:44.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/_myobj 2024-04-12T03:18:44.873 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6crRH1YGWT+WE0uBSILcUUkUqx4= 2024-04-12T03:18:44.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6crRH1YGWT+WE0uBSILcUUkUqx4=', 'Content-Length': '0'} 2024-04-12T03:18:44.875 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000086ed5c3cf6d23693-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.918 DEBUG:boto:path=/dkyges-7/_myobj 2024-04-12T03:18:44.919 DEBUG:boto:auth_path=/dkyges-7/_myobj 2024-04-12T03:18:44.919 DEBUG:boto:Method: GET 2024-04-12T03:18:44.919 DEBUG:boto:Path: /dkyges-7/_myobj 2024-04-12T03:18:44.919 DEBUG:boto:Data: 2024-04-12T03:18:44.919 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.919 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.919 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.920 DEBUG:boto:Params: {} 2024-04-12T03:18:44.920 DEBUG:boto:Token: None 2024-04-12T03:18:44.920 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/_myobj 2024-04-12T03:18:44.920 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6crRH1YGWT+WE0uBSILcUUkUqx4= 2024-04-12T03:18:44.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6crRH1YGWT+WE0uBSILcUUkUqx4=', 'Content-Length': '0'} 2024-04-12T03:18:44.922 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a31358ce0a0bca02-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.923 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.923 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.923 DEBUG:boto:path=/dkyges-7/?prefix=_myobj 2024-04-12T03:18:44.923 DEBUG:boto:auth_path=/dkyges-7/?prefix=_myobj 2024-04-12T03:18:44.923 DEBUG:boto:Method: GET 2024-04-12T03:18:44.923 DEBUG:boto:Path: /dkyges-7/?prefix=_myobj 2024-04-12T03:18:44.923 DEBUG:boto:Data: 2024-04-12T03:18:44.923 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.923 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.923 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.924 DEBUG:boto:Params: {} 2024-04-12T03:18:44.924 DEBUG:boto:Token: None 2024-04-12T03:18:44.924 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/ 2024-04-12T03:18:44.924 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY= 2024-04-12T03:18:44.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY=', 'Content-Length': '0'} 2024-04-12T03:18:44.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016f746a92ca382b4-006618a814-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.927 DEBUG:boto:b'dkyges-7_myobj1000false_myobj2024-04-12T03:15:03.887Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:44.927 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:44.927 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:44.928 DEBUG:boto:path=/dkyges-7/?prefix=_myobj 2024-04-12T03:18:44.928 DEBUG:boto:auth_path=/dkyges-7/?prefix=_myobj 2024-04-12T03:18:44.928 DEBUG:boto:Method: GET 2024-04-12T03:18:44.928 DEBUG:boto:Path: /dkyges-7/?prefix=_myobj 2024-04-12T03:18:44.928 DEBUG:boto:Data: 2024-04-12T03:18:44.928 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.928 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.928 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.928 DEBUG:boto:Params: {} 2024-04-12T03:18:44.928 DEBUG:boto:Token: None 2024-04-12T03:18:44.929 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/ 2024-04-12T03:18:44.929 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY= 2024-04-12T03:18:44.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mSHZ3ljZrj2KIhWZzMu0bM3n4FY=', 'Content-Length': '0'} 2024-04-12T03:18:44.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007de4bceaf0cca64e-006618a814-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.931 DEBUG:boto:b'dkyges-7_myobj1000false_myobj2024-04-12T03:15:03.887Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:44.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T03:18:44.932 DEBUG:boto:path=/dkyges-7/myobj 2024-04-12T03:18:44.932 DEBUG:boto:auth_path=/dkyges-7/myobj 2024-04-12T03:18:44.932 DEBUG:boto:path=/dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.932 DEBUG:boto:auth_path=/dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.932 DEBUG:boto:Method: GET 2024-04-12T03:18:44.932 DEBUG:boto:Path: /dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.932 DEBUG:boto:Data: 2024-04-12T03:18:44.932 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.932 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.933 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.933 DEBUG:boto:Params: {} 2024-04-12T03:18:44.933 DEBUG:boto:Token: None 2024-04-12T03:18:44.933 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.933 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8SY6O10BcsZfMTr4a5+sYd3xFGo= 2024-04-12T03:18:44.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8SY6O10BcsZfMTr4a5+sYd3xFGo=', 'Content-Length': '0'} 2024-04-12T03:18:44.935 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be63e035a49d4217-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.935 DEBUG:boto:path=/dkyges-7/myobj 2024-04-12T03:18:44.936 DEBUG:boto:auth_path=/dkyges-7/myobj 2024-04-12T03:18:44.936 DEBUG:boto:path=/dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.936 DEBUG:boto:auth_path=/dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.936 DEBUG:boto:Method: GET 2024-04-12T03:18:44.936 DEBUG:boto:Path: /dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.936 DEBUG:boto:Data: 2024-04-12T03:18:44.936 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.936 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.936 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.936 DEBUG:boto:Params: {} 2024-04-12T03:18:44.937 DEBUG:boto:Token: None 2024-04-12T03:18:44.937 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.937 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8SY6O10BcsZfMTr4a5+sYd3xFGo= 2024-04-12T03:18:44.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8SY6O10BcsZfMTr4a5+sYd3xFGo=', 'Content-Length': '0'} 2024-04-12T03:18:44.939 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069cd8db8c4f21599-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.986 DEBUG:boto:path=/dkyges-7/myobj 2024-04-12T03:18:44.987 DEBUG:boto:auth_path=/dkyges-7/myobj 2024-04-12T03:18:44.987 DEBUG:boto:path=/dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.987 DEBUG:boto:auth_path=/dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.987 DEBUG:boto:Method: HEAD 2024-04-12T03:18:44.987 DEBUG:boto:Path: /dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.987 DEBUG:boto:Data: 2024-04-12T03:18:44.987 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.987 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.987 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.987 DEBUG:boto:Params: {} 2024-04-12T03:18:44.987 DEBUG:boto:Token: None 2024-04-12T03:18:44.988 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.988 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3RAZdCZlDOiWBtx2LmwHS8EZ6HU= 2024-04-12T03:18:44.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3RAZdCZlDOiWBtx2LmwHS8EZ6HU=', 'Content-Length': '0'} 2024-04-12T03:18:44.990 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f182c0f7b88e37e7-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.990 DEBUG:boto:path=/dkyges-7/myobj 2024-04-12T03:18:44.990 DEBUG:boto:auth_path=/dkyges-7/myobj 2024-04-12T03:18:44.990 DEBUG:boto:path=/dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.990 DEBUG:boto:auth_path=/dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.991 DEBUG:boto:Method: HEAD 2024-04-12T03:18:44.991 DEBUG:boto:Path: /dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.991 DEBUG:boto:Data: 2024-04-12T03:18:44.991 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.991 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.991 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.991 DEBUG:boto:Params: {} 2024-04-12T03:18:44.991 DEBUG:boto:Token: None 2024-04-12T03:18:44.991 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.991 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3RAZdCZlDOiWBtx2LmwHS8EZ6HU= 2024-04-12T03:18:44.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3RAZdCZlDOiWBtx2LmwHS8EZ6HU=', 'Content-Length': '0'} 2024-04-12T03:18:44.993 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d0d9a8f3c063193f-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T03:18:44.994 DEBUG:boto:path=/dkyges-7/myobj 2024-04-12T03:18:44.994 DEBUG:boto:auth_path=/dkyges-7/myobj 2024-04-12T03:18:44.994 DEBUG:boto:path=/dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.994 DEBUG:boto:auth_path=/dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.994 DEBUG:boto:Method: GET 2024-04-12T03:18:44.994 DEBUG:boto:Path: /dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.994 DEBUG:boto:Data: 2024-04-12T03:18:44.994 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.995 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:44.995 DEBUG:boto:Port: 8001 2024-04-12T03:18:44.995 DEBUG:boto:Params: {} 2024-04-12T03:18:44.995 DEBUG:boto:Token: None 2024-04-12T03:18:44.995 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.995 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8SY6O10BcsZfMTr4a5+sYd3xFGo= 2024-04-12T03:18:44.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8SY6O10BcsZfMTr4a5+sYd3xFGo=', 'Content-Length': '0'} 2024-04-12T03:18:44.997 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052a8b76cbaa327e4-006618a814-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:44.997 DEBUG:boto:path=/dkyges-7/myobj 2024-04-12T03:18:44.998 DEBUG:boto:auth_path=/dkyges-7/myobj 2024-04-12T03:18:44.998 DEBUG:boto:path=/dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.998 DEBUG:boto:auth_path=/dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.998 DEBUG:boto:Method: GET 2024-04-12T03:18:44.998 DEBUG:boto:Path: /dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.998 DEBUG:boto:Data: 2024-04-12T03:18:44.998 DEBUG:boto:Headers: {} 2024-04-12T03:18:44.998 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:44.998 DEBUG:boto:Port: 8000 2024-04-12T03:18:44.998 DEBUG:boto:Params: {} 2024-04-12T03:18:44.999 DEBUG:boto:Token: None 2024-04-12T03:18:44.999 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:44 GMT /dkyges-7/myobj?versionId=null 2024-04-12T03:18:44.999 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8SY6O10BcsZfMTr4a5+sYd3xFGo= 2024-04-12T03:18:44.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8SY6O10BcsZfMTr4a5+sYd3xFGo=', 'Content-Length': '0'} 2024-04-12T03:18:45.001 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000abbaf207cde68a01-006618a814-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.001 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:45.001 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:45.001 DEBUG:boto:path=/dkyges-7/?prefix=myobj 2024-04-12T03:18:45.002 DEBUG:boto:auth_path=/dkyges-7/?prefix=myobj 2024-04-12T03:18:45.002 DEBUG:boto:Method: GET 2024-04-12T03:18:45.002 DEBUG:boto:Path: /dkyges-7/?prefix=myobj 2024-04-12T03:18:45.002 DEBUG:boto:Data: 2024-04-12T03:18:45.002 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.002 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.002 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.002 DEBUG:boto:Params: {} 2024-04-12T03:18:45.002 DEBUG:boto:Token: None 2024-04-12T03:18:45.002 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-7/ 2024-04-12T03:18:45.002 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rMCROs8eF0aAVftJQaTn054mP5M= 2024-04-12T03:18:45.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rMCROs8eF0aAVftJQaTn054mP5M=', 'Content-Length': '0'} 2024-04-12T03:18:45.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088553a5d0582cd44-006618a815-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.005 DEBUG:boto:b'dkyges-7myobj1000falsemyobj2024-04-12T03:15:03.872Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.006 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:45.006 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:45.006 DEBUG:boto:path=/dkyges-7/?prefix=myobj 2024-04-12T03:18:45.006 DEBUG:boto:auth_path=/dkyges-7/?prefix=myobj 2024-04-12T03:18:45.006 DEBUG:boto:Method: GET 2024-04-12T03:18:45.006 DEBUG:boto:Path: /dkyges-7/?prefix=myobj 2024-04-12T03:18:45.006 DEBUG:boto:Data: 2024-04-12T03:18:45.006 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.006 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.006 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.007 DEBUG:boto:Params: {} 2024-04-12T03:18:45.007 DEBUG:boto:Token: None 2024-04-12T03:18:45.007 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-7/ 2024-04-12T03:18:45.007 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rMCROs8eF0aAVftJQaTn054mP5M= 2024-04-12T03:18:45.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rMCROs8eF0aAVftJQaTn054mP5M=', 'Content-Length': '0'} 2024-04-12T03:18:45.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009557b2900bbbb25a-006618a815-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.010 DEBUG:boto:b'dkyges-7myobj1000falsemyobj2024-04-12T03:15:03.872Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.010 DEBUG:boto:path=/dkyges-7/myobj 2024-04-12T03:18:45.010 DEBUG:boto:auth_path=/dkyges-7/myobj 2024-04-12T03:18:45.010 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.010 DEBUG:boto:Path: /dkyges-7/myobj 2024-04-12T03:18:45.011 DEBUG:boto:Data: 2024-04-12T03:18:45.011 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.011 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.011 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.011 DEBUG:boto:Params: {} 2024-04-12T03:18:45.011 DEBUG:boto:Token: None 2024-04-12T03:18:45.011 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-7/myobj 2024-04-12T03:18:45.011 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TXAgcuqhsD2mTf5A+zmFW+bCvI8= 2024-04-12T03:18:45.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TXAgcuqhsD2mTf5A+zmFW+bCvI8=', 'Content-Length': '0'} 2024-04-12T03:18:45.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c21ac902509d782a-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.013 DEBUG:boto:path=/dkyges-7/myobj 2024-04-12T03:18:45.013 DEBUG:boto:auth_path=/dkyges-7/myobj 2024-04-12T03:18:45.013 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.014 DEBUG:boto:Path: /dkyges-7/myobj 2024-04-12T03:18:45.014 DEBUG:boto:Data: 2024-04-12T03:18:45.014 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.014 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.014 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.014 DEBUG:boto:Params: {} 2024-04-12T03:18:45.014 DEBUG:boto:Token: None 2024-04-12T03:18:45.014 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-7/myobj 2024-04-12T03:18:45.014 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TXAgcuqhsD2mTf5A+zmFW+bCvI8= 2024-04-12T03:18:45.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TXAgcuqhsD2mTf5A+zmFW+bCvI8=', 'Content-Length': '0'} 2024-04-12T03:18:45.016 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030cc597b2cdc83e7-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T03:18:45.017 DEBUG:boto:path=/dkyges-7/myobj 2024-04-12T03:18:45.017 DEBUG:boto:auth_path=/dkyges-7/myobj 2024-04-12T03:18:45.017 DEBUG:boto:Method: GET 2024-04-12T03:18:45.017 DEBUG:boto:Path: /dkyges-7/myobj 2024-04-12T03:18:45.017 DEBUG:boto:Data: 2024-04-12T03:18:45.017 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.017 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.017 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.017 DEBUG:boto:Params: {} 2024-04-12T03:18:45.018 DEBUG:boto:Token: None 2024-04-12T03:18:45.018 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-7/myobj 2024-04-12T03:18:45.018 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2zh/pPXNzSVcKUz/+qc60wippAI= 2024-04-12T03:18:45.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2zh/pPXNzSVcKUz/+qc60wippAI=', 'Content-Length': '0'} 2024-04-12T03:18:45.020 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000437e1f238c657696-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.020 DEBUG:boto:path=/dkyges-7/myobj 2024-04-12T03:18:45.020 DEBUG:boto:auth_path=/dkyges-7/myobj 2024-04-12T03:18:45.020 DEBUG:boto:Method: GET 2024-04-12T03:18:45.020 DEBUG:boto:Path: /dkyges-7/myobj 2024-04-12T03:18:45.020 DEBUG:boto:Data: 2024-04-12T03:18:45.021 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.021 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.021 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.021 DEBUG:boto:Params: {} 2024-04-12T03:18:45.021 DEBUG:boto:Token: None 2024-04-12T03:18:45.021 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-7/myobj 2024-04-12T03:18:45.021 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2zh/pPXNzSVcKUz/+qc60wippAI= 2024-04-12T03:18:45.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2zh/pPXNzSVcKUz/+qc60wippAI=', 'Content-Length': '0'} 2024-04-12T03:18:45.023 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000414eb74028880215-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.023 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:45.023 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:45.023 DEBUG:boto:path=/dkyges-7/?prefix=myobj 2024-04-12T03:18:45.023 DEBUG:boto:auth_path=/dkyges-7/?prefix=myobj 2024-04-12T03:18:45.024 DEBUG:boto:Method: GET 2024-04-12T03:18:45.024 DEBUG:boto:Path: /dkyges-7/?prefix=myobj 2024-04-12T03:18:45.024 DEBUG:boto:Data: 2024-04-12T03:18:45.024 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.024 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.024 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.024 DEBUG:boto:Params: {} 2024-04-12T03:18:45.024 DEBUG:boto:Token: None 2024-04-12T03:18:45.024 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-7/ 2024-04-12T03:18:45.024 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rMCROs8eF0aAVftJQaTn054mP5M= 2024-04-12T03:18:45.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rMCROs8eF0aAVftJQaTn054mP5M=', 'Content-Length': '0'} 2024-04-12T03:18:45.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcfa25b7a19fe4d0-006618a815-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.027 DEBUG:boto:b'dkyges-7myobj1000falsemyobj2024-04-12T03:15:03.872Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.027 DEBUG:boto:path=/dkyges-7/ 2024-04-12T03:18:45.027 DEBUG:boto:auth_path=/dkyges-7/ 2024-04-12T03:18:45.028 DEBUG:boto:path=/dkyges-7/?prefix=myobj 2024-04-12T03:18:45.028 DEBUG:boto:auth_path=/dkyges-7/?prefix=myobj 2024-04-12T03:18:45.028 DEBUG:boto:Method: GET 2024-04-12T03:18:45.028 DEBUG:boto:Path: /dkyges-7/?prefix=myobj 2024-04-12T03:18:45.028 DEBUG:boto:Data: 2024-04-12T03:18:45.028 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.028 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.028 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.028 DEBUG:boto:Params: {} 2024-04-12T03:18:45.028 DEBUG:boto:Token: None 2024-04-12T03:18:45.028 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-7/ 2024-04-12T03:18:45.029 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rMCROs8eF0aAVftJQaTn054mP5M= 2024-04-12T03:18:45.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rMCROs8eF0aAVftJQaTn054mP5M=', 'Content-Length': '0'} 2024-04-12T03:18:45.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab66830580b06d17-006618a815-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.031 DEBUG:boto:b'dkyges-7myobj1000falsemyobj2024-04-12T03:15:03.872Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.032 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-7 zones={a2, a1} 2024-04-12T03:18:45.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:18:45.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:18:45.032 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-8 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:18:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:18:45.105 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:18:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:45.146+0000 7f0df6710780 1 bucket sync caught up with source: 2024-04-12T03:18:45.148 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.5.6, , 00000000001.11.6, , , , 00000000001.3.6, 00000000001.6.6, , ] 2024-04-12T03:18:45.148 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.5.6, , 00000000001.11.6, , , , 00000000001.3.6, 00000000001.6.6, , ] 2024-04-12T03:18:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:45.146+0000 7f0df6710780 0 bucket checkpoint complete 2024-04-12T03:18:45.162 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-8 zones={a1, a2} 2024-04-12T03:18:45.162 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.162 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.162 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.163 DEBUG:boto:Path: /dkyges-8/ 2024-04-12T03:18:45.163 DEBUG:boto:Data: 2024-04-12T03:18:45.163 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.163 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.163 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.163 DEBUG:boto:Params: {} 2024-04-12T03:18:45.163 DEBUG:boto:Token: None 2024-04-12T03:18:45.163 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.163 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mUDPwGu3+a2+6b7kAEcKEXADq80= 2024-04-12T03:18:45.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mUDPwGu3+a2+6b7kAEcKEXADq80=', 'Content-Length': '0'} 2024-04-12T03:18:45.166 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000725e6f5c624a43cb-006618a815-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.166 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.166 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.166 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.166 DEBUG:boto:Path: /dkyges-8/ 2024-04-12T03:18:45.166 DEBUG:boto:Data: 2024-04-12T03:18:45.166 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.167 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.167 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.167 DEBUG:boto:Params: {} 2024-04-12T03:18:45.167 DEBUG:boto:Token: None 2024-04-12T03:18:45.167 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.167 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mUDPwGu3+a2+6b7kAEcKEXADq80= 2024-04-12T03:18:45.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mUDPwGu3+a2+6b7kAEcKEXADq80=', 'Content-Length': '0'} 2024-04-12T03:18:45.169 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4a2eafe755adf21-006618a815-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.170 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:18:45.170 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.170 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.170 DEBUG:boto:path=/dkyges-8/?versions&max-keys=999 2024-04-12T03:18:45.170 DEBUG:boto:auth_path=/dkyges-8/?versions&max-keys=999 2024-04-12T03:18:45.170 DEBUG:boto:Method: GET 2024-04-12T03:18:45.170 DEBUG:boto:Path: /dkyges-8/?versions&max-keys=999 2024-04-12T03:18:45.170 DEBUG:boto:Data: 2024-04-12T03:18:45.170 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.170 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.171 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.171 DEBUG:boto:Params: {} 2024-04-12T03:18:45.171 DEBUG:boto:Token: None 2024-04-12T03:18:45.171 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/?versions 2024-04-12T03:18:45.171 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4xDZe61CvQZAo+Zy3tnGtlto5o0= 2024-04-12T03:18:45.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4xDZe61CvQZAo+Zy3tnGtlto5o0=', 'Content-Length': '0'} 2024-04-12T03:18:45.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000488c1dfb2bd96c90-006618a815-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.173 DEBUG:boto:b'dkyges-8999false&nulltrue2024-04-12T03:15:04.081Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-12T03:15:04.063Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-12T03:15:04.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-12T03:15:04.002Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.174 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-12T03:18:45.174 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-12T03:18:45.174 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-12T03:18:45.174 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-12T03:18:45.174 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:18:45.174 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.174 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.174 DEBUG:boto:path=/dkyges-8/?versions&max-keys=999 2024-04-12T03:18:45.175 DEBUG:boto:auth_path=/dkyges-8/?versions&max-keys=999 2024-04-12T03:18:45.175 DEBUG:boto:Method: GET 2024-04-12T03:18:45.175 DEBUG:boto:Path: /dkyges-8/?versions&max-keys=999 2024-04-12T03:18:45.175 DEBUG:boto:Data: 2024-04-12T03:18:45.175 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.175 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.175 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.175 DEBUG:boto:Params: {} 2024-04-12T03:18:45.175 DEBUG:boto:Token: None 2024-04-12T03:18:45.175 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/?versions 2024-04-12T03:18:45.175 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4xDZe61CvQZAo+Zy3tnGtlto5o0= 2024-04-12T03:18:45.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4xDZe61CvQZAo+Zy3tnGtlto5o0=', 'Content-Length': '0'} 2024-04-12T03:18:45.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dd36058e9681707-006618a815-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.178 DEBUG:boto:b'dkyges-8999false&nulltrue2024-04-12T03:15:04.081Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-12T03:15:04.063Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-12T03:15:04.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-12T03:15:04.002Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.178 DEBUG:tasks.rgw_multi.zone_rados:o=& 2024-04-12T03:18:45.179 DEBUG:tasks.rgw_multi.zone_rados:o=: 2024-04-12T03:18:45.179 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2024-04-12T03:18:45.179 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-12T03:18:45.179 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.179 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.179 DEBUG:boto:path=/dkyges-8/?versions&max-keys=999 2024-04-12T03:18:45.179 DEBUG:boto:auth_path=/dkyges-8/?versions&max-keys=999 2024-04-12T03:18:45.179 DEBUG:boto:Method: GET 2024-04-12T03:18:45.179 DEBUG:boto:Path: /dkyges-8/?versions&max-keys=999 2024-04-12T03:18:45.179 DEBUG:boto:Data: 2024-04-12T03:18:45.179 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.179 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.180 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.180 DEBUG:boto:Params: {} 2024-04-12T03:18:45.180 DEBUG:boto:Token: None 2024-04-12T03:18:45.180 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/?versions 2024-04-12T03:18:45.180 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4xDZe61CvQZAo+Zy3tnGtlto5o0= 2024-04-12T03:18:45.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4xDZe61CvQZAo+Zy3tnGtlto5o0=', 'Content-Length': '0'} 2024-04-12T03:18:45.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b36a0d64a18511c5-006618a815-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.182 DEBUG:boto:b'dkyges-8999false&nulltrue2024-04-12T03:15:04.081Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-12T03:15:04.063Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-12T03:15:04.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-12T03:15:04.002Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.183 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.183 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.183 DEBUG:boto:path=/dkyges-8/?versions&max-keys=999 2024-04-12T03:18:45.183 DEBUG:boto:auth_path=/dkyges-8/?versions&max-keys=999 2024-04-12T03:18:45.183 DEBUG:boto:Method: GET 2024-04-12T03:18:45.183 DEBUG:boto:Path: /dkyges-8/?versions&max-keys=999 2024-04-12T03:18:45.183 DEBUG:boto:Data: 2024-04-12T03:18:45.183 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.183 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.184 DEBUG:boto:Params: {} 2024-04-12T03:18:45.184 DEBUG:boto:Token: None 2024-04-12T03:18:45.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/?versions 2024-04-12T03:18:45.184 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4xDZe61CvQZAo+Zy3tnGtlto5o0= 2024-04-12T03:18:45.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4xDZe61CvQZAo+Zy3tnGtlto5o0=', 'Content-Length': '0'} 2024-04-12T03:18:45.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065ddfe540b523103-006618a815-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.186 DEBUG:boto:b'dkyges-8999false&nulltrue2024-04-12T03:15:04.081Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2024-04-12T03:15:04.063Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2024-04-12T03:15:04.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2024-04-12T03:15:04.002Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-12T03:18:45.187 DEBUG:boto:path=/dkyges-8/%26 2024-04-12T03:18:45.187 DEBUG:boto:auth_path=/dkyges-8/%26 2024-04-12T03:18:45.187 DEBUG:boto:path=/dkyges-8/%26?versionId=null 2024-04-12T03:18:45.187 DEBUG:boto:auth_path=/dkyges-8/%26?versionId=null 2024-04-12T03:18:45.187 DEBUG:boto:Method: GET 2024-04-12T03:18:45.187 DEBUG:boto:Path: /dkyges-8/%26?versionId=null 2024-04-12T03:18:45.188 DEBUG:boto:Data: 2024-04-12T03:18:45.188 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.188 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.188 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.188 DEBUG:boto:Params: {} 2024-04-12T03:18:45.188 DEBUG:boto:Token: None 2024-04-12T03:18:45.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%26?versionId=null 2024-04-12T03:18:45.188 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:swQNZucWBMLl19dSic7/QWWbOag= 2024-04-12T03:18:45.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:swQNZucWBMLl19dSic7/QWWbOag=', 'Content-Length': '0'} 2024-04-12T03:18:45.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b9f113941cd803b-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.230 DEBUG:boto:path=/dkyges-8/%26 2024-04-12T03:18:45.230 DEBUG:boto:auth_path=/dkyges-8/%26 2024-04-12T03:18:45.230 DEBUG:boto:path=/dkyges-8/%26?versionId=null 2024-04-12T03:18:45.231 DEBUG:boto:auth_path=/dkyges-8/%26?versionId=null 2024-04-12T03:18:45.231 DEBUG:boto:Method: GET 2024-04-12T03:18:45.231 DEBUG:boto:Path: /dkyges-8/%26?versionId=null 2024-04-12T03:18:45.231 DEBUG:boto:Data: 2024-04-12T03:18:45.231 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.231 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.231 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.231 DEBUG:boto:Params: {} 2024-04-12T03:18:45.231 DEBUG:boto:Token: None 2024-04-12T03:18:45.231 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%26?versionId=null 2024-04-12T03:18:45.232 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:swQNZucWBMLl19dSic7/QWWbOag= 2024-04-12T03:18:45.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:swQNZucWBMLl19dSic7/QWWbOag=', 'Content-Length': '0'} 2024-04-12T03:18:45.233 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe51b245d1db313c-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.234 DEBUG:boto:path=/dkyges-8/%26 2024-04-12T03:18:45.234 DEBUG:boto:auth_path=/dkyges-8/%26 2024-04-12T03:18:45.234 DEBUG:boto:path=/dkyges-8/%26?versionId=null 2024-04-12T03:18:45.234 DEBUG:boto:auth_path=/dkyges-8/%26?versionId=null 2024-04-12T03:18:45.234 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.234 DEBUG:boto:Path: /dkyges-8/%26?versionId=null 2024-04-12T03:18:45.234 DEBUG:boto:Data: 2024-04-12T03:18:45.234 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.234 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.234 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.234 DEBUG:boto:Params: {} 2024-04-12T03:18:45.235 DEBUG:boto:Token: None 2024-04-12T03:18:45.235 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%26?versionId=null 2024-04-12T03:18:45.235 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NHIkcemww7Xpo/zDKWXbSvj475M= 2024-04-12T03:18:45.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NHIkcemww7Xpo/zDKWXbSvj475M=', 'Content-Length': '0'} 2024-04-12T03:18:45.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fa04121a5a79beb-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.237 DEBUG:boto:path=/dkyges-8/%26 2024-04-12T03:18:45.237 DEBUG:boto:auth_path=/dkyges-8/%26 2024-04-12T03:18:45.237 DEBUG:boto:path=/dkyges-8/%26?versionId=null 2024-04-12T03:18:45.237 DEBUG:boto:auth_path=/dkyges-8/%26?versionId=null 2024-04-12T03:18:45.237 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.237 DEBUG:boto:Path: /dkyges-8/%26?versionId=null 2024-04-12T03:18:45.238 DEBUG:boto:Data: 2024-04-12T03:18:45.238 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.238 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.238 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.238 DEBUG:boto:Params: {} 2024-04-12T03:18:45.238 DEBUG:boto:Token: None 2024-04-12T03:18:45.238 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%26?versionId=null 2024-04-12T03:18:45.238 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NHIkcemww7Xpo/zDKWXbSvj475M= 2024-04-12T03:18:45.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NHIkcemww7Xpo/zDKWXbSvj475M=', 'Content-Length': '0'} 2024-04-12T03:18:45.240 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bafd2cb49b125850-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-12T03:18:45.240 DEBUG:boto:path=/dkyges-8/%26 2024-04-12T03:18:45.241 DEBUG:boto:auth_path=/dkyges-8/%26 2024-04-12T03:18:45.241 DEBUG:boto:path=/dkyges-8/%26?versionId=null 2024-04-12T03:18:45.241 DEBUG:boto:auth_path=/dkyges-8/%26?versionId=null 2024-04-12T03:18:45.241 DEBUG:boto:Method: GET 2024-04-12T03:18:45.241 DEBUG:boto:Path: /dkyges-8/%26?versionId=null 2024-04-12T03:18:45.241 DEBUG:boto:Data: 2024-04-12T03:18:45.241 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.241 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.241 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.241 DEBUG:boto:Params: {} 2024-04-12T03:18:45.241 DEBUG:boto:Token: None 2024-04-12T03:18:45.242 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%26?versionId=null 2024-04-12T03:18:45.242 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:swQNZucWBMLl19dSic7/QWWbOag= 2024-04-12T03:18:45.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:swQNZucWBMLl19dSic7/QWWbOag=', 'Content-Length': '0'} 2024-04-12T03:18:45.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000323a38d3cfe9a019-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.244 DEBUG:boto:path=/dkyges-8/%26 2024-04-12T03:18:45.244 DEBUG:boto:auth_path=/dkyges-8/%26 2024-04-12T03:18:45.244 DEBUG:boto:path=/dkyges-8/%26?versionId=null 2024-04-12T03:18:45.244 DEBUG:boto:auth_path=/dkyges-8/%26?versionId=null 2024-04-12T03:18:45.244 DEBUG:boto:Method: GET 2024-04-12T03:18:45.244 DEBUG:boto:Path: /dkyges-8/%26?versionId=null 2024-04-12T03:18:45.244 DEBUG:boto:Data: 2024-04-12T03:18:45.245 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.245 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.245 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.245 DEBUG:boto:Params: {} 2024-04-12T03:18:45.245 DEBUG:boto:Token: None 2024-04-12T03:18:45.245 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%26?versionId=null 2024-04-12T03:18:45.245 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:swQNZucWBMLl19dSic7/QWWbOag= 2024-04-12T03:18:45.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:swQNZucWBMLl19dSic7/QWWbOag=', 'Content-Length': '0'} 2024-04-12T03:18:45.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf583ebb979f79de-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.247 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.247 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.247 DEBUG:boto:path=/dkyges-8/?prefix=%26 2024-04-12T03:18:45.247 DEBUG:boto:auth_path=/dkyges-8/?prefix=%26 2024-04-12T03:18:45.248 DEBUG:boto:Method: GET 2024-04-12T03:18:45.248 DEBUG:boto:Path: /dkyges-8/?prefix=%26 2024-04-12T03:18:45.248 DEBUG:boto:Data: 2024-04-12T03:18:45.248 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.248 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.248 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.248 DEBUG:boto:Params: {} 2024-04-12T03:18:45.248 DEBUG:boto:Token: None 2024-04-12T03:18:45.248 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.248 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ= 2024-04-12T03:18:45.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006889e3b4c763667-006618a815-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.251 DEBUG:boto:b'dkyges-8&1000false&2024-04-12T03:15:04.081Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.251 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.251 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.251 DEBUG:boto:path=/dkyges-8/?prefix=%26 2024-04-12T03:18:45.251 DEBUG:boto:auth_path=/dkyges-8/?prefix=%26 2024-04-12T03:18:45.252 DEBUG:boto:Method: GET 2024-04-12T03:18:45.252 DEBUG:boto:Path: /dkyges-8/?prefix=%26 2024-04-12T03:18:45.252 DEBUG:boto:Data: 2024-04-12T03:18:45.252 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.252 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.252 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.252 DEBUG:boto:Params: {} 2024-04-12T03:18:45.252 DEBUG:boto:Token: None 2024-04-12T03:18:45.252 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.252 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ= 2024-04-12T03:18:45.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a73e082d16201cbe-006618a815-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.298 DEBUG:boto:b'dkyges-8&1000false&2024-04-12T03:15:04.081Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.299 DEBUG:boto:path=/dkyges-8/%26 2024-04-12T03:18:45.299 DEBUG:boto:auth_path=/dkyges-8/%26 2024-04-12T03:18:45.299 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.299 DEBUG:boto:Path: /dkyges-8/%26 2024-04-12T03:18:45.299 DEBUG:boto:Data: 2024-04-12T03:18:45.299 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.299 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.299 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.299 DEBUG:boto:Params: {} 2024-04-12T03:18:45.299 DEBUG:boto:Token: None 2024-04-12T03:18:45.300 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%26 2024-04-12T03:18:45.300 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PX8uA4D4rUYuqytiLv82AN2Gth4= 2024-04-12T03:18:45.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PX8uA4D4rUYuqytiLv82AN2Gth4=', 'Content-Length': '0'} 2024-04-12T03:18:45.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a825452d2d2ebfab-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.302 DEBUG:boto:path=/dkyges-8/%26 2024-04-12T03:18:45.302 DEBUG:boto:auth_path=/dkyges-8/%26 2024-04-12T03:18:45.302 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.302 DEBUG:boto:Path: /dkyges-8/%26 2024-04-12T03:18:45.302 DEBUG:boto:Data: 2024-04-12T03:18:45.302 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.302 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.302 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.303 DEBUG:boto:Params: {} 2024-04-12T03:18:45.303 DEBUG:boto:Token: None 2024-04-12T03:18:45.303 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%26 2024-04-12T03:18:45.303 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PX8uA4D4rUYuqytiLv82AN2Gth4= 2024-04-12T03:18:45.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PX8uA4D4rUYuqytiLv82AN2Gth4=', 'Content-Length': '0'} 2024-04-12T03:18:45.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000739d19fb4fb6f408-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2024-04-12T03:18:45.305 DEBUG:boto:path=/dkyges-8/%26 2024-04-12T03:18:45.305 DEBUG:boto:auth_path=/dkyges-8/%26 2024-04-12T03:18:45.305 DEBUG:boto:Method: GET 2024-04-12T03:18:45.305 DEBUG:boto:Path: /dkyges-8/%26 2024-04-12T03:18:45.305 DEBUG:boto:Data: 2024-04-12T03:18:45.305 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.305 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.306 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.306 DEBUG:boto:Params: {} 2024-04-12T03:18:45.306 DEBUG:boto:Token: None 2024-04-12T03:18:45.306 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%26 2024-04-12T03:18:45.306 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lG7tdIFWK31lZPa923Pbd+Wi0Ww= 2024-04-12T03:18:45.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lG7tdIFWK31lZPa923Pbd+Wi0Ww=', 'Content-Length': '0'} 2024-04-12T03:18:45.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bace4eed920c7b6-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.308 DEBUG:boto:path=/dkyges-8/%26 2024-04-12T03:18:45.308 DEBUG:boto:auth_path=/dkyges-8/%26 2024-04-12T03:18:45.309 DEBUG:boto:Method: GET 2024-04-12T03:18:45.309 DEBUG:boto:Path: /dkyges-8/%26 2024-04-12T03:18:45.309 DEBUG:boto:Data: 2024-04-12T03:18:45.309 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.309 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.309 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.309 DEBUG:boto:Params: {} 2024-04-12T03:18:45.309 DEBUG:boto:Token: None 2024-04-12T03:18:45.309 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%26 2024-04-12T03:18:45.310 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lG7tdIFWK31lZPa923Pbd+Wi0Ww= 2024-04-12T03:18:45.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lG7tdIFWK31lZPa923Pbd+Wi0Ww=', 'Content-Length': '0'} 2024-04-12T03:18:45.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d26289c53d021e0-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.312 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.312 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.312 DEBUG:boto:path=/dkyges-8/?prefix=%26 2024-04-12T03:18:45.312 DEBUG:boto:auth_path=/dkyges-8/?prefix=%26 2024-04-12T03:18:45.312 DEBUG:boto:Method: GET 2024-04-12T03:18:45.312 DEBUG:boto:Path: /dkyges-8/?prefix=%26 2024-04-12T03:18:45.312 DEBUG:boto:Data: 2024-04-12T03:18:45.313 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.313 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.313 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.313 DEBUG:boto:Params: {} 2024-04-12T03:18:45.313 DEBUG:boto:Token: None 2024-04-12T03:18:45.313 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.313 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ= 2024-04-12T03:18:45.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e7aa4f6f1e07216-006618a815-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.316 DEBUG:boto:b'dkyges-8&1000false&2024-04-12T03:15:04.081Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.316 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.316 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.316 DEBUG:boto:path=/dkyges-8/?prefix=%26 2024-04-12T03:18:45.316 DEBUG:boto:auth_path=/dkyges-8/?prefix=%26 2024-04-12T03:18:45.316 DEBUG:boto:Method: GET 2024-04-12T03:18:45.317 DEBUG:boto:Path: /dkyges-8/?prefix=%26 2024-04-12T03:18:45.317 DEBUG:boto:Data: 2024-04-12T03:18:45.317 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.317 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.317 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.317 DEBUG:boto:Params: {} 2024-04-12T03:18:45.317 DEBUG:boto:Token: None 2024-04-12T03:18:45.317 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.317 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ= 2024-04-12T03:18:45.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d0a3c5041d3109e-006618a815-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.320 DEBUG:boto:b'dkyges-8&1000false&2024-04-12T03:15:04.081Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-12T03:18:45.320 DEBUG:boto:path=/dkyges-8/%3A 2024-04-12T03:18:45.321 DEBUG:boto:auth_path=/dkyges-8/%3A 2024-04-12T03:18:45.321 DEBUG:boto:path=/dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.321 DEBUG:boto:auth_path=/dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.321 DEBUG:boto:Method: GET 2024-04-12T03:18:45.321 DEBUG:boto:Path: /dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.321 DEBUG:boto:Data: 2024-04-12T03:18:45.321 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.321 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.321 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.321 DEBUG:boto:Params: {} 2024-04-12T03:18:45.321 DEBUG:boto:Token: None 2024-04-12T03:18:45.322 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.322 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n0uiKZPbzu1oJdi6fOfaCbq4dfw= 2024-04-12T03:18:45.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n0uiKZPbzu1oJdi6fOfaCbq4dfw=', 'Content-Length': '0'} 2024-04-12T03:18:45.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c66be31990690de3-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.324 DEBUG:boto:path=/dkyges-8/%3A 2024-04-12T03:18:45.324 DEBUG:boto:auth_path=/dkyges-8/%3A 2024-04-12T03:18:45.324 DEBUG:boto:path=/dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.324 DEBUG:boto:auth_path=/dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.324 DEBUG:boto:Method: GET 2024-04-12T03:18:45.324 DEBUG:boto:Path: /dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.325 DEBUG:boto:Data: 2024-04-12T03:18:45.325 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.325 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.325 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.325 DEBUG:boto:Params: {} 2024-04-12T03:18:45.325 DEBUG:boto:Token: None 2024-04-12T03:18:45.325 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.325 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n0uiKZPbzu1oJdi6fOfaCbq4dfw= 2024-04-12T03:18:45.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n0uiKZPbzu1oJdi6fOfaCbq4dfw=', 'Content-Length': '0'} 2024-04-12T03:18:45.327 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000076c94ddfd90be4d6-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.327 DEBUG:boto:path=/dkyges-8/%3A 2024-04-12T03:18:45.327 DEBUG:boto:auth_path=/dkyges-8/%3A 2024-04-12T03:18:45.328 DEBUG:boto:path=/dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.328 DEBUG:boto:auth_path=/dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.328 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.328 DEBUG:boto:Path: /dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.328 DEBUG:boto:Data: 2024-04-12T03:18:45.328 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.328 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.328 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.328 DEBUG:boto:Params: {} 2024-04-12T03:18:45.328 DEBUG:boto:Token: None 2024-04-12T03:18:45.328 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.329 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:K7Q7bbs89q9U1hEjEbI7v5JXpOw= 2024-04-12T03:18:45.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:K7Q7bbs89q9U1hEjEbI7v5JXpOw=', 'Content-Length': '0'} 2024-04-12T03:18:45.330 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005848584b8a198979-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.331 DEBUG:boto:path=/dkyges-8/%3A 2024-04-12T03:18:45.331 DEBUG:boto:auth_path=/dkyges-8/%3A 2024-04-12T03:18:45.331 DEBUG:boto:path=/dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.331 DEBUG:boto:auth_path=/dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.331 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.331 DEBUG:boto:Path: /dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.331 DEBUG:boto:Data: 2024-04-12T03:18:45.331 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.332 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.332 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.332 DEBUG:boto:Params: {} 2024-04-12T03:18:45.332 DEBUG:boto:Token: None 2024-04-12T03:18:45.332 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.332 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:K7Q7bbs89q9U1hEjEbI7v5JXpOw= 2024-04-12T03:18:45.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:K7Q7bbs89q9U1hEjEbI7v5JXpOw=', 'Content-Length': '0'} 2024-04-12T03:18:45.334 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001bd1f3e9fa819dd3-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-12T03:18:45.334 DEBUG:boto:path=/dkyges-8/%3A 2024-04-12T03:18:45.334 DEBUG:boto:auth_path=/dkyges-8/%3A 2024-04-12T03:18:45.334 DEBUG:boto:path=/dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.334 DEBUG:boto:auth_path=/dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.334 DEBUG:boto:Method: GET 2024-04-12T03:18:45.334 DEBUG:boto:Path: /dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.334 DEBUG:boto:Data: 2024-04-12T03:18:45.335 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.335 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.335 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.335 DEBUG:boto:Params: {} 2024-04-12T03:18:45.335 DEBUG:boto:Token: None 2024-04-12T03:18:45.335 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.335 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n0uiKZPbzu1oJdi6fOfaCbq4dfw= 2024-04-12T03:18:45.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n0uiKZPbzu1oJdi6fOfaCbq4dfw=', 'Content-Length': '0'} 2024-04-12T03:18:45.337 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e05d95d23ebb7cfa-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.378 DEBUG:boto:path=/dkyges-8/%3A 2024-04-12T03:18:45.378 DEBUG:boto:auth_path=/dkyges-8/%3A 2024-04-12T03:18:45.379 DEBUG:boto:path=/dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.379 DEBUG:boto:auth_path=/dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.379 DEBUG:boto:Method: GET 2024-04-12T03:18:45.379 DEBUG:boto:Path: /dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.379 DEBUG:boto:Data: 2024-04-12T03:18:45.379 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.379 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.379 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.379 DEBUG:boto:Params: {} 2024-04-12T03:18:45.379 DEBUG:boto:Token: None 2024-04-12T03:18:45.379 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%3A?versionId=null 2024-04-12T03:18:45.380 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n0uiKZPbzu1oJdi6fOfaCbq4dfw= 2024-04-12T03:18:45.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n0uiKZPbzu1oJdi6fOfaCbq4dfw=', 'Content-Length': '0'} 2024-04-12T03:18:45.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000a7c9980ca95042-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.382 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.382 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.382 DEBUG:boto:path=/dkyges-8/?prefix=%3A 2024-04-12T03:18:45.382 DEBUG:boto:auth_path=/dkyges-8/?prefix=%3A 2024-04-12T03:18:45.382 DEBUG:boto:Method: GET 2024-04-12T03:18:45.382 DEBUG:boto:Path: /dkyges-8/?prefix=%3A 2024-04-12T03:18:45.382 DEBUG:boto:Data: 2024-04-12T03:18:45.382 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.382 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.383 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.383 DEBUG:boto:Params: {} 2024-04-12T03:18:45.383 DEBUG:boto:Token: None 2024-04-12T03:18:45.383 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.383 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ= 2024-04-12T03:18:45.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df22ff33e8b8d666-006618a815-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.386 DEBUG:boto:b'dkyges-8:1000false:2024-04-12T03:15:04.063Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.386 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.386 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.386 DEBUG:boto:path=/dkyges-8/?prefix=%3A 2024-04-12T03:18:45.386 DEBUG:boto:auth_path=/dkyges-8/?prefix=%3A 2024-04-12T03:18:45.387 DEBUG:boto:Method: GET 2024-04-12T03:18:45.387 DEBUG:boto:Path: /dkyges-8/?prefix=%3A 2024-04-12T03:18:45.387 DEBUG:boto:Data: 2024-04-12T03:18:45.387 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.387 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.387 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.387 DEBUG:boto:Params: {} 2024-04-12T03:18:45.387 DEBUG:boto:Token: None 2024-04-12T03:18:45.387 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.387 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ= 2024-04-12T03:18:45.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c1a664621f8025b-006618a815-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.390 DEBUG:boto:b'dkyges-8:1000false:2024-04-12T03:15:04.063Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.390 DEBUG:boto:path=/dkyges-8/%3A 2024-04-12T03:18:45.390 DEBUG:boto:auth_path=/dkyges-8/%3A 2024-04-12T03:18:45.390 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.390 DEBUG:boto:Path: /dkyges-8/%3A 2024-04-12T03:18:45.390 DEBUG:boto:Data: 2024-04-12T03:18:45.391 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.391 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.391 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.391 DEBUG:boto:Params: {} 2024-04-12T03:18:45.391 DEBUG:boto:Token: None 2024-04-12T03:18:45.391 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%3A 2024-04-12T03:18:45.391 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7MdZNYzyT5+mftSbmro0ixoMDkY= 2024-04-12T03:18:45.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7MdZNYzyT5+mftSbmro0ixoMDkY=', 'Content-Length': '0'} 2024-04-12T03:18:45.393 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0d4257c8c843da5-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.393 DEBUG:boto:path=/dkyges-8/%3A 2024-04-12T03:18:45.393 DEBUG:boto:auth_path=/dkyges-8/%3A 2024-04-12T03:18:45.394 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.394 DEBUG:boto:Path: /dkyges-8/%3A 2024-04-12T03:18:45.394 DEBUG:boto:Data: 2024-04-12T03:18:45.394 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.394 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.394 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.394 DEBUG:boto:Params: {} 2024-04-12T03:18:45.394 DEBUG:boto:Token: None 2024-04-12T03:18:45.394 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%3A 2024-04-12T03:18:45.394 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7MdZNYzyT5+mftSbmro0ixoMDkY= 2024-04-12T03:18:45.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7MdZNYzyT5+mftSbmro0ixoMDkY=', 'Content-Length': '0'} 2024-04-12T03:18:45.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001832b7cf72b8aaf3-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2024-04-12T03:18:45.396 DEBUG:boto:path=/dkyges-8/%3A 2024-04-12T03:18:45.397 DEBUG:boto:auth_path=/dkyges-8/%3A 2024-04-12T03:18:45.397 DEBUG:boto:Method: GET 2024-04-12T03:18:45.397 DEBUG:boto:Path: /dkyges-8/%3A 2024-04-12T03:18:45.397 DEBUG:boto:Data: 2024-04-12T03:18:45.397 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.397 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.397 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.397 DEBUG:boto:Params: {} 2024-04-12T03:18:45.397 DEBUG:boto:Token: None 2024-04-12T03:18:45.398 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%3A 2024-04-12T03:18:45.398 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sL+dYI52nF5e/mCETwFIfaL2Zms= 2024-04-12T03:18:45.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sL+dYI52nF5e/mCETwFIfaL2Zms=', 'Content-Length': '0'} 2024-04-12T03:18:45.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080ccba5a7e8c7d51-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.400 DEBUG:boto:path=/dkyges-8/%3A 2024-04-12T03:18:45.400 DEBUG:boto:auth_path=/dkyges-8/%3A 2024-04-12T03:18:45.400 DEBUG:boto:Method: GET 2024-04-12T03:18:45.400 DEBUG:boto:Path: /dkyges-8/%3A 2024-04-12T03:18:45.400 DEBUG:boto:Data: 2024-04-12T03:18:45.400 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.400 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.400 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.400 DEBUG:boto:Params: {} 2024-04-12T03:18:45.401 DEBUG:boto:Token: None 2024-04-12T03:18:45.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/%3A 2024-04-12T03:18:45.401 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sL+dYI52nF5e/mCETwFIfaL2Zms= 2024-04-12T03:18:45.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sL+dYI52nF5e/mCETwFIfaL2Zms=', 'Content-Length': '0'} 2024-04-12T03:18:45.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f1c0b8ed805e5e5-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.403 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.403 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.403 DEBUG:boto:path=/dkyges-8/?prefix=%3A 2024-04-12T03:18:45.403 DEBUG:boto:auth_path=/dkyges-8/?prefix=%3A 2024-04-12T03:18:45.403 DEBUG:boto:Method: GET 2024-04-12T03:18:45.403 DEBUG:boto:Path: /dkyges-8/?prefix=%3A 2024-04-12T03:18:45.403 DEBUG:boto:Data: 2024-04-12T03:18:45.404 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.404 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.404 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.404 DEBUG:boto:Params: {} 2024-04-12T03:18:45.404 DEBUG:boto:Token: None 2024-04-12T03:18:45.404 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.404 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ= 2024-04-12T03:18:45.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ac55e26bc3b8962-006618a815-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.407 DEBUG:boto:b'dkyges-8:1000false:2024-04-12T03:15:04.063Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.407 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.407 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.407 DEBUG:boto:path=/dkyges-8/?prefix=%3A 2024-04-12T03:18:45.408 DEBUG:boto:auth_path=/dkyges-8/?prefix=%3A 2024-04-12T03:18:45.408 DEBUG:boto:Method: GET 2024-04-12T03:18:45.408 DEBUG:boto:Path: /dkyges-8/?prefix=%3A 2024-04-12T03:18:45.408 DEBUG:boto:Data: 2024-04-12T03:18:45.408 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.408 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.408 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.408 DEBUG:boto:Params: {} 2024-04-12T03:18:45.408 DEBUG:boto:Token: None 2024-04-12T03:18:45.408 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.409 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ= 2024-04-12T03:18:45.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e3f40551c35c07c-006618a815-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.411 DEBUG:boto:b'dkyges-8:1000false:2024-04-12T03:15:04.063Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-12T03:18:45.411 DEBUG:boto:path=/dkyges-8/_myobj 2024-04-12T03:18:45.411 DEBUG:boto:auth_path=/dkyges-8/_myobj 2024-04-12T03:18:45.411 DEBUG:boto:path=/dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.412 DEBUG:boto:auth_path=/dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.412 DEBUG:boto:Method: GET 2024-04-12T03:18:45.412 DEBUG:boto:Path: /dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.412 DEBUG:boto:Data: 2024-04-12T03:18:45.412 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.412 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.412 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.412 DEBUG:boto:Params: {} 2024-04-12T03:18:45.412 DEBUG:boto:Token: None 2024-04-12T03:18:45.412 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.413 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qQ28bMIrBl/vcf2LDUfHoed+/AY= 2024-04-12T03:18:45.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qQ28bMIrBl/vcf2LDUfHoed+/AY=', 'Content-Length': '0'} 2024-04-12T03:18:45.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b942d5284068226d-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.415 DEBUG:boto:path=/dkyges-8/_myobj 2024-04-12T03:18:45.415 DEBUG:boto:auth_path=/dkyges-8/_myobj 2024-04-12T03:18:45.415 DEBUG:boto:path=/dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.415 DEBUG:boto:auth_path=/dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.415 DEBUG:boto:Method: GET 2024-04-12T03:18:45.415 DEBUG:boto:Path: /dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.415 DEBUG:boto:Data: 2024-04-12T03:18:45.416 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.416 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.416 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.416 DEBUG:boto:Params: {} 2024-04-12T03:18:45.416 DEBUG:boto:Token: None 2024-04-12T03:18:45.416 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.416 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qQ28bMIrBl/vcf2LDUfHoed+/AY= 2024-04-12T03:18:45.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qQ28bMIrBl/vcf2LDUfHoed+/AY=', 'Content-Length': '0'} 2024-04-12T03:18:45.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024b41d7c4727a2c9-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.418 DEBUG:boto:path=/dkyges-8/_myobj 2024-04-12T03:18:45.418 DEBUG:boto:auth_path=/dkyges-8/_myobj 2024-04-12T03:18:45.418 DEBUG:boto:path=/dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.418 DEBUG:boto:auth_path=/dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.418 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.419 DEBUG:boto:Path: /dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.419 DEBUG:boto:Data: 2024-04-12T03:18:45.419 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.419 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.419 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.419 DEBUG:boto:Params: {} 2024-04-12T03:18:45.419 DEBUG:boto:Token: None 2024-04-12T03:18:45.419 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.419 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HqNii3R0VSRW0wdV+CuOIXk1mxg= 2024-04-12T03:18:45.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HqNii3R0VSRW0wdV+CuOIXk1mxg=', 'Content-Length': '0'} 2024-04-12T03:18:45.421 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068d7ddf2c4faf361-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.421 DEBUG:boto:path=/dkyges-8/_myobj 2024-04-12T03:18:45.421 DEBUG:boto:auth_path=/dkyges-8/_myobj 2024-04-12T03:18:45.421 DEBUG:boto:path=/dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.421 DEBUG:boto:auth_path=/dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.422 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.422 DEBUG:boto:Path: /dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.422 DEBUG:boto:Data: 2024-04-12T03:18:45.422 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.422 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.422 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.422 DEBUG:boto:Params: {} 2024-04-12T03:18:45.422 DEBUG:boto:Token: None 2024-04-12T03:18:45.422 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.423 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HqNii3R0VSRW0wdV+CuOIXk1mxg= 2024-04-12T03:18:45.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HqNii3R0VSRW0wdV+CuOIXk1mxg=', 'Content-Length': '0'} 2024-04-12T03:18:45.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7fd22f198116e67-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-12T03:18:45.425 DEBUG:boto:path=/dkyges-8/_myobj 2024-04-12T03:18:45.425 DEBUG:boto:auth_path=/dkyges-8/_myobj 2024-04-12T03:18:45.425 DEBUG:boto:path=/dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.425 DEBUG:boto:auth_path=/dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.425 DEBUG:boto:Method: GET 2024-04-12T03:18:45.425 DEBUG:boto:Path: /dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.425 DEBUG:boto:Data: 2024-04-12T03:18:45.425 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.425 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.425 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.425 DEBUG:boto:Params: {} 2024-04-12T03:18:45.425 DEBUG:boto:Token: None 2024-04-12T03:18:45.426 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.426 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qQ28bMIrBl/vcf2LDUfHoed+/AY= 2024-04-12T03:18:45.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qQ28bMIrBl/vcf2LDUfHoed+/AY=', 'Content-Length': '0'} 2024-04-12T03:18:45.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3603bda7874d16e-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.470 DEBUG:boto:path=/dkyges-8/_myobj 2024-04-12T03:18:45.470 DEBUG:boto:auth_path=/dkyges-8/_myobj 2024-04-12T03:18:45.471 DEBUG:boto:path=/dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.471 DEBUG:boto:auth_path=/dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.471 DEBUG:boto:Method: GET 2024-04-12T03:18:45.471 DEBUG:boto:Path: /dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.471 DEBUG:boto:Data: 2024-04-12T03:18:45.471 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.471 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.471 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.471 DEBUG:boto:Params: {} 2024-04-12T03:18:45.471 DEBUG:boto:Token: None 2024-04-12T03:18:45.471 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/_myobj?versionId=null 2024-04-12T03:18:45.472 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qQ28bMIrBl/vcf2LDUfHoed+/AY= 2024-04-12T03:18:45.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qQ28bMIrBl/vcf2LDUfHoed+/AY=', 'Content-Length': '0'} 2024-04-12T03:18:45.473 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a73198c93587bdc9-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.474 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.474 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.474 DEBUG:boto:path=/dkyges-8/?prefix=_myobj 2024-04-12T03:18:45.474 DEBUG:boto:auth_path=/dkyges-8/?prefix=_myobj 2024-04-12T03:18:45.474 DEBUG:boto:Method: GET 2024-04-12T03:18:45.474 DEBUG:boto:Path: /dkyges-8/?prefix=_myobj 2024-04-12T03:18:45.474 DEBUG:boto:Data: 2024-04-12T03:18:45.475 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.475 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.475 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.475 DEBUG:boto:Params: {} 2024-04-12T03:18:45.475 DEBUG:boto:Token: None 2024-04-12T03:18:45.475 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.475 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ= 2024-04-12T03:18:45.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f64525ec928cb16-006618a815-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.478 DEBUG:boto:b'dkyges-8_myobj1000false_myobj2024-04-12T03:15:04.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.478 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.479 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.479 DEBUG:boto:path=/dkyges-8/?prefix=_myobj 2024-04-12T03:18:45.479 DEBUG:boto:auth_path=/dkyges-8/?prefix=_myobj 2024-04-12T03:18:45.479 DEBUG:boto:Method: GET 2024-04-12T03:18:45.479 DEBUG:boto:Path: /dkyges-8/?prefix=_myobj 2024-04-12T03:18:45.479 DEBUG:boto:Data: 2024-04-12T03:18:45.479 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.479 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.479 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.479 DEBUG:boto:Params: {} 2024-04-12T03:18:45.479 DEBUG:boto:Token: None 2024-04-12T03:18:45.480 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.480 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ= 2024-04-12T03:18:45.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cebdb9a4d8f528a-006618a815-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.482 DEBUG:boto:b'dkyges-8_myobj1000false_myobj2024-04-12T03:15:04.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.483 DEBUG:boto:path=/dkyges-8/_myobj 2024-04-12T03:18:45.483 DEBUG:boto:auth_path=/dkyges-8/_myobj 2024-04-12T03:18:45.483 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.483 DEBUG:boto:Path: /dkyges-8/_myobj 2024-04-12T03:18:45.483 DEBUG:boto:Data: 2024-04-12T03:18:45.483 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.483 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.483 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.483 DEBUG:boto:Params: {} 2024-04-12T03:18:45.484 DEBUG:boto:Token: None 2024-04-12T03:18:45.484 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/_myobj 2024-04-12T03:18:45.484 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YDBMdAdmETtBCMbDqSh9w31RsuM= 2024-04-12T03:18:45.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YDBMdAdmETtBCMbDqSh9w31RsuM=', 'Content-Length': '0'} 2024-04-12T03:18:45.486 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c60f46dd91b1302c-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.486 DEBUG:boto:path=/dkyges-8/_myobj 2024-04-12T03:18:45.486 DEBUG:boto:auth_path=/dkyges-8/_myobj 2024-04-12T03:18:45.486 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.486 DEBUG:boto:Path: /dkyges-8/_myobj 2024-04-12T03:18:45.486 DEBUG:boto:Data: 2024-04-12T03:18:45.486 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.486 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.487 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.487 DEBUG:boto:Params: {} 2024-04-12T03:18:45.487 DEBUG:boto:Token: None 2024-04-12T03:18:45.487 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/_myobj 2024-04-12T03:18:45.487 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YDBMdAdmETtBCMbDqSh9w31RsuM= 2024-04-12T03:18:45.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YDBMdAdmETtBCMbDqSh9w31RsuM=', 'Content-Length': '0'} 2024-04-12T03:18:45.489 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000227f93e5c18fd785-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2024-04-12T03:18:45.489 DEBUG:boto:path=/dkyges-8/_myobj 2024-04-12T03:18:45.489 DEBUG:boto:auth_path=/dkyges-8/_myobj 2024-04-12T03:18:45.489 DEBUG:boto:Method: GET 2024-04-12T03:18:45.489 DEBUG:boto:Path: /dkyges-8/_myobj 2024-04-12T03:18:45.490 DEBUG:boto:Data: 2024-04-12T03:18:45.490 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.490 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.490 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.490 DEBUG:boto:Params: {} 2024-04-12T03:18:45.490 DEBUG:boto:Token: None 2024-04-12T03:18:45.490 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/_myobj 2024-04-12T03:18:45.490 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:49Y91hUXREMoKLzWv49cnf5eYgc= 2024-04-12T03:18:45.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:49Y91hUXREMoKLzWv49cnf5eYgc=', 'Content-Length': '0'} 2024-04-12T03:18:45.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000085936529096358f-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.493 DEBUG:boto:path=/dkyges-8/_myobj 2024-04-12T03:18:45.493 DEBUG:boto:auth_path=/dkyges-8/_myobj 2024-04-12T03:18:45.493 DEBUG:boto:Method: GET 2024-04-12T03:18:45.493 DEBUG:boto:Path: /dkyges-8/_myobj 2024-04-12T03:18:45.493 DEBUG:boto:Data: 2024-04-12T03:18:45.493 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.493 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.493 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.493 DEBUG:boto:Params: {} 2024-04-12T03:18:45.493 DEBUG:boto:Token: None 2024-04-12T03:18:45.494 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/_myobj 2024-04-12T03:18:45.494 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:49Y91hUXREMoKLzWv49cnf5eYgc= 2024-04-12T03:18:45.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:49Y91hUXREMoKLzWv49cnf5eYgc=', 'Content-Length': '0'} 2024-04-12T03:18:45.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cdf184b31ebb5879-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.538 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.539 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.539 DEBUG:boto:path=/dkyges-8/?prefix=_myobj 2024-04-12T03:18:45.539 DEBUG:boto:auth_path=/dkyges-8/?prefix=_myobj 2024-04-12T03:18:45.539 DEBUG:boto:Method: GET 2024-04-12T03:18:45.539 DEBUG:boto:Path: /dkyges-8/?prefix=_myobj 2024-04-12T03:18:45.539 DEBUG:boto:Data: 2024-04-12T03:18:45.539 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.539 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.539 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.539 DEBUG:boto:Params: {} 2024-04-12T03:18:45.540 DEBUG:boto:Token: None 2024-04-12T03:18:45.540 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.540 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ= 2024-04-12T03:18:45.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009850e608ed1b8161-006618a815-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.542 DEBUG:boto:b'dkyges-8_myobj1000false_myobj2024-04-12T03:15:04.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.543 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.543 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.543 DEBUG:boto:path=/dkyges-8/?prefix=_myobj 2024-04-12T03:18:45.543 DEBUG:boto:auth_path=/dkyges-8/?prefix=_myobj 2024-04-12T03:18:45.543 DEBUG:boto:Method: GET 2024-04-12T03:18:45.543 DEBUG:boto:Path: /dkyges-8/?prefix=_myobj 2024-04-12T03:18:45.543 DEBUG:boto:Data: 2024-04-12T03:18:45.543 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.544 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.544 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.544 DEBUG:boto:Params: {} 2024-04-12T03:18:45.544 DEBUG:boto:Token: None 2024-04-12T03:18:45.544 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.544 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ= 2024-04-12T03:18:45.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ade7dc82eadf8670-006618a815-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.547 DEBUG:boto:b'dkyges-8_myobj1000false_myobj2024-04-12T03:15:04.049Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T03:18:45.547 DEBUG:boto:path=/dkyges-8/myobj 2024-04-12T03:18:45.547 DEBUG:boto:auth_path=/dkyges-8/myobj 2024-04-12T03:18:45.547 DEBUG:boto:path=/dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.548 DEBUG:boto:auth_path=/dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.548 DEBUG:boto:Method: GET 2024-04-12T03:18:45.548 DEBUG:boto:Path: /dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.548 DEBUG:boto:Data: 2024-04-12T03:18:45.548 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.548 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.548 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.548 DEBUG:boto:Params: {} 2024-04-12T03:18:45.548 DEBUG:boto:Token: None 2024-04-12T03:18:45.548 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.549 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b+Obo86zvmb5XY4PPM+5EXSMuuQ= 2024-04-12T03:18:45.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b+Obo86zvmb5XY4PPM+5EXSMuuQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.550 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020a307c9dc1d0184-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.551 DEBUG:boto:path=/dkyges-8/myobj 2024-04-12T03:18:45.551 DEBUG:boto:auth_path=/dkyges-8/myobj 2024-04-12T03:18:45.551 DEBUG:boto:path=/dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.551 DEBUG:boto:auth_path=/dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.551 DEBUG:boto:Method: GET 2024-04-12T03:18:45.551 DEBUG:boto:Path: /dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.552 DEBUG:boto:Data: 2024-04-12T03:18:45.552 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.552 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.552 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.552 DEBUG:boto:Params: {} 2024-04-12T03:18:45.552 DEBUG:boto:Token: None 2024-04-12T03:18:45.552 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.552 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b+Obo86zvmb5XY4PPM+5EXSMuuQ= 2024-04-12T03:18:45.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b+Obo86zvmb5XY4PPM+5EXSMuuQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c635a606bf7cfc0-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.554 DEBUG:boto:path=/dkyges-8/myobj 2024-04-12T03:18:45.555 DEBUG:boto:auth_path=/dkyges-8/myobj 2024-04-12T03:18:45.555 DEBUG:boto:path=/dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.555 DEBUG:boto:auth_path=/dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.555 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.555 DEBUG:boto:Path: /dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.555 DEBUG:boto:Data: 2024-04-12T03:18:45.555 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.555 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.555 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.555 DEBUG:boto:Params: {} 2024-04-12T03:18:45.555 DEBUG:boto:Token: None 2024-04-12T03:18:45.556 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.556 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m5p9NO7+i6ZfED9zysQSYWRTKYA= 2024-04-12T03:18:45.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m5p9NO7+i6ZfED9zysQSYWRTKYA=', 'Content-Length': '0'} 2024-04-12T03:18:45.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000275ecddee4c14b24-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.557 DEBUG:boto:path=/dkyges-8/myobj 2024-04-12T03:18:45.558 DEBUG:boto:auth_path=/dkyges-8/myobj 2024-04-12T03:18:45.558 DEBUG:boto:path=/dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.558 DEBUG:boto:auth_path=/dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.558 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.558 DEBUG:boto:Path: /dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.558 DEBUG:boto:Data: 2024-04-12T03:18:45.558 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.558 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.558 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.558 DEBUG:boto:Params: {} 2024-04-12T03:18:45.558 DEBUG:boto:Token: None 2024-04-12T03:18:45.559 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.559 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m5p9NO7+i6ZfED9zysQSYWRTKYA= 2024-04-12T03:18:45.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m5p9NO7+i6ZfED9zysQSYWRTKYA=', 'Content-Length': '0'} 2024-04-12T03:18:45.560 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b4366535edab970-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T03:18:45.561 DEBUG:boto:path=/dkyges-8/myobj 2024-04-12T03:18:45.561 DEBUG:boto:auth_path=/dkyges-8/myobj 2024-04-12T03:18:45.561 DEBUG:boto:path=/dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.561 DEBUG:boto:auth_path=/dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.561 DEBUG:boto:Method: GET 2024-04-12T03:18:45.561 DEBUG:boto:Path: /dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.561 DEBUG:boto:Data: 2024-04-12T03:18:45.562 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.562 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.562 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.562 DEBUG:boto:Params: {} 2024-04-12T03:18:45.562 DEBUG:boto:Token: None 2024-04-12T03:18:45.562 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.562 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b+Obo86zvmb5XY4PPM+5EXSMuuQ= 2024-04-12T03:18:45.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b+Obo86zvmb5XY4PPM+5EXSMuuQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.564 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1ff7bfd904f4322-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.606 DEBUG:boto:path=/dkyges-8/myobj 2024-04-12T03:18:45.607 DEBUG:boto:auth_path=/dkyges-8/myobj 2024-04-12T03:18:45.607 DEBUG:boto:path=/dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.607 DEBUG:boto:auth_path=/dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.607 DEBUG:boto:Method: GET 2024-04-12T03:18:45.607 DEBUG:boto:Path: /dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.607 DEBUG:boto:Data: 2024-04-12T03:18:45.607 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.607 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.607 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.607 DEBUG:boto:Params: {} 2024-04-12T03:18:45.608 DEBUG:boto:Token: None 2024-04-12T03:18:45.608 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/myobj?versionId=null 2024-04-12T03:18:45.608 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b+Obo86zvmb5XY4PPM+5EXSMuuQ= 2024-04-12T03:18:45.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b+Obo86zvmb5XY4PPM+5EXSMuuQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.610 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efa194cbf3dd2ee7-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.610 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.610 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.610 DEBUG:boto:path=/dkyges-8/?prefix=myobj 2024-04-12T03:18:45.611 DEBUG:boto:auth_path=/dkyges-8/?prefix=myobj 2024-04-12T03:18:45.611 DEBUG:boto:Method: GET 2024-04-12T03:18:45.611 DEBUG:boto:Path: /dkyges-8/?prefix=myobj 2024-04-12T03:18:45.611 DEBUG:boto:Data: 2024-04-12T03:18:45.611 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.611 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.611 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.611 DEBUG:boto:Params: {} 2024-04-12T03:18:45.611 DEBUG:boto:Token: None 2024-04-12T03:18:45.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.611 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ= 2024-04-12T03:18:45.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c2808dea01d0bfe-006618a815-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.614 DEBUG:boto:b'dkyges-8myobj1000falsemyobj2024-04-12T03:15:04.002Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.614 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.615 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.615 DEBUG:boto:path=/dkyges-8/?prefix=myobj 2024-04-12T03:18:45.615 DEBUG:boto:auth_path=/dkyges-8/?prefix=myobj 2024-04-12T03:18:45.615 DEBUG:boto:Method: GET 2024-04-12T03:18:45.615 DEBUG:boto:Path: /dkyges-8/?prefix=myobj 2024-04-12T03:18:45.615 DEBUG:boto:Data: 2024-04-12T03:18:45.615 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.615 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.615 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.615 DEBUG:boto:Params: {} 2024-04-12T03:18:45.615 DEBUG:boto:Token: None 2024-04-12T03:18:45.616 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.616 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ= 2024-04-12T03:18:45.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020bf78af721b1478-006618a815-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.618 DEBUG:boto:b'dkyges-8myobj1000falsemyobj2024-04-12T03:15:04.002Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.619 DEBUG:boto:path=/dkyges-8/myobj 2024-04-12T03:18:45.619 DEBUG:boto:auth_path=/dkyges-8/myobj 2024-04-12T03:18:45.619 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.619 DEBUG:boto:Path: /dkyges-8/myobj 2024-04-12T03:18:45.619 DEBUG:boto:Data: 2024-04-12T03:18:45.619 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.619 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.619 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.619 DEBUG:boto:Params: {} 2024-04-12T03:18:45.619 DEBUG:boto:Token: None 2024-04-12T03:18:45.620 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/myobj 2024-04-12T03:18:45.620 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JY6L8zWfmLNu0qcxsOpM/vQJ9GM= 2024-04-12T03:18:45.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JY6L8zWfmLNu0qcxsOpM/vQJ9GM=', 'Content-Length': '0'} 2024-04-12T03:18:45.621 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b7642405416758d-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.622 DEBUG:boto:path=/dkyges-8/myobj 2024-04-12T03:18:45.622 DEBUG:boto:auth_path=/dkyges-8/myobj 2024-04-12T03:18:45.622 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.622 DEBUG:boto:Path: /dkyges-8/myobj 2024-04-12T03:18:45.622 DEBUG:boto:Data: 2024-04-12T03:18:45.622 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.622 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.622 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.622 DEBUG:boto:Params: {} 2024-04-12T03:18:45.623 DEBUG:boto:Token: None 2024-04-12T03:18:45.623 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/myobj 2024-04-12T03:18:45.623 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JY6L8zWfmLNu0qcxsOpM/vQJ9GM= 2024-04-12T03:18:45.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JY6L8zWfmLNu0qcxsOpM/vQJ9GM=', 'Content-Length': '0'} 2024-04-12T03:18:45.624 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ce7452d7341096b-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T03:18:45.625 DEBUG:boto:path=/dkyges-8/myobj 2024-04-12T03:18:45.625 DEBUG:boto:auth_path=/dkyges-8/myobj 2024-04-12T03:18:45.625 DEBUG:boto:Method: GET 2024-04-12T03:18:45.625 DEBUG:boto:Path: /dkyges-8/myobj 2024-04-12T03:18:45.625 DEBUG:boto:Data: 2024-04-12T03:18:45.625 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.625 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.625 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.626 DEBUG:boto:Params: {} 2024-04-12T03:18:45.626 DEBUG:boto:Token: None 2024-04-12T03:18:45.626 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/myobj 2024-04-12T03:18:45.626 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U1A8p2/LnBWsDxOYIrNk9jKPY90= 2024-04-12T03:18:45.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U1A8p2/LnBWsDxOYIrNk9jKPY90=', 'Content-Length': '0'} 2024-04-12T03:18:45.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000709e93073314d0c8-006618a815-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.628 DEBUG:boto:path=/dkyges-8/myobj 2024-04-12T03:18:45.628 DEBUG:boto:auth_path=/dkyges-8/myobj 2024-04-12T03:18:45.628 DEBUG:boto:Method: GET 2024-04-12T03:18:45.628 DEBUG:boto:Path: /dkyges-8/myobj 2024-04-12T03:18:45.628 DEBUG:boto:Data: 2024-04-12T03:18:45.628 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.628 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.628 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.629 DEBUG:boto:Params: {} 2024-04-12T03:18:45.629 DEBUG:boto:Token: None 2024-04-12T03:18:45.629 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/myobj 2024-04-12T03:18:45.629 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U1A8p2/LnBWsDxOYIrNk9jKPY90= 2024-04-12T03:18:45.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U1A8p2/LnBWsDxOYIrNk9jKPY90=', 'Content-Length': '0'} 2024-04-12T03:18:45.631 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:15:04 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f72685c627b4b35-006618a815-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.631 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.631 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.631 DEBUG:boto:path=/dkyges-8/?prefix=myobj 2024-04-12T03:18:45.631 DEBUG:boto:auth_path=/dkyges-8/?prefix=myobj 2024-04-12T03:18:45.631 DEBUG:boto:Method: GET 2024-04-12T03:18:45.631 DEBUG:boto:Path: /dkyges-8/?prefix=myobj 2024-04-12T03:18:45.631 DEBUG:boto:Data: 2024-04-12T03:18:45.631 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.632 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.632 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.632 DEBUG:boto:Params: {} 2024-04-12T03:18:45.632 DEBUG:boto:Token: None 2024-04-12T03:18:45.632 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.632 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ= 2024-04-12T03:18:45.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb692207f8d0bbbc-006618a815-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.634 DEBUG:boto:b'dkyges-8myobj1000falsemyobj2024-04-12T03:15:04.002Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.635 DEBUG:boto:path=/dkyges-8/ 2024-04-12T03:18:45.635 DEBUG:boto:auth_path=/dkyges-8/ 2024-04-12T03:18:45.635 DEBUG:boto:path=/dkyges-8/?prefix=myobj 2024-04-12T03:18:45.635 DEBUG:boto:auth_path=/dkyges-8/?prefix=myobj 2024-04-12T03:18:45.635 DEBUG:boto:Method: GET 2024-04-12T03:18:45.635 DEBUG:boto:Path: /dkyges-8/?prefix=myobj 2024-04-12T03:18:45.635 DEBUG:boto:Data: 2024-04-12T03:18:45.635 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.636 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.636 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.636 DEBUG:boto:Params: {} 2024-04-12T03:18:45.636 DEBUG:boto:Token: None 2024-04-12T03:18:45.636 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:18:45 GMT /dkyges-8/ 2024-04-12T03:18:45.636 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ= 2024-04-12T03:18:45.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2XLl0vtPc9hI+l5fD/zcFB9/sZQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050a37c4977dce30e-006618a815-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.639 DEBUG:boto:b'dkyges-8myobj1000falsemyobj2024-04-12T03:15:04.002Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:18:45.639 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-8 zones={a1, a2} 2024-04-12T03:18:45.640 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2024-04-12T03:18:45.641 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-9 2024-04-12T03:18:45.641 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:18:45.641 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:18:45.641 DEBUG:boto:Method: PUT 2024-04-12T03:18:45.641 DEBUG:boto:Path: /dkyges-9/ 2024-04-12T03:18:45.641 DEBUG:boto:Data: 2024-04-12T03:18:45.641 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.641 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.641 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.642 DEBUG:boto:Params: {} 2024-04-12T03:18:45.642 DEBUG:boto:Token: None 2024-04-12T03:18:45.642 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:18:45 GMT /dkyges-9/ 2024-04-12T03:18:45.642 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SBi2fyR/mYEAEMcYjsdLA5+wgkE= 2024-04-12T03:18:45.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SBi2fyR/mYEAEMcYjsdLA5+wgkE=', 'Content-Length': '0'} 2024-04-12T03:18:45.658 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fdc117fa2919f455-006618a815-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.658 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-10 2024-04-12T03:18:45.658 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:18:45.658 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:18:45.658 DEBUG:boto:Method: PUT 2024-04-12T03:18:45.658 DEBUG:boto:Path: /dkyges-10/ 2024-04-12T03:18:45.658 DEBUG:boto:Data: 2024-04-12T03:18:45.658 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.659 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.659 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.659 DEBUG:boto:Params: {} 2024-04-12T03:18:45.659 DEBUG:boto:Token: None 2024-04-12T03:18:45.659 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:18:45 GMT /dkyges-10/ 2024-04-12T03:18:45.659 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UeqyaujXqKntItWuoIE05jfZNwY= 2024-04-12T03:18:45.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UeqyaujXqKntItWuoIE05jfZNwY=', 'Content-Length': '0'} 2024-04-12T03:18:45.685 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d1196d6274566cf1-006618a815-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.685 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:18:45.685 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:18:45.685 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.686 DEBUG:boto:Path: /dkyges-9/ 2024-04-12T03:18:45.686 DEBUG:boto:Data: 2024-04-12T03:18:45.686 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.686 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.686 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.686 DEBUG:boto:Params: {} 2024-04-12T03:18:45.686 DEBUG:boto:Token: None 2024-04-12T03:18:45.686 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-9/ 2024-04-12T03:18:45.687 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UY+IAappIWMWWVhWSNdKso1j8LQ= 2024-04-12T03:18:45.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UY+IAappIWMWWVhWSNdKso1j8LQ=', 'Content-Length': '0'} 2024-04-12T03:18:45.689 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b55d0f9a8be55961-006618a815-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.689 DEBUG:boto:path=/dkyges-9/myobj 2024-04-12T03:18:45.690 DEBUG:boto:auth_path=/dkyges-9/myobj 2024-04-12T03:18:45.690 DEBUG:boto:Method: PUT 2024-04-12T03:18:45.690 DEBUG:boto:Path: /dkyges-9/myobj 2024-04-12T03:18:45.690 DEBUG:boto:Data: 2024-04-12T03:18:45.691 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:18:45.691 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:18:45.691 DEBUG:boto:Port: 8000 2024-04-12T03:18:45.691 DEBUG:boto:Params: {} 2024-04-12T03:18:45.691 DEBUG:boto:Token: None 2024-04-12T03:18:45.691 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:18:45 GMT /dkyges-9/myobj 2024-04-12T03:18:45.691 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TDuBmHC0mWA+yqKn8DaFJEZmxd4= 2024-04-12T03:18:45.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TDuBmHC0mWA+yqKn8DaFJEZmxd4='} 2024-04-12T03:18:45.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b09411b44fdf3dfd-006618a815-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.739 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:18:45.739 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:18:45.739 DEBUG:boto:Method: HEAD 2024-04-12T03:18:45.739 DEBUG:boto:Path: /dkyges-10/ 2024-04-12T03:18:45.739 DEBUG:boto:Data: 2024-04-12T03:18:45.739 DEBUG:boto:Headers: {} 2024-04-12T03:18:45.739 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.739 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.740 DEBUG:boto:Params: {} 2024-04-12T03:18:45.740 DEBUG:boto:Token: None 2024-04-12T03:18:45.740 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:18:45 GMT /dkyges-10/ 2024-04-12T03:18:45.740 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:g9cW9oy2sqvf4+jXOovoWLplnkk= 2024-04-12T03:18:45.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:g9cW9oy2sqvf4+jXOovoWLplnkk=', 'Content-Length': '0'} 2024-04-12T03:18:45.743 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d5ddf75467c7ab5-006618a815-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.744 DEBUG:boto:path=/dkyges-10/myobj 2024-04-12T03:18:45.744 DEBUG:boto:auth_path=/dkyges-10/myobj 2024-04-12T03:18:45.744 DEBUG:boto:Method: PUT 2024-04-12T03:18:45.744 DEBUG:boto:Path: /dkyges-10/myobj 2024-04-12T03:18:45.744 DEBUG:boto:Data: 2024-04-12T03:18:45.744 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:18:45.744 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:18:45.744 DEBUG:boto:Port: 8001 2024-04-12T03:18:45.744 DEBUG:boto:Params: {} 2024-04-12T03:18:45.744 DEBUG:boto:Token: None 2024-04-12T03:18:45.745 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:18:45 GMT /dkyges-10/myobj 2024-04-12T03:18:45.745 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:urz80yToDj7fEZDKnC/l07mTgd4= 2024-04-12T03:18:45.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:18:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:urz80yToDj7fEZDKnC/l07mTgd4='} 2024-04-12T03:18:45.752 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e240dcfa97ed549-006618a815-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:18:45.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:45.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:45.753 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:18:45.812 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:18:45.812 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:18:45.847 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:18:45.860 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:18:45.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:18:45.861 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:18:45.861 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:18:45.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:18:45.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:18:45.861 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:18:45.861 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:18:45.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:18:45.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:18:45.861 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:18:45.862 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:18:45.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:18:45.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:18:45.862 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.674231_47.1", 2024-04-12T03:18:45.862 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.674231Z" 2024-04-12T03:18:45.862 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:18:45.862 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:18:45.863 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712891925.674231_47.1'} 2024-04-12T03:18:45.863 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:18:45.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:45.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:18:45.864 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:18:45.919 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:18:45.919 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:18:45.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:45.951+0000 7f0568bf5780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:45.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:45.953+0000 7f0568bf5780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:18:45.967 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:18:45.968 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:18:45.968 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:18:45.968 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:18:45.968 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:18:45.968 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:18:45.968 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:18:45.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:18:45.968 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:18:45.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:18:45.969 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:18:45.969 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:18:45.969 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:18:45.969 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:18:45.969 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:18:45.969 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:18:45.969 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:18:45.969 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:18:45.969 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:18:45.969 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:18:45.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:18:45.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:18:45.970 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:18:45.970 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:18:45.970 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:18:45.970 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:18:45.970 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:18:45.970 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:18:45.970 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:18:45.970 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:18:45.970 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:18:45.970 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:18:45.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:18:45.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:18:45.971 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:18:45.971 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:18:45.971 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:18:45.971 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:18:45.971 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:18:45.971 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:18:45.971 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:18:45.971 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:18:45.971 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:18:45.971 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:18:45.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:18:45.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:18:45.972 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:18:45.972 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:18:45.972 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:18:45.972 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.674231_47.1", 2024-04-12T03:18:45.972 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:18:45.972 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:18:45.972 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:18:45.972 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.674231Z", 2024-04-12T03:18:45.972 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:18:45.972 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:18:45.972 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:18:45.973 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:18:45.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:18:45.973 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:18:45.973 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:18:45.973 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:18:45.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:18:45.973 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:18:45Z" 2024-04-12T03:18:45.973 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:18:45.974 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891925.674231_47.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.674231Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:18:45Z" } 2024-04-12T03:18:45.974 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891925.674231_47.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.674231Z', 'realm_epoch': 2}}] 2024-04-12T03:18:45.974 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712891925.674231_47.1'} 2024-04-12T03:18:45.974 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712891925.674231_47.1'} 2024-04-12T03:18:45.975 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:18:45.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:18:45.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:18:45.975 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:18:46.039 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:18:46.040 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:18:47.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:47.076+0000 7f7dbb7ef780 1 waiting to reach incremental sync.. 2024-04-12T03:18:52.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:52.077+0000 7f7dbb7ef780 1 waiting to reach incremental sync.. 2024-04-12T03:18:57.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:18:57.078+0000 7f7dbb7ef780 1 waiting to reach incremental sync.. 2024-04-12T03:19:02.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:02.080+0000 7f7dbb7ef780 1 bucket sync caught up with source: 2024-04-12T03:19:02.081 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2024-04-12T03:19:02.081 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.5.6, , , , , , , , , ] 2024-04-12T03:19:02.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:02.080+0000 7f7dbb7ef780 0 bucket checkpoint complete 2024-04-12T03:19:02.102 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-9 zones={a2, a1} 2024-04-12T03:19:02.102 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:02.102 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:02.102 DEBUG:boto:Method: HEAD 2024-04-12T03:19:02.103 DEBUG:boto:Path: /dkyges-9/ 2024-04-12T03:19:02.103 DEBUG:boto:Data: 2024-04-12T03:19:02.103 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.103 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.103 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.103 DEBUG:boto:Params: {} 2024-04-12T03:19:02.103 DEBUG:boto:Token: None 2024-04-12T03:19:02.103 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/ 2024-04-12T03:19:02.104 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mtPftfSNDHSgAcE1mgJVDnw+GPg= 2024-04-12T03:19:02.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mtPftfSNDHSgAcE1mgJVDnw+GPg=', 'Content-Length': '0'} 2024-04-12T03:19:02.106 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1a386798718d59d-006618a826-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.106 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:02.106 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:02.107 DEBUG:boto:Method: HEAD 2024-04-12T03:19:02.107 DEBUG:boto:Path: /dkyges-9/ 2024-04-12T03:19:02.107 DEBUG:boto:Data: 2024-04-12T03:19:02.107 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.107 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.107 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.107 DEBUG:boto:Params: {} 2024-04-12T03:19:02.107 DEBUG:boto:Token: None 2024-04-12T03:19:02.107 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/ 2024-04-12T03:19:02.107 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mtPftfSNDHSgAcE1mgJVDnw+GPg= 2024-04-12T03:19:02.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mtPftfSNDHSgAcE1mgJVDnw+GPg=', 'Content-Length': '0'} 2024-04-12T03:19:02.110 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc274910fdc1a019-006618a826-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.110 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:19:02.110 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:02.110 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:02.110 DEBUG:boto:path=/dkyges-9/?versions&max-keys=999 2024-04-12T03:19:02.110 DEBUG:boto:auth_path=/dkyges-9/?versions&max-keys=999 2024-04-12T03:19:02.111 DEBUG:boto:Method: GET 2024-04-12T03:19:02.111 DEBUG:boto:Path: /dkyges-9/?versions&max-keys=999 2024-04-12T03:19:02.111 DEBUG:boto:Data: 2024-04-12T03:19:02.111 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.111 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.111 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.111 DEBUG:boto:Params: {} 2024-04-12T03:19:02.111 DEBUG:boto:Token: None 2024-04-12T03:19:02.111 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/?versions 2024-04-12T03:19:02.112 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6jnbcNsTP6QIMYzMI/tY8xOY/Ms= 2024-04-12T03:19:02.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6jnbcNsTP6QIMYzMI/tY8xOY/Ms=', 'Content-Length': '0'} 2024-04-12T03:19:02.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d95a2d4df95d9a1-006618a826-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.114 DEBUG:boto:b'dkyges-9999falsemyobjnulltrue2024-04-12T03:18:45.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:19:02.115 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-12T03:19:02.115 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:19:02.115 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:02.115 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:02.115 DEBUG:boto:path=/dkyges-9/?versions&max-keys=999 2024-04-12T03:19:02.115 DEBUG:boto:auth_path=/dkyges-9/?versions&max-keys=999 2024-04-12T03:19:02.115 DEBUG:boto:Method: GET 2024-04-12T03:19:02.115 DEBUG:boto:Path: /dkyges-9/?versions&max-keys=999 2024-04-12T03:19:02.115 DEBUG:boto:Data: 2024-04-12T03:19:02.115 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.116 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.116 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.116 DEBUG:boto:Params: {} 2024-04-12T03:19:02.116 DEBUG:boto:Token: None 2024-04-12T03:19:02.116 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/?versions 2024-04-12T03:19:02.116 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6jnbcNsTP6QIMYzMI/tY8xOY/Ms= 2024-04-12T03:19:02.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6jnbcNsTP6QIMYzMI/tY8xOY/Ms=', 'Content-Length': '0'} 2024-04-12T03:19:02.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d83046b73239fad-006618a826-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.118 DEBUG:boto:b'dkyges-9999falsemyobjnulltrue2024-04-12T03:18:45.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:19:02.119 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-12T03:19:02.119 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:02.119 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:02.119 DEBUG:boto:path=/dkyges-9/?versions&max-keys=999 2024-04-12T03:19:02.119 DEBUG:boto:auth_path=/dkyges-9/?versions&max-keys=999 2024-04-12T03:19:02.119 DEBUG:boto:Method: GET 2024-04-12T03:19:02.119 DEBUG:boto:Path: /dkyges-9/?versions&max-keys=999 2024-04-12T03:19:02.119 DEBUG:boto:Data: 2024-04-12T03:19:02.120 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.120 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.120 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.120 DEBUG:boto:Params: {} 2024-04-12T03:19:02.120 DEBUG:boto:Token: None 2024-04-12T03:19:02.120 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/?versions 2024-04-12T03:19:02.120 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6jnbcNsTP6QIMYzMI/tY8xOY/Ms= 2024-04-12T03:19:02.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6jnbcNsTP6QIMYzMI/tY8xOY/Ms=', 'Content-Length': '0'} 2024-04-12T03:19:02.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071af5a2497d278e7-006618a826-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.123 DEBUG:boto:b'dkyges-9999falsemyobjnulltrue2024-04-12T03:18:45.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:19:02.123 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:02.123 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:02.123 DEBUG:boto:path=/dkyges-9/?versions&max-keys=999 2024-04-12T03:19:02.123 DEBUG:boto:auth_path=/dkyges-9/?versions&max-keys=999 2024-04-12T03:19:02.123 DEBUG:boto:Method: GET 2024-04-12T03:19:02.124 DEBUG:boto:Path: /dkyges-9/?versions&max-keys=999 2024-04-12T03:19:02.124 DEBUG:boto:Data: 2024-04-12T03:19:02.124 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.124 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.124 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.124 DEBUG:boto:Params: {} 2024-04-12T03:19:02.124 DEBUG:boto:Token: None 2024-04-12T03:19:02.124 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/?versions 2024-04-12T03:19:02.124 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6jnbcNsTP6QIMYzMI/tY8xOY/Ms= 2024-04-12T03:19:02.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6jnbcNsTP6QIMYzMI/tY8xOY/Ms=', 'Content-Length': '0'} 2024-04-12T03:19:02.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000758d6c7cd96713f4-006618a826-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.126 DEBUG:boto:b'dkyges-9999falsemyobjnulltrue2024-04-12T03:18:45.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:19:02.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T03:19:02.127 DEBUG:boto:path=/dkyges-9/myobj 2024-04-12T03:19:02.127 DEBUG:boto:auth_path=/dkyges-9/myobj 2024-04-12T03:19:02.127 DEBUG:boto:path=/dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.127 DEBUG:boto:auth_path=/dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.127 DEBUG:boto:Method: GET 2024-04-12T03:19:02.127 DEBUG:boto:Path: /dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.127 DEBUG:boto:Data: 2024-04-12T03:19:02.128 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.128 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.128 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.128 DEBUG:boto:Params: {} 2024-04-12T03:19:02.128 DEBUG:boto:Token: None 2024-04-12T03:19:02.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.128 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U6ii02XEd10V+ExbcUXr9QIzfjE= 2024-04-12T03:19:02.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U6ii02XEd10V+ExbcUXr9QIzfjE=', 'Content-Length': '0'} 2024-04-12T03:19:02.130 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021909090adc460fe-006618a826-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.130 DEBUG:boto:path=/dkyges-9/myobj 2024-04-12T03:19:02.131 DEBUG:boto:auth_path=/dkyges-9/myobj 2024-04-12T03:19:02.131 DEBUG:boto:path=/dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.131 DEBUG:boto:auth_path=/dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.131 DEBUG:boto:Method: GET 2024-04-12T03:19:02.131 DEBUG:boto:Path: /dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.131 DEBUG:boto:Data: 2024-04-12T03:19:02.131 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.131 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.131 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.131 DEBUG:boto:Params: {} 2024-04-12T03:19:02.131 DEBUG:boto:Token: None 2024-04-12T03:19:02.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.132 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U6ii02XEd10V+ExbcUXr9QIzfjE= 2024-04-12T03:19:02.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U6ii02XEd10V+ExbcUXr9QIzfjE=', 'Content-Length': '0'} 2024-04-12T03:19:02.133 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0e2900c113790b0-006618a826-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.133 DEBUG:boto:path=/dkyges-9/myobj 2024-04-12T03:19:02.134 DEBUG:boto:auth_path=/dkyges-9/myobj 2024-04-12T03:19:02.134 DEBUG:boto:path=/dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.134 DEBUG:boto:auth_path=/dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.134 DEBUG:boto:Method: HEAD 2024-04-12T03:19:02.134 DEBUG:boto:Path: /dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.134 DEBUG:boto:Data: 2024-04-12T03:19:02.134 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.134 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.134 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.134 DEBUG:boto:Params: {} 2024-04-12T03:19:02.135 DEBUG:boto:Token: None 2024-04-12T03:19:02.135 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.135 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rROV8w9dsrzYOCzbGYABNC4Hokw= 2024-04-12T03:19:02.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rROV8w9dsrzYOCzbGYABNC4Hokw=', 'Content-Length': '0'} 2024-04-12T03:19:02.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8237fc244a6f453-006618a826-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.137 DEBUG:boto:path=/dkyges-9/myobj 2024-04-12T03:19:02.137 DEBUG:boto:auth_path=/dkyges-9/myobj 2024-04-12T03:19:02.137 DEBUG:boto:path=/dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.137 DEBUG:boto:auth_path=/dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.137 DEBUG:boto:Method: HEAD 2024-04-12T03:19:02.137 DEBUG:boto:Path: /dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.137 DEBUG:boto:Data: 2024-04-12T03:19:02.137 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.137 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.137 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.138 DEBUG:boto:Params: {} 2024-04-12T03:19:02.138 DEBUG:boto:Token: None 2024-04-12T03:19:02.138 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.138 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rROV8w9dsrzYOCzbGYABNC4Hokw= 2024-04-12T03:19:02.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rROV8w9dsrzYOCzbGYABNC4Hokw=', 'Content-Length': '0'} 2024-04-12T03:19:02.140 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099fc3984145995a5-006618a826-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T03:19:02.140 DEBUG:boto:path=/dkyges-9/myobj 2024-04-12T03:19:02.140 DEBUG:boto:auth_path=/dkyges-9/myobj 2024-04-12T03:19:02.140 DEBUG:boto:path=/dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.140 DEBUG:boto:auth_path=/dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.140 DEBUG:boto:Method: GET 2024-04-12T03:19:02.141 DEBUG:boto:Path: /dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.141 DEBUG:boto:Data: 2024-04-12T03:19:02.141 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.141 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.141 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.141 DEBUG:boto:Params: {} 2024-04-12T03:19:02.141 DEBUG:boto:Token: None 2024-04-12T03:19:02.141 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.141 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U6ii02XEd10V+ExbcUXr9QIzfjE= 2024-04-12T03:19:02.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U6ii02XEd10V+ExbcUXr9QIzfjE=', 'Content-Length': '0'} 2024-04-12T03:19:02.143 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000068abb4061ec28db-006618a826-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.186 DEBUG:boto:path=/dkyges-9/myobj 2024-04-12T03:19:02.186 DEBUG:boto:auth_path=/dkyges-9/myobj 2024-04-12T03:19:02.187 DEBUG:boto:path=/dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.187 DEBUG:boto:auth_path=/dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.187 DEBUG:boto:Method: GET 2024-04-12T03:19:02.187 DEBUG:boto:Path: /dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.187 DEBUG:boto:Data: 2024-04-12T03:19:02.187 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.187 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.187 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.188 DEBUG:boto:Params: {} 2024-04-12T03:19:02.188 DEBUG:boto:Token: None 2024-04-12T03:19:02.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/myobj?versionId=null 2024-04-12T03:19:02.188 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U6ii02XEd10V+ExbcUXr9QIzfjE= 2024-04-12T03:19:02.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U6ii02XEd10V+ExbcUXr9QIzfjE=', 'Content-Length': '0'} 2024-04-12T03:19:02.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bb3230e4cba02c2-006618a826-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.190 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:02.191 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:02.191 DEBUG:boto:path=/dkyges-9/?prefix=myobj 2024-04-12T03:19:02.191 DEBUG:boto:auth_path=/dkyges-9/?prefix=myobj 2024-04-12T03:19:02.191 DEBUG:boto:Method: GET 2024-04-12T03:19:02.191 DEBUG:boto:Path: /dkyges-9/?prefix=myobj 2024-04-12T03:19:02.191 DEBUG:boto:Data: 2024-04-12T03:19:02.191 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.191 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.191 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.191 DEBUG:boto:Params: {} 2024-04-12T03:19:02.191 DEBUG:boto:Token: None 2024-04-12T03:19:02.192 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/ 2024-04-12T03:19:02.192 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u/IbMiLhKaiCcMkuriG+kplN1gU= 2024-04-12T03:19:02.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u/IbMiLhKaiCcMkuriG+kplN1gU=', 'Content-Length': '0'} 2024-04-12T03:19:02.194 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba2baf71ab6ad91b-006618a826-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.195 DEBUG:boto:b'dkyges-9myobj1000falsemyobj2024-04-12T03:18:45.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:19:02.197 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:02.197 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:02.197 DEBUG:boto:path=/dkyges-9/?prefix=myobj 2024-04-12T03:19:02.197 DEBUG:boto:auth_path=/dkyges-9/?prefix=myobj 2024-04-12T03:19:02.198 DEBUG:boto:Method: GET 2024-04-12T03:19:02.198 DEBUG:boto:Path: /dkyges-9/?prefix=myobj 2024-04-12T03:19:02.198 DEBUG:boto:Data: 2024-04-12T03:19:02.198 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.198 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.198 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.198 DEBUG:boto:Params: {} 2024-04-12T03:19:02.198 DEBUG:boto:Token: None 2024-04-12T03:19:02.198 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/ 2024-04-12T03:19:02.199 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u/IbMiLhKaiCcMkuriG+kplN1gU= 2024-04-12T03:19:02.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u/IbMiLhKaiCcMkuriG+kplN1gU=', 'Content-Length': '0'} 2024-04-12T03:19:02.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099c942c7715bcd74-006618a826-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.242 DEBUG:boto:b'dkyges-9myobj1000falsemyobj2024-04-12T03:18:45.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:19:02.243 DEBUG:boto:path=/dkyges-9/myobj 2024-04-12T03:19:02.243 DEBUG:boto:auth_path=/dkyges-9/myobj 2024-04-12T03:19:02.243 DEBUG:boto:Method: HEAD 2024-04-12T03:19:02.243 DEBUG:boto:Path: /dkyges-9/myobj 2024-04-12T03:19:02.243 DEBUG:boto:Data: 2024-04-12T03:19:02.243 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.243 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.243 DEBUG:boto:Params: {} 2024-04-12T03:19:02.244 DEBUG:boto:Token: None 2024-04-12T03:19:02.244 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/myobj 2024-04-12T03:19:02.244 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y4pnqLgpUyM+/3P6vI9fHx/Tncg= 2024-04-12T03:19:02.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y4pnqLgpUyM+/3P6vI9fHx/Tncg=', 'Content-Length': '0'} 2024-04-12T03:19:02.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001436c04e5c3c1893-006618a826-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.246 DEBUG:boto:path=/dkyges-9/myobj 2024-04-12T03:19:02.246 DEBUG:boto:auth_path=/dkyges-9/myobj 2024-04-12T03:19:02.247 DEBUG:boto:Method: HEAD 2024-04-12T03:19:02.247 DEBUG:boto:Path: /dkyges-9/myobj 2024-04-12T03:19:02.247 DEBUG:boto:Data: 2024-04-12T03:19:02.247 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.247 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.247 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.247 DEBUG:boto:Params: {} 2024-04-12T03:19:02.247 DEBUG:boto:Token: None 2024-04-12T03:19:02.247 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/myobj 2024-04-12T03:19:02.247 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y4pnqLgpUyM+/3P6vI9fHx/Tncg= 2024-04-12T03:19:02.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y4pnqLgpUyM+/3P6vI9fHx/Tncg=', 'Content-Length': '0'} 2024-04-12T03:19:02.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009bea5c9d7f25c97f-006618a826-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T03:19:02.249 DEBUG:boto:path=/dkyges-9/myobj 2024-04-12T03:19:02.250 DEBUG:boto:auth_path=/dkyges-9/myobj 2024-04-12T03:19:02.250 DEBUG:boto:Method: GET 2024-04-12T03:19:02.250 DEBUG:boto:Path: /dkyges-9/myobj 2024-04-12T03:19:02.250 DEBUG:boto:Data: 2024-04-12T03:19:02.250 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.250 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.250 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.250 DEBUG:boto:Params: {} 2024-04-12T03:19:02.250 DEBUG:boto:Token: None 2024-04-12T03:19:02.251 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/myobj 2024-04-12T03:19:02.251 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8GP5TGXS9A2FTPmmdBKwLEOeoHs= 2024-04-12T03:19:02.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8GP5TGXS9A2FTPmmdBKwLEOeoHs=', 'Content-Length': '0'} 2024-04-12T03:19:02.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009cdd83bbb273b4ca-006618a826-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.253 DEBUG:boto:path=/dkyges-9/myobj 2024-04-12T03:19:02.253 DEBUG:boto:auth_path=/dkyges-9/myobj 2024-04-12T03:19:02.253 DEBUG:boto:Method: GET 2024-04-12T03:19:02.253 DEBUG:boto:Path: /dkyges-9/myobj 2024-04-12T03:19:02.253 DEBUG:boto:Data: 2024-04-12T03:19:02.254 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.254 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.254 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.254 DEBUG:boto:Params: {} 2024-04-12T03:19:02.254 DEBUG:boto:Token: None 2024-04-12T03:19:02.254 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/myobj 2024-04-12T03:19:02.254 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8GP5TGXS9A2FTPmmdBKwLEOeoHs= 2024-04-12T03:19:02.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8GP5TGXS9A2FTPmmdBKwLEOeoHs=', 'Content-Length': '0'} 2024-04-12T03:19:02.256 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000427121039899a042-006618a826-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.256 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:02.256 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:02.257 DEBUG:boto:path=/dkyges-9/?prefix=myobj 2024-04-12T03:19:02.257 DEBUG:boto:auth_path=/dkyges-9/?prefix=myobj 2024-04-12T03:19:02.257 DEBUG:boto:Method: GET 2024-04-12T03:19:02.257 DEBUG:boto:Path: /dkyges-9/?prefix=myobj 2024-04-12T03:19:02.257 DEBUG:boto:Data: 2024-04-12T03:19:02.257 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.257 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.257 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.258 DEBUG:boto:Params: {} 2024-04-12T03:19:02.258 DEBUG:boto:Token: None 2024-04-12T03:19:02.258 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/ 2024-04-12T03:19:02.258 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u/IbMiLhKaiCcMkuriG+kplN1gU= 2024-04-12T03:19:02.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u/IbMiLhKaiCcMkuriG+kplN1gU=', 'Content-Length': '0'} 2024-04-12T03:19:02.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085c2a433c44c2621-006618a826-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.261 DEBUG:boto:b'dkyges-9myobj1000falsemyobj2024-04-12T03:18:45.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:19:02.261 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:02.261 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:02.261 DEBUG:boto:path=/dkyges-9/?prefix=myobj 2024-04-12T03:19:02.261 DEBUG:boto:auth_path=/dkyges-9/?prefix=myobj 2024-04-12T03:19:02.262 DEBUG:boto:Method: GET 2024-04-12T03:19:02.262 DEBUG:boto:Path: /dkyges-9/?prefix=myobj 2024-04-12T03:19:02.262 DEBUG:boto:Data: 2024-04-12T03:19:02.262 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.262 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.262 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.262 DEBUG:boto:Params: {} 2024-04-12T03:19:02.262 DEBUG:boto:Token: None 2024-04-12T03:19:02.262 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/ 2024-04-12T03:19:02.262 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u/IbMiLhKaiCcMkuriG+kplN1gU= 2024-04-12T03:19:02.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u/IbMiLhKaiCcMkuriG+kplN1gU=', 'Content-Length': '0'} 2024-04-12T03:19:02.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1b2083117aac855-006618a826-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.265 DEBUG:boto:b'dkyges-9myobj1000falsemyobj2024-04-12T03:18:45.693Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:19:02.265 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-9 zones={a2, a1} 2024-04-12T03:19:02.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:19:02.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:19:02.266 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:19:02.332 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:02.332 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:02.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:02.373+0000 7f69f2b2a780 1 bucket sync caught up with source: 2024-04-12T03:19:02.373 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.2.6, , , , , , , , , ] 2024-04-12T03:19:02.373 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.2.6, , , , , , , , , ] 2024-04-12T03:19:02.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:02.373+0000 7f69f2b2a780 0 bucket checkpoint complete 2024-04-12T03:19:02.387 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-10 zones={a1, a2} 2024-04-12T03:19:02.387 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:02.387 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:02.387 DEBUG:boto:Method: HEAD 2024-04-12T03:19:02.387 DEBUG:boto:Path: /dkyges-10/ 2024-04-12T03:19:02.387 DEBUG:boto:Data: 2024-04-12T03:19:02.387 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.387 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.387 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.388 DEBUG:boto:Params: {} 2024-04-12T03:19:02.388 DEBUG:boto:Token: None 2024-04-12T03:19:02.388 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/ 2024-04-12T03:19:02.388 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wL4llj60WBQ+c3R2kDykoQMNZGE= 2024-04-12T03:19:02.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wL4llj60WBQ+c3R2kDykoQMNZGE=', 'Content-Length': '0'} 2024-04-12T03:19:02.390 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b37c2cc66cf5f5e-006618a826-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.390 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:02.390 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:02.391 DEBUG:boto:Method: HEAD 2024-04-12T03:19:02.391 DEBUG:boto:Path: /dkyges-10/ 2024-04-12T03:19:02.391 DEBUG:boto:Data: 2024-04-12T03:19:02.391 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.391 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.391 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.391 DEBUG:boto:Params: {} 2024-04-12T03:19:02.391 DEBUG:boto:Token: None 2024-04-12T03:19:02.391 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/ 2024-04-12T03:19:02.391 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wL4llj60WBQ+c3R2kDykoQMNZGE= 2024-04-12T03:19:02.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wL4llj60WBQ+c3R2kDykoQMNZGE=', 'Content-Length': '0'} 2024-04-12T03:19:02.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027a2219cec11ba55-006618a826-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.394 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:19:02.394 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:02.394 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:02.394 DEBUG:boto:path=/dkyges-10/?versions&max-keys=999 2024-04-12T03:19:02.394 DEBUG:boto:auth_path=/dkyges-10/?versions&max-keys=999 2024-04-12T03:19:02.394 DEBUG:boto:Method: GET 2024-04-12T03:19:02.394 DEBUG:boto:Path: /dkyges-10/?versions&max-keys=999 2024-04-12T03:19:02.395 DEBUG:boto:Data: 2024-04-12T03:19:02.395 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.395 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.395 DEBUG:boto:Params: {} 2024-04-12T03:19:02.395 DEBUG:boto:Token: None 2024-04-12T03:19:02.395 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/?versions 2024-04-12T03:19:02.395 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L2GfzV5dg8ElLY/w3I5xAXmHCO0= 2024-04-12T03:19:02.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L2GfzV5dg8ElLY/w3I5xAXmHCO0=', 'Content-Length': '0'} 2024-04-12T03:19:02.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a6d0a0b2fa9890d-006618a826-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.398 DEBUG:boto:b'dkyges-10999falsemyobjnulltrue2024-04-12T03:18:45.747Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:19:02.398 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-12T03:19:02.398 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:19:02.398 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:02.399 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:02.399 DEBUG:boto:path=/dkyges-10/?versions&max-keys=999 2024-04-12T03:19:02.399 DEBUG:boto:auth_path=/dkyges-10/?versions&max-keys=999 2024-04-12T03:19:02.399 DEBUG:boto:Method: GET 2024-04-12T03:19:02.399 DEBUG:boto:Path: /dkyges-10/?versions&max-keys=999 2024-04-12T03:19:02.399 DEBUG:boto:Data: 2024-04-12T03:19:02.399 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.399 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.399 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.399 DEBUG:boto:Params: {} 2024-04-12T03:19:02.400 DEBUG:boto:Token: None 2024-04-12T03:19:02.400 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/?versions 2024-04-12T03:19:02.400 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L2GfzV5dg8ElLY/w3I5xAXmHCO0= 2024-04-12T03:19:02.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L2GfzV5dg8ElLY/w3I5xAXmHCO0=', 'Content-Length': '0'} 2024-04-12T03:19:02.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002474a899bbfff18f-006618a826-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.402 DEBUG:boto:b'dkyges-10999falsemyobjnulltrue2024-04-12T03:18:45.747Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:19:02.403 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2024-04-12T03:19:02.403 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:02.403 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:02.403 DEBUG:boto:path=/dkyges-10/?versions&max-keys=999 2024-04-12T03:19:02.403 DEBUG:boto:auth_path=/dkyges-10/?versions&max-keys=999 2024-04-12T03:19:02.403 DEBUG:boto:Method: GET 2024-04-12T03:19:02.403 DEBUG:boto:Path: /dkyges-10/?versions&max-keys=999 2024-04-12T03:19:02.404 DEBUG:boto:Data: 2024-04-12T03:19:02.404 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.404 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.404 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.404 DEBUG:boto:Params: {} 2024-04-12T03:19:02.404 DEBUG:boto:Token: None 2024-04-12T03:19:02.404 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/?versions 2024-04-12T03:19:02.404 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L2GfzV5dg8ElLY/w3I5xAXmHCO0= 2024-04-12T03:19:02.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L2GfzV5dg8ElLY/w3I5xAXmHCO0=', 'Content-Length': '0'} 2024-04-12T03:19:02.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1b50bbe235af61a-006618a826-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.407 DEBUG:boto:b'dkyges-10999falsemyobjnulltrue2024-04-12T03:18:45.747Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:19:02.407 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:02.408 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:02.408 DEBUG:boto:path=/dkyges-10/?versions&max-keys=999 2024-04-12T03:19:02.408 DEBUG:boto:auth_path=/dkyges-10/?versions&max-keys=999 2024-04-12T03:19:02.408 DEBUG:boto:Method: GET 2024-04-12T03:19:02.408 DEBUG:boto:Path: /dkyges-10/?versions&max-keys=999 2024-04-12T03:19:02.408 DEBUG:boto:Data: 2024-04-12T03:19:02.409 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.409 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.409 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.410 DEBUG:boto:Params: {} 2024-04-12T03:19:02.410 DEBUG:boto:Token: None 2024-04-12T03:19:02.410 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/?versions 2024-04-12T03:19:02.410 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L2GfzV5dg8ElLY/w3I5xAXmHCO0= 2024-04-12T03:19:02.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L2GfzV5dg8ElLY/w3I5xAXmHCO0=', 'Content-Length': '0'} 2024-04-12T03:19:02.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000756db13213638f25-006618a826-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.412 DEBUG:boto:b'dkyges-10999falsemyobjnulltrue2024-04-12T03:18:45.747Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:19:02.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T03:19:02.413 DEBUG:boto:path=/dkyges-10/myobj 2024-04-12T03:19:02.413 DEBUG:boto:auth_path=/dkyges-10/myobj 2024-04-12T03:19:02.413 DEBUG:boto:path=/dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.413 DEBUG:boto:auth_path=/dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.413 DEBUG:boto:Method: GET 2024-04-12T03:19:02.413 DEBUG:boto:Path: /dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.413 DEBUG:boto:Data: 2024-04-12T03:19:02.414 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.414 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.414 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.414 DEBUG:boto:Params: {} 2024-04-12T03:19:02.414 DEBUG:boto:Token: None 2024-04-12T03:19:02.414 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.414 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MEPXkhphe9SnPW3C34gIGCDw0eE= 2024-04-12T03:19:02.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MEPXkhphe9SnPW3C34gIGCDw0eE=', 'Content-Length': '0'} 2024-04-12T03:19:02.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d4d9fb8b12be41ff-006618a826-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.458 DEBUG:boto:path=/dkyges-10/myobj 2024-04-12T03:19:02.459 DEBUG:boto:auth_path=/dkyges-10/myobj 2024-04-12T03:19:02.459 DEBUG:boto:path=/dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.459 DEBUG:boto:auth_path=/dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.459 DEBUG:boto:Method: GET 2024-04-12T03:19:02.459 DEBUG:boto:Path: /dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.459 DEBUG:boto:Data: 2024-04-12T03:19:02.459 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.459 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.459 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.459 DEBUG:boto:Params: {} 2024-04-12T03:19:02.459 DEBUG:boto:Token: None 2024-04-12T03:19:02.460 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.460 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MEPXkhphe9SnPW3C34gIGCDw0eE= 2024-04-12T03:19:02.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MEPXkhphe9SnPW3C34gIGCDw0eE=', 'Content-Length': '0'} 2024-04-12T03:19:02.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9302044286f2c72-006618a826-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.462 DEBUG:boto:path=/dkyges-10/myobj 2024-04-12T03:19:02.462 DEBUG:boto:auth_path=/dkyges-10/myobj 2024-04-12T03:19:02.462 DEBUG:boto:path=/dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.462 DEBUG:boto:auth_path=/dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.463 DEBUG:boto:Method: HEAD 2024-04-12T03:19:02.463 DEBUG:boto:Path: /dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.463 DEBUG:boto:Data: 2024-04-12T03:19:02.463 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.463 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.463 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.463 DEBUG:boto:Params: {} 2024-04-12T03:19:02.463 DEBUG:boto:Token: None 2024-04-12T03:19:02.463 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.463 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:96vBk5EO+GAoOdjUtl2PsROdUQM= 2024-04-12T03:19:02.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:96vBk5EO+GAoOdjUtl2PsROdUQM=', 'Content-Length': '0'} 2024-04-12T03:19:02.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007122956de6783816-006618a826-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.465 DEBUG:boto:path=/dkyges-10/myobj 2024-04-12T03:19:02.465 DEBUG:boto:auth_path=/dkyges-10/myobj 2024-04-12T03:19:02.466 DEBUG:boto:path=/dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.466 DEBUG:boto:auth_path=/dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.466 DEBUG:boto:Method: HEAD 2024-04-12T03:19:02.466 DEBUG:boto:Path: /dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.466 DEBUG:boto:Data: 2024-04-12T03:19:02.466 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.466 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.466 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.466 DEBUG:boto:Params: {} 2024-04-12T03:19:02.466 DEBUG:boto:Token: None 2024-04-12T03:19:02.467 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.467 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:96vBk5EO+GAoOdjUtl2PsROdUQM= 2024-04-12T03:19:02.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:96vBk5EO+GAoOdjUtl2PsROdUQM=', 'Content-Length': '0'} 2024-04-12T03:19:02.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c7f4524dd2e57c3-006618a826-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T03:19:02.469 DEBUG:boto:path=/dkyges-10/myobj 2024-04-12T03:19:02.469 DEBUG:boto:auth_path=/dkyges-10/myobj 2024-04-12T03:19:02.469 DEBUG:boto:path=/dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.469 DEBUG:boto:auth_path=/dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.469 DEBUG:boto:Method: GET 2024-04-12T03:19:02.469 DEBUG:boto:Path: /dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.469 DEBUG:boto:Data: 2024-04-12T03:19:02.470 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.470 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.470 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.470 DEBUG:boto:Params: {} 2024-04-12T03:19:02.470 DEBUG:boto:Token: None 2024-04-12T03:19:02.470 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.470 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MEPXkhphe9SnPW3C34gIGCDw0eE= 2024-04-12T03:19:02.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MEPXkhphe9SnPW3C34gIGCDw0eE=', 'Content-Length': '0'} 2024-04-12T03:19:02.472 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000483fa26ef28511ca-006618a826-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.472 DEBUG:boto:path=/dkyges-10/myobj 2024-04-12T03:19:02.472 DEBUG:boto:auth_path=/dkyges-10/myobj 2024-04-12T03:19:02.472 DEBUG:boto:path=/dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.473 DEBUG:boto:auth_path=/dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.473 DEBUG:boto:Method: GET 2024-04-12T03:19:02.473 DEBUG:boto:Path: /dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.473 DEBUG:boto:Data: 2024-04-12T03:19:02.473 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.473 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.473 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.473 DEBUG:boto:Params: {} 2024-04-12T03:19:02.473 DEBUG:boto:Token: None 2024-04-12T03:19:02.473 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/myobj?versionId=null 2024-04-12T03:19:02.474 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MEPXkhphe9SnPW3C34gIGCDw0eE= 2024-04-12T03:19:02.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MEPXkhphe9SnPW3C34gIGCDw0eE=', 'Content-Length': '0'} 2024-04-12T03:19:02.475 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044e90816aa812748-006618a826-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.476 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:02.476 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:02.476 DEBUG:boto:path=/dkyges-10/?prefix=myobj 2024-04-12T03:19:02.476 DEBUG:boto:auth_path=/dkyges-10/?prefix=myobj 2024-04-12T03:19:02.476 DEBUG:boto:Method: GET 2024-04-12T03:19:02.476 DEBUG:boto:Path: /dkyges-10/?prefix=myobj 2024-04-12T03:19:02.476 DEBUG:boto:Data: 2024-04-12T03:19:02.476 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.476 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.476 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.476 DEBUG:boto:Params: {} 2024-04-12T03:19:02.477 DEBUG:boto:Token: None 2024-04-12T03:19:02.477 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/ 2024-04-12T03:19:02.477 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wnQ+DJeO1n7qP4a2cbEqA0hhX0Q= 2024-04-12T03:19:02.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wnQ+DJeO1n7qP4a2cbEqA0hhX0Q=', 'Content-Length': '0'} 2024-04-12T03:19:02.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089618b5f20863eeb-006618a826-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.479 DEBUG:boto:b'dkyges-10myobj1000falsemyobj2024-04-12T03:18:45.747Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:19:02.480 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:02.480 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:02.480 DEBUG:boto:path=/dkyges-10/?prefix=myobj 2024-04-12T03:19:02.480 DEBUG:boto:auth_path=/dkyges-10/?prefix=myobj 2024-04-12T03:19:02.480 DEBUG:boto:Method: GET 2024-04-12T03:19:02.480 DEBUG:boto:Path: /dkyges-10/?prefix=myobj 2024-04-12T03:19:02.480 DEBUG:boto:Data: 2024-04-12T03:19:02.481 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.481 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.481 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.481 DEBUG:boto:Params: {} 2024-04-12T03:19:02.481 DEBUG:boto:Token: None 2024-04-12T03:19:02.481 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/ 2024-04-12T03:19:02.481 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wnQ+DJeO1n7qP4a2cbEqA0hhX0Q= 2024-04-12T03:19:02.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wnQ+DJeO1n7qP4a2cbEqA0hhX0Q=', 'Content-Length': '0'} 2024-04-12T03:19:02.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e1dcb8473c40f87-006618a826-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.484 DEBUG:boto:b'dkyges-10myobj1000falsemyobj2024-04-12T03:18:45.747Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:19:02.484 DEBUG:boto:path=/dkyges-10/myobj 2024-04-12T03:19:02.484 DEBUG:boto:auth_path=/dkyges-10/myobj 2024-04-12T03:19:02.484 DEBUG:boto:Method: HEAD 2024-04-12T03:19:02.484 DEBUG:boto:Path: /dkyges-10/myobj 2024-04-12T03:19:02.484 DEBUG:boto:Data: 2024-04-12T03:19:02.484 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.484 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.485 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.485 DEBUG:boto:Params: {} 2024-04-12T03:19:02.485 DEBUG:boto:Token: None 2024-04-12T03:19:02.485 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/myobj 2024-04-12T03:19:02.485 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dzsTI2pYG3oAXeDCK9MX8VqFDJw= 2024-04-12T03:19:02.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dzsTI2pYG3oAXeDCK9MX8VqFDJw=', 'Content-Length': '0'} 2024-04-12T03:19:02.487 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8ed8855a17087cd-006618a826-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.487 DEBUG:boto:path=/dkyges-10/myobj 2024-04-12T03:19:02.487 DEBUG:boto:auth_path=/dkyges-10/myobj 2024-04-12T03:19:02.487 DEBUG:boto:Method: HEAD 2024-04-12T03:19:02.487 DEBUG:boto:Path: /dkyges-10/myobj 2024-04-12T03:19:02.487 DEBUG:boto:Data: 2024-04-12T03:19:02.487 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.487 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.488 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.488 DEBUG:boto:Params: {} 2024-04-12T03:19:02.488 DEBUG:boto:Token: None 2024-04-12T03:19:02.488 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/myobj 2024-04-12T03:19:02.488 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dzsTI2pYG3oAXeDCK9MX8VqFDJw= 2024-04-12T03:19:02.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dzsTI2pYG3oAXeDCK9MX8VqFDJw=', 'Content-Length': '0'} 2024-04-12T03:19:02.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003688ac0eb38e2492-006618a826-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2024-04-12T03:19:02.490 DEBUG:boto:path=/dkyges-10/myobj 2024-04-12T03:19:02.490 DEBUG:boto:auth_path=/dkyges-10/myobj 2024-04-12T03:19:02.490 DEBUG:boto:Method: GET 2024-04-12T03:19:02.490 DEBUG:boto:Path: /dkyges-10/myobj 2024-04-12T03:19:02.490 DEBUG:boto:Data: 2024-04-12T03:19:02.490 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.491 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.491 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.491 DEBUG:boto:Params: {} 2024-04-12T03:19:02.491 DEBUG:boto:Token: None 2024-04-12T03:19:02.491 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/myobj 2024-04-12T03:19:02.491 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bFatBzQCxWuPDzgjN1wE55wlkpg= 2024-04-12T03:19:02.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bFatBzQCxWuPDzgjN1wE55wlkpg=', 'Content-Length': '0'} 2024-04-12T03:19:02.493 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001981f3b9df48fd94-006618a826-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.534 DEBUG:boto:path=/dkyges-10/myobj 2024-04-12T03:19:02.535 DEBUG:boto:auth_path=/dkyges-10/myobj 2024-04-12T03:19:02.535 DEBUG:boto:Method: GET 2024-04-12T03:19:02.535 DEBUG:boto:Path: /dkyges-10/myobj 2024-04-12T03:19:02.535 DEBUG:boto:Data: 2024-04-12T03:19:02.535 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.535 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.535 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.535 DEBUG:boto:Params: {} 2024-04-12T03:19:02.535 DEBUG:boto:Token: None 2024-04-12T03:19:02.536 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/myobj 2024-04-12T03:19:02.536 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bFatBzQCxWuPDzgjN1wE55wlkpg= 2024-04-12T03:19:02.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bFatBzQCxWuPDzgjN1wE55wlkpg=', 'Content-Length': '0'} 2024-04-12T03:19:02.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089767e4e5ee506bc-006618a826-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.538 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:02.538 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:02.538 DEBUG:boto:path=/dkyges-10/?prefix=myobj 2024-04-12T03:19:02.538 DEBUG:boto:auth_path=/dkyges-10/?prefix=myobj 2024-04-12T03:19:02.539 DEBUG:boto:Method: GET 2024-04-12T03:19:02.539 DEBUG:boto:Path: /dkyges-10/?prefix=myobj 2024-04-12T03:19:02.539 DEBUG:boto:Data: 2024-04-12T03:19:02.539 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.539 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.539 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.539 DEBUG:boto:Params: {} 2024-04-12T03:19:02.539 DEBUG:boto:Token: None 2024-04-12T03:19:02.539 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/ 2024-04-12T03:19:02.539 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wnQ+DJeO1n7qP4a2cbEqA0hhX0Q= 2024-04-12T03:19:02.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wnQ+DJeO1n7qP4a2cbEqA0hhX0Q=', 'Content-Length': '0'} 2024-04-12T03:19:02.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000958e728d1fcda8b4-006618a826-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.542 DEBUG:boto:b'dkyges-10myobj1000falsemyobj2024-04-12T03:18:45.747Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:19:02.543 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:02.543 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:02.543 DEBUG:boto:path=/dkyges-10/?prefix=myobj 2024-04-12T03:19:02.543 DEBUG:boto:auth_path=/dkyges-10/?prefix=myobj 2024-04-12T03:19:02.543 DEBUG:boto:Method: GET 2024-04-12T03:19:02.543 DEBUG:boto:Path: /dkyges-10/?prefix=myobj 2024-04-12T03:19:02.543 DEBUG:boto:Data: 2024-04-12T03:19:02.543 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.544 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:02.544 DEBUG:boto:Port: 8001 2024-04-12T03:19:02.544 DEBUG:boto:Params: {} 2024-04-12T03:19:02.544 DEBUG:boto:Token: None 2024-04-12T03:19:02.544 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:02 GMT /dkyges-10/ 2024-04-12T03:19:02.544 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wnQ+DJeO1n7qP4a2cbEqA0hhX0Q= 2024-04-12T03:19:02.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wnQ+DJeO1n7qP4a2cbEqA0hhX0Q=', 'Content-Length': '0'} 2024-04-12T03:19:02.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fec1a76bbcaa80ab-006618a826-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.546 DEBUG:boto:b'dkyges-10myobj1000falsemyobj2024-04-12T03:18:45.747Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:19:02.547 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-10 zones={a1, a2} 2024-04-12T03:19:02.547 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:02.547 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:02.547 DEBUG:boto:Method: HEAD 2024-04-12T03:19:02.547 DEBUG:boto:Path: /dkyges-9/ 2024-04-12T03:19:02.547 DEBUG:boto:Data: 2024-04-12T03:19:02.547 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.547 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.548 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.548 DEBUG:boto:Params: {} 2024-04-12T03:19:02.548 DEBUG:boto:Token: None 2024-04-12T03:19:02.548 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/ 2024-04-12T03:19:02.548 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mtPftfSNDHSgAcE1mgJVDnw+GPg= 2024-04-12T03:19:02.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mtPftfSNDHSgAcE1mgJVDnw+GPg=', 'Content-Length': '0'} 2024-04-12T03:19:02.550 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eed10478fdb8fd13-006618a826-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.550 DEBUG:boto:path=/dkyges-9/myobj 2024-04-12T03:19:02.550 DEBUG:boto:auth_path=/dkyges-9/myobj 2024-04-12T03:19:02.550 DEBUG:boto:Method: HEAD 2024-04-12T03:19:02.550 DEBUG:boto:Path: /dkyges-9/myobj 2024-04-12T03:19:02.551 DEBUG:boto:Data: 2024-04-12T03:19:02.551 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.551 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.551 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.551 DEBUG:boto:Params: {} 2024-04-12T03:19:02.551 DEBUG:boto:Token: None 2024-04-12T03:19:02.551 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/myobj 2024-04-12T03:19:02.551 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y4pnqLgpUyM+/3P6vI9fHx/Tncg= 2024-04-12T03:19:02.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y4pnqLgpUyM+/3P6vI9fHx/Tncg=', 'Content-Length': '0'} 2024-04-12T03:19:02.553 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004958a4008f8cfb79-006618a826-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.553 DEBUG:boto:path=/dkyges-9/myobj 2024-04-12T03:19:02.554 DEBUG:boto:auth_path=/dkyges-9/myobj 2024-04-12T03:19:02.554 DEBUG:boto:Method: DELETE 2024-04-12T03:19:02.554 DEBUG:boto:Path: /dkyges-9/myobj 2024-04-12T03:19:02.554 DEBUG:boto:Data: 2024-04-12T03:19:02.554 DEBUG:boto:Headers: {} 2024-04-12T03:19:02.554 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:02.554 DEBUG:boto:Port: 8000 2024-04-12T03:19:02.554 DEBUG:boto:Params: {} 2024-04-12T03:19:02.554 DEBUG:boto:Token: None 2024-04-12T03:19:02.554 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:19:02 GMT /dkyges-9/myobj 2024-04-12T03:19:02.555 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pcczvpsqdwj1qkadkXid1qVWwRI= 2024-04-12T03:19:02.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pcczvpsqdwj1qkadkXid1qVWwRI=', 'Content-Length': '0'} 2024-04-12T03:19:02.559 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000006f7b77d468049ed-006618a826-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:02.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:19:02.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:19:02.560 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:19:02.628 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:02.629 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:02.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:02.670+0000 7ff535139780 1 waiting for incremental sync to catch up: 2024-04-12T03:19:02.671 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2024-04-12T03:19:02.671 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2024-04-12T03:19:07.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:07.671+0000 7ff535139780 1 waiting for incremental sync to catch up: 2024-04-12T03:19:07.672 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2024-04-12T03:19:07.672 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2024-04-12T03:19:12.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:12.672+0000 7ff535139780 1 waiting for incremental sync to catch up: 2024-04-12T03:19:12.673 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2024-04-12T03:19:12.674 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2024-04-12T03:19:17.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:17.673+0000 7ff535139780 1 waiting for incremental sync to catch up: 2024-04-12T03:19:17.675 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2024-04-12T03:19:17.675 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2024-04-12T03:19:22.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:22.675+0000 7ff535139780 1 bucket sync caught up with source: 2024-04-12T03:19:22.676 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.7.6, , , , , , , , , ] 2024-04-12T03:19:22.676 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2024-04-12T03:19:22.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:22.675+0000 7ff535139780 0 bucket checkpoint complete 2024-04-12T03:19:22.693 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-9 zones={a2, a1} 2024-04-12T03:19:22.693 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:22.693 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:22.693 DEBUG:boto:Method: HEAD 2024-04-12T03:19:22.693 DEBUG:boto:Path: /dkyges-9/ 2024-04-12T03:19:22.693 DEBUG:boto:Data: 2024-04-12T03:19:22.693 DEBUG:boto:Headers: {} 2024-04-12T03:19:22.693 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:22.693 DEBUG:boto:Port: 8001 2024-04-12T03:19:22.694 DEBUG:boto:Params: {} 2024-04-12T03:19:22.694 DEBUG:boto:Token: None 2024-04-12T03:19:22.694 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:22 GMT /dkyges-9/ 2024-04-12T03:19:22.694 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YBPS/x9cIefTLlMtNbmXGh9C9uA= 2024-04-12T03:19:22.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YBPS/x9cIefTLlMtNbmXGh9C9uA=', 'Content-Length': '0'} 2024-04-12T03:19:22.697 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b8e43a533bd4617a-006618a83a-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:22.697 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:22.697 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:22.697 DEBUG:boto:Method: HEAD 2024-04-12T03:19:22.697 DEBUG:boto:Path: /dkyges-9/ 2024-04-12T03:19:22.697 DEBUG:boto:Data: 2024-04-12T03:19:22.697 DEBUG:boto:Headers: {} 2024-04-12T03:19:22.697 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:22.698 DEBUG:boto:Port: 8000 2024-04-12T03:19:22.698 DEBUG:boto:Params: {} 2024-04-12T03:19:22.698 DEBUG:boto:Token: None 2024-04-12T03:19:22.698 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:22 GMT /dkyges-9/ 2024-04-12T03:19:22.698 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YBPS/x9cIefTLlMtNbmXGh9C9uA= 2024-04-12T03:19:22.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YBPS/x9cIefTLlMtNbmXGh9C9uA=', 'Content-Length': '0'} 2024-04-12T03:19:22.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5134b4037188912-006618a83a-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:22.700 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:19:22.701 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:22.701 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:22.701 DEBUG:boto:path=/dkyges-9/?versions&max-keys=999 2024-04-12T03:19:22.701 DEBUG:boto:auth_path=/dkyges-9/?versions&max-keys=999 2024-04-12T03:19:22.701 DEBUG:boto:Method: GET 2024-04-12T03:19:22.701 DEBUG:boto:Path: /dkyges-9/?versions&max-keys=999 2024-04-12T03:19:22.701 DEBUG:boto:Data: 2024-04-12T03:19:22.701 DEBUG:boto:Headers: {} 2024-04-12T03:19:22.702 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:22.702 DEBUG:boto:Port: 8001 2024-04-12T03:19:22.702 DEBUG:boto:Params: {} 2024-04-12T03:19:22.702 DEBUG:boto:Token: None 2024-04-12T03:19:22.702 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:22 GMT /dkyges-9/?versions 2024-04-12T03:19:22.702 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rr+AvnG3T+3ANSXUe08HQvTGN3A= 2024-04-12T03:19:22.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rr+AvnG3T+3ANSXUe08HQvTGN3A=', 'Content-Length': '0'} 2024-04-12T03:19:22.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e73722810c9bbb97-006618a83a-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:22.705 DEBUG:boto:b'dkyges-9999false' 2024-04-12T03:19:22.705 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:19:22.705 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:22.705 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:22.705 DEBUG:boto:path=/dkyges-9/?versions&max-keys=999 2024-04-12T03:19:22.706 DEBUG:boto:auth_path=/dkyges-9/?versions&max-keys=999 2024-04-12T03:19:22.706 DEBUG:boto:Method: GET 2024-04-12T03:19:22.706 DEBUG:boto:Path: /dkyges-9/?versions&max-keys=999 2024-04-12T03:19:22.706 DEBUG:boto:Data: 2024-04-12T03:19:22.706 DEBUG:boto:Headers: {} 2024-04-12T03:19:22.706 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:22.706 DEBUG:boto:Port: 8000 2024-04-12T03:19:22.706 DEBUG:boto:Params: {} 2024-04-12T03:19:22.706 DEBUG:boto:Token: None 2024-04-12T03:19:22.706 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:22 GMT /dkyges-9/?versions 2024-04-12T03:19:22.707 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rr+AvnG3T+3ANSXUe08HQvTGN3A= 2024-04-12T03:19:22.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rr+AvnG3T+3ANSXUe08HQvTGN3A=', 'Content-Length': '0'} 2024-04-12T03:19:22.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034aba15f8d313d6d-006618a83a-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:22.709 DEBUG:boto:b'dkyges-9999false' 2024-04-12T03:19:22.709 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:22.709 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:22.709 DEBUG:boto:path=/dkyges-9/?versions&max-keys=999 2024-04-12T03:19:22.710 DEBUG:boto:auth_path=/dkyges-9/?versions&max-keys=999 2024-04-12T03:19:22.710 DEBUG:boto:Method: GET 2024-04-12T03:19:22.710 DEBUG:boto:Path: /dkyges-9/?versions&max-keys=999 2024-04-12T03:19:22.710 DEBUG:boto:Data: 2024-04-12T03:19:22.710 DEBUG:boto:Headers: {} 2024-04-12T03:19:22.710 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:22.710 DEBUG:boto:Port: 8001 2024-04-12T03:19:22.710 DEBUG:boto:Params: {} 2024-04-12T03:19:22.710 DEBUG:boto:Token: None 2024-04-12T03:19:22.710 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:22 GMT /dkyges-9/?versions 2024-04-12T03:19:22.711 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rr+AvnG3T+3ANSXUe08HQvTGN3A= 2024-04-12T03:19:22.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rr+AvnG3T+3ANSXUe08HQvTGN3A=', 'Content-Length': '0'} 2024-04-12T03:19:22.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce648d9d1d8baa93-006618a83a-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:22.713 DEBUG:boto:b'dkyges-9999false' 2024-04-12T03:19:22.713 DEBUG:boto:path=/dkyges-9/ 2024-04-12T03:19:22.713 DEBUG:boto:auth_path=/dkyges-9/ 2024-04-12T03:19:22.713 DEBUG:boto:path=/dkyges-9/?versions&max-keys=999 2024-04-12T03:19:22.713 DEBUG:boto:auth_path=/dkyges-9/?versions&max-keys=999 2024-04-12T03:19:22.713 DEBUG:boto:Method: GET 2024-04-12T03:19:22.713 DEBUG:boto:Path: /dkyges-9/?versions&max-keys=999 2024-04-12T03:19:22.714 DEBUG:boto:Data: 2024-04-12T03:19:22.714 DEBUG:boto:Headers: {} 2024-04-12T03:19:22.714 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:22.714 DEBUG:boto:Port: 8000 2024-04-12T03:19:22.714 DEBUG:boto:Params: {} 2024-04-12T03:19:22.714 DEBUG:boto:Token: None 2024-04-12T03:19:22.714 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:22 GMT /dkyges-9/?versions 2024-04-12T03:19:22.714 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rr+AvnG3T+3ANSXUe08HQvTGN3A= 2024-04-12T03:19:22.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rr+AvnG3T+3ANSXUe08HQvTGN3A=', 'Content-Length': '0'} 2024-04-12T03:19:22.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2bf47fdcd253215-006618a83a-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:22.716 DEBUG:boto:b'dkyges-9999false' 2024-04-12T03:19:22.717 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-9 zones={a2, a1} 2024-04-12T03:19:22.717 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:22.717 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:22.717 DEBUG:boto:Method: HEAD 2024-04-12T03:19:22.717 DEBUG:boto:Path: /dkyges-10/ 2024-04-12T03:19:22.717 DEBUG:boto:Data: 2024-04-12T03:19:22.717 DEBUG:boto:Headers: {} 2024-04-12T03:19:22.717 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:22.718 DEBUG:boto:Port: 8001 2024-04-12T03:19:22.718 DEBUG:boto:Params: {} 2024-04-12T03:19:22.718 DEBUG:boto:Token: None 2024-04-12T03:19:22.718 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:22 GMT /dkyges-10/ 2024-04-12T03:19:22.718 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Uj22paEyjOXMECxUZnwdcR/ENlE= 2024-04-12T03:19:22.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Uj22paEyjOXMECxUZnwdcR/ENlE=', 'Content-Length': '0'} 2024-04-12T03:19:22.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000afedfa1121c3bd8f-006618a83a-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:22.720 DEBUG:boto:path=/dkyges-10/myobj 2024-04-12T03:19:22.720 DEBUG:boto:auth_path=/dkyges-10/myobj 2024-04-12T03:19:22.720 DEBUG:boto:Method: HEAD 2024-04-12T03:19:22.721 DEBUG:boto:Path: /dkyges-10/myobj 2024-04-12T03:19:22.721 DEBUG:boto:Data: 2024-04-12T03:19:22.721 DEBUG:boto:Headers: {} 2024-04-12T03:19:22.721 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:22.721 DEBUG:boto:Port: 8001 2024-04-12T03:19:22.721 DEBUG:boto:Params: {} 2024-04-12T03:19:22.721 DEBUG:boto:Token: None 2024-04-12T03:19:22.721 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:22 GMT /dkyges-10/myobj 2024-04-12T03:19:22.722 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UbXJwRcpBPvhcGuK9Fw9A7MIzVo= 2024-04-12T03:19:22.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UbXJwRcpBPvhcGuK9Fw9A7MIzVo=', 'Content-Length': '0'} 2024-04-12T03:19:22.724 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:18:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c74bbab4dc71b296-006618a83a-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:19:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:22.725 DEBUG:boto:path=/dkyges-10/myobj 2024-04-12T03:19:22.725 DEBUG:boto:auth_path=/dkyges-10/myobj 2024-04-12T03:19:22.725 DEBUG:boto:Method: DELETE 2024-04-12T03:19:22.725 DEBUG:boto:Path: /dkyges-10/myobj 2024-04-12T03:19:22.725 DEBUG:boto:Data: 2024-04-12T03:19:22.725 DEBUG:boto:Headers: {} 2024-04-12T03:19:22.726 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:22.726 DEBUG:boto:Port: 8001 2024-04-12T03:19:22.726 DEBUG:boto:Params: {} 2024-04-12T03:19:22.726 DEBUG:boto:Token: None 2024-04-12T03:19:22.726 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:19:22 GMT /dkyges-10/myobj 2024-04-12T03:19:22.726 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:slmDnKYguPXIeRLj61RoDYRkzIg= 2024-04-12T03:19:22.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:slmDnKYguPXIeRLj61RoDYRkzIg=', 'Content-Length': '0'} 2024-04-12T03:19:22.732 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a460be1f054387c2-006618a83a-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:22.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:19:22.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:19:22.733 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:19:22.799 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:22.799 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:22.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:22.839+0000 7fa2f458d780 1 waiting for incremental sync to catch up: 2024-04-12T03:19:22.841 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.2.6, , , , , , , , , ] 2024-04-12T03:19:22.841 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.5.6, , , , , , , , , ] 2024-04-12T03:19:27.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:27.841+0000 7fa2f458d780 1 waiting for incremental sync to catch up: 2024-04-12T03:19:27.843 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.2.6, , , , , , , , , ] 2024-04-12T03:19:27.843 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.5.6, , , , , , , , , ] 2024-04-12T03:19:32.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:32.843+0000 7fa2f458d780 1 waiting for incremental sync to catch up: 2024-04-12T03:19:32.844 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.2.6, , , , , , , , , ] 2024-04-12T03:19:32.844 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.5.6, , , , , , , , , ] 2024-04-12T03:19:37.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:37.844+0000 7fa2f458d780 1 waiting for incremental sync to catch up: 2024-04-12T03:19:37.846 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.2.6, , , , , , , , , ] 2024-04-12T03:19:37.846 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.5.6, , , , , , , , , ] 2024-04-12T03:19:42.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:42.846+0000 7fa2f458d780 1 bucket sync caught up with source: 2024-04-12T03:19:42.848 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.5.6, , , , , , , , , ] 2024-04-12T03:19:42.848 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.5.6, , , , , , , , , ] 2024-04-12T03:19:42.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:42.846+0000 7fa2f458d780 0 bucket checkpoint complete 2024-04-12T03:19:42.864 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-10 zones={a1, a2} 2024-04-12T03:19:42.864 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:42.864 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:42.864 DEBUG:boto:Method: HEAD 2024-04-12T03:19:42.864 DEBUG:boto:Path: /dkyges-10/ 2024-04-12T03:19:42.864 DEBUG:boto:Data: 2024-04-12T03:19:42.865 DEBUG:boto:Headers: {} 2024-04-12T03:19:42.865 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:42.865 DEBUG:boto:Port: 8000 2024-04-12T03:19:42.865 DEBUG:boto:Params: {} 2024-04-12T03:19:42.865 DEBUG:boto:Token: None 2024-04-12T03:19:42.865 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:42 GMT /dkyges-10/ 2024-04-12T03:19:42.866 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fxwuQlMrsoo5W0m85Y9zLQAoac8= 2024-04-12T03:19:42.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fxwuQlMrsoo5W0m85Y9zLQAoac8=', 'Content-Length': '0'} 2024-04-12T03:19:42.868 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc8974a4566eb3a1-006618a84e-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:42.868 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:42.869 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:42.869 DEBUG:boto:Method: HEAD 2024-04-12T03:19:42.869 DEBUG:boto:Path: /dkyges-10/ 2024-04-12T03:19:42.869 DEBUG:boto:Data: 2024-04-12T03:19:42.869 DEBUG:boto:Headers: {} 2024-04-12T03:19:42.869 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:42.869 DEBUG:boto:Port: 8001 2024-04-12T03:19:42.869 DEBUG:boto:Params: {} 2024-04-12T03:19:42.869 DEBUG:boto:Token: None 2024-04-12T03:19:42.869 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:42 GMT /dkyges-10/ 2024-04-12T03:19:42.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fxwuQlMrsoo5W0m85Y9zLQAoac8= 2024-04-12T03:19:42.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fxwuQlMrsoo5W0m85Y9zLQAoac8=', 'Content-Length': '0'} 2024-04-12T03:19:42.872 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a47066968cff60dd-006618a84e-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:42.872 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:19:42.872 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:42.872 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:42.872 DEBUG:boto:path=/dkyges-10/?versions&max-keys=999 2024-04-12T03:19:42.873 DEBUG:boto:auth_path=/dkyges-10/?versions&max-keys=999 2024-04-12T03:19:42.873 DEBUG:boto:Method: GET 2024-04-12T03:19:42.873 DEBUG:boto:Path: /dkyges-10/?versions&max-keys=999 2024-04-12T03:19:42.873 DEBUG:boto:Data: 2024-04-12T03:19:42.873 DEBUG:boto:Headers: {} 2024-04-12T03:19:42.873 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:42.873 DEBUG:boto:Port: 8000 2024-04-12T03:19:42.873 DEBUG:boto:Params: {} 2024-04-12T03:19:42.873 DEBUG:boto:Token: None 2024-04-12T03:19:42.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:42 GMT /dkyges-10/?versions 2024-04-12T03:19:42.874 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X5dkSXEnyBashb7dUjm4spUqGLI= 2024-04-12T03:19:42.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X5dkSXEnyBashb7dUjm4spUqGLI=', 'Content-Length': '0'} 2024-04-12T03:19:42.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eea5c145cdd47729-006618a84e-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:42.876 DEBUG:boto:b'dkyges-10999false' 2024-04-12T03:19:42.877 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:19:42.877 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:42.877 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:42.877 DEBUG:boto:path=/dkyges-10/?versions&max-keys=999 2024-04-12T03:19:42.877 DEBUG:boto:auth_path=/dkyges-10/?versions&max-keys=999 2024-04-12T03:19:42.877 DEBUG:boto:Method: GET 2024-04-12T03:19:42.877 DEBUG:boto:Path: /dkyges-10/?versions&max-keys=999 2024-04-12T03:19:42.877 DEBUG:boto:Data: 2024-04-12T03:19:42.878 DEBUG:boto:Headers: {} 2024-04-12T03:19:42.878 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:42.878 DEBUG:boto:Port: 8001 2024-04-12T03:19:42.878 DEBUG:boto:Params: {} 2024-04-12T03:19:42.878 DEBUG:boto:Token: None 2024-04-12T03:19:42.878 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:42 GMT /dkyges-10/?versions 2024-04-12T03:19:42.878 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X5dkSXEnyBashb7dUjm4spUqGLI= 2024-04-12T03:19:42.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X5dkSXEnyBashb7dUjm4spUqGLI=', 'Content-Length': '0'} 2024-04-12T03:19:42.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002e6338422e7b4c7-006618a84e-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:42.880 DEBUG:boto:b'dkyges-10999false' 2024-04-12T03:19:42.881 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:42.881 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:42.881 DEBUG:boto:path=/dkyges-10/?versions&max-keys=999 2024-04-12T03:19:42.881 DEBUG:boto:auth_path=/dkyges-10/?versions&max-keys=999 2024-04-12T03:19:42.881 DEBUG:boto:Method: GET 2024-04-12T03:19:42.881 DEBUG:boto:Path: /dkyges-10/?versions&max-keys=999 2024-04-12T03:19:42.881 DEBUG:boto:Data: 2024-04-12T03:19:42.881 DEBUG:boto:Headers: {} 2024-04-12T03:19:42.881 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:42.882 DEBUG:boto:Port: 8000 2024-04-12T03:19:42.882 DEBUG:boto:Params: {} 2024-04-12T03:19:42.882 DEBUG:boto:Token: None 2024-04-12T03:19:42.882 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:42 GMT /dkyges-10/?versions 2024-04-12T03:19:42.882 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X5dkSXEnyBashb7dUjm4spUqGLI= 2024-04-12T03:19:42.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X5dkSXEnyBashb7dUjm4spUqGLI=', 'Content-Length': '0'} 2024-04-12T03:19:42.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006535cbf648ed872a-006618a84e-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:42.926 DEBUG:boto:b'dkyges-10999false' 2024-04-12T03:19:42.927 DEBUG:boto:path=/dkyges-10/ 2024-04-12T03:19:42.927 DEBUG:boto:auth_path=/dkyges-10/ 2024-04-12T03:19:42.927 DEBUG:boto:path=/dkyges-10/?versions&max-keys=999 2024-04-12T03:19:42.927 DEBUG:boto:auth_path=/dkyges-10/?versions&max-keys=999 2024-04-12T03:19:42.927 DEBUG:boto:Method: GET 2024-04-12T03:19:42.927 DEBUG:boto:Path: /dkyges-10/?versions&max-keys=999 2024-04-12T03:19:42.927 DEBUG:boto:Data: 2024-04-12T03:19:42.927 DEBUG:boto:Headers: {} 2024-04-12T03:19:42.927 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:42.928 DEBUG:boto:Port: 8001 2024-04-12T03:19:42.928 DEBUG:boto:Params: {} 2024-04-12T03:19:42.928 DEBUG:boto:Token: None 2024-04-12T03:19:42.928 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:19:42 GMT /dkyges-10/?versions 2024-04-12T03:19:42.928 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X5dkSXEnyBashb7dUjm4spUqGLI= 2024-04-12T03:19:42.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X5dkSXEnyBashb7dUjm4spUqGLI=', 'Content-Length': '0'} 2024-04-12T03:19:42.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dab9cb004cb139c-006618a84e-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:19:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:42.931 DEBUG:boto:b'dkyges-10999false' 2024-04-12T03:19:42.931 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-10 zones={a1, a2} 2024-04-12T03:19:42.931 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2024-04-12T03:19:42.933 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-11 2024-04-12T03:19:42.933 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:42.933 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:42.933 DEBUG:boto:Method: PUT 2024-04-12T03:19:42.933 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:42.933 DEBUG:boto:Data: 2024-04-12T03:19:42.933 DEBUG:boto:Headers: {} 2024-04-12T03:19:42.933 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:42.933 DEBUG:boto:Port: 8000 2024-04-12T03:19:42.933 DEBUG:boto:Params: {} 2024-04-12T03:19:42.934 DEBUG:boto:Token: None 2024-04-12T03:19:42.934 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:19:42 GMT /dkyges-11/ 2024-04-12T03:19:42.934 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ObmF7ZPgDmoetIO297QL1jpvMGY= 2024-04-12T03:19:42.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ObmF7ZPgDmoetIO297QL1jpvMGY=', 'Content-Length': '0'} 2024-04-12T03:19:42.950 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000037444dabed7e33b4-006618a84e-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:42.950 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-12 2024-04-12T03:19:42.950 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:42.951 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:42.951 DEBUG:boto:Method: PUT 2024-04-12T03:19:42.951 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:42.951 DEBUG:boto:Data: 2024-04-12T03:19:42.951 DEBUG:boto:Headers: {} 2024-04-12T03:19:42.951 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:42.951 DEBUG:boto:Port: 8001 2024-04-12T03:19:42.951 DEBUG:boto:Params: {} 2024-04-12T03:19:42.951 DEBUG:boto:Token: None 2024-04-12T03:19:42.951 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:19:42 GMT /dkyges-12/ 2024-04-12T03:19:42.952 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gdpmZWGj/iRGRCWfBNByD18q3ww= 2024-04-12T03:19:42.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gdpmZWGj/iRGRCWfBNByD18q3ww=', 'Content-Length': '0'} 2024-04-12T03:19:42.979 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006f2ed29c19506f61-006618a84e-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:42.979 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:42.979 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:42.979 DEBUG:boto:Method: HEAD 2024-04-12T03:19:42.979 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:42.979 DEBUG:boto:Data: 2024-04-12T03:19:42.980 DEBUG:boto:Headers: {} 2024-04-12T03:19:42.980 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:42.980 DEBUG:boto:Port: 8000 2024-04-12T03:19:42.980 DEBUG:boto:Params: {} 2024-04-12T03:19:42.980 DEBUG:boto:Token: None 2024-04-12T03:19:42.980 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:42 GMT /dkyges-11/ 2024-04-12T03:19:42.980 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xDPVQOae4G9S5DTOgeK1bAfY0Cw= 2024-04-12T03:19:42.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xDPVQOae4G9S5DTOgeK1bAfY0Cw=', 'Content-Length': '0'} 2024-04-12T03:19:42.983 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec05f64bc6c5988e-006618a84e-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:42.983 DEBUG:boto:path=/dkyges-11/obj1 2024-04-12T03:19:42.983 DEBUG:boto:auth_path=/dkyges-11/obj1 2024-04-12T03:19:42.983 DEBUG:boto:Method: PUT 2024-04-12T03:19:42.983 DEBUG:boto:Path: /dkyges-11/obj1 2024-04-12T03:19:42.983 DEBUG:boto:Data: 2024-04-12T03:19:42.984 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:42.984 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:42.984 DEBUG:boto:Port: 8000 2024-04-12T03:19:42.984 DEBUG:boto:Params: {} 2024-04-12T03:19:42.984 DEBUG:boto:Token: None 2024-04-12T03:19:42.984 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:42 GMT /dkyges-11/obj1 2024-04-12T03:19:42.984 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NCnRWL4u2UCsgzQqX8WBdhcewUM= 2024-04-12T03:19:42.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NCnRWL4u2UCsgzQqX8WBdhcewUM='} 2024-04-12T03:19:43.027 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000227e1415fcb9b709-006618a84e-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.027 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.027 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.027 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.027 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.028 DEBUG:boto:Data: 2024-04-12T03:19:43.028 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.028 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.028 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.028 DEBUG:boto:Params: {} 2024-04-12T03:19:43.028 DEBUG:boto:Token: None 2024-04-12T03:19:43.028 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.029 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.031 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa95f3608ae5778f-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.031 DEBUG:boto:path=/dkyges-11/obj2 2024-04-12T03:19:43.031 DEBUG:boto:auth_path=/dkyges-11/obj2 2024-04-12T03:19:43.031 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.032 DEBUG:boto:Path: /dkyges-11/obj2 2024-04-12T03:19:43.032 DEBUG:boto:Data: 2024-04-12T03:19:43.032 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.032 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.032 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.032 DEBUG:boto:Params: {} 2024-04-12T03:19:43.032 DEBUG:boto:Token: None 2024-04-12T03:19:43.033 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj2 2024-04-12T03:19:43.033 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5wWSt4aobjCTIpOdfL2ewfI+GNQ= 2024-04-12T03:19:43.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5wWSt4aobjCTIpOdfL2ewfI+GNQ='} 2024-04-12T03:19:43.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f7662a96cb605e3-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.039 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.039 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.039 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.039 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.039 DEBUG:boto:Data: 2024-04-12T03:19:43.039 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.039 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.039 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.039 DEBUG:boto:Params: {} 2024-04-12T03:19:43.040 DEBUG:boto:Token: None 2024-04-12T03:19:43.040 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.040 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.042 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064713742f802f1a3-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.043 DEBUG:boto:path=/dkyges-11/obj3 2024-04-12T03:19:43.043 DEBUG:boto:auth_path=/dkyges-11/obj3 2024-04-12T03:19:43.043 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.043 DEBUG:boto:Path: /dkyges-11/obj3 2024-04-12T03:19:43.043 DEBUG:boto:Data: 2024-04-12T03:19:43.043 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.043 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.043 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.043 DEBUG:boto:Params: {} 2024-04-12T03:19:43.044 DEBUG:boto:Token: None 2024-04-12T03:19:43.044 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj3 2024-04-12T03:19:43.044 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aCaet8mYtOEx4xZydzW3tMEMYXU= 2024-04-12T03:19:43.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aCaet8mYtOEx4xZydzW3tMEMYXU='} 2024-04-12T03:19:43.087 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dea8973b22229e94-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.087 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.087 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.087 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.087 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.087 DEBUG:boto:Data: 2024-04-12T03:19:43.088 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.088 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.088 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.088 DEBUG:boto:Params: {} 2024-04-12T03:19:43.088 DEBUG:boto:Token: None 2024-04-12T03:19:43.088 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.088 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.091 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000752eb90b48342044-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.091 DEBUG:boto:path=/dkyges-11/obj4 2024-04-12T03:19:43.092 DEBUG:boto:auth_path=/dkyges-11/obj4 2024-04-12T03:19:43.092 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.092 DEBUG:boto:Path: /dkyges-11/obj4 2024-04-12T03:19:43.092 DEBUG:boto:Data: 2024-04-12T03:19:43.092 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.092 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.092 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.092 DEBUG:boto:Params: {} 2024-04-12T03:19:43.092 DEBUG:boto:Token: None 2024-04-12T03:19:43.093 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj4 2024-04-12T03:19:43.093 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EghNKGR95MNwc8rkY4UFybIDHKU= 2024-04-12T03:19:43.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EghNKGR95MNwc8rkY4UFybIDHKU='} 2024-04-12T03:19:43.104 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000011d31e232f72f30c-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.104 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.104 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.104 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.104 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.104 DEBUG:boto:Data: 2024-04-12T03:19:43.105 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.105 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.105 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.105 DEBUG:boto:Params: {} 2024-04-12T03:19:43.105 DEBUG:boto:Token: None 2024-04-12T03:19:43.105 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.105 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.107 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000056c48cbfa8db62e-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.108 DEBUG:boto:path=/dkyges-11/obj5 2024-04-12T03:19:43.108 DEBUG:boto:auth_path=/dkyges-11/obj5 2024-04-12T03:19:43.108 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.108 DEBUG:boto:Path: /dkyges-11/obj5 2024-04-12T03:19:43.108 DEBUG:boto:Data: 2024-04-12T03:19:43.108 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.108 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.108 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.108 DEBUG:boto:Params: {} 2024-04-12T03:19:43.109 DEBUG:boto:Token: None 2024-04-12T03:19:43.109 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj5 2024-04-12T03:19:43.109 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5BanJtk7qwcsmxUbMKwX5971dzI= 2024-04-12T03:19:43.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5BanJtk7qwcsmxUbMKwX5971dzI='} 2024-04-12T03:19:43.150 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009029d4f494130b97-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.151 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.151 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.151 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.151 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.151 DEBUG:boto:Data: 2024-04-12T03:19:43.151 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.151 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.151 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.152 DEBUG:boto:Params: {} 2024-04-12T03:19:43.152 DEBUG:boto:Token: None 2024-04-12T03:19:43.152 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.152 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009acc5ac4d1877645-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.155 DEBUG:boto:path=/dkyges-11/obj6 2024-04-12T03:19:43.155 DEBUG:boto:auth_path=/dkyges-11/obj6 2024-04-12T03:19:43.155 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.155 DEBUG:boto:Path: /dkyges-11/obj6 2024-04-12T03:19:43.155 DEBUG:boto:Data: 2024-04-12T03:19:43.155 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.155 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.155 DEBUG:boto:Params: {} 2024-04-12T03:19:43.156 DEBUG:boto:Token: None 2024-04-12T03:19:43.156 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj6 2024-04-12T03:19:43.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3yHkarfswFEa+MY3vTQQOqF8+N0= 2024-04-12T03:19:43.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3yHkarfswFEa+MY3vTQQOqF8+N0='} 2024-04-12T03:19:43.161 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6a84693e8c5f1ec-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.161 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.161 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.161 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.161 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.162 DEBUG:boto:Data: 2024-04-12T03:19:43.162 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.162 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.162 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.162 DEBUG:boto:Params: {} 2024-04-12T03:19:43.162 DEBUG:boto:Token: None 2024-04-12T03:19:43.162 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.162 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.165 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c5c01e5e792f5e0-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.165 DEBUG:boto:path=/dkyges-11/obj7 2024-04-12T03:19:43.165 DEBUG:boto:auth_path=/dkyges-11/obj7 2024-04-12T03:19:43.165 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.165 DEBUG:boto:Path: /dkyges-11/obj7 2024-04-12T03:19:43.165 DEBUG:boto:Data: 2024-04-12T03:19:43.166 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.166 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.166 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.166 DEBUG:boto:Params: {} 2024-04-12T03:19:43.166 DEBUG:boto:Token: None 2024-04-12T03:19:43.166 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj7 2024-04-12T03:19:43.166 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:izQ8461tBXgVuqf+R1s9BhJvVLI= 2024-04-12T03:19:43.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:izQ8461tBXgVuqf+R1s9BhJvVLI='} 2024-04-12T03:19:43.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0dd79bad552519a-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.211 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.211 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.211 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.211 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.211 DEBUG:boto:Data: 2024-04-12T03:19:43.211 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.211 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.211 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.211 DEBUG:boto:Params: {} 2024-04-12T03:19:43.212 DEBUG:boto:Token: None 2024-04-12T03:19:43.212 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.215 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fdbcaf106623e2ac-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.215 DEBUG:boto:path=/dkyges-11/obj8 2024-04-12T03:19:43.215 DEBUG:boto:auth_path=/dkyges-11/obj8 2024-04-12T03:19:43.215 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.215 DEBUG:boto:Path: /dkyges-11/obj8 2024-04-12T03:19:43.215 DEBUG:boto:Data: 2024-04-12T03:19:43.215 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.215 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.215 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.216 DEBUG:boto:Params: {} 2024-04-12T03:19:43.216 DEBUG:boto:Token: None 2024-04-12T03:19:43.216 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj8 2024-04-12T03:19:43.216 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nwHhiouhgQ7KT2tf2eBV9s6f/ZE= 2024-04-12T03:19:43.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nwHhiouhgQ7KT2tf2eBV9s6f/ZE='} 2024-04-12T03:19:43.220 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025f3b0a3abd563fc-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.221 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.221 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.221 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.221 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.221 DEBUG:boto:Data: 2024-04-12T03:19:43.221 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.221 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.221 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.222 DEBUG:boto:Params: {} 2024-04-12T03:19:43.222 DEBUG:boto:Token: None 2024-04-12T03:19:43.222 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.222 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.225 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a87b5e9bd515a574-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.225 DEBUG:boto:path=/dkyges-11/obj9 2024-04-12T03:19:43.225 DEBUG:boto:auth_path=/dkyges-11/obj9 2024-04-12T03:19:43.225 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.225 DEBUG:boto:Path: /dkyges-11/obj9 2024-04-12T03:19:43.225 DEBUG:boto:Data: 2024-04-12T03:19:43.225 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.225 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.226 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.226 DEBUG:boto:Params: {} 2024-04-12T03:19:43.227 DEBUG:boto:Token: None 2024-04-12T03:19:43.227 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj9 2024-04-12T03:19:43.227 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6JRxZxW671VDGxHyUgR6vaH9Cl0= 2024-04-12T03:19:43.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6JRxZxW671VDGxHyUgR6vaH9Cl0='} 2024-04-12T03:19:43.270 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000977d100f4960f9e9-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.271 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.271 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.271 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.271 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.271 DEBUG:boto:Data: 2024-04-12T03:19:43.271 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.271 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.272 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.272 DEBUG:boto:Params: {} 2024-04-12T03:19:43.272 DEBUG:boto:Token: None 2024-04-12T03:19:43.272 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.272 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.275 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000062d75b68c60b23d-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.275 DEBUG:boto:path=/dkyges-11/obj10 2024-04-12T03:19:43.275 DEBUG:boto:auth_path=/dkyges-11/obj10 2024-04-12T03:19:43.275 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.275 DEBUG:boto:Path: /dkyges-11/obj10 2024-04-12T03:19:43.275 DEBUG:boto:Data: 2024-04-12T03:19:43.275 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.276 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.276 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.276 DEBUG:boto:Params: {} 2024-04-12T03:19:43.276 DEBUG:boto:Token: None 2024-04-12T03:19:43.276 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj10 2024-04-12T03:19:43.276 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lfQacQMHZyZHwX0hJRtcpqpC/zc= 2024-04-12T03:19:43.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lfQacQMHZyZHwX0hJRtcpqpC/zc='} 2024-04-12T03:19:43.280 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000703ef13c27a52ab8-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.281 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.281 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.281 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.281 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.281 DEBUG:boto:Data: 2024-04-12T03:19:43.281 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.281 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.281 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.281 DEBUG:boto:Params: {} 2024-04-12T03:19:43.282 DEBUG:boto:Token: None 2024-04-12T03:19:43.282 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.282 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.284 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009275fc2cb6280c2-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.285 DEBUG:boto:path=/dkyges-11/obj11 2024-04-12T03:19:43.285 DEBUG:boto:auth_path=/dkyges-11/obj11 2024-04-12T03:19:43.285 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.285 DEBUG:boto:Path: /dkyges-11/obj11 2024-04-12T03:19:43.285 DEBUG:boto:Data: 2024-04-12T03:19:43.285 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.285 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.285 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.285 DEBUG:boto:Params: {} 2024-04-12T03:19:43.285 DEBUG:boto:Token: None 2024-04-12T03:19:43.286 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj11 2024-04-12T03:19:43.286 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2vpBmV/6oENe2ntHFjPL6j9zjGU= 2024-04-12T03:19:43.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2vpBmV/6oENe2ntHFjPL6j9zjGU='} 2024-04-12T03:19:43.334 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021886eb00214b491-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.334 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.334 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.335 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.335 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.335 DEBUG:boto:Data: 2024-04-12T03:19:43.335 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.335 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.335 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.335 DEBUG:boto:Params: {} 2024-04-12T03:19:43.335 DEBUG:boto:Token: None 2024-04-12T03:19:43.335 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.335 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.338 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f9e26e2919dc8b3-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.338 DEBUG:boto:path=/dkyges-11/obj12 2024-04-12T03:19:43.338 DEBUG:boto:auth_path=/dkyges-11/obj12 2024-04-12T03:19:43.338 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.339 DEBUG:boto:Path: /dkyges-11/obj12 2024-04-12T03:19:43.339 DEBUG:boto:Data: 2024-04-12T03:19:43.339 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.339 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.339 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.339 DEBUG:boto:Params: {} 2024-04-12T03:19:43.339 DEBUG:boto:Token: None 2024-04-12T03:19:43.339 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj12 2024-04-12T03:19:43.339 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zp0A8sBQDknwnsdSBFNtG7SEHcY= 2024-04-12T03:19:43.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zp0A8sBQDknwnsdSBFNtG7SEHcY='} 2024-04-12T03:19:43.344 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000710ca795276b6c45-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.344 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.344 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.344 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.344 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.344 DEBUG:boto:Data: 2024-04-12T03:19:43.344 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.345 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.345 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.345 DEBUG:boto:Params: {} 2024-04-12T03:19:43.345 DEBUG:boto:Token: None 2024-04-12T03:19:43.345 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.345 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.347 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003997855eaf0ce8d5-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.348 DEBUG:boto:path=/dkyges-11/obj13 2024-04-12T03:19:43.348 DEBUG:boto:auth_path=/dkyges-11/obj13 2024-04-12T03:19:43.348 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.348 DEBUG:boto:Path: /dkyges-11/obj13 2024-04-12T03:19:43.348 DEBUG:boto:Data: 2024-04-12T03:19:43.348 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.348 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.348 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.348 DEBUG:boto:Params: {} 2024-04-12T03:19:43.348 DEBUG:boto:Token: None 2024-04-12T03:19:43.348 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj13 2024-04-12T03:19:43.349 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h+Mf1NUu3Mvmk9Ob/6L25rJR/AY= 2024-04-12T03:19:43.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h+Mf1NUu3Mvmk9Ob/6L25rJR/AY='} 2024-04-12T03:19:43.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d02300039323317e-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.391 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.391 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.391 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.391 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.391 DEBUG:boto:Data: 2024-04-12T03:19:43.391 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.391 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.391 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.391 DEBUG:boto:Params: {} 2024-04-12T03:19:43.392 DEBUG:boto:Token: None 2024-04-12T03:19:43.392 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.392 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ffc4e535b038bc19-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.394 DEBUG:boto:path=/dkyges-11/obj14 2024-04-12T03:19:43.394 DEBUG:boto:auth_path=/dkyges-11/obj14 2024-04-12T03:19:43.395 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.395 DEBUG:boto:Path: /dkyges-11/obj14 2024-04-12T03:19:43.395 DEBUG:boto:Data: 2024-04-12T03:19:43.395 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.395 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.395 DEBUG:boto:Params: {} 2024-04-12T03:19:43.395 DEBUG:boto:Token: None 2024-04-12T03:19:43.395 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj14 2024-04-12T03:19:43.395 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UfqeKfP7M6dFNgnLpkUCan/NWgE= 2024-04-12T03:19:43.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UfqeKfP7M6dFNgnLpkUCan/NWgE='} 2024-04-12T03:19:43.400 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1d47e5ec41a3a1f-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.400 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.400 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.400 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.400 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.400 DEBUG:boto:Data: 2024-04-12T03:19:43.401 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.401 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.401 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.401 DEBUG:boto:Params: {} 2024-04-12T03:19:43.401 DEBUG:boto:Token: None 2024-04-12T03:19:43.401 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.401 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.403 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d4c5e7ecc70167a-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.404 DEBUG:boto:path=/dkyges-11/obj15 2024-04-12T03:19:43.404 DEBUG:boto:auth_path=/dkyges-11/obj15 2024-04-12T03:19:43.404 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.404 DEBUG:boto:Path: /dkyges-11/obj15 2024-04-12T03:19:43.404 DEBUG:boto:Data: 2024-04-12T03:19:43.404 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.404 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.404 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.405 DEBUG:boto:Params: {} 2024-04-12T03:19:43.405 DEBUG:boto:Token: None 2024-04-12T03:19:43.405 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj15 2024-04-12T03:19:43.405 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Fpoo9Bkx2X+xOxOhAkRNlnIybjE= 2024-04-12T03:19:43.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Fpoo9Bkx2X+xOxOhAkRNlnIybjE='} 2024-04-12T03:19:43.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac8078199a9a8200-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.451 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.451 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.451 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.451 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.451 DEBUG:boto:Data: 2024-04-12T03:19:43.451 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.451 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.451 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.451 DEBUG:boto:Params: {} 2024-04-12T03:19:43.452 DEBUG:boto:Token: None 2024-04-12T03:19:43.452 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.452 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.454 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0606f16d4bcaa59-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.455 DEBUG:boto:path=/dkyges-11/obj16 2024-04-12T03:19:43.455 DEBUG:boto:auth_path=/dkyges-11/obj16 2024-04-12T03:19:43.455 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.455 DEBUG:boto:Path: /dkyges-11/obj16 2024-04-12T03:19:43.455 DEBUG:boto:Data: 2024-04-12T03:19:43.455 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.455 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.455 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.455 DEBUG:boto:Params: {} 2024-04-12T03:19:43.456 DEBUG:boto:Token: None 2024-04-12T03:19:43.456 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj16 2024-04-12T03:19:43.456 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VwLT5RYP7vXy6BgSzNofPRCKIIE= 2024-04-12T03:19:43.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VwLT5RYP7vXy6BgSzNofPRCKIIE='} 2024-04-12T03:19:43.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d16b1b72581bc09-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.462 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.462 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.462 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.462 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.462 DEBUG:boto:Data: 2024-04-12T03:19:43.463 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.463 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.463 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.463 DEBUG:boto:Params: {} 2024-04-12T03:19:43.463 DEBUG:boto:Token: None 2024-04-12T03:19:43.463 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.463 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.465 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072f78267e348a83b-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.465 DEBUG:boto:path=/dkyges-11/obj17 2024-04-12T03:19:43.466 DEBUG:boto:auth_path=/dkyges-11/obj17 2024-04-12T03:19:43.466 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.466 DEBUG:boto:Path: /dkyges-11/obj17 2024-04-12T03:19:43.466 DEBUG:boto:Data: 2024-04-12T03:19:43.466 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.466 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.466 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.466 DEBUG:boto:Params: {} 2024-04-12T03:19:43.466 DEBUG:boto:Token: None 2024-04-12T03:19:43.466 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj17 2024-04-12T03:19:43.467 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2mAebRhafTBwALFD6dVkP87KLbE= 2024-04-12T03:19:43.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2mAebRhafTBwALFD6dVkP87KLbE='} 2024-04-12T03:19:43.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055e1bb19d5ffc234-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.511 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.511 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.511 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.511 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.511 DEBUG:boto:Data: 2024-04-12T03:19:43.511 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.511 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.511 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.511 DEBUG:boto:Params: {} 2024-04-12T03:19:43.512 DEBUG:boto:Token: None 2024-04-12T03:19:43.512 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.512 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.514 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009de2b078a3d75b78-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.514 DEBUG:boto:path=/dkyges-11/obj18 2024-04-12T03:19:43.514 DEBUG:boto:auth_path=/dkyges-11/obj18 2024-04-12T03:19:43.514 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.515 DEBUG:boto:Path: /dkyges-11/obj18 2024-04-12T03:19:43.515 DEBUG:boto:Data: 2024-04-12T03:19:43.515 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.515 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.515 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.515 DEBUG:boto:Params: {} 2024-04-12T03:19:43.515 DEBUG:boto:Token: None 2024-04-12T03:19:43.515 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj18 2024-04-12T03:19:43.515 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mEOJWmMGNFl8hueiAts2j4KqPP0= 2024-04-12T03:19:43.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mEOJWmMGNFl8hueiAts2j4KqPP0='} 2024-04-12T03:19:43.520 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002905748ea6394a10-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.520 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.520 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.520 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.520 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.521 DEBUG:boto:Data: 2024-04-12T03:19:43.521 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.521 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.521 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.521 DEBUG:boto:Params: {} 2024-04-12T03:19:43.521 DEBUG:boto:Token: None 2024-04-12T03:19:43.521 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.521 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098aab3097af80562-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.524 DEBUG:boto:path=/dkyges-11/obj19 2024-04-12T03:19:43.524 DEBUG:boto:auth_path=/dkyges-11/obj19 2024-04-12T03:19:43.524 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.524 DEBUG:boto:Path: /dkyges-11/obj19 2024-04-12T03:19:43.524 DEBUG:boto:Data: 2024-04-12T03:19:43.524 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.524 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.524 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.524 DEBUG:boto:Params: {} 2024-04-12T03:19:43.524 DEBUG:boto:Token: None 2024-04-12T03:19:43.525 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj19 2024-04-12T03:19:43.525 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:itEUQwg9AGclj4ZM86Cnx9n/KcM= 2024-04-12T03:19:43.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:itEUQwg9AGclj4ZM86Cnx9n/KcM='} 2024-04-12T03:19:43.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002208e94b65f576e2-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.571 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.571 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.571 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.571 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.571 DEBUG:boto:Data: 2024-04-12T03:19:43.571 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.571 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.572 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.572 DEBUG:boto:Params: {} 2024-04-12T03:19:43.572 DEBUG:boto:Token: None 2024-04-12T03:19:43.572 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.572 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.574 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a49de8d063ca722a-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.575 DEBUG:boto:path=/dkyges-11/obj20 2024-04-12T03:19:43.575 DEBUG:boto:auth_path=/dkyges-11/obj20 2024-04-12T03:19:43.575 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.575 DEBUG:boto:Path: /dkyges-11/obj20 2024-04-12T03:19:43.575 DEBUG:boto:Data: 2024-04-12T03:19:43.575 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.575 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.575 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.575 DEBUG:boto:Params: {} 2024-04-12T03:19:43.575 DEBUG:boto:Token: None 2024-04-12T03:19:43.576 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj20 2024-04-12T03:19:43.576 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lqmGbMyLkdEPNlvUWyk5tp8uAlE= 2024-04-12T03:19:43.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lqmGbMyLkdEPNlvUWyk5tp8uAlE='} 2024-04-12T03:19:43.582 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000653e7fe23f834d6b-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.582 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.582 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.582 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.582 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.582 DEBUG:boto:Data: 2024-04-12T03:19:43.583 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.583 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.583 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.583 DEBUG:boto:Params: {} 2024-04-12T03:19:43.583 DEBUG:boto:Token: None 2024-04-12T03:19:43.583 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.583 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008cbb4af60581e901-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.586 DEBUG:boto:path=/dkyges-11/obj21 2024-04-12T03:19:43.586 DEBUG:boto:auth_path=/dkyges-11/obj21 2024-04-12T03:19:43.586 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.586 DEBUG:boto:Path: /dkyges-11/obj21 2024-04-12T03:19:43.586 DEBUG:boto:Data: 2024-04-12T03:19:43.586 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.586 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.586 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.587 DEBUG:boto:Params: {} 2024-04-12T03:19:43.587 DEBUG:boto:Token: None 2024-04-12T03:19:43.587 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj21 2024-04-12T03:19:43.587 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Lg1QYV5C6h3fgbQgnk5AfvaO4rI= 2024-04-12T03:19:43.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Lg1QYV5C6h3fgbQgnk5AfvaO4rI='} 2024-04-12T03:19:43.634 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000033cc4df884f8c13-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.635 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.635 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.635 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.635 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.635 DEBUG:boto:Data: 2024-04-12T03:19:43.635 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.635 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.635 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.635 DEBUG:boto:Params: {} 2024-04-12T03:19:43.636 DEBUG:boto:Token: None 2024-04-12T03:19:43.636 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.636 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.638 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000265957bfcdbb067c-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.638 DEBUG:boto:path=/dkyges-11/obj22 2024-04-12T03:19:43.639 DEBUG:boto:auth_path=/dkyges-11/obj22 2024-04-12T03:19:43.639 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.639 DEBUG:boto:Path: /dkyges-11/obj22 2024-04-12T03:19:43.639 DEBUG:boto:Data: 2024-04-12T03:19:43.639 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.639 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.639 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.639 DEBUG:boto:Params: {} 2024-04-12T03:19:43.639 DEBUG:boto:Token: None 2024-04-12T03:19:43.639 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj22 2024-04-12T03:19:43.640 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KIlhalNP/COI2aOP0Bj7FP8ANrQ= 2024-04-12T03:19:43.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KIlhalNP/COI2aOP0Bj7FP8ANrQ='} 2024-04-12T03:19:43.644 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000848f1d218d48e3b2-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.644 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.644 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.644 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.644 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.644 DEBUG:boto:Data: 2024-04-12T03:19:43.645 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.645 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.645 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.645 DEBUG:boto:Params: {} 2024-04-12T03:19:43.645 DEBUG:boto:Token: None 2024-04-12T03:19:43.645 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.645 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d84d77eaaac5594a-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.648 DEBUG:boto:path=/dkyges-11/obj23 2024-04-12T03:19:43.648 DEBUG:boto:auth_path=/dkyges-11/obj23 2024-04-12T03:19:43.649 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.649 DEBUG:boto:Path: /dkyges-11/obj23 2024-04-12T03:19:43.649 DEBUG:boto:Data: 2024-04-12T03:19:43.649 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.649 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.649 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.649 DEBUG:boto:Params: {} 2024-04-12T03:19:43.649 DEBUG:boto:Token: None 2024-04-12T03:19:43.650 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj23 2024-04-12T03:19:43.650 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8+2nrC/QdIGYVH638acuYD2yXRw= 2024-04-12T03:19:43.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8+2nrC/QdIGYVH638acuYD2yXRw='} 2024-04-12T03:19:43.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b27d0c0f955dac3-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.699 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.699 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.699 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.699 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.699 DEBUG:boto:Data: 2024-04-12T03:19:43.699 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.699 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.699 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.699 DEBUG:boto:Params: {} 2024-04-12T03:19:43.700 DEBUG:boto:Token: None 2024-04-12T03:19:43.700 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.700 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.702 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbe3add891c9bdc4-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.702 DEBUG:boto:path=/dkyges-11/obj24 2024-04-12T03:19:43.703 DEBUG:boto:auth_path=/dkyges-11/obj24 2024-04-12T03:19:43.703 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.703 DEBUG:boto:Path: /dkyges-11/obj24 2024-04-12T03:19:43.703 DEBUG:boto:Data: 2024-04-12T03:19:43.703 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.703 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.703 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.703 DEBUG:boto:Params: {} 2024-04-12T03:19:43.703 DEBUG:boto:Token: None 2024-04-12T03:19:43.703 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj24 2024-04-12T03:19:43.704 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/rZ1RRgcQKTeTFAxmebBcfa8yuI= 2024-04-12T03:19:43.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/rZ1RRgcQKTeTFAxmebBcfa8yuI='} 2024-04-12T03:19:43.708 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c21d1a73011187d-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.708 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.708 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.708 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.708 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.709 DEBUG:boto:Data: 2024-04-12T03:19:43.709 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.709 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.709 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.709 DEBUG:boto:Params: {} 2024-04-12T03:19:43.709 DEBUG:boto:Token: None 2024-04-12T03:19:43.709 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.709 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.712 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037ba770606cf3631-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.712 DEBUG:boto:path=/dkyges-11/obj25 2024-04-12T03:19:43.712 DEBUG:boto:auth_path=/dkyges-11/obj25 2024-04-12T03:19:43.712 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.712 DEBUG:boto:Path: /dkyges-11/obj25 2024-04-12T03:19:43.712 DEBUG:boto:Data: 2024-04-12T03:19:43.712 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.712 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.712 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.713 DEBUG:boto:Params: {} 2024-04-12T03:19:43.713 DEBUG:boto:Token: None 2024-04-12T03:19:43.713 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj25 2024-04-12T03:19:43.713 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FVmMBXm78/nKdEUB0qy/SgsO2dQ= 2024-04-12T03:19:43.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FVmMBXm78/nKdEUB0qy/SgsO2dQ='} 2024-04-12T03:19:43.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c19bf748c6a2eb89-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.763 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.763 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.763 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.763 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.763 DEBUG:boto:Data: 2024-04-12T03:19:43.763 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.763 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.763 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.764 DEBUG:boto:Params: {} 2024-04-12T03:19:43.764 DEBUG:boto:Token: None 2024-04-12T03:19:43.764 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.764 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.766 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000395c3958c601dcdc-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.767 DEBUG:boto:path=/dkyges-11/obj26 2024-04-12T03:19:43.767 DEBUG:boto:auth_path=/dkyges-11/obj26 2024-04-12T03:19:43.767 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.767 DEBUG:boto:Path: /dkyges-11/obj26 2024-04-12T03:19:43.767 DEBUG:boto:Data: 2024-04-12T03:19:43.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.767 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.767 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.768 DEBUG:boto:Params: {} 2024-04-12T03:19:43.768 DEBUG:boto:Token: None 2024-04-12T03:19:43.768 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj26 2024-04-12T03:19:43.768 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LddhUHoTPD15FCxkZKUwFFO8mvI= 2024-04-12T03:19:43.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LddhUHoTPD15FCxkZKUwFFO8mvI='} 2024-04-12T03:19:43.773 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8d1824a223f9a65-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.773 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.773 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.773 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.774 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.774 DEBUG:boto:Data: 2024-04-12T03:19:43.774 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.774 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.774 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.774 DEBUG:boto:Params: {} 2024-04-12T03:19:43.774 DEBUG:boto:Token: None 2024-04-12T03:19:43.774 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.775 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.777 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4047d7155801d26-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.778 DEBUG:boto:path=/dkyges-11/obj27 2024-04-12T03:19:43.778 DEBUG:boto:auth_path=/dkyges-11/obj27 2024-04-12T03:19:43.778 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.778 DEBUG:boto:Path: /dkyges-11/obj27 2024-04-12T03:19:43.778 DEBUG:boto:Data: 2024-04-12T03:19:43.778 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.778 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.778 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.779 DEBUG:boto:Params: {} 2024-04-12T03:19:43.779 DEBUG:boto:Token: None 2024-04-12T03:19:43.779 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj27 2024-04-12T03:19:43.779 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Kn/A4cGFUShfI/byazE38CGkfP8= 2024-04-12T03:19:43.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Kn/A4cGFUShfI/byazE38CGkfP8='} 2024-04-12T03:19:43.823 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7d29e7469fb3b83-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.823 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.823 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.823 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.823 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.824 DEBUG:boto:Data: 2024-04-12T03:19:43.824 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.824 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.824 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.824 DEBUG:boto:Params: {} 2024-04-12T03:19:43.824 DEBUG:boto:Token: None 2024-04-12T03:19:43.824 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.825 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c45823e45bdb65ca-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.827 DEBUG:boto:path=/dkyges-11/obj28 2024-04-12T03:19:43.828 DEBUG:boto:auth_path=/dkyges-11/obj28 2024-04-12T03:19:43.828 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.828 DEBUG:boto:Path: /dkyges-11/obj28 2024-04-12T03:19:43.828 DEBUG:boto:Data: 2024-04-12T03:19:43.828 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.828 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.828 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.828 DEBUG:boto:Params: {} 2024-04-12T03:19:43.828 DEBUG:boto:Token: None 2024-04-12T03:19:43.828 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj28 2024-04-12T03:19:43.829 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HtoY+JuD4fn9Aw3nSTAp6+RkXgc= 2024-04-12T03:19:43.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HtoY+JuD4fn9Aw3nSTAp6+RkXgc='} 2024-04-12T03:19:43.833 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e73871e8b8a5d42-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.833 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.833 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.833 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.833 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.833 DEBUG:boto:Data: 2024-04-12T03:19:43.833 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.833 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.834 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.834 DEBUG:boto:Params: {} 2024-04-12T03:19:43.834 DEBUG:boto:Token: None 2024-04-12T03:19:43.834 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.834 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.836 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000836ac96ba1f28dea-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.836 DEBUG:boto:path=/dkyges-11/obj29 2024-04-12T03:19:43.837 DEBUG:boto:auth_path=/dkyges-11/obj29 2024-04-12T03:19:43.837 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.837 DEBUG:boto:Path: /dkyges-11/obj29 2024-04-12T03:19:43.837 DEBUG:boto:Data: 2024-04-12T03:19:43.837 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.837 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.837 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.837 DEBUG:boto:Params: {} 2024-04-12T03:19:43.837 DEBUG:boto:Token: None 2024-04-12T03:19:43.837 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj29 2024-04-12T03:19:43.838 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rqSn+4Is4buxj3kH4po7XGRyKyI= 2024-04-12T03:19:43.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rqSn+4Is4buxj3kH4po7XGRyKyI='} 2024-04-12T03:19:43.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000118132f2ac6b2533-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.887 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.887 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.887 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.887 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.887 DEBUG:boto:Data: 2024-04-12T03:19:43.887 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.887 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.887 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.887 DEBUG:boto:Params: {} 2024-04-12T03:19:43.888 DEBUG:boto:Token: None 2024-04-12T03:19:43.888 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.888 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.890 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc37dcbe9e1eac87-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.891 DEBUG:boto:path=/dkyges-11/obj30 2024-04-12T03:19:43.891 DEBUG:boto:auth_path=/dkyges-11/obj30 2024-04-12T03:19:43.891 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.891 DEBUG:boto:Path: /dkyges-11/obj30 2024-04-12T03:19:43.891 DEBUG:boto:Data: 2024-04-12T03:19:43.891 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.891 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.891 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.891 DEBUG:boto:Params: {} 2024-04-12T03:19:43.891 DEBUG:boto:Token: None 2024-04-12T03:19:43.892 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj30 2024-04-12T03:19:43.892 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QCzWw5MMkh3texDZPmiwKuVVZJ0= 2024-04-12T03:19:43.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QCzWw5MMkh3texDZPmiwKuVVZJ0='} 2024-04-12T03:19:43.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab59fa2a13088b17-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.896 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.897 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.897 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.897 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.897 DEBUG:boto:Data: 2024-04-12T03:19:43.897 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.897 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.897 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.897 DEBUG:boto:Params: {} 2024-04-12T03:19:43.898 DEBUG:boto:Token: None 2024-04-12T03:19:43.898 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.898 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e6bdf5c1e7c5e19-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.901 DEBUG:boto:path=/dkyges-11/obj31 2024-04-12T03:19:43.901 DEBUG:boto:auth_path=/dkyges-11/obj31 2024-04-12T03:19:43.901 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.901 DEBUG:boto:Path: /dkyges-11/obj31 2024-04-12T03:19:43.901 DEBUG:boto:Data: 2024-04-12T03:19:43.901 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.901 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.902 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.902 DEBUG:boto:Params: {} 2024-04-12T03:19:43.902 DEBUG:boto:Token: None 2024-04-12T03:19:43.902 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj31 2024-04-12T03:19:43.903 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:97wxdRiwnUvfrpGZmT1EOthGQr8= 2024-04-12T03:19:43.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:97wxdRiwnUvfrpGZmT1EOthGQr8='} 2024-04-12T03:19:43.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d55ff1b6466fad9c-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.947 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.947 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.947 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.947 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.947 DEBUG:boto:Data: 2024-04-12T03:19:43.947 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.947 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.947 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.948 DEBUG:boto:Params: {} 2024-04-12T03:19:43.948 DEBUG:boto:Token: None 2024-04-12T03:19:43.948 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.948 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.950 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1bde25b6bbbb8fd-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.950 DEBUG:boto:path=/dkyges-11/obj32 2024-04-12T03:19:43.951 DEBUG:boto:auth_path=/dkyges-11/obj32 2024-04-12T03:19:43.951 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.951 DEBUG:boto:Path: /dkyges-11/obj32 2024-04-12T03:19:43.951 DEBUG:boto:Data: 2024-04-12T03:19:43.951 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.951 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.951 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.951 DEBUG:boto:Params: {} 2024-04-12T03:19:43.951 DEBUG:boto:Token: None 2024-04-12T03:19:43.951 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj32 2024-04-12T03:19:43.952 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0F1fOyCgvoaGiRQ6YfyFp+K4U+o= 2024-04-12T03:19:43.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0F1fOyCgvoaGiRQ6YfyFp+K4U+o='} 2024-04-12T03:19:43.956 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028082c7bc120ba8e-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.956 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:43.956 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:43.956 DEBUG:boto:Method: HEAD 2024-04-12T03:19:43.956 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:43.956 DEBUG:boto:Data: 2024-04-12T03:19:43.956 DEBUG:boto:Headers: {} 2024-04-12T03:19:43.957 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.957 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.957 DEBUG:boto:Params: {} 2024-04-12T03:19:43.957 DEBUG:boto:Token: None 2024-04-12T03:19:43.957 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/ 2024-04-12T03:19:43.957 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T03:19:43.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T03:19:43.959 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081893cc72bc7b6ef-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:43.960 DEBUG:boto:path=/dkyges-11/obj33 2024-04-12T03:19:43.960 DEBUG:boto:auth_path=/dkyges-11/obj33 2024-04-12T03:19:43.960 DEBUG:boto:Method: PUT 2024-04-12T03:19:43.960 DEBUG:boto:Path: /dkyges-11/obj33 2024-04-12T03:19:43.960 DEBUG:boto:Data: 2024-04-12T03:19:43.960 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:43.960 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:43.960 DEBUG:boto:Port: 8000 2024-04-12T03:19:43.960 DEBUG:boto:Params: {} 2024-04-12T03:19:43.960 DEBUG:boto:Token: None 2024-04-12T03:19:43.961 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:43 GMT /dkyges-11/obj33 2024-04-12T03:19:43.961 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8TJ91LlBh65Qm2SmEgrCwv6+1Yw= 2024-04-12T03:19:43.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8TJ91LlBh65Qm2SmEgrCwv6+1Yw='} 2024-04-12T03:19:44.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002deb7ff6c88f14cc-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.003 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:44.003 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:44.003 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.003 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:44.003 DEBUG:boto:Data: 2024-04-12T03:19:44.003 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.003 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.003 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.003 DEBUG:boto:Params: {} 2024-04-12T03:19:44.004 DEBUG:boto:Token: None 2024-04-12T03:19:44.004 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/ 2024-04-12T03:19:44.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T03:19:44.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T03:19:44.006 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fcaeec328d88847-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.007 DEBUG:boto:path=/dkyges-11/obj34 2024-04-12T03:19:44.007 DEBUG:boto:auth_path=/dkyges-11/obj34 2024-04-12T03:19:44.007 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.007 DEBUG:boto:Path: /dkyges-11/obj34 2024-04-12T03:19:44.007 DEBUG:boto:Data: 2024-04-12T03:19:44.007 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.007 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.007 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.007 DEBUG:boto:Params: {} 2024-04-12T03:19:44.008 DEBUG:boto:Token: None 2024-04-12T03:19:44.008 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/obj34 2024-04-12T03:19:44.008 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0VwER8EiO++IWqvaIFyg+0DATlE= 2024-04-12T03:19:44.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0VwER8EiO++IWqvaIFyg+0DATlE='} 2024-04-12T03:19:44.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000067730255c1721f9-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.013 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:44.013 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:44.013 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.013 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:44.013 DEBUG:boto:Data: 2024-04-12T03:19:44.013 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.014 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.014 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.014 DEBUG:boto:Params: {} 2024-04-12T03:19:44.014 DEBUG:boto:Token: None 2024-04-12T03:19:44.014 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/ 2024-04-12T03:19:44.015 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T03:19:44.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T03:19:44.017 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000047e5992745c221b-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.017 DEBUG:boto:path=/dkyges-11/obj35 2024-04-12T03:19:44.018 DEBUG:boto:auth_path=/dkyges-11/obj35 2024-04-12T03:19:44.018 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.018 DEBUG:boto:Path: /dkyges-11/obj35 2024-04-12T03:19:44.018 DEBUG:boto:Data: 2024-04-12T03:19:44.018 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.018 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.018 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.019 DEBUG:boto:Params: {} 2024-04-12T03:19:44.019 DEBUG:boto:Token: None 2024-04-12T03:19:44.019 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/obj35 2024-04-12T03:19:44.019 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:as9REBOJ+xJ/y+/lnXeyhXPRdk4= 2024-04-12T03:19:44.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:as9REBOJ+xJ/y+/lnXeyhXPRdk4='} 2024-04-12T03:19:44.024 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013948c89acdb51fa-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.024 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:44.024 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:44.024 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.024 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:44.025 DEBUG:boto:Data: 2024-04-12T03:19:44.025 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.025 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.025 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.025 DEBUG:boto:Params: {} 2024-04-12T03:19:44.025 DEBUG:boto:Token: None 2024-04-12T03:19:44.025 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/ 2024-04-12T03:19:44.025 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T03:19:44.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T03:19:44.028 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000342297c6c5ab6d4c-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.028 DEBUG:boto:path=/dkyges-11/obj36 2024-04-12T03:19:44.028 DEBUG:boto:auth_path=/dkyges-11/obj36 2024-04-12T03:19:44.028 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.028 DEBUG:boto:Path: /dkyges-11/obj36 2024-04-12T03:19:44.028 DEBUG:boto:Data: 2024-04-12T03:19:44.029 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.029 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.029 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.029 DEBUG:boto:Params: {} 2024-04-12T03:19:44.029 DEBUG:boto:Token: None 2024-04-12T03:19:44.029 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/obj36 2024-04-12T03:19:44.029 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U5YMNLJZkCeuYjWFVu1vZyvCJYo= 2024-04-12T03:19:44.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U5YMNLJZkCeuYjWFVu1vZyvCJYo='} 2024-04-12T03:19:44.075 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc24c685343b87dc-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.075 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:44.075 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:44.075 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.075 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:44.075 DEBUG:boto:Data: 2024-04-12T03:19:44.075 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.076 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.076 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.076 DEBUG:boto:Params: {} 2024-04-12T03:19:44.076 DEBUG:boto:Token: None 2024-04-12T03:19:44.076 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/ 2024-04-12T03:19:44.076 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T03:19:44.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T03:19:44.079 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c10e0715fbcac66-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.080 DEBUG:boto:path=/dkyges-11/obj37 2024-04-12T03:19:44.080 DEBUG:boto:auth_path=/dkyges-11/obj37 2024-04-12T03:19:44.080 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.080 DEBUG:boto:Path: /dkyges-11/obj37 2024-04-12T03:19:44.080 DEBUG:boto:Data: 2024-04-12T03:19:44.080 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.080 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.080 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.081 DEBUG:boto:Params: {} 2024-04-12T03:19:44.081 DEBUG:boto:Token: None 2024-04-12T03:19:44.082 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/obj37 2024-04-12T03:19:44.082 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:36dCdMGvj3WxNR2dgyq3h8R81hg= 2024-04-12T03:19:44.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:36dCdMGvj3WxNR2dgyq3h8R81hg='} 2024-04-12T03:19:44.091 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056bd06e54d2cce68-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.092 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:44.092 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:44.092 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.092 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:44.093 DEBUG:boto:Data: 2024-04-12T03:19:44.093 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.094 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.094 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.094 DEBUG:boto:Params: {} 2024-04-12T03:19:44.095 DEBUG:boto:Token: None 2024-04-12T03:19:44.096 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/ 2024-04-12T03:19:44.097 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T03:19:44.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T03:19:44.109 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003501098c5a3061c7-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.110 DEBUG:boto:path=/dkyges-11/obj38 2024-04-12T03:19:44.110 DEBUG:boto:auth_path=/dkyges-11/obj38 2024-04-12T03:19:44.110 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.110 DEBUG:boto:Path: /dkyges-11/obj38 2024-04-12T03:19:44.111 DEBUG:boto:Data: 2024-04-12T03:19:44.111 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.111 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.111 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.111 DEBUG:boto:Params: {} 2024-04-12T03:19:44.111 DEBUG:boto:Token: None 2024-04-12T03:19:44.112 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/obj38 2024-04-12T03:19:44.114 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4DpKWPaZBPKcgJ9SJLxR+XS1XPM= 2024-04-12T03:19:44.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4DpKWPaZBPKcgJ9SJLxR+XS1XPM='} 2024-04-12T03:19:44.161 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000516c30a9662e279c-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.161 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:44.161 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:44.161 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.161 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:44.161 DEBUG:boto:Data: 2024-04-12T03:19:44.161 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.161 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.162 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.162 DEBUG:boto:Params: {} 2024-04-12T03:19:44.162 DEBUG:boto:Token: None 2024-04-12T03:19:44.162 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/ 2024-04-12T03:19:44.162 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T03:19:44.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T03:19:44.165 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dec81a2fcb4196e1-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.165 DEBUG:boto:path=/dkyges-11/obj39 2024-04-12T03:19:44.165 DEBUG:boto:auth_path=/dkyges-11/obj39 2024-04-12T03:19:44.165 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.165 DEBUG:boto:Path: /dkyges-11/obj39 2024-04-12T03:19:44.165 DEBUG:boto:Data: 2024-04-12T03:19:44.166 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.166 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.166 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.166 DEBUG:boto:Params: {} 2024-04-12T03:19:44.166 DEBUG:boto:Token: None 2024-04-12T03:19:44.166 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/obj39 2024-04-12T03:19:44.166 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:utygtF/ZtbScdedqS6VAc8M1Hgc= 2024-04-12T03:19:44.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:utygtF/ZtbScdedqS6VAc8M1Hgc='} 2024-04-12T03:19:44.176 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7507505292773e8-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.176 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:44.177 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:44.177 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.177 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:44.177 DEBUG:boto:Data: 2024-04-12T03:19:44.177 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.178 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.178 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.178 DEBUG:boto:Params: {} 2024-04-12T03:19:44.178 DEBUG:boto:Token: None 2024-04-12T03:19:44.179 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/ 2024-04-12T03:19:44.179 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T03:19:44.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T03:19:44.182 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041c407d352ac860b-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.183 DEBUG:boto:path=/dkyges-11/obj40 2024-04-12T03:19:44.183 DEBUG:boto:auth_path=/dkyges-11/obj40 2024-04-12T03:19:44.183 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.184 DEBUG:boto:Path: /dkyges-11/obj40 2024-04-12T03:19:44.184 DEBUG:boto:Data: 2024-04-12T03:19:44.184 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.184 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.184 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.184 DEBUG:boto:Params: {} 2024-04-12T03:19:44.185 DEBUG:boto:Token: None 2024-04-12T03:19:44.185 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/obj40 2024-04-12T03:19:44.185 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UUggWyiojPRvvfSUSi1/u/Gu0m0= 2024-04-12T03:19:44.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UUggWyiojPRvvfSUSi1/u/Gu0m0='} 2024-04-12T03:19:44.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031503d375085ce6c-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.199 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:44.199 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:44.199 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.199 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:44.199 DEBUG:boto:Data: 2024-04-12T03:19:44.200 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.200 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.200 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.200 DEBUG:boto:Params: {} 2024-04-12T03:19:44.200 DEBUG:boto:Token: None 2024-04-12T03:19:44.201 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/ 2024-04-12T03:19:44.201 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T03:19:44.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T03:19:44.207 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8eba9346bb78122-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.208 DEBUG:boto:path=/dkyges-11/obj41 2024-04-12T03:19:44.208 DEBUG:boto:auth_path=/dkyges-11/obj41 2024-04-12T03:19:44.208 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.209 DEBUG:boto:Path: /dkyges-11/obj41 2024-04-12T03:19:44.209 DEBUG:boto:Data: 2024-04-12T03:19:44.209 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.209 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.209 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.209 DEBUG:boto:Params: {} 2024-04-12T03:19:44.210 DEBUG:boto:Token: None 2024-04-12T03:19:44.210 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/obj41 2024-04-12T03:19:44.210 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:y2uarvryy8SHjdJtlDpwzRGkx8g= 2024-04-12T03:19:44.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y2uarvryy8SHjdJtlDpwzRGkx8g='} 2024-04-12T03:19:44.261 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c27ab103000f0bd7-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.261 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:44.262 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:44.262 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.262 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:44.262 DEBUG:boto:Data: 2024-04-12T03:19:44.262 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.262 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.263 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.263 DEBUG:boto:Params: {} 2024-04-12T03:19:44.263 DEBUG:boto:Token: None 2024-04-12T03:19:44.263 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/ 2024-04-12T03:19:44.263 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T03:19:44.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T03:19:44.269 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005240687dbc180963-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.269 DEBUG:boto:path=/dkyges-11/obj42 2024-04-12T03:19:44.269 DEBUG:boto:auth_path=/dkyges-11/obj42 2024-04-12T03:19:44.269 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.270 DEBUG:boto:Path: /dkyges-11/obj42 2024-04-12T03:19:44.270 DEBUG:boto:Data: 2024-04-12T03:19:44.270 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.270 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.270 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.270 DEBUG:boto:Params: {} 2024-04-12T03:19:44.271 DEBUG:boto:Token: None 2024-04-12T03:19:44.271 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/obj42 2024-04-12T03:19:44.271 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l/YoM2VQHWcxIQLRZ4i99q+ju2s= 2024-04-12T03:19:44.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l/YoM2VQHWcxIQLRZ4i99q+ju2s='} 2024-04-12T03:19:44.279 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a7a8e45de3d5c3b-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.279 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:44.279 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:44.280 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.280 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:44.280 DEBUG:boto:Data: 2024-04-12T03:19:44.280 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.280 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.280 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.280 DEBUG:boto:Params: {} 2024-04-12T03:19:44.280 DEBUG:boto:Token: None 2024-04-12T03:19:44.280 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/ 2024-04-12T03:19:44.281 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T03:19:44.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T03:19:44.283 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a61f10d5ea9b1772-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.284 DEBUG:boto:path=/dkyges-11/obj43 2024-04-12T03:19:44.284 DEBUG:boto:auth_path=/dkyges-11/obj43 2024-04-12T03:19:44.284 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.284 DEBUG:boto:Path: /dkyges-11/obj43 2024-04-12T03:19:44.284 DEBUG:boto:Data: 2024-04-12T03:19:44.284 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.284 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.284 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.284 DEBUG:boto:Params: {} 2024-04-12T03:19:44.284 DEBUG:boto:Token: None 2024-04-12T03:19:44.285 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/obj43 2024-04-12T03:19:44.285 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6lIxmEh3YhFhAlmV9vYBx3mQgDk= 2024-04-12T03:19:44.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6lIxmEh3YhFhAlmV9vYBx3mQgDk='} 2024-04-12T03:19:44.331 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a79a620272d662c-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.332 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:44.332 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:44.332 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.332 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:44.332 DEBUG:boto:Data: 2024-04-12T03:19:44.332 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.332 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.333 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.333 DEBUG:boto:Params: {} 2024-04-12T03:19:44.333 DEBUG:boto:Token: None 2024-04-12T03:19:44.333 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/ 2024-04-12T03:19:44.333 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T03:19:44.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T03:19:44.336 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ea0164165abad26-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.337 DEBUG:boto:path=/dkyges-11/obj44 2024-04-12T03:19:44.337 DEBUG:boto:auth_path=/dkyges-11/obj44 2024-04-12T03:19:44.337 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.337 DEBUG:boto:Path: /dkyges-11/obj44 2024-04-12T03:19:44.338 DEBUG:boto:Data: 2024-04-12T03:19:44.338 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.338 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.338 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.338 DEBUG:boto:Params: {} 2024-04-12T03:19:44.338 DEBUG:boto:Token: None 2024-04-12T03:19:44.338 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/obj44 2024-04-12T03:19:44.339 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DGr4zG6HCxc995i0C6j3e77l5BM= 2024-04-12T03:19:44.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DGr4zG6HCxc995i0C6j3e77l5BM='} 2024-04-12T03:19:44.345 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6af4d5f83480b46-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.345 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:44.345 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:44.346 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.346 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:44.346 DEBUG:boto:Data: 2024-04-12T03:19:44.346 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.346 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.346 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.346 DEBUG:boto:Params: {} 2024-04-12T03:19:44.346 DEBUG:boto:Token: None 2024-04-12T03:19:44.347 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/ 2024-04-12T03:19:44.347 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T03:19:44.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T03:19:44.350 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000118d8aff010afe5d-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.350 DEBUG:boto:path=/dkyges-11/obj45 2024-04-12T03:19:44.350 DEBUG:boto:auth_path=/dkyges-11/obj45 2024-04-12T03:19:44.350 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.351 DEBUG:boto:Path: /dkyges-11/obj45 2024-04-12T03:19:44.351 DEBUG:boto:Data: 2024-04-12T03:19:44.351 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.351 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.351 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.351 DEBUG:boto:Params: {} 2024-04-12T03:19:44.351 DEBUG:boto:Token: None 2024-04-12T03:19:44.352 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/obj45 2024-04-12T03:19:44.352 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YOIT4pZKtkIvNLGAxvv9kvnrXwU= 2024-04-12T03:19:44.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YOIT4pZKtkIvNLGAxvv9kvnrXwU='} 2024-04-12T03:19:44.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000099aad73257837543-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.357 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:44.357 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:44.357 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.357 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:44.357 DEBUG:boto:Data: 2024-04-12T03:19:44.357 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.357 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.357 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.357 DEBUG:boto:Params: {} 2024-04-12T03:19:44.358 DEBUG:boto:Token: None 2024-04-12T03:19:44.358 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/ 2024-04-12T03:19:44.358 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T03:19:44.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T03:19:44.361 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f63b15f269533e4d-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.361 DEBUG:boto:path=/dkyges-11/obj46 2024-04-12T03:19:44.362 DEBUG:boto:auth_path=/dkyges-11/obj46 2024-04-12T03:19:44.362 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.362 DEBUG:boto:Path: /dkyges-11/obj46 2024-04-12T03:19:44.362 DEBUG:boto:Data: 2024-04-12T03:19:44.362 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.362 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.362 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.362 DEBUG:boto:Params: {} 2024-04-12T03:19:44.363 DEBUG:boto:Token: None 2024-04-12T03:19:44.363 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/obj46 2024-04-12T03:19:44.363 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kpv+KU73udWLB5pj8OLkclxixFQ= 2024-04-12T03:19:44.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kpv+KU73udWLB5pj8OLkclxixFQ='} 2024-04-12T03:19:44.407 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e59f2bcb536f4758-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.407 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:44.407 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:44.407 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.407 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:44.407 DEBUG:boto:Data: 2024-04-12T03:19:44.408 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.408 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.408 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.408 DEBUG:boto:Params: {} 2024-04-12T03:19:44.408 DEBUG:boto:Token: None 2024-04-12T03:19:44.408 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/ 2024-04-12T03:19:44.408 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T03:19:44.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T03:19:44.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d839c2f40f78218-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.412 DEBUG:boto:path=/dkyges-11/obj47 2024-04-12T03:19:44.413 DEBUG:boto:auth_path=/dkyges-11/obj47 2024-04-12T03:19:44.413 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.413 DEBUG:boto:Path: /dkyges-11/obj47 2024-04-12T03:19:44.413 DEBUG:boto:Data: 2024-04-12T03:19:44.413 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.413 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.414 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.414 DEBUG:boto:Params: {} 2024-04-12T03:19:44.414 DEBUG:boto:Token: None 2024-04-12T03:19:44.414 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/obj47 2024-04-12T03:19:44.414 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uVzhM9ouvg+OBo74hifbl3tXEUI= 2024-04-12T03:19:44.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uVzhM9ouvg+OBo74hifbl3tXEUI='} 2024-04-12T03:19:44.420 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7f985fa5db3ba67-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.420 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:44.420 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:44.421 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.421 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:44.421 DEBUG:boto:Data: 2024-04-12T03:19:44.421 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.421 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.421 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.421 DEBUG:boto:Params: {} 2024-04-12T03:19:44.421 DEBUG:boto:Token: None 2024-04-12T03:19:44.422 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/ 2024-04-12T03:19:44.422 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T03:19:44.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T03:19:44.425 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a74bc216203e2f8-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.425 DEBUG:boto:path=/dkyges-11/obj48 2024-04-12T03:19:44.425 DEBUG:boto:auth_path=/dkyges-11/obj48 2024-04-12T03:19:44.425 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.425 DEBUG:boto:Path: /dkyges-11/obj48 2024-04-12T03:19:44.425 DEBUG:boto:Data: 2024-04-12T03:19:44.426 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.426 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.426 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.426 DEBUG:boto:Params: {} 2024-04-12T03:19:44.426 DEBUG:boto:Token: None 2024-04-12T03:19:44.426 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/obj48 2024-04-12T03:19:44.426 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VKlsaUBc5SvMyT7KJfE5NZRowG4= 2024-04-12T03:19:44.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VKlsaUBc5SvMyT7KJfE5NZRowG4='} 2024-04-12T03:19:44.475 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d92a3ac7625eb87-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.475 DEBUG:boto:path=/dkyges-11/ 2024-04-12T03:19:44.475 DEBUG:boto:auth_path=/dkyges-11/ 2024-04-12T03:19:44.476 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.476 DEBUG:boto:Path: /dkyges-11/ 2024-04-12T03:19:44.476 DEBUG:boto:Data: 2024-04-12T03:19:44.476 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.476 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.476 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.476 DEBUG:boto:Params: {} 2024-04-12T03:19:44.476 DEBUG:boto:Token: None 2024-04-12T03:19:44.477 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/ 2024-04-12T03:19:44.477 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T03:19:44.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T03:19:44.480 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080cb53e446b63b8e-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.480 DEBUG:boto:path=/dkyges-11/obj49 2024-04-12T03:19:44.480 DEBUG:boto:auth_path=/dkyges-11/obj49 2024-04-12T03:19:44.481 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.481 DEBUG:boto:Path: /dkyges-11/obj49 2024-04-12T03:19:44.481 DEBUG:boto:Data: 2024-04-12T03:19:44.481 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.481 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:19:44.481 DEBUG:boto:Port: 8000 2024-04-12T03:19:44.481 DEBUG:boto:Params: {} 2024-04-12T03:19:44.482 DEBUG:boto:Token: None 2024-04-12T03:19:44.482 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-11/obj49 2024-04-12T03:19:44.482 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZZLdNKc1LHvY51slZRdSY6ElTRU= 2024-04-12T03:19:44.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZZLdNKc1LHvY51slZRdSY6ElTRU='} 2024-04-12T03:19:44.487 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b5953beddaeb8fa-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.488 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.488 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.488 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.488 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.488 DEBUG:boto:Data: 2024-04-12T03:19:44.488 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.488 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.488 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.489 DEBUG:boto:Params: {} 2024-04-12T03:19:44.489 DEBUG:boto:Token: None 2024-04-12T03:19:44.489 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.489 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.492 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b32606363e38747-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.493 DEBUG:boto:path=/dkyges-12/obj1 2024-04-12T03:19:44.493 DEBUG:boto:auth_path=/dkyges-12/obj1 2024-04-12T03:19:44.493 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.493 DEBUG:boto:Path: /dkyges-12/obj1 2024-04-12T03:19:44.493 DEBUG:boto:Data: 2024-04-12T03:19:44.493 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.493 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.494 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.494 DEBUG:boto:Params: {} 2024-04-12T03:19:44.494 DEBUG:boto:Token: None 2024-04-12T03:19:44.494 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj1 2024-04-12T03:19:44.494 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NqXG9TTTiseiusyRhR1rwfOFDXY= 2024-04-12T03:19:44.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NqXG9TTTiseiusyRhR1rwfOFDXY='} 2024-04-12T03:19:44.504 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a04ea9e897d56d7-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.504 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.504 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.505 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.505 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.505 DEBUG:boto:Data: 2024-04-12T03:19:44.505 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.505 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.505 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.506 DEBUG:boto:Params: {} 2024-04-12T03:19:44.506 DEBUG:boto:Token: None 2024-04-12T03:19:44.506 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.506 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cad09d2ee0bfdd1c-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.511 DEBUG:boto:path=/dkyges-12/obj2 2024-04-12T03:19:44.511 DEBUG:boto:auth_path=/dkyges-12/obj2 2024-04-12T03:19:44.511 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.511 DEBUG:boto:Path: /dkyges-12/obj2 2024-04-12T03:19:44.511 DEBUG:boto:Data: 2024-04-12T03:19:44.512 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.512 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.512 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.512 DEBUG:boto:Params: {} 2024-04-12T03:19:44.512 DEBUG:boto:Token: None 2024-04-12T03:19:44.513 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj2 2024-04-12T03:19:44.513 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kn4mUiNbzJluDTec/6GIqFaT+J0= 2024-04-12T03:19:44.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kn4mUiNbzJluDTec/6GIqFaT+J0='} 2024-04-12T03:19:44.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004578f5dc7ac83518-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.559 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.559 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.560 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.560 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.560 DEBUG:boto:Data: 2024-04-12T03:19:44.560 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.560 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.560 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.560 DEBUG:boto:Params: {} 2024-04-12T03:19:44.561 DEBUG:boto:Token: None 2024-04-12T03:19:44.561 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.561 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2f3ae30ac969463-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.565 DEBUG:boto:path=/dkyges-12/obj3 2024-04-12T03:19:44.565 DEBUG:boto:auth_path=/dkyges-12/obj3 2024-04-12T03:19:44.565 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.565 DEBUG:boto:Path: /dkyges-12/obj3 2024-04-12T03:19:44.565 DEBUG:boto:Data: 2024-04-12T03:19:44.566 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.566 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.566 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.566 DEBUG:boto:Params: {} 2024-04-12T03:19:44.566 DEBUG:boto:Token: None 2024-04-12T03:19:44.566 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj3 2024-04-12T03:19:44.567 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RKwbP1p6J+FnENsO8d6uDV1wdSs= 2024-04-12T03:19:44.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RKwbP1p6J+FnENsO8d6uDV1wdSs='} 2024-04-12T03:19:44.573 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2c5f486f95c8896-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.573 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.573 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.574 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.574 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.574 DEBUG:boto:Data: 2024-04-12T03:19:44.574 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.574 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.574 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.574 DEBUG:boto:Params: {} 2024-04-12T03:19:44.574 DEBUG:boto:Token: None 2024-04-12T03:19:44.574 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.575 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000437b0a6cf7e81652-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.577 DEBUG:boto:path=/dkyges-12/obj4 2024-04-12T03:19:44.577 DEBUG:boto:auth_path=/dkyges-12/obj4 2024-04-12T03:19:44.577 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.577 DEBUG:boto:Path: /dkyges-12/obj4 2024-04-12T03:19:44.577 DEBUG:boto:Data: 2024-04-12T03:19:44.578 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.578 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.578 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.578 DEBUG:boto:Params: {} 2024-04-12T03:19:44.578 DEBUG:boto:Token: None 2024-04-12T03:19:44.578 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj4 2024-04-12T03:19:44.578 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y7ya2D8d41f3AgpUy4bE0/SZERc= 2024-04-12T03:19:44.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y7ya2D8d41f3AgpUy4bE0/SZERc='} 2024-04-12T03:19:44.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3f98b3278607e04-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.623 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.623 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.623 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.623 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.623 DEBUG:boto:Data: 2024-04-12T03:19:44.623 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.623 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.623 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.624 DEBUG:boto:Params: {} 2024-04-12T03:19:44.624 DEBUG:boto:Token: None 2024-04-12T03:19:44.624 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.624 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe779de6c464b0ed-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.627 DEBUG:boto:path=/dkyges-12/obj5 2024-04-12T03:19:44.627 DEBUG:boto:auth_path=/dkyges-12/obj5 2024-04-12T03:19:44.627 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.627 DEBUG:boto:Path: /dkyges-12/obj5 2024-04-12T03:19:44.627 DEBUG:boto:Data: 2024-04-12T03:19:44.627 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.627 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.627 DEBUG:boto:Params: {} 2024-04-12T03:19:44.627 DEBUG:boto:Token: None 2024-04-12T03:19:44.628 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj5 2024-04-12T03:19:44.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z/6jw5JCXFD/ab8nk7vl/YPw5dE= 2024-04-12T03:19:44.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z/6jw5JCXFD/ab8nk7vl/YPw5dE='} 2024-04-12T03:19:44.632 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006120456b16db7dcc-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.632 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.632 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.632 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.633 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.633 DEBUG:boto:Data: 2024-04-12T03:19:44.633 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.633 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.633 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.633 DEBUG:boto:Params: {} 2024-04-12T03:19:44.633 DEBUG:boto:Token: None 2024-04-12T03:19:44.633 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.633 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.636 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e04a1fd143b8dd9c-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.636 DEBUG:boto:path=/dkyges-12/obj6 2024-04-12T03:19:44.636 DEBUG:boto:auth_path=/dkyges-12/obj6 2024-04-12T03:19:44.636 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.636 DEBUG:boto:Path: /dkyges-12/obj6 2024-04-12T03:19:44.636 DEBUG:boto:Data: 2024-04-12T03:19:44.636 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.636 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.637 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.637 DEBUG:boto:Params: {} 2024-04-12T03:19:44.637 DEBUG:boto:Token: None 2024-04-12T03:19:44.637 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj6 2024-04-12T03:19:44.637 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qpXvnTY42Jhx+1LAv2Zv6hyS8Y4= 2024-04-12T03:19:44.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qpXvnTY42Jhx+1LAv2Zv6hyS8Y4='} 2024-04-12T03:19:44.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ff1220d0020d0c2-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.683 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.683 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.683 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.683 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.683 DEBUG:boto:Data: 2024-04-12T03:19:44.683 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.683 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.683 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.684 DEBUG:boto:Params: {} 2024-04-12T03:19:44.684 DEBUG:boto:Token: None 2024-04-12T03:19:44.684 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.684 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.686 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed0d3ba74b5d788f-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.687 DEBUG:boto:path=/dkyges-12/obj7 2024-04-12T03:19:44.687 DEBUG:boto:auth_path=/dkyges-12/obj7 2024-04-12T03:19:44.687 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.687 DEBUG:boto:Path: /dkyges-12/obj7 2024-04-12T03:19:44.687 DEBUG:boto:Data: 2024-04-12T03:19:44.687 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.687 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.687 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.687 DEBUG:boto:Params: {} 2024-04-12T03:19:44.688 DEBUG:boto:Token: None 2024-04-12T03:19:44.688 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj7 2024-04-12T03:19:44.688 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7nwG1CznDOPhxLJNwoa3Si3jxIY= 2024-04-12T03:19:44.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7nwG1CznDOPhxLJNwoa3Si3jxIY='} 2024-04-12T03:19:44.693 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000686fa4dbd1a3ad50-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.693 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.693 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.693 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.693 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.693 DEBUG:boto:Data: 2024-04-12T03:19:44.693 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.694 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.694 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.694 DEBUG:boto:Params: {} 2024-04-12T03:19:44.694 DEBUG:boto:Token: None 2024-04-12T03:19:44.694 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.694 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.696 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007db1c156cb533152-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.697 DEBUG:boto:path=/dkyges-12/obj8 2024-04-12T03:19:44.697 DEBUG:boto:auth_path=/dkyges-12/obj8 2024-04-12T03:19:44.697 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.697 DEBUG:boto:Path: /dkyges-12/obj8 2024-04-12T03:19:44.697 DEBUG:boto:Data: 2024-04-12T03:19:44.697 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.697 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.697 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.698 DEBUG:boto:Params: {} 2024-04-12T03:19:44.698 DEBUG:boto:Token: None 2024-04-12T03:19:44.698 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj8 2024-04-12T03:19:44.698 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j86hssB4cD5ntc1FlvFrovbizog= 2024-04-12T03:19:44.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j86hssB4cD5ntc1FlvFrovbizog='} 2024-04-12T03:19:44.743 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4b02bc46156016c-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.743 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.743 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.743 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.743 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.743 DEBUG:boto:Data: 2024-04-12T03:19:44.743 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.743 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.744 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.744 DEBUG:boto:Params: {} 2024-04-12T03:19:44.744 DEBUG:boto:Token: None 2024-04-12T03:19:44.744 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.744 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.747 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a99c514204e3b6d-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.747 DEBUG:boto:path=/dkyges-12/obj9 2024-04-12T03:19:44.747 DEBUG:boto:auth_path=/dkyges-12/obj9 2024-04-12T03:19:44.747 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.747 DEBUG:boto:Path: /dkyges-12/obj9 2024-04-12T03:19:44.747 DEBUG:boto:Data: 2024-04-12T03:19:44.748 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.748 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.748 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.748 DEBUG:boto:Params: {} 2024-04-12T03:19:44.748 DEBUG:boto:Token: None 2024-04-12T03:19:44.748 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj9 2024-04-12T03:19:44.748 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CsXKGfbrEIDSQUYZesNFbEOg2LY= 2024-04-12T03:19:44.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CsXKGfbrEIDSQUYZesNFbEOg2LY='} 2024-04-12T03:19:44.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6739e934794ce02-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.755 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.755 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.755 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.755 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.755 DEBUG:boto:Data: 2024-04-12T03:19:44.755 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.755 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.755 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.755 DEBUG:boto:Params: {} 2024-04-12T03:19:44.756 DEBUG:boto:Token: None 2024-04-12T03:19:44.756 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.756 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.758 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053bc65069c28e3dd-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.759 DEBUG:boto:path=/dkyges-12/obj10 2024-04-12T03:19:44.759 DEBUG:boto:auth_path=/dkyges-12/obj10 2024-04-12T03:19:44.759 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.759 DEBUG:boto:Path: /dkyges-12/obj10 2024-04-12T03:19:44.759 DEBUG:boto:Data: 2024-04-12T03:19:44.759 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.759 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.759 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.759 DEBUG:boto:Params: {} 2024-04-12T03:19:44.759 DEBUG:boto:Token: None 2024-04-12T03:19:44.760 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj10 2024-04-12T03:19:44.760 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Z3k7JLDyyosv5ON58EeA0n9ElK0= 2024-04-12T03:19:44.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Z3k7JLDyyosv5ON58EeA0n9ElK0='} 2024-04-12T03:19:44.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007093bc2223fb838b-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.803 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.803 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.803 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.803 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.803 DEBUG:boto:Data: 2024-04-12T03:19:44.803 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.803 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.803 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.804 DEBUG:boto:Params: {} 2024-04-12T03:19:44.804 DEBUG:boto:Token: None 2024-04-12T03:19:44.804 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.804 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b34b205ac548b067-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.807 DEBUG:boto:path=/dkyges-12/obj11 2024-04-12T03:19:44.807 DEBUG:boto:auth_path=/dkyges-12/obj11 2024-04-12T03:19:44.807 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.807 DEBUG:boto:Path: /dkyges-12/obj11 2024-04-12T03:19:44.807 DEBUG:boto:Data: 2024-04-12T03:19:44.807 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.808 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.808 DEBUG:boto:Params: {} 2024-04-12T03:19:44.808 DEBUG:boto:Token: None 2024-04-12T03:19:44.808 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj11 2024-04-12T03:19:44.808 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:twBAwyacwShaY98wwdpx5PZfRfo= 2024-04-12T03:19:44.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:twBAwyacwShaY98wwdpx5PZfRfo='} 2024-04-12T03:19:44.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020e1619756171fab-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.815 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.815 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.815 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.815 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.815 DEBUG:boto:Data: 2024-04-12T03:19:44.815 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.815 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.815 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.815 DEBUG:boto:Params: {} 2024-04-12T03:19:44.816 DEBUG:boto:Token: None 2024-04-12T03:19:44.816 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.816 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.818 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060095ff90ed3d3ba-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.818 DEBUG:boto:path=/dkyges-12/obj12 2024-04-12T03:19:44.818 DEBUG:boto:auth_path=/dkyges-12/obj12 2024-04-12T03:19:44.819 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.819 DEBUG:boto:Path: /dkyges-12/obj12 2024-04-12T03:19:44.819 DEBUG:boto:Data: 2024-04-12T03:19:44.819 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.819 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.819 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.819 DEBUG:boto:Params: {} 2024-04-12T03:19:44.819 DEBUG:boto:Token: None 2024-04-12T03:19:44.819 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj12 2024-04-12T03:19:44.819 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8YgdE6UHNUyd0w6UYsG12U7e5S4= 2024-04-12T03:19:44.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8YgdE6UHNUyd0w6UYsG12U7e5S4='} 2024-04-12T03:19:44.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000895d92c39728cf74-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.863 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.863 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.863 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.864 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.864 DEBUG:boto:Data: 2024-04-12T03:19:44.864 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.864 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.864 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.864 DEBUG:boto:Params: {} 2024-04-12T03:19:44.864 DEBUG:boto:Token: None 2024-04-12T03:19:44.865 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.865 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.868 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e77a58ee8506603-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.869 DEBUG:boto:path=/dkyges-12/obj13 2024-04-12T03:19:44.869 DEBUG:boto:auth_path=/dkyges-12/obj13 2024-04-12T03:19:44.869 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.869 DEBUG:boto:Path: /dkyges-12/obj13 2024-04-12T03:19:44.869 DEBUG:boto:Data: 2024-04-12T03:19:44.869 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.869 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.869 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.869 DEBUG:boto:Params: {} 2024-04-12T03:19:44.869 DEBUG:boto:Token: None 2024-04-12T03:19:44.870 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj13 2024-04-12T03:19:44.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/mgLGh0Wy/jMG/vJWrLAzqC220A= 2024-04-12T03:19:44.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/mgLGh0Wy/jMG/vJWrLAzqC220A='} 2024-04-12T03:19:44.875 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1e3b2ce105470ea-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.875 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.875 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.875 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.876 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.876 DEBUG:boto:Data: 2024-04-12T03:19:44.876 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.876 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.876 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.876 DEBUG:boto:Params: {} 2024-04-12T03:19:44.876 DEBUG:boto:Token: None 2024-04-12T03:19:44.876 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.876 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.879 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000473e0f92d9e9c784-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.879 DEBUG:boto:path=/dkyges-12/obj14 2024-04-12T03:19:44.879 DEBUG:boto:auth_path=/dkyges-12/obj14 2024-04-12T03:19:44.879 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.879 DEBUG:boto:Path: /dkyges-12/obj14 2024-04-12T03:19:44.879 DEBUG:boto:Data: 2024-04-12T03:19:44.880 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.880 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.880 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.880 DEBUG:boto:Params: {} 2024-04-12T03:19:44.880 DEBUG:boto:Token: None 2024-04-12T03:19:44.880 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj14 2024-04-12T03:19:44.880 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KqN+ImU72TahQ+RMjSf0cRBIdI4= 2024-04-12T03:19:44.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KqN+ImU72TahQ+RMjSf0cRBIdI4='} 2024-04-12T03:19:44.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9b98cce82b31a79-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.923 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.923 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.923 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.923 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.923 DEBUG:boto:Data: 2024-04-12T03:19:44.923 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.923 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.923 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.924 DEBUG:boto:Params: {} 2024-04-12T03:19:44.924 DEBUG:boto:Token: None 2024-04-12T03:19:44.924 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.924 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b99d142388b8c857-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.927 DEBUG:boto:path=/dkyges-12/obj15 2024-04-12T03:19:44.927 DEBUG:boto:auth_path=/dkyges-12/obj15 2024-04-12T03:19:44.927 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.927 DEBUG:boto:Path: /dkyges-12/obj15 2024-04-12T03:19:44.927 DEBUG:boto:Data: 2024-04-12T03:19:44.927 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.927 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.927 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.927 DEBUG:boto:Params: {} 2024-04-12T03:19:44.927 DEBUG:boto:Token: None 2024-04-12T03:19:44.928 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj15 2024-04-12T03:19:44.928 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tz+wUWYrGaMtZkBuVCZ3ktd9KE8= 2024-04-12T03:19:44.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tz+wUWYrGaMtZkBuVCZ3ktd9KE8='} 2024-04-12T03:19:44.933 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022668c866b8dd750-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.934 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.934 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.934 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.934 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.934 DEBUG:boto:Data: 2024-04-12T03:19:44.934 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.934 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.934 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.934 DEBUG:boto:Params: {} 2024-04-12T03:19:44.935 DEBUG:boto:Token: None 2024-04-12T03:19:44.935 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.935 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.937 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c87af6cfdfdb2641-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.937 DEBUG:boto:path=/dkyges-12/obj16 2024-04-12T03:19:44.937 DEBUG:boto:auth_path=/dkyges-12/obj16 2024-04-12T03:19:44.937 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.937 DEBUG:boto:Path: /dkyges-12/obj16 2024-04-12T03:19:44.938 DEBUG:boto:Data: 2024-04-12T03:19:44.938 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.938 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.938 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.938 DEBUG:boto:Params: {} 2024-04-12T03:19:44.938 DEBUG:boto:Token: None 2024-04-12T03:19:44.938 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj16 2024-04-12T03:19:44.938 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SNlfw4b4yFKqO5Oj6qebvFxGGqw= 2024-04-12T03:19:44.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SNlfw4b4yFKqO5Oj6qebvFxGGqw='} 2024-04-12T03:19:44.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000323ff521c0edf8ab-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.983 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.983 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.983 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.983 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.983 DEBUG:boto:Data: 2024-04-12T03:19:44.983 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.983 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.983 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.983 DEBUG:boto:Params: {} 2024-04-12T03:19:44.984 DEBUG:boto:Token: None 2024-04-12T03:19:44.984 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.984 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fd6e4006acaee6c-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.987 DEBUG:boto:path=/dkyges-12/obj17 2024-04-12T03:19:44.987 DEBUG:boto:auth_path=/dkyges-12/obj17 2024-04-12T03:19:44.987 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.987 DEBUG:boto:Path: /dkyges-12/obj17 2024-04-12T03:19:44.987 DEBUG:boto:Data: 2024-04-12T03:19:44.987 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.987 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.987 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.987 DEBUG:boto:Params: {} 2024-04-12T03:19:44.987 DEBUG:boto:Token: None 2024-04-12T03:19:44.988 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj17 2024-04-12T03:19:44.988 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xsXRAXWkQjQnaYjuDeXvBJYEZ5Q= 2024-04-12T03:19:44.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xsXRAXWkQjQnaYjuDeXvBJYEZ5Q='} 2024-04-12T03:19:44.993 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000462883b2831bb7a8-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.994 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:44.994 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:44.994 DEBUG:boto:Method: HEAD 2024-04-12T03:19:44.994 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:44.994 DEBUG:boto:Data: 2024-04-12T03:19:44.994 DEBUG:boto:Headers: {} 2024-04-12T03:19:44.994 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.994 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.994 DEBUG:boto:Params: {} 2024-04-12T03:19:44.995 DEBUG:boto:Token: None 2024-04-12T03:19:44.995 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/ 2024-04-12T03:19:44.995 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T03:19:44.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T03:19:44.998 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c314fc2da11f863e-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:44.998 DEBUG:boto:path=/dkyges-12/obj18 2024-04-12T03:19:44.998 DEBUG:boto:auth_path=/dkyges-12/obj18 2024-04-12T03:19:44.998 DEBUG:boto:Method: PUT 2024-04-12T03:19:44.998 DEBUG:boto:Path: /dkyges-12/obj18 2024-04-12T03:19:44.998 DEBUG:boto:Data: 2024-04-12T03:19:44.998 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:44.998 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:44.998 DEBUG:boto:Port: 8001 2024-04-12T03:19:44.999 DEBUG:boto:Params: {} 2024-04-12T03:19:44.999 DEBUG:boto:Token: None 2024-04-12T03:19:44.999 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:44 GMT /dkyges-12/obj18 2024-04-12T03:19:44.999 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uzncsSU6PMmtVmUlengPSxOT2Ck= 2024-04-12T03:19:44.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uzncsSU6PMmtVmUlengPSxOT2Ck='} 2024-04-12T03:19:45.003 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c98697e620a5f9af-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.004 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.004 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.004 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.004 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.004 DEBUG:boto:Data: 2024-04-12T03:19:45.004 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.004 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.004 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.004 DEBUG:boto:Params: {} 2024-04-12T03:19:45.004 DEBUG:boto:Token: None 2024-04-12T03:19:45.005 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.005 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.007 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c267a2513a07b64f-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.007 DEBUG:boto:path=/dkyges-12/obj19 2024-04-12T03:19:45.007 DEBUG:boto:auth_path=/dkyges-12/obj19 2024-04-12T03:19:45.007 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.007 DEBUG:boto:Path: /dkyges-12/obj19 2024-04-12T03:19:45.007 DEBUG:boto:Data: 2024-04-12T03:19:45.007 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.007 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.008 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.008 DEBUG:boto:Params: {} 2024-04-12T03:19:45.008 DEBUG:boto:Token: None 2024-04-12T03:19:45.008 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj19 2024-04-12T03:19:45.008 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nvyB353nwTv+w3JVHf4Fca5/iSo= 2024-04-12T03:19:45.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nvyB353nwTv+w3JVHf4Fca5/iSo='} 2024-04-12T03:19:45.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5ef649277adc7d6-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.050 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.051 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.051 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.051 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.051 DEBUG:boto:Data: 2024-04-12T03:19:45.051 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.051 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.051 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.051 DEBUG:boto:Params: {} 2024-04-12T03:19:45.051 DEBUG:boto:Token: None 2024-04-12T03:19:45.051 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.052 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.054 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba893e8c26b0c6a4-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.054 DEBUG:boto:path=/dkyges-12/obj20 2024-04-12T03:19:45.054 DEBUG:boto:auth_path=/dkyges-12/obj20 2024-04-12T03:19:45.054 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.054 DEBUG:boto:Path: /dkyges-12/obj20 2024-04-12T03:19:45.054 DEBUG:boto:Data: 2024-04-12T03:19:45.054 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.055 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.055 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.055 DEBUG:boto:Params: {} 2024-04-12T03:19:45.055 DEBUG:boto:Token: None 2024-04-12T03:19:45.055 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj20 2024-04-12T03:19:45.055 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y9zQJCZujqg13DTvCeVlvAUUiUI= 2024-04-12T03:19:45.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y9zQJCZujqg13DTvCeVlvAUUiUI='} 2024-04-12T03:19:45.061 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c46c72f846392f1-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.061 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.062 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.062 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.062 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.062 DEBUG:boto:Data: 2024-04-12T03:19:45.062 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.062 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.062 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.062 DEBUG:boto:Params: {} 2024-04-12T03:19:45.062 DEBUG:boto:Token: None 2024-04-12T03:19:45.062 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.063 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.065 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6deff5a7924ef05-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.065 DEBUG:boto:path=/dkyges-12/obj21 2024-04-12T03:19:45.065 DEBUG:boto:auth_path=/dkyges-12/obj21 2024-04-12T03:19:45.065 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.066 DEBUG:boto:Path: /dkyges-12/obj21 2024-04-12T03:19:45.066 DEBUG:boto:Data: 2024-04-12T03:19:45.066 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.066 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.066 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.066 DEBUG:boto:Params: {} 2024-04-12T03:19:45.066 DEBUG:boto:Token: None 2024-04-12T03:19:45.066 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj21 2024-04-12T03:19:45.066 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:c5hKndp2fes7ygUdNPvc7rC8Ioo= 2024-04-12T03:19:45.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:c5hKndp2fes7ygUdNPvc7rC8Ioo='} 2024-04-12T03:19:45.072 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da969076bb76181a-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.072 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.072 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.072 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.072 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.073 DEBUG:boto:Data: 2024-04-12T03:19:45.073 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.073 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.073 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.073 DEBUG:boto:Params: {} 2024-04-12T03:19:45.073 DEBUG:boto:Token: None 2024-04-12T03:19:45.073 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.073 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.075 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000845e91f8a80aaea6-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.076 DEBUG:boto:path=/dkyges-12/obj22 2024-04-12T03:19:45.076 DEBUG:boto:auth_path=/dkyges-12/obj22 2024-04-12T03:19:45.076 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.076 DEBUG:boto:Path: /dkyges-12/obj22 2024-04-12T03:19:45.076 DEBUG:boto:Data: 2024-04-12T03:19:45.076 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.076 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.076 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.076 DEBUG:boto:Params: {} 2024-04-12T03:19:45.077 DEBUG:boto:Token: None 2024-04-12T03:19:45.077 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj22 2024-04-12T03:19:45.077 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9BSXkbaWjGT+2ydVfatrHb8YNEE= 2024-04-12T03:19:45.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9BSXkbaWjGT+2ydVfatrHb8YNEE='} 2024-04-12T03:19:45.118 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e18f00a960b751d-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.119 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.119 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.119 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.119 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.119 DEBUG:boto:Data: 2024-04-12T03:19:45.119 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.119 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.119 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.119 DEBUG:boto:Params: {} 2024-04-12T03:19:45.120 DEBUG:boto:Token: None 2024-04-12T03:19:45.120 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.120 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.122 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1daf2617354b84e-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.123 DEBUG:boto:path=/dkyges-12/obj23 2024-04-12T03:19:45.123 DEBUG:boto:auth_path=/dkyges-12/obj23 2024-04-12T03:19:45.123 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.123 DEBUG:boto:Path: /dkyges-12/obj23 2024-04-12T03:19:45.123 DEBUG:boto:Data: 2024-04-12T03:19:45.123 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.123 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.123 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.123 DEBUG:boto:Params: {} 2024-04-12T03:19:45.123 DEBUG:boto:Token: None 2024-04-12T03:19:45.124 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj23 2024-04-12T03:19:45.124 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i8rIss1OnXwdsIcFdSSmScFEvJ4= 2024-04-12T03:19:45.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i8rIss1OnXwdsIcFdSSmScFEvJ4='} 2024-04-12T03:19:45.128 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d54f5c174537b5b-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.128 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.128 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.128 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.128 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.128 DEBUG:boto:Data: 2024-04-12T03:19:45.128 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.129 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.129 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.129 DEBUG:boto:Params: {} 2024-04-12T03:19:45.129 DEBUG:boto:Token: None 2024-04-12T03:19:45.129 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.129 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.131 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4782f0fcad5e2fe-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.131 DEBUG:boto:path=/dkyges-12/obj24 2024-04-12T03:19:45.132 DEBUG:boto:auth_path=/dkyges-12/obj24 2024-04-12T03:19:45.132 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.132 DEBUG:boto:Path: /dkyges-12/obj24 2024-04-12T03:19:45.132 DEBUG:boto:Data: 2024-04-12T03:19:45.132 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.132 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.132 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.132 DEBUG:boto:Params: {} 2024-04-12T03:19:45.132 DEBUG:boto:Token: None 2024-04-12T03:19:45.132 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj24 2024-04-12T03:19:45.133 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:O20hFh9zkb+YVzxYSyfKh1FITOU= 2024-04-12T03:19:45.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:O20hFh9zkb+YVzxYSyfKh1FITOU='} 2024-04-12T03:19:45.178 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bab2753849949886-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.178 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.178 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.179 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.179 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.179 DEBUG:boto:Data: 2024-04-12T03:19:45.179 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.179 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.179 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.179 DEBUG:boto:Params: {} 2024-04-12T03:19:45.179 DEBUG:boto:Token: None 2024-04-12T03:19:45.179 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.180 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.182 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bad5f2177cd6b49-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.182 DEBUG:boto:path=/dkyges-12/obj25 2024-04-12T03:19:45.182 DEBUG:boto:auth_path=/dkyges-12/obj25 2024-04-12T03:19:45.183 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.183 DEBUG:boto:Path: /dkyges-12/obj25 2024-04-12T03:19:45.183 DEBUG:boto:Data: 2024-04-12T03:19:45.183 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.183 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.183 DEBUG:boto:Params: {} 2024-04-12T03:19:45.183 DEBUG:boto:Token: None 2024-04-12T03:19:45.183 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj25 2024-04-12T03:19:45.183 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ap37GXl6irP3oZ8/uj4QGBWJsaw= 2024-04-12T03:19:45.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ap37GXl6irP3oZ8/uj4QGBWJsaw='} 2024-04-12T03:19:45.188 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2162cf1d36c9e68-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.188 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.189 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.189 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.189 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.189 DEBUG:boto:Data: 2024-04-12T03:19:45.189 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.189 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.189 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.189 DEBUG:boto:Params: {} 2024-04-12T03:19:45.189 DEBUG:boto:Token: None 2024-04-12T03:19:45.189 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.190 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.192 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058f70e06a39904e0-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.192 DEBUG:boto:path=/dkyges-12/obj26 2024-04-12T03:19:45.192 DEBUG:boto:auth_path=/dkyges-12/obj26 2024-04-12T03:19:45.192 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.192 DEBUG:boto:Path: /dkyges-12/obj26 2024-04-12T03:19:45.192 DEBUG:boto:Data: 2024-04-12T03:19:45.192 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.192 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.192 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.193 DEBUG:boto:Params: {} 2024-04-12T03:19:45.193 DEBUG:boto:Token: None 2024-04-12T03:19:45.193 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj26 2024-04-12T03:19:45.193 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AgK9gY1AkqZ8bZBpcQcWGIErqJY= 2024-04-12T03:19:45.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AgK9gY1AkqZ8bZBpcQcWGIErqJY='} 2024-04-12T03:19:45.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b15d61180dc3995-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.239 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.239 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.239 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.239 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.239 DEBUG:boto:Data: 2024-04-12T03:19:45.239 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.239 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.239 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.239 DEBUG:boto:Params: {} 2024-04-12T03:19:45.240 DEBUG:boto:Token: None 2024-04-12T03:19:45.240 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.240 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.242 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001430c251c9887810-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.243 DEBUG:boto:path=/dkyges-12/obj27 2024-04-12T03:19:45.243 DEBUG:boto:auth_path=/dkyges-12/obj27 2024-04-12T03:19:45.243 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.243 DEBUG:boto:Path: /dkyges-12/obj27 2024-04-12T03:19:45.243 DEBUG:boto:Data: 2024-04-12T03:19:45.243 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.243 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.243 DEBUG:boto:Params: {} 2024-04-12T03:19:45.243 DEBUG:boto:Token: None 2024-04-12T03:19:45.244 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj27 2024-04-12T03:19:45.244 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ltN3GCI8OHX43+aZAUIbRvqnwQE= 2024-04-12T03:19:45.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ltN3GCI8OHX43+aZAUIbRvqnwQE='} 2024-04-12T03:19:45.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b85fecde6216c34-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.248 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.249 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.249 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.249 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.249 DEBUG:boto:Data: 2024-04-12T03:19:45.249 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.249 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.249 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.249 DEBUG:boto:Params: {} 2024-04-12T03:19:45.249 DEBUG:boto:Token: None 2024-04-12T03:19:45.249 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.249 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.252 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047832f644d2d2eda-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.252 DEBUG:boto:path=/dkyges-12/obj28 2024-04-12T03:19:45.252 DEBUG:boto:auth_path=/dkyges-12/obj28 2024-04-12T03:19:45.253 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.253 DEBUG:boto:Path: /dkyges-12/obj28 2024-04-12T03:19:45.253 DEBUG:boto:Data: 2024-04-12T03:19:45.253 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.254 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.254 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.254 DEBUG:boto:Params: {} 2024-04-12T03:19:45.255 DEBUG:boto:Token: None 2024-04-12T03:19:45.255 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj28 2024-04-12T03:19:45.255 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:owqDEIp4Gv7htdP8uv0DeGNZQ4s= 2024-04-12T03:19:45.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:owqDEIp4Gv7htdP8uv0DeGNZQ4s='} 2024-04-12T03:19:45.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008bb88dce53ec040d-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.299 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.299 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.300 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.300 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.300 DEBUG:boto:Data: 2024-04-12T03:19:45.300 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.300 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.300 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.300 DEBUG:boto:Params: {} 2024-04-12T03:19:45.301 DEBUG:boto:Token: None 2024-04-12T03:19:45.301 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.301 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.303 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b09757187d55447-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.304 DEBUG:boto:path=/dkyges-12/obj29 2024-04-12T03:19:45.304 DEBUG:boto:auth_path=/dkyges-12/obj29 2024-04-12T03:19:45.304 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.304 DEBUG:boto:Path: /dkyges-12/obj29 2024-04-12T03:19:45.304 DEBUG:boto:Data: 2024-04-12T03:19:45.304 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.304 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.304 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.304 DEBUG:boto:Params: {} 2024-04-12T03:19:45.305 DEBUG:boto:Token: None 2024-04-12T03:19:45.305 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj29 2024-04-12T03:19:45.305 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aXSlDvYwgMWShTLniheZXWJU3r0= 2024-04-12T03:19:45.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aXSlDvYwgMWShTLniheZXWJU3r0='} 2024-04-12T03:19:45.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085a5b7beeef02ed7-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.309 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.310 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.310 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.310 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.310 DEBUG:boto:Data: 2024-04-12T03:19:45.310 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.310 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.310 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.310 DEBUG:boto:Params: {} 2024-04-12T03:19:45.310 DEBUG:boto:Token: None 2024-04-12T03:19:45.310 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.311 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.313 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000343dbe6432472f6f-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.313 DEBUG:boto:path=/dkyges-12/obj30 2024-04-12T03:19:45.313 DEBUG:boto:auth_path=/dkyges-12/obj30 2024-04-12T03:19:45.313 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.313 DEBUG:boto:Path: /dkyges-12/obj30 2024-04-12T03:19:45.313 DEBUG:boto:Data: 2024-04-12T03:19:45.313 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.313 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.314 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.314 DEBUG:boto:Params: {} 2024-04-12T03:19:45.314 DEBUG:boto:Token: None 2024-04-12T03:19:45.314 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj30 2024-04-12T03:19:45.314 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OHd7dc54/OpdVlDxXZgg1L/1l6w= 2024-04-12T03:19:45.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OHd7dc54/OpdVlDxXZgg1L/1l6w='} 2024-04-12T03:19:45.363 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071519a8983acc4b9-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.363 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.363 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.363 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.363 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.363 DEBUG:boto:Data: 2024-04-12T03:19:45.364 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.364 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.364 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.364 DEBUG:boto:Params: {} 2024-04-12T03:19:45.364 DEBUG:boto:Token: None 2024-04-12T03:19:45.364 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.364 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.367 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005d61a8c3f79981a-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.367 DEBUG:boto:path=/dkyges-12/obj31 2024-04-12T03:19:45.367 DEBUG:boto:auth_path=/dkyges-12/obj31 2024-04-12T03:19:45.367 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.368 DEBUG:boto:Path: /dkyges-12/obj31 2024-04-12T03:19:45.368 DEBUG:boto:Data: 2024-04-12T03:19:45.368 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.368 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.368 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.368 DEBUG:boto:Params: {} 2024-04-12T03:19:45.368 DEBUG:boto:Token: None 2024-04-12T03:19:45.368 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj31 2024-04-12T03:19:45.369 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:p+T322FY82AvXYzG69wv8pWEssk= 2024-04-12T03:19:45.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p+T322FY82AvXYzG69wv8pWEssk='} 2024-04-12T03:19:45.373 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d9d98c774bd02aa-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.373 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.373 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.374 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.374 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.374 DEBUG:boto:Data: 2024-04-12T03:19:45.374 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.374 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.374 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.374 DEBUG:boto:Params: {} 2024-04-12T03:19:45.374 DEBUG:boto:Token: None 2024-04-12T03:19:45.375 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.375 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.377 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc8f1b603c642097-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.377 DEBUG:boto:path=/dkyges-12/obj32 2024-04-12T03:19:45.377 DEBUG:boto:auth_path=/dkyges-12/obj32 2024-04-12T03:19:45.377 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.378 DEBUG:boto:Path: /dkyges-12/obj32 2024-04-12T03:19:45.378 DEBUG:boto:Data: 2024-04-12T03:19:45.378 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.378 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.378 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.378 DEBUG:boto:Params: {} 2024-04-12T03:19:45.378 DEBUG:boto:Token: None 2024-04-12T03:19:45.378 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj32 2024-04-12T03:19:45.382 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cqfKYNRSOoW4Egop18iOBHAy4wg= 2024-04-12T03:19:45.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cqfKYNRSOoW4Egop18iOBHAy4wg='} 2024-04-12T03:19:45.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023d6ebb2d6bd2127-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.427 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.427 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.427 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.427 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.427 DEBUG:boto:Data: 2024-04-12T03:19:45.427 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.428 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.428 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.428 DEBUG:boto:Params: {} 2024-04-12T03:19:45.428 DEBUG:boto:Token: None 2024-04-12T03:19:45.429 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.429 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.431 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078d95d7829ab7289-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.431 DEBUG:boto:path=/dkyges-12/obj33 2024-04-12T03:19:45.431 DEBUG:boto:auth_path=/dkyges-12/obj33 2024-04-12T03:19:45.432 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.432 DEBUG:boto:Path: /dkyges-12/obj33 2024-04-12T03:19:45.432 DEBUG:boto:Data: 2024-04-12T03:19:45.432 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.432 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.432 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.432 DEBUG:boto:Params: {} 2024-04-12T03:19:45.433 DEBUG:boto:Token: None 2024-04-12T03:19:45.433 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj33 2024-04-12T03:19:45.433 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:exZaFqkuNjsJbbBJNhLIFohEI9g= 2024-04-12T03:19:45.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:exZaFqkuNjsJbbBJNhLIFohEI9g='} 2024-04-12T03:19:45.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094d4a6a3d6c07b9c-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.438 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.438 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.438 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.438 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.438 DEBUG:boto:Data: 2024-04-12T03:19:45.439 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.439 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.439 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.439 DEBUG:boto:Params: {} 2024-04-12T03:19:45.439 DEBUG:boto:Token: None 2024-04-12T03:19:45.439 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.439 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.441 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ace9e053a98444d-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.441 DEBUG:boto:path=/dkyges-12/obj34 2024-04-12T03:19:45.442 DEBUG:boto:auth_path=/dkyges-12/obj34 2024-04-12T03:19:45.442 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.442 DEBUG:boto:Path: /dkyges-12/obj34 2024-04-12T03:19:45.442 DEBUG:boto:Data: 2024-04-12T03:19:45.442 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.442 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.442 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.442 DEBUG:boto:Params: {} 2024-04-12T03:19:45.442 DEBUG:boto:Token: None 2024-04-12T03:19:45.442 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj34 2024-04-12T03:19:45.443 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nzinLDtf9DfVqesRnRPTeYRw7SQ= 2024-04-12T03:19:45.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nzinLDtf9DfVqesRnRPTeYRw7SQ='} 2024-04-12T03:19:45.486 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f7530560f9da504-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.487 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.487 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.487 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.487 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.487 DEBUG:boto:Data: 2024-04-12T03:19:45.487 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.487 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.488 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.488 DEBUG:boto:Params: {} 2024-04-12T03:19:45.488 DEBUG:boto:Token: None 2024-04-12T03:19:45.488 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.488 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.491 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b24cc1c8d15ca32e-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.491 DEBUG:boto:path=/dkyges-12/obj35 2024-04-12T03:19:45.491 DEBUG:boto:auth_path=/dkyges-12/obj35 2024-04-12T03:19:45.491 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.491 DEBUG:boto:Path: /dkyges-12/obj35 2024-04-12T03:19:45.491 DEBUG:boto:Data: 2024-04-12T03:19:45.492 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.492 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.492 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.492 DEBUG:boto:Params: {} 2024-04-12T03:19:45.492 DEBUG:boto:Token: None 2024-04-12T03:19:45.492 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj35 2024-04-12T03:19:45.492 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Is/FZorg5Ljp3ebk8dDMUL72Edc= 2024-04-12T03:19:45.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Is/FZorg5Ljp3ebk8dDMUL72Edc='} 2024-04-12T03:19:45.497 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b64089e235398b1b-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.497 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.497 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.498 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.498 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.498 DEBUG:boto:Data: 2024-04-12T03:19:45.498 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.498 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.498 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.498 DEBUG:boto:Params: {} 2024-04-12T03:19:45.498 DEBUG:boto:Token: None 2024-04-12T03:19:45.498 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.499 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.501 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1a29ffd70e46fb5-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.501 DEBUG:boto:path=/dkyges-12/obj36 2024-04-12T03:19:45.502 DEBUG:boto:auth_path=/dkyges-12/obj36 2024-04-12T03:19:45.502 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.502 DEBUG:boto:Path: /dkyges-12/obj36 2024-04-12T03:19:45.502 DEBUG:boto:Data: 2024-04-12T03:19:45.502 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.502 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.502 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.502 DEBUG:boto:Params: {} 2024-04-12T03:19:45.502 DEBUG:boto:Token: None 2024-04-12T03:19:45.503 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj36 2024-04-12T03:19:45.503 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pTFA+vF1Ux3LJeKx91c/OT8qaQ8= 2024-04-12T03:19:45.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pTFA+vF1Ux3LJeKx91c/OT8qaQ8='} 2024-04-12T03:19:45.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bce603ed4e827a1a-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.547 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.547 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.547 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.547 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.547 DEBUG:boto:Data: 2024-04-12T03:19:45.547 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.547 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.548 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.548 DEBUG:boto:Params: {} 2024-04-12T03:19:45.548 DEBUG:boto:Token: None 2024-04-12T03:19:45.548 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.548 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.550 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058f7feb79f092b0b-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.551 DEBUG:boto:path=/dkyges-12/obj37 2024-04-12T03:19:45.551 DEBUG:boto:auth_path=/dkyges-12/obj37 2024-04-12T03:19:45.551 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.552 DEBUG:boto:Path: /dkyges-12/obj37 2024-04-12T03:19:45.552 DEBUG:boto:Data: 2024-04-12T03:19:45.552 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.552 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.552 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.552 DEBUG:boto:Params: {} 2024-04-12T03:19:45.552 DEBUG:boto:Token: None 2024-04-12T03:19:45.552 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj37 2024-04-12T03:19:45.553 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9aSzN3R3R4uXNL8YM7aMk2Fq/D0= 2024-04-12T03:19:45.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9aSzN3R3R4uXNL8YM7aMk2Fq/D0='} 2024-04-12T03:19:45.557 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df166aa675117cf2-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.557 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.557 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.557 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.557 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.558 DEBUG:boto:Data: 2024-04-12T03:19:45.558 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.558 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.558 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.558 DEBUG:boto:Params: {} 2024-04-12T03:19:45.558 DEBUG:boto:Token: None 2024-04-12T03:19:45.558 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.558 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.561 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e67ddfdb89f3c682-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.561 DEBUG:boto:path=/dkyges-12/obj38 2024-04-12T03:19:45.561 DEBUG:boto:auth_path=/dkyges-12/obj38 2024-04-12T03:19:45.561 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.561 DEBUG:boto:Path: /dkyges-12/obj38 2024-04-12T03:19:45.561 DEBUG:boto:Data: 2024-04-12T03:19:45.562 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.562 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.562 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.562 DEBUG:boto:Params: {} 2024-04-12T03:19:45.562 DEBUG:boto:Token: None 2024-04-12T03:19:45.562 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj38 2024-04-12T03:19:45.562 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1oo2dpv3oIfZ2B11/cpe8FtEk/0= 2024-04-12T03:19:45.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1oo2dpv3oIfZ2B11/cpe8FtEk/0='} 2024-04-12T03:19:45.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000360effeecd4e02d5-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.607 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.607 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.607 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.607 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.607 DEBUG:boto:Data: 2024-04-12T03:19:45.608 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.608 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.608 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.608 DEBUG:boto:Params: {} 2024-04-12T03:19:45.608 DEBUG:boto:Token: None 2024-04-12T03:19:45.608 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.609 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.611 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd4e6dc6fadbf39a-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.612 DEBUG:boto:path=/dkyges-12/obj39 2024-04-12T03:19:45.612 DEBUG:boto:auth_path=/dkyges-12/obj39 2024-04-12T03:19:45.612 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.612 DEBUG:boto:Path: /dkyges-12/obj39 2024-04-12T03:19:45.612 DEBUG:boto:Data: 2024-04-12T03:19:45.612 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.612 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.613 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.613 DEBUG:boto:Params: {} 2024-04-12T03:19:45.613 DEBUG:boto:Token: None 2024-04-12T03:19:45.613 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj39 2024-04-12T03:19:45.613 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Sf2hawxvEl7wZV00DFaLMrV2ecs= 2024-04-12T03:19:45.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Sf2hawxvEl7wZV00DFaLMrV2ecs='} 2024-04-12T03:19:45.617 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000024945f0cff87c993-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.618 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.618 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.618 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.618 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.618 DEBUG:boto:Data: 2024-04-12T03:19:45.618 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.618 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.618 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.618 DEBUG:boto:Params: {} 2024-04-12T03:19:45.619 DEBUG:boto:Token: None 2024-04-12T03:19:45.619 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.619 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.621 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cef590edfc291392-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.621 DEBUG:boto:path=/dkyges-12/obj40 2024-04-12T03:19:45.621 DEBUG:boto:auth_path=/dkyges-12/obj40 2024-04-12T03:19:45.621 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.621 DEBUG:boto:Path: /dkyges-12/obj40 2024-04-12T03:19:45.621 DEBUG:boto:Data: 2024-04-12T03:19:45.622 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.622 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.622 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.622 DEBUG:boto:Params: {} 2024-04-12T03:19:45.622 DEBUG:boto:Token: None 2024-04-12T03:19:45.622 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj40 2024-04-12T03:19:45.622 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:50nY986B8eWq42wyHAztEc6TjEU= 2024-04-12T03:19:45.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:50nY986B8eWq42wyHAztEc6TjEU='} 2024-04-12T03:19:45.666 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029c8ad4244ab893b-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.667 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.667 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.667 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.667 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.667 DEBUG:boto:Data: 2024-04-12T03:19:45.667 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.667 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.667 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.668 DEBUG:boto:Params: {} 2024-04-12T03:19:45.668 DEBUG:boto:Token: None 2024-04-12T03:19:45.668 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.668 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.670 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c8c18b311247453-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.670 DEBUG:boto:path=/dkyges-12/obj41 2024-04-12T03:19:45.671 DEBUG:boto:auth_path=/dkyges-12/obj41 2024-04-12T03:19:45.671 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.671 DEBUG:boto:Path: /dkyges-12/obj41 2024-04-12T03:19:45.671 DEBUG:boto:Data: 2024-04-12T03:19:45.671 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.671 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.671 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.671 DEBUG:boto:Params: {} 2024-04-12T03:19:45.671 DEBUG:boto:Token: None 2024-04-12T03:19:45.671 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj41 2024-04-12T03:19:45.672 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vbwJ8fz9vwHCVMo952IYVtHS4vs= 2024-04-12T03:19:45.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vbwJ8fz9vwHCVMo952IYVtHS4vs='} 2024-04-12T03:19:45.677 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca5df6d6947c7d91-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.677 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.677 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.677 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.677 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.677 DEBUG:boto:Data: 2024-04-12T03:19:45.677 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.677 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.677 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.678 DEBUG:boto:Params: {} 2024-04-12T03:19:45.678 DEBUG:boto:Token: None 2024-04-12T03:19:45.678 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.678 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.680 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e083510618935e9-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.680 DEBUG:boto:path=/dkyges-12/obj42 2024-04-12T03:19:45.680 DEBUG:boto:auth_path=/dkyges-12/obj42 2024-04-12T03:19:45.680 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.681 DEBUG:boto:Path: /dkyges-12/obj42 2024-04-12T03:19:45.681 DEBUG:boto:Data: 2024-04-12T03:19:45.681 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.681 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.681 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.681 DEBUG:boto:Params: {} 2024-04-12T03:19:45.681 DEBUG:boto:Token: None 2024-04-12T03:19:45.681 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj42 2024-04-12T03:19:45.681 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nrrWfhYFr+zPkceLsIYn7y2EpMc= 2024-04-12T03:19:45.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nrrWfhYFr+zPkceLsIYn7y2EpMc='} 2024-04-12T03:19:45.726 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c0ef858f56981eb9-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.727 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.727 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.727 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.727 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.727 DEBUG:boto:Data: 2024-04-12T03:19:45.727 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.727 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.727 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.728 DEBUG:boto:Params: {} 2024-04-12T03:19:45.728 DEBUG:boto:Token: None 2024-04-12T03:19:45.728 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.728 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.730 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007408ba6201e7c597-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.730 DEBUG:boto:path=/dkyges-12/obj43 2024-04-12T03:19:45.731 DEBUG:boto:auth_path=/dkyges-12/obj43 2024-04-12T03:19:45.731 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.731 DEBUG:boto:Path: /dkyges-12/obj43 2024-04-12T03:19:45.731 DEBUG:boto:Data: 2024-04-12T03:19:45.731 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.731 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.731 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.731 DEBUG:boto:Params: {} 2024-04-12T03:19:45.731 DEBUG:boto:Token: None 2024-04-12T03:19:45.731 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj43 2024-04-12T03:19:45.732 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sy9tvttK0/k0mansIzzUj7AvYbE= 2024-04-12T03:19:45.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sy9tvttK0/k0mansIzzUj7AvYbE='} 2024-04-12T03:19:45.736 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000049ccb211144e0eb-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.737 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.737 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.737 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.737 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.737 DEBUG:boto:Data: 2024-04-12T03:19:45.737 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.737 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.737 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.737 DEBUG:boto:Params: {} 2024-04-12T03:19:45.737 DEBUG:boto:Token: None 2024-04-12T03:19:45.738 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.738 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.740 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2c1b7294cbcf9a3-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.740 DEBUG:boto:path=/dkyges-12/obj44 2024-04-12T03:19:45.740 DEBUG:boto:auth_path=/dkyges-12/obj44 2024-04-12T03:19:45.740 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.741 DEBUG:boto:Path: /dkyges-12/obj44 2024-04-12T03:19:45.741 DEBUG:boto:Data: 2024-04-12T03:19:45.741 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.741 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.741 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.741 DEBUG:boto:Params: {} 2024-04-12T03:19:45.741 DEBUG:boto:Token: None 2024-04-12T03:19:45.741 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj44 2024-04-12T03:19:45.741 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JIxp7a9T4lMF86xr9LTqnaISSbI= 2024-04-12T03:19:45.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JIxp7a9T4lMF86xr9LTqnaISSbI='} 2024-04-12T03:19:45.791 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000467cc0e936c84d3f-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.791 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.791 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.791 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.791 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.791 DEBUG:boto:Data: 2024-04-12T03:19:45.791 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.791 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.791 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.792 DEBUG:boto:Params: {} 2024-04-12T03:19:45.794 DEBUG:boto:Token: None 2024-04-12T03:19:45.794 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.794 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.797 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f070d5cd3e097d05-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.797 DEBUG:boto:path=/dkyges-12/obj45 2024-04-12T03:19:45.797 DEBUG:boto:auth_path=/dkyges-12/obj45 2024-04-12T03:19:45.798 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.798 DEBUG:boto:Path: /dkyges-12/obj45 2024-04-12T03:19:45.798 DEBUG:boto:Data: 2024-04-12T03:19:45.798 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.798 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.798 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.798 DEBUG:boto:Params: {} 2024-04-12T03:19:45.798 DEBUG:boto:Token: None 2024-04-12T03:19:45.798 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj45 2024-04-12T03:19:45.799 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ucJWZFAXb4fhtRRfzbKvdmlgSTU= 2024-04-12T03:19:45.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ucJWZFAXb4fhtRRfzbKvdmlgSTU='} 2024-04-12T03:19:45.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab46265467e70ecc-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.803 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.804 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.804 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.804 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.804 DEBUG:boto:Data: 2024-04-12T03:19:45.804 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.804 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.804 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.804 DEBUG:boto:Params: {} 2024-04-12T03:19:45.804 DEBUG:boto:Token: None 2024-04-12T03:19:45.804 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.805 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000241f540bf91117d1-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.807 DEBUG:boto:path=/dkyges-12/obj46 2024-04-12T03:19:45.807 DEBUG:boto:auth_path=/dkyges-12/obj46 2024-04-12T03:19:45.807 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.807 DEBUG:boto:Path: /dkyges-12/obj46 2024-04-12T03:19:45.807 DEBUG:boto:Data: 2024-04-12T03:19:45.807 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.808 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.808 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.808 DEBUG:boto:Params: {} 2024-04-12T03:19:45.808 DEBUG:boto:Token: None 2024-04-12T03:19:45.808 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj46 2024-04-12T03:19:45.808 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fv38QD/jF7ZcuFs3zAO5WyoMpOs= 2024-04-12T03:19:45.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fv38QD/jF7ZcuFs3zAO5WyoMpOs='} 2024-04-12T03:19:45.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d762c43accfa5779-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.851 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.851 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.851 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.851 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.851 DEBUG:boto:Data: 2024-04-12T03:19:45.851 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.851 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.851 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.852 DEBUG:boto:Params: {} 2024-04-12T03:19:45.852 DEBUG:boto:Token: None 2024-04-12T03:19:45.852 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.852 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ff32f460d7edb6e-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.855 DEBUG:boto:path=/dkyges-12/obj47 2024-04-12T03:19:45.855 DEBUG:boto:auth_path=/dkyges-12/obj47 2024-04-12T03:19:45.855 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.855 DEBUG:boto:Path: /dkyges-12/obj47 2024-04-12T03:19:45.855 DEBUG:boto:Data: 2024-04-12T03:19:45.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.855 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.855 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.855 DEBUG:boto:Params: {} 2024-04-12T03:19:45.855 DEBUG:boto:Token: None 2024-04-12T03:19:45.856 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj47 2024-04-12T03:19:45.856 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z6AH6QjWWAzOTkbFHcAc+PlNsUY= 2024-04-12T03:19:45.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z6AH6QjWWAzOTkbFHcAc+PlNsUY='} 2024-04-12T03:19:45.860 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042034defc5d76009-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.860 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.860 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.860 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.860 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.860 DEBUG:boto:Data: 2024-04-12T03:19:45.861 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.861 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.861 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.861 DEBUG:boto:Params: {} 2024-04-12T03:19:45.861 DEBUG:boto:Token: None 2024-04-12T03:19:45.861 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.861 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.863 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd9264e3f5f549ab-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.864 DEBUG:boto:path=/dkyges-12/obj48 2024-04-12T03:19:45.864 DEBUG:boto:auth_path=/dkyges-12/obj48 2024-04-12T03:19:45.864 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.864 DEBUG:boto:Path: /dkyges-12/obj48 2024-04-12T03:19:45.864 DEBUG:boto:Data: 2024-04-12T03:19:45.864 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.864 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.865 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.865 DEBUG:boto:Params: {} 2024-04-12T03:19:45.865 DEBUG:boto:Token: None 2024-04-12T03:19:45.865 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj48 2024-04-12T03:19:45.865 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BuyfECLpa96bhAWIf0/KqCE6tN8= 2024-04-12T03:19:45.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BuyfECLpa96bhAWIf0/KqCE6tN8='} 2024-04-12T03:19:45.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000213dab6a2a8b7008-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.911 DEBUG:boto:path=/dkyges-12/ 2024-04-12T03:19:45.911 DEBUG:boto:auth_path=/dkyges-12/ 2024-04-12T03:19:45.911 DEBUG:boto:Method: HEAD 2024-04-12T03:19:45.911 DEBUG:boto:Path: /dkyges-12/ 2024-04-12T03:19:45.911 DEBUG:boto:Data: 2024-04-12T03:19:45.911 DEBUG:boto:Headers: {} 2024-04-12T03:19:45.911 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.911 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.911 DEBUG:boto:Params: {} 2024-04-12T03:19:45.912 DEBUG:boto:Token: None 2024-04-12T03:19:45.912 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/ 2024-04-12T03:19:45.912 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T03:19:45.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T03:19:45.914 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a73cb569e864675f-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.915 DEBUG:boto:path=/dkyges-12/obj49 2024-04-12T03:19:45.915 DEBUG:boto:auth_path=/dkyges-12/obj49 2024-04-12T03:19:45.915 DEBUG:boto:Method: PUT 2024-04-12T03:19:45.915 DEBUG:boto:Path: /dkyges-12/obj49 2024-04-12T03:19:45.915 DEBUG:boto:Data: 2024-04-12T03:19:45.915 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:19:45.915 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:19:45.915 DEBUG:boto:Port: 8001 2024-04-12T03:19:45.915 DEBUG:boto:Params: {} 2024-04-12T03:19:45.916 DEBUG:boto:Token: None 2024-04-12T03:19:45.916 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:19:45 GMT /dkyges-12/obj49 2024-04-12T03:19:45.916 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ST+8b+8ZknZL/7ij++g87iDUV9k= 2024-04-12T03:19:45.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ST+8b+8ZknZL/7ij++g87iDUV9k='} 2024-04-12T03:19:45.920 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c6469e73f3828566-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:19:45.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:45.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:45.921 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:19:45.980 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:45.980 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:46.015 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:19:46.028 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:19:46.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:19:46.029 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:19:46.029 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:19:46.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:19:46.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:19:46.029 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:19:46.029 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:19:46.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:19:46.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:19:46.029 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:19:46.029 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:19:46.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:19:46.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:19:46.030 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891982.967859_56.1", 2024-04-12T03:19:46.030 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:19:42.967859Z" 2024-04-12T03:19:46.030 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:19:46.030 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:19:46.031 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712891982.967859_56.1'} 2024-04-12T03:19:46.031 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:19:46.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:46.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:19:46.032 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:19:46.086 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:46.087 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:46.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:46.118+0000 7f050ea7e780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:46.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:46.119+0000 7f050ea7e780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:19:46.133 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:19:46.133 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:19:46.133 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:19:46.133 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:19:46.133 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:19:46.134 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:19:46.134 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:19:46.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:19:46.134 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:19:46.134 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:19:46.134 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:19:46.134 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:19:46.134 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:19:46.134 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:19:46.134 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:19:46.134 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:19:46.135 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:19:46.135 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:19:46.135 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:19:46.135 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:19:46.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:19:46.135 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:19:46.135 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:19:46.135 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:19:46.135 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:19:46.135 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:19:46.135 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:19:46.136 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:19:46.136 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:19:46.136 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:19:46.136 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:19:46.136 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:19:46.136 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:19:46.136 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:19:46.136 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:19:46.136 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:19:46.136 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:19:46.136 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:19:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:19:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:19:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:19:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:19:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:19:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:19:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:19:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:19:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:19:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:19:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:19:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891982.967859_56.1", 2024-04-12T03:19:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:19:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:19:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:19:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:19:42.967859Z", 2024-04-12T03:19:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:19:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:19:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:19:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:19:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:19:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:19:46.139 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:19:46.139 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:19:46.139 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:19:46.139 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:19:46Z" 2024-04-12T03:19:46.139 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:19:46.140 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712891982.967859_56.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:19:42.967859Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:19:46Z" } 2024-04-12T03:19:46.140 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891982.967859_56.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:19:42.967859Z', 'realm_epoch': 2}}] 2024-04-12T03:19:46.140 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712891982.967859_56.1'} 2024-04-12T03:19:46.140 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712891982.967859_56.1'} 2024-04-12T03:19:46.140 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:19:46.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:19:46.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:19:46.141 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:19:46.206 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:19:46.207 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:19:46.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:46.241+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:19:51.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:51.242+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:19:56.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:19:56.242+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:20:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:20:01.243+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:20:06.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:20:06.244+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:20:11.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:20:11.245+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:20:16.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:20:16.246+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:20:21.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:20:21.245+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:20:26.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:20:26.246+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:20:31.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:20:31.247+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:20:36.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:20:36.247+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:20:41.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:20:41.248+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:20:46.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:20:46.249+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:20:51.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:20:51.250+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:20:56.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:20:56.250+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:21:01.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:21:01.251+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:21:06.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:21:06.251+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:21:11.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:21:11.252+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:21:16.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:21:16.253+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:21:21.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:21:21.254+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:21:26.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:21:26.254+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:21:31.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:21:31.255+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:21:36.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:21:36.255+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:21:41.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:21:41.256+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:21:46.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:21:46.257+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:21:51.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:21:51.257+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:21:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:21:56.258+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:22:01.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:22:01.259+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:22:06.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:22:06.259+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:22:11.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:22:11.260+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:22:16.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:22:16.260+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:22:21.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:22:21.261+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:22:26.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:22:26.262+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:22:31.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:22:31.262+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:22:36.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:22:36.263+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:22:41.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:22:41.263+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:22:46.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:22:46.265+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:22:51.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:22:51.265+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:22:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:22:56.266+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:23:01.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:23:01.266+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:23:06.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:23:06.267+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:23:11.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:23:11.268+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:23:16.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:23:16.269+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:23:21.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:23:21.269+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:23:26.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:23:26.270+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:23:31.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:23:31.271+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:23:36.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:23:36.271+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:23:41.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:23:41.272+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:23:46.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:23:46.273+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:23:51.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:23:51.273+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:23:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:23:56.274+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:24:01.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:01.276+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:24:06.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:06.275+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:24:11.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:11.276+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:24:16.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:16.277+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:24:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:21.277+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:24:26.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:26.278+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:24:31.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:31.279+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:24:36.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:36.279+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T03:24:41.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:41.280+0000 7f1275b0f780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-12T03:24:41.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:41.280+0000 7f1275b0f780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T03:24:41.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:41.280+0000 7f1275b0f780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T03:24:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-12T03:24:41.303 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_object_delete ... FAIL 2024-04-12T03:24:41.304 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-13 2024-04-12T03:24:41.304 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:24:41.304 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:24:41.304 DEBUG:boto:Method: PUT 2024-04-12T03:24:41.304 DEBUG:boto:Path: /dkyges-13/ 2024-04-12T03:24:41.305 DEBUG:boto:Data: 2024-04-12T03:24:41.305 DEBUG:boto:Headers: {} 2024-04-12T03:24:41.305 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:24:41.305 DEBUG:boto:Port: 8000 2024-04-12T03:24:41.305 DEBUG:boto:Params: {} 2024-04-12T03:24:41.305 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:24:41.306 DEBUG:boto:Token: None 2024-04-12T03:24:41.306 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:24:41 GMT /dkyges-13/ 2024-04-12T03:24:41.306 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rfamg/qS1cDvsdrZdirJndiwJW8= 2024-04-12T03:24:41.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:24:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rfamg/qS1cDvsdrZdirJndiwJW8=', 'Content-Length': '0'} 2024-04-12T03:24:41.351 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c6c7ea71c2c535b1-006618a979-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:24:41.352 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-14 2024-04-12T03:24:41.352 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:24:41.352 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:24:41.352 DEBUG:boto:Method: PUT 2024-04-12T03:24:41.352 DEBUG:boto:Path: /dkyges-14/ 2024-04-12T03:24:41.352 DEBUG:boto:Data: 2024-04-12T03:24:41.352 DEBUG:boto:Headers: {} 2024-04-12T03:24:41.352 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:24:41.352 DEBUG:boto:Port: 8001 2024-04-12T03:24:41.352 DEBUG:boto:Params: {} 2024-04-12T03:24:41.353 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:24:41.353 DEBUG:boto:Token: None 2024-04-12T03:24:41.353 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:24:41 GMT /dkyges-14/ 2024-04-12T03:24:41.353 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:huYnZrARiLou+GxQeyvzP3acDwk= 2024-04-12T03:24:41.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:24:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:huYnZrARiLou+GxQeyvzP3acDwk=', 'Content-Length': '0'} 2024-04-12T03:24:41.407 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000080df62b41016f3c3-006618a979-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:24:41.407 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:24:41.407 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:24:41.407 DEBUG:boto:path=/dkyges-13/?versioning 2024-04-12T03:24:41.408 DEBUG:boto:auth_path=/dkyges-13/?versioning 2024-04-12T03:24:41.408 DEBUG:boto:Method: PUT 2024-04-12T03:24:41.408 DEBUG:boto:Path: /dkyges-13/?versioning 2024-04-12T03:24:41.408 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:24:41.408 DEBUG:boto:Headers: {} 2024-04-12T03:24:41.408 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:24:41.408 DEBUG:boto:Port: 8000 2024-04-12T03:24:41.408 DEBUG:boto:Params: {} 2024-04-12T03:24:41.408 DEBUG:boto:Token: None 2024-04-12T03:24:41.408 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:24:41 GMT /dkyges-13/?versioning 2024-04-12T03:24:41.409 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AyKe8FdXpWB708Ecag3ygjk71AA= 2024-04-12T03:24:41.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:24:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AyKe8FdXpWB708Ecag3ygjk71AA=', 'Content-Length': '228'} 2024-04-12T03:24:41.414 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000db5670a822a63bae-006618a979-110f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:24:41.414 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:24:41.414 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:24:41.414 DEBUG:boto:path=/dkyges-14/?versioning 2024-04-12T03:24:41.415 DEBUG:boto:auth_path=/dkyges-14/?versioning 2024-04-12T03:24:41.415 DEBUG:boto:Method: PUT 2024-04-12T03:24:41.415 DEBUG:boto:Path: /dkyges-14/?versioning 2024-04-12T03:24:41.415 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:24:41.415 DEBUG:boto:Headers: {} 2024-04-12T03:24:41.415 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:24:41.415 DEBUG:boto:Port: 8001 2024-04-12T03:24:41.415 DEBUG:boto:Params: {} 2024-04-12T03:24:41.415 DEBUG:boto:Token: None 2024-04-12T03:24:41.415 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:24:41 GMT /dkyges-14/?versioning 2024-04-12T03:24:41.415 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AIkZMzTFW9OMMDja0xnDVyn1IMM= 2024-04-12T03:24:41.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:24:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AIkZMzTFW9OMMDja0xnDVyn1IMM=', 'Content-Length': '228'} 2024-04-12T03:24:41.424 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001f6ce540cecae9ac-006618a979-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:24:41.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:24:41.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:24:41.425 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:24:41.482 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:24:41.482 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:24:41.517 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:24:41.538 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:24:41.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:24:41.539 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:24:41.539 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:24:41.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:24:41.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:24:41.539 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:24:41.539 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:24:41.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:24:41.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:24:41.539 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:24:41.540 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:24:41.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:24:41.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:24:41.540 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892281.421615_116.1", 2024-04-12T03:24:41.540 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:24:41.421615Z" 2024-04-12T03:24:41.540 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:24:41.540 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:24:41.541 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892281.421615_116.1'} 2024-04-12T03:24:41.541 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:24:41.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:24:41.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:24:41.542 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:24:41.599 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:24:41.599 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:24:41.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:41.629+0000 7f7d08070780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:24:41.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:41.630+0000 7f7d08070780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:24:41.644 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:24:41.644 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:24:41.645 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:24:41.645 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:24:41.645 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:24:41.645 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:24:41.645 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:24:41.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:24:41.645 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:24:41.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:24:41.645 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:24:41.645 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:24:41.645 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:24:41.646 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:24:41.646 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:24:41.646 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:24:41.646 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:24:41.646 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:24:41.646 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:24:41.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:24:41.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:24:41.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:24:41.646 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:24:41.647 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:24:41.647 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:24:41.647 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:24:41.647 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:24:41.647 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:24:41.647 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:24:41.647 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:24:41.647 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:24:41.647 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:24:41.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:24:41.648 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:24:41.648 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:24:41.648 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:24:41.648 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:24:41.648 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:24:41.648 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:24:41.648 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:24:41.648 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:24:41.648 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:24:41.648 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:24:41.648 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:24:41.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:24:41.649 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:24:41.649 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:24:41.649 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:24:41.649 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:24:41.649 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892281.421615_116.1", 2024-04-12T03:24:41.649 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:24:41.649 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:24:41.649 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:24:41.649 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:24:41.421615Z", 2024-04-12T03:24:41.649 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:24:41.650 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:24:41.650 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:24:41.650 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:24:41.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:24:41.650 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:24:41.650 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:24:41.650 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:24:41.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:24:41.650 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:24:41Z" 2024-04-12T03:24:41.650 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:24:41.652 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892281.421615_116.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:24:41.421615Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:24:41Z" } 2024-04-12T03:24:41.652 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892281.421615_116.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:24:41.421615Z', 'realm_epoch': 2}}] 2024-04-12T03:24:41.652 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892281.421615_116.1'} 2024-04-12T03:24:41.652 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892281.421615_116.1'} 2024-04-12T03:24:41.652 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:24:41.653 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:24:41.653 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:24:41.653 DEBUG:boto:Method: HEAD 2024-04-12T03:24:41.653 DEBUG:boto:Path: /dkyges-13/ 2024-04-12T03:24:41.653 DEBUG:boto:Data: 2024-04-12T03:24:41.653 DEBUG:boto:Headers: {} 2024-04-12T03:24:41.653 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:24:41.653 DEBUG:boto:Port: 8000 2024-04-12T03:24:41.653 DEBUG:boto:Params: {} 2024-04-12T03:24:41.653 DEBUG:boto:Token: None 2024-04-12T03:24:41.654 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:24:41 GMT /dkyges-13/ 2024-04-12T03:24:41.654 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cbWlqFC2aHcmDdeeCKQcCZB0/y8= 2024-04-12T03:24:41.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:24:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cbWlqFC2aHcmDdeeCKQcCZB0/y8=', 'Content-Length': '0'} 2024-04-12T03:24:41.656 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ed581c30b1e9c3b-006618a979-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:24:41.656 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:24:41.656 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:24:41.657 DEBUG:boto:Method: PUT 2024-04-12T03:24:41.657 DEBUG:boto:Path: /dkyges-13/dummy 2024-04-12T03:24:41.657 DEBUG:boto:Data: 2024-04-12T03:24:41.657 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:24:41.657 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:24:41.657 DEBUG:boto:Port: 8000 2024-04-12T03:24:41.657 DEBUG:boto:Params: {} 2024-04-12T03:24:41.657 DEBUG:boto:Token: None 2024-04-12T03:24:41.658 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:24:41 GMT /dkyges-13/dummy 2024-04-12T03:24:41.658 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nVmPCzdfH1rT4IMKfhNpOP9QMzE= 2024-04-12T03:24:41.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:24:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nVmPCzdfH1rT4IMKfhNpOP9QMzE='} 2024-04-12T03:24:41.669 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-amz-request-id', 'tx000002865f01a5755623f-006618a979-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:24:41.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:24:41.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:24:41.670 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:24:41.733 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:24:41.733 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:24:41.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:41.770+0000 7f1ce16ef780 1 waiting to reach incremental sync.. 2024-04-12T03:24:46.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:46.772+0000 7f1ce16ef780 1 bucket sync caught up with source: 2024-04-12T03:24:46.773 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.10.11, , , , , ] 2024-04-12T03:24:46.773 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.10.11, , , , , ] 2024-04-12T03:24:46.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:46.772+0000 7f1ce16ef780 0 bucket checkpoint complete 2024-04-12T03:24:46.791 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:24:46.791 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:24:46.791 DEBUG:boto:Method: HEAD 2024-04-12T03:24:46.791 DEBUG:boto:Path: /dkyges-14/ 2024-04-12T03:24:46.791 DEBUG:boto:Data: 2024-04-12T03:24:46.791 DEBUG:boto:Headers: {} 2024-04-12T03:24:46.791 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:24:46.791 DEBUG:boto:Port: 8001 2024-04-12T03:24:46.792 DEBUG:boto:Params: {} 2024-04-12T03:24:46.792 DEBUG:boto:Token: None 2024-04-12T03:24:46.792 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:24:46 GMT /dkyges-14/ 2024-04-12T03:24:46.792 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BEOevk/U5e/aQkveJbtsjtL1tUo= 2024-04-12T03:24:46.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:24:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BEOevk/U5e/aQkveJbtsjtL1tUo=', 'Content-Length': '0'} 2024-04-12T03:24:46.795 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c49d35b947f28ca3-006618a97e-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:24:46.795 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:24:46.795 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:24:46.795 DEBUG:boto:Method: PUT 2024-04-12T03:24:46.796 DEBUG:boto:Path: /dkyges-14/dummy 2024-04-12T03:24:46.796 DEBUG:boto:Data: 2024-04-12T03:24:46.796 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:24:46.796 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:24:46.796 DEBUG:boto:Port: 8001 2024-04-12T03:24:46.796 DEBUG:boto:Params: {} 2024-04-12T03:24:46.796 DEBUG:boto:Token: None 2024-04-12T03:24:46.796 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:24:46 GMT /dkyges-14/dummy 2024-04-12T03:24:46.797 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y0CG+NPN6ACivRFpPnPHuXM+NUU= 2024-04-12T03:24:46.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:24:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y0CG+NPN6ACivRFpPnPHuXM+NUU='} 2024-04-12T03:24:46.809 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-amz-request-id', 'tx0000083e75e7486393e04-006618a97e-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:24:46.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:24:46.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:24:46.810 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:24:46.876 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:24:46.876 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:24:46.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:46.914+0000 7fc5846f7780 1 waiting to reach incremental sync.. 2024-04-12T03:24:51.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:51.915+0000 7fc5846f7780 1 waiting to reach incremental sync.. 2024-04-12T03:24:56.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:24:56.916+0000 7fc5846f7780 1 waiting to reach incremental sync.. 2024-04-12T03:25:01.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:01.916+0000 7fc5846f7780 1 waiting to reach incremental sync.. 2024-04-12T03:25:06.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:06.918+0000 7fc5846f7780 1 bucket sync caught up with source: 2024-04-12T03:25:06.919 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.29.11, , , , , ] 2024-04-12T03:25:06.919 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.29.11, , , , , ] 2024-04-12T03:25:06.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:06.918+0000 7fc5846f7780 0 bucket checkpoint complete 2024-04-12T03:25:06.936 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:06.937 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:06.937 DEBUG:boto:Method: HEAD 2024-04-12T03:25:06.937 DEBUG:boto:Path: /dkyges-13/ 2024-04-12T03:25:06.937 DEBUG:boto:Data: 2024-04-12T03:25:06.937 DEBUG:boto:Headers: {} 2024-04-12T03:25:06.937 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:06.937 DEBUG:boto:Port: 8000 2024-04-12T03:25:06.937 DEBUG:boto:Params: {} 2024-04-12T03:25:06.938 DEBUG:boto:Token: None 2024-04-12T03:25:06.938 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:06 GMT /dkyges-13/ 2024-04-12T03:25:06.938 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:chWXaq43WVP9Mc3+ZQydv1adIfY= 2024-04-12T03:25:06.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:chWXaq43WVP9Mc3+ZQydv1adIfY=', 'Content-Length': '0'} 2024-04-12T03:25:06.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fd6f076512eda1f-006618a992-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:06.941 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:25:06.941 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:25:06.941 DEBUG:boto:Method: PUT 2024-04-12T03:25:06.941 DEBUG:boto:Path: /dkyges-13/obj-a1 2024-04-12T03:25:06.941 DEBUG:boto:Data: 2024-04-12T03:25:06.941 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:25:06.941 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:06.941 DEBUG:boto:Port: 8000 2024-04-12T03:25:06.941 DEBUG:boto:Params: {} 2024-04-12T03:25:06.942 DEBUG:boto:Token: None 2024-04-12T03:25:06.942 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:25:06 GMT /dkyges-13/obj-a1 2024-04-12T03:25:06.942 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Z4BGxa5NEnFI7GBMDtX3JTrLjbc= 2024-04-12T03:25:06.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:25:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Z4BGxa5NEnFI7GBMDtX3JTrLjbc='} 2024-04-12T03:25:06.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-amz-request-id', 'tx00000828dbedc45a6110c-006618a992-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:06.954 DEBUG:rgw_multi.tests:version1 id=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:06.954 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:25:06.954 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:25:06.955 DEBUG:boto:Method: PUT 2024-04-12T03:25:06.955 DEBUG:boto:Path: /dkyges-13/obj-a1 2024-04-12T03:25:06.955 DEBUG:boto:Data: 2024-04-12T03:25:06.955 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:25:06.955 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:06.955 DEBUG:boto:Port: 8000 2024-04-12T03:25:06.955 DEBUG:boto:Params: {} 2024-04-12T03:25:06.955 DEBUG:boto:Token: None 2024-04-12T03:25:06.955 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 12 Apr 2024 03:25:06 GMT /dkyges-13/obj-a1 2024-04-12T03:25:06.955 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SKipnVr/L4Q7ZURiufofFTl0FjU= 2024-04-12T03:25:06.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:25:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SKipnVr/L4Q7ZURiufofFTl0FjU='} 2024-04-12T03:25:06.966 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'RMAhKrPBqjf3y37Gh3DS7AzujaizJGp'), ('x-amz-request-id', 'tx00000f2b655cbf69ffc10-006618a992-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:06.966 DEBUG:rgw_multi.tests:version2 id=RMAhKrPBqjf3y37Gh3DS7AzujaizJGp 2024-04-12T03:25:06.966 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:25:06.966 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:25:06.966 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=RMAhKrPBqjf3y37Gh3DS7AzujaizJGp 2024-04-12T03:25:06.966 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=RMAhKrPBqjf3y37Gh3DS7AzujaizJGp 2024-04-12T03:25:06.967 DEBUG:boto:Method: DELETE 2024-04-12T03:25:06.967 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=RMAhKrPBqjf3y37Gh3DS7AzujaizJGp 2024-04-12T03:25:06.967 DEBUG:boto:Data: 2024-04-12T03:25:06.967 DEBUG:boto:Headers: {} 2024-04-12T03:25:06.967 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:06.967 DEBUG:boto:Port: 8000 2024-04-12T03:25:06.967 DEBUG:boto:Params: {} 2024-04-12T03:25:06.967 DEBUG:boto:Token: None 2024-04-12T03:25:06.967 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:25:06 GMT /dkyges-13/obj-a1?versionId=RMAhKrPBqjf3y37Gh3DS7AzujaizJGp 2024-04-12T03:25:06.967 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wcbqSE9xLOWgqS4ctc0mqL11+w8= 2024-04-12T03:25:06.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wcbqSE9xLOWgqS4ctc0mqL11+w8=', 'Content-Length': '0'} 2024-04-12T03:25:06.978 DEBUG:boto:Response headers: [('x-amz-version-id', 'RMAhKrPBqjf3y37Gh3DS7AzujaizJGp'), ('x-amz-request-id', 'tx000006609f94919741b02-006618a992-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:06.979 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:25:06.979 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:25:06.979 DEBUG:boto:Method: PUT 2024-04-12T03:25:06.979 DEBUG:boto:Path: /dkyges-13/obj-a1 2024-04-12T03:25:06.979 DEBUG:boto:Data: 2024-04-12T03:25:06.979 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:25:06.979 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:06.979 DEBUG:boto:Port: 8000 2024-04-12T03:25:06.979 DEBUG:boto:Params: {} 2024-04-12T03:25:06.980 DEBUG:boto:Token: None 2024-04-12T03:25:06.980 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 12 Apr 2024 03:25:06 GMT /dkyges-13/obj-a1 2024-04-12T03:25:06.980 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n7Nvwwoh4RQuv3Bo+4cQPqX2B4o= 2024-04-12T03:25:06.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:25:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n7Nvwwoh4RQuv3Bo+4cQPqX2B4o='} 2024-04-12T03:25:07.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '0HULt92-o6fgtks3FUCRgh0P2OuSFR4'), ('x-amz-request-id', 'tx00000fd02fad7293c9e74-006618a992-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.027 DEBUG:rgw_multi.tests:version3 id=0HULt92-o6fgtks3FUCRgh0P2OuSFR4 2024-04-12T03:25:07.027 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:25:07.027 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:25:07.027 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=0HULt92-o6fgtks3FUCRgh0P2OuSFR4 2024-04-12T03:25:07.027 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=0HULt92-o6fgtks3FUCRgh0P2OuSFR4 2024-04-12T03:25:07.027 DEBUG:boto:Method: DELETE 2024-04-12T03:25:07.027 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=0HULt92-o6fgtks3FUCRgh0P2OuSFR4 2024-04-12T03:25:07.028 DEBUG:boto:Data: 2024-04-12T03:25:07.028 DEBUG:boto:Headers: {} 2024-04-12T03:25:07.028 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:07.028 DEBUG:boto:Port: 8000 2024-04-12T03:25:07.028 DEBUG:boto:Params: {} 2024-04-12T03:25:07.028 DEBUG:boto:Token: None 2024-04-12T03:25:07.028 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:25:07 GMT /dkyges-13/obj-a1?versionId=0HULt92-o6fgtks3FUCRgh0P2OuSFR4 2024-04-12T03:25:07.029 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ts78QtR3+7qFWIZ6PCFMzwRS7Qs= 2024-04-12T03:25:07.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ts78QtR3+7qFWIZ6PCFMzwRS7Qs=', 'Content-Length': '0'} 2024-04-12T03:25:07.040 DEBUG:boto:Response headers: [('x-amz-version-id', '0HULt92-o6fgtks3FUCRgh0P2OuSFR4'), ('x-amz-request-id', 'tx00000d0a15c5460e7546a-006618a993-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.040 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:07.040 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:07.040 DEBUG:boto:Method: HEAD 2024-04-12T03:25:07.040 DEBUG:boto:Path: /dkyges-13/ 2024-04-12T03:25:07.040 DEBUG:boto:Data: 2024-04-12T03:25:07.041 DEBUG:boto:Headers: {} 2024-04-12T03:25:07.041 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:07.041 DEBUG:boto:Port: 8001 2024-04-12T03:25:07.041 DEBUG:boto:Params: {} 2024-04-12T03:25:07.041 DEBUG:boto:Token: None 2024-04-12T03:25:07.041 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:07 GMT /dkyges-13/ 2024-04-12T03:25:07.041 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EuNtWCEAa12THo1oVx3nfSAtnjU= 2024-04-12T03:25:07.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EuNtWCEAa12THo1oVx3nfSAtnjU=', 'Content-Length': '0'} 2024-04-12T03:25:07.044 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b016b39a047a7a53-006618a993-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.044 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:25:07.044 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:25:07.044 DEBUG:boto:Method: PUT 2024-04-12T03:25:07.045 DEBUG:boto:Path: /dkyges-13/obj-a2 2024-04-12T03:25:07.045 DEBUG:boto:Data: 2024-04-12T03:25:07.045 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:25:07.045 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:07.045 DEBUG:boto:Port: 8001 2024-04-12T03:25:07.045 DEBUG:boto:Params: {} 2024-04-12T03:25:07.045 DEBUG:boto:Token: None 2024-04-12T03:25:07.045 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:25:07 GMT /dkyges-13/obj-a2 2024-04-12T03:25:07.046 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AJ852wLSTR0UFlD8dnCM9KVDwB8= 2024-04-12T03:25:07.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AJ852wLSTR0UFlD8dnCM9KVDwB8='} 2024-04-12T03:25:07.091 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-amz-request-id', 'tx00000a2a43e5e0f66d530-006618a993-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.091 DEBUG:rgw_multi.tests:version1 id=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:07.091 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:25:07.091 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:25:07.091 DEBUG:boto:Method: PUT 2024-04-12T03:25:07.091 DEBUG:boto:Path: /dkyges-13/obj-a2 2024-04-12T03:25:07.092 DEBUG:boto:Data: 2024-04-12T03:25:07.092 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:25:07.092 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:07.092 DEBUG:boto:Port: 8001 2024-04-12T03:25:07.092 DEBUG:boto:Params: {} 2024-04-12T03:25:07.092 DEBUG:boto:Token: None 2024-04-12T03:25:07.092 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 12 Apr 2024 03:25:07 GMT /dkyges-13/obj-a2 2024-04-12T03:25:07.092 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:w1UvqH/BamO5eoPv1Kqul9O8rkM= 2024-04-12T03:25:07.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:w1UvqH/BamO5eoPv1Kqul9O8rkM='} 2024-04-12T03:25:07.104 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '1A4ym0fL04b4lYE1hY7NPPPaablbNqx'), ('x-amz-request-id', 'tx00000d18c74537223c211-006618a993-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.104 DEBUG:rgw_multi.tests:version2 id=1A4ym0fL04b4lYE1hY7NPPPaablbNqx 2024-04-12T03:25:07.104 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:25:07.104 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:25:07.105 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=1A4ym0fL04b4lYE1hY7NPPPaablbNqx 2024-04-12T03:25:07.105 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=1A4ym0fL04b4lYE1hY7NPPPaablbNqx 2024-04-12T03:25:07.105 DEBUG:boto:Method: DELETE 2024-04-12T03:25:07.105 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=1A4ym0fL04b4lYE1hY7NPPPaablbNqx 2024-04-12T03:25:07.105 DEBUG:boto:Data: 2024-04-12T03:25:07.105 DEBUG:boto:Headers: {} 2024-04-12T03:25:07.105 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:07.105 DEBUG:boto:Port: 8001 2024-04-12T03:25:07.105 DEBUG:boto:Params: {} 2024-04-12T03:25:07.105 DEBUG:boto:Token: None 2024-04-12T03:25:07.105 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:25:07 GMT /dkyges-13/obj-a2?versionId=1A4ym0fL04b4lYE1hY7NPPPaablbNqx 2024-04-12T03:25:07.106 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:s5K400+ZGlhlPXbvA9ojbjSmO4s= 2024-04-12T03:25:07.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:s5K400+ZGlhlPXbvA9ojbjSmO4s=', 'Content-Length': '0'} 2024-04-12T03:25:07.117 DEBUG:boto:Response headers: [('x-amz-version-id', '1A4ym0fL04b4lYE1hY7NPPPaablbNqx'), ('x-amz-request-id', 'tx00000bbc0aea6df14bf0c-006618a993-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.117 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:25:07.117 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:25:07.117 DEBUG:boto:Method: PUT 2024-04-12T03:25:07.117 DEBUG:boto:Path: /dkyges-13/obj-a2 2024-04-12T03:25:07.118 DEBUG:boto:Data: 2024-04-12T03:25:07.118 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:25:07.118 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:07.118 DEBUG:boto:Port: 8001 2024-04-12T03:25:07.118 DEBUG:boto:Params: {} 2024-04-12T03:25:07.118 DEBUG:boto:Token: None 2024-04-12T03:25:07.118 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 12 Apr 2024 03:25:07 GMT /dkyges-13/obj-a2 2024-04-12T03:25:07.118 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:w0YNB+wAvvQg87CksEUAwPq/IHg= 2024-04-12T03:25:07.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:w0YNB+wAvvQg87CksEUAwPq/IHg='} 2024-04-12T03:25:07.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WItAnt9YVuV-bJr8.twjqNcc9SISwhh'), ('x-amz-request-id', 'tx000004cb476a984e45ec5-006618a993-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.167 DEBUG:rgw_multi.tests:version3 id=WItAnt9YVuV-bJr8.twjqNcc9SISwhh 2024-04-12T03:25:07.167 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:25:07.167 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:25:07.167 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=WItAnt9YVuV-bJr8.twjqNcc9SISwhh 2024-04-12T03:25:07.167 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=WItAnt9YVuV-bJr8.twjqNcc9SISwhh 2024-04-12T03:25:07.167 DEBUG:boto:Method: DELETE 2024-04-12T03:25:07.167 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=WItAnt9YVuV-bJr8.twjqNcc9SISwhh 2024-04-12T03:25:07.167 DEBUG:boto:Data: 2024-04-12T03:25:07.167 DEBUG:boto:Headers: {} 2024-04-12T03:25:07.168 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:07.168 DEBUG:boto:Port: 8001 2024-04-12T03:25:07.168 DEBUG:boto:Params: {} 2024-04-12T03:25:07.168 DEBUG:boto:Token: None 2024-04-12T03:25:07.168 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:25:07 GMT /dkyges-13/obj-a2?versionId=WItAnt9YVuV-bJr8.twjqNcc9SISwhh 2024-04-12T03:25:07.168 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P/dAeZoX9aKbh3kRp9vmZNtt8G4= 2024-04-12T03:25:07.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P/dAeZoX9aKbh3kRp9vmZNtt8G4=', 'Content-Length': '0'} 2024-04-12T03:25:07.180 DEBUG:boto:Response headers: [('x-amz-version-id', 'WItAnt9YVuV-bJr8.twjqNcc9SISwhh'), ('x-amz-request-id', 'tx00000cff9d894f8103fe3-006618a993-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.180 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:07.180 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:07.180 DEBUG:boto:Method: HEAD 2024-04-12T03:25:07.180 DEBUG:boto:Path: /dkyges-14/ 2024-04-12T03:25:07.180 DEBUG:boto:Data: 2024-04-12T03:25:07.180 DEBUG:boto:Headers: {} 2024-04-12T03:25:07.180 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:07.180 DEBUG:boto:Port: 8000 2024-04-12T03:25:07.181 DEBUG:boto:Params: {} 2024-04-12T03:25:07.181 DEBUG:boto:Token: None 2024-04-12T03:25:07.181 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:07 GMT /dkyges-14/ 2024-04-12T03:25:07.181 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Fu8sHkhAK7yeDzUbrCnPs0opk5E= 2024-04-12T03:25:07.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Fu8sHkhAK7yeDzUbrCnPs0opk5E=', 'Content-Length': '0'} 2024-04-12T03:25:07.184 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000023e618e91f4e5f-006618a993-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.184 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:25:07.184 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:25:07.184 DEBUG:boto:Method: PUT 2024-04-12T03:25:07.184 DEBUG:boto:Path: /dkyges-14/obj-a1 2024-04-12T03:25:07.184 DEBUG:boto:Data: 2024-04-12T03:25:07.185 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:25:07.185 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:07.185 DEBUG:boto:Port: 8000 2024-04-12T03:25:07.185 DEBUG:boto:Params: {} 2024-04-12T03:25:07.185 DEBUG:boto:Token: None 2024-04-12T03:25:07.185 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:25:07 GMT /dkyges-14/obj-a1 2024-04-12T03:25:07.185 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PUhkWi/XH2wk1t26QMPeXYmyEgA= 2024-04-12T03:25:07.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PUhkWi/XH2wk1t26QMPeXYmyEgA='} 2024-04-12T03:25:07.197 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-amz-request-id', 'tx0000031e3c645fa4dd8b4-006618a993-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.197 DEBUG:rgw_multi.tests:version1 id=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:07.197 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:25:07.198 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:25:07.198 DEBUG:boto:Method: PUT 2024-04-12T03:25:07.198 DEBUG:boto:Path: /dkyges-14/obj-a1 2024-04-12T03:25:07.198 DEBUG:boto:Data: 2024-04-12T03:25:07.198 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:25:07.198 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:07.198 DEBUG:boto:Port: 8000 2024-04-12T03:25:07.198 DEBUG:boto:Params: {} 2024-04-12T03:25:07.198 DEBUG:boto:Token: None 2024-04-12T03:25:07.198 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 12 Apr 2024 03:25:07 GMT /dkyges-14/obj-a1 2024-04-12T03:25:07.199 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S2gDSIAsXGiDs+Uj6LFmHw0S0pU= 2024-04-12T03:25:07.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S2gDSIAsXGiDs+Uj6LFmHw0S0pU='} 2024-04-12T03:25:07.246 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '5v0YCxIsYISPkpXopGLRgrmPE91z.Kz'), ('x-amz-request-id', 'tx000003010f45811f05e09-006618a993-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.247 DEBUG:rgw_multi.tests:version2 id=5v0YCxIsYISPkpXopGLRgrmPE91z.Kz 2024-04-12T03:25:07.247 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:25:07.247 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:25:07.247 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=5v0YCxIsYISPkpXopGLRgrmPE91z.Kz 2024-04-12T03:25:07.247 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=5v0YCxIsYISPkpXopGLRgrmPE91z.Kz 2024-04-12T03:25:07.248 DEBUG:boto:Method: DELETE 2024-04-12T03:25:07.248 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=5v0YCxIsYISPkpXopGLRgrmPE91z.Kz 2024-04-12T03:25:07.248 DEBUG:boto:Data: 2024-04-12T03:25:07.248 DEBUG:boto:Headers: {} 2024-04-12T03:25:07.248 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:07.248 DEBUG:boto:Port: 8000 2024-04-12T03:25:07.248 DEBUG:boto:Params: {} 2024-04-12T03:25:07.248 DEBUG:boto:Token: None 2024-04-12T03:25:07.249 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:25:07 GMT /dkyges-14/obj-a1?versionId=5v0YCxIsYISPkpXopGLRgrmPE91z.Kz 2024-04-12T03:25:07.249 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YOP2hz9dtWB8JMOVLNPQzBqoewA= 2024-04-12T03:25:07.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YOP2hz9dtWB8JMOVLNPQzBqoewA=', 'Content-Length': '0'} 2024-04-12T03:25:07.260 DEBUG:boto:Response headers: [('x-amz-version-id', '5v0YCxIsYISPkpXopGLRgrmPE91z.Kz'), ('x-amz-request-id', 'tx00000914b1d37efac7ac3-006618a993-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.261 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:25:07.261 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:25:07.261 DEBUG:boto:Method: PUT 2024-04-12T03:25:07.261 DEBUG:boto:Path: /dkyges-14/obj-a1 2024-04-12T03:25:07.261 DEBUG:boto:Data: 2024-04-12T03:25:07.261 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:25:07.261 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:07.261 DEBUG:boto:Port: 8000 2024-04-12T03:25:07.261 DEBUG:boto:Params: {} 2024-04-12T03:25:07.262 DEBUG:boto:Token: None 2024-04-12T03:25:07.262 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 12 Apr 2024 03:25:07 GMT /dkyges-14/obj-a1 2024-04-12T03:25:07.262 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CC7ptH19xkDPIdq14l6x1HB6HQ0= 2024-04-12T03:25:07.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CC7ptH19xkDPIdq14l6x1HB6HQ0='} 2024-04-12T03:25:07.274 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'XXM6SWxYYsZL76XVEQWXscFasXxmTGj'), ('x-amz-request-id', 'tx00000e4f236d820181f69-006618a993-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.274 DEBUG:rgw_multi.tests:version3 id=XXM6SWxYYsZL76XVEQWXscFasXxmTGj 2024-04-12T03:25:07.274 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:25:07.274 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:25:07.274 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=XXM6SWxYYsZL76XVEQWXscFasXxmTGj 2024-04-12T03:25:07.274 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=XXM6SWxYYsZL76XVEQWXscFasXxmTGj 2024-04-12T03:25:07.275 DEBUG:boto:Method: DELETE 2024-04-12T03:25:07.275 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=XXM6SWxYYsZL76XVEQWXscFasXxmTGj 2024-04-12T03:25:07.275 DEBUG:boto:Data: 2024-04-12T03:25:07.275 DEBUG:boto:Headers: {} 2024-04-12T03:25:07.275 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:07.275 DEBUG:boto:Port: 8000 2024-04-12T03:25:07.275 DEBUG:boto:Params: {} 2024-04-12T03:25:07.275 DEBUG:boto:Token: None 2024-04-12T03:25:07.275 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:25:07 GMT /dkyges-14/obj-a1?versionId=XXM6SWxYYsZL76XVEQWXscFasXxmTGj 2024-04-12T03:25:07.276 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bti/LWfhocgUNWBUpEshcALoNGY= 2024-04-12T03:25:07.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bti/LWfhocgUNWBUpEshcALoNGY=', 'Content-Length': '0'} 2024-04-12T03:25:07.287 DEBUG:boto:Response headers: [('x-amz-version-id', 'XXM6SWxYYsZL76XVEQWXscFasXxmTGj'), ('x-amz-request-id', 'tx000001bae142a14a3f1d4-006618a993-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.287 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:07.287 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:07.287 DEBUG:boto:Method: HEAD 2024-04-12T03:25:07.287 DEBUG:boto:Path: /dkyges-14/ 2024-04-12T03:25:07.287 DEBUG:boto:Data: 2024-04-12T03:25:07.288 DEBUG:boto:Headers: {} 2024-04-12T03:25:07.288 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:07.288 DEBUG:boto:Port: 8001 2024-04-12T03:25:07.288 DEBUG:boto:Params: {} 2024-04-12T03:25:07.288 DEBUG:boto:Token: None 2024-04-12T03:25:07.288 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:07 GMT /dkyges-14/ 2024-04-12T03:25:07.288 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Fu8sHkhAK7yeDzUbrCnPs0opk5E= 2024-04-12T03:25:07.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Fu8sHkhAK7yeDzUbrCnPs0opk5E=', 'Content-Length': '0'} 2024-04-12T03:25:07.291 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d627356cf7afa4a9-006618a993-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.291 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:25:07.291 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:25:07.291 DEBUG:boto:Method: PUT 2024-04-12T03:25:07.292 DEBUG:boto:Path: /dkyges-14/obj-a2 2024-04-12T03:25:07.292 DEBUG:boto:Data: 2024-04-12T03:25:07.292 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:25:07.292 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:07.292 DEBUG:boto:Port: 8001 2024-04-12T03:25:07.292 DEBUG:boto:Params: {} 2024-04-12T03:25:07.292 DEBUG:boto:Token: None 2024-04-12T03:25:07.292 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:25:07 GMT /dkyges-14/obj-a2 2024-04-12T03:25:07.293 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qvmPOP0mzYWHFaBjzS522sWwHXg= 2024-04-12T03:25:07.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qvmPOP0mzYWHFaBjzS522sWwHXg='} 2024-04-12T03:25:07.305 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-amz-request-id', 'tx00000e7529f7d2582d08b-006618a993-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.305 DEBUG:rgw_multi.tests:version1 id=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:07.305 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:25:07.306 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:25:07.306 DEBUG:boto:Method: PUT 2024-04-12T03:25:07.306 DEBUG:boto:Path: /dkyges-14/obj-a2 2024-04-12T03:25:07.306 DEBUG:boto:Data: 2024-04-12T03:25:07.306 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:25:07.306 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:07.306 DEBUG:boto:Port: 8001 2024-04-12T03:25:07.306 DEBUG:boto:Params: {} 2024-04-12T03:25:07.306 DEBUG:boto:Token: None 2024-04-12T03:25:07.307 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 12 Apr 2024 03:25:07 GMT /dkyges-14/obj-a2 2024-04-12T03:25:07.307 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:g8IH3G5Zca33es977kBvlMYYiZ4= 2024-04-12T03:25:07.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:g8IH3G5Zca33es977kBvlMYYiZ4='} 2024-04-12T03:25:07.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'fV.WLvP0RlYSzTnD7ZcTJHgd0CHm2J8'), ('x-amz-request-id', 'tx00000d87cd9e6f79ba052-006618a993-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.351 DEBUG:rgw_multi.tests:version2 id=fV.WLvP0RlYSzTnD7ZcTJHgd0CHm2J8 2024-04-12T03:25:07.351 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:25:07.351 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:25:07.351 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=fV.WLvP0RlYSzTnD7ZcTJHgd0CHm2J8 2024-04-12T03:25:07.351 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=fV.WLvP0RlYSzTnD7ZcTJHgd0CHm2J8 2024-04-12T03:25:07.351 DEBUG:boto:Method: DELETE 2024-04-12T03:25:07.351 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=fV.WLvP0RlYSzTnD7ZcTJHgd0CHm2J8 2024-04-12T03:25:07.351 DEBUG:boto:Data: 2024-04-12T03:25:07.351 DEBUG:boto:Headers: {} 2024-04-12T03:25:07.352 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:07.352 DEBUG:boto:Port: 8001 2024-04-12T03:25:07.352 DEBUG:boto:Params: {} 2024-04-12T03:25:07.352 DEBUG:boto:Token: None 2024-04-12T03:25:07.352 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:25:07 GMT /dkyges-14/obj-a2?versionId=fV.WLvP0RlYSzTnD7ZcTJHgd0CHm2J8 2024-04-12T03:25:07.352 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U5jELye3lydm7byK6L6c/6BqaEo= 2024-04-12T03:25:07.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U5jELye3lydm7byK6L6c/6BqaEo=', 'Content-Length': '0'} 2024-04-12T03:25:07.363 DEBUG:boto:Response headers: [('x-amz-version-id', 'fV.WLvP0RlYSzTnD7ZcTJHgd0CHm2J8'), ('x-amz-request-id', 'tx00000e44346d468b4b18d-006618a993-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.364 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:25:07.364 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:25:07.364 DEBUG:boto:Method: PUT 2024-04-12T03:25:07.364 DEBUG:boto:Path: /dkyges-14/obj-a2 2024-04-12T03:25:07.364 DEBUG:boto:Data: 2024-04-12T03:25:07.364 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:25:07.365 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:07.365 DEBUG:boto:Port: 8001 2024-04-12T03:25:07.365 DEBUG:boto:Params: {} 2024-04-12T03:25:07.365 DEBUG:boto:Token: None 2024-04-12T03:25:07.365 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 12 Apr 2024 03:25:07 GMT /dkyges-14/obj-a2 2024-04-12T03:25:07.365 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DPWT9jX03Cre6zvG3sk+/QN8GNI= 2024-04-12T03:25:07.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DPWT9jX03Cre6zvG3sk+/QN8GNI='} 2024-04-12T03:25:07.376 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'd0qA82rDC7CBXJac7hi25dagCLHdRa9'), ('x-amz-request-id', 'tx0000090ae85da20f39ef6-006618a993-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.376 DEBUG:rgw_multi.tests:version3 id=d0qA82rDC7CBXJac7hi25dagCLHdRa9 2024-04-12T03:25:07.377 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:25:07.377 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:25:07.377 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=d0qA82rDC7CBXJac7hi25dagCLHdRa9 2024-04-12T03:25:07.377 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=d0qA82rDC7CBXJac7hi25dagCLHdRa9 2024-04-12T03:25:07.377 DEBUG:boto:Method: DELETE 2024-04-12T03:25:07.377 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=d0qA82rDC7CBXJac7hi25dagCLHdRa9 2024-04-12T03:25:07.377 DEBUG:boto:Data: 2024-04-12T03:25:07.377 DEBUG:boto:Headers: {} 2024-04-12T03:25:07.377 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:07.377 DEBUG:boto:Port: 8001 2024-04-12T03:25:07.377 DEBUG:boto:Params: {} 2024-04-12T03:25:07.378 DEBUG:boto:Token: None 2024-04-12T03:25:07.378 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:25:07 GMT /dkyges-14/obj-a2?versionId=d0qA82rDC7CBXJac7hi25dagCLHdRa9 2024-04-12T03:25:07.378 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dTmYLmaR48En1EsZna5AwHUnZa0= 2024-04-12T03:25:07.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dTmYLmaR48En1EsZna5AwHUnZa0=', 'Content-Length': '0'} 2024-04-12T03:25:07.389 DEBUG:boto:Response headers: [('x-amz-version-id', 'd0qA82rDC7CBXJac7hi25dagCLHdRa9'), ('x-amz-request-id', 'tx000005218e03075eafa53-006618a993-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:07.389 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-13 2024-04-12T03:25:07.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:25:07.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:25:07.390 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:25:07.456 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:07.457 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:07.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:07.496+0000 7fed8ad92780 1 waiting for incremental sync to catch up: 2024-04-12T03:25:07.498 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.12.10, , , , , ] 2024-04-12T03:25:07.498 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.12.10, , , , 00000000010.22.6, ] 2024-04-12T03:25:12.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:12.498+0000 7fed8ad92780 1 waiting for incremental sync to catch up: 2024-04-12T03:25:12.499 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.12.10, , , , , ] 2024-04-12T03:25:12.499 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.12.10, , , , 00000000010.22.6, ] 2024-04-12T03:25:17.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:17.500+0000 7fed8ad92780 1 waiting for incremental sync to catch up: 2024-04-12T03:25:17.500 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.12.10, , , , , ] 2024-04-12T03:25:17.500 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.12.10, , , , 00000000010.22.6, ] 2024-04-12T03:25:22.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:22.500+0000 7fed8ad92780 1 waiting for incremental sync to catch up: 2024-04-12T03:25:22.502 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.12.10, , , , , ] 2024-04-12T03:25:22.502 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.12.10, , , , 00000000010.22.6, ] 2024-04-12T03:25:27.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:27.501+0000 7fed8ad92780 1 bucket sync caught up with source: 2024-04-12T03:25:27.503 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.12.10, , , , 00000000010.22.6, ] 2024-04-12T03:25:27.503 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.12.10, , , , 00000000010.22.6, ] 2024-04-12T03:25:27.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:27.501+0000 7fed8ad92780 0 bucket checkpoint complete 2024-04-12T03:25:27.520 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-13 2024-04-12T03:25:27.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:25:27.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:25:27.520 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:25:27.584 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:27.585 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:27.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:27.623+0000 7f7b9be76780 1 waiting for incremental sync to catch up: 2024-04-12T03:25:27.624 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.12.11, , , 00000000010.34.6, , ] 2024-04-12T03:25:27.624 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000010.34.6, 00000000002.14.11, ] 2024-04-12T03:25:32.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:32.624+0000 7f7b9be76780 1 waiting for incremental sync to catch up: 2024-04-12T03:25:32.626 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.12.11, , , 00000000010.34.6, , ] 2024-04-12T03:25:32.626 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000010.34.6, 00000000002.14.11, ] 2024-04-12T03:25:37.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:37.625+0000 7f7b9be76780 1 waiting for incremental sync to catch up: 2024-04-12T03:25:37.627 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.12.11, , , 00000000010.34.6, , ] 2024-04-12T03:25:37.627 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000010.34.6, 00000000002.14.11, ] 2024-04-12T03:25:42.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:42.627+0000 7f7b9be76780 1 waiting for incremental sync to catch up: 2024-04-12T03:25:42.628 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.12.11, , , 00000000010.34.6, , ] 2024-04-12T03:25:42.628 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000010.34.6, 00000000002.14.11, ] 2024-04-12T03:25:47.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:47.628+0000 7f7b9be76780 1 bucket sync caught up with source: 2024-04-12T03:25:47.630 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.12.11, , , 00000000010.34.6, 00000000002.14.11, ] 2024-04-12T03:25:47.630 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000010.34.6, 00000000002.14.11, ] 2024-04-12T03:25:47.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:47.628+0000 7f7b9be76780 0 bucket checkpoint complete 2024-04-12T03:25:47.651 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-13 zones={a2, a1} 2024-04-12T03:25:47.652 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:47.652 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:47.653 DEBUG:boto:Method: HEAD 2024-04-12T03:25:47.653 DEBUG:boto:Path: /dkyges-13/ 2024-04-12T03:25:47.653 DEBUG:boto:Data: 2024-04-12T03:25:47.653 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.653 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:47.653 DEBUG:boto:Port: 8001 2024-04-12T03:25:47.654 DEBUG:boto:Params: {} 2024-04-12T03:25:47.654 DEBUG:boto:Token: None 2024-04-12T03:25:47.655 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/ 2024-04-12T03:25:47.655 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QDbZbWhNxKwWsm7eAfLXgFJ0Ggo= 2024-04-12T03:25:47.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QDbZbWhNxKwWsm7eAfLXgFJ0Ggo=', 'Content-Length': '0'} 2024-04-12T03:25:47.658 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a57f3c13f28e784-006618a9bb-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.659 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:47.659 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:47.659 DEBUG:boto:Method: HEAD 2024-04-12T03:25:47.659 DEBUG:boto:Path: /dkyges-13/ 2024-04-12T03:25:47.659 DEBUG:boto:Data: 2024-04-12T03:25:47.659 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.659 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:47.659 DEBUG:boto:Port: 8000 2024-04-12T03:25:47.659 DEBUG:boto:Params: {} 2024-04-12T03:25:47.660 DEBUG:boto:Token: None 2024-04-12T03:25:47.660 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/ 2024-04-12T03:25:47.660 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QDbZbWhNxKwWsm7eAfLXgFJ0Ggo= 2024-04-12T03:25:47.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QDbZbWhNxKwWsm7eAfLXgFJ0Ggo=', 'Content-Length': '0'} 2024-04-12T03:25:47.662 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a236a80edbd8b38-006618a9bb-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.663 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:25:47.663 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:47.663 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:47.663 DEBUG:boto:path=/dkyges-13/?versions&max-keys=999 2024-04-12T03:25:47.663 DEBUG:boto:auth_path=/dkyges-13/?versions&max-keys=999 2024-04-12T03:25:47.664 DEBUG:boto:Method: GET 2024-04-12T03:25:47.664 DEBUG:boto:Path: /dkyges-13/?versions&max-keys=999 2024-04-12T03:25:47.664 DEBUG:boto:Data: 2024-04-12T03:25:47.664 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.664 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:47.664 DEBUG:boto:Port: 8001 2024-04-12T03:25:47.665 DEBUG:boto:Params: {} 2024-04-12T03:25:47.665 DEBUG:boto:Token: None 2024-04-12T03:25:47.665 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/?versions 2024-04-12T03:25:47.665 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b0nM3xA2k794NGU6AL2X7G5IKts= 2024-04-12T03:25:47.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b0nM3xA2k794NGU6AL2X7G5IKts=', 'Content-Length': '0'} 2024-04-12T03:25:47.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000118c82187930783b-006618a9bb-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.672 DEBUG:boto:b'dkyges-13999falsedummySwFL1c7KE7MLvXFD1.wmtEUphs6xWXUtrue2024-04-12T03:24:41.660Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1No.FwkW62CECdIaWakbZwZXAPvIeOmOtrue2024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a29EpHGMR9P8CQ61qU0DObgl0ylooJjuatrue2024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:47.673 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:25:47.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-12T03:25:47.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-12T03:25:47.673 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:25:47.673 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:47.673 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:47.673 DEBUG:boto:path=/dkyges-13/?versions&max-keys=999 2024-04-12T03:25:47.673 DEBUG:boto:auth_path=/dkyges-13/?versions&max-keys=999 2024-04-12T03:25:47.673 DEBUG:boto:Method: GET 2024-04-12T03:25:47.674 DEBUG:boto:Path: /dkyges-13/?versions&max-keys=999 2024-04-12T03:25:47.674 DEBUG:boto:Data: 2024-04-12T03:25:47.674 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.674 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:47.674 DEBUG:boto:Port: 8000 2024-04-12T03:25:47.674 DEBUG:boto:Params: {} 2024-04-12T03:25:47.674 DEBUG:boto:Token: None 2024-04-12T03:25:47.674 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/?versions 2024-04-12T03:25:47.674 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b0nM3xA2k794NGU6AL2X7G5IKts= 2024-04-12T03:25:47.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b0nM3xA2k794NGU6AL2X7G5IKts=', 'Content-Length': '0'} 2024-04-12T03:25:47.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c251ea1d167ddb31-006618a9bb-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.685 DEBUG:boto:b'dkyges-13999falsedummySwFL1c7KE7MLvXFD1.wmtEUphs6xWXUtrue2024-04-12T03:24:41.660Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1No.FwkW62CECdIaWakbZwZXAPvIeOmOtrue2024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a29EpHGMR9P8CQ61qU0DObgl0ylooJjuatrue2024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:47.686 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:25:47.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-12T03:25:47.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-12T03:25:47.686 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:47.687 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:47.687 DEBUG:boto:path=/dkyges-13/?versions&max-keys=999 2024-04-12T03:25:47.687 DEBUG:boto:auth_path=/dkyges-13/?versions&max-keys=999 2024-04-12T03:25:47.687 DEBUG:boto:Method: GET 2024-04-12T03:25:47.687 DEBUG:boto:Path: /dkyges-13/?versions&max-keys=999 2024-04-12T03:25:47.687 DEBUG:boto:Data: 2024-04-12T03:25:47.687 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.687 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:47.687 DEBUG:boto:Port: 8001 2024-04-12T03:25:47.688 DEBUG:boto:Params: {} 2024-04-12T03:25:47.688 DEBUG:boto:Token: None 2024-04-12T03:25:47.688 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/?versions 2024-04-12T03:25:47.688 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b0nM3xA2k794NGU6AL2X7G5IKts= 2024-04-12T03:25:47.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b0nM3xA2k794NGU6AL2X7G5IKts=', 'Content-Length': '0'} 2024-04-12T03:25:47.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005429f8b94e6a2f0c-006618a9bb-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.696 DEBUG:boto:b'dkyges-13999falsedummySwFL1c7KE7MLvXFD1.wmtEUphs6xWXUtrue2024-04-12T03:24:41.660Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1No.FwkW62CECdIaWakbZwZXAPvIeOmOtrue2024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a29EpHGMR9P8CQ61qU0DObgl0ylooJjuatrue2024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:47.697 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:47.697 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:47.697 DEBUG:boto:path=/dkyges-13/?versions&max-keys=999 2024-04-12T03:25:47.697 DEBUG:boto:auth_path=/dkyges-13/?versions&max-keys=999 2024-04-12T03:25:47.698 DEBUG:boto:Method: GET 2024-04-12T03:25:47.698 DEBUG:boto:Path: /dkyges-13/?versions&max-keys=999 2024-04-12T03:25:47.698 DEBUG:boto:Data: 2024-04-12T03:25:47.698 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.698 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:47.698 DEBUG:boto:Port: 8000 2024-04-12T03:25:47.698 DEBUG:boto:Params: {} 2024-04-12T03:25:47.698 DEBUG:boto:Token: None 2024-04-12T03:25:47.698 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/?versions 2024-04-12T03:25:47.699 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b0nM3xA2k794NGU6AL2X7G5IKts= 2024-04-12T03:25:47.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b0nM3xA2k794NGU6AL2X7G5IKts=', 'Content-Length': '0'} 2024-04-12T03:25:47.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000704d19926a611748-006618a9bb-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.703 DEBUG:boto:b'dkyges-13999falsedummySwFL1c7KE7MLvXFD1.wmtEUphs6xWXUtrue2024-04-12T03:24:41.660Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1No.FwkW62CECdIaWakbZwZXAPvIeOmOtrue2024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a29EpHGMR9P8CQ61qU0DObgl0ylooJjuatrue2024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:47.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:25:47.704 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:25:47.704 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:25:47.704 DEBUG:boto:path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.705 DEBUG:boto:auth_path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.705 DEBUG:boto:Method: GET 2024-04-12T03:25:47.705 DEBUG:boto:Path: /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.705 DEBUG:boto:Data: 2024-04-12T03:25:47.705 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.705 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:47.705 DEBUG:boto:Port: 8001 2024-04-12T03:25:47.705 DEBUG:boto:Params: {} 2024-04-12T03:25:47.705 DEBUG:boto:Token: None 2024-04-12T03:25:47.706 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.706 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6iybqmp+c+8gmST/VJPjZ2CPeBk= 2024-04-12T03:25:47.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6iybqmp+c+8gmST/VJPjZ2CPeBk=', 'Content-Length': '0'} 2024-04-12T03:25:47.708 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cd12e050f041ad41-006618a9bb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.708 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:25:47.708 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:25:47.709 DEBUG:boto:path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.709 DEBUG:boto:auth_path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.709 DEBUG:boto:Method: GET 2024-04-12T03:25:47.709 DEBUG:boto:Path: /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.709 DEBUG:boto:Data: 2024-04-12T03:25:47.709 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.709 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:47.709 DEBUG:boto:Port: 8000 2024-04-12T03:25:47.709 DEBUG:boto:Params: {} 2024-04-12T03:25:47.709 DEBUG:boto:Token: None 2024-04-12T03:25:47.709 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.710 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6iybqmp+c+8gmST/VJPjZ2CPeBk= 2024-04-12T03:25:47.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6iybqmp+c+8gmST/VJPjZ2CPeBk=', 'Content-Length': '0'} 2024-04-12T03:25:47.711 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000113c887f755906af-006618a9bb-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.712 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:25:47.712 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:25:47.712 DEBUG:boto:path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.712 DEBUG:boto:auth_path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.712 DEBUG:boto:Method: HEAD 2024-04-12T03:25:47.712 DEBUG:boto:Path: /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.713 DEBUG:boto:Data: 2024-04-12T03:25:47.713 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.713 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:47.713 DEBUG:boto:Port: 8001 2024-04-12T03:25:47.713 DEBUG:boto:Params: {} 2024-04-12T03:25:47.713 DEBUG:boto:Token: None 2024-04-12T03:25:47.713 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.713 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l5OiF0FCsTA3fa0TzyFcbvE1//o= 2024-04-12T03:25:47.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l5OiF0FCsTA3fa0TzyFcbvE1//o=', 'Content-Length': '0'} 2024-04-12T03:25:47.716 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004ea7a10083d922eb-006618a9bb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.716 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:25:47.716 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:25:47.716 DEBUG:boto:path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.716 DEBUG:boto:auth_path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.716 DEBUG:boto:Method: HEAD 2024-04-12T03:25:47.717 DEBUG:boto:Path: /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.717 DEBUG:boto:Data: 2024-04-12T03:25:47.717 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.717 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:47.717 DEBUG:boto:Port: 8000 2024-04-12T03:25:47.717 DEBUG:boto:Params: {} 2024-04-12T03:25:47.717 DEBUG:boto:Token: None 2024-04-12T03:25:47.717 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.718 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l5OiF0FCsTA3fa0TzyFcbvE1//o= 2024-04-12T03:25:47.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l5OiF0FCsTA3fa0TzyFcbvE1//o=', 'Content-Length': '0'} 2024-04-12T03:25:47.720 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b67c42aa560b4d28-006618a9bb-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:25:47.720 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:25:47.721 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:25:47.721 DEBUG:boto:path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.721 DEBUG:boto:auth_path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.721 DEBUG:boto:Method: GET 2024-04-12T03:25:47.721 DEBUG:boto:Path: /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.721 DEBUG:boto:Data: 2024-04-12T03:25:47.721 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.721 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:47.721 DEBUG:boto:Port: 8001 2024-04-12T03:25:47.722 DEBUG:boto:Params: {} 2024-04-12T03:25:47.722 DEBUG:boto:Token: None 2024-04-12T03:25:47.722 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.722 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6iybqmp+c+8gmST/VJPjZ2CPeBk= 2024-04-12T03:25:47.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6iybqmp+c+8gmST/VJPjZ2CPeBk=', 'Content-Length': '0'} 2024-04-12T03:25:47.724 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e5e2a4f86df29745-006618a9bb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.724 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:25:47.725 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:25:47.725 DEBUG:boto:path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.725 DEBUG:boto:auth_path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.725 DEBUG:boto:Method: GET 2024-04-12T03:25:47.725 DEBUG:boto:Path: /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.725 DEBUG:boto:Data: 2024-04-12T03:25:47.725 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.725 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:47.725 DEBUG:boto:Port: 8000 2024-04-12T03:25:47.725 DEBUG:boto:Params: {} 2024-04-12T03:25:47.726 DEBUG:boto:Token: None 2024-04-12T03:25:47.726 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.726 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6iybqmp+c+8gmST/VJPjZ2CPeBk= 2024-04-12T03:25:47.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6iybqmp+c+8gmST/VJPjZ2CPeBk=', 'Content-Length': '0'} 2024-04-12T03:25:47.728 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bc151e8cacbfc9df-006618a9bb-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.728 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:47.729 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:47.729 DEBUG:boto:path=/dkyges-13/?prefix=dummy 2024-04-12T03:25:47.729 DEBUG:boto:auth_path=/dkyges-13/?prefix=dummy 2024-04-12T03:25:47.729 DEBUG:boto:Method: GET 2024-04-12T03:25:47.729 DEBUG:boto:Path: /dkyges-13/?prefix=dummy 2024-04-12T03:25:47.729 DEBUG:boto:Data: 2024-04-12T03:25:47.729 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.729 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:47.729 DEBUG:boto:Port: 8001 2024-04-12T03:25:47.729 DEBUG:boto:Params: {} 2024-04-12T03:25:47.730 DEBUG:boto:Token: None 2024-04-12T03:25:47.730 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/ 2024-04-12T03:25:47.730 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sDfSkj73p74fkR2hh04y8nhDlzc= 2024-04-12T03:25:47.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sDfSkj73p74fkR2hh04y8nhDlzc=', 'Content-Length': '0'} 2024-04-12T03:25:47.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4c4de8dfecdc5a8-006618a9bb-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.774 DEBUG:boto:b'dkyges-13dummy1000falsedummy2024-04-12T03:24:41.660Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:47.775 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:47.775 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:47.775 DEBUG:boto:path=/dkyges-13/?prefix=dummy 2024-04-12T03:25:47.775 DEBUG:boto:auth_path=/dkyges-13/?prefix=dummy 2024-04-12T03:25:47.776 DEBUG:boto:Method: GET 2024-04-12T03:25:47.776 DEBUG:boto:Path: /dkyges-13/?prefix=dummy 2024-04-12T03:25:47.776 DEBUG:boto:Data: 2024-04-12T03:25:47.776 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.776 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:47.776 DEBUG:boto:Port: 8000 2024-04-12T03:25:47.776 DEBUG:boto:Params: {} 2024-04-12T03:25:47.776 DEBUG:boto:Token: None 2024-04-12T03:25:47.777 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/ 2024-04-12T03:25:47.777 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sDfSkj73p74fkR2hh04y8nhDlzc= 2024-04-12T03:25:47.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sDfSkj73p74fkR2hh04y8nhDlzc=', 'Content-Length': '0'} 2024-04-12T03:25:47.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4def72040ec8c25-006618a9bb-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.780 DEBUG:boto:b'dkyges-13dummy1000falsedummy2024-04-12T03:24:41.660Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:47.780 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:25:47.780 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:25:47.780 DEBUG:boto:Method: HEAD 2024-04-12T03:25:47.780 DEBUG:boto:Path: /dkyges-13/dummy 2024-04-12T03:25:47.780 DEBUG:boto:Data: 2024-04-12T03:25:47.781 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.781 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:47.781 DEBUG:boto:Port: 8001 2024-04-12T03:25:47.781 DEBUG:boto:Params: {} 2024-04-12T03:25:47.781 DEBUG:boto:Token: None 2024-04-12T03:25:47.781 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/dummy 2024-04-12T03:25:47.781 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IBL9+zJCepqZlVKgzopZjfHRmK0= 2024-04-12T03:25:47.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IBL9+zJCepqZlVKgzopZjfHRmK0=', 'Content-Length': '0'} 2024-04-12T03:25:47.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d0bb9c93d75e6560-006618a9bb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.784 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:25:47.784 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:25:47.784 DEBUG:boto:Method: HEAD 2024-04-12T03:25:47.784 DEBUG:boto:Path: /dkyges-13/dummy 2024-04-12T03:25:47.784 DEBUG:boto:Data: 2024-04-12T03:25:47.784 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.784 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:47.784 DEBUG:boto:Port: 8000 2024-04-12T03:25:47.785 DEBUG:boto:Params: {} 2024-04-12T03:25:47.785 DEBUG:boto:Token: None 2024-04-12T03:25:47.785 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/dummy 2024-04-12T03:25:47.785 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IBL9+zJCepqZlVKgzopZjfHRmK0= 2024-04-12T03:25:47.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IBL9+zJCepqZlVKgzopZjfHRmK0=', 'Content-Length': '0'} 2024-04-12T03:25:47.787 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009b3a8e3755b5ed4f-006618a9bb-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:25:47.788 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:25:47.788 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:25:47.788 DEBUG:boto:path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.788 DEBUG:boto:auth_path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.788 DEBUG:boto:Method: GET 2024-04-12T03:25:47.788 DEBUG:boto:Path: /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.788 DEBUG:boto:Data: 2024-04-12T03:25:47.788 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.788 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:47.788 DEBUG:boto:Port: 8001 2024-04-12T03:25:47.788 DEBUG:boto:Params: {} 2024-04-12T03:25:47.789 DEBUG:boto:Token: None 2024-04-12T03:25:47.789 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.789 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6iybqmp+c+8gmST/VJPjZ2CPeBk= 2024-04-12T03:25:47.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6iybqmp+c+8gmST/VJPjZ2CPeBk=', 'Content-Length': '0'} 2024-04-12T03:25:47.791 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000d9940f7da8bff49-006618a9bb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.791 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:25:47.791 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:25:47.792 DEBUG:boto:path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.792 DEBUG:boto:auth_path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.792 DEBUG:boto:Method: GET 2024-04-12T03:25:47.792 DEBUG:boto:Path: /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.792 DEBUG:boto:Data: 2024-04-12T03:25:47.792 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.792 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:47.792 DEBUG:boto:Port: 8000 2024-04-12T03:25:47.792 DEBUG:boto:Params: {} 2024-04-12T03:25:47.792 DEBUG:boto:Token: None 2024-04-12T03:25:47.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:25:47.793 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6iybqmp+c+8gmST/VJPjZ2CPeBk= 2024-04-12T03:25:47.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6iybqmp+c+8gmST/VJPjZ2CPeBk=', 'Content-Length': '0'} 2024-04-12T03:25:47.795 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c82121a04dcbf604-006618a9bb-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.795 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:47.795 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:47.795 DEBUG:boto:path=/dkyges-13/?prefix=dummy 2024-04-12T03:25:47.796 DEBUG:boto:auth_path=/dkyges-13/?prefix=dummy 2024-04-12T03:25:47.796 DEBUG:boto:Method: GET 2024-04-12T03:25:47.796 DEBUG:boto:Path: /dkyges-13/?prefix=dummy 2024-04-12T03:25:47.796 DEBUG:boto:Data: 2024-04-12T03:25:47.796 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.796 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:47.796 DEBUG:boto:Port: 8001 2024-04-12T03:25:47.796 DEBUG:boto:Params: {} 2024-04-12T03:25:47.796 DEBUG:boto:Token: None 2024-04-12T03:25:47.796 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/ 2024-04-12T03:25:47.797 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sDfSkj73p74fkR2hh04y8nhDlzc= 2024-04-12T03:25:47.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sDfSkj73p74fkR2hh04y8nhDlzc=', 'Content-Length': '0'} 2024-04-12T03:25:47.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed5b40ebce9d4dd2-006618a9bb-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.846 DEBUG:boto:b'dkyges-13dummy1000falsedummy2024-04-12T03:24:41.660Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:47.846 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:47.846 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:47.847 DEBUG:boto:path=/dkyges-13/?prefix=dummy 2024-04-12T03:25:47.847 DEBUG:boto:auth_path=/dkyges-13/?prefix=dummy 2024-04-12T03:25:47.847 DEBUG:boto:Method: GET 2024-04-12T03:25:47.847 DEBUG:boto:Path: /dkyges-13/?prefix=dummy 2024-04-12T03:25:47.847 DEBUG:boto:Data: 2024-04-12T03:25:47.847 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.847 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:47.847 DEBUG:boto:Port: 8000 2024-04-12T03:25:47.847 DEBUG:boto:Params: {} 2024-04-12T03:25:47.848 DEBUG:boto:Token: None 2024-04-12T03:25:47.848 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/ 2024-04-12T03:25:47.848 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sDfSkj73p74fkR2hh04y8nhDlzc= 2024-04-12T03:25:47.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sDfSkj73p74fkR2hh04y8nhDlzc=', 'Content-Length': '0'} 2024-04-12T03:25:47.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ed0bf1b1a225fa6-006618a9bb-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.851 DEBUG:boto:b'dkyges-13dummy1000falsedummy2024-04-12T03:24:41.660Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:47.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:25:47.851 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:25:47.852 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:25:47.852 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.852 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.852 DEBUG:boto:Method: GET 2024-04-12T03:25:47.852 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.852 DEBUG:boto:Data: 2024-04-12T03:25:47.852 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.852 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:47.852 DEBUG:boto:Port: 8001 2024-04-12T03:25:47.853 DEBUG:boto:Params: {} 2024-04-12T03:25:47.853 DEBUG:boto:Token: None 2024-04-12T03:25:47.853 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.853 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BMNsaMC853irQUoYJ+mpoTnuXEI= 2024-04-12T03:25:47.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BMNsaMC853irQUoYJ+mpoTnuXEI=', 'Content-Length': '0'} 2024-04-12T03:25:47.856 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cc8b62367391f86-006618a9bb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.856 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:25:47.856 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:25:47.856 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.856 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.857 DEBUG:boto:Method: GET 2024-04-12T03:25:47.857 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.857 DEBUG:boto:Data: 2024-04-12T03:25:47.857 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.857 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:47.857 DEBUG:boto:Port: 8000 2024-04-12T03:25:47.857 DEBUG:boto:Params: {} 2024-04-12T03:25:47.857 DEBUG:boto:Token: None 2024-04-12T03:25:47.858 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.858 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BMNsaMC853irQUoYJ+mpoTnuXEI= 2024-04-12T03:25:47.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BMNsaMC853irQUoYJ+mpoTnuXEI=', 'Content-Length': '0'} 2024-04-12T03:25:47.860 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000325a160c27533a69-006618a9bb-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.903 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:25:47.903 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:25:47.903 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.903 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.903 DEBUG:boto:Method: HEAD 2024-04-12T03:25:47.903 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.903 DEBUG:boto:Data: 2024-04-12T03:25:47.903 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.903 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:47.903 DEBUG:boto:Port: 8001 2024-04-12T03:25:47.904 DEBUG:boto:Params: {} 2024-04-12T03:25:47.904 DEBUG:boto:Token: None 2024-04-12T03:25:47.904 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.904 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Bg8lgiQrVngg94IZk0mPDH1E5t8= 2024-04-12T03:25:47.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bg8lgiQrVngg94IZk0mPDH1E5t8=', 'Content-Length': '0'} 2024-04-12T03:25:47.906 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dc62483cff638b2d-006618a9bb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.906 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:25:47.906 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:25:47.907 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.907 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.907 DEBUG:boto:Method: HEAD 2024-04-12T03:25:47.907 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.907 DEBUG:boto:Data: 2024-04-12T03:25:47.907 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.907 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:47.907 DEBUG:boto:Port: 8000 2024-04-12T03:25:47.907 DEBUG:boto:Params: {} 2024-04-12T03:25:47.907 DEBUG:boto:Token: None 2024-04-12T03:25:47.908 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.908 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Bg8lgiQrVngg94IZk0mPDH1E5t8= 2024-04-12T03:25:47.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bg8lgiQrVngg94IZk0mPDH1E5t8=', 'Content-Length': '0'} 2024-04-12T03:25:47.910 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000331d254e30846a45-006618a9bb-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:25:47.910 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:25:47.910 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:25:47.910 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.910 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.910 DEBUG:boto:Method: GET 2024-04-12T03:25:47.910 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.911 DEBUG:boto:Data: 2024-04-12T03:25:47.911 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.911 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:47.911 DEBUG:boto:Port: 8001 2024-04-12T03:25:47.911 DEBUG:boto:Params: {} 2024-04-12T03:25:47.911 DEBUG:boto:Token: None 2024-04-12T03:25:47.911 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.911 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BMNsaMC853irQUoYJ+mpoTnuXEI= 2024-04-12T03:25:47.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BMNsaMC853irQUoYJ+mpoTnuXEI=', 'Content-Length': '0'} 2024-04-12T03:25:47.913 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b3a90f5d71c26b0-006618a9bb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.960 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:25:47.960 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:25:47.960 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.960 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.960 DEBUG:boto:Method: GET 2024-04-12T03:25:47.960 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.960 DEBUG:boto:Data: 2024-04-12T03:25:47.960 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.961 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:47.961 DEBUG:boto:Port: 8000 2024-04-12T03:25:47.961 DEBUG:boto:Params: {} 2024-04-12T03:25:47.961 DEBUG:boto:Token: None 2024-04-12T03:25:47.961 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.961 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BMNsaMC853irQUoYJ+mpoTnuXEI= 2024-04-12T03:25:47.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BMNsaMC853irQUoYJ+mpoTnuXEI=', 'Content-Length': '0'} 2024-04-12T03:25:47.964 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055f1f16826795ed0-006618a9bb-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.964 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:47.964 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:47.964 DEBUG:boto:path=/dkyges-13/?prefix=obj-a1 2024-04-12T03:25:47.964 DEBUG:boto:auth_path=/dkyges-13/?prefix=obj-a1 2024-04-12T03:25:47.964 DEBUG:boto:Method: GET 2024-04-12T03:25:47.965 DEBUG:boto:Path: /dkyges-13/?prefix=obj-a1 2024-04-12T03:25:47.965 DEBUG:boto:Data: 2024-04-12T03:25:47.965 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.965 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:47.965 DEBUG:boto:Port: 8001 2024-04-12T03:25:47.965 DEBUG:boto:Params: {} 2024-04-12T03:25:47.965 DEBUG:boto:Token: None 2024-04-12T03:25:47.965 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/ 2024-04-12T03:25:47.965 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sDfSkj73p74fkR2hh04y8nhDlzc= 2024-04-12T03:25:47.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sDfSkj73p74fkR2hh04y8nhDlzc=', 'Content-Length': '0'} 2024-04-12T03:25:47.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f42b77a1d2cc476b-006618a9bb-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.968 DEBUG:boto:b'dkyges-13obj-a11000falseobj-a12024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:47.969 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:47.969 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:47.969 DEBUG:boto:path=/dkyges-13/?prefix=obj-a1 2024-04-12T03:25:47.969 DEBUG:boto:auth_path=/dkyges-13/?prefix=obj-a1 2024-04-12T03:25:47.969 DEBUG:boto:Method: GET 2024-04-12T03:25:47.969 DEBUG:boto:Path: /dkyges-13/?prefix=obj-a1 2024-04-12T03:25:47.969 DEBUG:boto:Data: 2024-04-12T03:25:47.969 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.969 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:47.969 DEBUG:boto:Port: 8000 2024-04-12T03:25:47.970 DEBUG:boto:Params: {} 2024-04-12T03:25:47.970 DEBUG:boto:Token: None 2024-04-12T03:25:47.970 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/ 2024-04-12T03:25:47.970 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sDfSkj73p74fkR2hh04y8nhDlzc= 2024-04-12T03:25:47.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sDfSkj73p74fkR2hh04y8nhDlzc=', 'Content-Length': '0'} 2024-04-12T03:25:47.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ede852dea79395fb-006618a9bb-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.973 DEBUG:boto:b'dkyges-13obj-a11000falseobj-a12024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:47.973 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:25:47.973 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:25:47.973 DEBUG:boto:Method: HEAD 2024-04-12T03:25:47.973 DEBUG:boto:Path: /dkyges-13/obj-a1 2024-04-12T03:25:47.974 DEBUG:boto:Data: 2024-04-12T03:25:47.974 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.974 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:47.974 DEBUG:boto:Port: 8001 2024-04-12T03:25:47.974 DEBUG:boto:Params: {} 2024-04-12T03:25:47.974 DEBUG:boto:Token: None 2024-04-12T03:25:47.974 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/obj-a1 2024-04-12T03:25:47.974 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mRkK1ShxklRZ/n9HFe2NnhmJZiM= 2024-04-12T03:25:47.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mRkK1ShxklRZ/n9HFe2NnhmJZiM=', 'Content-Length': '0'} 2024-04-12T03:25:47.977 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5edba37bd064a95-006618a9bb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.977 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:25:47.977 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:25:47.977 DEBUG:boto:Method: HEAD 2024-04-12T03:25:47.977 DEBUG:boto:Path: /dkyges-13/obj-a1 2024-04-12T03:25:47.978 DEBUG:boto:Data: 2024-04-12T03:25:47.978 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.980 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:47.980 DEBUG:boto:Port: 8000 2024-04-12T03:25:47.980 DEBUG:boto:Params: {} 2024-04-12T03:25:47.981 DEBUG:boto:Token: None 2024-04-12T03:25:47.981 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/obj-a1 2024-04-12T03:25:47.981 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mRkK1ShxklRZ/n9HFe2NnhmJZiM= 2024-04-12T03:25:47.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mRkK1ShxklRZ/n9HFe2NnhmJZiM=', 'Content-Length': '0'} 2024-04-12T03:25:47.984 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ae131ddc19dc3dd-006618a9bb-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:47.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:25:47.984 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:25:47.984 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:25:47.984 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.984 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.985 DEBUG:boto:Method: GET 2024-04-12T03:25:47.985 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.985 DEBUG:boto:Data: 2024-04-12T03:25:47.985 DEBUG:boto:Headers: {} 2024-04-12T03:25:47.985 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:47.985 DEBUG:boto:Port: 8001 2024-04-12T03:25:47.985 DEBUG:boto:Params: {} 2024-04-12T03:25:47.985 DEBUG:boto:Token: None 2024-04-12T03:25:47.985 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:47 GMT /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:47.985 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BMNsaMC853irQUoYJ+mpoTnuXEI= 2024-04-12T03:25:47.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BMNsaMC853irQUoYJ+mpoTnuXEI=', 'Content-Length': '0'} 2024-04-12T03:25:47.988 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fc059313b3f9608-006618a9bb-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.030 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:25:48.031 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:25:48.031 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:48.031 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:48.031 DEBUG:boto:Method: GET 2024-04-12T03:25:48.031 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:48.031 DEBUG:boto:Data: 2024-04-12T03:25:48.031 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.031 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:48.031 DEBUG:boto:Port: 8000 2024-04-12T03:25:48.032 DEBUG:boto:Params: {} 2024-04-12T03:25:48.032 DEBUG:boto:Token: None 2024-04-12T03:25:48.032 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:25:48.032 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sKdyvGDM/6YOlPT31MShxcpiRhc= 2024-04-12T03:25:48.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sKdyvGDM/6YOlPT31MShxcpiRhc=', 'Content-Length': '0'} 2024-04-12T03:25:48.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c5873f694e8e6a73-006618a9bc-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.035 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:48.035 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:48.035 DEBUG:boto:path=/dkyges-13/?prefix=obj-a1 2024-04-12T03:25:48.035 DEBUG:boto:auth_path=/dkyges-13/?prefix=obj-a1 2024-04-12T03:25:48.035 DEBUG:boto:Method: GET 2024-04-12T03:25:48.035 DEBUG:boto:Path: /dkyges-13/?prefix=obj-a1 2024-04-12T03:25:48.035 DEBUG:boto:Data: 2024-04-12T03:25:48.035 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.035 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:48.035 DEBUG:boto:Port: 8001 2024-04-12T03:25:48.036 DEBUG:boto:Params: {} 2024-04-12T03:25:48.036 DEBUG:boto:Token: None 2024-04-12T03:25:48.036 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/ 2024-04-12T03:25:48.036 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ca+bvswxp6dg9qHEogj9R+QcDL8= 2024-04-12T03:25:48.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ca+bvswxp6dg9qHEogj9R+QcDL8=', 'Content-Length': '0'} 2024-04-12T03:25:48.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5773034d2218396-006618a9bc-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.039 DEBUG:boto:b'dkyges-13obj-a11000falseobj-a12024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:48.040 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:48.040 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:48.041 DEBUG:boto:path=/dkyges-13/?prefix=obj-a1 2024-04-12T03:25:48.041 DEBUG:boto:auth_path=/dkyges-13/?prefix=obj-a1 2024-04-12T03:25:48.041 DEBUG:boto:Method: GET 2024-04-12T03:25:48.041 DEBUG:boto:Path: /dkyges-13/?prefix=obj-a1 2024-04-12T03:25:48.042 DEBUG:boto:Data: 2024-04-12T03:25:48.042 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.042 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:48.043 DEBUG:boto:Port: 8000 2024-04-12T03:25:48.043 DEBUG:boto:Params: {} 2024-04-12T03:25:48.043 DEBUG:boto:Token: None 2024-04-12T03:25:48.044 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/ 2024-04-12T03:25:48.044 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ca+bvswxp6dg9qHEogj9R+QcDL8= 2024-04-12T03:25:48.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ca+bvswxp6dg9qHEogj9R+QcDL8=', 'Content-Length': '0'} 2024-04-12T03:25:48.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f6b6ae74cc3285c-006618a9bc-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.094 DEBUG:boto:b'dkyges-13obj-a11000falseobj-a12024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:48.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:25:48.095 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:25:48.095 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:25:48.096 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.096 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.096 DEBUG:boto:Method: GET 2024-04-12T03:25:48.096 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.096 DEBUG:boto:Data: 2024-04-12T03:25:48.096 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.096 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:48.096 DEBUG:boto:Port: 8001 2024-04-12T03:25:48.096 DEBUG:boto:Params: {} 2024-04-12T03:25:48.096 DEBUG:boto:Token: None 2024-04-12T03:25:48.097 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.097 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4Ze7eFZS+g1T3CZQ2wIUPaE0E/A= 2024-04-12T03:25:48.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4Ze7eFZS+g1T3CZQ2wIUPaE0E/A=', 'Content-Length': '0'} 2024-04-12T03:25:48.099 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a3ef9d5700d26d8-006618a9bc-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.099 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:25:48.099 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:25:48.099 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.100 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.100 DEBUG:boto:Method: GET 2024-04-12T03:25:48.100 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.100 DEBUG:boto:Data: 2024-04-12T03:25:48.100 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.100 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:48.100 DEBUG:boto:Port: 8000 2024-04-12T03:25:48.100 DEBUG:boto:Params: {} 2024-04-12T03:25:48.100 DEBUG:boto:Token: None 2024-04-12T03:25:48.101 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.101 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4Ze7eFZS+g1T3CZQ2wIUPaE0E/A= 2024-04-12T03:25:48.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4Ze7eFZS+g1T3CZQ2wIUPaE0E/A=', 'Content-Length': '0'} 2024-04-12T03:25:48.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003aea63357885cdb1-006618a9bc-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.103 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:25:48.103 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:25:48.104 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.104 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.104 DEBUG:boto:Method: HEAD 2024-04-12T03:25:48.104 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.104 DEBUG:boto:Data: 2024-04-12T03:25:48.104 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.104 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:48.104 DEBUG:boto:Port: 8001 2024-04-12T03:25:48.104 DEBUG:boto:Params: {} 2024-04-12T03:25:48.105 DEBUG:boto:Token: None 2024-04-12T03:25:48.105 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.105 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vWnDIpOZsqWNStcaCqfjIkb9Kg8= 2024-04-12T03:25:48.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vWnDIpOZsqWNStcaCqfjIkb9Kg8=', 'Content-Length': '0'} 2024-04-12T03:25:48.107 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6b7b2fcd1b50356-006618a9bc-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.107 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:25:48.108 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:25:48.108 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.108 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.108 DEBUG:boto:Method: HEAD 2024-04-12T03:25:48.108 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.108 DEBUG:boto:Data: 2024-04-12T03:25:48.108 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.108 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:48.109 DEBUG:boto:Port: 8000 2024-04-12T03:25:48.109 DEBUG:boto:Params: {} 2024-04-12T03:25:48.109 DEBUG:boto:Token: None 2024-04-12T03:25:48.109 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.109 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vWnDIpOZsqWNStcaCqfjIkb9Kg8= 2024-04-12T03:25:48.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vWnDIpOZsqWNStcaCqfjIkb9Kg8=', 'Content-Length': '0'} 2024-04-12T03:25:48.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef55bcd48e6d1a8e-006618a9bc-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:25:48.112 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:25:48.112 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:25:48.112 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.112 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.112 DEBUG:boto:Method: GET 2024-04-12T03:25:48.112 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.112 DEBUG:boto:Data: 2024-04-12T03:25:48.113 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.113 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:48.113 DEBUG:boto:Port: 8001 2024-04-12T03:25:48.113 DEBUG:boto:Params: {} 2024-04-12T03:25:48.113 DEBUG:boto:Token: None 2024-04-12T03:25:48.113 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.113 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4Ze7eFZS+g1T3CZQ2wIUPaE0E/A= 2024-04-12T03:25:48.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4Ze7eFZS+g1T3CZQ2wIUPaE0E/A=', 'Content-Length': '0'} 2024-04-12T03:25:48.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000441907237de48735-006618a9bc-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.161 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:25:48.161 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:25:48.161 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.161 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.161 DEBUG:boto:Method: GET 2024-04-12T03:25:48.161 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.161 DEBUG:boto:Data: 2024-04-12T03:25:48.161 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.161 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:48.162 DEBUG:boto:Port: 8000 2024-04-12T03:25:48.162 DEBUG:boto:Params: {} 2024-04-12T03:25:48.162 DEBUG:boto:Token: None 2024-04-12T03:25:48.162 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.162 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4Ze7eFZS+g1T3CZQ2wIUPaE0E/A= 2024-04-12T03:25:48.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4Ze7eFZS+g1T3CZQ2wIUPaE0E/A=', 'Content-Length': '0'} 2024-04-12T03:25:48.164 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000914e59e998aada95-006618a9bc-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.207 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:48.207 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:48.207 DEBUG:boto:path=/dkyges-13/?prefix=obj-a2 2024-04-12T03:25:48.207 DEBUG:boto:auth_path=/dkyges-13/?prefix=obj-a2 2024-04-12T03:25:48.208 DEBUG:boto:Method: GET 2024-04-12T03:25:48.208 DEBUG:boto:Path: /dkyges-13/?prefix=obj-a2 2024-04-12T03:25:48.208 DEBUG:boto:Data: 2024-04-12T03:25:48.208 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.208 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:48.208 DEBUG:boto:Port: 8001 2024-04-12T03:25:48.209 DEBUG:boto:Params: {} 2024-04-12T03:25:48.209 DEBUG:boto:Token: None 2024-04-12T03:25:48.209 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/ 2024-04-12T03:25:48.210 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ca+bvswxp6dg9qHEogj9R+QcDL8= 2024-04-12T03:25:48.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ca+bvswxp6dg9qHEogj9R+QcDL8=', 'Content-Length': '0'} 2024-04-12T03:25:48.213 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7ebea8a63397561-006618a9bc-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.213 DEBUG:boto:b'dkyges-13obj-a21000falseobj-a22024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:48.214 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:48.214 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:48.214 DEBUG:boto:path=/dkyges-13/?prefix=obj-a2 2024-04-12T03:25:48.214 DEBUG:boto:auth_path=/dkyges-13/?prefix=obj-a2 2024-04-12T03:25:48.214 DEBUG:boto:Method: GET 2024-04-12T03:25:48.214 DEBUG:boto:Path: /dkyges-13/?prefix=obj-a2 2024-04-12T03:25:48.214 DEBUG:boto:Data: 2024-04-12T03:25:48.214 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.214 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:48.214 DEBUG:boto:Port: 8000 2024-04-12T03:25:48.215 DEBUG:boto:Params: {} 2024-04-12T03:25:48.215 DEBUG:boto:Token: None 2024-04-12T03:25:48.215 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/ 2024-04-12T03:25:48.215 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ca+bvswxp6dg9qHEogj9R+QcDL8= 2024-04-12T03:25:48.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ca+bvswxp6dg9qHEogj9R+QcDL8=', 'Content-Length': '0'} 2024-04-12T03:25:48.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca1ebb032697f7ad-006618a9bc-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.259 DEBUG:boto:b'dkyges-13obj-a21000falseobj-a22024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:48.259 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:25:48.260 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:25:48.260 DEBUG:boto:Method: HEAD 2024-04-12T03:25:48.260 DEBUG:boto:Path: /dkyges-13/obj-a2 2024-04-12T03:25:48.260 DEBUG:boto:Data: 2024-04-12T03:25:48.260 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.260 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:48.260 DEBUG:boto:Port: 8001 2024-04-12T03:25:48.260 DEBUG:boto:Params: {} 2024-04-12T03:25:48.260 DEBUG:boto:Token: None 2024-04-12T03:25:48.260 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/obj-a2 2024-04-12T03:25:48.261 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ttwpS7uvsmc4PKqNI7+7LdJDu7Y= 2024-04-12T03:25:48.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ttwpS7uvsmc4PKqNI7+7LdJDu7Y=', 'Content-Length': '0'} 2024-04-12T03:25:48.263 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d5ebb9757610fa8-006618a9bc-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.263 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:25:48.263 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:25:48.264 DEBUG:boto:Method: HEAD 2024-04-12T03:25:48.264 DEBUG:boto:Path: /dkyges-13/obj-a2 2024-04-12T03:25:48.264 DEBUG:boto:Data: 2024-04-12T03:25:48.264 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.264 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:48.264 DEBUG:boto:Port: 8000 2024-04-12T03:25:48.264 DEBUG:boto:Params: {} 2024-04-12T03:25:48.264 DEBUG:boto:Token: None 2024-04-12T03:25:48.264 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/obj-a2 2024-04-12T03:25:48.264 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ttwpS7uvsmc4PKqNI7+7LdJDu7Y= 2024-04-12T03:25:48.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ttwpS7uvsmc4PKqNI7+7LdJDu7Y=', 'Content-Length': '0'} 2024-04-12T03:25:48.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005d6b56826e4d2fa4-006618a9bc-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:25:48.267 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:25:48.267 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:25:48.267 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.268 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.268 DEBUG:boto:Method: GET 2024-04-12T03:25:48.268 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.268 DEBUG:boto:Data: 2024-04-12T03:25:48.268 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.268 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:48.268 DEBUG:boto:Port: 8001 2024-04-12T03:25:48.268 DEBUG:boto:Params: {} 2024-04-12T03:25:48.268 DEBUG:boto:Token: None 2024-04-12T03:25:48.269 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.269 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4Ze7eFZS+g1T3CZQ2wIUPaE0E/A= 2024-04-12T03:25:48.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4Ze7eFZS+g1T3CZQ2wIUPaE0E/A=', 'Content-Length': '0'} 2024-04-12T03:25:48.271 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ba29849f0ee07f1a-006618a9bc-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.272 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:25:48.272 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:25:48.272 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.272 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.272 DEBUG:boto:Method: GET 2024-04-12T03:25:48.272 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.272 DEBUG:boto:Data: 2024-04-12T03:25:48.272 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.272 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:48.272 DEBUG:boto:Port: 8000 2024-04-12T03:25:48.272 DEBUG:boto:Params: {} 2024-04-12T03:25:48.273 DEBUG:boto:Token: None 2024-04-12T03:25:48.273 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:25:48.273 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4Ze7eFZS+g1T3CZQ2wIUPaE0E/A= 2024-04-12T03:25:48.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4Ze7eFZS+g1T3CZQ2wIUPaE0E/A=', 'Content-Length': '0'} 2024-04-12T03:25:48.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076f4fb7a3f82232c-006618a9bc-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.275 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:48.275 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:48.275 DEBUG:boto:path=/dkyges-13/?prefix=obj-a2 2024-04-12T03:25:48.276 DEBUG:boto:auth_path=/dkyges-13/?prefix=obj-a2 2024-04-12T03:25:48.276 DEBUG:boto:Method: GET 2024-04-12T03:25:48.276 DEBUG:boto:Path: /dkyges-13/?prefix=obj-a2 2024-04-12T03:25:48.276 DEBUG:boto:Data: 2024-04-12T03:25:48.276 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.276 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:48.276 DEBUG:boto:Port: 8001 2024-04-12T03:25:48.276 DEBUG:boto:Params: {} 2024-04-12T03:25:48.276 DEBUG:boto:Token: None 2024-04-12T03:25:48.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/ 2024-04-12T03:25:48.276 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ca+bvswxp6dg9qHEogj9R+QcDL8= 2024-04-12T03:25:48.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ca+bvswxp6dg9qHEogj9R+QcDL8=', 'Content-Length': '0'} 2024-04-12T03:25:48.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b41195a18ca9ab13-006618a9bc-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.279 DEBUG:boto:b'dkyges-13obj-a21000falseobj-a22024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:48.279 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:48.280 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:48.280 DEBUG:boto:path=/dkyges-13/?prefix=obj-a2 2024-04-12T03:25:48.280 DEBUG:boto:auth_path=/dkyges-13/?prefix=obj-a2 2024-04-12T03:25:48.280 DEBUG:boto:Method: GET 2024-04-12T03:25:48.280 DEBUG:boto:Path: /dkyges-13/?prefix=obj-a2 2024-04-12T03:25:48.280 DEBUG:boto:Data: 2024-04-12T03:25:48.280 DEBUG:boto:Headers: {} 2024-04-12T03:25:48.280 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:48.280 DEBUG:boto:Port: 8000 2024-04-12T03:25:48.280 DEBUG:boto:Params: {} 2024-04-12T03:25:48.280 DEBUG:boto:Token: None 2024-04-12T03:25:48.281 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:48 GMT /dkyges-13/ 2024-04-12T03:25:48.281 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ca+bvswxp6dg9qHEogj9R+QcDL8= 2024-04-12T03:25:48.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ca+bvswxp6dg9qHEogj9R+QcDL8=', 'Content-Length': '0'} 2024-04-12T03:25:48.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cd57fd7f622cb40-006618a9bc-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:48.326 DEBUG:boto:b'dkyges-13obj-a21000falseobj-a22024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:48.327 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-13 zones={a2, a1} 2024-04-12T03:25:48.327 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-14 2024-04-12T03:25:48.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:25:48.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:25:48.328 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:25:48.396 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:48.397 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:49.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:49.435+0000 7f1865bc7780 1 bucket sync caught up with source: 2024-04-12T03:25:49.437 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.19.11, , , 00000000002.27.11, 00000000010.21.6, ] 2024-04-12T03:25:49.437 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.19.11, , , 00000000002.27.11, 00000000010.21.6, ] 2024-04-12T03:25:49.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:49.435+0000 7f1865bc7780 0 bucket checkpoint complete 2024-04-12T03:25:49.454 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-14 2024-04-12T03:25:49.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:25:49.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:25:49.455 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:25:49.521 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:49.521 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:50.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:50.562+0000 7f6458d6b780 1 bucket sync caught up with source: 2024-04-12T03:25:50.563 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.31.10, , , 00000000010.54.6, 00000000002.10.11, ] 2024-04-12T03:25:50.563 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.31.10, , , 00000000010.54.6, 00000000002.10.11, ] 2024-04-12T03:25:50.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:50.562+0000 7f6458d6b780 0 bucket checkpoint complete 2024-04-12T03:25:50.580 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-14 zones={a2, a1} 2024-04-12T03:25:50.580 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:50.580 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:50.580 DEBUG:boto:Method: HEAD 2024-04-12T03:25:50.580 DEBUG:boto:Path: /dkyges-14/ 2024-04-12T03:25:50.580 DEBUG:boto:Data: 2024-04-12T03:25:50.581 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.581 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.581 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.581 DEBUG:boto:Params: {} 2024-04-12T03:25:50.581 DEBUG:boto:Token: None 2024-04-12T03:25:50.581 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/ 2024-04-12T03:25:50.581 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hYopheMGM6caxQAx4I61ImcX64M= 2024-04-12T03:25:50.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hYopheMGM6caxQAx4I61ImcX64M=', 'Content-Length': '0'} 2024-04-12T03:25:50.584 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000225960bb2a564d95-006618a9be-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.584 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:50.584 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:50.585 DEBUG:boto:Method: HEAD 2024-04-12T03:25:50.585 DEBUG:boto:Path: /dkyges-14/ 2024-04-12T03:25:50.585 DEBUG:boto:Data: 2024-04-12T03:25:50.585 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.585 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.585 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.585 DEBUG:boto:Params: {} 2024-04-12T03:25:50.585 DEBUG:boto:Token: None 2024-04-12T03:25:50.585 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/ 2024-04-12T03:25:50.586 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hYopheMGM6caxQAx4I61ImcX64M= 2024-04-12T03:25:50.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hYopheMGM6caxQAx4I61ImcX64M=', 'Content-Length': '0'} 2024-04-12T03:25:50.588 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e855816fe9f77ad3-006618a9be-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.588 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:25:50.588 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:50.588 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:50.588 DEBUG:boto:path=/dkyges-14/?versions&max-keys=999 2024-04-12T03:25:50.588 DEBUG:boto:auth_path=/dkyges-14/?versions&max-keys=999 2024-04-12T03:25:50.588 DEBUG:boto:Method: GET 2024-04-12T03:25:50.589 DEBUG:boto:Path: /dkyges-14/?versions&max-keys=999 2024-04-12T03:25:50.589 DEBUG:boto:Data: 2024-04-12T03:25:50.589 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.589 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.589 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.589 DEBUG:boto:Params: {} 2024-04-12T03:25:50.589 DEBUG:boto:Token: None 2024-04-12T03:25:50.589 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/?versions 2024-04-12T03:25:50.589 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vsxjXn3E4FGnBbnHVxx66y4pe4w= 2024-04-12T03:25:50.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vsxjXn3E4FGnBbnHVxx66y4pe4w=', 'Content-Length': '0'} 2024-04-12T03:25:50.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3250b07dd5d5f52-006618a9be-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.592 DEBUG:boto:b'dkyges-14999falsedummyH5mOPWoRlWY-PmXX4nbTLN5Wikt68oetrue2024-04-12T03:24:46.798Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Cqfn87I7vLNs1MeceQitarGsoRdD1pEtrue2024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2N26z7de3t6pW-GMELJHxxRYWiKhUwcFtrue2024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:50.592 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:25:50.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-12T03:25:50.593 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-12T03:25:50.593 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:25:50.593 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:50.593 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:50.593 DEBUG:boto:path=/dkyges-14/?versions&max-keys=999 2024-04-12T03:25:50.593 DEBUG:boto:auth_path=/dkyges-14/?versions&max-keys=999 2024-04-12T03:25:50.593 DEBUG:boto:Method: GET 2024-04-12T03:25:50.593 DEBUG:boto:Path: /dkyges-14/?versions&max-keys=999 2024-04-12T03:25:50.593 DEBUG:boto:Data: 2024-04-12T03:25:50.593 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.593 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.594 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.594 DEBUG:boto:Params: {} 2024-04-12T03:25:50.594 DEBUG:boto:Token: None 2024-04-12T03:25:50.594 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/?versions 2024-04-12T03:25:50.594 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vsxjXn3E4FGnBbnHVxx66y4pe4w= 2024-04-12T03:25:50.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vsxjXn3E4FGnBbnHVxx66y4pe4w=', 'Content-Length': '0'} 2024-04-12T03:25:50.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1633ab20dbbd730-006618a9be-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.597 DEBUG:boto:b'dkyges-14999falsedummyH5mOPWoRlWY-PmXX4nbTLN5Wikt68oetrue2024-04-12T03:24:46.798Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Cqfn87I7vLNs1MeceQitarGsoRdD1pEtrue2024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2N26z7de3t6pW-GMELJHxxRYWiKhUwcFtrue2024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:50.597 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:25:50.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-12T03:25:50.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-12T03:25:50.598 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:50.598 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:50.598 DEBUG:boto:path=/dkyges-14/?versions&max-keys=999 2024-04-12T03:25:50.598 DEBUG:boto:auth_path=/dkyges-14/?versions&max-keys=999 2024-04-12T03:25:50.598 DEBUG:boto:Method: GET 2024-04-12T03:25:50.598 DEBUG:boto:Path: /dkyges-14/?versions&max-keys=999 2024-04-12T03:25:50.598 DEBUG:boto:Data: 2024-04-12T03:25:50.598 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.598 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.598 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.598 DEBUG:boto:Params: {} 2024-04-12T03:25:50.599 DEBUG:boto:Token: None 2024-04-12T03:25:50.599 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/?versions 2024-04-12T03:25:50.599 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vsxjXn3E4FGnBbnHVxx66y4pe4w= 2024-04-12T03:25:50.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vsxjXn3E4FGnBbnHVxx66y4pe4w=', 'Content-Length': '0'} 2024-04-12T03:25:50.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085d90d66c29ba7f6-006618a9be-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.602 DEBUG:boto:b'dkyges-14999falsedummyH5mOPWoRlWY-PmXX4nbTLN5Wikt68oetrue2024-04-12T03:24:46.798Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Cqfn87I7vLNs1MeceQitarGsoRdD1pEtrue2024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2N26z7de3t6pW-GMELJHxxRYWiKhUwcFtrue2024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:50.602 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:50.603 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:50.603 DEBUG:boto:path=/dkyges-14/?versions&max-keys=999 2024-04-12T03:25:50.603 DEBUG:boto:auth_path=/dkyges-14/?versions&max-keys=999 2024-04-12T03:25:50.603 DEBUG:boto:Method: GET 2024-04-12T03:25:50.603 DEBUG:boto:Path: /dkyges-14/?versions&max-keys=999 2024-04-12T03:25:50.603 DEBUG:boto:Data: 2024-04-12T03:25:50.603 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.603 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.603 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.603 DEBUG:boto:Params: {} 2024-04-12T03:25:50.603 DEBUG:boto:Token: None 2024-04-12T03:25:50.604 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/?versions 2024-04-12T03:25:50.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vsxjXn3E4FGnBbnHVxx66y4pe4w= 2024-04-12T03:25:50.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vsxjXn3E4FGnBbnHVxx66y4pe4w=', 'Content-Length': '0'} 2024-04-12T03:25:50.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caa37aa63b07c653-006618a9be-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.606 DEBUG:boto:b'dkyges-14999falsedummyH5mOPWoRlWY-PmXX4nbTLN5Wikt68oetrue2024-04-12T03:24:46.798Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Cqfn87I7vLNs1MeceQitarGsoRdD1pEtrue2024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2N26z7de3t6pW-GMELJHxxRYWiKhUwcFtrue2024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:50.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:25:50.607 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:25:50.607 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:25:50.607 DEBUG:boto:path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.607 DEBUG:boto:auth_path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.607 DEBUG:boto:Method: GET 2024-04-12T03:25:50.607 DEBUG:boto:Path: /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.607 DEBUG:boto:Data: 2024-04-12T03:25:50.607 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.608 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.608 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.608 DEBUG:boto:Params: {} 2024-04-12T03:25:50.608 DEBUG:boto:Token: None 2024-04-12T03:25:50.608 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.608 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CSYG9ju+qtKQqrGJp4v0kaOQPkk= 2024-04-12T03:25:50.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CSYG9ju+qtKQqrGJp4v0kaOQPkk=', 'Content-Length': '0'} 2024-04-12T03:25:50.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000070def8feb9dff115-006618a9be-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.610 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:25:50.610 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:25:50.610 DEBUG:boto:path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.610 DEBUG:boto:auth_path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.610 DEBUG:boto:Method: GET 2024-04-12T03:25:50.611 DEBUG:boto:Path: /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.611 DEBUG:boto:Data: 2024-04-12T03:25:50.611 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.611 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.611 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.611 DEBUG:boto:Params: {} 2024-04-12T03:25:50.611 DEBUG:boto:Token: None 2024-04-12T03:25:50.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.611 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CSYG9ju+qtKQqrGJp4v0kaOQPkk= 2024-04-12T03:25:50.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CSYG9ju+qtKQqrGJp4v0kaOQPkk=', 'Content-Length': '0'} 2024-04-12T03:25:50.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec521516d088a230-006618a9be-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.613 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:25:50.613 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:25:50.614 DEBUG:boto:path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.614 DEBUG:boto:auth_path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.614 DEBUG:boto:Method: HEAD 2024-04-12T03:25:50.614 DEBUG:boto:Path: /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.614 DEBUG:boto:Data: 2024-04-12T03:25:50.614 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.614 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.614 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.614 DEBUG:boto:Params: {} 2024-04-12T03:25:50.614 DEBUG:boto:Token: None 2024-04-12T03:25:50.614 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.615 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DnTHnrL6rwsqFtZ5H/at8zS7LMI= 2024-04-12T03:25:50.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DnTHnrL6rwsqFtZ5H/at8zS7LMI=', 'Content-Length': '0'} 2024-04-12T03:25:50.616 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b65852a867d6e66f-006618a9be-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.617 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:25:50.617 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:25:50.617 DEBUG:boto:path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.617 DEBUG:boto:auth_path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.617 DEBUG:boto:Method: HEAD 2024-04-12T03:25:50.617 DEBUG:boto:Path: /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.617 DEBUG:boto:Data: 2024-04-12T03:25:50.617 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.617 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.617 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.617 DEBUG:boto:Params: {} 2024-04-12T03:25:50.618 DEBUG:boto:Token: None 2024-04-12T03:25:50.618 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.618 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DnTHnrL6rwsqFtZ5H/at8zS7LMI= 2024-04-12T03:25:50.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DnTHnrL6rwsqFtZ5H/at8zS7LMI=', 'Content-Length': '0'} 2024-04-12T03:25:50.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007cfe24f42e6194a1-006618a9be-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:25:50.620 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:25:50.620 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:25:50.620 DEBUG:boto:path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.620 DEBUG:boto:auth_path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.620 DEBUG:boto:Method: GET 2024-04-12T03:25:50.620 DEBUG:boto:Path: /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.620 DEBUG:boto:Data: 2024-04-12T03:25:50.621 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.621 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.621 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.621 DEBUG:boto:Params: {} 2024-04-12T03:25:50.621 DEBUG:boto:Token: None 2024-04-12T03:25:50.621 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.621 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CSYG9ju+qtKQqrGJp4v0kaOQPkk= 2024-04-12T03:25:50.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CSYG9ju+qtKQqrGJp4v0kaOQPkk=', 'Content-Length': '0'} 2024-04-12T03:25:50.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d0dd47209b5f979f-006618a9be-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.623 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:25:50.624 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:25:50.624 DEBUG:boto:path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.624 DEBUG:boto:auth_path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.624 DEBUG:boto:Method: GET 2024-04-12T03:25:50.624 DEBUG:boto:Path: /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.624 DEBUG:boto:Data: 2024-04-12T03:25:50.624 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.624 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.624 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.624 DEBUG:boto:Params: {} 2024-04-12T03:25:50.624 DEBUG:boto:Token: None 2024-04-12T03:25:50.625 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.625 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CSYG9ju+qtKQqrGJp4v0kaOQPkk= 2024-04-12T03:25:50.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CSYG9ju+qtKQqrGJp4v0kaOQPkk=', 'Content-Length': '0'} 2024-04-12T03:25:50.627 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000070100e779c0d103b-006618a9be-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.627 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:50.627 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:50.627 DEBUG:boto:path=/dkyges-14/?prefix=dummy 2024-04-12T03:25:50.627 DEBUG:boto:auth_path=/dkyges-14/?prefix=dummy 2024-04-12T03:25:50.627 DEBUG:boto:Method: GET 2024-04-12T03:25:50.628 DEBUG:boto:Path: /dkyges-14/?prefix=dummy 2024-04-12T03:25:50.628 DEBUG:boto:Data: 2024-04-12T03:25:50.628 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.628 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.628 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.628 DEBUG:boto:Params: {} 2024-04-12T03:25:50.628 DEBUG:boto:Token: None 2024-04-12T03:25:50.628 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/ 2024-04-12T03:25:50.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0= 2024-04-12T03:25:50.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0=', 'Content-Length': '0'} 2024-04-12T03:25:50.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab65b9801956dbff-006618a9be-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.678 DEBUG:boto:b'dkyges-14dummy1000falsedummy2024-04-12T03:24:46.798Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:50.679 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:50.679 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:50.679 DEBUG:boto:path=/dkyges-14/?prefix=dummy 2024-04-12T03:25:50.679 DEBUG:boto:auth_path=/dkyges-14/?prefix=dummy 2024-04-12T03:25:50.679 DEBUG:boto:Method: GET 2024-04-12T03:25:50.679 DEBUG:boto:Path: /dkyges-14/?prefix=dummy 2024-04-12T03:25:50.679 DEBUG:boto:Data: 2024-04-12T03:25:50.680 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.680 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.680 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.680 DEBUG:boto:Params: {} 2024-04-12T03:25:50.680 DEBUG:boto:Token: None 2024-04-12T03:25:50.680 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/ 2024-04-12T03:25:50.680 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0= 2024-04-12T03:25:50.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0=', 'Content-Length': '0'} 2024-04-12T03:25:50.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d002e7df9d849d1-006618a9be-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.683 DEBUG:boto:b'dkyges-14dummy1000falsedummy2024-04-12T03:24:46.798Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:50.683 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:25:50.684 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:25:50.684 DEBUG:boto:Method: HEAD 2024-04-12T03:25:50.684 DEBUG:boto:Path: /dkyges-14/dummy 2024-04-12T03:25:50.684 DEBUG:boto:Data: 2024-04-12T03:25:50.684 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.684 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.684 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.684 DEBUG:boto:Params: {} 2024-04-12T03:25:50.684 DEBUG:boto:Token: None 2024-04-12T03:25:50.684 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/dummy 2024-04-12T03:25:50.685 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7wP79WaPJ72oK5XNJeY8iS/fMbY= 2024-04-12T03:25:50.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7wP79WaPJ72oK5XNJeY8iS/fMbY=', 'Content-Length': '0'} 2024-04-12T03:25:50.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000004b5c0fc5e4b0352-006618a9be-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.687 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:25:50.687 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:25:50.687 DEBUG:boto:Method: HEAD 2024-04-12T03:25:50.687 DEBUG:boto:Path: /dkyges-14/dummy 2024-04-12T03:25:50.687 DEBUG:boto:Data: 2024-04-12T03:25:50.688 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.688 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.688 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.688 DEBUG:boto:Params: {} 2024-04-12T03:25:50.688 DEBUG:boto:Token: None 2024-04-12T03:25:50.688 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/dummy 2024-04-12T03:25:50.688 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7wP79WaPJ72oK5XNJeY8iS/fMbY= 2024-04-12T03:25:50.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7wP79WaPJ72oK5XNJeY8iS/fMbY=', 'Content-Length': '0'} 2024-04-12T03:25:50.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005a3de50c421afe46-006618a9be-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:25:50.691 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:25:50.691 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:25:50.691 DEBUG:boto:path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.691 DEBUG:boto:auth_path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.691 DEBUG:boto:Method: GET 2024-04-12T03:25:50.691 DEBUG:boto:Path: /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.691 DEBUG:boto:Data: 2024-04-12T03:25:50.691 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.692 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.692 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.692 DEBUG:boto:Params: {} 2024-04-12T03:25:50.692 DEBUG:boto:Token: None 2024-04-12T03:25:50.692 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.692 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CSYG9ju+qtKQqrGJp4v0kaOQPkk= 2024-04-12T03:25:50.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CSYG9ju+qtKQqrGJp4v0kaOQPkk=', 'Content-Length': '0'} 2024-04-12T03:25:50.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ca9fed8168c74c43-006618a9be-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.694 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:25:50.695 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:25:50.695 DEBUG:boto:path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.695 DEBUG:boto:auth_path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.695 DEBUG:boto:Method: GET 2024-04-12T03:25:50.695 DEBUG:boto:Path: /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.695 DEBUG:boto:Data: 2024-04-12T03:25:50.695 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.695 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.695 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.695 DEBUG:boto:Params: {} 2024-04-12T03:25:50.695 DEBUG:boto:Token: None 2024-04-12T03:25:50.696 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:25:50.696 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CSYG9ju+qtKQqrGJp4v0kaOQPkk= 2024-04-12T03:25:50.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CSYG9ju+qtKQqrGJp4v0kaOQPkk=', 'Content-Length': '0'} 2024-04-12T03:25:50.698 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000936a62dc0d598a58-006618a9be-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.698 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:50.698 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:50.698 DEBUG:boto:path=/dkyges-14/?prefix=dummy 2024-04-12T03:25:50.698 DEBUG:boto:auth_path=/dkyges-14/?prefix=dummy 2024-04-12T03:25:50.698 DEBUG:boto:Method: GET 2024-04-12T03:25:50.698 DEBUG:boto:Path: /dkyges-14/?prefix=dummy 2024-04-12T03:25:50.698 DEBUG:boto:Data: 2024-04-12T03:25:50.698 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.699 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.699 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.699 DEBUG:boto:Params: {} 2024-04-12T03:25:50.699 DEBUG:boto:Token: None 2024-04-12T03:25:50.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/ 2024-04-12T03:25:50.699 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0= 2024-04-12T03:25:50.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0=', 'Content-Length': '0'} 2024-04-12T03:25:50.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c80b03f2f637daec-006618a9be-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.701 DEBUG:boto:b'dkyges-14dummy1000falsedummy2024-04-12T03:24:46.798Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:50.702 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:50.702 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:50.702 DEBUG:boto:path=/dkyges-14/?prefix=dummy 2024-04-12T03:25:50.702 DEBUG:boto:auth_path=/dkyges-14/?prefix=dummy 2024-04-12T03:25:50.702 DEBUG:boto:Method: GET 2024-04-12T03:25:50.702 DEBUG:boto:Path: /dkyges-14/?prefix=dummy 2024-04-12T03:25:50.702 DEBUG:boto:Data: 2024-04-12T03:25:50.703 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.703 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.703 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.703 DEBUG:boto:Params: {} 2024-04-12T03:25:50.703 DEBUG:boto:Token: None 2024-04-12T03:25:50.703 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/ 2024-04-12T03:25:50.703 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0= 2024-04-12T03:25:50.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0=', 'Content-Length': '0'} 2024-04-12T03:25:50.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceac5f3bf2929fd3-006618a9be-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.746 DEBUG:boto:b'dkyges-14dummy1000falsedummy2024-04-12T03:24:46.798Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:50.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:25:50.747 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:25:50.747 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:25:50.747 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.747 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.747 DEBUG:boto:Method: GET 2024-04-12T03:25:50.747 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.747 DEBUG:boto:Data: 2024-04-12T03:25:50.747 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.748 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.748 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.748 DEBUG:boto:Params: {} 2024-04-12T03:25:50.748 DEBUG:boto:Token: None 2024-04-12T03:25:50.748 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.748 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zdkCBYYNRZd2YTiIU69Ta2BB04w= 2024-04-12T03:25:50.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zdkCBYYNRZd2YTiIU69Ta2BB04w=', 'Content-Length': '0'} 2024-04-12T03:25:50.750 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2f197495a442314-006618a9be-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.750 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:25:50.751 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:25:50.751 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.751 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.751 DEBUG:boto:Method: GET 2024-04-12T03:25:50.751 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.751 DEBUG:boto:Data: 2024-04-12T03:25:50.751 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.751 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.751 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.751 DEBUG:boto:Params: {} 2024-04-12T03:25:50.751 DEBUG:boto:Token: None 2024-04-12T03:25:50.752 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.752 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zdkCBYYNRZd2YTiIU69Ta2BB04w= 2024-04-12T03:25:50.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zdkCBYYNRZd2YTiIU69Ta2BB04w=', 'Content-Length': '0'} 2024-04-12T03:25:50.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000042b5e215db45a5bc-006618a9be-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.754 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:25:50.754 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:25:50.754 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.754 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.754 DEBUG:boto:Method: HEAD 2024-04-12T03:25:50.754 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.754 DEBUG:boto:Data: 2024-04-12T03:25:50.754 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.754 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.754 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.755 DEBUG:boto:Params: {} 2024-04-12T03:25:50.755 DEBUG:boto:Token: None 2024-04-12T03:25:50.755 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.755 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZHieT5WUZ8lCX0zkQwQlWJjXb9E= 2024-04-12T03:25:50.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZHieT5WUZ8lCX0zkQwQlWJjXb9E=', 'Content-Length': '0'} 2024-04-12T03:25:50.757 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f822efb41ac636b0-006618a9be-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.757 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:25:50.757 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:25:50.757 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.757 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.757 DEBUG:boto:Method: HEAD 2024-04-12T03:25:50.757 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.757 DEBUG:boto:Data: 2024-04-12T03:25:50.758 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.758 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.758 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.758 DEBUG:boto:Params: {} 2024-04-12T03:25:50.758 DEBUG:boto:Token: None 2024-04-12T03:25:50.758 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.758 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZHieT5WUZ8lCX0zkQwQlWJjXb9E= 2024-04-12T03:25:50.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZHieT5WUZ8lCX0zkQwQlWJjXb9E=', 'Content-Length': '0'} 2024-04-12T03:25:50.760 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f0f876094c3cc48e-006618a9be-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:25:50.760 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:25:50.760 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:25:50.760 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.761 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.761 DEBUG:boto:Method: GET 2024-04-12T03:25:50.761 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.761 DEBUG:boto:Data: 2024-04-12T03:25:50.761 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.761 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.761 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.761 DEBUG:boto:Params: {} 2024-04-12T03:25:50.761 DEBUG:boto:Token: None 2024-04-12T03:25:50.761 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.761 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zdkCBYYNRZd2YTiIU69Ta2BB04w= 2024-04-12T03:25:50.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zdkCBYYNRZd2YTiIU69Ta2BB04w=', 'Content-Length': '0'} 2024-04-12T03:25:50.763 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a4c58b82b786b75-006618a9be-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.806 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:25:50.806 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:25:50.807 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.807 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.807 DEBUG:boto:Method: GET 2024-04-12T03:25:50.807 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.807 DEBUG:boto:Data: 2024-04-12T03:25:50.807 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.807 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.807 DEBUG:boto:Params: {} 2024-04-12T03:25:50.807 DEBUG:boto:Token: None 2024-04-12T03:25:50.807 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.808 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zdkCBYYNRZd2YTiIU69Ta2BB04w= 2024-04-12T03:25:50.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zdkCBYYNRZd2YTiIU69Ta2BB04w=', 'Content-Length': '0'} 2024-04-12T03:25:50.810 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046da0fae91102336-006618a9be-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.810 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:50.810 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:50.810 DEBUG:boto:path=/dkyges-14/?prefix=obj-a1 2024-04-12T03:25:50.810 DEBUG:boto:auth_path=/dkyges-14/?prefix=obj-a1 2024-04-12T03:25:50.810 DEBUG:boto:Method: GET 2024-04-12T03:25:50.810 DEBUG:boto:Path: /dkyges-14/?prefix=obj-a1 2024-04-12T03:25:50.810 DEBUG:boto:Data: 2024-04-12T03:25:50.811 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.811 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.811 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.811 DEBUG:boto:Params: {} 2024-04-12T03:25:50.811 DEBUG:boto:Token: None 2024-04-12T03:25:50.811 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/ 2024-04-12T03:25:50.811 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0= 2024-04-12T03:25:50.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0=', 'Content-Length': '0'} 2024-04-12T03:25:50.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbb7863a2cbf8a26-006618a9be-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.814 DEBUG:boto:b'dkyges-14obj-a11000falseobj-a12024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:50.814 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:50.814 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:50.814 DEBUG:boto:path=/dkyges-14/?prefix=obj-a1 2024-04-12T03:25:50.814 DEBUG:boto:auth_path=/dkyges-14/?prefix=obj-a1 2024-04-12T03:25:50.814 DEBUG:boto:Method: GET 2024-04-12T03:25:50.815 DEBUG:boto:Path: /dkyges-14/?prefix=obj-a1 2024-04-12T03:25:50.815 DEBUG:boto:Data: 2024-04-12T03:25:50.815 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.815 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.815 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.815 DEBUG:boto:Params: {} 2024-04-12T03:25:50.815 DEBUG:boto:Token: None 2024-04-12T03:25:50.815 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/ 2024-04-12T03:25:50.815 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0= 2024-04-12T03:25:50.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0=', 'Content-Length': '0'} 2024-04-12T03:25:50.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec22585a55279e02-006618a9be-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.818 DEBUG:boto:b'dkyges-14obj-a11000falseobj-a12024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:50.818 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:25:50.818 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:25:50.818 DEBUG:boto:Method: HEAD 2024-04-12T03:25:50.819 DEBUG:boto:Path: /dkyges-14/obj-a1 2024-04-12T03:25:50.819 DEBUG:boto:Data: 2024-04-12T03:25:50.819 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.819 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.819 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.819 DEBUG:boto:Params: {} 2024-04-12T03:25:50.819 DEBUG:boto:Token: None 2024-04-12T03:25:50.819 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/obj-a1 2024-04-12T03:25:50.819 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UQUC0bnZZD0xwesltm64MaH2f5A= 2024-04-12T03:25:50.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UQUC0bnZZD0xwesltm64MaH2f5A=', 'Content-Length': '0'} 2024-04-12T03:25:50.822 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da489435bcd5ea00-006618a9be-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.822 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:25:50.822 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:25:50.822 DEBUG:boto:Method: HEAD 2024-04-12T03:25:50.822 DEBUG:boto:Path: /dkyges-14/obj-a1 2024-04-12T03:25:50.822 DEBUG:boto:Data: 2024-04-12T03:25:50.822 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.822 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.822 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.823 DEBUG:boto:Params: {} 2024-04-12T03:25:50.823 DEBUG:boto:Token: None 2024-04-12T03:25:50.823 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/obj-a1 2024-04-12T03:25:50.823 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UQUC0bnZZD0xwesltm64MaH2f5A= 2024-04-12T03:25:50.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UQUC0bnZZD0xwesltm64MaH2f5A=', 'Content-Length': '0'} 2024-04-12T03:25:50.825 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ea6caedb60fda7b-006618a9be-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:25:50.825 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:25:50.826 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:25:50.826 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.826 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.826 DEBUG:boto:Method: GET 2024-04-12T03:25:50.826 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.826 DEBUG:boto:Data: 2024-04-12T03:25:50.826 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.826 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.826 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.826 DEBUG:boto:Params: {} 2024-04-12T03:25:50.826 DEBUG:boto:Token: None 2024-04-12T03:25:50.827 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.827 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zdkCBYYNRZd2YTiIU69Ta2BB04w= 2024-04-12T03:25:50.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zdkCBYYNRZd2YTiIU69Ta2BB04w=', 'Content-Length': '0'} 2024-04-12T03:25:50.829 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000078ffa08dc90e95c-006618a9be-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.870 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:25:50.871 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:25:50.871 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.871 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.871 DEBUG:boto:Method: GET 2024-04-12T03:25:50.871 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.871 DEBUG:boto:Data: 2024-04-12T03:25:50.871 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.871 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.871 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.871 DEBUG:boto:Params: {} 2024-04-12T03:25:50.871 DEBUG:boto:Token: None 2024-04-12T03:25:50.872 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:25:50.872 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zdkCBYYNRZd2YTiIU69Ta2BB04w= 2024-04-12T03:25:50.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zdkCBYYNRZd2YTiIU69Ta2BB04w=', 'Content-Length': '0'} 2024-04-12T03:25:50.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004a19579363251dbd-006618a9be-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.874 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:50.874 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:50.874 DEBUG:boto:path=/dkyges-14/?prefix=obj-a1 2024-04-12T03:25:50.874 DEBUG:boto:auth_path=/dkyges-14/?prefix=obj-a1 2024-04-12T03:25:50.874 DEBUG:boto:Method: GET 2024-04-12T03:25:50.875 DEBUG:boto:Path: /dkyges-14/?prefix=obj-a1 2024-04-12T03:25:50.875 DEBUG:boto:Data: 2024-04-12T03:25:50.875 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.875 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.875 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.875 DEBUG:boto:Params: {} 2024-04-12T03:25:50.875 DEBUG:boto:Token: None 2024-04-12T03:25:50.875 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/ 2024-04-12T03:25:50.875 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0= 2024-04-12T03:25:50.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0=', 'Content-Length': '0'} 2024-04-12T03:25:50.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b158362fe73cfcf9-006618a9be-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.878 DEBUG:boto:b'dkyges-14obj-a11000falseobj-a12024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:50.878 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:50.879 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:50.879 DEBUG:boto:path=/dkyges-14/?prefix=obj-a1 2024-04-12T03:25:50.879 DEBUG:boto:auth_path=/dkyges-14/?prefix=obj-a1 2024-04-12T03:25:50.879 DEBUG:boto:Method: GET 2024-04-12T03:25:50.879 DEBUG:boto:Path: /dkyges-14/?prefix=obj-a1 2024-04-12T03:25:50.879 DEBUG:boto:Data: 2024-04-12T03:25:50.879 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.879 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.879 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.880 DEBUG:boto:Params: {} 2024-04-12T03:25:50.880 DEBUG:boto:Token: None 2024-04-12T03:25:50.880 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/ 2024-04-12T03:25:50.880 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0= 2024-04-12T03:25:50.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0=', 'Content-Length': '0'} 2024-04-12T03:25:50.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ba929444f04bbcb-006618a9be-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.883 DEBUG:boto:b'dkyges-14obj-a11000falseobj-a12024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:50.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:25:50.883 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:25:50.883 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:25:50.883 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.883 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.883 DEBUG:boto:Method: GET 2024-04-12T03:25:50.883 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.884 DEBUG:boto:Data: 2024-04-12T03:25:50.884 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.884 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.884 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.884 DEBUG:boto:Params: {} 2024-04-12T03:25:50.884 DEBUG:boto:Token: None 2024-04-12T03:25:50.884 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.884 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sP9YwHXsQsGsldAfTIcJgwDOPy4= 2024-04-12T03:25:50.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sP9YwHXsQsGsldAfTIcJgwDOPy4=', 'Content-Length': '0'} 2024-04-12T03:25:50.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000195b94cd319e8790-006618a9be-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.886 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:25:50.887 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:25:50.887 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.887 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.887 DEBUG:boto:Method: GET 2024-04-12T03:25:50.887 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.887 DEBUG:boto:Data: 2024-04-12T03:25:50.887 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.887 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.887 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.887 DEBUG:boto:Params: {} 2024-04-12T03:25:50.887 DEBUG:boto:Token: None 2024-04-12T03:25:50.888 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.888 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sP9YwHXsQsGsldAfTIcJgwDOPy4= 2024-04-12T03:25:50.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sP9YwHXsQsGsldAfTIcJgwDOPy4=', 'Content-Length': '0'} 2024-04-12T03:25:50.890 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a391e10dee57ab5e-006618a9be-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.890 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:25:50.890 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:25:50.890 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.890 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.890 DEBUG:boto:Method: HEAD 2024-04-12T03:25:50.890 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.890 DEBUG:boto:Data: 2024-04-12T03:25:50.891 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.891 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.891 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.891 DEBUG:boto:Params: {} 2024-04-12T03:25:50.891 DEBUG:boto:Token: None 2024-04-12T03:25:50.891 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.891 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yddD4zeH2Z0CUv9xpbJouaK21oI= 2024-04-12T03:25:50.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yddD4zeH2Z0CUv9xpbJouaK21oI=', 'Content-Length': '0'} 2024-04-12T03:25:50.893 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c8b8eee8a7cb18d-006618a9be-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.893 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:25:50.893 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:25:50.893 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.893 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.893 DEBUG:boto:Method: HEAD 2024-04-12T03:25:50.894 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.894 DEBUG:boto:Data: 2024-04-12T03:25:50.894 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.894 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.894 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.894 DEBUG:boto:Params: {} 2024-04-12T03:25:50.894 DEBUG:boto:Token: None 2024-04-12T03:25:50.894 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.894 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yddD4zeH2Z0CUv9xpbJouaK21oI= 2024-04-12T03:25:50.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yddD4zeH2Z0CUv9xpbJouaK21oI=', 'Content-Length': '0'} 2024-04-12T03:25:50.896 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0d908506e00ff7b-006618a9be-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:25:50.897 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:25:50.897 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:25:50.897 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.897 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.897 DEBUG:boto:Method: GET 2024-04-12T03:25:50.897 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.897 DEBUG:boto:Data: 2024-04-12T03:25:50.897 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.897 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.897 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.897 DEBUG:boto:Params: {} 2024-04-12T03:25:50.898 DEBUG:boto:Token: None 2024-04-12T03:25:50.898 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.898 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sP9YwHXsQsGsldAfTIcJgwDOPy4= 2024-04-12T03:25:50.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sP9YwHXsQsGsldAfTIcJgwDOPy4=', 'Content-Length': '0'} 2024-04-12T03:25:50.900 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000932c5bbd3dc6dfc8-006618a9be-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.942 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:25:50.943 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:25:50.943 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.943 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.943 DEBUG:boto:Method: GET 2024-04-12T03:25:50.943 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.943 DEBUG:boto:Data: 2024-04-12T03:25:50.943 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.943 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.943 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.943 DEBUG:boto:Params: {} 2024-04-12T03:25:50.944 DEBUG:boto:Token: None 2024-04-12T03:25:50.944 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:50.944 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sP9YwHXsQsGsldAfTIcJgwDOPy4= 2024-04-12T03:25:50.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sP9YwHXsQsGsldAfTIcJgwDOPy4=', 'Content-Length': '0'} 2024-04-12T03:25:50.946 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000770894564c64e6d8-006618a9be-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.986 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:50.987 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:50.987 DEBUG:boto:path=/dkyges-14/?prefix=obj-a2 2024-04-12T03:25:50.987 DEBUG:boto:auth_path=/dkyges-14/?prefix=obj-a2 2024-04-12T03:25:50.987 DEBUG:boto:Method: GET 2024-04-12T03:25:50.987 DEBUG:boto:Path: /dkyges-14/?prefix=obj-a2 2024-04-12T03:25:50.987 DEBUG:boto:Data: 2024-04-12T03:25:50.987 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.987 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:50.987 DEBUG:boto:Port: 8001 2024-04-12T03:25:50.987 DEBUG:boto:Params: {} 2024-04-12T03:25:50.988 DEBUG:boto:Token: None 2024-04-12T03:25:50.988 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/ 2024-04-12T03:25:50.988 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0= 2024-04-12T03:25:50.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0=', 'Content-Length': '0'} 2024-04-12T03:25:50.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ed3a380f5d90a07-006618a9be-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:50.991 DEBUG:boto:b'dkyges-14obj-a21000falseobj-a22024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:50.991 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:50.991 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:50.991 DEBUG:boto:path=/dkyges-14/?prefix=obj-a2 2024-04-12T03:25:50.991 DEBUG:boto:auth_path=/dkyges-14/?prefix=obj-a2 2024-04-12T03:25:50.991 DEBUG:boto:Method: GET 2024-04-12T03:25:50.992 DEBUG:boto:Path: /dkyges-14/?prefix=obj-a2 2024-04-12T03:25:50.992 DEBUG:boto:Data: 2024-04-12T03:25:50.992 DEBUG:boto:Headers: {} 2024-04-12T03:25:50.992 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:50.992 DEBUG:boto:Port: 8000 2024-04-12T03:25:50.992 DEBUG:boto:Params: {} 2024-04-12T03:25:50.992 DEBUG:boto:Token: None 2024-04-12T03:25:50.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:50 GMT /dkyges-14/ 2024-04-12T03:25:50.992 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0= 2024-04-12T03:25:50.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EX0VKqab8uvYwh1dskCpQgUqDv0=', 'Content-Length': '0'} 2024-04-12T03:25:50.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdef2d1feb3dcef9-006618a9be-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.038 DEBUG:boto:b'dkyges-14obj-a21000falseobj-a22024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:51.039 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:25:51.039 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:25:51.039 DEBUG:boto:Method: HEAD 2024-04-12T03:25:51.039 DEBUG:boto:Path: /dkyges-14/obj-a2 2024-04-12T03:25:51.039 DEBUG:boto:Data: 2024-04-12T03:25:51.039 DEBUG:boto:Headers: {} 2024-04-12T03:25:51.039 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:51.040 DEBUG:boto:Port: 8001 2024-04-12T03:25:51.040 DEBUG:boto:Params: {} 2024-04-12T03:25:51.040 DEBUG:boto:Token: None 2024-04-12T03:25:51.040 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:51 GMT /dkyges-14/obj-a2 2024-04-12T03:25:51.040 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7YGxATSytv7VwMSIacSq4YbTOOM= 2024-04-12T03:25:51.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7YGxATSytv7VwMSIacSq4YbTOOM=', 'Content-Length': '0'} 2024-04-12T03:25:51.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c493bf506da0212-006618a9bf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.043 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:25:51.044 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:25:51.044 DEBUG:boto:Method: HEAD 2024-04-12T03:25:51.044 DEBUG:boto:Path: /dkyges-14/obj-a2 2024-04-12T03:25:51.044 DEBUG:boto:Data: 2024-04-12T03:25:51.044 DEBUG:boto:Headers: {} 2024-04-12T03:25:51.044 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:51.044 DEBUG:boto:Port: 8000 2024-04-12T03:25:51.044 DEBUG:boto:Params: {} 2024-04-12T03:25:51.044 DEBUG:boto:Token: None 2024-04-12T03:25:51.044 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:51 GMT /dkyges-14/obj-a2 2024-04-12T03:25:51.044 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7YGxATSytv7VwMSIacSq4YbTOOM= 2024-04-12T03:25:51.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7YGxATSytv7VwMSIacSq4YbTOOM=', 'Content-Length': '0'} 2024-04-12T03:25:51.047 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000001cd5f631d2d7f1d-006618a9bf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:25:51.047 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:25:51.047 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:25:51.048 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:51.048 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:51.048 DEBUG:boto:Method: GET 2024-04-12T03:25:51.048 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:51.048 DEBUG:boto:Data: 2024-04-12T03:25:51.048 DEBUG:boto:Headers: {} 2024-04-12T03:25:51.048 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:51.049 DEBUG:boto:Port: 8001 2024-04-12T03:25:51.049 DEBUG:boto:Params: {} 2024-04-12T03:25:51.049 DEBUG:boto:Token: None 2024-04-12T03:25:51.049 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:51 GMT /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:51.049 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d2AbQgvCaAfeL7VnUoL/tbR1eX0= 2024-04-12T03:25:51.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d2AbQgvCaAfeL7VnUoL/tbR1eX0=', 'Content-Length': '0'} 2024-04-12T03:25:51.051 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dcde05c6f525f390-006618a9bf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.051 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:25:51.051 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:25:51.051 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:51.051 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:51.051 DEBUG:boto:Method: GET 2024-04-12T03:25:51.052 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:51.052 DEBUG:boto:Data: 2024-04-12T03:25:51.052 DEBUG:boto:Headers: {} 2024-04-12T03:25:51.052 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:51.052 DEBUG:boto:Port: 8000 2024-04-12T03:25:51.052 DEBUG:boto:Params: {} 2024-04-12T03:25:51.052 DEBUG:boto:Token: None 2024-04-12T03:25:51.052 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:51 GMT /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:25:51.052 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d2AbQgvCaAfeL7VnUoL/tbR1eX0= 2024-04-12T03:25:51.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d2AbQgvCaAfeL7VnUoL/tbR1eX0=', 'Content-Length': '0'} 2024-04-12T03:25:51.054 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000670114e4540267cd-006618a9bf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.102 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:51.102 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:51.103 DEBUG:boto:path=/dkyges-14/?prefix=obj-a2 2024-04-12T03:25:51.103 DEBUG:boto:auth_path=/dkyges-14/?prefix=obj-a2 2024-04-12T03:25:51.103 DEBUG:boto:Method: GET 2024-04-12T03:25:51.103 DEBUG:boto:Path: /dkyges-14/?prefix=obj-a2 2024-04-12T03:25:51.103 DEBUG:boto:Data: 2024-04-12T03:25:51.103 DEBUG:boto:Headers: {} 2024-04-12T03:25:51.103 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:51.103 DEBUG:boto:Port: 8001 2024-04-12T03:25:51.103 DEBUG:boto:Params: {} 2024-04-12T03:25:51.104 DEBUG:boto:Token: None 2024-04-12T03:25:51.104 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:51 GMT /dkyges-14/ 2024-04-12T03:25:51.104 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:deb/U3yJLb5f14+ReO67LZDJW94= 2024-04-12T03:25:51.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:deb/U3yJLb5f14+ReO67LZDJW94=', 'Content-Length': '0'} 2024-04-12T03:25:51.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000093239acdc087e17-006618a9bf-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.106 DEBUG:boto:b'dkyges-14obj-a21000falseobj-a22024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:51.107 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:51.107 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:51.107 DEBUG:boto:path=/dkyges-14/?prefix=obj-a2 2024-04-12T03:25:51.107 DEBUG:boto:auth_path=/dkyges-14/?prefix=obj-a2 2024-04-12T03:25:51.107 DEBUG:boto:Method: GET 2024-04-12T03:25:51.108 DEBUG:boto:Path: /dkyges-14/?prefix=obj-a2 2024-04-12T03:25:51.108 DEBUG:boto:Data: 2024-04-12T03:25:51.108 DEBUG:boto:Headers: {} 2024-04-12T03:25:51.108 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:51.108 DEBUG:boto:Port: 8000 2024-04-12T03:25:51.108 DEBUG:boto:Params: {} 2024-04-12T03:25:51.108 DEBUG:boto:Token: None 2024-04-12T03:25:51.108 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:51 GMT /dkyges-14/ 2024-04-12T03:25:51.108 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:deb/U3yJLb5f14+ReO67LZDJW94= 2024-04-12T03:25:51.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:deb/U3yJLb5f14+ReO67LZDJW94=', 'Content-Length': '0'} 2024-04-12T03:25:51.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed4ffc87075bd994-006618a9bf-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.158 DEBUG:boto:b'dkyges-14obj-a21000falseobj-a22024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:51.159 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-14 zones={a2, a1} 2024-04-12T03:25:51.159 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:51.159 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:51.159 DEBUG:boto:Method: HEAD 2024-04-12T03:25:51.159 DEBUG:boto:Path: /dkyges-13/ 2024-04-12T03:25:51.159 DEBUG:boto:Data: 2024-04-12T03:25:51.159 DEBUG:boto:Headers: {} 2024-04-12T03:25:51.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:51.159 DEBUG:boto:Port: 8000 2024-04-12T03:25:51.159 DEBUG:boto:Params: {} 2024-04-12T03:25:51.160 DEBUG:boto:Token: None 2024-04-12T03:25:51.160 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:51 GMT /dkyges-13/ 2024-04-12T03:25:51.160 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H/F+BQ3YdXF+815B6FrEGTl5RSI= 2024-04-12T03:25:51.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H/F+BQ3YdXF+815B6FrEGTl5RSI=', 'Content-Length': '0'} 2024-04-12T03:25:51.163 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017df0b4c8941c3bf-006618a9bf-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.163 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:51.163 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:51.163 DEBUG:boto:path=/dkyges-13/?versions&max-keys=999&prefix=obj-a1 2024-04-12T03:25:51.163 DEBUG:boto:auth_path=/dkyges-13/?versions&max-keys=999&prefix=obj-a1 2024-04-12T03:25:51.163 DEBUG:boto:Method: GET 2024-04-12T03:25:51.163 DEBUG:boto:Path: /dkyges-13/?versions&max-keys=999&prefix=obj-a1 2024-04-12T03:25:51.163 DEBUG:boto:Data: 2024-04-12T03:25:51.164 DEBUG:boto:Headers: {} 2024-04-12T03:25:51.164 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:51.164 DEBUG:boto:Port: 8000 2024-04-12T03:25:51.164 DEBUG:boto:Params: {} 2024-04-12T03:25:51.164 DEBUG:boto:Token: None 2024-04-12T03:25:51.164 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:51 GMT /dkyges-13/?versions 2024-04-12T03:25:51.164 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:c1ejy7dA7FCMJKidvMDXsMnEzCA= 2024-04-12T03:25:51.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:c1ejy7dA7FCMJKidvMDXsMnEzCA=', 'Content-Length': '0'} 2024-04-12T03:25:51.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eadb67b0a97f0d8e-006618a9bf-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.167 DEBUG:boto:b'dkyges-13obj-a1999falseobj-a1No.FwkW62CECdIaWakbZwZXAPvIeOmOtrue2024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:51.167 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:25:51.167 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:25:51.167 DEBUG:boto:path=/dkyges-13/obj-a1?acl 2024-04-12T03:25:51.168 DEBUG:boto:auth_path=/dkyges-13/obj-a1?acl 2024-04-12T03:25:51.168 DEBUG:boto:Method: PUT 2024-04-12T03:25:51.168 DEBUG:boto:Path: /dkyges-13/obj-a1?acl 2024-04-12T03:25:51.168 DEBUG:boto:Data: 2024-04-12T03:25:51.168 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-12T03:25:51.168 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:51.168 DEBUG:boto:Port: 8000 2024-04-12T03:25:51.168 DEBUG:boto:Params: {} 2024-04-12T03:25:51.168 DEBUG:boto:Token: None 2024-04-12T03:25:51.168 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:51 GMT x-amz-acl:public-read /dkyges-13/obj-a1?acl 2024-04-12T03:25:51.169 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Phae7XbqkmWzSc0JzF5OpyX16q8= 2024-04-12T03:25:51.169 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Phae7XbqkmWzSc0JzF5OpyX16q8=', 'Content-Length': '0'} 2024-04-12T03:25:51.176 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000579281b5c8bb0108-006618a9bf-110f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.176 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:51.176 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:51.176 DEBUG:boto:Method: HEAD 2024-04-12T03:25:51.176 DEBUG:boto:Path: /dkyges-13/ 2024-04-12T03:25:51.176 DEBUG:boto:Data: 2024-04-12T03:25:51.176 DEBUG:boto:Headers: {} 2024-04-12T03:25:51.176 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:51.176 DEBUG:boto:Port: 8001 2024-04-12T03:25:51.176 DEBUG:boto:Params: {} 2024-04-12T03:25:51.177 DEBUG:boto:Token: None 2024-04-12T03:25:51.177 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:51 GMT /dkyges-13/ 2024-04-12T03:25:51.177 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H/F+BQ3YdXF+815B6FrEGTl5RSI= 2024-04-12T03:25:51.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H/F+BQ3YdXF+815B6FrEGTl5RSI=', 'Content-Length': '0'} 2024-04-12T03:25:51.179 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009daabcc1f1f44358-006618a9bf-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.179 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:25:51.180 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:25:51.180 DEBUG:boto:path=/dkyges-13/?versions&max-keys=999&prefix=obj-a2 2024-04-12T03:25:51.180 DEBUG:boto:auth_path=/dkyges-13/?versions&max-keys=999&prefix=obj-a2 2024-04-12T03:25:51.180 DEBUG:boto:Method: GET 2024-04-12T03:25:51.180 DEBUG:boto:Path: /dkyges-13/?versions&max-keys=999&prefix=obj-a2 2024-04-12T03:25:51.180 DEBUG:boto:Data: 2024-04-12T03:25:51.180 DEBUG:boto:Headers: {} 2024-04-12T03:25:51.180 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:51.180 DEBUG:boto:Port: 8001 2024-04-12T03:25:51.180 DEBUG:boto:Params: {} 2024-04-12T03:25:51.180 DEBUG:boto:Token: None 2024-04-12T03:25:51.181 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:51 GMT /dkyges-13/?versions 2024-04-12T03:25:51.181 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:c1ejy7dA7FCMJKidvMDXsMnEzCA= 2024-04-12T03:25:51.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:c1ejy7dA7FCMJKidvMDXsMnEzCA=', 'Content-Length': '0'} 2024-04-12T03:25:51.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000117e2ec020661717-006618a9bf-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.230 DEBUG:boto:b'dkyges-13obj-a2999falseobj-a29EpHGMR9P8CQ61qU0DObgl0ylooJjuatrue2024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:51.231 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:25:51.231 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:25:51.231 DEBUG:boto:path=/dkyges-13/obj-a2?acl 2024-04-12T03:25:51.231 DEBUG:boto:auth_path=/dkyges-13/obj-a2?acl 2024-04-12T03:25:51.231 DEBUG:boto:Method: PUT 2024-04-12T03:25:51.231 DEBUG:boto:Path: /dkyges-13/obj-a2?acl 2024-04-12T03:25:51.231 DEBUG:boto:Data: 2024-04-12T03:25:51.231 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-12T03:25:51.231 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:51.231 DEBUG:boto:Port: 8001 2024-04-12T03:25:51.232 DEBUG:boto:Params: {} 2024-04-12T03:25:51.232 DEBUG:boto:Token: None 2024-04-12T03:25:51.232 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:51 GMT x-amz-acl:public-read /dkyges-13/obj-a2?acl 2024-04-12T03:25:51.232 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5EvCyEWyZVnDI46vJhnSmALKfig= 2024-04-12T03:25:51.232 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5EvCyEWyZVnDI46vJhnSmALKfig=', 'Content-Length': '0'} 2024-04-12T03:25:51.239 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000018efd90a716577c3-006618a9bf-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.239 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:51.239 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:51.240 DEBUG:boto:Method: HEAD 2024-04-12T03:25:51.240 DEBUG:boto:Path: /dkyges-14/ 2024-04-12T03:25:51.240 DEBUG:boto:Data: 2024-04-12T03:25:51.240 DEBUG:boto:Headers: {} 2024-04-12T03:25:51.240 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:51.240 DEBUG:boto:Port: 8000 2024-04-12T03:25:51.240 DEBUG:boto:Params: {} 2024-04-12T03:25:51.240 DEBUG:boto:Token: None 2024-04-12T03:25:51.241 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:51 GMT /dkyges-14/ 2024-04-12T03:25:51.241 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P3pIthtqNfOeOGVYGRtqRDaWWfw= 2024-04-12T03:25:51.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P3pIthtqNfOeOGVYGRtqRDaWWfw=', 'Content-Length': '0'} 2024-04-12T03:25:51.243 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d9078adb985b556-006618a9bf-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.243 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:51.243 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:51.244 DEBUG:boto:path=/dkyges-14/?versions&max-keys=999&prefix=obj-a1 2024-04-12T03:25:51.244 DEBUG:boto:auth_path=/dkyges-14/?versions&max-keys=999&prefix=obj-a1 2024-04-12T03:25:51.244 DEBUG:boto:Method: GET 2024-04-12T03:25:51.244 DEBUG:boto:Path: /dkyges-14/?versions&max-keys=999&prefix=obj-a1 2024-04-12T03:25:51.244 DEBUG:boto:Data: 2024-04-12T03:25:51.244 DEBUG:boto:Headers: {} 2024-04-12T03:25:51.244 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:51.244 DEBUG:boto:Port: 8000 2024-04-12T03:25:51.244 DEBUG:boto:Params: {} 2024-04-12T03:25:51.244 DEBUG:boto:Token: None 2024-04-12T03:25:51.245 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:51 GMT /dkyges-14/?versions 2024-04-12T03:25:51.245 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XteyuN++L7K6hDiWgdkLNpE0xWc= 2024-04-12T03:25:51.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XteyuN++L7K6hDiWgdkLNpE0xWc=', 'Content-Length': '0'} 2024-04-12T03:25:51.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba5d1eca85af5262-006618a9bf-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.247 DEBUG:boto:b'dkyges-14obj-a1999falseobj-a1Cqfn87I7vLNs1MeceQitarGsoRdD1pEtrue2024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:51.247 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:25:51.248 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:25:51.248 DEBUG:boto:path=/dkyges-14/obj-a1?acl 2024-04-12T03:25:51.248 DEBUG:boto:auth_path=/dkyges-14/obj-a1?acl 2024-04-12T03:25:51.248 DEBUG:boto:Method: PUT 2024-04-12T03:25:51.248 DEBUG:boto:Path: /dkyges-14/obj-a1?acl 2024-04-12T03:25:51.248 DEBUG:boto:Data: 2024-04-12T03:25:51.248 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-12T03:25:51.248 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:25:51.248 DEBUG:boto:Port: 8000 2024-04-12T03:25:51.248 DEBUG:boto:Params: {} 2024-04-12T03:25:51.248 DEBUG:boto:Token: None 2024-04-12T03:25:51.249 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:51 GMT x-amz-acl:public-read /dkyges-14/obj-a1?acl 2024-04-12T03:25:51.249 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PiMqm3lJRmxd90o2CVnekEC47bA= 2024-04-12T03:25:51.249 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PiMqm3lJRmxd90o2CVnekEC47bA=', 'Content-Length': '0'} 2024-04-12T03:25:51.255 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000090c5f1fca40e8887-006618a9bf-110f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.256 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:51.256 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:51.256 DEBUG:boto:Method: HEAD 2024-04-12T03:25:51.256 DEBUG:boto:Path: /dkyges-14/ 2024-04-12T03:25:51.256 DEBUG:boto:Data: 2024-04-12T03:25:51.256 DEBUG:boto:Headers: {} 2024-04-12T03:25:51.256 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:51.256 DEBUG:boto:Port: 8001 2024-04-12T03:25:51.256 DEBUG:boto:Params: {} 2024-04-12T03:25:51.256 DEBUG:boto:Token: None 2024-04-12T03:25:51.257 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:25:51 GMT /dkyges-14/ 2024-04-12T03:25:51.257 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P3pIthtqNfOeOGVYGRtqRDaWWfw= 2024-04-12T03:25:51.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P3pIthtqNfOeOGVYGRtqRDaWWfw=', 'Content-Length': '0'} 2024-04-12T03:25:51.259 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c0f091d2e03cdfe-006618a9bf-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.259 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:25:51.259 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:25:51.259 DEBUG:boto:path=/dkyges-14/?versions&max-keys=999&prefix=obj-a2 2024-04-12T03:25:51.259 DEBUG:boto:auth_path=/dkyges-14/?versions&max-keys=999&prefix=obj-a2 2024-04-12T03:25:51.259 DEBUG:boto:Method: GET 2024-04-12T03:25:51.260 DEBUG:boto:Path: /dkyges-14/?versions&max-keys=999&prefix=obj-a2 2024-04-12T03:25:51.260 DEBUG:boto:Data: 2024-04-12T03:25:51.260 DEBUG:boto:Headers: {} 2024-04-12T03:25:51.260 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:51.260 DEBUG:boto:Port: 8001 2024-04-12T03:25:51.260 DEBUG:boto:Params: {} 2024-04-12T03:25:51.260 DEBUG:boto:Token: None 2024-04-12T03:25:51.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:25:51 GMT /dkyges-14/?versions 2024-04-12T03:25:51.260 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XteyuN++L7K6hDiWgdkLNpE0xWc= 2024-04-12T03:25:51.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XteyuN++L7K6hDiWgdkLNpE0xWc=', 'Content-Length': '0'} 2024-04-12T03:25:51.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ded85c7921e121a-006618a9bf-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.306 DEBUG:boto:b'dkyges-14obj-a2999falseobj-a2N26z7de3t6pW-GMELJHxxRYWiKhUwcFtrue2024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:25:51.307 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:25:51.307 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:25:51.307 DEBUG:boto:path=/dkyges-14/obj-a2?acl 2024-04-12T03:25:51.307 DEBUG:boto:auth_path=/dkyges-14/obj-a2?acl 2024-04-12T03:25:51.307 DEBUG:boto:Method: PUT 2024-04-12T03:25:51.307 DEBUG:boto:Path: /dkyges-14/obj-a2?acl 2024-04-12T03:25:51.307 DEBUG:boto:Data: 2024-04-12T03:25:51.307 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-12T03:25:51.308 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:25:51.308 DEBUG:boto:Port: 8001 2024-04-12T03:25:51.308 DEBUG:boto:Params: {} 2024-04-12T03:25:51.308 DEBUG:boto:Token: None 2024-04-12T03:25:51.308 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:25:51 GMT x-amz-acl:public-read /dkyges-14/obj-a2?acl 2024-04-12T03:25:51.308 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mV1IbFCizCOxD9EPxHXMVn47HTU= 2024-04-12T03:25:51.308 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:25:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mV1IbFCizCOxD9EPxHXMVn47HTU=', 'Content-Length': '0'} 2024-04-12T03:25:51.316 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a685c75a97b1a7a8-006618a9bf-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:25:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:25:51.316 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-13 2024-04-12T03:25:51.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:25:51.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:25:51.317 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:25:51.383 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:25:51.384 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:25:51.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:51.424+0000 7f0447350780 1 waiting for incremental sync to catch up: 2024-04-12T03:25:51.425 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.12.10, , , 00000000002.8.11, 00000000010.22.6, ] 2024-04-12T03:25:51.425 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.12.10, , , 00000000002.8.11, 00000000011.25.6, ] 2024-04-12T03:25:56.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:25:56.425+0000 7f0447350780 1 waiting for incremental sync to catch up: 2024-04-12T03:25:56.427 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.12.10, , , 00000000002.8.11, 00000000010.22.6, ] 2024-04-12T03:25:56.427 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.12.10, , , 00000000002.8.11, 00000000011.25.6, ] 2024-04-12T03:26:01.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:01.427+0000 7f0447350780 1 waiting for incremental sync to catch up: 2024-04-12T03:26:01.428 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.12.10, , , 00000000002.8.11, 00000000010.22.6, ] 2024-04-12T03:26:01.428 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.12.10, , , 00000000002.8.11, 00000000011.25.6, ] 2024-04-12T03:26:06.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:06.428+0000 7f0447350780 1 bucket sync caught up with source: 2024-04-12T03:26:06.429 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.12.10, , , 00000000002.8.11, 00000000011.25.6, ] 2024-04-12T03:26:06.429 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.12.10, , , 00000000002.8.11, 00000000011.25.6, ] 2024-04-12T03:26:06.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:06.428+0000 7f0447350780 0 bucket checkpoint complete 2024-04-12T03:26:06.446 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-13 2024-04-12T03:26:06.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:26:06.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:26:06.446 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:26:06.514 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:06.514 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:06.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:06.554+0000 7efe8b1ff780 1 bucket sync caught up with source: 2024-04-12T03:26:06.556 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.12.11, , , 00000000011.37.6, 00000000002.14.11, ] 2024-04-12T03:26:06.556 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000011.37.6, 00000000002.14.11, ] 2024-04-12T03:26:06.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:06.554+0000 7efe8b1ff780 0 bucket checkpoint complete 2024-04-12T03:26:06.569 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-13 zones={a2, a1} 2024-04-12T03:26:06.569 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:06.569 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:06.570 DEBUG:boto:Method: HEAD 2024-04-12T03:26:06.570 DEBUG:boto:Path: /dkyges-13/ 2024-04-12T03:26:06.570 DEBUG:boto:Data: 2024-04-12T03:26:06.570 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.570 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.570 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.570 DEBUG:boto:Params: {} 2024-04-12T03:26:06.570 DEBUG:boto:Token: None 2024-04-12T03:26:06.570 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/ 2024-04-12T03:26:06.571 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:59pDmKY5p9J9s/93HeyuoyUQosg= 2024-04-12T03:26:06.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:59pDmKY5p9J9s/93HeyuoyUQosg=', 'Content-Length': '0'} 2024-04-12T03:26:06.573 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1128430c9025e0c-006618a9ce-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.573 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:06.573 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:06.573 DEBUG:boto:Method: HEAD 2024-04-12T03:26:06.573 DEBUG:boto:Path: /dkyges-13/ 2024-04-12T03:26:06.573 DEBUG:boto:Data: 2024-04-12T03:26:06.574 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.574 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.574 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.574 DEBUG:boto:Params: {} 2024-04-12T03:26:06.574 DEBUG:boto:Token: None 2024-04-12T03:26:06.574 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/ 2024-04-12T03:26:06.574 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:59pDmKY5p9J9s/93HeyuoyUQosg= 2024-04-12T03:26:06.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:59pDmKY5p9J9s/93HeyuoyUQosg=', 'Content-Length': '0'} 2024-04-12T03:26:06.576 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009bcdb594cd64276d-006618a9ce-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.577 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:26:06.577 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:06.577 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:06.577 DEBUG:boto:path=/dkyges-13/?versions&max-keys=999 2024-04-12T03:26:06.577 DEBUG:boto:auth_path=/dkyges-13/?versions&max-keys=999 2024-04-12T03:26:06.577 DEBUG:boto:Method: GET 2024-04-12T03:26:06.577 DEBUG:boto:Path: /dkyges-13/?versions&max-keys=999 2024-04-12T03:26:06.577 DEBUG:boto:Data: 2024-04-12T03:26:06.577 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.578 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.578 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.578 DEBUG:boto:Params: {} 2024-04-12T03:26:06.578 DEBUG:boto:Token: None 2024-04-12T03:26:06.578 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/?versions 2024-04-12T03:26:06.578 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vMwV3/c239ZJftFYrTV8rAwoLaQ= 2024-04-12T03:26:06.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vMwV3/c239ZJftFYrTV8rAwoLaQ=', 'Content-Length': '0'} 2024-04-12T03:26:06.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9172e437d6348c4-006618a9ce-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.580 DEBUG:boto:b'dkyges-13999falsedummySwFL1c7KE7MLvXFD1.wmtEUphs6xWXUtrue2024-04-12T03:24:41.660Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1No.FwkW62CECdIaWakbZwZXAPvIeOmOtrue2024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a29EpHGMR9P8CQ61qU0DObgl0ylooJjuatrue2024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:06.581 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:26:06.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-12T03:26:06.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-12T03:26:06.581 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:26:06.581 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:06.581 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:06.581 DEBUG:boto:path=/dkyges-13/?versions&max-keys=999 2024-04-12T03:26:06.582 DEBUG:boto:auth_path=/dkyges-13/?versions&max-keys=999 2024-04-12T03:26:06.582 DEBUG:boto:Method: GET 2024-04-12T03:26:06.582 DEBUG:boto:Path: /dkyges-13/?versions&max-keys=999 2024-04-12T03:26:06.582 DEBUG:boto:Data: 2024-04-12T03:26:06.582 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.582 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.582 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.582 DEBUG:boto:Params: {} 2024-04-12T03:26:06.582 DEBUG:boto:Token: None 2024-04-12T03:26:06.582 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/?versions 2024-04-12T03:26:06.583 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vMwV3/c239ZJftFYrTV8rAwoLaQ= 2024-04-12T03:26:06.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vMwV3/c239ZJftFYrTV8rAwoLaQ=', 'Content-Length': '0'} 2024-04-12T03:26:06.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027ac07fc7fd1307a-006618a9ce-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.585 DEBUG:boto:b'dkyges-13999falsedummySwFL1c7KE7MLvXFD1.wmtEUphs6xWXUtrue2024-04-12T03:24:41.660Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1No.FwkW62CECdIaWakbZwZXAPvIeOmOtrue2024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a29EpHGMR9P8CQ61qU0DObgl0ylooJjuatrue2024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:06.585 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:26:06.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-12T03:26:06.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-12T03:26:06.586 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:06.586 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:06.586 DEBUG:boto:path=/dkyges-13/?versions&max-keys=999 2024-04-12T03:26:06.586 DEBUG:boto:auth_path=/dkyges-13/?versions&max-keys=999 2024-04-12T03:26:06.586 DEBUG:boto:Method: GET 2024-04-12T03:26:06.586 DEBUG:boto:Path: /dkyges-13/?versions&max-keys=999 2024-04-12T03:26:06.586 DEBUG:boto:Data: 2024-04-12T03:26:06.586 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.586 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.587 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.587 DEBUG:boto:Params: {} 2024-04-12T03:26:06.587 DEBUG:boto:Token: None 2024-04-12T03:26:06.587 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/?versions 2024-04-12T03:26:06.587 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vMwV3/c239ZJftFYrTV8rAwoLaQ= 2024-04-12T03:26:06.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vMwV3/c239ZJftFYrTV8rAwoLaQ=', 'Content-Length': '0'} 2024-04-12T03:26:06.589 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdbea19b65d11fdc-006618a9ce-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.589 DEBUG:boto:b'dkyges-13999falsedummySwFL1c7KE7MLvXFD1.wmtEUphs6xWXUtrue2024-04-12T03:24:41.660Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1No.FwkW62CECdIaWakbZwZXAPvIeOmOtrue2024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a29EpHGMR9P8CQ61qU0DObgl0ylooJjuatrue2024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:06.590 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:06.590 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:06.590 DEBUG:boto:path=/dkyges-13/?versions&max-keys=999 2024-04-12T03:26:06.590 DEBUG:boto:auth_path=/dkyges-13/?versions&max-keys=999 2024-04-12T03:26:06.590 DEBUG:boto:Method: GET 2024-04-12T03:26:06.590 DEBUG:boto:Path: /dkyges-13/?versions&max-keys=999 2024-04-12T03:26:06.590 DEBUG:boto:Data: 2024-04-12T03:26:06.591 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.591 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.591 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.591 DEBUG:boto:Params: {} 2024-04-12T03:26:06.591 DEBUG:boto:Token: None 2024-04-12T03:26:06.591 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/?versions 2024-04-12T03:26:06.591 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vMwV3/c239ZJftFYrTV8rAwoLaQ= 2024-04-12T03:26:06.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vMwV3/c239ZJftFYrTV8rAwoLaQ=', 'Content-Length': '0'} 2024-04-12T03:26:06.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000138c583a77b7c61c-006618a9ce-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.594 DEBUG:boto:b'dkyges-13999falsedummySwFL1c7KE7MLvXFD1.wmtEUphs6xWXUtrue2024-04-12T03:24:41.660Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1No.FwkW62CECdIaWakbZwZXAPvIeOmOtrue2024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a29EpHGMR9P8CQ61qU0DObgl0ylooJjuatrue2024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:06.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:26:06.594 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:26:06.594 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:26:06.594 DEBUG:boto:path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.594 DEBUG:boto:auth_path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.594 DEBUG:boto:Method: GET 2024-04-12T03:26:06.595 DEBUG:boto:Path: /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.595 DEBUG:boto:Data: 2024-04-12T03:26:06.595 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.595 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.595 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.595 DEBUG:boto:Params: {} 2024-04-12T03:26:06.595 DEBUG:boto:Token: None 2024-04-12T03:26:06.595 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.595 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mcshqtXVIGdAHvZf66M54qLbJfE= 2024-04-12T03:26:06.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mcshqtXVIGdAHvZf66M54qLbJfE=', 'Content-Length': '0'} 2024-04-12T03:26:06.597 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000079d2e8fca4329556-006618a9ce-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.597 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:26:06.597 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:26:06.597 DEBUG:boto:path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.597 DEBUG:boto:auth_path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.597 DEBUG:boto:Method: GET 2024-04-12T03:26:06.598 DEBUG:boto:Path: /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.598 DEBUG:boto:Data: 2024-04-12T03:26:06.598 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.598 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.598 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.598 DEBUG:boto:Params: {} 2024-04-12T03:26:06.598 DEBUG:boto:Token: None 2024-04-12T03:26:06.598 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.598 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mcshqtXVIGdAHvZf66M54qLbJfE= 2024-04-12T03:26:06.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mcshqtXVIGdAHvZf66M54qLbJfE=', 'Content-Length': '0'} 2024-04-12T03:26:06.600 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cbdc4a8ca7d6d3b5-006618a9ce-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.600 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:26:06.600 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:26:06.600 DEBUG:boto:path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.601 DEBUG:boto:auth_path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.601 DEBUG:boto:Method: HEAD 2024-04-12T03:26:06.601 DEBUG:boto:Path: /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.601 DEBUG:boto:Data: 2024-04-12T03:26:06.601 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.601 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.601 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.601 DEBUG:boto:Params: {} 2024-04-12T03:26:06.601 DEBUG:boto:Token: None 2024-04-12T03:26:06.601 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.601 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GDRur82HsvJ4i7t4OZK863KhfS4= 2024-04-12T03:26:06.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GDRur82HsvJ4i7t4OZK863KhfS4=', 'Content-Length': '0'} 2024-04-12T03:26:06.603 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa7417f2add2bfa0-006618a9ce-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.603 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:26:06.603 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:26:06.604 DEBUG:boto:path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.604 DEBUG:boto:auth_path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.604 DEBUG:boto:Method: HEAD 2024-04-12T03:26:06.604 DEBUG:boto:Path: /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.604 DEBUG:boto:Data: 2024-04-12T03:26:06.604 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.604 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.604 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.604 DEBUG:boto:Params: {} 2024-04-12T03:26:06.604 DEBUG:boto:Token: None 2024-04-12T03:26:06.604 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GDRur82HsvJ4i7t4OZK863KhfS4= 2024-04-12T03:26:06.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GDRur82HsvJ4i7t4OZK863KhfS4=', 'Content-Length': '0'} 2024-04-12T03:26:06.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b354213be35c3c0-006618a9ce-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:26:06.606 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:26:06.607 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:26:06.607 DEBUG:boto:path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.607 DEBUG:boto:auth_path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.607 DEBUG:boto:Method: GET 2024-04-12T03:26:06.607 DEBUG:boto:Path: /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.607 DEBUG:boto:Data: 2024-04-12T03:26:06.607 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.607 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.607 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.607 DEBUG:boto:Params: {} 2024-04-12T03:26:06.607 DEBUG:boto:Token: None 2024-04-12T03:26:06.608 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.608 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mcshqtXVIGdAHvZf66M54qLbJfE= 2024-04-12T03:26:06.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mcshqtXVIGdAHvZf66M54qLbJfE=', 'Content-Length': '0'} 2024-04-12T03:26:06.610 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b32c0f2f56847734-006618a9ce-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.610 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:26:06.610 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:26:06.610 DEBUG:boto:path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.610 DEBUG:boto:auth_path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.610 DEBUG:boto:Method: GET 2024-04-12T03:26:06.610 DEBUG:boto:Path: /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.610 DEBUG:boto:Data: 2024-04-12T03:26:06.611 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.611 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.611 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.611 DEBUG:boto:Params: {} 2024-04-12T03:26:06.611 DEBUG:boto:Token: None 2024-04-12T03:26:06.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.611 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mcshqtXVIGdAHvZf66M54qLbJfE= 2024-04-12T03:26:06.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mcshqtXVIGdAHvZf66M54qLbJfE=', 'Content-Length': '0'} 2024-04-12T03:26:06.613 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002202c635403b1080-006618a9ce-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.613 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:06.613 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:06.613 DEBUG:boto:path=/dkyges-13/?prefix=dummy 2024-04-12T03:26:06.613 DEBUG:boto:auth_path=/dkyges-13/?prefix=dummy 2024-04-12T03:26:06.613 DEBUG:boto:Method: GET 2024-04-12T03:26:06.613 DEBUG:boto:Path: /dkyges-13/?prefix=dummy 2024-04-12T03:26:06.613 DEBUG:boto:Data: 2024-04-12T03:26:06.613 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.614 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.614 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.614 DEBUG:boto:Params: {} 2024-04-12T03:26:06.614 DEBUG:boto:Token: None 2024-04-12T03:26:06.614 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/ 2024-04-12T03:26:06.614 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xtW5wYhC0ahXxlO/jFV7s5GuHcA= 2024-04-12T03:26:06.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xtW5wYhC0ahXxlO/jFV7s5GuHcA=', 'Content-Length': '0'} 2024-04-12T03:26:06.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0c333ae7be1fe71-006618a9ce-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.617 DEBUG:boto:b'dkyges-13dummy1000falsedummy2024-04-12T03:24:41.660Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:06.617 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:06.617 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:06.617 DEBUG:boto:path=/dkyges-13/?prefix=dummy 2024-04-12T03:26:06.617 DEBUG:boto:auth_path=/dkyges-13/?prefix=dummy 2024-04-12T03:26:06.618 DEBUG:boto:Method: GET 2024-04-12T03:26:06.618 DEBUG:boto:Path: /dkyges-13/?prefix=dummy 2024-04-12T03:26:06.618 DEBUG:boto:Data: 2024-04-12T03:26:06.618 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.618 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.618 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.618 DEBUG:boto:Params: {} 2024-04-12T03:26:06.618 DEBUG:boto:Token: None 2024-04-12T03:26:06.618 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/ 2024-04-12T03:26:06.618 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xtW5wYhC0ahXxlO/jFV7s5GuHcA= 2024-04-12T03:26:06.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xtW5wYhC0ahXxlO/jFV7s5GuHcA=', 'Content-Length': '0'} 2024-04-12T03:26:06.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fc2364d0fefca51-006618a9ce-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.666 DEBUG:boto:b'dkyges-13dummy1000falsedummy2024-04-12T03:24:41.660Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:06.667 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:26:06.667 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:26:06.667 DEBUG:boto:Method: HEAD 2024-04-12T03:26:06.667 DEBUG:boto:Path: /dkyges-13/dummy 2024-04-12T03:26:06.667 DEBUG:boto:Data: 2024-04-12T03:26:06.667 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.667 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.667 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.667 DEBUG:boto:Params: {} 2024-04-12T03:26:06.667 DEBUG:boto:Token: None 2024-04-12T03:26:06.668 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/dummy 2024-04-12T03:26:06.668 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lAFC8Zf5RdKnluSNH0spEW9bfKk= 2024-04-12T03:26:06.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lAFC8Zf5RdKnluSNH0spEW9bfKk=', 'Content-Length': '0'} 2024-04-12T03:26:06.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003a70f778b34b3338-006618a9ce-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.670 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:26:06.670 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:26:06.670 DEBUG:boto:Method: HEAD 2024-04-12T03:26:06.670 DEBUG:boto:Path: /dkyges-13/dummy 2024-04-12T03:26:06.670 DEBUG:boto:Data: 2024-04-12T03:26:06.670 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.671 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.671 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.671 DEBUG:boto:Params: {} 2024-04-12T03:26:06.671 DEBUG:boto:Token: None 2024-04-12T03:26:06.671 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/dummy 2024-04-12T03:26:06.671 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lAFC8Zf5RdKnluSNH0spEW9bfKk= 2024-04-12T03:26:06.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lAFC8Zf5RdKnluSNH0spEW9bfKk=', 'Content-Length': '0'} 2024-04-12T03:26:06.673 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad321f083660cf27-006618a9ce-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:26:06.673 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:26:06.674 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:26:06.674 DEBUG:boto:path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.674 DEBUG:boto:auth_path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.674 DEBUG:boto:Method: GET 2024-04-12T03:26:06.674 DEBUG:boto:Path: /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.674 DEBUG:boto:Data: 2024-04-12T03:26:06.674 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.674 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.674 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.674 DEBUG:boto:Params: {} 2024-04-12T03:26:06.674 DEBUG:boto:Token: None 2024-04-12T03:26:06.675 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.675 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mcshqtXVIGdAHvZf66M54qLbJfE= 2024-04-12T03:26:06.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mcshqtXVIGdAHvZf66M54qLbJfE=', 'Content-Length': '0'} 2024-04-12T03:26:06.676 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d7f85554817655f1-006618a9ce-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.677 DEBUG:boto:path=/dkyges-13/dummy 2024-04-12T03:26:06.677 DEBUG:boto:auth_path=/dkyges-13/dummy 2024-04-12T03:26:06.677 DEBUG:boto:path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.677 DEBUG:boto:auth_path=/dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.677 DEBUG:boto:Method: GET 2024-04-12T03:26:06.677 DEBUG:boto:Path: /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.677 DEBUG:boto:Data: 2024-04-12T03:26:06.677 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.677 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.677 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.678 DEBUG:boto:Params: {} 2024-04-12T03:26:06.678 DEBUG:boto:Token: None 2024-04-12T03:26:06.678 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/dummy?versionId=SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU 2024-04-12T03:26:06.678 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mcshqtXVIGdAHvZf66M54qLbJfE= 2024-04-12T03:26:06.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mcshqtXVIGdAHvZf66M54qLbJfE=', 'Content-Length': '0'} 2024-04-12T03:26:06.680 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:41 GMT'), ('x-amz-version-id', 'SwFL1c7KE7MLvXFD1.wmtEUphs6xWXU'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a3503cb3c4441b17-006618a9ce-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.680 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:06.680 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:06.680 DEBUG:boto:path=/dkyges-13/?prefix=dummy 2024-04-12T03:26:06.680 DEBUG:boto:auth_path=/dkyges-13/?prefix=dummy 2024-04-12T03:26:06.680 DEBUG:boto:Method: GET 2024-04-12T03:26:06.681 DEBUG:boto:Path: /dkyges-13/?prefix=dummy 2024-04-12T03:26:06.681 DEBUG:boto:Data: 2024-04-12T03:26:06.681 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.681 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.681 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.681 DEBUG:boto:Params: {} 2024-04-12T03:26:06.681 DEBUG:boto:Token: None 2024-04-12T03:26:06.681 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/ 2024-04-12T03:26:06.681 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xtW5wYhC0ahXxlO/jFV7s5GuHcA= 2024-04-12T03:26:06.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xtW5wYhC0ahXxlO/jFV7s5GuHcA=', 'Content-Length': '0'} 2024-04-12T03:26:06.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000777dbf0fe3202d6f-006618a9ce-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.684 DEBUG:boto:b'dkyges-13dummy1000falsedummy2024-04-12T03:24:41.660Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:06.684 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:06.684 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:06.684 DEBUG:boto:path=/dkyges-13/?prefix=dummy 2024-04-12T03:26:06.685 DEBUG:boto:auth_path=/dkyges-13/?prefix=dummy 2024-04-12T03:26:06.685 DEBUG:boto:Method: GET 2024-04-12T03:26:06.685 DEBUG:boto:Path: /dkyges-13/?prefix=dummy 2024-04-12T03:26:06.685 DEBUG:boto:Data: 2024-04-12T03:26:06.685 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.685 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.685 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.685 DEBUG:boto:Params: {} 2024-04-12T03:26:06.685 DEBUG:boto:Token: None 2024-04-12T03:26:06.685 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/ 2024-04-12T03:26:06.685 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xtW5wYhC0ahXxlO/jFV7s5GuHcA= 2024-04-12T03:26:06.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xtW5wYhC0ahXxlO/jFV7s5GuHcA=', 'Content-Length': '0'} 2024-04-12T03:26:06.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a05590be21cfc292-006618a9ce-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.688 DEBUG:boto:b'dkyges-13dummy1000falsedummy2024-04-12T03:24:41.660Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:06.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:26:06.689 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:26:06.689 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:26:06.689 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.689 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.689 DEBUG:boto:Method: GET 2024-04-12T03:26:06.689 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.689 DEBUG:boto:Data: 2024-04-12T03:26:06.689 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.689 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.689 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.689 DEBUG:boto:Params: {} 2024-04-12T03:26:06.690 DEBUG:boto:Token: None 2024-04-12T03:26:06.690 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.690 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hkgh5UNhl7FPZGkqyT+aq4C4GD8= 2024-04-12T03:26:06.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hkgh5UNhl7FPZGkqyT+aq4C4GD8=', 'Content-Length': '0'} 2024-04-12T03:26:06.692 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000019af14d80446ea6e-006618a9ce-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.734 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:26:06.734 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:26:06.734 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.735 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.735 DEBUG:boto:Method: GET 2024-04-12T03:26:06.735 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.735 DEBUG:boto:Data: 2024-04-12T03:26:06.735 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.735 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.735 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.735 DEBUG:boto:Params: {} 2024-04-12T03:26:06.735 DEBUG:boto:Token: None 2024-04-12T03:26:06.735 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.736 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hkgh5UNhl7FPZGkqyT+aq4C4GD8= 2024-04-12T03:26:06.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hkgh5UNhl7FPZGkqyT+aq4C4GD8=', 'Content-Length': '0'} 2024-04-12T03:26:06.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd0b7155a050e606-006618a9ce-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.738 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:26:06.738 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:26:06.738 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.738 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.738 DEBUG:boto:Method: HEAD 2024-04-12T03:26:06.739 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.739 DEBUG:boto:Data: 2024-04-12T03:26:06.739 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.739 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.739 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.739 DEBUG:boto:Params: {} 2024-04-12T03:26:06.739 DEBUG:boto:Token: None 2024-04-12T03:26:06.739 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.739 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BMRpm0wQi5vWf8TKJ85jdhbWuAQ= 2024-04-12T03:26:06.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BMRpm0wQi5vWf8TKJ85jdhbWuAQ=', 'Content-Length': '0'} 2024-04-12T03:26:06.741 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001a362491c85da4bd-006618a9ce-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.741 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:26:06.742 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:26:06.742 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.742 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.742 DEBUG:boto:Method: HEAD 2024-04-12T03:26:06.742 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.742 DEBUG:boto:Data: 2024-04-12T03:26:06.742 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.742 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.742 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.742 DEBUG:boto:Params: {} 2024-04-12T03:26:06.743 DEBUG:boto:Token: None 2024-04-12T03:26:06.743 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.743 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BMRpm0wQi5vWf8TKJ85jdhbWuAQ= 2024-04-12T03:26:06.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BMRpm0wQi5vWf8TKJ85jdhbWuAQ=', 'Content-Length': '0'} 2024-04-12T03:26:06.744 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc7e7902b47f58ff-006618a9ce-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:26:06.745 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:26:06.745 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:26:06.745 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.745 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.745 DEBUG:boto:Method: GET 2024-04-12T03:26:06.745 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.745 DEBUG:boto:Data: 2024-04-12T03:26:06.745 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.746 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.746 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.746 DEBUG:boto:Params: {} 2024-04-12T03:26:06.746 DEBUG:boto:Token: None 2024-04-12T03:26:06.746 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.746 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hkgh5UNhl7FPZGkqyT+aq4C4GD8= 2024-04-12T03:26:06.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hkgh5UNhl7FPZGkqyT+aq4C4GD8=', 'Content-Length': '0'} 2024-04-12T03:26:06.748 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fbb971e4a90052b-006618a9ce-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.790 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:26:06.791 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:26:06.791 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.791 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.791 DEBUG:boto:Method: GET 2024-04-12T03:26:06.791 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.791 DEBUG:boto:Data: 2024-04-12T03:26:06.791 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.791 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.791 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.791 DEBUG:boto:Params: {} 2024-04-12T03:26:06.792 DEBUG:boto:Token: None 2024-04-12T03:26:06.792 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.792 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hkgh5UNhl7FPZGkqyT+aq4C4GD8= 2024-04-12T03:26:06.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hkgh5UNhl7FPZGkqyT+aq4C4GD8=', 'Content-Length': '0'} 2024-04-12T03:26:06.794 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fce8c45637c4da0f-006618a9ce-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.794 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:06.794 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:06.794 DEBUG:boto:path=/dkyges-13/?prefix=obj-a1 2024-04-12T03:26:06.794 DEBUG:boto:auth_path=/dkyges-13/?prefix=obj-a1 2024-04-12T03:26:06.795 DEBUG:boto:Method: GET 2024-04-12T03:26:06.795 DEBUG:boto:Path: /dkyges-13/?prefix=obj-a1 2024-04-12T03:26:06.795 DEBUG:boto:Data: 2024-04-12T03:26:06.795 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.795 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.795 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.795 DEBUG:boto:Params: {} 2024-04-12T03:26:06.795 DEBUG:boto:Token: None 2024-04-12T03:26:06.795 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/ 2024-04-12T03:26:06.795 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xtW5wYhC0ahXxlO/jFV7s5GuHcA= 2024-04-12T03:26:06.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xtW5wYhC0ahXxlO/jFV7s5GuHcA=', 'Content-Length': '0'} 2024-04-12T03:26:06.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f35f5c8525ec9937-006618a9ce-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.798 DEBUG:boto:b'dkyges-13obj-a11000falseobj-a12024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:06.798 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:06.798 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:06.799 DEBUG:boto:path=/dkyges-13/?prefix=obj-a1 2024-04-12T03:26:06.799 DEBUG:boto:auth_path=/dkyges-13/?prefix=obj-a1 2024-04-12T03:26:06.799 DEBUG:boto:Method: GET 2024-04-12T03:26:06.799 DEBUG:boto:Path: /dkyges-13/?prefix=obj-a1 2024-04-12T03:26:06.799 DEBUG:boto:Data: 2024-04-12T03:26:06.799 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.799 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.799 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.799 DEBUG:boto:Params: {} 2024-04-12T03:26:06.799 DEBUG:boto:Token: None 2024-04-12T03:26:06.799 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/ 2024-04-12T03:26:06.800 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xtW5wYhC0ahXxlO/jFV7s5GuHcA= 2024-04-12T03:26:06.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xtW5wYhC0ahXxlO/jFV7s5GuHcA=', 'Content-Length': '0'} 2024-04-12T03:26:06.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068f65c4fe8755308-006618a9ce-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.802 DEBUG:boto:b'dkyges-13obj-a11000falseobj-a12024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:06.803 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:26:06.803 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:26:06.803 DEBUG:boto:Method: HEAD 2024-04-12T03:26:06.803 DEBUG:boto:Path: /dkyges-13/obj-a1 2024-04-12T03:26:06.803 DEBUG:boto:Data: 2024-04-12T03:26:06.803 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.803 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.803 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.803 DEBUG:boto:Params: {} 2024-04-12T03:26:06.804 DEBUG:boto:Token: None 2024-04-12T03:26:06.804 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/obj-a1 2024-04-12T03:26:06.804 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B2d7IuKEE/TX3fJY0i5IxP2VQys= 2024-04-12T03:26:06.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B2d7IuKEE/TX3fJY0i5IxP2VQys=', 'Content-Length': '0'} 2024-04-12T03:26:06.806 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001b76f9b5d22cae6e-006618a9ce-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.806 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:26:06.806 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:26:06.806 DEBUG:boto:Method: HEAD 2024-04-12T03:26:06.806 DEBUG:boto:Path: /dkyges-13/obj-a1 2024-04-12T03:26:06.807 DEBUG:boto:Data: 2024-04-12T03:26:06.807 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.807 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.807 DEBUG:boto:Params: {} 2024-04-12T03:26:06.807 DEBUG:boto:Token: None 2024-04-12T03:26:06.807 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/obj-a1 2024-04-12T03:26:06.807 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B2d7IuKEE/TX3fJY0i5IxP2VQys= 2024-04-12T03:26:06.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B2d7IuKEE/TX3fJY0i5IxP2VQys=', 'Content-Length': '0'} 2024-04-12T03:26:06.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058393cc3caa53dbf-006618a9ce-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:26:06.810 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:26:06.810 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:26:06.810 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.810 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.810 DEBUG:boto:Method: GET 2024-04-12T03:26:06.810 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.810 DEBUG:boto:Data: 2024-04-12T03:26:06.810 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.811 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.811 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.811 DEBUG:boto:Params: {} 2024-04-12T03:26:06.811 DEBUG:boto:Token: None 2024-04-12T03:26:06.811 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.811 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hkgh5UNhl7FPZGkqyT+aq4C4GD8= 2024-04-12T03:26:06.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hkgh5UNhl7FPZGkqyT+aq4C4GD8=', 'Content-Length': '0'} 2024-04-12T03:26:06.813 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7e8f5ef7f340346-006618a9ce-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.854 DEBUG:boto:path=/dkyges-13/obj-a1 2024-04-12T03:26:06.855 DEBUG:boto:auth_path=/dkyges-13/obj-a1 2024-04-12T03:26:06.855 DEBUG:boto:path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.855 DEBUG:boto:auth_path=/dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.855 DEBUG:boto:Method: GET 2024-04-12T03:26:06.855 DEBUG:boto:Path: /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.855 DEBUG:boto:Data: 2024-04-12T03:26:06.855 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.855 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.855 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.855 DEBUG:boto:Params: {} 2024-04-12T03:26:06.856 DEBUG:boto:Token: None 2024-04-12T03:26:06.856 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/obj-a1?versionId=No.FwkW62CECdIaWakbZwZXAPvIeOmO 2024-04-12T03:26:06.856 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hkgh5UNhl7FPZGkqyT+aq4C4GD8= 2024-04-12T03:26:06.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hkgh5UNhl7FPZGkqyT+aq4C4GD8=', 'Content-Length': '0'} 2024-04-12T03:26:06.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:06 GMT'), ('x-amz-version-id', 'No.FwkW62CECdIaWakbZwZXAPvIeOmO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000011e7057e6498296-006618a9ce-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.858 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:06.858 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:06.858 DEBUG:boto:path=/dkyges-13/?prefix=obj-a1 2024-04-12T03:26:06.858 DEBUG:boto:auth_path=/dkyges-13/?prefix=obj-a1 2024-04-12T03:26:06.859 DEBUG:boto:Method: GET 2024-04-12T03:26:06.859 DEBUG:boto:Path: /dkyges-13/?prefix=obj-a1 2024-04-12T03:26:06.859 DEBUG:boto:Data: 2024-04-12T03:26:06.859 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.859 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.859 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.859 DEBUG:boto:Params: {} 2024-04-12T03:26:06.859 DEBUG:boto:Token: None 2024-04-12T03:26:06.859 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/ 2024-04-12T03:26:06.859 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xtW5wYhC0ahXxlO/jFV7s5GuHcA= 2024-04-12T03:26:06.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xtW5wYhC0ahXxlO/jFV7s5GuHcA=', 'Content-Length': '0'} 2024-04-12T03:26:06.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008569f92bcb259d76-006618a9ce-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.902 DEBUG:boto:b'dkyges-13obj-a11000falseobj-a12024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:06.903 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:06.903 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:06.903 DEBUG:boto:path=/dkyges-13/?prefix=obj-a1 2024-04-12T03:26:06.903 DEBUG:boto:auth_path=/dkyges-13/?prefix=obj-a1 2024-04-12T03:26:06.903 DEBUG:boto:Method: GET 2024-04-12T03:26:06.903 DEBUG:boto:Path: /dkyges-13/?prefix=obj-a1 2024-04-12T03:26:06.903 DEBUG:boto:Data: 2024-04-12T03:26:06.904 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.904 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.904 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.904 DEBUG:boto:Params: {} 2024-04-12T03:26:06.904 DEBUG:boto:Token: None 2024-04-12T03:26:06.904 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/ 2024-04-12T03:26:06.904 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xtW5wYhC0ahXxlO/jFV7s5GuHcA= 2024-04-12T03:26:06.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xtW5wYhC0ahXxlO/jFV7s5GuHcA=', 'Content-Length': '0'} 2024-04-12T03:26:06.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025ec73ebf5abf06a-006618a9ce-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.907 DEBUG:boto:b'dkyges-13obj-a11000falseobj-a12024-04-12T03:25:06.944Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:06.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:26:06.908 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:26:06.908 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:26:06.908 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.908 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.908 DEBUG:boto:Method: GET 2024-04-12T03:26:06.908 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.908 DEBUG:boto:Data: 2024-04-12T03:26:06.908 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.908 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.908 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.908 DEBUG:boto:Params: {} 2024-04-12T03:26:06.909 DEBUG:boto:Token: None 2024-04-12T03:26:06.909 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.909 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ah5e4Gqi++ceA2nM2paJ0sLspvU= 2024-04-12T03:26:06.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ah5e4Gqi++ceA2nM2paJ0sLspvU=', 'Content-Length': '0'} 2024-04-12T03:26:06.911 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d0ca3539d3cfacb-006618a9ce-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.911 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:26:06.911 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:26:06.911 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.912 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.912 DEBUG:boto:Method: GET 2024-04-12T03:26:06.912 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.912 DEBUG:boto:Data: 2024-04-12T03:26:06.912 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.912 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.912 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.912 DEBUG:boto:Params: {} 2024-04-12T03:26:06.912 DEBUG:boto:Token: None 2024-04-12T03:26:06.912 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.912 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ah5e4Gqi++ceA2nM2paJ0sLspvU= 2024-04-12T03:26:06.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ah5e4Gqi++ceA2nM2paJ0sLspvU=', 'Content-Length': '0'} 2024-04-12T03:26:06.914 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000dc7f790cf7d12b3-006618a9ce-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.958 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:26:06.959 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:26:06.959 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.959 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.959 DEBUG:boto:Method: HEAD 2024-04-12T03:26:06.959 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.959 DEBUG:boto:Data: 2024-04-12T03:26:06.959 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.959 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.959 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.960 DEBUG:boto:Params: {} 2024-04-12T03:26:06.960 DEBUG:boto:Token: None 2024-04-12T03:26:06.960 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.960 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r/6JKgN0eyfWPREZYyGef3tG3Ws= 2024-04-12T03:26:06.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r/6JKgN0eyfWPREZYyGef3tG3Ws=', 'Content-Length': '0'} 2024-04-12T03:26:06.962 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000070808cbf4d30697f-006618a9ce-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.963 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:26:06.963 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:26:06.963 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.963 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.963 DEBUG:boto:Method: HEAD 2024-04-12T03:26:06.963 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.964 DEBUG:boto:Data: 2024-04-12T03:26:06.964 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.964 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.964 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.965 DEBUG:boto:Params: {} 2024-04-12T03:26:06.965 DEBUG:boto:Token: None 2024-04-12T03:26:06.965 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.965 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r/6JKgN0eyfWPREZYyGef3tG3Ws= 2024-04-12T03:26:06.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r/6JKgN0eyfWPREZYyGef3tG3Ws=', 'Content-Length': '0'} 2024-04-12T03:26:06.968 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068c5db210854ee40-006618a9ce-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:26:06.968 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:26:06.968 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:26:06.968 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.968 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.968 DEBUG:boto:Method: GET 2024-04-12T03:26:06.968 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.969 DEBUG:boto:Data: 2024-04-12T03:26:06.969 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.969 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:06.969 DEBUG:boto:Port: 8001 2024-04-12T03:26:06.969 DEBUG:boto:Params: {} 2024-04-12T03:26:06.969 DEBUG:boto:Token: None 2024-04-12T03:26:06.969 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.969 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ah5e4Gqi++ceA2nM2paJ0sLspvU= 2024-04-12T03:26:06.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ah5e4Gqi++ceA2nM2paJ0sLspvU=', 'Content-Length': '0'} 2024-04-12T03:26:06.971 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eedfafd503a699ba-006618a9ce-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:06.971 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:26:06.971 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:26:06.972 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.972 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.972 DEBUG:boto:Method: GET 2024-04-12T03:26:06.972 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.972 DEBUG:boto:Data: 2024-04-12T03:26:06.972 DEBUG:boto:Headers: {} 2024-04-12T03:26:06.972 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:06.972 DEBUG:boto:Port: 8000 2024-04-12T03:26:06.972 DEBUG:boto:Params: {} 2024-04-12T03:26:06.972 DEBUG:boto:Token: None 2024-04-12T03:26:06.972 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:06 GMT /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:06.972 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ah5e4Gqi++ceA2nM2paJ0sLspvU= 2024-04-12T03:26:06.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ah5e4Gqi++ceA2nM2paJ0sLspvU=', 'Content-Length': '0'} 2024-04-12T03:26:06.974 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007abfb52f9523b607-006618a9ce-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.018 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:07.019 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:07.019 DEBUG:boto:path=/dkyges-13/?prefix=obj-a2 2024-04-12T03:26:07.019 DEBUG:boto:auth_path=/dkyges-13/?prefix=obj-a2 2024-04-12T03:26:07.019 DEBUG:boto:Method: GET 2024-04-12T03:26:07.019 DEBUG:boto:Path: /dkyges-13/?prefix=obj-a2 2024-04-12T03:26:07.019 DEBUG:boto:Data: 2024-04-12T03:26:07.019 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.019 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.019 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.019 DEBUG:boto:Params: {} 2024-04-12T03:26:07.020 DEBUG:boto:Token: None 2024-04-12T03:26:07.020 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-13/ 2024-04-12T03:26:07.020 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dWTVcd6AJbLXzQGbGzMLzPKmeVI= 2024-04-12T03:26:07.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dWTVcd6AJbLXzQGbGzMLzPKmeVI=', 'Content-Length': '0'} 2024-04-12T03:26:07.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5eebfea233bf5a0-006618a9cf-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.023 DEBUG:boto:b'dkyges-13obj-a21000falseobj-a22024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.023 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:07.023 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:07.023 DEBUG:boto:path=/dkyges-13/?prefix=obj-a2 2024-04-12T03:26:07.023 DEBUG:boto:auth_path=/dkyges-13/?prefix=obj-a2 2024-04-12T03:26:07.024 DEBUG:boto:Method: GET 2024-04-12T03:26:07.024 DEBUG:boto:Path: /dkyges-13/?prefix=obj-a2 2024-04-12T03:26:07.024 DEBUG:boto:Data: 2024-04-12T03:26:07.024 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.024 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.024 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.024 DEBUG:boto:Params: {} 2024-04-12T03:26:07.024 DEBUG:boto:Token: None 2024-04-12T03:26:07.024 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-13/ 2024-04-12T03:26:07.025 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dWTVcd6AJbLXzQGbGzMLzPKmeVI= 2024-04-12T03:26:07.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dWTVcd6AJbLXzQGbGzMLzPKmeVI=', 'Content-Length': '0'} 2024-04-12T03:26:07.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c840da8d9298f34c-006618a9cf-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.027 DEBUG:boto:b'dkyges-13obj-a21000falseobj-a22024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.028 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:26:07.028 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:26:07.028 DEBUG:boto:Method: HEAD 2024-04-12T03:26:07.028 DEBUG:boto:Path: /dkyges-13/obj-a2 2024-04-12T03:26:07.028 DEBUG:boto:Data: 2024-04-12T03:26:07.028 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.028 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.028 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.029 DEBUG:boto:Params: {} 2024-04-12T03:26:07.029 DEBUG:boto:Token: None 2024-04-12T03:26:07.029 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:07 GMT /dkyges-13/obj-a2 2024-04-12T03:26:07.029 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OIVwah7RqPRnuezHIEe9NZ1KcTA= 2024-04-12T03:26:07.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OIVwah7RqPRnuezHIEe9NZ1KcTA=', 'Content-Length': '0'} 2024-04-12T03:26:07.031 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056dc849ff39d95f1-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.032 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:26:07.032 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:26:07.032 DEBUG:boto:Method: HEAD 2024-04-12T03:26:07.032 DEBUG:boto:Path: /dkyges-13/obj-a2 2024-04-12T03:26:07.032 DEBUG:boto:Data: 2024-04-12T03:26:07.032 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.032 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.032 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.032 DEBUG:boto:Params: {} 2024-04-12T03:26:07.032 DEBUG:boto:Token: None 2024-04-12T03:26:07.033 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:07 GMT /dkyges-13/obj-a2 2024-04-12T03:26:07.033 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OIVwah7RqPRnuezHIEe9NZ1KcTA= 2024-04-12T03:26:07.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OIVwah7RqPRnuezHIEe9NZ1KcTA=', 'Content-Length': '0'} 2024-04-12T03:26:07.036 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b9829fd7f79087e9-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:26:07.036 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:26:07.036 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:26:07.036 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:07.036 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:07.036 DEBUG:boto:Method: GET 2024-04-12T03:26:07.037 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:07.037 DEBUG:boto:Data: 2024-04-12T03:26:07.037 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.037 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.037 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.037 DEBUG:boto:Params: {} 2024-04-12T03:26:07.037 DEBUG:boto:Token: None 2024-04-12T03:26:07.037 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:07.037 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v5BZRTFjblV73BTiY64KAW8goBk= 2024-04-12T03:26:07.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v5BZRTFjblV73BTiY64KAW8goBk=', 'Content-Length': '0'} 2024-04-12T03:26:07.039 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009605d8e45bb004fa-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.082 DEBUG:boto:path=/dkyges-13/obj-a2 2024-04-12T03:26:07.083 DEBUG:boto:auth_path=/dkyges-13/obj-a2 2024-04-12T03:26:07.083 DEBUG:boto:path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:07.083 DEBUG:boto:auth_path=/dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:07.083 DEBUG:boto:Method: GET 2024-04-12T03:26:07.083 DEBUG:boto:Path: /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:07.083 DEBUG:boto:Data: 2024-04-12T03:26:07.083 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.083 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.083 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.083 DEBUG:boto:Params: {} 2024-04-12T03:26:07.083 DEBUG:boto:Token: None 2024-04-12T03:26:07.084 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-13/obj-a2?versionId=9EpHGMR9P8CQ61qU0DObgl0ylooJjua 2024-04-12T03:26:07.084 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v5BZRTFjblV73BTiY64KAW8goBk= 2024-04-12T03:26:07.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v5BZRTFjblV73BTiY64KAW8goBk=', 'Content-Length': '0'} 2024-04-12T03:26:07.086 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', '9EpHGMR9P8CQ61qU0DObgl0ylooJjua'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a7ddf8431c4dda1-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.126 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:07.127 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:07.127 DEBUG:boto:path=/dkyges-13/?prefix=obj-a2 2024-04-12T03:26:07.127 DEBUG:boto:auth_path=/dkyges-13/?prefix=obj-a2 2024-04-12T03:26:07.127 DEBUG:boto:Method: GET 2024-04-12T03:26:07.127 DEBUG:boto:Path: /dkyges-13/?prefix=obj-a2 2024-04-12T03:26:07.127 DEBUG:boto:Data: 2024-04-12T03:26:07.127 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.127 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.127 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.127 DEBUG:boto:Params: {} 2024-04-12T03:26:07.128 DEBUG:boto:Token: None 2024-04-12T03:26:07.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-13/ 2024-04-12T03:26:07.128 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dWTVcd6AJbLXzQGbGzMLzPKmeVI= 2024-04-12T03:26:07.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dWTVcd6AJbLXzQGbGzMLzPKmeVI=', 'Content-Length': '0'} 2024-04-12T03:26:07.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f4f958c868a2c97-006618a9cf-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.131 DEBUG:boto:b'dkyges-13obj-a21000falseobj-a22024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.131 DEBUG:boto:path=/dkyges-13/ 2024-04-12T03:26:07.131 DEBUG:boto:auth_path=/dkyges-13/ 2024-04-12T03:26:07.132 DEBUG:boto:path=/dkyges-13/?prefix=obj-a2 2024-04-12T03:26:07.132 DEBUG:boto:auth_path=/dkyges-13/?prefix=obj-a2 2024-04-12T03:26:07.132 DEBUG:boto:Method: GET 2024-04-12T03:26:07.132 DEBUG:boto:Path: /dkyges-13/?prefix=obj-a2 2024-04-12T03:26:07.132 DEBUG:boto:Data: 2024-04-12T03:26:07.132 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.132 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.132 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.132 DEBUG:boto:Params: {} 2024-04-12T03:26:07.132 DEBUG:boto:Token: None 2024-04-12T03:26:07.133 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-13/ 2024-04-12T03:26:07.133 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dWTVcd6AJbLXzQGbGzMLzPKmeVI= 2024-04-12T03:26:07.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dWTVcd6AJbLXzQGbGzMLzPKmeVI=', 'Content-Length': '0'} 2024-04-12T03:26:07.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9a39d0702a95857-006618a9cf-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.135 DEBUG:boto:b'dkyges-13obj-a21000falseobj-a22024-04-12T03:25:07.047Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.136 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-13 zones={a2, a1} 2024-04-12T03:26:07.136 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-14 2024-04-12T03:26:07.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:26:07.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:26:07.137 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:26:07.205 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:07.205 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:07.245+0000 7f35c4eae780 1 bucket sync caught up with source: 2024-04-12T03:26:07.246 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.19.11, , , 00000000002.27.11, 00000000011.24.6, ] 2024-04-12T03:26:07.247 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.19.11, , , 00000000002.27.11, 00000000011.24.6, ] 2024-04-12T03:26:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:07.245+0000 7f35c4eae780 0 bucket checkpoint complete 2024-04-12T03:26:07.260 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-14 2024-04-12T03:26:07.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:26:07.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:26:07.261 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:26:07.323 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:07.324 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:07.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:07.363+0000 7fca48d9f780 1 bucket sync caught up with source: 2024-04-12T03:26:07.364 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.31.10, , , 00000000011.57.6, 00000000002.10.11, ] 2024-04-12T03:26:07.364 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.31.10, , , 00000000011.57.6, 00000000002.10.11, ] 2024-04-12T03:26:07.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:07.363+0000 7fca48d9f780 0 bucket checkpoint complete 2024-04-12T03:26:07.378 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-14 zones={a2, a1} 2024-04-12T03:26:07.378 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.378 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.378 DEBUG:boto:Method: HEAD 2024-04-12T03:26:07.378 DEBUG:boto:Path: /dkyges-14/ 2024-04-12T03:26:07.379 DEBUG:boto:Data: 2024-04-12T03:26:07.379 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.379 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.379 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.379 DEBUG:boto:Params: {} 2024-04-12T03:26:07.379 DEBUG:boto:Token: None 2024-04-12T03:26:07.379 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/ 2024-04-12T03:26:07.379 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kDjIhxhxW4/WPwuS93fpWqN3U7I= 2024-04-12T03:26:07.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kDjIhxhxW4/WPwuS93fpWqN3U7I=', 'Content-Length': '0'} 2024-04-12T03:26:07.382 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cad0ef688aeb0e3b-006618a9cf-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.382 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.382 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.382 DEBUG:boto:Method: HEAD 2024-04-12T03:26:07.383 DEBUG:boto:Path: /dkyges-14/ 2024-04-12T03:26:07.383 DEBUG:boto:Data: 2024-04-12T03:26:07.383 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.383 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.383 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.383 DEBUG:boto:Params: {} 2024-04-12T03:26:07.383 DEBUG:boto:Token: None 2024-04-12T03:26:07.383 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/ 2024-04-12T03:26:07.383 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kDjIhxhxW4/WPwuS93fpWqN3U7I= 2024-04-12T03:26:07.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kDjIhxhxW4/WPwuS93fpWqN3U7I=', 'Content-Length': '0'} 2024-04-12T03:26:07.386 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b2413062e29decf-006618a9cf-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.386 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:26:07.386 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.386 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.386 DEBUG:boto:path=/dkyges-14/?versions&max-keys=999 2024-04-12T03:26:07.386 DEBUG:boto:auth_path=/dkyges-14/?versions&max-keys=999 2024-04-12T03:26:07.386 DEBUG:boto:Method: GET 2024-04-12T03:26:07.386 DEBUG:boto:Path: /dkyges-14/?versions&max-keys=999 2024-04-12T03:26:07.386 DEBUG:boto:Data: 2024-04-12T03:26:07.387 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.387 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.387 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.387 DEBUG:boto:Params: {} 2024-04-12T03:26:07.387 DEBUG:boto:Token: None 2024-04-12T03:26:07.387 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/?versions 2024-04-12T03:26:07.387 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+nwNQ7OgNPxBKd70CfoFtCs/sTY= 2024-04-12T03:26:07.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+nwNQ7OgNPxBKd70CfoFtCs/sTY=', 'Content-Length': '0'} 2024-04-12T03:26:07.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003bd6c988353b311-006618a9cf-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.390 DEBUG:boto:b'dkyges-14999falsedummyH5mOPWoRlWY-PmXX4nbTLN5Wikt68oetrue2024-04-12T03:24:46.798Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Cqfn87I7vLNs1MeceQitarGsoRdD1pEtrue2024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2N26z7de3t6pW-GMELJHxxRYWiKhUwcFtrue2024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.390 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:26:07.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-12T03:26:07.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-12T03:26:07.390 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:26:07.390 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.391 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.391 DEBUG:boto:path=/dkyges-14/?versions&max-keys=999 2024-04-12T03:26:07.391 DEBUG:boto:auth_path=/dkyges-14/?versions&max-keys=999 2024-04-12T03:26:07.391 DEBUG:boto:Method: GET 2024-04-12T03:26:07.391 DEBUG:boto:Path: /dkyges-14/?versions&max-keys=999 2024-04-12T03:26:07.391 DEBUG:boto:Data: 2024-04-12T03:26:07.391 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.391 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.391 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.391 DEBUG:boto:Params: {} 2024-04-12T03:26:07.391 DEBUG:boto:Token: None 2024-04-12T03:26:07.392 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/?versions 2024-04-12T03:26:07.392 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+nwNQ7OgNPxBKd70CfoFtCs/sTY= 2024-04-12T03:26:07.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+nwNQ7OgNPxBKd70CfoFtCs/sTY=', 'Content-Length': '0'} 2024-04-12T03:26:07.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015426bfba27540a7-006618a9cf-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.394 DEBUG:boto:b'dkyges-14999falsedummyH5mOPWoRlWY-PmXX4nbTLN5Wikt68oetrue2024-04-12T03:24:46.798Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Cqfn87I7vLNs1MeceQitarGsoRdD1pEtrue2024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2N26z7de3t6pW-GMELJHxxRYWiKhUwcFtrue2024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.394 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:26:07.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2024-04-12T03:26:07.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2024-04-12T03:26:07.395 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.395 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.395 DEBUG:boto:path=/dkyges-14/?versions&max-keys=999 2024-04-12T03:26:07.395 DEBUG:boto:auth_path=/dkyges-14/?versions&max-keys=999 2024-04-12T03:26:07.395 DEBUG:boto:Method: GET 2024-04-12T03:26:07.395 DEBUG:boto:Path: /dkyges-14/?versions&max-keys=999 2024-04-12T03:26:07.395 DEBUG:boto:Data: 2024-04-12T03:26:07.395 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.395 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.395 DEBUG:boto:Params: {} 2024-04-12T03:26:07.396 DEBUG:boto:Token: None 2024-04-12T03:26:07.396 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/?versions 2024-04-12T03:26:07.396 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+nwNQ7OgNPxBKd70CfoFtCs/sTY= 2024-04-12T03:26:07.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+nwNQ7OgNPxBKd70CfoFtCs/sTY=', 'Content-Length': '0'} 2024-04-12T03:26:07.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aa57a2c615c8088-006618a9cf-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.398 DEBUG:boto:b'dkyges-14999falsedummyH5mOPWoRlWY-PmXX4nbTLN5Wikt68oetrue2024-04-12T03:24:46.798Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Cqfn87I7vLNs1MeceQitarGsoRdD1pEtrue2024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2N26z7de3t6pW-GMELJHxxRYWiKhUwcFtrue2024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.399 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.399 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.399 DEBUG:boto:path=/dkyges-14/?versions&max-keys=999 2024-04-12T03:26:07.399 DEBUG:boto:auth_path=/dkyges-14/?versions&max-keys=999 2024-04-12T03:26:07.399 DEBUG:boto:Method: GET 2024-04-12T03:26:07.399 DEBUG:boto:Path: /dkyges-14/?versions&max-keys=999 2024-04-12T03:26:07.399 DEBUG:boto:Data: 2024-04-12T03:26:07.399 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.399 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.400 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.400 DEBUG:boto:Params: {} 2024-04-12T03:26:07.400 DEBUG:boto:Token: None 2024-04-12T03:26:07.400 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/?versions 2024-04-12T03:26:07.400 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+nwNQ7OgNPxBKd70CfoFtCs/sTY= 2024-04-12T03:26:07.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+nwNQ7OgNPxBKd70CfoFtCs/sTY=', 'Content-Length': '0'} 2024-04-12T03:26:07.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003523657b792a4200-006618a9cf-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.402 DEBUG:boto:b'dkyges-14999falsedummyH5mOPWoRlWY-PmXX4nbTLN5Wikt68oetrue2024-04-12T03:24:46.798Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1Cqfn87I7vLNs1MeceQitarGsoRdD1pEtrue2024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2N26z7de3t6pW-GMELJHxxRYWiKhUwcFtrue2024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:26:07.403 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:26:07.403 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:26:07.403 DEBUG:boto:path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.403 DEBUG:boto:auth_path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.403 DEBUG:boto:Method: GET 2024-04-12T03:26:07.403 DEBUG:boto:Path: /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.403 DEBUG:boto:Data: 2024-04-12T03:26:07.403 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.403 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.404 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.404 DEBUG:boto:Params: {} 2024-04-12T03:26:07.404 DEBUG:boto:Token: None 2024-04-12T03:26:07.404 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.404 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/RNG/+VIFgZPps4LaIfmKSsRHeo= 2024-04-12T03:26:07.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/RNG/+VIFgZPps4LaIfmKSsRHeo=', 'Content-Length': '0'} 2024-04-12T03:26:07.406 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a8db5ac58ede07fb-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.406 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:26:07.406 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:26:07.406 DEBUG:boto:path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.406 DEBUG:boto:auth_path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.406 DEBUG:boto:Method: GET 2024-04-12T03:26:07.406 DEBUG:boto:Path: /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.406 DEBUG:boto:Data: 2024-04-12T03:26:07.407 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.407 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.407 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.407 DEBUG:boto:Params: {} 2024-04-12T03:26:07.407 DEBUG:boto:Token: None 2024-04-12T03:26:07.407 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.407 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/RNG/+VIFgZPps4LaIfmKSsRHeo= 2024-04-12T03:26:07.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/RNG/+VIFgZPps4LaIfmKSsRHeo=', 'Content-Length': '0'} 2024-04-12T03:26:07.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b3be35c24a87c3c-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.409 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:26:07.409 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:26:07.409 DEBUG:boto:path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.409 DEBUG:boto:auth_path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.409 DEBUG:boto:Method: HEAD 2024-04-12T03:26:07.410 DEBUG:boto:Path: /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.410 DEBUG:boto:Data: 2024-04-12T03:26:07.410 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.410 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.410 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.410 DEBUG:boto:Params: {} 2024-04-12T03:26:07.410 DEBUG:boto:Token: None 2024-04-12T03:26:07.410 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.410 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iZaRoM0ryoTkuxW/QD0pfa6JDEw= 2024-04-12T03:26:07.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iZaRoM0ryoTkuxW/QD0pfa6JDEw=', 'Content-Length': '0'} 2024-04-12T03:26:07.412 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009ad1bbfd13d5387a-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.412 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:26:07.412 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:26:07.412 DEBUG:boto:path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.412 DEBUG:boto:auth_path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.412 DEBUG:boto:Method: HEAD 2024-04-12T03:26:07.412 DEBUG:boto:Path: /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.413 DEBUG:boto:Data: 2024-04-12T03:26:07.413 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.413 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.413 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.413 DEBUG:boto:Params: {} 2024-04-12T03:26:07.413 DEBUG:boto:Token: None 2024-04-12T03:26:07.413 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.413 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iZaRoM0ryoTkuxW/QD0pfa6JDEw= 2024-04-12T03:26:07.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iZaRoM0ryoTkuxW/QD0pfa6JDEw=', 'Content-Length': '0'} 2024-04-12T03:26:07.415 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004ff44ba41f542203-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:26:07.416 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:26:07.416 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:26:07.416 DEBUG:boto:path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.416 DEBUG:boto:auth_path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.416 DEBUG:boto:Method: GET 2024-04-12T03:26:07.416 DEBUG:boto:Path: /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.416 DEBUG:boto:Data: 2024-04-12T03:26:07.416 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.416 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.416 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.416 DEBUG:boto:Params: {} 2024-04-12T03:26:07.417 DEBUG:boto:Token: None 2024-04-12T03:26:07.417 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.417 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/RNG/+VIFgZPps4LaIfmKSsRHeo= 2024-04-12T03:26:07.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/RNG/+VIFgZPps4LaIfmKSsRHeo=', 'Content-Length': '0'} 2024-04-12T03:26:07.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b6badd0df124d8ab-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.419 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:26:07.419 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:26:07.419 DEBUG:boto:path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.419 DEBUG:boto:auth_path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.419 DEBUG:boto:Method: GET 2024-04-12T03:26:07.419 DEBUG:boto:Path: /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.419 DEBUG:boto:Data: 2024-04-12T03:26:07.419 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.419 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.419 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.419 DEBUG:boto:Params: {} 2024-04-12T03:26:07.420 DEBUG:boto:Token: None 2024-04-12T03:26:07.420 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.420 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/RNG/+VIFgZPps4LaIfmKSsRHeo= 2024-04-12T03:26:07.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/RNG/+VIFgZPps4LaIfmKSsRHeo=', 'Content-Length': '0'} 2024-04-12T03:26:07.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005271c7c68e1fae0a-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.422 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.422 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.422 DEBUG:boto:path=/dkyges-14/?prefix=dummy 2024-04-12T03:26:07.422 DEBUG:boto:auth_path=/dkyges-14/?prefix=dummy 2024-04-12T03:26:07.422 DEBUG:boto:Method: GET 2024-04-12T03:26:07.422 DEBUG:boto:Path: /dkyges-14/?prefix=dummy 2024-04-12T03:26:07.423 DEBUG:boto:Data: 2024-04-12T03:26:07.423 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.423 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.423 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.423 DEBUG:boto:Params: {} 2024-04-12T03:26:07.423 DEBUG:boto:Token: None 2024-04-12T03:26:07.423 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/ 2024-04-12T03:26:07.423 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o= 2024-04-12T03:26:07.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o=', 'Content-Length': '0'} 2024-04-12T03:26:07.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000689c3b01af5c867a-006618a9cf-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.426 DEBUG:boto:b'dkyges-14dummy1000falsedummy2024-04-12T03:24:46.798Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.426 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.426 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.426 DEBUG:boto:path=/dkyges-14/?prefix=dummy 2024-04-12T03:26:07.426 DEBUG:boto:auth_path=/dkyges-14/?prefix=dummy 2024-04-12T03:26:07.426 DEBUG:boto:Method: GET 2024-04-12T03:26:07.427 DEBUG:boto:Path: /dkyges-14/?prefix=dummy 2024-04-12T03:26:07.427 DEBUG:boto:Data: 2024-04-12T03:26:07.427 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.427 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.427 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.427 DEBUG:boto:Params: {} 2024-04-12T03:26:07.427 DEBUG:boto:Token: None 2024-04-12T03:26:07.427 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/ 2024-04-12T03:26:07.427 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o= 2024-04-12T03:26:07.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o=', 'Content-Length': '0'} 2024-04-12T03:26:07.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068efd6a8102db871-006618a9cf-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.470 DEBUG:boto:b'dkyges-14dummy1000falsedummy2024-04-12T03:24:46.798Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.471 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:26:07.471 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:26:07.471 DEBUG:boto:Method: HEAD 2024-04-12T03:26:07.471 DEBUG:boto:Path: /dkyges-14/dummy 2024-04-12T03:26:07.471 DEBUG:boto:Data: 2024-04-12T03:26:07.471 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.471 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.471 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.471 DEBUG:boto:Params: {} 2024-04-12T03:26:07.472 DEBUG:boto:Token: None 2024-04-12T03:26:07.472 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/dummy 2024-04-12T03:26:07.472 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EASVBVrRakrfvIYr55F2Drb1sDM= 2024-04-12T03:26:07.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EASVBVrRakrfvIYr55F2Drb1sDM=', 'Content-Length': '0'} 2024-04-12T03:26:07.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000020a550bd881af40f-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.474 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:26:07.475 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:26:07.475 DEBUG:boto:Method: HEAD 2024-04-12T03:26:07.475 DEBUG:boto:Path: /dkyges-14/dummy 2024-04-12T03:26:07.475 DEBUG:boto:Data: 2024-04-12T03:26:07.475 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.475 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.475 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.475 DEBUG:boto:Params: {} 2024-04-12T03:26:07.475 DEBUG:boto:Token: None 2024-04-12T03:26:07.475 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/dummy 2024-04-12T03:26:07.476 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EASVBVrRakrfvIYr55F2Drb1sDM= 2024-04-12T03:26:07.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EASVBVrRakrfvIYr55F2Drb1sDM=', 'Content-Length': '0'} 2024-04-12T03:26:07.478 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000cce7015ef07d6520-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:26:07.478 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:26:07.478 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:26:07.478 DEBUG:boto:path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.478 DEBUG:boto:auth_path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.478 DEBUG:boto:Method: GET 2024-04-12T03:26:07.478 DEBUG:boto:Path: /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.479 DEBUG:boto:Data: 2024-04-12T03:26:07.479 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.479 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.479 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.479 DEBUG:boto:Params: {} 2024-04-12T03:26:07.479 DEBUG:boto:Token: None 2024-04-12T03:26:07.479 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.479 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/RNG/+VIFgZPps4LaIfmKSsRHeo= 2024-04-12T03:26:07.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/RNG/+VIFgZPps4LaIfmKSsRHeo=', 'Content-Length': '0'} 2024-04-12T03:26:07.481 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000a3d98d77b04c0f2-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.481 DEBUG:boto:path=/dkyges-14/dummy 2024-04-12T03:26:07.481 DEBUG:boto:auth_path=/dkyges-14/dummy 2024-04-12T03:26:07.482 DEBUG:boto:path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.482 DEBUG:boto:auth_path=/dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.482 DEBUG:boto:Method: GET 2024-04-12T03:26:07.482 DEBUG:boto:Path: /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.482 DEBUG:boto:Data: 2024-04-12T03:26:07.482 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.482 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.482 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.483 DEBUG:boto:Params: {} 2024-04-12T03:26:07.483 DEBUG:boto:Token: None 2024-04-12T03:26:07.483 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/dummy?versionId=H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe 2024-04-12T03:26:07.483 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/RNG/+VIFgZPps4LaIfmKSsRHeo= 2024-04-12T03:26:07.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/RNG/+VIFgZPps4LaIfmKSsRHeo=', 'Content-Length': '0'} 2024-04-12T03:26:07.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:24:46 GMT'), ('x-amz-version-id', 'H5mOPWoRlWY-PmXX4nbTLN5Wikt68oe'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001666d8111a4b808a-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.485 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.485 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.485 DEBUG:boto:path=/dkyges-14/?prefix=dummy 2024-04-12T03:26:07.485 DEBUG:boto:auth_path=/dkyges-14/?prefix=dummy 2024-04-12T03:26:07.485 DEBUG:boto:Method: GET 2024-04-12T03:26:07.485 DEBUG:boto:Path: /dkyges-14/?prefix=dummy 2024-04-12T03:26:07.485 DEBUG:boto:Data: 2024-04-12T03:26:07.486 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.486 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.486 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.486 DEBUG:boto:Params: {} 2024-04-12T03:26:07.486 DEBUG:boto:Token: None 2024-04-12T03:26:07.486 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/ 2024-04-12T03:26:07.486 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o= 2024-04-12T03:26:07.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o=', 'Content-Length': '0'} 2024-04-12T03:26:07.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009423d358278382ee-006618a9cf-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.489 DEBUG:boto:b'dkyges-14dummy1000falsedummy2024-04-12T03:24:46.798Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.489 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.489 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.489 DEBUG:boto:path=/dkyges-14/?prefix=dummy 2024-04-12T03:26:07.489 DEBUG:boto:auth_path=/dkyges-14/?prefix=dummy 2024-04-12T03:26:07.490 DEBUG:boto:Method: GET 2024-04-12T03:26:07.490 DEBUG:boto:Path: /dkyges-14/?prefix=dummy 2024-04-12T03:26:07.490 DEBUG:boto:Data: 2024-04-12T03:26:07.490 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.490 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.490 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.490 DEBUG:boto:Params: {} 2024-04-12T03:26:07.490 DEBUG:boto:Token: None 2024-04-12T03:26:07.490 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/ 2024-04-12T03:26:07.490 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o= 2024-04-12T03:26:07.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o=', 'Content-Length': '0'} 2024-04-12T03:26:07.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e01b3adab706b23-006618a9cf-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.493 DEBUG:boto:b'dkyges-14dummy1000falsedummy2024-04-12T03:24:46.798Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:26:07.493 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:26:07.494 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:26:07.494 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.494 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.494 DEBUG:boto:Method: GET 2024-04-12T03:26:07.494 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.494 DEBUG:boto:Data: 2024-04-12T03:26:07.494 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.494 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.494 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.494 DEBUG:boto:Params: {} 2024-04-12T03:26:07.494 DEBUG:boto:Token: None 2024-04-12T03:26:07.495 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.495 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gnlLk2QVcrm8rVnAP8dcnZFf4QU= 2024-04-12T03:26:07.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gnlLk2QVcrm8rVnAP8dcnZFf4QU=', 'Content-Length': '0'} 2024-04-12T03:26:07.497 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060b39ba75c931eb4-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.497 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:26:07.497 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:26:07.497 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.497 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.497 DEBUG:boto:Method: GET 2024-04-12T03:26:07.497 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.497 DEBUG:boto:Data: 2024-04-12T03:26:07.497 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.498 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.498 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.498 DEBUG:boto:Params: {} 2024-04-12T03:26:07.498 DEBUG:boto:Token: None 2024-04-12T03:26:07.498 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.498 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gnlLk2QVcrm8rVnAP8dcnZFf4QU= 2024-04-12T03:26:07.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gnlLk2QVcrm8rVnAP8dcnZFf4QU=', 'Content-Length': '0'} 2024-04-12T03:26:07.500 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eb41bd5e24eb9a9e-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.500 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:26:07.500 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:26:07.500 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.500 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.500 DEBUG:boto:Method: HEAD 2024-04-12T03:26:07.501 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.501 DEBUG:boto:Data: 2024-04-12T03:26:07.501 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.501 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.501 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.501 DEBUG:boto:Params: {} 2024-04-12T03:26:07.501 DEBUG:boto:Token: None 2024-04-12T03:26:07.501 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.501 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rF/1/hAUdIbdAwcnbnGB4Ujrm8g= 2024-04-12T03:26:07.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rF/1/hAUdIbdAwcnbnGB4Ujrm8g=', 'Content-Length': '0'} 2024-04-12T03:26:07.503 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009161c6558ec375e1-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.503 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:26:07.503 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:26:07.503 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.504 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.504 DEBUG:boto:Method: HEAD 2024-04-12T03:26:07.504 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.504 DEBUG:boto:Data: 2024-04-12T03:26:07.504 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.504 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.504 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.504 DEBUG:boto:Params: {} 2024-04-12T03:26:07.504 DEBUG:boto:Token: None 2024-04-12T03:26:07.504 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.504 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rF/1/hAUdIbdAwcnbnGB4Ujrm8g= 2024-04-12T03:26:07.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rF/1/hAUdIbdAwcnbnGB4Ujrm8g=', 'Content-Length': '0'} 2024-04-12T03:26:07.506 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e40e2a84abddd3dc-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:26:07.506 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:26:07.506 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:26:07.507 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.507 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.507 DEBUG:boto:Method: GET 2024-04-12T03:26:07.507 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.507 DEBUG:boto:Data: 2024-04-12T03:26:07.507 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.507 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.507 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.507 DEBUG:boto:Params: {} 2024-04-12T03:26:07.507 DEBUG:boto:Token: None 2024-04-12T03:26:07.507 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.508 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gnlLk2QVcrm8rVnAP8dcnZFf4QU= 2024-04-12T03:26:07.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gnlLk2QVcrm8rVnAP8dcnZFf4QU=', 'Content-Length': '0'} 2024-04-12T03:26:07.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f56acd6317fcd57c-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.550 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:26:07.551 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:26:07.551 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.551 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.551 DEBUG:boto:Method: GET 2024-04-12T03:26:07.551 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.551 DEBUG:boto:Data: 2024-04-12T03:26:07.551 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.551 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.551 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.551 DEBUG:boto:Params: {} 2024-04-12T03:26:07.552 DEBUG:boto:Token: None 2024-04-12T03:26:07.552 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.552 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gnlLk2QVcrm8rVnAP8dcnZFf4QU= 2024-04-12T03:26:07.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gnlLk2QVcrm8rVnAP8dcnZFf4QU=', 'Content-Length': '0'} 2024-04-12T03:26:07.554 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000920419ecce7626d3-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.554 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.554 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.554 DEBUG:boto:path=/dkyges-14/?prefix=obj-a1 2024-04-12T03:26:07.554 DEBUG:boto:auth_path=/dkyges-14/?prefix=obj-a1 2024-04-12T03:26:07.554 DEBUG:boto:Method: GET 2024-04-12T03:26:07.555 DEBUG:boto:Path: /dkyges-14/?prefix=obj-a1 2024-04-12T03:26:07.555 DEBUG:boto:Data: 2024-04-12T03:26:07.555 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.555 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.555 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.555 DEBUG:boto:Params: {} 2024-04-12T03:26:07.555 DEBUG:boto:Token: None 2024-04-12T03:26:07.555 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/ 2024-04-12T03:26:07.555 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o= 2024-04-12T03:26:07.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o=', 'Content-Length': '0'} 2024-04-12T03:26:07.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000345fbf93a2b69427-006618a9cf-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.558 DEBUG:boto:b'dkyges-14obj-a11000falseobj-a12024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.558 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.558 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.559 DEBUG:boto:path=/dkyges-14/?prefix=obj-a1 2024-04-12T03:26:07.559 DEBUG:boto:auth_path=/dkyges-14/?prefix=obj-a1 2024-04-12T03:26:07.559 DEBUG:boto:Method: GET 2024-04-12T03:26:07.559 DEBUG:boto:Path: /dkyges-14/?prefix=obj-a1 2024-04-12T03:26:07.559 DEBUG:boto:Data: 2024-04-12T03:26:07.559 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.559 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.559 DEBUG:boto:Params: {} 2024-04-12T03:26:07.559 DEBUG:boto:Token: None 2024-04-12T03:26:07.559 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/ 2024-04-12T03:26:07.560 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o= 2024-04-12T03:26:07.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o=', 'Content-Length': '0'} 2024-04-12T03:26:07.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002635cc1112fa5a67-006618a9cf-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.562 DEBUG:boto:b'dkyges-14obj-a11000falseobj-a12024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.562 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:26:07.565 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:26:07.565 DEBUG:boto:Method: HEAD 2024-04-12T03:26:07.565 DEBUG:boto:Path: /dkyges-14/obj-a1 2024-04-12T03:26:07.565 DEBUG:boto:Data: 2024-04-12T03:26:07.565 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.566 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.566 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.566 DEBUG:boto:Params: {} 2024-04-12T03:26:07.566 DEBUG:boto:Token: None 2024-04-12T03:26:07.566 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a1 2024-04-12T03:26:07.566 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8LDMRdRiSUH3GFi2tNv5fJ4xaMU= 2024-04-12T03:26:07.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8LDMRdRiSUH3GFi2tNv5fJ4xaMU=', 'Content-Length': '0'} 2024-04-12T03:26:07.569 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f3d6887f4403e31-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.569 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:26:07.569 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:26:07.569 DEBUG:boto:Method: HEAD 2024-04-12T03:26:07.569 DEBUG:boto:Path: /dkyges-14/obj-a1 2024-04-12T03:26:07.569 DEBUG:boto:Data: 2024-04-12T03:26:07.569 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.569 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.570 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.570 DEBUG:boto:Params: {} 2024-04-12T03:26:07.570 DEBUG:boto:Token: None 2024-04-12T03:26:07.570 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a1 2024-04-12T03:26:07.570 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8LDMRdRiSUH3GFi2tNv5fJ4xaMU= 2024-04-12T03:26:07.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8LDMRdRiSUH3GFi2tNv5fJ4xaMU=', 'Content-Length': '0'} 2024-04-12T03:26:07.572 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086b6954edf30a9e9-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2024-04-12T03:26:07.573 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:26:07.573 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:26:07.573 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.573 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.573 DEBUG:boto:Method: GET 2024-04-12T03:26:07.573 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.573 DEBUG:boto:Data: 2024-04-12T03:26:07.573 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.573 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.573 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.573 DEBUG:boto:Params: {} 2024-04-12T03:26:07.574 DEBUG:boto:Token: None 2024-04-12T03:26:07.574 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.574 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gnlLk2QVcrm8rVnAP8dcnZFf4QU= 2024-04-12T03:26:07.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gnlLk2QVcrm8rVnAP8dcnZFf4QU=', 'Content-Length': '0'} 2024-04-12T03:26:07.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1f7d8624561b7b6-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.618 DEBUG:boto:path=/dkyges-14/obj-a1 2024-04-12T03:26:07.618 DEBUG:boto:auth_path=/dkyges-14/obj-a1 2024-04-12T03:26:07.618 DEBUG:boto:path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.619 DEBUG:boto:auth_path=/dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.619 DEBUG:boto:Method: GET 2024-04-12T03:26:07.619 DEBUG:boto:Path: /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.619 DEBUG:boto:Data: 2024-04-12T03:26:07.619 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.619 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.619 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.619 DEBUG:boto:Params: {} 2024-04-12T03:26:07.619 DEBUG:boto:Token: None 2024-04-12T03:26:07.619 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a1?versionId=Cqfn87I7vLNs1MeceQitarGsoRdD1pE 2024-04-12T03:26:07.619 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gnlLk2QVcrm8rVnAP8dcnZFf4QU= 2024-04-12T03:26:07.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gnlLk2QVcrm8rVnAP8dcnZFf4QU=', 'Content-Length': '0'} 2024-04-12T03:26:07.621 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'Cqfn87I7vLNs1MeceQitarGsoRdD1pE'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000052d3f31ce6ad78f0-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.622 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.622 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.622 DEBUG:boto:path=/dkyges-14/?prefix=obj-a1 2024-04-12T03:26:07.622 DEBUG:boto:auth_path=/dkyges-14/?prefix=obj-a1 2024-04-12T03:26:07.622 DEBUG:boto:Method: GET 2024-04-12T03:26:07.622 DEBUG:boto:Path: /dkyges-14/?prefix=obj-a1 2024-04-12T03:26:07.622 DEBUG:boto:Data: 2024-04-12T03:26:07.622 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.622 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.623 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.623 DEBUG:boto:Params: {} 2024-04-12T03:26:07.623 DEBUG:boto:Token: None 2024-04-12T03:26:07.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/ 2024-04-12T03:26:07.623 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o= 2024-04-12T03:26:07.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o=', 'Content-Length': '0'} 2024-04-12T03:26:07.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000694fbe3dbf682a8a-006618a9cf-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.625 DEBUG:boto:b'dkyges-14obj-a11000falseobj-a12024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.626 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.626 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.626 DEBUG:boto:path=/dkyges-14/?prefix=obj-a1 2024-04-12T03:26:07.626 DEBUG:boto:auth_path=/dkyges-14/?prefix=obj-a1 2024-04-12T03:26:07.626 DEBUG:boto:Method: GET 2024-04-12T03:26:07.626 DEBUG:boto:Path: /dkyges-14/?prefix=obj-a1 2024-04-12T03:26:07.626 DEBUG:boto:Data: 2024-04-12T03:26:07.626 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.627 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.627 DEBUG:boto:Params: {} 2024-04-12T03:26:07.627 DEBUG:boto:Token: None 2024-04-12T03:26:07.627 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/ 2024-04-12T03:26:07.627 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o= 2024-04-12T03:26:07.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o=', 'Content-Length': '0'} 2024-04-12T03:26:07.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fc3e6a0cda665f4-006618a9cf-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.630 DEBUG:boto:b'dkyges-14obj-a11000falseobj-a12024-04-12T03:25:07.187Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:26:07.630 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:26:07.630 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:26:07.630 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.630 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.630 DEBUG:boto:Method: GET 2024-04-12T03:26:07.631 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.631 DEBUG:boto:Data: 2024-04-12T03:26:07.631 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.631 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.631 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.631 DEBUG:boto:Params: {} 2024-04-12T03:26:07.631 DEBUG:boto:Token: None 2024-04-12T03:26:07.631 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.631 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WaRdF66VU+L50Ry1gKepEB4FvWg= 2024-04-12T03:26:07.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WaRdF66VU+L50Ry1gKepEB4FvWg=', 'Content-Length': '0'} 2024-04-12T03:26:07.633 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012d2fe2ae90dce63-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.633 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:26:07.633 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:26:07.633 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.634 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.634 DEBUG:boto:Method: GET 2024-04-12T03:26:07.634 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.634 DEBUG:boto:Data: 2024-04-12T03:26:07.634 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.634 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.634 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.634 DEBUG:boto:Params: {} 2024-04-12T03:26:07.634 DEBUG:boto:Token: None 2024-04-12T03:26:07.634 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.634 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WaRdF66VU+L50Ry1gKepEB4FvWg= 2024-04-12T03:26:07.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WaRdF66VU+L50Ry1gKepEB4FvWg=', 'Content-Length': '0'} 2024-04-12T03:26:07.636 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009d384556bdc1fcf5-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.636 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:26:07.637 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:26:07.637 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.637 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.637 DEBUG:boto:Method: HEAD 2024-04-12T03:26:07.637 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.637 DEBUG:boto:Data: 2024-04-12T03:26:07.637 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.637 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.637 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.637 DEBUG:boto:Params: {} 2024-04-12T03:26:07.637 DEBUG:boto:Token: None 2024-04-12T03:26:07.638 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.638 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wz2dWtIaIGeQ1Uh4+nGlbOkw1LU= 2024-04-12T03:26:07.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wz2dWtIaIGeQ1Uh4+nGlbOkw1LU=', 'Content-Length': '0'} 2024-04-12T03:26:07.639 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000109bc419025fcce9-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.640 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:26:07.640 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:26:07.640 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.640 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.640 DEBUG:boto:Method: HEAD 2024-04-12T03:26:07.640 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.640 DEBUG:boto:Data: 2024-04-12T03:26:07.640 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.640 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.640 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.640 DEBUG:boto:Params: {} 2024-04-12T03:26:07.641 DEBUG:boto:Token: None 2024-04-12T03:26:07.641 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.641 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wz2dWtIaIGeQ1Uh4+nGlbOkw1LU= 2024-04-12T03:26:07.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wz2dWtIaIGeQ1Uh4+nGlbOkw1LU=', 'Content-Length': '0'} 2024-04-12T03:26:07.642 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cb9d1c0e40dca7f-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:26:07.643 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:26:07.643 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:26:07.643 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.643 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.643 DEBUG:boto:Method: GET 2024-04-12T03:26:07.643 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.643 DEBUG:boto:Data: 2024-04-12T03:26:07.643 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.643 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.643 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.644 DEBUG:boto:Params: {} 2024-04-12T03:26:07.644 DEBUG:boto:Token: None 2024-04-12T03:26:07.644 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.644 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WaRdF66VU+L50Ry1gKepEB4FvWg= 2024-04-12T03:26:07.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WaRdF66VU+L50Ry1gKepEB4FvWg=', 'Content-Length': '0'} 2024-04-12T03:26:07.646 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040bf1536ff188fe8-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.646 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:26:07.646 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:26:07.647 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.647 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.647 DEBUG:boto:Method: GET 2024-04-12T03:26:07.647 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.647 DEBUG:boto:Data: 2024-04-12T03:26:07.647 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.647 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.647 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.647 DEBUG:boto:Params: {} 2024-04-12T03:26:07.647 DEBUG:boto:Token: None 2024-04-12T03:26:07.648 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.648 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WaRdF66VU+L50Ry1gKepEB4FvWg= 2024-04-12T03:26:07.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WaRdF66VU+L50Ry1gKepEB4FvWg=', 'Content-Length': '0'} 2024-04-12T03:26:07.650 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e39100e9fb65bd33-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.694 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.694 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.695 DEBUG:boto:path=/dkyges-14/?prefix=obj-a2 2024-04-12T03:26:07.695 DEBUG:boto:auth_path=/dkyges-14/?prefix=obj-a2 2024-04-12T03:26:07.695 DEBUG:boto:Method: GET 2024-04-12T03:26:07.695 DEBUG:boto:Path: /dkyges-14/?prefix=obj-a2 2024-04-12T03:26:07.695 DEBUG:boto:Data: 2024-04-12T03:26:07.695 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.695 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.695 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.695 DEBUG:boto:Params: {} 2024-04-12T03:26:07.695 DEBUG:boto:Token: None 2024-04-12T03:26:07.695 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/ 2024-04-12T03:26:07.696 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o= 2024-04-12T03:26:07.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o=', 'Content-Length': '0'} 2024-04-12T03:26:07.698 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fbe88420ccf8810-006618a9cf-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.698 DEBUG:boto:b'dkyges-14obj-a21000falseobj-a22024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.698 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.699 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.699 DEBUG:boto:path=/dkyges-14/?prefix=obj-a2 2024-04-12T03:26:07.699 DEBUG:boto:auth_path=/dkyges-14/?prefix=obj-a2 2024-04-12T03:26:07.699 DEBUG:boto:Method: GET 2024-04-12T03:26:07.699 DEBUG:boto:Path: /dkyges-14/?prefix=obj-a2 2024-04-12T03:26:07.699 DEBUG:boto:Data: 2024-04-12T03:26:07.699 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.699 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.699 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.699 DEBUG:boto:Params: {} 2024-04-12T03:26:07.699 DEBUG:boto:Token: None 2024-04-12T03:26:07.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/ 2024-04-12T03:26:07.700 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o= 2024-04-12T03:26:07.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o=', 'Content-Length': '0'} 2024-04-12T03:26:07.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028811e63c7e0601a-006618a9cf-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.742 DEBUG:boto:b'dkyges-14obj-a21000falseobj-a22024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.743 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:26:07.743 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:26:07.743 DEBUG:boto:Method: HEAD 2024-04-12T03:26:07.743 DEBUG:boto:Path: /dkyges-14/obj-a2 2024-04-12T03:26:07.743 DEBUG:boto:Data: 2024-04-12T03:26:07.743 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.743 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.743 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.743 DEBUG:boto:Params: {} 2024-04-12T03:26:07.743 DEBUG:boto:Token: None 2024-04-12T03:26:07.744 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a2 2024-04-12T03:26:07.744 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TfR8Gl/zWys8LGwWq2Q/x4JEepc= 2024-04-12T03:26:07.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TfR8Gl/zWys8LGwWq2Q/x4JEepc=', 'Content-Length': '0'} 2024-04-12T03:26:07.746 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029195866d990ae94-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.746 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:26:07.746 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:26:07.747 DEBUG:boto:Method: HEAD 2024-04-12T03:26:07.747 DEBUG:boto:Path: /dkyges-14/obj-a2 2024-04-12T03:26:07.747 DEBUG:boto:Data: 2024-04-12T03:26:07.747 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.747 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.747 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.747 DEBUG:boto:Params: {} 2024-04-12T03:26:07.747 DEBUG:boto:Token: None 2024-04-12T03:26:07.747 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a2 2024-04-12T03:26:07.747 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TfR8Gl/zWys8LGwWq2Q/x4JEepc= 2024-04-12T03:26:07.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TfR8Gl/zWys8LGwWq2Q/x4JEepc=', 'Content-Length': '0'} 2024-04-12T03:26:07.749 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009a618b610a8756ef-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2024-04-12T03:26:07.750 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:26:07.750 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:26:07.750 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.750 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.750 DEBUG:boto:Method: GET 2024-04-12T03:26:07.750 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.750 DEBUG:boto:Data: 2024-04-12T03:26:07.751 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.751 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.751 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.751 DEBUG:boto:Params: {} 2024-04-12T03:26:07.751 DEBUG:boto:Token: None 2024-04-12T03:26:07.751 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.751 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WaRdF66VU+L50Ry1gKepEB4FvWg= 2024-04-12T03:26:07.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WaRdF66VU+L50Ry1gKepEB4FvWg=', 'Content-Length': '0'} 2024-04-12T03:26:07.753 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4a6728a04bd1417-006618a9cf-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.753 DEBUG:boto:path=/dkyges-14/obj-a2 2024-04-12T03:26:07.753 DEBUG:boto:auth_path=/dkyges-14/obj-a2 2024-04-12T03:26:07.753 DEBUG:boto:path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.754 DEBUG:boto:auth_path=/dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.754 DEBUG:boto:Method: GET 2024-04-12T03:26:07.754 DEBUG:boto:Path: /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.754 DEBUG:boto:Data: 2024-04-12T03:26:07.754 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.754 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.754 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.754 DEBUG:boto:Params: {} 2024-04-12T03:26:07.754 DEBUG:boto:Token: None 2024-04-12T03:26:07.754 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/obj-a2?versionId=N26z7de3t6pW-GMELJHxxRYWiKhUwcF 2024-04-12T03:26:07.755 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WaRdF66VU+L50Ry1gKepEB4FvWg= 2024-04-12T03:26:07.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WaRdF66VU+L50Ry1gKepEB4FvWg=', 'Content-Length': '0'} 2024-04-12T03:26:07.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:25:07 GMT'), ('x-amz-version-id', 'N26z7de3t6pW-GMELJHxxRYWiKhUwcF'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f7ba286dde83ca09-006618a9cf-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.798 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.798 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.798 DEBUG:boto:path=/dkyges-14/?prefix=obj-a2 2024-04-12T03:26:07.799 DEBUG:boto:auth_path=/dkyges-14/?prefix=obj-a2 2024-04-12T03:26:07.799 DEBUG:boto:Method: GET 2024-04-12T03:26:07.799 DEBUG:boto:Path: /dkyges-14/?prefix=obj-a2 2024-04-12T03:26:07.799 DEBUG:boto:Data: 2024-04-12T03:26:07.799 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.799 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:07.799 DEBUG:boto:Port: 8001 2024-04-12T03:26:07.799 DEBUG:boto:Params: {} 2024-04-12T03:26:07.799 DEBUG:boto:Token: None 2024-04-12T03:26:07.799 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/ 2024-04-12T03:26:07.799 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o= 2024-04-12T03:26:07.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o=', 'Content-Length': '0'} 2024-04-12T03:26:07.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042a74588dcf03deb-006618a9cf-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.802 DEBUG:boto:b'dkyges-14obj-a21000falseobj-a22024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.803 DEBUG:boto:path=/dkyges-14/ 2024-04-12T03:26:07.803 DEBUG:boto:auth_path=/dkyges-14/ 2024-04-12T03:26:07.803 DEBUG:boto:path=/dkyges-14/?prefix=obj-a2 2024-04-12T03:26:07.803 DEBUG:boto:auth_path=/dkyges-14/?prefix=obj-a2 2024-04-12T03:26:07.803 DEBUG:boto:Method: GET 2024-04-12T03:26:07.803 DEBUG:boto:Path: /dkyges-14/?prefix=obj-a2 2024-04-12T03:26:07.803 DEBUG:boto:Data: 2024-04-12T03:26:07.803 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.803 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.803 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.803 DEBUG:boto:Params: {} 2024-04-12T03:26:07.804 DEBUG:boto:Token: None 2024-04-12T03:26:07.804 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:07 GMT /dkyges-14/ 2024-04-12T03:26:07.804 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o= 2024-04-12T03:26:07.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ORVCjSMGOJkMk23asTKaQKg5i8o=', 'Content-Length': '0'} 2024-04-12T03:26:07.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fb025a52c2ea26f-006618a9cf-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.806 DEBUG:boto:b'dkyges-14obj-a21000falseobj-a22024-04-12T03:25:07.295Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:07.806 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-14 zones={a2, a1} 2024-04-12T03:26:07.807 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2024-04-12T03:26:07.808 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:07.808 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:07.808 DEBUG:boto:Method: PUT 2024-04-12T03:26:07.808 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:07.808 DEBUG:boto:Data: 2024-04-12T03:26:07.808 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.809 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.809 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.809 DEBUG:boto:Params: {} 2024-04-12T03:26:07.809 DEBUG:boto:Token: None 2024-04-12T03:26:07.809 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:26:07 GMT /dkyges-15/ 2024-04-12T03:26:07.809 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:64QTPjdvLsrarXtBwT43iAbMS/w= 2024-04-12T03:26:07.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:64QTPjdvLsrarXtBwT43iAbMS/w=', 'Content-Length': '0'} 2024-04-12T03:26:07.826 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f031a831576fbb52-006618a9cf-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.826 DEBUG:rgw_multi.tests:created bucket=dkyges-15 2024-04-12T03:26:07.826 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:07.826 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:07.826 DEBUG:boto:path=/dkyges-15/?versioning 2024-04-12T03:26:07.826 DEBUG:boto:auth_path=/dkyges-15/?versioning 2024-04-12T03:26:07.826 DEBUG:boto:Method: PUT 2024-04-12T03:26:07.826 DEBUG:boto:Path: /dkyges-15/?versioning 2024-04-12T03:26:07.826 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:26:07.827 DEBUG:boto:Headers: {} 2024-04-12T03:26:07.827 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:07.827 DEBUG:boto:Port: 8000 2024-04-12T03:26:07.827 DEBUG:boto:Params: {} 2024-04-12T03:26:07.827 DEBUG:boto:Token: None 2024-04-12T03:26:07.827 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:26:07 GMT /dkyges-15/?versioning 2024-04-12T03:26:07.827 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:699T4ETACGzL1BVTr//kF06Atl8= 2024-04-12T03:26:07.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:699T4ETACGzL1BVTr//kF06Atl8=', 'Content-Length': '228'} 2024-04-12T03:26:07.833 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fc0008335c388e02-006618a9cf-110f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:07.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:07.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:07.833 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:26:07.890 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:07.890 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:07.925 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:26:07.939 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:26:07.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:26:07.939 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:26:07.940 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:26:07.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:26:07.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:26:07.940 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:26:07.940 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:26:07.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:26:07.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:26:07.940 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:26:07.940 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:26:07.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:26:07.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:26:07.940 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892367.832700_136.1", 2024-04-12T03:26:07.941 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:26:07.832700Z" 2024-04-12T03:26:07.941 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:26:07.941 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:26:07.942 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892367.832700_136.1'} 2024-04-12T03:26:07.942 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:26:07.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:07.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:07.942 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:26:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:08.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:08.035+0000 7fb459212780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:08.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:08.036+0000 7fb459212780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:08.050 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:26:08.050 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:26:08.050 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:26:08.051 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:26:08.051 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:26:08.051 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:26:08.051 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:26:08.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:26:08.051 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:26:08.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:26:08.051 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:26:08.051 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:26:08.051 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:26:08.051 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:26:08.052 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:26:08.052 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:26:08.052 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:26:08.052 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:26:08.052 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:26:08.052 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:26:08.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:26:08.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:26:08.052 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:26:08.052 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:26:08.052 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:26:08.053 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:26:08.053 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:26:08.053 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:26:08.053 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:26:08.053 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:26:08.053 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:26:08.053 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:26:08.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:26:08.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:26:08.053 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:26:08.053 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:26:08.053 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:26:08.054 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:26:08.054 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:26:08.054 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:26:08.054 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:26:08.054 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:26:08.054 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:26:08.054 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:26:08.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:26:08.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:26:08.054 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:26:08.054 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:26:08.054 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:26:08.055 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892367.816483_131.1", 2024-04-12T03:26:08.055 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:26:08.055 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:26:08.055 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:26:08.055 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:26:07.816483Z", 2024-04-12T03:26:08.055 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:26:08.055 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:26:08.055 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:26:08.055 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:26:08.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:26:08.055 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:26:08.055 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:26:08.055 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:26:08.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:26:08.056 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:26:08Z" 2024-04-12T03:26:08.056 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:26:08.057 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892367.816483_131.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:26:07.816483Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:26:08Z" } 2024-04-12T03:26:08.057 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892367.816483_131.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:26:07.816483Z', 'realm_epoch': 2}}] 2024-04-12T03:26:08.057 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892367.832700_136.1'} 2024-04-12T03:26:08.057 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892367.816483_131.1'} 2024-04-12T03:26:08.057 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892367.832700_136.1 target=1_1712892367.816483_131.1 2024-04-12T03:26:13.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:13.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:26:13.058 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:26:13.120 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:13.120 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:13.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:13.152+0000 7f8c0521b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:13.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:13.154+0000 7f8c0521b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:26:13.169 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:26:13.169 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:26:13.170 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:26:13.170 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:26:13.170 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:26:13.170 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:26:13.170 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:26:13.170 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:26:13.170 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:26:13.170 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:26:13.170 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:26:13.170 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:26:13.171 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:26:13.171 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:26:13.171 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:26:13.171 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:26:13.171 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:26:13.171 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:26:13.171 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:26:13.171 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:26:13.171 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:26:13.171 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:26:13.171 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:26:13.171 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:26:13.172 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:26:13.172 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:26:13.172 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:26:13.172 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:26:13.172 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:26:13.172 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:26:13.172 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:26:13.172 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:26:13.172 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:26:13.172 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:26:13.172 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:26:13.172 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:26:13.173 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:26:13.173 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:26:13.173 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:26:13.173 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:26:13.173 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:26:13.173 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:26:13.173 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:26:13.173 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:26:13.173 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:26:13.173 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:26:13.173 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:26:13.174 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:26:13.174 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:26:13.174 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892367.832700_136.1", 2024-04-12T03:26:13.174 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:26:13.174 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:26:13.174 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:26:13.174 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:26:07.832700Z", 2024-04-12T03:26:13.174 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:26:13.174 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:26:13.174 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:26:13.175 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:26:13.175 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:26:13.175 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:26:13.175 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:26:13.175 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:26:13.175 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:26:13.175 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:26:13Z" 2024-04-12T03:26:13.175 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:26:13.176 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892367.832700_136.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:26:07.832700Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:26:13Z" } 2024-04-12T03:26:13.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892367.832700_136.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:26:07.832700Z', 'realm_epoch': 2}}] 2024-04-12T03:26:13.176 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892367.832700_136.1'} 2024-04-12T03:26:13.177 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892367.832700_136.1'} 2024-04-12T03:26:13.177 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:26:13.177 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:13.177 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:13.177 DEBUG:boto:Method: HEAD 2024-04-12T03:26:13.177 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:13.177 DEBUG:boto:Data: 2024-04-12T03:26:13.177 DEBUG:boto:Headers: {} 2024-04-12T03:26:13.177 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:13.178 DEBUG:boto:Port: 8000 2024-04-12T03:26:13.178 DEBUG:boto:Params: {} 2024-04-12T03:26:13.178 DEBUG:boto:Token: None 2024-04-12T03:26:13.178 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:13 GMT /dkyges-15/ 2024-04-12T03:26:13.178 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RF44jI5OQHVegkfqPipupsdWHI0= 2024-04-12T03:26:13.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RF44jI5OQHVegkfqPipupsdWHI0=', 'Content-Length': '0'} 2024-04-12T03:26:13.181 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e843e553e218f3b-006618a9d5-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:13.181 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:26:13.181 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:26:13.181 DEBUG:boto:Method: PUT 2024-04-12T03:26:13.181 DEBUG:boto:Path: /dkyges-15/dummy 2024-04-12T03:26:13.181 DEBUG:boto:Data: 2024-04-12T03:26:13.181 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:26:13.181 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:13.182 DEBUG:boto:Port: 8000 2024-04-12T03:26:13.182 DEBUG:boto:Params: {} 2024-04-12T03:26:13.182 DEBUG:boto:Token: None 2024-04-12T03:26:13.182 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:26:13 GMT /dkyges-15/dummy 2024-04-12T03:26:13.182 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lTMgz16BVrW7RpBM+ZXGCFeXFa0= 2024-04-12T03:26:13.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lTMgz16BVrW7RpBM+ZXGCFeXFa0='} 2024-04-12T03:26:13.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-amz-request-id', 'tx000009b9aa3191c2b9d06-006618a9d5-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:13.194 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-15 2024-04-12T03:26:13.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:26:13.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:26:13.195 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:26:13.257 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:13.257 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:13.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:13.297+0000 7f4f24398780 1 waiting to reach incremental sync.. 2024-04-12T03:26:18.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:18.297+0000 7f4f24398780 1 waiting to reach incremental sync.. 2024-04-12T03:26:23.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:23.298+0000 7f4f24398780 1 waiting to reach incremental sync.. 2024-04-12T03:26:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:28.300+0000 7f4f24398780 1 bucket sync caught up with source: 2024-04-12T03:26:28.301 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.37.11, , , , , ] 2024-04-12T03:26:28.301 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.37.11, , , , , ] 2024-04-12T03:26:28.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:28.300+0000 7f4f24398780 0 bucket checkpoint complete 2024-04-12T03:26:28.317 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-15 2024-04-12T03:26:28.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:26:28.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:26:28.318 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:26:28.383 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:28.384 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:28.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:28.423+0000 7f09165a5780 1 waiting to reach incremental sync.. 2024-04-12T03:26:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:33.422+0000 7f09165a5780 1 waiting to reach incremental sync.. 2024-04-12T03:26:38.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:38.422+0000 7f09165a5780 1 waiting to reach incremental sync.. 2024-04-12T03:26:43.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:43.423+0000 7f09165a5780 1 waiting to reach incremental sync.. 2024-04-12T03:26:48.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:48.425+0000 7f09165a5780 1 bucket sync caught up with source: 2024-04-12T03:26:48.427 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.49.11, , , , , ] 2024-04-12T03:26:48.427 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.49.11, , , , , ] 2024-04-12T03:26:48.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:48.425+0000 7f09165a5780 0 bucket checkpoint complete 2024-04-12T03:26:48.443 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-15 zones={a2, a1} 2024-04-12T03:26:48.443 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.443 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.443 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.443 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:48.443 DEBUG:boto:Data: 2024-04-12T03:26:48.443 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.443 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.444 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.444 DEBUG:boto:Params: {} 2024-04-12T03:26:48.444 DEBUG:boto:Token: None 2024-04-12T03:26:48.444 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.444 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo= 2024-04-12T03:26:48.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo=', 'Content-Length': '0'} 2024-04-12T03:26:48.447 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec9c28cece097dac-006618a9f8-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.448 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.448 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.448 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.448 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:48.448 DEBUG:boto:Data: 2024-04-12T03:26:48.448 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.448 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.448 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.448 DEBUG:boto:Params: {} 2024-04-12T03:26:48.448 DEBUG:boto:Token: None 2024-04-12T03:26:48.449 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.449 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo= 2024-04-12T03:26:48.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo=', 'Content-Length': '0'} 2024-04-12T03:26:48.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3e63fc1b33127bd-006618a9f8-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.451 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:26:48.451 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.451 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.451 DEBUG:boto:path=/dkyges-15/?versions&max-keys=999 2024-04-12T03:26:48.451 DEBUG:boto:auth_path=/dkyges-15/?versions&max-keys=999 2024-04-12T03:26:48.452 DEBUG:boto:Method: GET 2024-04-12T03:26:48.452 DEBUG:boto:Path: /dkyges-15/?versions&max-keys=999 2024-04-12T03:26:48.452 DEBUG:boto:Data: 2024-04-12T03:26:48.452 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.452 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.452 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.452 DEBUG:boto:Params: {} 2024-04-12T03:26:48.452 DEBUG:boto:Token: None 2024-04-12T03:26:48.452 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/?versions 2024-04-12T03:26:48.452 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rfye7jBJa2fec7dp5rAGyBFDpHY= 2024-04-12T03:26:48.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rfye7jBJa2fec7dp5rAGyBFDpHY=', 'Content-Length': '0'} 2024-04-12T03:26:48.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a52a2cfa5343f3f-006618a9f8-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.498 DEBUG:boto:b'dkyges-15999falsedummy6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-true2024-04-12T03:26:13.184Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:48.499 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:26:48.499 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:26:48.499 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.499 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.499 DEBUG:boto:path=/dkyges-15/?versions&max-keys=999 2024-04-12T03:26:48.500 DEBUG:boto:auth_path=/dkyges-15/?versions&max-keys=999 2024-04-12T03:26:48.500 DEBUG:boto:Method: GET 2024-04-12T03:26:48.500 DEBUG:boto:Path: /dkyges-15/?versions&max-keys=999 2024-04-12T03:26:48.500 DEBUG:boto:Data: 2024-04-12T03:26:48.500 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.500 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.500 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.500 DEBUG:boto:Params: {} 2024-04-12T03:26:48.500 DEBUG:boto:Token: None 2024-04-12T03:26:48.500 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/?versions 2024-04-12T03:26:48.501 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rfye7jBJa2fec7dp5rAGyBFDpHY= 2024-04-12T03:26:48.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rfye7jBJa2fec7dp5rAGyBFDpHY=', 'Content-Length': '0'} 2024-04-12T03:26:48.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e81580e1972e65e0-006618a9f8-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.503 DEBUG:boto:b'dkyges-15999falsedummy6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-true2024-04-12T03:26:13.184Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:48.503 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:26:48.504 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.504 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.504 DEBUG:boto:path=/dkyges-15/?versions&max-keys=999 2024-04-12T03:26:48.504 DEBUG:boto:auth_path=/dkyges-15/?versions&max-keys=999 2024-04-12T03:26:48.504 DEBUG:boto:Method: GET 2024-04-12T03:26:48.504 DEBUG:boto:Path: /dkyges-15/?versions&max-keys=999 2024-04-12T03:26:48.504 DEBUG:boto:Data: 2024-04-12T03:26:48.504 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.504 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.504 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.505 DEBUG:boto:Params: {} 2024-04-12T03:26:48.505 DEBUG:boto:Token: None 2024-04-12T03:26:48.505 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/?versions 2024-04-12T03:26:48.505 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rfye7jBJa2fec7dp5rAGyBFDpHY= 2024-04-12T03:26:48.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rfye7jBJa2fec7dp5rAGyBFDpHY=', 'Content-Length': '0'} 2024-04-12T03:26:48.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008616aa77dbcac21f-006618a9f8-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.508 DEBUG:boto:b'dkyges-15999falsedummy6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-true2024-04-12T03:26:13.184Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:48.508 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.508 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.508 DEBUG:boto:path=/dkyges-15/?versions&max-keys=999 2024-04-12T03:26:48.508 DEBUG:boto:auth_path=/dkyges-15/?versions&max-keys=999 2024-04-12T03:26:48.508 DEBUG:boto:Method: GET 2024-04-12T03:26:48.508 DEBUG:boto:Path: /dkyges-15/?versions&max-keys=999 2024-04-12T03:26:48.508 DEBUG:boto:Data: 2024-04-12T03:26:48.508 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.509 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.509 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.509 DEBUG:boto:Params: {} 2024-04-12T03:26:48.509 DEBUG:boto:Token: None 2024-04-12T03:26:48.509 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/?versions 2024-04-12T03:26:48.509 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rfye7jBJa2fec7dp5rAGyBFDpHY= 2024-04-12T03:26:48.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rfye7jBJa2fec7dp5rAGyBFDpHY=', 'Content-Length': '0'} 2024-04-12T03:26:48.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e77a49c27d322a5-006618a9f8-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.512 DEBUG:boto:b'dkyges-15999falsedummy6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-true2024-04-12T03:26:13.184Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:48.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:26:48.512 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:26:48.512 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:26:48.512 DEBUG:boto:path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.512 DEBUG:boto:auth_path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.513 DEBUG:boto:Method: GET 2024-04-12T03:26:48.513 DEBUG:boto:Path: /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.513 DEBUG:boto:Data: 2024-04-12T03:26:48.513 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.513 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.513 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.513 DEBUG:boto:Params: {} 2024-04-12T03:26:48.513 DEBUG:boto:Token: None 2024-04-12T03:26:48.513 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.513 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+fHKTlbP0qVOz97R/rM0Aky5pgg= 2024-04-12T03:26:48.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+fHKTlbP0qVOz97R/rM0Aky5pgg=', 'Content-Length': '0'} 2024-04-12T03:26:48.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006353227f40b0a3d5-006618a9f8-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.516 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:26:48.516 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:26:48.516 DEBUG:boto:path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.516 DEBUG:boto:auth_path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.516 DEBUG:boto:Method: GET 2024-04-12T03:26:48.516 DEBUG:boto:Path: /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.516 DEBUG:boto:Data: 2024-04-12T03:26:48.516 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.516 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.516 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.516 DEBUG:boto:Params: {} 2024-04-12T03:26:48.517 DEBUG:boto:Token: None 2024-04-12T03:26:48.517 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.517 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+fHKTlbP0qVOz97R/rM0Aky5pgg= 2024-04-12T03:26:48.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+fHKTlbP0qVOz97R/rM0Aky5pgg=', 'Content-Length': '0'} 2024-04-12T03:26:48.519 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c7dd9e282e0316f-006618a9f8-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.519 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:26:48.519 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:26:48.519 DEBUG:boto:path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.519 DEBUG:boto:auth_path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.519 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.519 DEBUG:boto:Path: /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.520 DEBUG:boto:Data: 2024-04-12T03:26:48.520 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.520 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.520 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.520 DEBUG:boto:Params: {} 2024-04-12T03:26:48.520 DEBUG:boto:Token: None 2024-04-12T03:26:48.520 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.520 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tCzkOatNIY5kPe0GFlqEYjSQBmA= 2024-04-12T03:26:48.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tCzkOatNIY5kPe0GFlqEYjSQBmA=', 'Content-Length': '0'} 2024-04-12T03:26:48.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000076a5758676f98e52-006618a9f8-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.522 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:26:48.522 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:26:48.522 DEBUG:boto:path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.522 DEBUG:boto:auth_path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.522 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.523 DEBUG:boto:Path: /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.523 DEBUG:boto:Data: 2024-04-12T03:26:48.523 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.523 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.523 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.523 DEBUG:boto:Params: {} 2024-04-12T03:26:48.523 DEBUG:boto:Token: None 2024-04-12T03:26:48.523 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.523 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tCzkOatNIY5kPe0GFlqEYjSQBmA= 2024-04-12T03:26:48.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tCzkOatNIY5kPe0GFlqEYjSQBmA=', 'Content-Length': '0'} 2024-04-12T03:26:48.525 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000607d8e68242c0d47-006618a9f8-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:26:48.526 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:26:48.526 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:26:48.526 DEBUG:boto:path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.526 DEBUG:boto:auth_path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.526 DEBUG:boto:Method: GET 2024-04-12T03:26:48.526 DEBUG:boto:Path: /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.526 DEBUG:boto:Data: 2024-04-12T03:26:48.526 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.526 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.526 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.527 DEBUG:boto:Params: {} 2024-04-12T03:26:48.527 DEBUG:boto:Token: None 2024-04-12T03:26:48.527 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.527 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+fHKTlbP0qVOz97R/rM0Aky5pgg= 2024-04-12T03:26:48.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+fHKTlbP0qVOz97R/rM0Aky5pgg=', 'Content-Length': '0'} 2024-04-12T03:26:48.529 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000975a95df9ae317e5-006618a9f8-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.529 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:26:48.529 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:26:48.529 DEBUG:boto:path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.529 DEBUG:boto:auth_path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.529 DEBUG:boto:Method: GET 2024-04-12T03:26:48.529 DEBUG:boto:Path: /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.530 DEBUG:boto:Data: 2024-04-12T03:26:48.530 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.530 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.530 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.530 DEBUG:boto:Params: {} 2024-04-12T03:26:48.530 DEBUG:boto:Token: None 2024-04-12T03:26:48.530 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.530 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+fHKTlbP0qVOz97R/rM0Aky5pgg= 2024-04-12T03:26:48.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+fHKTlbP0qVOz97R/rM0Aky5pgg=', 'Content-Length': '0'} 2024-04-12T03:26:48.532 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000af3d4a2e84affde1-006618a9f8-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.532 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.532 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.532 DEBUG:boto:path=/dkyges-15/?prefix=dummy 2024-04-12T03:26:48.533 DEBUG:boto:auth_path=/dkyges-15/?prefix=dummy 2024-04-12T03:26:48.533 DEBUG:boto:Method: GET 2024-04-12T03:26:48.533 DEBUG:boto:Path: /dkyges-15/?prefix=dummy 2024-04-12T03:26:48.533 DEBUG:boto:Data: 2024-04-12T03:26:48.533 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.533 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.533 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.533 DEBUG:boto:Params: {} 2024-04-12T03:26:48.533 DEBUG:boto:Token: None 2024-04-12T03:26:48.533 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.533 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9y/T0bUrVFxbDzPtqjMMULtNCDc= 2024-04-12T03:26:48.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9y/T0bUrVFxbDzPtqjMMULtNCDc=', 'Content-Length': '0'} 2024-04-12T03:26:48.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f5a53fae9177204-006618a9f8-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.578 DEBUG:boto:b'dkyges-15dummy1000falsedummy2024-04-12T03:26:13.184Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:48.579 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.579 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.579 DEBUG:boto:path=/dkyges-15/?prefix=dummy 2024-04-12T03:26:48.579 DEBUG:boto:auth_path=/dkyges-15/?prefix=dummy 2024-04-12T03:26:48.579 DEBUG:boto:Method: GET 2024-04-12T03:26:48.579 DEBUG:boto:Path: /dkyges-15/?prefix=dummy 2024-04-12T03:26:48.579 DEBUG:boto:Data: 2024-04-12T03:26:48.579 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.580 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.580 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.580 DEBUG:boto:Params: {} 2024-04-12T03:26:48.580 DEBUG:boto:Token: None 2024-04-12T03:26:48.580 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.580 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9y/T0bUrVFxbDzPtqjMMULtNCDc= 2024-04-12T03:26:48.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9y/T0bUrVFxbDzPtqjMMULtNCDc=', 'Content-Length': '0'} 2024-04-12T03:26:48.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f302b5bda1cc4cad-006618a9f8-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.582 DEBUG:boto:b'dkyges-15dummy1000falsedummy2024-04-12T03:26:13.184Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:48.583 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:26:48.583 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:26:48.583 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.583 DEBUG:boto:Path: /dkyges-15/dummy 2024-04-12T03:26:48.583 DEBUG:boto:Data: 2024-04-12T03:26:48.583 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.583 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.583 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.583 DEBUG:boto:Params: {} 2024-04-12T03:26:48.584 DEBUG:boto:Token: None 2024-04-12T03:26:48.584 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/dummy 2024-04-12T03:26:48.584 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D1RP2/UiUpS6dye/42pXfsD9UVk= 2024-04-12T03:26:48.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D1RP2/UiUpS6dye/42pXfsD9UVk=', 'Content-Length': '0'} 2024-04-12T03:26:48.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b74ac7ba3fdada8-006618a9f8-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.586 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:26:48.586 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:26:48.586 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.587 DEBUG:boto:Path: /dkyges-15/dummy 2024-04-12T03:26:48.587 DEBUG:boto:Data: 2024-04-12T03:26:48.587 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.587 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.587 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.587 DEBUG:boto:Params: {} 2024-04-12T03:26:48.587 DEBUG:boto:Token: None 2024-04-12T03:26:48.587 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/dummy 2024-04-12T03:26:48.587 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D1RP2/UiUpS6dye/42pXfsD9UVk= 2024-04-12T03:26:48.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D1RP2/UiUpS6dye/42pXfsD9UVk=', 'Content-Length': '0'} 2024-04-12T03:26:48.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a60bbb8e8060da33-006618a9f8-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:26:48.590 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:26:48.590 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:26:48.590 DEBUG:boto:path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.590 DEBUG:boto:auth_path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.590 DEBUG:boto:Method: GET 2024-04-12T03:26:48.590 DEBUG:boto:Path: /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.590 DEBUG:boto:Data: 2024-04-12T03:26:48.591 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.591 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.591 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.591 DEBUG:boto:Params: {} 2024-04-12T03:26:48.591 DEBUG:boto:Token: None 2024-04-12T03:26:48.591 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.591 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+fHKTlbP0qVOz97R/rM0Aky5pgg= 2024-04-12T03:26:48.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+fHKTlbP0qVOz97R/rM0Aky5pgg=', 'Content-Length': '0'} 2024-04-12T03:26:48.593 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bb094a1c519ecec7-006618a9f8-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.593 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:26:48.593 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:26:48.593 DEBUG:boto:path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.594 DEBUG:boto:auth_path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.594 DEBUG:boto:Method: GET 2024-04-12T03:26:48.594 DEBUG:boto:Path: /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.594 DEBUG:boto:Data: 2024-04-12T03:26:48.594 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.594 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.594 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.594 DEBUG:boto:Params: {} 2024-04-12T03:26:48.594 DEBUG:boto:Token: None 2024-04-12T03:26:48.594 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:26:48.595 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+fHKTlbP0qVOz97R/rM0Aky5pgg= 2024-04-12T03:26:48.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+fHKTlbP0qVOz97R/rM0Aky5pgg=', 'Content-Length': '0'} 2024-04-12T03:26:48.596 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006641e1f7c5aea92c-006618a9f8-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.596 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.597 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.597 DEBUG:boto:path=/dkyges-15/?prefix=dummy 2024-04-12T03:26:48.597 DEBUG:boto:auth_path=/dkyges-15/?prefix=dummy 2024-04-12T03:26:48.597 DEBUG:boto:Method: GET 2024-04-12T03:26:48.597 DEBUG:boto:Path: /dkyges-15/?prefix=dummy 2024-04-12T03:26:48.597 DEBUG:boto:Data: 2024-04-12T03:26:48.597 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.597 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.597 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.597 DEBUG:boto:Params: {} 2024-04-12T03:26:48.597 DEBUG:boto:Token: None 2024-04-12T03:26:48.598 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.598 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9y/T0bUrVFxbDzPtqjMMULtNCDc= 2024-04-12T03:26:48.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9y/T0bUrVFxbDzPtqjMMULtNCDc=', 'Content-Length': '0'} 2024-04-12T03:26:48.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006484d10d055fc41b-006618a9f8-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.600 DEBUG:boto:b'dkyges-15dummy1000falsedummy2024-04-12T03:26:13.184Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:48.600 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.601 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.601 DEBUG:boto:path=/dkyges-15/?prefix=dummy 2024-04-12T03:26:48.601 DEBUG:boto:auth_path=/dkyges-15/?prefix=dummy 2024-04-12T03:26:48.601 DEBUG:boto:Method: GET 2024-04-12T03:26:48.601 DEBUG:boto:Path: /dkyges-15/?prefix=dummy 2024-04-12T03:26:48.601 DEBUG:boto:Data: 2024-04-12T03:26:48.601 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.601 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.601 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.601 DEBUG:boto:Params: {} 2024-04-12T03:26:48.601 DEBUG:boto:Token: None 2024-04-12T03:26:48.602 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.602 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9y/T0bUrVFxbDzPtqjMMULtNCDc= 2024-04-12T03:26:48.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9y/T0bUrVFxbDzPtqjMMULtNCDc=', 'Content-Length': '0'} 2024-04-12T03:26:48.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000119415f55c0b4d25-006618a9f8-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.604 DEBUG:boto:b'dkyges-15dummy1000falsedummy2024-04-12T03:26:13.184Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:26:48.605 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-15 zones={a2, a1} 2024-04-12T03:26:48.605 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.605 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.605 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.605 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:48.605 DEBUG:boto:Data: 2024-04-12T03:26:48.605 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.605 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.605 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.606 DEBUG:boto:Params: {} 2024-04-12T03:26:48.606 DEBUG:boto:Token: None 2024-04-12T03:26:48.606 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.606 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo= 2024-04-12T03:26:48.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo=', 'Content-Length': '0'} 2024-04-12T03:26:48.608 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094a271fc974b8c7d-006618a9f8-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.608 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:48.609 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:48.609 DEBUG:boto:Method: PUT 2024-04-12T03:26:48.609 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:48.609 DEBUG:boto:Data: 2024-04-12T03:26:48.609 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:48.609 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.609 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.609 DEBUG:boto:Params: {} 2024-04-12T03:26:48.609 DEBUG:boto:Token: None 2024-04-12T03:26:48.609 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/obj 2024-04-12T03:26:48.610 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs= 2024-04-12T03:26:48.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs='} 2024-04-12T03:26:48.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'm0JnneswTZ9PHcgOelxxc25lL6S0Dz8'), ('x-amz-request-id', 'tx000007624435eb4f6d7e3-006618a9f8-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.622 DEBUG:rgw_multi.tests:zone=a1 version=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:26:48.623 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.623 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.623 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.623 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:48.623 DEBUG:boto:Data: 2024-04-12T03:26:48.623 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.623 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.623 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.623 DEBUG:boto:Params: {} 2024-04-12T03:26:48.623 DEBUG:boto:Token: None 2024-04-12T03:26:48.623 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.624 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo= 2024-04-12T03:26:48.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo=', 'Content-Length': '0'} 2024-04-12T03:26:48.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000012136c452a22f0-006618a9f8-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.626 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:48.627 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:48.627 DEBUG:boto:Method: PUT 2024-04-12T03:26:48.627 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:48.627 DEBUG:boto:Data: 2024-04-12T03:26:48.627 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:48.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.627 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.627 DEBUG:boto:Params: {} 2024-04-12T03:26:48.627 DEBUG:boto:Token: None 2024-04-12T03:26:48.627 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/obj 2024-04-12T03:26:48.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs= 2024-04-12T03:26:48.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs='} 2024-04-12T03:26:48.640 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'NNLDujprZgDl--7sM1Pv37sfHNfrmJ5'), ('x-amz-request-id', 'tx00000d7be5c46c6426f4f-006618a9f8-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.640 DEBUG:rgw_multi.tests:zone=a2 version=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:26:48.640 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.640 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.640 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.640 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:48.640 DEBUG:boto:Data: 2024-04-12T03:26:48.640 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.641 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.641 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.641 DEBUG:boto:Params: {} 2024-04-12T03:26:48.641 DEBUG:boto:Token: None 2024-04-12T03:26:48.641 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.641 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo= 2024-04-12T03:26:48.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo=', 'Content-Length': '0'} 2024-04-12T03:26:48.644 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac485cdcd472acca-006618a9f8-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.644 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:48.644 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:48.644 DEBUG:boto:Method: PUT 2024-04-12T03:26:48.644 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:48.644 DEBUG:boto:Data: 2024-04-12T03:26:48.644 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:48.644 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.645 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.645 DEBUG:boto:Params: {} 2024-04-12T03:26:48.645 DEBUG:boto:Token: None 2024-04-12T03:26:48.645 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/obj 2024-04-12T03:26:48.645 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs= 2024-04-12T03:26:48.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs='} 2024-04-12T03:26:48.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'nA8qPHogisi3arF3skk4QGJgaM9iq.b'), ('x-amz-request-id', 'tx000008e3d99d7a38a1c3c-006618a9f8-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.691 DEBUG:rgw_multi.tests:zone=a1 version=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:26:48.691 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.691 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.691 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.691 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:48.691 DEBUG:boto:Data: 2024-04-12T03:26:48.691 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.691 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.691 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.691 DEBUG:boto:Params: {} 2024-04-12T03:26:48.692 DEBUG:boto:Token: None 2024-04-12T03:26:48.692 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.692 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo= 2024-04-12T03:26:48.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo=', 'Content-Length': '0'} 2024-04-12T03:26:48.694 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008639f3b37477a19c-006618a9f8-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.695 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:48.695 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:48.695 DEBUG:boto:Method: PUT 2024-04-12T03:26:48.695 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:48.695 DEBUG:boto:Data: 2024-04-12T03:26:48.695 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:48.695 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.695 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.695 DEBUG:boto:Params: {} 2024-04-12T03:26:48.695 DEBUG:boto:Token: None 2024-04-12T03:26:48.695 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/obj 2024-04-12T03:26:48.696 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs= 2024-04-12T03:26:48.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs='} 2024-04-12T03:26:48.743 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'L9iWynXqdS0BqqL22bTZDCGwDwt2itq'), ('x-amz-request-id', 'tx00000cfedd747cfed7895-006618a9f8-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.743 DEBUG:rgw_multi.tests:zone=a2 version=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:26:48.743 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.743 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.743 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.743 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:48.743 DEBUG:boto:Data: 2024-04-12T03:26:48.743 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.744 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.744 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.744 DEBUG:boto:Params: {} 2024-04-12T03:26:48.744 DEBUG:boto:Token: None 2024-04-12T03:26:48.744 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.744 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo= 2024-04-12T03:26:48.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo=', 'Content-Length': '0'} 2024-04-12T03:26:48.747 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007e8f2959578c8a6e-006618a9f8-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.747 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:48.747 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:48.747 DEBUG:boto:Method: PUT 2024-04-12T03:26:48.747 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:48.747 DEBUG:boto:Data: 2024-04-12T03:26:48.748 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:48.748 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.748 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.748 DEBUG:boto:Params: {} 2024-04-12T03:26:48.748 DEBUG:boto:Token: None 2024-04-12T03:26:48.748 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/obj 2024-04-12T03:26:48.748 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs= 2024-04-12T03:26:48.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs='} 2024-04-12T03:26:48.759 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl.'), ('x-amz-request-id', 'tx00000a3310138bfccf869-006618a9f8-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.760 DEBUG:rgw_multi.tests:zone=a1 version=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:26:48.760 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.760 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.760 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.760 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:48.760 DEBUG:boto:Data: 2024-04-12T03:26:48.760 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.760 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.760 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.760 DEBUG:boto:Params: {} 2024-04-12T03:26:48.760 DEBUG:boto:Token: None 2024-04-12T03:26:48.761 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.761 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo= 2024-04-12T03:26:48.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo=', 'Content-Length': '0'} 2024-04-12T03:26:48.763 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bbebf1500ec4f5c-006618a9f8-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.763 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:48.763 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:48.763 DEBUG:boto:Method: PUT 2024-04-12T03:26:48.764 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:48.764 DEBUG:boto:Data: 2024-04-12T03:26:48.764 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:48.764 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.764 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.764 DEBUG:boto:Params: {} 2024-04-12T03:26:48.764 DEBUG:boto:Token: None 2024-04-12T03:26:48.764 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/obj 2024-04-12T03:26:48.764 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs= 2024-04-12T03:26:48.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs='} 2024-04-12T03:26:48.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV'), ('x-amz-request-id', 'tx000005d5d8651d8c63011-006618a9f8-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.775 DEBUG:rgw_multi.tests:zone=a2 version=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:26:48.775 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.775 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.775 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.775 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:48.775 DEBUG:boto:Data: 2024-04-12T03:26:48.775 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.776 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.776 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.776 DEBUG:boto:Params: {} 2024-04-12T03:26:48.776 DEBUG:boto:Token: None 2024-04-12T03:26:48.776 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.776 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo= 2024-04-12T03:26:48.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo=', 'Content-Length': '0'} 2024-04-12T03:26:48.779 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3c4ece21c27db18-006618a9f8-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.779 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:48.779 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:48.779 DEBUG:boto:Method: PUT 2024-04-12T03:26:48.779 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:48.779 DEBUG:boto:Data: 2024-04-12T03:26:48.779 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:48.779 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.779 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.780 DEBUG:boto:Params: {} 2024-04-12T03:26:48.780 DEBUG:boto:Token: None 2024-04-12T03:26:48.780 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/obj 2024-04-12T03:26:48.780 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs= 2024-04-12T03:26:48.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs='} 2024-04-12T03:26:48.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj'), ('x-amz-request-id', 'tx00000da41f1d255d20eb6-006618a9f8-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.827 DEBUG:rgw_multi.tests:zone=a1 version=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:26:48.827 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.827 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.827 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.827 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:48.827 DEBUG:boto:Data: 2024-04-12T03:26:48.827 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.827 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.828 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.828 DEBUG:boto:Params: {} 2024-04-12T03:26:48.828 DEBUG:boto:Token: None 2024-04-12T03:26:48.828 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.828 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo= 2024-04-12T03:26:48.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo=', 'Content-Length': '0'} 2024-04-12T03:26:48.831 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000368db5cdc8ffb2cd-006618a9f8-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.831 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:48.831 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:48.831 DEBUG:boto:Method: PUT 2024-04-12T03:26:48.831 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:48.831 DEBUG:boto:Data: 2024-04-12T03:26:48.831 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:48.831 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.832 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.832 DEBUG:boto:Params: {} 2024-04-12T03:26:48.832 DEBUG:boto:Token: None 2024-04-12T03:26:48.832 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/obj 2024-04-12T03:26:48.832 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs= 2024-04-12T03:26:48.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs='} 2024-04-12T03:26:48.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW'), ('x-amz-request-id', 'tx00000915fb94c017fd4a0-006618a9f8-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.879 DEBUG:rgw_multi.tests:zone=a2 version=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:26:48.879 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.879 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.879 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.879 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:48.879 DEBUG:boto:Data: 2024-04-12T03:26:48.879 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.879 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.880 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.880 DEBUG:boto:Params: {} 2024-04-12T03:26:48.880 DEBUG:boto:Token: None 2024-04-12T03:26:48.880 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.880 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo= 2024-04-12T03:26:48.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo=', 'Content-Length': '0'} 2024-04-12T03:26:48.883 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000074884f7b7015f29-006618a9f8-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.883 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:48.883 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:48.883 DEBUG:boto:Method: PUT 2024-04-12T03:26:48.883 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:48.883 DEBUG:boto:Data: 2024-04-12T03:26:48.883 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:48.884 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.884 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.884 DEBUG:boto:Params: {} 2024-04-12T03:26:48.884 DEBUG:boto:Token: None 2024-04-12T03:26:48.884 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/obj 2024-04-12T03:26:48.884 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs= 2024-04-12T03:26:48.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs='} 2024-04-12T03:26:48.895 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0'), ('x-amz-request-id', 'tx00000867340ce577c7aff-006618a9f8-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.895 DEBUG:rgw_multi.tests:zone=a1 version=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:26:48.895 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.895 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.895 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.895 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:48.895 DEBUG:boto:Data: 2024-04-12T03:26:48.895 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.896 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.896 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.896 DEBUG:boto:Params: {} 2024-04-12T03:26:48.896 DEBUG:boto:Token: None 2024-04-12T03:26:48.896 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.896 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo= 2024-04-12T03:26:48.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo=', 'Content-Length': '0'} 2024-04-12T03:26:48.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f84594755f1c725-006618a9f8-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.899 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:48.899 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:48.899 DEBUG:boto:Method: PUT 2024-04-12T03:26:48.899 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:48.899 DEBUG:boto:Data: 2024-04-12T03:26:48.899 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:48.899 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.900 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.900 DEBUG:boto:Params: {} 2024-04-12T03:26:48.900 DEBUG:boto:Token: None 2024-04-12T03:26:48.900 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/obj 2024-04-12T03:26:48.900 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs= 2024-04-12T03:26:48.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs='} 2024-04-12T03:26:48.911 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl'), ('x-amz-request-id', 'tx000001eeb54eddc7cb829-006618a9f8-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.911 DEBUG:rgw_multi.tests:zone=a2 version=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:26:48.911 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.911 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.911 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.912 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:48.912 DEBUG:boto:Data: 2024-04-12T03:26:48.912 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.912 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.912 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.912 DEBUG:boto:Params: {} 2024-04-12T03:26:48.912 DEBUG:boto:Token: None 2024-04-12T03:26:48.912 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.912 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo= 2024-04-12T03:26:48.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo=', 'Content-Length': '0'} 2024-04-12T03:26:48.915 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012b281033f9e6a57-006618a9f8-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.915 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:48.915 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:48.915 DEBUG:boto:Method: PUT 2024-04-12T03:26:48.915 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:48.915 DEBUG:boto:Data: 2024-04-12T03:26:48.915 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:48.916 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:48.916 DEBUG:boto:Port: 8000 2024-04-12T03:26:48.916 DEBUG:boto:Params: {} 2024-04-12T03:26:48.916 DEBUG:boto:Token: None 2024-04-12T03:26:48.916 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/obj 2024-04-12T03:26:48.916 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs= 2024-04-12T03:26:48.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs='} 2024-04-12T03:26:48.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv'), ('x-amz-request-id', 'tx00000b32fca01c0322f91-006618a9f8-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.959 DEBUG:rgw_multi.tests:zone=a1 version=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:26:48.959 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:48.959 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:48.959 DEBUG:boto:Method: HEAD 2024-04-12T03:26:48.959 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:48.959 DEBUG:boto:Data: 2024-04-12T03:26:48.959 DEBUG:boto:Headers: {} 2024-04-12T03:26:48.959 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.959 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.960 DEBUG:boto:Params: {} 2024-04-12T03:26:48.960 DEBUG:boto:Token: None 2024-04-12T03:26:48.960 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/ 2024-04-12T03:26:48.960 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo= 2024-04-12T03:26:48.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OYfa77icPwRuFgKVUghO3XKE5jo=', 'Content-Length': '0'} 2024-04-12T03:26:48.962 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084c20c212c4eb841-006618a9f8-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:48.963 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:48.963 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:48.963 DEBUG:boto:Method: PUT 2024-04-12T03:26:48.963 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:48.963 DEBUG:boto:Data: 2024-04-12T03:26:48.963 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:48.963 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:48.963 DEBUG:boto:Port: 8001 2024-04-12T03:26:48.963 DEBUG:boto:Params: {} 2024-04-12T03:26:48.963 DEBUG:boto:Token: None 2024-04-12T03:26:48.964 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:48 GMT /dkyges-15/obj 2024-04-12T03:26:48.964 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs= 2024-04-12T03:26:48.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7SCuUpcE9Yrx0kbwqaXfhjV8Rxs='} 2024-04-12T03:26:49.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj'), ('x-amz-request-id', 'tx000000acc28329795c2d2-006618a9f8-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.011 DEBUG:rgw_multi.tests:zone=a2 version=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:26:49.011 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:49.011 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:49.011 DEBUG:boto:Method: HEAD 2024-04-12T03:26:49.011 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:49.011 DEBUG:boto:Data: 2024-04-12T03:26:49.011 DEBUG:boto:Headers: {} 2024-04-12T03:26:49.011 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:49.011 DEBUG:boto:Port: 8000 2024-04-12T03:26:49.011 DEBUG:boto:Params: {} 2024-04-12T03:26:49.011 DEBUG:boto:Token: None 2024-04-12T03:26:49.012 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:49 GMT /dkyges-15/ 2024-04-12T03:26:49.012 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PHlCeuj0+M1EJbzmvgNyVEWjegI= 2024-04-12T03:26:49.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PHlCeuj0+M1EJbzmvgNyVEWjegI=', 'Content-Length': '0'} 2024-04-12T03:26:49.014 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ef533c44c5ed54b-006618a9f9-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.014 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:49.014 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:49.014 DEBUG:boto:Method: PUT 2024-04-12T03:26:49.015 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:49.015 DEBUG:boto:Data: 2024-04-12T03:26:49.015 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:49.015 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:49.015 DEBUG:boto:Port: 8000 2024-04-12T03:26:49.015 DEBUG:boto:Params: {} 2024-04-12T03:26:49.015 DEBUG:boto:Token: None 2024-04-12T03:26:49.015 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:49 GMT /dkyges-15/obj 2024-04-12T03:26:49.015 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BaWzRN7a1cggF5SLq7vBvD/eVMc= 2024-04-12T03:26:49.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BaWzRN7a1cggF5SLq7vBvD/eVMc='} 2024-04-12T03:26:49.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'cXajPIrN3ztkkbq6VakQTR3vdZhlxgl'), ('x-amz-request-id', 'tx0000036ce66ee765687f4-006618a9f9-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.027 DEBUG:rgw_multi.tests:zone=a1 version=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:26:49.027 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:49.027 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:49.027 DEBUG:boto:Method: HEAD 2024-04-12T03:26:49.027 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:49.027 DEBUG:boto:Data: 2024-04-12T03:26:49.027 DEBUG:boto:Headers: {} 2024-04-12T03:26:49.027 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:49.027 DEBUG:boto:Port: 8001 2024-04-12T03:26:49.027 DEBUG:boto:Params: {} 2024-04-12T03:26:49.028 DEBUG:boto:Token: None 2024-04-12T03:26:49.028 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:49 GMT /dkyges-15/ 2024-04-12T03:26:49.028 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PHlCeuj0+M1EJbzmvgNyVEWjegI= 2024-04-12T03:26:49.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PHlCeuj0+M1EJbzmvgNyVEWjegI=', 'Content-Length': '0'} 2024-04-12T03:26:49.030 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d61e81a981555c8-006618a9f9-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.030 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:49.031 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:49.031 DEBUG:boto:Method: PUT 2024-04-12T03:26:49.031 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:49.031 DEBUG:boto:Data: 2024-04-12T03:26:49.031 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:49.031 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:49.031 DEBUG:boto:Port: 8001 2024-04-12T03:26:49.031 DEBUG:boto:Params: {} 2024-04-12T03:26:49.031 DEBUG:boto:Token: None 2024-04-12T03:26:49.031 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:49 GMT /dkyges-15/obj 2024-04-12T03:26:49.032 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BaWzRN7a1cggF5SLq7vBvD/eVMc= 2024-04-12T03:26:49.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BaWzRN7a1cggF5SLq7vBvD/eVMc='} 2024-04-12T03:26:49.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '6gubSYrBrkePUl8NKnDsGwX3jArFN7d'), ('x-amz-request-id', 'tx000005d39c16680aac159-006618a9f9-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.042 DEBUG:rgw_multi.tests:zone=a2 version=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:26:49.042 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:49.043 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:49.043 DEBUG:boto:Method: HEAD 2024-04-12T03:26:49.043 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:49.043 DEBUG:boto:Data: 2024-04-12T03:26:49.043 DEBUG:boto:Headers: {} 2024-04-12T03:26:49.043 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:49.043 DEBUG:boto:Port: 8000 2024-04-12T03:26:49.043 DEBUG:boto:Params: {} 2024-04-12T03:26:49.043 DEBUG:boto:Token: None 2024-04-12T03:26:49.043 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:49 GMT /dkyges-15/ 2024-04-12T03:26:49.043 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PHlCeuj0+M1EJbzmvgNyVEWjegI= 2024-04-12T03:26:49.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PHlCeuj0+M1EJbzmvgNyVEWjegI=', 'Content-Length': '0'} 2024-04-12T03:26:49.046 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008e2175dc34fe449-006618a9f9-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.046 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:49.046 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:49.046 DEBUG:boto:Method: PUT 2024-04-12T03:26:49.046 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:49.046 DEBUG:boto:Data: 2024-04-12T03:26:49.047 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:49.047 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:49.047 DEBUG:boto:Port: 8000 2024-04-12T03:26:49.047 DEBUG:boto:Params: {} 2024-04-12T03:26:49.047 DEBUG:boto:Token: None 2024-04-12T03:26:49.047 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:49 GMT /dkyges-15/obj 2024-04-12T03:26:49.047 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BaWzRN7a1cggF5SLq7vBvD/eVMc= 2024-04-12T03:26:49.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BaWzRN7a1cggF5SLq7vBvD/eVMc='} 2024-04-12T03:26:49.058 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i'), ('x-amz-request-id', 'tx0000071766ab5e4d7b73c-006618a9f9-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.058 DEBUG:rgw_multi.tests:zone=a1 version=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:26:49.058 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:49.058 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:49.059 DEBUG:boto:Method: HEAD 2024-04-12T03:26:49.059 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:49.059 DEBUG:boto:Data: 2024-04-12T03:26:49.059 DEBUG:boto:Headers: {} 2024-04-12T03:26:49.059 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:49.059 DEBUG:boto:Port: 8001 2024-04-12T03:26:49.059 DEBUG:boto:Params: {} 2024-04-12T03:26:49.059 DEBUG:boto:Token: None 2024-04-12T03:26:49.059 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:49 GMT /dkyges-15/ 2024-04-12T03:26:49.059 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PHlCeuj0+M1EJbzmvgNyVEWjegI= 2024-04-12T03:26:49.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PHlCeuj0+M1EJbzmvgNyVEWjegI=', 'Content-Length': '0'} 2024-04-12T03:26:49.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4a4a350f72d757d-006618a9f9-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.062 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:49.062 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:49.062 DEBUG:boto:Method: PUT 2024-04-12T03:26:49.062 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:49.062 DEBUG:boto:Data: 2024-04-12T03:26:49.062 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:49.062 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:49.062 DEBUG:boto:Port: 8001 2024-04-12T03:26:49.063 DEBUG:boto:Params: {} 2024-04-12T03:26:49.063 DEBUG:boto:Token: None 2024-04-12T03:26:49.063 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:49 GMT /dkyges-15/obj 2024-04-12T03:26:49.063 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BaWzRN7a1cggF5SLq7vBvD/eVMc= 2024-04-12T03:26:49.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BaWzRN7a1cggF5SLq7vBvD/eVMc='} 2024-04-12T03:26:49.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'crJW1cr7W5PZhyrypuyIfUhJav0.RMm'), ('x-amz-request-id', 'tx00000a59667f654afa625-006618a9f9-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.074 DEBUG:rgw_multi.tests:zone=a2 version=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:26:49.074 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:49.074 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:49.075 DEBUG:boto:Method: HEAD 2024-04-12T03:26:49.075 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:49.075 DEBUG:boto:Data: 2024-04-12T03:26:49.075 DEBUG:boto:Headers: {} 2024-04-12T03:26:49.075 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:49.075 DEBUG:boto:Port: 8000 2024-04-12T03:26:49.075 DEBUG:boto:Params: {} 2024-04-12T03:26:49.075 DEBUG:boto:Token: None 2024-04-12T03:26:49.075 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:49 GMT /dkyges-15/ 2024-04-12T03:26:49.075 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PHlCeuj0+M1EJbzmvgNyVEWjegI= 2024-04-12T03:26:49.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PHlCeuj0+M1EJbzmvgNyVEWjegI=', 'Content-Length': '0'} 2024-04-12T03:26:49.078 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b87bea3ebcb2bda2-006618a9f9-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.078 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:49.078 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:49.078 DEBUG:boto:Method: PUT 2024-04-12T03:26:49.078 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:49.078 DEBUG:boto:Data: 2024-04-12T03:26:49.078 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:49.079 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:49.079 DEBUG:boto:Port: 8000 2024-04-12T03:26:49.079 DEBUG:boto:Params: {} 2024-04-12T03:26:49.079 DEBUG:boto:Token: None 2024-04-12T03:26:49.079 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:49 GMT /dkyges-15/obj 2024-04-12T03:26:49.079 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BaWzRN7a1cggF5SLq7vBvD/eVMc= 2024-04-12T03:26:49.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BaWzRN7a1cggF5SLq7vBvD/eVMc='} 2024-04-12T03:26:49.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG'), ('x-amz-request-id', 'tx00000114157f46ce5dd86-006618a9f9-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.090 DEBUG:rgw_multi.tests:zone=a1 version=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:26:49.090 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:49.090 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:49.090 DEBUG:boto:Method: HEAD 2024-04-12T03:26:49.090 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:49.090 DEBUG:boto:Data: 2024-04-12T03:26:49.090 DEBUG:boto:Headers: {} 2024-04-12T03:26:49.090 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:49.091 DEBUG:boto:Port: 8001 2024-04-12T03:26:49.091 DEBUG:boto:Params: {} 2024-04-12T03:26:49.091 DEBUG:boto:Token: None 2024-04-12T03:26:49.091 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:49 GMT /dkyges-15/ 2024-04-12T03:26:49.091 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PHlCeuj0+M1EJbzmvgNyVEWjegI= 2024-04-12T03:26:49.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PHlCeuj0+M1EJbzmvgNyVEWjegI=', 'Content-Length': '0'} 2024-04-12T03:26:49.093 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081dd1014a923742c-006618a9f9-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.094 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:49.094 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:49.094 DEBUG:boto:Method: PUT 2024-04-12T03:26:49.094 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:49.094 DEBUG:boto:Data: 2024-04-12T03:26:49.094 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:49.094 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:49.094 DEBUG:boto:Port: 8001 2024-04-12T03:26:49.094 DEBUG:boto:Params: {} 2024-04-12T03:26:49.094 DEBUG:boto:Token: None 2024-04-12T03:26:49.095 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:49 GMT /dkyges-15/obj 2024-04-12T03:26:49.095 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BaWzRN7a1cggF5SLq7vBvD/eVMc= 2024-04-12T03:26:49.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BaWzRN7a1cggF5SLq7vBvD/eVMc='} 2024-04-12T03:26:49.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Xu58tKQgwRtuVVd31MDUlibB214glR3'), ('x-amz-request-id', 'tx000007513dbb6812da696-006618a9f9-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.143 DEBUG:rgw_multi.tests:zone=a2 version=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:26:49.143 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:49.143 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:49.143 DEBUG:boto:Method: HEAD 2024-04-12T03:26:49.143 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:49.143 DEBUG:boto:Data: 2024-04-12T03:26:49.143 DEBUG:boto:Headers: {} 2024-04-12T03:26:49.143 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:49.143 DEBUG:boto:Port: 8000 2024-04-12T03:26:49.144 DEBUG:boto:Params: {} 2024-04-12T03:26:49.144 DEBUG:boto:Token: None 2024-04-12T03:26:49.144 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:49 GMT /dkyges-15/ 2024-04-12T03:26:49.144 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PHlCeuj0+M1EJbzmvgNyVEWjegI= 2024-04-12T03:26:49.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PHlCeuj0+M1EJbzmvgNyVEWjegI=', 'Content-Length': '0'} 2024-04-12T03:26:49.146 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000033567e7419b82b6-006618a9f9-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.147 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:49.147 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:49.147 DEBUG:boto:Method: PUT 2024-04-12T03:26:49.147 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:49.147 DEBUG:boto:Data: 2024-04-12T03:26:49.147 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:49.147 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:26:49.147 DEBUG:boto:Port: 8000 2024-04-12T03:26:49.147 DEBUG:boto:Params: {} 2024-04-12T03:26:49.147 DEBUG:boto:Token: None 2024-04-12T03:26:49.148 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:49 GMT /dkyges-15/obj 2024-04-12T03:26:49.148 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BaWzRN7a1cggF5SLq7vBvD/eVMc= 2024-04-12T03:26:49.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BaWzRN7a1cggF5SLq7vBvD/eVMc='} 2024-04-12T03:26:49.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM'), ('x-amz-request-id', 'tx000008338c12c7943ee81-006618a9f9-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.195 DEBUG:rgw_multi.tests:zone=a1 version=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:26:49.195 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:26:49.195 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:26:49.195 DEBUG:boto:Method: HEAD 2024-04-12T03:26:49.195 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:26:49.195 DEBUG:boto:Data: 2024-04-12T03:26:49.195 DEBUG:boto:Headers: {} 2024-04-12T03:26:49.195 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:49.195 DEBUG:boto:Port: 8001 2024-04-12T03:26:49.195 DEBUG:boto:Params: {} 2024-04-12T03:26:49.195 DEBUG:boto:Token: None 2024-04-12T03:26:49.196 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:26:49 GMT /dkyges-15/ 2024-04-12T03:26:49.196 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PHlCeuj0+M1EJbzmvgNyVEWjegI= 2024-04-12T03:26:49.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:26:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PHlCeuj0+M1EJbzmvgNyVEWjegI=', 'Content-Length': '0'} 2024-04-12T03:26:49.198 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a30a55376a9e126-006618a9f9-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.198 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:26:49.199 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:26:49.199 DEBUG:boto:Method: PUT 2024-04-12T03:26:49.199 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:26:49.199 DEBUG:boto:Data: 2024-04-12T03:26:49.199 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2024-04-12T03:26:49.199 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:26:49.199 DEBUG:boto:Port: 8001 2024-04-12T03:26:49.199 DEBUG:boto:Params: {} 2024-04-12T03:26:49.199 DEBUG:boto:Token: None 2024-04-12T03:26:49.199 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 12 Apr 2024 03:26:49 GMT /dkyges-15/obj 2024-04-12T03:26:49.200 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BaWzRN7a1cggF5SLq7vBvD/eVMc= 2024-04-12T03:26:49.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:26:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BaWzRN7a1cggF5SLq7vBvD/eVMc='} 2024-04-12T03:26:49.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-amz-request-id', 'tx00000c366051f301fb0e0-006618a9f9-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:26:49.210 DEBUG:rgw_multi.tests:zone=a2 version=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:26:49.210 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-15 2024-04-12T03:26:49.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:26:49.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:26:49.211 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:26:49.277 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:26:49.278 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:26:49.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:49.318+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:26:49.319 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.37.11, , , , , ] 2024-04-12T03:26:49.319 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:26:54.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:54.319+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:26:54.321 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.37.11, , , , , ] 2024-04-12T03:26:54.321 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:26:59.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:26:59.320+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:26:59.322 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.37.11, , , , , ] 2024-04-12T03:26:59.322 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:27:04.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:27:04.322+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:27:04.323 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.37.11, , , 00000000003.32.6, , ] 2024-04-12T03:27:04.323 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:27:09.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:27:09.323+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:27:09.325 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.39.10, , , 00000000003.32.6, , ] 2024-04-12T03:27:09.325 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:27:14.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:27:14.324+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:27:14.326 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.39.10, , , 00000000003.32.6, , ] 2024-04-12T03:27:14.326 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:27:19.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:27:19.326+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:27:19.327 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.39.10, , , 00000000003.32.6, , ] 2024-04-12T03:27:19.327 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:27:24.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:27:24.327+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:27:24.329 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.39.10, , , 00000000007.40.6, , ] 2024-04-12T03:27:24.329 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:27:29.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:27:29.328+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:27:29.330 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.39.10, , , 00000000017.60.6, , ] 2024-04-12T03:27:29.331 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:27:34.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:27:34.329+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:27:34.330 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.39.10, , , 00000000017.60.6, , ] 2024-04-12T03:27:34.331 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:27:39.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:27:39.330+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:27:39.332 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.39.10, , , 00000000017.60.6, , ] 2024-04-12T03:27:39.332 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:27:44.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:27:44.332+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:27:44.333 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.39.10, , , 00000000017.60.6, , ] 2024-04-12T03:27:44.333 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:27:49.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:27:49.333+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:27:49.334 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.39.10, , , 00000000017.60.6, , ] 2024-04-12T03:27:49.334 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:27:54.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:27:54.334+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:27:54.336 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.39.10, , , 00000000017.60.6, , ] 2024-04-12T03:27:54.336 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:27:59.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:27:59.336+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:27:59.337 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.39.10, , , 00000000017.60.6, , ] 2024-04-12T03:27:59.337 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:28:04.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:04.337+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:28:04.338 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.39.10, , , 00000000017.60.6, , ] 2024-04-12T03:28:04.339 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:28:09.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:09.339+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:28:09.340 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.39.10, , , 00000000017.60.6, , ] 2024-04-12T03:28:09.340 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:28:14.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:14.340+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:28:14.341 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.39.10, , , 00000000017.60.6, , ] 2024-04-12T03:28:14.341 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:28:19.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:19.341+0000 7f97d978a780 1 waiting for incremental sync to catch up: 2024-04-12T03:28:19.343 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.39.10, , , 00000000017.60.6, , ] 2024-04-12T03:28:19.343 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:28:24.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:24.343+0000 7f97d978a780 1 bucket sync caught up with source: 2024-04-12T03:28:24.344 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000003.39.10, , , 00000000031.96.12, , ] 2024-04-12T03:28:24.344 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000003.39.10, , , 00000000020.65.12, , ] 2024-04-12T03:28:24.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:24.343+0000 7f97d978a780 0 bucket checkpoint complete 2024-04-12T03:28:24.359 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-15 2024-04-12T03:28:24.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:28:24.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:28:24.360 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:28:24.432 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:24.433 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:24.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:24.479+0000 7fc82a7d3780 1 waiting for incremental sync to catch up: 2024-04-12T03:28:24.481 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.49.11, , , 00000000028.84.6, , ] 2024-04-12T03:28:24.481 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.49.11, , , 00000000031.91.9, , ] 2024-04-12T03:28:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:29.481+0000 7fc82a7d3780 1 waiting for incremental sync to catch up: 2024-04-12T03:28:29.482 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.49.11, , , 00000000028.84.6, , ] 2024-04-12T03:28:29.482 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.49.11, , , 00000000031.91.9, , ] 2024-04-12T03:28:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:34.482+0000 7fc82a7d3780 1 waiting for incremental sync to catch up: 2024-04-12T03:28:34.484 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.49.11, , , 00000000028.84.6, , ] 2024-04-12T03:28:34.484 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.49.11, , , 00000000031.91.9, , ] 2024-04-12T03:28:39.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:39.484+0000 7fc82a7d3780 1 waiting for incremental sync to catch up: 2024-04-12T03:28:39.495 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.49.11, , , 00000000028.84.6, , ] 2024-04-12T03:28:39.495 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.49.11, , , 00000000031.91.9, , ] 2024-04-12T03:28:44.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:44.486+0000 7fc82a7d3780 1 bucket sync caught up with source: 2024-04-12T03:28:44.487 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000002.49.11, , , 00000000031.91.9, , ] 2024-04-12T03:28:44.487 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000002.49.11, , , 00000000031.91.9, , ] 2024-04-12T03:28:44.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:44.486+0000 7fc82a7d3780 0 bucket checkpoint complete 2024-04-12T03:28:44.503 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-15 zones={a2, a1} 2024-04-12T03:28:44.504 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:44.504 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:44.504 DEBUG:boto:Method: HEAD 2024-04-12T03:28:44.504 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:28:44.504 DEBUG:boto:Data: 2024-04-12T03:28:44.504 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.504 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.504 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.504 DEBUG:boto:Params: {} 2024-04-12T03:28:44.505 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:28:44.505 DEBUG:boto:Token: None 2024-04-12T03:28:44.505 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/ 2024-04-12T03:28:44.505 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FReDk++iBJjNEfnr3V5DiiJSqq0= 2024-04-12T03:28:44.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FReDk++iBJjNEfnr3V5DiiJSqq0=', 'Content-Length': '0'} 2024-04-12T03:28:44.536 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008843c62311fd866e-006618aa6c-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.536 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:44.536 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:44.536 DEBUG:boto:Method: HEAD 2024-04-12T03:28:44.536 DEBUG:boto:Path: /dkyges-15/ 2024-04-12T03:28:44.536 DEBUG:boto:Data: 2024-04-12T03:28:44.536 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.536 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.536 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.537 DEBUG:boto:Params: {} 2024-04-12T03:28:44.537 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:28:44.537 DEBUG:boto:Token: None 2024-04-12T03:28:44.537 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/ 2024-04-12T03:28:44.537 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FReDk++iBJjNEfnr3V5DiiJSqq0= 2024-04-12T03:28:44.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FReDk++iBJjNEfnr3V5DiiJSqq0=', 'Content-Length': '0'} 2024-04-12T03:28:44.567 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8decff814d76437-006618aa6c-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.568 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:28:44.568 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:44.568 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:44.568 DEBUG:boto:path=/dkyges-15/?versions&max-keys=999 2024-04-12T03:28:44.568 DEBUG:boto:auth_path=/dkyges-15/?versions&max-keys=999 2024-04-12T03:28:44.568 DEBUG:boto:Method: GET 2024-04-12T03:28:44.568 DEBUG:boto:Path: /dkyges-15/?versions&max-keys=999 2024-04-12T03:28:44.568 DEBUG:boto:Data: 2024-04-12T03:28:44.568 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.568 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.568 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.569 DEBUG:boto:Params: {} 2024-04-12T03:28:44.569 DEBUG:boto:Token: None 2024-04-12T03:28:44.569 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/?versions 2024-04-12T03:28:44.569 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cW9C/qfwvncHeZgaULvG0tWkSLc= 2024-04-12T03:28:44.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cW9C/qfwvncHeZgaULvG0tWkSLc=', 'Content-Length': '0'} 2024-04-12T03:28:44.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa8216bbdb3f826e-006618aa6c-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.572 DEBUG:boto:b'dkyges-15999falsedummy6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-true2024-04-12T03:26:13.184Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDCRUzLijiY355CP0VujBj35OATE10fgtrue2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjIM0gY0UkHgbX4pgoOgWmEMjY7n4stIMfalse2024-04-12T03:26:49.149Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXu58tKQgwRtuVVd31MDUlibB214glR3false2024-04-12T03:26:49.096Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTGfalse2024-04-12T03:26:49.081Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcrJW1cr7W5PZhyrypuyIfUhJav0.RMmfalse2024-04-12T03:26:49.065Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpYCRYGlATwh0QVzhGZxDIKJ9LcInd2ifalse2024-04-12T03:26:49.049Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6gubSYrBrkePUl8NKnDsGwX3jArFN7dfalse2024-04-12T03:26:49.033Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcXajPIrN3ztkkbq6VakQTR3vdZhlxglfalse2024-04-12T03:26:49.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Svfalse2024-04-12T03:26:48.918Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjW7uVB2AgTJyCAtTi2lB4dP4PR6fEVsjfalse2024-04-12T03:26:48.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vlfalse2024-04-12T03:26:48.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgXiwRoSqiWpOKPjQCipYwB9V5k1dqF0false2024-04-12T03:26:48.886Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCvjy9irUJDLukcVm26x3nBQ9pyVqTRWfalse2024-04-12T03:26:48.834Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ3mUi5oz1AZqDhZS-0eePbcQI7nb4Fjfalse2024-04-12T03:26:48.782Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZVfalse2024-04-12T03:26:48.766Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSQ4-mXfRGBqGjQRWOEV5qPda3uaZwl.false2024-04-12T03:26:48.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjL9iWynXqdS0BqqL22bTZDCGwDwt2itqfalse2024-04-12T03:26:48.697Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnA8qPHogisi3arF3skk4QGJgaM9iq.bfalse2024-04-12T03:26:48.647Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNNLDujprZgDl--7sM1Pv37sfHNfrmJ5false2024-04-12T03:26:48.629Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm0JnneswTZ9PHcgOelxxc25lL6S0Dz8false2024-04-12T03:26:48.611Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:44.574 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:28:44.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.575 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:28:44.575 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:44.578 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:44.578 DEBUG:boto:path=/dkyges-15/?versions&max-keys=999 2024-04-12T03:28:44.578 DEBUG:boto:auth_path=/dkyges-15/?versions&max-keys=999 2024-04-12T03:28:44.579 DEBUG:boto:Method: GET 2024-04-12T03:28:44.579 DEBUG:boto:Path: /dkyges-15/?versions&max-keys=999 2024-04-12T03:28:44.579 DEBUG:boto:Data: 2024-04-12T03:28:44.579 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.579 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.579 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.579 DEBUG:boto:Params: {} 2024-04-12T03:28:44.580 DEBUG:boto:Token: None 2024-04-12T03:28:44.580 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/?versions 2024-04-12T03:28:44.580 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cW9C/qfwvncHeZgaULvG0tWkSLc= 2024-04-12T03:28:44.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cW9C/qfwvncHeZgaULvG0tWkSLc=', 'Content-Length': '0'} 2024-04-12T03:28:44.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074c6c7cf79b6b081-006618aa6c-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.584 DEBUG:boto:b'dkyges-15999falsedummy6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-true2024-04-12T03:26:13.184Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDCRUzLijiY355CP0VujBj35OATE10fgtrue2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjIM0gY0UkHgbX4pgoOgWmEMjY7n4stIMfalse2024-04-12T03:26:49.149Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXu58tKQgwRtuVVd31MDUlibB214glR3false2024-04-12T03:26:49.096Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTGfalse2024-04-12T03:26:49.081Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcrJW1cr7W5PZhyrypuyIfUhJav0.RMmfalse2024-04-12T03:26:49.065Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpYCRYGlATwh0QVzhGZxDIKJ9LcInd2ifalse2024-04-12T03:26:49.049Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6gubSYrBrkePUl8NKnDsGwX3jArFN7dfalse2024-04-12T03:26:49.033Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcXajPIrN3ztkkbq6VakQTR3vdZhlxglfalse2024-04-12T03:26:49.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Svfalse2024-04-12T03:26:48.918Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjW7uVB2AgTJyCAtTi2lB4dP4PR6fEVsjfalse2024-04-12T03:26:48.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vlfalse2024-04-12T03:26:48.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgXiwRoSqiWpOKPjQCipYwB9V5k1dqF0false2024-04-12T03:26:48.886Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCvjy9irUJDLukcVm26x3nBQ9pyVqTRWfalse2024-04-12T03:26:48.834Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ3mUi5oz1AZqDhZS-0eePbcQI7nb4Fjfalse2024-04-12T03:26:48.782Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZVfalse2024-04-12T03:26:48.766Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSQ4-mXfRGBqGjQRWOEV5qPda3uaZwl.false2024-04-12T03:26:48.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjL9iWynXqdS0BqqL22bTZDCGwDwt2itqfalse2024-04-12T03:26:48.697Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnA8qPHogisi3arF3skk4QGJgaM9iq.bfalse2024-04-12T03:26:48.647Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNNLDujprZgDl--7sM1Pv37sfHNfrmJ5false2024-04-12T03:26:48.629Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm0JnneswTZ9PHcgOelxxc25lL6S0Dz8false2024-04-12T03:26:48.611Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:44.585 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T03:28:44.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:28:44.588 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:44.588 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:44.588 DEBUG:boto:path=/dkyges-15/?versions&max-keys=999 2024-04-12T03:28:44.588 DEBUG:boto:auth_path=/dkyges-15/?versions&max-keys=999 2024-04-12T03:28:44.588 DEBUG:boto:Method: GET 2024-04-12T03:28:44.588 DEBUG:boto:Path: /dkyges-15/?versions&max-keys=999 2024-04-12T03:28:44.588 DEBUG:boto:Data: 2024-04-12T03:28:44.588 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.588 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.589 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.589 DEBUG:boto:Params: {} 2024-04-12T03:28:44.589 DEBUG:boto:Token: None 2024-04-12T03:28:44.589 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/?versions 2024-04-12T03:28:44.589 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cW9C/qfwvncHeZgaULvG0tWkSLc= 2024-04-12T03:28:44.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cW9C/qfwvncHeZgaULvG0tWkSLc=', 'Content-Length': '0'} 2024-04-12T03:28:44.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bcb890f4ffa0225-006618aa6c-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.592 DEBUG:boto:b'dkyges-15999falsedummy6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-true2024-04-12T03:26:13.184Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDCRUzLijiY355CP0VujBj35OATE10fgtrue2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjIM0gY0UkHgbX4pgoOgWmEMjY7n4stIMfalse2024-04-12T03:26:49.149Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXu58tKQgwRtuVVd31MDUlibB214glR3false2024-04-12T03:26:49.096Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTGfalse2024-04-12T03:26:49.081Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcrJW1cr7W5PZhyrypuyIfUhJav0.RMmfalse2024-04-12T03:26:49.065Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpYCRYGlATwh0QVzhGZxDIKJ9LcInd2ifalse2024-04-12T03:26:49.049Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6gubSYrBrkePUl8NKnDsGwX3jArFN7dfalse2024-04-12T03:26:49.033Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcXajPIrN3ztkkbq6VakQTR3vdZhlxglfalse2024-04-12T03:26:49.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Svfalse2024-04-12T03:26:48.918Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjW7uVB2AgTJyCAtTi2lB4dP4PR6fEVsjfalse2024-04-12T03:26:48.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vlfalse2024-04-12T03:26:48.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgXiwRoSqiWpOKPjQCipYwB9V5k1dqF0false2024-04-12T03:26:48.886Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCvjy9irUJDLukcVm26x3nBQ9pyVqTRWfalse2024-04-12T03:26:48.834Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ3mUi5oz1AZqDhZS-0eePbcQI7nb4Fjfalse2024-04-12T03:26:48.782Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZVfalse2024-04-12T03:26:48.766Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSQ4-mXfRGBqGjQRWOEV5qPda3uaZwl.false2024-04-12T03:26:48.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjL9iWynXqdS0BqqL22bTZDCGwDwt2itqfalse2024-04-12T03:26:48.697Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnA8qPHogisi3arF3skk4QGJgaM9iq.bfalse2024-04-12T03:26:48.647Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNNLDujprZgDl--7sM1Pv37sfHNfrmJ5false2024-04-12T03:26:48.629Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm0JnneswTZ9PHcgOelxxc25lL6S0Dz8false2024-04-12T03:26:48.611Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:44.594 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:44.594 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:44.594 DEBUG:boto:path=/dkyges-15/?versions&max-keys=999 2024-04-12T03:28:44.594 DEBUG:boto:auth_path=/dkyges-15/?versions&max-keys=999 2024-04-12T03:28:44.594 DEBUG:boto:Method: GET 2024-04-12T03:28:44.594 DEBUG:boto:Path: /dkyges-15/?versions&max-keys=999 2024-04-12T03:28:44.595 DEBUG:boto:Data: 2024-04-12T03:28:44.595 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.595 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.595 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.595 DEBUG:boto:Params: {} 2024-04-12T03:28:44.595 DEBUG:boto:Token: None 2024-04-12T03:28:44.596 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/?versions 2024-04-12T03:28:44.596 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cW9C/qfwvncHeZgaULvG0tWkSLc= 2024-04-12T03:28:44.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cW9C/qfwvncHeZgaULvG0tWkSLc=', 'Content-Length': '0'} 2024-04-12T03:28:44.599 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a837b9b0a423b841-006618aa6c-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.599 DEBUG:boto:b'dkyges-15999falsedummy6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-true2024-04-12T03:26:13.184Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDCRUzLijiY355CP0VujBj35OATE10fgtrue2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjIM0gY0UkHgbX4pgoOgWmEMjY7n4stIMfalse2024-04-12T03:26:49.149Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjXu58tKQgwRtuVVd31MDUlibB214glR3false2024-04-12T03:26:49.096Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTGfalse2024-04-12T03:26:49.081Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcrJW1cr7W5PZhyrypuyIfUhJav0.RMmfalse2024-04-12T03:26:49.065Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjpYCRYGlATwh0QVzhGZxDIKJ9LcInd2ifalse2024-04-12T03:26:49.049Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6gubSYrBrkePUl8NKnDsGwX3jArFN7dfalse2024-04-12T03:26:49.033Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjcXajPIrN3ztkkbq6VakQTR3vdZhlxglfalse2024-04-12T03:26:49.017Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Svfalse2024-04-12T03:26:48.918Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjW7uVB2AgTJyCAtTi2lB4dP4PR6fEVsjfalse2024-04-12T03:26:48.965Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vlfalse2024-04-12T03:26:48.902Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjgXiwRoSqiWpOKPjQCipYwB9V5k1dqF0false2024-04-12T03:26:48.886Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjCvjy9irUJDLukcVm26x3nBQ9pyVqTRWfalse2024-04-12T03:26:48.834Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZ3mUi5oz1AZqDhZS-0eePbcQI7nb4Fjfalse2024-04-12T03:26:48.782Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZVfalse2024-04-12T03:26:48.766Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjSQ4-mXfRGBqGjQRWOEV5qPda3uaZwl.false2024-04-12T03:26:48.750Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjL9iWynXqdS0BqqL22bTZDCGwDwt2itqfalse2024-04-12T03:26:48.697Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnA8qPHogisi3arF3skk4QGJgaM9iq.bfalse2024-04-12T03:26:48.647Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjNNLDujprZgDl--7sM1Pv37sfHNfrmJ5false2024-04-12T03:26:48.629Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjm0JnneswTZ9PHcgOelxxc25lL6S0Dz8false2024-04-12T03:26:48.611Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:44.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:28:44.601 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:28:44.601 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:28:44.601 DEBUG:boto:path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.601 DEBUG:boto:auth_path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.601 DEBUG:boto:Method: GET 2024-04-12T03:28:44.601 DEBUG:boto:Path: /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.602 DEBUG:boto:Data: 2024-04-12T03:28:44.602 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.602 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.602 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.602 DEBUG:boto:Params: {} 2024-04-12T03:28:44.602 DEBUG:boto:Token: None 2024-04-12T03:28:44.602 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.603 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ds9Mh1to4VtOu0gxm3xhleDMH+E= 2024-04-12T03:28:44.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ds9Mh1to4VtOu0gxm3xhleDMH+E=', 'Content-Length': '0'} 2024-04-12T03:28:44.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001180ad7ce7f970bc-006618aa6c-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.605 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:28:44.605 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:28:44.605 DEBUG:boto:path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.605 DEBUG:boto:auth_path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.605 DEBUG:boto:Method: GET 2024-04-12T03:28:44.605 DEBUG:boto:Path: /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.605 DEBUG:boto:Data: 2024-04-12T03:28:44.606 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.606 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.606 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.606 DEBUG:boto:Params: {} 2024-04-12T03:28:44.606 DEBUG:boto:Token: None 2024-04-12T03:28:44.606 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.607 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ds9Mh1to4VtOu0gxm3xhleDMH+E= 2024-04-12T03:28:44.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ds9Mh1to4VtOu0gxm3xhleDMH+E=', 'Content-Length': '0'} 2024-04-12T03:28:44.608 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000003a114f37a48ee42-006618aa6c-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.609 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:28:44.609 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:28:44.609 DEBUG:boto:path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.609 DEBUG:boto:auth_path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.609 DEBUG:boto:Method: HEAD 2024-04-12T03:28:44.609 DEBUG:boto:Path: /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.609 DEBUG:boto:Data: 2024-04-12T03:28:44.609 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.610 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.610 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.610 DEBUG:boto:Params: {} 2024-04-12T03:28:44.610 DEBUG:boto:Token: None 2024-04-12T03:28:44.610 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.611 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:38Ix5VDEqe788gRd/KuWan1nOJQ= 2024-04-12T03:28:44.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:38Ix5VDEqe788gRd/KuWan1nOJQ=', 'Content-Length': '0'} 2024-04-12T03:28:44.612 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dade530d08c719bf-006618aa6c-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.613 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:28:44.613 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:28:44.613 DEBUG:boto:path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.613 DEBUG:boto:auth_path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.613 DEBUG:boto:Method: HEAD 2024-04-12T03:28:44.613 DEBUG:boto:Path: /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.613 DEBUG:boto:Data: 2024-04-12T03:28:44.613 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.613 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.613 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.614 DEBUG:boto:Params: {} 2024-04-12T03:28:44.614 DEBUG:boto:Token: None 2024-04-12T03:28:44.614 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.614 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:38Ix5VDEqe788gRd/KuWan1nOJQ= 2024-04-12T03:28:44.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:38Ix5VDEqe788gRd/KuWan1nOJQ=', 'Content-Length': '0'} 2024-04-12T03:28:44.616 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000448e8af1424ec994-006618aa6c-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:28:44.616 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:28:44.616 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:28:44.617 DEBUG:boto:path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.617 DEBUG:boto:auth_path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.617 DEBUG:boto:Method: GET 2024-04-12T03:28:44.617 DEBUG:boto:Path: /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.617 DEBUG:boto:Data: 2024-04-12T03:28:44.617 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.617 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.617 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.617 DEBUG:boto:Params: {} 2024-04-12T03:28:44.618 DEBUG:boto:Token: None 2024-04-12T03:28:44.618 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.618 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ds9Mh1to4VtOu0gxm3xhleDMH+E= 2024-04-12T03:28:44.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ds9Mh1to4VtOu0gxm3xhleDMH+E=', 'Content-Length': '0'} 2024-04-12T03:28:44.620 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004e6e3d91156055f5-006618aa6c-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.620 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:28:44.620 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:28:44.620 DEBUG:boto:path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.620 DEBUG:boto:auth_path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.620 DEBUG:boto:Method: GET 2024-04-12T03:28:44.621 DEBUG:boto:Path: /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.621 DEBUG:boto:Data: 2024-04-12T03:28:44.621 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.621 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.621 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.621 DEBUG:boto:Params: {} 2024-04-12T03:28:44.621 DEBUG:boto:Token: None 2024-04-12T03:28:44.621 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.622 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ds9Mh1to4VtOu0gxm3xhleDMH+E= 2024-04-12T03:28:44.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ds9Mh1to4VtOu0gxm3xhleDMH+E=', 'Content-Length': '0'} 2024-04-12T03:28:44.623 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006a2dee3ab2308190-006618aa6c-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.624 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:44.624 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:44.624 DEBUG:boto:path=/dkyges-15/?prefix=dummy 2024-04-12T03:28:44.624 DEBUG:boto:auth_path=/dkyges-15/?prefix=dummy 2024-04-12T03:28:44.624 DEBUG:boto:Method: GET 2024-04-12T03:28:44.624 DEBUG:boto:Path: /dkyges-15/?prefix=dummy 2024-04-12T03:28:44.624 DEBUG:boto:Data: 2024-04-12T03:28:44.624 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.624 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.625 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.625 DEBUG:boto:Params: {} 2024-04-12T03:28:44.625 DEBUG:boto:Token: None 2024-04-12T03:28:44.625 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/ 2024-04-12T03:28:44.625 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4= 2024-04-12T03:28:44.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4=', 'Content-Length': '0'} 2024-04-12T03:28:44.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2a88b0883e0e21d-006618aa6c-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.670 DEBUG:boto:b'dkyges-15dummy1000falsedummy2024-04-12T03:26:13.184Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:44.671 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:44.671 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:44.671 DEBUG:boto:path=/dkyges-15/?prefix=dummy 2024-04-12T03:28:44.671 DEBUG:boto:auth_path=/dkyges-15/?prefix=dummy 2024-04-12T03:28:44.671 DEBUG:boto:Method: GET 2024-04-12T03:28:44.671 DEBUG:boto:Path: /dkyges-15/?prefix=dummy 2024-04-12T03:28:44.671 DEBUG:boto:Data: 2024-04-12T03:28:44.671 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.671 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.672 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.672 DEBUG:boto:Params: {} 2024-04-12T03:28:44.672 DEBUG:boto:Token: None 2024-04-12T03:28:44.672 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/ 2024-04-12T03:28:44.672 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4= 2024-04-12T03:28:44.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4=', 'Content-Length': '0'} 2024-04-12T03:28:44.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000582ce0adf3f35d56-006618aa6c-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.675 DEBUG:boto:b'dkyges-15dummy1000falsedummy2024-04-12T03:26:13.184Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:44.675 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:28:44.675 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:28:44.675 DEBUG:boto:Method: HEAD 2024-04-12T03:28:44.675 DEBUG:boto:Path: /dkyges-15/dummy 2024-04-12T03:28:44.675 DEBUG:boto:Data: 2024-04-12T03:28:44.675 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.676 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.676 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.676 DEBUG:boto:Params: {} 2024-04-12T03:28:44.676 DEBUG:boto:Token: None 2024-04-12T03:28:44.676 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/dummy 2024-04-12T03:28:44.676 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dlC5Wz/Gyfv4cMha2avVsv8V1zs= 2024-04-12T03:28:44.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dlC5Wz/Gyfv4cMha2avVsv8V1zs=', 'Content-Length': '0'} 2024-04-12T03:28:44.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000066827d2a67c8291-006618aa6c-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.679 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:28:44.679 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:28:44.679 DEBUG:boto:Method: HEAD 2024-04-12T03:28:44.679 DEBUG:boto:Path: /dkyges-15/dummy 2024-04-12T03:28:44.679 DEBUG:boto:Data: 2024-04-12T03:28:44.679 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.679 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.679 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.679 DEBUG:boto:Params: {} 2024-04-12T03:28:44.680 DEBUG:boto:Token: None 2024-04-12T03:28:44.680 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/dummy 2024-04-12T03:28:44.680 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dlC5Wz/Gyfv4cMha2avVsv8V1zs= 2024-04-12T03:28:44.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dlC5Wz/Gyfv4cMha2avVsv8V1zs=', 'Content-Length': '0'} 2024-04-12T03:28:44.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000074bdc098d869b737-006618aa6c-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T03:28:44.682 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:28:44.683 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:28:44.683 DEBUG:boto:path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.683 DEBUG:boto:auth_path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.683 DEBUG:boto:Method: GET 2024-04-12T03:28:44.683 DEBUG:boto:Path: /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.683 DEBUG:boto:Data: 2024-04-12T03:28:44.683 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.683 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.683 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.683 DEBUG:boto:Params: {} 2024-04-12T03:28:44.683 DEBUG:boto:Token: None 2024-04-12T03:28:44.684 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.684 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ds9Mh1to4VtOu0gxm3xhleDMH+E= 2024-04-12T03:28:44.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ds9Mh1to4VtOu0gxm3xhleDMH+E=', 'Content-Length': '0'} 2024-04-12T03:28:44.686 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000531b2f40ef0b47a1-006618aa6c-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.686 DEBUG:boto:path=/dkyges-15/dummy 2024-04-12T03:28:44.686 DEBUG:boto:auth_path=/dkyges-15/dummy 2024-04-12T03:28:44.686 DEBUG:boto:path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.686 DEBUG:boto:auth_path=/dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.686 DEBUG:boto:Method: GET 2024-04-12T03:28:44.686 DEBUG:boto:Path: /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.686 DEBUG:boto:Data: 2024-04-12T03:28:44.687 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.687 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.687 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.687 DEBUG:boto:Params: {} 2024-04-12T03:28:44.687 DEBUG:boto:Token: None 2024-04-12T03:28:44.687 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/dummy?versionId=6v6vFUNe41NgUAF9IDH2X1ylAjDo-0- 2024-04-12T03:28:44.687 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ds9Mh1to4VtOu0gxm3xhleDMH+E= 2024-04-12T03:28:44.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ds9Mh1to4VtOu0gxm3xhleDMH+E=', 'Content-Length': '0'} 2024-04-12T03:28:44.689 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:13 GMT'), ('x-amz-version-id', '6v6vFUNe41NgUAF9IDH2X1ylAjDo-0-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e52da1d59904c56d-006618aa6c-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.689 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:44.689 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:44.689 DEBUG:boto:path=/dkyges-15/?prefix=dummy 2024-04-12T03:28:44.689 DEBUG:boto:auth_path=/dkyges-15/?prefix=dummy 2024-04-12T03:28:44.689 DEBUG:boto:Method: GET 2024-04-12T03:28:44.690 DEBUG:boto:Path: /dkyges-15/?prefix=dummy 2024-04-12T03:28:44.690 DEBUG:boto:Data: 2024-04-12T03:28:44.690 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.690 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.690 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.690 DEBUG:boto:Params: {} 2024-04-12T03:28:44.690 DEBUG:boto:Token: None 2024-04-12T03:28:44.690 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/ 2024-04-12T03:28:44.690 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4= 2024-04-12T03:28:44.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4=', 'Content-Length': '0'} 2024-04-12T03:28:44.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb98fad4e297445e-006618aa6c-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.693 DEBUG:boto:b'dkyges-15dummy1000falsedummy2024-04-12T03:26:13.184Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:44.693 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:44.693 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:44.694 DEBUG:boto:path=/dkyges-15/?prefix=dummy 2024-04-12T03:28:44.694 DEBUG:boto:auth_path=/dkyges-15/?prefix=dummy 2024-04-12T03:28:44.694 DEBUG:boto:Method: GET 2024-04-12T03:28:44.694 DEBUG:boto:Path: /dkyges-15/?prefix=dummy 2024-04-12T03:28:44.694 DEBUG:boto:Data: 2024-04-12T03:28:44.694 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.694 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.694 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.694 DEBUG:boto:Params: {} 2024-04-12T03:28:44.694 DEBUG:boto:Token: None 2024-04-12T03:28:44.695 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/ 2024-04-12T03:28:44.695 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4= 2024-04-12T03:28:44.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4=', 'Content-Length': '0'} 2024-04-12T03:28:44.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064eb0cc07379e0e7-006618aa6c-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.697 DEBUG:boto:b'dkyges-15dummy1000falsedummy2024-04-12T03:26:13.184Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:44.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:44.698 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.698 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.698 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.698 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.698 DEBUG:boto:Method: GET 2024-04-12T03:28:44.698 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.698 DEBUG:boto:Data: 2024-04-12T03:28:44.698 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.698 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.699 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.699 DEBUG:boto:Params: {} 2024-04-12T03:28:44.699 DEBUG:boto:Token: None 2024-04-12T03:28:44.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.699 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:71mqJ2s8gHSPYTdsnCqgeL4ckYI= 2024-04-12T03:28:44.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:71mqJ2s8gHSPYTdsnCqgeL4ckYI=', 'Content-Length': '0'} 2024-04-12T03:28:44.701 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e20d44bd22064fa3-006618aa6c-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.701 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.701 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.701 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.701 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.701 DEBUG:boto:Method: GET 2024-04-12T03:28:44.702 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.702 DEBUG:boto:Data: 2024-04-12T03:28:44.702 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.702 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.702 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.702 DEBUG:boto:Params: {} 2024-04-12T03:28:44.702 DEBUG:boto:Token: None 2024-04-12T03:28:44.702 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.702 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:71mqJ2s8gHSPYTdsnCqgeL4ckYI= 2024-04-12T03:28:44.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:71mqJ2s8gHSPYTdsnCqgeL4ckYI=', 'Content-Length': '0'} 2024-04-12T03:28:44.704 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e9b24ab649c87c74-006618aa6c-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.705 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.705 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.705 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.705 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.705 DEBUG:boto:Method: HEAD 2024-04-12T03:28:44.705 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.705 DEBUG:boto:Data: 2024-04-12T03:28:44.705 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.705 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.706 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.706 DEBUG:boto:Params: {} 2024-04-12T03:28:44.706 DEBUG:boto:Token: None 2024-04-12T03:28:44.706 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.706 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JkUmCPPMv9bur1aJCj85jjfKL2E= 2024-04-12T03:28:44.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JkUmCPPMv9bur1aJCj85jjfKL2E=', 'Content-Length': '0'} 2024-04-12T03:28:44.708 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000df9262f208f6276a-006618aa6c-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.708 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.708 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.708 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.708 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.709 DEBUG:boto:Method: HEAD 2024-04-12T03:28:44.709 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.709 DEBUG:boto:Data: 2024-04-12T03:28:44.709 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.709 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.709 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.709 DEBUG:boto:Params: {} 2024-04-12T03:28:44.709 DEBUG:boto:Token: None 2024-04-12T03:28:44.709 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.709 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JkUmCPPMv9bur1aJCj85jjfKL2E= 2024-04-12T03:28:44.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JkUmCPPMv9bur1aJCj85jjfKL2E=', 'Content-Length': '0'} 2024-04-12T03:28:44.711 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000902e55c8634cc4e4-006618aa6c-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:44.711 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.711 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.712 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.712 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.712 DEBUG:boto:Method: GET 2024-04-12T03:28:44.712 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.712 DEBUG:boto:Data: 2024-04-12T03:28:44.712 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.712 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.712 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.712 DEBUG:boto:Params: {} 2024-04-12T03:28:44.712 DEBUG:boto:Token: None 2024-04-12T03:28:44.713 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.713 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:71mqJ2s8gHSPYTdsnCqgeL4ckYI= 2024-04-12T03:28:44.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:71mqJ2s8gHSPYTdsnCqgeL4ckYI=', 'Content-Length': '0'} 2024-04-12T03:28:44.714 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004e4ae6b7f5b83017-006618aa6c-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.715 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.715 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.715 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.715 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.715 DEBUG:boto:Method: GET 2024-04-12T03:28:44.715 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.715 DEBUG:boto:Data: 2024-04-12T03:28:44.715 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.715 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.715 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.715 DEBUG:boto:Params: {} 2024-04-12T03:28:44.716 DEBUG:boto:Token: None 2024-04-12T03:28:44.716 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.716 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:71mqJ2s8gHSPYTdsnCqgeL4ckYI= 2024-04-12T03:28:44.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:71mqJ2s8gHSPYTdsnCqgeL4ckYI=', 'Content-Length': '0'} 2024-04-12T03:28:44.718 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efd20bca4ec3d02a-006618aa6c-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.758 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:44.758 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:44.759 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:44.759 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:44.759 DEBUG:boto:Method: GET 2024-04-12T03:28:44.759 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:44.759 DEBUG:boto:Data: 2024-04-12T03:28:44.759 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.759 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.759 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.759 DEBUG:boto:Params: {} 2024-04-12T03:28:44.759 DEBUG:boto:Token: None 2024-04-12T03:28:44.760 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/ 2024-04-12T03:28:44.760 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4= 2024-04-12T03:28:44.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4=', 'Content-Length': '0'} 2024-04-12T03:28:44.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ccd35ed95eb2276-006618aa6c-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.762 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:44.763 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:44.763 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:44.763 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:44.763 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:44.763 DEBUG:boto:Method: GET 2024-04-12T03:28:44.763 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:44.763 DEBUG:boto:Data: 2024-04-12T03:28:44.763 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.764 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.764 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.764 DEBUG:boto:Params: {} 2024-04-12T03:28:44.764 DEBUG:boto:Token: None 2024-04-12T03:28:44.764 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/ 2024-04-12T03:28:44.764 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4= 2024-04-12T03:28:44.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4=', 'Content-Length': '0'} 2024-04-12T03:28:44.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001528dd5887149c6f-006618aa6c-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.810 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:44.811 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.811 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.811 DEBUG:boto:Method: HEAD 2024-04-12T03:28:44.811 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:44.811 DEBUG:boto:Data: 2024-04-12T03:28:44.811 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.811 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.811 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.812 DEBUG:boto:Params: {} 2024-04-12T03:28:44.812 DEBUG:boto:Token: None 2024-04-12T03:28:44.812 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj 2024-04-12T03:28:44.812 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6uBdnxXCFtG/tfzCJCHP4n/X00A= 2024-04-12T03:28:44.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6uBdnxXCFtG/tfzCJCHP4n/X00A=', 'Content-Length': '0'} 2024-04-12T03:28:44.815 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000efcf6debf7395256-006618aa6c-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.816 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.816 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.816 DEBUG:boto:Method: HEAD 2024-04-12T03:28:44.816 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:44.816 DEBUG:boto:Data: 2024-04-12T03:28:44.816 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.816 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.816 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.816 DEBUG:boto:Params: {} 2024-04-12T03:28:44.817 DEBUG:boto:Token: None 2024-04-12T03:28:44.817 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj 2024-04-12T03:28:44.817 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6uBdnxXCFtG/tfzCJCHP4n/X00A= 2024-04-12T03:28:44.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6uBdnxXCFtG/tfzCJCHP4n/X00A=', 'Content-Length': '0'} 2024-04-12T03:28:44.820 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff4061c83aa9b56a-006618aa6c-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:44.820 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.820 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.821 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.821 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.821 DEBUG:boto:Method: GET 2024-04-12T03:28:44.821 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.821 DEBUG:boto:Data: 2024-04-12T03:28:44.821 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.821 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.821 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.821 DEBUG:boto:Params: {} 2024-04-12T03:28:44.821 DEBUG:boto:Token: None 2024-04-12T03:28:44.821 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.822 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:71mqJ2s8gHSPYTdsnCqgeL4ckYI= 2024-04-12T03:28:44.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:71mqJ2s8gHSPYTdsnCqgeL4ckYI=', 'Content-Length': '0'} 2024-04-12T03:28:44.823 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d542930cb68bd5e-006618aa6c-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.824 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.824 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.824 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.824 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.824 DEBUG:boto:Method: GET 2024-04-12T03:28:44.824 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.824 DEBUG:boto:Data: 2024-04-12T03:28:44.824 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.824 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.825 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.825 DEBUG:boto:Params: {} 2024-04-12T03:28:44.825 DEBUG:boto:Token: None 2024-04-12T03:28:44.825 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.825 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:71mqJ2s8gHSPYTdsnCqgeL4ckYI= 2024-04-12T03:28:44.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:71mqJ2s8gHSPYTdsnCqgeL4ckYI=', 'Content-Length': '0'} 2024-04-12T03:28:44.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000a0f4551b8e30a06-006618aa6c-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.827 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:44.827 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:44.827 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:44.827 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:44.827 DEBUG:boto:Method: GET 2024-04-12T03:28:44.828 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:44.828 DEBUG:boto:Data: 2024-04-12T03:28:44.828 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.828 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.828 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.828 DEBUG:boto:Params: {} 2024-04-12T03:28:44.828 DEBUG:boto:Token: None 2024-04-12T03:28:44.828 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/ 2024-04-12T03:28:44.828 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4= 2024-04-12T03:28:44.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4=', 'Content-Length': '0'} 2024-04-12T03:28:44.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e287269c23d6b796-006618aa6c-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.874 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:44.875 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:44.875 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:44.875 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:44.875 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:44.875 DEBUG:boto:Method: GET 2024-04-12T03:28:44.875 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:44.875 DEBUG:boto:Data: 2024-04-12T03:28:44.875 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.875 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.875 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.876 DEBUG:boto:Params: {} 2024-04-12T03:28:44.876 DEBUG:boto:Token: None 2024-04-12T03:28:44.876 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/ 2024-04-12T03:28:44.876 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4= 2024-04-12T03:28:44.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4=', 'Content-Length': '0'} 2024-04-12T03:28:44.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bd4d8db768066bd-006618aa6c-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.879 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:44.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:44.879 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.879 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.879 DEBUG:boto:path=/dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.880 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.880 DEBUG:boto:Method: GET 2024-04-12T03:28:44.880 DEBUG:boto:Path: /dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.880 DEBUG:boto:Data: 2024-04-12T03:28:44.880 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.880 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.880 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.880 DEBUG:boto:Params: {} 2024-04-12T03:28:44.880 DEBUG:boto:Token: None 2024-04-12T03:28:44.880 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.880 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JhCWcQRwsQlUQwifhZzQL8WY8jQ= 2024-04-12T03:28:44.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JhCWcQRwsQlUQwifhZzQL8WY8jQ=', 'Content-Length': '0'} 2024-04-12T03:28:44.882 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cad94dff55bdaaea-006618aa6c-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.883 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.883 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.883 DEBUG:boto:path=/dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.883 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.883 DEBUG:boto:Method: GET 2024-04-12T03:28:44.883 DEBUG:boto:Path: /dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.883 DEBUG:boto:Data: 2024-04-12T03:28:44.883 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.883 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.883 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.883 DEBUG:boto:Params: {} 2024-04-12T03:28:44.884 DEBUG:boto:Token: None 2024-04-12T03:28:44.884 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.884 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JhCWcQRwsQlUQwifhZzQL8WY8jQ= 2024-04-12T03:28:44.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JhCWcQRwsQlUQwifhZzQL8WY8jQ=', 'Content-Length': '0'} 2024-04-12T03:28:44.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf2c10ef4f6a4503-006618aa6c-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.926 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.926 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.926 DEBUG:boto:path=/dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.927 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.927 DEBUG:boto:Method: HEAD 2024-04-12T03:28:44.927 DEBUG:boto:Path: /dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.927 DEBUG:boto:Data: 2024-04-12T03:28:44.927 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.927 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.927 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.927 DEBUG:boto:Params: {} 2024-04-12T03:28:44.927 DEBUG:boto:Token: None 2024-04-12T03:28:44.927 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.927 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qJ9jfJKWbFbbPjmiNTqFztG8oH4= 2024-04-12T03:28:44.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qJ9jfJKWbFbbPjmiNTqFztG8oH4=', 'Content-Length': '0'} 2024-04-12T03:28:44.929 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbb31c64895103fb-006618aa6c-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.930 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.930 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.930 DEBUG:boto:path=/dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.930 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.930 DEBUG:boto:Method: HEAD 2024-04-12T03:28:44.930 DEBUG:boto:Path: /dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.930 DEBUG:boto:Data: 2024-04-12T03:28:44.930 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.930 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.930 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.930 DEBUG:boto:Params: {} 2024-04-12T03:28:44.931 DEBUG:boto:Token: None 2024-04-12T03:28:44.931 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.931 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qJ9jfJKWbFbbPjmiNTqFztG8oH4= 2024-04-12T03:28:44.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qJ9jfJKWbFbbPjmiNTqFztG8oH4=', 'Content-Length': '0'} 2024-04-12T03:28:44.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1e18b4155778915-006618aa6c-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:44.933 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.933 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.933 DEBUG:boto:path=/dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.933 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.933 DEBUG:boto:Method: GET 2024-04-12T03:28:44.933 DEBUG:boto:Path: /dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.933 DEBUG:boto:Data: 2024-04-12T03:28:44.933 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.933 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.934 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.934 DEBUG:boto:Params: {} 2024-04-12T03:28:44.934 DEBUG:boto:Token: None 2024-04-12T03:28:44.934 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.934 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JhCWcQRwsQlUQwifhZzQL8WY8jQ= 2024-04-12T03:28:44.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JhCWcQRwsQlUQwifhZzQL8WY8jQ=', 'Content-Length': '0'} 2024-04-12T03:28:44.936 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031db161385a11292-006618aa6c-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.936 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.936 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.936 DEBUG:boto:path=/dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.936 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.936 DEBUG:boto:Method: GET 2024-04-12T03:28:44.936 DEBUG:boto:Path: /dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.937 DEBUG:boto:Data: 2024-04-12T03:28:44.937 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.937 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.937 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.937 DEBUG:boto:Params: {} 2024-04-12T03:28:44.937 DEBUG:boto:Token: None 2024-04-12T03:28:44.937 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj?versionId=IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM 2024-04-12T03:28:44.937 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JhCWcQRwsQlUQwifhZzQL8WY8jQ= 2024-04-12T03:28:44.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JhCWcQRwsQlUQwifhZzQL8WY8jQ=', 'Content-Length': '0'} 2024-04-12T03:28:44.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'IM0gY0UkHgbX4pgoOgWmEMjY7n4stIM'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b95d375c48492190-006618aa6c-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.939 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:44.940 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:44.940 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:44.940 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:44.940 DEBUG:boto:Method: GET 2024-04-12T03:28:44.940 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:44.940 DEBUG:boto:Data: 2024-04-12T03:28:44.940 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.940 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.940 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.940 DEBUG:boto:Params: {} 2024-04-12T03:28:44.940 DEBUG:boto:Token: None 2024-04-12T03:28:44.940 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/ 2024-04-12T03:28:44.941 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4= 2024-04-12T03:28:44.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4=', 'Content-Length': '0'} 2024-04-12T03:28:44.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3daa804c1329aa7-006618aa6c-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.943 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:44.944 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:44.944 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:44.944 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:44.944 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:44.944 DEBUG:boto:Method: GET 2024-04-12T03:28:44.944 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:44.944 DEBUG:boto:Data: 2024-04-12T03:28:44.944 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.944 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.944 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.944 DEBUG:boto:Params: {} 2024-04-12T03:28:44.945 DEBUG:boto:Token: None 2024-04-12T03:28:44.945 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/ 2024-04-12T03:28:44.945 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4= 2024-04-12T03:28:44.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jlThcopy2FYBwG4txMkswxAzmQ4=', 'Content-Length': '0'} 2024-04-12T03:28:44.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014b3610f468553dd-006618aa6c-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.947 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:44.948 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.948 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.948 DEBUG:boto:Method: HEAD 2024-04-12T03:28:44.948 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:44.948 DEBUG:boto:Data: 2024-04-12T03:28:44.948 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.948 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.948 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.948 DEBUG:boto:Params: {} 2024-04-12T03:28:44.948 DEBUG:boto:Token: None 2024-04-12T03:28:44.949 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj 2024-04-12T03:28:44.949 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6uBdnxXCFtG/tfzCJCHP4n/X00A= 2024-04-12T03:28:44.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6uBdnxXCFtG/tfzCJCHP4n/X00A=', 'Content-Length': '0'} 2024-04-12T03:28:44.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002e83a906518de263-006618aa6c-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.952 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.952 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.952 DEBUG:boto:Method: HEAD 2024-04-12T03:28:44.952 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:44.952 DEBUG:boto:Data: 2024-04-12T03:28:44.952 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.952 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:44.952 DEBUG:boto:Port: 8000 2024-04-12T03:28:44.952 DEBUG:boto:Params: {} 2024-04-12T03:28:44.953 DEBUG:boto:Token: None 2024-04-12T03:28:44.953 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj 2024-04-12T03:28:44.953 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6uBdnxXCFtG/tfzCJCHP4n/X00A= 2024-04-12T03:28:44.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6uBdnxXCFtG/tfzCJCHP4n/X00A=', 'Content-Length': '0'} 2024-04-12T03:28:44.956 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006795101a3855a801-006618aa6c-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:44.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:44.956 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:44.956 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:44.956 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.957 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.957 DEBUG:boto:Method: GET 2024-04-12T03:28:44.957 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.957 DEBUG:boto:Data: 2024-04-12T03:28:44.957 DEBUG:boto:Headers: {} 2024-04-12T03:28:44.957 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:44.957 DEBUG:boto:Port: 8001 2024-04-12T03:28:44.957 DEBUG:boto:Params: {} 2024-04-12T03:28:44.957 DEBUG:boto:Token: None 2024-04-12T03:28:44.957 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:44 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:44.958 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:71mqJ2s8gHSPYTdsnCqgeL4ckYI= 2024-04-12T03:28:44.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:71mqJ2s8gHSPYTdsnCqgeL4ckYI=', 'Content-Length': '0'} 2024-04-12T03:28:44.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8fdbc3dd0aff575-006618aa6c-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.002 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.003 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.003 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.003 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.003 DEBUG:boto:Method: GET 2024-04-12T03:28:45.003 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.003 DEBUG:boto:Data: 2024-04-12T03:28:45.003 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.003 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.003 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.003 DEBUG:boto:Params: {} 2024-04-12T03:28:45.003 DEBUG:boto:Token: None 2024-04-12T03:28:45.004 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ= 2024-04-12T03:28:45.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000483e856ae059a2a1-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.006 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.006 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.006 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.006 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.006 DEBUG:boto:Method: GET 2024-04-12T03:28:45.006 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.007 DEBUG:boto:Data: 2024-04-12T03:28:45.007 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.007 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.007 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.007 DEBUG:boto:Params: {} 2024-04-12T03:28:45.007 DEBUG:boto:Token: None 2024-04-12T03:28:45.007 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.007 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000004e3b731f094a34-006618aa6d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.010 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.010 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.011 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.011 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.011 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.011 DEBUG:boto:Method: GET 2024-04-12T03:28:45.011 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.011 DEBUG:boto:Data: 2024-04-12T03:28:45.011 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.011 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.011 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.011 DEBUG:boto:Params: {} 2024-04-12T03:28:45.011 DEBUG:boto:Token: None 2024-04-12T03:28:45.012 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.012 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce84ea66b6619973-006618aa6d-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.014 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.015 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.015 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.015 DEBUG:boto:path=/dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.015 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.015 DEBUG:boto:Method: GET 2024-04-12T03:28:45.015 DEBUG:boto:Path: /dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.015 DEBUG:boto:Data: 2024-04-12T03:28:45.015 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.015 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.015 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.016 DEBUG:boto:Params: {} 2024-04-12T03:28:45.016 DEBUG:boto:Token: None 2024-04-12T03:28:45.016 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.016 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kbBE59yAullGaNlcZU8JJORbSuY= 2024-04-12T03:28:45.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kbBE59yAullGaNlcZU8JJORbSuY=', 'Content-Length': '0'} 2024-04-12T03:28:45.018 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'Xu58tKQgwRtuVVd31MDUlibB214glR3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fbdcbf66c3c4dd41-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.018 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.018 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.018 DEBUG:boto:path=/dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.018 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.018 DEBUG:boto:Method: GET 2024-04-12T03:28:45.018 DEBUG:boto:Path: /dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.019 DEBUG:boto:Data: 2024-04-12T03:28:45.019 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.019 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.019 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.019 DEBUG:boto:Params: {} 2024-04-12T03:28:45.019 DEBUG:boto:Token: None 2024-04-12T03:28:45.019 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.019 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kbBE59yAullGaNlcZU8JJORbSuY= 2024-04-12T03:28:45.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kbBE59yAullGaNlcZU8JJORbSuY=', 'Content-Length': '0'} 2024-04-12T03:28:45.021 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'Xu58tKQgwRtuVVd31MDUlibB214glR3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000670f2b418d86b763-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.021 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.021 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.022 DEBUG:boto:path=/dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.022 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.022 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.022 DEBUG:boto:Path: /dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.022 DEBUG:boto:Data: 2024-04-12T03:28:45.022 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.022 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.022 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.022 DEBUG:boto:Params: {} 2024-04-12T03:28:45.022 DEBUG:boto:Token: None 2024-04-12T03:28:45.023 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.023 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cCeVreZIgySCrAxrDYXoKFxlLdQ= 2024-04-12T03:28:45.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cCeVreZIgySCrAxrDYXoKFxlLdQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.024 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'Xu58tKQgwRtuVVd31MDUlibB214glR3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007d7f75527fe2ea51-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.024 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.025 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.025 DEBUG:boto:path=/dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.025 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.025 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.025 DEBUG:boto:Path: /dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.025 DEBUG:boto:Data: 2024-04-12T03:28:45.025 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.025 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.025 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.025 DEBUG:boto:Params: {} 2024-04-12T03:28:45.025 DEBUG:boto:Token: None 2024-04-12T03:28:45.026 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.026 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cCeVreZIgySCrAxrDYXoKFxlLdQ= 2024-04-12T03:28:45.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cCeVreZIgySCrAxrDYXoKFxlLdQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.027 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'Xu58tKQgwRtuVVd31MDUlibB214glR3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018deff6d4d2b3950-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.028 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.028 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.028 DEBUG:boto:path=/dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.028 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.028 DEBUG:boto:Method: GET 2024-04-12T03:28:45.028 DEBUG:boto:Path: /dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.028 DEBUG:boto:Data: 2024-04-12T03:28:45.028 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.029 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.029 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.029 DEBUG:boto:Params: {} 2024-04-12T03:28:45.029 DEBUG:boto:Token: None 2024-04-12T03:28:45.029 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.029 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kbBE59yAullGaNlcZU8JJORbSuY= 2024-04-12T03:28:45.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kbBE59yAullGaNlcZU8JJORbSuY=', 'Content-Length': '0'} 2024-04-12T03:28:45.031 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'Xu58tKQgwRtuVVd31MDUlibB214glR3'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fd6b704f3bad587-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.031 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.031 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.031 DEBUG:boto:path=/dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.032 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.032 DEBUG:boto:Method: GET 2024-04-12T03:28:45.032 DEBUG:boto:Path: /dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.032 DEBUG:boto:Data: 2024-04-12T03:28:45.032 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.032 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.032 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.032 DEBUG:boto:Params: {} 2024-04-12T03:28:45.032 DEBUG:boto:Token: None 2024-04-12T03:28:45.032 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=Xu58tKQgwRtuVVd31MDUlibB214glR3 2024-04-12T03:28:45.032 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kbBE59yAullGaNlcZU8JJORbSuY= 2024-04-12T03:28:45.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kbBE59yAullGaNlcZU8JJORbSuY=', 'Content-Length': '0'} 2024-04-12T03:28:45.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'Xu58tKQgwRtuVVd31MDUlibB214glR3'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000cf50ddda4eea977-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.034 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.035 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.035 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.035 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.035 DEBUG:boto:Method: GET 2024-04-12T03:28:45.035 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.035 DEBUG:boto:Data: 2024-04-12T03:28:45.035 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.035 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.035 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.035 DEBUG:boto:Params: {} 2024-04-12T03:28:45.035 DEBUG:boto:Token: None 2024-04-12T03:28:45.036 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.036 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bade3755ba8fcdba-006618aa6d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.040 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.040 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.040 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.040 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.040 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.040 DEBUG:boto:Method: GET 2024-04-12T03:28:45.040 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.041 DEBUG:boto:Data: 2024-04-12T03:28:45.041 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.041 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.041 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.041 DEBUG:boto:Params: {} 2024-04-12T03:28:45.041 DEBUG:boto:Token: None 2024-04-12T03:28:45.041 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.041 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000473666ab6c44bfb7-006618aa6d-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.044 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.044 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.044 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.044 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.045 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:45.045 DEBUG:boto:Data: 2024-04-12T03:28:45.045 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.045 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.045 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.045 DEBUG:boto:Params: {} 2024-04-12T03:28:45.045 DEBUG:boto:Token: None 2024-04-12T03:28:45.045 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj 2024-04-12T03:28:45.045 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk= 2024-04-12T03:28:45.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk=', 'Content-Length': '0'} 2024-04-12T03:28:45.049 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007840a7b758852522-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.049 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.049 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.049 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.049 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:45.049 DEBUG:boto:Data: 2024-04-12T03:28:45.049 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.049 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.049 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.050 DEBUG:boto:Params: {} 2024-04-12T03:28:45.050 DEBUG:boto:Token: None 2024-04-12T03:28:45.050 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj 2024-04-12T03:28:45.050 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk= 2024-04-12T03:28:45.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk=', 'Content-Length': '0'} 2024-04-12T03:28:45.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ccbdfbbee6c0e4b6-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.053 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.054 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.054 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.054 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.054 DEBUG:boto:Method: GET 2024-04-12T03:28:45.054 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.054 DEBUG:boto:Data: 2024-04-12T03:28:45.054 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.054 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.054 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.054 DEBUG:boto:Params: {} 2024-04-12T03:28:45.054 DEBUG:boto:Token: None 2024-04-12T03:28:45.055 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.055 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ= 2024-04-12T03:28:45.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056de62c968d70a72-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.098 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.099 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.099 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.099 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.099 DEBUG:boto:Method: GET 2024-04-12T03:28:45.099 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.099 DEBUG:boto:Data: 2024-04-12T03:28:45.099 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.099 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.099 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.099 DEBUG:boto:Params: {} 2024-04-12T03:28:45.099 DEBUG:boto:Token: None 2024-04-12T03:28:45.100 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.100 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ= 2024-04-12T03:28:45.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000956ed4a168da8001-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.102 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.103 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.103 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.103 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.103 DEBUG:boto:Method: GET 2024-04-12T03:28:45.103 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.103 DEBUG:boto:Data: 2024-04-12T03:28:45.103 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.103 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.103 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.103 DEBUG:boto:Params: {} 2024-04-12T03:28:45.103 DEBUG:boto:Token: None 2024-04-12T03:28:45.104 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.104 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b140bbc95ce7fa7-006618aa6d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.106 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.107 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.107 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.107 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.107 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.107 DEBUG:boto:Method: GET 2024-04-12T03:28:45.107 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.107 DEBUG:boto:Data: 2024-04-12T03:28:45.107 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.108 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.108 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.108 DEBUG:boto:Params: {} 2024-04-12T03:28:45.108 DEBUG:boto:Token: None 2024-04-12T03:28:45.108 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.108 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000124671095434dda4-006618aa6d-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.112 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.112 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.112 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.112 DEBUG:boto:path=/dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.113 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.113 DEBUG:boto:Method: GET 2024-04-12T03:28:45.113 DEBUG:boto:Path: /dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.113 DEBUG:boto:Data: 2024-04-12T03:28:45.113 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.113 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.113 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.113 DEBUG:boto:Params: {} 2024-04-12T03:28:45.113 DEBUG:boto:Token: None 2024-04-12T03:28:45.113 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.114 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pEPtyM7eAUF59wT03P7gUHts1fw= 2024-04-12T03:28:45.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pEPtyM7eAUF59wT03P7gUHts1fw=', 'Content-Length': '0'} 2024-04-12T03:28:45.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000005a83d46b9fd94a-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.116 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.116 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.116 DEBUG:boto:path=/dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.116 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.116 DEBUG:boto:Method: GET 2024-04-12T03:28:45.116 DEBUG:boto:Path: /dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.116 DEBUG:boto:Data: 2024-04-12T03:28:45.116 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.116 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.117 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.117 DEBUG:boto:Params: {} 2024-04-12T03:28:45.117 DEBUG:boto:Token: None 2024-04-12T03:28:45.117 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.117 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pEPtyM7eAUF59wT03P7gUHts1fw= 2024-04-12T03:28:45.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pEPtyM7eAUF59wT03P7gUHts1fw=', 'Content-Length': '0'} 2024-04-12T03:28:45.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d47dabe9bb71306-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.119 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.119 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.119 DEBUG:boto:path=/dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.120 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.120 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.120 DEBUG:boto:Path: /dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.120 DEBUG:boto:Data: 2024-04-12T03:28:45.120 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.120 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.120 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.120 DEBUG:boto:Params: {} 2024-04-12T03:28:45.120 DEBUG:boto:Token: None 2024-04-12T03:28:45.120 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.120 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OCvP1BM/DcveAuXS16WG+Wy59F0= 2024-04-12T03:28:45.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OCvP1BM/DcveAuXS16WG+Wy59F0=', 'Content-Length': '0'} 2024-04-12T03:28:45.122 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9473ddde34336ff-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.123 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.123 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.123 DEBUG:boto:path=/dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.123 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.123 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.123 DEBUG:boto:Path: /dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.123 DEBUG:boto:Data: 2024-04-12T03:28:45.123 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.123 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.124 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.124 DEBUG:boto:Params: {} 2024-04-12T03:28:45.124 DEBUG:boto:Token: None 2024-04-12T03:28:45.124 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.124 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OCvP1BM/DcveAuXS16WG+Wy59F0= 2024-04-12T03:28:45.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OCvP1BM/DcveAuXS16WG+Wy59F0=', 'Content-Length': '0'} 2024-04-12T03:28:45.126 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8275297daa47205-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.126 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.126 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.126 DEBUG:boto:path=/dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.126 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.126 DEBUG:boto:Method: GET 2024-04-12T03:28:45.127 DEBUG:boto:Path: /dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.127 DEBUG:boto:Data: 2024-04-12T03:28:45.127 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.127 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.127 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.127 DEBUG:boto:Params: {} 2024-04-12T03:28:45.127 DEBUG:boto:Token: None 2024-04-12T03:28:45.127 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.127 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pEPtyM7eAUF59wT03P7gUHts1fw= 2024-04-12T03:28:45.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pEPtyM7eAUF59wT03P7gUHts1fw=', 'Content-Length': '0'} 2024-04-12T03:28:45.129 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0336cb7df82a92d-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.170 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.170 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.170 DEBUG:boto:path=/dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.171 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.171 DEBUG:boto:Method: GET 2024-04-12T03:28:45.171 DEBUG:boto:Path: /dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.171 DEBUG:boto:Data: 2024-04-12T03:28:45.171 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.171 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.171 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.171 DEBUG:boto:Params: {} 2024-04-12T03:28:45.171 DEBUG:boto:Token: None 2024-04-12T03:28:45.171 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG 2024-04-12T03:28:45.172 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pEPtyM7eAUF59wT03P7gUHts1fw= 2024-04-12T03:28:45.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pEPtyM7eAUF59wT03P7gUHts1fw=', 'Content-Length': '0'} 2024-04-12T03:28:45.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'qcfWRWm3IR8-4Qy5hjwNTT.c3BvKLTG'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6d838125f342387-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.174 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.174 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.174 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.174 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.174 DEBUG:boto:Method: GET 2024-04-12T03:28:45.174 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.174 DEBUG:boto:Data: 2024-04-12T03:28:45.174 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.174 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.175 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.175 DEBUG:boto:Params: {} 2024-04-12T03:28:45.175 DEBUG:boto:Token: None 2024-04-12T03:28:45.175 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.175 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d143f27642d300c-006618aa6d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.177 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.178 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.178 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.178 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.178 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.178 DEBUG:boto:Method: GET 2024-04-12T03:28:45.178 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.178 DEBUG:boto:Data: 2024-04-12T03:28:45.179 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.179 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.179 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.179 DEBUG:boto:Params: {} 2024-04-12T03:28:45.179 DEBUG:boto:Token: None 2024-04-12T03:28:45.179 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.179 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bbdd24a877c3d52-006618aa6d-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.182 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.182 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.182 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.183 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.183 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:45.183 DEBUG:boto:Data: 2024-04-12T03:28:45.183 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.183 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.183 DEBUG:boto:Params: {} 2024-04-12T03:28:45.183 DEBUG:boto:Token: None 2024-04-12T03:28:45.183 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj 2024-04-12T03:28:45.183 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk= 2024-04-12T03:28:45.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk=', 'Content-Length': '0'} 2024-04-12T03:28:45.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000db8be5040ca4b0d7-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.187 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.187 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.187 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.187 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:45.187 DEBUG:boto:Data: 2024-04-12T03:28:45.188 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.188 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.188 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.188 DEBUG:boto:Params: {} 2024-04-12T03:28:45.188 DEBUG:boto:Token: None 2024-04-12T03:28:45.188 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj 2024-04-12T03:28:45.188 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk= 2024-04-12T03:28:45.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk=', 'Content-Length': '0'} 2024-04-12T03:28:45.191 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000098b93148e9c3024c-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.192 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.192 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.192 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.192 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.192 DEBUG:boto:Method: GET 2024-04-12T03:28:45.192 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.192 DEBUG:boto:Data: 2024-04-12T03:28:45.192 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.192 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.193 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.193 DEBUG:boto:Params: {} 2024-04-12T03:28:45.193 DEBUG:boto:Token: None 2024-04-12T03:28:45.193 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.193 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ= 2024-04-12T03:28:45.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.195 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000970daf8d2acc8de3-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.195 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.195 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.195 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.195 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.196 DEBUG:boto:Method: GET 2024-04-12T03:28:45.196 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.196 DEBUG:boto:Data: 2024-04-12T03:28:45.196 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.196 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.196 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.196 DEBUG:boto:Params: {} 2024-04-12T03:28:45.196 DEBUG:boto:Token: None 2024-04-12T03:28:45.196 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.196 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ= 2024-04-12T03:28:45.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.199 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000063684a411a2667ff-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.242 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.243 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.243 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.243 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.243 DEBUG:boto:Method: GET 2024-04-12T03:28:45.243 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.243 DEBUG:boto:Data: 2024-04-12T03:28:45.243 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.243 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.243 DEBUG:boto:Params: {} 2024-04-12T03:28:45.244 DEBUG:boto:Token: None 2024-04-12T03:28:45.244 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.244 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9b56ca6c50e4d13-006618aa6d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.247 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.247 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.247 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.247 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.247 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.247 DEBUG:boto:Method: GET 2024-04-12T03:28:45.247 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.248 DEBUG:boto:Data: 2024-04-12T03:28:45.248 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.248 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.248 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.248 DEBUG:boto:Params: {} 2024-04-12T03:28:45.248 DEBUG:boto:Token: None 2024-04-12T03:28:45.248 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.248 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095c58bfd61bf94f2-006618aa6d-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.251 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.252 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.252 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.252 DEBUG:boto:path=/dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.252 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.252 DEBUG:boto:Method: GET 2024-04-12T03:28:45.252 DEBUG:boto:Path: /dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.252 DEBUG:boto:Data: 2024-04-12T03:28:45.252 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.252 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.252 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.252 DEBUG:boto:Params: {} 2024-04-12T03:28:45.252 DEBUG:boto:Token: None 2024-04-12T03:28:45.253 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.253 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oQEbduDGFMzJFRTXoHo0AVMoHxw= 2024-04-12T03:28:45.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oQEbduDGFMzJFRTXoHo0AVMoHxw=', 'Content-Length': '0'} 2024-04-12T03:28:45.255 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'crJW1cr7W5PZhyrypuyIfUhJav0.RMm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000042b2082461a3180-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.298 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.299 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.299 DEBUG:boto:path=/dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.299 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.299 DEBUG:boto:Method: GET 2024-04-12T03:28:45.299 DEBUG:boto:Path: /dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.299 DEBUG:boto:Data: 2024-04-12T03:28:45.299 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.299 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.299 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.299 DEBUG:boto:Params: {} 2024-04-12T03:28:45.300 DEBUG:boto:Token: None 2024-04-12T03:28:45.300 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.300 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oQEbduDGFMzJFRTXoHo0AVMoHxw= 2024-04-12T03:28:45.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oQEbduDGFMzJFRTXoHo0AVMoHxw=', 'Content-Length': '0'} 2024-04-12T03:28:45.302 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'crJW1cr7W5PZhyrypuyIfUhJav0.RMm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af8a74579791290a-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.302 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.302 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.302 DEBUG:boto:path=/dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.303 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.303 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.303 DEBUG:boto:Path: /dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.303 DEBUG:boto:Data: 2024-04-12T03:28:45.303 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.303 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.303 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.303 DEBUG:boto:Params: {} 2024-04-12T03:28:45.303 DEBUG:boto:Token: None 2024-04-12T03:28:45.303 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.303 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FFx4JCD+xx6AcQLs6/ARFISC1qE= 2024-04-12T03:28:45.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FFx4JCD+xx6AcQLs6/ARFISC1qE=', 'Content-Length': '0'} 2024-04-12T03:28:45.305 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'crJW1cr7W5PZhyrypuyIfUhJav0.RMm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c931abe81356c7ec-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.306 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.306 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.306 DEBUG:boto:path=/dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.306 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.306 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.306 DEBUG:boto:Path: /dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.306 DEBUG:boto:Data: 2024-04-12T03:28:45.306 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.306 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.306 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.307 DEBUG:boto:Params: {} 2024-04-12T03:28:45.307 DEBUG:boto:Token: None 2024-04-12T03:28:45.307 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.307 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FFx4JCD+xx6AcQLs6/ARFISC1qE= 2024-04-12T03:28:45.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FFx4JCD+xx6AcQLs6/ARFISC1qE=', 'Content-Length': '0'} 2024-04-12T03:28:45.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'crJW1cr7W5PZhyrypuyIfUhJav0.RMm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9e301e3a599a7e4-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.309 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.309 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.309 DEBUG:boto:path=/dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.309 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.310 DEBUG:boto:Method: GET 2024-04-12T03:28:45.310 DEBUG:boto:Path: /dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.310 DEBUG:boto:Data: 2024-04-12T03:28:45.310 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.310 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.310 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.310 DEBUG:boto:Params: {} 2024-04-12T03:28:45.310 DEBUG:boto:Token: None 2024-04-12T03:28:45.310 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.310 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oQEbduDGFMzJFRTXoHo0AVMoHxw= 2024-04-12T03:28:45.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oQEbduDGFMzJFRTXoHo0AVMoHxw=', 'Content-Length': '0'} 2024-04-12T03:28:45.312 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'crJW1cr7W5PZhyrypuyIfUhJav0.RMm'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d99a087e38e29fc2-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.312 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.313 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.313 DEBUG:boto:path=/dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.313 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.313 DEBUG:boto:Method: GET 2024-04-12T03:28:45.313 DEBUG:boto:Path: /dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.313 DEBUG:boto:Data: 2024-04-12T03:28:45.313 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.313 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.313 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.313 DEBUG:boto:Params: {} 2024-04-12T03:28:45.314 DEBUG:boto:Token: None 2024-04-12T03:28:45.314 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=crJW1cr7W5PZhyrypuyIfUhJav0.RMm 2024-04-12T03:28:45.314 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oQEbduDGFMzJFRTXoHo0AVMoHxw= 2024-04-12T03:28:45.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oQEbduDGFMzJFRTXoHo0AVMoHxw=', 'Content-Length': '0'} 2024-04-12T03:28:45.316 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'crJW1cr7W5PZhyrypuyIfUhJav0.RMm'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000092b45127ada90df0-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.316 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.316 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.316 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.316 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.316 DEBUG:boto:Method: GET 2024-04-12T03:28:45.316 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.317 DEBUG:boto:Data: 2024-04-12T03:28:45.317 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.317 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.317 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.317 DEBUG:boto:Params: {} 2024-04-12T03:28:45.317 DEBUG:boto:Token: None 2024-04-12T03:28:45.317 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.317 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f1438ef14fbc7bb-006618aa6d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.320 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.320 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.321 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.321 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.321 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.321 DEBUG:boto:Method: GET 2024-04-12T03:28:45.321 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.321 DEBUG:boto:Data: 2024-04-12T03:28:45.321 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.321 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.321 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.321 DEBUG:boto:Params: {} 2024-04-12T03:28:45.321 DEBUG:boto:Token: None 2024-04-12T03:28:45.322 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.322 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ac4bfeceb172535-006618aa6d-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.324 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.325 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.325 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.325 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.325 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:45.325 DEBUG:boto:Data: 2024-04-12T03:28:45.325 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.325 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.325 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.325 DEBUG:boto:Params: {} 2024-04-12T03:28:45.326 DEBUG:boto:Token: None 2024-04-12T03:28:45.326 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj 2024-04-12T03:28:45.326 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk= 2024-04-12T03:28:45.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk=', 'Content-Length': '0'} 2024-04-12T03:28:45.329 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004fc1bcbab19555d0-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.329 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.330 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.330 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.330 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:45.330 DEBUG:boto:Data: 2024-04-12T03:28:45.330 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.330 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.330 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.330 DEBUG:boto:Params: {} 2024-04-12T03:28:45.330 DEBUG:boto:Token: None 2024-04-12T03:28:45.330 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj 2024-04-12T03:28:45.331 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk= 2024-04-12T03:28:45.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk=', 'Content-Length': '0'} 2024-04-12T03:28:45.334 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f027ad54b92e2065-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.334 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.334 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.334 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.334 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.335 DEBUG:boto:Method: GET 2024-04-12T03:28:45.335 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.335 DEBUG:boto:Data: 2024-04-12T03:28:45.335 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.335 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.335 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.335 DEBUG:boto:Params: {} 2024-04-12T03:28:45.335 DEBUG:boto:Token: None 2024-04-12T03:28:45.335 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.335 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ= 2024-04-12T03:28:45.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000087974c0b9da83472-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.337 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.338 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.338 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.338 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.338 DEBUG:boto:Method: GET 2024-04-12T03:28:45.338 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.338 DEBUG:boto:Data: 2024-04-12T03:28:45.338 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.338 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.338 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.338 DEBUG:boto:Params: {} 2024-04-12T03:28:45.339 DEBUG:boto:Token: None 2024-04-12T03:28:45.339 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.339 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ= 2024-04-12T03:28:45.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068ab58396faea0ba-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.341 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.341 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.341 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.341 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.341 DEBUG:boto:Method: GET 2024-04-12T03:28:45.341 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.341 DEBUG:boto:Data: 2024-04-12T03:28:45.342 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.342 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.342 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.342 DEBUG:boto:Params: {} 2024-04-12T03:28:45.342 DEBUG:boto:Token: None 2024-04-12T03:28:45.342 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.342 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007effadc08bd3c74d-006618aa6d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.345 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.345 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.345 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.346 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.346 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.346 DEBUG:boto:Method: GET 2024-04-12T03:28:45.346 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.346 DEBUG:boto:Data: 2024-04-12T03:28:45.346 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.346 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.346 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.346 DEBUG:boto:Params: {} 2024-04-12T03:28:45.346 DEBUG:boto:Token: None 2024-04-12T03:28:45.347 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.347 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa766046597ea3eb-006618aa6d-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.350 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.350 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.350 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.350 DEBUG:boto:path=/dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.351 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.351 DEBUG:boto:Method: GET 2024-04-12T03:28:45.351 DEBUG:boto:Path: /dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.351 DEBUG:boto:Data: 2024-04-12T03:28:45.351 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.351 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.351 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.351 DEBUG:boto:Params: {} 2024-04-12T03:28:45.351 DEBUG:boto:Token: None 2024-04-12T03:28:45.351 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.351 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MoBxM7Jcto7JzurCGwwSQc+n0SY= 2024-04-12T03:28:45.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MoBxM7Jcto7JzurCGwwSQc+n0SY=', 'Content-Length': '0'} 2024-04-12T03:28:45.354 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007623eaec079a6da4-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.354 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.354 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.354 DEBUG:boto:path=/dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.355 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.355 DEBUG:boto:Method: GET 2024-04-12T03:28:45.355 DEBUG:boto:Path: /dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.355 DEBUG:boto:Data: 2024-04-12T03:28:45.355 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.355 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.355 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.355 DEBUG:boto:Params: {} 2024-04-12T03:28:45.355 DEBUG:boto:Token: None 2024-04-12T03:28:45.355 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.355 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MoBxM7Jcto7JzurCGwwSQc+n0SY= 2024-04-12T03:28:45.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MoBxM7Jcto7JzurCGwwSQc+n0SY=', 'Content-Length': '0'} 2024-04-12T03:28:45.357 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008be4e34590789da0-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.357 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.358 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.358 DEBUG:boto:path=/dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.358 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.358 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.358 DEBUG:boto:Path: /dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.358 DEBUG:boto:Data: 2024-04-12T03:28:45.358 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.358 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.358 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.358 DEBUG:boto:Params: {} 2024-04-12T03:28:45.359 DEBUG:boto:Token: None 2024-04-12T03:28:45.359 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.359 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KSvMXNL7nsJ9V0thyKVUISLZjSA= 2024-04-12T03:28:45.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KSvMXNL7nsJ9V0thyKVUISLZjSA=', 'Content-Length': '0'} 2024-04-12T03:28:45.361 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0fd58d56db40303-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.361 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.361 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.361 DEBUG:boto:path=/dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.361 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.361 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.361 DEBUG:boto:Path: /dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.361 DEBUG:boto:Data: 2024-04-12T03:28:45.361 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.361 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.362 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.362 DEBUG:boto:Params: {} 2024-04-12T03:28:45.362 DEBUG:boto:Token: None 2024-04-12T03:28:45.362 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.362 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KSvMXNL7nsJ9V0thyKVUISLZjSA= 2024-04-12T03:28:45.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KSvMXNL7nsJ9V0thyKVUISLZjSA=', 'Content-Length': '0'} 2024-04-12T03:28:45.364 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6ab321e6825e778-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.364 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.364 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.364 DEBUG:boto:path=/dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.364 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.364 DEBUG:boto:Method: GET 2024-04-12T03:28:45.365 DEBUG:boto:Path: /dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.365 DEBUG:boto:Data: 2024-04-12T03:28:45.365 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.365 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.365 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.365 DEBUG:boto:Params: {} 2024-04-12T03:28:45.365 DEBUG:boto:Token: None 2024-04-12T03:28:45.365 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.365 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MoBxM7Jcto7JzurCGwwSQc+n0SY= 2024-04-12T03:28:45.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MoBxM7Jcto7JzurCGwwSQc+n0SY=', 'Content-Length': '0'} 2024-04-12T03:28:45.367 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b7d2e8640ff06b2-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.410 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.410 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.411 DEBUG:boto:path=/dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.411 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.411 DEBUG:boto:Method: GET 2024-04-12T03:28:45.411 DEBUG:boto:Path: /dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.411 DEBUG:boto:Data: 2024-04-12T03:28:45.411 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.411 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.411 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.411 DEBUG:boto:Params: {} 2024-04-12T03:28:45.411 DEBUG:boto:Token: None 2024-04-12T03:28:45.411 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i 2024-04-12T03:28:45.412 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MoBxM7Jcto7JzurCGwwSQc+n0SY= 2024-04-12T03:28:45.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MoBxM7Jcto7JzurCGwwSQc+n0SY=', 'Content-Length': '0'} 2024-04-12T03:28:45.414 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'pYCRYGlATwh0QVzhGZxDIKJ9LcInd2i'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da87af9c9a198b07-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.414 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.414 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.414 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.414 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.414 DEBUG:boto:Method: GET 2024-04-12T03:28:45.415 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.415 DEBUG:boto:Data: 2024-04-12T03:28:45.415 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.415 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.415 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.415 DEBUG:boto:Params: {} 2024-04-12T03:28:45.415 DEBUG:boto:Token: None 2024-04-12T03:28:45.415 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.415 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032d69f8e85b0c222-006618aa6d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.458 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.459 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.459 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.459 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.459 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.459 DEBUG:boto:Method: GET 2024-04-12T03:28:45.459 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.459 DEBUG:boto:Data: 2024-04-12T03:28:45.459 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.459 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.459 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.459 DEBUG:boto:Params: {} 2024-04-12T03:28:45.460 DEBUG:boto:Token: None 2024-04-12T03:28:45.460 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.460 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fe793539956afc9-006618aa6d-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.466 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.467 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.467 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.467 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.467 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:45.467 DEBUG:boto:Data: 2024-04-12T03:28:45.467 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.467 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.467 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.468 DEBUG:boto:Params: {} 2024-04-12T03:28:45.468 DEBUG:boto:Token: None 2024-04-12T03:28:45.468 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj 2024-04-12T03:28:45.468 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk= 2024-04-12T03:28:45.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk=', 'Content-Length': '0'} 2024-04-12T03:28:45.474 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000425bdf9eb1fe09fa-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.475 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.475 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.475 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.475 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:45.475 DEBUG:boto:Data: 2024-04-12T03:28:45.475 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.475 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.475 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.475 DEBUG:boto:Params: {} 2024-04-12T03:28:45.475 DEBUG:boto:Token: None 2024-04-12T03:28:45.475 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj 2024-04-12T03:28:45.476 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk= 2024-04-12T03:28:45.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk=', 'Content-Length': '0'} 2024-04-12T03:28:45.479 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a6d3c928992a9784-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.479 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.479 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.479 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.479 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.480 DEBUG:boto:Method: GET 2024-04-12T03:28:45.480 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.480 DEBUG:boto:Data: 2024-04-12T03:28:45.480 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.480 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.480 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.480 DEBUG:boto:Params: {} 2024-04-12T03:28:45.480 DEBUG:boto:Token: None 2024-04-12T03:28:45.480 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.480 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ= 2024-04-12T03:28:45.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.482 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ea6fed050aabadc-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.482 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.482 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.482 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.483 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.483 DEBUG:boto:Method: GET 2024-04-12T03:28:45.483 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.483 DEBUG:boto:Data: 2024-04-12T03:28:45.483 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.483 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.483 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.483 DEBUG:boto:Params: {} 2024-04-12T03:28:45.483 DEBUG:boto:Token: None 2024-04-12T03:28:45.483 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.483 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ= 2024-04-12T03:28:45.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.485 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009af19a046ba93d46-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.526 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.527 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.527 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.527 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.527 DEBUG:boto:Method: GET 2024-04-12T03:28:45.527 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.527 DEBUG:boto:Data: 2024-04-12T03:28:45.527 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.527 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.527 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.527 DEBUG:boto:Params: {} 2024-04-12T03:28:45.528 DEBUG:boto:Token: None 2024-04-12T03:28:45.528 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.528 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fd8fd8cbe7ae143-006618aa6d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.530 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.531 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.531 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.531 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.531 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.531 DEBUG:boto:Method: GET 2024-04-12T03:28:45.531 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.531 DEBUG:boto:Data: 2024-04-12T03:28:45.532 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.532 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.532 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.532 DEBUG:boto:Params: {} 2024-04-12T03:28:45.532 DEBUG:boto:Token: None 2024-04-12T03:28:45.532 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.532 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047875217ec2152e7-006618aa6d-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.535 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.535 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.535 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.535 DEBUG:boto:path=/dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.536 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.536 DEBUG:boto:Method: GET 2024-04-12T03:28:45.536 DEBUG:boto:Path: /dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.536 DEBUG:boto:Data: 2024-04-12T03:28:45.536 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.536 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.536 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.536 DEBUG:boto:Params: {} 2024-04-12T03:28:45.536 DEBUG:boto:Token: None 2024-04-12T03:28:45.536 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.537 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TcaMBLIj/vYtay0JMnEvc5w89qE= 2024-04-12T03:28:45.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TcaMBLIj/vYtay0JMnEvc5w89qE=', 'Content-Length': '0'} 2024-04-12T03:28:45.539 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', '6gubSYrBrkePUl8NKnDsGwX3jArFN7d'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073ac7049ad4fd0a3-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.582 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.582 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.583 DEBUG:boto:path=/dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.583 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.583 DEBUG:boto:Method: GET 2024-04-12T03:28:45.583 DEBUG:boto:Path: /dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.583 DEBUG:boto:Data: 2024-04-12T03:28:45.583 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.583 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.583 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.583 DEBUG:boto:Params: {} 2024-04-12T03:28:45.583 DEBUG:boto:Token: None 2024-04-12T03:28:45.583 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.584 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TcaMBLIj/vYtay0JMnEvc5w89qE= 2024-04-12T03:28:45.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TcaMBLIj/vYtay0JMnEvc5w89qE=', 'Content-Length': '0'} 2024-04-12T03:28:45.586 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', '6gubSYrBrkePUl8NKnDsGwX3jArFN7d'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cee2a1d8c658386d-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.586 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.586 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.586 DEBUG:boto:path=/dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.586 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.586 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.587 DEBUG:boto:Path: /dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.587 DEBUG:boto:Data: 2024-04-12T03:28:45.587 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.587 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.587 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.587 DEBUG:boto:Params: {} 2024-04-12T03:28:45.587 DEBUG:boto:Token: None 2024-04-12T03:28:45.587 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.587 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tv3mIS+We9xCoz9NUeswnVS0zJI= 2024-04-12T03:28:45.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tv3mIS+We9xCoz9NUeswnVS0zJI=', 'Content-Length': '0'} 2024-04-12T03:28:45.589 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', '6gubSYrBrkePUl8NKnDsGwX3jArFN7d'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d101b03047a92665-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.590 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.590 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.590 DEBUG:boto:path=/dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.590 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.590 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.590 DEBUG:boto:Path: /dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.590 DEBUG:boto:Data: 2024-04-12T03:28:45.590 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.590 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.590 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.591 DEBUG:boto:Params: {} 2024-04-12T03:28:45.591 DEBUG:boto:Token: None 2024-04-12T03:28:45.591 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.591 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tv3mIS+We9xCoz9NUeswnVS0zJI= 2024-04-12T03:28:45.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tv3mIS+We9xCoz9NUeswnVS0zJI=', 'Content-Length': '0'} 2024-04-12T03:28:45.593 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', '6gubSYrBrkePUl8NKnDsGwX3jArFN7d'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069cba26248f6cb16-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.593 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.593 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.593 DEBUG:boto:path=/dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.593 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.593 DEBUG:boto:Method: GET 2024-04-12T03:28:45.593 DEBUG:boto:Path: /dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.594 DEBUG:boto:Data: 2024-04-12T03:28:45.594 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.594 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.594 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.594 DEBUG:boto:Params: {} 2024-04-12T03:28:45.594 DEBUG:boto:Token: None 2024-04-12T03:28:45.594 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.594 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TcaMBLIj/vYtay0JMnEvc5w89qE= 2024-04-12T03:28:45.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TcaMBLIj/vYtay0JMnEvc5w89qE=', 'Content-Length': '0'} 2024-04-12T03:28:45.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', '6gubSYrBrkePUl8NKnDsGwX3jArFN7d'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e180195a3c041b74-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.596 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.596 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.596 DEBUG:boto:path=/dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.597 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.597 DEBUG:boto:Method: GET 2024-04-12T03:28:45.597 DEBUG:boto:Path: /dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.597 DEBUG:boto:Data: 2024-04-12T03:28:45.597 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.597 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.597 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.597 DEBUG:boto:Params: {} 2024-04-12T03:28:45.597 DEBUG:boto:Token: None 2024-04-12T03:28:45.597 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=6gubSYrBrkePUl8NKnDsGwX3jArFN7d 2024-04-12T03:28:45.598 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TcaMBLIj/vYtay0JMnEvc5w89qE= 2024-04-12T03:28:45.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TcaMBLIj/vYtay0JMnEvc5w89qE=', 'Content-Length': '0'} 2024-04-12T03:28:45.599 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', '6gubSYrBrkePUl8NKnDsGwX3jArFN7d'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a91d5c2ce8aa66aa-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.600 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.600 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.600 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.600 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.600 DEBUG:boto:Method: GET 2024-04-12T03:28:45.600 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.600 DEBUG:boto:Data: 2024-04-12T03:28:45.600 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.600 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.600 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.600 DEBUG:boto:Params: {} 2024-04-12T03:28:45.600 DEBUG:boto:Token: None 2024-04-12T03:28:45.601 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.601 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000202f80051c3e34c1-006618aa6d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.603 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.604 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.604 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.604 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.604 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.604 DEBUG:boto:Method: GET 2024-04-12T03:28:45.604 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.604 DEBUG:boto:Data: 2024-04-12T03:28:45.604 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.604 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.605 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.605 DEBUG:boto:Params: {} 2024-04-12T03:28:45.605 DEBUG:boto:Token: None 2024-04-12T03:28:45.605 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.605 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da606bb4f70031ce-006618aa6d-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.608 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.608 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.608 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.608 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.608 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:45.608 DEBUG:boto:Data: 2024-04-12T03:28:45.608 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.608 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.609 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.609 DEBUG:boto:Params: {} 2024-04-12T03:28:45.609 DEBUG:boto:Token: None 2024-04-12T03:28:45.609 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj 2024-04-12T03:28:45.609 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk= 2024-04-12T03:28:45.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk=', 'Content-Length': '0'} 2024-04-12T03:28:45.612 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000489f65a00f159156-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.612 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.612 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.612 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.612 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:45.613 DEBUG:boto:Data: 2024-04-12T03:28:45.613 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.613 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.613 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.613 DEBUG:boto:Params: {} 2024-04-12T03:28:45.613 DEBUG:boto:Token: None 2024-04-12T03:28:45.613 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj 2024-04-12T03:28:45.613 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk= 2024-04-12T03:28:45.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk=', 'Content-Length': '0'} 2024-04-12T03:28:45.616 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000bb5ccc0ea544073-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.617 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.617 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.617 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.617 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.617 DEBUG:boto:Method: GET 2024-04-12T03:28:45.617 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.617 DEBUG:boto:Data: 2024-04-12T03:28:45.617 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.617 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.618 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.618 DEBUG:boto:Params: {} 2024-04-12T03:28:45.618 DEBUG:boto:Token: None 2024-04-12T03:28:45.618 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.618 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ= 2024-04-12T03:28:45.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.620 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000051e5e60b512dbc6d-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.662 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.662 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.663 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.663 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.663 DEBUG:boto:Method: GET 2024-04-12T03:28:45.663 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.663 DEBUG:boto:Data: 2024-04-12T03:28:45.663 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.663 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.663 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.663 DEBUG:boto:Params: {} 2024-04-12T03:28:45.663 DEBUG:boto:Token: None 2024-04-12T03:28:45.663 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.664 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ= 2024-04-12T03:28:45.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.666 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000966f77e106932fe3-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.666 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.666 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.666 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.666 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.666 DEBUG:boto:Method: GET 2024-04-12T03:28:45.666 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.667 DEBUG:boto:Data: 2024-04-12T03:28:45.667 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.667 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.667 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.667 DEBUG:boto:Params: {} 2024-04-12T03:28:45.667 DEBUG:boto:Token: None 2024-04-12T03:28:45.667 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.667 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000654ba4d07d3605ea-006618aa6d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.670 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.670 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.670 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.670 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.671 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.671 DEBUG:boto:Method: GET 2024-04-12T03:28:45.671 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.671 DEBUG:boto:Data: 2024-04-12T03:28:45.671 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.671 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.671 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.671 DEBUG:boto:Params: {} 2024-04-12T03:28:45.671 DEBUG:boto:Token: None 2024-04-12T03:28:45.671 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.672 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9507a4b5fdb5b4f-006618aa6d-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.674 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.675 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.675 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.675 DEBUG:boto:path=/dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.675 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.675 DEBUG:boto:Method: GET 2024-04-12T03:28:45.675 DEBUG:boto:Path: /dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.675 DEBUG:boto:Data: 2024-04-12T03:28:45.675 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.675 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.675 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.675 DEBUG:boto:Params: {} 2024-04-12T03:28:45.676 DEBUG:boto:Token: None 2024-04-12T03:28:45.676 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.676 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yYa3A8+AsGwpvTPE2kFrvPDllUM= 2024-04-12T03:28:45.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yYa3A8+AsGwpvTPE2kFrvPDllUM=', 'Content-Length': '0'} 2024-04-12T03:28:45.678 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'cXajPIrN3ztkkbq6VakQTR3vdZhlxgl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fdee84b3f4de4ea2-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.678 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.678 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.678 DEBUG:boto:path=/dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.678 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.678 DEBUG:boto:Method: GET 2024-04-12T03:28:45.678 DEBUG:boto:Path: /dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.679 DEBUG:boto:Data: 2024-04-12T03:28:45.679 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.679 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.679 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.679 DEBUG:boto:Params: {} 2024-04-12T03:28:45.679 DEBUG:boto:Token: None 2024-04-12T03:28:45.679 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.679 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yYa3A8+AsGwpvTPE2kFrvPDllUM= 2024-04-12T03:28:45.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yYa3A8+AsGwpvTPE2kFrvPDllUM=', 'Content-Length': '0'} 2024-04-12T03:28:45.681 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'cXajPIrN3ztkkbq6VakQTR3vdZhlxgl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007750ebaae5fe630f-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.681 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.682 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.682 DEBUG:boto:path=/dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.682 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.682 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.682 DEBUG:boto:Path: /dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.682 DEBUG:boto:Data: 2024-04-12T03:28:45.682 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.682 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.682 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.682 DEBUG:boto:Params: {} 2024-04-12T03:28:45.683 DEBUG:boto:Token: None 2024-04-12T03:28:45.683 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.683 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1jfLibtjd4h0MloL/+4PtmxaLkk= 2024-04-12T03:28:45.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1jfLibtjd4h0MloL/+4PtmxaLkk=', 'Content-Length': '0'} 2024-04-12T03:28:45.684 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'cXajPIrN3ztkkbq6VakQTR3vdZhlxgl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d31fc69c49d4e7c9-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.685 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.685 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.685 DEBUG:boto:path=/dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.685 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.685 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.685 DEBUG:boto:Path: /dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.685 DEBUG:boto:Data: 2024-04-12T03:28:45.685 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.685 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.685 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.686 DEBUG:boto:Params: {} 2024-04-12T03:28:45.686 DEBUG:boto:Token: None 2024-04-12T03:28:45.686 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.686 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1jfLibtjd4h0MloL/+4PtmxaLkk= 2024-04-12T03:28:45.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1jfLibtjd4h0MloL/+4PtmxaLkk=', 'Content-Length': '0'} 2024-04-12T03:28:45.688 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'cXajPIrN3ztkkbq6VakQTR3vdZhlxgl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048d9fc089afc8547-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.688 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.688 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.688 DEBUG:boto:path=/dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.688 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.689 DEBUG:boto:Method: GET 2024-04-12T03:28:45.689 DEBUG:boto:Path: /dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.689 DEBUG:boto:Data: 2024-04-12T03:28:45.689 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.689 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.689 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.689 DEBUG:boto:Params: {} 2024-04-12T03:28:45.689 DEBUG:boto:Token: None 2024-04-12T03:28:45.689 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.689 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yYa3A8+AsGwpvTPE2kFrvPDllUM= 2024-04-12T03:28:45.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yYa3A8+AsGwpvTPE2kFrvPDllUM=', 'Content-Length': '0'} 2024-04-12T03:28:45.691 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'cXajPIrN3ztkkbq6VakQTR3vdZhlxgl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056a1ba8ccd377337-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.734 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.734 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.734 DEBUG:boto:path=/dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.735 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.735 DEBUG:boto:Method: GET 2024-04-12T03:28:45.735 DEBUG:boto:Path: /dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.735 DEBUG:boto:Data: 2024-04-12T03:28:45.735 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.735 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.735 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.735 DEBUG:boto:Params: {} 2024-04-12T03:28:45.735 DEBUG:boto:Token: None 2024-04-12T03:28:45.735 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=cXajPIrN3ztkkbq6VakQTR3vdZhlxgl 2024-04-12T03:28:45.736 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yYa3A8+AsGwpvTPE2kFrvPDllUM= 2024-04-12T03:28:45.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yYa3A8+AsGwpvTPE2kFrvPDllUM=', 'Content-Length': '0'} 2024-04-12T03:28:45.737 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'cXajPIrN3ztkkbq6VakQTR3vdZhlxgl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000069ac6a3b46349ab0-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.738 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.738 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.738 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.738 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.738 DEBUG:boto:Method: GET 2024-04-12T03:28:45.738 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.738 DEBUG:boto:Data: 2024-04-12T03:28:45.738 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.738 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.738 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.738 DEBUG:boto:Params: {} 2024-04-12T03:28:45.739 DEBUG:boto:Token: None 2024-04-12T03:28:45.739 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.739 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076e6ac7ae8f28ec2-006618aa6d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.741 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.742 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.742 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.742 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.742 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.742 DEBUG:boto:Method: GET 2024-04-12T03:28:45.742 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.742 DEBUG:boto:Data: 2024-04-12T03:28:45.742 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.742 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.742 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.743 DEBUG:boto:Params: {} 2024-04-12T03:28:45.743 DEBUG:boto:Token: None 2024-04-12T03:28:45.743 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.743 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000073e75fb8e6e1647-006618aa6d-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.746 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.746 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.746 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.746 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.746 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:45.747 DEBUG:boto:Data: 2024-04-12T03:28:45.747 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.747 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.747 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.747 DEBUG:boto:Params: {} 2024-04-12T03:28:45.747 DEBUG:boto:Token: None 2024-04-12T03:28:45.747 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj 2024-04-12T03:28:45.747 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk= 2024-04-12T03:28:45.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk=', 'Content-Length': '0'} 2024-04-12T03:28:45.750 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab212cf9719d6958-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.751 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.751 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.751 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.751 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:45.751 DEBUG:boto:Data: 2024-04-12T03:28:45.751 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.751 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.751 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.751 DEBUG:boto:Params: {} 2024-04-12T03:28:45.752 DEBUG:boto:Token: None 2024-04-12T03:28:45.752 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj 2024-04-12T03:28:45.752 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk= 2024-04-12T03:28:45.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk=', 'Content-Length': '0'} 2024-04-12T03:28:45.755 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b722ba25354dc861-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.755 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.755 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.755 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.755 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.755 DEBUG:boto:Method: GET 2024-04-12T03:28:45.755 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.756 DEBUG:boto:Data: 2024-04-12T03:28:45.756 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.756 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.756 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.756 DEBUG:boto:Params: {} 2024-04-12T03:28:45.756 DEBUG:boto:Token: None 2024-04-12T03:28:45.756 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.756 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ= 2024-04-12T03:28:45.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.758 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a99ff059e053dad-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.798 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.798 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.799 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.799 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.799 DEBUG:boto:Method: GET 2024-04-12T03:28:45.799 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.799 DEBUG:boto:Data: 2024-04-12T03:28:45.799 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.799 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.799 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.799 DEBUG:boto:Params: {} 2024-04-12T03:28:45.799 DEBUG:boto:Token: None 2024-04-12T03:28:45.800 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.800 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ= 2024-04-12T03:28:45.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.801 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a37e08b8c10a424-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.802 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.802 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.802 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.802 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.802 DEBUG:boto:Method: GET 2024-04-12T03:28:45.802 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.803 DEBUG:boto:Data: 2024-04-12T03:28:45.803 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.803 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.803 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.803 DEBUG:boto:Params: {} 2024-04-12T03:28:45.803 DEBUG:boto:Token: None 2024-04-12T03:28:45.803 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.803 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d9df4fd240852df-006618aa6d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.806 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.807 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.807 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.807 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.807 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.807 DEBUG:boto:Method: GET 2024-04-12T03:28:45.807 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.807 DEBUG:boto:Data: 2024-04-12T03:28:45.807 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.807 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.808 DEBUG:boto:Params: {} 2024-04-12T03:28:45.808 DEBUG:boto:Token: None 2024-04-12T03:28:45.808 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.808 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db258f3811287f76-006618aa6d-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.811 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.811 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.811 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.811 DEBUG:boto:path=/dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.811 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.811 DEBUG:boto:Method: GET 2024-04-12T03:28:45.811 DEBUG:boto:Path: /dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.812 DEBUG:boto:Data: 2024-04-12T03:28:45.812 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.812 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.812 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.812 DEBUG:boto:Params: {} 2024-04-12T03:28:45.812 DEBUG:boto:Token: None 2024-04-12T03:28:45.812 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.812 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+Lrw/241v1ohcqnx3Ufl4iHcFto= 2024-04-12T03:28:45.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+Lrw/241v1ohcqnx3Ufl4iHcFto=', 'Content-Length': '0'} 2024-04-12T03:28:45.814 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000057ea9d67f85f18d6-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.814 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.814 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.815 DEBUG:boto:path=/dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.815 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.815 DEBUG:boto:Method: GET 2024-04-12T03:28:45.815 DEBUG:boto:Path: /dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.815 DEBUG:boto:Data: 2024-04-12T03:28:45.815 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.815 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.815 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.815 DEBUG:boto:Params: {} 2024-04-12T03:28:45.815 DEBUG:boto:Token: None 2024-04-12T03:28:45.815 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.816 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+Lrw/241v1ohcqnx3Ufl4iHcFto= 2024-04-12T03:28:45.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+Lrw/241v1ohcqnx3Ufl4iHcFto=', 'Content-Length': '0'} 2024-04-12T03:28:45.817 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c97dc6ea1b289cce-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.818 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.818 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.818 DEBUG:boto:path=/dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.818 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.818 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.818 DEBUG:boto:Path: /dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.818 DEBUG:boto:Data: 2024-04-12T03:28:45.818 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.819 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.819 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.819 DEBUG:boto:Params: {} 2024-04-12T03:28:45.819 DEBUG:boto:Token: None 2024-04-12T03:28:45.819 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.819 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UQt6WFlCqnEa/yXCDfBfj6/ail4= 2024-04-12T03:28:45.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UQt6WFlCqnEa/yXCDfBfj6/ail4=', 'Content-Length': '0'} 2024-04-12T03:28:45.821 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2c105cd71c69a72-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.821 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.821 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.822 DEBUG:boto:path=/dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.822 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.822 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.822 DEBUG:boto:Path: /dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.822 DEBUG:boto:Data: 2024-04-12T03:28:45.822 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.822 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.822 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.822 DEBUG:boto:Params: {} 2024-04-12T03:28:45.822 DEBUG:boto:Token: None 2024-04-12T03:28:45.823 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.823 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UQt6WFlCqnEa/yXCDfBfj6/ail4= 2024-04-12T03:28:45.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UQt6WFlCqnEa/yXCDfBfj6/ail4=', 'Content-Length': '0'} 2024-04-12T03:28:45.824 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084af75ac23f15b5c-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.825 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.825 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.825 DEBUG:boto:path=/dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.825 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.825 DEBUG:boto:Method: GET 2024-04-12T03:28:45.825 DEBUG:boto:Path: /dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.825 DEBUG:boto:Data: 2024-04-12T03:28:45.825 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.825 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.825 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.826 DEBUG:boto:Params: {} 2024-04-12T03:28:45.826 DEBUG:boto:Token: None 2024-04-12T03:28:45.826 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.826 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+Lrw/241v1ohcqnx3Ufl4iHcFto= 2024-04-12T03:28:45.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+Lrw/241v1ohcqnx3Ufl4iHcFto=', 'Content-Length': '0'} 2024-04-12T03:28:45.828 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f09973d412a4c035-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.870 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.871 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.871 DEBUG:boto:path=/dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.871 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.871 DEBUG:boto:Method: GET 2024-04-12T03:28:45.871 DEBUG:boto:Path: /dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.871 DEBUG:boto:Data: 2024-04-12T03:28:45.871 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.871 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.871 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.871 DEBUG:boto:Params: {} 2024-04-12T03:28:45.872 DEBUG:boto:Token: None 2024-04-12T03:28:45.872 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv 2024-04-12T03:28:45.872 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+Lrw/241v1ohcqnx3Ufl4iHcFto= 2024-04-12T03:28:45.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+Lrw/241v1ohcqnx3Ufl4iHcFto=', 'Content-Length': '0'} 2024-04-12T03:28:45.874 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6DGm-fKiIUWNNAVI-ZpNeeIaHRqc7Sv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003571b7e123f80427-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.874 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.874 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.875 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.875 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.875 DEBUG:boto:Method: GET 2024-04-12T03:28:45.875 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.875 DEBUG:boto:Data: 2024-04-12T03:28:45.875 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.875 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.875 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.875 DEBUG:boto:Params: {} 2024-04-12T03:28:45.875 DEBUG:boto:Token: None 2024-04-12T03:28:45.876 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.876 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032646985f306c9b8-006618aa6d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.918 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.919 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.919 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.919 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.919 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.919 DEBUG:boto:Method: GET 2024-04-12T03:28:45.919 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.919 DEBUG:boto:Data: 2024-04-12T03:28:45.919 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.920 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.920 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.920 DEBUG:boto:Params: {} 2024-04-12T03:28:45.920 DEBUG:boto:Token: None 2024-04-12T03:28:45.920 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.920 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044af562013c2924c-006618aa6d-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.923 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.923 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.923 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.924 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.924 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:45.924 DEBUG:boto:Data: 2024-04-12T03:28:45.924 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.924 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.924 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.924 DEBUG:boto:Params: {} 2024-04-12T03:28:45.924 DEBUG:boto:Token: None 2024-04-12T03:28:45.924 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj 2024-04-12T03:28:45.924 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk= 2024-04-12T03:28:45.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk=', 'Content-Length': '0'} 2024-04-12T03:28:45.927 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000064e28027110eafff-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.928 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.928 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.928 DEBUG:boto:Method: HEAD 2024-04-12T03:28:45.928 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:45.928 DEBUG:boto:Data: 2024-04-12T03:28:45.928 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.928 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.928 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.928 DEBUG:boto:Params: {} 2024-04-12T03:28:45.929 DEBUG:boto:Token: None 2024-04-12T03:28:45.929 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj 2024-04-12T03:28:45.929 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk= 2024-04-12T03:28:45.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h8jR9lJ5lPc0AQUIHmhIUdMfDZk=', 'Content-Length': '0'} 2024-04-12T03:28:45.932 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009fa52d0fabd0a5ec-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.932 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.932 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.933 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.933 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.933 DEBUG:boto:Method: GET 2024-04-12T03:28:45.933 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.933 DEBUG:boto:Data: 2024-04-12T03:28:45.933 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.933 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.933 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.933 DEBUG:boto:Params: {} 2024-04-12T03:28:45.933 DEBUG:boto:Token: None 2024-04-12T03:28:45.933 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.934 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ= 2024-04-12T03:28:45.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000272e4b92840e7b55-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.936 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.936 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.936 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.936 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.936 DEBUG:boto:Method: GET 2024-04-12T03:28:45.936 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.936 DEBUG:boto:Data: 2024-04-12T03:28:45.936 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.936 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.936 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.937 DEBUG:boto:Params: {} 2024-04-12T03:28:45.937 DEBUG:boto:Token: None 2024-04-12T03:28:45.937 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:45.937 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ= 2024-04-12T03:28:45.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xbWgicVXWZfZnSsSYCvkqkBsPqQ=', 'Content-Length': '0'} 2024-04-12T03:28:45.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe9bdbed5a9bf273-006618aa6d-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.982 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.982 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.983 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.983 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.983 DEBUG:boto:Method: GET 2024-04-12T03:28:45.983 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.983 DEBUG:boto:Data: 2024-04-12T03:28:45.983 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.983 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.983 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.983 DEBUG:boto:Params: {} 2024-04-12T03:28:45.983 DEBUG:boto:Token: None 2024-04-12T03:28:45.984 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.984 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023bacd356b021843-006618aa6d-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.986 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.987 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:45.987 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:45.987 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.987 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:45.987 DEBUG:boto:Method: GET 2024-04-12T03:28:45.987 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:45.987 DEBUG:boto:Data: 2024-04-12T03:28:45.987 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.987 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:45.988 DEBUG:boto:Port: 8000 2024-04-12T03:28:45.988 DEBUG:boto:Params: {} 2024-04-12T03:28:45.988 DEBUG:boto:Token: None 2024-04-12T03:28:45.988 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/ 2024-04-12T03:28:45.988 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA= 2024-04-12T03:28:45.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0RB3s2gK19ZzM1F3GLwv+eu4GFA=', 'Content-Length': '0'} 2024-04-12T03:28:45.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b6f43ad7dcc0880-006618aa6d-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:45.991 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:45.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:45.991 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:45.991 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:45.992 DEBUG:boto:path=/dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:45.992 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:45.992 DEBUG:boto:Method: GET 2024-04-12T03:28:45.992 DEBUG:boto:Path: /dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:45.992 DEBUG:boto:Data: 2024-04-12T03:28:45.992 DEBUG:boto:Headers: {} 2024-04-12T03:28:45.992 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:45.992 DEBUG:boto:Port: 8001 2024-04-12T03:28:45.992 DEBUG:boto:Params: {} 2024-04-12T03:28:45.992 DEBUG:boto:Token: None 2024-04-12T03:28:45.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:45 GMT /dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:45.993 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:23cSQq0FR5KYTAfPMD5z+pp5mrA= 2024-04-12T03:28:45.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:23cSQq0FR5KYTAfPMD5z+pp5mrA=', 'Content-Length': '0'} 2024-04-12T03:28:45.994 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ebf6d6b9053e80a6-006618aa6d-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.038 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.039 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.039 DEBUG:boto:path=/dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.039 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.039 DEBUG:boto:Method: GET 2024-04-12T03:28:46.039 DEBUG:boto:Path: /dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.039 DEBUG:boto:Data: 2024-04-12T03:28:46.039 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.039 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.039 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.039 DEBUG:boto:Params: {} 2024-04-12T03:28:46.039 DEBUG:boto:Token: None 2024-04-12T03:28:46.040 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.040 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z8rgHsASqdci63kGCismOpYqYQw= 2024-04-12T03:28:46.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z8rgHsASqdci63kGCismOpYqYQw=', 'Content-Length': '0'} 2024-04-12T03:28:46.042 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a8e00edf78636584-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.042 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.043 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.043 DEBUG:boto:path=/dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.043 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.043 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.043 DEBUG:boto:Path: /dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.043 DEBUG:boto:Data: 2024-04-12T03:28:46.043 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.043 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.043 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.043 DEBUG:boto:Params: {} 2024-04-12T03:28:46.043 DEBUG:boto:Token: None 2024-04-12T03:28:46.044 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.044 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JqJvrPQVPIts7+zwmlfe2UoWTGw= 2024-04-12T03:28:46.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JqJvrPQVPIts7+zwmlfe2UoWTGw=', 'Content-Length': '0'} 2024-04-12T03:28:46.045 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2647643799284c7-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.046 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.046 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.046 DEBUG:boto:path=/dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.046 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.046 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.046 DEBUG:boto:Path: /dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.046 DEBUG:boto:Data: 2024-04-12T03:28:46.046 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.046 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.047 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.047 DEBUG:boto:Params: {} 2024-04-12T03:28:46.047 DEBUG:boto:Token: None 2024-04-12T03:28:46.047 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.047 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JqJvrPQVPIts7+zwmlfe2UoWTGw= 2024-04-12T03:28:46.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JqJvrPQVPIts7+zwmlfe2UoWTGw=', 'Content-Length': '0'} 2024-04-12T03:28:46.049 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c35f174a226b759-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:46.049 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.049 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.049 DEBUG:boto:path=/dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.049 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.049 DEBUG:boto:Method: GET 2024-04-12T03:28:46.050 DEBUG:boto:Path: /dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.050 DEBUG:boto:Data: 2024-04-12T03:28:46.050 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.050 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.050 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.050 DEBUG:boto:Params: {} 2024-04-12T03:28:46.050 DEBUG:boto:Token: None 2024-04-12T03:28:46.050 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.050 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z8rgHsASqdci63kGCismOpYqYQw= 2024-04-12T03:28:46.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z8rgHsASqdci63kGCismOpYqYQw=', 'Content-Length': '0'} 2024-04-12T03:28:46.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007707c747633af381-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.053 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.053 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.053 DEBUG:boto:path=/dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.053 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.053 DEBUG:boto:Method: GET 2024-04-12T03:28:46.053 DEBUG:boto:Path: /dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.053 DEBUG:boto:Data: 2024-04-12T03:28:46.053 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.054 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.054 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.054 DEBUG:boto:Params: {} 2024-04-12T03:28:46.054 DEBUG:boto:Token: None 2024-04-12T03:28:46.054 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj 2024-04-12T03:28:46.054 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z8rgHsASqdci63kGCismOpYqYQw= 2024-04-12T03:28:46.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z8rgHsASqdci63kGCismOpYqYQw=', 'Content-Length': '0'} 2024-04-12T03:28:46.056 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'W7uVB2AgTJyCAtTi2lB4dP4PR6fEVsj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cc8733854e75975f-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.056 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.056 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.057 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.057 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.057 DEBUG:boto:Method: GET 2024-04-12T03:28:46.057 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.057 DEBUG:boto:Data: 2024-04-12T03:28:46.057 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.057 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.057 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.057 DEBUG:boto:Params: {} 2024-04-12T03:28:46.057 DEBUG:boto:Token: None 2024-04-12T03:28:46.058 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.058 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050dde107ea4b7ddb-006618aa6e-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.060 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.060 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.061 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.061 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.061 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.061 DEBUG:boto:Method: GET 2024-04-12T03:28:46.061 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.061 DEBUG:boto:Data: 2024-04-12T03:28:46.061 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.061 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.061 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.061 DEBUG:boto:Params: {} 2024-04-12T03:28:46.061 DEBUG:boto:Token: None 2024-04-12T03:28:46.061 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.062 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1124fb9ee1ab11e-006618aa6e-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.064 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.065 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.065 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.065 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.065 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:46.065 DEBUG:boto:Data: 2024-04-12T03:28:46.065 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.065 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.065 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.065 DEBUG:boto:Params: {} 2024-04-12T03:28:46.065 DEBUG:boto:Token: None 2024-04-12T03:28:46.065 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj 2024-04-12T03:28:46.066 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU= 2024-04-12T03:28:46.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU=', 'Content-Length': '0'} 2024-04-12T03:28:46.068 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000018474cb300dc1f4b-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.069 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.069 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.069 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.069 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:46.069 DEBUG:boto:Data: 2024-04-12T03:28:46.069 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.069 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.069 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.069 DEBUG:boto:Params: {} 2024-04-12T03:28:46.069 DEBUG:boto:Token: None 2024-04-12T03:28:46.070 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj 2024-04-12T03:28:46.070 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU= 2024-04-12T03:28:46.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU=', 'Content-Length': '0'} 2024-04-12T03:28:46.073 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d08f375d92e1e51-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:46.073 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.073 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.073 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.073 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.073 DEBUG:boto:Method: GET 2024-04-12T03:28:46.074 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.074 DEBUG:boto:Data: 2024-04-12T03:28:46.074 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.074 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.074 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.074 DEBUG:boto:Params: {} 2024-04-12T03:28:46.074 DEBUG:boto:Token: None 2024-04-12T03:28:46.074 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.074 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok= 2024-04-12T03:28:46.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok=', 'Content-Length': '0'} 2024-04-12T03:28:46.076 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000909cf1b69d10a221-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.076 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.076 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.076 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.076 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.077 DEBUG:boto:Method: GET 2024-04-12T03:28:46.077 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.077 DEBUG:boto:Data: 2024-04-12T03:28:46.077 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.077 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.077 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.077 DEBUG:boto:Params: {} 2024-04-12T03:28:46.077 DEBUG:boto:Token: None 2024-04-12T03:28:46.077 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.077 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok= 2024-04-12T03:28:46.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok=', 'Content-Length': '0'} 2024-04-12T03:28:46.079 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000130f42fa51afe11c-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.080 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.080 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.080 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.080 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.080 DEBUG:boto:Method: GET 2024-04-12T03:28:46.080 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.080 DEBUG:boto:Data: 2024-04-12T03:28:46.080 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.080 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.080 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.080 DEBUG:boto:Params: {} 2024-04-12T03:28:46.081 DEBUG:boto:Token: None 2024-04-12T03:28:46.081 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.081 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.083 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eccfda8750ec46b6-006618aa6e-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.083 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.084 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.084 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.084 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.084 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.084 DEBUG:boto:Method: GET 2024-04-12T03:28:46.084 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.084 DEBUG:boto:Data: 2024-04-12T03:28:46.084 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.084 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.084 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.085 DEBUG:boto:Params: {} 2024-04-12T03:28:46.085 DEBUG:boto:Token: None 2024-04-12T03:28:46.085 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.085 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3751dbcce161b7e-006618aa6e-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.130 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:46.131 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.131 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.131 DEBUG:boto:path=/dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.131 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.131 DEBUG:boto:Method: GET 2024-04-12T03:28:46.131 DEBUG:boto:Path: /dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.131 DEBUG:boto:Data: 2024-04-12T03:28:46.131 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.131 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.131 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.132 DEBUG:boto:Params: {} 2024-04-12T03:28:46.132 DEBUG:boto:Token: None 2024-04-12T03:28:46.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.132 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2WAo8fbx2obpEVMPBrI+7ZOT/ZE= 2024-04-12T03:28:46.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2WAo8fbx2obpEVMPBrI+7ZOT/ZE=', 'Content-Length': '0'} 2024-04-12T03:28:46.134 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d76da366e0525e3c-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.174 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.174 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.174 DEBUG:boto:path=/dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.175 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.175 DEBUG:boto:Method: GET 2024-04-12T03:28:46.175 DEBUG:boto:Path: /dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.175 DEBUG:boto:Data: 2024-04-12T03:28:46.175 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.175 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.175 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.175 DEBUG:boto:Params: {} 2024-04-12T03:28:46.175 DEBUG:boto:Token: None 2024-04-12T03:28:46.175 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.175 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2WAo8fbx2obpEVMPBrI+7ZOT/ZE= 2024-04-12T03:28:46.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2WAo8fbx2obpEVMPBrI+7ZOT/ZE=', 'Content-Length': '0'} 2024-04-12T03:28:46.177 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003682b7f1a5ea7b84-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.178 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.178 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.178 DEBUG:boto:path=/dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.178 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.178 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.178 DEBUG:boto:Path: /dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.178 DEBUG:boto:Data: 2024-04-12T03:28:46.178 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.178 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.178 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.178 DEBUG:boto:Params: {} 2024-04-12T03:28:46.179 DEBUG:boto:Token: None 2024-04-12T03:28:46.179 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.179 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tI4fugkIBbmaRo1vhS8nuj0YAQA= 2024-04-12T03:28:46.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tI4fugkIBbmaRo1vhS8nuj0YAQA=', 'Content-Length': '0'} 2024-04-12T03:28:46.181 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ff74646a672c317-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.181 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.181 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.181 DEBUG:boto:path=/dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.181 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.181 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.181 DEBUG:boto:Path: /dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.181 DEBUG:boto:Data: 2024-04-12T03:28:46.181 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.181 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.182 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.182 DEBUG:boto:Params: {} 2024-04-12T03:28:46.182 DEBUG:boto:Token: None 2024-04-12T03:28:46.182 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.182 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tI4fugkIBbmaRo1vhS8nuj0YAQA= 2024-04-12T03:28:46.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tI4fugkIBbmaRo1vhS8nuj0YAQA=', 'Content-Length': '0'} 2024-04-12T03:28:46.184 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000000ef2f10aa394ee0-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:46.184 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.184 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.184 DEBUG:boto:path=/dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.184 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.184 DEBUG:boto:Method: GET 2024-04-12T03:28:46.184 DEBUG:boto:Path: /dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.184 DEBUG:boto:Data: 2024-04-12T03:28:46.185 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.185 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.185 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.185 DEBUG:boto:Params: {} 2024-04-12T03:28:46.185 DEBUG:boto:Token: None 2024-04-12T03:28:46.185 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.185 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2WAo8fbx2obpEVMPBrI+7ZOT/ZE= 2024-04-12T03:28:46.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2WAo8fbx2obpEVMPBrI+7ZOT/ZE=', 'Content-Length': '0'} 2024-04-12T03:28:46.187 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003c1180f7520eaef4-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.187 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.187 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.187 DEBUG:boto:path=/dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.188 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.188 DEBUG:boto:Method: GET 2024-04-12T03:28:46.188 DEBUG:boto:Path: /dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.188 DEBUG:boto:Data: 2024-04-12T03:28:46.188 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.188 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.188 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.188 DEBUG:boto:Params: {} 2024-04-12T03:28:46.188 DEBUG:boto:Token: None 2024-04-12T03:28:46.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl 2024-04-12T03:28:46.188 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2WAo8fbx2obpEVMPBrI+7ZOT/ZE= 2024-04-12T03:28:46.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2WAo8fbx2obpEVMPBrI+7ZOT/ZE=', 'Content-Length': '0'} 2024-04-12T03:28:46.190 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6MdpsOyXVm-vH5bb6NFC2s50lRZo.Vl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a73db9446da85409-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.234 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.234 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.234 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.235 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.235 DEBUG:boto:Method: GET 2024-04-12T03:28:46.235 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.235 DEBUG:boto:Data: 2024-04-12T03:28:46.235 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.235 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.235 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.235 DEBUG:boto:Params: {} 2024-04-12T03:28:46.235 DEBUG:boto:Token: None 2024-04-12T03:28:46.235 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.235 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000462843f2e26bd5b-006618aa6e-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.238 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.239 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.239 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.239 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.239 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.239 DEBUG:boto:Method: GET 2024-04-12T03:28:46.239 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.239 DEBUG:boto:Data: 2024-04-12T03:28:46.239 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.239 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.239 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.239 DEBUG:boto:Params: {} 2024-04-12T03:28:46.240 DEBUG:boto:Token: None 2024-04-12T03:28:46.240 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.240 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dea0fa6abab098c7-006618aa6e-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.242 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.243 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.243 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.243 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.243 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:46.243 DEBUG:boto:Data: 2024-04-12T03:28:46.243 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.243 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.243 DEBUG:boto:Params: {} 2024-04-12T03:28:46.244 DEBUG:boto:Token: None 2024-04-12T03:28:46.244 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj 2024-04-12T03:28:46.244 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU= 2024-04-12T03:28:46.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU=', 'Content-Length': '0'} 2024-04-12T03:28:46.247 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032d610483c3685e6-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.247 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.247 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.247 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.248 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:46.248 DEBUG:boto:Data: 2024-04-12T03:28:46.248 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.248 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.248 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.248 DEBUG:boto:Params: {} 2024-04-12T03:28:46.248 DEBUG:boto:Token: None 2024-04-12T03:28:46.248 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj 2024-04-12T03:28:46.248 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU= 2024-04-12T03:28:46.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU=', 'Content-Length': '0'} 2024-04-12T03:28:46.252 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032a13b3fa3ed5615-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:46.252 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.252 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.252 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.252 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.252 DEBUG:boto:Method: GET 2024-04-12T03:28:46.252 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.252 DEBUG:boto:Data: 2024-04-12T03:28:46.252 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.253 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.253 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.253 DEBUG:boto:Params: {} 2024-04-12T03:28:46.253 DEBUG:boto:Token: None 2024-04-12T03:28:46.253 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.253 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok= 2024-04-12T03:28:46.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok=', 'Content-Length': '0'} 2024-04-12T03:28:46.255 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b1038b997b78a9a1-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.298 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.299 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.299 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.299 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.299 DEBUG:boto:Method: GET 2024-04-12T03:28:46.299 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.299 DEBUG:boto:Data: 2024-04-12T03:28:46.299 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.299 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.299 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.299 DEBUG:boto:Params: {} 2024-04-12T03:28:46.299 DEBUG:boto:Token: None 2024-04-12T03:28:46.300 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.300 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok= 2024-04-12T03:28:46.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok=', 'Content-Length': '0'} 2024-04-12T03:28:46.302 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e454b7c45a8a26ff-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.342 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.342 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.343 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.343 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.343 DEBUG:boto:Method: GET 2024-04-12T03:28:46.343 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.343 DEBUG:boto:Data: 2024-04-12T03:28:46.343 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.343 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.343 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.343 DEBUG:boto:Params: {} 2024-04-12T03:28:46.343 DEBUG:boto:Token: None 2024-04-12T03:28:46.344 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.344 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5bbfe271b692a0e-006618aa6e-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.347 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.347 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.348 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.348 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.348 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.348 DEBUG:boto:Method: GET 2024-04-12T03:28:46.348 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.348 DEBUG:boto:Data: 2024-04-12T03:28:46.348 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.348 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.348 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.348 DEBUG:boto:Params: {} 2024-04-12T03:28:46.349 DEBUG:boto:Token: None 2024-04-12T03:28:46.349 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.349 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be9d655e24617987-006618aa6e-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.394 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:46.395 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.395 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.395 DEBUG:boto:path=/dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.395 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.396 DEBUG:boto:Method: GET 2024-04-12T03:28:46.396 DEBUG:boto:Path: /dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.396 DEBUG:boto:Data: 2024-04-12T03:28:46.396 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.396 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.396 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.396 DEBUG:boto:Params: {} 2024-04-12T03:28:46.396 DEBUG:boto:Token: None 2024-04-12T03:28:46.397 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.397 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dK4LvMf4OiB22apCGMjn9KByc1M= 2024-04-12T03:28:46.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dK4LvMf4OiB22apCGMjn9KByc1M=', 'Content-Length': '0'} 2024-04-12T03:28:46.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005e23d05a89fa3277-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.400 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.400 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.400 DEBUG:boto:path=/dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.400 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.400 DEBUG:boto:Method: GET 2024-04-12T03:28:46.400 DEBUG:boto:Path: /dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.400 DEBUG:boto:Data: 2024-04-12T03:28:46.400 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.401 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.401 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.401 DEBUG:boto:Params: {} 2024-04-12T03:28:46.401 DEBUG:boto:Token: None 2024-04-12T03:28:46.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.401 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dK4LvMf4OiB22apCGMjn9KByc1M= 2024-04-12T03:28:46.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dK4LvMf4OiB22apCGMjn9KByc1M=', 'Content-Length': '0'} 2024-04-12T03:28:46.403 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f9350d19af0de946-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.403 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.403 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.403 DEBUG:boto:path=/dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.404 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.404 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.404 DEBUG:boto:Path: /dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.404 DEBUG:boto:Data: 2024-04-12T03:28:46.404 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.404 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.404 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.404 DEBUG:boto:Params: {} 2024-04-12T03:28:46.404 DEBUG:boto:Token: None 2024-04-12T03:28:46.404 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.404 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:E7m6E6jsStAiDBR5gIwp92MfdaU= 2024-04-12T03:28:46.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:E7m6E6jsStAiDBR5gIwp92MfdaU=', 'Content-Length': '0'} 2024-04-12T03:28:46.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c08606ae3c2fe55-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.406 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.406 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.406 DEBUG:boto:path=/dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.407 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.407 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.407 DEBUG:boto:Path: /dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.407 DEBUG:boto:Data: 2024-04-12T03:28:46.407 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.407 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.407 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.407 DEBUG:boto:Params: {} 2024-04-12T03:28:46.407 DEBUG:boto:Token: None 2024-04-12T03:28:46.407 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.408 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:E7m6E6jsStAiDBR5gIwp92MfdaU= 2024-04-12T03:28:46.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:E7m6E6jsStAiDBR5gIwp92MfdaU=', 'Content-Length': '0'} 2024-04-12T03:28:46.409 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e58c8f3b47f055a5-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:46.410 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.410 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.410 DEBUG:boto:path=/dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.410 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.410 DEBUG:boto:Method: GET 2024-04-12T03:28:46.410 DEBUG:boto:Path: /dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.410 DEBUG:boto:Data: 2024-04-12T03:28:46.410 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.410 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.410 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.411 DEBUG:boto:Params: {} 2024-04-12T03:28:46.411 DEBUG:boto:Token: None 2024-04-12T03:28:46.411 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.411 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dK4LvMf4OiB22apCGMjn9KByc1M= 2024-04-12T03:28:46.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dK4LvMf4OiB22apCGMjn9KByc1M=', 'Content-Length': '0'} 2024-04-12T03:28:46.412 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005af0c8a4804790ed-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.454 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.454 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.455 DEBUG:boto:path=/dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.455 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.455 DEBUG:boto:Method: GET 2024-04-12T03:28:46.455 DEBUG:boto:Path: /dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.455 DEBUG:boto:Data: 2024-04-12T03:28:46.455 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.455 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.455 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.455 DEBUG:boto:Params: {} 2024-04-12T03:28:46.455 DEBUG:boto:Token: None 2024-04-12T03:28:46.456 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0 2024-04-12T03:28:46.456 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dK4LvMf4OiB22apCGMjn9KByc1M= 2024-04-12T03:28:46.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dK4LvMf4OiB22apCGMjn9KByc1M=', 'Content-Length': '0'} 2024-04-12T03:28:46.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'gXiwRoSqiWpOKPjQCipYwB9V5k1dqF0'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005053e188a4a963f2-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.458 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.458 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.458 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.458 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.458 DEBUG:boto:Method: GET 2024-04-12T03:28:46.459 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.459 DEBUG:boto:Data: 2024-04-12T03:28:46.459 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.459 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.459 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.459 DEBUG:boto:Params: {} 2024-04-12T03:28:46.459 DEBUG:boto:Token: None 2024-04-12T03:28:46.459 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.459 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b73faff88f9133a-006618aa6e-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.502 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.503 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.503 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.503 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.503 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.503 DEBUG:boto:Method: GET 2024-04-12T03:28:46.503 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.503 DEBUG:boto:Data: 2024-04-12T03:28:46.503 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.503 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.504 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.504 DEBUG:boto:Params: {} 2024-04-12T03:28:46.504 DEBUG:boto:Token: None 2024-04-12T03:28:46.504 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.504 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6a44242b965e6d7-006618aa6e-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.507 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.507 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.507 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.507 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.507 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:46.507 DEBUG:boto:Data: 2024-04-12T03:28:46.507 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.508 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.508 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.508 DEBUG:boto:Params: {} 2024-04-12T03:28:46.508 DEBUG:boto:Token: None 2024-04-12T03:28:46.508 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj 2024-04-12T03:28:46.508 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU= 2024-04-12T03:28:46.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU=', 'Content-Length': '0'} 2024-04-12T03:28:46.511 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000073a4860a1e279bc0-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.511 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.511 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.512 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.512 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:46.512 DEBUG:boto:Data: 2024-04-12T03:28:46.512 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.512 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.512 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.512 DEBUG:boto:Params: {} 2024-04-12T03:28:46.512 DEBUG:boto:Token: None 2024-04-12T03:28:46.512 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj 2024-04-12T03:28:46.512 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU= 2024-04-12T03:28:46.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU=', 'Content-Length': '0'} 2024-04-12T03:28:46.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001ab16f3683336648-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:46.516 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.516 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.516 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.516 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.516 DEBUG:boto:Method: GET 2024-04-12T03:28:46.516 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.516 DEBUG:boto:Data: 2024-04-12T03:28:46.516 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.516 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.517 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.517 DEBUG:boto:Params: {} 2024-04-12T03:28:46.517 DEBUG:boto:Token: None 2024-04-12T03:28:46.517 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.517 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok= 2024-04-12T03:28:46.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok=', 'Content-Length': '0'} 2024-04-12T03:28:46.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9f99b085c7ee57f-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.519 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.519 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.519 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.519 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.519 DEBUG:boto:Method: GET 2024-04-12T03:28:46.520 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.520 DEBUG:boto:Data: 2024-04-12T03:28:46.520 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.520 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.520 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.520 DEBUG:boto:Params: {} 2024-04-12T03:28:46.520 DEBUG:boto:Token: None 2024-04-12T03:28:46.520 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.520 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok= 2024-04-12T03:28:46.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok=', 'Content-Length': '0'} 2024-04-12T03:28:46.522 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000890b28d85324fd6-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.562 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.563 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.563 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.563 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.563 DEBUG:boto:Method: GET 2024-04-12T03:28:46.563 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.563 DEBUG:boto:Data: 2024-04-12T03:28:46.563 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.563 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.563 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.563 DEBUG:boto:Params: {} 2024-04-12T03:28:46.564 DEBUG:boto:Token: None 2024-04-12T03:28:46.564 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.564 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c19b943893d518e9-006618aa6e-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.567 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.567 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.567 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.567 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.567 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.567 DEBUG:boto:Method: GET 2024-04-12T03:28:46.568 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.568 DEBUG:boto:Data: 2024-04-12T03:28:46.568 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.568 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.568 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.568 DEBUG:boto:Params: {} 2024-04-12T03:28:46.568 DEBUG:boto:Token: None 2024-04-12T03:28:46.568 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.568 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ade9ca89683ba7d3-006618aa6e-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.571 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:46.571 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.572 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.572 DEBUG:boto:path=/dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.572 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.572 DEBUG:boto:Method: GET 2024-04-12T03:28:46.572 DEBUG:boto:Path: /dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.572 DEBUG:boto:Data: 2024-04-12T03:28:46.572 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.572 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.573 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.573 DEBUG:boto:Params: {} 2024-04-12T03:28:46.573 DEBUG:boto:Token: None 2024-04-12T03:28:46.573 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.573 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sELiVxjdjBZwuHUSNZsyGtChvcM= 2024-04-12T03:28:46.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sELiVxjdjBZwuHUSNZsyGtChvcM=', 'Content-Length': '0'} 2024-04-12T03:28:46.575 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000d52a66a250ca7f4-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.618 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.619 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.619 DEBUG:boto:path=/dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.619 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.619 DEBUG:boto:Method: GET 2024-04-12T03:28:46.619 DEBUG:boto:Path: /dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.619 DEBUG:boto:Data: 2024-04-12T03:28:46.619 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.619 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.619 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.619 DEBUG:boto:Params: {} 2024-04-12T03:28:46.620 DEBUG:boto:Token: None 2024-04-12T03:28:46.620 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.620 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sELiVxjdjBZwuHUSNZsyGtChvcM= 2024-04-12T03:28:46.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sELiVxjdjBZwuHUSNZsyGtChvcM=', 'Content-Length': '0'} 2024-04-12T03:28:46.622 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000297aebf2e4463b16-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.622 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.623 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.623 DEBUG:boto:path=/dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.623 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.623 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.623 DEBUG:boto:Path: /dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.623 DEBUG:boto:Data: 2024-04-12T03:28:46.623 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.623 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.624 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.624 DEBUG:boto:Params: {} 2024-04-12T03:28:46.624 DEBUG:boto:Token: None 2024-04-12T03:28:46.624 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.624 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wzxR5AB4BTgzrHKlN1CT/5rfGxQ= 2024-04-12T03:28:46.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wzxR5AB4BTgzrHKlN1CT/5rfGxQ=', 'Content-Length': '0'} 2024-04-12T03:28:46.626 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000708195cfd52079c5-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.626 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.626 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.627 DEBUG:boto:path=/dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.627 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.627 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.627 DEBUG:boto:Path: /dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.627 DEBUG:boto:Data: 2024-04-12T03:28:46.627 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.627 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.627 DEBUG:boto:Params: {} 2024-04-12T03:28:46.628 DEBUG:boto:Token: None 2024-04-12T03:28:46.628 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wzxR5AB4BTgzrHKlN1CT/5rfGxQ= 2024-04-12T03:28:46.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wzxR5AB4BTgzrHKlN1CT/5rfGxQ=', 'Content-Length': '0'} 2024-04-12T03:28:46.629 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a1850e02c6a40716-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:46.630 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.630 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.630 DEBUG:boto:path=/dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.630 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.630 DEBUG:boto:Method: GET 2024-04-12T03:28:46.630 DEBUG:boto:Path: /dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.631 DEBUG:boto:Data: 2024-04-12T03:28:46.631 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.631 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.631 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.631 DEBUG:boto:Params: {} 2024-04-12T03:28:46.631 DEBUG:boto:Token: None 2024-04-12T03:28:46.631 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.631 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sELiVxjdjBZwuHUSNZsyGtChvcM= 2024-04-12T03:28:46.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sELiVxjdjBZwuHUSNZsyGtChvcM=', 'Content-Length': '0'} 2024-04-12T03:28:46.633 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008659915dab4932e0-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.633 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.634 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.634 DEBUG:boto:path=/dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.634 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.634 DEBUG:boto:Method: GET 2024-04-12T03:28:46.634 DEBUG:boto:Path: /dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.634 DEBUG:boto:Data: 2024-04-12T03:28:46.634 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.634 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.634 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.634 DEBUG:boto:Params: {} 2024-04-12T03:28:46.635 DEBUG:boto:Token: None 2024-04-12T03:28:46.635 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW 2024-04-12T03:28:46.635 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sELiVxjdjBZwuHUSNZsyGtChvcM= 2024-04-12T03:28:46.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sELiVxjdjBZwuHUSNZsyGtChvcM=', 'Content-Length': '0'} 2024-04-12T03:28:46.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'Cvjy9irUJDLukcVm26x3nBQ9pyVqTRW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001057967c0b534641-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.678 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.678 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.679 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.679 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.679 DEBUG:boto:Method: GET 2024-04-12T03:28:46.679 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.679 DEBUG:boto:Data: 2024-04-12T03:28:46.679 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.679 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.679 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.679 DEBUG:boto:Params: {} 2024-04-12T03:28:46.679 DEBUG:boto:Token: None 2024-04-12T03:28:46.680 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.680 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088072c8b7de099e5-006618aa6e-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.682 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.683 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.683 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.683 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.683 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.683 DEBUG:boto:Method: GET 2024-04-12T03:28:46.683 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.683 DEBUG:boto:Data: 2024-04-12T03:28:46.683 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.684 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.684 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.684 DEBUG:boto:Params: {} 2024-04-12T03:28:46.684 DEBUG:boto:Token: None 2024-04-12T03:28:46.684 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.684 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e78dfa8516403918-006618aa6e-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.730 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.730 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.731 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.731 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.731 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:46.731 DEBUG:boto:Data: 2024-04-12T03:28:46.731 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.731 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.731 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.731 DEBUG:boto:Params: {} 2024-04-12T03:28:46.731 DEBUG:boto:Token: None 2024-04-12T03:28:46.731 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj 2024-04-12T03:28:46.732 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU= 2024-04-12T03:28:46.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU=', 'Content-Length': '0'} 2024-04-12T03:28:46.735 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005029f6de25764efe-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.735 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.735 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.735 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.735 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:46.735 DEBUG:boto:Data: 2024-04-12T03:28:46.735 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.735 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.735 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.736 DEBUG:boto:Params: {} 2024-04-12T03:28:46.736 DEBUG:boto:Token: None 2024-04-12T03:28:46.736 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj 2024-04-12T03:28:46.736 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU= 2024-04-12T03:28:46.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU=', 'Content-Length': '0'} 2024-04-12T03:28:46.739 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b935ef9c23474bc-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:46.739 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.739 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.740 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.740 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.740 DEBUG:boto:Method: GET 2024-04-12T03:28:46.740 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.740 DEBUG:boto:Data: 2024-04-12T03:28:46.740 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.740 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.740 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.740 DEBUG:boto:Params: {} 2024-04-12T03:28:46.740 DEBUG:boto:Token: None 2024-04-12T03:28:46.740 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.740 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok= 2024-04-12T03:28:46.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok=', 'Content-Length': '0'} 2024-04-12T03:28:46.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000017b542bc1b8bd84f-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.782 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.782 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.783 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.783 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.783 DEBUG:boto:Method: GET 2024-04-12T03:28:46.783 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.783 DEBUG:boto:Data: 2024-04-12T03:28:46.783 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.783 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.783 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.783 DEBUG:boto:Params: {} 2024-04-12T03:28:46.783 DEBUG:boto:Token: None 2024-04-12T03:28:46.783 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.784 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok= 2024-04-12T03:28:46.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok=', 'Content-Length': '0'} 2024-04-12T03:28:46.785 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000749c8dd851d889f0-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.826 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.826 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.827 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.827 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.827 DEBUG:boto:Method: GET 2024-04-12T03:28:46.827 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.827 DEBUG:boto:Data: 2024-04-12T03:28:46.827 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.827 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.827 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.827 DEBUG:boto:Params: {} 2024-04-12T03:28:46.827 DEBUG:boto:Token: None 2024-04-12T03:28:46.828 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.828 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aa755ee155966d9-006618aa6e-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.830 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.831 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.831 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.831 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.831 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.831 DEBUG:boto:Method: GET 2024-04-12T03:28:46.831 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.831 DEBUG:boto:Data: 2024-04-12T03:28:46.831 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.832 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.832 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.832 DEBUG:boto:Params: {} 2024-04-12T03:28:46.832 DEBUG:boto:Token: None 2024-04-12T03:28:46.832 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.832 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6033bee249cd445-006618aa6e-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.835 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:46.835 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.835 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.836 DEBUG:boto:path=/dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.836 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.836 DEBUG:boto:Method: GET 2024-04-12T03:28:46.836 DEBUG:boto:Path: /dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.836 DEBUG:boto:Data: 2024-04-12T03:28:46.836 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.836 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.836 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.836 DEBUG:boto:Params: {} 2024-04-12T03:28:46.836 DEBUG:boto:Token: None 2024-04-12T03:28:46.837 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.837 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qcO22msf6vjoA05dbJ/HA7bt2g4= 2024-04-12T03:28:46.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qcO22msf6vjoA05dbJ/HA7bt2g4=', 'Content-Length': '0'} 2024-04-12T03:28:46.838 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c4cd38a7e766ace-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.839 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.839 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.839 DEBUG:boto:path=/dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.839 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.839 DEBUG:boto:Method: GET 2024-04-12T03:28:46.839 DEBUG:boto:Path: /dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.839 DEBUG:boto:Data: 2024-04-12T03:28:46.839 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.839 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.840 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.840 DEBUG:boto:Params: {} 2024-04-12T03:28:46.840 DEBUG:boto:Token: None 2024-04-12T03:28:46.840 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.840 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qcO22msf6vjoA05dbJ/HA7bt2g4= 2024-04-12T03:28:46.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qcO22msf6vjoA05dbJ/HA7bt2g4=', 'Content-Length': '0'} 2024-04-12T03:28:46.842 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6e682feb3add80b-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.882 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.883 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.883 DEBUG:boto:path=/dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.883 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.883 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.883 DEBUG:boto:Path: /dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.883 DEBUG:boto:Data: 2024-04-12T03:28:46.883 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.883 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.883 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.883 DEBUG:boto:Params: {} 2024-04-12T03:28:46.883 DEBUG:boto:Token: None 2024-04-12T03:28:46.884 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.884 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VlaF7MyseAmyof46eNgcIg2cVL8= 2024-04-12T03:28:46.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VlaF7MyseAmyof46eNgcIg2cVL8=', 'Content-Length': '0'} 2024-04-12T03:28:46.886 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002f3b9167a2a0b229-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.886 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.886 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.886 DEBUG:boto:path=/dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.886 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.887 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.887 DEBUG:boto:Path: /dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.887 DEBUG:boto:Data: 2024-04-12T03:28:46.887 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.887 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.887 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.887 DEBUG:boto:Params: {} 2024-04-12T03:28:46.887 DEBUG:boto:Token: None 2024-04-12T03:28:46.887 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.887 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VlaF7MyseAmyof46eNgcIg2cVL8= 2024-04-12T03:28:46.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VlaF7MyseAmyof46eNgcIg2cVL8=', 'Content-Length': '0'} 2024-04-12T03:28:46.889 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c17d6c5e1cb8c421-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:46.889 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.890 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.890 DEBUG:boto:path=/dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.890 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.890 DEBUG:boto:Method: GET 2024-04-12T03:28:46.890 DEBUG:boto:Path: /dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.890 DEBUG:boto:Data: 2024-04-12T03:28:46.890 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.890 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.890 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.890 DEBUG:boto:Params: {} 2024-04-12T03:28:46.890 DEBUG:boto:Token: None 2024-04-12T03:28:46.891 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.891 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qcO22msf6vjoA05dbJ/HA7bt2g4= 2024-04-12T03:28:46.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qcO22msf6vjoA05dbJ/HA7bt2g4=', 'Content-Length': '0'} 2024-04-12T03:28:46.892 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000056a5111d037a28da-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.934 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.934 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.934 DEBUG:boto:path=/dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.935 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.935 DEBUG:boto:Method: GET 2024-04-12T03:28:46.935 DEBUG:boto:Path: /dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.935 DEBUG:boto:Data: 2024-04-12T03:28:46.935 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.935 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.935 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.935 DEBUG:boto:Params: {} 2024-04-12T03:28:46.935 DEBUG:boto:Token: None 2024-04-12T03:28:46.935 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj 2024-04-12T03:28:46.936 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qcO22msf6vjoA05dbJ/HA7bt2g4= 2024-04-12T03:28:46.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qcO22msf6vjoA05dbJ/HA7bt2g4=', 'Content-Length': '0'} 2024-04-12T03:28:46.937 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'Z3mUi5oz1AZqDhZS-0eePbcQI7nb4Fj'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe52ba08591ff6f6-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.938 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.938 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.938 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.938 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.938 DEBUG:boto:Method: GET 2024-04-12T03:28:46.938 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.938 DEBUG:boto:Data: 2024-04-12T03:28:46.939 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.939 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.939 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.939 DEBUG:boto:Params: {} 2024-04-12T03:28:46.939 DEBUG:boto:Token: None 2024-04-12T03:28:46.939 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.939 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e217dd59112e40c6-006618aa6e-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.942 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.942 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:46.942 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:46.943 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.943 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:46.943 DEBUG:boto:Method: GET 2024-04-12T03:28:46.943 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:46.943 DEBUG:boto:Data: 2024-04-12T03:28:46.943 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.943 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.943 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.943 DEBUG:boto:Params: {} 2024-04-12T03:28:46.943 DEBUG:boto:Token: None 2024-04-12T03:28:46.943 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/ 2024-04-12T03:28:46.944 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw= 2024-04-12T03:28:46.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP3Mbub6s4HzBXtLyKBk1yi4BVw=', 'Content-Length': '0'} 2024-04-12T03:28:46.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fc7859ca90e13c2-006618aa6e-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.946 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:46.947 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.947 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.947 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.947 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:46.947 DEBUG:boto:Data: 2024-04-12T03:28:46.947 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.947 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.948 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.948 DEBUG:boto:Params: {} 2024-04-12T03:28:46.948 DEBUG:boto:Token: None 2024-04-12T03:28:46.948 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj 2024-04-12T03:28:46.948 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU= 2024-04-12T03:28:46.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU=', 'Content-Length': '0'} 2024-04-12T03:28:46.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008842b2500488d26f-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.951 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.951 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.951 DEBUG:boto:Method: HEAD 2024-04-12T03:28:46.951 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:46.952 DEBUG:boto:Data: 2024-04-12T03:28:46.952 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.952 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:46.952 DEBUG:boto:Port: 8000 2024-04-12T03:28:46.952 DEBUG:boto:Params: {} 2024-04-12T03:28:46.952 DEBUG:boto:Token: None 2024-04-12T03:28:46.952 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj 2024-04-12T03:28:46.952 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU= 2024-04-12T03:28:46.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DcOMxJK3ba9VK2yd5MQ15XKrZaU=', 'Content-Length': '0'} 2024-04-12T03:28:46.955 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a331343959f4f80e-006618aa6e-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:46.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:46.956 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:46.956 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:46.956 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.956 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.956 DEBUG:boto:Method: GET 2024-04-12T03:28:46.956 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.956 DEBUG:boto:Data: 2024-04-12T03:28:46.956 DEBUG:boto:Headers: {} 2024-04-12T03:28:46.956 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:46.956 DEBUG:boto:Port: 8001 2024-04-12T03:28:46.957 DEBUG:boto:Params: {} 2024-04-12T03:28:46.957 DEBUG:boto:Token: None 2024-04-12T03:28:46.957 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:46 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:46.957 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok= 2024-04-12T03:28:46.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/gJbw3Dink/fQw6zv8JWeqndhok=', 'Content-Length': '0'} 2024-04-12T03:28:46.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000961dd352a7c4f484-006618aa6e-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.002 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.002 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.003 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.003 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.003 DEBUG:boto:Method: GET 2024-04-12T03:28:47.003 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.003 DEBUG:boto:Data: 2024-04-12T03:28:47.003 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.003 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.003 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.003 DEBUG:boto:Params: {} 2024-04-12T03:28:47.003 DEBUG:boto:Token: None 2024-04-12T03:28:47.003 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw= 2024-04-12T03:28:47.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw=', 'Content-Length': '0'} 2024-04-12T03:28:47.006 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c30b20ba92fa3df6-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.046 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.047 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.047 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.047 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.047 DEBUG:boto:Method: GET 2024-04-12T03:28:47.047 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.047 DEBUG:boto:Data: 2024-04-12T03:28:47.047 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.047 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.047 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.047 DEBUG:boto:Params: {} 2024-04-12T03:28:47.048 DEBUG:boto:Token: None 2024-04-12T03:28:47.048 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.048 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c57781553bbd3c9b-006618aa6f-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.051 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.051 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.051 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.051 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.051 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.051 DEBUG:boto:Method: GET 2024-04-12T03:28:47.052 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.052 DEBUG:boto:Data: 2024-04-12T03:28:47.052 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.052 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.052 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.052 DEBUG:boto:Params: {} 2024-04-12T03:28:47.052 DEBUG:boto:Token: None 2024-04-12T03:28:47.052 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.052 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002994538feba38592-006618aa6f-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.098 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:47.099 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.099 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.099 DEBUG:boto:path=/dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.099 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.099 DEBUG:boto:Method: GET 2024-04-12T03:28:47.099 DEBUG:boto:Path: /dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.099 DEBUG:boto:Data: 2024-04-12T03:28:47.099 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.099 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.100 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.100 DEBUG:boto:Params: {} 2024-04-12T03:28:47.100 DEBUG:boto:Token: None 2024-04-12T03:28:47.100 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.100 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gRw9GRAIbind6b5yrpESsvMCcZU= 2024-04-12T03:28:47.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gRw9GRAIbind6b5yrpESsvMCcZU=', 'Content-Length': '0'} 2024-04-12T03:28:47.102 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e296970d0ed70b50-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.102 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.102 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.102 DEBUG:boto:path=/dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.102 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.103 DEBUG:boto:Method: GET 2024-04-12T03:28:47.103 DEBUG:boto:Path: /dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.103 DEBUG:boto:Data: 2024-04-12T03:28:47.103 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.103 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.103 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.103 DEBUG:boto:Params: {} 2024-04-12T03:28:47.103 DEBUG:boto:Token: None 2024-04-12T03:28:47.103 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.103 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gRw9GRAIbind6b5yrpESsvMCcZU= 2024-04-12T03:28:47.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gRw9GRAIbind6b5yrpESsvMCcZU=', 'Content-Length': '0'} 2024-04-12T03:28:47.105 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e5b8e41e04310871-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.106 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.106 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.106 DEBUG:boto:path=/dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.106 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.106 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.106 DEBUG:boto:Path: /dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.106 DEBUG:boto:Data: 2024-04-12T03:28:47.106 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.107 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.107 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.107 DEBUG:boto:Params: {} 2024-04-12T03:28:47.107 DEBUG:boto:Token: None 2024-04-12T03:28:47.107 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.107 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:F14bmSYdrVf5jUKkCvUYYXwOZeA= 2024-04-12T03:28:47.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:F14bmSYdrVf5jUKkCvUYYXwOZeA=', 'Content-Length': '0'} 2024-04-12T03:28:47.109 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007430ea4b1e09da1a-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.109 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.109 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.109 DEBUG:boto:path=/dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.109 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.109 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.109 DEBUG:boto:Path: /dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.109 DEBUG:boto:Data: 2024-04-12T03:28:47.110 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.110 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.110 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.110 DEBUG:boto:Params: {} 2024-04-12T03:28:47.110 DEBUG:boto:Token: None 2024-04-12T03:28:47.110 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.110 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:F14bmSYdrVf5jUKkCvUYYXwOZeA= 2024-04-12T03:28:47.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:F14bmSYdrVf5jUKkCvUYYXwOZeA=', 'Content-Length': '0'} 2024-04-12T03:28:47.112 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004b7234c14005e74-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:47.112 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.113 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.113 DEBUG:boto:path=/dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.113 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.113 DEBUG:boto:Method: GET 2024-04-12T03:28:47.113 DEBUG:boto:Path: /dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.113 DEBUG:boto:Data: 2024-04-12T03:28:47.113 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.113 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.113 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.113 DEBUG:boto:Params: {} 2024-04-12T03:28:47.113 DEBUG:boto:Token: None 2024-04-12T03:28:47.114 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.114 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gRw9GRAIbind6b5yrpESsvMCcZU= 2024-04-12T03:28:47.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gRw9GRAIbind6b5yrpESsvMCcZU=', 'Content-Length': '0'} 2024-04-12T03:28:47.116 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001dcda65f59955653-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.116 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.117 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.117 DEBUG:boto:path=/dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.117 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.117 DEBUG:boto:Method: GET 2024-04-12T03:28:47.117 DEBUG:boto:Path: /dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.117 DEBUG:boto:Data: 2024-04-12T03:28:47.117 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.117 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.117 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.117 DEBUG:boto:Params: {} 2024-04-12T03:28:47.118 DEBUG:boto:Token: None 2024-04-12T03:28:47.118 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV 2024-04-12T03:28:47.118 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gRw9GRAIbind6b5yrpESsvMCcZU= 2024-04-12T03:28:47.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gRw9GRAIbind6b5yrpESsvMCcZU=', 'Content-Length': '0'} 2024-04-12T03:28:47.120 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', '6bZtxwV7yUc-Hr7bHVHXh3ex1mdQAZV'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4ada99097fbc261-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.162 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.162 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.162 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.163 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.163 DEBUG:boto:Method: GET 2024-04-12T03:28:47.163 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.163 DEBUG:boto:Data: 2024-04-12T03:28:47.163 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.163 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.163 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.163 DEBUG:boto:Params: {} 2024-04-12T03:28:47.163 DEBUG:boto:Token: None 2024-04-12T03:28:47.163 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.164 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaad4f25859cd431-006618aa6f-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.166 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.167 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.167 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.167 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.167 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.167 DEBUG:boto:Method: GET 2024-04-12T03:28:47.167 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.167 DEBUG:boto:Data: 2024-04-12T03:28:47.167 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.167 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.167 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.168 DEBUG:boto:Params: {} 2024-04-12T03:28:47.168 DEBUG:boto:Token: None 2024-04-12T03:28:47.168 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.168 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d3ce79d1e21f243-006618aa6f-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.171 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.171 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.171 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.171 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.171 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:47.171 DEBUG:boto:Data: 2024-04-12T03:28:47.171 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.171 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.172 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.172 DEBUG:boto:Params: {} 2024-04-12T03:28:47.172 DEBUG:boto:Token: None 2024-04-12T03:28:47.172 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj 2024-04-12T03:28:47.172 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9g1CY1rkUxbfZHxbYBfEDKYk5Ds= 2024-04-12T03:28:47.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9g1CY1rkUxbfZHxbYBfEDKYk5Ds=', 'Content-Length': '0'} 2024-04-12T03:28:47.175 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ec2a16eccde96a3d-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.175 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.176 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.176 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.176 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:47.176 DEBUG:boto:Data: 2024-04-12T03:28:47.176 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.176 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.176 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.176 DEBUG:boto:Params: {} 2024-04-12T03:28:47.176 DEBUG:boto:Token: None 2024-04-12T03:28:47.176 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj 2024-04-12T03:28:47.176 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9g1CY1rkUxbfZHxbYBfEDKYk5Ds= 2024-04-12T03:28:47.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9g1CY1rkUxbfZHxbYBfEDKYk5Ds=', 'Content-Length': '0'} 2024-04-12T03:28:47.179 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e4fbf4911efea1ed-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:47.180 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.180 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.180 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.180 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.180 DEBUG:boto:Method: GET 2024-04-12T03:28:47.180 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.180 DEBUG:boto:Data: 2024-04-12T03:28:47.180 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.180 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.181 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.181 DEBUG:boto:Params: {} 2024-04-12T03:28:47.181 DEBUG:boto:Token: None 2024-04-12T03:28:47.181 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.181 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw= 2024-04-12T03:28:47.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw=', 'Content-Length': '0'} 2024-04-12T03:28:47.183 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000570969aaf9cc6c4d-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.226 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.226 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.227 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.227 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.227 DEBUG:boto:Method: GET 2024-04-12T03:28:47.227 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.227 DEBUG:boto:Data: 2024-04-12T03:28:47.227 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.227 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.227 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.227 DEBUG:boto:Params: {} 2024-04-12T03:28:47.227 DEBUG:boto:Token: None 2024-04-12T03:28:47.227 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.228 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw= 2024-04-12T03:28:47.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw=', 'Content-Length': '0'} 2024-04-12T03:28:47.230 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cff5cd0264f8f006-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.270 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.270 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.271 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.271 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.271 DEBUG:boto:Method: GET 2024-04-12T03:28:47.271 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.271 DEBUG:boto:Data: 2024-04-12T03:28:47.271 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.271 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.271 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.271 DEBUG:boto:Params: {} 2024-04-12T03:28:47.271 DEBUG:boto:Token: None 2024-04-12T03:28:47.271 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.272 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a398c62a7793bcc8-006618aa6f-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.274 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.275 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.275 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.275 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.275 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.275 DEBUG:boto:Method: GET 2024-04-12T03:28:47.275 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.275 DEBUG:boto:Data: 2024-04-12T03:28:47.275 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.275 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.276 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.276 DEBUG:boto:Params: {} 2024-04-12T03:28:47.276 DEBUG:boto:Token: None 2024-04-12T03:28:47.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.276 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d9d97a3542731d9-006618aa6f-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.279 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:47.279 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.279 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.279 DEBUG:boto:path=/dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.279 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.280 DEBUG:boto:Method: GET 2024-04-12T03:28:47.280 DEBUG:boto:Path: /dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.280 DEBUG:boto:Data: 2024-04-12T03:28:47.280 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.280 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.280 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.280 DEBUG:boto:Params: {} 2024-04-12T03:28:47.280 DEBUG:boto:Token: None 2024-04-12T03:28:47.280 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.280 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+HGo516VIYRqGuhAjpmnCZC00mw= 2024-04-12T03:28:47.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+HGo516VIYRqGuhAjpmnCZC00mw=', 'Content-Length': '0'} 2024-04-12T03:28:47.282 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000095aec224b1df72c6-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.283 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.283 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.283 DEBUG:boto:path=/dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.283 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.283 DEBUG:boto:Method: GET 2024-04-12T03:28:47.283 DEBUG:boto:Path: /dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.283 DEBUG:boto:Data: 2024-04-12T03:28:47.283 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.283 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.283 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.283 DEBUG:boto:Params: {} 2024-04-12T03:28:47.284 DEBUG:boto:Token: None 2024-04-12T03:28:47.284 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.284 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+HGo516VIYRqGuhAjpmnCZC00mw= 2024-04-12T03:28:47.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+HGo516VIYRqGuhAjpmnCZC00mw=', 'Content-Length': '0'} 2024-04-12T03:28:47.286 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000040315275cb753ee7-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.286 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.286 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.286 DEBUG:boto:path=/dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.286 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.286 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.286 DEBUG:boto:Path: /dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.287 DEBUG:boto:Data: 2024-04-12T03:28:47.287 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.287 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.287 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.287 DEBUG:boto:Params: {} 2024-04-12T03:28:47.287 DEBUG:boto:Token: None 2024-04-12T03:28:47.287 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.287 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BeaCHnH/mHVDvlk4PzGzd2ZBrP8= 2024-04-12T03:28:47.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BeaCHnH/mHVDvlk4PzGzd2ZBrP8=', 'Content-Length': '0'} 2024-04-12T03:28:47.289 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000471b85cba36d1edb-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.289 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.289 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.289 DEBUG:boto:path=/dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.290 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.290 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.290 DEBUG:boto:Path: /dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.290 DEBUG:boto:Data: 2024-04-12T03:28:47.290 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.290 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.290 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.290 DEBUG:boto:Params: {} 2024-04-12T03:28:47.290 DEBUG:boto:Token: None 2024-04-12T03:28:47.290 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.291 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BeaCHnH/mHVDvlk4PzGzd2ZBrP8= 2024-04-12T03:28:47.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BeaCHnH/mHVDvlk4PzGzd2ZBrP8=', 'Content-Length': '0'} 2024-04-12T03:28:47.292 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008180cf29d9406edf-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:47.293 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.293 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.293 DEBUG:boto:path=/dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.293 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.293 DEBUG:boto:Method: GET 2024-04-12T03:28:47.293 DEBUG:boto:Path: /dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.293 DEBUG:boto:Data: 2024-04-12T03:28:47.293 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.293 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.294 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.294 DEBUG:boto:Params: {} 2024-04-12T03:28:47.294 DEBUG:boto:Token: None 2024-04-12T03:28:47.294 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.294 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+HGo516VIYRqGuhAjpmnCZC00mw= 2024-04-12T03:28:47.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+HGo516VIYRqGuhAjpmnCZC00mw=', 'Content-Length': '0'} 2024-04-12T03:28:47.296 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl.'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026899dc50f8414c5-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.338 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.338 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.339 DEBUG:boto:path=/dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.339 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.339 DEBUG:boto:Method: GET 2024-04-12T03:28:47.339 DEBUG:boto:Path: /dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.339 DEBUG:boto:Data: 2024-04-12T03:28:47.339 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.339 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.339 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.339 DEBUG:boto:Params: {} 2024-04-12T03:28:47.339 DEBUG:boto:Token: None 2024-04-12T03:28:47.340 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl. 2024-04-12T03:28:47.340 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+HGo516VIYRqGuhAjpmnCZC00mw= 2024-04-12T03:28:47.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+HGo516VIYRqGuhAjpmnCZC00mw=', 'Content-Length': '0'} 2024-04-12T03:28:47.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'SQ4-mXfRGBqGjQRWOEV5qPda3uaZwl.'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008601f9f63f7d781b-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.342 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.342 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.342 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.342 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.342 DEBUG:boto:Method: GET 2024-04-12T03:28:47.342 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.342 DEBUG:boto:Data: 2024-04-12T03:28:47.343 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.343 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.343 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.343 DEBUG:boto:Params: {} 2024-04-12T03:28:47.343 DEBUG:boto:Token: None 2024-04-12T03:28:47.343 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.343 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000483a0fd816e56d1e-006618aa6f-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.386 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.387 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.387 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.387 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.387 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.387 DEBUG:boto:Method: GET 2024-04-12T03:28:47.387 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.387 DEBUG:boto:Data: 2024-04-12T03:28:47.387 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.387 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.388 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.388 DEBUG:boto:Params: {} 2024-04-12T03:28:47.388 DEBUG:boto:Token: None 2024-04-12T03:28:47.388 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.388 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009903ab281ba75a0e-006618aa6f-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.391 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.391 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.391 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.392 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.392 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:47.392 DEBUG:boto:Data: 2024-04-12T03:28:47.392 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.392 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.392 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.392 DEBUG:boto:Params: {} 2024-04-12T03:28:47.392 DEBUG:boto:Token: None 2024-04-12T03:28:47.392 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj 2024-04-12T03:28:47.392 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9g1CY1rkUxbfZHxbYBfEDKYk5Ds= 2024-04-12T03:28:47.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9g1CY1rkUxbfZHxbYBfEDKYk5Ds=', 'Content-Length': '0'} 2024-04-12T03:28:47.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000196dfc83a202e5e4-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.396 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.396 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.396 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.396 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:47.396 DEBUG:boto:Data: 2024-04-12T03:28:47.396 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.396 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.396 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.396 DEBUG:boto:Params: {} 2024-04-12T03:28:47.396 DEBUG:boto:Token: None 2024-04-12T03:28:47.396 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj 2024-04-12T03:28:47.397 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9g1CY1rkUxbfZHxbYBfEDKYk5Ds= 2024-04-12T03:28:47.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9g1CY1rkUxbfZHxbYBfEDKYk5Ds=', 'Content-Length': '0'} 2024-04-12T03:28:47.399 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003a9554af5981089e-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:47.400 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.400 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.400 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.400 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.400 DEBUG:boto:Method: GET 2024-04-12T03:28:47.400 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.400 DEBUG:boto:Data: 2024-04-12T03:28:47.400 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.401 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.401 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.401 DEBUG:boto:Params: {} 2024-04-12T03:28:47.401 DEBUG:boto:Token: None 2024-04-12T03:28:47.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.401 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw= 2024-04-12T03:28:47.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw=', 'Content-Length': '0'} 2024-04-12T03:28:47.403 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009be4279e503bfac3-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.403 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.403 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.403 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.403 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.403 DEBUG:boto:Method: GET 2024-04-12T03:28:47.403 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.404 DEBUG:boto:Data: 2024-04-12T03:28:47.404 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.404 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.404 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.404 DEBUG:boto:Params: {} 2024-04-12T03:28:47.404 DEBUG:boto:Token: None 2024-04-12T03:28:47.404 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.404 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw= 2024-04-12T03:28:47.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw=', 'Content-Length': '0'} 2024-04-12T03:28:47.406 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054a1671b0e6bfc35-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.446 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.447 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.447 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.447 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.447 DEBUG:boto:Method: GET 2024-04-12T03:28:47.447 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.447 DEBUG:boto:Data: 2024-04-12T03:28:47.447 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.447 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.447 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.447 DEBUG:boto:Params: {} 2024-04-12T03:28:47.447 DEBUG:boto:Token: None 2024-04-12T03:28:47.448 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.448 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bf6f604d54900f6-006618aa6f-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.451 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.451 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.451 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.451 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.451 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.451 DEBUG:boto:Method: GET 2024-04-12T03:28:47.451 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.452 DEBUG:boto:Data: 2024-04-12T03:28:47.452 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.452 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.452 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.452 DEBUG:boto:Params: {} 2024-04-12T03:28:47.452 DEBUG:boto:Token: None 2024-04-12T03:28:47.452 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.452 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e95411f777829983-006618aa6f-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.455 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:47.455 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.455 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.456 DEBUG:boto:path=/dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.456 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.456 DEBUG:boto:Method: GET 2024-04-12T03:28:47.456 DEBUG:boto:Path: /dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.456 DEBUG:boto:Data: 2024-04-12T03:28:47.456 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.456 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.456 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.456 DEBUG:boto:Params: {} 2024-04-12T03:28:47.456 DEBUG:boto:Token: None 2024-04-12T03:28:47.456 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.457 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Eqmv8Sv+4Mty0KqS7mzkzvTvrRc= 2024-04-12T03:28:47.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Eqmv8Sv+4Mty0KqS7mzkzvTvrRc=', 'Content-Length': '0'} 2024-04-12T03:28:47.458 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'L9iWynXqdS0BqqL22bTZDCGwDwt2itq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009998094c32b73eb4-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.502 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.502 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.503 DEBUG:boto:path=/dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.503 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.503 DEBUG:boto:Method: GET 2024-04-12T03:28:47.503 DEBUG:boto:Path: /dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.503 DEBUG:boto:Data: 2024-04-12T03:28:47.503 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.503 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.503 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.503 DEBUG:boto:Params: {} 2024-04-12T03:28:47.503 DEBUG:boto:Token: None 2024-04-12T03:28:47.504 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.504 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Eqmv8Sv+4Mty0KqS7mzkzvTvrRc= 2024-04-12T03:28:47.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Eqmv8Sv+4Mty0KqS7mzkzvTvrRc=', 'Content-Length': '0'} 2024-04-12T03:28:47.506 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'L9iWynXqdS0BqqL22bTZDCGwDwt2itq'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac6139d614b5f7dd-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.506 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.506 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.506 DEBUG:boto:path=/dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.506 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.506 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.507 DEBUG:boto:Path: /dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.507 DEBUG:boto:Data: 2024-04-12T03:28:47.507 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.507 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.507 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.507 DEBUG:boto:Params: {} 2024-04-12T03:28:47.507 DEBUG:boto:Token: None 2024-04-12T03:28:47.507 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.507 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WNlRp7h1S2wfC98IuV2rGY1q01k= 2024-04-12T03:28:47.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WNlRp7h1S2wfC98IuV2rGY1q01k=', 'Content-Length': '0'} 2024-04-12T03:28:47.509 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'L9iWynXqdS0BqqL22bTZDCGwDwt2itq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002eb58af06693ee3a-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.509 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.510 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.510 DEBUG:boto:path=/dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.510 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.510 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.510 DEBUG:boto:Path: /dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.510 DEBUG:boto:Data: 2024-04-12T03:28:47.510 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.510 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.510 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.510 DEBUG:boto:Params: {} 2024-04-12T03:28:47.511 DEBUG:boto:Token: None 2024-04-12T03:28:47.511 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.511 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WNlRp7h1S2wfC98IuV2rGY1q01k= 2024-04-12T03:28:47.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WNlRp7h1S2wfC98IuV2rGY1q01k=', 'Content-Length': '0'} 2024-04-12T03:28:47.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'L9iWynXqdS0BqqL22bTZDCGwDwt2itq'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007ad7821fc9e695a-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:47.513 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.513 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.513 DEBUG:boto:path=/dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.513 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.514 DEBUG:boto:Method: GET 2024-04-12T03:28:47.514 DEBUG:boto:Path: /dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.514 DEBUG:boto:Data: 2024-04-12T03:28:47.514 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.514 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.514 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.514 DEBUG:boto:Params: {} 2024-04-12T03:28:47.514 DEBUG:boto:Token: None 2024-04-12T03:28:47.514 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.515 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Eqmv8Sv+4Mty0KqS7mzkzvTvrRc= 2024-04-12T03:28:47.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Eqmv8Sv+4Mty0KqS7mzkzvTvrRc=', 'Content-Length': '0'} 2024-04-12T03:28:47.516 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'L9iWynXqdS0BqqL22bTZDCGwDwt2itq'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068ef5612ce3c0cb9-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.517 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.517 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.517 DEBUG:boto:path=/dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.517 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.517 DEBUG:boto:Method: GET 2024-04-12T03:28:47.517 DEBUG:boto:Path: /dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.517 DEBUG:boto:Data: 2024-04-12T03:28:47.517 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.517 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.518 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.518 DEBUG:boto:Params: {} 2024-04-12T03:28:47.518 DEBUG:boto:Token: None 2024-04-12T03:28:47.518 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=L9iWynXqdS0BqqL22bTZDCGwDwt2itq 2024-04-12T03:28:47.518 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Eqmv8Sv+4Mty0KqS7mzkzvTvrRc= 2024-04-12T03:28:47.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Eqmv8Sv+4Mty0KqS7mzkzvTvrRc=', 'Content-Length': '0'} 2024-04-12T03:28:47.520 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'L9iWynXqdS0BqqL22bTZDCGwDwt2itq'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ce1d7b892ac03482-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.562 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.562 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.563 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.563 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.563 DEBUG:boto:Method: GET 2024-04-12T03:28:47.563 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.563 DEBUG:boto:Data: 2024-04-12T03:28:47.563 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.563 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.563 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.563 DEBUG:boto:Params: {} 2024-04-12T03:28:47.563 DEBUG:boto:Token: None 2024-04-12T03:28:47.564 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.564 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a0b783845d5b423-006618aa6f-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.567 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.567 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.567 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.567 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.567 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.568 DEBUG:boto:Method: GET 2024-04-12T03:28:47.568 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.568 DEBUG:boto:Data: 2024-04-12T03:28:47.568 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.568 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.568 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.568 DEBUG:boto:Params: {} 2024-04-12T03:28:47.568 DEBUG:boto:Token: None 2024-04-12T03:28:47.568 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.568 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e09d1d77538221ed-006618aa6f-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.614 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.615 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.615 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.615 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.615 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:47.615 DEBUG:boto:Data: 2024-04-12T03:28:47.615 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.615 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.615 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.615 DEBUG:boto:Params: {} 2024-04-12T03:28:47.616 DEBUG:boto:Token: None 2024-04-12T03:28:47.616 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj 2024-04-12T03:28:47.616 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9g1CY1rkUxbfZHxbYBfEDKYk5Ds= 2024-04-12T03:28:47.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9g1CY1rkUxbfZHxbYBfEDKYk5Ds=', 'Content-Length': '0'} 2024-04-12T03:28:47.619 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c0dd2cfdedf81017-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.619 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.619 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.620 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.620 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:47.620 DEBUG:boto:Data: 2024-04-12T03:28:47.620 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.620 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.620 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.620 DEBUG:boto:Params: {} 2024-04-12T03:28:47.620 DEBUG:boto:Token: None 2024-04-12T03:28:47.620 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj 2024-04-12T03:28:47.620 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9g1CY1rkUxbfZHxbYBfEDKYk5Ds= 2024-04-12T03:28:47.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9g1CY1rkUxbfZHxbYBfEDKYk5Ds=', 'Content-Length': '0'} 2024-04-12T03:28:47.624 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c87bd542d1f00ada-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:47.624 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.624 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.624 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.624 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.624 DEBUG:boto:Method: GET 2024-04-12T03:28:47.624 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.625 DEBUG:boto:Data: 2024-04-12T03:28:47.625 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.625 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.625 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.625 DEBUG:boto:Params: {} 2024-04-12T03:28:47.625 DEBUG:boto:Token: None 2024-04-12T03:28:47.625 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.625 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw= 2024-04-12T03:28:47.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw=', 'Content-Length': '0'} 2024-04-12T03:28:47.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007bb9d7805d00d3cc-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.670 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.670 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.671 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.671 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.671 DEBUG:boto:Method: GET 2024-04-12T03:28:47.671 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.671 DEBUG:boto:Data: 2024-04-12T03:28:47.671 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.671 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.671 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.671 DEBUG:boto:Params: {} 2024-04-12T03:28:47.671 DEBUG:boto:Token: None 2024-04-12T03:28:47.672 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.672 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw= 2024-04-12T03:28:47.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw=', 'Content-Length': '0'} 2024-04-12T03:28:47.674 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000370a13ce910d20f1-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.714 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.715 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.715 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.715 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.715 DEBUG:boto:Method: GET 2024-04-12T03:28:47.715 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.715 DEBUG:boto:Data: 2024-04-12T03:28:47.715 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.715 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.715 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.715 DEBUG:boto:Params: {} 2024-04-12T03:28:47.715 DEBUG:boto:Token: None 2024-04-12T03:28:47.716 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.716 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016ca26e208553c5a-006618aa6f-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.718 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.719 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.719 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.719 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.719 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.719 DEBUG:boto:Method: GET 2024-04-12T03:28:47.719 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.719 DEBUG:boto:Data: 2024-04-12T03:28:47.720 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.720 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.720 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.720 DEBUG:boto:Params: {} 2024-04-12T03:28:47.720 DEBUG:boto:Token: None 2024-04-12T03:28:47.720 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.720 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d09b2de4b55a9803-006618aa6f-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.723 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:47.723 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.724 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.724 DEBUG:boto:path=/dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.724 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.724 DEBUG:boto:Method: GET 2024-04-12T03:28:47.724 DEBUG:boto:Path: /dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.724 DEBUG:boto:Data: 2024-04-12T03:28:47.724 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.724 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.724 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.724 DEBUG:boto:Params: {} 2024-04-12T03:28:47.724 DEBUG:boto:Token: None 2024-04-12T03:28:47.725 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.725 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tin1HWmCSwOoVax0A8Frc3E0bpE= 2024-04-12T03:28:47.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tin1HWmCSwOoVax0A8Frc3E0bpE=', 'Content-Length': '0'} 2024-04-12T03:28:47.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'nA8qPHogisi3arF3skk4QGJgaM9iq.b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005126555092be349a-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.727 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.727 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.727 DEBUG:boto:path=/dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.728 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.728 DEBUG:boto:Method: GET 2024-04-12T03:28:47.728 DEBUG:boto:Path: /dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.728 DEBUG:boto:Data: 2024-04-12T03:28:47.728 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.728 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.728 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.728 DEBUG:boto:Params: {} 2024-04-12T03:28:47.728 DEBUG:boto:Token: None 2024-04-12T03:28:47.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.729 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tin1HWmCSwOoVax0A8Frc3E0bpE= 2024-04-12T03:28:47.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tin1HWmCSwOoVax0A8Frc3E0bpE=', 'Content-Length': '0'} 2024-04-12T03:28:47.730 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'nA8qPHogisi3arF3skk4QGJgaM9iq.b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1e60a8423e4606a-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.774 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.775 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.775 DEBUG:boto:path=/dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.775 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.775 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.775 DEBUG:boto:Path: /dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.775 DEBUG:boto:Data: 2024-04-12T03:28:47.775 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.775 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.775 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.775 DEBUG:boto:Params: {} 2024-04-12T03:28:47.776 DEBUG:boto:Token: None 2024-04-12T03:28:47.776 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.776 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:86W5f9pAjZgI+N9nE31j+rDjQfg= 2024-04-12T03:28:47.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:86W5f9pAjZgI+N9nE31j+rDjQfg=', 'Content-Length': '0'} 2024-04-12T03:28:47.778 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'nA8qPHogisi3arF3skk4QGJgaM9iq.b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000035f70d6d8549a4cd-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.778 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.778 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.778 DEBUG:boto:path=/dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.778 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.778 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.778 DEBUG:boto:Path: /dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.779 DEBUG:boto:Data: 2024-04-12T03:28:47.779 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.779 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.779 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.779 DEBUG:boto:Params: {} 2024-04-12T03:28:47.779 DEBUG:boto:Token: None 2024-04-12T03:28:47.779 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.779 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:86W5f9pAjZgI+N9nE31j+rDjQfg= 2024-04-12T03:28:47.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:86W5f9pAjZgI+N9nE31j+rDjQfg=', 'Content-Length': '0'} 2024-04-12T03:28:47.781 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'nA8qPHogisi3arF3skk4QGJgaM9iq.b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0d4998ca80aaece-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:47.781 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.782 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.782 DEBUG:boto:path=/dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.782 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.782 DEBUG:boto:Method: GET 2024-04-12T03:28:47.782 DEBUG:boto:Path: /dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.782 DEBUG:boto:Data: 2024-04-12T03:28:47.782 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.782 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.782 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.782 DEBUG:boto:Params: {} 2024-04-12T03:28:47.783 DEBUG:boto:Token: None 2024-04-12T03:28:47.783 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.783 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tin1HWmCSwOoVax0A8Frc3E0bpE= 2024-04-12T03:28:47.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tin1HWmCSwOoVax0A8Frc3E0bpE=', 'Content-Length': '0'} 2024-04-12T03:28:47.785 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'nA8qPHogisi3arF3skk4QGJgaM9iq.b'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000be527e703a8eee6e-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.826 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.826 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.827 DEBUG:boto:path=/dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.827 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.827 DEBUG:boto:Method: GET 2024-04-12T03:28:47.827 DEBUG:boto:Path: /dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.827 DEBUG:boto:Data: 2024-04-12T03:28:47.827 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.827 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.827 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.827 DEBUG:boto:Params: {} 2024-04-12T03:28:47.827 DEBUG:boto:Token: None 2024-04-12T03:28:47.828 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=nA8qPHogisi3arF3skk4QGJgaM9iq.b 2024-04-12T03:28:47.828 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tin1HWmCSwOoVax0A8Frc3E0bpE= 2024-04-12T03:28:47.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tin1HWmCSwOoVax0A8Frc3E0bpE=', 'Content-Length': '0'} 2024-04-12T03:28:47.830 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'nA8qPHogisi3arF3skk4QGJgaM9iq.b'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d666484450af6bc-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.830 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.830 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.830 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.830 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.830 DEBUG:boto:Method: GET 2024-04-12T03:28:47.830 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.831 DEBUG:boto:Data: 2024-04-12T03:28:47.831 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.831 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.831 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.831 DEBUG:boto:Params: {} 2024-04-12T03:28:47.831 DEBUG:boto:Token: None 2024-04-12T03:28:47.831 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.831 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1fb08a3c7fedd19-006618aa6f-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.834 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.834 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.835 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.835 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.835 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.835 DEBUG:boto:Method: GET 2024-04-12T03:28:47.835 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.835 DEBUG:boto:Data: 2024-04-12T03:28:47.835 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.835 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.835 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.835 DEBUG:boto:Params: {} 2024-04-12T03:28:47.835 DEBUG:boto:Token: None 2024-04-12T03:28:47.836 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.836 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000585a6c31cf823dad-006618aa6f-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.838 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.839 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.839 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.839 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.839 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:47.839 DEBUG:boto:Data: 2024-04-12T03:28:47.839 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.839 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.839 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.839 DEBUG:boto:Params: {} 2024-04-12T03:28:47.842 DEBUG:boto:Token: None 2024-04-12T03:28:47.842 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj 2024-04-12T03:28:47.842 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9g1CY1rkUxbfZHxbYBfEDKYk5Ds= 2024-04-12T03:28:47.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9g1CY1rkUxbfZHxbYBfEDKYk5Ds=', 'Content-Length': '0'} 2024-04-12T03:28:47.845 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000163024138c60c08b-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.845 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.845 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.846 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.846 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:47.846 DEBUG:boto:Data: 2024-04-12T03:28:47.846 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.846 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.846 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.846 DEBUG:boto:Params: {} 2024-04-12T03:28:47.846 DEBUG:boto:Token: None 2024-04-12T03:28:47.846 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj 2024-04-12T03:28:47.846 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9g1CY1rkUxbfZHxbYBfEDKYk5Ds= 2024-04-12T03:28:47.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9g1CY1rkUxbfZHxbYBfEDKYk5Ds=', 'Content-Length': '0'} 2024-04-12T03:28:47.850 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bff00c24d19d43c3-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:47.850 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.850 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.850 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.851 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.851 DEBUG:boto:Method: GET 2024-04-12T03:28:47.851 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.851 DEBUG:boto:Data: 2024-04-12T03:28:47.851 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.851 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.851 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.851 DEBUG:boto:Params: {} 2024-04-12T03:28:47.851 DEBUG:boto:Token: None 2024-04-12T03:28:47.851 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.852 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw= 2024-04-12T03:28:47.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw=', 'Content-Length': '0'} 2024-04-12T03:28:47.853 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8467a15d4dc5d29-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.854 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.854 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.854 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.854 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.854 DEBUG:boto:Method: GET 2024-04-12T03:28:47.854 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.854 DEBUG:boto:Data: 2024-04-12T03:28:47.855 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.855 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.855 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.855 DEBUG:boto:Params: {} 2024-04-12T03:28:47.855 DEBUG:boto:Token: None 2024-04-12T03:28:47.855 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:47.855 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw= 2024-04-12T03:28:47.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MZylAOg09ZwxNj1Vi/CyndlF9cw=', 'Content-Length': '0'} 2024-04-12T03:28:47.857 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001d7822d07e8b21a5-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.898 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.899 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.899 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.899 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.899 DEBUG:boto:Method: GET 2024-04-12T03:28:47.899 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.899 DEBUG:boto:Data: 2024-04-12T03:28:47.899 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.899 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.899 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.899 DEBUG:boto:Params: {} 2024-04-12T03:28:47.899 DEBUG:boto:Token: None 2024-04-12T03:28:47.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.900 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007de3817acf0727ed-006618aa6f-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.903 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.903 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:47.903 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:47.903 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.903 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:47.903 DEBUG:boto:Method: GET 2024-04-12T03:28:47.903 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:47.904 DEBUG:boto:Data: 2024-04-12T03:28:47.904 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.904 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.904 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.904 DEBUG:boto:Params: {} 2024-04-12T03:28:47.904 DEBUG:boto:Token: None 2024-04-12T03:28:47.904 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/ 2024-04-12T03:28:47.904 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ= 2024-04-12T03:28:47.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Djaf6Y1s1eBc7otZzs4y+5oiSaQ=', 'Content-Length': '0'} 2024-04-12T03:28:47.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca7da885d5926747-006618aa6f-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.950 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:47.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:47.951 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.951 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.951 DEBUG:boto:path=/dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:47.951 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:47.951 DEBUG:boto:Method: GET 2024-04-12T03:28:47.951 DEBUG:boto:Path: /dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:47.951 DEBUG:boto:Data: 2024-04-12T03:28:47.951 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.952 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.952 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.952 DEBUG:boto:Params: {} 2024-04-12T03:28:47.952 DEBUG:boto:Token: None 2024-04-12T03:28:47.952 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:47.952 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:31n3I1K4TZgMO//Qf1M0Hx/sKBI= 2024-04-12T03:28:47.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:31n3I1K4TZgMO//Qf1M0Hx/sKBI=', 'Content-Length': '0'} 2024-04-12T03:28:47.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'NNLDujprZgDl--7sM1Pv37sfHNfrmJ5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062b03d5bb6593c6b-006618aa6f-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.994 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.995 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.995 DEBUG:boto:path=/dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:47.995 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:47.995 DEBUG:boto:Method: GET 2024-04-12T03:28:47.995 DEBUG:boto:Path: /dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:47.995 DEBUG:boto:Data: 2024-04-12T03:28:47.995 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.995 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:47.995 DEBUG:boto:Port: 8000 2024-04-12T03:28:47.995 DEBUG:boto:Params: {} 2024-04-12T03:28:47.996 DEBUG:boto:Token: None 2024-04-12T03:28:47.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:47 GMT /dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:47.996 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:31n3I1K4TZgMO//Qf1M0Hx/sKBI= 2024-04-12T03:28:47.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:31n3I1K4TZgMO//Qf1M0Hx/sKBI=', 'Content-Length': '0'} 2024-04-12T03:28:47.998 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'NNLDujprZgDl--7sM1Pv37sfHNfrmJ5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007772b103b7bdf9ec-006618aa6f-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:47.998 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:47.999 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:47.999 DEBUG:boto:path=/dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:47.999 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:47.999 DEBUG:boto:Method: HEAD 2024-04-12T03:28:47.999 DEBUG:boto:Path: /dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:47.999 DEBUG:boto:Data: 2024-04-12T03:28:47.999 DEBUG:boto:Headers: {} 2024-04-12T03:28:47.999 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:47.999 DEBUG:boto:Port: 8001 2024-04-12T03:28:47.999 DEBUG:boto:Params: {} 2024-04-12T03:28:48.000 DEBUG:boto:Token: None 2024-04-12T03:28:48.000 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:48.000 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Sd6KiU6FaDg+XA2kf0Bi/6mGisg= 2024-04-12T03:28:48.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Sd6KiU6FaDg+XA2kf0Bi/6mGisg=', 'Content-Length': '0'} 2024-04-12T03:28:48.001 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'NNLDujprZgDl--7sM1Pv37sfHNfrmJ5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c938118ddb3cd3f-006618aa70-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.002 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.002 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.002 DEBUG:boto:path=/dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:48.002 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:48.002 DEBUG:boto:Method: HEAD 2024-04-12T03:28:48.002 DEBUG:boto:Path: /dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:48.002 DEBUG:boto:Data: 2024-04-12T03:28:48.002 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.002 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:48.003 DEBUG:boto:Port: 8000 2024-04-12T03:28:48.003 DEBUG:boto:Params: {} 2024-04-12T03:28:48.003 DEBUG:boto:Token: None 2024-04-12T03:28:48.003 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:48.003 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Sd6KiU6FaDg+XA2kf0Bi/6mGisg= 2024-04-12T03:28:48.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Sd6KiU6FaDg+XA2kf0Bi/6mGisg=', 'Content-Length': '0'} 2024-04-12T03:28:48.005 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'NNLDujprZgDl--7sM1Pv37sfHNfrmJ5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000338031aafc79948d-006618aa70-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:48.005 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.005 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.005 DEBUG:boto:path=/dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:48.005 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:48.006 DEBUG:boto:Method: GET 2024-04-12T03:28:48.006 DEBUG:boto:Path: /dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:48.006 DEBUG:boto:Data: 2024-04-12T03:28:48.006 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.006 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:48.006 DEBUG:boto:Port: 8001 2024-04-12T03:28:48.006 DEBUG:boto:Params: {} 2024-04-12T03:28:48.006 DEBUG:boto:Token: None 2024-04-12T03:28:48.006 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:48.006 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qZbEOCkciX5yBVG3U5CZnx+80yY= 2024-04-12T03:28:48.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qZbEOCkciX5yBVG3U5CZnx+80yY=', 'Content-Length': '0'} 2024-04-12T03:28:48.008 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'NNLDujprZgDl--7sM1Pv37sfHNfrmJ5'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c944ebcb7298aa43-006618aa70-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.009 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.009 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.009 DEBUG:boto:path=/dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:48.009 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:48.009 DEBUG:boto:Method: GET 2024-04-12T03:28:48.009 DEBUG:boto:Path: /dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:48.009 DEBUG:boto:Data: 2024-04-12T03:28:48.009 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.010 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:48.010 DEBUG:boto:Port: 8000 2024-04-12T03:28:48.010 DEBUG:boto:Params: {} 2024-04-12T03:28:48.010 DEBUG:boto:Token: None 2024-04-12T03:28:48.010 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj?versionId=NNLDujprZgDl--7sM1Pv37sfHNfrmJ5 2024-04-12T03:28:48.010 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qZbEOCkciX5yBVG3U5CZnx+80yY= 2024-04-12T03:28:48.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qZbEOCkciX5yBVG3U5CZnx+80yY=', 'Content-Length': '0'} 2024-04-12T03:28:48.012 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'NNLDujprZgDl--7sM1Pv37sfHNfrmJ5'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c82da9123aba7305-006618aa70-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.054 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:48.055 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:48.055 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:48.055 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:48.055 DEBUG:boto:Method: GET 2024-04-12T03:28:48.055 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:48.055 DEBUG:boto:Data: 2024-04-12T03:28:48.055 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.055 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:48.055 DEBUG:boto:Port: 8001 2024-04-12T03:28:48.055 DEBUG:boto:Params: {} 2024-04-12T03:28:48.056 DEBUG:boto:Token: None 2024-04-12T03:28:48.056 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/ 2024-04-12T03:28:48.056 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QwDOTKzPbYQJ3s1GCbC9RWoF5B0= 2024-04-12T03:28:48.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QwDOTKzPbYQJ3s1GCbC9RWoF5B0=', 'Content-Length': '0'} 2024-04-12T03:28:48.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c292f4d571d057f-006618aa70-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.059 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:48.059 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:48.059 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:48.059 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:48.059 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:48.059 DEBUG:boto:Method: GET 2024-04-12T03:28:48.059 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:48.060 DEBUG:boto:Data: 2024-04-12T03:28:48.060 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.060 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:48.060 DEBUG:boto:Port: 8000 2024-04-12T03:28:48.060 DEBUG:boto:Params: {} 2024-04-12T03:28:48.060 DEBUG:boto:Token: None 2024-04-12T03:28:48.060 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/ 2024-04-12T03:28:48.060 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QwDOTKzPbYQJ3s1GCbC9RWoF5B0= 2024-04-12T03:28:48.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QwDOTKzPbYQJ3s1GCbC9RWoF5B0=', 'Content-Length': '0'} 2024-04-12T03:28:48.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000432c4c5115dccde9-006618aa70-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.106 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:48.107 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.107 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.107 DEBUG:boto:Method: HEAD 2024-04-12T03:28:48.107 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:48.107 DEBUG:boto:Data: 2024-04-12T03:28:48.107 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.107 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:48.107 DEBUG:boto:Port: 8001 2024-04-12T03:28:48.108 DEBUG:boto:Params: {} 2024-04-12T03:28:48.108 DEBUG:boto:Token: None 2024-04-12T03:28:48.108 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj 2024-04-12T03:28:48.108 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Rrsy8yXwftnYXEHtQ+tSSF5eHx8= 2024-04-12T03:28:48.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Rrsy8yXwftnYXEHtQ+tSSF5eHx8=', 'Content-Length': '0'} 2024-04-12T03:28:48.111 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007f160134fefc849-006618aa70-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.111 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.111 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.111 DEBUG:boto:Method: HEAD 2024-04-12T03:28:48.112 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:48.112 DEBUG:boto:Data: 2024-04-12T03:28:48.112 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.112 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:48.112 DEBUG:boto:Port: 8000 2024-04-12T03:28:48.112 DEBUG:boto:Params: {} 2024-04-12T03:28:48.112 DEBUG:boto:Token: None 2024-04-12T03:28:48.112 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj 2024-04-12T03:28:48.112 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Rrsy8yXwftnYXEHtQ+tSSF5eHx8= 2024-04-12T03:28:48.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Rrsy8yXwftnYXEHtQ+tSSF5eHx8=', 'Content-Length': '0'} 2024-04-12T03:28:48.116 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c8ccd3badba1ad0-006618aa70-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:48.116 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.116 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.116 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:48.116 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:48.117 DEBUG:boto:Method: GET 2024-04-12T03:28:48.117 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:48.117 DEBUG:boto:Data: 2024-04-12T03:28:48.117 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.117 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:48.117 DEBUG:boto:Port: 8001 2024-04-12T03:28:48.117 DEBUG:boto:Params: {} 2024-04-12T03:28:48.117 DEBUG:boto:Token: None 2024-04-12T03:28:48.117 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:48.117 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+NJPHeG41rGHmdFHvfEMpP5m9Io= 2024-04-12T03:28:48.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+NJPHeG41rGHmdFHvfEMpP5m9Io=', 'Content-Length': '0'} 2024-04-12T03:28:48.119 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003fd0688f97c0ae61-006618aa70-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.162 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.163 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.163 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:48.163 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:48.163 DEBUG:boto:Method: GET 2024-04-12T03:28:48.163 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:48.163 DEBUG:boto:Data: 2024-04-12T03:28:48.163 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.163 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:48.163 DEBUG:boto:Port: 8000 2024-04-12T03:28:48.163 DEBUG:boto:Params: {} 2024-04-12T03:28:48.163 DEBUG:boto:Token: None 2024-04-12T03:28:48.164 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:48.164 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+NJPHeG41rGHmdFHvfEMpP5m9Io= 2024-04-12T03:28:48.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+NJPHeG41rGHmdFHvfEMpP5m9Io=', 'Content-Length': '0'} 2024-04-12T03:28:48.166 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000eb03eeaef24c1a5-006618aa70-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.206 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:48.206 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:48.207 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:48.207 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:48.207 DEBUG:boto:Method: GET 2024-04-12T03:28:48.207 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:48.207 DEBUG:boto:Data: 2024-04-12T03:28:48.207 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.207 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:48.207 DEBUG:boto:Port: 8001 2024-04-12T03:28:48.207 DEBUG:boto:Params: {} 2024-04-12T03:28:48.207 DEBUG:boto:Token: None 2024-04-12T03:28:48.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/ 2024-04-12T03:28:48.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QwDOTKzPbYQJ3s1GCbC9RWoF5B0= 2024-04-12T03:28:48.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QwDOTKzPbYQJ3s1GCbC9RWoF5B0=', 'Content-Length': '0'} 2024-04-12T03:28:48.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000472ed333099d755a-006618aa70-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.211 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:48.211 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:48.212 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:48.212 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:48.212 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:48.212 DEBUG:boto:Method: GET 2024-04-12T03:28:48.212 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:48.212 DEBUG:boto:Data: 2024-04-12T03:28:48.212 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.212 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:48.212 DEBUG:boto:Port: 8000 2024-04-12T03:28:48.212 DEBUG:boto:Params: {} 2024-04-12T03:28:48.212 DEBUG:boto:Token: None 2024-04-12T03:28:48.213 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/ 2024-04-12T03:28:48.213 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QwDOTKzPbYQJ3s1GCbC9RWoF5B0= 2024-04-12T03:28:48.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QwDOTKzPbYQJ3s1GCbC9RWoF5B0=', 'Content-Length': '0'} 2024-04-12T03:28:48.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6cac8041afbac6f-006618aa70-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.215 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:48.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:48.216 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.216 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.216 DEBUG:boto:path=/dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.216 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.216 DEBUG:boto:Method: GET 2024-04-12T03:28:48.216 DEBUG:boto:Path: /dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.216 DEBUG:boto:Data: 2024-04-12T03:28:48.216 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.216 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:48.217 DEBUG:boto:Port: 8001 2024-04-12T03:28:48.217 DEBUG:boto:Params: {} 2024-04-12T03:28:48.217 DEBUG:boto:Token: None 2024-04-12T03:28:48.217 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.217 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vTRGzPeQK/f6MezTf4CU9NQX4GQ= 2024-04-12T03:28:48.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vTRGzPeQK/f6MezTf4CU9NQX4GQ=', 'Content-Length': '0'} 2024-04-12T03:28:48.219 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'm0JnneswTZ9PHcgOelxxc25lL6S0Dz8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003e38f69422a6078c-006618aa70-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.219 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.219 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.220 DEBUG:boto:path=/dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.220 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.220 DEBUG:boto:Method: GET 2024-04-12T03:28:48.220 DEBUG:boto:Path: /dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.220 DEBUG:boto:Data: 2024-04-12T03:28:48.220 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.220 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:48.220 DEBUG:boto:Port: 8000 2024-04-12T03:28:48.220 DEBUG:boto:Params: {} 2024-04-12T03:28:48.220 DEBUG:boto:Token: None 2024-04-12T03:28:48.220 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.221 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vTRGzPeQK/f6MezTf4CU9NQX4GQ= 2024-04-12T03:28:48.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vTRGzPeQK/f6MezTf4CU9NQX4GQ=', 'Content-Length': '0'} 2024-04-12T03:28:48.223 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'm0JnneswTZ9PHcgOelxxc25lL6S0Dz8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7d4902869392ef5-006618aa70-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.266 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.267 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.267 DEBUG:boto:path=/dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.267 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.267 DEBUG:boto:Method: HEAD 2024-04-12T03:28:48.267 DEBUG:boto:Path: /dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.267 DEBUG:boto:Data: 2024-04-12T03:28:48.267 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.267 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:48.267 DEBUG:boto:Port: 8001 2024-04-12T03:28:48.267 DEBUG:boto:Params: {} 2024-04-12T03:28:48.268 DEBUG:boto:Token: None 2024-04-12T03:28:48.268 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.268 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1bSHcXRcUy2UK37fzorwZ6aSiio= 2024-04-12T03:28:48.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1bSHcXRcUy2UK37fzorwZ6aSiio=', 'Content-Length': '0'} 2024-04-12T03:28:48.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'm0JnneswTZ9PHcgOelxxc25lL6S0Dz8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048d9f01814dd0042-006618aa70-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.270 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.270 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.270 DEBUG:boto:path=/dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.270 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.270 DEBUG:boto:Method: HEAD 2024-04-12T03:28:48.270 DEBUG:boto:Path: /dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.271 DEBUG:boto:Data: 2024-04-12T03:28:48.271 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.271 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:48.271 DEBUG:boto:Port: 8000 2024-04-12T03:28:48.271 DEBUG:boto:Params: {} 2024-04-12T03:28:48.271 DEBUG:boto:Token: None 2024-04-12T03:28:48.271 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.271 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1bSHcXRcUy2UK37fzorwZ6aSiio= 2024-04-12T03:28:48.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1bSHcXRcUy2UK37fzorwZ6aSiio=', 'Content-Length': '0'} 2024-04-12T03:28:48.273 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'm0JnneswTZ9PHcgOelxxc25lL6S0Dz8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c349843911f2e3d-006618aa70-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:48.273 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.273 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.273 DEBUG:boto:path=/dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.273 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.274 DEBUG:boto:Method: GET 2024-04-12T03:28:48.274 DEBUG:boto:Path: /dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.274 DEBUG:boto:Data: 2024-04-12T03:28:48.274 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.274 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:48.274 DEBUG:boto:Port: 8001 2024-04-12T03:28:48.274 DEBUG:boto:Params: {} 2024-04-12T03:28:48.274 DEBUG:boto:Token: None 2024-04-12T03:28:48.274 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.274 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vTRGzPeQK/f6MezTf4CU9NQX4GQ= 2024-04-12T03:28:48.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vTRGzPeQK/f6MezTf4CU9NQX4GQ=', 'Content-Length': '0'} 2024-04-12T03:28:48.276 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'm0JnneswTZ9PHcgOelxxc25lL6S0Dz8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000428d9fb435aa90e4-006618aa70-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.318 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.319 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.319 DEBUG:boto:path=/dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.319 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.319 DEBUG:boto:Method: GET 2024-04-12T03:28:48.319 DEBUG:boto:Path: /dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.319 DEBUG:boto:Data: 2024-04-12T03:28:48.319 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.319 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:48.319 DEBUG:boto:Port: 8000 2024-04-12T03:28:48.319 DEBUG:boto:Params: {} 2024-04-12T03:28:48.319 DEBUG:boto:Token: None 2024-04-12T03:28:48.320 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj?versionId=m0JnneswTZ9PHcgOelxxc25lL6S0Dz8 2024-04-12T03:28:48.320 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vTRGzPeQK/f6MezTf4CU9NQX4GQ= 2024-04-12T03:28:48.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vTRGzPeQK/f6MezTf4CU9NQX4GQ=', 'Content-Length': '0'} 2024-04-12T03:28:48.322 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:48 GMT'), ('x-amz-version-id', 'm0JnneswTZ9PHcgOelxxc25lL6S0Dz8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc066a57a2499a16-006618aa70-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.322 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:48.322 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:48.323 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:48.323 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:48.323 DEBUG:boto:Method: GET 2024-04-12T03:28:48.323 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:48.323 DEBUG:boto:Data: 2024-04-12T03:28:48.323 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.323 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:48.323 DEBUG:boto:Port: 8001 2024-04-12T03:28:48.323 DEBUG:boto:Params: {} 2024-04-12T03:28:48.323 DEBUG:boto:Token: None 2024-04-12T03:28:48.324 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/ 2024-04-12T03:28:48.324 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QwDOTKzPbYQJ3s1GCbC9RWoF5B0= 2024-04-12T03:28:48.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QwDOTKzPbYQJ3s1GCbC9RWoF5B0=', 'Content-Length': '0'} 2024-04-12T03:28:48.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b4e06f6321f38f2-006618aa70-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.326 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:48.327 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:48.327 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:48.327 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:48.327 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:48.327 DEBUG:boto:Method: GET 2024-04-12T03:28:48.327 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:48.327 DEBUG:boto:Data: 2024-04-12T03:28:48.327 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.327 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:48.327 DEBUG:boto:Port: 8000 2024-04-12T03:28:48.327 DEBUG:boto:Params: {} 2024-04-12T03:28:48.328 DEBUG:boto:Token: None 2024-04-12T03:28:48.328 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/ 2024-04-12T03:28:48.328 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QwDOTKzPbYQJ3s1GCbC9RWoF5B0= 2024-04-12T03:28:48.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QwDOTKzPbYQJ3s1GCbC9RWoF5B0=', 'Content-Length': '0'} 2024-04-12T03:28:48.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d42ed81d4e3e5e5-006618aa70-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.331 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:48.331 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.331 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.331 DEBUG:boto:Method: HEAD 2024-04-12T03:28:48.331 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:48.331 DEBUG:boto:Data: 2024-04-12T03:28:48.331 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.332 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:48.332 DEBUG:boto:Port: 8001 2024-04-12T03:28:48.332 DEBUG:boto:Params: {} 2024-04-12T03:28:48.332 DEBUG:boto:Token: None 2024-04-12T03:28:48.332 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj 2024-04-12T03:28:48.332 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Rrsy8yXwftnYXEHtQ+tSSF5eHx8= 2024-04-12T03:28:48.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Rrsy8yXwftnYXEHtQ+tSSF5eHx8=', 'Content-Length': '0'} 2024-04-12T03:28:48.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8fc9a8b1c1d8e4c-006618aa70-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.336 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.336 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.336 DEBUG:boto:Method: HEAD 2024-04-12T03:28:48.336 DEBUG:boto:Path: /dkyges-15/obj 2024-04-12T03:28:48.336 DEBUG:boto:Data: 2024-04-12T03:28:48.336 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.336 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:48.336 DEBUG:boto:Port: 8000 2024-04-12T03:28:48.336 DEBUG:boto:Params: {} 2024-04-12T03:28:48.337 DEBUG:boto:Token: None 2024-04-12T03:28:48.337 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj 2024-04-12T03:28:48.337 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Rrsy8yXwftnYXEHtQ+tSSF5eHx8= 2024-04-12T03:28:48.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Rrsy8yXwftnYXEHtQ+tSSF5eHx8=', 'Content-Length': '0'} 2024-04-12T03:28:48.340 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000232b9e517ca01f6e-006618aa70-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:28:48.340 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.340 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.340 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:48.341 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:48.341 DEBUG:boto:Method: GET 2024-04-12T03:28:48.341 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:48.341 DEBUG:boto:Data: 2024-04-12T03:28:48.341 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.341 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:48.341 DEBUG:boto:Port: 8001 2024-04-12T03:28:48.341 DEBUG:boto:Params: {} 2024-04-12T03:28:48.341 DEBUG:boto:Token: None 2024-04-12T03:28:48.341 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:48.341 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+NJPHeG41rGHmdFHvfEMpP5m9Io= 2024-04-12T03:28:48.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+NJPHeG41rGHmdFHvfEMpP5m9Io=', 'Content-Length': '0'} 2024-04-12T03:28:48.343 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fc40fe98ad15b81c-006618aa70-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.386 DEBUG:boto:path=/dkyges-15/obj 2024-04-12T03:28:48.387 DEBUG:boto:auth_path=/dkyges-15/obj 2024-04-12T03:28:48.387 DEBUG:boto:path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:48.387 DEBUG:boto:auth_path=/dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:48.387 DEBUG:boto:Method: GET 2024-04-12T03:28:48.387 DEBUG:boto:Path: /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:48.387 DEBUG:boto:Data: 2024-04-12T03:28:48.387 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.387 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:48.387 DEBUG:boto:Port: 8000 2024-04-12T03:28:48.387 DEBUG:boto:Params: {} 2024-04-12T03:28:48.387 DEBUG:boto:Token: None 2024-04-12T03:28:48.388 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/obj?versionId=DCRUzLijiY355CP0VujBj35OATE10fg 2024-04-12T03:28:48.388 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+NJPHeG41rGHmdFHvfEMpP5m9Io= 2024-04-12T03:28:48.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+NJPHeG41rGHmdFHvfEMpP5m9Io=', 'Content-Length': '0'} 2024-04-12T03:28:48.390 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:26:49 GMT'), ('x-amz-version-id', 'DCRUzLijiY355CP0VujBj35OATE10fg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000540b47b590f5f7c4-006618aa70-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.430 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:48.430 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:48.431 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:48.431 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:48.431 DEBUG:boto:Method: GET 2024-04-12T03:28:48.431 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:48.431 DEBUG:boto:Data: 2024-04-12T03:28:48.431 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.431 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:28:48.431 DEBUG:boto:Port: 8001 2024-04-12T03:28:48.431 DEBUG:boto:Params: {} 2024-04-12T03:28:48.431 DEBUG:boto:Token: None 2024-04-12T03:28:48.432 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/ 2024-04-12T03:28:48.432 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QwDOTKzPbYQJ3s1GCbC9RWoF5B0= 2024-04-12T03:28:48.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QwDOTKzPbYQJ3s1GCbC9RWoF5B0=', 'Content-Length': '0'} 2024-04-12T03:28:48.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e4b1dba31ed6822-006618aa70-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.434 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:48.435 DEBUG:boto:path=/dkyges-15/ 2024-04-12T03:28:48.435 DEBUG:boto:auth_path=/dkyges-15/ 2024-04-12T03:28:48.435 DEBUG:boto:path=/dkyges-15/?prefix=obj 2024-04-12T03:28:48.435 DEBUG:boto:auth_path=/dkyges-15/?prefix=obj 2024-04-12T03:28:48.435 DEBUG:boto:Method: GET 2024-04-12T03:28:48.435 DEBUG:boto:Path: /dkyges-15/?prefix=obj 2024-04-12T03:28:48.435 DEBUG:boto:Data: 2024-04-12T03:28:48.435 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.435 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:48.436 DEBUG:boto:Port: 8000 2024-04-12T03:28:48.436 DEBUG:boto:Params: {} 2024-04-12T03:28:48.436 DEBUG:boto:Token: None 2024-04-12T03:28:48.436 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:28:48 GMT /dkyges-15/ 2024-04-12T03:28:48.436 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QwDOTKzPbYQJ3s1GCbC9RWoF5B0= 2024-04-12T03:28:48.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QwDOTKzPbYQJ3s1GCbC9RWoF5B0=', 'Content-Length': '0'} 2024-04-12T03:28:48.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcc4cdcac6363317-006618aa70-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.482 DEBUG:boto:b'dkyges-15obj1000falseobj2024-04-12T03:26:49.201Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:28:48.483 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-15 zones={a2, a1} 2024-04-12T03:28:48.483 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-12T03:28:48.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:48.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:48.484 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:48.542 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:48.542 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:48.596 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:28:48.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:48.596 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-12T03:28:48.597 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:20:22.704353Z" 2024-04-12T03:28:48.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.597 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:48.597 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:28:48.597 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:20:22.705876Z" 2024-04-12T03:28:48.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.597 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:48.597 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004029", 2024-04-12T03:28:48.597 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:28:04.729440Z" 2024-04-12T03:28:48.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:48.598 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004597", 2024-04-12T03:28:48.598 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:26:58.723769Z" 2024-04-12T03:28:48.598 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:48.598 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:28:48.599 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000002622', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000004029', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:28:48.599 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:28:48.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:48.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:48.600 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:48.657 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:48.657 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:48.705 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:28:48.705 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:28:48.705 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:28:48.705 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:28:48.705 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:28:48.705 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 16340188497730315678 2024-04-12T03:28:48.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.705 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:28:48.706 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:48.706 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:28:48.706 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:28:48.706 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:28:48.706 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-12T03:28:48.706 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:28:48.706 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:28:48.706 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:28:48.706 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.704353Z" 2024-04-12T03:28:48.706 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:48.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.707 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:48.707 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:28:48.707 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:28:48.707 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:28:48.707 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:28:48.707 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:28:48.707 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:28:48.707 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:28:48.707 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.705876Z" 2024-04-12T03:28:48.707 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:48.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:28:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:28:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:28:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004029", 2024-04-12T03:28:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:28:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:28:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:28:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:28:04.729440Z" 2024-04-12T03:28:48.708 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:28:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:28:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:28:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004597", 2024-04-12T03:28:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:28:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:28:48.709 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:28:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:26:58.723769Z" 2024-04-12T03:28:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:28:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:28:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:28:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:28:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.710 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:28:48Z" 2024-04-12T03:28:48.711 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:28:48.712 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16340188497730315678 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002622", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.704353Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.705876Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004029", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:04.729440Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004597", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:26:58.723769Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:28:48Z" } 2024-04-12T03:28:48.712 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002622', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.704353Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.705876Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004029', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:04.729440Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004597', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:26:58.723769Z'}}] 2024-04-12T03:28:48.712 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002622', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000004029', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:28:48.712 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002622', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000004029', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:28:48.712 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:28:48.712 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-12T03:28:48.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:48.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:48.713 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:48.766 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:48.767 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:48.819 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:28:48.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:48.819 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-12T03:28:48.819 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:20:22.849581Z" 2024-04-12T03:28:48.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:48.819 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:28:48.819 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:20:22.851157Z" 2024-04-12T03:28:48.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:48.820 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004311", 2024-04-12T03:28:48.820 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:28:26.872065Z" 2024-04-12T03:28:48.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:48.820 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004174", 2024-04-12T03:28:48.820 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:26:36.864390Z" 2024-04-12T03:28:48.820 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:48.820 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:28:48.821 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000002622', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000004311', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:28:48.822 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:28:48.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:48.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:48.822 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:48.879 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:48.879 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:48.925 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:28:48.926 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:28:48.926 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:28:48.926 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:28:48.926 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:28:48.926 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5944041172307441597 2024-04-12T03:28:48.926 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.926 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:28:48.926 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:48.926 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:28:48.926 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:28:48.926 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:28:48.927 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-12T03:28:48.927 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:28:48.927 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:28:48.927 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:28:48.927 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.849581Z" 2024-04-12T03:28:48.927 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:48.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:48.927 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:28:48.927 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:28:48.927 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:28:48.928 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:28:48.928 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:28:48.928 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:28:48.928 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:28:48.928 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.851157Z" 2024-04-12T03:28:48.928 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:48.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:48.928 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:28:48.928 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:28:48.928 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:28:48.928 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004311", 2024-04-12T03:28:48.929 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:28:48.929 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:28:48.929 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:28:48.929 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:28:26.872065Z" 2024-04-12T03:28:48.929 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:48.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.929 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:48.929 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:28:48.929 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:28:48.929 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:28:48.929 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004174", 2024-04-12T03:28:48.929 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:28:48.930 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:28:48.930 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:28:48.930 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:26:36.864390Z" 2024-04-12T03:28:48.930 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:48.930 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:48.930 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:28:48.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.930 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:28:48.930 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:28:48.930 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:28:48.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:48.931 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:28:48Z" 2024-04-12T03:28:48.931 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:28:48.931 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5944041172307441597 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002622", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.849581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.851157Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004311", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:26.872065Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004174", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:26:36.864390Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:28:48Z" } 2024-04-12T03:28:48.932 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002622', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.849581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.851157Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004311', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:26.872065Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004174', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:26:36.864390Z'}}] 2024-04-12T03:28:48.932 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002622', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000004311', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:28:48.932 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002622', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000004311', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:28:48.932 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:28:48.932 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2024-04-12T03:28:48.934 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:28:48.934 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:28:48.934 DEBUG:boto:Method: PUT 2024-04-12T03:28:48.934 DEBUG:boto:Path: /dkyges-16/ 2024-04-12T03:28:48.934 DEBUG:boto:Data: 2024-04-12T03:28:48.934 DEBUG:boto:Headers: {} 2024-04-12T03:28:48.934 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:48.934 DEBUG:boto:Port: 8000 2024-04-12T03:28:48.934 DEBUG:boto:Params: {} 2024-04-12T03:28:48.935 DEBUG:boto:Token: None 2024-04-12T03:28:48.935 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:28:48 GMT /dkyges-16/ 2024-04-12T03:28:48.935 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GmUGnxin3oijvCvagShmrFmwbDw= 2024-04-12T03:28:48.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GmUGnxin3oijvCvagShmrFmwbDw=', 'Content-Length': '0'} 2024-04-12T03:28:48.949 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000719ee4ae26f4e115-006618aa70-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:48.949 DEBUG:rgw_multi.tests:created bucket=dkyges-16 2024-04-12T03:28:48.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:48.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:48.950 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:49.004 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:49.004 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:49.040 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:28:49.055 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:28:49.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:49.056 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:28:49.056 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:28:49.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:49.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:49.056 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:28:49.056 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:28:49.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:49.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:49.056 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:28:49.056 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:28:49.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:49.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:49.057 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892528.947677_159.1", 2024-04-12T03:28:49.057 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:28:48.947677Z" 2024-04-12T03:28:49.057 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:49.057 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:28:49.058 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892528.947677_159.1'} 2024-04-12T03:28:49.058 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:28:49.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:49.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:28:49.059 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:28:49.116 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:49.116 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:49.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:49.149+0000 7fbf9e879780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:28:49.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:49.151+0000 7fbf9e879780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:28:49.165 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:28:49.165 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:28:49.165 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:28:49.165 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:28:49.165 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:28:49.166 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:28:49.166 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:28:49.166 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:49.166 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:28:49.166 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:49.166 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:28:49.166 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:28:49.166 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:28:49.166 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:28:49.166 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:28:49.166 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:28:49.166 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:28:49.166 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:28:49.167 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:28:49.167 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:49.167 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:49.167 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:49.167 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:28:49.167 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:28:49.167 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:28:49.167 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:28:49.167 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:28:49.167 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:28:49.167 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:28:49.167 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:28:49.168 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:28:49.168 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:49.168 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:49.168 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:49.168 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:28:49.168 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:28:49.168 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:28:49.168 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:28:49.168 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:28:49.168 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:28:49.168 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:28:49.168 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:28:49.168 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:28:49.169 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:49.169 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:49.169 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:28:49.169 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:28:49.169 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:28:49.169 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:28:49.169 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892528.947677_159.1", 2024-04-12T03:28:49.169 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:28:49.169 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:28:49.169 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:28:49.169 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:28:48.947677Z", 2024-04-12T03:28:49.169 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:28:49.169 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:49.169 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:28:49.170 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:28:49.170 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:49.170 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:28:49.170 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:28:49.170 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:28:49.170 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:28:49.170 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:28:49Z" 2024-04-12T03:28:49.170 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:28:49.171 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892528.947677_159.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:28:48.947677Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:28:49Z" } 2024-04-12T03:28:49.171 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892528.947677_159.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:28:48.947677Z', 'realm_epoch': 2}}] 2024-04-12T03:28:49.171 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892528.947677_159.1'} 2024-04-12T03:28:49.171 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892528.947677_159.1'} 2024-04-12T03:28:49.171 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:28:49.172 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:28:49.172 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:28:49.172 DEBUG:boto:Method: HEAD 2024-04-12T03:28:49.172 DEBUG:boto:Path: /dkyges-16/ 2024-04-12T03:28:49.172 DEBUG:boto:Data: 2024-04-12T03:28:49.172 DEBUG:boto:Headers: {} 2024-04-12T03:28:49.172 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:49.172 DEBUG:boto:Port: 8000 2024-04-12T03:28:49.172 DEBUG:boto:Params: {} 2024-04-12T03:28:49.172 DEBUG:boto:Token: None 2024-04-12T03:28:49.172 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:28:49 GMT /dkyges-16/ 2024-04-12T03:28:49.173 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pME9XmpAeG3ePNObc/Iw8c66LwY= 2024-04-12T03:28:49.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:28:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pME9XmpAeG3ePNObc/Iw8c66LwY=', 'Content-Length': '0'} 2024-04-12T03:28:49.175 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e1636d27079e072-006618aa71-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:49.175 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:28:49.175 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:28:49.175 DEBUG:boto:Method: PUT 2024-04-12T03:28:49.176 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:28:49.176 DEBUG:boto:Data: 2024-04-12T03:28:49.176 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:28:49.176 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:28:49.176 DEBUG:boto:Port: 8000 2024-04-12T03:28:49.176 DEBUG:boto:Params: {} 2024-04-12T03:28:49.176 DEBUG:boto:Token: None 2024-04-12T03:28:49.176 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:28:49 GMT /dkyges-16/obj 2024-04-12T03:28:49.176 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/cNTO2ueiQw3x9i7SAYH783K2xc= 2024-04-12T03:28:49.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:28:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/cNTO2ueiQw3x9i7SAYH783K2xc='} 2024-04-12T03:28:49.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000222f4d2796282a47-006618aa71-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:28:49.183 DEBUG:rgw_multi.tests:created initial version id=None 2024-04-12T03:28:49.183 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-16 2024-04-12T03:28:49.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:28:49.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:28:49.184 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:28:49.245 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:28:49.245 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:28:49.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:49.281+0000 7f21effdc780 1 waiting to reach incremental sync.. 2024-04-12T03:28:54.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:54.283+0000 7f21effdc780 1 waiting to reach incremental sync.. 2024-04-12T03:28:59.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:28:59.284+0000 7f21effdc780 1 waiting to reach incremental sync.. 2024-04-12T03:29:04.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:29:04.283+0000 7f21effdc780 1 waiting to reach incremental sync.. 2024-04-12T03:29:09.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:29:09.285+0000 7f21effdc780 1 bucket sync caught up with source: 2024-04-12T03:29:09.287 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000001.30.6, , ] 2024-04-12T03:29:09.287 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000001.30.6, , ] 2024-04-12T03:29:09.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:29:09.285+0000 7f21effdc780 0 bucket checkpoint complete 2024-04-12T03:29:09.303 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-16 2024-04-12T03:29:09.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:29:09.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:29:09.304 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:29:09.368 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:29:09.368 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:29:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:29:09.405+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:29:14.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:29:14.406+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:29:19.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:29:19.407+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:29:24.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:29:24.407+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:29:29.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:29:29.408+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:29:34.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:29:34.408+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:29:39.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:29:39.409+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:29:44.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:29:44.410+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:29:49.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:29:49.411+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:29:54.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:29:54.411+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:29:59.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:29:59.412+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:30:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:30:04.413+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:30:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:30:09.414+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:30:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:30:14.414+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:30:19.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:30:19.415+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:30:24.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:30:24.415+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:30:29.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:30:29.416+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:30:34.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:30:34.416+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:30:39.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:30:39.417+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:30:44.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:30:44.418+0000 7f98836ef780 1 waiting to reach incremental sync.. 2024-04-12T03:30:49.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:30:49.419+0000 7f98836ef780 1 bucket sync caught up with source: 2024-04-12T03:30:49.421 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000001.20.6, , ] 2024-04-12T03:30:49.421 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000001.20.6, , ] 2024-04-12T03:30:49.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:30:49.419+0000 7f98836ef780 0 bucket checkpoint complete 2024-04-12T03:30:49.438 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-16 zones={a2, a1} 2024-04-12T03:30:49.438 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:30:49.438 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:30:49.438 DEBUG:boto:Method: HEAD 2024-04-12T03:30:49.438 DEBUG:boto:Path: /dkyges-16/ 2024-04-12T03:30:49.439 DEBUG:boto:Data: 2024-04-12T03:30:49.439 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.439 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:30:49.439 DEBUG:boto:Port: 8001 2024-04-12T03:30:49.439 DEBUG:boto:Params: {} 2024-04-12T03:30:49.439 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:30:49.440 DEBUG:boto:Token: None 2024-04-12T03:30:49.440 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/ 2024-04-12T03:30:49.440 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D5UJjjrGP45p+oe+emdlwrSbYjU= 2024-04-12T03:30:49.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D5UJjjrGP45p+oe+emdlwrSbYjU=', 'Content-Length': '0'} 2024-04-12T03:30:49.457 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003529ce0dc28773e0-006618aae9-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.457 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:30:49.458 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:30:49.458 DEBUG:boto:Method: HEAD 2024-04-12T03:30:49.458 DEBUG:boto:Path: /dkyges-16/ 2024-04-12T03:30:49.458 DEBUG:boto:Data: 2024-04-12T03:30:49.458 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.458 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:30:49.458 DEBUG:boto:Port: 8000 2024-04-12T03:30:49.458 DEBUG:boto:Params: {} 2024-04-12T03:30:49.458 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:30:49.459 DEBUG:boto:Token: None 2024-04-12T03:30:49.459 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/ 2024-04-12T03:30:49.459 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D5UJjjrGP45p+oe+emdlwrSbYjU= 2024-04-12T03:30:49.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D5UJjjrGP45p+oe+emdlwrSbYjU=', 'Content-Length': '0'} 2024-04-12T03:30:49.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f04bb52233968a71-006618aae9-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.471 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:30:49.471 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:30:49.471 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:30:49.471 DEBUG:boto:path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:30:49.471 DEBUG:boto:auth_path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:30:49.472 DEBUG:boto:Method: GET 2024-04-12T03:30:49.472 DEBUG:boto:Path: /dkyges-16/?versions&max-keys=999 2024-04-12T03:30:49.472 DEBUG:boto:Data: 2024-04-12T03:30:49.472 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.472 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:30:49.472 DEBUG:boto:Port: 8001 2024-04-12T03:30:49.472 DEBUG:boto:Params: {} 2024-04-12T03:30:49.472 DEBUG:boto:Token: None 2024-04-12T03:30:49.472 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/?versions 2024-04-12T03:30:49.472 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3F6v/pOCo+jn5vxbpc9th3dZdFU= 2024-04-12T03:30:49.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3F6v/pOCo+jn5vxbpc9th3dZdFU=', 'Content-Length': '0'} 2024-04-12T03:30:49.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef34d855affe10b6-006618aae9-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.475 DEBUG:boto:b'dkyges-16999falseobjnulltrue2024-04-12T03:28:49.178Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:49.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:30:49.476 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:30:49.476 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:30:49.476 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:30:49.476 DEBUG:boto:path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:30:49.476 DEBUG:boto:auth_path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:30:49.477 DEBUG:boto:Method: GET 2024-04-12T03:30:49.477 DEBUG:boto:Path: /dkyges-16/?versions&max-keys=999 2024-04-12T03:30:49.477 DEBUG:boto:Data: 2024-04-12T03:30:49.477 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.477 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:30:49.477 DEBUG:boto:Port: 8000 2024-04-12T03:30:49.477 DEBUG:boto:Params: {} 2024-04-12T03:30:49.477 DEBUG:boto:Token: None 2024-04-12T03:30:49.477 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/?versions 2024-04-12T03:30:49.478 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3F6v/pOCo+jn5vxbpc9th3dZdFU= 2024-04-12T03:30:49.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3F6v/pOCo+jn5vxbpc9th3dZdFU=', 'Content-Length': '0'} 2024-04-12T03:30:49.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000503e7cd1abf0a1f6-006618aae9-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.480 DEBUG:boto:b'dkyges-16999falseobjnulltrue2024-04-12T03:28:49.178Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:49.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:30:49.480 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:30:49.481 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:30:49.481 DEBUG:boto:path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:30:49.481 DEBUG:boto:auth_path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:30:49.481 DEBUG:boto:Method: GET 2024-04-12T03:30:49.481 DEBUG:boto:Path: /dkyges-16/?versions&max-keys=999 2024-04-12T03:30:49.481 DEBUG:boto:Data: 2024-04-12T03:30:49.481 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.481 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:30:49.481 DEBUG:boto:Port: 8001 2024-04-12T03:30:49.481 DEBUG:boto:Params: {} 2024-04-12T03:30:49.481 DEBUG:boto:Token: None 2024-04-12T03:30:49.482 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/?versions 2024-04-12T03:30:49.482 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3F6v/pOCo+jn5vxbpc9th3dZdFU= 2024-04-12T03:30:49.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3F6v/pOCo+jn5vxbpc9th3dZdFU=', 'Content-Length': '0'} 2024-04-12T03:30:49.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000294d4da7ff161839-006618aae9-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.484 DEBUG:boto:b'dkyges-16999falseobjnulltrue2024-04-12T03:28:49.178Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:49.485 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:30:49.485 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:30:49.485 DEBUG:boto:path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:30:49.485 DEBUG:boto:auth_path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:30:49.485 DEBUG:boto:Method: GET 2024-04-12T03:30:49.485 DEBUG:boto:Path: /dkyges-16/?versions&max-keys=999 2024-04-12T03:30:49.485 DEBUG:boto:Data: 2024-04-12T03:30:49.485 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.485 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:30:49.486 DEBUG:boto:Port: 8000 2024-04-12T03:30:49.486 DEBUG:boto:Params: {} 2024-04-12T03:30:49.486 DEBUG:boto:Token: None 2024-04-12T03:30:49.486 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/?versions 2024-04-12T03:30:49.486 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3F6v/pOCo+jn5vxbpc9th3dZdFU= 2024-04-12T03:30:49.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3F6v/pOCo+jn5vxbpc9th3dZdFU=', 'Content-Length': '0'} 2024-04-12T03:30:49.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b428a9d2e2c70b4-006618aae9-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.489 DEBUG:boto:b'dkyges-16999falseobjnulltrue2024-04-12T03:28:49.178Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:49.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:30:49.489 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:30:49.490 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:30:49.490 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:30:49.490 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:30:49.490 DEBUG:boto:Method: GET 2024-04-12T03:30:49.490 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:30:49.490 DEBUG:boto:Data: 2024-04-12T03:30:49.490 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.490 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:30:49.490 DEBUG:boto:Port: 8001 2024-04-12T03:30:49.490 DEBUG:boto:Params: {} 2024-04-12T03:30:49.490 DEBUG:boto:Token: None 2024-04-12T03:30:49.491 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:30:49.491 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OnMJVeyWSNxP3dFFtUpwzVXgJfs= 2024-04-12T03:30:49.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OnMJVeyWSNxP3dFFtUpwzVXgJfs=', 'Content-Length': '0'} 2024-04-12T03:30:49.493 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000058f6885a5f0b1b82-006618aae9-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.493 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:30:49.493 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:30:49.493 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:30:49.493 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:30:49.493 DEBUG:boto:Method: GET 2024-04-12T03:30:49.494 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:30:49.494 DEBUG:boto:Data: 2024-04-12T03:30:49.494 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.494 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:30:49.494 DEBUG:boto:Port: 8000 2024-04-12T03:30:49.494 DEBUG:boto:Params: {} 2024-04-12T03:30:49.494 DEBUG:boto:Token: None 2024-04-12T03:30:49.494 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:30:49.494 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OnMJVeyWSNxP3dFFtUpwzVXgJfs= 2024-04-12T03:30:49.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OnMJVeyWSNxP3dFFtUpwzVXgJfs=', 'Content-Length': '0'} 2024-04-12T03:30:49.497 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aaaef2295c7ff8c3-006618aae9-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.497 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:30:49.497 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:30:49.497 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:30:49.497 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:30:49.497 DEBUG:boto:Method: HEAD 2024-04-12T03:30:49.497 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:30:49.497 DEBUG:boto:Data: 2024-04-12T03:30:49.497 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.498 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:30:49.498 DEBUG:boto:Port: 8001 2024-04-12T03:30:49.498 DEBUG:boto:Params: {} 2024-04-12T03:30:49.498 DEBUG:boto:Token: None 2024-04-12T03:30:49.498 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:30:49.498 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8k6BQIMtPsu0kkWwFb9PHz6FQnM= 2024-04-12T03:30:49.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8k6BQIMtPsu0kkWwFb9PHz6FQnM=', 'Content-Length': '0'} 2024-04-12T03:30:49.500 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000026571e1629949efb-006618aae9-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.500 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:30:49.500 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:30:49.500 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:30:49.500 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:30:49.501 DEBUG:boto:Method: HEAD 2024-04-12T03:30:49.501 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:30:49.501 DEBUG:boto:Data: 2024-04-12T03:30:49.501 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.501 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:30:49.501 DEBUG:boto:Port: 8000 2024-04-12T03:30:49.501 DEBUG:boto:Params: {} 2024-04-12T03:30:49.501 DEBUG:boto:Token: None 2024-04-12T03:30:49.501 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:30:49.501 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8k6BQIMtPsu0kkWwFb9PHz6FQnM= 2024-04-12T03:30:49.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8k6BQIMtPsu0kkWwFb9PHz6FQnM=', 'Content-Length': '0'} 2024-04-12T03:30:49.503 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ddc156bf894ac774-006618aae9-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:30:49.503 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:30:49.504 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:30:49.504 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:30:49.504 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:30:49.504 DEBUG:boto:Method: GET 2024-04-12T03:30:49.504 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:30:49.504 DEBUG:boto:Data: 2024-04-12T03:30:49.504 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.504 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:30:49.504 DEBUG:boto:Port: 8001 2024-04-12T03:30:49.504 DEBUG:boto:Params: {} 2024-04-12T03:30:49.504 DEBUG:boto:Token: None 2024-04-12T03:30:49.505 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:30:49.505 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OnMJVeyWSNxP3dFFtUpwzVXgJfs= 2024-04-12T03:30:49.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OnMJVeyWSNxP3dFFtUpwzVXgJfs=', 'Content-Length': '0'} 2024-04-12T03:30:49.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a2b0eef91171330b-006618aae9-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.506 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:30:49.507 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:30:49.507 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:30:49.507 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:30:49.507 DEBUG:boto:Method: GET 2024-04-12T03:30:49.507 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:30:49.507 DEBUG:boto:Data: 2024-04-12T03:30:49.507 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.507 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:30:49.507 DEBUG:boto:Port: 8000 2024-04-12T03:30:49.508 DEBUG:boto:Params: {} 2024-04-12T03:30:49.508 DEBUG:boto:Token: None 2024-04-12T03:30:49.508 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:30:49.508 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OnMJVeyWSNxP3dFFtUpwzVXgJfs= 2024-04-12T03:30:49.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OnMJVeyWSNxP3dFFtUpwzVXgJfs=', 'Content-Length': '0'} 2024-04-12T03:30:49.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000005fd800b73cd2d9e-006618aae9-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.510 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:30:49.510 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:30:49.510 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:30:49.510 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:30:49.510 DEBUG:boto:Method: GET 2024-04-12T03:30:49.510 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:30:49.511 DEBUG:boto:Data: 2024-04-12T03:30:49.511 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.511 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:30:49.511 DEBUG:boto:Port: 8001 2024-04-12T03:30:49.511 DEBUG:boto:Params: {} 2024-04-12T03:30:49.511 DEBUG:boto:Token: None 2024-04-12T03:30:49.511 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/ 2024-04-12T03:30:49.511 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RV1wUrSWWvxwFGfJwspdBK8edSw= 2024-04-12T03:30:49.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RV1wUrSWWvxwFGfJwspdBK8edSw=', 'Content-Length': '0'} 2024-04-12T03:30:49.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000688653bd73afe0bc-006618aae9-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.554 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:28:49.178Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:49.555 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:30:49.555 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:30:49.555 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:30:49.555 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:30:49.555 DEBUG:boto:Method: GET 2024-04-12T03:30:49.555 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:30:49.555 DEBUG:boto:Data: 2024-04-12T03:30:49.555 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.555 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:30:49.556 DEBUG:boto:Port: 8000 2024-04-12T03:30:49.556 DEBUG:boto:Params: {} 2024-04-12T03:30:49.556 DEBUG:boto:Token: None 2024-04-12T03:30:49.556 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/ 2024-04-12T03:30:49.556 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RV1wUrSWWvxwFGfJwspdBK8edSw= 2024-04-12T03:30:49.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RV1wUrSWWvxwFGfJwspdBK8edSw=', 'Content-Length': '0'} 2024-04-12T03:30:49.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004493989859c130df-006618aae9-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.559 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:28:49.178Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:49.559 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:30:49.559 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:30:49.559 DEBUG:boto:Method: HEAD 2024-04-12T03:30:49.559 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:30:49.559 DEBUG:boto:Data: 2024-04-12T03:30:49.559 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.560 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:30:49.560 DEBUG:boto:Port: 8001 2024-04-12T03:30:49.560 DEBUG:boto:Params: {} 2024-04-12T03:30:49.560 DEBUG:boto:Token: None 2024-04-12T03:30:49.560 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/obj 2024-04-12T03:30:49.560 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KcoILpCDkYFjAXliyaPExsthjiE= 2024-04-12T03:30:49.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KcoILpCDkYFjAXliyaPExsthjiE=', 'Content-Length': '0'} 2024-04-12T03:30:49.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000005109bce7687fc09-006618aae9-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.562 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:30:49.562 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:30:49.562 DEBUG:boto:Method: HEAD 2024-04-12T03:30:49.563 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:30:49.563 DEBUG:boto:Data: 2024-04-12T03:30:49.563 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.563 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:30:49.563 DEBUG:boto:Port: 8000 2024-04-12T03:30:49.563 DEBUG:boto:Params: {} 2024-04-12T03:30:49.563 DEBUG:boto:Token: None 2024-04-12T03:30:49.563 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/obj 2024-04-12T03:30:49.563 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KcoILpCDkYFjAXliyaPExsthjiE= 2024-04-12T03:30:49.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KcoILpCDkYFjAXliyaPExsthjiE=', 'Content-Length': '0'} 2024-04-12T03:30:49.565 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a6829328730efb31-006618aae9-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:30:49.565 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:30:49.565 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:30:49.565 DEBUG:boto:Method: GET 2024-04-12T03:30:49.566 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:30:49.566 DEBUG:boto:Data: 2024-04-12T03:30:49.566 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.566 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:30:49.566 DEBUG:boto:Port: 8001 2024-04-12T03:30:49.566 DEBUG:boto:Params: {} 2024-04-12T03:30:49.566 DEBUG:boto:Token: None 2024-04-12T03:30:49.566 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/obj 2024-04-12T03:30:49.566 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hal6DlVMg4luUD8LB5MD09H7BiY= 2024-04-12T03:30:49.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hal6DlVMg4luUD8LB5MD09H7BiY=', 'Content-Length': '0'} 2024-04-12T03:30:49.568 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008bad54bd9279d217-006618aae9-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.569 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:30:49.569 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:30:49.569 DEBUG:boto:Method: GET 2024-04-12T03:30:49.569 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:30:49.569 DEBUG:boto:Data: 2024-04-12T03:30:49.569 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.569 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:30:49.569 DEBUG:boto:Port: 8000 2024-04-12T03:30:49.569 DEBUG:boto:Params: {} 2024-04-12T03:30:49.569 DEBUG:boto:Token: None 2024-04-12T03:30:49.569 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/obj 2024-04-12T03:30:49.570 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hal6DlVMg4luUD8LB5MD09H7BiY= 2024-04-12T03:30:49.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hal6DlVMg4luUD8LB5MD09H7BiY=', 'Content-Length': '0'} 2024-04-12T03:30:49.571 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aa569c990f2e4b23-006618aae9-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.572 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:30:49.572 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:30:49.572 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:30:49.572 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:30:49.572 DEBUG:boto:Method: GET 2024-04-12T03:30:49.572 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:30:49.572 DEBUG:boto:Data: 2024-04-12T03:30:49.572 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.572 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:30:49.572 DEBUG:boto:Port: 8001 2024-04-12T03:30:49.573 DEBUG:boto:Params: {} 2024-04-12T03:30:49.573 DEBUG:boto:Token: None 2024-04-12T03:30:49.573 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/ 2024-04-12T03:30:49.573 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RV1wUrSWWvxwFGfJwspdBK8edSw= 2024-04-12T03:30:49.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RV1wUrSWWvxwFGfJwspdBK8edSw=', 'Content-Length': '0'} 2024-04-12T03:30:49.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f17853ca5fb2c96-006618aae9-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.576 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:28:49.178Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:49.576 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:30:49.576 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:30:49.576 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:30:49.576 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:30:49.576 DEBUG:boto:Method: GET 2024-04-12T03:30:49.576 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:30:49.576 DEBUG:boto:Data: 2024-04-12T03:30:49.577 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.577 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:30:49.577 DEBUG:boto:Port: 8000 2024-04-12T03:30:49.577 DEBUG:boto:Params: {} 2024-04-12T03:30:49.577 DEBUG:boto:Token: None 2024-04-12T03:30:49.577 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/ 2024-04-12T03:30:49.577 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RV1wUrSWWvxwFGfJwspdBK8edSw= 2024-04-12T03:30:49.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RV1wUrSWWvxwFGfJwspdBK8edSw=', 'Content-Length': '0'} 2024-04-12T03:30:49.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000712cb99c99ea7911-006618aae9-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.580 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:28:49.178Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:30:49.580 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-16 zones={a2, a1} 2024-04-12T03:30:49.580 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:30:49.580 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:30:49.581 DEBUG:boto:path=/dkyges-16/?versioning 2024-04-12T03:30:49.581 DEBUG:boto:auth_path=/dkyges-16/?versioning 2024-04-12T03:30:49.581 DEBUG:boto:Method: PUT 2024-04-12T03:30:49.581 DEBUG:boto:Path: /dkyges-16/?versioning 2024-04-12T03:30:49.581 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:30:49.581 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.581 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:30:49.581 DEBUG:boto:Port: 8000 2024-04-12T03:30:49.581 DEBUG:boto:Params: {} 2024-04-12T03:30:49.581 DEBUG:boto:Token: None 2024-04-12T03:30:49.581 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/?versioning 2024-04-12T03:30:49.582 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GUSycYXbgxnlV+9L6P4Iq5HnabM= 2024-04-12T03:30:49.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GUSycYXbgxnlV+9L6P4Iq5HnabM=', 'Content-Length': '228'} 2024-04-12T03:30:49.587 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e2f22ac405e66394-006618aae9-110f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:49.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:49.588 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:30:49.646 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:49.646 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:49.681 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:30:49.694 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:30:49.694 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:30:49.695 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:30:49.695 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:30:49.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:30:49.695 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:30:49.695 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:30:49.695 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:30:49.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:30:49.695 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:30:49.695 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:30:49.696 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:30:49.696 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:30:49.696 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:30:49.696 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892649.587092_167.1", 2024-04-12T03:30:49.696 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:30:49.587092Z" 2024-04-12T03:30:49.696 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:30:49.696 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:30:49.697 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892649.587092_167.1'} 2024-04-12T03:30:49.697 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:30:49.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:49.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:30:49.698 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:30:49.754 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:49.754 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:49.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:30:49.786+0000 7fe300276780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:30:49.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:30:49.787+0000 7fe300276780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:30:49.802 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:30:49.802 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:30:49.802 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:30:49.802 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:30:49.802 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:30:49.802 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:30:49.802 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:30:49.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:30:49.803 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:30:49.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:30:49.803 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:30:49.803 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:30:49.803 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:30:49.803 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:30:49.803 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:30:49.803 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:30:49.803 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:30:49.803 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:30:49.803 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:30:49.803 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:30:49.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:30:49.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:30:49.804 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:30:49.804 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:30:49.804 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:30:49.804 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:30:49.804 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:30:49.804 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:30:49.804 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:30:49.804 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:30:49.804 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:30:49.804 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:30:49.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:30:49.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:30:49.805 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:30:49.805 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:30:49.805 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:30:49.805 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:30:49.805 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:30:49.805 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:30:49.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:30:49.805 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:30:49.805 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:30:49.805 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:30:49.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:30:49.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:30:49.806 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:30:49.806 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:30:49.806 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:30:49.806 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892649.587092_167.1", 2024-04-12T03:30:49.806 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:30:49.806 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:30:49.806 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:30:49.806 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:30:49.587092Z", 2024-04-12T03:30:49.806 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:30:49.806 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:30:49.806 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:30:49.807 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:30:49.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:30:49.807 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:30:49.807 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:30:49.807 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:30:49.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:30:49.807 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:30:49Z" 2024-04-12T03:30:49.807 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:30:49.808 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892649.587092_167.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:30:49.587092Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:30:49Z" } 2024-04-12T03:30:49.808 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892649.587092_167.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:30:49.587092Z', 'realm_epoch': 2}}] 2024-04-12T03:30:49.808 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892649.587092_167.1'} 2024-04-12T03:30:49.808 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892649.587092_167.1'} 2024-04-12T03:30:49.809 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:30:49.809 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:30:49.809 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:30:49.809 DEBUG:boto:Method: HEAD 2024-04-12T03:30:49.809 DEBUG:boto:Path: /dkyges-16/ 2024-04-12T03:30:49.809 DEBUG:boto:Data: 2024-04-12T03:30:49.809 DEBUG:boto:Headers: {} 2024-04-12T03:30:49.809 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:30:49.809 DEBUG:boto:Port: 8000 2024-04-12T03:30:49.809 DEBUG:boto:Params: {} 2024-04-12T03:30:49.810 DEBUG:boto:Token: None 2024-04-12T03:30:49.810 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/ 2024-04-12T03:30:49.810 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D5UJjjrGP45p+oe+emdlwrSbYjU= 2024-04-12T03:30:49.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D5UJjjrGP45p+oe+emdlwrSbYjU=', 'Content-Length': '0'} 2024-04-12T03:30:49.812 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2fb39996b1b49b6-006618aae9-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.813 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:30:49.813 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:30:49.813 DEBUG:boto:Method: PUT 2024-04-12T03:30:49.813 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:30:49.813 DEBUG:boto:Data: 2024-04-12T03:30:49.813 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:30:49.813 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:30:49.813 DEBUG:boto:Port: 8000 2024-04-12T03:30:49.813 DEBUG:boto:Params: {} 2024-04-12T03:30:49.813 DEBUG:boto:Token: None 2024-04-12T03:30:49.813 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:30:49 GMT /dkyges-16/obj 2024-04-12T03:30:49.814 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:N5an0s+RxkvSUzNcu7TEC9AbYsU= 2024-04-12T03:30:49.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:30:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:N5an0s+RxkvSUzNcu7TEC9AbYsU='} 2024-04-12T03:30:49.825 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-amz-request-id', 'tx00000b0465715f2e8305c-006618aae9-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:30:49.825 DEBUG:rgw_multi.tests:created new version id=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:30:49.825 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-16 2024-04-12T03:30:49.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:30:49.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:30:49.826 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:30:49.888 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:30:49.888 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:30:49.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:30:49.926+0000 7f0d7d99e780 1 waiting for incremental sync to catch up: 2024-04-12T03:30:49.927 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000001.30.6, , ] 2024-04-12T03:30:49.927 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000003.33.13, , ] 2024-04-12T03:30:54.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:30:54.928+0000 7f0d7d99e780 1 waiting for incremental sync to catch up: 2024-04-12T03:30:54.929 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000001.30.6, , ] 2024-04-12T03:30:54.929 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000003.33.13, , ] 2024-04-12T03:30:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:30:59.929+0000 7f0d7d99e780 1 waiting for incremental sync to catch up: 2024-04-12T03:30:59.930 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000001.30.6, , ] 2024-04-12T03:30:59.930 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000003.33.13, , ] 2024-04-12T03:31:04.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:04.930+0000 7f0d7d99e780 1 bucket sync caught up with source: 2024-04-12T03:31:04.935 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000003.33.13, , ] 2024-04-12T03:31:04.935 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000003.33.13, , ] 2024-04-12T03:31:04.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:04.930+0000 7f0d7d99e780 0 bucket checkpoint complete 2024-04-12T03:31:04.948 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-16 2024-04-12T03:31:04.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:04.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:04.949 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:31:05.018 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:05.018 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:05.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:05.059+0000 7f3cd32e7780 1 waiting for incremental sync to catch up: 2024-04-12T03:31:05.060 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000001.20.6, , ] 2024-04-12T03:31:05.061 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000003.26.13, , ] 2024-04-12T03:31:10.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:10.060+0000 7f3cd32e7780 1 bucket sync caught up with source: 2024-04-12T03:31:10.062 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000003.26.13, , ] 2024-04-12T03:31:10.062 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000003.26.13, , ] 2024-04-12T03:31:10.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:10.060+0000 7f3cd32e7780 0 bucket checkpoint complete 2024-04-12T03:31:10.078 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-16 zones={a2, a1} 2024-04-12T03:31:10.079 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:10.079 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:10.079 DEBUG:boto:Method: HEAD 2024-04-12T03:31:10.079 DEBUG:boto:Path: /dkyges-16/ 2024-04-12T03:31:10.079 DEBUG:boto:Data: 2024-04-12T03:31:10.079 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.079 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.079 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.079 DEBUG:boto:Params: {} 2024-04-12T03:31:10.080 DEBUG:boto:Token: None 2024-04-12T03:31:10.080 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/ 2024-04-12T03:31:10.080 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GGruYTfVc04WvxXuf+IjD5cM1UA= 2024-04-12T03:31:10.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GGruYTfVc04WvxXuf+IjD5cM1UA=', 'Content-Length': '0'} 2024-04-12T03:31:10.083 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b6db8fe33622343-006618aafe-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.083 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:10.083 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:10.083 DEBUG:boto:Method: HEAD 2024-04-12T03:31:10.083 DEBUG:boto:Path: /dkyges-16/ 2024-04-12T03:31:10.083 DEBUG:boto:Data: 2024-04-12T03:31:10.083 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.083 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.083 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.084 DEBUG:boto:Params: {} 2024-04-12T03:31:10.084 DEBUG:boto:Token: None 2024-04-12T03:31:10.084 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/ 2024-04-12T03:31:10.084 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GGruYTfVc04WvxXuf+IjD5cM1UA= 2024-04-12T03:31:10.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GGruYTfVc04WvxXuf+IjD5cM1UA=', 'Content-Length': '0'} 2024-04-12T03:31:10.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000078cb12cad5545c1-006618aafe-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.087 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:31:10.087 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:10.087 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:10.087 DEBUG:boto:path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:31:10.087 DEBUG:boto:auth_path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:31:10.087 DEBUG:boto:Method: GET 2024-04-12T03:31:10.087 DEBUG:boto:Path: /dkyges-16/?versions&max-keys=999 2024-04-12T03:31:10.087 DEBUG:boto:Data: 2024-04-12T03:31:10.087 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.087 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.087 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.088 DEBUG:boto:Params: {} 2024-04-12T03:31:10.088 DEBUG:boto:Token: None 2024-04-12T03:31:10.088 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/?versions 2024-04-12T03:31:10.088 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d5N7jvDnIU9FphvUjasSESb52UI= 2024-04-12T03:31:10.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d5N7jvDnIU9FphvUjasSESb52UI=', 'Content-Length': '0'} 2024-04-12T03:31:10.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000178257607e63fd28-006618aafe-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.091 DEBUG:boto:b'dkyges-16999falseobjujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vbtrue2024-04-12T03:30:49.815Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-12T03:28:49.178Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:10.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:31:10.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:31:10.091 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:31:10.091 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:10.092 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:10.092 DEBUG:boto:path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:31:10.092 DEBUG:boto:auth_path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:31:10.092 DEBUG:boto:Method: GET 2024-04-12T03:31:10.092 DEBUG:boto:Path: /dkyges-16/?versions&max-keys=999 2024-04-12T03:31:10.092 DEBUG:boto:Data: 2024-04-12T03:31:10.092 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.092 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.092 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.092 DEBUG:boto:Params: {} 2024-04-12T03:31:10.092 DEBUG:boto:Token: None 2024-04-12T03:31:10.093 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/?versions 2024-04-12T03:31:10.093 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d5N7jvDnIU9FphvUjasSESb52UI= 2024-04-12T03:31:10.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d5N7jvDnIU9FphvUjasSESb52UI=', 'Content-Length': '0'} 2024-04-12T03:31:10.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dad404f43fd5fd20-006618aafe-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.095 DEBUG:boto:b'dkyges-16999falseobjujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vbtrue2024-04-12T03:30:49.815Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-12T03:28:49.178Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:10.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:31:10.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:31:10.096 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:10.096 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:10.096 DEBUG:boto:path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:31:10.096 DEBUG:boto:auth_path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:31:10.096 DEBUG:boto:Method: GET 2024-04-12T03:31:10.096 DEBUG:boto:Path: /dkyges-16/?versions&max-keys=999 2024-04-12T03:31:10.097 DEBUG:boto:Data: 2024-04-12T03:31:10.097 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.097 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.097 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.097 DEBUG:boto:Params: {} 2024-04-12T03:31:10.097 DEBUG:boto:Token: None 2024-04-12T03:31:10.097 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/?versions 2024-04-12T03:31:10.097 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d5N7jvDnIU9FphvUjasSESb52UI= 2024-04-12T03:31:10.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d5N7jvDnIU9FphvUjasSESb52UI=', 'Content-Length': '0'} 2024-04-12T03:31:10.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e28d4e47cc0d4e9-006618aafe-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.142 DEBUG:boto:b'dkyges-16999falseobjujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vbtrue2024-04-12T03:30:49.815Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-12T03:28:49.178Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:10.143 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:10.143 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:10.143 DEBUG:boto:path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:31:10.143 DEBUG:boto:auth_path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:31:10.143 DEBUG:boto:Method: GET 2024-04-12T03:31:10.143 DEBUG:boto:Path: /dkyges-16/?versions&max-keys=999 2024-04-12T03:31:10.143 DEBUG:boto:Data: 2024-04-12T03:31:10.144 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.144 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.144 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.144 DEBUG:boto:Params: {} 2024-04-12T03:31:10.144 DEBUG:boto:Token: None 2024-04-12T03:31:10.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/?versions 2024-04-12T03:31:10.144 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d5N7jvDnIU9FphvUjasSESb52UI= 2024-04-12T03:31:10.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d5N7jvDnIU9FphvUjasSESb52UI=', 'Content-Length': '0'} 2024-04-12T03:31:10.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004929f4f51e7b7f76-006618aafe-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.147 DEBUG:boto:b'dkyges-16999falseobjujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vbtrue2024-04-12T03:30:49.815Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2024-04-12T03:28:49.178Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:10.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:31:10.148 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.148 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.148 DEBUG:boto:path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.148 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.148 DEBUG:boto:Method: GET 2024-04-12T03:31:10.148 DEBUG:boto:Path: /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.149 DEBUG:boto:Data: 2024-04-12T03:31:10.149 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.149 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.149 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.149 DEBUG:boto:Params: {} 2024-04-12T03:31:10.149 DEBUG:boto:Token: None 2024-04-12T03:31:10.149 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.149 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t8+N5VnaZgDeaL11cQjCWo4m304= 2024-04-12T03:31:10.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t8+N5VnaZgDeaL11cQjCWo4m304=', 'Content-Length': '0'} 2024-04-12T03:31:10.151 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000731dbd5f7e825a6d-006618aafe-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.151 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.152 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.152 DEBUG:boto:path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.152 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.152 DEBUG:boto:Method: GET 2024-04-12T03:31:10.152 DEBUG:boto:Path: /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.152 DEBUG:boto:Data: 2024-04-12T03:31:10.152 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.152 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.152 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.152 DEBUG:boto:Params: {} 2024-04-12T03:31:10.152 DEBUG:boto:Token: None 2024-04-12T03:31:10.153 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.153 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t8+N5VnaZgDeaL11cQjCWo4m304= 2024-04-12T03:31:10.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t8+N5VnaZgDeaL11cQjCWo4m304=', 'Content-Length': '0'} 2024-04-12T03:31:10.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001c8164ccf3fc6925-006618aafe-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.155 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.155 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.155 DEBUG:boto:path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.155 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.155 DEBUG:boto:Method: HEAD 2024-04-12T03:31:10.155 DEBUG:boto:Path: /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.155 DEBUG:boto:Data: 2024-04-12T03:31:10.155 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.155 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.156 DEBUG:boto:Params: {} 2024-04-12T03:31:10.156 DEBUG:boto:Token: None 2024-04-12T03:31:10.156 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8ywRvD1m6tn2ksHAIX08n8FoA/c= 2024-04-12T03:31:10.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8ywRvD1m6tn2ksHAIX08n8FoA/c=', 'Content-Length': '0'} 2024-04-12T03:31:10.157 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001fec79d03cb14a2a-006618aafe-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.158 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.158 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.158 DEBUG:boto:path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.158 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.158 DEBUG:boto:Method: HEAD 2024-04-12T03:31:10.158 DEBUG:boto:Path: /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.158 DEBUG:boto:Data: 2024-04-12T03:31:10.158 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.159 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.159 DEBUG:boto:Params: {} 2024-04-12T03:31:10.159 DEBUG:boto:Token: None 2024-04-12T03:31:10.159 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.159 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8ywRvD1m6tn2ksHAIX08n8FoA/c= 2024-04-12T03:31:10.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8ywRvD1m6tn2ksHAIX08n8FoA/c=', 'Content-Length': '0'} 2024-04-12T03:31:10.161 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f1978ca6e2cbe738-006618aafe-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:31:10.161 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.161 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.162 DEBUG:boto:path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.162 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.162 DEBUG:boto:Method: GET 2024-04-12T03:31:10.162 DEBUG:boto:Path: /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.162 DEBUG:boto:Data: 2024-04-12T03:31:10.162 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.162 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.162 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.162 DEBUG:boto:Params: {} 2024-04-12T03:31:10.162 DEBUG:boto:Token: None 2024-04-12T03:31:10.162 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.163 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t8+N5VnaZgDeaL11cQjCWo4m304= 2024-04-12T03:31:10.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t8+N5VnaZgDeaL11cQjCWo4m304=', 'Content-Length': '0'} 2024-04-12T03:31:10.164 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005edc15220a702b5f-006618aafe-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.165 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.165 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.165 DEBUG:boto:path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.165 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.165 DEBUG:boto:Method: GET 2024-04-12T03:31:10.165 DEBUG:boto:Path: /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.165 DEBUG:boto:Data: 2024-04-12T03:31:10.165 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.165 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.165 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.166 DEBUG:boto:Params: {} 2024-04-12T03:31:10.166 DEBUG:boto:Token: None 2024-04-12T03:31:10.166 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.166 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t8+N5VnaZgDeaL11cQjCWo4m304= 2024-04-12T03:31:10.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t8+N5VnaZgDeaL11cQjCWo4m304=', 'Content-Length': '0'} 2024-04-12T03:31:10.168 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007f8f05bd0f496c5d-006618aafe-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.168 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:10.168 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:10.168 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:31:10.168 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:31:10.168 DEBUG:boto:Method: GET 2024-04-12T03:31:10.168 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:31:10.168 DEBUG:boto:Data: 2024-04-12T03:31:10.169 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.169 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.169 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.169 DEBUG:boto:Params: {} 2024-04-12T03:31:10.169 DEBUG:boto:Token: None 2024-04-12T03:31:10.169 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/ 2024-04-12T03:31:10.169 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZkfsNloBgAjXLIZwJqYSi2F2vWM= 2024-04-12T03:31:10.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZkfsNloBgAjXLIZwJqYSi2F2vWM=', 'Content-Length': '0'} 2024-04-12T03:31:10.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b61c734a333b3b06-006618aafe-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.172 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:30:49.815Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:10.173 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:10.173 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:10.173 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:31:10.173 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:31:10.173 DEBUG:boto:Method: GET 2024-04-12T03:31:10.173 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:31:10.174 DEBUG:boto:Data: 2024-04-12T03:31:10.174 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.174 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.174 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.174 DEBUG:boto:Params: {} 2024-04-12T03:31:10.174 DEBUG:boto:Token: None 2024-04-12T03:31:10.174 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/ 2024-04-12T03:31:10.174 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZkfsNloBgAjXLIZwJqYSi2F2vWM= 2024-04-12T03:31:10.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZkfsNloBgAjXLIZwJqYSi2F2vWM=', 'Content-Length': '0'} 2024-04-12T03:31:10.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042b2d9ae4d18414e-006618aafe-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.218 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:30:49.815Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:10.219 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.219 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.219 DEBUG:boto:Method: HEAD 2024-04-12T03:31:10.219 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:31:10.220 DEBUG:boto:Data: 2024-04-12T03:31:10.220 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.220 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.220 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.220 DEBUG:boto:Params: {} 2024-04-12T03:31:10.220 DEBUG:boto:Token: None 2024-04-12T03:31:10.220 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj 2024-04-12T03:31:10.220 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gvgRybwZXIdETr/Wfc7Pfhyp/8k= 2024-04-12T03:31:10.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gvgRybwZXIdETr/Wfc7Pfhyp/8k=', 'Content-Length': '0'} 2024-04-12T03:31:10.223 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006a486e2b49518eb0-006618aafe-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.223 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.223 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.223 DEBUG:boto:Method: HEAD 2024-04-12T03:31:10.223 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:31:10.224 DEBUG:boto:Data: 2024-04-12T03:31:10.224 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.224 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.224 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.224 DEBUG:boto:Params: {} 2024-04-12T03:31:10.224 DEBUG:boto:Token: None 2024-04-12T03:31:10.224 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj 2024-04-12T03:31:10.224 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gvgRybwZXIdETr/Wfc7Pfhyp/8k= 2024-04-12T03:31:10.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gvgRybwZXIdETr/Wfc7Pfhyp/8k=', 'Content-Length': '0'} 2024-04-12T03:31:10.226 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000059a14ae2f1f33126-006618aafe-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:31:10.227 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.227 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.227 DEBUG:boto:path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.227 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.227 DEBUG:boto:Method: GET 2024-04-12T03:31:10.227 DEBUG:boto:Path: /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.227 DEBUG:boto:Data: 2024-04-12T03:31:10.227 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.227 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.228 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.228 DEBUG:boto:Params: {} 2024-04-12T03:31:10.228 DEBUG:boto:Token: None 2024-04-12T03:31:10.228 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.228 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t8+N5VnaZgDeaL11cQjCWo4m304= 2024-04-12T03:31:10.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t8+N5VnaZgDeaL11cQjCWo4m304=', 'Content-Length': '0'} 2024-04-12T03:31:10.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008780f2c5a783941f-006618aafe-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.230 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.231 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.231 DEBUG:boto:path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.231 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.231 DEBUG:boto:Method: GET 2024-04-12T03:31:10.231 DEBUG:boto:Path: /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.231 DEBUG:boto:Data: 2024-04-12T03:31:10.231 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.231 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.231 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.231 DEBUG:boto:Params: {} 2024-04-12T03:31:10.231 DEBUG:boto:Token: None 2024-04-12T03:31:10.232 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.232 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t8+N5VnaZgDeaL11cQjCWo4m304= 2024-04-12T03:31:10.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t8+N5VnaZgDeaL11cQjCWo4m304=', 'Content-Length': '0'} 2024-04-12T03:31:10.233 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000048a86c96b041b820-006618aafe-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.233 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:10.234 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:10.234 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:31:10.234 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:31:10.234 DEBUG:boto:Method: GET 2024-04-12T03:31:10.234 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:31:10.234 DEBUG:boto:Data: 2024-04-12T03:31:10.234 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.234 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.234 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.234 DEBUG:boto:Params: {} 2024-04-12T03:31:10.235 DEBUG:boto:Token: None 2024-04-12T03:31:10.235 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/ 2024-04-12T03:31:10.235 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZkfsNloBgAjXLIZwJqYSi2F2vWM= 2024-04-12T03:31:10.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZkfsNloBgAjXLIZwJqYSi2F2vWM=', 'Content-Length': '0'} 2024-04-12T03:31:10.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b32232f2480eec8-006618aafe-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.278 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:30:49.815Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:10.279 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:10.279 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:10.279 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:31:10.279 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:31:10.279 DEBUG:boto:Method: GET 2024-04-12T03:31:10.279 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:31:10.280 DEBUG:boto:Data: 2024-04-12T03:31:10.280 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.280 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.280 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.280 DEBUG:boto:Params: {} 2024-04-12T03:31:10.281 DEBUG:boto:Token: None 2024-04-12T03:31:10.281 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/ 2024-04-12T03:31:10.281 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZkfsNloBgAjXLIZwJqYSi2F2vWM= 2024-04-12T03:31:10.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZkfsNloBgAjXLIZwJqYSi2F2vWM=', 'Content-Length': '0'} 2024-04-12T03:31:10.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cb1112ba6fa42dd-006618aafe-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.284 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:30:49.815Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:10.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:31:10.285 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.285 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.285 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:10.285 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:10.285 DEBUG:boto:Method: GET 2024-04-12T03:31:10.285 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:31:10.285 DEBUG:boto:Data: 2024-04-12T03:31:10.285 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.285 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.286 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.286 DEBUG:boto:Params: {} 2024-04-12T03:31:10.286 DEBUG:boto:Token: None 2024-04-12T03:31:10.286 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:31:10.286 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aa4h2ENuIZPb0tmmA8mhX29jtjQ= 2024-04-12T03:31:10.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aa4h2ENuIZPb0tmmA8mhX29jtjQ=', 'Content-Length': '0'} 2024-04-12T03:31:10.288 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004966e969e1320887-006618aafe-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.288 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.288 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.289 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:10.289 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:10.289 DEBUG:boto:Method: GET 2024-04-12T03:31:10.289 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:31:10.289 DEBUG:boto:Data: 2024-04-12T03:31:10.289 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.289 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.289 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.289 DEBUG:boto:Params: {} 2024-04-12T03:31:10.290 DEBUG:boto:Token: None 2024-04-12T03:31:10.290 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:31:10.290 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aa4h2ENuIZPb0tmmA8mhX29jtjQ= 2024-04-12T03:31:10.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aa4h2ENuIZPb0tmmA8mhX29jtjQ=', 'Content-Length': '0'} 2024-04-12T03:31:10.291 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000608a3cd325173942-006618aafe-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.292 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.292 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.292 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:10.292 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:10.292 DEBUG:boto:Method: HEAD 2024-04-12T03:31:10.292 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:31:10.292 DEBUG:boto:Data: 2024-04-12T03:31:10.292 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.293 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.293 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.293 DEBUG:boto:Params: {} 2024-04-12T03:31:10.293 DEBUG:boto:Token: None 2024-04-12T03:31:10.293 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:31:10.293 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Gq+4EkoaopQaooP2f84zzWVk0uo= 2024-04-12T03:31:10.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Gq+4EkoaopQaooP2f84zzWVk0uo=', 'Content-Length': '0'} 2024-04-12T03:31:10.295 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000026cddc4b8f3cfb6c-006618aafe-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.296 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.296 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.296 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:10.296 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:10.296 DEBUG:boto:Method: HEAD 2024-04-12T03:31:10.296 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:31:10.296 DEBUG:boto:Data: 2024-04-12T03:31:10.296 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.296 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.297 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.297 DEBUG:boto:Params: {} 2024-04-12T03:31:10.297 DEBUG:boto:Token: None 2024-04-12T03:31:10.297 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:31:10.297 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Gq+4EkoaopQaooP2f84zzWVk0uo= 2024-04-12T03:31:10.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Gq+4EkoaopQaooP2f84zzWVk0uo=', 'Content-Length': '0'} 2024-04-12T03:31:10.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000570ca0149f92234f-006618aafe-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:31:10.299 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.299 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.299 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:10.299 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:10.299 DEBUG:boto:Method: GET 2024-04-12T03:31:10.299 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:31:10.300 DEBUG:boto:Data: 2024-04-12T03:31:10.300 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.300 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.300 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.300 DEBUG:boto:Params: {} 2024-04-12T03:31:10.300 DEBUG:boto:Token: None 2024-04-12T03:31:10.300 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:31:10.300 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aa4h2ENuIZPb0tmmA8mhX29jtjQ= 2024-04-12T03:31:10.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aa4h2ENuIZPb0tmmA8mhX29jtjQ=', 'Content-Length': '0'} 2024-04-12T03:31:10.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c82393dc8ef5f104-006618aafe-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.302 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.302 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.302 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:10.302 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:10.302 DEBUG:boto:Method: GET 2024-04-12T03:31:10.303 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:31:10.303 DEBUG:boto:Data: 2024-04-12T03:31:10.303 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.303 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.303 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.303 DEBUG:boto:Params: {} 2024-04-12T03:31:10.303 DEBUG:boto:Token: None 2024-04-12T03:31:10.303 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:31:10.303 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aa4h2ENuIZPb0tmmA8mhX29jtjQ= 2024-04-12T03:31:10.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aa4h2ENuIZPb0tmmA8mhX29jtjQ=', 'Content-Length': '0'} 2024-04-12T03:31:10.305 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:28:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000026bb665b978d13f9-006618aafe-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.305 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:10.305 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:10.305 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:31:10.306 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:31:10.306 DEBUG:boto:Method: GET 2024-04-12T03:31:10.306 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:31:10.306 DEBUG:boto:Data: 2024-04-12T03:31:10.306 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.306 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.306 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.306 DEBUG:boto:Params: {} 2024-04-12T03:31:10.306 DEBUG:boto:Token: None 2024-04-12T03:31:10.306 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/ 2024-04-12T03:31:10.307 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZkfsNloBgAjXLIZwJqYSi2F2vWM= 2024-04-12T03:31:10.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZkfsNloBgAjXLIZwJqYSi2F2vWM=', 'Content-Length': '0'} 2024-04-12T03:31:10.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073c23e7b5c21012e-006618aafe-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.309 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:30:49.815Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:10.310 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:10.310 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:10.310 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:31:10.310 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:31:10.310 DEBUG:boto:Method: GET 2024-04-12T03:31:10.310 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:31:10.311 DEBUG:boto:Data: 2024-04-12T03:31:10.311 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.311 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.311 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.311 DEBUG:boto:Params: {} 2024-04-12T03:31:10.311 DEBUG:boto:Token: None 2024-04-12T03:31:10.311 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/ 2024-04-12T03:31:10.311 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZkfsNloBgAjXLIZwJqYSi2F2vWM= 2024-04-12T03:31:10.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZkfsNloBgAjXLIZwJqYSi2F2vWM=', 'Content-Length': '0'} 2024-04-12T03:31:10.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019be2539be03ccd2-006618aafe-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.354 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:30:49.815Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:10.355 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.355 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.355 DEBUG:boto:Method: HEAD 2024-04-12T03:31:10.355 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:31:10.355 DEBUG:boto:Data: 2024-04-12T03:31:10.355 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.355 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.355 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.355 DEBUG:boto:Params: {} 2024-04-12T03:31:10.356 DEBUG:boto:Token: None 2024-04-12T03:31:10.356 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj 2024-04-12T03:31:10.356 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gvgRybwZXIdETr/Wfc7Pfhyp/8k= 2024-04-12T03:31:10.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gvgRybwZXIdETr/Wfc7Pfhyp/8k=', 'Content-Length': '0'} 2024-04-12T03:31:10.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000042b7020c6ef4067e-006618aafe-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.359 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.359 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.359 DEBUG:boto:Method: HEAD 2024-04-12T03:31:10.359 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:31:10.359 DEBUG:boto:Data: 2024-04-12T03:31:10.359 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.360 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.360 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.360 DEBUG:boto:Params: {} 2024-04-12T03:31:10.360 DEBUG:boto:Token: None 2024-04-12T03:31:10.360 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj 2024-04-12T03:31:10.360 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gvgRybwZXIdETr/Wfc7Pfhyp/8k= 2024-04-12T03:31:10.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gvgRybwZXIdETr/Wfc7Pfhyp/8k=', 'Content-Length': '0'} 2024-04-12T03:31:10.363 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000024be4a1cfeeeec5e-006618aafe-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:31:10.363 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.363 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.363 DEBUG:boto:path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.363 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.363 DEBUG:boto:Method: GET 2024-04-12T03:31:10.363 DEBUG:boto:Path: /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.363 DEBUG:boto:Data: 2024-04-12T03:31:10.364 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.364 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.364 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.364 DEBUG:boto:Params: {} 2024-04-12T03:31:10.364 DEBUG:boto:Token: None 2024-04-12T03:31:10.364 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.364 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t8+N5VnaZgDeaL11cQjCWo4m304= 2024-04-12T03:31:10.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t8+N5VnaZgDeaL11cQjCWo4m304=', 'Content-Length': '0'} 2024-04-12T03:31:10.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000972298df643cf5ac-006618aafe-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.366 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.366 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.366 DEBUG:boto:path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.367 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.367 DEBUG:boto:Method: GET 2024-04-12T03:31:10.367 DEBUG:boto:Path: /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.367 DEBUG:boto:Data: 2024-04-12T03:31:10.367 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.367 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.367 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.367 DEBUG:boto:Params: {} 2024-04-12T03:31:10.367 DEBUG:boto:Token: None 2024-04-12T03:31:10.367 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:10.368 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t8+N5VnaZgDeaL11cQjCWo4m304= 2024-04-12T03:31:10.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t8+N5VnaZgDeaL11cQjCWo4m304=', 'Content-Length': '0'} 2024-04-12T03:31:10.369 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002d911fee6f4af87a-006618aafe-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.370 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:10.370 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:10.370 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:31:10.370 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:31:10.370 DEBUG:boto:Method: GET 2024-04-12T03:31:10.370 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:31:10.370 DEBUG:boto:Data: 2024-04-12T03:31:10.370 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.370 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:10.371 DEBUG:boto:Port: 8001 2024-04-12T03:31:10.371 DEBUG:boto:Params: {} 2024-04-12T03:31:10.371 DEBUG:boto:Token: None 2024-04-12T03:31:10.371 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/ 2024-04-12T03:31:10.371 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZkfsNloBgAjXLIZwJqYSi2F2vWM= 2024-04-12T03:31:10.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZkfsNloBgAjXLIZwJqYSi2F2vWM=', 'Content-Length': '0'} 2024-04-12T03:31:10.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000505ba6e620e7498-006618aafe-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.414 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:30:49.815Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:10.415 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:10.415 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:10.415 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:31:10.415 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:31:10.415 DEBUG:boto:Method: GET 2024-04-12T03:31:10.415 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:31:10.415 DEBUG:boto:Data: 2024-04-12T03:31:10.416 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.416 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.416 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.416 DEBUG:boto:Params: {} 2024-04-12T03:31:10.416 DEBUG:boto:Token: None 2024-04-12T03:31:10.416 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/ 2024-04-12T03:31:10.416 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZkfsNloBgAjXLIZwJqYSi2F2vWM= 2024-04-12T03:31:10.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZkfsNloBgAjXLIZwJqYSi2F2vWM=', 'Content-Length': '0'} 2024-04-12T03:31:10.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c792d2c9d25167f1-006618aafe-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.419 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:30:49.815Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:10.420 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-16 zones={a2, a1} 2024-04-12T03:31:10.420 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:10.420 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:10.420 DEBUG:boto:path=/dkyges-16/?versioning 2024-04-12T03:31:10.420 DEBUG:boto:auth_path=/dkyges-16/?versioning 2024-04-12T03:31:10.420 DEBUG:boto:Method: PUT 2024-04-12T03:31:10.420 DEBUG:boto:Path: /dkyges-16/?versioning 2024-04-12T03:31:10.420 DEBUG:boto:Data: Suspended Disabled 2024-04-12T03:31:10.421 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.421 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.421 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.421 DEBUG:boto:Params: {} 2024-04-12T03:31:10.421 DEBUG:boto:Token: None 2024-04-12T03:31:10.421 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/?versioning 2024-04-12T03:31:10.421 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lUgrQucSrSxmSqup6umdOSidrGg= 2024-04-12T03:31:10.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lUgrQucSrSxmSqup6umdOSidrGg=', 'Content-Length': '230'} 2024-04-12T03:31:10.428 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004a876d835831458e-006618aafe-110f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:10.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:10.428 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:31:10.485 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:10.486 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:10.521 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:31:10.534 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:31:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:31:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:31:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:31:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:31:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:31:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:31:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892670.426988_169.1", 2024-04-12T03:31:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:31:10.426988Z" 2024-04-12T03:31:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:10.536 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:31:10.537 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892670.426988_169.1'} 2024-04-12T03:31:10.538 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:31:10.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:10.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:10.538 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:31:10.594 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:10.595 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:10.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:10.625+0000 7f025d99c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:31:10.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:10.626+0000 7f025d99c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:31:10.641 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:31:10.641 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:31:10.641 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:31:10.641 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:31:10.641 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:31:10.641 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:31:10.641 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:31:10.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:10.642 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:31:10.642 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:10.642 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:31:10.642 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:31:10.642 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:31:10.642 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:31:10.642 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:31:10.642 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:31:10.642 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:31:10.643 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:31:10.643 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:31:10.643 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:10.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:10.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:10.643 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:31:10.643 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:31:10.643 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:31:10.643 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:31:10.643 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:31:10.643 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:31:10.644 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:31:10.644 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:31:10.644 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:31:10.644 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:10.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:10.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:10.644 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:31:10.644 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:31:10.644 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:31:10.644 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:31:10.644 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:31:10.645 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:31:10.645 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:31:10.645 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:31:10.645 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:31:10.645 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:10.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:10.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:10.645 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:31:10.645 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:31:10.645 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:31:10.646 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892670.426988_169.1", 2024-04-12T03:31:10.646 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:31:10.646 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:31:10.646 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:31:10.646 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:31:10.426988Z", 2024-04-12T03:31:10.646 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:31:10.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:10.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:10.646 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:31:10.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:10.647 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:31:10.647 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:31:10.647 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:31:10.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:10.647 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:31:10Z" 2024-04-12T03:31:10.647 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:31:10.648 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892670.426988_169.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:31:10.426988Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:31:10Z" } 2024-04-12T03:31:10.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892670.426988_169.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:31:10.426988Z', 'realm_epoch': 2}}] 2024-04-12T03:31:10.648 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892670.426988_169.1'} 2024-04-12T03:31:10.648 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892670.426988_169.1'} 2024-04-12T03:31:10.649 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:31:10.649 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:10.649 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:10.649 DEBUG:boto:Method: HEAD 2024-04-12T03:31:10.649 DEBUG:boto:Path: /dkyges-16/ 2024-04-12T03:31:10.649 DEBUG:boto:Data: 2024-04-12T03:31:10.649 DEBUG:boto:Headers: {} 2024-04-12T03:31:10.649 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.649 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.649 DEBUG:boto:Params: {} 2024-04-12T03:31:10.650 DEBUG:boto:Token: None 2024-04-12T03:31:10.650 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/ 2024-04-12T03:31:10.650 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GGruYTfVc04WvxXuf+IjD5cM1UA= 2024-04-12T03:31:10.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GGruYTfVc04WvxXuf+IjD5cM1UA=', 'Content-Length': '0'} 2024-04-12T03:31:10.652 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccc2035df7d175b0-006618aafe-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.653 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:10.653 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:10.653 DEBUG:boto:Method: PUT 2024-04-12T03:31:10.653 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:31:10.653 DEBUG:boto:Data: 2024-04-12T03:31:10.653 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:31:10.653 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:10.653 DEBUG:boto:Port: 8000 2024-04-12T03:31:10.654 DEBUG:boto:Params: {} 2024-04-12T03:31:10.654 DEBUG:boto:Token: None 2024-04-12T03:31:10.654 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:31:10 GMT /dkyges-16/obj 2024-04-12T03:31:10.654 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jfwf2+diFmzVV5hfBuGn5zwJV6I= 2024-04-12T03:31:10.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:31:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jfwf2+diFmzVV5hfBuGn5zwJV6I='} 2024-04-12T03:31:10.665 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c89abbfb2af446c-006618aafe-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:10.665 DEBUG:rgw_multi.tests:created null version id=None 2024-04-12T03:31:10.665 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-16 2024-04-12T03:31:10.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:10.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:10.666 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:31:10.729 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:10.729 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:10.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:10.767+0000 7f10b912f780 1 waiting for incremental sync to catch up: 2024-04-12T03:31:10.768 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000003.33.13, , ] 2024-04-12T03:31:10.768 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000005.37.18, , ] 2024-04-12T03:31:15.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:15.768+0000 7f10b912f780 1 waiting for incremental sync to catch up: 2024-04-12T03:31:15.770 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000003.33.13, , ] 2024-04-12T03:31:15.770 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000005.37.18, , ] 2024-04-12T03:31:20.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:20.770+0000 7f10b912f780 1 waiting for incremental sync to catch up: 2024-04-12T03:31:20.771 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000003.33.13, , ] 2024-04-12T03:31:20.771 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000005.37.18, , ] 2024-04-12T03:31:25.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:25.772+0000 7f10b912f780 1 bucket sync caught up with source: 2024-04-12T03:31:25.773 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000005.37.18, , ] 2024-04-12T03:31:25.773 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000005.37.18, , ] 2024-04-12T03:31:25.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:25.772+0000 7f10b912f780 0 bucket checkpoint complete 2024-04-12T03:31:25.790 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-16 2024-04-12T03:31:25.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:25.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:25.791 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:31:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:25.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:25.898+0000 7fd6a39ca780 1 waiting for incremental sync to catch up: 2024-04-12T03:31:25.899 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000003.26.13, , ] 2024-04-12T03:31:25.899 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000005.33.18, , ] 2024-04-12T03:31:30.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:30.899+0000 7fd6a39ca780 1 waiting for incremental sync to catch up: 2024-04-12T03:31:30.900 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000003.26.13, , ] 2024-04-12T03:31:30.900 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000005.33.18, , ] 2024-04-12T03:31:35.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:35.900+0000 7fd6a39ca780 1 waiting for incremental sync to catch up: 2024-04-12T03:31:35.901 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000003.26.13, , ] 2024-04-12T03:31:35.902 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000005.33.18, , ] 2024-04-12T03:31:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:40.901+0000 7fd6a39ca780 1 waiting for incremental sync to catch up: 2024-04-12T03:31:40.903 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000003.26.13, , ] 2024-04-12T03:31:40.903 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000005.33.18, , ] 2024-04-12T03:31:45.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:45.903+0000 7fd6a39ca780 1 bucket sync caught up with source: 2024-04-12T03:31:45.904 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000005.33.18, , ] 2024-04-12T03:31:45.904 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000005.33.18, , ] 2024-04-12T03:31:45.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:45.903+0000 7fd6a39ca780 0 bucket checkpoint complete 2024-04-12T03:31:45.921 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-16 zones={a2, a1} 2024-04-12T03:31:45.922 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:45.922 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:45.922 DEBUG:boto:Method: HEAD 2024-04-12T03:31:45.922 DEBUG:boto:Path: /dkyges-16/ 2024-04-12T03:31:45.922 DEBUG:boto:Data: 2024-04-12T03:31:45.922 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.923 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:45.923 DEBUG:boto:Port: 8001 2024-04-12T03:31:45.923 DEBUG:boto:Params: {} 2024-04-12T03:31:45.923 DEBUG:boto:Token: None 2024-04-12T03:31:45.923 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/ 2024-04-12T03:31:45.923 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:g/EZgr1CjoNYH5s7iCZrAy3bKvE= 2024-04-12T03:31:45.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:g/EZgr1CjoNYH5s7iCZrAy3bKvE=', 'Content-Length': '0'} 2024-04-12T03:31:45.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084ba2e77a39ae8ba-006618ab21-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.927 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:45.927 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:45.927 DEBUG:boto:Method: HEAD 2024-04-12T03:31:45.927 DEBUG:boto:Path: /dkyges-16/ 2024-04-12T03:31:45.927 DEBUG:boto:Data: 2024-04-12T03:31:45.927 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.927 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:45.927 DEBUG:boto:Port: 8000 2024-04-12T03:31:45.927 DEBUG:boto:Params: {} 2024-04-12T03:31:45.927 DEBUG:boto:Token: None 2024-04-12T03:31:45.928 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/ 2024-04-12T03:31:45.928 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:g/EZgr1CjoNYH5s7iCZrAy3bKvE= 2024-04-12T03:31:45.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:g/EZgr1CjoNYH5s7iCZrAy3bKvE=', 'Content-Length': '0'} 2024-04-12T03:31:45.930 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037038b66375c7adb-006618ab21-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.930 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:31:45.930 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:45.930 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:45.930 DEBUG:boto:path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:31:45.931 DEBUG:boto:auth_path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:31:45.931 DEBUG:boto:Method: GET 2024-04-12T03:31:45.931 DEBUG:boto:Path: /dkyges-16/?versions&max-keys=999 2024-04-12T03:31:45.931 DEBUG:boto:Data: 2024-04-12T03:31:45.931 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.931 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:45.931 DEBUG:boto:Port: 8001 2024-04-12T03:31:45.931 DEBUG:boto:Params: {} 2024-04-12T03:31:45.931 DEBUG:boto:Token: None 2024-04-12T03:31:45.932 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/?versions 2024-04-12T03:31:45.932 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hiMu5v0Vdnq6WTqzSrSw4c5Pe0g= 2024-04-12T03:31:45.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hiMu5v0Vdnq6WTqzSrSw4c5Pe0g=', 'Content-Length': '0'} 2024-04-12T03:31:45.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd6b662f76594bd6-006618ab21-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.934 DEBUG:boto:b'dkyges-16999falseobjnulltrue2024-04-12T03:31:10.655Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vbfalse2024-04-12T03:30:49.815Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:45.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:31:45.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:31:45.935 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:31:45.935 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:45.935 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:45.936 DEBUG:boto:path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:31:45.936 DEBUG:boto:auth_path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:31:45.936 DEBUG:boto:Method: GET 2024-04-12T03:31:45.936 DEBUG:boto:Path: /dkyges-16/?versions&max-keys=999 2024-04-12T03:31:45.936 DEBUG:boto:Data: 2024-04-12T03:31:45.936 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.936 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:45.936 DEBUG:boto:Port: 8000 2024-04-12T03:31:45.936 DEBUG:boto:Params: {} 2024-04-12T03:31:45.936 DEBUG:boto:Token: None 2024-04-12T03:31:45.937 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/?versions 2024-04-12T03:31:45.937 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hiMu5v0Vdnq6WTqzSrSw4c5Pe0g= 2024-04-12T03:31:45.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hiMu5v0Vdnq6WTqzSrSw4c5Pe0g=', 'Content-Length': '0'} 2024-04-12T03:31:45.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000969642356b52e9f8-006618ab21-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.939 DEBUG:boto:b'dkyges-16999falseobjnulltrue2024-04-12T03:31:10.655Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vbfalse2024-04-12T03:30:49.815Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:45.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:31:45.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:31:45.941 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:45.942 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:45.942 DEBUG:boto:path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:31:45.942 DEBUG:boto:auth_path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:31:45.942 DEBUG:boto:Method: GET 2024-04-12T03:31:45.942 DEBUG:boto:Path: /dkyges-16/?versions&max-keys=999 2024-04-12T03:31:45.942 DEBUG:boto:Data: 2024-04-12T03:31:45.942 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.942 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:45.942 DEBUG:boto:Port: 8001 2024-04-12T03:31:45.942 DEBUG:boto:Params: {} 2024-04-12T03:31:45.943 DEBUG:boto:Token: None 2024-04-12T03:31:45.943 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/?versions 2024-04-12T03:31:45.943 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hiMu5v0Vdnq6WTqzSrSw4c5Pe0g= 2024-04-12T03:31:45.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hiMu5v0Vdnq6WTqzSrSw4c5Pe0g=', 'Content-Length': '0'} 2024-04-12T03:31:45.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0d6c50804ebd57c-006618ab21-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.946 DEBUG:boto:b'dkyges-16999falseobjnulltrue2024-04-12T03:31:10.655Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vbfalse2024-04-12T03:30:49.815Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:45.946 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:45.946 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:45.946 DEBUG:boto:path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:31:45.947 DEBUG:boto:auth_path=/dkyges-16/?versions&max-keys=999 2024-04-12T03:31:45.947 DEBUG:boto:Method: GET 2024-04-12T03:31:45.947 DEBUG:boto:Path: /dkyges-16/?versions&max-keys=999 2024-04-12T03:31:45.947 DEBUG:boto:Data: 2024-04-12T03:31:45.947 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.947 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:45.947 DEBUG:boto:Port: 8000 2024-04-12T03:31:45.947 DEBUG:boto:Params: {} 2024-04-12T03:31:45.947 DEBUG:boto:Token: None 2024-04-12T03:31:45.948 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/?versions 2024-04-12T03:31:45.948 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hiMu5v0Vdnq6WTqzSrSw4c5Pe0g= 2024-04-12T03:31:45.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hiMu5v0Vdnq6WTqzSrSw4c5Pe0g=', 'Content-Length': '0'} 2024-04-12T03:31:45.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b0a119b57ee039c-006618ab21-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.950 DEBUG:boto:b'dkyges-16999falseobjnulltrue2024-04-12T03:31:10.655Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vbfalse2024-04-12T03:30:49.815Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:45.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:31:45.951 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:45.951 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:45.951 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:45.952 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:45.952 DEBUG:boto:Method: GET 2024-04-12T03:31:45.952 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:31:45.952 DEBUG:boto:Data: 2024-04-12T03:31:45.952 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.952 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:45.952 DEBUG:boto:Port: 8001 2024-04-12T03:31:45.952 DEBUG:boto:Params: {} 2024-04-12T03:31:45.952 DEBUG:boto:Token: None 2024-04-12T03:31:45.953 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:31:45.953 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7aXoaZ6ENfygBHF1DlAssSec3z8= 2024-04-12T03:31:45.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7aXoaZ6ENfygBHF1DlAssSec3z8=', 'Content-Length': '0'} 2024-04-12T03:31:45.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000188a1d3cc4e30574-006618ab21-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.955 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:45.955 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:45.955 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:45.955 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:45.955 DEBUG:boto:Method: GET 2024-04-12T03:31:45.955 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:31:45.955 DEBUG:boto:Data: 2024-04-12T03:31:45.955 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.955 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:45.955 DEBUG:boto:Port: 8000 2024-04-12T03:31:45.956 DEBUG:boto:Params: {} 2024-04-12T03:31:45.956 DEBUG:boto:Token: None 2024-04-12T03:31:45.956 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:31:45.956 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7aXoaZ6ENfygBHF1DlAssSec3z8= 2024-04-12T03:31:45.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7aXoaZ6ENfygBHF1DlAssSec3z8=', 'Content-Length': '0'} 2024-04-12T03:31:45.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000521b8a77d3d2c981-006618ab21-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.958 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:45.958 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:45.958 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:45.958 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:45.958 DEBUG:boto:Method: HEAD 2024-04-12T03:31:45.959 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:31:45.959 DEBUG:boto:Data: 2024-04-12T03:31:45.959 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.959 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:45.959 DEBUG:boto:Port: 8001 2024-04-12T03:31:45.959 DEBUG:boto:Params: {} 2024-04-12T03:31:45.959 DEBUG:boto:Token: None 2024-04-12T03:31:45.959 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:31:45.959 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Gp5w7vjQjEM3AKwtv8F0zzZ88yE= 2024-04-12T03:31:45.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Gp5w7vjQjEM3AKwtv8F0zzZ88yE=', 'Content-Length': '0'} 2024-04-12T03:31:45.961 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d04dcf1607798483-006618ab21-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.962 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:45.962 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:45.962 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:45.962 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:45.962 DEBUG:boto:Method: HEAD 2024-04-12T03:31:45.962 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:31:45.962 DEBUG:boto:Data: 2024-04-12T03:31:45.963 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.963 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:45.963 DEBUG:boto:Port: 8000 2024-04-12T03:31:45.963 DEBUG:boto:Params: {} 2024-04-12T03:31:45.963 DEBUG:boto:Token: None 2024-04-12T03:31:45.963 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:31:45.963 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Gp5w7vjQjEM3AKwtv8F0zzZ88yE= 2024-04-12T03:31:45.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Gp5w7vjQjEM3AKwtv8F0zzZ88yE=', 'Content-Length': '0'} 2024-04-12T03:31:45.965 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fb3f955da9d41b06-006618ab21-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:31:45.966 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:45.966 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:45.966 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:45.966 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:45.966 DEBUG:boto:Method: GET 2024-04-12T03:31:45.966 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:31:45.966 DEBUG:boto:Data: 2024-04-12T03:31:45.966 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.966 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:45.967 DEBUG:boto:Port: 8001 2024-04-12T03:31:45.967 DEBUG:boto:Params: {} 2024-04-12T03:31:45.967 DEBUG:boto:Token: None 2024-04-12T03:31:45.967 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:31:45.967 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7aXoaZ6ENfygBHF1DlAssSec3z8= 2024-04-12T03:31:45.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7aXoaZ6ENfygBHF1DlAssSec3z8=', 'Content-Length': '0'} 2024-04-12T03:31:45.969 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006a91251d13f446ef-006618ab21-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.969 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:45.969 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:45.969 DEBUG:boto:path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:45.969 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=null 2024-04-12T03:31:45.970 DEBUG:boto:Method: GET 2024-04-12T03:31:45.970 DEBUG:boto:Path: /dkyges-16/obj?versionId=null 2024-04-12T03:31:45.970 DEBUG:boto:Data: 2024-04-12T03:31:45.970 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.970 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:45.970 DEBUG:boto:Port: 8000 2024-04-12T03:31:45.970 DEBUG:boto:Params: {} 2024-04-12T03:31:45.970 DEBUG:boto:Token: None 2024-04-12T03:31:45.970 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/obj?versionId=null 2024-04-12T03:31:45.971 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7aXoaZ6ENfygBHF1DlAssSec3z8= 2024-04-12T03:31:45.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7aXoaZ6ENfygBHF1DlAssSec3z8=', 'Content-Length': '0'} 2024-04-12T03:31:45.972 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003f74761df01c430e-006618ab21-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.973 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:45.973 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:45.973 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:31:45.973 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:31:45.973 DEBUG:boto:Method: GET 2024-04-12T03:31:45.973 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:31:45.973 DEBUG:boto:Data: 2024-04-12T03:31:45.973 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.973 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:45.974 DEBUG:boto:Port: 8001 2024-04-12T03:31:45.974 DEBUG:boto:Params: {} 2024-04-12T03:31:45.974 DEBUG:boto:Token: None 2024-04-12T03:31:45.974 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/ 2024-04-12T03:31:45.974 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NmqyTITiOM1w6S9mXY5/ncFP3UU= 2024-04-12T03:31:45.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NmqyTITiOM1w6S9mXY5/ncFP3UU=', 'Content-Length': '0'} 2024-04-12T03:31:45.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2c89bdf55c43846-006618ab21-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.977 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:31:10.655Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:45.977 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:45.978 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:45.978 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:31:45.978 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:31:45.978 DEBUG:boto:Method: GET 2024-04-12T03:31:45.978 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:31:45.978 DEBUG:boto:Data: 2024-04-12T03:31:45.978 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.978 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:45.978 DEBUG:boto:Port: 8000 2024-04-12T03:31:45.978 DEBUG:boto:Params: {} 2024-04-12T03:31:45.979 DEBUG:boto:Token: None 2024-04-12T03:31:45.979 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/ 2024-04-12T03:31:45.979 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NmqyTITiOM1w6S9mXY5/ncFP3UU= 2024-04-12T03:31:45.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NmqyTITiOM1w6S9mXY5/ncFP3UU=', 'Content-Length': '0'} 2024-04-12T03:31:45.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a41b3ddb609fb59-006618ab21-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.981 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:31:10.655Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:45.982 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:45.982 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:45.982 DEBUG:boto:Method: HEAD 2024-04-12T03:31:45.982 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:31:45.982 DEBUG:boto:Data: 2024-04-12T03:31:45.982 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.982 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:45.982 DEBUG:boto:Port: 8001 2024-04-12T03:31:45.982 DEBUG:boto:Params: {} 2024-04-12T03:31:45.983 DEBUG:boto:Token: None 2024-04-12T03:31:45.983 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/obj 2024-04-12T03:31:45.983 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:25gBPyKU3LqdzvvDv6Fo6BHsEpw= 2024-04-12T03:31:45.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:25gBPyKU3LqdzvvDv6Fo6BHsEpw=', 'Content-Length': '0'} 2024-04-12T03:31:45.984 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f7a325e69191b846-006618ab21-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.985 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:45.985 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:45.985 DEBUG:boto:Method: HEAD 2024-04-12T03:31:45.985 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:31:45.985 DEBUG:boto:Data: 2024-04-12T03:31:45.985 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.985 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:45.985 DEBUG:boto:Port: 8000 2024-04-12T03:31:45.985 DEBUG:boto:Params: {} 2024-04-12T03:31:45.985 DEBUG:boto:Token: None 2024-04-12T03:31:45.986 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/obj 2024-04-12T03:31:45.986 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:25gBPyKU3LqdzvvDv6Fo6BHsEpw= 2024-04-12T03:31:45.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:25gBPyKU3LqdzvvDv6Fo6BHsEpw=', 'Content-Length': '0'} 2024-04-12T03:31:45.987 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005af8dee3a6007dd1-006618ab21-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:31:45.988 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:45.988 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:45.988 DEBUG:boto:Method: GET 2024-04-12T03:31:45.988 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:31:45.988 DEBUG:boto:Data: 2024-04-12T03:31:45.988 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.988 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:45.988 DEBUG:boto:Port: 8001 2024-04-12T03:31:45.988 DEBUG:boto:Params: {} 2024-04-12T03:31:45.989 DEBUG:boto:Token: None 2024-04-12T03:31:45.989 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/obj 2024-04-12T03:31:45.989 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/SwGAaM2XfzcdjwbxhIlw19+jMI= 2024-04-12T03:31:45.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/SwGAaM2XfzcdjwbxhIlw19+jMI=', 'Content-Length': '0'} 2024-04-12T03:31:45.991 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000a9ecb51d3f1c699-006618ab21-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.991 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:45.991 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:45.991 DEBUG:boto:Method: GET 2024-04-12T03:31:45.991 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:31:45.991 DEBUG:boto:Data: 2024-04-12T03:31:45.991 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.991 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:45.991 DEBUG:boto:Port: 8000 2024-04-12T03:31:45.991 DEBUG:boto:Params: {} 2024-04-12T03:31:45.992 DEBUG:boto:Token: None 2024-04-12T03:31:45.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/obj 2024-04-12T03:31:45.992 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/SwGAaM2XfzcdjwbxhIlw19+jMI= 2024-04-12T03:31:45.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/SwGAaM2XfzcdjwbxhIlw19+jMI=', 'Content-Length': '0'} 2024-04-12T03:31:45.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c7797d145422c0e0-006618ab21-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:45.994 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:45.994 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:45.994 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:31:45.994 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:31:45.994 DEBUG:boto:Method: GET 2024-04-12T03:31:45.994 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:31:45.994 DEBUG:boto:Data: 2024-04-12T03:31:45.995 DEBUG:boto:Headers: {} 2024-04-12T03:31:45.995 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:45.995 DEBUG:boto:Port: 8001 2024-04-12T03:31:45.995 DEBUG:boto:Params: {} 2024-04-12T03:31:45.995 DEBUG:boto:Token: None 2024-04-12T03:31:45.995 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:45 GMT /dkyges-16/ 2024-04-12T03:31:45.995 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NmqyTITiOM1w6S9mXY5/ncFP3UU= 2024-04-12T03:31:45.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NmqyTITiOM1w6S9mXY5/ncFP3UU=', 'Content-Length': '0'} 2024-04-12T03:31:45.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efdbaae3bdf5986d-006618ab21-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.042 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:31:10.655Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:46.043 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:46.043 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:46.043 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:31:46.043 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:31:46.043 DEBUG:boto:Method: GET 2024-04-12T03:31:46.043 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:31:46.043 DEBUG:boto:Data: 2024-04-12T03:31:46.043 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.043 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:46.044 DEBUG:boto:Port: 8000 2024-04-12T03:31:46.044 DEBUG:boto:Params: {} 2024-04-12T03:31:46.044 DEBUG:boto:Token: None 2024-04-12T03:31:46.044 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:46 GMT /dkyges-16/ 2024-04-12T03:31:46.044 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:k0YCT2XK3XfplAFlVyfB4msB3lc= 2024-04-12T03:31:46.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:k0YCT2XK3XfplAFlVyfB4msB3lc=', 'Content-Length': '0'} 2024-04-12T03:31:46.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6c99f40c2c9b898-006618ab22-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.047 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:31:10.655Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:46.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:31:46.047 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:46.047 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:46.047 DEBUG:boto:path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.047 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.048 DEBUG:boto:Method: GET 2024-04-12T03:31:46.048 DEBUG:boto:Path: /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.048 DEBUG:boto:Data: 2024-04-12T03:31:46.048 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.048 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:46.048 DEBUG:boto:Port: 8001 2024-04-12T03:31:46.048 DEBUG:boto:Params: {} 2024-04-12T03:31:46.048 DEBUG:boto:Token: None 2024-04-12T03:31:46.048 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:46 GMT /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.049 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3qQgwLLiMN/BcmS8N24htPDLQHs= 2024-04-12T03:31:46.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3qQgwLLiMN/BcmS8N24htPDLQHs=', 'Content-Length': '0'} 2024-04-12T03:31:46.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bbc714e1a73e7d21-006618ab22-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.051 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:46.051 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:46.051 DEBUG:boto:path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.051 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.051 DEBUG:boto:Method: GET 2024-04-12T03:31:46.051 DEBUG:boto:Path: /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.051 DEBUG:boto:Data: 2024-04-12T03:31:46.052 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.052 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:46.052 DEBUG:boto:Port: 8000 2024-04-12T03:31:46.052 DEBUG:boto:Params: {} 2024-04-12T03:31:46.052 DEBUG:boto:Token: None 2024-04-12T03:31:46.052 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:46 GMT /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.053 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3qQgwLLiMN/BcmS8N24htPDLQHs= 2024-04-12T03:31:46.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3qQgwLLiMN/BcmS8N24htPDLQHs=', 'Content-Length': '0'} 2024-04-12T03:31:46.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ad0febc52c42b4a6-006618ab22-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.055 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:46.055 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:46.055 DEBUG:boto:path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.055 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.055 DEBUG:boto:Method: HEAD 2024-04-12T03:31:46.055 DEBUG:boto:Path: /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.055 DEBUG:boto:Data: 2024-04-12T03:31:46.055 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.055 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:46.055 DEBUG:boto:Port: 8001 2024-04-12T03:31:46.056 DEBUG:boto:Params: {} 2024-04-12T03:31:46.056 DEBUG:boto:Token: None 2024-04-12T03:31:46.056 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:46 GMT /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.056 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Z+YMXRF04F3jwUzl+kq4P4ytMtM= 2024-04-12T03:31:46.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Z+YMXRF04F3jwUzl+kq4P4ytMtM=', 'Content-Length': '0'} 2024-04-12T03:31:46.058 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006f544909cfbef554-006618ab22-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.058 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:46.059 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:46.059 DEBUG:boto:path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.059 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.059 DEBUG:boto:Method: HEAD 2024-04-12T03:31:46.059 DEBUG:boto:Path: /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.059 DEBUG:boto:Data: 2024-04-12T03:31:46.059 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.059 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:46.059 DEBUG:boto:Port: 8000 2024-04-12T03:31:46.059 DEBUG:boto:Params: {} 2024-04-12T03:31:46.060 DEBUG:boto:Token: None 2024-04-12T03:31:46.060 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:46 GMT /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.060 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Z+YMXRF04F3jwUzl+kq4P4ytMtM= 2024-04-12T03:31:46.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Z+YMXRF04F3jwUzl+kq4P4ytMtM=', 'Content-Length': '0'} 2024-04-12T03:31:46.062 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003dd99f553d7191f4-006618ab22-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:31:46.062 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:46.062 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:46.062 DEBUG:boto:path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.062 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.062 DEBUG:boto:Method: GET 2024-04-12T03:31:46.063 DEBUG:boto:Path: /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.063 DEBUG:boto:Data: 2024-04-12T03:31:46.063 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.063 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:46.063 DEBUG:boto:Port: 8001 2024-04-12T03:31:46.063 DEBUG:boto:Params: {} 2024-04-12T03:31:46.063 DEBUG:boto:Token: None 2024-04-12T03:31:46.063 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:46 GMT /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.063 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3qQgwLLiMN/BcmS8N24htPDLQHs= 2024-04-12T03:31:46.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3qQgwLLiMN/BcmS8N24htPDLQHs=', 'Content-Length': '0'} 2024-04-12T03:31:46.065 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000006977c24ec511a1b-006618ab22-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.065 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:46.066 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:46.066 DEBUG:boto:path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.066 DEBUG:boto:auth_path=/dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.066 DEBUG:boto:Method: GET 2024-04-12T03:31:46.066 DEBUG:boto:Path: /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.066 DEBUG:boto:Data: 2024-04-12T03:31:46.066 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.066 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:46.066 DEBUG:boto:Port: 8000 2024-04-12T03:31:46.066 DEBUG:boto:Params: {} 2024-04-12T03:31:46.067 DEBUG:boto:Token: None 2024-04-12T03:31:46.067 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:46 GMT /dkyges-16/obj?versionId=ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb 2024-04-12T03:31:46.067 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3qQgwLLiMN/BcmS8N24htPDLQHs= 2024-04-12T03:31:46.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3qQgwLLiMN/BcmS8N24htPDLQHs=', 'Content-Length': '0'} 2024-04-12T03:31:46.069 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:30:49 GMT'), ('x-amz-version-id', 'ujkyi1n5OiJWNKf8JjuDMMZA6HH8-Vb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000775adb8dfe9ea21-006618ab22-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.069 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:46.069 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:46.069 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:31:46.069 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:31:46.069 DEBUG:boto:Method: GET 2024-04-12T03:31:46.069 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:31:46.069 DEBUG:boto:Data: 2024-04-12T03:31:46.070 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.070 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:46.070 DEBUG:boto:Port: 8001 2024-04-12T03:31:46.070 DEBUG:boto:Params: {} 2024-04-12T03:31:46.070 DEBUG:boto:Token: None 2024-04-12T03:31:46.070 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:46 GMT /dkyges-16/ 2024-04-12T03:31:46.070 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:k0YCT2XK3XfplAFlVyfB4msB3lc= 2024-04-12T03:31:46.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:k0YCT2XK3XfplAFlVyfB4msB3lc=', 'Content-Length': '0'} 2024-04-12T03:31:46.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de45c178722e0ad0-006618ab22-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.073 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:31:10.655Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:46.073 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:46.074 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:46.074 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:31:46.074 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:31:46.074 DEBUG:boto:Method: GET 2024-04-12T03:31:46.074 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:31:46.074 DEBUG:boto:Data: 2024-04-12T03:31:46.074 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.074 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:46.074 DEBUG:boto:Port: 8000 2024-04-12T03:31:46.074 DEBUG:boto:Params: {} 2024-04-12T03:31:46.075 DEBUG:boto:Token: None 2024-04-12T03:31:46.075 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:46 GMT /dkyges-16/ 2024-04-12T03:31:46.075 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:k0YCT2XK3XfplAFlVyfB4msB3lc= 2024-04-12T03:31:46.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:k0YCT2XK3XfplAFlVyfB4msB3lc=', 'Content-Length': '0'} 2024-04-12T03:31:46.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039634aa165e4f558-006618ab22-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.119 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:31:10.655Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:46.119 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:46.119 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:46.119 DEBUG:boto:Method: HEAD 2024-04-12T03:31:46.119 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:31:46.120 DEBUG:boto:Data: 2024-04-12T03:31:46.120 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.120 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:46.120 DEBUG:boto:Port: 8001 2024-04-12T03:31:46.120 DEBUG:boto:Params: {} 2024-04-12T03:31:46.120 DEBUG:boto:Token: None 2024-04-12T03:31:46.120 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:46 GMT /dkyges-16/obj 2024-04-12T03:31:46.120 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LX9v6x/5RZDvYseDH9n/McN/oo4= 2024-04-12T03:31:46.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LX9v6x/5RZDvYseDH9n/McN/oo4=', 'Content-Length': '0'} 2024-04-12T03:31:46.123 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ee19225aafdc0405-006618ab22-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.124 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:46.124 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:46.124 DEBUG:boto:Method: HEAD 2024-04-12T03:31:46.124 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:31:46.124 DEBUG:boto:Data: 2024-04-12T03:31:46.124 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.124 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:46.124 DEBUG:boto:Port: 8000 2024-04-12T03:31:46.124 DEBUG:boto:Params: {} 2024-04-12T03:31:46.124 DEBUG:boto:Token: None 2024-04-12T03:31:46.125 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:46 GMT /dkyges-16/obj 2024-04-12T03:31:46.125 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LX9v6x/5RZDvYseDH9n/McN/oo4= 2024-04-12T03:31:46.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LX9v6x/5RZDvYseDH9n/McN/oo4=', 'Content-Length': '0'} 2024-04-12T03:31:46.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007b66ea8255c2580b-006618ab22-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:31:46.127 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:46.127 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:46.127 DEBUG:boto:Method: GET 2024-04-12T03:31:46.127 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:31:46.127 DEBUG:boto:Data: 2024-04-12T03:31:46.127 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.127 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:46.127 DEBUG:boto:Port: 8001 2024-04-12T03:31:46.128 DEBUG:boto:Params: {} 2024-04-12T03:31:46.128 DEBUG:boto:Token: None 2024-04-12T03:31:46.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:46 GMT /dkyges-16/obj 2024-04-12T03:31:46.128 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MxbVQ9QC770BU7X5OBzOx66TlQE= 2024-04-12T03:31:46.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MxbVQ9QC770BU7X5OBzOx66TlQE=', 'Content-Length': '0'} 2024-04-12T03:31:46.129 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c0ab93f4fe5279f0-006618ab22-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.130 DEBUG:boto:path=/dkyges-16/obj 2024-04-12T03:31:46.130 DEBUG:boto:auth_path=/dkyges-16/obj 2024-04-12T03:31:46.130 DEBUG:boto:Method: GET 2024-04-12T03:31:46.130 DEBUG:boto:Path: /dkyges-16/obj 2024-04-12T03:31:46.130 DEBUG:boto:Data: 2024-04-12T03:31:46.130 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.130 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:46.130 DEBUG:boto:Port: 8000 2024-04-12T03:31:46.131 DEBUG:boto:Params: {} 2024-04-12T03:31:46.131 DEBUG:boto:Token: None 2024-04-12T03:31:46.131 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:46 GMT /dkyges-16/obj 2024-04-12T03:31:46.131 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MxbVQ9QC770BU7X5OBzOx66TlQE= 2024-04-12T03:31:46.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MxbVQ9QC770BU7X5OBzOx66TlQE=', 'Content-Length': '0'} 2024-04-12T03:31:46.133 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:10 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000edf9f98834f67f8c-006618ab22-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.133 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:46.133 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:46.133 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:31:46.133 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:31:46.134 DEBUG:boto:Method: GET 2024-04-12T03:31:46.134 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:31:46.134 DEBUG:boto:Data: 2024-04-12T03:31:46.134 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.134 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:46.134 DEBUG:boto:Port: 8001 2024-04-12T03:31:46.134 DEBUG:boto:Params: {} 2024-04-12T03:31:46.134 DEBUG:boto:Token: None 2024-04-12T03:31:46.134 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:46 GMT /dkyges-16/ 2024-04-12T03:31:46.134 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:k0YCT2XK3XfplAFlVyfB4msB3lc= 2024-04-12T03:31:46.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:k0YCT2XK3XfplAFlVyfB4msB3lc=', 'Content-Length': '0'} 2024-04-12T03:31:46.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000490759aedf05cb17-006618ab22-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.178 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:31:10.655Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:46.179 DEBUG:boto:path=/dkyges-16/ 2024-04-12T03:31:46.179 DEBUG:boto:auth_path=/dkyges-16/ 2024-04-12T03:31:46.179 DEBUG:boto:path=/dkyges-16/?prefix=obj 2024-04-12T03:31:46.179 DEBUG:boto:auth_path=/dkyges-16/?prefix=obj 2024-04-12T03:31:46.179 DEBUG:boto:Method: GET 2024-04-12T03:31:46.179 DEBUG:boto:Path: /dkyges-16/?prefix=obj 2024-04-12T03:31:46.179 DEBUG:boto:Data: 2024-04-12T03:31:46.180 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.180 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:46.180 DEBUG:boto:Port: 8000 2024-04-12T03:31:46.180 DEBUG:boto:Params: {} 2024-04-12T03:31:46.180 DEBUG:boto:Token: None 2024-04-12T03:31:46.180 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:31:46 GMT /dkyges-16/ 2024-04-12T03:31:46.180 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:k0YCT2XK3XfplAFlVyfB4msB3lc= 2024-04-12T03:31:46.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:k0YCT2XK3XfplAFlVyfB4msB3lc=', 'Content-Length': '0'} 2024-04-12T03:31:46.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d788508e8ea3a242-006618ab22-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.183 DEBUG:boto:b'dkyges-16obj1000falseobj2024-04-12T03:31:10.655Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:31:46.184 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-16 zones={a2, a1} 2024-04-12T03:31:46.184 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2024-04-12T03:31:46.185 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-17 2024-04-12T03:31:46.185 DEBUG:boto:path=/dkyges-17/ 2024-04-12T03:31:46.185 DEBUG:boto:auth_path=/dkyges-17/ 2024-04-12T03:31:46.186 DEBUG:boto:Method: PUT 2024-04-12T03:31:46.186 DEBUG:boto:Path: /dkyges-17/ 2024-04-12T03:31:46.186 DEBUG:boto:Data: 2024-04-12T03:31:46.186 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.186 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:46.186 DEBUG:boto:Port: 8000 2024-04-12T03:31:46.186 DEBUG:boto:Params: {} 2024-04-12T03:31:46.186 DEBUG:boto:Token: None 2024-04-12T03:31:46.186 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:31:46 GMT /dkyges-17/ 2024-04-12T03:31:46.187 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cHACNWNBWQk0KHB3l4AAOcXUBQw= 2024-04-12T03:31:46.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cHACNWNBWQk0KHB3l4AAOcXUBQw=', 'Content-Length': '0'} 2024-04-12T03:31:46.203 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000097158830b0ecd7ae-006618ab22-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.203 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-18 2024-04-12T03:31:46.203 DEBUG:boto:path=/dkyges-18/ 2024-04-12T03:31:46.203 DEBUG:boto:auth_path=/dkyges-18/ 2024-04-12T03:31:46.204 DEBUG:boto:Method: PUT 2024-04-12T03:31:46.204 DEBUG:boto:Path: /dkyges-18/ 2024-04-12T03:31:46.204 DEBUG:boto:Data: 2024-04-12T03:31:46.204 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.204 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:46.204 DEBUG:boto:Port: 8001 2024-04-12T03:31:46.204 DEBUG:boto:Params: {} 2024-04-12T03:31:46.204 DEBUG:boto:Token: None 2024-04-12T03:31:46.204 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:31:46 GMT /dkyges-18/ 2024-04-12T03:31:46.204 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cVWTCX/kXFzvx6C5QSC9ArYyPwc= 2024-04-12T03:31:46.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cVWTCX/kXFzvx6C5QSC9ArYyPwc=', 'Content-Length': '0'} 2024-04-12T03:31:46.231 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000008af344c0a682bb36-006618ab22-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.231 DEBUG:boto:path=/dkyges-17/ 2024-04-12T03:31:46.231 DEBUG:boto:auth_path=/dkyges-17/ 2024-04-12T03:31:46.231 DEBUG:boto:path=/dkyges-17/?versioning 2024-04-12T03:31:46.231 DEBUG:boto:auth_path=/dkyges-17/?versioning 2024-04-12T03:31:46.232 DEBUG:boto:Method: PUT 2024-04-12T03:31:46.232 DEBUG:boto:Path: /dkyges-17/?versioning 2024-04-12T03:31:46.232 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:31:46.232 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.232 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:46.232 DEBUG:boto:Port: 8000 2024-04-12T03:31:46.232 DEBUG:boto:Params: {} 2024-04-12T03:31:46.232 DEBUG:boto:Token: None 2024-04-12T03:31:46.232 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:31:46 GMT /dkyges-17/?versioning 2024-04-12T03:31:46.233 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:s/awkAi39YZL1elvIW2moig6Kao= 2024-04-12T03:31:46.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:s/awkAi39YZL1elvIW2moig6Kao=', 'Content-Length': '228'} 2024-04-12T03:31:46.238 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000840215c7b9c1b51d-006618ab22-110f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.238 DEBUG:boto:path=/dkyges-18/ 2024-04-12T03:31:46.239 DEBUG:boto:auth_path=/dkyges-18/ 2024-04-12T03:31:46.239 DEBUG:boto:path=/dkyges-18/?versioning 2024-04-12T03:31:46.239 DEBUG:boto:auth_path=/dkyges-18/?versioning 2024-04-12T03:31:46.239 DEBUG:boto:Method: PUT 2024-04-12T03:31:46.239 DEBUG:boto:Path: /dkyges-18/?versioning 2024-04-12T03:31:46.239 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:31:46.239 DEBUG:boto:Headers: {} 2024-04-12T03:31:46.239 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:46.239 DEBUG:boto:Port: 8001 2024-04-12T03:31:46.239 DEBUG:boto:Params: {} 2024-04-12T03:31:46.239 DEBUG:boto:Token: None 2024-04-12T03:31:46.240 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:31:46 GMT /dkyges-18/?versioning 2024-04-12T03:31:46.240 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qkdO59Amb9opoMr+GdUWy6pHFTg= 2024-04-12T03:31:46.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qkdO59Amb9opoMr+GdUWy6pHFTg=', 'Content-Length': '228'} 2024-04-12T03:31:46.247 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fe85297336b7ed18-006618ab22-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:46.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:46.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:46.248 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:31:46.304 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:46.304 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:46.341 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:31:46.358 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:31:46.358 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:46.358 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:31:46.359 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:31:46.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:46.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:46.359 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:31:46.359 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:31:46.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:46.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:46.359 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:31:46.359 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:31:46.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:46.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:46.359 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892706.245340_181.1", 2024-04-12T03:31:46.360 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:31:46.245340Z" 2024-04-12T03:31:46.360 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:46.360 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:31:46.361 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892706.245340_181.1'} 2024-04-12T03:31:46.361 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:31:46.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:46.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:46.361 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:31:46.420 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:46.420 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:46.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:46.450+0000 7fa8e9f04780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:31:46.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:46.451+0000 7fa8e9f04780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:31:46.466 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:31:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:31:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:31:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:31:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:31:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:31:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:31:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:31:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:31:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:31:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:31:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:31:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:31:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:31:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:31:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:31:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:31:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:31:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:31:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:31:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:31:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:31:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:31:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:31:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:31:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:31:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:46.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:46.469 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:31:46.469 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:31:46.469 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:31:46.469 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:31:46.469 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:31:46.469 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:31:46.469 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:31:46.469 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:31:46.469 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:31:46.469 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:31:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:31:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:31:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892706.193683_170.1", 2024-04-12T03:31:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:31:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:31:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:31:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:31:46.193683Z", 2024-04-12T03:31:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:31:46.471 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:46.471 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:46.471 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:31:46.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:46.471 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:31:46.471 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:31:46.471 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:31:46.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:46.471 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:31:46Z" 2024-04-12T03:31:46.471 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:31:46.472 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892706.193683_170.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:31:46.193683Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:31:46Z" } 2024-04-12T03:31:46.472 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892706.193683_170.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:31:46.193683Z', 'realm_epoch': 2}}] 2024-04-12T03:31:46.472 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892706.245340_181.1'} 2024-04-12T03:31:46.473 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892706.193683_170.1'} 2024-04-12T03:31:46.473 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892706.245340_181.1 target=1_1712892706.193683_170.1 2024-04-12T03:31:51.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:51.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:31:51.475 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:31:51.535 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:51.535 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:51.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:51.571+0000 7f79045f0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:31:51.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:51.572+0000 7f79045f0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:31:51.587 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:31:51.587 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:31:51.587 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:31:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:31:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:31:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:31:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:31:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:31:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:31:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:31:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:31:51.588 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:31:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:31:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:31:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:31:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:31:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:31:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:31:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:31:51.589 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:31:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:31:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:31:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:31:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:31:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:31:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:31:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:31:51.590 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:31:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:31:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:31:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:31:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:31:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:31:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:31:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:31:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:31:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:31:51.591 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:31:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:31:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892706.245340_181.1", 2024-04-12T03:31:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:31:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:31:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:31:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:31:46.245340Z", 2024-04-12T03:31:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:31:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:31:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:31:51.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:51.593 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:31:51.593 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:31:51.593 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:31:51.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:31:51.593 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:31:51Z" 2024-04-12T03:31:51.593 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:31:51.594 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892706.245340_181.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:31:46.245340Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:31:51Z" } 2024-04-12T03:31:51.594 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892706.245340_181.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:31:46.245340Z', 'realm_epoch': 2}}] 2024-04-12T03:31:51.594 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892706.245340_181.1'} 2024-04-12T03:31:51.594 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892706.245340_181.1'} 2024-04-12T03:31:51.594 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:31:51.595 DEBUG:boto:path=/dkyges-17/ 2024-04-12T03:31:51.595 DEBUG:boto:auth_path=/dkyges-17/ 2024-04-12T03:31:51.595 DEBUG:boto:Method: HEAD 2024-04-12T03:31:51.595 DEBUG:boto:Path: /dkyges-17/ 2024-04-12T03:31:51.595 DEBUG:boto:Data: 2024-04-12T03:31:51.595 DEBUG:boto:Headers: {} 2024-04-12T03:31:51.595 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:51.595 DEBUG:boto:Port: 8000 2024-04-12T03:31:51.595 DEBUG:boto:Params: {} 2024-04-12T03:31:51.595 DEBUG:boto:Token: None 2024-04-12T03:31:51.596 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:51 GMT /dkyges-17/ 2024-04-12T03:31:51.596 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DW7phCzMhk7p3RIK5BqdmMdQ+Y8= 2024-04-12T03:31:51.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DW7phCzMhk7p3RIK5BqdmMdQ+Y8=', 'Content-Length': '0'} 2024-04-12T03:31:51.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077cfef01896a9e12-006618ab27-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:51.599 DEBUG:boto:path=/dkyges-17/obj 2024-04-12T03:31:51.599 DEBUG:boto:auth_path=/dkyges-17/obj 2024-04-12T03:31:51.599 DEBUG:boto:Method: PUT 2024-04-12T03:31:51.599 DEBUG:boto:Path: /dkyges-17/obj 2024-04-12T03:31:51.599 DEBUG:boto:Data: 2024-04-12T03:31:51.599 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:31:51.599 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:51.599 DEBUG:boto:Port: 8000 2024-04-12T03:31:51.599 DEBUG:boto:Params: {} 2024-04-12T03:31:51.599 DEBUG:boto:Token: None 2024-04-12T03:31:51.600 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:31:51 GMT /dkyges-17/obj 2024-04-12T03:31:51.600 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vaqiEs8i10fxCRGktig/D+mSqP0= 2024-04-12T03:31:51.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:31:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vaqiEs8i10fxCRGktig/D+mSqP0='} 2024-04-12T03:31:51.611 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc'), ('x-amz-request-id', 'tx0000042643625f8ae63ae-006618ab27-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:51.611 DEBUG:boto:path=/dkyges-17/ 2024-04-12T03:31:51.611 DEBUG:boto:auth_path=/dkyges-17/ 2024-04-12T03:31:51.611 DEBUG:boto:Method: HEAD 2024-04-12T03:31:51.611 DEBUG:boto:Path: /dkyges-17/ 2024-04-12T03:31:51.612 DEBUG:boto:Data: 2024-04-12T03:31:51.612 DEBUG:boto:Headers: {} 2024-04-12T03:31:51.612 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:51.612 DEBUG:boto:Port: 8000 2024-04-12T03:31:51.612 DEBUG:boto:Params: {} 2024-04-12T03:31:51.612 DEBUG:boto:Token: None 2024-04-12T03:31:51.612 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:51 GMT /dkyges-17/ 2024-04-12T03:31:51.612 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DW7phCzMhk7p3RIK5BqdmMdQ+Y8= 2024-04-12T03:31:51.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DW7phCzMhk7p3RIK5BqdmMdQ+Y8=', 'Content-Length': '0'} 2024-04-12T03:31:51.614 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a65079d0c19b9f8d-006618ab27-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:51.614 DEBUG:boto:path=/dkyges-17/obj 2024-04-12T03:31:51.615 DEBUG:boto:auth_path=/dkyges-17/obj 2024-04-12T03:31:51.615 DEBUG:boto:Method: DELETE 2024-04-12T03:31:51.615 DEBUG:boto:Path: /dkyges-17/obj 2024-04-12T03:31:51.615 DEBUG:boto:Data: 2024-04-12T03:31:51.615 DEBUG:boto:Headers: {} 2024-04-12T03:31:51.615 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:31:51.615 DEBUG:boto:Port: 8000 2024-04-12T03:31:51.615 DEBUG:boto:Params: {} 2024-04-12T03:31:51.615 DEBUG:boto:Token: None 2024-04-12T03:31:51.616 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:31:51 GMT /dkyges-17/obj 2024-04-12T03:31:51.616 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XbIvZPsyVmik29oDiBPDN+Csw9w= 2024-04-12T03:31:51.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XbIvZPsyVmik29oDiBPDN+Csw9w=', 'Content-Length': '0'} 2024-04-12T03:31:51.622 DEBUG:boto:Response headers: [('x-amz-version-id', 'Mv.1hpEcMojSBrlrpzaJE0cpIoNogN5'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx00000c117d51f02f5ce83-006618ab27-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:51.623 DEBUG:boto:path=/dkyges-18/ 2024-04-12T03:31:51.623 DEBUG:boto:auth_path=/dkyges-18/ 2024-04-12T03:31:51.623 DEBUG:boto:Method: HEAD 2024-04-12T03:31:51.623 DEBUG:boto:Path: /dkyges-18/ 2024-04-12T03:31:51.623 DEBUG:boto:Data: 2024-04-12T03:31:51.623 DEBUG:boto:Headers: {} 2024-04-12T03:31:51.623 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:51.623 DEBUG:boto:Port: 8001 2024-04-12T03:31:51.623 DEBUG:boto:Params: {} 2024-04-12T03:31:51.624 DEBUG:boto:Token: None 2024-04-12T03:31:51.624 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:51 GMT /dkyges-18/ 2024-04-12T03:31:51.624 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SaR8kiFUZKdFRyT6uXTAqbQd6yQ= 2024-04-12T03:31:51.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SaR8kiFUZKdFRyT6uXTAqbQd6yQ=', 'Content-Length': '0'} 2024-04-12T03:31:51.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074c6e71809c1d4f6-006618ab27-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:51.626 DEBUG:boto:path=/dkyges-18/obj 2024-04-12T03:31:51.627 DEBUG:boto:auth_path=/dkyges-18/obj 2024-04-12T03:31:51.627 DEBUG:boto:Method: PUT 2024-04-12T03:31:51.627 DEBUG:boto:Path: /dkyges-18/obj 2024-04-12T03:31:51.627 DEBUG:boto:Data: 2024-04-12T03:31:51.627 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:31:51.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:51.627 DEBUG:boto:Port: 8001 2024-04-12T03:31:51.627 DEBUG:boto:Params: {} 2024-04-12T03:31:51.627 DEBUG:boto:Token: None 2024-04-12T03:31:51.627 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:31:51 GMT /dkyges-18/obj 2024-04-12T03:31:51.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NLEbIffffh6GzkbijUSCjorIA74= 2024-04-12T03:31:51.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:31:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NLEbIffffh6GzkbijUSCjorIA74='} 2024-04-12T03:31:51.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy'), ('x-amz-request-id', 'tx00000f62e91b47e094a54-006618ab27-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:51.671 DEBUG:boto:path=/dkyges-18/ 2024-04-12T03:31:51.671 DEBUG:boto:auth_path=/dkyges-18/ 2024-04-12T03:31:51.671 DEBUG:boto:Method: HEAD 2024-04-12T03:31:51.671 DEBUG:boto:Path: /dkyges-18/ 2024-04-12T03:31:51.671 DEBUG:boto:Data: 2024-04-12T03:31:51.671 DEBUG:boto:Headers: {} 2024-04-12T03:31:51.671 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:51.672 DEBUG:boto:Port: 8001 2024-04-12T03:31:51.672 DEBUG:boto:Params: {} 2024-04-12T03:31:51.672 DEBUG:boto:Token: None 2024-04-12T03:31:51.672 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:31:51 GMT /dkyges-18/ 2024-04-12T03:31:51.672 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SaR8kiFUZKdFRyT6uXTAqbQd6yQ= 2024-04-12T03:31:51.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SaR8kiFUZKdFRyT6uXTAqbQd6yQ=', 'Content-Length': '0'} 2024-04-12T03:31:51.674 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbb5d7e3d00de83b-006618ab27-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:51.675 DEBUG:boto:path=/dkyges-18/obj 2024-04-12T03:31:51.675 DEBUG:boto:auth_path=/dkyges-18/obj 2024-04-12T03:31:51.675 DEBUG:boto:Method: DELETE 2024-04-12T03:31:51.675 DEBUG:boto:Path: /dkyges-18/obj 2024-04-12T03:31:51.675 DEBUG:boto:Data: 2024-04-12T03:31:51.675 DEBUG:boto:Headers: {} 2024-04-12T03:31:51.675 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:31:51.675 DEBUG:boto:Port: 8001 2024-04-12T03:31:51.675 DEBUG:boto:Params: {} 2024-04-12T03:31:51.675 DEBUG:boto:Token: None 2024-04-12T03:31:51.676 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:31:51 GMT /dkyges-18/obj 2024-04-12T03:31:51.676 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Yip876wxbcf+s33itzsytiAf2Ac= 2024-04-12T03:31:51.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:31:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Yip876wxbcf+s33itzsytiAf2Ac=', 'Content-Length': '0'} 2024-04-12T03:31:51.683 DEBUG:boto:Response headers: [('x-amz-version-id', '8WfLuNSvJpTfoqbI1a5Mlql48GPBA31'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000034304200c7ba501e-006618ab27-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:31:51.683 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-17 2024-04-12T03:31:51.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:51.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:31:51.684 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-17 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:31:51.749 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:31:51.749 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:31:51.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:51.786+0000 7f3b0ee25780 1 waiting to reach incremental sync.. 2024-04-12T03:31:56.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:31:56.787+0000 7f3b0ee25780 1 waiting to reach incremental sync.. 2024-04-12T03:32:01.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:32:01.788+0000 7f3b0ee25780 1 waiting to reach incremental sync.. 2024-04-12T03:32:06.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:32:06.790+0000 7f3b0ee25780 1 bucket sync caught up with source: 2024-04-12T03:32:06.791 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000003.10.12, , ] 2024-04-12T03:32:06.791 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000003.10.12, , ] 2024-04-12T03:32:06.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:32:06.790+0000 7f3b0ee25780 0 bucket checkpoint complete 2024-04-12T03:32:06.808 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-17 2024-04-12T03:32:06.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:32:06.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:32:06.808 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-17 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:32:06.873 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:06.874 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:06.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:32:06.912+0000 7f679f48c780 1 waiting to reach incremental sync.. 2024-04-12T03:32:11.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:32:11.913+0000 7f679f48c780 1 waiting to reach incremental sync.. 2024-04-12T03:32:16.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:32:16.913+0000 7f679f48c780 1 waiting to reach incremental sync.. 2024-04-12T03:32:21.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:32:21.914+0000 7f679f48c780 1 waiting to reach incremental sync.. 2024-04-12T03:32:26.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:32:26.916+0000 7f679f48c780 1 bucket sync caught up with source: 2024-04-12T03:32:26.918 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000002.18.6, , ] 2024-04-12T03:32:26.918 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000002.18.6, , ] 2024-04-12T03:32:26.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:32:26.916+0000 7f679f48c780 0 bucket checkpoint complete 2024-04-12T03:32:26.934 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-17 zones={a2, a1} 2024-04-12T03:32:26.935 DEBUG:boto:path=/dkyges-17/ 2024-04-12T03:32:26.935 DEBUG:boto:auth_path=/dkyges-17/ 2024-04-12T03:32:26.935 DEBUG:boto:Method: HEAD 2024-04-12T03:32:26.935 DEBUG:boto:Path: /dkyges-17/ 2024-04-12T03:32:26.935 DEBUG:boto:Data: 2024-04-12T03:32:26.935 DEBUG:boto:Headers: {} 2024-04-12T03:32:26.935 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:32:26.935 DEBUG:boto:Port: 8001 2024-04-12T03:32:26.935 DEBUG:boto:Params: {} 2024-04-12T03:32:26.936 DEBUG:boto:Token: None 2024-04-12T03:32:26.936 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:26 GMT /dkyges-17/ 2024-04-12T03:32:26.936 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:E8YpGRwQHZgIhvwfdgArIjRwctU= 2024-04-12T03:32:26.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:E8YpGRwQHZgIhvwfdgArIjRwctU=', 'Content-Length': '0'} 2024-04-12T03:32:26.939 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000694c9be023ea6300-006618ab4a-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:26.940 DEBUG:boto:path=/dkyges-17/ 2024-04-12T03:32:26.940 DEBUG:boto:auth_path=/dkyges-17/ 2024-04-12T03:32:26.940 DEBUG:boto:Method: HEAD 2024-04-12T03:32:26.940 DEBUG:boto:Path: /dkyges-17/ 2024-04-12T03:32:26.940 DEBUG:boto:Data: 2024-04-12T03:32:26.940 DEBUG:boto:Headers: {} 2024-04-12T03:32:26.940 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:32:26.940 DEBUG:boto:Port: 8000 2024-04-12T03:32:26.940 DEBUG:boto:Params: {} 2024-04-12T03:32:26.941 DEBUG:boto:Token: None 2024-04-12T03:32:26.941 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:26 GMT /dkyges-17/ 2024-04-12T03:32:26.941 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:E8YpGRwQHZgIhvwfdgArIjRwctU= 2024-04-12T03:32:26.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:E8YpGRwQHZgIhvwfdgArIjRwctU=', 'Content-Length': '0'} 2024-04-12T03:32:26.943 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000354175cd6434efd3-006618ab4a-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:32:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:26.944 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:32:26.944 DEBUG:boto:path=/dkyges-17/ 2024-04-12T03:32:26.944 DEBUG:boto:auth_path=/dkyges-17/ 2024-04-12T03:32:26.944 DEBUG:boto:path=/dkyges-17/?versions&max-keys=999 2024-04-12T03:32:26.944 DEBUG:boto:auth_path=/dkyges-17/?versions&max-keys=999 2024-04-12T03:32:26.944 DEBUG:boto:Method: GET 2024-04-12T03:32:26.944 DEBUG:boto:Path: /dkyges-17/?versions&max-keys=999 2024-04-12T03:32:26.944 DEBUG:boto:Data: 2024-04-12T03:32:26.944 DEBUG:boto:Headers: {} 2024-04-12T03:32:26.944 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:32:26.945 DEBUG:boto:Port: 8001 2024-04-12T03:32:26.945 DEBUG:boto:Params: {} 2024-04-12T03:32:26.945 DEBUG:boto:Token: None 2024-04-12T03:32:26.946 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:26 GMT /dkyges-17/?versions 2024-04-12T03:32:26.946 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S/hT5lBb13kQaM8SYjuf1PtiHxg= 2024-04-12T03:32:26.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S/hT5lBb13kQaM8SYjuf1PtiHxg=', 'Content-Length': '0'} 2024-04-12T03:32:26.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f61de304efd2fd08-006618ab4a-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:26.990 DEBUG:boto:b'dkyges-17999falseobjMv.1hpEcMojSBrlrpzaJE0cpIoNogN5true2024-04-12T03:31:51.618Zrgw-multisite-test-userMultisite Test UserNormalobjJjuPE9h1dYPFfGaIE-.zkwLYkVfftncfalse2024-04-12T03:31:51.602Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:26.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:32:26.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:32:26.992 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:32:26.992 DEBUG:boto:path=/dkyges-17/ 2024-04-12T03:32:26.992 DEBUG:boto:auth_path=/dkyges-17/ 2024-04-12T03:32:26.992 DEBUG:boto:path=/dkyges-17/?versions&max-keys=999 2024-04-12T03:32:26.992 DEBUG:boto:auth_path=/dkyges-17/?versions&max-keys=999 2024-04-12T03:32:26.992 DEBUG:boto:Method: GET 2024-04-12T03:32:26.992 DEBUG:boto:Path: /dkyges-17/?versions&max-keys=999 2024-04-12T03:32:26.993 DEBUG:boto:Data: 2024-04-12T03:32:26.993 DEBUG:boto:Headers: {} 2024-04-12T03:32:26.993 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:32:26.993 DEBUG:boto:Port: 8000 2024-04-12T03:32:26.993 DEBUG:boto:Params: {} 2024-04-12T03:32:26.993 DEBUG:boto:Token: None 2024-04-12T03:32:26.993 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:26 GMT /dkyges-17/?versions 2024-04-12T03:32:26.993 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S/hT5lBb13kQaM8SYjuf1PtiHxg= 2024-04-12T03:32:26.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S/hT5lBb13kQaM8SYjuf1PtiHxg=', 'Content-Length': '0'} 2024-04-12T03:32:26.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005137b7547cec95d4-006618ab4a-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:26.996 DEBUG:boto:b'dkyges-17999falseobjMv.1hpEcMojSBrlrpzaJE0cpIoNogN5true2024-04-12T03:31:51.618Zrgw-multisite-test-userMultisite Test UserNormalobjJjuPE9h1dYPFfGaIE-.zkwLYkVfftncfalse2024-04-12T03:31:51.602Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:26.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:32:26.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:32:26.997 DEBUG:boto:path=/dkyges-17/ 2024-04-12T03:32:26.997 DEBUG:boto:auth_path=/dkyges-17/ 2024-04-12T03:32:26.997 DEBUG:boto:path=/dkyges-17/?versions&max-keys=999 2024-04-12T03:32:26.997 DEBUG:boto:auth_path=/dkyges-17/?versions&max-keys=999 2024-04-12T03:32:26.997 DEBUG:boto:Method: GET 2024-04-12T03:32:26.997 DEBUG:boto:Path: /dkyges-17/?versions&max-keys=999 2024-04-12T03:32:26.997 DEBUG:boto:Data: 2024-04-12T03:32:26.997 DEBUG:boto:Headers: {} 2024-04-12T03:32:26.997 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:32:26.997 DEBUG:boto:Port: 8001 2024-04-12T03:32:26.997 DEBUG:boto:Params: {} 2024-04-12T03:32:26.998 DEBUG:boto:Token: None 2024-04-12T03:32:26.998 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:26 GMT /dkyges-17/?versions 2024-04-12T03:32:26.998 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S/hT5lBb13kQaM8SYjuf1PtiHxg= 2024-04-12T03:32:26.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S/hT5lBb13kQaM8SYjuf1PtiHxg=', 'Content-Length': '0'} 2024-04-12T03:32:27.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ffbf6f76e36cef8-006618ab4a-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:27.000 DEBUG:boto:b'dkyges-17999falseobjMv.1hpEcMojSBrlrpzaJE0cpIoNogN5true2024-04-12T03:31:51.618Zrgw-multisite-test-userMultisite Test UserNormalobjJjuPE9h1dYPFfGaIE-.zkwLYkVfftncfalse2024-04-12T03:31:51.602Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:27.001 DEBUG:boto:path=/dkyges-17/ 2024-04-12T03:32:27.001 DEBUG:boto:auth_path=/dkyges-17/ 2024-04-12T03:32:27.001 DEBUG:boto:path=/dkyges-17/?versions&max-keys=999 2024-04-12T03:32:27.001 DEBUG:boto:auth_path=/dkyges-17/?versions&max-keys=999 2024-04-12T03:32:27.001 DEBUG:boto:Method: GET 2024-04-12T03:32:27.001 DEBUG:boto:Path: /dkyges-17/?versions&max-keys=999 2024-04-12T03:32:27.001 DEBUG:boto:Data: 2024-04-12T03:32:27.001 DEBUG:boto:Headers: {} 2024-04-12T03:32:27.001 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:32:27.002 DEBUG:boto:Port: 8000 2024-04-12T03:32:27.002 DEBUG:boto:Params: {} 2024-04-12T03:32:27.002 DEBUG:boto:Token: None 2024-04-12T03:32:27.002 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:27 GMT /dkyges-17/?versions 2024-04-12T03:32:27.002 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+ZdxVAf+jOQGcScvFgeZqLv/K9U= 2024-04-12T03:32:27.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+ZdxVAf+jOQGcScvFgeZqLv/K9U=', 'Content-Length': '0'} 2024-04-12T03:32:27.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdee1b545ed387a5-006618ab4b-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:27.005 DEBUG:boto:b'dkyges-17999falseobjMv.1hpEcMojSBrlrpzaJE0cpIoNogN5true2024-04-12T03:31:51.618Zrgw-multisite-test-userMultisite Test UserNormalobjJjuPE9h1dYPFfGaIE-.zkwLYkVfftncfalse2024-04-12T03:31:51.602Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:32:27.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:32:27.005 DEBUG:boto:path=/dkyges-17/obj 2024-04-12T03:32:27.006 DEBUG:boto:auth_path=/dkyges-17/obj 2024-04-12T03:32:27.006 DEBUG:boto:Method: HEAD 2024-04-12T03:32:27.006 DEBUG:boto:Path: /dkyges-17/obj 2024-04-12T03:32:27.006 DEBUG:boto:Data: 2024-04-12T03:32:27.006 DEBUG:boto:Headers: {} 2024-04-12T03:32:27.006 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:32:27.006 DEBUG:boto:Port: 8001 2024-04-12T03:32:27.006 DEBUG:boto:Params: {} 2024-04-12T03:32:27.006 DEBUG:boto:Token: None 2024-04-12T03:32:27.006 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:27 GMT /dkyges-17/obj 2024-04-12T03:32:27.006 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4bryHLACadBPGBa1y+2GG7nCjTU= 2024-04-12T03:32:27.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4bryHLACadBPGBa1y+2GG7nCjTU=', 'Content-Length': '0'} 2024-04-12T03:32:27.008 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000cb6d12859e050c3d-006618ab4b-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:27.009 DEBUG:boto:path=/dkyges-17/obj 2024-04-12T03:32:27.009 DEBUG:boto:auth_path=/dkyges-17/obj 2024-04-12T03:32:27.009 DEBUG:boto:Method: HEAD 2024-04-12T03:32:27.009 DEBUG:boto:Path: /dkyges-17/obj 2024-04-12T03:32:27.009 DEBUG:boto:Data: 2024-04-12T03:32:27.009 DEBUG:boto:Headers: {} 2024-04-12T03:32:27.009 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:32:27.009 DEBUG:boto:Port: 8000 2024-04-12T03:32:27.009 DEBUG:boto:Params: {} 2024-04-12T03:32:27.010 DEBUG:boto:Token: None 2024-04-12T03:32:27.010 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:27 GMT /dkyges-17/obj 2024-04-12T03:32:27.010 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4bryHLACadBPGBa1y+2GG7nCjTU= 2024-04-12T03:32:27.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4bryHLACadBPGBa1y+2GG7nCjTU=', 'Content-Length': '0'} 2024-04-12T03:32:27.012 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000003fc64fbdea220ba2-006618ab4b-110f-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:27.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:32:27.012 DEBUG:boto:path=/dkyges-17/obj 2024-04-12T03:32:27.012 DEBUG:boto:auth_path=/dkyges-17/obj 2024-04-12T03:32:27.012 DEBUG:boto:path=/dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.012 DEBUG:boto:auth_path=/dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.012 DEBUG:boto:Method: GET 2024-04-12T03:32:27.012 DEBUG:boto:Path: /dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.013 DEBUG:boto:Data: 2024-04-12T03:32:27.013 DEBUG:boto:Headers: {} 2024-04-12T03:32:27.013 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:32:27.013 DEBUG:boto:Port: 8001 2024-04-12T03:32:27.013 DEBUG:boto:Params: {} 2024-04-12T03:32:27.013 DEBUG:boto:Token: None 2024-04-12T03:32:27.013 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:27 GMT /dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.013 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OiiyYgCD+hcDs/2hbTbPKK7Au8w= 2024-04-12T03:32:27.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OiiyYgCD+hcDs/2hbTbPKK7Au8w=', 'Content-Length': '0'} 2024-04-12T03:32:27.015 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('x-amz-version-id', 'JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000066687c3d11c9139-006618ab4b-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:27.015 DEBUG:boto:path=/dkyges-17/obj 2024-04-12T03:32:27.016 DEBUG:boto:auth_path=/dkyges-17/obj 2024-04-12T03:32:27.016 DEBUG:boto:path=/dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.016 DEBUG:boto:auth_path=/dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.016 DEBUG:boto:Method: GET 2024-04-12T03:32:27.016 DEBUG:boto:Path: /dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.016 DEBUG:boto:Data: 2024-04-12T03:32:27.016 DEBUG:boto:Headers: {} 2024-04-12T03:32:27.016 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:32:27.016 DEBUG:boto:Port: 8000 2024-04-12T03:32:27.016 DEBUG:boto:Params: {} 2024-04-12T03:32:27.016 DEBUG:boto:Token: None 2024-04-12T03:32:27.017 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:27 GMT /dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.017 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OiiyYgCD+hcDs/2hbTbPKK7Au8w= 2024-04-12T03:32:27.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OiiyYgCD+hcDs/2hbTbPKK7Au8w=', 'Content-Length': '0'} 2024-04-12T03:32:27.019 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('x-amz-version-id', 'JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008f6491b8e9a6291a-006618ab4b-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:27.019 DEBUG:boto:path=/dkyges-17/obj 2024-04-12T03:32:27.019 DEBUG:boto:auth_path=/dkyges-17/obj 2024-04-12T03:32:27.019 DEBUG:boto:path=/dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.019 DEBUG:boto:auth_path=/dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.019 DEBUG:boto:Method: HEAD 2024-04-12T03:32:27.019 DEBUG:boto:Path: /dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.019 DEBUG:boto:Data: 2024-04-12T03:32:27.019 DEBUG:boto:Headers: {} 2024-04-12T03:32:27.020 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:32:27.020 DEBUG:boto:Port: 8001 2024-04-12T03:32:27.020 DEBUG:boto:Params: {} 2024-04-12T03:32:27.020 DEBUG:boto:Token: None 2024-04-12T03:32:27.020 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:27 GMT /dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.020 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6://LIv3R2dKW4xC/XrAdwtiTV8c4= 2024-04-12T03:32:27.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6://LIv3R2dKW4xC/XrAdwtiTV8c4=', 'Content-Length': '0'} 2024-04-12T03:32:27.022 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('x-amz-version-id', 'JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000542462a170fe8416-006618ab4b-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:27.022 DEBUG:boto:path=/dkyges-17/obj 2024-04-12T03:32:27.022 DEBUG:boto:auth_path=/dkyges-17/obj 2024-04-12T03:32:27.022 DEBUG:boto:path=/dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.022 DEBUG:boto:auth_path=/dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.022 DEBUG:boto:Method: HEAD 2024-04-12T03:32:27.023 DEBUG:boto:Path: /dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.023 DEBUG:boto:Data: 2024-04-12T03:32:27.023 DEBUG:boto:Headers: {} 2024-04-12T03:32:27.023 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:32:27.023 DEBUG:boto:Port: 8000 2024-04-12T03:32:27.023 DEBUG:boto:Params: {} 2024-04-12T03:32:27.023 DEBUG:boto:Token: None 2024-04-12T03:32:27.023 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:27 GMT /dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.023 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6://LIv3R2dKW4xC/XrAdwtiTV8c4= 2024-04-12T03:32:27.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6://LIv3R2dKW4xC/XrAdwtiTV8c4=', 'Content-Length': '0'} 2024-04-12T03:32:27.025 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('x-amz-version-id', 'JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000738bbc6367cd1e06-006618ab4b-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:27.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:32:27.026 DEBUG:boto:path=/dkyges-17/obj 2024-04-12T03:32:27.026 DEBUG:boto:auth_path=/dkyges-17/obj 2024-04-12T03:32:27.026 DEBUG:boto:path=/dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.026 DEBUG:boto:auth_path=/dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.026 DEBUG:boto:Method: GET 2024-04-12T03:32:27.026 DEBUG:boto:Path: /dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.026 DEBUG:boto:Data: 2024-04-12T03:32:27.026 DEBUG:boto:Headers: {} 2024-04-12T03:32:27.026 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:32:27.027 DEBUG:boto:Port: 8001 2024-04-12T03:32:27.027 DEBUG:boto:Params: {} 2024-04-12T03:32:27.027 DEBUG:boto:Token: None 2024-04-12T03:32:27.027 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:27 GMT /dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.027 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OiiyYgCD+hcDs/2hbTbPKK7Au8w= 2024-04-12T03:32:27.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OiiyYgCD+hcDs/2hbTbPKK7Au8w=', 'Content-Length': '0'} 2024-04-12T03:32:27.029 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('x-amz-version-id', 'JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000042f46bf86de868f5-006618ab4b-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:27.029 DEBUG:boto:path=/dkyges-17/obj 2024-04-12T03:32:27.029 DEBUG:boto:auth_path=/dkyges-17/obj 2024-04-12T03:32:27.029 DEBUG:boto:path=/dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.029 DEBUG:boto:auth_path=/dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.029 DEBUG:boto:Method: GET 2024-04-12T03:32:27.029 DEBUG:boto:Path: /dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.029 DEBUG:boto:Data: 2024-04-12T03:32:27.030 DEBUG:boto:Headers: {} 2024-04-12T03:32:27.030 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:32:27.030 DEBUG:boto:Port: 8000 2024-04-12T03:32:27.030 DEBUG:boto:Params: {} 2024-04-12T03:32:27.030 DEBUG:boto:Token: None 2024-04-12T03:32:27.030 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:27 GMT /dkyges-17/obj?versionId=JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc 2024-04-12T03:32:27.030 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OiiyYgCD+hcDs/2hbTbPKK7Au8w= 2024-04-12T03:32:27.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OiiyYgCD+hcDs/2hbTbPKK7Au8w=', 'Content-Length': '0'} 2024-04-12T03:32:27.033 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('x-amz-version-id', 'JjuPE9h1dYPFfGaIE-.zkwLYkVfftnc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000667256e31f84f7f1-006618ab4b-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:27.033 DEBUG:boto:path=/dkyges-17/ 2024-04-12T03:32:27.033 DEBUG:boto:auth_path=/dkyges-17/ 2024-04-12T03:32:27.033 DEBUG:boto:path=/dkyges-17/?prefix=obj 2024-04-12T03:32:27.033 DEBUG:boto:auth_path=/dkyges-17/?prefix=obj 2024-04-12T03:32:27.033 DEBUG:boto:Method: GET 2024-04-12T03:32:27.034 DEBUG:boto:Path: /dkyges-17/?prefix=obj 2024-04-12T03:32:27.034 DEBUG:boto:Data: 2024-04-12T03:32:27.034 DEBUG:boto:Headers: {} 2024-04-12T03:32:27.034 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:32:27.034 DEBUG:boto:Port: 8001 2024-04-12T03:32:27.034 DEBUG:boto:Params: {} 2024-04-12T03:32:27.034 DEBUG:boto:Token: None 2024-04-12T03:32:27.034 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:27 GMT /dkyges-17/ 2024-04-12T03:32:27.034 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BI4tQS0/ZMtSQN/J07kwc/t7WIE= 2024-04-12T03:32:27.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BI4tQS0/ZMtSQN/J07kwc/t7WIE=', 'Content-Length': '0'} 2024-04-12T03:32:27.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d64d530803da849e-006618ab4b-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:27.037 DEBUG:boto:b'dkyges-17obj1000false' 2024-04-12T03:32:27.037 DEBUG:boto:path=/dkyges-17/ 2024-04-12T03:32:27.037 DEBUG:boto:auth_path=/dkyges-17/ 2024-04-12T03:32:27.038 DEBUG:boto:path=/dkyges-17/?prefix=obj 2024-04-12T03:32:27.038 DEBUG:boto:auth_path=/dkyges-17/?prefix=obj 2024-04-12T03:32:27.038 DEBUG:boto:Method: GET 2024-04-12T03:32:27.038 DEBUG:boto:Path: /dkyges-17/?prefix=obj 2024-04-12T03:32:27.038 DEBUG:boto:Data: 2024-04-12T03:32:27.038 DEBUG:boto:Headers: {} 2024-04-12T03:32:27.038 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:32:27.038 DEBUG:boto:Port: 8000 2024-04-12T03:32:27.038 DEBUG:boto:Params: {} 2024-04-12T03:32:27.038 DEBUG:boto:Token: None 2024-04-12T03:32:27.039 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:32:27 GMT /dkyges-17/ 2024-04-12T03:32:27.039 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BI4tQS0/ZMtSQN/J07kwc/t7WIE= 2024-04-12T03:32:27.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BI4tQS0/ZMtSQN/J07kwc/t7WIE=', 'Content-Length': '0'} 2024-04-12T03:32:27.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000543817388c19fcd0-006618ab4b-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:27.042 DEBUG:boto:b'dkyges-17obj1000false' 2024-04-12T03:32:27.042 DEBUG:boto:path=/dkyges-17/obj 2024-04-12T03:32:27.042 DEBUG:boto:auth_path=/dkyges-17/obj 2024-04-12T03:32:27.042 DEBUG:boto:Method: HEAD 2024-04-12T03:32:27.042 DEBUG:boto:Path: /dkyges-17/obj 2024-04-12T03:32:27.042 DEBUG:boto:Data: 2024-04-12T03:32:27.042 DEBUG:boto:Headers: {} 2024-04-12T03:32:27.042 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:32:27.043 DEBUG:boto:Port: 8001 2024-04-12T03:32:27.043 DEBUG:boto:Params: {} 2024-04-12T03:32:27.043 DEBUG:boto:Token: None 2024-04-12T03:32:27.043 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:27 GMT /dkyges-17/obj 2024-04-12T03:32:27.043 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4bryHLACadBPGBa1y+2GG7nCjTU= 2024-04-12T03:32:27.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4bryHLACadBPGBa1y+2GG7nCjTU=', 'Content-Length': '0'} 2024-04-12T03:32:27.045 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d2546e3e80d541cc-006618ab4b-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:27.045 DEBUG:boto:path=/dkyges-17/obj 2024-04-12T03:32:27.045 DEBUG:boto:auth_path=/dkyges-17/obj 2024-04-12T03:32:27.046 DEBUG:boto:Method: HEAD 2024-04-12T03:32:27.046 DEBUG:boto:Path: /dkyges-17/obj 2024-04-12T03:32:27.046 DEBUG:boto:Data: 2024-04-12T03:32:27.046 DEBUG:boto:Headers: {} 2024-04-12T03:32:27.046 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:32:27.046 DEBUG:boto:Port: 8000 2024-04-12T03:32:27.046 DEBUG:boto:Params: {} 2024-04-12T03:32:27.046 DEBUG:boto:Token: None 2024-04-12T03:32:27.046 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:32:27 GMT /dkyges-17/obj 2024-04-12T03:32:27.046 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4bryHLACadBPGBa1y+2GG7nCjTU= 2024-04-12T03:32:27.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:32:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4bryHLACadBPGBa1y+2GG7nCjTU=', 'Content-Length': '0'} 2024-04-12T03:32:27.048 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000001339bb6b999884f1-006618ab4b-110f-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:32:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:32:27.049 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-17 zones={a2, a1} 2024-04-12T03:32:27.049 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-18 2024-04-12T03:32:27.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:32:27.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:32:27.049 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-18 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:32:27.117 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:32:27.118 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:32:27.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:32:27.162+0000 7f44fe8e9780 1 waiting to reach incremental sync.. 2024-04-12T03:32:32.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:32:32.163+0000 7f44fe8e9780 1 waiting to reach incremental sync.. 2024-04-12T03:32:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:32:37.164+0000 7f44fe8e9780 1 waiting to reach incremental sync.. 2024-04-12T03:32:42.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:32:42.163+0000 7f44fe8e9780 1 waiting to reach incremental sync.. 2024-04-12T03:32:47.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:32:47.164+0000 7f44fe8e9780 1 waiting to reach incremental sync.. 2024-04-12T03:32:52.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:32:52.165+0000 7f44fe8e9780 1 waiting to reach incremental sync.. 2024-04-12T03:32:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:32:57.165+0000 7f44fe8e9780 1 waiting to reach incremental sync.. 2024-04-12T03:33:02.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:02.166+0000 7f44fe8e9780 1 waiting to reach incremental sync.. 2024-04-12T03:33:07.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:07.168+0000 7f44fe8e9780 1 bucket sync caught up with source: 2024-04-12T03:33:07.170 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000002.24.6, , ] 2024-04-12T03:33:07.170 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000002.24.6, , ] 2024-04-12T03:33:07.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:07.168+0000 7f44fe8e9780 0 bucket checkpoint complete 2024-04-12T03:33:07.188 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-18 2024-04-12T03:33:07.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:33:07.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:33:07.189 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-18 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:33:07.254 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:07.255 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:07.294+0000 7f6b99c56780 1 bucket sync caught up with source: 2024-04-12T03:33:07.295 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000003.37.12, , ] 2024-04-12T03:33:07.295 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000003.37.12, , ] 2024-04-12T03:33:07.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:07.294+0000 7f6b99c56780 0 bucket checkpoint complete 2024-04-12T03:33:07.311 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-18 zones={a2, a1} 2024-04-12T03:33:07.311 DEBUG:boto:path=/dkyges-18/ 2024-04-12T03:33:07.311 DEBUG:boto:auth_path=/dkyges-18/ 2024-04-12T03:33:07.311 DEBUG:boto:Method: HEAD 2024-04-12T03:33:07.312 DEBUG:boto:Path: /dkyges-18/ 2024-04-12T03:33:07.312 DEBUG:boto:Data: 2024-04-12T03:33:07.312 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.312 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:33:07.312 DEBUG:boto:Port: 8001 2024-04-12T03:33:07.312 DEBUG:boto:Params: {} 2024-04-12T03:33:07.312 DEBUG:boto:Token: None 2024-04-12T03:33:07.312 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/ 2024-04-12T03:33:07.313 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SdjVWdljSV6SMROYHXDpvArQPgw= 2024-04-12T03:33:07.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SdjVWdljSV6SMROYHXDpvArQPgw=', 'Content-Length': '0'} 2024-04-12T03:33:07.318 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000beac61e453ef0a52-006618ab73-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.318 DEBUG:boto:path=/dkyges-18/ 2024-04-12T03:33:07.318 DEBUG:boto:auth_path=/dkyges-18/ 2024-04-12T03:33:07.318 DEBUG:boto:Method: HEAD 2024-04-12T03:33:07.318 DEBUG:boto:Path: /dkyges-18/ 2024-04-12T03:33:07.318 DEBUG:boto:Data: 2024-04-12T03:33:07.319 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.319 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:33:07.319 DEBUG:boto:Port: 8000 2024-04-12T03:33:07.319 DEBUG:boto:Params: {} 2024-04-12T03:33:07.319 DEBUG:boto:Token: None 2024-04-12T03:33:07.319 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/ 2024-04-12T03:33:07.319 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SdjVWdljSV6SMROYHXDpvArQPgw= 2024-04-12T03:33:07.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SdjVWdljSV6SMROYHXDpvArQPgw=', 'Content-Length': '0'} 2024-04-12T03:33:07.322 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a7788579152ef0f-006618ab73-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.322 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:33:07.322 DEBUG:boto:path=/dkyges-18/ 2024-04-12T03:33:07.322 DEBUG:boto:auth_path=/dkyges-18/ 2024-04-12T03:33:07.322 DEBUG:boto:path=/dkyges-18/?versions&max-keys=999 2024-04-12T03:33:07.322 DEBUG:boto:auth_path=/dkyges-18/?versions&max-keys=999 2024-04-12T03:33:07.322 DEBUG:boto:Method: GET 2024-04-12T03:33:07.322 DEBUG:boto:Path: /dkyges-18/?versions&max-keys=999 2024-04-12T03:33:07.323 DEBUG:boto:Data: 2024-04-12T03:33:07.323 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.323 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:33:07.323 DEBUG:boto:Port: 8001 2024-04-12T03:33:07.323 DEBUG:boto:Params: {} 2024-04-12T03:33:07.323 DEBUG:boto:Token: None 2024-04-12T03:33:07.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/?versions 2024-04-12T03:33:07.323 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/3OMP2nLqHUY8lMSOxHMQTrAexk= 2024-04-12T03:33:07.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/3OMP2nLqHUY8lMSOxHMQTrAexk=', 'Content-Length': '0'} 2024-04-12T03:33:07.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aef9c1b96256acca-006618ab73-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.326 DEBUG:boto:b'dkyges-18999falseobj8WfLuNSvJpTfoqbI1a5Mlql48GPBA31true2024-04-12T03:31:51.678Zrgw-multisite-test-userMultisite Test UserNormalobjGUJuHw9TDS57.uhhhnXWqwHOG8H1AFyfalse2024-04-12T03:31:51.630Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:07.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:33:07.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:33:07.327 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:33:07.327 DEBUG:boto:path=/dkyges-18/ 2024-04-12T03:33:07.327 DEBUG:boto:auth_path=/dkyges-18/ 2024-04-12T03:33:07.327 DEBUG:boto:path=/dkyges-18/?versions&max-keys=999 2024-04-12T03:33:07.327 DEBUG:boto:auth_path=/dkyges-18/?versions&max-keys=999 2024-04-12T03:33:07.327 DEBUG:boto:Method: GET 2024-04-12T03:33:07.327 DEBUG:boto:Path: /dkyges-18/?versions&max-keys=999 2024-04-12T03:33:07.327 DEBUG:boto:Data: 2024-04-12T03:33:07.327 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.327 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:33:07.327 DEBUG:boto:Port: 8000 2024-04-12T03:33:07.328 DEBUG:boto:Params: {} 2024-04-12T03:33:07.328 DEBUG:boto:Token: None 2024-04-12T03:33:07.328 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/?versions 2024-04-12T03:33:07.328 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/3OMP2nLqHUY8lMSOxHMQTrAexk= 2024-04-12T03:33:07.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/3OMP2nLqHUY8lMSOxHMQTrAexk=', 'Content-Length': '0'} 2024-04-12T03:33:07.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da45ca3ccead1d03-006618ab73-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.330 DEBUG:boto:b'dkyges-18999falseobj8WfLuNSvJpTfoqbI1a5Mlql48GPBA31true2024-04-12T03:31:51.678Zrgw-multisite-test-userMultisite Test UserNormalobjGUJuHw9TDS57.uhhhnXWqwHOG8H1AFyfalse2024-04-12T03:31:51.630Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:07.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:33:07.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:33:07.331 DEBUG:boto:path=/dkyges-18/ 2024-04-12T03:33:07.331 DEBUG:boto:auth_path=/dkyges-18/ 2024-04-12T03:33:07.331 DEBUG:boto:path=/dkyges-18/?versions&max-keys=999 2024-04-12T03:33:07.331 DEBUG:boto:auth_path=/dkyges-18/?versions&max-keys=999 2024-04-12T03:33:07.331 DEBUG:boto:Method: GET 2024-04-12T03:33:07.331 DEBUG:boto:Path: /dkyges-18/?versions&max-keys=999 2024-04-12T03:33:07.332 DEBUG:boto:Data: 2024-04-12T03:33:07.332 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.332 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:33:07.332 DEBUG:boto:Port: 8001 2024-04-12T03:33:07.332 DEBUG:boto:Params: {} 2024-04-12T03:33:07.332 DEBUG:boto:Token: None 2024-04-12T03:33:07.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/?versions 2024-04-12T03:33:07.332 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/3OMP2nLqHUY8lMSOxHMQTrAexk= 2024-04-12T03:33:07.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/3OMP2nLqHUY8lMSOxHMQTrAexk=', 'Content-Length': '0'} 2024-04-12T03:33:07.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fcb1a617f35d688-006618ab73-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.335 DEBUG:boto:b'dkyges-18999falseobj8WfLuNSvJpTfoqbI1a5Mlql48GPBA31true2024-04-12T03:31:51.678Zrgw-multisite-test-userMultisite Test UserNormalobjGUJuHw9TDS57.uhhhnXWqwHOG8H1AFyfalse2024-04-12T03:31:51.630Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:07.335 DEBUG:boto:path=/dkyges-18/ 2024-04-12T03:33:07.335 DEBUG:boto:auth_path=/dkyges-18/ 2024-04-12T03:33:07.335 DEBUG:boto:path=/dkyges-18/?versions&max-keys=999 2024-04-12T03:33:07.335 DEBUG:boto:auth_path=/dkyges-18/?versions&max-keys=999 2024-04-12T03:33:07.336 DEBUG:boto:Method: GET 2024-04-12T03:33:07.336 DEBUG:boto:Path: /dkyges-18/?versions&max-keys=999 2024-04-12T03:33:07.336 DEBUG:boto:Data: 2024-04-12T03:33:07.336 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.336 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:33:07.336 DEBUG:boto:Port: 8000 2024-04-12T03:33:07.336 DEBUG:boto:Params: {} 2024-04-12T03:33:07.336 DEBUG:boto:Token: None 2024-04-12T03:33:07.336 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/?versions 2024-04-12T03:33:07.336 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/3OMP2nLqHUY8lMSOxHMQTrAexk= 2024-04-12T03:33:07.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/3OMP2nLqHUY8lMSOxHMQTrAexk=', 'Content-Length': '0'} 2024-04-12T03:33:07.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040eceee582bdc808-006618ab73-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.382 DEBUG:boto:b'dkyges-18999falseobj8WfLuNSvJpTfoqbI1a5Mlql48GPBA31true2024-04-12T03:31:51.678Zrgw-multisite-test-userMultisite Test UserNormalobjGUJuHw9TDS57.uhhhnXWqwHOG8H1AFyfalse2024-04-12T03:31:51.630Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:33:07.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:33:07.383 DEBUG:boto:path=/dkyges-18/obj 2024-04-12T03:33:07.383 DEBUG:boto:auth_path=/dkyges-18/obj 2024-04-12T03:33:07.383 DEBUG:boto:Method: HEAD 2024-04-12T03:33:07.383 DEBUG:boto:Path: /dkyges-18/obj 2024-04-12T03:33:07.383 DEBUG:boto:Data: 2024-04-12T03:33:07.384 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.384 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:33:07.384 DEBUG:boto:Port: 8001 2024-04-12T03:33:07.384 DEBUG:boto:Params: {} 2024-04-12T03:33:07.384 DEBUG:boto:Token: None 2024-04-12T03:33:07.384 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/obj 2024-04-12T03:33:07.384 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b0Xiqs0ymqEI8D/0iRzaqhzzPC4= 2024-04-12T03:33:07.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b0Xiqs0ymqEI8D/0iRzaqhzzPC4=', 'Content-Length': '0'} 2024-04-12T03:33:07.386 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000ebfdd3d41ac87b7b-006618ab73-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.386 DEBUG:boto:path=/dkyges-18/obj 2024-04-12T03:33:07.387 DEBUG:boto:auth_path=/dkyges-18/obj 2024-04-12T03:33:07.387 DEBUG:boto:Method: HEAD 2024-04-12T03:33:07.387 DEBUG:boto:Path: /dkyges-18/obj 2024-04-12T03:33:07.387 DEBUG:boto:Data: 2024-04-12T03:33:07.387 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.387 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:33:07.387 DEBUG:boto:Port: 8000 2024-04-12T03:33:07.387 DEBUG:boto:Params: {} 2024-04-12T03:33:07.387 DEBUG:boto:Token: None 2024-04-12T03:33:07.387 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/obj 2024-04-12T03:33:07.388 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b0Xiqs0ymqEI8D/0iRzaqhzzPC4= 2024-04-12T03:33:07.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b0Xiqs0ymqEI8D/0iRzaqhzzPC4=', 'Content-Length': '0'} 2024-04-12T03:33:07.389 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000affdf9d05642a97b-006618ab73-110f-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:33:07.390 DEBUG:boto:path=/dkyges-18/obj 2024-04-12T03:33:07.390 DEBUG:boto:auth_path=/dkyges-18/obj 2024-04-12T03:33:07.390 DEBUG:boto:path=/dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.390 DEBUG:boto:auth_path=/dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.390 DEBUG:boto:Method: GET 2024-04-12T03:33:07.390 DEBUG:boto:Path: /dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.391 DEBUG:boto:Data: 2024-04-12T03:33:07.391 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.391 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:33:07.391 DEBUG:boto:Port: 8001 2024-04-12T03:33:07.391 DEBUG:boto:Params: {} 2024-04-12T03:33:07.391 DEBUG:boto:Token: None 2024-04-12T03:33:07.391 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.391 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eYk4xL30DtjlJKZKiPrShgUSQ1g= 2024-04-12T03:33:07.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eYk4xL30DtjlJKZKiPrShgUSQ1g=', 'Content-Length': '0'} 2024-04-12T03:33:07.393 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('x-amz-version-id', 'GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001bd23b661f2afd51-006618ab73-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.394 DEBUG:boto:path=/dkyges-18/obj 2024-04-12T03:33:07.394 DEBUG:boto:auth_path=/dkyges-18/obj 2024-04-12T03:33:07.394 DEBUG:boto:path=/dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.394 DEBUG:boto:auth_path=/dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.394 DEBUG:boto:Method: GET 2024-04-12T03:33:07.394 DEBUG:boto:Path: /dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.394 DEBUG:boto:Data: 2024-04-12T03:33:07.394 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.394 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:33:07.394 DEBUG:boto:Port: 8000 2024-04-12T03:33:07.394 DEBUG:boto:Params: {} 2024-04-12T03:33:07.395 DEBUG:boto:Token: None 2024-04-12T03:33:07.395 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.395 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eYk4xL30DtjlJKZKiPrShgUSQ1g= 2024-04-12T03:33:07.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eYk4xL30DtjlJKZKiPrShgUSQ1g=', 'Content-Length': '0'} 2024-04-12T03:33:07.397 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('x-amz-version-id', 'GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001757e2afb867f078-006618ab73-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.397 DEBUG:boto:path=/dkyges-18/obj 2024-04-12T03:33:07.397 DEBUG:boto:auth_path=/dkyges-18/obj 2024-04-12T03:33:07.397 DEBUG:boto:path=/dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.397 DEBUG:boto:auth_path=/dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.397 DEBUG:boto:Method: HEAD 2024-04-12T03:33:07.398 DEBUG:boto:Path: /dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.398 DEBUG:boto:Data: 2024-04-12T03:33:07.398 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.398 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:33:07.398 DEBUG:boto:Port: 8001 2024-04-12T03:33:07.398 DEBUG:boto:Params: {} 2024-04-12T03:33:07.398 DEBUG:boto:Token: None 2024-04-12T03:33:07.398 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.398 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Pn2eeDOCow/mGCT5IRBVB306LQU= 2024-04-12T03:33:07.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Pn2eeDOCow/mGCT5IRBVB306LQU=', 'Content-Length': '0'} 2024-04-12T03:33:07.400 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('x-amz-version-id', 'GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e3a33721d267cfd0-006618ab73-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.401 DEBUG:boto:path=/dkyges-18/obj 2024-04-12T03:33:07.401 DEBUG:boto:auth_path=/dkyges-18/obj 2024-04-12T03:33:07.401 DEBUG:boto:path=/dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.401 DEBUG:boto:auth_path=/dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.401 DEBUG:boto:Method: HEAD 2024-04-12T03:33:07.401 DEBUG:boto:Path: /dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.401 DEBUG:boto:Data: 2024-04-12T03:33:07.401 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.401 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:33:07.401 DEBUG:boto:Port: 8000 2024-04-12T03:33:07.401 DEBUG:boto:Params: {} 2024-04-12T03:33:07.402 DEBUG:boto:Token: None 2024-04-12T03:33:07.402 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.402 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Pn2eeDOCow/mGCT5IRBVB306LQU= 2024-04-12T03:33:07.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Pn2eeDOCow/mGCT5IRBVB306LQU=', 'Content-Length': '0'} 2024-04-12T03:33:07.404 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('x-amz-version-id', 'GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d540b9b3c3c03543-006618ab73-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:33:07.404 DEBUG:boto:path=/dkyges-18/obj 2024-04-12T03:33:07.404 DEBUG:boto:auth_path=/dkyges-18/obj 2024-04-12T03:33:07.404 DEBUG:boto:path=/dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.404 DEBUG:boto:auth_path=/dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.404 DEBUG:boto:Method: GET 2024-04-12T03:33:07.405 DEBUG:boto:Path: /dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.405 DEBUG:boto:Data: 2024-04-12T03:33:07.405 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.405 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:33:07.405 DEBUG:boto:Port: 8001 2024-04-12T03:33:07.405 DEBUG:boto:Params: {} 2024-04-12T03:33:07.405 DEBUG:boto:Token: None 2024-04-12T03:33:07.405 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.405 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eYk4xL30DtjlJKZKiPrShgUSQ1g= 2024-04-12T03:33:07.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eYk4xL30DtjlJKZKiPrShgUSQ1g=', 'Content-Length': '0'} 2024-04-12T03:33:07.407 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('x-amz-version-id', 'GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000443cf785ec4a29ad-006618ab73-1113-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.407 DEBUG:boto:path=/dkyges-18/obj 2024-04-12T03:33:07.408 DEBUG:boto:auth_path=/dkyges-18/obj 2024-04-12T03:33:07.408 DEBUG:boto:path=/dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.408 DEBUG:boto:auth_path=/dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.408 DEBUG:boto:Method: GET 2024-04-12T03:33:07.408 DEBUG:boto:Path: /dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.408 DEBUG:boto:Data: 2024-04-12T03:33:07.408 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.408 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:33:07.408 DEBUG:boto:Port: 8000 2024-04-12T03:33:07.408 DEBUG:boto:Params: {} 2024-04-12T03:33:07.408 DEBUG:boto:Token: None 2024-04-12T03:33:07.409 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/obj?versionId=GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy 2024-04-12T03:33:07.409 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eYk4xL30DtjlJKZKiPrShgUSQ1g= 2024-04-12T03:33:07.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eYk4xL30DtjlJKZKiPrShgUSQ1g=', 'Content-Length': '0'} 2024-04-12T03:33:07.411 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:31:51 GMT'), ('x-amz-version-id', 'GUJuHw9TDS57.uhhhnXWqwHOG8H1AFy'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000012c772967d4a77ea-006618ab73-110f-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.411 DEBUG:boto:path=/dkyges-18/ 2024-04-12T03:33:07.411 DEBUG:boto:auth_path=/dkyges-18/ 2024-04-12T03:33:07.411 DEBUG:boto:path=/dkyges-18/?prefix=obj 2024-04-12T03:33:07.411 DEBUG:boto:auth_path=/dkyges-18/?prefix=obj 2024-04-12T03:33:07.411 DEBUG:boto:Method: GET 2024-04-12T03:33:07.411 DEBUG:boto:Path: /dkyges-18/?prefix=obj 2024-04-12T03:33:07.411 DEBUG:boto:Data: 2024-04-12T03:33:07.411 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.412 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:33:07.412 DEBUG:boto:Port: 8001 2024-04-12T03:33:07.412 DEBUG:boto:Params: {} 2024-04-12T03:33:07.412 DEBUG:boto:Token: None 2024-04-12T03:33:07.412 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/ 2024-04-12T03:33:07.412 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gU/JQeUJ39kMF1iV2L2sJuN7xxg= 2024-04-12T03:33:07.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gU/JQeUJ39kMF1iV2L2sJuN7xxg=', 'Content-Length': '0'} 2024-04-12T03:33:07.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005536f75bc279f93-006618ab73-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.458 DEBUG:boto:b'dkyges-18obj1000false' 2024-04-12T03:33:07.459 DEBUG:boto:path=/dkyges-18/ 2024-04-12T03:33:07.459 DEBUG:boto:auth_path=/dkyges-18/ 2024-04-12T03:33:07.459 DEBUG:boto:path=/dkyges-18/?prefix=obj 2024-04-12T03:33:07.459 DEBUG:boto:auth_path=/dkyges-18/?prefix=obj 2024-04-12T03:33:07.459 DEBUG:boto:Method: GET 2024-04-12T03:33:07.459 DEBUG:boto:Path: /dkyges-18/?prefix=obj 2024-04-12T03:33:07.459 DEBUG:boto:Data: 2024-04-12T03:33:07.459 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.460 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:33:07.460 DEBUG:boto:Port: 8000 2024-04-12T03:33:07.460 DEBUG:boto:Params: {} 2024-04-12T03:33:07.460 DEBUG:boto:Token: None 2024-04-12T03:33:07.460 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/ 2024-04-12T03:33:07.460 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gU/JQeUJ39kMF1iV2L2sJuN7xxg= 2024-04-12T03:33:07.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gU/JQeUJ39kMF1iV2L2sJuN7xxg=', 'Content-Length': '0'} 2024-04-12T03:33:07.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008669c8df7c60c485-006618ab73-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.463 DEBUG:boto:b'dkyges-18obj1000false' 2024-04-12T03:33:07.463 DEBUG:boto:path=/dkyges-18/obj 2024-04-12T03:33:07.463 DEBUG:boto:auth_path=/dkyges-18/obj 2024-04-12T03:33:07.463 DEBUG:boto:Method: HEAD 2024-04-12T03:33:07.463 DEBUG:boto:Path: /dkyges-18/obj 2024-04-12T03:33:07.464 DEBUG:boto:Data: 2024-04-12T03:33:07.464 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.464 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:33:07.464 DEBUG:boto:Port: 8001 2024-04-12T03:33:07.464 DEBUG:boto:Params: {} 2024-04-12T03:33:07.464 DEBUG:boto:Token: None 2024-04-12T03:33:07.464 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/obj 2024-04-12T03:33:07.464 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b0Xiqs0ymqEI8D/0iRzaqhzzPC4= 2024-04-12T03:33:07.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b0Xiqs0ymqEI8D/0iRzaqhzzPC4=', 'Content-Length': '0'} 2024-04-12T03:33:07.466 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000dbfafa1d048c9583-006618ab73-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.466 DEBUG:boto:path=/dkyges-18/obj 2024-04-12T03:33:07.466 DEBUG:boto:auth_path=/dkyges-18/obj 2024-04-12T03:33:07.467 DEBUG:boto:Method: HEAD 2024-04-12T03:33:07.467 DEBUG:boto:Path: /dkyges-18/obj 2024-04-12T03:33:07.467 DEBUG:boto:Data: 2024-04-12T03:33:07.467 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.467 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:33:07.467 DEBUG:boto:Port: 8000 2024-04-12T03:33:07.467 DEBUG:boto:Params: {} 2024-04-12T03:33:07.467 DEBUG:boto:Token: None 2024-04-12T03:33:07.467 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:07 GMT /dkyges-18/obj 2024-04-12T03:33:07.468 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b0Xiqs0ymqEI8D/0iRzaqhzzPC4= 2024-04-12T03:33:07.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b0Xiqs0ymqEI8D/0iRzaqhzzPC4=', 'Content-Length': '0'} 2024-04-12T03:33:07.470 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000001436c81b603c40d5-006618ab73-110f-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.470 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-18 zones={a2, a1} 2024-04-12T03:33:07.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2024-04-12T03:33:07.471 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-19 2024-04-12T03:33:07.472 DEBUG:boto:path=/dkyges-19/ 2024-04-12T03:33:07.472 DEBUG:boto:auth_path=/dkyges-19/ 2024-04-12T03:33:07.472 DEBUG:boto:Method: PUT 2024-04-12T03:33:07.472 DEBUG:boto:Path: /dkyges-19/ 2024-04-12T03:33:07.472 DEBUG:boto:Data: 2024-04-12T03:33:07.472 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.472 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:33:07.472 DEBUG:boto:Port: 8000 2024-04-12T03:33:07.472 DEBUG:boto:Params: {} 2024-04-12T03:33:07.472 DEBUG:boto:Token: None 2024-04-12T03:33:07.472 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:33:07 GMT /dkyges-19/ 2024-04-12T03:33:07.473 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WQkhc4WZX1nM3bVafyVibV2g+zY= 2024-04-12T03:33:07.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WQkhc4WZX1nM3bVafyVibV2g+zY=', 'Content-Length': '0'} 2024-04-12T03:33:07.489 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000839a5bf507af465a-006618ab73-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.490 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-20 2024-04-12T03:33:07.490 DEBUG:boto:path=/dkyges-20/ 2024-04-12T03:33:07.490 DEBUG:boto:auth_path=/dkyges-20/ 2024-04-12T03:33:07.490 DEBUG:boto:Method: PUT 2024-04-12T03:33:07.490 DEBUG:boto:Path: /dkyges-20/ 2024-04-12T03:33:07.490 DEBUG:boto:Data: 2024-04-12T03:33:07.490 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.490 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:33:07.490 DEBUG:boto:Port: 8001 2024-04-12T03:33:07.490 DEBUG:boto:Params: {} 2024-04-12T03:33:07.491 DEBUG:boto:Token: None 2024-04-12T03:33:07.491 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:33:07 GMT /dkyges-20/ 2024-04-12T03:33:07.491 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GOJRG47rHV+9578j5bQ9apHDKns= 2024-04-12T03:33:07.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GOJRG47rHV+9578j5bQ9apHDKns=', 'Content-Length': '0'} 2024-04-12T03:33:07.517 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ef529fa1759590cd-006618ab73-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.517 DEBUG:boto:path=/dkyges-19/ 2024-04-12T03:33:07.517 DEBUG:boto:auth_path=/dkyges-19/ 2024-04-12T03:33:07.517 DEBUG:boto:path=/dkyges-19/?versioning 2024-04-12T03:33:07.518 DEBUG:boto:auth_path=/dkyges-19/?versioning 2024-04-12T03:33:07.518 DEBUG:boto:Method: PUT 2024-04-12T03:33:07.518 DEBUG:boto:Path: /dkyges-19/?versioning 2024-04-12T03:33:07.518 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:33:07.518 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.518 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:33:07.518 DEBUG:boto:Port: 8000 2024-04-12T03:33:07.518 DEBUG:boto:Params: {} 2024-04-12T03:33:07.518 DEBUG:boto:Token: None 2024-04-12T03:33:07.519 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:33:07 GMT /dkyges-19/?versioning 2024-04-12T03:33:07.519 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:luigcH7IlUPV06otUXB470TrkzE= 2024-04-12T03:33:07.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:luigcH7IlUPV06otUXB470TrkzE=', 'Content-Length': '228'} 2024-04-12T03:33:07.525 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000046babb65ffe0aca1-006618ab73-110f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.525 DEBUG:boto:path=/dkyges-19/ 2024-04-12T03:33:07.525 DEBUG:boto:auth_path=/dkyges-19/ 2024-04-12T03:33:07.525 DEBUG:boto:path=/dkyges-19/?versioning 2024-04-12T03:33:07.525 DEBUG:boto:auth_path=/dkyges-19/?versioning 2024-04-12T03:33:07.525 DEBUG:boto:Method: PUT 2024-04-12T03:33:07.525 DEBUG:boto:Path: /dkyges-19/?versioning 2024-04-12T03:33:07.525 DEBUG:boto:Data: Suspended Disabled 2024-04-12T03:33:07.526 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.526 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:33:07.526 DEBUG:boto:Port: 8000 2024-04-12T03:33:07.526 DEBUG:boto:Params: {} 2024-04-12T03:33:07.526 DEBUG:boto:Token: None 2024-04-12T03:33:07.526 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:33:07 GMT /dkyges-19/?versioning 2024-04-12T03:33:07.526 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:luigcH7IlUPV06otUXB470TrkzE= 2024-04-12T03:33:07.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:luigcH7IlUPV06otUXB470TrkzE=', 'Content-Length': '230'} 2024-04-12T03:33:07.531 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000548036f90932ac3f-006618ab73-110f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.532 DEBUG:boto:path=/dkyges-20/ 2024-04-12T03:33:07.532 DEBUG:boto:auth_path=/dkyges-20/ 2024-04-12T03:33:07.532 DEBUG:boto:path=/dkyges-20/?versioning 2024-04-12T03:33:07.532 DEBUG:boto:auth_path=/dkyges-20/?versioning 2024-04-12T03:33:07.532 DEBUG:boto:Method: PUT 2024-04-12T03:33:07.532 DEBUG:boto:Path: /dkyges-20/?versioning 2024-04-12T03:33:07.532 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:33:07.532 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.532 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:33:07.532 DEBUG:boto:Port: 8001 2024-04-12T03:33:07.533 DEBUG:boto:Params: {} 2024-04-12T03:33:07.533 DEBUG:boto:Token: None 2024-04-12T03:33:07.533 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:33:07 GMT /dkyges-20/?versioning 2024-04-12T03:33:07.533 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:al8R+ey+oXW8XaNTcfFWGHlmnfk= 2024-04-12T03:33:07.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:al8R+ey+oXW8XaNTcfFWGHlmnfk=', 'Content-Length': '228'} 2024-04-12T03:33:07.541 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009d5caef66a5523b5-006618ab73-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.541 DEBUG:boto:path=/dkyges-20/ 2024-04-12T03:33:07.541 DEBUG:boto:auth_path=/dkyges-20/ 2024-04-12T03:33:07.542 DEBUG:boto:path=/dkyges-20/?versioning 2024-04-12T03:33:07.542 DEBUG:boto:auth_path=/dkyges-20/?versioning 2024-04-12T03:33:07.542 DEBUG:boto:Method: PUT 2024-04-12T03:33:07.542 DEBUG:boto:Path: /dkyges-20/?versioning 2024-04-12T03:33:07.542 DEBUG:boto:Data: Suspended Disabled 2024-04-12T03:33:07.542 DEBUG:boto:Headers: {} 2024-04-12T03:33:07.542 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:33:07.542 DEBUG:boto:Port: 8001 2024-04-12T03:33:07.542 DEBUG:boto:Params: {} 2024-04-12T03:33:07.542 DEBUG:boto:Token: None 2024-04-12T03:33:07.543 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:33:07 GMT /dkyges-20/?versioning 2024-04-12T03:33:07.543 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:al8R+ey+oXW8XaNTcfFWGHlmnfk= 2024-04-12T03:33:07.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:al8R+ey+oXW8XaNTcfFWGHlmnfk=', 'Content-Length': '230'} 2024-04-12T03:33:07.551 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a825ef7c8486a927-006618ab73-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:07.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:33:07.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:33:07.552 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:33:07.616 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:07.617 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:07.656 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:33:07.670 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:33:07.670 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:33:07.670 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:33:07.670 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:33:07.670 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:33:07.670 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:33:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:33:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:33:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:33:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:33:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:33:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:33:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:33:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:33:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892787.548455_203.1", 2024-04-12T03:33:07.671 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:33:07.548455Z" 2024-04-12T03:33:07.672 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:33:07.672 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:33:07.673 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892787.548455_203.1'} 2024-04-12T03:33:07.673 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:33:07.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:33:07.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:33:07.674 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:33:07.729 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:07.729 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:07.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:07.761+0000 7f0566918780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:33:07.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:07.762+0000 7f0566918780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:33:07.776 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:33:07.776 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:33:07.776 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:33:07.776 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:33:07.776 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:33:07.776 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:33:07.777 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:33:07.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:33:07.777 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:33:07.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:33:07.777 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:33:07.777 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:33:07.777 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:33:07.777 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:33:07.777 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:33:07.777 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:33:07.778 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:33:07.778 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:33:07.778 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:33:07.778 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:33:07.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:33:07.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:33:07.778 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:33:07.778 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:33:07.778 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:33:07.778 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:33:07.778 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:33:07.779 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:33:07.779 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:33:07.779 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:33:07.779 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:33:07.779 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:33:07.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:33:07.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:33:07.779 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:33:07.779 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:33:07.779 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:33:07.779 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:33:07.779 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:33:07.780 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:33:07.780 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:33:07.780 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:33:07.780 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:33:07.780 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:33:07.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:33:07.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:33:07.780 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:33:07.780 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:33:07.780 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:33:07.780 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892787.487687_191.1", 2024-04-12T03:33:07.781 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:33:07.781 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:33:07.781 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:33:07.781 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:33:07.487687Z", 2024-04-12T03:33:07.781 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:33:07.781 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:33:07.781 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:33:07.781 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:33:07.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:33:07.781 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:33:07.781 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:33:07.781 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:33:07.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:33:07.782 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:33:07Z" 2024-04-12T03:33:07.782 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:33:07.783 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892787.487687_191.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:33:07.487687Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:33:07Z" } 2024-04-12T03:33:07.783 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892787.487687_191.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:33:07.487687Z', 'realm_epoch': 2}}] 2024-04-12T03:33:07.783 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892787.548455_203.1'} 2024-04-12T03:33:07.783 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892787.487687_191.1'} 2024-04-12T03:33:07.783 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892787.548455_203.1 target=1_1712892787.487687_191.1 2024-04-12T03:33:12.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:33:12.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:33:12.784 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:33:12.844 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:12.844 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:12.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:12.877+0000 7f50c4b2c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:33:12.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:12.878+0000 7f50c4b2c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:33:12.893 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:33:12.893 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:33:12.893 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:33:12.893 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:33:12.893 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:33:12.893 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:33:12.893 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:33:12.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:33:12.893 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:33:12.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:33:12.894 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:33:12.894 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:33:12.894 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:33:12.894 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:33:12.894 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:33:12.894 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:33:12.894 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:33:12.894 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:33:12.894 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:33:12.894 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:33:12.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:33:12.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:33:12.895 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:33:12.895 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:33:12.895 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:33:12.895 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:33:12.895 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:33:12.895 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:33:12.895 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:33:12.895 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:33:12.895 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:33:12.896 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:33:12.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:33:12.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:33:12.896 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:33:12.896 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:33:12.896 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:33:12.896 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:33:12.896 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:33:12.896 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:33:12.896 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:33:12.896 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:33:12.897 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:33:12.897 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:33:12.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:33:12.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:33:12.897 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:33:12.897 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:33:12.897 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:33:12.897 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892787.548455_203.1", 2024-04-12T03:33:12.897 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:33:12.897 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:33:12.897 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:33:12.897 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:33:07.548455Z", 2024-04-12T03:33:12.898 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:33:12.898 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:33:12.898 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:33:12.898 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:33:12.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:33:12.898 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:33:12.898 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:33:12.898 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:33:12.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:33:12.898 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:33:12Z" 2024-04-12T03:33:12.898 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:33:12.899 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892787.548455_203.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:33:07.548455Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:33:12Z" } 2024-04-12T03:33:12.900 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892787.548455_203.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:33:07.548455Z', 'realm_epoch': 2}}] 2024-04-12T03:33:12.900 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892787.548455_203.1'} 2024-04-12T03:33:12.900 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892787.548455_203.1'} 2024-04-12T03:33:12.900 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:33:12.900 DEBUG:boto:path=/dkyges-19/ 2024-04-12T03:33:12.900 DEBUG:boto:auth_path=/dkyges-19/ 2024-04-12T03:33:12.900 DEBUG:boto:Method: HEAD 2024-04-12T03:33:12.900 DEBUG:boto:Path: /dkyges-19/ 2024-04-12T03:33:12.900 DEBUG:boto:Data: 2024-04-12T03:33:12.901 DEBUG:boto:Headers: {} 2024-04-12T03:33:12.901 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:33:12.901 DEBUG:boto:Port: 8000 2024-04-12T03:33:12.901 DEBUG:boto:Params: {} 2024-04-12T03:33:12.901 DEBUG:boto:Token: None 2024-04-12T03:33:12.901 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:12 GMT /dkyges-19/ 2024-04-12T03:33:12.901 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/LH465shk1/64J8Az8rH71HwTUs= 2024-04-12T03:33:12.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:12 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/LH465shk1/64J8Az8rH71HwTUs=', 'Content-Length': '0'} 2024-04-12T03:33:12.904 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000536b7ca29dda8aa2-006618ab78-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:12.904 DEBUG:boto:path=/dkyges-19/obj 2024-04-12T03:33:12.904 DEBUG:boto:auth_path=/dkyges-19/obj 2024-04-12T03:33:12.904 DEBUG:boto:Method: PUT 2024-04-12T03:33:12.904 DEBUG:boto:Path: /dkyges-19/obj 2024-04-12T03:33:12.905 DEBUG:boto:Data: 2024-04-12T03:33:12.905 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:33:12.905 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:33:12.905 DEBUG:boto:Port: 8000 2024-04-12T03:33:12.905 DEBUG:boto:Params: {} 2024-04-12T03:33:12.905 DEBUG:boto:Token: None 2024-04-12T03:33:12.905 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:33:12 GMT /dkyges-19/obj 2024-04-12T03:33:12.905 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:buS8jH2Ejc/7O+RP05HluoywBLQ= 2024-04-12T03:33:12.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:33:12 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:buS8jH2Ejc/7O+RP05HluoywBLQ='} 2024-04-12T03:33:12.912 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d9571e7467e2c7a-006618ab78-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:12.912 DEBUG:boto:path=/dkyges-19/ 2024-04-12T03:33:12.912 DEBUG:boto:auth_path=/dkyges-19/ 2024-04-12T03:33:12.912 DEBUG:boto:Method: HEAD 2024-04-12T03:33:12.912 DEBUG:boto:Path: /dkyges-19/ 2024-04-12T03:33:12.912 DEBUG:boto:Data: 2024-04-12T03:33:12.912 DEBUG:boto:Headers: {} 2024-04-12T03:33:12.912 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:33:12.912 DEBUG:boto:Port: 8000 2024-04-12T03:33:12.912 DEBUG:boto:Params: {} 2024-04-12T03:33:12.913 DEBUG:boto:Token: None 2024-04-12T03:33:12.913 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:12 GMT /dkyges-19/ 2024-04-12T03:33:12.913 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/LH465shk1/64J8Az8rH71HwTUs= 2024-04-12T03:33:12.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:12 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/LH465shk1/64J8Az8rH71HwTUs=', 'Content-Length': '0'} 2024-04-12T03:33:12.915 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f6d337a16ea09cd-006618ab78-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:12.915 DEBUG:boto:path=/dkyges-19/obj 2024-04-12T03:33:12.915 DEBUG:boto:auth_path=/dkyges-19/obj 2024-04-12T03:33:12.916 DEBUG:boto:Method: DELETE 2024-04-12T03:33:12.916 DEBUG:boto:Path: /dkyges-19/obj 2024-04-12T03:33:12.916 DEBUG:boto:Data: 2024-04-12T03:33:12.916 DEBUG:boto:Headers: {} 2024-04-12T03:33:12.916 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:33:12.916 DEBUG:boto:Port: 8000 2024-04-12T03:33:12.916 DEBUG:boto:Params: {} 2024-04-12T03:33:12.916 DEBUG:boto:Token: None 2024-04-12T03:33:12.916 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:33:12 GMT /dkyges-19/obj 2024-04-12T03:33:12.916 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GghbmCsoVkKuR+N18wFL1ZRx2Fk= 2024-04-12T03:33:12.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:12 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GghbmCsoVkKuR+N18wFL1ZRx2Fk=', 'Content-Length': '0'} 2024-04-12T03:33:12.925 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000003c4d1b9763694171-006618ab78-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:12.925 DEBUG:boto:path=/dkyges-20/ 2024-04-12T03:33:12.925 DEBUG:boto:auth_path=/dkyges-20/ 2024-04-12T03:33:12.926 DEBUG:boto:Method: HEAD 2024-04-12T03:33:12.926 DEBUG:boto:Path: /dkyges-20/ 2024-04-12T03:33:12.926 DEBUG:boto:Data: 2024-04-12T03:33:12.926 DEBUG:boto:Headers: {} 2024-04-12T03:33:12.926 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:33:12.926 DEBUG:boto:Port: 8001 2024-04-12T03:33:12.926 DEBUG:boto:Params: {} 2024-04-12T03:33:12.926 DEBUG:boto:Token: None 2024-04-12T03:33:12.926 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:12 GMT /dkyges-20/ 2024-04-12T03:33:12.926 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cv/X3MB5EvaBK9m7c/Yb6PQm2L0= 2024-04-12T03:33:12.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:12 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cv/X3MB5EvaBK9m7c/Yb6PQm2L0=', 'Content-Length': '0'} 2024-04-12T03:33:12.929 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006afd259c776daab9-006618ab78-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:12.929 DEBUG:boto:path=/dkyges-20/obj 2024-04-12T03:33:12.929 DEBUG:boto:auth_path=/dkyges-20/obj 2024-04-12T03:33:12.929 DEBUG:boto:Method: PUT 2024-04-12T03:33:12.929 DEBUG:boto:Path: /dkyges-20/obj 2024-04-12T03:33:12.929 DEBUG:boto:Data: 2024-04-12T03:33:12.930 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2024-04-12T03:33:12.930 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:33:12.930 DEBUG:boto:Port: 8001 2024-04-12T03:33:12.930 DEBUG:boto:Params: {} 2024-04-12T03:33:12.930 DEBUG:boto:Token: None 2024-04-12T03:33:12.930 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 12 Apr 2024 03:33:12 GMT /dkyges-20/obj 2024-04-12T03:33:12.930 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cGYN7OEL3Q7S16yX1NTUCTPWxHM= 2024-04-12T03:33:12.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:33:12 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cGYN7OEL3Q7S16yX1NTUCTPWxHM='} 2024-04-12T03:33:12.936 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000890e0cc9ce9eb0c-006618ab78-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:12.937 DEBUG:boto:path=/dkyges-20/ 2024-04-12T03:33:12.937 DEBUG:boto:auth_path=/dkyges-20/ 2024-04-12T03:33:12.937 DEBUG:boto:Method: HEAD 2024-04-12T03:33:12.937 DEBUG:boto:Path: /dkyges-20/ 2024-04-12T03:33:12.937 DEBUG:boto:Data: 2024-04-12T03:33:12.937 DEBUG:boto:Headers: {} 2024-04-12T03:33:12.937 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:33:12.937 DEBUG:boto:Port: 8001 2024-04-12T03:33:12.937 DEBUG:boto:Params: {} 2024-04-12T03:33:12.937 DEBUG:boto:Token: None 2024-04-12T03:33:12.937 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:33:12 GMT /dkyges-20/ 2024-04-12T03:33:12.938 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cv/X3MB5EvaBK9m7c/Yb6PQm2L0= 2024-04-12T03:33:12.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:12 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cv/X3MB5EvaBK9m7c/Yb6PQm2L0=', 'Content-Length': '0'} 2024-04-12T03:33:12.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073c1593e6690868b-006618ab78-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:12.940 DEBUG:boto:path=/dkyges-20/obj 2024-04-12T03:33:12.940 DEBUG:boto:auth_path=/dkyges-20/obj 2024-04-12T03:33:12.940 DEBUG:boto:Method: DELETE 2024-04-12T03:33:12.940 DEBUG:boto:Path: /dkyges-20/obj 2024-04-12T03:33:12.940 DEBUG:boto:Data: 2024-04-12T03:33:12.940 DEBUG:boto:Headers: {} 2024-04-12T03:33:12.940 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:33:12.940 DEBUG:boto:Port: 8001 2024-04-12T03:33:12.940 DEBUG:boto:Params: {} 2024-04-12T03:33:12.941 DEBUG:boto:Token: None 2024-04-12T03:33:12.941 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:33:12 GMT /dkyges-20/obj 2024-04-12T03:33:12.941 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zHsqGX/HqGB/9loQ/7VTVTDla5M= 2024-04-12T03:33:12.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:33:12 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zHsqGX/HqGB/9loQ/7VTVTDla5M=', 'Content-Length': '0'} 2024-04-12T03:33:12.950 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000009078562256deed34-006618ab78-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:33:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:33:12.950 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-19 2024-04-12T03:33:12.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-19', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:33:12.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-19', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:33:12.951 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-19 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:33:13.014 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:33:13.014 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:33:13.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:13.054+0000 7f732aaf9780 1 waiting to reach incremental sync.. 2024-04-12T03:33:18.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:18.055+0000 7f732aaf9780 1 waiting to reach incremental sync.. 2024-04-12T03:33:23.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:23.055+0000 7f732aaf9780 1 waiting to reach incremental sync.. 2024-04-12T03:33:28.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:28.056+0000 7f732aaf9780 1 waiting to reach incremental sync.. 2024-04-12T03:33:33.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:33.057+0000 7f732aaf9780 1 waiting to reach incremental sync.. 2024-04-12T03:33:38.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:38.057+0000 7f732aaf9780 1 waiting to reach incremental sync.. 2024-04-12T03:33:43.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:43.058+0000 7f732aaf9780 1 waiting to reach incremental sync.. 2024-04-12T03:33:48.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:48.059+0000 7f732aaf9780 1 waiting to reach incremental sync.. 2024-04-12T03:33:53.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:53.060+0000 7f732aaf9780 1 waiting to reach incremental sync.. 2024-04-12T03:33:58.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:33:58.060+0000 7f732aaf9780 1 waiting to reach incremental sync.. 2024-04-12T03:34:03.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:03.061+0000 7f732aaf9780 1 waiting to reach incremental sync.. 2024-04-12T03:34:08.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:08.063+0000 7f732aaf9780 1 bucket sync caught up with source: 2024-04-12T03:34:08.064 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000003.13.7, , ] 2024-04-12T03:34:08.065 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000003.13.7, , ] 2024-04-12T03:34:08.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:08.063+0000 7f732aaf9780 0 bucket checkpoint complete 2024-04-12T03:34:08.082 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-19 2024-04-12T03:34:08.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-19', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:34:08.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-19', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:34:08.083 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-19 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:34:08.150 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:34:08.150 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:34:08.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:08.189+0000 7f51fb6e7780 1 bucket sync caught up with source: 2024-04-12T03:34:08.190 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000001.14.14, , ] 2024-04-12T03:34:08.191 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000001.14.14, , ] 2024-04-12T03:34:08.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:08.189+0000 7f51fb6e7780 0 bucket checkpoint complete 2024-04-12T03:34:08.205 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-19 zones={a2, a1} 2024-04-12T03:34:08.205 DEBUG:boto:path=/dkyges-19/ 2024-04-12T03:34:08.205 DEBUG:boto:auth_path=/dkyges-19/ 2024-04-12T03:34:08.205 DEBUG:boto:Method: HEAD 2024-04-12T03:34:08.205 DEBUG:boto:Path: /dkyges-19/ 2024-04-12T03:34:08.205 DEBUG:boto:Data: 2024-04-12T03:34:08.205 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.205 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:08.206 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.206 DEBUG:boto:Params: {} 2024-04-12T03:34:08.206 DEBUG:boto:Token: None 2024-04-12T03:34:08.206 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:34:08 GMT /dkyges-19/ 2024-04-12T03:34:08.207 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wk90YYVET2/0NFVqLl0ckE3mY3o= 2024-04-12T03:34:08.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wk90YYVET2/0NFVqLl0ckE3mY3o=', 'Content-Length': '0'} 2024-04-12T03:34:08.209 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2ccf5bb71006d3b-006618abb0-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.210 DEBUG:boto:path=/dkyges-19/ 2024-04-12T03:34:08.210 DEBUG:boto:auth_path=/dkyges-19/ 2024-04-12T03:34:08.210 DEBUG:boto:Method: HEAD 2024-04-12T03:34:08.210 DEBUG:boto:Path: /dkyges-19/ 2024-04-12T03:34:08.210 DEBUG:boto:Data: 2024-04-12T03:34:08.210 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.210 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:08.210 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.210 DEBUG:boto:Params: {} 2024-04-12T03:34:08.211 DEBUG:boto:Token: None 2024-04-12T03:34:08.211 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:34:08 GMT /dkyges-19/ 2024-04-12T03:34:08.211 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wk90YYVET2/0NFVqLl0ckE3mY3o= 2024-04-12T03:34:08.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wk90YYVET2/0NFVqLl0ckE3mY3o=', 'Content-Length': '0'} 2024-04-12T03:34:08.213 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbe0ddc6208020e8-006618abb0-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.213 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:34:08.213 DEBUG:boto:path=/dkyges-19/ 2024-04-12T03:34:08.213 DEBUG:boto:auth_path=/dkyges-19/ 2024-04-12T03:34:08.214 DEBUG:boto:path=/dkyges-19/?versions&max-keys=999 2024-04-12T03:34:08.214 DEBUG:boto:auth_path=/dkyges-19/?versions&max-keys=999 2024-04-12T03:34:08.214 DEBUG:boto:Method: GET 2024-04-12T03:34:08.214 DEBUG:boto:Path: /dkyges-19/?versions&max-keys=999 2024-04-12T03:34:08.214 DEBUG:boto:Data: 2024-04-12T03:34:08.214 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.214 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:08.214 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.214 DEBUG:boto:Params: {} 2024-04-12T03:34:08.214 DEBUG:boto:Token: None 2024-04-12T03:34:08.215 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:08 GMT /dkyges-19/?versions 2024-04-12T03:34:08.215 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:k2SOJvnn/K7KNo8g5r9FeYsaZm8= 2024-04-12T03:34:08.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:k2SOJvnn/K7KNo8g5r9FeYsaZm8=', 'Content-Length': '0'} 2024-04-12T03:34:08.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fded38e2cab27137-006618abb0-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.217 DEBUG:boto:b'dkyges-19999falseobjnulltrue2024-04-12T03:33:12.918Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:34:08.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:34:08.218 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:34:08.218 DEBUG:boto:path=/dkyges-19/ 2024-04-12T03:34:08.218 DEBUG:boto:auth_path=/dkyges-19/ 2024-04-12T03:34:08.218 DEBUG:boto:path=/dkyges-19/?versions&max-keys=999 2024-04-12T03:34:08.218 DEBUG:boto:auth_path=/dkyges-19/?versions&max-keys=999 2024-04-12T03:34:08.218 DEBUG:boto:Method: GET 2024-04-12T03:34:08.218 DEBUG:boto:Path: /dkyges-19/?versions&max-keys=999 2024-04-12T03:34:08.218 DEBUG:boto:Data: 2024-04-12T03:34:08.218 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:08.219 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.219 DEBUG:boto:Params: {} 2024-04-12T03:34:08.219 DEBUG:boto:Token: None 2024-04-12T03:34:08.219 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:08 GMT /dkyges-19/?versions 2024-04-12T03:34:08.219 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:k2SOJvnn/K7KNo8g5r9FeYsaZm8= 2024-04-12T03:34:08.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:k2SOJvnn/K7KNo8g5r9FeYsaZm8=', 'Content-Length': '0'} 2024-04-12T03:34:08.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056fc462e3cd774c3-006618abb0-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.221 DEBUG:boto:b'dkyges-19999falseobjnulltrue2024-04-12T03:33:12.918Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:34:08.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:34:08.222 DEBUG:boto:path=/dkyges-19/ 2024-04-12T03:34:08.222 DEBUG:boto:auth_path=/dkyges-19/ 2024-04-12T03:34:08.222 DEBUG:boto:path=/dkyges-19/?versions&max-keys=999 2024-04-12T03:34:08.222 DEBUG:boto:auth_path=/dkyges-19/?versions&max-keys=999 2024-04-12T03:34:08.222 DEBUG:boto:Method: GET 2024-04-12T03:34:08.222 DEBUG:boto:Path: /dkyges-19/?versions&max-keys=999 2024-04-12T03:34:08.223 DEBUG:boto:Data: 2024-04-12T03:34:08.223 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.223 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:08.223 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.223 DEBUG:boto:Params: {} 2024-04-12T03:34:08.223 DEBUG:boto:Token: None 2024-04-12T03:34:08.223 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:08 GMT /dkyges-19/?versions 2024-04-12T03:34:08.223 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:k2SOJvnn/K7KNo8g5r9FeYsaZm8= 2024-04-12T03:34:08.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:k2SOJvnn/K7KNo8g5r9FeYsaZm8=', 'Content-Length': '0'} 2024-04-12T03:34:08.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c321e27c3eb4447c-006618abb0-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.225 DEBUG:boto:b'dkyges-19999falseobjnulltrue2024-04-12T03:33:12.918Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:34:08.226 DEBUG:boto:path=/dkyges-19/ 2024-04-12T03:34:08.226 DEBUG:boto:auth_path=/dkyges-19/ 2024-04-12T03:34:08.226 DEBUG:boto:path=/dkyges-19/?versions&max-keys=999 2024-04-12T03:34:08.226 DEBUG:boto:auth_path=/dkyges-19/?versions&max-keys=999 2024-04-12T03:34:08.226 DEBUG:boto:Method: GET 2024-04-12T03:34:08.226 DEBUG:boto:Path: /dkyges-19/?versions&max-keys=999 2024-04-12T03:34:08.226 DEBUG:boto:Data: 2024-04-12T03:34:08.226 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.226 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:08.227 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.227 DEBUG:boto:Params: {} 2024-04-12T03:34:08.227 DEBUG:boto:Token: None 2024-04-12T03:34:08.227 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:08 GMT /dkyges-19/?versions 2024-04-12T03:34:08.227 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:k2SOJvnn/K7KNo8g5r9FeYsaZm8= 2024-04-12T03:34:08.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:k2SOJvnn/K7KNo8g5r9FeYsaZm8=', 'Content-Length': '0'} 2024-04-12T03:34:08.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000263ced09bae90bc3-006618abb0-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.229 DEBUG:boto:b'dkyges-19999falseobjnulltrue2024-04-12T03:33:12.918Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:34:08.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:34:08.230 DEBUG:boto:path=/dkyges-19/obj 2024-04-12T03:34:08.230 DEBUG:boto:auth_path=/dkyges-19/obj 2024-04-12T03:34:08.230 DEBUG:boto:Method: HEAD 2024-04-12T03:34:08.230 DEBUG:boto:Path: /dkyges-19/obj 2024-04-12T03:34:08.230 DEBUG:boto:Data: 2024-04-12T03:34:08.230 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.230 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:08.230 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.230 DEBUG:boto:Params: {} 2024-04-12T03:34:08.230 DEBUG:boto:Token: None 2024-04-12T03:34:08.231 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:34:08 GMT /dkyges-19/obj 2024-04-12T03:34:08.231 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:J1Ynq2TsxDnqmd5z2+tJi4Q1SiY= 2024-04-12T03:34:08.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:J1Ynq2TsxDnqmd5z2+tJi4Q1SiY=', 'Content-Length': '0'} 2024-04-12T03:34:08.232 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000fd7ecac9a86fd28c-006618abb0-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.232 DEBUG:boto:path=/dkyges-19/obj 2024-04-12T03:34:08.232 DEBUG:boto:auth_path=/dkyges-19/obj 2024-04-12T03:34:08.232 DEBUG:boto:Method: HEAD 2024-04-12T03:34:08.232 DEBUG:boto:Path: /dkyges-19/obj 2024-04-12T03:34:08.233 DEBUG:boto:Data: 2024-04-12T03:34:08.233 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.233 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:08.233 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.233 DEBUG:boto:Params: {} 2024-04-12T03:34:08.233 DEBUG:boto:Token: None 2024-04-12T03:34:08.233 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:34:08 GMT /dkyges-19/obj 2024-04-12T03:34:08.233 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:J1Ynq2TsxDnqmd5z2+tJi4Q1SiY= 2024-04-12T03:34:08.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:J1Ynq2TsxDnqmd5z2+tJi4Q1SiY=', 'Content-Length': '0'} 2024-04-12T03:34:08.235 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000865aba56c39675db-006618abb0-110f-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.235 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-19 zones={a2, a1} 2024-04-12T03:34:08.235 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-20 2024-04-12T03:34:08.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-20', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:34:08.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-20', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:34:08.236 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-20 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:34:08.298 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:34:08.299 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:34:08.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:08.337+0000 7f1488508780 1 bucket sync caught up with source: 2024-04-12T03:34:08.339 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000001.14.14, , ] 2024-04-12T03:34:08.339 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000001.14.14, , ] 2024-04-12T03:34:08.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:08.337+0000 7f1488508780 0 bucket checkpoint complete 2024-04-12T03:34:08.353 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-20 2024-04-12T03:34:08.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-20', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:34:08.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-20', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:34:08.353 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-20 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:34:08.415 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:34:08.416 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:34:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:08.453+0000 7f16b2579780 1 bucket sync caught up with source: 2024-04-12T03:34:08.454 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000003.56.7, , ] 2024-04-12T03:34:08.454 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000003.56.7, , ] 2024-04-12T03:34:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:08.453+0000 7f16b2579780 0 bucket checkpoint complete 2024-04-12T03:34:08.468 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-20 zones={a2, a1} 2024-04-12T03:34:08.468 DEBUG:boto:path=/dkyges-20/ 2024-04-12T03:34:08.468 DEBUG:boto:auth_path=/dkyges-20/ 2024-04-12T03:34:08.468 DEBUG:boto:Method: HEAD 2024-04-12T03:34:08.468 DEBUG:boto:Path: /dkyges-20/ 2024-04-12T03:34:08.468 DEBUG:boto:Data: 2024-04-12T03:34:08.469 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.469 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:08.469 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.469 DEBUG:boto:Params: {} 2024-04-12T03:34:08.469 DEBUG:boto:Token: None 2024-04-12T03:34:08.469 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:34:08 GMT /dkyges-20/ 2024-04-12T03:34:08.469 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r0gmPMRXtLnG9x3LF7OtddxhRig= 2024-04-12T03:34:08.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r0gmPMRXtLnG9x3LF7OtddxhRig=', 'Content-Length': '0'} 2024-04-12T03:34:08.471 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9d45ba0424817a3-006618abb0-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.472 DEBUG:boto:path=/dkyges-20/ 2024-04-12T03:34:08.472 DEBUG:boto:auth_path=/dkyges-20/ 2024-04-12T03:34:08.472 DEBUG:boto:Method: HEAD 2024-04-12T03:34:08.472 DEBUG:boto:Path: /dkyges-20/ 2024-04-12T03:34:08.472 DEBUG:boto:Data: 2024-04-12T03:34:08.472 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.472 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:08.472 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.472 DEBUG:boto:Params: {} 2024-04-12T03:34:08.472 DEBUG:boto:Token: None 2024-04-12T03:34:08.473 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:34:08 GMT /dkyges-20/ 2024-04-12T03:34:08.473 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r0gmPMRXtLnG9x3LF7OtddxhRig= 2024-04-12T03:34:08.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r0gmPMRXtLnG9x3LF7OtddxhRig=', 'Content-Length': '0'} 2024-04-12T03:34:08.475 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e373dc090919f10b-006618abb0-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.475 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:34:08.475 DEBUG:boto:path=/dkyges-20/ 2024-04-12T03:34:08.475 DEBUG:boto:auth_path=/dkyges-20/ 2024-04-12T03:34:08.475 DEBUG:boto:path=/dkyges-20/?versions&max-keys=999 2024-04-12T03:34:08.475 DEBUG:boto:auth_path=/dkyges-20/?versions&max-keys=999 2024-04-12T03:34:08.475 DEBUG:boto:Method: GET 2024-04-12T03:34:08.476 DEBUG:boto:Path: /dkyges-20/?versions&max-keys=999 2024-04-12T03:34:08.476 DEBUG:boto:Data: 2024-04-12T03:34:08.476 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.476 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:08.476 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.476 DEBUG:boto:Params: {} 2024-04-12T03:34:08.476 DEBUG:boto:Token: None 2024-04-12T03:34:08.476 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:08 GMT /dkyges-20/?versions 2024-04-12T03:34:08.476 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S9J/NFPHHo/Bj0jz3aVIXDPH7Mo= 2024-04-12T03:34:08.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S9J/NFPHHo/Bj0jz3aVIXDPH7Mo=', 'Content-Length': '0'} 2024-04-12T03:34:08.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab136bf363dd8b8e-006618abb0-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.479 DEBUG:boto:b'dkyges-20999falseobjnulltrue2024-04-12T03:33:12.942Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:34:08.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:34:08.479 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:34:08.479 DEBUG:boto:path=/dkyges-20/ 2024-04-12T03:34:08.480 DEBUG:boto:auth_path=/dkyges-20/ 2024-04-12T03:34:08.480 DEBUG:boto:path=/dkyges-20/?versions&max-keys=999 2024-04-12T03:34:08.480 DEBUG:boto:auth_path=/dkyges-20/?versions&max-keys=999 2024-04-12T03:34:08.480 DEBUG:boto:Method: GET 2024-04-12T03:34:08.480 DEBUG:boto:Path: /dkyges-20/?versions&max-keys=999 2024-04-12T03:34:08.480 DEBUG:boto:Data: 2024-04-12T03:34:08.481 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.481 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:08.481 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.481 DEBUG:boto:Params: {} 2024-04-12T03:34:08.481 DEBUG:boto:Token: None 2024-04-12T03:34:08.481 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:08 GMT /dkyges-20/?versions 2024-04-12T03:34:08.481 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S9J/NFPHHo/Bj0jz3aVIXDPH7Mo= 2024-04-12T03:34:08.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S9J/NFPHHo/Bj0jz3aVIXDPH7Mo=', 'Content-Length': '0'} 2024-04-12T03:34:08.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbff466118d15cc6-006618abb0-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.484 DEBUG:boto:b'dkyges-20999falseobjnulltrue2024-04-12T03:33:12.942Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:34:08.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2024-04-12T03:34:08.484 DEBUG:boto:path=/dkyges-20/ 2024-04-12T03:34:08.484 DEBUG:boto:auth_path=/dkyges-20/ 2024-04-12T03:34:08.484 DEBUG:boto:path=/dkyges-20/?versions&max-keys=999 2024-04-12T03:34:08.484 DEBUG:boto:auth_path=/dkyges-20/?versions&max-keys=999 2024-04-12T03:34:08.485 DEBUG:boto:Method: GET 2024-04-12T03:34:08.485 DEBUG:boto:Path: /dkyges-20/?versions&max-keys=999 2024-04-12T03:34:08.485 DEBUG:boto:Data: 2024-04-12T03:34:08.485 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.485 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:08.485 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.485 DEBUG:boto:Params: {} 2024-04-12T03:34:08.485 DEBUG:boto:Token: None 2024-04-12T03:34:08.485 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:08 GMT /dkyges-20/?versions 2024-04-12T03:34:08.485 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S9J/NFPHHo/Bj0jz3aVIXDPH7Mo= 2024-04-12T03:34:08.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S9J/NFPHHo/Bj0jz3aVIXDPH7Mo=', 'Content-Length': '0'} 2024-04-12T03:34:08.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c1802d20863fdd2-006618abb0-1113-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.530 DEBUG:boto:b'dkyges-20999falseobjnulltrue2024-04-12T03:33:12.942Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:34:08.531 DEBUG:boto:path=/dkyges-20/ 2024-04-12T03:34:08.531 DEBUG:boto:auth_path=/dkyges-20/ 2024-04-12T03:34:08.531 DEBUG:boto:path=/dkyges-20/?versions&max-keys=999 2024-04-12T03:34:08.531 DEBUG:boto:auth_path=/dkyges-20/?versions&max-keys=999 2024-04-12T03:34:08.531 DEBUG:boto:Method: GET 2024-04-12T03:34:08.531 DEBUG:boto:Path: /dkyges-20/?versions&max-keys=999 2024-04-12T03:34:08.532 DEBUG:boto:Data: 2024-04-12T03:34:08.532 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.532 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:08.532 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.532 DEBUG:boto:Params: {} 2024-04-12T03:34:08.532 DEBUG:boto:Token: None 2024-04-12T03:34:08.532 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:08 GMT /dkyges-20/?versions 2024-04-12T03:34:08.532 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S9J/NFPHHo/Bj0jz3aVIXDPH7Mo= 2024-04-12T03:34:08.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S9J/NFPHHo/Bj0jz3aVIXDPH7Mo=', 'Content-Length': '0'} 2024-04-12T03:34:08.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d4601f2493c0746-006618abb0-110f-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.535 DEBUG:boto:b'dkyges-20999falseobjnulltrue2024-04-12T03:33:12.942Zrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:34:08.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2024-04-12T03:34:08.535 DEBUG:boto:path=/dkyges-20/obj 2024-04-12T03:34:08.535 DEBUG:boto:auth_path=/dkyges-20/obj 2024-04-12T03:34:08.535 DEBUG:boto:Method: HEAD 2024-04-12T03:34:08.536 DEBUG:boto:Path: /dkyges-20/obj 2024-04-12T03:34:08.536 DEBUG:boto:Data: 2024-04-12T03:34:08.536 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.536 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:08.536 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.536 DEBUG:boto:Params: {} 2024-04-12T03:34:08.536 DEBUG:boto:Token: None 2024-04-12T03:34:08.536 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:34:08 GMT /dkyges-20/obj 2024-04-12T03:34:08.536 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sfMIkVzcItvUuqgUy+JUQ/oXL84= 2024-04-12T03:34:08.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sfMIkVzcItvUuqgUy+JUQ/oXL84=', 'Content-Length': '0'} 2024-04-12T03:34:08.538 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c6591d3eefbd0d27-006618abb0-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.538 DEBUG:boto:path=/dkyges-20/obj 2024-04-12T03:34:08.538 DEBUG:boto:auth_path=/dkyges-20/obj 2024-04-12T03:34:08.538 DEBUG:boto:Method: HEAD 2024-04-12T03:34:08.538 DEBUG:boto:Path: /dkyges-20/obj 2024-04-12T03:34:08.538 DEBUG:boto:Data: 2024-04-12T03:34:08.539 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.539 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:08.539 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.539 DEBUG:boto:Params: {} 2024-04-12T03:34:08.539 DEBUG:boto:Token: None 2024-04-12T03:34:08.539 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:34:08 GMT /dkyges-20/obj 2024-04-12T03:34:08.539 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sfMIkVzcItvUuqgUy+JUQ/oXL84= 2024-04-12T03:34:08.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sfMIkVzcItvUuqgUy+JUQ/oXL84=', 'Content-Length': '0'} 2024-04-12T03:34:08.541 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007a36e697ebe0358a-006618abb0-110f-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.541 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-20 zones={a2, a1} 2024-04-12T03:34:08.542 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2024-04-12T03:34:08.543 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-21 2024-04-12T03:34:08.543 DEBUG:boto:path=/dkyges-21/ 2024-04-12T03:34:08.543 DEBUG:boto:auth_path=/dkyges-21/ 2024-04-12T03:34:08.543 DEBUG:boto:Method: PUT 2024-04-12T03:34:08.543 DEBUG:boto:Path: /dkyges-21/ 2024-04-12T03:34:08.543 DEBUG:boto:Data: 2024-04-12T03:34:08.543 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.544 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:08.544 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.544 DEBUG:boto:Params: {} 2024-04-12T03:34:08.544 DEBUG:boto:Token: None 2024-04-12T03:34:08.544 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:08 GMT /dkyges-21/ 2024-04-12T03:34:08.544 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:infBdX2+jeaHdtSAXsC3NKtd/dw= 2024-04-12T03:34:08.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:infBdX2+jeaHdtSAXsC3NKtd/dw=', 'Content-Length': '0'} 2024-04-12T03:34:08.560 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000be7b2a65164f026a-006618abb0-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.560 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-22 2024-04-12T03:34:08.560 DEBUG:boto:path=/dkyges-22/ 2024-04-12T03:34:08.560 DEBUG:boto:auth_path=/dkyges-22/ 2024-04-12T03:34:08.560 DEBUG:boto:Method: PUT 2024-04-12T03:34:08.560 DEBUG:boto:Path: /dkyges-22/ 2024-04-12T03:34:08.560 DEBUG:boto:Data: 2024-04-12T03:34:08.561 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.561 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:08.561 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.561 DEBUG:boto:Params: {} 2024-04-12T03:34:08.561 DEBUG:boto:Token: None 2024-04-12T03:34:08.561 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:08 GMT /dkyges-22/ 2024-04-12T03:34:08.561 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X/yQKUjfacLjbWS/qsBFzIVNfCo= 2024-04-12T03:34:08.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X/yQKUjfacLjbWS/qsBFzIVNfCo=', 'Content-Length': '0'} 2024-04-12T03:34:08.587 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000098b7631efcbdec26-006618abb0-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.588 DEBUG:boto:Using access key provided by client. 2024-04-12T03:34:08.588 DEBUG:boto:Using secret key provided by client. 2024-04-12T03:34:08.589 DEBUG:boto:Using access key provided by client. 2024-04-12T03:34:08.589 DEBUG:boto:Using secret key provided by client. 2024-04-12T03:34:08.589 DEBUG:boto:Using access key provided by client. 2024-04-12T03:34:08.589 DEBUG:boto:Using secret key provided by client. 2024-04-12T03:34:08.590 DEBUG:boto:Using access key provided by client. 2024-04-12T03:34:08.590 DEBUG:boto:Using secret key provided by client. 2024-04-12T03:34:08.590 INFO:rgw_multi.tests:create bucket zone=b1 name=dkyges-23 2024-04-12T03:34:08.590 DEBUG:boto:path=/dkyges-23/ 2024-04-12T03:34:08.590 DEBUG:boto:auth_path=/dkyges-23/ 2024-04-12T03:34:08.590 DEBUG:boto:Method: PUT 2024-04-12T03:34:08.591 DEBUG:boto:Path: /dkyges-23/ 2024-04-12T03:34:08.591 DEBUG:boto:Data: 2024-04-12T03:34:08.591 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.591 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8000 2024-04-12T03:34:08.591 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.591 DEBUG:boto:Params: {} 2024-04-12T03:34:08.591 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:34:08.591 DEBUG:boto:Token: None 2024-04-12T03:34:08.591 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:08 GMT /dkyges-23/ 2024-04-12T03:34:08.592 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X3W8woMOZW/LvihMA3hWdgMV/yo= 2024-04-12T03:34:08.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X3W8woMOZW/LvihMA3hWdgMV/yo=', 'Content-Length': '0'} 2024-04-12T03:34:08.652 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000496e841a478ccf6e-006618abb0-10fc-b1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.653 INFO:rgw_multi.tests:create bucket zone=b2 name=dkyges-24 2024-04-12T03:34:08.653 DEBUG:boto:path=/dkyges-24/ 2024-04-12T03:34:08.653 DEBUG:boto:auth_path=/dkyges-24/ 2024-04-12T03:34:08.653 DEBUG:boto:Method: PUT 2024-04-12T03:34:08.653 DEBUG:boto:Path: /dkyges-24/ 2024-04-12T03:34:08.653 DEBUG:boto:Data: 2024-04-12T03:34:08.653 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.653 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8001 2024-04-12T03:34:08.653 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.653 DEBUG:boto:Params: {} 2024-04-12T03:34:08.654 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:34:08.654 DEBUG:boto:Token: None 2024-04-12T03:34:08.654 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:08 GMT /dkyges-24/ 2024-04-12T03:34:08.654 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0lCOFBrzeIGhYqIMuc9POHZ54cE= 2024-04-12T03:34:08.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0lCOFBrzeIGhYqIMuc9POHZ54cE=', 'Content-Length': '0'} 2024-04-12T03:34:08.719 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000086e9fff03d9e214d-006618abb0-1101-b2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.720 DEBUG:boto:path=/dkyges-21/ 2024-04-12T03:34:08.720 DEBUG:boto:auth_path=/dkyges-21/ 2024-04-12T03:34:08.720 DEBUG:boto:path=/dkyges-21/?versioning 2024-04-12T03:34:08.720 DEBUG:boto:auth_path=/dkyges-21/?versioning 2024-04-12T03:34:08.720 DEBUG:boto:Method: PUT 2024-04-12T03:34:08.720 DEBUG:boto:Path: /dkyges-21/?versioning 2024-04-12T03:34:08.720 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:34:08.720 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.720 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:08.720 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.721 DEBUG:boto:Params: {} 2024-04-12T03:34:08.721 DEBUG:boto:Token: None 2024-04-12T03:34:08.721 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:08 GMT /dkyges-21/?versioning 2024-04-12T03:34:08.721 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+3ah8b9LvEzbkJsiXpAbPaXfUCY= 2024-04-12T03:34:08.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+3ah8b9LvEzbkJsiXpAbPaXfUCY=', 'Content-Length': '228'} 2024-04-12T03:34:08.727 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b2c47673356a1c72-006618abb0-110f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.727 DEBUG:boto:path=/dkyges-21/ 2024-04-12T03:34:08.727 DEBUG:boto:auth_path=/dkyges-21/ 2024-04-12T03:34:08.727 DEBUG:boto:path=/dkyges-21/?versioning 2024-04-12T03:34:08.727 DEBUG:boto:auth_path=/dkyges-21/?versioning 2024-04-12T03:34:08.727 DEBUG:boto:Method: GET 2024-04-12T03:34:08.727 DEBUG:boto:Path: /dkyges-21/?versioning 2024-04-12T03:34:08.727 DEBUG:boto:Data: 2024-04-12T03:34:08.727 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.727 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:08.728 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.728 DEBUG:boto:Params: {} 2024-04-12T03:34:08.728 DEBUG:boto:Token: None 2024-04-12T03:34:08.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:08 GMT /dkyges-21/?versioning 2024-04-12T03:34:08.728 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oMs8V4/f9HLLOF22GPHWip8FpI= 2024-04-12T03:34:08.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oMs8V4/f9HLLOF22GPHWip8FpI=', 'Content-Length': '0'} 2024-04-12T03:34:08.729 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dbe44873e113bf92-006618abb0-110f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.774 DEBUG:boto:EnabledDisabled 2024-04-12T03:34:08.775 DEBUG:boto:path=/dkyges-22/ 2024-04-12T03:34:08.776 DEBUG:boto:auth_path=/dkyges-22/ 2024-04-12T03:34:08.776 DEBUG:boto:path=/dkyges-22/?versioning 2024-04-12T03:34:08.776 DEBUG:boto:auth_path=/dkyges-22/?versioning 2024-04-12T03:34:08.776 DEBUG:boto:Method: PUT 2024-04-12T03:34:08.776 DEBUG:boto:Path: /dkyges-22/?versioning 2024-04-12T03:34:08.776 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:34:08.777 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.777 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:08.777 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.777 DEBUG:boto:Params: {} 2024-04-12T03:34:08.777 DEBUG:boto:Token: None 2024-04-12T03:34:08.777 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:08 GMT /dkyges-22/?versioning 2024-04-12T03:34:08.777 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3DNFEa7vrNt4NXAMRue+k7DUlrk= 2024-04-12T03:34:08.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3DNFEa7vrNt4NXAMRue+k7DUlrk=', 'Content-Length': '228'} 2024-04-12T03:34:08.787 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000781eb555efad3050-006618abb0-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.787 DEBUG:boto:path=/dkyges-22/ 2024-04-12T03:34:08.787 DEBUG:boto:auth_path=/dkyges-22/ 2024-04-12T03:34:08.787 DEBUG:boto:path=/dkyges-22/?versioning 2024-04-12T03:34:08.788 DEBUG:boto:auth_path=/dkyges-22/?versioning 2024-04-12T03:34:08.788 DEBUG:boto:Method: GET 2024-04-12T03:34:08.788 DEBUG:boto:Path: /dkyges-22/?versioning 2024-04-12T03:34:08.788 DEBUG:boto:Data: 2024-04-12T03:34:08.788 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.789 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:08.789 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.789 DEBUG:boto:Params: {} 2024-04-12T03:34:08.789 DEBUG:boto:Token: None 2024-04-12T03:34:08.789 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:08 GMT /dkyges-22/?versioning 2024-04-12T03:34:08.790 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jYKFvN4N8ZJcqksjf0RgymZbz0I= 2024-04-12T03:34:08.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jYKFvN4N8ZJcqksjf0RgymZbz0I=', 'Content-Length': '0'} 2024-04-12T03:34:08.791 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f2278f59bdce488a-006618abb0-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.792 DEBUG:boto:EnabledDisabled 2024-04-12T03:34:08.792 DEBUG:boto:path=/dkyges-23/ 2024-04-12T03:34:08.792 DEBUG:boto:auth_path=/dkyges-23/ 2024-04-12T03:34:08.792 DEBUG:boto:path=/dkyges-23/?versioning 2024-04-12T03:34:08.792 DEBUG:boto:auth_path=/dkyges-23/?versioning 2024-04-12T03:34:08.793 DEBUG:boto:Method: PUT 2024-04-12T03:34:08.793 DEBUG:boto:Path: /dkyges-23/?versioning 2024-04-12T03:34:08.793 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:34:08.793 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.793 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8000 2024-04-12T03:34:08.794 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.794 DEBUG:boto:Params: {} 2024-04-12T03:34:08.794 DEBUG:boto:Token: None 2024-04-12T03:34:08.794 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:08 GMT /dkyges-23/?versioning 2024-04-12T03:34:08.795 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2pCFpJGkXE0MpMJGeVgDv8rgTZ4= 2024-04-12T03:34:08.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2pCFpJGkXE0MpMJGeVgDv8rgTZ4=', 'Content-Length': '228'} 2024-04-12T03:34:08.804 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a7114a96242d09ea-006618abb0-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.805 DEBUG:boto:path=/dkyges-23/ 2024-04-12T03:34:08.805 DEBUG:boto:auth_path=/dkyges-23/ 2024-04-12T03:34:08.805 DEBUG:boto:path=/dkyges-23/?versioning 2024-04-12T03:34:08.805 DEBUG:boto:auth_path=/dkyges-23/?versioning 2024-04-12T03:34:08.805 DEBUG:boto:Method: GET 2024-04-12T03:34:08.806 DEBUG:boto:Path: /dkyges-23/?versioning 2024-04-12T03:34:08.806 DEBUG:boto:Data: 2024-04-12T03:34:08.806 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.806 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8000 2024-04-12T03:34:08.806 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.806 DEBUG:boto:Params: {} 2024-04-12T03:34:08.806 DEBUG:boto:Token: None 2024-04-12T03:34:08.806 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:08 GMT /dkyges-23/?versioning 2024-04-12T03:34:08.807 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T94yO+Ld1bFigqxKxH9Hr2TF0fY= 2024-04-12T03:34:08.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T94yO+Ld1bFigqxKxH9Hr2TF0fY=', 'Content-Length': '0'} 2024-04-12T03:34:08.808 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f86dd73484e3e3a2-006618abb0-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.808 DEBUG:boto:EnabledDisabled 2024-04-12T03:34:08.808 DEBUG:boto:path=/dkyges-24/ 2024-04-12T03:34:08.808 DEBUG:boto:auth_path=/dkyges-24/ 2024-04-12T03:34:08.809 DEBUG:boto:path=/dkyges-24/?versioning 2024-04-12T03:34:08.809 DEBUG:boto:auth_path=/dkyges-24/?versioning 2024-04-12T03:34:08.809 DEBUG:boto:Method: PUT 2024-04-12T03:34:08.809 DEBUG:boto:Path: /dkyges-24/?versioning 2024-04-12T03:34:08.809 DEBUG:boto:Data: Enabled Disabled 2024-04-12T03:34:08.809 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.809 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8001 2024-04-12T03:34:08.809 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.809 DEBUG:boto:Params: {} 2024-04-12T03:34:08.810 DEBUG:boto:Token: None 2024-04-12T03:34:08.810 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:08 GMT /dkyges-24/?versioning 2024-04-12T03:34:08.810 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oSfPhULs7oTRC2SnXarecOavvp0= 2024-04-12T03:34:08.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oSfPhULs7oTRC2SnXarecOavvp0=', 'Content-Length': '228'} 2024-04-12T03:34:08.824 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f66994a8c19b42be-006618abb0-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.824 DEBUG:boto:path=/dkyges-24/ 2024-04-12T03:34:08.825 DEBUG:boto:auth_path=/dkyges-24/ 2024-04-12T03:34:08.825 DEBUG:boto:path=/dkyges-24/?versioning 2024-04-12T03:34:08.825 DEBUG:boto:auth_path=/dkyges-24/?versioning 2024-04-12T03:34:08.825 DEBUG:boto:Method: GET 2024-04-12T03:34:08.825 DEBUG:boto:Path: /dkyges-24/?versioning 2024-04-12T03:34:08.825 DEBUG:boto:Data: 2024-04-12T03:34:08.825 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.825 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8001 2024-04-12T03:34:08.825 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.825 DEBUG:boto:Params: {} 2024-04-12T03:34:08.825 DEBUG:boto:Token: None 2024-04-12T03:34:08.826 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:08 GMT /dkyges-24/?versioning 2024-04-12T03:34:08.826 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MgnUuqIENadRGMcOpxqTZZ2zjag= 2024-04-12T03:34:08.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MgnUuqIENadRGMcOpxqTZZ2zjag=', 'Content-Length': '0'} 2024-04-12T03:34:08.827 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002dd2c6e19ab393fb-006618abb0-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.827 DEBUG:boto:EnabledDisabled 2024-04-12T03:34:08.827 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2024-04-12T03:34:08.828 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-25 2024-04-12T03:34:08.828 DEBUG:boto:path=/dkyges-25/ 2024-04-12T03:34:08.828 DEBUG:boto:auth_path=/dkyges-25/ 2024-04-12T03:34:08.828 DEBUG:boto:Method: PUT 2024-04-12T03:34:08.828 DEBUG:boto:Path: /dkyges-25/ 2024-04-12T03:34:08.828 DEBUG:boto:Data: 2024-04-12T03:34:08.828 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.828 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:08.829 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.829 DEBUG:boto:Params: {} 2024-04-12T03:34:08.829 DEBUG:boto:Token: None 2024-04-12T03:34:08.829 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:08 GMT /dkyges-25/ 2024-04-12T03:34:08.829 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zWu6En89OaTroabT2ni6Ug+nOzs= 2024-04-12T03:34:08.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zWu6En89OaTroabT2ni6Ug+nOzs=', 'Content-Length': '0'} 2024-04-12T03:34:08.845 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a8a03bac3cdcfddf-006618abb0-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.845 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-26 2024-04-12T03:34:08.845 DEBUG:boto:path=/dkyges-26/ 2024-04-12T03:34:08.845 DEBUG:boto:auth_path=/dkyges-26/ 2024-04-12T03:34:08.845 DEBUG:boto:Method: PUT 2024-04-12T03:34:08.845 DEBUG:boto:Path: /dkyges-26/ 2024-04-12T03:34:08.845 DEBUG:boto:Data: 2024-04-12T03:34:08.845 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.845 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:08.845 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.845 DEBUG:boto:Params: {} 2024-04-12T03:34:08.846 DEBUG:boto:Token: None 2024-04-12T03:34:08.846 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:08 GMT /dkyges-26/ 2024-04-12T03:34:08.846 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vO13wth/aF/dWJ/3edXt7otubec= 2024-04-12T03:34:08.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vO13wth/aF/dWJ/3edXt7otubec=', 'Content-Length': '0'} 2024-04-12T03:34:08.879 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000086ff8f954c5b1615-006618abb0-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.879 INFO:rgw_multi.tests:create bucket zone=b1 name=dkyges-27 2024-04-12T03:34:08.879 DEBUG:boto:path=/dkyges-27/ 2024-04-12T03:34:08.879 DEBUG:boto:auth_path=/dkyges-27/ 2024-04-12T03:34:08.880 DEBUG:boto:Method: PUT 2024-04-12T03:34:08.880 DEBUG:boto:Path: /dkyges-27/ 2024-04-12T03:34:08.880 DEBUG:boto:Data: 2024-04-12T03:34:08.880 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.880 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8000 2024-04-12T03:34:08.880 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.880 DEBUG:boto:Params: {} 2024-04-12T03:34:08.880 DEBUG:boto:Token: None 2024-04-12T03:34:08.881 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:08 GMT /dkyges-27/ 2024-04-12T03:34:08.881 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6WFLnt6ylGwVDml2kxk62hMS+dw= 2024-04-12T03:34:08.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6WFLnt6ylGwVDml2kxk62hMS+dw=', 'Content-Length': '0'} 2024-04-12T03:34:08.907 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000023e6187c1f0b2ff6-006618abb0-10fc-b1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.907 INFO:rgw_multi.tests:create bucket zone=b2 name=dkyges-28 2024-04-12T03:34:08.907 DEBUG:boto:path=/dkyges-28/ 2024-04-12T03:34:08.907 DEBUG:boto:auth_path=/dkyges-28/ 2024-04-12T03:34:08.907 DEBUG:boto:Method: PUT 2024-04-12T03:34:08.908 DEBUG:boto:Path: /dkyges-28/ 2024-04-12T03:34:08.908 DEBUG:boto:Data: 2024-04-12T03:34:08.908 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.908 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8001 2024-04-12T03:34:08.908 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.908 DEBUG:boto:Params: {} 2024-04-12T03:34:08.908 DEBUG:boto:Token: None 2024-04-12T03:34:08.908 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:08 GMT /dkyges-28/ 2024-04-12T03:34:08.908 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LWRqmoat6ZjMSkZNoM4rZfgIBLA= 2024-04-12T03:34:08.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LWRqmoat6ZjMSkZNoM4rZfgIBLA=', 'Content-Length': '0'} 2024-04-12T03:34:08.934 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b8e54430442d525d-006618abb0-1101-b2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.934 DEBUG:boto:path=/dkyges-25/ 2024-04-12T03:34:08.934 DEBUG:boto:auth_path=/dkyges-25/ 2024-04-12T03:34:08.934 DEBUG:boto:path=/dkyges-25/?acl 2024-04-12T03:34:08.935 DEBUG:boto:auth_path=/dkyges-25/?acl 2024-04-12T03:34:08.935 DEBUG:boto:Method: GET 2024-04-12T03:34:08.935 DEBUG:boto:Path: /dkyges-25/?acl 2024-04-12T03:34:08.935 DEBUG:boto:Data: 2024-04-12T03:34:08.935 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.935 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:08.935 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.935 DEBUG:boto:Params: {} 2024-04-12T03:34:08.935 DEBUG:boto:Token: None 2024-04-12T03:34:08.935 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:08 GMT /dkyges-25/?acl 2024-04-12T03:34:08.936 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mCP5NRkZGNm0SPGYRuGFTK9nw40= 2024-04-12T03:34:08.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mCP5NRkZGNm0SPGYRuGFTK9nw40=', 'Content-Length': '0'} 2024-04-12T03:34:08.937 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a16302cf51116f90-006618abb0-110f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.937 DEBUG:boto:path=/dkyges-25/ 2024-04-12T03:34:08.938 DEBUG:boto:auth_path=/dkyges-25/ 2024-04-12T03:34:08.938 DEBUG:boto:path=/dkyges-25/?acl 2024-04-12T03:34:08.938 DEBUG:boto:auth_path=/dkyges-25/?acl 2024-04-12T03:34:08.938 DEBUG:boto:Method: PUT 2024-04-12T03:34:08.938 DEBUG:boto:Path: /dkyges-25/?acl 2024-04-12T03:34:08.938 DEBUG:boto:Data: 2024-04-12T03:34:08.938 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-12T03:34:08.938 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:08.938 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.938 DEBUG:boto:Params: {} 2024-04-12T03:34:08.938 DEBUG:boto:Token: None 2024-04-12T03:34:08.939 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:08 GMT x-amz-acl:public-read /dkyges-25/?acl 2024-04-12T03:34:08.939 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AmRZWf4tbf9Ko/yHSllYlesLtC4= 2024-04-12T03:34:08.939 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AmRZWf4tbf9Ko/yHSllYlesLtC4=', 'Content-Length': '0'} 2024-04-12T03:34:08.944 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009e0c3fc3742de6ab-006618abb0-110f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.945 DEBUG:boto:path=/dkyges-25/ 2024-04-12T03:34:08.945 DEBUG:boto:auth_path=/dkyges-25/ 2024-04-12T03:34:08.945 DEBUG:boto:path=/dkyges-25/?acl 2024-04-12T03:34:08.945 DEBUG:boto:auth_path=/dkyges-25/?acl 2024-04-12T03:34:08.945 DEBUG:boto:Method: GET 2024-04-12T03:34:08.945 DEBUG:boto:Path: /dkyges-25/?acl 2024-04-12T03:34:08.945 DEBUG:boto:Data: 2024-04-12T03:34:08.945 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.945 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:08.945 DEBUG:boto:Port: 8000 2024-04-12T03:34:08.945 DEBUG:boto:Params: {} 2024-04-12T03:34:08.946 DEBUG:boto:Token: None 2024-04-12T03:34:08.946 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:08 GMT /dkyges-25/?acl 2024-04-12T03:34:08.946 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mCP5NRkZGNm0SPGYRuGFTK9nw40= 2024-04-12T03:34:08.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mCP5NRkZGNm0SPGYRuGFTK9nw40=', 'Content-Length': '0'} 2024-04-12T03:34:08.947 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ba493c32d2fc4c03-006618abb0-110f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.947 DEBUG:boto:path=/dkyges-26/ 2024-04-12T03:34:08.947 DEBUG:boto:auth_path=/dkyges-26/ 2024-04-12T03:34:08.947 DEBUG:boto:path=/dkyges-26/?acl 2024-04-12T03:34:08.948 DEBUG:boto:auth_path=/dkyges-26/?acl 2024-04-12T03:34:08.948 DEBUG:boto:Method: GET 2024-04-12T03:34:08.948 DEBUG:boto:Path: /dkyges-26/?acl 2024-04-12T03:34:08.948 DEBUG:boto:Data: 2024-04-12T03:34:08.948 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.948 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:08.948 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.948 DEBUG:boto:Params: {} 2024-04-12T03:34:08.948 DEBUG:boto:Token: None 2024-04-12T03:34:08.948 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:08 GMT /dkyges-26/?acl 2024-04-12T03:34:08.948 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VB7C/qCBrlzXkyayIPpRkdM7lrs= 2024-04-12T03:34:08.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VB7C/qCBrlzXkyayIPpRkdM7lrs=', 'Content-Length': '0'} 2024-04-12T03:34:08.950 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a4c2dca5d73f2e87-006618abb0-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.950 DEBUG:boto:path=/dkyges-26/ 2024-04-12T03:34:08.950 DEBUG:boto:auth_path=/dkyges-26/ 2024-04-12T03:34:08.950 DEBUG:boto:path=/dkyges-26/?acl 2024-04-12T03:34:08.950 DEBUG:boto:auth_path=/dkyges-26/?acl 2024-04-12T03:34:08.951 DEBUG:boto:Method: PUT 2024-04-12T03:34:08.951 DEBUG:boto:Path: /dkyges-26/?acl 2024-04-12T03:34:08.951 DEBUG:boto:Data: 2024-04-12T03:34:08.951 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-12T03:34:08.951 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:08.951 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.951 DEBUG:boto:Params: {} 2024-04-12T03:34:08.951 DEBUG:boto:Token: None 2024-04-12T03:34:08.951 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:08 GMT x-amz-acl:public-read /dkyges-26/?acl 2024-04-12T03:34:08.951 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e3+FHidx0cr7IRjc7bUxf6Q8rm4= 2024-04-12T03:34:08.952 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e3+FHidx0cr7IRjc7bUxf6Q8rm4=', 'Content-Length': '0'} 2024-04-12T03:34:08.960 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003ab8fc90a2b7f472-006618abb0-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:08.960 DEBUG:boto:path=/dkyges-26/ 2024-04-12T03:34:08.960 DEBUG:boto:auth_path=/dkyges-26/ 2024-04-12T03:34:08.960 DEBUG:boto:path=/dkyges-26/?acl 2024-04-12T03:34:08.961 DEBUG:boto:auth_path=/dkyges-26/?acl 2024-04-12T03:34:08.961 DEBUG:boto:Method: GET 2024-04-12T03:34:08.961 DEBUG:boto:Path: /dkyges-26/?acl 2024-04-12T03:34:08.961 DEBUG:boto:Data: 2024-04-12T03:34:08.961 DEBUG:boto:Headers: {} 2024-04-12T03:34:08.961 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:08.961 DEBUG:boto:Port: 8001 2024-04-12T03:34:08.961 DEBUG:boto:Params: {} 2024-04-12T03:34:08.961 DEBUG:boto:Token: None 2024-04-12T03:34:08.961 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:08 GMT /dkyges-26/?acl 2024-04-12T03:34:08.961 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VB7C/qCBrlzXkyayIPpRkdM7lrs= 2024-04-12T03:34:08.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VB7C/qCBrlzXkyayIPpRkdM7lrs=', 'Content-Length': '0'} 2024-04-12T03:34:08.963 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000015bbc783df609019-006618abb0-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 12 Apr 2024 03:34:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:09.006 DEBUG:boto:path=/dkyges-27/ 2024-04-12T03:34:09.007 DEBUG:boto:auth_path=/dkyges-27/ 2024-04-12T03:34:09.007 DEBUG:boto:path=/dkyges-27/?acl 2024-04-12T03:34:09.007 DEBUG:boto:auth_path=/dkyges-27/?acl 2024-04-12T03:34:09.007 DEBUG:boto:Method: GET 2024-04-12T03:34:09.007 DEBUG:boto:Path: /dkyges-27/?acl 2024-04-12T03:34:09.007 DEBUG:boto:Data: 2024-04-12T03:34:09.007 DEBUG:boto:Headers: {} 2024-04-12T03:34:09.007 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8000 2024-04-12T03:34:09.007 DEBUG:boto:Port: 8000 2024-04-12T03:34:09.007 DEBUG:boto:Params: {} 2024-04-12T03:34:09.008 DEBUG:boto:Token: None 2024-04-12T03:34:09.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:09 GMT /dkyges-27/?acl 2024-04-12T03:34:09.008 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cJfw6HnfO4t8IIq+JOXsopIQVyA= 2024-04-12T03:34:09.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cJfw6HnfO4t8IIq+JOXsopIQVyA=', 'Content-Length': '0'} 2024-04-12T03:34:09.009 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000eecd4d8d160b58d7-006618abb1-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 12 Apr 2024 03:34:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:09.010 DEBUG:boto:path=/dkyges-27/ 2024-04-12T03:34:09.010 DEBUG:boto:auth_path=/dkyges-27/ 2024-04-12T03:34:09.010 DEBUG:boto:path=/dkyges-27/?acl 2024-04-12T03:34:09.010 DEBUG:boto:auth_path=/dkyges-27/?acl 2024-04-12T03:34:09.010 DEBUG:boto:Method: PUT 2024-04-12T03:34:09.010 DEBUG:boto:Path: /dkyges-27/?acl 2024-04-12T03:34:09.010 DEBUG:boto:Data: 2024-04-12T03:34:09.010 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-12T03:34:09.011 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8000 2024-04-12T03:34:09.011 DEBUG:boto:Port: 8000 2024-04-12T03:34:09.011 DEBUG:boto:Params: {} 2024-04-12T03:34:09.011 DEBUG:boto:Token: None 2024-04-12T03:34:09.011 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:09 GMT x-amz-acl:public-read /dkyges-27/?acl 2024-04-12T03:34:09.011 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/UX34HC1vBeYu8ZLUe2/AmDpRj0= 2024-04-12T03:34:09.011 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/UX34HC1vBeYu8ZLUe2/AmDpRj0=', 'Content-Length': '0'} 2024-04-12T03:34:09.041 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000052708bc26c0e48e1-006618abb1-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:09.042 DEBUG:boto:path=/dkyges-27/ 2024-04-12T03:34:09.042 DEBUG:boto:auth_path=/dkyges-27/ 2024-04-12T03:34:09.042 DEBUG:boto:path=/dkyges-27/?acl 2024-04-12T03:34:09.042 DEBUG:boto:auth_path=/dkyges-27/?acl 2024-04-12T03:34:09.042 DEBUG:boto:Method: GET 2024-04-12T03:34:09.042 DEBUG:boto:Path: /dkyges-27/?acl 2024-04-12T03:34:09.042 DEBUG:boto:Data: 2024-04-12T03:34:09.042 DEBUG:boto:Headers: {} 2024-04-12T03:34:09.043 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8000 2024-04-12T03:34:09.043 DEBUG:boto:Port: 8000 2024-04-12T03:34:09.043 DEBUG:boto:Params: {} 2024-04-12T03:34:09.043 DEBUG:boto:Token: None 2024-04-12T03:34:09.043 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:09 GMT /dkyges-27/?acl 2024-04-12T03:34:09.043 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cJfw6HnfO4t8IIq+JOXsopIQVyA= 2024-04-12T03:34:09.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cJfw6HnfO4t8IIq+JOXsopIQVyA=', 'Content-Length': '0'} 2024-04-12T03:34:09.044 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000071ea4c5b52cea20a-006618abb1-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 12 Apr 2024 03:34:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:09.045 DEBUG:boto:path=/dkyges-28/ 2024-04-12T03:34:09.045 DEBUG:boto:auth_path=/dkyges-28/ 2024-04-12T03:34:09.045 DEBUG:boto:path=/dkyges-28/?acl 2024-04-12T03:34:09.045 DEBUG:boto:auth_path=/dkyges-28/?acl 2024-04-12T03:34:09.045 DEBUG:boto:Method: GET 2024-04-12T03:34:09.045 DEBUG:boto:Path: /dkyges-28/?acl 2024-04-12T03:34:09.045 DEBUG:boto:Data: 2024-04-12T03:34:09.045 DEBUG:boto:Headers: {} 2024-04-12T03:34:09.045 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8001 2024-04-12T03:34:09.045 DEBUG:boto:Port: 8001 2024-04-12T03:34:09.045 DEBUG:boto:Params: {} 2024-04-12T03:34:09.046 DEBUG:boto:Token: None 2024-04-12T03:34:09.046 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:09 GMT /dkyges-28/?acl 2024-04-12T03:34:09.046 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C6BgKw2zpooadQ4glnBgotsHveI= 2024-04-12T03:34:09.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C6BgKw2zpooadQ4glnBgotsHveI=', 'Content-Length': '0'} 2024-04-12T03:34:09.047 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000d08f5c05a58da2a-006618abb1-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 12 Apr 2024 03:34:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:09.047 DEBUG:boto:path=/dkyges-28/ 2024-04-12T03:34:09.047 DEBUG:boto:auth_path=/dkyges-28/ 2024-04-12T03:34:09.048 DEBUG:boto:path=/dkyges-28/?acl 2024-04-12T03:34:09.048 DEBUG:boto:auth_path=/dkyges-28/?acl 2024-04-12T03:34:09.048 DEBUG:boto:Method: PUT 2024-04-12T03:34:09.048 DEBUG:boto:Path: /dkyges-28/?acl 2024-04-12T03:34:09.048 DEBUG:boto:Data: 2024-04-12T03:34:09.048 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2024-04-12T03:34:09.048 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8001 2024-04-12T03:34:09.048 DEBUG:boto:Port: 8001 2024-04-12T03:34:09.048 DEBUG:boto:Params: {} 2024-04-12T03:34:09.048 DEBUG:boto:Token: None 2024-04-12T03:34:09.048 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:09 GMT x-amz-acl:public-read /dkyges-28/?acl 2024-04-12T03:34:09.049 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+5Agnv4aUPtZ4Z5snrNV6uyQHM= 2024-04-12T03:34:09.049 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+5Agnv4aUPtZ4Z5snrNV6uyQHM=', 'Content-Length': '0'} 2024-04-12T03:34:09.060 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003ed50e27afd5534f-006618abb1-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:09.060 DEBUG:boto:path=/dkyges-28/ 2024-04-12T03:34:09.060 DEBUG:boto:auth_path=/dkyges-28/ 2024-04-12T03:34:09.060 DEBUG:boto:path=/dkyges-28/?acl 2024-04-12T03:34:09.060 DEBUG:boto:auth_path=/dkyges-28/?acl 2024-04-12T03:34:09.060 DEBUG:boto:Method: GET 2024-04-12T03:34:09.060 DEBUG:boto:Path: /dkyges-28/?acl 2024-04-12T03:34:09.060 DEBUG:boto:Data: 2024-04-12T03:34:09.060 DEBUG:boto:Headers: {} 2024-04-12T03:34:09.061 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8001 2024-04-12T03:34:09.061 DEBUG:boto:Port: 8001 2024-04-12T03:34:09.061 DEBUG:boto:Params: {} 2024-04-12T03:34:09.061 DEBUG:boto:Token: None 2024-04-12T03:34:09.061 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:09 GMT /dkyges-28/?acl 2024-04-12T03:34:09.061 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C6BgKw2zpooadQ4glnBgotsHveI= 2024-04-12T03:34:09.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C6BgKw2zpooadQ4glnBgotsHveI=', 'Content-Length': '0'} 2024-04-12T03:34:09.062 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a2305e81a25a497c-006618abb1-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 12 Apr 2024 03:34:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:09.063 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2024-04-12T03:34:09.063 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-29 2024-04-12T03:34:09.063 DEBUG:boto:path=/dkyges-29/ 2024-04-12T03:34:09.063 DEBUG:boto:auth_path=/dkyges-29/ 2024-04-12T03:34:09.063 DEBUG:boto:Method: PUT 2024-04-12T03:34:09.063 DEBUG:boto:Path: /dkyges-29/ 2024-04-12T03:34:09.064 DEBUG:boto:Data: 2024-04-12T03:34:09.064 DEBUG:boto:Headers: {} 2024-04-12T03:34:09.064 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:09.064 DEBUG:boto:Port: 8000 2024-04-12T03:34:09.064 DEBUG:boto:Params: {} 2024-04-12T03:34:09.064 DEBUG:boto:Token: None 2024-04-12T03:34:09.064 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:09 GMT /dkyges-29/ 2024-04-12T03:34:09.064 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:39Ns6wjiamhlXz1DB1C776BO7oI= 2024-04-12T03:34:09.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:39Ns6wjiamhlXz1DB1C776BO7oI=', 'Content-Length': '0'} 2024-04-12T03:34:09.086 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000de017b6d6e9a2a5c-006618abb1-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:09.086 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-30 2024-04-12T03:34:09.087 DEBUG:boto:path=/dkyges-30/ 2024-04-12T03:34:09.087 DEBUG:boto:auth_path=/dkyges-30/ 2024-04-12T03:34:09.087 DEBUG:boto:Method: PUT 2024-04-12T03:34:09.087 DEBUG:boto:Path: /dkyges-30/ 2024-04-12T03:34:09.087 DEBUG:boto:Data: 2024-04-12T03:34:09.087 DEBUG:boto:Headers: {} 2024-04-12T03:34:09.087 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:09.087 DEBUG:boto:Port: 8001 2024-04-12T03:34:09.087 DEBUG:boto:Params: {} 2024-04-12T03:34:09.087 DEBUG:boto:Token: None 2024-04-12T03:34:09.088 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:09 GMT /dkyges-30/ 2024-04-12T03:34:09.088 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vdVMsdf8HZOFJSNdvmPP2kNXAQc= 2024-04-12T03:34:09.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vdVMsdf8HZOFJSNdvmPP2kNXAQc=', 'Content-Length': '0'} 2024-04-12T03:34:09.113 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001583eb99b9b2403c-006618abb1-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:09.113 INFO:rgw_multi.tests:create bucket zone=b1 name=dkyges-31 2024-04-12T03:34:09.113 DEBUG:boto:path=/dkyges-31/ 2024-04-12T03:34:09.113 DEBUG:boto:auth_path=/dkyges-31/ 2024-04-12T03:34:09.113 DEBUG:boto:Method: PUT 2024-04-12T03:34:09.113 DEBUG:boto:Path: /dkyges-31/ 2024-04-12T03:34:09.114 DEBUG:boto:Data: 2024-04-12T03:34:09.114 DEBUG:boto:Headers: {} 2024-04-12T03:34:09.114 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8000 2024-04-12T03:34:09.114 DEBUG:boto:Port: 8000 2024-04-12T03:34:09.114 DEBUG:boto:Params: {} 2024-04-12T03:34:09.114 DEBUG:boto:Token: None 2024-04-12T03:34:09.114 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:09 GMT /dkyges-31/ 2024-04-12T03:34:09.114 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kWXQL+rqo/EZVW7Ob1Cq17Vi7cw= 2024-04-12T03:34:09.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kWXQL+rqo/EZVW7Ob1Cq17Vi7cw=', 'Content-Length': '0'} 2024-04-12T03:34:09.140 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003c2b68b99b2434c8-006618abb1-10fc-b1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:09.141 INFO:rgw_multi.tests:create bucket zone=b2 name=dkyges-32 2024-04-12T03:34:09.141 DEBUG:boto:path=/dkyges-32/ 2024-04-12T03:34:09.141 DEBUG:boto:auth_path=/dkyges-32/ 2024-04-12T03:34:09.141 DEBUG:boto:Method: PUT 2024-04-12T03:34:09.141 DEBUG:boto:Path: /dkyges-32/ 2024-04-12T03:34:09.141 DEBUG:boto:Data: 2024-04-12T03:34:09.141 DEBUG:boto:Headers: {} 2024-04-12T03:34:09.141 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8001 2024-04-12T03:34:09.141 DEBUG:boto:Port: 8001 2024-04-12T03:34:09.141 DEBUG:boto:Params: {} 2024-04-12T03:34:09.142 DEBUG:boto:Token: None 2024-04-12T03:34:09.142 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:09 GMT /dkyges-32/ 2024-04-12T03:34:09.142 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7yk61zqv6A73VXxE9OJV2rGBFnw= 2024-04-12T03:34:09.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7yk61zqv6A73VXxE9OJV2rGBFnw=', 'Content-Length': '0'} 2024-04-12T03:34:09.168 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000079291e889084717a-006618abb1-1101-b2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:09.168 DEBUG:boto:path=/dkyges-29/ 2024-04-12T03:34:09.168 DEBUG:boto:auth_path=/dkyges-29/ 2024-04-12T03:34:09.168 DEBUG:boto:path=/dkyges-29/?cors 2024-04-12T03:34:09.168 DEBUG:boto:auth_path=/dkyges-29/?cors 2024-04-12T03:34:09.168 DEBUG:boto:Method: PUT 2024-04-12T03:34:09.168 DEBUG:boto:Path: /dkyges-29/?cors 2024-04-12T03:34:09.169 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-12T03:34:09.169 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-12T03:34:09.169 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:09.169 DEBUG:boto:Port: 8000 2024-04-12T03:34:09.169 DEBUG:boto:Params: {} 2024-04-12T03:34:09.169 DEBUG:boto:Token: None 2024-04-12T03:34:09.169 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 12 Apr 2024 03:34:09 GMT /dkyges-29/?cors 2024-04-12T03:34:09.169 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nmLxJPr/L7MojlKYvGV1bf3N/bo= 2024-04-12T03:34:09.169 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nmLxJPr/L7MojlKYvGV1bf3N/bo=', 'Content-Length': '218'} 2024-04-12T03:34:09.175 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002bfefef5b42e03e3-006618abb1-110f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:09.175 DEBUG:boto:path=/dkyges-29/ 2024-04-12T03:34:09.175 DEBUG:boto:auth_path=/dkyges-29/ 2024-04-12T03:34:09.175 DEBUG:boto:path=/dkyges-29/?cors 2024-04-12T03:34:09.175 DEBUG:boto:auth_path=/dkyges-29/?cors 2024-04-12T03:34:09.176 DEBUG:boto:Method: GET 2024-04-12T03:34:09.176 DEBUG:boto:Path: /dkyges-29/?cors 2024-04-12T03:34:09.176 DEBUG:boto:Data: 2024-04-12T03:34:09.176 DEBUG:boto:Headers: {} 2024-04-12T03:34:09.176 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:09.176 DEBUG:boto:Port: 8000 2024-04-12T03:34:09.176 DEBUG:boto:Params: {} 2024-04-12T03:34:09.176 DEBUG:boto:Token: None 2024-04-12T03:34:09.176 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:09 GMT /dkyges-29/?cors 2024-04-12T03:34:09.177 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H3uAaEFZI607biUWnBXCqdXH9ic= 2024-04-12T03:34:09.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H3uAaEFZI607biUWnBXCqdXH9ic=', 'Content-Length': '0'} 2024-04-12T03:34:09.179 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000aee29f428503d14c-006618abb1-110f-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 12 Apr 2024 03:34:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:09.179 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-12T03:34:09.179 DEBUG:boto:path=/dkyges-30/ 2024-04-12T03:34:09.179 DEBUG:boto:auth_path=/dkyges-30/ 2024-04-12T03:34:09.179 DEBUG:boto:path=/dkyges-30/?cors 2024-04-12T03:34:09.180 DEBUG:boto:auth_path=/dkyges-30/?cors 2024-04-12T03:34:09.180 DEBUG:boto:Method: PUT 2024-04-12T03:34:09.180 DEBUG:boto:Path: /dkyges-30/?cors 2024-04-12T03:34:09.180 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-12T03:34:09.180 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-12T03:34:09.180 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:09.180 DEBUG:boto:Port: 8001 2024-04-12T03:34:09.180 DEBUG:boto:Params: {} 2024-04-12T03:34:09.180 DEBUG:boto:Token: None 2024-04-12T03:34:09.180 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 12 Apr 2024 03:34:09 GMT /dkyges-30/?cors 2024-04-12T03:34:09.181 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CxQWGcUQImXQJE5y85w72C3Gmdw= 2024-04-12T03:34:09.181 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CxQWGcUQImXQJE5y85w72C3Gmdw=', 'Content-Length': '218'} 2024-04-12T03:34:10.194 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000063f260fc6bb2ab9a-006618abb1-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:10.194 DEBUG:boto:path=/dkyges-30/ 2024-04-12T03:34:10.194 DEBUG:boto:auth_path=/dkyges-30/ 2024-04-12T03:34:10.194 DEBUG:boto:path=/dkyges-30/?cors 2024-04-12T03:34:10.194 DEBUG:boto:auth_path=/dkyges-30/?cors 2024-04-12T03:34:10.195 DEBUG:boto:Method: GET 2024-04-12T03:34:10.195 DEBUG:boto:Path: /dkyges-30/?cors 2024-04-12T03:34:10.195 DEBUG:boto:Data: 2024-04-12T03:34:10.195 DEBUG:boto:Headers: {} 2024-04-12T03:34:10.195 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:10.195 DEBUG:boto:Port: 8001 2024-04-12T03:34:10.195 DEBUG:boto:Params: {} 2024-04-12T03:34:10.195 DEBUG:boto:Token: None 2024-04-12T03:34:10.195 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:10 GMT /dkyges-30/?cors 2024-04-12T03:34:10.196 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bQvqiU1SP5AaDj+LH3GdgekBLzY= 2024-04-12T03:34:10.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bQvqiU1SP5AaDj+LH3GdgekBLzY=', 'Content-Length': '0'} 2024-04-12T03:34:10.198 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005d0ae202f6ce43e6-006618abb2-1113-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 12 Apr 2024 03:34:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:10.198 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-12T03:34:10.198 DEBUG:boto:path=/dkyges-31/ 2024-04-12T03:34:10.198 DEBUG:boto:auth_path=/dkyges-31/ 2024-04-12T03:34:10.198 DEBUG:boto:path=/dkyges-31/?cors 2024-04-12T03:34:10.198 DEBUG:boto:auth_path=/dkyges-31/?cors 2024-04-12T03:34:10.199 DEBUG:boto:Method: PUT 2024-04-12T03:34:10.199 DEBUG:boto:Path: /dkyges-31/?cors 2024-04-12T03:34:10.199 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-12T03:34:10.199 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-12T03:34:10.199 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8000 2024-04-12T03:34:10.199 DEBUG:boto:Port: 8000 2024-04-12T03:34:10.199 DEBUG:boto:Params: {} 2024-04-12T03:34:10.199 DEBUG:boto:Token: None 2024-04-12T03:34:10.199 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 12 Apr 2024 03:34:10 GMT /dkyges-31/?cors 2024-04-12T03:34:10.199 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aZHWqHr7KshgLxffPfgWgkYZSPU= 2024-04-12T03:34:10.199 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aZHWqHr7KshgLxffPfgWgkYZSPU=', 'Content-Length': '218'} 2024-04-12T03:34:11.210 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000523cc671df1f0de7-006618abb2-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:11.211 DEBUG:boto:path=/dkyges-31/ 2024-04-12T03:34:11.211 DEBUG:boto:auth_path=/dkyges-31/ 2024-04-12T03:34:11.211 DEBUG:boto:path=/dkyges-31/?cors 2024-04-12T03:34:11.211 DEBUG:boto:auth_path=/dkyges-31/?cors 2024-04-12T03:34:11.211 DEBUG:boto:Method: GET 2024-04-12T03:34:11.211 DEBUG:boto:Path: /dkyges-31/?cors 2024-04-12T03:34:11.211 DEBUG:boto:Data: 2024-04-12T03:34:11.211 DEBUG:boto:Headers: {} 2024-04-12T03:34:11.211 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8000 2024-04-12T03:34:11.211 DEBUG:boto:Port: 8000 2024-04-12T03:34:11.212 DEBUG:boto:Params: {} 2024-04-12T03:34:11.212 DEBUG:boto:Token: None 2024-04-12T03:34:11.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:11 GMT /dkyges-31/?cors 2024-04-12T03:34:11.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bNfCJVVWiCBLOjQBkhugCi2LEC0= 2024-04-12T03:34:11.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:11 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bNfCJVVWiCBLOjQBkhugCi2LEC0=', 'Content-Length': '0'} 2024-04-12T03:34:11.213 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000089a71a544cc8f104-006618abb3-10fc-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 12 Apr 2024 03:34:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:11.258 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-12T03:34:11.259 DEBUG:boto:path=/dkyges-32/ 2024-04-12T03:34:11.259 DEBUG:boto:auth_path=/dkyges-32/ 2024-04-12T03:34:11.259 DEBUG:boto:path=/dkyges-32/?cors 2024-04-12T03:34:11.259 DEBUG:boto:auth_path=/dkyges-32/?cors 2024-04-12T03:34:11.259 DEBUG:boto:Method: PUT 2024-04-12T03:34:11.259 DEBUG:boto:Path: /dkyges-32/?cors 2024-04-12T03:34:11.259 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2024-04-12T03:34:11.260 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2024-04-12T03:34:11.260 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8001 2024-04-12T03:34:11.260 DEBUG:boto:Port: 8001 2024-04-12T03:34:11.260 DEBUG:boto:Params: {} 2024-04-12T03:34:11.260 DEBUG:boto:Token: None 2024-04-12T03:34:11.260 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 12 Apr 2024 03:34:11 GMT /dkyges-32/?cors 2024-04-12T03:34:11.260 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DfUUhqaHSuVdPjgFL1l23y/ijPc= 2024-04-12T03:34:11.260 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:11 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DfUUhqaHSuVdPjgFL1l23y/ijPc=', 'Content-Length': '218'} 2024-04-12T03:34:12.270 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009d28056c685a839b-006618abb3-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:12.270 DEBUG:boto:path=/dkyges-32/ 2024-04-12T03:34:12.270 DEBUG:boto:auth_path=/dkyges-32/ 2024-04-12T03:34:12.271 DEBUG:boto:path=/dkyges-32/?cors 2024-04-12T03:34:12.271 DEBUG:boto:auth_path=/dkyges-32/?cors 2024-04-12T03:34:12.271 DEBUG:boto:Method: GET 2024-04-12T03:34:12.271 DEBUG:boto:Path: /dkyges-32/?cors 2024-04-12T03:34:12.271 DEBUG:boto:Data: 2024-04-12T03:34:12.271 DEBUG:boto:Headers: {} 2024-04-12T03:34:12.271 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8001 2024-04-12T03:34:12.271 DEBUG:boto:Port: 8001 2024-04-12T03:34:12.271 DEBUG:boto:Params: {} 2024-04-12T03:34:12.272 DEBUG:boto:Token: None 2024-04-12T03:34:12.272 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:34:12 GMT /dkyges-32/?cors 2024-04-12T03:34:12.272 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VpJekt5EY4ioql0MMooMr7nsB74= 2024-04-12T03:34:12.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:12 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VpJekt5EY4ioql0MMooMr7nsB74=', 'Content-Length': '0'} 2024-04-12T03:34:12.273 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000002571b33401b88a-006618abb4-1101-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 12 Apr 2024 03:34:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:12.318 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2024-04-12T03:34:12.319 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2024-04-12T03:34:12.319 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-33 2024-04-12T03:34:12.320 DEBUG:boto:path=/dkyges-33/ 2024-04-12T03:34:12.320 DEBUG:boto:auth_path=/dkyges-33/ 2024-04-12T03:34:12.320 DEBUG:boto:Method: PUT 2024-04-12T03:34:12.320 DEBUG:boto:Path: /dkyges-33/ 2024-04-12T03:34:12.320 DEBUG:boto:Data: 2024-04-12T03:34:12.320 DEBUG:boto:Headers: {} 2024-04-12T03:34:12.320 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:34:12.320 DEBUG:boto:Port: 8000 2024-04-12T03:34:12.320 DEBUG:boto:Params: {} 2024-04-12T03:34:12.320 DEBUG:boto:Token: None 2024-04-12T03:34:12.321 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:12 GMT /dkyges-33/ 2024-04-12T03:34:12.321 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BQRwp/sfFXVv86cP//ZZwrVi/9g= 2024-04-12T03:34:12.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:12 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BQRwp/sfFXVv86cP//ZZwrVi/9g=', 'Content-Length': '0'} 2024-04-12T03:34:12.337 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e2b4978df1c4f213-006618abb4-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:12.338 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-34 2024-04-12T03:34:12.338 DEBUG:boto:path=/dkyges-34/ 2024-04-12T03:34:12.338 DEBUG:boto:auth_path=/dkyges-34/ 2024-04-12T03:34:12.338 DEBUG:boto:Method: PUT 2024-04-12T03:34:12.338 DEBUG:boto:Path: /dkyges-34/ 2024-04-12T03:34:12.338 DEBUG:boto:Data: 2024-04-12T03:34:12.338 DEBUG:boto:Headers: {} 2024-04-12T03:34:12.338 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:34:12.338 DEBUG:boto:Port: 8001 2024-04-12T03:34:12.338 DEBUG:boto:Params: {} 2024-04-12T03:34:12.339 DEBUG:boto:Token: None 2024-04-12T03:34:12.339 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:34:12 GMT /dkyges-34/ 2024-04-12T03:34:12.339 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B7b5std3zpSOLB9YqBfHfbIKm6g= 2024-04-12T03:34:12.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:34:12 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B7b5std3zpSOLB9YqBfHfbIKm6g=', 'Content-Length': '0'} 2024-04-12T03:34:12.365 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000507323e42ca76505-006618abb4-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:34:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:34:12.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:12.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:12.366 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:34:12.437 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:34:12.437 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:34:12.516 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:34:12.529 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:34:12.529 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:12.529 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:34:12.529 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:34:12.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:12.529 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:12.530 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:34:12.530 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:34:12.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:12.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:12.530 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:34:12.530 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:34:12.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:12.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:12.530 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892852.354438_285.1", 2024-04-12T03:34:12.530 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:34:12.354438Z" 2024-04-12T03:34:12.531 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:12.531 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:34:12.532 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:34:12.532 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:34:12.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:12.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:12.533 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:34:12.589 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:34:12.589 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:34:12.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:12.620+0000 7fadee385780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:12.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:12.621+0000 7fadee385780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:12.636 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:34:12.636 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:34:12.636 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:34:12.636 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:34:12.636 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:34:12.636 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:34:12.636 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:12.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:12.637 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:34:12.637 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:12.637 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:34:12.637 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:12.637 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:12.637 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:34:12.637 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:12.637 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:34:12.637 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:12.637 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:34:12.637 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:12.638 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:12.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:12.638 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:12.638 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:34:12.638 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:12.638 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:12.638 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:34:12.638 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:12.638 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:12.638 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:12.639 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:34:12.639 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:12.639 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:12.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:12.639 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:12.639 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:34:12.639 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:12.639 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:12.639 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:34:12.639 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:12.640 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:12.640 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:12.640 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:34:12.640 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:12.640 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:12.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:12.640 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:12.640 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:34:12.640 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:12.641 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:12.641 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.705488_221.1", 2024-04-12T03:34:12.641 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:12.641 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:12.641 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:12.641 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.705488Z", 2024-04-12T03:34:12.641 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:12.641 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:12.641 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:12.641 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:34:12.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:12.642 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:34:12.642 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:34:12.642 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:34:12.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:12.642 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:34:12Z" 2024-04-12T03:34:12.642 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:34:12.643 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.705488_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.705488Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:34:12Z" } 2024-04-12T03:34:12.643 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.705488_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.705488Z', 'realm_epoch': 2}}] 2024-04-12T03:34:12.643 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:34:12.643 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.705488_221.1'} 2024-04-12T03:34:12.644 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.705488_221.1 2024-04-12T03:34:17.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:17.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:17.645 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:34:17.705 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:34:17.705 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:34:17.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:17.739+0000 7fe52bb43780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:17.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:17.740+0000 7fe52bb43780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:17.755 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:34:17.755 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:34:17.755 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:34:17.756 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:34:17.756 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:34:17.756 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:34:17.756 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:17.756 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:17.756 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:34:17.756 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:17.756 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:34:17.756 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:17.756 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:17.757 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:34:17.757 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:17.757 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:34:17.757 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:17.757 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:34:17.757 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:17.757 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:17.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:17.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:17.757 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:34:17.757 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:17.758 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:17.758 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:34:17.758 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:17.758 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:17.758 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:17.758 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:34:17.758 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:17.758 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:17.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:17.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:17.759 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:34:17.759 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:17.759 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:17.759 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:34:17.759 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:17.759 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:17.759 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:17.759 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:34:17.759 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:17.759 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:17.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:17.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:17.760 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:34:17.760 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:17.760 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:17.760 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.635345_216.1", 2024-04-12T03:34:17.760 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:17.760 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:17.760 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:17.760 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.635345Z", 2024-04-12T03:34:17.760 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:17.760 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:17.760 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:17.761 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:34:17.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:17.761 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:34:17.761 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:34:17.761 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:34:17.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:17.761 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:34:17Z" 2024-04-12T03:34:17.761 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:34:17.762 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.635345_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.635345Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:34:17Z" } 2024-04-12T03:34:17.762 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.635345_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.635345Z', 'realm_epoch': 2}}] 2024-04-12T03:34:17.762 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:34:17.763 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.635345_216.1'} 2024-04-12T03:34:17.763 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.635345_216.1 2024-04-12T03:34:22.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:22.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:22.764 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:34:22.854 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:34:22.854 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:34:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:22.920+0000 7f933154d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:22.921+0000 7f933154d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:22.935 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:34:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:34:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:34:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:34:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:34:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:34:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:34:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:34:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:34:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:34:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:34:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:34:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:34:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:34:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:34:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:34:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:34:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:34:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.638183_217.1", 2024-04-12T03:34:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.638183Z", 2024-04-12T03:34:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:34:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:34:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:34:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:34:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:34:22Z" 2024-04-12T03:34:22.941 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:34:22.942 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.638183_217.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.638183Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:34:22Z" } 2024-04-12T03:34:22.942 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.638183_217.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.638183Z', 'realm_epoch': 2}}] 2024-04-12T03:34:22.942 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:34:22.943 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.638183_217.1'} 2024-04-12T03:34:22.943 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.638183_217.1 2024-04-12T03:34:27.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:27.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:27.944 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:34:28.005 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:34:28.006 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:34:28.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:28.039+0000 7f01145e6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:28.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:28.040+0000 7f01145e6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:28.055 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:34:28.055 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:34:28.056 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:34:28.056 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:34:28.056 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:34:28.056 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:34:28.056 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:28.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:28.056 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:34:28.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:28.056 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:34:28.056 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:28.056 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:28.057 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:34:28.057 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:28.057 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:34:28.057 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:28.057 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:34:28.057 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:28.057 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:28.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:28.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:28.057 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:34:28.057 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:28.057 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:34:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:34:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:34:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:34:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:34:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:34:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.705488_221.1", 2024-04-12T03:34:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.705488Z", 2024-04-12T03:34:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:34:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:28.061 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:34:28.061 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:34:28.061 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:34:28.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:28.061 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:34:28Z" 2024-04-12T03:34:28.061 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:34:28.062 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.705488_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.705488Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:34:28Z" } 2024-04-12T03:34:28.062 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.705488_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.705488Z', 'realm_epoch': 2}}] 2024-04-12T03:34:28.062 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:34:28.062 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.705488_221.1'} 2024-04-12T03:34:28.062 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.705488_221.1 2024-04-12T03:34:33.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:33.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:33.064 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:34:33.162 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:34:33.163 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:34:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:33.205+0000 7fe1cb722780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:33.206+0000 7fe1cb722780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:33.225 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:34:33.225 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:34:33.225 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:34:33.225 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:34:33.225 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:34:33.225 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:34:33.226 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:33.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:33.226 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:34:33.226 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:33.226 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:34:33.226 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:33.226 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:33.226 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:34:33.226 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:33.227 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:34:33.227 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:33.227 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:34:33.227 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:33.227 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:33.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:33.227 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:33.227 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:34:33.227 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:33.227 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:33.228 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:34:33.228 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:33.228 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:33.228 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:33.228 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:34:33.228 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:33.228 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:33.228 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:33.228 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:33.228 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:34:33.228 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:33.229 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:33.229 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:34:33.229 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:33.229 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:33.229 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:33.229 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:34:33.229 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:33.229 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:33.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:33.229 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:33.229 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:34:33.230 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:33.230 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:33.230 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.780445_224.1", 2024-04-12T03:34:33.230 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:33.230 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:33.230 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:33.230 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.780445Z", 2024-04-12T03:34:33.230 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:33.230 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:33.230 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:33.230 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:34:33.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:33.231 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:34:33.231 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:34:33.231 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:34:33.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:33.231 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:34:33Z" 2024-04-12T03:34:33.231 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:34:33.232 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.780445_224.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.780445Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:34:33Z" } 2024-04-12T03:34:33.232 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.780445_224.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.780445Z', 'realm_epoch': 2}}] 2024-04-12T03:34:33.232 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:34:33.232 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.780445_224.1'} 2024-04-12T03:34:33.233 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.780445_224.1 2024-04-12T03:34:38.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:38.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:38.234 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:34:38.293 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:34:38.293 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:34:38.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:38.325+0000 7f074ac40780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:38.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:38.327+0000 7f074ac40780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:38.342 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:34:38.342 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:34:38.342 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:34:38.343 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:34:38.343 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:34:38.343 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:34:38.343 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:38.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:38.343 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:34:38.343 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:38.343 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:34:38.343 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:38.343 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:38.343 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:34:38.344 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:38.344 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:34:38.344 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:38.344 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:34:38.344 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:38.344 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:38.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:38.344 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:38.344 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:34:38.344 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:38.344 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:38.344 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:34:38.345 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:38.345 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:38.345 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:38.345 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:34:38.345 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:38.345 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:38.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:38.345 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:38.345 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:34:38.345 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:38.345 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:38.345 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:34:38.346 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:38.346 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:38.346 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:38.346 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:34:38.346 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:38.346 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:38.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:38.346 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:38.346 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:34:38.347 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:38.347 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:38.347 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.702568_220.1", 2024-04-12T03:34:38.347 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:38.347 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:38.347 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:38.347 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.702568Z", 2024-04-12T03:34:38.347 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:38.347 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:38.347 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:38.347 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:34:38.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:38.348 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:34:38.348 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:34:38.348 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:34:38.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:38.348 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:34:38Z" 2024-04-12T03:34:38.348 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:34:38.350 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.702568_220.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.702568Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:34:38Z" } 2024-04-12T03:34:38.350 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.702568_220.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.702568Z', 'realm_epoch': 2}}] 2024-04-12T03:34:38.350 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:34:38.350 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.702568_220.1'} 2024-04-12T03:34:38.350 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.702568_220.1 2024-04-12T03:34:43.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:43.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:43.351 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:34:43.421 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:34:43.421 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:34:43.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:43.528+0000 7fb12dcc9780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:43.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:43.529+0000 7fb12dcc9780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:43.556 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:34:43.556 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:34:43.556 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:34:43.556 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:34:43.557 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:34:43.557 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:34:43.557 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:43.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:43.557 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:34:43.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:43.557 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:34:43.557 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:43.557 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:43.557 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:34:43.558 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:43.558 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:34:43.558 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:43.558 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:34:43.558 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:43.558 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:43.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:43.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:43.558 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:34:43.558 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:43.558 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:43.559 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:34:43.559 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:43.559 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:43.559 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:43.559 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:34:43.559 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:43.559 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:43.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:43.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:43.559 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:34:43.559 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:43.559 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:43.560 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:34:43.560 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:43.560 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:43.560 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:43.560 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:34:43.560 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:43.560 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:43.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:43.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:43.560 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:34:43.560 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:43.561 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:43.561 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.635345_216.1", 2024-04-12T03:34:43.561 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:43.561 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:43.561 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:43.561 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.635345Z", 2024-04-12T03:34:43.561 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:43.561 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:43.561 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:43.561 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:34:43.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:43.561 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:34:43.562 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:34:43.562 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:34:43.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:43.562 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:34:43Z" 2024-04-12T03:34:43.562 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:34:43.563 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.635345_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.635345Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:34:43Z" } 2024-04-12T03:34:43.563 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.635345_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.635345Z', 'realm_epoch': 2}}] 2024-04-12T03:34:43.563 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:34:43.563 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.635345_216.1'} 2024-04-12T03:34:43.563 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.635345_216.1 2024-04-12T03:34:48.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:48.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:48.565 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:34:48.624 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:34:48.624 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:34:48.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:48.657+0000 7f3c462b4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:48.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:48.658+0000 7f3c462b4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:48.673 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:34:48.673 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:34:48.673 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:34:48.674 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:34:48.674 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:34:48.674 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:34:48.674 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:48.674 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:48.674 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:34:48.674 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:48.674 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:34:48.674 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:48.674 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:48.675 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:34:48.675 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:48.675 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:34:48.675 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:48.675 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:34:48.675 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:48.675 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:48.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:48.675 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:48.675 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:34:48.675 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:48.675 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:48.676 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:34:48.676 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:48.676 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:48.676 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:48.676 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:34:48.676 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:48.676 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:48.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:48.676 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:48.677 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:34:48.677 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:48.677 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:48.677 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:34:48.677 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:48.677 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:48.677 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:48.677 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:34:48.677 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:48.677 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:48.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:48.678 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:48.678 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:34:48.678 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:48.678 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:48.678 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.722990_222.1", 2024-04-12T03:34:48.678 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:48.678 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:48.678 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:48.679 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.722990Z", 2024-04-12T03:34:48.679 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:48.679 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:48.679 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:48.679 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:34:48.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:48.679 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:34:48.679 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:34:48.679 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:34:48.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:48.679 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:34:48Z" 2024-04-12T03:34:48.680 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:34:48.680 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.722990_222.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.722990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:34:48Z" } 2024-04-12T03:34:48.681 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.722990_222.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.722990Z', 'realm_epoch': 2}}] 2024-04-12T03:34:48.681 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:34:48.681 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.722990_222.1'} 2024-04-12T03:34:48.681 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.722990_222.1 2024-04-12T03:34:53.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:53.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:53.683 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:34:53.742 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:34:53.743 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:34:53.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:53.776+0000 7f8649421780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:53.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:53.777+0000 7f8649421780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:53.793 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:34:53.793 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:34:53.793 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:34:53.793 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:34:53.793 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:34:53.793 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:34:53.793 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:53.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:53.793 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:34:53.794 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:53.794 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:34:53.794 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:53.794 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:53.794 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:34:53.794 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:53.794 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:34:53.794 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:53.794 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:34:53.794 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:53.794 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:53.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:53.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:53.795 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:34:53.795 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:53.796 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:53.796 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:34:53.796 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:53.796 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:53.796 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:53.796 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:34:53.796 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:53.796 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:53.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:53.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:53.797 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:34:53.797 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:53.797 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:53.797 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:34:53.797 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:53.797 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:53.797 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:53.797 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:34:53.797 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:53.797 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:53.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:53.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:53.798 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:34:53.798 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:53.798 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:53.798 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.634004_215.1", 2024-04-12T03:34:53.798 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:53.798 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:53.798 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:53.798 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.634004Z", 2024-04-12T03:34:53.798 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:53.799 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:53.799 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:53.799 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:34:53.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:53.799 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:34:53.799 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:34:53.799 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:34:53.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:53.799 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:34:53Z" 2024-04-12T03:34:53.799 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:34:53.800 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.634004_215.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.634004Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:34:53Z" } 2024-04-12T03:34:53.800 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.634004_215.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.634004Z', 'realm_epoch': 2}}] 2024-04-12T03:34:53.801 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:34:53.801 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.634004_215.1'} 2024-04-12T03:34:53.801 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.634004_215.1 2024-04-12T03:34:58.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:58.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:34:58.802 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:34:58.912 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:34:58.912 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:34:58.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:58.976+0000 7fbf4c8fa780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:34:58.978+0000 7fbf4c8fa780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:34:58.992 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:34:58.993 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:34:58.993 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:34:58.993 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:34:58.993 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:34:58.994 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:34:58.994 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:58.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:58.994 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:34:58.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:58.994 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:34:58.994 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:58.994 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:58.994 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:34:58.995 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:58.995 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:34:58.995 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:58.995 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:34:58.995 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:58.995 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:58.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:58.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:58.995 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:34:58.995 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:58.995 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:58.995 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:34:58.996 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:58.996 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:58.996 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:58.996 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:34:58.996 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:58.996 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:58.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:58.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:58.996 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:34:58.996 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:58.996 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:58.997 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:34:58.997 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:58.997 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:58.997 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:58.997 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:34:58.997 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:58.997 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:58.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:58.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:34:58.997 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:34:58.997 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:34:58.997 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:34:58.998 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.722990_222.1", 2024-04-12T03:34:58.998 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:34:58.998 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:34:58.998 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:34:58.998 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.722990Z", 2024-04-12T03:34:58.998 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:34:58.998 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:58.998 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:34:58.998 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:34:58.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:58.999 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:34:58.999 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:34:58.999 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:34:58.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:34:58.999 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:34:58Z" 2024-04-12T03:34:58.999 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:34:59.000 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.722990_222.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.722990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:34:58Z" } 2024-04-12T03:34:59.000 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.722990_222.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.722990Z', 'realm_epoch': 2}}] 2024-04-12T03:34:59.000 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:34:59.001 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.722990_222.1'} 2024-04-12T03:34:59.001 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.722990_222.1 2024-04-12T03:35:04.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:04.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:04.007 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:35:04.064 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:35:04.065 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:35:04.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:04.098+0000 7fc16f486780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:04.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:04.100+0000 7fc16f486780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:04.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:35:04.116 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:35:04.117 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:04.118 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:04.119 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.722990_222.1", 2024-04-12T03:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.722990Z", 2024-04-12T03:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:35:04.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:35:04Z" 2024-04-12T03:35:04.121 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:35:04.122 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.722990_222.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.722990Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:35:04Z" } 2024-04-12T03:35:04.123 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.722990_222.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.722990Z', 'realm_epoch': 2}}] 2024-04-12T03:35:04.123 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:35:04.123 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.722990_222.1'} 2024-04-12T03:35:04.123 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.722990_222.1 2024-04-12T03:35:09.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:09.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:09.124 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:35:09.184 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:35:09.184 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:35:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:09.218+0000 7f51cd4d4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:09.219+0000 7f51cd4d4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:09.234 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:35:09.234 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:35:09.234 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:35:09.234 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:35:09.235 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:35:09.235 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:35:09.235 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:09.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:09.236 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:35:09.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:09.236 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:35:09.236 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:09.236 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:09.237 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:35:09.237 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:09.237 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:35:09.237 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:09.237 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:35:09.238 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:09.238 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:09.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:09.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:09.239 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:35:09.239 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:09.239 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:09.239 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:35:09.239 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:09.240 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:09.240 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:09.240 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:35:09.240 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:09.241 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:09.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:09.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:09.241 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:35:09.241 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:09.242 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:09.242 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:35:09.242 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:09.242 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:09.243 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:09.243 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:35:09.243 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:09.243 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:09.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:09.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:09.244 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:35:09.244 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:09.244 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:09.244 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.798136_226.1", 2024-04-12T03:35:09.245 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:09.245 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:09.245 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:09.245 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.798136Z", 2024-04-12T03:35:09.246 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:09.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:09.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:09.246 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:35:09.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:09.247 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:35:09.247 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:35:09.247 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:35:09.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:09.247 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:35:09Z" 2024-04-12T03:35:09.248 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:35:09.249 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.798136_226.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.798136Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:35:09Z" } 2024-04-12T03:35:09.250 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.798136_226.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.798136Z', 'realm_epoch': 2}}] 2024-04-12T03:35:09.250 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:35:09.250 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.798136_226.1'} 2024-04-12T03:35:09.250 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.798136_226.1 2024-04-12T03:35:14.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:14.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:14.251 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:35:14.311 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:35:14.311 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:35:14.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:14.344+0000 7fe2c59a3780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:14.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:14.345+0000 7fe2c59a3780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:14.396 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:35:14.397 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:35:14.397 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:35:14.397 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:35:14.397 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:35:14.397 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:35:14.397 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:14.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:14.397 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:35:14.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:14.397 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:35:14.398 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:14.398 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:14.398 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:35:14.398 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:14.398 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:35:14.398 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:14.398 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:35:14.398 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:14.398 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:14.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:14.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:14.399 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:35:14.399 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:14.399 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:14.399 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:35:14.399 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:14.399 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:14.399 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:14.399 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:35:14.399 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:14.399 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:14.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:14.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:14.400 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:35:14.400 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:14.400 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:14.400 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:35:14.400 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:14.400 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:14.400 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:14.400 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:35:14.400 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:14.400 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:14.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:14.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:14.401 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:35:14.401 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:14.401 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:14.401 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.801301_227.1", 2024-04-12T03:35:14.401 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:14.401 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:14.401 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:14.401 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.801301Z", 2024-04-12T03:35:14.401 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:14.401 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:14.401 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:14.402 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:35:14.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:14.402 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:35:14.402 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:35:14.402 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:35:14.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:14.402 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:35:14Z" 2024-04-12T03:35:14.402 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:35:14.402 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.801301_227.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.801301Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:35:14Z" } 2024-04-12T03:35:14.403 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.801301_227.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.801301Z', 'realm_epoch': 2}}] 2024-04-12T03:35:14.403 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:35:14.403 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.801301_227.1'} 2024-04-12T03:35:14.403 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.801301_227.1 2024-04-12T03:35:19.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:19.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:19.404 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:35:19.467 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:35:19.467 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:35:19.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:19.499+0000 7ff1502ec780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:19.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:19.501+0000 7ff1502ec780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:19.516 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:35:19.516 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:35:19.516 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:35:19.516 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:35:19.516 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:35:19.516 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:35:19.517 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:19.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:19.517 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:35:19.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:19.517 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:35:19.517 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:19.517 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:19.517 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:35:19.517 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:19.517 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:35:19.517 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:19.518 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:35:19.518 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:19.518 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:19.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:19.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:19.518 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:35:19.518 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:19.518 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:19.518 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:35:19.518 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:19.518 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:19.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:19.519 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:35:19.519 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:19.519 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:19.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:19.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:19.519 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:35:19.519 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:19.519 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:19.519 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:35:19.519 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:19.520 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:19.520 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:19.520 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:35:19.520 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:19.520 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:19.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:19.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:19.520 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:35:19.520 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:19.520 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:19.521 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.635345_216.1", 2024-04-12T03:35:19.521 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:19.521 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:19.521 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:19.521 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.635345Z", 2024-04-12T03:35:19.521 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:19.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:19.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:19.521 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:35:19.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:19.521 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:35:19.521 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:35:19.522 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:35:19.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:19.522 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:35:19Z" 2024-04-12T03:35:19.522 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:35:19.523 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.635345_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.635345Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:35:19Z" } 2024-04-12T03:35:19.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.635345_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.635345Z', 'realm_epoch': 2}}] 2024-04-12T03:35:19.523 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:35:19.523 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.635345_216.1'} 2024-04-12T03:35:19.523 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.635345_216.1 2024-04-12T03:35:24.524 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:24.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:24.525 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:35:24.582 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:35:24.582 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:35:24.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:24.615+0000 7efcfa204780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:24.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:24.616+0000 7efcfa204780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:24.631 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:35:24.631 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:35:24.631 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:35:24.631 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:35:24.632 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:35:24.632 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:35:24.632 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:24.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:24.632 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:35:24.632 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:24.632 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:35:24.632 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:24.632 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:24.632 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:35:24.632 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:24.632 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:35:24.633 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:24.633 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:35:24.633 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:24.633 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:24.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:24.633 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:24.633 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:35:24.633 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:24.633 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:24.633 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:35:24.633 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:24.634 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:24.634 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:24.634 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:35:24.634 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:24.634 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:24.634 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:24.634 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:24.634 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:35:24.634 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:24.634 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:24.634 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:35:24.635 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:24.635 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:24.635 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:24.635 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:35:24.635 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:24.635 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:24.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:24.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:24.635 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:35:24.635 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:24.635 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:24.635 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.820874_229.1", 2024-04-12T03:35:24.636 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:24.636 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:24.636 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:24.636 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.820874Z", 2024-04-12T03:35:24.636 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:24.636 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:24.636 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:24.636 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:35:24.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:24.636 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:35:24.636 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:35:24.636 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:35:24.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:24.637 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:35:24Z" 2024-04-12T03:35:24.637 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:35:24.638 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.820874_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.820874Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:35:24Z" } 2024-04-12T03:35:24.638 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.820874_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.820874Z', 'realm_epoch': 2}}] 2024-04-12T03:35:24.638 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:35:24.638 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.820874_229.1'} 2024-04-12T03:35:24.638 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.820874_229.1 2024-04-12T03:35:29.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:29.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:29.640 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:35:29.700 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:35:29.700 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:35:29.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:29.735+0000 7fed127bf780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:29.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:29.736+0000 7fed127bf780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:29.752 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:35:29.752 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:35:29.752 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:35:29.752 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:35:29.752 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:35:29.753 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:35:29.753 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:29.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:29.754 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:35:29.754 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:29.755 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:35:29.755 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:29.755 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:29.755 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:35:29.755 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:29.756 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:35:29.756 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:29.756 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:35:29.756 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:29.756 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:29.756 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:29.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:29.757 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:35:29.757 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:29.757 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:29.757 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:35:29.757 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:29.757 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:29.757 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:29.757 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:35:29.757 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:29.757 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:29.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:29.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:29.758 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:35:29.758 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:29.758 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:29.758 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:35:29.759 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:29.759 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:29.759 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:29.759 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:35:29.759 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:29.759 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:29.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:29.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:29.760 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:35:29.760 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:29.760 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:29.760 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.783635_225.1", 2024-04-12T03:35:29.760 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:29.760 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:29.760 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:29.760 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.783635Z", 2024-04-12T03:35:29.760 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:29.760 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:29.761 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:29.761 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:35:29.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:29.761 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:35:29.761 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:35:29.761 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:35:29.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:29.761 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:35:29Z" 2024-04-12T03:35:29.761 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:35:29.763 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.783635_225.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.783635Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:35:29Z" } 2024-04-12T03:35:29.763 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.783635_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.783635Z', 'realm_epoch': 2}}] 2024-04-12T03:35:29.763 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:35:29.763 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.783635_225.1'} 2024-04-12T03:35:29.763 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.783635_225.1 2024-04-12T03:35:34.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:34.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:34.765 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:35:34.875 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:35:34.876 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:35:34.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:34.933+0000 7fa526b2c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:34.934+0000 7fa526b2c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:34.949 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:35:34.949 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:35:34.949 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:35:34.950 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:35:34.950 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:35:34.950 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:35:34.950 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:34.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:34.950 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:35:34.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:34.950 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:35:34.950 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:34.950 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:34.951 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:35:34.951 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:34.951 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:35:34.951 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:34.951 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:35:34.951 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:34.951 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:34.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:34.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:34.951 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:35:34.951 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:34.952 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:34.952 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:35:34.952 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:34.952 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:34.952 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:34.952 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:35:34.952 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:34.952 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:34.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:34.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:34.952 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:35:34.953 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:34.953 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:34.953 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:35:34.953 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:34.953 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:34.953 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:34.953 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:35:34.953 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:34.953 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:34.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:34.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:34.954 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:35:34.954 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:34.954 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:34.954 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.783635_225.1", 2024-04-12T03:35:34.954 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:34.954 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:34.954 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:34.954 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.783635Z", 2024-04-12T03:35:34.954 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:34.954 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:34.954 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:34.955 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:35:34.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:34.955 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:35:34.955 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:35:34.955 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:35:34.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:34.955 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:35:34Z" 2024-04-12T03:35:34.955 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:35:34.956 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.783635_225.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.783635Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:35:34Z" } 2024-04-12T03:35:34.956 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.783635_225.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.783635Z', 'realm_epoch': 2}}] 2024-04-12T03:35:34.956 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:35:34.957 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.783635_225.1'} 2024-04-12T03:35:34.957 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.783635_225.1 2024-04-12T03:35:39.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:39.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:39.958 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:35:40.018 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:35:40.019 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:35:40.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:40.052+0000 7f0202c04780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:40.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:40.053+0000 7f0202c04780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:40.069 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:35:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:35:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:35:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:35:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:35:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:35:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:35:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:35:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:35:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:35:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:35:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:35:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:35:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:35:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:35:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:35:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:35:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:35:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.635345_216.1", 2024-04-12T03:35:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.635345Z", 2024-04-12T03:35:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:35:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:35:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:35:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:35:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:35:40Z" 2024-04-12T03:35:40.074 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:35:40.075 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.635345_216.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.635345Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:35:40Z" } 2024-04-12T03:35:40.075 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.635345_216.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.635345Z', 'realm_epoch': 2}}] 2024-04-12T03:35:40.076 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:35:40.076 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.635345_216.1'} 2024-04-12T03:35:40.076 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.635345_216.1 2024-04-12T03:35:45.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:45.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:45.077 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:35:45.138 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:35:45.138 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:35:45.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:45.172+0000 7fe59a032780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:45.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:45.173+0000 7fe59a032780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:45.188 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:35:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:35:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:35:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:35:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:35:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:35:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:35:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:35:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:35:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:35:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:35:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:35:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:35:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:35:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:35:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:35:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:35:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:35:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.705488_221.1", 2024-04-12T03:35:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.705488Z", 2024-04-12T03:35:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:35:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:35:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:35:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:35:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:45.194 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:35:45Z" 2024-04-12T03:35:45.194 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:35:45.195 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.705488_221.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.705488Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:35:45Z" } 2024-04-12T03:35:45.195 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.705488_221.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.705488Z', 'realm_epoch': 2}}] 2024-04-12T03:35:45.195 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:35:45.195 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.705488_221.1'} 2024-04-12T03:35:45.195 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.705488_221.1 2024-04-12T03:35:50.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:50.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:50.196 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:35:50.281 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:35:50.281 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:35:50.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:50.360+0000 7f4e1bcfc780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:50.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:50.363+0000 7f4e1bcfc780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:50.391 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:35:50.391 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:35:50.391 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:35:50.391 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:35:50.391 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:35:50.391 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:35:50.391 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:50.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:50.391 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:35:50.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:50.392 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:35:50.392 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:50.392 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:50.392 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:35:50.392 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:50.392 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:35:50.392 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:50.392 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:35:50.392 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:50.392 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:50.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:50.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:50.393 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:35:50.393 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:50.393 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:50.393 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:35:50.393 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:50.393 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:50.393 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:50.394 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:35:50.394 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:50.394 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:50.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:50.394 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:50.394 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:35:50.394 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:50.394 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:50.394 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:35:50.394 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:50.394 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:50.395 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:50.395 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:35:50.395 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:50.395 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:50.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:50.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:50.395 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:35:50.395 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:50.395 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:50.395 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892848.726205_223.1", 2024-04-12T03:35:50.395 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:50.395 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:08.726205Z", 2024-04-12T03:35:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:35:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:35:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:35:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:35:50.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:50.397 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:35:50Z" 2024-04-12T03:35:50.397 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:35:50.398 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892848.726205_223.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:08.726205Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:35:50Z" } 2024-04-12T03:35:50.398 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892848.726205_223.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:08.726205Z', 'realm_epoch': 2}}] 2024-04-12T03:35:50.398 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:35:50.398 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892848.726205_223.1'} 2024-04-12T03:35:50.398 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712892852.354438_285.1 target=1_1712892848.726205_223.1 2024-04-12T03:35:55.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:55.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:55.400 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:35:55.459 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:35:55.459 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:35:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:55.493+0000 7f540e2b8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:55.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:55.494+0000 7f540e2b8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:55.510 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:35:55.510 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:35:55.510 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:35:55.510 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:35:55.510 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:35:55.511 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:35:55.511 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:55.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:55.511 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:35:55.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:55.511 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:35:55.511 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:55.511 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:55.511 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:35:55.511 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:55.511 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:35:55.511 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:55.512 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:35:55.512 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:55.512 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:55.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:55.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:55.512 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:35:55.512 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:55.512 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:55.512 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:35:55.512 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:55.512 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:55.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:55.513 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:35:55.513 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:55.513 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:55.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:55.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:55.513 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:35:55.513 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:55.513 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:55.513 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:35:55.513 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:55.514 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:55.514 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:55.514 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:35:55.514 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:55.514 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:55.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:55.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:55.514 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:35:55.514 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:55.514 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:55.514 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892852.354438_285.1", 2024-04-12T03:35:55.515 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:55.515 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:55.515 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:55.515 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:34:12.354438Z", 2024-04-12T03:35:55.515 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:55.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:55.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:55.515 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:35:55.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:55.516 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:35:55.516 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:35:55.516 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:35:55.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:55.516 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:35:55Z" 2024-04-12T03:35:55.516 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:35:55.517 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892852.354438_285.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:34:12.354438Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:35:55Z" } 2024-04-12T03:35:55.518 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892852.354438_285.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:34:12.354438Z', 'realm_epoch': 2}}] 2024-04-12T03:35:55.518 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:35:55.518 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892852.354438_285.1'} 2024-04-12T03:35:55.518 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:35:55.518 DEBUG:boto:path=/dkyges-33/ 2024-04-12T03:35:55.518 DEBUG:boto:auth_path=/dkyges-33/ 2024-04-12T03:35:55.518 DEBUG:boto:Method: HEAD 2024-04-12T03:35:55.519 DEBUG:boto:Path: /dkyges-33/ 2024-04-12T03:35:55.519 DEBUG:boto:Data: 2024-04-12T03:35:55.519 DEBUG:boto:Headers: {} 2024-04-12T03:35:55.519 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:35:55.519 DEBUG:boto:Port: 8000 2024-04-12T03:35:55.519 DEBUG:boto:Params: {} 2024-04-12T03:35:55.519 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:35:55.520 DEBUG:boto:Token: None 2024-04-12T03:35:55.520 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:35:55 GMT /dkyges-33/ 2024-04-12T03:35:55.520 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UoPzvJeIMk1PFpdGWSfE9V9J5Ds= 2024-04-12T03:35:55.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:35:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UoPzvJeIMk1PFpdGWSfE9V9J5Ds=', 'Content-Length': '0'} 2024-04-12T03:35:55.553 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000577bafbd94b0259b-006618ac1b-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:35:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:35:55.553 DEBUG:boto:path=/dkyges-33/foo 2024-04-12T03:35:55.553 DEBUG:boto:auth_path=/dkyges-33/foo 2024-04-12T03:35:55.554 DEBUG:boto:Method: PUT 2024-04-12T03:35:55.554 DEBUG:boto:Path: /dkyges-33/foo 2024-04-12T03:35:55.554 DEBUG:boto:Data: 2024-04-12T03:35:55.554 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:35:55.554 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:35:55.554 DEBUG:boto:Port: 8000 2024-04-12T03:35:55.554 DEBUG:boto:Params: {} 2024-04-12T03:35:55.554 DEBUG:boto:Token: None 2024-04-12T03:35:55.554 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Fri, 12 Apr 2024 03:35:55 GMT /dkyges-33/foo 2024-04-12T03:35:55.555 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TpJGVX7N1iWGV3rwvN5yAKLDZUo= 2024-04-12T03:35:55.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:35:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TpJGVX7N1iWGV3rwvN5yAKLDZUo='} 2024-04-12T03:35:55.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f1651f43863e5dc1-006618ac1b-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:35:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:35:55.563 DEBUG:boto:path=/dkyges-33/ 2024-04-12T03:35:55.563 DEBUG:boto:auth_path=/dkyges-33/ 2024-04-12T03:35:55.563 DEBUG:boto:Method: DELETE 2024-04-12T03:35:55.563 DEBUG:boto:Path: /dkyges-33/ 2024-04-12T03:35:55.563 DEBUG:boto:Data: 2024-04-12T03:35:55.563 DEBUG:boto:Headers: {} 2024-04-12T03:35:55.563 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:35:55.563 DEBUG:boto:Port: 8000 2024-04-12T03:35:55.563 DEBUG:boto:Params: {} 2024-04-12T03:35:55.563 DEBUG:boto:Token: None 2024-04-12T03:35:55.564 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:35:55 GMT /dkyges-33/ 2024-04-12T03:35:55.564 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+YIHV26B75QMdKktWOsrXbS1C20= 2024-04-12T03:35:55.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:35:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+YIHV26B75QMdKktWOsrXbS1C20=', 'Content-Length': '0'} 2024-04-12T03:35:55.571 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx000000553f8a48282542a-006618ac1b-110f-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:35:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:35:55.572 DEBUG:boto:path=/dkyges-34/ 2024-04-12T03:35:55.572 DEBUG:boto:auth_path=/dkyges-34/ 2024-04-12T03:35:55.572 DEBUG:boto:Method: HEAD 2024-04-12T03:35:55.572 DEBUG:boto:Path: /dkyges-34/ 2024-04-12T03:35:55.572 DEBUG:boto:Data: 2024-04-12T03:35:55.573 DEBUG:boto:Headers: {} 2024-04-12T03:35:55.573 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:35:55.573 DEBUG:boto:Port: 8001 2024-04-12T03:35:55.573 DEBUG:boto:Params: {} 2024-04-12T03:35:55.573 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:35:55.573 DEBUG:boto:Token: None 2024-04-12T03:35:55.573 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:35:55 GMT /dkyges-34/ 2024-04-12T03:35:55.573 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:reu/2NCPPVbINaehq4dQSPPHmoY= 2024-04-12T03:35:55.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:35:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:reu/2NCPPVbINaehq4dQSPPHmoY=', 'Content-Length': '0'} 2024-04-12T03:35:55.602 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa55b8533a1dd58d-006618ac1b-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:35:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:35:55.603 DEBUG:boto:path=/dkyges-34/foo 2024-04-12T03:35:55.603 DEBUG:boto:auth_path=/dkyges-34/foo 2024-04-12T03:35:55.603 DEBUG:boto:Method: PUT 2024-04-12T03:35:55.603 DEBUG:boto:Path: /dkyges-34/foo 2024-04-12T03:35:55.603 DEBUG:boto:Data: 2024-04-12T03:35:55.603 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:35:55.603 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:35:55.604 DEBUG:boto:Port: 8001 2024-04-12T03:35:55.604 DEBUG:boto:Params: {} 2024-04-12T03:35:55.604 DEBUG:boto:Token: None 2024-04-12T03:35:55.604 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Fri, 12 Apr 2024 03:35:55 GMT /dkyges-34/foo 2024-04-12T03:35:55.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KqoVJomkB90k/Nla+Kd+nW9MD+U= 2024-04-12T03:35:55.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:35:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KqoVJomkB90k/Nla+Kd+nW9MD+U='} 2024-04-12T03:35:55.612 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000062302c8888b865ea-006618ac1b-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:35:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:35:55.612 DEBUG:boto:path=/dkyges-34/ 2024-04-12T03:35:55.612 DEBUG:boto:auth_path=/dkyges-34/ 2024-04-12T03:35:55.612 DEBUG:boto:Method: DELETE 2024-04-12T03:35:55.613 DEBUG:boto:Path: /dkyges-34/ 2024-04-12T03:35:55.613 DEBUG:boto:Data: 2024-04-12T03:35:55.613 DEBUG:boto:Headers: {} 2024-04-12T03:35:55.613 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:35:55.613 DEBUG:boto:Port: 8001 2024-04-12T03:35:55.613 DEBUG:boto:Params: {} 2024-04-12T03:35:55.613 DEBUG:boto:Token: None 2024-04-12T03:35:55.613 DEBUG:boto:StringToSign: DELETE Fri, 12 Apr 2024 03:35:55 GMT /dkyges-34/ 2024-04-12T03:35:55.613 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TFMDmNFrBFSIa7CEI8++sp+dirY= 2024-04-12T03:35:55.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:35:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TFMDmNFrBFSIa7CEI8++sp+dirY=', 'Content-Length': '0'} 2024-04-12T03:35:55.622 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx00000bb64f8105622f3ad-006618ac1b-1113-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:35:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:35:55.622 DEBUG:boto:path=/dkyges-33/ 2024-04-12T03:35:55.622 DEBUG:boto:auth_path=/dkyges-33/ 2024-04-12T03:35:55.623 DEBUG:boto:Method: HEAD 2024-04-12T03:35:55.623 DEBUG:boto:Path: /dkyges-33/ 2024-04-12T03:35:55.623 DEBUG:boto:Data: 2024-04-12T03:35:55.623 DEBUG:boto:Headers: {} 2024-04-12T03:35:55.623 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:35:55.623 DEBUG:boto:Port: 8000 2024-04-12T03:35:55.623 DEBUG:boto:Params: {} 2024-04-12T03:35:55.623 DEBUG:boto:Token: None 2024-04-12T03:35:55.623 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:35:55 GMT /dkyges-33/ 2024-04-12T03:35:55.623 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UoPzvJeIMk1PFpdGWSfE9V9J5Ds= 2024-04-12T03:35:55.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:35:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UoPzvJeIMk1PFpdGWSfE9V9J5Ds=', 'Content-Length': '0'} 2024-04-12T03:35:55.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000897c48c7d72d78bb-006618ac1b-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:35:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:35:55.626 DEBUG:boto:path=/dkyges-34/ 2024-04-12T03:35:55.626 DEBUG:boto:auth_path=/dkyges-34/ 2024-04-12T03:35:55.626 DEBUG:boto:Method: HEAD 2024-04-12T03:35:55.626 DEBUG:boto:Path: /dkyges-34/ 2024-04-12T03:35:55.626 DEBUG:boto:Data: 2024-04-12T03:35:55.626 DEBUG:boto:Headers: {} 2024-04-12T03:35:55.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:35:55.627 DEBUG:boto:Port: 8000 2024-04-12T03:35:55.627 DEBUG:boto:Params: {} 2024-04-12T03:35:55.627 DEBUG:boto:Token: None 2024-04-12T03:35:55.627 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:35:55 GMT /dkyges-34/ 2024-04-12T03:35:55.627 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:reu/2NCPPVbINaehq4dQSPPHmoY= 2024-04-12T03:35:55.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:35:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:reu/2NCPPVbINaehq4dQSPPHmoY=', 'Content-Length': '0'} 2024-04-12T03:35:55.629 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d91553ada9ffa975-006618ac1b-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:35:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:35:55.630 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2024-04-12T03:35:55.631 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... SKIP: test_multi_period_incremental_sync skipped. Requires 3 or more zones in master zonegroup. 2024-04-12T03:35:55.631 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-35 2024-04-12T03:35:55.631 DEBUG:boto:path=/dkyges-35/ 2024-04-12T03:35:55.632 DEBUG:boto:auth_path=/dkyges-35/ 2024-04-12T03:35:55.632 DEBUG:boto:Method: PUT 2024-04-12T03:35:55.632 DEBUG:boto:Path: /dkyges-35/ 2024-04-12T03:35:55.632 DEBUG:boto:Data: 2024-04-12T03:35:55.632 DEBUG:boto:Headers: {} 2024-04-12T03:35:55.632 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:35:55.632 DEBUG:boto:Port: 8000 2024-04-12T03:35:55.632 DEBUG:boto:Params: {} 2024-04-12T03:35:55.632 DEBUG:boto:Token: None 2024-04-12T03:35:55.632 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:35:55 GMT /dkyges-35/ 2024-04-12T03:35:55.633 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:irxVdx7VLwX9RxpLCSeUO0r/7qY= 2024-04-12T03:35:55.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:35:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:irxVdx7VLwX9RxpLCSeUO0r/7qY=', 'Content-Length': '0'} 2024-04-12T03:35:55.651 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b6a0edb5a42b5f7b-006618ac1b-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:35:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:35:55.651 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-36 2024-04-12T03:35:55.652 DEBUG:boto:path=/dkyges-36/ 2024-04-12T03:35:55.652 DEBUG:boto:auth_path=/dkyges-36/ 2024-04-12T03:35:55.652 DEBUG:boto:Method: PUT 2024-04-12T03:35:55.652 DEBUG:boto:Path: /dkyges-36/ 2024-04-12T03:35:55.652 DEBUG:boto:Data: 2024-04-12T03:35:55.652 DEBUG:boto:Headers: {} 2024-04-12T03:35:55.652 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:35:55.652 DEBUG:boto:Port: 8001 2024-04-12T03:35:55.652 DEBUG:boto:Params: {} 2024-04-12T03:35:55.652 DEBUG:boto:Token: None 2024-04-12T03:35:55.653 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:35:55 GMT /dkyges-36/ 2024-04-12T03:35:55.653 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3Lqhb4DN0byUJ0HLztWMTRYjqyY= 2024-04-12T03:35:55.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:35:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3Lqhb4DN0byUJ0HLztWMTRYjqyY=', 'Content-Length': '0'} 2024-04-12T03:35:55.685 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f9ff769e7d8f37e2-006618ac1b-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:35:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:35:55.685 DEBUG:boto:path=/dkyges-35/ 2024-04-12T03:35:55.686 DEBUG:boto:auth_path=/dkyges-35/ 2024-04-12T03:35:55.686 DEBUG:boto:Method: HEAD 2024-04-12T03:35:55.686 DEBUG:boto:Path: /dkyges-35/ 2024-04-12T03:35:55.686 DEBUG:boto:Data: 2024-04-12T03:35:55.686 DEBUG:boto:Headers: {} 2024-04-12T03:35:55.686 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:35:55.686 DEBUG:boto:Port: 8000 2024-04-12T03:35:55.686 DEBUG:boto:Params: {} 2024-04-12T03:35:55.686 DEBUG:boto:Token: None 2024-04-12T03:35:55.687 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:35:55 GMT /dkyges-35/ 2024-04-12T03:35:55.687 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RVnd3reZrMq0ktgz50thQ/KVjOc= 2024-04-12T03:35:55.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:35:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RVnd3reZrMq0ktgz50thQ/KVjOc=', 'Content-Length': '0'} 2024-04-12T03:35:55.689 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f37d3cabf843599c-006618ac1b-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:35:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:35:55.689 DEBUG:boto:path=/dkyges-35/key 2024-04-12T03:35:55.689 DEBUG:boto:auth_path=/dkyges-35/key 2024-04-12T03:35:55.690 DEBUG:boto:Method: PUT 2024-04-12T03:35:55.690 DEBUG:boto:Path: /dkyges-35/key 2024-04-12T03:35:55.690 DEBUG:boto:Data: 2024-04-12T03:35:55.690 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-12T03:35:55.690 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:35:55.690 DEBUG:boto:Port: 8000 2024-04-12T03:35:55.690 DEBUG:boto:Params: {} 2024-04-12T03:35:55.690 DEBUG:boto:Token: None 2024-04-12T03:35:55.690 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 12 Apr 2024 03:35:55 GMT /dkyges-35/key 2024-04-12T03:35:55.690 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HmmA92HMOeA7FGXWm9QnlATrMps= 2024-04-12T03:35:55.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:35:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HmmA92HMOeA7FGXWm9QnlATrMps='} 2024-04-12T03:35:55.697 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0de8204bcdd8b2e-006618ac1b-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:35:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:35:55.697 DEBUG:boto:path=/dkyges-36/ 2024-04-12T03:35:55.697 DEBUG:boto:auth_path=/dkyges-36/ 2024-04-12T03:35:55.697 DEBUG:boto:Method: HEAD 2024-04-12T03:35:55.698 DEBUG:boto:Path: /dkyges-36/ 2024-04-12T03:35:55.698 DEBUG:boto:Data: 2024-04-12T03:35:55.698 DEBUG:boto:Headers: {} 2024-04-12T03:35:55.698 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:35:55.698 DEBUG:boto:Port: 8001 2024-04-12T03:35:55.698 DEBUG:boto:Params: {} 2024-04-12T03:35:55.698 DEBUG:boto:Token: None 2024-04-12T03:35:55.698 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:35:55 GMT /dkyges-36/ 2024-04-12T03:35:55.698 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZZVWpyyjofcf6fXGWUg4uCabAlc= 2024-04-12T03:35:55.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:35:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZZVWpyyjofcf6fXGWUg4uCabAlc=', 'Content-Length': '0'} 2024-04-12T03:35:55.701 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054af4ca6aee6fabc-006618ac1b-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:35:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:35:55.701 DEBUG:boto:path=/dkyges-36/key 2024-04-12T03:35:55.701 DEBUG:boto:auth_path=/dkyges-36/key 2024-04-12T03:35:55.701 DEBUG:boto:Method: PUT 2024-04-12T03:35:55.701 DEBUG:boto:Path: /dkyges-36/key 2024-04-12T03:35:55.701 DEBUG:boto:Data: 2024-04-12T03:35:55.701 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2024-04-12T03:35:55.702 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:35:55.702 DEBUG:boto:Port: 8001 2024-04-12T03:35:55.702 DEBUG:boto:Params: {} 2024-04-12T03:35:55.702 DEBUG:boto:Token: None 2024-04-12T03:35:55.702 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 12 Apr 2024 03:35:55 GMT /dkyges-36/key 2024-04-12T03:35:55.702 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YFH1BEZ1DPGylyTcpVIKYhNUWps= 2024-04-12T03:35:55.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:35:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YFH1BEZ1DPGylyTcpVIKYhNUWps='} 2024-04-12T03:35:55.746 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003e6266d66312827-006618ac1b-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:35:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:35:55.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:55.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:55.747 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:35:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:35:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:35:55.841 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:35:55.855 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:35:55.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:55.855 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:35:55.855 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:35:55.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:55.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:55.856 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:35:55.856 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:35:55.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:55.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:55.856 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:35:55.856 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:35:55.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:55.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:55.856 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892955.671477_293.1", 2024-04-12T03:35:55.857 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:35:55.671477Z" 2024-04-12T03:35:55.857 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:55.857 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:35:55.858 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892955.671477_293.1'} 2024-04-12T03:35:55.858 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:35:55.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:55.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:55.858 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:35:55.915 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:35:55.916 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:35:55.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:55.949+0000 7f30e97ad780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:55.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:35:55.950+0000 7f30e97ad780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:35:55.965 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:35:55.965 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:35:55.965 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:35:55.965 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:35:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:35:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:35:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:35:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:35:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:35:55.967 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:55.967 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:35:55.967 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:55.967 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:35:55.967 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:55.967 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:55.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:55.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:55.967 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:35:55.967 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:55.967 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:55.967 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:35:55.968 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:55.968 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:55.968 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:55.968 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:35:55.968 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:55.968 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:55.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:55.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:55.968 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:35:55.968 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:55.968 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:55.969 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:35:55.969 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:55.969 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:55.969 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:55.969 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:35:55.969 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:55.969 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:55.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:55.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:55.969 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:35:55.969 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:55.970 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:35:55.970 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892955.671477_293.1", 2024-04-12T03:35:55.970 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:55.970 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:55.970 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:55.970 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:35:55.671477Z", 2024-04-12T03:35:55.970 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:35:55.970 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:55.970 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:55.970 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:35:55.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:55.971 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:35:55.971 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:35:55.971 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:35:55.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:55.971 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:35:55Z" 2024-04-12T03:35:55.971 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:35:55.972 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892955.671477_293.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:35:55.671477Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:35:55Z" } 2024-04-12T03:35:55.972 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892955.671477_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:35:55.671477Z', 'realm_epoch': 2}}] 2024-04-12T03:35:55.972 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892955.671477_293.1'} 2024-04-12T03:35:55.972 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892955.671477_293.1'} 2024-04-12T03:35:55.972 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:35:55.973 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-12T03:35:55.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:55.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:55.973 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:35:56.027 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:35:56.027 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:35:56.083 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:35:56.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-12T03:35:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:35:55.561872Z" 2024-04-12T03:35:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:35:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:20:22.705876Z" 2024-04-12T03:35:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006003", 2024-04-12T03:35:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:35:55.696807Z" 2024-04-12T03:35:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:56.085 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004597", 2024-04-12T03:35:56.085 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:26:58.723769Z" 2024-04-12T03:35:56.085 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:56.085 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:35:56.086 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006003', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:35:56.086 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:35:56.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:56.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:35:56.086 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:35:56.144 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:35:56.144 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:35:56.195 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:35:56.195 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:35:56.196 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:35:56.196 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:35:56.196 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:35:56.196 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 16340188497730315678 2024-04-12T03:35:56.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:56.196 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:35:56.196 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:56.196 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:35:56.196 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:56.196 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:35:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-12T03:35:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.704353Z" 2024-04-12T03:35:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:35:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:35:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:35:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.705876Z" 2024-04-12T03:35:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:35:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:35:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000005862", 2024-04-12T03:35:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:33:56.741039Z" 2024-04-12T03:35:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:35:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:35:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:35:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:35:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004597", 2024-04-12T03:35:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:35:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:35:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:35:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:26:58.723769Z" 2024-04-12T03:35:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:35:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:35:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:35:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:35:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:35:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:35:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:35:56Z" 2024-04-12T03:35:56.201 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:35:56.202 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16340188497730315678 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002622", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.704353Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.705876Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005862", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:33:56.741039Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004597", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:26:58.723769Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:35:56Z" } 2024-04-12T03:35:56.202 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002622', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.704353Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.705876Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005862', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:33:56.741039Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004597', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:26:58.723769Z'}}] 2024-04-12T03:35:56.202 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006003', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:35:56.202 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002622', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005862', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:35:56.203 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000002764 target=00000000000000000000:00000000000000002622, shard=2 master=00000000000000000000:00000000000000006003 target=00000000000000000000:00000000000000005862 2024-04-12T03:36:01.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:01.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:01.204 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:01.267 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:01.267 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:02.324 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:36:02.324 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:36:02.324 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:36:02.324 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:36:02.325 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:36:02.325 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 16340188497730315678 2024-04-12T03:36:02.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:02.325 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:36:02.325 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:02.325 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:36:02.325 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:02.325 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:02.325 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002622", 2024-04-12T03:36:02.325 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:02.325 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:02.326 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:02.326 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.704353Z" 2024-04-12T03:36:02.326 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:02.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:02.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:02.326 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:36:02.326 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:02.326 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:02.326 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:36:02.326 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:02.326 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:02.327 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:02.327 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.705876Z" 2024-04-12T03:36:02.327 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:02.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:02.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:02.327 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:36:02.327 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:02.327 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:02.327 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000005862", 2024-04-12T03:36:02.327 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:02.328 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:02.328 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:02.328 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:33:56.741039Z" 2024-04-12T03:36:02.328 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:02.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:02.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:02.328 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:36:02.328 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:02.328 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:02.328 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004597", 2024-04-12T03:36:02.328 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:02.328 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:02.329 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:02.329 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:26:58.723769Z" 2024-04-12T03:36:02.329 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:02.329 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:02.329 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:36:02.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:02.329 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:36:02.329 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:36:02.329 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:36:02.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:02.329 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:36:02Z" 2024-04-12T03:36:02.329 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:36:02.330 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16340188497730315678 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002622", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.704353Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.705876Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005862", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:33:56.741039Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004597", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:26:58.723769Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:36:02Z" } 2024-04-12T03:36:02.331 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002622', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.704353Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.705876Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005862', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:33:56.741039Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004597', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:26:58.723769Z'}}] 2024-04-12T03:36:02.331 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006003', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:36:02.331 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002622', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000005862', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:36:02.331 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000002764 target=00000000000000000000:00000000000000002622, shard=2 master=00000000000000000000:00000000000000006003 target=00000000000000000000:00000000000000005862 2024-04-12T03:36:07.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:07.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:07.333 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:07.396 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:07.396 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:07.447 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:36:07.448 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:36:07.448 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:36:07.448 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:36:07.448 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:36:07.448 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 16340188497730315678 2024-04-12T03:36:07.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:07.448 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:36:07.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:07.448 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:36:07.448 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:07.449 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:07.449 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002906", 2024-04-12T03:36:07.449 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:07.449 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:07.449 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:07.449 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:36:03.087572Z" 2024-04-12T03:36:07.449 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:07.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:07.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:07.449 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:36:07.449 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:07.449 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:07.450 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:36:07.450 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:07.450 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:07.450 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:07.450 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.705876Z" 2024-04-12T03:36:07.450 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:07.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:07.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:07.450 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:36:07.450 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:07.450 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006003", 2024-04-12T03:36:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:35:55.696807Z" 2024-04-12T03:36:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:36:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004597", 2024-04-12T03:36:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:26:58.723769Z" 2024-04-12T03:36:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:36:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:36:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:36:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:36:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:36:07Z" 2024-04-12T03:36:07.453 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:36:07.454 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16340188497730315678 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002906", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:36:03.087572Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.705876Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006003", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:35:55.696807Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004597", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:26:58.723769Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:36:07Z" } 2024-04-12T03:36:07.454 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002906', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:36:03.087572Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.705876Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006003', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:35:55.696807Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004597', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:26:58.723769Z'}}] 2024-04-12T03:36:07.455 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006003', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:36:07.455 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002906', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006003', 3: '00000000000000000000:00000000000000004597'} 2024-04-12T03:36:07.455 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:36:07.455 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-12T03:36:07.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:07.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:07.455 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:07.509 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:07.509 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:07.562 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:36:07.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:07.562 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002906", 2024-04-12T03:36:07.562 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:36:04.449581Z" 2024-04-12T03:36:07.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:07.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:07.563 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:36:07.563 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:20:22.851157Z" 2024-04-12T03:36:07.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:07.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:07.563 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006708", 2024-04-12T03:36:07.563 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:36:06.614078Z" 2024-04-12T03:36:07.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:07.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:07.564 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004174", 2024-04-12T03:36:07.564 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:26:36.864390Z" 2024-04-12T03:36:07.564 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:07.564 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:36:07.565 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000002906', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006708', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:36:07.565 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:36:07.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:07.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:07.565 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:07.674 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:36:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:36:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:36:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:36:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:36:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5944041172307441597 2024-04-12T03:36:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:36:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:36:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:07.675 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-12T03:36:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:35:55.611652Z" 2024-04-12T03:36:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:36:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:07.676 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:36:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.851157Z" 2024-04-12T03:36:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:36:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:07.677 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006567", 2024-04-12T03:36:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:35:55.708539Z" 2024-04-12T03:36:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:36:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:07.678 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004174", 2024-04-12T03:36:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:26:36.864390Z" 2024-04-12T03:36:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:36:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:36:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:36:07.679 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:36:07.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:07.680 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:36:07Z" 2024-04-12T03:36:07.680 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:36:07.681 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5944041172307441597 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:35:55.611652Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.851157Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006567", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:35:55.708539Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004174", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:26:36.864390Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:36:07Z" } 2024-04-12T03:36:07.681 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:35:55.611652Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.851157Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006567', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:35:55.708539Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004174', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:26:36.864390Z'}}] 2024-04-12T03:36:07.681 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002906', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006708', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:36:07.681 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006567', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:36:07.681 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000002906 target=00000000000000000000:00000000000000002764, shard=2 master=00000000000000000000:00000000000000006708 target=00000000000000000000:00000000000000006567 2024-04-12T03:36:12.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:12.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:12.683 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:12.744 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:12.744 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:12.794 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:36:12.795 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:36:12.795 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:36:12.795 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:36:12.795 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:36:12.795 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5944041172307441597 2024-04-12T03:36:12.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:12.796 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:36:12.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:12.796 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:36:12.796 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:12.796 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:12.796 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-12T03:36:12.796 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:12.796 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:12.796 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:12.797 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:35:55.611652Z" 2024-04-12T03:36:12.797 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:12.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:12.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:12.797 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:36:12.797 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:12.797 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:12.797 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:36:12.798 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:12.798 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:12.798 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:12.798 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.851157Z" 2024-04-12T03:36:12.798 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:12.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:12.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:12.799 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:36:12.799 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:12.799 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:12.799 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006567", 2024-04-12T03:36:12.799 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:12.799 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:12.799 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:12.799 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:35:55.708539Z" 2024-04-12T03:36:12.799 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:12.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:12.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:12.800 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:36:12.800 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:12.800 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:12.800 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004174", 2024-04-12T03:36:12.800 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:12.801 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:12.801 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:12.801 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:26:36.864390Z" 2024-04-12T03:36:12.801 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:12.801 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:12.801 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:36:12.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:12.801 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:36:12.801 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:36:12.802 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:36:12.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:12.802 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:36:12Z" 2024-04-12T03:36:12.802 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:36:12.803 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5944041172307441597 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:35:55.611652Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.851157Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006567", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:35:55.708539Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004174", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:26:36.864390Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:36:12Z" } 2024-04-12T03:36:12.803 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:35:55.611652Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.851157Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006567', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:35:55.708539Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004174', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:26:36.864390Z'}}] 2024-04-12T03:36:12.804 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002906', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006708', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:36:12.804 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006567', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:36:12.804 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000002906 target=00000000000000000000:00000000000000002764, shard=2 master=00000000000000000000:00000000000000006708 target=00000000000000000000:00000000000000006567 2024-04-12T03:36:17.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:17.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:17.805 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:17.872 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:17.872 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:17.931 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:36:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:36:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:36:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:36:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:36:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5944041172307441597 2024-04-12T03:36:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:17.932 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:36:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:36:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-12T03:36:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:35:55.611652Z" 2024-04-12T03:36:17.933 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:36:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:36:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:17.934 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.851157Z" 2024-04-12T03:36:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:36:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006567", 2024-04-12T03:36:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:17.935 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:35:55.708539Z" 2024-04-12T03:36:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:36:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004174", 2024-04-12T03:36:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:17.936 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:26:36.864390Z" 2024-04-12T03:36:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:36:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:36:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:36:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:36:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:17.937 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:36:17Z" 2024-04-12T03:36:17.938 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:36:17.939 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5944041172307441597 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:35:55.611652Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.851157Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006567", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:35:55.708539Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004174", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:26:36.864390Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:36:17Z" } 2024-04-12T03:36:17.939 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:35:55.611652Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.851157Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006567', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:35:55.708539Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004174', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:26:36.864390Z'}}] 2024-04-12T03:36:17.939 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002906', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006708', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:36:17.939 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006567', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:36:17.939 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000002906 target=00000000000000000000:00000000000000002764, shard=2 master=00000000000000000000:00000000000000006708 target=00000000000000000000:00000000000000006567 2024-04-12T03:36:22.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:22.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:22.941 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:23.002 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:23.002 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:23.055 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:36:23.056 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:36:23.056 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:36:23.056 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:36:23.056 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:36:23.056 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5944041172307441597 2024-04-12T03:36:23.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:23.056 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:36:23.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:23.056 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:36:23.056 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:23.056 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:23.056 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002764", 2024-04-12T03:36:23.057 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:23.057 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:23.057 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:23.057 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:35:55.611652Z" 2024-04-12T03:36:23.057 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:23.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:23.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:23.057 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:36:23.057 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:23.057 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:23.058 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:36:23.058 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:23.058 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:23.058 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:23.058 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.851157Z" 2024-04-12T03:36:23.058 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:23.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:23.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:23.058 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:36:23.058 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:23.059 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:23.059 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006567", 2024-04-12T03:36:23.059 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:23.059 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:23.059 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:23.059 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:35:55.708539Z" 2024-04-12T03:36:23.059 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:23.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:23.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:23.059 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:36:23.059 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:23.059 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:23.060 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004174", 2024-04-12T03:36:23.060 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:23.060 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:23.060 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:23.060 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:26:36.864390Z" 2024-04-12T03:36:23.060 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:23.060 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:23.060 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:36:23.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:23.060 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:36:23.060 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:36:23.061 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:36:23.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:23.061 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:36:23Z" 2024-04-12T03:36:23.061 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:36:23.062 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5944041172307441597 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002764", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:35:55.611652Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.851157Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006567", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:35:55.708539Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004174", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:26:36.864390Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:36:23Z" } 2024-04-12T03:36:23.062 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002764', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:35:55.611652Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.851157Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006567', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:35:55.708539Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004174', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:26:36.864390Z'}}] 2024-04-12T03:36:23.062 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002906', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006708', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:36:23.062 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002764', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006567', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:36:23.062 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000002906 target=00000000000000000000:00000000000000002764, shard=2 master=00000000000000000000:00000000000000006708 target=00000000000000000000:00000000000000006567 2024-04-12T03:36:28.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:28.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:28.064 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:28.127 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:28.127 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:28.180 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:36:28.181 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:36:28.181 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:36:28.181 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:36:28.181 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:36:28.181 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5944041172307441597 2024-04-12T03:36:28.181 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:28.181 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:36:28.181 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:28.181 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:36:28.182 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:28.182 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:28.182 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002906", 2024-04-12T03:36:28.182 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:28.182 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:28.182 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:28.182 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:36:04.449581Z" 2024-04-12T03:36:28.182 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:28.182 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:28.182 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:28.183 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:36:28.183 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:28.183 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:28.183 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:36:28.183 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:28.183 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:28.183 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:28.183 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.851157Z" 2024-04-12T03:36:28.183 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:28.183 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:28.184 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:28.184 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:36:28.184 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:28.184 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:28.184 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006708", 2024-04-12T03:36:28.184 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:28.184 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:28.184 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:28.185 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:36:06.614078Z" 2024-04-12T03:36:28.185 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:28.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:28.185 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:28.185 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:36:28.185 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:28.185 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:36:28.185 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004174", 2024-04-12T03:36:28.185 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:36:28.186 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:36:28.186 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:36:28.186 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:26:36.864390Z" 2024-04-12T03:36:28.186 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:28.186 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:28.186 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:36:28.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:28.186 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:36:28.186 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:36:28.186 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:36:28.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:28.187 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:36:28Z" 2024-04-12T03:36:28.187 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:36:28.188 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5944041172307441597 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002906", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:36:04.449581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.851157Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006708", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:36:06.614078Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004174", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:26:36.864390Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:36:28Z" } 2024-04-12T03:36:28.188 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002906', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:36:04.449581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.851157Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006708', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:36:06.614078Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004174', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:26:36.864390Z'}}] 2024-04-12T03:36:28.189 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002906', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006708', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:36:28.189 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002906', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006708', 3: '00000000000000000000:00000000000000004174'} 2024-04-12T03:36:28.189 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:36:28.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:28.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:28.189 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:28.243 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:28.243 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:28.297 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:36:28.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:28.297 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002906", 2024-04-12T03:36:28.297 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:36:03.087572Z" 2024-04-12T03:36:28.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:28.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:28.298 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:36:28.298 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:20:22.705876Z" 2024-04-12T03:36:28.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:28.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:28.298 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006144", 2024-04-12T03:36:28.298 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:36:06.623407Z" 2024-04-12T03:36:28.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:28.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:28.299 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004597", 2024-04-12T03:36:28.299 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:26:58.723769Z" 2024-04-12T03:36:28.299 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:28.299 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:36:28.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:36:28.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:36:28.301 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:36:28.353 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:28.353 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:28.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:28.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:28.412 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:28.466 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:28.467 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:28.514 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-04-12T03:36:28.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:28.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:28.518 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:28.574 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:28.574 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:28.621 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-04-12T03:36:28.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:28.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:28.623 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:28.677 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:28.677 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:28.725 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-04-12T03:36:28.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:28.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:28.727 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:28.827 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-04-12T03:36:28.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:28.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:28.829 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:28.879 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:28.879 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:28.933 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:36:28.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:28.933 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002906", 2024-04-12T03:36:28.933 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:36:03.087572Z" 2024-04-12T03:36:28.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:28.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:28.934 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:36:28.934 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:20:22.705876Z" 2024-04-12T03:36:28.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:28.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:28.934 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006144", 2024-04-12T03:36:28.934 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:36:06.623407Z" 2024-04-12T03:36:28.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:28.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:28.934 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004597", 2024-04-12T03:36:28.934 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:26:58.723769Z" 2024-04-12T03:36:28.934 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:28.935 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:36:28.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:36:28.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:36:28.936 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:36:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:29.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:29.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:29.046 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:29.102 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:29.102 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:29.154 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-04-12T03:36:29.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:29.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:29.156 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:29.211 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:29.211 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:29.258 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-04-12T03:36:29.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:29.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:29.260 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:29.315 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:29.315 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:29.363 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-04-12T03:36:29.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:29.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '3', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:36:29.366 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 3 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:36:29.420 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:29.420 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:29.467 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-04-12T03:36:29.469 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... ok 2024-04-12T03:36:29.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:36:29.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:36:29.470 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:36:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:29.547 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:36:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:36:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T03:36:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-12T03:36:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "control_pool": "a2.rgw.control", 2024-04-12T03:36:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-12T03:36:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-12T03:36:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "log_pool": "a2.rgw.log", 2024-04-12T03:36:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-12T03:36:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-12T03:36:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-12T03:36:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-12T03:36:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-12T03:36:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-12T03:36:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-12T03:36:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-12T03:36:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "otp_pool": "a2.rgw.otp", 2024-04-12T03:36:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "system_key": { 2024-04-12T03:36:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "access_key": "6ESEI28JZW8UW9BY", 2024-04-12T03:36:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "secret_key": "oVIj7wCADBnecpdBF65ZyUvXEbg14PDu" 2024-04-12T03:36:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_pools": [ 2024-04-12T03:36:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "default-placement", 2024-04-12T03:36:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:36:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-12T03:36:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": { 2024-04-12T03:36:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD": { 2024-04-12T03:36:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-12T03:36:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "compression_type": "random" 2024-04-12T03:36:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-12T03:36:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "index_type": 0, 2024-04-12T03:36:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "inline_data": true 2024-04-12T03:36:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:36:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:36:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-12T03:36:29.551 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:36:29.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:36:29.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:36:29.552 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:36:29.605 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:36:29.605 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:36:29.652 INFO:teuthology.orchestra.run.smithi043.stderr:Sending period to new master zone 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61 2024-04-12T03:36:29.681 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:36:29.681 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:36:29.681 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5, 2024-04-12T03:36:29.681 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T03:36:29.681 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T03:36:29.682 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T03:36:29.682 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:36:29.682 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T03:36:29.682 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:29.682 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T03:36:29.682 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T03:36:29.682 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:36:29.682 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T03:36:29.682 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:36:29.682 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:36:29.683 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:36:29.683 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:36:29.683 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:36:29.683 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:36:29.683 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:36:29.683 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:29.683 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:36:29.683 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T03:36:29.683 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:36:29.683 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T03:36:29.683 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:36:29.684 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:36:29.684 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:36:29.684 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:36:29.684 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:36:29.684 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:36:29.684 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": false, 2024-04-12T03:36:29.684 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:36:29.684 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:36:29.684 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:36:29.684 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:36:29.684 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:36:29.684 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:36:29.685 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:29.685 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:29.685 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:36:29.685 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T03:36:29.685 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:36:29.685 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:36:29.685 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:36:29.685 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:36:29.685 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:36:29.685 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:36:29.685 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:36:29.685 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:36:29.686 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:36:29.686 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:36:29.686 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:36:29.686 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:36:29.686 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:36:29.686 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:36:29.686 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:36:29.686 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:29.686 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:36:29.686 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:36:29.686 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:29.687 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:36:29.687 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:36:29.687 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:36:29.687 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:36:29.687 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:36:29.687 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:29.687 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:36:29.687 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:36:29.687 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:36:29.687 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:36:29.687 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:36:29.687 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:29.688 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:36:29.688 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:36:29.688 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:36:29.688 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:29.688 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:29.688 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T03:36:29.688 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T03:36:29.688 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:36:29.688 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T03:36:29.688 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:36:29.688 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:36:29.689 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:36:29.689 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:36:29.689 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:36:29.689 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:36:29.689 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:36:29.689 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:29.689 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:36:29.689 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T03:36:29.689 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:36:29.689 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:36:29.689 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:36:29.690 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:36:29.690 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:36:29.690 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:36:29.690 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:36:29.690 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:36:29.690 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:36:29.690 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:36:29.690 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:36:29.690 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:36:29.690 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:36:29.695 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:36:29.695 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:36:29.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:29.695 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:29.695 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:36:29.695 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T03:36:29.695 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:36:29.695 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T03:36:29.695 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:36:29.696 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:36:29.696 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:36:29.696 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:36:29.696 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:36:29.696 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:36:29.696 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:36:29.696 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:36:29.696 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:36:29.696 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:36:29.696 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:36:29.696 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:36:29.696 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:36:29.697 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:29.697 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:36:29.697 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:36:29.697 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:29.697 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:36:29.697 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:36:29.697 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:36:29.697 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:36:29.697 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:36:29.697 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:29.697 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:36:29.698 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:36:29.698 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:36:29.698 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:36:29.698 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:36:29.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:29.698 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:36:29.698 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:36:29.698 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:36:29.698 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:29.698 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:36:29.698 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T03:36:29.699 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:29.699 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:36:29.699 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T03:36:29.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:29.699 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:29.699 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:36:29.699 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T03:36:29.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:29.699 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:29.699 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:36:29.699 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T03:36:29.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:29.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:36:29.700 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:36:29.700 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T03:36:29.700 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:29.700 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:36:29.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:29.700 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T03:36:29.700 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:36:29.700 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T03:36:29.700 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T03:36:29.700 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:36:29.701 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:36:29.701 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:36:29.701 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:36:29.701 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:36:29.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:29.701 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T03:36:29.701 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:36:29.701 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:36:29.701 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:36:29.701 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:36:29.701 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:36:29.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:29.702 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T03:36:29.702 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:36:29.702 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:36:29.702 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:36:29.702 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:36:29.702 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:36:29.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:29.702 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T03:36:29.702 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:36:29.702 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:36:29.703 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:36:29.703 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:36:29.703 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:36:29.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:29.703 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T03:36:29.703 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:36:29.703 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:36:29.703 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:36:29.703 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:36:29.703 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:36:29.703 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:36:29.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:36:29.704 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:36:29.704 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T03:36:29.704 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:36:29.704 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:36:29.705 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to false 2024-04-12T03:37:29.705 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-37 2024-04-12T03:37:29.706 DEBUG:boto:path=/dkyges-37/ 2024-04-12T03:37:29.706 DEBUG:boto:auth_path=/dkyges-37/ 2024-04-12T03:37:29.707 DEBUG:boto:Method: PUT 2024-04-12T03:37:29.707 DEBUG:boto:Path: /dkyges-37/ 2024-04-12T03:37:29.707 DEBUG:boto:Data: 2024-04-12T03:37:29.707 DEBUG:boto:Headers: {} 2024-04-12T03:37:29.707 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:37:29.707 DEBUG:boto:Port: 8000 2024-04-12T03:37:29.707 DEBUG:boto:Params: {} 2024-04-12T03:37:29.708 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:37:29.708 DEBUG:boto:Token: None 2024-04-12T03:37:29.708 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:37:29 GMT /dkyges-37/ 2024-04-12T03:37:29.709 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2xj467mxRFhd63Ak1hX3bgCkz5k= 2024-04-12T03:37:29.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:37:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2xj467mxRFhd63Ak1hX3bgCkz5k=', 'Content-Length': '0'} 2024-04-12T03:37:29.738 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001ba4cd98d2d9173b-006618ac79-15c6-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:37:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:37:29.739 DEBUG:boto:path=/dkyges-37/testredirect 2024-04-12T03:37:29.739 DEBUG:boto:auth_path=/dkyges-37/testredirect 2024-04-12T03:37:29.739 DEBUG:boto:Method: PUT 2024-04-12T03:37:29.739 DEBUG:boto:Path: /dkyges-37/testredirect 2024-04-12T03:37:29.739 DEBUG:boto:Data: 2024-04-12T03:37:29.739 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-12T03:37:29.739 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:37:29.740 DEBUG:boto:Port: 8000 2024-04-12T03:37:29.740 DEBUG:boto:Params: {} 2024-04-12T03:37:29.740 DEBUG:boto:Token: None 2024-04-12T03:37:29.740 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 12 Apr 2024 03:37:29 GMT /dkyges-37/testredirect 2024-04-12T03:37:29.740 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WTVv6jy9ywXg7kv6ovkb73hCeM0= 2024-04-12T03:37:29.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:37:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WTVv6jy9ywXg7kv6ovkb73hCeM0='} 2024-04-12T03:37:29.749 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e59d140bc0c272a-006618ac79-15c6-a1'), ('Date', 'Fri, 12 Apr 2024 03:37:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:37:29.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:37:29.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:37:29.750 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:37:29.808 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:37:29.809 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:37:29.847 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:37:29.861 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:37:29.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:29.861 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:37:29.861 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:37:29.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:29.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:29.861 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:37:29.861 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:37:29.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:29.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:29.862 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:37:29.862 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:37:29.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:29.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:29.862 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:37:29.862 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:37:29.736308Z" 2024-04-12T03:37:29.862 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:29.862 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:37:29.863 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:37:29.864 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:37:29.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:37:29.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:37:29.864 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:37:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:37:29.919 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:37:29.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:37:29.950+0000 7f36bb4a2780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:37:29.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:37:29.951+0000 7f36bb4a2780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:37:29.966 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:37:29.967 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:37:29.967 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:37:29.967 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:37:29.967 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:37:29.967 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:37:29.967 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:29.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:29.967 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:37:29.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:29.967 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:37:29.968 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:29.968 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:29.968 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:37:29.968 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:29.968 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:37:29.968 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:29.968 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:37:29.968 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:29.968 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:29.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:29.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:29.969 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:37:29.969 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:29.969 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:29.969 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:37:29.969 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:29.969 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:29.969 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:29.969 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:37:29.969 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:29.970 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:29.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:29.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:29.970 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:37:29.970 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:29.970 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:29.970 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:37:29.970 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:29.970 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:29.971 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:29.971 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:37:29.971 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:29.971 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:29.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:29.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:29.971 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:37:29.971 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:29.971 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:29.971 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892955.671477_293.1", 2024-04-12T03:37:29.972 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:29.972 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:29.972 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:29.972 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:35:55.671477Z", 2024-04-12T03:37:29.972 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:29.972 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:29.972 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:29.972 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:37:29.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:29.972 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:37:29.973 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:37:29.973 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:37:29.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:29.973 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:37:29Z" 2024-04-12T03:37:29.973 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:37:29.974 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892955.671477_293.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:35:55.671477Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:37:29Z" } 2024-04-12T03:37:29.974 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892955.671477_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:35:55.671477Z', 'realm_epoch': 2}}] 2024-04-12T03:37:29.974 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:37:29.975 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892955.671477_293.1'} 2024-04-12T03:37:29.975 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893049.736308_308.1 target=1_1712892955.671477_293.1 2024-04-12T03:37:34.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:37:34.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:37:34.977 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:37:35.037 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:37:35.037 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:37:35.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:37:35.075+0000 7ff4657e8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:37:35.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:37:35.076+0000 7ff4657e8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:37:35.091 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:37:35.091 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:37:35.091 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:37:35.091 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:37:35.092 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:37:35.092 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:37:35.092 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:35.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:35.092 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:37:35.092 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:35.092 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:37:35.092 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:35.092 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:35.092 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:37:35.092 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:35.093 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:37:35.093 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:35.093 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:37:35.093 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:35.093 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:35.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:35.093 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:35.093 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:37:35.093 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:35.093 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:35.093 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:37:35.094 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:35.094 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:35.094 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:35.094 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:37:35.094 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:35.094 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:35.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:35.094 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:35.094 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:37:35.094 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:35.094 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:35.095 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:37:35.095 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:35.095 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:35.095 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:35.095 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:37:35.095 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:35.095 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:35.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:35.095 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:35.095 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:37:35.095 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:35.095 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:35.096 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892955.671477_293.1", 2024-04-12T03:37:35.096 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:35.096 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:35.096 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:35.096 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:35:55.671477Z", 2024-04-12T03:37:35.096 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:35.096 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:35.096 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:35.096 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:37:35.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:35.096 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:37:35.097 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:37:35.097 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:37:35.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:35.097 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:37:35Z" 2024-04-12T03:37:35.097 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:37:35.098 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892955.671477_293.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:35:55.671477Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:37:35Z" } 2024-04-12T03:37:35.098 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892955.671477_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:35:55.671477Z', 'realm_epoch': 2}}] 2024-04-12T03:37:35.098 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:37:35.098 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892955.671477_293.1'} 2024-04-12T03:37:35.098 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893049.736308_308.1 target=1_1712892955.671477_293.1 2024-04-12T03:37:40.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:37:40.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:37:40.100 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:37:40.160 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:37:40.160 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:37:40.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:37:40.193+0000 7f468192d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:37:40.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:37:40.194+0000 7f468192d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:37:40.210 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:37:40.210 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:37:40.210 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:37:40.210 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:37:40.210 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:37:40.210 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:37:40.210 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:40.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:40.210 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:37:40.210 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:40.211 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:37:40.211 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:40.211 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:40.211 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:37:40.211 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:40.211 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:37:40.211 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:40.211 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:37:40.211 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:40.211 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:40.211 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:40.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:40.212 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:37:40.212 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:40.212 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:40.212 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:37:40.212 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:40.212 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:40.212 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:40.212 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:37:40.212 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:40.212 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:40.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:40.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:40.213 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:37:40.213 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:40.213 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:40.213 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:37:40.213 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:40.213 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:40.213 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:40.213 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:37:40.213 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:40.213 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:40.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:40.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:40.214 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:37:40.214 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:40.214 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:40.214 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892955.671477_293.1", 2024-04-12T03:37:40.214 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:40.214 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:40.214 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:40.214 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:35:55.671477Z", 2024-04-12T03:37:40.214 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:40.215 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:40.215 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:40.215 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:37:40.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:40.215 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:37:40.215 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:37:40.215 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:37:40.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:40.215 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:37:40Z" 2024-04-12T03:37:40.215 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:37:40.216 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892955.671477_293.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:35:55.671477Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:37:40Z" } 2024-04-12T03:37:40.216 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892955.671477_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:35:55.671477Z', 'realm_epoch': 2}}] 2024-04-12T03:37:40.217 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:37:40.217 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892955.671477_293.1'} 2024-04-12T03:37:40.217 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893049.736308_308.1 target=1_1712892955.671477_293.1 2024-04-12T03:37:45.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:37:45.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:37:45.218 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:37:45.278 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:37:45.278 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:37:45.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:37:45.310+0000 7f17cf6ea780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:37:45.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:37:45.311+0000 7f17cf6ea780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:37:45.328 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:37:45.328 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:37:45.328 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:37:45.328 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:37:45.328 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:37:45.328 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:37:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:37:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:37:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:37:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:37:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:37:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:37:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:37:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:37:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:37:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:37:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:37:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:37:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892955.671477_293.1", 2024-04-12T03:37:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:35:55.671477Z", 2024-04-12T03:37:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:37:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:37:45.334 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:37:45.334 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:37:45.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:45.334 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:37:45Z" 2024-04-12T03:37:45.334 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:37:45.335 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892955.671477_293.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:35:55.671477Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:37:45Z" } 2024-04-12T03:37:45.335 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892955.671477_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:35:55.671477Z', 'realm_epoch': 2}}] 2024-04-12T03:37:45.335 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:37:45.336 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892955.671477_293.1'} 2024-04-12T03:37:45.336 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893049.736308_308.1 target=1_1712892955.671477_293.1 2024-04-12T03:37:50.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:37:50.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:37:50.338 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:37:50.397 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:37:50.398 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:37:50.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:37:50.431+0000 7fa05f157780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:37:50.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:37:50.432+0000 7fa05f157780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:37:50.448 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:37:50.448 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:37:50.448 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:37:50.448 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:37:50.448 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:37:50.449 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:37:50.449 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:50.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:50.450 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:37:50.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:50.450 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:37:50.450 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:50.450 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:50.450 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:37:50.450 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:50.450 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:37:50.450 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:50.450 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:37:50.451 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:50.451 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:50.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:50.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:50.451 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:37:50.451 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:50.451 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:50.451 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:37:50.451 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:50.451 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:50.451 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:50.451 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:37:50.452 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:50.452 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:50.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:50.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:50.452 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:37:50.452 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:50.452 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:50.452 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:37:50.452 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:50.452 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:50.452 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:37:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:37:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892955.671477_293.1", 2024-04-12T03:37:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:50.453 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:35:55.671477Z", 2024-04-12T03:37:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:37:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:37:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:37:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:37:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:50.454 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:37:50Z" 2024-04-12T03:37:50.454 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:37:50.455 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892955.671477_293.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:35:55.671477Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:37:50Z" } 2024-04-12T03:37:50.456 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892955.671477_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:35:55.671477Z', 'realm_epoch': 2}}] 2024-04-12T03:37:50.456 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:37:50.456 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892955.671477_293.1'} 2024-04-12T03:37:50.456 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893049.736308_308.1 target=1_1712892955.671477_293.1 2024-04-12T03:37:55.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:37:55.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:37:55.458 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:37:55.517 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:37:55.517 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:37:55.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:37:55.550+0000 7efdf3aac780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:37:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:37:55.552+0000 7efdf3aac780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:37:55.567 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:37:55.567 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:37:55.567 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:37:55.567 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:37:55.567 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:37:55.568 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:37:55.568 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:55.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:55.568 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:37:55.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:55.568 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:37:55.568 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:55.568 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:55.568 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:37:55.568 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:55.568 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:37:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:37:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:37:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:37:55.569 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:37:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:37:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:55.570 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:37:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:37:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:37:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:37:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:37:55.571 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:37:55.572 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712892955.671477_293.1", 2024-04-12T03:37:55.572 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:37:55.572 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:37:55.572 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:37:55.572 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:35:55.671477Z", 2024-04-12T03:37:55.572 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:37:55.572 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:55.572 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:37:55.572 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:37:55.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:55.572 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:37:55.573 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:37:55.573 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:37:55.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:37:55.573 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:37:55Z" 2024-04-12T03:37:55.573 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:37:55.574 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712892955.671477_293.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:35:55.671477Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:37:55Z" } 2024-04-12T03:37:55.574 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712892955.671477_293.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:35:55.671477Z', 'realm_epoch': 2}}] 2024-04-12T03:37:55.574 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:37:55.574 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712892955.671477_293.1'} 2024-04-12T03:37:55.574 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893049.736308_308.1 target=1_1712892955.671477_293.1 2024-04-12T03:38:00.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:38:00.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:38:00.576 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:38:00.635 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:38:00.635 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:38:00.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:38:00.668+0000 7f4cfcf71780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:38:00.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:38:00.669+0000 7f4cfcf71780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:38:00.684 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:38:00.684 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:38:00.684 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:38:00.684 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:38:00.684 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:38:00.684 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:38:00.684 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:38:00.685 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.685 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:38:00.685 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.685 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:38:00.685 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:38:00.685 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:38:00.685 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:38:00.685 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:38:00.685 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:38:00.685 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:38:00.685 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:38:00.685 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:38:00.686 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:38:00.686 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.686 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.686 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:38:00.686 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:38:00.686 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:38:00.686 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:38:00.686 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:38:00.686 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:38:00.687 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:38:00.687 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:38:00.687 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:38:00.687 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:38:00.687 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.687 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.687 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:38:00.687 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:38:00.687 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:38:00.687 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:38:00.688 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:38:00.688 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:38:00.688 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:38:00.688 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:38:00.688 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:38:00.688 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:38:00.688 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.688 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.688 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:38:00.689 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:38:00.689 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:38:00.689 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:38:00.689 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:38:00.689 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:38:00.689 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:38:00.689 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:38:00.689 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:38:00.689 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:38:00.689 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:38:00.689 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:38:00.689 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.690 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:38:00.690 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:38:00.690 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:38:00.690 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.690 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:38:00Z" 2024-04-12T03:38:00.690 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:38:00.691 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:38:00Z" } 2024-04-12T03:38:00.691 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:38:00.691 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:38:00.691 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:38:00.691 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:38:00.692 DEBUG:boto:path=/dkyges-37/ 2024-04-12T03:38:00.692 DEBUG:boto:auth_path=/dkyges-37/ 2024-04-12T03:38:00.692 DEBUG:boto:Method: HEAD 2024-04-12T03:38:00.692 DEBUG:boto:Path: /dkyges-37/ 2024-04-12T03:38:00.692 DEBUG:boto:Data: 2024-04-12T03:38:00.692 DEBUG:boto:Headers: {} 2024-04-12T03:38:00.692 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:38:00.692 DEBUG:boto:Port: 8001 2024-04-12T03:38:00.692 DEBUG:boto:Params: {} 2024-04-12T03:38:00.693 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:38:00.693 DEBUG:boto:Token: None 2024-04-12T03:38:00.693 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:38:00 GMT /dkyges-37/ 2024-04-12T03:38:00.693 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AitkiDy6bj6LEf47IRT6AJFU+Wg= 2024-04-12T03:38:00.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:38:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AitkiDy6bj6LEf47IRT6AJFU+Wg=', 'Content-Length': '0'} 2024-04-12T03:38:00.724 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cbcd1f6be062bac1-006618ac98-15c4-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:38:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:38:00.724 DEBUG:boto:path=/dkyges-37/testredirect 2024-04-12T03:38:00.724 DEBUG:boto:auth_path=/dkyges-37/testredirect 2024-04-12T03:38:00.724 DEBUG:boto:Method: HEAD 2024-04-12T03:38:00.724 DEBUG:boto:Path: /dkyges-37/testredirect 2024-04-12T03:38:00.724 DEBUG:boto:Data: 2024-04-12T03:38:00.724 DEBUG:boto:Headers: {} 2024-04-12T03:38:00.725 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:38:00.725 DEBUG:boto:Port: 8001 2024-04-12T03:38:00.725 DEBUG:boto:Params: {} 2024-04-12T03:38:00.725 DEBUG:boto:Token: None 2024-04-12T03:38:00.725 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:38:00 GMT /dkyges-37/testredirect 2024-04-12T03:38:00.725 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pKJqafbHxlvBbxankqf62PZOI8E= 2024-04-12T03:38:00.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:38:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pKJqafbHxlvBbxankqf62PZOI8E=', 'Content-Length': '0'} 2024-04-12T03:38:00.727 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000001abc70e6283f768-006618ac98-15c4-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:38:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:38:00.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:38:00.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:38:00.727 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:38:00.783 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:38:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:38:00.809 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:38:00.809 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:38:00.809 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T03:38:00.809 INFO:teuthology.orchestra.run.smithi043.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-12T03:38:00.809 INFO:teuthology.orchestra.run.smithi043.stdout: "control_pool": "a2.rgw.control", 2024-04-12T03:38:00.809 INFO:teuthology.orchestra.run.smithi043.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-12T03:38:00.809 INFO:teuthology.orchestra.run.smithi043.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-12T03:38:00.809 INFO:teuthology.orchestra.run.smithi043.stdout: "log_pool": "a2.rgw.log", 2024-04-12T03:38:00.809 INFO:teuthology.orchestra.run.smithi043.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-12T03:38:00.809 INFO:teuthology.orchestra.run.smithi043.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-12T03:38:00.810 INFO:teuthology.orchestra.run.smithi043.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-12T03:38:00.810 INFO:teuthology.orchestra.run.smithi043.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-12T03:38:00.810 INFO:teuthology.orchestra.run.smithi043.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-12T03:38:00.810 INFO:teuthology.orchestra.run.smithi043.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-12T03:38:00.810 INFO:teuthology.orchestra.run.smithi043.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-12T03:38:00.810 INFO:teuthology.orchestra.run.smithi043.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-12T03:38:00.810 INFO:teuthology.orchestra.run.smithi043.stdout: "otp_pool": "a2.rgw.otp", 2024-04-12T03:38:00.810 INFO:teuthology.orchestra.run.smithi043.stdout: "system_key": { 2024-04-12T03:38:00.810 INFO:teuthology.orchestra.run.smithi043.stdout: "access_key": "6ESEI28JZW8UW9BY", 2024-04-12T03:38:00.810 INFO:teuthology.orchestra.run.smithi043.stdout: "secret_key": "oVIj7wCADBnecpdBF65ZyUvXEbg14PDu" 2024-04-12T03:38:00.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.811 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_pools": [ 2024-04-12T03:38:00.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.811 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "default-placement", 2024-04-12T03:38:00.811 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:38:00.811 INFO:teuthology.orchestra.run.smithi043.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-12T03:38:00.811 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": { 2024-04-12T03:38:00.811 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD": { 2024-04-12T03:38:00.811 INFO:teuthology.orchestra.run.smithi043.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-12T03:38:00.811 INFO:teuthology.orchestra.run.smithi043.stdout: "compression_type": "random" 2024-04-12T03:38:00.811 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:38:00.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.812 INFO:teuthology.orchestra.run.smithi043.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-12T03:38:00.812 INFO:teuthology.orchestra.run.smithi043.stdout: "index_type": 0, 2024-04-12T03:38:00.812 INFO:teuthology.orchestra.run.smithi043.stdout: "inline_data": true 2024-04-12T03:38:00.812 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:38:00.812 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:38:00.812 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:38:00.812 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:38:00.812 INFO:teuthology.orchestra.run.smithi043.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-12T03:38:00.812 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:38:00.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:38:00.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:38:00.813 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:38:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:38:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:38:00.917 INFO:teuthology.orchestra.run.smithi043.stderr:Sending period to new master zone 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61 2024-04-12T03:38:00.945 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:38:00.945 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:38:00.945 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6, 2024-04-12T03:38:00.945 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T03:38:00.946 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T03:38:00.946 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T03:38:00.946 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:38:00.946 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T03:38:00.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.946 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T03:38:00.946 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T03:38:00.946 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:38:00.946 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T03:38:00.946 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:38:00.947 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:38:00.947 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:38:00.947 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:38:00.947 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:38:00.947 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:38:00.947 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:38:00.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.947 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:38:00.947 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T03:38:00.947 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:38:00.947 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T03:38:00.948 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:38:00.948 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:38:00.948 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:38:00.948 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:38:00.948 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:38:00.948 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:38:00.948 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": false, 2024-04-12T03:38:00.948 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:38:00.948 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:38:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:38:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:38:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:38:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:38:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:38:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T03:38:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:38:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:38:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:38:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:38:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:38:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:38:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:38:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:38:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:38:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:38:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:38:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:38:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:38:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:38:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:38:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:38:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:38:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:38:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:38:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:38:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:38:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:38:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:38:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:38:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:38:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:38:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:38:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:38:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:38:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:38:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:38:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:38:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T03:38:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T03:38:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:38:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T03:38:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:38:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:38:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:38:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:38:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:38:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:38:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:38:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:38:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T03:38:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:38:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:38:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:38:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:38:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:38:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:38:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:38:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:38:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:38:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:38:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:38:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:38:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:38:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:38:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:38:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:38:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T03:38:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:38:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T03:38:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:38:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:38:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:38:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:38:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:38:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:38:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:38:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:38:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:38:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:38:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:38:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:38:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:38:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:38:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:38:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:38:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:38:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:38:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:38:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:38:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:38:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:38:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:38:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:38:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:38:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:38:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:38:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:38:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:38:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:38:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:38:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:38:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T03:38:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:38:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T03:38:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:38:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T03:38:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:38:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T03:38:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:38:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:38:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T03:38:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:38:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:38:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T03:38:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:38:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T03:38:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T03:38:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:38:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:38:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:38:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:38:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:38:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T03:38:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:38:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:38:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:38:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:38:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:38:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T03:38:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:38:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:38:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:38:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:38:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:38:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T03:38:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:38:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:38:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:38:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:38:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:38:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T03:38:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:38:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:38:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:38:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:38:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:38:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:38:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:38:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:38:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T03:38:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:38:00.968 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:38:00.969 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T03:39:00.970 DEBUG:boto:path=/dkyges-37/testredirect 2024-04-12T03:39:00.970 DEBUG:boto:auth_path=/dkyges-37/testredirect 2024-04-12T03:39:00.971 DEBUG:boto:Method: HEAD 2024-04-12T03:39:00.971 DEBUG:boto:Path: /dkyges-37/testredirect 2024-04-12T03:39:00.971 DEBUG:boto:Data: 2024-04-12T03:39:00.971 DEBUG:boto:Headers: {} 2024-04-12T03:39:00.971 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:39:00.971 DEBUG:boto:Port: 8001 2024-04-12T03:39:00.971 DEBUG:boto:Params: {} 2024-04-12T03:39:00.972 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:39:00.972 DEBUG:boto:Token: None 2024-04-12T03:39:00.972 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:00 GMT /dkyges-37/testredirect 2024-04-12T03:39:00.972 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZrXCJ8xEbNrfBIDJMKR9xcU6FYo= 2024-04-12T03:39:00.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZrXCJ8xEbNrfBIDJMKR9xcU6FYo=', 'Content-Length': '0'} 2024-04-12T03:39:00.987 DEBUG:boto:Response headers: [('Location', 'http://smithi043.front.sepia.ceph.com:8000/dkyges-37/testredirect?'), ('x-amz-request-id', 'tx00000034126ac479f76b4-006618acd4-161e-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:00.987 DEBUG:boto:Redirecting: http://smithi043.front.sepia.ceph.com/dkyges-37/testredirect 2024-04-12T03:39:00.987 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2024-04-12T03:39:00.987 DEBUG:boto:Token: None 2024-04-12T03:39:00.987 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:39:00 GMT /dkyges-37/testredirect 2024-04-12T03:39:00.988 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZrXCJ8xEbNrfBIDJMKR9xcU6FYo= 2024-04-12T03:39:00.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZrXCJ8xEbNrfBIDJMKR9xcU6FYo=', 'Content-Length': '0'} 2024-04-12T03:39:01.002 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:37:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000005d1b8be5028e054b-006618acd4-161c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:01.003 DEBUG:boto:path=/dkyges-37/testredirect 2024-04-12T03:39:01.003 DEBUG:boto:auth_path=/dkyges-37/testredirect 2024-04-12T03:39:01.003 DEBUG:boto:Method: GET 2024-04-12T03:39:01.003 DEBUG:boto:Path: /dkyges-37/testredirect 2024-04-12T03:39:01.003 DEBUG:boto:Data: 2024-04-12T03:39:01.003 DEBUG:boto:Headers: {} 2024-04-12T03:39:01.003 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:39:01.003 DEBUG:boto:Port: 8001 2024-04-12T03:39:01.003 DEBUG:boto:Params: {} 2024-04-12T03:39:01.003 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:39:01.004 DEBUG:boto:Token: None 2024-04-12T03:39:01.004 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:01 GMT /dkyges-37/testredirect 2024-04-12T03:39:01.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs= 2024-04-12T03:39:01.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs=', 'Content-Length': '0'} 2024-04-12T03:39:01.015 DEBUG:boto:Response headers: [('Location', 'http://smithi043.front.sepia.ceph.com:8000/dkyges-37/testredirect?'), ('x-amz-request-id', 'tx00000db8720841ab6a578-006618acd5-161e-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:01.015 DEBUG:boto:Redirecting: http://smithi043.front.sepia.ceph.com/dkyges-37/testredirect 2024-04-12T03:39:01.015 DEBUG:boto:Token: None 2024-04-12T03:39:01.015 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:01 GMT /dkyges-37/testredirect 2024-04-12T03:39:01.016 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs= 2024-04-12T03:39:01.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs=', 'Content-Length': '0'} 2024-04-12T03:39:01.018 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:37:29 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000a01f25fab972db57-006618acd5-161c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:01.018 DEBUG:boto:path=/dkyges-37/testredirect 2024-04-12T03:39:01.018 DEBUG:boto:auth_path=/dkyges-37/testredirect 2024-04-12T03:39:01.018 DEBUG:boto:Method: PUT 2024-04-12T03:39:01.018 DEBUG:boto:Path: /dkyges-37/testredirect 2024-04-12T03:39:01.018 DEBUG:boto:Data: 2024-04-12T03:39:01.019 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-12T03:39:01.019 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:39:01.019 DEBUG:boto:Port: 8000 2024-04-12T03:39:01.019 DEBUG:boto:Params: {} 2024-04-12T03:39:01.019 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:39:01.019 DEBUG:boto:Token: None 2024-04-12T03:39:01.019 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Fri, 12 Apr 2024 03:39:01 GMT /dkyges-37/testredirect 2024-04-12T03:39:01.019 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FVBzd3gYRNY1Fl1M0GqcVbTGCnk= 2024-04-12T03:39:01.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FVBzd3gYRNY1Fl1M0GqcVbTGCnk='} 2024-04-12T03:39:01.042 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000297f41c21d6ff5f0-006618acd5-161c-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:01.043 DEBUG:boto:path=/dkyges-37/testredirect 2024-04-12T03:39:01.043 DEBUG:boto:auth_path=/dkyges-37/testredirect 2024-04-12T03:39:01.043 DEBUG:boto:Method: GET 2024-04-12T03:39:01.043 DEBUG:boto:Path: /dkyges-37/testredirect 2024-04-12T03:39:01.043 DEBUG:boto:Data: 2024-04-12T03:39:01.043 DEBUG:boto:Headers: {} 2024-04-12T03:39:01.043 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:39:01.043 DEBUG:boto:Port: 8001 2024-04-12T03:39:01.043 DEBUG:boto:Params: {} 2024-04-12T03:39:01.043 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:39:01.044 DEBUG:boto:Token: None 2024-04-12T03:39:01.044 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:01 GMT /dkyges-37/testredirect 2024-04-12T03:39:01.044 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs= 2024-04-12T03:39:01.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs=', 'Content-Length': '0'} 2024-04-12T03:39:01.055 DEBUG:boto:Response headers: [('Location', 'http://smithi043.front.sepia.ceph.com:8000/dkyges-37/testredirect?'), ('x-amz-request-id', 'tx00000801e60a7143fb43b-006618acd5-161e-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:01.055 DEBUG:boto:Redirecting: http://smithi043.front.sepia.ceph.com/dkyges-37/testredirect 2024-04-12T03:39:01.056 DEBUG:boto:Token: None 2024-04-12T03:39:01.056 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:01 GMT /dkyges-37/testredirect 2024-04-12T03:39:01.056 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs= 2024-04-12T03:39:01.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs=', 'Content-Length': '0'} 2024-04-12T03:39:01.058 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx00000765a01eb4501d237-006618acd5-161c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:01.058 DEBUG:boto:path=/dkyges-37/testredirect 2024-04-12T03:39:01.058 DEBUG:boto:auth_path=/dkyges-37/testredirect 2024-04-12T03:39:01.058 DEBUG:boto:Method: PUT 2024-04-12T03:39:01.058 DEBUG:boto:Path: /dkyges-37/testredirect 2024-04-12T03:39:01.059 DEBUG:boto:Data: 2024-04-12T03:39:01.059 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-12T03:39:01.059 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:39:01.059 DEBUG:boto:Port: 8000 2024-04-12T03:39:01.059 DEBUG:boto:Params: {} 2024-04-12T03:39:01.059 DEBUG:boto:Token: None 2024-04-12T03:39:01.059 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Fri, 12 Apr 2024 03:39:01 GMT /dkyges-37/testredirect 2024-04-12T03:39:01.059 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0LSudXGsA4wu2SXnRFATGns8eUg= 2024-04-12T03:39:01.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0LSudXGsA4wu2SXnRFATGns8eUg='} 2024-04-12T03:39:01.066 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3ffe9957c83cb89-006618acd5-161c-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:01.066 DEBUG:boto:path=/dkyges-37/testredirect 2024-04-12T03:39:01.066 DEBUG:boto:auth_path=/dkyges-37/testredirect 2024-04-12T03:39:01.066 DEBUG:boto:Method: GET 2024-04-12T03:39:01.066 DEBUG:boto:Path: /dkyges-37/testredirect 2024-04-12T03:39:01.066 DEBUG:boto:Data: 2024-04-12T03:39:01.066 DEBUG:boto:Headers: {} 2024-04-12T03:39:01.066 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:39:01.067 DEBUG:boto:Port: 8001 2024-04-12T03:39:01.067 DEBUG:boto:Params: {} 2024-04-12T03:39:01.067 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:39:01.067 DEBUG:boto:Token: None 2024-04-12T03:39:01.067 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:01 GMT /dkyges-37/testredirect 2024-04-12T03:39:01.067 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs= 2024-04-12T03:39:01.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs=', 'Content-Length': '0'} 2024-04-12T03:39:01.098 DEBUG:boto:Response headers: [('Location', 'http://smithi043.front.sepia.ceph.com:8000/dkyges-37/testredirect?'), ('x-amz-request-id', 'tx00000047339a23fccd50a-006618acd5-161e-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:01.098 DEBUG:boto:Redirecting: http://smithi043.front.sepia.ceph.com/dkyges-37/testredirect 2024-04-12T03:39:01.099 DEBUG:boto:Token: None 2024-04-12T03:39:01.099 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:01 GMT /dkyges-37/testredirect 2024-04-12T03:39:01.099 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs= 2024-04-12T03:39:01.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs=', 'Content-Length': '0'} 2024-04-12T03:39:01.101 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx00000591f16efabbad363-006618acd5-161c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:01.101 DEBUG:boto:path=/dkyges-37/testredirect 2024-04-12T03:39:01.101 DEBUG:boto:auth_path=/dkyges-37/testredirect 2024-04-12T03:39:01.101 DEBUG:boto:Method: PUT 2024-04-12T03:39:01.101 DEBUG:boto:Path: /dkyges-37/testredirect 2024-04-12T03:39:01.102 DEBUG:boto:Data: 2024-04-12T03:39:01.102 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-12T03:39:01.102 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:39:01.102 DEBUG:boto:Port: 8000 2024-04-12T03:39:01.102 DEBUG:boto:Params: {} 2024-04-12T03:39:01.102 DEBUG:boto:Token: None 2024-04-12T03:39:01.102 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Fri, 12 Apr 2024 03:39:01 GMT /dkyges-37/testredirect 2024-04-12T03:39:01.102 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:baM1fdrBmaBrbdRwJJw/x0gWLT0= 2024-04-12T03:39:01.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:baM1fdrBmaBrbdRwJJw/x0gWLT0='} 2024-04-12T03:39:01.109 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042084082cf66b78c-006618acd5-161c-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:01.109 DEBUG:boto:path=/dkyges-37/testredirect 2024-04-12T03:39:01.109 DEBUG:boto:auth_path=/dkyges-37/testredirect 2024-04-12T03:39:01.109 DEBUG:boto:Method: GET 2024-04-12T03:39:01.109 DEBUG:boto:Path: /dkyges-37/testredirect 2024-04-12T03:39:01.109 DEBUG:boto:Data: 2024-04-12T03:39:01.109 DEBUG:boto:Headers: {} 2024-04-12T03:39:01.109 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:39:01.110 DEBUG:boto:Port: 8001 2024-04-12T03:39:01.110 DEBUG:boto:Params: {} 2024-04-12T03:39:01.110 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:39:01.110 DEBUG:boto:Token: None 2024-04-12T03:39:01.110 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:01 GMT /dkyges-37/testredirect 2024-04-12T03:39:01.110 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs= 2024-04-12T03:39:01.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs=', 'Content-Length': '0'} 2024-04-12T03:39:01.123 DEBUG:boto:Response headers: [('Location', 'http://smithi043.front.sepia.ceph.com:8000/dkyges-37/testredirect?'), ('x-amz-request-id', 'tx00000387f607c833256bc-006618acd5-161e-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:01.123 DEBUG:boto:Redirecting: http://smithi043.front.sepia.ceph.com/dkyges-37/testredirect 2024-04-12T03:39:01.124 DEBUG:boto:Token: None 2024-04-12T03:39:01.124 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:01 GMT /dkyges-37/testredirect 2024-04-12T03:39:01.124 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs= 2024-04-12T03:39:01.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs=', 'Content-Length': '0'} 2024-04-12T03:39:01.126 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx0000005fbf0d3270fddc7-006618acd5-161c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:01.126 DEBUG:boto:path=/dkyges-37/testredirect 2024-04-12T03:39:01.126 DEBUG:boto:auth_path=/dkyges-37/testredirect 2024-04-12T03:39:01.126 DEBUG:boto:Method: PUT 2024-04-12T03:39:01.126 DEBUG:boto:Path: /dkyges-37/testredirect 2024-04-12T03:39:01.126 DEBUG:boto:Data: 2024-04-12T03:39:01.126 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-12T03:39:01.127 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:39:01.127 DEBUG:boto:Port: 8000 2024-04-12T03:39:01.127 DEBUG:boto:Params: {} 2024-04-12T03:39:01.127 DEBUG:boto:Token: None 2024-04-12T03:39:01.127 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Fri, 12 Apr 2024 03:39:01 GMT /dkyges-37/testredirect 2024-04-12T03:39:01.127 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l5wtaobvcmm8RS1oL08FsRAhtbo= 2024-04-12T03:39:01.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:39:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l5wtaobvcmm8RS1oL08FsRAhtbo='} 2024-04-12T03:39:01.136 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000693bb71079f21074-006618acd5-161c-a1'), ('Date', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:01.136 DEBUG:boto:path=/dkyges-37/testredirect 2024-04-12T03:39:01.136 DEBUG:boto:auth_path=/dkyges-37/testredirect 2024-04-12T03:39:01.136 DEBUG:boto:Method: GET 2024-04-12T03:39:01.136 DEBUG:boto:Path: /dkyges-37/testredirect 2024-04-12T03:39:01.136 DEBUG:boto:Data: 2024-04-12T03:39:01.136 DEBUG:boto:Headers: {} 2024-04-12T03:39:01.136 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:39:01.137 DEBUG:boto:Port: 8001 2024-04-12T03:39:01.137 DEBUG:boto:Params: {} 2024-04-12T03:39:01.137 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:39:01.137 DEBUG:boto:Token: None 2024-04-12T03:39:01.137 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:01 GMT /dkyges-37/testredirect 2024-04-12T03:39:01.137 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs= 2024-04-12T03:39:01.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs=', 'Content-Length': '0'} 2024-04-12T03:39:01.148 DEBUG:boto:Response headers: [('Location', 'http://smithi043.front.sepia.ceph.com:8000/dkyges-37/testredirect?'), ('x-amz-request-id', 'tx00000c0ba1640ef30c651-006618acd5-161e-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:01.149 DEBUG:boto:Redirecting: http://smithi043.front.sepia.ceph.com/dkyges-37/testredirect 2024-04-12T03:39:01.149 DEBUG:boto:Token: None 2024-04-12T03:39:01.149 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:39:01 GMT /dkyges-37/testredirect 2024-04-12T03:39:01.149 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs= 2024-04-12T03:39:01.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:39:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cDKQyCcL3fZqMfj2UCZNxNW0tNs=', 'Content-Length': '0'} 2024-04-12T03:39:01.151 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx000003d4ea325f6385d2a-006618acd5-161c-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:39:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:39:01.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:01.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:01.152 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=true --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:39:01.213 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:01.213 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:01.239 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:39:01.239 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:39:01.239 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T03:39:01.239 INFO:teuthology.orchestra.run.smithi043.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-12T03:39:01.240 INFO:teuthology.orchestra.run.smithi043.stdout: "control_pool": "a2.rgw.control", 2024-04-12T03:39:01.240 INFO:teuthology.orchestra.run.smithi043.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-12T03:39:01.240 INFO:teuthology.orchestra.run.smithi043.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-12T03:39:01.240 INFO:teuthology.orchestra.run.smithi043.stdout: "log_pool": "a2.rgw.log", 2024-04-12T03:39:01.240 INFO:teuthology.orchestra.run.smithi043.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-12T03:39:01.240 INFO:teuthology.orchestra.run.smithi043.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-12T03:39:01.240 INFO:teuthology.orchestra.run.smithi043.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-12T03:39:01.240 INFO:teuthology.orchestra.run.smithi043.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-12T03:39:01.240 INFO:teuthology.orchestra.run.smithi043.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-12T03:39:01.240 INFO:teuthology.orchestra.run.smithi043.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-12T03:39:01.240 INFO:teuthology.orchestra.run.smithi043.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-12T03:39:01.241 INFO:teuthology.orchestra.run.smithi043.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-12T03:39:01.241 INFO:teuthology.orchestra.run.smithi043.stdout: "otp_pool": "a2.rgw.otp", 2024-04-12T03:39:01.241 INFO:teuthology.orchestra.run.smithi043.stdout: "system_key": { 2024-04-12T03:39:01.241 INFO:teuthology.orchestra.run.smithi043.stdout: "access_key": "6ESEI28JZW8UW9BY", 2024-04-12T03:39:01.241 INFO:teuthology.orchestra.run.smithi043.stdout: "secret_key": "oVIj7wCADBnecpdBF65ZyUvXEbg14PDu" 2024-04-12T03:39:01.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:39:01.241 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_pools": [ 2024-04-12T03:39:01.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:39:01.241 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "default-placement", 2024-04-12T03:39:01.241 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:39:01.241 INFO:teuthology.orchestra.run.smithi043.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-12T03:39:01.241 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": { 2024-04-12T03:39:01.242 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD": { 2024-04-12T03:39:01.242 INFO:teuthology.orchestra.run.smithi043.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-12T03:39:01.242 INFO:teuthology.orchestra.run.smithi043.stdout: "compression_type": "random" 2024-04-12T03:39:01.242 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:39:01.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:39:01.242 INFO:teuthology.orchestra.run.smithi043.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-12T03:39:01.242 INFO:teuthology.orchestra.run.smithi043.stdout: "index_type": 0, 2024-04-12T03:39:01.242 INFO:teuthology.orchestra.run.smithi043.stdout: "inline_data": true 2024-04-12T03:39:01.242 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:39:01.242 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:39:01.242 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:39:01.242 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:39:01.243 INFO:teuthology.orchestra.run.smithi043.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-12T03:39:01.243 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:39:01.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:01.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:39:01.244 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:39:01.298 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:39:01.298 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:39:01.347 INFO:teuthology.orchestra.run.smithi043.stderr:Sending period to new master zone 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61 2024-04-12T03:39:01.376 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:39:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:39:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7, 2024-04-12T03:39:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T03:39:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T03:39:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T03:39:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:39:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T03:39:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:39:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T03:39:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T03:39:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:39:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T03:39:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:39:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:39:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:39:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:39:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:39:01.378 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:39:01.378 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:39:01.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:39:01.378 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:39:01.378 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T03:39:01.378 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:39:01.378 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T03:39:01.378 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:39:01.378 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:39:01.378 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:39:01.378 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:39:01.379 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:39:01.379 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:39:01.379 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:39:01.379 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:39:01.379 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:39:01.379 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:39:01.379 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:39:01.379 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:39:01.379 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:39:01.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:39:01.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:39:01.380 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:39:01.380 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T03:39:01.380 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:39:01.380 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:39:01.380 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:39:01.380 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:39:01.380 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:39:01.380 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:39:01.380 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:39:01.380 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:39:01.380 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:39:01.380 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:39:01.381 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:39:01.381 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:39:01.381 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:39:01.381 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:39:01.381 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:39:01.381 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:39:01.381 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:39:01.381 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:39:01.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:39:01.381 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:39:01.381 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:39:01.382 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:39:01.382 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:39:01.382 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:39:01.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:39:01.382 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:39:01.382 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:39:01.382 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:39:01.382 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:39:01.382 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:39:01.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:39:01.382 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:39:01.383 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:39:01.383 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:39:01.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:39:01.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:39:01.383 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T03:39:01.383 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T03:39:01.383 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:39:01.383 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T03:39:01.383 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:39:01.383 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:39:01.383 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:39:01.383 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:39:01.384 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:39:01.384 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:39:01.384 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:39:01.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:39:01.384 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:39:01.384 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T03:39:01.384 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:39:01.384 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:39:01.384 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:39:01.384 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:39:01.384 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:39:01.385 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:39:01.385 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:39:01.385 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:39:01.385 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:39:01.385 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:39:01.385 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:39:01.385 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:39:01.395 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:39:01.395 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:39:01.395 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:39:01.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:39:01.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:39:01.395 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:39:01.395 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T03:39:01.395 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:39:01.396 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T03:39:01.396 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:39:01.396 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:39:01.396 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:39:01.396 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:39:01.396 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:39:01.396 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:39:01.396 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:39:01.396 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:39:01.396 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:39:01.396 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:39:01.397 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:39:01.397 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:39:01.397 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:39:01.397 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:39:01.397 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:39:01.397 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:39:01.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:39:01.397 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:39:01.397 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:39:01.397 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:39:01.397 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:39:01.398 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:39:01.398 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:39:01.398 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:39:01.398 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:39:01.398 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:39:01.398 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:39:01.398 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:39:01.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:39:01.398 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:39:01.398 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:39:01.398 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:39:01.399 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:39:01.399 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:39:01.399 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T03:39:01.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:39:01.399 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:39:01.399 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T03:39:01.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:39:01.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:39:01.399 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:39:01.399 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T03:39:01.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:39:01.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:39:01.400 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:39:01.400 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T03:39:01.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:39:01.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:39:01.400 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:39:01.400 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T03:39:01.400 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:39:01.400 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:39:01.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:39:01.400 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T03:39:01.400 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:39:01.401 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T03:39:01.401 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T03:39:01.401 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:39:01.401 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:39:01.401 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:39:01.401 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:39:01.401 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:39:01.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:39:01.401 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T03:39:01.401 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:39:01.401 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:39:01.401 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:39:01.402 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:39:01.402 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:39:01.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:39:01.402 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T03:39:01.402 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:39:01.402 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:39:01.402 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:39:01.402 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:39:01.402 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:39:01.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:39:01.402 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T03:39:01.402 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:39:01.403 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:39:01.403 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:39:01.403 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:39:01.403 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:39:01.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:39:01.403 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T03:39:01.403 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:39:01.403 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:39:01.403 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:39:01.403 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:39:01.403 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:39:01.404 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:39:01.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:39:01.404 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:39:01.404 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T03:39:01.404 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:39:01.404 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:39:01.404 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to true 2024-04-12T03:40:01.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:40:01.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:40:01.406 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone= --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:40:01.466 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:40:01.466 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:40:01.489 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:40:01.490 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:40:01.490 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T03:40:01.490 INFO:teuthology.orchestra.run.smithi043.stdout: "domain_root": "a2.rgw.meta:root", 2024-04-12T03:40:01.490 INFO:teuthology.orchestra.run.smithi043.stdout: "control_pool": "a2.rgw.control", 2024-04-12T03:40:01.490 INFO:teuthology.orchestra.run.smithi043.stdout: "gc_pool": "a2.rgw.log:gc", 2024-04-12T03:40:01.490 INFO:teuthology.orchestra.run.smithi043.stdout: "lc_pool": "a2.rgw.log:lc", 2024-04-12T03:40:01.490 INFO:teuthology.orchestra.run.smithi043.stdout: "log_pool": "a2.rgw.log", 2024-04-12T03:40:01.490 INFO:teuthology.orchestra.run.smithi043.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-04-12T03:40:01.490 INFO:teuthology.orchestra.run.smithi043.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-04-12T03:40:01.491 INFO:teuthology.orchestra.run.smithi043.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-04-12T03:40:01.491 INFO:teuthology.orchestra.run.smithi043.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-04-12T03:40:01.491 INFO:teuthology.orchestra.run.smithi043.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-04-12T03:40:01.491 INFO:teuthology.orchestra.run.smithi043.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-04-12T03:40:01.491 INFO:teuthology.orchestra.run.smithi043.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-04-12T03:40:01.491 INFO:teuthology.orchestra.run.smithi043.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-04-12T03:40:01.491 INFO:teuthology.orchestra.run.smithi043.stdout: "otp_pool": "a2.rgw.otp", 2024-04-12T03:40:01.491 INFO:teuthology.orchestra.run.smithi043.stdout: "system_key": { 2024-04-12T03:40:01.491 INFO:teuthology.orchestra.run.smithi043.stdout: "access_key": "6ESEI28JZW8UW9BY", 2024-04-12T03:40:01.491 INFO:teuthology.orchestra.run.smithi043.stdout: "secret_key": "oVIj7wCADBnecpdBF65ZyUvXEbg14PDu" 2024-04-12T03:40:01.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:40:01.492 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_pools": [ 2024-04-12T03:40:01.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:40:01.492 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "default-placement", 2024-04-12T03:40:01.492 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:40:01.492 INFO:teuthology.orchestra.run.smithi043.stdout: "index_pool": "a2.rgw.buckets.index", 2024-04-12T03:40:01.492 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": { 2024-04-12T03:40:01.492 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD": { 2024-04-12T03:40:01.492 INFO:teuthology.orchestra.run.smithi043.stdout: "data_pool": "a2.rgw.buckets.data", 2024-04-12T03:40:01.492 INFO:teuthology.orchestra.run.smithi043.stdout: "compression_type": "random" 2024-04-12T03:40:01.493 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:40:01.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:40:01.493 INFO:teuthology.orchestra.run.smithi043.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-04-12T03:40:01.493 INFO:teuthology.orchestra.run.smithi043.stdout: "index_type": 0, 2024-04-12T03:40:01.493 INFO:teuthology.orchestra.run.smithi043.stdout: "inline_data": true 2024-04-12T03:40:01.493 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:40:01.493 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:40:01.493 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:40:01.493 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:40:01.493 INFO:teuthology.orchestra.run.smithi043.stdout: "notif_pool": "a2.rgw.log:notif" 2024-04-12T03:40:01.493 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:40:01.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:40:01.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:40:01.495 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:40:01.549 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:40:01.550 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:40:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:Sending period to new master zone 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61 2024-04-12T03:40:01.627 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:40:01.627 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:40:01.627 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 8, 2024-04-12T03:40:01.628 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T03:40:01.628 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T03:40:01.628 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T03:40:01.628 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:40:01.628 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T03:40:01.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:40:01.628 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T03:40:01.628 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T03:40:01.628 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:40:01.628 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T03:40:01.628 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:40:01.628 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:40:01.629 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:40:01.629 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:40:01.629 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:40:01.629 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:40:01.629 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:40:01.629 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:40:01.629 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:40:01.629 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T03:40:01.629 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:40:01.629 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T03:40:01.629 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:40:01.630 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:40:01.630 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:40:01.630 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:40:01.630 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:40:01.630 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:40:01.630 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:40:01.630 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:40:01.630 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:40:01.630 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:40:01.630 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:40:01.630 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:40:01.630 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:40:01.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:40:01.631 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:40:01.631 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:40:01.631 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T03:40:01.631 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:40:01.631 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:40:01.631 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:40:01.631 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:40:01.631 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:40:01.631 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:40:01.631 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:40:01.631 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:40:01.631 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:40:01.632 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:40:01.632 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:40:01.632 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:40:01.632 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:40:01.632 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:40:01.632 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:40:01.632 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:40:01.632 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:40:01.632 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:40:01.632 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:40:01.632 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:40:01.632 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:40:01.632 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:40:01.633 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:40:01.633 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:40:01.633 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:40:01.633 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:40:01.633 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:40:01.633 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:40:01.633 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:40:01.633 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:40:01.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:40:01.633 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:40:01.633 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:40:01.633 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:40:01.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:40:01.634 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:40:01.634 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T03:40:01.634 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T03:40:01.634 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:40:01.634 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T03:40:01.634 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:40:01.634 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:40:01.634 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:40:01.634 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:40:01.634 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:40:01.634 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:40:01.634 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:40:01.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:40:01.635 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:40:01.635 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T03:40:01.635 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:40:01.635 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:40:01.635 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:40:01.635 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:40:01.635 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:40:01.635 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:40:01.635 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:40:01.635 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:40:01.635 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:40:01.635 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:40:01.636 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:40:01.636 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:40:01.636 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:40:01.641 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:40:01.641 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:40:01.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:40:01.641 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:40:01.641 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:40:01.641 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T03:40:01.641 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:40:01.642 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T03:40:01.642 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:40:01.642 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:40:01.642 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:40:01.642 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:40:01.642 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:40:01.642 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:40:01.642 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:40:01.642 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:40:01.642 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:40:01.642 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:40:01.643 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:40:01.643 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:40:01.643 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:40:01.643 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:40:01.643 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:40:01.643 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:40:01.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:40:01.643 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:40:01.643 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:40:01.643 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:40:01.643 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:40:01.643 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:40:01.644 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:40:01.644 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:40:01.644 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:40:01.644 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:40:01.644 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:40:01.644 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:40:01.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:40:01.644 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:40:01.644 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:40:01.644 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:40:01.644 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:40:01.644 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:40:01.645 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T03:40:01.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:40:01.645 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:40:01.645 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T03:40:01.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:40:01.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:40:01.645 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:40:01.645 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T03:40:01.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:40:01.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:40:01.645 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:40:01.646 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T03:40:01.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:40:01.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:40:01.646 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:40:01.646 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T03:40:01.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:40:01.646 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:40:01.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:40:01.646 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T03:40:01.646 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:40:01.646 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T03:40:01.646 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T03:40:01.647 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:40:01.647 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:40:01.647 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:40:01.647 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:40:01.647 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:40:01.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:40:01.647 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T03:40:01.647 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:40:01.647 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:40:01.647 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:40:01.647 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:40:01.648 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:40:01.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:40:01.648 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T03:40:01.648 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:40:01.648 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:40:01.648 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:40:01.648 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:40:01.648 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:40:01.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:40:01.648 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T03:40:01.648 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:40:01.648 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:40:01.648 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:40:01.649 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:40:01.649 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:40:01.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:40:01.649 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T03:40:01.649 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:40:01.649 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:40:01.649 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:40:01.649 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:40:01.649 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:40:01.649 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:40:01.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:40:01.649 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:40:01.649 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T03:40:01.650 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:40:01.650 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:40:01.651 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "" 2024-04-12T03:41:01.652 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2024-04-12T03:41:01.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:01.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:01.655 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:41:01.714 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:01.714 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:01.751 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:41:01.765 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:41:01.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:01.766 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:41:01.766 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:41:01.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:01.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:01.766 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:41:01.766 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:41:01.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:01.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:01.767 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:41:01.767 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:41:01.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:01.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:01.767 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:41:01.767 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:37:29.736308Z" 2024-04-12T03:41:01.767 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:01.767 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:41:01.768 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:41:01.768 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:41:01.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:01.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:01.769 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:41:01.824 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:01.824 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:01.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:01.856+0000 7fa08a072780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:01.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:01.857+0000 7fa08a072780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:01.873 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:01.873 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:41:01.873 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:41:01.873 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:41:01.874 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:41:01.874 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:01.874 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:01.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:01.874 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:41:01.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:01.874 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:41:01.874 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:01.875 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:01.875 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:41:01.875 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:01.875 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:41:01.875 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:01.875 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:41:01.875 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:01.875 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:01.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:01.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:01.875 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:41:01.875 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:01.876 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:01.876 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:41:01.876 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:01.876 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:01.876 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:01.876 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:41:01.876 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:01.876 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:01.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:01.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:01.876 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:41:01.876 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:01.877 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:01.877 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:41:01.877 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:01.877 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:01.877 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:01.877 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:41:01.877 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:01.877 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:01.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:01.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:01.878 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:41:01.878 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:01.878 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:01.878 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:41:01.878 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:01.878 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:01.878 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:01.878 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:41:01.878 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:01.878 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:01.878 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:01.879 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:01.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:01.879 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:41:01.879 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:41:01.879 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:41:01.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:01.879 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:41:01Z" 2024-04-12T03:41:01.879 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:01.880 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:41:01Z" } 2024-04-12T03:41:01.880 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:41:01.880 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:41:01.881 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:41:01.881 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:41:01.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:01.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', '6ESEI28JZW8UW9BY', '--secret', 'oVIj7wCADBnecpdBF65ZyUvXEbg14PDu', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:01.881 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --access-key 6ESEI28JZW8UW9BY --secret oVIj7wCADBnecpdBF65ZyUvXEbg14PDu --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:41:01.938 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:01.938 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:01.956 INFO:teuthology.orchestra.run.smithi043.stderr:NOTICE: set zone's realm_id=a55da930-a452-44e6-b1f3-2c34e822e09c 2024-04-12T03:41:01.972 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:01.972 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "dd685840-3ecb-4d91-90df-1d645af583cc", 2024-04-12T03:41:01.973 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "remove", 2024-04-12T03:41:01.973 INFO:teuthology.orchestra.run.smithi043.stdout: "domain_root": "remove.rgw.meta:root", 2024-04-12T03:41:01.973 INFO:teuthology.orchestra.run.smithi043.stdout: "control_pool": "remove.rgw.control", 2024-04-12T03:41:01.973 INFO:teuthology.orchestra.run.smithi043.stdout: "gc_pool": "remove.rgw.log:gc", 2024-04-12T03:41:01.973 INFO:teuthology.orchestra.run.smithi043.stdout: "lc_pool": "remove.rgw.log:lc", 2024-04-12T03:41:01.973 INFO:teuthology.orchestra.run.smithi043.stdout: "log_pool": "remove.rgw.log", 2024-04-12T03:41:01.973 INFO:teuthology.orchestra.run.smithi043.stdout: "intent_log_pool": "remove.rgw.log:intent", 2024-04-12T03:41:01.973 INFO:teuthology.orchestra.run.smithi043.stdout: "usage_log_pool": "remove.rgw.log:usage", 2024-04-12T03:41:01.973 INFO:teuthology.orchestra.run.smithi043.stdout: "roles_pool": "remove.rgw.meta:roles", 2024-04-12T03:41:01.973 INFO:teuthology.orchestra.run.smithi043.stdout: "reshard_pool": "remove.rgw.log:reshard", 2024-04-12T03:41:01.973 INFO:teuthology.orchestra.run.smithi043.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2024-04-12T03:41:01.974 INFO:teuthology.orchestra.run.smithi043.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2024-04-12T03:41:01.974 INFO:teuthology.orchestra.run.smithi043.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2024-04-12T03:41:01.974 INFO:teuthology.orchestra.run.smithi043.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2024-04-12T03:41:01.974 INFO:teuthology.orchestra.run.smithi043.stdout: "otp_pool": "remove.rgw.otp", 2024-04-12T03:41:01.974 INFO:teuthology.orchestra.run.smithi043.stdout: "system_key": { 2024-04-12T03:41:01.974 INFO:teuthology.orchestra.run.smithi043.stdout: "access_key": "6ESEI28JZW8UW9BY", 2024-04-12T03:41:01.974 INFO:teuthology.orchestra.run.smithi043.stdout: "secret_key": "oVIj7wCADBnecpdBF65ZyUvXEbg14PDu" 2024-04-12T03:41:01.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:01.974 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_pools": [ 2024-04-12T03:41:01.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:01.974 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "default-placement", 2024-04-12T03:41:01.975 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:01.975 INFO:teuthology.orchestra.run.smithi043.stdout: "index_pool": "remove.rgw.buckets.index", 2024-04-12T03:41:01.975 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": { 2024-04-12T03:41:01.975 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD": { 2024-04-12T03:41:01.975 INFO:teuthology.orchestra.run.smithi043.stdout: "data_pool": "remove.rgw.buckets.data" 2024-04-12T03:41:01.975 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:01.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:01.975 INFO:teuthology.orchestra.run.smithi043.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2024-04-12T03:41:01.975 INFO:teuthology.orchestra.run.smithi043.stdout: "index_type": 0, 2024-04-12T03:41:01.975 INFO:teuthology.orchestra.run.smithi043.stdout: "inline_data": true 2024-04-12T03:41:01.975 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:01.975 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:01.976 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:01.976 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:01.976 INFO:teuthology.orchestra.run.smithi043.stdout: "notif_pool": "remove.rgw.log:notif" 2024-04-12T03:41:01.976 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:01.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:01.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:01.977 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:41:02.031 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:02.031 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:02.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:02.044+0000 7fce655b1780 0 period (b69a664f-970b-4d72-be0d-0071b991ee5c does not have zone dd685840-3ecb-4d91-90df-1d645af583cc configured 2024-04-12T03:41:08.312 INFO:teuthology.orchestra.run.smithi043.stderr:Sending period to new master zone 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61 2024-04-12T03:41:08.341 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:08.341 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:08.341 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9, 2024-04-12T03:41:08.341 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T03:41:08.341 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T03:41:08.341 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T03:41:08.342 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:08.342 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T03:41:08.342 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.342 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T03:41:08.342 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T03:41:08.342 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:41:08.342 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T03:41:08.342 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:08.342 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:41:08.343 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.343 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:41:08.343 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:41:08.343 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:08.343 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:41:08.343 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.343 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:41:08.344 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T03:41:08.344 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:08.344 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T03:41:08.344 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.344 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:08.345 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:08.345 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:08.345 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:08.345 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:08.345 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:08.345 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:08.346 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:08.346 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:08.346 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:08.346 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:08.346 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.346 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.346 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:08.347 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T03:41:08.347 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:08.347 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:41:08.347 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.347 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:08.347 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:08.347 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:08.348 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:08.348 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:08.348 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:08.348 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:08.348 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:08.348 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:08.348 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:08.348 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:08.349 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.349 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.349 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "dd685840-3ecb-4d91-90df-1d645af583cc", 2024-04-12T03:41:08.349 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "remove", 2024-04-12T03:41:08.349 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [], 2024-04-12T03:41:08.349 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:08.349 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:08.349 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:08.350 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:08.350 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:08.350 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:08.350 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:08.350 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:08.350 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:08.350 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:08.350 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:08.350 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.350 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:08.350 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.351 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:41:08.351 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.351 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:41:08.351 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:41:08.351 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:41:08.351 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:41:08.351 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.351 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:08.351 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.351 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:41:08.351 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:08.351 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:41:08.351 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:41:08.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.352 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:41:08.352 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:08.352 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.352 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.352 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T03:41:08.352 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T03:41:08.352 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:41:08.352 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T03:41:08.352 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:08.352 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:41:08.352 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.353 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:41:08.353 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:41:08.353 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:41:08.353 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:41:08.353 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.353 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:41:08.353 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T03:41:08.353 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:08.353 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:41:08.353 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.353 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:08.353 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:08.354 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:08.354 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:08.354 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:08.354 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:08.354 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:08.354 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:08.354 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:08.354 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:08.354 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:08.355 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.355 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.355 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.355 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:41:08.355 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T03:41:08.355 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:08.355 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T03:41:08.355 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.355 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:08.355 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:08.355 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:08.355 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:08.355 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:08.356 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:08.356 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:08.356 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:08.356 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:08.356 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:08.356 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:08.356 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.356 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:08.356 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.356 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:41:08.356 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.356 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:41:08.357 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:41:08.357 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:41:08.357 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:41:08.357 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.357 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:08.357 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.357 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:41:08.357 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:08.357 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:41:08.357 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:41:08.357 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.357 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:41:08.358 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:08.358 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.358 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:08.358 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.358 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T03:41:08.358 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.358 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:41:08.358 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T03:41:08.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.359 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:41:08.359 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T03:41:08.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.359 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:41:08.359 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T03:41:08.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.359 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:08.360 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T03:41:08.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.360 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.360 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "dd685840-3ecb-4d91-90df-1d645af583cc", 2024-04-12T03:41:08.360 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 853955540 2024-04-12T03:41:08.360 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:08.360 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.360 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T03:41:08.360 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:08.360 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T03:41:08.360 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T03:41:08.361 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:41:08.361 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:41:08.361 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:41:08.361 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:41:08.361 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:41:08.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.361 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T03:41:08.361 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:41:08.361 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:41:08.361 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:41:08.361 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:41:08.361 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:41:08.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.362 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T03:41:08.362 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:41:08.362 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:41:08.362 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:41:08.362 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:41:08.362 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:41:08.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.362 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T03:41:08.362 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:41:08.362 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:41:08.363 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:41:08.363 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:41:08.363 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:41:08.363 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.363 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T03:41:08.363 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:41:08.363 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:41:08.363 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:41:08.363 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:41:08.363 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:41:08.364 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:08.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.364 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:08.365 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T03:41:08.365 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:08.365 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:08.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:08.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:08.365 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:41:08.425 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:08.425 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:08.460 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:08.460 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T03:41:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T03:41:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:41:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T03:41:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:41:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:41:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:41:08.461 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:41:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:41:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T03:41:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T03:41:08.462 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:08.463 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T03:41:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:41:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:08.464 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.465 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:41:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:41:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:41:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:41:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:41:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:41:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:41:08.466 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:41:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:41:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:08.467 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.467 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:08.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:08.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:08.468 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:41:08.535 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:08.535 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:08.590 INFO:teuthology.orchestra.run.smithi043.stderr:zone name remove not found in zonegroup a 2024-04-12T03:41:08.601 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-12T03:41:08.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:08.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:08.602 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:41:08.684 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:08.684 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:08.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:08.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:08.781 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:41:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:08.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:08.899+0000 7fa19ec2f780 0 WARNING: can't generate connection for zone dd685840-3ecb-4d91-90df-1d645af583cc id remove: no endpoints defined 2024-04-12T03:41:08.969 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:08.969 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:08.970 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 10, 2024-04-12T03:41:08.970 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T03:41:08.970 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T03:41:08.970 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T03:41:08.970 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:08.970 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T03:41:08.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.970 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T03:41:08.970 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T03:41:08.970 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:41:08.971 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T03:41:08.971 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:08.971 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:41:08.971 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.971 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:41:08.971 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:41:08.971 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:08.971 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:41:08.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.971 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:41:08.971 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T03:41:08.971 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:08.972 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T03:41:08.972 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.972 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:08.972 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:08.972 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:08.972 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:08.972 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:08.972 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:08.972 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:08.972 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:08.972 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:08.972 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:08.973 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:08.973 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.973 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:08.973 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T03:41:08.973 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:08.973 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:41:08.973 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.973 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:08.973 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:08.973 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:08.974 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:08.974 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:08.974 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:08.974 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:08.974 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:08.974 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:08.974 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:08.974 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:08.974 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.974 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:08.974 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.975 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:41:08.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.975 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:41:08.975 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:41:08.975 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:41:08.975 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:41:08.975 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.975 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:08.975 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.975 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:41:08.975 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:08.976 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:41:08.976 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:41:08.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.976 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:41:08.976 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:08.976 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.976 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T03:41:08.976 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T03:41:08.976 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:41:08.977 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T03:41:08.977 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:08.977 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:41:08.977 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.977 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:41:08.977 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:41:08.977 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:41:08.977 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:41:08.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.977 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:41:08.977 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T03:41:08.978 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:08.978 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:41:08.978 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.978 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:08.978 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:08.978 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:08.978 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:08.978 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:08.978 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:08.978 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:08.978 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:08.979 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:08.979 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:08.988 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:08.988 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.988 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:41:08.988 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T03:41:08.988 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:08.988 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T03:41:08.988 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.988 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:08.988 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:08.988 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:08.989 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:08.989 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:08.989 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:08.989 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:08.989 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:08.989 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:08.989 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:08.989 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:08.989 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.989 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:08.989 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.989 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:41:08.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.990 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:41:08.990 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:41:08.990 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:41:08.990 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:41:08.990 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.990 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:08.990 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.990 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:41:08.990 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:08.990 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:41:08.990 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:41:08.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.991 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:41:08.991 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:08.991 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.991 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:08.991 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:08.991 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T03:41:08.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.991 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:41:08.991 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T03:41:08.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.992 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:41:08.992 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T03:41:08.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.992 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:41:08.992 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T03:41:08.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:08.992 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:08.992 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T03:41:08.992 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:08.992 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:08.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.993 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T03:41:08.993 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:08.993 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T03:41:08.993 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T03:41:08.993 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:41:08.993 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:41:08.993 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:41:08.993 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:41:08.993 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:41:08.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.993 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T03:41:08.994 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:41:08.994 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:41:08.994 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:41:08.994 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:41:08.994 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:41:08.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.994 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T03:41:08.994 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:41:08.994 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:41:08.994 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:41:08.994 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:41:08.995 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:41:08.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.995 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T03:41:08.995 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:41:08.995 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:41:08.995 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:41:08.995 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:41:08.995 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:41:08.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.995 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T03:41:08.995 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:41:08.995 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:41:08.996 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:41:08.996 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:41:08.996 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:41:08.996 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:08.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:08.996 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:08.996 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T03:41:08.996 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:08.996 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:08.997 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2024-04-12T03:41:08.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:08.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:08.998 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:41:09.064 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:09.064 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:09.098 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:09.098 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "d8c1d810-e2c8-4c75-a0aa-befec4669c98", 2024-04-12T03:41:09.098 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "remove_zg", 2024-04-12T03:41:09.098 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "remove_zg", 2024-04-12T03:41:09.098 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T03:41:09.099 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [], 2024-04-12T03:41:09.099 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:41:09.099 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:41:09.099 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "", 2024-04-12T03:41:09.099 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [], 2024-04-12T03:41:09.099 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:41:09.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.099 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:41:09.099 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:41:09.099 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [] 2024-04-12T03:41:09.099 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:09.099 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:09.100 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:41:09.100 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:09.100 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:41:09.100 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:41:09.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.100 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:41:09.100 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:09.100 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:09.100 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:09.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:09.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:09.101 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:41:09.169 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:09.169 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:09.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:09.221+0000 7f7931066780 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2024-04-12T03:41:09.230 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:09.230 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "68067140-307d-43bd-9409-e52009a68f44", 2024-04-12T03:41:09.230 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "remove", 2024-04-12T03:41:09.230 INFO:teuthology.orchestra.run.smithi043.stdout: "domain_root": "remove.rgw.meta:root", 2024-04-12T03:41:09.230 INFO:teuthology.orchestra.run.smithi043.stdout: "control_pool": "remove.rgw.control", 2024-04-12T03:41:09.230 INFO:teuthology.orchestra.run.smithi043.stdout: "gc_pool": "remove.rgw.log:gc", 2024-04-12T03:41:09.230 INFO:teuthology.orchestra.run.smithi043.stdout: "lc_pool": "remove.rgw.log:lc", 2024-04-12T03:41:09.230 INFO:teuthology.orchestra.run.smithi043.stdout: "log_pool": "remove.rgw.log", 2024-04-12T03:41:09.230 INFO:teuthology.orchestra.run.smithi043.stdout: "intent_log_pool": "remove.rgw.log:intent", 2024-04-12T03:41:09.230 INFO:teuthology.orchestra.run.smithi043.stdout: "usage_log_pool": "remove.rgw.log:usage", 2024-04-12T03:41:09.231 INFO:teuthology.orchestra.run.smithi043.stdout: "roles_pool": "remove.rgw.meta:roles", 2024-04-12T03:41:09.231 INFO:teuthology.orchestra.run.smithi043.stdout: "reshard_pool": "remove.rgw.log:reshard", 2024-04-12T03:41:09.231 INFO:teuthology.orchestra.run.smithi043.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2024-04-12T03:41:09.231 INFO:teuthology.orchestra.run.smithi043.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2024-04-12T03:41:09.231 INFO:teuthology.orchestra.run.smithi043.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2024-04-12T03:41:09.231 INFO:teuthology.orchestra.run.smithi043.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2024-04-12T03:41:09.231 INFO:teuthology.orchestra.run.smithi043.stdout: "otp_pool": "remove.rgw.otp", 2024-04-12T03:41:09.231 INFO:teuthology.orchestra.run.smithi043.stdout: "system_key": { 2024-04-12T03:41:09.231 INFO:teuthology.orchestra.run.smithi043.stdout: "access_key": "", 2024-04-12T03:41:09.231 INFO:teuthology.orchestra.run.smithi043.stdout: "secret_key": "" 2024-04-12T03:41:09.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.232 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_pools": [ 2024-04-12T03:41:09.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.232 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "default-placement", 2024-04-12T03:41:09.232 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:09.232 INFO:teuthology.orchestra.run.smithi043.stdout: "index_pool": "remove.rgw.buckets.index", 2024-04-12T03:41:09.232 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": { 2024-04-12T03:41:09.232 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD": { 2024-04-12T03:41:09.232 INFO:teuthology.orchestra.run.smithi043.stdout: "data_pool": "remove.rgw.buckets.data" 2024-04-12T03:41:09.232 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:09.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.232 INFO:teuthology.orchestra.run.smithi043.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2024-04-12T03:41:09.232 INFO:teuthology.orchestra.run.smithi043.stdout: "index_type": 0, 2024-04-12T03:41:09.233 INFO:teuthology.orchestra.run.smithi043.stdout: "inline_data": true 2024-04-12T03:41:09.233 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:09.233 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:09.233 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:09.233 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:09.233 INFO:teuthology.orchestra.run.smithi043.stdout: "notif_pool": "remove.rgw.log:notif" 2024-04-12T03:41:09.233 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:09.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:09.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:09.234 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:41:09.302 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:09.302 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:09.461 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:09.461 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:09.461 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 11, 2024-04-12T03:41:09.461 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T03:41:09.461 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T03:41:09.461 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T03:41:09.461 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:09.462 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T03:41:09.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.462 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T03:41:09.462 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T03:41:09.462 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:41:09.462 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T03:41:09.462 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:09.462 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:41:09.462 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:09.462 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:41:09.462 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:41:09.463 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:09.463 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:41:09.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.463 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:41:09.463 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T03:41:09.463 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:09.463 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T03:41:09.463 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:09.463 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:09.463 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:09.463 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:09.463 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:09.464 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:09.464 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:09.464 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:09.464 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:09.464 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:09.464 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:09.464 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:09.464 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:09.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.464 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:09.465 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T03:41:09.465 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:09.465 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:41:09.465 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:09.465 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:09.465 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:09.465 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:09.465 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:09.465 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:09.465 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:09.465 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:09.465 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:09.466 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:09.466 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:09.466 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:09.466 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:09.466 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:09.466 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:09.466 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:41:09.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.466 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:41:09.466 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:41:09.466 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:41:09.467 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:41:09.467 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:09.467 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:09.467 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:09.467 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:41:09.467 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:09.467 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:41:09.467 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:41:09.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.467 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:41:09.467 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:09.467 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:09.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.468 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T03:41:09.468 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T03:41:09.468 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:41:09.468 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T03:41:09.468 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:09.468 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:41:09.468 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:09.468 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:41:09.468 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:41:09.468 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:41:09.468 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:41:09.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.469 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:41:09.469 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T03:41:09.469 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:09.469 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:41:09.469 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:09.469 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:09.469 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:09.469 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:09.469 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:09.469 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:09.470 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:09.470 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:09.470 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:09.470 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:09.470 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:09.470 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:09.470 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:09.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.470 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:41:09.470 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T03:41:09.470 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:09.471 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T03:41:09.471 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:09.471 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:09.471 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:09.471 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:09.471 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:09.471 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:09.471 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:09.471 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:09.471 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:09.471 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:09.471 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:09.472 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:09.472 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:09.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:09.472 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:09.472 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:41:09.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.472 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:41:09.472 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:41:09.472 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:41:09.472 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:41:09.472 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:09.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:09.473 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:09.473 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:41:09.473 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:09.473 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:41:09.473 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:41:09.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.473 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:41:09.473 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:09.473 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:09.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.473 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "d8c1d810-e2c8-4c75-a0aa-befec4669c98", 2024-04-12T03:41:09.474 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "remove_zg", 2024-04-12T03:41:09.474 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "remove_zg", 2024-04-12T03:41:09.474 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T03:41:09.474 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [], 2024-04-12T03:41:09.474 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:41:09.474 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:41:09.474 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "68067140-307d-43bd-9409-e52009a68f44", 2024-04-12T03:41:09.474 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:41:09.474 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.474 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "68067140-307d-43bd-9409-e52009a68f44", 2024-04-12T03:41:09.474 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "remove", 2024-04-12T03:41:09.474 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [], 2024-04-12T03:41:09.475 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:09.475 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": false, 2024-04-12T03:41:09.475 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:09.475 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:09.475 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:09.475 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:09.475 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:09.475 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:09.475 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:09.475 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:09.475 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:09.475 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:09.476 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:09.476 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:09.476 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:41:09.476 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.476 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:41:09.476 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:41:09.476 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:41:09.476 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:41:09.476 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:09.476 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:09.476 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:09.476 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:41:09.477 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:09.477 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:41:09.477 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:41:09.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.477 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:41:09.477 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:09.477 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:09.477 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:09.477 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:09.477 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T03:41:09.477 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.477 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:41:09.478 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T03:41:09.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.478 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.478 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:41:09.478 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T03:41:09.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.478 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.478 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:41:09.478 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T03:41:09.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.478 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.478 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:09.479 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T03:41:09.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.479 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:09.479 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "68067140-307d-43bd-9409-e52009a68f44", 2024-04-12T03:41:09.479 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 1525166125 2024-04-12T03:41:09.479 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:09.479 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:09.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.479 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T03:41:09.479 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:09.479 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T03:41:09.498 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T03:41:09.498 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:41:09.498 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:41:09.498 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:41:09.498 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:41:09.498 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:41:09.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.498 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T03:41:09.499 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:41:09.499 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:41:09.499 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:41:09.499 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:41:09.499 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:41:09.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.499 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T03:41:09.499 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:41:09.499 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:41:09.499 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:41:09.499 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:41:09.499 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:41:09.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.500 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T03:41:09.500 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:41:09.500 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:41:09.500 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:41:09.500 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:41:09.500 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:41:09.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.500 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T03:41:09.500 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:41:09.500 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:41:09.500 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:41:09.501 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:41:09.501 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:41:09.501 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:09.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:09.501 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:09.501 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T03:41:09.501 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:09.501 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:09.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:09.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:09.502 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:41:09.579 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:09.579 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:09.635+0000 7ff0e5aab780 0 NOTICE: removed master_zone of zonegroup remove_zg 2024-04-12T03:41:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:09.639+0000 7ff0e5aab780 0 Removed zone from zonegroup remove_zg 2024-04-12T03:41:09.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:09.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:09.664 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:41:09.741 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:09.741 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:09.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:09.780+0000 7f250a8eb780 0 ERROR: zonegroup remove_zg has a non existent master zone 2024-04-12T03:41:09.787 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-12T03:41:09.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:09.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:09.788 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:41:09.853 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:09.853 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:09.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:09.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:41:09.891 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:41:09.960 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:09.961 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:10.057 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:10.058 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:10.058 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12, 2024-04-12T03:41:10.058 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T03:41:10.058 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T03:41:10.058 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T03:41:10.058 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:10.058 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T03:41:10.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:10.058 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T03:41:10.058 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T03:41:10.059 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:41:10.059 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T03:41:10.059 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:10.059 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:41:10.059 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:10.059 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:41:10.059 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:41:10.059 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:10.059 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:41:10.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:10.059 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:41:10.060 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T03:41:10.060 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:10.060 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T03:41:10.060 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:10.060 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:10.060 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:10.060 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:10.060 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:10.060 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:10.060 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:10.060 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:10.061 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:10.061 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:10.061 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:10.061 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:10.061 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:10.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:10.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:10.061 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:10.061 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T03:41:10.061 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:10.061 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T03:41:10.061 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:10.062 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:10.062 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:10.062 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:10.062 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:10.062 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:10.062 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:10.062 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:10.062 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:10.062 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:10.062 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:10.062 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:10.063 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:10.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:10.063 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:10.063 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:41:10.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:10.063 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:41:10.063 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:41:10.063 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:41:10.063 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:41:10.063 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:10.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:10.063 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:10.064 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:41:10.064 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:10.064 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:41:10.064 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:41:10.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:10.064 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:41:10.064 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:10.064 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:10.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:10.064 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:10.064 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T03:41:10.065 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T03:41:10.065 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T03:41:10.065 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T03:41:10.065 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:10.065 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:41:10.065 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:10.065 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T03:41:10.065 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T03:41:10.065 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:41:10.065 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T03:41:10.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:10.065 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:41:10.066 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T03:41:10.066 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:10.066 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T03:41:10.066 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:10.066 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:10.066 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:10.066 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:10.066 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:10.066 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:10.066 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:10.066 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:10.067 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:10.067 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:10.067 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:41:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T03:41:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T03:41:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T03:41:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T03:41:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T03:41:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T03:41:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T03:41:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T03:41:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T03:41:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T03:41:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T03:41:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T03:41:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T03:41:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T03:41:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T03:41:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T03:41:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T03:41:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T03:41:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T03:41:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T03:41:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T03:41:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T03:41:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T03:41:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T03:41:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T03:41:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T03:41:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T03:41:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:41:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T03:41:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T03:41:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T03:41:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T03:41:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T03:41:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:41:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T03:41:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T03:41:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:41:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:41:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:41:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:41:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:41:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T03:41:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T03:41:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T03:41:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T03:41:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T03:41:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T03:41:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T03:41:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:41:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:41:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:41:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:41:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:41:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T03:41:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:41:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:41:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:41:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:41:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:41:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T03:41:10.101 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T03:41:10.101 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T03:41:10.101 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T03:41:10.101 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T03:41:10.101 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T03:41:10.101 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:10.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:10.101 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T03:41:10.101 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T03:41:10.101 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:10.101 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:10.102 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2024-04-12T03:41:10.103 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-38 2024-04-12T03:41:10.103 DEBUG:boto:path=/dkyges-38/ 2024-04-12T03:41:10.103 DEBUG:boto:auth_path=/dkyges-38/ 2024-04-12T03:41:10.103 DEBUG:boto:Method: PUT 2024-04-12T03:41:10.104 DEBUG:boto:Path: /dkyges-38/ 2024-04-12T03:41:10.104 DEBUG:boto:Data: 2024-04-12T03:41:10.104 DEBUG:boto:Headers: {} 2024-04-12T03:41:10.104 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:41:10.104 DEBUG:boto:Port: 8000 2024-04-12T03:41:10.104 DEBUG:boto:Params: {} 2024-04-12T03:41:10.105 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:41:10.105 DEBUG:boto:Token: None 2024-04-12T03:41:10.105 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:41:10 GMT /dkyges-38/ 2024-04-12T03:41:10.105 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6lyKCx0yVXfpSWrbsUZ7wLNNiYA= 2024-04-12T03:41:10.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6lyKCx0yVXfpSWrbsUZ7wLNNiYA=', 'Content-Length': '0'} 2024-04-12T03:41:10.174 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000584e97ebbe770f65-006618ad56-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:10.174 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-39 2024-04-12T03:41:10.174 DEBUG:boto:path=/dkyges-39/ 2024-04-12T03:41:10.174 DEBUG:boto:auth_path=/dkyges-39/ 2024-04-12T03:41:10.174 DEBUG:boto:Method: PUT 2024-04-12T03:41:10.174 DEBUG:boto:Path: /dkyges-39/ 2024-04-12T03:41:10.174 DEBUG:boto:Data: 2024-04-12T03:41:10.174 DEBUG:boto:Headers: {} 2024-04-12T03:41:10.174 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:41:10.175 DEBUG:boto:Port: 8001 2024-04-12T03:41:10.175 DEBUG:boto:Params: {} 2024-04-12T03:41:10.175 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:41:10.175 DEBUG:boto:Token: None 2024-04-12T03:41:10.175 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:41:10 GMT /dkyges-39/ 2024-04-12T03:41:10.175 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TEtsuQnFl96TJp7qUP+o0tIxARM= 2024-04-12T03:41:10.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TEtsuQnFl96TJp7qUP+o0tIxARM=', 'Content-Length': '0'} 2024-04-12T03:41:10.300 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000121632d8966fd048-006618ad56-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:10.301 INFO:rgw_multi.tests:create bucket zone=b1 name=dkyges-40 2024-04-12T03:41:10.301 DEBUG:boto:path=/dkyges-40/ 2024-04-12T03:41:10.301 DEBUG:boto:auth_path=/dkyges-40/ 2024-04-12T03:41:10.301 DEBUG:boto:Method: PUT 2024-04-12T03:41:10.301 DEBUG:boto:Path: /dkyges-40/ 2024-04-12T03:41:10.301 DEBUG:boto:Data: 2024-04-12T03:41:10.301 DEBUG:boto:Headers: {} 2024-04-12T03:41:10.301 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8000 2024-04-12T03:41:10.301 DEBUG:boto:Port: 8000 2024-04-12T03:41:10.301 DEBUG:boto:Params: {} 2024-04-12T03:41:10.302 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:41:10.302 DEBUG:boto:Token: None 2024-04-12T03:41:10.302 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:41:10 GMT /dkyges-40/ 2024-04-12T03:41:10.302 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KwvrNSFrn9fdKA7HTri8Mu0CzuI= 2024-04-12T03:41:10.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KwvrNSFrn9fdKA7HTri8Mu0CzuI=', 'Content-Length': '0'} 2024-04-12T03:41:10.812 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000096b18769679e871c-006618ad56-1157-b1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:10.813 INFO:rgw_multi.tests:create bucket zone=b2 name=dkyges-41 2024-04-12T03:41:10.813 DEBUG:boto:path=/dkyges-41/ 2024-04-12T03:41:10.813 DEBUG:boto:auth_path=/dkyges-41/ 2024-04-12T03:41:10.813 DEBUG:boto:Method: PUT 2024-04-12T03:41:10.813 DEBUG:boto:Path: /dkyges-41/ 2024-04-12T03:41:10.813 DEBUG:boto:Data: 2024-04-12T03:41:10.813 DEBUG:boto:Headers: {} 2024-04-12T03:41:10.813 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8001 2024-04-12T03:41:10.813 DEBUG:boto:Port: 8001 2024-04-12T03:41:10.814 DEBUG:boto:Params: {} 2024-04-12T03:41:10.814 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:41:10.814 DEBUG:boto:Token: None 2024-04-12T03:41:10.814 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:41:10 GMT /dkyges-41/ 2024-04-12T03:41:10.814 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mVWbbOLTaheBTnA+64pxtsON87g= 2024-04-12T03:41:10.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mVWbbOLTaheBTnA+64pxtsON87g=', 'Content-Length': '0'} 2024-04-12T03:41:11.201 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e69d66bcb515e6ac-006618ad57-115b-b2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:11.202 DEBUG:boto:path=/dkyges-38/ 2024-04-12T03:41:11.202 DEBUG:boto:auth_path=/dkyges-38/ 2024-04-12T03:41:11.202 DEBUG:boto:path=/dkyges-38/?website 2024-04-12T03:41:11.202 DEBUG:boto:auth_path=/dkyges-38/?website 2024-04-12T03:41:11.202 DEBUG:boto:Method: PUT 2024-04-12T03:41:11.202 DEBUG:boto:Path: /dkyges-38/?website 2024-04-12T03:41:11.202 DEBUG:boto:Data: index.htmlerror.html 2024-04-12T03:41:11.202 DEBUG:boto:Headers: {} 2024-04-12T03:41:11.202 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:41:11.202 DEBUG:boto:Port: 8000 2024-04-12T03:41:11.202 DEBUG:boto:Params: {} 2024-04-12T03:41:11.203 DEBUG:boto:Token: None 2024-04-12T03:41:11.203 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:41:11 GMT /dkyges-38/?website 2024-04-12T03:41:11.203 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RM7sGeN693KNZce93BnrQwo0K7E= 2024-04-12T03:41:11.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:11 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RM7sGeN693KNZce93BnrQwo0K7E=', 'Content-Length': '241'} 2024-04-12T03:41:11.204 DEBUG:boto:Response headers: [('Content-Length', '192'), ('x-amz-request-id', 'tx00000b09aa1c6ad7e1937-006618ad57-16d5-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:41:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:11.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2024-04-12T03:41:11.205 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-42 2024-04-12T03:41:11.205 DEBUG:boto:path=/dkyges-42/ 2024-04-12T03:41:11.205 DEBUG:boto:auth_path=/dkyges-42/ 2024-04-12T03:41:11.206 DEBUG:boto:Method: PUT 2024-04-12T03:41:11.206 DEBUG:boto:Path: /dkyges-42/ 2024-04-12T03:41:11.206 DEBUG:boto:Data: 2024-04-12T03:41:11.206 DEBUG:boto:Headers: {} 2024-04-12T03:41:11.206 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:41:11.206 DEBUG:boto:Port: 8000 2024-04-12T03:41:11.206 DEBUG:boto:Params: {} 2024-04-12T03:41:11.206 DEBUG:boto:Token: None 2024-04-12T03:41:11.206 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:41:11 GMT /dkyges-42/ 2024-04-12T03:41:11.207 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IW+muECDryh41PigtN8z8BXIuMA= 2024-04-12T03:41:11.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:11 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IW+muECDryh41PigtN8z8BXIuMA=', 'Content-Length': '0'} 2024-04-12T03:41:11.224 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dbf7abfdd29e238a-006618ad57-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:11.224 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-43 2024-04-12T03:41:11.224 DEBUG:boto:path=/dkyges-43/ 2024-04-12T03:41:11.224 DEBUG:boto:auth_path=/dkyges-43/ 2024-04-12T03:41:11.225 DEBUG:boto:Method: PUT 2024-04-12T03:41:11.225 DEBUG:boto:Path: /dkyges-43/ 2024-04-12T03:41:11.225 DEBUG:boto:Data: 2024-04-12T03:41:11.225 DEBUG:boto:Headers: {} 2024-04-12T03:41:11.225 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:41:11.225 DEBUG:boto:Port: 8001 2024-04-12T03:41:11.225 DEBUG:boto:Params: {} 2024-04-12T03:41:11.225 DEBUG:boto:Token: None 2024-04-12T03:41:11.225 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:41:11 GMT /dkyges-43/ 2024-04-12T03:41:11.225 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GyfOkIEHKqpKzBald/tXPFA0LRE= 2024-04-12T03:41:11.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:11 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GyfOkIEHKqpKzBald/tXPFA0LRE=', 'Content-Length': '0'} 2024-04-12T03:41:11.258 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003ce72912e4d75a42-006618ad57-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:11.258 INFO:rgw_multi.tests:create bucket zone=b1 name=dkyges-44 2024-04-12T03:41:11.258 DEBUG:boto:path=/dkyges-44/ 2024-04-12T03:41:11.258 DEBUG:boto:auth_path=/dkyges-44/ 2024-04-12T03:41:11.258 DEBUG:boto:Method: PUT 2024-04-12T03:41:11.259 DEBUG:boto:Path: /dkyges-44/ 2024-04-12T03:41:11.259 DEBUG:boto:Data: 2024-04-12T03:41:11.259 DEBUG:boto:Headers: {} 2024-04-12T03:41:11.259 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8000 2024-04-12T03:41:11.259 DEBUG:boto:Port: 8000 2024-04-12T03:41:11.259 DEBUG:boto:Params: {} 2024-04-12T03:41:11.259 DEBUG:boto:Token: None 2024-04-12T03:41:11.259 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:41:11 GMT /dkyges-44/ 2024-04-12T03:41:11.259 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OKwc2c6nHe4q37dM8OcA8DBOFl4= 2024-04-12T03:41:11.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:11 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OKwc2c6nHe4q37dM8OcA8DBOFl4=', 'Content-Length': '0'} 2024-04-12T03:41:11.808 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a6e08328a0065b91-006618ad57-115f-b1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:11.809 INFO:rgw_multi.tests:create bucket zone=b2 name=dkyges-45 2024-04-12T03:41:11.809 DEBUG:boto:path=/dkyges-45/ 2024-04-12T03:41:11.809 DEBUG:boto:auth_path=/dkyges-45/ 2024-04-12T03:41:11.809 DEBUG:boto:Method: PUT 2024-04-12T03:41:11.809 DEBUG:boto:Path: /dkyges-45/ 2024-04-12T03:41:11.809 DEBUG:boto:Data: 2024-04-12T03:41:11.809 DEBUG:boto:Headers: {} 2024-04-12T03:41:11.809 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8001 2024-04-12T03:41:11.809 DEBUG:boto:Port: 8001 2024-04-12T03:41:11.809 DEBUG:boto:Params: {} 2024-04-12T03:41:11.810 DEBUG:boto:Token: None 2024-04-12T03:41:11.810 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:41:11 GMT /dkyges-45/ 2024-04-12T03:41:11.810 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IWq3Whdf08i9Rfe7FoQxRDyi1cc= 2024-04-12T03:41:11.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:11 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IWq3Whdf08i9Rfe7FoQxRDyi1cc=', 'Content-Length': '0'} 2024-04-12T03:41:11.856 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d47fd500d6f67fbf-006618ad57-115b-b2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:11.856 DEBUG:boto:path=/dkyges-42/ 2024-04-12T03:41:11.856 DEBUG:boto:auth_path=/dkyges-42/ 2024-04-12T03:41:11.857 DEBUG:boto:path=/dkyges-42/?policy 2024-04-12T03:41:11.857 DEBUG:boto:auth_path=/dkyges-42/?policy 2024-04-12T03:41:11.857 DEBUG:boto:Method: PUT 2024-04-12T03:41:11.857 DEBUG:boto:Path: /dkyges-42/?policy 2024-04-12T03:41:11.857 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-12T03:41:11.857 DEBUG:boto:Headers: {} 2024-04-12T03:41:11.857 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:41:11.857 DEBUG:boto:Port: 8000 2024-04-12T03:41:11.857 DEBUG:boto:Params: {} 2024-04-12T03:41:11.857 DEBUG:boto:Token: None 2024-04-12T03:41:11.858 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:41:11 GMT /dkyges-42/?policy 2024-04-12T03:41:11.858 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jSLUDOyAIbSV3bVg/I0p+ncj2j8= 2024-04-12T03:41:11.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:11 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jSLUDOyAIbSV3bVg/I0p+ncj2j8=', 'Content-Length': '97'} 2024-04-12T03:41:11.864 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004fc3cb74bfb6c596-006618ad57-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:41:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:11.864 DEBUG:boto:path=/dkyges-42/ 2024-04-12T03:41:11.864 DEBUG:boto:auth_path=/dkyges-42/ 2024-04-12T03:41:11.864 DEBUG:boto:path=/dkyges-42/?policy 2024-04-12T03:41:11.864 DEBUG:boto:auth_path=/dkyges-42/?policy 2024-04-12T03:41:11.864 DEBUG:boto:Method: GET 2024-04-12T03:41:11.865 DEBUG:boto:Path: /dkyges-42/?policy 2024-04-12T03:41:11.865 DEBUG:boto:Data: 2024-04-12T03:41:11.865 DEBUG:boto:Headers: {} 2024-04-12T03:41:11.865 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:41:11.865 DEBUG:boto:Port: 8000 2024-04-12T03:41:11.865 DEBUG:boto:Params: {} 2024-04-12T03:41:11.865 DEBUG:boto:Token: None 2024-04-12T03:41:11.865 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:11 GMT /dkyges-42/?policy 2024-04-12T03:41:11.865 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2z+ZqkZ/ZS8CNGYe/1Py56+8QFc= 2024-04-12T03:41:11.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:11 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2z+ZqkZ/ZS8CNGYe/1Py56+8QFc=', 'Content-Length': '0'} 2024-04-12T03:41:11.867 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009230b89fb0915e19-006618ad57-16d5-a1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 12 Apr 2024 03:41:11 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:11.867 DEBUG:boto:path=/dkyges-43/ 2024-04-12T03:41:11.867 DEBUG:boto:auth_path=/dkyges-43/ 2024-04-12T03:41:11.867 DEBUG:boto:path=/dkyges-43/?policy 2024-04-12T03:41:11.867 DEBUG:boto:auth_path=/dkyges-43/?policy 2024-04-12T03:41:11.867 DEBUG:boto:Method: PUT 2024-04-12T03:41:11.867 DEBUG:boto:Path: /dkyges-43/?policy 2024-04-12T03:41:11.867 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-12T03:41:11.867 DEBUG:boto:Headers: {} 2024-04-12T03:41:11.867 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:41:11.868 DEBUG:boto:Port: 8001 2024-04-12T03:41:11.868 DEBUG:boto:Params: {} 2024-04-12T03:41:11.868 DEBUG:boto:Token: None 2024-04-12T03:41:11.868 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:41:11 GMT /dkyges-43/?policy 2024-04-12T03:41:11.868 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Gja7bn5OUjUSkDljLMW+NMYjxus= 2024-04-12T03:41:11.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:11 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Gja7bn5OUjUSkDljLMW+NMYjxus=', 'Content-Length': '97'} 2024-04-12T03:41:12.878 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000a162302be50b237-006618ad57-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:41:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:12.878 DEBUG:boto:path=/dkyges-43/ 2024-04-12T03:41:12.878 DEBUG:boto:auth_path=/dkyges-43/ 2024-04-12T03:41:12.879 DEBUG:boto:path=/dkyges-43/?policy 2024-04-12T03:41:12.879 DEBUG:boto:auth_path=/dkyges-43/?policy 2024-04-12T03:41:12.879 DEBUG:boto:Method: GET 2024-04-12T03:41:12.879 DEBUG:boto:Path: /dkyges-43/?policy 2024-04-12T03:41:12.879 DEBUG:boto:Data: 2024-04-12T03:41:12.879 DEBUG:boto:Headers: {} 2024-04-12T03:41:12.879 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:41:12.879 DEBUG:boto:Port: 8001 2024-04-12T03:41:12.879 DEBUG:boto:Params: {} 2024-04-12T03:41:12.879 DEBUG:boto:Token: None 2024-04-12T03:41:12.880 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:12 GMT /dkyges-43/?policy 2024-04-12T03:41:12.880 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cc/DMxMgfINLFVMZBRZzYOiUOyk= 2024-04-12T03:41:12.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:12 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cc/DMxMgfINLFVMZBRZzYOiUOyk=', 'Content-Length': '0'} 2024-04-12T03:41:12.881 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005bec22fde82b7c4e-006618ad58-16de-a2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 12 Apr 2024 03:41:12 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:12.881 DEBUG:boto:path=/dkyges-44/ 2024-04-12T03:41:12.882 DEBUG:boto:auth_path=/dkyges-44/ 2024-04-12T03:41:12.882 DEBUG:boto:path=/dkyges-44/?policy 2024-04-12T03:41:12.882 DEBUG:boto:auth_path=/dkyges-44/?policy 2024-04-12T03:41:12.882 DEBUG:boto:Method: PUT 2024-04-12T03:41:12.882 DEBUG:boto:Path: /dkyges-44/?policy 2024-04-12T03:41:12.882 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-12T03:41:12.882 DEBUG:boto:Headers: {} 2024-04-12T03:41:12.882 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8000 2024-04-12T03:41:12.882 DEBUG:boto:Port: 8000 2024-04-12T03:41:12.882 DEBUG:boto:Params: {} 2024-04-12T03:41:12.882 DEBUG:boto:Token: None 2024-04-12T03:41:12.883 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:41:12 GMT /dkyges-44/?policy 2024-04-12T03:41:12.883 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mEFDO3IpvvLJp4Wa5QYe21PxQd8= 2024-04-12T03:41:12.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:12 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mEFDO3IpvvLJp4Wa5QYe21PxQd8=', 'Content-Length': '97'} 2024-04-12T03:41:13.895 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000020767955825677cf-006618ad58-115f-b1'), ('Date', 'Fri, 12 Apr 2024 03:41:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:13.895 DEBUG:boto:path=/dkyges-44/ 2024-04-12T03:41:13.895 DEBUG:boto:auth_path=/dkyges-44/ 2024-04-12T03:41:13.895 DEBUG:boto:path=/dkyges-44/?policy 2024-04-12T03:41:13.896 DEBUG:boto:auth_path=/dkyges-44/?policy 2024-04-12T03:41:13.896 DEBUG:boto:Method: GET 2024-04-12T03:41:13.896 DEBUG:boto:Path: /dkyges-44/?policy 2024-04-12T03:41:13.896 DEBUG:boto:Data: 2024-04-12T03:41:13.896 DEBUG:boto:Headers: {} 2024-04-12T03:41:13.896 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8000 2024-04-12T03:41:13.896 DEBUG:boto:Port: 8000 2024-04-12T03:41:13.896 DEBUG:boto:Params: {} 2024-04-12T03:41:13.896 DEBUG:boto:Token: None 2024-04-12T03:41:13.896 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:13 GMT /dkyges-44/?policy 2024-04-12T03:41:13.897 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:73C6ZNhcEqpvH3N36ElNYofb11Q= 2024-04-12T03:41:13.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:73C6ZNhcEqpvH3N36ElNYofb11Q=', 'Content-Length': '0'} 2024-04-12T03:41:13.898 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ed47fa8bec07d5fe-006618ad59-115f-b1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 12 Apr 2024 03:41:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:13.898 DEBUG:boto:path=/dkyges-45/ 2024-04-12T03:41:13.898 DEBUG:boto:auth_path=/dkyges-45/ 2024-04-12T03:41:13.898 DEBUG:boto:path=/dkyges-45/?policy 2024-04-12T03:41:13.899 DEBUG:boto:auth_path=/dkyges-45/?policy 2024-04-12T03:41:13.899 DEBUG:boto:Method: PUT 2024-04-12T03:41:13.899 DEBUG:boto:Path: /dkyges-45/?policy 2024-04-12T03:41:13.899 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2024-04-12T03:41:13.899 DEBUG:boto:Headers: {} 2024-04-12T03:41:13.899 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8001 2024-04-12T03:41:13.899 DEBUG:boto:Port: 8001 2024-04-12T03:41:13.899 DEBUG:boto:Params: {} 2024-04-12T03:41:13.899 DEBUG:boto:Token: None 2024-04-12T03:41:13.899 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:41:13 GMT /dkyges-45/?policy 2024-04-12T03:41:13.899 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fbNi65z4i64zwHrN9m+f4WQ0IPE= 2024-04-12T03:41:13.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fbNi65z4i64zwHrN9m+f4WQ0IPE=', 'Content-Length': '97'} 2024-04-12T03:41:14.911 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006b56678f3df9d2ad-006618ad59-115b-b2'), ('Date', 'Fri, 12 Apr 2024 03:41:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:14.911 DEBUG:boto:path=/dkyges-45/ 2024-04-12T03:41:14.911 DEBUG:boto:auth_path=/dkyges-45/ 2024-04-12T03:41:14.911 DEBUG:boto:path=/dkyges-45/?policy 2024-04-12T03:41:14.911 DEBUG:boto:auth_path=/dkyges-45/?policy 2024-04-12T03:41:14.911 DEBUG:boto:Method: GET 2024-04-12T03:41:14.912 DEBUG:boto:Path: /dkyges-45/?policy 2024-04-12T03:41:14.912 DEBUG:boto:Data: 2024-04-12T03:41:14.912 DEBUG:boto:Headers: {} 2024-04-12T03:41:14.912 DEBUG:boto:Host: smithi149.front.sepia.ceph.com:8001 2024-04-12T03:41:14.912 DEBUG:boto:Port: 8001 2024-04-12T03:41:14.912 DEBUG:boto:Params: {} 2024-04-12T03:41:14.912 DEBUG:boto:Token: None 2024-04-12T03:41:14.912 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:41:14 GMT /dkyges-45/?policy 2024-04-12T03:41:14.912 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kBYEJXu15HRUNm8+XkTDbQI1Q4= 2024-04-12T03:41:14.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kBYEJXu15HRUNm8+XkTDbQI1Q4=', 'Content-Length': '0'} 2024-04-12T03:41:14.914 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000567d54372253dc35-006618ad5a-115b-b2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 12 Apr 2024 03:41:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:14.914 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2024-04-12T03:41:14.915 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-46 2024-04-12T03:41:14.915 DEBUG:boto:path=/dkyges-46/ 2024-04-12T03:41:14.915 DEBUG:boto:auth_path=/dkyges-46/ 2024-04-12T03:41:14.915 DEBUG:boto:Method: PUT 2024-04-12T03:41:14.915 DEBUG:boto:Path: /dkyges-46/ 2024-04-12T03:41:14.915 DEBUG:boto:Data: 2024-04-12T03:41:14.915 DEBUG:boto:Headers: {} 2024-04-12T03:41:14.916 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:41:14.916 DEBUG:boto:Port: 8000 2024-04-12T03:41:14.916 DEBUG:boto:Params: {} 2024-04-12T03:41:14.916 DEBUG:boto:Token: None 2024-04-12T03:41:14.916 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:41:14 GMT /dkyges-46/ 2024-04-12T03:41:14.916 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aznVToqrsJjZjtF32iQSVBBiDY8= 2024-04-12T03:41:14.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aznVToqrsJjZjtF32iQSVBBiDY8=', 'Content-Length': '0'} 2024-04-12T03:41:14.937 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000091b875878378b0ae-006618ad5a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:14.937 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-47 2024-04-12T03:41:14.937 DEBUG:boto:path=/dkyges-47/ 2024-04-12T03:41:14.937 DEBUG:boto:auth_path=/dkyges-47/ 2024-04-12T03:41:14.937 DEBUG:boto:Method: PUT 2024-04-12T03:41:14.938 DEBUG:boto:Path: /dkyges-47/ 2024-04-12T03:41:14.938 DEBUG:boto:Data: 2024-04-12T03:41:14.938 DEBUG:boto:Headers: {} 2024-04-12T03:41:14.938 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:41:14.938 DEBUG:boto:Port: 8001 2024-04-12T03:41:14.938 DEBUG:boto:Params: {} 2024-04-12T03:41:14.938 DEBUG:boto:Token: None 2024-04-12T03:41:14.938 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:41:14 GMT /dkyges-47/ 2024-04-12T03:41:14.938 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XSwF95ShqYVCLrEIuYCpdMwlOV8= 2024-04-12T03:41:14.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:41:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XSwF95ShqYVCLrEIuYCpdMwlOV8=', 'Content-Length': '0'} 2024-04-12T03:41:14.971 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000051ac337c3a20f1dc-006618ad5a-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:41:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:41:14.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:14.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:14.972 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:41:15.030 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:15.030 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:15.065 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:41:15.079 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:41:15.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:15.079 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:41:15.079 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:41:15.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:15.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:15.080 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:41:15.080 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:41:15.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:15.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:15.080 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:41:15.080 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:41:15.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:15.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:15.080 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893274.957117_411.1", 2024-04-12T03:41:15.080 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:41:14.957117Z" 2024-04-12T03:41:15.080 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:15.081 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:41:15.082 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:41:15.082 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:41:15.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:15.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:15.082 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:41:15.137 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:15.137 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:15.174+0000 7fdadacba780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:15.175+0000 7fdadacba780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:15.191 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:15.191 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:41:15.191 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:41:15.191 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:41:15.191 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:41:15.191 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:15.191 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:15.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:15.191 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:41:15.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:15.192 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:41:15.192 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:15.192 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:15.192 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:41:15.192 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:15.192 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:41:15.192 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:15.192 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:41:15.192 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:15.192 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:15.193 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:15.193 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:15.193 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:41:15.193 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:15.193 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:15.193 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:41:15.193 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:15.193 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:15.193 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:15.193 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:41:15.193 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:15.193 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:15.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:15.194 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:15.194 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:41:15.194 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:15.194 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:15.194 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:41:15.194 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:15.194 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:15.194 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:15.194 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:41:15.194 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:15.195 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:15.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:15.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:15.195 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:41:15.195 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:15.195 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:15.195 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:41:15.195 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:15.195 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:15.195 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:15.195 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:41:15.196 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:15.196 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:15.196 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:15.196 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:15.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:15.196 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:41:15.196 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:41:15.196 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:41:15.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:15.196 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:41:15Z" 2024-04-12T03:41:15.196 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:15.197 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:41:15Z" } 2024-04-12T03:41:15.197 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:41:15.198 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:41:15.198 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:41:15.198 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:41:20.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:20.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:20.199 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:41:20.260 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:20.260 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:20.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:20.295+0000 7f352981b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:20.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:20.296+0000 7f352981b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:20.311 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:41:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:41:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:41:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:41:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:20.311 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:41:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:41:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:41:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:41:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:41:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:20.312 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:41:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:41:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:41:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:20.313 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:41:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:41:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:41:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:20.314 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:41:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:41:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:41:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:20.315 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:41:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:41:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:41:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:20.316 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:41:20Z" 2024-04-12T03:41:20.316 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:20.317 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:41:20Z" } 2024-04-12T03:41:20.317 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:41:20.317 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:41:20.317 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:41:20.317 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:41:25.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:25.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:25.318 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:41:25.376 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:25.377 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:25.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:25.411+0000 7f93d27b0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:25.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:25.412+0000 7f93d27b0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:25.427 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:25.428 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:41:25.428 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:41:25.428 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:41:25.428 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:41:25.428 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:25.428 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:25.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:25.428 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:41:25.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:25.428 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:41:25.428 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:25.429 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:25.429 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:41:25.429 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:25.429 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:41:25.429 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:25.429 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:41:25.429 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:25.429 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:25.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:25.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:25.429 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:41:25.429 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:25.430 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:25.430 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:41:25.430 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:25.430 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:25.430 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:25.430 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:41:25.430 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:25.430 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:25.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:25.430 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:25.430 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:41:25.430 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:25.431 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:25.431 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:41:25.431 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:25.431 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:25.431 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:25.431 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:41:25.431 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:25.431 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:25.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:25.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:25.431 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:41:25.431 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:25.432 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:25.432 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:41:25.432 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:25.432 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:25.432 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:25.432 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:41:25.432 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:25.432 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:25.432 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:25.432 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:25.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:25.432 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:41:25.432 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:41:25.433 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:41:25.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:25.433 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:41:25Z" 2024-04-12T03:41:25.433 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:25.434 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:41:25Z" } 2024-04-12T03:41:25.434 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:41:25.434 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:41:25.434 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:41:25.434 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:41:30.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:30.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:30.435 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:41:30.494 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:30.494 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:30.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:30.528+0000 7ff1eb33c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:30.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:30.529+0000 7ff1eb33c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:30.545 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:41:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:41:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:41:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:41:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:30.545 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:41:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:41:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:41:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:41:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:41:30.546 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:41:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:41:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:41:30.547 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:41:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:41:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:41:30.548 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:41:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:41:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:41:30.549 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:41:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:41:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:41:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:30.550 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:41:30Z" 2024-04-12T03:41:30.550 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:30.551 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:41:30Z" } 2024-04-12T03:41:30.551 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:41:30.551 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:41:30.552 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:41:30.552 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:41:35.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:35.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:35.554 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:41:35.613 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:35.613 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:35.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:35.647+0000 7f977616a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:35.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:35.648+0000 7f977616a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:35.664 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:35.664 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:41:35.664 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:41:35.664 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:41:35.664 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:41:35.664 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:35.664 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:35.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:35.665 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:41:35.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:35.665 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:41:35.665 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:35.665 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:35.665 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:41:35.665 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:35.665 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:41:35.665 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:35.665 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:41:35.665 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:35.666 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:35.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:35.666 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:35.666 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:41:35.666 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:35.666 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:35.666 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:41:35.666 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:35.666 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:35.666 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:35.666 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:41:35.666 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:35.667 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:35.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:35.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:35.667 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:41:35.667 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:35.667 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:35.667 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:41:35.667 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:35.667 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:35.667 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:35.667 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:41:35.668 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:35.668 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:35.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:35.668 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:35.668 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:41:35.668 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:35.668 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:35.668 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:41:35.668 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:35.668 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:35.668 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:35.668 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:41:35.669 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:35.669 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:35.669 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:35.669 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:35.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:35.669 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:41:35.669 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:41:35.669 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:41:35.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:35.669 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:41:35Z" 2024-04-12T03:41:35.669 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:35.670 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:41:35Z" } 2024-04-12T03:41:35.671 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:41:35.671 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:41:35.671 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:41:35.671 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:41:40.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:40.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:40.672 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:41:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:40.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:40.765+0000 7fab0ad78780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:40.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:40.766+0000 7fab0ad78780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:40.782 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:40.782 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:41:40.782 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:41:40.782 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:41:40.782 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:41:40.782 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:40.782 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:40.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:40.783 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:41:40.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:40.783 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:41:40.783 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:40.783 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:40.783 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:41:40.783 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:40.783 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:41:40.783 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:40.783 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:41:40.784 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:40.784 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:40.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:40.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:40.784 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:41:40.784 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:40.784 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:40.784 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:41:40.784 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:40.784 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:40.784 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:40.785 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:41:40.785 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:40.785 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:40.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:40.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:40.785 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:41:40.785 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:40.785 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:40.785 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:41:40.785 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:40.786 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:40.786 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:40.786 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:41:40.786 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:40.786 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:40.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:40.786 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:40.786 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:41:40.786 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:40.786 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:40.787 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:41:40.787 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:40.787 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:40.787 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:40.787 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:41:40.787 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:40.787 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:40.787 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:40.787 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:40.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:40.787 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:41:40.787 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:41:40.788 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:41:40.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:40.788 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:41:40Z" 2024-04-12T03:41:40.788 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:40.789 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:41:40Z" } 2024-04-12T03:41:40.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:41:40.789 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:41:40.789 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:41:40.789 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:41:45.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:45.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:45.791 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:41:45.849 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:45.849 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:45.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:45.884+0000 7f239ab3c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:45.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:45.885+0000 7f239ab3c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:45.900 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:45.901 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:41:45.901 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:41:45.901 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:41:45.901 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:41:45.901 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:45.901 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:45.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:45.901 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:41:45.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:45.901 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:41:45.901 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:45.902 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:45.902 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:41:45.902 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:45.902 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:41:45.902 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:45.902 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:41:45.902 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:45.902 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:45.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:45.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:45.902 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:41:45.902 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:45.903 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:45.903 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:41:45.903 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:45.903 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:45.903 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:45.903 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:41:45.903 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:45.903 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:45.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:45.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:45.903 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:41:45.904 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:45.904 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:45.904 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:41:45.904 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:45.904 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:45.904 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:45.904 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:41:45.904 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:45.904 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:45.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:45.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:45.905 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:41:45.905 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:45.905 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:45.905 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:41:45.905 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:45.905 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:45.905 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:45.905 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:41:45.905 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:45.905 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:45.906 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:45.906 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:45.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:45.906 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:41:45.906 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:41:45.906 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:41:45.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:45.906 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:41:45Z" 2024-04-12T03:41:45.906 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:45.907 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:41:45Z" } 2024-04-12T03:41:45.907 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:41:45.907 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:41:45.908 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:41:45.908 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:41:50.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:50.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:50.909 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:41:50.970 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:50.970 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:51.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:51.003+0000 7f9d5b137780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:51.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:51.005+0000 7f9d5b137780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:51.021 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:51.021 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:41:51.021 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:41:51.021 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:41:51.021 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:41:51.021 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:51.021 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:41:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:41:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:41:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:41:51.022 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:41:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:41:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:41:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:51.023 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:41:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:41:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:51.024 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:41:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:41:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:41:51.025 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:41:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:41:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:51.026 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:51.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:51.027 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:41:51.027 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:41:51.027 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:41:51.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:51.027 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:41:51Z" 2024-04-12T03:41:51.027 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:51.028 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:41:51Z" } 2024-04-12T03:41:51.028 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:41:51.028 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:41:51.029 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:41:51.029 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:41:56.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:56.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:41:56.030 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:41:56.090 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:41:56.090 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:41:56.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:56.123+0000 7f5fee98f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:56.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:41:56.124+0000 7f5fee98f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:41:56.140 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:41:56.140 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:41:56.140 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:41:56.140 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:41:56.140 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:41:56.140 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:41:56.140 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:56.141 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:56.141 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:41:56.141 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:56.141 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:41:56.141 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:56.141 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:56.141 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:41:56.141 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:56.141 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:41:56.141 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:56.141 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:41:56.142 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:56.142 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:56.142 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:56.142 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:56.142 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:41:56.142 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:56.142 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:56.142 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:41:56.142 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:56.142 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:56.142 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:56.142 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:41:56.143 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:56.143 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:56.143 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:56.143 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:56.143 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:41:56.143 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:56.143 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:56.143 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:41:56.143 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:56.143 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:56.143 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:56.143 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:41:56.144 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:56.144 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:56.144 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:56.144 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:41:56.144 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:41:56.144 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:41:56.144 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:41:56.144 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:41:56.144 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:41:56.144 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:41:56.144 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:41:56.144 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:41:56.145 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:41:56.145 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:56.145 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:41:56.145 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:41:56.145 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:56.145 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:41:56.145 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:41:56.145 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:41:56.145 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:41:56.145 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:41:56Z" 2024-04-12T03:41:56.145 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:41:56.146 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:41:56Z" } 2024-04-12T03:41:56.146 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:41:56.147 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:41:56.147 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:41:56.147 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:42:01.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:01.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:01.148 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:42:01.209 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:42:01.209 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:42:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:01.242+0000 7f17147e4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:01.243+0000 7f17147e4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:01.258 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:42:01.258 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:42:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:42:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:42:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:42:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:42:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:42:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:42:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:01.259 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:01.260 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:42:01.260 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:01.260 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:42:01.260 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:01.260 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:42:01.260 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:01.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:01.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:01.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:01.260 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:42:01.260 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:01.260 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:01.260 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:42:01.261 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:01.261 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:01.261 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:01.261 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:42:01.261 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:01.261 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:01.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:01.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:01.261 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:42:01.261 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:01.261 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:01.261 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:42:01.261 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:01.262 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:01.262 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:01.262 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:42:01.262 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:01.262 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:01.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:01.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:01.262 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:42:01.262 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:01.262 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:01.262 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:42:01.263 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:01.263 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:01.263 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:01.263 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:42:01.263 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:01.263 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:01.263 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:01.263 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:42:01.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:01.263 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:42:01.263 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:42:01.263 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:42:01.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:01.264 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:42:01Z" 2024-04-12T03:42:01.264 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:42:01.264 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:42:01Z" } 2024-04-12T03:42:01.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:42:01.265 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:42:01.265 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:42:01.265 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:42:06.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:06.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:06.267 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:42:06.326 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:42:06.327 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:42:06.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:06.360+0000 7f386f11b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:06.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:06.361+0000 7f386f11b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:06.376 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:42:06.376 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:42:06.377 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:42:06.377 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:42:06.377 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:42:06.377 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:42:06.377 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:06.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:06.377 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:42:06.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:06.377 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:42:06.377 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:06.377 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:06.378 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:42:06.378 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:06.378 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:42:06.378 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:06.378 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:42:06.378 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:06.378 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:06.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:06.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:06.378 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:42:06.378 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:06.378 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:06.379 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:42:06.379 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:06.379 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:06.379 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:06.379 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:42:06.379 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:06.379 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:06.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:06.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:06.379 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:42:06.379 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:06.380 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:06.380 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:42:06.380 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:06.380 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:06.380 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:06.380 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:42:06.380 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:06.380 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:06.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:06.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:06.380 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:42:06.381 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:06.381 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:06.381 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:42:06.381 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:06.381 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:06.381 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:06.381 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:42:06.381 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:06.381 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:06.381 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:06.381 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:42:06.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:06.382 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:42:06.382 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:42:06.382 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:42:06.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:06.382 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:42:06Z" 2024-04-12T03:42:06.382 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:42:06.383 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:42:06Z" } 2024-04-12T03:42:06.383 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:42:06.383 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:42:06.383 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:42:06.383 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:42:11.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:11.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:11.385 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:42:11.446 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:42:11.446 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:42:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:11.480+0000 7f2fbb8d4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:11.481+0000 7f2fbb8d4780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:11.497 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:42:11.497 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:42:11.497 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:42:11.497 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:42:11.498 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:42:11.498 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:42:11.498 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:11.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:11.498 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:42:11.498 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:11.498 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:42:11.498 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:11.498 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:11.498 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:42:11.498 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:11.499 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:42:11.499 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:11.499 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:42:11.499 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:11.499 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:11.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:11.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:11.499 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:42:11.499 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:11.499 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:11.499 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:42:11.500 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:11.500 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:11.500 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:11.500 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:42:11.500 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:11.500 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:11.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:11.500 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:11.500 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:42:11.500 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:11.500 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:11.501 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:42:11.501 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:11.501 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:11.501 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:11.501 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:42:11.501 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:11.501 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:11.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:11.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:11.501 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:42:11.501 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:11.502 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:11.502 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:42:11.502 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:11.502 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:11.502 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:11.502 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:42:11.502 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:11.502 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:11.502 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:11.502 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:42:11.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:11.503 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:42:11.503 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:42:11.503 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:42:11.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:11.503 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:42:11Z" 2024-04-12T03:42:11.503 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:42:11.504 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:42:11Z" } 2024-04-12T03:42:11.504 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:42:11.504 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:42:11.504 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:42:11.504 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:42:16.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:16.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:16.505 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:42:16.565 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:42:16.565 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:42:16.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:16.600+0000 7fe1e74be780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:16.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:16.601+0000 7fe1e74be780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:16.616 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:42:16.616 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:42:16.616 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:42:16.616 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:42:16.616 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:42:16.617 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:42:16.617 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:16.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:16.617 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:42:16.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:16.617 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:42:16.617 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:16.617 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:16.617 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:42:16.617 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:16.617 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:42:16.618 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:16.618 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:42:16.618 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:16.618 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:16.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:16.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:16.618 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:42:16.618 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:16.618 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:16.618 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:42:16.618 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:16.619 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:16.619 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:16.619 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:42:16.619 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:16.619 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:16.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:16.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:16.619 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:42:16.619 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:16.619 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:16.619 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:42:16.619 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:16.620 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:16.620 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:16.620 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:42:16.620 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:16.620 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:16.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:16.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:16.620 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:42:16.620 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:16.620 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:16.620 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:42:16.620 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:16.621 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:16.621 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:16.621 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:42:16.621 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:16.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:16.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:16.621 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:42:16.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:16.621 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:42:16.621 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:42:16.621 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:42:16.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:16.622 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:42:16Z" 2024-04-12T03:42:16.622 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:42:16.622 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:42:16Z" } 2024-04-12T03:42:16.623 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:42:16.623 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:42:16.623 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:42:16.623 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:42:21.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:21.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:21.624 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:42:21.683 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:42:21.684 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:42:21.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:21.718+0000 7f19b0b66780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:21.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:21.719+0000 7f19b0b66780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:21.735 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:42:21.735 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:42:21.735 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:42:21.735 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:42:21.735 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:42:21.736 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:42:21.736 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:21.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:21.736 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:42:21.736 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:21.736 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:42:21.736 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:21.736 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:21.736 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:42:21.736 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:21.736 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:42:21.737 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:21.737 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:42:21.737 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:21.737 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:21.737 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:21.737 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:21.737 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:42:21.737 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:21.737 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:21.737 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:42:21.737 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:21.737 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:21.738 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:21.738 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:42:21.738 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:21.738 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:21.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:21.738 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:21.738 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:42:21.738 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:21.738 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:21.738 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:42:21.739 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:21.739 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:21.739 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:21.739 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:42:21.739 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:21.739 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:21.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:21.739 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:21.739 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:42:21.739 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:21.739 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:21.740 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:42:21.740 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:21.740 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:21.740 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:21.740 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:42:21.740 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:21.740 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:21.740 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:21.740 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:42:21.740 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:21.740 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:42:21.740 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:42:21.741 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:42:21.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:21.741 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:42:21Z" 2024-04-12T03:42:21.741 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:42:21.742 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:42:21Z" } 2024-04-12T03:42:21.742 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:42:21.742 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:42:21.742 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:42:21.742 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:42:26.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:26.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:26.744 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:42:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:42:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:42:26.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:26.839+0000 7f8c1dc3f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:26.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:26.840+0000 7f8c1dc3f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:26.855 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:42:26.855 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:42:26.856 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:42:26.856 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:42:26.856 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:42:26.856 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:42:26.856 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:26.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:26.856 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:42:26.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:26.856 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:42:26.856 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:26.856 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:26.857 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:42:26.857 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:26.857 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:42:26.857 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:26.857 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:42:26.857 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:26.857 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:26.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:26.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:26.857 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:42:26.857 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:26.858 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:26.858 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:42:26.858 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:26.858 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:26.858 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:26.858 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:42:26.858 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:26.858 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:26.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:26.858 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:26.859 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:42:26.859 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:26.859 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:26.859 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:42:26.859 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:26.859 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:26.859 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:26.859 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:42:26.859 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:26.859 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:26.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:26.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:26.859 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:42:26.860 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:26.860 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:26.860 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:42:26.860 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:26.860 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:26.860 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:26.860 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:42:26.860 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:26.860 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:26.860 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:26.860 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:42:26.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:26.861 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:42:26.861 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:42:26.861 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:42:26.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:26.861 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:42:26Z" 2024-04-12T03:42:26.861 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:42:26.862 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:42:26Z" } 2024-04-12T03:42:26.862 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:42:26.862 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:42:26.862 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:42:26.862 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:42:31.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:31.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:31.864 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:42:31.923 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:42:31.923 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:42:31.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:31.958+0000 7fa3608b0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:31.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:31.959+0000 7fa3608b0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:31.975 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:42:31.975 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:42:31.975 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:42:31.975 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:42:31.975 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:42:31.975 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:42:31.975 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:31.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:31.975 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:42:31.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:31.976 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:42:31.976 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:31.976 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:31.976 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:42:31.976 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:31.976 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:42:31.976 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:31.976 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:42:31.976 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:31.976 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:31.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:31.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:31.977 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:42:31.977 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:31.977 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:31.977 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:42:31.977 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:31.977 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:31.977 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:31.977 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:42:31.977 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:31.977 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:31.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:31.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:31.978 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:42:31.978 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:31.978 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:31.978 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:42:31.978 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:31.978 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:31.978 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:31.978 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:42:31.978 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:31.979 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:31.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:31.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:31.979 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:42:31.979 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:31.979 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:31.979 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:42:31.979 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:31.979 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:31.979 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:31.979 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:42:31.980 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:31.980 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:31.980 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:31.980 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:42:31.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:31.980 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:42:31.980 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:42:31.980 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:42:31.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:31.980 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:42:31Z" 2024-04-12T03:42:31.980 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:42:31.981 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:42:31Z" } 2024-04-12T03:42:31.982 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:42:31.982 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:42:31.982 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:42:31.982 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:42:36.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:36.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:36.984 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:42:37.043 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:42:37.043 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:42:37.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:37.080+0000 7f5962dda780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:37.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:37.081+0000 7f5962dda780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:37.096 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:42:37.096 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:42:37.096 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:42:37.097 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:42:37.097 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:42:37.097 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:42:37.097 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:37.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:37.097 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:42:37.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:37.097 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:42:37.097 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:37.097 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:37.097 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:42:37.098 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:37.098 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:42:37.098 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:37.098 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:42:37.098 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:37.098 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:37.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:37.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:37.098 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:42:37.098 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:37.099 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:37.099 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:42:37.099 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:37.099 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:37.099 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:37.099 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:42:37.099 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:37.099 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:37.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:37.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:37.099 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:42:37.099 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:37.100 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:37.100 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:42:37.100 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:37.100 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:37.100 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:37.100 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:42:37.100 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:37.100 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:37.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:37.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:37.100 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:42:37.100 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:37.101 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:37.101 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:42:37.101 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:37.101 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:37.101 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:37.101 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:42:37.101 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:37.101 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:37.101 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:37.101 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:42:37.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:37.102 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:42:37.102 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:42:37.102 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:42:37.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:37.102 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:42:37Z" 2024-04-12T03:42:37.102 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:42:37.103 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:42:37Z" } 2024-04-12T03:42:37.103 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:42:37.103 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:42:37.103 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:42:37.103 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:42:42.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:42.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:42.105 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:42:42.165 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:42:42.165 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:42:42.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:42.199+0000 7fe3101ca780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:42.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:42.200+0000 7fe3101ca780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:42.216 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:42:42.216 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:42:42.216 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:42:42.216 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:42:42.216 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:42:42.216 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:42:42.216 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:42.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:42.216 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:42:42.217 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:42.217 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:42:42.217 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:42.217 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:42.217 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:42:42.217 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:42.217 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:42:42.217 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:42.217 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:42:42.217 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:42.218 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:42.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:42.218 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:42.218 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:42:42.218 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:42.218 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:42.218 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:42:42.218 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:42.218 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:42.218 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:42.219 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:42:42.219 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:42.219 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:42.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:42.219 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:42.219 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:42:42.219 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:42.219 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:42.219 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:42:42.219 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:42.219 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:42.220 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:42.220 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:42:42.220 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:42.220 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:42.220 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:42.220 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:42.220 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:42:42.220 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:42.220 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:42.220 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:42:42.220 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:42.220 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:42.220 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:42.221 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:42:42.221 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:42.221 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:42.221 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:42.221 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:42:42.221 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:42.221 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:42:42.221 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:42:42.221 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:42:42.221 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:42.221 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:42:42Z" 2024-04-12T03:42:42.221 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:42:42.222 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:42:42Z" } 2024-04-12T03:42:42.223 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:42:42.223 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:42:42.223 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:42:42.223 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:42:47.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:47.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:47.224 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:42:47.285 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:42:47.285 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:42:47.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:47.319+0000 7f62cad93780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:47.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:47.321+0000 7f62cad93780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:47.337 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:42:47.337 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:42:47.337 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:42:47.337 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:42:47.337 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:42:47.337 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:42:47.338 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:47.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:47.338 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:42:47.338 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:47.338 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:42:47.338 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:47.338 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:47.338 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:42:47.338 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:47.338 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:42:47.338 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:47.339 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:42:47.339 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:47.339 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:47.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:47.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:47.339 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:42:47.339 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:47.339 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:47.339 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:42:47.339 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:47.339 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:47.340 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:47.340 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:42:47.340 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:47.340 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:47.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:47.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:47.340 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:42:47.340 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:47.340 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:47.340 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:42:47.340 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:47.340 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:47.341 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:47.341 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:42:47.341 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:47.341 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:47.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:47.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:47.341 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:42:47.341 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:47.341 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:47.341 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:42:47.341 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:47.342 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:47.342 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:47.342 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:42:47.342 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:47.342 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:47.342 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:47.342 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:42:47.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:47.342 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:42:47.342 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:42:47.343 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:42:47.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:47.343 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:42:47Z" 2024-04-12T03:42:47.343 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:42:47.344 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:42:47Z" } 2024-04-12T03:42:47.344 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:42:47.344 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:42:47.344 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:42:47.344 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:42:52.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:52.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:52.346 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:42:52.404 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:42:52.405 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:42:52.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:52.439+0000 7f4b224a2780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:52.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:52.440+0000 7f4b224a2780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:52.456 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:42:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:42:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:42:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:42:52.456 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:42:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:42:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:42:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:42:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:42:52.457 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:42:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:42:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:42:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:42:52.458 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:42:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:42:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:42:52.459 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:42:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:42:52.460 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:42:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:42:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:42:52.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:42:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:42:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:42:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:52.462 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:42:52Z" 2024-04-12T03:42:52.462 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:42:52.463 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:42:52Z" } 2024-04-12T03:42:52.463 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:42:52.464 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:42:52.464 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:42:52.464 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:42:57.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:57.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:42:57.465 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:42:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:42:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:42:57.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:57.559+0000 7f77b2278780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:57.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:42:57.560+0000 7f77b2278780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:42:57.576 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:42:57.576 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:42:57.576 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:42:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:42:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:42:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:42:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:42:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:42:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:57.577 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:42:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:42:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:42:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:42:57.578 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:42:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:42:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:57.579 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:42:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:42:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:42:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:57.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:42:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:42:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:42:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:42:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:42:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:42:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:42:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:42:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:42:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:42:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:42:57.581 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:42:57.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:57.582 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:42:57.582 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:42:57.582 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:42:57.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:42:57.582 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:42:57Z" 2024-04-12T03:42:57.582 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:42:57.583 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:42:57Z" } 2024-04-12T03:42:57.583 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:42:57.583 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:42:57.583 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:42:57.584 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:43:02.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:02.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:02.585 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:43:02.647 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:43:02.647 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:43:02.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:02.682+0000 7f12cee31780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:02.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:02.683+0000 7f12cee31780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:02.699 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:43:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:43:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:43:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:43:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:43:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:43:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:43:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:43:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:43:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:43:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:43:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:43:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:43:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:43:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:43:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:43:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:43:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:43:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:43:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:43:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:43:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:43:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:43:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:43:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:43:02Z" 2024-04-12T03:43:02.705 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:43:02.706 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:43:02Z" } 2024-04-12T03:43:02.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:43:02.706 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:43:02.707 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:43:02.707 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:43:07.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:07.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:07.708 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:43:07.768 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:43:07.768 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:43:07.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:07.803+0000 7fd3fcb50780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:07.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:07.804+0000 7fd3fcb50780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:07.820 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:43:07.820 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:43:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:43:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:43:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:43:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:43:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:43:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:43:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:43:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:43:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:43:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:43:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:43:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:43:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:43:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:43:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:43:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:43:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893049.736308_308.1", 2024-04-12T03:43:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:37:29.736308Z", 2024-04-12T03:43:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:43:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:43:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:43:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:43:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:43:07Z" 2024-04-12T03:43:07.826 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:43:07.827 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893049.736308_308.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:37:29.736308Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:43:07Z" } 2024-04-12T03:43:07.827 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893049.736308_308.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:37:29.736308Z', 'realm_epoch': 2}}] 2024-04-12T03:43:07.827 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:43:07.828 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893049.736308_308.1'} 2024-04-12T03:43:07.828 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893049.736308_308.1 2024-04-12T03:43:12.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:12.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:12.829 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:43:12.889 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:43:12.889 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:43:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:12.923+0000 7f1f6fb94780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:12.925+0000 7f1f6fb94780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:12.941 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:43:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:43:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:43:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:43:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:43:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:43:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:12.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:43:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:43:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:43:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:43:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:12.942 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:43:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:43:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:43:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:12.943 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:43:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:43:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:43:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:12.944 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:43:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:43:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.738847_372.1", 2024-04-12T03:43:12.945 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.738847Z", 2024-04-12T03:43:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:43:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:43:12.946 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:43:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:43:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:12.947 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:43:12Z" 2024-04-12T03:43:12.947 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:43:12.948 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.738847_372.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.738847Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:43:12Z" } 2024-04-12T03:43:12.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.738847_372.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.738847Z', 'realm_epoch': 2}}] 2024-04-12T03:43:12.948 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:43:12.948 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.738847_372.1'} 2024-04-12T03:43:12.948 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.738847_372.1 2024-04-12T03:43:17.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:17.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:17.950 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:43:18.007 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:43:18.007 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:43:18.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:18.040+0000 7f89416fa780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:18.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:18.042+0000 7f89416fa780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:18.058 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:43:18.058 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:43:18.058 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:43:18.058 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:43:18.058 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:43:18.058 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:43:18.059 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:18.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:18.059 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:43:18.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:18.059 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:43:18.059 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:18.059 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:18.059 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:43:18.059 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:18.059 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:43:18.059 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:18.060 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:43:18.060 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:18.060 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:18.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:18.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:18.060 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:43:18.060 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:18.060 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:18.060 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:43:18.060 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:18.060 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:18.060 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:18.061 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:43:18.061 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:18.061 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:18.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:18.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:18.061 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:43:18.061 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:18.061 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:18.061 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:43:18.061 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:18.061 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:18.061 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:18.062 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:43:18.062 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:18.062 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:18.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:18.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:18.062 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:43:18.062 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:18.062 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:18.062 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.145385_357.1", 2024-04-12T03:43:18.062 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:18.062 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:18.063 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:18.063 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.145385Z", 2024-04-12T03:43:18.063 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:18.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:18.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:18.063 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:43:18.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:18.063 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:43:18.063 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:43:18.063 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:43:18.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:18.064 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:43:18Z" 2024-04-12T03:43:18.064 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:43:18.064 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.145385_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.145385Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:43:18Z" } 2024-04-12T03:43:18.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.145385_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.145385Z', 'realm_epoch': 2}}] 2024-04-12T03:43:18.065 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:43:18.065 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.145385_357.1'} 2024-04-12T03:43:18.065 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.145385_357.1 2024-04-12T03:43:23.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:23.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:23.067 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:43:23.143 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:43:23.143 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:43:23.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:23.214+0000 7f2706cec780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:23.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:23.215+0000 7f2706cec780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:23.230 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:43:23.231 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:43:23.231 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:43:23.231 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:43:23.231 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:43:23.231 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:43:23.231 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:23.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:23.231 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:43:23.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:23.232 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:43:23.232 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:23.232 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:23.232 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:43:23.232 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:23.232 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:43:23.232 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:23.232 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:43:23.232 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:23.233 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:23.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:23.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:23.233 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:43:23.233 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:23.233 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:23.233 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:43:23.233 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:23.233 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:23.233 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:23.233 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:43:23.234 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:23.234 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:23.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:23.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:23.234 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:43:23.234 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:23.234 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:23.234 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:43:23.234 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:23.235 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:23.235 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:23.235 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:43:23.235 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:23.235 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:23.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:23.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:23.235 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:43:23.235 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:23.235 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:23.235 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.206242_362.1", 2024-04-12T03:43:23.236 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:23.236 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:23.236 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:23.236 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.206242Z", 2024-04-12T03:43:23.236 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:23.236 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:23.236 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:23.236 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:43:23.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:23.236 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:43:23.237 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:43:23.237 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:43:23.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:23.237 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:43:23Z" 2024-04-12T03:43:23.237 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:43:23.238 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.206242_362.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.206242Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:43:23Z" } 2024-04-12T03:43:23.238 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.206242_362.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.206242Z', 'realm_epoch': 2}}] 2024-04-12T03:43:23.238 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:43:23.238 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.206242_362.1'} 2024-04-12T03:43:23.238 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.206242_362.1 2024-04-12T03:43:28.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:28.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:28.240 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:43:28.299 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:43:28.299 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:43:28.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:28.333+0000 7f4c8b811780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:28.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:28.334+0000 7f4c8b811780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:28.351 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:43:28.351 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:43:28.351 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:43:28.351 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:43:28.351 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:43:28.351 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:43:28.351 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:28.351 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:28.351 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:43:28.351 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:28.352 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:43:28.352 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:28.352 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:28.352 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:43:28.352 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:28.352 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:43:28.352 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:28.352 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:43:28.352 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:28.352 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:28.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:28.352 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:28.353 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:43:28.353 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:28.353 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:28.353 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:43:28.353 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:28.353 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:28.353 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:28.353 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:43:28.353 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:28.353 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:28.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:28.354 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:28.354 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:43:28.354 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:28.354 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:28.354 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:43:28.354 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:28.354 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:28.354 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:28.354 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:43:28.354 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:28.355 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:28.355 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:28.355 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:28.355 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:43:28.355 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:28.355 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:28.355 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.206242_362.1", 2024-04-12T03:43:28.355 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:28.355 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:28.355 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:28.355 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.206242Z", 2024-04-12T03:43:28.355 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:28.356 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:28.356 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:28.356 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:43:28.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:28.356 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:43:28.356 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:43:28.356 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:43:28.356 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:28.356 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:43:28Z" 2024-04-12T03:43:28.356 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:43:28.357 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.206242_362.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.206242Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:43:28Z" } 2024-04-12T03:43:28.357 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.206242_362.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.206242Z', 'realm_epoch': 2}}] 2024-04-12T03:43:28.357 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:43:28.358 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.206242_362.1'} 2024-04-12T03:43:28.358 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.206242_362.1 2024-04-12T03:43:33.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:33.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:33.359 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:43:33.418 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:43:33.418 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:43:33.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:33.452+0000 7fcf32af8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:33.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:33.453+0000 7fcf32af8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:33.469 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:43:33.469 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:43:33.469 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:43:33.469 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:43:33.469 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:43:33.470 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:43:33.470 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:33.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:33.470 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:43:33.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:33.470 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:43:33.470 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:33.470 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:33.470 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:43:33.470 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:33.470 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:43:33.471 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:33.471 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:43:33.471 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:33.471 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:33.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:33.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:33.471 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:43:33.471 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:33.471 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:33.471 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:43:33.471 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:33.471 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:33.472 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:33.472 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:43:33.472 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:33.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:33.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:33.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:33.472 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:43:33.472 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:33.472 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:33.472 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:43:33.472 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:33.473 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:33.473 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:33.473 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:43:33.473 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:33.473 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:33.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:33.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:33.473 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:43:33.473 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:33.473 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:33.473 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.149960_358.1", 2024-04-12T03:43:33.473 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:33.474 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:33.474 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:33.474 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.149960Z", 2024-04-12T03:43:33.474 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:33.474 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:33.474 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:33.474 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:43:33.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:33.474 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:43:33.474 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:43:33.474 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:43:33.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:33.475 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:43:33Z" 2024-04-12T03:43:33.475 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:43:33.476 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.149960_358.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.149960Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:43:33Z" } 2024-04-12T03:43:33.476 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.149960_358.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.149960Z', 'realm_epoch': 2}}] 2024-04-12T03:43:33.476 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:43:33.476 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.149960_358.1'} 2024-04-12T03:43:33.476 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.149960_358.1 2024-04-12T03:43:38.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:38.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:38.478 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:43:38.542 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:43:38.542 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:43:38.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:38.574+0000 7ff822340780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:38.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:38.575+0000 7ff822340780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:38.593 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:43:38.593 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:43:38.593 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:43:38.593 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:43:38.593 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:43:38.593 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:43:38.594 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:38.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:38.594 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:43:38.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:38.594 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:43:38.594 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:38.594 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:38.595 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:43:38.595 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:38.595 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:43:38.595 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:38.595 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:43:38.595 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:38.595 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:38.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:38.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:38.596 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:43:38.596 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:38.596 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:38.596 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:43:38.596 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:38.596 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:38.596 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:38.597 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:43:38.597 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:38.597 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:38.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:38.597 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:38.597 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:43:38.597 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:38.597 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:38.598 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:43:38.598 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:38.598 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:38.598 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:38.598 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:43:38.598 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:38.598 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:38.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:38.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:38.599 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:43:38.599 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:38.599 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:38.599 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.214735_363.1", 2024-04-12T03:43:38.599 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:38.600 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:38.600 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:38.600 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.214735Z", 2024-04-12T03:43:38.600 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:38.600 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:38.600 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:38.600 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:43:38.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:38.601 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:43:38.601 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:43:38.601 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:43:38.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:38.601 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:43:38Z" 2024-04-12T03:43:38.601 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:43:38.602 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.214735_363.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.214735Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:43:38Z" } 2024-04-12T03:43:38.603 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.214735_363.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.214735Z', 'realm_epoch': 2}}] 2024-04-12T03:43:38.603 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:43:38.603 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.214735_363.1'} 2024-04-12T03:43:38.603 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.214735_363.1 2024-04-12T03:43:43.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:43.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:43.605 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:43:43.666 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:43:43.666 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:43:43.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:43.702+0000 7f3862e18780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:43.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:43.703+0000 7f3862e18780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:43.719 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:43:43.719 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:43:43.719 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:43:43.720 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:43:43.720 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:43:43.720 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:43:43.720 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:43.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:43.720 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:43:43.720 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:43.720 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:43:43.720 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:43.721 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:43.721 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:43:43.721 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:43.721 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:43:43.721 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:43.721 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:43:43.721 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:43.721 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:43.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:43.722 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:43.722 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:43:43.722 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:43.722 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:43.722 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:43:43.722 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:43.722 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:43.722 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:43.722 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:43:43.723 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:43.723 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:43.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:43.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:43.723 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:43:43.723 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:43.723 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:43.723 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:43:43.724 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:43.724 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:43.724 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:43.724 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:43:43.724 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:43.724 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:43.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:43.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:43.724 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:43:43.725 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:43.725 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:43.725 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.743463_374.1", 2024-04-12T03:43:43.725 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:43.725 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:43.725 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:43.725 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.743463Z", 2024-04-12T03:43:43.726 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:43.726 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:43.726 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:43.726 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:43:43.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:43.726 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:43:43.726 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:43:43.727 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:43:43.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:43.727 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:43:43Z" 2024-04-12T03:43:43.727 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:43:43.728 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.743463_374.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.743463Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:43:43Z" } 2024-04-12T03:43:43.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.743463_374.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.743463Z', 'realm_epoch': 2}}] 2024-04-12T03:43:43.728 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:43:43.728 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.743463_374.1'} 2024-04-12T03:43:43.729 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.743463_374.1 2024-04-12T03:43:48.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:48.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:48.731 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:43:48.789 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:43:48.789 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:43:48.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:48.821+0000 7f6083b53780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:48.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:48.822+0000 7f6083b53780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:48.838 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:43:48.838 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:43:48.838 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:43:48.838 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:43:48.838 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:43:48.838 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:43:48.838 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:48.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:48.839 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:43:48.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:48.839 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:43:48.839 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:48.839 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:48.839 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:43:48.839 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:48.839 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:43:48.839 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:48.839 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:43:48.839 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:48.840 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:48.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:48.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:48.840 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:43:48.840 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:48.840 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:48.840 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:43:48.840 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:48.840 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:48.840 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:48.840 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:43:48.840 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:48.841 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:48.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:48.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:48.841 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:43:48.841 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:48.841 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:48.841 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:43:48.841 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:48.841 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:48.841 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:48.841 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:43:48.842 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:48.842 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:48.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:48.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:48.842 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:43:48.842 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:48.842 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:48.842 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.214735_363.1", 2024-04-12T03:43:48.842 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:48.842 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:48.842 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:48.842 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.214735Z", 2024-04-12T03:43:48.843 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:48.843 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:48.843 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:48.843 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:43:48.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:48.843 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:43:48.843 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:43:48.843 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:43:48.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:48.843 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:43:48Z" 2024-04-12T03:43:48.843 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:43:48.845 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.214735_363.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.214735Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:43:48Z" } 2024-04-12T03:43:48.845 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.214735_363.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.214735Z', 'realm_epoch': 2}}] 2024-04-12T03:43:48.845 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:43:48.845 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.214735_363.1'} 2024-04-12T03:43:48.845 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.214735_363.1 2024-04-12T03:43:53.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:53.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:53.847 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:43:53.906 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:43:53.906 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:43:53.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:53.942+0000 7f58fd629780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:53.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:53.943+0000 7f58fd629780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:53.958 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:43:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:43:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:43:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:43:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:43:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:43:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:43:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:43:53.959 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:43:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:43:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:43:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:53.960 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:43:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:43:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:43:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:53.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:43:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:43:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:43:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:53.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:43:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.166984_359.1", 2024-04-12T03:43:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.166984Z", 2024-04-12T03:43:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:53.963 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:43:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:43:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:43:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:43:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:53.964 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:43:53Z" 2024-04-12T03:43:53.964 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:43:53.965 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.166984_359.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.166984Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:43:53Z" } 2024-04-12T03:43:53.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.166984_359.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.166984Z', 'realm_epoch': 2}}] 2024-04-12T03:43:53.966 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:43:53.966 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.166984_359.1'} 2024-04-12T03:43:53.966 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.166984_359.1 2024-04-12T03:43:58.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:58.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:43:58.967 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:43:59.026 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:43:59.027 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:43:59.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:59.060+0000 7f6ea6b56780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:59.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:43:59.062+0000 7f6ea6b56780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:43:59.079 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:43:59.079 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:43:59.079 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:43:59.080 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:43:59.080 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:43:59.080 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:43:59.080 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:59.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:59.080 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:43:59.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:59.080 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:43:59.080 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:59.080 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:59.080 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:43:59.080 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:59.081 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:43:59.081 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:59.081 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:43:59.081 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:59.081 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:59.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:59.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:59.081 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:43:59.081 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:59.081 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:59.082 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:43:59.082 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:59.082 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:59.082 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:59.082 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:43:59.082 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:59.082 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:59.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:59.082 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:59.082 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:43:59.082 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:59.083 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:59.083 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:43:59.083 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:59.083 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:59.083 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:59.083 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:43:59.083 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:59.083 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:59.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:59.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:43:59.083 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:43:59.084 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:43:59.084 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:43:59.084 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.166984_359.1", 2024-04-12T03:43:59.084 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:43:59.084 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:43:59.084 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:43:59.084 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.166984Z", 2024-04-12T03:43:59.084 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:43:59.084 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:59.084 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:43:59.084 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:43:59.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:59.085 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:43:59.085 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:43:59.085 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:43:59.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:43:59.085 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:43:59Z" 2024-04-12T03:43:59.085 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:43:59.086 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.166984_359.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.166984Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:43:59Z" } 2024-04-12T03:43:59.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.166984_359.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.166984Z', 'realm_epoch': 2}}] 2024-04-12T03:43:59.086 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:43:59.086 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.166984_359.1'} 2024-04-12T03:43:59.087 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.166984_359.1 2024-04-12T03:44:04.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:04.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:04.089 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:44:04.143 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:44:04.144 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:44:04.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:04.179+0000 7f3b6e9fb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:04.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:04.180+0000 7f3b6e9fb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:04.195 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:44:04.196 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:44:04.196 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:44:04.196 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:44:04.196 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:44:04.196 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:44:04.196 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:04.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:04.196 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:44:04.196 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:04.197 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:44:04.197 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:04.197 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:04.197 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:44:04.197 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:04.197 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:44:04.197 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:04.197 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:44:04.197 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:04.197 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:04.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:04.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:04.198 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:44:04.198 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:04.198 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:04.198 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:44:04.198 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:04.198 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:04.198 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:04.198 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:44:04.198 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:04.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:04.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:04.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:04.199 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:44:04.199 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:04.199 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:04.199 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:44:04.199 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:04.199 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:04.199 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:04.200 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:44:04.200 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:04.200 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:04.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:04.200 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:04.200 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:44:04.200 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:04.200 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:04.200 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.145385_357.1", 2024-04-12T03:44:04.200 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:04.201 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:04.201 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:04.201 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.145385Z", 2024-04-12T03:44:04.201 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:04.201 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:04.201 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:04.201 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:44:04.201 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:04.201 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:44:04.201 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:44:04.201 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:44:04.202 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:04.202 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:44:04Z" 2024-04-12T03:44:04.202 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:44:04.203 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.145385_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.145385Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:44:04Z" } 2024-04-12T03:44:04.203 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.145385_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.145385Z', 'realm_epoch': 2}}] 2024-04-12T03:44:04.203 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:44:04.203 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.145385_357.1'} 2024-04-12T03:44:04.204 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.145385_357.1 2024-04-12T03:44:09.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:09.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:09.205 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:44:09.267 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:44:09.267 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:44:09.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:09.305+0000 7f5df5d3f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:09.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:09.307+0000 7f5df5d3f780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:09.323 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:44:09.323 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:44:09.323 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:44:09.323 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:44:09.323 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:44:09.323 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:44:09.323 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:09.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:09.324 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:44:09.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:09.324 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:44:09.324 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:09.324 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:09.324 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:44:09.324 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:09.324 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:44:09.324 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:09.324 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:44:09.324 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:09.325 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:09.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:09.325 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:09.325 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:44:09.325 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:09.325 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:09.325 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:44:09.325 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:09.325 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:09.325 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:09.325 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:44:09.326 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:09.326 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:09.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:09.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:09.326 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:44:09.326 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:09.326 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:09.326 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:44:09.326 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:09.327 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:09.327 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:09.327 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:44:09.327 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:09.327 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:09.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:09.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:09.328 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:44:09.328 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:09.328 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:09.328 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.742193_373.1", 2024-04-12T03:44:09.328 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:09.328 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:09.328 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:09.328 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.742193Z", 2024-04-12T03:44:09.328 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:09.328 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:09.328 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:09.329 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:44:09.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:09.329 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:44:09.329 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:44:09.329 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:44:09.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:09.329 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:44:09Z" 2024-04-12T03:44:09.329 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:44:09.330 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.742193_373.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.742193Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:44:09Z" } 2024-04-12T03:44:09.330 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.742193_373.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.742193Z', 'realm_epoch': 2}}] 2024-04-12T03:44:09.331 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:44:09.331 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.742193_373.1'} 2024-04-12T03:44:09.331 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.742193_373.1 2024-04-12T03:44:14.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:14.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:14.333 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:44:14.393 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:44:14.394 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:44:14.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:14.428+0000 7f0dc2adf780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:14.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:14.429+0000 7f0dc2adf780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:14.445 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:44:14.445 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:44:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:44:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:44:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:44:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:44:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:44:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:44:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:44:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:44:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:44:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:44:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:44:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:44:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:44:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:44:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:44:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:14.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:14.450 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:44:14.450 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:14.450 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:14.450 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.738847_372.1", 2024-04-12T03:44:14.450 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:14.450 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:14.450 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:14.450 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.738847Z", 2024-04-12T03:44:14.450 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:14.450 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:14.451 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:14.451 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:44:14.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:14.451 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:44:14.451 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:44:14.451 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:44:14.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:14.451 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:44:14Z" 2024-04-12T03:44:14.451 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:44:14.452 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.738847_372.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.738847Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:44:14Z" } 2024-04-12T03:44:14.452 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.738847_372.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.738847Z', 'realm_epoch': 2}}] 2024-04-12T03:44:14.453 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:44:14.453 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.738847_372.1'} 2024-04-12T03:44:14.453 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.738847_372.1 2024-04-12T03:44:19.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:19.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:19.460 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:44:19.517 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:44:19.518 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:44:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:19.550+0000 7f377619a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:19.551+0000 7f377619a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:19.567 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:44:19.567 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:44:19.567 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:44:19.567 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:44:19.567 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:44:19.567 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:44:19.567 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:44:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:44:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:44:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:44:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:44:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:44:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:44:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:44:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:44:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:44:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:44:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:44:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.218795_364.1", 2024-04-12T03:44:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.218795Z", 2024-04-12T03:44:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:44:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:44:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:44:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:44:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:44:19Z" 2024-04-12T03:44:19.573 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:44:19.574 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.218795_364.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.218795Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:44:19Z" } 2024-04-12T03:44:19.574 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.218795_364.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.218795Z', 'realm_epoch': 2}}] 2024-04-12T03:44:19.574 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:44:19.574 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.218795_364.1'} 2024-04-12T03:44:19.574 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.218795_364.1 2024-04-12T03:44:24.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:24.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:24.577 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:44:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:44:24.637 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:44:24.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:24.672+0000 7f5fb359b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:24.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:24.673+0000 7f5fb359b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:24.689 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:44:24.689 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:44:24.689 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:44:24.689 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:44:24.689 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:44:24.689 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:44:24.690 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:24.690 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:24.690 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:44:24.690 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:24.690 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:44:24.690 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:24.690 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:24.690 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:44:24.690 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:24.691 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:44:24.691 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:24.691 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:44:24.691 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:24.691 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:24.691 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:24.691 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:24.691 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:44:24.691 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:24.691 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:24.692 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:44:24.692 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:24.692 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:24.692 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:24.692 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:44:24.692 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:24.692 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:24.692 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:24.692 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:24.692 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:44:24.692 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:24.693 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:24.693 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:44:24.693 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:24.693 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:24.693 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:24.693 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:44:24.693 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:24.693 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:24.693 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:24.693 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:24.693 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:44:24.694 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:24.694 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:24.694 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.145385_357.1", 2024-04-12T03:44:24.694 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:24.694 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:24.694 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:24.694 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.145385Z", 2024-04-12T03:44:24.694 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:24.694 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:24.694 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:24.695 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:44:24.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:24.695 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:44:24.695 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:44:24.695 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:44:24.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:24.695 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:44:24Z" 2024-04-12T03:44:24.695 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:44:24.696 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.145385_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.145385Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:44:24Z" } 2024-04-12T03:44:24.696 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.145385_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.145385Z', 'realm_epoch': 2}}] 2024-04-12T03:44:24.696 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:44:24.696 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.145385_357.1'} 2024-04-12T03:44:24.697 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.145385_357.1 2024-04-12T03:44:29.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:29.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:29.698 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:44:29.758 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:44:29.758 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:44:29.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:29.793+0000 7f62356d6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:29.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:29.794+0000 7f62356d6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:29.810 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:44:29.810 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:44:29.811 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:44:29.811 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:44:29.811 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:44:29.811 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:44:29.811 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:29.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:29.811 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:44:29.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:29.811 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:44:29.811 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:29.811 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:29.812 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:44:29.812 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:29.812 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:44:29.812 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:29.812 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:44:29.812 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:29.812 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:29.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:29.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:29.812 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:44:29.812 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:29.812 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:29.813 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:44:29.813 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:29.813 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:29.813 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:29.813 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:44:29.813 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:29.813 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:29.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:29.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:29.813 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:44:29.813 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:29.814 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:29.814 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:44:29.814 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:29.814 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:29.814 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:29.814 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:44:29.814 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:29.814 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:29.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:29.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:29.814 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:44:29.814 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:29.815 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:29.815 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.226327_365.1", 2024-04-12T03:44:29.815 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:29.815 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:29.815 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:29.815 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.226327Z", 2024-04-12T03:44:29.815 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:29.815 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:29.815 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:29.815 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:44:29.815 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:29.816 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:44:29.816 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:44:29.816 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:44:29.816 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:29.816 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:44:29Z" 2024-04-12T03:44:29.816 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:44:29.817 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.226327_365.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.226327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:44:29Z" } 2024-04-12T03:44:29.817 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.226327_365.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.226327Z', 'realm_epoch': 2}}] 2024-04-12T03:44:29.817 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:44:29.817 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.226327_365.1'} 2024-04-12T03:44:29.818 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.226327_365.1 2024-04-12T03:44:34.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:34.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:34.819 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:44:34.878 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:44:34.878 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:44:34.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:34.912+0000 7fec34c89780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:34.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:34.913+0000 7fec34c89780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:34.929 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:44:34.929 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:44:34.929 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:44:34.929 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:44:34.930 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:44:34.930 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:44:34.930 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:34.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:34.930 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:44:34.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:34.930 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:44:34.930 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:34.930 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:34.930 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:44:34.930 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:34.931 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:44:34.931 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:34.931 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:44:34.931 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:34.931 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:34.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:34.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:34.931 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:44:34.931 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:34.931 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:34.931 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:44:34.932 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:34.932 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:34.932 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:34.932 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:44:34.932 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:34.932 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:34.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:34.932 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:34.932 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:44:34.932 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:34.932 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:34.933 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:44:34.933 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:34.933 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:34.933 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:34.933 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:44:34.933 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:34.933 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:34.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:34.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:34.933 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:44:34.933 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:34.934 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:34.934 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.145385_357.1", 2024-04-12T03:44:34.934 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:34.934 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:34.934 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:34.934 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.145385Z", 2024-04-12T03:44:34.934 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:34.934 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:34.934 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:34.934 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:44:34.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:34.935 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:44:34.935 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:44:34.935 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:44:34.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:34.935 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:44:34Z" 2024-04-12T03:44:34.935 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:44:34.936 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.145385_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.145385Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:44:34Z" } 2024-04-12T03:44:34.936 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.145385_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.145385Z', 'realm_epoch': 2}}] 2024-04-12T03:44:34.936 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:44:34.937 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.145385_357.1'} 2024-04-12T03:44:34.937 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.145385_357.1 2024-04-12T03:44:39.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:39.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:39.938 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:44:39.999 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:44:39.999 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:44:40.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:40.034+0000 7f2d86e7a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:40.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:40.035+0000 7f2d86e7a780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:40.051 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:44:40.051 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:44:40.051 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:44:40.052 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:44:40.052 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:44:40.052 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:44:40.052 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:40.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:40.052 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:44:40.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:40.052 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:44:40.052 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:40.052 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:40.053 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:44:40.053 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:40.053 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:44:40.053 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:40.053 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:44:40.053 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:40.053 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:40.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:40.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:40.053 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:44:40.053 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:40.054 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:40.054 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:44:40.054 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:40.054 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:40.054 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:40.054 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:44:40.054 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:40.054 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:40.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:40.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:40.055 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:44:40.055 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:40.055 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:40.055 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:44:40.055 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:40.055 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:40.055 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:40.055 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:44:40.055 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:40.055 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:40.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:40.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:40.056 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:44:40.056 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:40.056 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:40.056 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.214735_363.1", 2024-04-12T03:44:40.056 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:40.056 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:40.056 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:40.056 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.214735Z", 2024-04-12T03:44:40.056 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:40.056 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:40.056 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:40.057 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:44:40.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:40.057 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:44:40.057 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:44:40.057 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:44:40.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:40.057 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:44:40Z" 2024-04-12T03:44:40.057 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:44:40.058 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.214735_363.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.214735Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:44:40Z" } 2024-04-12T03:44:40.058 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.214735_363.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.214735Z', 'realm_epoch': 2}}] 2024-04-12T03:44:40.058 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:44:40.059 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.214735_363.1'} 2024-04-12T03:44:40.059 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.214735_363.1 2024-04-12T03:44:45.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:45.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:45.060 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:44:45.120 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:44:45.121 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:44:45.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:45.160+0000 7fd262457780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:45.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:45.161+0000 7fd262457780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:45.177 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:44:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:44:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:44:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:44:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:44:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:44:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:44:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:44:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:44:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:44:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:44:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:44:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:44:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:44:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:44:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:44:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:44:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:44:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893271.129434_376.1", 2024-04-12T03:44:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:11.129434Z", 2024-04-12T03:44:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:44:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:44:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:44:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:44:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:44:45Z" 2024-04-12T03:44:45.183 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:44:45.184 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893271.129434_376.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:11.129434Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:44:45Z" } 2024-04-12T03:44:45.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893271.129434_376.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:11.129434Z', 'realm_epoch': 2}}] 2024-04-12T03:44:45.185 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:44:45.185 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893271.129434_376.1'} 2024-04-12T03:44:45.185 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893271.129434_376.1 2024-04-12T03:44:50.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:50.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:50.187 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:44:50.244 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:44:50.244 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:44:50.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:50.279+0000 7f0145a06780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:50.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:50.280+0000 7f0145a06780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:50.296 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:44:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:44:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:44:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:44:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:44:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:44:50.296 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:44:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:44:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:44:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:50.297 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:44:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:44:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:44:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:44:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:50.298 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:44:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:44:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:44:50.299 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:44:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:44:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.743463_374.1", 2024-04-12T03:44:50.300 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.743463Z", 2024-04-12T03:44:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:44:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:44:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:44:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:44:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:44:50Z" 2024-04-12T03:44:50.302 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:44:50.303 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.743463_374.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.743463Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:44:50Z" } 2024-04-12T03:44:50.303 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.743463_374.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.743463Z', 'realm_epoch': 2}}] 2024-04-12T03:44:50.303 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:44:50.303 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.743463_374.1'} 2024-04-12T03:44:50.303 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.743463_374.1 2024-04-12T03:44:55.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:55.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:44:55.305 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:44:55.365 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:44:55.366 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:44:55.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:55.400+0000 7fd61c3bb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:44:55.401+0000 7fd61c3bb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:44:55.417 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:44:55.417 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:44:55.417 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:44:55.418 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:44:55.418 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:44:55.418 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:44:55.418 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:55.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:55.418 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:44:55.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:55.418 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:44:55.418 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:55.418 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:55.419 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:44:55.419 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:55.419 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:44:55.419 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:55.419 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:44:55.419 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:55.419 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:55.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:55.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:55.419 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:44:55.419 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:55.419 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:55.420 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:44:55.420 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:55.420 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:55.420 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:55.420 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:44:55.420 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:55.420 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:55.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:55.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:55.420 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:44:55.420 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:55.421 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:55.421 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:44:55.421 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:55.421 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:55.421 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:55.421 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:44:55.421 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:55.421 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:55.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:55.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:44:55.421 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:44:55.422 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:44:55.422 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:44:55.422 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.738847_372.1", 2024-04-12T03:44:55.422 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:44:55.422 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:44:55.422 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:44:55.422 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.738847Z", 2024-04-12T03:44:55.422 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:44:55.422 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:55.422 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:44:55.422 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:44:55.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:55.423 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:44:55.423 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:44:55.423 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:44:55.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:44:55.423 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:44:55Z" 2024-04-12T03:44:55.423 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:44:55.424 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.738847_372.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.738847Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:44:55Z" } 2024-04-12T03:44:55.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.738847_372.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.738847Z', 'realm_epoch': 2}}] 2024-04-12T03:44:55.424 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:44:55.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.738847_372.1'} 2024-04-12T03:44:55.425 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.738847_372.1 2024-04-12T03:45:00.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:00.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:00.426 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:45:00.487 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:00.487 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:00.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:00.522+0000 7f230d527780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:00.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:00.524+0000 7f230d527780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:00.540 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:45:00.540 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:45:00.540 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:45:00.540 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:45:00.540 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:45:00.540 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:45:00.540 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:00.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:00.540 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:45:00.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:00.541 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:45:00.541 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:00.541 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:00.541 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:45:00.541 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:00.541 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:45:00.541 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:00.541 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:45:00.541 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:00.541 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:00.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:00.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:00.542 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:45:00.542 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:00.542 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:00.542 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:45:00.542 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:00.542 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:00.542 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:00.542 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:45:00.542 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:00.542 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:00.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:00.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:00.543 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:45:00.543 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:00.543 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:00.543 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:45:00.543 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:00.543 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:00.543 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:00.543 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:45:00.543 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:00.544 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:00.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:00.544 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:00.544 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:45:00.544 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:00.544 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:00.544 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.145385_357.1", 2024-04-12T03:45:00.544 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:00.544 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:00.544 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:00.544 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.145385Z", 2024-04-12T03:45:00.544 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:00.545 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:00.545 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:00.545 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:45:00.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:00.545 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:45:00.545 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:45:00.545 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:45:00.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:00.545 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:45:00Z" 2024-04-12T03:45:00.545 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:45:00.546 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.145385_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.145385Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:45:00Z" } 2024-04-12T03:45:00.547 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.145385_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.145385Z', 'realm_epoch': 2}}] 2024-04-12T03:45:00.547 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:45:00.547 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.145385_357.1'} 2024-04-12T03:45:00.547 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.145385_357.1 2024-04-12T03:45:05.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:05.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:05.548 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:45:05.618 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:05.618 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:05.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:05.691+0000 7ff4f2f7e780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:05.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:05.692+0000 7ff4f2f7e780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:05.707 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:45:05.708 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:45:05.708 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:45:05.708 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:45:05.708 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:45:05.708 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:45:05.708 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:05.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:05.708 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:45:05.708 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:05.708 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:45:05.708 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:05.709 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:05.709 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:45:05.709 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:05.709 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:45:05.709 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:05.709 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:45:05.709 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:05.709 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:05.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:05.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:05.709 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:45:05.709 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:05.710 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:05.710 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:45:05.710 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:05.710 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:05.710 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:05.710 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:45:05.710 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:05.710 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:05.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:05.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:05.710 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:45:05.711 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:05.711 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:05.711 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:45:05.711 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:05.711 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:05.711 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:05.711 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:45:05.711 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:05.711 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:05.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:05.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:05.712 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:45:05.712 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:05.712 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:05.712 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.738847_372.1", 2024-04-12T03:45:05.712 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:05.712 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:05.712 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:05.712 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.738847Z", 2024-04-12T03:45:05.712 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:05.712 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:05.712 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:05.712 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:45:05.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:05.713 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:45:05.713 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:45:05.713 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:45:05.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:05.713 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:45:05Z" 2024-04-12T03:45:05.713 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:45:05.714 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.738847_372.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.738847Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:45:05Z" } 2024-04-12T03:45:05.714 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.738847_372.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.738847Z', 'realm_epoch': 2}}] 2024-04-12T03:45:05.714 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:45:05.715 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.738847_372.1'} 2024-04-12T03:45:05.715 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.738847_372.1 2024-04-12T03:45:10.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:10.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:10.716 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:45:10.776 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:10.776 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:10.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:10.811+0000 7f2842d5c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:10.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:10.812+0000 7f2842d5c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:10.827 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:45:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:45:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:45:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:45:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:45:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:45:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:45:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:10.828 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:45:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:45:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:45:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:45:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:10.829 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:45:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:45:10.830 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:45:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:10.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:10.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:10.832 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:45:10.832 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:10.832 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:10.832 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:45:10.833 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:10.833 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:10.833 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:10.833 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:45:10.833 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:10.834 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:10.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:10.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:10.834 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:45:10.834 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:10.835 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:10.835 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.166984_359.1", 2024-04-12T03:45:10.835 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:10.835 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:10.836 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:10.836 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.166984Z", 2024-04-12T03:45:10.836 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:10.836 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:10.837 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:10.837 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:45:10.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:10.837 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:45:10.838 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:45:10.838 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:45:10.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:10.838 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:45:10Z" 2024-04-12T03:45:10.838 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:45:10.840 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.166984_359.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.166984Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:45:10Z" } 2024-04-12T03:45:10.840 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.166984_359.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.166984Z', 'realm_epoch': 2}}] 2024-04-12T03:45:10.841 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:45:10.841 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.166984_359.1'} 2024-04-12T03:45:10.841 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.166984_359.1 2024-04-12T03:45:15.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:15.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:15.843 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:45:15.906 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:15.906 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:15.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:15.940+0000 7f40757fb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:15.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:15.941+0000 7f40757fb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:15.957 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:45:15.957 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:45:15.957 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:45:15.958 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:45:15.958 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:45:15.958 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:45:15.958 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:15.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:15.958 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:45:15.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:15.958 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:45:15.958 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:15.958 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:15.958 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:45:15.959 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:15.959 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:45:15.959 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:15.959 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:45:15.959 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:15.959 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:15.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:15.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:15.959 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:45:15.959 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:15.959 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:15.960 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:45:15.960 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:15.960 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:15.960 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:15.960 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:45:15.960 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:15.960 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:15.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:15.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:15.960 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:45:15.960 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:15.961 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:15.961 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:45:15.961 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:15.961 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:15.961 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:15.961 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:45:15.961 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:15.961 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:15.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:15.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:15.961 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:45:15.962 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:15.962 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:15.962 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.206242_362.1", 2024-04-12T03:45:15.962 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:15.962 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:15.963 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:15.963 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.206242Z", 2024-04-12T03:45:15.963 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:15.963 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:15.963 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:15.963 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:45:15.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:15.964 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:45:15.964 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:45:15.964 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:45:15.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:15.964 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:45:15Z" 2024-04-12T03:45:15.964 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:45:15.965 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.206242_362.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.206242Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:45:15Z" } 2024-04-12T03:45:15.965 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.206242_362.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.206242Z', 'realm_epoch': 2}}] 2024-04-12T03:45:15.965 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:45:15.966 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.206242_362.1'} 2024-04-12T03:45:15.966 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.206242_362.1 2024-04-12T03:45:20.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:20.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:20.967 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:45:21.027 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:21.027 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:21.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:21.139+0000 7fb794f42780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:21.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:21.141+0000 7fb794f42780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:21.157 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:45:21.157 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:45:21.157 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:45:21.158 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:45:21.158 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:45:21.158 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:45:21.158 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:21.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:21.158 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:45:21.158 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:21.158 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:45:21.158 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:21.158 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:21.159 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:45:21.159 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:21.159 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:45:21.159 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:21.159 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:45:21.159 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:21.159 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:21.159 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:21.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:21.159 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:45:21.159 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:21.160 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:21.160 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:45:21.160 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:21.160 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:21.160 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:21.160 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:45:21.160 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:21.160 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:21.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:21.160 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:21.160 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:45:21.161 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:21.161 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:21.161 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:45:21.161 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:21.161 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:21.161 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:21.161 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:45:21.161 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:21.161 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:21.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:21.161 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:21.162 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:45:21.162 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:21.162 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:21.162 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.145385_357.1", 2024-04-12T03:45:21.162 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:21.162 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:21.162 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:21.162 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.145385Z", 2024-04-12T03:45:21.162 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:21.162 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:21.163 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:21.163 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:45:21.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:21.163 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:45:21.163 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:45:21.163 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:45:21.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:21.163 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:45:21Z" 2024-04-12T03:45:21.163 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:45:21.164 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.145385_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.145385Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:45:21Z" } 2024-04-12T03:45:21.164 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.145385_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.145385Z', 'realm_epoch': 2}}] 2024-04-12T03:45:21.165 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:45:21.165 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.145385_357.1'} 2024-04-12T03:45:21.165 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.145385_357.1 2024-04-12T03:45:26.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:26.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:26.166 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:45:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:26.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:26.258+0000 7f61ddb48780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:26.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:26.259+0000 7f61ddb48780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:26.275 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:45:26.275 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:45:26.275 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:45:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:45:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:45:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:45:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:45:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:45:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:45:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:45:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:45:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:45:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:45:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:45:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:45:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:45:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:45:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:45:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893271.214527_380.1", 2024-04-12T03:45:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:11.214527Z", 2024-04-12T03:45:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:26.281 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:45:26.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:26.281 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:45:26.281 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:45:26.281 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:45:26.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:26.281 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:45:26Z" 2024-04-12T03:45:26.281 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:45:26.282 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893271.214527_380.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:11.214527Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:45:26Z" } 2024-04-12T03:45:26.282 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893271.214527_380.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:11.214527Z', 'realm_epoch': 2}}] 2024-04-12T03:45:26.283 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:45:26.283 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893271.214527_380.1'} 2024-04-12T03:45:26.283 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893271.214527_380.1 2024-04-12T03:45:31.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:31.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:31.284 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:45:31.347 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:31.348 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:31.383+0000 7f5cd61ee780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:31.384+0000 7f5cd61ee780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:31.400 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:45:31.400 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:45:31.400 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:45:31.400 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:45:31.400 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:45:31.400 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:45:31.400 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:31.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:31.400 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:45:31.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:31.401 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:45:31.401 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:31.401 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:31.401 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:45:31.401 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:31.401 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:45:31.401 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:31.401 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:45:31.401 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:31.401 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:31.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:31.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:31.402 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:45:31.402 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:31.402 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:31.402 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:45:31.402 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:31.402 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:31.402 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:31.402 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:45:31.402 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:31.403 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:31.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:31.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:31.403 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:45:31.403 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:31.403 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:31.403 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:45:31.404 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:31.404 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:31.404 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:31.404 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:45:31.404 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:31.404 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:31.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:31.404 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:31.404 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:45:31.404 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:31.405 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:31.405 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.226327_365.1", 2024-04-12T03:45:31.405 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:31.405 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:31.405 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:31.405 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.226327Z", 2024-04-12T03:45:31.405 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:31.405 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:31.405 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:31.405 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:45:31.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:31.406 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:45:31.406 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:45:31.406 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:45:31.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:31.406 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:45:31Z" 2024-04-12T03:45:31.406 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:45:31.407 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.226327_365.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.226327Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:45:31Z" } 2024-04-12T03:45:31.407 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.226327_365.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.226327Z', 'realm_epoch': 2}}] 2024-04-12T03:45:31.407 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:45:31.408 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.226327_365.1'} 2024-04-12T03:45:31.408 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.226327_365.1 2024-04-12T03:45:36.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:36.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:36.409 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:45:36.469 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:36.469 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:36.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:36.503+0000 7f14ff5b6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:36.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:36.504+0000 7f14ff5b6780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:36.520 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:45:36.521 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:45:36.521 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:45:36.521 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:45:36.521 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:45:36.521 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:45:36.521 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:36.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:36.521 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:45:36.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:36.521 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:45:36.521 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:36.521 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:36.522 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:45:36.522 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:36.522 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:45:36.522 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:36.522 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:45:36.522 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:36.522 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:36.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:36.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:36.522 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:45:36.522 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:36.523 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:36.523 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:45:36.523 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:36.523 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:36.523 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:36.523 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:45:36.523 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:36.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:36.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:36.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:36.523 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:45:36.523 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:36.524 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:36.524 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:45:36.524 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:36.524 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:36.524 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:36.524 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:45:36.524 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:36.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:36.524 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:36.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:36.524 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:45:36.524 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:36.525 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:36.525 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.145385_357.1", 2024-04-12T03:45:36.525 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:36.525 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:36.525 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:36.525 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.145385Z", 2024-04-12T03:45:36.525 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:36.525 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:36.525 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:36.525 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:45:36.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:36.525 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:45:36.526 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:45:36.526 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:45:36.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:36.526 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:45:36Z" 2024-04-12T03:45:36.526 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:45:36.527 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.145385_357.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.145385Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:45:36Z" } 2024-04-12T03:45:36.527 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.145385_357.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.145385Z', 'realm_epoch': 2}}] 2024-04-12T03:45:36.527 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:45:36.528 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.145385_357.1'} 2024-04-12T03:45:36.528 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.145385_357.1 2024-04-12T03:45:41.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:41.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:41.529 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:45:41.621 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:41.622 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:41.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:41.663+0000 7f418b91c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:41.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:41.664+0000 7f418b91c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:41.679 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:45:41.679 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:45:41.680 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:45:41.680 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:45:41.680 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:45:41.680 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:45:41.680 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:41.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:41.680 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:45:41.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:41.680 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:45:41.680 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:41.681 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:41.681 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:45:41.681 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:41.681 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:45:41.681 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:41.681 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:45:41.681 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:41.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:41.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:41.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:41.681 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:45:41.682 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:41.682 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:41.682 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:45:41.682 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:41.682 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:41.682 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:41.682 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:45:41.682 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:41.682 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:41.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:41.683 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:41.683 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:45:41.683 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:41.683 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:41.683 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:45:41.683 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:41.683 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:41.683 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:41.683 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:45:41.683 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:41.683 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:41.683 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:41.684 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:41.684 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:45:41.684 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:41.684 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:41.684 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.738847_372.1", 2024-04-12T03:45:41.684 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:41.684 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:41.684 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:41.684 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.738847Z", 2024-04-12T03:45:41.684 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:41.684 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:41.685 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:41.685 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:45:41.685 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:41.685 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:45:41.685 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:45:41.685 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:45:41.685 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:41.685 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:45:41Z" 2024-04-12T03:45:41.685 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:45:41.686 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.738847_372.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.738847Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:45:41Z" } 2024-04-12T03:45:41.686 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.738847_372.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.738847Z', 'realm_epoch': 2}}] 2024-04-12T03:45:41.687 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:45:41.687 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.738847_372.1'} 2024-04-12T03:45:41.687 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.738847_372.1 2024-04-12T03:45:46.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:46.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:46.688 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:45:46.748 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:46.748 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:46.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:46.787+0000 7f7ddd2db780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:46.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:46.788+0000 7f7ddd2db780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:46.804 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:45:46.804 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:45:46.804 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:45:46.804 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:45:46.804 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:45:46.804 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:45:46.805 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:46.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:46.805 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:45:46.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:46.805 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:45:46.805 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:46.805 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:46.805 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:45:46.805 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:46.805 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:45:46.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:46.806 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:45:46.806 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:46.806 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:46.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:46.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:46.806 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:45:46.806 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:46.806 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:46.806 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:45:46.806 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:46.806 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:46.806 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:46.807 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:45:46.807 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:46.807 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:46.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:46.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:46.807 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:45:46.807 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:46.807 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:46.807 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:45:46.807 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:46.807 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:46.808 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:46.808 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:45:46.808 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:46.808 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:46.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:46.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:46.808 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:45:46.808 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:46.808 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:46.808 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.166984_359.1", 2024-04-12T03:45:46.808 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:46.809 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:46.809 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:46.809 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.166984Z", 2024-04-12T03:45:46.809 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:46.809 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:46.809 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:46.809 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:45:46.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:46.809 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:45:46.809 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:45:46.809 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:45:46.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:46.810 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:45:46Z" 2024-04-12T03:45:46.810 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:45:46.811 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.166984_359.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.166984Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:45:46Z" } 2024-04-12T03:45:46.811 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.166984_359.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.166984Z', 'realm_epoch': 2}}] 2024-04-12T03:45:46.811 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:45:46.811 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.166984_359.1'} 2024-04-12T03:45:46.811 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.166984_359.1 2024-04-12T03:45:51.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:51.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:51.812 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:45:51.872 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:51.872 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:51.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:51.907+0000 7f3d8e1bb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:51.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:51.909+0000 7f3d8e1bb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:51.924 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:45:51.924 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:45:51.924 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:45:51.924 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:45:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:45:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:45:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:45:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:45:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:51.925 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:51.926 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:45:51.926 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:51.926 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:45:51.926 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:51.926 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:45:51.926 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:51.926 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:51.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:51.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:51.927 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:45:51.927 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:51.927 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:51.927 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:45:51.928 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:51.928 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:51.928 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:51.928 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:45:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:45:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:45:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:51.929 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:45:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:45:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893270.214735_363.1", 2024-04-12T03:45:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:51.930 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:10.214735Z", 2024-04-12T03:45:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:45:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:45:51.931 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:45:51.932 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:45:51.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:51.932 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:45:51Z" 2024-04-12T03:45:51.932 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:45:51.933 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893270.214735_363.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:10.214735Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:45:51Z" } 2024-04-12T03:45:51.933 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893270.214735_363.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:10.214735Z', 'realm_epoch': 2}}] 2024-04-12T03:45:51.934 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:45:51.934 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893270.214735_363.1'} 2024-04-12T03:45:51.934 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893274.957117_411.1 target=1_1712893270.214735_363.1 2024-04-12T03:45:56.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:56.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:56.936 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:45:56.997 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:56.997 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:57.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:57.033+0000 7f7bfa3be780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:57.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:45:57.034+0000 7f7bfa3be780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:45:57.049 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:45:57.049 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:45:57.049 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:45:57.049 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:45:57.049 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:45:57.049 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:45:57.049 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:57.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:57.050 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:45:57.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:57.050 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:45:57.050 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:57.050 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:57.050 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:45:57.051 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:57.051 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:45:57.051 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:57.051 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:45:57.051 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:57.051 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:57.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:57.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:57.051 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:45:57.051 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:57.052 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:57.052 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:45:57.052 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:57.052 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:57.052 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:57.052 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:45:57.052 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:57.052 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:57.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:57.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:57.053 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:45:57.053 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:57.053 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:57.053 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:45:57.053 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:57.053 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:57.053 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:57.053 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:45:57.053 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:57.054 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:57.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:57.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:57.054 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:45:57.054 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:57.054 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:45:57.054 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893274.957117_411.1", 2024-04-12T03:45:57.054 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:57.054 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:57.054 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:57.054 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:41:14.957117Z", 2024-04-12T03:45:57.054 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:45:57.055 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:57.055 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:57.055 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:45:57.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:57.055 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:45:57.055 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:45:57.055 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:45:57.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:57.056 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:45:57Z" 2024-04-12T03:45:57.056 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:45:57.057 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893274.957117_411.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:41:14.957117Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:45:57Z" } 2024-04-12T03:45:57.057 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893274.957117_411.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:41:14.957117Z', 'realm_epoch': 2}}] 2024-04-12T03:45:57.057 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:45:57.057 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893274.957117_411.1'} 2024-04-12T03:45:57.057 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:45:57.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dkyges-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:45:57.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dkyges-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:45:57.058 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dkyges-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:45:57.115 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:57.115 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:57.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dkyges-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:45:57.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dkyges-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:45:57.202 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dkyges-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:45:57.258 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:57.258 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:57.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:57.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:57.333 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:45:57.385 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:57.385 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:57.435 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:45:57.435 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.29", 2024-04-12T03:45:57.435 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000078", 2024-04-12T03:45:57.435 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:1", 2024-04-12T03:45:57.435 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:10", 2024-04-12T03:45:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "meta.history", 2024-04-12T03:45:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:4", 2024-04-12T03:45:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:4", 2024-04-12T03:45:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19", 2024-04-12T03:45:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.23", 2024-04-12T03:45:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000070", 2024-04-12T03:45:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000104", 2024-04-12T03:45:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:7", 2024-04-12T03:45:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:7", 2024-04-12T03:45:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000026", 2024-04-12T03:45:57.436 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.8", 2024-04-12T03:45:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000028", 2024-04-12T03:45:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:0", 2024-04-12T03:45:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:6", 2024-04-12T03:45:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:8", 2024-04-12T03:45:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.30", 2024-04-12T03:45:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:2", 2024-04-12T03:45:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000040", 2024-04-12T03:45:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:9", 2024-04-12T03:45:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8", 2024-04-12T03:45:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.6", 2024-04-12T03:45:57.437 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18", 2024-04-12T03:45:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000015", 2024-04-12T03:45:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:8", 2024-04-12T03:45:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000069", 2024-04-12T03:45:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000095", 2024-04-12T03:45:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000003", 2024-04-12T03:45:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000047", 2024-04-12T03:45:57.438 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.shard.1ba046e0-d0ae-437d-b741-28b4299423c2.3.retry", 2024-04-12T03:45:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:0", 2024-04-12T03:45:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:2", 2024-04-12T03:45:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000035", 2024-04-12T03:45:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:6", 2024-04-12T03:45:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000037", 2024-04-12T03:45:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:7", 2024-04-12T03:45:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:10", 2024-04-12T03:45:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:7", 2024-04-12T03:45:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:5", 2024-04-12T03:45:57.439 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:4", 2024-04-12T03:45:57.440 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000024", 2024-04-12T03:45:57.440 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000000", 2024-04-12T03:45:57.440 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000031", 2024-04-12T03:45:57.440 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:2", 2024-04-12T03:45:57.440 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:10", 2024-04-12T03:45:57.441 INFO:teuthology.orchestra.run.smithi043.stdout: "data_log.3", 2024-04-12T03:45:57.441 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:1", 2024-04-12T03:45:57.441 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:8", 2024-04-12T03:45:57.441 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:6", 2024-04-12T03:45:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34", 2024-04-12T03:45:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "data_loggenerations_metadata", 2024-04-12T03:45:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:9", 2024-04-12T03:45:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:5", 2024-04-12T03:45:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:1", 2024-04-12T03:45:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000076", 2024-04-12T03:45:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000113", 2024-04-12T03:45:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000005", 2024-04-12T03:45:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:9", 2024-04-12T03:45:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000011", 2024-04-12T03:45:57.442 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:1", 2024-04-12T03:45:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000058", 2024-04-12T03:45:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:4", 2024-04-12T03:45:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000052", 2024-04-12T03:45:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:9", 2024-04-12T03:45:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:2", 2024-04-12T03:45:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:9", 2024-04-12T03:45:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.21", 2024-04-12T03:45:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:3", 2024-04-12T03:45:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17", 2024-04-12T03:45:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:6", 2024-04-12T03:45:57.443 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000088", 2024-04-12T03:45:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000080", 2024-04-12T03:45:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000090", 2024-04-12T03:45:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:3", 2024-04-12T03:45:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:2", 2024-04-12T03:45:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000110", 2024-04-12T03:45:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:5", 2024-04-12T03:45:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:8", 2024-04-12T03:45:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.0", 2024-04-12T03:45:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000096", 2024-04-12T03:45:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:10", 2024-04-12T03:45:57.444 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:4", 2024-04-12T03:45:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000087", 2024-04-12T03:45:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000008", 2024-04-12T03:45:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000006", 2024-04-12T03:45:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12", 2024-04-12T03:45:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000029", 2024-04-12T03:45:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33", 2024-04-12T03:45:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20", 2024-04-12T03:45:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:10", 2024-04-12T03:45:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000089", 2024-04-12T03:45:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:9", 2024-04-12T03:45:57.445 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000022", 2024-04-12T03:45:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:3", 2024-04-12T03:45:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:7", 2024-04-12T03:45:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:10", 2024-04-12T03:45:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "bilog.trim", 2024-04-12T03:45:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000016", 2024-04-12T03:45:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:5", 2024-04-12T03:45:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000041", 2024-04-12T03:45:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:6", 2024-04-12T03:45:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7", 2024-04-12T03:45:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000018", 2024-04-12T03:45:57.446 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000092", 2024-04-12T03:45:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:1", 2024-04-12T03:45:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:0", 2024-04-12T03:45:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.27", 2024-04-12T03:45:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:9", 2024-04-12T03:45:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.4", 2024-04-12T03:45:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:8", 2024-04-12T03:45:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:7", 2024-04-12T03:45:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000014", 2024-04-12T03:45:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:7", 2024-04-12T03:45:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:6", 2024-04-12T03:45:57.447 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000112", 2024-04-12T03:45:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000007", 2024-04-12T03:45:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000021", 2024-04-12T03:45:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:9", 2024-04-12T03:45:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:2", 2024-04-12T03:45:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:0", 2024-04-12T03:45:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.16", 2024-04-12T03:45:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000064", 2024-04-12T03:45:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000071", 2024-04-12T03:45:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:10", 2024-04-12T03:45:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000074", 2024-04-12T03:45:57.448 INFO:teuthology.orchestra.run.smithi043.stdout: "meta.log.b69a664f-970b-4d72-be0d-0071b991ee5c.3", 2024-04-12T03:45:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.shard.1ba046e0-d0ae-437d-b741-28b4299423c2.2.retry", 2024-04-12T03:45:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:1", 2024-04-12T03:45:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000081", 2024-04-12T03:45:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000009", 2024-04-12T03:45:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:4", 2024-04-12T03:45:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:2", 2024-04-12T03:45:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000121", 2024-04-12T03:45:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "data_log.1", 2024-04-12T03:45:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1", 2024-04-12T03:45:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:1", 2024-04-12T03:45:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:3", 2024-04-12T03:45:57.449 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.17", 2024-04-12T03:45:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000125", 2024-04-12T03:45:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000082", 2024-04-12T03:45:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9", 2024-04-12T03:45:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:0", 2024-04-12T03:45:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:5", 2024-04-12T03:45:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:7", 2024-04-12T03:45:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000105", 2024-04-12T03:45:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000059", 2024-04-12T03:45:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10", 2024-04-12T03:45:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:1", 2024-04-12T03:45:57.450 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:7", 2024-04-12T03:45:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:10", 2024-04-12T03:45:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000077", 2024-04-12T03:45:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000032", 2024-04-12T03:45:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.24", 2024-04-12T03:45:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:4", 2024-04-12T03:45:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000053", 2024-04-12T03:45:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000091", 2024-04-12T03:45:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:0", 2024-04-12T03:45:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:4", 2024-04-12T03:45:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11", 2024-04-12T03:45:57.451 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000065", 2024-04-12T03:45:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:5", 2024-04-12T03:45:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000083", 2024-04-12T03:45:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000010", 2024-04-12T03:45:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:3", 2024-04-12T03:45:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:7", 2024-04-12T03:45:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "data_log.3.DC67N25C/+5.0", 2024-04-12T03:45:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:10", 2024-04-12T03:45:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:3", 2024-04-12T03:45:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.7", 2024-04-12T03:45:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.3", 2024-04-12T03:45:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:7", 2024-04-12T03:45:57.452 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:4", 2024-04-12T03:45:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "data_log.2.D9sMDKuVO/O.0", 2024-04-12T03:45:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:5", 2024-04-12T03:45:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15", 2024-04-12T03:45:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:6", 2024-04-12T03:45:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000045", 2024-04-12T03:45:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:1", 2024-04-12T03:45:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14", 2024-04-12T03:45:57.453 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000002", 2024-04-12T03:45:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13", 2024-04-12T03:45:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.12", 2024-04-12T03:45:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:6", 2024-04-12T03:45:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000116", 2024-04-12T03:45:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.25", 2024-04-12T03:45:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:4", 2024-04-12T03:45:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:0", 2024-04-12T03:45:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:8", 2024-04-12T03:45:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000034", 2024-04-12T03:45:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000101", 2024-04-12T03:45:57.454 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.13", 2024-04-12T03:45:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:4", 2024-04-12T03:45:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:7", 2024-04-12T03:45:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:6", 2024-04-12T03:45:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000079", 2024-04-12T03:45:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:1", 2024-04-12T03:45:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000049", 2024-04-12T03:45:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:9", 2024-04-12T03:45:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000117", 2024-04-12T03:45:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000044", 2024-04-12T03:45:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:2", 2024-04-12T03:45:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:6", 2024-04-12T03:45:57.455 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:6", 2024-04-12T03:45:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:8", 2024-04-12T03:45:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.14", 2024-04-12T03:45:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000066", 2024-04-12T03:45:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:4", 2024-04-12T03:45:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:7", 2024-04-12T03:45:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000068", 2024-04-12T03:45:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:8", 2024-04-12T03:45:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000085", 2024-04-12T03:45:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:9", 2024-04-12T03:45:57.456 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:4", 2024-04-12T03:45:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.shard.1ba046e0-d0ae-437d-b741-28b4299423c2.1", 2024-04-12T03:45:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:0", 2024-04-12T03:45:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:6", 2024-04-12T03:45:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000073", 2024-04-12T03:45:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000038", 2024-04-12T03:45:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.10", 2024-04-12T03:45:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000118", 2024-04-12T03:45:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000036", 2024-04-12T03:45:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:0", 2024-04-12T03:45:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:3", 2024-04-12T03:45:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000103", 2024-04-12T03:45:57.457 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.shard.1ba046e0-d0ae-437d-b741-28b4299423c2.0.retry", 2024-04-12T03:45:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000119", 2024-04-12T03:45:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000098", 2024-04-12T03:45:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:5", 2024-04-12T03:45:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:1", 2024-04-12T03:45:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:0", 2024-04-12T03:45:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:4", 2024-04-12T03:45:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000027", 2024-04-12T03:45:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "meta.log.b69a664f-970b-4d72-be0d-0071b991ee5c.0", 2024-04-12T03:45:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:4", 2024-04-12T03:45:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000019", 2024-04-12T03:45:57.458 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000039", 2024-04-12T03:45:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:0", 2024-04-12T03:45:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:8", 2024-04-12T03:45:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.shard.1ba046e0-d0ae-437d-b741-28b4299423c2.1.retry", 2024-04-12T03:45:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:3", 2024-04-12T03:45:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:9", 2024-04-12T03:45:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "data_log.0.EhRAW8EBARY.0", 2024-04-12T03:45:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000100", 2024-04-12T03:45:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:8", 2024-04-12T03:45:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000093", 2024-04-12T03:45:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:1", 2024-04-12T03:45:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:2", 2024-04-12T03:45:57.459 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000004", 2024-04-12T03:45:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000063", 2024-04-12T03:45:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000122", 2024-04-12T03:45:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:0", 2024-04-12T03:45:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000057", 2024-04-12T03:45:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000054", 2024-04-12T03:45:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000114", 2024-04-12T03:45:57.460 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.2", 2024-04-12T03:45:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: "data_log.0", 2024-04-12T03:45:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:0", 2024-04-12T03:45:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000012", 2024-04-12T03:45:57.461 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:1", 2024-04-12T03:45:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:0", 2024-04-12T03:45:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000084", 2024-04-12T03:45:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000043", 2024-04-12T03:45:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:9", 2024-04-12T03:45:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:3", 2024-04-12T03:45:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.11", 2024-04-12T03:45:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:10", 2024-04-12T03:45:57.462 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000111", 2024-04-12T03:45:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000048", 2024-04-12T03:45:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000020", 2024-04-12T03:45:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:2", 2024-04-12T03:45:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:2", 2024-04-12T03:45:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000099", 2024-04-12T03:45:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000056", 2024-04-12T03:45:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:8", 2024-04-12T03:45:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:5", 2024-04-12T03:45:57.463 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.19", 2024-04-12T03:45:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.28", 2024-04-12T03:45:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000072", 2024-04-12T03:45:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:0", 2024-04-12T03:45:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000062", 2024-04-12T03:45:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:2", 2024-04-12T03:45:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:5", 2024-04-12T03:45:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000115", 2024-04-12T03:45:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.1", 2024-04-12T03:45:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000033", 2024-04-12T03:45:57.464 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:9", 2024-04-12T03:45:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:10", 2024-04-12T03:45:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:1", 2024-04-12T03:45:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:4", 2024-04-12T03:45:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:10", 2024-04-12T03:45:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:3", 2024-04-12T03:45:57.465 INFO:teuthology.orchestra.run.smithi043.stdout: "data_log.1.vaHRMz+CsDh.0", 2024-04-12T03:45:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:9", 2024-04-12T03:45:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:6", 2024-04-12T03:45:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:3", 2024-04-12T03:45:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:7", 2024-04-12T03:45:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35", 2024-04-12T03:45:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000025", 2024-04-12T03:45:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000123", 2024-04-12T03:45:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:6", 2024-04-12T03:45:57.466 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.31", 2024-04-12T03:45:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:8", 2024-04-12T03:45:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:5", 2024-04-12T03:45:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.18", 2024-04-12T03:45:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000108", 2024-04-12T03:45:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000094", 2024-04-12T03:45:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:4", 2024-04-12T03:45:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000060", 2024-04-12T03:45:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:10", 2024-04-12T03:45:57.467 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:0", 2024-04-12T03:45:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:8", 2024-04-12T03:45:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000109", 2024-04-12T03:45:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.15", 2024-04-12T03:45:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:5", 2024-04-12T03:45:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:5", 2024-04-12T03:45:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:7", 2024-04-12T03:45:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000013", 2024-04-12T03:45:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:10", 2024-04-12T03:45:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:10", 2024-04-12T03:45:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:6", 2024-04-12T03:45:57.468 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:2", 2024-04-12T03:45:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:8", 2024-04-12T03:45:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16", 2024-04-12T03:45:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000051", 2024-04-12T03:45:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:9", 2024-04-12T03:45:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000106", 2024-04-12T03:45:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:1", 2024-04-12T03:45:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.shard.1ba046e0-d0ae-437d-b741-28b4299423c2.3", 2024-04-12T03:45:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:2", 2024-04-12T03:45:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000017", 2024-04-12T03:45:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:0", 2024-04-12T03:45:57.469 INFO:teuthology.orchestra.run.smithi043.stdout: "meta.log.b69a664f-970b-4d72-be0d-0071b991ee5c.2", 2024-04-12T03:45:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000023", 2024-04-12T03:45:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:1", 2024-04-12T03:45:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:3", 2024-04-12T03:45:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:10", 2024-04-12T03:45:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:6", 2024-04-12T03:45:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:8", 2024-04-12T03:45:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000107", 2024-04-12T03:45:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:1", 2024-04-12T03:45:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000046", 2024-04-12T03:45:57.470 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.26", 2024-04-12T03:45:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:3", 2024-04-12T03:45:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:8", 2024-04-12T03:45:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:1", 2024-04-12T03:45:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.22", 2024-04-12T03:45:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:9", 2024-04-12T03:45:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:2", 2024-04-12T03:45:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "data_log.2", 2024-04-12T03:45:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:10", 2024-04-12T03:45:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:3", 2024-04-12T03:45:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000067", 2024-04-12T03:45:57.471 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:9", 2024-04-12T03:45:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:7", 2024-04-12T03:45:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:3", 2024-04-12T03:45:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.9", 2024-04-12T03:45:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000086", 2024-04-12T03:45:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.5", 2024-04-12T03:45:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:1", 2024-04-12T03:45:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:2", 2024-04-12T03:45:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:7", 2024-04-12T03:45:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:3", 2024-04-12T03:45:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:3", 2024-04-12T03:45:57.472 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.20", 2024-04-12T03:45:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:8", 2024-04-12T03:45:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:4", 2024-04-12T03:45:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T03:45:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:7", 2024-04-12T03:45:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:8", 2024-04-12T03:45:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36", 2024-04-12T03:45:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:6", 2024-04-12T03:45:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:0", 2024-04-12T03:45:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000050", 2024-04-12T03:45:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:10", 2024-04-12T03:45:57.473 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.shard.1ba046e0-d0ae-437d-b741-28b4299423c2.2", 2024-04-12T03:45:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000102", 2024-04-12T03:45:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:2", 2024-04-12T03:45:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:9", 2024-04-12T03:45:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:2", 2024-04-12T03:45:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000124", 2024-04-12T03:45:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:5", 2024-04-12T03:45:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000001", 2024-04-12T03:45:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:7", 2024-04-12T03:45:57.474 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000126", 2024-04-12T03:45:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000055", 2024-04-12T03:45:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000061", 2024-04-12T03:45:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:8", 2024-04-12T03:45:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:5", 2024-04-12T03:45:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000097", 2024-04-12T03:45:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:5", 2024-04-12T03:45:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.shard.1ba046e0-d0ae-437d-b741-28b4299423c2.0", 2024-04-12T03:45:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:5", 2024-04-12T03:45:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:4", 2024-04-12T03:45:57.475 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000042", 2024-04-12T03:45:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:5", 2024-04-12T03:45:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000075", 2024-04-12T03:45:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:6", 2024-04-12T03:45:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:0", 2024-04-12T03:45:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000120", 2024-04-12T03:45:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000030", 2024-04-12T03:45:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:9", 2024-04-12T03:45:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:5", 2024-04-12T03:45:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:6", 2024-04-12T03:45:57.476 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:3", 2024-04-12T03:45:57.477 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:3", 2024-04-12T03:45:57.477 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:2", 2024-04-12T03:45:57.477 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:10" 2024-04-12T03:45:57.477 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:45:57.477 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:45:57.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:57.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:57.479 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:45:57.532 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:57.533 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:57.586 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:45:57.586 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.29", 2024-04-12T03:45:57.586 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000078", 2024-04-12T03:45:57.586 INFO:teuthology.orchestra.run.smithi043.stdout: "meta.history", 2024-04-12T03:45:57.586 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:8", 2024-04-12T03:45:57.586 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.23", 2024-04-12T03:45:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:8", 2024-04-12T03:45:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000070", 2024-04-12T03:45:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000104", 2024-04-12T03:45:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:9", 2024-04-12T03:45:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000026", 2024-04-12T03:45:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.8", 2024-04-12T03:45:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:1", 2024-04-12T03:45:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:7", 2024-04-12T03:45:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000028", 2024-04-12T03:45:57.587 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:4", 2024-04-12T03:45:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.30", 2024-04-12T03:45:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:7", 2024-04-12T03:45:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000040", 2024-04-12T03:45:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:0", 2024-04-12T03:45:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.6", 2024-04-12T03:45:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:7", 2024-04-12T03:45:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:10", 2024-04-12T03:45:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000015", 2024-04-12T03:45:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:0", 2024-04-12T03:45:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000069", 2024-04-12T03:45:57.588 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000095", 2024-04-12T03:45:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000003", 2024-04-12T03:45:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:5", 2024-04-12T03:45:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000047", 2024-04-12T03:45:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:10", 2024-04-12T03:45:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000035", 2024-04-12T03:45:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000037", 2024-04-12T03:45:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:5", 2024-04-12T03:45:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:10", 2024-04-12T03:45:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:3", 2024-04-12T03:45:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15", 2024-04-12T03:45:57.589 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:3", 2024-04-12T03:45:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:0", 2024-04-12T03:45:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:2", 2024-04-12T03:45:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000024", 2024-04-12T03:45:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:2", 2024-04-12T03:45:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000000", 2024-04-12T03:45:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:8", 2024-04-12T03:45:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000031", 2024-04-12T03:45:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:2", 2024-04-12T03:45:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:0", 2024-04-12T03:45:57.590 INFO:teuthology.orchestra.run.smithi043.stdout: "data_log.3", 2024-04-12T03:45:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:5", 2024-04-12T03:45:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20", 2024-04-12T03:45:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.shard.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.2.retry", 2024-04-12T03:45:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "data_loggenerations_metadata", 2024-04-12T03:45:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:8", 2024-04-12T03:45:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:10", 2024-04-12T03:45:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000076", 2024-04-12T03:45:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000113", 2024-04-12T03:45:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000005", 2024-04-12T03:45:57.591 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:2", 2024-04-12T03:45:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000011", 2024-04-12T03:45:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000058", 2024-04-12T03:45:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18", 2024-04-12T03:45:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000052", 2024-04-12T03:45:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:5", 2024-04-12T03:45:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:8", 2024-04-12T03:45:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.21", 2024-04-12T03:45:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:10", 2024-04-12T03:45:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:1", 2024-04-12T03:45:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:9", 2024-04-12T03:45:57.592 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:1", 2024-04-12T03:45:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:10", 2024-04-12T03:45:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:4", 2024-04-12T03:45:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:6", 2024-04-12T03:45:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000088", 2024-04-12T03:45:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000080", 2024-04-12T03:45:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000090", 2024-04-12T03:45:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:1", 2024-04-12T03:45:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "data_log.3.4WHk9YLCxzY.0", 2024-04-12T03:45:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000110", 2024-04-12T03:45:57.593 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:4", 2024-04-12T03:45:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:4", 2024-04-12T03:45:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.0", 2024-04-12T03:45:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:10", 2024-04-12T03:45:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000096", 2024-04-12T03:45:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:0", 2024-04-12T03:45:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000087", 2024-04-12T03:45:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000008", 2024-04-12T03:45:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:9", 2024-04-12T03:45:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:1", 2024-04-12T03:45:57.594 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:8", 2024-04-12T03:45:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:0", 2024-04-12T03:45:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000006", 2024-04-12T03:45:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000029", 2024-04-12T03:45:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:5", 2024-04-12T03:45:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000089", 2024-04-12T03:45:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000022", 2024-04-12T03:45:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:2", 2024-04-12T03:45:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:4", 2024-04-12T03:45:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "bilog.trim", 2024-04-12T03:45:57.595 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000016", 2024-04-12T03:45:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:7", 2024-04-12T03:45:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:2", 2024-04-12T03:45:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "data_log.2.ss2P02AgxOo.0", 2024-04-12T03:45:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:6", 2024-04-12T03:45:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:6", 2024-04-12T03:45:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000041", 2024-04-12T03:45:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:10", 2024-04-12T03:45:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000018", 2024-04-12T03:45:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:9", 2024-04-12T03:45:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000092", 2024-04-12T03:45:57.596 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:4", 2024-04-12T03:45:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:7", 2024-04-12T03:45:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.27", 2024-04-12T03:45:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:3", 2024-04-12T03:45:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.4", 2024-04-12T03:45:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:4", 2024-04-12T03:45:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:1", 2024-04-12T03:45:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:7", 2024-04-12T03:45:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000014", 2024-04-12T03:45:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:5", 2024-04-12T03:45:57.597 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000112", 2024-04-12T03:45:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:7", 2024-04-12T03:45:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000007", 2024-04-12T03:45:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000021", 2024-04-12T03:45:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.16", 2024-04-12T03:45:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000064", 2024-04-12T03:45:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.shard.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.0.retry", 2024-04-12T03:45:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:8", 2024-04-12T03:45:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000071", 2024-04-12T03:45:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000074", 2024-04-12T03:45:57.598 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:8", 2024-04-12T03:45:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:4", 2024-04-12T03:45:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "meta.log.b69a664f-970b-4d72-be0d-0071b991ee5c.3", 2024-04-12T03:45:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:4", 2024-04-12T03:45:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000081", 2024-04-12T03:45:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000009", 2024-04-12T03:45:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000121", 2024-04-12T03:45:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "data_log.1", 2024-04-12T03:45:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:5", 2024-04-12T03:45:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:10", 2024-04-12T03:45:57.599 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.17", 2024-04-12T03:45:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:4", 2024-04-12T03:45:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000125", 2024-04-12T03:45:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000082", 2024-04-12T03:45:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:6", 2024-04-12T03:45:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16", 2024-04-12T03:45:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:4", 2024-04-12T03:45:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:0", 2024-04-12T03:45:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000105", 2024-04-12T03:45:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000059", 2024-04-12T03:45:57.600 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:6", 2024-04-12T03:45:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T03:45:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000077", 2024-04-12T03:45:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:6", 2024-04-12T03:45:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000032", 2024-04-12T03:45:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:2", 2024-04-12T03:45:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.24", 2024-04-12T03:45:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:6", 2024-04-12T03:45:57.601 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000053", 2024-04-12T03:45:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000091", 2024-04-12T03:45:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11", 2024-04-12T03:45:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000065", 2024-04-12T03:45:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000083", 2024-04-12T03:45:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000010", 2024-04-12T03:45:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.7", 2024-04-12T03:45:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.3", 2024-04-12T03:45:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:9", 2024-04-12T03:45:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:7", 2024-04-12T03:45:57.602 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:1", 2024-04-12T03:45:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:0", 2024-04-12T03:45:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:4", 2024-04-12T03:45:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:9", 2024-04-12T03:45:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:10", 2024-04-12T03:45:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mdlog.sync-status.shard.2", 2024-04-12T03:45:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.shard.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.1", 2024-04-12T03:45:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000045", 2024-04-12T03:45:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13", 2024-04-12T03:45:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000002", 2024-04-12T03:45:57.603 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.12", 2024-04-12T03:45:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:5", 2024-04-12T03:45:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000116", 2024-04-12T03:45:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.25", 2024-04-12T03:45:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000034", 2024-04-12T03:45:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:1", 2024-04-12T03:45:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:0", 2024-04-12T03:45:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14", 2024-04-12T03:45:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000101", 2024-04-12T03:45:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.13", 2024-04-12T03:45:57.604 INFO:teuthology.orchestra.run.smithi043.stdout: "mdlog.sync-status.shard.1", 2024-04-12T03:45:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7", 2024-04-12T03:45:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000079", 2024-04-12T03:45:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000049", 2024-04-12T03:45:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36", 2024-04-12T03:45:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000117", 2024-04-12T03:45:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000044", 2024-04-12T03:45:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:9", 2024-04-12T03:45:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.shard.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.3", 2024-04-12T03:45:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.14", 2024-04-12T03:45:57.605 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:6", 2024-04-12T03:45:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:5", 2024-04-12T03:45:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000066", 2024-04-12T03:45:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:4", 2024-04-12T03:45:57.606 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34", 2024-04-12T03:45:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000068", 2024-04-12T03:45:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000085", 2024-04-12T03:45:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:2", 2024-04-12T03:45:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:3", 2024-04-12T03:45:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:3", 2024-04-12T03:45:57.607 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000073", 2024-04-12T03:45:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:5", 2024-04-12T03:45:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000038", 2024-04-12T03:45:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "data_log.1.fVBlpQNCQmA.0", 2024-04-12T03:45:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:3", 2024-04-12T03:45:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:7", 2024-04-12T03:45:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.10", 2024-04-12T03:45:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:0", 2024-04-12T03:45:57.608 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000118", 2024-04-12T03:45:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:3", 2024-04-12T03:45:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:1", 2024-04-12T03:45:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000036", 2024-04-12T03:45:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:3", 2024-04-12T03:45:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:7", 2024-04-12T03:45:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000103", 2024-04-12T03:45:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "mdlog.sync-status.shard.3", 2024-04-12T03:45:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:4", 2024-04-12T03:45:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:2", 2024-04-12T03:45:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000119", 2024-04-12T03:45:57.609 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000098", 2024-04-12T03:45:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:0", 2024-04-12T03:45:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35", 2024-04-12T03:45:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:6", 2024-04-12T03:45:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:3", 2024-04-12T03:45:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.shard.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.1.retry", 2024-04-12T03:45:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mdlog.sync-status.shard.0", 2024-04-12T03:45:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000027", 2024-04-12T03:45:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "meta.log.b69a664f-970b-4d72-be0d-0071b991ee5c.0", 2024-04-12T03:45:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:6", 2024-04-12T03:45:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:3", 2024-04-12T03:45:57.610 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:5", 2024-04-12T03:45:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:10", 2024-04-12T03:45:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "mdlog.sync-status", 2024-04-12T03:45:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000019", 2024-04-12T03:45:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:2", 2024-04-12T03:45:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:3", 2024-04-12T03:45:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:10", 2024-04-12T03:45:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000039", 2024-04-12T03:45:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:9", 2024-04-12T03:45:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8", 2024-04-12T03:45:57.611 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33", 2024-04-12T03:45:57.612 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.shard.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.3.retry", 2024-04-12T03:45:57.612 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000100", 2024-04-12T03:45:57.612 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:2", 2024-04-12T03:45:57.612 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:0", 2024-04-12T03:45:57.612 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000093", 2024-04-12T03:45:57.612 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.shard.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.2", 2024-04-12T03:45:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000004", 2024-04-12T03:45:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000063", 2024-04-12T03:45:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000122", 2024-04-12T03:45:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000057", 2024-04-12T03:45:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:1", 2024-04-12T03:45:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000054", 2024-04-12T03:45:57.613 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000114", 2024-04-12T03:45:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.2", 2024-04-12T03:45:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "data_log.0", 2024-04-12T03:45:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:8", 2024-04-12T03:45:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:1", 2024-04-12T03:45:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "datalog.sync-status.shard.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.0", 2024-04-12T03:45:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000012", 2024-04-12T03:45:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:6", 2024-04-12T03:45:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:9", 2024-04-12T03:45:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000084", 2024-04-12T03:45:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000043", 2024-04-12T03:45:57.614 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:6", 2024-04-12T03:45:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:0", 2024-04-12T03:45:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.11", 2024-04-12T03:45:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:9", 2024-04-12T03:45:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:7", 2024-04-12T03:45:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:0", 2024-04-12T03:45:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:7", 2024-04-12T03:45:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:9", 2024-04-12T03:45:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000111", 2024-04-12T03:45:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000048", 2024-04-12T03:45:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:10", 2024-04-12T03:45:57.615 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000020", 2024-04-12T03:45:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:4", 2024-04-12T03:45:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10", 2024-04-12T03:45:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:5", 2024-04-12T03:45:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000099", 2024-04-12T03:45:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000056", 2024-04-12T03:45:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:0", 2024-04-12T03:45:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.19", 2024-04-12T03:45:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.28", 2024-04-12T03:45:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:7", 2024-04-12T03:45:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:2", 2024-04-12T03:45:57.616 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:2", 2024-04-12T03:45:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000072", 2024-04-12T03:45:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000062", 2024-04-12T03:45:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000115", 2024-04-12T03:45:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:6", 2024-04-12T03:45:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.1", 2024-04-12T03:45:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000033", 2024-04-12T03:45:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:10", 2024-04-12T03:45:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:7", 2024-04-12T03:45:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:2", 2024-04-12T03:45:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:9", 2024-04-12T03:45:57.617 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:0", 2024-04-12T03:45:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000025", 2024-04-12T03:45:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:6", 2024-04-12T03:45:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:5", 2024-04-12T03:45:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000123", 2024-04-12T03:45:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.31", 2024-04-12T03:45:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:4", 2024-04-12T03:45:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:4", 2024-04-12T03:45:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.18", 2024-04-12T03:45:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:10", 2024-04-12T03:45:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19", 2024-04-12T03:45:57.618 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000108", 2024-04-12T03:45:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:8", 2024-04-12T03:45:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000094", 2024-04-12T03:45:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:1", 2024-04-12T03:45:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:8", 2024-04-12T03:45:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000060", 2024-04-12T03:45:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000109", 2024-04-12T03:45:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.15", 2024-04-12T03:45:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:3", 2024-04-12T03:45:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:5", 2024-04-12T03:45:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:6", 2024-04-12T03:45:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:9", 2024-04-12T03:45:57.619 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000013", 2024-04-12T03:45:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:5", 2024-04-12T03:45:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:8", 2024-04-12T03:45:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:8", 2024-04-12T03:45:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:0", 2024-04-12T03:45:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:1", 2024-04-12T03:45:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:1", 2024-04-12T03:45:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:6", 2024-04-12T03:45:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000051", 2024-04-12T03:45:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1", 2024-04-12T03:45:57.620 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000106", 2024-04-12T03:45:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:2", 2024-04-12T03:45:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:7", 2024-04-12T03:45:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:8", 2024-04-12T03:45:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000017", 2024-04-12T03:45:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:3", 2024-04-12T03:45:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "meta.log.b69a664f-970b-4d72-be0d-0071b991ee5c.2", 2024-04-12T03:45:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:1", 2024-04-12T03:45:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000023", 2024-04-12T03:45:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:5", 2024-04-12T03:45:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:4", 2024-04-12T03:45:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-16:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.16:2", 2024-04-12T03:45:57.621 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:7", 2024-04-12T03:45:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:7", 2024-04-12T03:45:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000107", 2024-04-12T03:45:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:3", 2024-04-12T03:45:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000046", 2024-04-12T03:45:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:1", 2024-04-12T03:45:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.26", 2024-04-12T03:45:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:8", 2024-04-12T03:45:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:2", 2024-04-12T03:45:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.22", 2024-04-12T03:45:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-33:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.33:9", 2024-04-12T03:45:57.622 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:6", 2024-04-12T03:45:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "data_log.2", 2024-04-12T03:45:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:8", 2024-04-12T03:45:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000067", 2024-04-12T03:45:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:1", 2024-04-12T03:45:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:3", 2024-04-12T03:45:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:1", 2024-04-12T03:45:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:0", 2024-04-12T03:45:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.9", 2024-04-12T03:45:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000086", 2024-04-12T03:45:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.5", 2024-04-12T03:45:57.623 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:7", 2024-04-12T03:45:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:7", 2024-04-12T03:45:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:9", 2024-04-12T03:45:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-7:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.7:9", 2024-04-12T03:45:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-14:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.14:10", 2024-04-12T03:45:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:2", 2024-04-12T03:45:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:9", 2024-04-12T03:45:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "sync.error-log.20", 2024-04-12T03:45:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-36:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.36:1", 2024-04-12T03:45:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:3", 2024-04-12T03:45:57.624 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-11:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.11:9", 2024-04-12T03:45:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9", 2024-04-12T03:45:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:2", 2024-04-12T03:45:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17", 2024-04-12T03:45:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-20:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.20:5", 2024-04-12T03:45:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:9", 2024-04-12T03:45:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000050", 2024-04-12T03:45:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000102", 2024-04-12T03:45:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-19:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.19:3", 2024-04-12T03:45:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-17:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.17:3", 2024-04-12T03:45:57.625 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-18:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.18:8", 2024-04-12T03:45:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12", 2024-04-12T03:45:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000124", 2024-04-12T03:45:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:3", 2024-04-12T03:45:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:8", 2024-04-12T03:45:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000001", 2024-04-12T03:45:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-15:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.15:5", 2024-04-12T03:45:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000126", 2024-04-12T03:45:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000055", 2024-04-12T03:45:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000061", 2024-04-12T03:45:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-35:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.35:10", 2024-04-12T03:45:57.626 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000097", 2024-04-12T03:45:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:6", 2024-04-12T03:45:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-12:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.12:8", 2024-04-12T03:45:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-8:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.8:6", 2024-04-12T03:45:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-13:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.13:5", 2024-04-12T03:45:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000042", 2024-04-12T03:45:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-37:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5578.1:10", 2024-04-12T03:45:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "data_log.0.4C+qgaj8nTT.0", 2024-04-12T03:45:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-34:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.34:10", 2024-04-12T03:45:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-9:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.9:4", 2024-04-12T03:45:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000075", 2024-04-12T03:45:57.627 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000120", 2024-04-12T03:45:57.628 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-10:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.10:0", 2024-04-12T03:45:57.628 INFO:teuthology.orchestra.run.smithi043.stdout: "obj_delete_at_hint.0000000030" 2024-04-12T03:45:57.628 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:45:57.628 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-12T03:45:57.629 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-12T03:45:57.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:57.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:57.629 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:45:57.685 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:57.685 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:57.740 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:45:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000003751", 2024-04-12T03:45:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:45:57.314445Z" 2024-04-12T03:45:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002061", 2024-04-12T03:45:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:45:57.311159Z" 2024-04-12T03:45:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:57.741 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006987", 2024-04-12T03:45:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:45:57.312248Z" 2024-04-12T03:45:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000005860", 2024-04-12T03:45:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:45:57.313349Z" 2024-04-12T03:45:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:57.742 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:45:57.747 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000003751', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006987', 3: '00000000000000000000:00000000000000005860'} 2024-04-12T03:45:57.747 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:45:57.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:57.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:45:57.747 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:45:57.806 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:45:57.806 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:45:57.860 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:45:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:45:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:45:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:45:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:45:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 16340188497730315678 2024-04-12T03:45:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:45:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:57.861 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:45:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:45:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002906", 2024-04-12T03:45:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:36:03.087572Z" 2024-04-12T03:45:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:57.862 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:45:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:45:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:45:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.705876Z" 2024-04-12T03:45:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:45:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:57.863 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:45:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006144", 2024-04-12T03:45:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:36:06.623407Z" 2024-04-12T03:45:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:45:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:45:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:45:57.864 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:45:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000005018", 2024-04-12T03:45:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:45:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:45:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:45:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:39:01.370893Z" 2024-04-12T03:45:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:45:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:45:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:45:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:45:57.865 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:45:57.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:45:57.866 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:45:57Z" 2024-04-12T03:45:57.866 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:45:57.867 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16340188497730315678 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002906", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:36:03.087572Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.705876Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006144", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:36:06.623407Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:39:01.370893Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:45:57Z" } 2024-04-12T03:45:57.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002906', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:36:03.087572Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.705876Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006144', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:36:06.623407Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:39:01.370893Z'}}] 2024-04-12T03:45:57.867 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003751', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006987', 3: '00000000000000000000:00000000000000005860'} 2024-04-12T03:45:57.867 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002906', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006144', 3: '00000000000000000000:00000000000000005018'} 2024-04-12T03:45:57.867 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003751 target=00000000000000000000:00000000000000002906, shard=1 master=00000000000000000000:00000000000000002061 target=00000000000000000000:00000000000000001499, shard=2 master=00000000000000000000:00000000000000006987 target=00000000000000000000:00000000000000006144, shard=3 master=00000000000000000000:00000000000000005860 target=00000000000000000000:00000000000000005018 2024-04-12T03:46:02.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:46:02.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:46:02.869 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:46:02.932 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:02.932 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:02.985 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:46:02.986 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:46:02.986 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:46:02.986 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:46:02.986 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:46:02.986 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 16340188497730315678 2024-04-12T03:46:02.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:02.986 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:46:02.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:02.987 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:46:02.987 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:02.987 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:46:02.987 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002906", 2024-04-12T03:46:02.987 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:02.987 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:02.987 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:02.987 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:36:03.087572Z" 2024-04-12T03:46:02.987 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:02.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:02.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:02.988 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:46:02.988 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:02.988 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:46:02.989 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:46:02.989 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:02.989 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:02.989 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:02.989 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.705876Z" 2024-04-12T03:46:02.990 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:02.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:02.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:02.990 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:46:02.990 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:02.990 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:46:02.990 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006144", 2024-04-12T03:46:02.990 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:02.991 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:02.991 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:02.991 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:36:06.623407Z" 2024-04-12T03:46:02.991 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:02.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:02.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:02.991 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:46:02.991 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:02.991 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:46:02.991 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000005018", 2024-04-12T03:46:02.992 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:02.992 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:02.992 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:02.992 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:39:01.370893Z" 2024-04-12T03:46:02.992 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:02.992 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:02.992 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:46:02.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:02.992 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:46:02.993 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:46:02.993 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:46:02.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:02.993 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:46:02Z" 2024-04-12T03:46:02.993 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:46:02.994 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16340188497730315678 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002906", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:36:03.087572Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.705876Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006144", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:36:06.623407Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:39:01.370893Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:46:02Z" } 2024-04-12T03:46:02.994 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002906', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:36:03.087572Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.705876Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006144', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:36:06.623407Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:39:01.370893Z'}}] 2024-04-12T03:46:02.994 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003751', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006987', 3: '00000000000000000000:00000000000000005860'} 2024-04-12T03:46:02.994 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002906', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006144', 3: '00000000000000000000:00000000000000005018'} 2024-04-12T03:46:02.994 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003751 target=00000000000000000000:00000000000000002906, shard=1 master=00000000000000000000:00000000000000002061 target=00000000000000000000:00000000000000001499, shard=2 master=00000000000000000000:00000000000000006987 target=00000000000000000000:00000000000000006144, shard=3 master=00000000000000000000:00000000000000005860 target=00000000000000000000:00000000000000005018 2024-04-12T03:46:07.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:46:07.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:46:07.996 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:46:08.058 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:08.058 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:08.111 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:46:08.112 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:46:08.112 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:46:08.112 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:46:08.112 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:46:08.112 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 16340188497730315678 2024-04-12T03:46:08.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:08.112 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:46:08.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:08.112 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:46:08.112 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:08.113 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:46:08.113 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002906", 2024-04-12T03:46:08.113 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:08.113 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:08.113 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:08.113 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:36:03.087572Z" 2024-04-12T03:46:08.113 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:08.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:08.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:08.113 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:46:08.113 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:08.114 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:46:08.114 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:46:08.114 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:08.114 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:08.114 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:08.114 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.705876Z" 2024-04-12T03:46:08.114 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:08.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:08.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:08.114 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:46:08.114 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:08.115 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:46:08.115 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006144", 2024-04-12T03:46:08.115 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:08.115 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:08.115 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:08.115 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:36:06.623407Z" 2024-04-12T03:46:08.115 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:08.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:08.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:08.115 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:46:08.116 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:08.116 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:46:08.116 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000005018", 2024-04-12T03:46:08.116 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:08.116 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:08.116 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:08.116 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:39:01.370893Z" 2024-04-12T03:46:08.116 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:08.116 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:46:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:46:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:46:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:46:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:46:08Z" 2024-04-12T03:46:08.117 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:46:08.118 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16340188497730315678 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002906", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:36:03.087572Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.705876Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006144", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:36:06.623407Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005018", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:39:01.370893Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:46:08Z" } 2024-04-12T03:46:08.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002906', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:36:03.087572Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.705876Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006144', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:36:06.623407Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005018', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:39:01.370893Z'}}] 2024-04-12T03:46:08.118 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003751', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006987', 3: '00000000000000000000:00000000000000005860'} 2024-04-12T03:46:08.118 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002906', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006144', 3: '00000000000000000000:00000000000000005018'} 2024-04-12T03:46:08.119 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003751 target=00000000000000000000:00000000000000002906, shard=1 master=00000000000000000000:00000000000000002061 target=00000000000000000000:00000000000000001499, shard=2 master=00000000000000000000:00000000000000006987 target=00000000000000000000:00000000000000006144, shard=3 master=00000000000000000000:00000000000000005860 target=00000000000000000000:00000000000000005018 2024-04-12T03:46:12.750 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T03:46:12.750+0000 7fd55285f700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T03:46:12.750 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T03:46:12.750+0000 7fd54b050700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T03:46:12.750 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T03:46:12.750+0000 7fd54b851700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T03:46:12.750 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T03:46:12.750+0000 7fd54c052700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T03:46:13.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:46:13.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:46:13.120 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:46:13.185 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:13.185 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:13.241 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:46:13.241 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:46:13.241 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:46:13.242 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:46:13.242 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:46:13.242 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 16340188497730315678 2024-04-12T03:46:13.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.242 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:46:13.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.242 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:46:13.242 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:13.242 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:46:13.242 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000003751", 2024-04-12T03:46:13.242 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:13.243 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:13.243 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:13.243 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:45:57.314445Z" 2024-04-12T03:46:13.243 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.243 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:46:13.243 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:13.243 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:46:13.243 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002061", 2024-04-12T03:46:13.243 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:13.243 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:13.244 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:13.244 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:45:57.311159Z" 2024-04-12T03:46:13.244 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.244 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:46:13.244 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:13.244 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:46:13.244 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006987", 2024-04-12T03:46:13.244 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:13.244 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:13.245 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:13.245 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:45:57.312248Z" 2024-04-12T03:46:13.245 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.245 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:46:13.245 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:13.245 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:46:13.245 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000005860", 2024-04-12T03:46:13.245 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:13.245 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:13.245 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:13.246 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:45:57.313349Z" 2024-04-12T03:46:13.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.246 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:46:13.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.246 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:46:13.246 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:46:13.246 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:46:13.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.246 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:46:13Z" 2024-04-12T03:46:13.246 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:46:13.247 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16340188497730315678 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003751", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:45:57.314445Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002061", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:45:57.311159Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006987", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:45:57.312248Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005860", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:45:57.313349Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:46:13Z" } 2024-04-12T03:46:13.247 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003751', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:45:57.314445Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002061', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:45:57.311159Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006987', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:45:57.312248Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005860', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:45:57.313349Z'}}] 2024-04-12T03:46:13.248 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003751', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006987', 3: '00000000000000000000:00000000000000005860'} 2024-04-12T03:46:13.248 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003751', 1: '00000000000000000000:00000000000000002061', 2: '00000000000000000000:00000000000000006987', 3: '00000000000000000000:00000000000000005860'} 2024-04-12T03:46:13.248 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:46:13.248 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-12T03:46:13.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:46:13.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:46:13.248 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:46:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:13.359 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:46:13.360 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.360 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002906", 2024-04-12T03:46:13.360 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:36:04.449581Z" 2024-04-12T03:46:13.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.360 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.360 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:46:13.360 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:20:22.851157Z" 2024-04-12T03:46:13.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.360 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.360 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006708", 2024-04-12T03:46:13.361 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:36:06.614078Z" 2024-04-12T03:46:13.361 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.361 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.361 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004315", 2024-04-12T03:46:13.361 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:39:01.834142Z" 2024-04-12T03:46:13.361 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.361 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:46:13.362 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000002906', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006708', 3: '00000000000000000000:00000000000000004315'} 2024-04-12T03:46:13.362 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:46:13.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:46:13.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:46:13.363 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:46:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:13.420 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:13.469 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:46:13.469 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:46:13.469 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:46:13.469 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:46:13.469 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:46:13.469 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5944041172307441597 2024-04-12T03:46:13.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.469 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:46:13.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.470 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:46:13.470 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:13.470 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:46:13.470 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002906", 2024-04-12T03:46:13.470 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:13.470 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:13.470 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:13.470 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:36:04.449581Z" 2024-04-12T03:46:13.470 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.471 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:46:13.471 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:13.471 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:46:13.471 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000001499", 2024-04-12T03:46:13.471 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:13.471 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:13.471 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:13.471 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:20:22.851157Z" 2024-04-12T03:46:13.471 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.472 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:46:13.472 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:13.472 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:46:13.472 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006708", 2024-04-12T03:46:13.472 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:13.472 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:13.472 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:13.472 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:36:06.614078Z" 2024-04-12T03:46:13.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.473 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:46:13.473 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:13.473 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:46:13.473 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004315", 2024-04-12T03:46:13.473 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:13.473 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:13.473 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:13.473 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:39:01.834142Z" 2024-04-12T03:46:13.473 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.473 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.473 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:46:13.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.474 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:46:13.474 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:46:13.474 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:46:13.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.474 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:46:13Z" 2024-04-12T03:46:13.474 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:46:13.475 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5944041172307441597 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002906", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:36:04.449581Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001499", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:20:22.851157Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006708", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:36:06.614078Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004315", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:39:01.834142Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:46:13Z" } 2024-04-12T03:46:13.475 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002906', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:36:04.449581Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001499', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:20:22.851157Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006708', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:36:06.614078Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004315', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:39:01.834142Z'}}] 2024-04-12T03:46:13.475 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002906', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006708', 3: '00000000000000000000:00000000000000004315'} 2024-04-12T03:46:13.475 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002906', 1: '00000000000000000000:00000000000000001499', 2: '00000000000000000000:00000000000000006708', 3: '00000000000000000000:00000000000000004315'} 2024-04-12T03:46:13.475 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:46:13.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2024-04-12T03:46:13.478 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-48 2024-04-12T03:46:13.478 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:13.478 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:13.478 DEBUG:boto:Method: PUT 2024-04-12T03:46:13.478 DEBUG:boto:Path: /dkyges-48/ 2024-04-12T03:46:13.478 DEBUG:boto:Data: 2024-04-12T03:46:13.478 DEBUG:boto:Headers: {} 2024-04-12T03:46:13.479 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:13.479 DEBUG:boto:Port: 8000 2024-04-12T03:46:13.479 DEBUG:boto:Params: {} 2024-04-12T03:46:13.479 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:46:13.479 DEBUG:boto:Token: None 2024-04-12T03:46:13.480 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:46:13 GMT /dkyges-48/ 2024-04-12T03:46:13.480 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zsmJtPBA0jPiPMLLnIuVYSM61I4= 2024-04-12T03:46:13.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zsmJtPBA0jPiPMLLnIuVYSM61I4=', 'Content-Length': '0'} 2024-04-12T03:46:13.508 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000035a779c6b5bac2db-006618ae85-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.508 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-49 2024-04-12T03:46:13.508 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:13.508 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:13.508 DEBUG:boto:Method: PUT 2024-04-12T03:46:13.508 DEBUG:boto:Path: /dkyges-49/ 2024-04-12T03:46:13.508 DEBUG:boto:Data: 2024-04-12T03:46:13.509 DEBUG:boto:Headers: {} 2024-04-12T03:46:13.509 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:13.509 DEBUG:boto:Port: 8001 2024-04-12T03:46:13.509 DEBUG:boto:Params: {} 2024-04-12T03:46:13.509 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:46:13.509 DEBUG:boto:Token: None 2024-04-12T03:46:13.509 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:46:13 GMT /dkyges-49/ 2024-04-12T03:46:13.509 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2P5N37Af+WUXQr0PFz5eGiLslW0= 2024-04-12T03:46:13.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2P5N37Af+WUXQr0PFz5eGiLslW0=', 'Content-Length': '0'} 2024-04-12T03:46:13.550 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005f369a6d76d01469-006618ae85-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.550 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:13.550 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:13.550 DEBUG:boto:Method: HEAD 2024-04-12T03:46:13.550 DEBUG:boto:Path: /dkyges-48/ 2024-04-12T03:46:13.550 DEBUG:boto:Data: 2024-04-12T03:46:13.550 DEBUG:boto:Headers: {} 2024-04-12T03:46:13.550 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:13.551 DEBUG:boto:Port: 8000 2024-04-12T03:46:13.551 DEBUG:boto:Params: {} 2024-04-12T03:46:13.551 DEBUG:boto:Token: None 2024-04-12T03:46:13.551 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:13 GMT /dkyges-48/ 2024-04-12T03:46:13.551 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kTKHv0tzu48Lr32T46s9HzWv/VQ= 2024-04-12T03:46:13.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kTKHv0tzu48Lr32T46s9HzWv/VQ=', 'Content-Length': '0'} 2024-04-12T03:46:13.554 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000487af7d9e06183f5-006618ae85-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.554 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:46:13.554 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:46:13.554 DEBUG:boto:Method: PUT 2024-04-12T03:46:13.554 DEBUG:boto:Path: /dkyges-48/obj1 2024-04-12T03:46:13.554 DEBUG:boto:Data: 2024-04-12T03:46:13.555 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:46:13.555 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:13.555 DEBUG:boto:Port: 8000 2024-04-12T03:46:13.555 DEBUG:boto:Params: {} 2024-04-12T03:46:13.555 DEBUG:boto:Token: None 2024-04-12T03:46:13.555 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:46:13 GMT /dkyges-48/obj1 2024-04-12T03:46:13.556 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rjlLDwk5FP/rKN6BYHNOVISlcyY= 2024-04-12T03:46:13.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rjlLDwk5FP/rKN6BYHNOVISlcyY='} 2024-04-12T03:46:13.563 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008535f7289f157f63-006618ae85-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.563 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:13.563 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:13.563 DEBUG:boto:Method: HEAD 2024-04-12T03:46:13.563 DEBUG:boto:Path: /dkyges-48/ 2024-04-12T03:46:13.563 DEBUG:boto:Data: 2024-04-12T03:46:13.564 DEBUG:boto:Headers: {} 2024-04-12T03:46:13.564 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:13.564 DEBUG:boto:Port: 8000 2024-04-12T03:46:13.564 DEBUG:boto:Params: {} 2024-04-12T03:46:13.564 DEBUG:boto:Token: None 2024-04-12T03:46:13.564 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:13 GMT /dkyges-48/ 2024-04-12T03:46:13.564 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kTKHv0tzu48Lr32T46s9HzWv/VQ= 2024-04-12T03:46:13.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kTKHv0tzu48Lr32T46s9HzWv/VQ=', 'Content-Length': '0'} 2024-04-12T03:46:13.566 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd0f48f938c637dc-006618ae85-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.567 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:46:13.567 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:46:13.567 DEBUG:boto:Method: PUT 2024-04-12T03:46:13.567 DEBUG:boto:Path: /dkyges-48/obj2 2024-04-12T03:46:13.567 DEBUG:boto:Data: 2024-04-12T03:46:13.567 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:46:13.567 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:13.567 DEBUG:boto:Port: 8000 2024-04-12T03:46:13.567 DEBUG:boto:Params: {} 2024-04-12T03:46:13.567 DEBUG:boto:Token: None 2024-04-12T03:46:13.568 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:46:13 GMT /dkyges-48/obj2 2024-04-12T03:46:13.568 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fuIujBg0LtskX+K4cszKeNwkoK8= 2024-04-12T03:46:13.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fuIujBg0LtskX+K4cszKeNwkoK8='} 2024-04-12T03:46:13.573 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a1d75eeffdfbd4c-006618ae85-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.573 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:13.574 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:13.574 DEBUG:boto:Method: HEAD 2024-04-12T03:46:13.574 DEBUG:boto:Path: /dkyges-48/ 2024-04-12T03:46:13.574 DEBUG:boto:Data: 2024-04-12T03:46:13.574 DEBUG:boto:Headers: {} 2024-04-12T03:46:13.574 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:13.574 DEBUG:boto:Port: 8000 2024-04-12T03:46:13.574 DEBUG:boto:Params: {} 2024-04-12T03:46:13.574 DEBUG:boto:Token: None 2024-04-12T03:46:13.574 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:13 GMT /dkyges-48/ 2024-04-12T03:46:13.575 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kTKHv0tzu48Lr32T46s9HzWv/VQ= 2024-04-12T03:46:13.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kTKHv0tzu48Lr32T46s9HzWv/VQ=', 'Content-Length': '0'} 2024-04-12T03:46:13.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e6c7f69509031da-006618ae85-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.577 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:46:13.577 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:46:13.578 DEBUG:boto:Method: PUT 2024-04-12T03:46:13.578 DEBUG:boto:Path: /dkyges-48/obj3 2024-04-12T03:46:13.578 DEBUG:boto:Data: 2024-04-12T03:46:13.578 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:46:13.578 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:13.578 DEBUG:boto:Port: 8000 2024-04-12T03:46:13.578 DEBUG:boto:Params: {} 2024-04-12T03:46:13.578 DEBUG:boto:Token: None 2024-04-12T03:46:13.578 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:46:13 GMT /dkyges-48/obj3 2024-04-12T03:46:13.578 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6VFNIHjTFxSdtd+K/hTEr9yEdNg= 2024-04-12T03:46:13.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6VFNIHjTFxSdtd+K/hTEr9yEdNg='} 2024-04-12T03:46:13.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000278fa9fc7384045d-006618ae85-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.623 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:13.623 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:13.623 DEBUG:boto:Method: HEAD 2024-04-12T03:46:13.623 DEBUG:boto:Path: /dkyges-48/ 2024-04-12T03:46:13.623 DEBUG:boto:Data: 2024-04-12T03:46:13.623 DEBUG:boto:Headers: {} 2024-04-12T03:46:13.623 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:13.623 DEBUG:boto:Port: 8000 2024-04-12T03:46:13.623 DEBUG:boto:Params: {} 2024-04-12T03:46:13.624 DEBUG:boto:Token: None 2024-04-12T03:46:13.624 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:13 GMT /dkyges-48/ 2024-04-12T03:46:13.624 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kTKHv0tzu48Lr32T46s9HzWv/VQ= 2024-04-12T03:46:13.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kTKHv0tzu48Lr32T46s9HzWv/VQ=', 'Content-Length': '0'} 2024-04-12T03:46:13.626 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbe9029acdb383dd-006618ae85-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.627 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:46:13.627 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:46:13.627 DEBUG:boto:Method: PUT 2024-04-12T03:46:13.627 DEBUG:boto:Path: /dkyges-48/obj4 2024-04-12T03:46:13.627 DEBUG:boto:Data: 2024-04-12T03:46:13.627 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:46:13.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:13.627 DEBUG:boto:Port: 8000 2024-04-12T03:46:13.627 DEBUG:boto:Params: {} 2024-04-12T03:46:13.628 DEBUG:boto:Token: None 2024-04-12T03:46:13.628 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:46:13 GMT /dkyges-48/obj4 2024-04-12T03:46:13.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rFHbEOXIXIfaMajB6LBbbEG2MV4= 2024-04-12T03:46:13.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rFHbEOXIXIfaMajB6LBbbEG2MV4='} 2024-04-12T03:46:13.637 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d864cc9221d5c9f-006618ae85-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.637 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:13.637 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:13.637 DEBUG:boto:Method: HEAD 2024-04-12T03:46:13.637 DEBUG:boto:Path: /dkyges-49/ 2024-04-12T03:46:13.638 DEBUG:boto:Data: 2024-04-12T03:46:13.638 DEBUG:boto:Headers: {} 2024-04-12T03:46:13.638 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:13.638 DEBUG:boto:Port: 8001 2024-04-12T03:46:13.638 DEBUG:boto:Params: {} 2024-04-12T03:46:13.638 DEBUG:boto:Token: None 2024-04-12T03:46:13.638 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:13 GMT /dkyges-49/ 2024-04-12T03:46:13.638 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MTmbH8VU2E/Fq77ZYKNxZwMBGMk= 2024-04-12T03:46:13.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MTmbH8VU2E/Fq77ZYKNxZwMBGMk=', 'Content-Length': '0'} 2024-04-12T03:46:13.642 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000230ad1adaa6d0be4-006618ae85-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.642 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:46:13.642 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:46:13.642 DEBUG:boto:Method: PUT 2024-04-12T03:46:13.642 DEBUG:boto:Path: /dkyges-49/obj1 2024-04-12T03:46:13.642 DEBUG:boto:Data: 2024-04-12T03:46:13.642 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:46:13.643 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:13.643 DEBUG:boto:Port: 8001 2024-04-12T03:46:13.643 DEBUG:boto:Params: {} 2024-04-12T03:46:13.643 DEBUG:boto:Token: None 2024-04-12T03:46:13.643 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:46:13 GMT /dkyges-49/obj1 2024-04-12T03:46:13.643 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NTqItqc7hPRkr4HHJyajDs4K7qA= 2024-04-12T03:46:13.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NTqItqc7hPRkr4HHJyajDs4K7qA='} 2024-04-12T03:46:13.656 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000251d3e9bb17e54d0-006618ae85-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.656 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:13.656 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:13.657 DEBUG:boto:Method: HEAD 2024-04-12T03:46:13.657 DEBUG:boto:Path: /dkyges-49/ 2024-04-12T03:46:13.657 DEBUG:boto:Data: 2024-04-12T03:46:13.657 DEBUG:boto:Headers: {} 2024-04-12T03:46:13.657 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:13.657 DEBUG:boto:Port: 8001 2024-04-12T03:46:13.657 DEBUG:boto:Params: {} 2024-04-12T03:46:13.657 DEBUG:boto:Token: None 2024-04-12T03:46:13.657 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:13 GMT /dkyges-49/ 2024-04-12T03:46:13.657 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MTmbH8VU2E/Fq77ZYKNxZwMBGMk= 2024-04-12T03:46:13.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MTmbH8VU2E/Fq77ZYKNxZwMBGMk=', 'Content-Length': '0'} 2024-04-12T03:46:13.660 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b79533b0ea974da9-006618ae85-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.660 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:46:13.660 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:46:13.660 DEBUG:boto:Method: PUT 2024-04-12T03:46:13.660 DEBUG:boto:Path: /dkyges-49/obj2 2024-04-12T03:46:13.660 DEBUG:boto:Data: 2024-04-12T03:46:13.660 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:46:13.660 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:13.661 DEBUG:boto:Port: 8001 2024-04-12T03:46:13.661 DEBUG:boto:Params: {} 2024-04-12T03:46:13.661 DEBUG:boto:Token: None 2024-04-12T03:46:13.661 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:46:13 GMT /dkyges-49/obj2 2024-04-12T03:46:13.661 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/9JV+afL2lODEXNKdCmV9D8cA6g= 2024-04-12T03:46:13.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/9JV+afL2lODEXNKdCmV9D8cA6g='} 2024-04-12T03:46:13.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a04f2210d550fbe4-006618ae85-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.667 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:13.667 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:13.667 DEBUG:boto:Method: HEAD 2024-04-12T03:46:13.668 DEBUG:boto:Path: /dkyges-49/ 2024-04-12T03:46:13.668 DEBUG:boto:Data: 2024-04-12T03:46:13.668 DEBUG:boto:Headers: {} 2024-04-12T03:46:13.668 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:13.668 DEBUG:boto:Port: 8001 2024-04-12T03:46:13.668 DEBUG:boto:Params: {} 2024-04-12T03:46:13.668 DEBUG:boto:Token: None 2024-04-12T03:46:13.668 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:13 GMT /dkyges-49/ 2024-04-12T03:46:13.668 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MTmbH8VU2E/Fq77ZYKNxZwMBGMk= 2024-04-12T03:46:13.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MTmbH8VU2E/Fq77ZYKNxZwMBGMk=', 'Content-Length': '0'} 2024-04-12T03:46:13.672 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f50899343ae5f427-006618ae85-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.672 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:46:13.672 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:46:13.672 DEBUG:boto:Method: PUT 2024-04-12T03:46:13.672 DEBUG:boto:Path: /dkyges-49/obj3 2024-04-12T03:46:13.672 DEBUG:boto:Data: 2024-04-12T03:46:13.672 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:46:13.673 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:13.673 DEBUG:boto:Port: 8001 2024-04-12T03:46:13.673 DEBUG:boto:Params: {} 2024-04-12T03:46:13.673 DEBUG:boto:Token: None 2024-04-12T03:46:13.673 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:46:13 GMT /dkyges-49/obj3 2024-04-12T03:46:13.673 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8pouNeEgm2tBTpQ+hrcKsuUUNbw= 2024-04-12T03:46:13.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8pouNeEgm2tBTpQ+hrcKsuUUNbw='} 2024-04-12T03:46:13.722 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f15b660d89c25a3-006618ae85-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.723 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:13.723 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:13.723 DEBUG:boto:Method: HEAD 2024-04-12T03:46:13.723 DEBUG:boto:Path: /dkyges-49/ 2024-04-12T03:46:13.723 DEBUG:boto:Data: 2024-04-12T03:46:13.723 DEBUG:boto:Headers: {} 2024-04-12T03:46:13.723 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:13.723 DEBUG:boto:Port: 8001 2024-04-12T03:46:13.723 DEBUG:boto:Params: {} 2024-04-12T03:46:13.724 DEBUG:boto:Token: None 2024-04-12T03:46:13.724 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:13 GMT /dkyges-49/ 2024-04-12T03:46:13.724 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MTmbH8VU2E/Fq77ZYKNxZwMBGMk= 2024-04-12T03:46:13.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MTmbH8VU2E/Fq77ZYKNxZwMBGMk=', 'Content-Length': '0'} 2024-04-12T03:46:13.726 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082dd5119147b8e00-006618ae85-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.726 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:46:13.726 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:46:13.726 DEBUG:boto:Method: PUT 2024-04-12T03:46:13.726 DEBUG:boto:Path: /dkyges-49/obj4 2024-04-12T03:46:13.727 DEBUG:boto:Data: 2024-04-12T03:46:13.727 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:46:13.727 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:13.727 DEBUG:boto:Port: 8001 2024-04-12T03:46:13.727 DEBUG:boto:Params: {} 2024-04-12T03:46:13.727 DEBUG:boto:Token: None 2024-04-12T03:46:13.727 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:46:13 GMT /dkyges-49/obj4 2024-04-12T03:46:13.727 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5izE0NnGHQi52OaMsMcYCaglg2U= 2024-04-12T03:46:13.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5izE0NnGHQi52OaMsMcYCaglg2U='} 2024-04-12T03:46:13.733 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cc611911c3e895af-006618ae85-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:13.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:46:13.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:46:13.734 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:46:13.788 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:13.788 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:13.823 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:46:13.838 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:46:13.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.838 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:46:13.838 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:46:13.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.838 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:46:13.839 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:46:13.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.839 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:46:13.839 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:46:13.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.839 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893573.538340_423.1", 2024-04-12T03:46:13.839 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:46:13.538340Z" 2024-04-12T03:46:13.839 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.839 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:46:13.840 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893573.538340_423.1'} 2024-04-12T03:46:13.841 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:46:13.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:46:13.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:46:13.841 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:46:13.895 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:13.895 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:13.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:13.930+0000 7f5310ebf780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:46:13.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:13.931+0000 7f5310ebf780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:46:13.945 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:46:13.946 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:46:13.946 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:46:13.946 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:46:13.946 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:46:13.946 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:46:13.946 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:46:13.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.946 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:46:13.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.946 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:46:13.947 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:13.947 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:46:13.947 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:46:13.947 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:13.947 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:46:13.947 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:13.947 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:46:13.947 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:46:13.947 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.948 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:46:13.948 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:13.948 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:46:13.948 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:46:13.948 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:13.948 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:13.948 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:13.948 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:46:13.948 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:46:13.948 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.949 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:46:13.949 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:13.949 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:46:13.949 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:46:13.949 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:13.949 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:13.949 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:13.949 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:46:13.949 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:46:13.949 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:46:13.950 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:46:13.950 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:46:13.950 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:46:13.950 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893573.538340_423.1", 2024-04-12T03:46:13.950 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:46:13.950 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:46:13.950 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:46:13.950 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:46:13.538340Z", 2024-04-12T03:46:13.950 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:46:13.950 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.951 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:46:13.951 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:46:13.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.951 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:46:13.951 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:46:13.951 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:46:13.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:46:13.951 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:46:13Z" 2024-04-12T03:46:13.951 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:46:13.952 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893573.538340_423.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:46:13.538340Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:46:13Z" } 2024-04-12T03:46:13.952 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893573.538340_423.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:46:13.538340Z', 'realm_epoch': 2}}] 2024-04-12T03:46:13.953 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893573.538340_423.1'} 2024-04-12T03:46:13.953 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893573.538340_423.1'} 2024-04-12T03:46:13.953 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:46:13.953 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-48 2024-04-12T03:46:13.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:46:13.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:46:13.953 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:46:14.016 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:14.016 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:14.057+0000 7f2d506f6780 1 waiting to reach incremental sync.. 2024-04-12T03:46:19.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:19.058+0000 7f2d506f6780 1 waiting to reach incremental sync.. 2024-04-12T03:46:21.184 INFO:tasks.rgw.c1.client.0.smithi043.stdout:2024-04-12T03:46:21.185+0000 7f1e272bd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T03:46:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:24.059+0000 7f2d506f6780 1 waiting to reach incremental sync.. 2024-04-12T03:46:29.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:29.058+0000 7f2d506f6780 1 waiting to reach incremental sync.. 2024-04-12T03:46:34.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:34.060+0000 7f2d506f6780 1 bucket sync caught up with source: 2024-04-12T03:46:34.060 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.22.6, , 00000000001.37.6, , 00000000001.41.6, , , 00000000001.21.6, , , ] 2024-04-12T03:46:34.060 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.22.6, , 00000000001.37.6, , 00000000001.41.6, , , 00000000001.21.6, , , ] 2024-04-12T03:46:34.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:34.060+0000 7f2d506f6780 0 bucket checkpoint complete 2024-04-12T03:46:34.078 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-48 2024-04-12T03:46:34.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:46:34.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:46:34.079 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:46:34.145 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:34.145 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:34.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:34.186+0000 7fdc033b5780 1 waiting for incremental sync to catch up: 2024-04-12T03:46:34.185 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.1861.6, , 00000000001.3585.6, , 00000000001.2429.6, , , , , , ] 2024-04-12T03:46:34.185 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.1861.6, , 00000000001.3585.6, , 00000000001.2429.6, , , 00000000001.2521.6, , , ] 2024-04-12T03:46:39.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:39.187+0000 7fdc033b5780 1 waiting for incremental sync to catch up: 2024-04-12T03:46:39.187 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.1861.6, , 00000000001.3585.6, , 00000000001.2429.6, , , , , , ] 2024-04-12T03:46:39.187 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.1861.6, , 00000000001.3585.6, , 00000000001.2429.6, , , 00000000001.2521.6, , , ] 2024-04-12T03:46:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:44.189+0000 7fdc033b5780 1 waiting for incremental sync to catch up: 2024-04-12T03:46:44.188 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.1861.6, , 00000000001.3585.6, , 00000000001.2429.6, , , , , , ] 2024-04-12T03:46:44.188 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.1861.6, , 00000000001.3585.6, , 00000000001.2429.6, , , 00000000001.2521.6, , , ] 2024-04-12T03:46:49.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:49.191+0000 7fdc033b5780 1 waiting for incremental sync to catch up: 2024-04-12T03:46:49.191 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.1861.6, , 00000000001.3585.6, , 00000000001.2429.6, , , , , , ] 2024-04-12T03:46:49.191 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.1861.6, , 00000000001.3585.6, , 00000000001.2429.6, , , 00000000001.2521.6, , , ] 2024-04-12T03:46:54.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:54.192+0000 7fdc033b5780 1 bucket sync caught up with source: 2024-04-12T03:46:54.191 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.1861.6, , 00000000001.3585.6, , 00000000001.2429.6, , , 00000000001.2521.6, , , ] 2024-04-12T03:46:54.191 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.1861.6, , 00000000001.3585.6, , 00000000001.2429.6, , , 00000000001.2521.6, , , ] 2024-04-12T03:46:54.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:54.192+0000 7fdc033b5780 0 bucket checkpoint complete 2024-04-12T03:46:54.209 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-48 zones={a2, a1} 2024-04-12T03:46:54.209 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.209 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.209 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.209 DEBUG:boto:Path: /dkyges-48/ 2024-04-12T03:46:54.210 DEBUG:boto:Data: 2024-04-12T03:46:54.210 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.210 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.210 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.210 DEBUG:boto:Params: {} 2024-04-12T03:46:54.210 DEBUG:boto:Token: None 2024-04-12T03:46:54.211 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.211 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IQDqA+t6etYFkG6oIqq8smu7bj0= 2024-04-12T03:46:54.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IQDqA+t6etYFkG6oIqq8smu7bj0=', 'Content-Length': '0'} 2024-04-12T03:46:54.214 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb28f57def7a33c5-006618aeae-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.214 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.215 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.215 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.215 DEBUG:boto:Path: /dkyges-48/ 2024-04-12T03:46:54.215 DEBUG:boto:Data: 2024-04-12T03:46:54.215 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.215 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.215 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.215 DEBUG:boto:Params: {} 2024-04-12T03:46:54.215 DEBUG:boto:Token: None 2024-04-12T03:46:54.216 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.216 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IQDqA+t6etYFkG6oIqq8smu7bj0= 2024-04-12T03:46:54.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IQDqA+t6etYFkG6oIqq8smu7bj0=', 'Content-Length': '0'} 2024-04-12T03:46:54.218 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1284f16762274f9-006618aeae-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.218 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:46:54.219 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.219 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.219 DEBUG:boto:path=/dkyges-48/?versions&max-keys=999 2024-04-12T03:46:54.219 DEBUG:boto:auth_path=/dkyges-48/?versions&max-keys=999 2024-04-12T03:46:54.219 DEBUG:boto:Method: GET 2024-04-12T03:46:54.219 DEBUG:boto:Path: /dkyges-48/?versions&max-keys=999 2024-04-12T03:46:54.219 DEBUG:boto:Data: 2024-04-12T03:46:54.219 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.219 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.219 DEBUG:boto:Params: {} 2024-04-12T03:46:54.220 DEBUG:boto:Token: None 2024-04-12T03:46:54.220 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/?versions 2024-04-12T03:46:54.220 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0JwW5IzmggN0kYh9pQ8gl+Yust4= 2024-04-12T03:46:54.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0JwW5IzmggN0kYh9pQ8gl+Yust4=', 'Content-Length': '0'} 2024-04-12T03:46:54.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d44e765b8547528-006618aeae-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.223 DEBUG:boto:b'dkyges-48999falseobj1nulltrue2024-04-12T03:46:13.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:46:13.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:46:13.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:46:13.631Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:46:54.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:46:54.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:46:54.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:46:54.224 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:46:54.224 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.224 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.224 DEBUG:boto:path=/dkyges-48/?versions&max-keys=999 2024-04-12T03:46:54.224 DEBUG:boto:auth_path=/dkyges-48/?versions&max-keys=999 2024-04-12T03:46:54.225 DEBUG:boto:Method: GET 2024-04-12T03:46:54.225 DEBUG:boto:Path: /dkyges-48/?versions&max-keys=999 2024-04-12T03:46:54.225 DEBUG:boto:Data: 2024-04-12T03:46:54.225 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.225 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.225 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.225 DEBUG:boto:Params: {} 2024-04-12T03:46:54.225 DEBUG:boto:Token: None 2024-04-12T03:46:54.225 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/?versions 2024-04-12T03:46:54.225 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0JwW5IzmggN0kYh9pQ8gl+Yust4= 2024-04-12T03:46:54.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0JwW5IzmggN0kYh9pQ8gl+Yust4=', 'Content-Length': '0'} 2024-04-12T03:46:54.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a06ce36617dcb30-006618aeae-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.228 DEBUG:boto:b'dkyges-48999falseobj1nulltrue2024-04-12T03:46:13.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:46:13.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:46:13.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:46:13.631Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:46:54.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:46:54.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:46:54.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:46:54.229 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.229 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.229 DEBUG:boto:path=/dkyges-48/?versions&max-keys=999 2024-04-12T03:46:54.229 DEBUG:boto:auth_path=/dkyges-48/?versions&max-keys=999 2024-04-12T03:46:54.229 DEBUG:boto:Method: GET 2024-04-12T03:46:54.230 DEBUG:boto:Path: /dkyges-48/?versions&max-keys=999 2024-04-12T03:46:54.230 DEBUG:boto:Data: 2024-04-12T03:46:54.230 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.230 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.230 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.230 DEBUG:boto:Params: {} 2024-04-12T03:46:54.230 DEBUG:boto:Token: None 2024-04-12T03:46:54.230 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/?versions 2024-04-12T03:46:54.230 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0JwW5IzmggN0kYh9pQ8gl+Yust4= 2024-04-12T03:46:54.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0JwW5IzmggN0kYh9pQ8gl+Yust4=', 'Content-Length': '0'} 2024-04-12T03:46:54.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004721cbffdb2a05e9-006618aeae-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.233 DEBUG:boto:b'dkyges-48999falseobj1nulltrue2024-04-12T03:46:13.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:46:13.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:46:13.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:46:13.631Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.233 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.234 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.234 DEBUG:boto:path=/dkyges-48/?versions&max-keys=999 2024-04-12T03:46:54.234 DEBUG:boto:auth_path=/dkyges-48/?versions&max-keys=999 2024-04-12T03:46:54.234 DEBUG:boto:Method: GET 2024-04-12T03:46:54.234 DEBUG:boto:Path: /dkyges-48/?versions&max-keys=999 2024-04-12T03:46:54.234 DEBUG:boto:Data: 2024-04-12T03:46:54.234 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.234 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.234 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.234 DEBUG:boto:Params: {} 2024-04-12T03:46:54.234 DEBUG:boto:Token: None 2024-04-12T03:46:54.235 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/?versions 2024-04-12T03:46:54.235 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0JwW5IzmggN0kYh9pQ8gl+Yust4= 2024-04-12T03:46:54.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0JwW5IzmggN0kYh9pQ8gl+Yust4=', 'Content-Length': '0'} 2024-04-12T03:46:54.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aad8e4f7dfdfd41f-006618aeae-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.237 DEBUG:boto:b'dkyges-48999falseobj1nulltrue2024-04-12T03:46:13.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:46:13.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:46:13.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:46:13.631Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:46:54.238 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:46:54.238 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:46:54.238 DEBUG:boto:path=/dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.238 DEBUG:boto:auth_path=/dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.238 DEBUG:boto:Method: GET 2024-04-12T03:46:54.238 DEBUG:boto:Path: /dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.238 DEBUG:boto:Data: 2024-04-12T03:46:54.238 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.239 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.239 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.239 DEBUG:boto:Params: {} 2024-04-12T03:46:54.239 DEBUG:boto:Token: None 2024-04-12T03:46:54.239 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.239 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mJbwH0t5NG5+xUBJg11JtO6HU1M= 2024-04-12T03:46:54.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mJbwH0t5NG5+xUBJg11JtO6HU1M=', 'Content-Length': '0'} 2024-04-12T03:46:54.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab0a43c431a79f49-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.241 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:46:54.241 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:46:54.241 DEBUG:boto:path=/dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.241 DEBUG:boto:auth_path=/dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.241 DEBUG:boto:Method: GET 2024-04-12T03:46:54.241 DEBUG:boto:Path: /dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.242 DEBUG:boto:Data: 2024-04-12T03:46:54.242 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.242 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.242 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.242 DEBUG:boto:Params: {} 2024-04-12T03:46:54.242 DEBUG:boto:Token: None 2024-04-12T03:46:54.242 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.242 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mJbwH0t5NG5+xUBJg11JtO6HU1M= 2024-04-12T03:46:54.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mJbwH0t5NG5+xUBJg11JtO6HU1M=', 'Content-Length': '0'} 2024-04-12T03:46:54.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f28e84290ad6f9c9-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.290 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:46:54.291 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:46:54.291 DEBUG:boto:path=/dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.291 DEBUG:boto:auth_path=/dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.291 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.291 DEBUG:boto:Path: /dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.291 DEBUG:boto:Data: 2024-04-12T03:46:54.291 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.291 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.291 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.291 DEBUG:boto:Params: {} 2024-04-12T03:46:54.292 DEBUG:boto:Token: None 2024-04-12T03:46:54.292 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.292 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8IZ4i57ECePZtHazmnv9S1F+0sc= 2024-04-12T03:46:54.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8IZ4i57ECePZtHazmnv9S1F+0sc=', 'Content-Length': '0'} 2024-04-12T03:46:54.294 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000395dfccb2d87f377-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.294 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:46:54.294 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:46:54.294 DEBUG:boto:path=/dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.294 DEBUG:boto:auth_path=/dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.294 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.294 DEBUG:boto:Path: /dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.295 DEBUG:boto:Data: 2024-04-12T03:46:54.295 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.295 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.295 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.295 DEBUG:boto:Params: {} 2024-04-12T03:46:54.295 DEBUG:boto:Token: None 2024-04-12T03:46:54.295 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.295 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8IZ4i57ECePZtHazmnv9S1F+0sc= 2024-04-12T03:46:54.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8IZ4i57ECePZtHazmnv9S1F+0sc=', 'Content-Length': '0'} 2024-04-12T03:46:54.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0d0e65d6247e69e-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:46:54.298 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:46:54.298 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:46:54.298 DEBUG:boto:path=/dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.298 DEBUG:boto:auth_path=/dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.298 DEBUG:boto:Method: GET 2024-04-12T03:46:54.298 DEBUG:boto:Path: /dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.298 DEBUG:boto:Data: 2024-04-12T03:46:54.298 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.298 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.299 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.299 DEBUG:boto:Params: {} 2024-04-12T03:46:54.299 DEBUG:boto:Token: None 2024-04-12T03:46:54.299 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.299 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mJbwH0t5NG5+xUBJg11JtO6HU1M= 2024-04-12T03:46:54.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mJbwH0t5NG5+xUBJg11JtO6HU1M=', 'Content-Length': '0'} 2024-04-12T03:46:54.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cf6422003062eaa-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.301 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:46:54.301 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:46:54.301 DEBUG:boto:path=/dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.301 DEBUG:boto:auth_path=/dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.301 DEBUG:boto:Method: GET 2024-04-12T03:46:54.301 DEBUG:boto:Path: /dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.302 DEBUG:boto:Data: 2024-04-12T03:46:54.302 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.302 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.302 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.302 DEBUG:boto:Params: {} 2024-04-12T03:46:54.302 DEBUG:boto:Token: None 2024-04-12T03:46:54.302 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj1?versionId=null 2024-04-12T03:46:54.302 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mJbwH0t5NG5+xUBJg11JtO6HU1M= 2024-04-12T03:46:54.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mJbwH0t5NG5+xUBJg11JtO6HU1M=', 'Content-Length': '0'} 2024-04-12T03:46:54.304 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038a6e4a7c740cdf8-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.304 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.304 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.305 DEBUG:boto:path=/dkyges-48/?prefix=obj1 2024-04-12T03:46:54.305 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj1 2024-04-12T03:46:54.305 DEBUG:boto:Method: GET 2024-04-12T03:46:54.305 DEBUG:boto:Path: /dkyges-48/?prefix=obj1 2024-04-12T03:46:54.305 DEBUG:boto:Data: 2024-04-12T03:46:54.305 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.305 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.305 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.305 DEBUG:boto:Params: {} 2024-04-12T03:46:54.305 DEBUG:boto:Token: None 2024-04-12T03:46:54.305 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.306 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc= 2024-04-12T03:46:54.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc=', 'Content-Length': '0'} 2024-04-12T03:46:54.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070128ade626dbb04-006618aeae-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.308 DEBUG:boto:b'dkyges-48obj11000falseobj12024-04-12T03:46:13.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.309 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.309 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.309 DEBUG:boto:path=/dkyges-48/?prefix=obj1 2024-04-12T03:46:54.309 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj1 2024-04-12T03:46:54.309 DEBUG:boto:Method: GET 2024-04-12T03:46:54.309 DEBUG:boto:Path: /dkyges-48/?prefix=obj1 2024-04-12T03:46:54.309 DEBUG:boto:Data: 2024-04-12T03:46:54.309 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.310 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.310 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.310 DEBUG:boto:Params: {} 2024-04-12T03:46:54.310 DEBUG:boto:Token: None 2024-04-12T03:46:54.310 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.310 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc= 2024-04-12T03:46:54.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc=', 'Content-Length': '0'} 2024-04-12T03:46:54.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfdbce729031aa4b-006618aeae-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.313 DEBUG:boto:b'dkyges-48obj11000falseobj12024-04-12T03:46:13.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.313 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:46:54.313 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:46:54.313 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.313 DEBUG:boto:Path: /dkyges-48/obj1 2024-04-12T03:46:54.313 DEBUG:boto:Data: 2024-04-12T03:46:54.313 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.313 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.314 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.314 DEBUG:boto:Params: {} 2024-04-12T03:46:54.314 DEBUG:boto:Token: None 2024-04-12T03:46:54.314 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj1 2024-04-12T03:46:54.314 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vg5uO+Zh2xvdl6369scItsbtrxE= 2024-04-12T03:46:54.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vg5uO+Zh2xvdl6369scItsbtrxE=', 'Content-Length': '0'} 2024-04-12T03:46:54.316 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002469e649377494bf-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.316 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:46:54.316 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:46:54.316 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.316 DEBUG:boto:Path: /dkyges-48/obj1 2024-04-12T03:46:54.316 DEBUG:boto:Data: 2024-04-12T03:46:54.317 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.317 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.317 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.317 DEBUG:boto:Params: {} 2024-04-12T03:46:54.317 DEBUG:boto:Token: None 2024-04-12T03:46:54.317 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj1 2024-04-12T03:46:54.317 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vg5uO+Zh2xvdl6369scItsbtrxE= 2024-04-12T03:46:54.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vg5uO+Zh2xvdl6369scItsbtrxE=', 'Content-Length': '0'} 2024-04-12T03:46:54.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf61efed101906b7-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:46:54.319 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:46:54.319 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:46:54.319 DEBUG:boto:Method: GET 2024-04-12T03:46:54.319 DEBUG:boto:Path: /dkyges-48/obj1 2024-04-12T03:46:54.320 DEBUG:boto:Data: 2024-04-12T03:46:54.320 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.320 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.320 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.320 DEBUG:boto:Params: {} 2024-04-12T03:46:54.320 DEBUG:boto:Token: None 2024-04-12T03:46:54.320 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj1 2024-04-12T03:46:54.320 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6mT59xrOM5xl68aCYWCRjxg4fFc= 2024-04-12T03:46:54.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6mT59xrOM5xl68aCYWCRjxg4fFc=', 'Content-Length': '0'} 2024-04-12T03:46:54.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef865296791aaad4-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.322 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:46:54.322 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:46:54.323 DEBUG:boto:Method: GET 2024-04-12T03:46:54.323 DEBUG:boto:Path: /dkyges-48/obj1 2024-04-12T03:46:54.323 DEBUG:boto:Data: 2024-04-12T03:46:54.323 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.323 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.323 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.323 DEBUG:boto:Params: {} 2024-04-12T03:46:54.323 DEBUG:boto:Token: None 2024-04-12T03:46:54.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj1 2024-04-12T03:46:54.323 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6mT59xrOM5xl68aCYWCRjxg4fFc= 2024-04-12T03:46:54.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6mT59xrOM5xl68aCYWCRjxg4fFc=', 'Content-Length': '0'} 2024-04-12T03:46:54.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9ac7c76cf905f93-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.326 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.326 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.326 DEBUG:boto:path=/dkyges-48/?prefix=obj1 2024-04-12T03:46:54.326 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj1 2024-04-12T03:46:54.326 DEBUG:boto:Method: GET 2024-04-12T03:46:54.326 DEBUG:boto:Path: /dkyges-48/?prefix=obj1 2024-04-12T03:46:54.326 DEBUG:boto:Data: 2024-04-12T03:46:54.326 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.326 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.326 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.327 DEBUG:boto:Params: {} 2024-04-12T03:46:54.327 DEBUG:boto:Token: None 2024-04-12T03:46:54.327 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.327 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc= 2024-04-12T03:46:54.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc=', 'Content-Length': '0'} 2024-04-12T03:46:54.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005041d24661a234cc-006618aeae-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.374 DEBUG:boto:b'dkyges-48obj11000falseobj12024-04-12T03:46:13.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.375 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.375 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.375 DEBUG:boto:path=/dkyges-48/?prefix=obj1 2024-04-12T03:46:54.375 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj1 2024-04-12T03:46:54.375 DEBUG:boto:Method: GET 2024-04-12T03:46:54.375 DEBUG:boto:Path: /dkyges-48/?prefix=obj1 2024-04-12T03:46:54.375 DEBUG:boto:Data: 2024-04-12T03:46:54.376 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.376 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.376 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.376 DEBUG:boto:Params: {} 2024-04-12T03:46:54.376 DEBUG:boto:Token: None 2024-04-12T03:46:54.376 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.376 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc= 2024-04-12T03:46:54.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc=', 'Content-Length': '0'} 2024-04-12T03:46:54.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc26356a10641088-006618aeae-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.379 DEBUG:boto:b'dkyges-48obj11000falseobj12024-04-12T03:46:13.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:46:54.380 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:46:54.380 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:46:54.380 DEBUG:boto:path=/dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.380 DEBUG:boto:auth_path=/dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.380 DEBUG:boto:Method: GET 2024-04-12T03:46:54.380 DEBUG:boto:Path: /dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.380 DEBUG:boto:Data: 2024-04-12T03:46:54.381 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.381 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.381 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.381 DEBUG:boto:Params: {} 2024-04-12T03:46:54.381 DEBUG:boto:Token: None 2024-04-12T03:46:54.381 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.381 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PJEtr/lsE8zikCjSS6FWuw1zYHU= 2024-04-12T03:46:54.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PJEtr/lsE8zikCjSS6FWuw1zYHU=', 'Content-Length': '0'} 2024-04-12T03:46:54.383 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000853a1428bb1ce167-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.383 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:46:54.383 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:46:54.384 DEBUG:boto:path=/dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.384 DEBUG:boto:auth_path=/dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.384 DEBUG:boto:Method: GET 2024-04-12T03:46:54.384 DEBUG:boto:Path: /dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.384 DEBUG:boto:Data: 2024-04-12T03:46:54.384 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.384 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.384 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.384 DEBUG:boto:Params: {} 2024-04-12T03:46:54.384 DEBUG:boto:Token: None 2024-04-12T03:46:54.384 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.385 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PJEtr/lsE8zikCjSS6FWuw1zYHU= 2024-04-12T03:46:54.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PJEtr/lsE8zikCjSS6FWuw1zYHU=', 'Content-Length': '0'} 2024-04-12T03:46:54.387 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd926c2cb9b2223e-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.434 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:46:54.435 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:46:54.435 DEBUG:boto:path=/dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.435 DEBUG:boto:auth_path=/dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.435 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.435 DEBUG:boto:Path: /dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.435 DEBUG:boto:Data: 2024-04-12T03:46:54.435 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.435 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.435 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.435 DEBUG:boto:Params: {} 2024-04-12T03:46:54.436 DEBUG:boto:Token: None 2024-04-12T03:46:54.436 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.436 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9E9EKk6APX4AceISlkRtm//FF+g= 2024-04-12T03:46:54.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9E9EKk6APX4AceISlkRtm//FF+g=', 'Content-Length': '0'} 2024-04-12T03:46:54.438 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001045aa77da5f0107-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.438 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:46:54.438 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:46:54.438 DEBUG:boto:path=/dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.438 DEBUG:boto:auth_path=/dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.439 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.439 DEBUG:boto:Path: /dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.439 DEBUG:boto:Data: 2024-04-12T03:46:54.439 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.439 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.439 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.439 DEBUG:boto:Params: {} 2024-04-12T03:46:54.439 DEBUG:boto:Token: None 2024-04-12T03:46:54.439 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.440 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9E9EKk6APX4AceISlkRtm//FF+g= 2024-04-12T03:46:54.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9E9EKk6APX4AceISlkRtm//FF+g=', 'Content-Length': '0'} 2024-04-12T03:46:54.442 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000430908237fc7d01e-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:46:54.442 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:46:54.442 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:46:54.442 DEBUG:boto:path=/dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.443 DEBUG:boto:auth_path=/dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.443 DEBUG:boto:Method: GET 2024-04-12T03:46:54.443 DEBUG:boto:Path: /dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.443 DEBUG:boto:Data: 2024-04-12T03:46:54.443 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.443 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.443 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.443 DEBUG:boto:Params: {} 2024-04-12T03:46:54.444 DEBUG:boto:Token: None 2024-04-12T03:46:54.444 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.444 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PJEtr/lsE8zikCjSS6FWuw1zYHU= 2024-04-12T03:46:54.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PJEtr/lsE8zikCjSS6FWuw1zYHU=', 'Content-Length': '0'} 2024-04-12T03:46:54.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cf41589c594373d-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.447 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:46:54.447 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:46:54.447 DEBUG:boto:path=/dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.447 DEBUG:boto:auth_path=/dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.447 DEBUG:boto:Method: GET 2024-04-12T03:46:54.447 DEBUG:boto:Path: /dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.447 DEBUG:boto:Data: 2024-04-12T03:46:54.447 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.447 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.447 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.448 DEBUG:boto:Params: {} 2024-04-12T03:46:54.448 DEBUG:boto:Token: None 2024-04-12T03:46:54.448 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj2?versionId=null 2024-04-12T03:46:54.448 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PJEtr/lsE8zikCjSS6FWuw1zYHU= 2024-04-12T03:46:54.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PJEtr/lsE8zikCjSS6FWuw1zYHU=', 'Content-Length': '0'} 2024-04-12T03:46:54.450 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf3bf7d7972c4fd9-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.450 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.450 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.450 DEBUG:boto:path=/dkyges-48/?prefix=obj2 2024-04-12T03:46:54.450 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj2 2024-04-12T03:46:54.451 DEBUG:boto:Method: GET 2024-04-12T03:46:54.451 DEBUG:boto:Path: /dkyges-48/?prefix=obj2 2024-04-12T03:46:54.451 DEBUG:boto:Data: 2024-04-12T03:46:54.451 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.451 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.451 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.451 DEBUG:boto:Params: {} 2024-04-12T03:46:54.451 DEBUG:boto:Token: None 2024-04-12T03:46:54.451 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.451 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc= 2024-04-12T03:46:54.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc=', 'Content-Length': '0'} 2024-04-12T03:46:54.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009119d51b2986cb2e-006618aeae-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.454 DEBUG:boto:b'dkyges-48obj21000falseobj22024-04-12T03:46:13.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.454 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.455 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.455 DEBUG:boto:path=/dkyges-48/?prefix=obj2 2024-04-12T03:46:54.455 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj2 2024-04-12T03:46:54.455 DEBUG:boto:Method: GET 2024-04-12T03:46:54.455 DEBUG:boto:Path: /dkyges-48/?prefix=obj2 2024-04-12T03:46:54.455 DEBUG:boto:Data: 2024-04-12T03:46:54.455 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.455 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.455 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.455 DEBUG:boto:Params: {} 2024-04-12T03:46:54.455 DEBUG:boto:Token: None 2024-04-12T03:46:54.456 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.456 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc= 2024-04-12T03:46:54.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc=', 'Content-Length': '0'} 2024-04-12T03:46:54.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004effeedacd3673ef-006618aeae-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.458 DEBUG:boto:b'dkyges-48obj21000falseobj22024-04-12T03:46:13.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.459 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:46:54.459 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:46:54.459 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.459 DEBUG:boto:Path: /dkyges-48/obj2 2024-04-12T03:46:54.459 DEBUG:boto:Data: 2024-04-12T03:46:54.459 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.459 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.459 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.459 DEBUG:boto:Params: {} 2024-04-12T03:46:54.459 DEBUG:boto:Token: None 2024-04-12T03:46:54.460 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj2 2024-04-12T03:46:54.460 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Dcm4GpW/hYkMXc1DB4Kh5MFvRYc= 2024-04-12T03:46:54.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Dcm4GpW/hYkMXc1DB4Kh5MFvRYc=', 'Content-Length': '0'} 2024-04-12T03:46:54.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000005c4dcd6e7e9e111-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.462 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:46:54.462 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:46:54.462 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.462 DEBUG:boto:Path: /dkyges-48/obj2 2024-04-12T03:46:54.462 DEBUG:boto:Data: 2024-04-12T03:46:54.462 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.462 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.462 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.463 DEBUG:boto:Params: {} 2024-04-12T03:46:54.463 DEBUG:boto:Token: None 2024-04-12T03:46:54.463 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj2 2024-04-12T03:46:54.463 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Dcm4GpW/hYkMXc1DB4Kh5MFvRYc= 2024-04-12T03:46:54.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Dcm4GpW/hYkMXc1DB4Kh5MFvRYc=', 'Content-Length': '0'} 2024-04-12T03:46:54.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d98ea7a7e30ade35-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:46:54.465 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:46:54.465 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:46:54.465 DEBUG:boto:Method: GET 2024-04-12T03:46:54.466 DEBUG:boto:Path: /dkyges-48/obj2 2024-04-12T03:46:54.466 DEBUG:boto:Data: 2024-04-12T03:46:54.466 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.466 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.466 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.466 DEBUG:boto:Params: {} 2024-04-12T03:46:54.466 DEBUG:boto:Token: None 2024-04-12T03:46:54.466 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj2 2024-04-12T03:46:54.466 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OfPuAw1WK+VVY0p8H+PRH2cVImY= 2024-04-12T03:46:54.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OfPuAw1WK+VVY0p8H+PRH2cVImY=', 'Content-Length': '0'} 2024-04-12T03:46:54.468 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0cab9d937b77c70-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.468 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:46:54.468 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:46:54.469 DEBUG:boto:Method: GET 2024-04-12T03:46:54.469 DEBUG:boto:Path: /dkyges-48/obj2 2024-04-12T03:46:54.469 DEBUG:boto:Data: 2024-04-12T03:46:54.469 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.469 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.469 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.469 DEBUG:boto:Params: {} 2024-04-12T03:46:54.469 DEBUG:boto:Token: None 2024-04-12T03:46:54.469 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj2 2024-04-12T03:46:54.469 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OfPuAw1WK+VVY0p8H+PRH2cVImY= 2024-04-12T03:46:54.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OfPuAw1WK+VVY0p8H+PRH2cVImY=', 'Content-Length': '0'} 2024-04-12T03:46:54.471 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c380cfa76d8a090-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.472 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.472 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.472 DEBUG:boto:path=/dkyges-48/?prefix=obj2 2024-04-12T03:46:54.472 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj2 2024-04-12T03:46:54.472 DEBUG:boto:Method: GET 2024-04-12T03:46:54.472 DEBUG:boto:Path: /dkyges-48/?prefix=obj2 2024-04-12T03:46:54.472 DEBUG:boto:Data: 2024-04-12T03:46:54.472 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.472 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.472 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.472 DEBUG:boto:Params: {} 2024-04-12T03:46:54.473 DEBUG:boto:Token: None 2024-04-12T03:46:54.473 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.473 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc= 2024-04-12T03:46:54.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc=', 'Content-Length': '0'} 2024-04-12T03:46:54.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005aa666625a7f638d-006618aeae-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.475 DEBUG:boto:b'dkyges-48obj21000falseobj22024-04-12T03:46:13.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.476 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.476 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.476 DEBUG:boto:path=/dkyges-48/?prefix=obj2 2024-04-12T03:46:54.476 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj2 2024-04-12T03:46:54.476 DEBUG:boto:Method: GET 2024-04-12T03:46:54.476 DEBUG:boto:Path: /dkyges-48/?prefix=obj2 2024-04-12T03:46:54.476 DEBUG:boto:Data: 2024-04-12T03:46:54.476 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.477 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.477 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.477 DEBUG:boto:Params: {} 2024-04-12T03:46:54.477 DEBUG:boto:Token: None 2024-04-12T03:46:54.477 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.477 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc= 2024-04-12T03:46:54.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc=', 'Content-Length': '0'} 2024-04-12T03:46:54.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000629dd23ec4366ef6-006618aeae-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.522 DEBUG:boto:b'dkyges-48obj21000falseobj22024-04-12T03:46:13.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:46:54.523 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:46:54.523 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:46:54.523 DEBUG:boto:path=/dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.523 DEBUG:boto:auth_path=/dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.524 DEBUG:boto:Method: GET 2024-04-12T03:46:54.524 DEBUG:boto:Path: /dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.524 DEBUG:boto:Data: 2024-04-12T03:46:54.524 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.524 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.524 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.524 DEBUG:boto:Params: {} 2024-04-12T03:46:54.524 DEBUG:boto:Token: None 2024-04-12T03:46:54.524 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.525 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9kwTdU2wHowfW1eVIkGAz6++Z3U= 2024-04-12T03:46:54.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9kwTdU2wHowfW1eVIkGAz6++Z3U=', 'Content-Length': '0'} 2024-04-12T03:46:54.527 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0247851b41fd1a2-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.527 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:46:54.527 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:46:54.527 DEBUG:boto:path=/dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.527 DEBUG:boto:auth_path=/dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.527 DEBUG:boto:Method: GET 2024-04-12T03:46:54.528 DEBUG:boto:Path: /dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.528 DEBUG:boto:Data: 2024-04-12T03:46:54.528 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.528 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.528 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.528 DEBUG:boto:Params: {} 2024-04-12T03:46:54.528 DEBUG:boto:Token: None 2024-04-12T03:46:54.528 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.528 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9kwTdU2wHowfW1eVIkGAz6++Z3U= 2024-04-12T03:46:54.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9kwTdU2wHowfW1eVIkGAz6++Z3U=', 'Content-Length': '0'} 2024-04-12T03:46:54.530 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006eef37197a59b3e7-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.531 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:46:54.531 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:46:54.531 DEBUG:boto:path=/dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.531 DEBUG:boto:auth_path=/dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.531 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.531 DEBUG:boto:Path: /dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.531 DEBUG:boto:Data: 2024-04-12T03:46:54.531 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.531 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.531 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.532 DEBUG:boto:Params: {} 2024-04-12T03:46:54.532 DEBUG:boto:Token: None 2024-04-12T03:46:54.532 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.532 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yPpNzkaWWAglUaNC5zv5ymOTpxg= 2024-04-12T03:46:54.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yPpNzkaWWAglUaNC5zv5ymOTpxg=', 'Content-Length': '0'} 2024-04-12T03:46:54.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1bd07dc9bade555-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.534 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:46:54.534 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:46:54.534 DEBUG:boto:path=/dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.534 DEBUG:boto:auth_path=/dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.534 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.534 DEBUG:boto:Path: /dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.534 DEBUG:boto:Data: 2024-04-12T03:46:54.535 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.535 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.535 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.535 DEBUG:boto:Params: {} 2024-04-12T03:46:54.535 DEBUG:boto:Token: None 2024-04-12T03:46:54.535 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.535 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yPpNzkaWWAglUaNC5zv5ymOTpxg= 2024-04-12T03:46:54.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yPpNzkaWWAglUaNC5zv5ymOTpxg=', 'Content-Length': '0'} 2024-04-12T03:46:54.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003645aa8d566a705e-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:46:54.537 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:46:54.537 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:46:54.537 DEBUG:boto:path=/dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.538 DEBUG:boto:auth_path=/dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.538 DEBUG:boto:Method: GET 2024-04-12T03:46:54.538 DEBUG:boto:Path: /dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.538 DEBUG:boto:Data: 2024-04-12T03:46:54.538 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.538 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.538 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.538 DEBUG:boto:Params: {} 2024-04-12T03:46:54.538 DEBUG:boto:Token: None 2024-04-12T03:46:54.538 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.539 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9kwTdU2wHowfW1eVIkGAz6++Z3U= 2024-04-12T03:46:54.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9kwTdU2wHowfW1eVIkGAz6++Z3U=', 'Content-Length': '0'} 2024-04-12T03:46:54.540 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d567cecbced7965b-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.582 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:46:54.583 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:46:54.583 DEBUG:boto:path=/dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.583 DEBUG:boto:auth_path=/dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.583 DEBUG:boto:Method: GET 2024-04-12T03:46:54.583 DEBUG:boto:Path: /dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.583 DEBUG:boto:Data: 2024-04-12T03:46:54.583 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.583 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.583 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.583 DEBUG:boto:Params: {} 2024-04-12T03:46:54.584 DEBUG:boto:Token: None 2024-04-12T03:46:54.584 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj3?versionId=null 2024-04-12T03:46:54.584 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9kwTdU2wHowfW1eVIkGAz6++Z3U= 2024-04-12T03:46:54.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9kwTdU2wHowfW1eVIkGAz6++Z3U=', 'Content-Length': '0'} 2024-04-12T03:46:54.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd1b879fc05fea02-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.586 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.586 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.586 DEBUG:boto:path=/dkyges-48/?prefix=obj3 2024-04-12T03:46:54.586 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj3 2024-04-12T03:46:54.587 DEBUG:boto:Method: GET 2024-04-12T03:46:54.587 DEBUG:boto:Path: /dkyges-48/?prefix=obj3 2024-04-12T03:46:54.587 DEBUG:boto:Data: 2024-04-12T03:46:54.587 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.587 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.587 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.587 DEBUG:boto:Params: {} 2024-04-12T03:46:54.587 DEBUG:boto:Token: None 2024-04-12T03:46:54.587 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.588 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc= 2024-04-12T03:46:54.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc=', 'Content-Length': '0'} 2024-04-12T03:46:54.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba2188697682371b-006618aeae-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.590 DEBUG:boto:b'dkyges-48obj31000falseobj32024-04-12T03:46:13.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.591 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.591 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.591 DEBUG:boto:path=/dkyges-48/?prefix=obj3 2024-04-12T03:46:54.591 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj3 2024-04-12T03:46:54.591 DEBUG:boto:Method: GET 2024-04-12T03:46:54.591 DEBUG:boto:Path: /dkyges-48/?prefix=obj3 2024-04-12T03:46:54.591 DEBUG:boto:Data: 2024-04-12T03:46:54.591 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.592 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.592 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.592 DEBUG:boto:Params: {} 2024-04-12T03:46:54.592 DEBUG:boto:Token: None 2024-04-12T03:46:54.592 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.592 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc= 2024-04-12T03:46:54.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc=', 'Content-Length': '0'} 2024-04-12T03:46:54.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065f4b5b9b1bc24f1-006618aeae-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.595 DEBUG:boto:b'dkyges-48obj31000falseobj32024-04-12T03:46:13.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.595 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:46:54.595 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:46:54.595 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.595 DEBUG:boto:Path: /dkyges-48/obj3 2024-04-12T03:46:54.596 DEBUG:boto:Data: 2024-04-12T03:46:54.596 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.596 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.596 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.596 DEBUG:boto:Params: {} 2024-04-12T03:46:54.596 DEBUG:boto:Token: None 2024-04-12T03:46:54.596 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj3 2024-04-12T03:46:54.596 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:geiiNWS+zjWYnvChcbCRU9HarR0= 2024-04-12T03:46:54.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:geiiNWS+zjWYnvChcbCRU9HarR0=', 'Content-Length': '0'} 2024-04-12T03:46:54.598 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000939c8efdc86fb260-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.598 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:46:54.599 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:46:54.599 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.599 DEBUG:boto:Path: /dkyges-48/obj3 2024-04-12T03:46:54.599 DEBUG:boto:Data: 2024-04-12T03:46:54.599 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.599 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.599 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.599 DEBUG:boto:Params: {} 2024-04-12T03:46:54.599 DEBUG:boto:Token: None 2024-04-12T03:46:54.599 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj3 2024-04-12T03:46:54.600 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:geiiNWS+zjWYnvChcbCRU9HarR0= 2024-04-12T03:46:54.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:geiiNWS+zjWYnvChcbCRU9HarR0=', 'Content-Length': '0'} 2024-04-12T03:46:54.601 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf62b005d29cc53c-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:46:54.601 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:46:54.602 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:46:54.602 DEBUG:boto:Method: GET 2024-04-12T03:46:54.602 DEBUG:boto:Path: /dkyges-48/obj3 2024-04-12T03:46:54.602 DEBUG:boto:Data: 2024-04-12T03:46:54.602 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.602 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.602 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.602 DEBUG:boto:Params: {} 2024-04-12T03:46:54.602 DEBUG:boto:Token: None 2024-04-12T03:46:54.602 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj3 2024-04-12T03:46:54.603 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1yG8TOu7KbaV6hEsKkhFVHeTJQw= 2024-04-12T03:46:54.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1yG8TOu7KbaV6hEsKkhFVHeTJQw=', 'Content-Length': '0'} 2024-04-12T03:46:54.604 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0dcda88c3ae5b45-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.646 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:46:54.647 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:46:54.647 DEBUG:boto:Method: GET 2024-04-12T03:46:54.647 DEBUG:boto:Path: /dkyges-48/obj3 2024-04-12T03:46:54.647 DEBUG:boto:Data: 2024-04-12T03:46:54.647 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.647 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.647 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.647 DEBUG:boto:Params: {} 2024-04-12T03:46:54.647 DEBUG:boto:Token: None 2024-04-12T03:46:54.648 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj3 2024-04-12T03:46:54.648 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1yG8TOu7KbaV6hEsKkhFVHeTJQw= 2024-04-12T03:46:54.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1yG8TOu7KbaV6hEsKkhFVHeTJQw=', 'Content-Length': '0'} 2024-04-12T03:46:54.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eac2c09f84e10f8b-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.650 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.651 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.651 DEBUG:boto:path=/dkyges-48/?prefix=obj3 2024-04-12T03:46:54.651 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj3 2024-04-12T03:46:54.651 DEBUG:boto:Method: GET 2024-04-12T03:46:54.651 DEBUG:boto:Path: /dkyges-48/?prefix=obj3 2024-04-12T03:46:54.651 DEBUG:boto:Data: 2024-04-12T03:46:54.651 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.651 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.651 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.651 DEBUG:boto:Params: {} 2024-04-12T03:46:54.652 DEBUG:boto:Token: None 2024-04-12T03:46:54.652 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.652 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc= 2024-04-12T03:46:54.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc=', 'Content-Length': '0'} 2024-04-12T03:46:54.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cddb67e4c79db53-006618aeae-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.655 DEBUG:boto:b'dkyges-48obj31000falseobj32024-04-12T03:46:13.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.655 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.655 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.655 DEBUG:boto:path=/dkyges-48/?prefix=obj3 2024-04-12T03:46:54.655 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj3 2024-04-12T03:46:54.655 DEBUG:boto:Method: GET 2024-04-12T03:46:54.656 DEBUG:boto:Path: /dkyges-48/?prefix=obj3 2024-04-12T03:46:54.656 DEBUG:boto:Data: 2024-04-12T03:46:54.656 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.656 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.656 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.656 DEBUG:boto:Params: {} 2024-04-12T03:46:54.656 DEBUG:boto:Token: None 2024-04-12T03:46:54.656 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc= 2024-04-12T03:46:54.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc=', 'Content-Length': '0'} 2024-04-12T03:46:54.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c2f00aef2f18735-006618aeae-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.659 DEBUG:boto:b'dkyges-48obj31000falseobj32024-04-12T03:46:13.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:46:54.659 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:46:54.659 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:46:54.659 DEBUG:boto:path=/dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.660 DEBUG:boto:auth_path=/dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.660 DEBUG:boto:Method: GET 2024-04-12T03:46:54.660 DEBUG:boto:Path: /dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.660 DEBUG:boto:Data: 2024-04-12T03:46:54.660 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.660 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.660 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.660 DEBUG:boto:Params: {} 2024-04-12T03:46:54.660 DEBUG:boto:Token: None 2024-04-12T03:46:54.660 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.661 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qIWr+ikQU2pjwi3guWBB3TKlT6o= 2024-04-12T03:46:54.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qIWr+ikQU2pjwi3guWBB3TKlT6o=', 'Content-Length': '0'} 2024-04-12T03:46:54.663 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e761091f89d4a29-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.663 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:46:54.663 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:46:54.663 DEBUG:boto:path=/dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.663 DEBUG:boto:auth_path=/dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.663 DEBUG:boto:Method: GET 2024-04-12T03:46:54.664 DEBUG:boto:Path: /dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.664 DEBUG:boto:Data: 2024-04-12T03:46:54.664 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.664 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.664 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.664 DEBUG:boto:Params: {} 2024-04-12T03:46:54.664 DEBUG:boto:Token: None 2024-04-12T03:46:54.664 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.664 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qIWr+ikQU2pjwi3guWBB3TKlT6o= 2024-04-12T03:46:54.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qIWr+ikQU2pjwi3guWBB3TKlT6o=', 'Content-Length': '0'} 2024-04-12T03:46:54.667 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a03f162d9a986336-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.667 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:46:54.667 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:46:54.667 DEBUG:boto:path=/dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.667 DEBUG:boto:auth_path=/dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.667 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.668 DEBUG:boto:Path: /dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.668 DEBUG:boto:Data: 2024-04-12T03:46:54.668 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.668 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.668 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.668 DEBUG:boto:Params: {} 2024-04-12T03:46:54.668 DEBUG:boto:Token: None 2024-04-12T03:46:54.668 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.668 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pZd+E86H/xkoj83JQbox90vwldg= 2024-04-12T03:46:54.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pZd+E86H/xkoj83JQbox90vwldg=', 'Content-Length': '0'} 2024-04-12T03:46:54.670 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004766c067b829e440-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.670 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:46:54.671 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:46:54.671 DEBUG:boto:path=/dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.671 DEBUG:boto:auth_path=/dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.671 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.671 DEBUG:boto:Path: /dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.671 DEBUG:boto:Data: 2024-04-12T03:46:54.671 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.671 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.671 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.671 DEBUG:boto:Params: {} 2024-04-12T03:46:54.672 DEBUG:boto:Token: None 2024-04-12T03:46:54.672 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.672 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pZd+E86H/xkoj83JQbox90vwldg= 2024-04-12T03:46:54.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pZd+E86H/xkoj83JQbox90vwldg=', 'Content-Length': '0'} 2024-04-12T03:46:54.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000820c7961f9bbd58-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:46:54.674 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:46:54.674 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:46:54.674 DEBUG:boto:path=/dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.674 DEBUG:boto:auth_path=/dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.674 DEBUG:boto:Method: GET 2024-04-12T03:46:54.675 DEBUG:boto:Path: /dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.675 DEBUG:boto:Data: 2024-04-12T03:46:54.675 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.675 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.675 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.675 DEBUG:boto:Params: {} 2024-04-12T03:46:54.675 DEBUG:boto:Token: None 2024-04-12T03:46:54.675 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.675 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qIWr+ikQU2pjwi3guWBB3TKlT6o= 2024-04-12T03:46:54.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qIWr+ikQU2pjwi3guWBB3TKlT6o=', 'Content-Length': '0'} 2024-04-12T03:46:54.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007eac5c5f51195c48-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.718 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:46:54.719 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:46:54.719 DEBUG:boto:path=/dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.719 DEBUG:boto:auth_path=/dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.719 DEBUG:boto:Method: GET 2024-04-12T03:46:54.719 DEBUG:boto:Path: /dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.719 DEBUG:boto:Data: 2024-04-12T03:46:54.719 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.719 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.719 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.719 DEBUG:boto:Params: {} 2024-04-12T03:46:54.720 DEBUG:boto:Token: None 2024-04-12T03:46:54.720 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj4?versionId=null 2024-04-12T03:46:54.720 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qIWr+ikQU2pjwi3guWBB3TKlT6o= 2024-04-12T03:46:54.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qIWr+ikQU2pjwi3guWBB3TKlT6o=', 'Content-Length': '0'} 2024-04-12T03:46:54.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a7b1ffac8f51cc7-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.723 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.723 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.723 DEBUG:boto:path=/dkyges-48/?prefix=obj4 2024-04-12T03:46:54.723 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj4 2024-04-12T03:46:54.723 DEBUG:boto:Method: GET 2024-04-12T03:46:54.723 DEBUG:boto:Path: /dkyges-48/?prefix=obj4 2024-04-12T03:46:54.723 DEBUG:boto:Data: 2024-04-12T03:46:54.723 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.723 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.723 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.723 DEBUG:boto:Params: {} 2024-04-12T03:46:54.724 DEBUG:boto:Token: None 2024-04-12T03:46:54.724 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.724 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc= 2024-04-12T03:46:54.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc=', 'Content-Length': '0'} 2024-04-12T03:46:54.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081aa190805fca4db-006618aeae-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.726 DEBUG:boto:b'dkyges-48obj41000falseobj42024-04-12T03:46:13.631Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.727 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.727 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.727 DEBUG:boto:path=/dkyges-48/?prefix=obj4 2024-04-12T03:46:54.727 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj4 2024-04-12T03:46:54.728 DEBUG:boto:Method: GET 2024-04-12T03:46:54.728 DEBUG:boto:Path: /dkyges-48/?prefix=obj4 2024-04-12T03:46:54.728 DEBUG:boto:Data: 2024-04-12T03:46:54.728 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.728 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.728 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.728 DEBUG:boto:Params: {} 2024-04-12T03:46:54.728 DEBUG:boto:Token: None 2024-04-12T03:46:54.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.729 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc= 2024-04-12T03:46:54.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc=', 'Content-Length': '0'} 2024-04-12T03:46:54.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdb1a025da4f2076-006618aeae-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.731 DEBUG:boto:b'dkyges-48obj41000falseobj42024-04-12T03:46:13.631Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.732 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:46:54.732 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:46:54.732 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.732 DEBUG:boto:Path: /dkyges-48/obj4 2024-04-12T03:46:54.732 DEBUG:boto:Data: 2024-04-12T03:46:54.732 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.732 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.732 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.733 DEBUG:boto:Params: {} 2024-04-12T03:46:54.733 DEBUG:boto:Token: None 2024-04-12T03:46:54.733 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj4 2024-04-12T03:46:54.733 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tgknu8p0MUCt7Ui7IjTcMp89C9g= 2024-04-12T03:46:54.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tgknu8p0MUCt7Ui7IjTcMp89C9g=', 'Content-Length': '0'} 2024-04-12T03:46:54.735 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ceaf6aa7826e8df-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.735 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:46:54.735 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:46:54.735 DEBUG:boto:Method: HEAD 2024-04-12T03:46:54.735 DEBUG:boto:Path: /dkyges-48/obj4 2024-04-12T03:46:54.735 DEBUG:boto:Data: 2024-04-12T03:46:54.735 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.735 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.736 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.736 DEBUG:boto:Params: {} 2024-04-12T03:46:54.736 DEBUG:boto:Token: None 2024-04-12T03:46:54.736 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj4 2024-04-12T03:46:54.736 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tgknu8p0MUCt7Ui7IjTcMp89C9g= 2024-04-12T03:46:54.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tgknu8p0MUCt7Ui7IjTcMp89C9g=', 'Content-Length': '0'} 2024-04-12T03:46:54.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025d48abe326fff05-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:46:54.738 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:46:54.738 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:46:54.739 DEBUG:boto:Method: GET 2024-04-12T03:46:54.739 DEBUG:boto:Path: /dkyges-48/obj4 2024-04-12T03:46:54.739 DEBUG:boto:Data: 2024-04-12T03:46:54.739 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.739 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.739 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.739 DEBUG:boto:Params: {} 2024-04-12T03:46:54.739 DEBUG:boto:Token: None 2024-04-12T03:46:54.739 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj4 2024-04-12T03:46:54.739 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D4+rzw11N8Pl0BL4Bo7ZbDvPyKM= 2024-04-12T03:46:54.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D4+rzw11N8Pl0BL4Bo7ZbDvPyKM=', 'Content-Length': '0'} 2024-04-12T03:46:54.741 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b303b336a3dcb1b-006618aeae-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.786 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:46:54.787 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:46:54.787 DEBUG:boto:Method: GET 2024-04-12T03:46:54.787 DEBUG:boto:Path: /dkyges-48/obj4 2024-04-12T03:46:54.787 DEBUG:boto:Data: 2024-04-12T03:46:54.787 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.787 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.787 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.787 DEBUG:boto:Params: {} 2024-04-12T03:46:54.787 DEBUG:boto:Token: None 2024-04-12T03:46:54.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/obj4 2024-04-12T03:46:54.788 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D4+rzw11N8Pl0BL4Bo7ZbDvPyKM= 2024-04-12T03:46:54.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D4+rzw11N8Pl0BL4Bo7ZbDvPyKM=', 'Content-Length': '0'} 2024-04-12T03:46:54.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000501cff11fee949f9-006618aeae-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.790 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.790 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.790 DEBUG:boto:path=/dkyges-48/?prefix=obj4 2024-04-12T03:46:54.790 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj4 2024-04-12T03:46:54.791 DEBUG:boto:Method: GET 2024-04-12T03:46:54.791 DEBUG:boto:Path: /dkyges-48/?prefix=obj4 2024-04-12T03:46:54.791 DEBUG:boto:Data: 2024-04-12T03:46:54.791 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.791 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:54.791 DEBUG:boto:Port: 8001 2024-04-12T03:46:54.791 DEBUG:boto:Params: {} 2024-04-12T03:46:54.791 DEBUG:boto:Token: None 2024-04-12T03:46:54.791 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.791 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc= 2024-04-12T03:46:54.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc=', 'Content-Length': '0'} 2024-04-12T03:46:54.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d04d5003c64e934f-006618aeae-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.794 DEBUG:boto:b'dkyges-48obj41000falseobj42024-04-12T03:46:13.631Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.795 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:54.795 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:54.795 DEBUG:boto:path=/dkyges-48/?prefix=obj4 2024-04-12T03:46:54.795 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj4 2024-04-12T03:46:54.795 DEBUG:boto:Method: GET 2024-04-12T03:46:54.795 DEBUG:boto:Path: /dkyges-48/?prefix=obj4 2024-04-12T03:46:54.795 DEBUG:boto:Data: 2024-04-12T03:46:54.795 DEBUG:boto:Headers: {} 2024-04-12T03:46:54.795 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:54.795 DEBUG:boto:Port: 8000 2024-04-12T03:46:54.795 DEBUG:boto:Params: {} 2024-04-12T03:46:54.796 DEBUG:boto:Token: None 2024-04-12T03:46:54.796 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:54 GMT /dkyges-48/ 2024-04-12T03:46:54.796 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc= 2024-04-12T03:46:54.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B/Hmy9CKiiuPXG7uISpzqK7ktsc=', 'Content-Length': '0'} 2024-04-12T03:46:54.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fb2d4612c61b809-006618aeae-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:54.799 DEBUG:boto:b'dkyges-48obj41000falseobj42024-04-12T03:46:13.631Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:54.799 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-48 zones={a2, a1} 2024-04-12T03:46:54.799 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-49 2024-04-12T03:46:54.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:46:54.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:46:54.800 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:46:54.866 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:54.866 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:54.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:54.910+0000 7f00b685b780 1 bucket sync caught up with source: 2024-04-12T03:46:54.909 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.36.6, , 00000000001.39.6, , 00000000001.38.6, , , 00000000001.19.6, , , ] 2024-04-12T03:46:54.909 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.36.6, , 00000000001.39.6, , 00000000001.38.6, , , 00000000001.19.6, , , ] 2024-04-12T03:46:54.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:54.910+0000 7f00b685b780 0 bucket checkpoint complete 2024-04-12T03:46:54.925 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-49 2024-04-12T03:46:54.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:46:54.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:46:54.925 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:46:54.989 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:54.989 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:55.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:55.032+0000 7fb0a9db2780 1 bucket sync caught up with source: 2024-04-12T03:46:55.031 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.1540.6, , 00000000001.3579.6, , 00000000001.2561.6, , , 00000000001.1876.6, , , ] 2024-04-12T03:46:55.031 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.1540.6, , 00000000001.3579.6, , 00000000001.2561.6, , , 00000000001.1876.6, , , ] 2024-04-12T03:46:55.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:55.032+0000 7fb0a9db2780 0 bucket checkpoint complete 2024-04-12T03:46:55.046 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-49 zones={a2, a1} 2024-04-12T03:46:55.046 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.046 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.046 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.046 DEBUG:boto:Path: /dkyges-49/ 2024-04-12T03:46:55.046 DEBUG:boto:Data: 2024-04-12T03:46:55.046 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.046 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.046 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.047 DEBUG:boto:Params: {} 2024-04-12T03:46:55.047 DEBUG:boto:Token: None 2024-04-12T03:46:55.047 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.047 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SZr7VCwfA2IFjhC2rhasRM6O9sE= 2024-04-12T03:46:55.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SZr7VCwfA2IFjhC2rhasRM6O9sE=', 'Content-Length': '0'} 2024-04-12T03:46:55.050 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bce060a36a6e5c5-006618aeaf-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.050 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.050 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.050 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.050 DEBUG:boto:Path: /dkyges-49/ 2024-04-12T03:46:55.050 DEBUG:boto:Data: 2024-04-12T03:46:55.051 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.051 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.051 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.051 DEBUG:boto:Params: {} 2024-04-12T03:46:55.051 DEBUG:boto:Token: None 2024-04-12T03:46:55.051 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.051 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SZr7VCwfA2IFjhC2rhasRM6O9sE= 2024-04-12T03:46:55.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SZr7VCwfA2IFjhC2rhasRM6O9sE=', 'Content-Length': '0'} 2024-04-12T03:46:55.055 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba073e3afaf2fa0f-006618aeaf-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.055 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:46:55.055 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.056 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.056 DEBUG:boto:path=/dkyges-49/?versions&max-keys=999 2024-04-12T03:46:55.057 DEBUG:boto:auth_path=/dkyges-49/?versions&max-keys=999 2024-04-12T03:46:55.057 DEBUG:boto:Method: GET 2024-04-12T03:46:55.057 DEBUG:boto:Path: /dkyges-49/?versions&max-keys=999 2024-04-12T03:46:55.057 DEBUG:boto:Data: 2024-04-12T03:46:55.058 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.058 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.058 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.058 DEBUG:boto:Params: {} 2024-04-12T03:46:55.058 DEBUG:boto:Token: None 2024-04-12T03:46:55.058 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/?versions 2024-04-12T03:46:55.058 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UNuJ5LjolCVGxkHg/rto/uE8Ygg= 2024-04-12T03:46:55.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UNuJ5LjolCVGxkHg/rto/uE8Ygg=', 'Content-Length': '0'} 2024-04-12T03:46:55.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5f9f0846056d02f-006618aeaf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.061 DEBUG:boto:b'dkyges-49999falseobj1nulltrue2024-04-12T03:46:13.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:46:13.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:46:13.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:46:13.731Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:46:55.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:46:55.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:46:55.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:46:55.062 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:46:55.062 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.062 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.062 DEBUG:boto:path=/dkyges-49/?versions&max-keys=999 2024-04-12T03:46:55.062 DEBUG:boto:auth_path=/dkyges-49/?versions&max-keys=999 2024-04-12T03:46:55.062 DEBUG:boto:Method: GET 2024-04-12T03:46:55.062 DEBUG:boto:Path: /dkyges-49/?versions&max-keys=999 2024-04-12T03:46:55.063 DEBUG:boto:Data: 2024-04-12T03:46:55.063 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.063 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.063 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.063 DEBUG:boto:Params: {} 2024-04-12T03:46:55.063 DEBUG:boto:Token: None 2024-04-12T03:46:55.063 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/?versions 2024-04-12T03:46:55.063 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UNuJ5LjolCVGxkHg/rto/uE8Ygg= 2024-04-12T03:46:55.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UNuJ5LjolCVGxkHg/rto/uE8Ygg=', 'Content-Length': '0'} 2024-04-12T03:46:55.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d33feb7dc5fef599-006618aeaf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.066 DEBUG:boto:b'dkyges-49999falseobj1nulltrue2024-04-12T03:46:13.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:46:13.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:46:13.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:46:13.731Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:46:55.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:46:55.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:46:55.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:46:55.067 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.067 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.067 DEBUG:boto:path=/dkyges-49/?versions&max-keys=999 2024-04-12T03:46:55.067 DEBUG:boto:auth_path=/dkyges-49/?versions&max-keys=999 2024-04-12T03:46:55.067 DEBUG:boto:Method: GET 2024-04-12T03:46:55.067 DEBUG:boto:Path: /dkyges-49/?versions&max-keys=999 2024-04-12T03:46:55.067 DEBUG:boto:Data: 2024-04-12T03:46:55.067 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.067 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.067 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.067 DEBUG:boto:Params: {} 2024-04-12T03:46:55.068 DEBUG:boto:Token: None 2024-04-12T03:46:55.068 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/?versions 2024-04-12T03:46:55.068 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UNuJ5LjolCVGxkHg/rto/uE8Ygg= 2024-04-12T03:46:55.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UNuJ5LjolCVGxkHg/rto/uE8Ygg=', 'Content-Length': '0'} 2024-04-12T03:46:55.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d343723c950d304-006618aeaf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.070 DEBUG:boto:b'dkyges-49999falseobj1nulltrue2024-04-12T03:46:13.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:46:13.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:46:13.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:46:13.731Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.071 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.071 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.071 DEBUG:boto:path=/dkyges-49/?versions&max-keys=999 2024-04-12T03:46:55.071 DEBUG:boto:auth_path=/dkyges-49/?versions&max-keys=999 2024-04-12T03:46:55.071 DEBUG:boto:Method: GET 2024-04-12T03:46:55.071 DEBUG:boto:Path: /dkyges-49/?versions&max-keys=999 2024-04-12T03:46:55.071 DEBUG:boto:Data: 2024-04-12T03:46:55.071 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.071 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.072 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.072 DEBUG:boto:Params: {} 2024-04-12T03:46:55.072 DEBUG:boto:Token: None 2024-04-12T03:46:55.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/?versions 2024-04-12T03:46:55.072 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UNuJ5LjolCVGxkHg/rto/uE8Ygg= 2024-04-12T03:46:55.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UNuJ5LjolCVGxkHg/rto/uE8Ygg=', 'Content-Length': '0'} 2024-04-12T03:46:55.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa48ff1433a50c4a-006618aeaf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.074 DEBUG:boto:b'dkyges-49999falseobj1nulltrue2024-04-12T03:46:13.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:46:13.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:46:13.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:46:13.731Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:46:55.075 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:46:55.075 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:46:55.075 DEBUG:boto:path=/dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.075 DEBUG:boto:auth_path=/dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.075 DEBUG:boto:Method: GET 2024-04-12T03:46:55.075 DEBUG:boto:Path: /dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.075 DEBUG:boto:Data: 2024-04-12T03:46:55.076 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.076 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.076 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.076 DEBUG:boto:Params: {} 2024-04-12T03:46:55.076 DEBUG:boto:Token: None 2024-04-12T03:46:55.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.076 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NwlMCZCLgzWZM7YFwThkz++6nQk= 2024-04-12T03:46:55.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NwlMCZCLgzWZM7YFwThkz++6nQk=', 'Content-Length': '0'} 2024-04-12T03:46:55.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000268ff1dbce5fe6cc-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.078 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:46:55.078 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:46:55.078 DEBUG:boto:path=/dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.078 DEBUG:boto:auth_path=/dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.079 DEBUG:boto:Method: GET 2024-04-12T03:46:55.079 DEBUG:boto:Path: /dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.079 DEBUG:boto:Data: 2024-04-12T03:46:55.079 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.079 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.079 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.080 DEBUG:boto:Params: {} 2024-04-12T03:46:55.080 DEBUG:boto:Token: None 2024-04-12T03:46:55.080 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.080 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NwlMCZCLgzWZM7YFwThkz++6nQk= 2024-04-12T03:46:55.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NwlMCZCLgzWZM7YFwThkz++6nQk=', 'Content-Length': '0'} 2024-04-12T03:46:55.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b269515b771c845-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.082 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:46:55.082 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:46:55.082 DEBUG:boto:path=/dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.082 DEBUG:boto:auth_path=/dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.082 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.082 DEBUG:boto:Path: /dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.082 DEBUG:boto:Data: 2024-04-12T03:46:55.082 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.082 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.083 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.083 DEBUG:boto:Params: {} 2024-04-12T03:46:55.083 DEBUG:boto:Token: None 2024-04-12T03:46:55.083 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.083 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3BVF18c0H6fYZ4Dnw6F1fdAB1O8= 2024-04-12T03:46:55.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3BVF18c0H6fYZ4Dnw6F1fdAB1O8=', 'Content-Length': '0'} 2024-04-12T03:46:55.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002812cb72b38b12ff-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.085 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:46:55.085 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:46:55.085 DEBUG:boto:path=/dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.085 DEBUG:boto:auth_path=/dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.085 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.085 DEBUG:boto:Path: /dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.085 DEBUG:boto:Data: 2024-04-12T03:46:55.085 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.085 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.085 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.086 DEBUG:boto:Params: {} 2024-04-12T03:46:55.086 DEBUG:boto:Token: None 2024-04-12T03:46:55.086 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.086 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3BVF18c0H6fYZ4Dnw6F1fdAB1O8= 2024-04-12T03:46:55.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3BVF18c0H6fYZ4Dnw6F1fdAB1O8=', 'Content-Length': '0'} 2024-04-12T03:46:55.087 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e209faae72308fb-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:46:55.088 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:46:55.088 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:46:55.088 DEBUG:boto:path=/dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.088 DEBUG:boto:auth_path=/dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.088 DEBUG:boto:Method: GET 2024-04-12T03:46:55.088 DEBUG:boto:Path: /dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.088 DEBUG:boto:Data: 2024-04-12T03:46:55.089 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.089 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.089 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.089 DEBUG:boto:Params: {} 2024-04-12T03:46:55.089 DEBUG:boto:Token: None 2024-04-12T03:46:55.089 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.089 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NwlMCZCLgzWZM7YFwThkz++6nQk= 2024-04-12T03:46:55.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NwlMCZCLgzWZM7YFwThkz++6nQk=', 'Content-Length': '0'} 2024-04-12T03:46:55.091 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a744e0f2190794b7-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.091 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:46:55.092 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:46:55.092 DEBUG:boto:path=/dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.092 DEBUG:boto:auth_path=/dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.092 DEBUG:boto:Method: GET 2024-04-12T03:46:55.092 DEBUG:boto:Path: /dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.092 DEBUG:boto:Data: 2024-04-12T03:46:55.092 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.092 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.092 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.092 DEBUG:boto:Params: {} 2024-04-12T03:46:55.092 DEBUG:boto:Token: None 2024-04-12T03:46:55.093 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj1?versionId=null 2024-04-12T03:46:55.093 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NwlMCZCLgzWZM7YFwThkz++6nQk= 2024-04-12T03:46:55.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NwlMCZCLgzWZM7YFwThkz++6nQk=', 'Content-Length': '0'} 2024-04-12T03:46:55.095 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb9d0b8ab85096e0-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.095 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.095 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.095 DEBUG:boto:path=/dkyges-49/?prefix=obj1 2024-04-12T03:46:55.095 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj1 2024-04-12T03:46:55.095 DEBUG:boto:Method: GET 2024-04-12T03:46:55.096 DEBUG:boto:Path: /dkyges-49/?prefix=obj1 2024-04-12T03:46:55.096 DEBUG:boto:Data: 2024-04-12T03:46:55.096 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.096 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.096 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.096 DEBUG:boto:Params: {} 2024-04-12T03:46:55.096 DEBUG:boto:Token: None 2024-04-12T03:46:55.096 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.096 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA= 2024-04-12T03:46:55.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA=', 'Content-Length': '0'} 2024-04-12T03:46:55.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f55a34ee952d263e-006618aeaf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.098 DEBUG:boto:b'dkyges-49obj11000falseobj12024-04-12T03:46:13.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.099 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.099 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.099 DEBUG:boto:path=/dkyges-49/?prefix=obj1 2024-04-12T03:46:55.099 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj1 2024-04-12T03:46:55.099 DEBUG:boto:Method: GET 2024-04-12T03:46:55.099 DEBUG:boto:Path: /dkyges-49/?prefix=obj1 2024-04-12T03:46:55.099 DEBUG:boto:Data: 2024-04-12T03:46:55.100 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.100 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.100 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.100 DEBUG:boto:Params: {} 2024-04-12T03:46:55.100 DEBUG:boto:Token: None 2024-04-12T03:46:55.100 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.100 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA= 2024-04-12T03:46:55.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA=', 'Content-Length': '0'} 2024-04-12T03:46:55.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3d992edb317bd00-006618aeaf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.102 DEBUG:boto:b'dkyges-49obj11000falseobj12024-04-12T03:46:13.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.103 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:46:55.103 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:46:55.103 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.103 DEBUG:boto:Path: /dkyges-49/obj1 2024-04-12T03:46:55.103 DEBUG:boto:Data: 2024-04-12T03:46:55.103 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.103 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.103 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.103 DEBUG:boto:Params: {} 2024-04-12T03:46:55.104 DEBUG:boto:Token: None 2024-04-12T03:46:55.104 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj1 2024-04-12T03:46:55.104 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3O6jSRa4t7gqCrO5JR4OeDFYuIY= 2024-04-12T03:46:55.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3O6jSRa4t7gqCrO5JR4OeDFYuIY=', 'Content-Length': '0'} 2024-04-12T03:46:55.105 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005557613b6204ebd5-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.106 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:46:55.106 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:46:55.106 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.106 DEBUG:boto:Path: /dkyges-49/obj1 2024-04-12T03:46:55.106 DEBUG:boto:Data: 2024-04-12T03:46:55.106 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.106 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.106 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.106 DEBUG:boto:Params: {} 2024-04-12T03:46:55.106 DEBUG:boto:Token: None 2024-04-12T03:46:55.107 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj1 2024-04-12T03:46:55.107 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3O6jSRa4t7gqCrO5JR4OeDFYuIY= 2024-04-12T03:46:55.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3O6jSRa4t7gqCrO5JR4OeDFYuIY=', 'Content-Length': '0'} 2024-04-12T03:46:55.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001663843f65711178-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:46:55.109 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:46:55.109 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:46:55.109 DEBUG:boto:Method: GET 2024-04-12T03:46:55.109 DEBUG:boto:Path: /dkyges-49/obj1 2024-04-12T03:46:55.109 DEBUG:boto:Data: 2024-04-12T03:46:55.109 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.109 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.109 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.109 DEBUG:boto:Params: {} 2024-04-12T03:46:55.109 DEBUG:boto:Token: None 2024-04-12T03:46:55.110 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj1 2024-04-12T03:46:55.110 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:neofmNVL0Bu2fKUxmEGNb3K0fVE= 2024-04-12T03:46:55.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:neofmNVL0Bu2fKUxmEGNb3K0fVE=', 'Content-Length': '0'} 2024-04-12T03:46:55.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000089edbd7606ae7f21-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.112 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:46:55.112 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:46:55.112 DEBUG:boto:Method: GET 2024-04-12T03:46:55.112 DEBUG:boto:Path: /dkyges-49/obj1 2024-04-12T03:46:55.112 DEBUG:boto:Data: 2024-04-12T03:46:55.112 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.112 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.112 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.112 DEBUG:boto:Params: {} 2024-04-12T03:46:55.112 DEBUG:boto:Token: None 2024-04-12T03:46:55.113 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj1 2024-04-12T03:46:55.113 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:neofmNVL0Bu2fKUxmEGNb3K0fVE= 2024-04-12T03:46:55.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:neofmNVL0Bu2fKUxmEGNb3K0fVE=', 'Content-Length': '0'} 2024-04-12T03:46:55.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc10b22107fdc258-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.158 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.159 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.159 DEBUG:boto:path=/dkyges-49/?prefix=obj1 2024-04-12T03:46:55.159 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj1 2024-04-12T03:46:55.159 DEBUG:boto:Method: GET 2024-04-12T03:46:55.159 DEBUG:boto:Path: /dkyges-49/?prefix=obj1 2024-04-12T03:46:55.159 DEBUG:boto:Data: 2024-04-12T03:46:55.159 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.159 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.159 DEBUG:boto:Params: {} 2024-04-12T03:46:55.159 DEBUG:boto:Token: None 2024-04-12T03:46:55.160 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.160 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA= 2024-04-12T03:46:55.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA=', 'Content-Length': '0'} 2024-04-12T03:46:55.163 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000159d564f42a5f94d-006618aeaf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.163 DEBUG:boto:b'dkyges-49obj11000falseobj12024-04-12T03:46:13.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.164 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.164 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.164 DEBUG:boto:path=/dkyges-49/?prefix=obj1 2024-04-12T03:46:55.164 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj1 2024-04-12T03:46:55.164 DEBUG:boto:Method: GET 2024-04-12T03:46:55.164 DEBUG:boto:Path: /dkyges-49/?prefix=obj1 2024-04-12T03:46:55.164 DEBUG:boto:Data: 2024-04-12T03:46:55.164 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.164 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.165 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.165 DEBUG:boto:Params: {} 2024-04-12T03:46:55.165 DEBUG:boto:Token: None 2024-04-12T03:46:55.165 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.165 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA= 2024-04-12T03:46:55.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA=', 'Content-Length': '0'} 2024-04-12T03:46:55.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f671e89d86fd3c0-006618aeaf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.168 DEBUG:boto:b'dkyges-49obj11000falseobj12024-04-12T03:46:13.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:46:55.168 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:46:55.168 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:46:55.169 DEBUG:boto:path=/dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.169 DEBUG:boto:auth_path=/dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.169 DEBUG:boto:Method: GET 2024-04-12T03:46:55.169 DEBUG:boto:Path: /dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.169 DEBUG:boto:Data: 2024-04-12T03:46:55.169 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.169 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.169 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.169 DEBUG:boto:Params: {} 2024-04-12T03:46:55.169 DEBUG:boto:Token: None 2024-04-12T03:46:55.169 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.170 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZXsnUEStoSHzOlE8y5EmqhhGOO4= 2024-04-12T03:46:55.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZXsnUEStoSHzOlE8y5EmqhhGOO4=', 'Content-Length': '0'} 2024-04-12T03:46:55.171 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2a109702a646280-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.214 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:46:55.215 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:46:55.215 DEBUG:boto:path=/dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.215 DEBUG:boto:auth_path=/dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.215 DEBUG:boto:Method: GET 2024-04-12T03:46:55.215 DEBUG:boto:Path: /dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.215 DEBUG:boto:Data: 2024-04-12T03:46:55.215 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.215 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.215 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.215 DEBUG:boto:Params: {} 2024-04-12T03:46:55.216 DEBUG:boto:Token: None 2024-04-12T03:46:55.216 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.216 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZXsnUEStoSHzOlE8y5EmqhhGOO4= 2024-04-12T03:46:55.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZXsnUEStoSHzOlE8y5EmqhhGOO4=', 'Content-Length': '0'} 2024-04-12T03:46:55.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e079688903c0c3e-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.219 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:46:55.219 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:46:55.219 DEBUG:boto:path=/dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.219 DEBUG:boto:auth_path=/dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.219 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.219 DEBUG:boto:Path: /dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.219 DEBUG:boto:Data: 2024-04-12T03:46:55.219 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.220 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.220 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.220 DEBUG:boto:Params: {} 2024-04-12T03:46:55.220 DEBUG:boto:Token: None 2024-04-12T03:46:55.220 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.220 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:J+LVgAckkf8VXl5Gvawe6WGpA8Y= 2024-04-12T03:46:55.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:J+LVgAckkf8VXl5Gvawe6WGpA8Y=', 'Content-Length': '0'} 2024-04-12T03:46:55.222 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007af95afa25a4424b-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.222 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:46:55.222 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:46:55.223 DEBUG:boto:path=/dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.223 DEBUG:boto:auth_path=/dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.223 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.223 DEBUG:boto:Path: /dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.223 DEBUG:boto:Data: 2024-04-12T03:46:55.223 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.223 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.223 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.223 DEBUG:boto:Params: {} 2024-04-12T03:46:55.223 DEBUG:boto:Token: None 2024-04-12T03:46:55.223 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.224 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:J+LVgAckkf8VXl5Gvawe6WGpA8Y= 2024-04-12T03:46:55.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:J+LVgAckkf8VXl5Gvawe6WGpA8Y=', 'Content-Length': '0'} 2024-04-12T03:46:55.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011bd92b1a2dd8fa7-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:46:55.226 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:46:55.226 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:46:55.226 DEBUG:boto:path=/dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.226 DEBUG:boto:auth_path=/dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.226 DEBUG:boto:Method: GET 2024-04-12T03:46:55.226 DEBUG:boto:Path: /dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.226 DEBUG:boto:Data: 2024-04-12T03:46:55.226 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.227 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.227 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.227 DEBUG:boto:Params: {} 2024-04-12T03:46:55.227 DEBUG:boto:Token: None 2024-04-12T03:46:55.227 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.227 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZXsnUEStoSHzOlE8y5EmqhhGOO4= 2024-04-12T03:46:55.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZXsnUEStoSHzOlE8y5EmqhhGOO4=', 'Content-Length': '0'} 2024-04-12T03:46:55.229 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008513c07c13376be3-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.229 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:46:55.230 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:46:55.230 DEBUG:boto:path=/dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.230 DEBUG:boto:auth_path=/dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.230 DEBUG:boto:Method: GET 2024-04-12T03:46:55.230 DEBUG:boto:Path: /dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.230 DEBUG:boto:Data: 2024-04-12T03:46:55.230 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.230 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.230 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.230 DEBUG:boto:Params: {} 2024-04-12T03:46:55.231 DEBUG:boto:Token: None 2024-04-12T03:46:55.231 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj2?versionId=null 2024-04-12T03:46:55.231 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZXsnUEStoSHzOlE8y5EmqhhGOO4= 2024-04-12T03:46:55.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZXsnUEStoSHzOlE8y5EmqhhGOO4=', 'Content-Length': '0'} 2024-04-12T03:46:55.233 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000528bafbd5750208b-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.233 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.233 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.233 DEBUG:boto:path=/dkyges-49/?prefix=obj2 2024-04-12T03:46:55.233 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj2 2024-04-12T03:46:55.234 DEBUG:boto:Method: GET 2024-04-12T03:46:55.234 DEBUG:boto:Path: /dkyges-49/?prefix=obj2 2024-04-12T03:46:55.234 DEBUG:boto:Data: 2024-04-12T03:46:55.234 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.234 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.234 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.234 DEBUG:boto:Params: {} 2024-04-12T03:46:55.234 DEBUG:boto:Token: None 2024-04-12T03:46:55.234 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.234 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA= 2024-04-12T03:46:55.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA=', 'Content-Length': '0'} 2024-04-12T03:46:55.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb6b3143a7352927-006618aeaf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.237 DEBUG:boto:b'dkyges-49obj21000falseobj22024-04-12T03:46:13.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.238 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.238 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.238 DEBUG:boto:path=/dkyges-49/?prefix=obj2 2024-04-12T03:46:55.238 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj2 2024-04-12T03:46:55.238 DEBUG:boto:Method: GET 2024-04-12T03:46:55.238 DEBUG:boto:Path: /dkyges-49/?prefix=obj2 2024-04-12T03:46:55.238 DEBUG:boto:Data: 2024-04-12T03:46:55.238 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.238 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.238 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.238 DEBUG:boto:Params: {} 2024-04-12T03:46:55.239 DEBUG:boto:Token: None 2024-04-12T03:46:55.239 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.239 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA= 2024-04-12T03:46:55.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA=', 'Content-Length': '0'} 2024-04-12T03:46:55.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000402a69a54053411b-006618aeaf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.242 DEBUG:boto:b'dkyges-49obj21000falseobj22024-04-12T03:46:13.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.242 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:46:55.242 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:46:55.242 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.242 DEBUG:boto:Path: /dkyges-49/obj2 2024-04-12T03:46:55.242 DEBUG:boto:Data: 2024-04-12T03:46:55.242 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.243 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.243 DEBUG:boto:Params: {} 2024-04-12T03:46:55.243 DEBUG:boto:Token: None 2024-04-12T03:46:55.243 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj2 2024-04-12T03:46:55.243 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A5ctuJSSJHqf+6rgmrzA4m8UrsE= 2024-04-12T03:46:55.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A5ctuJSSJHqf+6rgmrzA4m8UrsE=', 'Content-Length': '0'} 2024-04-12T03:46:55.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4a7d338399c1064-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.245 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:46:55.245 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:46:55.245 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.245 DEBUG:boto:Path: /dkyges-49/obj2 2024-04-12T03:46:55.245 DEBUG:boto:Data: 2024-04-12T03:46:55.245 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.245 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.245 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.246 DEBUG:boto:Params: {} 2024-04-12T03:46:55.246 DEBUG:boto:Token: None 2024-04-12T03:46:55.246 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj2 2024-04-12T03:46:55.246 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A5ctuJSSJHqf+6rgmrzA4m8UrsE= 2024-04-12T03:46:55.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A5ctuJSSJHqf+6rgmrzA4m8UrsE=', 'Content-Length': '0'} 2024-04-12T03:46:55.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079b04f4a658bd019-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:46:55.248 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:46:55.248 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:46:55.248 DEBUG:boto:Method: GET 2024-04-12T03:46:55.248 DEBUG:boto:Path: /dkyges-49/obj2 2024-04-12T03:46:55.249 DEBUG:boto:Data: 2024-04-12T03:46:55.249 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.249 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.249 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.249 DEBUG:boto:Params: {} 2024-04-12T03:46:55.249 DEBUG:boto:Token: None 2024-04-12T03:46:55.249 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj2 2024-04-12T03:46:55.249 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IgXfx110fFDLdMWvYFq2khrfOpI= 2024-04-12T03:46:55.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IgXfx110fFDLdMWvYFq2khrfOpI=', 'Content-Length': '0'} 2024-04-12T03:46:55.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c48bdabce15f0a14-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.251 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:46:55.251 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:46:55.251 DEBUG:boto:Method: GET 2024-04-12T03:46:55.251 DEBUG:boto:Path: /dkyges-49/obj2 2024-04-12T03:46:55.251 DEBUG:boto:Data: 2024-04-12T03:46:55.252 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.252 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.252 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.252 DEBUG:boto:Params: {} 2024-04-12T03:46:55.252 DEBUG:boto:Token: None 2024-04-12T03:46:55.252 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj2 2024-04-12T03:46:55.252 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IgXfx110fFDLdMWvYFq2khrfOpI= 2024-04-12T03:46:55.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IgXfx110fFDLdMWvYFq2khrfOpI=', 'Content-Length': '0'} 2024-04-12T03:46:55.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd3b7800a19c0295-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.255 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.255 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.255 DEBUG:boto:path=/dkyges-49/?prefix=obj2 2024-04-12T03:46:55.255 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj2 2024-04-12T03:46:55.255 DEBUG:boto:Method: GET 2024-04-12T03:46:55.255 DEBUG:boto:Path: /dkyges-49/?prefix=obj2 2024-04-12T03:46:55.255 DEBUG:boto:Data: 2024-04-12T03:46:55.256 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.256 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.256 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.256 DEBUG:boto:Params: {} 2024-04-12T03:46:55.256 DEBUG:boto:Token: None 2024-04-12T03:46:55.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.256 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA= 2024-04-12T03:46:55.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA=', 'Content-Length': '0'} 2024-04-12T03:46:55.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba7b931fa404b3b9-006618aeaf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.259 DEBUG:boto:b'dkyges-49obj21000falseobj22024-04-12T03:46:13.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.259 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.259 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.259 DEBUG:boto:path=/dkyges-49/?prefix=obj2 2024-04-12T03:46:55.260 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj2 2024-04-12T03:46:55.260 DEBUG:boto:Method: GET 2024-04-12T03:46:55.260 DEBUG:boto:Path: /dkyges-49/?prefix=obj2 2024-04-12T03:46:55.260 DEBUG:boto:Data: 2024-04-12T03:46:55.260 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.260 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.260 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.261 DEBUG:boto:Params: {} 2024-04-12T03:46:55.261 DEBUG:boto:Token: None 2024-04-12T03:46:55.261 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.261 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA= 2024-04-12T03:46:55.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA=', 'Content-Length': '0'} 2024-04-12T03:46:55.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095adf85b4a630206-006618aeaf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.306 DEBUG:boto:b'dkyges-49obj21000falseobj22024-04-12T03:46:13.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:46:55.307 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:46:55.307 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:46:55.307 DEBUG:boto:path=/dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.307 DEBUG:boto:auth_path=/dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.307 DEBUG:boto:Method: GET 2024-04-12T03:46:55.307 DEBUG:boto:Path: /dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.308 DEBUG:boto:Data: 2024-04-12T03:46:55.308 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.308 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.308 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.308 DEBUG:boto:Params: {} 2024-04-12T03:46:55.308 DEBUG:boto:Token: None 2024-04-12T03:46:55.308 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.308 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wWPG9goR8rcYIyE7+jPjZuCulj0= 2024-04-12T03:46:55.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wWPG9goR8rcYIyE7+jPjZuCulj0=', 'Content-Length': '0'} 2024-04-12T03:46:55.310 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005672f700d1a2488f-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.311 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:46:55.311 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:46:55.311 DEBUG:boto:path=/dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.311 DEBUG:boto:auth_path=/dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.311 DEBUG:boto:Method: GET 2024-04-12T03:46:55.311 DEBUG:boto:Path: /dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.311 DEBUG:boto:Data: 2024-04-12T03:46:55.311 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.311 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.312 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.312 DEBUG:boto:Params: {} 2024-04-12T03:46:55.312 DEBUG:boto:Token: None 2024-04-12T03:46:55.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.312 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wWPG9goR8rcYIyE7+jPjZuCulj0= 2024-04-12T03:46:55.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wWPG9goR8rcYIyE7+jPjZuCulj0=', 'Content-Length': '0'} 2024-04-12T03:46:55.314 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ecabba3249ef2d59-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.314 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:46:55.315 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:46:55.315 DEBUG:boto:path=/dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.315 DEBUG:boto:auth_path=/dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.315 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.315 DEBUG:boto:Path: /dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.315 DEBUG:boto:Data: 2024-04-12T03:46:55.315 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.315 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.315 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.315 DEBUG:boto:Params: {} 2024-04-12T03:46:55.315 DEBUG:boto:Token: None 2024-04-12T03:46:55.316 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.316 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MYpQRgIWgVOagnBPvNW1boUgI/o= 2024-04-12T03:46:55.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MYpQRgIWgVOagnBPvNW1boUgI/o=', 'Content-Length': '0'} 2024-04-12T03:46:55.318 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf39c472d63c7394-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.318 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:46:55.318 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:46:55.318 DEBUG:boto:path=/dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.319 DEBUG:boto:auth_path=/dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.319 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.319 DEBUG:boto:Path: /dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.319 DEBUG:boto:Data: 2024-04-12T03:46:55.319 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.320 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.320 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.320 DEBUG:boto:Params: {} 2024-04-12T03:46:55.321 DEBUG:boto:Token: None 2024-04-12T03:46:55.321 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.321 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MYpQRgIWgVOagnBPvNW1boUgI/o= 2024-04-12T03:46:55.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MYpQRgIWgVOagnBPvNW1boUgI/o=', 'Content-Length': '0'} 2024-04-12T03:46:55.323 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be05d87a6b7a1303-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:46:55.324 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:46:55.324 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:46:55.324 DEBUG:boto:path=/dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.325 DEBUG:boto:auth_path=/dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.325 DEBUG:boto:Method: GET 2024-04-12T03:46:55.325 DEBUG:boto:Path: /dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.325 DEBUG:boto:Data: 2024-04-12T03:46:55.325 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.326 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.326 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.326 DEBUG:boto:Params: {} 2024-04-12T03:46:55.326 DEBUG:boto:Token: None 2024-04-12T03:46:55.327 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.327 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wWPG9goR8rcYIyE7+jPjZuCulj0= 2024-04-12T03:46:55.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wWPG9goR8rcYIyE7+jPjZuCulj0=', 'Content-Length': '0'} 2024-04-12T03:46:55.329 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f636f4f74b64a83-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.370 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:46:55.370 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:46:55.371 DEBUG:boto:path=/dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.371 DEBUG:boto:auth_path=/dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.371 DEBUG:boto:Method: GET 2024-04-12T03:46:55.371 DEBUG:boto:Path: /dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.371 DEBUG:boto:Data: 2024-04-12T03:46:55.371 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.371 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.371 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.371 DEBUG:boto:Params: {} 2024-04-12T03:46:55.371 DEBUG:boto:Token: None 2024-04-12T03:46:55.372 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj3?versionId=null 2024-04-12T03:46:55.372 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wWPG9goR8rcYIyE7+jPjZuCulj0= 2024-04-12T03:46:55.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wWPG9goR8rcYIyE7+jPjZuCulj0=', 'Content-Length': '0'} 2024-04-12T03:46:55.374 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e441f899bd51c96-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.374 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.374 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.374 DEBUG:boto:path=/dkyges-49/?prefix=obj3 2024-04-12T03:46:55.374 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj3 2024-04-12T03:46:55.374 DEBUG:boto:Method: GET 2024-04-12T03:46:55.375 DEBUG:boto:Path: /dkyges-49/?prefix=obj3 2024-04-12T03:46:55.375 DEBUG:boto:Data: 2024-04-12T03:46:55.375 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.375 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.375 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.375 DEBUG:boto:Params: {} 2024-04-12T03:46:55.375 DEBUG:boto:Token: None 2024-04-12T03:46:55.375 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.375 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA= 2024-04-12T03:46:55.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA=', 'Content-Length': '0'} 2024-04-12T03:46:55.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007496ca5b5d2510c8-006618aeaf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.378 DEBUG:boto:b'dkyges-49obj31000falseobj32024-04-12T03:46:13.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.378 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.379 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.379 DEBUG:boto:path=/dkyges-49/?prefix=obj3 2024-04-12T03:46:55.379 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj3 2024-04-12T03:46:55.379 DEBUG:boto:Method: GET 2024-04-12T03:46:55.379 DEBUG:boto:Path: /dkyges-49/?prefix=obj3 2024-04-12T03:46:55.379 DEBUG:boto:Data: 2024-04-12T03:46:55.379 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.379 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.379 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.379 DEBUG:boto:Params: {} 2024-04-12T03:46:55.380 DEBUG:boto:Token: None 2024-04-12T03:46:55.380 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.380 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA= 2024-04-12T03:46:55.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA=', 'Content-Length': '0'} 2024-04-12T03:46:55.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000395e35c44abfd671-006618aeaf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.382 DEBUG:boto:b'dkyges-49obj31000falseobj32024-04-12T03:46:13.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.383 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:46:55.383 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:46:55.383 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.383 DEBUG:boto:Path: /dkyges-49/obj3 2024-04-12T03:46:55.383 DEBUG:boto:Data: 2024-04-12T03:46:55.383 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.383 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.383 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.383 DEBUG:boto:Params: {} 2024-04-12T03:46:55.383 DEBUG:boto:Token: None 2024-04-12T03:46:55.384 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj3 2024-04-12T03:46:55.384 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BT1n975LbSFuPj7BG0xPi6Dkbts= 2024-04-12T03:46:55.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BT1n975LbSFuPj7BG0xPi6Dkbts=', 'Content-Length': '0'} 2024-04-12T03:46:55.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008add49aa60abe4c4-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.386 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:46:55.386 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:46:55.386 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.386 DEBUG:boto:Path: /dkyges-49/obj3 2024-04-12T03:46:55.386 DEBUG:boto:Data: 2024-04-12T03:46:55.386 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.386 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.386 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.387 DEBUG:boto:Params: {} 2024-04-12T03:46:55.387 DEBUG:boto:Token: None 2024-04-12T03:46:55.387 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj3 2024-04-12T03:46:55.387 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BT1n975LbSFuPj7BG0xPi6Dkbts= 2024-04-12T03:46:55.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BT1n975LbSFuPj7BG0xPi6Dkbts=', 'Content-Length': '0'} 2024-04-12T03:46:55.388 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a91501a69fed6840-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:46:55.389 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:46:55.389 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:46:55.389 DEBUG:boto:Method: GET 2024-04-12T03:46:55.389 DEBUG:boto:Path: /dkyges-49/obj3 2024-04-12T03:46:55.389 DEBUG:boto:Data: 2024-04-12T03:46:55.389 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.389 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.389 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.390 DEBUG:boto:Params: {} 2024-04-12T03:46:55.390 DEBUG:boto:Token: None 2024-04-12T03:46:55.390 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj3 2024-04-12T03:46:55.390 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zmN+cBhZEM6pp6BJIomFXCiIQFs= 2024-04-12T03:46:55.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zmN+cBhZEM6pp6BJIomFXCiIQFs=', 'Content-Length': '0'} 2024-04-12T03:46:55.392 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f8bfda301438530-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.392 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:46:55.392 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:46:55.392 DEBUG:boto:Method: GET 2024-04-12T03:46:55.392 DEBUG:boto:Path: /dkyges-49/obj3 2024-04-12T03:46:55.392 DEBUG:boto:Data: 2024-04-12T03:46:55.393 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.393 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.393 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.393 DEBUG:boto:Params: {} 2024-04-12T03:46:55.393 DEBUG:boto:Token: None 2024-04-12T03:46:55.393 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj3 2024-04-12T03:46:55.393 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zmN+cBhZEM6pp6BJIomFXCiIQFs= 2024-04-12T03:46:55.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zmN+cBhZEM6pp6BJIomFXCiIQFs=', 'Content-Length': '0'} 2024-04-12T03:46:55.395 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b8a1c76827f386d-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.442 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.442 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.443 DEBUG:boto:path=/dkyges-49/?prefix=obj3 2024-04-12T03:46:55.443 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj3 2024-04-12T03:46:55.443 DEBUG:boto:Method: GET 2024-04-12T03:46:55.443 DEBUG:boto:Path: /dkyges-49/?prefix=obj3 2024-04-12T03:46:55.443 DEBUG:boto:Data: 2024-04-12T03:46:55.443 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.443 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.443 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.443 DEBUG:boto:Params: {} 2024-04-12T03:46:55.443 DEBUG:boto:Token: None 2024-04-12T03:46:55.444 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.444 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA= 2024-04-12T03:46:55.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA=', 'Content-Length': '0'} 2024-04-12T03:46:55.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ef127b2d9077ab9-006618aeaf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.446 DEBUG:boto:b'dkyges-49obj31000falseobj32024-04-12T03:46:13.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.447 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.447 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.447 DEBUG:boto:path=/dkyges-49/?prefix=obj3 2024-04-12T03:46:55.447 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj3 2024-04-12T03:46:55.447 DEBUG:boto:Method: GET 2024-04-12T03:46:55.447 DEBUG:boto:Path: /dkyges-49/?prefix=obj3 2024-04-12T03:46:55.447 DEBUG:boto:Data: 2024-04-12T03:46:55.447 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.448 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.448 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.448 DEBUG:boto:Params: {} 2024-04-12T03:46:55.448 DEBUG:boto:Token: None 2024-04-12T03:46:55.448 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.448 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA= 2024-04-12T03:46:55.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA=', 'Content-Length': '0'} 2024-04-12T03:46:55.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c021bbd44ce11cd8-006618aeaf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.451 DEBUG:boto:b'dkyges-49obj31000falseobj32024-04-12T03:46:13.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:46:55.451 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:46:55.452 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:46:55.452 DEBUG:boto:path=/dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.452 DEBUG:boto:auth_path=/dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.452 DEBUG:boto:Method: GET 2024-04-12T03:46:55.452 DEBUG:boto:Path: /dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.452 DEBUG:boto:Data: 2024-04-12T03:46:55.452 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.452 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.452 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.453 DEBUG:boto:Params: {} 2024-04-12T03:46:55.453 DEBUG:boto:Token: None 2024-04-12T03:46:55.453 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.453 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vTTe6vyHzHqvIQTyczfzXPKtseY= 2024-04-12T03:46:55.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vTTe6vyHzHqvIQTyczfzXPKtseY=', 'Content-Length': '0'} 2024-04-12T03:46:55.455 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075b0849a58e3251f-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.455 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:46:55.455 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:46:55.455 DEBUG:boto:path=/dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.456 DEBUG:boto:auth_path=/dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.456 DEBUG:boto:Method: GET 2024-04-12T03:46:55.456 DEBUG:boto:Path: /dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.456 DEBUG:boto:Data: 2024-04-12T03:46:55.456 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.456 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.456 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.456 DEBUG:boto:Params: {} 2024-04-12T03:46:55.456 DEBUG:boto:Token: None 2024-04-12T03:46:55.456 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.457 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vTTe6vyHzHqvIQTyczfzXPKtseY= 2024-04-12T03:46:55.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vTTe6vyHzHqvIQTyczfzXPKtseY=', 'Content-Length': '0'} 2024-04-12T03:46:55.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a62a04b0888e6426-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.459 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:46:55.459 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:46:55.459 DEBUG:boto:path=/dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.459 DEBUG:boto:auth_path=/dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.459 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.459 DEBUG:boto:Path: /dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.460 DEBUG:boto:Data: 2024-04-12T03:46:55.460 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.460 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.460 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.460 DEBUG:boto:Params: {} 2024-04-12T03:46:55.460 DEBUG:boto:Token: None 2024-04-12T03:46:55.460 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.460 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VB++chzchLtUS7c4Fn+neMA3DQc= 2024-04-12T03:46:55.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VB++chzchLtUS7c4Fn+neMA3DQc=', 'Content-Length': '0'} 2024-04-12T03:46:55.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000032525e37b2065974-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.462 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:46:55.462 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:46:55.462 DEBUG:boto:path=/dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.462 DEBUG:boto:auth_path=/dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.463 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.463 DEBUG:boto:Path: /dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.463 DEBUG:boto:Data: 2024-04-12T03:46:55.463 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.463 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.463 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.463 DEBUG:boto:Params: {} 2024-04-12T03:46:55.463 DEBUG:boto:Token: None 2024-04-12T03:46:55.463 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.463 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VB++chzchLtUS7c4Fn+neMA3DQc= 2024-04-12T03:46:55.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VB++chzchLtUS7c4Fn+neMA3DQc=', 'Content-Length': '0'} 2024-04-12T03:46:55.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4d1985b858fa39c-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:46:55.466 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:46:55.466 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:46:55.466 DEBUG:boto:path=/dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.466 DEBUG:boto:auth_path=/dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.466 DEBUG:boto:Method: GET 2024-04-12T03:46:55.466 DEBUG:boto:Path: /dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.466 DEBUG:boto:Data: 2024-04-12T03:46:55.466 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.466 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.466 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.467 DEBUG:boto:Params: {} 2024-04-12T03:46:55.467 DEBUG:boto:Token: None 2024-04-12T03:46:55.467 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.467 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vTTe6vyHzHqvIQTyczfzXPKtseY= 2024-04-12T03:46:55.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vTTe6vyHzHqvIQTyczfzXPKtseY=', 'Content-Length': '0'} 2024-04-12T03:46:55.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b463d7573c29abe-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.514 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:46:55.514 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:46:55.515 DEBUG:boto:path=/dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.515 DEBUG:boto:auth_path=/dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.515 DEBUG:boto:Method: GET 2024-04-12T03:46:55.515 DEBUG:boto:Path: /dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.515 DEBUG:boto:Data: 2024-04-12T03:46:55.515 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.515 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.515 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.515 DEBUG:boto:Params: {} 2024-04-12T03:46:55.515 DEBUG:boto:Token: None 2024-04-12T03:46:55.516 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj4?versionId=null 2024-04-12T03:46:55.516 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vTTe6vyHzHqvIQTyczfzXPKtseY= 2024-04-12T03:46:55.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vTTe6vyHzHqvIQTyczfzXPKtseY=', 'Content-Length': '0'} 2024-04-12T03:46:55.518 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004b8aa6f5c985793-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.518 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.518 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.518 DEBUG:boto:path=/dkyges-49/?prefix=obj4 2024-04-12T03:46:55.518 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj4 2024-04-12T03:46:55.518 DEBUG:boto:Method: GET 2024-04-12T03:46:55.519 DEBUG:boto:Path: /dkyges-49/?prefix=obj4 2024-04-12T03:46:55.519 DEBUG:boto:Data: 2024-04-12T03:46:55.519 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.519 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.519 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.519 DEBUG:boto:Params: {} 2024-04-12T03:46:55.519 DEBUG:boto:Token: None 2024-04-12T03:46:55.519 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.519 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA= 2024-04-12T03:46:55.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA=', 'Content-Length': '0'} 2024-04-12T03:46:55.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000628af80b191b0180-006618aeaf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.522 DEBUG:boto:b'dkyges-49obj41000falseobj42024-04-12T03:46:13.731Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.523 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.523 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.523 DEBUG:boto:path=/dkyges-49/?prefix=obj4 2024-04-12T03:46:55.523 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj4 2024-04-12T03:46:55.523 DEBUG:boto:Method: GET 2024-04-12T03:46:55.523 DEBUG:boto:Path: /dkyges-49/?prefix=obj4 2024-04-12T03:46:55.523 DEBUG:boto:Data: 2024-04-12T03:46:55.523 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.523 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.523 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.523 DEBUG:boto:Params: {} 2024-04-12T03:46:55.524 DEBUG:boto:Token: None 2024-04-12T03:46:55.524 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.524 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA= 2024-04-12T03:46:55.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA=', 'Content-Length': '0'} 2024-04-12T03:46:55.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039c392e546b4a563-006618aeaf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.526 DEBUG:boto:b'dkyges-49obj41000falseobj42024-04-12T03:46:13.731Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.527 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:46:55.527 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:46:55.527 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.527 DEBUG:boto:Path: /dkyges-49/obj4 2024-04-12T03:46:55.527 DEBUG:boto:Data: 2024-04-12T03:46:55.527 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.527 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.527 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.528 DEBUG:boto:Params: {} 2024-04-12T03:46:55.528 DEBUG:boto:Token: None 2024-04-12T03:46:55.528 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj4 2024-04-12T03:46:55.528 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9cuxavcHeNnw1rGBKwAnMYRFjOc= 2024-04-12T03:46:55.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9cuxavcHeNnw1rGBKwAnMYRFjOc=', 'Content-Length': '0'} 2024-04-12T03:46:55.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050d9c65acf7540d5-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.530 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:46:55.530 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:46:55.530 DEBUG:boto:Method: HEAD 2024-04-12T03:46:55.530 DEBUG:boto:Path: /dkyges-49/obj4 2024-04-12T03:46:55.530 DEBUG:boto:Data: 2024-04-12T03:46:55.530 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.530 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.530 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.531 DEBUG:boto:Params: {} 2024-04-12T03:46:55.531 DEBUG:boto:Token: None 2024-04-12T03:46:55.531 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj4 2024-04-12T03:46:55.531 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9cuxavcHeNnw1rGBKwAnMYRFjOc= 2024-04-12T03:46:55.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9cuxavcHeNnw1rGBKwAnMYRFjOc=', 'Content-Length': '0'} 2024-04-12T03:46:55.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca386cd3d4030f8c-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:46:55.533 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:46:55.533 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:46:55.533 DEBUG:boto:Method: GET 2024-04-12T03:46:55.533 DEBUG:boto:Path: /dkyges-49/obj4 2024-04-12T03:46:55.533 DEBUG:boto:Data: 2024-04-12T03:46:55.533 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.533 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.534 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.535 DEBUG:boto:Params: {} 2024-04-12T03:46:55.535 DEBUG:boto:Token: None 2024-04-12T03:46:55.536 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj4 2024-04-12T03:46:55.536 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SOtZ7c71VvWjJmHr0bf9YpuqSCU= 2024-04-12T03:46:55.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SOtZ7c71VvWjJmHr0bf9YpuqSCU=', 'Content-Length': '0'} 2024-04-12T03:46:55.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd29e1961c8f0ed2-006618aeaf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.538 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:46:55.538 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:46:55.538 DEBUG:boto:Method: GET 2024-04-12T03:46:55.539 DEBUG:boto:Path: /dkyges-49/obj4 2024-04-12T03:46:55.539 DEBUG:boto:Data: 2024-04-12T03:46:55.539 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.539 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.539 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.539 DEBUG:boto:Params: {} 2024-04-12T03:46:55.539 DEBUG:boto:Token: None 2024-04-12T03:46:55.539 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/obj4 2024-04-12T03:46:55.539 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SOtZ7c71VvWjJmHr0bf9YpuqSCU= 2024-04-12T03:46:55.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SOtZ7c71VvWjJmHr0bf9YpuqSCU=', 'Content-Length': '0'} 2024-04-12T03:46:55.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004bd1c22ac0ff3426-006618aeaf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.541 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.542 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.542 DEBUG:boto:path=/dkyges-49/?prefix=obj4 2024-04-12T03:46:55.542 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj4 2024-04-12T03:46:55.543 DEBUG:boto:Method: GET 2024-04-12T03:46:55.543 DEBUG:boto:Path: /dkyges-49/?prefix=obj4 2024-04-12T03:46:55.543 DEBUG:boto:Data: 2024-04-12T03:46:55.543 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.543 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:55.543 DEBUG:boto:Port: 8001 2024-04-12T03:46:55.543 DEBUG:boto:Params: {} 2024-04-12T03:46:55.543 DEBUG:boto:Token: None 2024-04-12T03:46:55.543 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.543 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA= 2024-04-12T03:46:55.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA=', 'Content-Length': '0'} 2024-04-12T03:46:55.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cca2b1af67f78d9f-006618aeaf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.546 DEBUG:boto:b'dkyges-49obj41000falseobj42024-04-12T03:46:13.731Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.547 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:55.547 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:55.547 DEBUG:boto:path=/dkyges-49/?prefix=obj4 2024-04-12T03:46:55.547 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj4 2024-04-12T03:46:55.547 DEBUG:boto:Method: GET 2024-04-12T03:46:55.547 DEBUG:boto:Path: /dkyges-49/?prefix=obj4 2024-04-12T03:46:55.547 DEBUG:boto:Data: 2024-04-12T03:46:55.547 DEBUG:boto:Headers: {} 2024-04-12T03:46:55.547 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:55.547 DEBUG:boto:Port: 8000 2024-04-12T03:46:55.547 DEBUG:boto:Params: {} 2024-04-12T03:46:55.548 DEBUG:boto:Token: None 2024-04-12T03:46:55.548 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:46:55 GMT /dkyges-49/ 2024-04-12T03:46:55.548 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA= 2024-04-12T03:46:55.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67apzH+RyZeTu/67PUS+JwkRPcA=', 'Content-Length': '0'} 2024-04-12T03:46:55.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f9319620683306c-006618aeaf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:46:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:55.550 DEBUG:boto:b'dkyges-49obj41000falseobj42024-04-12T03:46:13.731Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:46:55.551 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-49 zones={a2, a1} 2024-04-12T03:46:55.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dkyges-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:46:55.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dkyges-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:46:55.551 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dkyges-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:46:55.612 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:55.613 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:55.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'dkyges-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:46:55.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'dkyges-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:46:55.691 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket dkyges-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:46:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:55.747 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:55.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dkyges-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:46:55.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dkyges-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:46:55.824 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dkyges-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:46:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:55.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'dkyges-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:46:55.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'dkyges-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:46:55.958 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket dkyges-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:46:56.014 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:56.014 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:56.090 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:56.090 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:56.090 DEBUG:boto:Method: HEAD 2024-04-12T03:46:56.090 DEBUG:boto:Path: /dkyges-48/ 2024-04-12T03:46:56.090 DEBUG:boto:Data: 2024-04-12T03:46:56.090 DEBUG:boto:Headers: {} 2024-04-12T03:46:56.090 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:56.090 DEBUG:boto:Port: 8000 2024-04-12T03:46:56.091 DEBUG:boto:Params: {} 2024-04-12T03:46:56.091 DEBUG:boto:Token: None 2024-04-12T03:46:56.091 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:56 GMT /dkyges-48/ 2024-04-12T03:46:56.091 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X50QXjmDcriVKcsO2sfKo9lwfQo= 2024-04-12T03:46:56.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X50QXjmDcriVKcsO2sfKo9lwfQo=', 'Content-Length': '0'} 2024-04-12T03:46:56.093 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a42311db6448efda-006618aeb0-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:56.094 DEBUG:boto:path=/dkyges-48/obj5 2024-04-12T03:46:56.094 DEBUG:boto:auth_path=/dkyges-48/obj5 2024-04-12T03:46:56.094 DEBUG:boto:Method: PUT 2024-04-12T03:46:56.094 DEBUG:boto:Path: /dkyges-48/obj5 2024-04-12T03:46:56.094 DEBUG:boto:Data: 2024-04-12T03:46:56.094 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:46:56.094 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:56.095 DEBUG:boto:Port: 8000 2024-04-12T03:46:56.095 DEBUG:boto:Params: {} 2024-04-12T03:46:56.095 DEBUG:boto:Token: None 2024-04-12T03:46:56.095 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:46:56 GMT /dkyges-48/obj5 2024-04-12T03:46:56.095 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1I7FM1agS1MCfRATztH56f4TtZU= 2024-04-12T03:46:56.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1I7FM1agS1MCfRATztH56f4TtZU='} 2024-04-12T03:46:56.100 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000904913ea9a5320d3-006618aeb0-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:56.101 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:56.101 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:56.101 DEBUG:boto:Method: HEAD 2024-04-12T03:46:56.101 DEBUG:boto:Path: /dkyges-48/ 2024-04-12T03:46:56.101 DEBUG:boto:Data: 2024-04-12T03:46:56.101 DEBUG:boto:Headers: {} 2024-04-12T03:46:56.101 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:56.101 DEBUG:boto:Port: 8000 2024-04-12T03:46:56.101 DEBUG:boto:Params: {} 2024-04-12T03:46:56.101 DEBUG:boto:Token: None 2024-04-12T03:46:56.102 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:56 GMT /dkyges-48/ 2024-04-12T03:46:56.102 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X50QXjmDcriVKcsO2sfKo9lwfQo= 2024-04-12T03:46:56.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X50QXjmDcriVKcsO2sfKo9lwfQo=', 'Content-Length': '0'} 2024-04-12T03:46:56.104 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057c610c5b9d78bdd-006618aeb0-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:56.104 DEBUG:boto:path=/dkyges-48/obj6 2024-04-12T03:46:56.104 DEBUG:boto:auth_path=/dkyges-48/obj6 2024-04-12T03:46:56.105 DEBUG:boto:Method: PUT 2024-04-12T03:46:56.105 DEBUG:boto:Path: /dkyges-48/obj6 2024-04-12T03:46:56.105 DEBUG:boto:Data: 2024-04-12T03:46:56.105 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:46:56.105 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:56.105 DEBUG:boto:Port: 8000 2024-04-12T03:46:56.105 DEBUG:boto:Params: {} 2024-04-12T03:46:56.105 DEBUG:boto:Token: None 2024-04-12T03:46:56.105 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:46:56 GMT /dkyges-48/obj6 2024-04-12T03:46:56.105 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ha7Gmo33jHlSBEcQ2fpbB/e1YyM= 2024-04-12T03:46:56.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ha7Gmo33jHlSBEcQ2fpbB/e1YyM='} 2024-04-12T03:46:56.111 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b09b281e7263f6a8-006618aeb0-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:56.111 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:56.112 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:56.112 DEBUG:boto:Method: HEAD 2024-04-12T03:46:56.112 DEBUG:boto:Path: /dkyges-48/ 2024-04-12T03:46:56.112 DEBUG:boto:Data: 2024-04-12T03:46:56.112 DEBUG:boto:Headers: {} 2024-04-12T03:46:56.112 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:56.112 DEBUG:boto:Port: 8000 2024-04-12T03:46:56.112 DEBUG:boto:Params: {} 2024-04-12T03:46:56.112 DEBUG:boto:Token: None 2024-04-12T03:46:56.112 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:56 GMT /dkyges-48/ 2024-04-12T03:46:56.113 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X50QXjmDcriVKcsO2sfKo9lwfQo= 2024-04-12T03:46:56.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X50QXjmDcriVKcsO2sfKo9lwfQo=', 'Content-Length': '0'} 2024-04-12T03:46:56.115 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca798f5dbb83a9b3-006618aeb0-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:56.115 DEBUG:boto:path=/dkyges-48/obj7 2024-04-12T03:46:56.115 DEBUG:boto:auth_path=/dkyges-48/obj7 2024-04-12T03:46:56.115 DEBUG:boto:Method: PUT 2024-04-12T03:46:56.116 DEBUG:boto:Path: /dkyges-48/obj7 2024-04-12T03:46:56.116 DEBUG:boto:Data: 2024-04-12T03:46:56.116 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:46:56.116 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:56.116 DEBUG:boto:Port: 8000 2024-04-12T03:46:56.116 DEBUG:boto:Params: {} 2024-04-12T03:46:56.116 DEBUG:boto:Token: None 2024-04-12T03:46:56.116 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:46:56 GMT /dkyges-48/obj7 2024-04-12T03:46:56.116 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f5BAyD/6vfzWTKyigtXAGDgSPzo= 2024-04-12T03:46:56.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f5BAyD/6vfzWTKyigtXAGDgSPzo='} 2024-04-12T03:46:56.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bc6ae4e92a0c7674-006618aeb0-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:56.163 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:46:56.163 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:46:56.163 DEBUG:boto:Method: HEAD 2024-04-12T03:46:56.163 DEBUG:boto:Path: /dkyges-48/ 2024-04-12T03:46:56.163 DEBUG:boto:Data: 2024-04-12T03:46:56.163 DEBUG:boto:Headers: {} 2024-04-12T03:46:56.163 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:56.163 DEBUG:boto:Port: 8000 2024-04-12T03:46:56.163 DEBUG:boto:Params: {} 2024-04-12T03:46:56.163 DEBUG:boto:Token: None 2024-04-12T03:46:56.164 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:56 GMT /dkyges-48/ 2024-04-12T03:46:56.164 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X50QXjmDcriVKcsO2sfKo9lwfQo= 2024-04-12T03:46:56.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X50QXjmDcriVKcsO2sfKo9lwfQo=', 'Content-Length': '0'} 2024-04-12T03:46:56.166 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000107f8fcc0b75478a-006618aeb0-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:56.166 DEBUG:boto:path=/dkyges-48/obj8 2024-04-12T03:46:56.166 DEBUG:boto:auth_path=/dkyges-48/obj8 2024-04-12T03:46:56.166 DEBUG:boto:Method: PUT 2024-04-12T03:46:56.167 DEBUG:boto:Path: /dkyges-48/obj8 2024-04-12T03:46:56.167 DEBUG:boto:Data: 2024-04-12T03:46:56.167 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:46:56.167 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:46:56.167 DEBUG:boto:Port: 8000 2024-04-12T03:46:56.167 DEBUG:boto:Params: {} 2024-04-12T03:46:56.167 DEBUG:boto:Token: None 2024-04-12T03:46:56.167 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:46:56 GMT /dkyges-48/obj8 2024-04-12T03:46:56.167 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MSe8UnPVaZwyOIrPPxw2fI9ILyw= 2024-04-12T03:46:56.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MSe8UnPVaZwyOIrPPxw2fI9ILyw='} 2024-04-12T03:46:56.173 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001bf8084d01f22b98-006618aeb0-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:56.174 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:56.174 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:56.174 DEBUG:boto:Method: HEAD 2024-04-12T03:46:56.174 DEBUG:boto:Path: /dkyges-49/ 2024-04-12T03:46:56.174 DEBUG:boto:Data: 2024-04-12T03:46:56.174 DEBUG:boto:Headers: {} 2024-04-12T03:46:56.174 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:56.174 DEBUG:boto:Port: 8001 2024-04-12T03:46:56.174 DEBUG:boto:Params: {} 2024-04-12T03:46:56.175 DEBUG:boto:Token: None 2024-04-12T03:46:56.175 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:56 GMT /dkyges-49/ 2024-04-12T03:46:56.175 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HQIJ7qgYyuqY8hHbkrL6oneYBRY= 2024-04-12T03:46:56.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HQIJ7qgYyuqY8hHbkrL6oneYBRY=', 'Content-Length': '0'} 2024-04-12T03:46:56.177 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b20d9f0e706129c-006618aeb0-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:56.178 DEBUG:boto:path=/dkyges-49/obj5 2024-04-12T03:46:56.178 DEBUG:boto:auth_path=/dkyges-49/obj5 2024-04-12T03:46:56.178 DEBUG:boto:Method: PUT 2024-04-12T03:46:56.178 DEBUG:boto:Path: /dkyges-49/obj5 2024-04-12T03:46:56.178 DEBUG:boto:Data: 2024-04-12T03:46:56.178 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:46:56.178 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:56.178 DEBUG:boto:Port: 8001 2024-04-12T03:46:56.179 DEBUG:boto:Params: {} 2024-04-12T03:46:56.179 DEBUG:boto:Token: None 2024-04-12T03:46:56.179 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:46:56 GMT /dkyges-49/obj5 2024-04-12T03:46:56.179 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9I8+F82HIsrg6NHZbpST4SfvaM4= 2024-04-12T03:46:56.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9I8+F82HIsrg6NHZbpST4SfvaM4='} 2024-04-12T03:46:56.185 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001af07aaad84953b6-006618aeb0-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:56.185 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:56.185 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:56.185 DEBUG:boto:Method: HEAD 2024-04-12T03:46:56.185 DEBUG:boto:Path: /dkyges-49/ 2024-04-12T03:46:56.185 DEBUG:boto:Data: 2024-04-12T03:46:56.185 DEBUG:boto:Headers: {} 2024-04-12T03:46:56.186 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:56.186 DEBUG:boto:Port: 8001 2024-04-12T03:46:56.186 DEBUG:boto:Params: {} 2024-04-12T03:46:56.186 DEBUG:boto:Token: None 2024-04-12T03:46:56.186 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:56 GMT /dkyges-49/ 2024-04-12T03:46:56.186 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HQIJ7qgYyuqY8hHbkrL6oneYBRY= 2024-04-12T03:46:56.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HQIJ7qgYyuqY8hHbkrL6oneYBRY=', 'Content-Length': '0'} 2024-04-12T03:46:56.188 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b27fdac505ef6de-006618aeb0-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:56.188 DEBUG:boto:path=/dkyges-49/obj6 2024-04-12T03:46:56.189 DEBUG:boto:auth_path=/dkyges-49/obj6 2024-04-12T03:46:56.189 DEBUG:boto:Method: PUT 2024-04-12T03:46:56.189 DEBUG:boto:Path: /dkyges-49/obj6 2024-04-12T03:46:56.189 DEBUG:boto:Data: 2024-04-12T03:46:56.189 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:46:56.189 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:56.189 DEBUG:boto:Port: 8001 2024-04-12T03:46:56.189 DEBUG:boto:Params: {} 2024-04-12T03:46:56.189 DEBUG:boto:Token: None 2024-04-12T03:46:56.189 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:46:56 GMT /dkyges-49/obj6 2024-04-12T03:46:56.190 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TETM1ZC3AbfEw0nYtRkqnrAmFj0= 2024-04-12T03:46:56.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TETM1ZC3AbfEw0nYtRkqnrAmFj0='} 2024-04-12T03:46:56.195 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008965f6abe4b5aa32-006618aeb0-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:56.196 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:56.196 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:56.196 DEBUG:boto:Method: HEAD 2024-04-12T03:46:56.196 DEBUG:boto:Path: /dkyges-49/ 2024-04-12T03:46:56.196 DEBUG:boto:Data: 2024-04-12T03:46:56.196 DEBUG:boto:Headers: {} 2024-04-12T03:46:56.196 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:56.196 DEBUG:boto:Port: 8001 2024-04-12T03:46:56.196 DEBUG:boto:Params: {} 2024-04-12T03:46:56.196 DEBUG:boto:Token: None 2024-04-12T03:46:56.196 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:56 GMT /dkyges-49/ 2024-04-12T03:46:56.197 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HQIJ7qgYyuqY8hHbkrL6oneYBRY= 2024-04-12T03:46:56.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HQIJ7qgYyuqY8hHbkrL6oneYBRY=', 'Content-Length': '0'} 2024-04-12T03:46:56.199 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000972ab7860e450555-006618aeb0-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:56.199 DEBUG:boto:path=/dkyges-49/obj7 2024-04-12T03:46:56.199 DEBUG:boto:auth_path=/dkyges-49/obj7 2024-04-12T03:46:56.199 DEBUG:boto:Method: PUT 2024-04-12T03:46:56.199 DEBUG:boto:Path: /dkyges-49/obj7 2024-04-12T03:46:56.199 DEBUG:boto:Data: 2024-04-12T03:46:56.199 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:46:56.200 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:56.200 DEBUG:boto:Port: 8001 2024-04-12T03:46:56.200 DEBUG:boto:Params: {} 2024-04-12T03:46:56.200 DEBUG:boto:Token: None 2024-04-12T03:46:56.200 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:46:56 GMT /dkyges-49/obj7 2024-04-12T03:46:56.200 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VCb3b/xF8hlOaM5h+QmrE9a06QY= 2024-04-12T03:46:56.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VCb3b/xF8hlOaM5h+QmrE9a06QY='} 2024-04-12T03:46:56.242 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ccdbbc2f28d3d534-006618aeb0-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:56.243 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:46:56.243 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:46:56.243 DEBUG:boto:Method: HEAD 2024-04-12T03:46:56.243 DEBUG:boto:Path: /dkyges-49/ 2024-04-12T03:46:56.243 DEBUG:boto:Data: 2024-04-12T03:46:56.243 DEBUG:boto:Headers: {} 2024-04-12T03:46:56.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:56.243 DEBUG:boto:Port: 8001 2024-04-12T03:46:56.244 DEBUG:boto:Params: {} 2024-04-12T03:46:56.244 DEBUG:boto:Token: None 2024-04-12T03:46:56.244 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:46:56 GMT /dkyges-49/ 2024-04-12T03:46:56.244 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HQIJ7qgYyuqY8hHbkrL6oneYBRY= 2024-04-12T03:46:56.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:46:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HQIJ7qgYyuqY8hHbkrL6oneYBRY=', 'Content-Length': '0'} 2024-04-12T03:46:56.246 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3a5b18457bf3553-006618aeb0-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:56.247 DEBUG:boto:path=/dkyges-49/obj8 2024-04-12T03:46:56.247 DEBUG:boto:auth_path=/dkyges-49/obj8 2024-04-12T03:46:56.247 DEBUG:boto:Method: PUT 2024-04-12T03:46:56.247 DEBUG:boto:Path: /dkyges-49/obj8 2024-04-12T03:46:56.247 DEBUG:boto:Data: 2024-04-12T03:46:56.247 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:46:56.247 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:46:56.247 DEBUG:boto:Port: 8001 2024-04-12T03:46:56.247 DEBUG:boto:Params: {} 2024-04-12T03:46:56.247 DEBUG:boto:Token: None 2024-04-12T03:46:56.248 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:46:56 GMT /dkyges-49/obj8 2024-04-12T03:46:56.248 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tP3AD5TDuPdPICMnJJSvVyubrIs= 2024-04-12T03:46:56.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:46:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tP3AD5TDuPdPICMnJJSvVyubrIs='} 2024-04-12T03:46:56.255 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001127764626ee557a-006618aeb0-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:46:56.255 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-48 2024-04-12T03:46:56.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:46:56.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:46:56.255 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:46:56.322 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:46:56.323 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:46:56.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:46:56.366+0000 7fac61467780 1 waiting for incremental sync to catch up: 2024-04-12T03:46:56.365 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.22.6, , 00000000001.37.6, , 00000000001.41.6, , , 00000000001.21.6, , , ] 2024-04-12T03:46:56.365 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000004.28.6, 00000000002.42.1, 00000000004.45.6, 00000000002.17.1, 00000000004.45.6, 00000000002.24.1, 00000000002.20.1, 00000000004.25.6, 00000000002.38.1, 00000000002.26.1, 00000000002.13.1] 2024-04-12T03:47:01.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:01.368+0000 7fac61467780 1 waiting for incremental sync to catch up: 2024-04-12T03:47:01.366 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.22.6, , 00000000001.37.6, , 00000000001.41.6, , , 00000000001.21.6, , , ] 2024-04-12T03:47:01.367 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000004.28.6, 00000000002.42.1, 00000000004.45.6, 00000000002.17.1, 00000000004.45.6, 00000000002.24.1, 00000000002.20.1, 00000000004.25.6, 00000000002.38.1, 00000000002.26.1, 00000000002.13.1] 2024-04-12T03:47:06.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:06.369+0000 7fac61467780 1 waiting for incremental sync to catch up: 2024-04-12T03:47:06.368 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.22.6, , 00000000001.37.6, , 00000000001.41.6, , , 00000000001.21.6, , , ] 2024-04-12T03:47:06.368 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000004.28.6, 00000000002.42.1, 00000000004.45.6, 00000000002.17.1, 00000000004.45.6, 00000000002.24.1, 00000000002.20.1, 00000000004.25.6, 00000000002.38.1, 00000000002.26.1, 00000000002.13.1] 2024-04-12T03:47:11.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:11.370+0000 7fac61467780 1 bucket sync caught up with source: 2024-04-12T03:47:11.369 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000004.28.6, 00000000002.42.1, 00000000004.45.6, 00000000002.17.1, 00000000004.45.6, 00000000002.24.1, 00000000002.20.1, 00000000004.25.6, 00000000002.38.1, 00000000002.26.1, 00000000002.13.1] 2024-04-12T03:47:11.369 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000004.28.6, 00000000002.42.1, 00000000004.45.6, 00000000002.17.1, 00000000004.45.6, 00000000002.24.1, 00000000002.20.1, 00000000004.25.6, 00000000002.38.1, 00000000002.26.1, 00000000002.13.1] 2024-04-12T03:47:11.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:11.370+0000 7fac61467780 0 bucket checkpoint complete 2024-04-12T03:47:11.390 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-48 2024-04-12T03:47:11.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:11.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:11.391 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:47:11.460 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:11.460 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:11.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:11.504+0000 7f69aaf0e780 1 waiting for incremental sync to catch up: 2024-04-12T03:47:11.502 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.1861.6, , 00000000001.3585.6, , 00000000001.2429.6, , , 00000000001.2521.6, , , ] 2024-04-12T03:47:11.502 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000002.1863.6, , 00000000002.3589.6, , 00000000002.2431.6, , , 00000000002.2523.6, , , ] 2024-04-12T03:47:16.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:16.505+0000 7f69aaf0e780 1 waiting for incremental sync to catch up: 2024-04-12T03:47:16.503 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.1861.6, , 00000000002.3589.6, , 00000000001.2429.6, , , 00000000002.2523.6, , , ] 2024-04-12T03:47:16.503 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000002.1863.6, , 00000000002.3589.6, , 00000000002.2431.6, , , 00000000002.2523.6, , , ] 2024-04-12T03:47:21.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:21.506+0000 7f69aaf0e780 1 waiting for incremental sync to catch up: 2024-04-12T03:47:21.504 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.1861.6, , 00000000002.3589.6, , 00000000001.2429.6, , , 00000000002.2523.6, , , ] 2024-04-12T03:47:21.505 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000002.1863.6, , 00000000002.3589.6, , 00000000002.2431.6, , , 00000000002.2523.6, , , ] 2024-04-12T03:47:26.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:26.508+0000 7f69aaf0e780 1 bucket sync caught up with source: 2024-04-12T03:47:26.506 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000002.1863.6, , 00000000002.3589.6, , 00000000002.2431.6, , , 00000000002.2523.6, , , ] 2024-04-12T03:47:26.506 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000002.1863.6, , 00000000002.3589.6, , 00000000002.2431.6, , , 00000000002.2523.6, , , ] 2024-04-12T03:47:26.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:26.508+0000 7f69aaf0e780 0 bucket checkpoint complete 2024-04-12T03:47:26.523 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-48 zones={a2, a1} 2024-04-12T03:47:26.523 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.523 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.524 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.524 DEBUG:boto:Path: /dkyges-48/ 2024-04-12T03:47:26.524 DEBUG:boto:Data: 2024-04-12T03:47:26.524 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.524 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.524 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.524 DEBUG:boto:Params: {} 2024-04-12T03:47:26.524 DEBUG:boto:Token: None 2024-04-12T03:47:26.524 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/ 2024-04-12T03:47:26.525 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aLJ3AD8g2ffxf7JFUpcOu6i0dFc= 2024-04-12T03:47:26.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aLJ3AD8g2ffxf7JFUpcOu6i0dFc=', 'Content-Length': '0'} 2024-04-12T03:47:26.527 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000165bdd72f8e034d3-006618aece-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.527 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.528 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.528 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.528 DEBUG:boto:Path: /dkyges-48/ 2024-04-12T03:47:26.528 DEBUG:boto:Data: 2024-04-12T03:47:26.528 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.528 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.528 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.528 DEBUG:boto:Params: {} 2024-04-12T03:47:26.528 DEBUG:boto:Token: None 2024-04-12T03:47:26.528 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/ 2024-04-12T03:47:26.529 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aLJ3AD8g2ffxf7JFUpcOu6i0dFc= 2024-04-12T03:47:26.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aLJ3AD8g2ffxf7JFUpcOu6i0dFc=', 'Content-Length': '0'} 2024-04-12T03:47:26.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1d0955182b797d2-006618aece-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.531 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:47:26.531 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.531 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.531 DEBUG:boto:path=/dkyges-48/?versions&max-keys=999 2024-04-12T03:47:26.531 DEBUG:boto:auth_path=/dkyges-48/?versions&max-keys=999 2024-04-12T03:47:26.531 DEBUG:boto:Method: GET 2024-04-12T03:47:26.532 DEBUG:boto:Path: /dkyges-48/?versions&max-keys=999 2024-04-12T03:47:26.532 DEBUG:boto:Data: 2024-04-12T03:47:26.532 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.532 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.532 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.532 DEBUG:boto:Params: {} 2024-04-12T03:47:26.532 DEBUG:boto:Token: None 2024-04-12T03:47:26.532 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/?versions 2024-04-12T03:47:26.532 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QcksrVvsCpGOymhR5aS61OaNf40= 2024-04-12T03:47:26.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QcksrVvsCpGOymhR5aS61OaNf40=', 'Content-Length': '0'} 2024-04-12T03:47:26.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000352e4e58117a6748-006618aece-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.535 DEBUG:boto:b'dkyges-48999falseobj1nulltrue2024-04-12T03:46:13.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:46:13.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:46:13.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:46:13.631Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:46:56.099Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:46:56.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:46:56.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:46:56.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:47:26.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:47:26.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:47:26.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:47:26.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-12T03:47:26.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-12T03:47:26.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-12T03:47:26.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-12T03:47:26.537 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:47:26.537 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.537 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.537 DEBUG:boto:path=/dkyges-48/?versions&max-keys=999 2024-04-12T03:47:26.537 DEBUG:boto:auth_path=/dkyges-48/?versions&max-keys=999 2024-04-12T03:47:26.537 DEBUG:boto:Method: GET 2024-04-12T03:47:26.538 DEBUG:boto:Path: /dkyges-48/?versions&max-keys=999 2024-04-12T03:47:26.538 DEBUG:boto:Data: 2024-04-12T03:47:26.538 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.538 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.538 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.538 DEBUG:boto:Params: {} 2024-04-12T03:47:26.538 DEBUG:boto:Token: None 2024-04-12T03:47:26.538 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/?versions 2024-04-12T03:47:26.539 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QcksrVvsCpGOymhR5aS61OaNf40= 2024-04-12T03:47:26.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QcksrVvsCpGOymhR5aS61OaNf40=', 'Content-Length': '0'} 2024-04-12T03:47:26.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a62af21e026cb9b-006618aece-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.541 DEBUG:boto:b'dkyges-48999falseobj1nulltrue2024-04-12T03:46:13.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:46:13.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:46:13.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:46:13.631Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:46:56.099Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:46:56.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:46:56.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:46:56.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:47:26.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:47:26.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:47:26.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:47:26.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-12T03:47:26.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-12T03:47:26.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-12T03:47:26.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-12T03:47:26.543 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.543 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.543 DEBUG:boto:path=/dkyges-48/?versions&max-keys=999 2024-04-12T03:47:26.543 DEBUG:boto:auth_path=/dkyges-48/?versions&max-keys=999 2024-04-12T03:47:26.543 DEBUG:boto:Method: GET 2024-04-12T03:47:26.543 DEBUG:boto:Path: /dkyges-48/?versions&max-keys=999 2024-04-12T03:47:26.543 DEBUG:boto:Data: 2024-04-12T03:47:26.543 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.544 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.544 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.544 DEBUG:boto:Params: {} 2024-04-12T03:47:26.544 DEBUG:boto:Token: None 2024-04-12T03:47:26.544 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/?versions 2024-04-12T03:47:26.544 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QcksrVvsCpGOymhR5aS61OaNf40= 2024-04-12T03:47:26.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QcksrVvsCpGOymhR5aS61OaNf40=', 'Content-Length': '0'} 2024-04-12T03:47:26.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000959dbd4ff77b0e22-006618aece-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.547 DEBUG:boto:b'dkyges-48999falseobj1nulltrue2024-04-12T03:46:13.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:46:13.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:46:13.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:46:13.631Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:46:56.099Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:46:56.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:46:56.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:46:56.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.547 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.548 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.548 DEBUG:boto:path=/dkyges-48/?versions&max-keys=999 2024-04-12T03:47:26.548 DEBUG:boto:auth_path=/dkyges-48/?versions&max-keys=999 2024-04-12T03:47:26.548 DEBUG:boto:Method: GET 2024-04-12T03:47:26.548 DEBUG:boto:Path: /dkyges-48/?versions&max-keys=999 2024-04-12T03:47:26.548 DEBUG:boto:Data: 2024-04-12T03:47:26.548 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.548 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.548 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.548 DEBUG:boto:Params: {} 2024-04-12T03:47:26.549 DEBUG:boto:Token: None 2024-04-12T03:47:26.549 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/?versions 2024-04-12T03:47:26.549 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QcksrVvsCpGOymhR5aS61OaNf40= 2024-04-12T03:47:26.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QcksrVvsCpGOymhR5aS61OaNf40=', 'Content-Length': '0'} 2024-04-12T03:47:26.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fc598bb4c467af0-006618aece-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.552 DEBUG:boto:b'dkyges-48999falseobj1nulltrue2024-04-12T03:46:13.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:46:13.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:46:13.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:46:13.631Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:46:56.099Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:46:56.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:46:56.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:46:56.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:47:26.553 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:47:26.553 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:47:26.553 DEBUG:boto:path=/dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.553 DEBUG:boto:auth_path=/dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.553 DEBUG:boto:Method: GET 2024-04-12T03:47:26.553 DEBUG:boto:Path: /dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.553 DEBUG:boto:Data: 2024-04-12T03:47:26.553 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.553 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.554 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.554 DEBUG:boto:Params: {} 2024-04-12T03:47:26.554 DEBUG:boto:Token: None 2024-04-12T03:47:26.554 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.554 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0kskrpuCheo2BlgCjX1V7YCpmpo= 2024-04-12T03:47:26.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0kskrpuCheo2BlgCjX1V7YCpmpo=', 'Content-Length': '0'} 2024-04-12T03:47:26.556 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000008cf9be9c8cbd8ae-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.602 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:47:26.602 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:47:26.603 DEBUG:boto:path=/dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.603 DEBUG:boto:auth_path=/dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.603 DEBUG:boto:Method: GET 2024-04-12T03:47:26.603 DEBUG:boto:Path: /dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.603 DEBUG:boto:Data: 2024-04-12T03:47:26.603 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.603 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.603 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.603 DEBUG:boto:Params: {} 2024-04-12T03:47:26.603 DEBUG:boto:Token: None 2024-04-12T03:47:26.604 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0kskrpuCheo2BlgCjX1V7YCpmpo= 2024-04-12T03:47:26.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0kskrpuCheo2BlgCjX1V7YCpmpo=', 'Content-Length': '0'} 2024-04-12T03:47:26.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003934239137467a88-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.606 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:47:26.606 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:47:26.606 DEBUG:boto:path=/dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.606 DEBUG:boto:auth_path=/dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.606 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.607 DEBUG:boto:Path: /dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.607 DEBUG:boto:Data: 2024-04-12T03:47:26.607 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.607 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.607 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.607 DEBUG:boto:Params: {} 2024-04-12T03:47:26.607 DEBUG:boto:Token: None 2024-04-12T03:47:26.607 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.607 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rzqkbw9mWHHXhj23y11C97p5iPY= 2024-04-12T03:47:26.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rzqkbw9mWHHXhj23y11C97p5iPY=', 'Content-Length': '0'} 2024-04-12T03:47:26.609 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000898637056ef610b2-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.609 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:47:26.609 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:47:26.609 DEBUG:boto:path=/dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.609 DEBUG:boto:auth_path=/dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.610 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.610 DEBUG:boto:Path: /dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.610 DEBUG:boto:Data: 2024-04-12T03:47:26.610 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.610 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.610 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.610 DEBUG:boto:Params: {} 2024-04-12T03:47:26.610 DEBUG:boto:Token: None 2024-04-12T03:47:26.610 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.611 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rzqkbw9mWHHXhj23y11C97p5iPY= 2024-04-12T03:47:26.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rzqkbw9mWHHXhj23y11C97p5iPY=', 'Content-Length': '0'} 2024-04-12T03:47:26.612 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b357dafa4c76410f-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:47:26.613 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:47:26.613 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:47:26.613 DEBUG:boto:path=/dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.613 DEBUG:boto:auth_path=/dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.613 DEBUG:boto:Method: GET 2024-04-12T03:47:26.613 DEBUG:boto:Path: /dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.613 DEBUG:boto:Data: 2024-04-12T03:47:26.613 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.613 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.613 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.613 DEBUG:boto:Params: {} 2024-04-12T03:47:26.614 DEBUG:boto:Token: None 2024-04-12T03:47:26.614 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.614 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0kskrpuCheo2BlgCjX1V7YCpmpo= 2024-04-12T03:47:26.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0kskrpuCheo2BlgCjX1V7YCpmpo=', 'Content-Length': '0'} 2024-04-12T03:47:26.616 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f1ce4d9c77823c4-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.616 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:47:26.616 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:47:26.616 DEBUG:boto:path=/dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.616 DEBUG:boto:auth_path=/dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.616 DEBUG:boto:Method: GET 2024-04-12T03:47:26.616 DEBUG:boto:Path: /dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.616 DEBUG:boto:Data: 2024-04-12T03:47:26.616 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.617 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.617 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.617 DEBUG:boto:Params: {} 2024-04-12T03:47:26.617 DEBUG:boto:Token: None 2024-04-12T03:47:26.617 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj1?versionId=null 2024-04-12T03:47:26.617 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0kskrpuCheo2BlgCjX1V7YCpmpo= 2024-04-12T03:47:26.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0kskrpuCheo2BlgCjX1V7YCpmpo=', 'Content-Length': '0'} 2024-04-12T03:47:26.619 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af31d382a3a5ac1d-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.619 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.619 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.619 DEBUG:boto:path=/dkyges-48/?prefix=obj1 2024-04-12T03:47:26.620 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj1 2024-04-12T03:47:26.620 DEBUG:boto:Method: GET 2024-04-12T03:47:26.620 DEBUG:boto:Path: /dkyges-48/?prefix=obj1 2024-04-12T03:47:26.620 DEBUG:boto:Data: 2024-04-12T03:47:26.620 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.620 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.620 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.620 DEBUG:boto:Params: {} 2024-04-12T03:47:26.620 DEBUG:boto:Token: None 2024-04-12T03:47:26.620 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/ 2024-04-12T03:47:26.620 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4= 2024-04-12T03:47:26.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4=', 'Content-Length': '0'} 2024-04-12T03:47:26.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002909f82276dc0edd-006618aece-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.623 DEBUG:boto:b'dkyges-48obj11000falseobj12024-04-12T03:46:13.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.624 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.624 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.624 DEBUG:boto:path=/dkyges-48/?prefix=obj1 2024-04-12T03:47:26.624 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj1 2024-04-12T03:47:26.624 DEBUG:boto:Method: GET 2024-04-12T03:47:26.624 DEBUG:boto:Path: /dkyges-48/?prefix=obj1 2024-04-12T03:47:26.624 DEBUG:boto:Data: 2024-04-12T03:47:26.624 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.624 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.624 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.625 DEBUG:boto:Params: {} 2024-04-12T03:47:26.625 DEBUG:boto:Token: None 2024-04-12T03:47:26.625 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/ 2024-04-12T03:47:26.625 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4= 2024-04-12T03:47:26.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4=', 'Content-Length': '0'} 2024-04-12T03:47:26.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fca94967298303b8-006618aece-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.670 DEBUG:boto:b'dkyges-48obj11000falseobj12024-04-12T03:46:13.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.671 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:47:26.671 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:47:26.671 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.671 DEBUG:boto:Path: /dkyges-48/obj1 2024-04-12T03:47:26.671 DEBUG:boto:Data: 2024-04-12T03:47:26.671 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.671 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.672 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.672 DEBUG:boto:Params: {} 2024-04-12T03:47:26.672 DEBUG:boto:Token: None 2024-04-12T03:47:26.672 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj1 2024-04-12T03:47:26.672 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WgJlUrPvI+7kzqyoQpnKI/Qtry4= 2024-04-12T03:47:26.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WgJlUrPvI+7kzqyoQpnKI/Qtry4=', 'Content-Length': '0'} 2024-04-12T03:47:26.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f79dbdd4cbc0aeb2-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.674 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:47:26.675 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:47:26.675 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.675 DEBUG:boto:Path: /dkyges-48/obj1 2024-04-12T03:47:26.675 DEBUG:boto:Data: 2024-04-12T03:47:26.675 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.675 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.675 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.675 DEBUG:boto:Params: {} 2024-04-12T03:47:26.675 DEBUG:boto:Token: None 2024-04-12T03:47:26.675 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj1 2024-04-12T03:47:26.676 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WgJlUrPvI+7kzqyoQpnKI/Qtry4= 2024-04-12T03:47:26.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WgJlUrPvI+7kzqyoQpnKI/Qtry4=', 'Content-Length': '0'} 2024-04-12T03:47:26.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a76879725a5b29db-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:47:26.678 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:47:26.678 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:47:26.678 DEBUG:boto:Method: GET 2024-04-12T03:47:26.678 DEBUG:boto:Path: /dkyges-48/obj1 2024-04-12T03:47:26.678 DEBUG:boto:Data: 2024-04-12T03:47:26.678 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.678 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.679 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.679 DEBUG:boto:Params: {} 2024-04-12T03:47:26.679 DEBUG:boto:Token: None 2024-04-12T03:47:26.679 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj1 2024-04-12T03:47:26.679 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2BVXqVkS3IWK148EtN/C8lKhyDA= 2024-04-12T03:47:26.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2BVXqVkS3IWK148EtN/C8lKhyDA=', 'Content-Length': '0'} 2024-04-12T03:47:26.681 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003e7655e09c2acd3-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.681 DEBUG:boto:path=/dkyges-48/obj1 2024-04-12T03:47:26.681 DEBUG:boto:auth_path=/dkyges-48/obj1 2024-04-12T03:47:26.681 DEBUG:boto:Method: GET 2024-04-12T03:47:26.682 DEBUG:boto:Path: /dkyges-48/obj1 2024-04-12T03:47:26.682 DEBUG:boto:Data: 2024-04-12T03:47:26.682 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.682 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.682 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.682 DEBUG:boto:Params: {} 2024-04-12T03:47:26.682 DEBUG:boto:Token: None 2024-04-12T03:47:26.682 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj1 2024-04-12T03:47:26.682 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2BVXqVkS3IWK148EtN/C8lKhyDA= 2024-04-12T03:47:26.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2BVXqVkS3IWK148EtN/C8lKhyDA=', 'Content-Length': '0'} 2024-04-12T03:47:26.684 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3a2871a1821278e-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.684 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.685 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.685 DEBUG:boto:path=/dkyges-48/?prefix=obj1 2024-04-12T03:47:26.685 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj1 2024-04-12T03:47:26.685 DEBUG:boto:Method: GET 2024-04-12T03:47:26.685 DEBUG:boto:Path: /dkyges-48/?prefix=obj1 2024-04-12T03:47:26.685 DEBUG:boto:Data: 2024-04-12T03:47:26.685 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.685 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.685 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.685 DEBUG:boto:Params: {} 2024-04-12T03:47:26.685 DEBUG:boto:Token: None 2024-04-12T03:47:26.686 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/ 2024-04-12T03:47:26.686 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4= 2024-04-12T03:47:26.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4=', 'Content-Length': '0'} 2024-04-12T03:47:26.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f585ac6e5fec662-006618aece-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.689 DEBUG:boto:b'dkyges-48obj11000falseobj12024-04-12T03:46:13.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.689 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.689 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.689 DEBUG:boto:path=/dkyges-48/?prefix=obj1 2024-04-12T03:47:26.689 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj1 2024-04-12T03:47:26.689 DEBUG:boto:Method: GET 2024-04-12T03:47:26.689 DEBUG:boto:Path: /dkyges-48/?prefix=obj1 2024-04-12T03:47:26.689 DEBUG:boto:Data: 2024-04-12T03:47:26.690 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.690 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.690 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.690 DEBUG:boto:Params: {} 2024-04-12T03:47:26.690 DEBUG:boto:Token: None 2024-04-12T03:47:26.690 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/ 2024-04-12T03:47:26.690 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4= 2024-04-12T03:47:26.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4=', 'Content-Length': '0'} 2024-04-12T03:47:26.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fffe81953be4ce6f-006618aece-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.693 DEBUG:boto:b'dkyges-48obj11000falseobj12024-04-12T03:46:13.560Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:47:26.693 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:47:26.693 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:47:26.693 DEBUG:boto:path=/dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.693 DEBUG:boto:auth_path=/dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.694 DEBUG:boto:Method: GET 2024-04-12T03:47:26.694 DEBUG:boto:Path: /dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.694 DEBUG:boto:Data: 2024-04-12T03:47:26.694 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.694 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.694 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.694 DEBUG:boto:Params: {} 2024-04-12T03:47:26.694 DEBUG:boto:Token: None 2024-04-12T03:47:26.694 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.695 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e4I/HON611T1SsMaoEb2GhTjyM4= 2024-04-12T03:47:26.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e4I/HON611T1SsMaoEb2GhTjyM4=', 'Content-Length': '0'} 2024-04-12T03:47:26.696 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b385f2e9701a2b1-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.697 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:47:26.697 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:47:26.697 DEBUG:boto:path=/dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.697 DEBUG:boto:auth_path=/dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.697 DEBUG:boto:Method: GET 2024-04-12T03:47:26.697 DEBUG:boto:Path: /dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.697 DEBUG:boto:Data: 2024-04-12T03:47:26.697 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.697 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.697 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.697 DEBUG:boto:Params: {} 2024-04-12T03:47:26.698 DEBUG:boto:Token: None 2024-04-12T03:47:26.698 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.698 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e4I/HON611T1SsMaoEb2GhTjyM4= 2024-04-12T03:47:26.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e4I/HON611T1SsMaoEb2GhTjyM4=', 'Content-Length': '0'} 2024-04-12T03:47:26.700 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9adcad7bb0c4826-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.700 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:47:26.700 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:47:26.700 DEBUG:boto:path=/dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.701 DEBUG:boto:auth_path=/dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.701 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.701 DEBUG:boto:Path: /dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.701 DEBUG:boto:Data: 2024-04-12T03:47:26.701 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.701 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.701 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.701 DEBUG:boto:Params: {} 2024-04-12T03:47:26.701 DEBUG:boto:Token: None 2024-04-12T03:47:26.701 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.702 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zuWN2S+wkMa9JCVoINRdljNP0us= 2024-04-12T03:47:26.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zuWN2S+wkMa9JCVoINRdljNP0us=', 'Content-Length': '0'} 2024-04-12T03:47:26.703 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b10dfff0d8071177-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.703 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:47:26.704 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:47:26.704 DEBUG:boto:path=/dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.704 DEBUG:boto:auth_path=/dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.704 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.704 DEBUG:boto:Path: /dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.704 DEBUG:boto:Data: 2024-04-12T03:47:26.704 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.704 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.704 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.704 DEBUG:boto:Params: {} 2024-04-12T03:47:26.705 DEBUG:boto:Token: None 2024-04-12T03:47:26.705 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.705 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zuWN2S+wkMa9JCVoINRdljNP0us= 2024-04-12T03:47:26.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zuWN2S+wkMa9JCVoINRdljNP0us=', 'Content-Length': '0'} 2024-04-12T03:47:26.707 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b588f1468d0f9946-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:47:26.707 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:47:26.707 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:47:26.707 DEBUG:boto:path=/dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.707 DEBUG:boto:auth_path=/dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.707 DEBUG:boto:Method: GET 2024-04-12T03:47:26.708 DEBUG:boto:Path: /dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.708 DEBUG:boto:Data: 2024-04-12T03:47:26.708 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.708 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.708 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.708 DEBUG:boto:Params: {} 2024-04-12T03:47:26.708 DEBUG:boto:Token: None 2024-04-12T03:47:26.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.708 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e4I/HON611T1SsMaoEb2GhTjyM4= 2024-04-12T03:47:26.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e4I/HON611T1SsMaoEb2GhTjyM4=', 'Content-Length': '0'} 2024-04-12T03:47:26.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e08c083b10b44a84-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.750 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:47:26.750 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:47:26.751 DEBUG:boto:path=/dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.751 DEBUG:boto:auth_path=/dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.751 DEBUG:boto:Method: GET 2024-04-12T03:47:26.751 DEBUG:boto:Path: /dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.751 DEBUG:boto:Data: 2024-04-12T03:47:26.751 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.751 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.751 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.751 DEBUG:boto:Params: {} 2024-04-12T03:47:26.751 DEBUG:boto:Token: None 2024-04-12T03:47:26.752 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj2?versionId=null 2024-04-12T03:47:26.752 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e4I/HON611T1SsMaoEb2GhTjyM4= 2024-04-12T03:47:26.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e4I/HON611T1SsMaoEb2GhTjyM4=', 'Content-Length': '0'} 2024-04-12T03:47:26.754 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063b98360c1dc09ed-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.754 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.754 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.754 DEBUG:boto:path=/dkyges-48/?prefix=obj2 2024-04-12T03:47:26.754 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj2 2024-04-12T03:47:26.755 DEBUG:boto:Method: GET 2024-04-12T03:47:26.755 DEBUG:boto:Path: /dkyges-48/?prefix=obj2 2024-04-12T03:47:26.755 DEBUG:boto:Data: 2024-04-12T03:47:26.755 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.755 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.755 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.755 DEBUG:boto:Params: {} 2024-04-12T03:47:26.755 DEBUG:boto:Token: None 2024-04-12T03:47:26.755 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/ 2024-04-12T03:47:26.755 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4= 2024-04-12T03:47:26.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4=', 'Content-Length': '0'} 2024-04-12T03:47:26.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b852c5f2a33b5b80-006618aece-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.758 DEBUG:boto:b'dkyges-48obj21000falseobj22024-04-12T03:46:13.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.758 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.759 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.759 DEBUG:boto:path=/dkyges-48/?prefix=obj2 2024-04-12T03:47:26.759 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj2 2024-04-12T03:47:26.759 DEBUG:boto:Method: GET 2024-04-12T03:47:26.759 DEBUG:boto:Path: /dkyges-48/?prefix=obj2 2024-04-12T03:47:26.759 DEBUG:boto:Data: 2024-04-12T03:47:26.759 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.759 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.759 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.759 DEBUG:boto:Params: {} 2024-04-12T03:47:26.759 DEBUG:boto:Token: None 2024-04-12T03:47:26.760 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/ 2024-04-12T03:47:26.760 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4= 2024-04-12T03:47:26.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4=', 'Content-Length': '0'} 2024-04-12T03:47:26.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f73ae3a299a6973-006618aece-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.762 DEBUG:boto:b'dkyges-48obj21000falseobj22024-04-12T03:46:13.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.763 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:47:26.763 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:47:26.763 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.763 DEBUG:boto:Path: /dkyges-48/obj2 2024-04-12T03:47:26.763 DEBUG:boto:Data: 2024-04-12T03:47:26.763 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.763 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.763 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.763 DEBUG:boto:Params: {} 2024-04-12T03:47:26.764 DEBUG:boto:Token: None 2024-04-12T03:47:26.764 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj2 2024-04-12T03:47:26.764 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ie37YV3BwuIKfSCfuMwKq5bZAmw= 2024-04-12T03:47:26.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ie37YV3BwuIKfSCfuMwKq5bZAmw=', 'Content-Length': '0'} 2024-04-12T03:47:26.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036d36b8d0926f94a-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.766 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:47:26.766 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:47:26.766 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.766 DEBUG:boto:Path: /dkyges-48/obj2 2024-04-12T03:47:26.766 DEBUG:boto:Data: 2024-04-12T03:47:26.766 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.767 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.767 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.767 DEBUG:boto:Params: {} 2024-04-12T03:47:26.767 DEBUG:boto:Token: None 2024-04-12T03:47:26.767 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj2 2024-04-12T03:47:26.767 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ie37YV3BwuIKfSCfuMwKq5bZAmw= 2024-04-12T03:47:26.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ie37YV3BwuIKfSCfuMwKq5bZAmw=', 'Content-Length': '0'} 2024-04-12T03:47:26.769 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c953fbdfac2d8c1a-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:47:26.769 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:47:26.769 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:47:26.770 DEBUG:boto:Method: GET 2024-04-12T03:47:26.770 DEBUG:boto:Path: /dkyges-48/obj2 2024-04-12T03:47:26.770 DEBUG:boto:Data: 2024-04-12T03:47:26.770 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.770 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.770 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.770 DEBUG:boto:Params: {} 2024-04-12T03:47:26.770 DEBUG:boto:Token: None 2024-04-12T03:47:26.770 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj2 2024-04-12T03:47:26.770 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UVg17Qyi2fM9ix0subHdYPtlxwM= 2024-04-12T03:47:26.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UVg17Qyi2fM9ix0subHdYPtlxwM=', 'Content-Length': '0'} 2024-04-12T03:47:26.772 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa86eca5761ceb73-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.772 DEBUG:boto:path=/dkyges-48/obj2 2024-04-12T03:47:26.773 DEBUG:boto:auth_path=/dkyges-48/obj2 2024-04-12T03:47:26.773 DEBUG:boto:Method: GET 2024-04-12T03:47:26.773 DEBUG:boto:Path: /dkyges-48/obj2 2024-04-12T03:47:26.773 DEBUG:boto:Data: 2024-04-12T03:47:26.773 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.773 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.773 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.773 DEBUG:boto:Params: {} 2024-04-12T03:47:26.773 DEBUG:boto:Token: None 2024-04-12T03:47:26.773 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj2 2024-04-12T03:47:26.774 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UVg17Qyi2fM9ix0subHdYPtlxwM= 2024-04-12T03:47:26.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UVg17Qyi2fM9ix0subHdYPtlxwM=', 'Content-Length': '0'} 2024-04-12T03:47:26.776 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eed139fb1929faa7-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.776 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.776 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.776 DEBUG:boto:path=/dkyges-48/?prefix=obj2 2024-04-12T03:47:26.776 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj2 2024-04-12T03:47:26.776 DEBUG:boto:Method: GET 2024-04-12T03:47:26.777 DEBUG:boto:Path: /dkyges-48/?prefix=obj2 2024-04-12T03:47:26.777 DEBUG:boto:Data: 2024-04-12T03:47:26.777 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.777 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.777 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.777 DEBUG:boto:Params: {} 2024-04-12T03:47:26.777 DEBUG:boto:Token: None 2024-04-12T03:47:26.777 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/ 2024-04-12T03:47:26.777 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4= 2024-04-12T03:47:26.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4=', 'Content-Length': '0'} 2024-04-12T03:47:26.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daf4f37da36b4d81-006618aece-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.780 DEBUG:boto:b'dkyges-48obj21000falseobj22024-04-12T03:46:13.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.781 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.781 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.781 DEBUG:boto:path=/dkyges-48/?prefix=obj2 2024-04-12T03:47:26.781 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj2 2024-04-12T03:47:26.781 DEBUG:boto:Method: GET 2024-04-12T03:47:26.781 DEBUG:boto:Path: /dkyges-48/?prefix=obj2 2024-04-12T03:47:26.781 DEBUG:boto:Data: 2024-04-12T03:47:26.781 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.781 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.781 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.781 DEBUG:boto:Params: {} 2024-04-12T03:47:26.782 DEBUG:boto:Token: None 2024-04-12T03:47:26.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/ 2024-04-12T03:47:26.782 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4= 2024-04-12T03:47:26.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4=', 'Content-Length': '0'} 2024-04-12T03:47:26.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000196b8529de3957dd-006618aece-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.785 DEBUG:boto:b'dkyges-48obj21000falseobj22024-04-12T03:46:13.571Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:47:26.785 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:47:26.785 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:47:26.785 DEBUG:boto:path=/dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.785 DEBUG:boto:auth_path=/dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.785 DEBUG:boto:Method: GET 2024-04-12T03:47:26.785 DEBUG:boto:Path: /dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.786 DEBUG:boto:Data: 2024-04-12T03:47:26.786 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.786 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.786 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.786 DEBUG:boto:Params: {} 2024-04-12T03:47:26.786 DEBUG:boto:Token: None 2024-04-12T03:47:26.786 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.786 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UvqYJ8mM62WTYGnBmaRbsbeOJGo= 2024-04-12T03:47:26.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UvqYJ8mM62WTYGnBmaRbsbeOJGo=', 'Content-Length': '0'} 2024-04-12T03:47:26.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030b3aedd866c99c7-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.788 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:47:26.788 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:47:26.789 DEBUG:boto:path=/dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.789 DEBUG:boto:auth_path=/dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.789 DEBUG:boto:Method: GET 2024-04-12T03:47:26.789 DEBUG:boto:Path: /dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.789 DEBUG:boto:Data: 2024-04-12T03:47:26.789 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.789 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.789 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.789 DEBUG:boto:Params: {} 2024-04-12T03:47:26.789 DEBUG:boto:Token: None 2024-04-12T03:47:26.790 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.790 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UvqYJ8mM62WTYGnBmaRbsbeOJGo= 2024-04-12T03:47:26.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UvqYJ8mM62WTYGnBmaRbsbeOJGo=', 'Content-Length': '0'} 2024-04-12T03:47:26.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030090cb88420cb66-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.792 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:47:26.792 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:47:26.792 DEBUG:boto:path=/dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.792 DEBUG:boto:auth_path=/dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.792 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.792 DEBUG:boto:Path: /dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.792 DEBUG:boto:Data: 2024-04-12T03:47:26.793 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.793 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.793 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.793 DEBUG:boto:Params: {} 2024-04-12T03:47:26.793 DEBUG:boto:Token: None 2024-04-12T03:47:26.793 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.793 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A1te52ehTL6wwAueDaJNqhs5gXY= 2024-04-12T03:47:26.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A1te52ehTL6wwAueDaJNqhs5gXY=', 'Content-Length': '0'} 2024-04-12T03:47:26.795 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000056b40c7f3275905-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.795 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:47:26.795 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:47:26.795 DEBUG:boto:path=/dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.795 DEBUG:boto:auth_path=/dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.795 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.796 DEBUG:boto:Path: /dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.796 DEBUG:boto:Data: 2024-04-12T03:47:26.796 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.796 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.796 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.796 DEBUG:boto:Params: {} 2024-04-12T03:47:26.796 DEBUG:boto:Token: None 2024-04-12T03:47:26.796 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.796 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A1te52ehTL6wwAueDaJNqhs5gXY= 2024-04-12T03:47:26.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A1te52ehTL6wwAueDaJNqhs5gXY=', 'Content-Length': '0'} 2024-04-12T03:47:26.798 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d207bc3b2b9201fa-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:47:26.799 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:47:26.799 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:47:26.799 DEBUG:boto:path=/dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.799 DEBUG:boto:auth_path=/dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.799 DEBUG:boto:Method: GET 2024-04-12T03:47:26.799 DEBUG:boto:Path: /dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.799 DEBUG:boto:Data: 2024-04-12T03:47:26.799 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.800 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.800 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.800 DEBUG:boto:Params: {} 2024-04-12T03:47:26.800 DEBUG:boto:Token: None 2024-04-12T03:47:26.800 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.800 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UvqYJ8mM62WTYGnBmaRbsbeOJGo= 2024-04-12T03:47:26.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UvqYJ8mM62WTYGnBmaRbsbeOJGo=', 'Content-Length': '0'} 2024-04-12T03:47:26.802 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb1c64fafde564da-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.846 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:47:26.847 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:47:26.847 DEBUG:boto:path=/dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.847 DEBUG:boto:auth_path=/dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.847 DEBUG:boto:Method: GET 2024-04-12T03:47:26.847 DEBUG:boto:Path: /dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.847 DEBUG:boto:Data: 2024-04-12T03:47:26.847 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.847 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.847 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.847 DEBUG:boto:Params: {} 2024-04-12T03:47:26.847 DEBUG:boto:Token: None 2024-04-12T03:47:26.848 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj3?versionId=null 2024-04-12T03:47:26.848 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UvqYJ8mM62WTYGnBmaRbsbeOJGo= 2024-04-12T03:47:26.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UvqYJ8mM62WTYGnBmaRbsbeOJGo=', 'Content-Length': '0'} 2024-04-12T03:47:26.850 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd657002e7a75d73-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.850 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.850 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.850 DEBUG:boto:path=/dkyges-48/?prefix=obj3 2024-04-12T03:47:26.850 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj3 2024-04-12T03:47:26.850 DEBUG:boto:Method: GET 2024-04-12T03:47:26.851 DEBUG:boto:Path: /dkyges-48/?prefix=obj3 2024-04-12T03:47:26.851 DEBUG:boto:Data: 2024-04-12T03:47:26.851 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.851 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.851 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.851 DEBUG:boto:Params: {} 2024-04-12T03:47:26.851 DEBUG:boto:Token: None 2024-04-12T03:47:26.851 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/ 2024-04-12T03:47:26.851 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4= 2024-04-12T03:47:26.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4=', 'Content-Length': '0'} 2024-04-12T03:47:26.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026a026039dbf726d-006618aece-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.854 DEBUG:boto:b'dkyges-48obj31000falseobj32024-04-12T03:46:13.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.854 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.854 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.854 DEBUG:boto:path=/dkyges-48/?prefix=obj3 2024-04-12T03:47:26.854 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj3 2024-04-12T03:47:26.855 DEBUG:boto:Method: GET 2024-04-12T03:47:26.855 DEBUG:boto:Path: /dkyges-48/?prefix=obj3 2024-04-12T03:47:26.855 DEBUG:boto:Data: 2024-04-12T03:47:26.855 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.855 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.855 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.855 DEBUG:boto:Params: {} 2024-04-12T03:47:26.855 DEBUG:boto:Token: None 2024-04-12T03:47:26.855 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/ 2024-04-12T03:47:26.855 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4= 2024-04-12T03:47:26.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4=', 'Content-Length': '0'} 2024-04-12T03:47:26.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa22aef88d93900a-006618aece-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.858 DEBUG:boto:b'dkyges-48obj31000falseobj32024-04-12T03:46:13.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.858 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:47:26.858 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:47:26.858 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.859 DEBUG:boto:Path: /dkyges-48/obj3 2024-04-12T03:47:26.859 DEBUG:boto:Data: 2024-04-12T03:47:26.859 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.859 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.859 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.859 DEBUG:boto:Params: {} 2024-04-12T03:47:26.859 DEBUG:boto:Token: None 2024-04-12T03:47:26.859 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj3 2024-04-12T03:47:26.859 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yBUw9SgNiN4pHdu/6k0yuAFqX+s= 2024-04-12T03:47:26.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yBUw9SgNiN4pHdu/6k0yuAFqX+s=', 'Content-Length': '0'} 2024-04-12T03:47:26.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005ce4d6f1f4255dd5-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.861 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:47:26.861 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:47:26.861 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.861 DEBUG:boto:Path: /dkyges-48/obj3 2024-04-12T03:47:26.862 DEBUG:boto:Data: 2024-04-12T03:47:26.862 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.862 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.862 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.862 DEBUG:boto:Params: {} 2024-04-12T03:47:26.862 DEBUG:boto:Token: None 2024-04-12T03:47:26.862 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj3 2024-04-12T03:47:26.862 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yBUw9SgNiN4pHdu/6k0yuAFqX+s= 2024-04-12T03:47:26.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yBUw9SgNiN4pHdu/6k0yuAFqX+s=', 'Content-Length': '0'} 2024-04-12T03:47:26.864 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ce1ca8663e7bda7-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:47:26.864 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:47:26.865 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:47:26.865 DEBUG:boto:Method: GET 2024-04-12T03:47:26.865 DEBUG:boto:Path: /dkyges-48/obj3 2024-04-12T03:47:26.865 DEBUG:boto:Data: 2024-04-12T03:47:26.865 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.865 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.865 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.865 DEBUG:boto:Params: {} 2024-04-12T03:47:26.865 DEBUG:boto:Token: None 2024-04-12T03:47:26.865 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj3 2024-04-12T03:47:26.866 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:74mb0lKDN+thPCCDwAbZuybZmZY= 2024-04-12T03:47:26.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:74mb0lKDN+thPCCDwAbZuybZmZY=', 'Content-Length': '0'} 2024-04-12T03:47:26.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000284bbad8668178b6-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.868 DEBUG:boto:path=/dkyges-48/obj3 2024-04-12T03:47:26.868 DEBUG:boto:auth_path=/dkyges-48/obj3 2024-04-12T03:47:26.868 DEBUG:boto:Method: GET 2024-04-12T03:47:26.868 DEBUG:boto:Path: /dkyges-48/obj3 2024-04-12T03:47:26.868 DEBUG:boto:Data: 2024-04-12T03:47:26.868 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.868 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.868 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.868 DEBUG:boto:Params: {} 2024-04-12T03:47:26.868 DEBUG:boto:Token: None 2024-04-12T03:47:26.869 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj3 2024-04-12T03:47:26.869 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:74mb0lKDN+thPCCDwAbZuybZmZY= 2024-04-12T03:47:26.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:74mb0lKDN+thPCCDwAbZuybZmZY=', 'Content-Length': '0'} 2024-04-12T03:47:26.871 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2382c1382ce2041-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.871 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.871 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.871 DEBUG:boto:path=/dkyges-48/?prefix=obj3 2024-04-12T03:47:26.872 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj3 2024-04-12T03:47:26.872 DEBUG:boto:Method: GET 2024-04-12T03:47:26.872 DEBUG:boto:Path: /dkyges-48/?prefix=obj3 2024-04-12T03:47:26.872 DEBUG:boto:Data: 2024-04-12T03:47:26.872 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.872 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.872 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.872 DEBUG:boto:Params: {} 2024-04-12T03:47:26.872 DEBUG:boto:Token: None 2024-04-12T03:47:26.872 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/ 2024-04-12T03:47:26.873 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4= 2024-04-12T03:47:26.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4=', 'Content-Length': '0'} 2024-04-12T03:47:26.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001415feb60fb318dc-006618aece-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.875 DEBUG:boto:b'dkyges-48obj31000falseobj32024-04-12T03:46:13.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.875 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.875 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.875 DEBUG:boto:path=/dkyges-48/?prefix=obj3 2024-04-12T03:47:26.875 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj3 2024-04-12T03:47:26.876 DEBUG:boto:Method: GET 2024-04-12T03:47:26.876 DEBUG:boto:Path: /dkyges-48/?prefix=obj3 2024-04-12T03:47:26.876 DEBUG:boto:Data: 2024-04-12T03:47:26.876 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.876 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.876 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.876 DEBUG:boto:Params: {} 2024-04-12T03:47:26.876 DEBUG:boto:Token: None 2024-04-12T03:47:26.876 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/ 2024-04-12T03:47:26.876 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4= 2024-04-12T03:47:26.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4=', 'Content-Length': '0'} 2024-04-12T03:47:26.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e937c0a2f1c5015-006618aece-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.922 DEBUG:boto:b'dkyges-48obj31000falseobj32024-04-12T03:46:13.582Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:47:26.923 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:47:26.923 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:47:26.923 DEBUG:boto:path=/dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.923 DEBUG:boto:auth_path=/dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.923 DEBUG:boto:Method: GET 2024-04-12T03:47:26.923 DEBUG:boto:Path: /dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.923 DEBUG:boto:Data: 2024-04-12T03:47:26.923 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.923 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.923 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.924 DEBUG:boto:Params: {} 2024-04-12T03:47:26.924 DEBUG:boto:Token: None 2024-04-12T03:47:26.924 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.924 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WsiHkCM8z7M2kCqxUyC96wkmum0= 2024-04-12T03:47:26.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WsiHkCM8z7M2kCqxUyC96wkmum0=', 'Content-Length': '0'} 2024-04-12T03:47:26.926 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6a3c66c19e367bf-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.926 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:47:26.927 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:47:26.927 DEBUG:boto:path=/dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.927 DEBUG:boto:auth_path=/dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.927 DEBUG:boto:Method: GET 2024-04-12T03:47:26.927 DEBUG:boto:Path: /dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.927 DEBUG:boto:Data: 2024-04-12T03:47:26.927 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.927 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.927 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.927 DEBUG:boto:Params: {} 2024-04-12T03:47:26.928 DEBUG:boto:Token: None 2024-04-12T03:47:26.928 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.928 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WsiHkCM8z7M2kCqxUyC96wkmum0= 2024-04-12T03:47:26.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WsiHkCM8z7M2kCqxUyC96wkmum0=', 'Content-Length': '0'} 2024-04-12T03:47:26.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042c6924fea464491-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.931 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:47:26.931 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:47:26.931 DEBUG:boto:path=/dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.931 DEBUG:boto:auth_path=/dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.931 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.931 DEBUG:boto:Path: /dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.931 DEBUG:boto:Data: 2024-04-12T03:47:26.931 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.931 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.931 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.931 DEBUG:boto:Params: {} 2024-04-12T03:47:26.932 DEBUG:boto:Token: None 2024-04-12T03:47:26.932 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.932 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kpVskXv2A1QB/eQpENq3PrnpCcY= 2024-04-12T03:47:26.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kpVskXv2A1QB/eQpENq3PrnpCcY=', 'Content-Length': '0'} 2024-04-12T03:47:26.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8b9ec9f89d7d5de-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.934 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:47:26.934 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:47:26.934 DEBUG:boto:path=/dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.934 DEBUG:boto:auth_path=/dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.935 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.935 DEBUG:boto:Path: /dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.935 DEBUG:boto:Data: 2024-04-12T03:47:26.935 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.935 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.935 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.935 DEBUG:boto:Params: {} 2024-04-12T03:47:26.935 DEBUG:boto:Token: None 2024-04-12T03:47:26.935 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.935 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kpVskXv2A1QB/eQpENq3PrnpCcY= 2024-04-12T03:47:26.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kpVskXv2A1QB/eQpENq3PrnpCcY=', 'Content-Length': '0'} 2024-04-12T03:47:26.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a2556822a1232e1-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:47:26.938 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:47:26.938 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:47:26.938 DEBUG:boto:path=/dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.938 DEBUG:boto:auth_path=/dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.938 DEBUG:boto:Method: GET 2024-04-12T03:47:26.938 DEBUG:boto:Path: /dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.938 DEBUG:boto:Data: 2024-04-12T03:47:26.938 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.938 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.939 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.939 DEBUG:boto:Params: {} 2024-04-12T03:47:26.939 DEBUG:boto:Token: None 2024-04-12T03:47:26.939 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.939 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WsiHkCM8z7M2kCqxUyC96wkmum0= 2024-04-12T03:47:26.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WsiHkCM8z7M2kCqxUyC96wkmum0=', 'Content-Length': '0'} 2024-04-12T03:47:26.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6ac5702fe5a786e-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.982 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:47:26.982 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:47:26.982 DEBUG:boto:path=/dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.983 DEBUG:boto:auth_path=/dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.983 DEBUG:boto:Method: GET 2024-04-12T03:47:26.983 DEBUG:boto:Path: /dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.983 DEBUG:boto:Data: 2024-04-12T03:47:26.983 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.983 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.983 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.983 DEBUG:boto:Params: {} 2024-04-12T03:47:26.983 DEBUG:boto:Token: None 2024-04-12T03:47:26.983 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj4?versionId=null 2024-04-12T03:47:26.984 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WsiHkCM8z7M2kCqxUyC96wkmum0= 2024-04-12T03:47:26.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WsiHkCM8z7M2kCqxUyC96wkmum0=', 'Content-Length': '0'} 2024-04-12T03:47:26.985 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000996385d744445fac-006618aece-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.986 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.986 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.986 DEBUG:boto:path=/dkyges-48/?prefix=obj4 2024-04-12T03:47:26.986 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj4 2024-04-12T03:47:26.986 DEBUG:boto:Method: GET 2024-04-12T03:47:26.986 DEBUG:boto:Path: /dkyges-48/?prefix=obj4 2024-04-12T03:47:26.986 DEBUG:boto:Data: 2024-04-12T03:47:26.986 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.987 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.987 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.987 DEBUG:boto:Params: {} 2024-04-12T03:47:26.987 DEBUG:boto:Token: None 2024-04-12T03:47:26.987 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/ 2024-04-12T03:47:26.987 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4= 2024-04-12T03:47:26.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4=', 'Content-Length': '0'} 2024-04-12T03:47:26.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067e3609762292519-006618aece-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.990 DEBUG:boto:b'dkyges-48obj41000falseobj42024-04-12T03:46:13.631Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.990 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:26.990 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:26.991 DEBUG:boto:path=/dkyges-48/?prefix=obj4 2024-04-12T03:47:26.991 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj4 2024-04-12T03:47:26.991 DEBUG:boto:Method: GET 2024-04-12T03:47:26.991 DEBUG:boto:Path: /dkyges-48/?prefix=obj4 2024-04-12T03:47:26.991 DEBUG:boto:Data: 2024-04-12T03:47:26.991 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.991 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.991 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.991 DEBUG:boto:Params: {} 2024-04-12T03:47:26.991 DEBUG:boto:Token: None 2024-04-12T03:47:26.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/ 2024-04-12T03:47:26.992 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4= 2024-04-12T03:47:26.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vfrbntoWIIbfnVN9hzXmugAF6k4=', 'Content-Length': '0'} 2024-04-12T03:47:26.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8e6c5218998f1a0-006618aece-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.994 DEBUG:boto:b'dkyges-48obj41000falseobj42024-04-12T03:46:13.631Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:26.995 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:47:26.995 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:47:26.995 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.995 DEBUG:boto:Path: /dkyges-48/obj4 2024-04-12T03:47:26.995 DEBUG:boto:Data: 2024-04-12T03:47:26.995 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.995 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:26.995 DEBUG:boto:Port: 8001 2024-04-12T03:47:26.995 DEBUG:boto:Params: {} 2024-04-12T03:47:26.996 DEBUG:boto:Token: None 2024-04-12T03:47:26.996 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj4 2024-04-12T03:47:26.996 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RTIyjoTqu4gt9257ZHaEo8caePk= 2024-04-12T03:47:26.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RTIyjoTqu4gt9257ZHaEo8caePk=', 'Content-Length': '0'} 2024-04-12T03:47:26.997 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f722c772bc214fe0-006618aece-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:26.998 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:47:26.998 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:47:26.998 DEBUG:boto:Method: HEAD 2024-04-12T03:47:26.998 DEBUG:boto:Path: /dkyges-48/obj4 2024-04-12T03:47:26.998 DEBUG:boto:Data: 2024-04-12T03:47:26.998 DEBUG:boto:Headers: {} 2024-04-12T03:47:26.998 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:26.998 DEBUG:boto:Port: 8000 2024-04-12T03:47:26.998 DEBUG:boto:Params: {} 2024-04-12T03:47:26.999 DEBUG:boto:Token: None 2024-04-12T03:47:26.999 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:26 GMT /dkyges-48/obj4 2024-04-12T03:47:26.999 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RTIyjoTqu4gt9257ZHaEo8caePk= 2024-04-12T03:47:26.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RTIyjoTqu4gt9257ZHaEo8caePk=', 'Content-Length': '0'} 2024-04-12T03:47:27.001 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084f844d37afe7392-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:47:27.001 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:47:27.001 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:47:27.001 DEBUG:boto:Method: GET 2024-04-12T03:47:27.001 DEBUG:boto:Path: /dkyges-48/obj4 2024-04-12T03:47:27.001 DEBUG:boto:Data: 2024-04-12T03:47:27.002 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.002 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.002 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.002 DEBUG:boto:Params: {} 2024-04-12T03:47:27.002 DEBUG:boto:Token: None 2024-04-12T03:47:27.002 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj4 2024-04-12T03:47:27.002 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uxCNDHygpCben5TkhMn57PyzWOY= 2024-04-12T03:47:27.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uxCNDHygpCben5TkhMn57PyzWOY=', 'Content-Length': '0'} 2024-04-12T03:47:27.004 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a81d80da317986a5-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.004 DEBUG:boto:path=/dkyges-48/obj4 2024-04-12T03:47:27.004 DEBUG:boto:auth_path=/dkyges-48/obj4 2024-04-12T03:47:27.005 DEBUG:boto:Method: GET 2024-04-12T03:47:27.005 DEBUG:boto:Path: /dkyges-48/obj4 2024-04-12T03:47:27.005 DEBUG:boto:Data: 2024-04-12T03:47:27.005 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.005 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.005 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.005 DEBUG:boto:Params: {} 2024-04-12T03:47:27.005 DEBUG:boto:Token: None 2024-04-12T03:47:27.005 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj4 2024-04-12T03:47:27.005 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uxCNDHygpCben5TkhMn57PyzWOY= 2024-04-12T03:47:27.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uxCNDHygpCben5TkhMn57PyzWOY=', 'Content-Length': '0'} 2024-04-12T03:47:27.007 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f601f5921d3e80ad-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.050 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.051 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.051 DEBUG:boto:path=/dkyges-48/?prefix=obj4 2024-04-12T03:47:27.051 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj4 2024-04-12T03:47:27.051 DEBUG:boto:Method: GET 2024-04-12T03:47:27.051 DEBUG:boto:Path: /dkyges-48/?prefix=obj4 2024-04-12T03:47:27.051 DEBUG:boto:Data: 2024-04-12T03:47:27.051 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.051 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.051 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.052 DEBUG:boto:Params: {} 2024-04-12T03:47:27.052 DEBUG:boto:Token: None 2024-04-12T03:47:27.052 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.052 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce893cd2edcef255-006618aecf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.055 DEBUG:boto:b'dkyges-48obj41000falseobj42024-04-12T03:46:13.631Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.055 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.055 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.055 DEBUG:boto:path=/dkyges-48/?prefix=obj4 2024-04-12T03:47:27.055 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj4 2024-04-12T03:47:27.055 DEBUG:boto:Method: GET 2024-04-12T03:47:27.056 DEBUG:boto:Path: /dkyges-48/?prefix=obj4 2024-04-12T03:47:27.056 DEBUG:boto:Data: 2024-04-12T03:47:27.056 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.056 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.056 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.056 DEBUG:boto:Params: {} 2024-04-12T03:47:27.056 DEBUG:boto:Token: None 2024-04-12T03:47:27.056 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.056 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085b978e4907c3442-006618aecf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.059 DEBUG:boto:b'dkyges-48obj41000falseobj42024-04-12T03:46:13.631Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:47:27.059 DEBUG:boto:path=/dkyges-48/obj5 2024-04-12T03:47:27.060 DEBUG:boto:auth_path=/dkyges-48/obj5 2024-04-12T03:47:27.060 DEBUG:boto:path=/dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.060 DEBUG:boto:auth_path=/dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.060 DEBUG:boto:Method: GET 2024-04-12T03:47:27.060 DEBUG:boto:Path: /dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.060 DEBUG:boto:Data: 2024-04-12T03:47:27.060 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.060 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.060 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.060 DEBUG:boto:Params: {} 2024-04-12T03:47:27.061 DEBUG:boto:Token: None 2024-04-12T03:47:27.061 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.061 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GYYIw532MKTSKukxcYXwo4GDC64= 2024-04-12T03:47:27.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GYYIw532MKTSKukxcYXwo4GDC64=', 'Content-Length': '0'} 2024-04-12T03:47:27.063 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eba3152d90fe2b7d-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.063 DEBUG:boto:path=/dkyges-48/obj5 2024-04-12T03:47:27.063 DEBUG:boto:auth_path=/dkyges-48/obj5 2024-04-12T03:47:27.063 DEBUG:boto:path=/dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.063 DEBUG:boto:auth_path=/dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.064 DEBUG:boto:Method: GET 2024-04-12T03:47:27.064 DEBUG:boto:Path: /dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.064 DEBUG:boto:Data: 2024-04-12T03:47:27.064 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.064 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.064 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.064 DEBUG:boto:Params: {} 2024-04-12T03:47:27.064 DEBUG:boto:Token: None 2024-04-12T03:47:27.064 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.064 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GYYIw532MKTSKukxcYXwo4GDC64= 2024-04-12T03:47:27.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GYYIw532MKTSKukxcYXwo4GDC64=', 'Content-Length': '0'} 2024-04-12T03:47:27.067 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005652e2c35e00f627-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.067 DEBUG:boto:path=/dkyges-48/obj5 2024-04-12T03:47:27.067 DEBUG:boto:auth_path=/dkyges-48/obj5 2024-04-12T03:47:27.067 DEBUG:boto:path=/dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.067 DEBUG:boto:auth_path=/dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.067 DEBUG:boto:Method: HEAD 2024-04-12T03:47:27.067 DEBUG:boto:Path: /dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.067 DEBUG:boto:Data: 2024-04-12T03:47:27.068 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.068 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.068 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.068 DEBUG:boto:Params: {} 2024-04-12T03:47:27.068 DEBUG:boto:Token: None 2024-04-12T03:47:27.068 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.068 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SmFLRrizs3Clk4tdX742Lzvhk9c= 2024-04-12T03:47:27.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SmFLRrizs3Clk4tdX742Lzvhk9c=', 'Content-Length': '0'} 2024-04-12T03:47:27.070 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030546f302d4e68be-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.070 DEBUG:boto:path=/dkyges-48/obj5 2024-04-12T03:47:27.070 DEBUG:boto:auth_path=/dkyges-48/obj5 2024-04-12T03:47:27.070 DEBUG:boto:path=/dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.071 DEBUG:boto:auth_path=/dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.071 DEBUG:boto:Method: HEAD 2024-04-12T03:47:27.071 DEBUG:boto:Path: /dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.071 DEBUG:boto:Data: 2024-04-12T03:47:27.071 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.071 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.071 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.071 DEBUG:boto:Params: {} 2024-04-12T03:47:27.071 DEBUG:boto:Token: None 2024-04-12T03:47:27.071 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.072 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SmFLRrizs3Clk4tdX742Lzvhk9c= 2024-04-12T03:47:27.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SmFLRrizs3Clk4tdX742Lzvhk9c=', 'Content-Length': '0'} 2024-04-12T03:47:27.074 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9f35322a3e36afc-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:47:27.074 DEBUG:boto:path=/dkyges-48/obj5 2024-04-12T03:47:27.074 DEBUG:boto:auth_path=/dkyges-48/obj5 2024-04-12T03:47:27.074 DEBUG:boto:path=/dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.074 DEBUG:boto:auth_path=/dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.075 DEBUG:boto:Method: GET 2024-04-12T03:47:27.075 DEBUG:boto:Path: /dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.075 DEBUG:boto:Data: 2024-04-12T03:47:27.075 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.075 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.075 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.075 DEBUG:boto:Params: {} 2024-04-12T03:47:27.075 DEBUG:boto:Token: None 2024-04-12T03:47:27.075 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.075 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GYYIw532MKTSKukxcYXwo4GDC64= 2024-04-12T03:47:27.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GYYIw532MKTSKukxcYXwo4GDC64=', 'Content-Length': '0'} 2024-04-12T03:47:27.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c94c72da4efe554-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.118 DEBUG:boto:path=/dkyges-48/obj5 2024-04-12T03:47:27.119 DEBUG:boto:auth_path=/dkyges-48/obj5 2024-04-12T03:47:27.119 DEBUG:boto:path=/dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.119 DEBUG:boto:auth_path=/dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.119 DEBUG:boto:Method: GET 2024-04-12T03:47:27.119 DEBUG:boto:Path: /dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.119 DEBUG:boto:Data: 2024-04-12T03:47:27.119 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.119 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.119 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.119 DEBUG:boto:Params: {} 2024-04-12T03:47:27.119 DEBUG:boto:Token: None 2024-04-12T03:47:27.120 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj5?versionId=null 2024-04-12T03:47:27.120 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GYYIw532MKTSKukxcYXwo4GDC64= 2024-04-12T03:47:27.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GYYIw532MKTSKukxcYXwo4GDC64=', 'Content-Length': '0'} 2024-04-12T03:47:27.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002699e56c68e381c1-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.122 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.123 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.123 DEBUG:boto:path=/dkyges-48/?prefix=obj5 2024-04-12T03:47:27.123 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj5 2024-04-12T03:47:27.123 DEBUG:boto:Method: GET 2024-04-12T03:47:27.123 DEBUG:boto:Path: /dkyges-48/?prefix=obj5 2024-04-12T03:47:27.123 DEBUG:boto:Data: 2024-04-12T03:47:27.123 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.123 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.123 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.123 DEBUG:boto:Params: {} 2024-04-12T03:47:27.124 DEBUG:boto:Token: None 2024-04-12T03:47:27.124 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.124 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aab47d50344ee569-006618aecf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.127 DEBUG:boto:b'dkyges-48obj51000falseobj52024-04-12T03:46:56.099Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.127 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.127 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.127 DEBUG:boto:path=/dkyges-48/?prefix=obj5 2024-04-12T03:47:27.128 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj5 2024-04-12T03:47:27.128 DEBUG:boto:Method: GET 2024-04-12T03:47:27.128 DEBUG:boto:Path: /dkyges-48/?prefix=obj5 2024-04-12T03:47:27.128 DEBUG:boto:Data: 2024-04-12T03:47:27.128 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.128 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.128 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.128 DEBUG:boto:Params: {} 2024-04-12T03:47:27.128 DEBUG:boto:Token: None 2024-04-12T03:47:27.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.129 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000450b3923b37cfc2d-006618aecf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.131 DEBUG:boto:b'dkyges-48obj51000falseobj52024-04-12T03:46:56.099Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.132 DEBUG:boto:path=/dkyges-48/obj5 2024-04-12T03:47:27.132 DEBUG:boto:auth_path=/dkyges-48/obj5 2024-04-12T03:47:27.132 DEBUG:boto:Method: HEAD 2024-04-12T03:47:27.132 DEBUG:boto:Path: /dkyges-48/obj5 2024-04-12T03:47:27.132 DEBUG:boto:Data: 2024-04-12T03:47:27.132 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.132 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.133 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.133 DEBUG:boto:Params: {} 2024-04-12T03:47:27.133 DEBUG:boto:Token: None 2024-04-12T03:47:27.133 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj5 2024-04-12T03:47:27.133 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PIWZaWlG/IBAo03+R2dtjbdDSU4= 2024-04-12T03:47:27.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PIWZaWlG/IBAo03+R2dtjbdDSU4=', 'Content-Length': '0'} 2024-04-12T03:47:27.135 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f6bb80a745156e2-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.135 DEBUG:boto:path=/dkyges-48/obj5 2024-04-12T03:47:27.136 DEBUG:boto:auth_path=/dkyges-48/obj5 2024-04-12T03:47:27.136 DEBUG:boto:Method: HEAD 2024-04-12T03:47:27.136 DEBUG:boto:Path: /dkyges-48/obj5 2024-04-12T03:47:27.136 DEBUG:boto:Data: 2024-04-12T03:47:27.136 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.136 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.136 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.136 DEBUG:boto:Params: {} 2024-04-12T03:47:27.136 DEBUG:boto:Token: None 2024-04-12T03:47:27.136 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj5 2024-04-12T03:47:27.137 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PIWZaWlG/IBAo03+R2dtjbdDSU4= 2024-04-12T03:47:27.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PIWZaWlG/IBAo03+R2dtjbdDSU4=', 'Content-Length': '0'} 2024-04-12T03:47:27.138 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5e492706b05a3f7-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:47:27.139 DEBUG:boto:path=/dkyges-48/obj5 2024-04-12T03:47:27.139 DEBUG:boto:auth_path=/dkyges-48/obj5 2024-04-12T03:47:27.139 DEBUG:boto:Method: GET 2024-04-12T03:47:27.139 DEBUG:boto:Path: /dkyges-48/obj5 2024-04-12T03:47:27.139 DEBUG:boto:Data: 2024-04-12T03:47:27.139 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.139 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.139 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.139 DEBUG:boto:Params: {} 2024-04-12T03:47:27.139 DEBUG:boto:Token: None 2024-04-12T03:47:27.140 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj5 2024-04-12T03:47:27.140 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9139hSzGF7C5W6kuPxH2Ss0ULkM= 2024-04-12T03:47:27.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9139hSzGF7C5W6kuPxH2Ss0ULkM=', 'Content-Length': '0'} 2024-04-12T03:47:27.141 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fe5f83ec682512b-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.182 DEBUG:boto:path=/dkyges-48/obj5 2024-04-12T03:47:27.182 DEBUG:boto:auth_path=/dkyges-48/obj5 2024-04-12T03:47:27.183 DEBUG:boto:Method: GET 2024-04-12T03:47:27.183 DEBUG:boto:Path: /dkyges-48/obj5 2024-04-12T03:47:27.183 DEBUG:boto:Data: 2024-04-12T03:47:27.183 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.183 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.183 DEBUG:boto:Params: {} 2024-04-12T03:47:27.183 DEBUG:boto:Token: None 2024-04-12T03:47:27.183 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj5 2024-04-12T03:47:27.184 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9139hSzGF7C5W6kuPxH2Ss0ULkM= 2024-04-12T03:47:27.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9139hSzGF7C5W6kuPxH2Ss0ULkM=', 'Content-Length': '0'} 2024-04-12T03:47:27.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb7cc2176965e4fa-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.186 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.186 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.186 DEBUG:boto:path=/dkyges-48/?prefix=obj5 2024-04-12T03:47:27.187 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj5 2024-04-12T03:47:27.187 DEBUG:boto:Method: GET 2024-04-12T03:47:27.187 DEBUG:boto:Path: /dkyges-48/?prefix=obj5 2024-04-12T03:47:27.187 DEBUG:boto:Data: 2024-04-12T03:47:27.187 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.187 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.187 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.187 DEBUG:boto:Params: {} 2024-04-12T03:47:27.187 DEBUG:boto:Token: None 2024-04-12T03:47:27.187 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.187 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014a60c9cf0745336-006618aecf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.190 DEBUG:boto:b'dkyges-48obj51000falseobj52024-04-12T03:46:56.099Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.191 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.191 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.191 DEBUG:boto:path=/dkyges-48/?prefix=obj5 2024-04-12T03:47:27.191 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj5 2024-04-12T03:47:27.191 DEBUG:boto:Method: GET 2024-04-12T03:47:27.191 DEBUG:boto:Path: /dkyges-48/?prefix=obj5 2024-04-12T03:47:27.191 DEBUG:boto:Data: 2024-04-12T03:47:27.192 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.192 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.192 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.192 DEBUG:boto:Params: {} 2024-04-12T03:47:27.192 DEBUG:boto:Token: None 2024-04-12T03:47:27.192 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.192 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fda725895a8a0c90-006618aecf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.195 DEBUG:boto:b'dkyges-48obj51000falseobj52024-04-12T03:46:56.099Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:47:27.195 DEBUG:boto:path=/dkyges-48/obj6 2024-04-12T03:47:27.195 DEBUG:boto:auth_path=/dkyges-48/obj6 2024-04-12T03:47:27.195 DEBUG:boto:path=/dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.195 DEBUG:boto:auth_path=/dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.196 DEBUG:boto:Method: GET 2024-04-12T03:47:27.196 DEBUG:boto:Path: /dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.196 DEBUG:boto:Data: 2024-04-12T03:47:27.196 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.196 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.196 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.196 DEBUG:boto:Params: {} 2024-04-12T03:47:27.196 DEBUG:boto:Token: None 2024-04-12T03:47:27.196 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.196 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NQ9yKwzPSxduOhM/OMw0JlWusGY= 2024-04-12T03:47:27.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NQ9yKwzPSxduOhM/OMw0JlWusGY=', 'Content-Length': '0'} 2024-04-12T03:47:27.198 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5405ffa95bc5923-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.199 DEBUG:boto:path=/dkyges-48/obj6 2024-04-12T03:47:27.199 DEBUG:boto:auth_path=/dkyges-48/obj6 2024-04-12T03:47:27.199 DEBUG:boto:path=/dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.199 DEBUG:boto:auth_path=/dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.199 DEBUG:boto:Method: GET 2024-04-12T03:47:27.199 DEBUG:boto:Path: /dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.199 DEBUG:boto:Data: 2024-04-12T03:47:27.199 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.199 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.199 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.200 DEBUG:boto:Params: {} 2024-04-12T03:47:27.200 DEBUG:boto:Token: None 2024-04-12T03:47:27.200 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.200 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NQ9yKwzPSxduOhM/OMw0JlWusGY= 2024-04-12T03:47:27.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NQ9yKwzPSxduOhM/OMw0JlWusGY=', 'Content-Length': '0'} 2024-04-12T03:47:27.202 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000324c62c5a0a0547a-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.242 DEBUG:boto:path=/dkyges-48/obj6 2024-04-12T03:47:27.243 DEBUG:boto:auth_path=/dkyges-48/obj6 2024-04-12T03:47:27.243 DEBUG:boto:path=/dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.243 DEBUG:boto:auth_path=/dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.243 DEBUG:boto:Method: HEAD 2024-04-12T03:47:27.243 DEBUG:boto:Path: /dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.243 DEBUG:boto:Data: 2024-04-12T03:47:27.243 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.243 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.243 DEBUG:boto:Params: {} 2024-04-12T03:47:27.244 DEBUG:boto:Token: None 2024-04-12T03:47:27.244 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.244 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wuf9qiDBV9unHZ9qFev9ZIRwl9o= 2024-04-12T03:47:27.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wuf9qiDBV9unHZ9qFev9ZIRwl9o=', 'Content-Length': '0'} 2024-04-12T03:47:27.246 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005fd93e385fa44ebe-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.246 DEBUG:boto:path=/dkyges-48/obj6 2024-04-12T03:47:27.246 DEBUG:boto:auth_path=/dkyges-48/obj6 2024-04-12T03:47:27.246 DEBUG:boto:path=/dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.246 DEBUG:boto:auth_path=/dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.246 DEBUG:boto:Method: HEAD 2024-04-12T03:47:27.247 DEBUG:boto:Path: /dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.247 DEBUG:boto:Data: 2024-04-12T03:47:27.247 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.247 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.247 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.247 DEBUG:boto:Params: {} 2024-04-12T03:47:27.247 DEBUG:boto:Token: None 2024-04-12T03:47:27.247 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.247 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wuf9qiDBV9unHZ9qFev9ZIRwl9o= 2024-04-12T03:47:27.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wuf9qiDBV9unHZ9qFev9ZIRwl9o=', 'Content-Length': '0'} 2024-04-12T03:47:27.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e3121ebfe914b7d-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:47:27.250 DEBUG:boto:path=/dkyges-48/obj6 2024-04-12T03:47:27.250 DEBUG:boto:auth_path=/dkyges-48/obj6 2024-04-12T03:47:27.250 DEBUG:boto:path=/dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.250 DEBUG:boto:auth_path=/dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.250 DEBUG:boto:Method: GET 2024-04-12T03:47:27.250 DEBUG:boto:Path: /dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.250 DEBUG:boto:Data: 2024-04-12T03:47:27.250 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.251 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.251 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.251 DEBUG:boto:Params: {} 2024-04-12T03:47:27.251 DEBUG:boto:Token: None 2024-04-12T03:47:27.251 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.251 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NQ9yKwzPSxduOhM/OMw0JlWusGY= 2024-04-12T03:47:27.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NQ9yKwzPSxduOhM/OMw0JlWusGY=', 'Content-Length': '0'} 2024-04-12T03:47:27.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d8a2630eb7a4be09-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.253 DEBUG:boto:path=/dkyges-48/obj6 2024-04-12T03:47:27.253 DEBUG:boto:auth_path=/dkyges-48/obj6 2024-04-12T03:47:27.253 DEBUG:boto:path=/dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.253 DEBUG:boto:auth_path=/dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.253 DEBUG:boto:Method: GET 2024-04-12T03:47:27.254 DEBUG:boto:Path: /dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.254 DEBUG:boto:Data: 2024-04-12T03:47:27.254 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.254 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.254 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.254 DEBUG:boto:Params: {} 2024-04-12T03:47:27.254 DEBUG:boto:Token: None 2024-04-12T03:47:27.254 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj6?versionId=null 2024-04-12T03:47:27.254 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NQ9yKwzPSxduOhM/OMw0JlWusGY= 2024-04-12T03:47:27.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NQ9yKwzPSxduOhM/OMw0JlWusGY=', 'Content-Length': '0'} 2024-04-12T03:47:27.256 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5ff91bb83766e48-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.256 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.257 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.257 DEBUG:boto:path=/dkyges-48/?prefix=obj6 2024-04-12T03:47:27.257 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj6 2024-04-12T03:47:27.257 DEBUG:boto:Method: GET 2024-04-12T03:47:27.257 DEBUG:boto:Path: /dkyges-48/?prefix=obj6 2024-04-12T03:47:27.257 DEBUG:boto:Data: 2024-04-12T03:47:27.257 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.257 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.257 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.257 DEBUG:boto:Params: {} 2024-04-12T03:47:27.258 DEBUG:boto:Token: None 2024-04-12T03:47:27.258 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.258 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b443178270701e4-006618aecf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.261 DEBUG:boto:b'dkyges-48obj61000falseobj62024-04-12T03:46:56.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.261 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.261 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.261 DEBUG:boto:path=/dkyges-48/?prefix=obj6 2024-04-12T03:47:27.261 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj6 2024-04-12T03:47:27.261 DEBUG:boto:Method: GET 2024-04-12T03:47:27.262 DEBUG:boto:Path: /dkyges-48/?prefix=obj6 2024-04-12T03:47:27.262 DEBUG:boto:Data: 2024-04-12T03:47:27.262 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.262 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.262 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.262 DEBUG:boto:Params: {} 2024-04-12T03:47:27.262 DEBUG:boto:Token: None 2024-04-12T03:47:27.262 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.262 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e9f7a4515ab5a10-006618aecf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.265 DEBUG:boto:b'dkyges-48obj61000falseobj62024-04-12T03:46:56.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.266 DEBUG:boto:path=/dkyges-48/obj6 2024-04-12T03:47:27.266 DEBUG:boto:auth_path=/dkyges-48/obj6 2024-04-12T03:47:27.266 DEBUG:boto:Method: HEAD 2024-04-12T03:47:27.266 DEBUG:boto:Path: /dkyges-48/obj6 2024-04-12T03:47:27.266 DEBUG:boto:Data: 2024-04-12T03:47:27.266 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.266 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.266 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.266 DEBUG:boto:Params: {} 2024-04-12T03:47:27.267 DEBUG:boto:Token: None 2024-04-12T03:47:27.267 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj6 2024-04-12T03:47:27.267 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hnZdiadDMpLWrJlyzzRNXi2Iz0I= 2024-04-12T03:47:27.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hnZdiadDMpLWrJlyzzRNXi2Iz0I=', 'Content-Length': '0'} 2024-04-12T03:47:27.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b88daa5fd7283e0-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.269 DEBUG:boto:path=/dkyges-48/obj6 2024-04-12T03:47:27.269 DEBUG:boto:auth_path=/dkyges-48/obj6 2024-04-12T03:47:27.269 DEBUG:boto:Method: HEAD 2024-04-12T03:47:27.269 DEBUG:boto:Path: /dkyges-48/obj6 2024-04-12T03:47:27.269 DEBUG:boto:Data: 2024-04-12T03:47:27.269 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.270 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.270 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.270 DEBUG:boto:Params: {} 2024-04-12T03:47:27.270 DEBUG:boto:Token: None 2024-04-12T03:47:27.270 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj6 2024-04-12T03:47:27.270 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hnZdiadDMpLWrJlyzzRNXi2Iz0I= 2024-04-12T03:47:27.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hnZdiadDMpLWrJlyzzRNXi2Iz0I=', 'Content-Length': '0'} 2024-04-12T03:47:27.272 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d221ad557972433e-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:47:27.272 DEBUG:boto:path=/dkyges-48/obj6 2024-04-12T03:47:27.272 DEBUG:boto:auth_path=/dkyges-48/obj6 2024-04-12T03:47:27.272 DEBUG:boto:Method: GET 2024-04-12T03:47:27.272 DEBUG:boto:Path: /dkyges-48/obj6 2024-04-12T03:47:27.273 DEBUG:boto:Data: 2024-04-12T03:47:27.273 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.273 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.273 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.273 DEBUG:boto:Params: {} 2024-04-12T03:47:27.273 DEBUG:boto:Token: None 2024-04-12T03:47:27.273 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj6 2024-04-12T03:47:27.273 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RWNbq8xC0vM/GaX3TWh1T4JTPfk= 2024-04-12T03:47:27.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RWNbq8xC0vM/GaX3TWh1T4JTPfk=', 'Content-Length': '0'} 2024-04-12T03:47:27.275 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050c862b6647145b6-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.275 DEBUG:boto:path=/dkyges-48/obj6 2024-04-12T03:47:27.275 DEBUG:boto:auth_path=/dkyges-48/obj6 2024-04-12T03:47:27.275 DEBUG:boto:Method: GET 2024-04-12T03:47:27.276 DEBUG:boto:Path: /dkyges-48/obj6 2024-04-12T03:47:27.276 DEBUG:boto:Data: 2024-04-12T03:47:27.276 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.276 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.276 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.276 DEBUG:boto:Params: {} 2024-04-12T03:47:27.276 DEBUG:boto:Token: None 2024-04-12T03:47:27.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj6 2024-04-12T03:47:27.276 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RWNbq8xC0vM/GaX3TWh1T4JTPfk= 2024-04-12T03:47:27.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RWNbq8xC0vM/GaX3TWh1T4JTPfk=', 'Content-Length': '0'} 2024-04-12T03:47:27.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045819f1b6be45e57-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.278 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.278 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.279 DEBUG:boto:path=/dkyges-48/?prefix=obj6 2024-04-12T03:47:27.279 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj6 2024-04-12T03:47:27.279 DEBUG:boto:Method: GET 2024-04-12T03:47:27.279 DEBUG:boto:Path: /dkyges-48/?prefix=obj6 2024-04-12T03:47:27.279 DEBUG:boto:Data: 2024-04-12T03:47:27.279 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.279 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.279 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.279 DEBUG:boto:Params: {} 2024-04-12T03:47:27.279 DEBUG:boto:Token: None 2024-04-12T03:47:27.280 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.280 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029265fa0080731c1-006618aecf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.330 DEBUG:boto:b'dkyges-48obj61000falseobj62024-04-12T03:46:56.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.331 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.331 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.331 DEBUG:boto:path=/dkyges-48/?prefix=obj6 2024-04-12T03:47:27.331 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj6 2024-04-12T03:47:27.331 DEBUG:boto:Method: GET 2024-04-12T03:47:27.331 DEBUG:boto:Path: /dkyges-48/?prefix=obj6 2024-04-12T03:47:27.331 DEBUG:boto:Data: 2024-04-12T03:47:27.331 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.332 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.332 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.332 DEBUG:boto:Params: {} 2024-04-12T03:47:27.332 DEBUG:boto:Token: None 2024-04-12T03:47:27.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.332 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005854084e888bf363-006618aecf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.335 DEBUG:boto:b'dkyges-48obj61000falseobj62024-04-12T03:46:56.110Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:47:27.335 DEBUG:boto:path=/dkyges-48/obj7 2024-04-12T03:47:27.335 DEBUG:boto:auth_path=/dkyges-48/obj7 2024-04-12T03:47:27.336 DEBUG:boto:path=/dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.336 DEBUG:boto:auth_path=/dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.336 DEBUG:boto:Method: GET 2024-04-12T03:47:27.336 DEBUG:boto:Path: /dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.336 DEBUG:boto:Data: 2024-04-12T03:47:27.336 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.336 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.336 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.336 DEBUG:boto:Params: {} 2024-04-12T03:47:27.336 DEBUG:boto:Token: None 2024-04-12T03:47:27.337 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.337 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bC4F4IShoOD+q7Oc1PO+CpW3ors= 2024-04-12T03:47:27.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bC4F4IShoOD+q7Oc1PO+CpW3ors=', 'Content-Length': '0'} 2024-04-12T03:47:27.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075a65a042227d213-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.339 DEBUG:boto:path=/dkyges-48/obj7 2024-04-12T03:47:27.339 DEBUG:boto:auth_path=/dkyges-48/obj7 2024-04-12T03:47:27.339 DEBUG:boto:path=/dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.340 DEBUG:boto:auth_path=/dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.340 DEBUG:boto:Method: GET 2024-04-12T03:47:27.340 DEBUG:boto:Path: /dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.340 DEBUG:boto:Data: 2024-04-12T03:47:27.340 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.340 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.340 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.340 DEBUG:boto:Params: {} 2024-04-12T03:47:27.340 DEBUG:boto:Token: None 2024-04-12T03:47:27.340 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.341 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bC4F4IShoOD+q7Oc1PO+CpW3ors= 2024-04-12T03:47:27.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bC4F4IShoOD+q7Oc1PO+CpW3ors=', 'Content-Length': '0'} 2024-04-12T03:47:27.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eeae7a3646fcfe37-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.343 DEBUG:boto:path=/dkyges-48/obj7 2024-04-12T03:47:27.343 DEBUG:boto:auth_path=/dkyges-48/obj7 2024-04-12T03:47:27.343 DEBUG:boto:path=/dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.343 DEBUG:boto:auth_path=/dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.343 DEBUG:boto:Method: HEAD 2024-04-12T03:47:27.343 DEBUG:boto:Path: /dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.343 DEBUG:boto:Data: 2024-04-12T03:47:27.343 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.343 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.343 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.344 DEBUG:boto:Params: {} 2024-04-12T03:47:27.344 DEBUG:boto:Token: None 2024-04-12T03:47:27.344 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.344 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DvHMWGIGlSd3cxVXg/bzRzZLSmY= 2024-04-12T03:47:27.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DvHMWGIGlSd3cxVXg/bzRzZLSmY=', 'Content-Length': '0'} 2024-04-12T03:47:27.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000978b699d255a8704-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.346 DEBUG:boto:path=/dkyges-48/obj7 2024-04-12T03:47:27.346 DEBUG:boto:auth_path=/dkyges-48/obj7 2024-04-12T03:47:27.346 DEBUG:boto:path=/dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.346 DEBUG:boto:auth_path=/dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.346 DEBUG:boto:Method: HEAD 2024-04-12T03:47:27.346 DEBUG:boto:Path: /dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.347 DEBUG:boto:Data: 2024-04-12T03:47:27.347 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.347 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.347 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.347 DEBUG:boto:Params: {} 2024-04-12T03:47:27.347 DEBUG:boto:Token: None 2024-04-12T03:47:27.347 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.347 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DvHMWGIGlSd3cxVXg/bzRzZLSmY= 2024-04-12T03:47:27.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DvHMWGIGlSd3cxVXg/bzRzZLSmY=', 'Content-Length': '0'} 2024-04-12T03:47:27.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000117722e18349cd3f-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:47:27.349 DEBUG:boto:path=/dkyges-48/obj7 2024-04-12T03:47:27.349 DEBUG:boto:auth_path=/dkyges-48/obj7 2024-04-12T03:47:27.350 DEBUG:boto:path=/dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.350 DEBUG:boto:auth_path=/dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.350 DEBUG:boto:Method: GET 2024-04-12T03:47:27.350 DEBUG:boto:Path: /dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.350 DEBUG:boto:Data: 2024-04-12T03:47:27.350 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.350 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.350 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.350 DEBUG:boto:Params: {} 2024-04-12T03:47:27.350 DEBUG:boto:Token: None 2024-04-12T03:47:27.351 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.351 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bC4F4IShoOD+q7Oc1PO+CpW3ors= 2024-04-12T03:47:27.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bC4F4IShoOD+q7Oc1PO+CpW3ors=', 'Content-Length': '0'} 2024-04-12T03:47:27.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e56f805860ebdee9-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.353 DEBUG:boto:path=/dkyges-48/obj7 2024-04-12T03:47:27.353 DEBUG:boto:auth_path=/dkyges-48/obj7 2024-04-12T03:47:27.353 DEBUG:boto:path=/dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.353 DEBUG:boto:auth_path=/dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.353 DEBUG:boto:Method: GET 2024-04-12T03:47:27.353 DEBUG:boto:Path: /dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.353 DEBUG:boto:Data: 2024-04-12T03:47:27.353 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.354 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.354 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.354 DEBUG:boto:Params: {} 2024-04-12T03:47:27.354 DEBUG:boto:Token: None 2024-04-12T03:47:27.354 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj7?versionId=null 2024-04-12T03:47:27.354 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bC4F4IShoOD+q7Oc1PO+CpW3ors= 2024-04-12T03:47:27.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bC4F4IShoOD+q7Oc1PO+CpW3ors=', 'Content-Length': '0'} 2024-04-12T03:47:27.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000550d13e646f811f7-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.398 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.399 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.399 DEBUG:boto:path=/dkyges-48/?prefix=obj7 2024-04-12T03:47:27.399 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj7 2024-04-12T03:47:27.399 DEBUG:boto:Method: GET 2024-04-12T03:47:27.399 DEBUG:boto:Path: /dkyges-48/?prefix=obj7 2024-04-12T03:47:27.399 DEBUG:boto:Data: 2024-04-12T03:47:27.399 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.399 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.399 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.399 DEBUG:boto:Params: {} 2024-04-12T03:47:27.399 DEBUG:boto:Token: None 2024-04-12T03:47:27.400 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.400 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f17b37f9db4c48b-006618aecf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.403 DEBUG:boto:b'dkyges-48obj71000falseobj72024-04-12T03:46:56.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.403 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.403 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.404 DEBUG:boto:path=/dkyges-48/?prefix=obj7 2024-04-12T03:47:27.404 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj7 2024-04-12T03:47:27.404 DEBUG:boto:Method: GET 2024-04-12T03:47:27.404 DEBUG:boto:Path: /dkyges-48/?prefix=obj7 2024-04-12T03:47:27.404 DEBUG:boto:Data: 2024-04-12T03:47:27.404 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.404 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.404 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.404 DEBUG:boto:Params: {} 2024-04-12T03:47:27.405 DEBUG:boto:Token: None 2024-04-12T03:47:27.405 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.405 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002adf95774e1dc8ab-006618aecf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.408 DEBUG:boto:b'dkyges-48obj71000falseobj72024-04-12T03:46:56.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.408 DEBUG:boto:path=/dkyges-48/obj7 2024-04-12T03:47:27.408 DEBUG:boto:auth_path=/dkyges-48/obj7 2024-04-12T03:47:27.408 DEBUG:boto:Method: HEAD 2024-04-12T03:47:27.408 DEBUG:boto:Path: /dkyges-48/obj7 2024-04-12T03:47:27.408 DEBUG:boto:Data: 2024-04-12T03:47:27.409 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.409 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.409 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.409 DEBUG:boto:Params: {} 2024-04-12T03:47:27.409 DEBUG:boto:Token: None 2024-04-12T03:47:27.409 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj7 2024-04-12T03:47:27.409 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IdyTD9zqdvHOdPbr/obkTq89hA8= 2024-04-12T03:47:27.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IdyTD9zqdvHOdPbr/obkTq89hA8=', 'Content-Length': '0'} 2024-04-12T03:47:27.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ced1d77da9f0e768-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.411 DEBUG:boto:path=/dkyges-48/obj7 2024-04-12T03:47:27.411 DEBUG:boto:auth_path=/dkyges-48/obj7 2024-04-12T03:47:27.411 DEBUG:boto:Method: HEAD 2024-04-12T03:47:27.411 DEBUG:boto:Path: /dkyges-48/obj7 2024-04-12T03:47:27.412 DEBUG:boto:Data: 2024-04-12T03:47:27.412 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.412 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.412 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.412 DEBUG:boto:Params: {} 2024-04-12T03:47:27.412 DEBUG:boto:Token: None 2024-04-12T03:47:27.412 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj7 2024-04-12T03:47:27.412 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IdyTD9zqdvHOdPbr/obkTq89hA8= 2024-04-12T03:47:27.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IdyTD9zqdvHOdPbr/obkTq89hA8=', 'Content-Length': '0'} 2024-04-12T03:47:27.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4ac99ae2c8bd085-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:47:27.414 DEBUG:boto:path=/dkyges-48/obj7 2024-04-12T03:47:27.414 DEBUG:boto:auth_path=/dkyges-48/obj7 2024-04-12T03:47:27.415 DEBUG:boto:Method: GET 2024-04-12T03:47:27.415 DEBUG:boto:Path: /dkyges-48/obj7 2024-04-12T03:47:27.415 DEBUG:boto:Data: 2024-04-12T03:47:27.415 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.415 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.415 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.415 DEBUG:boto:Params: {} 2024-04-12T03:47:27.415 DEBUG:boto:Token: None 2024-04-12T03:47:27.415 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj7 2024-04-12T03:47:27.415 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4eV5i7s96ox0iJGG5KzNFPuKJmY= 2024-04-12T03:47:27.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4eV5i7s96ox0iJGG5KzNFPuKJmY=', 'Content-Length': '0'} 2024-04-12T03:47:27.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018901cce58665a09-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.462 DEBUG:boto:path=/dkyges-48/obj7 2024-04-12T03:47:27.463 DEBUG:boto:auth_path=/dkyges-48/obj7 2024-04-12T03:47:27.463 DEBUG:boto:Method: GET 2024-04-12T03:47:27.463 DEBUG:boto:Path: /dkyges-48/obj7 2024-04-12T03:47:27.463 DEBUG:boto:Data: 2024-04-12T03:47:27.463 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.463 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.463 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.463 DEBUG:boto:Params: {} 2024-04-12T03:47:27.463 DEBUG:boto:Token: None 2024-04-12T03:47:27.464 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj7 2024-04-12T03:47:27.464 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4eV5i7s96ox0iJGG5KzNFPuKJmY= 2024-04-12T03:47:27.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4eV5i7s96ox0iJGG5KzNFPuKJmY=', 'Content-Length': '0'} 2024-04-12T03:47:27.466 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d12675e52379c49-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.466 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.466 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.466 DEBUG:boto:path=/dkyges-48/?prefix=obj7 2024-04-12T03:47:27.466 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj7 2024-04-12T03:47:27.467 DEBUG:boto:Method: GET 2024-04-12T03:47:27.467 DEBUG:boto:Path: /dkyges-48/?prefix=obj7 2024-04-12T03:47:27.467 DEBUG:boto:Data: 2024-04-12T03:47:27.467 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.467 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.467 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.467 DEBUG:boto:Params: {} 2024-04-12T03:47:27.467 DEBUG:boto:Token: None 2024-04-12T03:47:27.467 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.467 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009118ab7290df86f8-006618aecf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.470 DEBUG:boto:b'dkyges-48obj71000falseobj72024-04-12T03:46:56.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.471 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.471 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.471 DEBUG:boto:path=/dkyges-48/?prefix=obj7 2024-04-12T03:47:27.471 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj7 2024-04-12T03:47:27.471 DEBUG:boto:Method: GET 2024-04-12T03:47:27.471 DEBUG:boto:Path: /dkyges-48/?prefix=obj7 2024-04-12T03:47:27.471 DEBUG:boto:Data: 2024-04-12T03:47:27.471 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.471 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.471 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.471 DEBUG:boto:Params: {} 2024-04-12T03:47:27.472 DEBUG:boto:Token: None 2024-04-12T03:47:27.472 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.472 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044f11d3fcfab402a-006618aecf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.474 DEBUG:boto:b'dkyges-48obj71000falseobj72024-04-12T03:46:56.120Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:47:27.475 DEBUG:boto:path=/dkyges-48/obj8 2024-04-12T03:47:27.475 DEBUG:boto:auth_path=/dkyges-48/obj8 2024-04-12T03:47:27.475 DEBUG:boto:path=/dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.475 DEBUG:boto:auth_path=/dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.475 DEBUG:boto:Method: GET 2024-04-12T03:47:27.475 DEBUG:boto:Path: /dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.475 DEBUG:boto:Data: 2024-04-12T03:47:27.475 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.476 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.476 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.476 DEBUG:boto:Params: {} 2024-04-12T03:47:27.476 DEBUG:boto:Token: None 2024-04-12T03:47:27.476 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.476 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JZFan7nu25Gf7BBkljZLZpUK3hk= 2024-04-12T03:47:27.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JZFan7nu25Gf7BBkljZLZpUK3hk=', 'Content-Length': '0'} 2024-04-12T03:47:27.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000428fdc7edb6bc2ef-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.478 DEBUG:boto:path=/dkyges-48/obj8 2024-04-12T03:47:27.479 DEBUG:boto:auth_path=/dkyges-48/obj8 2024-04-12T03:47:27.479 DEBUG:boto:path=/dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.479 DEBUG:boto:auth_path=/dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.479 DEBUG:boto:Method: GET 2024-04-12T03:47:27.479 DEBUG:boto:Path: /dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.479 DEBUG:boto:Data: 2024-04-12T03:47:27.479 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.479 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.479 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.479 DEBUG:boto:Params: {} 2024-04-12T03:47:27.479 DEBUG:boto:Token: None 2024-04-12T03:47:27.480 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.480 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JZFan7nu25Gf7BBkljZLZpUK3hk= 2024-04-12T03:47:27.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JZFan7nu25Gf7BBkljZLZpUK3hk=', 'Content-Length': '0'} 2024-04-12T03:47:27.482 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e74bc5aca21c6aa-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.522 DEBUG:boto:path=/dkyges-48/obj8 2024-04-12T03:47:27.523 DEBUG:boto:auth_path=/dkyges-48/obj8 2024-04-12T03:47:27.523 DEBUG:boto:path=/dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.523 DEBUG:boto:auth_path=/dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.523 DEBUG:boto:Method: HEAD 2024-04-12T03:47:27.523 DEBUG:boto:Path: /dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.523 DEBUG:boto:Data: 2024-04-12T03:47:27.523 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.523 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.523 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.523 DEBUG:boto:Params: {} 2024-04-12T03:47:27.524 DEBUG:boto:Token: None 2024-04-12T03:47:27.524 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.524 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:de7ClOjoTT0VCOqwE1x0C8Vz6/0= 2024-04-12T03:47:27.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:de7ClOjoTT0VCOqwE1x0C8Vz6/0=', 'Content-Length': '0'} 2024-04-12T03:47:27.526 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004692a3556ff94633-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.526 DEBUG:boto:path=/dkyges-48/obj8 2024-04-12T03:47:27.526 DEBUG:boto:auth_path=/dkyges-48/obj8 2024-04-12T03:47:27.527 DEBUG:boto:path=/dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.527 DEBUG:boto:auth_path=/dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.527 DEBUG:boto:Method: HEAD 2024-04-12T03:47:27.527 DEBUG:boto:Path: /dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.527 DEBUG:boto:Data: 2024-04-12T03:47:27.527 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.527 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.527 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.527 DEBUG:boto:Params: {} 2024-04-12T03:47:27.527 DEBUG:boto:Token: None 2024-04-12T03:47:27.527 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.527 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:de7ClOjoTT0VCOqwE1x0C8Vz6/0= 2024-04-12T03:47:27.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:de7ClOjoTT0VCOqwE1x0C8Vz6/0=', 'Content-Length': '0'} 2024-04-12T03:47:27.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000940b373e08851551-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:47:27.529 DEBUG:boto:path=/dkyges-48/obj8 2024-04-12T03:47:27.530 DEBUG:boto:auth_path=/dkyges-48/obj8 2024-04-12T03:47:27.530 DEBUG:boto:path=/dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.530 DEBUG:boto:auth_path=/dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.530 DEBUG:boto:Method: GET 2024-04-12T03:47:27.530 DEBUG:boto:Path: /dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.530 DEBUG:boto:Data: 2024-04-12T03:47:27.530 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.530 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.530 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.530 DEBUG:boto:Params: {} 2024-04-12T03:47:27.530 DEBUG:boto:Token: None 2024-04-12T03:47:27.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.531 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JZFan7nu25Gf7BBkljZLZpUK3hk= 2024-04-12T03:47:27.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JZFan7nu25Gf7BBkljZLZpUK3hk=', 'Content-Length': '0'} 2024-04-12T03:47:27.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af89504081c4f563-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.533 DEBUG:boto:path=/dkyges-48/obj8 2024-04-12T03:47:27.533 DEBUG:boto:auth_path=/dkyges-48/obj8 2024-04-12T03:47:27.533 DEBUG:boto:path=/dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.533 DEBUG:boto:auth_path=/dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.533 DEBUG:boto:Method: GET 2024-04-12T03:47:27.533 DEBUG:boto:Path: /dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.533 DEBUG:boto:Data: 2024-04-12T03:47:27.533 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.534 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.534 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.534 DEBUG:boto:Params: {} 2024-04-12T03:47:27.534 DEBUG:boto:Token: None 2024-04-12T03:47:27.534 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj8?versionId=null 2024-04-12T03:47:27.534 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JZFan7nu25Gf7BBkljZLZpUK3hk= 2024-04-12T03:47:27.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JZFan7nu25Gf7BBkljZLZpUK3hk=', 'Content-Length': '0'} 2024-04-12T03:47:27.536 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2bbd65528fc27ae-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.536 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.537 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.537 DEBUG:boto:path=/dkyges-48/?prefix=obj8 2024-04-12T03:47:27.537 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj8 2024-04-12T03:47:27.537 DEBUG:boto:Method: GET 2024-04-12T03:47:27.537 DEBUG:boto:Path: /dkyges-48/?prefix=obj8 2024-04-12T03:47:27.537 DEBUG:boto:Data: 2024-04-12T03:47:27.537 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.537 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.537 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.537 DEBUG:boto:Params: {} 2024-04-12T03:47:27.537 DEBUG:boto:Token: None 2024-04-12T03:47:27.538 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.538 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7bae39f3e843ad9-006618aecf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.540 DEBUG:boto:b'dkyges-48obj81000falseobj82024-04-12T03:46:56.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.541 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.541 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.541 DEBUG:boto:path=/dkyges-48/?prefix=obj8 2024-04-12T03:47:27.541 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj8 2024-04-12T03:47:27.541 DEBUG:boto:Method: GET 2024-04-12T03:47:27.541 DEBUG:boto:Path: /dkyges-48/?prefix=obj8 2024-04-12T03:47:27.541 DEBUG:boto:Data: 2024-04-12T03:47:27.541 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.541 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.541 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.542 DEBUG:boto:Params: {} 2024-04-12T03:47:27.542 DEBUG:boto:Token: None 2024-04-12T03:47:27.542 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.542 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a5144a571a04e0c-006618aecf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.545 DEBUG:boto:b'dkyges-48obj81000falseobj82024-04-12T03:46:56.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.545 DEBUG:boto:path=/dkyges-48/obj8 2024-04-12T03:47:27.545 DEBUG:boto:auth_path=/dkyges-48/obj8 2024-04-12T03:47:27.545 DEBUG:boto:Method: HEAD 2024-04-12T03:47:27.545 DEBUG:boto:Path: /dkyges-48/obj8 2024-04-12T03:47:27.545 DEBUG:boto:Data: 2024-04-12T03:47:27.545 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.546 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.546 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.546 DEBUG:boto:Params: {} 2024-04-12T03:47:27.546 DEBUG:boto:Token: None 2024-04-12T03:47:27.546 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj8 2024-04-12T03:47:27.546 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:anQlAZZjAydcsqIFcVzvFAuB/JE= 2024-04-12T03:47:27.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:anQlAZZjAydcsqIFcVzvFAuB/JE=', 'Content-Length': '0'} 2024-04-12T03:47:27.548 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1b4e0708d25c933-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.548 DEBUG:boto:path=/dkyges-48/obj8 2024-04-12T03:47:27.548 DEBUG:boto:auth_path=/dkyges-48/obj8 2024-04-12T03:47:27.549 DEBUG:boto:Method: HEAD 2024-04-12T03:47:27.549 DEBUG:boto:Path: /dkyges-48/obj8 2024-04-12T03:47:27.549 DEBUG:boto:Data: 2024-04-12T03:47:27.549 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.549 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.549 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.549 DEBUG:boto:Params: {} 2024-04-12T03:47:27.549 DEBUG:boto:Token: None 2024-04-12T03:47:27.549 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj8 2024-04-12T03:47:27.549 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:anQlAZZjAydcsqIFcVzvFAuB/JE= 2024-04-12T03:47:27.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:anQlAZZjAydcsqIFcVzvFAuB/JE=', 'Content-Length': '0'} 2024-04-12T03:47:27.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f88b39c316d14c71-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:47:27.551 DEBUG:boto:path=/dkyges-48/obj8 2024-04-12T03:47:27.552 DEBUG:boto:auth_path=/dkyges-48/obj8 2024-04-12T03:47:27.552 DEBUG:boto:Method: GET 2024-04-12T03:47:27.552 DEBUG:boto:Path: /dkyges-48/obj8 2024-04-12T03:47:27.552 DEBUG:boto:Data: 2024-04-12T03:47:27.552 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.552 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.552 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.552 DEBUG:boto:Params: {} 2024-04-12T03:47:27.552 DEBUG:boto:Token: None 2024-04-12T03:47:27.553 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj8 2024-04-12T03:47:27.553 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gAXkBku5rH7Vb6oDcbFglzPOHZU= 2024-04-12T03:47:27.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gAXkBku5rH7Vb6oDcbFglzPOHZU=', 'Content-Length': '0'} 2024-04-12T03:47:27.555 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-48:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008371fa1e51f1f90a-006618aecf-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.555 DEBUG:boto:path=/dkyges-48/obj8 2024-04-12T03:47:27.555 DEBUG:boto:auth_path=/dkyges-48/obj8 2024-04-12T03:47:27.555 DEBUG:boto:Method: GET 2024-04-12T03:47:27.555 DEBUG:boto:Path: /dkyges-48/obj8 2024-04-12T03:47:27.555 DEBUG:boto:Data: 2024-04-12T03:47:27.555 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.556 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.556 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.556 DEBUG:boto:Params: {} 2024-04-12T03:47:27.556 DEBUG:boto:Token: None 2024-04-12T03:47:27.556 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/obj8 2024-04-12T03:47:27.556 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gAXkBku5rH7Vb6oDcbFglzPOHZU= 2024-04-12T03:47:27.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gAXkBku5rH7Vb6oDcbFglzPOHZU=', 'Content-Length': '0'} 2024-04-12T03:47:27.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000364b96755fd85974-006618aecf-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.558 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.558 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.558 DEBUG:boto:path=/dkyges-48/?prefix=obj8 2024-04-12T03:47:27.559 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj8 2024-04-12T03:47:27.559 DEBUG:boto:Method: GET 2024-04-12T03:47:27.559 DEBUG:boto:Path: /dkyges-48/?prefix=obj8 2024-04-12T03:47:27.559 DEBUG:boto:Data: 2024-04-12T03:47:27.559 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:27.559 DEBUG:boto:Port: 8001 2024-04-12T03:47:27.559 DEBUG:boto:Params: {} 2024-04-12T03:47:27.559 DEBUG:boto:Token: None 2024-04-12T03:47:27.559 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.560 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bf77b8b39c27729-006618aecf-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.602 DEBUG:boto:b'dkyges-48obj81000falseobj82024-04-12T03:46:56.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.603 DEBUG:boto:path=/dkyges-48/ 2024-04-12T03:47:27.603 DEBUG:boto:auth_path=/dkyges-48/ 2024-04-12T03:47:27.603 DEBUG:boto:path=/dkyges-48/?prefix=obj8 2024-04-12T03:47:27.603 DEBUG:boto:auth_path=/dkyges-48/?prefix=obj8 2024-04-12T03:47:27.603 DEBUG:boto:Method: GET 2024-04-12T03:47:27.603 DEBUG:boto:Path: /dkyges-48/?prefix=obj8 2024-04-12T03:47:27.603 DEBUG:boto:Data: 2024-04-12T03:47:27.603 DEBUG:boto:Headers: {} 2024-04-12T03:47:27.604 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:27.604 DEBUG:boto:Port: 8000 2024-04-12T03:47:27.604 DEBUG:boto:Params: {} 2024-04-12T03:47:27.604 DEBUG:boto:Token: None 2024-04-12T03:47:27.604 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:27 GMT /dkyges-48/ 2024-04-12T03:47:27.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc= 2024-04-12T03:47:27.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:27 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AXo6/M1lBpV7ux6oqaa+M0Xolmc=', 'Content-Length': '0'} 2024-04-12T03:47:27.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e0bd7083e75e26d-006618aecf-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:27 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:27.607 DEBUG:boto:b'dkyges-48obj81000falseobj82024-04-12T03:46:56.172Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:27.607 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-48 zones={a2, a1} 2024-04-12T03:47:27.607 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-49 2024-04-12T03:47:27.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:27.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:27.608 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:47:27.676 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:27.676 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:27.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:27.723+0000 7f53c5869780 1 waiting for incremental sync to catch up: 2024-04-12T03:47:27.721 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000004.44.6, 00000000002.18.1, 00000000003.43.1, 00000000002.33.1, 00000000004.42.6, 00000000002.110.1, 00000000002.56.1, 00000000004.23.6, 00000000002.37.1, 00000000002.42.1, 00000000002.34.1] 2024-04-12T03:47:27.721 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000004.44.6, 00000000002.18.1, 00000000004.47.6, 00000000002.33.1, 00000000004.42.6, 00000000002.110.1, 00000000002.56.1, 00000000004.23.6, 00000000002.37.1, 00000000002.42.1, 00000000002.34.1] 2024-04-12T03:47:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:32.724+0000 7f53c5869780 1 bucket sync caught up with source: 2024-04-12T03:47:32.722 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000004.44.6, 00000000002.18.1, 00000000004.47.6, 00000000002.33.1, 00000000004.42.6, 00000000002.110.1, 00000000002.56.1, 00000000004.23.6, 00000000002.37.1, 00000000002.42.1, 00000000002.34.1] 2024-04-12T03:47:32.722 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000004.44.6, 00000000002.18.1, 00000000004.47.6, 00000000002.33.1, 00000000004.42.6, 00000000002.110.1, 00000000002.56.1, 00000000004.23.6, 00000000002.37.1, 00000000002.42.1, 00000000002.34.1] 2024-04-12T03:47:32.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:32.724+0000 7f53c5869780 0 bucket checkpoint complete 2024-04-12T03:47:32.740 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-49 2024-04-12T03:47:32.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:32.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:32.741 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:47:32.807 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:32.807 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:32.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:32.851+0000 7f0ecf745780 1 bucket sync caught up with source: 2024-04-12T03:47:32.849 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000002.1544.6, , 00000000002.3587.6, , 00000000002.2567.6, , , 00000000002.1878.6, , , ] 2024-04-12T03:47:32.849 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000002.1544.6, , 00000000002.3587.6, , 00000000002.2567.6, , , 00000000002.1878.6, , , ] 2024-04-12T03:47:32.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:32.851+0000 7f0ecf745780 0 bucket checkpoint complete 2024-04-12T03:47:32.864 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-49 zones={a2, a1} 2024-04-12T03:47:32.864 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:32.864 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:32.864 DEBUG:boto:Method: HEAD 2024-04-12T03:47:32.865 DEBUG:boto:Path: /dkyges-49/ 2024-04-12T03:47:32.865 DEBUG:boto:Data: 2024-04-12T03:47:32.865 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.865 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:32.865 DEBUG:boto:Port: 8001 2024-04-12T03:47:32.865 DEBUG:boto:Params: {} 2024-04-12T03:47:32.865 DEBUG:boto:Token: None 2024-04-12T03:47:32.865 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/ 2024-04-12T03:47:32.866 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lpSKf6uIcNdKT1FkVU2QuTib9sU= 2024-04-12T03:47:32.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lpSKf6uIcNdKT1FkVU2QuTib9sU=', 'Content-Length': '0'} 2024-04-12T03:47:32.868 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085a26fb6c02102de-006618aed4-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.868 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:32.868 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:32.868 DEBUG:boto:Method: HEAD 2024-04-12T03:47:32.868 DEBUG:boto:Path: /dkyges-49/ 2024-04-12T03:47:32.869 DEBUG:boto:Data: 2024-04-12T03:47:32.869 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.869 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:32.869 DEBUG:boto:Port: 8000 2024-04-12T03:47:32.869 DEBUG:boto:Params: {} 2024-04-12T03:47:32.869 DEBUG:boto:Token: None 2024-04-12T03:47:32.869 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/ 2024-04-12T03:47:32.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lpSKf6uIcNdKT1FkVU2QuTib9sU= 2024-04-12T03:47:32.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lpSKf6uIcNdKT1FkVU2QuTib9sU=', 'Content-Length': '0'} 2024-04-12T03:47:32.872 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0ee58727c2a9936-006618aed4-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.872 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:47:32.872 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:32.873 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:32.873 DEBUG:boto:path=/dkyges-49/?versions&max-keys=999 2024-04-12T03:47:32.873 DEBUG:boto:auth_path=/dkyges-49/?versions&max-keys=999 2024-04-12T03:47:32.873 DEBUG:boto:Method: GET 2024-04-12T03:47:32.873 DEBUG:boto:Path: /dkyges-49/?versions&max-keys=999 2024-04-12T03:47:32.873 DEBUG:boto:Data: 2024-04-12T03:47:32.873 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.873 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:32.873 DEBUG:boto:Port: 8001 2024-04-12T03:47:32.873 DEBUG:boto:Params: {} 2024-04-12T03:47:32.873 DEBUG:boto:Token: None 2024-04-12T03:47:32.874 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/?versions 2024-04-12T03:47:32.874 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vwpNQJ7vr0mJWTjpQg0ozBhNdV0= 2024-04-12T03:47:32.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vwpNQJ7vr0mJWTjpQg0ozBhNdV0=', 'Content-Length': '0'} 2024-04-12T03:47:32.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3c42d2539d1b0dd-006618aed4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.876 DEBUG:boto:b'dkyges-49999falseobj1nulltrue2024-04-12T03:46:13.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:46:13.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:46:13.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:46:13.731Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:46:56.183Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:46:56.194Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:46:56.204Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:46:56.252Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:32.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:47:32.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:47:32.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:47:32.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:47:32.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-12T03:47:32.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-12T03:47:32.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-12T03:47:32.878 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-12T03:47:32.878 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:47:32.878 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:32.878 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:32.879 DEBUG:boto:path=/dkyges-49/?versions&max-keys=999 2024-04-12T03:47:32.879 DEBUG:boto:auth_path=/dkyges-49/?versions&max-keys=999 2024-04-12T03:47:32.879 DEBUG:boto:Method: GET 2024-04-12T03:47:32.879 DEBUG:boto:Path: /dkyges-49/?versions&max-keys=999 2024-04-12T03:47:32.879 DEBUG:boto:Data: 2024-04-12T03:47:32.879 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.879 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:32.879 DEBUG:boto:Port: 8000 2024-04-12T03:47:32.879 DEBUG:boto:Params: {} 2024-04-12T03:47:32.879 DEBUG:boto:Token: None 2024-04-12T03:47:32.879 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/?versions 2024-04-12T03:47:32.880 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vwpNQJ7vr0mJWTjpQg0ozBhNdV0= 2024-04-12T03:47:32.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vwpNQJ7vr0mJWTjpQg0ozBhNdV0=', 'Content-Length': '0'} 2024-04-12T03:47:32.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff239751f6bc8b8b-006618aed4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.882 DEBUG:boto:b'dkyges-49999falseobj1nulltrue2024-04-12T03:46:13.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:46:13.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:46:13.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:46:13.731Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:46:56.183Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:46:56.194Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:46:56.204Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:46:56.252Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:32.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:47:32.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:47:32.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:47:32.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:47:32.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-12T03:47:32.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-12T03:47:32.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-12T03:47:32.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-12T03:47:32.884 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:32.884 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:32.884 DEBUG:boto:path=/dkyges-49/?versions&max-keys=999 2024-04-12T03:47:32.884 DEBUG:boto:auth_path=/dkyges-49/?versions&max-keys=999 2024-04-12T03:47:32.884 DEBUG:boto:Method: GET 2024-04-12T03:47:32.884 DEBUG:boto:Path: /dkyges-49/?versions&max-keys=999 2024-04-12T03:47:32.884 DEBUG:boto:Data: 2024-04-12T03:47:32.884 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.884 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:32.884 DEBUG:boto:Port: 8001 2024-04-12T03:47:32.885 DEBUG:boto:Params: {} 2024-04-12T03:47:32.885 DEBUG:boto:Token: None 2024-04-12T03:47:32.885 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/?versions 2024-04-12T03:47:32.885 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vwpNQJ7vr0mJWTjpQg0ozBhNdV0= 2024-04-12T03:47:32.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vwpNQJ7vr0mJWTjpQg0ozBhNdV0=', 'Content-Length': '0'} 2024-04-12T03:47:32.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5f4f1f8f0a324ce-006618aed4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.887 DEBUG:boto:b'dkyges-49999falseobj1nulltrue2024-04-12T03:46:13.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:46:13.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:46:13.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:46:13.731Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:46:56.183Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:46:56.194Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:46:56.204Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:46:56.252Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:32.888 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:32.888 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:32.888 DEBUG:boto:path=/dkyges-49/?versions&max-keys=999 2024-04-12T03:47:32.888 DEBUG:boto:auth_path=/dkyges-49/?versions&max-keys=999 2024-04-12T03:47:32.888 DEBUG:boto:Method: GET 2024-04-12T03:47:32.888 DEBUG:boto:Path: /dkyges-49/?versions&max-keys=999 2024-04-12T03:47:32.888 DEBUG:boto:Data: 2024-04-12T03:47:32.889 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.889 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:32.889 DEBUG:boto:Port: 8000 2024-04-12T03:47:32.889 DEBUG:boto:Params: {} 2024-04-12T03:47:32.889 DEBUG:boto:Token: None 2024-04-12T03:47:32.889 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/?versions 2024-04-12T03:47:32.889 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vwpNQJ7vr0mJWTjpQg0ozBhNdV0= 2024-04-12T03:47:32.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vwpNQJ7vr0mJWTjpQg0ozBhNdV0=', 'Content-Length': '0'} 2024-04-12T03:47:32.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098b4f5d83e1f4243-006618aed4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.892 DEBUG:boto:b'dkyges-49999falseobj1nulltrue2024-04-12T03:46:13.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:46:13.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:46:13.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:46:13.731Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:46:56.183Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:46:56.194Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:46:56.204Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:46:56.252Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:32.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:47:32.892 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:47:32.893 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:47:32.893 DEBUG:boto:path=/dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.893 DEBUG:boto:auth_path=/dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.893 DEBUG:boto:Method: GET 2024-04-12T03:47:32.893 DEBUG:boto:Path: /dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.893 DEBUG:boto:Data: 2024-04-12T03:47:32.893 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.893 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:32.893 DEBUG:boto:Port: 8001 2024-04-12T03:47:32.893 DEBUG:boto:Params: {} 2024-04-12T03:47:32.894 DEBUG:boto:Token: None 2024-04-12T03:47:32.894 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.894 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mr0+pe9VVCYdSrSwkjNW+KOfp8Q= 2024-04-12T03:47:32.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mr0+pe9VVCYdSrSwkjNW+KOfp8Q=', 'Content-Length': '0'} 2024-04-12T03:47:32.896 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d0312a6956f4918-006618aed4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.896 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:47:32.896 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:47:32.896 DEBUG:boto:path=/dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.896 DEBUG:boto:auth_path=/dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.896 DEBUG:boto:Method: GET 2024-04-12T03:47:32.896 DEBUG:boto:Path: /dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.896 DEBUG:boto:Data: 2024-04-12T03:47:32.896 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.897 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:32.897 DEBUG:boto:Port: 8000 2024-04-12T03:47:32.897 DEBUG:boto:Params: {} 2024-04-12T03:47:32.897 DEBUG:boto:Token: None 2024-04-12T03:47:32.897 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.897 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mr0+pe9VVCYdSrSwkjNW+KOfp8Q= 2024-04-12T03:47:32.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mr0+pe9VVCYdSrSwkjNW+KOfp8Q=', 'Content-Length': '0'} 2024-04-12T03:47:32.899 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a28e91fdf1ed9fb4-006618aed4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.942 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:47:32.942 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:47:32.943 DEBUG:boto:path=/dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.943 DEBUG:boto:auth_path=/dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.943 DEBUG:boto:Method: HEAD 2024-04-12T03:47:32.943 DEBUG:boto:Path: /dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.943 DEBUG:boto:Data: 2024-04-12T03:47:32.943 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.943 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:32.943 DEBUG:boto:Port: 8001 2024-04-12T03:47:32.945 DEBUG:boto:Params: {} 2024-04-12T03:47:32.945 DEBUG:boto:Token: None 2024-04-12T03:47:32.945 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.945 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l2fErKnYqoh70hgnaTJ5Vx9Y5eI= 2024-04-12T03:47:32.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l2fErKnYqoh70hgnaTJ5Vx9Y5eI=', 'Content-Length': '0'} 2024-04-12T03:47:32.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ae946788680625c-006618aed4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.947 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:47:32.947 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:47:32.948 DEBUG:boto:path=/dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.948 DEBUG:boto:auth_path=/dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.948 DEBUG:boto:Method: HEAD 2024-04-12T03:47:32.948 DEBUG:boto:Path: /dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.948 DEBUG:boto:Data: 2024-04-12T03:47:32.948 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.948 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:32.948 DEBUG:boto:Port: 8000 2024-04-12T03:47:32.948 DEBUG:boto:Params: {} 2024-04-12T03:47:32.948 DEBUG:boto:Token: None 2024-04-12T03:47:32.949 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.949 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l2fErKnYqoh70hgnaTJ5Vx9Y5eI= 2024-04-12T03:47:32.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l2fErKnYqoh70hgnaTJ5Vx9Y5eI=', 'Content-Length': '0'} 2024-04-12T03:47:32.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000638ecc4859d1932c-006618aed4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:47:32.951 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:47:32.951 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:47:32.951 DEBUG:boto:path=/dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.951 DEBUG:boto:auth_path=/dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.951 DEBUG:boto:Method: GET 2024-04-12T03:47:32.951 DEBUG:boto:Path: /dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.951 DEBUG:boto:Data: 2024-04-12T03:47:32.951 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.951 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:32.951 DEBUG:boto:Port: 8001 2024-04-12T03:47:32.952 DEBUG:boto:Params: {} 2024-04-12T03:47:32.952 DEBUG:boto:Token: None 2024-04-12T03:47:32.952 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.952 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mr0+pe9VVCYdSrSwkjNW+KOfp8Q= 2024-04-12T03:47:32.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mr0+pe9VVCYdSrSwkjNW+KOfp8Q=', 'Content-Length': '0'} 2024-04-12T03:47:32.954 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070855677c1cc2259-006618aed4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.954 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:47:32.954 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:47:32.954 DEBUG:boto:path=/dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.954 DEBUG:boto:auth_path=/dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.954 DEBUG:boto:Method: GET 2024-04-12T03:47:32.954 DEBUG:boto:Path: /dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.954 DEBUG:boto:Data: 2024-04-12T03:47:32.955 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.955 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:32.955 DEBUG:boto:Port: 8000 2024-04-12T03:47:32.955 DEBUG:boto:Params: {} 2024-04-12T03:47:32.955 DEBUG:boto:Token: None 2024-04-12T03:47:32.955 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/obj1?versionId=null 2024-04-12T03:47:32.955 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mr0+pe9VVCYdSrSwkjNW+KOfp8Q= 2024-04-12T03:47:32.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mr0+pe9VVCYdSrSwkjNW+KOfp8Q=', 'Content-Length': '0'} 2024-04-12T03:47:32.957 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b9ef130f15e7b34-006618aed4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.957 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:32.958 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:32.958 DEBUG:boto:path=/dkyges-49/?prefix=obj1 2024-04-12T03:47:32.958 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj1 2024-04-12T03:47:32.958 DEBUG:boto:Method: GET 2024-04-12T03:47:32.958 DEBUG:boto:Path: /dkyges-49/?prefix=obj1 2024-04-12T03:47:32.958 DEBUG:boto:Data: 2024-04-12T03:47:32.958 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.958 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:32.958 DEBUG:boto:Port: 8001 2024-04-12T03:47:32.958 DEBUG:boto:Params: {} 2024-04-12T03:47:32.959 DEBUG:boto:Token: None 2024-04-12T03:47:32.959 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/ 2024-04-12T03:47:32.959 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WUnWO1BW8+lfdQYWOEORRgRSKXg= 2024-04-12T03:47:32.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WUnWO1BW8+lfdQYWOEORRgRSKXg=', 'Content-Length': '0'} 2024-04-12T03:47:32.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e375e8ad163384b-006618aed4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.961 DEBUG:boto:b'dkyges-49obj11000falseobj12024-04-12T03:46:13.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:32.962 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:32.962 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:32.962 DEBUG:boto:path=/dkyges-49/?prefix=obj1 2024-04-12T03:47:32.962 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj1 2024-04-12T03:47:32.962 DEBUG:boto:Method: GET 2024-04-12T03:47:32.962 DEBUG:boto:Path: /dkyges-49/?prefix=obj1 2024-04-12T03:47:32.963 DEBUG:boto:Data: 2024-04-12T03:47:32.963 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.963 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:32.963 DEBUG:boto:Port: 8000 2024-04-12T03:47:32.963 DEBUG:boto:Params: {} 2024-04-12T03:47:32.963 DEBUG:boto:Token: None 2024-04-12T03:47:32.963 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/ 2024-04-12T03:47:32.963 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WUnWO1BW8+lfdQYWOEORRgRSKXg= 2024-04-12T03:47:32.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WUnWO1BW8+lfdQYWOEORRgRSKXg=', 'Content-Length': '0'} 2024-04-12T03:47:32.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e601fba5805c2c67-006618aed4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.966 DEBUG:boto:b'dkyges-49obj11000falseobj12024-04-12T03:46:13.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:32.966 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:47:32.966 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:47:32.966 DEBUG:boto:Method: HEAD 2024-04-12T03:47:32.966 DEBUG:boto:Path: /dkyges-49/obj1 2024-04-12T03:47:32.966 DEBUG:boto:Data: 2024-04-12T03:47:32.967 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.967 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:32.967 DEBUG:boto:Port: 8001 2024-04-12T03:47:32.967 DEBUG:boto:Params: {} 2024-04-12T03:47:32.967 DEBUG:boto:Token: None 2024-04-12T03:47:32.967 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/obj1 2024-04-12T03:47:32.967 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WsvPL7nZ76UYYjt0REaCFO176TM= 2024-04-12T03:47:32.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WsvPL7nZ76UYYjt0REaCFO176TM=', 'Content-Length': '0'} 2024-04-12T03:47:32.969 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f18003d4511f902-006618aed4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.969 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:47:32.969 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:47:32.969 DEBUG:boto:Method: HEAD 2024-04-12T03:47:32.969 DEBUG:boto:Path: /dkyges-49/obj1 2024-04-12T03:47:32.970 DEBUG:boto:Data: 2024-04-12T03:47:32.970 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.970 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:32.970 DEBUG:boto:Port: 8000 2024-04-12T03:47:32.970 DEBUG:boto:Params: {} 2024-04-12T03:47:32.970 DEBUG:boto:Token: None 2024-04-12T03:47:32.970 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/obj1 2024-04-12T03:47:32.970 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WsvPL7nZ76UYYjt0REaCFO176TM= 2024-04-12T03:47:32.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WsvPL7nZ76UYYjt0REaCFO176TM=', 'Content-Length': '0'} 2024-04-12T03:47:32.972 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cec47e4ed172546a-006618aed4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:47:32.973 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:47:32.973 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:47:32.973 DEBUG:boto:Method: GET 2024-04-12T03:47:32.973 DEBUG:boto:Path: /dkyges-49/obj1 2024-04-12T03:47:32.973 DEBUG:boto:Data: 2024-04-12T03:47:32.973 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.973 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:32.973 DEBUG:boto:Port: 8001 2024-04-12T03:47:32.974 DEBUG:boto:Params: {} 2024-04-12T03:47:32.974 DEBUG:boto:Token: None 2024-04-12T03:47:32.974 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/obj1 2024-04-12T03:47:32.974 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Sr1wuHcJ0EaoiVVPh61dMi0Z2hk= 2024-04-12T03:47:32.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Sr1wuHcJ0EaoiVVPh61dMi0Z2hk=', 'Content-Length': '0'} 2024-04-12T03:47:32.976 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000067bcf09babbfd8b-006618aed4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.976 DEBUG:boto:path=/dkyges-49/obj1 2024-04-12T03:47:32.976 DEBUG:boto:auth_path=/dkyges-49/obj1 2024-04-12T03:47:32.976 DEBUG:boto:Method: GET 2024-04-12T03:47:32.976 DEBUG:boto:Path: /dkyges-49/obj1 2024-04-12T03:47:32.976 DEBUG:boto:Data: 2024-04-12T03:47:32.976 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.976 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:32.977 DEBUG:boto:Port: 8000 2024-04-12T03:47:32.977 DEBUG:boto:Params: {} 2024-04-12T03:47:32.977 DEBUG:boto:Token: None 2024-04-12T03:47:32.977 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/obj1 2024-04-12T03:47:32.977 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Sr1wuHcJ0EaoiVVPh61dMi0Z2hk= 2024-04-12T03:47:32.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Sr1wuHcJ0EaoiVVPh61dMi0Z2hk=', 'Content-Length': '0'} 2024-04-12T03:47:32.979 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec1f12b1278fbe7f-006618aed4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.979 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:32.979 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:32.979 DEBUG:boto:path=/dkyges-49/?prefix=obj1 2024-04-12T03:47:32.979 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj1 2024-04-12T03:47:32.980 DEBUG:boto:Method: GET 2024-04-12T03:47:32.980 DEBUG:boto:Path: /dkyges-49/?prefix=obj1 2024-04-12T03:47:32.980 DEBUG:boto:Data: 2024-04-12T03:47:32.980 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.980 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:32.980 DEBUG:boto:Port: 8001 2024-04-12T03:47:32.980 DEBUG:boto:Params: {} 2024-04-12T03:47:32.980 DEBUG:boto:Token: None 2024-04-12T03:47:32.980 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/ 2024-04-12T03:47:32.981 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WUnWO1BW8+lfdQYWOEORRgRSKXg= 2024-04-12T03:47:32.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WUnWO1BW8+lfdQYWOEORRgRSKXg=', 'Content-Length': '0'} 2024-04-12T03:47:32.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043284bf108839225-006618aed4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:32.983 DEBUG:boto:b'dkyges-49obj11000falseobj12024-04-12T03:46:13.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:32.984 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:32.984 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:32.984 DEBUG:boto:path=/dkyges-49/?prefix=obj1 2024-04-12T03:47:32.984 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj1 2024-04-12T03:47:32.984 DEBUG:boto:Method: GET 2024-04-12T03:47:32.984 DEBUG:boto:Path: /dkyges-49/?prefix=obj1 2024-04-12T03:47:32.984 DEBUG:boto:Data: 2024-04-12T03:47:32.984 DEBUG:boto:Headers: {} 2024-04-12T03:47:32.984 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:32.984 DEBUG:boto:Port: 8000 2024-04-12T03:47:32.984 DEBUG:boto:Params: {} 2024-04-12T03:47:32.984 DEBUG:boto:Token: None 2024-04-12T03:47:32.985 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:32 GMT /dkyges-49/ 2024-04-12T03:47:32.985 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WUnWO1BW8+lfdQYWOEORRgRSKXg= 2024-04-12T03:47:32.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WUnWO1BW8+lfdQYWOEORRgRSKXg=', 'Content-Length': '0'} 2024-04-12T03:47:32.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000988607442c788f4b-006618aed4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.030 DEBUG:boto:b'dkyges-49obj11000falseobj12024-04-12T03:46:13.649Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:47:33.031 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:47:33.031 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:47:33.031 DEBUG:boto:path=/dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.031 DEBUG:boto:auth_path=/dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.031 DEBUG:boto:Method: GET 2024-04-12T03:47:33.031 DEBUG:boto:Path: /dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.031 DEBUG:boto:Data: 2024-04-12T03:47:33.032 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.032 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.032 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.032 DEBUG:boto:Params: {} 2024-04-12T03:47:33.032 DEBUG:boto:Token: None 2024-04-12T03:47:33.032 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.032 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YRU5cC9ERDs7M0g/7GQljSk9YiM= 2024-04-12T03:47:33.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YRU5cC9ERDs7M0g/7GQljSk9YiM=', 'Content-Length': '0'} 2024-04-12T03:47:33.034 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004847fa2a39d31ee0-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.034 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:47:33.035 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:47:33.035 DEBUG:boto:path=/dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.035 DEBUG:boto:auth_path=/dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.035 DEBUG:boto:Method: GET 2024-04-12T03:47:33.035 DEBUG:boto:Path: /dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.035 DEBUG:boto:Data: 2024-04-12T03:47:33.035 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.035 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.035 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.035 DEBUG:boto:Params: {} 2024-04-12T03:47:33.035 DEBUG:boto:Token: None 2024-04-12T03:47:33.036 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.036 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YRU5cC9ERDs7M0g/7GQljSk9YiM= 2024-04-12T03:47:33.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YRU5cC9ERDs7M0g/7GQljSk9YiM=', 'Content-Length': '0'} 2024-04-12T03:47:33.038 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf99746508eeb9a0-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.038 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:47:33.038 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:47:33.038 DEBUG:boto:path=/dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.038 DEBUG:boto:auth_path=/dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.038 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.038 DEBUG:boto:Path: /dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.039 DEBUG:boto:Data: 2024-04-12T03:47:33.039 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.039 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.039 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.039 DEBUG:boto:Params: {} 2024-04-12T03:47:33.039 DEBUG:boto:Token: None 2024-04-12T03:47:33.039 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.039 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6X+/jQrPyyyMM01DCu9GQIeF/KI= 2024-04-12T03:47:33.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6X+/jQrPyyyMM01DCu9GQIeF/KI=', 'Content-Length': '0'} 2024-04-12T03:47:33.041 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000346a35fa00534153-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.041 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:47:33.041 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:47:33.041 DEBUG:boto:path=/dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.041 DEBUG:boto:auth_path=/dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.042 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.042 DEBUG:boto:Path: /dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.042 DEBUG:boto:Data: 2024-04-12T03:47:33.042 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.042 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.042 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.042 DEBUG:boto:Params: {} 2024-04-12T03:47:33.042 DEBUG:boto:Token: None 2024-04-12T03:47:33.042 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.042 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6X+/jQrPyyyMM01DCu9GQIeF/KI= 2024-04-12T03:47:33.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6X+/jQrPyyyMM01DCu9GQIeF/KI=', 'Content-Length': '0'} 2024-04-12T03:47:33.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000734028bc2565b8cd-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:47:33.045 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:47:33.045 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:47:33.045 DEBUG:boto:path=/dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.045 DEBUG:boto:auth_path=/dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.045 DEBUG:boto:Method: GET 2024-04-12T03:47:33.045 DEBUG:boto:Path: /dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.045 DEBUG:boto:Data: 2024-04-12T03:47:33.045 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.045 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.046 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.046 DEBUG:boto:Params: {} 2024-04-12T03:47:33.046 DEBUG:boto:Token: None 2024-04-12T03:47:33.046 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.046 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YRU5cC9ERDs7M0g/7GQljSk9YiM= 2024-04-12T03:47:33.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YRU5cC9ERDs7M0g/7GQljSk9YiM=', 'Content-Length': '0'} 2024-04-12T03:47:33.048 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d22acf8aa8ad11b-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.094 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:47:33.095 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:47:33.095 DEBUG:boto:path=/dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.095 DEBUG:boto:auth_path=/dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.095 DEBUG:boto:Method: GET 2024-04-12T03:47:33.095 DEBUG:boto:Path: /dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.095 DEBUG:boto:Data: 2024-04-12T03:47:33.095 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.095 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.095 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.095 DEBUG:boto:Params: {} 2024-04-12T03:47:33.096 DEBUG:boto:Token: None 2024-04-12T03:47:33.096 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj2?versionId=null 2024-04-12T03:47:33.096 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YRU5cC9ERDs7M0g/7GQljSk9YiM= 2024-04-12T03:47:33.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YRU5cC9ERDs7M0g/7GQljSk9YiM=', 'Content-Length': '0'} 2024-04-12T03:47:33.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5f4a0a9a6f0eb81-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.098 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.099 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.099 DEBUG:boto:path=/dkyges-49/?prefix=obj2 2024-04-12T03:47:33.099 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj2 2024-04-12T03:47:33.099 DEBUG:boto:Method: GET 2024-04-12T03:47:33.099 DEBUG:boto:Path: /dkyges-49/?prefix=obj2 2024-04-12T03:47:33.099 DEBUG:boto:Data: 2024-04-12T03:47:33.099 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.099 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.099 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.099 DEBUG:boto:Params: {} 2024-04-12T03:47:33.099 DEBUG:boto:Token: None 2024-04-12T03:47:33.100 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.100 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068d93753761b6c6d-006618aed5-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.102 DEBUG:boto:b'dkyges-49obj21000falseobj22024-04-12T03:46:13.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.103 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.103 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.103 DEBUG:boto:path=/dkyges-49/?prefix=obj2 2024-04-12T03:47:33.103 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj2 2024-04-12T03:47:33.103 DEBUG:boto:Method: GET 2024-04-12T03:47:33.103 DEBUG:boto:Path: /dkyges-49/?prefix=obj2 2024-04-12T03:47:33.103 DEBUG:boto:Data: 2024-04-12T03:47:33.104 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.104 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.104 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.104 DEBUG:boto:Params: {} 2024-04-12T03:47:33.104 DEBUG:boto:Token: None 2024-04-12T03:47:33.104 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.104 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bfc7657478a60be-006618aed5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.107 DEBUG:boto:b'dkyges-49obj21000falseobj22024-04-12T03:46:13.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.107 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:47:33.108 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:47:33.108 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.108 DEBUG:boto:Path: /dkyges-49/obj2 2024-04-12T03:47:33.108 DEBUG:boto:Data: 2024-04-12T03:47:33.108 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.108 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.108 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.108 DEBUG:boto:Params: {} 2024-04-12T03:47:33.108 DEBUG:boto:Token: None 2024-04-12T03:47:33.108 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj2 2024-04-12T03:47:33.109 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9INpFnnmbTZyt+w/dBn1P0PY6rA= 2024-04-12T03:47:33.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9INpFnnmbTZyt+w/dBn1P0PY6rA=', 'Content-Length': '0'} 2024-04-12T03:47:33.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000083fbfae8983dd3a1-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.111 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:47:33.111 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:47:33.111 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.111 DEBUG:boto:Path: /dkyges-49/obj2 2024-04-12T03:47:33.111 DEBUG:boto:Data: 2024-04-12T03:47:33.111 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.111 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.111 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.111 DEBUG:boto:Params: {} 2024-04-12T03:47:33.111 DEBUG:boto:Token: None 2024-04-12T03:47:33.112 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj2 2024-04-12T03:47:33.112 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9INpFnnmbTZyt+w/dBn1P0PY6rA= 2024-04-12T03:47:33.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9INpFnnmbTZyt+w/dBn1P0PY6rA=', 'Content-Length': '0'} 2024-04-12T03:47:33.113 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e0b9ec8219de685-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:47:33.114 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:47:33.114 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:47:33.114 DEBUG:boto:Method: GET 2024-04-12T03:47:33.114 DEBUG:boto:Path: /dkyges-49/obj2 2024-04-12T03:47:33.114 DEBUG:boto:Data: 2024-04-12T03:47:33.114 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.114 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.114 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.114 DEBUG:boto:Params: {} 2024-04-12T03:47:33.115 DEBUG:boto:Token: None 2024-04-12T03:47:33.115 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj2 2024-04-12T03:47:33.115 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wpH/F5SSHc8z6cx5RZ203a0jlZE= 2024-04-12T03:47:33.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wpH/F5SSHc8z6cx5RZ203a0jlZE=', 'Content-Length': '0'} 2024-04-12T03:47:33.116 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000122c9c4e14e16cd6-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.158 DEBUG:boto:path=/dkyges-49/obj2 2024-04-12T03:47:33.158 DEBUG:boto:auth_path=/dkyges-49/obj2 2024-04-12T03:47:33.158 DEBUG:boto:Method: GET 2024-04-12T03:47:33.159 DEBUG:boto:Path: /dkyges-49/obj2 2024-04-12T03:47:33.159 DEBUG:boto:Data: 2024-04-12T03:47:33.159 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.159 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.159 DEBUG:boto:Params: {} 2024-04-12T03:47:33.159 DEBUG:boto:Token: None 2024-04-12T03:47:33.159 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj2 2024-04-12T03:47:33.159 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wpH/F5SSHc8z6cx5RZ203a0jlZE= 2024-04-12T03:47:33.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wpH/F5SSHc8z6cx5RZ203a0jlZE=', 'Content-Length': '0'} 2024-04-12T03:47:33.161 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000608b3f1ba98cfbb8-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.162 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.162 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.162 DEBUG:boto:path=/dkyges-49/?prefix=obj2 2024-04-12T03:47:33.162 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj2 2024-04-12T03:47:33.162 DEBUG:boto:Method: GET 2024-04-12T03:47:33.162 DEBUG:boto:Path: /dkyges-49/?prefix=obj2 2024-04-12T03:47:33.162 DEBUG:boto:Data: 2024-04-12T03:47:33.162 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.165 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.165 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.165 DEBUG:boto:Params: {} 2024-04-12T03:47:33.166 DEBUG:boto:Token: None 2024-04-12T03:47:33.166 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.166 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061cb330d3fde154f-006618aed5-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.169 DEBUG:boto:b'dkyges-49obj21000falseobj22024-04-12T03:46:13.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.169 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.169 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.169 DEBUG:boto:path=/dkyges-49/?prefix=obj2 2024-04-12T03:47:33.170 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj2 2024-04-12T03:47:33.170 DEBUG:boto:Method: GET 2024-04-12T03:47:33.170 DEBUG:boto:Path: /dkyges-49/?prefix=obj2 2024-04-12T03:47:33.170 DEBUG:boto:Data: 2024-04-12T03:47:33.170 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.170 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.170 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.170 DEBUG:boto:Params: {} 2024-04-12T03:47:33.170 DEBUG:boto:Token: None 2024-04-12T03:47:33.171 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.171 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5af2b4b87ccdd76-006618aed5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.174 DEBUG:boto:b'dkyges-49obj21000falseobj22024-04-12T03:46:13.664Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:47:33.174 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:47:33.174 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:47:33.174 DEBUG:boto:path=/dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.174 DEBUG:boto:auth_path=/dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.174 DEBUG:boto:Method: GET 2024-04-12T03:47:33.175 DEBUG:boto:Path: /dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.175 DEBUG:boto:Data: 2024-04-12T03:47:33.175 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.175 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.175 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.175 DEBUG:boto:Params: {} 2024-04-12T03:47:33.175 DEBUG:boto:Token: None 2024-04-12T03:47:33.175 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.175 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+U3lxz4HE5kK2R1pRdd0kPkGwSk= 2024-04-12T03:47:33.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+U3lxz4HE5kK2R1pRdd0kPkGwSk=', 'Content-Length': '0'} 2024-04-12T03:47:33.177 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f06468905e8bf3fc-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.177 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:47:33.177 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:47:33.178 DEBUG:boto:path=/dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.178 DEBUG:boto:auth_path=/dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.178 DEBUG:boto:Method: GET 2024-04-12T03:47:33.178 DEBUG:boto:Path: /dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.178 DEBUG:boto:Data: 2024-04-12T03:47:33.178 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.178 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.178 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.178 DEBUG:boto:Params: {} 2024-04-12T03:47:33.178 DEBUG:boto:Token: None 2024-04-12T03:47:33.178 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.179 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+U3lxz4HE5kK2R1pRdd0kPkGwSk= 2024-04-12T03:47:33.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+U3lxz4HE5kK2R1pRdd0kPkGwSk=', 'Content-Length': '0'} 2024-04-12T03:47:33.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b45b75996fa0510f-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.181 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:47:33.181 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:47:33.181 DEBUG:boto:path=/dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.181 DEBUG:boto:auth_path=/dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.181 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.181 DEBUG:boto:Path: /dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.181 DEBUG:boto:Data: 2024-04-12T03:47:33.181 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.181 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.182 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.182 DEBUG:boto:Params: {} 2024-04-12T03:47:33.182 DEBUG:boto:Token: None 2024-04-12T03:47:33.182 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.182 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hlTP0c3GgTp8KnhU2NMUp/HO2g4= 2024-04-12T03:47:33.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hlTP0c3GgTp8KnhU2NMUp/HO2g4=', 'Content-Length': '0'} 2024-04-12T03:47:33.184 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000313c2cfe066bb138-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.184 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:47:33.184 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:47:33.184 DEBUG:boto:path=/dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.184 DEBUG:boto:auth_path=/dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.184 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.185 DEBUG:boto:Path: /dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.185 DEBUG:boto:Data: 2024-04-12T03:47:33.185 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.185 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.185 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.185 DEBUG:boto:Params: {} 2024-04-12T03:47:33.185 DEBUG:boto:Token: None 2024-04-12T03:47:33.185 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.185 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hlTP0c3GgTp8KnhU2NMUp/HO2g4= 2024-04-12T03:47:33.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hlTP0c3GgTp8KnhU2NMUp/HO2g4=', 'Content-Length': '0'} 2024-04-12T03:47:33.187 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cd8e6b874c4d2b9-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:47:33.187 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:47:33.187 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:47:33.188 DEBUG:boto:path=/dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.188 DEBUG:boto:auth_path=/dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.188 DEBUG:boto:Method: GET 2024-04-12T03:47:33.188 DEBUG:boto:Path: /dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.188 DEBUG:boto:Data: 2024-04-12T03:47:33.188 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.188 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.188 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.188 DEBUG:boto:Params: {} 2024-04-12T03:47:33.188 DEBUG:boto:Token: None 2024-04-12T03:47:33.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.189 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+U3lxz4HE5kK2R1pRdd0kPkGwSk= 2024-04-12T03:47:33.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+U3lxz4HE5kK2R1pRdd0kPkGwSk=', 'Content-Length': '0'} 2024-04-12T03:47:33.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b26bd730424cc9b9-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.191 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:47:33.191 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:47:33.191 DEBUG:boto:path=/dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.191 DEBUG:boto:auth_path=/dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.191 DEBUG:boto:Method: GET 2024-04-12T03:47:33.191 DEBUG:boto:Path: /dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.191 DEBUG:boto:Data: 2024-04-12T03:47:33.191 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.191 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.192 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.192 DEBUG:boto:Params: {} 2024-04-12T03:47:33.192 DEBUG:boto:Token: None 2024-04-12T03:47:33.192 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj3?versionId=null 2024-04-12T03:47:33.192 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+U3lxz4HE5kK2R1pRdd0kPkGwSk= 2024-04-12T03:47:33.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+U3lxz4HE5kK2R1pRdd0kPkGwSk=', 'Content-Length': '0'} 2024-04-12T03:47:33.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f68f629da4ece911-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.194 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.194 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.194 DEBUG:boto:path=/dkyges-49/?prefix=obj3 2024-04-12T03:47:33.195 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj3 2024-04-12T03:47:33.195 DEBUG:boto:Method: GET 2024-04-12T03:47:33.195 DEBUG:boto:Path: /dkyges-49/?prefix=obj3 2024-04-12T03:47:33.195 DEBUG:boto:Data: 2024-04-12T03:47:33.195 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.195 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.195 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.195 DEBUG:boto:Params: {} 2024-04-12T03:47:33.195 DEBUG:boto:Token: None 2024-04-12T03:47:33.195 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.196 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9d8bd6ea19a6180-006618aed5-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.199 DEBUG:boto:b'dkyges-49obj31000falseobj32024-04-12T03:46:13.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.200 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.200 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.200 DEBUG:boto:path=/dkyges-49/?prefix=obj3 2024-04-12T03:47:33.200 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj3 2024-04-12T03:47:33.200 DEBUG:boto:Method: GET 2024-04-12T03:47:33.200 DEBUG:boto:Path: /dkyges-49/?prefix=obj3 2024-04-12T03:47:33.200 DEBUG:boto:Data: 2024-04-12T03:47:33.200 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.200 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.201 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.201 DEBUG:boto:Params: {} 2024-04-12T03:47:33.201 DEBUG:boto:Token: None 2024-04-12T03:47:33.201 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.201 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084770fdf378c06e0-006618aed5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.204 DEBUG:boto:b'dkyges-49obj31000falseobj32024-04-12T03:46:13.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.204 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:47:33.205 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:47:33.205 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.205 DEBUG:boto:Path: /dkyges-49/obj3 2024-04-12T03:47:33.205 DEBUG:boto:Data: 2024-04-12T03:47:33.205 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.205 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.205 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.205 DEBUG:boto:Params: {} 2024-04-12T03:47:33.205 DEBUG:boto:Token: None 2024-04-12T03:47:33.205 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj3 2024-04-12T03:47:33.206 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:246fln3GnJe8q79DN+usjTfeSh0= 2024-04-12T03:47:33.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:246fln3GnJe8q79DN+usjTfeSh0=', 'Content-Length': '0'} 2024-04-12T03:47:33.207 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bae2b33e074356ce-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.207 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:47:33.208 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:47:33.208 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.208 DEBUG:boto:Path: /dkyges-49/obj3 2024-04-12T03:47:33.208 DEBUG:boto:Data: 2024-04-12T03:47:33.208 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.208 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.208 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.208 DEBUG:boto:Params: {} 2024-04-12T03:47:33.208 DEBUG:boto:Token: None 2024-04-12T03:47:33.208 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj3 2024-04-12T03:47:33.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:246fln3GnJe8q79DN+usjTfeSh0= 2024-04-12T03:47:33.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:246fln3GnJe8q79DN+usjTfeSh0=', 'Content-Length': '0'} 2024-04-12T03:47:33.210 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c58c124b47dd05cc-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:47:33.211 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:47:33.211 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:47:33.211 DEBUG:boto:Method: GET 2024-04-12T03:47:33.211 DEBUG:boto:Path: /dkyges-49/obj3 2024-04-12T03:47:33.211 DEBUG:boto:Data: 2024-04-12T03:47:33.211 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.211 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.211 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.211 DEBUG:boto:Params: {} 2024-04-12T03:47:33.211 DEBUG:boto:Token: None 2024-04-12T03:47:33.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj3 2024-04-12T03:47:33.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e+/lWhyeUbxbU8XCGoa56XW/vx8= 2024-04-12T03:47:33.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e+/lWhyeUbxbU8XCGoa56XW/vx8=', 'Content-Length': '0'} 2024-04-12T03:47:33.213 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042366478f924407d-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.258 DEBUG:boto:path=/dkyges-49/obj3 2024-04-12T03:47:33.259 DEBUG:boto:auth_path=/dkyges-49/obj3 2024-04-12T03:47:33.259 DEBUG:boto:Method: GET 2024-04-12T03:47:33.259 DEBUG:boto:Path: /dkyges-49/obj3 2024-04-12T03:47:33.259 DEBUG:boto:Data: 2024-04-12T03:47:33.259 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.259 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.259 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.259 DEBUG:boto:Params: {} 2024-04-12T03:47:33.259 DEBUG:boto:Token: None 2024-04-12T03:47:33.259 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj3 2024-04-12T03:47:33.260 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e+/lWhyeUbxbU8XCGoa56XW/vx8= 2024-04-12T03:47:33.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e+/lWhyeUbxbU8XCGoa56XW/vx8=', 'Content-Length': '0'} 2024-04-12T03:47:33.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ced29f11905c82a-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.262 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.262 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.262 DEBUG:boto:path=/dkyges-49/?prefix=obj3 2024-04-12T03:47:33.262 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj3 2024-04-12T03:47:33.262 DEBUG:boto:Method: GET 2024-04-12T03:47:33.262 DEBUG:boto:Path: /dkyges-49/?prefix=obj3 2024-04-12T03:47:33.263 DEBUG:boto:Data: 2024-04-12T03:47:33.263 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.263 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.263 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.263 DEBUG:boto:Params: {} 2024-04-12T03:47:33.263 DEBUG:boto:Token: None 2024-04-12T03:47:33.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.263 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e41803ffb900a274-006618aed5-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.266 DEBUG:boto:b'dkyges-49obj31000falseobj32024-04-12T03:46:13.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.266 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.267 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.267 DEBUG:boto:path=/dkyges-49/?prefix=obj3 2024-04-12T03:47:33.267 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj3 2024-04-12T03:47:33.267 DEBUG:boto:Method: GET 2024-04-12T03:47:33.267 DEBUG:boto:Path: /dkyges-49/?prefix=obj3 2024-04-12T03:47:33.267 DEBUG:boto:Data: 2024-04-12T03:47:33.267 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.267 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.267 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.267 DEBUG:boto:Params: {} 2024-04-12T03:47:33.267 DEBUG:boto:Token: None 2024-04-12T03:47:33.268 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.268 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a15801beac57af40-006618aed5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.270 DEBUG:boto:b'dkyges-49obj31000falseobj32024-04-12T03:46:13.676Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:47:33.271 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:47:33.271 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:47:33.271 DEBUG:boto:path=/dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.271 DEBUG:boto:auth_path=/dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.271 DEBUG:boto:Method: GET 2024-04-12T03:47:33.271 DEBUG:boto:Path: /dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.271 DEBUG:boto:Data: 2024-04-12T03:47:33.271 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.271 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.272 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.272 DEBUG:boto:Params: {} 2024-04-12T03:47:33.272 DEBUG:boto:Token: None 2024-04-12T03:47:33.272 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.272 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nEUMpvn7KC5jkSNmHMuhsxkCkpE= 2024-04-12T03:47:33.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nEUMpvn7KC5jkSNmHMuhsxkCkpE=', 'Content-Length': '0'} 2024-04-12T03:47:33.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e4906a7bf8455b0-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.274 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:47:33.274 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:47:33.275 DEBUG:boto:path=/dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.275 DEBUG:boto:auth_path=/dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.275 DEBUG:boto:Method: GET 2024-04-12T03:47:33.275 DEBUG:boto:Path: /dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.275 DEBUG:boto:Data: 2024-04-12T03:47:33.275 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.275 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.275 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.275 DEBUG:boto:Params: {} 2024-04-12T03:47:33.275 DEBUG:boto:Token: None 2024-04-12T03:47:33.275 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.276 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nEUMpvn7KC5jkSNmHMuhsxkCkpE= 2024-04-12T03:47:33.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nEUMpvn7KC5jkSNmHMuhsxkCkpE=', 'Content-Length': '0'} 2024-04-12T03:47:33.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0d945ce6d89edd8-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.318 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:47:33.318 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:47:33.319 DEBUG:boto:path=/dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.319 DEBUG:boto:auth_path=/dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.319 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.319 DEBUG:boto:Path: /dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.319 DEBUG:boto:Data: 2024-04-12T03:47:33.319 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.319 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.319 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.319 DEBUG:boto:Params: {} 2024-04-12T03:47:33.319 DEBUG:boto:Token: None 2024-04-12T03:47:33.319 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.320 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AJZpSWwtSXolHof852MYlagmn68= 2024-04-12T03:47:33.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AJZpSWwtSXolHof852MYlagmn68=', 'Content-Length': '0'} 2024-04-12T03:47:33.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cb08ea5c4089d79-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.322 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:47:33.322 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:47:33.322 DEBUG:boto:path=/dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.322 DEBUG:boto:auth_path=/dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.322 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.322 DEBUG:boto:Path: /dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.322 DEBUG:boto:Data: 2024-04-12T03:47:33.322 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.322 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.323 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.323 DEBUG:boto:Params: {} 2024-04-12T03:47:33.323 DEBUG:boto:Token: None 2024-04-12T03:47:33.323 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.323 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AJZpSWwtSXolHof852MYlagmn68= 2024-04-12T03:47:33.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AJZpSWwtSXolHof852MYlagmn68=', 'Content-Length': '0'} 2024-04-12T03:47:33.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003609dbba2a4aae23-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:47:33.325 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:47:33.325 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:47:33.325 DEBUG:boto:path=/dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.325 DEBUG:boto:auth_path=/dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.326 DEBUG:boto:Method: GET 2024-04-12T03:47:33.326 DEBUG:boto:Path: /dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.326 DEBUG:boto:Data: 2024-04-12T03:47:33.326 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.326 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.326 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.326 DEBUG:boto:Params: {} 2024-04-12T03:47:33.326 DEBUG:boto:Token: None 2024-04-12T03:47:33.326 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.326 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nEUMpvn7KC5jkSNmHMuhsxkCkpE= 2024-04-12T03:47:33.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nEUMpvn7KC5jkSNmHMuhsxkCkpE=', 'Content-Length': '0'} 2024-04-12T03:47:33.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a919a366800969c5-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.328 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:47:33.329 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:47:33.329 DEBUG:boto:path=/dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.329 DEBUG:boto:auth_path=/dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.329 DEBUG:boto:Method: GET 2024-04-12T03:47:33.329 DEBUG:boto:Path: /dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.329 DEBUG:boto:Data: 2024-04-12T03:47:33.329 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.329 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.329 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.329 DEBUG:boto:Params: {} 2024-04-12T03:47:33.330 DEBUG:boto:Token: None 2024-04-12T03:47:33.330 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj4?versionId=null 2024-04-12T03:47:33.330 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nEUMpvn7KC5jkSNmHMuhsxkCkpE= 2024-04-12T03:47:33.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nEUMpvn7KC5jkSNmHMuhsxkCkpE=', 'Content-Length': '0'} 2024-04-12T03:47:33.332 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e626c4c35de86793-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.332 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.332 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.332 DEBUG:boto:path=/dkyges-49/?prefix=obj4 2024-04-12T03:47:33.332 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj4 2024-04-12T03:47:33.333 DEBUG:boto:Method: GET 2024-04-12T03:47:33.333 DEBUG:boto:Path: /dkyges-49/?prefix=obj4 2024-04-12T03:47:33.333 DEBUG:boto:Data: 2024-04-12T03:47:33.333 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.333 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.333 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.333 DEBUG:boto:Params: {} 2024-04-12T03:47:33.333 DEBUG:boto:Token: None 2024-04-12T03:47:33.333 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.333 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dda5d90ff00f13e-006618aed5-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.336 DEBUG:boto:b'dkyges-49obj41000falseobj42024-04-12T03:46:13.731Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.337 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.337 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.337 DEBUG:boto:path=/dkyges-49/?prefix=obj4 2024-04-12T03:47:33.337 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj4 2024-04-12T03:47:33.337 DEBUG:boto:Method: GET 2024-04-12T03:47:33.337 DEBUG:boto:Path: /dkyges-49/?prefix=obj4 2024-04-12T03:47:33.337 DEBUG:boto:Data: 2024-04-12T03:47:33.337 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.337 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.337 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.337 DEBUG:boto:Params: {} 2024-04-12T03:47:33.338 DEBUG:boto:Token: None 2024-04-12T03:47:33.338 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.338 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b62c00132ce0bbd-006618aed5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.341 DEBUG:boto:b'dkyges-49obj41000falseobj42024-04-12T03:46:13.731Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.341 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:47:33.341 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:47:33.341 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.341 DEBUG:boto:Path: /dkyges-49/obj4 2024-04-12T03:47:33.342 DEBUG:boto:Data: 2024-04-12T03:47:33.342 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.342 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.342 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.342 DEBUG:boto:Params: {} 2024-04-12T03:47:33.342 DEBUG:boto:Token: None 2024-04-12T03:47:33.342 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj4 2024-04-12T03:47:33.342 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lsafdpSnj9kBpye6Y5GREgE9q+0= 2024-04-12T03:47:33.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lsafdpSnj9kBpye6Y5GREgE9q+0=', 'Content-Length': '0'} 2024-04-12T03:47:33.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012fe1a5684ee35a2-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.344 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:47:33.344 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:47:33.345 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.345 DEBUG:boto:Path: /dkyges-49/obj4 2024-04-12T03:47:33.345 DEBUG:boto:Data: 2024-04-12T03:47:33.345 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.345 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.345 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.345 DEBUG:boto:Params: {} 2024-04-12T03:47:33.345 DEBUG:boto:Token: None 2024-04-12T03:47:33.345 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj4 2024-04-12T03:47:33.345 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lsafdpSnj9kBpye6Y5GREgE9q+0= 2024-04-12T03:47:33.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lsafdpSnj9kBpye6Y5GREgE9q+0=', 'Content-Length': '0'} 2024-04-12T03:47:33.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcd27f064bdbda40-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:47:33.347 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:47:33.348 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:47:33.348 DEBUG:boto:Method: GET 2024-04-12T03:47:33.348 DEBUG:boto:Path: /dkyges-49/obj4 2024-04-12T03:47:33.348 DEBUG:boto:Data: 2024-04-12T03:47:33.348 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.348 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.348 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.348 DEBUG:boto:Params: {} 2024-04-12T03:47:33.348 DEBUG:boto:Token: None 2024-04-12T03:47:33.349 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj4 2024-04-12T03:47:33.349 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8nOK9g93cHeOb/Er1NHjB4kjGho= 2024-04-12T03:47:33.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8nOK9g93cHeOb/Er1NHjB4kjGho=', 'Content-Length': '0'} 2024-04-12T03:47:33.350 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4a6e7d740e56359-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.351 DEBUG:boto:path=/dkyges-49/obj4 2024-04-12T03:47:33.351 DEBUG:boto:auth_path=/dkyges-49/obj4 2024-04-12T03:47:33.351 DEBUG:boto:Method: GET 2024-04-12T03:47:33.351 DEBUG:boto:Path: /dkyges-49/obj4 2024-04-12T03:47:33.351 DEBUG:boto:Data: 2024-04-12T03:47:33.351 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.351 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.351 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.351 DEBUG:boto:Params: {} 2024-04-12T03:47:33.352 DEBUG:boto:Token: None 2024-04-12T03:47:33.352 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj4 2024-04-12T03:47:33.352 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8nOK9g93cHeOb/Er1NHjB4kjGho= 2024-04-12T03:47:33.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8nOK9g93cHeOb/Er1NHjB4kjGho=', 'Content-Length': '0'} 2024-04-12T03:47:33.354 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ab4a7c70a15ae57-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.354 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.354 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.354 DEBUG:boto:path=/dkyges-49/?prefix=obj4 2024-04-12T03:47:33.354 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj4 2024-04-12T03:47:33.354 DEBUG:boto:Method: GET 2024-04-12T03:47:33.355 DEBUG:boto:Path: /dkyges-49/?prefix=obj4 2024-04-12T03:47:33.355 DEBUG:boto:Data: 2024-04-12T03:47:33.355 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.355 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.355 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.355 DEBUG:boto:Params: {} 2024-04-12T03:47:33.355 DEBUG:boto:Token: None 2024-04-12T03:47:33.355 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.355 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002510ffa169b9f0da-006618aed5-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.398 DEBUG:boto:b'dkyges-49obj41000falseobj42024-04-12T03:46:13.731Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.399 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.399 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.399 DEBUG:boto:path=/dkyges-49/?prefix=obj4 2024-04-12T03:47:33.399 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj4 2024-04-12T03:47:33.399 DEBUG:boto:Method: GET 2024-04-12T03:47:33.399 DEBUG:boto:Path: /dkyges-49/?prefix=obj4 2024-04-12T03:47:33.400 DEBUG:boto:Data: 2024-04-12T03:47:33.400 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.400 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.400 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.400 DEBUG:boto:Params: {} 2024-04-12T03:47:33.400 DEBUG:boto:Token: None 2024-04-12T03:47:33.400 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.400 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006360b017b4109b78-006618aed5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.403 DEBUG:boto:b'dkyges-49obj41000falseobj42024-04-12T03:46:13.731Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:47:33.404 DEBUG:boto:path=/dkyges-49/obj5 2024-04-12T03:47:33.404 DEBUG:boto:auth_path=/dkyges-49/obj5 2024-04-12T03:47:33.404 DEBUG:boto:path=/dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.404 DEBUG:boto:auth_path=/dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.404 DEBUG:boto:Method: GET 2024-04-12T03:47:33.404 DEBUG:boto:Path: /dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.404 DEBUG:boto:Data: 2024-04-12T03:47:33.404 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.404 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.404 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.405 DEBUG:boto:Params: {} 2024-04-12T03:47:33.405 DEBUG:boto:Token: None 2024-04-12T03:47:33.405 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.405 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VYlEkNkLyOEllSqTv8kiyCeSS/U= 2024-04-12T03:47:33.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VYlEkNkLyOEllSqTv8kiyCeSS/U=', 'Content-Length': '0'} 2024-04-12T03:47:33.407 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e73b04850a631cc3-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.407 DEBUG:boto:path=/dkyges-49/obj5 2024-04-12T03:47:33.407 DEBUG:boto:auth_path=/dkyges-49/obj5 2024-04-12T03:47:33.407 DEBUG:boto:path=/dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.407 DEBUG:boto:auth_path=/dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.407 DEBUG:boto:Method: GET 2024-04-12T03:47:33.407 DEBUG:boto:Path: /dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.407 DEBUG:boto:Data: 2024-04-12T03:47:33.408 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.408 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.408 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.408 DEBUG:boto:Params: {} 2024-04-12T03:47:33.408 DEBUG:boto:Token: None 2024-04-12T03:47:33.408 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.408 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VYlEkNkLyOEllSqTv8kiyCeSS/U= 2024-04-12T03:47:33.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VYlEkNkLyOEllSqTv8kiyCeSS/U=', 'Content-Length': '0'} 2024-04-12T03:47:33.410 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070c16871105bbf90-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.454 DEBUG:boto:path=/dkyges-49/obj5 2024-04-12T03:47:33.455 DEBUG:boto:auth_path=/dkyges-49/obj5 2024-04-12T03:47:33.455 DEBUG:boto:path=/dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.455 DEBUG:boto:auth_path=/dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.455 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.455 DEBUG:boto:Path: /dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.455 DEBUG:boto:Data: 2024-04-12T03:47:33.455 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.455 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.455 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.455 DEBUG:boto:Params: {} 2024-04-12T03:47:33.456 DEBUG:boto:Token: None 2024-04-12T03:47:33.456 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.456 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nx4YTi7utxallQ/UxaQ9K2oIU9c= 2024-04-12T03:47:33.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nx4YTi7utxallQ/UxaQ9K2oIU9c=', 'Content-Length': '0'} 2024-04-12T03:47:33.458 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d26cb8b775f6b3df-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.458 DEBUG:boto:path=/dkyges-49/obj5 2024-04-12T03:47:33.458 DEBUG:boto:auth_path=/dkyges-49/obj5 2024-04-12T03:47:33.458 DEBUG:boto:path=/dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.459 DEBUG:boto:auth_path=/dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.459 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.459 DEBUG:boto:Path: /dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.459 DEBUG:boto:Data: 2024-04-12T03:47:33.459 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.459 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.459 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.459 DEBUG:boto:Params: {} 2024-04-12T03:47:33.459 DEBUG:boto:Token: None 2024-04-12T03:47:33.459 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.459 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nx4YTi7utxallQ/UxaQ9K2oIU9c= 2024-04-12T03:47:33.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nx4YTi7utxallQ/UxaQ9K2oIU9c=', 'Content-Length': '0'} 2024-04-12T03:47:33.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a10623eeb040630-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:47:33.462 DEBUG:boto:path=/dkyges-49/obj5 2024-04-12T03:47:33.462 DEBUG:boto:auth_path=/dkyges-49/obj5 2024-04-12T03:47:33.462 DEBUG:boto:path=/dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.462 DEBUG:boto:auth_path=/dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.462 DEBUG:boto:Method: GET 2024-04-12T03:47:33.462 DEBUG:boto:Path: /dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.463 DEBUG:boto:Data: 2024-04-12T03:47:33.463 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.463 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.463 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.463 DEBUG:boto:Params: {} 2024-04-12T03:47:33.463 DEBUG:boto:Token: None 2024-04-12T03:47:33.463 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.463 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VYlEkNkLyOEllSqTv8kiyCeSS/U= 2024-04-12T03:47:33.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VYlEkNkLyOEllSqTv8kiyCeSS/U=', 'Content-Length': '0'} 2024-04-12T03:47:33.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e54e91e6fd631f1e-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.465 DEBUG:boto:path=/dkyges-49/obj5 2024-04-12T03:47:33.466 DEBUG:boto:auth_path=/dkyges-49/obj5 2024-04-12T03:47:33.466 DEBUG:boto:path=/dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.466 DEBUG:boto:auth_path=/dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.466 DEBUG:boto:Method: GET 2024-04-12T03:47:33.466 DEBUG:boto:Path: /dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.466 DEBUG:boto:Data: 2024-04-12T03:47:33.466 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.466 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.466 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.466 DEBUG:boto:Params: {} 2024-04-12T03:47:33.467 DEBUG:boto:Token: None 2024-04-12T03:47:33.467 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj5?versionId=null 2024-04-12T03:47:33.467 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VYlEkNkLyOEllSqTv8kiyCeSS/U= 2024-04-12T03:47:33.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VYlEkNkLyOEllSqTv8kiyCeSS/U=', 'Content-Length': '0'} 2024-04-12T03:47:33.469 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd2d7292f0c82d52-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.469 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.469 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.469 DEBUG:boto:path=/dkyges-49/?prefix=obj5 2024-04-12T03:47:33.469 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj5 2024-04-12T03:47:33.469 DEBUG:boto:Method: GET 2024-04-12T03:47:33.470 DEBUG:boto:Path: /dkyges-49/?prefix=obj5 2024-04-12T03:47:33.470 DEBUG:boto:Data: 2024-04-12T03:47:33.470 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.470 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.470 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.470 DEBUG:boto:Params: {} 2024-04-12T03:47:33.470 DEBUG:boto:Token: None 2024-04-12T03:47:33.470 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.470 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa01af5d813c4519-006618aed5-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.473 DEBUG:boto:b'dkyges-49obj51000falseobj52024-04-12T03:46:56.183Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.473 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.473 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.474 DEBUG:boto:path=/dkyges-49/?prefix=obj5 2024-04-12T03:47:33.474 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj5 2024-04-12T03:47:33.474 DEBUG:boto:Method: GET 2024-04-12T03:47:33.474 DEBUG:boto:Path: /dkyges-49/?prefix=obj5 2024-04-12T03:47:33.474 DEBUG:boto:Data: 2024-04-12T03:47:33.474 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.474 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.474 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.474 DEBUG:boto:Params: {} 2024-04-12T03:47:33.474 DEBUG:boto:Token: None 2024-04-12T03:47:33.475 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.475 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006018a59263db3d03-006618aed5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.477 DEBUG:boto:b'dkyges-49obj51000falseobj52024-04-12T03:46:56.183Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.478 DEBUG:boto:path=/dkyges-49/obj5 2024-04-12T03:47:33.478 DEBUG:boto:auth_path=/dkyges-49/obj5 2024-04-12T03:47:33.478 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.478 DEBUG:boto:Path: /dkyges-49/obj5 2024-04-12T03:47:33.478 DEBUG:boto:Data: 2024-04-12T03:47:33.478 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.478 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.478 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.478 DEBUG:boto:Params: {} 2024-04-12T03:47:33.478 DEBUG:boto:Token: None 2024-04-12T03:47:33.479 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj5 2024-04-12T03:47:33.479 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ka5d1jqXM6XT+wIasijfQekLQXg= 2024-04-12T03:47:33.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ka5d1jqXM6XT+wIasijfQekLQXg=', 'Content-Length': '0'} 2024-04-12T03:47:33.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dce2ecf7f8f0ce2-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.481 DEBUG:boto:path=/dkyges-49/obj5 2024-04-12T03:47:33.481 DEBUG:boto:auth_path=/dkyges-49/obj5 2024-04-12T03:47:33.481 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.481 DEBUG:boto:Path: /dkyges-49/obj5 2024-04-12T03:47:33.481 DEBUG:boto:Data: 2024-04-12T03:47:33.481 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.481 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.481 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.481 DEBUG:boto:Params: {} 2024-04-12T03:47:33.482 DEBUG:boto:Token: None 2024-04-12T03:47:33.482 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj5 2024-04-12T03:47:33.482 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ka5d1jqXM6XT+wIasijfQekLQXg= 2024-04-12T03:47:33.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ka5d1jqXM6XT+wIasijfQekLQXg=', 'Content-Length': '0'} 2024-04-12T03:47:33.484 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef017b10586acb90-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:47:33.484 DEBUG:boto:path=/dkyges-49/obj5 2024-04-12T03:47:33.484 DEBUG:boto:auth_path=/dkyges-49/obj5 2024-04-12T03:47:33.484 DEBUG:boto:Method: GET 2024-04-12T03:47:33.484 DEBUG:boto:Path: /dkyges-49/obj5 2024-04-12T03:47:33.484 DEBUG:boto:Data: 2024-04-12T03:47:33.485 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.485 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.485 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.485 DEBUG:boto:Params: {} 2024-04-12T03:47:33.485 DEBUG:boto:Token: None 2024-04-12T03:47:33.485 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj5 2024-04-12T03:47:33.485 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Cotl3SN3S/OKVOk4cFHe0GnPYHA= 2024-04-12T03:47:33.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Cotl3SN3S/OKVOk4cFHe0GnPYHA=', 'Content-Length': '0'} 2024-04-12T03:47:33.487 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010fa346cf1274192-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.487 DEBUG:boto:path=/dkyges-49/obj5 2024-04-12T03:47:33.487 DEBUG:boto:auth_path=/dkyges-49/obj5 2024-04-12T03:47:33.487 DEBUG:boto:Method: GET 2024-04-12T03:47:33.487 DEBUG:boto:Path: /dkyges-49/obj5 2024-04-12T03:47:33.487 DEBUG:boto:Data: 2024-04-12T03:47:33.488 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.488 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.488 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.488 DEBUG:boto:Params: {} 2024-04-12T03:47:33.488 DEBUG:boto:Token: None 2024-04-12T03:47:33.488 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj5 2024-04-12T03:47:33.488 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Cotl3SN3S/OKVOk4cFHe0GnPYHA= 2024-04-12T03:47:33.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Cotl3SN3S/OKVOk4cFHe0GnPYHA=', 'Content-Length': '0'} 2024-04-12T03:47:33.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044759a847811c49e-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.491 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.491 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.491 DEBUG:boto:path=/dkyges-49/?prefix=obj5 2024-04-12T03:47:33.491 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj5 2024-04-12T03:47:33.491 DEBUG:boto:Method: GET 2024-04-12T03:47:33.491 DEBUG:boto:Path: /dkyges-49/?prefix=obj5 2024-04-12T03:47:33.491 DEBUG:boto:Data: 2024-04-12T03:47:33.491 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.491 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.491 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.491 DEBUG:boto:Params: {} 2024-04-12T03:47:33.492 DEBUG:boto:Token: None 2024-04-12T03:47:33.492 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.492 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055639eb5a9041f25-006618aed5-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.494 DEBUG:boto:b'dkyges-49obj51000falseobj52024-04-12T03:46:56.183Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.495 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.495 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.495 DEBUG:boto:path=/dkyges-49/?prefix=obj5 2024-04-12T03:47:33.495 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj5 2024-04-12T03:47:33.495 DEBUG:boto:Method: GET 2024-04-12T03:47:33.495 DEBUG:boto:Path: /dkyges-49/?prefix=obj5 2024-04-12T03:47:33.495 DEBUG:boto:Data: 2024-04-12T03:47:33.495 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.495 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.495 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.495 DEBUG:boto:Params: {} 2024-04-12T03:47:33.496 DEBUG:boto:Token: None 2024-04-12T03:47:33.496 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.496 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a8796d24d611646-006618aed5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.499 DEBUG:boto:b'dkyges-49obj51000falseobj52024-04-12T03:46:56.183Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:47:33.499 DEBUG:boto:path=/dkyges-49/obj6 2024-04-12T03:47:33.499 DEBUG:boto:auth_path=/dkyges-49/obj6 2024-04-12T03:47:33.499 DEBUG:boto:path=/dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.499 DEBUG:boto:auth_path=/dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.499 DEBUG:boto:Method: GET 2024-04-12T03:47:33.499 DEBUG:boto:Path: /dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.499 DEBUG:boto:Data: 2024-04-12T03:47:33.500 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.500 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.500 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.500 DEBUG:boto:Params: {} 2024-04-12T03:47:33.500 DEBUG:boto:Token: None 2024-04-12T03:47:33.500 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.500 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LVwIHFPxLRfVZ5dwBVF3fWkeSWw= 2024-04-12T03:47:33.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LVwIHFPxLRfVZ5dwBVF3fWkeSWw=', 'Content-Length': '0'} 2024-04-12T03:47:33.502 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ef8cc6584840b60-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.542 DEBUG:boto:path=/dkyges-49/obj6 2024-04-12T03:47:33.543 DEBUG:boto:auth_path=/dkyges-49/obj6 2024-04-12T03:47:33.543 DEBUG:boto:path=/dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.543 DEBUG:boto:auth_path=/dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.543 DEBUG:boto:Method: GET 2024-04-12T03:47:33.543 DEBUG:boto:Path: /dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.543 DEBUG:boto:Data: 2024-04-12T03:47:33.543 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.543 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.543 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.543 DEBUG:boto:Params: {} 2024-04-12T03:47:33.544 DEBUG:boto:Token: None 2024-04-12T03:47:33.544 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.544 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LVwIHFPxLRfVZ5dwBVF3fWkeSWw= 2024-04-12T03:47:33.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LVwIHFPxLRfVZ5dwBVF3fWkeSWw=', 'Content-Length': '0'} 2024-04-12T03:47:33.546 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e6d94c7ded0c144-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.546 DEBUG:boto:path=/dkyges-49/obj6 2024-04-12T03:47:33.546 DEBUG:boto:auth_path=/dkyges-49/obj6 2024-04-12T03:47:33.547 DEBUG:boto:path=/dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.547 DEBUG:boto:auth_path=/dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.547 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.547 DEBUG:boto:Path: /dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.547 DEBUG:boto:Data: 2024-04-12T03:47:33.547 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.547 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.547 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.547 DEBUG:boto:Params: {} 2024-04-12T03:47:33.547 DEBUG:boto:Token: None 2024-04-12T03:47:33.547 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.548 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GQSKJL1QlqkxvAsN9qkP9uoSuo4= 2024-04-12T03:47:33.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GQSKJL1QlqkxvAsN9qkP9uoSuo4=', 'Content-Length': '0'} 2024-04-12T03:47:33.550 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b94c9355bfa50f8-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.550 DEBUG:boto:path=/dkyges-49/obj6 2024-04-12T03:47:33.550 DEBUG:boto:auth_path=/dkyges-49/obj6 2024-04-12T03:47:33.550 DEBUG:boto:path=/dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.550 DEBUG:boto:auth_path=/dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.551 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.551 DEBUG:boto:Path: /dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.551 DEBUG:boto:Data: 2024-04-12T03:47:33.551 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.551 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.551 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.551 DEBUG:boto:Params: {} 2024-04-12T03:47:33.551 DEBUG:boto:Token: None 2024-04-12T03:47:33.552 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.552 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GQSKJL1QlqkxvAsN9qkP9uoSuo4= 2024-04-12T03:47:33.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GQSKJL1QlqkxvAsN9qkP9uoSuo4=', 'Content-Length': '0'} 2024-04-12T03:47:33.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2b721005ed696be-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:47:33.554 DEBUG:boto:path=/dkyges-49/obj6 2024-04-12T03:47:33.554 DEBUG:boto:auth_path=/dkyges-49/obj6 2024-04-12T03:47:33.554 DEBUG:boto:path=/dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.554 DEBUG:boto:auth_path=/dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.555 DEBUG:boto:Method: GET 2024-04-12T03:47:33.555 DEBUG:boto:Path: /dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.555 DEBUG:boto:Data: 2024-04-12T03:47:33.555 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.555 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.555 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.555 DEBUG:boto:Params: {} 2024-04-12T03:47:33.555 DEBUG:boto:Token: None 2024-04-12T03:47:33.555 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.555 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LVwIHFPxLRfVZ5dwBVF3fWkeSWw= 2024-04-12T03:47:33.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LVwIHFPxLRfVZ5dwBVF3fWkeSWw=', 'Content-Length': '0'} 2024-04-12T03:47:33.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f4de7944a084cf6-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.558 DEBUG:boto:path=/dkyges-49/obj6 2024-04-12T03:47:33.558 DEBUG:boto:auth_path=/dkyges-49/obj6 2024-04-12T03:47:33.558 DEBUG:boto:path=/dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.558 DEBUG:boto:auth_path=/dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.558 DEBUG:boto:Method: GET 2024-04-12T03:47:33.558 DEBUG:boto:Path: /dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.558 DEBUG:boto:Data: 2024-04-12T03:47:33.559 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.559 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.559 DEBUG:boto:Params: {} 2024-04-12T03:47:33.559 DEBUG:boto:Token: None 2024-04-12T03:47:33.559 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj6?versionId=null 2024-04-12T03:47:33.559 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LVwIHFPxLRfVZ5dwBVF3fWkeSWw= 2024-04-12T03:47:33.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LVwIHFPxLRfVZ5dwBVF3fWkeSWw=', 'Content-Length': '0'} 2024-04-12T03:47:33.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073b6f9b114ab7f19-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.562 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.562 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.562 DEBUG:boto:path=/dkyges-49/?prefix=obj6 2024-04-12T03:47:33.562 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj6 2024-04-12T03:47:33.563 DEBUG:boto:Method: GET 2024-04-12T03:47:33.563 DEBUG:boto:Path: /dkyges-49/?prefix=obj6 2024-04-12T03:47:33.563 DEBUG:boto:Data: 2024-04-12T03:47:33.563 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.563 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.563 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.563 DEBUG:boto:Params: {} 2024-04-12T03:47:33.563 DEBUG:boto:Token: None 2024-04-12T03:47:33.564 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.564 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054b14f3eb6cf82ae-006618aed5-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.567 DEBUG:boto:b'dkyges-49obj61000falseobj62024-04-12T03:46:56.194Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.567 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.567 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.567 DEBUG:boto:path=/dkyges-49/?prefix=obj6 2024-04-12T03:47:33.567 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj6 2024-04-12T03:47:33.567 DEBUG:boto:Method: GET 2024-04-12T03:47:33.567 DEBUG:boto:Path: /dkyges-49/?prefix=obj6 2024-04-12T03:47:33.568 DEBUG:boto:Data: 2024-04-12T03:47:33.568 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.568 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.568 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.568 DEBUG:boto:Params: {} 2024-04-12T03:47:33.568 DEBUG:boto:Token: None 2024-04-12T03:47:33.568 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.568 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007310dfe5e57528c1-006618aed5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.571 DEBUG:boto:b'dkyges-49obj61000falseobj62024-04-12T03:46:56.194Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.571 DEBUG:boto:path=/dkyges-49/obj6 2024-04-12T03:47:33.572 DEBUG:boto:auth_path=/dkyges-49/obj6 2024-04-12T03:47:33.572 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.572 DEBUG:boto:Path: /dkyges-49/obj6 2024-04-12T03:47:33.572 DEBUG:boto:Data: 2024-04-12T03:47:33.572 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.572 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.572 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.572 DEBUG:boto:Params: {} 2024-04-12T03:47:33.572 DEBUG:boto:Token: None 2024-04-12T03:47:33.573 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj6 2024-04-12T03:47:33.573 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oLVTDB/Uaoa9iQTNP4dgcpRbZq4= 2024-04-12T03:47:33.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oLVTDB/Uaoa9iQTNP4dgcpRbZq4=', 'Content-Length': '0'} 2024-04-12T03:47:33.574 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9c597018f66dacc-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.574 DEBUG:boto:path=/dkyges-49/obj6 2024-04-12T03:47:33.575 DEBUG:boto:auth_path=/dkyges-49/obj6 2024-04-12T03:47:33.575 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.575 DEBUG:boto:Path: /dkyges-49/obj6 2024-04-12T03:47:33.575 DEBUG:boto:Data: 2024-04-12T03:47:33.575 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.575 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.575 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.575 DEBUG:boto:Params: {} 2024-04-12T03:47:33.575 DEBUG:boto:Token: None 2024-04-12T03:47:33.575 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj6 2024-04-12T03:47:33.576 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oLVTDB/Uaoa9iQTNP4dgcpRbZq4= 2024-04-12T03:47:33.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oLVTDB/Uaoa9iQTNP4dgcpRbZq4=', 'Content-Length': '0'} 2024-04-12T03:47:33.577 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b97d099d1a132b7b-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:47:33.578 DEBUG:boto:path=/dkyges-49/obj6 2024-04-12T03:47:33.578 DEBUG:boto:auth_path=/dkyges-49/obj6 2024-04-12T03:47:33.578 DEBUG:boto:Method: GET 2024-04-12T03:47:33.578 DEBUG:boto:Path: /dkyges-49/obj6 2024-04-12T03:47:33.578 DEBUG:boto:Data: 2024-04-12T03:47:33.578 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.578 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.579 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.579 DEBUG:boto:Params: {} 2024-04-12T03:47:33.579 DEBUG:boto:Token: None 2024-04-12T03:47:33.579 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj6 2024-04-12T03:47:33.579 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DzrjOEyMtMIjA/OgjhqINpO6FSg= 2024-04-12T03:47:33.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DzrjOEyMtMIjA/OgjhqINpO6FSg=', 'Content-Length': '0'} 2024-04-12T03:47:33.581 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007903283a988154c7-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.622 DEBUG:boto:path=/dkyges-49/obj6 2024-04-12T03:47:33.622 DEBUG:boto:auth_path=/dkyges-49/obj6 2024-04-12T03:47:33.623 DEBUG:boto:Method: GET 2024-04-12T03:47:33.623 DEBUG:boto:Path: /dkyges-49/obj6 2024-04-12T03:47:33.623 DEBUG:boto:Data: 2024-04-12T03:47:33.623 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.623 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.623 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.623 DEBUG:boto:Params: {} 2024-04-12T03:47:33.623 DEBUG:boto:Token: None 2024-04-12T03:47:33.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj6 2024-04-12T03:47:33.624 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DzrjOEyMtMIjA/OgjhqINpO6FSg= 2024-04-12T03:47:33.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DzrjOEyMtMIjA/OgjhqINpO6FSg=', 'Content-Length': '0'} 2024-04-12T03:47:33.625 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002143484b0b71ec18-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.626 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.626 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.626 DEBUG:boto:path=/dkyges-49/?prefix=obj6 2024-04-12T03:47:33.626 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj6 2024-04-12T03:47:33.626 DEBUG:boto:Method: GET 2024-04-12T03:47:33.626 DEBUG:boto:Path: /dkyges-49/?prefix=obj6 2024-04-12T03:47:33.626 DEBUG:boto:Data: 2024-04-12T03:47:33.626 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.627 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.627 DEBUG:boto:Params: {} 2024-04-12T03:47:33.627 DEBUG:boto:Token: None 2024-04-12T03:47:33.627 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.627 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e67aab18e506bd20-006618aed5-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.630 DEBUG:boto:b'dkyges-49obj61000falseobj62024-04-12T03:46:56.194Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.630 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.630 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.631 DEBUG:boto:path=/dkyges-49/?prefix=obj6 2024-04-12T03:47:33.631 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj6 2024-04-12T03:47:33.631 DEBUG:boto:Method: GET 2024-04-12T03:47:33.631 DEBUG:boto:Path: /dkyges-49/?prefix=obj6 2024-04-12T03:47:33.631 DEBUG:boto:Data: 2024-04-12T03:47:33.631 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.631 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.631 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.631 DEBUG:boto:Params: {} 2024-04-12T03:47:33.631 DEBUG:boto:Token: None 2024-04-12T03:47:33.631 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.632 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.634 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046ff57ef2fd00781-006618aed5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.634 DEBUG:boto:b'dkyges-49obj61000falseobj62024-04-12T03:46:56.194Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:47:33.635 DEBUG:boto:path=/dkyges-49/obj7 2024-04-12T03:47:33.635 DEBUG:boto:auth_path=/dkyges-49/obj7 2024-04-12T03:47:33.635 DEBUG:boto:path=/dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.635 DEBUG:boto:auth_path=/dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.635 DEBUG:boto:Method: GET 2024-04-12T03:47:33.635 DEBUG:boto:Path: /dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.635 DEBUG:boto:Data: 2024-04-12T03:47:33.636 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.636 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.636 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.636 DEBUG:boto:Params: {} 2024-04-12T03:47:33.636 DEBUG:boto:Token: None 2024-04-12T03:47:33.636 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.636 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:flHsuqRoeM9+PFlfbErVx0JhOkg= 2024-04-12T03:47:33.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:flHsuqRoeM9+PFlfbErVx0JhOkg=', 'Content-Length': '0'} 2024-04-12T03:47:33.638 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fb624d35f814f00-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.638 DEBUG:boto:path=/dkyges-49/obj7 2024-04-12T03:47:33.639 DEBUG:boto:auth_path=/dkyges-49/obj7 2024-04-12T03:47:33.639 DEBUG:boto:path=/dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.639 DEBUG:boto:auth_path=/dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.639 DEBUG:boto:Method: GET 2024-04-12T03:47:33.639 DEBUG:boto:Path: /dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.639 DEBUG:boto:Data: 2024-04-12T03:47:33.639 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.639 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.639 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.639 DEBUG:boto:Params: {} 2024-04-12T03:47:33.639 DEBUG:boto:Token: None 2024-04-12T03:47:33.640 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.640 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:flHsuqRoeM9+PFlfbErVx0JhOkg= 2024-04-12T03:47:33.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:flHsuqRoeM9+PFlfbErVx0JhOkg=', 'Content-Length': '0'} 2024-04-12T03:47:33.642 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2a60076563c2e50-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.642 DEBUG:boto:path=/dkyges-49/obj7 2024-04-12T03:47:33.642 DEBUG:boto:auth_path=/dkyges-49/obj7 2024-04-12T03:47:33.642 DEBUG:boto:path=/dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.642 DEBUG:boto:auth_path=/dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.642 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.643 DEBUG:boto:Path: /dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.643 DEBUG:boto:Data: 2024-04-12T03:47:33.643 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.643 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.643 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.643 DEBUG:boto:Params: {} 2024-04-12T03:47:33.643 DEBUG:boto:Token: None 2024-04-12T03:47:33.643 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.643 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EsKmpl78jZm6OYoOuvq5i3F40Xk= 2024-04-12T03:47:33.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EsKmpl78jZm6OYoOuvq5i3F40Xk=', 'Content-Length': '0'} 2024-04-12T03:47:33.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acc964ad80bea113-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.645 DEBUG:boto:path=/dkyges-49/obj7 2024-04-12T03:47:33.645 DEBUG:boto:auth_path=/dkyges-49/obj7 2024-04-12T03:47:33.645 DEBUG:boto:path=/dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.646 DEBUG:boto:auth_path=/dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.646 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.646 DEBUG:boto:Path: /dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.646 DEBUG:boto:Data: 2024-04-12T03:47:33.646 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.646 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.646 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.646 DEBUG:boto:Params: {} 2024-04-12T03:47:33.646 DEBUG:boto:Token: None 2024-04-12T03:47:33.646 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.647 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EsKmpl78jZm6OYoOuvq5i3F40Xk= 2024-04-12T03:47:33.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EsKmpl78jZm6OYoOuvq5i3F40Xk=', 'Content-Length': '0'} 2024-04-12T03:47:33.649 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e1bfb6e8dc079266-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:47:33.649 DEBUG:boto:path=/dkyges-49/obj7 2024-04-12T03:47:33.649 DEBUG:boto:auth_path=/dkyges-49/obj7 2024-04-12T03:47:33.649 DEBUG:boto:path=/dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.649 DEBUG:boto:auth_path=/dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.649 DEBUG:boto:Method: GET 2024-04-12T03:47:33.650 DEBUG:boto:Path: /dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.650 DEBUG:boto:Data: 2024-04-12T03:47:33.650 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.650 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.650 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.650 DEBUG:boto:Params: {} 2024-04-12T03:47:33.650 DEBUG:boto:Token: None 2024-04-12T03:47:33.650 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.650 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:flHsuqRoeM9+PFlfbErVx0JhOkg= 2024-04-12T03:47:33.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:flHsuqRoeM9+PFlfbErVx0JhOkg=', 'Content-Length': '0'} 2024-04-12T03:47:33.652 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006df38ae1e879845a-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.653 DEBUG:boto:path=/dkyges-49/obj7 2024-04-12T03:47:33.653 DEBUG:boto:auth_path=/dkyges-49/obj7 2024-04-12T03:47:33.653 DEBUG:boto:path=/dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.653 DEBUG:boto:auth_path=/dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.653 DEBUG:boto:Method: GET 2024-04-12T03:47:33.653 DEBUG:boto:Path: /dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.653 DEBUG:boto:Data: 2024-04-12T03:47:33.653 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.653 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.653 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.653 DEBUG:boto:Params: {} 2024-04-12T03:47:33.654 DEBUG:boto:Token: None 2024-04-12T03:47:33.654 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj7?versionId=null 2024-04-12T03:47:33.654 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:flHsuqRoeM9+PFlfbErVx0JhOkg= 2024-04-12T03:47:33.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:flHsuqRoeM9+PFlfbErVx0JhOkg=', 'Content-Length': '0'} 2024-04-12T03:47:33.656 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001234ec95d85d79af-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.656 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.656 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.656 DEBUG:boto:path=/dkyges-49/?prefix=obj7 2024-04-12T03:47:33.656 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj7 2024-04-12T03:47:33.657 DEBUG:boto:Method: GET 2024-04-12T03:47:33.657 DEBUG:boto:Path: /dkyges-49/?prefix=obj7 2024-04-12T03:47:33.657 DEBUG:boto:Data: 2024-04-12T03:47:33.657 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.657 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.657 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.657 DEBUG:boto:Params: {} 2024-04-12T03:47:33.657 DEBUG:boto:Token: None 2024-04-12T03:47:33.657 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.657 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086b2590d3321ba76-006618aed5-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.660 DEBUG:boto:b'dkyges-49obj71000falseobj72024-04-12T03:46:56.204Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.661 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.661 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.661 DEBUG:boto:path=/dkyges-49/?prefix=obj7 2024-04-12T03:47:33.661 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj7 2024-04-12T03:47:33.661 DEBUG:boto:Method: GET 2024-04-12T03:47:33.661 DEBUG:boto:Path: /dkyges-49/?prefix=obj7 2024-04-12T03:47:33.661 DEBUG:boto:Data: 2024-04-12T03:47:33.661 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.661 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.661 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.661 DEBUG:boto:Params: {} 2024-04-12T03:47:33.661 DEBUG:boto:Token: None 2024-04-12T03:47:33.662 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.662 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e59ac1050f851818-006618aed5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.664 DEBUG:boto:b'dkyges-49obj71000falseobj72024-04-12T03:46:56.204Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.665 DEBUG:boto:path=/dkyges-49/obj7 2024-04-12T03:47:33.665 DEBUG:boto:auth_path=/dkyges-49/obj7 2024-04-12T03:47:33.665 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.665 DEBUG:boto:Path: /dkyges-49/obj7 2024-04-12T03:47:33.665 DEBUG:boto:Data: 2024-04-12T03:47:33.665 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.665 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.665 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.665 DEBUG:boto:Params: {} 2024-04-12T03:47:33.666 DEBUG:boto:Token: None 2024-04-12T03:47:33.666 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj7 2024-04-12T03:47:33.666 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WF/t9cotSKvJYPComa82f2drXLA= 2024-04-12T03:47:33.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WF/t9cotSKvJYPComa82f2drXLA=', 'Content-Length': '0'} 2024-04-12T03:47:33.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007fb8b33b6f1d0cf3-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.668 DEBUG:boto:path=/dkyges-49/obj7 2024-04-12T03:47:33.668 DEBUG:boto:auth_path=/dkyges-49/obj7 2024-04-12T03:47:33.668 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.668 DEBUG:boto:Path: /dkyges-49/obj7 2024-04-12T03:47:33.668 DEBUG:boto:Data: 2024-04-12T03:47:33.668 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.668 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.668 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.668 DEBUG:boto:Params: {} 2024-04-12T03:47:33.669 DEBUG:boto:Token: None 2024-04-12T03:47:33.669 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj7 2024-04-12T03:47:33.669 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WF/t9cotSKvJYPComa82f2drXLA= 2024-04-12T03:47:33.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WF/t9cotSKvJYPComa82f2drXLA=', 'Content-Length': '0'} 2024-04-12T03:47:33.671 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ed9761bf24e4273-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:47:33.671 DEBUG:boto:path=/dkyges-49/obj7 2024-04-12T03:47:33.671 DEBUG:boto:auth_path=/dkyges-49/obj7 2024-04-12T03:47:33.671 DEBUG:boto:Method: GET 2024-04-12T03:47:33.671 DEBUG:boto:Path: /dkyges-49/obj7 2024-04-12T03:47:33.672 DEBUG:boto:Data: 2024-04-12T03:47:33.672 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.672 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.672 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.672 DEBUG:boto:Params: {} 2024-04-12T03:47:33.672 DEBUG:boto:Token: None 2024-04-12T03:47:33.672 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj7 2024-04-12T03:47:33.672 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BPP61pJFzfmsGCxey79x/CERSVE= 2024-04-12T03:47:33.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BPP61pJFzfmsGCxey79x/CERSVE=', 'Content-Length': '0'} 2024-04-12T03:47:33.674 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea94339ca543c37d-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.714 DEBUG:boto:path=/dkyges-49/obj7 2024-04-12T03:47:33.714 DEBUG:boto:auth_path=/dkyges-49/obj7 2024-04-12T03:47:33.715 DEBUG:boto:Method: GET 2024-04-12T03:47:33.715 DEBUG:boto:Path: /dkyges-49/obj7 2024-04-12T03:47:33.715 DEBUG:boto:Data: 2024-04-12T03:47:33.715 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.715 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.715 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.715 DEBUG:boto:Params: {} 2024-04-12T03:47:33.715 DEBUG:boto:Token: None 2024-04-12T03:47:33.715 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj7 2024-04-12T03:47:33.715 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BPP61pJFzfmsGCxey79x/CERSVE= 2024-04-12T03:47:33.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BPP61pJFzfmsGCxey79x/CERSVE=', 'Content-Length': '0'} 2024-04-12T03:47:33.717 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad470af571278a64-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.718 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.718 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.718 DEBUG:boto:path=/dkyges-49/?prefix=obj7 2024-04-12T03:47:33.718 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj7 2024-04-12T03:47:33.718 DEBUG:boto:Method: GET 2024-04-12T03:47:33.718 DEBUG:boto:Path: /dkyges-49/?prefix=obj7 2024-04-12T03:47:33.718 DEBUG:boto:Data: 2024-04-12T03:47:33.718 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.719 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.719 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.719 DEBUG:boto:Params: {} 2024-04-12T03:47:33.719 DEBUG:boto:Token: None 2024-04-12T03:47:33.719 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.719 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee80090275a2a8ff-006618aed5-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.722 DEBUG:boto:b'dkyges-49obj71000falseobj72024-04-12T03:46:56.204Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.722 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.722 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.722 DEBUG:boto:path=/dkyges-49/?prefix=obj7 2024-04-12T03:47:33.722 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj7 2024-04-12T03:47:33.722 DEBUG:boto:Method: GET 2024-04-12T03:47:33.723 DEBUG:boto:Path: /dkyges-49/?prefix=obj7 2024-04-12T03:47:33.723 DEBUG:boto:Data: 2024-04-12T03:47:33.723 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.723 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.723 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.723 DEBUG:boto:Params: {} 2024-04-12T03:47:33.723 DEBUG:boto:Token: None 2024-04-12T03:47:33.723 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.723 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2a27812c66b32ea-006618aed5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.726 DEBUG:boto:b'dkyges-49obj71000falseobj72024-04-12T03:46:56.204Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:47:33.727 DEBUG:boto:path=/dkyges-49/obj8 2024-04-12T03:47:33.727 DEBUG:boto:auth_path=/dkyges-49/obj8 2024-04-12T03:47:33.727 DEBUG:boto:path=/dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.727 DEBUG:boto:auth_path=/dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.727 DEBUG:boto:Method: GET 2024-04-12T03:47:33.727 DEBUG:boto:Path: /dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.727 DEBUG:boto:Data: 2024-04-12T03:47:33.727 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.727 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.727 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.727 DEBUG:boto:Params: {} 2024-04-12T03:47:33.728 DEBUG:boto:Token: None 2024-04-12T03:47:33.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.728 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0uqNygTLPts7d3qUprfic/J+4u8= 2024-04-12T03:47:33.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0uqNygTLPts7d3qUprfic/J+4u8=', 'Content-Length': '0'} 2024-04-12T03:47:33.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000270c483e9285ca9-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.730 DEBUG:boto:path=/dkyges-49/obj8 2024-04-12T03:47:33.730 DEBUG:boto:auth_path=/dkyges-49/obj8 2024-04-12T03:47:33.730 DEBUG:boto:path=/dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.730 DEBUG:boto:auth_path=/dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.730 DEBUG:boto:Method: GET 2024-04-12T03:47:33.730 DEBUG:boto:Path: /dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.730 DEBUG:boto:Data: 2024-04-12T03:47:33.731 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.731 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.731 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.731 DEBUG:boto:Params: {} 2024-04-12T03:47:33.731 DEBUG:boto:Token: None 2024-04-12T03:47:33.731 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.731 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0uqNygTLPts7d3qUprfic/J+4u8= 2024-04-12T03:47:33.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0uqNygTLPts7d3qUprfic/J+4u8=', 'Content-Length': '0'} 2024-04-12T03:47:33.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004a8ffaed3b07a5e-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.734 DEBUG:boto:path=/dkyges-49/obj8 2024-04-12T03:47:33.734 DEBUG:boto:auth_path=/dkyges-49/obj8 2024-04-12T03:47:33.734 DEBUG:boto:path=/dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.734 DEBUG:boto:auth_path=/dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.734 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.734 DEBUG:boto:Path: /dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.734 DEBUG:boto:Data: 2024-04-12T03:47:33.734 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.734 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.734 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.735 DEBUG:boto:Params: {} 2024-04-12T03:47:33.735 DEBUG:boto:Token: None 2024-04-12T03:47:33.735 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.735 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r77tS5ir3MhczszCOdAEnqtnWH8= 2024-04-12T03:47:33.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r77tS5ir3MhczszCOdAEnqtnWH8=', 'Content-Length': '0'} 2024-04-12T03:47:33.737 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fd5a7ff08bab0a9-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.737 DEBUG:boto:path=/dkyges-49/obj8 2024-04-12T03:47:33.737 DEBUG:boto:auth_path=/dkyges-49/obj8 2024-04-12T03:47:33.737 DEBUG:boto:path=/dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.737 DEBUG:boto:auth_path=/dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.737 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.737 DEBUG:boto:Path: /dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.738 DEBUG:boto:Data: 2024-04-12T03:47:33.738 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.738 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.738 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.738 DEBUG:boto:Params: {} 2024-04-12T03:47:33.738 DEBUG:boto:Token: None 2024-04-12T03:47:33.738 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.738 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r77tS5ir3MhczszCOdAEnqtnWH8= 2024-04-12T03:47:33.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r77tS5ir3MhczszCOdAEnqtnWH8=', 'Content-Length': '0'} 2024-04-12T03:47:33.740 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e316337280c231aa-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:47:33.741 DEBUG:boto:path=/dkyges-49/obj8 2024-04-12T03:47:33.741 DEBUG:boto:auth_path=/dkyges-49/obj8 2024-04-12T03:47:33.741 DEBUG:boto:path=/dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.741 DEBUG:boto:auth_path=/dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.741 DEBUG:boto:Method: GET 2024-04-12T03:47:33.741 DEBUG:boto:Path: /dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.741 DEBUG:boto:Data: 2024-04-12T03:47:33.741 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.741 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.741 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.741 DEBUG:boto:Params: {} 2024-04-12T03:47:33.742 DEBUG:boto:Token: None 2024-04-12T03:47:33.742 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.742 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0uqNygTLPts7d3qUprfic/J+4u8= 2024-04-12T03:47:33.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0uqNygTLPts7d3qUprfic/J+4u8=', 'Content-Length': '0'} 2024-04-12T03:47:33.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ec6fdc0195ff0e6-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.744 DEBUG:boto:path=/dkyges-49/obj8 2024-04-12T03:47:33.744 DEBUG:boto:auth_path=/dkyges-49/obj8 2024-04-12T03:47:33.744 DEBUG:boto:path=/dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.744 DEBUG:boto:auth_path=/dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.744 DEBUG:boto:Method: GET 2024-04-12T03:47:33.745 DEBUG:boto:Path: /dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.745 DEBUG:boto:Data: 2024-04-12T03:47:33.745 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.745 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.745 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.745 DEBUG:boto:Params: {} 2024-04-12T03:47:33.745 DEBUG:boto:Token: None 2024-04-12T03:47:33.745 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj8?versionId=null 2024-04-12T03:47:33.745 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0uqNygTLPts7d3qUprfic/J+4u8= 2024-04-12T03:47:33.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0uqNygTLPts7d3qUprfic/J+4u8=', 'Content-Length': '0'} 2024-04-12T03:47:33.747 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002dc6e85e0467f76c-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.747 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.747 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.747 DEBUG:boto:path=/dkyges-49/?prefix=obj8 2024-04-12T03:47:33.748 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj8 2024-04-12T03:47:33.748 DEBUG:boto:Method: GET 2024-04-12T03:47:33.748 DEBUG:boto:Path: /dkyges-49/?prefix=obj8 2024-04-12T03:47:33.748 DEBUG:boto:Data: 2024-04-12T03:47:33.748 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.748 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.748 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.748 DEBUG:boto:Params: {} 2024-04-12T03:47:33.748 DEBUG:boto:Token: None 2024-04-12T03:47:33.748 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.749 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000312fba97a4482c0a-006618aed5-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.751 DEBUG:boto:b'dkyges-49obj81000falseobj82024-04-12T03:46:56.252Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.751 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.751 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.752 DEBUG:boto:path=/dkyges-49/?prefix=obj8 2024-04-12T03:47:33.752 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj8 2024-04-12T03:47:33.752 DEBUG:boto:Method: GET 2024-04-12T03:47:33.752 DEBUG:boto:Path: /dkyges-49/?prefix=obj8 2024-04-12T03:47:33.752 DEBUG:boto:Data: 2024-04-12T03:47:33.752 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.752 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.752 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.752 DEBUG:boto:Params: {} 2024-04-12T03:47:33.752 DEBUG:boto:Token: None 2024-04-12T03:47:33.752 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.753 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7e7f7ad7ccebe31-006618aed5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.798 DEBUG:boto:b'dkyges-49obj81000falseobj82024-04-12T03:46:56.252Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.799 DEBUG:boto:path=/dkyges-49/obj8 2024-04-12T03:47:33.799 DEBUG:boto:auth_path=/dkyges-49/obj8 2024-04-12T03:47:33.799 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.799 DEBUG:boto:Path: /dkyges-49/obj8 2024-04-12T03:47:33.799 DEBUG:boto:Data: 2024-04-12T03:47:33.799 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.799 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.799 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.799 DEBUG:boto:Params: {} 2024-04-12T03:47:33.800 DEBUG:boto:Token: None 2024-04-12T03:47:33.800 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj8 2024-04-12T03:47:33.800 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NVwhH9yR4M2ZUK9+gQ0ERjyFhj8= 2024-04-12T03:47:33.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NVwhH9yR4M2ZUK9+gQ0ERjyFhj8=', 'Content-Length': '0'} 2024-04-12T03:47:33.802 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a8521ec32d7d6735-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.802 DEBUG:boto:path=/dkyges-49/obj8 2024-04-12T03:47:33.802 DEBUG:boto:auth_path=/dkyges-49/obj8 2024-04-12T03:47:33.802 DEBUG:boto:Method: HEAD 2024-04-12T03:47:33.802 DEBUG:boto:Path: /dkyges-49/obj8 2024-04-12T03:47:33.802 DEBUG:boto:Data: 2024-04-12T03:47:33.802 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.802 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.803 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.803 DEBUG:boto:Params: {} 2024-04-12T03:47:33.803 DEBUG:boto:Token: None 2024-04-12T03:47:33.803 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj8 2024-04-12T03:47:33.803 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NVwhH9yR4M2ZUK9+gQ0ERjyFhj8= 2024-04-12T03:47:33.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NVwhH9yR4M2ZUK9+gQ0ERjyFhj8=', 'Content-Length': '0'} 2024-04-12T03:47:33.805 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c599bd334a4524ea-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:47:33.805 DEBUG:boto:path=/dkyges-49/obj8 2024-04-12T03:47:33.805 DEBUG:boto:auth_path=/dkyges-49/obj8 2024-04-12T03:47:33.805 DEBUG:boto:Method: GET 2024-04-12T03:47:33.806 DEBUG:boto:Path: /dkyges-49/obj8 2024-04-12T03:47:33.806 DEBUG:boto:Data: 2024-04-12T03:47:33.806 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.806 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.806 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.806 DEBUG:boto:Params: {} 2024-04-12T03:47:33.806 DEBUG:boto:Token: None 2024-04-12T03:47:33.806 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj8 2024-04-12T03:47:33.806 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AHi+DPoA6WEIEZyKAK4legfJPeg= 2024-04-12T03:47:33.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AHi+DPoA6WEIEZyKAK4legfJPeg=', 'Content-Length': '0'} 2024-04-12T03:47:33.808 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7612a487abcbfde-006618aed5-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.808 DEBUG:boto:path=/dkyges-49/obj8 2024-04-12T03:47:33.808 DEBUG:boto:auth_path=/dkyges-49/obj8 2024-04-12T03:47:33.809 DEBUG:boto:Method: GET 2024-04-12T03:47:33.809 DEBUG:boto:Path: /dkyges-49/obj8 2024-04-12T03:47:33.809 DEBUG:boto:Data: 2024-04-12T03:47:33.809 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.809 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.809 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.809 DEBUG:boto:Params: {} 2024-04-12T03:47:33.809 DEBUG:boto:Token: None 2024-04-12T03:47:33.809 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/obj8 2024-04-12T03:47:33.809 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AHi+DPoA6WEIEZyKAK4legfJPeg= 2024-04-12T03:47:33.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AHi+DPoA6WEIEZyKAK4legfJPeg=', 'Content-Length': '0'} 2024-04-12T03:47:33.811 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:46:56 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-49:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffeda5ca968ca655-006618aed5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.812 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.812 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.812 DEBUG:boto:path=/dkyges-49/?prefix=obj8 2024-04-12T03:47:33.812 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj8 2024-04-12T03:47:33.812 DEBUG:boto:Method: GET 2024-04-12T03:47:33.812 DEBUG:boto:Path: /dkyges-49/?prefix=obj8 2024-04-12T03:47:33.812 DEBUG:boto:Data: 2024-04-12T03:47:33.812 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.812 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:33.812 DEBUG:boto:Port: 8001 2024-04-12T03:47:33.813 DEBUG:boto:Params: {} 2024-04-12T03:47:33.813 DEBUG:boto:Token: None 2024-04-12T03:47:33.813 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.813 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c2a411fa38b4653-006618aed5-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.816 DEBUG:boto:b'dkyges-49obj81000falseobj82024-04-12T03:46:56.252Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.816 DEBUG:boto:path=/dkyges-49/ 2024-04-12T03:47:33.816 DEBUG:boto:auth_path=/dkyges-49/ 2024-04-12T03:47:33.816 DEBUG:boto:path=/dkyges-49/?prefix=obj8 2024-04-12T03:47:33.817 DEBUG:boto:auth_path=/dkyges-49/?prefix=obj8 2024-04-12T03:47:33.817 DEBUG:boto:Method: GET 2024-04-12T03:47:33.817 DEBUG:boto:Path: /dkyges-49/?prefix=obj8 2024-04-12T03:47:33.817 DEBUG:boto:Data: 2024-04-12T03:47:33.817 DEBUG:boto:Headers: {} 2024-04-12T03:47:33.817 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:33.817 DEBUG:boto:Port: 8000 2024-04-12T03:47:33.817 DEBUG:boto:Params: {} 2024-04-12T03:47:33.817 DEBUG:boto:Token: None 2024-04-12T03:47:33.817 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:33 GMT /dkyges-49/ 2024-04-12T03:47:33.818 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY= 2024-04-12T03:47:33.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cXlvZsbyHUXPhvjshvUY9AF4owY=', 'Content-Length': '0'} 2024-04-12T03:47:33.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088004bc4f6de0cfd-006618aed5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:33.820 DEBUG:boto:b'dkyges-49obj81000falseobj82024-04-12T03:46:56.252Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:33.821 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-49 zones={a2, a1} 2024-04-12T03:47:33.821 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-12T03:47:33.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:47:33.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:47:33.821 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:47:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:33.880 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:33.936 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:47:33.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:33.936 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006012", 2024-04-12T03:47:33.937 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:47:13.471954Z" 2024-04-12T03:47:33.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:33.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:33.937 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000003753", 2024-04-12T03:47:33.937 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:47:12.262341Z" 2024-04-12T03:47:33.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:33.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:33.937 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000009807", 2024-04-12T03:47:33.937 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:47:02.517106Z" 2024-04-12T03:47:33.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:33.938 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:33.938 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000007552", 2024-04-12T03:47:33.938 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:46:56.071395Z" 2024-04-12T03:47:33.938 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:33.938 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:47:33.939 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000006012', 1: '00000000000000000000:00000000000000003753', 2: '00000000000000000000:00000000000000009807', 3: '00000000000000000000:00000000000000007552'} 2024-04-12T03:47:33.939 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:47:33.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:47:33.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:47:33.940 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:47:33.996 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:33.996 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:34.050 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:47:34.050 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:47:34.050 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:47:34.050 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:47:34.050 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:47:34.051 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 16340188497730315678 2024-04-12T03:47:34.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.051 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:47:34.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.051 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:47:34.051 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:47:34.051 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:47:34.051 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006012", 2024-04-12T03:47:34.051 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:47:34.051 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:47:34.051 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:47:34.052 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:47:13.471954Z" 2024-04-12T03:47:34.052 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.052 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:47:34.052 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:47:34.052 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:47:34.052 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000003753", 2024-04-12T03:47:34.052 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:47:34.052 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:47:34.052 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:47:34.053 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:47:12.262341Z" 2024-04-12T03:47:34.053 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.053 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:47:34.053 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:47:34.053 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:47:34.053 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000009807", 2024-04-12T03:47:34.053 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:47:34.053 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:47:34.053 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:47:34.054 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:47:02.517106Z" 2024-04-12T03:47:34.054 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.054 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:47:34.054 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:47:34.054 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:47:34.054 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000007552", 2024-04-12T03:47:34.054 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:47:34.054 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:47:34.054 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:47:34.055 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:46:56.071395Z" 2024-04-12T03:47:34.055 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.055 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.055 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:47:34.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.055 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:47:34.055 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:47:34.055 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:47:34.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.055 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:47:34Z" 2024-04-12T03:47:34.055 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:47:34.056 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16340188497730315678 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006012", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:47:13.471954Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003753", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:47:12.262341Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009807", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:47:02.517106Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007552", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:46:56.071395Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:47:34Z" } 2024-04-12T03:47:34.056 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006012', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:47:13.471954Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003753', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:47:12.262341Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009807', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:47:02.517106Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007552', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:46:56.071395Z'}}] 2024-04-12T03:47:34.057 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006012', 1: '00000000000000000000:00000000000000003753', 2: '00000000000000000000:00000000000000009807', 3: '00000000000000000000:00000000000000007552'} 2024-04-12T03:47:34.057 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006012', 1: '00000000000000000000:00000000000000003753', 2: '00000000000000000000:00000000000000009807', 3: '00000000000000000000:00000000000000007552'} 2024-04-12T03:47:34.057 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:47:34.057 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-12T03:47:34.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:47:34.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:47:34.057 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:47:34.112 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:34.113 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:34.169 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:47:34.170 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.170 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004316", 2024-04-12T03:47:34.170 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:47:11.337327Z" 2024-04-12T03:47:34.170 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.170 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.170 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002625", 2024-04-12T03:47:34.170 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:47:11.336689Z" 2024-04-12T03:47:34.171 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.171 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.171 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000008679", 2024-04-12T03:47:34.171 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:47:11.378263Z" 2024-04-12T03:47:34.171 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.171 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.171 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000005158", 2024-04-12T03:47:34.171 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:46:13.698945Z" 2024-04-12T03:47:34.171 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.171 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:47:34.172 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000004316', 1: '00000000000000000000:00000000000000002625', 2: '00000000000000000000:00000000000000008679', 3: '00000000000000000000:00000000000000005158'} 2024-04-12T03:47:34.172 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:47:34.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:47:34.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:47:34.173 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:47:34.231 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:34.231 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:34.281 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:47:34.281 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:47:34.281 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:47:34.281 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:47:34.281 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:47:34.281 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5944041172307441597 2024-04-12T03:47:34.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.282 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:47:34.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.282 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:47:34.282 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:47:34.282 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:47:34.282 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004316", 2024-04-12T03:47:34.282 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:47:34.282 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:47:34.282 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:47:34.282 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:47:11.337327Z" 2024-04-12T03:47:34.283 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.283 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:47:34.283 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:47:34.283 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:47:34.283 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000002625", 2024-04-12T03:47:34.283 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:47:34.283 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:47:34.283 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:47:34.283 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:47:11.336689Z" 2024-04-12T03:47:34.284 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.284 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:47:34.284 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:47:34.284 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:47:34.284 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000008679", 2024-04-12T03:47:34.284 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:47:34.284 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:47:34.284 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:47:34.284 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:47:11.378263Z" 2024-04-12T03:47:34.285 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.285 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:47:34.285 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:47:34.285 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:47:34.285 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000005158", 2024-04-12T03:47:34.285 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:47:34.285 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:47:34.285 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:47:34.285 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:46:13.698945Z" 2024-04-12T03:47:34.285 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.286 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.286 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:47:34.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.286 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:47:34.286 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:47:34.286 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:47:34.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.286 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:47:34Z" 2024-04-12T03:47:34.286 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:47:34.287 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5944041172307441597 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004316", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:47:11.337327Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002625", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:47:11.336689Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008679", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:47:11.378263Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005158", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:46:13.698945Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:47:34Z" } 2024-04-12T03:47:34.288 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004316', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:47:11.337327Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002625', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:47:11.336689Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008679', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:47:11.378263Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005158', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:46:13.698945Z'}}] 2024-04-12T03:47:34.288 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004316', 1: '00000000000000000000:00000000000000002625', 2: '00000000000000000000:00000000000000008679', 3: '00000000000000000000:00000000000000005158'} 2024-04-12T03:47:34.288 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004316', 1: '00000000000000000000:00000000000000002625', 2: '00000000000000000000:00000000000000008679', 3: '00000000000000000000:00000000000000005158'} 2024-04-12T03:47:34.288 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:47:34.289 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2024-04-12T03:47:34.291 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-50 2024-04-12T03:47:34.291 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:34.291 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:34.291 DEBUG:boto:Method: PUT 2024-04-12T03:47:34.291 DEBUG:boto:Path: /dkyges-50/ 2024-04-12T03:47:34.291 DEBUG:boto:Data: 2024-04-12T03:47:34.291 DEBUG:boto:Headers: {} 2024-04-12T03:47:34.291 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:34.292 DEBUG:boto:Port: 8000 2024-04-12T03:47:34.292 DEBUG:boto:Params: {} 2024-04-12T03:47:34.292 DEBUG:boto:Token: None 2024-04-12T03:47:34.292 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:47:34 GMT /dkyges-50/ 2024-04-12T03:47:34.292 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DIPcvPwCcOeFS3KTLr0FHgoM8xk= 2024-04-12T03:47:34.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DIPcvPwCcOeFS3KTLr0FHgoM8xk=', 'Content-Length': '0'} 2024-04-12T03:47:34.309 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000be0a98c0db3dbae9-006618aed6-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.310 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-51 2024-04-12T03:47:34.310 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:34.310 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:34.310 DEBUG:boto:Method: PUT 2024-04-12T03:47:34.310 DEBUG:boto:Path: /dkyges-51/ 2024-04-12T03:47:34.310 DEBUG:boto:Data: 2024-04-12T03:47:34.310 DEBUG:boto:Headers: {} 2024-04-12T03:47:34.310 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:34.310 DEBUG:boto:Port: 8001 2024-04-12T03:47:34.311 DEBUG:boto:Params: {} 2024-04-12T03:47:34.311 DEBUG:boto:Token: None 2024-04-12T03:47:34.311 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:47:34 GMT /dkyges-51/ 2024-04-12T03:47:34.311 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OKBaiZeAnfzBGdOOZV6Y5UkJdM8= 2024-04-12T03:47:34.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OKBaiZeAnfzBGdOOZV6Y5UkJdM8=', 'Content-Length': '0'} 2024-04-12T03:47:34.340 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cd84c8b05b17b7d1-006618aed6-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.341 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:34.341 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:34.341 DEBUG:boto:Method: HEAD 2024-04-12T03:47:34.341 DEBUG:boto:Path: /dkyges-50/ 2024-04-12T03:47:34.341 DEBUG:boto:Data: 2024-04-12T03:47:34.341 DEBUG:boto:Headers: {} 2024-04-12T03:47:34.341 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:34.341 DEBUG:boto:Port: 8000 2024-04-12T03:47:34.342 DEBUG:boto:Params: {} 2024-04-12T03:47:34.342 DEBUG:boto:Token: None 2024-04-12T03:47:34.342 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:34 GMT /dkyges-50/ 2024-04-12T03:47:34.342 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S3Z39nnBDJWTNjMRj4JFGTJ/TO4= 2024-04-12T03:47:34.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S3Z39nnBDJWTNjMRj4JFGTJ/TO4=', 'Content-Length': '0'} 2024-04-12T03:47:34.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ffc3b1d99297cf80-006618aed6-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.345 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:47:34.345 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:47:34.345 DEBUG:boto:Method: PUT 2024-04-12T03:47:34.345 DEBUG:boto:Path: /dkyges-50/obj1 2024-04-12T03:47:34.345 DEBUG:boto:Data: 2024-04-12T03:47:34.345 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:47:34.345 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:34.345 DEBUG:boto:Port: 8000 2024-04-12T03:47:34.345 DEBUG:boto:Params: {} 2024-04-12T03:47:34.346 DEBUG:boto:Token: None 2024-04-12T03:47:34.346 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:47:34 GMT /dkyges-50/obj1 2024-04-12T03:47:34.346 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wrz1sb+W72hFM4/DM39IrXT71v8= 2024-04-12T03:47:34.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wrz1sb+W72hFM4/DM39IrXT71v8='} 2024-04-12T03:47:34.390 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f952d1f17363162f-006618aed6-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.391 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:34.391 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:34.391 DEBUG:boto:Method: HEAD 2024-04-12T03:47:34.391 DEBUG:boto:Path: /dkyges-50/ 2024-04-12T03:47:34.391 DEBUG:boto:Data: 2024-04-12T03:47:34.391 DEBUG:boto:Headers: {} 2024-04-12T03:47:34.391 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:34.392 DEBUG:boto:Port: 8000 2024-04-12T03:47:34.392 DEBUG:boto:Params: {} 2024-04-12T03:47:34.392 DEBUG:boto:Token: None 2024-04-12T03:47:34.392 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:34 GMT /dkyges-50/ 2024-04-12T03:47:34.392 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S3Z39nnBDJWTNjMRj4JFGTJ/TO4= 2024-04-12T03:47:34.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S3Z39nnBDJWTNjMRj4JFGTJ/TO4=', 'Content-Length': '0'} 2024-04-12T03:47:34.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000245041a590cb52e2-006618aed6-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.394 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:47:34.395 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:47:34.395 DEBUG:boto:Method: PUT 2024-04-12T03:47:34.395 DEBUG:boto:Path: /dkyges-50/obj2 2024-04-12T03:47:34.395 DEBUG:boto:Data: 2024-04-12T03:47:34.395 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:47:34.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:34.395 DEBUG:boto:Port: 8000 2024-04-12T03:47:34.395 DEBUG:boto:Params: {} 2024-04-12T03:47:34.395 DEBUG:boto:Token: None 2024-04-12T03:47:34.395 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:47:34 GMT /dkyges-50/obj2 2024-04-12T03:47:34.396 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:khMYsYeyUrvzTi69OSf9JW3GAz8= 2024-04-12T03:47:34.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:khMYsYeyUrvzTi69OSf9JW3GAz8='} 2024-04-12T03:47:34.402 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000b14a6e5c55ffbda-006618aed6-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.402 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:34.402 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:34.402 DEBUG:boto:Method: HEAD 2024-04-12T03:47:34.403 DEBUG:boto:Path: /dkyges-50/ 2024-04-12T03:47:34.403 DEBUG:boto:Data: 2024-04-12T03:47:34.403 DEBUG:boto:Headers: {} 2024-04-12T03:47:34.403 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:34.403 DEBUG:boto:Port: 8000 2024-04-12T03:47:34.403 DEBUG:boto:Params: {} 2024-04-12T03:47:34.403 DEBUG:boto:Token: None 2024-04-12T03:47:34.403 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:34 GMT /dkyges-50/ 2024-04-12T03:47:34.403 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S3Z39nnBDJWTNjMRj4JFGTJ/TO4= 2024-04-12T03:47:34.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S3Z39nnBDJWTNjMRj4JFGTJ/TO4=', 'Content-Length': '0'} 2024-04-12T03:47:34.406 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3534a54752085a6-006618aed6-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.406 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:47:34.406 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:47:34.406 DEBUG:boto:Method: PUT 2024-04-12T03:47:34.406 DEBUG:boto:Path: /dkyges-50/obj3 2024-04-12T03:47:34.407 DEBUG:boto:Data: 2024-04-12T03:47:34.407 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:47:34.407 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:34.407 DEBUG:boto:Port: 8000 2024-04-12T03:47:34.407 DEBUG:boto:Params: {} 2024-04-12T03:47:34.407 DEBUG:boto:Token: None 2024-04-12T03:47:34.408 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:47:34 GMT /dkyges-50/obj3 2024-04-12T03:47:34.408 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:w4TcLDXQ55Q5fxV3EdE7W4OmGtA= 2024-04-12T03:47:34.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:w4TcLDXQ55Q5fxV3EdE7W4OmGtA='} 2024-04-12T03:47:34.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc40afd044adf62e-006618aed6-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.451 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:34.451 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:34.451 DEBUG:boto:Method: HEAD 2024-04-12T03:47:34.451 DEBUG:boto:Path: /dkyges-50/ 2024-04-12T03:47:34.451 DEBUG:boto:Data: 2024-04-12T03:47:34.451 DEBUG:boto:Headers: {} 2024-04-12T03:47:34.451 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:34.451 DEBUG:boto:Port: 8000 2024-04-12T03:47:34.451 DEBUG:boto:Params: {} 2024-04-12T03:47:34.451 DEBUG:boto:Token: None 2024-04-12T03:47:34.451 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:34 GMT /dkyges-50/ 2024-04-12T03:47:34.452 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S3Z39nnBDJWTNjMRj4JFGTJ/TO4= 2024-04-12T03:47:34.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S3Z39nnBDJWTNjMRj4JFGTJ/TO4=', 'Content-Length': '0'} 2024-04-12T03:47:34.454 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022753dd4f4884390-006618aed6-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.454 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:47:34.454 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:47:34.455 DEBUG:boto:Method: PUT 2024-04-12T03:47:34.455 DEBUG:boto:Path: /dkyges-50/obj4 2024-04-12T03:47:34.455 DEBUG:boto:Data: 2024-04-12T03:47:34.455 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:47:34.455 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:34.455 DEBUG:boto:Port: 8000 2024-04-12T03:47:34.455 DEBUG:boto:Params: {} 2024-04-12T03:47:34.455 DEBUG:boto:Token: None 2024-04-12T03:47:34.455 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:47:34 GMT /dkyges-50/obj4 2024-04-12T03:47:34.456 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iqJaUeEbgcvCf8eC0uVnBHjAY2Q= 2024-04-12T03:47:34.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iqJaUeEbgcvCf8eC0uVnBHjAY2Q='} 2024-04-12T03:47:34.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009df412428de7fdee-006618aed6-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.462 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:34.462 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:34.463 DEBUG:boto:Method: HEAD 2024-04-12T03:47:34.463 DEBUG:boto:Path: /dkyges-51/ 2024-04-12T03:47:34.463 DEBUG:boto:Data: 2024-04-12T03:47:34.463 DEBUG:boto:Headers: {} 2024-04-12T03:47:34.463 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:34.463 DEBUG:boto:Port: 8001 2024-04-12T03:47:34.463 DEBUG:boto:Params: {} 2024-04-12T03:47:34.463 DEBUG:boto:Token: None 2024-04-12T03:47:34.463 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:34 GMT /dkyges-51/ 2024-04-12T03:47:34.463 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dm801Iyp+0XVxeffPndaTRZzgOI= 2024-04-12T03:47:34.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dm801Iyp+0XVxeffPndaTRZzgOI=', 'Content-Length': '0'} 2024-04-12T03:47:34.466 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c8d588c5d2b9f814-006618aed6-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.466 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:47:34.466 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:47:34.466 DEBUG:boto:Method: PUT 2024-04-12T03:47:34.467 DEBUG:boto:Path: /dkyges-51/obj1 2024-04-12T03:47:34.467 DEBUG:boto:Data: 2024-04-12T03:47:34.467 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:47:34.467 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:34.467 DEBUG:boto:Port: 8001 2024-04-12T03:47:34.467 DEBUG:boto:Params: {} 2024-04-12T03:47:34.467 DEBUG:boto:Token: None 2024-04-12T03:47:34.467 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:47:34 GMT /dkyges-51/obj1 2024-04-12T03:47:34.468 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C8bM8yntPPK45e/gXABAQfZTh9k= 2024-04-12T03:47:34.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C8bM8yntPPK45e/gXABAQfZTh9k='} 2024-04-12T03:47:34.478 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e933064104f6d320-006618aed6-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.478 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:34.478 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:34.478 DEBUG:boto:Method: HEAD 2024-04-12T03:47:34.478 DEBUG:boto:Path: /dkyges-51/ 2024-04-12T03:47:34.478 DEBUG:boto:Data: 2024-04-12T03:47:34.478 DEBUG:boto:Headers: {} 2024-04-12T03:47:34.478 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:34.478 DEBUG:boto:Port: 8001 2024-04-12T03:47:34.479 DEBUG:boto:Params: {} 2024-04-12T03:47:34.479 DEBUG:boto:Token: None 2024-04-12T03:47:34.479 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:34 GMT /dkyges-51/ 2024-04-12T03:47:34.479 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dm801Iyp+0XVxeffPndaTRZzgOI= 2024-04-12T03:47:34.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dm801Iyp+0XVxeffPndaTRZzgOI=', 'Content-Length': '0'} 2024-04-12T03:47:34.483 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000612e702c5ad49ca4-006618aed6-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.483 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:47:34.484 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:47:34.484 DEBUG:boto:Method: PUT 2024-04-12T03:47:34.484 DEBUG:boto:Path: /dkyges-51/obj2 2024-04-12T03:47:34.484 DEBUG:boto:Data: 2024-04-12T03:47:34.484 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:47:34.485 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:34.485 DEBUG:boto:Port: 8001 2024-04-12T03:47:34.485 DEBUG:boto:Params: {} 2024-04-12T03:47:34.485 DEBUG:boto:Token: None 2024-04-12T03:47:34.485 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:47:34 GMT /dkyges-51/obj2 2024-04-12T03:47:34.485 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yJ0NshJkV6hmB6ugYNrzdCMH89w= 2024-04-12T03:47:34.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yJ0NshJkV6hmB6ugYNrzdCMH89w='} 2024-04-12T03:47:34.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ef9d679ccfadbe0-006618aed6-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.531 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:34.531 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:34.531 DEBUG:boto:Method: HEAD 2024-04-12T03:47:34.531 DEBUG:boto:Path: /dkyges-51/ 2024-04-12T03:47:34.531 DEBUG:boto:Data: 2024-04-12T03:47:34.531 DEBUG:boto:Headers: {} 2024-04-12T03:47:34.531 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:34.531 DEBUG:boto:Port: 8001 2024-04-12T03:47:34.531 DEBUG:boto:Params: {} 2024-04-12T03:47:34.532 DEBUG:boto:Token: None 2024-04-12T03:47:34.532 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:34 GMT /dkyges-51/ 2024-04-12T03:47:34.532 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dm801Iyp+0XVxeffPndaTRZzgOI= 2024-04-12T03:47:34.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dm801Iyp+0XVxeffPndaTRZzgOI=', 'Content-Length': '0'} 2024-04-12T03:47:34.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042558dfcf4648b1f-006618aed6-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.534 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:47:34.535 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:47:34.535 DEBUG:boto:Method: PUT 2024-04-12T03:47:34.535 DEBUG:boto:Path: /dkyges-51/obj3 2024-04-12T03:47:34.535 DEBUG:boto:Data: 2024-04-12T03:47:34.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:47:34.535 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:34.535 DEBUG:boto:Port: 8001 2024-04-12T03:47:34.535 DEBUG:boto:Params: {} 2024-04-12T03:47:34.535 DEBUG:boto:Token: None 2024-04-12T03:47:34.535 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:47:34 GMT /dkyges-51/obj3 2024-04-12T03:47:34.536 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:43X2GOMiRoh9M58Ho6rhlVYNQaw= 2024-04-12T03:47:34.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:43X2GOMiRoh9M58Ho6rhlVYNQaw='} 2024-04-12T03:47:34.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d80cfec29a2299d2-006618aed6-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.542 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:34.542 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:34.542 DEBUG:boto:Method: HEAD 2024-04-12T03:47:34.542 DEBUG:boto:Path: /dkyges-51/ 2024-04-12T03:47:34.542 DEBUG:boto:Data: 2024-04-12T03:47:34.542 DEBUG:boto:Headers: {} 2024-04-12T03:47:34.543 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:34.543 DEBUG:boto:Port: 8001 2024-04-12T03:47:34.543 DEBUG:boto:Params: {} 2024-04-12T03:47:34.543 DEBUG:boto:Token: None 2024-04-12T03:47:34.543 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:34 GMT /dkyges-51/ 2024-04-12T03:47:34.543 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dm801Iyp+0XVxeffPndaTRZzgOI= 2024-04-12T03:47:34.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dm801Iyp+0XVxeffPndaTRZzgOI=', 'Content-Length': '0'} 2024-04-12T03:47:34.545 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b6f292eb03606e9-006618aed6-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.545 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:47:34.545 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:47:34.546 DEBUG:boto:Method: PUT 2024-04-12T03:47:34.546 DEBUG:boto:Path: /dkyges-51/obj4 2024-04-12T03:47:34.546 DEBUG:boto:Data: 2024-04-12T03:47:34.546 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:47:34.546 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:34.546 DEBUG:boto:Port: 8001 2024-04-12T03:47:34.546 DEBUG:boto:Params: {} 2024-04-12T03:47:34.546 DEBUG:boto:Token: None 2024-04-12T03:47:34.546 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:47:34 GMT /dkyges-51/obj4 2024-04-12T03:47:34.546 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QOw5k8ztuqvHSciQLWqvZkCcwuo= 2024-04-12T03:47:34.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:47:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QOw5k8ztuqvHSciQLWqvZkCcwuo='} 2024-04-12T03:47:34.590 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020d57153b6b1219c-006618aed6-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:34.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:47:34.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:47:34.591 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:47:34.649 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:34.650 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:34.685 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:47:34.700 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:47:34.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.700 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:47:34.700 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:47:34.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.700 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:47:34.700 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:47:34.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.701 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.701 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:47:34.701 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:47:34.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.701 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.701 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893654.331665_443.1", 2024-04-12T03:47:34.701 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:47:34.331665Z" 2024-04-12T03:47:34.701 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.701 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:47:34.702 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893654.331665_443.1'} 2024-04-12T03:47:34.703 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:47:34.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:47:34.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:47:34.703 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:47:34.758 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:34.758 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:34.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:34.795+0000 7f1966c4b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:47:34.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:34.796+0000 7f1966c4b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:47:34.809 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:47:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:47:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:47:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:47:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:47:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:47:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:47:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:47:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:47:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:47:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:47:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:47:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:47:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:47:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:47:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:47:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:47:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:47:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:47:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:47:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:47:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:47:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:47:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:47:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:47:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:47:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:47:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:47:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:47:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:47:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:47:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:47:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:47:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:47:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:47:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:47:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:47:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:47:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893654.331665_443.1", 2024-04-12T03:47:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:47:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:47:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:47:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:47:34.331665Z", 2024-04-12T03:47:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:47:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:47:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:47:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:47:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:47:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:47:34Z" 2024-04-12T03:47:34.815 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:47:34.815 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893654.331665_443.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:47:34.331665Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:47:34Z" } 2024-04-12T03:47:34.816 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893654.331665_443.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:47:34.331665Z', 'realm_epoch': 2}}] 2024-04-12T03:47:34.816 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893654.331665_443.1'} 2024-04-12T03:47:34.816 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893654.331665_443.1'} 2024-04-12T03:47:34.816 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:47:34.817 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-50 2024-04-12T03:47:34.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:34.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:34.817 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:47:34.879 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:34.880 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:34.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:34.922+0000 7f0174eee780 1 waiting to reach incremental sync.. 2024-04-12T03:47:39.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:39.924+0000 7f0174eee780 1 waiting to reach incremental sync.. 2024-04-12T03:47:44.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:44.923+0000 7f0174eee780 1 waiting to reach incremental sync.. 2024-04-12T03:47:49.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:49.924+0000 7f0174eee780 1 waiting to reach incremental sync.. 2024-04-12T03:47:54.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:54.926+0000 7f0174eee780 1 bucket sync caught up with source: 2024-04-12T03:47:54.924 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.27.6, , 00000000001.22.6, , 00000000001.50.6, , , 00000000001.50.6, , , ] 2024-04-12T03:47:54.924 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.27.6, , 00000000001.22.6, , 00000000001.50.6, , , 00000000001.50.6, , , ] 2024-04-12T03:47:54.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:54.926+0000 7f0174eee780 0 bucket checkpoint complete 2024-04-12T03:47:54.943 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-50 2024-04-12T03:47:54.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:54.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:54.944 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:47:55.010 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:55.010 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:55.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:55.054+0000 7fd0401eb780 1 bucket sync caught up with source: 2024-04-12T03:47:55.051 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.1886.6, , 00000000001.1386.6, , 00000000001.3600.6, , , 00000000001.1566.6, , , ] 2024-04-12T03:47:55.052 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.1886.6, , 00000000001.1386.6, , 00000000001.3600.6, , , 00000000001.1566.6, , , ] 2024-04-12T03:47:55.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:55.054+0000 7fd0401eb780 0 bucket checkpoint complete 2024-04-12T03:47:55.068 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-50 zones={a2, a1} 2024-04-12T03:47:55.068 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.068 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.068 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.068 DEBUG:boto:Path: /dkyges-50/ 2024-04-12T03:47:55.068 DEBUG:boto:Data: 2024-04-12T03:47:55.068 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.068 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.069 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.069 DEBUG:boto:Params: {} 2024-04-12T03:47:55.069 DEBUG:boto:Token: None 2024-04-12T03:47:55.069 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.069 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9DZinkVuRcb1xcupHx40XqD77n4= 2024-04-12T03:47:55.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9DZinkVuRcb1xcupHx40XqD77n4=', 'Content-Length': '0'} 2024-04-12T03:47:55.072 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d801bca9540eb38d-006618aeeb-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.072 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.072 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.072 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.073 DEBUG:boto:Path: /dkyges-50/ 2024-04-12T03:47:55.073 DEBUG:boto:Data: 2024-04-12T03:47:55.073 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.073 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.073 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.073 DEBUG:boto:Params: {} 2024-04-12T03:47:55.073 DEBUG:boto:Token: None 2024-04-12T03:47:55.073 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.073 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9DZinkVuRcb1xcupHx40XqD77n4= 2024-04-12T03:47:55.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9DZinkVuRcb1xcupHx40XqD77n4=', 'Content-Length': '0'} 2024-04-12T03:47:55.075 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3bbaec03eb06ed1-006618aeeb-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.076 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:47:55.076 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.076 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.076 DEBUG:boto:path=/dkyges-50/?versions&max-keys=999 2024-04-12T03:47:55.076 DEBUG:boto:auth_path=/dkyges-50/?versions&max-keys=999 2024-04-12T03:47:55.076 DEBUG:boto:Method: GET 2024-04-12T03:47:55.077 DEBUG:boto:Path: /dkyges-50/?versions&max-keys=999 2024-04-12T03:47:55.077 DEBUG:boto:Data: 2024-04-12T03:47:55.077 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.077 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.077 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.077 DEBUG:boto:Params: {} 2024-04-12T03:47:55.077 DEBUG:boto:Token: None 2024-04-12T03:47:55.077 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/?versions 2024-04-12T03:47:55.077 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a70VuJFDjE0hY4CT6B+ygS+FLow= 2024-04-12T03:47:55.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a70VuJFDjE0hY4CT6B+ygS+FLow=', 'Content-Length': '0'} 2024-04-12T03:47:55.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a12e3081ba2efc73-006618aeeb-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.080 DEBUG:boto:b'dkyges-50999falseobj1nulltrue2024-04-12T03:47:34.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:47:34.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:47:34.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:47:34.461Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:47:55.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:47:55.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:47:55.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:47:55.081 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:47:55.081 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.081 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.081 DEBUG:boto:path=/dkyges-50/?versions&max-keys=999 2024-04-12T03:47:55.081 DEBUG:boto:auth_path=/dkyges-50/?versions&max-keys=999 2024-04-12T03:47:55.081 DEBUG:boto:Method: GET 2024-04-12T03:47:55.082 DEBUG:boto:Path: /dkyges-50/?versions&max-keys=999 2024-04-12T03:47:55.082 DEBUG:boto:Data: 2024-04-12T03:47:55.082 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.082 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.082 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.082 DEBUG:boto:Params: {} 2024-04-12T03:47:55.082 DEBUG:boto:Token: None 2024-04-12T03:47:55.083 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/?versions 2024-04-12T03:47:55.083 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a70VuJFDjE0hY4CT6B+ygS+FLow= 2024-04-12T03:47:55.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a70VuJFDjE0hY4CT6B+ygS+FLow=', 'Content-Length': '0'} 2024-04-12T03:47:55.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038187800bc7bf020-006618aeeb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.085 DEBUG:boto:b'dkyges-50999falseobj1nulltrue2024-04-12T03:47:34.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:47:34.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:47:34.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:47:34.461Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:47:55.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:47:55.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:47:55.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:47:55.086 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.086 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.086 DEBUG:boto:path=/dkyges-50/?versions&max-keys=999 2024-04-12T03:47:55.087 DEBUG:boto:auth_path=/dkyges-50/?versions&max-keys=999 2024-04-12T03:47:55.087 DEBUG:boto:Method: GET 2024-04-12T03:47:55.087 DEBUG:boto:Path: /dkyges-50/?versions&max-keys=999 2024-04-12T03:47:55.087 DEBUG:boto:Data: 2024-04-12T03:47:55.087 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.087 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.087 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.087 DEBUG:boto:Params: {} 2024-04-12T03:47:55.087 DEBUG:boto:Token: None 2024-04-12T03:47:55.087 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/?versions 2024-04-12T03:47:55.088 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a70VuJFDjE0hY4CT6B+ygS+FLow= 2024-04-12T03:47:55.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a70VuJFDjE0hY4CT6B+ygS+FLow=', 'Content-Length': '0'} 2024-04-12T03:47:55.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000197305a437232ace-006618aeeb-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.090 DEBUG:boto:b'dkyges-50999falseobj1nulltrue2024-04-12T03:47:34.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:47:34.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:47:34.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:47:34.461Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.090 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.091 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.091 DEBUG:boto:path=/dkyges-50/?versions&max-keys=999 2024-04-12T03:47:55.091 DEBUG:boto:auth_path=/dkyges-50/?versions&max-keys=999 2024-04-12T03:47:55.091 DEBUG:boto:Method: GET 2024-04-12T03:47:55.091 DEBUG:boto:Path: /dkyges-50/?versions&max-keys=999 2024-04-12T03:47:55.091 DEBUG:boto:Data: 2024-04-12T03:47:55.091 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.091 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.091 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.091 DEBUG:boto:Params: {} 2024-04-12T03:47:55.092 DEBUG:boto:Token: None 2024-04-12T03:47:55.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/?versions 2024-04-12T03:47:55.092 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a70VuJFDjE0hY4CT6B+ygS+FLow= 2024-04-12T03:47:55.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a70VuJFDjE0hY4CT6B+ygS+FLow=', 'Content-Length': '0'} 2024-04-12T03:47:55.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbf4e063a39ed6e4-006618aeeb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.095 DEBUG:boto:b'dkyges-50999falseobj1nulltrue2024-04-12T03:47:34.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:47:34.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:47:34.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:47:34.461Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:47:55.095 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:47:55.095 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:47:55.096 DEBUG:boto:path=/dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.096 DEBUG:boto:auth_path=/dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.096 DEBUG:boto:Method: GET 2024-04-12T03:47:55.096 DEBUG:boto:Path: /dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.096 DEBUG:boto:Data: 2024-04-12T03:47:55.096 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.096 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.096 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.096 DEBUG:boto:Params: {} 2024-04-12T03:47:55.097 DEBUG:boto:Token: None 2024-04-12T03:47:55.097 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.097 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9ZY1ysLO+EXN2D/pw4FYML1qJuU= 2024-04-12T03:47:55.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9ZY1ysLO+EXN2D/pw4FYML1qJuU=', 'Content-Length': '0'} 2024-04-12T03:47:55.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc60422c75b358e6-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.100 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:47:55.100 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:47:55.100 DEBUG:boto:path=/dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.100 DEBUG:boto:auth_path=/dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.100 DEBUG:boto:Method: GET 2024-04-12T03:47:55.100 DEBUG:boto:Path: /dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.100 DEBUG:boto:Data: 2024-04-12T03:47:55.100 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.100 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.100 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.100 DEBUG:boto:Params: {} 2024-04-12T03:47:55.100 DEBUG:boto:Token: None 2024-04-12T03:47:55.101 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.101 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9ZY1ysLO+EXN2D/pw4FYML1qJuU= 2024-04-12T03:47:55.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9ZY1ysLO+EXN2D/pw4FYML1qJuU=', 'Content-Length': '0'} 2024-04-12T03:47:55.102 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7bd0b3f59698883-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.150 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:47:55.151 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:47:55.151 DEBUG:boto:path=/dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.151 DEBUG:boto:auth_path=/dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.151 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.151 DEBUG:boto:Path: /dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.151 DEBUG:boto:Data: 2024-04-12T03:47:55.151 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.151 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.151 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.151 DEBUG:boto:Params: {} 2024-04-12T03:47:55.151 DEBUG:boto:Token: None 2024-04-12T03:47:55.152 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.152 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u2/eiCI4Ye2QwmzpVLVPeohJH1k= 2024-04-12T03:47:55.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u2/eiCI4Ye2QwmzpVLVPeohJH1k=', 'Content-Length': '0'} 2024-04-12T03:47:55.153 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000bba217c22f06f85-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.154 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:47:55.154 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:47:55.154 DEBUG:boto:path=/dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.154 DEBUG:boto:auth_path=/dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.154 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.154 DEBUG:boto:Path: /dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.154 DEBUG:boto:Data: 2024-04-12T03:47:55.154 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.154 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.155 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.155 DEBUG:boto:Params: {} 2024-04-12T03:47:55.155 DEBUG:boto:Token: None 2024-04-12T03:47:55.155 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.155 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u2/eiCI4Ye2QwmzpVLVPeohJH1k= 2024-04-12T03:47:55.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u2/eiCI4Ye2QwmzpVLVPeohJH1k=', 'Content-Length': '0'} 2024-04-12T03:47:55.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9d0ba27059a900f-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:47:55.157 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:47:55.157 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:47:55.157 DEBUG:boto:path=/dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.157 DEBUG:boto:auth_path=/dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.158 DEBUG:boto:Method: GET 2024-04-12T03:47:55.158 DEBUG:boto:Path: /dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.158 DEBUG:boto:Data: 2024-04-12T03:47:55.158 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.158 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.158 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.158 DEBUG:boto:Params: {} 2024-04-12T03:47:55.158 DEBUG:boto:Token: None 2024-04-12T03:47:55.158 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.158 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9ZY1ysLO+EXN2D/pw4FYML1qJuU= 2024-04-12T03:47:55.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9ZY1ysLO+EXN2D/pw4FYML1qJuU=', 'Content-Length': '0'} 2024-04-12T03:47:55.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4f5e528c35b51b9-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.160 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:47:55.161 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:47:55.161 DEBUG:boto:path=/dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.161 DEBUG:boto:auth_path=/dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.161 DEBUG:boto:Method: GET 2024-04-12T03:47:55.161 DEBUG:boto:Path: /dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.161 DEBUG:boto:Data: 2024-04-12T03:47:55.161 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.161 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.161 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.161 DEBUG:boto:Params: {} 2024-04-12T03:47:55.161 DEBUG:boto:Token: None 2024-04-12T03:47:55.162 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj1?versionId=null 2024-04-12T03:47:55.162 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9ZY1ysLO+EXN2D/pw4FYML1qJuU= 2024-04-12T03:47:55.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9ZY1ysLO+EXN2D/pw4FYML1qJuU=', 'Content-Length': '0'} 2024-04-12T03:47:55.164 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f330eb8a54b4e21-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.164 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.165 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.165 DEBUG:boto:path=/dkyges-50/?prefix=obj1 2024-04-12T03:47:55.165 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj1 2024-04-12T03:47:55.165 DEBUG:boto:Method: GET 2024-04-12T03:47:55.165 DEBUG:boto:Path: /dkyges-50/?prefix=obj1 2024-04-12T03:47:55.165 DEBUG:boto:Data: 2024-04-12T03:47:55.165 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.165 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.165 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.165 DEBUG:boto:Params: {} 2024-04-12T03:47:55.165 DEBUG:boto:Token: None 2024-04-12T03:47:55.166 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.166 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE= 2024-04-12T03:47:55.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE=', 'Content-Length': '0'} 2024-04-12T03:47:55.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057570426c6488875-006618aeeb-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.169 DEBUG:boto:b'dkyges-50obj11000falseobj12024-04-12T03:47:34.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.169 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.169 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.169 DEBUG:boto:path=/dkyges-50/?prefix=obj1 2024-04-12T03:47:55.169 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj1 2024-04-12T03:47:55.169 DEBUG:boto:Method: GET 2024-04-12T03:47:55.170 DEBUG:boto:Path: /dkyges-50/?prefix=obj1 2024-04-12T03:47:55.170 DEBUG:boto:Data: 2024-04-12T03:47:55.170 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.170 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.170 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.170 DEBUG:boto:Params: {} 2024-04-12T03:47:55.170 DEBUG:boto:Token: None 2024-04-12T03:47:55.170 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.170 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE= 2024-04-12T03:47:55.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE=', 'Content-Length': '0'} 2024-04-12T03:47:55.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083f417028c57411d-006618aeeb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.173 DEBUG:boto:b'dkyges-50obj11000falseobj12024-04-12T03:47:34.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.173 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:47:55.173 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:47:55.173 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.173 DEBUG:boto:Path: /dkyges-50/obj1 2024-04-12T03:47:55.173 DEBUG:boto:Data: 2024-04-12T03:47:55.174 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.174 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.174 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.174 DEBUG:boto:Params: {} 2024-04-12T03:47:55.174 DEBUG:boto:Token: None 2024-04-12T03:47:55.174 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj1 2024-04-12T03:47:55.174 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZXCWjshCGNfWq2OQOR9SQS+Nt9g= 2024-04-12T03:47:55.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZXCWjshCGNfWq2OQOR9SQS+Nt9g=', 'Content-Length': '0'} 2024-04-12T03:47:55.176 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070b8c25620dbeab9-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.176 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:47:55.176 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:47:55.177 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.177 DEBUG:boto:Path: /dkyges-50/obj1 2024-04-12T03:47:55.177 DEBUG:boto:Data: 2024-04-12T03:47:55.177 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.177 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.177 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.177 DEBUG:boto:Params: {} 2024-04-12T03:47:55.177 DEBUG:boto:Token: None 2024-04-12T03:47:55.177 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj1 2024-04-12T03:47:55.177 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZXCWjshCGNfWq2OQOR9SQS+Nt9g= 2024-04-12T03:47:55.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZXCWjshCGNfWq2OQOR9SQS+Nt9g=', 'Content-Length': '0'} 2024-04-12T03:47:55.180 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024c694bfbf85a6df-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:47:55.180 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:47:55.180 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:47:55.180 DEBUG:boto:Method: GET 2024-04-12T03:47:55.180 DEBUG:boto:Path: /dkyges-50/obj1 2024-04-12T03:47:55.181 DEBUG:boto:Data: 2024-04-12T03:47:55.181 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.181 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.181 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.181 DEBUG:boto:Params: {} 2024-04-12T03:47:55.181 DEBUG:boto:Token: None 2024-04-12T03:47:55.181 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj1 2024-04-12T03:47:55.181 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PNyeU5hhvgkTYNm+hjJRaPTt5sU= 2024-04-12T03:47:55.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PNyeU5hhvgkTYNm+hjJRaPTt5sU=', 'Content-Length': '0'} 2024-04-12T03:47:55.183 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003a29858e8f5945a5-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.183 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:47:55.183 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:47:55.183 DEBUG:boto:Method: GET 2024-04-12T03:47:55.184 DEBUG:boto:Path: /dkyges-50/obj1 2024-04-12T03:47:55.184 DEBUG:boto:Data: 2024-04-12T03:47:55.184 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.184 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.184 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.184 DEBUG:boto:Params: {} 2024-04-12T03:47:55.184 DEBUG:boto:Token: None 2024-04-12T03:47:55.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj1 2024-04-12T03:47:55.184 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PNyeU5hhvgkTYNm+hjJRaPTt5sU= 2024-04-12T03:47:55.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PNyeU5hhvgkTYNm+hjJRaPTt5sU=', 'Content-Length': '0'} 2024-04-12T03:47:55.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000050edc3710fc13bc-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.186 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.187 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.187 DEBUG:boto:path=/dkyges-50/?prefix=obj1 2024-04-12T03:47:55.187 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj1 2024-04-12T03:47:55.187 DEBUG:boto:Method: GET 2024-04-12T03:47:55.187 DEBUG:boto:Path: /dkyges-50/?prefix=obj1 2024-04-12T03:47:55.187 DEBUG:boto:Data: 2024-04-12T03:47:55.187 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.187 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.187 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.187 DEBUG:boto:Params: {} 2024-04-12T03:47:55.187 DEBUG:boto:Token: None 2024-04-12T03:47:55.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.188 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE= 2024-04-12T03:47:55.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE=', 'Content-Length': '0'} 2024-04-12T03:47:55.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041bf486f635d071e-006618aeeb-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.234 DEBUG:boto:b'dkyges-50obj11000falseobj12024-04-12T03:47:34.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.235 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.235 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.235 DEBUG:boto:path=/dkyges-50/?prefix=obj1 2024-04-12T03:47:55.235 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj1 2024-04-12T03:47:55.235 DEBUG:boto:Method: GET 2024-04-12T03:47:55.235 DEBUG:boto:Path: /dkyges-50/?prefix=obj1 2024-04-12T03:47:55.235 DEBUG:boto:Data: 2024-04-12T03:47:55.236 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.236 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.236 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.236 DEBUG:boto:Params: {} 2024-04-12T03:47:55.236 DEBUG:boto:Token: None 2024-04-12T03:47:55.236 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.236 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE= 2024-04-12T03:47:55.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE=', 'Content-Length': '0'} 2024-04-12T03:47:55.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a11b2c2a7976f494-006618aeeb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.239 DEBUG:boto:b'dkyges-50obj11000falseobj12024-04-12T03:47:34.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:47:55.240 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:47:55.240 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:47:55.240 DEBUG:boto:path=/dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.240 DEBUG:boto:auth_path=/dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.240 DEBUG:boto:Method: GET 2024-04-12T03:47:55.240 DEBUG:boto:Path: /dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.241 DEBUG:boto:Data: 2024-04-12T03:47:55.241 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.241 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.241 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.241 DEBUG:boto:Params: {} 2024-04-12T03:47:55.241 DEBUG:boto:Token: None 2024-04-12T03:47:55.241 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.241 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1PlMf/tS7lOczxKc0sOmikkVOR4= 2024-04-12T03:47:55.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1PlMf/tS7lOczxKc0sOmikkVOR4=', 'Content-Length': '0'} 2024-04-12T03:47:55.243 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008dad225b8ca31dd0-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.244 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:47:55.244 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:47:55.244 DEBUG:boto:path=/dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.244 DEBUG:boto:auth_path=/dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.244 DEBUG:boto:Method: GET 2024-04-12T03:47:55.244 DEBUG:boto:Path: /dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.244 DEBUG:boto:Data: 2024-04-12T03:47:55.244 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.244 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.244 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.244 DEBUG:boto:Params: {} 2024-04-12T03:47:55.245 DEBUG:boto:Token: None 2024-04-12T03:47:55.245 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.245 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1PlMf/tS7lOczxKc0sOmikkVOR4= 2024-04-12T03:47:55.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1PlMf/tS7lOczxKc0sOmikkVOR4=', 'Content-Length': '0'} 2024-04-12T03:47:55.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f33e6a3d406bf9eb-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.290 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:47:55.291 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:47:55.291 DEBUG:boto:path=/dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.291 DEBUG:boto:auth_path=/dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.291 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.291 DEBUG:boto:Path: /dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.291 DEBUG:boto:Data: 2024-04-12T03:47:55.291 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.291 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.291 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.291 DEBUG:boto:Params: {} 2024-04-12T03:47:55.292 DEBUG:boto:Token: None 2024-04-12T03:47:55.292 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.292 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IRTwtjIECi0aMjbcW8Xk19u2JB8= 2024-04-12T03:47:55.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IRTwtjIECi0aMjbcW8Xk19u2JB8=', 'Content-Length': '0'} 2024-04-12T03:47:55.294 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a954ee72e894811f-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.295 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:47:55.295 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:47:55.295 DEBUG:boto:path=/dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.295 DEBUG:boto:auth_path=/dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.295 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.295 DEBUG:boto:Path: /dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.295 DEBUG:boto:Data: 2024-04-12T03:47:55.295 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.295 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.295 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.296 DEBUG:boto:Params: {} 2024-04-12T03:47:55.296 DEBUG:boto:Token: None 2024-04-12T03:47:55.296 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.296 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IRTwtjIECi0aMjbcW8Xk19u2JB8= 2024-04-12T03:47:55.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IRTwtjIECi0aMjbcW8Xk19u2JB8=', 'Content-Length': '0'} 2024-04-12T03:47:55.298 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000286aa556c16fbb41-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:47:55.299 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:47:55.299 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:47:55.299 DEBUG:boto:path=/dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.299 DEBUG:boto:auth_path=/dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.299 DEBUG:boto:Method: GET 2024-04-12T03:47:55.299 DEBUG:boto:Path: /dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.299 DEBUG:boto:Data: 2024-04-12T03:47:55.299 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.299 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.299 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.299 DEBUG:boto:Params: {} 2024-04-12T03:47:55.300 DEBUG:boto:Token: None 2024-04-12T03:47:55.300 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.300 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1PlMf/tS7lOczxKc0sOmikkVOR4= 2024-04-12T03:47:55.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1PlMf/tS7lOczxKc0sOmikkVOR4=', 'Content-Length': '0'} 2024-04-12T03:47:55.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000db8ed0198bc96a8-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.302 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:47:55.302 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:47:55.302 DEBUG:boto:path=/dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.302 DEBUG:boto:auth_path=/dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.302 DEBUG:boto:Method: GET 2024-04-12T03:47:55.303 DEBUG:boto:Path: /dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.303 DEBUG:boto:Data: 2024-04-12T03:47:55.303 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.303 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.303 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.303 DEBUG:boto:Params: {} 2024-04-12T03:47:55.303 DEBUG:boto:Token: None 2024-04-12T03:47:55.303 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj2?versionId=null 2024-04-12T03:47:55.303 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1PlMf/tS7lOczxKc0sOmikkVOR4= 2024-04-12T03:47:55.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1PlMf/tS7lOczxKc0sOmikkVOR4=', 'Content-Length': '0'} 2024-04-12T03:47:55.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e55eb6e7c4fd12d3-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.305 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.306 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.306 DEBUG:boto:path=/dkyges-50/?prefix=obj2 2024-04-12T03:47:55.306 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj2 2024-04-12T03:47:55.306 DEBUG:boto:Method: GET 2024-04-12T03:47:55.306 DEBUG:boto:Path: /dkyges-50/?prefix=obj2 2024-04-12T03:47:55.306 DEBUG:boto:Data: 2024-04-12T03:47:55.306 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.306 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.306 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.306 DEBUG:boto:Params: {} 2024-04-12T03:47:55.307 DEBUG:boto:Token: None 2024-04-12T03:47:55.307 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.307 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE= 2024-04-12T03:47:55.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE=', 'Content-Length': '0'} 2024-04-12T03:47:55.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051cf998fe1986e23-006618aeeb-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.310 DEBUG:boto:b'dkyges-50obj21000falseobj22024-04-12T03:47:34.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.311 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.311 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.311 DEBUG:boto:path=/dkyges-50/?prefix=obj2 2024-04-12T03:47:55.311 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj2 2024-04-12T03:47:55.311 DEBUG:boto:Method: GET 2024-04-12T03:47:55.311 DEBUG:boto:Path: /dkyges-50/?prefix=obj2 2024-04-12T03:47:55.311 DEBUG:boto:Data: 2024-04-12T03:47:55.311 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.311 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.312 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.312 DEBUG:boto:Params: {} 2024-04-12T03:47:55.312 DEBUG:boto:Token: None 2024-04-12T03:47:55.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.312 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE= 2024-04-12T03:47:55.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE=', 'Content-Length': '0'} 2024-04-12T03:47:55.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005903e3533874d28a-006618aeeb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.315 DEBUG:boto:b'dkyges-50obj21000falseobj22024-04-12T03:47:34.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.316 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:47:55.316 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:47:55.316 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.316 DEBUG:boto:Path: /dkyges-50/obj2 2024-04-12T03:47:55.316 DEBUG:boto:Data: 2024-04-12T03:47:55.316 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.316 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.316 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.316 DEBUG:boto:Params: {} 2024-04-12T03:47:55.316 DEBUG:boto:Token: None 2024-04-12T03:47:55.317 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj2 2024-04-12T03:47:55.317 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1Zdigz10idt+WjpN8RpZx8H7hek= 2024-04-12T03:47:55.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1Zdigz10idt+WjpN8RpZx8H7hek=', 'Content-Length': '0'} 2024-04-12T03:47:55.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d24c5a55dc583f1-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.319 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:47:55.319 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:47:55.319 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.319 DEBUG:boto:Path: /dkyges-50/obj2 2024-04-12T03:47:55.319 DEBUG:boto:Data: 2024-04-12T03:47:55.319 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.319 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.319 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.319 DEBUG:boto:Params: {} 2024-04-12T03:47:55.320 DEBUG:boto:Token: None 2024-04-12T03:47:55.320 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj2 2024-04-12T03:47:55.320 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1Zdigz10idt+WjpN8RpZx8H7hek= 2024-04-12T03:47:55.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1Zdigz10idt+WjpN8RpZx8H7hek=', 'Content-Length': '0'} 2024-04-12T03:47:55.322 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a04ef590bea2440-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:47:55.322 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:47:55.322 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:47:55.322 DEBUG:boto:Method: GET 2024-04-12T03:47:55.322 DEBUG:boto:Path: /dkyges-50/obj2 2024-04-12T03:47:55.322 DEBUG:boto:Data: 2024-04-12T03:47:55.322 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.323 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.323 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.323 DEBUG:boto:Params: {} 2024-04-12T03:47:55.323 DEBUG:boto:Token: None 2024-04-12T03:47:55.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj2 2024-04-12T03:47:55.323 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wr7UfnG61GO0LpvnuHq1LEdshVs= 2024-04-12T03:47:55.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wr7UfnG61GO0LpvnuHq1LEdshVs=', 'Content-Length': '0'} 2024-04-12T03:47:55.325 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079737e4eb506b4a8-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.366 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:47:55.367 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:47:55.367 DEBUG:boto:Method: GET 2024-04-12T03:47:55.367 DEBUG:boto:Path: /dkyges-50/obj2 2024-04-12T03:47:55.367 DEBUG:boto:Data: 2024-04-12T03:47:55.367 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.367 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.367 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.367 DEBUG:boto:Params: {} 2024-04-12T03:47:55.367 DEBUG:boto:Token: None 2024-04-12T03:47:55.368 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj2 2024-04-12T03:47:55.368 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wr7UfnG61GO0LpvnuHq1LEdshVs= 2024-04-12T03:47:55.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wr7UfnG61GO0LpvnuHq1LEdshVs=', 'Content-Length': '0'} 2024-04-12T03:47:55.370 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c2562d0645ff259-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.370 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.370 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.370 DEBUG:boto:path=/dkyges-50/?prefix=obj2 2024-04-12T03:47:55.370 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj2 2024-04-12T03:47:55.370 DEBUG:boto:Method: GET 2024-04-12T03:47:55.370 DEBUG:boto:Path: /dkyges-50/?prefix=obj2 2024-04-12T03:47:55.371 DEBUG:boto:Data: 2024-04-12T03:47:55.371 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.371 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.371 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.371 DEBUG:boto:Params: {} 2024-04-12T03:47:55.371 DEBUG:boto:Token: None 2024-04-12T03:47:55.371 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.371 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE= 2024-04-12T03:47:55.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE=', 'Content-Length': '0'} 2024-04-12T03:47:55.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000055c814d62e80e67-006618aeeb-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.374 DEBUG:boto:b'dkyges-50obj21000falseobj22024-04-12T03:47:34.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.374 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.375 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.375 DEBUG:boto:path=/dkyges-50/?prefix=obj2 2024-04-12T03:47:55.375 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj2 2024-04-12T03:47:55.375 DEBUG:boto:Method: GET 2024-04-12T03:47:55.375 DEBUG:boto:Path: /dkyges-50/?prefix=obj2 2024-04-12T03:47:55.375 DEBUG:boto:Data: 2024-04-12T03:47:55.375 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.375 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.375 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.375 DEBUG:boto:Params: {} 2024-04-12T03:47:55.375 DEBUG:boto:Token: None 2024-04-12T03:47:55.376 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.376 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE= 2024-04-12T03:47:55.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE=', 'Content-Length': '0'} 2024-04-12T03:47:55.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3178d51322200e4-006618aeeb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.379 DEBUG:boto:b'dkyges-50obj21000falseobj22024-04-12T03:47:34.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:47:55.379 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:47:55.379 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:47:55.379 DEBUG:boto:path=/dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.380 DEBUG:boto:auth_path=/dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.380 DEBUG:boto:Method: GET 2024-04-12T03:47:55.380 DEBUG:boto:Path: /dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.380 DEBUG:boto:Data: 2024-04-12T03:47:55.380 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.380 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.380 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.380 DEBUG:boto:Params: {} 2024-04-12T03:47:55.380 DEBUG:boto:Token: None 2024-04-12T03:47:55.380 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.380 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Bd/HbLRtf4G3RYGHRcVoI1sOJ+s= 2024-04-12T03:47:55.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bd/HbLRtf4G3RYGHRcVoI1sOJ+s=', 'Content-Length': '0'} 2024-04-12T03:47:55.382 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002eaf704d452e9070-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.383 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:47:55.383 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:47:55.383 DEBUG:boto:path=/dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.383 DEBUG:boto:auth_path=/dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.383 DEBUG:boto:Method: GET 2024-04-12T03:47:55.383 DEBUG:boto:Path: /dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.383 DEBUG:boto:Data: 2024-04-12T03:47:55.383 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.383 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.383 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.383 DEBUG:boto:Params: {} 2024-04-12T03:47:55.384 DEBUG:boto:Token: None 2024-04-12T03:47:55.384 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.384 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Bd/HbLRtf4G3RYGHRcVoI1sOJ+s= 2024-04-12T03:47:55.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bd/HbLRtf4G3RYGHRcVoI1sOJ+s=', 'Content-Length': '0'} 2024-04-12T03:47:55.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fdd7609796b38a73-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.386 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:47:55.386 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:47:55.386 DEBUG:boto:path=/dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.386 DEBUG:boto:auth_path=/dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.386 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.386 DEBUG:boto:Path: /dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.387 DEBUG:boto:Data: 2024-04-12T03:47:55.387 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.387 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.387 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.387 DEBUG:boto:Params: {} 2024-04-12T03:47:55.387 DEBUG:boto:Token: None 2024-04-12T03:47:55.387 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.387 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dqyZHjiK+MzmxuC6yv8XA9mksWg= 2024-04-12T03:47:55.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dqyZHjiK+MzmxuC6yv8XA9mksWg=', 'Content-Length': '0'} 2024-04-12T03:47:55.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091929ec81ba49eac-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.389 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:47:55.390 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:47:55.390 DEBUG:boto:path=/dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.390 DEBUG:boto:auth_path=/dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.390 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.390 DEBUG:boto:Path: /dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.390 DEBUG:boto:Data: 2024-04-12T03:47:55.390 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.390 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.390 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.390 DEBUG:boto:Params: {} 2024-04-12T03:47:55.390 DEBUG:boto:Token: None 2024-04-12T03:47:55.391 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.391 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dqyZHjiK+MzmxuC6yv8XA9mksWg= 2024-04-12T03:47:55.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dqyZHjiK+MzmxuC6yv8XA9mksWg=', 'Content-Length': '0'} 2024-04-12T03:47:55.392 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006929c0e0c3c65681-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:47:55.393 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:47:55.393 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:47:55.393 DEBUG:boto:path=/dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.393 DEBUG:boto:auth_path=/dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.393 DEBUG:boto:Method: GET 2024-04-12T03:47:55.393 DEBUG:boto:Path: /dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.393 DEBUG:boto:Data: 2024-04-12T03:47:55.393 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.393 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.393 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.394 DEBUG:boto:Params: {} 2024-04-12T03:47:55.394 DEBUG:boto:Token: None 2024-04-12T03:47:55.394 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.394 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Bd/HbLRtf4G3RYGHRcVoI1sOJ+s= 2024-04-12T03:47:55.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bd/HbLRtf4G3RYGHRcVoI1sOJ+s=', 'Content-Length': '0'} 2024-04-12T03:47:55.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a599742964d6f75-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.396 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:47:55.396 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:47:55.396 DEBUG:boto:path=/dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.397 DEBUG:boto:auth_path=/dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.397 DEBUG:boto:Method: GET 2024-04-12T03:47:55.397 DEBUG:boto:Path: /dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.397 DEBUG:boto:Data: 2024-04-12T03:47:55.397 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.397 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.397 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.397 DEBUG:boto:Params: {} 2024-04-12T03:47:55.397 DEBUG:boto:Token: None 2024-04-12T03:47:55.397 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj3?versionId=null 2024-04-12T03:47:55.398 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Bd/HbLRtf4G3RYGHRcVoI1sOJ+s= 2024-04-12T03:47:55.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bd/HbLRtf4G3RYGHRcVoI1sOJ+s=', 'Content-Length': '0'} 2024-04-12T03:47:55.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000180becee0db0a11c-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.400 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.400 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.400 DEBUG:boto:path=/dkyges-50/?prefix=obj3 2024-04-12T03:47:55.400 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj3 2024-04-12T03:47:55.400 DEBUG:boto:Method: GET 2024-04-12T03:47:55.400 DEBUG:boto:Path: /dkyges-50/?prefix=obj3 2024-04-12T03:47:55.400 DEBUG:boto:Data: 2024-04-12T03:47:55.400 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.400 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.400 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.400 DEBUG:boto:Params: {} 2024-04-12T03:47:55.401 DEBUG:boto:Token: None 2024-04-12T03:47:55.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.401 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE= 2024-04-12T03:47:55.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE=', 'Content-Length': '0'} 2024-04-12T03:47:55.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bbcdbf1a926d53e-006618aeeb-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.404 DEBUG:boto:b'dkyges-50obj31000falseobj32024-04-12T03:47:34.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.404 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.404 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.404 DEBUG:boto:path=/dkyges-50/?prefix=obj3 2024-04-12T03:47:55.404 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj3 2024-04-12T03:47:55.404 DEBUG:boto:Method: GET 2024-04-12T03:47:55.405 DEBUG:boto:Path: /dkyges-50/?prefix=obj3 2024-04-12T03:47:55.405 DEBUG:boto:Data: 2024-04-12T03:47:55.405 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.405 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.405 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.405 DEBUG:boto:Params: {} 2024-04-12T03:47:55.405 DEBUG:boto:Token: None 2024-04-12T03:47:55.405 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.405 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE= 2024-04-12T03:47:55.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE=', 'Content-Length': '0'} 2024-04-12T03:47:55.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084d86e10a6e49613-006618aeeb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.408 DEBUG:boto:b'dkyges-50obj31000falseobj32024-04-12T03:47:34.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.408 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:47:55.409 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:47:55.409 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.409 DEBUG:boto:Path: /dkyges-50/obj3 2024-04-12T03:47:55.409 DEBUG:boto:Data: 2024-04-12T03:47:55.409 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.409 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.409 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.409 DEBUG:boto:Params: {} 2024-04-12T03:47:55.409 DEBUG:boto:Token: None 2024-04-12T03:47:55.409 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj3 2024-04-12T03:47:55.410 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OIqzBjEMArOOrISnEmlLCD6+8QI= 2024-04-12T03:47:55.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OIqzBjEMArOOrISnEmlLCD6+8QI=', 'Content-Length': '0'} 2024-04-12T03:47:55.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000609d3bc6e30b8638-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.411 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:47:55.412 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:47:55.412 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.412 DEBUG:boto:Path: /dkyges-50/obj3 2024-04-12T03:47:55.412 DEBUG:boto:Data: 2024-04-12T03:47:55.412 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.412 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.412 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.412 DEBUG:boto:Params: {} 2024-04-12T03:47:55.412 DEBUG:boto:Token: None 2024-04-12T03:47:55.412 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj3 2024-04-12T03:47:55.413 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OIqzBjEMArOOrISnEmlLCD6+8QI= 2024-04-12T03:47:55.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OIqzBjEMArOOrISnEmlLCD6+8QI=', 'Content-Length': '0'} 2024-04-12T03:47:55.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a13b46c7f05f8d9-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:47:55.415 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:47:55.415 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:47:55.415 DEBUG:boto:Method: GET 2024-04-12T03:47:55.415 DEBUG:boto:Path: /dkyges-50/obj3 2024-04-12T03:47:55.415 DEBUG:boto:Data: 2024-04-12T03:47:55.415 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.415 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.415 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.415 DEBUG:boto:Params: {} 2024-04-12T03:47:55.415 DEBUG:boto:Token: None 2024-04-12T03:47:55.416 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj3 2024-04-12T03:47:55.416 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zFPFbFp7yN8GCM0XYZt87jAh/PQ= 2024-04-12T03:47:55.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zFPFbFp7yN8GCM0XYZt87jAh/PQ=', 'Content-Length': '0'} 2024-04-12T03:47:55.417 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5a715506236d03a-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.458 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:47:55.459 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:47:55.459 DEBUG:boto:Method: GET 2024-04-12T03:47:55.459 DEBUG:boto:Path: /dkyges-50/obj3 2024-04-12T03:47:55.459 DEBUG:boto:Data: 2024-04-12T03:47:55.459 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.459 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.459 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.459 DEBUG:boto:Params: {} 2024-04-12T03:47:55.459 DEBUG:boto:Token: None 2024-04-12T03:47:55.459 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj3 2024-04-12T03:47:55.460 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zFPFbFp7yN8GCM0XYZt87jAh/PQ= 2024-04-12T03:47:55.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zFPFbFp7yN8GCM0XYZt87jAh/PQ=', 'Content-Length': '0'} 2024-04-12T03:47:55.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005c8a8196205ade86-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.462 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.462 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.462 DEBUG:boto:path=/dkyges-50/?prefix=obj3 2024-04-12T03:47:55.462 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj3 2024-04-12T03:47:55.462 DEBUG:boto:Method: GET 2024-04-12T03:47:55.463 DEBUG:boto:Path: /dkyges-50/?prefix=obj3 2024-04-12T03:47:55.463 DEBUG:boto:Data: 2024-04-12T03:47:55.463 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.463 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.463 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.463 DEBUG:boto:Params: {} 2024-04-12T03:47:55.463 DEBUG:boto:Token: None 2024-04-12T03:47:55.463 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.463 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE= 2024-04-12T03:47:55.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE=', 'Content-Length': '0'} 2024-04-12T03:47:55.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3bd20da3c7f1acf-006618aeeb-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.466 DEBUG:boto:b'dkyges-50obj31000falseobj32024-04-12T03:47:34.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.467 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.467 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.467 DEBUG:boto:path=/dkyges-50/?prefix=obj3 2024-04-12T03:47:55.467 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj3 2024-04-12T03:47:55.467 DEBUG:boto:Method: GET 2024-04-12T03:47:55.467 DEBUG:boto:Path: /dkyges-50/?prefix=obj3 2024-04-12T03:47:55.467 DEBUG:boto:Data: 2024-04-12T03:47:55.467 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.467 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.467 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.467 DEBUG:boto:Params: {} 2024-04-12T03:47:55.468 DEBUG:boto:Token: None 2024-04-12T03:47:55.468 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.468 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE= 2024-04-12T03:47:55.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE=', 'Content-Length': '0'} 2024-04-12T03:47:55.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004215592b06ce11cf-006618aeeb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.470 DEBUG:boto:b'dkyges-50obj31000falseobj32024-04-12T03:47:34.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:47:55.471 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:47:55.471 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:47:55.471 DEBUG:boto:path=/dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.471 DEBUG:boto:auth_path=/dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.471 DEBUG:boto:Method: GET 2024-04-12T03:47:55.471 DEBUG:boto:Path: /dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.472 DEBUG:boto:Data: 2024-04-12T03:47:55.472 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.472 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.472 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.472 DEBUG:boto:Params: {} 2024-04-12T03:47:55.472 DEBUG:boto:Token: None 2024-04-12T03:47:55.472 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.472 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kxUxb+jNBbNWvoTi77q71QFTlWI= 2024-04-12T03:47:55.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kxUxb+jNBbNWvoTi77q71QFTlWI=', 'Content-Length': '0'} 2024-04-12T03:47:55.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d73322addf8a7d86-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.474 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:47:55.474 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:47:55.474 DEBUG:boto:path=/dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.475 DEBUG:boto:auth_path=/dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.475 DEBUG:boto:Method: GET 2024-04-12T03:47:55.475 DEBUG:boto:Path: /dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.475 DEBUG:boto:Data: 2024-04-12T03:47:55.475 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.475 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.475 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.475 DEBUG:boto:Params: {} 2024-04-12T03:47:55.475 DEBUG:boto:Token: None 2024-04-12T03:47:55.475 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.476 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kxUxb+jNBbNWvoTi77q71QFTlWI= 2024-04-12T03:47:55.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kxUxb+jNBbNWvoTi77q71QFTlWI=', 'Content-Length': '0'} 2024-04-12T03:47:55.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c24bee4b04f1e2f-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.478 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:47:55.478 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:47:55.478 DEBUG:boto:path=/dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.478 DEBUG:boto:auth_path=/dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.478 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.478 DEBUG:boto:Path: /dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.478 DEBUG:boto:Data: 2024-04-12T03:47:55.479 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.479 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.479 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.479 DEBUG:boto:Params: {} 2024-04-12T03:47:55.479 DEBUG:boto:Token: None 2024-04-12T03:47:55.479 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.479 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:su/TM83zWwXg6JaARP/4Z6JPHpU= 2024-04-12T03:47:55.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:su/TM83zWwXg6JaARP/4Z6JPHpU=', 'Content-Length': '0'} 2024-04-12T03:47:55.481 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f72e11ca067adc3-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.481 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:47:55.481 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:47:55.481 DEBUG:boto:path=/dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.481 DEBUG:boto:auth_path=/dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.482 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.482 DEBUG:boto:Path: /dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.482 DEBUG:boto:Data: 2024-04-12T03:47:55.482 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.482 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.482 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.482 DEBUG:boto:Params: {} 2024-04-12T03:47:55.482 DEBUG:boto:Token: None 2024-04-12T03:47:55.483 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.483 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:su/TM83zWwXg6JaARP/4Z6JPHpU= 2024-04-12T03:47:55.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:su/TM83zWwXg6JaARP/4Z6JPHpU=', 'Content-Length': '0'} 2024-04-12T03:47:55.484 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6d5f0df941d6bf5-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:47:55.485 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:47:55.485 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:47:55.485 DEBUG:boto:path=/dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.485 DEBUG:boto:auth_path=/dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.485 DEBUG:boto:Method: GET 2024-04-12T03:47:55.485 DEBUG:boto:Path: /dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.485 DEBUG:boto:Data: 2024-04-12T03:47:55.485 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.486 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.486 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.486 DEBUG:boto:Params: {} 2024-04-12T03:47:55.486 DEBUG:boto:Token: None 2024-04-12T03:47:55.486 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.486 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kxUxb+jNBbNWvoTi77q71QFTlWI= 2024-04-12T03:47:55.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kxUxb+jNBbNWvoTi77q71QFTlWI=', 'Content-Length': '0'} 2024-04-12T03:47:55.488 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000700bd443710ba12f-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.488 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:47:55.489 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:47:55.489 DEBUG:boto:path=/dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.489 DEBUG:boto:auth_path=/dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.489 DEBUG:boto:Method: GET 2024-04-12T03:47:55.489 DEBUG:boto:Path: /dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.489 DEBUG:boto:Data: 2024-04-12T03:47:55.489 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.489 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.489 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.489 DEBUG:boto:Params: {} 2024-04-12T03:47:55.490 DEBUG:boto:Token: None 2024-04-12T03:47:55.490 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj4?versionId=null 2024-04-12T03:47:55.490 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kxUxb+jNBbNWvoTi77q71QFTlWI= 2024-04-12T03:47:55.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kxUxb+jNBbNWvoTi77q71QFTlWI=', 'Content-Length': '0'} 2024-04-12T03:47:55.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000850edbd410cb5b4e-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.492 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.492 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.492 DEBUG:boto:path=/dkyges-50/?prefix=obj4 2024-04-12T03:47:55.492 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj4 2024-04-12T03:47:55.492 DEBUG:boto:Method: GET 2024-04-12T03:47:55.492 DEBUG:boto:Path: /dkyges-50/?prefix=obj4 2024-04-12T03:47:55.492 DEBUG:boto:Data: 2024-04-12T03:47:55.492 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.493 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.493 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.493 DEBUG:boto:Params: {} 2024-04-12T03:47:55.493 DEBUG:boto:Token: None 2024-04-12T03:47:55.493 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.493 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE= 2024-04-12T03:47:55.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE=', 'Content-Length': '0'} 2024-04-12T03:47:55.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dddd60066a248971-006618aeeb-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.496 DEBUG:boto:b'dkyges-50obj41000falseobj42024-04-12T03:47:34.461Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.496 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.496 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.496 DEBUG:boto:path=/dkyges-50/?prefix=obj4 2024-04-12T03:47:55.497 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj4 2024-04-12T03:47:55.497 DEBUG:boto:Method: GET 2024-04-12T03:47:55.497 DEBUG:boto:Path: /dkyges-50/?prefix=obj4 2024-04-12T03:47:55.497 DEBUG:boto:Data: 2024-04-12T03:47:55.497 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.497 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.497 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.497 DEBUG:boto:Params: {} 2024-04-12T03:47:55.497 DEBUG:boto:Token: None 2024-04-12T03:47:55.497 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.498 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE= 2024-04-12T03:47:55.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE=', 'Content-Length': '0'} 2024-04-12T03:47:55.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069f367adae6b7592-006618aeeb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.500 DEBUG:boto:b'dkyges-50obj41000falseobj42024-04-12T03:47:34.461Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.500 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:47:55.500 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:47:55.501 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.501 DEBUG:boto:Path: /dkyges-50/obj4 2024-04-12T03:47:55.501 DEBUG:boto:Data: 2024-04-12T03:47:55.501 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.501 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.501 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.501 DEBUG:boto:Params: {} 2024-04-12T03:47:55.501 DEBUG:boto:Token: None 2024-04-12T03:47:55.501 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj4 2024-04-12T03:47:55.501 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gSYDVE1WezuL4HolNxXrH/K1e8Q= 2024-04-12T03:47:55.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gSYDVE1WezuL4HolNxXrH/K1e8Q=', 'Content-Length': '0'} 2024-04-12T03:47:55.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074527c8a55949f18-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.503 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:47:55.503 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:47:55.504 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.504 DEBUG:boto:Path: /dkyges-50/obj4 2024-04-12T03:47:55.504 DEBUG:boto:Data: 2024-04-12T03:47:55.504 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.504 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.504 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.504 DEBUG:boto:Params: {} 2024-04-12T03:47:55.504 DEBUG:boto:Token: None 2024-04-12T03:47:55.504 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj4 2024-04-12T03:47:55.504 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gSYDVE1WezuL4HolNxXrH/K1e8Q= 2024-04-12T03:47:55.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gSYDVE1WezuL4HolNxXrH/K1e8Q=', 'Content-Length': '0'} 2024-04-12T03:47:55.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000665ce7baa1711ca6-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:47:55.507 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:47:55.507 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:47:55.507 DEBUG:boto:Method: GET 2024-04-12T03:47:55.507 DEBUG:boto:Path: /dkyges-50/obj4 2024-04-12T03:47:55.507 DEBUG:boto:Data: 2024-04-12T03:47:55.507 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.507 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.507 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.507 DEBUG:boto:Params: {} 2024-04-12T03:47:55.507 DEBUG:boto:Token: None 2024-04-12T03:47:55.508 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj4 2024-04-12T03:47:55.508 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JZow3QYl7/aE4/38u4NPxsF0e3c= 2024-04-12T03:47:55.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JZow3QYl7/aE4/38u4NPxsF0e3c=', 'Content-Length': '0'} 2024-04-12T03:47:55.509 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000145efeb4ffcc95dd-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.554 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:47:55.555 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:47:55.555 DEBUG:boto:Method: GET 2024-04-12T03:47:55.555 DEBUG:boto:Path: /dkyges-50/obj4 2024-04-12T03:47:55.555 DEBUG:boto:Data: 2024-04-12T03:47:55.555 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.555 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.555 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.555 DEBUG:boto:Params: {} 2024-04-12T03:47:55.555 DEBUG:boto:Token: None 2024-04-12T03:47:55.555 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/obj4 2024-04-12T03:47:55.556 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JZow3QYl7/aE4/38u4NPxsF0e3c= 2024-04-12T03:47:55.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JZow3QYl7/aE4/38u4NPxsF0e3c=', 'Content-Length': '0'} 2024-04-12T03:47:55.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021b3b0f2317da1d1-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.558 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.558 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.558 DEBUG:boto:path=/dkyges-50/?prefix=obj4 2024-04-12T03:47:55.558 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj4 2024-04-12T03:47:55.558 DEBUG:boto:Method: GET 2024-04-12T03:47:55.559 DEBUG:boto:Path: /dkyges-50/?prefix=obj4 2024-04-12T03:47:55.559 DEBUG:boto:Data: 2024-04-12T03:47:55.559 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.559 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.559 DEBUG:boto:Params: {} 2024-04-12T03:47:55.559 DEBUG:boto:Token: None 2024-04-12T03:47:55.559 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.559 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE= 2024-04-12T03:47:55.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE=', 'Content-Length': '0'} 2024-04-12T03:47:55.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062ed6f9c98c10674-006618aeeb-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.562 DEBUG:boto:b'dkyges-50obj41000falseobj42024-04-12T03:47:34.461Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.562 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:47:55.563 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:47:55.563 DEBUG:boto:path=/dkyges-50/?prefix=obj4 2024-04-12T03:47:55.563 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj4 2024-04-12T03:47:55.563 DEBUG:boto:Method: GET 2024-04-12T03:47:55.563 DEBUG:boto:Path: /dkyges-50/?prefix=obj4 2024-04-12T03:47:55.563 DEBUG:boto:Data: 2024-04-12T03:47:55.563 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.563 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.563 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.563 DEBUG:boto:Params: {} 2024-04-12T03:47:55.564 DEBUG:boto:Token: None 2024-04-12T03:47:55.564 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-50/ 2024-04-12T03:47:55.564 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE= 2024-04-12T03:47:55.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zb/eJ+iAHblB9uDPR7oMtiacaRE=', 'Content-Length': '0'} 2024-04-12T03:47:55.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000751bfd39acada122-006618aeeb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.567 DEBUG:boto:b'dkyges-50obj41000falseobj42024-04-12T03:47:34.461Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.567 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-50 zones={a2, a1} 2024-04-12T03:47:55.568 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-51 2024-04-12T03:47:55.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-51', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:55.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-51', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:55.568 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-51 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:47:55.634 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:55.634 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:55.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:55.679+0000 7fcf5b85a780 1 bucket sync caught up with source: 2024-04-12T03:47:55.677 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.21.6, , 00000000001.32.6, , 00000000001.43.6, , , 00000000001.52.6, , , ] 2024-04-12T03:47:55.677 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.21.6, , 00000000001.32.6, , 00000000001.43.6, , , 00000000001.52.6, , , ] 2024-04-12T03:47:55.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:55.679+0000 7fcf5b85a780 0 bucket checkpoint complete 2024-04-12T03:47:55.693 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-51 2024-04-12T03:47:55.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-51', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:55.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-51', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:47:55.693 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-51 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:47:55.757 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:55.757 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:55.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:55.800+0000 7f49ebfe7780 1 bucket sync caught up with source: 2024-04-12T03:47:55.797 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.3035.6, , 00000000001.3206.6, , 00000000001.2412.6, , , 00000000001.1564.6, , , ] 2024-04-12T03:47:55.797 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.3035.6, , 00000000001.3206.6, , 00000000001.2412.6, , , 00000000001.1564.6, , , ] 2024-04-12T03:47:55.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:55.800+0000 7f49ebfe7780 0 bucket checkpoint complete 2024-04-12T03:47:55.812 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-51 zones={a2, a1} 2024-04-12T03:47:55.812 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:55.812 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:55.813 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.813 DEBUG:boto:Path: /dkyges-51/ 2024-04-12T03:47:55.813 DEBUG:boto:Data: 2024-04-12T03:47:55.813 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.813 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.813 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.813 DEBUG:boto:Params: {} 2024-04-12T03:47:55.813 DEBUG:boto:Token: None 2024-04-12T03:47:55.813 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/ 2024-04-12T03:47:55.814 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XssrzHlVVXJ/iaj7OeVVWcNJ4+s= 2024-04-12T03:47:55.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XssrzHlVVXJ/iaj7OeVVWcNJ4+s=', 'Content-Length': '0'} 2024-04-12T03:47:55.816 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0598adf8bf5e94e-006618aeeb-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.816 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:55.816 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:55.816 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.816 DEBUG:boto:Path: /dkyges-51/ 2024-04-12T03:47:55.816 DEBUG:boto:Data: 2024-04-12T03:47:55.817 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.817 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.817 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.817 DEBUG:boto:Params: {} 2024-04-12T03:47:55.817 DEBUG:boto:Token: None 2024-04-12T03:47:55.817 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/ 2024-04-12T03:47:55.817 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XssrzHlVVXJ/iaj7OeVVWcNJ4+s= 2024-04-12T03:47:55.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XssrzHlVVXJ/iaj7OeVVWcNJ4+s=', 'Content-Length': '0'} 2024-04-12T03:47:55.819 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c18e3b1bbb5f7f79-006618aeeb-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.820 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:47:55.820 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:55.820 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:55.820 DEBUG:boto:path=/dkyges-51/?versions&max-keys=999 2024-04-12T03:47:55.820 DEBUG:boto:auth_path=/dkyges-51/?versions&max-keys=999 2024-04-12T03:47:55.820 DEBUG:boto:Method: GET 2024-04-12T03:47:55.820 DEBUG:boto:Path: /dkyges-51/?versions&max-keys=999 2024-04-12T03:47:55.821 DEBUG:boto:Data: 2024-04-12T03:47:55.821 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.821 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.821 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.821 DEBUG:boto:Params: {} 2024-04-12T03:47:55.821 DEBUG:boto:Token: None 2024-04-12T03:47:55.821 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/?versions 2024-04-12T03:47:55.821 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5Q4Lwg9StSL+LClWbHuGq2LbIbM= 2024-04-12T03:47:55.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5Q4Lwg9StSL+LClWbHuGq2LbIbM=', 'Content-Length': '0'} 2024-04-12T03:47:55.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025747551ebb3896b-006618aeeb-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.824 DEBUG:boto:b'dkyges-51999falseobj1nulltrue2024-04-12T03:47:34.473Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:47:34.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:47:34.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:47:34.552Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:47:55.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:47:55.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:47:55.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:47:55.825 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:47:55.825 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:55.825 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:55.825 DEBUG:boto:path=/dkyges-51/?versions&max-keys=999 2024-04-12T03:47:55.825 DEBUG:boto:auth_path=/dkyges-51/?versions&max-keys=999 2024-04-12T03:47:55.825 DEBUG:boto:Method: GET 2024-04-12T03:47:55.826 DEBUG:boto:Path: /dkyges-51/?versions&max-keys=999 2024-04-12T03:47:55.826 DEBUG:boto:Data: 2024-04-12T03:47:55.826 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.826 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.826 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.826 DEBUG:boto:Params: {} 2024-04-12T03:47:55.826 DEBUG:boto:Token: None 2024-04-12T03:47:55.826 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/?versions 2024-04-12T03:47:55.826 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5Q4Lwg9StSL+LClWbHuGq2LbIbM= 2024-04-12T03:47:55.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5Q4Lwg9StSL+LClWbHuGq2LbIbM=', 'Content-Length': '0'} 2024-04-12T03:47:55.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000001fc255499fed58-006618aeeb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.829 DEBUG:boto:b'dkyges-51999falseobj1nulltrue2024-04-12T03:47:34.473Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:47:34.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:47:34.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:47:34.552Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:47:55.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:47:55.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:47:55.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:47:55.830 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:55.830 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:55.830 DEBUG:boto:path=/dkyges-51/?versions&max-keys=999 2024-04-12T03:47:55.830 DEBUG:boto:auth_path=/dkyges-51/?versions&max-keys=999 2024-04-12T03:47:55.830 DEBUG:boto:Method: GET 2024-04-12T03:47:55.830 DEBUG:boto:Path: /dkyges-51/?versions&max-keys=999 2024-04-12T03:47:55.830 DEBUG:boto:Data: 2024-04-12T03:47:55.830 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.831 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.831 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.831 DEBUG:boto:Params: {} 2024-04-12T03:47:55.831 DEBUG:boto:Token: None 2024-04-12T03:47:55.831 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/?versions 2024-04-12T03:47:55.831 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5Q4Lwg9StSL+LClWbHuGq2LbIbM= 2024-04-12T03:47:55.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5Q4Lwg9StSL+LClWbHuGq2LbIbM=', 'Content-Length': '0'} 2024-04-12T03:47:55.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f275d24960ae3095-006618aeeb-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.834 DEBUG:boto:b'dkyges-51999falseobj1nulltrue2024-04-12T03:47:34.473Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:47:34.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:47:34.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:47:34.552Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.834 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:55.834 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:55.834 DEBUG:boto:path=/dkyges-51/?versions&max-keys=999 2024-04-12T03:47:55.835 DEBUG:boto:auth_path=/dkyges-51/?versions&max-keys=999 2024-04-12T03:47:55.835 DEBUG:boto:Method: GET 2024-04-12T03:47:55.835 DEBUG:boto:Path: /dkyges-51/?versions&max-keys=999 2024-04-12T03:47:55.835 DEBUG:boto:Data: 2024-04-12T03:47:55.835 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.835 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.835 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.836 DEBUG:boto:Params: {} 2024-04-12T03:47:55.836 DEBUG:boto:Token: None 2024-04-12T03:47:55.836 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/?versions 2024-04-12T03:47:55.836 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5Q4Lwg9StSL+LClWbHuGq2LbIbM= 2024-04-12T03:47:55.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5Q4Lwg9StSL+LClWbHuGq2LbIbM=', 'Content-Length': '0'} 2024-04-12T03:47:55.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000527050324d826141-006618aeeb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.839 DEBUG:boto:b'dkyges-51999falseobj1nulltrue2024-04-12T03:47:34.473Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:47:34.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:47:34.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:47:34.552Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:47:55.839 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:47:55.840 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:47:55.840 DEBUG:boto:path=/dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.840 DEBUG:boto:auth_path=/dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.840 DEBUG:boto:Method: GET 2024-04-12T03:47:55.840 DEBUG:boto:Path: /dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.840 DEBUG:boto:Data: 2024-04-12T03:47:55.840 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.840 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.841 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.841 DEBUG:boto:Params: {} 2024-04-12T03:47:55.841 DEBUG:boto:Token: None 2024-04-12T03:47:55.841 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.841 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ts3peqh17xkkeDdVqIZmziTbUyc= 2024-04-12T03:47:55.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ts3peqh17xkkeDdVqIZmziTbUyc=', 'Content-Length': '0'} 2024-04-12T03:47:55.843 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004a7869e024fd8dbb-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.843 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:47:55.843 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:47:55.844 DEBUG:boto:path=/dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.844 DEBUG:boto:auth_path=/dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.844 DEBUG:boto:Method: GET 2024-04-12T03:47:55.844 DEBUG:boto:Path: /dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.844 DEBUG:boto:Data: 2024-04-12T03:47:55.844 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.844 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.844 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.844 DEBUG:boto:Params: {} 2024-04-12T03:47:55.844 DEBUG:boto:Token: None 2024-04-12T03:47:55.845 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.845 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ts3peqh17xkkeDdVqIZmziTbUyc= 2024-04-12T03:47:55.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ts3peqh17xkkeDdVqIZmziTbUyc=', 'Content-Length': '0'} 2024-04-12T03:47:55.847 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004aa14e589db9423a-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.894 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:47:55.894 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:47:55.894 DEBUG:boto:path=/dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.894 DEBUG:boto:auth_path=/dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.894 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.894 DEBUG:boto:Path: /dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.894 DEBUG:boto:Data: 2024-04-12T03:47:55.894 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.894 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.894 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.895 DEBUG:boto:Params: {} 2024-04-12T03:47:55.895 DEBUG:boto:Token: None 2024-04-12T03:47:55.895 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.895 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4WstkBrR/xIK+LvVJBs+wRuogN0= 2024-04-12T03:47:55.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4WstkBrR/xIK+LvVJBs+wRuogN0=', 'Content-Length': '0'} 2024-04-12T03:47:55.897 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038151665808f26a9-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.897 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:47:55.898 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:47:55.898 DEBUG:boto:path=/dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.898 DEBUG:boto:auth_path=/dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.898 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.898 DEBUG:boto:Path: /dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.898 DEBUG:boto:Data: 2024-04-12T03:47:55.898 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.898 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.898 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.898 DEBUG:boto:Params: {} 2024-04-12T03:47:55.898 DEBUG:boto:Token: None 2024-04-12T03:47:55.899 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.899 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4WstkBrR/xIK+LvVJBs+wRuogN0= 2024-04-12T03:47:55.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4WstkBrR/xIK+LvVJBs+wRuogN0=', 'Content-Length': '0'} 2024-04-12T03:47:55.901 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cbfe4dd2196f6ac-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:47:55.901 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:47:55.901 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:47:55.901 DEBUG:boto:path=/dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.901 DEBUG:boto:auth_path=/dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.902 DEBUG:boto:Method: GET 2024-04-12T03:47:55.902 DEBUG:boto:Path: /dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.902 DEBUG:boto:Data: 2024-04-12T03:47:55.902 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.902 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.902 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.902 DEBUG:boto:Params: {} 2024-04-12T03:47:55.902 DEBUG:boto:Token: None 2024-04-12T03:47:55.902 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.902 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ts3peqh17xkkeDdVqIZmziTbUyc= 2024-04-12T03:47:55.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ts3peqh17xkkeDdVqIZmziTbUyc=', 'Content-Length': '0'} 2024-04-12T03:47:55.904 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f6c81ce3bcaa37c-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.905 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:47:55.905 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:47:55.905 DEBUG:boto:path=/dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.905 DEBUG:boto:auth_path=/dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.905 DEBUG:boto:Method: GET 2024-04-12T03:47:55.906 DEBUG:boto:Path: /dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.906 DEBUG:boto:Data: 2024-04-12T03:47:55.906 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.906 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.906 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.907 DEBUG:boto:Params: {} 2024-04-12T03:47:55.907 DEBUG:boto:Token: None 2024-04-12T03:47:55.907 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/obj1?versionId=null 2024-04-12T03:47:55.907 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ts3peqh17xkkeDdVqIZmziTbUyc= 2024-04-12T03:47:55.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ts3peqh17xkkeDdVqIZmziTbUyc=', 'Content-Length': '0'} 2024-04-12T03:47:55.909 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a41f253be110eb3b-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.909 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:55.909 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:55.910 DEBUG:boto:path=/dkyges-51/?prefix=obj1 2024-04-12T03:47:55.910 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj1 2024-04-12T03:47:55.910 DEBUG:boto:Method: GET 2024-04-12T03:47:55.910 DEBUG:boto:Path: /dkyges-51/?prefix=obj1 2024-04-12T03:47:55.910 DEBUG:boto:Data: 2024-04-12T03:47:55.910 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.910 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.910 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.910 DEBUG:boto:Params: {} 2024-04-12T03:47:55.910 DEBUG:boto:Token: None 2024-04-12T03:47:55.911 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/ 2024-04-12T03:47:55.911 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V40bHS8JO2P0hLL1S0eAA4jJHCA= 2024-04-12T03:47:55.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V40bHS8JO2P0hLL1S0eAA4jJHCA=', 'Content-Length': '0'} 2024-04-12T03:47:55.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ab3f359ad8c7797-006618aeeb-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.914 DEBUG:boto:b'dkyges-51obj11000falseobj12024-04-12T03:47:34.473Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.914 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:55.914 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:55.914 DEBUG:boto:path=/dkyges-51/?prefix=obj1 2024-04-12T03:47:55.915 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj1 2024-04-12T03:47:55.915 DEBUG:boto:Method: GET 2024-04-12T03:47:55.915 DEBUG:boto:Path: /dkyges-51/?prefix=obj1 2024-04-12T03:47:55.916 DEBUG:boto:Data: 2024-04-12T03:47:55.916 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.916 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.916 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.916 DEBUG:boto:Params: {} 2024-04-12T03:47:55.916 DEBUG:boto:Token: None 2024-04-12T03:47:55.917 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/ 2024-04-12T03:47:55.917 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V40bHS8JO2P0hLL1S0eAA4jJHCA= 2024-04-12T03:47:55.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V40bHS8JO2P0hLL1S0eAA4jJHCA=', 'Content-Length': '0'} 2024-04-12T03:47:55.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c58f45c46c21136f-006618aeeb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.920 DEBUG:boto:b'dkyges-51obj11000falseobj12024-04-12T03:47:34.473Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.920 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:47:55.920 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:47:55.920 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.921 DEBUG:boto:Path: /dkyges-51/obj1 2024-04-12T03:47:55.921 DEBUG:boto:Data: 2024-04-12T03:47:55.921 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.921 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.921 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.921 DEBUG:boto:Params: {} 2024-04-12T03:47:55.921 DEBUG:boto:Token: None 2024-04-12T03:47:55.921 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/obj1 2024-04-12T03:47:55.921 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hyz6CqbtqiJ4F2llROXNDjC5rw8= 2024-04-12T03:47:55.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hyz6CqbtqiJ4F2llROXNDjC5rw8=', 'Content-Length': '0'} 2024-04-12T03:47:55.923 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005195cc22bcc44566-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.923 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:47:55.923 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:47:55.924 DEBUG:boto:Method: HEAD 2024-04-12T03:47:55.924 DEBUG:boto:Path: /dkyges-51/obj1 2024-04-12T03:47:55.924 DEBUG:boto:Data: 2024-04-12T03:47:55.924 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.924 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.924 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.924 DEBUG:boto:Params: {} 2024-04-12T03:47:55.924 DEBUG:boto:Token: None 2024-04-12T03:47:55.924 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/obj1 2024-04-12T03:47:55.924 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hyz6CqbtqiJ4F2llROXNDjC5rw8= 2024-04-12T03:47:55.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hyz6CqbtqiJ4F2llROXNDjC5rw8=', 'Content-Length': '0'} 2024-04-12T03:47:55.926 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e31464ac136cd8c7-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:47:55.927 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:47:55.927 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:47:55.927 DEBUG:boto:Method: GET 2024-04-12T03:47:55.927 DEBUG:boto:Path: /dkyges-51/obj1 2024-04-12T03:47:55.927 DEBUG:boto:Data: 2024-04-12T03:47:55.927 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.927 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.927 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.928 DEBUG:boto:Params: {} 2024-04-12T03:47:55.928 DEBUG:boto:Token: None 2024-04-12T03:47:55.928 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/obj1 2024-04-12T03:47:55.928 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rrGa0yzPjVeMrFyyVLrPCMzuRWE= 2024-04-12T03:47:55.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rrGa0yzPjVeMrFyyVLrPCMzuRWE=', 'Content-Length': '0'} 2024-04-12T03:47:55.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000574723aaa1d1eee7-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.970 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:47:55.970 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:47:55.971 DEBUG:boto:Method: GET 2024-04-12T03:47:55.971 DEBUG:boto:Path: /dkyges-51/obj1 2024-04-12T03:47:55.971 DEBUG:boto:Data: 2024-04-12T03:47:55.971 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.971 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.971 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.971 DEBUG:boto:Params: {} 2024-04-12T03:47:55.971 DEBUG:boto:Token: None 2024-04-12T03:47:55.971 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/obj1 2024-04-12T03:47:55.972 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rrGa0yzPjVeMrFyyVLrPCMzuRWE= 2024-04-12T03:47:55.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rrGa0yzPjVeMrFyyVLrPCMzuRWE=', 'Content-Length': '0'} 2024-04-12T03:47:55.974 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005131b32fe710b5c1-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.974 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:55.974 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:55.974 DEBUG:boto:path=/dkyges-51/?prefix=obj1 2024-04-12T03:47:55.974 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj1 2024-04-12T03:47:55.975 DEBUG:boto:Method: GET 2024-04-12T03:47:55.975 DEBUG:boto:Path: /dkyges-51/?prefix=obj1 2024-04-12T03:47:55.975 DEBUG:boto:Data: 2024-04-12T03:47:55.975 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.975 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.975 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.975 DEBUG:boto:Params: {} 2024-04-12T03:47:55.975 DEBUG:boto:Token: None 2024-04-12T03:47:55.975 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/ 2024-04-12T03:47:55.975 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V40bHS8JO2P0hLL1S0eAA4jJHCA= 2024-04-12T03:47:55.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V40bHS8JO2P0hLL1S0eAA4jJHCA=', 'Content-Length': '0'} 2024-04-12T03:47:55.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077c5068ef150d56e-006618aeeb-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.978 DEBUG:boto:b'dkyges-51obj11000falseobj12024-04-12T03:47:34.473Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.979 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:55.979 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:55.979 DEBUG:boto:path=/dkyges-51/?prefix=obj1 2024-04-12T03:47:55.979 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj1 2024-04-12T03:47:55.979 DEBUG:boto:Method: GET 2024-04-12T03:47:55.979 DEBUG:boto:Path: /dkyges-51/?prefix=obj1 2024-04-12T03:47:55.979 DEBUG:boto:Data: 2024-04-12T03:47:55.979 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.980 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.980 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.980 DEBUG:boto:Params: {} 2024-04-12T03:47:55.980 DEBUG:boto:Token: None 2024-04-12T03:47:55.980 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/ 2024-04-12T03:47:55.980 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V40bHS8JO2P0hLL1S0eAA4jJHCA= 2024-04-12T03:47:55.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V40bHS8JO2P0hLL1S0eAA4jJHCA=', 'Content-Length': '0'} 2024-04-12T03:47:55.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000573a2ce4fbcb8943-006618aeeb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.983 DEBUG:boto:b'dkyges-51obj11000falseobj12024-04-12T03:47:34.473Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:55.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:47:55.984 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:47:55.984 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:47:55.984 DEBUG:boto:path=/dkyges-51/obj2?versionId=null 2024-04-12T03:47:55.984 DEBUG:boto:auth_path=/dkyges-51/obj2?versionId=null 2024-04-12T03:47:55.984 DEBUG:boto:Method: GET 2024-04-12T03:47:55.984 DEBUG:boto:Path: /dkyges-51/obj2?versionId=null 2024-04-12T03:47:55.984 DEBUG:boto:Data: 2024-04-12T03:47:55.984 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.985 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:55.985 DEBUG:boto:Port: 8001 2024-04-12T03:47:55.985 DEBUG:boto:Params: {} 2024-04-12T03:47:55.985 DEBUG:boto:Token: None 2024-04-12T03:47:55.985 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/obj2?versionId=null 2024-04-12T03:47:55.985 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:217dlofJpgGW/b//6/ne7V5Tnzg= 2024-04-12T03:47:55.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:217dlofJpgGW/b//6/ne7V5Tnzg=', 'Content-Length': '0'} 2024-04-12T03:47:55.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024683b635c857c41-006618aeeb-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:55.988 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:47:55.988 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:47:55.988 DEBUG:boto:path=/dkyges-51/obj2?versionId=null 2024-04-12T03:47:55.988 DEBUG:boto:auth_path=/dkyges-51/obj2?versionId=null 2024-04-12T03:47:55.988 DEBUG:boto:Method: GET 2024-04-12T03:47:55.989 DEBUG:boto:Path: /dkyges-51/obj2?versionId=null 2024-04-12T03:47:55.989 DEBUG:boto:Data: 2024-04-12T03:47:55.989 DEBUG:boto:Headers: {} 2024-04-12T03:47:55.989 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:55.989 DEBUG:boto:Port: 8000 2024-04-12T03:47:55.989 DEBUG:boto:Params: {} 2024-04-12T03:47:55.989 DEBUG:boto:Token: None 2024-04-12T03:47:55.989 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:55 GMT /dkyges-51/obj2?versionId=null 2024-04-12T03:47:55.989 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:217dlofJpgGW/b//6/ne7V5Tnzg= 2024-04-12T03:47:55.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:217dlofJpgGW/b//6/ne7V5Tnzg=', 'Content-Length': '0'} 2024-04-12T03:47:55.991 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d1e4071adb904696-006618aeeb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.034 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:47:56.035 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:47:56.035 DEBUG:boto:path=/dkyges-51/obj2?versionId=null 2024-04-12T03:47:56.035 DEBUG:boto:auth_path=/dkyges-51/obj2?versionId=null 2024-04-12T03:47:56.035 DEBUG:boto:Method: HEAD 2024-04-12T03:47:56.035 DEBUG:boto:Path: /dkyges-51/obj2?versionId=null 2024-04-12T03:47:56.035 DEBUG:boto:Data: 2024-04-12T03:47:56.035 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.035 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.035 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.036 DEBUG:boto:Params: {} 2024-04-12T03:47:56.036 DEBUG:boto:Token: None 2024-04-12T03:47:56.036 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj2?versionId=null 2024-04-12T03:47:56.036 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1WBGh0icT7AB8UBY0IZcaM9V91I= 2024-04-12T03:47:56.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1WBGh0icT7AB8UBY0IZcaM9V91I=', 'Content-Length': '0'} 2024-04-12T03:47:56.038 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4f791557ef2909f-006618aeec-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.039 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:47:56.039 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:47:56.039 DEBUG:boto:path=/dkyges-51/obj2?versionId=null 2024-04-12T03:47:56.039 DEBUG:boto:auth_path=/dkyges-51/obj2?versionId=null 2024-04-12T03:47:56.039 DEBUG:boto:Method: HEAD 2024-04-12T03:47:56.039 DEBUG:boto:Path: /dkyges-51/obj2?versionId=null 2024-04-12T03:47:56.039 DEBUG:boto:Data: 2024-04-12T03:47:56.039 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.040 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.040 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.040 DEBUG:boto:Params: {} 2024-04-12T03:47:56.040 DEBUG:boto:Token: None 2024-04-12T03:47:56.040 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj2?versionId=null 2024-04-12T03:47:56.040 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1WBGh0icT7AB8UBY0IZcaM9V91I= 2024-04-12T03:47:56.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1WBGh0icT7AB8UBY0IZcaM9V91I=', 'Content-Length': '0'} 2024-04-12T03:47:56.042 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001132c7caf4eaf4f8-006618aeec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:47:56.043 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:47:56.043 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:47:56.043 DEBUG:boto:path=/dkyges-51/obj2?versionId=null 2024-04-12T03:47:56.043 DEBUG:boto:auth_path=/dkyges-51/obj2?versionId=null 2024-04-12T03:47:56.043 DEBUG:boto:Method: GET 2024-04-12T03:47:56.043 DEBUG:boto:Path: /dkyges-51/obj2?versionId=null 2024-04-12T03:47:56.043 DEBUG:boto:Data: 2024-04-12T03:47:56.043 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.044 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.044 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.044 DEBUG:boto:Params: {} 2024-04-12T03:47:56.044 DEBUG:boto:Token: None 2024-04-12T03:47:56.044 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj2?versionId=null 2024-04-12T03:47:56.045 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jkC5JbmCmMAYtC+wJQJ4Qvu5gPI= 2024-04-12T03:47:56.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jkC5JbmCmMAYtC+wJQJ4Qvu5gPI=', 'Content-Length': '0'} 2024-04-12T03:47:56.047 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc841b7d1843f956-006618aeec-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.047 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:47:56.047 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:47:56.047 DEBUG:boto:path=/dkyges-51/obj2?versionId=null 2024-04-12T03:47:56.048 DEBUG:boto:auth_path=/dkyges-51/obj2?versionId=null 2024-04-12T03:47:56.048 DEBUG:boto:Method: GET 2024-04-12T03:47:56.048 DEBUG:boto:Path: /dkyges-51/obj2?versionId=null 2024-04-12T03:47:56.048 DEBUG:boto:Data: 2024-04-12T03:47:56.048 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.048 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.048 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.048 DEBUG:boto:Params: {} 2024-04-12T03:47:56.049 DEBUG:boto:Token: None 2024-04-12T03:47:56.049 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj2?versionId=null 2024-04-12T03:47:56.049 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jkC5JbmCmMAYtC+wJQJ4Qvu5gPI= 2024-04-12T03:47:56.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jkC5JbmCmMAYtC+wJQJ4Qvu5gPI=', 'Content-Length': '0'} 2024-04-12T03:47:56.051 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e07d39bbf8c15b19-006618aeec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.051 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:56.051 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:56.051 DEBUG:boto:path=/dkyges-51/?prefix=obj2 2024-04-12T03:47:56.052 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj2 2024-04-12T03:47:56.052 DEBUG:boto:Method: GET 2024-04-12T03:47:56.052 DEBUG:boto:Path: /dkyges-51/?prefix=obj2 2024-04-12T03:47:56.052 DEBUG:boto:Data: 2024-04-12T03:47:56.052 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.052 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.052 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.052 DEBUG:boto:Params: {} 2024-04-12T03:47:56.052 DEBUG:boto:Token: None 2024-04-12T03:47:56.052 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/ 2024-04-12T03:47:56.053 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0= 2024-04-12T03:47:56.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0=', 'Content-Length': '0'} 2024-04-12T03:47:56.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d277f21537906317-006618aeec-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.098 DEBUG:boto:b'dkyges-51obj21000falseobj22024-04-12T03:47:34.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:56.099 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:56.099 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:56.099 DEBUG:boto:path=/dkyges-51/?prefix=obj2 2024-04-12T03:47:56.099 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj2 2024-04-12T03:47:56.099 DEBUG:boto:Method: GET 2024-04-12T03:47:56.099 DEBUG:boto:Path: /dkyges-51/?prefix=obj2 2024-04-12T03:47:56.099 DEBUG:boto:Data: 2024-04-12T03:47:56.100 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.100 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.100 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.100 DEBUG:boto:Params: {} 2024-04-12T03:47:56.100 DEBUG:boto:Token: None 2024-04-12T03:47:56.100 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/ 2024-04-12T03:47:56.100 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0= 2024-04-12T03:47:56.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0=', 'Content-Length': '0'} 2024-04-12T03:47:56.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081db7a2758f59503-006618aeec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.103 DEBUG:boto:b'dkyges-51obj21000falseobj22024-04-12T03:47:34.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:56.103 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:47:56.104 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:47:56.104 DEBUG:boto:Method: HEAD 2024-04-12T03:47:56.104 DEBUG:boto:Path: /dkyges-51/obj2 2024-04-12T03:47:56.104 DEBUG:boto:Data: 2024-04-12T03:47:56.104 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.104 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.104 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.104 DEBUG:boto:Params: {} 2024-04-12T03:47:56.104 DEBUG:boto:Token: None 2024-04-12T03:47:56.105 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj2 2024-04-12T03:47:56.105 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZxLoN1H0kvFrVNHF5naHt3lNW/Y= 2024-04-12T03:47:56.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZxLoN1H0kvFrVNHF5naHt3lNW/Y=', 'Content-Length': '0'} 2024-04-12T03:47:56.106 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b319607f41b9e841-006618aeec-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.107 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:47:56.107 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:47:56.107 DEBUG:boto:Method: HEAD 2024-04-12T03:47:56.107 DEBUG:boto:Path: /dkyges-51/obj2 2024-04-12T03:47:56.107 DEBUG:boto:Data: 2024-04-12T03:47:56.107 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.107 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.107 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.107 DEBUG:boto:Params: {} 2024-04-12T03:47:56.108 DEBUG:boto:Token: None 2024-04-12T03:47:56.108 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj2 2024-04-12T03:47:56.109 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZxLoN1H0kvFrVNHF5naHt3lNW/Y= 2024-04-12T03:47:56.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZxLoN1H0kvFrVNHF5naHt3lNW/Y=', 'Content-Length': '0'} 2024-04-12T03:47:56.111 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000c5ba26376962996-006618aeec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:47:56.111 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:47:56.111 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:47:56.111 DEBUG:boto:Method: GET 2024-04-12T03:47:56.111 DEBUG:boto:Path: /dkyges-51/obj2 2024-04-12T03:47:56.111 DEBUG:boto:Data: 2024-04-12T03:47:56.111 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.112 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.112 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.112 DEBUG:boto:Params: {} 2024-04-12T03:47:56.112 DEBUG:boto:Token: None 2024-04-12T03:47:56.112 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj2 2024-04-12T03:47:56.112 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rP9Q+ETDx4thBqxTWutDhXPhckY= 2024-04-12T03:47:56.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rP9Q+ETDx4thBqxTWutDhXPhckY=', 'Content-Length': '0'} 2024-04-12T03:47:56.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7da6c9daabbad92-006618aeec-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.115 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:47:56.115 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:47:56.115 DEBUG:boto:Method: GET 2024-04-12T03:47:56.115 DEBUG:boto:Path: /dkyges-51/obj2 2024-04-12T03:47:56.115 DEBUG:boto:Data: 2024-04-12T03:47:56.115 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.115 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.115 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.115 DEBUG:boto:Params: {} 2024-04-12T03:47:56.116 DEBUG:boto:Token: None 2024-04-12T03:47:56.116 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj2 2024-04-12T03:47:56.116 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rP9Q+ETDx4thBqxTWutDhXPhckY= 2024-04-12T03:47:56.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rP9Q+ETDx4thBqxTWutDhXPhckY=', 'Content-Length': '0'} 2024-04-12T03:47:56.117 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004803a32a2d8cf89a-006618aeec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.158 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:56.159 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:56.159 DEBUG:boto:path=/dkyges-51/?prefix=obj2 2024-04-12T03:47:56.159 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj2 2024-04-12T03:47:56.159 DEBUG:boto:Method: GET 2024-04-12T03:47:56.159 DEBUG:boto:Path: /dkyges-51/?prefix=obj2 2024-04-12T03:47:56.159 DEBUG:boto:Data: 2024-04-12T03:47:56.159 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.159 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.159 DEBUG:boto:Params: {} 2024-04-12T03:47:56.160 DEBUG:boto:Token: None 2024-04-12T03:47:56.160 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/ 2024-04-12T03:47:56.160 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0= 2024-04-12T03:47:56.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0=', 'Content-Length': '0'} 2024-04-12T03:47:56.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041f84acf200ed719-006618aeec-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.162 DEBUG:boto:b'dkyges-51obj21000falseobj22024-04-12T03:47:34.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:56.163 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:56.163 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:56.163 DEBUG:boto:path=/dkyges-51/?prefix=obj2 2024-04-12T03:47:56.163 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj2 2024-04-12T03:47:56.163 DEBUG:boto:Method: GET 2024-04-12T03:47:56.163 DEBUG:boto:Path: /dkyges-51/?prefix=obj2 2024-04-12T03:47:56.163 DEBUG:boto:Data: 2024-04-12T03:47:56.164 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.164 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.164 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.164 DEBUG:boto:Params: {} 2024-04-12T03:47:56.164 DEBUG:boto:Token: None 2024-04-12T03:47:56.164 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/ 2024-04-12T03:47:56.164 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0= 2024-04-12T03:47:56.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0=', 'Content-Length': '0'} 2024-04-12T03:47:56.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d04e1e86305b34f-006618aeec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.166 DEBUG:boto:b'dkyges-51obj21000falseobj22024-04-12T03:47:34.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:56.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:47:56.167 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:47:56.167 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:47:56.167 DEBUG:boto:path=/dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.167 DEBUG:boto:auth_path=/dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.167 DEBUG:boto:Method: GET 2024-04-12T03:47:56.167 DEBUG:boto:Path: /dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.168 DEBUG:boto:Data: 2024-04-12T03:47:56.168 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.168 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.168 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.168 DEBUG:boto:Params: {} 2024-04-12T03:47:56.168 DEBUG:boto:Token: None 2024-04-12T03:47:56.168 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.168 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:22GC6FPCiDgmO+N4nNuBQJF0PSo= 2024-04-12T03:47:56.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:22GC6FPCiDgmO+N4nNuBQJF0PSo=', 'Content-Length': '0'} 2024-04-12T03:47:56.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001969b586f072e9bd-006618aeec-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.218 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:47:56.219 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:47:56.219 DEBUG:boto:path=/dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.219 DEBUG:boto:auth_path=/dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.219 DEBUG:boto:Method: GET 2024-04-12T03:47:56.219 DEBUG:boto:Path: /dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.219 DEBUG:boto:Data: 2024-04-12T03:47:56.219 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.219 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.219 DEBUG:boto:Params: {} 2024-04-12T03:47:56.219 DEBUG:boto:Token: None 2024-04-12T03:47:56.220 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.220 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:22GC6FPCiDgmO+N4nNuBQJF0PSo= 2024-04-12T03:47:56.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:22GC6FPCiDgmO+N4nNuBQJF0PSo=', 'Content-Length': '0'} 2024-04-12T03:47:56.222 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007636b14ba88ae578-006618aeec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.222 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:47:56.222 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:47:56.222 DEBUG:boto:path=/dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.223 DEBUG:boto:auth_path=/dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.223 DEBUG:boto:Method: HEAD 2024-04-12T03:47:56.223 DEBUG:boto:Path: /dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.223 DEBUG:boto:Data: 2024-04-12T03:47:56.223 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.223 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.223 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.223 DEBUG:boto:Params: {} 2024-04-12T03:47:56.223 DEBUG:boto:Token: None 2024-04-12T03:47:56.223 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.224 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hlDPh3yYIm5gd+IDH/Csy84qyXc= 2024-04-12T03:47:56.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hlDPh3yYIm5gd+IDH/Csy84qyXc=', 'Content-Length': '0'} 2024-04-12T03:47:56.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c34c58bd057db672-006618aeec-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.226 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:47:56.226 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:47:56.226 DEBUG:boto:path=/dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.226 DEBUG:boto:auth_path=/dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.226 DEBUG:boto:Method: HEAD 2024-04-12T03:47:56.226 DEBUG:boto:Path: /dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.226 DEBUG:boto:Data: 2024-04-12T03:47:56.226 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.226 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.227 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.227 DEBUG:boto:Params: {} 2024-04-12T03:47:56.227 DEBUG:boto:Token: None 2024-04-12T03:47:56.227 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.227 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hlDPh3yYIm5gd+IDH/Csy84qyXc= 2024-04-12T03:47:56.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hlDPh3yYIm5gd+IDH/Csy84qyXc=', 'Content-Length': '0'} 2024-04-12T03:47:56.229 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d294bb48fafee12-006618aeec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:47:56.229 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:47:56.230 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:47:56.230 DEBUG:boto:path=/dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.230 DEBUG:boto:auth_path=/dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.230 DEBUG:boto:Method: GET 2024-04-12T03:47:56.230 DEBUG:boto:Path: /dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.230 DEBUG:boto:Data: 2024-04-12T03:47:56.230 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.230 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.230 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.231 DEBUG:boto:Params: {} 2024-04-12T03:47:56.231 DEBUG:boto:Token: None 2024-04-12T03:47:56.231 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.231 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:22GC6FPCiDgmO+N4nNuBQJF0PSo= 2024-04-12T03:47:56.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:22GC6FPCiDgmO+N4nNuBQJF0PSo=', 'Content-Length': '0'} 2024-04-12T03:47:56.233 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b9d95ab1f26713d-006618aeec-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.233 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:47:56.233 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:47:56.233 DEBUG:boto:path=/dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.233 DEBUG:boto:auth_path=/dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.233 DEBUG:boto:Method: GET 2024-04-12T03:47:56.234 DEBUG:boto:Path: /dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.234 DEBUG:boto:Data: 2024-04-12T03:47:56.234 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.234 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.234 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.234 DEBUG:boto:Params: {} 2024-04-12T03:47:56.234 DEBUG:boto:Token: None 2024-04-12T03:47:56.234 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj3?versionId=null 2024-04-12T03:47:56.234 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:22GC6FPCiDgmO+N4nNuBQJF0PSo= 2024-04-12T03:47:56.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:22GC6FPCiDgmO+N4nNuBQJF0PSo=', 'Content-Length': '0'} 2024-04-12T03:47:56.237 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000782a2aa8500651b5-006618aeec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.237 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:56.237 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:56.237 DEBUG:boto:path=/dkyges-51/?prefix=obj3 2024-04-12T03:47:56.237 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj3 2024-04-12T03:47:56.237 DEBUG:boto:Method: GET 2024-04-12T03:47:56.238 DEBUG:boto:Path: /dkyges-51/?prefix=obj3 2024-04-12T03:47:56.238 DEBUG:boto:Data: 2024-04-12T03:47:56.238 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.238 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.238 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.238 DEBUG:boto:Params: {} 2024-04-12T03:47:56.238 DEBUG:boto:Token: None 2024-04-12T03:47:56.238 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/ 2024-04-12T03:47:56.239 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0= 2024-04-12T03:47:56.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0=', 'Content-Length': '0'} 2024-04-12T03:47:56.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2d2d9e11674e93f-006618aeec-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.241 DEBUG:boto:b'dkyges-51obj31000falseobj32024-04-12T03:47:34.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:56.242 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:56.242 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:56.242 DEBUG:boto:path=/dkyges-51/?prefix=obj3 2024-04-12T03:47:56.242 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj3 2024-04-12T03:47:56.242 DEBUG:boto:Method: GET 2024-04-12T03:47:56.242 DEBUG:boto:Path: /dkyges-51/?prefix=obj3 2024-04-12T03:47:56.242 DEBUG:boto:Data: 2024-04-12T03:47:56.243 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.243 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.243 DEBUG:boto:Params: {} 2024-04-12T03:47:56.243 DEBUG:boto:Token: None 2024-04-12T03:47:56.243 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/ 2024-04-12T03:47:56.243 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0= 2024-04-12T03:47:56.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0=', 'Content-Length': '0'} 2024-04-12T03:47:56.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029041a33f498dc66-006618aeec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.246 DEBUG:boto:b'dkyges-51obj31000falseobj32024-04-12T03:47:34.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:56.246 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:47:56.247 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:47:56.247 DEBUG:boto:Method: HEAD 2024-04-12T03:47:56.247 DEBUG:boto:Path: /dkyges-51/obj3 2024-04-12T03:47:56.247 DEBUG:boto:Data: 2024-04-12T03:47:56.247 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.247 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.247 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.247 DEBUG:boto:Params: {} 2024-04-12T03:47:56.247 DEBUG:boto:Token: None 2024-04-12T03:47:56.248 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj3 2024-04-12T03:47:56.248 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:keUNfxWr594E7ZYJqfwcGs5bMSo= 2024-04-12T03:47:56.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:keUNfxWr594E7ZYJqfwcGs5bMSo=', 'Content-Length': '0'} 2024-04-12T03:47:56.249 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef9ca2b16a8ba3b9-006618aeec-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.250 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:47:56.250 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:47:56.250 DEBUG:boto:Method: HEAD 2024-04-12T03:47:56.250 DEBUG:boto:Path: /dkyges-51/obj3 2024-04-12T03:47:56.250 DEBUG:boto:Data: 2024-04-12T03:47:56.250 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.250 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.250 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.251 DEBUG:boto:Params: {} 2024-04-12T03:47:56.251 DEBUG:boto:Token: None 2024-04-12T03:47:56.251 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj3 2024-04-12T03:47:56.251 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:keUNfxWr594E7ZYJqfwcGs5bMSo= 2024-04-12T03:47:56.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:keUNfxWr594E7ZYJqfwcGs5bMSo=', 'Content-Length': '0'} 2024-04-12T03:47:56.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062bfa44c1c7a0f80-006618aeec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:47:56.253 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:47:56.253 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:47:56.254 DEBUG:boto:Method: GET 2024-04-12T03:47:56.254 DEBUG:boto:Path: /dkyges-51/obj3 2024-04-12T03:47:56.254 DEBUG:boto:Data: 2024-04-12T03:47:56.254 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.254 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.254 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.254 DEBUG:boto:Params: {} 2024-04-12T03:47:56.254 DEBUG:boto:Token: None 2024-04-12T03:47:56.255 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj3 2024-04-12T03:47:56.255 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i/Xb4POAFk5JNplT3EKBmVCxNdc= 2024-04-12T03:47:56.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i/Xb4POAFk5JNplT3EKBmVCxNdc=', 'Content-Length': '0'} 2024-04-12T03:47:56.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa638c5fb19e2e11-006618aeec-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.257 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:47:56.257 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:47:56.257 DEBUG:boto:Method: GET 2024-04-12T03:47:56.257 DEBUG:boto:Path: /dkyges-51/obj3 2024-04-12T03:47:56.257 DEBUG:boto:Data: 2024-04-12T03:47:56.258 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.258 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.258 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.258 DEBUG:boto:Params: {} 2024-04-12T03:47:56.258 DEBUG:boto:Token: None 2024-04-12T03:47:56.258 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj3 2024-04-12T03:47:56.258 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i/Xb4POAFk5JNplT3EKBmVCxNdc= 2024-04-12T03:47:56.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i/Xb4POAFk5JNplT3EKBmVCxNdc=', 'Content-Length': '0'} 2024-04-12T03:47:56.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029b9f21936a37452-006618aeec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.261 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:56.261 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:56.261 DEBUG:boto:path=/dkyges-51/?prefix=obj3 2024-04-12T03:47:56.261 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj3 2024-04-12T03:47:56.261 DEBUG:boto:Method: GET 2024-04-12T03:47:56.261 DEBUG:boto:Path: /dkyges-51/?prefix=obj3 2024-04-12T03:47:56.261 DEBUG:boto:Data: 2024-04-12T03:47:56.261 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.261 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.262 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.262 DEBUG:boto:Params: {} 2024-04-12T03:47:56.262 DEBUG:boto:Token: None 2024-04-12T03:47:56.262 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/ 2024-04-12T03:47:56.262 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0= 2024-04-12T03:47:56.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0=', 'Content-Length': '0'} 2024-04-12T03:47:56.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f410a45510eea4c-006618aeec-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.265 DEBUG:boto:b'dkyges-51obj31000falseobj32024-04-12T03:47:34.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:56.265 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:56.265 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:56.265 DEBUG:boto:path=/dkyges-51/?prefix=obj3 2024-04-12T03:47:56.265 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj3 2024-04-12T03:47:56.266 DEBUG:boto:Method: GET 2024-04-12T03:47:56.266 DEBUG:boto:Path: /dkyges-51/?prefix=obj3 2024-04-12T03:47:56.266 DEBUG:boto:Data: 2024-04-12T03:47:56.266 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.266 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.266 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.266 DEBUG:boto:Params: {} 2024-04-12T03:47:56.266 DEBUG:boto:Token: None 2024-04-12T03:47:56.266 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/ 2024-04-12T03:47:56.267 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0= 2024-04-12T03:47:56.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0=', 'Content-Length': '0'} 2024-04-12T03:47:56.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003159b071a937cb39-006618aeec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.269 DEBUG:boto:b'dkyges-51obj31000falseobj32024-04-12T03:47:34.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:56.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:47:56.270 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:47:56.270 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:47:56.270 DEBUG:boto:path=/dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.270 DEBUG:boto:auth_path=/dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.270 DEBUG:boto:Method: GET 2024-04-12T03:47:56.271 DEBUG:boto:Path: /dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.271 DEBUG:boto:Data: 2024-04-12T03:47:56.271 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.271 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.271 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.271 DEBUG:boto:Params: {} 2024-04-12T03:47:56.271 DEBUG:boto:Token: None 2024-04-12T03:47:56.271 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.271 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dE97g6vaePw8FJfg30xctscjR0s= 2024-04-12T03:47:56.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dE97g6vaePw8FJfg30xctscjR0s=', 'Content-Length': '0'} 2024-04-12T03:47:56.273 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003039c5ce00ec4c4f-006618aeec-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.274 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:47:56.274 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:47:56.274 DEBUG:boto:path=/dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.274 DEBUG:boto:auth_path=/dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.274 DEBUG:boto:Method: GET 2024-04-12T03:47:56.275 DEBUG:boto:Path: /dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.275 DEBUG:boto:Data: 2024-04-12T03:47:56.275 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.275 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.275 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.275 DEBUG:boto:Params: {} 2024-04-12T03:47:56.275 DEBUG:boto:Token: None 2024-04-12T03:47:56.275 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.276 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dE97g6vaePw8FJfg30xctscjR0s= 2024-04-12T03:47:56.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dE97g6vaePw8FJfg30xctscjR0s=', 'Content-Length': '0'} 2024-04-12T03:47:56.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063619ba72cdf2174-006618aeec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.279 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:47:56.279 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:47:56.279 DEBUG:boto:path=/dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.279 DEBUG:boto:auth_path=/dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.279 DEBUG:boto:Method: HEAD 2024-04-12T03:47:56.279 DEBUG:boto:Path: /dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.279 DEBUG:boto:Data: 2024-04-12T03:47:56.279 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.279 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.280 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.280 DEBUG:boto:Params: {} 2024-04-12T03:47:56.280 DEBUG:boto:Token: None 2024-04-12T03:47:56.280 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.280 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sgP2OMBiRmbEhMD7meibL+bOLrQ= 2024-04-12T03:47:56.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sgP2OMBiRmbEhMD7meibL+bOLrQ=', 'Content-Length': '0'} 2024-04-12T03:47:56.282 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080960d66743d60eb-006618aeec-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.282 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:47:56.282 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:47:56.282 DEBUG:boto:path=/dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.282 DEBUG:boto:auth_path=/dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.283 DEBUG:boto:Method: HEAD 2024-04-12T03:47:56.283 DEBUG:boto:Path: /dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.283 DEBUG:boto:Data: 2024-04-12T03:47:56.283 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.283 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.283 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.283 DEBUG:boto:Params: {} 2024-04-12T03:47:56.283 DEBUG:boto:Token: None 2024-04-12T03:47:56.283 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.284 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sgP2OMBiRmbEhMD7meibL+bOLrQ= 2024-04-12T03:47:56.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sgP2OMBiRmbEhMD7meibL+bOLrQ=', 'Content-Length': '0'} 2024-04-12T03:47:56.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029153130b84cb6f4-006618aeec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:47:56.286 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:47:56.286 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:47:56.286 DEBUG:boto:path=/dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.286 DEBUG:boto:auth_path=/dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.286 DEBUG:boto:Method: GET 2024-04-12T03:47:56.286 DEBUG:boto:Path: /dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.286 DEBUG:boto:Data: 2024-04-12T03:47:56.286 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.287 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.287 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.287 DEBUG:boto:Params: {} 2024-04-12T03:47:56.287 DEBUG:boto:Token: None 2024-04-12T03:47:56.287 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.287 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dE97g6vaePw8FJfg30xctscjR0s= 2024-04-12T03:47:56.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dE97g6vaePw8FJfg30xctscjR0s=', 'Content-Length': '0'} 2024-04-12T03:47:56.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f477c10ff26ccb85-006618aeec-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.289 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:47:56.289 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:47:56.289 DEBUG:boto:path=/dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.289 DEBUG:boto:auth_path=/dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.290 DEBUG:boto:Method: GET 2024-04-12T03:47:56.290 DEBUG:boto:Path: /dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.290 DEBUG:boto:Data: 2024-04-12T03:47:56.290 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.290 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.290 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.290 DEBUG:boto:Params: {} 2024-04-12T03:47:56.290 DEBUG:boto:Token: None 2024-04-12T03:47:56.290 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj4?versionId=null 2024-04-12T03:47:56.291 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dE97g6vaePw8FJfg30xctscjR0s= 2024-04-12T03:47:56.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dE97g6vaePw8FJfg30xctscjR0s=', 'Content-Length': '0'} 2024-04-12T03:47:56.292 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb83e0b1d01d290c-006618aeec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.293 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:56.293 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:56.293 DEBUG:boto:path=/dkyges-51/?prefix=obj4 2024-04-12T03:47:56.293 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj4 2024-04-12T03:47:56.293 DEBUG:boto:Method: GET 2024-04-12T03:47:56.293 DEBUG:boto:Path: /dkyges-51/?prefix=obj4 2024-04-12T03:47:56.293 DEBUG:boto:Data: 2024-04-12T03:47:56.293 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.293 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.293 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.294 DEBUG:boto:Params: {} 2024-04-12T03:47:56.294 DEBUG:boto:Token: None 2024-04-12T03:47:56.294 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/ 2024-04-12T03:47:56.294 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0= 2024-04-12T03:47:56.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0=', 'Content-Length': '0'} 2024-04-12T03:47:56.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dce1f5d2fa1f7b0c-006618aeec-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.297 DEBUG:boto:b'dkyges-51obj41000falseobj42024-04-12T03:47:34.552Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:56.297 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:56.297 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:56.297 DEBUG:boto:path=/dkyges-51/?prefix=obj4 2024-04-12T03:47:56.297 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj4 2024-04-12T03:47:56.297 DEBUG:boto:Method: GET 2024-04-12T03:47:56.298 DEBUG:boto:Path: /dkyges-51/?prefix=obj4 2024-04-12T03:47:56.298 DEBUG:boto:Data: 2024-04-12T03:47:56.298 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.298 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.298 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.298 DEBUG:boto:Params: {} 2024-04-12T03:47:56.298 DEBUG:boto:Token: None 2024-04-12T03:47:56.298 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/ 2024-04-12T03:47:56.298 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0= 2024-04-12T03:47:56.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0=', 'Content-Length': '0'} 2024-04-12T03:47:56.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c79f91fcdbd7c5bc-006618aeec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.301 DEBUG:boto:b'dkyges-51obj41000falseobj42024-04-12T03:47:34.552Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:56.302 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:47:56.302 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:47:56.302 DEBUG:boto:Method: HEAD 2024-04-12T03:47:56.302 DEBUG:boto:Path: /dkyges-51/obj4 2024-04-12T03:47:56.302 DEBUG:boto:Data: 2024-04-12T03:47:56.302 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.302 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.302 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.303 DEBUG:boto:Params: {} 2024-04-12T03:47:56.303 DEBUG:boto:Token: None 2024-04-12T03:47:56.303 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj4 2024-04-12T03:47:56.303 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Vjp/qDX5A1dKnnaTr1AgqL5r9U4= 2024-04-12T03:47:56.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Vjp/qDX5A1dKnnaTr1AgqL5r9U4=', 'Content-Length': '0'} 2024-04-12T03:47:56.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000243f6ce86f111d2f-006618aeec-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.305 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:47:56.305 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:47:56.305 DEBUG:boto:Method: HEAD 2024-04-12T03:47:56.305 DEBUG:boto:Path: /dkyges-51/obj4 2024-04-12T03:47:56.306 DEBUG:boto:Data: 2024-04-12T03:47:56.306 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.306 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.306 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.306 DEBUG:boto:Params: {} 2024-04-12T03:47:56.306 DEBUG:boto:Token: None 2024-04-12T03:47:56.306 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj4 2024-04-12T03:47:56.306 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Vjp/qDX5A1dKnnaTr1AgqL5r9U4= 2024-04-12T03:47:56.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Vjp/qDX5A1dKnnaTr1AgqL5r9U4=', 'Content-Length': '0'} 2024-04-12T03:47:56.308 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c52af87ffb9b403-006618aeec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:47:56.309 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:47:56.309 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:47:56.309 DEBUG:boto:Method: GET 2024-04-12T03:47:56.309 DEBUG:boto:Path: /dkyges-51/obj4 2024-04-12T03:47:56.309 DEBUG:boto:Data: 2024-04-12T03:47:56.309 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.309 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.309 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.309 DEBUG:boto:Params: {} 2024-04-12T03:47:56.309 DEBUG:boto:Token: None 2024-04-12T03:47:56.309 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj4 2024-04-12T03:47:56.310 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OpYOztJpXmRxFmR+NxFBoF8JXSk= 2024-04-12T03:47:56.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OpYOztJpXmRxFmR+NxFBoF8JXSk=', 'Content-Length': '0'} 2024-04-12T03:47:56.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000049494b8fb0db97bf-006618aeec-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.312 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:47:56.312 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:47:56.312 DEBUG:boto:Method: GET 2024-04-12T03:47:56.312 DEBUG:boto:Path: /dkyges-51/obj4 2024-04-12T03:47:56.312 DEBUG:boto:Data: 2024-04-12T03:47:56.312 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.312 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.313 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.313 DEBUG:boto:Params: {} 2024-04-12T03:47:56.313 DEBUG:boto:Token: None 2024-04-12T03:47:56.313 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/obj4 2024-04-12T03:47:56.313 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OpYOztJpXmRxFmR+NxFBoF8JXSk= 2024-04-12T03:47:56.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OpYOztJpXmRxFmR+NxFBoF8JXSk=', 'Content-Length': '0'} 2024-04-12T03:47:56.315 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003589ba27f41301c6-006618aeec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.315 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:56.315 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:56.315 DEBUG:boto:path=/dkyges-51/?prefix=obj4 2024-04-12T03:47:56.315 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj4 2024-04-12T03:47:56.316 DEBUG:boto:Method: GET 2024-04-12T03:47:56.316 DEBUG:boto:Path: /dkyges-51/?prefix=obj4 2024-04-12T03:47:56.316 DEBUG:boto:Data: 2024-04-12T03:47:56.316 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.316 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:47:56.316 DEBUG:boto:Port: 8001 2024-04-12T03:47:56.316 DEBUG:boto:Params: {} 2024-04-12T03:47:56.316 DEBUG:boto:Token: None 2024-04-12T03:47:56.316 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/ 2024-04-12T03:47:56.316 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0= 2024-04-12T03:47:56.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0=', 'Content-Length': '0'} 2024-04-12T03:47:56.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008188a48c5db584e0-006618aeec-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.380 DEBUG:boto:b'dkyges-51obj41000falseobj42024-04-12T03:47:34.552Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:56.381 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:47:56.381 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:47:56.381 DEBUG:boto:path=/dkyges-51/?prefix=obj4 2024-04-12T03:47:56.381 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj4 2024-04-12T03:47:56.381 DEBUG:boto:Method: GET 2024-04-12T03:47:56.381 DEBUG:boto:Path: /dkyges-51/?prefix=obj4 2024-04-12T03:47:56.381 DEBUG:boto:Data: 2024-04-12T03:47:56.381 DEBUG:boto:Headers: {} 2024-04-12T03:47:56.381 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:47:56.381 DEBUG:boto:Port: 8000 2024-04-12T03:47:56.382 DEBUG:boto:Params: {} 2024-04-12T03:47:56.382 DEBUG:boto:Token: None 2024-04-12T03:47:56.382 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:47:56 GMT /dkyges-51/ 2024-04-12T03:47:56.382 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0= 2024-04-12T03:47:56.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:47:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ae+AiqpgEKNmv9kfeM00fwcZzb0=', 'Content-Length': '0'} 2024-04-12T03:47:56.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad704f729b214db0-006618aeec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:47:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:47:56.385 DEBUG:boto:b'dkyges-51obj41000falseobj42024-04-12T03:47:34.552Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:47:56.385 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-51 zones={a2, a1} 2024-04-12T03:47:56.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dkyges-50', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:47:56.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dkyges-50', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:47:56.386 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dkyges-50 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:47:56.446 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:56.446 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:56.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'dkyges-51', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:47:56.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'dkyges-51', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:47:56.529 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket dkyges-51 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:47:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:56.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:47:56.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:47:56.665 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:47:56.719 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:56.719 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:47:56.770 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:47:56.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:56.770 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:47:56.770 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:47:56.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:56.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:56.771 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:47:56.771 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:47:56.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:56.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:56.771 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:47:56.771 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:47:56.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:56.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:56.771 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893676.651969_450.1", 2024-04-12T03:47:56.771 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:47:56.651969Z" 2024-04-12T03:47:56.772 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:56.772 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:47:56.772 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893676.651969_450.1'} 2024-04-12T03:47:56.772 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:47:56.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:47:56.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:47:56.773 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:47:56.827 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:47:56.827 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:47:56.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:56.864+0000 7fe625cf5780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:47:56.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:47:56.865+0000 7fe625cf5780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:47:56.877 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:47:56.877 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:47:56.877 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:47:56.877 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:47:56.877 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:47:56.878 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:47:56.878 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:47:56.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:56.878 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:47:56.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:56.878 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:47:56.878 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:47:56.878 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:47:56.878 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:47:56.878 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:47:56.879 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:47:56.879 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:47:56.879 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:47:56.879 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:47:56.879 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:56.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:56.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:56.879 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:47:56.879 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:47:56.879 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:47:56.879 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:47:56.880 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:47:56.880 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:47:56.880 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:47:56.880 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:47:56.880 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:47:56.880 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:56.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:56.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:56.880 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:47:56.880 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:47:56.881 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:47:56.881 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:47:56.881 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:47:56.881 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:47:56.881 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:47:56.881 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:47:56.881 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:47:56.881 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:56.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:56.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:47:56.881 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:47:56.882 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:47:56.882 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:47:56.882 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893654.331665_443.1", 2024-04-12T03:47:56.882 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:47:56.882 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:47:56.882 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:47:56.882 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:47:34.331665Z", 2024-04-12T03:47:56.882 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:47:56.882 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:56.882 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:47:56.882 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:47:56.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:56.883 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:47:56.883 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:47:56.883 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:47:56.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:47:56.883 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:47:56Z" 2024-04-12T03:47:56.883 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:47:56.884 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893654.331665_443.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:47:34.331665Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:47:56Z" } 2024-04-12T03:47:56.884 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893654.331665_443.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:47:34.331665Z', 'realm_epoch': 2}}] 2024-04-12T03:47:56.884 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893676.651969_450.1'} 2024-04-12T03:47:56.884 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893654.331665_443.1'} 2024-04-12T03:47:56.885 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893676.651969_450.1 target=1_1712893654.331665_443.1 2024-04-12T03:48:01.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:48:01.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:48:01.887 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:48:01.947 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:48:01.947 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:48:01.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:01.986+0000 7ff543461780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:48:01.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:01.987+0000 7ff543461780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:48:02.000 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:48:02.000 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:48:02.000 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:48:02.000 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:48:02.000 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:48:02.000 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:48:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:48:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:48:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:48:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:48:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:48:02.001 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:48:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:48:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:48:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:48:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:02.002 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:48:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:48:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:48:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:48:02.003 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:48:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:48:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:48:02.004 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893654.331665_443.1", 2024-04-12T03:48:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:47:34.331665Z", 2024-04-12T03:48:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:48:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:48:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:48:02.005 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:48:02.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:02.006 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:48:01Z" 2024-04-12T03:48:02.006 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:48:02.007 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893654.331665_443.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:47:34.331665Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:48:01Z" } 2024-04-12T03:48:02.007 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893654.331665_443.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:47:34.331665Z', 'realm_epoch': 2}}] 2024-04-12T03:48:02.007 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893676.651969_450.1'} 2024-04-12T03:48:02.007 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893654.331665_443.1'} 2024-04-12T03:48:02.007 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893676.651969_450.1 target=1_1712893654.331665_443.1 2024-04-12T03:48:07.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:48:07.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:48:07.009 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:48:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:48:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:48:07.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:07.111+0000 7ffbddc35780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:48:07.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:07.112+0000 7ffbddc35780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:48:07.125 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:48:07.125 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:48:07.125 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:48:07.125 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:48:07.125 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:48:07.125 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:48:07.125 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:07.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:07.125 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:48:07.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:07.126 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:48:07.126 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:07.126 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:48:07.126 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:48:07.126 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:07.126 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:48:07.126 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:07.126 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:48:07.126 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:07.127 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:07.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:07.127 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:07.127 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:48:07.127 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:07.127 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:48:07.127 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:48:07.127 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:07.127 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:07.127 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:07.127 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:48:07.127 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:07.128 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:07.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:07.128 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:07.128 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:48:07.128 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:07.128 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:48:07.128 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:48:07.128 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:07.128 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:07.128 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:07.128 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:48:07.129 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:07.129 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:07.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:07.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:07.129 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:48:07.129 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:07.129 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:48:07.129 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893654.331665_443.1", 2024-04-12T03:48:07.129 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:07.129 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:07.129 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:07.130 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:47:34.331665Z", 2024-04-12T03:48:07.130 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:07.130 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:07.130 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:07.130 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:48:07.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:07.130 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:48:07.130 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:48:07.130 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:48:07.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:07.130 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:48:07Z" 2024-04-12T03:48:07.130 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:48:07.131 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893654.331665_443.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:47:34.331665Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:48:07Z" } 2024-04-12T03:48:07.132 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893654.331665_443.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:47:34.331665Z', 'realm_epoch': 2}}] 2024-04-12T03:48:07.132 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893676.651969_450.1'} 2024-04-12T03:48:07.132 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893654.331665_443.1'} 2024-04-12T03:48:07.132 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893676.651969_450.1 target=1_1712893654.331665_443.1 2024-04-12T03:48:12.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:48:12.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:48:12.133 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:48:12.193 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:48:12.194 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:48:12.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:12.235+0000 7f65d0eb2780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:48:12.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:12.236+0000 7f65d0eb2780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:48:12.249 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:48:12.249 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:48:12.249 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:48:12.249 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:48:12.249 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:48:12.249 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:48:12.250 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:12.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:12.250 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:48:12.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:12.250 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:48:12.250 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:12.250 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:48:12.250 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:48:12.251 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:12.251 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:48:12.251 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:12.251 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:48:12.251 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:12.251 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:12.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:12.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:12.251 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:48:12.251 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:12.251 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:48:12.252 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:48:12.252 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:12.252 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:12.252 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:12.252 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:48:12.252 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:12.252 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:12.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:12.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:12.252 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:48:12.252 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:12.253 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:48:12.253 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:48:12.253 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:12.253 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:12.253 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:12.253 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:48:12.253 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:12.253 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:12.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:12.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:12.253 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:48:12.253 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:12.254 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:48:12.254 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893654.331665_443.1", 2024-04-12T03:48:12.254 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:12.254 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:12.254 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:12.254 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:47:34.331665Z", 2024-04-12T03:48:12.254 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:12.254 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:12.254 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:12.254 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:48:12.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:12.254 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:48:12.255 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:48:12.255 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:48:12.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:12.255 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:48:12Z" 2024-04-12T03:48:12.255 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:48:12.256 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893654.331665_443.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:47:34.331665Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:48:12Z" } 2024-04-12T03:48:12.256 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893654.331665_443.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:47:34.331665Z', 'realm_epoch': 2}}] 2024-04-12T03:48:12.256 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893676.651969_450.1'} 2024-04-12T03:48:12.256 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893654.331665_443.1'} 2024-04-12T03:48:12.256 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893676.651969_450.1 target=1_1712893654.331665_443.1 2024-04-12T03:48:12.747 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T03:48:12.750+0000 7fd54c052700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T03:48:12.747 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T03:48:12.750+0000 7fd544042700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T03:48:12.748 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T03:48:12.750+0000 7fd545044700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T03:48:12.748 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T03:48:12.750+0000 7fd54a84f700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T03:48:17.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:48:17.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:48:17.258 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:48:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:48:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:48:17.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:17.361+0000 7fe6275c1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:48:17.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:17.362+0000 7fe6275c1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:48:17.373 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:48:17.374 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:48:17.374 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:48:17.374 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:48:17.374 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:48:17.374 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:48:17.374 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:17.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:17.374 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:48:17.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:17.375 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:48:17.375 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:17.375 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:48:17.375 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:48:17.375 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:17.375 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:48:17.375 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:17.375 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:48:17.375 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:17.375 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:17.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:17.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:17.376 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:48:17.376 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:17.376 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:48:17.376 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:48:17.376 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:17.376 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:17.376 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:17.376 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:48:17.376 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:17.376 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:17.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:17.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:17.377 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:48:17.377 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:17.377 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:48:17.377 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:48:17.377 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:17.377 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:17.377 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:17.377 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:48:17.377 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:17.377 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:17.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:17.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:17.378 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:48:17.378 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:17.378 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:48:17.378 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893676.651969_450.1", 2024-04-12T03:48:17.378 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:17.378 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:17.378 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:17.378 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:47:56.651969Z", 2024-04-12T03:48:17.378 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:48:17.378 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:17.379 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:17.379 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:48:17.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:17.379 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:48:17.379 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:48:17.379 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:48:17.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:17.379 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:48:17Z" 2024-04-12T03:48:17.379 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:48:17.381 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893676.651969_450.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:47:56.651969Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:48:17Z" } 2024-04-12T03:48:17.381 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893676.651969_450.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:47:56.651969Z', 'realm_epoch': 2}}] 2024-04-12T03:48:17.381 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893676.651969_450.1'} 2024-04-12T03:48:17.381 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893676.651969_450.1'} 2024-04-12T03:48:17.381 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:48:17.382 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:17.382 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:17.382 DEBUG:boto:Method: HEAD 2024-04-12T03:48:17.382 DEBUG:boto:Path: /dkyges-50/ 2024-04-12T03:48:17.382 DEBUG:boto:Data: 2024-04-12T03:48:17.382 DEBUG:boto:Headers: {} 2024-04-12T03:48:17.382 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:17.382 DEBUG:boto:Port: 8000 2024-04-12T03:48:17.382 DEBUG:boto:Params: {} 2024-04-12T03:48:17.382 DEBUG:boto:Token: None 2024-04-12T03:48:17.383 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:17 GMT /dkyges-50/ 2024-04-12T03:48:17.383 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NwKaQ1lj70sdiNPH+DpRoYUj2n8= 2024-04-12T03:48:17.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NwKaQ1lj70sdiNPH+DpRoYUj2n8=', 'Content-Length': '0'} 2024-04-12T03:48:17.385 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000537e11ce258eebc8-006618af01-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:17.386 DEBUG:boto:path=/dkyges-50/obj5 2024-04-12T03:48:17.386 DEBUG:boto:auth_path=/dkyges-50/obj5 2024-04-12T03:48:17.386 DEBUG:boto:Method: PUT 2024-04-12T03:48:17.386 DEBUG:boto:Path: /dkyges-50/obj5 2024-04-12T03:48:17.386 DEBUG:boto:Data: 2024-04-12T03:48:17.386 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:48:17.386 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:17.386 DEBUG:boto:Port: 8000 2024-04-12T03:48:17.386 DEBUG:boto:Params: {} 2024-04-12T03:48:17.386 DEBUG:boto:Token: None 2024-04-12T03:48:17.387 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:48:17 GMT /dkyges-50/obj5 2024-04-12T03:48:17.387 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gfgG7yS5qiCPjBLRMoA46//xBSI= 2024-04-12T03:48:17.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:48:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gfgG7yS5qiCPjBLRMoA46//xBSI='} 2024-04-12T03:48:17.393 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000949d895292d14d54-006618af01-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:17.393 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:17.393 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:17.393 DEBUG:boto:Method: HEAD 2024-04-12T03:48:17.393 DEBUG:boto:Path: /dkyges-50/ 2024-04-12T03:48:17.393 DEBUG:boto:Data: 2024-04-12T03:48:17.393 DEBUG:boto:Headers: {} 2024-04-12T03:48:17.394 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:17.394 DEBUG:boto:Port: 8000 2024-04-12T03:48:17.394 DEBUG:boto:Params: {} 2024-04-12T03:48:17.394 DEBUG:boto:Token: None 2024-04-12T03:48:17.394 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:17 GMT /dkyges-50/ 2024-04-12T03:48:17.394 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NwKaQ1lj70sdiNPH+DpRoYUj2n8= 2024-04-12T03:48:17.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NwKaQ1lj70sdiNPH+DpRoYUj2n8=', 'Content-Length': '0'} 2024-04-12T03:48:17.396 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e79b1af4d1a4a7e0-006618af01-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:17.396 DEBUG:boto:path=/dkyges-50/obj6 2024-04-12T03:48:17.396 DEBUG:boto:auth_path=/dkyges-50/obj6 2024-04-12T03:48:17.397 DEBUG:boto:Method: PUT 2024-04-12T03:48:17.397 DEBUG:boto:Path: /dkyges-50/obj6 2024-04-12T03:48:17.397 DEBUG:boto:Data: 2024-04-12T03:48:17.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:48:17.397 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:17.397 DEBUG:boto:Port: 8000 2024-04-12T03:48:17.397 DEBUG:boto:Params: {} 2024-04-12T03:48:17.397 DEBUG:boto:Token: None 2024-04-12T03:48:17.397 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:48:17 GMT /dkyges-50/obj6 2024-04-12T03:48:17.397 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W62eDVEXZrC54DRuLeD73veJqDA= 2024-04-12T03:48:17.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:48:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W62eDVEXZrC54DRuLeD73veJqDA='} 2024-04-12T03:48:17.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dec5080db816f30a-006618af01-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:17.443 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:17.443 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:17.443 DEBUG:boto:Method: HEAD 2024-04-12T03:48:17.443 DEBUG:boto:Path: /dkyges-50/ 2024-04-12T03:48:17.443 DEBUG:boto:Data: 2024-04-12T03:48:17.443 DEBUG:boto:Headers: {} 2024-04-12T03:48:17.443 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:17.443 DEBUG:boto:Port: 8000 2024-04-12T03:48:17.443 DEBUG:boto:Params: {} 2024-04-12T03:48:17.444 DEBUG:boto:Token: None 2024-04-12T03:48:17.444 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:17 GMT /dkyges-50/ 2024-04-12T03:48:17.444 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NwKaQ1lj70sdiNPH+DpRoYUj2n8= 2024-04-12T03:48:17.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NwKaQ1lj70sdiNPH+DpRoYUj2n8=', 'Content-Length': '0'} 2024-04-12T03:48:17.446 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b9b574ed894f764-006618af01-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:17.446 DEBUG:boto:path=/dkyges-50/obj7 2024-04-12T03:48:17.446 DEBUG:boto:auth_path=/dkyges-50/obj7 2024-04-12T03:48:17.447 DEBUG:boto:Method: PUT 2024-04-12T03:48:17.447 DEBUG:boto:Path: /dkyges-50/obj7 2024-04-12T03:48:17.447 DEBUG:boto:Data: 2024-04-12T03:48:17.447 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:48:17.447 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:17.447 DEBUG:boto:Port: 8000 2024-04-12T03:48:17.447 DEBUG:boto:Params: {} 2024-04-12T03:48:17.447 DEBUG:boto:Token: None 2024-04-12T03:48:17.447 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:48:17 GMT /dkyges-50/obj7 2024-04-12T03:48:17.447 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0uXzxyvq2aKdWmkvysdacsUo78Q= 2024-04-12T03:48:17.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:48:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0uXzxyvq2aKdWmkvysdacsUo78Q='} 2024-04-12T03:48:17.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084638d98363e0946-006618af01-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:17.454 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:17.454 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:17.454 DEBUG:boto:Method: HEAD 2024-04-12T03:48:17.454 DEBUG:boto:Path: /dkyges-50/ 2024-04-12T03:48:17.454 DEBUG:boto:Data: 2024-04-12T03:48:17.454 DEBUG:boto:Headers: {} 2024-04-12T03:48:17.454 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:17.455 DEBUG:boto:Port: 8000 2024-04-12T03:48:17.455 DEBUG:boto:Params: {} 2024-04-12T03:48:17.455 DEBUG:boto:Token: None 2024-04-12T03:48:17.455 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:17 GMT /dkyges-50/ 2024-04-12T03:48:17.455 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NwKaQ1lj70sdiNPH+DpRoYUj2n8= 2024-04-12T03:48:17.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NwKaQ1lj70sdiNPH+DpRoYUj2n8=', 'Content-Length': '0'} 2024-04-12T03:48:17.457 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027a66024994b2404-006618af01-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:17.458 DEBUG:boto:path=/dkyges-50/obj8 2024-04-12T03:48:17.458 DEBUG:boto:auth_path=/dkyges-50/obj8 2024-04-12T03:48:17.458 DEBUG:boto:Method: PUT 2024-04-12T03:48:17.458 DEBUG:boto:Path: /dkyges-50/obj8 2024-04-12T03:48:17.458 DEBUG:boto:Data: 2024-04-12T03:48:17.458 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:48:17.458 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:17.458 DEBUG:boto:Port: 8000 2024-04-12T03:48:17.458 DEBUG:boto:Params: {} 2024-04-12T03:48:17.459 DEBUG:boto:Token: None 2024-04-12T03:48:17.459 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:48:17 GMT /dkyges-50/obj8 2024-04-12T03:48:17.459 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4FUzhhmACeVFgz8AS0qJ4oXPf60= 2024-04-12T03:48:17.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:48:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4FUzhhmACeVFgz8AS0qJ4oXPf60='} 2024-04-12T03:48:17.465 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000931b7691139edb59-006618af01-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:17.465 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:17.465 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:17.465 DEBUG:boto:Method: HEAD 2024-04-12T03:48:17.465 DEBUG:boto:Path: /dkyges-51/ 2024-04-12T03:48:17.465 DEBUG:boto:Data: 2024-04-12T03:48:17.465 DEBUG:boto:Headers: {} 2024-04-12T03:48:17.465 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:17.465 DEBUG:boto:Port: 8001 2024-04-12T03:48:17.465 DEBUG:boto:Params: {} 2024-04-12T03:48:17.466 DEBUG:boto:Token: None 2024-04-12T03:48:17.466 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:17 GMT /dkyges-51/ 2024-04-12T03:48:17.466 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4so+XZAEMzXZRZ3JPfEbH950zMk= 2024-04-12T03:48:17.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4so+XZAEMzXZRZ3JPfEbH950zMk=', 'Content-Length': '0'} 2024-04-12T03:48:17.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c90ddd5c364b6f4f-006618af01-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:17.469 DEBUG:boto:path=/dkyges-51/obj5 2024-04-12T03:48:17.469 DEBUG:boto:auth_path=/dkyges-51/obj5 2024-04-12T03:48:17.469 DEBUG:boto:Method: PUT 2024-04-12T03:48:17.469 DEBUG:boto:Path: /dkyges-51/obj5 2024-04-12T03:48:17.469 DEBUG:boto:Data: 2024-04-12T03:48:17.469 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:48:17.469 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:17.469 DEBUG:boto:Port: 8001 2024-04-12T03:48:17.470 DEBUG:boto:Params: {} 2024-04-12T03:48:17.470 DEBUG:boto:Token: None 2024-04-12T03:48:17.470 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:48:17 GMT /dkyges-51/obj5 2024-04-12T03:48:17.470 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fEbYtIg5XJY3IoCaieKo5ErCNVI= 2024-04-12T03:48:17.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:48:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fEbYtIg5XJY3IoCaieKo5ErCNVI='} 2024-04-12T03:48:17.475 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af34f2223d3824bb-006618af01-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:17.475 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:17.475 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:17.475 DEBUG:boto:Method: HEAD 2024-04-12T03:48:17.475 DEBUG:boto:Path: /dkyges-51/ 2024-04-12T03:48:17.475 DEBUG:boto:Data: 2024-04-12T03:48:17.475 DEBUG:boto:Headers: {} 2024-04-12T03:48:17.475 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:17.475 DEBUG:boto:Port: 8001 2024-04-12T03:48:17.476 DEBUG:boto:Params: {} 2024-04-12T03:48:17.476 DEBUG:boto:Token: None 2024-04-12T03:48:17.476 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:17 GMT /dkyges-51/ 2024-04-12T03:48:17.476 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4so+XZAEMzXZRZ3JPfEbH950zMk= 2024-04-12T03:48:17.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4so+XZAEMzXZRZ3JPfEbH950zMk=', 'Content-Length': '0'} 2024-04-12T03:48:17.478 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d211ee0ee603e54d-006618af01-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:17.478 DEBUG:boto:path=/dkyges-51/obj6 2024-04-12T03:48:17.478 DEBUG:boto:auth_path=/dkyges-51/obj6 2024-04-12T03:48:17.478 DEBUG:boto:Method: PUT 2024-04-12T03:48:17.478 DEBUG:boto:Path: /dkyges-51/obj6 2024-04-12T03:48:17.479 DEBUG:boto:Data: 2024-04-12T03:48:17.479 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:48:17.479 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:17.479 DEBUG:boto:Port: 8001 2024-04-12T03:48:17.479 DEBUG:boto:Params: {} 2024-04-12T03:48:17.479 DEBUG:boto:Token: None 2024-04-12T03:48:17.479 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:48:17 GMT /dkyges-51/obj6 2024-04-12T03:48:17.479 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KvID//3QiziyADh+fR8LJ2zdEgI= 2024-04-12T03:48:17.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:48:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KvID//3QiziyADh+fR8LJ2zdEgI='} 2024-04-12T03:48:17.484 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073f8534398655b35-006618af01-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:17.484 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:17.484 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:17.484 DEBUG:boto:Method: HEAD 2024-04-12T03:48:17.484 DEBUG:boto:Path: /dkyges-51/ 2024-04-12T03:48:17.484 DEBUG:boto:Data: 2024-04-12T03:48:17.484 DEBUG:boto:Headers: {} 2024-04-12T03:48:17.485 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:17.485 DEBUG:boto:Port: 8001 2024-04-12T03:48:17.485 DEBUG:boto:Params: {} 2024-04-12T03:48:17.485 DEBUG:boto:Token: None 2024-04-12T03:48:17.485 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:17 GMT /dkyges-51/ 2024-04-12T03:48:17.485 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4so+XZAEMzXZRZ3JPfEbH950zMk= 2024-04-12T03:48:17.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4so+XZAEMzXZRZ3JPfEbH950zMk=', 'Content-Length': '0'} 2024-04-12T03:48:17.487 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000050873b8efa92bd88-006618af01-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:17.488 DEBUG:boto:path=/dkyges-51/obj7 2024-04-12T03:48:17.488 DEBUG:boto:auth_path=/dkyges-51/obj7 2024-04-12T03:48:17.488 DEBUG:boto:Method: PUT 2024-04-12T03:48:17.488 DEBUG:boto:Path: /dkyges-51/obj7 2024-04-12T03:48:17.488 DEBUG:boto:Data: 2024-04-12T03:48:17.488 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:48:17.488 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:17.488 DEBUG:boto:Port: 8001 2024-04-12T03:48:17.488 DEBUG:boto:Params: {} 2024-04-12T03:48:17.488 DEBUG:boto:Token: None 2024-04-12T03:48:17.489 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:48:17 GMT /dkyges-51/obj7 2024-04-12T03:48:17.489 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j3hb54CuSSvbIYHQP7SGuhfgMcg= 2024-04-12T03:48:17.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:48:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j3hb54CuSSvbIYHQP7SGuhfgMcg='} 2024-04-12T03:48:17.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c64a17eb3bab86cf-006618af01-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:17.535 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:17.535 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:17.535 DEBUG:boto:Method: HEAD 2024-04-12T03:48:17.535 DEBUG:boto:Path: /dkyges-51/ 2024-04-12T03:48:17.535 DEBUG:boto:Data: 2024-04-12T03:48:17.535 DEBUG:boto:Headers: {} 2024-04-12T03:48:17.535 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:17.535 DEBUG:boto:Port: 8001 2024-04-12T03:48:17.535 DEBUG:boto:Params: {} 2024-04-12T03:48:17.536 DEBUG:boto:Token: None 2024-04-12T03:48:17.536 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:17 GMT /dkyges-51/ 2024-04-12T03:48:17.536 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4so+XZAEMzXZRZ3JPfEbH950zMk= 2024-04-12T03:48:17.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4so+XZAEMzXZRZ3JPfEbH950zMk=', 'Content-Length': '0'} 2024-04-12T03:48:17.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080f5394d43374713-006618af01-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:17.538 DEBUG:boto:path=/dkyges-51/obj8 2024-04-12T03:48:17.538 DEBUG:boto:auth_path=/dkyges-51/obj8 2024-04-12T03:48:17.539 DEBUG:boto:Method: PUT 2024-04-12T03:48:17.539 DEBUG:boto:Path: /dkyges-51/obj8 2024-04-12T03:48:17.539 DEBUG:boto:Data: 2024-04-12T03:48:17.539 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T03:48:17.539 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:17.539 DEBUG:boto:Port: 8001 2024-04-12T03:48:17.539 DEBUG:boto:Params: {} 2024-04-12T03:48:17.539 DEBUG:boto:Token: None 2024-04-12T03:48:17.539 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 03:48:17 GMT /dkyges-51/obj8 2024-04-12T03:48:17.539 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tjzEGV9MfIirsZp3/j90MH4xsMc= 2024-04-12T03:48:17.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:48:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tjzEGV9MfIirsZp3/j90MH4xsMc='} 2024-04-12T03:48:17.544 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000008063bed82782508-006618af01-16de-a2'), ('Date', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:17.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'dkyges-50', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:48:17.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'dkyges-50', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:48:17.545 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket dkyges-50 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:48:17.605 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:48:17.605 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:48:17.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'dkyges-51', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:48:17.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'dkyges-51', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:48:17.686 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket dkyges-51 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:48:17.742 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:48:17.742 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:48:17.819 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-50 2024-04-12T03:48:17.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:48:17.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:48:17.820 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:48:17.882 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:48:17.882 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:48:17.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:17.925+0000 7f2639185780 1 waiting to reach incremental sync.. 2024-04-12T03:48:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:22.926+0000 7f2639185780 1 waiting to reach incremental sync.. 2024-04-12T03:48:27.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:27.927+0000 7f2639185780 1 waiting to reach incremental sync.. 2024-04-12T03:48:32.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:32.929+0000 7f2639185780 1 bucket sync caught up with source: 2024-04-12T03:48:32.926 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000004.32.1, 00000000002.37.1, 00000000004.27.1, 00000000002.17.1, 00000000004.54.1, 00000000002.13.1, 00000000002.59.1, 00000000004.60.1, 00000000002.33.1, 00000000002.27.1, 00000000002.18.1] 2024-04-12T03:48:32.926 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000004.32.1, 00000000002.37.1, 00000000004.27.1, 00000000002.17.1, 00000000004.54.1, 00000000002.13.1, 00000000002.59.1, 00000000004.60.1, 00000000002.33.1, 00000000002.27.1, 00000000002.18.1] 2024-04-12T03:48:32.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:32.929+0000 7f2639185780 0 bucket checkpoint complete 2024-04-12T03:48:32.944 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-50 2024-04-12T03:48:32.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:48:32.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:48:32.945 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:48:33.010 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:48:33.010 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:48:34.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:34.059+0000 7f7b5d831780 1 waiting to reach incremental sync.. 2024-04-12T03:48:39.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:39.060+0000 7f7b5d831780 1 waiting to reach incremental sync.. 2024-04-12T03:48:44.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:44.060+0000 7f7b5d831780 1 waiting to reach incremental sync.. 2024-04-12T03:48:49.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:49.062+0000 7f7b5d831780 1 bucket sync caught up with source: 2024-04-12T03:48:49.059 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000004.1895.1, 00000000002.2531.1, 00000000004.1396.1, 00000000002.1381.1, 00000000004.3606.1, 00000000002.1053.1, 00000000002.1585.1, 00000000004.1590.1, 00000000002.1896.1, 00000000002.2009.1, 00000000002.2512.1] 2024-04-12T03:48:49.059 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000002.1887.1, 00000000001.2530.1, 00000000002.1388.1, 00000000001.1380.1, 00000000002.3601.1, 00000000001.1052.1, 00000000001.1569.1, 00000000002.1572.1, 00000000001.1888.1, 00000000001.2008.1, 00000000001.2506.1] 2024-04-12T03:48:49.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:49.062+0000 7f7b5d831780 0 bucket checkpoint complete 2024-04-12T03:48:49.076 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-50 zones={a2, a1} 2024-04-12T03:48:49.077 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.077 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.077 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.077 DEBUG:boto:Path: /dkyges-50/ 2024-04-12T03:48:49.077 DEBUG:boto:Data: 2024-04-12T03:48:49.077 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.077 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.078 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.078 DEBUG:boto:Params: {} 2024-04-12T03:48:49.078 DEBUG:boto:Token: None 2024-04-12T03:48:49.078 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.078 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DfcgP6zqIHmAFiPIrjV5eMmdGb8= 2024-04-12T03:48:49.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DfcgP6zqIHmAFiPIrjV5eMmdGb8=', 'Content-Length': '0'} 2024-04-12T03:48:49.081 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2231f9c7758033e-006618af21-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.082 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.082 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.082 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.082 DEBUG:boto:Path: /dkyges-50/ 2024-04-12T03:48:49.082 DEBUG:boto:Data: 2024-04-12T03:48:49.082 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.082 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.082 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.082 DEBUG:boto:Params: {} 2024-04-12T03:48:49.083 DEBUG:boto:Token: None 2024-04-12T03:48:49.083 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.083 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DfcgP6zqIHmAFiPIrjV5eMmdGb8= 2024-04-12T03:48:49.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DfcgP6zqIHmAFiPIrjV5eMmdGb8=', 'Content-Length': '0'} 2024-04-12T03:48:49.086 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a58b307e140f356a-006618af21-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.086 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:48:49.087 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.087 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.087 DEBUG:boto:path=/dkyges-50/?versions&max-keys=999 2024-04-12T03:48:49.088 DEBUG:boto:auth_path=/dkyges-50/?versions&max-keys=999 2024-04-12T03:48:49.088 DEBUG:boto:Method: GET 2024-04-12T03:48:49.088 DEBUG:boto:Path: /dkyges-50/?versions&max-keys=999 2024-04-12T03:48:49.088 DEBUG:boto:Data: 2024-04-12T03:48:49.089 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.089 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.089 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.089 DEBUG:boto:Params: {} 2024-04-12T03:48:49.089 DEBUG:boto:Token: None 2024-04-12T03:48:49.089 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/?versions 2024-04-12T03:48:49.089 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FSdMmWUu97E1EhPyrqekSVWKJT8= 2024-04-12T03:48:49.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FSdMmWUu97E1EhPyrqekSVWKJT8=', 'Content-Length': '0'} 2024-04-12T03:48:49.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d925ba75935575b0-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.092 DEBUG:boto:b'dkyges-50999falseobj1nulltrue2024-04-12T03:47:34.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:47:34.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:47:34.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:47:34.461Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:48:17.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:48:17.403Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:48:17.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:48:17.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:48:49.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:48:49.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:48:49.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:48:49.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-12T03:48:49.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-12T03:48:49.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-12T03:48:49.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-12T03:48:49.094 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:48:49.094 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.094 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.094 DEBUG:boto:path=/dkyges-50/?versions&max-keys=999 2024-04-12T03:48:49.095 DEBUG:boto:auth_path=/dkyges-50/?versions&max-keys=999 2024-04-12T03:48:49.095 DEBUG:boto:Method: GET 2024-04-12T03:48:49.095 DEBUG:boto:Path: /dkyges-50/?versions&max-keys=999 2024-04-12T03:48:49.095 DEBUG:boto:Data: 2024-04-12T03:48:49.095 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.095 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.095 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.095 DEBUG:boto:Params: {} 2024-04-12T03:48:49.095 DEBUG:boto:Token: None 2024-04-12T03:48:49.095 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/?versions 2024-04-12T03:48:49.095 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FSdMmWUu97E1EhPyrqekSVWKJT8= 2024-04-12T03:48:49.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FSdMmWUu97E1EhPyrqekSVWKJT8=', 'Content-Length': '0'} 2024-04-12T03:48:49.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064d8832b71b6dcc2-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.099 DEBUG:boto:b'dkyges-50999falseobj1nulltrue2024-04-12T03:47:34.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:47:34.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:47:34.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:47:34.461Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:48:17.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:48:17.403Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:48:17.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:48:17.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:48:49.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:48:49.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:48:49.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:48:49.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-12T03:48:49.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-12T03:48:49.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-12T03:48:49.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-12T03:48:49.100 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.100 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.100 DEBUG:boto:path=/dkyges-50/?versions&max-keys=999 2024-04-12T03:48:49.100 DEBUG:boto:auth_path=/dkyges-50/?versions&max-keys=999 2024-04-12T03:48:49.100 DEBUG:boto:Method: GET 2024-04-12T03:48:49.100 DEBUG:boto:Path: /dkyges-50/?versions&max-keys=999 2024-04-12T03:48:49.100 DEBUG:boto:Data: 2024-04-12T03:48:49.101 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.101 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.101 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.101 DEBUG:boto:Params: {} 2024-04-12T03:48:49.101 DEBUG:boto:Token: None 2024-04-12T03:48:49.101 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/?versions 2024-04-12T03:48:49.101 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FSdMmWUu97E1EhPyrqekSVWKJT8= 2024-04-12T03:48:49.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FSdMmWUu97E1EhPyrqekSVWKJT8=', 'Content-Length': '0'} 2024-04-12T03:48:49.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dd984d4afdb2de7-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.104 DEBUG:boto:b'dkyges-50999falseobj1nulltrue2024-04-12T03:47:34.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:47:34.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:47:34.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:47:34.461Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:48:17.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:48:17.403Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:48:17.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:48:17.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.105 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.105 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.105 DEBUG:boto:path=/dkyges-50/?versions&max-keys=999 2024-04-12T03:48:49.105 DEBUG:boto:auth_path=/dkyges-50/?versions&max-keys=999 2024-04-12T03:48:49.105 DEBUG:boto:Method: GET 2024-04-12T03:48:49.105 DEBUG:boto:Path: /dkyges-50/?versions&max-keys=999 2024-04-12T03:48:49.105 DEBUG:boto:Data: 2024-04-12T03:48:49.105 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.105 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.106 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.106 DEBUG:boto:Params: {} 2024-04-12T03:48:49.106 DEBUG:boto:Token: None 2024-04-12T03:48:49.106 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/?versions 2024-04-12T03:48:49.106 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FSdMmWUu97E1EhPyrqekSVWKJT8= 2024-04-12T03:48:49.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FSdMmWUu97E1EhPyrqekSVWKJT8=', 'Content-Length': '0'} 2024-04-12T03:48:49.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006453f027f42992b7-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.110 DEBUG:boto:b'dkyges-50999falseobj1nulltrue2024-04-12T03:47:34.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:47:34.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:47:34.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:47:34.461Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:48:17.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:48:17.403Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:48:17.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:48:17.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:48:49.110 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:48:49.111 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:48:49.111 DEBUG:boto:path=/dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.111 DEBUG:boto:auth_path=/dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.111 DEBUG:boto:Method: GET 2024-04-12T03:48:49.111 DEBUG:boto:Path: /dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.111 DEBUG:boto:Data: 2024-04-12T03:48:49.111 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.111 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.111 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.111 DEBUG:boto:Params: {} 2024-04-12T03:48:49.111 DEBUG:boto:Token: None 2024-04-12T03:48:49.112 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.112 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BmHlbSh1t/aK8Y7f39Uhim1nlAE= 2024-04-12T03:48:49.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BmHlbSh1t/aK8Y7f39Uhim1nlAE=', 'Content-Length': '0'} 2024-04-12T03:48:49.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000078e1c7767a5aa71-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.154 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:48:49.155 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:48:49.155 DEBUG:boto:path=/dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.155 DEBUG:boto:auth_path=/dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.155 DEBUG:boto:Method: GET 2024-04-12T03:48:49.155 DEBUG:boto:Path: /dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.155 DEBUG:boto:Data: 2024-04-12T03:48:49.155 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.155 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.155 DEBUG:boto:Params: {} 2024-04-12T03:48:49.155 DEBUG:boto:Token: None 2024-04-12T03:48:49.156 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BmHlbSh1t/aK8Y7f39Uhim1nlAE= 2024-04-12T03:48:49.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BmHlbSh1t/aK8Y7f39Uhim1nlAE=', 'Content-Length': '0'} 2024-04-12T03:48:49.159 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c21ebdb1d7feede-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.159 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:48:49.159 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:48:49.159 DEBUG:boto:path=/dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.159 DEBUG:boto:auth_path=/dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.159 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.160 DEBUG:boto:Path: /dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.160 DEBUG:boto:Data: 2024-04-12T03:48:49.160 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.160 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.160 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.160 DEBUG:boto:Params: {} 2024-04-12T03:48:49.160 DEBUG:boto:Token: None 2024-04-12T03:48:49.160 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.160 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Iek1L3Z4Muim1jtHreJwgqm/ypU= 2024-04-12T03:48:49.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Iek1L3Z4Muim1jtHreJwgqm/ypU=', 'Content-Length': '0'} 2024-04-12T03:48:49.162 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e3a8bc092703763-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.162 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:48:49.163 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:48:49.163 DEBUG:boto:path=/dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.163 DEBUG:boto:auth_path=/dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.163 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.163 DEBUG:boto:Path: /dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.163 DEBUG:boto:Data: 2024-04-12T03:48:49.163 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.163 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.163 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.163 DEBUG:boto:Params: {} 2024-04-12T03:48:49.164 DEBUG:boto:Token: None 2024-04-12T03:48:49.164 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.164 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Iek1L3Z4Muim1jtHreJwgqm/ypU= 2024-04-12T03:48:49.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Iek1L3Z4Muim1jtHreJwgqm/ypU=', 'Content-Length': '0'} 2024-04-12T03:48:49.166 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014cc612218c8d0cf-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:48:49.166 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:48:49.166 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:48:49.166 DEBUG:boto:path=/dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.166 DEBUG:boto:auth_path=/dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.166 DEBUG:boto:Method: GET 2024-04-12T03:48:49.167 DEBUG:boto:Path: /dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.167 DEBUG:boto:Data: 2024-04-12T03:48:49.167 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.167 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.167 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.167 DEBUG:boto:Params: {} 2024-04-12T03:48:49.167 DEBUG:boto:Token: None 2024-04-12T03:48:49.167 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.167 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BmHlbSh1t/aK8Y7f39Uhim1nlAE= 2024-04-12T03:48:49.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BmHlbSh1t/aK8Y7f39Uhim1nlAE=', 'Content-Length': '0'} 2024-04-12T03:48:49.169 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee68c32b062b321c-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.169 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:48:49.169 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:48:49.170 DEBUG:boto:path=/dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.170 DEBUG:boto:auth_path=/dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.170 DEBUG:boto:Method: GET 2024-04-12T03:48:49.170 DEBUG:boto:Path: /dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.170 DEBUG:boto:Data: 2024-04-12T03:48:49.170 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.170 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.170 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.171 DEBUG:boto:Params: {} 2024-04-12T03:48:49.171 DEBUG:boto:Token: None 2024-04-12T03:48:49.171 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj1?versionId=null 2024-04-12T03:48:49.171 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BmHlbSh1t/aK8Y7f39Uhim1nlAE= 2024-04-12T03:48:49.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BmHlbSh1t/aK8Y7f39Uhim1nlAE=', 'Content-Length': '0'} 2024-04-12T03:48:49.173 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000018175aa2ea21f67a-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.173 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.173 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.173 DEBUG:boto:path=/dkyges-50/?prefix=obj1 2024-04-12T03:48:49.173 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj1 2024-04-12T03:48:49.174 DEBUG:boto:Method: GET 2024-04-12T03:48:49.174 DEBUG:boto:Path: /dkyges-50/?prefix=obj1 2024-04-12T03:48:49.174 DEBUG:boto:Data: 2024-04-12T03:48:49.174 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.174 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.174 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.174 DEBUG:boto:Params: {} 2024-04-12T03:48:49.174 DEBUG:boto:Token: None 2024-04-12T03:48:49.174 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.174 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2b86889a6cfec77-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.178 DEBUG:boto:b'dkyges-50obj11000falseobj12024-04-12T03:47:34.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.178 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.178 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.179 DEBUG:boto:path=/dkyges-50/?prefix=obj1 2024-04-12T03:48:49.179 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj1 2024-04-12T03:48:49.179 DEBUG:boto:Method: GET 2024-04-12T03:48:49.179 DEBUG:boto:Path: /dkyges-50/?prefix=obj1 2024-04-12T03:48:49.179 DEBUG:boto:Data: 2024-04-12T03:48:49.179 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.179 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.179 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.179 DEBUG:boto:Params: {} 2024-04-12T03:48:49.179 DEBUG:boto:Token: None 2024-04-12T03:48:49.179 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.180 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091df763b041abf39-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.182 DEBUG:boto:b'dkyges-50obj11000falseobj12024-04-12T03:47:34.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.183 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:48:49.183 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:48:49.183 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.183 DEBUG:boto:Path: /dkyges-50/obj1 2024-04-12T03:48:49.183 DEBUG:boto:Data: 2024-04-12T03:48:49.183 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.183 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.183 DEBUG:boto:Params: {} 2024-04-12T03:48:49.184 DEBUG:boto:Token: None 2024-04-12T03:48:49.184 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj1 2024-04-12T03:48:49.184 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Vbq6cayMV7WzbB2Ps5ecJZJl1I4= 2024-04-12T03:48:49.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Vbq6cayMV7WzbB2Ps5ecJZJl1I4=', 'Content-Length': '0'} 2024-04-12T03:48:49.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c8fd5c55bf83a89-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.186 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:48:49.186 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:48:49.186 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.186 DEBUG:boto:Path: /dkyges-50/obj1 2024-04-12T03:48:49.186 DEBUG:boto:Data: 2024-04-12T03:48:49.186 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.186 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.187 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.187 DEBUG:boto:Params: {} 2024-04-12T03:48:49.187 DEBUG:boto:Token: None 2024-04-12T03:48:49.187 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj1 2024-04-12T03:48:49.187 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Vbq6cayMV7WzbB2Ps5ecJZJl1I4= 2024-04-12T03:48:49.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Vbq6cayMV7WzbB2Ps5ecJZJl1I4=', 'Content-Length': '0'} 2024-04-12T03:48:49.189 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000699c9f612cd6699c-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:48:49.190 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:48:49.190 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:48:49.190 DEBUG:boto:Method: GET 2024-04-12T03:48:49.190 DEBUG:boto:Path: /dkyges-50/obj1 2024-04-12T03:48:49.190 DEBUG:boto:Data: 2024-04-12T03:48:49.190 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.190 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.190 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.190 DEBUG:boto:Params: {} 2024-04-12T03:48:49.191 DEBUG:boto:Token: None 2024-04-12T03:48:49.191 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj1 2024-04-12T03:48:49.191 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZtQU8y5i1I/XMStHnHRNTS5UGTs= 2024-04-12T03:48:49.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZtQU8y5i1I/XMStHnHRNTS5UGTs=', 'Content-Length': '0'} 2024-04-12T03:48:49.193 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035eb98d6a550ac1b-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.193 DEBUG:boto:path=/dkyges-50/obj1 2024-04-12T03:48:49.193 DEBUG:boto:auth_path=/dkyges-50/obj1 2024-04-12T03:48:49.193 DEBUG:boto:Method: GET 2024-04-12T03:48:49.193 DEBUG:boto:Path: /dkyges-50/obj1 2024-04-12T03:48:49.193 DEBUG:boto:Data: 2024-04-12T03:48:49.194 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.194 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.194 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.194 DEBUG:boto:Params: {} 2024-04-12T03:48:49.194 DEBUG:boto:Token: None 2024-04-12T03:48:49.194 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj1 2024-04-12T03:48:49.194 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZtQU8y5i1I/XMStHnHRNTS5UGTs= 2024-04-12T03:48:49.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZtQU8y5i1I/XMStHnHRNTS5UGTs=', 'Content-Length': '0'} 2024-04-12T03:48:49.196 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000038b39cb1cd1b1ee9-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.196 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.196 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.196 DEBUG:boto:path=/dkyges-50/?prefix=obj1 2024-04-12T03:48:49.196 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj1 2024-04-12T03:48:49.197 DEBUG:boto:Method: GET 2024-04-12T03:48:49.197 DEBUG:boto:Path: /dkyges-50/?prefix=obj1 2024-04-12T03:48:49.197 DEBUG:boto:Data: 2024-04-12T03:48:49.197 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.197 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.197 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.197 DEBUG:boto:Params: {} 2024-04-12T03:48:49.197 DEBUG:boto:Token: None 2024-04-12T03:48:49.198 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.198 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042312e6eccfb41ee-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.200 DEBUG:boto:b'dkyges-50obj11000falseobj12024-04-12T03:47:34.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.201 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.201 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.201 DEBUG:boto:path=/dkyges-50/?prefix=obj1 2024-04-12T03:48:49.201 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj1 2024-04-12T03:48:49.201 DEBUG:boto:Method: GET 2024-04-12T03:48:49.201 DEBUG:boto:Path: /dkyges-50/?prefix=obj1 2024-04-12T03:48:49.201 DEBUG:boto:Data: 2024-04-12T03:48:49.201 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.202 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.202 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.202 DEBUG:boto:Params: {} 2024-04-12T03:48:49.202 DEBUG:boto:Token: None 2024-04-12T03:48:49.202 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.202 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000724466780836c423-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.205 DEBUG:boto:b'dkyges-50obj11000falseobj12024-04-12T03:47:34.351Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:48:49.205 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:48:49.205 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:48:49.205 DEBUG:boto:path=/dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.205 DEBUG:boto:auth_path=/dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.205 DEBUG:boto:Method: GET 2024-04-12T03:48:49.205 DEBUG:boto:Path: /dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.206 DEBUG:boto:Data: 2024-04-12T03:48:49.206 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.206 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.206 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.206 DEBUG:boto:Params: {} 2024-04-12T03:48:49.206 DEBUG:boto:Token: None 2024-04-12T03:48:49.206 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.206 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8Wc2T7a3eNxYTKTHUZSLggKdp7k= 2024-04-12T03:48:49.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8Wc2T7a3eNxYTKTHUZSLggKdp7k=', 'Content-Length': '0'} 2024-04-12T03:48:49.208 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000654d8b8af6d42755-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.209 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:48:49.209 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:48:49.209 DEBUG:boto:path=/dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.209 DEBUG:boto:auth_path=/dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.209 DEBUG:boto:Method: GET 2024-04-12T03:48:49.209 DEBUG:boto:Path: /dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.209 DEBUG:boto:Data: 2024-04-12T03:48:49.209 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.209 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.209 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.209 DEBUG:boto:Params: {} 2024-04-12T03:48:49.209 DEBUG:boto:Token: None 2024-04-12T03:48:49.210 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.210 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8Wc2T7a3eNxYTKTHUZSLggKdp7k= 2024-04-12T03:48:49.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8Wc2T7a3eNxYTKTHUZSLggKdp7k=', 'Content-Length': '0'} 2024-04-12T03:48:49.211 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000172ceb3faf5e19ec-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.212 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:48:49.212 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:48:49.212 DEBUG:boto:path=/dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.212 DEBUG:boto:auth_path=/dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.212 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.212 DEBUG:boto:Path: /dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.212 DEBUG:boto:Data: 2024-04-12T03:48:49.212 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.212 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.212 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.213 DEBUG:boto:Params: {} 2024-04-12T03:48:49.213 DEBUG:boto:Token: None 2024-04-12T03:48:49.213 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.213 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wWj5ewuwQtw6fMMdB2IHmbrZBNQ= 2024-04-12T03:48:49.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wWj5ewuwQtw6fMMdB2IHmbrZBNQ=', 'Content-Length': '0'} 2024-04-12T03:48:49.215 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a9bb8c90093fab2-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.215 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:48:49.215 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:48:49.215 DEBUG:boto:path=/dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.215 DEBUG:boto:auth_path=/dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.216 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.216 DEBUG:boto:Path: /dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.216 DEBUG:boto:Data: 2024-04-12T03:48:49.216 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.216 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.216 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.216 DEBUG:boto:Params: {} 2024-04-12T03:48:49.216 DEBUG:boto:Token: None 2024-04-12T03:48:49.216 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.216 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wWj5ewuwQtw6fMMdB2IHmbrZBNQ= 2024-04-12T03:48:49.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wWj5ewuwQtw6fMMdB2IHmbrZBNQ=', 'Content-Length': '0'} 2024-04-12T03:48:49.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000d223b03501828a2-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:48:49.218 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:48:49.218 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:48:49.219 DEBUG:boto:path=/dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.219 DEBUG:boto:auth_path=/dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.219 DEBUG:boto:Method: GET 2024-04-12T03:48:49.219 DEBUG:boto:Path: /dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.219 DEBUG:boto:Data: 2024-04-12T03:48:49.219 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.219 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.219 DEBUG:boto:Params: {} 2024-04-12T03:48:49.219 DEBUG:boto:Token: None 2024-04-12T03:48:49.219 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.220 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8Wc2T7a3eNxYTKTHUZSLggKdp7k= 2024-04-12T03:48:49.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8Wc2T7a3eNxYTKTHUZSLggKdp7k=', 'Content-Length': '0'} 2024-04-12T03:48:49.221 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007909546fc2e97cc5-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.221 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:48:49.222 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:48:49.222 DEBUG:boto:path=/dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.222 DEBUG:boto:auth_path=/dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.222 DEBUG:boto:Method: GET 2024-04-12T03:48:49.222 DEBUG:boto:Path: /dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.222 DEBUG:boto:Data: 2024-04-12T03:48:49.222 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.222 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.222 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.222 DEBUG:boto:Params: {} 2024-04-12T03:48:49.222 DEBUG:boto:Token: None 2024-04-12T03:48:49.223 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj2?versionId=null 2024-04-12T03:48:49.223 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8Wc2T7a3eNxYTKTHUZSLggKdp7k= 2024-04-12T03:48:49.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8Wc2T7a3eNxYTKTHUZSLggKdp7k=', 'Content-Length': '0'} 2024-04-12T03:48:49.225 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db6c0838b572fd09-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.225 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.225 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.225 DEBUG:boto:path=/dkyges-50/?prefix=obj2 2024-04-12T03:48:49.225 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj2 2024-04-12T03:48:49.225 DEBUG:boto:Method: GET 2024-04-12T03:48:49.225 DEBUG:boto:Path: /dkyges-50/?prefix=obj2 2024-04-12T03:48:49.225 DEBUG:boto:Data: 2024-04-12T03:48:49.225 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.226 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.226 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.226 DEBUG:boto:Params: {} 2024-04-12T03:48:49.226 DEBUG:boto:Token: None 2024-04-12T03:48:49.226 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.226 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbc54283b8129a8b-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.229 DEBUG:boto:b'dkyges-50obj21000falseobj22024-04-12T03:47:34.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.229 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.229 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.229 DEBUG:boto:path=/dkyges-50/?prefix=obj2 2024-04-12T03:48:49.229 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj2 2024-04-12T03:48:49.229 DEBUG:boto:Method: GET 2024-04-12T03:48:49.229 DEBUG:boto:Path: /dkyges-50/?prefix=obj2 2024-04-12T03:48:49.230 DEBUG:boto:Data: 2024-04-12T03:48:49.230 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.230 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.230 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.230 DEBUG:boto:Params: {} 2024-04-12T03:48:49.231 DEBUG:boto:Token: None 2024-04-12T03:48:49.231 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.231 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011925519f5d93b30-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.235 DEBUG:boto:b'dkyges-50obj21000falseobj22024-04-12T03:47:34.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.235 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:48:49.235 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:48:49.236 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.236 DEBUG:boto:Path: /dkyges-50/obj2 2024-04-12T03:48:49.236 DEBUG:boto:Data: 2024-04-12T03:48:49.236 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.236 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.236 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.236 DEBUG:boto:Params: {} 2024-04-12T03:48:49.236 DEBUG:boto:Token: None 2024-04-12T03:48:49.236 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj2 2024-04-12T03:48:49.236 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DR/UDk0MiALWDg14Vemu9xYSa5k= 2024-04-12T03:48:49.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DR/UDk0MiALWDg14Vemu9xYSa5k=', 'Content-Length': '0'} 2024-04-12T03:48:49.238 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b7a2296e90164efd-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.238 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:48:49.238 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:48:49.239 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.239 DEBUG:boto:Path: /dkyges-50/obj2 2024-04-12T03:48:49.239 DEBUG:boto:Data: 2024-04-12T03:48:49.239 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.239 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.239 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.239 DEBUG:boto:Params: {} 2024-04-12T03:48:49.239 DEBUG:boto:Token: None 2024-04-12T03:48:49.239 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj2 2024-04-12T03:48:49.240 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DR/UDk0MiALWDg14Vemu9xYSa5k= 2024-04-12T03:48:49.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DR/UDk0MiALWDg14Vemu9xYSa5k=', 'Content-Length': '0'} 2024-04-12T03:48:49.241 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a01d85a8e3f08da1-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:48:49.242 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:48:49.242 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:48:49.242 DEBUG:boto:Method: GET 2024-04-12T03:48:49.242 DEBUG:boto:Path: /dkyges-50/obj2 2024-04-12T03:48:49.242 DEBUG:boto:Data: 2024-04-12T03:48:49.242 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.243 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.243 DEBUG:boto:Params: {} 2024-04-12T03:48:49.243 DEBUG:boto:Token: None 2024-04-12T03:48:49.243 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj2 2024-04-12T03:48:49.243 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z55kaGyPXQHhfPsFfQ2mjl8i03c= 2024-04-12T03:48:49.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z55kaGyPXQHhfPsFfQ2mjl8i03c=', 'Content-Length': '0'} 2024-04-12T03:48:49.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1745db6779a8f87-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.245 DEBUG:boto:path=/dkyges-50/obj2 2024-04-12T03:48:49.245 DEBUG:boto:auth_path=/dkyges-50/obj2 2024-04-12T03:48:49.245 DEBUG:boto:Method: GET 2024-04-12T03:48:49.246 DEBUG:boto:Path: /dkyges-50/obj2 2024-04-12T03:48:49.246 DEBUG:boto:Data: 2024-04-12T03:48:49.246 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.246 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.246 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.246 DEBUG:boto:Params: {} 2024-04-12T03:48:49.246 DEBUG:boto:Token: None 2024-04-12T03:48:49.246 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj2 2024-04-12T03:48:49.246 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z55kaGyPXQHhfPsFfQ2mjl8i03c= 2024-04-12T03:48:49.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z55kaGyPXQHhfPsFfQ2mjl8i03c=', 'Content-Length': '0'} 2024-04-12T03:48:49.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d21d1a8ba10276d5-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.294 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.295 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.295 DEBUG:boto:path=/dkyges-50/?prefix=obj2 2024-04-12T03:48:49.295 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj2 2024-04-12T03:48:49.295 DEBUG:boto:Method: GET 2024-04-12T03:48:49.295 DEBUG:boto:Path: /dkyges-50/?prefix=obj2 2024-04-12T03:48:49.295 DEBUG:boto:Data: 2024-04-12T03:48:49.295 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.295 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.295 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.295 DEBUG:boto:Params: {} 2024-04-12T03:48:49.296 DEBUG:boto:Token: None 2024-04-12T03:48:49.296 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.296 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d876dddde5e82ab2-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.299 DEBUG:boto:b'dkyges-50obj21000falseobj22024-04-12T03:47:34.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.299 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.299 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.300 DEBUG:boto:path=/dkyges-50/?prefix=obj2 2024-04-12T03:48:49.300 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj2 2024-04-12T03:48:49.300 DEBUG:boto:Method: GET 2024-04-12T03:48:49.300 DEBUG:boto:Path: /dkyges-50/?prefix=obj2 2024-04-12T03:48:49.300 DEBUG:boto:Data: 2024-04-12T03:48:49.300 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.300 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.300 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.300 DEBUG:boto:Params: {} 2024-04-12T03:48:49.300 DEBUG:boto:Token: None 2024-04-12T03:48:49.300 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.301 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083976e6607466e01-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.303 DEBUG:boto:b'dkyges-50obj21000falseobj22024-04-12T03:47:34.401Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:48:49.304 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:48:49.304 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:48:49.304 DEBUG:boto:path=/dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.304 DEBUG:boto:auth_path=/dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.304 DEBUG:boto:Method: GET 2024-04-12T03:48:49.304 DEBUG:boto:Path: /dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.304 DEBUG:boto:Data: 2024-04-12T03:48:49.305 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.305 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.305 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.305 DEBUG:boto:Params: {} 2024-04-12T03:48:49.305 DEBUG:boto:Token: None 2024-04-12T03:48:49.305 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.305 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yWs+R7YBrUfSZEGD6if3I6P+VmA= 2024-04-12T03:48:49.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yWs+R7YBrUfSZEGD6if3I6P+VmA=', 'Content-Length': '0'} 2024-04-12T03:48:49.307 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009df1eed5f528f969-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.350 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:48:49.351 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:48:49.351 DEBUG:boto:path=/dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.351 DEBUG:boto:auth_path=/dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.351 DEBUG:boto:Method: GET 2024-04-12T03:48:49.351 DEBUG:boto:Path: /dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.351 DEBUG:boto:Data: 2024-04-12T03:48:49.351 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.351 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.351 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.351 DEBUG:boto:Params: {} 2024-04-12T03:48:49.352 DEBUG:boto:Token: None 2024-04-12T03:48:49.352 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.352 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yWs+R7YBrUfSZEGD6if3I6P+VmA= 2024-04-12T03:48:49.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yWs+R7YBrUfSZEGD6if3I6P+VmA=', 'Content-Length': '0'} 2024-04-12T03:48:49.354 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a957cd48e1706cfe-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.354 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:48:49.354 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:48:49.354 DEBUG:boto:path=/dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.354 DEBUG:boto:auth_path=/dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.355 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.355 DEBUG:boto:Path: /dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.355 DEBUG:boto:Data: 2024-04-12T03:48:49.355 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.355 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.355 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.355 DEBUG:boto:Params: {} 2024-04-12T03:48:49.355 DEBUG:boto:Token: None 2024-04-12T03:48:49.355 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.355 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C2IkByt0h1LhbVJOlC7+gokr1RY= 2024-04-12T03:48:49.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C2IkByt0h1LhbVJOlC7+gokr1RY=', 'Content-Length': '0'} 2024-04-12T03:48:49.357 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000051bb42a0f33afff8-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.358 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:48:49.358 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:48:49.358 DEBUG:boto:path=/dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.358 DEBUG:boto:auth_path=/dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.358 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.358 DEBUG:boto:Path: /dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.358 DEBUG:boto:Data: 2024-04-12T03:48:49.358 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.358 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.358 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.359 DEBUG:boto:Params: {} 2024-04-12T03:48:49.359 DEBUG:boto:Token: None 2024-04-12T03:48:49.359 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.359 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C2IkByt0h1LhbVJOlC7+gokr1RY= 2024-04-12T03:48:49.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C2IkByt0h1LhbVJOlC7+gokr1RY=', 'Content-Length': '0'} 2024-04-12T03:48:49.361 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6ac4561b550a104-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:48:49.361 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:48:49.361 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:48:49.361 DEBUG:boto:path=/dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.361 DEBUG:boto:auth_path=/dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.361 DEBUG:boto:Method: GET 2024-04-12T03:48:49.361 DEBUG:boto:Path: /dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.362 DEBUG:boto:Data: 2024-04-12T03:48:49.362 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.362 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.362 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.362 DEBUG:boto:Params: {} 2024-04-12T03:48:49.362 DEBUG:boto:Token: None 2024-04-12T03:48:49.362 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.362 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yWs+R7YBrUfSZEGD6if3I6P+VmA= 2024-04-12T03:48:49.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yWs+R7YBrUfSZEGD6if3I6P+VmA=', 'Content-Length': '0'} 2024-04-12T03:48:49.364 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb0c5ab8c4026610-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.365 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:48:49.365 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:48:49.365 DEBUG:boto:path=/dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.365 DEBUG:boto:auth_path=/dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.365 DEBUG:boto:Method: GET 2024-04-12T03:48:49.365 DEBUG:boto:Path: /dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.365 DEBUG:boto:Data: 2024-04-12T03:48:49.365 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.365 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.365 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.366 DEBUG:boto:Params: {} 2024-04-12T03:48:49.366 DEBUG:boto:Token: None 2024-04-12T03:48:49.366 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj3?versionId=null 2024-04-12T03:48:49.366 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yWs+R7YBrUfSZEGD6if3I6P+VmA= 2024-04-12T03:48:49.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yWs+R7YBrUfSZEGD6if3I6P+VmA=', 'Content-Length': '0'} 2024-04-12T03:48:49.368 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095fa2c10dedfd54b-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.368 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.368 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.368 DEBUG:boto:path=/dkyges-50/?prefix=obj3 2024-04-12T03:48:49.368 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj3 2024-04-12T03:48:49.368 DEBUG:boto:Method: GET 2024-04-12T03:48:49.368 DEBUG:boto:Path: /dkyges-50/?prefix=obj3 2024-04-12T03:48:49.369 DEBUG:boto:Data: 2024-04-12T03:48:49.369 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.369 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.369 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.369 DEBUG:boto:Params: {} 2024-04-12T03:48:49.369 DEBUG:boto:Token: None 2024-04-12T03:48:49.369 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.369 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fedfd3c9b6176df6-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.372 DEBUG:boto:b'dkyges-50obj31000falseobj32024-04-12T03:47:34.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.372 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.372 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.373 DEBUG:boto:path=/dkyges-50/?prefix=obj3 2024-04-12T03:48:49.373 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj3 2024-04-12T03:48:49.373 DEBUG:boto:Method: GET 2024-04-12T03:48:49.373 DEBUG:boto:Path: /dkyges-50/?prefix=obj3 2024-04-12T03:48:49.373 DEBUG:boto:Data: 2024-04-12T03:48:49.373 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.373 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.373 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.373 DEBUG:boto:Params: {} 2024-04-12T03:48:49.373 DEBUG:boto:Token: None 2024-04-12T03:48:49.374 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.374 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb9dbb871291ab5e-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.377 DEBUG:boto:b'dkyges-50obj31000falseobj32024-04-12T03:47:34.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.377 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:48:49.377 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:48:49.377 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.377 DEBUG:boto:Path: /dkyges-50/obj3 2024-04-12T03:48:49.377 DEBUG:boto:Data: 2024-04-12T03:48:49.377 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.377 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.378 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.378 DEBUG:boto:Params: {} 2024-04-12T03:48:49.378 DEBUG:boto:Token: None 2024-04-12T03:48:49.378 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj3 2024-04-12T03:48:49.378 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u8RWzXO4pPjywlowtc6oDSgKY3Y= 2024-04-12T03:48:49.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u8RWzXO4pPjywlowtc6oDSgKY3Y=', 'Content-Length': '0'} 2024-04-12T03:48:49.381 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff5919fe6812971f-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.381 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:48:49.381 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:48:49.382 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.382 DEBUG:boto:Path: /dkyges-50/obj3 2024-04-12T03:48:49.382 DEBUG:boto:Data: 2024-04-12T03:48:49.382 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.382 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.382 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.382 DEBUG:boto:Params: {} 2024-04-12T03:48:49.382 DEBUG:boto:Token: None 2024-04-12T03:48:49.382 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj3 2024-04-12T03:48:49.383 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u8RWzXO4pPjywlowtc6oDSgKY3Y= 2024-04-12T03:48:49.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u8RWzXO4pPjywlowtc6oDSgKY3Y=', 'Content-Length': '0'} 2024-04-12T03:48:49.385 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be62822af11871e7-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:48:49.385 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:48:49.385 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:48:49.386 DEBUG:boto:Method: GET 2024-04-12T03:48:49.386 DEBUG:boto:Path: /dkyges-50/obj3 2024-04-12T03:48:49.386 DEBUG:boto:Data: 2024-04-12T03:48:49.386 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.386 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.386 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.386 DEBUG:boto:Params: {} 2024-04-12T03:48:49.386 DEBUG:boto:Token: None 2024-04-12T03:48:49.386 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj3 2024-04-12T03:48:49.386 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ei0PnacGMFmAXNvWe072DWgvWEk= 2024-04-12T03:48:49.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ei0PnacGMFmAXNvWe072DWgvWEk=', 'Content-Length': '0'} 2024-04-12T03:48:49.388 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1e8abaa5d1c5067-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.388 DEBUG:boto:path=/dkyges-50/obj3 2024-04-12T03:48:49.389 DEBUG:boto:auth_path=/dkyges-50/obj3 2024-04-12T03:48:49.389 DEBUG:boto:Method: GET 2024-04-12T03:48:49.389 DEBUG:boto:Path: /dkyges-50/obj3 2024-04-12T03:48:49.389 DEBUG:boto:Data: 2024-04-12T03:48:49.389 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.389 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.389 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.389 DEBUG:boto:Params: {} 2024-04-12T03:48:49.389 DEBUG:boto:Token: None 2024-04-12T03:48:49.389 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj3 2024-04-12T03:48:49.390 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ei0PnacGMFmAXNvWe072DWgvWEk= 2024-04-12T03:48:49.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ei0PnacGMFmAXNvWe072DWgvWEk=', 'Content-Length': '0'} 2024-04-12T03:48:49.391 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be59e01383a79093-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.392 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.392 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.392 DEBUG:boto:path=/dkyges-50/?prefix=obj3 2024-04-12T03:48:49.392 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj3 2024-04-12T03:48:49.392 DEBUG:boto:Method: GET 2024-04-12T03:48:49.392 DEBUG:boto:Path: /dkyges-50/?prefix=obj3 2024-04-12T03:48:49.392 DEBUG:boto:Data: 2024-04-12T03:48:49.392 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.392 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.392 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.392 DEBUG:boto:Params: {} 2024-04-12T03:48:49.393 DEBUG:boto:Token: None 2024-04-12T03:48:49.393 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.393 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000153bdc0044e23113-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.396 DEBUG:boto:b'dkyges-50obj31000falseobj32024-04-12T03:47:34.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.396 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.396 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.396 DEBUG:boto:path=/dkyges-50/?prefix=obj3 2024-04-12T03:48:49.397 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj3 2024-04-12T03:48:49.397 DEBUG:boto:Method: GET 2024-04-12T03:48:49.397 DEBUG:boto:Path: /dkyges-50/?prefix=obj3 2024-04-12T03:48:49.397 DEBUG:boto:Data: 2024-04-12T03:48:49.397 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.397 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.397 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.397 DEBUG:boto:Params: {} 2024-04-12T03:48:49.397 DEBUG:boto:Token: None 2024-04-12T03:48:49.397 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.397 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000760564565fe905ce-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.400 DEBUG:boto:b'dkyges-50obj31000falseobj32024-04-12T03:47:34.413Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:48:49.401 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:48:49.401 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:48:49.401 DEBUG:boto:path=/dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.401 DEBUG:boto:auth_path=/dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.401 DEBUG:boto:Method: GET 2024-04-12T03:48:49.401 DEBUG:boto:Path: /dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.401 DEBUG:boto:Data: 2024-04-12T03:48:49.401 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.401 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.402 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.402 DEBUG:boto:Params: {} 2024-04-12T03:48:49.402 DEBUG:boto:Token: None 2024-04-12T03:48:49.402 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.402 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:063rXlo4hVkdM7J8WeV920G5a8o= 2024-04-12T03:48:49.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:063rXlo4hVkdM7J8WeV920G5a8o=', 'Content-Length': '0'} 2024-04-12T03:48:49.404 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db2a13346200dbf8-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.446 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:48:49.447 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:48:49.447 DEBUG:boto:path=/dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.447 DEBUG:boto:auth_path=/dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.447 DEBUG:boto:Method: GET 2024-04-12T03:48:49.447 DEBUG:boto:Path: /dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.447 DEBUG:boto:Data: 2024-04-12T03:48:49.447 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.447 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.447 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.447 DEBUG:boto:Params: {} 2024-04-12T03:48:49.448 DEBUG:boto:Token: None 2024-04-12T03:48:49.448 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.448 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:063rXlo4hVkdM7J8WeV920G5a8o= 2024-04-12T03:48:49.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:063rXlo4hVkdM7J8WeV920G5a8o=', 'Content-Length': '0'} 2024-04-12T03:48:49.450 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000066ddee504d1b84e1-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.450 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:48:49.450 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:48:49.451 DEBUG:boto:path=/dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.451 DEBUG:boto:auth_path=/dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.451 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.451 DEBUG:boto:Path: /dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.451 DEBUG:boto:Data: 2024-04-12T03:48:49.451 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.451 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.451 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.451 DEBUG:boto:Params: {} 2024-04-12T03:48:49.451 DEBUG:boto:Token: None 2024-04-12T03:48:49.452 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.452 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NSXaFzjMx2VTbmGCSFI20w3m4rM= 2024-04-12T03:48:49.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NSXaFzjMx2VTbmGCSFI20w3m4rM=', 'Content-Length': '0'} 2024-04-12T03:48:49.454 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023eb191d2fcfc185-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.454 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:48:49.454 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:48:49.454 DEBUG:boto:path=/dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.454 DEBUG:boto:auth_path=/dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.454 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.454 DEBUG:boto:Path: /dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.455 DEBUG:boto:Data: 2024-04-12T03:48:49.455 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.455 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.455 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.455 DEBUG:boto:Params: {} 2024-04-12T03:48:49.455 DEBUG:boto:Token: None 2024-04-12T03:48:49.455 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.455 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NSXaFzjMx2VTbmGCSFI20w3m4rM= 2024-04-12T03:48:49.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NSXaFzjMx2VTbmGCSFI20w3m4rM=', 'Content-Length': '0'} 2024-04-12T03:48:49.457 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000440431d9ceba01bb-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:48:49.458 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:48:49.458 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:48:49.458 DEBUG:boto:path=/dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.458 DEBUG:boto:auth_path=/dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.458 DEBUG:boto:Method: GET 2024-04-12T03:48:49.458 DEBUG:boto:Path: /dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.458 DEBUG:boto:Data: 2024-04-12T03:48:49.458 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.459 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.459 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.459 DEBUG:boto:Params: {} 2024-04-12T03:48:49.459 DEBUG:boto:Token: None 2024-04-12T03:48:49.459 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.459 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:063rXlo4hVkdM7J8WeV920G5a8o= 2024-04-12T03:48:49.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:063rXlo4hVkdM7J8WeV920G5a8o=', 'Content-Length': '0'} 2024-04-12T03:48:49.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b784a0d36aa58111-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.462 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:48:49.462 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:48:49.462 DEBUG:boto:path=/dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.462 DEBUG:boto:auth_path=/dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.462 DEBUG:boto:Method: GET 2024-04-12T03:48:49.462 DEBUG:boto:Path: /dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.462 DEBUG:boto:Data: 2024-04-12T03:48:49.462 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.463 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.463 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.463 DEBUG:boto:Params: {} 2024-04-12T03:48:49.463 DEBUG:boto:Token: None 2024-04-12T03:48:49.463 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj4?versionId=null 2024-04-12T03:48:49.463 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:063rXlo4hVkdM7J8WeV920G5a8o= 2024-04-12T03:48:49.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:063rXlo4hVkdM7J8WeV920G5a8o=', 'Content-Length': '0'} 2024-04-12T03:48:49.466 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a0cb7aba9aa8880-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.466 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.467 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.467 DEBUG:boto:path=/dkyges-50/?prefix=obj4 2024-04-12T03:48:49.467 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj4 2024-04-12T03:48:49.467 DEBUG:boto:Method: GET 2024-04-12T03:48:49.467 DEBUG:boto:Path: /dkyges-50/?prefix=obj4 2024-04-12T03:48:49.467 DEBUG:boto:Data: 2024-04-12T03:48:49.467 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.467 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.467 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.467 DEBUG:boto:Params: {} 2024-04-12T03:48:49.467 DEBUG:boto:Token: None 2024-04-12T03:48:49.468 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.468 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000380d791c98df5487-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.470 DEBUG:boto:b'dkyges-50obj41000falseobj42024-04-12T03:47:34.461Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.471 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.471 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.471 DEBUG:boto:path=/dkyges-50/?prefix=obj4 2024-04-12T03:48:49.471 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj4 2024-04-12T03:48:49.471 DEBUG:boto:Method: GET 2024-04-12T03:48:49.471 DEBUG:boto:Path: /dkyges-50/?prefix=obj4 2024-04-12T03:48:49.471 DEBUG:boto:Data: 2024-04-12T03:48:49.471 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.472 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.472 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.472 DEBUG:boto:Params: {} 2024-04-12T03:48:49.472 DEBUG:boto:Token: None 2024-04-12T03:48:49.472 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.472 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7a3aa78158e1c5d-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.475 DEBUG:boto:b'dkyges-50obj41000falseobj42024-04-12T03:47:34.461Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.476 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:48:49.476 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:48:49.476 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.476 DEBUG:boto:Path: /dkyges-50/obj4 2024-04-12T03:48:49.476 DEBUG:boto:Data: 2024-04-12T03:48:49.476 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.476 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.476 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.476 DEBUG:boto:Params: {} 2024-04-12T03:48:49.477 DEBUG:boto:Token: None 2024-04-12T03:48:49.477 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj4 2024-04-12T03:48:49.477 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:M4lxZzBzP26KCBo8Iesm7HNbH4Q= 2024-04-12T03:48:49.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:M4lxZzBzP26KCBo8Iesm7HNbH4Q=', 'Content-Length': '0'} 2024-04-12T03:48:49.478 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009832dc2e0564ec6-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.479 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:48:49.479 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:48:49.479 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.479 DEBUG:boto:Path: /dkyges-50/obj4 2024-04-12T03:48:49.479 DEBUG:boto:Data: 2024-04-12T03:48:49.479 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.479 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.479 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.479 DEBUG:boto:Params: {} 2024-04-12T03:48:49.480 DEBUG:boto:Token: None 2024-04-12T03:48:49.480 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj4 2024-04-12T03:48:49.480 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:M4lxZzBzP26KCBo8Iesm7HNbH4Q= 2024-04-12T03:48:49.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:M4lxZzBzP26KCBo8Iesm7HNbH4Q=', 'Content-Length': '0'} 2024-04-12T03:48:49.481 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bc9e4f218860cd1-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:48:49.482 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:48:49.482 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:48:49.482 DEBUG:boto:Method: GET 2024-04-12T03:48:49.482 DEBUG:boto:Path: /dkyges-50/obj4 2024-04-12T03:48:49.482 DEBUG:boto:Data: 2024-04-12T03:48:49.482 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.483 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.483 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.483 DEBUG:boto:Params: {} 2024-04-12T03:48:49.483 DEBUG:boto:Token: None 2024-04-12T03:48:49.483 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj4 2024-04-12T03:48:49.483 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+hPYzAPzelQvouyOpcs/jPOxxD0= 2024-04-12T03:48:49.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+hPYzAPzelQvouyOpcs/jPOxxD0=', 'Content-Length': '0'} 2024-04-12T03:48:49.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009384ef55aae85212-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.526 DEBUG:boto:path=/dkyges-50/obj4 2024-04-12T03:48:49.527 DEBUG:boto:auth_path=/dkyges-50/obj4 2024-04-12T03:48:49.527 DEBUG:boto:Method: GET 2024-04-12T03:48:49.527 DEBUG:boto:Path: /dkyges-50/obj4 2024-04-12T03:48:49.527 DEBUG:boto:Data: 2024-04-12T03:48:49.527 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.527 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.527 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.527 DEBUG:boto:Params: {} 2024-04-12T03:48:49.527 DEBUG:boto:Token: None 2024-04-12T03:48:49.528 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj4 2024-04-12T03:48:49.528 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+hPYzAPzelQvouyOpcs/jPOxxD0= 2024-04-12T03:48:49.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+hPYzAPzelQvouyOpcs/jPOxxD0=', 'Content-Length': '0'} 2024-04-12T03:48:49.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1fec940f4a5859e-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.530 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.530 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.530 DEBUG:boto:path=/dkyges-50/?prefix=obj4 2024-04-12T03:48:49.530 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj4 2024-04-12T03:48:49.530 DEBUG:boto:Method: GET 2024-04-12T03:48:49.530 DEBUG:boto:Path: /dkyges-50/?prefix=obj4 2024-04-12T03:48:49.530 DEBUG:boto:Data: 2024-04-12T03:48:49.530 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.531 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.531 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.531 DEBUG:boto:Params: {} 2024-04-12T03:48:49.531 DEBUG:boto:Token: None 2024-04-12T03:48:49.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.531 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d791fe308549b920-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.534 DEBUG:boto:b'dkyges-50obj41000falseobj42024-04-12T03:47:34.461Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.535 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.535 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.535 DEBUG:boto:path=/dkyges-50/?prefix=obj4 2024-04-12T03:48:49.535 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj4 2024-04-12T03:48:49.535 DEBUG:boto:Method: GET 2024-04-12T03:48:49.535 DEBUG:boto:Path: /dkyges-50/?prefix=obj4 2024-04-12T03:48:49.535 DEBUG:boto:Data: 2024-04-12T03:48:49.535 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.536 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.536 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.536 DEBUG:boto:Params: {} 2024-04-12T03:48:49.536 DEBUG:boto:Token: None 2024-04-12T03:48:49.536 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.536 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db87e2a7fe46ee3b-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.539 DEBUG:boto:b'dkyges-50obj41000falseobj42024-04-12T03:47:34.461Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:48:49.539 DEBUG:boto:path=/dkyges-50/obj5 2024-04-12T03:48:49.539 DEBUG:boto:auth_path=/dkyges-50/obj5 2024-04-12T03:48:49.540 DEBUG:boto:path=/dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.540 DEBUG:boto:auth_path=/dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.540 DEBUG:boto:Method: GET 2024-04-12T03:48:49.540 DEBUG:boto:Path: /dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.540 DEBUG:boto:Data: 2024-04-12T03:48:49.540 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.540 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.540 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.540 DEBUG:boto:Params: {} 2024-04-12T03:48:49.540 DEBUG:boto:Token: None 2024-04-12T03:48:49.540 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.541 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AcqZ2abAjhuCGo2bN/jihqj8U7M= 2024-04-12T03:48:49.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AcqZ2abAjhuCGo2bN/jihqj8U7M=', 'Content-Length': '0'} 2024-04-12T03:48:49.542 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a526a51803a9b787-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.543 DEBUG:boto:path=/dkyges-50/obj5 2024-04-12T03:48:49.543 DEBUG:boto:auth_path=/dkyges-50/obj5 2024-04-12T03:48:49.543 DEBUG:boto:path=/dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.543 DEBUG:boto:auth_path=/dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.543 DEBUG:boto:Method: GET 2024-04-12T03:48:49.543 DEBUG:boto:Path: /dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.543 DEBUG:boto:Data: 2024-04-12T03:48:49.543 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.543 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.543 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.543 DEBUG:boto:Params: {} 2024-04-12T03:48:49.544 DEBUG:boto:Token: None 2024-04-12T03:48:49.544 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.544 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AcqZ2abAjhuCGo2bN/jihqj8U7M= 2024-04-12T03:48:49.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AcqZ2abAjhuCGo2bN/jihqj8U7M=', 'Content-Length': '0'} 2024-04-12T03:48:49.546 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000516438a44ff5e722-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.546 DEBUG:boto:path=/dkyges-50/obj5 2024-04-12T03:48:49.546 DEBUG:boto:auth_path=/dkyges-50/obj5 2024-04-12T03:48:49.546 DEBUG:boto:path=/dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.547 DEBUG:boto:auth_path=/dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.547 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.547 DEBUG:boto:Path: /dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.547 DEBUG:boto:Data: 2024-04-12T03:48:49.547 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.547 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.547 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.547 DEBUG:boto:Params: {} 2024-04-12T03:48:49.547 DEBUG:boto:Token: None 2024-04-12T03:48:49.547 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.547 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fFk7v8PINjOlHRwGq5xQkobQNbg= 2024-04-12T03:48:49.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fFk7v8PINjOlHRwGq5xQkobQNbg=', 'Content-Length': '0'} 2024-04-12T03:48:49.549 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9894d31c434667d-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.550 DEBUG:boto:path=/dkyges-50/obj5 2024-04-12T03:48:49.550 DEBUG:boto:auth_path=/dkyges-50/obj5 2024-04-12T03:48:49.550 DEBUG:boto:path=/dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.550 DEBUG:boto:auth_path=/dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.550 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.550 DEBUG:boto:Path: /dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.550 DEBUG:boto:Data: 2024-04-12T03:48:49.550 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.550 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.550 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.551 DEBUG:boto:Params: {} 2024-04-12T03:48:49.551 DEBUG:boto:Token: None 2024-04-12T03:48:49.551 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.551 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fFk7v8PINjOlHRwGq5xQkobQNbg= 2024-04-12T03:48:49.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fFk7v8PINjOlHRwGq5xQkobQNbg=', 'Content-Length': '0'} 2024-04-12T03:48:49.553 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000831cb7ed986d81e-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:48:49.553 DEBUG:boto:path=/dkyges-50/obj5 2024-04-12T03:48:49.553 DEBUG:boto:auth_path=/dkyges-50/obj5 2024-04-12T03:48:49.553 DEBUG:boto:path=/dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.553 DEBUG:boto:auth_path=/dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.554 DEBUG:boto:Method: GET 2024-04-12T03:48:49.554 DEBUG:boto:Path: /dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.554 DEBUG:boto:Data: 2024-04-12T03:48:49.554 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.554 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.554 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.554 DEBUG:boto:Params: {} 2024-04-12T03:48:49.554 DEBUG:boto:Token: None 2024-04-12T03:48:49.554 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.554 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AcqZ2abAjhuCGo2bN/jihqj8U7M= 2024-04-12T03:48:49.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AcqZ2abAjhuCGo2bN/jihqj8U7M=', 'Content-Length': '0'} 2024-04-12T03:48:49.556 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b8f79bca2c5a806-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.557 DEBUG:boto:path=/dkyges-50/obj5 2024-04-12T03:48:49.557 DEBUG:boto:auth_path=/dkyges-50/obj5 2024-04-12T03:48:49.557 DEBUG:boto:path=/dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.557 DEBUG:boto:auth_path=/dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.557 DEBUG:boto:Method: GET 2024-04-12T03:48:49.557 DEBUG:boto:Path: /dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.557 DEBUG:boto:Data: 2024-04-12T03:48:49.557 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.557 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.558 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.558 DEBUG:boto:Params: {} 2024-04-12T03:48:49.558 DEBUG:boto:Token: None 2024-04-12T03:48:49.558 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj5?versionId=null 2024-04-12T03:48:49.558 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AcqZ2abAjhuCGo2bN/jihqj8U7M= 2024-04-12T03:48:49.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AcqZ2abAjhuCGo2bN/jihqj8U7M=', 'Content-Length': '0'} 2024-04-12T03:48:49.560 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb78aa30cffbc72f-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.560 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.560 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.560 DEBUG:boto:path=/dkyges-50/?prefix=obj5 2024-04-12T03:48:49.560 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj5 2024-04-12T03:48:49.560 DEBUG:boto:Method: GET 2024-04-12T03:48:49.561 DEBUG:boto:Path: /dkyges-50/?prefix=obj5 2024-04-12T03:48:49.561 DEBUG:boto:Data: 2024-04-12T03:48:49.561 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.561 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.561 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.561 DEBUG:boto:Params: {} 2024-04-12T03:48:49.561 DEBUG:boto:Token: None 2024-04-12T03:48:49.561 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.561 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000398fb63eadcda98d-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.564 DEBUG:boto:b'dkyges-50obj51000falseobj52024-04-12T03:48:17.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.565 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.565 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.565 DEBUG:boto:path=/dkyges-50/?prefix=obj5 2024-04-12T03:48:49.565 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj5 2024-04-12T03:48:49.565 DEBUG:boto:Method: GET 2024-04-12T03:48:49.565 DEBUG:boto:Path: /dkyges-50/?prefix=obj5 2024-04-12T03:48:49.565 DEBUG:boto:Data: 2024-04-12T03:48:49.565 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.565 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.566 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.566 DEBUG:boto:Params: {} 2024-04-12T03:48:49.566 DEBUG:boto:Token: None 2024-04-12T03:48:49.566 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.566 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0bb5cd616adae0a-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.568 DEBUG:boto:b'dkyges-50obj51000falseobj52024-04-12T03:48:17.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.569 DEBUG:boto:path=/dkyges-50/obj5 2024-04-12T03:48:49.569 DEBUG:boto:auth_path=/dkyges-50/obj5 2024-04-12T03:48:49.569 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.569 DEBUG:boto:Path: /dkyges-50/obj5 2024-04-12T03:48:49.569 DEBUG:boto:Data: 2024-04-12T03:48:49.569 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.569 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.569 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.569 DEBUG:boto:Params: {} 2024-04-12T03:48:49.570 DEBUG:boto:Token: None 2024-04-12T03:48:49.570 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj5 2024-04-12T03:48:49.570 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pjvTBkD6zfQzb2L7vV0BL3O1gHA= 2024-04-12T03:48:49.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pjvTBkD6zfQzb2L7vV0BL3O1gHA=', 'Content-Length': '0'} 2024-04-12T03:48:49.572 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005026e4a91241eca9-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.572 DEBUG:boto:path=/dkyges-50/obj5 2024-04-12T03:48:49.572 DEBUG:boto:auth_path=/dkyges-50/obj5 2024-04-12T03:48:49.572 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.572 DEBUG:boto:Path: /dkyges-50/obj5 2024-04-12T03:48:49.572 DEBUG:boto:Data: 2024-04-12T03:48:49.573 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.573 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.573 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.573 DEBUG:boto:Params: {} 2024-04-12T03:48:49.573 DEBUG:boto:Token: None 2024-04-12T03:48:49.573 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj5 2024-04-12T03:48:49.573 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pjvTBkD6zfQzb2L7vV0BL3O1gHA= 2024-04-12T03:48:49.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pjvTBkD6zfQzb2L7vV0BL3O1gHA=', 'Content-Length': '0'} 2024-04-12T03:48:49.575 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000087f62ad7c516ffcb-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:48:49.575 DEBUG:boto:path=/dkyges-50/obj5 2024-04-12T03:48:49.575 DEBUG:boto:auth_path=/dkyges-50/obj5 2024-04-12T03:48:49.576 DEBUG:boto:Method: GET 2024-04-12T03:48:49.576 DEBUG:boto:Path: /dkyges-50/obj5 2024-04-12T03:48:49.576 DEBUG:boto:Data: 2024-04-12T03:48:49.576 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.576 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.576 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.576 DEBUG:boto:Params: {} 2024-04-12T03:48:49.576 DEBUG:boto:Token: None 2024-04-12T03:48:49.576 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj5 2024-04-12T03:48:49.576 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZIfT/n6CvM5TEQMr9OeBuOmuqD4= 2024-04-12T03:48:49.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZIfT/n6CvM5TEQMr9OeBuOmuqD4=', 'Content-Length': '0'} 2024-04-12T03:48:49.578 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000565f80c75682af5c-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.622 DEBUG:boto:path=/dkyges-50/obj5 2024-04-12T03:48:49.623 DEBUG:boto:auth_path=/dkyges-50/obj5 2024-04-12T03:48:49.623 DEBUG:boto:Method: GET 2024-04-12T03:48:49.623 DEBUG:boto:Path: /dkyges-50/obj5 2024-04-12T03:48:49.623 DEBUG:boto:Data: 2024-04-12T03:48:49.623 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.623 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.623 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.623 DEBUG:boto:Params: {} 2024-04-12T03:48:49.624 DEBUG:boto:Token: None 2024-04-12T03:48:49.624 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj5 2024-04-12T03:48:49.624 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZIfT/n6CvM5TEQMr9OeBuOmuqD4= 2024-04-12T03:48:49.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZIfT/n6CvM5TEQMr9OeBuOmuqD4=', 'Content-Length': '0'} 2024-04-12T03:48:49.626 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec2534adccb9c702-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.626 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.626 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.627 DEBUG:boto:path=/dkyges-50/?prefix=obj5 2024-04-12T03:48:49.627 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj5 2024-04-12T03:48:49.627 DEBUG:boto:Method: GET 2024-04-12T03:48:49.627 DEBUG:boto:Path: /dkyges-50/?prefix=obj5 2024-04-12T03:48:49.627 DEBUG:boto:Data: 2024-04-12T03:48:49.627 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.627 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.627 DEBUG:boto:Params: {} 2024-04-12T03:48:49.628 DEBUG:boto:Token: None 2024-04-12T03:48:49.628 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e49e34238a6cf966-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.630 DEBUG:boto:b'dkyges-50obj51000falseobj52024-04-12T03:48:17.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.631 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.631 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.631 DEBUG:boto:path=/dkyges-50/?prefix=obj5 2024-04-12T03:48:49.631 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj5 2024-04-12T03:48:49.631 DEBUG:boto:Method: GET 2024-04-12T03:48:49.632 DEBUG:boto:Path: /dkyges-50/?prefix=obj5 2024-04-12T03:48:49.632 DEBUG:boto:Data: 2024-04-12T03:48:49.632 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.632 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.632 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.632 DEBUG:boto:Params: {} 2024-04-12T03:48:49.633 DEBUG:boto:Token: None 2024-04-12T03:48:49.633 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.633 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f53d7d91db0bcfe4-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.636 DEBUG:boto:b'dkyges-50obj51000falseobj52024-04-12T03:48:17.393Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:48:49.636 DEBUG:boto:path=/dkyges-50/obj6 2024-04-12T03:48:49.636 DEBUG:boto:auth_path=/dkyges-50/obj6 2024-04-12T03:48:49.637 DEBUG:boto:path=/dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.637 DEBUG:boto:auth_path=/dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.637 DEBUG:boto:Method: GET 2024-04-12T03:48:49.637 DEBUG:boto:Path: /dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.637 DEBUG:boto:Data: 2024-04-12T03:48:49.637 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.637 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.637 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.637 DEBUG:boto:Params: {} 2024-04-12T03:48:49.637 DEBUG:boto:Token: None 2024-04-12T03:48:49.638 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.638 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y1mwnLxc886I8citsaavZfZ2fAE= 2024-04-12T03:48:49.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y1mwnLxc886I8citsaavZfZ2fAE=', 'Content-Length': '0'} 2024-04-12T03:48:49.639 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000233b416c019bd4a7-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.640 DEBUG:boto:path=/dkyges-50/obj6 2024-04-12T03:48:49.640 DEBUG:boto:auth_path=/dkyges-50/obj6 2024-04-12T03:48:49.640 DEBUG:boto:path=/dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.640 DEBUG:boto:auth_path=/dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.640 DEBUG:boto:Method: GET 2024-04-12T03:48:49.640 DEBUG:boto:Path: /dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.640 DEBUG:boto:Data: 2024-04-12T03:48:49.640 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.640 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.641 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.641 DEBUG:boto:Params: {} 2024-04-12T03:48:49.641 DEBUG:boto:Token: None 2024-04-12T03:48:49.641 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.641 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y1mwnLxc886I8citsaavZfZ2fAE= 2024-04-12T03:48:49.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y1mwnLxc886I8citsaavZfZ2fAE=', 'Content-Length': '0'} 2024-04-12T03:48:49.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a160abb61dd27754-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.643 DEBUG:boto:path=/dkyges-50/obj6 2024-04-12T03:48:49.643 DEBUG:boto:auth_path=/dkyges-50/obj6 2024-04-12T03:48:49.643 DEBUG:boto:path=/dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.644 DEBUG:boto:auth_path=/dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.644 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.644 DEBUG:boto:Path: /dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.644 DEBUG:boto:Data: 2024-04-12T03:48:49.644 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.644 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.644 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.644 DEBUG:boto:Params: {} 2024-04-12T03:48:49.644 DEBUG:boto:Token: None 2024-04-12T03:48:49.644 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.645 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:o9uls4ju5a4nOLbUhTx1eekXEWM= 2024-04-12T03:48:49.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:o9uls4ju5a4nOLbUhTx1eekXEWM=', 'Content-Length': '0'} 2024-04-12T03:48:49.646 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7fb9fadd174aa25-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.646 DEBUG:boto:path=/dkyges-50/obj6 2024-04-12T03:48:49.647 DEBUG:boto:auth_path=/dkyges-50/obj6 2024-04-12T03:48:49.647 DEBUG:boto:path=/dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.647 DEBUG:boto:auth_path=/dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.647 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.647 DEBUG:boto:Path: /dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.647 DEBUG:boto:Data: 2024-04-12T03:48:49.647 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.647 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.647 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.647 DEBUG:boto:Params: {} 2024-04-12T03:48:49.647 DEBUG:boto:Token: None 2024-04-12T03:48:49.648 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.648 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:o9uls4ju5a4nOLbUhTx1eekXEWM= 2024-04-12T03:48:49.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:o9uls4ju5a4nOLbUhTx1eekXEWM=', 'Content-Length': '0'} 2024-04-12T03:48:49.650 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b77ded8dfaf91e73-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:48:49.650 DEBUG:boto:path=/dkyges-50/obj6 2024-04-12T03:48:49.650 DEBUG:boto:auth_path=/dkyges-50/obj6 2024-04-12T03:48:49.650 DEBUG:boto:path=/dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.650 DEBUG:boto:auth_path=/dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.650 DEBUG:boto:Method: GET 2024-04-12T03:48:49.650 DEBUG:boto:Path: /dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.651 DEBUG:boto:Data: 2024-04-12T03:48:49.651 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.651 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.651 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.651 DEBUG:boto:Params: {} 2024-04-12T03:48:49.651 DEBUG:boto:Token: None 2024-04-12T03:48:49.651 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.651 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y1mwnLxc886I8citsaavZfZ2fAE= 2024-04-12T03:48:49.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y1mwnLxc886I8citsaavZfZ2fAE=', 'Content-Length': '0'} 2024-04-12T03:48:49.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007100f46b275b2415-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.694 DEBUG:boto:path=/dkyges-50/obj6 2024-04-12T03:48:49.695 DEBUG:boto:auth_path=/dkyges-50/obj6 2024-04-12T03:48:49.695 DEBUG:boto:path=/dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.695 DEBUG:boto:auth_path=/dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.695 DEBUG:boto:Method: GET 2024-04-12T03:48:49.695 DEBUG:boto:Path: /dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.695 DEBUG:boto:Data: 2024-04-12T03:48:49.695 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.695 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.695 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.695 DEBUG:boto:Params: {} 2024-04-12T03:48:49.696 DEBUG:boto:Token: None 2024-04-12T03:48:49.696 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj6?versionId=null 2024-04-12T03:48:49.696 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y1mwnLxc886I8citsaavZfZ2fAE= 2024-04-12T03:48:49.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y1mwnLxc886I8citsaavZfZ2fAE=', 'Content-Length': '0'} 2024-04-12T03:48:49.698 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad351b00be2dce7f-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.698 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.698 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.699 DEBUG:boto:path=/dkyges-50/?prefix=obj6 2024-04-12T03:48:49.699 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj6 2024-04-12T03:48:49.699 DEBUG:boto:Method: GET 2024-04-12T03:48:49.699 DEBUG:boto:Path: /dkyges-50/?prefix=obj6 2024-04-12T03:48:49.699 DEBUG:boto:Data: 2024-04-12T03:48:49.699 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.699 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.699 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.699 DEBUG:boto:Params: {} 2024-04-12T03:48:49.699 DEBUG:boto:Token: None 2024-04-12T03:48:49.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.700 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003334d63556e403c9-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.702 DEBUG:boto:b'dkyges-50obj61000falseobj62024-04-12T03:48:17.403Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.703 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.703 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.703 DEBUG:boto:path=/dkyges-50/?prefix=obj6 2024-04-12T03:48:49.703 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj6 2024-04-12T03:48:49.703 DEBUG:boto:Method: GET 2024-04-12T03:48:49.703 DEBUG:boto:Path: /dkyges-50/?prefix=obj6 2024-04-12T03:48:49.703 DEBUG:boto:Data: 2024-04-12T03:48:49.703 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.704 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.704 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.704 DEBUG:boto:Params: {} 2024-04-12T03:48:49.704 DEBUG:boto:Token: None 2024-04-12T03:48:49.704 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.704 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c22307ee0b326c6b-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.707 DEBUG:boto:b'dkyges-50obj61000falseobj62024-04-12T03:48:17.403Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.707 DEBUG:boto:path=/dkyges-50/obj6 2024-04-12T03:48:49.707 DEBUG:boto:auth_path=/dkyges-50/obj6 2024-04-12T03:48:49.707 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.707 DEBUG:boto:Path: /dkyges-50/obj6 2024-04-12T03:48:49.707 DEBUG:boto:Data: 2024-04-12T03:48:49.707 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.708 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.708 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.708 DEBUG:boto:Params: {} 2024-04-12T03:48:49.708 DEBUG:boto:Token: None 2024-04-12T03:48:49.708 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj6 2024-04-12T03:48:49.708 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P60+G2c1B4BLLUKgAgoUGYCYgZ0= 2024-04-12T03:48:49.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P60+G2c1B4BLLUKgAgoUGYCYgZ0=', 'Content-Length': '0'} 2024-04-12T03:48:49.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c83bc388f658fe7a-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.710 DEBUG:boto:path=/dkyges-50/obj6 2024-04-12T03:48:49.710 DEBUG:boto:auth_path=/dkyges-50/obj6 2024-04-12T03:48:49.710 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.710 DEBUG:boto:Path: /dkyges-50/obj6 2024-04-12T03:48:49.710 DEBUG:boto:Data: 2024-04-12T03:48:49.711 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.711 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.711 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.711 DEBUG:boto:Params: {} 2024-04-12T03:48:49.711 DEBUG:boto:Token: None 2024-04-12T03:48:49.711 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj6 2024-04-12T03:48:49.711 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P60+G2c1B4BLLUKgAgoUGYCYgZ0= 2024-04-12T03:48:49.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P60+G2c1B4BLLUKgAgoUGYCYgZ0=', 'Content-Length': '0'} 2024-04-12T03:48:49.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eff91109d1bad32d-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:48:49.713 DEBUG:boto:path=/dkyges-50/obj6 2024-04-12T03:48:49.713 DEBUG:boto:auth_path=/dkyges-50/obj6 2024-04-12T03:48:49.714 DEBUG:boto:Method: GET 2024-04-12T03:48:49.714 DEBUG:boto:Path: /dkyges-50/obj6 2024-04-12T03:48:49.714 DEBUG:boto:Data: 2024-04-12T03:48:49.714 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.714 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.714 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.714 DEBUG:boto:Params: {} 2024-04-12T03:48:49.714 DEBUG:boto:Token: None 2024-04-12T03:48:49.714 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj6 2024-04-12T03:48:49.714 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FNenWpAr5zkTiBq7N0SZ6BEgY2Y= 2024-04-12T03:48:49.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FNenWpAr5zkTiBq7N0SZ6BEgY2Y=', 'Content-Length': '0'} 2024-04-12T03:48:49.716 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c096a674c2ccb5c-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.716 DEBUG:boto:path=/dkyges-50/obj6 2024-04-12T03:48:49.716 DEBUG:boto:auth_path=/dkyges-50/obj6 2024-04-12T03:48:49.717 DEBUG:boto:Method: GET 2024-04-12T03:48:49.717 DEBUG:boto:Path: /dkyges-50/obj6 2024-04-12T03:48:49.717 DEBUG:boto:Data: 2024-04-12T03:48:49.717 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.717 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.717 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.717 DEBUG:boto:Params: {} 2024-04-12T03:48:49.717 DEBUG:boto:Token: None 2024-04-12T03:48:49.717 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj6 2024-04-12T03:48:49.717 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FNenWpAr5zkTiBq7N0SZ6BEgY2Y= 2024-04-12T03:48:49.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FNenWpAr5zkTiBq7N0SZ6BEgY2Y=', 'Content-Length': '0'} 2024-04-12T03:48:49.719 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2f6382459e9fedf-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.720 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.720 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.720 DEBUG:boto:path=/dkyges-50/?prefix=obj6 2024-04-12T03:48:49.720 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj6 2024-04-12T03:48:49.720 DEBUG:boto:Method: GET 2024-04-12T03:48:49.720 DEBUG:boto:Path: /dkyges-50/?prefix=obj6 2024-04-12T03:48:49.720 DEBUG:boto:Data: 2024-04-12T03:48:49.720 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.720 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.720 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.720 DEBUG:boto:Params: {} 2024-04-12T03:48:49.720 DEBUG:boto:Token: None 2024-04-12T03:48:49.721 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.721 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1c11bcb330dad32-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.724 DEBUG:boto:b'dkyges-50obj61000falseobj62024-04-12T03:48:17.403Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.724 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.724 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.724 DEBUG:boto:path=/dkyges-50/?prefix=obj6 2024-04-12T03:48:49.724 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj6 2024-04-12T03:48:49.724 DEBUG:boto:Method: GET 2024-04-12T03:48:49.724 DEBUG:boto:Path: /dkyges-50/?prefix=obj6 2024-04-12T03:48:49.725 DEBUG:boto:Data: 2024-04-12T03:48:49.725 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.725 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.725 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.725 DEBUG:boto:Params: {} 2024-04-12T03:48:49.725 DEBUG:boto:Token: None 2024-04-12T03:48:49.725 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.726 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f981957d6ef840d7-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.728 DEBUG:boto:b'dkyges-50obj61000falseobj62024-04-12T03:48:17.403Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:48:49.728 DEBUG:boto:path=/dkyges-50/obj7 2024-04-12T03:48:49.729 DEBUG:boto:auth_path=/dkyges-50/obj7 2024-04-12T03:48:49.729 DEBUG:boto:path=/dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.729 DEBUG:boto:auth_path=/dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.729 DEBUG:boto:Method: GET 2024-04-12T03:48:49.729 DEBUG:boto:Path: /dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.729 DEBUG:boto:Data: 2024-04-12T03:48:49.729 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.729 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.729 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.729 DEBUG:boto:Params: {} 2024-04-12T03:48:49.729 DEBUG:boto:Token: None 2024-04-12T03:48:49.730 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.730 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1BhSG6yXHZPjJFXUWiJc/nKL85Q= 2024-04-12T03:48:49.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1BhSG6yXHZPjJFXUWiJc/nKL85Q=', 'Content-Length': '0'} 2024-04-12T03:48:49.732 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006cca753a8c848a3e-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.774 DEBUG:boto:path=/dkyges-50/obj7 2024-04-12T03:48:49.775 DEBUG:boto:auth_path=/dkyges-50/obj7 2024-04-12T03:48:49.775 DEBUG:boto:path=/dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.775 DEBUG:boto:auth_path=/dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.775 DEBUG:boto:Method: GET 2024-04-12T03:48:49.775 DEBUG:boto:Path: /dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.775 DEBUG:boto:Data: 2024-04-12T03:48:49.775 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.775 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.775 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.775 DEBUG:boto:Params: {} 2024-04-12T03:48:49.776 DEBUG:boto:Token: None 2024-04-12T03:48:49.776 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.776 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1BhSG6yXHZPjJFXUWiJc/nKL85Q= 2024-04-12T03:48:49.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1BhSG6yXHZPjJFXUWiJc/nKL85Q=', 'Content-Length': '0'} 2024-04-12T03:48:49.778 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c97ff03eb711ee9-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.778 DEBUG:boto:path=/dkyges-50/obj7 2024-04-12T03:48:49.778 DEBUG:boto:auth_path=/dkyges-50/obj7 2024-04-12T03:48:49.778 DEBUG:boto:path=/dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.779 DEBUG:boto:auth_path=/dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.779 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.779 DEBUG:boto:Path: /dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.779 DEBUG:boto:Data: 2024-04-12T03:48:49.779 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.779 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.779 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.779 DEBUG:boto:Params: {} 2024-04-12T03:48:49.779 DEBUG:boto:Token: None 2024-04-12T03:48:49.779 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.780 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gZM1N74SJkcMr3WeW/1ged88prw= 2024-04-12T03:48:49.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gZM1N74SJkcMr3WeW/1ged88prw=', 'Content-Length': '0'} 2024-04-12T03:48:49.781 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9a8a3299640c5d5-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.782 DEBUG:boto:path=/dkyges-50/obj7 2024-04-12T03:48:49.782 DEBUG:boto:auth_path=/dkyges-50/obj7 2024-04-12T03:48:49.782 DEBUG:boto:path=/dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.782 DEBUG:boto:auth_path=/dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.782 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.782 DEBUG:boto:Path: /dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.782 DEBUG:boto:Data: 2024-04-12T03:48:49.782 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.782 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.782 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.782 DEBUG:boto:Params: {} 2024-04-12T03:48:49.783 DEBUG:boto:Token: None 2024-04-12T03:48:49.783 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.783 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gZM1N74SJkcMr3WeW/1ged88prw= 2024-04-12T03:48:49.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gZM1N74SJkcMr3WeW/1ged88prw=', 'Content-Length': '0'} 2024-04-12T03:48:49.785 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb23e4c4567471a1-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:48:49.785 DEBUG:boto:path=/dkyges-50/obj7 2024-04-12T03:48:49.785 DEBUG:boto:auth_path=/dkyges-50/obj7 2024-04-12T03:48:49.786 DEBUG:boto:path=/dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.786 DEBUG:boto:auth_path=/dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.786 DEBUG:boto:Method: GET 2024-04-12T03:48:49.786 DEBUG:boto:Path: /dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.786 DEBUG:boto:Data: 2024-04-12T03:48:49.786 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.786 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.786 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.786 DEBUG:boto:Params: {} 2024-04-12T03:48:49.786 DEBUG:boto:Token: None 2024-04-12T03:48:49.787 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.787 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1BhSG6yXHZPjJFXUWiJc/nKL85Q= 2024-04-12T03:48:49.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1BhSG6yXHZPjJFXUWiJc/nKL85Q=', 'Content-Length': '0'} 2024-04-12T03:48:49.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029ea32dff5bc030a-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.789 DEBUG:boto:path=/dkyges-50/obj7 2024-04-12T03:48:49.789 DEBUG:boto:auth_path=/dkyges-50/obj7 2024-04-12T03:48:49.789 DEBUG:boto:path=/dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.790 DEBUG:boto:auth_path=/dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.790 DEBUG:boto:Method: GET 2024-04-12T03:48:49.790 DEBUG:boto:Path: /dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.790 DEBUG:boto:Data: 2024-04-12T03:48:49.790 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.790 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.790 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.790 DEBUG:boto:Params: {} 2024-04-12T03:48:49.790 DEBUG:boto:Token: None 2024-04-12T03:48:49.790 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj7?versionId=null 2024-04-12T03:48:49.791 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1BhSG6yXHZPjJFXUWiJc/nKL85Q= 2024-04-12T03:48:49.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1BhSG6yXHZPjJFXUWiJc/nKL85Q=', 'Content-Length': '0'} 2024-04-12T03:48:49.794 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2e1e521ad901b0a-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.794 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.794 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.794 DEBUG:boto:path=/dkyges-50/?prefix=obj7 2024-04-12T03:48:49.794 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj7 2024-04-12T03:48:49.795 DEBUG:boto:Method: GET 2024-04-12T03:48:49.795 DEBUG:boto:Path: /dkyges-50/?prefix=obj7 2024-04-12T03:48:49.795 DEBUG:boto:Data: 2024-04-12T03:48:49.795 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.795 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.795 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.795 DEBUG:boto:Params: {} 2024-04-12T03:48:49.795 DEBUG:boto:Token: None 2024-04-12T03:48:49.795 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.795 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffe2bf1eff5bf491-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.798 DEBUG:boto:b'dkyges-50obj71000falseobj72024-04-12T03:48:17.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.798 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.799 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.799 DEBUG:boto:path=/dkyges-50/?prefix=obj7 2024-04-12T03:48:49.799 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj7 2024-04-12T03:48:49.799 DEBUG:boto:Method: GET 2024-04-12T03:48:49.799 DEBUG:boto:Path: /dkyges-50/?prefix=obj7 2024-04-12T03:48:49.799 DEBUG:boto:Data: 2024-04-12T03:48:49.799 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.800 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.800 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.800 DEBUG:boto:Params: {} 2024-04-12T03:48:49.800 DEBUG:boto:Token: None 2024-04-12T03:48:49.800 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.800 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2551ce01a64c374-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.803 DEBUG:boto:b'dkyges-50obj71000falseobj72024-04-12T03:48:17.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.803 DEBUG:boto:path=/dkyges-50/obj7 2024-04-12T03:48:49.803 DEBUG:boto:auth_path=/dkyges-50/obj7 2024-04-12T03:48:49.803 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.803 DEBUG:boto:Path: /dkyges-50/obj7 2024-04-12T03:48:49.804 DEBUG:boto:Data: 2024-04-12T03:48:49.804 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.804 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.804 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.804 DEBUG:boto:Params: {} 2024-04-12T03:48:49.804 DEBUG:boto:Token: None 2024-04-12T03:48:49.804 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj7 2024-04-12T03:48:49.804 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fGZ3QrkxgNnSE9hz4Iu/e/0rpyI= 2024-04-12T03:48:49.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fGZ3QrkxgNnSE9hz4Iu/e/0rpyI=', 'Content-Length': '0'} 2024-04-12T03:48:49.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8345b90ea016b3b-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.806 DEBUG:boto:path=/dkyges-50/obj7 2024-04-12T03:48:49.806 DEBUG:boto:auth_path=/dkyges-50/obj7 2024-04-12T03:48:49.807 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.807 DEBUG:boto:Path: /dkyges-50/obj7 2024-04-12T03:48:49.807 DEBUG:boto:Data: 2024-04-12T03:48:49.807 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.807 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.807 DEBUG:boto:Params: {} 2024-04-12T03:48:49.807 DEBUG:boto:Token: None 2024-04-12T03:48:49.807 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj7 2024-04-12T03:48:49.807 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fGZ3QrkxgNnSE9hz4Iu/e/0rpyI= 2024-04-12T03:48:49.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fGZ3QrkxgNnSE9hz4Iu/e/0rpyI=', 'Content-Length': '0'} 2024-04-12T03:48:49.809 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6720b69f8d24927-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:48:49.810 DEBUG:boto:path=/dkyges-50/obj7 2024-04-12T03:48:49.810 DEBUG:boto:auth_path=/dkyges-50/obj7 2024-04-12T03:48:49.810 DEBUG:boto:Method: GET 2024-04-12T03:48:49.810 DEBUG:boto:Path: /dkyges-50/obj7 2024-04-12T03:48:49.810 DEBUG:boto:Data: 2024-04-12T03:48:49.810 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.810 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.810 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.810 DEBUG:boto:Params: {} 2024-04-12T03:48:49.810 DEBUG:boto:Token: None 2024-04-12T03:48:49.810 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj7 2024-04-12T03:48:49.811 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nc3s75v/bNKT/L4cx0nmE2zjeqc= 2024-04-12T03:48:49.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nc3s75v/bNKT/L4cx0nmE2zjeqc=', 'Content-Length': '0'} 2024-04-12T03:48:49.812 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ebb1e69eaa738936-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.813 DEBUG:boto:path=/dkyges-50/obj7 2024-04-12T03:48:49.813 DEBUG:boto:auth_path=/dkyges-50/obj7 2024-04-12T03:48:49.813 DEBUG:boto:Method: GET 2024-04-12T03:48:49.813 DEBUG:boto:Path: /dkyges-50/obj7 2024-04-12T03:48:49.813 DEBUG:boto:Data: 2024-04-12T03:48:49.813 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.813 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.813 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.813 DEBUG:boto:Params: {} 2024-04-12T03:48:49.813 DEBUG:boto:Token: None 2024-04-12T03:48:49.813 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj7 2024-04-12T03:48:49.814 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nc3s75v/bNKT/L4cx0nmE2zjeqc= 2024-04-12T03:48:49.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nc3s75v/bNKT/L4cx0nmE2zjeqc=', 'Content-Length': '0'} 2024-04-12T03:48:49.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f76cbe7daaed5a5a-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.858 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.859 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.859 DEBUG:boto:path=/dkyges-50/?prefix=obj7 2024-04-12T03:48:49.859 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj7 2024-04-12T03:48:49.859 DEBUG:boto:Method: GET 2024-04-12T03:48:49.859 DEBUG:boto:Path: /dkyges-50/?prefix=obj7 2024-04-12T03:48:49.859 DEBUG:boto:Data: 2024-04-12T03:48:49.859 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.859 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.859 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.859 DEBUG:boto:Params: {} 2024-04-12T03:48:49.860 DEBUG:boto:Token: None 2024-04-12T03:48:49.860 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.860 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000445e5fb10e4088e3-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.862 DEBUG:boto:b'dkyges-50obj71000falseobj72024-04-12T03:48:17.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.863 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.863 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.863 DEBUG:boto:path=/dkyges-50/?prefix=obj7 2024-04-12T03:48:49.863 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj7 2024-04-12T03:48:49.863 DEBUG:boto:Method: GET 2024-04-12T03:48:49.863 DEBUG:boto:Path: /dkyges-50/?prefix=obj7 2024-04-12T03:48:49.863 DEBUG:boto:Data: 2024-04-12T03:48:49.864 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.864 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.864 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.864 DEBUG:boto:Params: {} 2024-04-12T03:48:49.864 DEBUG:boto:Token: None 2024-04-12T03:48:49.864 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.864 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001092074103ecae6f-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.867 DEBUG:boto:b'dkyges-50obj71000falseobj72024-04-12T03:48:17.454Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:48:49.867 DEBUG:boto:path=/dkyges-50/obj8 2024-04-12T03:48:49.868 DEBUG:boto:auth_path=/dkyges-50/obj8 2024-04-12T03:48:49.868 DEBUG:boto:path=/dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.868 DEBUG:boto:auth_path=/dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.868 DEBUG:boto:Method: GET 2024-04-12T03:48:49.868 DEBUG:boto:Path: /dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.868 DEBUG:boto:Data: 2024-04-12T03:48:49.868 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.868 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.868 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.868 DEBUG:boto:Params: {} 2024-04-12T03:48:49.869 DEBUG:boto:Token: None 2024-04-12T03:48:49.869 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.869 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X3JlkNnO1EB9XglfNxKEc+cNY9c= 2024-04-12T03:48:49.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X3JlkNnO1EB9XglfNxKEc+cNY9c=', 'Content-Length': '0'} 2024-04-12T03:48:49.871 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d629f76161c2bf5-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.871 DEBUG:boto:path=/dkyges-50/obj8 2024-04-12T03:48:49.872 DEBUG:boto:auth_path=/dkyges-50/obj8 2024-04-12T03:48:49.872 DEBUG:boto:path=/dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.872 DEBUG:boto:auth_path=/dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.872 DEBUG:boto:Method: GET 2024-04-12T03:48:49.872 DEBUG:boto:Path: /dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.872 DEBUG:boto:Data: 2024-04-12T03:48:49.872 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.872 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.872 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.872 DEBUG:boto:Params: {} 2024-04-12T03:48:49.872 DEBUG:boto:Token: None 2024-04-12T03:48:49.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.873 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X3JlkNnO1EB9XglfNxKEc+cNY9c= 2024-04-12T03:48:49.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X3JlkNnO1EB9XglfNxKEc+cNY9c=', 'Content-Length': '0'} 2024-04-12T03:48:49.875 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfeeb83be34b6b87-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.875 DEBUG:boto:path=/dkyges-50/obj8 2024-04-12T03:48:49.875 DEBUG:boto:auth_path=/dkyges-50/obj8 2024-04-12T03:48:49.875 DEBUG:boto:path=/dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.875 DEBUG:boto:auth_path=/dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.875 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.875 DEBUG:boto:Path: /dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.875 DEBUG:boto:Data: 2024-04-12T03:48:49.876 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.876 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.876 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.876 DEBUG:boto:Params: {} 2024-04-12T03:48:49.876 DEBUG:boto:Token: None 2024-04-12T03:48:49.876 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.876 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wsgio7uf1bRT9Y0IUdq/zsFNruE= 2024-04-12T03:48:49.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wsgio7uf1bRT9Y0IUdq/zsFNruE=', 'Content-Length': '0'} 2024-04-12T03:48:49.878 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d0b659d5a4cb01f-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.878 DEBUG:boto:path=/dkyges-50/obj8 2024-04-12T03:48:49.878 DEBUG:boto:auth_path=/dkyges-50/obj8 2024-04-12T03:48:49.878 DEBUG:boto:path=/dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.879 DEBUG:boto:auth_path=/dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.879 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.879 DEBUG:boto:Path: /dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.879 DEBUG:boto:Data: 2024-04-12T03:48:49.879 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.879 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.879 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.879 DEBUG:boto:Params: {} 2024-04-12T03:48:49.879 DEBUG:boto:Token: None 2024-04-12T03:48:49.879 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.880 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wsgio7uf1bRT9Y0IUdq/zsFNruE= 2024-04-12T03:48:49.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wsgio7uf1bRT9Y0IUdq/zsFNruE=', 'Content-Length': '0'} 2024-04-12T03:48:49.881 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000332878507f44b08a-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:48:49.882 DEBUG:boto:path=/dkyges-50/obj8 2024-04-12T03:48:49.882 DEBUG:boto:auth_path=/dkyges-50/obj8 2024-04-12T03:48:49.882 DEBUG:boto:path=/dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.882 DEBUG:boto:auth_path=/dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.882 DEBUG:boto:Method: GET 2024-04-12T03:48:49.882 DEBUG:boto:Path: /dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.882 DEBUG:boto:Data: 2024-04-12T03:48:49.882 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.882 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.883 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.883 DEBUG:boto:Params: {} 2024-04-12T03:48:49.883 DEBUG:boto:Token: None 2024-04-12T03:48:49.883 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.883 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X3JlkNnO1EB9XglfNxKEc+cNY9c= 2024-04-12T03:48:49.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X3JlkNnO1EB9XglfNxKEc+cNY9c=', 'Content-Length': '0'} 2024-04-12T03:48:49.885 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b82ea3ab9b3edc2f-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.926 DEBUG:boto:path=/dkyges-50/obj8 2024-04-12T03:48:49.927 DEBUG:boto:auth_path=/dkyges-50/obj8 2024-04-12T03:48:49.927 DEBUG:boto:path=/dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.927 DEBUG:boto:auth_path=/dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.927 DEBUG:boto:Method: GET 2024-04-12T03:48:49.927 DEBUG:boto:Path: /dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.927 DEBUG:boto:Data: 2024-04-12T03:48:49.927 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.927 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.927 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.927 DEBUG:boto:Params: {} 2024-04-12T03:48:49.928 DEBUG:boto:Token: None 2024-04-12T03:48:49.928 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj8?versionId=null 2024-04-12T03:48:49.928 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X3JlkNnO1EB9XglfNxKEc+cNY9c= 2024-04-12T03:48:49.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X3JlkNnO1EB9XglfNxKEc+cNY9c=', 'Content-Length': '0'} 2024-04-12T03:48:49.930 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d527ae8eebcf0d77-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.931 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.931 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.931 DEBUG:boto:path=/dkyges-50/?prefix=obj8 2024-04-12T03:48:49.931 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj8 2024-04-12T03:48:49.931 DEBUG:boto:Method: GET 2024-04-12T03:48:49.931 DEBUG:boto:Path: /dkyges-50/?prefix=obj8 2024-04-12T03:48:49.931 DEBUG:boto:Data: 2024-04-12T03:48:49.931 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.931 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.931 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.931 DEBUG:boto:Params: {} 2024-04-12T03:48:49.932 DEBUG:boto:Token: None 2024-04-12T03:48:49.932 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.932 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d2a536f36ee9cb5-006618af21-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.934 DEBUG:boto:b'dkyges-50obj81000falseobj82024-04-12T03:48:17.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.935 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:49.935 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:49.935 DEBUG:boto:path=/dkyges-50/?prefix=obj8 2024-04-12T03:48:49.935 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj8 2024-04-12T03:48:49.935 DEBUG:boto:Method: GET 2024-04-12T03:48:49.935 DEBUG:boto:Path: /dkyges-50/?prefix=obj8 2024-04-12T03:48:49.935 DEBUG:boto:Data: 2024-04-12T03:48:49.935 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.935 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.935 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.936 DEBUG:boto:Params: {} 2024-04-12T03:48:49.936 DEBUG:boto:Token: None 2024-04-12T03:48:49.936 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/ 2024-04-12T03:48:49.936 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0= 2024-04-12T03:48:49.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obRVuaIJjdKp/FghNH20m8ip3x0=', 'Content-Length': '0'} 2024-04-12T03:48:49.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000978b1fbaff76440a-006618af21-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.939 DEBUG:boto:b'dkyges-50obj81000falseobj82024-04-12T03:48:17.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:49.940 DEBUG:boto:path=/dkyges-50/obj8 2024-04-12T03:48:49.940 DEBUG:boto:auth_path=/dkyges-50/obj8 2024-04-12T03:48:49.940 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.940 DEBUG:boto:Path: /dkyges-50/obj8 2024-04-12T03:48:49.940 DEBUG:boto:Data: 2024-04-12T03:48:49.940 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.940 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.940 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.940 DEBUG:boto:Params: {} 2024-04-12T03:48:49.941 DEBUG:boto:Token: None 2024-04-12T03:48:49.941 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj8 2024-04-12T03:48:49.941 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8d5IY7jpxg3VrLd7HcYF2cMULBA= 2024-04-12T03:48:49.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8d5IY7jpxg3VrLd7HcYF2cMULBA=', 'Content-Length': '0'} 2024-04-12T03:48:49.943 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036782878bb725925-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.944 DEBUG:boto:path=/dkyges-50/obj8 2024-04-12T03:48:49.944 DEBUG:boto:auth_path=/dkyges-50/obj8 2024-04-12T03:48:49.944 DEBUG:boto:Method: HEAD 2024-04-12T03:48:49.944 DEBUG:boto:Path: /dkyges-50/obj8 2024-04-12T03:48:49.944 DEBUG:boto:Data: 2024-04-12T03:48:49.944 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.944 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.944 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.944 DEBUG:boto:Params: {} 2024-04-12T03:48:49.945 DEBUG:boto:Token: None 2024-04-12T03:48:49.945 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj8 2024-04-12T03:48:49.945 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8d5IY7jpxg3VrLd7HcYF2cMULBA= 2024-04-12T03:48:49.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8d5IY7jpxg3VrLd7HcYF2cMULBA=', 'Content-Length': '0'} 2024-04-12T03:48:49.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000088d7a6b968a97607-006618af21-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:48:49.947 DEBUG:boto:path=/dkyges-50/obj8 2024-04-12T03:48:49.947 DEBUG:boto:auth_path=/dkyges-50/obj8 2024-04-12T03:48:49.947 DEBUG:boto:Method: GET 2024-04-12T03:48:49.947 DEBUG:boto:Path: /dkyges-50/obj8 2024-04-12T03:48:49.948 DEBUG:boto:Data: 2024-04-12T03:48:49.948 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.948 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:49.948 DEBUG:boto:Port: 8001 2024-04-12T03:48:49.948 DEBUG:boto:Params: {} 2024-04-12T03:48:49.948 DEBUG:boto:Token: None 2024-04-12T03:48:49.948 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj8 2024-04-12T03:48:49.948 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IhPFhllTR/hsaVih34OY84tAxQ8= 2024-04-12T03:48:49.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IhPFhllTR/hsaVih34OY84tAxQ8=', 'Content-Length': '0'} 2024-04-12T03:48:49.950 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-50:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091d357c6cd0495a5-006618af21-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:49.995 DEBUG:boto:path=/dkyges-50/obj8 2024-04-12T03:48:49.995 DEBUG:boto:auth_path=/dkyges-50/obj8 2024-04-12T03:48:49.995 DEBUG:boto:Method: GET 2024-04-12T03:48:49.995 DEBUG:boto:Path: /dkyges-50/obj8 2024-04-12T03:48:49.995 DEBUG:boto:Data: 2024-04-12T03:48:49.995 DEBUG:boto:Headers: {} 2024-04-12T03:48:49.996 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:49.997 DEBUG:boto:Port: 8000 2024-04-12T03:48:49.998 DEBUG:boto:Params: {} 2024-04-12T03:48:49.998 DEBUG:boto:Token: None 2024-04-12T03:48:49.998 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:49 GMT /dkyges-50/obj8 2024-04-12T03:48:49.998 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IhPFhllTR/hsaVih34OY84tAxQ8= 2024-04-12T03:48:49.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IhPFhllTR/hsaVih34OY84tAxQ8=', 'Content-Length': '0'} 2024-04-12T03:48:50.000 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001fc74fb7ad68956f-006618af22-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:50.001 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:50.001 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:50.001 DEBUG:boto:path=/dkyges-50/?prefix=obj8 2024-04-12T03:48:50.002 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj8 2024-04-12T03:48:50.002 DEBUG:boto:Method: GET 2024-04-12T03:48:50.002 DEBUG:boto:Path: /dkyges-50/?prefix=obj8 2024-04-12T03:48:50.002 DEBUG:boto:Data: 2024-04-12T03:48:50.002 DEBUG:boto:Headers: {} 2024-04-12T03:48:50.003 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:50.003 DEBUG:boto:Port: 8001 2024-04-12T03:48:50.003 DEBUG:boto:Params: {} 2024-04-12T03:48:50.003 DEBUG:boto:Token: None 2024-04-12T03:48:50.003 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:50 GMT /dkyges-50/ 2024-04-12T03:48:50.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ofv7sdOuPSAxo3lindn4/4YMeLE= 2024-04-12T03:48:50.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ofv7sdOuPSAxo3lindn4/4YMeLE=', 'Content-Length': '0'} 2024-04-12T03:48:50.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ee857e2342364dc-006618af22-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:50.006 DEBUG:boto:b'dkyges-50obj81000falseobj82024-04-12T03:48:17.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:50.007 DEBUG:boto:path=/dkyges-50/ 2024-04-12T03:48:50.007 DEBUG:boto:auth_path=/dkyges-50/ 2024-04-12T03:48:50.007 DEBUG:boto:path=/dkyges-50/?prefix=obj8 2024-04-12T03:48:50.007 DEBUG:boto:auth_path=/dkyges-50/?prefix=obj8 2024-04-12T03:48:50.007 DEBUG:boto:Method: GET 2024-04-12T03:48:50.008 DEBUG:boto:Path: /dkyges-50/?prefix=obj8 2024-04-12T03:48:50.008 DEBUG:boto:Data: 2024-04-12T03:48:50.008 DEBUG:boto:Headers: {} 2024-04-12T03:48:50.008 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:50.008 DEBUG:boto:Port: 8000 2024-04-12T03:48:50.008 DEBUG:boto:Params: {} 2024-04-12T03:48:50.008 DEBUG:boto:Token: None 2024-04-12T03:48:50.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:50 GMT /dkyges-50/ 2024-04-12T03:48:50.008 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ofv7sdOuPSAxo3lindn4/4YMeLE= 2024-04-12T03:48:50.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ofv7sdOuPSAxo3lindn4/4YMeLE=', 'Content-Length': '0'} 2024-04-12T03:48:50.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000316be943818fae00-006618af22-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:50.011 DEBUG:boto:b'dkyges-50obj81000falseobj82024-04-12T03:48:17.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:50.011 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-50 zones={a2, a1} 2024-04-12T03:48:50.012 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-51 2024-04-12T03:48:50.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-51', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:48:50.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-51', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:48:50.012 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-51 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:48:50.080 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:48:50.080 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:48:50.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:50.130+0000 7f7b3587d780 1 waiting for incremental sync to catch up: 2024-04-12T03:48:50.126 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.24.1, 00000000002.61.1, 00000000003.35.1, 00000000002.45.1, 00000000003.45.1, 00000000002.36.1, 00000000002.19.1, 00000000003.62.1, 00000000002.59.1, 00000000002.28.1, 00000000002.25.1] 2024-04-12T03:48:50.127 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000004.27.6, 00000000002.61.1, 00000000004.38.6, 00000000002.45.1, 00000000004.47.6, 00000000002.36.1, 00000000002.19.1, 00000000004.64.6, 00000000002.59.1, 00000000002.28.1, 00000000002.25.1] 2024-04-12T03:48:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:55.133+0000 7f7b3587d780 1 bucket sync caught up with source: 2024-04-12T03:48:55.128 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000004.27.6, 00000000002.61.1, 00000000004.38.6, 00000000002.45.1, 00000000004.47.6, 00000000002.36.1, 00000000002.19.1, 00000000004.64.6, 00000000002.59.1, 00000000002.28.1, 00000000002.25.1] 2024-04-12T03:48:55.128 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000004.27.6, 00000000002.61.1, 00000000004.38.6, 00000000002.45.1, 00000000004.47.6, 00000000002.36.1, 00000000002.19.1, 00000000004.64.6, 00000000002.59.1, 00000000002.28.1, 00000000002.25.1] 2024-04-12T03:48:55.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:55.133+0000 7f7b3587d780 0 bucket checkpoint complete 2024-04-12T03:48:55.147 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-51 2024-04-12T03:48:55.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-51', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:48:55.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-51', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:48:55.148 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-51 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:48:55.219 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:48:55.219 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:48:55.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:55.266+0000 7efcd2937780 1 bucket sync caught up with source: 2024-04-12T03:48:55.261 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000004.3044.1, 00000000002.1583.1, 00000000004.3215.1, 00000000002.2575.1, 00000000004.2419.1, 00000000002.3216.1, 00000000002.2508.1, 00000000004.1584.1, 00000000002.2251.1, 00000000002.1395.1, 00000000002.3045.1] 2024-04-12T03:48:55.261 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000004.3044.1, 00000000002.1583.1, 00000000004.3215.1, 00000000002.2575.1, 00000000004.2419.1, 00000000002.3216.1, 00000000002.2508.1, 00000000004.1584.1, 00000000002.2251.1, 00000000002.1395.1, 00000000002.3045.1] 2024-04-12T03:48:55.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:48:55.266+0000 7efcd2937780 0 bucket checkpoint complete 2024-04-12T03:48:55.277 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-51 zones={a2, a1} 2024-04-12T03:48:55.278 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.278 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.278 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.278 DEBUG:boto:Path: /dkyges-51/ 2024-04-12T03:48:55.278 DEBUG:boto:Data: 2024-04-12T03:48:55.278 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.278 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.278 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.278 DEBUG:boto:Params: {} 2024-04-12T03:48:55.279 DEBUG:boto:Token: None 2024-04-12T03:48:55.279 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.279 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TtYS6Uf1O68BF2KkimJ9I6GyCP4= 2024-04-12T03:48:55.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TtYS6Uf1O68BF2KkimJ9I6GyCP4=', 'Content-Length': '0'} 2024-04-12T03:48:55.281 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c1937be46d2601a-006618af27-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.282 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.282 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.282 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.282 DEBUG:boto:Path: /dkyges-51/ 2024-04-12T03:48:55.282 DEBUG:boto:Data: 2024-04-12T03:48:55.282 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.282 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.282 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.282 DEBUG:boto:Params: {} 2024-04-12T03:48:55.282 DEBUG:boto:Token: None 2024-04-12T03:48:55.282 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.283 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TtYS6Uf1O68BF2KkimJ9I6GyCP4= 2024-04-12T03:48:55.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TtYS6Uf1O68BF2KkimJ9I6GyCP4=', 'Content-Length': '0'} 2024-04-12T03:48:55.285 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043b3dd449c394f57-006618af27-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.285 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:48:55.285 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.285 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.285 DEBUG:boto:path=/dkyges-51/?versions&max-keys=999 2024-04-12T03:48:55.286 DEBUG:boto:auth_path=/dkyges-51/?versions&max-keys=999 2024-04-12T03:48:55.286 DEBUG:boto:Method: GET 2024-04-12T03:48:55.286 DEBUG:boto:Path: /dkyges-51/?versions&max-keys=999 2024-04-12T03:48:55.286 DEBUG:boto:Data: 2024-04-12T03:48:55.286 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.286 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.286 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.286 DEBUG:boto:Params: {} 2024-04-12T03:48:55.286 DEBUG:boto:Token: None 2024-04-12T03:48:55.286 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/?versions 2024-04-12T03:48:55.287 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EOPFjf8l/Udx0HCCk84E33H5z3A= 2024-04-12T03:48:55.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EOPFjf8l/Udx0HCCk84E33H5z3A=', 'Content-Length': '0'} 2024-04-12T03:48:55.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6d19d4c9c196ceb-006618af27-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.289 DEBUG:boto:b'dkyges-51999falseobj1nulltrue2024-04-12T03:47:34.473Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:47:34.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:47:34.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:47:34.552Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:48:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:48:17.485Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:48:17.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:48:17.545Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:48:55.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:48:55.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:48:55.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:48:55.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-12T03:48:55.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-12T03:48:55.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-12T03:48:55.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-12T03:48:55.291 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:48:55.291 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.291 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.291 DEBUG:boto:path=/dkyges-51/?versions&max-keys=999 2024-04-12T03:48:55.291 DEBUG:boto:auth_path=/dkyges-51/?versions&max-keys=999 2024-04-12T03:48:55.291 DEBUG:boto:Method: GET 2024-04-12T03:48:55.291 DEBUG:boto:Path: /dkyges-51/?versions&max-keys=999 2024-04-12T03:48:55.291 DEBUG:boto:Data: 2024-04-12T03:48:55.291 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.291 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.292 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.292 DEBUG:boto:Params: {} 2024-04-12T03:48:55.292 DEBUG:boto:Token: None 2024-04-12T03:48:55.292 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/?versions 2024-04-12T03:48:55.292 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EOPFjf8l/Udx0HCCk84E33H5z3A= 2024-04-12T03:48:55.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EOPFjf8l/Udx0HCCk84E33H5z3A=', 'Content-Length': '0'} 2024-04-12T03:48:55.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000648426782de43608-006618af27-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.295 DEBUG:boto:b'dkyges-51999falseobj1nulltrue2024-04-12T03:47:34.473Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:47:34.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:47:34.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:47:34.552Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:48:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:48:17.485Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:48:17.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:48:17.545Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2024-04-12T03:48:55.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2024-04-12T03:48:55.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2024-04-12T03:48:55.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2024-04-12T03:48:55.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2024-04-12T03:48:55.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2024-04-12T03:48:55.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2024-04-12T03:48:55.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2024-04-12T03:48:55.296 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.296 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.296 DEBUG:boto:path=/dkyges-51/?versions&max-keys=999 2024-04-12T03:48:55.297 DEBUG:boto:auth_path=/dkyges-51/?versions&max-keys=999 2024-04-12T03:48:55.297 DEBUG:boto:Method: GET 2024-04-12T03:48:55.297 DEBUG:boto:Path: /dkyges-51/?versions&max-keys=999 2024-04-12T03:48:55.297 DEBUG:boto:Data: 2024-04-12T03:48:55.297 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.297 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.297 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.297 DEBUG:boto:Params: {} 2024-04-12T03:48:55.297 DEBUG:boto:Token: None 2024-04-12T03:48:55.297 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/?versions 2024-04-12T03:48:55.298 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EOPFjf8l/Udx0HCCk84E33H5z3A= 2024-04-12T03:48:55.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EOPFjf8l/Udx0HCCk84E33H5z3A=', 'Content-Length': '0'} 2024-04-12T03:48:55.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086f3b129361b86d1-006618af27-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.300 DEBUG:boto:b'dkyges-51999falseobj1nulltrue2024-04-12T03:47:34.473Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:47:34.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:47:34.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:47:34.552Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:48:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:48:17.485Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:48:17.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:48:17.545Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.301 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.301 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.301 DEBUG:boto:path=/dkyges-51/?versions&max-keys=999 2024-04-12T03:48:55.301 DEBUG:boto:auth_path=/dkyges-51/?versions&max-keys=999 2024-04-12T03:48:55.301 DEBUG:boto:Method: GET 2024-04-12T03:48:55.301 DEBUG:boto:Path: /dkyges-51/?versions&max-keys=999 2024-04-12T03:48:55.301 DEBUG:boto:Data: 2024-04-12T03:48:55.302 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.302 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.302 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.302 DEBUG:boto:Params: {} 2024-04-12T03:48:55.302 DEBUG:boto:Token: None 2024-04-12T03:48:55.302 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/?versions 2024-04-12T03:48:55.302 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EOPFjf8l/Udx0HCCk84E33H5z3A= 2024-04-12T03:48:55.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EOPFjf8l/Udx0HCCk84E33H5z3A=', 'Content-Length': '0'} 2024-04-12T03:48:55.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035e824ac5fdf99f4-006618af27-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.305 DEBUG:boto:b'dkyges-51999falseobj1nulltrue2024-04-12T03:47:34.473Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2024-04-12T03:47:34.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2024-04-12T03:47:34.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2024-04-12T03:47:34.552Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2024-04-12T03:48:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2024-04-12T03:48:17.485Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2024-04-12T03:48:17.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2024-04-12T03:48:17.545Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:48:55.306 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:48:55.306 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:48:55.306 DEBUG:boto:path=/dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.306 DEBUG:boto:auth_path=/dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.306 DEBUG:boto:Method: GET 2024-04-12T03:48:55.306 DEBUG:boto:Path: /dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.306 DEBUG:boto:Data: 2024-04-12T03:48:55.306 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.306 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.307 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.307 DEBUG:boto:Params: {} 2024-04-12T03:48:55.307 DEBUG:boto:Token: None 2024-04-12T03:48:55.307 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.307 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S6hwDuiXVLp51epGXIfZ/QOSgi4= 2024-04-12T03:48:55.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S6hwDuiXVLp51epGXIfZ/QOSgi4=', 'Content-Length': '0'} 2024-04-12T03:48:55.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004f028d3224febe17-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.309 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:48:55.309 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:48:55.309 DEBUG:boto:path=/dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.309 DEBUG:boto:auth_path=/dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.310 DEBUG:boto:Method: GET 2024-04-12T03:48:55.310 DEBUG:boto:Path: /dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.310 DEBUG:boto:Data: 2024-04-12T03:48:55.310 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.310 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.310 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.310 DEBUG:boto:Params: {} 2024-04-12T03:48:55.310 DEBUG:boto:Token: None 2024-04-12T03:48:55.311 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.311 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S6hwDuiXVLp51epGXIfZ/QOSgi4= 2024-04-12T03:48:55.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S6hwDuiXVLp51epGXIfZ/QOSgi4=', 'Content-Length': '0'} 2024-04-12T03:48:55.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098dca5a8988f848b-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.354 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:48:55.355 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:48:55.355 DEBUG:boto:path=/dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.355 DEBUG:boto:auth_path=/dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.355 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.355 DEBUG:boto:Path: /dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.355 DEBUG:boto:Data: 2024-04-12T03:48:55.355 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.355 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.355 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.356 DEBUG:boto:Params: {} 2024-04-12T03:48:55.356 DEBUG:boto:Token: None 2024-04-12T03:48:55.356 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.357 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:spnMjRtXZmsQkFPBRXxWRRPklGU= 2024-04-12T03:48:55.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:spnMjRtXZmsQkFPBRXxWRRPklGU=', 'Content-Length': '0'} 2024-04-12T03:48:55.359 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fb0181d7d3e4e40a-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.359 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:48:55.359 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:48:55.360 DEBUG:boto:path=/dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.360 DEBUG:boto:auth_path=/dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.360 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.360 DEBUG:boto:Path: /dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.360 DEBUG:boto:Data: 2024-04-12T03:48:55.361 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.361 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.361 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.361 DEBUG:boto:Params: {} 2024-04-12T03:48:55.361 DEBUG:boto:Token: None 2024-04-12T03:48:55.362 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.362 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:spnMjRtXZmsQkFPBRXxWRRPklGU= 2024-04-12T03:48:55.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:spnMjRtXZmsQkFPBRXxWRRPklGU=', 'Content-Length': '0'} 2024-04-12T03:48:55.364 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e4d21ec82a80faa8-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:48:55.365 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:48:55.365 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:48:55.365 DEBUG:boto:path=/dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.365 DEBUG:boto:auth_path=/dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.366 DEBUG:boto:Method: GET 2024-04-12T03:48:55.366 DEBUG:boto:Path: /dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.366 DEBUG:boto:Data: 2024-04-12T03:48:55.366 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.366 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.367 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.367 DEBUG:boto:Params: {} 2024-04-12T03:48:55.367 DEBUG:boto:Token: None 2024-04-12T03:48:55.367 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.368 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S6hwDuiXVLp51epGXIfZ/QOSgi4= 2024-04-12T03:48:55.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S6hwDuiXVLp51epGXIfZ/QOSgi4=', 'Content-Length': '0'} 2024-04-12T03:48:55.370 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a45225b78932abb-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.370 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:48:55.370 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:48:55.370 DEBUG:boto:path=/dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.370 DEBUG:boto:auth_path=/dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.370 DEBUG:boto:Method: GET 2024-04-12T03:48:55.370 DEBUG:boto:Path: /dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.370 DEBUG:boto:Data: 2024-04-12T03:48:55.371 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.371 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.371 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.371 DEBUG:boto:Params: {} 2024-04-12T03:48:55.371 DEBUG:boto:Token: None 2024-04-12T03:48:55.371 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj1?versionId=null 2024-04-12T03:48:55.371 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S6hwDuiXVLp51epGXIfZ/QOSgi4= 2024-04-12T03:48:55.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S6hwDuiXVLp51epGXIfZ/QOSgi4=', 'Content-Length': '0'} 2024-04-12T03:48:55.373 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cb7539f9cc96b5b-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.374 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.374 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.374 DEBUG:boto:path=/dkyges-51/?prefix=obj1 2024-04-12T03:48:55.374 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj1 2024-04-12T03:48:55.374 DEBUG:boto:Method: GET 2024-04-12T03:48:55.374 DEBUG:boto:Path: /dkyges-51/?prefix=obj1 2024-04-12T03:48:55.374 DEBUG:boto:Data: 2024-04-12T03:48:55.374 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.374 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.374 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.374 DEBUG:boto:Params: {} 2024-04-12T03:48:55.375 DEBUG:boto:Token: None 2024-04-12T03:48:55.375 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.375 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085b7d544b9480800-006618af27-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.378 DEBUG:boto:b'dkyges-51obj11000falseobj12024-04-12T03:47:34.473Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.379 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.379 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.379 DEBUG:boto:path=/dkyges-51/?prefix=obj1 2024-04-12T03:48:55.379 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj1 2024-04-12T03:48:55.379 DEBUG:boto:Method: GET 2024-04-12T03:48:55.379 DEBUG:boto:Path: /dkyges-51/?prefix=obj1 2024-04-12T03:48:55.379 DEBUG:boto:Data: 2024-04-12T03:48:55.379 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.379 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.379 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.380 DEBUG:boto:Params: {} 2024-04-12T03:48:55.380 DEBUG:boto:Token: None 2024-04-12T03:48:55.380 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.380 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083faab09ebfda029-006618af27-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.384 DEBUG:boto:b'dkyges-51obj11000falseobj12024-04-12T03:47:34.473Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.385 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:48:55.385 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:48:55.385 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.385 DEBUG:boto:Path: /dkyges-51/obj1 2024-04-12T03:48:55.385 DEBUG:boto:Data: 2024-04-12T03:48:55.386 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.386 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.386 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.386 DEBUG:boto:Params: {} 2024-04-12T03:48:55.386 DEBUG:boto:Token: None 2024-04-12T03:48:55.387 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj1 2024-04-12T03:48:55.387 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OLtVhIGVo8EuDBIsfzn4Y7rMto= 2024-04-12T03:48:55.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OLtVhIGVo8EuDBIsfzn4Y7rMto=', 'Content-Length': '0'} 2024-04-12T03:48:55.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a61a72a38c57ee8-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.390 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:48:55.390 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:48:55.390 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.390 DEBUG:boto:Path: /dkyges-51/obj1 2024-04-12T03:48:55.391 DEBUG:boto:Data: 2024-04-12T03:48:55.391 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.391 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.391 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.391 DEBUG:boto:Params: {} 2024-04-12T03:48:55.392 DEBUG:boto:Token: None 2024-04-12T03:48:55.392 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj1 2024-04-12T03:48:55.393 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OLtVhIGVo8EuDBIsfzn4Y7rMto= 2024-04-12T03:48:55.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OLtVhIGVo8EuDBIsfzn4Y7rMto=', 'Content-Length': '0'} 2024-04-12T03:48:55.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fe82bd4e78c0321-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2024-04-12T03:48:55.396 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:48:55.396 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:48:55.396 DEBUG:boto:Method: GET 2024-04-12T03:48:55.396 DEBUG:boto:Path: /dkyges-51/obj1 2024-04-12T03:48:55.397 DEBUG:boto:Data: 2024-04-12T03:48:55.397 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.397 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.397 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.397 DEBUG:boto:Params: {} 2024-04-12T03:48:55.397 DEBUG:boto:Token: None 2024-04-12T03:48:55.398 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj1 2024-04-12T03:48:55.398 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lHqipFGVnGdQTCqSlqAMsna+m6Q= 2024-04-12T03:48:55.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lHqipFGVnGdQTCqSlqAMsna+m6Q=', 'Content-Length': '0'} 2024-04-12T03:48:55.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030691e2ac8d36e62-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.401 DEBUG:boto:path=/dkyges-51/obj1 2024-04-12T03:48:55.401 DEBUG:boto:auth_path=/dkyges-51/obj1 2024-04-12T03:48:55.401 DEBUG:boto:Method: GET 2024-04-12T03:48:55.401 DEBUG:boto:Path: /dkyges-51/obj1 2024-04-12T03:48:55.401 DEBUG:boto:Data: 2024-04-12T03:48:55.401 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.401 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.402 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.402 DEBUG:boto:Params: {} 2024-04-12T03:48:55.402 DEBUG:boto:Token: None 2024-04-12T03:48:55.402 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj1 2024-04-12T03:48:55.402 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lHqipFGVnGdQTCqSlqAMsna+m6Q= 2024-04-12T03:48:55.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lHqipFGVnGdQTCqSlqAMsna+m6Q=', 'Content-Length': '0'} 2024-04-12T03:48:55.404 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000451545655aaa0ba6-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.446 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.447 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.447 DEBUG:boto:path=/dkyges-51/?prefix=obj1 2024-04-12T03:48:55.447 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj1 2024-04-12T03:48:55.448 DEBUG:boto:Method: GET 2024-04-12T03:48:55.448 DEBUG:boto:Path: /dkyges-51/?prefix=obj1 2024-04-12T03:48:55.448 DEBUG:boto:Data: 2024-04-12T03:48:55.448 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.448 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.448 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.448 DEBUG:boto:Params: {} 2024-04-12T03:48:55.448 DEBUG:boto:Token: None 2024-04-12T03:48:55.448 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.449 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c18a2b77a64b713-006618af27-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.451 DEBUG:boto:b'dkyges-51obj11000falseobj12024-04-12T03:47:34.473Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.452 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.452 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.452 DEBUG:boto:path=/dkyges-51/?prefix=obj1 2024-04-12T03:48:55.452 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj1 2024-04-12T03:48:55.452 DEBUG:boto:Method: GET 2024-04-12T03:48:55.452 DEBUG:boto:Path: /dkyges-51/?prefix=obj1 2024-04-12T03:48:55.452 DEBUG:boto:Data: 2024-04-12T03:48:55.452 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.453 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.453 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.453 DEBUG:boto:Params: {} 2024-04-12T03:48:55.453 DEBUG:boto:Token: None 2024-04-12T03:48:55.453 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.453 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4fff517cc6ae7c9-006618af27-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.456 DEBUG:boto:b'dkyges-51obj11000falseobj12024-04-12T03:47:34.473Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:48:55.457 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:48:55.458 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:48:55.458 DEBUG:boto:path=/dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.458 DEBUG:boto:auth_path=/dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.458 DEBUG:boto:Method: GET 2024-04-12T03:48:55.458 DEBUG:boto:Path: /dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.458 DEBUG:boto:Data: 2024-04-12T03:48:55.458 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.458 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.458 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.458 DEBUG:boto:Params: {} 2024-04-12T03:48:55.459 DEBUG:boto:Token: None 2024-04-12T03:48:55.459 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.459 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vSUt93QDPyNQMLT3A2dKfb2lC3U= 2024-04-12T03:48:55.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vSUt93QDPyNQMLT3A2dKfb2lC3U=', 'Content-Length': '0'} 2024-04-12T03:48:55.461 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008cc436f898db6943-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.502 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:48:55.503 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:48:55.503 DEBUG:boto:path=/dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.503 DEBUG:boto:auth_path=/dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.503 DEBUG:boto:Method: GET 2024-04-12T03:48:55.503 DEBUG:boto:Path: /dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.503 DEBUG:boto:Data: 2024-04-12T03:48:55.503 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.503 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.504 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.504 DEBUG:boto:Params: {} 2024-04-12T03:48:55.504 DEBUG:boto:Token: None 2024-04-12T03:48:55.504 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.504 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vSUt93QDPyNQMLT3A2dKfb2lC3U= 2024-04-12T03:48:55.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vSUt93QDPyNQMLT3A2dKfb2lC3U=', 'Content-Length': '0'} 2024-04-12T03:48:55.506 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bc72e07242124716-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.507 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:48:55.507 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:48:55.507 DEBUG:boto:path=/dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.507 DEBUG:boto:auth_path=/dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.507 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.507 DEBUG:boto:Path: /dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.507 DEBUG:boto:Data: 2024-04-12T03:48:55.507 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.507 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.507 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.507 DEBUG:boto:Params: {} 2024-04-12T03:48:55.508 DEBUG:boto:Token: None 2024-04-12T03:48:55.508 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.508 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rBBWARhMOHCXiJsY+YhNeYxm6xM= 2024-04-12T03:48:55.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rBBWARhMOHCXiJsY+YhNeYxm6xM=', 'Content-Length': '0'} 2024-04-12T03:48:55.509 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006517f40ffad26c90-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.510 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:48:55.510 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:48:55.510 DEBUG:boto:path=/dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.510 DEBUG:boto:auth_path=/dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.510 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.510 DEBUG:boto:Path: /dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.510 DEBUG:boto:Data: 2024-04-12T03:48:55.510 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.511 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.511 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.511 DEBUG:boto:Params: {} 2024-04-12T03:48:55.511 DEBUG:boto:Token: None 2024-04-12T03:48:55.511 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.511 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rBBWARhMOHCXiJsY+YhNeYxm6xM= 2024-04-12T03:48:55.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rBBWARhMOHCXiJsY+YhNeYxm6xM=', 'Content-Length': '0'} 2024-04-12T03:48:55.513 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de3a8777b0410520-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:48:55.514 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:48:55.514 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:48:55.514 DEBUG:boto:path=/dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.514 DEBUG:boto:auth_path=/dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.514 DEBUG:boto:Method: GET 2024-04-12T03:48:55.514 DEBUG:boto:Path: /dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.514 DEBUG:boto:Data: 2024-04-12T03:48:55.514 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.514 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.514 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.515 DEBUG:boto:Params: {} 2024-04-12T03:48:55.515 DEBUG:boto:Token: None 2024-04-12T03:48:55.517 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.517 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vSUt93QDPyNQMLT3A2dKfb2lC3U= 2024-04-12T03:48:55.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vSUt93QDPyNQMLT3A2dKfb2lC3U=', 'Content-Length': '0'} 2024-04-12T03:48:55.519 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000764cdb4b06708644-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.519 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:48:55.520 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:48:55.520 DEBUG:boto:path=/dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.520 DEBUG:boto:auth_path=/dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.520 DEBUG:boto:Method: GET 2024-04-12T03:48:55.520 DEBUG:boto:Path: /dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.520 DEBUG:boto:Data: 2024-04-12T03:48:55.520 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.520 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.520 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.520 DEBUG:boto:Params: {} 2024-04-12T03:48:55.520 DEBUG:boto:Token: None 2024-04-12T03:48:55.521 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj2?versionId=null 2024-04-12T03:48:55.521 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vSUt93QDPyNQMLT3A2dKfb2lC3U= 2024-04-12T03:48:55.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vSUt93QDPyNQMLT3A2dKfb2lC3U=', 'Content-Length': '0'} 2024-04-12T03:48:55.522 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008128b506aa68f767-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.523 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.523 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.523 DEBUG:boto:path=/dkyges-51/?prefix=obj2 2024-04-12T03:48:55.523 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj2 2024-04-12T03:48:55.523 DEBUG:boto:Method: GET 2024-04-12T03:48:55.523 DEBUG:boto:Path: /dkyges-51/?prefix=obj2 2024-04-12T03:48:55.523 DEBUG:boto:Data: 2024-04-12T03:48:55.523 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.524 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.524 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.524 DEBUG:boto:Params: {} 2024-04-12T03:48:55.524 DEBUG:boto:Token: None 2024-04-12T03:48:55.524 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.524 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002265cd9909a35fce-006618af27-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.527 DEBUG:boto:b'dkyges-51obj21000falseobj22024-04-12T03:47:34.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.527 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.527 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.527 DEBUG:boto:path=/dkyges-51/?prefix=obj2 2024-04-12T03:48:55.527 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj2 2024-04-12T03:48:55.528 DEBUG:boto:Method: GET 2024-04-12T03:48:55.528 DEBUG:boto:Path: /dkyges-51/?prefix=obj2 2024-04-12T03:48:55.528 DEBUG:boto:Data: 2024-04-12T03:48:55.528 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.528 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.528 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.528 DEBUG:boto:Params: {} 2024-04-12T03:48:55.528 DEBUG:boto:Token: None 2024-04-12T03:48:55.528 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.528 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcc72abc3c7442f0-006618af27-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.531 DEBUG:boto:b'dkyges-51obj21000falseobj22024-04-12T03:47:34.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.531 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:48:55.531 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:48:55.532 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.532 DEBUG:boto:Path: /dkyges-51/obj2 2024-04-12T03:48:55.532 DEBUG:boto:Data: 2024-04-12T03:48:55.532 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.532 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.532 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.532 DEBUG:boto:Params: {} 2024-04-12T03:48:55.532 DEBUG:boto:Token: None 2024-04-12T03:48:55.532 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj2 2024-04-12T03:48:55.532 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yrhLym/30j/h5etKExWoKuOIx+w= 2024-04-12T03:48:55.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yrhLym/30j/h5etKExWoKuOIx+w=', 'Content-Length': '0'} 2024-04-12T03:48:55.534 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b21374ccee676e1b-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.534 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:48:55.535 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:48:55.535 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.535 DEBUG:boto:Path: /dkyges-51/obj2 2024-04-12T03:48:55.535 DEBUG:boto:Data: 2024-04-12T03:48:55.535 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.535 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.535 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.535 DEBUG:boto:Params: {} 2024-04-12T03:48:55.535 DEBUG:boto:Token: None 2024-04-12T03:48:55.535 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj2 2024-04-12T03:48:55.535 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yrhLym/30j/h5etKExWoKuOIx+w= 2024-04-12T03:48:55.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yrhLym/30j/h5etKExWoKuOIx+w=', 'Content-Length': '0'} 2024-04-12T03:48:55.537 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9953c6f83135247-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2024-04-12T03:48:55.538 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:48:55.538 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:48:55.538 DEBUG:boto:Method: GET 2024-04-12T03:48:55.538 DEBUG:boto:Path: /dkyges-51/obj2 2024-04-12T03:48:55.538 DEBUG:boto:Data: 2024-04-12T03:48:55.538 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.538 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.538 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.538 DEBUG:boto:Params: {} 2024-04-12T03:48:55.538 DEBUG:boto:Token: None 2024-04-12T03:48:55.539 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj2 2024-04-12T03:48:55.539 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4UHGMsp62h83Z7eycj+Yj1/uRcE= 2024-04-12T03:48:55.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4UHGMsp62h83Z7eycj+Yj1/uRcE=', 'Content-Length': '0'} 2024-04-12T03:48:55.540 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6c8a9ae24d872af-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.582 DEBUG:boto:path=/dkyges-51/obj2 2024-04-12T03:48:55.583 DEBUG:boto:auth_path=/dkyges-51/obj2 2024-04-12T03:48:55.583 DEBUG:boto:Method: GET 2024-04-12T03:48:55.583 DEBUG:boto:Path: /dkyges-51/obj2 2024-04-12T03:48:55.583 DEBUG:boto:Data: 2024-04-12T03:48:55.583 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.583 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.583 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.583 DEBUG:boto:Params: {} 2024-04-12T03:48:55.583 DEBUG:boto:Token: None 2024-04-12T03:48:55.584 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj2 2024-04-12T03:48:55.584 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4UHGMsp62h83Z7eycj+Yj1/uRcE= 2024-04-12T03:48:55.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4UHGMsp62h83Z7eycj+Yj1/uRcE=', 'Content-Length': '0'} 2024-04-12T03:48:55.586 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d840aad41a1f4b96-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.587 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.587 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.587 DEBUG:boto:path=/dkyges-51/?prefix=obj2 2024-04-12T03:48:55.587 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj2 2024-04-12T03:48:55.587 DEBUG:boto:Method: GET 2024-04-12T03:48:55.587 DEBUG:boto:Path: /dkyges-51/?prefix=obj2 2024-04-12T03:48:55.587 DEBUG:boto:Data: 2024-04-12T03:48:55.587 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.587 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.588 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.588 DEBUG:boto:Params: {} 2024-04-12T03:48:55.588 DEBUG:boto:Token: None 2024-04-12T03:48:55.588 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.588 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a850ce83010b74b4-006618af27-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.591 DEBUG:boto:b'dkyges-51obj21000falseobj22024-04-12T03:47:34.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.591 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.591 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.591 DEBUG:boto:path=/dkyges-51/?prefix=obj2 2024-04-12T03:48:55.592 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj2 2024-04-12T03:48:55.592 DEBUG:boto:Method: GET 2024-04-12T03:48:55.592 DEBUG:boto:Path: /dkyges-51/?prefix=obj2 2024-04-12T03:48:55.592 DEBUG:boto:Data: 2024-04-12T03:48:55.592 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.592 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.592 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.592 DEBUG:boto:Params: {} 2024-04-12T03:48:55.592 DEBUG:boto:Token: None 2024-04-12T03:48:55.592 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.593 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e1f95097e65a3da-006618af27-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.595 DEBUG:boto:b'dkyges-51obj21000falseobj22024-04-12T03:47:34.490Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:48:55.596 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:48:55.596 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:48:55.596 DEBUG:boto:path=/dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.596 DEBUG:boto:auth_path=/dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.596 DEBUG:boto:Method: GET 2024-04-12T03:48:55.596 DEBUG:boto:Path: /dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.596 DEBUG:boto:Data: 2024-04-12T03:48:55.596 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.596 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.596 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.597 DEBUG:boto:Params: {} 2024-04-12T03:48:55.597 DEBUG:boto:Token: None 2024-04-12T03:48:55.597 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.597 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1MIgkbuipmtR+3ZsjQHuWelAn2c= 2024-04-12T03:48:55.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1MIgkbuipmtR+3ZsjQHuWelAn2c=', 'Content-Length': '0'} 2024-04-12T03:48:55.599 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ac0ce5345517043-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.599 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:48:55.599 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:48:55.600 DEBUG:boto:path=/dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.600 DEBUG:boto:auth_path=/dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.600 DEBUG:boto:Method: GET 2024-04-12T03:48:55.600 DEBUG:boto:Path: /dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.600 DEBUG:boto:Data: 2024-04-12T03:48:55.600 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.600 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.600 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.600 DEBUG:boto:Params: {} 2024-04-12T03:48:55.600 DEBUG:boto:Token: None 2024-04-12T03:48:55.601 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.601 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1MIgkbuipmtR+3ZsjQHuWelAn2c= 2024-04-12T03:48:55.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1MIgkbuipmtR+3ZsjQHuWelAn2c=', 'Content-Length': '0'} 2024-04-12T03:48:55.603 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e1e74522a53e5d5-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.603 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:48:55.603 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:48:55.603 DEBUG:boto:path=/dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.603 DEBUG:boto:auth_path=/dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.603 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.603 DEBUG:boto:Path: /dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.604 DEBUG:boto:Data: 2024-04-12T03:48:55.604 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.604 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.604 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.604 DEBUG:boto:Params: {} 2024-04-12T03:48:55.604 DEBUG:boto:Token: None 2024-04-12T03:48:55.604 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OdEpSFUUprnjV6fQZ7JkiGGb//E= 2024-04-12T03:48:55.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OdEpSFUUprnjV6fQZ7JkiGGb//E=', 'Content-Length': '0'} 2024-04-12T03:48:55.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099b2e0e076ca4b97-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.606 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:48:55.606 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:48:55.606 DEBUG:boto:path=/dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.606 DEBUG:boto:auth_path=/dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.606 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.607 DEBUG:boto:Path: /dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.607 DEBUG:boto:Data: 2024-04-12T03:48:55.607 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.607 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.607 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.607 DEBUG:boto:Params: {} 2024-04-12T03:48:55.607 DEBUG:boto:Token: None 2024-04-12T03:48:55.607 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.607 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OdEpSFUUprnjV6fQZ7JkiGGb//E= 2024-04-12T03:48:55.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OdEpSFUUprnjV6fQZ7JkiGGb//E=', 'Content-Length': '0'} 2024-04-12T03:48:55.609 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd606fa017033bee-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:48:55.609 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:48:55.610 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:48:55.610 DEBUG:boto:path=/dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.610 DEBUG:boto:auth_path=/dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.610 DEBUG:boto:Method: GET 2024-04-12T03:48:55.610 DEBUG:boto:Path: /dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.610 DEBUG:boto:Data: 2024-04-12T03:48:55.610 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.610 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.610 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.610 DEBUG:boto:Params: {} 2024-04-12T03:48:55.611 DEBUG:boto:Token: None 2024-04-12T03:48:55.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.611 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1MIgkbuipmtR+3ZsjQHuWelAn2c= 2024-04-12T03:48:55.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1MIgkbuipmtR+3ZsjQHuWelAn2c=', 'Content-Length': '0'} 2024-04-12T03:48:55.613 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002526bf5e11211493-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.654 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:48:55.655 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:48:55.655 DEBUG:boto:path=/dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.655 DEBUG:boto:auth_path=/dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.655 DEBUG:boto:Method: GET 2024-04-12T03:48:55.655 DEBUG:boto:Path: /dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.655 DEBUG:boto:Data: 2024-04-12T03:48:55.655 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.655 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.655 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.655 DEBUG:boto:Params: {} 2024-04-12T03:48:55.656 DEBUG:boto:Token: None 2024-04-12T03:48:55.656 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj3?versionId=null 2024-04-12T03:48:55.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1MIgkbuipmtR+3ZsjQHuWelAn2c= 2024-04-12T03:48:55.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1MIgkbuipmtR+3ZsjQHuWelAn2c=', 'Content-Length': '0'} 2024-04-12T03:48:55.658 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab2fee9082f1a40d-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.658 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.659 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.659 DEBUG:boto:path=/dkyges-51/?prefix=obj3 2024-04-12T03:48:55.659 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj3 2024-04-12T03:48:55.659 DEBUG:boto:Method: GET 2024-04-12T03:48:55.659 DEBUG:boto:Path: /dkyges-51/?prefix=obj3 2024-04-12T03:48:55.659 DEBUG:boto:Data: 2024-04-12T03:48:55.659 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.659 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.659 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.659 DEBUG:boto:Params: {} 2024-04-12T03:48:55.659 DEBUG:boto:Token: None 2024-04-12T03:48:55.660 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.660 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e16c4ccd949f1543-006618af27-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.663 DEBUG:boto:b'dkyges-51obj31000falseobj32024-04-12T03:47:34.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.663 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.663 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.663 DEBUG:boto:path=/dkyges-51/?prefix=obj3 2024-04-12T03:48:55.663 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj3 2024-04-12T03:48:55.663 DEBUG:boto:Method: GET 2024-04-12T03:48:55.664 DEBUG:boto:Path: /dkyges-51/?prefix=obj3 2024-04-12T03:48:55.664 DEBUG:boto:Data: 2024-04-12T03:48:55.664 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.664 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.664 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.664 DEBUG:boto:Params: {} 2024-04-12T03:48:55.664 DEBUG:boto:Token: None 2024-04-12T03:48:55.664 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.664 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bb6ec449dca0636-006618af27-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.667 DEBUG:boto:b'dkyges-51obj31000falseobj32024-04-12T03:47:34.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.668 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:48:55.668 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:48:55.668 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.668 DEBUG:boto:Path: /dkyges-51/obj3 2024-04-12T03:48:55.668 DEBUG:boto:Data: 2024-04-12T03:48:55.668 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.668 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.668 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.668 DEBUG:boto:Params: {} 2024-04-12T03:48:55.668 DEBUG:boto:Token: None 2024-04-12T03:48:55.669 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj3 2024-04-12T03:48:55.669 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2AlEx5aD1gru0cvEwsDC4knDlBg= 2024-04-12T03:48:55.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2AlEx5aD1gru0cvEwsDC4knDlBg=', 'Content-Length': '0'} 2024-04-12T03:48:55.670 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c8e0ddb5228a912-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.671 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:48:55.671 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:48:55.671 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.671 DEBUG:boto:Path: /dkyges-51/obj3 2024-04-12T03:48:55.671 DEBUG:boto:Data: 2024-04-12T03:48:55.671 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.671 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.671 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.671 DEBUG:boto:Params: {} 2024-04-12T03:48:55.671 DEBUG:boto:Token: None 2024-04-12T03:48:55.672 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj3 2024-04-12T03:48:55.672 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2AlEx5aD1gru0cvEwsDC4knDlBg= 2024-04-12T03:48:55.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2AlEx5aD1gru0cvEwsDC4knDlBg=', 'Content-Length': '0'} 2024-04-12T03:48:55.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045e50f4deb0ca4d6-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2024-04-12T03:48:55.674 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:48:55.674 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:48:55.674 DEBUG:boto:Method: GET 2024-04-12T03:48:55.674 DEBUG:boto:Path: /dkyges-51/obj3 2024-04-12T03:48:55.674 DEBUG:boto:Data: 2024-04-12T03:48:55.674 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.674 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.675 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.675 DEBUG:boto:Params: {} 2024-04-12T03:48:55.675 DEBUG:boto:Token: None 2024-04-12T03:48:55.675 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj3 2024-04-12T03:48:55.675 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JdXIShmbpZP3K/yRVq7WqFiCG48= 2024-04-12T03:48:55.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JdXIShmbpZP3K/yRVq7WqFiCG48=', 'Content-Length': '0'} 2024-04-12T03:48:55.677 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea8bf90b6ceaa28b-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.722 DEBUG:boto:path=/dkyges-51/obj3 2024-04-12T03:48:55.723 DEBUG:boto:auth_path=/dkyges-51/obj3 2024-04-12T03:48:55.723 DEBUG:boto:Method: GET 2024-04-12T03:48:55.723 DEBUG:boto:Path: /dkyges-51/obj3 2024-04-12T03:48:55.723 DEBUG:boto:Data: 2024-04-12T03:48:55.723 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.723 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.723 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.723 DEBUG:boto:Params: {} 2024-04-12T03:48:55.723 DEBUG:boto:Token: None 2024-04-12T03:48:55.724 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj3 2024-04-12T03:48:55.724 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JdXIShmbpZP3K/yRVq7WqFiCG48= 2024-04-12T03:48:55.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JdXIShmbpZP3K/yRVq7WqFiCG48=', 'Content-Length': '0'} 2024-04-12T03:48:55.726 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e05afc45e96834da-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.726 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.726 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.726 DEBUG:boto:path=/dkyges-51/?prefix=obj3 2024-04-12T03:48:55.726 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj3 2024-04-12T03:48:55.727 DEBUG:boto:Method: GET 2024-04-12T03:48:55.727 DEBUG:boto:Path: /dkyges-51/?prefix=obj3 2024-04-12T03:48:55.727 DEBUG:boto:Data: 2024-04-12T03:48:55.727 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.727 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.727 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.727 DEBUG:boto:Params: {} 2024-04-12T03:48:55.727 DEBUG:boto:Token: None 2024-04-12T03:48:55.727 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.727 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec2f22cb6471f5b9-006618af27-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.730 DEBUG:boto:b'dkyges-51obj31000falseobj32024-04-12T03:47:34.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.731 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.731 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.731 DEBUG:boto:path=/dkyges-51/?prefix=obj3 2024-04-12T03:48:55.731 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj3 2024-04-12T03:48:55.731 DEBUG:boto:Method: GET 2024-04-12T03:48:55.731 DEBUG:boto:Path: /dkyges-51/?prefix=obj3 2024-04-12T03:48:55.731 DEBUG:boto:Data: 2024-04-12T03:48:55.731 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.731 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.732 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.732 DEBUG:boto:Params: {} 2024-04-12T03:48:55.732 DEBUG:boto:Token: None 2024-04-12T03:48:55.732 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.732 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9dd19603de937c4-006618af27-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.735 DEBUG:boto:b'dkyges-51obj31000falseobj32024-04-12T03:47:34.541Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:48:55.735 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:48:55.735 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:48:55.735 DEBUG:boto:path=/dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.735 DEBUG:boto:auth_path=/dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.735 DEBUG:boto:Method: GET 2024-04-12T03:48:55.736 DEBUG:boto:Path: /dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.736 DEBUG:boto:Data: 2024-04-12T03:48:55.736 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.736 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.736 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.736 DEBUG:boto:Params: {} 2024-04-12T03:48:55.736 DEBUG:boto:Token: None 2024-04-12T03:48:55.736 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.736 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vlePlcnalLZzUzTs/Q91zurgVYo= 2024-04-12T03:48:55.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vlePlcnalLZzUzTs/Q91zurgVYo=', 'Content-Length': '0'} 2024-04-12T03:48:55.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000163cecea1ad89219-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.738 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:48:55.738 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:48:55.739 DEBUG:boto:path=/dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.739 DEBUG:boto:auth_path=/dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.739 DEBUG:boto:Method: GET 2024-04-12T03:48:55.739 DEBUG:boto:Path: /dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.739 DEBUG:boto:Data: 2024-04-12T03:48:55.739 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.739 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.739 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.739 DEBUG:boto:Params: {} 2024-04-12T03:48:55.739 DEBUG:boto:Token: None 2024-04-12T03:48:55.740 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.740 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vlePlcnalLZzUzTs/Q91zurgVYo= 2024-04-12T03:48:55.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vlePlcnalLZzUzTs/Q91zurgVYo=', 'Content-Length': '0'} 2024-04-12T03:48:55.742 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007401e9f403473bc4-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.742 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:48:55.742 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:48:55.742 DEBUG:boto:path=/dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.742 DEBUG:boto:auth_path=/dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.742 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.742 DEBUG:boto:Path: /dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.743 DEBUG:boto:Data: 2024-04-12T03:48:55.743 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.743 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.743 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.743 DEBUG:boto:Params: {} 2024-04-12T03:48:55.743 DEBUG:boto:Token: None 2024-04-12T03:48:55.743 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.743 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hyiyjwDCfGXaO4bopBxyKAsiCFY= 2024-04-12T03:48:55.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hyiyjwDCfGXaO4bopBxyKAsiCFY=', 'Content-Length': '0'} 2024-04-12T03:48:55.745 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a16146276d20108-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.745 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:48:55.745 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:48:55.745 DEBUG:boto:path=/dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.745 DEBUG:boto:auth_path=/dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.746 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.746 DEBUG:boto:Path: /dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.746 DEBUG:boto:Data: 2024-04-12T03:48:55.746 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.746 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.746 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.746 DEBUG:boto:Params: {} 2024-04-12T03:48:55.746 DEBUG:boto:Token: None 2024-04-12T03:48:55.746 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.746 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hyiyjwDCfGXaO4bopBxyKAsiCFY= 2024-04-12T03:48:55.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hyiyjwDCfGXaO4bopBxyKAsiCFY=', 'Content-Length': '0'} 2024-04-12T03:48:55.748 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfc769e23ad6f900-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:48:55.748 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:48:55.749 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:48:55.749 DEBUG:boto:path=/dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.749 DEBUG:boto:auth_path=/dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.749 DEBUG:boto:Method: GET 2024-04-12T03:48:55.749 DEBUG:boto:Path: /dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.749 DEBUG:boto:Data: 2024-04-12T03:48:55.749 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.749 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.749 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.749 DEBUG:boto:Params: {} 2024-04-12T03:48:55.749 DEBUG:boto:Token: None 2024-04-12T03:48:55.750 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.750 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vlePlcnalLZzUzTs/Q91zurgVYo= 2024-04-12T03:48:55.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vlePlcnalLZzUzTs/Q91zurgVYo=', 'Content-Length': '0'} 2024-04-12T03:48:55.752 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a567e8457be76b71-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.794 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:48:55.795 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:48:55.796 DEBUG:boto:path=/dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.796 DEBUG:boto:auth_path=/dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.796 DEBUG:boto:Method: GET 2024-04-12T03:48:55.796 DEBUG:boto:Path: /dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.796 DEBUG:boto:Data: 2024-04-12T03:48:55.796 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.796 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.796 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.797 DEBUG:boto:Params: {} 2024-04-12T03:48:55.797 DEBUG:boto:Token: None 2024-04-12T03:48:55.797 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj4?versionId=null 2024-04-12T03:48:55.797 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vlePlcnalLZzUzTs/Q91zurgVYo= 2024-04-12T03:48:55.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vlePlcnalLZzUzTs/Q91zurgVYo=', 'Content-Length': '0'} 2024-04-12T03:48:55.799 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000faff1fe19b13b0ec-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.799 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.800 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.800 DEBUG:boto:path=/dkyges-51/?prefix=obj4 2024-04-12T03:48:55.800 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj4 2024-04-12T03:48:55.800 DEBUG:boto:Method: GET 2024-04-12T03:48:55.800 DEBUG:boto:Path: /dkyges-51/?prefix=obj4 2024-04-12T03:48:55.800 DEBUG:boto:Data: 2024-04-12T03:48:55.800 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.800 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.800 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.800 DEBUG:boto:Params: {} 2024-04-12T03:48:55.800 DEBUG:boto:Token: None 2024-04-12T03:48:55.801 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.801 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b994afa0badcf316-006618af27-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.804 DEBUG:boto:b'dkyges-51obj41000falseobj42024-04-12T03:47:34.552Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.804 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.805 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.805 DEBUG:boto:path=/dkyges-51/?prefix=obj4 2024-04-12T03:48:55.805 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj4 2024-04-12T03:48:55.805 DEBUG:boto:Method: GET 2024-04-12T03:48:55.805 DEBUG:boto:Path: /dkyges-51/?prefix=obj4 2024-04-12T03:48:55.806 DEBUG:boto:Data: 2024-04-12T03:48:55.806 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.806 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.806 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.806 DEBUG:boto:Params: {} 2024-04-12T03:48:55.807 DEBUG:boto:Token: None 2024-04-12T03:48:55.807 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.807 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6559f15052b89e1-006618af27-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.810 DEBUG:boto:b'dkyges-51obj41000falseobj42024-04-12T03:47:34.552Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.810 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:48:55.810 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:48:55.810 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.811 DEBUG:boto:Path: /dkyges-51/obj4 2024-04-12T03:48:55.811 DEBUG:boto:Data: 2024-04-12T03:48:55.811 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.811 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.811 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.811 DEBUG:boto:Params: {} 2024-04-12T03:48:55.811 DEBUG:boto:Token: None 2024-04-12T03:48:55.812 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj4 2024-04-12T03:48:55.812 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:44ItNa4wRgNMJ6u9ZyKYKD8Fz1o= 2024-04-12T03:48:55.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:44ItNa4wRgNMJ6u9ZyKYKD8Fz1o=', 'Content-Length': '0'} 2024-04-12T03:48:55.814 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3b1f6ba02b64bde-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.814 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:48:55.814 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:48:55.814 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.814 DEBUG:boto:Path: /dkyges-51/obj4 2024-04-12T03:48:55.814 DEBUG:boto:Data: 2024-04-12T03:48:55.814 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.814 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.815 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.815 DEBUG:boto:Params: {} 2024-04-12T03:48:55.815 DEBUG:boto:Token: None 2024-04-12T03:48:55.815 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj4 2024-04-12T03:48:55.816 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:44ItNa4wRgNMJ6u9ZyKYKD8Fz1o= 2024-04-12T03:48:55.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:44ItNa4wRgNMJ6u9ZyKYKD8Fz1o=', 'Content-Length': '0'} 2024-04-12T03:48:55.818 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003325f4cd353ea3aa-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2024-04-12T03:48:55.819 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:48:55.819 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:48:55.820 DEBUG:boto:Method: GET 2024-04-12T03:48:55.820 DEBUG:boto:Path: /dkyges-51/obj4 2024-04-12T03:48:55.820 DEBUG:boto:Data: 2024-04-12T03:48:55.820 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.820 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.821 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.821 DEBUG:boto:Params: {} 2024-04-12T03:48:55.821 DEBUG:boto:Token: None 2024-04-12T03:48:55.821 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj4 2024-04-12T03:48:55.822 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:N5Ig0BSWoqTb+y1hthv617XzS7k= 2024-04-12T03:48:55.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:N5Ig0BSWoqTb+y1hthv617XzS7k=', 'Content-Length': '0'} 2024-04-12T03:48:55.824 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5cf71df753fb4e8-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.824 DEBUG:boto:path=/dkyges-51/obj4 2024-04-12T03:48:55.824 DEBUG:boto:auth_path=/dkyges-51/obj4 2024-04-12T03:48:55.824 DEBUG:boto:Method: GET 2024-04-12T03:48:55.825 DEBUG:boto:Path: /dkyges-51/obj4 2024-04-12T03:48:55.825 DEBUG:boto:Data: 2024-04-12T03:48:55.825 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.825 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.825 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.825 DEBUG:boto:Params: {} 2024-04-12T03:48:55.825 DEBUG:boto:Token: None 2024-04-12T03:48:55.825 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj4 2024-04-12T03:48:55.825 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:N5Ig0BSWoqTb+y1hthv617XzS7k= 2024-04-12T03:48:55.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:N5Ig0BSWoqTb+y1hthv617XzS7k=', 'Content-Length': '0'} 2024-04-12T03:48:55.827 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:47:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001939aacbcd121e7e-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.870 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.871 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.871 DEBUG:boto:path=/dkyges-51/?prefix=obj4 2024-04-12T03:48:55.871 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj4 2024-04-12T03:48:55.871 DEBUG:boto:Method: GET 2024-04-12T03:48:55.871 DEBUG:boto:Path: /dkyges-51/?prefix=obj4 2024-04-12T03:48:55.872 DEBUG:boto:Data: 2024-04-12T03:48:55.872 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.872 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.872 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.872 DEBUG:boto:Params: {} 2024-04-12T03:48:55.872 DEBUG:boto:Token: None 2024-04-12T03:48:55.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.873 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001036e230459c85cd-006618af27-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.876 DEBUG:boto:b'dkyges-51obj41000falseobj42024-04-12T03:47:34.552Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.876 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.877 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.877 DEBUG:boto:path=/dkyges-51/?prefix=obj4 2024-04-12T03:48:55.877 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj4 2024-04-12T03:48:55.877 DEBUG:boto:Method: GET 2024-04-12T03:48:55.877 DEBUG:boto:Path: /dkyges-51/?prefix=obj4 2024-04-12T03:48:55.877 DEBUG:boto:Data: 2024-04-12T03:48:55.877 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.877 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.877 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.877 DEBUG:boto:Params: {} 2024-04-12T03:48:55.877 DEBUG:boto:Token: None 2024-04-12T03:48:55.878 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.878 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000771e3e13e883f035-006618af27-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.881 DEBUG:boto:b'dkyges-51obj41000falseobj42024-04-12T03:47:34.552Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:48:55.881 DEBUG:boto:path=/dkyges-51/obj5 2024-04-12T03:48:55.881 DEBUG:boto:auth_path=/dkyges-51/obj5 2024-04-12T03:48:55.882 DEBUG:boto:path=/dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.882 DEBUG:boto:auth_path=/dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.882 DEBUG:boto:Method: GET 2024-04-12T03:48:55.882 DEBUG:boto:Path: /dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.882 DEBUG:boto:Data: 2024-04-12T03:48:55.882 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.882 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.882 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.882 DEBUG:boto:Params: {} 2024-04-12T03:48:55.882 DEBUG:boto:Token: None 2024-04-12T03:48:55.883 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.883 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q+H/jd72ULOYk/acSQVNL01Hdcw= 2024-04-12T03:48:55.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q+H/jd72ULOYk/acSQVNL01Hdcw=', 'Content-Length': '0'} 2024-04-12T03:48:55.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ef0c083041affd7-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.926 DEBUG:boto:path=/dkyges-51/obj5 2024-04-12T03:48:55.929 DEBUG:boto:auth_path=/dkyges-51/obj5 2024-04-12T03:48:55.930 DEBUG:boto:path=/dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.930 DEBUG:boto:auth_path=/dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.930 DEBUG:boto:Method: GET 2024-04-12T03:48:55.930 DEBUG:boto:Path: /dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.930 DEBUG:boto:Data: 2024-04-12T03:48:55.930 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.930 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.930 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.930 DEBUG:boto:Params: {} 2024-04-12T03:48:55.931 DEBUG:boto:Token: None 2024-04-12T03:48:55.931 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.931 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q+H/jd72ULOYk/acSQVNL01Hdcw= 2024-04-12T03:48:55.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q+H/jd72ULOYk/acSQVNL01Hdcw=', 'Content-Length': '0'} 2024-04-12T03:48:55.933 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d629cd134c09ae4d-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.934 DEBUG:boto:path=/dkyges-51/obj5 2024-04-12T03:48:55.934 DEBUG:boto:auth_path=/dkyges-51/obj5 2024-04-12T03:48:55.934 DEBUG:boto:path=/dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.934 DEBUG:boto:auth_path=/dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.934 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.934 DEBUG:boto:Path: /dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.934 DEBUG:boto:Data: 2024-04-12T03:48:55.935 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.935 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.935 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.935 DEBUG:boto:Params: {} 2024-04-12T03:48:55.935 DEBUG:boto:Token: None 2024-04-12T03:48:55.935 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.935 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:x2KhMZFtMmBgDAfkRph8x1coGW4= 2024-04-12T03:48:55.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:x2KhMZFtMmBgDAfkRph8x1coGW4=', 'Content-Length': '0'} 2024-04-12T03:48:55.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b2f899a3f27e1a2c-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.937 DEBUG:boto:path=/dkyges-51/obj5 2024-04-12T03:48:55.937 DEBUG:boto:auth_path=/dkyges-51/obj5 2024-04-12T03:48:55.937 DEBUG:boto:path=/dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.937 DEBUG:boto:auth_path=/dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.938 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.938 DEBUG:boto:Path: /dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.938 DEBUG:boto:Data: 2024-04-12T03:48:55.938 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.938 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.938 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.938 DEBUG:boto:Params: {} 2024-04-12T03:48:55.938 DEBUG:boto:Token: None 2024-04-12T03:48:55.939 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.939 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:x2KhMZFtMmBgDAfkRph8x1coGW4= 2024-04-12T03:48:55.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:x2KhMZFtMmBgDAfkRph8x1coGW4=', 'Content-Length': '0'} 2024-04-12T03:48:55.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5bdedffd9b3e5ac-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:48:55.941 DEBUG:boto:path=/dkyges-51/obj5 2024-04-12T03:48:55.941 DEBUG:boto:auth_path=/dkyges-51/obj5 2024-04-12T03:48:55.941 DEBUG:boto:path=/dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.941 DEBUG:boto:auth_path=/dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.942 DEBUG:boto:Method: GET 2024-04-12T03:48:55.942 DEBUG:boto:Path: /dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.942 DEBUG:boto:Data: 2024-04-12T03:48:55.942 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.942 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.942 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.942 DEBUG:boto:Params: {} 2024-04-12T03:48:55.942 DEBUG:boto:Token: None 2024-04-12T03:48:55.942 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.942 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q+H/jd72ULOYk/acSQVNL01Hdcw= 2024-04-12T03:48:55.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q+H/jd72ULOYk/acSQVNL01Hdcw=', 'Content-Length': '0'} 2024-04-12T03:48:55.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f09f2a8e00ccbfff-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.945 DEBUG:boto:path=/dkyges-51/obj5 2024-04-12T03:48:55.945 DEBUG:boto:auth_path=/dkyges-51/obj5 2024-04-12T03:48:55.945 DEBUG:boto:path=/dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.945 DEBUG:boto:auth_path=/dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.945 DEBUG:boto:Method: GET 2024-04-12T03:48:55.945 DEBUG:boto:Path: /dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.945 DEBUG:boto:Data: 2024-04-12T03:48:55.945 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.945 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.945 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.946 DEBUG:boto:Params: {} 2024-04-12T03:48:55.946 DEBUG:boto:Token: None 2024-04-12T03:48:55.946 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj5?versionId=null 2024-04-12T03:48:55.946 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q+H/jd72ULOYk/acSQVNL01Hdcw= 2024-04-12T03:48:55.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q+H/jd72ULOYk/acSQVNL01Hdcw=', 'Content-Length': '0'} 2024-04-12T03:48:55.948 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a653b7fb6b5b98ff-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.948 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.948 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.948 DEBUG:boto:path=/dkyges-51/?prefix=obj5 2024-04-12T03:48:55.949 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj5 2024-04-12T03:48:55.949 DEBUG:boto:Method: GET 2024-04-12T03:48:55.949 DEBUG:boto:Path: /dkyges-51/?prefix=obj5 2024-04-12T03:48:55.949 DEBUG:boto:Data: 2024-04-12T03:48:55.949 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.949 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.949 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.949 DEBUG:boto:Params: {} 2024-04-12T03:48:55.949 DEBUG:boto:Token: None 2024-04-12T03:48:55.949 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.949 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a5540b6bcc4c3bf-006618af27-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.952 DEBUG:boto:b'dkyges-51obj51000falseobj52024-04-12T03:48:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.953 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.953 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.953 DEBUG:boto:path=/dkyges-51/?prefix=obj5 2024-04-12T03:48:55.953 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj5 2024-04-12T03:48:55.953 DEBUG:boto:Method: GET 2024-04-12T03:48:55.953 DEBUG:boto:Path: /dkyges-51/?prefix=obj5 2024-04-12T03:48:55.953 DEBUG:boto:Data: 2024-04-12T03:48:55.953 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.953 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.953 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.954 DEBUG:boto:Params: {} 2024-04-12T03:48:55.954 DEBUG:boto:Token: None 2024-04-12T03:48:55.954 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.954 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b074ba7abd2d5918-006618af27-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.957 DEBUG:boto:b'dkyges-51obj51000falseobj52024-04-12T03:48:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:55.957 DEBUG:boto:path=/dkyges-51/obj5 2024-04-12T03:48:55.957 DEBUG:boto:auth_path=/dkyges-51/obj5 2024-04-12T03:48:55.957 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.957 DEBUG:boto:Path: /dkyges-51/obj5 2024-04-12T03:48:55.957 DEBUG:boto:Data: 2024-04-12T03:48:55.957 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.957 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.958 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.958 DEBUG:boto:Params: {} 2024-04-12T03:48:55.958 DEBUG:boto:Token: None 2024-04-12T03:48:55.958 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj5 2024-04-12T03:48:55.958 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zUfrrHMO9Ee+kfINTfgYsQMK+sQ= 2024-04-12T03:48:55.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zUfrrHMO9Ee+kfINTfgYsQMK+sQ=', 'Content-Length': '0'} 2024-04-12T03:48:55.960 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093cf193833c796fd-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.960 DEBUG:boto:path=/dkyges-51/obj5 2024-04-12T03:48:55.961 DEBUG:boto:auth_path=/dkyges-51/obj5 2024-04-12T03:48:55.961 DEBUG:boto:Method: HEAD 2024-04-12T03:48:55.961 DEBUG:boto:Path: /dkyges-51/obj5 2024-04-12T03:48:55.961 DEBUG:boto:Data: 2024-04-12T03:48:55.961 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.961 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.961 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.961 DEBUG:boto:Params: {} 2024-04-12T03:48:55.961 DEBUG:boto:Token: None 2024-04-12T03:48:55.962 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj5 2024-04-12T03:48:55.962 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zUfrrHMO9Ee+kfINTfgYsQMK+sQ= 2024-04-12T03:48:55.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zUfrrHMO9Ee+kfINTfgYsQMK+sQ=', 'Content-Length': '0'} 2024-04-12T03:48:55.964 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000851be0cebe1afb44-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2024-04-12T03:48:55.964 DEBUG:boto:path=/dkyges-51/obj5 2024-04-12T03:48:55.964 DEBUG:boto:auth_path=/dkyges-51/obj5 2024-04-12T03:48:55.964 DEBUG:boto:Method: GET 2024-04-12T03:48:55.964 DEBUG:boto:Path: /dkyges-51/obj5 2024-04-12T03:48:55.965 DEBUG:boto:Data: 2024-04-12T03:48:55.965 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.965 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.965 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.965 DEBUG:boto:Params: {} 2024-04-12T03:48:55.965 DEBUG:boto:Token: None 2024-04-12T03:48:55.965 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj5 2024-04-12T03:48:55.965 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ovVaTOQLX/rPc2vq8SYcfxHWFlQ= 2024-04-12T03:48:55.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ovVaTOQLX/rPc2vq8SYcfxHWFlQ=', 'Content-Length': '0'} 2024-04-12T03:48:55.967 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f08477edcfad3da4-006618af27-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.967 DEBUG:boto:path=/dkyges-51/obj5 2024-04-12T03:48:55.967 DEBUG:boto:auth_path=/dkyges-51/obj5 2024-04-12T03:48:55.967 DEBUG:boto:Method: GET 2024-04-12T03:48:55.967 DEBUG:boto:Path: /dkyges-51/obj5 2024-04-12T03:48:55.968 DEBUG:boto:Data: 2024-04-12T03:48:55.968 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.968 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:55.968 DEBUG:boto:Port: 8000 2024-04-12T03:48:55.968 DEBUG:boto:Params: {} 2024-04-12T03:48:55.968 DEBUG:boto:Token: None 2024-04-12T03:48:55.968 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/obj5 2024-04-12T03:48:55.968 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ovVaTOQLX/rPc2vq8SYcfxHWFlQ= 2024-04-12T03:48:55.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ovVaTOQLX/rPc2vq8SYcfxHWFlQ=', 'Content-Length': '0'} 2024-04-12T03:48:55.970 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5bcab82345074e4-006618af27-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:55.970 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:55.970 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:55.971 DEBUG:boto:path=/dkyges-51/?prefix=obj5 2024-04-12T03:48:55.971 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj5 2024-04-12T03:48:55.971 DEBUG:boto:Method: GET 2024-04-12T03:48:55.971 DEBUG:boto:Path: /dkyges-51/?prefix=obj5 2024-04-12T03:48:55.971 DEBUG:boto:Data: 2024-04-12T03:48:55.971 DEBUG:boto:Headers: {} 2024-04-12T03:48:55.971 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:55.971 DEBUG:boto:Port: 8001 2024-04-12T03:48:55.971 DEBUG:boto:Params: {} 2024-04-12T03:48:55.971 DEBUG:boto:Token: None 2024-04-12T03:48:55.971 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:55 GMT /dkyges-51/ 2024-04-12T03:48:55.972 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU= 2024-04-12T03:48:55.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zBu14JIUB75OY0EojWm6sZba4uU=', 'Content-Length': '0'} 2024-04-12T03:48:55.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044dbcceee16895aa-006618af27-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.018 DEBUG:boto:b'dkyges-51obj51000falseobj52024-04-12T03:48:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:56.019 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:56.019 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:56.019 DEBUG:boto:path=/dkyges-51/?prefix=obj5 2024-04-12T03:48:56.020 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj5 2024-04-12T03:48:56.020 DEBUG:boto:Method: GET 2024-04-12T03:48:56.020 DEBUG:boto:Path: /dkyges-51/?prefix=obj5 2024-04-12T03:48:56.020 DEBUG:boto:Data: 2024-04-12T03:48:56.020 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.020 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.020 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.020 DEBUG:boto:Params: {} 2024-04-12T03:48:56.021 DEBUG:boto:Token: None 2024-04-12T03:48:56.021 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/ 2024-04-12T03:48:56.021 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ= 2024-04-12T03:48:56.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a389b93cbfb7a64-006618af28-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.024 DEBUG:boto:b'dkyges-51obj51000falseobj52024-04-12T03:48:17.476Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:56.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:48:56.025 DEBUG:boto:path=/dkyges-51/obj6 2024-04-12T03:48:56.025 DEBUG:boto:auth_path=/dkyges-51/obj6 2024-04-12T03:48:56.025 DEBUG:boto:path=/dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.025 DEBUG:boto:auth_path=/dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.025 DEBUG:boto:Method: GET 2024-04-12T03:48:56.026 DEBUG:boto:Path: /dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.026 DEBUG:boto:Data: 2024-04-12T03:48:56.026 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.026 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.027 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.027 DEBUG:boto:Params: {} 2024-04-12T03:48:56.027 DEBUG:boto:Token: None 2024-04-12T03:48:56.027 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.027 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4nVnBcc4JnAQWmGFT5XxVXl1EkQ= 2024-04-12T03:48:56.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4nVnBcc4JnAQWmGFT5XxVXl1EkQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.030 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f58c02bb1fe2490e-006618af28-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.030 DEBUG:boto:path=/dkyges-51/obj6 2024-04-12T03:48:56.030 DEBUG:boto:auth_path=/dkyges-51/obj6 2024-04-12T03:48:56.030 DEBUG:boto:path=/dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.030 DEBUG:boto:auth_path=/dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.030 DEBUG:boto:Method: GET 2024-04-12T03:48:56.031 DEBUG:boto:Path: /dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.031 DEBUG:boto:Data: 2024-04-12T03:48:56.031 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.031 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.031 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.031 DEBUG:boto:Params: {} 2024-04-12T03:48:56.031 DEBUG:boto:Token: None 2024-04-12T03:48:56.031 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.031 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4nVnBcc4JnAQWmGFT5XxVXl1EkQ= 2024-04-12T03:48:56.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4nVnBcc4JnAQWmGFT5XxVXl1EkQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.033 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000015386fd6838b6eeb-006618af28-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.034 DEBUG:boto:path=/dkyges-51/obj6 2024-04-12T03:48:56.034 DEBUG:boto:auth_path=/dkyges-51/obj6 2024-04-12T03:48:56.034 DEBUG:boto:path=/dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.034 DEBUG:boto:auth_path=/dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.034 DEBUG:boto:Method: HEAD 2024-04-12T03:48:56.034 DEBUG:boto:Path: /dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.034 DEBUG:boto:Data: 2024-04-12T03:48:56.034 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.034 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.034 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.035 DEBUG:boto:Params: {} 2024-04-12T03:48:56.035 DEBUG:boto:Token: None 2024-04-12T03:48:56.035 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.035 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PgaSlLpPp0Kz68gzhGlECrtOqGE= 2024-04-12T03:48:56.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PgaSlLpPp0Kz68gzhGlECrtOqGE=', 'Content-Length': '0'} 2024-04-12T03:48:56.037 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc6c6d7933604435-006618af28-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.037 DEBUG:boto:path=/dkyges-51/obj6 2024-04-12T03:48:56.037 DEBUG:boto:auth_path=/dkyges-51/obj6 2024-04-12T03:48:56.037 DEBUG:boto:path=/dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.038 DEBUG:boto:auth_path=/dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.038 DEBUG:boto:Method: HEAD 2024-04-12T03:48:56.038 DEBUG:boto:Path: /dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.038 DEBUG:boto:Data: 2024-04-12T03:48:56.038 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.038 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.038 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.038 DEBUG:boto:Params: {} 2024-04-12T03:48:56.038 DEBUG:boto:Token: None 2024-04-12T03:48:56.039 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.039 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PgaSlLpPp0Kz68gzhGlECrtOqGE= 2024-04-12T03:48:56.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PgaSlLpPp0Kz68gzhGlECrtOqGE=', 'Content-Length': '0'} 2024-04-12T03:48:56.040 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007ba9ff61bcd4bf6b-006618af28-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:48:56.041 DEBUG:boto:path=/dkyges-51/obj6 2024-04-12T03:48:56.041 DEBUG:boto:auth_path=/dkyges-51/obj6 2024-04-12T03:48:56.041 DEBUG:boto:path=/dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.041 DEBUG:boto:auth_path=/dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.041 DEBUG:boto:Method: GET 2024-04-12T03:48:56.041 DEBUG:boto:Path: /dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.041 DEBUG:boto:Data: 2024-04-12T03:48:56.041 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.041 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.042 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.042 DEBUG:boto:Params: {} 2024-04-12T03:48:56.042 DEBUG:boto:Token: None 2024-04-12T03:48:56.042 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.042 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4nVnBcc4JnAQWmGFT5XxVXl1EkQ= 2024-04-12T03:48:56.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4nVnBcc4JnAQWmGFT5XxVXl1EkQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.044 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006858497ed722ad56-006618af28-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.044 DEBUG:boto:path=/dkyges-51/obj6 2024-04-12T03:48:56.044 DEBUG:boto:auth_path=/dkyges-51/obj6 2024-04-12T03:48:56.044 DEBUG:boto:path=/dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.044 DEBUG:boto:auth_path=/dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.045 DEBUG:boto:Method: GET 2024-04-12T03:48:56.045 DEBUG:boto:Path: /dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.045 DEBUG:boto:Data: 2024-04-12T03:48:56.045 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.045 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.045 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.045 DEBUG:boto:Params: {} 2024-04-12T03:48:56.045 DEBUG:boto:Token: None 2024-04-12T03:48:56.045 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj6?versionId=null 2024-04-12T03:48:56.046 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4nVnBcc4JnAQWmGFT5XxVXl1EkQ= 2024-04-12T03:48:56.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4nVnBcc4JnAQWmGFT5XxVXl1EkQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.047 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9779f0032f3b104-006618af28-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.048 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:56.048 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:56.048 DEBUG:boto:path=/dkyges-51/?prefix=obj6 2024-04-12T03:48:56.048 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj6 2024-04-12T03:48:56.048 DEBUG:boto:Method: GET 2024-04-12T03:48:56.048 DEBUG:boto:Path: /dkyges-51/?prefix=obj6 2024-04-12T03:48:56.048 DEBUG:boto:Data: 2024-04-12T03:48:56.048 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.049 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.049 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.049 DEBUG:boto:Params: {} 2024-04-12T03:48:56.049 DEBUG:boto:Token: None 2024-04-12T03:48:56.049 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/ 2024-04-12T03:48:56.049 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ= 2024-04-12T03:48:56.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076b7ec7e6f30da15-006618af28-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.052 DEBUG:boto:b'dkyges-51obj61000falseobj62024-04-12T03:48:17.485Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:56.052 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:56.052 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:56.053 DEBUG:boto:path=/dkyges-51/?prefix=obj6 2024-04-12T03:48:56.053 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj6 2024-04-12T03:48:56.053 DEBUG:boto:Method: GET 2024-04-12T03:48:56.053 DEBUG:boto:Path: /dkyges-51/?prefix=obj6 2024-04-12T03:48:56.053 DEBUG:boto:Data: 2024-04-12T03:48:56.053 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.053 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.053 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.053 DEBUG:boto:Params: {} 2024-04-12T03:48:56.053 DEBUG:boto:Token: None 2024-04-12T03:48:56.054 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/ 2024-04-12T03:48:56.054 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ= 2024-04-12T03:48:56.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c361724b57aac393-006618af28-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.098 DEBUG:boto:b'dkyges-51obj61000falseobj62024-04-12T03:48:17.485Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:56.099 DEBUG:boto:path=/dkyges-51/obj6 2024-04-12T03:48:56.099 DEBUG:boto:auth_path=/dkyges-51/obj6 2024-04-12T03:48:56.099 DEBUG:boto:Method: HEAD 2024-04-12T03:48:56.099 DEBUG:boto:Path: /dkyges-51/obj6 2024-04-12T03:48:56.099 DEBUG:boto:Data: 2024-04-12T03:48:56.099 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.099 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.099 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.100 DEBUG:boto:Params: {} 2024-04-12T03:48:56.100 DEBUG:boto:Token: None 2024-04-12T03:48:56.100 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj6 2024-04-12T03:48:56.100 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:doDfRTuFY9nhSUeG83uOwAi3zDA= 2024-04-12T03:48:56.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:doDfRTuFY9nhSUeG83uOwAi3zDA=', 'Content-Length': '0'} 2024-04-12T03:48:56.102 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093c6c5b3eea2a527-006618af28-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.102 DEBUG:boto:path=/dkyges-51/obj6 2024-04-12T03:48:56.102 DEBUG:boto:auth_path=/dkyges-51/obj6 2024-04-12T03:48:56.103 DEBUG:boto:Method: HEAD 2024-04-12T03:48:56.103 DEBUG:boto:Path: /dkyges-51/obj6 2024-04-12T03:48:56.103 DEBUG:boto:Data: 2024-04-12T03:48:56.103 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.103 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.103 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.103 DEBUG:boto:Params: {} 2024-04-12T03:48:56.103 DEBUG:boto:Token: None 2024-04-12T03:48:56.103 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj6 2024-04-12T03:48:56.103 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:doDfRTuFY9nhSUeG83uOwAi3zDA= 2024-04-12T03:48:56.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:doDfRTuFY9nhSUeG83uOwAi3zDA=', 'Content-Length': '0'} 2024-04-12T03:48:56.105 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e637bffd1379179-006618af28-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2024-04-12T03:48:56.106 DEBUG:boto:path=/dkyges-51/obj6 2024-04-12T03:48:56.106 DEBUG:boto:auth_path=/dkyges-51/obj6 2024-04-12T03:48:56.106 DEBUG:boto:Method: GET 2024-04-12T03:48:56.106 DEBUG:boto:Path: /dkyges-51/obj6 2024-04-12T03:48:56.106 DEBUG:boto:Data: 2024-04-12T03:48:56.106 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.106 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.106 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.106 DEBUG:boto:Params: {} 2024-04-12T03:48:56.106 DEBUG:boto:Token: None 2024-04-12T03:48:56.107 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj6 2024-04-12T03:48:56.107 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:thJ5j1CCTAnfQRvgEEcT+Cp+sz8= 2024-04-12T03:48:56.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:thJ5j1CCTAnfQRvgEEcT+Cp+sz8=', 'Content-Length': '0'} 2024-04-12T03:48:56.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000953dc3e13f19684f-006618af28-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.109 DEBUG:boto:path=/dkyges-51/obj6 2024-04-12T03:48:56.109 DEBUG:boto:auth_path=/dkyges-51/obj6 2024-04-12T03:48:56.109 DEBUG:boto:Method: GET 2024-04-12T03:48:56.109 DEBUG:boto:Path: /dkyges-51/obj6 2024-04-12T03:48:56.109 DEBUG:boto:Data: 2024-04-12T03:48:56.109 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.109 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.109 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.109 DEBUG:boto:Params: {} 2024-04-12T03:48:56.109 DEBUG:boto:Token: None 2024-04-12T03:48:56.110 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj6 2024-04-12T03:48:56.110 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:thJ5j1CCTAnfQRvgEEcT+Cp+sz8= 2024-04-12T03:48:56.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:thJ5j1CCTAnfQRvgEEcT+Cp+sz8=', 'Content-Length': '0'} 2024-04-12T03:48:56.112 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000080d3241dc885d8b5-006618af28-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.112 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:56.112 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:56.112 DEBUG:boto:path=/dkyges-51/?prefix=obj6 2024-04-12T03:48:56.112 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj6 2024-04-12T03:48:56.112 DEBUG:boto:Method: GET 2024-04-12T03:48:56.112 DEBUG:boto:Path: /dkyges-51/?prefix=obj6 2024-04-12T03:48:56.112 DEBUG:boto:Data: 2024-04-12T03:48:56.113 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.113 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.113 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.113 DEBUG:boto:Params: {} 2024-04-12T03:48:56.113 DEBUG:boto:Token: None 2024-04-12T03:48:56.113 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/ 2024-04-12T03:48:56.113 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ= 2024-04-12T03:48:56.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.116 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ed3f307dea2c02d-006618af28-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.116 DEBUG:boto:b'dkyges-51obj61000falseobj62024-04-12T03:48:17.485Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:56.116 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:56.116 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:56.117 DEBUG:boto:path=/dkyges-51/?prefix=obj6 2024-04-12T03:48:56.117 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj6 2024-04-12T03:48:56.117 DEBUG:boto:Method: GET 2024-04-12T03:48:56.117 DEBUG:boto:Path: /dkyges-51/?prefix=obj6 2024-04-12T03:48:56.117 DEBUG:boto:Data: 2024-04-12T03:48:56.117 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.117 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.117 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.117 DEBUG:boto:Params: {} 2024-04-12T03:48:56.117 DEBUG:boto:Token: None 2024-04-12T03:48:56.117 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/ 2024-04-12T03:48:56.118 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ= 2024-04-12T03:48:56.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2dfd475de6ee2d7-006618af28-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.120 DEBUG:boto:b'dkyges-51obj61000falseobj62024-04-12T03:48:17.485Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:56.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:48:56.121 DEBUG:boto:path=/dkyges-51/obj7 2024-04-12T03:48:56.121 DEBUG:boto:auth_path=/dkyges-51/obj7 2024-04-12T03:48:56.121 DEBUG:boto:path=/dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.121 DEBUG:boto:auth_path=/dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.121 DEBUG:boto:Method: GET 2024-04-12T03:48:56.121 DEBUG:boto:Path: /dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.121 DEBUG:boto:Data: 2024-04-12T03:48:56.121 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.121 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.122 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.122 DEBUG:boto:Params: {} 2024-04-12T03:48:56.122 DEBUG:boto:Token: None 2024-04-12T03:48:56.122 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.122 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P7Gfaz7glq8O6Hnv0GIeuod772E= 2024-04-12T03:48:56.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P7Gfaz7glq8O6Hnv0GIeuod772E=', 'Content-Length': '0'} 2024-04-12T03:48:56.124 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c45bdadfc398f01-006618af28-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.124 DEBUG:boto:path=/dkyges-51/obj7 2024-04-12T03:48:56.124 DEBUG:boto:auth_path=/dkyges-51/obj7 2024-04-12T03:48:56.124 DEBUG:boto:path=/dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.124 DEBUG:boto:auth_path=/dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.124 DEBUG:boto:Method: GET 2024-04-12T03:48:56.124 DEBUG:boto:Path: /dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.125 DEBUG:boto:Data: 2024-04-12T03:48:56.125 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.125 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.125 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.125 DEBUG:boto:Params: {} 2024-04-12T03:48:56.125 DEBUG:boto:Token: None 2024-04-12T03:48:56.125 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.125 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P7Gfaz7glq8O6Hnv0GIeuod772E= 2024-04-12T03:48:56.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P7Gfaz7glq8O6Hnv0GIeuod772E=', 'Content-Length': '0'} 2024-04-12T03:48:56.127 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a11c60e217ca59c6-006618af28-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.128 DEBUG:boto:path=/dkyges-51/obj7 2024-04-12T03:48:56.128 DEBUG:boto:auth_path=/dkyges-51/obj7 2024-04-12T03:48:56.128 DEBUG:boto:path=/dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.128 DEBUG:boto:auth_path=/dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.128 DEBUG:boto:Method: HEAD 2024-04-12T03:48:56.128 DEBUG:boto:Path: /dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.128 DEBUG:boto:Data: 2024-04-12T03:48:56.128 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.128 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.128 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.128 DEBUG:boto:Params: {} 2024-04-12T03:48:56.129 DEBUG:boto:Token: None 2024-04-12T03:48:56.129 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.129 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vwvggfpzvYPFCIEkuW53EFy/Xiw= 2024-04-12T03:48:56.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vwvggfpzvYPFCIEkuW53EFy/Xiw=', 'Content-Length': '0'} 2024-04-12T03:48:56.130 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e431e2ae7d6e8e25-006618af28-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.131 DEBUG:boto:path=/dkyges-51/obj7 2024-04-12T03:48:56.131 DEBUG:boto:auth_path=/dkyges-51/obj7 2024-04-12T03:48:56.131 DEBUG:boto:path=/dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.131 DEBUG:boto:auth_path=/dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.131 DEBUG:boto:Method: HEAD 2024-04-12T03:48:56.131 DEBUG:boto:Path: /dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.131 DEBUG:boto:Data: 2024-04-12T03:48:56.131 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.132 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.132 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.132 DEBUG:boto:Params: {} 2024-04-12T03:48:56.132 DEBUG:boto:Token: None 2024-04-12T03:48:56.132 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.132 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vwvggfpzvYPFCIEkuW53EFy/Xiw= 2024-04-12T03:48:56.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vwvggfpzvYPFCIEkuW53EFy/Xiw=', 'Content-Length': '0'} 2024-04-12T03:48:56.134 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ec3a41fa18a42de-006618af28-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:48:56.134 DEBUG:boto:path=/dkyges-51/obj7 2024-04-12T03:48:56.134 DEBUG:boto:auth_path=/dkyges-51/obj7 2024-04-12T03:48:56.135 DEBUG:boto:path=/dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.135 DEBUG:boto:auth_path=/dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.135 DEBUG:boto:Method: GET 2024-04-12T03:48:56.135 DEBUG:boto:Path: /dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.135 DEBUG:boto:Data: 2024-04-12T03:48:56.135 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.135 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.135 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.135 DEBUG:boto:Params: {} 2024-04-12T03:48:56.135 DEBUG:boto:Token: None 2024-04-12T03:48:56.135 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.136 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P7Gfaz7glq8O6Hnv0GIeuod772E= 2024-04-12T03:48:56.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P7Gfaz7glq8O6Hnv0GIeuod772E=', 'Content-Length': '0'} 2024-04-12T03:48:56.137 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079b9ea7552b9ab91-006618af28-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.178 DEBUG:boto:path=/dkyges-51/obj7 2024-04-12T03:48:56.179 DEBUG:boto:auth_path=/dkyges-51/obj7 2024-04-12T03:48:56.179 DEBUG:boto:path=/dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.179 DEBUG:boto:auth_path=/dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.179 DEBUG:boto:Method: GET 2024-04-12T03:48:56.179 DEBUG:boto:Path: /dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.179 DEBUG:boto:Data: 2024-04-12T03:48:56.179 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.179 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.179 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.179 DEBUG:boto:Params: {} 2024-04-12T03:48:56.180 DEBUG:boto:Token: None 2024-04-12T03:48:56.180 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj7?versionId=null 2024-04-12T03:48:56.180 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P7Gfaz7glq8O6Hnv0GIeuod772E= 2024-04-12T03:48:56.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P7Gfaz7glq8O6Hnv0GIeuod772E=', 'Content-Length': '0'} 2024-04-12T03:48:56.182 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b9f9593e5acb93c-006618af28-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.182 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:56.182 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:56.183 DEBUG:boto:path=/dkyges-51/?prefix=obj7 2024-04-12T03:48:56.183 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj7 2024-04-12T03:48:56.183 DEBUG:boto:Method: GET 2024-04-12T03:48:56.183 DEBUG:boto:Path: /dkyges-51/?prefix=obj7 2024-04-12T03:48:56.183 DEBUG:boto:Data: 2024-04-12T03:48:56.183 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.183 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.184 DEBUG:boto:Params: {} 2024-04-12T03:48:56.184 DEBUG:boto:Token: None 2024-04-12T03:48:56.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/ 2024-04-12T03:48:56.184 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ= 2024-04-12T03:48:56.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015e5f7c90245d336-006618af28-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.186 DEBUG:boto:b'dkyges-51obj71000falseobj72024-04-12T03:48:17.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:56.187 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:56.187 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:56.187 DEBUG:boto:path=/dkyges-51/?prefix=obj7 2024-04-12T03:48:56.187 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj7 2024-04-12T03:48:56.187 DEBUG:boto:Method: GET 2024-04-12T03:48:56.187 DEBUG:boto:Path: /dkyges-51/?prefix=obj7 2024-04-12T03:48:56.187 DEBUG:boto:Data: 2024-04-12T03:48:56.188 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.188 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.188 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.188 DEBUG:boto:Params: {} 2024-04-12T03:48:56.188 DEBUG:boto:Token: None 2024-04-12T03:48:56.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/ 2024-04-12T03:48:56.188 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ= 2024-04-12T03:48:56.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094e842d6e559bb6e-006618af28-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.191 DEBUG:boto:b'dkyges-51obj71000falseobj72024-04-12T03:48:17.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:56.191 DEBUG:boto:path=/dkyges-51/obj7 2024-04-12T03:48:56.191 DEBUG:boto:auth_path=/dkyges-51/obj7 2024-04-12T03:48:56.191 DEBUG:boto:Method: HEAD 2024-04-12T03:48:56.192 DEBUG:boto:Path: /dkyges-51/obj7 2024-04-12T03:48:56.192 DEBUG:boto:Data: 2024-04-12T03:48:56.192 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.192 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.192 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.192 DEBUG:boto:Params: {} 2024-04-12T03:48:56.192 DEBUG:boto:Token: None 2024-04-12T03:48:56.192 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj7 2024-04-12T03:48:56.192 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ylgWkq1+fD+NYnDJ6h+ZQyUb+Vw= 2024-04-12T03:48:56.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ylgWkq1+fD+NYnDJ6h+ZQyUb+Vw=', 'Content-Length': '0'} 2024-04-12T03:48:56.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffa1cef2b1b00397-006618af28-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.194 DEBUG:boto:path=/dkyges-51/obj7 2024-04-12T03:48:56.194 DEBUG:boto:auth_path=/dkyges-51/obj7 2024-04-12T03:48:56.195 DEBUG:boto:Method: HEAD 2024-04-12T03:48:56.195 DEBUG:boto:Path: /dkyges-51/obj7 2024-04-12T03:48:56.195 DEBUG:boto:Data: 2024-04-12T03:48:56.195 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.195 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.195 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.195 DEBUG:boto:Params: {} 2024-04-12T03:48:56.195 DEBUG:boto:Token: None 2024-04-12T03:48:56.195 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj7 2024-04-12T03:48:56.195 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ylgWkq1+fD+NYnDJ6h+ZQyUb+Vw= 2024-04-12T03:48:56.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ylgWkq1+fD+NYnDJ6h+ZQyUb+Vw=', 'Content-Length': '0'} 2024-04-12T03:48:56.199 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d46c1ce1ba0b69b5-006618af28-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2024-04-12T03:48:56.199 DEBUG:boto:path=/dkyges-51/obj7 2024-04-12T03:48:56.199 DEBUG:boto:auth_path=/dkyges-51/obj7 2024-04-12T03:48:56.199 DEBUG:boto:Method: GET 2024-04-12T03:48:56.200 DEBUG:boto:Path: /dkyges-51/obj7 2024-04-12T03:48:56.200 DEBUG:boto:Data: 2024-04-12T03:48:56.200 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.200 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.200 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.200 DEBUG:boto:Params: {} 2024-04-12T03:48:56.200 DEBUG:boto:Token: None 2024-04-12T03:48:56.200 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj7 2024-04-12T03:48:56.200 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiaBRqAeXf2XzXzfBUj4+zUpsVw= 2024-04-12T03:48:56.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiaBRqAeXf2XzXzfBUj4+zUpsVw=', 'Content-Length': '0'} 2024-04-12T03:48:56.202 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000456ad8ba3dcb2190-006618af28-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.203 DEBUG:boto:path=/dkyges-51/obj7 2024-04-12T03:48:56.203 DEBUG:boto:auth_path=/dkyges-51/obj7 2024-04-12T03:48:56.203 DEBUG:boto:Method: GET 2024-04-12T03:48:56.203 DEBUG:boto:Path: /dkyges-51/obj7 2024-04-12T03:48:56.203 DEBUG:boto:Data: 2024-04-12T03:48:56.203 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.203 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.203 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.203 DEBUG:boto:Params: {} 2024-04-12T03:48:56.203 DEBUG:boto:Token: None 2024-04-12T03:48:56.204 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj7 2024-04-12T03:48:56.204 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiaBRqAeXf2XzXzfBUj4+zUpsVw= 2024-04-12T03:48:56.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiaBRqAeXf2XzXzfBUj4+zUpsVw=', 'Content-Length': '0'} 2024-04-12T03:48:56.206 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000050ca48c6ddbbcf1-006618af28-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.206 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:56.206 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:56.206 DEBUG:boto:path=/dkyges-51/?prefix=obj7 2024-04-12T03:48:56.207 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj7 2024-04-12T03:48:56.207 DEBUG:boto:Method: GET 2024-04-12T03:48:56.207 DEBUG:boto:Path: /dkyges-51/?prefix=obj7 2024-04-12T03:48:56.207 DEBUG:boto:Data: 2024-04-12T03:48:56.207 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.207 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.207 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.207 DEBUG:boto:Params: {} 2024-04-12T03:48:56.207 DEBUG:boto:Token: None 2024-04-12T03:48:56.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/ 2024-04-12T03:48:56.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ= 2024-04-12T03:48:56.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067f40feaf0b38dcd-006618af28-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.250 DEBUG:boto:b'dkyges-51obj71000falseobj72024-04-12T03:48:17.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:56.251 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:56.251 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:56.251 DEBUG:boto:path=/dkyges-51/?prefix=obj7 2024-04-12T03:48:56.251 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj7 2024-04-12T03:48:56.251 DEBUG:boto:Method: GET 2024-04-12T03:48:56.251 DEBUG:boto:Path: /dkyges-51/?prefix=obj7 2024-04-12T03:48:56.251 DEBUG:boto:Data: 2024-04-12T03:48:56.251 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.252 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.252 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.252 DEBUG:boto:Params: {} 2024-04-12T03:48:56.252 DEBUG:boto:Token: None 2024-04-12T03:48:56.252 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/ 2024-04-12T03:48:56.252 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ= 2024-04-12T03:48:56.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084bccc8a7b40533b-006618af28-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.255 DEBUG:boto:b'dkyges-51obj71000falseobj72024-04-12T03:48:17.495Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:56.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:48:56.256 DEBUG:boto:path=/dkyges-51/obj8 2024-04-12T03:48:56.256 DEBUG:boto:auth_path=/dkyges-51/obj8 2024-04-12T03:48:56.256 DEBUG:boto:path=/dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.256 DEBUG:boto:auth_path=/dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.256 DEBUG:boto:Method: GET 2024-04-12T03:48:56.256 DEBUG:boto:Path: /dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.257 DEBUG:boto:Data: 2024-04-12T03:48:56.257 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.257 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.257 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.257 DEBUG:boto:Params: {} 2024-04-12T03:48:56.257 DEBUG:boto:Token: None 2024-04-12T03:48:56.257 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.257 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0EhJGup3krd7cHcvqY5vxVoeQ1g= 2024-04-12T03:48:56.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0EhJGup3krd7cHcvqY5vxVoeQ1g=', 'Content-Length': '0'} 2024-04-12T03:48:56.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043137118d67f85fa-006618af28-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.259 DEBUG:boto:path=/dkyges-51/obj8 2024-04-12T03:48:56.259 DEBUG:boto:auth_path=/dkyges-51/obj8 2024-04-12T03:48:56.260 DEBUG:boto:path=/dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.260 DEBUG:boto:auth_path=/dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.260 DEBUG:boto:Method: GET 2024-04-12T03:48:56.260 DEBUG:boto:Path: /dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.260 DEBUG:boto:Data: 2024-04-12T03:48:56.260 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.260 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.260 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.260 DEBUG:boto:Params: {} 2024-04-12T03:48:56.260 DEBUG:boto:Token: None 2024-04-12T03:48:56.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.261 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0EhJGup3krd7cHcvqY5vxVoeQ1g= 2024-04-12T03:48:56.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0EhJGup3krd7cHcvqY5vxVoeQ1g=', 'Content-Length': '0'} 2024-04-12T03:48:56.263 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ed5809dea88cb6b-006618af28-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.306 DEBUG:boto:path=/dkyges-51/obj8 2024-04-12T03:48:56.307 DEBUG:boto:auth_path=/dkyges-51/obj8 2024-04-12T03:48:56.307 DEBUG:boto:path=/dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.307 DEBUG:boto:auth_path=/dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.307 DEBUG:boto:Method: HEAD 2024-04-12T03:48:56.307 DEBUG:boto:Path: /dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.307 DEBUG:boto:Data: 2024-04-12T03:48:56.307 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.307 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.307 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.307 DEBUG:boto:Params: {} 2024-04-12T03:48:56.308 DEBUG:boto:Token: None 2024-04-12T03:48:56.308 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.308 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:O+i+Tm6ebafowPHsRG1hHQW/OWo= 2024-04-12T03:48:56.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:O+i+Tm6ebafowPHsRG1hHQW/OWo=', 'Content-Length': '0'} 2024-04-12T03:48:56.310 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000854b44858bd4735b-006618af28-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.310 DEBUG:boto:path=/dkyges-51/obj8 2024-04-12T03:48:56.310 DEBUG:boto:auth_path=/dkyges-51/obj8 2024-04-12T03:48:56.310 DEBUG:boto:path=/dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.310 DEBUG:boto:auth_path=/dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.311 DEBUG:boto:Method: HEAD 2024-04-12T03:48:56.311 DEBUG:boto:Path: /dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.311 DEBUG:boto:Data: 2024-04-12T03:48:56.311 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.311 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.311 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.311 DEBUG:boto:Params: {} 2024-04-12T03:48:56.311 DEBUG:boto:Token: None 2024-04-12T03:48:56.311 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.311 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:O+i+Tm6ebafowPHsRG1hHQW/OWo= 2024-04-12T03:48:56.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:O+i+Tm6ebafowPHsRG1hHQW/OWo=', 'Content-Length': '0'} 2024-04-12T03:48:56.313 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000caf619611a2359b6-006618af28-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:48:56.314 DEBUG:boto:path=/dkyges-51/obj8 2024-04-12T03:48:56.314 DEBUG:boto:auth_path=/dkyges-51/obj8 2024-04-12T03:48:56.314 DEBUG:boto:path=/dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.314 DEBUG:boto:auth_path=/dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.314 DEBUG:boto:Method: GET 2024-04-12T03:48:56.314 DEBUG:boto:Path: /dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.314 DEBUG:boto:Data: 2024-04-12T03:48:56.314 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.314 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.315 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.315 DEBUG:boto:Params: {} 2024-04-12T03:48:56.315 DEBUG:boto:Token: None 2024-04-12T03:48:56.315 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.315 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0EhJGup3krd7cHcvqY5vxVoeQ1g= 2024-04-12T03:48:56.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0EhJGup3krd7cHcvqY5vxVoeQ1g=', 'Content-Length': '0'} 2024-04-12T03:48:56.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fd7207029c90245-006618af28-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.318 DEBUG:boto:path=/dkyges-51/obj8 2024-04-12T03:48:56.318 DEBUG:boto:auth_path=/dkyges-51/obj8 2024-04-12T03:48:56.318 DEBUG:boto:path=/dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.318 DEBUG:boto:auth_path=/dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.318 DEBUG:boto:Method: GET 2024-04-12T03:48:56.318 DEBUG:boto:Path: /dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.318 DEBUG:boto:Data: 2024-04-12T03:48:56.319 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.319 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.319 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.319 DEBUG:boto:Params: {} 2024-04-12T03:48:56.319 DEBUG:boto:Token: None 2024-04-12T03:48:56.319 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj8?versionId=null 2024-04-12T03:48:56.319 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0EhJGup3krd7cHcvqY5vxVoeQ1g= 2024-04-12T03:48:56.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0EhJGup3krd7cHcvqY5vxVoeQ1g=', 'Content-Length': '0'} 2024-04-12T03:48:56.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dab38d9c1961758f-006618af28-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.322 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:56.322 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:56.322 DEBUG:boto:path=/dkyges-51/?prefix=obj8 2024-04-12T03:48:56.322 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj8 2024-04-12T03:48:56.322 DEBUG:boto:Method: GET 2024-04-12T03:48:56.322 DEBUG:boto:Path: /dkyges-51/?prefix=obj8 2024-04-12T03:48:56.322 DEBUG:boto:Data: 2024-04-12T03:48:56.322 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.322 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.322 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.322 DEBUG:boto:Params: {} 2024-04-12T03:48:56.323 DEBUG:boto:Token: None 2024-04-12T03:48:56.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/ 2024-04-12T03:48:56.323 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ= 2024-04-12T03:48:56.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab955e8e0892bc3c-006618af28-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.326 DEBUG:boto:b'dkyges-51obj81000falseobj82024-04-12T03:48:17.545Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:56.327 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:56.327 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:56.327 DEBUG:boto:path=/dkyges-51/?prefix=obj8 2024-04-12T03:48:56.327 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj8 2024-04-12T03:48:56.327 DEBUG:boto:Method: GET 2024-04-12T03:48:56.327 DEBUG:boto:Path: /dkyges-51/?prefix=obj8 2024-04-12T03:48:56.327 DEBUG:boto:Data: 2024-04-12T03:48:56.328 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.328 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.328 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.328 DEBUG:boto:Params: {} 2024-04-12T03:48:56.328 DEBUG:boto:Token: None 2024-04-12T03:48:56.328 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/ 2024-04-12T03:48:56.328 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ= 2024-04-12T03:48:56.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052d8506aa3af19db-006618af28-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.331 DEBUG:boto:b'dkyges-51obj81000falseobj82024-04-12T03:48:17.545Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:56.331 DEBUG:boto:path=/dkyges-51/obj8 2024-04-12T03:48:56.331 DEBUG:boto:auth_path=/dkyges-51/obj8 2024-04-12T03:48:56.332 DEBUG:boto:Method: HEAD 2024-04-12T03:48:56.332 DEBUG:boto:Path: /dkyges-51/obj8 2024-04-12T03:48:56.332 DEBUG:boto:Data: 2024-04-12T03:48:56.332 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.332 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.332 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.332 DEBUG:boto:Params: {} 2024-04-12T03:48:56.332 DEBUG:boto:Token: None 2024-04-12T03:48:56.332 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj8 2024-04-12T03:48:56.332 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZWDfICh4lRRG+VKQdfzP4Q8Jg3Y= 2024-04-12T03:48:56.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZWDfICh4lRRG+VKQdfzP4Q8Jg3Y=', 'Content-Length': '0'} 2024-04-12T03:48:56.334 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f9129ef686c0fc2-006618af28-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.334 DEBUG:boto:path=/dkyges-51/obj8 2024-04-12T03:48:56.335 DEBUG:boto:auth_path=/dkyges-51/obj8 2024-04-12T03:48:56.337 DEBUG:boto:Method: HEAD 2024-04-12T03:48:56.338 DEBUG:boto:Path: /dkyges-51/obj8 2024-04-12T03:48:56.338 DEBUG:boto:Data: 2024-04-12T03:48:56.338 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.338 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.338 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.338 DEBUG:boto:Params: {} 2024-04-12T03:48:56.338 DEBUG:boto:Token: None 2024-04-12T03:48:56.338 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj8 2024-04-12T03:48:56.338 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZWDfICh4lRRG+VKQdfzP4Q8Jg3Y= 2024-04-12T03:48:56.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZWDfICh4lRRG+VKQdfzP4Q8Jg3Y=', 'Content-Length': '0'} 2024-04-12T03:48:56.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071f3d25b35211c6e-006618af28-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2024-04-12T03:48:56.341 DEBUG:boto:path=/dkyges-51/obj8 2024-04-12T03:48:56.341 DEBUG:boto:auth_path=/dkyges-51/obj8 2024-04-12T03:48:56.341 DEBUG:boto:Method: GET 2024-04-12T03:48:56.341 DEBUG:boto:Path: /dkyges-51/obj8 2024-04-12T03:48:56.341 DEBUG:boto:Data: 2024-04-12T03:48:56.341 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.341 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.341 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.341 DEBUG:boto:Params: {} 2024-04-12T03:48:56.342 DEBUG:boto:Token: None 2024-04-12T03:48:56.342 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj8 2024-04-12T03:48:56.342 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:s0oHKPhCXV9liMz3TxtEYCxauAg= 2024-04-12T03:48:56.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:s0oHKPhCXV9liMz3TxtEYCxauAg=', 'Content-Length': '0'} 2024-04-12T03:48:56.344 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9cd0c64a2a11690-006618af28-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.344 DEBUG:boto:path=/dkyges-51/obj8 2024-04-12T03:48:56.344 DEBUG:boto:auth_path=/dkyges-51/obj8 2024-04-12T03:48:56.344 DEBUG:boto:Method: GET 2024-04-12T03:48:56.344 DEBUG:boto:Path: /dkyges-51/obj8 2024-04-12T03:48:56.344 DEBUG:boto:Data: 2024-04-12T03:48:56.344 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.344 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.344 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.345 DEBUG:boto:Params: {} 2024-04-12T03:48:56.345 DEBUG:boto:Token: None 2024-04-12T03:48:56.345 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/obj8 2024-04-12T03:48:56.345 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:s0oHKPhCXV9liMz3TxtEYCxauAg= 2024-04-12T03:48:56.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:s0oHKPhCXV9liMz3TxtEYCxauAg=', 'Content-Length': '0'} 2024-04-12T03:48:56.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:48:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-51:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095af6d9109c943fe-006618af28-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.347 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:56.347 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:56.348 DEBUG:boto:path=/dkyges-51/?prefix=obj8 2024-04-12T03:48:56.348 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj8 2024-04-12T03:48:56.348 DEBUG:boto:Method: GET 2024-04-12T03:48:56.348 DEBUG:boto:Path: /dkyges-51/?prefix=obj8 2024-04-12T03:48:56.348 DEBUG:boto:Data: 2024-04-12T03:48:56.348 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.348 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:48:56.348 DEBUG:boto:Port: 8001 2024-04-12T03:48:56.348 DEBUG:boto:Params: {} 2024-04-12T03:48:56.348 DEBUG:boto:Token: None 2024-04-12T03:48:56.348 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/ 2024-04-12T03:48:56.349 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ= 2024-04-12T03:48:56.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079bf4fd11726ca89-006618af28-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.351 DEBUG:boto:b'dkyges-51obj81000falseobj82024-04-12T03:48:17.545Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:56.351 DEBUG:boto:path=/dkyges-51/ 2024-04-12T03:48:56.352 DEBUG:boto:auth_path=/dkyges-51/ 2024-04-12T03:48:56.352 DEBUG:boto:path=/dkyges-51/?prefix=obj8 2024-04-12T03:48:56.352 DEBUG:boto:auth_path=/dkyges-51/?prefix=obj8 2024-04-12T03:48:56.352 DEBUG:boto:Method: GET 2024-04-12T03:48:56.352 DEBUG:boto:Path: /dkyges-51/?prefix=obj8 2024-04-12T03:48:56.352 DEBUG:boto:Data: 2024-04-12T03:48:56.352 DEBUG:boto:Headers: {} 2024-04-12T03:48:56.352 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:48:56.352 DEBUG:boto:Port: 8000 2024-04-12T03:48:56.352 DEBUG:boto:Params: {} 2024-04-12T03:48:56.352 DEBUG:boto:Token: None 2024-04-12T03:48:56.353 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:48:56 GMT /dkyges-51/ 2024-04-12T03:48:56.353 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ= 2024-04-12T03:48:56.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:48:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LtlbNHHFKj/xJHd+jecm4sxg/dQ=', 'Content-Length': '0'} 2024-04-12T03:48:56.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001103eca4ed3b29cc-006618af28-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:48:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:48:56.398 DEBUG:boto:b'dkyges-51obj81000falseobj82024-04-12T03:48:17.545Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:48:56.399 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-51 zones={a2, a1} 2024-04-12T03:48:56.399 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-12T03:48:56.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:48:56.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:48:56.399 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:48:56.457 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:48:56.458 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:48:56.514 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:48:56.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:56.514 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000008272", 2024-04-12T03:48:56.514 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:48:44.919830Z" 2024-04-12T03:48:56.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:56.514 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000005445", 2024-04-12T03:48:56.515 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:48:44.968729Z" 2024-04-12T03:48:56.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:56.515 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000012627", 2024-04-12T03:48:56.515 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:48:44.967620Z" 2024-04-12T03:48:56.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:56.515 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000009244", 2024-04-12T03:48:56.515 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:48:17.802289Z" 2024-04-12T03:48:56.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:56.515 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:48:56.516 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000008272', 1: '00000000000000000000:00000000000000005445', 2: '00000000000000000000:00000000000000012627', 3: '00000000000000000000:00000000000000009244'} 2024-04-12T03:48:56.516 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:48:56.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:48:56.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:48:56.517 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:48:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:48:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:48:56.626 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:48:56.626 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:48:56.626 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:48:56.626 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:48:56.627 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:48:56.627 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 16340188497730315678 2024-04-12T03:48:56.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.627 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:48:56.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:56.627 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:48:56.627 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:56.627 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:48:56.627 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000008272", 2024-04-12T03:48:56.627 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:56.627 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:56.628 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:56.628 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:48:44.919830Z" 2024-04-12T03:48:56.628 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:56.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:56.628 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:48:56.628 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:56.628 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:48:56.628 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000005445", 2024-04-12T03:48:56.628 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:56.628 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:56.628 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:56.629 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:48:44.968729Z" 2024-04-12T03:48:56.629 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:56.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.629 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:56.629 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:48:56.629 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:56.629 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:48:56.629 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000012627", 2024-04-12T03:48:56.629 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:56.629 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:56.629 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:56.630 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:48:44.967620Z" 2024-04-12T03:48:56.630 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:56.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:56.630 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:48:56.630 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:56.630 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:48:56.630 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000009244", 2024-04-12T03:48:56.630 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:56.630 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:56.630 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:56.630 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:48:17.802289Z" 2024-04-12T03:48:56.631 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:56.631 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:56.631 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:48:56.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.631 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:48:56.631 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:48:56.631 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:48:56.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.631 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:48:56Z" 2024-04-12T03:48:56.631 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:48:56.633 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 16340188497730315678 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008272", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:48:44.919830Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005445", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:48:44.968729Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012627", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:48:44.967620Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000009244", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:48:17.802289Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:48:56Z" } 2024-04-12T03:48:56.633 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008272', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:48:44.919830Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005445', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:48:44.968729Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012627', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:48:44.967620Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000009244', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:48:17.802289Z'}}] 2024-04-12T03:48:56.633 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000008272', 1: '00000000000000000000:00000000000000005445', 2: '00000000000000000000:00000000000000012627', 3: '00000000000000000000:00000000000000009244'} 2024-04-12T03:48:56.633 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000008272', 1: '00000000000000000000:00000000000000005445', 2: '00000000000000000000:00000000000000012627', 3: '00000000000000000000:00000000000000009244'} 2024-04-12T03:48:56.633 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T03:48:56.633 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-12T03:48:56.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:48:56.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:48:56.634 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:48:56.687 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:48:56.688 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:48:56.745 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:48:56.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:56.745 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006434", 2024-04-12T03:48:56.746 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:48:52.051606Z" 2024-04-12T03:48:56.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.746 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:56.746 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004176", 2024-04-12T03:48:56.746 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:48:52.053256Z" 2024-04-12T03:48:56.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.746 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:56.746 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000011217", 2024-04-12T03:48:56.746 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:48:52.054900Z" 2024-04-12T03:48:56.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.747 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:56.747 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006849", 2024-04-12T03:48:56.747 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:48:52.056607Z" 2024-04-12T03:48:56.747 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:56.747 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:48:56.749 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000006434', 1: '00000000000000000000:00000000000000004176', 2: '00000000000000000000:00000000000000011217', 3: '00000000000000000000:00000000000000006849'} 2024-04-12T03:48:56.750 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:48:56.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:48:56.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:48:56.750 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:48:56.807 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:48:56.807 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:48:56.858 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:48:56.859 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:48:56.859 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:48:56.859 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:48:56.859 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:48:56.859 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5944041172307441597 2024-04-12T03:48:56.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.859 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:48:56.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:56.859 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:48:56.860 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:56.860 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:48:56.860 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006434", 2024-04-12T03:48:56.860 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:56.860 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:56.860 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:56.860 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:48:52.051606Z" 2024-04-12T03:48:56.860 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:56.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:56.861 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:48:56.861 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:56.861 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:48:56.861 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004176", 2024-04-12T03:48:56.861 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:56.861 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:56.861 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:56.861 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:48:52.053256Z" 2024-04-12T03:48:56.861 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:56.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:56.861 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:48:56.862 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:56.862 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:48:56.862 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000010371", 2024-04-12T03:48:56.862 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:56.862 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:56.862 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:56.862 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:48:30.049519Z" 2024-04-12T03:48:56.862 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:56.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:48:56.862 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:48:56.863 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:48:56.863 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:48:56.863 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006849", 2024-04-12T03:48:56.863 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:48:56.863 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:48:56.863 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:48:56.863 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:48:52.056607Z" 2024-04-12T03:48:56.863 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:56.863 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:48:56.863 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:48:56.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.864 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:48:56.864 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:48:56.864 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:48:56.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:48:56.864 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:48:56Z" 2024-04-12T03:48:56.864 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:48:56.865 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5944041172307441597 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:48:52.051606Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004176", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:48:52.053256Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010371", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:48:30.049519Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006849", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:48:52.056607Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:48:56Z" } 2024-04-12T03:48:56.865 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:48:52.051606Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004176', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:48:52.053256Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010371', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:48:30.049519Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006849', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:48:52.056607Z'}}] 2024-04-12T03:48:56.865 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006434', 1: '00000000000000000000:00000000000000004176', 2: '00000000000000000000:00000000000000011217', 3: '00000000000000000000:00000000000000006849'} 2024-04-12T03:48:56.866 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006434', 1: '00000000000000000000:00000000000000004176', 2: '00000000000000000000:00000000000000010371', 3: '00000000000000000000:00000000000000006849'} 2024-04-12T03:48:56.866 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=2 master=00000000000000000000:00000000000000011217 target=00000000000000000000:00000000000000010371 2024-04-12T03:49:01.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:01.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:01.868 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:49:01.930 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:01.930 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:01.983 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:49:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:49:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:49:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:49:01.983 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:49:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5944041172307441597 2024-04-12T03:49:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:49:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:49:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:49:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006434", 2024-04-12T03:49:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:01.984 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:48:52.051606Z" 2024-04-12T03:49:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:49:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:49:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004176", 2024-04-12T03:49:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:01.985 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:48:52.053256Z" 2024-04-12T03:49:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:49:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:49:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000010371", 2024-04-12T03:49:01.986 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:48:30.049519Z" 2024-04-12T03:49:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:49:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:49:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006849", 2024-04-12T03:49:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:01.987 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:48:52.056607Z" 2024-04-12T03:49:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:49:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:49:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:49:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:49:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:01.988 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:49:01Z" 2024-04-12T03:49:01.988 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:49:01.989 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5944041172307441597 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:48:52.051606Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004176", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:48:52.053256Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010371", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:48:30.049519Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006849", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:48:52.056607Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:49:01Z" } 2024-04-12T03:49:01.990 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:48:52.051606Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004176', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:48:52.053256Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010371', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:48:30.049519Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006849', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:48:52.056607Z'}}] 2024-04-12T03:49:01.990 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006434', 1: '00000000000000000000:00000000000000004176', 2: '00000000000000000000:00000000000000011217', 3: '00000000000000000000:00000000000000006849'} 2024-04-12T03:49:01.990 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006434', 1: '00000000000000000000:00000000000000004176', 2: '00000000000000000000:00000000000000010371', 3: '00000000000000000000:00000000000000006849'} 2024-04-12T03:49:01.990 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=2 master=00000000000000000000:00000000000000011217 target=00000000000000000000:00000000000000010371 2024-04-12T03:49:06.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:06.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:06.992 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:49:07.056 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:07.057 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:08.115 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:49:08.116 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:49:08.116 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:49:08.116 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:49:08.116 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:49:08.116 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5944041172307441597 2024-04-12T03:49:08.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:08.116 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:49:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:49:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:49:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006434", 2024-04-12T03:49:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:48:52.051606Z" 2024-04-12T03:49:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:08.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:08.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:08.118 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:49:08.118 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:08.118 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:49:08.118 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000004176", 2024-04-12T03:49:08.118 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:08.118 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:08.118 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:08.118 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:48:52.053256Z" 2024-04-12T03:49:08.119 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:08.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:08.119 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:08.119 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:49:08.119 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:08.119 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:49:08.119 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000011217", 2024-04-12T03:49:08.119 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:08.119 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:08.119 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:08.119 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:48:52.054900Z" 2024-04-12T03:49:08.120 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:08.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:08.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:08.120 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:49:08.120 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:08.120 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T03:49:08.120 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006849", 2024-04-12T03:49:08.120 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:08.120 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:08.121 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:08.121 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:48:52.056607Z" 2024-04-12T03:49:08.121 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:08.121 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:08.121 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:49:08.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:08.121 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:49:08.121 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T03:49:08.121 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T03:49:08.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:08.121 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:49:08Z" 2024-04-12T03:49:08.121 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:49:08.122 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5944041172307441597 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006434", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:48:52.051606Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004176", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:48:52.053256Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011217", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:48:52.054900Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006849", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:48:52.056607Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T03:49:08Z" } 2024-04-12T03:49:08.123 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006434', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:48:52.051606Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004176', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:48:52.053256Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011217', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:48:52.054900Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006849', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:48:52.056607Z'}}] 2024-04-12T03:49:08.123 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006434', 1: '00000000000000000000:00000000000000004176', 2: '00000000000000000000:00000000000000011217', 3: '00000000000000000000:00000000000000006849'} 2024-04-12T03:49:08.123 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006434', 1: '00000000000000000000:00000000000000004176', 2: '00000000000000000000:00000000000000011217', 3: '00000000000000000000:00000000000000006849'} 2024-04-12T03:49:08.123 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T03:49:08.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2024-04-12T03:49:08.125 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-52 2024-04-12T03:49:08.126 DEBUG:boto:path=/dkyges-52/ 2024-04-12T03:49:08.126 DEBUG:boto:auth_path=/dkyges-52/ 2024-04-12T03:49:08.126 DEBUG:boto:Method: PUT 2024-04-12T03:49:08.126 DEBUG:boto:Path: /dkyges-52/ 2024-04-12T03:49:08.126 DEBUG:boto:Data: 2024-04-12T03:49:08.126 DEBUG:boto:Headers: {} 2024-04-12T03:49:08.126 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:08.126 DEBUG:boto:Port: 8000 2024-04-12T03:49:08.126 DEBUG:boto:Params: {} 2024-04-12T03:49:08.127 DEBUG:boto:Token: None 2024-04-12T03:49:08.127 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:49:08 GMT /dkyges-52/ 2024-04-12T03:49:08.127 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:op7VHLrV32mFVwRMbL5YekIs40g= 2024-04-12T03:49:08.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:op7VHLrV32mFVwRMbL5YekIs40g=', 'Content-Length': '0'} 2024-04-12T03:49:08.145 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000000ea7418bd2e5b2b-006618af34-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:08.145 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-53 2024-04-12T03:49:08.146 DEBUG:boto:path=/dkyges-53/ 2024-04-12T03:49:08.146 DEBUG:boto:auth_path=/dkyges-53/ 2024-04-12T03:49:08.146 DEBUG:boto:Method: PUT 2024-04-12T03:49:08.146 DEBUG:boto:Path: /dkyges-53/ 2024-04-12T03:49:08.146 DEBUG:boto:Data: 2024-04-12T03:49:08.146 DEBUG:boto:Headers: {} 2024-04-12T03:49:08.146 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:08.146 DEBUG:boto:Port: 8001 2024-04-12T03:49:08.146 DEBUG:boto:Params: {} 2024-04-12T03:49:08.147 DEBUG:boto:Token: None 2024-04-12T03:49:08.147 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:49:08 GMT /dkyges-53/ 2024-04-12T03:49:08.147 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YXlXU9GC7O5SZfQ9pIaCi9rHEZI= 2024-04-12T03:49:08.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YXlXU9GC7O5SZfQ9pIaCi9rHEZI=', 'Content-Length': '0'} 2024-04-12T03:49:08.178 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bf7816434c1dcb99-006618af34-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:08.178 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:08.178 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:08.178 DEBUG:boto:path=/dkyges-52/MULTIPART?uploads 2024-04-12T03:49:08.178 DEBUG:boto:auth_path=/dkyges-52/MULTIPART?uploads 2024-04-12T03:49:08.178 DEBUG:boto:Method: POST 2024-04-12T03:49:08.178 DEBUG:boto:Path: /dkyges-52/MULTIPART?uploads 2024-04-12T03:49:08.179 DEBUG:boto:Data: 2024-04-12T03:49:08.179 DEBUG:boto:Headers: {} 2024-04-12T03:49:08.179 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:08.179 DEBUG:boto:Port: 8000 2024-04-12T03:49:08.179 DEBUG:boto:Params: {} 2024-04-12T03:49:08.179 DEBUG:boto:Token: None 2024-04-12T03:49:08.179 DEBUG:boto:StringToSign: POST Fri, 12 Apr 2024 03:49:08 GMT /dkyges-52/MULTIPART?uploads 2024-04-12T03:49:08.179 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lPLM3QgBPjMWuBchDQUJmwO5L8o= 2024-04-12T03:49:08.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lPLM3QgBPjMWuBchDQUJmwO5L8o=', 'Content-Length': '0'} 2024-04-12T03:49:09.381 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000430e96d6d968a8e1-006618af34-16d5-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:09.381 DEBUG:boto:b'dkyges-52MULTIPART2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l' 2024-04-12T03:49:09.416 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:09.417 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:09.417 DEBUG:boto:path=/dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l&partNumber=1 2024-04-12T03:49:09.417 DEBUG:boto:auth_path=/dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l&partNumber=1 2024-04-12T03:49:09.417 DEBUG:boto:Method: PUT 2024-04-12T03:49:09.417 DEBUG:boto:Path: /dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l&partNumber=1 2024-04-12T03:49:09.417 DEBUG:boto:Data: 2024-04-12T03:49:09.417 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-12T03:49:09.417 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:09.417 DEBUG:boto:Port: 8000 2024-04-12T03:49:09.417 DEBUG:boto:Params: {} 2024-04-12T03:49:09.418 DEBUG:boto:Token: None 2024-04-12T03:49:09.418 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Fri, 12 Apr 2024 03:49:09 GMT /dkyges-52/MULTIPART?partNumber=1&uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l 2024-04-12T03:49:09.418 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bzxPlA8MfdkDbvyAqWG1Ftt40c4= 2024-04-12T03:49:09.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:49:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bzxPlA8MfdkDbvyAqWG1Ftt40c4='} 2024-04-12T03:49:09.453 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6d91e9f5571f5e2-006618af35-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:09.487 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:09.488 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:09.488 DEBUG:boto:path=/dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l&partNumber=2 2024-04-12T03:49:09.488 DEBUG:boto:auth_path=/dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l&partNumber=2 2024-04-12T03:49:09.488 DEBUG:boto:Method: PUT 2024-04-12T03:49:09.488 DEBUG:boto:Path: /dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l&partNumber=2 2024-04-12T03:49:09.488 DEBUG:boto:Data: 2024-04-12T03:49:09.488 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-12T03:49:09.488 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:09.488 DEBUG:boto:Port: 8000 2024-04-12T03:49:09.488 DEBUG:boto:Params: {} 2024-04-12T03:49:09.489 DEBUG:boto:Token: None 2024-04-12T03:49:09.489 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Fri, 12 Apr 2024 03:49:09 GMT /dkyges-52/MULTIPART?partNumber=2&uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l 2024-04-12T03:49:09.489 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zj9RMTUC1pms/oiCdgD1nxwDfWc= 2024-04-12T03:49:09.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:49:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zj9RMTUC1pms/oiCdgD1nxwDfWc='} 2024-04-12T03:49:09.521 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b058143b8bd30dc5-006618af35-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:09.540 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:09.541 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:09.541 DEBUG:boto:path=/dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l&partNumber=3 2024-04-12T03:49:09.541 DEBUG:boto:auth_path=/dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l&partNumber=3 2024-04-12T03:49:09.541 DEBUG:boto:Method: PUT 2024-04-12T03:49:09.541 DEBUG:boto:Path: /dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l&partNumber=3 2024-04-12T03:49:09.541 DEBUG:boto:Data: 2024-04-12T03:49:09.541 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-12T03:49:09.541 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:09.541 DEBUG:boto:Port: 8000 2024-04-12T03:49:09.542 DEBUG:boto:Params: {} 2024-04-12T03:49:09.542 DEBUG:boto:Token: None 2024-04-12T03:49:09.542 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Fri, 12 Apr 2024 03:49:09 GMT /dkyges-52/MULTIPART?partNumber=3&uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l 2024-04-12T03:49:09.542 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+CF/SO6I6hszreWjZ27xVPFxB/w= 2024-04-12T03:49:09.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:49:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+CF/SO6I6hszreWjZ27xVPFxB/w='} 2024-04-12T03:49:09.573 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e108c30208561d8a-006618af35-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:09.592 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:09.593 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:09.593 DEBUG:boto:path=/dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l&partNumber=4 2024-04-12T03:49:09.593 DEBUG:boto:auth_path=/dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l&partNumber=4 2024-04-12T03:49:09.593 DEBUG:boto:Method: PUT 2024-04-12T03:49:09.593 DEBUG:boto:Path: /dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l&partNumber=4 2024-04-12T03:49:09.593 DEBUG:boto:Data: 2024-04-12T03:49:09.593 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2024-04-12T03:49:09.593 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:09.593 DEBUG:boto:Port: 8000 2024-04-12T03:49:09.593 DEBUG:boto:Params: {} 2024-04-12T03:49:09.593 DEBUG:boto:Token: None 2024-04-12T03:49:09.594 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Fri, 12 Apr 2024 03:49:09 GMT /dkyges-52/MULTIPART?partNumber=4&uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l 2024-04-12T03:49:09.594 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cCfEsPim8aAnAC6aHQRIwwyA0/4= 2024-04-12T03:49:09.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:49:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cCfEsPim8aAnAC6aHQRIwwyA0/4='} 2024-04-12T03:49:09.625 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cacba93f567dcb5c-006618af35-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:09.626 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:09.626 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:09.626 DEBUG:boto:path=/dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l 2024-04-12T03:49:09.626 DEBUG:boto:auth_path=/dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l 2024-04-12T03:49:09.626 DEBUG:boto:Method: GET 2024-04-12T03:49:09.626 DEBUG:boto:Path: /dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l 2024-04-12T03:49:09.626 DEBUG:boto:Data: 2024-04-12T03:49:09.626 DEBUG:boto:Headers: {} 2024-04-12T03:49:09.626 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:09.627 DEBUG:boto:Port: 8000 2024-04-12T03:49:09.627 DEBUG:boto:Params: {} 2024-04-12T03:49:09.627 DEBUG:boto:Token: None 2024-04-12T03:49:09.627 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:09 GMT /dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l 2024-04-12T03:49:09.627 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qSiOK+UeKAA/l8tYMGsR01Hy3rU= 2024-04-12T03:49:09.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qSiOK+UeKAA/l8tYMGsR01Hy3rU=', 'Content-Length': '0'} 2024-04-12T03:49:09.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a44c060de5155e1d-006618af35-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:09.671 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:09.671 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:09.671 DEBUG:boto:path=/dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l 2024-04-12T03:49:09.671 DEBUG:boto:auth_path=/dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l 2024-04-12T03:49:09.671 DEBUG:boto:Method: POST 2024-04-12T03:49:09.672 DEBUG:boto:Path: /dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l 2024-04-12T03:49:09.672 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2024-04-12T03:49:09.672 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2024-04-12T03:49:09.672 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:09.672 DEBUG:boto:Port: 8000 2024-04-12T03:49:09.672 DEBUG:boto:Params: {} 2024-04-12T03:49:09.672 DEBUG:boto:Token: None 2024-04-12T03:49:09.673 DEBUG:boto:StringToSign: POST text/xml Fri, 12 Apr 2024 03:49:09 GMT /dkyges-52/MULTIPART?uploadId=2~XkThSp7AgRG0DGcSWmEcW8JDRarU76l 2024-04-12T03:49:09.673 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+Q76u6GLJrWS4r6p+8a9dZ+yPkI= 2024-04-12T03:49:09.673 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+Q76u6GLJrWS4r6p+8a9dZ+yPkI=', 'Content-Length': '460'} 2024-04-12T03:49:09.684 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e54fe4502af0ac79-006618af35-16d5-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '325'), ('Date', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:09.685 DEBUG:boto:http://smithi043.front.sepia.ceph.com:8000/dkyges-52/MULTIPARTdkyges-52MULTIPARTd07d411772772538cd4b657da660bfb5-4 2024-04-12T03:49:09.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:09.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:09.686 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:49:09.743 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:09.744 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:09.781 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:49:09.796 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:49:09.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:49:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:49:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:49:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:49:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:49:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:49:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893748.170502_474.1", 2024-04-12T03:49:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:49:08.170502Z" 2024-04-12T03:49:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:09.798 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:49:09.799 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893748.170502_474.1'} 2024-04-12T03:49:09.800 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:49:09.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:09.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:09.800 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:49:09.855 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:09.855 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:09.894+0000 7f7fd6333780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:49:09.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:09.895+0000 7f7fd6333780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:49:09.906 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:49:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:49:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:49:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:49:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:49:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:49:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:49:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:49:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:49:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:49:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:49:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:49:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:49:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:49:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:49:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:49:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:49:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:49:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893748.141222_467.1", 2024-04-12T03:49:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:49:08.141222Z", 2024-04-12T03:49:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:49:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:49:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:49:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:49:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:49:09Z" 2024-04-12T03:49:09.912 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:49:09.913 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893748.141222_467.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:49:08.141222Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:49:09Z" } 2024-04-12T03:49:09.913 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893748.141222_467.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:49:08.141222Z', 'realm_epoch': 2}}] 2024-04-12T03:49:09.914 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893748.170502_474.1'} 2024-04-12T03:49:09.914 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893748.141222_467.1'} 2024-04-12T03:49:09.914 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893748.170502_474.1 target=1_1712893748.141222_467.1 2024-04-12T03:49:14.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:14.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:14.916 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:49:14.978 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:14.978 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:15.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:15.016+0000 7f5e90e7b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:49:15.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:15.018+0000 7f5e90e7b780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:49:15.028 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:49:15.029 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:49:15.029 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:49:15.029 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:49:15.029 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:49:15.029 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:49:15.029 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:15.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:15.029 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:49:15.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:15.029 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:49:15.029 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:15.030 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:15.030 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:49:15.030 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:15.030 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:49:15.030 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:15.030 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:49:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:49:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:49:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:49:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:49:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:49:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:49:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:49:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893748.170502_474.1", 2024-04-12T03:49:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:49:08.170502Z", 2024-04-12T03:49:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:49:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:49:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:49:15.035 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:49:15.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:15.035 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:49:15Z" 2024-04-12T03:49:15.035 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:49:15.036 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893748.170502_474.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:49:08.170502Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:49:15Z" } 2024-04-12T03:49:15.036 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893748.170502_474.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:49:08.170502Z', 'realm_epoch': 2}}] 2024-04-12T03:49:15.036 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893748.170502_474.1'} 2024-04-12T03:49:15.036 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893748.170502_474.1'} 2024-04-12T03:49:15.036 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:49:15.037 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-52 2024-04-12T03:49:15.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:49:15.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:49:15.037 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:49:15.101 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:15.101 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:15.146+0000 7fa2a024c780 1 bucket sync caught up with source: 2024-04-12T03:49:15.142 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000011.79.6, , , , , ] 2024-04-12T03:49:15.142 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000011.79.6, , , , , ] 2024-04-12T03:49:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:15.146+0000 7fa2a024c780 0 bucket checkpoint complete 2024-04-12T03:49:15.159 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-52 2024-04-12T03:49:15.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-52', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:49:15.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-52', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:49:15.159 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-52 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:49:15.224 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:15.224 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:15.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:15.270+0000 7fe471141780 1 bucket sync caught up with source: 2024-04-12T03:49:15.265 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.1598.6, , , , , ] 2024-04-12T03:49:15.266 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.1598.6, , , , , ] 2024-04-12T03:49:15.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:15.270+0000 7fe471141780 0 bucket checkpoint complete 2024-04-12T03:49:15.281 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-52 zones={a2, a1} 2024-04-12T03:49:15.281 DEBUG:boto:path=/dkyges-52/ 2024-04-12T03:49:15.281 DEBUG:boto:auth_path=/dkyges-52/ 2024-04-12T03:49:15.282 DEBUG:boto:Method: HEAD 2024-04-12T03:49:15.282 DEBUG:boto:Path: /dkyges-52/ 2024-04-12T03:49:15.282 DEBUG:boto:Data: 2024-04-12T03:49:15.282 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.282 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:15.282 DEBUG:boto:Port: 8001 2024-04-12T03:49:15.282 DEBUG:boto:Params: {} 2024-04-12T03:49:15.282 DEBUG:boto:Token: None 2024-04-12T03:49:15.283 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/ 2024-04-12T03:49:15.283 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:p6fOUMiUGe2Ud4NbbGHilDz4Dms= 2024-04-12T03:49:15.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p6fOUMiUGe2Ud4NbbGHilDz4Dms=', 'Content-Length': '0'} 2024-04-12T03:49:15.285 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3b012e7aa9546a1-006618af3b-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.285 DEBUG:boto:path=/dkyges-52/ 2024-04-12T03:49:15.285 DEBUG:boto:auth_path=/dkyges-52/ 2024-04-12T03:49:15.286 DEBUG:boto:Method: HEAD 2024-04-12T03:49:15.286 DEBUG:boto:Path: /dkyges-52/ 2024-04-12T03:49:15.286 DEBUG:boto:Data: 2024-04-12T03:49:15.286 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.286 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:15.286 DEBUG:boto:Port: 8000 2024-04-12T03:49:15.286 DEBUG:boto:Params: {} 2024-04-12T03:49:15.286 DEBUG:boto:Token: None 2024-04-12T03:49:15.286 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/ 2024-04-12T03:49:15.286 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:p6fOUMiUGe2Ud4NbbGHilDz4Dms= 2024-04-12T03:49:15.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p6fOUMiUGe2Ud4NbbGHilDz4Dms=', 'Content-Length': '0'} 2024-04-12T03:49:15.290 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ffeb1c614995ff1-006618af3b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.290 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:49:15.290 DEBUG:boto:path=/dkyges-52/ 2024-04-12T03:49:15.290 DEBUG:boto:auth_path=/dkyges-52/ 2024-04-12T03:49:15.290 DEBUG:boto:path=/dkyges-52/?versions&max-keys=999 2024-04-12T03:49:15.290 DEBUG:boto:auth_path=/dkyges-52/?versions&max-keys=999 2024-04-12T03:49:15.290 DEBUG:boto:Method: GET 2024-04-12T03:49:15.291 DEBUG:boto:Path: /dkyges-52/?versions&max-keys=999 2024-04-12T03:49:15.291 DEBUG:boto:Data: 2024-04-12T03:49:15.291 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.291 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:15.291 DEBUG:boto:Port: 8001 2024-04-12T03:49:15.291 DEBUG:boto:Params: {} 2024-04-12T03:49:15.291 DEBUG:boto:Token: None 2024-04-12T03:49:15.291 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/?versions 2024-04-12T03:49:15.291 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Te77EF0Las2Sic5e1+pab5V4AH0= 2024-04-12T03:49:15.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Te77EF0Las2Sic5e1+pab5V4AH0=', 'Content-Length': '0'} 2024-04-12T03:49:15.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b6bde88600eed34-006618af3b-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.294 DEBUG:boto:b'dkyges-52999falseMULTIPARTnulltrue2024-04-12T03:49:09.682Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:15.294 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2024-04-12T03:49:15.294 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:49:15.295 DEBUG:boto:path=/dkyges-52/ 2024-04-12T03:49:15.295 DEBUG:boto:auth_path=/dkyges-52/ 2024-04-12T03:49:15.295 DEBUG:boto:path=/dkyges-52/?versions&max-keys=999 2024-04-12T03:49:15.295 DEBUG:boto:auth_path=/dkyges-52/?versions&max-keys=999 2024-04-12T03:49:15.295 DEBUG:boto:Method: GET 2024-04-12T03:49:15.295 DEBUG:boto:Path: /dkyges-52/?versions&max-keys=999 2024-04-12T03:49:15.295 DEBUG:boto:Data: 2024-04-12T03:49:15.295 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.295 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:15.295 DEBUG:boto:Port: 8000 2024-04-12T03:49:15.295 DEBUG:boto:Params: {} 2024-04-12T03:49:15.296 DEBUG:boto:Token: None 2024-04-12T03:49:15.296 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/?versions 2024-04-12T03:49:15.296 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Te77EF0Las2Sic5e1+pab5V4AH0= 2024-04-12T03:49:15.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Te77EF0Las2Sic5e1+pab5V4AH0=', 'Content-Length': '0'} 2024-04-12T03:49:15.298 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004982007ac5d424e7-006618af3b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.298 DEBUG:boto:b'dkyges-52999falseMULTIPARTnulltrue2024-04-12T03:49:09.682Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:15.298 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2024-04-12T03:49:15.299 DEBUG:boto:path=/dkyges-52/ 2024-04-12T03:49:15.299 DEBUG:boto:auth_path=/dkyges-52/ 2024-04-12T03:49:15.299 DEBUG:boto:path=/dkyges-52/?versions&max-keys=999 2024-04-12T03:49:15.299 DEBUG:boto:auth_path=/dkyges-52/?versions&max-keys=999 2024-04-12T03:49:15.299 DEBUG:boto:Method: GET 2024-04-12T03:49:15.299 DEBUG:boto:Path: /dkyges-52/?versions&max-keys=999 2024-04-12T03:49:15.299 DEBUG:boto:Data: 2024-04-12T03:49:15.299 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.299 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:15.299 DEBUG:boto:Port: 8001 2024-04-12T03:49:15.299 DEBUG:boto:Params: {} 2024-04-12T03:49:15.300 DEBUG:boto:Token: None 2024-04-12T03:49:15.300 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/?versions 2024-04-12T03:49:15.300 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Te77EF0Las2Sic5e1+pab5V4AH0= 2024-04-12T03:49:15.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Te77EF0Las2Sic5e1+pab5V4AH0=', 'Content-Length': '0'} 2024-04-12T03:49:15.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026dd804714ac14af-006618af3b-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.302 DEBUG:boto:b'dkyges-52999falseMULTIPARTnulltrue2024-04-12T03:49:09.682Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:15.303 DEBUG:boto:path=/dkyges-52/ 2024-04-12T03:49:15.303 DEBUG:boto:auth_path=/dkyges-52/ 2024-04-12T03:49:15.303 DEBUG:boto:path=/dkyges-52/?versions&max-keys=999 2024-04-12T03:49:15.303 DEBUG:boto:auth_path=/dkyges-52/?versions&max-keys=999 2024-04-12T03:49:15.303 DEBUG:boto:Method: GET 2024-04-12T03:49:15.303 DEBUG:boto:Path: /dkyges-52/?versions&max-keys=999 2024-04-12T03:49:15.303 DEBUG:boto:Data: 2024-04-12T03:49:15.303 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.303 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:15.303 DEBUG:boto:Port: 8000 2024-04-12T03:49:15.303 DEBUG:boto:Params: {} 2024-04-12T03:49:15.304 DEBUG:boto:Token: None 2024-04-12T03:49:15.304 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/?versions 2024-04-12T03:49:15.304 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Te77EF0Las2Sic5e1+pab5V4AH0= 2024-04-12T03:49:15.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Te77EF0Las2Sic5e1+pab5V4AH0=', 'Content-Length': '0'} 2024-04-12T03:49:15.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003945313a7631e873-006618af3b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.306 DEBUG:boto:b'dkyges-52999falseMULTIPARTnulltrue2024-04-12T03:49:09.682Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:15.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-04-12T03:49:15.307 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.307 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.307 DEBUG:boto:path=/dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.307 DEBUG:boto:auth_path=/dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.307 DEBUG:boto:Method: GET 2024-04-12T03:49:15.307 DEBUG:boto:Path: /dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.307 DEBUG:boto:Data: 2024-04-12T03:49:15.307 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.307 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:15.307 DEBUG:boto:Port: 8001 2024-04-12T03:49:15.307 DEBUG:boto:Params: {} 2024-04-12T03:49:15.308 DEBUG:boto:Token: None 2024-04-12T03:49:15.308 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.308 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mnjFkBp73DmLMT2TTJIzcaCzakc= 2024-04-12T03:49:15.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mnjFkBp73DmLMT2TTJIzcaCzakc=', 'Content-Length': '0'} 2024-04-12T03:49:15.310 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-52:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000494570253676ab72-006618af3b-16de-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.383 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.383 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.383 DEBUG:boto:path=/dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.383 DEBUG:boto:auth_path=/dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.383 DEBUG:boto:Method: GET 2024-04-12T03:49:15.383 DEBUG:boto:Path: /dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.383 DEBUG:boto:Data: 2024-04-12T03:49:15.383 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.383 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:15.383 DEBUG:boto:Port: 8000 2024-04-12T03:49:15.384 DEBUG:boto:Params: {} 2024-04-12T03:49:15.384 DEBUG:boto:Token: None 2024-04-12T03:49:15.384 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.384 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mnjFkBp73DmLMT2TTJIzcaCzakc= 2024-04-12T03:49:15.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mnjFkBp73DmLMT2TTJIzcaCzakc=', 'Content-Length': '0'} 2024-04-12T03:49:15.390 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000903404ddcd8961a1-006618af3b-16d5-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.476 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.476 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.476 DEBUG:boto:path=/dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.476 DEBUG:boto:auth_path=/dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.476 DEBUG:boto:Method: HEAD 2024-04-12T03:49:15.477 DEBUG:boto:Path: /dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.477 DEBUG:boto:Data: 2024-04-12T03:49:15.477 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.477 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:15.477 DEBUG:boto:Port: 8001 2024-04-12T03:49:15.477 DEBUG:boto:Params: {} 2024-04-12T03:49:15.477 DEBUG:boto:Token: None 2024-04-12T03:49:15.477 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.477 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EQyd9CD9fOAK1W+sW6inAasUEDA= 2024-04-12T03:49:15.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EQyd9CD9fOAK1W+sW6inAasUEDA=', 'Content-Length': '0'} 2024-04-12T03:49:15.480 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-52:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000001be7eb51be7559fa-006618af3b-16de-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.480 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.480 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.480 DEBUG:boto:path=/dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.480 DEBUG:boto:auth_path=/dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.480 DEBUG:boto:Method: HEAD 2024-04-12T03:49:15.480 DEBUG:boto:Path: /dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.480 DEBUG:boto:Data: 2024-04-12T03:49:15.480 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.481 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:15.481 DEBUG:boto:Port: 8000 2024-04-12T03:49:15.481 DEBUG:boto:Params: {} 2024-04-12T03:49:15.481 DEBUG:boto:Token: None 2024-04-12T03:49:15.481 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.481 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EQyd9CD9fOAK1W+sW6inAasUEDA= 2024-04-12T03:49:15.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EQyd9CD9fOAK1W+sW6inAasUEDA=', 'Content-Length': '0'} 2024-04-12T03:49:15.489 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000007bb499912ccb9fb1-006618af3b-16d5-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-04-12T03:49:15.490 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.490 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.490 DEBUG:boto:path=/dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.490 DEBUG:boto:auth_path=/dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.490 DEBUG:boto:Method: GET 2024-04-12T03:49:15.490 DEBUG:boto:Path: /dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.490 DEBUG:boto:Data: 2024-04-12T03:49:15.490 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.490 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:15.490 DEBUG:boto:Port: 8001 2024-04-12T03:49:15.490 DEBUG:boto:Params: {} 2024-04-12T03:49:15.491 DEBUG:boto:Token: None 2024-04-12T03:49:15.491 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.491 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mnjFkBp73DmLMT2TTJIzcaCzakc= 2024-04-12T03:49:15.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mnjFkBp73DmLMT2TTJIzcaCzakc=', 'Content-Length': '0'} 2024-04-12T03:49:15.493 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-52:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000ea87b73d3e02f155-006618af3b-16de-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.595 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.595 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.595 DEBUG:boto:path=/dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.595 DEBUG:boto:auth_path=/dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.596 DEBUG:boto:Method: GET 2024-04-12T03:49:15.596 DEBUG:boto:Path: /dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.596 DEBUG:boto:Data: 2024-04-12T03:49:15.596 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.596 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:15.596 DEBUG:boto:Port: 8000 2024-04-12T03:49:15.596 DEBUG:boto:Params: {} 2024-04-12T03:49:15.596 DEBUG:boto:Token: None 2024-04-12T03:49:15.596 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/MULTIPART?versionId=null 2024-04-12T03:49:15.597 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mnjFkBp73DmLMT2TTJIzcaCzakc= 2024-04-12T03:49:15.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mnjFkBp73DmLMT2TTJIzcaCzakc=', 'Content-Length': '0'} 2024-04-12T03:49:15.600 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000009abc8080304150e8-006618af3b-16d5-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.693 DEBUG:boto:path=/dkyges-52/ 2024-04-12T03:49:15.693 DEBUG:boto:auth_path=/dkyges-52/ 2024-04-12T03:49:15.693 DEBUG:boto:path=/dkyges-52/?prefix=MULTIPART 2024-04-12T03:49:15.693 DEBUG:boto:auth_path=/dkyges-52/?prefix=MULTIPART 2024-04-12T03:49:15.693 DEBUG:boto:Method: GET 2024-04-12T03:49:15.693 DEBUG:boto:Path: /dkyges-52/?prefix=MULTIPART 2024-04-12T03:49:15.693 DEBUG:boto:Data: 2024-04-12T03:49:15.693 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.693 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:15.694 DEBUG:boto:Port: 8001 2024-04-12T03:49:15.694 DEBUG:boto:Params: {} 2024-04-12T03:49:15.694 DEBUG:boto:Token: None 2024-04-12T03:49:15.694 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/ 2024-04-12T03:49:15.694 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EvIt8BJpetnrBm/PjiRm9UdsfoY= 2024-04-12T03:49:15.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EvIt8BJpetnrBm/PjiRm9UdsfoY=', 'Content-Length': '0'} 2024-04-12T03:49:15.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015ba47279a421fda-006618af3b-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.697 DEBUG:boto:b'dkyges-52MULTIPART1000falseMULTIPART2024-04-12T03:49:09.682Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:15.698 DEBUG:boto:path=/dkyges-52/ 2024-04-12T03:49:15.698 DEBUG:boto:auth_path=/dkyges-52/ 2024-04-12T03:49:15.698 DEBUG:boto:path=/dkyges-52/?prefix=MULTIPART 2024-04-12T03:49:15.698 DEBUG:boto:auth_path=/dkyges-52/?prefix=MULTIPART 2024-04-12T03:49:15.698 DEBUG:boto:Method: GET 2024-04-12T03:49:15.698 DEBUG:boto:Path: /dkyges-52/?prefix=MULTIPART 2024-04-12T03:49:15.698 DEBUG:boto:Data: 2024-04-12T03:49:15.698 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.698 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:15.699 DEBUG:boto:Port: 8000 2024-04-12T03:49:15.699 DEBUG:boto:Params: {} 2024-04-12T03:49:15.699 DEBUG:boto:Token: None 2024-04-12T03:49:15.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/ 2024-04-12T03:49:15.699 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EvIt8BJpetnrBm/PjiRm9UdsfoY= 2024-04-12T03:49:15.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EvIt8BJpetnrBm/PjiRm9UdsfoY=', 'Content-Length': '0'} 2024-04-12T03:49:15.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c510b6ad0452afa2-006618af3b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.702 DEBUG:boto:b'dkyges-52MULTIPART1000falseMULTIPART2024-04-12T03:49:09.682Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:15.702 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.702 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.702 DEBUG:boto:Method: HEAD 2024-04-12T03:49:15.702 DEBUG:boto:Path: /dkyges-52/MULTIPART 2024-04-12T03:49:15.702 DEBUG:boto:Data: 2024-04-12T03:49:15.702 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.703 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:15.703 DEBUG:boto:Port: 8001 2024-04-12T03:49:15.703 DEBUG:boto:Params: {} 2024-04-12T03:49:15.703 DEBUG:boto:Token: None 2024-04-12T03:49:15.703 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/MULTIPART 2024-04-12T03:49:15.703 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4sjSKKhFxDEov8TslqojaS4XDYU= 2024-04-12T03:49:15.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4sjSKKhFxDEov8TslqojaS4XDYU=', 'Content-Length': '0'} 2024-04-12T03:49:15.705 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-52:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000b7814757b2e370db-006618af3b-16de-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.705 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.705 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.705 DEBUG:boto:Method: HEAD 2024-04-12T03:49:15.705 DEBUG:boto:Path: /dkyges-52/MULTIPART 2024-04-12T03:49:15.706 DEBUG:boto:Data: 2024-04-12T03:49:15.706 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.706 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:15.706 DEBUG:boto:Port: 8000 2024-04-12T03:49:15.706 DEBUG:boto:Params: {} 2024-04-12T03:49:15.706 DEBUG:boto:Token: None 2024-04-12T03:49:15.706 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/MULTIPART 2024-04-12T03:49:15.706 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4sjSKKhFxDEov8TslqojaS4XDYU= 2024-04-12T03:49:15.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4sjSKKhFxDEov8TslqojaS4XDYU=', 'Content-Length': '0'} 2024-04-12T03:49:15.708 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000006c0f4b568ffac196-006618af3b-16d5-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2024-04-12T03:49:15.709 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.709 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.709 DEBUG:boto:Method: GET 2024-04-12T03:49:15.709 DEBUG:boto:Path: /dkyges-52/MULTIPART 2024-04-12T03:49:15.709 DEBUG:boto:Data: 2024-04-12T03:49:15.709 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.709 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:15.709 DEBUG:boto:Port: 8001 2024-04-12T03:49:15.710 DEBUG:boto:Params: {} 2024-04-12T03:49:15.710 DEBUG:boto:Token: None 2024-04-12T03:49:15.710 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/MULTIPART 2024-04-12T03:49:15.710 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KYdjxc6uOpWanmTyfj2aipGfYaY= 2024-04-12T03:49:15.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KYdjxc6uOpWanmTyfj2aipGfYaY=', 'Content-Length': '0'} 2024-04-12T03:49:15.712 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-52:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000a9b86cc7eefc8517-006618af3b-16de-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.803 DEBUG:boto:path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.804 DEBUG:boto:auth_path=/dkyges-52/MULTIPART 2024-04-12T03:49:15.804 DEBUG:boto:Method: GET 2024-04-12T03:49:15.804 DEBUG:boto:Path: /dkyges-52/MULTIPART 2024-04-12T03:49:15.804 DEBUG:boto:Data: 2024-04-12T03:49:15.804 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.804 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:15.804 DEBUG:boto:Port: 8000 2024-04-12T03:49:15.804 DEBUG:boto:Params: {} 2024-04-12T03:49:15.805 DEBUG:boto:Token: None 2024-04-12T03:49:15.805 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/MULTIPART 2024-04-12T03:49:15.805 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KYdjxc6uOpWanmTyfj2aipGfYaY= 2024-04-12T03:49:15.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KYdjxc6uOpWanmTyfj2aipGfYaY=', 'Content-Length': '0'} 2024-04-12T03:49:15.809 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000001a514affb4e10aa5-006618af3b-16d5-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.907 DEBUG:boto:path=/dkyges-52/ 2024-04-12T03:49:15.907 DEBUG:boto:auth_path=/dkyges-52/ 2024-04-12T03:49:15.907 DEBUG:boto:path=/dkyges-52/?prefix=MULTIPART 2024-04-12T03:49:15.907 DEBUG:boto:auth_path=/dkyges-52/?prefix=MULTIPART 2024-04-12T03:49:15.907 DEBUG:boto:Method: GET 2024-04-12T03:49:15.908 DEBUG:boto:Path: /dkyges-52/?prefix=MULTIPART 2024-04-12T03:49:15.908 DEBUG:boto:Data: 2024-04-12T03:49:15.908 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.908 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:15.908 DEBUG:boto:Port: 8001 2024-04-12T03:49:15.908 DEBUG:boto:Params: {} 2024-04-12T03:49:15.908 DEBUG:boto:Token: None 2024-04-12T03:49:15.908 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/ 2024-04-12T03:49:15.908 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EvIt8BJpetnrBm/PjiRm9UdsfoY= 2024-04-12T03:49:15.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EvIt8BJpetnrBm/PjiRm9UdsfoY=', 'Content-Length': '0'} 2024-04-12T03:49:15.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf676a985f743ed0-006618af3b-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.911 DEBUG:boto:b'dkyges-52MULTIPART1000falseMULTIPART2024-04-12T03:49:09.682Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:15.911 DEBUG:boto:path=/dkyges-52/ 2024-04-12T03:49:15.912 DEBUG:boto:auth_path=/dkyges-52/ 2024-04-12T03:49:15.912 DEBUG:boto:path=/dkyges-52/?prefix=MULTIPART 2024-04-12T03:49:15.912 DEBUG:boto:auth_path=/dkyges-52/?prefix=MULTIPART 2024-04-12T03:49:15.912 DEBUG:boto:Method: GET 2024-04-12T03:49:15.912 DEBUG:boto:Path: /dkyges-52/?prefix=MULTIPART 2024-04-12T03:49:15.912 DEBUG:boto:Data: 2024-04-12T03:49:15.912 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.912 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:15.912 DEBUG:boto:Port: 8000 2024-04-12T03:49:15.912 DEBUG:boto:Params: {} 2024-04-12T03:49:15.913 DEBUG:boto:Token: None 2024-04-12T03:49:15.913 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:15 GMT /dkyges-52/ 2024-04-12T03:49:15.913 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EvIt8BJpetnrBm/PjiRm9UdsfoY= 2024-04-12T03:49:15.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EvIt8BJpetnrBm/PjiRm9UdsfoY=', 'Content-Length': '0'} 2024-04-12T03:49:15.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee53babf3cc77ece-006618af3b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.915 DEBUG:boto:b'dkyges-52MULTIPART1000falseMULTIPART2024-04-12T03:49:09.682Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:15.916 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-52 zones={a2, a1} 2024-04-12T03:49:15.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2024-04-12T03:49:15.917 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-54 2024-04-12T03:49:15.917 DEBUG:boto:path=/dkyges-54/ 2024-04-12T03:49:15.917 DEBUG:boto:auth_path=/dkyges-54/ 2024-04-12T03:49:15.917 DEBUG:boto:Method: PUT 2024-04-12T03:49:15.917 DEBUG:boto:Path: /dkyges-54/ 2024-04-12T03:49:15.917 DEBUG:boto:Data: 2024-04-12T03:49:15.918 DEBUG:boto:Headers: {} 2024-04-12T03:49:15.918 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:15.918 DEBUG:boto:Port: 8000 2024-04-12T03:49:15.918 DEBUG:boto:Params: {} 2024-04-12T03:49:15.918 DEBUG:boto:Token: None 2024-04-12T03:49:15.918 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:49:15 GMT /dkyges-54/ 2024-04-12T03:49:15.918 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TTjqCu+iO4CBXLJcuYv3rz6nB64= 2024-04-12T03:49:15.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TTjqCu+iO4CBXLJcuYv3rz6nB64=', 'Content-Length': '0'} 2024-04-12T03:49:15.937 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009ab69162a52e19db-006618af3b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.937 DEBUG:boto:path=/dkyges-54/testobj-sse-c 2024-04-12T03:49:15.937 DEBUG:boto:auth_path=/dkyges-54/testobj-sse-c 2024-04-12T03:49:15.938 DEBUG:boto:Method: PUT 2024-04-12T03:49:15.938 DEBUG:boto:Path: /dkyges-54/testobj-sse-c 2024-04-12T03:49:15.938 DEBUG:boto:Data: 2024-04-12T03:49:15.938 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-12T03:49:15.938 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:15.938 DEBUG:boto:Port: 8000 2024-04-12T03:49:15.938 DEBUG:boto:Params: {} 2024-04-12T03:49:15.938 DEBUG:boto:Token: None 2024-04-12T03:49:15.938 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 12 Apr 2024 03:49:15 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /dkyges-54/testobj-sse-c 2024-04-12T03:49:15.939 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rBPMjZkVgzBUThAhrRqdd+uL2rM= 2024-04-12T03:49:15.939 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rBPMjZkVgzBUThAhrRqdd+uL2rM='} 2024-04-12T03:49:15.983 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx00000a27e0612758bab10-006618af3b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.983 DEBUG:boto:path=/dkyges-54/testobj-sse-kms 2024-04-12T03:49:15.983 DEBUG:boto:auth_path=/dkyges-54/testobj-sse-kms 2024-04-12T03:49:15.983 DEBUG:boto:Method: PUT 2024-04-12T03:49:15.983 DEBUG:boto:Path: /dkyges-54/testobj-sse-kms 2024-04-12T03:49:15.983 DEBUG:boto:Data: 2024-04-12T03:49:15.984 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2024-04-12T03:49:15.984 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:15.984 DEBUG:boto:Port: 8000 2024-04-12T03:49:15.984 DEBUG:boto:Params: {} 2024-04-12T03:49:15.984 DEBUG:boto:Token: None 2024-04-12T03:49:15.984 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 12 Apr 2024 03:49:15 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /dkyges-54/testobj-sse-kms 2024-04-12T03:49:15.984 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:622J1symOTQpef5/VuCrMHDlZKE= 2024-04-12T03:49:15.984 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:49:15 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:622J1symOTQpef5/VuCrMHDlZKE='} 2024-04-12T03:49:15.991 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6ZGt5Z2VzLTU0L3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx000000de64480d99e2032-006618af3b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:15.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:15.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:15.992 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:49:16.050 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:16.050 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:16.086 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:49:16.101 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:49:16.101 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:16.101 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:49:16.101 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:49:16.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:16.102 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:16.102 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:49:16.102 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:49:16.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:16.102 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:16.102 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:49:16.102 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:49:16.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:16.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:16.103 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893755.940791_479.1", 2024-04-12T03:49:16.103 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:49:15.940791Z" 2024-04-12T03:49:16.103 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:16.103 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:49:16.104 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893755.940791_479.1'} 2024-04-12T03:49:16.104 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:49:16.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:16.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:16.105 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:49:16.159 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:16.160 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:16.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:16.199+0000 7ff1eeafb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:49:16.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:16.200+0000 7ff1eeafb780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:49:16.210 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:49:16.211 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:49:16.211 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:49:16.211 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:49:16.211 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:49:16.211 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:49:16.211 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:16.211 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:16.211 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:49:16.211 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:16.211 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:49:16.211 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:16.212 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:16.212 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:49:16.212 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:16.212 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:49:16.212 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:16.212 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:49:16.212 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:16.212 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:16.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:16.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:16.212 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:49:16.213 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:16.213 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:16.213 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:49:16.213 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:16.213 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:16.213 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:16.213 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:49:16.213 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:16.213 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:16.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:16.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:16.213 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:49:16.214 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:16.214 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:16.214 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:49:16.214 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:16.214 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:16.214 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:16.214 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:49:16.214 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:16.214 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:16.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:16.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:16.215 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:49:16.215 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:16.215 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:16.215 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893755.932943_476.1", 2024-04-12T03:49:16.215 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:16.215 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:16.215 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:16.215 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:49:15.932943Z", 2024-04-12T03:49:16.215 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:16.215 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:16.215 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:16.216 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:49:16.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:16.216 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:49:16.216 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:49:16.216 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:49:16.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:16.216 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:49:16Z" 2024-04-12T03:49:16.216 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:49:16.218 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893755.932943_476.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:49:15.932943Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:49:16Z" } 2024-04-12T03:49:16.218 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893755.932943_476.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:49:15.932943Z', 'realm_epoch': 2}}] 2024-04-12T03:49:16.218 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893755.940791_479.1'} 2024-04-12T03:49:16.218 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893755.932943_476.1'} 2024-04-12T03:49:16.218 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893755.940791_479.1 target=1_1712893755.932943_476.1 2024-04-12T03:49:21.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:21.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:21.220 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:49:21.281 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:21.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:21.323+0000 7efdc8297780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:49:21.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:21.324+0000 7efdc8297780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:49:21.338 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:49:21.339 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:49:21.339 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:49:21.339 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:49:21.339 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:49:21.339 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:49:21.339 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:21.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:21.339 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:49:21.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:21.339 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:49:21.339 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:21.340 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:21.340 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:49:21.340 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:21.340 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:49:21.340 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:21.340 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:49:21.340 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:21.340 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:21.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:21.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:21.340 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:49:21.340 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:21.341 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:21.341 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:49:21.341 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:21.341 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:21.341 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:21.341 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:49:21.341 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:21.341 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:21.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:21.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:21.341 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:49:21.342 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:21.342 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:21.342 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:49:21.342 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:21.342 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:21.342 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:21.342 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:49:21.342 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:21.342 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:21.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:21.342 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:21.343 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:49:21.343 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:21.343 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:21.343 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893755.940791_479.1", 2024-04-12T03:49:21.343 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:21.343 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:21.343 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:21.343 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:49:15.940791Z", 2024-04-12T03:49:21.343 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:21.343 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:21.343 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:21.343 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:49:21.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:21.344 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:49:21.344 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:49:21.344 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:49:21.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:21.344 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:49:21Z" 2024-04-12T03:49:21.344 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:49:21.344 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893755.940791_479.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:49:15.940791Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:49:21Z" } 2024-04-12T03:49:21.344 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893755.940791_479.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:49:15.940791Z', 'realm_epoch': 2}}] 2024-04-12T03:49:21.345 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893755.940791_479.1'} 2024-04-12T03:49:21.345 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893755.940791_479.1'} 2024-04-12T03:49:21.345 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:49:21.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:49:21.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:49:21.345 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-54 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:49:21.408 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:21.408 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:21.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:21.454+0000 7f84e2132780 1 waiting to reach incremental sync.. 2024-04-12T03:49:26.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:26.455+0000 7f84e2132780 1 waiting to reach incremental sync.. 2024-04-12T03:49:31.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:31.455+0000 7f84e2132780 1 waiting to reach incremental sync.. 2024-04-12T03:49:36.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:36.457+0000 7f84e2132780 1 bucket sync caught up with source: 2024-04-12T03:49:36.452 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.32.6, , , , 00000000001.29.6, , , , , ] 2024-04-12T03:49:36.452 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.32.6, , , , 00000000001.29.6, , , , , ] 2024-04-12T03:49:36.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:36.457+0000 7f84e2132780 0 bucket checkpoint complete 2024-04-12T03:49:36.472 DEBUG:boto:path=/dkyges-54/ 2024-04-12T03:49:36.472 DEBUG:boto:auth_path=/dkyges-54/ 2024-04-12T03:49:36.472 DEBUG:boto:Method: HEAD 2024-04-12T03:49:36.472 DEBUG:boto:Path: /dkyges-54/ 2024-04-12T03:49:36.472 DEBUG:boto:Data: 2024-04-12T03:49:36.472 DEBUG:boto:Headers: {} 2024-04-12T03:49:36.472 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:36.473 DEBUG:boto:Port: 8001 2024-04-12T03:49:36.473 DEBUG:boto:Params: {} 2024-04-12T03:49:36.473 DEBUG:boto:Token: None 2024-04-12T03:49:36.473 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:36 GMT /dkyges-54/ 2024-04-12T03:49:36.473 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1Ljcy17kiZogsuWs4UOLH4jQx78= 2024-04-12T03:49:36.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1Ljcy17kiZogsuWs4UOLH4jQx78=', 'Content-Length': '0'} 2024-04-12T03:49:36.476 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '1024'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be7c901dcdc3d340-006618af50-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:36.476 DEBUG:boto:path=/dkyges-54/testobj-sse-c 2024-04-12T03:49:36.476 DEBUG:boto:auth_path=/dkyges-54/testobj-sse-c 2024-04-12T03:49:36.477 DEBUG:boto:Method: HEAD 2024-04-12T03:49:36.477 DEBUG:boto:Path: /dkyges-54/testobj-sse-c 2024-04-12T03:49:36.477 DEBUG:boto:Data: 2024-04-12T03:49:36.477 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2024-04-12T03:49:36.477 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:36.477 DEBUG:boto:Port: 8001 2024-04-12T03:49:36.477 DEBUG:boto:Params: {} 2024-04-12T03:49:36.477 DEBUG:boto:Token: None 2024-04-12T03:49:36.477 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:36 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /dkyges-54/testobj-sse-c 2024-04-12T03:49:36.477 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ig7Vt4JrBs8lKLA3SWV1WSfaOOw= 2024-04-12T03:49:36.478 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ig7Vt4JrBs8lKLA3SWV1WSfaOOw=', 'Content-Length': '0'} 2024-04-12T03:49:36.479 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-54:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000000a0356179a053536-006618af50-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:36.480 DEBUG:boto:path=/dkyges-54/testobj-sse-c 2024-04-12T03:49:36.480 DEBUG:boto:auth_path=/dkyges-54/testobj-sse-c 2024-04-12T03:49:36.480 DEBUG:boto:Method: GET 2024-04-12T03:49:36.480 DEBUG:boto:Path: /dkyges-54/testobj-sse-c 2024-04-12T03:49:36.480 DEBUG:boto:Data: 2024-04-12T03:49:36.480 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2024-04-12T03:49:36.480 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:36.480 DEBUG:boto:Port: 8001 2024-04-12T03:49:36.480 DEBUG:boto:Params: {} 2024-04-12T03:49:36.480 DEBUG:boto:Token: None 2024-04-12T03:49:36.481 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:36 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /dkyges-54/testobj-sse-c 2024-04-12T03:49:36.481 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4XLuwoeJYbkB9QEsmOmKyL4HcAk= 2024-04-12T03:49:36.481 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4XLuwoeJYbkB9QEsmOmKyL4HcAk=', 'Content-Length': '0'} 2024-04-12T03:49:36.483 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-54:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000004db222181cf1793a-006618af50-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:36.526 DEBUG:boto:path=/dkyges-54/testobj-sse-kms 2024-04-12T03:49:36.526 DEBUG:boto:auth_path=/dkyges-54/testobj-sse-kms 2024-04-12T03:49:36.527 DEBUG:boto:Method: HEAD 2024-04-12T03:49:36.527 DEBUG:boto:Path: /dkyges-54/testobj-sse-kms 2024-04-12T03:49:36.527 DEBUG:boto:Data: 2024-04-12T03:49:36.527 DEBUG:boto:Headers: {} 2024-04-12T03:49:36.527 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:36.527 DEBUG:boto:Port: 8001 2024-04-12T03:49:36.527 DEBUG:boto:Params: {} 2024-04-12T03:49:36.527 DEBUG:boto:Token: None 2024-04-12T03:49:36.527 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:36 GMT /dkyges-54/testobj-sse-kms 2024-04-12T03:49:36.528 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9EUmbvoDAzqu5/GUZamHe1AINL8= 2024-04-12T03:49:36.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9EUmbvoDAzqu5/GUZamHe1AINL8=', 'Content-Length': '0'} 2024-04-12T03:49:36.530 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-54:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000c0d4431cdbb65204-006618af50-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:36.530 DEBUG:boto:path=/dkyges-54/testobj-sse-kms 2024-04-12T03:49:36.530 DEBUG:boto:auth_path=/dkyges-54/testobj-sse-kms 2024-04-12T03:49:36.530 DEBUG:boto:Method: GET 2024-04-12T03:49:36.530 DEBUG:boto:Path: /dkyges-54/testobj-sse-kms 2024-04-12T03:49:36.530 DEBUG:boto:Data: 2024-04-12T03:49:36.530 DEBUG:boto:Headers: {} 2024-04-12T03:49:36.530 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:36.530 DEBUG:boto:Port: 8001 2024-04-12T03:49:36.531 DEBUG:boto:Params: {} 2024-04-12T03:49:36.531 DEBUG:boto:Token: None 2024-04-12T03:49:36.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:36 GMT /dkyges-54/testobj-sse-kms 2024-04-12T03:49:36.531 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:twzERHr/he5XaQkJ2cNSwxlWFUc= 2024-04-12T03:49:36.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:twzERHr/he5XaQkJ2cNSwxlWFUc=', 'Content-Length': '0'} 2024-04-12T03:49:36.533 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-54:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000481455a8bdb74cbc-006618af50-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:36.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ok 2024-04-12T03:49:36.534 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-55 2024-04-12T03:49:36.534 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:36.534 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:36.535 DEBUG:boto:Method: PUT 2024-04-12T03:49:36.535 DEBUG:boto:Path: /dkyges-55/ 2024-04-12T03:49:36.535 DEBUG:boto:Data: 2024-04-12T03:49:36.535 DEBUG:boto:Headers: {} 2024-04-12T03:49:36.535 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:36.535 DEBUG:boto:Port: 8000 2024-04-12T03:49:36.535 DEBUG:boto:Params: {} 2024-04-12T03:49:36.535 DEBUG:boto:Token: None 2024-04-12T03:49:36.535 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:49:36 GMT /dkyges-55/ 2024-04-12T03:49:36.535 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZQdFRhqPjW3EoktoyJRHIRJ+Bt4= 2024-04-12T03:49:36.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZQdFRhqPjW3EoktoyJRHIRJ+Bt4=', 'Content-Length': '0'} 2024-04-12T03:49:36.555 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000781f53e1c0fca8c9-006618af50-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:36.555 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:36.555 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:36.555 DEBUG:boto:Method: HEAD 2024-04-12T03:49:36.555 DEBUG:boto:Path: /dkyges-55/ 2024-04-12T03:49:36.556 DEBUG:boto:Data: 2024-04-12T03:49:36.556 DEBUG:boto:Headers: {} 2024-04-12T03:49:36.556 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:36.556 DEBUG:boto:Port: 8000 2024-04-12T03:49:36.556 DEBUG:boto:Params: {} 2024-04-12T03:49:36.556 DEBUG:boto:Token: None 2024-04-12T03:49:36.556 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:36 GMT /dkyges-55/ 2024-04-12T03:49:36.556 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3izlxt3goROUn9hIGUzpj5FyCoc= 2024-04-12T03:49:36.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3izlxt3goROUn9hIGUzpj5FyCoc=', 'Content-Length': '0'} 2024-04-12T03:49:36.559 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001c73d39f7ec6297-006618af50-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:36.559 DEBUG:boto:path=/dkyges-55/a 2024-04-12T03:49:36.559 DEBUG:boto:auth_path=/dkyges-55/a 2024-04-12T03:49:36.559 DEBUG:boto:Method: PUT 2024-04-12T03:49:36.559 DEBUG:boto:Path: /dkyges-55/a 2024-04-12T03:49:36.560 DEBUG:boto:Data: 2024-04-12T03:49:36.560 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:49:36.560 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:36.560 DEBUG:boto:Port: 8000 2024-04-12T03:49:36.560 DEBUG:boto:Params: {} 2024-04-12T03:49:36.560 DEBUG:boto:Token: None 2024-04-12T03:49:36.560 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:49:36 GMT /dkyges-55/a 2024-04-12T03:49:36.560 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1/FrQQCjuFfcK8J/mfjZYUfH7sc= 2024-04-12T03:49:36.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:49:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1/FrQQCjuFfcK8J/mfjZYUfH7sc='} 2024-04-12T03:49:36.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000865c098b6f7f7a2f-006618af50-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:36.603 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:36.603 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:36.603 DEBUG:boto:Method: HEAD 2024-04-12T03:49:36.603 DEBUG:boto:Path: /dkyges-55/ 2024-04-12T03:49:36.603 DEBUG:boto:Data: 2024-04-12T03:49:36.603 DEBUG:boto:Headers: {} 2024-04-12T03:49:36.603 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:36.603 DEBUG:boto:Port: 8000 2024-04-12T03:49:36.603 DEBUG:boto:Params: {} 2024-04-12T03:49:36.603 DEBUG:boto:Token: None 2024-04-12T03:49:36.604 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:36 GMT /dkyges-55/ 2024-04-12T03:49:36.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3izlxt3goROUn9hIGUzpj5FyCoc= 2024-04-12T03:49:36.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3izlxt3goROUn9hIGUzpj5FyCoc=', 'Content-Length': '0'} 2024-04-12T03:49:36.606 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8da6544b0f76657-006618af50-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:36.606 DEBUG:boto:path=/dkyges-55/b 2024-04-12T03:49:36.606 DEBUG:boto:auth_path=/dkyges-55/b 2024-04-12T03:49:36.607 DEBUG:boto:Method: PUT 2024-04-12T03:49:36.607 DEBUG:boto:Path: /dkyges-55/b 2024-04-12T03:49:36.607 DEBUG:boto:Data: 2024-04-12T03:49:36.607 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:49:36.607 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:36.607 DEBUG:boto:Port: 8000 2024-04-12T03:49:36.607 DEBUG:boto:Params: {} 2024-04-12T03:49:36.607 DEBUG:boto:Token: None 2024-04-12T03:49:36.607 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:49:36 GMT /dkyges-55/b 2024-04-12T03:49:36.608 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xfrXziAJehF0a6EFVjNDOfytzm8= 2024-04-12T03:49:36.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:49:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xfrXziAJehF0a6EFVjNDOfytzm8='} 2024-04-12T03:49:36.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006a149b9235926341-006618af50-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:36.614 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:36.614 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:36.614 DEBUG:boto:Method: HEAD 2024-04-12T03:49:36.615 DEBUG:boto:Path: /dkyges-55/ 2024-04-12T03:49:36.615 DEBUG:boto:Data: 2024-04-12T03:49:36.615 DEBUG:boto:Headers: {} 2024-04-12T03:49:36.615 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:36.615 DEBUG:boto:Port: 8000 2024-04-12T03:49:36.615 DEBUG:boto:Params: {} 2024-04-12T03:49:36.615 DEBUG:boto:Token: None 2024-04-12T03:49:36.615 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:36 GMT /dkyges-55/ 2024-04-12T03:49:36.615 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3izlxt3goROUn9hIGUzpj5FyCoc= 2024-04-12T03:49:36.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3izlxt3goROUn9hIGUzpj5FyCoc=', 'Content-Length': '0'} 2024-04-12T03:49:36.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042df542a9ca15ef9-006618af50-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:36.618 DEBUG:boto:path=/dkyges-55/c 2024-04-12T03:49:36.618 DEBUG:boto:auth_path=/dkyges-55/c 2024-04-12T03:49:36.618 DEBUG:boto:Method: PUT 2024-04-12T03:49:36.618 DEBUG:boto:Path: /dkyges-55/c 2024-04-12T03:49:36.618 DEBUG:boto:Data: 2024-04-12T03:49:36.618 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:49:36.619 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:36.619 DEBUG:boto:Port: 8000 2024-04-12T03:49:36.619 DEBUG:boto:Params: {} 2024-04-12T03:49:36.619 DEBUG:boto:Token: None 2024-04-12T03:49:36.619 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:49:36 GMT /dkyges-55/c 2024-04-12T03:49:36.619 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MX/ZkcrXvIwR29dY33wSHFFmb2E= 2024-04-12T03:49:36.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:49:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MX/ZkcrXvIwR29dY33wSHFFmb2E='} 2024-04-12T03:49:36.625 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae3cfc3c92b69ee6-006618af50-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:36.625 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:36.625 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:36.626 DEBUG:boto:Method: HEAD 2024-04-12T03:49:36.626 DEBUG:boto:Path: /dkyges-55/ 2024-04-12T03:49:36.626 DEBUG:boto:Data: 2024-04-12T03:49:36.626 DEBUG:boto:Headers: {} 2024-04-12T03:49:36.626 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:36.626 DEBUG:boto:Port: 8000 2024-04-12T03:49:36.626 DEBUG:boto:Params: {} 2024-04-12T03:49:36.626 DEBUG:boto:Token: None 2024-04-12T03:49:36.626 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:36 GMT /dkyges-55/ 2024-04-12T03:49:36.627 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3izlxt3goROUn9hIGUzpj5FyCoc= 2024-04-12T03:49:36.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3izlxt3goROUn9hIGUzpj5FyCoc=', 'Content-Length': '0'} 2024-04-12T03:49:36.629 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095aa55f86d3d6864-006618af50-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:36.629 DEBUG:boto:path=/dkyges-55/d 2024-04-12T03:49:36.629 DEBUG:boto:auth_path=/dkyges-55/d 2024-04-12T03:49:36.629 DEBUG:boto:Method: PUT 2024-04-12T03:49:36.629 DEBUG:boto:Path: /dkyges-55/d 2024-04-12T03:49:36.629 DEBUG:boto:Data: 2024-04-12T03:49:36.630 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:49:36.630 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:36.630 DEBUG:boto:Port: 8000 2024-04-12T03:49:36.630 DEBUG:boto:Params: {} 2024-04-12T03:49:36.630 DEBUG:boto:Token: None 2024-04-12T03:49:36.630 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:49:36 GMT /dkyges-55/d 2024-04-12T03:49:36.630 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vbf1to8TtYeZm/2omXnHrZf81/c= 2024-04-12T03:49:36.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:49:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vbf1to8TtYeZm/2omXnHrZf81/c='} 2024-04-12T03:49:36.674 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d523def2f78ec76-006618af50-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:36.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:36.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:36.675 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:49:36.735 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:36.735 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:36.772 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:49:36.792 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:49:36.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:36.792 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:49:36.792 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:49:36.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:36.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:36.793 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:49:36.793 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:49:36.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:36.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:36.793 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:49:36.793 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:49:36.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:36.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:36.793 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893776.559026_483.1", 2024-04-12T03:49:36.793 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:49:36.559026Z" 2024-04-12T03:49:36.794 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:36.794 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:49:36.795 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893776.559026_483.1'} 2024-04-12T03:49:36.795 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:49:36.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:36.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:36.795 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:49:36.850 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:36.850 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:36.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:36.891+0000 7f256c8a1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:49:36.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:36.892+0000 7f256c8a1780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:49:36.903 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:49:36.903 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:49:36.903 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:49:36.903 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:49:36.903 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:49:36.903 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:49:36.903 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:36.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:36.903 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:49:36.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:36.904 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:49:36.904 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:36.904 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:36.904 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:49:36.904 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:36.904 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:49:36.904 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:36.904 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:49:36.904 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:36.904 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:36.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:36.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:36.905 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:49:36.905 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:36.905 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:36.905 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:49:36.905 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:36.905 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:36.905 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:36.905 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:49:36.905 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:36.906 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:36.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:36.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:36.906 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:49:36.906 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:36.906 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:36.906 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:49:36.906 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:36.906 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:36.906 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:36.906 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:49:36.907 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:36.907 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:36.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:36.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:36.907 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:49:36.907 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:36.907 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:36.907 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893776.559026_483.1", 2024-04-12T03:49:36.907 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:36.907 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:36.907 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:36.907 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:49:36.559026Z", 2024-04-12T03:49:36.908 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:36.908 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:36.908 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:36.908 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:49:36.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:36.908 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:49:36.908 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:49:36.908 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:49:36.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:36.908 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:49:36Z" 2024-04-12T03:49:36.908 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:49:36.909 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893776.559026_483.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:49:36.559026Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:49:36Z" } 2024-04-12T03:49:36.909 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893776.559026_483.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:49:36.559026Z', 'realm_epoch': 2}}] 2024-04-12T03:49:36.910 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893776.559026_483.1'} 2024-04-12T03:49:36.910 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893776.559026_483.1'} 2024-04-12T03:49:36.910 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:49:36.910 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-55 2024-04-12T03:49:36.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:49:36.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:49:36.910 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:49:36.972 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:36.972 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:37.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:37.017+0000 7fb50ba02780 1 waiting to reach incremental sync.. 2024-04-12T03:49:42.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:42.018+0000 7fb50ba02780 1 waiting to reach incremental sync.. 2024-04-12T03:49:47.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:47.018+0000 7fb50ba02780 1 waiting to reach incremental sync.. 2024-04-12T03:49:52.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:52.019+0000 7fb50ba02780 1 waiting to reach incremental sync.. 2024-04-12T03:49:57.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:57.020+0000 7fb50ba02780 1 bucket sync caught up with source: 2024-04-12T03:49:57.015 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.36.6, , , , 00000000001.116.6, , , 00000000001.21.6, , 00000000001.114.6] 2024-04-12T03:49:57.016 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.36.6, , , , 00000000001.116.6, , , 00000000001.21.6, , 00000000001.114.6] 2024-04-12T03:49:57.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:57.020+0000 7fb50ba02780 0 bucket checkpoint complete 2024-04-12T03:49:57.036 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-55 2024-04-12T03:49:57.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:49:57.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:49:57.036 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:49:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:58.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:58.156+0000 7f4b8562a780 1 bucket sync caught up with source: 2024-04-12T03:49:58.151 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.1901.6, , , , 00000000001.2748.6, , , 00000000001.1391.6, , 00000000001.2747.6] 2024-04-12T03:49:58.151 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.1901.6, , , , 00000000001.2748.6, , , 00000000001.1391.6, , 00000000001.2747.6] 2024-04-12T03:49:58.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:58.156+0000 7f4b8562a780 0 bucket checkpoint complete 2024-04-12T03:49:58.168 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-55 zones={a2, a1} 2024-04-12T03:49:58.169 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.169 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.169 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.169 DEBUG:boto:Path: /dkyges-55/ 2024-04-12T03:49:58.169 DEBUG:boto:Data: 2024-04-12T03:49:58.169 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.169 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.169 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.169 DEBUG:boto:Params: {} 2024-04-12T03:49:58.169 DEBUG:boto:Token: None 2024-04-12T03:49:58.170 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.170 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mI0BwL3mP/qkcvr1FWOpsywEwrc= 2024-04-12T03:49:58.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mI0BwL3mP/qkcvr1FWOpsywEwrc=', 'Content-Length': '0'} 2024-04-12T03:49:58.173 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000641bba3c20523845-006618af66-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.173 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.173 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.173 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.173 DEBUG:boto:Path: /dkyges-55/ 2024-04-12T03:49:58.173 DEBUG:boto:Data: 2024-04-12T03:49:58.173 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.173 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.173 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.173 DEBUG:boto:Params: {} 2024-04-12T03:49:58.174 DEBUG:boto:Token: None 2024-04-12T03:49:58.174 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.174 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mI0BwL3mP/qkcvr1FWOpsywEwrc= 2024-04-12T03:49:58.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mI0BwL3mP/qkcvr1FWOpsywEwrc=', 'Content-Length': '0'} 2024-04-12T03:49:58.177 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000091c85c43ab239716-006618af66-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.177 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:49:58.177 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.177 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.177 DEBUG:boto:path=/dkyges-55/?versions&max-keys=999 2024-04-12T03:49:58.177 DEBUG:boto:auth_path=/dkyges-55/?versions&max-keys=999 2024-04-12T03:49:58.177 DEBUG:boto:Method: GET 2024-04-12T03:49:58.177 DEBUG:boto:Path: /dkyges-55/?versions&max-keys=999 2024-04-12T03:49:58.178 DEBUG:boto:Data: 2024-04-12T03:49:58.178 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.178 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.178 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.178 DEBUG:boto:Params: {} 2024-04-12T03:49:58.178 DEBUG:boto:Token: None 2024-04-12T03:49:58.178 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/?versions 2024-04-12T03:49:58.178 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CRVnCyWiinwiKuIzDvwHB9MSB6I= 2024-04-12T03:49:58.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CRVnCyWiinwiKuIzDvwHB9MSB6I=', 'Content-Length': '0'} 2024-04-12T03:49:58.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000783bffa9752d6711-006618af66-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.181 DEBUG:boto:b'dkyges-55999falseanulltrue2024-04-12T03:49:36.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:49:36.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:49:36.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:49:36.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.182 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:49:58.182 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:49:58.182 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:49:58.183 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:49:58.183 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:49:58.183 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.183 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.183 DEBUG:boto:path=/dkyges-55/?versions&max-keys=999 2024-04-12T03:49:58.183 DEBUG:boto:auth_path=/dkyges-55/?versions&max-keys=999 2024-04-12T03:49:58.183 DEBUG:boto:Method: GET 2024-04-12T03:49:58.183 DEBUG:boto:Path: /dkyges-55/?versions&max-keys=999 2024-04-12T03:49:58.183 DEBUG:boto:Data: 2024-04-12T03:49:58.183 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.184 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.184 DEBUG:boto:Params: {} 2024-04-12T03:49:58.184 DEBUG:boto:Token: None 2024-04-12T03:49:58.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/?versions 2024-04-12T03:49:58.184 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CRVnCyWiinwiKuIzDvwHB9MSB6I= 2024-04-12T03:49:58.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CRVnCyWiinwiKuIzDvwHB9MSB6I=', 'Content-Length': '0'} 2024-04-12T03:49:58.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072df468c75a6082f-006618af66-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.187 DEBUG:boto:b'dkyges-55999falseanulltrue2024-04-12T03:49:36.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:49:36.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:49:36.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:49:36.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.187 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:49:58.187 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:49:58.187 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:49:58.187 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:49:58.188 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.188 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.188 DEBUG:boto:path=/dkyges-55/?versions&max-keys=999 2024-04-12T03:49:58.188 DEBUG:boto:auth_path=/dkyges-55/?versions&max-keys=999 2024-04-12T03:49:58.188 DEBUG:boto:Method: GET 2024-04-12T03:49:58.188 DEBUG:boto:Path: /dkyges-55/?versions&max-keys=999 2024-04-12T03:49:58.188 DEBUG:boto:Data: 2024-04-12T03:49:58.188 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.188 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.188 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.188 DEBUG:boto:Params: {} 2024-04-12T03:49:58.189 DEBUG:boto:Token: None 2024-04-12T03:49:58.189 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/?versions 2024-04-12T03:49:58.189 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CRVnCyWiinwiKuIzDvwHB9MSB6I= 2024-04-12T03:49:58.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CRVnCyWiinwiKuIzDvwHB9MSB6I=', 'Content-Length': '0'} 2024-04-12T03:49:58.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fe27bbf7ccb15cc-006618af66-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.191 DEBUG:boto:b'dkyges-55999falseanulltrue2024-04-12T03:49:36.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:49:36.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:49:36.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:49:36.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.192 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.192 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.192 DEBUG:boto:path=/dkyges-55/?versions&max-keys=999 2024-04-12T03:49:58.192 DEBUG:boto:auth_path=/dkyges-55/?versions&max-keys=999 2024-04-12T03:49:58.192 DEBUG:boto:Method: GET 2024-04-12T03:49:58.192 DEBUG:boto:Path: /dkyges-55/?versions&max-keys=999 2024-04-12T03:49:58.192 DEBUG:boto:Data: 2024-04-12T03:49:58.192 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.192 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.193 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.193 DEBUG:boto:Params: {} 2024-04-12T03:49:58.193 DEBUG:boto:Token: None 2024-04-12T03:49:58.193 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/?versions 2024-04-12T03:49:58.193 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CRVnCyWiinwiKuIzDvwHB9MSB6I= 2024-04-12T03:49:58.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CRVnCyWiinwiKuIzDvwHB9MSB6I=', 'Content-Length': '0'} 2024-04-12T03:49:58.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e65d710267e5f92-006618af66-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.195 DEBUG:boto:b'dkyges-55999falseanulltrue2024-04-12T03:49:36.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:49:36.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:49:36.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:49:36.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:49:58.196 DEBUG:boto:path=/dkyges-55/a 2024-04-12T03:49:58.196 DEBUG:boto:auth_path=/dkyges-55/a 2024-04-12T03:49:58.196 DEBUG:boto:path=/dkyges-55/a?versionId=null 2024-04-12T03:49:58.196 DEBUG:boto:auth_path=/dkyges-55/a?versionId=null 2024-04-12T03:49:58.196 DEBUG:boto:Method: GET 2024-04-12T03:49:58.197 DEBUG:boto:Path: /dkyges-55/a?versionId=null 2024-04-12T03:49:58.197 DEBUG:boto:Data: 2024-04-12T03:49:58.197 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.197 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.197 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.197 DEBUG:boto:Params: {} 2024-04-12T03:49:58.197 DEBUG:boto:Token: None 2024-04-12T03:49:58.197 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/a?versionId=null 2024-04-12T03:49:58.197 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJL2H4pfwSe6AM/6pAg47yD+qJs= 2024-04-12T03:49:58.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJL2H4pfwSe6AM/6pAg47yD+qJs=', 'Content-Length': '0'} 2024-04-12T03:49:58.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c13337d50e87600-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.199 DEBUG:boto:path=/dkyges-55/a 2024-04-12T03:49:58.200 DEBUG:boto:auth_path=/dkyges-55/a 2024-04-12T03:49:58.200 DEBUG:boto:path=/dkyges-55/a?versionId=null 2024-04-12T03:49:58.200 DEBUG:boto:auth_path=/dkyges-55/a?versionId=null 2024-04-12T03:49:58.200 DEBUG:boto:Method: GET 2024-04-12T03:49:58.200 DEBUG:boto:Path: /dkyges-55/a?versionId=null 2024-04-12T03:49:58.200 DEBUG:boto:Data: 2024-04-12T03:49:58.200 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.200 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.200 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.200 DEBUG:boto:Params: {} 2024-04-12T03:49:58.201 DEBUG:boto:Token: None 2024-04-12T03:49:58.201 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/a?versionId=null 2024-04-12T03:49:58.201 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJL2H4pfwSe6AM/6pAg47yD+qJs= 2024-04-12T03:49:58.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJL2H4pfwSe6AM/6pAg47yD+qJs=', 'Content-Length': '0'} 2024-04-12T03:49:58.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcc0a769a55e7171-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.203 DEBUG:boto:path=/dkyges-55/a 2024-04-12T03:49:58.203 DEBUG:boto:auth_path=/dkyges-55/a 2024-04-12T03:49:58.203 DEBUG:boto:path=/dkyges-55/a?versionId=null 2024-04-12T03:49:58.203 DEBUG:boto:auth_path=/dkyges-55/a?versionId=null 2024-04-12T03:49:58.203 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.203 DEBUG:boto:Path: /dkyges-55/a?versionId=null 2024-04-12T03:49:58.204 DEBUG:boto:Data: 2024-04-12T03:49:58.204 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.204 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.204 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.204 DEBUG:boto:Params: {} 2024-04-12T03:49:58.204 DEBUG:boto:Token: None 2024-04-12T03:49:58.204 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/a?versionId=null 2024-04-12T03:49:58.204 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UsGrhGwLUYLj+6Lpzmugg2WV0yY= 2024-04-12T03:49:58.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UsGrhGwLUYLj+6Lpzmugg2WV0yY=', 'Content-Length': '0'} 2024-04-12T03:49:58.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059ba40683ecd7a57-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.206 DEBUG:boto:path=/dkyges-55/a 2024-04-12T03:49:58.207 DEBUG:boto:auth_path=/dkyges-55/a 2024-04-12T03:49:58.207 DEBUG:boto:path=/dkyges-55/a?versionId=null 2024-04-12T03:49:58.207 DEBUG:boto:auth_path=/dkyges-55/a?versionId=null 2024-04-12T03:49:58.207 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.207 DEBUG:boto:Path: /dkyges-55/a?versionId=null 2024-04-12T03:49:58.207 DEBUG:boto:Data: 2024-04-12T03:49:58.207 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.207 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.207 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.207 DEBUG:boto:Params: {} 2024-04-12T03:49:58.208 DEBUG:boto:Token: None 2024-04-12T03:49:58.208 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/a?versionId=null 2024-04-12T03:49:58.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UsGrhGwLUYLj+6Lpzmugg2WV0yY= 2024-04-12T03:49:58.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UsGrhGwLUYLj+6Lpzmugg2WV0yY=', 'Content-Length': '0'} 2024-04-12T03:49:58.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d817957a08521168-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:49:58.210 DEBUG:boto:path=/dkyges-55/a 2024-04-12T03:49:58.210 DEBUG:boto:auth_path=/dkyges-55/a 2024-04-12T03:49:58.210 DEBUG:boto:path=/dkyges-55/a?versionId=null 2024-04-12T03:49:58.210 DEBUG:boto:auth_path=/dkyges-55/a?versionId=null 2024-04-12T03:49:58.210 DEBUG:boto:Method: GET 2024-04-12T03:49:58.210 DEBUG:boto:Path: /dkyges-55/a?versionId=null 2024-04-12T03:49:58.210 DEBUG:boto:Data: 2024-04-12T03:49:58.210 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.211 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.211 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.211 DEBUG:boto:Params: {} 2024-04-12T03:49:58.211 DEBUG:boto:Token: None 2024-04-12T03:49:58.211 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/a?versionId=null 2024-04-12T03:49:58.211 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJL2H4pfwSe6AM/6pAg47yD+qJs= 2024-04-12T03:49:58.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJL2H4pfwSe6AM/6pAg47yD+qJs=', 'Content-Length': '0'} 2024-04-12T03:49:58.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d0429fa64b1ae19-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.254 DEBUG:boto:path=/dkyges-55/a 2024-04-12T03:49:58.254 DEBUG:boto:auth_path=/dkyges-55/a 2024-04-12T03:49:58.255 DEBUG:boto:path=/dkyges-55/a?versionId=null 2024-04-12T03:49:58.255 DEBUG:boto:auth_path=/dkyges-55/a?versionId=null 2024-04-12T03:49:58.255 DEBUG:boto:Method: GET 2024-04-12T03:49:58.255 DEBUG:boto:Path: /dkyges-55/a?versionId=null 2024-04-12T03:49:58.255 DEBUG:boto:Data: 2024-04-12T03:49:58.255 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.255 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.255 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.255 DEBUG:boto:Params: {} 2024-04-12T03:49:58.255 DEBUG:boto:Token: None 2024-04-12T03:49:58.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/a?versionId=null 2024-04-12T03:49:58.256 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJL2H4pfwSe6AM/6pAg47yD+qJs= 2024-04-12T03:49:58.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJL2H4pfwSe6AM/6pAg47yD+qJs=', 'Content-Length': '0'} 2024-04-12T03:49:58.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000829053a29886a5f0-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.258 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.258 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.258 DEBUG:boto:path=/dkyges-55/?prefix=a 2024-04-12T03:49:58.258 DEBUG:boto:auth_path=/dkyges-55/?prefix=a 2024-04-12T03:49:58.258 DEBUG:boto:Method: GET 2024-04-12T03:49:58.258 DEBUG:boto:Path: /dkyges-55/?prefix=a 2024-04-12T03:49:58.258 DEBUG:boto:Data: 2024-04-12T03:49:58.259 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.259 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.259 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.259 DEBUG:boto:Params: {} 2024-04-12T03:49:58.259 DEBUG:boto:Token: None 2024-04-12T03:49:58.259 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.259 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0= 2024-04-12T03:49:58.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0=', 'Content-Length': '0'} 2024-04-12T03:49:58.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed18a85a274a7d59-006618af66-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.262 DEBUG:boto:b'dkyges-55a1000falsea2024-04-12T03:49:36.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.262 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.262 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.262 DEBUG:boto:path=/dkyges-55/?prefix=a 2024-04-12T03:49:58.262 DEBUG:boto:auth_path=/dkyges-55/?prefix=a 2024-04-12T03:49:58.262 DEBUG:boto:Method: GET 2024-04-12T03:49:58.263 DEBUG:boto:Path: /dkyges-55/?prefix=a 2024-04-12T03:49:58.263 DEBUG:boto:Data: 2024-04-12T03:49:58.263 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.263 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.263 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.263 DEBUG:boto:Params: {} 2024-04-12T03:49:58.263 DEBUG:boto:Token: None 2024-04-12T03:49:58.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.263 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0= 2024-04-12T03:49:58.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0=', 'Content-Length': '0'} 2024-04-12T03:49:58.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055fc8df217a53cbf-006618af66-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.266 DEBUG:boto:b'dkyges-55a1000falsea2024-04-12T03:49:36.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.267 DEBUG:boto:path=/dkyges-55/a 2024-04-12T03:49:58.267 DEBUG:boto:auth_path=/dkyges-55/a 2024-04-12T03:49:58.267 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.267 DEBUG:boto:Path: /dkyges-55/a 2024-04-12T03:49:58.267 DEBUG:boto:Data: 2024-04-12T03:49:58.267 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.267 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.267 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.267 DEBUG:boto:Params: {} 2024-04-12T03:49:58.267 DEBUG:boto:Token: None 2024-04-12T03:49:58.268 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/a 2024-04-12T03:49:58.268 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9nIpyuoToM+tztXZ89wXPzice4w= 2024-04-12T03:49:58.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9nIpyuoToM+tztXZ89wXPzice4w=', 'Content-Length': '0'} 2024-04-12T03:49:58.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a610c4e9159edde1-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.270 DEBUG:boto:path=/dkyges-55/a 2024-04-12T03:49:58.270 DEBUG:boto:auth_path=/dkyges-55/a 2024-04-12T03:49:58.270 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.270 DEBUG:boto:Path: /dkyges-55/a 2024-04-12T03:49:58.270 DEBUG:boto:Data: 2024-04-12T03:49:58.270 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.270 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.270 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.270 DEBUG:boto:Params: {} 2024-04-12T03:49:58.270 DEBUG:boto:Token: None 2024-04-12T03:49:58.271 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/a 2024-04-12T03:49:58.271 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9nIpyuoToM+tztXZ89wXPzice4w= 2024-04-12T03:49:58.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9nIpyuoToM+tztXZ89wXPzice4w=', 'Content-Length': '0'} 2024-04-12T03:49:58.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d26abf8172de0841-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:49:58.273 DEBUG:boto:path=/dkyges-55/a 2024-04-12T03:49:58.273 DEBUG:boto:auth_path=/dkyges-55/a 2024-04-12T03:49:58.273 DEBUG:boto:Method: GET 2024-04-12T03:49:58.273 DEBUG:boto:Path: /dkyges-55/a 2024-04-12T03:49:58.273 DEBUG:boto:Data: 2024-04-12T03:49:58.273 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.274 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.274 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.274 DEBUG:boto:Params: {} 2024-04-12T03:49:58.274 DEBUG:boto:Token: None 2024-04-12T03:49:58.274 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/a 2024-04-12T03:49:58.274 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rYQ3IS8bTRyZOiYyh2rRzrEk3Sc= 2024-04-12T03:49:58.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rYQ3IS8bTRyZOiYyh2rRzrEk3Sc=', 'Content-Length': '0'} 2024-04-12T03:49:58.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa4972becef5626c-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.276 DEBUG:boto:path=/dkyges-55/a 2024-04-12T03:49:58.276 DEBUG:boto:auth_path=/dkyges-55/a 2024-04-12T03:49:58.277 DEBUG:boto:Method: GET 2024-04-12T03:49:58.277 DEBUG:boto:Path: /dkyges-55/a 2024-04-12T03:49:58.277 DEBUG:boto:Data: 2024-04-12T03:49:58.277 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.277 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.277 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.277 DEBUG:boto:Params: {} 2024-04-12T03:49:58.277 DEBUG:boto:Token: None 2024-04-12T03:49:58.277 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/a 2024-04-12T03:49:58.277 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rYQ3IS8bTRyZOiYyh2rRzrEk3Sc= 2024-04-12T03:49:58.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rYQ3IS8bTRyZOiYyh2rRzrEk3Sc=', 'Content-Length': '0'} 2024-04-12T03:49:58.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000115e71c1598339e6-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.280 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.280 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.280 DEBUG:boto:path=/dkyges-55/?prefix=a 2024-04-12T03:49:58.280 DEBUG:boto:auth_path=/dkyges-55/?prefix=a 2024-04-12T03:49:58.280 DEBUG:boto:Method: GET 2024-04-12T03:49:58.280 DEBUG:boto:Path: /dkyges-55/?prefix=a 2024-04-12T03:49:58.280 DEBUG:boto:Data: 2024-04-12T03:49:58.280 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.280 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.281 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.281 DEBUG:boto:Params: {} 2024-04-12T03:49:58.281 DEBUG:boto:Token: None 2024-04-12T03:49:58.281 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.281 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0= 2024-04-12T03:49:58.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0=', 'Content-Length': '0'} 2024-04-12T03:49:58.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c7f255eace17de0-006618af66-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.284 DEBUG:boto:b'dkyges-55a1000falsea2024-04-12T03:49:36.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.284 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.284 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.284 DEBUG:boto:path=/dkyges-55/?prefix=a 2024-04-12T03:49:58.284 DEBUG:boto:auth_path=/dkyges-55/?prefix=a 2024-04-12T03:49:58.284 DEBUG:boto:Method: GET 2024-04-12T03:49:58.284 DEBUG:boto:Path: /dkyges-55/?prefix=a 2024-04-12T03:49:58.284 DEBUG:boto:Data: 2024-04-12T03:49:58.285 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.285 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.285 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.285 DEBUG:boto:Params: {} 2024-04-12T03:49:58.285 DEBUG:boto:Token: None 2024-04-12T03:49:58.285 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.285 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0= 2024-04-12T03:49:58.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0=', 'Content-Length': '0'} 2024-04-12T03:49:58.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000023e0e766757990-006618af66-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.288 DEBUG:boto:b'dkyges-55a1000falsea2024-04-12T03:49:36.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:49:58.288 DEBUG:boto:path=/dkyges-55/b 2024-04-12T03:49:58.288 DEBUG:boto:auth_path=/dkyges-55/b 2024-04-12T03:49:58.288 DEBUG:boto:path=/dkyges-55/b?versionId=null 2024-04-12T03:49:58.288 DEBUG:boto:auth_path=/dkyges-55/b?versionId=null 2024-04-12T03:49:58.289 DEBUG:boto:Method: GET 2024-04-12T03:49:58.289 DEBUG:boto:Path: /dkyges-55/b?versionId=null 2024-04-12T03:49:58.289 DEBUG:boto:Data: 2024-04-12T03:49:58.289 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.289 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.289 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.289 DEBUG:boto:Params: {} 2024-04-12T03:49:58.289 DEBUG:boto:Token: None 2024-04-12T03:49:58.289 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/b?versionId=null 2024-04-12T03:49:58.289 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eS2jlmYd3Use8obWH+FMgZ2U/dA= 2024-04-12T03:49:58.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eS2jlmYd3Use8obWH+FMgZ2U/dA=', 'Content-Length': '0'} 2024-04-12T03:49:58.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcea4af0c0166fc1-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.292 DEBUG:boto:path=/dkyges-55/b 2024-04-12T03:49:58.292 DEBUG:boto:auth_path=/dkyges-55/b 2024-04-12T03:49:58.292 DEBUG:boto:path=/dkyges-55/b?versionId=null 2024-04-12T03:49:58.292 DEBUG:boto:auth_path=/dkyges-55/b?versionId=null 2024-04-12T03:49:58.292 DEBUG:boto:Method: GET 2024-04-12T03:49:58.292 DEBUG:boto:Path: /dkyges-55/b?versionId=null 2024-04-12T03:49:58.292 DEBUG:boto:Data: 2024-04-12T03:49:58.292 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.292 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.292 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.292 DEBUG:boto:Params: {} 2024-04-12T03:49:58.293 DEBUG:boto:Token: None 2024-04-12T03:49:58.293 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/b?versionId=null 2024-04-12T03:49:58.293 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eS2jlmYd3Use8obWH+FMgZ2U/dA= 2024-04-12T03:49:58.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eS2jlmYd3Use8obWH+FMgZ2U/dA=', 'Content-Length': '0'} 2024-04-12T03:49:58.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009a32f9cca892445-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.338 DEBUG:boto:path=/dkyges-55/b 2024-04-12T03:49:58.338 DEBUG:boto:auth_path=/dkyges-55/b 2024-04-12T03:49:58.339 DEBUG:boto:path=/dkyges-55/b?versionId=null 2024-04-12T03:49:58.339 DEBUG:boto:auth_path=/dkyges-55/b?versionId=null 2024-04-12T03:49:58.339 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.339 DEBUG:boto:Path: /dkyges-55/b?versionId=null 2024-04-12T03:49:58.339 DEBUG:boto:Data: 2024-04-12T03:49:58.339 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.339 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.339 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.339 DEBUG:boto:Params: {} 2024-04-12T03:49:58.339 DEBUG:boto:Token: None 2024-04-12T03:49:58.340 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/b?versionId=null 2024-04-12T03:49:58.340 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t/lm5O/Ko+LPTzUSvslaG9b4/9Q= 2024-04-12T03:49:58.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t/lm5O/Ko+LPTzUSvslaG9b4/9Q=', 'Content-Length': '0'} 2024-04-12T03:49:58.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acef910a947b0b1c-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.342 DEBUG:boto:path=/dkyges-55/b 2024-04-12T03:49:58.342 DEBUG:boto:auth_path=/dkyges-55/b 2024-04-12T03:49:58.342 DEBUG:boto:path=/dkyges-55/b?versionId=null 2024-04-12T03:49:58.342 DEBUG:boto:auth_path=/dkyges-55/b?versionId=null 2024-04-12T03:49:58.342 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.342 DEBUG:boto:Path: /dkyges-55/b?versionId=null 2024-04-12T03:49:58.342 DEBUG:boto:Data: 2024-04-12T03:49:58.342 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.342 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.343 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.343 DEBUG:boto:Params: {} 2024-04-12T03:49:58.343 DEBUG:boto:Token: None 2024-04-12T03:49:58.343 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/b?versionId=null 2024-04-12T03:49:58.343 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t/lm5O/Ko+LPTzUSvslaG9b4/9Q= 2024-04-12T03:49:58.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t/lm5O/Ko+LPTzUSvslaG9b4/9Q=', 'Content-Length': '0'} 2024-04-12T03:49:58.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cc566ec34bce027-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:49:58.345 DEBUG:boto:path=/dkyges-55/b 2024-04-12T03:49:58.345 DEBUG:boto:auth_path=/dkyges-55/b 2024-04-12T03:49:58.345 DEBUG:boto:path=/dkyges-55/b?versionId=null 2024-04-12T03:49:58.345 DEBUG:boto:auth_path=/dkyges-55/b?versionId=null 2024-04-12T03:49:58.345 DEBUG:boto:Method: GET 2024-04-12T03:49:58.345 DEBUG:boto:Path: /dkyges-55/b?versionId=null 2024-04-12T03:49:58.346 DEBUG:boto:Data: 2024-04-12T03:49:58.346 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.346 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.346 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.346 DEBUG:boto:Params: {} 2024-04-12T03:49:58.346 DEBUG:boto:Token: None 2024-04-12T03:49:58.346 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/b?versionId=null 2024-04-12T03:49:58.346 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eS2jlmYd3Use8obWH+FMgZ2U/dA= 2024-04-12T03:49:58.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eS2jlmYd3Use8obWH+FMgZ2U/dA=', 'Content-Length': '0'} 2024-04-12T03:49:58.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3e9bbfedcc73e91-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.348 DEBUG:boto:path=/dkyges-55/b 2024-04-12T03:49:58.348 DEBUG:boto:auth_path=/dkyges-55/b 2024-04-12T03:49:58.348 DEBUG:boto:path=/dkyges-55/b?versionId=null 2024-04-12T03:49:58.349 DEBUG:boto:auth_path=/dkyges-55/b?versionId=null 2024-04-12T03:49:58.349 DEBUG:boto:Method: GET 2024-04-12T03:49:58.349 DEBUG:boto:Path: /dkyges-55/b?versionId=null 2024-04-12T03:49:58.349 DEBUG:boto:Data: 2024-04-12T03:49:58.349 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.349 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.349 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.349 DEBUG:boto:Params: {} 2024-04-12T03:49:58.349 DEBUG:boto:Token: None 2024-04-12T03:49:58.349 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/b?versionId=null 2024-04-12T03:49:58.350 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eS2jlmYd3Use8obWH+FMgZ2U/dA= 2024-04-12T03:49:58.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eS2jlmYd3Use8obWH+FMgZ2U/dA=', 'Content-Length': '0'} 2024-04-12T03:49:58.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086f30bde61565e22-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.352 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.352 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.352 DEBUG:boto:path=/dkyges-55/?prefix=b 2024-04-12T03:49:58.352 DEBUG:boto:auth_path=/dkyges-55/?prefix=b 2024-04-12T03:49:58.352 DEBUG:boto:Method: GET 2024-04-12T03:49:58.352 DEBUG:boto:Path: /dkyges-55/?prefix=b 2024-04-12T03:49:58.353 DEBUG:boto:Data: 2024-04-12T03:49:58.353 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.353 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.353 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.353 DEBUG:boto:Params: {} 2024-04-12T03:49:58.353 DEBUG:boto:Token: None 2024-04-12T03:49:58.353 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.353 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0= 2024-04-12T03:49:58.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0=', 'Content-Length': '0'} 2024-04-12T03:49:58.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dc2e71a5699bd56-006618af66-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.356 DEBUG:boto:b'dkyges-55b1000falseb2024-04-12T03:49:36.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.356 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.356 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.357 DEBUG:boto:path=/dkyges-55/?prefix=b 2024-04-12T03:49:58.357 DEBUG:boto:auth_path=/dkyges-55/?prefix=b 2024-04-12T03:49:58.357 DEBUG:boto:Method: GET 2024-04-12T03:49:58.357 DEBUG:boto:Path: /dkyges-55/?prefix=b 2024-04-12T03:49:58.357 DEBUG:boto:Data: 2024-04-12T03:49:58.357 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.357 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.357 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.357 DEBUG:boto:Params: {} 2024-04-12T03:49:58.357 DEBUG:boto:Token: None 2024-04-12T03:49:58.357 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.358 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0= 2024-04-12T03:49:58.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0=', 'Content-Length': '0'} 2024-04-12T03:49:58.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebda325ed4dcf085-006618af66-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.360 DEBUG:boto:b'dkyges-55b1000falseb2024-04-12T03:49:36.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.361 DEBUG:boto:path=/dkyges-55/b 2024-04-12T03:49:58.361 DEBUG:boto:auth_path=/dkyges-55/b 2024-04-12T03:49:58.361 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.361 DEBUG:boto:Path: /dkyges-55/b 2024-04-12T03:49:58.361 DEBUG:boto:Data: 2024-04-12T03:49:58.361 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.361 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.361 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.362 DEBUG:boto:Params: {} 2024-04-12T03:49:58.362 DEBUG:boto:Token: None 2024-04-12T03:49:58.362 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/b 2024-04-12T03:49:58.362 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HNsFK5CuSvWxbYtGe1DLacDe6Yk= 2024-04-12T03:49:58.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HNsFK5CuSvWxbYtGe1DLacDe6Yk=', 'Content-Length': '0'} 2024-04-12T03:49:58.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e37f4366624a68f8-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.364 DEBUG:boto:path=/dkyges-55/b 2024-04-12T03:49:58.364 DEBUG:boto:auth_path=/dkyges-55/b 2024-04-12T03:49:58.364 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.364 DEBUG:boto:Path: /dkyges-55/b 2024-04-12T03:49:58.364 DEBUG:boto:Data: 2024-04-12T03:49:58.364 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.364 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.364 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.364 DEBUG:boto:Params: {} 2024-04-12T03:49:58.365 DEBUG:boto:Token: None 2024-04-12T03:49:58.365 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/b 2024-04-12T03:49:58.365 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HNsFK5CuSvWxbYtGe1DLacDe6Yk= 2024-04-12T03:49:58.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HNsFK5CuSvWxbYtGe1DLacDe6Yk=', 'Content-Length': '0'} 2024-04-12T03:49:58.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eaa012adbbfc5b8-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:49:58.367 DEBUG:boto:path=/dkyges-55/b 2024-04-12T03:49:58.367 DEBUG:boto:auth_path=/dkyges-55/b 2024-04-12T03:49:58.367 DEBUG:boto:Method: GET 2024-04-12T03:49:58.367 DEBUG:boto:Path: /dkyges-55/b 2024-04-12T03:49:58.367 DEBUG:boto:Data: 2024-04-12T03:49:58.367 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.367 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.367 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.367 DEBUG:boto:Params: {} 2024-04-12T03:49:58.368 DEBUG:boto:Token: None 2024-04-12T03:49:58.368 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/b 2024-04-12T03:49:58.368 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WUAiOTA+mtzhqGAUtSePNqne7Ho= 2024-04-12T03:49:58.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WUAiOTA+mtzhqGAUtSePNqne7Ho=', 'Content-Length': '0'} 2024-04-12T03:49:58.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d9495b775907d1f-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.410 DEBUG:boto:path=/dkyges-55/b 2024-04-12T03:49:58.410 DEBUG:boto:auth_path=/dkyges-55/b 2024-04-12T03:49:58.410 DEBUG:boto:Method: GET 2024-04-12T03:49:58.411 DEBUG:boto:Path: /dkyges-55/b 2024-04-12T03:49:58.411 DEBUG:boto:Data: 2024-04-12T03:49:58.411 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.411 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.411 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.411 DEBUG:boto:Params: {} 2024-04-12T03:49:58.411 DEBUG:boto:Token: None 2024-04-12T03:49:58.411 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/b 2024-04-12T03:49:58.411 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WUAiOTA+mtzhqGAUtSePNqne7Ho= 2024-04-12T03:49:58.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WUAiOTA+mtzhqGAUtSePNqne7Ho=', 'Content-Length': '0'} 2024-04-12T03:49:58.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045eb28840bb69078-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.414 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.414 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.414 DEBUG:boto:path=/dkyges-55/?prefix=b 2024-04-12T03:49:58.414 DEBUG:boto:auth_path=/dkyges-55/?prefix=b 2024-04-12T03:49:58.414 DEBUG:boto:Method: GET 2024-04-12T03:49:58.414 DEBUG:boto:Path: /dkyges-55/?prefix=b 2024-04-12T03:49:58.414 DEBUG:boto:Data: 2024-04-12T03:49:58.414 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.415 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.415 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.415 DEBUG:boto:Params: {} 2024-04-12T03:49:58.415 DEBUG:boto:Token: None 2024-04-12T03:49:58.415 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.415 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0= 2024-04-12T03:49:58.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0=', 'Content-Length': '0'} 2024-04-12T03:49:58.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aeabe88675b0cabc-006618af66-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.418 DEBUG:boto:b'dkyges-55b1000falseb2024-04-12T03:49:36.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.418 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.418 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.418 DEBUG:boto:path=/dkyges-55/?prefix=b 2024-04-12T03:49:58.418 DEBUG:boto:auth_path=/dkyges-55/?prefix=b 2024-04-12T03:49:58.419 DEBUG:boto:Method: GET 2024-04-12T03:49:58.419 DEBUG:boto:Path: /dkyges-55/?prefix=b 2024-04-12T03:49:58.419 DEBUG:boto:Data: 2024-04-12T03:49:58.419 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.419 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.419 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.419 DEBUG:boto:Params: {} 2024-04-12T03:49:58.419 DEBUG:boto:Token: None 2024-04-12T03:49:58.419 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.419 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0= 2024-04-12T03:49:58.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0=', 'Content-Length': '0'} 2024-04-12T03:49:58.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceca79b3e2d1aea2-006618af66-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.422 DEBUG:boto:b'dkyges-55b1000falseb2024-04-12T03:49:36.615Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:49:58.422 DEBUG:boto:path=/dkyges-55/c 2024-04-12T03:49:58.423 DEBUG:boto:auth_path=/dkyges-55/c 2024-04-12T03:49:58.423 DEBUG:boto:path=/dkyges-55/c?versionId=null 2024-04-12T03:49:58.423 DEBUG:boto:auth_path=/dkyges-55/c?versionId=null 2024-04-12T03:49:58.423 DEBUG:boto:Method: GET 2024-04-12T03:49:58.423 DEBUG:boto:Path: /dkyges-55/c?versionId=null 2024-04-12T03:49:58.423 DEBUG:boto:Data: 2024-04-12T03:49:58.423 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.423 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.423 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.423 DEBUG:boto:Params: {} 2024-04-12T03:49:58.423 DEBUG:boto:Token: None 2024-04-12T03:49:58.424 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/c?versionId=null 2024-04-12T03:49:58.424 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PfrGSf9WE2tRdtmdzljEldnHYX4= 2024-04-12T03:49:58.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PfrGSf9WE2tRdtmdzljEldnHYX4=', 'Content-Length': '0'} 2024-04-12T03:49:58.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059a5e94350545a90-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.426 DEBUG:boto:path=/dkyges-55/c 2024-04-12T03:49:58.426 DEBUG:boto:auth_path=/dkyges-55/c 2024-04-12T03:49:58.426 DEBUG:boto:path=/dkyges-55/c?versionId=null 2024-04-12T03:49:58.426 DEBUG:boto:auth_path=/dkyges-55/c?versionId=null 2024-04-12T03:49:58.426 DEBUG:boto:Method: GET 2024-04-12T03:49:58.426 DEBUG:boto:Path: /dkyges-55/c?versionId=null 2024-04-12T03:49:58.427 DEBUG:boto:Data: 2024-04-12T03:49:58.427 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.427 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.427 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.427 DEBUG:boto:Params: {} 2024-04-12T03:49:58.427 DEBUG:boto:Token: None 2024-04-12T03:49:58.427 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/c?versionId=null 2024-04-12T03:49:58.427 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PfrGSf9WE2tRdtmdzljEldnHYX4= 2024-04-12T03:49:58.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PfrGSf9WE2tRdtmdzljEldnHYX4=', 'Content-Length': '0'} 2024-04-12T03:49:58.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000190bec4157582ac6-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.429 DEBUG:boto:path=/dkyges-55/c 2024-04-12T03:49:58.429 DEBUG:boto:auth_path=/dkyges-55/c 2024-04-12T03:49:58.430 DEBUG:boto:path=/dkyges-55/c?versionId=null 2024-04-12T03:49:58.430 DEBUG:boto:auth_path=/dkyges-55/c?versionId=null 2024-04-12T03:49:58.430 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.430 DEBUG:boto:Path: /dkyges-55/c?versionId=null 2024-04-12T03:49:58.430 DEBUG:boto:Data: 2024-04-12T03:49:58.430 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.430 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.430 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.430 DEBUG:boto:Params: {} 2024-04-12T03:49:58.430 DEBUG:boto:Token: None 2024-04-12T03:49:58.431 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/c?versionId=null 2024-04-12T03:49:58.431 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MjlMlywIP88rhqszA8uU84bC+7I= 2024-04-12T03:49:58.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MjlMlywIP88rhqszA8uU84bC+7I=', 'Content-Length': '0'} 2024-04-12T03:49:58.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054042f2ded5b8a12-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.433 DEBUG:boto:path=/dkyges-55/c 2024-04-12T03:49:58.433 DEBUG:boto:auth_path=/dkyges-55/c 2024-04-12T03:49:58.433 DEBUG:boto:path=/dkyges-55/c?versionId=null 2024-04-12T03:49:58.433 DEBUG:boto:auth_path=/dkyges-55/c?versionId=null 2024-04-12T03:49:58.433 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.433 DEBUG:boto:Path: /dkyges-55/c?versionId=null 2024-04-12T03:49:58.433 DEBUG:boto:Data: 2024-04-12T03:49:58.433 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.434 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.434 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.434 DEBUG:boto:Params: {} 2024-04-12T03:49:58.434 DEBUG:boto:Token: None 2024-04-12T03:49:58.434 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/c?versionId=null 2024-04-12T03:49:58.434 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MjlMlywIP88rhqszA8uU84bC+7I= 2024-04-12T03:49:58.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MjlMlywIP88rhqszA8uU84bC+7I=', 'Content-Length': '0'} 2024-04-12T03:49:58.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a55389780980db75-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:49:58.436 DEBUG:boto:path=/dkyges-55/c 2024-04-12T03:49:58.437 DEBUG:boto:auth_path=/dkyges-55/c 2024-04-12T03:49:58.437 DEBUG:boto:path=/dkyges-55/c?versionId=null 2024-04-12T03:49:58.437 DEBUG:boto:auth_path=/dkyges-55/c?versionId=null 2024-04-12T03:49:58.437 DEBUG:boto:Method: GET 2024-04-12T03:49:58.437 DEBUG:boto:Path: /dkyges-55/c?versionId=null 2024-04-12T03:49:58.437 DEBUG:boto:Data: 2024-04-12T03:49:58.437 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.437 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.437 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.437 DEBUG:boto:Params: {} 2024-04-12T03:49:58.437 DEBUG:boto:Token: None 2024-04-12T03:49:58.437 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/c?versionId=null 2024-04-12T03:49:58.438 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PfrGSf9WE2tRdtmdzljEldnHYX4= 2024-04-12T03:49:58.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PfrGSf9WE2tRdtmdzljEldnHYX4=', 'Content-Length': '0'} 2024-04-12T03:49:58.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbf0b62440ccc624-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.440 DEBUG:boto:path=/dkyges-55/c 2024-04-12T03:49:58.440 DEBUG:boto:auth_path=/dkyges-55/c 2024-04-12T03:49:58.440 DEBUG:boto:path=/dkyges-55/c?versionId=null 2024-04-12T03:49:58.440 DEBUG:boto:auth_path=/dkyges-55/c?versionId=null 2024-04-12T03:49:58.440 DEBUG:boto:Method: GET 2024-04-12T03:49:58.440 DEBUG:boto:Path: /dkyges-55/c?versionId=null 2024-04-12T03:49:58.440 DEBUG:boto:Data: 2024-04-12T03:49:58.440 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.440 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.441 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.441 DEBUG:boto:Params: {} 2024-04-12T03:49:58.441 DEBUG:boto:Token: None 2024-04-12T03:49:58.441 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/c?versionId=null 2024-04-12T03:49:58.441 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PfrGSf9WE2tRdtmdzljEldnHYX4= 2024-04-12T03:49:58.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PfrGSf9WE2tRdtmdzljEldnHYX4=', 'Content-Length': '0'} 2024-04-12T03:49:58.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053b150bded93ed5d-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.443 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.443 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.444 DEBUG:boto:path=/dkyges-55/?prefix=c 2024-04-12T03:49:58.444 DEBUG:boto:auth_path=/dkyges-55/?prefix=c 2024-04-12T03:49:58.444 DEBUG:boto:Method: GET 2024-04-12T03:49:58.444 DEBUG:boto:Path: /dkyges-55/?prefix=c 2024-04-12T03:49:58.444 DEBUG:boto:Data: 2024-04-12T03:49:58.444 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.444 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.444 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.444 DEBUG:boto:Params: {} 2024-04-12T03:49:58.444 DEBUG:boto:Token: None 2024-04-12T03:49:58.444 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.445 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0= 2024-04-12T03:49:58.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0=', 'Content-Length': '0'} 2024-04-12T03:49:58.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e7b1ad98e541467-006618af66-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.450 DEBUG:boto:b'dkyges-55c1000falsec2024-04-12T03:49:36.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.450 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.450 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.450 DEBUG:boto:path=/dkyges-55/?prefix=c 2024-04-12T03:49:58.450 DEBUG:boto:auth_path=/dkyges-55/?prefix=c 2024-04-12T03:49:58.450 DEBUG:boto:Method: GET 2024-04-12T03:49:58.451 DEBUG:boto:Path: /dkyges-55/?prefix=c 2024-04-12T03:49:58.451 DEBUG:boto:Data: 2024-04-12T03:49:58.451 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.451 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.451 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.451 DEBUG:boto:Params: {} 2024-04-12T03:49:58.451 DEBUG:boto:Token: None 2024-04-12T03:49:58.452 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.452 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0= 2024-04-12T03:49:58.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0=', 'Content-Length': '0'} 2024-04-12T03:49:58.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ee7d2c96fe16247-006618af66-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.454 DEBUG:boto:b'dkyges-55c1000falsec2024-04-12T03:49:36.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.455 DEBUG:boto:path=/dkyges-55/c 2024-04-12T03:49:58.455 DEBUG:boto:auth_path=/dkyges-55/c 2024-04-12T03:49:58.455 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.455 DEBUG:boto:Path: /dkyges-55/c 2024-04-12T03:49:58.455 DEBUG:boto:Data: 2024-04-12T03:49:58.455 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.455 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.455 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.455 DEBUG:boto:Params: {} 2024-04-12T03:49:58.456 DEBUG:boto:Token: None 2024-04-12T03:49:58.456 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/c 2024-04-12T03:49:58.456 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Cv2j0kqe5JFSEMKPO7f9GqDzJ8Y= 2024-04-12T03:49:58.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Cv2j0kqe5JFSEMKPO7f9GqDzJ8Y=', 'Content-Length': '0'} 2024-04-12T03:49:58.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a17c0f9329ed7a1-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.458 DEBUG:boto:path=/dkyges-55/c 2024-04-12T03:49:58.458 DEBUG:boto:auth_path=/dkyges-55/c 2024-04-12T03:49:58.458 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.458 DEBUG:boto:Path: /dkyges-55/c 2024-04-12T03:49:58.459 DEBUG:boto:Data: 2024-04-12T03:49:58.459 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.459 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.459 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.459 DEBUG:boto:Params: {} 2024-04-12T03:49:58.459 DEBUG:boto:Token: None 2024-04-12T03:49:58.459 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/c 2024-04-12T03:49:58.459 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Cv2j0kqe5JFSEMKPO7f9GqDzJ8Y= 2024-04-12T03:49:58.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Cv2j0kqe5JFSEMKPO7f9GqDzJ8Y=', 'Content-Length': '0'} 2024-04-12T03:49:58.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daa00247936eafe8-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:49:58.462 DEBUG:boto:path=/dkyges-55/c 2024-04-12T03:49:58.462 DEBUG:boto:auth_path=/dkyges-55/c 2024-04-12T03:49:58.462 DEBUG:boto:Method: GET 2024-04-12T03:49:58.462 DEBUG:boto:Path: /dkyges-55/c 2024-04-12T03:49:58.462 DEBUG:boto:Data: 2024-04-12T03:49:58.462 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.462 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.462 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.462 DEBUG:boto:Params: {} 2024-04-12T03:49:58.463 DEBUG:boto:Token: None 2024-04-12T03:49:58.463 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/c 2024-04-12T03:49:58.463 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3vg2vstQfoJD/CfodVI0KlRcEwM= 2024-04-12T03:49:58.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3vg2vstQfoJD/CfodVI0KlRcEwM=', 'Content-Length': '0'} 2024-04-12T03:49:58.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006c6efbbc22e874c-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.465 DEBUG:boto:path=/dkyges-55/c 2024-04-12T03:49:58.465 DEBUG:boto:auth_path=/dkyges-55/c 2024-04-12T03:49:58.465 DEBUG:boto:Method: GET 2024-04-12T03:49:58.465 DEBUG:boto:Path: /dkyges-55/c 2024-04-12T03:49:58.465 DEBUG:boto:Data: 2024-04-12T03:49:58.466 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.466 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.466 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.466 DEBUG:boto:Params: {} 2024-04-12T03:49:58.466 DEBUG:boto:Token: None 2024-04-12T03:49:58.466 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/c 2024-04-12T03:49:58.466 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3vg2vstQfoJD/CfodVI0KlRcEwM= 2024-04-12T03:49:58.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3vg2vstQfoJD/CfodVI0KlRcEwM=', 'Content-Length': '0'} 2024-04-12T03:49:58.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba3b8ce74e53fda2-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.510 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.511 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.511 DEBUG:boto:path=/dkyges-55/?prefix=c 2024-04-12T03:49:58.511 DEBUG:boto:auth_path=/dkyges-55/?prefix=c 2024-04-12T03:49:58.511 DEBUG:boto:Method: GET 2024-04-12T03:49:58.511 DEBUG:boto:Path: /dkyges-55/?prefix=c 2024-04-12T03:49:58.511 DEBUG:boto:Data: 2024-04-12T03:49:58.511 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.511 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.511 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.511 DEBUG:boto:Params: {} 2024-04-12T03:49:58.512 DEBUG:boto:Token: None 2024-04-12T03:49:58.512 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.512 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0= 2024-04-12T03:49:58.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0=', 'Content-Length': '0'} 2024-04-12T03:49:58.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d4e307f1b6cba12-006618af66-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.514 DEBUG:boto:b'dkyges-55c1000falsec2024-04-12T03:49:36.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.515 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.515 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.515 DEBUG:boto:path=/dkyges-55/?prefix=c 2024-04-12T03:49:58.515 DEBUG:boto:auth_path=/dkyges-55/?prefix=c 2024-04-12T03:49:58.515 DEBUG:boto:Method: GET 2024-04-12T03:49:58.515 DEBUG:boto:Path: /dkyges-55/?prefix=c 2024-04-12T03:49:58.515 DEBUG:boto:Data: 2024-04-12T03:49:58.515 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.515 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.516 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.516 DEBUG:boto:Params: {} 2024-04-12T03:49:58.516 DEBUG:boto:Token: None 2024-04-12T03:49:58.516 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.516 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0= 2024-04-12T03:49:58.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0=', 'Content-Length': '0'} 2024-04-12T03:49:58.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae95f07331651a82-006618af66-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.518 DEBUG:boto:b'dkyges-55c1000falsec2024-04-12T03:49:36.627Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:49:58.519 DEBUG:boto:path=/dkyges-55/d 2024-04-12T03:49:58.519 DEBUG:boto:auth_path=/dkyges-55/d 2024-04-12T03:49:58.519 DEBUG:boto:path=/dkyges-55/d?versionId=null 2024-04-12T03:49:58.519 DEBUG:boto:auth_path=/dkyges-55/d?versionId=null 2024-04-12T03:49:58.519 DEBUG:boto:Method: GET 2024-04-12T03:49:58.519 DEBUG:boto:Path: /dkyges-55/d?versionId=null 2024-04-12T03:49:58.519 DEBUG:boto:Data: 2024-04-12T03:49:58.520 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.520 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.520 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.520 DEBUG:boto:Params: {} 2024-04-12T03:49:58.520 DEBUG:boto:Token: None 2024-04-12T03:49:58.520 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/d?versionId=null 2024-04-12T03:49:58.520 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m4cUrPznHQl68NjVnwUc5BtIH/E= 2024-04-12T03:49:58.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m4cUrPznHQl68NjVnwUc5BtIH/E=', 'Content-Length': '0'} 2024-04-12T03:49:58.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8182210906b144a-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.562 DEBUG:boto:path=/dkyges-55/d 2024-04-12T03:49:58.562 DEBUG:boto:auth_path=/dkyges-55/d 2024-04-12T03:49:58.563 DEBUG:boto:path=/dkyges-55/d?versionId=null 2024-04-12T03:49:58.563 DEBUG:boto:auth_path=/dkyges-55/d?versionId=null 2024-04-12T03:49:58.563 DEBUG:boto:Method: GET 2024-04-12T03:49:58.563 DEBUG:boto:Path: /dkyges-55/d?versionId=null 2024-04-12T03:49:58.563 DEBUG:boto:Data: 2024-04-12T03:49:58.563 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.563 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.563 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.563 DEBUG:boto:Params: {} 2024-04-12T03:49:58.563 DEBUG:boto:Token: None 2024-04-12T03:49:58.564 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/d?versionId=null 2024-04-12T03:49:58.564 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m4cUrPznHQl68NjVnwUc5BtIH/E= 2024-04-12T03:49:58.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m4cUrPznHQl68NjVnwUc5BtIH/E=', 'Content-Length': '0'} 2024-04-12T03:49:58.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047372d48d2208019-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.566 DEBUG:boto:path=/dkyges-55/d 2024-04-12T03:49:58.566 DEBUG:boto:auth_path=/dkyges-55/d 2024-04-12T03:49:58.566 DEBUG:boto:path=/dkyges-55/d?versionId=null 2024-04-12T03:49:58.566 DEBUG:boto:auth_path=/dkyges-55/d?versionId=null 2024-04-12T03:49:58.566 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.566 DEBUG:boto:Path: /dkyges-55/d?versionId=null 2024-04-12T03:49:58.566 DEBUG:boto:Data: 2024-04-12T03:49:58.567 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.567 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.567 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.567 DEBUG:boto:Params: {} 2024-04-12T03:49:58.567 DEBUG:boto:Token: None 2024-04-12T03:49:58.567 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/d?versionId=null 2024-04-12T03:49:58.567 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yKVUxDqkQ221NteRaGYg+8sBu6Q= 2024-04-12T03:49:58.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yKVUxDqkQ221NteRaGYg+8sBu6Q=', 'Content-Length': '0'} 2024-04-12T03:49:58.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9eeaf7522cb487f-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.569 DEBUG:boto:path=/dkyges-55/d 2024-04-12T03:49:58.569 DEBUG:boto:auth_path=/dkyges-55/d 2024-04-12T03:49:58.569 DEBUG:boto:path=/dkyges-55/d?versionId=null 2024-04-12T03:49:58.570 DEBUG:boto:auth_path=/dkyges-55/d?versionId=null 2024-04-12T03:49:58.570 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.570 DEBUG:boto:Path: /dkyges-55/d?versionId=null 2024-04-12T03:49:58.570 DEBUG:boto:Data: 2024-04-12T03:49:58.570 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.570 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.570 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.570 DEBUG:boto:Params: {} 2024-04-12T03:49:58.570 DEBUG:boto:Token: None 2024-04-12T03:49:58.570 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/d?versionId=null 2024-04-12T03:49:58.570 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yKVUxDqkQ221NteRaGYg+8sBu6Q= 2024-04-12T03:49:58.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yKVUxDqkQ221NteRaGYg+8sBu6Q=', 'Content-Length': '0'} 2024-04-12T03:49:58.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c611653aa7d78e10-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:49:58.573 DEBUG:boto:path=/dkyges-55/d 2024-04-12T03:49:58.573 DEBUG:boto:auth_path=/dkyges-55/d 2024-04-12T03:49:58.573 DEBUG:boto:path=/dkyges-55/d?versionId=null 2024-04-12T03:49:58.573 DEBUG:boto:auth_path=/dkyges-55/d?versionId=null 2024-04-12T03:49:58.573 DEBUG:boto:Method: GET 2024-04-12T03:49:58.573 DEBUG:boto:Path: /dkyges-55/d?versionId=null 2024-04-12T03:49:58.573 DEBUG:boto:Data: 2024-04-12T03:49:58.573 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.573 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.574 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.574 DEBUG:boto:Params: {} 2024-04-12T03:49:58.574 DEBUG:boto:Token: None 2024-04-12T03:49:58.574 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/d?versionId=null 2024-04-12T03:49:58.574 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m4cUrPznHQl68NjVnwUc5BtIH/E= 2024-04-12T03:49:58.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m4cUrPznHQl68NjVnwUc5BtIH/E=', 'Content-Length': '0'} 2024-04-12T03:49:58.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026235a2d9d5e0411-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.576 DEBUG:boto:path=/dkyges-55/d 2024-04-12T03:49:58.576 DEBUG:boto:auth_path=/dkyges-55/d 2024-04-12T03:49:58.576 DEBUG:boto:path=/dkyges-55/d?versionId=null 2024-04-12T03:49:58.576 DEBUG:boto:auth_path=/dkyges-55/d?versionId=null 2024-04-12T03:49:58.577 DEBUG:boto:Method: GET 2024-04-12T03:49:58.577 DEBUG:boto:Path: /dkyges-55/d?versionId=null 2024-04-12T03:49:58.577 DEBUG:boto:Data: 2024-04-12T03:49:58.577 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.577 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.577 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.577 DEBUG:boto:Params: {} 2024-04-12T03:49:58.577 DEBUG:boto:Token: None 2024-04-12T03:49:58.577 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/d?versionId=null 2024-04-12T03:49:58.577 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m4cUrPznHQl68NjVnwUc5BtIH/E= 2024-04-12T03:49:58.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m4cUrPznHQl68NjVnwUc5BtIH/E=', 'Content-Length': '0'} 2024-04-12T03:49:58.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee9c23d4df7c2175-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.579 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.580 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.580 DEBUG:boto:path=/dkyges-55/?prefix=d 2024-04-12T03:49:58.580 DEBUG:boto:auth_path=/dkyges-55/?prefix=d 2024-04-12T03:49:58.580 DEBUG:boto:Method: GET 2024-04-12T03:49:58.580 DEBUG:boto:Path: /dkyges-55/?prefix=d 2024-04-12T03:49:58.580 DEBUG:boto:Data: 2024-04-12T03:49:58.580 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.580 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.580 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.580 DEBUG:boto:Params: {} 2024-04-12T03:49:58.580 DEBUG:boto:Token: None 2024-04-12T03:49:58.581 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.581 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0= 2024-04-12T03:49:58.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0=', 'Content-Length': '0'} 2024-04-12T03:49:58.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3f01f3de36887ae-006618af66-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.583 DEBUG:boto:b'dkyges-55d1000falsed2024-04-12T03:49:36.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.584 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.584 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.584 DEBUG:boto:path=/dkyges-55/?prefix=d 2024-04-12T03:49:58.584 DEBUG:boto:auth_path=/dkyges-55/?prefix=d 2024-04-12T03:49:58.584 DEBUG:boto:Method: GET 2024-04-12T03:49:58.584 DEBUG:boto:Path: /dkyges-55/?prefix=d 2024-04-12T03:49:58.584 DEBUG:boto:Data: 2024-04-12T03:49:58.584 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.585 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.585 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.585 DEBUG:boto:Params: {} 2024-04-12T03:49:58.585 DEBUG:boto:Token: None 2024-04-12T03:49:58.585 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.585 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0= 2024-04-12T03:49:58.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0=', 'Content-Length': '0'} 2024-04-12T03:49:58.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000810e17780417809b-006618af66-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.588 DEBUG:boto:b'dkyges-55d1000falsed2024-04-12T03:49:36.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.588 DEBUG:boto:path=/dkyges-55/d 2024-04-12T03:49:58.588 DEBUG:boto:auth_path=/dkyges-55/d 2024-04-12T03:49:58.588 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.588 DEBUG:boto:Path: /dkyges-55/d 2024-04-12T03:49:58.588 DEBUG:boto:Data: 2024-04-12T03:49:58.589 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.589 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.589 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.589 DEBUG:boto:Params: {} 2024-04-12T03:49:58.589 DEBUG:boto:Token: None 2024-04-12T03:49:58.589 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/d 2024-04-12T03:49:58.589 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Tgyy8fk2kvkCFw53/LYknPw2Q0o= 2024-04-12T03:49:58.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Tgyy8fk2kvkCFw53/LYknPw2Q0o=', 'Content-Length': '0'} 2024-04-12T03:49:58.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0c426f898a7c38c-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.591 DEBUG:boto:path=/dkyges-55/d 2024-04-12T03:49:58.591 DEBUG:boto:auth_path=/dkyges-55/d 2024-04-12T03:49:58.591 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.591 DEBUG:boto:Path: /dkyges-55/d 2024-04-12T03:49:58.591 DEBUG:boto:Data: 2024-04-12T03:49:58.591 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.592 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.592 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.592 DEBUG:boto:Params: {} 2024-04-12T03:49:58.592 DEBUG:boto:Token: None 2024-04-12T03:49:58.592 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/d 2024-04-12T03:49:58.592 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Tgyy8fk2kvkCFw53/LYknPw2Q0o= 2024-04-12T03:49:58.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Tgyy8fk2kvkCFw53/LYknPw2Q0o=', 'Content-Length': '0'} 2024-04-12T03:49:58.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba42a8dff3b23cfe-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:49:58.594 DEBUG:boto:path=/dkyges-55/d 2024-04-12T03:49:58.594 DEBUG:boto:auth_path=/dkyges-55/d 2024-04-12T03:49:58.594 DEBUG:boto:Method: GET 2024-04-12T03:49:58.594 DEBUG:boto:Path: /dkyges-55/d 2024-04-12T03:49:58.594 DEBUG:boto:Data: 2024-04-12T03:49:58.595 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.595 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.595 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.595 DEBUG:boto:Params: {} 2024-04-12T03:49:58.595 DEBUG:boto:Token: None 2024-04-12T03:49:58.595 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/d 2024-04-12T03:49:58.595 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y4QI0oqWDB4CUy2qer3DPl5DpgU= 2024-04-12T03:49:58.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y4QI0oqWDB4CUy2qer3DPl5DpgU=', 'Content-Length': '0'} 2024-04-12T03:49:58.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b1c00148a22d078-006618af66-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.638 DEBUG:boto:path=/dkyges-55/d 2024-04-12T03:49:58.638 DEBUG:boto:auth_path=/dkyges-55/d 2024-04-12T03:49:58.639 DEBUG:boto:Method: GET 2024-04-12T03:49:58.639 DEBUG:boto:Path: /dkyges-55/d 2024-04-12T03:49:58.639 DEBUG:boto:Data: 2024-04-12T03:49:58.639 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.639 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.639 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.639 DEBUG:boto:Params: {} 2024-04-12T03:49:58.639 DEBUG:boto:Token: None 2024-04-12T03:49:58.639 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/d 2024-04-12T03:49:58.640 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y4QI0oqWDB4CUy2qer3DPl5DpgU= 2024-04-12T03:49:58.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y4QI0oqWDB4CUy2qer3DPl5DpgU=', 'Content-Length': '0'} 2024-04-12T03:49:58.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028b38989a347cb4a-006618af66-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.642 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.642 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.642 DEBUG:boto:path=/dkyges-55/?prefix=d 2024-04-12T03:49:58.642 DEBUG:boto:auth_path=/dkyges-55/?prefix=d 2024-04-12T03:49:58.642 DEBUG:boto:Method: GET 2024-04-12T03:49:58.643 DEBUG:boto:Path: /dkyges-55/?prefix=d 2024-04-12T03:49:58.643 DEBUG:boto:Data: 2024-04-12T03:49:58.643 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.643 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:49:58.643 DEBUG:boto:Port: 8001 2024-04-12T03:49:58.643 DEBUG:boto:Params: {} 2024-04-12T03:49:58.643 DEBUG:boto:Token: None 2024-04-12T03:49:58.643 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.643 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0= 2024-04-12T03:49:58.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0=', 'Content-Length': '0'} 2024-04-12T03:49:58.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007385ab8d4b6ebe24-006618af66-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.646 DEBUG:boto:b'dkyges-55d1000falsed2024-04-12T03:49:36.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.646 DEBUG:boto:path=/dkyges-55/ 2024-04-12T03:49:58.646 DEBUG:boto:auth_path=/dkyges-55/ 2024-04-12T03:49:58.646 DEBUG:boto:path=/dkyges-55/?prefix=d 2024-04-12T03:49:58.646 DEBUG:boto:auth_path=/dkyges-55/?prefix=d 2024-04-12T03:49:58.646 DEBUG:boto:Method: GET 2024-04-12T03:49:58.647 DEBUG:boto:Path: /dkyges-55/?prefix=d 2024-04-12T03:49:58.647 DEBUG:boto:Data: 2024-04-12T03:49:58.647 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.647 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.647 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.647 DEBUG:boto:Params: {} 2024-04-12T03:49:58.647 DEBUG:boto:Token: None 2024-04-12T03:49:58.647 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:49:58 GMT /dkyges-55/ 2024-04-12T03:49:58.647 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0= 2024-04-12T03:49:58.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHUuA3sAAGM7svffAwIYyEqduf0=', 'Content-Length': '0'} 2024-04-12T03:49:58.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0a9d5345934f59c-006618af66-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.650 DEBUG:boto:b'dkyges-55d1000falsed2024-04-12T03:49:36.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:49:58.653 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-55 zones={a2, a1} 2024-04-12T03:49:58.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:49:58.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:49:58.653 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:49:58.709 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:58.709 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:58.763 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-56 2024-04-12T03:49:58.763 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:49:58.763 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:49:58.763 DEBUG:boto:Method: PUT 2024-04-12T03:49:58.763 DEBUG:boto:Path: /dkyges-56/ 2024-04-12T03:49:58.763 DEBUG:boto:Data: 2024-04-12T03:49:58.763 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.763 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.764 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.764 DEBUG:boto:Params: {} 2024-04-12T03:49:58.764 DEBUG:boto:Token: None 2024-04-12T03:49:58.764 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:49:58 GMT /dkyges-56/ 2024-04-12T03:49:58.764 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SKD2Mfq0FMniajGNpticAQS1Wug= 2024-04-12T03:49:58.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SKD2Mfq0FMniajGNpticAQS1Wug=', 'Content-Length': '0'} 2024-04-12T03:49:58.781 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ca42f1c939171f0e-006618af66-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.781 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:49:58.781 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:49:58.781 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.781 DEBUG:boto:Path: /dkyges-56/ 2024-04-12T03:49:58.782 DEBUG:boto:Data: 2024-04-12T03:49:58.782 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.782 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.782 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.782 DEBUG:boto:Params: {} 2024-04-12T03:49:58.782 DEBUG:boto:Token: None 2024-04-12T03:49:58.782 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-56/ 2024-04-12T03:49:58.782 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YHvs1PRHW24cneBbQwsmbFNL28o= 2024-04-12T03:49:58.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YHvs1PRHW24cneBbQwsmbFNL28o=', 'Content-Length': '0'} 2024-04-12T03:49:58.785 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000519491f99ba638e5-006618af66-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.785 DEBUG:boto:path=/dkyges-56/a 2024-04-12T03:49:58.785 DEBUG:boto:auth_path=/dkyges-56/a 2024-04-12T03:49:58.785 DEBUG:boto:Method: PUT 2024-04-12T03:49:58.785 DEBUG:boto:Path: /dkyges-56/a 2024-04-12T03:49:58.785 DEBUG:boto:Data: 2024-04-12T03:49:58.786 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:49:58.786 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.786 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.786 DEBUG:boto:Params: {} 2024-04-12T03:49:58.786 DEBUG:boto:Token: None 2024-04-12T03:49:58.786 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:49:58 GMT /dkyges-56/a 2024-04-12T03:49:58.786 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kagnt2XSSeleWxhUnSgOicLtdM8= 2024-04-12T03:49:58.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kagnt2XSSeleWxhUnSgOicLtdM8='} 2024-04-12T03:49:58.792 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a1047e84d80d1226-006618af66-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.793 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:49:58.793 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:49:58.793 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.793 DEBUG:boto:Path: /dkyges-56/ 2024-04-12T03:49:58.793 DEBUG:boto:Data: 2024-04-12T03:49:58.793 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.793 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.793 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.793 DEBUG:boto:Params: {} 2024-04-12T03:49:58.793 DEBUG:boto:Token: None 2024-04-12T03:49:58.793 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-56/ 2024-04-12T03:49:58.794 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YHvs1PRHW24cneBbQwsmbFNL28o= 2024-04-12T03:49:58.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YHvs1PRHW24cneBbQwsmbFNL28o=', 'Content-Length': '0'} 2024-04-12T03:49:58.796 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017da142a7d5bede6-006618af66-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.796 DEBUG:boto:path=/dkyges-56/b 2024-04-12T03:49:58.796 DEBUG:boto:auth_path=/dkyges-56/b 2024-04-12T03:49:58.796 DEBUG:boto:Method: PUT 2024-04-12T03:49:58.796 DEBUG:boto:Path: /dkyges-56/b 2024-04-12T03:49:58.796 DEBUG:boto:Data: 2024-04-12T03:49:58.796 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:49:58.797 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.797 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.797 DEBUG:boto:Params: {} 2024-04-12T03:49:58.797 DEBUG:boto:Token: None 2024-04-12T03:49:58.797 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:49:58 GMT /dkyges-56/b 2024-04-12T03:49:58.797 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1ATrZTkfCocKcy6aaoo/oGZycpM= 2024-04-12T03:49:58.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1ATrZTkfCocKcy6aaoo/oGZycpM='} 2024-04-12T03:49:58.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000633754effe770b63-006618af66-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.803 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:49:58.803 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:49:58.803 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.803 DEBUG:boto:Path: /dkyges-56/ 2024-04-12T03:49:58.803 DEBUG:boto:Data: 2024-04-12T03:49:58.803 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.803 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.804 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.804 DEBUG:boto:Params: {} 2024-04-12T03:49:58.804 DEBUG:boto:Token: None 2024-04-12T03:49:58.804 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-56/ 2024-04-12T03:49:58.804 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YHvs1PRHW24cneBbQwsmbFNL28o= 2024-04-12T03:49:58.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YHvs1PRHW24cneBbQwsmbFNL28o=', 'Content-Length': '0'} 2024-04-12T03:49:58.806 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7d340921d1e1989-006618af66-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.806 DEBUG:boto:path=/dkyges-56/c 2024-04-12T03:49:58.806 DEBUG:boto:auth_path=/dkyges-56/c 2024-04-12T03:49:58.807 DEBUG:boto:Method: PUT 2024-04-12T03:49:58.807 DEBUG:boto:Path: /dkyges-56/c 2024-04-12T03:49:58.807 DEBUG:boto:Data: 2024-04-12T03:49:58.807 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:49:58.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.807 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.807 DEBUG:boto:Params: {} 2024-04-12T03:49:58.807 DEBUG:boto:Token: None 2024-04-12T03:49:58.807 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:49:58 GMT /dkyges-56/c 2024-04-12T03:49:58.807 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZT+js3GcHaju+X8XWNCAD6lDbAY= 2024-04-12T03:49:58.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZT+js3GcHaju+X8XWNCAD6lDbAY='} 2024-04-12T03:49:58.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049a478f0f36f9acc-006618af66-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.855 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:49:58.855 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:49:58.855 DEBUG:boto:Method: HEAD 2024-04-12T03:49:58.855 DEBUG:boto:Path: /dkyges-56/ 2024-04-12T03:49:58.855 DEBUG:boto:Data: 2024-04-12T03:49:58.855 DEBUG:boto:Headers: {} 2024-04-12T03:49:58.855 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.855 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.855 DEBUG:boto:Params: {} 2024-04-12T03:49:58.856 DEBUG:boto:Token: None 2024-04-12T03:49:58.856 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:49:58 GMT /dkyges-56/ 2024-04-12T03:49:58.856 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YHvs1PRHW24cneBbQwsmbFNL28o= 2024-04-12T03:49:58.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YHvs1PRHW24cneBbQwsmbFNL28o=', 'Content-Length': '0'} 2024-04-12T03:49:58.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000359f0af2e2943d8a-006618af66-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.859 DEBUG:boto:path=/dkyges-56/d 2024-04-12T03:49:58.859 DEBUG:boto:auth_path=/dkyges-56/d 2024-04-12T03:49:58.859 DEBUG:boto:Method: PUT 2024-04-12T03:49:58.859 DEBUG:boto:Path: /dkyges-56/d 2024-04-12T03:49:58.859 DEBUG:boto:Data: 2024-04-12T03:49:58.859 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:49:58.859 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:49:58.859 DEBUG:boto:Port: 8000 2024-04-12T03:49:58.859 DEBUG:boto:Params: {} 2024-04-12T03:49:58.859 DEBUG:boto:Token: None 2024-04-12T03:49:58.860 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:49:58 GMT /dkyges-56/d 2024-04-12T03:49:58.860 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xxz3ymRcoRPegrzFxSHhsm4JK+s= 2024-04-12T03:49:58.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:49:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xxz3ymRcoRPegrzFxSHhsm4JK+s='} 2024-04-12T03:49:58.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004bb654f8b0310d96-006618af66-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:49:58.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:58.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:58.866 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:49:58.923 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:58.923 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:49:58.976 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:49:58.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:58.976 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:49:58.976 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:49:58.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:58.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:58.977 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:49:58.977 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:49:58.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:58.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:58.977 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:49:58.977 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:49:58.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:58.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:58.977 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893798.786054_489.1", 2024-04-12T03:49:58.978 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:49:58.786054Z" 2024-04-12T03:49:58.978 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:58.978 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:49:58.979 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893798.786054_489.1'} 2024-04-12T03:49:58.979 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:49:58.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:58.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:49:58.979 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:49:59.038 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:49:59.038 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:49:59.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:59.077+0000 7f705e7d8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:49:59.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:49:59.079+0000 7f705e7d8780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:49:59.090 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:49:59.090 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:49:59.090 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:49:59.090 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:49:59.090 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:49:59.090 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:49:59.090 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:59.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:59.090 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:49:59.090 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:59.091 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:49:59.091 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:59.091 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:59.091 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:49:59.091 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:59.091 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:49:59.091 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:59.091 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:49:59.091 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:59.091 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:59.091 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:59.092 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:59.092 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:49:59.092 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:59.092 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:59.092 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:49:59.092 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:59.092 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:59.092 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:59.092 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:49:59.092 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:59.092 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:59.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:59.093 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:59.093 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:49:59.093 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:59.093 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:59.093 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:49:59.093 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:59.093 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:59.093 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:59.093 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:49:59.093 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:59.093 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:59.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:59.094 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:49:59.094 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:49:59.094 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:49:59.094 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:49:59.094 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893798.778988_486.1", 2024-04-12T03:49:59.094 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:49:59.094 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:49:59.094 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:49:59.094 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:49:58.778988Z", 2024-04-12T03:49:59.094 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:49:59.095 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:59.095 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:49:59.095 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:49:59.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:59.095 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:49:59.095 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:49:59.095 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:49:59.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:49:59.095 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:49:59Z" 2024-04-12T03:49:59.095 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:49:59.096 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893798.778988_486.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:49:58.778988Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:49:59Z" } 2024-04-12T03:49:59.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893798.778988_486.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:49:58.778988Z', 'realm_epoch': 2}}] 2024-04-12T03:49:59.097 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893798.786054_489.1'} 2024-04-12T03:49:59.097 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893798.778988_486.1'} 2024-04-12T03:49:59.097 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893798.786054_489.1 target=1_1712893798.778988_486.1 2024-04-12T03:50:04.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:04.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:04.098 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:50:04.157 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:04.157 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:04.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:04.198+0000 7f509ff19780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:50:04.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:04.199+0000 7f509ff19780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:50:04.209 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:50:04.210 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:50:04.210 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:50:04.210 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:50:04.210 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:50:04.210 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:50:04.210 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:50:04.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:04.210 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:50:04.210 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:04.211 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:50:04.211 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:50:04.211 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:50:04.211 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:50:04.211 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:50:04.211 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:50:04.211 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:50:04.211 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:50:04.211 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:50:04.211 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:04.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:04.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:04.212 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:50:04.212 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:50:04.212 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:50:04.212 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:50:04.212 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:50:04.212 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:50:04.212 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:50:04.212 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:50:04.213 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:50:04.213 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:04.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:04.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:04.213 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:50:04.213 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:50:04.213 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:50:04.213 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:50:04.213 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:50:04.213 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:50:04.214 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:50:04.214 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:50:04.214 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:50:04.214 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:04.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:04.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:04.214 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:50:04.214 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:50:04.214 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:50:04.214 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893798.786054_489.1", 2024-04-12T03:50:04.214 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:50:04.215 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:50:04.215 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:50:04.215 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:49:58.786054Z", 2024-04-12T03:50:04.215 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:50:04.215 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:04.215 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:04.215 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:50:04.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:04.215 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:50:04.215 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:50:04.215 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:50:04.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:04.216 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:50:04Z" 2024-04-12T03:50:04.216 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:50:04.217 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893798.786054_489.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:49:58.786054Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:50:04Z" } 2024-04-12T03:50:04.217 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893798.786054_489.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:49:58.786054Z', 'realm_epoch': 2}}] 2024-04-12T03:50:04.217 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893798.786054_489.1'} 2024-04-12T03:50:04.217 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893798.786054_489.1'} 2024-04-12T03:50:04.217 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:50:04.217 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-56 2024-04-12T03:50:04.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:50:04.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:50:04.218 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:50:04.282 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:04.282 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:04.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:04.329+0000 7fd968420780 1 waiting to reach incremental sync.. 2024-04-12T03:50:09.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:09.330+0000 7fd968420780 1 waiting to reach incremental sync.. 2024-04-12T03:50:14.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:14.332+0000 7fd968420780 1 waiting to reach incremental sync.. 2024-04-12T03:50:19.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:19.333+0000 7fd968420780 1 bucket sync caught up with source: 2024-04-12T03:50:19.328 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.24.6, , , , 00000000001.18.6, , , 00000000001.58.6, , 00000000001.59.6] 2024-04-12T03:50:19.328 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.24.6, , , , 00000000001.18.6, , , 00000000001.58.6, , 00000000001.59.6] 2024-04-12T03:50:19.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:19.333+0000 7fd968420780 0 bucket checkpoint complete 2024-04-12T03:50:19.346 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-56 2024-04-12T03:50:19.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:50:19.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:50:19.347 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:50:19.413 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:19.413 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:19.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:19.460+0000 7ff9cbf87780 1 bucket sync caught up with source: 2024-04-12T03:50:19.454 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.549.6, , , , 00000000001.2065.6, , , 00000000001.3615.6, , 00000000001.2786.6] 2024-04-12T03:50:19.454 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.549.6, , , , 00000000001.2065.6, , , 00000000001.3615.6, , 00000000001.2786.6] 2024-04-12T03:50:19.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:19.460+0000 7ff9cbf87780 0 bucket checkpoint complete 2024-04-12T03:50:19.470 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-56 zones={a2, a1} 2024-04-12T03:50:19.470 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.470 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.470 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.470 DEBUG:boto:Path: /dkyges-56/ 2024-04-12T03:50:19.470 DEBUG:boto:Data: 2024-04-12T03:50:19.470 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.470 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.470 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.471 DEBUG:boto:Params: {} 2024-04-12T03:50:19.471 DEBUG:boto:Token: None 2024-04-12T03:50:19.471 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.471 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C/+iJ7cHUOFP75esylt98lFbXCI= 2024-04-12T03:50:19.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C/+iJ7cHUOFP75esylt98lFbXCI=', 'Content-Length': '0'} 2024-04-12T03:50:19.474 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dfb8535f7df56e12-006618af7b-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.474 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.474 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.474 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.475 DEBUG:boto:Path: /dkyges-56/ 2024-04-12T03:50:19.475 DEBUG:boto:Data: 2024-04-12T03:50:19.475 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.475 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.475 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.475 DEBUG:boto:Params: {} 2024-04-12T03:50:19.475 DEBUG:boto:Token: None 2024-04-12T03:50:19.475 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.475 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C/+iJ7cHUOFP75esylt98lFbXCI= 2024-04-12T03:50:19.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C/+iJ7cHUOFP75esylt98lFbXCI=', 'Content-Length': '0'} 2024-04-12T03:50:19.478 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e7c983bf1ed130a-006618af7b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.478 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:50:19.478 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.478 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.478 DEBUG:boto:path=/dkyges-56/?versions&max-keys=999 2024-04-12T03:50:19.478 DEBUG:boto:auth_path=/dkyges-56/?versions&max-keys=999 2024-04-12T03:50:19.478 DEBUG:boto:Method: GET 2024-04-12T03:50:19.479 DEBUG:boto:Path: /dkyges-56/?versions&max-keys=999 2024-04-12T03:50:19.479 DEBUG:boto:Data: 2024-04-12T03:50:19.479 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.479 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.479 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.479 DEBUG:boto:Params: {} 2024-04-12T03:50:19.479 DEBUG:boto:Token: None 2024-04-12T03:50:19.479 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/?versions 2024-04-12T03:50:19.479 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UhtlNCXafzYgTHrgBV4wqGcqzAc= 2024-04-12T03:50:19.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UhtlNCXafzYgTHrgBV4wqGcqzAc=', 'Content-Length': '0'} 2024-04-12T03:50:19.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d18bf7dab24ddc16-006618af7b-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.482 DEBUG:boto:b'dkyges-56999falseanulltrue2024-04-12T03:49:58.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:49:58.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:49:58.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:49:58.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.482 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:50:19.482 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:50:19.483 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:50:19.483 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:50:19.483 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:50:19.483 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.483 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.483 DEBUG:boto:path=/dkyges-56/?versions&max-keys=999 2024-04-12T03:50:19.483 DEBUG:boto:auth_path=/dkyges-56/?versions&max-keys=999 2024-04-12T03:50:19.483 DEBUG:boto:Method: GET 2024-04-12T03:50:19.483 DEBUG:boto:Path: /dkyges-56/?versions&max-keys=999 2024-04-12T03:50:19.483 DEBUG:boto:Data: 2024-04-12T03:50:19.483 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.483 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.484 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.484 DEBUG:boto:Params: {} 2024-04-12T03:50:19.484 DEBUG:boto:Token: None 2024-04-12T03:50:19.484 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/?versions 2024-04-12T03:50:19.484 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UhtlNCXafzYgTHrgBV4wqGcqzAc= 2024-04-12T03:50:19.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UhtlNCXafzYgTHrgBV4wqGcqzAc=', 'Content-Length': '0'} 2024-04-12T03:50:19.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002337927a1e056caf-006618af7b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.487 DEBUG:boto:b'dkyges-56999falseanulltrue2024-04-12T03:49:58.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:49:58.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:49:58.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:49:58.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.487 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:50:19.487 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:50:19.488 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:50:19.488 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:50:19.488 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.488 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.488 DEBUG:boto:path=/dkyges-56/?versions&max-keys=999 2024-04-12T03:50:19.488 DEBUG:boto:auth_path=/dkyges-56/?versions&max-keys=999 2024-04-12T03:50:19.488 DEBUG:boto:Method: GET 2024-04-12T03:50:19.488 DEBUG:boto:Path: /dkyges-56/?versions&max-keys=999 2024-04-12T03:50:19.488 DEBUG:boto:Data: 2024-04-12T03:50:19.489 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.489 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.489 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.489 DEBUG:boto:Params: {} 2024-04-12T03:50:19.489 DEBUG:boto:Token: None 2024-04-12T03:50:19.489 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/?versions 2024-04-12T03:50:19.489 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UhtlNCXafzYgTHrgBV4wqGcqzAc= 2024-04-12T03:50:19.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UhtlNCXafzYgTHrgBV4wqGcqzAc=', 'Content-Length': '0'} 2024-04-12T03:50:19.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000591cd79404a86ec8-006618af7b-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.492 DEBUG:boto:b'dkyges-56999falseanulltrue2024-04-12T03:49:58.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:49:58.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:49:58.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:49:58.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.492 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.493 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.493 DEBUG:boto:path=/dkyges-56/?versions&max-keys=999 2024-04-12T03:50:19.493 DEBUG:boto:auth_path=/dkyges-56/?versions&max-keys=999 2024-04-12T03:50:19.493 DEBUG:boto:Method: GET 2024-04-12T03:50:19.493 DEBUG:boto:Path: /dkyges-56/?versions&max-keys=999 2024-04-12T03:50:19.493 DEBUG:boto:Data: 2024-04-12T03:50:19.493 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.493 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.493 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.493 DEBUG:boto:Params: {} 2024-04-12T03:50:19.494 DEBUG:boto:Token: None 2024-04-12T03:50:19.494 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/?versions 2024-04-12T03:50:19.494 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UhtlNCXafzYgTHrgBV4wqGcqzAc= 2024-04-12T03:50:19.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UhtlNCXafzYgTHrgBV4wqGcqzAc=', 'Content-Length': '0'} 2024-04-12T03:50:19.496 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091fc01122766e67c-006618af7b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.497 DEBUG:boto:b'dkyges-56999falseanulltrue2024-04-12T03:49:58.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:49:58.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:49:58.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:49:58.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:50:19.497 DEBUG:boto:path=/dkyges-56/a 2024-04-12T03:50:19.497 DEBUG:boto:auth_path=/dkyges-56/a 2024-04-12T03:50:19.498 DEBUG:boto:path=/dkyges-56/a?versionId=null 2024-04-12T03:50:19.498 DEBUG:boto:auth_path=/dkyges-56/a?versionId=null 2024-04-12T03:50:19.498 DEBUG:boto:Method: GET 2024-04-12T03:50:19.498 DEBUG:boto:Path: /dkyges-56/a?versionId=null 2024-04-12T03:50:19.498 DEBUG:boto:Data: 2024-04-12T03:50:19.498 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.498 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.498 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.498 DEBUG:boto:Params: {} 2024-04-12T03:50:19.498 DEBUG:boto:Token: None 2024-04-12T03:50:19.499 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/a?versionId=null 2024-04-12T03:50:19.499 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/a9FNAAontVW9hWykJklbmDmctk= 2024-04-12T03:50:19.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/a9FNAAontVW9hWykJklbmDmctk=', 'Content-Length': '0'} 2024-04-12T03:50:19.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fee799fd32c31361-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.501 DEBUG:boto:path=/dkyges-56/a 2024-04-12T03:50:19.501 DEBUG:boto:auth_path=/dkyges-56/a 2024-04-12T03:50:19.501 DEBUG:boto:path=/dkyges-56/a?versionId=null 2024-04-12T03:50:19.501 DEBUG:boto:auth_path=/dkyges-56/a?versionId=null 2024-04-12T03:50:19.501 DEBUG:boto:Method: GET 2024-04-12T03:50:19.501 DEBUG:boto:Path: /dkyges-56/a?versionId=null 2024-04-12T03:50:19.502 DEBUG:boto:Data: 2024-04-12T03:50:19.502 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.502 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.502 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.502 DEBUG:boto:Params: {} 2024-04-12T03:50:19.502 DEBUG:boto:Token: None 2024-04-12T03:50:19.502 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/a?versionId=null 2024-04-12T03:50:19.502 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/a9FNAAontVW9hWykJklbmDmctk= 2024-04-12T03:50:19.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/a9FNAAontVW9hWykJklbmDmctk=', 'Content-Length': '0'} 2024-04-12T03:50:19.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdce201058018cbe-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.550 DEBUG:boto:path=/dkyges-56/a 2024-04-12T03:50:19.551 DEBUG:boto:auth_path=/dkyges-56/a 2024-04-12T03:50:19.551 DEBUG:boto:path=/dkyges-56/a?versionId=null 2024-04-12T03:50:19.551 DEBUG:boto:auth_path=/dkyges-56/a?versionId=null 2024-04-12T03:50:19.551 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.551 DEBUG:boto:Path: /dkyges-56/a?versionId=null 2024-04-12T03:50:19.551 DEBUG:boto:Data: 2024-04-12T03:50:19.551 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.551 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.551 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.551 DEBUG:boto:Params: {} 2024-04-12T03:50:19.552 DEBUG:boto:Token: None 2024-04-12T03:50:19.552 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/a?versionId=null 2024-04-12T03:50:19.552 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JF1zmheUtfEyxgLYImUspQpHZSU= 2024-04-12T03:50:19.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JF1zmheUtfEyxgLYImUspQpHZSU=', 'Content-Length': '0'} 2024-04-12T03:50:19.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dca3f807af553d1f-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.554 DEBUG:boto:path=/dkyges-56/a 2024-04-12T03:50:19.554 DEBUG:boto:auth_path=/dkyges-56/a 2024-04-12T03:50:19.554 DEBUG:boto:path=/dkyges-56/a?versionId=null 2024-04-12T03:50:19.554 DEBUG:boto:auth_path=/dkyges-56/a?versionId=null 2024-04-12T03:50:19.555 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.555 DEBUG:boto:Path: /dkyges-56/a?versionId=null 2024-04-12T03:50:19.555 DEBUG:boto:Data: 2024-04-12T03:50:19.555 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.555 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.555 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.555 DEBUG:boto:Params: {} 2024-04-12T03:50:19.555 DEBUG:boto:Token: None 2024-04-12T03:50:19.555 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/a?versionId=null 2024-04-12T03:50:19.555 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JF1zmheUtfEyxgLYImUspQpHZSU= 2024-04-12T03:50:19.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JF1zmheUtfEyxgLYImUspQpHZSU=', 'Content-Length': '0'} 2024-04-12T03:50:19.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035bfcc912f2050fd-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:50:19.558 DEBUG:boto:path=/dkyges-56/a 2024-04-12T03:50:19.558 DEBUG:boto:auth_path=/dkyges-56/a 2024-04-12T03:50:19.558 DEBUG:boto:path=/dkyges-56/a?versionId=null 2024-04-12T03:50:19.558 DEBUG:boto:auth_path=/dkyges-56/a?versionId=null 2024-04-12T03:50:19.558 DEBUG:boto:Method: GET 2024-04-12T03:50:19.558 DEBUG:boto:Path: /dkyges-56/a?versionId=null 2024-04-12T03:50:19.558 DEBUG:boto:Data: 2024-04-12T03:50:19.558 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.559 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.559 DEBUG:boto:Params: {} 2024-04-12T03:50:19.559 DEBUG:boto:Token: None 2024-04-12T03:50:19.559 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/a?versionId=null 2024-04-12T03:50:19.559 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/a9FNAAontVW9hWykJklbmDmctk= 2024-04-12T03:50:19.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/a9FNAAontVW9hWykJklbmDmctk=', 'Content-Length': '0'} 2024-04-12T03:50:19.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfa470eacb545ad1-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.561 DEBUG:boto:path=/dkyges-56/a 2024-04-12T03:50:19.561 DEBUG:boto:auth_path=/dkyges-56/a 2024-04-12T03:50:19.561 DEBUG:boto:path=/dkyges-56/a?versionId=null 2024-04-12T03:50:19.562 DEBUG:boto:auth_path=/dkyges-56/a?versionId=null 2024-04-12T03:50:19.562 DEBUG:boto:Method: GET 2024-04-12T03:50:19.562 DEBUG:boto:Path: /dkyges-56/a?versionId=null 2024-04-12T03:50:19.562 DEBUG:boto:Data: 2024-04-12T03:50:19.562 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.562 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.562 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.562 DEBUG:boto:Params: {} 2024-04-12T03:50:19.562 DEBUG:boto:Token: None 2024-04-12T03:50:19.562 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/a?versionId=null 2024-04-12T03:50:19.563 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/a9FNAAontVW9hWykJklbmDmctk= 2024-04-12T03:50:19.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/a9FNAAontVW9hWykJklbmDmctk=', 'Content-Length': '0'} 2024-04-12T03:50:19.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025040e774693bca9-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.565 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.565 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.565 DEBUG:boto:path=/dkyges-56/?prefix=a 2024-04-12T03:50:19.565 DEBUG:boto:auth_path=/dkyges-56/?prefix=a 2024-04-12T03:50:19.565 DEBUG:boto:Method: GET 2024-04-12T03:50:19.565 DEBUG:boto:Path: /dkyges-56/?prefix=a 2024-04-12T03:50:19.565 DEBUG:boto:Data: 2024-04-12T03:50:19.565 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.565 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.565 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.565 DEBUG:boto:Params: {} 2024-04-12T03:50:19.566 DEBUG:boto:Token: None 2024-04-12T03:50:19.566 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.566 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50= 2024-04-12T03:50:19.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50=', 'Content-Length': '0'} 2024-04-12T03:50:19.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efc2d12cff5e4613-006618af7b-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.568 DEBUG:boto:b'dkyges-56a1000falsea2024-04-12T03:49:58.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.569 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.569 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.569 DEBUG:boto:path=/dkyges-56/?prefix=a 2024-04-12T03:50:19.569 DEBUG:boto:auth_path=/dkyges-56/?prefix=a 2024-04-12T03:50:19.569 DEBUG:boto:Method: GET 2024-04-12T03:50:19.570 DEBUG:boto:Path: /dkyges-56/?prefix=a 2024-04-12T03:50:19.570 DEBUG:boto:Data: 2024-04-12T03:50:19.570 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.570 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.570 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.570 DEBUG:boto:Params: {} 2024-04-12T03:50:19.570 DEBUG:boto:Token: None 2024-04-12T03:50:19.571 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.571 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50= 2024-04-12T03:50:19.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50=', 'Content-Length': '0'} 2024-04-12T03:50:19.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfdab966e21c1ec1-006618af7b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.573 DEBUG:boto:b'dkyges-56a1000falsea2024-04-12T03:49:58.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.574 DEBUG:boto:path=/dkyges-56/a 2024-04-12T03:50:19.574 DEBUG:boto:auth_path=/dkyges-56/a 2024-04-12T03:50:19.574 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.574 DEBUG:boto:Path: /dkyges-56/a 2024-04-12T03:50:19.574 DEBUG:boto:Data: 2024-04-12T03:50:19.575 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.575 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.575 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.575 DEBUG:boto:Params: {} 2024-04-12T03:50:19.575 DEBUG:boto:Token: None 2024-04-12T03:50:19.575 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/a 2024-04-12T03:50:19.575 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OAndqfGhWkTa3VJ//pfoDwj2I7s= 2024-04-12T03:50:19.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OAndqfGhWkTa3VJ//pfoDwj2I7s=', 'Content-Length': '0'} 2024-04-12T03:50:19.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c9a816b74e52337-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.577 DEBUG:boto:path=/dkyges-56/a 2024-04-12T03:50:19.578 DEBUG:boto:auth_path=/dkyges-56/a 2024-04-12T03:50:19.578 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.578 DEBUG:boto:Path: /dkyges-56/a 2024-04-12T03:50:19.578 DEBUG:boto:Data: 2024-04-12T03:50:19.578 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.578 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.578 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.578 DEBUG:boto:Params: {} 2024-04-12T03:50:19.578 DEBUG:boto:Token: None 2024-04-12T03:50:19.579 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/a 2024-04-12T03:50:19.579 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OAndqfGhWkTa3VJ//pfoDwj2I7s= 2024-04-12T03:50:19.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OAndqfGhWkTa3VJ//pfoDwj2I7s=', 'Content-Length': '0'} 2024-04-12T03:50:19.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011d06760cb24d8f6-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:50:19.581 DEBUG:boto:path=/dkyges-56/a 2024-04-12T03:50:19.581 DEBUG:boto:auth_path=/dkyges-56/a 2024-04-12T03:50:19.581 DEBUG:boto:Method: GET 2024-04-12T03:50:19.581 DEBUG:boto:Path: /dkyges-56/a 2024-04-12T03:50:19.581 DEBUG:boto:Data: 2024-04-12T03:50:19.582 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.582 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.582 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.582 DEBUG:boto:Params: {} 2024-04-12T03:50:19.582 DEBUG:boto:Token: None 2024-04-12T03:50:19.582 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/a 2024-04-12T03:50:19.582 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VZLm2WCXNt65J4S2iEerOmg5c78= 2024-04-12T03:50:19.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VZLm2WCXNt65J4S2iEerOmg5c78=', 'Content-Length': '0'} 2024-04-12T03:50:19.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c266c942cac737c-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.584 DEBUG:boto:path=/dkyges-56/a 2024-04-12T03:50:19.584 DEBUG:boto:auth_path=/dkyges-56/a 2024-04-12T03:50:19.585 DEBUG:boto:Method: GET 2024-04-12T03:50:19.585 DEBUG:boto:Path: /dkyges-56/a 2024-04-12T03:50:19.585 DEBUG:boto:Data: 2024-04-12T03:50:19.585 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.585 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.585 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.585 DEBUG:boto:Params: {} 2024-04-12T03:50:19.585 DEBUG:boto:Token: None 2024-04-12T03:50:19.585 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/a 2024-04-12T03:50:19.586 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VZLm2WCXNt65J4S2iEerOmg5c78= 2024-04-12T03:50:19.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VZLm2WCXNt65J4S2iEerOmg5c78=', 'Content-Length': '0'} 2024-04-12T03:50:19.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c474f15ba152f0f3-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.588 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.588 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.588 DEBUG:boto:path=/dkyges-56/?prefix=a 2024-04-12T03:50:19.588 DEBUG:boto:auth_path=/dkyges-56/?prefix=a 2024-04-12T03:50:19.588 DEBUG:boto:Method: GET 2024-04-12T03:50:19.588 DEBUG:boto:Path: /dkyges-56/?prefix=a 2024-04-12T03:50:19.588 DEBUG:boto:Data: 2024-04-12T03:50:19.589 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.589 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.589 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.589 DEBUG:boto:Params: {} 2024-04-12T03:50:19.589 DEBUG:boto:Token: None 2024-04-12T03:50:19.589 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.589 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50= 2024-04-12T03:50:19.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50=', 'Content-Length': '0'} 2024-04-12T03:50:19.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000963f27d7bfa77218-006618af7b-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.592 DEBUG:boto:b'dkyges-56a1000falsea2024-04-12T03:49:58.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.592 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.592 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.593 DEBUG:boto:path=/dkyges-56/?prefix=a 2024-04-12T03:50:19.593 DEBUG:boto:auth_path=/dkyges-56/?prefix=a 2024-04-12T03:50:19.593 DEBUG:boto:Method: GET 2024-04-12T03:50:19.593 DEBUG:boto:Path: /dkyges-56/?prefix=a 2024-04-12T03:50:19.593 DEBUG:boto:Data: 2024-04-12T03:50:19.593 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.593 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.593 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.593 DEBUG:boto:Params: {} 2024-04-12T03:50:19.594 DEBUG:boto:Token: None 2024-04-12T03:50:19.594 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.594 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50= 2024-04-12T03:50:19.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50=', 'Content-Length': '0'} 2024-04-12T03:50:19.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6a4f197535176e4-006618af7b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.597 DEBUG:boto:b'dkyges-56a1000falsea2024-04-12T03:49:58.794Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:50:19.597 DEBUG:boto:path=/dkyges-56/b 2024-04-12T03:50:19.597 DEBUG:boto:auth_path=/dkyges-56/b 2024-04-12T03:50:19.597 DEBUG:boto:path=/dkyges-56/b?versionId=null 2024-04-12T03:50:19.598 DEBUG:boto:auth_path=/dkyges-56/b?versionId=null 2024-04-12T03:50:19.598 DEBUG:boto:Method: GET 2024-04-12T03:50:19.598 DEBUG:boto:Path: /dkyges-56/b?versionId=null 2024-04-12T03:50:19.598 DEBUG:boto:Data: 2024-04-12T03:50:19.598 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.598 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.598 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.598 DEBUG:boto:Params: {} 2024-04-12T03:50:19.598 DEBUG:boto:Token: None 2024-04-12T03:50:19.599 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/b?versionId=null 2024-04-12T03:50:19.599 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5YI0e6tXkABtpPThWMBWhW1gtwY= 2024-04-12T03:50:19.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5YI0e6tXkABtpPThWMBWhW1gtwY=', 'Content-Length': '0'} 2024-04-12T03:50:19.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080f63cc0c84660f7-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.601 DEBUG:boto:path=/dkyges-56/b 2024-04-12T03:50:19.601 DEBUG:boto:auth_path=/dkyges-56/b 2024-04-12T03:50:19.601 DEBUG:boto:path=/dkyges-56/b?versionId=null 2024-04-12T03:50:19.602 DEBUG:boto:auth_path=/dkyges-56/b?versionId=null 2024-04-12T03:50:19.602 DEBUG:boto:Method: GET 2024-04-12T03:50:19.602 DEBUG:boto:Path: /dkyges-56/b?versionId=null 2024-04-12T03:50:19.602 DEBUG:boto:Data: 2024-04-12T03:50:19.602 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.602 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.602 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.602 DEBUG:boto:Params: {} 2024-04-12T03:50:19.602 DEBUG:boto:Token: None 2024-04-12T03:50:19.602 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/b?versionId=null 2024-04-12T03:50:19.602 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5YI0e6tXkABtpPThWMBWhW1gtwY= 2024-04-12T03:50:19.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5YI0e6tXkABtpPThWMBWhW1gtwY=', 'Content-Length': '0'} 2024-04-12T03:50:19.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040faa2d889c92554-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.604 DEBUG:boto:path=/dkyges-56/b 2024-04-12T03:50:19.604 DEBUG:boto:auth_path=/dkyges-56/b 2024-04-12T03:50:19.605 DEBUG:boto:path=/dkyges-56/b?versionId=null 2024-04-12T03:50:19.605 DEBUG:boto:auth_path=/dkyges-56/b?versionId=null 2024-04-12T03:50:19.605 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.605 DEBUG:boto:Path: /dkyges-56/b?versionId=null 2024-04-12T03:50:19.605 DEBUG:boto:Data: 2024-04-12T03:50:19.605 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.605 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.605 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.605 DEBUG:boto:Params: {} 2024-04-12T03:50:19.605 DEBUG:boto:Token: None 2024-04-12T03:50:19.605 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/b?versionId=null 2024-04-12T03:50:19.606 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2CCqvFTO9zwnMz/h7ms79XIkk7U= 2024-04-12T03:50:19.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2CCqvFTO9zwnMz/h7ms79XIkk7U=', 'Content-Length': '0'} 2024-04-12T03:50:19.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055363397de130ff0-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.607 DEBUG:boto:path=/dkyges-56/b 2024-04-12T03:50:19.608 DEBUG:boto:auth_path=/dkyges-56/b 2024-04-12T03:50:19.608 DEBUG:boto:path=/dkyges-56/b?versionId=null 2024-04-12T03:50:19.608 DEBUG:boto:auth_path=/dkyges-56/b?versionId=null 2024-04-12T03:50:19.608 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.608 DEBUG:boto:Path: /dkyges-56/b?versionId=null 2024-04-12T03:50:19.608 DEBUG:boto:Data: 2024-04-12T03:50:19.608 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.608 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.608 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.608 DEBUG:boto:Params: {} 2024-04-12T03:50:19.608 DEBUG:boto:Token: None 2024-04-12T03:50:19.608 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/b?versionId=null 2024-04-12T03:50:19.609 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2CCqvFTO9zwnMz/h7ms79XIkk7U= 2024-04-12T03:50:19.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2CCqvFTO9zwnMz/h7ms79XIkk7U=', 'Content-Length': '0'} 2024-04-12T03:50:19.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006023b29d0eb9a33d-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:50:19.611 DEBUG:boto:path=/dkyges-56/b 2024-04-12T03:50:19.611 DEBUG:boto:auth_path=/dkyges-56/b 2024-04-12T03:50:19.611 DEBUG:boto:path=/dkyges-56/b?versionId=null 2024-04-12T03:50:19.611 DEBUG:boto:auth_path=/dkyges-56/b?versionId=null 2024-04-12T03:50:19.611 DEBUG:boto:Method: GET 2024-04-12T03:50:19.611 DEBUG:boto:Path: /dkyges-56/b?versionId=null 2024-04-12T03:50:19.611 DEBUG:boto:Data: 2024-04-12T03:50:19.611 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.611 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.612 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.612 DEBUG:boto:Params: {} 2024-04-12T03:50:19.612 DEBUG:boto:Token: None 2024-04-12T03:50:19.612 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/b?versionId=null 2024-04-12T03:50:19.612 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5YI0e6tXkABtpPThWMBWhW1gtwY= 2024-04-12T03:50:19.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5YI0e6tXkABtpPThWMBWhW1gtwY=', 'Content-Length': '0'} 2024-04-12T03:50:19.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000020bf283b48da842-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.654 DEBUG:boto:path=/dkyges-56/b 2024-04-12T03:50:19.655 DEBUG:boto:auth_path=/dkyges-56/b 2024-04-12T03:50:19.655 DEBUG:boto:path=/dkyges-56/b?versionId=null 2024-04-12T03:50:19.655 DEBUG:boto:auth_path=/dkyges-56/b?versionId=null 2024-04-12T03:50:19.655 DEBUG:boto:Method: GET 2024-04-12T03:50:19.655 DEBUG:boto:Path: /dkyges-56/b?versionId=null 2024-04-12T03:50:19.655 DEBUG:boto:Data: 2024-04-12T03:50:19.655 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.655 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.655 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.655 DEBUG:boto:Params: {} 2024-04-12T03:50:19.656 DEBUG:boto:Token: None 2024-04-12T03:50:19.656 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/b?versionId=null 2024-04-12T03:50:19.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5YI0e6tXkABtpPThWMBWhW1gtwY= 2024-04-12T03:50:19.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5YI0e6tXkABtpPThWMBWhW1gtwY=', 'Content-Length': '0'} 2024-04-12T03:50:19.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015d3c0602aa7556f-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.658 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.659 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.659 DEBUG:boto:path=/dkyges-56/?prefix=b 2024-04-12T03:50:19.659 DEBUG:boto:auth_path=/dkyges-56/?prefix=b 2024-04-12T03:50:19.659 DEBUG:boto:Method: GET 2024-04-12T03:50:19.659 DEBUG:boto:Path: /dkyges-56/?prefix=b 2024-04-12T03:50:19.659 DEBUG:boto:Data: 2024-04-12T03:50:19.659 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.659 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.659 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.659 DEBUG:boto:Params: {} 2024-04-12T03:50:19.660 DEBUG:boto:Token: None 2024-04-12T03:50:19.660 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.660 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50= 2024-04-12T03:50:19.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50=', 'Content-Length': '0'} 2024-04-12T03:50:19.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdcddbb054f0361c-006618af7b-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.663 DEBUG:boto:b'dkyges-56b1000falseb2024-04-12T03:49:58.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.663 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.663 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.663 DEBUG:boto:path=/dkyges-56/?prefix=b 2024-04-12T03:50:19.663 DEBUG:boto:auth_path=/dkyges-56/?prefix=b 2024-04-12T03:50:19.663 DEBUG:boto:Method: GET 2024-04-12T03:50:19.664 DEBUG:boto:Path: /dkyges-56/?prefix=b 2024-04-12T03:50:19.664 DEBUG:boto:Data: 2024-04-12T03:50:19.664 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.664 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.664 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.664 DEBUG:boto:Params: {} 2024-04-12T03:50:19.664 DEBUG:boto:Token: None 2024-04-12T03:50:19.664 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.664 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50= 2024-04-12T03:50:19.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50=', 'Content-Length': '0'} 2024-04-12T03:50:19.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000429d3223b219f3af-006618af7b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.667 DEBUG:boto:b'dkyges-56b1000falseb2024-04-12T03:49:58.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.667 DEBUG:boto:path=/dkyges-56/b 2024-04-12T03:50:19.667 DEBUG:boto:auth_path=/dkyges-56/b 2024-04-12T03:50:19.667 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.667 DEBUG:boto:Path: /dkyges-56/b 2024-04-12T03:50:19.668 DEBUG:boto:Data: 2024-04-12T03:50:19.668 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.668 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.668 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.668 DEBUG:boto:Params: {} 2024-04-12T03:50:19.668 DEBUG:boto:Token: None 2024-04-12T03:50:19.668 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/b 2024-04-12T03:50:19.668 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8Qe08ZyN+afqjzn9+DILHlxrbh0= 2024-04-12T03:50:19.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8Qe08ZyN+afqjzn9+DILHlxrbh0=', 'Content-Length': '0'} 2024-04-12T03:50:19.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad142397560b8f04-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.670 DEBUG:boto:path=/dkyges-56/b 2024-04-12T03:50:19.671 DEBUG:boto:auth_path=/dkyges-56/b 2024-04-12T03:50:19.671 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.671 DEBUG:boto:Path: /dkyges-56/b 2024-04-12T03:50:19.671 DEBUG:boto:Data: 2024-04-12T03:50:19.671 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.671 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.671 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.671 DEBUG:boto:Params: {} 2024-04-12T03:50:19.671 DEBUG:boto:Token: None 2024-04-12T03:50:19.672 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/b 2024-04-12T03:50:19.672 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8Qe08ZyN+afqjzn9+DILHlxrbh0= 2024-04-12T03:50:19.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8Qe08ZyN+afqjzn9+DILHlxrbh0=', 'Content-Length': '0'} 2024-04-12T03:50:19.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fed201a91f97b34-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:50:19.674 DEBUG:boto:path=/dkyges-56/b 2024-04-12T03:50:19.674 DEBUG:boto:auth_path=/dkyges-56/b 2024-04-12T03:50:19.674 DEBUG:boto:Method: GET 2024-04-12T03:50:19.674 DEBUG:boto:Path: /dkyges-56/b 2024-04-12T03:50:19.674 DEBUG:boto:Data: 2024-04-12T03:50:19.674 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.675 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.675 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.675 DEBUG:boto:Params: {} 2024-04-12T03:50:19.675 DEBUG:boto:Token: None 2024-04-12T03:50:19.675 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/b 2024-04-12T03:50:19.675 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Lxuc9mDwRTpm/5FHt4dUAqvSAT0= 2024-04-12T03:50:19.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Lxuc9mDwRTpm/5FHt4dUAqvSAT0=', 'Content-Length': '0'} 2024-04-12T03:50:19.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f2b780060f9a553-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.677 DEBUG:boto:path=/dkyges-56/b 2024-04-12T03:50:19.677 DEBUG:boto:auth_path=/dkyges-56/b 2024-04-12T03:50:19.678 DEBUG:boto:Method: GET 2024-04-12T03:50:19.678 DEBUG:boto:Path: /dkyges-56/b 2024-04-12T03:50:19.678 DEBUG:boto:Data: 2024-04-12T03:50:19.678 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.678 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.678 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.678 DEBUG:boto:Params: {} 2024-04-12T03:50:19.678 DEBUG:boto:Token: None 2024-04-12T03:50:19.678 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/b 2024-04-12T03:50:19.678 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Lxuc9mDwRTpm/5FHt4dUAqvSAT0= 2024-04-12T03:50:19.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Lxuc9mDwRTpm/5FHt4dUAqvSAT0=', 'Content-Length': '0'} 2024-04-12T03:50:19.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057a493ed8d46aba5-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.681 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.681 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.681 DEBUG:boto:path=/dkyges-56/?prefix=b 2024-04-12T03:50:19.681 DEBUG:boto:auth_path=/dkyges-56/?prefix=b 2024-04-12T03:50:19.681 DEBUG:boto:Method: GET 2024-04-12T03:50:19.681 DEBUG:boto:Path: /dkyges-56/?prefix=b 2024-04-12T03:50:19.681 DEBUG:boto:Data: 2024-04-12T03:50:19.681 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.681 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.681 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.682 DEBUG:boto:Params: {} 2024-04-12T03:50:19.682 DEBUG:boto:Token: None 2024-04-12T03:50:19.682 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.682 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50= 2024-04-12T03:50:19.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50=', 'Content-Length': '0'} 2024-04-12T03:50:19.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f818b93c5e06cee-006618af7b-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.684 DEBUG:boto:b'dkyges-56b1000falseb2024-04-12T03:49:58.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.685 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.685 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.685 DEBUG:boto:path=/dkyges-56/?prefix=b 2024-04-12T03:50:19.685 DEBUG:boto:auth_path=/dkyges-56/?prefix=b 2024-04-12T03:50:19.685 DEBUG:boto:Method: GET 2024-04-12T03:50:19.685 DEBUG:boto:Path: /dkyges-56/?prefix=b 2024-04-12T03:50:19.685 DEBUG:boto:Data: 2024-04-12T03:50:19.685 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.685 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.686 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.686 DEBUG:boto:Params: {} 2024-04-12T03:50:19.686 DEBUG:boto:Token: None 2024-04-12T03:50:19.686 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.686 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50= 2024-04-12T03:50:19.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50=', 'Content-Length': '0'} 2024-04-12T03:50:19.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e02d1543def30c5-006618af7b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.689 DEBUG:boto:b'dkyges-56b1000falseb2024-04-12T03:49:58.805Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:50:19.689 DEBUG:boto:path=/dkyges-56/c 2024-04-12T03:50:19.689 DEBUG:boto:auth_path=/dkyges-56/c 2024-04-12T03:50:19.690 DEBUG:boto:path=/dkyges-56/c?versionId=null 2024-04-12T03:50:19.690 DEBUG:boto:auth_path=/dkyges-56/c?versionId=null 2024-04-12T03:50:19.690 DEBUG:boto:Method: GET 2024-04-12T03:50:19.690 DEBUG:boto:Path: /dkyges-56/c?versionId=null 2024-04-12T03:50:19.690 DEBUG:boto:Data: 2024-04-12T03:50:19.690 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.690 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.690 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.690 DEBUG:boto:Params: {} 2024-04-12T03:50:19.690 DEBUG:boto:Token: None 2024-04-12T03:50:19.691 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/c?versionId=null 2024-04-12T03:50:19.691 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VFZ70e4oiklF3oWavXtnkXQVAjI= 2024-04-12T03:50:19.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VFZ70e4oiklF3oWavXtnkXQVAjI=', 'Content-Length': '0'} 2024-04-12T03:50:19.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c2287cecd7b93f0-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.693 DEBUG:boto:path=/dkyges-56/c 2024-04-12T03:50:19.693 DEBUG:boto:auth_path=/dkyges-56/c 2024-04-12T03:50:19.693 DEBUG:boto:path=/dkyges-56/c?versionId=null 2024-04-12T03:50:19.693 DEBUG:boto:auth_path=/dkyges-56/c?versionId=null 2024-04-12T03:50:19.693 DEBUG:boto:Method: GET 2024-04-12T03:50:19.693 DEBUG:boto:Path: /dkyges-56/c?versionId=null 2024-04-12T03:50:19.693 DEBUG:boto:Data: 2024-04-12T03:50:19.693 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.694 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.694 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.694 DEBUG:boto:Params: {} 2024-04-12T03:50:19.694 DEBUG:boto:Token: None 2024-04-12T03:50:19.694 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/c?versionId=null 2024-04-12T03:50:19.694 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VFZ70e4oiklF3oWavXtnkXQVAjI= 2024-04-12T03:50:19.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VFZ70e4oiklF3oWavXtnkXQVAjI=', 'Content-Length': '0'} 2024-04-12T03:50:19.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d076bfb60670f2ed-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.696 DEBUG:boto:path=/dkyges-56/c 2024-04-12T03:50:19.697 DEBUG:boto:auth_path=/dkyges-56/c 2024-04-12T03:50:19.697 DEBUG:boto:path=/dkyges-56/c?versionId=null 2024-04-12T03:50:19.697 DEBUG:boto:auth_path=/dkyges-56/c?versionId=null 2024-04-12T03:50:19.697 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.697 DEBUG:boto:Path: /dkyges-56/c?versionId=null 2024-04-12T03:50:19.697 DEBUG:boto:Data: 2024-04-12T03:50:19.697 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.697 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.697 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.697 DEBUG:boto:Params: {} 2024-04-12T03:50:19.697 DEBUG:boto:Token: None 2024-04-12T03:50:19.698 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/c?versionId=null 2024-04-12T03:50:19.698 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Vo77x02b038Tlj4WxYxUSQQew34= 2024-04-12T03:50:19.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Vo77x02b038Tlj4WxYxUSQQew34=', 'Content-Length': '0'} 2024-04-12T03:50:19.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c31a79d0f212f8b-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.700 DEBUG:boto:path=/dkyges-56/c 2024-04-12T03:50:19.700 DEBUG:boto:auth_path=/dkyges-56/c 2024-04-12T03:50:19.700 DEBUG:boto:path=/dkyges-56/c?versionId=null 2024-04-12T03:50:19.700 DEBUG:boto:auth_path=/dkyges-56/c?versionId=null 2024-04-12T03:50:19.700 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.700 DEBUG:boto:Path: /dkyges-56/c?versionId=null 2024-04-12T03:50:19.700 DEBUG:boto:Data: 2024-04-12T03:50:19.700 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.700 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.701 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.701 DEBUG:boto:Params: {} 2024-04-12T03:50:19.701 DEBUG:boto:Token: None 2024-04-12T03:50:19.701 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/c?versionId=null 2024-04-12T03:50:19.701 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Vo77x02b038Tlj4WxYxUSQQew34= 2024-04-12T03:50:19.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Vo77x02b038Tlj4WxYxUSQQew34=', 'Content-Length': '0'} 2024-04-12T03:50:19.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008326b573380c9e4f-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:50:19.703 DEBUG:boto:path=/dkyges-56/c 2024-04-12T03:50:19.703 DEBUG:boto:auth_path=/dkyges-56/c 2024-04-12T03:50:19.703 DEBUG:boto:path=/dkyges-56/c?versionId=null 2024-04-12T03:50:19.703 DEBUG:boto:auth_path=/dkyges-56/c?versionId=null 2024-04-12T03:50:19.703 DEBUG:boto:Method: GET 2024-04-12T03:50:19.703 DEBUG:boto:Path: /dkyges-56/c?versionId=null 2024-04-12T03:50:19.704 DEBUG:boto:Data: 2024-04-12T03:50:19.704 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.704 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.704 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.704 DEBUG:boto:Params: {} 2024-04-12T03:50:19.704 DEBUG:boto:Token: None 2024-04-12T03:50:19.704 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/c?versionId=null 2024-04-12T03:50:19.704 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VFZ70e4oiklF3oWavXtnkXQVAjI= 2024-04-12T03:50:19.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VFZ70e4oiklF3oWavXtnkXQVAjI=', 'Content-Length': '0'} 2024-04-12T03:50:19.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de7649f5d60db7c9-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.707 DEBUG:boto:path=/dkyges-56/c 2024-04-12T03:50:19.707 DEBUG:boto:auth_path=/dkyges-56/c 2024-04-12T03:50:19.707 DEBUG:boto:path=/dkyges-56/c?versionId=null 2024-04-12T03:50:19.707 DEBUG:boto:auth_path=/dkyges-56/c?versionId=null 2024-04-12T03:50:19.707 DEBUG:boto:Method: GET 2024-04-12T03:50:19.707 DEBUG:boto:Path: /dkyges-56/c?versionId=null 2024-04-12T03:50:19.708 DEBUG:boto:Data: 2024-04-12T03:50:19.708 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.708 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.708 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.708 DEBUG:boto:Params: {} 2024-04-12T03:50:19.708 DEBUG:boto:Token: None 2024-04-12T03:50:19.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/c?versionId=null 2024-04-12T03:50:19.708 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VFZ70e4oiklF3oWavXtnkXQVAjI= 2024-04-12T03:50:19.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VFZ70e4oiklF3oWavXtnkXQVAjI=', 'Content-Length': '0'} 2024-04-12T03:50:19.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e75cc50559a641e5-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.754 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.755 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.755 DEBUG:boto:path=/dkyges-56/?prefix=c 2024-04-12T03:50:19.755 DEBUG:boto:auth_path=/dkyges-56/?prefix=c 2024-04-12T03:50:19.755 DEBUG:boto:Method: GET 2024-04-12T03:50:19.755 DEBUG:boto:Path: /dkyges-56/?prefix=c 2024-04-12T03:50:19.755 DEBUG:boto:Data: 2024-04-12T03:50:19.755 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.755 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.755 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.756 DEBUG:boto:Params: {} 2024-04-12T03:50:19.756 DEBUG:boto:Token: None 2024-04-12T03:50:19.756 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.756 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50= 2024-04-12T03:50:19.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50=', 'Content-Length': '0'} 2024-04-12T03:50:19.759 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cb91cf8a7574118-006618af7b-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.759 DEBUG:boto:b'dkyges-56c1000falsec2024-04-12T03:49:58.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.759 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.759 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.759 DEBUG:boto:path=/dkyges-56/?prefix=c 2024-04-12T03:50:19.759 DEBUG:boto:auth_path=/dkyges-56/?prefix=c 2024-04-12T03:50:19.760 DEBUG:boto:Method: GET 2024-04-12T03:50:19.760 DEBUG:boto:Path: /dkyges-56/?prefix=c 2024-04-12T03:50:19.760 DEBUG:boto:Data: 2024-04-12T03:50:19.760 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.760 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.760 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.760 DEBUG:boto:Params: {} 2024-04-12T03:50:19.760 DEBUG:boto:Token: None 2024-04-12T03:50:19.760 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.760 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50= 2024-04-12T03:50:19.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50=', 'Content-Length': '0'} 2024-04-12T03:50:19.763 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e55124834f275438-006618af7b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.763 DEBUG:boto:b'dkyges-56c1000falsec2024-04-12T03:49:58.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.763 DEBUG:boto:path=/dkyges-56/c 2024-04-12T03:50:19.763 DEBUG:boto:auth_path=/dkyges-56/c 2024-04-12T03:50:19.763 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.763 DEBUG:boto:Path: /dkyges-56/c 2024-04-12T03:50:19.764 DEBUG:boto:Data: 2024-04-12T03:50:19.764 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.764 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.764 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.764 DEBUG:boto:Params: {} 2024-04-12T03:50:19.764 DEBUG:boto:Token: None 2024-04-12T03:50:19.764 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/c 2024-04-12T03:50:19.764 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AVVGDGlE26HduNhhK3b6UBhougA= 2024-04-12T03:50:19.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AVVGDGlE26HduNhhK3b6UBhougA=', 'Content-Length': '0'} 2024-04-12T03:50:19.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d8743b3a0bebbe9-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.766 DEBUG:boto:path=/dkyges-56/c 2024-04-12T03:50:19.766 DEBUG:boto:auth_path=/dkyges-56/c 2024-04-12T03:50:19.767 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.767 DEBUG:boto:Path: /dkyges-56/c 2024-04-12T03:50:19.767 DEBUG:boto:Data: 2024-04-12T03:50:19.767 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.767 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.767 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.767 DEBUG:boto:Params: {} 2024-04-12T03:50:19.767 DEBUG:boto:Token: None 2024-04-12T03:50:19.767 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/c 2024-04-12T03:50:19.767 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AVVGDGlE26HduNhhK3b6UBhougA= 2024-04-12T03:50:19.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AVVGDGlE26HduNhhK3b6UBhougA=', 'Content-Length': '0'} 2024-04-12T03:50:19.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fe832706af0852c-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:50:19.769 DEBUG:boto:path=/dkyges-56/c 2024-04-12T03:50:19.769 DEBUG:boto:auth_path=/dkyges-56/c 2024-04-12T03:50:19.770 DEBUG:boto:Method: GET 2024-04-12T03:50:19.770 DEBUG:boto:Path: /dkyges-56/c 2024-04-12T03:50:19.770 DEBUG:boto:Data: 2024-04-12T03:50:19.770 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.770 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.770 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.770 DEBUG:boto:Params: {} 2024-04-12T03:50:19.770 DEBUG:boto:Token: None 2024-04-12T03:50:19.770 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/c 2024-04-12T03:50:19.770 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gBrljpDBa/Yo8AQf5iRe1NjWqxo= 2024-04-12T03:50:19.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gBrljpDBa/Yo8AQf5iRe1NjWqxo=', 'Content-Length': '0'} 2024-04-12T03:50:19.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8e0cb5d190ebe6b-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.814 DEBUG:boto:path=/dkyges-56/c 2024-04-12T03:50:19.814 DEBUG:boto:auth_path=/dkyges-56/c 2024-04-12T03:50:19.815 DEBUG:boto:Method: GET 2024-04-12T03:50:19.815 DEBUG:boto:Path: /dkyges-56/c 2024-04-12T03:50:19.815 DEBUG:boto:Data: 2024-04-12T03:50:19.815 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.815 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.815 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.815 DEBUG:boto:Params: {} 2024-04-12T03:50:19.815 DEBUG:boto:Token: None 2024-04-12T03:50:19.815 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/c 2024-04-12T03:50:19.815 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gBrljpDBa/Yo8AQf5iRe1NjWqxo= 2024-04-12T03:50:19.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gBrljpDBa/Yo8AQf5iRe1NjWqxo=', 'Content-Length': '0'} 2024-04-12T03:50:19.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1171fa8a5e5b6c1-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.818 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.818 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.818 DEBUG:boto:path=/dkyges-56/?prefix=c 2024-04-12T03:50:19.818 DEBUG:boto:auth_path=/dkyges-56/?prefix=c 2024-04-12T03:50:19.818 DEBUG:boto:Method: GET 2024-04-12T03:50:19.818 DEBUG:boto:Path: /dkyges-56/?prefix=c 2024-04-12T03:50:19.818 DEBUG:boto:Data: 2024-04-12T03:50:19.819 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.819 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.819 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.819 DEBUG:boto:Params: {} 2024-04-12T03:50:19.819 DEBUG:boto:Token: None 2024-04-12T03:50:19.819 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.819 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50= 2024-04-12T03:50:19.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50=', 'Content-Length': '0'} 2024-04-12T03:50:19.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038e56b39583028c7-006618af7b-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.822 DEBUG:boto:b'dkyges-56c1000falsec2024-04-12T03:49:58.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.822 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.822 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.822 DEBUG:boto:path=/dkyges-56/?prefix=c 2024-04-12T03:50:19.822 DEBUG:boto:auth_path=/dkyges-56/?prefix=c 2024-04-12T03:50:19.822 DEBUG:boto:Method: GET 2024-04-12T03:50:19.823 DEBUG:boto:Path: /dkyges-56/?prefix=c 2024-04-12T03:50:19.823 DEBUG:boto:Data: 2024-04-12T03:50:19.823 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.823 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.823 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.823 DEBUG:boto:Params: {} 2024-04-12T03:50:19.823 DEBUG:boto:Token: None 2024-04-12T03:50:19.823 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.823 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50= 2024-04-12T03:50:19.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50=', 'Content-Length': '0'} 2024-04-12T03:50:19.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061094495cac7ff7d-006618af7b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.826 DEBUG:boto:b'dkyges-56c1000falsec2024-04-12T03:49:58.815Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:50:19.826 DEBUG:boto:path=/dkyges-56/d 2024-04-12T03:50:19.826 DEBUG:boto:auth_path=/dkyges-56/d 2024-04-12T03:50:19.827 DEBUG:boto:path=/dkyges-56/d?versionId=null 2024-04-12T03:50:19.827 DEBUG:boto:auth_path=/dkyges-56/d?versionId=null 2024-04-12T03:50:19.827 DEBUG:boto:Method: GET 2024-04-12T03:50:19.827 DEBUG:boto:Path: /dkyges-56/d?versionId=null 2024-04-12T03:50:19.827 DEBUG:boto:Data: 2024-04-12T03:50:19.827 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.827 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.827 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.827 DEBUG:boto:Params: {} 2024-04-12T03:50:19.827 DEBUG:boto:Token: None 2024-04-12T03:50:19.827 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/d?versionId=null 2024-04-12T03:50:19.828 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bQNLLlW8byaQeH0MclpSoYxWdrA= 2024-04-12T03:50:19.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bQNLLlW8byaQeH0MclpSoYxWdrA=', 'Content-Length': '0'} 2024-04-12T03:50:19.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf80199808239344-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.830 DEBUG:boto:path=/dkyges-56/d 2024-04-12T03:50:19.830 DEBUG:boto:auth_path=/dkyges-56/d 2024-04-12T03:50:19.830 DEBUG:boto:path=/dkyges-56/d?versionId=null 2024-04-12T03:50:19.830 DEBUG:boto:auth_path=/dkyges-56/d?versionId=null 2024-04-12T03:50:19.830 DEBUG:boto:Method: GET 2024-04-12T03:50:19.830 DEBUG:boto:Path: /dkyges-56/d?versionId=null 2024-04-12T03:50:19.830 DEBUG:boto:Data: 2024-04-12T03:50:19.830 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.830 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.831 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.831 DEBUG:boto:Params: {} 2024-04-12T03:50:19.831 DEBUG:boto:Token: None 2024-04-12T03:50:19.831 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/d?versionId=null 2024-04-12T03:50:19.831 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bQNLLlW8byaQeH0MclpSoYxWdrA= 2024-04-12T03:50:19.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bQNLLlW8byaQeH0MclpSoYxWdrA=', 'Content-Length': '0'} 2024-04-12T03:50:19.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2bca807f48812f6-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.833 DEBUG:boto:path=/dkyges-56/d 2024-04-12T03:50:19.833 DEBUG:boto:auth_path=/dkyges-56/d 2024-04-12T03:50:19.833 DEBUG:boto:path=/dkyges-56/d?versionId=null 2024-04-12T03:50:19.833 DEBUG:boto:auth_path=/dkyges-56/d?versionId=null 2024-04-12T03:50:19.833 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.833 DEBUG:boto:Path: /dkyges-56/d?versionId=null 2024-04-12T03:50:19.834 DEBUG:boto:Data: 2024-04-12T03:50:19.834 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.834 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.834 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.834 DEBUG:boto:Params: {} 2024-04-12T03:50:19.834 DEBUG:boto:Token: None 2024-04-12T03:50:19.834 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/d?versionId=null 2024-04-12T03:50:19.834 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WUhwabLdoi7IHRq309vVpHRyXlU= 2024-04-12T03:50:19.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WUhwabLdoi7IHRq309vVpHRyXlU=', 'Content-Length': '0'} 2024-04-12T03:50:19.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000566127072f0f7dde-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.836 DEBUG:boto:path=/dkyges-56/d 2024-04-12T03:50:19.836 DEBUG:boto:auth_path=/dkyges-56/d 2024-04-12T03:50:19.837 DEBUG:boto:path=/dkyges-56/d?versionId=null 2024-04-12T03:50:19.837 DEBUG:boto:auth_path=/dkyges-56/d?versionId=null 2024-04-12T03:50:19.837 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.837 DEBUG:boto:Path: /dkyges-56/d?versionId=null 2024-04-12T03:50:19.837 DEBUG:boto:Data: 2024-04-12T03:50:19.837 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.837 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.837 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.837 DEBUG:boto:Params: {} 2024-04-12T03:50:19.837 DEBUG:boto:Token: None 2024-04-12T03:50:19.837 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/d?versionId=null 2024-04-12T03:50:19.838 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WUhwabLdoi7IHRq309vVpHRyXlU= 2024-04-12T03:50:19.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WUhwabLdoi7IHRq309vVpHRyXlU=', 'Content-Length': '0'} 2024-04-12T03:50:19.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c51bedf544da827d-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:50:19.840 DEBUG:boto:path=/dkyges-56/d 2024-04-12T03:50:19.840 DEBUG:boto:auth_path=/dkyges-56/d 2024-04-12T03:50:19.840 DEBUG:boto:path=/dkyges-56/d?versionId=null 2024-04-12T03:50:19.840 DEBUG:boto:auth_path=/dkyges-56/d?versionId=null 2024-04-12T03:50:19.840 DEBUG:boto:Method: GET 2024-04-12T03:50:19.840 DEBUG:boto:Path: /dkyges-56/d?versionId=null 2024-04-12T03:50:19.840 DEBUG:boto:Data: 2024-04-12T03:50:19.841 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.841 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.841 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.841 DEBUG:boto:Params: {} 2024-04-12T03:50:19.841 DEBUG:boto:Token: None 2024-04-12T03:50:19.841 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/d?versionId=null 2024-04-12T03:50:19.841 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bQNLLlW8byaQeH0MclpSoYxWdrA= 2024-04-12T03:50:19.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bQNLLlW8byaQeH0MclpSoYxWdrA=', 'Content-Length': '0'} 2024-04-12T03:50:19.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed702309045b8149-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.843 DEBUG:boto:path=/dkyges-56/d 2024-04-12T03:50:19.843 DEBUG:boto:auth_path=/dkyges-56/d 2024-04-12T03:50:19.843 DEBUG:boto:path=/dkyges-56/d?versionId=null 2024-04-12T03:50:19.843 DEBUG:boto:auth_path=/dkyges-56/d?versionId=null 2024-04-12T03:50:19.844 DEBUG:boto:Method: GET 2024-04-12T03:50:19.844 DEBUG:boto:Path: /dkyges-56/d?versionId=null 2024-04-12T03:50:19.844 DEBUG:boto:Data: 2024-04-12T03:50:19.844 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.844 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.844 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.844 DEBUG:boto:Params: {} 2024-04-12T03:50:19.844 DEBUG:boto:Token: None 2024-04-12T03:50:19.844 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/d?versionId=null 2024-04-12T03:50:19.844 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bQNLLlW8byaQeH0MclpSoYxWdrA= 2024-04-12T03:50:19.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bQNLLlW8byaQeH0MclpSoYxWdrA=', 'Content-Length': '0'} 2024-04-12T03:50:19.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000162eacc34be6536a-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.894 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.894 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.895 DEBUG:boto:path=/dkyges-56/?prefix=d 2024-04-12T03:50:19.895 DEBUG:boto:auth_path=/dkyges-56/?prefix=d 2024-04-12T03:50:19.895 DEBUG:boto:Method: GET 2024-04-12T03:50:19.895 DEBUG:boto:Path: /dkyges-56/?prefix=d 2024-04-12T03:50:19.895 DEBUG:boto:Data: 2024-04-12T03:50:19.895 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.895 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.895 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.895 DEBUG:boto:Params: {} 2024-04-12T03:50:19.895 DEBUG:boto:Token: None 2024-04-12T03:50:19.896 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.896 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50= 2024-04-12T03:50:19.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50=', 'Content-Length': '0'} 2024-04-12T03:50:19.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc0df2d3bf4e1f7d-006618af7b-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.898 DEBUG:boto:b'dkyges-56d1000falsed2024-04-12T03:49:58.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.899 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.899 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.899 DEBUG:boto:path=/dkyges-56/?prefix=d 2024-04-12T03:50:19.899 DEBUG:boto:auth_path=/dkyges-56/?prefix=d 2024-04-12T03:50:19.899 DEBUG:boto:Method: GET 2024-04-12T03:50:19.899 DEBUG:boto:Path: /dkyges-56/?prefix=d 2024-04-12T03:50:19.899 DEBUG:boto:Data: 2024-04-12T03:50:19.899 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.899 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.899 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.899 DEBUG:boto:Params: {} 2024-04-12T03:50:19.900 DEBUG:boto:Token: None 2024-04-12T03:50:19.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.900 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50= 2024-04-12T03:50:19.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50=', 'Content-Length': '0'} 2024-04-12T03:50:19.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e09ec5d1ad231b98-006618af7b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.902 DEBUG:boto:b'dkyges-56d1000falsed2024-04-12T03:49:58.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.903 DEBUG:boto:path=/dkyges-56/d 2024-04-12T03:50:19.903 DEBUG:boto:auth_path=/dkyges-56/d 2024-04-12T03:50:19.903 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.903 DEBUG:boto:Path: /dkyges-56/d 2024-04-12T03:50:19.903 DEBUG:boto:Data: 2024-04-12T03:50:19.903 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.903 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.903 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.903 DEBUG:boto:Params: {} 2024-04-12T03:50:19.904 DEBUG:boto:Token: None 2024-04-12T03:50:19.904 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/d 2024-04-12T03:50:19.904 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PyEe6FzkSMzUzHd5tI5poh+syk0= 2024-04-12T03:50:19.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PyEe6FzkSMzUzHd5tI5poh+syk0=', 'Content-Length': '0'} 2024-04-12T03:50:19.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ff576caf8beec2b-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.906 DEBUG:boto:path=/dkyges-56/d 2024-04-12T03:50:19.906 DEBUG:boto:auth_path=/dkyges-56/d 2024-04-12T03:50:19.906 DEBUG:boto:Method: HEAD 2024-04-12T03:50:19.906 DEBUG:boto:Path: /dkyges-56/d 2024-04-12T03:50:19.906 DEBUG:boto:Data: 2024-04-12T03:50:19.906 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.906 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.907 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.907 DEBUG:boto:Params: {} 2024-04-12T03:50:19.907 DEBUG:boto:Token: None 2024-04-12T03:50:19.907 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/d 2024-04-12T03:50:19.907 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PyEe6FzkSMzUzHd5tI5poh+syk0= 2024-04-12T03:50:19.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PyEe6FzkSMzUzHd5tI5poh+syk0=', 'Content-Length': '0'} 2024-04-12T03:50:19.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000103de601de97d4e3-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:50:19.909 DEBUG:boto:path=/dkyges-56/d 2024-04-12T03:50:19.909 DEBUG:boto:auth_path=/dkyges-56/d 2024-04-12T03:50:19.909 DEBUG:boto:Method: GET 2024-04-12T03:50:19.909 DEBUG:boto:Path: /dkyges-56/d 2024-04-12T03:50:19.909 DEBUG:boto:Data: 2024-04-12T03:50:19.910 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.910 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.910 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.910 DEBUG:boto:Params: {} 2024-04-12T03:50:19.910 DEBUG:boto:Token: None 2024-04-12T03:50:19.910 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/d 2024-04-12T03:50:19.910 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFX2hOlcrpFohR4BlYj2bW1hMNk= 2024-04-12T03:50:19.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFX2hOlcrpFohR4BlYj2bW1hMNk=', 'Content-Length': '0'} 2024-04-12T03:50:19.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-56:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002311e9f6cde76c05-006618af7b-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.912 DEBUG:boto:path=/dkyges-56/d 2024-04-12T03:50:19.912 DEBUG:boto:auth_path=/dkyges-56/d 2024-04-12T03:50:19.912 DEBUG:boto:Method: GET 2024-04-12T03:50:19.912 DEBUG:boto:Path: /dkyges-56/d 2024-04-12T03:50:19.913 DEBUG:boto:Data: 2024-04-12T03:50:19.913 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.913 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.913 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.913 DEBUG:boto:Params: {} 2024-04-12T03:50:19.913 DEBUG:boto:Token: None 2024-04-12T03:50:19.913 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/d 2024-04-12T03:50:19.913 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFX2hOlcrpFohR4BlYj2bW1hMNk= 2024-04-12T03:50:19.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFX2hOlcrpFohR4BlYj2bW1hMNk=', 'Content-Length': '0'} 2024-04-12T03:50:19.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:49:58 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc2c29da3e325afd-006618af7b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.915 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.915 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.915 DEBUG:boto:path=/dkyges-56/?prefix=d 2024-04-12T03:50:19.916 DEBUG:boto:auth_path=/dkyges-56/?prefix=d 2024-04-12T03:50:19.916 DEBUG:boto:Method: GET 2024-04-12T03:50:19.916 DEBUG:boto:Path: /dkyges-56/?prefix=d 2024-04-12T03:50:19.916 DEBUG:boto:Data: 2024-04-12T03:50:19.916 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.916 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:19.916 DEBUG:boto:Port: 8001 2024-04-12T03:50:19.916 DEBUG:boto:Params: {} 2024-04-12T03:50:19.916 DEBUG:boto:Token: None 2024-04-12T03:50:19.916 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.917 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50= 2024-04-12T03:50:19.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50=', 'Content-Length': '0'} 2024-04-12T03:50:19.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040ea3c42a08cbbe7-006618af7b-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.919 DEBUG:boto:b'dkyges-56d1000falsed2024-04-12T03:49:58.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.919 DEBUG:boto:path=/dkyges-56/ 2024-04-12T03:50:19.919 DEBUG:boto:auth_path=/dkyges-56/ 2024-04-12T03:50:19.919 DEBUG:boto:path=/dkyges-56/?prefix=d 2024-04-12T03:50:19.920 DEBUG:boto:auth_path=/dkyges-56/?prefix=d 2024-04-12T03:50:19.920 DEBUG:boto:Method: GET 2024-04-12T03:50:19.920 DEBUG:boto:Path: /dkyges-56/?prefix=d 2024-04-12T03:50:19.920 DEBUG:boto:Data: 2024-04-12T03:50:19.920 DEBUG:boto:Headers: {} 2024-04-12T03:50:19.920 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:19.920 DEBUG:boto:Port: 8000 2024-04-12T03:50:19.920 DEBUG:boto:Params: {} 2024-04-12T03:50:19.920 DEBUG:boto:Token: None 2024-04-12T03:50:19.920 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:19 GMT /dkyges-56/ 2024-04-12T03:50:19.920 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50= 2024-04-12T03:50:19.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WuyVO0i0BPKOurJSezjwL88OV50=', 'Content-Length': '0'} 2024-04-12T03:50:19.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a676aa0ad76a947-006618af7b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:19.966 DEBUG:boto:b'dkyges-56d1000falsed2024-04-12T03:49:58.868Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:19.967 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-56 zones={a2, a1} 2024-04-12T03:50:19.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:50:19.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:50:19.967 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:50:20.023 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:20.023 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:20.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dkyges-56', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:20.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dkyges-56', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:20.087 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dkyges-56 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:50:20.140 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:20.140 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:20.191 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-04-12T03:50:20.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dkyges-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:20.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dkyges-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:20.193 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dkyges-55 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:50:20.245 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:20.246 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:20.297 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:50:20.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:20.297 INFO:teuthology.orchestra.run.smithi043.stdout: "op_id": "1#00000000001.36.6", 2024-04-12T03:50:20.297 INFO:teuthology.orchestra.run.smithi043.stdout: "op_tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.9681623793415453231", 2024-04-12T03:50:20.297 INFO:teuthology.orchestra.run.smithi043.stdout: "op": "write", 2024-04-12T03:50:20.297 INFO:teuthology.orchestra.run.smithi043.stdout: "object": "a", 2024-04-12T03:50:20.297 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:50:20.297 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete", 2024-04-12T03:50:20.297 INFO:teuthology.orchestra.run.smithi043.stdout: "index_ver": 1, 2024-04-12T03:50:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:49:36.569014705Z", 2024-04-12T03:50:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:50:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:50:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:50:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: "bilog_flags": 0, 2024-04-12T03:50:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned": false, 2024-04-12T03:50:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "", 2024-04-12T03:50:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "", 2024-04-12T03:50:20.298 INFO:teuthology.orchestra.run.smithi043.stdout: "zones_trace": [ 2024-04-12T03:50:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "entry": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18" 2024-04-12T03:50:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:50:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "op_id": "5#00000000001.116.6", 2024-04-12T03:50:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "op_tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.7877426795727350902", 2024-04-12T03:50:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "op": "write", 2024-04-12T03:50:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "object": "d", 2024-04-12T03:50:20.299 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:50:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete", 2024-04-12T03:50:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "index_ver": 1, 2024-04-12T03:50:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:49:36.638480250Z", 2024-04-12T03:50:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:50:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:50:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2 2024-04-12T03:50:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "bilog_flags": 0, 2024-04-12T03:50:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned": false, 2024-04-12T03:50:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "", 2024-04-12T03:50:20.300 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "", 2024-04-12T03:50:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: "zones_trace": [ 2024-04-12T03:50:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: "entry": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18" 2024-04-12T03:50:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:50:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: "op_id": "8#00000000001.21.6", 2024-04-12T03:50:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: "op_tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.12555187198291779302", 2024-04-12T03:50:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: "op": "write", 2024-04-12T03:50:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: "object": "c", 2024-04-12T03:50:20.301 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:50:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete", 2024-04-12T03:50:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: "index_ver": 1, 2024-04-12T03:50:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:49:36.627499515Z", 2024-04-12T03:50:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:50:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:50:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T03:50:20.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: "bilog_flags": 0, 2024-04-12T03:50:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned": false, 2024-04-12T03:50:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "", 2024-04-12T03:50:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "", 2024-04-12T03:50:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: "zones_trace": [ 2024-04-12T03:50:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: "entry": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18" 2024-04-12T03:50:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:50:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:20.303 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: "op_id": "10#00000000001.114.6", 2024-04-12T03:50:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: "op_tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.7643905519820890945", 2024-04-12T03:50:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: "op": "write", 2024-04-12T03:50:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: "object": "b", 2024-04-12T03:50:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:50:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete", 2024-04-12T03:50:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: "index_ver": 1, 2024-04-12T03:50:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:49:36.615856032Z", 2024-04-12T03:50:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:50:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:50:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 16 2024-04-12T03:50:20.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:20.305 INFO:teuthology.orchestra.run.smithi043.stdout: "bilog_flags": 0, 2024-04-12T03:50:20.305 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned": false, 2024-04-12T03:50:20.305 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "", 2024-04-12T03:50:20.305 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "", 2024-04-12T03:50:20.305 INFO:teuthology.orchestra.run.smithi043.stdout: "zones_trace": [ 2024-04-12T03:50:20.305 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:20.305 INFO:teuthology.orchestra.run.smithi043.stdout: "entry": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-55:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.18" 2024-04-12T03:50:20.305 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:20.305 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:50:20.306 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:20.306 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:50:20.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:50:20.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:50:20.307 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:50:20.359 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:20.359 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:20.409 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.415+0000 7f331c95c700 -1 req 4222307144039376810 0.003999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-32:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.32,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-32:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.32,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.420 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.426+0000 7f33aaa78700 -1 req 6097112895761214499 0.002999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-24:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.24,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-24:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.24,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.471 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.477+0000 7f33ac27b700 -1 req 522059988696836290 0.001999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-25:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.25,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-25:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.25,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.520 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.526+0000 7f33ec2fb700 -1 req 3434763270824345318 0.001999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-21:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.21,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-21:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.21,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.562 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.568+0000 7f3396a50700 -1 req 3375596598922441939 0.002999961s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-30:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.30,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-30:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.30,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.566 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.572+0000 7f334c1bb700 -1 req 11964104571478388399 0.002999961s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-2:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.2,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-2:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.2,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.607 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.613+0000 7f33edafe700 -1 req 10000525667217571838 0.002999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-27:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.27,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-27:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.27,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.611 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.617+0000 7f33aea80700 -1 req 137631974738426205 0.002999961s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-1:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.1,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-1:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.1,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.612 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.618+0000 7f330a938700 -1 req 253381186149494102 0.001999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-43:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.6,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-43:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.6,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.652 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.658+0000 7f33be29f700 -1 req 16933809753549380437 0.002999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-4:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.4,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-4:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.4,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.655 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.661+0000 7f330d13d700 -1 req 12373862314626471482 0.001999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-38:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.1,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-38:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.1,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.659 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.665+0000 7f3338193700 -1 req 12446173368646912601 0.002999961s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-5:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.5,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-5:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.5,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.696 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.702+0000 7f332e980700 -1 req 10632110607847126065 0.001999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-45:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.8,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-45:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.8,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.704 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.710+0000 7f33d82d3700 -1 req 16893800301824716490 0.001999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-28:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.28,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-28:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.28,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.740 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.746+0000 7f330c13b700 -1 req 18307556574198166544 0.001999973s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-41:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.4,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-41:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.4,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.749 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.755+0000 7f3318153700 -1 req 540305917056811560 0.002999960s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-44:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.7,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-44:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.7,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.750 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.756+0000 7f33ee2ff700 -1 req 17452045445521756348 0.003999947s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-22:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.22,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-22:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.22,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.752 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.758+0000 7f335e9e0700 -1 req 211235624314098419 0.001999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-42:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.5,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-42:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.5,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.794 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.799+0000 7f33ebafa700 -1 req 4672591046848670136 0.000999987s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-40:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.3,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-40:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.3,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.794 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.800+0000 7f331e960700 -1 req 9199756023764317552 0.001999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-39:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.2,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-39:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.2,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.796 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.802+0000 7f339b259700 -1 req 18027886125591542284 0.001999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-29:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.29,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-29:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.29,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.839 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.845+0000 7f333f9a2700 -1 req 4400568568915784845 0.002999961s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-31:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.31,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-31:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.31,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.841 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.847+0000 7f33a6a70700 -1 req 10602159517812307243 0.001999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-26:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.26,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-26:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.26,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.888 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.893+0000 7f3399a56700 -1 req 10340203025195908840 0.000999987s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-53:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.16,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-53:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.16,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.933 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.938+0000 7f33c8ab4700 -1 req 3481962418773561682 0.003999948s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-23:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.23,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-23:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.23,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.978 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.983+0000 7f338ea40700 -1 req 17524652871784731897 0.001999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-6:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.6,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-6:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.6,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:20.988 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:50:20.994+0000 7f3312147700 -1 req 5230518285316150139 0.001999974s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-3:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.3,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-3:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.3,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:50:21.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dkyges-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:21.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dkyges-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:21.053 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dkyges-55 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:50:21.109 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:21.109 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:21.166 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-04-12T03:50:21.171 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ok 2024-04-12T03:50:21.172 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-57 2024-04-12T03:50:21.172 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:21.173 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:21.173 DEBUG:boto:Method: PUT 2024-04-12T03:50:21.173 DEBUG:boto:Path: /dkyges-57/ 2024-04-12T03:50:21.173 DEBUG:boto:Data: 2024-04-12T03:50:21.173 DEBUG:boto:Headers: {} 2024-04-12T03:50:21.173 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:21.173 DEBUG:boto:Port: 8000 2024-04-12T03:50:21.173 DEBUG:boto:Params: {} 2024-04-12T03:50:21.173 DEBUG:boto:Token: None 2024-04-12T03:50:21.174 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:50:21 GMT /dkyges-57/ 2024-04-12T03:50:21.174 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V4exqninCVTDFoP74elA6MPv0bo= 2024-04-12T03:50:21.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V4exqninCVTDFoP74elA6MPv0bo=', 'Content-Length': '0'} 2024-04-12T03:50:21.191 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006e0df52cf4809f23-006618af7d-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:21.191 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:21.191 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:21.191 DEBUG:boto:Method: HEAD 2024-04-12T03:50:21.191 DEBUG:boto:Path: /dkyges-57/ 2024-04-12T03:50:21.191 DEBUG:boto:Data: 2024-04-12T03:50:21.191 DEBUG:boto:Headers: {} 2024-04-12T03:50:21.191 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:21.191 DEBUG:boto:Port: 8000 2024-04-12T03:50:21.192 DEBUG:boto:Params: {} 2024-04-12T03:50:21.192 DEBUG:boto:Token: None 2024-04-12T03:50:21.192 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:21 GMT /dkyges-57/ 2024-04-12T03:50:21.192 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BTTHEelLQseAUKlARavSSnUsiWw= 2024-04-12T03:50:21.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BTTHEelLQseAUKlARavSSnUsiWw=', 'Content-Length': '0'} 2024-04-12T03:50:21.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000caf9acd36f9f9053-006618af7d-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:21.195 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:50:21.195 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:50:21.195 DEBUG:boto:Method: PUT 2024-04-12T03:50:21.195 DEBUG:boto:Path: /dkyges-57/a 2024-04-12T03:50:21.195 DEBUG:boto:Data: 2024-04-12T03:50:21.195 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:50:21.195 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:21.195 DEBUG:boto:Port: 8000 2024-04-12T03:50:21.195 DEBUG:boto:Params: {} 2024-04-12T03:50:21.196 DEBUG:boto:Token: None 2024-04-12T03:50:21.196 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:50:21 GMT /dkyges-57/a 2024-04-12T03:50:21.196 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CMrvJ8HERNzTGgXT+a5FBetggow= 2024-04-12T03:50:21.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CMrvJ8HERNzTGgXT+a5FBetggow='} 2024-04-12T03:50:21.204 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079c7ff4f8b7740a3-006618af7d-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:21.204 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:21.204 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:21.204 DEBUG:boto:Method: HEAD 2024-04-12T03:50:21.204 DEBUG:boto:Path: /dkyges-57/ 2024-04-12T03:50:21.204 DEBUG:boto:Data: 2024-04-12T03:50:21.204 DEBUG:boto:Headers: {} 2024-04-12T03:50:21.204 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:21.204 DEBUG:boto:Port: 8000 2024-04-12T03:50:21.205 DEBUG:boto:Params: {} 2024-04-12T03:50:21.205 DEBUG:boto:Token: None 2024-04-12T03:50:21.205 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:21 GMT /dkyges-57/ 2024-04-12T03:50:21.205 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BTTHEelLQseAUKlARavSSnUsiWw= 2024-04-12T03:50:21.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BTTHEelLQseAUKlARavSSnUsiWw=', 'Content-Length': '0'} 2024-04-12T03:50:21.207 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001aac41545b397cef-006618af7d-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:21.207 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:50:21.208 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:50:21.208 DEBUG:boto:Method: PUT 2024-04-12T03:50:21.208 DEBUG:boto:Path: /dkyges-57/b 2024-04-12T03:50:21.208 DEBUG:boto:Data: 2024-04-12T03:50:21.208 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:50:21.208 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:21.208 DEBUG:boto:Port: 8000 2024-04-12T03:50:21.208 DEBUG:boto:Params: {} 2024-04-12T03:50:21.208 DEBUG:boto:Token: None 2024-04-12T03:50:21.209 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:50:21 GMT /dkyges-57/b 2024-04-12T03:50:21.209 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qhWXqgWtUAZ1z/C2LfJarUTJNqk= 2024-04-12T03:50:21.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qhWXqgWtUAZ1z/C2LfJarUTJNqk='} 2024-04-12T03:50:21.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005bcd98c13e9a9132-006618af7d-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:21.251 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:21.251 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:21.251 DEBUG:boto:Method: HEAD 2024-04-12T03:50:21.251 DEBUG:boto:Path: /dkyges-57/ 2024-04-12T03:50:21.251 DEBUG:boto:Data: 2024-04-12T03:50:21.251 DEBUG:boto:Headers: {} 2024-04-12T03:50:21.251 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:21.251 DEBUG:boto:Port: 8000 2024-04-12T03:50:21.252 DEBUG:boto:Params: {} 2024-04-12T03:50:21.252 DEBUG:boto:Token: None 2024-04-12T03:50:21.252 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:21 GMT /dkyges-57/ 2024-04-12T03:50:21.252 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BTTHEelLQseAUKlARavSSnUsiWw= 2024-04-12T03:50:21.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BTTHEelLQseAUKlARavSSnUsiWw=', 'Content-Length': '0'} 2024-04-12T03:50:21.255 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007050fcce68c87aa4-006618af7d-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:21.255 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:50:21.255 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:50:21.255 DEBUG:boto:Method: PUT 2024-04-12T03:50:21.255 DEBUG:boto:Path: /dkyges-57/c 2024-04-12T03:50:21.255 DEBUG:boto:Data: 2024-04-12T03:50:21.255 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:50:21.255 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:21.255 DEBUG:boto:Port: 8000 2024-04-12T03:50:21.256 DEBUG:boto:Params: {} 2024-04-12T03:50:21.256 DEBUG:boto:Token: None 2024-04-12T03:50:21.256 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:50:21 GMT /dkyges-57/c 2024-04-12T03:50:21.256 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cW+tN0DQ1DZ5ZOpesFa5x3Vk8PQ= 2024-04-12T03:50:21.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cW+tN0DQ1DZ5ZOpesFa5x3Vk8PQ='} 2024-04-12T03:50:21.262 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002736c0cbda7c88d7-006618af7d-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:21.262 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:21.262 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:21.262 DEBUG:boto:Method: HEAD 2024-04-12T03:50:21.262 DEBUG:boto:Path: /dkyges-57/ 2024-04-12T03:50:21.262 DEBUG:boto:Data: 2024-04-12T03:50:21.263 DEBUG:boto:Headers: {} 2024-04-12T03:50:21.263 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:21.263 DEBUG:boto:Port: 8000 2024-04-12T03:50:21.263 DEBUG:boto:Params: {} 2024-04-12T03:50:21.263 DEBUG:boto:Token: None 2024-04-12T03:50:21.263 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:21 GMT /dkyges-57/ 2024-04-12T03:50:21.263 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BTTHEelLQseAUKlARavSSnUsiWw= 2024-04-12T03:50:21.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BTTHEelLQseAUKlARavSSnUsiWw=', 'Content-Length': '0'} 2024-04-12T03:50:21.265 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000902eb3b89379bc44-006618af7d-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:21.266 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:50:21.266 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:50:21.266 DEBUG:boto:Method: PUT 2024-04-12T03:50:21.266 DEBUG:boto:Path: /dkyges-57/d 2024-04-12T03:50:21.266 DEBUG:boto:Data: 2024-04-12T03:50:21.266 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:50:21.266 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:21.266 DEBUG:boto:Port: 8000 2024-04-12T03:50:21.267 DEBUG:boto:Params: {} 2024-04-12T03:50:21.267 DEBUG:boto:Token: None 2024-04-12T03:50:21.267 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:50:21 GMT /dkyges-57/d 2024-04-12T03:50:21.267 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rcgy2JrsO1NqV23yPl3X/MAI7Ro= 2024-04-12T03:50:21.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rcgy2JrsO1NqV23yPl3X/MAI7Ro='} 2024-04-12T03:50:21.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005cee5cffd67f47eb-006618af7d-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:21.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:21.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:21.311 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:50:21.373 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:21.374 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:21.418 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:50:21.433 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:50:21.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:21.433 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:50:21.433 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:50:21.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:21.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:21.434 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:50:21.434 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:50:21.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:21.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:21.434 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:50:21.434 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:50:21.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:21.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:21.434 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893821.196084_497.1", 2024-04-12T03:50:21.435 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:50:21.196084Z" 2024-04-12T03:50:21.435 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:21.435 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:50:21.436 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893821.196084_497.1'} 2024-04-12T03:50:21.436 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:50:21.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:21.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:21.436 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:50:21.491 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:21.491 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:21.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:21.531+0000 7fb2a6c59780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:50:21.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:21.532+0000 7fb2a6c59780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:50:21.541 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:50:21.541 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:50:21.541 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:50:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:50:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:50:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:50:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:50:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:50:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:50:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:50:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:50:21.542 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:50:21.543 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:50:21.543 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:50:21.543 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:50:21.543 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:50:21.543 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:50:21.543 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:21.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:21.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:21.543 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:50:21.543 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:50:21.543 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:50:21.543 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:50:21.544 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:50:21.544 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:50:21.544 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:50:21.544 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:50:21.544 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:50:21.544 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:21.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:21.544 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:21.544 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:50:21.544 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:50:21.544 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:50:21.545 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:50:21.545 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:50:21.545 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:50:21.545 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:50:21.545 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:50:21.545 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:50:21.545 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:21.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:21.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:21.545 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:50:21.545 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:50:21.545 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:50:21.546 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893821.189168_494.1", 2024-04-12T03:50:21.546 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:50:21.546 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:50:21.546 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:50:21.546 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:50:21.189168Z", 2024-04-12T03:50:21.546 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:50:21.546 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:21.546 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:21.546 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:50:21.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:21.546 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:50:21.547 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:50:21.547 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:50:21.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:21.547 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:50:21Z" 2024-04-12T03:50:21.547 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:50:21.548 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893821.189168_494.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:50:21.189168Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:50:21Z" } 2024-04-12T03:50:21.548 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893821.189168_494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:50:21.189168Z', 'realm_epoch': 2}}] 2024-04-12T03:50:21.548 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893821.196084_497.1'} 2024-04-12T03:50:21.548 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893821.189168_494.1'} 2024-04-12T03:50:21.548 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712893821.196084_497.1 target=1_1712893821.189168_494.1 2024-04-12T03:50:26.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:26.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:26.550 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:50:26.610 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:26.610 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:26.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:26.652+0000 7fafc273d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:50:26.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:26.653+0000 7fafc273d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:50:26.663 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:50:26.663 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:50:26.663 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:50:26.663 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:50:26.663 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:50:26.663 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:50:26.663 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:50:26.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:26.663 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:50:26.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:26.664 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:50:26.664 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:50:26.664 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:50:26.664 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:50:26.664 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:50:26.664 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:50:26.664 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:50:26.664 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:50:26.664 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:50:26.664 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:26.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:26.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:26.665 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:50:26.665 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:50:26.665 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:50:26.665 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:50:26.665 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:50:26.665 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:50:26.665 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:50:26.665 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:50:26.665 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:50:26.666 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:26.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:26.666 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:26.666 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:50:26.666 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:50:26.666 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:50:26.666 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:50:26.666 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:50:26.666 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:50:26.666 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:50:26.666 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:50:26.667 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:50:26.667 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:26.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:26.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:26.667 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:50:26.667 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:50:26.667 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:50:26.667 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893821.196084_497.1", 2024-04-12T03:50:26.667 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:50:26.667 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:50:26.667 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:50:26.668 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:50:21.196084Z", 2024-04-12T03:50:26.668 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:50:26.668 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:26.668 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:26.668 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:50:26.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:26.668 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:50:26.668 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:50:26.668 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:50:26.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:26.668 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:50:26Z" 2024-04-12T03:50:26.668 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:50:26.669 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893821.196084_497.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:50:21.196084Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:50:26Z" } 2024-04-12T03:50:26.670 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893821.196084_497.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:50:21.196084Z', 'realm_epoch': 2}}] 2024-04-12T03:50:26.670 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893821.196084_497.1'} 2024-04-12T03:50:26.670 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893821.196084_497.1'} 2024-04-12T03:50:26.670 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:50:26.670 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-57 2024-04-12T03:50:26.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:50:26.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:50:26.671 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:50:26.734 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:26.734 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:26.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:26.780+0000 7f3b9a929780 1 waiting to reach incremental sync.. 2024-04-12T03:50:31.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:31.781+0000 7f3b9a929780 1 waiting to reach incremental sync.. 2024-04-12T03:50:36.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:36.783+0000 7f3b9a929780 1 bucket sync caught up with source: 2024-04-12T03:50:36.777 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.53.6, , , , 00000000001.19.6, , , 00000000001.93.6, , 00000000001.36.6] 2024-04-12T03:50:36.777 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.53.6, , , , 00000000001.19.6, , , 00000000001.93.6, , 00000000001.36.6] 2024-04-12T03:50:36.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:36.783+0000 7f3b9a929780 0 bucket checkpoint complete 2024-04-12T03:50:36.796 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-57 2024-04-12T03:50:36.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:50:36.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:50:36.797 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:50:36.862 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:36.862 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:36.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:36.909+0000 7f4a53157780 1 waiting to reach incremental sync.. 2024-04-12T03:50:41.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:41.911+0000 7f4a53157780 1 bucket sync caught up with source: 2024-04-12T03:50:41.905 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.2906.6, , , , 00000000001.1066.6, , , 00000000001.1602.6, , 00000000001.3698.6] 2024-04-12T03:50:41.905 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.2906.6, , , , 00000000001.1066.6, , , 00000000001.1602.6, , 00000000001.3698.6] 2024-04-12T03:50:41.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:41.911+0000 7f4a53157780 0 bucket checkpoint complete 2024-04-12T03:50:41.923 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-57 zones={a2, a1} 2024-04-12T03:50:41.923 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:41.923 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:41.923 DEBUG:boto:Method: HEAD 2024-04-12T03:50:41.924 DEBUG:boto:Path: /dkyges-57/ 2024-04-12T03:50:41.924 DEBUG:boto:Data: 2024-04-12T03:50:41.924 DEBUG:boto:Headers: {} 2024-04-12T03:50:41.924 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:41.924 DEBUG:boto:Port: 8001 2024-04-12T03:50:41.924 DEBUG:boto:Params: {} 2024-04-12T03:50:41.924 DEBUG:boto:Token: None 2024-04-12T03:50:41.924 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:41 GMT /dkyges-57/ 2024-04-12T03:50:41.924 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gft4wKK0TzuNf39+w4XqPcng0w4= 2024-04-12T03:50:41.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gft4wKK0TzuNf39+w4XqPcng0w4=', 'Content-Length': '0'} 2024-04-12T03:50:41.927 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cd5e56c8a3b988f-006618af91-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:41.928 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:41.928 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:41.928 DEBUG:boto:Method: HEAD 2024-04-12T03:50:41.928 DEBUG:boto:Path: /dkyges-57/ 2024-04-12T03:50:41.928 DEBUG:boto:Data: 2024-04-12T03:50:41.928 DEBUG:boto:Headers: {} 2024-04-12T03:50:41.928 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:41.928 DEBUG:boto:Port: 8000 2024-04-12T03:50:41.928 DEBUG:boto:Params: {} 2024-04-12T03:50:41.928 DEBUG:boto:Token: None 2024-04-12T03:50:41.929 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:41 GMT /dkyges-57/ 2024-04-12T03:50:41.929 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gft4wKK0TzuNf39+w4XqPcng0w4= 2024-04-12T03:50:41.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gft4wKK0TzuNf39+w4XqPcng0w4=', 'Content-Length': '0'} 2024-04-12T03:50:41.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad85d690a57eb672-006618af91-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:41.931 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:50:41.931 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:41.932 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:41.932 DEBUG:boto:path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:50:41.932 DEBUG:boto:auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:50:41.932 DEBUG:boto:Method: GET 2024-04-12T03:50:41.932 DEBUG:boto:Path: /dkyges-57/?versions&max-keys=999 2024-04-12T03:50:41.932 DEBUG:boto:Data: 2024-04-12T03:50:41.932 DEBUG:boto:Headers: {} 2024-04-12T03:50:41.932 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:41.932 DEBUG:boto:Port: 8001 2024-04-12T03:50:41.932 DEBUG:boto:Params: {} 2024-04-12T03:50:41.933 DEBUG:boto:Token: None 2024-04-12T03:50:41.933 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:41 GMT /dkyges-57/?versions 2024-04-12T03:50:41.933 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fuHj8srEFFZQOgpqfd8gzn6gSUE= 2024-04-12T03:50:41.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fuHj8srEFFZQOgpqfd8gzn6gSUE=', 'Content-Length': '0'} 2024-04-12T03:50:41.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040a869e55ee0141c-006618af91-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:41.935 DEBUG:boto:b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:41.936 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:50:41.936 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:50:41.936 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:50:41.936 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:50:41.936 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:50:41.936 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:41.936 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:41.936 DEBUG:boto:path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:50:41.936 DEBUG:boto:auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:50:41.937 DEBUG:boto:Method: GET 2024-04-12T03:50:41.937 DEBUG:boto:Path: /dkyges-57/?versions&max-keys=999 2024-04-12T03:50:41.937 DEBUG:boto:Data: 2024-04-12T03:50:41.937 DEBUG:boto:Headers: {} 2024-04-12T03:50:41.937 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:41.937 DEBUG:boto:Port: 8000 2024-04-12T03:50:41.937 DEBUG:boto:Params: {} 2024-04-12T03:50:41.937 DEBUG:boto:Token: None 2024-04-12T03:50:41.937 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:41 GMT /dkyges-57/?versions 2024-04-12T03:50:41.937 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fuHj8srEFFZQOgpqfd8gzn6gSUE= 2024-04-12T03:50:41.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fuHj8srEFFZQOgpqfd8gzn6gSUE=', 'Content-Length': '0'} 2024-04-12T03:50:41.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2fa104ad2a22d84-006618af91-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:41.940 DEBUG:boto:b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:41.940 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:50:41.940 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:50:41.940 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:50:41.940 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:50:41.941 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:41.941 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:41.941 DEBUG:boto:path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:50:41.941 DEBUG:boto:auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:50:41.941 DEBUG:boto:Method: GET 2024-04-12T03:50:41.941 DEBUG:boto:Path: /dkyges-57/?versions&max-keys=999 2024-04-12T03:50:41.941 DEBUG:boto:Data: 2024-04-12T03:50:41.941 DEBUG:boto:Headers: {} 2024-04-12T03:50:41.941 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:41.941 DEBUG:boto:Port: 8001 2024-04-12T03:50:41.941 DEBUG:boto:Params: {} 2024-04-12T03:50:41.942 DEBUG:boto:Token: None 2024-04-12T03:50:41.942 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:41 GMT /dkyges-57/?versions 2024-04-12T03:50:41.942 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fuHj8srEFFZQOgpqfd8gzn6gSUE= 2024-04-12T03:50:41.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fuHj8srEFFZQOgpqfd8gzn6gSUE=', 'Content-Length': '0'} 2024-04-12T03:50:41.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cc759735466550e-006618af91-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:41.944 DEBUG:boto:b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:41.945 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:41.945 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:41.945 DEBUG:boto:path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:50:41.945 DEBUG:boto:auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:50:41.945 DEBUG:boto:Method: GET 2024-04-12T03:50:41.945 DEBUG:boto:Path: /dkyges-57/?versions&max-keys=999 2024-04-12T03:50:41.945 DEBUG:boto:Data: 2024-04-12T03:50:41.945 DEBUG:boto:Headers: {} 2024-04-12T03:50:41.945 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:41.945 DEBUG:boto:Port: 8000 2024-04-12T03:50:41.946 DEBUG:boto:Params: {} 2024-04-12T03:50:41.946 DEBUG:boto:Token: None 2024-04-12T03:50:41.946 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:41 GMT /dkyges-57/?versions 2024-04-12T03:50:41.946 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fuHj8srEFFZQOgpqfd8gzn6gSUE= 2024-04-12T03:50:41.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fuHj8srEFFZQOgpqfd8gzn6gSUE=', 'Content-Length': '0'} 2024-04-12T03:50:41.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a2f0ca1c510f9bf-006618af91-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:41.949 DEBUG:boto:b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:41.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:50:41.949 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:50:41.949 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:50:41.949 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:50:41.949 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:50:41.950 DEBUG:boto:Method: GET 2024-04-12T03:50:41.950 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:50:41.950 DEBUG:boto:Data: 2024-04-12T03:50:41.950 DEBUG:boto:Headers: {} 2024-04-12T03:50:41.950 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:41.950 DEBUG:boto:Port: 8001 2024-04-12T03:50:41.950 DEBUG:boto:Params: {} 2024-04-12T03:50:41.950 DEBUG:boto:Token: None 2024-04-12T03:50:41.950 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:41 GMT /dkyges-57/a?versionId=null 2024-04-12T03:50:41.950 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rejUZ20WKjx52D0KUZBOjLXK8yo= 2024-04-12T03:50:41.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rejUZ20WKjx52D0KUZBOjLXK8yo=', 'Content-Length': '0'} 2024-04-12T03:50:41.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004db7b54ff8e3af70-006618af91-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:41.994 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:50:41.994 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:50:41.994 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:50:41.995 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:50:41.995 DEBUG:boto:Method: GET 2024-04-12T03:50:41.995 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:50:41.995 DEBUG:boto:Data: 2024-04-12T03:50:41.995 DEBUG:boto:Headers: {} 2024-04-12T03:50:41.995 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:41.995 DEBUG:boto:Port: 8000 2024-04-12T03:50:41.995 DEBUG:boto:Params: {} 2024-04-12T03:50:41.995 DEBUG:boto:Token: None 2024-04-12T03:50:41.995 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:41 GMT /dkyges-57/a?versionId=null 2024-04-12T03:50:41.995 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rejUZ20WKjx52D0KUZBOjLXK8yo= 2024-04-12T03:50:41.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rejUZ20WKjx52D0KUZBOjLXK8yo=', 'Content-Length': '0'} 2024-04-12T03:50:41.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6e8a942ade1a34c-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:41.998 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:50:41.998 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:50:41.998 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:50:41.998 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:50:41.998 DEBUG:boto:Method: HEAD 2024-04-12T03:50:41.999 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:50:41.999 DEBUG:boto:Data: 2024-04-12T03:50:41.999 DEBUG:boto:Headers: {} 2024-04-12T03:50:41.999 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:41.999 DEBUG:boto:Port: 8001 2024-04-12T03:50:41.999 DEBUG:boto:Params: {} 2024-04-12T03:50:41.999 DEBUG:boto:Token: None 2024-04-12T03:50:41.999 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:41 GMT /dkyges-57/a?versionId=null 2024-04-12T03:50:41.999 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RNJbsCkns/BFabmw64vwQWeUz6g= 2024-04-12T03:50:41.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RNJbsCkns/BFabmw64vwQWeUz6g=', 'Content-Length': '0'} 2024-04-12T03:50:42.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9272988af9639c6-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.002 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:50:42.002 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:50:42.002 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:50:42.002 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:50:42.002 DEBUG:boto:Method: HEAD 2024-04-12T03:50:42.002 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:50:42.002 DEBUG:boto:Data: 2024-04-12T03:50:42.002 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.002 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.002 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.003 DEBUG:boto:Params: {} 2024-04-12T03:50:42.003 DEBUG:boto:Token: None 2024-04-12T03:50:42.003 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/a?versionId=null 2024-04-12T03:50:42.003 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C6vKTTFB62DMlSSVr+wffUWb9x0= 2024-04-12T03:50:42.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C6vKTTFB62DMlSSVr+wffUWb9x0=', 'Content-Length': '0'} 2024-04-12T03:50:42.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf25ccde18c5e513-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:50:42.005 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:50:42.005 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:50:42.005 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:50:42.005 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:50:42.005 DEBUG:boto:Method: GET 2024-04-12T03:50:42.005 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:50:42.006 DEBUG:boto:Data: 2024-04-12T03:50:42.006 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.006 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.006 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.006 DEBUG:boto:Params: {} 2024-04-12T03:50:42.006 DEBUG:boto:Token: None 2024-04-12T03:50:42.006 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/a?versionId=null 2024-04-12T03:50:42.006 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ccEwEMCzbuhjwijKm7iMAkyrqMo= 2024-04-12T03:50:42.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ccEwEMCzbuhjwijKm7iMAkyrqMo=', 'Content-Length': '0'} 2024-04-12T03:50:42.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003273fd2ff01b0e0-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.008 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:50:42.008 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:50:42.008 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:50:42.009 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:50:42.009 DEBUG:boto:Method: GET 2024-04-12T03:50:42.009 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:50:42.009 DEBUG:boto:Data: 2024-04-12T03:50:42.009 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.009 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.009 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.009 DEBUG:boto:Params: {} 2024-04-12T03:50:42.009 DEBUG:boto:Token: None 2024-04-12T03:50:42.009 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/a?versionId=null 2024-04-12T03:50:42.010 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ccEwEMCzbuhjwijKm7iMAkyrqMo= 2024-04-12T03:50:42.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ccEwEMCzbuhjwijKm7iMAkyrqMo=', 'Content-Length': '0'} 2024-04-12T03:50:42.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcd18d4a8fe3abac-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.012 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:42.012 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:42.012 DEBUG:boto:path=/dkyges-57/?prefix=a 2024-04-12T03:50:42.012 DEBUG:boto:auth_path=/dkyges-57/?prefix=a 2024-04-12T03:50:42.012 DEBUG:boto:Method: GET 2024-04-12T03:50:42.012 DEBUG:boto:Path: /dkyges-57/?prefix=a 2024-04-12T03:50:42.012 DEBUG:boto:Data: 2024-04-12T03:50:42.012 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.012 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.012 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.013 DEBUG:boto:Params: {} 2024-04-12T03:50:42.013 DEBUG:boto:Token: None 2024-04-12T03:50:42.013 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/ 2024-04-12T03:50:42.013 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T03:50:42.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T03:50:42.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e61c689b0dc10a5-006618af92-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.016 DEBUG:boto:b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:42.016 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:42.016 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:42.016 DEBUG:boto:path=/dkyges-57/?prefix=a 2024-04-12T03:50:42.017 DEBUG:boto:auth_path=/dkyges-57/?prefix=a 2024-04-12T03:50:42.017 DEBUG:boto:Method: GET 2024-04-12T03:50:42.017 DEBUG:boto:Path: /dkyges-57/?prefix=a 2024-04-12T03:50:42.017 DEBUG:boto:Data: 2024-04-12T03:50:42.017 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.017 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.017 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.017 DEBUG:boto:Params: {} 2024-04-12T03:50:42.017 DEBUG:boto:Token: None 2024-04-12T03:50:42.017 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/ 2024-04-12T03:50:42.018 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T03:50:42.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T03:50:42.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f91c8f671fa73c14-006618af92-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.062 DEBUG:boto:b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:42.063 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:50:42.063 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:50:42.063 DEBUG:boto:Method: HEAD 2024-04-12T03:50:42.063 DEBUG:boto:Path: /dkyges-57/a 2024-04-12T03:50:42.063 DEBUG:boto:Data: 2024-04-12T03:50:42.063 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.063 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.064 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.064 DEBUG:boto:Params: {} 2024-04-12T03:50:42.064 DEBUG:boto:Token: None 2024-04-12T03:50:42.064 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/a 2024-04-12T03:50:42.064 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gzUgsw3BZqyPgZ0tp30snRglmuQ= 2024-04-12T03:50:42.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gzUgsw3BZqyPgZ0tp30snRglmuQ=', 'Content-Length': '0'} 2024-04-12T03:50:42.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2165499c7c1b451-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.066 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:50:42.066 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:50:42.066 DEBUG:boto:Method: HEAD 2024-04-12T03:50:42.067 DEBUG:boto:Path: /dkyges-57/a 2024-04-12T03:50:42.067 DEBUG:boto:Data: 2024-04-12T03:50:42.067 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.067 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.067 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.067 DEBUG:boto:Params: {} 2024-04-12T03:50:42.067 DEBUG:boto:Token: None 2024-04-12T03:50:42.067 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/a 2024-04-12T03:50:42.067 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gzUgsw3BZqyPgZ0tp30snRglmuQ= 2024-04-12T03:50:42.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gzUgsw3BZqyPgZ0tp30snRglmuQ=', 'Content-Length': '0'} 2024-04-12T03:50:42.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d32978534292aa1-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:50:42.070 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:50:42.070 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:50:42.070 DEBUG:boto:Method: GET 2024-04-12T03:50:42.070 DEBUG:boto:Path: /dkyges-57/a 2024-04-12T03:50:42.070 DEBUG:boto:Data: 2024-04-12T03:50:42.070 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.070 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.070 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.070 DEBUG:boto:Params: {} 2024-04-12T03:50:42.070 DEBUG:boto:Token: None 2024-04-12T03:50:42.071 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/a 2024-04-12T03:50:42.071 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WsrnbViAVR9ZDdPYi8y7F3B8mnY= 2024-04-12T03:50:42.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WsrnbViAVR9ZDdPYi8y7F3B8mnY=', 'Content-Length': '0'} 2024-04-12T03:50:42.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cd37ca8cded458a-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.073 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:50:42.073 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:50:42.073 DEBUG:boto:Method: GET 2024-04-12T03:50:42.073 DEBUG:boto:Path: /dkyges-57/a 2024-04-12T03:50:42.073 DEBUG:boto:Data: 2024-04-12T03:50:42.073 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.073 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.073 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.074 DEBUG:boto:Params: {} 2024-04-12T03:50:42.074 DEBUG:boto:Token: None 2024-04-12T03:50:42.074 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/a 2024-04-12T03:50:42.074 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WsrnbViAVR9ZDdPYi8y7F3B8mnY= 2024-04-12T03:50:42.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WsrnbViAVR9ZDdPYi8y7F3B8mnY=', 'Content-Length': '0'} 2024-04-12T03:50:42.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dd0457df1c559b7-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.076 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:42.076 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:42.076 DEBUG:boto:path=/dkyges-57/?prefix=a 2024-04-12T03:50:42.077 DEBUG:boto:auth_path=/dkyges-57/?prefix=a 2024-04-12T03:50:42.077 DEBUG:boto:Method: GET 2024-04-12T03:50:42.077 DEBUG:boto:Path: /dkyges-57/?prefix=a 2024-04-12T03:50:42.077 DEBUG:boto:Data: 2024-04-12T03:50:42.077 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.077 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.077 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.077 DEBUG:boto:Params: {} 2024-04-12T03:50:42.077 DEBUG:boto:Token: None 2024-04-12T03:50:42.077 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/ 2024-04-12T03:50:42.078 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T03:50:42.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T03:50:42.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000299c43c1fdd88a0e-006618af92-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.080 DEBUG:boto:b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:42.081 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:42.081 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:42.081 DEBUG:boto:path=/dkyges-57/?prefix=a 2024-04-12T03:50:42.081 DEBUG:boto:auth_path=/dkyges-57/?prefix=a 2024-04-12T03:50:42.081 DEBUG:boto:Method: GET 2024-04-12T03:50:42.081 DEBUG:boto:Path: /dkyges-57/?prefix=a 2024-04-12T03:50:42.081 DEBUG:boto:Data: 2024-04-12T03:50:42.081 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.081 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.081 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.082 DEBUG:boto:Params: {} 2024-04-12T03:50:42.082 DEBUG:boto:Token: None 2024-04-12T03:50:42.082 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/ 2024-04-12T03:50:42.082 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T03:50:42.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T03:50:42.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d74f5c995956be36-006618af92-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.084 DEBUG:boto:b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:42.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:50:42.085 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:50:42.085 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:50:42.085 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:50:42.085 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:50:42.085 DEBUG:boto:Method: GET 2024-04-12T03:50:42.085 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:50:42.085 DEBUG:boto:Data: 2024-04-12T03:50:42.085 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.085 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.085 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.086 DEBUG:boto:Params: {} 2024-04-12T03:50:42.086 DEBUG:boto:Token: None 2024-04-12T03:50:42.086 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/b?versionId=null 2024-04-12T03:50:42.086 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1qDgvbqtWc+aikdQtfdLDXgcV4Y= 2024-04-12T03:50:42.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1qDgvbqtWc+aikdQtfdLDXgcV4Y=', 'Content-Length': '0'} 2024-04-12T03:50:42.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000514b272bf4c630b3-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.088 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:50:42.088 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:50:42.088 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:50:42.088 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:50:42.088 DEBUG:boto:Method: GET 2024-04-12T03:50:42.088 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:50:42.089 DEBUG:boto:Data: 2024-04-12T03:50:42.089 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.089 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.089 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.089 DEBUG:boto:Params: {} 2024-04-12T03:50:42.089 DEBUG:boto:Token: None 2024-04-12T03:50:42.089 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/b?versionId=null 2024-04-12T03:50:42.089 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1qDgvbqtWc+aikdQtfdLDXgcV4Y= 2024-04-12T03:50:42.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1qDgvbqtWc+aikdQtfdLDXgcV4Y=', 'Content-Length': '0'} 2024-04-12T03:50:42.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a18e767299ef1f7-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.092 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:50:42.092 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:50:42.092 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:50:42.092 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:50:42.092 DEBUG:boto:Method: HEAD 2024-04-12T03:50:42.092 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:50:42.092 DEBUG:boto:Data: 2024-04-12T03:50:42.092 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.092 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.092 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.093 DEBUG:boto:Params: {} 2024-04-12T03:50:42.093 DEBUG:boto:Token: None 2024-04-12T03:50:42.093 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/b?versionId=null 2024-04-12T03:50:42.093 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a+ww9uLwQ0NYwYvaZDXx3GDLhkU= 2024-04-12T03:50:42.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a+ww9uLwQ0NYwYvaZDXx3GDLhkU=', 'Content-Length': '0'} 2024-04-12T03:50:42.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cb781eeb4893b31-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.095 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:50:42.095 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:50:42.095 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:50:42.095 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:50:42.096 DEBUG:boto:Method: HEAD 2024-04-12T03:50:42.096 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:50:42.096 DEBUG:boto:Data: 2024-04-12T03:50:42.096 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.096 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.096 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.096 DEBUG:boto:Params: {} 2024-04-12T03:50:42.096 DEBUG:boto:Token: None 2024-04-12T03:50:42.096 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/b?versionId=null 2024-04-12T03:50:42.097 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a+ww9uLwQ0NYwYvaZDXx3GDLhkU= 2024-04-12T03:50:42.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a+ww9uLwQ0NYwYvaZDXx3GDLhkU=', 'Content-Length': '0'} 2024-04-12T03:50:42.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f1d434fd7e8bcdf-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:50:42.099 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:50:42.099 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:50:42.099 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:50:42.099 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:50:42.099 DEBUG:boto:Method: GET 2024-04-12T03:50:42.099 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:50:42.099 DEBUG:boto:Data: 2024-04-12T03:50:42.099 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.099 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.099 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.100 DEBUG:boto:Params: {} 2024-04-12T03:50:42.100 DEBUG:boto:Token: None 2024-04-12T03:50:42.100 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/b?versionId=null 2024-04-12T03:50:42.100 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1qDgvbqtWc+aikdQtfdLDXgcV4Y= 2024-04-12T03:50:42.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1qDgvbqtWc+aikdQtfdLDXgcV4Y=', 'Content-Length': '0'} 2024-04-12T03:50:42.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000706d911190e99f1f-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.142 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:50:42.143 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:50:42.143 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:50:42.143 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:50:42.143 DEBUG:boto:Method: GET 2024-04-12T03:50:42.143 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:50:42.143 DEBUG:boto:Data: 2024-04-12T03:50:42.143 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.143 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.143 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.143 DEBUG:boto:Params: {} 2024-04-12T03:50:42.144 DEBUG:boto:Token: None 2024-04-12T03:50:42.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/b?versionId=null 2024-04-12T03:50:42.144 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1qDgvbqtWc+aikdQtfdLDXgcV4Y= 2024-04-12T03:50:42.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1qDgvbqtWc+aikdQtfdLDXgcV4Y=', 'Content-Length': '0'} 2024-04-12T03:50:42.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075861a13181d0348-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.147 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:42.147 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:42.147 DEBUG:boto:path=/dkyges-57/?prefix=b 2024-04-12T03:50:42.147 DEBUG:boto:auth_path=/dkyges-57/?prefix=b 2024-04-12T03:50:42.147 DEBUG:boto:Method: GET 2024-04-12T03:50:42.147 DEBUG:boto:Path: /dkyges-57/?prefix=b 2024-04-12T03:50:42.147 DEBUG:boto:Data: 2024-04-12T03:50:42.147 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.147 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.147 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.147 DEBUG:boto:Params: {} 2024-04-12T03:50:42.148 DEBUG:boto:Token: None 2024-04-12T03:50:42.148 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/ 2024-04-12T03:50:42.148 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T03:50:42.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T03:50:42.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce89b457ab1a4c22-006618af92-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.151 DEBUG:boto:b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:42.151 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:42.151 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:42.151 DEBUG:boto:path=/dkyges-57/?prefix=b 2024-04-12T03:50:42.151 DEBUG:boto:auth_path=/dkyges-57/?prefix=b 2024-04-12T03:50:42.152 DEBUG:boto:Method: GET 2024-04-12T03:50:42.152 DEBUG:boto:Path: /dkyges-57/?prefix=b 2024-04-12T03:50:42.152 DEBUG:boto:Data: 2024-04-12T03:50:42.152 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.152 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.152 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.153 DEBUG:boto:Params: {} 2024-04-12T03:50:42.153 DEBUG:boto:Token: None 2024-04-12T03:50:42.153 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/ 2024-04-12T03:50:42.153 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T03:50:42.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T03:50:42.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045d2aeb0fe4c6061-006618af92-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.202 DEBUG:boto:b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:42.203 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:50:42.203 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:50:42.203 DEBUG:boto:Method: HEAD 2024-04-12T03:50:42.203 DEBUG:boto:Path: /dkyges-57/b 2024-04-12T03:50:42.203 DEBUG:boto:Data: 2024-04-12T03:50:42.203 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.203 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.204 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.204 DEBUG:boto:Params: {} 2024-04-12T03:50:42.204 DEBUG:boto:Token: None 2024-04-12T03:50:42.204 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/b 2024-04-12T03:50:42.204 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9By2kDgLYqGGul2UUYTS18X0mOg= 2024-04-12T03:50:42.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9By2kDgLYqGGul2UUYTS18X0mOg=', 'Content-Length': '0'} 2024-04-12T03:50:42.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd9b2ce456d74af0-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.206 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:50:42.207 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:50:42.207 DEBUG:boto:Method: HEAD 2024-04-12T03:50:42.207 DEBUG:boto:Path: /dkyges-57/b 2024-04-12T03:50:42.207 DEBUG:boto:Data: 2024-04-12T03:50:42.207 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.207 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.207 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.207 DEBUG:boto:Params: {} 2024-04-12T03:50:42.208 DEBUG:boto:Token: None 2024-04-12T03:50:42.208 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/b 2024-04-12T03:50:42.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9By2kDgLYqGGul2UUYTS18X0mOg= 2024-04-12T03:50:42.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9By2kDgLYqGGul2UUYTS18X0mOg=', 'Content-Length': '0'} 2024-04-12T03:50:42.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8846eeedb0fccaa-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:50:42.210 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:50:42.211 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:50:42.211 DEBUG:boto:Method: GET 2024-04-12T03:50:42.211 DEBUG:boto:Path: /dkyges-57/b 2024-04-12T03:50:42.211 DEBUG:boto:Data: 2024-04-12T03:50:42.211 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.211 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.211 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.211 DEBUG:boto:Params: {} 2024-04-12T03:50:42.211 DEBUG:boto:Token: None 2024-04-12T03:50:42.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/b 2024-04-12T03:50:42.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V3TKYnu9QCCYSnJg1D2MaPMfM6I= 2024-04-12T03:50:42.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V3TKYnu9QCCYSnJg1D2MaPMfM6I=', 'Content-Length': '0'} 2024-04-12T03:50:42.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4e030f701e059b6-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.214 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:50:42.214 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:50:42.214 DEBUG:boto:Method: GET 2024-04-12T03:50:42.214 DEBUG:boto:Path: /dkyges-57/b 2024-04-12T03:50:42.215 DEBUG:boto:Data: 2024-04-12T03:50:42.215 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.215 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.215 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.215 DEBUG:boto:Params: {} 2024-04-12T03:50:42.215 DEBUG:boto:Token: None 2024-04-12T03:50:42.215 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/b 2024-04-12T03:50:42.215 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V3TKYnu9QCCYSnJg1D2MaPMfM6I= 2024-04-12T03:50:42.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V3TKYnu9QCCYSnJg1D2MaPMfM6I=', 'Content-Length': '0'} 2024-04-12T03:50:42.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022f2e067630a39ea-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.218 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:42.218 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:42.218 DEBUG:boto:path=/dkyges-57/?prefix=b 2024-04-12T03:50:42.218 DEBUG:boto:auth_path=/dkyges-57/?prefix=b 2024-04-12T03:50:42.218 DEBUG:boto:Method: GET 2024-04-12T03:50:42.218 DEBUG:boto:Path: /dkyges-57/?prefix=b 2024-04-12T03:50:42.219 DEBUG:boto:Data: 2024-04-12T03:50:42.219 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.219 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.219 DEBUG:boto:Params: {} 2024-04-12T03:50:42.219 DEBUG:boto:Token: None 2024-04-12T03:50:42.219 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/ 2024-04-12T03:50:42.219 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T03:50:42.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T03:50:42.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012a23cee6813fd34-006618af92-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.222 DEBUG:boto:b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:42.222 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:42.222 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:42.223 DEBUG:boto:path=/dkyges-57/?prefix=b 2024-04-12T03:50:42.223 DEBUG:boto:auth_path=/dkyges-57/?prefix=b 2024-04-12T03:50:42.223 DEBUG:boto:Method: GET 2024-04-12T03:50:42.223 DEBUG:boto:Path: /dkyges-57/?prefix=b 2024-04-12T03:50:42.223 DEBUG:boto:Data: 2024-04-12T03:50:42.223 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.223 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.223 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.223 DEBUG:boto:Params: {} 2024-04-12T03:50:42.223 DEBUG:boto:Token: None 2024-04-12T03:50:42.224 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/ 2024-04-12T03:50:42.224 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T03:50:42.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T03:50:42.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053f16f6bcfcfd5eb-006618af92-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.227 DEBUG:boto:b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:42.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:50:42.227 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:50:42.227 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:50:42.227 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:50:42.227 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:50:42.227 DEBUG:boto:Method: GET 2024-04-12T03:50:42.228 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:50:42.228 DEBUG:boto:Data: 2024-04-12T03:50:42.228 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.228 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.228 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.228 DEBUG:boto:Params: {} 2024-04-12T03:50:42.228 DEBUG:boto:Token: None 2024-04-12T03:50:42.228 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/c?versionId=null 2024-04-12T03:50:42.228 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WKT0NTfM4GA8D5rQ5eDOV58cC80= 2024-04-12T03:50:42.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WKT0NTfM4GA8D5rQ5eDOV58cC80=', 'Content-Length': '0'} 2024-04-12T03:50:42.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe543b589ebb89bd-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.231 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:50:42.231 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:50:42.231 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:50:42.231 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:50:42.231 DEBUG:boto:Method: GET 2024-04-12T03:50:42.231 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:50:42.231 DEBUG:boto:Data: 2024-04-12T03:50:42.231 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.231 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.232 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.232 DEBUG:boto:Params: {} 2024-04-12T03:50:42.232 DEBUG:boto:Token: None 2024-04-12T03:50:42.232 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/c?versionId=null 2024-04-12T03:50:42.232 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WKT0NTfM4GA8D5rQ5eDOV58cC80= 2024-04-12T03:50:42.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WKT0NTfM4GA8D5rQ5eDOV58cC80=', 'Content-Length': '0'} 2024-04-12T03:50:42.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c59471b213261a6-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.235 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:50:42.235 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:50:42.235 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:50:42.235 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:50:42.235 DEBUG:boto:Method: HEAD 2024-04-12T03:50:42.235 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:50:42.235 DEBUG:boto:Data: 2024-04-12T03:50:42.235 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.236 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.236 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.236 DEBUG:boto:Params: {} 2024-04-12T03:50:42.236 DEBUG:boto:Token: None 2024-04-12T03:50:42.236 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/c?versionId=null 2024-04-12T03:50:42.236 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fTtQG5fbfKK+LScz8vwcHDP9zfg= 2024-04-12T03:50:42.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fTtQG5fbfKK+LScz8vwcHDP9zfg=', 'Content-Length': '0'} 2024-04-12T03:50:42.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a030922b28df9dc-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.238 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:50:42.239 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:50:42.239 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:50:42.239 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:50:42.239 DEBUG:boto:Method: HEAD 2024-04-12T03:50:42.239 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:50:42.239 DEBUG:boto:Data: 2024-04-12T03:50:42.239 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.239 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.239 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.239 DEBUG:boto:Params: {} 2024-04-12T03:50:42.240 DEBUG:boto:Token: None 2024-04-12T03:50:42.240 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/c?versionId=null 2024-04-12T03:50:42.240 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fTtQG5fbfKK+LScz8vwcHDP9zfg= 2024-04-12T03:50:42.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fTtQG5fbfKK+LScz8vwcHDP9zfg=', 'Content-Length': '0'} 2024-04-12T03:50:42.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d6c204c95c97f8f-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:50:42.242 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:50:42.242 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:50:42.242 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:50:42.242 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:50:42.242 DEBUG:boto:Method: GET 2024-04-12T03:50:42.242 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:50:42.242 DEBUG:boto:Data: 2024-04-12T03:50:42.242 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.243 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.243 DEBUG:boto:Params: {} 2024-04-12T03:50:42.243 DEBUG:boto:Token: None 2024-04-12T03:50:42.243 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/c?versionId=null 2024-04-12T03:50:42.243 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WKT0NTfM4GA8D5rQ5eDOV58cC80= 2024-04-12T03:50:42.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WKT0NTfM4GA8D5rQ5eDOV58cC80=', 'Content-Length': '0'} 2024-04-12T03:50:42.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0aca77c4b9f8fd7-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.245 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:50:42.245 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:50:42.245 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:50:42.245 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:50:42.246 DEBUG:boto:Method: GET 2024-04-12T03:50:42.246 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:50:42.246 DEBUG:boto:Data: 2024-04-12T03:50:42.246 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.246 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.246 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.246 DEBUG:boto:Params: {} 2024-04-12T03:50:42.246 DEBUG:boto:Token: None 2024-04-12T03:50:42.246 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/c?versionId=null 2024-04-12T03:50:42.247 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WKT0NTfM4GA8D5rQ5eDOV58cC80= 2024-04-12T03:50:42.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WKT0NTfM4GA8D5rQ5eDOV58cC80=', 'Content-Length': '0'} 2024-04-12T03:50:42.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044fed36d57db31df-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.249 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:42.249 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:42.249 DEBUG:boto:path=/dkyges-57/?prefix=c 2024-04-12T03:50:42.249 DEBUG:boto:auth_path=/dkyges-57/?prefix=c 2024-04-12T03:50:42.249 DEBUG:boto:Method: GET 2024-04-12T03:50:42.249 DEBUG:boto:Path: /dkyges-57/?prefix=c 2024-04-12T03:50:42.249 DEBUG:boto:Data: 2024-04-12T03:50:42.249 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.250 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.250 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.250 DEBUG:boto:Params: {} 2024-04-12T03:50:42.250 DEBUG:boto:Token: None 2024-04-12T03:50:42.250 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/ 2024-04-12T03:50:42.250 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T03:50:42.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T03:50:42.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001e948f16dc2ac45-006618af92-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.253 DEBUG:boto:b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:42.253 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:42.253 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:42.253 DEBUG:boto:path=/dkyges-57/?prefix=c 2024-04-12T03:50:42.253 DEBUG:boto:auth_path=/dkyges-57/?prefix=c 2024-04-12T03:50:42.253 DEBUG:boto:Method: GET 2024-04-12T03:50:42.254 DEBUG:boto:Path: /dkyges-57/?prefix=c 2024-04-12T03:50:42.254 DEBUG:boto:Data: 2024-04-12T03:50:42.254 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.254 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.254 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.254 DEBUG:boto:Params: {} 2024-04-12T03:50:42.254 DEBUG:boto:Token: None 2024-04-12T03:50:42.254 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/ 2024-04-12T03:50:42.254 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T03:50:42.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T03:50:42.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faa80cd1f326fdf2-006618af92-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.257 DEBUG:boto:b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:42.257 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:50:42.257 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:50:42.257 DEBUG:boto:Method: HEAD 2024-04-12T03:50:42.258 DEBUG:boto:Path: /dkyges-57/c 2024-04-12T03:50:42.258 DEBUG:boto:Data: 2024-04-12T03:50:42.258 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.258 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.258 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.258 DEBUG:boto:Params: {} 2024-04-12T03:50:42.258 DEBUG:boto:Token: None 2024-04-12T03:50:42.258 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/c 2024-04-12T03:50:42.258 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:14IQT2FGso7SmQ4LvrfyuYS4j9o= 2024-04-12T03:50:42.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:14IQT2FGso7SmQ4LvrfyuYS4j9o=', 'Content-Length': '0'} 2024-04-12T03:50:42.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2e07e0ef568d198-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.260 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:50:42.261 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:50:42.261 DEBUG:boto:Method: HEAD 2024-04-12T03:50:42.261 DEBUG:boto:Path: /dkyges-57/c 2024-04-12T03:50:42.261 DEBUG:boto:Data: 2024-04-12T03:50:42.261 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.261 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.261 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.261 DEBUG:boto:Params: {} 2024-04-12T03:50:42.261 DEBUG:boto:Token: None 2024-04-12T03:50:42.261 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/c 2024-04-12T03:50:42.262 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:14IQT2FGso7SmQ4LvrfyuYS4j9o= 2024-04-12T03:50:42.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:14IQT2FGso7SmQ4LvrfyuYS4j9o=', 'Content-Length': '0'} 2024-04-12T03:50:42.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc2ef24f8b941b68-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:50:42.264 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:50:42.264 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:50:42.264 DEBUG:boto:Method: GET 2024-04-12T03:50:42.264 DEBUG:boto:Path: /dkyges-57/c 2024-04-12T03:50:42.264 DEBUG:boto:Data: 2024-04-12T03:50:42.264 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.264 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.264 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.264 DEBUG:boto:Params: {} 2024-04-12T03:50:42.265 DEBUG:boto:Token: None 2024-04-12T03:50:42.265 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/c 2024-04-12T03:50:42.265 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MeUKTRJCsgdmxFOQ7vGdBIogcgY= 2024-04-12T03:50:42.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MeUKTRJCsgdmxFOQ7vGdBIogcgY=', 'Content-Length': '0'} 2024-04-12T03:50:42.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e2c1f5b6ccd959b-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.267 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:50:42.268 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:50:42.268 DEBUG:boto:Method: GET 2024-04-12T03:50:42.268 DEBUG:boto:Path: /dkyges-57/c 2024-04-12T03:50:42.268 DEBUG:boto:Data: 2024-04-12T03:50:42.268 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.268 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.268 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.268 DEBUG:boto:Params: {} 2024-04-12T03:50:42.268 DEBUG:boto:Token: None 2024-04-12T03:50:42.268 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/c 2024-04-12T03:50:42.269 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MeUKTRJCsgdmxFOQ7vGdBIogcgY= 2024-04-12T03:50:42.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MeUKTRJCsgdmxFOQ7vGdBIogcgY=', 'Content-Length': '0'} 2024-04-12T03:50:42.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c6602d1c8f0609c-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.271 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:42.271 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:42.273 DEBUG:boto:path=/dkyges-57/?prefix=c 2024-04-12T03:50:42.273 DEBUG:boto:auth_path=/dkyges-57/?prefix=c 2024-04-12T03:50:42.273 DEBUG:boto:Method: GET 2024-04-12T03:50:42.273 DEBUG:boto:Path: /dkyges-57/?prefix=c 2024-04-12T03:50:42.274 DEBUG:boto:Data: 2024-04-12T03:50:42.274 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.274 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.274 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.274 DEBUG:boto:Params: {} 2024-04-12T03:50:42.274 DEBUG:boto:Token: None 2024-04-12T03:50:42.274 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/ 2024-04-12T03:50:42.274 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T03:50:42.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T03:50:42.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006389a8a23717532b-006618af92-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.319 DEBUG:boto:b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:42.319 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:42.319 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:42.319 DEBUG:boto:path=/dkyges-57/?prefix=c 2024-04-12T03:50:42.320 DEBUG:boto:auth_path=/dkyges-57/?prefix=c 2024-04-12T03:50:42.320 DEBUG:boto:Method: GET 2024-04-12T03:50:42.320 DEBUG:boto:Path: /dkyges-57/?prefix=c 2024-04-12T03:50:42.320 DEBUG:boto:Data: 2024-04-12T03:50:42.320 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.320 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.320 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.320 DEBUG:boto:Params: {} 2024-04-12T03:50:42.320 DEBUG:boto:Token: None 2024-04-12T03:50:42.321 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/ 2024-04-12T03:50:42.321 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T03:50:42.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T03:50:42.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fd151424f18f432-006618af92-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.323 DEBUG:boto:b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:42.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:50:42.324 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:50:42.324 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:50:42.324 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:50:42.324 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:50:42.324 DEBUG:boto:Method: GET 2024-04-12T03:50:42.324 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:50:42.324 DEBUG:boto:Data: 2024-04-12T03:50:42.324 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.324 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.325 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.325 DEBUG:boto:Params: {} 2024-04-12T03:50:42.325 DEBUG:boto:Token: None 2024-04-12T03:50:42.325 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/d?versionId=null 2024-04-12T03:50:42.325 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Bw+//k5ZFvE2VOA7dl2hMrI6tUI= 2024-04-12T03:50:42.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bw+//k5ZFvE2VOA7dl2hMrI6tUI=', 'Content-Length': '0'} 2024-04-12T03:50:42.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000949716fc44723e7-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.327 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:50:42.327 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:50:42.327 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:50:42.327 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:50:42.328 DEBUG:boto:Method: GET 2024-04-12T03:50:42.328 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:50:42.328 DEBUG:boto:Data: 2024-04-12T03:50:42.328 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.328 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.328 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.328 DEBUG:boto:Params: {} 2024-04-12T03:50:42.328 DEBUG:boto:Token: None 2024-04-12T03:50:42.328 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/d?versionId=null 2024-04-12T03:50:42.328 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Bw+//k5ZFvE2VOA7dl2hMrI6tUI= 2024-04-12T03:50:42.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bw+//k5ZFvE2VOA7dl2hMrI6tUI=', 'Content-Length': '0'} 2024-04-12T03:50:42.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000670599aa25421b0e-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.374 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:50:42.374 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:50:42.375 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:50:42.375 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:50:42.375 DEBUG:boto:Method: HEAD 2024-04-12T03:50:42.375 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:50:42.375 DEBUG:boto:Data: 2024-04-12T03:50:42.375 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.375 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.375 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.375 DEBUG:boto:Params: {} 2024-04-12T03:50:42.375 DEBUG:boto:Token: None 2024-04-12T03:50:42.376 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/d?versionId=null 2024-04-12T03:50:42.376 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wV2XdrkxHBbG6qYqw5fmfrJVF34= 2024-04-12T03:50:42.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wV2XdrkxHBbG6qYqw5fmfrJVF34=', 'Content-Length': '0'} 2024-04-12T03:50:42.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000715d70b315afe944-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.378 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:50:42.378 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:50:42.378 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:50:42.378 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:50:42.378 DEBUG:boto:Method: HEAD 2024-04-12T03:50:42.378 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:50:42.378 DEBUG:boto:Data: 2024-04-12T03:50:42.379 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.379 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.379 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.379 DEBUG:boto:Params: {} 2024-04-12T03:50:42.379 DEBUG:boto:Token: None 2024-04-12T03:50:42.379 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/d?versionId=null 2024-04-12T03:50:42.379 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wV2XdrkxHBbG6qYqw5fmfrJVF34= 2024-04-12T03:50:42.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wV2XdrkxHBbG6qYqw5fmfrJVF34=', 'Content-Length': '0'} 2024-04-12T03:50:42.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9eff826cb5d204f-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:50:42.381 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:50:42.382 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:50:42.382 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:50:42.382 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:50:42.382 DEBUG:boto:Method: GET 2024-04-12T03:50:42.382 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:50:42.382 DEBUG:boto:Data: 2024-04-12T03:50:42.382 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.382 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.382 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.382 DEBUG:boto:Params: {} 2024-04-12T03:50:42.382 DEBUG:boto:Token: None 2024-04-12T03:50:42.383 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/d?versionId=null 2024-04-12T03:50:42.383 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Bw+//k5ZFvE2VOA7dl2hMrI6tUI= 2024-04-12T03:50:42.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bw+//k5ZFvE2VOA7dl2hMrI6tUI=', 'Content-Length': '0'} 2024-04-12T03:50:42.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a07da31eaecc7420-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.385 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:50:42.385 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:50:42.385 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:50:42.385 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:50:42.385 DEBUG:boto:Method: GET 2024-04-12T03:50:42.385 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:50:42.385 DEBUG:boto:Data: 2024-04-12T03:50:42.386 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.386 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.386 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.386 DEBUG:boto:Params: {} 2024-04-12T03:50:42.386 DEBUG:boto:Token: None 2024-04-12T03:50:42.386 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/d?versionId=null 2024-04-12T03:50:42.386 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Bw+//k5ZFvE2VOA7dl2hMrI6tUI= 2024-04-12T03:50:42.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bw+//k5ZFvE2VOA7dl2hMrI6tUI=', 'Content-Length': '0'} 2024-04-12T03:50:42.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e7dad277fef946e-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.388 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:42.388 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:42.389 DEBUG:boto:path=/dkyges-57/?prefix=d 2024-04-12T03:50:42.389 DEBUG:boto:auth_path=/dkyges-57/?prefix=d 2024-04-12T03:50:42.389 DEBUG:boto:Method: GET 2024-04-12T03:50:42.389 DEBUG:boto:Path: /dkyges-57/?prefix=d 2024-04-12T03:50:42.389 DEBUG:boto:Data: 2024-04-12T03:50:42.389 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.389 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.389 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.389 DEBUG:boto:Params: {} 2024-04-12T03:50:42.390 DEBUG:boto:Token: None 2024-04-12T03:50:42.390 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/ 2024-04-12T03:50:42.390 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T03:50:42.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T03:50:42.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bab24032ae63746f-006618af92-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.393 DEBUG:boto:b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:42.393 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:42.393 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:42.393 DEBUG:boto:path=/dkyges-57/?prefix=d 2024-04-12T03:50:42.393 DEBUG:boto:auth_path=/dkyges-57/?prefix=d 2024-04-12T03:50:42.394 DEBUG:boto:Method: GET 2024-04-12T03:50:42.394 DEBUG:boto:Path: /dkyges-57/?prefix=d 2024-04-12T03:50:42.394 DEBUG:boto:Data: 2024-04-12T03:50:42.394 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.394 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.394 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.394 DEBUG:boto:Params: {} 2024-04-12T03:50:42.394 DEBUG:boto:Token: None 2024-04-12T03:50:42.394 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/ 2024-04-12T03:50:42.394 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T03:50:42.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T03:50:42.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000595b399102d5392a-006618af92-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.398 DEBUG:boto:b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:42.398 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:50:42.398 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:50:42.398 DEBUG:boto:Method: HEAD 2024-04-12T03:50:42.399 DEBUG:boto:Path: /dkyges-57/d 2024-04-12T03:50:42.399 DEBUG:boto:Data: 2024-04-12T03:50:42.399 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.399 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.399 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.399 DEBUG:boto:Params: {} 2024-04-12T03:50:42.399 DEBUG:boto:Token: None 2024-04-12T03:50:42.399 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/d 2024-04-12T03:50:42.399 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D+NJpaaAT2F/zDlF1w2a5rTp0gg= 2024-04-12T03:50:42.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D+NJpaaAT2F/zDlF1w2a5rTp0gg=', 'Content-Length': '0'} 2024-04-12T03:50:42.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42fe89299def7dc-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.401 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:50:42.402 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:50:42.402 DEBUG:boto:Method: HEAD 2024-04-12T03:50:42.402 DEBUG:boto:Path: /dkyges-57/d 2024-04-12T03:50:42.402 DEBUG:boto:Data: 2024-04-12T03:50:42.402 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.402 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.402 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.402 DEBUG:boto:Params: {} 2024-04-12T03:50:42.402 DEBUG:boto:Token: None 2024-04-12T03:50:42.402 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/d 2024-04-12T03:50:42.403 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D+NJpaaAT2F/zDlF1w2a5rTp0gg= 2024-04-12T03:50:42.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D+NJpaaAT2F/zDlF1w2a5rTp0gg=', 'Content-Length': '0'} 2024-04-12T03:50:42.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000601d2a19b4fae2b3-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:50:42.405 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:50:42.405 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:50:42.405 DEBUG:boto:Method: GET 2024-04-12T03:50:42.405 DEBUG:boto:Path: /dkyges-57/d 2024-04-12T03:50:42.405 DEBUG:boto:Data: 2024-04-12T03:50:42.405 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.405 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.405 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.405 DEBUG:boto:Params: {} 2024-04-12T03:50:42.405 DEBUG:boto:Token: None 2024-04-12T03:50:42.406 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/d 2024-04-12T03:50:42.406 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JwYfbopNZSwXMd/9zNxL9zc0xWY= 2024-04-12T03:50:42.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JwYfbopNZSwXMd/9zNxL9zc0xWY=', 'Content-Length': '0'} 2024-04-12T03:50:42.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9cdbe80fdf7e010-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.408 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:50:42.408 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:50:42.408 DEBUG:boto:Method: GET 2024-04-12T03:50:42.408 DEBUG:boto:Path: /dkyges-57/d 2024-04-12T03:50:42.408 DEBUG:boto:Data: 2024-04-12T03:50:42.408 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.408 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.408 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.408 DEBUG:boto:Params: {} 2024-04-12T03:50:42.408 DEBUG:boto:Token: None 2024-04-12T03:50:42.409 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/d 2024-04-12T03:50:42.409 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JwYfbopNZSwXMd/9zNxL9zc0xWY= 2024-04-12T03:50:42.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JwYfbopNZSwXMd/9zNxL9zc0xWY=', 'Content-Length': '0'} 2024-04-12T03:50:42.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fb5b2bba4fe4db5-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.411 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:42.411 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:42.411 DEBUG:boto:path=/dkyges-57/?prefix=d 2024-04-12T03:50:42.412 DEBUG:boto:auth_path=/dkyges-57/?prefix=d 2024-04-12T03:50:42.412 DEBUG:boto:Method: GET 2024-04-12T03:50:42.412 DEBUG:boto:Path: /dkyges-57/?prefix=d 2024-04-12T03:50:42.412 DEBUG:boto:Data: 2024-04-12T03:50:42.412 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.412 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:50:42.412 DEBUG:boto:Port: 8001 2024-04-12T03:50:42.412 DEBUG:boto:Params: {} 2024-04-12T03:50:42.412 DEBUG:boto:Token: None 2024-04-12T03:50:42.412 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/ 2024-04-12T03:50:42.413 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T03:50:42.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T03:50:42.415 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050f80e44d702c898-006618af92-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.415 DEBUG:boto:b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:42.415 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:42.416 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:42.416 DEBUG:boto:path=/dkyges-57/?prefix=d 2024-04-12T03:50:42.416 DEBUG:boto:auth_path=/dkyges-57/?prefix=d 2024-04-12T03:50:42.416 DEBUG:boto:Method: GET 2024-04-12T03:50:42.416 DEBUG:boto:Path: /dkyges-57/?prefix=d 2024-04-12T03:50:42.416 DEBUG:boto:Data: 2024-04-12T03:50:42.416 DEBUG:boto:Headers: {} 2024-04-12T03:50:42.416 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:42.416 DEBUG:boto:Port: 8000 2024-04-12T03:50:42.416 DEBUG:boto:Params: {} 2024-04-12T03:50:42.416 DEBUG:boto:Token: None 2024-04-12T03:50:42.417 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:50:42 GMT /dkyges-57/ 2024-04-12T03:50:42.417 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T03:50:42.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T03:50:42.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a68a8a5808d9bd74-006618af92-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:42.420 DEBUG:boto:b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:50:42.420 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-57 zones={a2, a1} 2024-04-12T03:50:42.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:50:42.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:50:42.420 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dkyges-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:50:42.477 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:42.477 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:42.530 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:50:42.530 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:50:42.530 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding": "None", 2024-04-12T03:50:42.530 INFO:teuthology.orchestra.run.smithi043.stdout: "current_index": { 2024-04-12T03:50:42.530 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 0, 2024-04-12T03:50:42.531 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:50:42.531 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "Normal", 2024-04-12T03:50:42.531 INFO:teuthology.orchestra.run.smithi043.stdout: "normal": { 2024-04-12T03:50:42.532 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 11, 2024-04-12T03:50:42.532 INFO:teuthology.orchestra.run.smithi043.stdout: "hash_type": "Mod" 2024-04-12T03:50:42.532 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.532 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:42.533 INFO:teuthology.orchestra.run.smithi043.stdout: "logs": [ 2024-04-12T03:50:42.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:42.533 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 0, 2024-04-12T03:50:42.533 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:50:42.533 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "InIndex", 2024-04-12T03:50:42.533 INFO:teuthology.orchestra.run.smithi043.stdout: "in_index": { 2024-04-12T03:50:42.533 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 0, 2024-04-12T03:50:42.534 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:50:42.534 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 11, 2024-04-12T03:50:42.534 INFO:teuthology.orchestra.run.smithi043.stdout: "hash_type": "Mod" 2024-04-12T03:50:42.534 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.534 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.534 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.535 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.535 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:50:42.535 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.535 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:50:42.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dkyges-57', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:42.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dkyges-57', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:42.536 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dkyges-57 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:50:42.591 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:42.592 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:42.641 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:50:42.641 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:42.641 INFO:teuthology.orchestra.run.smithi043.stdout: "op_id": "1#00000000001.53.6", 2024-04-12T03:50:42.641 INFO:teuthology.orchestra.run.smithi043.stdout: "op_tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.8775263116062048419", 2024-04-12T03:50:42.641 INFO:teuthology.orchestra.run.smithi043.stdout: "op": "write", 2024-04-12T03:50:42.641 INFO:teuthology.orchestra.run.smithi043.stdout: "object": "a", 2024-04-12T03:50:42.641 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:50:42.641 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete", 2024-04-12T03:50:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: "index_ver": 1, 2024-04-12T03:50:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:50:21.205175995Z", 2024-04-12T03:50:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:50:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:50:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T03:50:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: "bilog_flags": 0, 2024-04-12T03:50:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned": false, 2024-04-12T03:50:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "", 2024-04-12T03:50:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "", 2024-04-12T03:50:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: "zones_trace": [ 2024-04-12T03:50:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: "entry": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20" 2024-04-12T03:50:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:50:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: "op_id": "5#00000000001.19.6", 2024-04-12T03:50:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: "op_tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.6696391949831587819", 2024-04-12T03:50:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: "op": "write", 2024-04-12T03:50:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: "object": "d", 2024-04-12T03:50:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:50:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete", 2024-04-12T03:50:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "index_ver": 1, 2024-04-12T03:50:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:50:21.275981344Z", 2024-04-12T03:50:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:50:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:50:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 8 2024-04-12T03:50:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "bilog_flags": 0, 2024-04-12T03:50:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned": false, 2024-04-12T03:50:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "", 2024-04-12T03:50:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "", 2024-04-12T03:50:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: "zones_trace": [ 2024-04-12T03:50:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: "entry": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20" 2024-04-12T03:50:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:50:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: "op_id": "8#00000000001.93.6", 2024-04-12T03:50:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: "op_tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.2825657797998053591", 2024-04-12T03:50:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: "op": "write", 2024-04-12T03:50:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: "object": "c", 2024-04-12T03:50:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:50:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete", 2024-04-12T03:50:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: "index_ver": 1, 2024-04-12T03:50:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:50:21.264841306Z", 2024-04-12T03:50:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:50:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:50:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 18 2024-04-12T03:50:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: "bilog_flags": 0, 2024-04-12T03:50:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned": false, 2024-04-12T03:50:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "", 2024-04-12T03:50:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "", 2024-04-12T03:50:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: "zones_trace": [ 2024-04-12T03:50:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: "entry": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20" 2024-04-12T03:50:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:50:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: "op_id": "10#00000000001.36.6", 2024-04-12T03:50:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: "op_tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.6615111383423553842", 2024-04-12T03:50:42.648 INFO:teuthology.orchestra.run.smithi043.stdout: "op": "write", 2024-04-12T03:50:42.648 INFO:teuthology.orchestra.run.smithi043.stdout: "object": "b", 2024-04-12T03:50:42.648 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:50:42.648 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "complete", 2024-04-12T03:50:42.648 INFO:teuthology.orchestra.run.smithi043.stdout: "index_ver": 1, 2024-04-12T03:50:42.648 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:50:21.217597097Z", 2024-04-12T03:50:42.648 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:50:42.648 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:50:42.648 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2 2024-04-12T03:50:42.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:42.648 INFO:teuthology.orchestra.run.smithi043.stdout: "bilog_flags": 0, 2024-04-12T03:50:42.649 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned": false, 2024-04-12T03:50:42.649 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "", 2024-04-12T03:50:42.649 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "", 2024-04-12T03:50:42.649 INFO:teuthology.orchestra.run.smithi043.stdout: "zones_trace": [ 2024-04-12T03:50:42.649 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:42.649 INFO:teuthology.orchestra.run.smithi043.stdout: "entry": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20" 2024-04-12T03:50:42.649 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.649 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:50:42.649 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.649 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:50:42.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dkyges-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:50:42.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dkyges-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:50:42.651 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dkyges-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:50:42.707 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:42.707 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:42.752 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T03:50:42.752 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-57 2024-04-12T03:50:42.757 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 4 2024-04-12T03:50:42.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:50:42.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:50:42.800 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dkyges-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:50:42.852 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:42.852 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:42.901 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:50:42.901 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:50:42.901 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding": "None", 2024-04-12T03:50:42.901 INFO:teuthology.orchestra.run.smithi043.stdout: "current_index": { 2024-04-12T03:50:42.901 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 1, 2024-04-12T03:50:42.901 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:50:42.901 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "Normal", 2024-04-12T03:50:42.901 INFO:teuthology.orchestra.run.smithi043.stdout: "normal": { 2024-04-12T03:50:42.902 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 3, 2024-04-12T03:50:42.902 INFO:teuthology.orchestra.run.smithi043.stdout: "hash_type": "Mod" 2024-04-12T03:50:42.902 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.902 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:42.902 INFO:teuthology.orchestra.run.smithi043.stdout: "logs": [ 2024-04-12T03:50:42.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:42.902 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 0, 2024-04-12T03:50:42.902 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:50:42.902 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "InIndex", 2024-04-12T03:50:42.903 INFO:teuthology.orchestra.run.smithi043.stdout: "in_index": { 2024-04-12T03:50:42.903 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 0, 2024-04-12T03:50:42.903 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:50:42.903 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 11, 2024-04-12T03:50:42.903 INFO:teuthology.orchestra.run.smithi043.stdout: "hash_type": "Mod" 2024-04-12T03:50:42.903 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.903 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.903 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:50:42.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:50:42.903 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 1, 2024-04-12T03:50:42.904 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:50:42.904 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "InIndex", 2024-04-12T03:50:42.904 INFO:teuthology.orchestra.run.smithi043.stdout: "in_index": { 2024-04-12T03:50:42.904 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 1, 2024-04-12T03:50:42.904 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:50:42.904 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 3, 2024-04-12T03:50:42.904 INFO:teuthology.orchestra.run.smithi043.stdout: "hash_type": "Mod" 2024-04-12T03:50:42.904 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.904 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.904 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.904 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.904 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:50:42.904 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:50:42.905 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:50:42.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dkyges-57', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:42.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dkyges-57', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:50:42.907 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dkyges-57 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:50:42.961 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:42.962 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:43.009 INFO:teuthology.orchestra.run.smithi043.stdout:[] 2024-04-12T03:50:43.011 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:43.011 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:43.011 DEBUG:boto:Method: HEAD 2024-04-12T03:50:43.012 DEBUG:boto:Path: /dkyges-57/ 2024-04-12T03:50:43.012 DEBUG:boto:Data: 2024-04-12T03:50:43.012 DEBUG:boto:Headers: {} 2024-04-12T03:50:43.012 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:43.012 DEBUG:boto:Port: 8000 2024-04-12T03:50:43.012 DEBUG:boto:Params: {} 2024-04-12T03:50:43.012 DEBUG:boto:Token: None 2024-04-12T03:50:43.012 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:43 GMT /dkyges-57/ 2024-04-12T03:50:43.012 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uzcilWOkK6VgOd7p8ce8aLkwfRU= 2024-04-12T03:50:43.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uzcilWOkK6VgOd7p8ce8aLkwfRU=', 'Content-Length': '0'} 2024-04-12T03:50:43.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074c539e8ada6b85f-006618af93-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:43.015 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:50:43.015 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:50:43.015 DEBUG:boto:Method: PUT 2024-04-12T03:50:43.015 DEBUG:boto:Path: /dkyges-57/e 2024-04-12T03:50:43.015 DEBUG:boto:Data: 2024-04-12T03:50:43.015 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:50:43.015 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:43.016 DEBUG:boto:Port: 8000 2024-04-12T03:50:43.016 DEBUG:boto:Params: {} 2024-04-12T03:50:43.016 DEBUG:boto:Token: None 2024-04-12T03:50:43.016 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:50:43 GMT /dkyges-57/e 2024-04-12T03:50:43.016 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YJHjnE2QCuKTeFrHao5whOc/bqI= 2024-04-12T03:50:43.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YJHjnE2QCuKTeFrHao5whOc/bqI='} 2024-04-12T03:50:43.059 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084764007ccf4ac14-006618af93-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:43.059 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:43.059 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:43.059 DEBUG:boto:Method: HEAD 2024-04-12T03:50:43.059 DEBUG:boto:Path: /dkyges-57/ 2024-04-12T03:50:43.060 DEBUG:boto:Data: 2024-04-12T03:50:43.060 DEBUG:boto:Headers: {} 2024-04-12T03:50:43.060 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:43.060 DEBUG:boto:Port: 8000 2024-04-12T03:50:43.060 DEBUG:boto:Params: {} 2024-04-12T03:50:43.060 DEBUG:boto:Token: None 2024-04-12T03:50:43.060 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:43 GMT /dkyges-57/ 2024-04-12T03:50:43.060 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uzcilWOkK6VgOd7p8ce8aLkwfRU= 2024-04-12T03:50:43.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uzcilWOkK6VgOd7p8ce8aLkwfRU=', 'Content-Length': '0'} 2024-04-12T03:50:43.062 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012ada3b8c0f95349-006618af93-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:43.063 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:50:43.063 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:50:43.063 DEBUG:boto:Method: PUT 2024-04-12T03:50:43.063 DEBUG:boto:Path: /dkyges-57/f 2024-04-12T03:50:43.063 DEBUG:boto:Data: 2024-04-12T03:50:43.063 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:50:43.063 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:43.063 DEBUG:boto:Port: 8000 2024-04-12T03:50:43.063 DEBUG:boto:Params: {} 2024-04-12T03:50:43.063 DEBUG:boto:Token: None 2024-04-12T03:50:43.064 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:50:43 GMT /dkyges-57/f 2024-04-12T03:50:43.064 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EyNxIdRQgOp2bMBpQeUVIYwLKjE= 2024-04-12T03:50:43.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EyNxIdRQgOp2bMBpQeUVIYwLKjE='} 2024-04-12T03:50:43.072 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ddc531ce0394441-006618af93-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:43.072 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:43.072 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:43.072 DEBUG:boto:Method: HEAD 2024-04-12T03:50:43.072 DEBUG:boto:Path: /dkyges-57/ 2024-04-12T03:50:43.072 DEBUG:boto:Data: 2024-04-12T03:50:43.072 DEBUG:boto:Headers: {} 2024-04-12T03:50:43.072 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:43.073 DEBUG:boto:Port: 8000 2024-04-12T03:50:43.073 DEBUG:boto:Params: {} 2024-04-12T03:50:43.073 DEBUG:boto:Token: None 2024-04-12T03:50:43.073 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:43 GMT /dkyges-57/ 2024-04-12T03:50:43.073 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uzcilWOkK6VgOd7p8ce8aLkwfRU= 2024-04-12T03:50:43.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uzcilWOkK6VgOd7p8ce8aLkwfRU=', 'Content-Length': '0'} 2024-04-12T03:50:43.075 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bff2da88c20ba29-006618af93-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:43.075 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:50:43.076 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:50:43.076 DEBUG:boto:Method: PUT 2024-04-12T03:50:43.076 DEBUG:boto:Path: /dkyges-57/g 2024-04-12T03:50:43.076 DEBUG:boto:Data: 2024-04-12T03:50:43.076 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:50:43.076 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:43.076 DEBUG:boto:Port: 8000 2024-04-12T03:50:43.076 DEBUG:boto:Params: {} 2024-04-12T03:50:43.076 DEBUG:boto:Token: None 2024-04-12T03:50:43.076 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:50:43 GMT /dkyges-57/g 2024-04-12T03:50:43.077 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BfCoICVOZIUzRB8uc4bNRxEZ2II= 2024-04-12T03:50:43.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BfCoICVOZIUzRB8uc4bNRxEZ2II='} 2024-04-12T03:50:43.118 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000201b931f64420830-006618af93-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:43.119 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:50:43.119 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:50:43.119 DEBUG:boto:Method: HEAD 2024-04-12T03:50:43.119 DEBUG:boto:Path: /dkyges-57/ 2024-04-12T03:50:43.119 DEBUG:boto:Data: 2024-04-12T03:50:43.119 DEBUG:boto:Headers: {} 2024-04-12T03:50:43.119 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:43.119 DEBUG:boto:Port: 8000 2024-04-12T03:50:43.120 DEBUG:boto:Params: {} 2024-04-12T03:50:43.120 DEBUG:boto:Token: None 2024-04-12T03:50:43.120 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:50:43 GMT /dkyges-57/ 2024-04-12T03:50:43.120 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uzcilWOkK6VgOd7p8ce8aLkwfRU= 2024-04-12T03:50:43.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uzcilWOkK6VgOd7p8ce8aLkwfRU=', 'Content-Length': '0'} 2024-04-12T03:50:43.122 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6d6e168af785a07-006618af93-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:43.122 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:50:43.123 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:50:43.123 DEBUG:boto:Method: PUT 2024-04-12T03:50:43.123 DEBUG:boto:Path: /dkyges-57/h 2024-04-12T03:50:43.123 DEBUG:boto:Data: 2024-04-12T03:50:43.123 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:50:43.123 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:50:43.123 DEBUG:boto:Port: 8000 2024-04-12T03:50:43.123 DEBUG:boto:Params: {} 2024-04-12T03:50:43.123 DEBUG:boto:Token: None 2024-04-12T03:50:43.124 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:50:43 GMT /dkyges-57/h 2024-04-12T03:50:43.124 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C04i4i/4Bm8dexUQuPfRTpwB41o= 2024-04-12T03:50:43.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C04i4i/4Bm8dexUQuPfRTpwB41o='} 2024-04-12T03:50:43.129 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0bbf016c232f4b7-006618af93-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:50:43.129 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-57 2024-04-12T03:50:43.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:50:43.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:50:43.129 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:50:43.197 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:50:43.198 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:50:43.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:43.246+0000 7f8986a60780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T03:50:48.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:48.247+0000 7f8986a60780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T03:50:53.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:53.247+0000 7f8986a60780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T03:50:58.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:50:58.249+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T03:50:58.243 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , ] 2024-04-12T03:50:58.243 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T03:51:03.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:51:03.250+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:03.244 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , ] 2024-04-12T03:51:03.244 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T03:51:08.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:51:08.251+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:08.245 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , ] 2024-04-12T03:51:08.245 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T03:51:13.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:51:13.252+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:13.245 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , ] 2024-04-12T03:51:13.246 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T03:51:18.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:51:18.252+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:18.246 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.39.6, , ] 2024-04-12T03:51:18.246 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T03:51:23.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:51:23.253+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:23.247 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.39.6, , ] 2024-04-12T03:51:23.247 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T03:51:27.123 INFO:tasks.rgw.c1.client.0.smithi043.stdout:2024-04-12T03:51:27.130+0000 7f1e272bd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T03:51:28.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:51:28.255+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:28.248 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.39.6, , ] 2024-04-12T03:51:28.248 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T03:51:33.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:51:33.255+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:33.248 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.39.6, , ] 2024-04-12T03:51:33.248 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T03:51:38.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:51:38.256+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:38.249 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.39.6, 00000000002.54.6, ] 2024-04-12T03:51:38.249 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T03:51:43.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:51:43.257+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:43.250 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.39.6, 00000000002.54.6, ] 2024-04-12T03:51:43.250 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T03:51:48.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:51:48.258+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:48.251 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.39.6, 00000000002.54.6, ] 2024-04-12T03:51:48.251 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T03:51:53.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:51:53.258+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:53.251 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.39.6, 00000000002.54.6, ] 2024-04-12T03:51:53.251 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T03:51:58.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:51:58.260+0000 7f8986a60780 1 bucket sync caught up with source: 2024-04-12T03:51:58.253 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T03:51:58.253 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T03:51:58.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:51:58.260+0000 7f8986a60780 0 bucket checkpoint complete 2024-04-12T03:51:58.272 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-57 2024-04-12T03:51:58.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:51:58.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:51:58.273 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:51:58.340 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:51:58.340 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:51:58.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:51:58.390+0000 7f934ab19780 1 waiting for incremental sync to catch up: 2024-04-12T03:51:58.383 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.2910.6, , , 00000000001.3705.6, 00000000001.1066.6, , , 00000000001.1602.6, , 00000000002.3706.6] 2024-04-12T03:51:58.383 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.2910.6, , , 00000000001.3705.6, 00000000001.1066.6, , , 00000000002.1606.6, , 00000000002.3706.6] 2024-04-12T03:52:03.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:03.391+0000 7f934ab19780 1 waiting for incremental sync to catch up: 2024-04-12T03:52:03.384 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.2910.6, , , 00000000001.3705.6, 00000000001.1066.6, , , 00000000001.1602.6, , 00000000002.3706.6] 2024-04-12T03:52:03.384 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.2910.6, , , 00000000001.3705.6, 00000000001.1066.6, , , 00000000002.1606.6, , 00000000002.3706.6] 2024-04-12T03:52:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:08.392+0000 7f934ab19780 1 bucket sync caught up with source: 2024-04-12T03:52:08.386 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.2910.6, , , 00000000001.3705.6, 00000000001.1066.6, , , 00000000002.1606.6, , 00000000002.3706.6] 2024-04-12T03:52:08.386 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.2910.6, , , 00000000001.3705.6, 00000000001.1066.6, , , 00000000002.1606.6, , 00000000002.3706.6] 2024-04-12T03:52:08.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:08.392+0000 7f934ab19780 0 bucket checkpoint complete 2024-04-12T03:52:08.404 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-57 zones={a2, a1} 2024-04-12T03:52:08.404 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.404 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.404 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.405 DEBUG:boto:Path: /dkyges-57/ 2024-04-12T03:52:08.405 DEBUG:boto:Data: 2024-04-12T03:52:08.405 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.405 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.405 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.405 DEBUG:boto:Params: {} 2024-04-12T03:52:08.405 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:52:08.406 DEBUG:boto:Token: None 2024-04-12T03:52:08.406 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.406 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7Xz8aTP9keMruBOZgpVAmE7K0uY= 2024-04-12T03:52:08.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7Xz8aTP9keMruBOZgpVAmE7K0uY=', 'Content-Length': '0'} 2024-04-12T03:52:08.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a44947787cc40809-006618afe8-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.438 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.438 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.438 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.438 DEBUG:boto:Path: /dkyges-57/ 2024-04-12T03:52:08.438 DEBUG:boto:Data: 2024-04-12T03:52:08.438 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.438 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.438 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.438 DEBUG:boto:Params: {} 2024-04-12T03:52:08.439 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:52:08.439 DEBUG:boto:Token: None 2024-04-12T03:52:08.439 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.439 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7Xz8aTP9keMruBOZgpVAmE7K0uY= 2024-04-12T03:52:08.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7Xz8aTP9keMruBOZgpVAmE7K0uY=', 'Content-Length': '0'} 2024-04-12T03:52:08.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018dd848c8adc9144-006618afe8-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.452 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:52:08.452 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.452 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.452 DEBUG:boto:path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:52:08.452 DEBUG:boto:auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:52:08.452 DEBUG:boto:Method: GET 2024-04-12T03:52:08.452 DEBUG:boto:Path: /dkyges-57/?versions&max-keys=999 2024-04-12T03:52:08.452 DEBUG:boto:Data: 2024-04-12T03:52:08.452 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.453 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.453 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.453 DEBUG:boto:Params: {} 2024-04-12T03:52:08.453 DEBUG:boto:Token: None 2024-04-12T03:52:08.453 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/?versions 2024-04-12T03:52:08.453 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMgTrz9yOLU+gZfkQZLyMnk0OUk= 2024-04-12T03:52:08.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMgTrz9yOLU+gZfkQZLyMnk0OUk=', 'Content-Length': '0'} 2024-04-12T03:52:08.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060ab74bcae4ddf63-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.456 DEBUG:boto:b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.457 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:52:08.457 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:52:08.457 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:52:08.457 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:52:08.457 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-12T03:52:08.457 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-12T03:52:08.457 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-12T03:52:08.458 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-12T03:52:08.458 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:52:08.458 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.458 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.458 DEBUG:boto:path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:52:08.458 DEBUG:boto:auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:52:08.458 DEBUG:boto:Method: GET 2024-04-12T03:52:08.458 DEBUG:boto:Path: /dkyges-57/?versions&max-keys=999 2024-04-12T03:52:08.458 DEBUG:boto:Data: 2024-04-12T03:52:08.459 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.459 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.459 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.459 DEBUG:boto:Params: {} 2024-04-12T03:52:08.459 DEBUG:boto:Token: None 2024-04-12T03:52:08.459 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/?versions 2024-04-12T03:52:08.459 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMgTrz9yOLU+gZfkQZLyMnk0OUk= 2024-04-12T03:52:08.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMgTrz9yOLU+gZfkQZLyMnk0OUk=', 'Content-Length': '0'} 2024-04-12T03:52:08.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001291255b2412b1ff-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.461 DEBUG:boto:b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.462 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:52:08.462 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:52:08.462 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:52:08.462 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:52:08.462 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-12T03:52:08.462 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-12T03:52:08.462 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-12T03:52:08.463 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-12T03:52:08.463 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.463 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.463 DEBUG:boto:path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:52:08.463 DEBUG:boto:auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:52:08.463 DEBUG:boto:Method: GET 2024-04-12T03:52:08.463 DEBUG:boto:Path: /dkyges-57/?versions&max-keys=999 2024-04-12T03:52:08.463 DEBUG:boto:Data: 2024-04-12T03:52:08.463 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.463 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.463 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.464 DEBUG:boto:Params: {} 2024-04-12T03:52:08.464 DEBUG:boto:Token: None 2024-04-12T03:52:08.464 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/?versions 2024-04-12T03:52:08.464 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMgTrz9yOLU+gZfkQZLyMnk0OUk= 2024-04-12T03:52:08.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMgTrz9yOLU+gZfkQZLyMnk0OUk=', 'Content-Length': '0'} 2024-04-12T03:52:08.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ec5a79609e5dcd0-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.466 DEBUG:boto:b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.467 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.467 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.467 DEBUG:boto:path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:52:08.467 DEBUG:boto:auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:52:08.467 DEBUG:boto:Method: GET 2024-04-12T03:52:08.467 DEBUG:boto:Path: /dkyges-57/?versions&max-keys=999 2024-04-12T03:52:08.468 DEBUG:boto:Data: 2024-04-12T03:52:08.468 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.468 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.468 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.468 DEBUG:boto:Params: {} 2024-04-12T03:52:08.468 DEBUG:boto:Token: None 2024-04-12T03:52:08.468 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/?versions 2024-04-12T03:52:08.468 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMgTrz9yOLU+gZfkQZLyMnk0OUk= 2024-04-12T03:52:08.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMgTrz9yOLU+gZfkQZLyMnk0OUk=', 'Content-Length': '0'} 2024-04-12T03:52:08.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fe6b25105f004e7-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.470 DEBUG:boto:b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:52:08.471 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:08.471 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:08.471 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:52:08.471 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:52:08.471 DEBUG:boto:Method: GET 2024-04-12T03:52:08.472 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:52:08.472 DEBUG:boto:Data: 2024-04-12T03:52:08.472 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.472 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.472 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.472 DEBUG:boto:Params: {} 2024-04-12T03:52:08.472 DEBUG:boto:Token: None 2024-04-12T03:52:08.472 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/a?versionId=null 2024-04-12T03:52:08.472 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8Ghuk8ws0gq7lXtyobRkBov+YRA= 2024-04-12T03:52:08.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8Ghuk8ws0gq7lXtyobRkBov+YRA=', 'Content-Length': '0'} 2024-04-12T03:52:08.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb12d855b82957f0-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.475 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:08.475 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:08.475 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:52:08.475 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:52:08.475 DEBUG:boto:Method: GET 2024-04-12T03:52:08.475 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:52:08.475 DEBUG:boto:Data: 2024-04-12T03:52:08.475 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.475 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.475 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.475 DEBUG:boto:Params: {} 2024-04-12T03:52:08.476 DEBUG:boto:Token: None 2024-04-12T03:52:08.476 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/a?versionId=null 2024-04-12T03:52:08.476 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8Ghuk8ws0gq7lXtyobRkBov+YRA= 2024-04-12T03:52:08.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8Ghuk8ws0gq7lXtyobRkBov+YRA=', 'Content-Length': '0'} 2024-04-12T03:52:08.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001964f53600aaa44-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.478 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:08.478 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:08.478 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:52:08.479 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:52:08.479 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.479 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:52:08.479 DEBUG:boto:Data: 2024-04-12T03:52:08.479 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.479 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.479 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.479 DEBUG:boto:Params: {} 2024-04-12T03:52:08.479 DEBUG:boto:Token: None 2024-04-12T03:52:08.479 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/a?versionId=null 2024-04-12T03:52:08.479 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:87zSg4f79b89o7axtVlYTIH2sXA= 2024-04-12T03:52:08.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:87zSg4f79b89o7axtVlYTIH2sXA=', 'Content-Length': '0'} 2024-04-12T03:52:08.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012172f7597fd2cab-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.481 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:08.481 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:08.481 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:52:08.482 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:52:08.482 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.482 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:52:08.482 DEBUG:boto:Data: 2024-04-12T03:52:08.482 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.482 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.482 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.482 DEBUG:boto:Params: {} 2024-04-12T03:52:08.482 DEBUG:boto:Token: None 2024-04-12T03:52:08.483 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/a?versionId=null 2024-04-12T03:52:08.483 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:87zSg4f79b89o7axtVlYTIH2sXA= 2024-04-12T03:52:08.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:87zSg4f79b89o7axtVlYTIH2sXA=', 'Content-Length': '0'} 2024-04-12T03:52:08.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9b3133a309cf805-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:52:08.485 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:08.485 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:08.485 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:52:08.485 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:52:08.485 DEBUG:boto:Method: GET 2024-04-12T03:52:08.485 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:52:08.485 DEBUG:boto:Data: 2024-04-12T03:52:08.485 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.486 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.486 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.486 DEBUG:boto:Params: {} 2024-04-12T03:52:08.486 DEBUG:boto:Token: None 2024-04-12T03:52:08.486 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/a?versionId=null 2024-04-12T03:52:08.486 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8Ghuk8ws0gq7lXtyobRkBov+YRA= 2024-04-12T03:52:08.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8Ghuk8ws0gq7lXtyobRkBov+YRA=', 'Content-Length': '0'} 2024-04-12T03:52:08.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044353fcc770b632c-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.534 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:08.535 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:08.535 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:52:08.535 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:52:08.536 DEBUG:boto:Method: GET 2024-04-12T03:52:08.536 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:52:08.536 DEBUG:boto:Data: 2024-04-12T03:52:08.536 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.536 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.536 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.537 DEBUG:boto:Params: {} 2024-04-12T03:52:08.537 DEBUG:boto:Token: None 2024-04-12T03:52:08.537 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/a?versionId=null 2024-04-12T03:52:08.538 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8Ghuk8ws0gq7lXtyobRkBov+YRA= 2024-04-12T03:52:08.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8Ghuk8ws0gq7lXtyobRkBov+YRA=', 'Content-Length': '0'} 2024-04-12T03:52:08.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003384be0b519f89f7-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.540 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.541 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.541 DEBUG:boto:path=/dkyges-57/?prefix=a 2024-04-12T03:52:08.541 DEBUG:boto:auth_path=/dkyges-57/?prefix=a 2024-04-12T03:52:08.541 DEBUG:boto:Method: GET 2024-04-12T03:52:08.541 DEBUG:boto:Path: /dkyges-57/?prefix=a 2024-04-12T03:52:08.541 DEBUG:boto:Data: 2024-04-12T03:52:08.541 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.542 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.542 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.542 DEBUG:boto:Params: {} 2024-04-12T03:52:08.542 DEBUG:boto:Token: None 2024-04-12T03:52:08.542 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.542 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T03:52:08.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T03:52:08.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca9a6bad15c00750-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.546 DEBUG:boto:b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.546 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.546 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.546 DEBUG:boto:path=/dkyges-57/?prefix=a 2024-04-12T03:52:08.546 DEBUG:boto:auth_path=/dkyges-57/?prefix=a 2024-04-12T03:52:08.547 DEBUG:boto:Method: GET 2024-04-12T03:52:08.547 DEBUG:boto:Path: /dkyges-57/?prefix=a 2024-04-12T03:52:08.547 DEBUG:boto:Data: 2024-04-12T03:52:08.547 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.547 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.547 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.547 DEBUG:boto:Params: {} 2024-04-12T03:52:08.547 DEBUG:boto:Token: None 2024-04-12T03:52:08.547 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.547 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T03:52:08.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T03:52:08.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a56ddc6909b18bb-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.550 DEBUG:boto:b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.550 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:08.550 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:08.550 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.550 DEBUG:boto:Path: /dkyges-57/a 2024-04-12T03:52:08.550 DEBUG:boto:Data: 2024-04-12T03:52:08.550 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.551 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.551 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.551 DEBUG:boto:Params: {} 2024-04-12T03:52:08.551 DEBUG:boto:Token: None 2024-04-12T03:52:08.551 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/a 2024-04-12T03:52:08.551 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:99v6ThYq2z7oU/3h2XwoHG67rks= 2024-04-12T03:52:08.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:99v6ThYq2z7oU/3h2XwoHG67rks=', 'Content-Length': '0'} 2024-04-12T03:52:08.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006931108083a23ac9-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.553 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:08.553 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:08.553 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.553 DEBUG:boto:Path: /dkyges-57/a 2024-04-12T03:52:08.553 DEBUG:boto:Data: 2024-04-12T03:52:08.553 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.554 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.554 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.554 DEBUG:boto:Params: {} 2024-04-12T03:52:08.554 DEBUG:boto:Token: None 2024-04-12T03:52:08.554 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/a 2024-04-12T03:52:08.554 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:99v6ThYq2z7oU/3h2XwoHG67rks= 2024-04-12T03:52:08.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:99v6ThYq2z7oU/3h2XwoHG67rks=', 'Content-Length': '0'} 2024-04-12T03:52:08.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9a7892825a0cef4-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:52:08.556 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:08.556 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:08.556 DEBUG:boto:Method: GET 2024-04-12T03:52:08.557 DEBUG:boto:Path: /dkyges-57/a 2024-04-12T03:52:08.557 DEBUG:boto:Data: 2024-04-12T03:52:08.557 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.557 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.557 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.557 DEBUG:boto:Params: {} 2024-04-12T03:52:08.557 DEBUG:boto:Token: None 2024-04-12T03:52:08.557 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/a 2024-04-12T03:52:08.557 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WHiCwMAZkwJ+CERBe8yXWZLC0Ww= 2024-04-12T03:52:08.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WHiCwMAZkwJ+CERBe8yXWZLC0Ww=', 'Content-Length': '0'} 2024-04-12T03:52:08.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000833158ddacbf5aa3-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.559 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:08.560 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:08.560 DEBUG:boto:Method: GET 2024-04-12T03:52:08.560 DEBUG:boto:Path: /dkyges-57/a 2024-04-12T03:52:08.560 DEBUG:boto:Data: 2024-04-12T03:52:08.560 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.560 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.560 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.560 DEBUG:boto:Params: {} 2024-04-12T03:52:08.560 DEBUG:boto:Token: None 2024-04-12T03:52:08.560 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/a 2024-04-12T03:52:08.561 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WHiCwMAZkwJ+CERBe8yXWZLC0Ww= 2024-04-12T03:52:08.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WHiCwMAZkwJ+CERBe8yXWZLC0Ww=', 'Content-Length': '0'} 2024-04-12T03:52:08.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c15a6a0d286aa593-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.563 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.563 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.563 DEBUG:boto:path=/dkyges-57/?prefix=a 2024-04-12T03:52:08.563 DEBUG:boto:auth_path=/dkyges-57/?prefix=a 2024-04-12T03:52:08.563 DEBUG:boto:Method: GET 2024-04-12T03:52:08.563 DEBUG:boto:Path: /dkyges-57/?prefix=a 2024-04-12T03:52:08.563 DEBUG:boto:Data: 2024-04-12T03:52:08.563 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.563 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.563 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.563 DEBUG:boto:Params: {} 2024-04-12T03:52:08.563 DEBUG:boto:Token: None 2024-04-12T03:52:08.564 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.564 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T03:52:08.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T03:52:08.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000265f84edf8a364c-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.566 DEBUG:boto:b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.567 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.567 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.567 DEBUG:boto:path=/dkyges-57/?prefix=a 2024-04-12T03:52:08.567 DEBUG:boto:auth_path=/dkyges-57/?prefix=a 2024-04-12T03:52:08.567 DEBUG:boto:Method: GET 2024-04-12T03:52:08.567 DEBUG:boto:Path: /dkyges-57/?prefix=a 2024-04-12T03:52:08.567 DEBUG:boto:Data: 2024-04-12T03:52:08.567 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.567 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.568 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.568 DEBUG:boto:Params: {} 2024-04-12T03:52:08.568 DEBUG:boto:Token: None 2024-04-12T03:52:08.568 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.568 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T03:52:08.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T03:52:08.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccb8e1cf49019550-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.614 DEBUG:boto:b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:52:08.615 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:08.615 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:08.615 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:52:08.615 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:52:08.615 DEBUG:boto:Method: GET 2024-04-12T03:52:08.616 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:52:08.616 DEBUG:boto:Data: 2024-04-12T03:52:08.616 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.616 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.616 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.616 DEBUG:boto:Params: {} 2024-04-12T03:52:08.616 DEBUG:boto:Token: None 2024-04-12T03:52:08.616 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/b?versionId=null 2024-04-12T03:52:08.617 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+AE+mjoWDAKGQwxjN5WWXcxRZVY= 2024-04-12T03:52:08.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+AE+mjoWDAKGQwxjN5WWXcxRZVY=', 'Content-Length': '0'} 2024-04-12T03:52:08.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e419abd9127beac-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.619 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:08.620 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:08.620 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:52:08.620 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:52:08.620 DEBUG:boto:Method: GET 2024-04-12T03:52:08.620 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:52:08.620 DEBUG:boto:Data: 2024-04-12T03:52:08.620 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.620 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.620 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.620 DEBUG:boto:Params: {} 2024-04-12T03:52:08.621 DEBUG:boto:Token: None 2024-04-12T03:52:08.621 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/b?versionId=null 2024-04-12T03:52:08.621 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+AE+mjoWDAKGQwxjN5WWXcxRZVY= 2024-04-12T03:52:08.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+AE+mjoWDAKGQwxjN5WWXcxRZVY=', 'Content-Length': '0'} 2024-04-12T03:52:08.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049bc91ea4f2e3dde-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.623 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:08.623 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:08.623 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:52:08.623 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:52:08.623 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.623 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:52:08.624 DEBUG:boto:Data: 2024-04-12T03:52:08.624 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.624 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.624 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.624 DEBUG:boto:Params: {} 2024-04-12T03:52:08.624 DEBUG:boto:Token: None 2024-04-12T03:52:08.624 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/b?versionId=null 2024-04-12T03:52:08.625 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7IHUpHLBpsZOev5a4f7xPAnI6os= 2024-04-12T03:52:08.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7IHUpHLBpsZOev5a4f7xPAnI6os=', 'Content-Length': '0'} 2024-04-12T03:52:08.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035f2266281a43a9a-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.626 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:08.627 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:08.627 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:52:08.627 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:52:08.627 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.627 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:52:08.627 DEBUG:boto:Data: 2024-04-12T03:52:08.627 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.628 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.628 DEBUG:boto:Params: {} 2024-04-12T03:52:08.628 DEBUG:boto:Token: None 2024-04-12T03:52:08.628 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/b?versionId=null 2024-04-12T03:52:08.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7IHUpHLBpsZOev5a4f7xPAnI6os= 2024-04-12T03:52:08.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7IHUpHLBpsZOev5a4f7xPAnI6os=', 'Content-Length': '0'} 2024-04-12T03:52:08.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce7fe3e02296e30d-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:52:08.630 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:08.630 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:08.630 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:52:08.631 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:52:08.631 DEBUG:boto:Method: GET 2024-04-12T03:52:08.631 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:52:08.631 DEBUG:boto:Data: 2024-04-12T03:52:08.631 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.631 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.631 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.631 DEBUG:boto:Params: {} 2024-04-12T03:52:08.632 DEBUG:boto:Token: None 2024-04-12T03:52:08.632 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/b?versionId=null 2024-04-12T03:52:08.632 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+AE+mjoWDAKGQwxjN5WWXcxRZVY= 2024-04-12T03:52:08.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+AE+mjoWDAKGQwxjN5WWXcxRZVY=', 'Content-Length': '0'} 2024-04-12T03:52:08.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000281902a8e397ec47-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.674 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:08.675 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:08.675 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:52:08.675 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:52:08.675 DEBUG:boto:Method: GET 2024-04-12T03:52:08.675 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:52:08.675 DEBUG:boto:Data: 2024-04-12T03:52:08.675 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.675 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.676 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.676 DEBUG:boto:Params: {} 2024-04-12T03:52:08.676 DEBUG:boto:Token: None 2024-04-12T03:52:08.676 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/b?versionId=null 2024-04-12T03:52:08.676 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+AE+mjoWDAKGQwxjN5WWXcxRZVY= 2024-04-12T03:52:08.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+AE+mjoWDAKGQwxjN5WWXcxRZVY=', 'Content-Length': '0'} 2024-04-12T03:52:08.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000005d90497be1521f-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.678 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.679 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.679 DEBUG:boto:path=/dkyges-57/?prefix=b 2024-04-12T03:52:08.679 DEBUG:boto:auth_path=/dkyges-57/?prefix=b 2024-04-12T03:52:08.679 DEBUG:boto:Method: GET 2024-04-12T03:52:08.679 DEBUG:boto:Path: /dkyges-57/?prefix=b 2024-04-12T03:52:08.679 DEBUG:boto:Data: 2024-04-12T03:52:08.679 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.679 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.679 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.679 DEBUG:boto:Params: {} 2024-04-12T03:52:08.680 DEBUG:boto:Token: None 2024-04-12T03:52:08.680 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.680 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T03:52:08.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T03:52:08.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a92b7fec290b78ee-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.683 DEBUG:boto:b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.683 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.683 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.683 DEBUG:boto:path=/dkyges-57/?prefix=b 2024-04-12T03:52:08.684 DEBUG:boto:auth_path=/dkyges-57/?prefix=b 2024-04-12T03:52:08.684 DEBUG:boto:Method: GET 2024-04-12T03:52:08.684 DEBUG:boto:Path: /dkyges-57/?prefix=b 2024-04-12T03:52:08.684 DEBUG:boto:Data: 2024-04-12T03:52:08.684 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.684 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.684 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.684 DEBUG:boto:Params: {} 2024-04-12T03:52:08.684 DEBUG:boto:Token: None 2024-04-12T03:52:08.684 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.685 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T03:52:08.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T03:52:08.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbd7c38755703d3f-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.734 DEBUG:boto:b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.735 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:08.735 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:08.735 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.735 DEBUG:boto:Path: /dkyges-57/b 2024-04-12T03:52:08.735 DEBUG:boto:Data: 2024-04-12T03:52:08.735 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.735 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.735 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.735 DEBUG:boto:Params: {} 2024-04-12T03:52:08.736 DEBUG:boto:Token: None 2024-04-12T03:52:08.736 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/b 2024-04-12T03:52:08.736 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H1nDxpbtrHt7+X3fDHfoEVMhjO0= 2024-04-12T03:52:08.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H1nDxpbtrHt7+X3fDHfoEVMhjO0=', 'Content-Length': '0'} 2024-04-12T03:52:08.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022f9020408034853-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.738 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:08.738 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:08.739 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.739 DEBUG:boto:Path: /dkyges-57/b 2024-04-12T03:52:08.739 DEBUG:boto:Data: 2024-04-12T03:52:08.739 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.739 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.739 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.739 DEBUG:boto:Params: {} 2024-04-12T03:52:08.739 DEBUG:boto:Token: None 2024-04-12T03:52:08.739 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/b 2024-04-12T03:52:08.739 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H1nDxpbtrHt7+X3fDHfoEVMhjO0= 2024-04-12T03:52:08.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H1nDxpbtrHt7+X3fDHfoEVMhjO0=', 'Content-Length': '0'} 2024-04-12T03:52:08.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009877591b251b3c62-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:52:08.741 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:08.742 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:08.742 DEBUG:boto:Method: GET 2024-04-12T03:52:08.742 DEBUG:boto:Path: /dkyges-57/b 2024-04-12T03:52:08.742 DEBUG:boto:Data: 2024-04-12T03:52:08.742 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.742 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.742 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.742 DEBUG:boto:Params: {} 2024-04-12T03:52:08.742 DEBUG:boto:Token: None 2024-04-12T03:52:08.742 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/b 2024-04-12T03:52:08.742 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wb+rG0+bvBQMSnWMPg8Yb7W1R4E= 2024-04-12T03:52:08.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wb+rG0+bvBQMSnWMPg8Yb7W1R4E=', 'Content-Length': '0'} 2024-04-12T03:52:08.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ba327a89a756043-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.745 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:08.745 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:08.745 DEBUG:boto:Method: GET 2024-04-12T03:52:08.745 DEBUG:boto:Path: /dkyges-57/b 2024-04-12T03:52:08.745 DEBUG:boto:Data: 2024-04-12T03:52:08.745 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.745 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.745 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.745 DEBUG:boto:Params: {} 2024-04-12T03:52:08.745 DEBUG:boto:Token: None 2024-04-12T03:52:08.745 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/b 2024-04-12T03:52:08.746 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wb+rG0+bvBQMSnWMPg8Yb7W1R4E= 2024-04-12T03:52:08.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wb+rG0+bvBQMSnWMPg8Yb7W1R4E=', 'Content-Length': '0'} 2024-04-12T03:52:08.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005ef5cf0b3c036c1-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.748 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.748 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.748 DEBUG:boto:path=/dkyges-57/?prefix=b 2024-04-12T03:52:08.748 DEBUG:boto:auth_path=/dkyges-57/?prefix=b 2024-04-12T03:52:08.748 DEBUG:boto:Method: GET 2024-04-12T03:52:08.748 DEBUG:boto:Path: /dkyges-57/?prefix=b 2024-04-12T03:52:08.748 DEBUG:boto:Data: 2024-04-12T03:52:08.748 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.748 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.748 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.749 DEBUG:boto:Params: {} 2024-04-12T03:52:08.749 DEBUG:boto:Token: None 2024-04-12T03:52:08.749 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.749 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T03:52:08.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T03:52:08.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae22acc41470fc76-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.751 DEBUG:boto:b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.752 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.752 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.752 DEBUG:boto:path=/dkyges-57/?prefix=b 2024-04-12T03:52:08.752 DEBUG:boto:auth_path=/dkyges-57/?prefix=b 2024-04-12T03:52:08.752 DEBUG:boto:Method: GET 2024-04-12T03:52:08.752 DEBUG:boto:Path: /dkyges-57/?prefix=b 2024-04-12T03:52:08.752 DEBUG:boto:Data: 2024-04-12T03:52:08.752 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.753 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.753 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.753 DEBUG:boto:Params: {} 2024-04-12T03:52:08.753 DEBUG:boto:Token: None 2024-04-12T03:52:08.753 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.753 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T03:52:08.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T03:52:08.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063ce2c125a27a1d1-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.755 DEBUG:boto:b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:52:08.756 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:08.756 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:08.756 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:52:08.756 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:52:08.756 DEBUG:boto:Method: GET 2024-04-12T03:52:08.756 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:52:08.756 DEBUG:boto:Data: 2024-04-12T03:52:08.756 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.756 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.756 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.756 DEBUG:boto:Params: {} 2024-04-12T03:52:08.756 DEBUG:boto:Token: None 2024-04-12T03:52:08.757 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/c?versionId=null 2024-04-12T03:52:08.757 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BzzoauNgcb8nZCKDjviLRVrZYZ8= 2024-04-12T03:52:08.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BzzoauNgcb8nZCKDjviLRVrZYZ8=', 'Content-Length': '0'} 2024-04-12T03:52:08.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055134680f4a7c226-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.759 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:08.759 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:08.759 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:52:08.759 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:52:08.759 DEBUG:boto:Method: GET 2024-04-12T03:52:08.759 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:52:08.759 DEBUG:boto:Data: 2024-04-12T03:52:08.759 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.759 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.760 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.760 DEBUG:boto:Params: {} 2024-04-12T03:52:08.760 DEBUG:boto:Token: None 2024-04-12T03:52:08.760 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/c?versionId=null 2024-04-12T03:52:08.760 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BzzoauNgcb8nZCKDjviLRVrZYZ8= 2024-04-12T03:52:08.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BzzoauNgcb8nZCKDjviLRVrZYZ8=', 'Content-Length': '0'} 2024-04-12T03:52:08.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ef7509e21bedb24-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.802 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:08.802 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:08.802 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:52:08.803 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:52:08.803 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.803 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:52:08.803 DEBUG:boto:Data: 2024-04-12T03:52:08.803 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.803 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.803 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.803 DEBUG:boto:Params: {} 2024-04-12T03:52:08.803 DEBUG:boto:Token: None 2024-04-12T03:52:08.803 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/c?versionId=null 2024-04-12T03:52:08.804 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:thbk7vN6RGbBd4cphI8UmkI3Oc0= 2024-04-12T03:52:08.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:thbk7vN6RGbBd4cphI8UmkI3Oc0=', 'Content-Length': '0'} 2024-04-12T03:52:08.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c1e4ae42114f20b-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.806 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:08.806 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:08.806 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:52:08.806 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:52:08.806 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.806 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:52:08.806 DEBUG:boto:Data: 2024-04-12T03:52:08.806 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.806 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.806 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.807 DEBUG:boto:Params: {} 2024-04-12T03:52:08.807 DEBUG:boto:Token: None 2024-04-12T03:52:08.807 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/c?versionId=null 2024-04-12T03:52:08.807 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:thbk7vN6RGbBd4cphI8UmkI3Oc0= 2024-04-12T03:52:08.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:thbk7vN6RGbBd4cphI8UmkI3Oc0=', 'Content-Length': '0'} 2024-04-12T03:52:08.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7280cb750180a8b-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:52:08.809 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:08.809 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:08.809 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:52:08.809 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:52:08.809 DEBUG:boto:Method: GET 2024-04-12T03:52:08.810 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:52:08.810 DEBUG:boto:Data: 2024-04-12T03:52:08.810 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.810 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.810 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.810 DEBUG:boto:Params: {} 2024-04-12T03:52:08.810 DEBUG:boto:Token: None 2024-04-12T03:52:08.810 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/c?versionId=null 2024-04-12T03:52:08.810 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BzzoauNgcb8nZCKDjviLRVrZYZ8= 2024-04-12T03:52:08.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BzzoauNgcb8nZCKDjviLRVrZYZ8=', 'Content-Length': '0'} 2024-04-12T03:52:08.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0b024b1d5b4c5f1-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.812 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:08.812 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:08.812 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:52:08.813 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:52:08.813 DEBUG:boto:Method: GET 2024-04-12T03:52:08.813 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:52:08.813 DEBUG:boto:Data: 2024-04-12T03:52:08.813 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.813 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.813 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.813 DEBUG:boto:Params: {} 2024-04-12T03:52:08.813 DEBUG:boto:Token: None 2024-04-12T03:52:08.813 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/c?versionId=null 2024-04-12T03:52:08.813 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BzzoauNgcb8nZCKDjviLRVrZYZ8= 2024-04-12T03:52:08.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BzzoauNgcb8nZCKDjviLRVrZYZ8=', 'Content-Length': '0'} 2024-04-12T03:52:08.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000180b9a68e35eb7aa-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.816 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.816 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.816 DEBUG:boto:path=/dkyges-57/?prefix=c 2024-04-12T03:52:08.816 DEBUG:boto:auth_path=/dkyges-57/?prefix=c 2024-04-12T03:52:08.816 DEBUG:boto:Method: GET 2024-04-12T03:52:08.816 DEBUG:boto:Path: /dkyges-57/?prefix=c 2024-04-12T03:52:08.816 DEBUG:boto:Data: 2024-04-12T03:52:08.816 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.817 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.817 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.817 DEBUG:boto:Params: {} 2024-04-12T03:52:08.817 DEBUG:boto:Token: None 2024-04-12T03:52:08.817 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.817 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T03:52:08.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T03:52:08.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd97ee6a0cf5681-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.820 DEBUG:boto:b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.820 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.820 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.820 DEBUG:boto:path=/dkyges-57/?prefix=c 2024-04-12T03:52:08.821 DEBUG:boto:auth_path=/dkyges-57/?prefix=c 2024-04-12T03:52:08.821 DEBUG:boto:Method: GET 2024-04-12T03:52:08.821 DEBUG:boto:Path: /dkyges-57/?prefix=c 2024-04-12T03:52:08.821 DEBUG:boto:Data: 2024-04-12T03:52:08.821 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.821 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.821 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.821 DEBUG:boto:Params: {} 2024-04-12T03:52:08.821 DEBUG:boto:Token: None 2024-04-12T03:52:08.821 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.822 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T03:52:08.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T03:52:08.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f1882cec80680da-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.824 DEBUG:boto:b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.824 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:08.824 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:08.824 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.824 DEBUG:boto:Path: /dkyges-57/c 2024-04-12T03:52:08.824 DEBUG:boto:Data: 2024-04-12T03:52:08.824 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.824 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.825 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.825 DEBUG:boto:Params: {} 2024-04-12T03:52:08.825 DEBUG:boto:Token: None 2024-04-12T03:52:08.825 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/c 2024-04-12T03:52:08.825 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H1HrQB6nCHDY57qlHoXHbJCVImc= 2024-04-12T03:52:08.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H1HrQB6nCHDY57qlHoXHbJCVImc=', 'Content-Length': '0'} 2024-04-12T03:52:08.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbcbc2888d74e2c6-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.830 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:08.830 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:08.830 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.830 DEBUG:boto:Path: /dkyges-57/c 2024-04-12T03:52:08.831 DEBUG:boto:Data: 2024-04-12T03:52:08.831 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.831 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.831 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.831 DEBUG:boto:Params: {} 2024-04-12T03:52:08.831 DEBUG:boto:Token: None 2024-04-12T03:52:08.831 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/c 2024-04-12T03:52:08.831 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H1HrQB6nCHDY57qlHoXHbJCVImc= 2024-04-12T03:52:08.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H1HrQB6nCHDY57qlHoXHbJCVImc=', 'Content-Length': '0'} 2024-04-12T03:52:08.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e8c9cb6e8d8ab89-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:52:08.834 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:08.834 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:08.834 DEBUG:boto:Method: GET 2024-04-12T03:52:08.834 DEBUG:boto:Path: /dkyges-57/c 2024-04-12T03:52:08.834 DEBUG:boto:Data: 2024-04-12T03:52:08.834 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.834 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.834 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.834 DEBUG:boto:Params: {} 2024-04-12T03:52:08.835 DEBUG:boto:Token: None 2024-04-12T03:52:08.835 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/c 2024-04-12T03:52:08.835 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HWtB/xNvrzzQ5BqpmjVtjhOvWSc= 2024-04-12T03:52:08.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HWtB/xNvrzzQ5BqpmjVtjhOvWSc=', 'Content-Length': '0'} 2024-04-12T03:52:08.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004abf7e0d18dc6a06-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.878 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:08.879 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:08.879 DEBUG:boto:Method: GET 2024-04-12T03:52:08.879 DEBUG:boto:Path: /dkyges-57/c 2024-04-12T03:52:08.879 DEBUG:boto:Data: 2024-04-12T03:52:08.879 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.879 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.879 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.879 DEBUG:boto:Params: {} 2024-04-12T03:52:08.879 DEBUG:boto:Token: None 2024-04-12T03:52:08.880 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/c 2024-04-12T03:52:08.880 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HWtB/xNvrzzQ5BqpmjVtjhOvWSc= 2024-04-12T03:52:08.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HWtB/xNvrzzQ5BqpmjVtjhOvWSc=', 'Content-Length': '0'} 2024-04-12T03:52:08.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c5867710e7d6a72-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.882 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.882 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.882 DEBUG:boto:path=/dkyges-57/?prefix=c 2024-04-12T03:52:08.883 DEBUG:boto:auth_path=/dkyges-57/?prefix=c 2024-04-12T03:52:08.883 DEBUG:boto:Method: GET 2024-04-12T03:52:08.883 DEBUG:boto:Path: /dkyges-57/?prefix=c 2024-04-12T03:52:08.883 DEBUG:boto:Data: 2024-04-12T03:52:08.883 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.883 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.883 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.883 DEBUG:boto:Params: {} 2024-04-12T03:52:08.883 DEBUG:boto:Token: None 2024-04-12T03:52:08.883 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.883 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T03:52:08.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T03:52:08.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000523b9b8745d2a69d-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.886 DEBUG:boto:b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.886 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.887 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.887 DEBUG:boto:path=/dkyges-57/?prefix=c 2024-04-12T03:52:08.887 DEBUG:boto:auth_path=/dkyges-57/?prefix=c 2024-04-12T03:52:08.887 DEBUG:boto:Method: GET 2024-04-12T03:52:08.887 DEBUG:boto:Path: /dkyges-57/?prefix=c 2024-04-12T03:52:08.887 DEBUG:boto:Data: 2024-04-12T03:52:08.887 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.887 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.887 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.887 DEBUG:boto:Params: {} 2024-04-12T03:52:08.887 DEBUG:boto:Token: None 2024-04-12T03:52:08.888 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.888 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T03:52:08.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T03:52:08.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8e201b3783612ce-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.890 DEBUG:boto:b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:52:08.891 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:08.891 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:08.891 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:52:08.891 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:52:08.891 DEBUG:boto:Method: GET 2024-04-12T03:52:08.891 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:52:08.891 DEBUG:boto:Data: 2024-04-12T03:52:08.891 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.891 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.891 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.891 DEBUG:boto:Params: {} 2024-04-12T03:52:08.892 DEBUG:boto:Token: None 2024-04-12T03:52:08.892 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/d?versionId=null 2024-04-12T03:52:08.892 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SaSRyJLItx11IubJLbqp44WAzyA= 2024-04-12T03:52:08.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SaSRyJLItx11IubJLbqp44WAzyA=', 'Content-Length': '0'} 2024-04-12T03:52:08.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d02e1e54c6dda999-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.894 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:08.894 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:08.894 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:52:08.894 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:52:08.895 DEBUG:boto:Method: GET 2024-04-12T03:52:08.895 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:52:08.895 DEBUG:boto:Data: 2024-04-12T03:52:08.895 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.895 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.895 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.895 DEBUG:boto:Params: {} 2024-04-12T03:52:08.895 DEBUG:boto:Token: None 2024-04-12T03:52:08.895 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/d?versionId=null 2024-04-12T03:52:08.895 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SaSRyJLItx11IubJLbqp44WAzyA= 2024-04-12T03:52:08.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SaSRyJLItx11IubJLbqp44WAzyA=', 'Content-Length': '0'} 2024-04-12T03:52:08.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f1736bb9db9d0f3-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.898 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:08.898 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:08.898 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:52:08.898 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:52:08.898 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.899 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:52:08.899 DEBUG:boto:Data: 2024-04-12T03:52:08.899 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.899 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.899 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.899 DEBUG:boto:Params: {} 2024-04-12T03:52:08.899 DEBUG:boto:Token: None 2024-04-12T03:52:08.899 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/d?versionId=null 2024-04-12T03:52:08.899 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1IpYRKFvBinT9/G2wOCxfTxBpzA= 2024-04-12T03:52:08.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1IpYRKFvBinT9/G2wOCxfTxBpzA=', 'Content-Length': '0'} 2024-04-12T03:52:08.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061d969b489f19849-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.901 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:08.902 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:08.902 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:52:08.902 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:52:08.902 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.902 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:52:08.902 DEBUG:boto:Data: 2024-04-12T03:52:08.902 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.902 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.902 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.902 DEBUG:boto:Params: {} 2024-04-12T03:52:08.903 DEBUG:boto:Token: None 2024-04-12T03:52:08.903 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/d?versionId=null 2024-04-12T03:52:08.903 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1IpYRKFvBinT9/G2wOCxfTxBpzA= 2024-04-12T03:52:08.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1IpYRKFvBinT9/G2wOCxfTxBpzA=', 'Content-Length': '0'} 2024-04-12T03:52:08.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a250e2b64f7adb9-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:52:08.905 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:08.905 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:08.905 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:52:08.905 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:52:08.905 DEBUG:boto:Method: GET 2024-04-12T03:52:08.905 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:52:08.905 DEBUG:boto:Data: 2024-04-12T03:52:08.905 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.906 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.906 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.906 DEBUG:boto:Params: {} 2024-04-12T03:52:08.906 DEBUG:boto:Token: None 2024-04-12T03:52:08.906 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/d?versionId=null 2024-04-12T03:52:08.906 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SaSRyJLItx11IubJLbqp44WAzyA= 2024-04-12T03:52:08.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SaSRyJLItx11IubJLbqp44WAzyA=', 'Content-Length': '0'} 2024-04-12T03:52:08.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c3673b5e4f5eff9-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.908 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:08.908 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:08.908 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:52:08.908 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:52:08.908 DEBUG:boto:Method: GET 2024-04-12T03:52:08.908 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:52:08.909 DEBUG:boto:Data: 2024-04-12T03:52:08.909 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.909 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.909 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.909 DEBUG:boto:Params: {} 2024-04-12T03:52:08.909 DEBUG:boto:Token: None 2024-04-12T03:52:08.909 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/d?versionId=null 2024-04-12T03:52:08.909 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SaSRyJLItx11IubJLbqp44WAzyA= 2024-04-12T03:52:08.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SaSRyJLItx11IubJLbqp44WAzyA=', 'Content-Length': '0'} 2024-04-12T03:52:08.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004cdc843b0f6b496-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.912 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.912 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.912 DEBUG:boto:path=/dkyges-57/?prefix=d 2024-04-12T03:52:08.912 DEBUG:boto:auth_path=/dkyges-57/?prefix=d 2024-04-12T03:52:08.912 DEBUG:boto:Method: GET 2024-04-12T03:52:08.912 DEBUG:boto:Path: /dkyges-57/?prefix=d 2024-04-12T03:52:08.912 DEBUG:boto:Data: 2024-04-12T03:52:08.912 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.913 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.913 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.913 DEBUG:boto:Params: {} 2024-04-12T03:52:08.913 DEBUG:boto:Token: None 2024-04-12T03:52:08.913 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.913 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T03:52:08.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T03:52:08.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7fce050f5e60a82-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.916 DEBUG:boto:b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.916 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.916 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.916 DEBUG:boto:path=/dkyges-57/?prefix=d 2024-04-12T03:52:08.916 DEBUG:boto:auth_path=/dkyges-57/?prefix=d 2024-04-12T03:52:08.917 DEBUG:boto:Method: GET 2024-04-12T03:52:08.917 DEBUG:boto:Path: /dkyges-57/?prefix=d 2024-04-12T03:52:08.917 DEBUG:boto:Data: 2024-04-12T03:52:08.917 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.917 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.917 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.917 DEBUG:boto:Params: {} 2024-04-12T03:52:08.917 DEBUG:boto:Token: None 2024-04-12T03:52:08.917 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.917 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T03:52:08.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T03:52:08.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a0edeb884461d64-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.920 DEBUG:boto:b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.920 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:08.920 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:08.920 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.920 DEBUG:boto:Path: /dkyges-57/d 2024-04-12T03:52:08.920 DEBUG:boto:Data: 2024-04-12T03:52:08.920 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.921 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.921 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.921 DEBUG:boto:Params: {} 2024-04-12T03:52:08.921 DEBUG:boto:Token: None 2024-04-12T03:52:08.921 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/d 2024-04-12T03:52:08.921 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ibIj1tvrWZYFFILmUa70Qj4kvrs= 2024-04-12T03:52:08.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ibIj1tvrWZYFFILmUa70Qj4kvrs=', 'Content-Length': '0'} 2024-04-12T03:52:08.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029eae1fbcf89b37d-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.923 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:08.923 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:08.924 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.924 DEBUG:boto:Path: /dkyges-57/d 2024-04-12T03:52:08.924 DEBUG:boto:Data: 2024-04-12T03:52:08.924 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.924 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.924 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.924 DEBUG:boto:Params: {} 2024-04-12T03:52:08.924 DEBUG:boto:Token: None 2024-04-12T03:52:08.924 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/d 2024-04-12T03:52:08.925 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ibIj1tvrWZYFFILmUa70Qj4kvrs= 2024-04-12T03:52:08.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ibIj1tvrWZYFFILmUa70Qj4kvrs=', 'Content-Length': '0'} 2024-04-12T03:52:08.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072fcf3c72ddf0ab6-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:52:08.927 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:08.927 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:08.927 DEBUG:boto:Method: GET 2024-04-12T03:52:08.927 DEBUG:boto:Path: /dkyges-57/d 2024-04-12T03:52:08.927 DEBUG:boto:Data: 2024-04-12T03:52:08.927 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.928 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.928 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.928 DEBUG:boto:Params: {} 2024-04-12T03:52:08.928 DEBUG:boto:Token: None 2024-04-12T03:52:08.928 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/d 2024-04-12T03:52:08.928 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:16DzJ6BEZR4Tp1UwNXII03TRNa0= 2024-04-12T03:52:08.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:16DzJ6BEZR4Tp1UwNXII03TRNa0=', 'Content-Length': '0'} 2024-04-12T03:52:08.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb0872f5e347ea25-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.970 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:08.970 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:08.970 DEBUG:boto:Method: GET 2024-04-12T03:52:08.971 DEBUG:boto:Path: /dkyges-57/d 2024-04-12T03:52:08.971 DEBUG:boto:Data: 2024-04-12T03:52:08.971 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.971 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.971 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.971 DEBUG:boto:Params: {} 2024-04-12T03:52:08.971 DEBUG:boto:Token: None 2024-04-12T03:52:08.971 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/d 2024-04-12T03:52:08.972 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:16DzJ6BEZR4Tp1UwNXII03TRNa0= 2024-04-12T03:52:08.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:16DzJ6BEZR4Tp1UwNXII03TRNa0=', 'Content-Length': '0'} 2024-04-12T03:52:08.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000460a2c1df385691c-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.974 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.974 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.974 DEBUG:boto:path=/dkyges-57/?prefix=d 2024-04-12T03:52:08.974 DEBUG:boto:auth_path=/dkyges-57/?prefix=d 2024-04-12T03:52:08.974 DEBUG:boto:Method: GET 2024-04-12T03:52:08.974 DEBUG:boto:Path: /dkyges-57/?prefix=d 2024-04-12T03:52:08.974 DEBUG:boto:Data: 2024-04-12T03:52:08.974 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.975 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.975 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.975 DEBUG:boto:Params: {} 2024-04-12T03:52:08.975 DEBUG:boto:Token: None 2024-04-12T03:52:08.975 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.975 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T03:52:08.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T03:52:08.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052a571110cafbd48-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.977 DEBUG:boto:b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.978 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:08.978 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:08.978 DEBUG:boto:path=/dkyges-57/?prefix=d 2024-04-12T03:52:08.978 DEBUG:boto:auth_path=/dkyges-57/?prefix=d 2024-04-12T03:52:08.978 DEBUG:boto:Method: GET 2024-04-12T03:52:08.978 DEBUG:boto:Path: /dkyges-57/?prefix=d 2024-04-12T03:52:08.978 DEBUG:boto:Data: 2024-04-12T03:52:08.978 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.979 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.979 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.979 DEBUG:boto:Params: {} 2024-04-12T03:52:08.979 DEBUG:boto:Token: None 2024-04-12T03:52:08.979 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/ 2024-04-12T03:52:08.979 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T03:52:08.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T03:52:08.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065981da435252ddb-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.982 DEBUG:boto:b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:08.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:52:08.982 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:08.982 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:08.982 DEBUG:boto:path=/dkyges-57/e?versionId=null 2024-04-12T03:52:08.982 DEBUG:boto:auth_path=/dkyges-57/e?versionId=null 2024-04-12T03:52:08.983 DEBUG:boto:Method: GET 2024-04-12T03:52:08.983 DEBUG:boto:Path: /dkyges-57/e?versionId=null 2024-04-12T03:52:08.983 DEBUG:boto:Data: 2024-04-12T03:52:08.983 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.983 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.983 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.983 DEBUG:boto:Params: {} 2024-04-12T03:52:08.983 DEBUG:boto:Token: None 2024-04-12T03:52:08.983 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/e?versionId=null 2024-04-12T03:52:08.983 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q89BiJwl2t9qUozc6JIu60aKJnU= 2024-04-12T03:52:08.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q89BiJwl2t9qUozc6JIu60aKJnU=', 'Content-Length': '0'} 2024-04-12T03:52:08.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089d335046b30f9b7-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.986 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:08.986 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:08.986 DEBUG:boto:path=/dkyges-57/e?versionId=null 2024-04-12T03:52:08.986 DEBUG:boto:auth_path=/dkyges-57/e?versionId=null 2024-04-12T03:52:08.986 DEBUG:boto:Method: GET 2024-04-12T03:52:08.986 DEBUG:boto:Path: /dkyges-57/e?versionId=null 2024-04-12T03:52:08.986 DEBUG:boto:Data: 2024-04-12T03:52:08.986 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.986 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.986 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.986 DEBUG:boto:Params: {} 2024-04-12T03:52:08.987 DEBUG:boto:Token: None 2024-04-12T03:52:08.987 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/e?versionId=null 2024-04-12T03:52:08.987 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q89BiJwl2t9qUozc6JIu60aKJnU= 2024-04-12T03:52:08.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q89BiJwl2t9qUozc6JIu60aKJnU=', 'Content-Length': '0'} 2024-04-12T03:52:08.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063df1acc0935d1a7-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.989 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:08.989 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:08.989 DEBUG:boto:path=/dkyges-57/e?versionId=null 2024-04-12T03:52:08.989 DEBUG:boto:auth_path=/dkyges-57/e?versionId=null 2024-04-12T03:52:08.989 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.989 DEBUG:boto:Path: /dkyges-57/e?versionId=null 2024-04-12T03:52:08.989 DEBUG:boto:Data: 2024-04-12T03:52:08.990 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.990 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.990 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.990 DEBUG:boto:Params: {} 2024-04-12T03:52:08.990 DEBUG:boto:Token: None 2024-04-12T03:52:08.990 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/e?versionId=null 2024-04-12T03:52:08.990 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pbGDEzqbsNxBbVjNC0Q9TyuZwMI= 2024-04-12T03:52:08.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pbGDEzqbsNxBbVjNC0Q9TyuZwMI=', 'Content-Length': '0'} 2024-04-12T03:52:08.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de0036acc80dc936-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.992 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:08.993 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:08.993 DEBUG:boto:path=/dkyges-57/e?versionId=null 2024-04-12T03:52:08.993 DEBUG:boto:auth_path=/dkyges-57/e?versionId=null 2024-04-12T03:52:08.993 DEBUG:boto:Method: HEAD 2024-04-12T03:52:08.993 DEBUG:boto:Path: /dkyges-57/e?versionId=null 2024-04-12T03:52:08.993 DEBUG:boto:Data: 2024-04-12T03:52:08.993 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.993 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:08.993 DEBUG:boto:Port: 8000 2024-04-12T03:52:08.993 DEBUG:boto:Params: {} 2024-04-12T03:52:08.993 DEBUG:boto:Token: None 2024-04-12T03:52:08.994 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/e?versionId=null 2024-04-12T03:52:08.994 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pbGDEzqbsNxBbVjNC0Q9TyuZwMI= 2024-04-12T03:52:08.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pbGDEzqbsNxBbVjNC0Q9TyuZwMI=', 'Content-Length': '0'} 2024-04-12T03:52:08.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fccdf9147a0352df-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:52:08.996 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:08.996 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:08.996 DEBUG:boto:path=/dkyges-57/e?versionId=null 2024-04-12T03:52:08.996 DEBUG:boto:auth_path=/dkyges-57/e?versionId=null 2024-04-12T03:52:08.996 DEBUG:boto:Method: GET 2024-04-12T03:52:08.996 DEBUG:boto:Path: /dkyges-57/e?versionId=null 2024-04-12T03:52:08.996 DEBUG:boto:Data: 2024-04-12T03:52:08.996 DEBUG:boto:Headers: {} 2024-04-12T03:52:08.996 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:08.997 DEBUG:boto:Port: 8001 2024-04-12T03:52:08.997 DEBUG:boto:Params: {} 2024-04-12T03:52:08.997 DEBUG:boto:Token: None 2024-04-12T03:52:08.997 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:08 GMT /dkyges-57/e?versionId=null 2024-04-12T03:52:08.997 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q89BiJwl2t9qUozc6JIu60aKJnU= 2024-04-12T03:52:08.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q89BiJwl2t9qUozc6JIu60aKJnU=', 'Content-Length': '0'} 2024-04-12T03:52:08.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f15598d78ff83e79-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:08.999 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:08.999 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:08.999 DEBUG:boto:path=/dkyges-57/e?versionId=null 2024-04-12T03:52:08.999 DEBUG:boto:auth_path=/dkyges-57/e?versionId=null 2024-04-12T03:52:08.999 DEBUG:boto:Method: GET 2024-04-12T03:52:09.000 DEBUG:boto:Path: /dkyges-57/e?versionId=null 2024-04-12T03:52:09.000 DEBUG:boto:Data: 2024-04-12T03:52:09.000 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.000 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.000 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.000 DEBUG:boto:Params: {} 2024-04-12T03:52:09.000 DEBUG:boto:Token: None 2024-04-12T03:52:09.000 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/e?versionId=null 2024-04-12T03:52:09.000 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Aw/Z4FqqhzVaZjuUJ3E+xwoIKnw= 2024-04-12T03:52:09.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Aw/Z4FqqhzVaZjuUJ3E+xwoIKnw=', 'Content-Length': '0'} 2024-04-12T03:52:09.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073731d79475d7049-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.002 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:09.003 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:09.003 DEBUG:boto:path=/dkyges-57/?prefix=e 2024-04-12T03:52:09.003 DEBUG:boto:auth_path=/dkyges-57/?prefix=e 2024-04-12T03:52:09.003 DEBUG:boto:Method: GET 2024-04-12T03:52:09.003 DEBUG:boto:Path: /dkyges-57/?prefix=e 2024-04-12T03:52:09.003 DEBUG:boto:Data: 2024-04-12T03:52:09.003 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.003 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.003 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.003 DEBUG:boto:Params: {} 2024-04-12T03:52:09.003 DEBUG:boto:Token: None 2024-04-12T03:52:09.004 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/ 2024-04-12T03:52:09.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T03:52:09.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T03:52:09.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000158c628c9102b9dd-006618afe9-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.006 DEBUG:boto:b'dkyges-57e1000falsee2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:09.007 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:09.007 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:09.007 DEBUG:boto:path=/dkyges-57/?prefix=e 2024-04-12T03:52:09.007 DEBUG:boto:auth_path=/dkyges-57/?prefix=e 2024-04-12T03:52:09.007 DEBUG:boto:Method: GET 2024-04-12T03:52:09.007 DEBUG:boto:Path: /dkyges-57/?prefix=e 2024-04-12T03:52:09.007 DEBUG:boto:Data: 2024-04-12T03:52:09.007 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.007 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.008 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.008 DEBUG:boto:Params: {} 2024-04-12T03:52:09.008 DEBUG:boto:Token: None 2024-04-12T03:52:09.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/ 2024-04-12T03:52:09.008 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T03:52:09.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T03:52:09.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000579c726f61c98331-006618afe9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.050 DEBUG:boto:b'dkyges-57e1000falsee2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:09.051 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:09.051 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:09.051 DEBUG:boto:Method: HEAD 2024-04-12T03:52:09.051 DEBUG:boto:Path: /dkyges-57/e 2024-04-12T03:52:09.051 DEBUG:boto:Data: 2024-04-12T03:52:09.051 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.051 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.051 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.051 DEBUG:boto:Params: {} 2024-04-12T03:52:09.051 DEBUG:boto:Token: None 2024-04-12T03:52:09.051 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/e 2024-04-12T03:52:09.052 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9QgWUojv7EAgIbj+/3mq4NkwK/A= 2024-04-12T03:52:09.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9QgWUojv7EAgIbj+/3mq4NkwK/A=', 'Content-Length': '0'} 2024-04-12T03:52:09.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dffab6bd09ff4183-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.054 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:09.054 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:09.054 DEBUG:boto:Method: HEAD 2024-04-12T03:52:09.054 DEBUG:boto:Path: /dkyges-57/e 2024-04-12T03:52:09.054 DEBUG:boto:Data: 2024-04-12T03:52:09.054 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.054 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.054 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.054 DEBUG:boto:Params: {} 2024-04-12T03:52:09.055 DEBUG:boto:Token: None 2024-04-12T03:52:09.055 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/e 2024-04-12T03:52:09.055 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9QgWUojv7EAgIbj+/3mq4NkwK/A= 2024-04-12T03:52:09.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9QgWUojv7EAgIbj+/3mq4NkwK/A=', 'Content-Length': '0'} 2024-04-12T03:52:09.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c993c93ae4ca45c-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:52:09.058 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:09.058 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:09.059 DEBUG:boto:Method: GET 2024-04-12T03:52:09.059 DEBUG:boto:Path: /dkyges-57/e 2024-04-12T03:52:09.059 DEBUG:boto:Data: 2024-04-12T03:52:09.059 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.059 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.059 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.059 DEBUG:boto:Params: {} 2024-04-12T03:52:09.059 DEBUG:boto:Token: None 2024-04-12T03:52:09.059 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/e 2024-04-12T03:52:09.060 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EPkGI6dpfggOwomWYLsVNfcMHwQ= 2024-04-12T03:52:09.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EPkGI6dpfggOwomWYLsVNfcMHwQ=', 'Content-Length': '0'} 2024-04-12T03:52:09.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0d0069e2f8819ef-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.062 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:09.062 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:09.062 DEBUG:boto:Method: GET 2024-04-12T03:52:09.062 DEBUG:boto:Path: /dkyges-57/e 2024-04-12T03:52:09.062 DEBUG:boto:Data: 2024-04-12T03:52:09.062 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.062 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.062 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.062 DEBUG:boto:Params: {} 2024-04-12T03:52:09.063 DEBUG:boto:Token: None 2024-04-12T03:52:09.063 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/e 2024-04-12T03:52:09.063 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EPkGI6dpfggOwomWYLsVNfcMHwQ= 2024-04-12T03:52:09.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EPkGI6dpfggOwomWYLsVNfcMHwQ=', 'Content-Length': '0'} 2024-04-12T03:52:09.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b7af509dcf15dea-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.065 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:09.065 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:09.065 DEBUG:boto:path=/dkyges-57/?prefix=e 2024-04-12T03:52:09.065 DEBUG:boto:auth_path=/dkyges-57/?prefix=e 2024-04-12T03:52:09.065 DEBUG:boto:Method: GET 2024-04-12T03:52:09.066 DEBUG:boto:Path: /dkyges-57/?prefix=e 2024-04-12T03:52:09.066 DEBUG:boto:Data: 2024-04-12T03:52:09.066 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.066 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.066 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.066 DEBUG:boto:Params: {} 2024-04-12T03:52:09.066 DEBUG:boto:Token: None 2024-04-12T03:52:09.066 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/ 2024-04-12T03:52:09.067 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T03:52:09.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T03:52:09.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068e9796ccca152b7-006618afe9-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.069 DEBUG:boto:b'dkyges-57e1000falsee2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:09.070 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:09.070 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:09.070 DEBUG:boto:path=/dkyges-57/?prefix=e 2024-04-12T03:52:09.070 DEBUG:boto:auth_path=/dkyges-57/?prefix=e 2024-04-12T03:52:09.070 DEBUG:boto:Method: GET 2024-04-12T03:52:09.070 DEBUG:boto:Path: /dkyges-57/?prefix=e 2024-04-12T03:52:09.070 DEBUG:boto:Data: 2024-04-12T03:52:09.070 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.070 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.070 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.071 DEBUG:boto:Params: {} 2024-04-12T03:52:09.071 DEBUG:boto:Token: None 2024-04-12T03:52:09.071 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/ 2024-04-12T03:52:09.071 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T03:52:09.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T03:52:09.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b73e4d3959898845-006618afe9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.073 DEBUG:boto:b'dkyges-57e1000falsee2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:09.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:52:09.073 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:09.074 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:09.074 DEBUG:boto:path=/dkyges-57/f?versionId=null 2024-04-12T03:52:09.074 DEBUG:boto:auth_path=/dkyges-57/f?versionId=null 2024-04-12T03:52:09.074 DEBUG:boto:Method: GET 2024-04-12T03:52:09.074 DEBUG:boto:Path: /dkyges-57/f?versionId=null 2024-04-12T03:52:09.074 DEBUG:boto:Data: 2024-04-12T03:52:09.074 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.075 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.075 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.075 DEBUG:boto:Params: {} 2024-04-12T03:52:09.075 DEBUG:boto:Token: None 2024-04-12T03:52:09.075 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/f?versionId=null 2024-04-12T03:52:09.075 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:njH1nDiczU9QU/NbNXKIXA9xlXc= 2024-04-12T03:52:09.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:njH1nDiczU9QU/NbNXKIXA9xlXc=', 'Content-Length': '0'} 2024-04-12T03:52:09.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd4e9cfd7311e925-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.078 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:09.078 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:09.078 DEBUG:boto:path=/dkyges-57/f?versionId=null 2024-04-12T03:52:09.078 DEBUG:boto:auth_path=/dkyges-57/f?versionId=null 2024-04-12T03:52:09.079 DEBUG:boto:Method: GET 2024-04-12T03:52:09.079 DEBUG:boto:Path: /dkyges-57/f?versionId=null 2024-04-12T03:52:09.079 DEBUG:boto:Data: 2024-04-12T03:52:09.079 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.079 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.079 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.079 DEBUG:boto:Params: {} 2024-04-12T03:52:09.079 DEBUG:boto:Token: None 2024-04-12T03:52:09.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/f?versionId=null 2024-04-12T03:52:09.079 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:njH1nDiczU9QU/NbNXKIXA9xlXc= 2024-04-12T03:52:09.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:njH1nDiczU9QU/NbNXKIXA9xlXc=', 'Content-Length': '0'} 2024-04-12T03:52:09.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000239c989bed5c5745-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.081 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:09.082 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:09.082 DEBUG:boto:path=/dkyges-57/f?versionId=null 2024-04-12T03:52:09.082 DEBUG:boto:auth_path=/dkyges-57/f?versionId=null 2024-04-12T03:52:09.082 DEBUG:boto:Method: HEAD 2024-04-12T03:52:09.082 DEBUG:boto:Path: /dkyges-57/f?versionId=null 2024-04-12T03:52:09.082 DEBUG:boto:Data: 2024-04-12T03:52:09.082 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.082 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.082 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.082 DEBUG:boto:Params: {} 2024-04-12T03:52:09.083 DEBUG:boto:Token: None 2024-04-12T03:52:09.083 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/f?versionId=null 2024-04-12T03:52:09.083 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fzppvKa60NzV9RgL3+6WfpTy4ZU= 2024-04-12T03:52:09.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fzppvKa60NzV9RgL3+6WfpTy4ZU=', 'Content-Length': '0'} 2024-04-12T03:52:09.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f31257f8a65562cb-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.085 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:09.085 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:09.085 DEBUG:boto:path=/dkyges-57/f?versionId=null 2024-04-12T03:52:09.085 DEBUG:boto:auth_path=/dkyges-57/f?versionId=null 2024-04-12T03:52:09.085 DEBUG:boto:Method: HEAD 2024-04-12T03:52:09.085 DEBUG:boto:Path: /dkyges-57/f?versionId=null 2024-04-12T03:52:09.085 DEBUG:boto:Data: 2024-04-12T03:52:09.085 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.085 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.086 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.086 DEBUG:boto:Params: {} 2024-04-12T03:52:09.086 DEBUG:boto:Token: None 2024-04-12T03:52:09.086 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/f?versionId=null 2024-04-12T03:52:09.086 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fzppvKa60NzV9RgL3+6WfpTy4ZU= 2024-04-12T03:52:09.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fzppvKa60NzV9RgL3+6WfpTy4ZU=', 'Content-Length': '0'} 2024-04-12T03:52:09.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c5e6c7ac4cf0dc6-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:52:09.088 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:09.088 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:09.089 DEBUG:boto:path=/dkyges-57/f?versionId=null 2024-04-12T03:52:09.089 DEBUG:boto:auth_path=/dkyges-57/f?versionId=null 2024-04-12T03:52:09.089 DEBUG:boto:Method: GET 2024-04-12T03:52:09.089 DEBUG:boto:Path: /dkyges-57/f?versionId=null 2024-04-12T03:52:09.089 DEBUG:boto:Data: 2024-04-12T03:52:09.089 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.089 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.089 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.089 DEBUG:boto:Params: {} 2024-04-12T03:52:09.089 DEBUG:boto:Token: None 2024-04-12T03:52:09.090 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/f?versionId=null 2024-04-12T03:52:09.090 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:njH1nDiczU9QU/NbNXKIXA9xlXc= 2024-04-12T03:52:09.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:njH1nDiczU9QU/NbNXKIXA9xlXc=', 'Content-Length': '0'} 2024-04-12T03:52:09.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004388f09198b14a6d-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.092 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:09.092 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:09.092 DEBUG:boto:path=/dkyges-57/f?versionId=null 2024-04-12T03:52:09.092 DEBUG:boto:auth_path=/dkyges-57/f?versionId=null 2024-04-12T03:52:09.092 DEBUG:boto:Method: GET 2024-04-12T03:52:09.092 DEBUG:boto:Path: /dkyges-57/f?versionId=null 2024-04-12T03:52:09.092 DEBUG:boto:Data: 2024-04-12T03:52:09.092 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.093 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.093 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.093 DEBUG:boto:Params: {} 2024-04-12T03:52:09.093 DEBUG:boto:Token: None 2024-04-12T03:52:09.093 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/f?versionId=null 2024-04-12T03:52:09.093 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:njH1nDiczU9QU/NbNXKIXA9xlXc= 2024-04-12T03:52:09.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:njH1nDiczU9QU/NbNXKIXA9xlXc=', 'Content-Length': '0'} 2024-04-12T03:52:09.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009af2af26f9ca00c0-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.095 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:09.096 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:09.096 DEBUG:boto:path=/dkyges-57/?prefix=f 2024-04-12T03:52:09.096 DEBUG:boto:auth_path=/dkyges-57/?prefix=f 2024-04-12T03:52:09.096 DEBUG:boto:Method: GET 2024-04-12T03:52:09.096 DEBUG:boto:Path: /dkyges-57/?prefix=f 2024-04-12T03:52:09.096 DEBUG:boto:Data: 2024-04-12T03:52:09.096 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.096 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.096 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.096 DEBUG:boto:Params: {} 2024-04-12T03:52:09.097 DEBUG:boto:Token: None 2024-04-12T03:52:09.097 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/ 2024-04-12T03:52:09.097 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T03:52:09.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T03:52:09.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eb971fd06f06f6a-006618afe9-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.099 DEBUG:boto:b'dkyges-57f1000falsef2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:09.100 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:09.100 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:09.100 DEBUG:boto:path=/dkyges-57/?prefix=f 2024-04-12T03:52:09.100 DEBUG:boto:auth_path=/dkyges-57/?prefix=f 2024-04-12T03:52:09.100 DEBUG:boto:Method: GET 2024-04-12T03:52:09.100 DEBUG:boto:Path: /dkyges-57/?prefix=f 2024-04-12T03:52:09.100 DEBUG:boto:Data: 2024-04-12T03:52:09.100 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.100 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.101 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.101 DEBUG:boto:Params: {} 2024-04-12T03:52:09.101 DEBUG:boto:Token: None 2024-04-12T03:52:09.101 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/ 2024-04-12T03:52:09.101 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T03:52:09.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T03:52:09.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065ddd1ef748dee76-006618afe9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.103 DEBUG:boto:b'dkyges-57f1000falsef2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:09.103 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:09.104 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:09.104 DEBUG:boto:Method: HEAD 2024-04-12T03:52:09.104 DEBUG:boto:Path: /dkyges-57/f 2024-04-12T03:52:09.104 DEBUG:boto:Data: 2024-04-12T03:52:09.104 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.104 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.104 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.104 DEBUG:boto:Params: {} 2024-04-12T03:52:09.104 DEBUG:boto:Token: None 2024-04-12T03:52:09.104 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/f 2024-04-12T03:52:09.105 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bckDdjNiGB0KGSyb9N62Se/noHY= 2024-04-12T03:52:09.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bckDdjNiGB0KGSyb9N62Se/noHY=', 'Content-Length': '0'} 2024-04-12T03:52:09.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000136bc644439cb2df-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.107 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:09.107 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:09.108 DEBUG:boto:Method: HEAD 2024-04-12T03:52:09.108 DEBUG:boto:Path: /dkyges-57/f 2024-04-12T03:52:09.108 DEBUG:boto:Data: 2024-04-12T03:52:09.108 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.108 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.108 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.108 DEBUG:boto:Params: {} 2024-04-12T03:52:09.108 DEBUG:boto:Token: None 2024-04-12T03:52:09.108 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/f 2024-04-12T03:52:09.108 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bckDdjNiGB0KGSyb9N62Se/noHY= 2024-04-12T03:52:09.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bckDdjNiGB0KGSyb9N62Se/noHY=', 'Content-Length': '0'} 2024-04-12T03:52:09.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bb995881b2b880d-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:52:09.111 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:09.111 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:09.111 DEBUG:boto:Method: GET 2024-04-12T03:52:09.111 DEBUG:boto:Path: /dkyges-57/f 2024-04-12T03:52:09.111 DEBUG:boto:Data: 2024-04-12T03:52:09.111 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.111 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.111 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.111 DEBUG:boto:Params: {} 2024-04-12T03:52:09.112 DEBUG:boto:Token: None 2024-04-12T03:52:09.112 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/f 2024-04-12T03:52:09.112 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZSXCjSvc/OjwM2iq0xYTDRZk4OM= 2024-04-12T03:52:09.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZSXCjSvc/OjwM2iq0xYTDRZk4OM=', 'Content-Length': '0'} 2024-04-12T03:52:09.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd3857cc8739d5e4-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.154 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:09.154 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:09.155 DEBUG:boto:Method: GET 2024-04-12T03:52:09.155 DEBUG:boto:Path: /dkyges-57/f 2024-04-12T03:52:09.155 DEBUG:boto:Data: 2024-04-12T03:52:09.155 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.155 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.155 DEBUG:boto:Params: {} 2024-04-12T03:52:09.155 DEBUG:boto:Token: None 2024-04-12T03:52:09.155 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/f 2024-04-12T03:52:09.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZSXCjSvc/OjwM2iq0xYTDRZk4OM= 2024-04-12T03:52:09.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZSXCjSvc/OjwM2iq0xYTDRZk4OM=', 'Content-Length': '0'} 2024-04-12T03:52:09.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d68ea6530471840-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.202 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:09.202 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:09.203 DEBUG:boto:path=/dkyges-57/?prefix=f 2024-04-12T03:52:09.203 DEBUG:boto:auth_path=/dkyges-57/?prefix=f 2024-04-12T03:52:09.203 DEBUG:boto:Method: GET 2024-04-12T03:52:09.203 DEBUG:boto:Path: /dkyges-57/?prefix=f 2024-04-12T03:52:09.203 DEBUG:boto:Data: 2024-04-12T03:52:09.203 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.203 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.203 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.203 DEBUG:boto:Params: {} 2024-04-12T03:52:09.203 DEBUG:boto:Token: None 2024-04-12T03:52:09.203 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/ 2024-04-12T03:52:09.204 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T03:52:09.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T03:52:09.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7a0024e1571f1cc-006618afe9-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.207 DEBUG:boto:b'dkyges-57f1000falsef2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:09.207 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:09.207 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:09.207 DEBUG:boto:path=/dkyges-57/?prefix=f 2024-04-12T03:52:09.207 DEBUG:boto:auth_path=/dkyges-57/?prefix=f 2024-04-12T03:52:09.207 DEBUG:boto:Method: GET 2024-04-12T03:52:09.208 DEBUG:boto:Path: /dkyges-57/?prefix=f 2024-04-12T03:52:09.208 DEBUG:boto:Data: 2024-04-12T03:52:09.208 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.208 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.208 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.208 DEBUG:boto:Params: {} 2024-04-12T03:52:09.208 DEBUG:boto:Token: None 2024-04-12T03:52:09.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/ 2024-04-12T03:52:09.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T03:52:09.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T03:52:09.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db15527bbd62bd0e-006618afe9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.210 DEBUG:boto:b'dkyges-57f1000falsef2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:09.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:52:09.211 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:09.211 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:09.211 DEBUG:boto:path=/dkyges-57/g?versionId=null 2024-04-12T03:52:09.211 DEBUG:boto:auth_path=/dkyges-57/g?versionId=null 2024-04-12T03:52:09.211 DEBUG:boto:Method: GET 2024-04-12T03:52:09.211 DEBUG:boto:Path: /dkyges-57/g?versionId=null 2024-04-12T03:52:09.211 DEBUG:boto:Data: 2024-04-12T03:52:09.211 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.211 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.212 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.212 DEBUG:boto:Params: {} 2024-04-12T03:52:09.212 DEBUG:boto:Token: None 2024-04-12T03:52:09.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/g?versionId=null 2024-04-12T03:52:09.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bh2BoZgaF1X9j9/w2zWsQ0Qpi5g= 2024-04-12T03:52:09.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bh2BoZgaF1X9j9/w2zWsQ0Qpi5g=', 'Content-Length': '0'} 2024-04-12T03:52:09.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049ed5d2cd54c06c9-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.214 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:09.214 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:09.215 DEBUG:boto:path=/dkyges-57/g?versionId=null 2024-04-12T03:52:09.215 DEBUG:boto:auth_path=/dkyges-57/g?versionId=null 2024-04-12T03:52:09.215 DEBUG:boto:Method: GET 2024-04-12T03:52:09.215 DEBUG:boto:Path: /dkyges-57/g?versionId=null 2024-04-12T03:52:09.215 DEBUG:boto:Data: 2024-04-12T03:52:09.215 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.215 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.215 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.215 DEBUG:boto:Params: {} 2024-04-12T03:52:09.215 DEBUG:boto:Token: None 2024-04-12T03:52:09.215 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/g?versionId=null 2024-04-12T03:52:09.216 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bh2BoZgaF1X9j9/w2zWsQ0Qpi5g= 2024-04-12T03:52:09.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bh2BoZgaF1X9j9/w2zWsQ0Qpi5g=', 'Content-Length': '0'} 2024-04-12T03:52:09.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097faad90354984b4-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.218 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:09.218 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:09.218 DEBUG:boto:path=/dkyges-57/g?versionId=null 2024-04-12T03:52:09.218 DEBUG:boto:auth_path=/dkyges-57/g?versionId=null 2024-04-12T03:52:09.218 DEBUG:boto:Method: HEAD 2024-04-12T03:52:09.218 DEBUG:boto:Path: /dkyges-57/g?versionId=null 2024-04-12T03:52:09.218 DEBUG:boto:Data: 2024-04-12T03:52:09.219 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.219 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.219 DEBUG:boto:Params: {} 2024-04-12T03:52:09.219 DEBUG:boto:Token: None 2024-04-12T03:52:09.219 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/g?versionId=null 2024-04-12T03:52:09.219 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gLo7WklETjIPHN+HYXO9FAprDEo= 2024-04-12T03:52:09.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gLo7WklETjIPHN+HYXO9FAprDEo=', 'Content-Length': '0'} 2024-04-12T03:52:09.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cc02bfd260fceeb-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.221 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:09.221 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:09.221 DEBUG:boto:path=/dkyges-57/g?versionId=null 2024-04-12T03:52:09.221 DEBUG:boto:auth_path=/dkyges-57/g?versionId=null 2024-04-12T03:52:09.222 DEBUG:boto:Method: HEAD 2024-04-12T03:52:09.222 DEBUG:boto:Path: /dkyges-57/g?versionId=null 2024-04-12T03:52:09.222 DEBUG:boto:Data: 2024-04-12T03:52:09.222 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.222 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.222 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.222 DEBUG:boto:Params: {} 2024-04-12T03:52:09.222 DEBUG:boto:Token: None 2024-04-12T03:52:09.222 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/g?versionId=null 2024-04-12T03:52:09.223 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gLo7WklETjIPHN+HYXO9FAprDEo= 2024-04-12T03:52:09.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gLo7WklETjIPHN+HYXO9FAprDEo=', 'Content-Length': '0'} 2024-04-12T03:52:09.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b98da4d31e62774-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:52:09.225 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:09.225 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:09.225 DEBUG:boto:path=/dkyges-57/g?versionId=null 2024-04-12T03:52:09.225 DEBUG:boto:auth_path=/dkyges-57/g?versionId=null 2024-04-12T03:52:09.225 DEBUG:boto:Method: GET 2024-04-12T03:52:09.225 DEBUG:boto:Path: /dkyges-57/g?versionId=null 2024-04-12T03:52:09.225 DEBUG:boto:Data: 2024-04-12T03:52:09.225 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.225 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.226 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.226 DEBUG:boto:Params: {} 2024-04-12T03:52:09.226 DEBUG:boto:Token: None 2024-04-12T03:52:09.226 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/g?versionId=null 2024-04-12T03:52:09.226 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bh2BoZgaF1X9j9/w2zWsQ0Qpi5g= 2024-04-12T03:52:09.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bh2BoZgaF1X9j9/w2zWsQ0Qpi5g=', 'Content-Length': '0'} 2024-04-12T03:52:09.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3c6b1a5a45ad686-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.228 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:09.228 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:09.228 DEBUG:boto:path=/dkyges-57/g?versionId=null 2024-04-12T03:52:09.228 DEBUG:boto:auth_path=/dkyges-57/g?versionId=null 2024-04-12T03:52:09.229 DEBUG:boto:Method: GET 2024-04-12T03:52:09.229 DEBUG:boto:Path: /dkyges-57/g?versionId=null 2024-04-12T03:52:09.229 DEBUG:boto:Data: 2024-04-12T03:52:09.229 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.229 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.229 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.229 DEBUG:boto:Params: {} 2024-04-12T03:52:09.229 DEBUG:boto:Token: None 2024-04-12T03:52:09.229 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/g?versionId=null 2024-04-12T03:52:09.229 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bh2BoZgaF1X9j9/w2zWsQ0Qpi5g= 2024-04-12T03:52:09.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bh2BoZgaF1X9j9/w2zWsQ0Qpi5g=', 'Content-Length': '0'} 2024-04-12T03:52:09.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023416118a596420a-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.232 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:09.232 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:09.232 DEBUG:boto:path=/dkyges-57/?prefix=g 2024-04-12T03:52:09.232 DEBUG:boto:auth_path=/dkyges-57/?prefix=g 2024-04-12T03:52:09.232 DEBUG:boto:Method: GET 2024-04-12T03:52:09.232 DEBUG:boto:Path: /dkyges-57/?prefix=g 2024-04-12T03:52:09.232 DEBUG:boto:Data: 2024-04-12T03:52:09.232 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.232 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.233 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.233 DEBUG:boto:Params: {} 2024-04-12T03:52:09.233 DEBUG:boto:Token: None 2024-04-12T03:52:09.233 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/ 2024-04-12T03:52:09.233 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T03:52:09.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T03:52:09.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002051cc43332e060-006618afe9-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.237 DEBUG:boto:b'dkyges-57g1000falseg2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:09.238 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:09.238 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:09.238 DEBUG:boto:path=/dkyges-57/?prefix=g 2024-04-12T03:52:09.238 DEBUG:boto:auth_path=/dkyges-57/?prefix=g 2024-04-12T03:52:09.238 DEBUG:boto:Method: GET 2024-04-12T03:52:09.238 DEBUG:boto:Path: /dkyges-57/?prefix=g 2024-04-12T03:52:09.239 DEBUG:boto:Data: 2024-04-12T03:52:09.239 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.239 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.239 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.239 DEBUG:boto:Params: {} 2024-04-12T03:52:09.240 DEBUG:boto:Token: None 2024-04-12T03:52:09.240 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/ 2024-04-12T03:52:09.240 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T03:52:09.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T03:52:09.242 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006eb319b125ecb9f4-006618afe9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.290 DEBUG:boto:b'dkyges-57g1000falseg2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:09.291 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:09.291 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:09.291 DEBUG:boto:Method: HEAD 2024-04-12T03:52:09.291 DEBUG:boto:Path: /dkyges-57/g 2024-04-12T03:52:09.291 DEBUG:boto:Data: 2024-04-12T03:52:09.291 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.291 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.292 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.292 DEBUG:boto:Params: {} 2024-04-12T03:52:09.292 DEBUG:boto:Token: None 2024-04-12T03:52:09.292 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/g 2024-04-12T03:52:09.292 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:46J5ArzLuULtB68uTm6zt/9YPOA= 2024-04-12T03:52:09.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:46J5ArzLuULtB68uTm6zt/9YPOA=', 'Content-Length': '0'} 2024-04-12T03:52:09.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062e0e10ba82b9d7d-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.294 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:09.295 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:09.295 DEBUG:boto:Method: HEAD 2024-04-12T03:52:09.295 DEBUG:boto:Path: /dkyges-57/g 2024-04-12T03:52:09.295 DEBUG:boto:Data: 2024-04-12T03:52:09.295 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.295 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.295 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.295 DEBUG:boto:Params: {} 2024-04-12T03:52:09.295 DEBUG:boto:Token: None 2024-04-12T03:52:09.296 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/g 2024-04-12T03:52:09.296 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:46J5ArzLuULtB68uTm6zt/9YPOA= 2024-04-12T03:52:09.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:46J5ArzLuULtB68uTm6zt/9YPOA=', 'Content-Length': '0'} 2024-04-12T03:52:09.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a80255d1c299bc94-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:52:09.298 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:09.298 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:09.298 DEBUG:boto:Method: GET 2024-04-12T03:52:09.299 DEBUG:boto:Path: /dkyges-57/g 2024-04-12T03:52:09.299 DEBUG:boto:Data: 2024-04-12T03:52:09.299 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.299 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.299 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.299 DEBUG:boto:Params: {} 2024-04-12T03:52:09.299 DEBUG:boto:Token: None 2024-04-12T03:52:09.299 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/g 2024-04-12T03:52:09.299 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sh8mSx1bcnkXXdsmf8OX7g689NM= 2024-04-12T03:52:09.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sh8mSx1bcnkXXdsmf8OX7g689NM=', 'Content-Length': '0'} 2024-04-12T03:52:09.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f72e877e42e0330-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.301 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:09.301 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:09.301 DEBUG:boto:Method: GET 2024-04-12T03:52:09.302 DEBUG:boto:Path: /dkyges-57/g 2024-04-12T03:52:09.302 DEBUG:boto:Data: 2024-04-12T03:52:09.302 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.302 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.302 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.302 DEBUG:boto:Params: {} 2024-04-12T03:52:09.302 DEBUG:boto:Token: None 2024-04-12T03:52:09.302 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/g 2024-04-12T03:52:09.302 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sh8mSx1bcnkXXdsmf8OX7g689NM= 2024-04-12T03:52:09.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sh8mSx1bcnkXXdsmf8OX7g689NM=', 'Content-Length': '0'} 2024-04-12T03:52:09.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ab3dbe8e1867b73-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.304 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:09.305 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:09.305 DEBUG:boto:path=/dkyges-57/?prefix=g 2024-04-12T03:52:09.305 DEBUG:boto:auth_path=/dkyges-57/?prefix=g 2024-04-12T03:52:09.305 DEBUG:boto:Method: GET 2024-04-12T03:52:09.305 DEBUG:boto:Path: /dkyges-57/?prefix=g 2024-04-12T03:52:09.305 DEBUG:boto:Data: 2024-04-12T03:52:09.305 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.305 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.305 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.305 DEBUG:boto:Params: {} 2024-04-12T03:52:09.305 DEBUG:boto:Token: None 2024-04-12T03:52:09.306 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/ 2024-04-12T03:52:09.306 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T03:52:09.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T03:52:09.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000407c66824c6d7d8-006618afe9-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.308 DEBUG:boto:b'dkyges-57g1000falseg2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:09.309 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:09.309 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:09.309 DEBUG:boto:path=/dkyges-57/?prefix=g 2024-04-12T03:52:09.309 DEBUG:boto:auth_path=/dkyges-57/?prefix=g 2024-04-12T03:52:09.309 DEBUG:boto:Method: GET 2024-04-12T03:52:09.309 DEBUG:boto:Path: /dkyges-57/?prefix=g 2024-04-12T03:52:09.309 DEBUG:boto:Data: 2024-04-12T03:52:09.309 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.310 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.310 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.310 DEBUG:boto:Params: {} 2024-04-12T03:52:09.310 DEBUG:boto:Token: None 2024-04-12T03:52:09.310 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/ 2024-04-12T03:52:09.310 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T03:52:09.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T03:52:09.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002191faa56775886c-006618afe9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.312 DEBUG:boto:b'dkyges-57g1000falseg2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:09.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:52:09.313 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:09.313 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:09.313 DEBUG:boto:path=/dkyges-57/h?versionId=null 2024-04-12T03:52:09.313 DEBUG:boto:auth_path=/dkyges-57/h?versionId=null 2024-04-12T03:52:09.313 DEBUG:boto:Method: GET 2024-04-12T03:52:09.313 DEBUG:boto:Path: /dkyges-57/h?versionId=null 2024-04-12T03:52:09.313 DEBUG:boto:Data: 2024-04-12T03:52:09.313 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.314 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.314 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.314 DEBUG:boto:Params: {} 2024-04-12T03:52:09.314 DEBUG:boto:Token: None 2024-04-12T03:52:09.314 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/h?versionId=null 2024-04-12T03:52:09.314 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JS3xVsvJtGhF8Ws0XtUa7GPAMPk= 2024-04-12T03:52:09.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JS3xVsvJtGhF8Ws0XtUa7GPAMPk=', 'Content-Length': '0'} 2024-04-12T03:52:09.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035f395213db9fad2-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.316 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:09.317 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:09.317 DEBUG:boto:path=/dkyges-57/h?versionId=null 2024-04-12T03:52:09.317 DEBUG:boto:auth_path=/dkyges-57/h?versionId=null 2024-04-12T03:52:09.317 DEBUG:boto:Method: GET 2024-04-12T03:52:09.317 DEBUG:boto:Path: /dkyges-57/h?versionId=null 2024-04-12T03:52:09.317 DEBUG:boto:Data: 2024-04-12T03:52:09.317 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.317 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.317 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.317 DEBUG:boto:Params: {} 2024-04-12T03:52:09.317 DEBUG:boto:Token: None 2024-04-12T03:52:09.318 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/h?versionId=null 2024-04-12T03:52:09.318 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JS3xVsvJtGhF8Ws0XtUa7GPAMPk= 2024-04-12T03:52:09.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JS3xVsvJtGhF8Ws0XtUa7GPAMPk=', 'Content-Length': '0'} 2024-04-12T03:52:09.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004139ac85ddf57896-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.320 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:09.320 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:09.320 DEBUG:boto:path=/dkyges-57/h?versionId=null 2024-04-12T03:52:09.320 DEBUG:boto:auth_path=/dkyges-57/h?versionId=null 2024-04-12T03:52:09.320 DEBUG:boto:Method: HEAD 2024-04-12T03:52:09.320 DEBUG:boto:Path: /dkyges-57/h?versionId=null 2024-04-12T03:52:09.320 DEBUG:boto:Data: 2024-04-12T03:52:09.320 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.320 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.321 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.321 DEBUG:boto:Params: {} 2024-04-12T03:52:09.321 DEBUG:boto:Token: None 2024-04-12T03:52:09.321 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/h?versionId=null 2024-04-12T03:52:09.321 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G8C9tnD8MgJOjrt3A5jPFeG+7rM= 2024-04-12T03:52:09.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G8C9tnD8MgJOjrt3A5jPFeG+7rM=', 'Content-Length': '0'} 2024-04-12T03:52:09.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e239d8cdfeadcfbb-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.323 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:09.323 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:09.323 DEBUG:boto:path=/dkyges-57/h?versionId=null 2024-04-12T03:52:09.323 DEBUG:boto:auth_path=/dkyges-57/h?versionId=null 2024-04-12T03:52:09.323 DEBUG:boto:Method: HEAD 2024-04-12T03:52:09.323 DEBUG:boto:Path: /dkyges-57/h?versionId=null 2024-04-12T03:52:09.323 DEBUG:boto:Data: 2024-04-12T03:52:09.323 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.324 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.324 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.324 DEBUG:boto:Params: {} 2024-04-12T03:52:09.324 DEBUG:boto:Token: None 2024-04-12T03:52:09.324 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/h?versionId=null 2024-04-12T03:52:09.324 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G8C9tnD8MgJOjrt3A5jPFeG+7rM= 2024-04-12T03:52:09.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G8C9tnD8MgJOjrt3A5jPFeG+7rM=', 'Content-Length': '0'} 2024-04-12T03:52:09.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027b9aea8800d6240-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:52:09.327 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:09.327 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:09.327 DEBUG:boto:path=/dkyges-57/h?versionId=null 2024-04-12T03:52:09.327 DEBUG:boto:auth_path=/dkyges-57/h?versionId=null 2024-04-12T03:52:09.327 DEBUG:boto:Method: GET 2024-04-12T03:52:09.327 DEBUG:boto:Path: /dkyges-57/h?versionId=null 2024-04-12T03:52:09.327 DEBUG:boto:Data: 2024-04-12T03:52:09.327 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.328 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.328 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.328 DEBUG:boto:Params: {} 2024-04-12T03:52:09.328 DEBUG:boto:Token: None 2024-04-12T03:52:09.328 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/h?versionId=null 2024-04-12T03:52:09.328 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JS3xVsvJtGhF8Ws0XtUa7GPAMPk= 2024-04-12T03:52:09.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JS3xVsvJtGhF8Ws0XtUa7GPAMPk=', 'Content-Length': '0'} 2024-04-12T03:52:09.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd6f8887951e82f0-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.330 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:09.331 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:09.331 DEBUG:boto:path=/dkyges-57/h?versionId=null 2024-04-12T03:52:09.331 DEBUG:boto:auth_path=/dkyges-57/h?versionId=null 2024-04-12T03:52:09.331 DEBUG:boto:Method: GET 2024-04-12T03:52:09.331 DEBUG:boto:Path: /dkyges-57/h?versionId=null 2024-04-12T03:52:09.331 DEBUG:boto:Data: 2024-04-12T03:52:09.331 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.331 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.331 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.331 DEBUG:boto:Params: {} 2024-04-12T03:52:09.331 DEBUG:boto:Token: None 2024-04-12T03:52:09.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/h?versionId=null 2024-04-12T03:52:09.332 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JS3xVsvJtGhF8Ws0XtUa7GPAMPk= 2024-04-12T03:52:09.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JS3xVsvJtGhF8Ws0XtUa7GPAMPk=', 'Content-Length': '0'} 2024-04-12T03:52:09.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054c37af7086f4df0-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.334 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:09.334 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:09.334 DEBUG:boto:path=/dkyges-57/?prefix=h 2024-04-12T03:52:09.334 DEBUG:boto:auth_path=/dkyges-57/?prefix=h 2024-04-12T03:52:09.334 DEBUG:boto:Method: GET 2024-04-12T03:52:09.334 DEBUG:boto:Path: /dkyges-57/?prefix=h 2024-04-12T03:52:09.334 DEBUG:boto:Data: 2024-04-12T03:52:09.334 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.335 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.335 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.335 DEBUG:boto:Params: {} 2024-04-12T03:52:09.335 DEBUG:boto:Token: None 2024-04-12T03:52:09.335 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/ 2024-04-12T03:52:09.335 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T03:52:09.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T03:52:09.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004300c004407ad87a-006618afe9-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.378 DEBUG:boto:b'dkyges-57h1000falseh2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:09.379 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:09.379 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:09.379 DEBUG:boto:path=/dkyges-57/?prefix=h 2024-04-12T03:52:09.379 DEBUG:boto:auth_path=/dkyges-57/?prefix=h 2024-04-12T03:52:09.379 DEBUG:boto:Method: GET 2024-04-12T03:52:09.379 DEBUG:boto:Path: /dkyges-57/?prefix=h 2024-04-12T03:52:09.379 DEBUG:boto:Data: 2024-04-12T03:52:09.379 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.379 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.379 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.380 DEBUG:boto:Params: {} 2024-04-12T03:52:09.380 DEBUG:boto:Token: None 2024-04-12T03:52:09.380 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/ 2024-04-12T03:52:09.380 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T03:52:09.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T03:52:09.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003db73fb21c9183b3-006618afe9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.382 DEBUG:boto:b'dkyges-57h1000falseh2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:09.382 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:09.383 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:09.383 DEBUG:boto:Method: HEAD 2024-04-12T03:52:09.383 DEBUG:boto:Path: /dkyges-57/h 2024-04-12T03:52:09.383 DEBUG:boto:Data: 2024-04-12T03:52:09.383 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.383 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.383 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.383 DEBUG:boto:Params: {} 2024-04-12T03:52:09.383 DEBUG:boto:Token: None 2024-04-12T03:52:09.384 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/h 2024-04-12T03:52:09.384 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zRpm0g/Dbe3JbrzoKhmChhv2GFs= 2024-04-12T03:52:09.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zRpm0g/Dbe3JbrzoKhmChhv2GFs=', 'Content-Length': '0'} 2024-04-12T03:52:09.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4bdbaa0555be112-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.386 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:09.386 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:09.386 DEBUG:boto:Method: HEAD 2024-04-12T03:52:09.386 DEBUG:boto:Path: /dkyges-57/h 2024-04-12T03:52:09.386 DEBUG:boto:Data: 2024-04-12T03:52:09.386 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.386 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.386 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.386 DEBUG:boto:Params: {} 2024-04-12T03:52:09.387 DEBUG:boto:Token: None 2024-04-12T03:52:09.387 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/h 2024-04-12T03:52:09.387 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zRpm0g/Dbe3JbrzoKhmChhv2GFs= 2024-04-12T03:52:09.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zRpm0g/Dbe3JbrzoKhmChhv2GFs=', 'Content-Length': '0'} 2024-04-12T03:52:09.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000330e44a40f407b12-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:52:09.389 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:09.389 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:09.389 DEBUG:boto:Method: GET 2024-04-12T03:52:09.389 DEBUG:boto:Path: /dkyges-57/h 2024-04-12T03:52:09.389 DEBUG:boto:Data: 2024-04-12T03:52:09.389 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.389 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.390 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.390 DEBUG:boto:Params: {} 2024-04-12T03:52:09.390 DEBUG:boto:Token: None 2024-04-12T03:52:09.390 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/h 2024-04-12T03:52:09.390 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:goIijJL+Jcrj/hMqRqfmNdfVup0= 2024-04-12T03:52:09.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:goIijJL+Jcrj/hMqRqfmNdfVup0=', 'Content-Length': '0'} 2024-04-12T03:52:09.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001755bd0e9e876b46-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.392 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:09.392 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:09.392 DEBUG:boto:Method: GET 2024-04-12T03:52:09.392 DEBUG:boto:Path: /dkyges-57/h 2024-04-12T03:52:09.393 DEBUG:boto:Data: 2024-04-12T03:52:09.393 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.393 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.393 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.393 DEBUG:boto:Params: {} 2024-04-12T03:52:09.393 DEBUG:boto:Token: None 2024-04-12T03:52:09.393 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/h 2024-04-12T03:52:09.393 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:goIijJL+Jcrj/hMqRqfmNdfVup0= 2024-04-12T03:52:09.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:goIijJL+Jcrj/hMqRqfmNdfVup0=', 'Content-Length': '0'} 2024-04-12T03:52:09.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d388f9793d44b317-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.395 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:09.395 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:09.396 DEBUG:boto:path=/dkyges-57/?prefix=h 2024-04-12T03:52:09.396 DEBUG:boto:auth_path=/dkyges-57/?prefix=h 2024-04-12T03:52:09.396 DEBUG:boto:Method: GET 2024-04-12T03:52:09.396 DEBUG:boto:Path: /dkyges-57/?prefix=h 2024-04-12T03:52:09.396 DEBUG:boto:Data: 2024-04-12T03:52:09.396 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.396 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:09.396 DEBUG:boto:Port: 8001 2024-04-12T03:52:09.396 DEBUG:boto:Params: {} 2024-04-12T03:52:09.396 DEBUG:boto:Token: None 2024-04-12T03:52:09.396 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/ 2024-04-12T03:52:09.396 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T03:52:09.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T03:52:09.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf4ee1602a2458d1-006618afe9-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.399 DEBUG:boto:b'dkyges-57h1000falseh2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:09.400 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:09.400 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:09.400 DEBUG:boto:path=/dkyges-57/?prefix=h 2024-04-12T03:52:09.400 DEBUG:boto:auth_path=/dkyges-57/?prefix=h 2024-04-12T03:52:09.400 DEBUG:boto:Method: GET 2024-04-12T03:52:09.400 DEBUG:boto:Path: /dkyges-57/?prefix=h 2024-04-12T03:52:09.400 DEBUG:boto:Data: 2024-04-12T03:52:09.400 DEBUG:boto:Headers: {} 2024-04-12T03:52:09.400 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:09.401 DEBUG:boto:Port: 8000 2024-04-12T03:52:09.401 DEBUG:boto:Params: {} 2024-04-12T03:52:09.401 DEBUG:boto:Token: None 2024-04-12T03:52:09.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:09 GMT /dkyges-57/ 2024-04-12T03:52:09.401 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T03:52:09.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T03:52:09.403 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb490b5303eb8c49-006618afe9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:09.403 DEBUG:boto:b'dkyges-57h1000falseh2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:09.403 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-57 zones={a2, a1} 2024-04-12T03:52:09.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dkyges-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:52:09.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dkyges-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:52:09.404 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dkyges-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:52:09.463 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:52:09.463 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:52:09.509 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T03:52:09.509 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-57 2024-04-12T03:52:09.510 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 8 2024-04-12T03:52:09.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:52:09.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:52:09.542 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dkyges-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:52:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:52:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:52:09.644 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:52:09.644 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:09.644 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding": "None", 2024-04-12T03:52:09.644 INFO:teuthology.orchestra.run.smithi043.stdout: "current_index": { 2024-04-12T03:52:09.644 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 2, 2024-04-12T03:52:09.645 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:09.645 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "Normal", 2024-04-12T03:52:09.645 INFO:teuthology.orchestra.run.smithi043.stdout: "normal": { 2024-04-12T03:52:09.645 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 3, 2024-04-12T03:52:09.645 INFO:teuthology.orchestra.run.smithi043.stdout: "hash_type": "Mod" 2024-04-12T03:52:09.645 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:09.645 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:09.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:52:09.645 INFO:teuthology.orchestra.run.smithi043.stdout: "logs": [ 2024-04-12T03:52:09.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:52:09.645 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 0, 2024-04-12T03:52:09.645 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:09.646 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "InIndex", 2024-04-12T03:52:09.646 INFO:teuthology.orchestra.run.smithi043.stdout: "in_index": { 2024-04-12T03:52:09.646 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 0, 2024-04-12T03:52:09.646 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:09.646 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 11, 2024-04-12T03:52:09.646 INFO:teuthology.orchestra.run.smithi043.stdout: "hash_type": "Mod" 2024-04-12T03:52:09.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:09.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:09.649 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:09.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:52:09.650 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:52:09.650 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 1, 2024-04-12T03:52:09.650 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:09.650 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "InIndex", 2024-04-12T03:52:09.650 INFO:teuthology.orchestra.run.smithi043.stdout: "in_index": { 2024-04-12T03:52:09.651 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 1, 2024-04-12T03:52:09.651 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:09.651 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 3, 2024-04-12T03:52:09.651 INFO:teuthology.orchestra.run.smithi043.stdout: "hash_type": "Mod" 2024-04-12T03:52:09.651 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:09.651 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:09.651 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:09.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:52:09.651 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:52:09.651 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 2, 2024-04-12T03:52:09.651 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:09.652 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "InIndex", 2024-04-12T03:52:09.652 INFO:teuthology.orchestra.run.smithi043.stdout: "in_index": { 2024-04-12T03:52:09.652 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 2, 2024-04-12T03:52:09.652 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:09.652 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 3, 2024-04-12T03:52:09.652 INFO:teuthology.orchestra.run.smithi043.stdout: "hash_type": "Mod" 2024-04-12T03:52:09.652 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:09.652 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:09.652 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:09.652 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:09.653 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:52:09.653 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:09.653 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:52:09.654 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-57 2024-04-12T03:52:09.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:52:09.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:52:09.655 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:52:09.718 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:52:09.718 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:52:09.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:09.767+0000 7f4422409780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:52:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:14.767+0000 7f4422409780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:52:19.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:19.768+0000 7f4422409780 1 bucket sync caught up with empty source 2024-04-12T03:52:19.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:19.768+0000 7f4422409780 0 bucket checkpoint complete 2024-04-12T03:52:19.780 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-57 2024-04-12T03:52:19.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:52:19.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:52:19.780 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:52:19.846 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:52:19.846 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:52:19.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:19.895+0000 7ff2727f9780 1 bucket sync caught up with source: 2024-04-12T03:52:19.887 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.2910.6, , , 00000000001.3705.6, 00000000001.1066.6, , , 00000000002.1606.6, , 00000000002.3706.6] 2024-04-12T03:52:19.887 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.2910.6, , , 00000000001.3705.6, 00000000001.1066.6, , , 00000000002.1606.6, , 00000000002.3706.6] 2024-04-12T03:52:19.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:19.895+0000 7ff2727f9780 0 bucket checkpoint complete 2024-04-12T03:52:19.903 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-57 zones={a2, a1} 2024-04-12T03:52:19.903 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:19.904 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:19.904 DEBUG:boto:Method: HEAD 2024-04-12T03:52:19.904 DEBUG:boto:Path: /dkyges-57/ 2024-04-12T03:52:19.904 DEBUG:boto:Data: 2024-04-12T03:52:19.904 DEBUG:boto:Headers: {} 2024-04-12T03:52:19.904 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:19.904 DEBUG:boto:Port: 8001 2024-04-12T03:52:19.904 DEBUG:boto:Params: {} 2024-04-12T03:52:19.904 DEBUG:boto:Token: None 2024-04-12T03:52:19.905 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:19 GMT /dkyges-57/ 2024-04-12T03:52:19.905 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qEnChDTCyFDPGew7fIvwxBdd6uk= 2024-04-12T03:52:19.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qEnChDTCyFDPGew7fIvwxBdd6uk=', 'Content-Length': '0'} 2024-04-12T03:52:19.907 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000403e8f629577ee0-006618aff3-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:19.907 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:19.907 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:19.908 DEBUG:boto:Method: HEAD 2024-04-12T03:52:19.908 DEBUG:boto:Path: /dkyges-57/ 2024-04-12T03:52:19.908 DEBUG:boto:Data: 2024-04-12T03:52:19.908 DEBUG:boto:Headers: {} 2024-04-12T03:52:19.908 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:19.908 DEBUG:boto:Port: 8000 2024-04-12T03:52:19.908 DEBUG:boto:Params: {} 2024-04-12T03:52:19.908 DEBUG:boto:Token: None 2024-04-12T03:52:19.908 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:19 GMT /dkyges-57/ 2024-04-12T03:52:19.908 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qEnChDTCyFDPGew7fIvwxBdd6uk= 2024-04-12T03:52:19.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qEnChDTCyFDPGew7fIvwxBdd6uk=', 'Content-Length': '0'} 2024-04-12T03:52:19.910 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dffdeeb8b09322b9-006618aff3-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:19.911 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:52:19.911 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:19.911 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:19.911 DEBUG:boto:path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:52:19.911 DEBUG:boto:auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:52:19.911 DEBUG:boto:Method: GET 2024-04-12T03:52:19.911 DEBUG:boto:Path: /dkyges-57/?versions&max-keys=999 2024-04-12T03:52:19.911 DEBUG:boto:Data: 2024-04-12T03:52:19.911 DEBUG:boto:Headers: {} 2024-04-12T03:52:19.911 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:19.911 DEBUG:boto:Port: 8001 2024-04-12T03:52:19.911 DEBUG:boto:Params: {} 2024-04-12T03:52:19.912 DEBUG:boto:Token: None 2024-04-12T03:52:19.912 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:19 GMT /dkyges-57/?versions 2024-04-12T03:52:19.912 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mMI7AYg6pH3d6cAPfLYn8jGhs2o= 2024-04-12T03:52:19.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mMI7AYg6pH3d6cAPfLYn8jGhs2o=', 'Content-Length': '0'} 2024-04-12T03:52:19.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf0aaa5d4e467c63-006618aff3-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:19.914 DEBUG:boto:b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:19.915 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:52:19.915 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:52:19.915 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:52:19.915 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:52:19.915 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-12T03:52:19.915 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-12T03:52:19.916 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-12T03:52:19.916 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-12T03:52:19.916 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:52:19.916 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:19.916 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:19.916 DEBUG:boto:path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:52:19.916 DEBUG:boto:auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:52:19.916 DEBUG:boto:Method: GET 2024-04-12T03:52:19.916 DEBUG:boto:Path: /dkyges-57/?versions&max-keys=999 2024-04-12T03:52:19.916 DEBUG:boto:Data: 2024-04-12T03:52:19.916 DEBUG:boto:Headers: {} 2024-04-12T03:52:19.916 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:19.917 DEBUG:boto:Port: 8000 2024-04-12T03:52:19.917 DEBUG:boto:Params: {} 2024-04-12T03:52:19.917 DEBUG:boto:Token: None 2024-04-12T03:52:19.917 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:19 GMT /dkyges-57/?versions 2024-04-12T03:52:19.917 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mMI7AYg6pH3d6cAPfLYn8jGhs2o= 2024-04-12T03:52:19.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mMI7AYg6pH3d6cAPfLYn8jGhs2o=', 'Content-Length': '0'} 2024-04-12T03:52:19.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008036eeadac0615b5-006618aff3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:19.919 DEBUG:boto:b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:19.920 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:52:19.920 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:52:19.920 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:52:19.920 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:52:19.920 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-12T03:52:19.921 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-12T03:52:19.921 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-12T03:52:19.921 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-12T03:52:19.921 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:19.921 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:19.921 DEBUG:boto:path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:52:19.921 DEBUG:boto:auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:52:19.921 DEBUG:boto:Method: GET 2024-04-12T03:52:19.921 DEBUG:boto:Path: /dkyges-57/?versions&max-keys=999 2024-04-12T03:52:19.921 DEBUG:boto:Data: 2024-04-12T03:52:19.921 DEBUG:boto:Headers: {} 2024-04-12T03:52:19.922 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:19.922 DEBUG:boto:Port: 8001 2024-04-12T03:52:19.922 DEBUG:boto:Params: {} 2024-04-12T03:52:19.922 DEBUG:boto:Token: None 2024-04-12T03:52:19.922 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:19 GMT /dkyges-57/?versions 2024-04-12T03:52:19.922 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mMI7AYg6pH3d6cAPfLYn8jGhs2o= 2024-04-12T03:52:19.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mMI7AYg6pH3d6cAPfLYn8jGhs2o=', 'Content-Length': '0'} 2024-04-12T03:52:19.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d128b873cc2c1490-006618aff3-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:19.925 DEBUG:boto:b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:19.925 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:19.926 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:19.926 DEBUG:boto:path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:52:19.926 DEBUG:boto:auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T03:52:19.926 DEBUG:boto:Method: GET 2024-04-12T03:52:19.926 DEBUG:boto:Path: /dkyges-57/?versions&max-keys=999 2024-04-12T03:52:19.926 DEBUG:boto:Data: 2024-04-12T03:52:19.926 DEBUG:boto:Headers: {} 2024-04-12T03:52:19.926 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:19.926 DEBUG:boto:Port: 8000 2024-04-12T03:52:19.926 DEBUG:boto:Params: {} 2024-04-12T03:52:19.927 DEBUG:boto:Token: None 2024-04-12T03:52:19.927 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:19 GMT /dkyges-57/?versions 2024-04-12T03:52:19.927 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mMI7AYg6pH3d6cAPfLYn8jGhs2o= 2024-04-12T03:52:19.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mMI7AYg6pH3d6cAPfLYn8jGhs2o=', 'Content-Length': '0'} 2024-04-12T03:52:19.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b8838d04a738c12-006618aff3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:19.929 DEBUG:boto:b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:19.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:52:19.929 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:19.930 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:19.930 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:52:19.930 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:52:19.930 DEBUG:boto:Method: GET 2024-04-12T03:52:19.930 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:52:19.930 DEBUG:boto:Data: 2024-04-12T03:52:19.930 DEBUG:boto:Headers: {} 2024-04-12T03:52:19.930 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:19.930 DEBUG:boto:Port: 8001 2024-04-12T03:52:19.930 DEBUG:boto:Params: {} 2024-04-12T03:52:19.931 DEBUG:boto:Token: None 2024-04-12T03:52:19.931 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:19 GMT /dkyges-57/a?versionId=null 2024-04-12T03:52:19.931 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Xkz/s9cbmNYHIDg02kUK1WFOG5I= 2024-04-12T03:52:19.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Xkz/s9cbmNYHIDg02kUK1WFOG5I=', 'Content-Length': '0'} 2024-04-12T03:52:19.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0f9279a074bf2d2-006618aff3-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:19.974 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:19.975 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:19.975 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:52:19.975 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:52:19.975 DEBUG:boto:Method: GET 2024-04-12T03:52:19.975 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:52:19.975 DEBUG:boto:Data: 2024-04-12T03:52:19.975 DEBUG:boto:Headers: {} 2024-04-12T03:52:19.975 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:19.975 DEBUG:boto:Port: 8000 2024-04-12T03:52:19.975 DEBUG:boto:Params: {} 2024-04-12T03:52:19.975 DEBUG:boto:Token: None 2024-04-12T03:52:19.976 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:19 GMT /dkyges-57/a?versionId=null 2024-04-12T03:52:19.976 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Xkz/s9cbmNYHIDg02kUK1WFOG5I= 2024-04-12T03:52:19.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Xkz/s9cbmNYHIDg02kUK1WFOG5I=', 'Content-Length': '0'} 2024-04-12T03:52:19.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0dc7d0413b88578-006618aff3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:19.978 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:19.979 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:19.979 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:52:19.979 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:52:19.979 DEBUG:boto:Method: HEAD 2024-04-12T03:52:19.979 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:52:19.979 DEBUG:boto:Data: 2024-04-12T03:52:19.979 DEBUG:boto:Headers: {} 2024-04-12T03:52:19.979 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:19.979 DEBUG:boto:Port: 8001 2024-04-12T03:52:19.979 DEBUG:boto:Params: {} 2024-04-12T03:52:19.979 DEBUG:boto:Token: None 2024-04-12T03:52:19.980 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:19 GMT /dkyges-57/a?versionId=null 2024-04-12T03:52:19.980 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nKDihOxaoLKZF6Z+AacKBolUg3M= 2024-04-12T03:52:19.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nKDihOxaoLKZF6Z+AacKBolUg3M=', 'Content-Length': '0'} 2024-04-12T03:52:19.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095e02ae288c0b23f-006618aff3-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:19.982 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:19.982 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:19.982 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:52:19.982 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:52:19.982 DEBUG:boto:Method: HEAD 2024-04-12T03:52:19.982 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:52:19.983 DEBUG:boto:Data: 2024-04-12T03:52:19.983 DEBUG:boto:Headers: {} 2024-04-12T03:52:19.983 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:19.983 DEBUG:boto:Port: 8000 2024-04-12T03:52:19.983 DEBUG:boto:Params: {} 2024-04-12T03:52:19.983 DEBUG:boto:Token: None 2024-04-12T03:52:19.983 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:19 GMT /dkyges-57/a?versionId=null 2024-04-12T03:52:19.983 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nKDihOxaoLKZF6Z+AacKBolUg3M= 2024-04-12T03:52:19.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nKDihOxaoLKZF6Z+AacKBolUg3M=', 'Content-Length': '0'} 2024-04-12T03:52:19.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000920ce556d11b03a-006618aff3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:19.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:52:19.985 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:19.985 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:19.985 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:52:19.985 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:52:19.986 DEBUG:boto:Method: GET 2024-04-12T03:52:19.986 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:52:19.986 DEBUG:boto:Data: 2024-04-12T03:52:19.986 DEBUG:boto:Headers: {} 2024-04-12T03:52:19.986 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:19.986 DEBUG:boto:Port: 8001 2024-04-12T03:52:19.986 DEBUG:boto:Params: {} 2024-04-12T03:52:19.986 DEBUG:boto:Token: None 2024-04-12T03:52:19.986 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:19 GMT /dkyges-57/a?versionId=null 2024-04-12T03:52:19.986 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Xkz/s9cbmNYHIDg02kUK1WFOG5I= 2024-04-12T03:52:19.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Xkz/s9cbmNYHIDg02kUK1WFOG5I=', 'Content-Length': '0'} 2024-04-12T03:52:19.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028821163447f5c76-006618aff3-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:19.988 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:19.989 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:19.989 DEBUG:boto:path=/dkyges-57/a?versionId=null 2024-04-12T03:52:19.989 DEBUG:boto:auth_path=/dkyges-57/a?versionId=null 2024-04-12T03:52:19.989 DEBUG:boto:Method: GET 2024-04-12T03:52:19.989 DEBUG:boto:Path: /dkyges-57/a?versionId=null 2024-04-12T03:52:19.989 DEBUG:boto:Data: 2024-04-12T03:52:19.989 DEBUG:boto:Headers: {} 2024-04-12T03:52:19.989 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:19.989 DEBUG:boto:Port: 8000 2024-04-12T03:52:19.989 DEBUG:boto:Params: {} 2024-04-12T03:52:19.990 DEBUG:boto:Token: None 2024-04-12T03:52:19.990 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:19 GMT /dkyges-57/a?versionId=null 2024-04-12T03:52:19.990 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Xkz/s9cbmNYHIDg02kUK1WFOG5I= 2024-04-12T03:52:19.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Xkz/s9cbmNYHIDg02kUK1WFOG5I=', 'Content-Length': '0'} 2024-04-12T03:52:19.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000babaa56afc9c5ea7-006618aff3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:19.992 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:19.992 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:19.992 DEBUG:boto:path=/dkyges-57/?prefix=a 2024-04-12T03:52:19.992 DEBUG:boto:auth_path=/dkyges-57/?prefix=a 2024-04-12T03:52:19.992 DEBUG:boto:Method: GET 2024-04-12T03:52:19.992 DEBUG:boto:Path: /dkyges-57/?prefix=a 2024-04-12T03:52:19.992 DEBUG:boto:Data: 2024-04-12T03:52:19.992 DEBUG:boto:Headers: {} 2024-04-12T03:52:19.993 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:19.993 DEBUG:boto:Port: 8001 2024-04-12T03:52:19.993 DEBUG:boto:Params: {} 2024-04-12T03:52:19.993 DEBUG:boto:Token: None 2024-04-12T03:52:19.993 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:19 GMT /dkyges-57/ 2024-04-12T03:52:19.993 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KOkL/T/PQrrhxIRPfS5JnavPqEc= 2024-04-12T03:52:19.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KOkL/T/PQrrhxIRPfS5JnavPqEc=', 'Content-Length': '0'} 2024-04-12T03:52:19.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1ff87e4966aef80-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:19.996 DEBUG:boto:b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:19.996 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:19.996 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:19.996 DEBUG:boto:path=/dkyges-57/?prefix=a 2024-04-12T03:52:19.997 DEBUG:boto:auth_path=/dkyges-57/?prefix=a 2024-04-12T03:52:19.997 DEBUG:boto:Method: GET 2024-04-12T03:52:19.997 DEBUG:boto:Path: /dkyges-57/?prefix=a 2024-04-12T03:52:19.997 DEBUG:boto:Data: 2024-04-12T03:52:19.997 DEBUG:boto:Headers: {} 2024-04-12T03:52:19.997 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:19.997 DEBUG:boto:Port: 8000 2024-04-12T03:52:19.997 DEBUG:boto:Params: {} 2024-04-12T03:52:19.997 DEBUG:boto:Token: None 2024-04-12T03:52:19.997 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:19 GMT /dkyges-57/ 2024-04-12T03:52:19.998 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KOkL/T/PQrrhxIRPfS5JnavPqEc= 2024-04-12T03:52:19.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KOkL/T/PQrrhxIRPfS5JnavPqEc=', 'Content-Length': '0'} 2024-04-12T03:52:19.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc54c86dd3160ec2-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.000 DEBUG:boto:b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.000 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:20.000 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:20.000 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.000 DEBUG:boto:Path: /dkyges-57/a 2024-04-12T03:52:20.000 DEBUG:boto:Data: 2024-04-12T03:52:20.001 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.001 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.001 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.001 DEBUG:boto:Params: {} 2024-04-12T03:52:20.001 DEBUG:boto:Token: None 2024-04-12T03:52:20.001 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/a 2024-04-12T03:52:20.001 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KkpdsuQx4uMpmRRp7sMKiiYLzgg= 2024-04-12T03:52:20.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KkpdsuQx4uMpmRRp7sMKiiYLzgg=', 'Content-Length': '0'} 2024-04-12T03:52:20.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e493c780b1d2293d-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.003 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:20.003 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:20.003 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.003 DEBUG:boto:Path: /dkyges-57/a 2024-04-12T03:52:20.003 DEBUG:boto:Data: 2024-04-12T03:52:20.004 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.004 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.004 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.004 DEBUG:boto:Params: {} 2024-04-12T03:52:20.004 DEBUG:boto:Token: None 2024-04-12T03:52:20.004 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/a 2024-04-12T03:52:20.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KkpdsuQx4uMpmRRp7sMKiiYLzgg= 2024-04-12T03:52:20.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KkpdsuQx4uMpmRRp7sMKiiYLzgg=', 'Content-Length': '0'} 2024-04-12T03:52:20.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078c30aa1b6d3bcf1-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:52:20.006 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:20.006 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:20.007 DEBUG:boto:Method: GET 2024-04-12T03:52:20.007 DEBUG:boto:Path: /dkyges-57/a 2024-04-12T03:52:20.007 DEBUG:boto:Data: 2024-04-12T03:52:20.007 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.007 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.007 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.007 DEBUG:boto:Params: {} 2024-04-12T03:52:20.007 DEBUG:boto:Token: None 2024-04-12T03:52:20.007 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/a 2024-04-12T03:52:20.007 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1GCpXGiP0W7VTDTktHVbK93Nuyw= 2024-04-12T03:52:20.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1GCpXGiP0W7VTDTktHVbK93Nuyw=', 'Content-Length': '0'} 2024-04-12T03:52:20.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000268f0183531d23b6-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.050 DEBUG:boto:path=/dkyges-57/a 2024-04-12T03:52:20.050 DEBUG:boto:auth_path=/dkyges-57/a 2024-04-12T03:52:20.051 DEBUG:boto:Method: GET 2024-04-12T03:52:20.051 DEBUG:boto:Path: /dkyges-57/a 2024-04-12T03:52:20.051 DEBUG:boto:Data: 2024-04-12T03:52:20.051 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.051 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.051 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.051 DEBUG:boto:Params: {} 2024-04-12T03:52:20.051 DEBUG:boto:Token: None 2024-04-12T03:52:20.051 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/a 2024-04-12T03:52:20.051 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1GCpXGiP0W7VTDTktHVbK93Nuyw= 2024-04-12T03:52:20.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1GCpXGiP0W7VTDTktHVbK93Nuyw=', 'Content-Length': '0'} 2024-04-12T03:52:20.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000972c19ba43ec4604-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.054 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.054 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.054 DEBUG:boto:path=/dkyges-57/?prefix=a 2024-04-12T03:52:20.054 DEBUG:boto:auth_path=/dkyges-57/?prefix=a 2024-04-12T03:52:20.054 DEBUG:boto:Method: GET 2024-04-12T03:52:20.054 DEBUG:boto:Path: /dkyges-57/?prefix=a 2024-04-12T03:52:20.054 DEBUG:boto:Data: 2024-04-12T03:52:20.054 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.055 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.055 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.055 DEBUG:boto:Params: {} 2024-04-12T03:52:20.055 DEBUG:boto:Token: None 2024-04-12T03:52:20.055 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.055 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2adf8ef6cafec52-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.058 DEBUG:boto:b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.058 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.058 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.058 DEBUG:boto:path=/dkyges-57/?prefix=a 2024-04-12T03:52:20.059 DEBUG:boto:auth_path=/dkyges-57/?prefix=a 2024-04-12T03:52:20.059 DEBUG:boto:Method: GET 2024-04-12T03:52:20.059 DEBUG:boto:Path: /dkyges-57/?prefix=a 2024-04-12T03:52:20.059 DEBUG:boto:Data: 2024-04-12T03:52:20.059 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.059 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.059 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.059 DEBUG:boto:Params: {} 2024-04-12T03:52:20.059 DEBUG:boto:Token: None 2024-04-12T03:52:20.059 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.059 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbcf86377bae0060-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.062 DEBUG:boto:b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:52:20.062 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:20.063 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:20.063 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:52:20.063 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:52:20.063 DEBUG:boto:Method: GET 2024-04-12T03:52:20.063 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:52:20.063 DEBUG:boto:Data: 2024-04-12T03:52:20.063 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.063 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.063 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.063 DEBUG:boto:Params: {} 2024-04-12T03:52:20.063 DEBUG:boto:Token: None 2024-04-12T03:52:20.063 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/b?versionId=null 2024-04-12T03:52:20.064 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dbnoRv6wiB0iGnTS2bdeY9W4QcU= 2024-04-12T03:52:20.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dbnoRv6wiB0iGnTS2bdeY9W4QcU=', 'Content-Length': '0'} 2024-04-12T03:52:20.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014a3213af12922df-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.066 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:20.066 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:20.066 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:52:20.066 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:52:20.066 DEBUG:boto:Method: GET 2024-04-12T03:52:20.066 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:52:20.066 DEBUG:boto:Data: 2024-04-12T03:52:20.066 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.066 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.067 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.067 DEBUG:boto:Params: {} 2024-04-12T03:52:20.067 DEBUG:boto:Token: None 2024-04-12T03:52:20.067 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/b?versionId=null 2024-04-12T03:52:20.067 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dbnoRv6wiB0iGnTS2bdeY9W4QcU= 2024-04-12T03:52:20.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dbnoRv6wiB0iGnTS2bdeY9W4QcU=', 'Content-Length': '0'} 2024-04-12T03:52:20.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a4815f27c12d2cc-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.069 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:20.069 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:20.069 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:52:20.070 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:52:20.070 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.070 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:52:20.070 DEBUG:boto:Data: 2024-04-12T03:52:20.070 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.070 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.070 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.070 DEBUG:boto:Params: {} 2024-04-12T03:52:20.070 DEBUG:boto:Token: None 2024-04-12T03:52:20.070 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/b?versionId=null 2024-04-12T03:52:20.071 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5TZQNWKJuZQooCkBzhIJgJLZsnE= 2024-04-12T03:52:20.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5TZQNWKJuZQooCkBzhIJgJLZsnE=', 'Content-Length': '0'} 2024-04-12T03:52:20.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b60a3943dac86d-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.073 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:20.073 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:20.073 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:52:20.073 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:52:20.073 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.073 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:52:20.073 DEBUG:boto:Data: 2024-04-12T03:52:20.073 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.073 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.073 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.073 DEBUG:boto:Params: {} 2024-04-12T03:52:20.074 DEBUG:boto:Token: None 2024-04-12T03:52:20.074 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/b?versionId=null 2024-04-12T03:52:20.074 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5TZQNWKJuZQooCkBzhIJgJLZsnE= 2024-04-12T03:52:20.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5TZQNWKJuZQooCkBzhIJgJLZsnE=', 'Content-Length': '0'} 2024-04-12T03:52:20.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ee86ee63ff0116c-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:52:20.076 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:20.076 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:20.076 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:52:20.076 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:52:20.076 DEBUG:boto:Method: GET 2024-04-12T03:52:20.076 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:52:20.076 DEBUG:boto:Data: 2024-04-12T03:52:20.077 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.077 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.077 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.077 DEBUG:boto:Params: {} 2024-04-12T03:52:20.077 DEBUG:boto:Token: None 2024-04-12T03:52:20.077 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/b?versionId=null 2024-04-12T03:52:20.077 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dbnoRv6wiB0iGnTS2bdeY9W4QcU= 2024-04-12T03:52:20.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dbnoRv6wiB0iGnTS2bdeY9W4QcU=', 'Content-Length': '0'} 2024-04-12T03:52:20.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbaf22a75f5e889f-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.079 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:20.079 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:20.079 DEBUG:boto:path=/dkyges-57/b?versionId=null 2024-04-12T03:52:20.080 DEBUG:boto:auth_path=/dkyges-57/b?versionId=null 2024-04-12T03:52:20.080 DEBUG:boto:Method: GET 2024-04-12T03:52:20.080 DEBUG:boto:Path: /dkyges-57/b?versionId=null 2024-04-12T03:52:20.080 DEBUG:boto:Data: 2024-04-12T03:52:20.080 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.080 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.080 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.080 DEBUG:boto:Params: {} 2024-04-12T03:52:20.080 DEBUG:boto:Token: None 2024-04-12T03:52:20.080 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/b?versionId=null 2024-04-12T03:52:20.080 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dbnoRv6wiB0iGnTS2bdeY9W4QcU= 2024-04-12T03:52:20.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dbnoRv6wiB0iGnTS2bdeY9W4QcU=', 'Content-Length': '0'} 2024-04-12T03:52:20.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ea5707ea463f335-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.083 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.083 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.083 DEBUG:boto:path=/dkyges-57/?prefix=b 2024-04-12T03:52:20.083 DEBUG:boto:auth_path=/dkyges-57/?prefix=b 2024-04-12T03:52:20.083 DEBUG:boto:Method: GET 2024-04-12T03:52:20.083 DEBUG:boto:Path: /dkyges-57/?prefix=b 2024-04-12T03:52:20.083 DEBUG:boto:Data: 2024-04-12T03:52:20.083 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.083 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.084 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.084 DEBUG:boto:Params: {} 2024-04-12T03:52:20.084 DEBUG:boto:Token: None 2024-04-12T03:52:20.084 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.084 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d96a5e4bc7039578-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.087 DEBUG:boto:b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.087 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.087 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.087 DEBUG:boto:path=/dkyges-57/?prefix=b 2024-04-12T03:52:20.087 DEBUG:boto:auth_path=/dkyges-57/?prefix=b 2024-04-12T03:52:20.087 DEBUG:boto:Method: GET 2024-04-12T03:52:20.087 DEBUG:boto:Path: /dkyges-57/?prefix=b 2024-04-12T03:52:20.088 DEBUG:boto:Data: 2024-04-12T03:52:20.088 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.088 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.088 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.088 DEBUG:boto:Params: {} 2024-04-12T03:52:20.088 DEBUG:boto:Token: None 2024-04-12T03:52:20.088 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.088 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa28ea791fd26db4-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.091 DEBUG:boto:b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.091 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:20.091 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:20.091 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.091 DEBUG:boto:Path: /dkyges-57/b 2024-04-12T03:52:20.092 DEBUG:boto:Data: 2024-04-12T03:52:20.092 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.092 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.092 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.092 DEBUG:boto:Params: {} 2024-04-12T03:52:20.092 DEBUG:boto:Token: None 2024-04-12T03:52:20.092 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/b 2024-04-12T03:52:20.092 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LJxDmkET03W9TRCVEi2ze3LOgbk= 2024-04-12T03:52:20.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LJxDmkET03W9TRCVEi2ze3LOgbk=', 'Content-Length': '0'} 2024-04-12T03:52:20.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072db870c89922d2b-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.094 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:20.094 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:20.095 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.095 DEBUG:boto:Path: /dkyges-57/b 2024-04-12T03:52:20.095 DEBUG:boto:Data: 2024-04-12T03:52:20.095 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.095 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.095 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.095 DEBUG:boto:Params: {} 2024-04-12T03:52:20.095 DEBUG:boto:Token: None 2024-04-12T03:52:20.095 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/b 2024-04-12T03:52:20.095 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LJxDmkET03W9TRCVEi2ze3LOgbk= 2024-04-12T03:52:20.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LJxDmkET03W9TRCVEi2ze3LOgbk=', 'Content-Length': '0'} 2024-04-12T03:52:20.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7e9e7039ad6ff78-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:52:20.098 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:20.098 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:20.098 DEBUG:boto:Method: GET 2024-04-12T03:52:20.098 DEBUG:boto:Path: /dkyges-57/b 2024-04-12T03:52:20.098 DEBUG:boto:Data: 2024-04-12T03:52:20.098 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.098 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.098 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.098 DEBUG:boto:Params: {} 2024-04-12T03:52:20.098 DEBUG:boto:Token: None 2024-04-12T03:52:20.099 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/b 2024-04-12T03:52:20.099 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SB4StwBBRjiM63/gZSQUxhhrZtA= 2024-04-12T03:52:20.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SB4StwBBRjiM63/gZSQUxhhrZtA=', 'Content-Length': '0'} 2024-04-12T03:52:20.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001122e461cef74fb-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.142 DEBUG:boto:path=/dkyges-57/b 2024-04-12T03:52:20.143 DEBUG:boto:auth_path=/dkyges-57/b 2024-04-12T03:52:20.143 DEBUG:boto:Method: GET 2024-04-12T03:52:20.143 DEBUG:boto:Path: /dkyges-57/b 2024-04-12T03:52:20.143 DEBUG:boto:Data: 2024-04-12T03:52:20.143 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.143 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.143 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.143 DEBUG:boto:Params: {} 2024-04-12T03:52:20.143 DEBUG:boto:Token: None 2024-04-12T03:52:20.143 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/b 2024-04-12T03:52:20.143 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SB4StwBBRjiM63/gZSQUxhhrZtA= 2024-04-12T03:52:20.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SB4StwBBRjiM63/gZSQUxhhrZtA=', 'Content-Length': '0'} 2024-04-12T03:52:20.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7a485b7c5ff9f57-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.146 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.146 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.146 DEBUG:boto:path=/dkyges-57/?prefix=b 2024-04-12T03:52:20.146 DEBUG:boto:auth_path=/dkyges-57/?prefix=b 2024-04-12T03:52:20.146 DEBUG:boto:Method: GET 2024-04-12T03:52:20.146 DEBUG:boto:Path: /dkyges-57/?prefix=b 2024-04-12T03:52:20.147 DEBUG:boto:Data: 2024-04-12T03:52:20.147 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.147 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.147 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.147 DEBUG:boto:Params: {} 2024-04-12T03:52:20.147 DEBUG:boto:Token: None 2024-04-12T03:52:20.147 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.147 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ae0830d27f72e37-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.150 DEBUG:boto:b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.151 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.151 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.151 DEBUG:boto:path=/dkyges-57/?prefix=b 2024-04-12T03:52:20.151 DEBUG:boto:auth_path=/dkyges-57/?prefix=b 2024-04-12T03:52:20.151 DEBUG:boto:Method: GET 2024-04-12T03:52:20.151 DEBUG:boto:Path: /dkyges-57/?prefix=b 2024-04-12T03:52:20.151 DEBUG:boto:Data: 2024-04-12T03:52:20.151 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.151 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.151 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.151 DEBUG:boto:Params: {} 2024-04-12T03:52:20.152 DEBUG:boto:Token: None 2024-04-12T03:52:20.152 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.152 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af3e174d6de225d8-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.154 DEBUG:boto:b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:52:20.155 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:20.155 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:20.155 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:52:20.155 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:52:20.155 DEBUG:boto:Method: GET 2024-04-12T03:52:20.155 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:52:20.155 DEBUG:boto:Data: 2024-04-12T03:52:20.155 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.156 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.156 DEBUG:boto:Params: {} 2024-04-12T03:52:20.156 DEBUG:boto:Token: None 2024-04-12T03:52:20.156 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/c?versionId=null 2024-04-12T03:52:20.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9MwANBy93fyvFAelbvyRqYAf+0k= 2024-04-12T03:52:20.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9MwANBy93fyvFAelbvyRqYAf+0k=', 'Content-Length': '0'} 2024-04-12T03:52:20.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000964e4b58833276cc-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.158 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:20.158 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:20.159 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:52:20.159 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:52:20.159 DEBUG:boto:Method: GET 2024-04-12T03:52:20.159 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:52:20.159 DEBUG:boto:Data: 2024-04-12T03:52:20.159 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.159 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.159 DEBUG:boto:Params: {} 2024-04-12T03:52:20.159 DEBUG:boto:Token: None 2024-04-12T03:52:20.160 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/c?versionId=null 2024-04-12T03:52:20.160 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9MwANBy93fyvFAelbvyRqYAf+0k= 2024-04-12T03:52:20.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9MwANBy93fyvFAelbvyRqYAf+0k=', 'Content-Length': '0'} 2024-04-12T03:52:20.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e014e1aef5893fec-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.162 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:20.162 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:20.162 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:52:20.162 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:52:20.162 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.162 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:52:20.162 DEBUG:boto:Data: 2024-04-12T03:52:20.162 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.163 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.163 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.163 DEBUG:boto:Params: {} 2024-04-12T03:52:20.163 DEBUG:boto:Token: None 2024-04-12T03:52:20.163 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/c?versionId=null 2024-04-12T03:52:20.163 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+mz3jgglbOhRJ1ITvrxJz1kvv7Q= 2024-04-12T03:52:20.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+mz3jgglbOhRJ1ITvrxJz1kvv7Q=', 'Content-Length': '0'} 2024-04-12T03:52:20.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040a688aa6813c80c-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.165 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:20.165 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:20.166 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:52:20.166 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:52:20.166 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.166 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:52:20.166 DEBUG:boto:Data: 2024-04-12T03:52:20.166 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.166 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.166 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.166 DEBUG:boto:Params: {} 2024-04-12T03:52:20.166 DEBUG:boto:Token: None 2024-04-12T03:52:20.166 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/c?versionId=null 2024-04-12T03:52:20.167 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+mz3jgglbOhRJ1ITvrxJz1kvv7Q= 2024-04-12T03:52:20.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+mz3jgglbOhRJ1ITvrxJz1kvv7Q=', 'Content-Length': '0'} 2024-04-12T03:52:20.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cec4d70d7ef576fb-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:52:20.169 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:20.169 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:20.169 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:52:20.169 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:52:20.169 DEBUG:boto:Method: GET 2024-04-12T03:52:20.169 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:52:20.169 DEBUG:boto:Data: 2024-04-12T03:52:20.169 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.170 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.170 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.170 DEBUG:boto:Params: {} 2024-04-12T03:52:20.170 DEBUG:boto:Token: None 2024-04-12T03:52:20.170 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/c?versionId=null 2024-04-12T03:52:20.170 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9MwANBy93fyvFAelbvyRqYAf+0k= 2024-04-12T03:52:20.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9MwANBy93fyvFAelbvyRqYAf+0k=', 'Content-Length': '0'} 2024-04-12T03:52:20.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5e8f73f4bdfbea3-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.172 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:20.172 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:20.172 DEBUG:boto:path=/dkyges-57/c?versionId=null 2024-04-12T03:52:20.172 DEBUG:boto:auth_path=/dkyges-57/c?versionId=null 2024-04-12T03:52:20.173 DEBUG:boto:Method: GET 2024-04-12T03:52:20.173 DEBUG:boto:Path: /dkyges-57/c?versionId=null 2024-04-12T03:52:20.173 DEBUG:boto:Data: 2024-04-12T03:52:20.173 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.173 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.173 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.173 DEBUG:boto:Params: {} 2024-04-12T03:52:20.173 DEBUG:boto:Token: None 2024-04-12T03:52:20.173 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/c?versionId=null 2024-04-12T03:52:20.173 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9MwANBy93fyvFAelbvyRqYAf+0k= 2024-04-12T03:52:20.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9MwANBy93fyvFAelbvyRqYAf+0k=', 'Content-Length': '0'} 2024-04-12T03:52:20.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000587cce71f1b96450-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.176 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.176 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.176 DEBUG:boto:path=/dkyges-57/?prefix=c 2024-04-12T03:52:20.176 DEBUG:boto:auth_path=/dkyges-57/?prefix=c 2024-04-12T03:52:20.176 DEBUG:boto:Method: GET 2024-04-12T03:52:20.176 DEBUG:boto:Path: /dkyges-57/?prefix=c 2024-04-12T03:52:20.176 DEBUG:boto:Data: 2024-04-12T03:52:20.176 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.177 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.177 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.177 DEBUG:boto:Params: {} 2024-04-12T03:52:20.177 DEBUG:boto:Token: None 2024-04-12T03:52:20.177 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.177 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1ba4ae1b42dcd2c-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.180 DEBUG:boto:b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.180 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.180 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.180 DEBUG:boto:path=/dkyges-57/?prefix=c 2024-04-12T03:52:20.180 DEBUG:boto:auth_path=/dkyges-57/?prefix=c 2024-04-12T03:52:20.181 DEBUG:boto:Method: GET 2024-04-12T03:52:20.181 DEBUG:boto:Path: /dkyges-57/?prefix=c 2024-04-12T03:52:20.181 DEBUG:boto:Data: 2024-04-12T03:52:20.181 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.181 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.181 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.181 DEBUG:boto:Params: {} 2024-04-12T03:52:20.181 DEBUG:boto:Token: None 2024-04-12T03:52:20.181 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.181 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094769600990c0178-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.184 DEBUG:boto:b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.184 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:20.184 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:20.184 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.185 DEBUG:boto:Path: /dkyges-57/c 2024-04-12T03:52:20.185 DEBUG:boto:Data: 2024-04-12T03:52:20.185 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.185 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.185 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.185 DEBUG:boto:Params: {} 2024-04-12T03:52:20.185 DEBUG:boto:Token: None 2024-04-12T03:52:20.185 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/c 2024-04-12T03:52:20.185 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uC/J7TmpxTM65GlEIGxoyQewVLU= 2024-04-12T03:52:20.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uC/J7TmpxTM65GlEIGxoyQewVLU=', 'Content-Length': '0'} 2024-04-12T03:52:20.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015857ebf0872b7ef-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.187 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:20.187 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:20.187 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.187 DEBUG:boto:Path: /dkyges-57/c 2024-04-12T03:52:20.188 DEBUG:boto:Data: 2024-04-12T03:52:20.188 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.188 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.188 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.188 DEBUG:boto:Params: {} 2024-04-12T03:52:20.188 DEBUG:boto:Token: None 2024-04-12T03:52:20.188 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/c 2024-04-12T03:52:20.188 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uC/J7TmpxTM65GlEIGxoyQewVLU= 2024-04-12T03:52:20.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uC/J7TmpxTM65GlEIGxoyQewVLU=', 'Content-Length': '0'} 2024-04-12T03:52:20.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008201f86a96b29f2e-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:52:20.191 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:20.191 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:20.191 DEBUG:boto:Method: GET 2024-04-12T03:52:20.191 DEBUG:boto:Path: /dkyges-57/c 2024-04-12T03:52:20.191 DEBUG:boto:Data: 2024-04-12T03:52:20.191 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.191 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.191 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.191 DEBUG:boto:Params: {} 2024-04-12T03:52:20.191 DEBUG:boto:Token: None 2024-04-12T03:52:20.191 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/c 2024-04-12T03:52:20.192 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:54Dh3xCO0LjtH2VC8DZRgrFpows= 2024-04-12T03:52:20.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:54Dh3xCO0LjtH2VC8DZRgrFpows=', 'Content-Length': '0'} 2024-04-12T03:52:20.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000800dfb692bb3b63e-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.234 DEBUG:boto:path=/dkyges-57/c 2024-04-12T03:52:20.235 DEBUG:boto:auth_path=/dkyges-57/c 2024-04-12T03:52:20.235 DEBUG:boto:Method: GET 2024-04-12T03:52:20.235 DEBUG:boto:Path: /dkyges-57/c 2024-04-12T03:52:20.235 DEBUG:boto:Data: 2024-04-12T03:52:20.235 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.236 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.236 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.236 DEBUG:boto:Params: {} 2024-04-12T03:52:20.236 DEBUG:boto:Token: None 2024-04-12T03:52:20.236 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/c 2024-04-12T03:52:20.237 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:54Dh3xCO0LjtH2VC8DZRgrFpows= 2024-04-12T03:52:20.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:54Dh3xCO0LjtH2VC8DZRgrFpows=', 'Content-Length': '0'} 2024-04-12T03:52:20.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000732e81fa17f63ea7-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.239 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.239 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.240 DEBUG:boto:path=/dkyges-57/?prefix=c 2024-04-12T03:52:20.240 DEBUG:boto:auth_path=/dkyges-57/?prefix=c 2024-04-12T03:52:20.240 DEBUG:boto:Method: GET 2024-04-12T03:52:20.240 DEBUG:boto:Path: /dkyges-57/?prefix=c 2024-04-12T03:52:20.240 DEBUG:boto:Data: 2024-04-12T03:52:20.240 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.240 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.241 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.241 DEBUG:boto:Params: {} 2024-04-12T03:52:20.241 DEBUG:boto:Token: None 2024-04-12T03:52:20.241 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.241 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051718274a092ad4f-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.244 DEBUG:boto:b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.245 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.245 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.245 DEBUG:boto:path=/dkyges-57/?prefix=c 2024-04-12T03:52:20.246 DEBUG:boto:auth_path=/dkyges-57/?prefix=c 2024-04-12T03:52:20.246 DEBUG:boto:Method: GET 2024-04-12T03:52:20.246 DEBUG:boto:Path: /dkyges-57/?prefix=c 2024-04-12T03:52:20.246 DEBUG:boto:Data: 2024-04-12T03:52:20.247 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.247 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.247 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.247 DEBUG:boto:Params: {} 2024-04-12T03:52:20.248 DEBUG:boto:Token: None 2024-04-12T03:52:20.248 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.248 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.251 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e0d7bf5965724c6-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.251 DEBUG:boto:b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:52:20.252 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:20.252 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:20.252 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:52:20.252 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:52:20.253 DEBUG:boto:Method: GET 2024-04-12T03:52:20.253 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:52:20.253 DEBUG:boto:Data: 2024-04-12T03:52:20.253 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.253 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.253 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.254 DEBUG:boto:Params: {} 2024-04-12T03:52:20.254 DEBUG:boto:Token: None 2024-04-12T03:52:20.254 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/d?versionId=null 2024-04-12T03:52:20.255 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R9h5vNJKcDAP88Pus1arlv7Tx/g= 2024-04-12T03:52:20.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R9h5vNJKcDAP88Pus1arlv7Tx/g=', 'Content-Length': '0'} 2024-04-12T03:52:20.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a5fbaa29c2db8d2-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.257 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:20.258 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:20.258 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:52:20.258 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:52:20.258 DEBUG:boto:Method: GET 2024-04-12T03:52:20.259 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:52:20.259 DEBUG:boto:Data: 2024-04-12T03:52:20.259 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.259 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.259 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.260 DEBUG:boto:Params: {} 2024-04-12T03:52:20.260 DEBUG:boto:Token: None 2024-04-12T03:52:20.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/d?versionId=null 2024-04-12T03:52:20.260 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R9h5vNJKcDAP88Pus1arlv7Tx/g= 2024-04-12T03:52:20.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R9h5vNJKcDAP88Pus1arlv7Tx/g=', 'Content-Length': '0'} 2024-04-12T03:52:20.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002202747e1652cc9a-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.263 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:20.263 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:20.263 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:52:20.263 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:52:20.264 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.264 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:52:20.264 DEBUG:boto:Data: 2024-04-12T03:52:20.264 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.264 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.264 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.264 DEBUG:boto:Params: {} 2024-04-12T03:52:20.265 DEBUG:boto:Token: None 2024-04-12T03:52:20.265 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/d?versionId=null 2024-04-12T03:52:20.265 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:p79gbhR3DNZxkdlWLzZCikgqqAU= 2024-04-12T03:52:20.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p79gbhR3DNZxkdlWLzZCikgqqAU=', 'Content-Length': '0'} 2024-04-12T03:52:20.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047d01d24c1136a4e-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.267 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:20.267 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:20.268 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:52:20.268 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:52:20.268 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.268 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:52:20.268 DEBUG:boto:Data: 2024-04-12T03:52:20.268 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.268 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.268 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.268 DEBUG:boto:Params: {} 2024-04-12T03:52:20.269 DEBUG:boto:Token: None 2024-04-12T03:52:20.269 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/d?versionId=null 2024-04-12T03:52:20.269 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:p79gbhR3DNZxkdlWLzZCikgqqAU= 2024-04-12T03:52:20.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p79gbhR3DNZxkdlWLzZCikgqqAU=', 'Content-Length': '0'} 2024-04-12T03:52:20.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d73b747923cd44e2-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:52:20.271 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:20.271 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:20.271 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:52:20.271 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:52:20.271 DEBUG:boto:Method: GET 2024-04-12T03:52:20.272 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:52:20.272 DEBUG:boto:Data: 2024-04-12T03:52:20.272 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.272 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.272 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.272 DEBUG:boto:Params: {} 2024-04-12T03:52:20.272 DEBUG:boto:Token: None 2024-04-12T03:52:20.272 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/d?versionId=null 2024-04-12T03:52:20.273 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R9h5vNJKcDAP88Pus1arlv7Tx/g= 2024-04-12T03:52:20.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R9h5vNJKcDAP88Pus1arlv7Tx/g=', 'Content-Length': '0'} 2024-04-12T03:52:20.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d82a83223d67fdb1-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.318 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:20.318 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:20.319 DEBUG:boto:path=/dkyges-57/d?versionId=null 2024-04-12T03:52:20.319 DEBUG:boto:auth_path=/dkyges-57/d?versionId=null 2024-04-12T03:52:20.319 DEBUG:boto:Method: GET 2024-04-12T03:52:20.319 DEBUG:boto:Path: /dkyges-57/d?versionId=null 2024-04-12T03:52:20.319 DEBUG:boto:Data: 2024-04-12T03:52:20.319 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.319 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.319 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.319 DEBUG:boto:Params: {} 2024-04-12T03:52:20.319 DEBUG:boto:Token: None 2024-04-12T03:52:20.319 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/d?versionId=null 2024-04-12T03:52:20.320 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R9h5vNJKcDAP88Pus1arlv7Tx/g= 2024-04-12T03:52:20.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R9h5vNJKcDAP88Pus1arlv7Tx/g=', 'Content-Length': '0'} 2024-04-12T03:52:20.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083c3e8a5d6b04309-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.322 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.322 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.322 DEBUG:boto:path=/dkyges-57/?prefix=d 2024-04-12T03:52:20.322 DEBUG:boto:auth_path=/dkyges-57/?prefix=d 2024-04-12T03:52:20.322 DEBUG:boto:Method: GET 2024-04-12T03:52:20.322 DEBUG:boto:Path: /dkyges-57/?prefix=d 2024-04-12T03:52:20.322 DEBUG:boto:Data: 2024-04-12T03:52:20.323 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.323 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.323 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.323 DEBUG:boto:Params: {} 2024-04-12T03:52:20.323 DEBUG:boto:Token: None 2024-04-12T03:52:20.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.323 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004585ecff20fe4c9e-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.326 DEBUG:boto:b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.326 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.327 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.327 DEBUG:boto:path=/dkyges-57/?prefix=d 2024-04-12T03:52:20.327 DEBUG:boto:auth_path=/dkyges-57/?prefix=d 2024-04-12T03:52:20.327 DEBUG:boto:Method: GET 2024-04-12T03:52:20.327 DEBUG:boto:Path: /dkyges-57/?prefix=d 2024-04-12T03:52:20.327 DEBUG:boto:Data: 2024-04-12T03:52:20.327 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.327 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.327 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.327 DEBUG:boto:Params: {} 2024-04-12T03:52:20.327 DEBUG:boto:Token: None 2024-04-12T03:52:20.328 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.328 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f2949e7e2c1d925-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.330 DEBUG:boto:b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.330 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:20.331 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:20.331 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.331 DEBUG:boto:Path: /dkyges-57/d 2024-04-12T03:52:20.331 DEBUG:boto:Data: 2024-04-12T03:52:20.331 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.331 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.331 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.331 DEBUG:boto:Params: {} 2024-04-12T03:52:20.331 DEBUG:boto:Token: None 2024-04-12T03:52:20.331 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/d 2024-04-12T03:52:20.332 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R3/mK+ze1RRTpuWqKD6GTCHw4fs= 2024-04-12T03:52:20.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R3/mK+ze1RRTpuWqKD6GTCHw4fs=', 'Content-Length': '0'} 2024-04-12T03:52:20.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e07a33f2ebebd3fc-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.333 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:20.333 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:20.334 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.334 DEBUG:boto:Path: /dkyges-57/d 2024-04-12T03:52:20.334 DEBUG:boto:Data: 2024-04-12T03:52:20.334 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.334 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.334 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.334 DEBUG:boto:Params: {} 2024-04-12T03:52:20.334 DEBUG:boto:Token: None 2024-04-12T03:52:20.334 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/d 2024-04-12T03:52:20.334 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R3/mK+ze1RRTpuWqKD6GTCHw4fs= 2024-04-12T03:52:20.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R3/mK+ze1RRTpuWqKD6GTCHw4fs=', 'Content-Length': '0'} 2024-04-12T03:52:20.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e359ef97bd9d83a1-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:52:20.337 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:20.337 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:20.337 DEBUG:boto:Method: GET 2024-04-12T03:52:20.337 DEBUG:boto:Path: /dkyges-57/d 2024-04-12T03:52:20.337 DEBUG:boto:Data: 2024-04-12T03:52:20.337 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.337 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.337 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.337 DEBUG:boto:Params: {} 2024-04-12T03:52:20.338 DEBUG:boto:Token: None 2024-04-12T03:52:20.338 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/d 2024-04-12T03:52:20.338 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5wFkjAGKT6oKfyZ7TO1AHMuO8oY= 2024-04-12T03:52:20.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5wFkjAGKT6oKfyZ7TO1AHMuO8oY=', 'Content-Length': '0'} 2024-04-12T03:52:20.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cfb426af39f59e6-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.340 DEBUG:boto:path=/dkyges-57/d 2024-04-12T03:52:20.340 DEBUG:boto:auth_path=/dkyges-57/d 2024-04-12T03:52:20.340 DEBUG:boto:Method: GET 2024-04-12T03:52:20.340 DEBUG:boto:Path: /dkyges-57/d 2024-04-12T03:52:20.340 DEBUG:boto:Data: 2024-04-12T03:52:20.340 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.340 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.341 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.341 DEBUG:boto:Params: {} 2024-04-12T03:52:20.341 DEBUG:boto:Token: None 2024-04-12T03:52:20.341 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/d 2024-04-12T03:52:20.341 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5wFkjAGKT6oKfyZ7TO1AHMuO8oY= 2024-04-12T03:52:20.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5wFkjAGKT6oKfyZ7TO1AHMuO8oY=', 'Content-Length': '0'} 2024-04-12T03:52:20.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afb951a8205d53ff-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.343 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.343 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.344 DEBUG:boto:path=/dkyges-57/?prefix=d 2024-04-12T03:52:20.344 DEBUG:boto:auth_path=/dkyges-57/?prefix=d 2024-04-12T03:52:20.344 DEBUG:boto:Method: GET 2024-04-12T03:52:20.344 DEBUG:boto:Path: /dkyges-57/?prefix=d 2024-04-12T03:52:20.344 DEBUG:boto:Data: 2024-04-12T03:52:20.344 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.344 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.344 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.344 DEBUG:boto:Params: {} 2024-04-12T03:52:20.344 DEBUG:boto:Token: None 2024-04-12T03:52:20.345 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.345 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002d123bdd5be91d0-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.347 DEBUG:boto:b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.348 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.348 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.348 DEBUG:boto:path=/dkyges-57/?prefix=d 2024-04-12T03:52:20.348 DEBUG:boto:auth_path=/dkyges-57/?prefix=d 2024-04-12T03:52:20.348 DEBUG:boto:Method: GET 2024-04-12T03:52:20.348 DEBUG:boto:Path: /dkyges-57/?prefix=d 2024-04-12T03:52:20.348 DEBUG:boto:Data: 2024-04-12T03:52:20.348 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.348 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.349 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.349 DEBUG:boto:Params: {} 2024-04-12T03:52:20.349 DEBUG:boto:Token: None 2024-04-12T03:52:20.349 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.349 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008414cce052dc2bc5-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.351 DEBUG:boto:b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:52:20.352 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:20.352 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:20.352 DEBUG:boto:path=/dkyges-57/e?versionId=null 2024-04-12T03:52:20.352 DEBUG:boto:auth_path=/dkyges-57/e?versionId=null 2024-04-12T03:52:20.352 DEBUG:boto:Method: GET 2024-04-12T03:52:20.352 DEBUG:boto:Path: /dkyges-57/e?versionId=null 2024-04-12T03:52:20.352 DEBUG:boto:Data: 2024-04-12T03:52:20.352 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.352 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.353 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.353 DEBUG:boto:Params: {} 2024-04-12T03:52:20.353 DEBUG:boto:Token: None 2024-04-12T03:52:20.353 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/e?versionId=null 2024-04-12T03:52:20.353 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P0HM4KjwepbpwfhFXwKXaBt7csE= 2024-04-12T03:52:20.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P0HM4KjwepbpwfhFXwKXaBt7csE=', 'Content-Length': '0'} 2024-04-12T03:52:20.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099e7c98e1926981a-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.355 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:20.356 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:20.356 DEBUG:boto:path=/dkyges-57/e?versionId=null 2024-04-12T03:52:20.356 DEBUG:boto:auth_path=/dkyges-57/e?versionId=null 2024-04-12T03:52:20.356 DEBUG:boto:Method: GET 2024-04-12T03:52:20.356 DEBUG:boto:Path: /dkyges-57/e?versionId=null 2024-04-12T03:52:20.356 DEBUG:boto:Data: 2024-04-12T03:52:20.356 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.356 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.356 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.356 DEBUG:boto:Params: {} 2024-04-12T03:52:20.356 DEBUG:boto:Token: None 2024-04-12T03:52:20.357 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/e?versionId=null 2024-04-12T03:52:20.357 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P0HM4KjwepbpwfhFXwKXaBt7csE= 2024-04-12T03:52:20.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P0HM4KjwepbpwfhFXwKXaBt7csE=', 'Content-Length': '0'} 2024-04-12T03:52:20.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f77123766bdfc0e-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.402 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:20.402 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:20.403 DEBUG:boto:path=/dkyges-57/e?versionId=null 2024-04-12T03:52:20.403 DEBUG:boto:auth_path=/dkyges-57/e?versionId=null 2024-04-12T03:52:20.403 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.403 DEBUG:boto:Path: /dkyges-57/e?versionId=null 2024-04-12T03:52:20.403 DEBUG:boto:Data: 2024-04-12T03:52:20.403 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.403 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.403 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.403 DEBUG:boto:Params: {} 2024-04-12T03:52:20.403 DEBUG:boto:Token: None 2024-04-12T03:52:20.404 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/e?versionId=null 2024-04-12T03:52:20.404 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0SPdO0BdTA2jYDRRj0VJntevMcM= 2024-04-12T03:52:20.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0SPdO0BdTA2jYDRRj0VJntevMcM=', 'Content-Length': '0'} 2024-04-12T03:52:20.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9621a9ba9fd04f9-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.406 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:20.406 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:20.406 DEBUG:boto:path=/dkyges-57/e?versionId=null 2024-04-12T03:52:20.406 DEBUG:boto:auth_path=/dkyges-57/e?versionId=null 2024-04-12T03:52:20.406 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.407 DEBUG:boto:Path: /dkyges-57/e?versionId=null 2024-04-12T03:52:20.407 DEBUG:boto:Data: 2024-04-12T03:52:20.407 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.407 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.407 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.407 DEBUG:boto:Params: {} 2024-04-12T03:52:20.407 DEBUG:boto:Token: None 2024-04-12T03:52:20.407 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/e?versionId=null 2024-04-12T03:52:20.407 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0SPdO0BdTA2jYDRRj0VJntevMcM= 2024-04-12T03:52:20.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0SPdO0BdTA2jYDRRj0VJntevMcM=', 'Content-Length': '0'} 2024-04-12T03:52:20.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f9a79b9aef3cd05-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:52:20.410 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:20.410 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:20.410 DEBUG:boto:path=/dkyges-57/e?versionId=null 2024-04-12T03:52:20.410 DEBUG:boto:auth_path=/dkyges-57/e?versionId=null 2024-04-12T03:52:20.410 DEBUG:boto:Method: GET 2024-04-12T03:52:20.410 DEBUG:boto:Path: /dkyges-57/e?versionId=null 2024-04-12T03:52:20.410 DEBUG:boto:Data: 2024-04-12T03:52:20.410 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.410 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.411 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.411 DEBUG:boto:Params: {} 2024-04-12T03:52:20.411 DEBUG:boto:Token: None 2024-04-12T03:52:20.411 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/e?versionId=null 2024-04-12T03:52:20.411 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P0HM4KjwepbpwfhFXwKXaBt7csE= 2024-04-12T03:52:20.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P0HM4KjwepbpwfhFXwKXaBt7csE=', 'Content-Length': '0'} 2024-04-12T03:52:20.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab6c08cbd5bdabfe-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.413 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:20.413 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:20.413 DEBUG:boto:path=/dkyges-57/e?versionId=null 2024-04-12T03:52:20.414 DEBUG:boto:auth_path=/dkyges-57/e?versionId=null 2024-04-12T03:52:20.414 DEBUG:boto:Method: GET 2024-04-12T03:52:20.414 DEBUG:boto:Path: /dkyges-57/e?versionId=null 2024-04-12T03:52:20.414 DEBUG:boto:Data: 2024-04-12T03:52:20.414 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.414 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.414 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.414 DEBUG:boto:Params: {} 2024-04-12T03:52:20.414 DEBUG:boto:Token: None 2024-04-12T03:52:20.414 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/e?versionId=null 2024-04-12T03:52:20.415 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P0HM4KjwepbpwfhFXwKXaBt7csE= 2024-04-12T03:52:20.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P0HM4KjwepbpwfhFXwKXaBt7csE=', 'Content-Length': '0'} 2024-04-12T03:52:20.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb89cc569969183d-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.417 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.417 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.417 DEBUG:boto:path=/dkyges-57/?prefix=e 2024-04-12T03:52:20.417 DEBUG:boto:auth_path=/dkyges-57/?prefix=e 2024-04-12T03:52:20.417 DEBUG:boto:Method: GET 2024-04-12T03:52:20.417 DEBUG:boto:Path: /dkyges-57/?prefix=e 2024-04-12T03:52:20.417 DEBUG:boto:Data: 2024-04-12T03:52:20.418 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.418 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.418 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.418 DEBUG:boto:Params: {} 2024-04-12T03:52:20.418 DEBUG:boto:Token: None 2024-04-12T03:52:20.418 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.418 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.421 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c897638073b0e2e3-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.421 DEBUG:boto:b'dkyges-57e1000falsee2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.421 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.422 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.422 DEBUG:boto:path=/dkyges-57/?prefix=e 2024-04-12T03:52:20.422 DEBUG:boto:auth_path=/dkyges-57/?prefix=e 2024-04-12T03:52:20.422 DEBUG:boto:Method: GET 2024-04-12T03:52:20.422 DEBUG:boto:Path: /dkyges-57/?prefix=e 2024-04-12T03:52:20.422 DEBUG:boto:Data: 2024-04-12T03:52:20.422 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.422 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.422 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.422 DEBUG:boto:Params: {} 2024-04-12T03:52:20.422 DEBUG:boto:Token: None 2024-04-12T03:52:20.423 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.423 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d500c466f6e0cc8-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.425 DEBUG:boto:b'dkyges-57e1000falsee2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.425 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:20.426 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:20.426 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.426 DEBUG:boto:Path: /dkyges-57/e 2024-04-12T03:52:20.426 DEBUG:boto:Data: 2024-04-12T03:52:20.426 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.426 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.426 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.426 DEBUG:boto:Params: {} 2024-04-12T03:52:20.426 DEBUG:boto:Token: None 2024-04-12T03:52:20.426 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/e 2024-04-12T03:52:20.426 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XUEEoyfysn9nVc7O0aw2RV6yeNk= 2024-04-12T03:52:20.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XUEEoyfysn9nVc7O0aw2RV6yeNk=', 'Content-Length': '0'} 2024-04-12T03:52:20.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049db07666e1ae97b-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.429 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:20.429 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:20.429 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.429 DEBUG:boto:Path: /dkyges-57/e 2024-04-12T03:52:20.429 DEBUG:boto:Data: 2024-04-12T03:52:20.429 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.429 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.429 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.429 DEBUG:boto:Params: {} 2024-04-12T03:52:20.429 DEBUG:boto:Token: None 2024-04-12T03:52:20.429 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/e 2024-04-12T03:52:20.430 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XUEEoyfysn9nVc7O0aw2RV6yeNk= 2024-04-12T03:52:20.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XUEEoyfysn9nVc7O0aw2RV6yeNk=', 'Content-Length': '0'} 2024-04-12T03:52:20.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff6ee3db14ab0818-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:52:20.432 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:20.432 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:20.432 DEBUG:boto:Method: GET 2024-04-12T03:52:20.432 DEBUG:boto:Path: /dkyges-57/e 2024-04-12T03:52:20.432 DEBUG:boto:Data: 2024-04-12T03:52:20.432 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.432 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.432 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.433 DEBUG:boto:Params: {} 2024-04-12T03:52:20.433 DEBUG:boto:Token: None 2024-04-12T03:52:20.433 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/e 2024-04-12T03:52:20.433 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Tj6X6BmSSLuiutQkvc73/5RjDus= 2024-04-12T03:52:20.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Tj6X6BmSSLuiutQkvc73/5RjDus=', 'Content-Length': '0'} 2024-04-12T03:52:20.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000164d31d7f2d621b-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.435 DEBUG:boto:path=/dkyges-57/e 2024-04-12T03:52:20.435 DEBUG:boto:auth_path=/dkyges-57/e 2024-04-12T03:52:20.435 DEBUG:boto:Method: GET 2024-04-12T03:52:20.435 DEBUG:boto:Path: /dkyges-57/e 2024-04-12T03:52:20.436 DEBUG:boto:Data: 2024-04-12T03:52:20.436 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.436 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.436 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.436 DEBUG:boto:Params: {} 2024-04-12T03:52:20.436 DEBUG:boto:Token: None 2024-04-12T03:52:20.436 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/e 2024-04-12T03:52:20.436 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Tj6X6BmSSLuiutQkvc73/5RjDus= 2024-04-12T03:52:20.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Tj6X6BmSSLuiutQkvc73/5RjDus=', 'Content-Length': '0'} 2024-04-12T03:52:20.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031215b224547b52a-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.438 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.438 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.439 DEBUG:boto:path=/dkyges-57/?prefix=e 2024-04-12T03:52:20.439 DEBUG:boto:auth_path=/dkyges-57/?prefix=e 2024-04-12T03:52:20.439 DEBUG:boto:Method: GET 2024-04-12T03:52:20.439 DEBUG:boto:Path: /dkyges-57/?prefix=e 2024-04-12T03:52:20.439 DEBUG:boto:Data: 2024-04-12T03:52:20.439 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.439 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.439 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.439 DEBUG:boto:Params: {} 2024-04-12T03:52:20.439 DEBUG:boto:Token: None 2024-04-12T03:52:20.439 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.440 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3d23c6bcfb424fb-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.482 DEBUG:boto:b'dkyges-57e1000falsee2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.483 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.483 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.483 DEBUG:boto:path=/dkyges-57/?prefix=e 2024-04-12T03:52:20.483 DEBUG:boto:auth_path=/dkyges-57/?prefix=e 2024-04-12T03:52:20.483 DEBUG:boto:Method: GET 2024-04-12T03:52:20.483 DEBUG:boto:Path: /dkyges-57/?prefix=e 2024-04-12T03:52:20.483 DEBUG:boto:Data: 2024-04-12T03:52:20.483 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.483 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.483 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.483 DEBUG:boto:Params: {} 2024-04-12T03:52:20.484 DEBUG:boto:Token: None 2024-04-12T03:52:20.484 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.484 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca1e0704ca48b49b-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.486 DEBUG:boto:b'dkyges-57e1000falsee2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:52:20.486 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:20.487 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:20.487 DEBUG:boto:path=/dkyges-57/f?versionId=null 2024-04-12T03:52:20.487 DEBUG:boto:auth_path=/dkyges-57/f?versionId=null 2024-04-12T03:52:20.487 DEBUG:boto:Method: GET 2024-04-12T03:52:20.487 DEBUG:boto:Path: /dkyges-57/f?versionId=null 2024-04-12T03:52:20.487 DEBUG:boto:Data: 2024-04-12T03:52:20.487 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.487 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.487 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.487 DEBUG:boto:Params: {} 2024-04-12T03:52:20.488 DEBUG:boto:Token: None 2024-04-12T03:52:20.488 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/f?versionId=null 2024-04-12T03:52:20.488 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rj0UNc3Bo+rAI7dQnnpoDLAS/gU= 2024-04-12T03:52:20.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rj0UNc3Bo+rAI7dQnnpoDLAS/gU=', 'Content-Length': '0'} 2024-04-12T03:52:20.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42d664d507c00e8-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.490 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:20.490 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:20.490 DEBUG:boto:path=/dkyges-57/f?versionId=null 2024-04-12T03:52:20.490 DEBUG:boto:auth_path=/dkyges-57/f?versionId=null 2024-04-12T03:52:20.490 DEBUG:boto:Method: GET 2024-04-12T03:52:20.490 DEBUG:boto:Path: /dkyges-57/f?versionId=null 2024-04-12T03:52:20.490 DEBUG:boto:Data: 2024-04-12T03:52:20.490 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.491 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.491 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.491 DEBUG:boto:Params: {} 2024-04-12T03:52:20.491 DEBUG:boto:Token: None 2024-04-12T03:52:20.491 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/f?versionId=null 2024-04-12T03:52:20.491 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rj0UNc3Bo+rAI7dQnnpoDLAS/gU= 2024-04-12T03:52:20.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rj0UNc3Bo+rAI7dQnnpoDLAS/gU=', 'Content-Length': '0'} 2024-04-12T03:52:20.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047555ea3ce1c5586-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.493 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:20.493 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:20.493 DEBUG:boto:path=/dkyges-57/f?versionId=null 2024-04-12T03:52:20.493 DEBUG:boto:auth_path=/dkyges-57/f?versionId=null 2024-04-12T03:52:20.494 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.494 DEBUG:boto:Path: /dkyges-57/f?versionId=null 2024-04-12T03:52:20.494 DEBUG:boto:Data: 2024-04-12T03:52:20.494 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.494 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.494 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.494 DEBUG:boto:Params: {} 2024-04-12T03:52:20.494 DEBUG:boto:Token: None 2024-04-12T03:52:20.494 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/f?versionId=null 2024-04-12T03:52:20.494 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qrG2eOKeUozlt4BmAbAJXlS4Gw0= 2024-04-12T03:52:20.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qrG2eOKeUozlt4BmAbAJXlS4Gw0=', 'Content-Length': '0'} 2024-04-12T03:52:20.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f08c9eb47de0068d-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.496 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:20.496 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:20.496 DEBUG:boto:path=/dkyges-57/f?versionId=null 2024-04-12T03:52:20.496 DEBUG:boto:auth_path=/dkyges-57/f?versionId=null 2024-04-12T03:52:20.497 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.497 DEBUG:boto:Path: /dkyges-57/f?versionId=null 2024-04-12T03:52:20.497 DEBUG:boto:Data: 2024-04-12T03:52:20.497 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.497 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.497 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.497 DEBUG:boto:Params: {} 2024-04-12T03:52:20.497 DEBUG:boto:Token: None 2024-04-12T03:52:20.497 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/f?versionId=null 2024-04-12T03:52:20.497 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qrG2eOKeUozlt4BmAbAJXlS4Gw0= 2024-04-12T03:52:20.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qrG2eOKeUozlt4BmAbAJXlS4Gw0=', 'Content-Length': '0'} 2024-04-12T03:52:20.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb6520690ad47a78-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:52:20.500 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:20.500 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:20.500 DEBUG:boto:path=/dkyges-57/f?versionId=null 2024-04-12T03:52:20.500 DEBUG:boto:auth_path=/dkyges-57/f?versionId=null 2024-04-12T03:52:20.500 DEBUG:boto:Method: GET 2024-04-12T03:52:20.500 DEBUG:boto:Path: /dkyges-57/f?versionId=null 2024-04-12T03:52:20.500 DEBUG:boto:Data: 2024-04-12T03:52:20.500 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.500 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.500 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.500 DEBUG:boto:Params: {} 2024-04-12T03:52:20.501 DEBUG:boto:Token: None 2024-04-12T03:52:20.501 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/f?versionId=null 2024-04-12T03:52:20.501 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rj0UNc3Bo+rAI7dQnnpoDLAS/gU= 2024-04-12T03:52:20.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rj0UNc3Bo+rAI7dQnnpoDLAS/gU=', 'Content-Length': '0'} 2024-04-12T03:52:20.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010f4eb00bb0d82af-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.503 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:20.503 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:20.503 DEBUG:boto:path=/dkyges-57/f?versionId=null 2024-04-12T03:52:20.503 DEBUG:boto:auth_path=/dkyges-57/f?versionId=null 2024-04-12T03:52:20.503 DEBUG:boto:Method: GET 2024-04-12T03:52:20.503 DEBUG:boto:Path: /dkyges-57/f?versionId=null 2024-04-12T03:52:20.503 DEBUG:boto:Data: 2024-04-12T03:52:20.503 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.503 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.503 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.504 DEBUG:boto:Params: {} 2024-04-12T03:52:20.504 DEBUG:boto:Token: None 2024-04-12T03:52:20.504 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/f?versionId=null 2024-04-12T03:52:20.504 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rj0UNc3Bo+rAI7dQnnpoDLAS/gU= 2024-04-12T03:52:20.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rj0UNc3Bo+rAI7dQnnpoDLAS/gU=', 'Content-Length': '0'} 2024-04-12T03:52:20.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9550af25732b103-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.546 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.546 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.547 DEBUG:boto:path=/dkyges-57/?prefix=f 2024-04-12T03:52:20.547 DEBUG:boto:auth_path=/dkyges-57/?prefix=f 2024-04-12T03:52:20.547 DEBUG:boto:Method: GET 2024-04-12T03:52:20.547 DEBUG:boto:Path: /dkyges-57/?prefix=f 2024-04-12T03:52:20.547 DEBUG:boto:Data: 2024-04-12T03:52:20.547 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.547 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.547 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.547 DEBUG:boto:Params: {} 2024-04-12T03:52:20.547 DEBUG:boto:Token: None 2024-04-12T03:52:20.547 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.548 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c4f3b64da4c8af1-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.550 DEBUG:boto:b'dkyges-57f1000falsef2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.551 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.551 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.551 DEBUG:boto:path=/dkyges-57/?prefix=f 2024-04-12T03:52:20.551 DEBUG:boto:auth_path=/dkyges-57/?prefix=f 2024-04-12T03:52:20.551 DEBUG:boto:Method: GET 2024-04-12T03:52:20.551 DEBUG:boto:Path: /dkyges-57/?prefix=f 2024-04-12T03:52:20.551 DEBUG:boto:Data: 2024-04-12T03:52:20.551 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.551 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.551 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.551 DEBUG:boto:Params: {} 2024-04-12T03:52:20.551 DEBUG:boto:Token: None 2024-04-12T03:52:20.552 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.552 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b714183cb439019d-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.554 DEBUG:boto:b'dkyges-57f1000falsef2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.554 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:20.554 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:20.555 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.555 DEBUG:boto:Path: /dkyges-57/f 2024-04-12T03:52:20.555 DEBUG:boto:Data: 2024-04-12T03:52:20.555 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.555 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.555 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.555 DEBUG:boto:Params: {} 2024-04-12T03:52:20.555 DEBUG:boto:Token: None 2024-04-12T03:52:20.555 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/f 2024-04-12T03:52:20.555 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qVjB51adJCZd5r4CjJh3EHhrKFA= 2024-04-12T03:52:20.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qVjB51adJCZd5r4CjJh3EHhrKFA=', 'Content-Length': '0'} 2024-04-12T03:52:20.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cb0e49fb0200345-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.557 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:20.557 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:20.558 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.558 DEBUG:boto:Path: /dkyges-57/f 2024-04-12T03:52:20.558 DEBUG:boto:Data: 2024-04-12T03:52:20.558 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.558 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.558 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.558 DEBUG:boto:Params: {} 2024-04-12T03:52:20.558 DEBUG:boto:Token: None 2024-04-12T03:52:20.558 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/f 2024-04-12T03:52:20.558 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qVjB51adJCZd5r4CjJh3EHhrKFA= 2024-04-12T03:52:20.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qVjB51adJCZd5r4CjJh3EHhrKFA=', 'Content-Length': '0'} 2024-04-12T03:52:20.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7da9933819973c1-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:52:20.561 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:20.561 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:20.561 DEBUG:boto:Method: GET 2024-04-12T03:52:20.561 DEBUG:boto:Path: /dkyges-57/f 2024-04-12T03:52:20.561 DEBUG:boto:Data: 2024-04-12T03:52:20.561 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.561 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.561 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.561 DEBUG:boto:Params: {} 2024-04-12T03:52:20.561 DEBUG:boto:Token: None 2024-04-12T03:52:20.561 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/f 2024-04-12T03:52:20.562 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OD5noZlc58kXco8sAsy75Alyosk= 2024-04-12T03:52:20.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OD5noZlc58kXco8sAsy75Alyosk=', 'Content-Length': '0'} 2024-04-12T03:52:20.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000787ebcd22f441030-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.564 DEBUG:boto:path=/dkyges-57/f 2024-04-12T03:52:20.564 DEBUG:boto:auth_path=/dkyges-57/f 2024-04-12T03:52:20.564 DEBUG:boto:Method: GET 2024-04-12T03:52:20.564 DEBUG:boto:Path: /dkyges-57/f 2024-04-12T03:52:20.564 DEBUG:boto:Data: 2024-04-12T03:52:20.564 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.564 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.565 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.565 DEBUG:boto:Params: {} 2024-04-12T03:52:20.565 DEBUG:boto:Token: None 2024-04-12T03:52:20.565 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/f 2024-04-12T03:52:20.565 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OD5noZlc58kXco8sAsy75Alyosk= 2024-04-12T03:52:20.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OD5noZlc58kXco8sAsy75Alyosk=', 'Content-Length': '0'} 2024-04-12T03:52:20.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000823788938ef3463f-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.567 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.567 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.567 DEBUG:boto:path=/dkyges-57/?prefix=f 2024-04-12T03:52:20.567 DEBUG:boto:auth_path=/dkyges-57/?prefix=f 2024-04-12T03:52:20.567 DEBUG:boto:Method: GET 2024-04-12T03:52:20.567 DEBUG:boto:Path: /dkyges-57/?prefix=f 2024-04-12T03:52:20.567 DEBUG:boto:Data: 2024-04-12T03:52:20.567 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.567 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.568 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.568 DEBUG:boto:Params: {} 2024-04-12T03:52:20.568 DEBUG:boto:Token: None 2024-04-12T03:52:20.568 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.568 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c82de1d157f670d-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.614 DEBUG:boto:b'dkyges-57f1000falsef2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.615 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.615 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.615 DEBUG:boto:path=/dkyges-57/?prefix=f 2024-04-12T03:52:20.615 DEBUG:boto:auth_path=/dkyges-57/?prefix=f 2024-04-12T03:52:20.615 DEBUG:boto:Method: GET 2024-04-12T03:52:20.615 DEBUG:boto:Path: /dkyges-57/?prefix=f 2024-04-12T03:52:20.615 DEBUG:boto:Data: 2024-04-12T03:52:20.615 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.616 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.616 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.616 DEBUG:boto:Params: {} 2024-04-12T03:52:20.616 DEBUG:boto:Token: None 2024-04-12T03:52:20.616 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.616 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d1e1f94b4df741f-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.618 DEBUG:boto:b'dkyges-57f1000falsef2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:52:20.619 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:20.619 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:20.619 DEBUG:boto:path=/dkyges-57/g?versionId=null 2024-04-12T03:52:20.619 DEBUG:boto:auth_path=/dkyges-57/g?versionId=null 2024-04-12T03:52:20.619 DEBUG:boto:Method: GET 2024-04-12T03:52:20.619 DEBUG:boto:Path: /dkyges-57/g?versionId=null 2024-04-12T03:52:20.619 DEBUG:boto:Data: 2024-04-12T03:52:20.619 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.619 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.620 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.620 DEBUG:boto:Params: {} 2024-04-12T03:52:20.620 DEBUG:boto:Token: None 2024-04-12T03:52:20.620 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/g?versionId=null 2024-04-12T03:52:20.620 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A8NTHgT3wEjIMhpj6w7Wic4AuiQ= 2024-04-12T03:52:20.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A8NTHgT3wEjIMhpj6w7Wic4AuiQ=', 'Content-Length': '0'} 2024-04-12T03:52:20.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f9c0b44fce0f99c-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.622 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:20.623 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:20.623 DEBUG:boto:path=/dkyges-57/g?versionId=null 2024-04-12T03:52:20.623 DEBUG:boto:auth_path=/dkyges-57/g?versionId=null 2024-04-12T03:52:20.623 DEBUG:boto:Method: GET 2024-04-12T03:52:20.623 DEBUG:boto:Path: /dkyges-57/g?versionId=null 2024-04-12T03:52:20.623 DEBUG:boto:Data: 2024-04-12T03:52:20.623 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.623 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.623 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.623 DEBUG:boto:Params: {} 2024-04-12T03:52:20.623 DEBUG:boto:Token: None 2024-04-12T03:52:20.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/g?versionId=null 2024-04-12T03:52:20.624 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A8NTHgT3wEjIMhpj6w7Wic4AuiQ= 2024-04-12T03:52:20.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A8NTHgT3wEjIMhpj6w7Wic4AuiQ=', 'Content-Length': '0'} 2024-04-12T03:52:20.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a2fd0ac299c0ae1-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.626 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:20.626 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:20.626 DEBUG:boto:path=/dkyges-57/g?versionId=null 2024-04-12T03:52:20.626 DEBUG:boto:auth_path=/dkyges-57/g?versionId=null 2024-04-12T03:52:20.626 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.626 DEBUG:boto:Path: /dkyges-57/g?versionId=null 2024-04-12T03:52:20.626 DEBUG:boto:Data: 2024-04-12T03:52:20.626 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.627 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.627 DEBUG:boto:Params: {} 2024-04-12T03:52:20.627 DEBUG:boto:Token: None 2024-04-12T03:52:20.627 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/g?versionId=null 2024-04-12T03:52:20.627 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8R3JCDN1Cvg2U7toLTSJYVQBVoY= 2024-04-12T03:52:20.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8R3JCDN1Cvg2U7toLTSJYVQBVoY=', 'Content-Length': '0'} 2024-04-12T03:52:20.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f422c3e5dc4f1f0a-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.630 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:20.630 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:20.630 DEBUG:boto:path=/dkyges-57/g?versionId=null 2024-04-12T03:52:20.630 DEBUG:boto:auth_path=/dkyges-57/g?versionId=null 2024-04-12T03:52:20.630 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.630 DEBUG:boto:Path: /dkyges-57/g?versionId=null 2024-04-12T03:52:20.630 DEBUG:boto:Data: 2024-04-12T03:52:20.630 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.630 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.630 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.630 DEBUG:boto:Params: {} 2024-04-12T03:52:20.631 DEBUG:boto:Token: None 2024-04-12T03:52:20.631 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/g?versionId=null 2024-04-12T03:52:20.631 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8R3JCDN1Cvg2U7toLTSJYVQBVoY= 2024-04-12T03:52:20.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8R3JCDN1Cvg2U7toLTSJYVQBVoY=', 'Content-Length': '0'} 2024-04-12T03:52:20.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000751fca791fd0813-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:52:20.633 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:20.634 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:20.634 DEBUG:boto:path=/dkyges-57/g?versionId=null 2024-04-12T03:52:20.634 DEBUG:boto:auth_path=/dkyges-57/g?versionId=null 2024-04-12T03:52:20.634 DEBUG:boto:Method: GET 2024-04-12T03:52:20.634 DEBUG:boto:Path: /dkyges-57/g?versionId=null 2024-04-12T03:52:20.634 DEBUG:boto:Data: 2024-04-12T03:52:20.634 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.634 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.634 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.634 DEBUG:boto:Params: {} 2024-04-12T03:52:20.634 DEBUG:boto:Token: None 2024-04-12T03:52:20.635 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/g?versionId=null 2024-04-12T03:52:20.635 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A8NTHgT3wEjIMhpj6w7Wic4AuiQ= 2024-04-12T03:52:20.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A8NTHgT3wEjIMhpj6w7Wic4AuiQ=', 'Content-Length': '0'} 2024-04-12T03:52:20.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000035f341e3130afc7-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.637 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:20.637 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:20.637 DEBUG:boto:path=/dkyges-57/g?versionId=null 2024-04-12T03:52:20.637 DEBUG:boto:auth_path=/dkyges-57/g?versionId=null 2024-04-12T03:52:20.637 DEBUG:boto:Method: GET 2024-04-12T03:52:20.637 DEBUG:boto:Path: /dkyges-57/g?versionId=null 2024-04-12T03:52:20.637 DEBUG:boto:Data: 2024-04-12T03:52:20.637 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.637 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.637 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.638 DEBUG:boto:Params: {} 2024-04-12T03:52:20.638 DEBUG:boto:Token: None 2024-04-12T03:52:20.638 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/g?versionId=null 2024-04-12T03:52:20.638 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A8NTHgT3wEjIMhpj6w7Wic4AuiQ= 2024-04-12T03:52:20.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A8NTHgT3wEjIMhpj6w7Wic4AuiQ=', 'Content-Length': '0'} 2024-04-12T03:52:20.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b4967c153515641-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.640 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.640 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.640 DEBUG:boto:path=/dkyges-57/?prefix=g 2024-04-12T03:52:20.640 DEBUG:boto:auth_path=/dkyges-57/?prefix=g 2024-04-12T03:52:20.641 DEBUG:boto:Method: GET 2024-04-12T03:52:20.641 DEBUG:boto:Path: /dkyges-57/?prefix=g 2024-04-12T03:52:20.641 DEBUG:boto:Data: 2024-04-12T03:52:20.641 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.641 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.641 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.641 DEBUG:boto:Params: {} 2024-04-12T03:52:20.641 DEBUG:boto:Token: None 2024-04-12T03:52:20.641 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.641 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3d297905e71d8f9-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.644 DEBUG:boto:b'dkyges-57g1000falseg2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.644 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.644 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.644 DEBUG:boto:path=/dkyges-57/?prefix=g 2024-04-12T03:52:20.644 DEBUG:boto:auth_path=/dkyges-57/?prefix=g 2024-04-12T03:52:20.644 DEBUG:boto:Method: GET 2024-04-12T03:52:20.645 DEBUG:boto:Path: /dkyges-57/?prefix=g 2024-04-12T03:52:20.645 DEBUG:boto:Data: 2024-04-12T03:52:20.645 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.645 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.645 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.645 DEBUG:boto:Params: {} 2024-04-12T03:52:20.645 DEBUG:boto:Token: None 2024-04-12T03:52:20.645 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.645 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb71dc1c928f96e8-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.648 DEBUG:boto:b'dkyges-57g1000falseg2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.648 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:20.648 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:20.648 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.648 DEBUG:boto:Path: /dkyges-57/g 2024-04-12T03:52:20.648 DEBUG:boto:Data: 2024-04-12T03:52:20.648 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.648 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.648 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.649 DEBUG:boto:Params: {} 2024-04-12T03:52:20.649 DEBUG:boto:Token: None 2024-04-12T03:52:20.649 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/g 2024-04-12T03:52:20.649 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ruwT++u4nyrdQZmPxKSZqgx7Wzw= 2024-04-12T03:52:20.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ruwT++u4nyrdQZmPxKSZqgx7Wzw=', 'Content-Length': '0'} 2024-04-12T03:52:20.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044eea00608dea109-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.651 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:20.651 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:20.651 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.651 DEBUG:boto:Path: /dkyges-57/g 2024-04-12T03:52:20.651 DEBUG:boto:Data: 2024-04-12T03:52:20.651 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.651 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.651 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.651 DEBUG:boto:Params: {} 2024-04-12T03:52:20.651 DEBUG:boto:Token: None 2024-04-12T03:52:20.652 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/g 2024-04-12T03:52:20.652 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ruwT++u4nyrdQZmPxKSZqgx7Wzw= 2024-04-12T03:52:20.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ruwT++u4nyrdQZmPxKSZqgx7Wzw=', 'Content-Length': '0'} 2024-04-12T03:52:20.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcb82dcb0caf9091-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:52:20.654 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:20.654 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:20.654 DEBUG:boto:Method: GET 2024-04-12T03:52:20.654 DEBUG:boto:Path: /dkyges-57/g 2024-04-12T03:52:20.654 DEBUG:boto:Data: 2024-04-12T03:52:20.654 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.655 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.655 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.655 DEBUG:boto:Params: {} 2024-04-12T03:52:20.655 DEBUG:boto:Token: None 2024-04-12T03:52:20.655 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/g 2024-04-12T03:52:20.655 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KNOSbjHn2f+DAQOKw1ABhGp3hdU= 2024-04-12T03:52:20.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KNOSbjHn2f+DAQOKw1ABhGp3hdU=', 'Content-Length': '0'} 2024-04-12T03:52:20.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d92d4e66763a8c62-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.698 DEBUG:boto:path=/dkyges-57/g 2024-04-12T03:52:20.699 DEBUG:boto:auth_path=/dkyges-57/g 2024-04-12T03:52:20.699 DEBUG:boto:Method: GET 2024-04-12T03:52:20.699 DEBUG:boto:Path: /dkyges-57/g 2024-04-12T03:52:20.699 DEBUG:boto:Data: 2024-04-12T03:52:20.699 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.699 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.699 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.699 DEBUG:boto:Params: {} 2024-04-12T03:52:20.699 DEBUG:boto:Token: None 2024-04-12T03:52:20.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/g 2024-04-12T03:52:20.700 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KNOSbjHn2f+DAQOKw1ABhGp3hdU= 2024-04-12T03:52:20.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KNOSbjHn2f+DAQOKw1ABhGp3hdU=', 'Content-Length': '0'} 2024-04-12T03:52:20.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078a81ffce8f97a23-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.702 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.702 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.702 DEBUG:boto:path=/dkyges-57/?prefix=g 2024-04-12T03:52:20.702 DEBUG:boto:auth_path=/dkyges-57/?prefix=g 2024-04-12T03:52:20.702 DEBUG:boto:Method: GET 2024-04-12T03:52:20.703 DEBUG:boto:Path: /dkyges-57/?prefix=g 2024-04-12T03:52:20.703 DEBUG:boto:Data: 2024-04-12T03:52:20.703 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.703 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.703 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.703 DEBUG:boto:Params: {} 2024-04-12T03:52:20.703 DEBUG:boto:Token: None 2024-04-12T03:52:20.703 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.703 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038eeb3ff9b1c840c-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.706 DEBUG:boto:b'dkyges-57g1000falseg2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.706 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.707 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.707 DEBUG:boto:path=/dkyges-57/?prefix=g 2024-04-12T03:52:20.707 DEBUG:boto:auth_path=/dkyges-57/?prefix=g 2024-04-12T03:52:20.707 DEBUG:boto:Method: GET 2024-04-12T03:52:20.707 DEBUG:boto:Path: /dkyges-57/?prefix=g 2024-04-12T03:52:20.707 DEBUG:boto:Data: 2024-04-12T03:52:20.707 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.707 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.707 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.707 DEBUG:boto:Params: {} 2024-04-12T03:52:20.708 DEBUG:boto:Token: None 2024-04-12T03:52:20.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.708 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dfaee78529261cc-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.710 DEBUG:boto:b'dkyges-57g1000falseg2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:52:20.711 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:20.713 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:20.713 DEBUG:boto:path=/dkyges-57/h?versionId=null 2024-04-12T03:52:20.713 DEBUG:boto:auth_path=/dkyges-57/h?versionId=null 2024-04-12T03:52:20.713 DEBUG:boto:Method: GET 2024-04-12T03:52:20.714 DEBUG:boto:Path: /dkyges-57/h?versionId=null 2024-04-12T03:52:20.714 DEBUG:boto:Data: 2024-04-12T03:52:20.714 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.714 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.714 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.714 DEBUG:boto:Params: {} 2024-04-12T03:52:20.714 DEBUG:boto:Token: None 2024-04-12T03:52:20.714 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/h?versionId=null 2024-04-12T03:52:20.714 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YxpXfG7sECYtQ4gY/nju4HSR+rM= 2024-04-12T03:52:20.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YxpXfG7sECYtQ4gY/nju4HSR+rM=', 'Content-Length': '0'} 2024-04-12T03:52:20.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049196797a15bc9de-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.717 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:20.717 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:20.717 DEBUG:boto:path=/dkyges-57/h?versionId=null 2024-04-12T03:52:20.717 DEBUG:boto:auth_path=/dkyges-57/h?versionId=null 2024-04-12T03:52:20.717 DEBUG:boto:Method: GET 2024-04-12T03:52:20.718 DEBUG:boto:Path: /dkyges-57/h?versionId=null 2024-04-12T03:52:20.718 DEBUG:boto:Data: 2024-04-12T03:52:20.718 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.718 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.718 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.718 DEBUG:boto:Params: {} 2024-04-12T03:52:20.718 DEBUG:boto:Token: None 2024-04-12T03:52:20.718 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/h?versionId=null 2024-04-12T03:52:20.718 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YxpXfG7sECYtQ4gY/nju4HSR+rM= 2024-04-12T03:52:20.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YxpXfG7sECYtQ4gY/nju4HSR+rM=', 'Content-Length': '0'} 2024-04-12T03:52:20.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000589faee354a9a20-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.721 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:20.721 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:20.721 DEBUG:boto:path=/dkyges-57/h?versionId=null 2024-04-12T03:52:20.721 DEBUG:boto:auth_path=/dkyges-57/h?versionId=null 2024-04-12T03:52:20.721 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.721 DEBUG:boto:Path: /dkyges-57/h?versionId=null 2024-04-12T03:52:20.721 DEBUG:boto:Data: 2024-04-12T03:52:20.721 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.721 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.722 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.722 DEBUG:boto:Params: {} 2024-04-12T03:52:20.722 DEBUG:boto:Token: None 2024-04-12T03:52:20.722 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/h?versionId=null 2024-04-12T03:52:20.722 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:36/V0M7cNOGzenjsTkBdPPUTCBY= 2024-04-12T03:52:20.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:36/V0M7cNOGzenjsTkBdPPUTCBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006695d56a5386a01b-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.724 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:20.724 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:20.724 DEBUG:boto:path=/dkyges-57/h?versionId=null 2024-04-12T03:52:20.724 DEBUG:boto:auth_path=/dkyges-57/h?versionId=null 2024-04-12T03:52:20.724 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.724 DEBUG:boto:Path: /dkyges-57/h?versionId=null 2024-04-12T03:52:20.725 DEBUG:boto:Data: 2024-04-12T03:52:20.725 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.725 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.725 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.725 DEBUG:boto:Params: {} 2024-04-12T03:52:20.725 DEBUG:boto:Token: None 2024-04-12T03:52:20.725 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/h?versionId=null 2024-04-12T03:52:20.725 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:36/V0M7cNOGzenjsTkBdPPUTCBY= 2024-04-12T03:52:20.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:36/V0M7cNOGzenjsTkBdPPUTCBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000818e3db3bcb300d1-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:52:20.727 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:20.728 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:20.728 DEBUG:boto:path=/dkyges-57/h?versionId=null 2024-04-12T03:52:20.728 DEBUG:boto:auth_path=/dkyges-57/h?versionId=null 2024-04-12T03:52:20.728 DEBUG:boto:Method: GET 2024-04-12T03:52:20.728 DEBUG:boto:Path: /dkyges-57/h?versionId=null 2024-04-12T03:52:20.728 DEBUG:boto:Data: 2024-04-12T03:52:20.728 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.728 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.728 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.728 DEBUG:boto:Params: {} 2024-04-12T03:52:20.728 DEBUG:boto:Token: None 2024-04-12T03:52:20.729 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/h?versionId=null 2024-04-12T03:52:20.729 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YxpXfG7sECYtQ4gY/nju4HSR+rM= 2024-04-12T03:52:20.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YxpXfG7sECYtQ4gY/nju4HSR+rM=', 'Content-Length': '0'} 2024-04-12T03:52:20.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008530c12f72f57a64-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.731 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:20.731 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:20.731 DEBUG:boto:path=/dkyges-57/h?versionId=null 2024-04-12T03:52:20.731 DEBUG:boto:auth_path=/dkyges-57/h?versionId=null 2024-04-12T03:52:20.731 DEBUG:boto:Method: GET 2024-04-12T03:52:20.732 DEBUG:boto:Path: /dkyges-57/h?versionId=null 2024-04-12T03:52:20.732 DEBUG:boto:Data: 2024-04-12T03:52:20.732 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.732 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.732 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.732 DEBUG:boto:Params: {} 2024-04-12T03:52:20.732 DEBUG:boto:Token: None 2024-04-12T03:52:20.732 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/h?versionId=null 2024-04-12T03:52:20.732 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YxpXfG7sECYtQ4gY/nju4HSR+rM= 2024-04-12T03:52:20.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YxpXfG7sECYtQ4gY/nju4HSR+rM=', 'Content-Length': '0'} 2024-04-12T03:52:20.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e45b08a697c16fff-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.735 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.735 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.735 DEBUG:boto:path=/dkyges-57/?prefix=h 2024-04-12T03:52:20.735 DEBUG:boto:auth_path=/dkyges-57/?prefix=h 2024-04-12T03:52:20.736 DEBUG:boto:Method: GET 2024-04-12T03:52:20.736 DEBUG:boto:Path: /dkyges-57/?prefix=h 2024-04-12T03:52:20.736 DEBUG:boto:Data: 2024-04-12T03:52:20.736 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.736 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.736 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.736 DEBUG:boto:Params: {} 2024-04-12T03:52:20.736 DEBUG:boto:Token: None 2024-04-12T03:52:20.736 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.736 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.739 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000641b05bb43c007fa-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.739 DEBUG:boto:b'dkyges-57h1000falseh2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.740 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.740 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.740 DEBUG:boto:path=/dkyges-57/?prefix=h 2024-04-12T03:52:20.740 DEBUG:boto:auth_path=/dkyges-57/?prefix=h 2024-04-12T03:52:20.740 DEBUG:boto:Method: GET 2024-04-12T03:52:20.740 DEBUG:boto:Path: /dkyges-57/?prefix=h 2024-04-12T03:52:20.740 DEBUG:boto:Data: 2024-04-12T03:52:20.740 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.740 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.741 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.741 DEBUG:boto:Params: {} 2024-04-12T03:52:20.741 DEBUG:boto:Token: None 2024-04-12T03:52:20.741 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.741 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caf0374995bdf865-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.743 DEBUG:boto:b'dkyges-57h1000falseh2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.743 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:20.744 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:20.744 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.744 DEBUG:boto:Path: /dkyges-57/h 2024-04-12T03:52:20.744 DEBUG:boto:Data: 2024-04-12T03:52:20.744 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.744 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.744 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.744 DEBUG:boto:Params: {} 2024-04-12T03:52:20.744 DEBUG:boto:Token: None 2024-04-12T03:52:20.745 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/h 2024-04-12T03:52:20.745 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2WUhWd1Og+bZSh2IYdGPBM0qAN4= 2024-04-12T03:52:20.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2WUhWd1Og+bZSh2IYdGPBM0qAN4=', 'Content-Length': '0'} 2024-04-12T03:52:20.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff3150f800b42795-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.747 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:20.747 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:20.747 DEBUG:boto:Method: HEAD 2024-04-12T03:52:20.747 DEBUG:boto:Path: /dkyges-57/h 2024-04-12T03:52:20.747 DEBUG:boto:Data: 2024-04-12T03:52:20.747 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.747 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.747 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.747 DEBUG:boto:Params: {} 2024-04-12T03:52:20.747 DEBUG:boto:Token: None 2024-04-12T03:52:20.747 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/h 2024-04-12T03:52:20.748 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2WUhWd1Og+bZSh2IYdGPBM0qAN4= 2024-04-12T03:52:20.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2WUhWd1Og+bZSh2IYdGPBM0qAN4=', 'Content-Length': '0'} 2024-04-12T03:52:20.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000032be85ce298c523-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:52:20.750 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:20.750 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:20.750 DEBUG:boto:Method: GET 2024-04-12T03:52:20.750 DEBUG:boto:Path: /dkyges-57/h 2024-04-12T03:52:20.750 DEBUG:boto:Data: 2024-04-12T03:52:20.750 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.750 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.751 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.751 DEBUG:boto:Params: {} 2024-04-12T03:52:20.751 DEBUG:boto:Token: None 2024-04-12T03:52:20.751 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/h 2024-04-12T03:52:20.751 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HEzcfchZ59vlqVhpnHzVcY2+Avc= 2024-04-12T03:52:20.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HEzcfchZ59vlqVhpnHzVcY2+Avc=', 'Content-Length': '0'} 2024-04-12T03:52:20.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaff7f812c1da178-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.794 DEBUG:boto:path=/dkyges-57/h 2024-04-12T03:52:20.794 DEBUG:boto:auth_path=/dkyges-57/h 2024-04-12T03:52:20.795 DEBUG:boto:Method: GET 2024-04-12T03:52:20.795 DEBUG:boto:Path: /dkyges-57/h 2024-04-12T03:52:20.795 DEBUG:boto:Data: 2024-04-12T03:52:20.795 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.795 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.795 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.795 DEBUG:boto:Params: {} 2024-04-12T03:52:20.795 DEBUG:boto:Token: None 2024-04-12T03:52:20.795 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/h 2024-04-12T03:52:20.795 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HEzcfchZ59vlqVhpnHzVcY2+Avc= 2024-04-12T03:52:20.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HEzcfchZ59vlqVhpnHzVcY2+Avc=', 'Content-Length': '0'} 2024-04-12T03:52:20.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af59f1f6b6d3d5ce-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.798 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.798 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.798 DEBUG:boto:path=/dkyges-57/?prefix=h 2024-04-12T03:52:20.798 DEBUG:boto:auth_path=/dkyges-57/?prefix=h 2024-04-12T03:52:20.798 DEBUG:boto:Method: GET 2024-04-12T03:52:20.798 DEBUG:boto:Path: /dkyges-57/?prefix=h 2024-04-12T03:52:20.798 DEBUG:boto:Data: 2024-04-12T03:52:20.798 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.798 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:20.798 DEBUG:boto:Port: 8001 2024-04-12T03:52:20.798 DEBUG:boto:Params: {} 2024-04-12T03:52:20.799 DEBUG:boto:Token: None 2024-04-12T03:52:20.799 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.799 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b6f8c8a632bda1a-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.802 DEBUG:boto:b'dkyges-57h1000falseh2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.802 DEBUG:boto:path=/dkyges-57/ 2024-04-12T03:52:20.802 DEBUG:boto:auth_path=/dkyges-57/ 2024-04-12T03:52:20.802 DEBUG:boto:path=/dkyges-57/?prefix=h 2024-04-12T03:52:20.802 DEBUG:boto:auth_path=/dkyges-57/?prefix=h 2024-04-12T03:52:20.802 DEBUG:boto:Method: GET 2024-04-12T03:52:20.802 DEBUG:boto:Path: /dkyges-57/?prefix=h 2024-04-12T03:52:20.802 DEBUG:boto:Data: 2024-04-12T03:52:20.803 DEBUG:boto:Headers: {} 2024-04-12T03:52:20.803 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:20.803 DEBUG:boto:Port: 8000 2024-04-12T03:52:20.803 DEBUG:boto:Params: {} 2024-04-12T03:52:20.803 DEBUG:boto:Token: None 2024-04-12T03:52:20.803 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:20 GMT /dkyges-57/ 2024-04-12T03:52:20.803 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T03:52:20.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T03:52:20.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8d3616b77f6e5ef-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:20.805 DEBUG:boto:b'dkyges-57h1000falseh2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:20.805 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-57 zones={a2, a1} 2024-04-12T03:52:20.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:52:20.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:52:20.806 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:52:20.860 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:52:20.860 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:52:20.924 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:52:20.932+0000 7f3330984700 -1 req 1858292394514936859 0.000999987s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-32:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.32,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-32:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.32,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:52:20.926 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:52:20.933+0000 7f3391245700 -1 req 12650732993877336180 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-24:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.24,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-24:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.24,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:52:20.976 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:52:20.984+0000 7f337520d700 -1 req 789324565558273563 0.000999987s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-25:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.25,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-25:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.25,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:52:21.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:52:21.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:52:21.088 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dkyges-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:52:21.144 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:52:21.144 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:52:21.195 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:52:21.196 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:21.196 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding": "None", 2024-04-12T03:52:21.196 INFO:teuthology.orchestra.run.smithi043.stdout: "current_index": { 2024-04-12T03:52:21.196 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 2, 2024-04-12T03:52:21.196 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:21.196 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "Normal", 2024-04-12T03:52:21.196 INFO:teuthology.orchestra.run.smithi043.stdout: "normal": { 2024-04-12T03:52:21.196 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 3, 2024-04-12T03:52:21.196 INFO:teuthology.orchestra.run.smithi043.stdout: "hash_type": "Mod" 2024-04-12T03:52:21.196 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.196 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:52:21.197 INFO:teuthology.orchestra.run.smithi043.stdout: "logs": [ 2024-04-12T03:52:21.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:52:21.197 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 1, 2024-04-12T03:52:21.197 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:21.197 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "InIndex", 2024-04-12T03:52:21.197 INFO:teuthology.orchestra.run.smithi043.stdout: "in_index": { 2024-04-12T03:52:21.197 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 1, 2024-04-12T03:52:21.197 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:21.197 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 3, 2024-04-12T03:52:21.197 INFO:teuthology.orchestra.run.smithi043.stdout: "hash_type": "Mod" 2024-04-12T03:52:21.197 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.198 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.198 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:52:21.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:52:21.198 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 2, 2024-04-12T03:52:21.198 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:21.198 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "InIndex", 2024-04-12T03:52:21.198 INFO:teuthology.orchestra.run.smithi043.stdout: "in_index": { 2024-04-12T03:52:21.198 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 2, 2024-04-12T03:52:21.198 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:21.198 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 3, 2024-04-12T03:52:21.198 INFO:teuthology.orchestra.run.smithi043.stdout: "hash_type": "Mod" 2024-04-12T03:52:21.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.199 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:52:21.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.199 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:52:21.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:52:21.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:52:21.200 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:52:21.251 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:52:21.251 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:52:21.299 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:52:21.306+0000 7f3337992700 -1 req 8728977924978192002 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-2:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.2,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-2:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.2,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:52:21.299 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:52:21.306+0000 7f33bb299700 -1 req 637262317642509595 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-21:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.21,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-21:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.21,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:52:21.299 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:52:21.307+0000 7f338ea40700 -1 req 17551905316112605668 0.000999987s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-30:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.30,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-30:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.30,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:52:21.302 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:52:21.310+0000 7f32fd11d700 -1 req 12505786100015442675 0.000999987s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-27:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.27,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-27:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.27,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:52:21.303 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:52:21.310+0000 7f3330183700 -1 req 15303215955899797734 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-1:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.1,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-1:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.1,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:52:21.305 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:52:21.313+0000 7f33b3289700 -1 req 186565308761057666 0.000999987s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-43:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.6,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-43:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.6,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:52:21.346 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:52:21.354+0000 7f33449ac700 -1 req 15481524183401078451 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-4:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.4,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-4:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.4,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:52:21.347 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:52:21.355+0000 7f3312948700 -1 req 4994374400251301308 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-38:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.1,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-38:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.1,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:52:21.388 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:52:21.396+0000 7f337ba1a700 -1 req 13232022127411770878 0.000999987s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-5:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.5,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-5:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.5,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:52:21.389 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:52:21.396+0000 7f333418b700 -1 req 16761627938963292835 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-45:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.8,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-45:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.8,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:52:21.394 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:52:21.402+0000 7f330d93e700 -1 req 3233970052148653791 0.000999987s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-28:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.28,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-28:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.28,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T03:52:21.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:52:21.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:52:21.461 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dkyges-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:52:21.514 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:52:21.514 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:52:21.564 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:52:21.565 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:21.565 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding": "None", 2024-04-12T03:52:21.565 INFO:teuthology.orchestra.run.smithi043.stdout: "current_index": { 2024-04-12T03:52:21.565 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 2, 2024-04-12T03:52:21.565 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:21.565 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "Normal", 2024-04-12T03:52:21.565 INFO:teuthology.orchestra.run.smithi043.stdout: "normal": { 2024-04-12T03:52:21.565 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 3, 2024-04-12T03:52:21.565 INFO:teuthology.orchestra.run.smithi043.stdout: "hash_type": "Mod" 2024-04-12T03:52:21.565 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.565 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:52:21.566 INFO:teuthology.orchestra.run.smithi043.stdout: "logs": [ 2024-04-12T03:52:21.566 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:52:21.566 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 1, 2024-04-12T03:52:21.566 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:21.566 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "InIndex", 2024-04-12T03:52:21.566 INFO:teuthology.orchestra.run.smithi043.stdout: "in_index": { 2024-04-12T03:52:21.566 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 1, 2024-04-12T03:52:21.566 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:21.566 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 3, 2024-04-12T03:52:21.566 INFO:teuthology.orchestra.run.smithi043.stdout: "hash_type": "Mod" 2024-04-12T03:52:21.566 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.567 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.567 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:52:21.567 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:52:21.567 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 2, 2024-04-12T03:52:21.567 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:21.567 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "InIndex", 2024-04-12T03:52:21.567 INFO:teuthology.orchestra.run.smithi043.stdout: "in_index": { 2024-04-12T03:52:21.567 INFO:teuthology.orchestra.run.smithi043.stdout: "gen": 2, 2024-04-12T03:52:21.567 INFO:teuthology.orchestra.run.smithi043.stdout: "layout": { 2024-04-12T03:52:21.567 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 3, 2024-04-12T03:52:21.567 INFO:teuthology.orchestra.run.smithi043.stdout: "hash_type": "Mod" 2024-04-12T03:52:21.568 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.568 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.568 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.568 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.568 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:52:21.568 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.568 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:52:21.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2024-04-12T03:52:21.573 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:21.573 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:21.573 DEBUG:boto:Method: PUT 2024-04-12T03:52:21.573 DEBUG:boto:Path: /dkyges-58/ 2024-04-12T03:52:21.573 DEBUG:boto:Data: 2024-04-12T03:52:21.574 DEBUG:boto:Headers: {} 2024-04-12T03:52:21.574 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:21.574 DEBUG:boto:Port: 8000 2024-04-12T03:52:21.574 DEBUG:boto:Params: {} 2024-04-12T03:52:21.574 DEBUG:boto:Token: None 2024-04-12T03:52:21.574 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:52:21 GMT /dkyges-58/ 2024-04-12T03:52:21.574 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6s1CNcMxStka1pDOD/Wxdzb5Ofw= 2024-04-12T03:52:21.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6s1CNcMxStka1pDOD/Wxdzb5Ofw=', 'Content-Length': '0'} 2024-04-12T03:52:21.591 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005dfc0c77253e280b-006618aff5-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:21.592 DEBUG:rgw_multi.tests:created bucket=dkyges-58 2024-04-12T03:52:21.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:52:21.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:52:21.592 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:52:21.650 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:52:21.650 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:52:21.690 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:52:21.705 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:52:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:52:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:52:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:52:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:52:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:52:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:52:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:52:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:52:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:52:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:52:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:52:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:52:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:52:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893941.598904_522.1", 2024-04-12T03:52:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:52:21.598904Z" 2024-04-12T03:52:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.706 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:52:21.707 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893941.598904_522.1'} 2024-04-12T03:52:21.707 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:52:21.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:52:21.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:52:21.708 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:52:21.762 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:52:21.762 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:52:21.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:21.804+0000 7f2cde2ca780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:52:21.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:21.805+0000 7f2cde2ca780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:52:21.812 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:52:21.813 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:52:21.814 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.815 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:52:21.816 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712893941.598904_522.1", 2024-04-12T03:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:52:21.598904Z", 2024-04-12T03:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:52:21.817 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:52:21Z" 2024-04-12T03:52:21.818 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:52:21.819 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712893941.598904_522.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:52:21.598904Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:52:21Z" } 2024-04-12T03:52:21.819 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893941.598904_522.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:52:21.598904Z', 'realm_epoch': 2}}] 2024-04-12T03:52:21.819 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893941.598904_522.1'} 2024-04-12T03:52:21.820 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893941.598904_522.1'} 2024-04-12T03:52:21.820 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:52:21.820 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:21.820 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:21.820 DEBUG:boto:Method: HEAD 2024-04-12T03:52:21.820 DEBUG:boto:Path: /dkyges-58/ 2024-04-12T03:52:21.820 DEBUG:boto:Data: 2024-04-12T03:52:21.820 DEBUG:boto:Headers: {} 2024-04-12T03:52:21.820 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:21.820 DEBUG:boto:Port: 8000 2024-04-12T03:52:21.820 DEBUG:boto:Params: {} 2024-04-12T03:52:21.821 DEBUG:boto:Token: None 2024-04-12T03:52:21.821 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:21 GMT /dkyges-58/ 2024-04-12T03:52:21.821 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ktQpmGuZAvCCzwd0xricOITAteY= 2024-04-12T03:52:21.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ktQpmGuZAvCCzwd0xricOITAteY=', 'Content-Length': '0'} 2024-04-12T03:52:21.823 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a70d24e003c69f8-006618aff5-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:21.824 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:52:21.824 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:52:21.824 DEBUG:boto:Method: PUT 2024-04-12T03:52:21.824 DEBUG:boto:Path: /dkyges-58/a 2024-04-12T03:52:21.824 DEBUG:boto:Data: 2024-04-12T03:52:21.824 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:52:21.824 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:21.824 DEBUG:boto:Port: 8000 2024-04-12T03:52:21.824 DEBUG:boto:Params: {} 2024-04-12T03:52:21.824 DEBUG:boto:Token: None 2024-04-12T03:52:21.825 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:52:21 GMT /dkyges-58/a 2024-04-12T03:52:21.825 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wpetMFaCbFT1wxI0NovXw/JKTV8= 2024-04-12T03:52:21.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:52:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wpetMFaCbFT1wxI0NovXw/JKTV8='} 2024-04-12T03:52:21.831 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000026eaac5d560b1bb-006618aff5-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:21.831 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:21.832 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:21.832 DEBUG:boto:Method: HEAD 2024-04-12T03:52:21.832 DEBUG:boto:Path: /dkyges-58/ 2024-04-12T03:52:21.832 DEBUG:boto:Data: 2024-04-12T03:52:21.832 DEBUG:boto:Headers: {} 2024-04-12T03:52:21.832 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:21.832 DEBUG:boto:Port: 8000 2024-04-12T03:52:21.832 DEBUG:boto:Params: {} 2024-04-12T03:52:21.832 DEBUG:boto:Token: None 2024-04-12T03:52:21.832 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:21 GMT /dkyges-58/ 2024-04-12T03:52:21.833 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ktQpmGuZAvCCzwd0xricOITAteY= 2024-04-12T03:52:21.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ktQpmGuZAvCCzwd0xricOITAteY=', 'Content-Length': '0'} 2024-04-12T03:52:21.834 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b2c1e018e1c2083-006618aff5-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:21.835 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:52:21.835 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:52:21.835 DEBUG:boto:Method: PUT 2024-04-12T03:52:21.835 DEBUG:boto:Path: /dkyges-58/b 2024-04-12T03:52:21.835 DEBUG:boto:Data: 2024-04-12T03:52:21.835 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:52:21.835 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:21.835 DEBUG:boto:Port: 8000 2024-04-12T03:52:21.835 DEBUG:boto:Params: {} 2024-04-12T03:52:21.836 DEBUG:boto:Token: None 2024-04-12T03:52:21.836 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:52:21 GMT /dkyges-58/b 2024-04-12T03:52:21.836 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jDx7EPQmh4kS17R1hXhWomBgoEc= 2024-04-12T03:52:21.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:52:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jDx7EPQmh4kS17R1hXhWomBgoEc='} 2024-04-12T03:52:21.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f1abe34cd11d487-006618aff5-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:21.879 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:21.879 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:21.879 DEBUG:boto:Method: HEAD 2024-04-12T03:52:21.879 DEBUG:boto:Path: /dkyges-58/ 2024-04-12T03:52:21.879 DEBUG:boto:Data: 2024-04-12T03:52:21.879 DEBUG:boto:Headers: {} 2024-04-12T03:52:21.879 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:21.879 DEBUG:boto:Port: 8000 2024-04-12T03:52:21.879 DEBUG:boto:Params: {} 2024-04-12T03:52:21.879 DEBUG:boto:Token: None 2024-04-12T03:52:21.879 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:21 GMT /dkyges-58/ 2024-04-12T03:52:21.880 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ktQpmGuZAvCCzwd0xricOITAteY= 2024-04-12T03:52:21.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ktQpmGuZAvCCzwd0xricOITAteY=', 'Content-Length': '0'} 2024-04-12T03:52:21.882 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021b9ac0642b3719f-006618aff5-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:21.882 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:52:21.882 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:52:21.882 DEBUG:boto:Method: PUT 2024-04-12T03:52:21.883 DEBUG:boto:Path: /dkyges-58/c 2024-04-12T03:52:21.883 DEBUG:boto:Data: 2024-04-12T03:52:21.883 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:52:21.883 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:21.883 DEBUG:boto:Port: 8000 2024-04-12T03:52:21.883 DEBUG:boto:Params: {} 2024-04-12T03:52:21.883 DEBUG:boto:Token: None 2024-04-12T03:52:21.883 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:52:21 GMT /dkyges-58/c 2024-04-12T03:52:21.883 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wZRoLlbuZ5ybJL1F8XqDZAZGY44= 2024-04-12T03:52:21.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:52:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wZRoLlbuZ5ybJL1F8XqDZAZGY44='} 2024-04-12T03:52:21.889 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e15bab490d35a4d-006618aff5-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:21.889 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:21.889 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:21.889 DEBUG:boto:Method: HEAD 2024-04-12T03:52:21.890 DEBUG:boto:Path: /dkyges-58/ 2024-04-12T03:52:21.890 DEBUG:boto:Data: 2024-04-12T03:52:21.890 DEBUG:boto:Headers: {} 2024-04-12T03:52:21.890 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:21.890 DEBUG:boto:Port: 8000 2024-04-12T03:52:21.890 DEBUG:boto:Params: {} 2024-04-12T03:52:21.890 DEBUG:boto:Token: None 2024-04-12T03:52:21.890 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:21 GMT /dkyges-58/ 2024-04-12T03:52:21.890 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ktQpmGuZAvCCzwd0xricOITAteY= 2024-04-12T03:52:21.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ktQpmGuZAvCCzwd0xricOITAteY=', 'Content-Length': '0'} 2024-04-12T03:52:21.893 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000456b5d2301b322eb-006618aff5-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:21.893 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:52:21.893 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:52:21.893 DEBUG:boto:Method: PUT 2024-04-12T03:52:21.893 DEBUG:boto:Path: /dkyges-58/d 2024-04-12T03:52:21.893 DEBUG:boto:Data: 2024-04-12T03:52:21.893 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:52:21.894 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:21.894 DEBUG:boto:Port: 8000 2024-04-12T03:52:21.894 DEBUG:boto:Params: {} 2024-04-12T03:52:21.894 DEBUG:boto:Token: None 2024-04-12T03:52:21.894 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:52:21 GMT /dkyges-58/d 2024-04-12T03:52:21.894 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UUtGTJotwYE1DeX2ZuchtOPY3VE= 2024-04-12T03:52:21.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:52:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UUtGTJotwYE1DeX2ZuchtOPY3VE='} 2024-04-12T03:52:21.900 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd03880b31a6a904-006618aff5-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:21.900 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-58 2024-04-12T03:52:21.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:52:21.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:52:21.900 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:52:21.965 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:52:21.965 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:52:22.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:22.014+0000 7fb68914a780 1 waiting to reach incremental sync.. 2024-04-12T03:52:27.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:27.015+0000 7fb68914a780 1 waiting to reach incremental sync.. 2024-04-12T03:52:32.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:32.015+0000 7fb68914a780 1 waiting to reach incremental sync.. 2024-04-12T03:52:37.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:37.017+0000 7fb68914a780 1 bucket sync caught up with source: 2024-04-12T03:52:37.010 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.42.6, , , , 00000000001.38.6, , , 00000000001.61.6, , 00000000001.45.6] 2024-04-12T03:52:37.010 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.42.6, , , , 00000000001.38.6, , , 00000000001.61.6, , 00000000001.45.6] 2024-04-12T03:52:37.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:37.017+0000 7fb68914a780 0 bucket checkpoint complete 2024-04-12T03:52:37.029 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-58 2024-04-12T03:52:37.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:52:37.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:52:37.029 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:52:37.099 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:52:37.099 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:52:37.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:37.148+0000 7f3c80240780 1 waiting to reach incremental sync.. 2024-04-12T03:52:42.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:42.149+0000 7f3c80240780 1 bucket sync caught up with source: 2024-04-12T03:52:42.142 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.1878.6, , , , 00000000001.1403.6, , , 00000000001.2586.6, , 00000000001.1881.6] 2024-04-12T03:52:42.142 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.1878.6, , , , 00000000001.1403.6, , , 00000000001.2586.6, , 00000000001.1881.6] 2024-04-12T03:52:42.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:42.149+0000 7f3c80240780 0 bucket checkpoint complete 2024-04-12T03:52:42.160 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-58 zones={a2, a1} 2024-04-12T03:52:42.160 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.160 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.161 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.161 DEBUG:boto:Path: /dkyges-58/ 2024-04-12T03:52:42.161 DEBUG:boto:Data: 2024-04-12T03:52:42.161 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.161 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.161 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.161 DEBUG:boto:Params: {} 2024-04-12T03:52:42.161 DEBUG:boto:Token: None 2024-04-12T03:52:42.162 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.162 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L2lLRSWYnjeau4OtJHskFYk4S9g= 2024-04-12T03:52:42.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L2lLRSWYnjeau4OtJHskFYk4S9g=', 'Content-Length': '0'} 2024-04-12T03:52:42.165 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000814b43a6e365259-006618b00a-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.165 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.165 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.165 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.165 DEBUG:boto:Path: /dkyges-58/ 2024-04-12T03:52:42.165 DEBUG:boto:Data: 2024-04-12T03:52:42.166 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.166 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.166 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.166 DEBUG:boto:Params: {} 2024-04-12T03:52:42.166 DEBUG:boto:Token: None 2024-04-12T03:52:42.166 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.166 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L2lLRSWYnjeau4OtJHskFYk4S9g= 2024-04-12T03:52:42.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L2lLRSWYnjeau4OtJHskFYk4S9g=', 'Content-Length': '0'} 2024-04-12T03:52:42.169 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc8660e58270e587-006618b00a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.169 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:52:42.169 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.169 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.169 DEBUG:boto:path=/dkyges-58/?versions&max-keys=999 2024-04-12T03:52:42.169 DEBUG:boto:auth_path=/dkyges-58/?versions&max-keys=999 2024-04-12T03:52:42.169 DEBUG:boto:Method: GET 2024-04-12T03:52:42.170 DEBUG:boto:Path: /dkyges-58/?versions&max-keys=999 2024-04-12T03:52:42.170 DEBUG:boto:Data: 2024-04-12T03:52:42.170 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.170 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.170 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.170 DEBUG:boto:Params: {} 2024-04-12T03:52:42.170 DEBUG:boto:Token: None 2024-04-12T03:52:42.170 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/?versions 2024-04-12T03:52:42.170 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kzmfkD6EpAnBRpSRhRwwoS4Z1ak= 2024-04-12T03:52:42.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kzmfkD6EpAnBRpSRhRwwoS4Z1ak=', 'Content-Length': '0'} 2024-04-12T03:52:42.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4d23351d508925e-006618b00a-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.173 DEBUG:boto:b'dkyges-58999falseanulltrue2024-04-12T03:52:21.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:52:21.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:52:21.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:52:21.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.174 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:52:42.174 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:52:42.174 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:52:42.174 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:52:42.174 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:52:42.175 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.175 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.175 DEBUG:boto:path=/dkyges-58/?versions&max-keys=999 2024-04-12T03:52:42.175 DEBUG:boto:auth_path=/dkyges-58/?versions&max-keys=999 2024-04-12T03:52:42.175 DEBUG:boto:Method: GET 2024-04-12T03:52:42.175 DEBUG:boto:Path: /dkyges-58/?versions&max-keys=999 2024-04-12T03:52:42.175 DEBUG:boto:Data: 2024-04-12T03:52:42.175 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.175 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.175 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.175 DEBUG:boto:Params: {} 2024-04-12T03:52:42.176 DEBUG:boto:Token: None 2024-04-12T03:52:42.176 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/?versions 2024-04-12T03:52:42.176 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kzmfkD6EpAnBRpSRhRwwoS4Z1ak= 2024-04-12T03:52:42.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kzmfkD6EpAnBRpSRhRwwoS4Z1ak=', 'Content-Length': '0'} 2024-04-12T03:52:42.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f03cfdaa277319b2-006618b00a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.178 DEBUG:boto:b'dkyges-58999falseanulltrue2024-04-12T03:52:21.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:52:21.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:52:21.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:52:21.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.179 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:52:42.179 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:52:42.179 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:52:42.179 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:52:42.179 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.179 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.179 DEBUG:boto:path=/dkyges-58/?versions&max-keys=999 2024-04-12T03:52:42.179 DEBUG:boto:auth_path=/dkyges-58/?versions&max-keys=999 2024-04-12T03:52:42.179 DEBUG:boto:Method: GET 2024-04-12T03:52:42.180 DEBUG:boto:Path: /dkyges-58/?versions&max-keys=999 2024-04-12T03:52:42.180 DEBUG:boto:Data: 2024-04-12T03:52:42.180 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.180 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.180 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.180 DEBUG:boto:Params: {} 2024-04-12T03:52:42.180 DEBUG:boto:Token: None 2024-04-12T03:52:42.180 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/?versions 2024-04-12T03:52:42.180 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kzmfkD6EpAnBRpSRhRwwoS4Z1ak= 2024-04-12T03:52:42.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kzmfkD6EpAnBRpSRhRwwoS4Z1ak=', 'Content-Length': '0'} 2024-04-12T03:52:42.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e85a1295418f5e89-006618b00a-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.183 DEBUG:boto:b'dkyges-58999falseanulltrue2024-04-12T03:52:21.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:52:21.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:52:21.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:52:21.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.183 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.184 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.184 DEBUG:boto:path=/dkyges-58/?versions&max-keys=999 2024-04-12T03:52:42.184 DEBUG:boto:auth_path=/dkyges-58/?versions&max-keys=999 2024-04-12T03:52:42.184 DEBUG:boto:Method: GET 2024-04-12T03:52:42.184 DEBUG:boto:Path: /dkyges-58/?versions&max-keys=999 2024-04-12T03:52:42.184 DEBUG:boto:Data: 2024-04-12T03:52:42.184 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.184 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.184 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.184 DEBUG:boto:Params: {} 2024-04-12T03:52:42.184 DEBUG:boto:Token: None 2024-04-12T03:52:42.185 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/?versions 2024-04-12T03:52:42.185 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kzmfkD6EpAnBRpSRhRwwoS4Z1ak= 2024-04-12T03:52:42.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kzmfkD6EpAnBRpSRhRwwoS4Z1ak=', 'Content-Length': '0'} 2024-04-12T03:52:42.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a5027257f33cd91-006618b00a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.187 DEBUG:boto:b'dkyges-58999falseanulltrue2024-04-12T03:52:21.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:52:21.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:52:21.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:52:21.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:52:42.188 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:52:42.188 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:52:42.188 DEBUG:boto:path=/dkyges-58/a?versionId=null 2024-04-12T03:52:42.188 DEBUG:boto:auth_path=/dkyges-58/a?versionId=null 2024-04-12T03:52:42.189 DEBUG:boto:Method: GET 2024-04-12T03:52:42.189 DEBUG:boto:Path: /dkyges-58/a?versionId=null 2024-04-12T03:52:42.189 DEBUG:boto:Data: 2024-04-12T03:52:42.189 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.189 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.189 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.189 DEBUG:boto:Params: {} 2024-04-12T03:52:42.189 DEBUG:boto:Token: None 2024-04-12T03:52:42.189 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/a?versionId=null 2024-04-12T03:52:42.189 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kQmZJjTzkElsN8p4dLqdBDXWmTs= 2024-04-12T03:52:42.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kQmZJjTzkElsN8p4dLqdBDXWmTs=', 'Content-Length': '0'} 2024-04-12T03:52:42.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008db79341e76db3fc-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.192 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:52:42.192 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:52:42.192 DEBUG:boto:path=/dkyges-58/a?versionId=null 2024-04-12T03:52:42.192 DEBUG:boto:auth_path=/dkyges-58/a?versionId=null 2024-04-12T03:52:42.192 DEBUG:boto:Method: GET 2024-04-12T03:52:42.192 DEBUG:boto:Path: /dkyges-58/a?versionId=null 2024-04-12T03:52:42.192 DEBUG:boto:Data: 2024-04-12T03:52:42.192 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.192 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.192 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.193 DEBUG:boto:Params: {} 2024-04-12T03:52:42.193 DEBUG:boto:Token: None 2024-04-12T03:52:42.193 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/a?versionId=null 2024-04-12T03:52:42.193 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kQmZJjTzkElsN8p4dLqdBDXWmTs= 2024-04-12T03:52:42.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kQmZJjTzkElsN8p4dLqdBDXWmTs=', 'Content-Length': '0'} 2024-04-12T03:52:42.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8fc913ef758a845-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.242 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:52:42.243 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:52:42.243 DEBUG:boto:path=/dkyges-58/a?versionId=null 2024-04-12T03:52:42.243 DEBUG:boto:auth_path=/dkyges-58/a?versionId=null 2024-04-12T03:52:42.243 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.243 DEBUG:boto:Path: /dkyges-58/a?versionId=null 2024-04-12T03:52:42.243 DEBUG:boto:Data: 2024-04-12T03:52:42.243 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.243 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.243 DEBUG:boto:Params: {} 2024-04-12T03:52:42.244 DEBUG:boto:Token: None 2024-04-12T03:52:42.244 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/a?versionId=null 2024-04-12T03:52:42.244 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pCaCkg4dKhYKXcSi0gaFH1afetY= 2024-04-12T03:52:42.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pCaCkg4dKhYKXcSi0gaFH1afetY=', 'Content-Length': '0'} 2024-04-12T03:52:42.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000323d0448bed54eee-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.246 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:52:42.246 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:52:42.246 DEBUG:boto:path=/dkyges-58/a?versionId=null 2024-04-12T03:52:42.246 DEBUG:boto:auth_path=/dkyges-58/a?versionId=null 2024-04-12T03:52:42.246 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.247 DEBUG:boto:Path: /dkyges-58/a?versionId=null 2024-04-12T03:52:42.247 DEBUG:boto:Data: 2024-04-12T03:52:42.247 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.247 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.247 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.247 DEBUG:boto:Params: {} 2024-04-12T03:52:42.247 DEBUG:boto:Token: None 2024-04-12T03:52:42.247 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/a?versionId=null 2024-04-12T03:52:42.247 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pCaCkg4dKhYKXcSi0gaFH1afetY= 2024-04-12T03:52:42.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pCaCkg4dKhYKXcSi0gaFH1afetY=', 'Content-Length': '0'} 2024-04-12T03:52:42.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d708122a49a46cd7-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:52:42.250 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:52:42.250 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:52:42.250 DEBUG:boto:path=/dkyges-58/a?versionId=null 2024-04-12T03:52:42.250 DEBUG:boto:auth_path=/dkyges-58/a?versionId=null 2024-04-12T03:52:42.250 DEBUG:boto:Method: GET 2024-04-12T03:52:42.250 DEBUG:boto:Path: /dkyges-58/a?versionId=null 2024-04-12T03:52:42.250 DEBUG:boto:Data: 2024-04-12T03:52:42.250 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.250 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.250 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.251 DEBUG:boto:Params: {} 2024-04-12T03:52:42.251 DEBUG:boto:Token: None 2024-04-12T03:52:42.251 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/a?versionId=null 2024-04-12T03:52:42.251 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kQmZJjTzkElsN8p4dLqdBDXWmTs= 2024-04-12T03:52:42.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kQmZJjTzkElsN8p4dLqdBDXWmTs=', 'Content-Length': '0'} 2024-04-12T03:52:42.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df94107fe541bad2-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.253 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:52:42.253 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:52:42.253 DEBUG:boto:path=/dkyges-58/a?versionId=null 2024-04-12T03:52:42.253 DEBUG:boto:auth_path=/dkyges-58/a?versionId=null 2024-04-12T03:52:42.254 DEBUG:boto:Method: GET 2024-04-12T03:52:42.254 DEBUG:boto:Path: /dkyges-58/a?versionId=null 2024-04-12T03:52:42.254 DEBUG:boto:Data: 2024-04-12T03:52:42.254 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.254 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.254 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.254 DEBUG:boto:Params: {} 2024-04-12T03:52:42.254 DEBUG:boto:Token: None 2024-04-12T03:52:42.254 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/a?versionId=null 2024-04-12T03:52:42.255 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kQmZJjTzkElsN8p4dLqdBDXWmTs= 2024-04-12T03:52:42.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kQmZJjTzkElsN8p4dLqdBDXWmTs=', 'Content-Length': '0'} 2024-04-12T03:52:42.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad2171e0a8f38b2b-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.256 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.257 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.257 DEBUG:boto:path=/dkyges-58/?prefix=a 2024-04-12T03:52:42.257 DEBUG:boto:auth_path=/dkyges-58/?prefix=a 2024-04-12T03:52:42.257 DEBUG:boto:Method: GET 2024-04-12T03:52:42.257 DEBUG:boto:Path: /dkyges-58/?prefix=a 2024-04-12T03:52:42.257 DEBUG:boto:Data: 2024-04-12T03:52:42.257 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.257 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.257 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.257 DEBUG:boto:Params: {} 2024-04-12T03:52:42.257 DEBUG:boto:Token: None 2024-04-12T03:52:42.258 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.258 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q= 2024-04-12T03:52:42.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4fe24aa3e3d5429-006618b00a-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.260 DEBUG:boto:b'dkyges-58a1000falsea2024-04-12T03:52:21.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.261 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.261 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.261 DEBUG:boto:path=/dkyges-58/?prefix=a 2024-04-12T03:52:42.261 DEBUG:boto:auth_path=/dkyges-58/?prefix=a 2024-04-12T03:52:42.261 DEBUG:boto:Method: GET 2024-04-12T03:52:42.261 DEBUG:boto:Path: /dkyges-58/?prefix=a 2024-04-12T03:52:42.261 DEBUG:boto:Data: 2024-04-12T03:52:42.261 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.262 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.262 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.262 DEBUG:boto:Params: {} 2024-04-12T03:52:42.262 DEBUG:boto:Token: None 2024-04-12T03:52:42.262 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.262 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q= 2024-04-12T03:52:42.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079f8bd950aebd16c-006618b00a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.265 DEBUG:boto:b'dkyges-58a1000falsea2024-04-12T03:52:21.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.265 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:52:42.265 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:52:42.265 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.265 DEBUG:boto:Path: /dkyges-58/a 2024-04-12T03:52:42.266 DEBUG:boto:Data: 2024-04-12T03:52:42.266 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.266 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.266 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.266 DEBUG:boto:Params: {} 2024-04-12T03:52:42.266 DEBUG:boto:Token: None 2024-04-12T03:52:42.266 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/a 2024-04-12T03:52:42.266 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hFGC8qNumqSMzsD6DzAVvgAIEUY= 2024-04-12T03:52:42.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hFGC8qNumqSMzsD6DzAVvgAIEUY=', 'Content-Length': '0'} 2024-04-12T03:52:42.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000201b360d3486eafc-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.268 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:52:42.269 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:52:42.269 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.269 DEBUG:boto:Path: /dkyges-58/a 2024-04-12T03:52:42.269 DEBUG:boto:Data: 2024-04-12T03:52:42.269 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.269 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.269 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.269 DEBUG:boto:Params: {} 2024-04-12T03:52:42.269 DEBUG:boto:Token: None 2024-04-12T03:52:42.270 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/a 2024-04-12T03:52:42.270 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hFGC8qNumqSMzsD6DzAVvgAIEUY= 2024-04-12T03:52:42.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hFGC8qNumqSMzsD6DzAVvgAIEUY=', 'Content-Length': '0'} 2024-04-12T03:52:42.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000002bcde57705f504-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:52:42.272 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:52:42.272 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:52:42.272 DEBUG:boto:Method: GET 2024-04-12T03:52:42.272 DEBUG:boto:Path: /dkyges-58/a 2024-04-12T03:52:42.273 DEBUG:boto:Data: 2024-04-12T03:52:42.273 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.273 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.273 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.273 DEBUG:boto:Params: {} 2024-04-12T03:52:42.273 DEBUG:boto:Token: None 2024-04-12T03:52:42.273 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/a 2024-04-12T03:52:42.273 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LxpDFM3am7wT7+oUV6UzLXovdGE= 2024-04-12T03:52:42.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LxpDFM3am7wT7+oUV6UzLXovdGE=', 'Content-Length': '0'} 2024-04-12T03:52:42.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007144cf0ef3da43c3-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.275 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:52:42.275 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:52:42.275 DEBUG:boto:Method: GET 2024-04-12T03:52:42.276 DEBUG:boto:Path: /dkyges-58/a 2024-04-12T03:52:42.276 DEBUG:boto:Data: 2024-04-12T03:52:42.276 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.276 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.276 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.276 DEBUG:boto:Params: {} 2024-04-12T03:52:42.276 DEBUG:boto:Token: None 2024-04-12T03:52:42.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/a 2024-04-12T03:52:42.276 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LxpDFM3am7wT7+oUV6UzLXovdGE= 2024-04-12T03:52:42.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LxpDFM3am7wT7+oUV6UzLXovdGE=', 'Content-Length': '0'} 2024-04-12T03:52:42.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a4e6f37dd967ce1-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.279 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.279 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.279 DEBUG:boto:path=/dkyges-58/?prefix=a 2024-04-12T03:52:42.279 DEBUG:boto:auth_path=/dkyges-58/?prefix=a 2024-04-12T03:52:42.279 DEBUG:boto:Method: GET 2024-04-12T03:52:42.279 DEBUG:boto:Path: /dkyges-58/?prefix=a 2024-04-12T03:52:42.279 DEBUG:boto:Data: 2024-04-12T03:52:42.279 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.279 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.279 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.280 DEBUG:boto:Params: {} 2024-04-12T03:52:42.280 DEBUG:boto:Token: None 2024-04-12T03:52:42.280 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.280 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q= 2024-04-12T03:52:42.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000427010330cb0127c-006618b00a-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.326 DEBUG:boto:b'dkyges-58a1000falsea2024-04-12T03:52:21.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.327 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.327 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.327 DEBUG:boto:path=/dkyges-58/?prefix=a 2024-04-12T03:52:42.327 DEBUG:boto:auth_path=/dkyges-58/?prefix=a 2024-04-12T03:52:42.327 DEBUG:boto:Method: GET 2024-04-12T03:52:42.327 DEBUG:boto:Path: /dkyges-58/?prefix=a 2024-04-12T03:52:42.327 DEBUG:boto:Data: 2024-04-12T03:52:42.328 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.328 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.328 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.328 DEBUG:boto:Params: {} 2024-04-12T03:52:42.328 DEBUG:boto:Token: None 2024-04-12T03:52:42.328 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.328 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q= 2024-04-12T03:52:42.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050bbe5917ccb29f7-006618b00a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.331 DEBUG:boto:b'dkyges-58a1000falsea2024-04-12T03:52:21.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:52:42.332 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:52:42.332 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:52:42.332 DEBUG:boto:path=/dkyges-58/b?versionId=null 2024-04-12T03:52:42.332 DEBUG:boto:auth_path=/dkyges-58/b?versionId=null 2024-04-12T03:52:42.332 DEBUG:boto:Method: GET 2024-04-12T03:52:42.332 DEBUG:boto:Path: /dkyges-58/b?versionId=null 2024-04-12T03:52:42.332 DEBUG:boto:Data: 2024-04-12T03:52:42.332 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.332 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.332 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.332 DEBUG:boto:Params: {} 2024-04-12T03:52:42.333 DEBUG:boto:Token: None 2024-04-12T03:52:42.333 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/b?versionId=null 2024-04-12T03:52:42.333 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gUiyd6HHcpaWnoWznueAooFnnUs= 2024-04-12T03:52:42.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gUiyd6HHcpaWnoWznueAooFnnUs=', 'Content-Length': '0'} 2024-04-12T03:52:42.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002db2f812caae20aa-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.335 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:52:42.335 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:52:42.335 DEBUG:boto:path=/dkyges-58/b?versionId=null 2024-04-12T03:52:42.335 DEBUG:boto:auth_path=/dkyges-58/b?versionId=null 2024-04-12T03:52:42.336 DEBUG:boto:Method: GET 2024-04-12T03:52:42.336 DEBUG:boto:Path: /dkyges-58/b?versionId=null 2024-04-12T03:52:42.336 DEBUG:boto:Data: 2024-04-12T03:52:42.336 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.336 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.336 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.336 DEBUG:boto:Params: {} 2024-04-12T03:52:42.336 DEBUG:boto:Token: None 2024-04-12T03:52:42.336 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/b?versionId=null 2024-04-12T03:52:42.336 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gUiyd6HHcpaWnoWznueAooFnnUs= 2024-04-12T03:52:42.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gUiyd6HHcpaWnoWznueAooFnnUs=', 'Content-Length': '0'} 2024-04-12T03:52:42.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c32bcbd9d412e17-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.339 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:52:42.339 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:52:42.339 DEBUG:boto:path=/dkyges-58/b?versionId=null 2024-04-12T03:52:42.339 DEBUG:boto:auth_path=/dkyges-58/b?versionId=null 2024-04-12T03:52:42.339 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.339 DEBUG:boto:Path: /dkyges-58/b?versionId=null 2024-04-12T03:52:42.339 DEBUG:boto:Data: 2024-04-12T03:52:42.339 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.339 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.339 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.340 DEBUG:boto:Params: {} 2024-04-12T03:52:42.340 DEBUG:boto:Token: None 2024-04-12T03:52:42.340 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/b?versionId=null 2024-04-12T03:52:42.340 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YDuVgp36qhI+sX7/7vV8jzIkXno= 2024-04-12T03:52:42.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YDuVgp36qhI+sX7/7vV8jzIkXno=', 'Content-Length': '0'} 2024-04-12T03:52:42.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eadf13160b438969-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.342 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:52:42.342 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:52:42.342 DEBUG:boto:path=/dkyges-58/b?versionId=null 2024-04-12T03:52:42.342 DEBUG:boto:auth_path=/dkyges-58/b?versionId=null 2024-04-12T03:52:42.342 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.343 DEBUG:boto:Path: /dkyges-58/b?versionId=null 2024-04-12T03:52:42.343 DEBUG:boto:Data: 2024-04-12T03:52:42.343 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.343 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.343 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.343 DEBUG:boto:Params: {} 2024-04-12T03:52:42.343 DEBUG:boto:Token: None 2024-04-12T03:52:42.343 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/b?versionId=null 2024-04-12T03:52:42.343 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YDuVgp36qhI+sX7/7vV8jzIkXno= 2024-04-12T03:52:42.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YDuVgp36qhI+sX7/7vV8jzIkXno=', 'Content-Length': '0'} 2024-04-12T03:52:42.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b06deae13efa0bd8-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:52:42.346 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:52:42.346 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:52:42.346 DEBUG:boto:path=/dkyges-58/b?versionId=null 2024-04-12T03:52:42.346 DEBUG:boto:auth_path=/dkyges-58/b?versionId=null 2024-04-12T03:52:42.346 DEBUG:boto:Method: GET 2024-04-12T03:52:42.346 DEBUG:boto:Path: /dkyges-58/b?versionId=null 2024-04-12T03:52:42.346 DEBUG:boto:Data: 2024-04-12T03:52:42.346 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.346 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.347 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.347 DEBUG:boto:Params: {} 2024-04-12T03:52:42.347 DEBUG:boto:Token: None 2024-04-12T03:52:42.347 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/b?versionId=null 2024-04-12T03:52:42.347 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gUiyd6HHcpaWnoWznueAooFnnUs= 2024-04-12T03:52:42.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gUiyd6HHcpaWnoWznueAooFnnUs=', 'Content-Length': '0'} 2024-04-12T03:52:42.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004231d0e9a04f2a7d-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.350 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:52:42.350 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:52:42.350 DEBUG:boto:path=/dkyges-58/b?versionId=null 2024-04-12T03:52:42.350 DEBUG:boto:auth_path=/dkyges-58/b?versionId=null 2024-04-12T03:52:42.350 DEBUG:boto:Method: GET 2024-04-12T03:52:42.351 DEBUG:boto:Path: /dkyges-58/b?versionId=null 2024-04-12T03:52:42.351 DEBUG:boto:Data: 2024-04-12T03:52:42.351 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.351 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.351 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.351 DEBUG:boto:Params: {} 2024-04-12T03:52:42.351 DEBUG:boto:Token: None 2024-04-12T03:52:42.351 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/b?versionId=null 2024-04-12T03:52:42.352 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gUiyd6HHcpaWnoWznueAooFnnUs= 2024-04-12T03:52:42.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gUiyd6HHcpaWnoWznueAooFnnUs=', 'Content-Length': '0'} 2024-04-12T03:52:42.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a278b5f8bc1eda5-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.394 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.395 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.395 DEBUG:boto:path=/dkyges-58/?prefix=b 2024-04-12T03:52:42.395 DEBUG:boto:auth_path=/dkyges-58/?prefix=b 2024-04-12T03:52:42.395 DEBUG:boto:Method: GET 2024-04-12T03:52:42.395 DEBUG:boto:Path: /dkyges-58/?prefix=b 2024-04-12T03:52:42.395 DEBUG:boto:Data: 2024-04-12T03:52:42.395 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.395 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.395 DEBUG:boto:Params: {} 2024-04-12T03:52:42.396 DEBUG:boto:Token: None 2024-04-12T03:52:42.396 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.396 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q= 2024-04-12T03:52:42.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9dd2027a1633386-006618b00a-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.400 DEBUG:boto:b'dkyges-58b1000falseb2024-04-12T03:52:21.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.400 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.400 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.400 DEBUG:boto:path=/dkyges-58/?prefix=b 2024-04-12T03:52:42.400 DEBUG:boto:auth_path=/dkyges-58/?prefix=b 2024-04-12T03:52:42.400 DEBUG:boto:Method: GET 2024-04-12T03:52:42.400 DEBUG:boto:Path: /dkyges-58/?prefix=b 2024-04-12T03:52:42.401 DEBUG:boto:Data: 2024-04-12T03:52:42.401 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.401 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.401 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.401 DEBUG:boto:Params: {} 2024-04-12T03:52:42.401 DEBUG:boto:Token: None 2024-04-12T03:52:42.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.401 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q= 2024-04-12T03:52:42.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e55ccd8eca071417-006618b00a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.404 DEBUG:boto:b'dkyges-58b1000falseb2024-04-12T03:52:21.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.404 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:52:42.404 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:52:42.404 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.404 DEBUG:boto:Path: /dkyges-58/b 2024-04-12T03:52:42.405 DEBUG:boto:Data: 2024-04-12T03:52:42.405 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.405 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.405 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.405 DEBUG:boto:Params: {} 2024-04-12T03:52:42.405 DEBUG:boto:Token: None 2024-04-12T03:52:42.405 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/b 2024-04-12T03:52:42.405 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ijSf49xilCgbrSi57l7mdEuk/tM= 2024-04-12T03:52:42.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ijSf49xilCgbrSi57l7mdEuk/tM=', 'Content-Length': '0'} 2024-04-12T03:52:42.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000363dd9a9d443a413-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.407 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:52:42.408 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:52:42.408 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.408 DEBUG:boto:Path: /dkyges-58/b 2024-04-12T03:52:42.408 DEBUG:boto:Data: 2024-04-12T03:52:42.408 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.408 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.408 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.408 DEBUG:boto:Params: {} 2024-04-12T03:52:42.408 DEBUG:boto:Token: None 2024-04-12T03:52:42.409 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/b 2024-04-12T03:52:42.409 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ijSf49xilCgbrSi57l7mdEuk/tM= 2024-04-12T03:52:42.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ijSf49xilCgbrSi57l7mdEuk/tM=', 'Content-Length': '0'} 2024-04-12T03:52:42.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcceb2e859508628-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:52:42.411 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:52:42.411 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:52:42.411 DEBUG:boto:Method: GET 2024-04-12T03:52:42.411 DEBUG:boto:Path: /dkyges-58/b 2024-04-12T03:52:42.411 DEBUG:boto:Data: 2024-04-12T03:52:42.411 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.411 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.411 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.411 DEBUG:boto:Params: {} 2024-04-12T03:52:42.412 DEBUG:boto:Token: None 2024-04-12T03:52:42.412 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/b 2024-04-12T03:52:42.412 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:84LWCxW6v7C+GS5vkgmY8abPf9o= 2024-04-12T03:52:42.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:84LWCxW6v7C+GS5vkgmY8abPf9o=', 'Content-Length': '0'} 2024-04-12T03:52:42.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001847dccccac314d0-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.458 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:52:42.458 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:52:42.459 DEBUG:boto:Method: GET 2024-04-12T03:52:42.459 DEBUG:boto:Path: /dkyges-58/b 2024-04-12T03:52:42.459 DEBUG:boto:Data: 2024-04-12T03:52:42.459 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.459 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.459 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.459 DEBUG:boto:Params: {} 2024-04-12T03:52:42.459 DEBUG:boto:Token: None 2024-04-12T03:52:42.459 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/b 2024-04-12T03:52:42.459 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:84LWCxW6v7C+GS5vkgmY8abPf9o= 2024-04-12T03:52:42.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:84LWCxW6v7C+GS5vkgmY8abPf9o=', 'Content-Length': '0'} 2024-04-12T03:52:42.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad3b73b17f6604b0-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.462 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.462 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.462 DEBUG:boto:path=/dkyges-58/?prefix=b 2024-04-12T03:52:42.462 DEBUG:boto:auth_path=/dkyges-58/?prefix=b 2024-04-12T03:52:42.462 DEBUG:boto:Method: GET 2024-04-12T03:52:42.462 DEBUG:boto:Path: /dkyges-58/?prefix=b 2024-04-12T03:52:42.462 DEBUG:boto:Data: 2024-04-12T03:52:42.462 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.463 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.463 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.463 DEBUG:boto:Params: {} 2024-04-12T03:52:42.463 DEBUG:boto:Token: None 2024-04-12T03:52:42.463 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.463 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q= 2024-04-12T03:52:42.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002970afcaa4881dd6-006618b00a-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.465 DEBUG:boto:b'dkyges-58b1000falseb2024-04-12T03:52:21.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.466 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.466 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.466 DEBUG:boto:path=/dkyges-58/?prefix=b 2024-04-12T03:52:42.466 DEBUG:boto:auth_path=/dkyges-58/?prefix=b 2024-04-12T03:52:42.466 DEBUG:boto:Method: GET 2024-04-12T03:52:42.467 DEBUG:boto:Path: /dkyges-58/?prefix=b 2024-04-12T03:52:42.467 DEBUG:boto:Data: 2024-04-12T03:52:42.467 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.467 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.467 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.467 DEBUG:boto:Params: {} 2024-04-12T03:52:42.467 DEBUG:boto:Token: None 2024-04-12T03:52:42.467 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.467 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q= 2024-04-12T03:52:42.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037cf38d32bfb642e-006618b00a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.470 DEBUG:boto:b'dkyges-58b1000falseb2024-04-12T03:52:21.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:52:42.471 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:52:42.471 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:52:42.471 DEBUG:boto:path=/dkyges-58/c?versionId=null 2024-04-12T03:52:42.471 DEBUG:boto:auth_path=/dkyges-58/c?versionId=null 2024-04-12T03:52:42.471 DEBUG:boto:Method: GET 2024-04-12T03:52:42.471 DEBUG:boto:Path: /dkyges-58/c?versionId=null 2024-04-12T03:52:42.471 DEBUG:boto:Data: 2024-04-12T03:52:42.471 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.471 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.471 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.471 DEBUG:boto:Params: {} 2024-04-12T03:52:42.472 DEBUG:boto:Token: None 2024-04-12T03:52:42.472 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/c?versionId=null 2024-04-12T03:52:42.472 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QlmW3ur6qvd5LnesRwpQ5hKww9Q= 2024-04-12T03:52:42.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QlmW3ur6qvd5LnesRwpQ5hKww9Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d3e4938537b80aa-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.474 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:52:42.474 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:52:42.474 DEBUG:boto:path=/dkyges-58/c?versionId=null 2024-04-12T03:52:42.474 DEBUG:boto:auth_path=/dkyges-58/c?versionId=null 2024-04-12T03:52:42.474 DEBUG:boto:Method: GET 2024-04-12T03:52:42.474 DEBUG:boto:Path: /dkyges-58/c?versionId=null 2024-04-12T03:52:42.474 DEBUG:boto:Data: 2024-04-12T03:52:42.475 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.475 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.475 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.475 DEBUG:boto:Params: {} 2024-04-12T03:52:42.475 DEBUG:boto:Token: None 2024-04-12T03:52:42.475 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/c?versionId=null 2024-04-12T03:52:42.475 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QlmW3ur6qvd5LnesRwpQ5hKww9Q= 2024-04-12T03:52:42.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QlmW3ur6qvd5LnesRwpQ5hKww9Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccd51d59c3a2aaf6-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.477 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:52:42.477 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:52:42.478 DEBUG:boto:path=/dkyges-58/c?versionId=null 2024-04-12T03:52:42.478 DEBUG:boto:auth_path=/dkyges-58/c?versionId=null 2024-04-12T03:52:42.478 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.478 DEBUG:boto:Path: /dkyges-58/c?versionId=null 2024-04-12T03:52:42.478 DEBUG:boto:Data: 2024-04-12T03:52:42.478 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.478 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.478 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.478 DEBUG:boto:Params: {} 2024-04-12T03:52:42.478 DEBUG:boto:Token: None 2024-04-12T03:52:42.478 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/c?versionId=null 2024-04-12T03:52:42.479 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:blLOWUgsrEzqTQOwnM853Dxd/QU= 2024-04-12T03:52:42.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:blLOWUgsrEzqTQOwnM853Dxd/QU=', 'Content-Length': '0'} 2024-04-12T03:52:42.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072b63ad9e8cc4eec-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.481 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:52:42.481 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:52:42.481 DEBUG:boto:path=/dkyges-58/c?versionId=null 2024-04-12T03:52:42.481 DEBUG:boto:auth_path=/dkyges-58/c?versionId=null 2024-04-12T03:52:42.481 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.481 DEBUG:boto:Path: /dkyges-58/c?versionId=null 2024-04-12T03:52:42.481 DEBUG:boto:Data: 2024-04-12T03:52:42.481 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.481 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.481 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.481 DEBUG:boto:Params: {} 2024-04-12T03:52:42.481 DEBUG:boto:Token: None 2024-04-12T03:52:42.482 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/c?versionId=null 2024-04-12T03:52:42.482 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:blLOWUgsrEzqTQOwnM853Dxd/QU= 2024-04-12T03:52:42.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:blLOWUgsrEzqTQOwnM853Dxd/QU=', 'Content-Length': '0'} 2024-04-12T03:52:42.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afb9d3f3632fa053-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:52:42.484 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:52:42.484 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:52:42.484 DEBUG:boto:path=/dkyges-58/c?versionId=null 2024-04-12T03:52:42.484 DEBUG:boto:auth_path=/dkyges-58/c?versionId=null 2024-04-12T03:52:42.484 DEBUG:boto:Method: GET 2024-04-12T03:52:42.484 DEBUG:boto:Path: /dkyges-58/c?versionId=null 2024-04-12T03:52:42.484 DEBUG:boto:Data: 2024-04-12T03:52:42.484 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.485 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.485 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.485 DEBUG:boto:Params: {} 2024-04-12T03:52:42.485 DEBUG:boto:Token: None 2024-04-12T03:52:42.485 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/c?versionId=null 2024-04-12T03:52:42.485 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QlmW3ur6qvd5LnesRwpQ5hKww9Q= 2024-04-12T03:52:42.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QlmW3ur6qvd5LnesRwpQ5hKww9Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060d9958056676295-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.487 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:52:42.487 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:52:42.487 DEBUG:boto:path=/dkyges-58/c?versionId=null 2024-04-12T03:52:42.487 DEBUG:boto:auth_path=/dkyges-58/c?versionId=null 2024-04-12T03:52:42.488 DEBUG:boto:Method: GET 2024-04-12T03:52:42.488 DEBUG:boto:Path: /dkyges-58/c?versionId=null 2024-04-12T03:52:42.488 DEBUG:boto:Data: 2024-04-12T03:52:42.488 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.488 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.488 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.488 DEBUG:boto:Params: {} 2024-04-12T03:52:42.488 DEBUG:boto:Token: None 2024-04-12T03:52:42.488 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/c?versionId=null 2024-04-12T03:52:42.488 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QlmW3ur6qvd5LnesRwpQ5hKww9Q= 2024-04-12T03:52:42.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QlmW3ur6qvd5LnesRwpQ5hKww9Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3191b94de62a117-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.491 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.491 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.491 DEBUG:boto:path=/dkyges-58/?prefix=c 2024-04-12T03:52:42.491 DEBUG:boto:auth_path=/dkyges-58/?prefix=c 2024-04-12T03:52:42.491 DEBUG:boto:Method: GET 2024-04-12T03:52:42.491 DEBUG:boto:Path: /dkyges-58/?prefix=c 2024-04-12T03:52:42.491 DEBUG:boto:Data: 2024-04-12T03:52:42.491 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.492 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.492 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.492 DEBUG:boto:Params: {} 2024-04-12T03:52:42.492 DEBUG:boto:Token: None 2024-04-12T03:52:42.492 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.492 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q= 2024-04-12T03:52:42.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089a8956a8f330907-006618b00a-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.495 DEBUG:boto:b'dkyges-58c1000falsec2024-04-12T03:52:21.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.495 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.495 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.495 DEBUG:boto:path=/dkyges-58/?prefix=c 2024-04-12T03:52:42.495 DEBUG:boto:auth_path=/dkyges-58/?prefix=c 2024-04-12T03:52:42.496 DEBUG:boto:Method: GET 2024-04-12T03:52:42.496 DEBUG:boto:Path: /dkyges-58/?prefix=c 2024-04-12T03:52:42.496 DEBUG:boto:Data: 2024-04-12T03:52:42.496 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.496 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.496 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.496 DEBUG:boto:Params: {} 2024-04-12T03:52:42.496 DEBUG:boto:Token: None 2024-04-12T03:52:42.496 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.497 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q= 2024-04-12T03:52:42.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad4b4f6fa16dc3ac-006618b00a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.542 DEBUG:boto:b'dkyges-58c1000falsec2024-04-12T03:52:21.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.543 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:52:42.543 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:52:42.543 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.543 DEBUG:boto:Path: /dkyges-58/c 2024-04-12T03:52:42.543 DEBUG:boto:Data: 2024-04-12T03:52:42.543 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.544 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.544 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.544 DEBUG:boto:Params: {} 2024-04-12T03:52:42.544 DEBUG:boto:Token: None 2024-04-12T03:52:42.544 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/c 2024-04-12T03:52:42.544 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2gJh9+dZ3hyBpro190QnafPzJiI= 2024-04-12T03:52:42.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2gJh9+dZ3hyBpro190QnafPzJiI=', 'Content-Length': '0'} 2024-04-12T03:52:42.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc6cfb69dc2b4cd1-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.546 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:52:42.547 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:52:42.547 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.547 DEBUG:boto:Path: /dkyges-58/c 2024-04-12T03:52:42.547 DEBUG:boto:Data: 2024-04-12T03:52:42.547 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.547 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.547 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.547 DEBUG:boto:Params: {} 2024-04-12T03:52:42.547 DEBUG:boto:Token: None 2024-04-12T03:52:42.547 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/c 2024-04-12T03:52:42.548 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2gJh9+dZ3hyBpro190QnafPzJiI= 2024-04-12T03:52:42.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2gJh9+dZ3hyBpro190QnafPzJiI=', 'Content-Length': '0'} 2024-04-12T03:52:42.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c11dd9781905bcc-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:52:42.550 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:52:42.550 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:52:42.550 DEBUG:boto:Method: GET 2024-04-12T03:52:42.550 DEBUG:boto:Path: /dkyges-58/c 2024-04-12T03:52:42.550 DEBUG:boto:Data: 2024-04-12T03:52:42.550 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.550 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.551 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.551 DEBUG:boto:Params: {} 2024-04-12T03:52:42.551 DEBUG:boto:Token: None 2024-04-12T03:52:42.551 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/c 2024-04-12T03:52:42.551 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gvuU3y4pCT1eO6iYU95l2W9yM9I= 2024-04-12T03:52:42.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gvuU3y4pCT1eO6iYU95l2W9yM9I=', 'Content-Length': '0'} 2024-04-12T03:52:42.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c63e1b3d15e77277-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.554 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:52:42.554 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:52:42.554 DEBUG:boto:Method: GET 2024-04-12T03:52:42.554 DEBUG:boto:Path: /dkyges-58/c 2024-04-12T03:52:42.554 DEBUG:boto:Data: 2024-04-12T03:52:42.554 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.554 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.554 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.554 DEBUG:boto:Params: {} 2024-04-12T03:52:42.555 DEBUG:boto:Token: None 2024-04-12T03:52:42.555 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/c 2024-04-12T03:52:42.555 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gvuU3y4pCT1eO6iYU95l2W9yM9I= 2024-04-12T03:52:42.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gvuU3y4pCT1eO6iYU95l2W9yM9I=', 'Content-Length': '0'} 2024-04-12T03:52:42.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf9c48f5dc85cad7-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.557 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.557 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.557 DEBUG:boto:path=/dkyges-58/?prefix=c 2024-04-12T03:52:42.557 DEBUG:boto:auth_path=/dkyges-58/?prefix=c 2024-04-12T03:52:42.558 DEBUG:boto:Method: GET 2024-04-12T03:52:42.558 DEBUG:boto:Path: /dkyges-58/?prefix=c 2024-04-12T03:52:42.558 DEBUG:boto:Data: 2024-04-12T03:52:42.558 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.558 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.558 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.558 DEBUG:boto:Params: {} 2024-04-12T03:52:42.558 DEBUG:boto:Token: None 2024-04-12T03:52:42.559 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.559 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q= 2024-04-12T03:52:42.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043d1f71a0c2507ad-006618b00a-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.561 DEBUG:boto:b'dkyges-58c1000falsec2024-04-12T03:52:21.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.562 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.562 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.562 DEBUG:boto:path=/dkyges-58/?prefix=c 2024-04-12T03:52:42.562 DEBUG:boto:auth_path=/dkyges-58/?prefix=c 2024-04-12T03:52:42.562 DEBUG:boto:Method: GET 2024-04-12T03:52:42.562 DEBUG:boto:Path: /dkyges-58/?prefix=c 2024-04-12T03:52:42.562 DEBUG:boto:Data: 2024-04-12T03:52:42.562 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.562 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.562 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.562 DEBUG:boto:Params: {} 2024-04-12T03:52:42.563 DEBUG:boto:Token: None 2024-04-12T03:52:42.563 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.563 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q= 2024-04-12T03:52:42.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bc9cb21eb48d4a2-006618b00a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.565 DEBUG:boto:b'dkyges-58c1000falsec2024-04-12T03:52:21.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:52:42.566 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:52:42.566 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:52:42.566 DEBUG:boto:path=/dkyges-58/d?versionId=null 2024-04-12T03:52:42.566 DEBUG:boto:auth_path=/dkyges-58/d?versionId=null 2024-04-12T03:52:42.566 DEBUG:boto:Method: GET 2024-04-12T03:52:42.566 DEBUG:boto:Path: /dkyges-58/d?versionId=null 2024-04-12T03:52:42.566 DEBUG:boto:Data: 2024-04-12T03:52:42.567 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.567 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.567 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.567 DEBUG:boto:Params: {} 2024-04-12T03:52:42.567 DEBUG:boto:Token: None 2024-04-12T03:52:42.567 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/d?versionId=null 2024-04-12T03:52:42.567 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1F4GuU7TDEWg+/OoYTBr2AnxsDI= 2024-04-12T03:52:42.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1F4GuU7TDEWg+/OoYTBr2AnxsDI=', 'Content-Length': '0'} 2024-04-12T03:52:42.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d7e5b4baf40b50f-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.570 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:52:42.570 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:52:42.570 DEBUG:boto:path=/dkyges-58/d?versionId=null 2024-04-12T03:52:42.570 DEBUG:boto:auth_path=/dkyges-58/d?versionId=null 2024-04-12T03:52:42.570 DEBUG:boto:Method: GET 2024-04-12T03:52:42.570 DEBUG:boto:Path: /dkyges-58/d?versionId=null 2024-04-12T03:52:42.570 DEBUG:boto:Data: 2024-04-12T03:52:42.570 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.570 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.570 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.571 DEBUG:boto:Params: {} 2024-04-12T03:52:42.571 DEBUG:boto:Token: None 2024-04-12T03:52:42.571 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/d?versionId=null 2024-04-12T03:52:42.571 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1F4GuU7TDEWg+/OoYTBr2AnxsDI= 2024-04-12T03:52:42.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1F4GuU7TDEWg+/OoYTBr2AnxsDI=', 'Content-Length': '0'} 2024-04-12T03:52:42.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048af87e55548c858-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.573 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:52:42.573 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:52:42.573 DEBUG:boto:path=/dkyges-58/d?versionId=null 2024-04-12T03:52:42.574 DEBUG:boto:auth_path=/dkyges-58/d?versionId=null 2024-04-12T03:52:42.574 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.574 DEBUG:boto:Path: /dkyges-58/d?versionId=null 2024-04-12T03:52:42.574 DEBUG:boto:Data: 2024-04-12T03:52:42.574 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.574 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.574 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.574 DEBUG:boto:Params: {} 2024-04-12T03:52:42.574 DEBUG:boto:Token: None 2024-04-12T03:52:42.574 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/d?versionId=null 2024-04-12T03:52:42.574 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iDfzvqN3VDM4plJIkUFyBELBu3U= 2024-04-12T03:52:42.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iDfzvqN3VDM4plJIkUFyBELBu3U=', 'Content-Length': '0'} 2024-04-12T03:52:42.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa46836c4da4fd10-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.577 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:52:42.577 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:52:42.577 DEBUG:boto:path=/dkyges-58/d?versionId=null 2024-04-12T03:52:42.577 DEBUG:boto:auth_path=/dkyges-58/d?versionId=null 2024-04-12T03:52:42.577 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.577 DEBUG:boto:Path: /dkyges-58/d?versionId=null 2024-04-12T03:52:42.577 DEBUG:boto:Data: 2024-04-12T03:52:42.577 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.577 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.577 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.578 DEBUG:boto:Params: {} 2024-04-12T03:52:42.578 DEBUG:boto:Token: None 2024-04-12T03:52:42.578 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/d?versionId=null 2024-04-12T03:52:42.578 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iDfzvqN3VDM4plJIkUFyBELBu3U= 2024-04-12T03:52:42.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iDfzvqN3VDM4plJIkUFyBELBu3U=', 'Content-Length': '0'} 2024-04-12T03:52:42.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081a656729853e520-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:52:42.580 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:52:42.580 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:52:42.580 DEBUG:boto:path=/dkyges-58/d?versionId=null 2024-04-12T03:52:42.580 DEBUG:boto:auth_path=/dkyges-58/d?versionId=null 2024-04-12T03:52:42.580 DEBUG:boto:Method: GET 2024-04-12T03:52:42.581 DEBUG:boto:Path: /dkyges-58/d?versionId=null 2024-04-12T03:52:42.581 DEBUG:boto:Data: 2024-04-12T03:52:42.581 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.581 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.581 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.581 DEBUG:boto:Params: {} 2024-04-12T03:52:42.581 DEBUG:boto:Token: None 2024-04-12T03:52:42.581 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/d?versionId=null 2024-04-12T03:52:42.581 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1F4GuU7TDEWg+/OoYTBr2AnxsDI= 2024-04-12T03:52:42.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1F4GuU7TDEWg+/OoYTBr2AnxsDI=', 'Content-Length': '0'} 2024-04-12T03:52:42.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019d192ae34a40e13-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.584 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:52:42.584 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:52:42.584 DEBUG:boto:path=/dkyges-58/d?versionId=null 2024-04-12T03:52:42.584 DEBUG:boto:auth_path=/dkyges-58/d?versionId=null 2024-04-12T03:52:42.584 DEBUG:boto:Method: GET 2024-04-12T03:52:42.584 DEBUG:boto:Path: /dkyges-58/d?versionId=null 2024-04-12T03:52:42.584 DEBUG:boto:Data: 2024-04-12T03:52:42.584 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.585 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.585 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.585 DEBUG:boto:Params: {} 2024-04-12T03:52:42.585 DEBUG:boto:Token: None 2024-04-12T03:52:42.585 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/d?versionId=null 2024-04-12T03:52:42.585 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1F4GuU7TDEWg+/OoYTBr2AnxsDI= 2024-04-12T03:52:42.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1F4GuU7TDEWg+/OoYTBr2AnxsDI=', 'Content-Length': '0'} 2024-04-12T03:52:42.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1ce90f219e5861e-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.587 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.587 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.588 DEBUG:boto:path=/dkyges-58/?prefix=d 2024-04-12T03:52:42.588 DEBUG:boto:auth_path=/dkyges-58/?prefix=d 2024-04-12T03:52:42.588 DEBUG:boto:Method: GET 2024-04-12T03:52:42.588 DEBUG:boto:Path: /dkyges-58/?prefix=d 2024-04-12T03:52:42.588 DEBUG:boto:Data: 2024-04-12T03:52:42.588 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.588 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.588 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.588 DEBUG:boto:Params: {} 2024-04-12T03:52:42.588 DEBUG:boto:Token: None 2024-04-12T03:52:42.588 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.589 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q= 2024-04-12T03:52:42.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000419675f473db3e66-006618b00a-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.634 DEBUG:boto:b'dkyges-58d1000falsed2024-04-12T03:52:21.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.635 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.635 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.635 DEBUG:boto:path=/dkyges-58/?prefix=d 2024-04-12T03:52:42.635 DEBUG:boto:auth_path=/dkyges-58/?prefix=d 2024-04-12T03:52:42.635 DEBUG:boto:Method: GET 2024-04-12T03:52:42.635 DEBUG:boto:Path: /dkyges-58/?prefix=d 2024-04-12T03:52:42.635 DEBUG:boto:Data: 2024-04-12T03:52:42.635 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.636 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.636 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.636 DEBUG:boto:Params: {} 2024-04-12T03:52:42.636 DEBUG:boto:Token: None 2024-04-12T03:52:42.636 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.636 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q= 2024-04-12T03:52:42.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6b308f70d5a58b3-006618b00a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.639 DEBUG:boto:b'dkyges-58d1000falsed2024-04-12T03:52:21.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.639 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:52:42.639 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:52:42.639 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.639 DEBUG:boto:Path: /dkyges-58/d 2024-04-12T03:52:42.639 DEBUG:boto:Data: 2024-04-12T03:52:42.639 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.640 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.640 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.640 DEBUG:boto:Params: {} 2024-04-12T03:52:42.640 DEBUG:boto:Token: None 2024-04-12T03:52:42.640 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/d 2024-04-12T03:52:42.640 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Pvtcx7XcIJChgqj8XCY9VtFRgz8= 2024-04-12T03:52:42.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Pvtcx7XcIJChgqj8XCY9VtFRgz8=', 'Content-Length': '0'} 2024-04-12T03:52:42.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abec0aaa7bb31252-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.642 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:52:42.642 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:52:42.642 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.642 DEBUG:boto:Path: /dkyges-58/d 2024-04-12T03:52:42.642 DEBUG:boto:Data: 2024-04-12T03:52:42.642 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.642 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.643 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.643 DEBUG:boto:Params: {} 2024-04-12T03:52:42.643 DEBUG:boto:Token: None 2024-04-12T03:52:42.643 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/d 2024-04-12T03:52:42.643 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Pvtcx7XcIJChgqj8XCY9VtFRgz8= 2024-04-12T03:52:42.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Pvtcx7XcIJChgqj8XCY9VtFRgz8=', 'Content-Length': '0'} 2024-04-12T03:52:42.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000508f4291987b0bc9-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:52:42.645 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:52:42.645 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:52:42.645 DEBUG:boto:Method: GET 2024-04-12T03:52:42.645 DEBUG:boto:Path: /dkyges-58/d 2024-04-12T03:52:42.646 DEBUG:boto:Data: 2024-04-12T03:52:42.646 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.646 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.646 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.646 DEBUG:boto:Params: {} 2024-04-12T03:52:42.646 DEBUG:boto:Token: None 2024-04-12T03:52:42.646 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/d 2024-04-12T03:52:42.646 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2vwiejKRCrqVxWFDVTx92FkteN0= 2024-04-12T03:52:42.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2vwiejKRCrqVxWFDVTx92FkteN0=', 'Content-Length': '0'} 2024-04-12T03:52:42.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000787ceb6d4096d9fc-006618b00a-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.648 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:52:42.648 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:52:42.649 DEBUG:boto:Method: GET 2024-04-12T03:52:42.649 DEBUG:boto:Path: /dkyges-58/d 2024-04-12T03:52:42.649 DEBUG:boto:Data: 2024-04-12T03:52:42.649 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.649 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.649 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.649 DEBUG:boto:Params: {} 2024-04-12T03:52:42.649 DEBUG:boto:Token: None 2024-04-12T03:52:42.649 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/d 2024-04-12T03:52:42.649 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2vwiejKRCrqVxWFDVTx92FkteN0= 2024-04-12T03:52:42.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2vwiejKRCrqVxWFDVTx92FkteN0=', 'Content-Length': '0'} 2024-04-12T03:52:42.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfe408aea85ee19b-006618b00a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.652 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.652 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.652 DEBUG:boto:path=/dkyges-58/?prefix=d 2024-04-12T03:52:42.652 DEBUG:boto:auth_path=/dkyges-58/?prefix=d 2024-04-12T03:52:42.652 DEBUG:boto:Method: GET 2024-04-12T03:52:42.652 DEBUG:boto:Path: /dkyges-58/?prefix=d 2024-04-12T03:52:42.652 DEBUG:boto:Data: 2024-04-12T03:52:42.652 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.652 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:52:42.652 DEBUG:boto:Port: 8001 2024-04-12T03:52:42.653 DEBUG:boto:Params: {} 2024-04-12T03:52:42.653 DEBUG:boto:Token: None 2024-04-12T03:52:42.653 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.653 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q= 2024-04-12T03:52:42.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005502e6210e9d56d-006618b00a-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.656 DEBUG:boto:b'dkyges-58d1000falsed2024-04-12T03:52:21.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.656 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.656 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.656 DEBUG:boto:path=/dkyges-58/?prefix=d 2024-04-12T03:52:42.656 DEBUG:boto:auth_path=/dkyges-58/?prefix=d 2024-04-12T03:52:42.656 DEBUG:boto:Method: GET 2024-04-12T03:52:42.656 DEBUG:boto:Path: /dkyges-58/?prefix=d 2024-04-12T03:52:42.656 DEBUG:boto:Data: 2024-04-12T03:52:42.657 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.657 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.657 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.657 DEBUG:boto:Params: {} 2024-04-12T03:52:42.657 DEBUG:boto:Token: None 2024-04-12T03:52:42.657 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.657 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q= 2024-04-12T03:52:42.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zJNwfPQbb9I5FJILSguTOFfAK6Q=', 'Content-Length': '0'} 2024-04-12T03:52:42.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a61b839218509d5-006618b00a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.702 DEBUG:boto:b'dkyges-58d1000falsed2024-04-12T03:52:21.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:52:42.703 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-58 zones={a2, a1} 2024-04-12T03:52:42.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dkyges-58', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:52:42.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dkyges-58', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:52:42.704 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dkyges-58 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:52:42.763 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:52:42.763 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:52:42.811 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T03:52:42.811 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-58 2024-04-12T03:52:42.816 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 4 2024-04-12T03:52:42.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dkyges-58', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:52:42.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dkyges-58', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:52:42.858 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dkyges-58 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:52:42.914 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:52:42.915 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:52:42.959 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T03:52:42.959 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-58 2024-04-12T03:52:42.960 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 4 2024-04-12T03:52:42.991 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:42.992 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:42.992 DEBUG:boto:Method: HEAD 2024-04-12T03:52:42.992 DEBUG:boto:Path: /dkyges-58/ 2024-04-12T03:52:42.992 DEBUG:boto:Data: 2024-04-12T03:52:42.992 DEBUG:boto:Headers: {} 2024-04-12T03:52:42.992 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.992 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.992 DEBUG:boto:Params: {} 2024-04-12T03:52:42.992 DEBUG:boto:Token: None 2024-04-12T03:52:42.993 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/ 2024-04-12T03:52:42.993 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L2lLRSWYnjeau4OtJHskFYk4S9g= 2024-04-12T03:52:42.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L2lLRSWYnjeau4OtJHskFYk4S9g=', 'Content-Length': '0'} 2024-04-12T03:52:42.995 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f04ae6e3c9f1409-006618b00b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:42.995 DEBUG:boto:path=/dkyges-58/e 2024-04-12T03:52:42.995 DEBUG:boto:auth_path=/dkyges-58/e 2024-04-12T03:52:42.995 DEBUG:boto:Method: PUT 2024-04-12T03:52:42.996 DEBUG:boto:Path: /dkyges-58/e 2024-04-12T03:52:42.996 DEBUG:boto:Data: 2024-04-12T03:52:42.996 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:52:42.996 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:42.996 DEBUG:boto:Port: 8000 2024-04-12T03:52:42.996 DEBUG:boto:Params: {} 2024-04-12T03:52:42.996 DEBUG:boto:Token: None 2024-04-12T03:52:42.996 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:52:42 GMT /dkyges-58/e 2024-04-12T03:52:42.996 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3rGbc3pdfr4FxkRMj/OZhSsmipk= 2024-04-12T03:52:42.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:52:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3rGbc3pdfr4FxkRMj/OZhSsmipk='} 2024-04-12T03:52:43.002 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000001a95e9c265f7072-006618b00b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:43.003 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:43.003 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:43.003 DEBUG:boto:Method: HEAD 2024-04-12T03:52:43.003 DEBUG:boto:Path: /dkyges-58/ 2024-04-12T03:52:43.003 DEBUG:boto:Data: 2024-04-12T03:52:43.003 DEBUG:boto:Headers: {} 2024-04-12T03:52:43.003 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:43.003 DEBUG:boto:Port: 8000 2024-04-12T03:52:43.004 DEBUG:boto:Params: {} 2024-04-12T03:52:43.004 DEBUG:boto:Token: None 2024-04-12T03:52:43.004 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:43 GMT /dkyges-58/ 2024-04-12T03:52:43.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t1kitMlPA2IRvTSSy2gtVpHdKWE= 2024-04-12T03:52:43.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t1kitMlPA2IRvTSSy2gtVpHdKWE=', 'Content-Length': '0'} 2024-04-12T03:52:43.006 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b87183e9676a3b1-006618b00b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:43.006 DEBUG:boto:path=/dkyges-58/f 2024-04-12T03:52:43.006 DEBUG:boto:auth_path=/dkyges-58/f 2024-04-12T03:52:43.007 DEBUG:boto:Method: PUT 2024-04-12T03:52:43.007 DEBUG:boto:Path: /dkyges-58/f 2024-04-12T03:52:43.007 DEBUG:boto:Data: 2024-04-12T03:52:43.007 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:52:43.007 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:43.007 DEBUG:boto:Port: 8000 2024-04-12T03:52:43.007 DEBUG:boto:Params: {} 2024-04-12T03:52:43.007 DEBUG:boto:Token: None 2024-04-12T03:52:43.007 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:52:43 GMT /dkyges-58/f 2024-04-12T03:52:43.008 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8EDoDQS3sY4ihDPUZToktrieX/s= 2024-04-12T03:52:43.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:52:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8EDoDQS3sY4ihDPUZToktrieX/s='} 2024-04-12T03:52:43.013 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b197a0a56d0be3b1-006618b00b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:43.013 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:43.013 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:43.013 DEBUG:boto:Method: HEAD 2024-04-12T03:52:43.014 DEBUG:boto:Path: /dkyges-58/ 2024-04-12T03:52:43.014 DEBUG:boto:Data: 2024-04-12T03:52:43.014 DEBUG:boto:Headers: {} 2024-04-12T03:52:43.014 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:43.014 DEBUG:boto:Port: 8000 2024-04-12T03:52:43.014 DEBUG:boto:Params: {} 2024-04-12T03:52:43.014 DEBUG:boto:Token: None 2024-04-12T03:52:43.014 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:43 GMT /dkyges-58/ 2024-04-12T03:52:43.014 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t1kitMlPA2IRvTSSy2gtVpHdKWE= 2024-04-12T03:52:43.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t1kitMlPA2IRvTSSy2gtVpHdKWE=', 'Content-Length': '0'} 2024-04-12T03:52:43.016 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000605a9b1ab791103c-006618b00b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:43.016 DEBUG:boto:path=/dkyges-58/g 2024-04-12T03:52:43.016 DEBUG:boto:auth_path=/dkyges-58/g 2024-04-12T03:52:43.017 DEBUG:boto:Method: PUT 2024-04-12T03:52:43.017 DEBUG:boto:Path: /dkyges-58/g 2024-04-12T03:52:43.017 DEBUG:boto:Data: 2024-04-12T03:52:43.017 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:52:43.017 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:43.017 DEBUG:boto:Port: 8000 2024-04-12T03:52:43.017 DEBUG:boto:Params: {} 2024-04-12T03:52:43.017 DEBUG:boto:Token: None 2024-04-12T03:52:43.017 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:52:43 GMT /dkyges-58/g 2024-04-12T03:52:43.018 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VUz7GbCd0q2iKvQl4SxHwa12E7g= 2024-04-12T03:52:43.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:52:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VUz7GbCd0q2iKvQl4SxHwa12E7g='} 2024-04-12T03:52:43.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009bc6d921c8999528-006618b00b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:43.063 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:52:43.063 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:52:43.063 DEBUG:boto:Method: HEAD 2024-04-12T03:52:43.063 DEBUG:boto:Path: /dkyges-58/ 2024-04-12T03:52:43.063 DEBUG:boto:Data: 2024-04-12T03:52:43.063 DEBUG:boto:Headers: {} 2024-04-12T03:52:43.063 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:43.063 DEBUG:boto:Port: 8000 2024-04-12T03:52:43.063 DEBUG:boto:Params: {} 2024-04-12T03:52:43.064 DEBUG:boto:Token: None 2024-04-12T03:52:43.064 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:52:43 GMT /dkyges-58/ 2024-04-12T03:52:43.064 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t1kitMlPA2IRvTSSy2gtVpHdKWE= 2024-04-12T03:52:43.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t1kitMlPA2IRvTSSy2gtVpHdKWE=', 'Content-Length': '0'} 2024-04-12T03:52:43.066 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1b8c36eeb6d0e92-006618b00b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:43.066 DEBUG:boto:path=/dkyges-58/h 2024-04-12T03:52:43.066 DEBUG:boto:auth_path=/dkyges-58/h 2024-04-12T03:52:43.066 DEBUG:boto:Method: PUT 2024-04-12T03:52:43.066 DEBUG:boto:Path: /dkyges-58/h 2024-04-12T03:52:43.066 DEBUG:boto:Data: 2024-04-12T03:52:43.066 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:52:43.067 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:52:43.067 DEBUG:boto:Port: 8000 2024-04-12T03:52:43.067 DEBUG:boto:Params: {} 2024-04-12T03:52:43.067 DEBUG:boto:Token: None 2024-04-12T03:52:43.067 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:52:43 GMT /dkyges-58/h 2024-04-12T03:52:43.067 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZT0IGe8ldRvcdSXcN3Hffj8WCTM= 2024-04-12T03:52:43.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:52:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZT0IGe8ldRvcdSXcN3Hffj8WCTM='} 2024-04-12T03:52:43.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e66616aed7219867-006618b00b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:52:43.073 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-58 2024-04-12T03:52:43.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:52:43.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:52:43.073 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:52:43.139 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:52:43.139 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:52:43.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:43.191+0000 7fed06f3e780 1 waiting to reach latest gen 2, on 0.. 2024-04-12T03:52:48.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:48.191+0000 7fed06f3e780 1 waiting to reach latest gen 2, on 0.. 2024-04-12T03:52:53.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:53.191+0000 7fed06f3e780 1 waiting to reach latest gen 2, on 0.. 2024-04-12T03:52:58.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:52:58.192+0000 7fed06f3e780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:53:03.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:53:03.193+0000 7fed06f3e780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:53:08.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:53:08.193+0000 7fed06f3e780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:53:13.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:53:13.194+0000 7fed06f3e780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:53:18.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:53:18.195+0000 7fed06f3e780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:53:23.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:53:23.195+0000 7fed06f3e780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:53:28.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:53:28.196+0000 7fed06f3e780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:53:33.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:53:33.196+0000 7fed06f3e780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:53:38.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:53:38.197+0000 7fed06f3e780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:53:43.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:53:43.198+0000 7fed06f3e780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:53:48.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:53:48.199+0000 7fed06f3e780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:53:53.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:53:53.199+0000 7fed06f3e780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T03:53:58.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:53:58.200+0000 7fed06f3e780 1 waiting for incremental sync to catch up: 2024-04-12T03:53:58.192 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.37.6, , ] 2024-04-12T03:53:58.192 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.37.6, 00000000002.33.6, 00000000002.72.6] 2024-04-12T03:54:03.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:54:03.202+0000 7fed06f3e780 1 waiting for incremental sync to catch up: 2024-04-12T03:54:03.192 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.37.6, , ] 2024-04-12T03:54:03.193 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.37.6, 00000000002.33.6, 00000000002.72.6] 2024-04-12T03:54:08.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:54:08.202+0000 7fed06f3e780 1 waiting for incremental sync to catch up: 2024-04-12T03:54:08.193 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.37.6, , ] 2024-04-12T03:54:08.193 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.37.6, 00000000002.33.6, 00000000002.72.6] 2024-04-12T03:54:13.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:54:13.203+0000 7fed06f3e780 1 waiting for incremental sync to catch up: 2024-04-12T03:54:13.194 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.37.6, , ] 2024-04-12T03:54:13.194 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.37.6, 00000000002.33.6, 00000000002.72.6] 2024-04-12T03:54:18.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:54:18.204+0000 7fed06f3e780 1 waiting for incremental sync to catch up: 2024-04-12T03:54:18.194 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.37.6, , ] 2024-04-12T03:54:18.195 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.37.6, 00000000002.33.6, 00000000002.72.6] 2024-04-12T03:54:23.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:54:23.204+0000 7fed06f3e780 1 waiting for incremental sync to catch up: 2024-04-12T03:54:23.195 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.37.6, , ] 2024-04-12T03:54:23.195 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.37.6, 00000000002.33.6, 00000000002.72.6] 2024-04-12T03:54:28.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:54:28.205+0000 7fed06f3e780 1 waiting for incremental sync to catch up: 2024-04-12T03:54:28.196 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.37.6, , ] 2024-04-12T03:54:28.196 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.37.6, 00000000002.33.6, 00000000002.72.6] 2024-04-12T03:54:33.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:54:33.206+0000 7fed06f3e780 1 waiting for incremental sync to catch up: 2024-04-12T03:54:33.197 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.37.6, , ] 2024-04-12T03:54:33.197 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.37.6, 00000000002.33.6, 00000000002.72.6] 2024-04-12T03:54:38.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:54:38.207+0000 7fed06f3e780 1 waiting for incremental sync to catch up: 2024-04-12T03:54:38.198 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.37.6, , ] 2024-04-12T03:54:38.198 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.37.6, 00000000002.33.6, 00000000002.72.6] 2024-04-12T03:54:43.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:54:43.208+0000 7fed06f3e780 1 waiting for incremental sync to catch up: 2024-04-12T03:54:43.199 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.37.6, , ] 2024-04-12T03:54:43.199 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.37.6, 00000000002.33.6, 00000000002.72.6] 2024-04-12T03:54:48.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:54:48.209+0000 7fed06f3e780 1 waiting for incremental sync to catch up: 2024-04-12T03:54:48.199 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.37.6, , ] 2024-04-12T03:54:48.199 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.37.6, 00000000002.33.6, 00000000002.72.6] 2024-04-12T03:54:53.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:54:53.210+0000 7fed06f3e780 1 waiting for incremental sync to catch up: 2024-04-12T03:54:53.201 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.37.6, , ] 2024-04-12T03:54:53.201 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.37.6, 00000000002.33.6, 00000000002.72.6] 2024-04-12T03:54:58.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:54:58.210+0000 7fed06f3e780 1 bucket sync caught up with source: 2024-04-12T03:54:58.201 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.37.6, 00000000002.33.6, 00000000002.72.6] 2024-04-12T03:54:58.201 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.37.6, 00000000002.33.6, 00000000002.72.6] 2024-04-12T03:54:58.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:54:58.210+0000 7fed06f3e780 0 bucket checkpoint complete 2024-04-12T03:54:58.220 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-58 2024-04-12T03:54:58.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:54:58.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:54:58.221 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:54:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:54:58.288 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:54:59.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:54:59.342+0000 7f01e501a780 1 waiting for incremental sync to catch up: 2024-04-12T03:54:59.332 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.1878.6, , , 00000000001.2549.6, 00000000001.1403.6, , , 00000000001.2586.6, , 00000000002.1899.6] 2024-04-12T03:54:59.332 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.1888.6, , , 00000000001.2549.6, 00000000001.1403.6, , , 00000000002.2596.6, , 00000000002.1899.6] 2024-04-12T03:55:04.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:55:04.343+0000 7f01e501a780 1 waiting for incremental sync to catch up: 2024-04-12T03:55:04.334 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.1888.6, , , 00000000001.2549.6, 00000000001.1403.6, , , 00000000001.2586.6, , 00000000002.1899.6] 2024-04-12T03:55:04.334 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.1888.6, , , 00000000001.2549.6, 00000000001.1403.6, , , 00000000002.2596.6, , 00000000002.1899.6] 2024-04-12T03:55:09.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:55:09.344+0000 7f01e501a780 1 bucket sync caught up with source: 2024-04-12T03:55:09.335 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.1888.6, , , 00000000001.2549.6, 00000000001.1403.6, , , 00000000002.2596.6, , 00000000002.1899.6] 2024-04-12T03:55:09.335 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.1888.6, , , 00000000001.2549.6, 00000000001.1403.6, , , 00000000002.2596.6, , 00000000002.1899.6] 2024-04-12T03:55:09.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:55:09.344+0000 7f01e501a780 0 bucket checkpoint complete 2024-04-12T03:55:09.353 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-58 zones={a2, a1} 2024-04-12T03:55:09.353 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.353 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.354 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.354 DEBUG:boto:Path: /dkyges-58/ 2024-04-12T03:55:09.354 DEBUG:boto:Data: 2024-04-12T03:55:09.354 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.354 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.354 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.354 DEBUG:boto:Params: {} 2024-04-12T03:55:09.355 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:55:09.355 DEBUG:boto:Token: None 2024-04-12T03:55:09.356 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.356 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlOQ4VQx1NYIB2D4AefoAMza564= 2024-04-12T03:55:09.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlOQ4VQx1NYIB2D4AefoAMza564=', 'Content-Length': '0'} 2024-04-12T03:55:09.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000705627cc8f47656-006618b09d-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.370 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.370 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.370 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.370 DEBUG:boto:Path: /dkyges-58/ 2024-04-12T03:55:09.370 DEBUG:boto:Data: 2024-04-12T03:55:09.370 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.370 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.370 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.371 DEBUG:boto:Params: {} 2024-04-12T03:55:09.371 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:55:09.371 DEBUG:boto:Token: None 2024-04-12T03:55:09.371 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.371 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlOQ4VQx1NYIB2D4AefoAMza564= 2024-04-12T03:55:09.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlOQ4VQx1NYIB2D4AefoAMza564=', 'Content-Length': '0'} 2024-04-12T03:55:09.400 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb7c75b6c23a2def-006618b09d-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.401 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:55:09.401 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.401 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.401 DEBUG:boto:path=/dkyges-58/?versions&max-keys=999 2024-04-12T03:55:09.401 DEBUG:boto:auth_path=/dkyges-58/?versions&max-keys=999 2024-04-12T03:55:09.401 DEBUG:boto:Method: GET 2024-04-12T03:55:09.402 DEBUG:boto:Path: /dkyges-58/?versions&max-keys=999 2024-04-12T03:55:09.402 DEBUG:boto:Data: 2024-04-12T03:55:09.402 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.402 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.402 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.402 DEBUG:boto:Params: {} 2024-04-12T03:55:09.402 DEBUG:boto:Token: None 2024-04-12T03:55:09.402 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/?versions 2024-04-12T03:55:09.402 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WPmxu25K7rCx61wRzmjCvxJ2eAo= 2024-04-12T03:55:09.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WPmxu25K7rCx61wRzmjCvxJ2eAo=', 'Content-Length': '0'} 2024-04-12T03:55:09.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a4603faac4756a0-006618b09d-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.405 DEBUG:boto:b'dkyges-58999falseanulltrue2024-04-12T03:52:21.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:52:21.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:52:21.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:52:21.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:52:43.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:52:43.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:52:43.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:52:43.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.406 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:55:09.406 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:55:09.406 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:55:09.407 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:55:09.407 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-12T03:55:09.407 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-12T03:55:09.407 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-12T03:55:09.407 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-12T03:55:09.407 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:55:09.407 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.407 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.407 DEBUG:boto:path=/dkyges-58/?versions&max-keys=999 2024-04-12T03:55:09.407 DEBUG:boto:auth_path=/dkyges-58/?versions&max-keys=999 2024-04-12T03:55:09.408 DEBUG:boto:Method: GET 2024-04-12T03:55:09.408 DEBUG:boto:Path: /dkyges-58/?versions&max-keys=999 2024-04-12T03:55:09.408 DEBUG:boto:Data: 2024-04-12T03:55:09.408 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.408 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.408 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.408 DEBUG:boto:Params: {} 2024-04-12T03:55:09.408 DEBUG:boto:Token: None 2024-04-12T03:55:09.408 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/?versions 2024-04-12T03:55:09.408 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WPmxu25K7rCx61wRzmjCvxJ2eAo= 2024-04-12T03:55:09.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WPmxu25K7rCx61wRzmjCvxJ2eAo=', 'Content-Length': '0'} 2024-04-12T03:55:09.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9990ae496d372d3-006618b09d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.411 DEBUG:boto:b'dkyges-58999falseanulltrue2024-04-12T03:52:21.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:52:21.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:52:21.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:52:21.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:52:43.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:52:43.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:52:43.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:52:43.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.411 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:55:09.411 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:55:09.412 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:55:09.412 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:55:09.412 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-12T03:55:09.412 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-12T03:55:09.412 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-12T03:55:09.412 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-12T03:55:09.412 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.412 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.412 DEBUG:boto:path=/dkyges-58/?versions&max-keys=999 2024-04-12T03:55:09.412 DEBUG:boto:auth_path=/dkyges-58/?versions&max-keys=999 2024-04-12T03:55:09.412 DEBUG:boto:Method: GET 2024-04-12T03:55:09.413 DEBUG:boto:Path: /dkyges-58/?versions&max-keys=999 2024-04-12T03:55:09.413 DEBUG:boto:Data: 2024-04-12T03:55:09.413 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.413 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.413 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.413 DEBUG:boto:Params: {} 2024-04-12T03:55:09.413 DEBUG:boto:Token: None 2024-04-12T03:55:09.413 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/?versions 2024-04-12T03:55:09.413 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WPmxu25K7rCx61wRzmjCvxJ2eAo= 2024-04-12T03:55:09.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WPmxu25K7rCx61wRzmjCvxJ2eAo=', 'Content-Length': '0'} 2024-04-12T03:55:09.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b81fc9827d50d146-006618b09d-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.416 DEBUG:boto:b'dkyges-58999falseanulltrue2024-04-12T03:52:21.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:52:21.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:52:21.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:52:21.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:52:43.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:52:43.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:52:43.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:52:43.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.417 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.417 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.417 DEBUG:boto:path=/dkyges-58/?versions&max-keys=999 2024-04-12T03:55:09.417 DEBUG:boto:auth_path=/dkyges-58/?versions&max-keys=999 2024-04-12T03:55:09.417 DEBUG:boto:Method: GET 2024-04-12T03:55:09.417 DEBUG:boto:Path: /dkyges-58/?versions&max-keys=999 2024-04-12T03:55:09.417 DEBUG:boto:Data: 2024-04-12T03:55:09.417 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.417 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.417 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.418 DEBUG:boto:Params: {} 2024-04-12T03:55:09.418 DEBUG:boto:Token: None 2024-04-12T03:55:09.418 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/?versions 2024-04-12T03:55:09.418 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WPmxu25K7rCx61wRzmjCvxJ2eAo= 2024-04-12T03:55:09.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WPmxu25K7rCx61wRzmjCvxJ2eAo=', 'Content-Length': '0'} 2024-04-12T03:55:09.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035a3e706e09a1757-006618b09d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.420 DEBUG:boto:b'dkyges-58999falseanulltrue2024-04-12T03:52:21.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:52:21.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:52:21.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:52:21.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:52:43.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:52:43.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:52:43.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:52:43.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:55:09.421 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:55:09.421 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:55:09.421 DEBUG:boto:path=/dkyges-58/a?versionId=null 2024-04-12T03:55:09.421 DEBUG:boto:auth_path=/dkyges-58/a?versionId=null 2024-04-12T03:55:09.421 DEBUG:boto:Method: GET 2024-04-12T03:55:09.422 DEBUG:boto:Path: /dkyges-58/a?versionId=null 2024-04-12T03:55:09.422 DEBUG:boto:Data: 2024-04-12T03:55:09.422 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.422 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.422 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.422 DEBUG:boto:Params: {} 2024-04-12T03:55:09.422 DEBUG:boto:Token: None 2024-04-12T03:55:09.422 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/a?versionId=null 2024-04-12T03:55:09.422 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q1F2Ro2akBWSYWLHtbkXG+AdYww= 2024-04-12T03:55:09.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q1F2Ro2akBWSYWLHtbkXG+AdYww=', 'Content-Length': '0'} 2024-04-12T03:55:09.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f896c064800adbac-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.425 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:55:09.425 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:55:09.425 DEBUG:boto:path=/dkyges-58/a?versionId=null 2024-04-12T03:55:09.425 DEBUG:boto:auth_path=/dkyges-58/a?versionId=null 2024-04-12T03:55:09.425 DEBUG:boto:Method: GET 2024-04-12T03:55:09.425 DEBUG:boto:Path: /dkyges-58/a?versionId=null 2024-04-12T03:55:09.425 DEBUG:boto:Data: 2024-04-12T03:55:09.425 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.425 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.425 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.425 DEBUG:boto:Params: {} 2024-04-12T03:55:09.426 DEBUG:boto:Token: None 2024-04-12T03:55:09.426 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/a?versionId=null 2024-04-12T03:55:09.426 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q1F2Ro2akBWSYWLHtbkXG+AdYww= 2024-04-12T03:55:09.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q1F2Ro2akBWSYWLHtbkXG+AdYww=', 'Content-Length': '0'} 2024-04-12T03:55:09.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2c8a6025af70f64-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.428 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:55:09.428 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:55:09.428 DEBUG:boto:path=/dkyges-58/a?versionId=null 2024-04-12T03:55:09.428 DEBUG:boto:auth_path=/dkyges-58/a?versionId=null 2024-04-12T03:55:09.429 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.429 DEBUG:boto:Path: /dkyges-58/a?versionId=null 2024-04-12T03:55:09.429 DEBUG:boto:Data: 2024-04-12T03:55:09.429 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.429 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.429 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.429 DEBUG:boto:Params: {} 2024-04-12T03:55:09.429 DEBUG:boto:Token: None 2024-04-12T03:55:09.429 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/a?versionId=null 2024-04-12T03:55:09.429 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JyJvbDm2CHyr/vMw3fw7gQPqvhk= 2024-04-12T03:55:09.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JyJvbDm2CHyr/vMw3fw7gQPqvhk=', 'Content-Length': '0'} 2024-04-12T03:55:09.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dcb08dd023b0daf-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.431 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:55:09.431 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:55:09.431 DEBUG:boto:path=/dkyges-58/a?versionId=null 2024-04-12T03:55:09.432 DEBUG:boto:auth_path=/dkyges-58/a?versionId=null 2024-04-12T03:55:09.432 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.432 DEBUG:boto:Path: /dkyges-58/a?versionId=null 2024-04-12T03:55:09.432 DEBUG:boto:Data: 2024-04-12T03:55:09.432 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.432 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.432 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.432 DEBUG:boto:Params: {} 2024-04-12T03:55:09.432 DEBUG:boto:Token: None 2024-04-12T03:55:09.432 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/a?versionId=null 2024-04-12T03:55:09.432 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JyJvbDm2CHyr/vMw3fw7gQPqvhk= 2024-04-12T03:55:09.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JyJvbDm2CHyr/vMw3fw7gQPqvhk=', 'Content-Length': '0'} 2024-04-12T03:55:09.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ee6d60d2e433a29-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:55:09.435 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:55:09.435 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:55:09.435 DEBUG:boto:path=/dkyges-58/a?versionId=null 2024-04-12T03:55:09.435 DEBUG:boto:auth_path=/dkyges-58/a?versionId=null 2024-04-12T03:55:09.435 DEBUG:boto:Method: GET 2024-04-12T03:55:09.435 DEBUG:boto:Path: /dkyges-58/a?versionId=null 2024-04-12T03:55:09.435 DEBUG:boto:Data: 2024-04-12T03:55:09.435 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.435 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.435 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.435 DEBUG:boto:Params: {} 2024-04-12T03:55:09.435 DEBUG:boto:Token: None 2024-04-12T03:55:09.436 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/a?versionId=null 2024-04-12T03:55:09.436 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q1F2Ro2akBWSYWLHtbkXG+AdYww= 2024-04-12T03:55:09.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q1F2Ro2akBWSYWLHtbkXG+AdYww=', 'Content-Length': '0'} 2024-04-12T03:55:09.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f785854d5fd300ee-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.438 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:55:09.438 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:55:09.438 DEBUG:boto:path=/dkyges-58/a?versionId=null 2024-04-12T03:55:09.438 DEBUG:boto:auth_path=/dkyges-58/a?versionId=null 2024-04-12T03:55:09.438 DEBUG:boto:Method: GET 2024-04-12T03:55:09.438 DEBUG:boto:Path: /dkyges-58/a?versionId=null 2024-04-12T03:55:09.438 DEBUG:boto:Data: 2024-04-12T03:55:09.438 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.438 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.438 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.439 DEBUG:boto:Params: {} 2024-04-12T03:55:09.439 DEBUG:boto:Token: None 2024-04-12T03:55:09.439 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/a?versionId=null 2024-04-12T03:55:09.439 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q1F2Ro2akBWSYWLHtbkXG+AdYww= 2024-04-12T03:55:09.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q1F2Ro2akBWSYWLHtbkXG+AdYww=', 'Content-Length': '0'} 2024-04-12T03:55:09.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a26b155fbeff9fc-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.441 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.441 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.441 DEBUG:boto:path=/dkyges-58/?prefix=a 2024-04-12T03:55:09.441 DEBUG:boto:auth_path=/dkyges-58/?prefix=a 2024-04-12T03:55:09.442 DEBUG:boto:Method: GET 2024-04-12T03:55:09.442 DEBUG:boto:Path: /dkyges-58/?prefix=a 2024-04-12T03:55:09.442 DEBUG:boto:Data: 2024-04-12T03:55:09.442 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.442 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.442 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.442 DEBUG:boto:Params: {} 2024-04-12T03:55:09.442 DEBUG:boto:Token: None 2024-04-12T03:55:09.442 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.442 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e557d1843bb7c69-006618b09d-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.445 DEBUG:boto:b'dkyges-58a1000falsea2024-04-12T03:52:21.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.445 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.445 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.445 DEBUG:boto:path=/dkyges-58/?prefix=a 2024-04-12T03:55:09.446 DEBUG:boto:auth_path=/dkyges-58/?prefix=a 2024-04-12T03:55:09.446 DEBUG:boto:Method: GET 2024-04-12T03:55:09.446 DEBUG:boto:Path: /dkyges-58/?prefix=a 2024-04-12T03:55:09.446 DEBUG:boto:Data: 2024-04-12T03:55:09.446 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.446 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.446 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.446 DEBUG:boto:Params: {} 2024-04-12T03:55:09.446 DEBUG:boto:Token: None 2024-04-12T03:55:09.446 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.447 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b27e33492cbbf260-006618b09d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.449 DEBUG:boto:b'dkyges-58a1000falsea2024-04-12T03:52:21.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.449 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:55:09.449 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:55:09.449 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.449 DEBUG:boto:Path: /dkyges-58/a 2024-04-12T03:55:09.449 DEBUG:boto:Data: 2024-04-12T03:55:09.449 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.449 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.449 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.450 DEBUG:boto:Params: {} 2024-04-12T03:55:09.450 DEBUG:boto:Token: None 2024-04-12T03:55:09.450 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/a 2024-04-12T03:55:09.450 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6nXvFlqGdIm6Wpa7Iibiw0n4jQw= 2024-04-12T03:55:09.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6nXvFlqGdIm6Wpa7Iibiw0n4jQw=', 'Content-Length': '0'} 2024-04-12T03:55:09.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047f880e1f7ca37bf-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.452 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:55:09.452 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:55:09.452 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.452 DEBUG:boto:Path: /dkyges-58/a 2024-04-12T03:55:09.452 DEBUG:boto:Data: 2024-04-12T03:55:09.452 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.452 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.452 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.452 DEBUG:boto:Params: {} 2024-04-12T03:55:09.452 DEBUG:boto:Token: None 2024-04-12T03:55:09.453 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/a 2024-04-12T03:55:09.453 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6nXvFlqGdIm6Wpa7Iibiw0n4jQw= 2024-04-12T03:55:09.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6nXvFlqGdIm6Wpa7Iibiw0n4jQw=', 'Content-Length': '0'} 2024-04-12T03:55:09.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f546882efe17384f-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:55:09.458 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:55:09.458 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:55:09.458 DEBUG:boto:Method: GET 2024-04-12T03:55:09.458 DEBUG:boto:Path: /dkyges-58/a 2024-04-12T03:55:09.458 DEBUG:boto:Data: 2024-04-12T03:55:09.458 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.458 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.459 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.459 DEBUG:boto:Params: {} 2024-04-12T03:55:09.459 DEBUG:boto:Token: None 2024-04-12T03:55:09.459 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/a 2024-04-12T03:55:09.459 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BgXfHzj3pojxvgjT8XBTqdGB1pU= 2024-04-12T03:55:09.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BgXfHzj3pojxvgjT8XBTqdGB1pU=', 'Content-Length': '0'} 2024-04-12T03:55:09.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000280626e837b0a369-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.461 DEBUG:boto:path=/dkyges-58/a 2024-04-12T03:55:09.461 DEBUG:boto:auth_path=/dkyges-58/a 2024-04-12T03:55:09.461 DEBUG:boto:Method: GET 2024-04-12T03:55:09.461 DEBUG:boto:Path: /dkyges-58/a 2024-04-12T03:55:09.461 DEBUG:boto:Data: 2024-04-12T03:55:09.462 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.462 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.462 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.462 DEBUG:boto:Params: {} 2024-04-12T03:55:09.462 DEBUG:boto:Token: None 2024-04-12T03:55:09.462 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/a 2024-04-12T03:55:09.462 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BgXfHzj3pojxvgjT8XBTqdGB1pU= 2024-04-12T03:55:09.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BgXfHzj3pojxvgjT8XBTqdGB1pU=', 'Content-Length': '0'} 2024-04-12T03:55:09.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a05eb169502ab509-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.506 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.506 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.507 DEBUG:boto:path=/dkyges-58/?prefix=a 2024-04-12T03:55:09.507 DEBUG:boto:auth_path=/dkyges-58/?prefix=a 2024-04-12T03:55:09.507 DEBUG:boto:Method: GET 2024-04-12T03:55:09.507 DEBUG:boto:Path: /dkyges-58/?prefix=a 2024-04-12T03:55:09.507 DEBUG:boto:Data: 2024-04-12T03:55:09.507 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.507 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.507 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.507 DEBUG:boto:Params: {} 2024-04-12T03:55:09.507 DEBUG:boto:Token: None 2024-04-12T03:55:09.507 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.508 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6a862f67753874d-006618b09d-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.510 DEBUG:boto:b'dkyges-58a1000falsea2024-04-12T03:52:21.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.511 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.511 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.511 DEBUG:boto:path=/dkyges-58/?prefix=a 2024-04-12T03:55:09.511 DEBUG:boto:auth_path=/dkyges-58/?prefix=a 2024-04-12T03:55:09.511 DEBUG:boto:Method: GET 2024-04-12T03:55:09.511 DEBUG:boto:Path: /dkyges-58/?prefix=a 2024-04-12T03:55:09.511 DEBUG:boto:Data: 2024-04-12T03:55:09.511 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.511 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.511 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.511 DEBUG:boto:Params: {} 2024-04-12T03:55:09.512 DEBUG:boto:Token: None 2024-04-12T03:55:09.512 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.512 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018df9ffc1f82cfa7-006618b09d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.514 DEBUG:boto:b'dkyges-58a1000falsea2024-04-12T03:52:21.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:55:09.514 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:55:09.514 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:55:09.515 DEBUG:boto:path=/dkyges-58/b?versionId=null 2024-04-12T03:55:09.515 DEBUG:boto:auth_path=/dkyges-58/b?versionId=null 2024-04-12T03:55:09.515 DEBUG:boto:Method: GET 2024-04-12T03:55:09.515 DEBUG:boto:Path: /dkyges-58/b?versionId=null 2024-04-12T03:55:09.515 DEBUG:boto:Data: 2024-04-12T03:55:09.515 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.515 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.515 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.515 DEBUG:boto:Params: {} 2024-04-12T03:55:09.515 DEBUG:boto:Token: None 2024-04-12T03:55:09.516 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/b?versionId=null 2024-04-12T03:55:09.516 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MqVswMPWpeWcYA1dSq0jcj6tKhU= 2024-04-12T03:55:09.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MqVswMPWpeWcYA1dSq0jcj6tKhU=', 'Content-Length': '0'} 2024-04-12T03:55:09.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ba7278a878dab8c-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.558 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:55:09.558 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:55:09.558 DEBUG:boto:path=/dkyges-58/b?versionId=null 2024-04-12T03:55:09.559 DEBUG:boto:auth_path=/dkyges-58/b?versionId=null 2024-04-12T03:55:09.559 DEBUG:boto:Method: GET 2024-04-12T03:55:09.559 DEBUG:boto:Path: /dkyges-58/b?versionId=null 2024-04-12T03:55:09.559 DEBUG:boto:Data: 2024-04-12T03:55:09.559 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.559 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.559 DEBUG:boto:Params: {} 2024-04-12T03:55:09.559 DEBUG:boto:Token: None 2024-04-12T03:55:09.559 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/b?versionId=null 2024-04-12T03:55:09.559 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MqVswMPWpeWcYA1dSq0jcj6tKhU= 2024-04-12T03:55:09.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MqVswMPWpeWcYA1dSq0jcj6tKhU=', 'Content-Length': '0'} 2024-04-12T03:55:09.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005403df21c44e429a-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.562 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:55:09.562 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:55:09.562 DEBUG:boto:path=/dkyges-58/b?versionId=null 2024-04-12T03:55:09.562 DEBUG:boto:auth_path=/dkyges-58/b?versionId=null 2024-04-12T03:55:09.562 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.562 DEBUG:boto:Path: /dkyges-58/b?versionId=null 2024-04-12T03:55:09.562 DEBUG:boto:Data: 2024-04-12T03:55:09.562 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.562 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.562 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.562 DEBUG:boto:Params: {} 2024-04-12T03:55:09.563 DEBUG:boto:Token: None 2024-04-12T03:55:09.563 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/b?versionId=null 2024-04-12T03:55:09.563 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y3GUbupbHiQ9aoL/Hw/f4F2ulQo= 2024-04-12T03:55:09.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y3GUbupbHiQ9aoL/Hw/f4F2ulQo=', 'Content-Length': '0'} 2024-04-12T03:55:09.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067aa8428b8ec9fc8-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.565 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:55:09.565 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:55:09.565 DEBUG:boto:path=/dkyges-58/b?versionId=null 2024-04-12T03:55:09.565 DEBUG:boto:auth_path=/dkyges-58/b?versionId=null 2024-04-12T03:55:09.565 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.566 DEBUG:boto:Path: /dkyges-58/b?versionId=null 2024-04-12T03:55:09.566 DEBUG:boto:Data: 2024-04-12T03:55:09.566 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.566 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.566 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.566 DEBUG:boto:Params: {} 2024-04-12T03:55:09.566 DEBUG:boto:Token: None 2024-04-12T03:55:09.566 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/b?versionId=null 2024-04-12T03:55:09.566 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y3GUbupbHiQ9aoL/Hw/f4F2ulQo= 2024-04-12T03:55:09.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y3GUbupbHiQ9aoL/Hw/f4F2ulQo=', 'Content-Length': '0'} 2024-04-12T03:55:09.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000713d2619862dcc44-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:55:09.569 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:55:09.569 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:55:09.569 DEBUG:boto:path=/dkyges-58/b?versionId=null 2024-04-12T03:55:09.569 DEBUG:boto:auth_path=/dkyges-58/b?versionId=null 2024-04-12T03:55:09.569 DEBUG:boto:Method: GET 2024-04-12T03:55:09.569 DEBUG:boto:Path: /dkyges-58/b?versionId=null 2024-04-12T03:55:09.569 DEBUG:boto:Data: 2024-04-12T03:55:09.569 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.569 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.570 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.570 DEBUG:boto:Params: {} 2024-04-12T03:55:09.570 DEBUG:boto:Token: None 2024-04-12T03:55:09.570 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/b?versionId=null 2024-04-12T03:55:09.570 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MqVswMPWpeWcYA1dSq0jcj6tKhU= 2024-04-12T03:55:09.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MqVswMPWpeWcYA1dSq0jcj6tKhU=', 'Content-Length': '0'} 2024-04-12T03:55:09.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed308517642544b9-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.572 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:55:09.572 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:55:09.572 DEBUG:boto:path=/dkyges-58/b?versionId=null 2024-04-12T03:55:09.573 DEBUG:boto:auth_path=/dkyges-58/b?versionId=null 2024-04-12T03:55:09.573 DEBUG:boto:Method: GET 2024-04-12T03:55:09.573 DEBUG:boto:Path: /dkyges-58/b?versionId=null 2024-04-12T03:55:09.573 DEBUG:boto:Data: 2024-04-12T03:55:09.573 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.573 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.573 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.573 DEBUG:boto:Params: {} 2024-04-12T03:55:09.573 DEBUG:boto:Token: None 2024-04-12T03:55:09.573 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/b?versionId=null 2024-04-12T03:55:09.573 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MqVswMPWpeWcYA1dSq0jcj6tKhU= 2024-04-12T03:55:09.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MqVswMPWpeWcYA1dSq0jcj6tKhU=', 'Content-Length': '0'} 2024-04-12T03:55:09.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1ba0ad613566df1-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.576 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.576 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.576 DEBUG:boto:path=/dkyges-58/?prefix=b 2024-04-12T03:55:09.576 DEBUG:boto:auth_path=/dkyges-58/?prefix=b 2024-04-12T03:55:09.576 DEBUG:boto:Method: GET 2024-04-12T03:55:09.576 DEBUG:boto:Path: /dkyges-58/?prefix=b 2024-04-12T03:55:09.576 DEBUG:boto:Data: 2024-04-12T03:55:09.577 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.577 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.577 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.577 DEBUG:boto:Params: {} 2024-04-12T03:55:09.577 DEBUG:boto:Token: None 2024-04-12T03:55:09.577 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.577 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b68e4f7997d0ee10-006618b09d-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.580 DEBUG:boto:b'dkyges-58b1000falseb2024-04-12T03:52:21.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.580 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.580 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.580 DEBUG:boto:path=/dkyges-58/?prefix=b 2024-04-12T03:55:09.580 DEBUG:boto:auth_path=/dkyges-58/?prefix=b 2024-04-12T03:55:09.580 DEBUG:boto:Method: GET 2024-04-12T03:55:09.580 DEBUG:boto:Path: /dkyges-58/?prefix=b 2024-04-12T03:55:09.580 DEBUG:boto:Data: 2024-04-12T03:55:09.580 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.580 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.581 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.581 DEBUG:boto:Params: {} 2024-04-12T03:55:09.581 DEBUG:boto:Token: None 2024-04-12T03:55:09.581 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.581 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d5c2e91e87bf508-006618b09d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.583 DEBUG:boto:b'dkyges-58b1000falseb2024-04-12T03:52:21.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.583 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:55:09.583 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:55:09.583 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.584 DEBUG:boto:Path: /dkyges-58/b 2024-04-12T03:55:09.584 DEBUG:boto:Data: 2024-04-12T03:55:09.584 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.584 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.584 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.584 DEBUG:boto:Params: {} 2024-04-12T03:55:09.584 DEBUG:boto:Token: None 2024-04-12T03:55:09.584 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/b 2024-04-12T03:55:09.584 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuV4Ub8l6s7Bebf+miMSakbuw8A= 2024-04-12T03:55:09.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuV4Ub8l6s7Bebf+miMSakbuw8A=', 'Content-Length': '0'} 2024-04-12T03:55:09.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c58204529fdf7d9-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.586 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:55:09.586 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:55:09.586 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.587 DEBUG:boto:Path: /dkyges-58/b 2024-04-12T03:55:09.587 DEBUG:boto:Data: 2024-04-12T03:55:09.587 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.587 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.587 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.587 DEBUG:boto:Params: {} 2024-04-12T03:55:09.587 DEBUG:boto:Token: None 2024-04-12T03:55:09.587 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/b 2024-04-12T03:55:09.587 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuV4Ub8l6s7Bebf+miMSakbuw8A= 2024-04-12T03:55:09.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuV4Ub8l6s7Bebf+miMSakbuw8A=', 'Content-Length': '0'} 2024-04-12T03:55:09.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000446d88d2d4ae9121-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:55:09.589 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:55:09.590 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:55:09.590 DEBUG:boto:Method: GET 2024-04-12T03:55:09.590 DEBUG:boto:Path: /dkyges-58/b 2024-04-12T03:55:09.590 DEBUG:boto:Data: 2024-04-12T03:55:09.590 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.590 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.590 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.590 DEBUG:boto:Params: {} 2024-04-12T03:55:09.590 DEBUG:boto:Token: None 2024-04-12T03:55:09.590 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/b 2024-04-12T03:55:09.590 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:80phI8dPtZ0AOtIUZyr+47AzEIw= 2024-04-12T03:55:09.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:80phI8dPtZ0AOtIUZyr+47AzEIw=', 'Content-Length': '0'} 2024-04-12T03:55:09.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000503ea4ddc07a9b14-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.592 DEBUG:boto:path=/dkyges-58/b 2024-04-12T03:55:09.592 DEBUG:boto:auth_path=/dkyges-58/b 2024-04-12T03:55:09.592 DEBUG:boto:Method: GET 2024-04-12T03:55:09.593 DEBUG:boto:Path: /dkyges-58/b 2024-04-12T03:55:09.593 DEBUG:boto:Data: 2024-04-12T03:55:09.593 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.593 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.593 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.593 DEBUG:boto:Params: {} 2024-04-12T03:55:09.593 DEBUG:boto:Token: None 2024-04-12T03:55:09.593 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/b 2024-04-12T03:55:09.593 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:80phI8dPtZ0AOtIUZyr+47AzEIw= 2024-04-12T03:55:09.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:80phI8dPtZ0AOtIUZyr+47AzEIw=', 'Content-Length': '0'} 2024-04-12T03:55:09.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000960b17fb8c7d06a5-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.638 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.638 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.639 DEBUG:boto:path=/dkyges-58/?prefix=b 2024-04-12T03:55:09.639 DEBUG:boto:auth_path=/dkyges-58/?prefix=b 2024-04-12T03:55:09.639 DEBUG:boto:Method: GET 2024-04-12T03:55:09.639 DEBUG:boto:Path: /dkyges-58/?prefix=b 2024-04-12T03:55:09.639 DEBUG:boto:Data: 2024-04-12T03:55:09.639 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.639 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.639 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.639 DEBUG:boto:Params: {} 2024-04-12T03:55:09.639 DEBUG:boto:Token: None 2024-04-12T03:55:09.639 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.640 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfc70ab54a06cade-006618b09d-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.642 DEBUG:boto:b'dkyges-58b1000falseb2024-04-12T03:52:21.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.643 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.643 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.643 DEBUG:boto:path=/dkyges-58/?prefix=b 2024-04-12T03:55:09.643 DEBUG:boto:auth_path=/dkyges-58/?prefix=b 2024-04-12T03:55:09.643 DEBUG:boto:Method: GET 2024-04-12T03:55:09.643 DEBUG:boto:Path: /dkyges-58/?prefix=b 2024-04-12T03:55:09.643 DEBUG:boto:Data: 2024-04-12T03:55:09.643 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.643 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.643 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.643 DEBUG:boto:Params: {} 2024-04-12T03:55:09.644 DEBUG:boto:Token: None 2024-04-12T03:55:09.644 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.644 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b51e6e24ed4093d-006618b09d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.646 DEBUG:boto:b'dkyges-58b1000falseb2024-04-12T03:52:21.846Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:55:09.646 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:55:09.646 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:55:09.646 DEBUG:boto:path=/dkyges-58/c?versionId=null 2024-04-12T03:55:09.646 DEBUG:boto:auth_path=/dkyges-58/c?versionId=null 2024-04-12T03:55:09.646 DEBUG:boto:Method: GET 2024-04-12T03:55:09.647 DEBUG:boto:Path: /dkyges-58/c?versionId=null 2024-04-12T03:55:09.647 DEBUG:boto:Data: 2024-04-12T03:55:09.647 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.647 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.647 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.647 DEBUG:boto:Params: {} 2024-04-12T03:55:09.647 DEBUG:boto:Token: None 2024-04-12T03:55:09.647 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/c?versionId=null 2024-04-12T03:55:09.647 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jL0XAvUHGEgDuXWQ5dQmIdoGnDk= 2024-04-12T03:55:09.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jL0XAvUHGEgDuXWQ5dQmIdoGnDk=', 'Content-Length': '0'} 2024-04-12T03:55:09.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004420ed7c8b0ca7f7-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.690 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:55:09.690 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:55:09.691 DEBUG:boto:path=/dkyges-58/c?versionId=null 2024-04-12T03:55:09.691 DEBUG:boto:auth_path=/dkyges-58/c?versionId=null 2024-04-12T03:55:09.691 DEBUG:boto:Method: GET 2024-04-12T03:55:09.691 DEBUG:boto:Path: /dkyges-58/c?versionId=null 2024-04-12T03:55:09.691 DEBUG:boto:Data: 2024-04-12T03:55:09.691 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.691 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.691 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.691 DEBUG:boto:Params: {} 2024-04-12T03:55:09.691 DEBUG:boto:Token: None 2024-04-12T03:55:09.692 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/c?versionId=null 2024-04-12T03:55:09.692 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jL0XAvUHGEgDuXWQ5dQmIdoGnDk= 2024-04-12T03:55:09.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jL0XAvUHGEgDuXWQ5dQmIdoGnDk=', 'Content-Length': '0'} 2024-04-12T03:55:09.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7c8eb7530949e4b-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.694 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:55:09.694 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:55:09.694 DEBUG:boto:path=/dkyges-58/c?versionId=null 2024-04-12T03:55:09.694 DEBUG:boto:auth_path=/dkyges-58/c?versionId=null 2024-04-12T03:55:09.694 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.694 DEBUG:boto:Path: /dkyges-58/c?versionId=null 2024-04-12T03:55:09.695 DEBUG:boto:Data: 2024-04-12T03:55:09.695 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.695 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.695 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.695 DEBUG:boto:Params: {} 2024-04-12T03:55:09.695 DEBUG:boto:Token: None 2024-04-12T03:55:09.695 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/c?versionId=null 2024-04-12T03:55:09.695 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7x60YIjpFjBRyi3Ky2g487sT5ew= 2024-04-12T03:55:09.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7x60YIjpFjBRyi3Ky2g487sT5ew=', 'Content-Length': '0'} 2024-04-12T03:55:09.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a4a63bd2a8c307d-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.697 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:55:09.698 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:55:09.698 DEBUG:boto:path=/dkyges-58/c?versionId=null 2024-04-12T03:55:09.698 DEBUG:boto:auth_path=/dkyges-58/c?versionId=null 2024-04-12T03:55:09.698 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.698 DEBUG:boto:Path: /dkyges-58/c?versionId=null 2024-04-12T03:55:09.698 DEBUG:boto:Data: 2024-04-12T03:55:09.698 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.699 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.699 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.699 DEBUG:boto:Params: {} 2024-04-12T03:55:09.699 DEBUG:boto:Token: None 2024-04-12T03:55:09.699 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/c?versionId=null 2024-04-12T03:55:09.699 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7x60YIjpFjBRyi3Ky2g487sT5ew= 2024-04-12T03:55:09.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7x60YIjpFjBRyi3Ky2g487sT5ew=', 'Content-Length': '0'} 2024-04-12T03:55:09.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e826c83ff3b262c3-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:55:09.702 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:55:09.702 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:55:09.702 DEBUG:boto:path=/dkyges-58/c?versionId=null 2024-04-12T03:55:09.702 DEBUG:boto:auth_path=/dkyges-58/c?versionId=null 2024-04-12T03:55:09.702 DEBUG:boto:Method: GET 2024-04-12T03:55:09.702 DEBUG:boto:Path: /dkyges-58/c?versionId=null 2024-04-12T03:55:09.702 DEBUG:boto:Data: 2024-04-12T03:55:09.702 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.702 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.703 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.703 DEBUG:boto:Params: {} 2024-04-12T03:55:09.703 DEBUG:boto:Token: None 2024-04-12T03:55:09.703 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/c?versionId=null 2024-04-12T03:55:09.703 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jL0XAvUHGEgDuXWQ5dQmIdoGnDk= 2024-04-12T03:55:09.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jL0XAvUHGEgDuXWQ5dQmIdoGnDk=', 'Content-Length': '0'} 2024-04-12T03:55:09.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c831a3bcc52627d2-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.705 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:55:09.705 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:55:09.705 DEBUG:boto:path=/dkyges-58/c?versionId=null 2024-04-12T03:55:09.705 DEBUG:boto:auth_path=/dkyges-58/c?versionId=null 2024-04-12T03:55:09.705 DEBUG:boto:Method: GET 2024-04-12T03:55:09.705 DEBUG:boto:Path: /dkyges-58/c?versionId=null 2024-04-12T03:55:09.706 DEBUG:boto:Data: 2024-04-12T03:55:09.706 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.706 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.706 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.706 DEBUG:boto:Params: {} 2024-04-12T03:55:09.706 DEBUG:boto:Token: None 2024-04-12T03:55:09.706 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/c?versionId=null 2024-04-12T03:55:09.706 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jL0XAvUHGEgDuXWQ5dQmIdoGnDk= 2024-04-12T03:55:09.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jL0XAvUHGEgDuXWQ5dQmIdoGnDk=', 'Content-Length': '0'} 2024-04-12T03:55:09.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071b6891c46eef49f-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.708 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.709 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.709 DEBUG:boto:path=/dkyges-58/?prefix=c 2024-04-12T03:55:09.709 DEBUG:boto:auth_path=/dkyges-58/?prefix=c 2024-04-12T03:55:09.709 DEBUG:boto:Method: GET 2024-04-12T03:55:09.709 DEBUG:boto:Path: /dkyges-58/?prefix=c 2024-04-12T03:55:09.709 DEBUG:boto:Data: 2024-04-12T03:55:09.709 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.709 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.709 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.709 DEBUG:boto:Params: {} 2024-04-12T03:55:09.709 DEBUG:boto:Token: None 2024-04-12T03:55:09.710 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.710 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000238d2fb8063b9af3-006618b09d-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.712 DEBUG:boto:b'dkyges-58c1000falsec2024-04-12T03:52:21.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.713 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.713 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.713 DEBUG:boto:path=/dkyges-58/?prefix=c 2024-04-12T03:55:09.713 DEBUG:boto:auth_path=/dkyges-58/?prefix=c 2024-04-12T03:55:09.713 DEBUG:boto:Method: GET 2024-04-12T03:55:09.713 DEBUG:boto:Path: /dkyges-58/?prefix=c 2024-04-12T03:55:09.713 DEBUG:boto:Data: 2024-04-12T03:55:09.713 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.713 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.713 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.713 DEBUG:boto:Params: {} 2024-04-12T03:55:09.713 DEBUG:boto:Token: None 2024-04-12T03:55:09.714 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.714 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3e1b744600a8189-006618b09d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.716 DEBUG:boto:b'dkyges-58c1000falsec2024-04-12T03:52:21.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.716 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:55:09.716 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:55:09.716 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.716 DEBUG:boto:Path: /dkyges-58/c 2024-04-12T03:55:09.716 DEBUG:boto:Data: 2024-04-12T03:55:09.717 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.717 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.717 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.717 DEBUG:boto:Params: {} 2024-04-12T03:55:09.717 DEBUG:boto:Token: None 2024-04-12T03:55:09.717 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/c 2024-04-12T03:55:09.717 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UQM01slxmuNB/yIT1bU63sQcYXU= 2024-04-12T03:55:09.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UQM01slxmuNB/yIT1bU63sQcYXU=', 'Content-Length': '0'} 2024-04-12T03:55:09.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000722f603ebff3cc94-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.719 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:55:09.719 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:55:09.720 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.720 DEBUG:boto:Path: /dkyges-58/c 2024-04-12T03:55:09.720 DEBUG:boto:Data: 2024-04-12T03:55:09.720 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.720 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.720 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.720 DEBUG:boto:Params: {} 2024-04-12T03:55:09.720 DEBUG:boto:Token: None 2024-04-12T03:55:09.720 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/c 2024-04-12T03:55:09.720 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UQM01slxmuNB/yIT1bU63sQcYXU= 2024-04-12T03:55:09.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UQM01slxmuNB/yIT1bU63sQcYXU=', 'Content-Length': '0'} 2024-04-12T03:55:09.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d370a2c7c1e0591-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:55:09.723 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:55:09.723 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:55:09.723 DEBUG:boto:Method: GET 2024-04-12T03:55:09.723 DEBUG:boto:Path: /dkyges-58/c 2024-04-12T03:55:09.723 DEBUG:boto:Data: 2024-04-12T03:55:09.723 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.723 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.723 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.723 DEBUG:boto:Params: {} 2024-04-12T03:55:09.723 DEBUG:boto:Token: None 2024-04-12T03:55:09.723 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/c 2024-04-12T03:55:09.724 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:k+0CueRKiHliip+PDAHei8jdv0A= 2024-04-12T03:55:09.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:k+0CueRKiHliip+PDAHei8jdv0A=', 'Content-Length': '0'} 2024-04-12T03:55:09.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee41cfb48c532e33-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.726 DEBUG:boto:path=/dkyges-58/c 2024-04-12T03:55:09.726 DEBUG:boto:auth_path=/dkyges-58/c 2024-04-12T03:55:09.726 DEBUG:boto:Method: GET 2024-04-12T03:55:09.726 DEBUG:boto:Path: /dkyges-58/c 2024-04-12T03:55:09.726 DEBUG:boto:Data: 2024-04-12T03:55:09.726 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.726 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.726 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.726 DEBUG:boto:Params: {} 2024-04-12T03:55:09.726 DEBUG:boto:Token: None 2024-04-12T03:55:09.727 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/c 2024-04-12T03:55:09.727 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:k+0CueRKiHliip+PDAHei8jdv0A= 2024-04-12T03:55:09.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:k+0CueRKiHliip+PDAHei8jdv0A=', 'Content-Length': '0'} 2024-04-12T03:55:09.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e37ba940097747d-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.729 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.729 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.729 DEBUG:boto:path=/dkyges-58/?prefix=c 2024-04-12T03:55:09.729 DEBUG:boto:auth_path=/dkyges-58/?prefix=c 2024-04-12T03:55:09.729 DEBUG:boto:Method: GET 2024-04-12T03:55:09.729 DEBUG:boto:Path: /dkyges-58/?prefix=c 2024-04-12T03:55:09.729 DEBUG:boto:Data: 2024-04-12T03:55:09.729 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.729 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.730 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.730 DEBUG:boto:Params: {} 2024-04-12T03:55:09.730 DEBUG:boto:Token: None 2024-04-12T03:55:09.730 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.730 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a4888b496ec2ddf-006618b09d-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.733 DEBUG:boto:b'dkyges-58c1000falsec2024-04-12T03:52:21.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.733 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.733 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.733 DEBUG:boto:path=/dkyges-58/?prefix=c 2024-04-12T03:55:09.733 DEBUG:boto:auth_path=/dkyges-58/?prefix=c 2024-04-12T03:55:09.733 DEBUG:boto:Method: GET 2024-04-12T03:55:09.733 DEBUG:boto:Path: /dkyges-58/?prefix=c 2024-04-12T03:55:09.733 DEBUG:boto:Data: 2024-04-12T03:55:09.734 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.734 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.734 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.734 DEBUG:boto:Params: {} 2024-04-12T03:55:09.734 DEBUG:boto:Token: None 2024-04-12T03:55:09.734 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.734 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000290823da86b3da0b-006618b09d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.736 DEBUG:boto:b'dkyges-58c1000falsec2024-04-12T03:52:21.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:55:09.737 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:55:09.737 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:55:09.737 DEBUG:boto:path=/dkyges-58/d?versionId=null 2024-04-12T03:55:09.737 DEBUG:boto:auth_path=/dkyges-58/d?versionId=null 2024-04-12T03:55:09.737 DEBUG:boto:Method: GET 2024-04-12T03:55:09.737 DEBUG:boto:Path: /dkyges-58/d?versionId=null 2024-04-12T03:55:09.737 DEBUG:boto:Data: 2024-04-12T03:55:09.737 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.737 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.737 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.737 DEBUG:boto:Params: {} 2024-04-12T03:55:09.738 DEBUG:boto:Token: None 2024-04-12T03:55:09.738 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/d?versionId=null 2024-04-12T03:55:09.738 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4bwolbtkGCp2ym3ErkElwTeaW80= 2024-04-12T03:55:09.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4bwolbtkGCp2ym3ErkElwTeaW80=', 'Content-Length': '0'} 2024-04-12T03:55:09.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094cb15ad6ece70d1-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.740 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:55:09.740 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:55:09.740 DEBUG:boto:path=/dkyges-58/d?versionId=null 2024-04-12T03:55:09.740 DEBUG:boto:auth_path=/dkyges-58/d?versionId=null 2024-04-12T03:55:09.740 DEBUG:boto:Method: GET 2024-04-12T03:55:09.740 DEBUG:boto:Path: /dkyges-58/d?versionId=null 2024-04-12T03:55:09.740 DEBUG:boto:Data: 2024-04-12T03:55:09.740 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.740 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.740 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.740 DEBUG:boto:Params: {} 2024-04-12T03:55:09.741 DEBUG:boto:Token: None 2024-04-12T03:55:09.741 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/d?versionId=null 2024-04-12T03:55:09.741 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4bwolbtkGCp2ym3ErkElwTeaW80= 2024-04-12T03:55:09.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4bwolbtkGCp2ym3ErkElwTeaW80=', 'Content-Length': '0'} 2024-04-12T03:55:09.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e81144146ad7eb9a-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.743 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:55:09.743 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:55:09.743 DEBUG:boto:path=/dkyges-58/d?versionId=null 2024-04-12T03:55:09.743 DEBUG:boto:auth_path=/dkyges-58/d?versionId=null 2024-04-12T03:55:09.743 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.743 DEBUG:boto:Path: /dkyges-58/d?versionId=null 2024-04-12T03:55:09.743 DEBUG:boto:Data: 2024-04-12T03:55:09.743 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.743 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.743 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.744 DEBUG:boto:Params: {} 2024-04-12T03:55:09.744 DEBUG:boto:Token: None 2024-04-12T03:55:09.744 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/d?versionId=null 2024-04-12T03:55:09.744 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eK6NRGp6H9CIKsom9tb3YEohHwo= 2024-04-12T03:55:09.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eK6NRGp6H9CIKsom9tb3YEohHwo=', 'Content-Length': '0'} 2024-04-12T03:55:09.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf02463d609c9764-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.746 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:55:09.746 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:55:09.746 DEBUG:boto:path=/dkyges-58/d?versionId=null 2024-04-12T03:55:09.746 DEBUG:boto:auth_path=/dkyges-58/d?versionId=null 2024-04-12T03:55:09.746 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.747 DEBUG:boto:Path: /dkyges-58/d?versionId=null 2024-04-12T03:55:09.747 DEBUG:boto:Data: 2024-04-12T03:55:09.747 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.747 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.747 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.747 DEBUG:boto:Params: {} 2024-04-12T03:55:09.747 DEBUG:boto:Token: None 2024-04-12T03:55:09.747 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/d?versionId=null 2024-04-12T03:55:09.747 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eK6NRGp6H9CIKsom9tb3YEohHwo= 2024-04-12T03:55:09.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eK6NRGp6H9CIKsom9tb3YEohHwo=', 'Content-Length': '0'} 2024-04-12T03:55:09.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bfd4410d10b1055-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:55:09.749 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:55:09.749 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:55:09.749 DEBUG:boto:path=/dkyges-58/d?versionId=null 2024-04-12T03:55:09.749 DEBUG:boto:auth_path=/dkyges-58/d?versionId=null 2024-04-12T03:55:09.750 DEBUG:boto:Method: GET 2024-04-12T03:55:09.750 DEBUG:boto:Path: /dkyges-58/d?versionId=null 2024-04-12T03:55:09.750 DEBUG:boto:Data: 2024-04-12T03:55:09.750 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.750 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.750 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.750 DEBUG:boto:Params: {} 2024-04-12T03:55:09.750 DEBUG:boto:Token: None 2024-04-12T03:55:09.750 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/d?versionId=null 2024-04-12T03:55:09.750 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4bwolbtkGCp2ym3ErkElwTeaW80= 2024-04-12T03:55:09.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4bwolbtkGCp2ym3ErkElwTeaW80=', 'Content-Length': '0'} 2024-04-12T03:55:09.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e47ec994eddb40d-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.753 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:55:09.753 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:55:09.753 DEBUG:boto:path=/dkyges-58/d?versionId=null 2024-04-12T03:55:09.753 DEBUG:boto:auth_path=/dkyges-58/d?versionId=null 2024-04-12T03:55:09.753 DEBUG:boto:Method: GET 2024-04-12T03:55:09.753 DEBUG:boto:Path: /dkyges-58/d?versionId=null 2024-04-12T03:55:09.753 DEBUG:boto:Data: 2024-04-12T03:55:09.753 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.753 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.753 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.753 DEBUG:boto:Params: {} 2024-04-12T03:55:09.753 DEBUG:boto:Token: None 2024-04-12T03:55:09.754 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/d?versionId=null 2024-04-12T03:55:09.754 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4bwolbtkGCp2ym3ErkElwTeaW80= 2024-04-12T03:55:09.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4bwolbtkGCp2ym3ErkElwTeaW80=', 'Content-Length': '0'} 2024-04-12T03:55:09.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046ae0615fb6c5374-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.798 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.798 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.798 DEBUG:boto:path=/dkyges-58/?prefix=d 2024-04-12T03:55:09.799 DEBUG:boto:auth_path=/dkyges-58/?prefix=d 2024-04-12T03:55:09.799 DEBUG:boto:Method: GET 2024-04-12T03:55:09.799 DEBUG:boto:Path: /dkyges-58/?prefix=d 2024-04-12T03:55:09.799 DEBUG:boto:Data: 2024-04-12T03:55:09.799 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.799 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.799 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.799 DEBUG:boto:Params: {} 2024-04-12T03:55:09.799 DEBUG:boto:Token: None 2024-04-12T03:55:09.799 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.800 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5b9aed84c28bfcc-006618b09d-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.803 DEBUG:boto:b'dkyges-58d1000falsed2024-04-12T03:52:21.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.803 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.803 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.803 DEBUG:boto:path=/dkyges-58/?prefix=d 2024-04-12T03:55:09.803 DEBUG:boto:auth_path=/dkyges-58/?prefix=d 2024-04-12T03:55:09.803 DEBUG:boto:Method: GET 2024-04-12T03:55:09.803 DEBUG:boto:Path: /dkyges-58/?prefix=d 2024-04-12T03:55:09.803 DEBUG:boto:Data: 2024-04-12T03:55:09.803 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.804 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.804 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.804 DEBUG:boto:Params: {} 2024-04-12T03:55:09.804 DEBUG:boto:Token: None 2024-04-12T03:55:09.804 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.804 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031820880d9f12e5e-006618b09d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.806 DEBUG:boto:b'dkyges-58d1000falsed2024-04-12T03:52:21.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.807 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:55:09.807 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:55:09.807 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.807 DEBUG:boto:Path: /dkyges-58/d 2024-04-12T03:55:09.807 DEBUG:boto:Data: 2024-04-12T03:55:09.807 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.807 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.807 DEBUG:boto:Params: {} 2024-04-12T03:55:09.807 DEBUG:boto:Token: None 2024-04-12T03:55:09.808 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/d 2024-04-12T03:55:09.808 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sqUXwJD7BgwIS061XPobcRVsYxw= 2024-04-12T03:55:09.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sqUXwJD7BgwIS061XPobcRVsYxw=', 'Content-Length': '0'} 2024-04-12T03:55:09.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5cf4efa035a8020-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.810 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:55:09.810 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:55:09.810 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.810 DEBUG:boto:Path: /dkyges-58/d 2024-04-12T03:55:09.810 DEBUG:boto:Data: 2024-04-12T03:55:09.810 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.810 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.810 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.810 DEBUG:boto:Params: {} 2024-04-12T03:55:09.811 DEBUG:boto:Token: None 2024-04-12T03:55:09.811 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/d 2024-04-12T03:55:09.811 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sqUXwJD7BgwIS061XPobcRVsYxw= 2024-04-12T03:55:09.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sqUXwJD7BgwIS061XPobcRVsYxw=', 'Content-Length': '0'} 2024-04-12T03:55:09.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000136bcbdb467a9585-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:55:09.813 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:55:09.813 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:55:09.813 DEBUG:boto:Method: GET 2024-04-12T03:55:09.813 DEBUG:boto:Path: /dkyges-58/d 2024-04-12T03:55:09.813 DEBUG:boto:Data: 2024-04-12T03:55:09.813 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.813 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.813 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.813 DEBUG:boto:Params: {} 2024-04-12T03:55:09.813 DEBUG:boto:Token: None 2024-04-12T03:55:09.814 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/d 2024-04-12T03:55:09.814 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8p0S8JAT8PHcTja4X4Rj6NgjTpE= 2024-04-12T03:55:09.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8p0S8JAT8PHcTja4X4Rj6NgjTpE=', 'Content-Length': '0'} 2024-04-12T03:55:09.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ac0a576505bd4d8-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.858 DEBUG:boto:path=/dkyges-58/d 2024-04-12T03:55:09.858 DEBUG:boto:auth_path=/dkyges-58/d 2024-04-12T03:55:09.859 DEBUG:boto:Method: GET 2024-04-12T03:55:09.859 DEBUG:boto:Path: /dkyges-58/d 2024-04-12T03:55:09.859 DEBUG:boto:Data: 2024-04-12T03:55:09.859 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.859 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.859 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.859 DEBUG:boto:Params: {} 2024-04-12T03:55:09.859 DEBUG:boto:Token: None 2024-04-12T03:55:09.859 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/d 2024-04-12T03:55:09.859 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8p0S8JAT8PHcTja4X4Rj6NgjTpE= 2024-04-12T03:55:09.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8p0S8JAT8PHcTja4X4Rj6NgjTpE=', 'Content-Length': '0'} 2024-04-12T03:55:09.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075614a87b744e3f5-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.861 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.862 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.862 DEBUG:boto:path=/dkyges-58/?prefix=d 2024-04-12T03:55:09.862 DEBUG:boto:auth_path=/dkyges-58/?prefix=d 2024-04-12T03:55:09.862 DEBUG:boto:Method: GET 2024-04-12T03:55:09.862 DEBUG:boto:Path: /dkyges-58/?prefix=d 2024-04-12T03:55:09.862 DEBUG:boto:Data: 2024-04-12T03:55:09.862 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.862 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.862 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.862 DEBUG:boto:Params: {} 2024-04-12T03:55:09.862 DEBUG:boto:Token: None 2024-04-12T03:55:09.863 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.863 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007962f7ed3b7bf7e8-006618b09d-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.865 DEBUG:boto:b'dkyges-58d1000falsed2024-04-12T03:52:21.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.865 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.866 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.866 DEBUG:boto:path=/dkyges-58/?prefix=d 2024-04-12T03:55:09.866 DEBUG:boto:auth_path=/dkyges-58/?prefix=d 2024-04-12T03:55:09.866 DEBUG:boto:Method: GET 2024-04-12T03:55:09.866 DEBUG:boto:Path: /dkyges-58/?prefix=d 2024-04-12T03:55:09.866 DEBUG:boto:Data: 2024-04-12T03:55:09.866 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.866 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.866 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.866 DEBUG:boto:Params: {} 2024-04-12T03:55:09.866 DEBUG:boto:Token: None 2024-04-12T03:55:09.867 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.867 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001253abc21d144086-006618b09d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.869 DEBUG:boto:b'dkyges-58d1000falsed2024-04-12T03:52:21.905Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:55:09.869 DEBUG:boto:path=/dkyges-58/e 2024-04-12T03:55:09.869 DEBUG:boto:auth_path=/dkyges-58/e 2024-04-12T03:55:09.869 DEBUG:boto:path=/dkyges-58/e?versionId=null 2024-04-12T03:55:09.869 DEBUG:boto:auth_path=/dkyges-58/e?versionId=null 2024-04-12T03:55:09.869 DEBUG:boto:Method: GET 2024-04-12T03:55:09.870 DEBUG:boto:Path: /dkyges-58/e?versionId=null 2024-04-12T03:55:09.870 DEBUG:boto:Data: 2024-04-12T03:55:09.870 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.870 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.870 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.870 DEBUG:boto:Params: {} 2024-04-12T03:55:09.870 DEBUG:boto:Token: None 2024-04-12T03:55:09.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/e?versionId=null 2024-04-12T03:55:09.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aCDvSh9pLC5BPRO+diug7vkS7iE= 2024-04-12T03:55:09.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aCDvSh9pLC5BPRO+diug7vkS7iE=', 'Content-Length': '0'} 2024-04-12T03:55:09.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005829241f8961f0a1-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.873 DEBUG:boto:path=/dkyges-58/e 2024-04-12T03:55:09.873 DEBUG:boto:auth_path=/dkyges-58/e 2024-04-12T03:55:09.873 DEBUG:boto:path=/dkyges-58/e?versionId=null 2024-04-12T03:55:09.873 DEBUG:boto:auth_path=/dkyges-58/e?versionId=null 2024-04-12T03:55:09.873 DEBUG:boto:Method: GET 2024-04-12T03:55:09.873 DEBUG:boto:Path: /dkyges-58/e?versionId=null 2024-04-12T03:55:09.873 DEBUG:boto:Data: 2024-04-12T03:55:09.873 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.873 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.873 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.873 DEBUG:boto:Params: {} 2024-04-12T03:55:09.874 DEBUG:boto:Token: None 2024-04-12T03:55:09.874 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/e?versionId=null 2024-04-12T03:55:09.874 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aCDvSh9pLC5BPRO+diug7vkS7iE= 2024-04-12T03:55:09.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aCDvSh9pLC5BPRO+diug7vkS7iE=', 'Content-Length': '0'} 2024-04-12T03:55:09.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094254881e6e3df14-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.876 DEBUG:boto:path=/dkyges-58/e 2024-04-12T03:55:09.876 DEBUG:boto:auth_path=/dkyges-58/e 2024-04-12T03:55:09.876 DEBUG:boto:path=/dkyges-58/e?versionId=null 2024-04-12T03:55:09.876 DEBUG:boto:auth_path=/dkyges-58/e?versionId=null 2024-04-12T03:55:09.876 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.877 DEBUG:boto:Path: /dkyges-58/e?versionId=null 2024-04-12T03:55:09.877 DEBUG:boto:Data: 2024-04-12T03:55:09.877 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.877 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.877 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.877 DEBUG:boto:Params: {} 2024-04-12T03:55:09.877 DEBUG:boto:Token: None 2024-04-12T03:55:09.877 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/e?versionId=null 2024-04-12T03:55:09.877 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eWTJfHjVc5uoOFL62Hay2SOwasA= 2024-04-12T03:55:09.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eWTJfHjVc5uoOFL62Hay2SOwasA=', 'Content-Length': '0'} 2024-04-12T03:55:09.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008477d2389b380140-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.879 DEBUG:boto:path=/dkyges-58/e 2024-04-12T03:55:09.880 DEBUG:boto:auth_path=/dkyges-58/e 2024-04-12T03:55:09.880 DEBUG:boto:path=/dkyges-58/e?versionId=null 2024-04-12T03:55:09.880 DEBUG:boto:auth_path=/dkyges-58/e?versionId=null 2024-04-12T03:55:09.880 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.880 DEBUG:boto:Path: /dkyges-58/e?versionId=null 2024-04-12T03:55:09.880 DEBUG:boto:Data: 2024-04-12T03:55:09.880 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.880 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.880 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.880 DEBUG:boto:Params: {} 2024-04-12T03:55:09.880 DEBUG:boto:Token: None 2024-04-12T03:55:09.881 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/e?versionId=null 2024-04-12T03:55:09.881 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eWTJfHjVc5uoOFL62Hay2SOwasA= 2024-04-12T03:55:09.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eWTJfHjVc5uoOFL62Hay2SOwasA=', 'Content-Length': '0'} 2024-04-12T03:55:09.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c88eca988af5f7ca-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:55:09.883 DEBUG:boto:path=/dkyges-58/e 2024-04-12T03:55:09.883 DEBUG:boto:auth_path=/dkyges-58/e 2024-04-12T03:55:09.883 DEBUG:boto:path=/dkyges-58/e?versionId=null 2024-04-12T03:55:09.883 DEBUG:boto:auth_path=/dkyges-58/e?versionId=null 2024-04-12T03:55:09.883 DEBUG:boto:Method: GET 2024-04-12T03:55:09.883 DEBUG:boto:Path: /dkyges-58/e?versionId=null 2024-04-12T03:55:09.883 DEBUG:boto:Data: 2024-04-12T03:55:09.883 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.883 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.884 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.884 DEBUG:boto:Params: {} 2024-04-12T03:55:09.884 DEBUG:boto:Token: None 2024-04-12T03:55:09.884 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/e?versionId=null 2024-04-12T03:55:09.884 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aCDvSh9pLC5BPRO+diug7vkS7iE= 2024-04-12T03:55:09.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aCDvSh9pLC5BPRO+diug7vkS7iE=', 'Content-Length': '0'} 2024-04-12T03:55:09.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045c2653923febba0-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.886 DEBUG:boto:path=/dkyges-58/e 2024-04-12T03:55:09.886 DEBUG:boto:auth_path=/dkyges-58/e 2024-04-12T03:55:09.886 DEBUG:boto:path=/dkyges-58/e?versionId=null 2024-04-12T03:55:09.887 DEBUG:boto:auth_path=/dkyges-58/e?versionId=null 2024-04-12T03:55:09.887 DEBUG:boto:Method: GET 2024-04-12T03:55:09.887 DEBUG:boto:Path: /dkyges-58/e?versionId=null 2024-04-12T03:55:09.887 DEBUG:boto:Data: 2024-04-12T03:55:09.887 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.887 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.887 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.887 DEBUG:boto:Params: {} 2024-04-12T03:55:09.887 DEBUG:boto:Token: None 2024-04-12T03:55:09.887 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/e?versionId=null 2024-04-12T03:55:09.887 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aCDvSh9pLC5BPRO+diug7vkS7iE= 2024-04-12T03:55:09.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aCDvSh9pLC5BPRO+diug7vkS7iE=', 'Content-Length': '0'} 2024-04-12T03:55:09.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f415ecbfc6a39dd-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.890 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.890 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.890 DEBUG:boto:path=/dkyges-58/?prefix=e 2024-04-12T03:55:09.890 DEBUG:boto:auth_path=/dkyges-58/?prefix=e 2024-04-12T03:55:09.890 DEBUG:boto:Method: GET 2024-04-12T03:55:09.890 DEBUG:boto:Path: /dkyges-58/?prefix=e 2024-04-12T03:55:09.890 DEBUG:boto:Data: 2024-04-12T03:55:09.890 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.890 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.891 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.891 DEBUG:boto:Params: {} 2024-04-12T03:55:09.891 DEBUG:boto:Token: None 2024-04-12T03:55:09.891 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.891 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000128c0371023c2656-006618b09d-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.894 DEBUG:boto:b'dkyges-58e1000falsee2024-04-12T03:52:43.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.894 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.894 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.894 DEBUG:boto:path=/dkyges-58/?prefix=e 2024-04-12T03:55:09.894 DEBUG:boto:auth_path=/dkyges-58/?prefix=e 2024-04-12T03:55:09.895 DEBUG:boto:Method: GET 2024-04-12T03:55:09.895 DEBUG:boto:Path: /dkyges-58/?prefix=e 2024-04-12T03:55:09.895 DEBUG:boto:Data: 2024-04-12T03:55:09.895 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.895 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.895 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.895 DEBUG:boto:Params: {} 2024-04-12T03:55:09.895 DEBUG:boto:Token: None 2024-04-12T03:55:09.895 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.895 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c967a8944352dd0-006618b09d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.898 DEBUG:boto:b'dkyges-58e1000falsee2024-04-12T03:52:43.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.898 DEBUG:boto:path=/dkyges-58/e 2024-04-12T03:55:09.898 DEBUG:boto:auth_path=/dkyges-58/e 2024-04-12T03:55:09.898 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.898 DEBUG:boto:Path: /dkyges-58/e 2024-04-12T03:55:09.898 DEBUG:boto:Data: 2024-04-12T03:55:09.898 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.898 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.899 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.899 DEBUG:boto:Params: {} 2024-04-12T03:55:09.899 DEBUG:boto:Token: None 2024-04-12T03:55:09.899 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/e 2024-04-12T03:55:09.899 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQdZm1Wb3DlCKhNDMQTlnRU/DEs= 2024-04-12T03:55:09.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQdZm1Wb3DlCKhNDMQTlnRU/DEs=', 'Content-Length': '0'} 2024-04-12T03:55:09.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000117113bfe7cb490a-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.901 DEBUG:boto:path=/dkyges-58/e 2024-04-12T03:55:09.901 DEBUG:boto:auth_path=/dkyges-58/e 2024-04-12T03:55:09.901 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.901 DEBUG:boto:Path: /dkyges-58/e 2024-04-12T03:55:09.901 DEBUG:boto:Data: 2024-04-12T03:55:09.902 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.902 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.902 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.902 DEBUG:boto:Params: {} 2024-04-12T03:55:09.902 DEBUG:boto:Token: None 2024-04-12T03:55:09.902 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/e 2024-04-12T03:55:09.902 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQdZm1Wb3DlCKhNDMQTlnRU/DEs= 2024-04-12T03:55:09.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQdZm1Wb3DlCKhNDMQTlnRU/DEs=', 'Content-Length': '0'} 2024-04-12T03:55:09.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c59f70f036e4d092-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:55:09.904 DEBUG:boto:path=/dkyges-58/e 2024-04-12T03:55:09.905 DEBUG:boto:auth_path=/dkyges-58/e 2024-04-12T03:55:09.905 DEBUG:boto:Method: GET 2024-04-12T03:55:09.905 DEBUG:boto:Path: /dkyges-58/e 2024-04-12T03:55:09.905 DEBUG:boto:Data: 2024-04-12T03:55:09.905 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.905 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.905 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.905 DEBUG:boto:Params: {} 2024-04-12T03:55:09.905 DEBUG:boto:Token: None 2024-04-12T03:55:09.905 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/e 2024-04-12T03:55:09.905 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NcsaHFncLZ1FR39G256xReob+ys= 2024-04-12T03:55:09.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NcsaHFncLZ1FR39G256xReob+ys=', 'Content-Length': '0'} 2024-04-12T03:55:09.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b1d5a01db8b680a-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.908 DEBUG:boto:path=/dkyges-58/e 2024-04-12T03:55:09.908 DEBUG:boto:auth_path=/dkyges-58/e 2024-04-12T03:55:09.908 DEBUG:boto:Method: GET 2024-04-12T03:55:09.908 DEBUG:boto:Path: /dkyges-58/e 2024-04-12T03:55:09.908 DEBUG:boto:Data: 2024-04-12T03:55:09.908 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.908 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.908 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.908 DEBUG:boto:Params: {} 2024-04-12T03:55:09.909 DEBUG:boto:Token: None 2024-04-12T03:55:09.909 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/e 2024-04-12T03:55:09.909 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NcsaHFncLZ1FR39G256xReob+ys= 2024-04-12T03:55:09.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NcsaHFncLZ1FR39G256xReob+ys=', 'Content-Length': '0'} 2024-04-12T03:55:09.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000befdd0aa172861e2-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.911 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.911 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.911 DEBUG:boto:path=/dkyges-58/?prefix=e 2024-04-12T03:55:09.911 DEBUG:boto:auth_path=/dkyges-58/?prefix=e 2024-04-12T03:55:09.911 DEBUG:boto:Method: GET 2024-04-12T03:55:09.911 DEBUG:boto:Path: /dkyges-58/?prefix=e 2024-04-12T03:55:09.912 DEBUG:boto:Data: 2024-04-12T03:55:09.912 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.912 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.912 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.912 DEBUG:boto:Params: {} 2024-04-12T03:55:09.912 DEBUG:boto:Token: None 2024-04-12T03:55:09.912 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.912 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f95e894d759a2ad3-006618b09d-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.958 DEBUG:boto:b'dkyges-58e1000falsee2024-04-12T03:52:43.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.959 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:09.959 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:09.959 DEBUG:boto:path=/dkyges-58/?prefix=e 2024-04-12T03:55:09.959 DEBUG:boto:auth_path=/dkyges-58/?prefix=e 2024-04-12T03:55:09.959 DEBUG:boto:Method: GET 2024-04-12T03:55:09.959 DEBUG:boto:Path: /dkyges-58/?prefix=e 2024-04-12T03:55:09.959 DEBUG:boto:Data: 2024-04-12T03:55:09.959 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.959 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.959 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.959 DEBUG:boto:Params: {} 2024-04-12T03:55:09.960 DEBUG:boto:Token: None 2024-04-12T03:55:09.960 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/ 2024-04-12T03:55:09.960 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4= 2024-04-12T03:55:09.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XhFwGTXnxoOmsxtXH3Q3RdZW1O4=', 'Content-Length': '0'} 2024-04-12T03:55:09.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d7664cfb9d10b00-006618b09d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.962 DEBUG:boto:b'dkyges-58e1000falsee2024-04-12T03:52:43.007Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:09.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:55:09.962 DEBUG:boto:path=/dkyges-58/f 2024-04-12T03:55:09.962 DEBUG:boto:auth_path=/dkyges-58/f 2024-04-12T03:55:09.962 DEBUG:boto:path=/dkyges-58/f?versionId=null 2024-04-12T03:55:09.963 DEBUG:boto:auth_path=/dkyges-58/f?versionId=null 2024-04-12T03:55:09.963 DEBUG:boto:Method: GET 2024-04-12T03:55:09.963 DEBUG:boto:Path: /dkyges-58/f?versionId=null 2024-04-12T03:55:09.963 DEBUG:boto:Data: 2024-04-12T03:55:09.963 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.963 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.963 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.963 DEBUG:boto:Params: {} 2024-04-12T03:55:09.963 DEBUG:boto:Token: None 2024-04-12T03:55:09.963 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/f?versionId=null 2024-04-12T03:55:09.963 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qGakY0/o+xPVfd0Vl6bt7zg/+GI= 2024-04-12T03:55:09.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qGakY0/o+xPVfd0Vl6bt7zg/+GI=', 'Content-Length': '0'} 2024-04-12T03:55:09.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f22078349aec7af-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.966 DEBUG:boto:path=/dkyges-58/f 2024-04-12T03:55:09.966 DEBUG:boto:auth_path=/dkyges-58/f 2024-04-12T03:55:09.966 DEBUG:boto:path=/dkyges-58/f?versionId=null 2024-04-12T03:55:09.966 DEBUG:boto:auth_path=/dkyges-58/f?versionId=null 2024-04-12T03:55:09.966 DEBUG:boto:Method: GET 2024-04-12T03:55:09.966 DEBUG:boto:Path: /dkyges-58/f?versionId=null 2024-04-12T03:55:09.966 DEBUG:boto:Data: 2024-04-12T03:55:09.966 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.966 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.966 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.966 DEBUG:boto:Params: {} 2024-04-12T03:55:09.967 DEBUG:boto:Token: None 2024-04-12T03:55:09.967 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/f?versionId=null 2024-04-12T03:55:09.967 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qGakY0/o+xPVfd0Vl6bt7zg/+GI= 2024-04-12T03:55:09.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qGakY0/o+xPVfd0Vl6bt7zg/+GI=', 'Content-Length': '0'} 2024-04-12T03:55:09.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ff3e96eacf5b9ec-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.969 DEBUG:boto:path=/dkyges-58/f 2024-04-12T03:55:09.969 DEBUG:boto:auth_path=/dkyges-58/f 2024-04-12T03:55:09.969 DEBUG:boto:path=/dkyges-58/f?versionId=null 2024-04-12T03:55:09.969 DEBUG:boto:auth_path=/dkyges-58/f?versionId=null 2024-04-12T03:55:09.969 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.969 DEBUG:boto:Path: /dkyges-58/f?versionId=null 2024-04-12T03:55:09.969 DEBUG:boto:Data: 2024-04-12T03:55:09.969 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.969 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.970 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.970 DEBUG:boto:Params: {} 2024-04-12T03:55:09.970 DEBUG:boto:Token: None 2024-04-12T03:55:09.970 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/f?versionId=null 2024-04-12T03:55:09.970 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pRTDCb2Xe70Kw2rTwpnqZxQ09Lc= 2024-04-12T03:55:09.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pRTDCb2Xe70Kw2rTwpnqZxQ09Lc=', 'Content-Length': '0'} 2024-04-12T03:55:09.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000581a332fdfc5bcd7-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.972 DEBUG:boto:path=/dkyges-58/f 2024-04-12T03:55:09.972 DEBUG:boto:auth_path=/dkyges-58/f 2024-04-12T03:55:09.972 DEBUG:boto:path=/dkyges-58/f?versionId=null 2024-04-12T03:55:09.972 DEBUG:boto:auth_path=/dkyges-58/f?versionId=null 2024-04-12T03:55:09.972 DEBUG:boto:Method: HEAD 2024-04-12T03:55:09.972 DEBUG:boto:Path: /dkyges-58/f?versionId=null 2024-04-12T03:55:09.973 DEBUG:boto:Data: 2024-04-12T03:55:09.973 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.973 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.973 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.973 DEBUG:boto:Params: {} 2024-04-12T03:55:09.973 DEBUG:boto:Token: None 2024-04-12T03:55:09.973 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/f?versionId=null 2024-04-12T03:55:09.973 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pRTDCb2Xe70Kw2rTwpnqZxQ09Lc= 2024-04-12T03:55:09.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pRTDCb2Xe70Kw2rTwpnqZxQ09Lc=', 'Content-Length': '0'} 2024-04-12T03:55:09.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7f3dc0cf61b0ca2-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:55:09.976 DEBUG:boto:path=/dkyges-58/f 2024-04-12T03:55:09.976 DEBUG:boto:auth_path=/dkyges-58/f 2024-04-12T03:55:09.976 DEBUG:boto:path=/dkyges-58/f?versionId=null 2024-04-12T03:55:09.976 DEBUG:boto:auth_path=/dkyges-58/f?versionId=null 2024-04-12T03:55:09.976 DEBUG:boto:Method: GET 2024-04-12T03:55:09.976 DEBUG:boto:Path: /dkyges-58/f?versionId=null 2024-04-12T03:55:09.976 DEBUG:boto:Data: 2024-04-12T03:55:09.976 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.976 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:09.976 DEBUG:boto:Port: 8001 2024-04-12T03:55:09.976 DEBUG:boto:Params: {} 2024-04-12T03:55:09.977 DEBUG:boto:Token: None 2024-04-12T03:55:09.977 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/f?versionId=null 2024-04-12T03:55:09.977 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qGakY0/o+xPVfd0Vl6bt7zg/+GI= 2024-04-12T03:55:09.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qGakY0/o+xPVfd0Vl6bt7zg/+GI=', 'Content-Length': '0'} 2024-04-12T03:55:09.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b3898bc5484f98f-006618b09d-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:09.979 DEBUG:boto:path=/dkyges-58/f 2024-04-12T03:55:09.979 DEBUG:boto:auth_path=/dkyges-58/f 2024-04-12T03:55:09.979 DEBUG:boto:path=/dkyges-58/f?versionId=null 2024-04-12T03:55:09.979 DEBUG:boto:auth_path=/dkyges-58/f?versionId=null 2024-04-12T03:55:09.979 DEBUG:boto:Method: GET 2024-04-12T03:55:09.979 DEBUG:boto:Path: /dkyges-58/f?versionId=null 2024-04-12T03:55:09.979 DEBUG:boto:Data: 2024-04-12T03:55:09.979 DEBUG:boto:Headers: {} 2024-04-12T03:55:09.980 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:09.980 DEBUG:boto:Port: 8000 2024-04-12T03:55:09.980 DEBUG:boto:Params: {} 2024-04-12T03:55:09.980 DEBUG:boto:Token: None 2024-04-12T03:55:09.980 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:09 GMT /dkyges-58/f?versionId=null 2024-04-12T03:55:09.980 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qGakY0/o+xPVfd0Vl6bt7zg/+GI= 2024-04-12T03:55:09.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qGakY0/o+xPVfd0Vl6bt7zg/+GI=', 'Content-Length': '0'} 2024-04-12T03:55:09.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dceb91906798785d-006618b09d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.022 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:10.023 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:10.023 DEBUG:boto:path=/dkyges-58/?prefix=f 2024-04-12T03:55:10.023 DEBUG:boto:auth_path=/dkyges-58/?prefix=f 2024-04-12T03:55:10.023 DEBUG:boto:Method: GET 2024-04-12T03:55:10.023 DEBUG:boto:Path: /dkyges-58/?prefix=f 2024-04-12T03:55:10.023 DEBUG:boto:Data: 2024-04-12T03:55:10.023 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.023 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.023 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.023 DEBUG:boto:Params: {} 2024-04-12T03:55:10.024 DEBUG:boto:Token: None 2024-04-12T03:55:10.024 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/ 2024-04-12T03:55:10.024 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U= 2024-04-12T03:55:10.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U=', 'Content-Length': '0'} 2024-04-12T03:55:10.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f212f1fbab6c9e5-006618b09e-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.027 DEBUG:boto:b'dkyges-58f1000falsef2024-04-12T03:52:43.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:10.027 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:10.027 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:10.027 DEBUG:boto:path=/dkyges-58/?prefix=f 2024-04-12T03:55:10.027 DEBUG:boto:auth_path=/dkyges-58/?prefix=f 2024-04-12T03:55:10.027 DEBUG:boto:Method: GET 2024-04-12T03:55:10.028 DEBUG:boto:Path: /dkyges-58/?prefix=f 2024-04-12T03:55:10.028 DEBUG:boto:Data: 2024-04-12T03:55:10.028 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.028 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.028 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.028 DEBUG:boto:Params: {} 2024-04-12T03:55:10.028 DEBUG:boto:Token: None 2024-04-12T03:55:10.028 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/ 2024-04-12T03:55:10.028 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U= 2024-04-12T03:55:10.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U=', 'Content-Length': '0'} 2024-04-12T03:55:10.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c65a6e519c9af401-006618b09e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.031 DEBUG:boto:b'dkyges-58f1000falsef2024-04-12T03:52:43.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:10.031 DEBUG:boto:path=/dkyges-58/f 2024-04-12T03:55:10.031 DEBUG:boto:auth_path=/dkyges-58/f 2024-04-12T03:55:10.031 DEBUG:boto:Method: HEAD 2024-04-12T03:55:10.031 DEBUG:boto:Path: /dkyges-58/f 2024-04-12T03:55:10.031 DEBUG:boto:Data: 2024-04-12T03:55:10.031 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.032 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.032 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.032 DEBUG:boto:Params: {} 2024-04-12T03:55:10.032 DEBUG:boto:Token: None 2024-04-12T03:55:10.032 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/f 2024-04-12T03:55:10.032 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D+s/qSW4aWgF2lg2o7CMkM0j7gs= 2024-04-12T03:55:10.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D+s/qSW4aWgF2lg2o7CMkM0j7gs=', 'Content-Length': '0'} 2024-04-12T03:55:10.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017cebacb2b587b65-006618b09e-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.034 DEBUG:boto:path=/dkyges-58/f 2024-04-12T03:55:10.034 DEBUG:boto:auth_path=/dkyges-58/f 2024-04-12T03:55:10.034 DEBUG:boto:Method: HEAD 2024-04-12T03:55:10.035 DEBUG:boto:Path: /dkyges-58/f 2024-04-12T03:55:10.035 DEBUG:boto:Data: 2024-04-12T03:55:10.035 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.035 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.035 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.035 DEBUG:boto:Params: {} 2024-04-12T03:55:10.035 DEBUG:boto:Token: None 2024-04-12T03:55:10.035 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/f 2024-04-12T03:55:10.035 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D+s/qSW4aWgF2lg2o7CMkM0j7gs= 2024-04-12T03:55:10.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D+s/qSW4aWgF2lg2o7CMkM0j7gs=', 'Content-Length': '0'} 2024-04-12T03:55:10.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c4a997b1a1c57ef-006618b09e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:55:10.038 DEBUG:boto:path=/dkyges-58/f 2024-04-12T03:55:10.038 DEBUG:boto:auth_path=/dkyges-58/f 2024-04-12T03:55:10.038 DEBUG:boto:Method: GET 2024-04-12T03:55:10.038 DEBUG:boto:Path: /dkyges-58/f 2024-04-12T03:55:10.038 DEBUG:boto:Data: 2024-04-12T03:55:10.038 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.038 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.038 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.038 DEBUG:boto:Params: {} 2024-04-12T03:55:10.039 DEBUG:boto:Token: None 2024-04-12T03:55:10.039 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/f 2024-04-12T03:55:10.039 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TzWeG50f1POxgUOQp6ExhlLb5rI= 2024-04-12T03:55:10.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TzWeG50f1POxgUOQp6ExhlLb5rI=', 'Content-Length': '0'} 2024-04-12T03:55:10.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e5a44a9d73b2783-006618b09e-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.041 DEBUG:boto:path=/dkyges-58/f 2024-04-12T03:55:10.041 DEBUG:boto:auth_path=/dkyges-58/f 2024-04-12T03:55:10.041 DEBUG:boto:Method: GET 2024-04-12T03:55:10.041 DEBUG:boto:Path: /dkyges-58/f 2024-04-12T03:55:10.041 DEBUG:boto:Data: 2024-04-12T03:55:10.041 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.041 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.042 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.042 DEBUG:boto:Params: {} 2024-04-12T03:55:10.042 DEBUG:boto:Token: None 2024-04-12T03:55:10.042 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/f 2024-04-12T03:55:10.042 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TzWeG50f1POxgUOQp6ExhlLb5rI= 2024-04-12T03:55:10.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TzWeG50f1POxgUOQp6ExhlLb5rI=', 'Content-Length': '0'} 2024-04-12T03:55:10.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006136aee0954d1a5a-006618b09e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.044 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:10.044 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:10.044 DEBUG:boto:path=/dkyges-58/?prefix=f 2024-04-12T03:55:10.044 DEBUG:boto:auth_path=/dkyges-58/?prefix=f 2024-04-12T03:55:10.045 DEBUG:boto:Method: GET 2024-04-12T03:55:10.045 DEBUG:boto:Path: /dkyges-58/?prefix=f 2024-04-12T03:55:10.045 DEBUG:boto:Data: 2024-04-12T03:55:10.045 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.045 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.045 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.045 DEBUG:boto:Params: {} 2024-04-12T03:55:10.045 DEBUG:boto:Token: None 2024-04-12T03:55:10.045 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/ 2024-04-12T03:55:10.045 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U= 2024-04-12T03:55:10.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U=', 'Content-Length': '0'} 2024-04-12T03:55:10.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5eac5d2591b0033-006618b09e-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.048 DEBUG:boto:b'dkyges-58f1000falsef2024-04-12T03:52:43.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:10.048 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:10.048 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:10.049 DEBUG:boto:path=/dkyges-58/?prefix=f 2024-04-12T03:55:10.049 DEBUG:boto:auth_path=/dkyges-58/?prefix=f 2024-04-12T03:55:10.049 DEBUG:boto:Method: GET 2024-04-12T03:55:10.049 DEBUG:boto:Path: /dkyges-58/?prefix=f 2024-04-12T03:55:10.049 DEBUG:boto:Data: 2024-04-12T03:55:10.049 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.049 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.049 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.049 DEBUG:boto:Params: {} 2024-04-12T03:55:10.049 DEBUG:boto:Token: None 2024-04-12T03:55:10.049 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/ 2024-04-12T03:55:10.049 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U= 2024-04-12T03:55:10.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U=', 'Content-Length': '0'} 2024-04-12T03:55:10.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffba8927e01ef602-006618b09e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.052 DEBUG:boto:b'dkyges-58f1000falsef2024-04-12T03:52:43.018Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:10.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:55:10.052 DEBUG:boto:path=/dkyges-58/g 2024-04-12T03:55:10.052 DEBUG:boto:auth_path=/dkyges-58/g 2024-04-12T03:55:10.052 DEBUG:boto:path=/dkyges-58/g?versionId=null 2024-04-12T03:55:10.052 DEBUG:boto:auth_path=/dkyges-58/g?versionId=null 2024-04-12T03:55:10.053 DEBUG:boto:Method: GET 2024-04-12T03:55:10.053 DEBUG:boto:Path: /dkyges-58/g?versionId=null 2024-04-12T03:55:10.053 DEBUG:boto:Data: 2024-04-12T03:55:10.053 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.053 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.053 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.053 DEBUG:boto:Params: {} 2024-04-12T03:55:10.053 DEBUG:boto:Token: None 2024-04-12T03:55:10.053 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/g?versionId=null 2024-04-12T03:55:10.053 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ESvn6u1xtitAVbv4ZmMLGqVt5j4= 2024-04-12T03:55:10.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ESvn6u1xtitAVbv4ZmMLGqVt5j4=', 'Content-Length': '0'} 2024-04-12T03:55:10.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f0b314d937739fb-006618b09e-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.098 DEBUG:boto:path=/dkyges-58/g 2024-04-12T03:55:10.098 DEBUG:boto:auth_path=/dkyges-58/g 2024-04-12T03:55:10.098 DEBUG:boto:path=/dkyges-58/g?versionId=null 2024-04-12T03:55:10.099 DEBUG:boto:auth_path=/dkyges-58/g?versionId=null 2024-04-12T03:55:10.099 DEBUG:boto:Method: GET 2024-04-12T03:55:10.099 DEBUG:boto:Path: /dkyges-58/g?versionId=null 2024-04-12T03:55:10.099 DEBUG:boto:Data: 2024-04-12T03:55:10.099 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.099 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.099 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.099 DEBUG:boto:Params: {} 2024-04-12T03:55:10.099 DEBUG:boto:Token: None 2024-04-12T03:55:10.099 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/g?versionId=null 2024-04-12T03:55:10.100 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ESvn6u1xtitAVbv4ZmMLGqVt5j4= 2024-04-12T03:55:10.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ESvn6u1xtitAVbv4ZmMLGqVt5j4=', 'Content-Length': '0'} 2024-04-12T03:55:10.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebca485c2b83c296-006618b09e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.102 DEBUG:boto:path=/dkyges-58/g 2024-04-12T03:55:10.102 DEBUG:boto:auth_path=/dkyges-58/g 2024-04-12T03:55:10.102 DEBUG:boto:path=/dkyges-58/g?versionId=null 2024-04-12T03:55:10.102 DEBUG:boto:auth_path=/dkyges-58/g?versionId=null 2024-04-12T03:55:10.102 DEBUG:boto:Method: HEAD 2024-04-12T03:55:10.102 DEBUG:boto:Path: /dkyges-58/g?versionId=null 2024-04-12T03:55:10.102 DEBUG:boto:Data: 2024-04-12T03:55:10.103 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.103 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.103 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.103 DEBUG:boto:Params: {} 2024-04-12T03:55:10.103 DEBUG:boto:Token: None 2024-04-12T03:55:10.103 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/g?versionId=null 2024-04-12T03:55:10.103 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ROYA9bpASOIS5wde/i00Kyyl43A= 2024-04-12T03:55:10.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ROYA9bpASOIS5wde/i00Kyyl43A=', 'Content-Length': '0'} 2024-04-12T03:55:10.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ef1698f1acd63c8-006618b09e-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.106 DEBUG:boto:path=/dkyges-58/g 2024-04-12T03:55:10.106 DEBUG:boto:auth_path=/dkyges-58/g 2024-04-12T03:55:10.106 DEBUG:boto:path=/dkyges-58/g?versionId=null 2024-04-12T03:55:10.106 DEBUG:boto:auth_path=/dkyges-58/g?versionId=null 2024-04-12T03:55:10.106 DEBUG:boto:Method: HEAD 2024-04-12T03:55:10.106 DEBUG:boto:Path: /dkyges-58/g?versionId=null 2024-04-12T03:55:10.106 DEBUG:boto:Data: 2024-04-12T03:55:10.106 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.106 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.106 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.106 DEBUG:boto:Params: {} 2024-04-12T03:55:10.107 DEBUG:boto:Token: None 2024-04-12T03:55:10.107 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/g?versionId=null 2024-04-12T03:55:10.107 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ROYA9bpASOIS5wde/i00Kyyl43A= 2024-04-12T03:55:10.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ROYA9bpASOIS5wde/i00Kyyl43A=', 'Content-Length': '0'} 2024-04-12T03:55:10.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025195e313488f683-006618b09e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:55:10.109 DEBUG:boto:path=/dkyges-58/g 2024-04-12T03:55:10.109 DEBUG:boto:auth_path=/dkyges-58/g 2024-04-12T03:55:10.109 DEBUG:boto:path=/dkyges-58/g?versionId=null 2024-04-12T03:55:10.109 DEBUG:boto:auth_path=/dkyges-58/g?versionId=null 2024-04-12T03:55:10.109 DEBUG:boto:Method: GET 2024-04-12T03:55:10.109 DEBUG:boto:Path: /dkyges-58/g?versionId=null 2024-04-12T03:55:10.110 DEBUG:boto:Data: 2024-04-12T03:55:10.110 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.110 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.110 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.110 DEBUG:boto:Params: {} 2024-04-12T03:55:10.110 DEBUG:boto:Token: None 2024-04-12T03:55:10.110 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/g?versionId=null 2024-04-12T03:55:10.110 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ESvn6u1xtitAVbv4ZmMLGqVt5j4= 2024-04-12T03:55:10.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ESvn6u1xtitAVbv4ZmMLGqVt5j4=', 'Content-Length': '0'} 2024-04-12T03:55:10.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e958369617724ad8-006618b09e-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.113 DEBUG:boto:path=/dkyges-58/g 2024-04-12T03:55:10.113 DEBUG:boto:auth_path=/dkyges-58/g 2024-04-12T03:55:10.113 DEBUG:boto:path=/dkyges-58/g?versionId=null 2024-04-12T03:55:10.113 DEBUG:boto:auth_path=/dkyges-58/g?versionId=null 2024-04-12T03:55:10.113 DEBUG:boto:Method: GET 2024-04-12T03:55:10.113 DEBUG:boto:Path: /dkyges-58/g?versionId=null 2024-04-12T03:55:10.113 DEBUG:boto:Data: 2024-04-12T03:55:10.114 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.114 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.114 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.114 DEBUG:boto:Params: {} 2024-04-12T03:55:10.114 DEBUG:boto:Token: None 2024-04-12T03:55:10.114 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/g?versionId=null 2024-04-12T03:55:10.114 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ESvn6u1xtitAVbv4ZmMLGqVt5j4= 2024-04-12T03:55:10.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ESvn6u1xtitAVbv4ZmMLGqVt5j4=', 'Content-Length': '0'} 2024-04-12T03:55:10.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adcc04bfa36a1d50-006618b09e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.116 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:10.117 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:10.117 DEBUG:boto:path=/dkyges-58/?prefix=g 2024-04-12T03:55:10.117 DEBUG:boto:auth_path=/dkyges-58/?prefix=g 2024-04-12T03:55:10.117 DEBUG:boto:Method: GET 2024-04-12T03:55:10.117 DEBUG:boto:Path: /dkyges-58/?prefix=g 2024-04-12T03:55:10.117 DEBUG:boto:Data: 2024-04-12T03:55:10.117 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.117 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.117 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.117 DEBUG:boto:Params: {} 2024-04-12T03:55:10.117 DEBUG:boto:Token: None 2024-04-12T03:55:10.118 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/ 2024-04-12T03:55:10.118 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U= 2024-04-12T03:55:10.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U=', 'Content-Length': '0'} 2024-04-12T03:55:10.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000158954ac8007171a-006618b09e-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.121 DEBUG:boto:b'dkyges-58g1000falseg2024-04-12T03:52:43.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:10.121 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:10.121 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:10.121 DEBUG:boto:path=/dkyges-58/?prefix=g 2024-04-12T03:55:10.121 DEBUG:boto:auth_path=/dkyges-58/?prefix=g 2024-04-12T03:55:10.121 DEBUG:boto:Method: GET 2024-04-12T03:55:10.121 DEBUG:boto:Path: /dkyges-58/?prefix=g 2024-04-12T03:55:10.121 DEBUG:boto:Data: 2024-04-12T03:55:10.121 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.122 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.122 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.122 DEBUG:boto:Params: {} 2024-04-12T03:55:10.122 DEBUG:boto:Token: None 2024-04-12T03:55:10.122 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/ 2024-04-12T03:55:10.122 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U= 2024-04-12T03:55:10.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U=', 'Content-Length': '0'} 2024-04-12T03:55:10.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ea00a7c9fa214bb-006618b09e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.124 DEBUG:boto:b'dkyges-58g1000falseg2024-04-12T03:52:43.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:10.124 DEBUG:boto:path=/dkyges-58/g 2024-04-12T03:55:10.125 DEBUG:boto:auth_path=/dkyges-58/g 2024-04-12T03:55:10.125 DEBUG:boto:Method: HEAD 2024-04-12T03:55:10.125 DEBUG:boto:Path: /dkyges-58/g 2024-04-12T03:55:10.125 DEBUG:boto:Data: 2024-04-12T03:55:10.125 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.125 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.125 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.125 DEBUG:boto:Params: {} 2024-04-12T03:55:10.125 DEBUG:boto:Token: None 2024-04-12T03:55:10.125 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/g 2024-04-12T03:55:10.126 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gFK+vzbRzlr8Sw32LlxESNsEWWI= 2024-04-12T03:55:10.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gFK+vzbRzlr8Sw32LlxESNsEWWI=', 'Content-Length': '0'} 2024-04-12T03:55:10.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad0b09dd2b3faf91-006618b09e-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.128 DEBUG:boto:path=/dkyges-58/g 2024-04-12T03:55:10.128 DEBUG:boto:auth_path=/dkyges-58/g 2024-04-12T03:55:10.128 DEBUG:boto:Method: HEAD 2024-04-12T03:55:10.128 DEBUG:boto:Path: /dkyges-58/g 2024-04-12T03:55:10.128 DEBUG:boto:Data: 2024-04-12T03:55:10.128 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.128 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.128 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.128 DEBUG:boto:Params: {} 2024-04-12T03:55:10.129 DEBUG:boto:Token: None 2024-04-12T03:55:10.129 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/g 2024-04-12T03:55:10.129 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gFK+vzbRzlr8Sw32LlxESNsEWWI= 2024-04-12T03:55:10.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gFK+vzbRzlr8Sw32LlxESNsEWWI=', 'Content-Length': '0'} 2024-04-12T03:55:10.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2636c683139f65a-006618b09e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:55:10.131 DEBUG:boto:path=/dkyges-58/g 2024-04-12T03:55:10.131 DEBUG:boto:auth_path=/dkyges-58/g 2024-04-12T03:55:10.131 DEBUG:boto:Method: GET 2024-04-12T03:55:10.131 DEBUG:boto:Path: /dkyges-58/g 2024-04-12T03:55:10.131 DEBUG:boto:Data: 2024-04-12T03:55:10.132 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.132 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.132 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.132 DEBUG:boto:Params: {} 2024-04-12T03:55:10.132 DEBUG:boto:Token: None 2024-04-12T03:55:10.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/g 2024-04-12T03:55:10.132 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cNPNFpPsM77DEJIqYdHQ2Rui5GM= 2024-04-12T03:55:10.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cNPNFpPsM77DEJIqYdHQ2Rui5GM=', 'Content-Length': '0'} 2024-04-12T03:55:10.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000519c1864f79c1680-006618b09e-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.134 DEBUG:boto:path=/dkyges-58/g 2024-04-12T03:55:10.134 DEBUG:boto:auth_path=/dkyges-58/g 2024-04-12T03:55:10.134 DEBUG:boto:Method: GET 2024-04-12T03:55:10.135 DEBUG:boto:Path: /dkyges-58/g 2024-04-12T03:55:10.135 DEBUG:boto:Data: 2024-04-12T03:55:10.135 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.135 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.135 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.135 DEBUG:boto:Params: {} 2024-04-12T03:55:10.135 DEBUG:boto:Token: None 2024-04-12T03:55:10.135 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/g 2024-04-12T03:55:10.135 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cNPNFpPsM77DEJIqYdHQ2Rui5GM= 2024-04-12T03:55:10.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cNPNFpPsM77DEJIqYdHQ2Rui5GM=', 'Content-Length': '0'} 2024-04-12T03:55:10.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d85ddca49328508-006618b09e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.178 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:10.179 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:10.179 DEBUG:boto:path=/dkyges-58/?prefix=g 2024-04-12T03:55:10.179 DEBUG:boto:auth_path=/dkyges-58/?prefix=g 2024-04-12T03:55:10.179 DEBUG:boto:Method: GET 2024-04-12T03:55:10.179 DEBUG:boto:Path: /dkyges-58/?prefix=g 2024-04-12T03:55:10.179 DEBUG:boto:Data: 2024-04-12T03:55:10.179 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.179 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.179 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.179 DEBUG:boto:Params: {} 2024-04-12T03:55:10.179 DEBUG:boto:Token: None 2024-04-12T03:55:10.180 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/ 2024-04-12T03:55:10.180 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U= 2024-04-12T03:55:10.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U=', 'Content-Length': '0'} 2024-04-12T03:55:10.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f19b0058a7ff2d0a-006618b09e-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.182 DEBUG:boto:b'dkyges-58g1000falseg2024-04-12T03:52:43.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:10.183 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:10.183 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:10.183 DEBUG:boto:path=/dkyges-58/?prefix=g 2024-04-12T03:55:10.183 DEBUG:boto:auth_path=/dkyges-58/?prefix=g 2024-04-12T03:55:10.183 DEBUG:boto:Method: GET 2024-04-12T03:55:10.183 DEBUG:boto:Path: /dkyges-58/?prefix=g 2024-04-12T03:55:10.183 DEBUG:boto:Data: 2024-04-12T03:55:10.183 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.183 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.183 DEBUG:boto:Params: {} 2024-04-12T03:55:10.183 DEBUG:boto:Token: None 2024-04-12T03:55:10.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/ 2024-04-12T03:55:10.184 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U= 2024-04-12T03:55:10.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U=', 'Content-Length': '0'} 2024-04-12T03:55:10.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6ed5e929805bacb-006618b09e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.186 DEBUG:boto:b'dkyges-58g1000falseg2024-04-12T03:52:43.028Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:10.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:55:10.187 DEBUG:boto:path=/dkyges-58/h 2024-04-12T03:55:10.187 DEBUG:boto:auth_path=/dkyges-58/h 2024-04-12T03:55:10.187 DEBUG:boto:path=/dkyges-58/h?versionId=null 2024-04-12T03:55:10.187 DEBUG:boto:auth_path=/dkyges-58/h?versionId=null 2024-04-12T03:55:10.187 DEBUG:boto:Method: GET 2024-04-12T03:55:10.187 DEBUG:boto:Path: /dkyges-58/h?versionId=null 2024-04-12T03:55:10.187 DEBUG:boto:Data: 2024-04-12T03:55:10.187 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.187 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.187 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.187 DEBUG:boto:Params: {} 2024-04-12T03:55:10.187 DEBUG:boto:Token: None 2024-04-12T03:55:10.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/h?versionId=null 2024-04-12T03:55:10.188 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iiK3g5UDmlotDqsvMUA1nP0ApwE= 2024-04-12T03:55:10.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iiK3g5UDmlotDqsvMUA1nP0ApwE=', 'Content-Length': '0'} 2024-04-12T03:55:10.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062eb32ef730eb3ca-006618b09e-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.190 DEBUG:boto:path=/dkyges-58/h 2024-04-12T03:55:10.190 DEBUG:boto:auth_path=/dkyges-58/h 2024-04-12T03:55:10.190 DEBUG:boto:path=/dkyges-58/h?versionId=null 2024-04-12T03:55:10.190 DEBUG:boto:auth_path=/dkyges-58/h?versionId=null 2024-04-12T03:55:10.190 DEBUG:boto:Method: GET 2024-04-12T03:55:10.190 DEBUG:boto:Path: /dkyges-58/h?versionId=null 2024-04-12T03:55:10.190 DEBUG:boto:Data: 2024-04-12T03:55:10.190 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.190 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.191 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.191 DEBUG:boto:Params: {} 2024-04-12T03:55:10.191 DEBUG:boto:Token: None 2024-04-12T03:55:10.191 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/h?versionId=null 2024-04-12T03:55:10.191 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iiK3g5UDmlotDqsvMUA1nP0ApwE= 2024-04-12T03:55:10.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iiK3g5UDmlotDqsvMUA1nP0ApwE=', 'Content-Length': '0'} 2024-04-12T03:55:10.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ce19872ca3a501c-006618b09e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.193 DEBUG:boto:path=/dkyges-58/h 2024-04-12T03:55:10.194 DEBUG:boto:auth_path=/dkyges-58/h 2024-04-12T03:55:10.194 DEBUG:boto:path=/dkyges-58/h?versionId=null 2024-04-12T03:55:10.194 DEBUG:boto:auth_path=/dkyges-58/h?versionId=null 2024-04-12T03:55:10.194 DEBUG:boto:Method: HEAD 2024-04-12T03:55:10.194 DEBUG:boto:Path: /dkyges-58/h?versionId=null 2024-04-12T03:55:10.194 DEBUG:boto:Data: 2024-04-12T03:55:10.194 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.194 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.194 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.194 DEBUG:boto:Params: {} 2024-04-12T03:55:10.194 DEBUG:boto:Token: None 2024-04-12T03:55:10.195 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/h?versionId=null 2024-04-12T03:55:10.195 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lLET7yJpI+KTIYaKOUi8UPyLE8g= 2024-04-12T03:55:10.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lLET7yJpI+KTIYaKOUi8UPyLE8g=', 'Content-Length': '0'} 2024-04-12T03:55:10.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000214d1604128d0e9c-006618b09e-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.197 DEBUG:boto:path=/dkyges-58/h 2024-04-12T03:55:10.197 DEBUG:boto:auth_path=/dkyges-58/h 2024-04-12T03:55:10.197 DEBUG:boto:path=/dkyges-58/h?versionId=null 2024-04-12T03:55:10.197 DEBUG:boto:auth_path=/dkyges-58/h?versionId=null 2024-04-12T03:55:10.197 DEBUG:boto:Method: HEAD 2024-04-12T03:55:10.197 DEBUG:boto:Path: /dkyges-58/h?versionId=null 2024-04-12T03:55:10.197 DEBUG:boto:Data: 2024-04-12T03:55:10.197 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.197 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.198 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.198 DEBUG:boto:Params: {} 2024-04-12T03:55:10.198 DEBUG:boto:Token: None 2024-04-12T03:55:10.198 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/h?versionId=null 2024-04-12T03:55:10.198 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lLET7yJpI+KTIYaKOUi8UPyLE8g= 2024-04-12T03:55:10.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lLET7yJpI+KTIYaKOUi8UPyLE8g=', 'Content-Length': '0'} 2024-04-12T03:55:10.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000137e6e9f5527ee40-006618b09e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:55:10.200 DEBUG:boto:path=/dkyges-58/h 2024-04-12T03:55:10.200 DEBUG:boto:auth_path=/dkyges-58/h 2024-04-12T03:55:10.200 DEBUG:boto:path=/dkyges-58/h?versionId=null 2024-04-12T03:55:10.200 DEBUG:boto:auth_path=/dkyges-58/h?versionId=null 2024-04-12T03:55:10.200 DEBUG:boto:Method: GET 2024-04-12T03:55:10.200 DEBUG:boto:Path: /dkyges-58/h?versionId=null 2024-04-12T03:55:10.201 DEBUG:boto:Data: 2024-04-12T03:55:10.201 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.201 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.201 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.201 DEBUG:boto:Params: {} 2024-04-12T03:55:10.201 DEBUG:boto:Token: None 2024-04-12T03:55:10.201 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/h?versionId=null 2024-04-12T03:55:10.201 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iiK3g5UDmlotDqsvMUA1nP0ApwE= 2024-04-12T03:55:10.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iiK3g5UDmlotDqsvMUA1nP0ApwE=', 'Content-Length': '0'} 2024-04-12T03:55:10.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7079b70ecc11e4e-006618b09e-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.203 DEBUG:boto:path=/dkyges-58/h 2024-04-12T03:55:10.203 DEBUG:boto:auth_path=/dkyges-58/h 2024-04-12T03:55:10.204 DEBUG:boto:path=/dkyges-58/h?versionId=null 2024-04-12T03:55:10.204 DEBUG:boto:auth_path=/dkyges-58/h?versionId=null 2024-04-12T03:55:10.204 DEBUG:boto:Method: GET 2024-04-12T03:55:10.204 DEBUG:boto:Path: /dkyges-58/h?versionId=null 2024-04-12T03:55:10.204 DEBUG:boto:Data: 2024-04-12T03:55:10.204 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.204 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.204 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.204 DEBUG:boto:Params: {} 2024-04-12T03:55:10.204 DEBUG:boto:Token: None 2024-04-12T03:55:10.204 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/h?versionId=null 2024-04-12T03:55:10.205 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iiK3g5UDmlotDqsvMUA1nP0ApwE= 2024-04-12T03:55:10.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iiK3g5UDmlotDqsvMUA1nP0ApwE=', 'Content-Length': '0'} 2024-04-12T03:55:10.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2ccfbffb437eea8-006618b09e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.207 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:10.207 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:10.207 DEBUG:boto:path=/dkyges-58/?prefix=h 2024-04-12T03:55:10.207 DEBUG:boto:auth_path=/dkyges-58/?prefix=h 2024-04-12T03:55:10.207 DEBUG:boto:Method: GET 2024-04-12T03:55:10.207 DEBUG:boto:Path: /dkyges-58/?prefix=h 2024-04-12T03:55:10.207 DEBUG:boto:Data: 2024-04-12T03:55:10.208 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.208 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.208 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.208 DEBUG:boto:Params: {} 2024-04-12T03:55:10.208 DEBUG:boto:Token: None 2024-04-12T03:55:10.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/ 2024-04-12T03:55:10.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U= 2024-04-12T03:55:10.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U=', 'Content-Length': '0'} 2024-04-12T03:55:10.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b09d1f8374af3d8-006618b09e-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.211 DEBUG:boto:b'dkyges-58h1000falseh2024-04-12T03:52:43.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:10.211 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:10.211 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:10.211 DEBUG:boto:path=/dkyges-58/?prefix=h 2024-04-12T03:55:10.211 DEBUG:boto:auth_path=/dkyges-58/?prefix=h 2024-04-12T03:55:10.211 DEBUG:boto:Method: GET 2024-04-12T03:55:10.212 DEBUG:boto:Path: /dkyges-58/?prefix=h 2024-04-12T03:55:10.212 DEBUG:boto:Data: 2024-04-12T03:55:10.212 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.212 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.212 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.212 DEBUG:boto:Params: {} 2024-04-12T03:55:10.212 DEBUG:boto:Token: None 2024-04-12T03:55:10.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/ 2024-04-12T03:55:10.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U= 2024-04-12T03:55:10.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U=', 'Content-Length': '0'} 2024-04-12T03:55:10.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc4f62318e358bbf-006618b09e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.214 DEBUG:boto:b'dkyges-58h1000falseh2024-04-12T03:52:43.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:10.214 DEBUG:boto:path=/dkyges-58/h 2024-04-12T03:55:10.215 DEBUG:boto:auth_path=/dkyges-58/h 2024-04-12T03:55:10.215 DEBUG:boto:Method: HEAD 2024-04-12T03:55:10.215 DEBUG:boto:Path: /dkyges-58/h 2024-04-12T03:55:10.215 DEBUG:boto:Data: 2024-04-12T03:55:10.215 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.215 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.215 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.215 DEBUG:boto:Params: {} 2024-04-12T03:55:10.215 DEBUG:boto:Token: None 2024-04-12T03:55:10.215 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/h 2024-04-12T03:55:10.215 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DSQS15abYc1mjvaZtV5QCd+TWtw= 2024-04-12T03:55:10.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DSQS15abYc1mjvaZtV5QCd+TWtw=', 'Content-Length': '0'} 2024-04-12T03:55:10.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3514c98efb53459-006618b09e-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.217 DEBUG:boto:path=/dkyges-58/h 2024-04-12T03:55:10.217 DEBUG:boto:auth_path=/dkyges-58/h 2024-04-12T03:55:10.218 DEBUG:boto:Method: HEAD 2024-04-12T03:55:10.218 DEBUG:boto:Path: /dkyges-58/h 2024-04-12T03:55:10.218 DEBUG:boto:Data: 2024-04-12T03:55:10.218 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.218 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.218 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.218 DEBUG:boto:Params: {} 2024-04-12T03:55:10.218 DEBUG:boto:Token: None 2024-04-12T03:55:10.218 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/h 2024-04-12T03:55:10.218 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DSQS15abYc1mjvaZtV5QCd+TWtw= 2024-04-12T03:55:10.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DSQS15abYc1mjvaZtV5QCd+TWtw=', 'Content-Length': '0'} 2024-04-12T03:55:10.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f3dbec400b695c-006618b09e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:55:10.221 DEBUG:boto:path=/dkyges-58/h 2024-04-12T03:55:10.221 DEBUG:boto:auth_path=/dkyges-58/h 2024-04-12T03:55:10.221 DEBUG:boto:Method: GET 2024-04-12T03:55:10.221 DEBUG:boto:Path: /dkyges-58/h 2024-04-12T03:55:10.221 DEBUG:boto:Data: 2024-04-12T03:55:10.221 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.221 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.221 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.221 DEBUG:boto:Params: {} 2024-04-12T03:55:10.221 DEBUG:boto:Token: None 2024-04-12T03:55:10.221 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/h 2024-04-12T03:55:10.222 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2FIrs/Vs3bI9D8EACUflHawWi8U= 2024-04-12T03:55:10.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2FIrs/Vs3bI9D8EACUflHawWi8U=', 'Content-Length': '0'} 2024-04-12T03:55:10.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-58:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b12168c74c4b71e-006618b09e-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.266 DEBUG:boto:path=/dkyges-58/h 2024-04-12T03:55:10.266 DEBUG:boto:auth_path=/dkyges-58/h 2024-04-12T03:55:10.267 DEBUG:boto:Method: GET 2024-04-12T03:55:10.267 DEBUG:boto:Path: /dkyges-58/h 2024-04-12T03:55:10.267 DEBUG:boto:Data: 2024-04-12T03:55:10.267 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.267 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.267 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.267 DEBUG:boto:Params: {} 2024-04-12T03:55:10.267 DEBUG:boto:Token: None 2024-04-12T03:55:10.267 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/h 2024-04-12T03:55:10.267 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2FIrs/Vs3bI9D8EACUflHawWi8U= 2024-04-12T03:55:10.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2FIrs/Vs3bI9D8EACUflHawWi8U=', 'Content-Length': '0'} 2024-04-12T03:55:10.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:52:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001467164b26352941-006618b09e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.270 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:10.270 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:10.270 DEBUG:boto:path=/dkyges-58/?prefix=h 2024-04-12T03:55:10.270 DEBUG:boto:auth_path=/dkyges-58/?prefix=h 2024-04-12T03:55:10.270 DEBUG:boto:Method: GET 2024-04-12T03:55:10.270 DEBUG:boto:Path: /dkyges-58/?prefix=h 2024-04-12T03:55:10.270 DEBUG:boto:Data: 2024-04-12T03:55:10.270 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.270 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:10.270 DEBUG:boto:Port: 8001 2024-04-12T03:55:10.270 DEBUG:boto:Params: {} 2024-04-12T03:55:10.271 DEBUG:boto:Token: None 2024-04-12T03:55:10.271 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/ 2024-04-12T03:55:10.271 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U= 2024-04-12T03:55:10.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U=', 'Content-Length': '0'} 2024-04-12T03:55:10.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e1a65d635bf9aa0-006618b09e-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.273 DEBUG:boto:b'dkyges-58h1000falseh2024-04-12T03:52:43.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:10.274 DEBUG:boto:path=/dkyges-58/ 2024-04-12T03:55:10.274 DEBUG:boto:auth_path=/dkyges-58/ 2024-04-12T03:55:10.274 DEBUG:boto:path=/dkyges-58/?prefix=h 2024-04-12T03:55:10.274 DEBUG:boto:auth_path=/dkyges-58/?prefix=h 2024-04-12T03:55:10.274 DEBUG:boto:Method: GET 2024-04-12T03:55:10.274 DEBUG:boto:Path: /dkyges-58/?prefix=h 2024-04-12T03:55:10.274 DEBUG:boto:Data: 2024-04-12T03:55:10.274 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.274 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.274 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.274 DEBUG:boto:Params: {} 2024-04-12T03:55:10.275 DEBUG:boto:Token: None 2024-04-12T03:55:10.275 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:55:10 GMT /dkyges-58/ 2024-04-12T03:55:10.275 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U= 2024-04-12T03:55:10.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2o0MrFtN5KCmcZPBBtl7NBVS94U=', 'Content-Length': '0'} 2024-04-12T03:55:10.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009218fbf6042a1882-006618b09e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.277 DEBUG:boto:b'dkyges-58h1000falseh2024-04-12T03:52:43.078Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:55:10.277 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-58 zones={a2, a1} 2024-04-12T03:55:10.278 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ok 2024-04-12T03:55:10.279 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:55:10.279 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:55:10.279 DEBUG:boto:Method: PUT 2024-04-12T03:55:10.279 DEBUG:boto:Path: /dkyges-59/ 2024-04-12T03:55:10.279 DEBUG:boto:Data: 2024-04-12T03:55:10.279 DEBUG:boto:Headers: {} 2024-04-12T03:55:10.279 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:10.280 DEBUG:boto:Port: 8000 2024-04-12T03:55:10.280 DEBUG:boto:Params: {} 2024-04-12T03:55:10.280 DEBUG:boto:Token: None 2024-04-12T03:55:10.280 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:55:10 GMT /dkyges-59/ 2024-04-12T03:55:10.280 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:phlq+thVu2KXddG0gWmhRGi83Qc= 2024-04-12T03:55:10.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:10 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:phlq+thVu2KXddG0gWmhRGi83Qc=', 'Content-Length': '0'} 2024-04-12T03:55:10.300 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000656a107f971bb39f-006618b09e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:10 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:10.300 DEBUG:rgw_multi.tests:created bucket=dkyges-59 2024-04-12T03:55:10.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:10.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:10.300 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:55:10.362 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:10.362 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:10.399 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:55:10.414 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:55:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:55:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:55:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:55:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:55:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:55:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:55:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:55:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:55:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:55:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:55:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:55:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:55:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:55:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894110.308772_537.1", 2024-04-12T03:55:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:55:10.308772Z" 2024-04-12T03:55:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:55:10.415 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:55:10.416 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894110.308772_537.1'} 2024-04-12T03:55:10.416 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:55:10.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:10.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:55:10.417 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:55:10.472 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:10.472 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:10.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:55:10.517+0000 7f7ce5849780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:10.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:55:10.518+0000 7f7ce5849780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:55:10.524 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:55:10.524 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:55:10.524 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:55:10.524 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:55:10.524 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:55:10.524 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:55:10.524 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:55:10.524 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:55:10.524 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:55:10.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:55:10.525 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:55:10.525 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:55:10.525 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:55:10.525 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:55:10.525 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:55:10.525 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:55:10.525 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:55:10.525 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:55:10.525 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:55:10.525 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:55:10.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:55:10.526 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:55:10.526 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:55:10.526 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:55:10.526 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:55:10.526 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:55:10.526 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:55:10.526 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:55:10.526 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:55:10.526 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:55:10.526 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:55:10.526 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:55:10.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:55:10.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:55:10.527 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:55:10.527 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:55:10.527 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:55:10.527 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:55:10.527 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:55:10.527 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:55:10.527 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:55:10.527 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:55:10.527 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:55:10.527 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:55:10.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:55:10.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:55:10.528 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:55:10.528 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:55:10.528 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:55:10.528 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894110.308772_537.1", 2024-04-12T03:55:10.528 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:55:10.528 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:55:10.528 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:55:10.528 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:55:10.308772Z", 2024-04-12T03:55:10.528 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:55:10.529 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:55:10.529 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:55:10.529 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:55:10.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:55:10.529 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:55:10.529 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:55:10.529 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:55:10.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:55:10.529 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:55:10Z" 2024-04-12T03:55:10.529 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:55:10.530 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894110.308772_537.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:55:10.308772Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:55:10Z" } 2024-04-12T03:55:10.530 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894110.308772_537.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:55:10.308772Z', 'realm_epoch': 2}}] 2024-04-12T03:55:10.531 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894110.308772_537.1'} 2024-04-12T03:55:10.531 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894110.308772_537.1'} 2024-04-12T03:55:10.531 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:55:10.531 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T03:55:10.531 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T03:55:10.532 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:55:10.542+0000 7f3427834700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 48986) UID: 0 2024-04-12T03:55:10.532 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:55:10.542+0000 7f342ca097c0 -1 shutting down 2024-04-12T03:55:15.210 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T03:55:16.633 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T03:55:16.633 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:55:16.633 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:55:16.633 DEBUG:boto:Method: HEAD 2024-04-12T03:55:16.633 DEBUG:boto:Path: /dkyges-59/ 2024-04-12T03:55:16.633 DEBUG:boto:Data: 2024-04-12T03:55:16.633 DEBUG:boto:Headers: {} 2024-04-12T03:55:16.634 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:16.634 DEBUG:boto:Port: 8000 2024-04-12T03:55:16.634 DEBUG:boto:Params: {} 2024-04-12T03:55:16.634 DEBUG:boto:Token: None 2024-04-12T03:55:16.634 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:16 GMT /dkyges-59/ 2024-04-12T03:55:16.634 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S7paIpyF6rONvdVUz6sKF75a+88= 2024-04-12T03:55:16.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:16 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S7paIpyF6rONvdVUz6sKF75a+88=', 'Content-Length': '0'} 2024-04-12T03:55:16.637 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be17f574b58aa2ff-006618b0a4-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:16.638 DEBUG:boto:path=/dkyges-59/a 2024-04-12T03:55:16.638 DEBUG:boto:auth_path=/dkyges-59/a 2024-04-12T03:55:16.638 DEBUG:boto:Method: PUT 2024-04-12T03:55:16.638 DEBUG:boto:Path: /dkyges-59/a 2024-04-12T03:55:16.638 DEBUG:boto:Data: 2024-04-12T03:55:16.638 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:55:16.638 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:16.638 DEBUG:boto:Port: 8000 2024-04-12T03:55:16.639 DEBUG:boto:Params: {} 2024-04-12T03:55:16.639 DEBUG:boto:Token: None 2024-04-12T03:55:16.639 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:55:16 GMT /dkyges-59/a 2024-04-12T03:55:16.639 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:N0UC27hoAIfqJeB2dKGyVSVd/pg= 2024-04-12T03:55:16.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:55:16 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:N0UC27hoAIfqJeB2dKGyVSVd/pg='} 2024-04-12T03:55:16.647 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d7ff66de217d54b-006618b0a4-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:16.647 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:55:16.647 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:55:16.647 DEBUG:boto:Method: HEAD 2024-04-12T03:55:16.647 DEBUG:boto:Path: /dkyges-59/ 2024-04-12T03:55:16.647 DEBUG:boto:Data: 2024-04-12T03:55:16.647 DEBUG:boto:Headers: {} 2024-04-12T03:55:16.647 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:16.647 DEBUG:boto:Port: 8000 2024-04-12T03:55:16.648 DEBUG:boto:Params: {} 2024-04-12T03:55:16.648 DEBUG:boto:Token: None 2024-04-12T03:55:16.648 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:16 GMT /dkyges-59/ 2024-04-12T03:55:16.648 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S7paIpyF6rONvdVUz6sKF75a+88= 2024-04-12T03:55:16.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:16 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S7paIpyF6rONvdVUz6sKF75a+88=', 'Content-Length': '0'} 2024-04-12T03:55:16.650 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6411a4c9f4a1b75-006618b0a4-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:16.651 DEBUG:boto:path=/dkyges-59/b 2024-04-12T03:55:16.651 DEBUG:boto:auth_path=/dkyges-59/b 2024-04-12T03:55:16.651 DEBUG:boto:Method: PUT 2024-04-12T03:55:16.651 DEBUG:boto:Path: /dkyges-59/b 2024-04-12T03:55:16.651 DEBUG:boto:Data: 2024-04-12T03:55:16.651 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:55:16.651 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:16.651 DEBUG:boto:Port: 8000 2024-04-12T03:55:16.651 DEBUG:boto:Params: {} 2024-04-12T03:55:16.652 DEBUG:boto:Token: None 2024-04-12T03:55:16.652 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:55:16 GMT /dkyges-59/b 2024-04-12T03:55:16.652 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hRkYoU22CDDQ1H0Z2eV1KJjufXc= 2024-04-12T03:55:16.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:55:16 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hRkYoU22CDDQ1H0Z2eV1KJjufXc='} 2024-04-12T03:55:16.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000076a23ee45116dd7b-006618b0a4-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:16.695 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:55:16.695 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:55:16.695 DEBUG:boto:Method: HEAD 2024-04-12T03:55:16.695 DEBUG:boto:Path: /dkyges-59/ 2024-04-12T03:55:16.695 DEBUG:boto:Data: 2024-04-12T03:55:16.695 DEBUG:boto:Headers: {} 2024-04-12T03:55:16.695 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:16.695 DEBUG:boto:Port: 8000 2024-04-12T03:55:16.695 DEBUG:boto:Params: {} 2024-04-12T03:55:16.696 DEBUG:boto:Token: None 2024-04-12T03:55:16.696 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:16 GMT /dkyges-59/ 2024-04-12T03:55:16.696 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S7paIpyF6rONvdVUz6sKF75a+88= 2024-04-12T03:55:16.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:16 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S7paIpyF6rONvdVUz6sKF75a+88=', 'Content-Length': '0'} 2024-04-12T03:55:16.698 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c82e35a502d24e84-006618b0a4-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:16.698 DEBUG:boto:path=/dkyges-59/c 2024-04-12T03:55:16.699 DEBUG:boto:auth_path=/dkyges-59/c 2024-04-12T03:55:16.699 DEBUG:boto:Method: PUT 2024-04-12T03:55:16.699 DEBUG:boto:Path: /dkyges-59/c 2024-04-12T03:55:16.699 DEBUG:boto:Data: 2024-04-12T03:55:16.699 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:55:16.699 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:16.699 DEBUG:boto:Port: 8000 2024-04-12T03:55:16.699 DEBUG:boto:Params: {} 2024-04-12T03:55:16.699 DEBUG:boto:Token: None 2024-04-12T03:55:16.700 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:55:16 GMT /dkyges-59/c 2024-04-12T03:55:16.700 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u6XRzdaEf/aA4CD8NsSZFHxK2FI= 2024-04-12T03:55:16.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:55:16 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u6XRzdaEf/aA4CD8NsSZFHxK2FI='} 2024-04-12T03:55:16.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0364ba5986a3fdd-006618b0a4-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:16.706 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:55:16.706 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:55:16.707 DEBUG:boto:Method: HEAD 2024-04-12T03:55:16.707 DEBUG:boto:Path: /dkyges-59/ 2024-04-12T03:55:16.707 DEBUG:boto:Data: 2024-04-12T03:55:16.707 DEBUG:boto:Headers: {} 2024-04-12T03:55:16.707 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:16.707 DEBUG:boto:Port: 8000 2024-04-12T03:55:16.707 DEBUG:boto:Params: {} 2024-04-12T03:55:16.707 DEBUG:boto:Token: None 2024-04-12T03:55:16.707 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:16 GMT /dkyges-59/ 2024-04-12T03:55:16.708 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S7paIpyF6rONvdVUz6sKF75a+88= 2024-04-12T03:55:16.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:16 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S7paIpyF6rONvdVUz6sKF75a+88=', 'Content-Length': '0'} 2024-04-12T03:55:16.710 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000420d52dfe9439d73-006618b0a4-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:16.710 DEBUG:boto:path=/dkyges-59/d 2024-04-12T03:55:16.710 DEBUG:boto:auth_path=/dkyges-59/d 2024-04-12T03:55:16.710 DEBUG:boto:Method: PUT 2024-04-12T03:55:16.711 DEBUG:boto:Path: /dkyges-59/d 2024-04-12T03:55:16.711 DEBUG:boto:Data: 2024-04-12T03:55:16.711 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:55:16.711 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:16.711 DEBUG:boto:Port: 8000 2024-04-12T03:55:16.711 DEBUG:boto:Params: {} 2024-04-12T03:55:16.711 DEBUG:boto:Token: None 2024-04-12T03:55:16.711 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:55:16 GMT /dkyges-59/d 2024-04-12T03:55:16.712 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eyST6XImmo+8373UM8fq+rLTa8U= 2024-04-12T03:55:16.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:55:16 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eyST6XImmo+8373UM8fq+rLTa8U='} 2024-04-12T03:55:16.718 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e54272b74604e7c-006618b0a4-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:16.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dkyges-59', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:55:16.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dkyges-59', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:55:16.719 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dkyges-59 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:55:16.778 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:16.778 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:16.825 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T03:55:16.825 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-59 2024-04-12T03:55:16.830 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 4 2024-04-12T03:55:16.875 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:55:16.875 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:55:16.875 DEBUG:boto:Method: HEAD 2024-04-12T03:55:16.875 DEBUG:boto:Path: /dkyges-59/ 2024-04-12T03:55:16.875 DEBUG:boto:Data: 2024-04-12T03:55:16.875 DEBUG:boto:Headers: {} 2024-04-12T03:55:16.875 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:16.875 DEBUG:boto:Port: 8000 2024-04-12T03:55:16.876 DEBUG:boto:Params: {} 2024-04-12T03:55:16.876 DEBUG:boto:Token: None 2024-04-12T03:55:16.876 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:16 GMT /dkyges-59/ 2024-04-12T03:55:16.876 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S7paIpyF6rONvdVUz6sKF75a+88= 2024-04-12T03:55:16.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:16 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S7paIpyF6rONvdVUz6sKF75a+88=', 'Content-Length': '0'} 2024-04-12T03:55:16.878 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079aca80485e4a3d9-006618b0a4-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:16.879 DEBUG:boto:path=/dkyges-59/e 2024-04-12T03:55:16.879 DEBUG:boto:auth_path=/dkyges-59/e 2024-04-12T03:55:16.879 DEBUG:boto:Method: PUT 2024-04-12T03:55:16.879 DEBUG:boto:Path: /dkyges-59/e 2024-04-12T03:55:16.879 DEBUG:boto:Data: 2024-04-12T03:55:16.879 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:55:16.879 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:16.879 DEBUG:boto:Port: 8000 2024-04-12T03:55:16.880 DEBUG:boto:Params: {} 2024-04-12T03:55:16.880 DEBUG:boto:Token: None 2024-04-12T03:55:16.880 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:55:16 GMT /dkyges-59/e 2024-04-12T03:55:16.880 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:K20oa+vezM2343zlAKl583catVg= 2024-04-12T03:55:16.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:55:16 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:K20oa+vezM2343zlAKl583catVg='} 2024-04-12T03:55:16.923 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b289bda417de81ba-006618b0a4-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:16.924 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:55:16.924 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:55:16.924 DEBUG:boto:Method: HEAD 2024-04-12T03:55:16.924 DEBUG:boto:Path: /dkyges-59/ 2024-04-12T03:55:16.925 DEBUG:boto:Data: 2024-04-12T03:55:16.925 DEBUG:boto:Headers: {} 2024-04-12T03:55:16.925 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:16.925 DEBUG:boto:Port: 8000 2024-04-12T03:55:16.926 DEBUG:boto:Params: {} 2024-04-12T03:55:16.926 DEBUG:boto:Token: None 2024-04-12T03:55:16.926 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:16 GMT /dkyges-59/ 2024-04-12T03:55:16.926 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S7paIpyF6rONvdVUz6sKF75a+88= 2024-04-12T03:55:16.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:16 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S7paIpyF6rONvdVUz6sKF75a+88=', 'Content-Length': '0'} 2024-04-12T03:55:16.931 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005ec722304c101b55-006618b0a4-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:16.931 DEBUG:boto:path=/dkyges-59/f 2024-04-12T03:55:16.931 DEBUG:boto:auth_path=/dkyges-59/f 2024-04-12T03:55:16.931 DEBUG:boto:Method: PUT 2024-04-12T03:55:16.932 DEBUG:boto:Path: /dkyges-59/f 2024-04-12T03:55:16.932 DEBUG:boto:Data: 2024-04-12T03:55:16.932 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:55:16.932 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:16.932 DEBUG:boto:Port: 8000 2024-04-12T03:55:16.932 DEBUG:boto:Params: {} 2024-04-12T03:55:16.932 DEBUG:boto:Token: None 2024-04-12T03:55:16.932 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:55:16 GMT /dkyges-59/f 2024-04-12T03:55:16.932 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2ohHaRGOjUL1LeY480Y4gJB3Sj0= 2024-04-12T03:55:16.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:55:16 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2ohHaRGOjUL1LeY480Y4gJB3Sj0='} 2024-04-12T03:55:16.944 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c0ba1572eecd89f-006618b0a4-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:16.944 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:55:16.944 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:55:16.945 DEBUG:boto:Method: HEAD 2024-04-12T03:55:16.945 DEBUG:boto:Path: /dkyges-59/ 2024-04-12T03:55:16.945 DEBUG:boto:Data: 2024-04-12T03:55:16.945 DEBUG:boto:Headers: {} 2024-04-12T03:55:16.945 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:16.945 DEBUG:boto:Port: 8000 2024-04-12T03:55:16.945 DEBUG:boto:Params: {} 2024-04-12T03:55:16.945 DEBUG:boto:Token: None 2024-04-12T03:55:16.946 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:16 GMT /dkyges-59/ 2024-04-12T03:55:16.946 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S7paIpyF6rONvdVUz6sKF75a+88= 2024-04-12T03:55:16.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:16 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S7paIpyF6rONvdVUz6sKF75a+88=', 'Content-Length': '0'} 2024-04-12T03:55:16.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bda5279255003790-006618b0a4-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:16.958 DEBUG:boto:path=/dkyges-59/g 2024-04-12T03:55:16.958 DEBUG:boto:auth_path=/dkyges-59/g 2024-04-12T03:55:16.958 DEBUG:boto:Method: PUT 2024-04-12T03:55:16.958 DEBUG:boto:Path: /dkyges-59/g 2024-04-12T03:55:16.958 DEBUG:boto:Data: 2024-04-12T03:55:16.958 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:55:16.959 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:16.959 DEBUG:boto:Port: 8000 2024-04-12T03:55:16.959 DEBUG:boto:Params: {} 2024-04-12T03:55:16.959 DEBUG:boto:Token: None 2024-04-12T03:55:16.959 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:55:16 GMT /dkyges-59/g 2024-04-12T03:55:16.959 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZNHswCSaEIwphq4wVZf3XaHLRiw= 2024-04-12T03:55:16.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:55:16 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZNHswCSaEIwphq4wVZf3XaHLRiw='} 2024-04-12T03:55:17.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2c235bd6da6fa5c-006618b0a4-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:17.011 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:55:17.012 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:55:17.012 DEBUG:boto:Method: HEAD 2024-04-12T03:55:17.012 DEBUG:boto:Path: /dkyges-59/ 2024-04-12T03:55:17.012 DEBUG:boto:Data: 2024-04-12T03:55:17.012 DEBUG:boto:Headers: {} 2024-04-12T03:55:17.012 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:17.012 DEBUG:boto:Port: 8000 2024-04-12T03:55:17.012 DEBUG:boto:Params: {} 2024-04-12T03:55:17.013 DEBUG:boto:Token: None 2024-04-12T03:55:17.013 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:55:17 GMT /dkyges-59/ 2024-04-12T03:55:17.013 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RVtFzXVZY9wBvKYjYiwne0j0q0s= 2024-04-12T03:55:17.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:55:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RVtFzXVZY9wBvKYjYiwne0j0q0s=', 'Content-Length': '0'} 2024-04-12T03:55:17.018 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bdf96de20d577b07-006618b0a5-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:55:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:17.018 DEBUG:boto:path=/dkyges-59/h 2024-04-12T03:55:17.019 DEBUG:boto:auth_path=/dkyges-59/h 2024-04-12T03:55:17.019 DEBUG:boto:Method: PUT 2024-04-12T03:55:17.019 DEBUG:boto:Path: /dkyges-59/h 2024-04-12T03:55:17.020 DEBUG:boto:Data: 2024-04-12T03:55:17.020 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:55:17.020 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:55:17.020 DEBUG:boto:Port: 8000 2024-04-12T03:55:17.020 DEBUG:boto:Params: {} 2024-04-12T03:55:17.020 DEBUG:boto:Token: None 2024-04-12T03:55:17.021 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:55:17 GMT /dkyges-59/h 2024-04-12T03:55:17.021 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3JY3Yuqme8+sDA6bsjp5RQF2MpE= 2024-04-12T03:55:17.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:55:17 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3JY3Yuqme8+sDA6bsjp5RQF2MpE='} 2024-04-12T03:55:17.032 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a8bf9e12d55b119-006618b0a5-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:55:17 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:55:17.032 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T03:55:17.032 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T03:55:17.038 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T03:55:17.038 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:17.064 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:55:17.064 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:55:17.065 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T03:55:17.066 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi043.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T03:55:18.067 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T03:55:18.098 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:55:18.099 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:55:18.101 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T03:55:18.102 INFO:teuthology.orchestra.run.smithi043.stdout:anonymous 2024-04-12T03:55:18.102 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-59 2024-04-12T03:55:18.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:55:18.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:55:18.103 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:55:18.210 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:55:18.211 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:55:18.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:55:18.263+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:55:20.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T03:55:23.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:55:23.263+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:55:28.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:55:28.263+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:55:33.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:55:33.264+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:55:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:55:38.265+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:55:43.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:55:43.265+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:55:48.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:55:48.266+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:55:53.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:55:53.266+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:55:58.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:55:58.267+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:56:03.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:03.268+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:56:08.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:08.268+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:56:13.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:13.269+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:56:18.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:18.270+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:56:23.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:23.270+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:56:28.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:28.271+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:56:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:33.272+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:56:38.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:38.272+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:56:43.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:43.272+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:56:48.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:48.273+0000 7fb0b4565780 1 waiting to reach incremental sync.. 2024-04-12T03:56:53.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:53.274+0000 7fb0b4565780 1 bucket sync caught up with source: 2024-04-12T03:56:53.263 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000003.48.6, 00000000002.78.6, 00000000002.80.6] 2024-04-12T03:56:53.264 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000003.48.6, 00000000002.78.6, 00000000002.80.6] 2024-04-12T03:56:53.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:53.274+0000 7fb0b4565780 0 bucket checkpoint complete 2024-04-12T03:56:53.283 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-59 2024-04-12T03:56:53.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:56:53.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:56:53.284 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:56:53.349 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:56:53.350 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:56:53.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:53.407+0000 7f158e04a780 1 waiting to reach incremental sync.. 2024-04-12T03:56:58.040 INFO:tasks.rgw.c1.client.0.smithi043.stdout:2024-04-12T03:56:58.051+0000 7f1e272bd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T03:56:58.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:58.408+0000 7f158e04a780 1 bucket sync caught up with source: 2024-04-12T03:56:58.398 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.1419.6, , , 00000000001.1416.6, 00000000001.2761.6, , , 00000000002.1736.6, , 00000000002.2554.6] 2024-04-12T03:56:58.398 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.1419.6, , , 00000000001.1416.6, 00000000001.2761.6, , , 00000000002.1736.6, , 00000000002.2554.6] 2024-04-12T03:56:58.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:58.408+0000 7f158e04a780 0 bucket checkpoint complete 2024-04-12T03:56:58.416 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-59 zones={a2, a1} 2024-04-12T03:56:58.416 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.417 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.417 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.417 DEBUG:boto:Path: /dkyges-59/ 2024-04-12T03:56:58.417 DEBUG:boto:Data: 2024-04-12T03:56:58.417 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.417 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.417 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.417 DEBUG:boto:Params: {} 2024-04-12T03:56:58.418 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:56:58.418 DEBUG:boto:Token: None 2024-04-12T03:56:58.419 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.419 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Erg8lbF1nkL09gZ7xbz2atA1Rbo= 2024-04-12T03:56:58.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Erg8lbF1nkL09gZ7xbz2atA1Rbo=', 'Content-Length': '0'} 2024-04-12T03:56:58.435 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9c69a61b46b4004-006618b10a-1c1a-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.435 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.435 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.435 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.435 DEBUG:boto:Path: /dkyges-59/ 2024-04-12T03:56:58.435 DEBUG:boto:Data: 2024-04-12T03:56:58.435 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.435 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.435 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.436 DEBUG:boto:Params: {} 2024-04-12T03:56:58.436 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T03:56:58.436 DEBUG:boto:Token: None 2024-04-12T03:56:58.436 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.436 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Erg8lbF1nkL09gZ7xbz2atA1Rbo= 2024-04-12T03:56:58.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Erg8lbF1nkL09gZ7xbz2atA1Rbo=', 'Content-Length': '0'} 2024-04-12T03:56:58.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017ea41bbc95a8e86-006618b10a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.468 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:56:58.468 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.469 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.469 DEBUG:boto:path=/dkyges-59/?versions&max-keys=999 2024-04-12T03:56:58.469 DEBUG:boto:auth_path=/dkyges-59/?versions&max-keys=999 2024-04-12T03:56:58.469 DEBUG:boto:Method: GET 2024-04-12T03:56:58.469 DEBUG:boto:Path: /dkyges-59/?versions&max-keys=999 2024-04-12T03:56:58.469 DEBUG:boto:Data: 2024-04-12T03:56:58.469 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.469 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.469 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.469 DEBUG:boto:Params: {} 2024-04-12T03:56:58.470 DEBUG:boto:Token: None 2024-04-12T03:56:58.470 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/?versions 2024-04-12T03:56:58.470 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6t6ojIdayVR2rZU9+PaUArHw4eY= 2024-04-12T03:56:58.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6t6ojIdayVR2rZU9+PaUArHw4eY=', 'Content-Length': '0'} 2024-04-12T03:56:58.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dee9ab6a02572c9-006618b10a-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.473 DEBUG:boto:b'dkyges-59999falseanulltrue2024-04-12T03:55:16.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:55:16.665Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:55:16.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:55:16.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:55:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:55:16.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:55:16.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:55:17.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.474 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:56:58.475 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:56:58.475 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:56:58.475 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:56:58.475 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-12T03:56:58.475 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-12T03:56:58.475 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-12T03:56:58.475 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-12T03:56:58.475 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:56:58.475 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.476 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.476 DEBUG:boto:path=/dkyges-59/?versions&max-keys=999 2024-04-12T03:56:58.476 DEBUG:boto:auth_path=/dkyges-59/?versions&max-keys=999 2024-04-12T03:56:58.476 DEBUG:boto:Method: GET 2024-04-12T03:56:58.476 DEBUG:boto:Path: /dkyges-59/?versions&max-keys=999 2024-04-12T03:56:58.476 DEBUG:boto:Data: 2024-04-12T03:56:58.476 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.476 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.476 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.476 DEBUG:boto:Params: {} 2024-04-12T03:56:58.477 DEBUG:boto:Token: None 2024-04-12T03:56:58.477 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/?versions 2024-04-12T03:56:58.477 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6t6ojIdayVR2rZU9+PaUArHw4eY= 2024-04-12T03:56:58.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6t6ojIdayVR2rZU9+PaUArHw4eY=', 'Content-Length': '0'} 2024-04-12T03:56:58.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ca6ca33c1b08513-006618b10a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.480 DEBUG:boto:b'dkyges-59999falseanulltrue2024-04-12T03:55:16.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:55:16.665Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:55:16.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:55:16.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:55:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:55:16.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:55:16.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:55:17.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.480 DEBUG:tasks.rgw_multi.zone_rados:o=a 2024-04-12T03:56:58.480 DEBUG:tasks.rgw_multi.zone_rados:o=b 2024-04-12T03:56:58.480 DEBUG:tasks.rgw_multi.zone_rados:o=c 2024-04-12T03:56:58.481 DEBUG:tasks.rgw_multi.zone_rados:o=d 2024-04-12T03:56:58.481 DEBUG:tasks.rgw_multi.zone_rados:o=e 2024-04-12T03:56:58.481 DEBUG:tasks.rgw_multi.zone_rados:o=f 2024-04-12T03:56:58.481 DEBUG:tasks.rgw_multi.zone_rados:o=g 2024-04-12T03:56:58.481 DEBUG:tasks.rgw_multi.zone_rados:o=h 2024-04-12T03:56:58.481 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.481 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.481 DEBUG:boto:path=/dkyges-59/?versions&max-keys=999 2024-04-12T03:56:58.481 DEBUG:boto:auth_path=/dkyges-59/?versions&max-keys=999 2024-04-12T03:56:58.481 DEBUG:boto:Method: GET 2024-04-12T03:56:58.481 DEBUG:boto:Path: /dkyges-59/?versions&max-keys=999 2024-04-12T03:56:58.481 DEBUG:boto:Data: 2024-04-12T03:56:58.482 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.482 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.482 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.482 DEBUG:boto:Params: {} 2024-04-12T03:56:58.482 DEBUG:boto:Token: None 2024-04-12T03:56:58.482 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/?versions 2024-04-12T03:56:58.482 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6t6ojIdayVR2rZU9+PaUArHw4eY= 2024-04-12T03:56:58.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6t6ojIdayVR2rZU9+PaUArHw4eY=', 'Content-Length': '0'} 2024-04-12T03:56:58.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ec091dfae519502-006618b10a-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.485 DEBUG:boto:b'dkyges-59999falseanulltrue2024-04-12T03:55:16.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:55:16.665Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:55:16.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:55:16.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:55:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:55:16.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:55:16.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:55:17.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.486 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.486 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.486 DEBUG:boto:path=/dkyges-59/?versions&max-keys=999 2024-04-12T03:56:58.486 DEBUG:boto:auth_path=/dkyges-59/?versions&max-keys=999 2024-04-12T03:56:58.487 DEBUG:boto:Method: GET 2024-04-12T03:56:58.487 DEBUG:boto:Path: /dkyges-59/?versions&max-keys=999 2024-04-12T03:56:58.487 DEBUG:boto:Data: 2024-04-12T03:56:58.487 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.487 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.487 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.487 DEBUG:boto:Params: {} 2024-04-12T03:56:58.487 DEBUG:boto:Token: None 2024-04-12T03:56:58.487 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/?versions 2024-04-12T03:56:58.487 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6t6ojIdayVR2rZU9+PaUArHw4eY= 2024-04-12T03:56:58.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6t6ojIdayVR2rZU9+PaUArHw4eY=', 'Content-Length': '0'} 2024-04-12T03:56:58.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7041eb462e9b488-006618b10a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.490 DEBUG:boto:b'dkyges-59999falseanulltrue2024-04-12T03:55:16.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:55:16.665Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:55:16.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:55:16.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:55:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:55:16.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:55:16.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:55:17.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:56:58.491 DEBUG:boto:path=/dkyges-59/a 2024-04-12T03:56:58.491 DEBUG:boto:auth_path=/dkyges-59/a 2024-04-12T03:56:58.491 DEBUG:boto:path=/dkyges-59/a?versionId=null 2024-04-12T03:56:58.491 DEBUG:boto:auth_path=/dkyges-59/a?versionId=null 2024-04-12T03:56:58.491 DEBUG:boto:Method: GET 2024-04-12T03:56:58.491 DEBUG:boto:Path: /dkyges-59/a?versionId=null 2024-04-12T03:56:58.491 DEBUG:boto:Data: 2024-04-12T03:56:58.491 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.491 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.491 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.491 DEBUG:boto:Params: {} 2024-04-12T03:56:58.492 DEBUG:boto:Token: None 2024-04-12T03:56:58.492 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/a?versionId=null 2024-04-12T03:56:58.492 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xK+cAcLepeLuP9DVh5nHLO/ZOnY= 2024-04-12T03:56:58.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xK+cAcLepeLuP9DVh5nHLO/ZOnY=', 'Content-Length': '0'} 2024-04-12T03:56:58.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001588650b05222dea-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.495 DEBUG:boto:path=/dkyges-59/a 2024-04-12T03:56:58.495 DEBUG:boto:auth_path=/dkyges-59/a 2024-04-12T03:56:58.495 DEBUG:boto:path=/dkyges-59/a?versionId=null 2024-04-12T03:56:58.495 DEBUG:boto:auth_path=/dkyges-59/a?versionId=null 2024-04-12T03:56:58.495 DEBUG:boto:Method: GET 2024-04-12T03:56:58.495 DEBUG:boto:Path: /dkyges-59/a?versionId=null 2024-04-12T03:56:58.495 DEBUG:boto:Data: 2024-04-12T03:56:58.495 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.495 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.495 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.496 DEBUG:boto:Params: {} 2024-04-12T03:56:58.496 DEBUG:boto:Token: None 2024-04-12T03:56:58.496 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/a?versionId=null 2024-04-12T03:56:58.496 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xK+cAcLepeLuP9DVh5nHLO/ZOnY= 2024-04-12T03:56:58.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xK+cAcLepeLuP9DVh5nHLO/ZOnY=', 'Content-Length': '0'} 2024-04-12T03:56:58.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd360e351ae8ef5c-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.498 DEBUG:boto:path=/dkyges-59/a 2024-04-12T03:56:58.498 DEBUG:boto:auth_path=/dkyges-59/a 2024-04-12T03:56:58.498 DEBUG:boto:path=/dkyges-59/a?versionId=null 2024-04-12T03:56:58.498 DEBUG:boto:auth_path=/dkyges-59/a?versionId=null 2024-04-12T03:56:58.499 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.499 DEBUG:boto:Path: /dkyges-59/a?versionId=null 2024-04-12T03:56:58.499 DEBUG:boto:Data: 2024-04-12T03:56:58.499 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.499 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.499 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.499 DEBUG:boto:Params: {} 2024-04-12T03:56:58.499 DEBUG:boto:Token: None 2024-04-12T03:56:58.499 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/a?versionId=null 2024-04-12T03:56:58.500 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mo6BFXbPMZa4SVYlSJIsl5VdhnY= 2024-04-12T03:56:58.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mo6BFXbPMZa4SVYlSJIsl5VdhnY=', 'Content-Length': '0'} 2024-04-12T03:56:58.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000532bf17407438370-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.501 DEBUG:boto:path=/dkyges-59/a 2024-04-12T03:56:58.502 DEBUG:boto:auth_path=/dkyges-59/a 2024-04-12T03:56:58.502 DEBUG:boto:path=/dkyges-59/a?versionId=null 2024-04-12T03:56:58.502 DEBUG:boto:auth_path=/dkyges-59/a?versionId=null 2024-04-12T03:56:58.502 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.502 DEBUG:boto:Path: /dkyges-59/a?versionId=null 2024-04-12T03:56:58.502 DEBUG:boto:Data: 2024-04-12T03:56:58.502 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.502 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.502 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.502 DEBUG:boto:Params: {} 2024-04-12T03:56:58.503 DEBUG:boto:Token: None 2024-04-12T03:56:58.503 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/a?versionId=null 2024-04-12T03:56:58.503 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mo6BFXbPMZa4SVYlSJIsl5VdhnY= 2024-04-12T03:56:58.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mo6BFXbPMZa4SVYlSJIsl5VdhnY=', 'Content-Length': '0'} 2024-04-12T03:56:58.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f1ea9306f717d8f-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:56:58.505 DEBUG:boto:path=/dkyges-59/a 2024-04-12T03:56:58.505 DEBUG:boto:auth_path=/dkyges-59/a 2024-04-12T03:56:58.505 DEBUG:boto:path=/dkyges-59/a?versionId=null 2024-04-12T03:56:58.505 DEBUG:boto:auth_path=/dkyges-59/a?versionId=null 2024-04-12T03:56:58.506 DEBUG:boto:Method: GET 2024-04-12T03:56:58.506 DEBUG:boto:Path: /dkyges-59/a?versionId=null 2024-04-12T03:56:58.506 DEBUG:boto:Data: 2024-04-12T03:56:58.506 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.506 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.506 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.506 DEBUG:boto:Params: {} 2024-04-12T03:56:58.506 DEBUG:boto:Token: None 2024-04-12T03:56:58.506 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/a?versionId=null 2024-04-12T03:56:58.507 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xK+cAcLepeLuP9DVh5nHLO/ZOnY= 2024-04-12T03:56:58.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xK+cAcLepeLuP9DVh5nHLO/ZOnY=', 'Content-Length': '0'} 2024-04-12T03:56:58.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6bed0d2d807d32a-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.509 DEBUG:boto:path=/dkyges-59/a 2024-04-12T03:56:58.509 DEBUG:boto:auth_path=/dkyges-59/a 2024-04-12T03:56:58.509 DEBUG:boto:path=/dkyges-59/a?versionId=null 2024-04-12T03:56:58.509 DEBUG:boto:auth_path=/dkyges-59/a?versionId=null 2024-04-12T03:56:58.509 DEBUG:boto:Method: GET 2024-04-12T03:56:58.509 DEBUG:boto:Path: /dkyges-59/a?versionId=null 2024-04-12T03:56:58.509 DEBUG:boto:Data: 2024-04-12T03:56:58.509 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.509 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.509 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.510 DEBUG:boto:Params: {} 2024-04-12T03:56:58.510 DEBUG:boto:Token: None 2024-04-12T03:56:58.510 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/a?versionId=null 2024-04-12T03:56:58.510 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xK+cAcLepeLuP9DVh5nHLO/ZOnY= 2024-04-12T03:56:58.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xK+cAcLepeLuP9DVh5nHLO/ZOnY=', 'Content-Length': '0'} 2024-04-12T03:56:58.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c42582cf64cb1684-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.512 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.513 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.513 DEBUG:boto:path=/dkyges-59/?prefix=a 2024-04-12T03:56:58.513 DEBUG:boto:auth_path=/dkyges-59/?prefix=a 2024-04-12T03:56:58.513 DEBUG:boto:Method: GET 2024-04-12T03:56:58.513 DEBUG:boto:Path: /dkyges-59/?prefix=a 2024-04-12T03:56:58.513 DEBUG:boto:Data: 2024-04-12T03:56:58.513 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.513 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.513 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.513 DEBUG:boto:Params: {} 2024-04-12T03:56:58.513 DEBUG:boto:Token: None 2024-04-12T03:56:58.513 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.514 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb007f6acf5c6987-006618b10a-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.516 DEBUG:boto:b'dkyges-59a1000falsea2024-04-12T03:55:16.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.517 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.517 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.517 DEBUG:boto:path=/dkyges-59/?prefix=a 2024-04-12T03:56:58.517 DEBUG:boto:auth_path=/dkyges-59/?prefix=a 2024-04-12T03:56:58.517 DEBUG:boto:Method: GET 2024-04-12T03:56:58.517 DEBUG:boto:Path: /dkyges-59/?prefix=a 2024-04-12T03:56:58.517 DEBUG:boto:Data: 2024-04-12T03:56:58.517 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.517 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.517 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.517 DEBUG:boto:Params: {} 2024-04-12T03:56:58.518 DEBUG:boto:Token: None 2024-04-12T03:56:58.518 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.518 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043cd294b15709cd2-006618b10a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.520 DEBUG:boto:b'dkyges-59a1000falsea2024-04-12T03:55:16.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.520 DEBUG:boto:path=/dkyges-59/a 2024-04-12T03:56:58.520 DEBUG:boto:auth_path=/dkyges-59/a 2024-04-12T03:56:58.520 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.521 DEBUG:boto:Path: /dkyges-59/a 2024-04-12T03:56:58.521 DEBUG:boto:Data: 2024-04-12T03:56:58.521 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.521 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.521 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.521 DEBUG:boto:Params: {} 2024-04-12T03:56:58.521 DEBUG:boto:Token: None 2024-04-12T03:56:58.521 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/a 2024-04-12T03:56:58.521 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BGSwckRPIuQarUN1AC54iTjte7s= 2024-04-12T03:56:58.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BGSwckRPIuQarUN1AC54iTjte7s=', 'Content-Length': '0'} 2024-04-12T03:56:58.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b42286c7bcff4f1-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.523 DEBUG:boto:path=/dkyges-59/a 2024-04-12T03:56:58.524 DEBUG:boto:auth_path=/dkyges-59/a 2024-04-12T03:56:58.524 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.524 DEBUG:boto:Path: /dkyges-59/a 2024-04-12T03:56:58.524 DEBUG:boto:Data: 2024-04-12T03:56:58.524 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.524 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.524 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.524 DEBUG:boto:Params: {} 2024-04-12T03:56:58.524 DEBUG:boto:Token: None 2024-04-12T03:56:58.524 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/a 2024-04-12T03:56:58.525 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BGSwckRPIuQarUN1AC54iTjte7s= 2024-04-12T03:56:58.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BGSwckRPIuQarUN1AC54iTjte7s=', 'Content-Length': '0'} 2024-04-12T03:56:58.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b6f356fb67dce1b-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2024-04-12T03:56:58.527 DEBUG:boto:path=/dkyges-59/a 2024-04-12T03:56:58.527 DEBUG:boto:auth_path=/dkyges-59/a 2024-04-12T03:56:58.527 DEBUG:boto:Method: GET 2024-04-12T03:56:58.527 DEBUG:boto:Path: /dkyges-59/a 2024-04-12T03:56:58.527 DEBUG:boto:Data: 2024-04-12T03:56:58.527 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.527 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.527 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.527 DEBUG:boto:Params: {} 2024-04-12T03:56:58.528 DEBUG:boto:Token: None 2024-04-12T03:56:58.528 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/a 2024-04-12T03:56:58.528 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2k4xmvP5SR88iptAdCiAN9GTGwo= 2024-04-12T03:56:58.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2k4xmvP5SR88iptAdCiAN9GTGwo=', 'Content-Length': '0'} 2024-04-12T03:56:58.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf10f46067f6d691-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.570 DEBUG:boto:path=/dkyges-59/a 2024-04-12T03:56:58.570 DEBUG:boto:auth_path=/dkyges-59/a 2024-04-12T03:56:58.570 DEBUG:boto:Method: GET 2024-04-12T03:56:58.571 DEBUG:boto:Path: /dkyges-59/a 2024-04-12T03:56:58.571 DEBUG:boto:Data: 2024-04-12T03:56:58.571 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.571 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.571 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.571 DEBUG:boto:Params: {} 2024-04-12T03:56:58.571 DEBUG:boto:Token: None 2024-04-12T03:56:58.571 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/a 2024-04-12T03:56:58.571 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2k4xmvP5SR88iptAdCiAN9GTGwo= 2024-04-12T03:56:58.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2k4xmvP5SR88iptAdCiAN9GTGwo=', 'Content-Length': '0'} 2024-04-12T03:56:58.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5e83e8e2d286c98-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.573 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.573 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.574 DEBUG:boto:path=/dkyges-59/?prefix=a 2024-04-12T03:56:58.574 DEBUG:boto:auth_path=/dkyges-59/?prefix=a 2024-04-12T03:56:58.574 DEBUG:boto:Method: GET 2024-04-12T03:56:58.574 DEBUG:boto:Path: /dkyges-59/?prefix=a 2024-04-12T03:56:58.574 DEBUG:boto:Data: 2024-04-12T03:56:58.574 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.574 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.574 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.574 DEBUG:boto:Params: {} 2024-04-12T03:56:58.574 DEBUG:boto:Token: None 2024-04-12T03:56:58.575 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.575 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068ce2dadccfab065-006618b10a-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.578 DEBUG:boto:b'dkyges-59a1000falsea2024-04-12T03:55:16.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.578 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.578 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.578 DEBUG:boto:path=/dkyges-59/?prefix=a 2024-04-12T03:56:58.579 DEBUG:boto:auth_path=/dkyges-59/?prefix=a 2024-04-12T03:56:58.579 DEBUG:boto:Method: GET 2024-04-12T03:56:58.579 DEBUG:boto:Path: /dkyges-59/?prefix=a 2024-04-12T03:56:58.579 DEBUG:boto:Data: 2024-04-12T03:56:58.579 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.579 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.579 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.579 DEBUG:boto:Params: {} 2024-04-12T03:56:58.579 DEBUG:boto:Token: None 2024-04-12T03:56:58.580 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.580 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002926620b8944e754-006618b10a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.582 DEBUG:boto:b'dkyges-59a1000falsea2024-04-12T03:55:16.652Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:56:58.582 DEBUG:boto:path=/dkyges-59/b 2024-04-12T03:56:58.582 DEBUG:boto:auth_path=/dkyges-59/b 2024-04-12T03:56:58.582 DEBUG:boto:path=/dkyges-59/b?versionId=null 2024-04-12T03:56:58.583 DEBUG:boto:auth_path=/dkyges-59/b?versionId=null 2024-04-12T03:56:58.583 DEBUG:boto:Method: GET 2024-04-12T03:56:58.583 DEBUG:boto:Path: /dkyges-59/b?versionId=null 2024-04-12T03:56:58.583 DEBUG:boto:Data: 2024-04-12T03:56:58.583 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.583 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.583 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.583 DEBUG:boto:Params: {} 2024-04-12T03:56:58.583 DEBUG:boto:Token: None 2024-04-12T03:56:58.583 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/b?versionId=null 2024-04-12T03:56:58.584 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BIyEdbGUdBQDQ1VvMFcWTGp2SXk= 2024-04-12T03:56:58.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BIyEdbGUdBQDQ1VvMFcWTGp2SXk=', 'Content-Length': '0'} 2024-04-12T03:56:58.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4fd9f0d99b7afd5-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.586 DEBUG:boto:path=/dkyges-59/b 2024-04-12T03:56:58.586 DEBUG:boto:auth_path=/dkyges-59/b 2024-04-12T03:56:58.586 DEBUG:boto:path=/dkyges-59/b?versionId=null 2024-04-12T03:56:58.586 DEBUG:boto:auth_path=/dkyges-59/b?versionId=null 2024-04-12T03:56:58.587 DEBUG:boto:Method: GET 2024-04-12T03:56:58.587 DEBUG:boto:Path: /dkyges-59/b?versionId=null 2024-04-12T03:56:58.587 DEBUG:boto:Data: 2024-04-12T03:56:58.587 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.587 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.587 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.587 DEBUG:boto:Params: {} 2024-04-12T03:56:58.587 DEBUG:boto:Token: None 2024-04-12T03:56:58.587 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/b?versionId=null 2024-04-12T03:56:58.587 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BIyEdbGUdBQDQ1VvMFcWTGp2SXk= 2024-04-12T03:56:58.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BIyEdbGUdBQDQ1VvMFcWTGp2SXk=', 'Content-Length': '0'} 2024-04-12T03:56:58.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000281f9767439c2539-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.590 DEBUG:boto:path=/dkyges-59/b 2024-04-12T03:56:58.590 DEBUG:boto:auth_path=/dkyges-59/b 2024-04-12T03:56:58.590 DEBUG:boto:path=/dkyges-59/b?versionId=null 2024-04-12T03:56:58.590 DEBUG:boto:auth_path=/dkyges-59/b?versionId=null 2024-04-12T03:56:58.590 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.590 DEBUG:boto:Path: /dkyges-59/b?versionId=null 2024-04-12T03:56:58.590 DEBUG:boto:Data: 2024-04-12T03:56:58.590 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.590 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.590 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.590 DEBUG:boto:Params: {} 2024-04-12T03:56:58.591 DEBUG:boto:Token: None 2024-04-12T03:56:58.591 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/b?versionId=null 2024-04-12T03:56:58.591 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GEfDjxw589sckXVDVKeBw+f0hAY= 2024-04-12T03:56:58.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GEfDjxw589sckXVDVKeBw+f0hAY=', 'Content-Length': '0'} 2024-04-12T03:56:58.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a859869f79c7427a-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.593 DEBUG:boto:path=/dkyges-59/b 2024-04-12T03:56:58.593 DEBUG:boto:auth_path=/dkyges-59/b 2024-04-12T03:56:58.593 DEBUG:boto:path=/dkyges-59/b?versionId=null 2024-04-12T03:56:58.593 DEBUG:boto:auth_path=/dkyges-59/b?versionId=null 2024-04-12T03:56:58.593 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.594 DEBUG:boto:Path: /dkyges-59/b?versionId=null 2024-04-12T03:56:58.594 DEBUG:boto:Data: 2024-04-12T03:56:58.594 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.594 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.594 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.594 DEBUG:boto:Params: {} 2024-04-12T03:56:58.594 DEBUG:boto:Token: None 2024-04-12T03:56:58.594 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/b?versionId=null 2024-04-12T03:56:58.594 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GEfDjxw589sckXVDVKeBw+f0hAY= 2024-04-12T03:56:58.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GEfDjxw589sckXVDVKeBw+f0hAY=', 'Content-Length': '0'} 2024-04-12T03:56:58.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000995cf47065f6a2df-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:56:58.597 DEBUG:boto:path=/dkyges-59/b 2024-04-12T03:56:58.597 DEBUG:boto:auth_path=/dkyges-59/b 2024-04-12T03:56:58.597 DEBUG:boto:path=/dkyges-59/b?versionId=null 2024-04-12T03:56:58.597 DEBUG:boto:auth_path=/dkyges-59/b?versionId=null 2024-04-12T03:56:58.597 DEBUG:boto:Method: GET 2024-04-12T03:56:58.597 DEBUG:boto:Path: /dkyges-59/b?versionId=null 2024-04-12T03:56:58.597 DEBUG:boto:Data: 2024-04-12T03:56:58.597 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.597 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.597 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.597 DEBUG:boto:Params: {} 2024-04-12T03:56:58.597 DEBUG:boto:Token: None 2024-04-12T03:56:58.598 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/b?versionId=null 2024-04-12T03:56:58.598 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BIyEdbGUdBQDQ1VvMFcWTGp2SXk= 2024-04-12T03:56:58.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BIyEdbGUdBQDQ1VvMFcWTGp2SXk=', 'Content-Length': '0'} 2024-04-12T03:56:58.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000362ce8e2a427f3d6-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.600 DEBUG:boto:path=/dkyges-59/b 2024-04-12T03:56:58.600 DEBUG:boto:auth_path=/dkyges-59/b 2024-04-12T03:56:58.600 DEBUG:boto:path=/dkyges-59/b?versionId=null 2024-04-12T03:56:58.600 DEBUG:boto:auth_path=/dkyges-59/b?versionId=null 2024-04-12T03:56:58.600 DEBUG:boto:Method: GET 2024-04-12T03:56:58.600 DEBUG:boto:Path: /dkyges-59/b?versionId=null 2024-04-12T03:56:58.600 DEBUG:boto:Data: 2024-04-12T03:56:58.601 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.601 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.601 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.601 DEBUG:boto:Params: {} 2024-04-12T03:56:58.601 DEBUG:boto:Token: None 2024-04-12T03:56:58.601 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/b?versionId=null 2024-04-12T03:56:58.601 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BIyEdbGUdBQDQ1VvMFcWTGp2SXk= 2024-04-12T03:56:58.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BIyEdbGUdBQDQ1VvMFcWTGp2SXk=', 'Content-Length': '0'} 2024-04-12T03:56:58.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9c756f6b5225426-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.603 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.603 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.603 DEBUG:boto:path=/dkyges-59/?prefix=b 2024-04-12T03:56:58.604 DEBUG:boto:auth_path=/dkyges-59/?prefix=b 2024-04-12T03:56:58.604 DEBUG:boto:Method: GET 2024-04-12T03:56:58.604 DEBUG:boto:Path: /dkyges-59/?prefix=b 2024-04-12T03:56:58.604 DEBUG:boto:Data: 2024-04-12T03:56:58.604 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.604 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.604 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.604 DEBUG:boto:Params: {} 2024-04-12T03:56:58.604 DEBUG:boto:Token: None 2024-04-12T03:56:58.604 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.607 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcb6a06b00e3eeb0-006618b10a-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.607 DEBUG:boto:b'dkyges-59b1000falseb2024-04-12T03:55:16.665Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.607 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.607 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.608 DEBUG:boto:path=/dkyges-59/?prefix=b 2024-04-12T03:56:58.608 DEBUG:boto:auth_path=/dkyges-59/?prefix=b 2024-04-12T03:56:58.608 DEBUG:boto:Method: GET 2024-04-12T03:56:58.608 DEBUG:boto:Path: /dkyges-59/?prefix=b 2024-04-12T03:56:58.608 DEBUG:boto:Data: 2024-04-12T03:56:58.608 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.608 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.608 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.608 DEBUG:boto:Params: {} 2024-04-12T03:56:58.608 DEBUG:boto:Token: None 2024-04-12T03:56:58.608 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.609 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c82a6c8a91d19d57-006618b10a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.654 DEBUG:boto:b'dkyges-59b1000falseb2024-04-12T03:55:16.665Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.655 DEBUG:boto:path=/dkyges-59/b 2024-04-12T03:56:58.655 DEBUG:boto:auth_path=/dkyges-59/b 2024-04-12T03:56:58.655 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.655 DEBUG:boto:Path: /dkyges-59/b 2024-04-12T03:56:58.655 DEBUG:boto:Data: 2024-04-12T03:56:58.655 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.655 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.655 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.655 DEBUG:boto:Params: {} 2024-04-12T03:56:58.655 DEBUG:boto:Token: None 2024-04-12T03:56:58.656 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/b 2024-04-12T03:56:58.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rqCjKmVo46pP17AcinHH9TzQHWI= 2024-04-12T03:56:58.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rqCjKmVo46pP17AcinHH9TzQHWI=', 'Content-Length': '0'} 2024-04-12T03:56:58.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001241917ed4686016-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.658 DEBUG:boto:path=/dkyges-59/b 2024-04-12T03:56:58.658 DEBUG:boto:auth_path=/dkyges-59/b 2024-04-12T03:56:58.658 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.658 DEBUG:boto:Path: /dkyges-59/b 2024-04-12T03:56:58.658 DEBUG:boto:Data: 2024-04-12T03:56:58.658 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.658 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.659 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.659 DEBUG:boto:Params: {} 2024-04-12T03:56:58.659 DEBUG:boto:Token: None 2024-04-12T03:56:58.659 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/b 2024-04-12T03:56:58.659 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rqCjKmVo46pP17AcinHH9TzQHWI= 2024-04-12T03:56:58.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rqCjKmVo46pP17AcinHH9TzQHWI=', 'Content-Length': '0'} 2024-04-12T03:56:58.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3c6ca1790e42d11-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2024-04-12T03:56:58.661 DEBUG:boto:path=/dkyges-59/b 2024-04-12T03:56:58.661 DEBUG:boto:auth_path=/dkyges-59/b 2024-04-12T03:56:58.662 DEBUG:boto:Method: GET 2024-04-12T03:56:58.662 DEBUG:boto:Path: /dkyges-59/b 2024-04-12T03:56:58.662 DEBUG:boto:Data: 2024-04-12T03:56:58.662 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.662 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.662 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.662 DEBUG:boto:Params: {} 2024-04-12T03:56:58.662 DEBUG:boto:Token: None 2024-04-12T03:56:58.662 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/b 2024-04-12T03:56:58.662 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Irg5chGdsaNjmx3zewylZpp/pAA= 2024-04-12T03:56:58.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Irg5chGdsaNjmx3zewylZpp/pAA=', 'Content-Length': '0'} 2024-04-12T03:56:58.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000121f5304c489e2af-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.665 DEBUG:boto:path=/dkyges-59/b 2024-04-12T03:56:58.665 DEBUG:boto:auth_path=/dkyges-59/b 2024-04-12T03:56:58.665 DEBUG:boto:Method: GET 2024-04-12T03:56:58.665 DEBUG:boto:Path: /dkyges-59/b 2024-04-12T03:56:58.665 DEBUG:boto:Data: 2024-04-12T03:56:58.665 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.665 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.665 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.665 DEBUG:boto:Params: {} 2024-04-12T03:56:58.665 DEBUG:boto:Token: None 2024-04-12T03:56:58.666 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/b 2024-04-12T03:56:58.666 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Irg5chGdsaNjmx3zewylZpp/pAA= 2024-04-12T03:56:58.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Irg5chGdsaNjmx3zewylZpp/pAA=', 'Content-Length': '0'} 2024-04-12T03:56:58.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0153e2c44f886c3-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.668 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.668 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.668 DEBUG:boto:path=/dkyges-59/?prefix=b 2024-04-12T03:56:58.668 DEBUG:boto:auth_path=/dkyges-59/?prefix=b 2024-04-12T03:56:58.668 DEBUG:boto:Method: GET 2024-04-12T03:56:58.668 DEBUG:boto:Path: /dkyges-59/?prefix=b 2024-04-12T03:56:58.668 DEBUG:boto:Data: 2024-04-12T03:56:58.668 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.668 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.669 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.669 DEBUG:boto:Params: {} 2024-04-12T03:56:58.669 DEBUG:boto:Token: None 2024-04-12T03:56:58.669 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.669 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dd0234b403b4d8f-006618b10a-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.672 DEBUG:boto:b'dkyges-59b1000falseb2024-04-12T03:55:16.665Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.672 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.672 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.673 DEBUG:boto:path=/dkyges-59/?prefix=b 2024-04-12T03:56:58.673 DEBUG:boto:auth_path=/dkyges-59/?prefix=b 2024-04-12T03:56:58.673 DEBUG:boto:Method: GET 2024-04-12T03:56:58.673 DEBUG:boto:Path: /dkyges-59/?prefix=b 2024-04-12T03:56:58.673 DEBUG:boto:Data: 2024-04-12T03:56:58.673 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.673 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.673 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.673 DEBUG:boto:Params: {} 2024-04-12T03:56:58.673 DEBUG:boto:Token: None 2024-04-12T03:56:58.673 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.674 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048a038eb7b7247aa-006618b10a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.676 DEBUG:boto:b'dkyges-59b1000falseb2024-04-12T03:55:16.665Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:56:58.676 DEBUG:boto:path=/dkyges-59/c 2024-04-12T03:56:58.677 DEBUG:boto:auth_path=/dkyges-59/c 2024-04-12T03:56:58.677 DEBUG:boto:path=/dkyges-59/c?versionId=null 2024-04-12T03:56:58.677 DEBUG:boto:auth_path=/dkyges-59/c?versionId=null 2024-04-12T03:56:58.677 DEBUG:boto:Method: GET 2024-04-12T03:56:58.677 DEBUG:boto:Path: /dkyges-59/c?versionId=null 2024-04-12T03:56:58.677 DEBUG:boto:Data: 2024-04-12T03:56:58.677 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.677 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.677 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.677 DEBUG:boto:Params: {} 2024-04-12T03:56:58.677 DEBUG:boto:Token: None 2024-04-12T03:56:58.677 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/c?versionId=null 2024-04-12T03:56:58.678 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9avykvxNUQZA9HLKNwCYPZTHXG4= 2024-04-12T03:56:58.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9avykvxNUQZA9HLKNwCYPZTHXG4=', 'Content-Length': '0'} 2024-04-12T03:56:58.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b51e5c0ae3e30326-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.680 DEBUG:boto:path=/dkyges-59/c 2024-04-12T03:56:58.680 DEBUG:boto:auth_path=/dkyges-59/c 2024-04-12T03:56:58.680 DEBUG:boto:path=/dkyges-59/c?versionId=null 2024-04-12T03:56:58.680 DEBUG:boto:auth_path=/dkyges-59/c?versionId=null 2024-04-12T03:56:58.680 DEBUG:boto:Method: GET 2024-04-12T03:56:58.680 DEBUG:boto:Path: /dkyges-59/c?versionId=null 2024-04-12T03:56:58.680 DEBUG:boto:Data: 2024-04-12T03:56:58.680 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.680 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.680 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.680 DEBUG:boto:Params: {} 2024-04-12T03:56:58.681 DEBUG:boto:Token: None 2024-04-12T03:56:58.681 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/c?versionId=null 2024-04-12T03:56:58.681 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9avykvxNUQZA9HLKNwCYPZTHXG4= 2024-04-12T03:56:58.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9avykvxNUQZA9HLKNwCYPZTHXG4=', 'Content-Length': '0'} 2024-04-12T03:56:58.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fcb1d7580c9f650-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.683 DEBUG:boto:path=/dkyges-59/c 2024-04-12T03:56:58.683 DEBUG:boto:auth_path=/dkyges-59/c 2024-04-12T03:56:58.683 DEBUG:boto:path=/dkyges-59/c?versionId=null 2024-04-12T03:56:58.684 DEBUG:boto:auth_path=/dkyges-59/c?versionId=null 2024-04-12T03:56:58.684 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.684 DEBUG:boto:Path: /dkyges-59/c?versionId=null 2024-04-12T03:56:58.684 DEBUG:boto:Data: 2024-04-12T03:56:58.684 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.684 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.684 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.684 DEBUG:boto:Params: {} 2024-04-12T03:56:58.684 DEBUG:boto:Token: None 2024-04-12T03:56:58.684 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/c?versionId=null 2024-04-12T03:56:58.685 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VdA6daD9CLlWPEyHe2ZMukBgC+Q= 2024-04-12T03:56:58.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VdA6daD9CLlWPEyHe2ZMukBgC+Q=', 'Content-Length': '0'} 2024-04-12T03:56:58.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064c6f346f0015723-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.686 DEBUG:boto:path=/dkyges-59/c 2024-04-12T03:56:58.686 DEBUG:boto:auth_path=/dkyges-59/c 2024-04-12T03:56:58.687 DEBUG:boto:path=/dkyges-59/c?versionId=null 2024-04-12T03:56:58.687 DEBUG:boto:auth_path=/dkyges-59/c?versionId=null 2024-04-12T03:56:58.687 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.687 DEBUG:boto:Path: /dkyges-59/c?versionId=null 2024-04-12T03:56:58.687 DEBUG:boto:Data: 2024-04-12T03:56:58.687 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.687 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.687 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.687 DEBUG:boto:Params: {} 2024-04-12T03:56:58.687 DEBUG:boto:Token: None 2024-04-12T03:56:58.687 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/c?versionId=null 2024-04-12T03:56:58.688 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VdA6daD9CLlWPEyHe2ZMukBgC+Q= 2024-04-12T03:56:58.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VdA6daD9CLlWPEyHe2ZMukBgC+Q=', 'Content-Length': '0'} 2024-04-12T03:56:58.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046233d344e829b91-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:56:58.689 DEBUG:boto:path=/dkyges-59/c 2024-04-12T03:56:58.690 DEBUG:boto:auth_path=/dkyges-59/c 2024-04-12T03:56:58.690 DEBUG:boto:path=/dkyges-59/c?versionId=null 2024-04-12T03:56:58.690 DEBUG:boto:auth_path=/dkyges-59/c?versionId=null 2024-04-12T03:56:58.690 DEBUG:boto:Method: GET 2024-04-12T03:56:58.690 DEBUG:boto:Path: /dkyges-59/c?versionId=null 2024-04-12T03:56:58.690 DEBUG:boto:Data: 2024-04-12T03:56:58.690 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.690 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.690 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.690 DEBUG:boto:Params: {} 2024-04-12T03:56:58.690 DEBUG:boto:Token: None 2024-04-12T03:56:58.691 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/c?versionId=null 2024-04-12T03:56:58.691 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9avykvxNUQZA9HLKNwCYPZTHXG4= 2024-04-12T03:56:58.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9avykvxNUQZA9HLKNwCYPZTHXG4=', 'Content-Length': '0'} 2024-04-12T03:56:58.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068e9fb028f168a14-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.734 DEBUG:boto:path=/dkyges-59/c 2024-04-12T03:56:58.734 DEBUG:boto:auth_path=/dkyges-59/c 2024-04-12T03:56:58.735 DEBUG:boto:path=/dkyges-59/c?versionId=null 2024-04-12T03:56:58.735 DEBUG:boto:auth_path=/dkyges-59/c?versionId=null 2024-04-12T03:56:58.735 DEBUG:boto:Method: GET 2024-04-12T03:56:58.735 DEBUG:boto:Path: /dkyges-59/c?versionId=null 2024-04-12T03:56:58.735 DEBUG:boto:Data: 2024-04-12T03:56:58.735 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.735 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.735 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.735 DEBUG:boto:Params: {} 2024-04-12T03:56:58.735 DEBUG:boto:Token: None 2024-04-12T03:56:58.736 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/c?versionId=null 2024-04-12T03:56:58.736 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9avykvxNUQZA9HLKNwCYPZTHXG4= 2024-04-12T03:56:58.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9avykvxNUQZA9HLKNwCYPZTHXG4=', 'Content-Length': '0'} 2024-04-12T03:56:58.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c57d2cfc243cf42-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.738 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.738 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.738 DEBUG:boto:path=/dkyges-59/?prefix=c 2024-04-12T03:56:58.738 DEBUG:boto:auth_path=/dkyges-59/?prefix=c 2024-04-12T03:56:58.739 DEBUG:boto:Method: GET 2024-04-12T03:56:58.739 DEBUG:boto:Path: /dkyges-59/?prefix=c 2024-04-12T03:56:58.739 DEBUG:boto:Data: 2024-04-12T03:56:58.739 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.739 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.739 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.739 DEBUG:boto:Params: {} 2024-04-12T03:56:58.739 DEBUG:boto:Token: None 2024-04-12T03:56:58.739 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.739 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb33fcfed780ffa2-006618b10a-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.742 DEBUG:boto:b'dkyges-59c1000falsec2024-04-12T03:55:16.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.742 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.742 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.742 DEBUG:boto:path=/dkyges-59/?prefix=c 2024-04-12T03:56:58.743 DEBUG:boto:auth_path=/dkyges-59/?prefix=c 2024-04-12T03:56:58.743 DEBUG:boto:Method: GET 2024-04-12T03:56:58.743 DEBUG:boto:Path: /dkyges-59/?prefix=c 2024-04-12T03:56:58.743 DEBUG:boto:Data: 2024-04-12T03:56:58.743 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.743 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.743 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.743 DEBUG:boto:Params: {} 2024-04-12T03:56:58.743 DEBUG:boto:Token: None 2024-04-12T03:56:58.743 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.743 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f1252c75ba865b0-006618b10a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.746 DEBUG:boto:b'dkyges-59c1000falsec2024-04-12T03:55:16.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.746 DEBUG:boto:path=/dkyges-59/c 2024-04-12T03:56:58.746 DEBUG:boto:auth_path=/dkyges-59/c 2024-04-12T03:56:58.746 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.746 DEBUG:boto:Path: /dkyges-59/c 2024-04-12T03:56:58.747 DEBUG:boto:Data: 2024-04-12T03:56:58.747 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.747 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.747 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.747 DEBUG:boto:Params: {} 2024-04-12T03:56:58.747 DEBUG:boto:Token: None 2024-04-12T03:56:58.747 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/c 2024-04-12T03:56:58.747 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rULwoUZR0BG0y9Y3bqGR0O5/3eA= 2024-04-12T03:56:58.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rULwoUZR0BG0y9Y3bqGR0O5/3eA=', 'Content-Length': '0'} 2024-04-12T03:56:58.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daed95845c3376da-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.749 DEBUG:boto:path=/dkyges-59/c 2024-04-12T03:56:58.749 DEBUG:boto:auth_path=/dkyges-59/c 2024-04-12T03:56:58.749 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.750 DEBUG:boto:Path: /dkyges-59/c 2024-04-12T03:56:58.750 DEBUG:boto:Data: 2024-04-12T03:56:58.750 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.750 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.750 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.750 DEBUG:boto:Params: {} 2024-04-12T03:56:58.750 DEBUG:boto:Token: None 2024-04-12T03:56:58.750 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/c 2024-04-12T03:56:58.750 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rULwoUZR0BG0y9Y3bqGR0O5/3eA= 2024-04-12T03:56:58.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rULwoUZR0BG0y9Y3bqGR0O5/3eA=', 'Content-Length': '0'} 2024-04-12T03:56:58.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b31abf89f5c1c0cb-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2024-04-12T03:56:58.752 DEBUG:boto:path=/dkyges-59/c 2024-04-12T03:56:58.753 DEBUG:boto:auth_path=/dkyges-59/c 2024-04-12T03:56:58.753 DEBUG:boto:Method: GET 2024-04-12T03:56:58.753 DEBUG:boto:Path: /dkyges-59/c 2024-04-12T03:56:58.753 DEBUG:boto:Data: 2024-04-12T03:56:58.753 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.753 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.753 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.753 DEBUG:boto:Params: {} 2024-04-12T03:56:58.753 DEBUG:boto:Token: None 2024-04-12T03:56:58.753 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/c 2024-04-12T03:56:58.754 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RQS23HihV70eO/MHaVtveKLRA30= 2024-04-12T03:56:58.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RQS23HihV70eO/MHaVtveKLRA30=', 'Content-Length': '0'} 2024-04-12T03:56:58.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5715eedf65c0219-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.756 DEBUG:boto:path=/dkyges-59/c 2024-04-12T03:56:58.756 DEBUG:boto:auth_path=/dkyges-59/c 2024-04-12T03:56:58.756 DEBUG:boto:Method: GET 2024-04-12T03:56:58.756 DEBUG:boto:Path: /dkyges-59/c 2024-04-12T03:56:58.756 DEBUG:boto:Data: 2024-04-12T03:56:58.756 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.756 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.756 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.756 DEBUG:boto:Params: {} 2024-04-12T03:56:58.756 DEBUG:boto:Token: None 2024-04-12T03:56:58.757 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/c 2024-04-12T03:56:58.757 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RQS23HihV70eO/MHaVtveKLRA30= 2024-04-12T03:56:58.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RQS23HihV70eO/MHaVtveKLRA30=', 'Content-Length': '0'} 2024-04-12T03:56:58.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081716901f3866f3f-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.759 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.759 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.759 DEBUG:boto:path=/dkyges-59/?prefix=c 2024-04-12T03:56:58.759 DEBUG:boto:auth_path=/dkyges-59/?prefix=c 2024-04-12T03:56:58.759 DEBUG:boto:Method: GET 2024-04-12T03:56:58.759 DEBUG:boto:Path: /dkyges-59/?prefix=c 2024-04-12T03:56:58.759 DEBUG:boto:Data: 2024-04-12T03:56:58.759 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.759 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.759 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.759 DEBUG:boto:Params: {} 2024-04-12T03:56:58.760 DEBUG:boto:Token: None 2024-04-12T03:56:58.760 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.760 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e7a948a89727218-006618b10a-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.763 DEBUG:boto:b'dkyges-59c1000falsec2024-04-12T03:55:16.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.763 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.763 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.763 DEBUG:boto:path=/dkyges-59/?prefix=c 2024-04-12T03:56:58.763 DEBUG:boto:auth_path=/dkyges-59/?prefix=c 2024-04-12T03:56:58.763 DEBUG:boto:Method: GET 2024-04-12T03:56:58.763 DEBUG:boto:Path: /dkyges-59/?prefix=c 2024-04-12T03:56:58.764 DEBUG:boto:Data: 2024-04-12T03:56:58.764 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.764 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.764 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.764 DEBUG:boto:Params: {} 2024-04-12T03:56:58.764 DEBUG:boto:Token: None 2024-04-12T03:56:58.764 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.764 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5ff44527bc36357-006618b10a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.766 DEBUG:boto:b'dkyges-59c1000falsec2024-04-12T03:55:16.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:56:58.767 DEBUG:boto:path=/dkyges-59/d 2024-04-12T03:56:58.767 DEBUG:boto:auth_path=/dkyges-59/d 2024-04-12T03:56:58.767 DEBUG:boto:path=/dkyges-59/d?versionId=null 2024-04-12T03:56:58.767 DEBUG:boto:auth_path=/dkyges-59/d?versionId=null 2024-04-12T03:56:58.767 DEBUG:boto:Method: GET 2024-04-12T03:56:58.767 DEBUG:boto:Path: /dkyges-59/d?versionId=null 2024-04-12T03:56:58.767 DEBUG:boto:Data: 2024-04-12T03:56:58.768 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.768 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.768 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.768 DEBUG:boto:Params: {} 2024-04-12T03:56:58.768 DEBUG:boto:Token: None 2024-04-12T03:56:58.768 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/d?versionId=null 2024-04-12T03:56:58.768 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d6zoeZIYGOw1x3vQIPdUd+SO04g= 2024-04-12T03:56:58.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d6zoeZIYGOw1x3vQIPdUd+SO04g=', 'Content-Length': '0'} 2024-04-12T03:56:58.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052c6ca3226ef63f4-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.770 DEBUG:boto:path=/dkyges-59/d 2024-04-12T03:56:58.770 DEBUG:boto:auth_path=/dkyges-59/d 2024-04-12T03:56:58.771 DEBUG:boto:path=/dkyges-59/d?versionId=null 2024-04-12T03:56:58.771 DEBUG:boto:auth_path=/dkyges-59/d?versionId=null 2024-04-12T03:56:58.771 DEBUG:boto:Method: GET 2024-04-12T03:56:58.771 DEBUG:boto:Path: /dkyges-59/d?versionId=null 2024-04-12T03:56:58.771 DEBUG:boto:Data: 2024-04-12T03:56:58.771 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.771 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.771 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.771 DEBUG:boto:Params: {} 2024-04-12T03:56:58.771 DEBUG:boto:Token: None 2024-04-12T03:56:58.771 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/d?versionId=null 2024-04-12T03:56:58.772 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d6zoeZIYGOw1x3vQIPdUd+SO04g= 2024-04-12T03:56:58.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d6zoeZIYGOw1x3vQIPdUd+SO04g=', 'Content-Length': '0'} 2024-04-12T03:56:58.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000636643228febbdac-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.774 DEBUG:boto:path=/dkyges-59/d 2024-04-12T03:56:58.774 DEBUG:boto:auth_path=/dkyges-59/d 2024-04-12T03:56:58.774 DEBUG:boto:path=/dkyges-59/d?versionId=null 2024-04-12T03:56:58.774 DEBUG:boto:auth_path=/dkyges-59/d?versionId=null 2024-04-12T03:56:58.774 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.775 DEBUG:boto:Path: /dkyges-59/d?versionId=null 2024-04-12T03:56:58.775 DEBUG:boto:Data: 2024-04-12T03:56:58.775 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.775 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.775 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.775 DEBUG:boto:Params: {} 2024-04-12T03:56:58.775 DEBUG:boto:Token: None 2024-04-12T03:56:58.775 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/d?versionId=null 2024-04-12T03:56:58.775 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XYsfgrhvcccITb+PJCw4Rucl70I= 2024-04-12T03:56:58.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XYsfgrhvcccITb+PJCw4Rucl70I=', 'Content-Length': '0'} 2024-04-12T03:56:58.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a142f61d614eadfb-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.777 DEBUG:boto:path=/dkyges-59/d 2024-04-12T03:56:58.777 DEBUG:boto:auth_path=/dkyges-59/d 2024-04-12T03:56:58.778 DEBUG:boto:path=/dkyges-59/d?versionId=null 2024-04-12T03:56:58.778 DEBUG:boto:auth_path=/dkyges-59/d?versionId=null 2024-04-12T03:56:58.778 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.778 DEBUG:boto:Path: /dkyges-59/d?versionId=null 2024-04-12T03:56:58.778 DEBUG:boto:Data: 2024-04-12T03:56:58.778 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.778 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.778 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.778 DEBUG:boto:Params: {} 2024-04-12T03:56:58.778 DEBUG:boto:Token: None 2024-04-12T03:56:58.778 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/d?versionId=null 2024-04-12T03:56:58.779 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XYsfgrhvcccITb+PJCw4Rucl70I= 2024-04-12T03:56:58.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XYsfgrhvcccITb+PJCw4Rucl70I=', 'Content-Length': '0'} 2024-04-12T03:56:58.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e387f0645568308-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:56:58.781 DEBUG:boto:path=/dkyges-59/d 2024-04-12T03:56:58.781 DEBUG:boto:auth_path=/dkyges-59/d 2024-04-12T03:56:58.781 DEBUG:boto:path=/dkyges-59/d?versionId=null 2024-04-12T03:56:58.781 DEBUG:boto:auth_path=/dkyges-59/d?versionId=null 2024-04-12T03:56:58.781 DEBUG:boto:Method: GET 2024-04-12T03:56:58.781 DEBUG:boto:Path: /dkyges-59/d?versionId=null 2024-04-12T03:56:58.781 DEBUG:boto:Data: 2024-04-12T03:56:58.781 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.781 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.781 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.782 DEBUG:boto:Params: {} 2024-04-12T03:56:58.782 DEBUG:boto:Token: None 2024-04-12T03:56:58.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/d?versionId=null 2024-04-12T03:56:58.782 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d6zoeZIYGOw1x3vQIPdUd+SO04g= 2024-04-12T03:56:58.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d6zoeZIYGOw1x3vQIPdUd+SO04g=', 'Content-Length': '0'} 2024-04-12T03:56:58.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaf14bf9e3a3a0cc-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.826 DEBUG:boto:path=/dkyges-59/d 2024-04-12T03:56:58.826 DEBUG:boto:auth_path=/dkyges-59/d 2024-04-12T03:56:58.827 DEBUG:boto:path=/dkyges-59/d?versionId=null 2024-04-12T03:56:58.827 DEBUG:boto:auth_path=/dkyges-59/d?versionId=null 2024-04-12T03:56:58.827 DEBUG:boto:Method: GET 2024-04-12T03:56:58.827 DEBUG:boto:Path: /dkyges-59/d?versionId=null 2024-04-12T03:56:58.827 DEBUG:boto:Data: 2024-04-12T03:56:58.827 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.827 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.827 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.827 DEBUG:boto:Params: {} 2024-04-12T03:56:58.827 DEBUG:boto:Token: None 2024-04-12T03:56:58.827 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/d?versionId=null 2024-04-12T03:56:58.828 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d6zoeZIYGOw1x3vQIPdUd+SO04g= 2024-04-12T03:56:58.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d6zoeZIYGOw1x3vQIPdUd+SO04g=', 'Content-Length': '0'} 2024-04-12T03:56:58.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000837e386c195de5a3-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.830 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.830 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.830 DEBUG:boto:path=/dkyges-59/?prefix=d 2024-04-12T03:56:58.830 DEBUG:boto:auth_path=/dkyges-59/?prefix=d 2024-04-12T03:56:58.830 DEBUG:boto:Method: GET 2024-04-12T03:56:58.830 DEBUG:boto:Path: /dkyges-59/?prefix=d 2024-04-12T03:56:58.830 DEBUG:boto:Data: 2024-04-12T03:56:58.830 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.830 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.831 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.831 DEBUG:boto:Params: {} 2024-04-12T03:56:58.831 DEBUG:boto:Token: None 2024-04-12T03:56:58.831 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.831 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fa1823b2a552e54-006618b10a-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.834 DEBUG:boto:b'dkyges-59d1000falsed2024-04-12T03:55:16.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.834 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.834 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.834 DEBUG:boto:path=/dkyges-59/?prefix=d 2024-04-12T03:56:58.834 DEBUG:boto:auth_path=/dkyges-59/?prefix=d 2024-04-12T03:56:58.834 DEBUG:boto:Method: GET 2024-04-12T03:56:58.834 DEBUG:boto:Path: /dkyges-59/?prefix=d 2024-04-12T03:56:58.834 DEBUG:boto:Data: 2024-04-12T03:56:58.834 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.835 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.835 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.835 DEBUG:boto:Params: {} 2024-04-12T03:56:58.835 DEBUG:boto:Token: None 2024-04-12T03:56:58.835 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.835 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009902e88a37d88007-006618b10a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.837 DEBUG:boto:b'dkyges-59d1000falsed2024-04-12T03:55:16.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.838 DEBUG:boto:path=/dkyges-59/d 2024-04-12T03:56:58.838 DEBUG:boto:auth_path=/dkyges-59/d 2024-04-12T03:56:58.838 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.838 DEBUG:boto:Path: /dkyges-59/d 2024-04-12T03:56:58.838 DEBUG:boto:Data: 2024-04-12T03:56:58.838 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.838 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.838 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.838 DEBUG:boto:Params: {} 2024-04-12T03:56:58.839 DEBUG:boto:Token: None 2024-04-12T03:56:58.839 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/d 2024-04-12T03:56:58.839 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OMrkHphsUmPPnP+3il9ycCU3SRs= 2024-04-12T03:56:58.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OMrkHphsUmPPnP+3il9ycCU3SRs=', 'Content-Length': '0'} 2024-04-12T03:56:58.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003671110898670118-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.841 DEBUG:boto:path=/dkyges-59/d 2024-04-12T03:56:58.841 DEBUG:boto:auth_path=/dkyges-59/d 2024-04-12T03:56:58.841 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.841 DEBUG:boto:Path: /dkyges-59/d 2024-04-12T03:56:58.841 DEBUG:boto:Data: 2024-04-12T03:56:58.841 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.841 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.841 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.841 DEBUG:boto:Params: {} 2024-04-12T03:56:58.841 DEBUG:boto:Token: None 2024-04-12T03:56:58.842 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/d 2024-04-12T03:56:58.842 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OMrkHphsUmPPnP+3il9ycCU3SRs= 2024-04-12T03:56:58.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OMrkHphsUmPPnP+3il9ycCU3SRs=', 'Content-Length': '0'} 2024-04-12T03:56:58.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaa682fe892cd54c-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2024-04-12T03:56:58.844 DEBUG:boto:path=/dkyges-59/d 2024-04-12T03:56:58.844 DEBUG:boto:auth_path=/dkyges-59/d 2024-04-12T03:56:58.844 DEBUG:boto:Method: GET 2024-04-12T03:56:58.844 DEBUG:boto:Path: /dkyges-59/d 2024-04-12T03:56:58.845 DEBUG:boto:Data: 2024-04-12T03:56:58.845 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.845 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.845 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.845 DEBUG:boto:Params: {} 2024-04-12T03:56:58.845 DEBUG:boto:Token: None 2024-04-12T03:56:58.845 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/d 2024-04-12T03:56:58.845 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8fvCq1uK9iiKb4c6drUYUpEdzig= 2024-04-12T03:56:58.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8fvCq1uK9iiKb4c6drUYUpEdzig=', 'Content-Length': '0'} 2024-04-12T03:56:58.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008effa5e4a7cc3fc5-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.847 DEBUG:boto:path=/dkyges-59/d 2024-04-12T03:56:58.847 DEBUG:boto:auth_path=/dkyges-59/d 2024-04-12T03:56:58.847 DEBUG:boto:Method: GET 2024-04-12T03:56:58.847 DEBUG:boto:Path: /dkyges-59/d 2024-04-12T03:56:58.848 DEBUG:boto:Data: 2024-04-12T03:56:58.848 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.848 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.848 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.848 DEBUG:boto:Params: {} 2024-04-12T03:56:58.848 DEBUG:boto:Token: None 2024-04-12T03:56:58.848 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/d 2024-04-12T03:56:58.848 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8fvCq1uK9iiKb4c6drUYUpEdzig= 2024-04-12T03:56:58.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8fvCq1uK9iiKb4c6drUYUpEdzig=', 'Content-Length': '0'} 2024-04-12T03:56:58.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aea38577a8937b5d-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.851 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.851 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.851 DEBUG:boto:path=/dkyges-59/?prefix=d 2024-04-12T03:56:58.851 DEBUG:boto:auth_path=/dkyges-59/?prefix=d 2024-04-12T03:56:58.851 DEBUG:boto:Method: GET 2024-04-12T03:56:58.851 DEBUG:boto:Path: /dkyges-59/?prefix=d 2024-04-12T03:56:58.851 DEBUG:boto:Data: 2024-04-12T03:56:58.851 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.851 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.851 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.851 DEBUG:boto:Params: {} 2024-04-12T03:56:58.852 DEBUG:boto:Token: None 2024-04-12T03:56:58.852 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.852 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b5b7b16386e75b7-006618b10a-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.854 DEBUG:boto:b'dkyges-59d1000falsed2024-04-12T03:55:16.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.855 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.855 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.855 DEBUG:boto:path=/dkyges-59/?prefix=d 2024-04-12T03:56:58.855 DEBUG:boto:auth_path=/dkyges-59/?prefix=d 2024-04-12T03:56:58.855 DEBUG:boto:Method: GET 2024-04-12T03:56:58.855 DEBUG:boto:Path: /dkyges-59/?prefix=d 2024-04-12T03:56:58.855 DEBUG:boto:Data: 2024-04-12T03:56:58.855 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.855 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.855 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.855 DEBUG:boto:Params: {} 2024-04-12T03:56:58.856 DEBUG:boto:Token: None 2024-04-12T03:56:58.856 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.856 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000716c7427c8cbc826-006618b10a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.898 DEBUG:boto:b'dkyges-59d1000falsed2024-04-12T03:55:16.724Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:56:58.899 DEBUG:boto:path=/dkyges-59/e 2024-04-12T03:56:58.899 DEBUG:boto:auth_path=/dkyges-59/e 2024-04-12T03:56:58.899 DEBUG:boto:path=/dkyges-59/e?versionId=null 2024-04-12T03:56:58.899 DEBUG:boto:auth_path=/dkyges-59/e?versionId=null 2024-04-12T03:56:58.899 DEBUG:boto:Method: GET 2024-04-12T03:56:58.899 DEBUG:boto:Path: /dkyges-59/e?versionId=null 2024-04-12T03:56:58.899 DEBUG:boto:Data: 2024-04-12T03:56:58.899 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.899 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.899 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.899 DEBUG:boto:Params: {} 2024-04-12T03:56:58.900 DEBUG:boto:Token: None 2024-04-12T03:56:58.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/e?versionId=null 2024-04-12T03:56:58.900 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iNXoVrDEa81jNCvFkWOpxHxZL2I= 2024-04-12T03:56:58.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iNXoVrDEa81jNCvFkWOpxHxZL2I=', 'Content-Length': '0'} 2024-04-12T03:56:58.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003230b1056ca9b2de-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.902 DEBUG:boto:path=/dkyges-59/e 2024-04-12T03:56:58.902 DEBUG:boto:auth_path=/dkyges-59/e 2024-04-12T03:56:58.902 DEBUG:boto:path=/dkyges-59/e?versionId=null 2024-04-12T03:56:58.902 DEBUG:boto:auth_path=/dkyges-59/e?versionId=null 2024-04-12T03:56:58.902 DEBUG:boto:Method: GET 2024-04-12T03:56:58.902 DEBUG:boto:Path: /dkyges-59/e?versionId=null 2024-04-12T03:56:58.902 DEBUG:boto:Data: 2024-04-12T03:56:58.903 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.903 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.903 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.903 DEBUG:boto:Params: {} 2024-04-12T03:56:58.903 DEBUG:boto:Token: None 2024-04-12T03:56:58.903 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/e?versionId=null 2024-04-12T03:56:58.903 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iNXoVrDEa81jNCvFkWOpxHxZL2I= 2024-04-12T03:56:58.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iNXoVrDEa81jNCvFkWOpxHxZL2I=', 'Content-Length': '0'} 2024-04-12T03:56:58.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073a3c61abbb8d3aa-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.905 DEBUG:boto:path=/dkyges-59/e 2024-04-12T03:56:58.906 DEBUG:boto:auth_path=/dkyges-59/e 2024-04-12T03:56:58.906 DEBUG:boto:path=/dkyges-59/e?versionId=null 2024-04-12T03:56:58.906 DEBUG:boto:auth_path=/dkyges-59/e?versionId=null 2024-04-12T03:56:58.906 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.906 DEBUG:boto:Path: /dkyges-59/e?versionId=null 2024-04-12T03:56:58.906 DEBUG:boto:Data: 2024-04-12T03:56:58.906 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.906 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.906 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.907 DEBUG:boto:Params: {} 2024-04-12T03:56:58.907 DEBUG:boto:Token: None 2024-04-12T03:56:58.907 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/e?versionId=null 2024-04-12T03:56:58.907 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5GTU+84VaKY9lVgik7C6ViiHUeY= 2024-04-12T03:56:58.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5GTU+84VaKY9lVgik7C6ViiHUeY=', 'Content-Length': '0'} 2024-04-12T03:56:58.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000683067a163074c4c-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.909 DEBUG:boto:path=/dkyges-59/e 2024-04-12T03:56:58.909 DEBUG:boto:auth_path=/dkyges-59/e 2024-04-12T03:56:58.909 DEBUG:boto:path=/dkyges-59/e?versionId=null 2024-04-12T03:56:58.909 DEBUG:boto:auth_path=/dkyges-59/e?versionId=null 2024-04-12T03:56:58.909 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.909 DEBUG:boto:Path: /dkyges-59/e?versionId=null 2024-04-12T03:56:58.909 DEBUG:boto:Data: 2024-04-12T03:56:58.910 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.910 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.910 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.910 DEBUG:boto:Params: {} 2024-04-12T03:56:58.910 DEBUG:boto:Token: None 2024-04-12T03:56:58.910 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/e?versionId=null 2024-04-12T03:56:58.910 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5GTU+84VaKY9lVgik7C6ViiHUeY= 2024-04-12T03:56:58.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5GTU+84VaKY9lVgik7C6ViiHUeY=', 'Content-Length': '0'} 2024-04-12T03:56:58.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004170c69ea03dcf09-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:56:58.912 DEBUG:boto:path=/dkyges-59/e 2024-04-12T03:56:58.913 DEBUG:boto:auth_path=/dkyges-59/e 2024-04-12T03:56:58.913 DEBUG:boto:path=/dkyges-59/e?versionId=null 2024-04-12T03:56:58.913 DEBUG:boto:auth_path=/dkyges-59/e?versionId=null 2024-04-12T03:56:58.913 DEBUG:boto:Method: GET 2024-04-12T03:56:58.913 DEBUG:boto:Path: /dkyges-59/e?versionId=null 2024-04-12T03:56:58.913 DEBUG:boto:Data: 2024-04-12T03:56:58.913 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.913 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.913 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.913 DEBUG:boto:Params: {} 2024-04-12T03:56:58.913 DEBUG:boto:Token: None 2024-04-12T03:56:58.914 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/e?versionId=null 2024-04-12T03:56:58.914 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iNXoVrDEa81jNCvFkWOpxHxZL2I= 2024-04-12T03:56:58.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iNXoVrDEa81jNCvFkWOpxHxZL2I=', 'Content-Length': '0'} 2024-04-12T03:56:58.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000939119ece838286a-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.916 DEBUG:boto:path=/dkyges-59/e 2024-04-12T03:56:58.916 DEBUG:boto:auth_path=/dkyges-59/e 2024-04-12T03:56:58.916 DEBUG:boto:path=/dkyges-59/e?versionId=null 2024-04-12T03:56:58.916 DEBUG:boto:auth_path=/dkyges-59/e?versionId=null 2024-04-12T03:56:58.916 DEBUG:boto:Method: GET 2024-04-12T03:56:58.916 DEBUG:boto:Path: /dkyges-59/e?versionId=null 2024-04-12T03:56:58.916 DEBUG:boto:Data: 2024-04-12T03:56:58.917 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.917 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.917 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.917 DEBUG:boto:Params: {} 2024-04-12T03:56:58.917 DEBUG:boto:Token: None 2024-04-12T03:56:58.917 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/e?versionId=null 2024-04-12T03:56:58.917 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iNXoVrDEa81jNCvFkWOpxHxZL2I= 2024-04-12T03:56:58.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iNXoVrDEa81jNCvFkWOpxHxZL2I=', 'Content-Length': '0'} 2024-04-12T03:56:58.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034daba62b49b4321-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.920 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.920 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.920 DEBUG:boto:path=/dkyges-59/?prefix=e 2024-04-12T03:56:58.920 DEBUG:boto:auth_path=/dkyges-59/?prefix=e 2024-04-12T03:56:58.920 DEBUG:boto:Method: GET 2024-04-12T03:56:58.920 DEBUG:boto:Path: /dkyges-59/?prefix=e 2024-04-12T03:56:58.920 DEBUG:boto:Data: 2024-04-12T03:56:58.921 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.921 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.921 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.921 DEBUG:boto:Params: {} 2024-04-12T03:56:58.921 DEBUG:boto:Token: None 2024-04-12T03:56:58.921 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.921 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000191a432c31f78bc3-006618b10a-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.924 DEBUG:boto:b'dkyges-59e1000falsee2024-04-12T03:55:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.925 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.925 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.925 DEBUG:boto:path=/dkyges-59/?prefix=e 2024-04-12T03:56:58.925 DEBUG:boto:auth_path=/dkyges-59/?prefix=e 2024-04-12T03:56:58.925 DEBUG:boto:Method: GET 2024-04-12T03:56:58.925 DEBUG:boto:Path: /dkyges-59/?prefix=e 2024-04-12T03:56:58.925 DEBUG:boto:Data: 2024-04-12T03:56:58.925 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.925 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.926 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.926 DEBUG:boto:Params: {} 2024-04-12T03:56:58.926 DEBUG:boto:Token: None 2024-04-12T03:56:58.926 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.926 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a85956c1613ad24e-006618b10a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.928 DEBUG:boto:b'dkyges-59e1000falsee2024-04-12T03:55:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.929 DEBUG:boto:path=/dkyges-59/e 2024-04-12T03:56:58.929 DEBUG:boto:auth_path=/dkyges-59/e 2024-04-12T03:56:58.929 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.929 DEBUG:boto:Path: /dkyges-59/e 2024-04-12T03:56:58.929 DEBUG:boto:Data: 2024-04-12T03:56:58.929 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.929 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.929 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.929 DEBUG:boto:Params: {} 2024-04-12T03:56:58.930 DEBUG:boto:Token: None 2024-04-12T03:56:58.930 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/e 2024-04-12T03:56:58.930 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OsWP+3xAvPnrFrZdIdmqKg7FGLg= 2024-04-12T03:56:58.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OsWP+3xAvPnrFrZdIdmqKg7FGLg=', 'Content-Length': '0'} 2024-04-12T03:56:58.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af7f452f5b340f6f-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.932 DEBUG:boto:path=/dkyges-59/e 2024-04-12T03:56:58.932 DEBUG:boto:auth_path=/dkyges-59/e 2024-04-12T03:56:58.932 DEBUG:boto:Method: HEAD 2024-04-12T03:56:58.932 DEBUG:boto:Path: /dkyges-59/e 2024-04-12T03:56:58.932 DEBUG:boto:Data: 2024-04-12T03:56:58.932 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.932 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.932 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.932 DEBUG:boto:Params: {} 2024-04-12T03:56:58.933 DEBUG:boto:Token: None 2024-04-12T03:56:58.933 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/e 2024-04-12T03:56:58.933 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OsWP+3xAvPnrFrZdIdmqKg7FGLg= 2024-04-12T03:56:58.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OsWP+3xAvPnrFrZdIdmqKg7FGLg=', 'Content-Length': '0'} 2024-04-12T03:56:58.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043ad85c1db9bfa72-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2024-04-12T03:56:58.935 DEBUG:boto:path=/dkyges-59/e 2024-04-12T03:56:58.935 DEBUG:boto:auth_path=/dkyges-59/e 2024-04-12T03:56:58.935 DEBUG:boto:Method: GET 2024-04-12T03:56:58.936 DEBUG:boto:Path: /dkyges-59/e 2024-04-12T03:56:58.936 DEBUG:boto:Data: 2024-04-12T03:56:58.936 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.936 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.936 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.936 DEBUG:boto:Params: {} 2024-04-12T03:56:58.936 DEBUG:boto:Token: None 2024-04-12T03:56:58.936 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/e 2024-04-12T03:56:58.936 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MrQHlW98dGQrY8odcCKLHa7pzXk= 2024-04-12T03:56:58.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MrQHlW98dGQrY8odcCKLHa7pzXk=', 'Content-Length': '0'} 2024-04-12T03:56:58.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002908ceedde70482f-006618b10a-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.982 DEBUG:boto:path=/dkyges-59/e 2024-04-12T03:56:58.982 DEBUG:boto:auth_path=/dkyges-59/e 2024-04-12T03:56:58.982 DEBUG:boto:Method: GET 2024-04-12T03:56:58.983 DEBUG:boto:Path: /dkyges-59/e 2024-04-12T03:56:58.983 DEBUG:boto:Data: 2024-04-12T03:56:58.983 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.983 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.983 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.983 DEBUG:boto:Params: {} 2024-04-12T03:56:58.983 DEBUG:boto:Token: None 2024-04-12T03:56:58.983 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/e 2024-04-12T03:56:58.983 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MrQHlW98dGQrY8odcCKLHa7pzXk= 2024-04-12T03:56:58.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MrQHlW98dGQrY8odcCKLHa7pzXk=', 'Content-Length': '0'} 2024-04-12T03:56:58.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005340a9de279f1029-006618b10a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.986 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.986 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.986 DEBUG:boto:path=/dkyges-59/?prefix=e 2024-04-12T03:56:58.986 DEBUG:boto:auth_path=/dkyges-59/?prefix=e 2024-04-12T03:56:58.986 DEBUG:boto:Method: GET 2024-04-12T03:56:58.986 DEBUG:boto:Path: /dkyges-59/?prefix=e 2024-04-12T03:56:58.986 DEBUG:boto:Data: 2024-04-12T03:56:58.987 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.987 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.987 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.987 DEBUG:boto:Params: {} 2024-04-12T03:56:58.987 DEBUG:boto:Token: None 2024-04-12T03:56:58.987 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.987 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c15e434ba1b26d28-006618b10b-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.990 DEBUG:boto:b'dkyges-59e1000falsee2024-04-12T03:55:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.990 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:58.990 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:58.990 DEBUG:boto:path=/dkyges-59/?prefix=e 2024-04-12T03:56:58.990 DEBUG:boto:auth_path=/dkyges-59/?prefix=e 2024-04-12T03:56:58.990 DEBUG:boto:Method: GET 2024-04-12T03:56:58.991 DEBUG:boto:Path: /dkyges-59/?prefix=e 2024-04-12T03:56:58.991 DEBUG:boto:Data: 2024-04-12T03:56:58.991 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.991 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.991 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.991 DEBUG:boto:Params: {} 2024-04-12T03:56:58.991 DEBUG:boto:Token: None 2024-04-12T03:56:58.991 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/ 2024-04-12T03:56:58.991 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI= 2024-04-12T03:56:58.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tQVSaUCcTfCIk4RYfM49rD25RhI=', 'Content-Length': '0'} 2024-04-12T03:56:58.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbe65b7e516b8fb4-006618b10b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.994 DEBUG:boto:b'dkyges-59e1000falsee2024-04-12T03:55:16.893Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:58.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:56:58.994 DEBUG:boto:path=/dkyges-59/f 2024-04-12T03:56:58.994 DEBUG:boto:auth_path=/dkyges-59/f 2024-04-12T03:56:58.994 DEBUG:boto:path=/dkyges-59/f?versionId=null 2024-04-12T03:56:58.995 DEBUG:boto:auth_path=/dkyges-59/f?versionId=null 2024-04-12T03:56:58.995 DEBUG:boto:Method: GET 2024-04-12T03:56:58.995 DEBUG:boto:Path: /dkyges-59/f?versionId=null 2024-04-12T03:56:58.995 DEBUG:boto:Data: 2024-04-12T03:56:58.995 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.995 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:58.995 DEBUG:boto:Port: 8001 2024-04-12T03:56:58.995 DEBUG:boto:Params: {} 2024-04-12T03:56:58.995 DEBUG:boto:Token: None 2024-04-12T03:56:58.995 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/f?versionId=null 2024-04-12T03:56:58.995 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SYHok14ypSNWOo4MC3BtndfZ+M4= 2024-04-12T03:56:58.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SYHok14ypSNWOo4MC3BtndfZ+M4=', 'Content-Length': '0'} 2024-04-12T03:56:58.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010854f957e42d44f-006618b10b-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:58.998 DEBUG:boto:path=/dkyges-59/f 2024-04-12T03:56:58.998 DEBUG:boto:auth_path=/dkyges-59/f 2024-04-12T03:56:58.998 DEBUG:boto:path=/dkyges-59/f?versionId=null 2024-04-12T03:56:58.998 DEBUG:boto:auth_path=/dkyges-59/f?versionId=null 2024-04-12T03:56:58.998 DEBUG:boto:Method: GET 2024-04-12T03:56:58.998 DEBUG:boto:Path: /dkyges-59/f?versionId=null 2024-04-12T03:56:58.998 DEBUG:boto:Data: 2024-04-12T03:56:58.998 DEBUG:boto:Headers: {} 2024-04-12T03:56:58.998 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:58.998 DEBUG:boto:Port: 8000 2024-04-12T03:56:58.999 DEBUG:boto:Params: {} 2024-04-12T03:56:58.999 DEBUG:boto:Token: None 2024-04-12T03:56:58.999 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:58 GMT /dkyges-59/f?versionId=null 2024-04-12T03:56:58.999 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SYHok14ypSNWOo4MC3BtndfZ+M4= 2024-04-12T03:56:58.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SYHok14ypSNWOo4MC3BtndfZ+M4=', 'Content-Length': '0'} 2024-04-12T03:56:59.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e815cf203f4eff71-006618b10b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.001 DEBUG:boto:path=/dkyges-59/f 2024-04-12T03:56:59.001 DEBUG:boto:auth_path=/dkyges-59/f 2024-04-12T03:56:59.001 DEBUG:boto:path=/dkyges-59/f?versionId=null 2024-04-12T03:56:59.001 DEBUG:boto:auth_path=/dkyges-59/f?versionId=null 2024-04-12T03:56:59.001 DEBUG:boto:Method: HEAD 2024-04-12T03:56:59.002 DEBUG:boto:Path: /dkyges-59/f?versionId=null 2024-04-12T03:56:59.002 DEBUG:boto:Data: 2024-04-12T03:56:59.002 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.002 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.002 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.002 DEBUG:boto:Params: {} 2024-04-12T03:56:59.002 DEBUG:boto:Token: None 2024-04-12T03:56:59.002 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/f?versionId=null 2024-04-12T03:56:59.002 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TvR2CGJgjGIkHvm1o65Urh3DXPg= 2024-04-12T03:56:59.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TvR2CGJgjGIkHvm1o65Urh3DXPg=', 'Content-Length': '0'} 2024-04-12T03:56:59.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000349b28c69ab81622-006618b10b-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.004 DEBUG:boto:path=/dkyges-59/f 2024-04-12T03:56:59.004 DEBUG:boto:auth_path=/dkyges-59/f 2024-04-12T03:56:59.004 DEBUG:boto:path=/dkyges-59/f?versionId=null 2024-04-12T03:56:59.005 DEBUG:boto:auth_path=/dkyges-59/f?versionId=null 2024-04-12T03:56:59.005 DEBUG:boto:Method: HEAD 2024-04-12T03:56:59.005 DEBUG:boto:Path: /dkyges-59/f?versionId=null 2024-04-12T03:56:59.005 DEBUG:boto:Data: 2024-04-12T03:56:59.005 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.005 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.005 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.005 DEBUG:boto:Params: {} 2024-04-12T03:56:59.005 DEBUG:boto:Token: None 2024-04-12T03:56:59.005 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/f?versionId=null 2024-04-12T03:56:59.006 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TvR2CGJgjGIkHvm1o65Urh3DXPg= 2024-04-12T03:56:59.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TvR2CGJgjGIkHvm1o65Urh3DXPg=', 'Content-Length': '0'} 2024-04-12T03:56:59.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000618474a4f7fb9151-006618b10b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:56:59.008 DEBUG:boto:path=/dkyges-59/f 2024-04-12T03:56:59.008 DEBUG:boto:auth_path=/dkyges-59/f 2024-04-12T03:56:59.008 DEBUG:boto:path=/dkyges-59/f?versionId=null 2024-04-12T03:56:59.008 DEBUG:boto:auth_path=/dkyges-59/f?versionId=null 2024-04-12T03:56:59.008 DEBUG:boto:Method: GET 2024-04-12T03:56:59.008 DEBUG:boto:Path: /dkyges-59/f?versionId=null 2024-04-12T03:56:59.008 DEBUG:boto:Data: 2024-04-12T03:56:59.008 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.008 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.009 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.009 DEBUG:boto:Params: {} 2024-04-12T03:56:59.009 DEBUG:boto:Token: None 2024-04-12T03:56:59.009 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/f?versionId=null 2024-04-12T03:56:59.009 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oysbgx/HqZp91hawriMdyphWr2k= 2024-04-12T03:56:59.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oysbgx/HqZp91hawriMdyphWr2k=', 'Content-Length': '0'} 2024-04-12T03:56:59.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035aac22964b816d7-006618b10b-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.011 DEBUG:boto:path=/dkyges-59/f 2024-04-12T03:56:59.011 DEBUG:boto:auth_path=/dkyges-59/f 2024-04-12T03:56:59.011 DEBUG:boto:path=/dkyges-59/f?versionId=null 2024-04-12T03:56:59.011 DEBUG:boto:auth_path=/dkyges-59/f?versionId=null 2024-04-12T03:56:59.011 DEBUG:boto:Method: GET 2024-04-12T03:56:59.011 DEBUG:boto:Path: /dkyges-59/f?versionId=null 2024-04-12T03:56:59.012 DEBUG:boto:Data: 2024-04-12T03:56:59.012 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.012 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.012 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.012 DEBUG:boto:Params: {} 2024-04-12T03:56:59.012 DEBUG:boto:Token: None 2024-04-12T03:56:59.012 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/f?versionId=null 2024-04-12T03:56:59.012 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oysbgx/HqZp91hawriMdyphWr2k= 2024-04-12T03:56:59.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oysbgx/HqZp91hawriMdyphWr2k=', 'Content-Length': '0'} 2024-04-12T03:56:59.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6469be872346693-006618b10b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.054 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:59.054 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:59.055 DEBUG:boto:path=/dkyges-59/?prefix=f 2024-04-12T03:56:59.055 DEBUG:boto:auth_path=/dkyges-59/?prefix=f 2024-04-12T03:56:59.055 DEBUG:boto:Method: GET 2024-04-12T03:56:59.055 DEBUG:boto:Path: /dkyges-59/?prefix=f 2024-04-12T03:56:59.055 DEBUG:boto:Data: 2024-04-12T03:56:59.055 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.055 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.055 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.055 DEBUG:boto:Params: {} 2024-04-12T03:56:59.055 DEBUG:boto:Token: None 2024-04-12T03:56:59.056 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/ 2024-04-12T03:56:59.056 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ= 2024-04-12T03:56:59.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ=', 'Content-Length': '0'} 2024-04-12T03:56:59.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb012b9c27ab8457-006618b10b-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.058 DEBUG:boto:b'dkyges-59f1000falsef2024-04-12T03:55:16.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:59.059 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:59.059 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:59.059 DEBUG:boto:path=/dkyges-59/?prefix=f 2024-04-12T03:56:59.059 DEBUG:boto:auth_path=/dkyges-59/?prefix=f 2024-04-12T03:56:59.059 DEBUG:boto:Method: GET 2024-04-12T03:56:59.059 DEBUG:boto:Path: /dkyges-59/?prefix=f 2024-04-12T03:56:59.060 DEBUG:boto:Data: 2024-04-12T03:56:59.060 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.060 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.060 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.060 DEBUG:boto:Params: {} 2024-04-12T03:56:59.060 DEBUG:boto:Token: None 2024-04-12T03:56:59.060 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/ 2024-04-12T03:56:59.060 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ= 2024-04-12T03:56:59.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ=', 'Content-Length': '0'} 2024-04-12T03:56:59.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009320e2d2db95ec73-006618b10b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.063 DEBUG:boto:b'dkyges-59f1000falsef2024-04-12T03:55:16.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:59.063 DEBUG:boto:path=/dkyges-59/f 2024-04-12T03:56:59.063 DEBUG:boto:auth_path=/dkyges-59/f 2024-04-12T03:56:59.063 DEBUG:boto:Method: HEAD 2024-04-12T03:56:59.063 DEBUG:boto:Path: /dkyges-59/f 2024-04-12T03:56:59.063 DEBUG:boto:Data: 2024-04-12T03:56:59.063 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.064 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.064 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.064 DEBUG:boto:Params: {} 2024-04-12T03:56:59.064 DEBUG:boto:Token: None 2024-04-12T03:56:59.064 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/f 2024-04-12T03:56:59.064 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:J4RCwMy1gvgLA0K4d3B+cAlMrFo= 2024-04-12T03:56:59.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:J4RCwMy1gvgLA0K4d3B+cAlMrFo=', 'Content-Length': '0'} 2024-04-12T03:56:59.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000206b2ab635d61f28-006618b10b-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.066 DEBUG:boto:path=/dkyges-59/f 2024-04-12T03:56:59.066 DEBUG:boto:auth_path=/dkyges-59/f 2024-04-12T03:56:59.066 DEBUG:boto:Method: HEAD 2024-04-12T03:56:59.067 DEBUG:boto:Path: /dkyges-59/f 2024-04-12T03:56:59.067 DEBUG:boto:Data: 2024-04-12T03:56:59.067 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.067 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.067 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.067 DEBUG:boto:Params: {} 2024-04-12T03:56:59.067 DEBUG:boto:Token: None 2024-04-12T03:56:59.067 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/f 2024-04-12T03:56:59.067 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:J4RCwMy1gvgLA0K4d3B+cAlMrFo= 2024-04-12T03:56:59.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:J4RCwMy1gvgLA0K4d3B+cAlMrFo=', 'Content-Length': '0'} 2024-04-12T03:56:59.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001415115615f65b1b-006618b10b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2024-04-12T03:56:59.070 DEBUG:boto:path=/dkyges-59/f 2024-04-12T03:56:59.070 DEBUG:boto:auth_path=/dkyges-59/f 2024-04-12T03:56:59.070 DEBUG:boto:Method: GET 2024-04-12T03:56:59.070 DEBUG:boto:Path: /dkyges-59/f 2024-04-12T03:56:59.070 DEBUG:boto:Data: 2024-04-12T03:56:59.070 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.070 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.070 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.070 DEBUG:boto:Params: {} 2024-04-12T03:56:59.071 DEBUG:boto:Token: None 2024-04-12T03:56:59.071 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/f 2024-04-12T03:56:59.071 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QVx544OTAnb8cf7zJiwzX8yr9ww= 2024-04-12T03:56:59.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QVx544OTAnb8cf7zJiwzX8yr9ww=', 'Content-Length': '0'} 2024-04-12T03:56:59.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001482c958dad7156a-006618b10b-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.073 DEBUG:boto:path=/dkyges-59/f 2024-04-12T03:56:59.073 DEBUG:boto:auth_path=/dkyges-59/f 2024-04-12T03:56:59.073 DEBUG:boto:Method: GET 2024-04-12T03:56:59.073 DEBUG:boto:Path: /dkyges-59/f 2024-04-12T03:56:59.073 DEBUG:boto:Data: 2024-04-12T03:56:59.073 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.073 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.074 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.074 DEBUG:boto:Params: {} 2024-04-12T03:56:59.074 DEBUG:boto:Token: None 2024-04-12T03:56:59.074 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/f 2024-04-12T03:56:59.074 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QVx544OTAnb8cf7zJiwzX8yr9ww= 2024-04-12T03:56:59.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QVx544OTAnb8cf7zJiwzX8yr9ww=', 'Content-Length': '0'} 2024-04-12T03:56:59.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f55710daf8cebe18-006618b10b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.076 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:59.076 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:59.076 DEBUG:boto:path=/dkyges-59/?prefix=f 2024-04-12T03:56:59.076 DEBUG:boto:auth_path=/dkyges-59/?prefix=f 2024-04-12T03:56:59.076 DEBUG:boto:Method: GET 2024-04-12T03:56:59.076 DEBUG:boto:Path: /dkyges-59/?prefix=f 2024-04-12T03:56:59.077 DEBUG:boto:Data: 2024-04-12T03:56:59.077 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.077 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.077 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.077 DEBUG:boto:Params: {} 2024-04-12T03:56:59.077 DEBUG:boto:Token: None 2024-04-12T03:56:59.077 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/ 2024-04-12T03:56:59.077 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ= 2024-04-12T03:56:59.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ=', 'Content-Length': '0'} 2024-04-12T03:56:59.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f1e171044806d6d-006618b10b-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.081 DEBUG:boto:b'dkyges-59f1000falsef2024-04-12T03:55:16.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:59.081 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:59.081 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:59.082 DEBUG:boto:path=/dkyges-59/?prefix=f 2024-04-12T03:56:59.082 DEBUG:boto:auth_path=/dkyges-59/?prefix=f 2024-04-12T03:56:59.082 DEBUG:boto:Method: GET 2024-04-12T03:56:59.082 DEBUG:boto:Path: /dkyges-59/?prefix=f 2024-04-12T03:56:59.082 DEBUG:boto:Data: 2024-04-12T03:56:59.082 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.082 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.082 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.082 DEBUG:boto:Params: {} 2024-04-12T03:56:59.082 DEBUG:boto:Token: None 2024-04-12T03:56:59.082 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/ 2024-04-12T03:56:59.083 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ= 2024-04-12T03:56:59.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ=', 'Content-Length': '0'} 2024-04-12T03:56:59.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f6331c69527036f-006618b10b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.085 DEBUG:boto:b'dkyges-59f1000falsef2024-04-12T03:55:16.949Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:59.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:56:59.085 DEBUG:boto:path=/dkyges-59/g 2024-04-12T03:56:59.085 DEBUG:boto:auth_path=/dkyges-59/g 2024-04-12T03:56:59.085 DEBUG:boto:path=/dkyges-59/g?versionId=null 2024-04-12T03:56:59.085 DEBUG:boto:auth_path=/dkyges-59/g?versionId=null 2024-04-12T03:56:59.086 DEBUG:boto:Method: GET 2024-04-12T03:56:59.086 DEBUG:boto:Path: /dkyges-59/g?versionId=null 2024-04-12T03:56:59.086 DEBUG:boto:Data: 2024-04-12T03:56:59.086 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.086 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.086 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.086 DEBUG:boto:Params: {} 2024-04-12T03:56:59.086 DEBUG:boto:Token: None 2024-04-12T03:56:59.086 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/g?versionId=null 2024-04-12T03:56:59.086 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e7G3aLmuRYQZ2ZsEt1i3lImaKCQ= 2024-04-12T03:56:59.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e7G3aLmuRYQZ2ZsEt1i3lImaKCQ=', 'Content-Length': '0'} 2024-04-12T03:56:59.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e648b46345568bf6-006618b10b-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.130 DEBUG:boto:path=/dkyges-59/g 2024-04-12T03:56:59.130 DEBUG:boto:auth_path=/dkyges-59/g 2024-04-12T03:56:59.131 DEBUG:boto:path=/dkyges-59/g?versionId=null 2024-04-12T03:56:59.131 DEBUG:boto:auth_path=/dkyges-59/g?versionId=null 2024-04-12T03:56:59.131 DEBUG:boto:Method: GET 2024-04-12T03:56:59.131 DEBUG:boto:Path: /dkyges-59/g?versionId=null 2024-04-12T03:56:59.131 DEBUG:boto:Data: 2024-04-12T03:56:59.131 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.131 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.131 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.131 DEBUG:boto:Params: {} 2024-04-12T03:56:59.131 DEBUG:boto:Token: None 2024-04-12T03:56:59.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/g?versionId=null 2024-04-12T03:56:59.132 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e7G3aLmuRYQZ2ZsEt1i3lImaKCQ= 2024-04-12T03:56:59.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e7G3aLmuRYQZ2ZsEt1i3lImaKCQ=', 'Content-Length': '0'} 2024-04-12T03:56:59.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038e078f4cec91375-006618b10b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.134 DEBUG:boto:path=/dkyges-59/g 2024-04-12T03:56:59.134 DEBUG:boto:auth_path=/dkyges-59/g 2024-04-12T03:56:59.134 DEBUG:boto:path=/dkyges-59/g?versionId=null 2024-04-12T03:56:59.135 DEBUG:boto:auth_path=/dkyges-59/g?versionId=null 2024-04-12T03:56:59.135 DEBUG:boto:Method: HEAD 2024-04-12T03:56:59.135 DEBUG:boto:Path: /dkyges-59/g?versionId=null 2024-04-12T03:56:59.135 DEBUG:boto:Data: 2024-04-12T03:56:59.135 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.135 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.135 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.135 DEBUG:boto:Params: {} 2024-04-12T03:56:59.135 DEBUG:boto:Token: None 2024-04-12T03:56:59.135 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/g?versionId=null 2024-04-12T03:56:59.136 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e2EI+t+zUFNCkv43CitrvV4gZUI= 2024-04-12T03:56:59.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e2EI+t+zUFNCkv43CitrvV4gZUI=', 'Content-Length': '0'} 2024-04-12T03:56:59.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab673561dc295155-006618b10b-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.137 DEBUG:boto:path=/dkyges-59/g 2024-04-12T03:56:59.138 DEBUG:boto:auth_path=/dkyges-59/g 2024-04-12T03:56:59.138 DEBUG:boto:path=/dkyges-59/g?versionId=null 2024-04-12T03:56:59.138 DEBUG:boto:auth_path=/dkyges-59/g?versionId=null 2024-04-12T03:56:59.138 DEBUG:boto:Method: HEAD 2024-04-12T03:56:59.138 DEBUG:boto:Path: /dkyges-59/g?versionId=null 2024-04-12T03:56:59.138 DEBUG:boto:Data: 2024-04-12T03:56:59.138 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.138 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.138 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.138 DEBUG:boto:Params: {} 2024-04-12T03:56:59.139 DEBUG:boto:Token: None 2024-04-12T03:56:59.139 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/g?versionId=null 2024-04-12T03:56:59.139 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e2EI+t+zUFNCkv43CitrvV4gZUI= 2024-04-12T03:56:59.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e2EI+t+zUFNCkv43CitrvV4gZUI=', 'Content-Length': '0'} 2024-04-12T03:56:59.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdd3002a78172f65-006618b10b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:56:59.141 DEBUG:boto:path=/dkyges-59/g 2024-04-12T03:56:59.141 DEBUG:boto:auth_path=/dkyges-59/g 2024-04-12T03:56:59.142 DEBUG:boto:path=/dkyges-59/g?versionId=null 2024-04-12T03:56:59.142 DEBUG:boto:auth_path=/dkyges-59/g?versionId=null 2024-04-12T03:56:59.142 DEBUG:boto:Method: GET 2024-04-12T03:56:59.142 DEBUG:boto:Path: /dkyges-59/g?versionId=null 2024-04-12T03:56:59.142 DEBUG:boto:Data: 2024-04-12T03:56:59.142 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.142 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.142 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.142 DEBUG:boto:Params: {} 2024-04-12T03:56:59.142 DEBUG:boto:Token: None 2024-04-12T03:56:59.142 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/g?versionId=null 2024-04-12T03:56:59.143 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e7G3aLmuRYQZ2ZsEt1i3lImaKCQ= 2024-04-12T03:56:59.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e7G3aLmuRYQZ2ZsEt1i3lImaKCQ=', 'Content-Length': '0'} 2024-04-12T03:56:59.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e30d97d616a420e4-006618b10b-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.145 DEBUG:boto:path=/dkyges-59/g 2024-04-12T03:56:59.145 DEBUG:boto:auth_path=/dkyges-59/g 2024-04-12T03:56:59.145 DEBUG:boto:path=/dkyges-59/g?versionId=null 2024-04-12T03:56:59.145 DEBUG:boto:auth_path=/dkyges-59/g?versionId=null 2024-04-12T03:56:59.145 DEBUG:boto:Method: GET 2024-04-12T03:56:59.145 DEBUG:boto:Path: /dkyges-59/g?versionId=null 2024-04-12T03:56:59.146 DEBUG:boto:Data: 2024-04-12T03:56:59.146 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.146 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.146 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.146 DEBUG:boto:Params: {} 2024-04-12T03:56:59.146 DEBUG:boto:Token: None 2024-04-12T03:56:59.146 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/g?versionId=null 2024-04-12T03:56:59.146 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e7G3aLmuRYQZ2ZsEt1i3lImaKCQ= 2024-04-12T03:56:59.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e7G3aLmuRYQZ2ZsEt1i3lImaKCQ=', 'Content-Length': '0'} 2024-04-12T03:56:59.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bd039dcb7ce4611-006618b10b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.149 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:59.149 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:59.149 DEBUG:boto:path=/dkyges-59/?prefix=g 2024-04-12T03:56:59.149 DEBUG:boto:auth_path=/dkyges-59/?prefix=g 2024-04-12T03:56:59.149 DEBUG:boto:Method: GET 2024-04-12T03:56:59.149 DEBUG:boto:Path: /dkyges-59/?prefix=g 2024-04-12T03:56:59.149 DEBUG:boto:Data: 2024-04-12T03:56:59.149 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.150 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.150 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.150 DEBUG:boto:Params: {} 2024-04-12T03:56:59.150 DEBUG:boto:Token: None 2024-04-12T03:56:59.150 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/ 2024-04-12T03:56:59.150 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ= 2024-04-12T03:56:59.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ=', 'Content-Length': '0'} 2024-04-12T03:56:59.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b94365a20a80120c-006618b10b-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.153 DEBUG:boto:b'dkyges-59g1000falseg2024-04-12T03:55:16.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:59.153 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:59.153 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:59.153 DEBUG:boto:path=/dkyges-59/?prefix=g 2024-04-12T03:56:59.153 DEBUG:boto:auth_path=/dkyges-59/?prefix=g 2024-04-12T03:56:59.153 DEBUG:boto:Method: GET 2024-04-12T03:56:59.154 DEBUG:boto:Path: /dkyges-59/?prefix=g 2024-04-12T03:56:59.154 DEBUG:boto:Data: 2024-04-12T03:56:59.154 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.154 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.154 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.154 DEBUG:boto:Params: {} 2024-04-12T03:56:59.154 DEBUG:boto:Token: None 2024-04-12T03:56:59.154 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/ 2024-04-12T03:56:59.154 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ= 2024-04-12T03:56:59.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ=', 'Content-Length': '0'} 2024-04-12T03:56:59.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c030f6f29c88cf39-006618b10b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.157 DEBUG:boto:b'dkyges-59g1000falseg2024-04-12T03:55:16.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:59.157 DEBUG:boto:path=/dkyges-59/g 2024-04-12T03:56:59.157 DEBUG:boto:auth_path=/dkyges-59/g 2024-04-12T03:56:59.157 DEBUG:boto:Method: HEAD 2024-04-12T03:56:59.157 DEBUG:boto:Path: /dkyges-59/g 2024-04-12T03:56:59.157 DEBUG:boto:Data: 2024-04-12T03:56:59.157 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.157 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.158 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.158 DEBUG:boto:Params: {} 2024-04-12T03:56:59.158 DEBUG:boto:Token: None 2024-04-12T03:56:59.158 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/g 2024-04-12T03:56:59.158 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FrSRjgWb2HuN3kdDqWrMLYdNh2M= 2024-04-12T03:56:59.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FrSRjgWb2HuN3kdDqWrMLYdNh2M=', 'Content-Length': '0'} 2024-04-12T03:56:59.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009058e196352356f6-006618b10b-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.160 DEBUG:boto:path=/dkyges-59/g 2024-04-12T03:56:59.160 DEBUG:boto:auth_path=/dkyges-59/g 2024-04-12T03:56:59.160 DEBUG:boto:Method: HEAD 2024-04-12T03:56:59.160 DEBUG:boto:Path: /dkyges-59/g 2024-04-12T03:56:59.160 DEBUG:boto:Data: 2024-04-12T03:56:59.160 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.161 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.161 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.161 DEBUG:boto:Params: {} 2024-04-12T03:56:59.161 DEBUG:boto:Token: None 2024-04-12T03:56:59.161 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/g 2024-04-12T03:56:59.161 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FrSRjgWb2HuN3kdDqWrMLYdNh2M= 2024-04-12T03:56:59.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FrSRjgWb2HuN3kdDqWrMLYdNh2M=', 'Content-Length': '0'} 2024-04-12T03:56:59.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c9c9e7c0a60fb7e-006618b10b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2024-04-12T03:56:59.164 DEBUG:boto:path=/dkyges-59/g 2024-04-12T03:56:59.164 DEBUG:boto:auth_path=/dkyges-59/g 2024-04-12T03:56:59.164 DEBUG:boto:Method: GET 2024-04-12T03:56:59.164 DEBUG:boto:Path: /dkyges-59/g 2024-04-12T03:56:59.164 DEBUG:boto:Data: 2024-04-12T03:56:59.164 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.164 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.164 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.164 DEBUG:boto:Params: {} 2024-04-12T03:56:59.164 DEBUG:boto:Token: None 2024-04-12T03:56:59.165 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/g 2024-04-12T03:56:59.165 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:N0ataDtzoLgBHBOhijhLbMdnrjo= 2024-04-12T03:56:59.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:N0ataDtzoLgBHBOhijhLbMdnrjo=', 'Content-Length': '0'} 2024-04-12T03:56:59.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1587d7b6bcce93d-006618b10b-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.167 DEBUG:boto:path=/dkyges-59/g 2024-04-12T03:56:59.167 DEBUG:boto:auth_path=/dkyges-59/g 2024-04-12T03:56:59.167 DEBUG:boto:Method: GET 2024-04-12T03:56:59.167 DEBUG:boto:Path: /dkyges-59/g 2024-04-12T03:56:59.167 DEBUG:boto:Data: 2024-04-12T03:56:59.167 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.167 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.167 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.167 DEBUG:boto:Params: {} 2024-04-12T03:56:59.167 DEBUG:boto:Token: None 2024-04-12T03:56:59.168 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/g 2024-04-12T03:56:59.168 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:N0ataDtzoLgBHBOhijhLbMdnrjo= 2024-04-12T03:56:59.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:N0ataDtzoLgBHBOhijhLbMdnrjo=', 'Content-Length': '0'} 2024-04-12T03:56:59.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000506c5f9ab4410b3f-006618b10b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.210 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:59.210 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:59.211 DEBUG:boto:path=/dkyges-59/?prefix=g 2024-04-12T03:56:59.211 DEBUG:boto:auth_path=/dkyges-59/?prefix=g 2024-04-12T03:56:59.211 DEBUG:boto:Method: GET 2024-04-12T03:56:59.211 DEBUG:boto:Path: /dkyges-59/?prefix=g 2024-04-12T03:56:59.211 DEBUG:boto:Data: 2024-04-12T03:56:59.211 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.211 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.211 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.212 DEBUG:boto:Params: {} 2024-04-12T03:56:59.212 DEBUG:boto:Token: None 2024-04-12T03:56:59.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/ 2024-04-12T03:56:59.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ= 2024-04-12T03:56:59.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ=', 'Content-Length': '0'} 2024-04-12T03:56:59.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a373d3f784fe174-006618b10b-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.215 DEBUG:boto:b'dkyges-59g1000falseg2024-04-12T03:55:16.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:59.216 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:59.216 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:59.216 DEBUG:boto:path=/dkyges-59/?prefix=g 2024-04-12T03:56:59.216 DEBUG:boto:auth_path=/dkyges-59/?prefix=g 2024-04-12T03:56:59.216 DEBUG:boto:Method: GET 2024-04-12T03:56:59.216 DEBUG:boto:Path: /dkyges-59/?prefix=g 2024-04-12T03:56:59.216 DEBUG:boto:Data: 2024-04-12T03:56:59.217 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.217 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.217 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.217 DEBUG:boto:Params: {} 2024-04-12T03:56:59.217 DEBUG:boto:Token: None 2024-04-12T03:56:59.217 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/ 2024-04-12T03:56:59.217 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ= 2024-04-12T03:56:59.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ=', 'Content-Length': '0'} 2024-04-12T03:56:59.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e51a6d8ebfb753d3-006618b10b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.220 DEBUG:boto:b'dkyges-59g1000falseg2024-04-12T03:55:16.972Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:59.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:56:59.220 DEBUG:boto:path=/dkyges-59/h 2024-04-12T03:56:59.220 DEBUG:boto:auth_path=/dkyges-59/h 2024-04-12T03:56:59.221 DEBUG:boto:path=/dkyges-59/h?versionId=null 2024-04-12T03:56:59.221 DEBUG:boto:auth_path=/dkyges-59/h?versionId=null 2024-04-12T03:56:59.221 DEBUG:boto:Method: GET 2024-04-12T03:56:59.221 DEBUG:boto:Path: /dkyges-59/h?versionId=null 2024-04-12T03:56:59.221 DEBUG:boto:Data: 2024-04-12T03:56:59.221 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.221 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.221 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.221 DEBUG:boto:Params: {} 2024-04-12T03:56:59.221 DEBUG:boto:Token: None 2024-04-12T03:56:59.221 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/h?versionId=null 2024-04-12T03:56:59.222 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XVL+0x/IzTMSVNXXuVLK9Igashg= 2024-04-12T03:56:59.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XVL+0x/IzTMSVNXXuVLK9Igashg=', 'Content-Length': '0'} 2024-04-12T03:56:59.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e27db9b1ce87b38-006618b10b-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.224 DEBUG:boto:path=/dkyges-59/h 2024-04-12T03:56:59.224 DEBUG:boto:auth_path=/dkyges-59/h 2024-04-12T03:56:59.224 DEBUG:boto:path=/dkyges-59/h?versionId=null 2024-04-12T03:56:59.225 DEBUG:boto:auth_path=/dkyges-59/h?versionId=null 2024-04-12T03:56:59.225 DEBUG:boto:Method: GET 2024-04-12T03:56:59.225 DEBUG:boto:Path: /dkyges-59/h?versionId=null 2024-04-12T03:56:59.225 DEBUG:boto:Data: 2024-04-12T03:56:59.225 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.225 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.225 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.225 DEBUG:boto:Params: {} 2024-04-12T03:56:59.225 DEBUG:boto:Token: None 2024-04-12T03:56:59.226 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/h?versionId=null 2024-04-12T03:56:59.226 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XVL+0x/IzTMSVNXXuVLK9Igashg= 2024-04-12T03:56:59.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XVL+0x/IzTMSVNXXuVLK9Igashg=', 'Content-Length': '0'} 2024-04-12T03:56:59.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b37540d0e013fe9-006618b10b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.228 DEBUG:boto:path=/dkyges-59/h 2024-04-12T03:56:59.228 DEBUG:boto:auth_path=/dkyges-59/h 2024-04-12T03:56:59.228 DEBUG:boto:path=/dkyges-59/h?versionId=null 2024-04-12T03:56:59.228 DEBUG:boto:auth_path=/dkyges-59/h?versionId=null 2024-04-12T03:56:59.229 DEBUG:boto:Method: HEAD 2024-04-12T03:56:59.229 DEBUG:boto:Path: /dkyges-59/h?versionId=null 2024-04-12T03:56:59.229 DEBUG:boto:Data: 2024-04-12T03:56:59.229 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.229 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.229 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.229 DEBUG:boto:Params: {} 2024-04-12T03:56:59.229 DEBUG:boto:Token: None 2024-04-12T03:56:59.229 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/h?versionId=null 2024-04-12T03:56:59.229 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AOliOFUvTPDLggq2AVZNPSzEo8I= 2024-04-12T03:56:59.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AOliOFUvTPDLggq2AVZNPSzEo8I=', 'Content-Length': '0'} 2024-04-12T03:56:59.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e36d52eeaf0dbfdb-006618b10b-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.232 DEBUG:boto:path=/dkyges-59/h 2024-04-12T03:56:59.232 DEBUG:boto:auth_path=/dkyges-59/h 2024-04-12T03:56:59.232 DEBUG:boto:path=/dkyges-59/h?versionId=null 2024-04-12T03:56:59.232 DEBUG:boto:auth_path=/dkyges-59/h?versionId=null 2024-04-12T03:56:59.232 DEBUG:boto:Method: HEAD 2024-04-12T03:56:59.232 DEBUG:boto:Path: /dkyges-59/h?versionId=null 2024-04-12T03:56:59.232 DEBUG:boto:Data: 2024-04-12T03:56:59.232 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.232 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.232 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.232 DEBUG:boto:Params: {} 2024-04-12T03:56:59.233 DEBUG:boto:Token: None 2024-04-12T03:56:59.233 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/h?versionId=null 2024-04-12T03:56:59.233 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AOliOFUvTPDLggq2AVZNPSzEo8I= 2024-04-12T03:56:59.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AOliOFUvTPDLggq2AVZNPSzEo8I=', 'Content-Length': '0'} 2024-04-12T03:56:59.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d50301a21cb7c9cf-006618b10b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:56:59.235 DEBUG:boto:path=/dkyges-59/h 2024-04-12T03:56:59.235 DEBUG:boto:auth_path=/dkyges-59/h 2024-04-12T03:56:59.235 DEBUG:boto:path=/dkyges-59/h?versionId=null 2024-04-12T03:56:59.235 DEBUG:boto:auth_path=/dkyges-59/h?versionId=null 2024-04-12T03:56:59.235 DEBUG:boto:Method: GET 2024-04-12T03:56:59.236 DEBUG:boto:Path: /dkyges-59/h?versionId=null 2024-04-12T03:56:59.236 DEBUG:boto:Data: 2024-04-12T03:56:59.236 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.236 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.236 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.236 DEBUG:boto:Params: {} 2024-04-12T03:56:59.236 DEBUG:boto:Token: None 2024-04-12T03:56:59.236 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/h?versionId=null 2024-04-12T03:56:59.236 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XVL+0x/IzTMSVNXXuVLK9Igashg= 2024-04-12T03:56:59.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XVL+0x/IzTMSVNXXuVLK9Igashg=', 'Content-Length': '0'} 2024-04-12T03:56:59.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000975fa3ceb27aada6-006618b10b-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.278 DEBUG:boto:path=/dkyges-59/h 2024-04-12T03:56:59.278 DEBUG:boto:auth_path=/dkyges-59/h 2024-04-12T03:56:59.278 DEBUG:boto:path=/dkyges-59/h?versionId=null 2024-04-12T03:56:59.279 DEBUG:boto:auth_path=/dkyges-59/h?versionId=null 2024-04-12T03:56:59.279 DEBUG:boto:Method: GET 2024-04-12T03:56:59.279 DEBUG:boto:Path: /dkyges-59/h?versionId=null 2024-04-12T03:56:59.279 DEBUG:boto:Data: 2024-04-12T03:56:59.279 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.279 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.279 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.279 DEBUG:boto:Params: {} 2024-04-12T03:56:59.279 DEBUG:boto:Token: None 2024-04-12T03:56:59.279 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/h?versionId=null 2024-04-12T03:56:59.280 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XVL+0x/IzTMSVNXXuVLK9Igashg= 2024-04-12T03:56:59.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XVL+0x/IzTMSVNXXuVLK9Igashg=', 'Content-Length': '0'} 2024-04-12T03:56:59.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:17 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006427d7ddd6fc6d80-006618b10b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.282 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:59.282 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:59.282 DEBUG:boto:path=/dkyges-59/?prefix=h 2024-04-12T03:56:59.282 DEBUG:boto:auth_path=/dkyges-59/?prefix=h 2024-04-12T03:56:59.282 DEBUG:boto:Method: GET 2024-04-12T03:56:59.282 DEBUG:boto:Path: /dkyges-59/?prefix=h 2024-04-12T03:56:59.283 DEBUG:boto:Data: 2024-04-12T03:56:59.283 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.283 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.283 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.283 DEBUG:boto:Params: {} 2024-04-12T03:56:59.283 DEBUG:boto:Token: None 2024-04-12T03:56:59.283 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/ 2024-04-12T03:56:59.283 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ= 2024-04-12T03:56:59.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ=', 'Content-Length': '0'} 2024-04-12T03:56:59.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a18f5fd3117d5ee6-006618b10b-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.286 DEBUG:boto:b'dkyges-59h1000falseh2024-04-12T03:55:17.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:59.287 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:59.287 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:59.287 DEBUG:boto:path=/dkyges-59/?prefix=h 2024-04-12T03:56:59.287 DEBUG:boto:auth_path=/dkyges-59/?prefix=h 2024-04-12T03:56:59.287 DEBUG:boto:Method: GET 2024-04-12T03:56:59.287 DEBUG:boto:Path: /dkyges-59/?prefix=h 2024-04-12T03:56:59.287 DEBUG:boto:Data: 2024-04-12T03:56:59.287 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.288 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.288 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.288 DEBUG:boto:Params: {} 2024-04-12T03:56:59.288 DEBUG:boto:Token: None 2024-04-12T03:56:59.288 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/ 2024-04-12T03:56:59.288 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ= 2024-04-12T03:56:59.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ=', 'Content-Length': '0'} 2024-04-12T03:56:59.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c574dff5e28b643-006618b10b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.290 DEBUG:boto:b'dkyges-59h1000falseh2024-04-12T03:55:17.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:59.291 DEBUG:boto:path=/dkyges-59/h 2024-04-12T03:56:59.291 DEBUG:boto:auth_path=/dkyges-59/h 2024-04-12T03:56:59.291 DEBUG:boto:Method: HEAD 2024-04-12T03:56:59.291 DEBUG:boto:Path: /dkyges-59/h 2024-04-12T03:56:59.291 DEBUG:boto:Data: 2024-04-12T03:56:59.291 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.291 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.291 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.291 DEBUG:boto:Params: {} 2024-04-12T03:56:59.292 DEBUG:boto:Token: None 2024-04-12T03:56:59.292 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/h 2024-04-12T03:56:59.292 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+4RkxJC70IlK4yNojtemZBgO7H4= 2024-04-12T03:56:59.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+4RkxJC70IlK4yNojtemZBgO7H4=', 'Content-Length': '0'} 2024-04-12T03:56:59.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d763720b261e473f-006618b10b-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.294 DEBUG:boto:path=/dkyges-59/h 2024-04-12T03:56:59.294 DEBUG:boto:auth_path=/dkyges-59/h 2024-04-12T03:56:59.294 DEBUG:boto:Method: HEAD 2024-04-12T03:56:59.294 DEBUG:boto:Path: /dkyges-59/h 2024-04-12T03:56:59.295 DEBUG:boto:Data: 2024-04-12T03:56:59.295 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.295 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.295 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.295 DEBUG:boto:Params: {} 2024-04-12T03:56:59.295 DEBUG:boto:Token: None 2024-04-12T03:56:59.295 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/h 2024-04-12T03:56:59.295 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+4RkxJC70IlK4yNojtemZBgO7H4= 2024-04-12T03:56:59.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+4RkxJC70IlK4yNojtemZBgO7H4=', 'Content-Length': '0'} 2024-04-12T03:56:59.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073b8a4616f7f27a0-006618b10b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2024-04-12T03:56:59.298 DEBUG:boto:path=/dkyges-59/h 2024-04-12T03:56:59.298 DEBUG:boto:auth_path=/dkyges-59/h 2024-04-12T03:56:59.298 DEBUG:boto:Method: GET 2024-04-12T03:56:59.298 DEBUG:boto:Path: /dkyges-59/h 2024-04-12T03:56:59.298 DEBUG:boto:Data: 2024-04-12T03:56:59.298 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.298 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.298 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.298 DEBUG:boto:Params: {} 2024-04-12T03:56:59.298 DEBUG:boto:Token: None 2024-04-12T03:56:59.299 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/h 2024-04-12T03:56:59.299 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:taV3mkCFzJLZ2dM0IAPQIQ19Wp0= 2024-04-12T03:56:59.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:taV3mkCFzJLZ2dM0IAPQIQ19Wp0=', 'Content-Length': '0'} 2024-04-12T03:56:59.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:17 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-59:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b7fe2a743f346b7-006618b10b-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.301 DEBUG:boto:path=/dkyges-59/h 2024-04-12T03:56:59.301 DEBUG:boto:auth_path=/dkyges-59/h 2024-04-12T03:56:59.301 DEBUG:boto:Method: GET 2024-04-12T03:56:59.301 DEBUG:boto:Path: /dkyges-59/h 2024-04-12T03:56:59.301 DEBUG:boto:Data: 2024-04-12T03:56:59.301 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.301 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.301 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.301 DEBUG:boto:Params: {} 2024-04-12T03:56:59.301 DEBUG:boto:Token: None 2024-04-12T03:56:59.302 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/h 2024-04-12T03:56:59.302 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:taV3mkCFzJLZ2dM0IAPQIQ19Wp0= 2024-04-12T03:56:59.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:taV3mkCFzJLZ2dM0IAPQIQ19Wp0=', 'Content-Length': '0'} 2024-04-12T03:56:59.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:55:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da143792624b7807-006618b10b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.346 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:59.346 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:59.346 DEBUG:boto:path=/dkyges-59/?prefix=h 2024-04-12T03:56:59.347 DEBUG:boto:auth_path=/dkyges-59/?prefix=h 2024-04-12T03:56:59.347 DEBUG:boto:Method: GET 2024-04-12T03:56:59.347 DEBUG:boto:Path: /dkyges-59/?prefix=h 2024-04-12T03:56:59.347 DEBUG:boto:Data: 2024-04-12T03:56:59.347 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.347 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:56:59.347 DEBUG:boto:Port: 8001 2024-04-12T03:56:59.347 DEBUG:boto:Params: {} 2024-04-12T03:56:59.347 DEBUG:boto:Token: None 2024-04-12T03:56:59.347 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/ 2024-04-12T03:56:59.348 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ= 2024-04-12T03:56:59.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ=', 'Content-Length': '0'} 2024-04-12T03:56:59.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028e8a14fddcb3e62-006618b10b-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.350 DEBUG:boto:b'dkyges-59h1000falseh2024-04-12T03:55:17.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:59.351 DEBUG:boto:path=/dkyges-59/ 2024-04-12T03:56:59.351 DEBUG:boto:auth_path=/dkyges-59/ 2024-04-12T03:56:59.351 DEBUG:boto:path=/dkyges-59/?prefix=h 2024-04-12T03:56:59.351 DEBUG:boto:auth_path=/dkyges-59/?prefix=h 2024-04-12T03:56:59.352 DEBUG:boto:Method: GET 2024-04-12T03:56:59.352 DEBUG:boto:Path: /dkyges-59/?prefix=h 2024-04-12T03:56:59.352 DEBUG:boto:Data: 2024-04-12T03:56:59.352 DEBUG:boto:Headers: {} 2024-04-12T03:56:59.352 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:56:59.352 DEBUG:boto:Port: 8000 2024-04-12T03:56:59.352 DEBUG:boto:Params: {} 2024-04-12T03:56:59.352 DEBUG:boto:Token: None 2024-04-12T03:56:59.352 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:56:59 GMT /dkyges-59/ 2024-04-12T03:56:59.352 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ= 2024-04-12T03:56:59.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:56:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pvr26jwnR1TyrMWdoP2MnSgf2zQ=', 'Content-Length': '0'} 2024-04-12T03:56:59.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000139fc6f3eefeb44f-006618b10b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:56:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:56:59.355 DEBUG:boto:b'dkyges-59h1000falseh2024-04-12T03:55:17.036Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:56:59.355 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-59 zones={a2, a1} 2024-04-12T03:56:59.356 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ok 2024-04-12T03:56:59.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:56:59.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:56:59.357 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:56:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:56:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:56:59.457 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:56:59.472 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:56:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:56:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:56:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:56:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:56:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:56:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:56:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:56:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:56:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:56:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:56:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:56:59.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:56:59.474 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:56:59.474 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894116.846810_541.1", 2024-04-12T03:56:59.474 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:55:16.846810Z" 2024-04-12T03:56:59.474 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:56:59.474 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:56:59.475 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894116.846810_541.1'} 2024-04-12T03:56:59.475 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:56:59.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:56:59.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:56:59.476 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:56:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:56:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:56:59.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:59.576+0000 7f7642755780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:56:59.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:56:59.577+0000 7f7642755780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:56:59.581 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:56:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:56:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:56:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:56:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:56:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:56:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:56:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:56:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:56:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:56:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:56:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:56:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:56:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:56:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:56:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:56:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:56:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:56:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:56:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:56:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:56:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:56:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:56:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:56:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:56:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:56:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:56:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:56:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:56:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:56:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:56:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:56:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:56:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:56:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:56:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:56:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:56:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:56:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:56:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:56:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:56:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:56:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:56:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:56:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:56:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:56:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:56:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:56:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:56:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894110.308772_537.1", 2024-04-12T03:56:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:56:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:56:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:56:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:55:10.308772Z", 2024-04-12T03:56:59.586 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:56:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:56:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:56:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:56:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:56:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:56:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:56:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:56:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:56:59.587 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:56:59Z" 2024-04-12T03:56:59.587 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:56:59.588 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894110.308772_537.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:55:10.308772Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:56:59Z" } 2024-04-12T03:56:59.588 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894110.308772_537.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:55:10.308772Z', 'realm_epoch': 2}}] 2024-04-12T03:56:59.589 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894116.846810_541.1'} 2024-04-12T03:56:59.589 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894110.308772_537.1'} 2024-04-12T03:56:59.589 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894116.846810_541.1 target=1_1712894110.308772_537.1 2024-04-12T03:57:04.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:57:04.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:57:04.591 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:57:04.651 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:57:04.652 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:57:04.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:04.701+0000 7fe57bb16780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:57:04.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:04.702+0000 7fe57bb16780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:57:04.707 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:57:04.707 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:57:04.707 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:57:04.707 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:57:04.707 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:57:04.707 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:57:04.707 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:04.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:04.707 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:57:04.707 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:04.708 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:57:04.708 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:04.708 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:04.708 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:57:04.708 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:04.708 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:57:04.708 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:04.708 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:57:04.708 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:04.708 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:04.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:04.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:04.709 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:57:04.709 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:04.709 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:04.709 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:57:04.709 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:04.709 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:04.709 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:04.709 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:57:04.709 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:04.709 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:04.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:04.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:04.710 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:57:04.710 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:04.710 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:04.710 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:57:04.710 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:04.710 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:04.710 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:04.710 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:57:04.710 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:04.711 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:04.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:04.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:04.711 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:57:04.711 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:04.711 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:04.711 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894110.308772_537.1", 2024-04-12T03:57:04.711 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:04.711 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:04.711 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:04.711 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:55:10.308772Z", 2024-04-12T03:57:04.711 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:04.712 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:04.712 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:04.712 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:57:04.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:04.712 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:57:04.712 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:57:04.712 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:57:04.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:04.712 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:57:04Z" 2024-04-12T03:57:04.712 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:57:04.713 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894110.308772_537.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:55:10.308772Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:57:04Z" } 2024-04-12T03:57:04.713 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894110.308772_537.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:55:10.308772Z', 'realm_epoch': 2}}] 2024-04-12T03:57:04.714 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894116.846810_541.1'} 2024-04-12T03:57:04.714 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894110.308772_537.1'} 2024-04-12T03:57:04.714 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894116.846810_541.1 target=1_1712894110.308772_537.1 2024-04-12T03:57:09.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:57:09.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:57:09.715 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:57:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:57:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:57:09.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:09.822+0000 7fb1e3a24780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:57:09.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:09.823+0000 7fb1e3a24780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:57:09.827 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:57:09.827 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:57:09.828 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:57:09.828 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:57:09.828 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:57:09.828 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:57:09.828 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:09.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:09.828 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:57:09.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:09.828 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:57:09.829 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:09.829 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:09.829 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:57:09.829 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:09.829 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:57:09.829 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:09.829 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:57:09.829 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:09.829 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:09.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:09.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:09.830 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:57:09.830 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:09.830 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:09.830 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:57:09.830 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:09.830 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:09.830 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:09.830 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:57:09.830 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:09.830 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:09.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:09.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:09.831 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:57:09.831 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:09.831 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:09.831 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:57:09.831 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:09.831 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:09.831 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:09.831 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:57:09.831 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:09.832 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:09.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:09.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:09.832 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:57:09.832 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:09.832 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:09.832 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894110.308772_537.1", 2024-04-12T03:57:09.832 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:09.832 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:09.832 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:09.832 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:55:10.308772Z", 2024-04-12T03:57:09.832 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:09.833 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:09.833 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:09.833 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:57:09.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:09.833 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:57:09.833 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:57:09.833 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:57:09.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:09.833 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:57:09Z" 2024-04-12T03:57:09.833 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:57:09.834 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894110.308772_537.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:55:10.308772Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:57:09Z" } 2024-04-12T03:57:09.834 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894110.308772_537.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:55:10.308772Z', 'realm_epoch': 2}}] 2024-04-12T03:57:09.834 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894116.846810_541.1'} 2024-04-12T03:57:09.835 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894110.308772_537.1'} 2024-04-12T03:57:09.835 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894116.846810_541.1 target=1_1712894110.308772_537.1 2024-04-12T03:57:14.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:57:14.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:57:14.836 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:57:14.894 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:57:14.895 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:57:14.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:14.944+0000 7f3df042c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:57:14.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:14.945+0000 7f3df042c780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:57:14.949 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:57:14.949 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:57:14.949 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:57:14.949 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:57:14.949 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:57:14.949 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:57:14.950 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:14.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:14.950 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:57:14.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:14.950 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:57:14.950 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:14.950 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:14.950 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:57:14.950 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:14.950 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:57:14.950 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:14.950 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:57:14.951 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:14.951 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:14.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:14.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:14.951 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:57:14.951 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:14.951 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:14.951 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:57:14.951 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:14.951 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:14.951 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:14.951 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:57:14.952 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:14.952 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:14.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:14.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:14.952 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:57:14.952 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:14.952 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:14.952 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:57:14.952 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:14.952 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:14.952 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:14.953 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:57:14.953 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:14.953 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:14.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:14.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:14.953 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:57:14.953 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:14.953 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:14.953 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894110.308772_537.1", 2024-04-12T03:57:14.953 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:14.953 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:14.953 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:14.954 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:55:10.308772Z", 2024-04-12T03:57:14.954 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:14.954 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:14.954 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:14.954 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:57:14.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:14.954 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:57:14.954 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:57:14.954 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:57:14.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:14.955 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:57:14Z" 2024-04-12T03:57:14.955 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:57:14.955 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894110.308772_537.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:55:10.308772Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:57:14Z" } 2024-04-12T03:57:14.956 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894110.308772_537.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:55:10.308772Z', 'realm_epoch': 2}}] 2024-04-12T03:57:14.956 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894116.846810_541.1'} 2024-04-12T03:57:14.956 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894110.308772_537.1'} 2024-04-12T03:57:14.956 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894116.846810_541.1 target=1_1712894110.308772_537.1 2024-04-12T03:57:19.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:57:19.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:57:19.957 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:57:20.017 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:57:20.017 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:57:20.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:20.065+0000 7ff08b40d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:57:20.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:20.066+0000 7ff08b40d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:57:20.070 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:57:20.070 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:57:20.070 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:57:20.070 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:57:20.070 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:57:20.071 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:57:20.071 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:20.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:20.071 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:57:20.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:20.071 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:57:20.071 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:20.071 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:20.071 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:57:20.071 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:20.071 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:57:20.071 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:20.072 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:57:20.072 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:20.072 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:20.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:20.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:20.072 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:57:20.072 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:20.072 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:20.072 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:57:20.072 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:20.072 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:20.072 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:20.072 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:57:20.073 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:20.073 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:20.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:20.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:20.073 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:57:20.073 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:20.073 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:20.073 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:57:20.073 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:20.073 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:20.073 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:20.074 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:57:20.074 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:20.074 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:20.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:20.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:20.074 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:57:20.074 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:20.074 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:20.074 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894116.846810_541.1", 2024-04-12T03:57:20.075 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:20.075 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:20.075 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:20.075 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:55:16.846810Z", 2024-04-12T03:57:20.075 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:20.075 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:20.075 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:20.075 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:57:20.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:20.075 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:57:20.075 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:57:20.075 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:57:20.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:20.076 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:57:20Z" 2024-04-12T03:57:20.076 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:57:20.077 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894116.846810_541.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:55:16.846810Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:57:20Z" } 2024-04-12T03:57:20.077 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894116.846810_541.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:55:16.846810Z', 'realm_epoch': 2}}] 2024-04-12T03:57:20.077 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894116.846810_541.1'} 2024-04-12T03:57:20.077 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894116.846810_541.1'} 2024-04-12T03:57:20.078 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:57:20.078 DEBUG:boto:path=/ 2024-04-12T03:57:20.078 DEBUG:boto:auth_path=/ 2024-04-12T03:57:20.078 DEBUG:boto:Method: GET 2024-04-12T03:57:20.078 DEBUG:boto:Path: / 2024-04-12T03:57:20.078 DEBUG:boto:Data: 2024-04-12T03:57:20.078 DEBUG:boto:Headers: {} 2024-04-12T03:57:20.078 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:20.078 DEBUG:boto:Port: 8000 2024-04-12T03:57:20.078 DEBUG:boto:Params: {} 2024-04-12T03:57:20.079 DEBUG:boto:Token: None 2024-04-12T03:57:20.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:57:20 GMT / 2024-04-12T03:57:20.079 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LJCCD4LHL1jYpG/kPGocKkJ1YVk= 2024-04-12T03:57:20.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:57:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LJCCD4LHL1jYpG/kPGocKkJ1YVk=', 'Content-Length': '0'} 2024-04-12T03:57:20.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000944efc5ac9f9e6f9-006618b120-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:57:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:20.085 DEBUG:boto:path=/ 2024-04-12T03:57:20.085 DEBUG:boto:auth_path=/ 2024-04-12T03:57:20.085 DEBUG:boto:Method: GET 2024-04-12T03:57:20.085 DEBUG:boto:Path: / 2024-04-12T03:57:20.085 DEBUG:boto:Data: 2024-04-12T03:57:20.085 DEBUG:boto:Headers: {} 2024-04-12T03:57:20.085 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:57:20.085 DEBUG:boto:Port: 8001 2024-04-12T03:57:20.085 DEBUG:boto:Params: {} 2024-04-12T03:57:20.086 DEBUG:boto:Token: None 2024-04-12T03:57:20.086 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:57:20 GMT / 2024-04-12T03:57:20.086 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LJCCD4LHL1jYpG/kPGocKkJ1YVk= 2024-04-12T03:57:20.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:57:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LJCCD4LHL1jYpG/kPGocKkJ1YVk=', 'Content-Length': '0'} 2024-04-12T03:57:20.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f324096e394210fa-006618b120-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:57:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:20.090 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2024-04-12T03:57:20.091 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:57:20.091 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:57:20.091 DEBUG:boto:Method: PUT 2024-04-12T03:57:20.091 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:57:20.091 DEBUG:boto:Data: 2024-04-12T03:57:20.091 DEBUG:boto:Headers: {} 2024-04-12T03:57:20.091 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:20.091 DEBUG:boto:Port: 8000 2024-04-12T03:57:20.091 DEBUG:boto:Params: {} 2024-04-12T03:57:20.091 DEBUG:boto:Token: None 2024-04-12T03:57:20.092 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:57:20 GMT /dkyges-60/ 2024-04-12T03:57:20.092 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XHHy9wtX3KqUYvRvRbcZ9ZPtujw= 2024-04-12T03:57:20.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:57:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XHHy9wtX3KqUYvRvRbcZ9ZPtujw=', 'Content-Length': '0'} 2024-04-12T03:57:20.110 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a50b46427a390d34-006618b120-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:57:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:20.110 DEBUG:rgw_multi.tests:created bucket=dkyges-60 2024-04-12T03:57:20.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:57:20.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:57:20.110 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:57:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:57:20.165 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:57:20.203 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:57:20.217 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:57:20.218 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:20.218 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:57:20.218 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:57:20.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:20.218 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:20.218 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:57:20.218 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:57:20.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:20.218 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:20.219 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:57:20.219 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:57:20.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:20.219 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:20.219 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T03:57:20.219 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:57:20.120233Z" 2024-04-12T03:57:20.219 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:20.219 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:57:20.220 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:57:20.220 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:57:20.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:57:20.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:57:20.221 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:57:20.275 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:57:20.275 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:57:20.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:20.322+0000 7fd0f7619780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:57:20.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:20.323+0000 7fd0f7619780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:57:20.328 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:57:20.328 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:57:20.328 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:57:20.329 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:57:20.329 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:57:20.329 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:57:20.329 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:20.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:57:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:57:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:57:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:57:20.330 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:57:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:57:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:57:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:20.331 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:57:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:57:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:57:20.332 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:20.333 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:20.333 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:20.333 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:57:20.333 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:20.333 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:20.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:20.333 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:20.333 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:57:20.333 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:20.333 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:20.333 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.112952_542.1", 2024-04-12T03:57:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.112952Z", 2024-04-12T03:57:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:57:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:20.334 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:57:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:57:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:57:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:20.335 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:57:20Z" 2024-04-12T03:57:20.335 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:57:20.336 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.112952_542.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.112952Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:57:20Z" } 2024-04-12T03:57:20.336 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.112952_542.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.112952Z', 'realm_epoch': 2}}] 2024-04-12T03:57:20.336 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:57:20.336 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.112952_542.1'} 2024-04-12T03:57:20.336 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894240.120233_545.1 target=1_1712894240.112952_542.1 2024-04-12T03:57:25.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:57:25.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:57:25.338 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:57:25.398 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:57:25.398 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:57:25.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:25.446+0000 7fad8f260780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:57:25.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:25.447+0000 7fad8f260780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:57:25.451 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:57:25.451 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:57:25.451 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:57:25.451 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:57:25.451 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:57:25.451 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:57:25.451 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:25.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:25.452 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:57:25.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:25.452 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:57:25.452 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:25.452 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:25.452 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:57:25.452 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:25.452 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:57:25.452 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:25.452 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:57:25.452 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:25.453 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:25.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:25.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:25.453 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:57:25.453 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:25.453 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:25.453 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:57:25.453 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:25.453 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:25.453 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:25.453 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:57:25.454 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:25.454 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:25.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:25.454 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:25.454 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:57:25.454 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:25.454 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:25.454 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:57:25.454 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:25.454 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:25.454 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:25.455 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:57:25.455 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:25.455 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:25.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:25.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:57:25.455 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:57:25.455 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:57:25.455 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:57:25.455 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T03:57:25.455 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:57:25.455 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:57:25.456 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:57:25.456 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T03:57:25.456 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:57:25.456 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:25.456 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:57:25.456 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:57:25.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:25.456 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:57:25.456 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:57:25.456 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:57:25.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:57:25.456 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:57:25Z" 2024-04-12T03:57:25.457 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:57:25.457 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:57:25Z" } 2024-04-12T03:57:25.458 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T03:57:25.458 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:57:25.458 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:57:25.458 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T03:57:25.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:57:25.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:57:25.458 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:57:25.517 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:57:25.517 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:57:25.565 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:57:25.565 INFO:teuthology.orchestra.run.smithi043.stdout: "objs": [ 2024-04-12T03:57:25.565 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000001", 2024-04-12T03:57:25.565 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000005", 2024-04-12T03:57:25.565 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000012", 2024-04-12T03:57:25.565 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000004", 2024-04-12T03:57:25.565 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000011", 2024-04-12T03:57:25.565 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000000", 2024-04-12T03:57:25.565 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000010", 2024-04-12T03:57:25.565 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000003", 2024-04-12T03:57:25.566 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000013", 2024-04-12T03:57:25.566 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000002", 2024-04-12T03:57:25.566 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000006" 2024-04-12T03:57:25.566 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:57:25.566 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:57:25.567 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:57:25.567 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:57:25.567 DEBUG:boto:Method: HEAD 2024-04-12T03:57:25.567 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:57:25.567 DEBUG:boto:Data: 2024-04-12T03:57:25.567 DEBUG:boto:Headers: {} 2024-04-12T03:57:25.568 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.568 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.568 DEBUG:boto:Params: {} 2024-04-12T03:57:25.568 DEBUG:boto:Token: None 2024-04-12T03:57:25.568 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/ 2024-04-12T03:57:25.568 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4= 2024-04-12T03:57:25.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4=', 'Content-Length': '0'} 2024-04-12T03:57:25.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c0718b61bb29498-006618b125-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.571 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:57:25.571 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:57:25.571 DEBUG:boto:Method: PUT 2024-04-12T03:57:25.571 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010 2024-04-12T03:57:25.571 DEBUG:boto:Data: 2024-04-12T03:57:25.571 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:57:25.572 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.572 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.572 DEBUG:boto:Params: {} 2024-04-12T03:57:25.572 DEBUG:boto:Token: None 2024-04-12T03:57:25.572 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/obj00000000000000000010 2024-04-12T03:57:25.572 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I2qQitafo88UioBXnjenA8luyiQ= 2024-04-12T03:57:25.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I2qQitafo88UioBXnjenA8luyiQ='} 2024-04-12T03:57:25.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d7719459e5c25d7e-006618b125-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.615 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:57:25.615 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:57:25.615 DEBUG:boto:Method: HEAD 2024-04-12T03:57:25.615 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:57:25.615 DEBUG:boto:Data: 2024-04-12T03:57:25.615 DEBUG:boto:Headers: {} 2024-04-12T03:57:25.615 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.615 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.615 DEBUG:boto:Params: {} 2024-04-12T03:57:25.615 DEBUG:boto:Token: None 2024-04-12T03:57:25.616 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/ 2024-04-12T03:57:25.616 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4= 2024-04-12T03:57:25.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4=', 'Content-Length': '0'} 2024-04-12T03:57:25.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7cf0b9594c9a371-006618b125-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.618 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:57:25.618 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:57:25.618 DEBUG:boto:Method: PUT 2024-04-12T03:57:25.619 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013 2024-04-12T03:57:25.619 DEBUG:boto:Data: 2024-04-12T03:57:25.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:57:25.619 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.619 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.619 DEBUG:boto:Params: {} 2024-04-12T03:57:25.619 DEBUG:boto:Token: None 2024-04-12T03:57:25.619 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/obj00000000000000000013 2024-04-12T03:57:25.619 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kgAYhELRHJ4JYEbya0aKI6eWmso= 2024-04-12T03:57:25.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kgAYhELRHJ4JYEbya0aKI6eWmso='} 2024-04-12T03:57:25.625 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005dadf51998b68e34-006618b125-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.625 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:57:25.625 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:57:25.626 DEBUG:boto:Method: HEAD 2024-04-12T03:57:25.626 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:57:25.626 DEBUG:boto:Data: 2024-04-12T03:57:25.626 DEBUG:boto:Headers: {} 2024-04-12T03:57:25.626 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.626 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.626 DEBUG:boto:Params: {} 2024-04-12T03:57:25.626 DEBUG:boto:Token: None 2024-04-12T03:57:25.626 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/ 2024-04-12T03:57:25.626 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4= 2024-04-12T03:57:25.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4=', 'Content-Length': '0'} 2024-04-12T03:57:25.629 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ffae76002ed4360b-006618b125-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.629 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:57:25.629 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:57:25.630 DEBUG:boto:Method: PUT 2024-04-12T03:57:25.630 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012 2024-04-12T03:57:25.630 DEBUG:boto:Data: 2024-04-12T03:57:25.630 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:57:25.630 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.630 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.630 DEBUG:boto:Params: {} 2024-04-12T03:57:25.630 DEBUG:boto:Token: None 2024-04-12T03:57:25.630 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/obj00000000000000000012 2024-04-12T03:57:25.630 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:daxj+FZ5vLzDdT5xPkFHn1y/OmU= 2024-04-12T03:57:25.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:daxj+FZ5vLzDdT5xPkFHn1y/OmU='} 2024-04-12T03:57:25.636 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005d5ce89b87e435cf-006618b125-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.636 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:57:25.636 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:57:25.637 DEBUG:boto:Method: HEAD 2024-04-12T03:57:25.637 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:57:25.637 DEBUG:boto:Data: 2024-04-12T03:57:25.637 DEBUG:boto:Headers: {} 2024-04-12T03:57:25.637 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.637 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.637 DEBUG:boto:Params: {} 2024-04-12T03:57:25.637 DEBUG:boto:Token: None 2024-04-12T03:57:25.637 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/ 2024-04-12T03:57:25.637 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4= 2024-04-12T03:57:25.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4=', 'Content-Length': '0'} 2024-04-12T03:57:25.639 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f70317267bc13422-006618b125-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.640 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:57:25.640 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:57:25.640 DEBUG:boto:Method: PUT 2024-04-12T03:57:25.640 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011 2024-04-12T03:57:25.640 DEBUG:boto:Data: 2024-04-12T03:57:25.640 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:57:25.640 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.640 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.640 DEBUG:boto:Params: {} 2024-04-12T03:57:25.640 DEBUG:boto:Token: None 2024-04-12T03:57:25.641 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/obj00000000000000000011 2024-04-12T03:57:25.641 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VS8tUmZsR4f+sGTE+6bRsVouxnc= 2024-04-12T03:57:25.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VS8tUmZsR4f+sGTE+6bRsVouxnc='} 2024-04-12T03:57:25.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000afd25094f20a4b9b-006618b125-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.683 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:57:25.683 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:57:25.683 DEBUG:boto:Method: HEAD 2024-04-12T03:57:25.683 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:57:25.683 DEBUG:boto:Data: 2024-04-12T03:57:25.683 DEBUG:boto:Headers: {} 2024-04-12T03:57:25.683 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.683 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.683 DEBUG:boto:Params: {} 2024-04-12T03:57:25.683 DEBUG:boto:Token: None 2024-04-12T03:57:25.683 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/ 2024-04-12T03:57:25.684 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4= 2024-04-12T03:57:25.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4=', 'Content-Length': '0'} 2024-04-12T03:57:25.686 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c993203e8e24934-006618b125-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.686 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:57:25.686 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:57:25.687 DEBUG:boto:Method: PUT 2024-04-12T03:57:25.687 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001 2024-04-12T03:57:25.687 DEBUG:boto:Data: 2024-04-12T03:57:25.687 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:57:25.687 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.687 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.687 DEBUG:boto:Params: {} 2024-04-12T03:57:25.687 DEBUG:boto:Token: None 2024-04-12T03:57:25.687 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/obj00000000000000000001 2024-04-12T03:57:25.687 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hF8FZ0M3TQxVSa/QBBj8bw6cBk8= 2024-04-12T03:57:25.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hF8FZ0M3TQxVSa/QBBj8bw6cBk8='} 2024-04-12T03:57:25.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c25fae91260000c7-006618b125-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.694 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:57:25.694 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:57:25.694 DEBUG:boto:Method: HEAD 2024-04-12T03:57:25.694 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:57:25.694 DEBUG:boto:Data: 2024-04-12T03:57:25.694 DEBUG:boto:Headers: {} 2024-04-12T03:57:25.694 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.694 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.694 DEBUG:boto:Params: {} 2024-04-12T03:57:25.695 DEBUG:boto:Token: None 2024-04-12T03:57:25.695 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/ 2024-04-12T03:57:25.695 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4= 2024-04-12T03:57:25.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4=', 'Content-Length': '0'} 2024-04-12T03:57:25.697 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc00e0f5eb624ada-006618b125-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.697 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:57:25.698 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:57:25.698 DEBUG:boto:Method: PUT 2024-04-12T03:57:25.698 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002 2024-04-12T03:57:25.698 DEBUG:boto:Data: 2024-04-12T03:57:25.698 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:57:25.698 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.698 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.698 DEBUG:boto:Params: {} 2024-04-12T03:57:25.698 DEBUG:boto:Token: None 2024-04-12T03:57:25.699 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/obj00000000000000000002 2024-04-12T03:57:25.699 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+Tyb23790Nu1XfrJaNAS6WpJfDo= 2024-04-12T03:57:25.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+Tyb23790Nu1XfrJaNAS6WpJfDo='} 2024-04-12T03:57:25.705 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d95b7be8a4dd7bcb-006618b125-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.705 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:57:25.705 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:57:25.705 DEBUG:boto:Method: HEAD 2024-04-12T03:57:25.705 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:57:25.705 DEBUG:boto:Data: 2024-04-12T03:57:25.705 DEBUG:boto:Headers: {} 2024-04-12T03:57:25.705 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.705 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.705 DEBUG:boto:Params: {} 2024-04-12T03:57:25.706 DEBUG:boto:Token: None 2024-04-12T03:57:25.706 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/ 2024-04-12T03:57:25.706 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4= 2024-04-12T03:57:25.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4=', 'Content-Length': '0'} 2024-04-12T03:57:25.708 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048740502501ac97d-006618b125-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.708 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:57:25.708 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:57:25.709 DEBUG:boto:Method: PUT 2024-04-12T03:57:25.709 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004 2024-04-12T03:57:25.709 DEBUG:boto:Data: 2024-04-12T03:57:25.709 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:57:25.709 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.709 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.709 DEBUG:boto:Params: {} 2024-04-12T03:57:25.709 DEBUG:boto:Token: None 2024-04-12T03:57:25.709 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/obj00000000000000000004 2024-04-12T03:57:25.709 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBFP6svYlGydjCdZrqJpjl1H7+Y= 2024-04-12T03:57:25.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBFP6svYlGydjCdZrqJpjl1H7+Y='} 2024-04-12T03:57:25.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079184fe671ebd095-006618b125-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.755 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:57:25.755 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:57:25.755 DEBUG:boto:Method: HEAD 2024-04-12T03:57:25.755 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:57:25.755 DEBUG:boto:Data: 2024-04-12T03:57:25.755 DEBUG:boto:Headers: {} 2024-04-12T03:57:25.755 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.755 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.755 DEBUG:boto:Params: {} 2024-04-12T03:57:25.755 DEBUG:boto:Token: None 2024-04-12T03:57:25.756 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/ 2024-04-12T03:57:25.756 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4= 2024-04-12T03:57:25.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4=', 'Content-Length': '0'} 2024-04-12T03:57:25.758 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c2637d511a07dde-006618b125-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.758 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:57:25.759 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:57:25.759 DEBUG:boto:Method: PUT 2024-04-12T03:57:25.759 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006 2024-04-12T03:57:25.759 DEBUG:boto:Data: 2024-04-12T03:57:25.759 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:57:25.759 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.759 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.759 DEBUG:boto:Params: {} 2024-04-12T03:57:25.759 DEBUG:boto:Token: None 2024-04-12T03:57:25.759 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/obj00000000000000000006 2024-04-12T03:57:25.760 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zILPbpYA7WLGnTDmvBLg7+ZEihg= 2024-04-12T03:57:25.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zILPbpYA7WLGnTDmvBLg7+ZEihg='} 2024-04-12T03:57:25.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f3d62737120f82c-006618b125-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.766 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:57:25.766 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:57:25.766 DEBUG:boto:Method: HEAD 2024-04-12T03:57:25.766 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:57:25.766 DEBUG:boto:Data: 2024-04-12T03:57:25.767 DEBUG:boto:Headers: {} 2024-04-12T03:57:25.767 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.767 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.767 DEBUG:boto:Params: {} 2024-04-12T03:57:25.767 DEBUG:boto:Token: None 2024-04-12T03:57:25.767 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/ 2024-04-12T03:57:25.767 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4= 2024-04-12T03:57:25.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4=', 'Content-Length': '0'} 2024-04-12T03:57:25.770 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce776cb7cb2af975-006618b125-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.770 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:57:25.770 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:57:25.770 DEBUG:boto:Method: PUT 2024-04-12T03:57:25.770 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000 2024-04-12T03:57:25.771 DEBUG:boto:Data: 2024-04-12T03:57:25.771 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:57:25.771 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.771 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.771 DEBUG:boto:Params: {} 2024-04-12T03:57:25.771 DEBUG:boto:Token: None 2024-04-12T03:57:25.771 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/obj00000000000000000000 2024-04-12T03:57:25.771 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+GWoGW0jV3KBYp4/1OipuylSxRM= 2024-04-12T03:57:25.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+GWoGW0jV3KBYp4/1OipuylSxRM='} 2024-04-12T03:57:25.777 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d92ae0913752cdce-006618b125-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.777 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:57:25.777 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:57:25.777 DEBUG:boto:Method: HEAD 2024-04-12T03:57:25.778 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:57:25.778 DEBUG:boto:Data: 2024-04-12T03:57:25.778 DEBUG:boto:Headers: {} 2024-04-12T03:57:25.778 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.778 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.778 DEBUG:boto:Params: {} 2024-04-12T03:57:25.778 DEBUG:boto:Token: None 2024-04-12T03:57:25.778 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/ 2024-04-12T03:57:25.778 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4= 2024-04-12T03:57:25.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:elouIS6D+NsmD+Cn0egXkldCQF4=', 'Content-Length': '0'} 2024-04-12T03:57:25.781 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a430466caa009677-006618b125-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.781 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:57:25.781 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:57:25.781 DEBUG:boto:Method: PUT 2024-04-12T03:57:25.781 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005 2024-04-12T03:57:25.781 DEBUG:boto:Data: 2024-04-12T03:57:25.781 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:57:25.782 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:57:25.782 DEBUG:boto:Port: 8000 2024-04-12T03:57:25.782 DEBUG:boto:Params: {} 2024-04-12T03:57:25.782 DEBUG:boto:Token: None 2024-04-12T03:57:25.782 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:57:25 GMT /dkyges-60/obj00000000000000000005 2024-04-12T03:57:25.782 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LGS4akMGDmgV9p6ikX0rUuBh/28= 2024-04-12T03:57:25.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:57:25 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LGS4akMGDmgV9p6ikX0rUuBh/28='} 2024-04-12T03:57:25.830 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c615b6c25ed6209-006618b125-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:57:25.831 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-60 2024-04-12T03:57:25.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:57:25.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:57:25.831 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:57:25.897 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:57:25.897 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:57:26.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:26.952+0000 7efd546e7780 1 waiting to reach incremental sync.. 2024-04-12T03:57:31.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:31.952+0000 7efd546e7780 1 waiting to reach incremental sync.. 2024-04-12T03:57:36.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:36.953+0000 7efd546e7780 1 waiting to reach incremental sync.. 2024-04-12T03:57:41.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:41.954+0000 7efd546e7780 1 waiting to reach incremental sync.. 2024-04-12T03:57:46.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:46.955+0000 7efd546e7780 1 waiting to reach incremental sync.. 2024-04-12T03:57:51.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:51.958+0000 7efd546e7780 1 bucket sync caught up with source: 2024-04-12T03:57:51.946 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.134.6, 00000000001.62.6, 00000000001.32.6, 00000000001.54.6, 00000000001.56.6, 00000000001.58.6, 00000000001.54.6, , 00000000001.51.6, 00000000001.63.6, 00000000001.56.6] 2024-04-12T03:57:51.946 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.134.6, 00000000001.62.6, 00000000001.32.6, 00000000001.54.6, 00000000001.56.6, 00000000001.58.6, 00000000001.54.6, , 00000000001.51.6, 00000000001.63.6, 00000000001.56.6] 2024-04-12T03:57:51.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:51.958+0000 7efd546e7780 0 bucket checkpoint complete 2024-04-12T03:57:51.964 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-60 2024-04-12T03:57:51.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:57:51.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:57:51.965 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:57:52.030 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:57:52.030 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:57:52.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:52.085+0000 7f003507c780 1 waiting to reach incremental sync.. 2024-04-12T03:57:57.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:57:57.084+0000 7f003507c780 1 waiting to reach incremental sync.. 2024-04-12T03:58:02.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:02.085+0000 7f003507c780 1 waiting to reach incremental sync.. 2024-04-12T03:58:07.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:07.086+0000 7f003507c780 1 waiting to reach incremental sync.. 2024-04-12T03:58:12.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:12.086+0000 7f003507c780 1 waiting to reach incremental sync.. 2024-04-12T03:58:17.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:17.087+0000 7f003507c780 1 waiting to reach incremental sync.. 2024-04-12T03:58:22.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:22.089+0000 7f003507c780 1 bucket sync caught up with source: 2024-04-12T03:58:22.077 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.2768.6, 00000000001.2433.6, 00000000001.2532.6, 00000000001.1743.6, 00000000001.2444.6, 00000000001.2595.6, 00000000001.2445.6, , 00000000001.1913.6, 00000000001.2927.6, 00000000001.1744.6] 2024-04-12T03:58:22.077 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.2768.6, 00000000001.2433.6, 00000000001.2532.6, 00000000001.1743.6, 00000000001.2444.6, 00000000001.2595.6, 00000000001.2445.6, , 00000000001.1913.6, 00000000001.2927.6, 00000000001.1744.6] 2024-04-12T03:58:22.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:22.089+0000 7f003507c780 0 bucket checkpoint complete 2024-04-12T03:58:22.097 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-60 zones={a2, a1} 2024-04-12T03:58:22.097 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.097 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.097 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.097 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:22.097 DEBUG:boto:Data: 2024-04-12T03:58:22.098 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.098 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.098 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.098 DEBUG:boto:Params: {} 2024-04-12T03:58:22.098 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:58:22.098 DEBUG:boto:Token: None 2024-04-12T03:58:22.099 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.099 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XMJ6ByVm9lpaWLMrLvtAOersE+g= 2024-04-12T03:58:22.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XMJ6ByVm9lpaWLMrLvtAOersE+g=', 'Content-Length': '0'} 2024-04-12T03:58:22.112 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002c5adcb5d1359f67-006618b15e-1c1a-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.112 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.112 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.112 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.113 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:22.113 DEBUG:boto:Data: 2024-04-12T03:58:22.113 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.113 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.113 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.113 DEBUG:boto:Params: {} 2024-04-12T03:58:22.113 DEBUG:boto:Token: None 2024-04-12T03:58:22.113 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.113 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XMJ6ByVm9lpaWLMrLvtAOersE+g= 2024-04-12T03:58:22.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XMJ6ByVm9lpaWLMrLvtAOersE+g=', 'Content-Length': '0'} 2024-04-12T03:58:22.116 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000359602ba40c2f3ac-006618b15e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.116 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:58:22.116 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.116 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.116 DEBUG:boto:path=/dkyges-60/?versions&max-keys=999 2024-04-12T03:58:22.117 DEBUG:boto:auth_path=/dkyges-60/?versions&max-keys=999 2024-04-12T03:58:22.117 DEBUG:boto:Method: GET 2024-04-12T03:58:22.117 DEBUG:boto:Path: /dkyges-60/?versions&max-keys=999 2024-04-12T03:58:22.117 DEBUG:boto:Data: 2024-04-12T03:58:22.117 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.117 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.117 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.117 DEBUG:boto:Params: {} 2024-04-12T03:58:22.117 DEBUG:boto:Token: None 2024-04-12T03:58:22.117 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/?versions 2024-04-12T03:58:22.117 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3vRm3gfv+vwM/AjqF1FF1iTMxJI= 2024-04-12T03:58:22.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3vRm3gfv+vwM/AjqF1FF1iTMxJI=', 'Content-Length': '0'} 2024-04-12T03:58:22.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005086e557ce5baca2-006618b15e-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.120 DEBUG:boto:b'dkyges-60999falseobj00000000000000000000nulltrue2024-04-12T03:57:25.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-12T03:57:25.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-12T03:57:25.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-12T03:57:25.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-12T03:57:25.796Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-12T03:57:25.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-12T03:57:25.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-12T03:57:25.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-12T03:57:25.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-12T03:57:25.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-04-12T03:58:22.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-04-12T03:58:22.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-04-12T03:58:22.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-04-12T03:58:22.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-04-12T03:58:22.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-04-12T03:58:22.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-04-12T03:58:22.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-04-12T03:58:22.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-04-12T03:58:22.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-04-12T03:58:22.122 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:58:22.122 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.122 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.122 DEBUG:boto:path=/dkyges-60/?versions&max-keys=999 2024-04-12T03:58:22.122 DEBUG:boto:auth_path=/dkyges-60/?versions&max-keys=999 2024-04-12T03:58:22.122 DEBUG:boto:Method: GET 2024-04-12T03:58:22.122 DEBUG:boto:Path: /dkyges-60/?versions&max-keys=999 2024-04-12T03:58:22.123 DEBUG:boto:Data: 2024-04-12T03:58:22.123 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.123 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.123 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.123 DEBUG:boto:Params: {} 2024-04-12T03:58:22.123 DEBUG:boto:Token: None 2024-04-12T03:58:22.123 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/?versions 2024-04-12T03:58:22.123 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3vRm3gfv+vwM/AjqF1FF1iTMxJI= 2024-04-12T03:58:22.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3vRm3gfv+vwM/AjqF1FF1iTMxJI=', 'Content-Length': '0'} 2024-04-12T03:58:22.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e9f6a7c6d94ebbc-006618b15e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.126 DEBUG:boto:b'dkyges-60999falseobj00000000000000000000nulltrue2024-04-12T03:57:25.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-12T03:57:25.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-12T03:57:25.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-12T03:57:25.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-12T03:57:25.796Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-12T03:57:25.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-12T03:57:25.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-12T03:57:25.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-12T03:57:25.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-12T03:57:25.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-04-12T03:58:22.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-04-12T03:58:22.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-04-12T03:58:22.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-04-12T03:58:22.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-04-12T03:58:22.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-04-12T03:58:22.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-04-12T03:58:22.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-04-12T03:58:22.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-04-12T03:58:22.128 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-04-12T03:58:22.128 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.128 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.128 DEBUG:boto:path=/dkyges-60/?versions&max-keys=999 2024-04-12T03:58:22.128 DEBUG:boto:auth_path=/dkyges-60/?versions&max-keys=999 2024-04-12T03:58:22.128 DEBUG:boto:Method: GET 2024-04-12T03:58:22.128 DEBUG:boto:Path: /dkyges-60/?versions&max-keys=999 2024-04-12T03:58:22.128 DEBUG:boto:Data: 2024-04-12T03:58:22.129 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.129 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.129 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.129 DEBUG:boto:Params: {} 2024-04-12T03:58:22.129 DEBUG:boto:Token: None 2024-04-12T03:58:22.129 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/?versions 2024-04-12T03:58:22.129 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3vRm3gfv+vwM/AjqF1FF1iTMxJI= 2024-04-12T03:58:22.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3vRm3gfv+vwM/AjqF1FF1iTMxJI=', 'Content-Length': '0'} 2024-04-12T03:58:22.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b557997dbee09f9f-006618b15e-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.132 DEBUG:boto:b'dkyges-60999falseobj00000000000000000000nulltrue2024-04-12T03:57:25.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-12T03:57:25.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-12T03:57:25.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-12T03:57:25.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-12T03:57:25.796Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-12T03:57:25.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-12T03:57:25.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-12T03:57:25.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-12T03:57:25.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-12T03:57:25.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.132 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.133 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.133 DEBUG:boto:path=/dkyges-60/?versions&max-keys=999 2024-04-12T03:58:22.133 DEBUG:boto:auth_path=/dkyges-60/?versions&max-keys=999 2024-04-12T03:58:22.133 DEBUG:boto:Method: GET 2024-04-12T03:58:22.133 DEBUG:boto:Path: /dkyges-60/?versions&max-keys=999 2024-04-12T03:58:22.133 DEBUG:boto:Data: 2024-04-12T03:58:22.133 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.133 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.133 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.133 DEBUG:boto:Params: {} 2024-04-12T03:58:22.134 DEBUG:boto:Token: None 2024-04-12T03:58:22.134 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/?versions 2024-04-12T03:58:22.134 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3vRm3gfv+vwM/AjqF1FF1iTMxJI= 2024-04-12T03:58:22.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3vRm3gfv+vwM/AjqF1FF1iTMxJI=', 'Content-Length': '0'} 2024-04-12T03:58:22.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022909776fb4e5426-006618b15e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.137 DEBUG:boto:b'dkyges-60999falseobj00000000000000000000nulltrue2024-04-12T03:57:25.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-12T03:57:25.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-12T03:57:25.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-12T03:57:25.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-12T03:57:25.796Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-12T03:57:25.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-12T03:57:25.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-12T03:57:25.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-12T03:57:25.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-12T03:57:25.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-12T03:58:22.137 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.138 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.138 DEBUG:boto:path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.138 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.138 DEBUG:boto:Method: GET 2024-04-12T03:58:22.138 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.138 DEBUG:boto:Data: 2024-04-12T03:58:22.138 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.138 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.138 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.139 DEBUG:boto:Params: {} 2024-04-12T03:58:22.139 DEBUG:boto:Token: None 2024-04-12T03:58:22.139 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.139 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7n4FbURFua8pcDWoHLPCa/bQTF4= 2024-04-12T03:58:22.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7n4FbURFua8pcDWoHLPCa/bQTF4=', 'Content-Length': '0'} 2024-04-12T03:58:22.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d55b9c438d74dfb-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.182 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.183 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.183 DEBUG:boto:path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.183 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.183 DEBUG:boto:Method: GET 2024-04-12T03:58:22.183 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.183 DEBUG:boto:Data: 2024-04-12T03:58:22.183 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.183 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.183 DEBUG:boto:Params: {} 2024-04-12T03:58:22.183 DEBUG:boto:Token: None 2024-04-12T03:58:22.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.184 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7n4FbURFua8pcDWoHLPCa/bQTF4= 2024-04-12T03:58:22.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7n4FbURFua8pcDWoHLPCa/bQTF4=', 'Content-Length': '0'} 2024-04-12T03:58:22.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022538a3d2be3d323-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.186 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.186 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.187 DEBUG:boto:path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.187 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.187 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.187 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.187 DEBUG:boto:Data: 2024-04-12T03:58:22.187 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.187 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.187 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.187 DEBUG:boto:Params: {} 2024-04-12T03:58:22.187 DEBUG:boto:Token: None 2024-04-12T03:58:22.188 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.188 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0cGXO00F7t854618tTFMqbwnLpY= 2024-04-12T03:58:22.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0cGXO00F7t854618tTFMqbwnLpY=', 'Content-Length': '0'} 2024-04-12T03:58:22.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf8f6b4fe3625a84-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.190 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.190 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.190 DEBUG:boto:path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.190 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.190 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.190 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.190 DEBUG:boto:Data: 2024-04-12T03:58:22.190 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.190 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.190 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.191 DEBUG:boto:Params: {} 2024-04-12T03:58:22.191 DEBUG:boto:Token: None 2024-04-12T03:58:22.191 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.191 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0cGXO00F7t854618tTFMqbwnLpY= 2024-04-12T03:58:22.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0cGXO00F7t854618tTFMqbwnLpY=', 'Content-Length': '0'} 2024-04-12T03:58:22.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c882c872e2642867-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-12T03:58:22.193 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.193 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.193 DEBUG:boto:path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.193 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.194 DEBUG:boto:Method: GET 2024-04-12T03:58:22.194 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.194 DEBUG:boto:Data: 2024-04-12T03:58:22.194 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.194 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.194 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.194 DEBUG:boto:Params: {} 2024-04-12T03:58:22.194 DEBUG:boto:Token: None 2024-04-12T03:58:22.194 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.194 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7n4FbURFua8pcDWoHLPCa/bQTF4= 2024-04-12T03:58:22.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7n4FbURFua8pcDWoHLPCa/bQTF4=', 'Content-Length': '0'} 2024-04-12T03:58:22.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000768fbfb554a90cd2-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.197 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.197 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.197 DEBUG:boto:path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.197 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.197 DEBUG:boto:Method: GET 2024-04-12T03:58:22.197 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.197 DEBUG:boto:Data: 2024-04-12T03:58:22.197 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.197 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.197 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.197 DEBUG:boto:Params: {} 2024-04-12T03:58:22.198 DEBUG:boto:Token: None 2024-04-12T03:58:22.198 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:22.198 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7n4FbURFua8pcDWoHLPCa/bQTF4= 2024-04-12T03:58:22.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7n4FbURFua8pcDWoHLPCa/bQTF4=', 'Content-Length': '0'} 2024-04-12T03:58:22.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f12a5abe62b38d4-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.200 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.200 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.200 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:22.200 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:22.201 DEBUG:boto:Method: GET 2024-04-12T03:58:22.201 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:22.201 DEBUG:boto:Data: 2024-04-12T03:58:22.201 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.201 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.201 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.201 DEBUG:boto:Params: {} 2024-04-12T03:58:22.201 DEBUG:boto:Token: None 2024-04-12T03:58:22.201 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.201 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac69c1e174cb2db9-006618b15e-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.204 DEBUG:boto:b'dkyges-60obj000000000000000000001000falseobj000000000000000000002024-04-12T03:57:25.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.204 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.205 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.205 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:22.205 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:22.205 DEBUG:boto:Method: GET 2024-04-12T03:58:22.205 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:22.205 DEBUG:boto:Data: 2024-04-12T03:58:22.205 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.205 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.205 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.205 DEBUG:boto:Params: {} 2024-04-12T03:58:22.205 DEBUG:boto:Token: None 2024-04-12T03:58:22.205 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.206 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008db2c31a6f85520c-006618b15e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.209 DEBUG:boto:b'dkyges-60obj000000000000000000001000falseobj000000000000000000002024-04-12T03:57:25.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.209 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.209 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.209 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.209 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.209 DEBUG:boto:Data: 2024-04-12T03:58:22.209 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.210 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.210 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.210 DEBUG:boto:Params: {} 2024-04-12T03:58:22.210 DEBUG:boto:Token: None 2024-04-12T03:58:22.210 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.210 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u6Y2SaSo9n7hh3Gt7vSrDewsoCs= 2024-04-12T03:58:22.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u6Y2SaSo9n7hh3Gt7vSrDewsoCs=', 'Content-Length': '0'} 2024-04-12T03:58:22.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbadc9872e606bf6-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.212 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.212 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.212 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.213 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.213 DEBUG:boto:Data: 2024-04-12T03:58:22.213 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.213 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.213 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.213 DEBUG:boto:Params: {} 2024-04-12T03:58:22.213 DEBUG:boto:Token: None 2024-04-12T03:58:22.213 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.213 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u6Y2SaSo9n7hh3Gt7vSrDewsoCs= 2024-04-12T03:58:22.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u6Y2SaSo9n7hh3Gt7vSrDewsoCs=', 'Content-Length': '0'} 2024-04-12T03:58:22.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009834f77819f4585a-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-12T03:58:22.216 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.216 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.216 DEBUG:boto:Method: GET 2024-04-12T03:58:22.216 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.216 DEBUG:boto:Data: 2024-04-12T03:58:22.216 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.216 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.216 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.216 DEBUG:boto:Params: {} 2024-04-12T03:58:22.216 DEBUG:boto:Token: None 2024-04-12T03:58:22.216 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.217 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L0HeqLS0hCWaQE+kif54GD9HGk0= 2024-04-12T03:58:22.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L0HeqLS0hCWaQE+kif54GD9HGk0=', 'Content-Length': '0'} 2024-04-12T03:58:22.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b623e8af8675d93f-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.218 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.219 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.219 DEBUG:boto:Method: GET 2024-04-12T03:58:22.219 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.219 DEBUG:boto:Data: 2024-04-12T03:58:22.219 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.219 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.219 DEBUG:boto:Params: {} 2024-04-12T03:58:22.219 DEBUG:boto:Token: None 2024-04-12T03:58:22.219 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000000 2024-04-12T03:58:22.220 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L0HeqLS0hCWaQE+kif54GD9HGk0= 2024-04-12T03:58:22.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L0HeqLS0hCWaQE+kif54GD9HGk0=', 'Content-Length': '0'} 2024-04-12T03:58:22.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1c3bf0e4d5d0d58-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.262 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.262 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.263 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:22.263 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:22.263 DEBUG:boto:Method: GET 2024-04-12T03:58:22.263 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:22.263 DEBUG:boto:Data: 2024-04-12T03:58:22.263 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.263 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.263 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.263 DEBUG:boto:Params: {} 2024-04-12T03:58:22.263 DEBUG:boto:Token: None 2024-04-12T03:58:22.264 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.264 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bb23afc15b44e7d-006618b15e-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.267 DEBUG:boto:b'dkyges-60obj000000000000000000001000falseobj000000000000000000002024-04-12T03:57:25.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.267 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.267 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.267 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:22.267 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:22.267 DEBUG:boto:Method: GET 2024-04-12T03:58:22.268 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:22.268 DEBUG:boto:Data: 2024-04-12T03:58:22.268 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.268 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.268 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.268 DEBUG:boto:Params: {} 2024-04-12T03:58:22.268 DEBUG:boto:Token: None 2024-04-12T03:58:22.268 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.268 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b6aeac5630cb46d-006618b15e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.271 DEBUG:boto:b'dkyges-60obj000000000000000000001000falseobj000000000000000000002024-04-12T03:57:25.785Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-12T03:58:22.271 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.272 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.272 DEBUG:boto:path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.272 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.272 DEBUG:boto:Method: GET 2024-04-12T03:58:22.272 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.272 DEBUG:boto:Data: 2024-04-12T03:58:22.272 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.272 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.272 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.272 DEBUG:boto:Params: {} 2024-04-12T03:58:22.272 DEBUG:boto:Token: None 2024-04-12T03:58:22.273 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.273 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zqjm+9A5e4AP4GwZnrIvFwhWeOE= 2024-04-12T03:58:22.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zqjm+9A5e4AP4GwZnrIvFwhWeOE=', 'Content-Length': '0'} 2024-04-12T03:58:22.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000caf4b16b6188ac3e-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.318 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.318 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.318 DEBUG:boto:path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.319 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.319 DEBUG:boto:Method: GET 2024-04-12T03:58:22.319 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.319 DEBUG:boto:Data: 2024-04-12T03:58:22.319 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.319 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.319 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.319 DEBUG:boto:Params: {} 2024-04-12T03:58:22.319 DEBUG:boto:Token: None 2024-04-12T03:58:22.319 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.320 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zqjm+9A5e4AP4GwZnrIvFwhWeOE= 2024-04-12T03:58:22.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zqjm+9A5e4AP4GwZnrIvFwhWeOE=', 'Content-Length': '0'} 2024-04-12T03:58:22.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063324ac1e8cc4e6c-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.322 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.322 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.322 DEBUG:boto:path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.322 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.322 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.322 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.323 DEBUG:boto:Data: 2024-04-12T03:58:22.323 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.323 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.323 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.323 DEBUG:boto:Params: {} 2024-04-12T03:58:22.323 DEBUG:boto:Token: None 2024-04-12T03:58:22.323 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.323 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jKdJy9aeulQ0+U41LqKVGbMryHw= 2024-04-12T03:58:22.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jKdJy9aeulQ0+U41LqKVGbMryHw=', 'Content-Length': '0'} 2024-04-12T03:58:22.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a3a2f06fae3e23a-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.325 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.326 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.326 DEBUG:boto:path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.326 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.326 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.326 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.326 DEBUG:boto:Data: 2024-04-12T03:58:22.326 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.326 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.326 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.326 DEBUG:boto:Params: {} 2024-04-12T03:58:22.326 DEBUG:boto:Token: None 2024-04-12T03:58:22.327 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.327 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jKdJy9aeulQ0+U41LqKVGbMryHw= 2024-04-12T03:58:22.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jKdJy9aeulQ0+U41LqKVGbMryHw=', 'Content-Length': '0'} 2024-04-12T03:58:22.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c7b00a6cce4148c-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-12T03:58:22.329 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.329 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.329 DEBUG:boto:path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.329 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.329 DEBUG:boto:Method: GET 2024-04-12T03:58:22.329 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.329 DEBUG:boto:Data: 2024-04-12T03:58:22.329 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.329 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.329 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.330 DEBUG:boto:Params: {} 2024-04-12T03:58:22.330 DEBUG:boto:Token: None 2024-04-12T03:58:22.330 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.330 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zqjm+9A5e4AP4GwZnrIvFwhWeOE= 2024-04-12T03:58:22.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zqjm+9A5e4AP4GwZnrIvFwhWeOE=', 'Content-Length': '0'} 2024-04-12T03:58:22.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005106251290c0888d-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.332 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.332 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.332 DEBUG:boto:path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.332 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.333 DEBUG:boto:Method: GET 2024-04-12T03:58:22.333 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.333 DEBUG:boto:Data: 2024-04-12T03:58:22.333 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.333 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.333 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.333 DEBUG:boto:Params: {} 2024-04-12T03:58:22.333 DEBUG:boto:Token: None 2024-04-12T03:58:22.333 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:22.333 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zqjm+9A5e4AP4GwZnrIvFwhWeOE= 2024-04-12T03:58:22.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zqjm+9A5e4AP4GwZnrIvFwhWeOE=', 'Content-Length': '0'} 2024-04-12T03:58:22.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e82a4ff8c33c77b-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.336 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.336 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.336 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:22.336 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:22.336 DEBUG:boto:Method: GET 2024-04-12T03:58:22.336 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:22.336 DEBUG:boto:Data: 2024-04-12T03:58:22.336 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.336 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.336 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.336 DEBUG:boto:Params: {} 2024-04-12T03:58:22.337 DEBUG:boto:Token: None 2024-04-12T03:58:22.337 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.337 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e92faff6cefaa24-006618b15e-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.340 DEBUG:boto:b'dkyges-60obj000000000000000000011000falseobj000000000000000000012024-04-12T03:57:25.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.340 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.340 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.340 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:22.340 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:22.340 DEBUG:boto:Method: GET 2024-04-12T03:58:22.341 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:22.341 DEBUG:boto:Data: 2024-04-12T03:58:22.341 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.341 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.341 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.341 DEBUG:boto:Params: {} 2024-04-12T03:58:22.341 DEBUG:boto:Token: None 2024-04-12T03:58:22.341 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.341 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d44fbbb82c1ed573-006618b15e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.345 DEBUG:boto:b'dkyges-60obj000000000000000000011000falseobj000000000000000000012024-04-12T03:57:25.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.345 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.345 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.345 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.345 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.345 DEBUG:boto:Data: 2024-04-12T03:58:22.345 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.346 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.346 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.346 DEBUG:boto:Params: {} 2024-04-12T03:58:22.346 DEBUG:boto:Token: None 2024-04-12T03:58:22.346 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.346 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kPdJ+sqfL71HYWs1/eUlTNqmr7Q= 2024-04-12T03:58:22.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kPdJ+sqfL71HYWs1/eUlTNqmr7Q=', 'Content-Length': '0'} 2024-04-12T03:58:22.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e42d67a73130f99-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.348 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.348 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.348 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.349 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.349 DEBUG:boto:Data: 2024-04-12T03:58:22.349 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.349 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.349 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.349 DEBUG:boto:Params: {} 2024-04-12T03:58:22.349 DEBUG:boto:Token: None 2024-04-12T03:58:22.349 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.349 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kPdJ+sqfL71HYWs1/eUlTNqmr7Q= 2024-04-12T03:58:22.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kPdJ+sqfL71HYWs1/eUlTNqmr7Q=', 'Content-Length': '0'} 2024-04-12T03:58:22.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7a317a8aa436d78-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-12T03:58:22.352 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.352 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.352 DEBUG:boto:Method: GET 2024-04-12T03:58:22.352 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.352 DEBUG:boto:Data: 2024-04-12T03:58:22.352 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.352 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.352 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.352 DEBUG:boto:Params: {} 2024-04-12T03:58:22.352 DEBUG:boto:Token: None 2024-04-12T03:58:22.352 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.353 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nBUYjYBxkI4B5DGA6BKWuE3D3vA= 2024-04-12T03:58:22.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nBUYjYBxkI4B5DGA6BKWuE3D3vA=', 'Content-Length': '0'} 2024-04-12T03:58:22.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001495cc989aff8458-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.355 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.355 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.355 DEBUG:boto:Method: GET 2024-04-12T03:58:22.355 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.355 DEBUG:boto:Data: 2024-04-12T03:58:22.355 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.355 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.355 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.355 DEBUG:boto:Params: {} 2024-04-12T03:58:22.355 DEBUG:boto:Token: None 2024-04-12T03:58:22.356 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000001 2024-04-12T03:58:22.356 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nBUYjYBxkI4B5DGA6BKWuE3D3vA= 2024-04-12T03:58:22.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nBUYjYBxkI4B5DGA6BKWuE3D3vA=', 'Content-Length': '0'} 2024-04-12T03:58:22.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010f3c510244e79a9-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.398 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.398 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.399 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:22.399 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:22.399 DEBUG:boto:Method: GET 2024-04-12T03:58:22.399 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:22.399 DEBUG:boto:Data: 2024-04-12T03:58:22.399 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.399 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.399 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.399 DEBUG:boto:Params: {} 2024-04-12T03:58:22.399 DEBUG:boto:Token: None 2024-04-12T03:58:22.399 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.400 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a1b18c18981e183-006618b15e-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.402 DEBUG:boto:b'dkyges-60obj000000000000000000011000falseobj000000000000000000012024-04-12T03:57:25.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.403 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.403 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.403 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:22.403 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:22.403 DEBUG:boto:Method: GET 2024-04-12T03:58:22.403 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:22.403 DEBUG:boto:Data: 2024-04-12T03:58:22.403 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.403 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.403 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.404 DEBUG:boto:Params: {} 2024-04-12T03:58:22.404 DEBUG:boto:Token: None 2024-04-12T03:58:22.404 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.404 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007764830d721f8d1c-006618b15e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.407 DEBUG:boto:b'dkyges-60obj000000000000000000011000falseobj000000000000000000012024-04-12T03:57:25.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-12T03:58:22.408 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.408 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.408 DEBUG:boto:path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.408 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.408 DEBUG:boto:Method: GET 2024-04-12T03:58:22.408 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.408 DEBUG:boto:Data: 2024-04-12T03:58:22.408 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.408 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.408 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.408 DEBUG:boto:Params: {} 2024-04-12T03:58:22.409 DEBUG:boto:Token: None 2024-04-12T03:58:22.409 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.409 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e9D4fAcFAzugbucMNlHGOEccDBU= 2024-04-12T03:58:22.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e9D4fAcFAzugbucMNlHGOEccDBU=', 'Content-Length': '0'} 2024-04-12T03:58:22.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007c28349c768d95a-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.454 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.454 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.455 DEBUG:boto:path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.455 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.455 DEBUG:boto:Method: GET 2024-04-12T03:58:22.455 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.455 DEBUG:boto:Data: 2024-04-12T03:58:22.455 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.455 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.455 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.455 DEBUG:boto:Params: {} 2024-04-12T03:58:22.455 DEBUG:boto:Token: None 2024-04-12T03:58:22.455 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.456 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e9D4fAcFAzugbucMNlHGOEccDBU= 2024-04-12T03:58:22.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e9D4fAcFAzugbucMNlHGOEccDBU=', 'Content-Length': '0'} 2024-04-12T03:58:22.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7323f388ae3ef1d-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.458 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.458 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.458 DEBUG:boto:path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.458 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.458 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.458 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.459 DEBUG:boto:Data: 2024-04-12T03:58:22.459 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.459 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.459 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.459 DEBUG:boto:Params: {} 2024-04-12T03:58:22.459 DEBUG:boto:Token: None 2024-04-12T03:58:22.459 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.459 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZOQoCMbOv8hb4TmShRNyXsYNFPs= 2024-04-12T03:58:22.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZOQoCMbOv8hb4TmShRNyXsYNFPs=', 'Content-Length': '0'} 2024-04-12T03:58:22.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046c6cc04afde95fe-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.461 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.461 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.461 DEBUG:boto:path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.462 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.462 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.462 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.462 DEBUG:boto:Data: 2024-04-12T03:58:22.462 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.462 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.462 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.462 DEBUG:boto:Params: {} 2024-04-12T03:58:22.462 DEBUG:boto:Token: None 2024-04-12T03:58:22.462 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.463 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZOQoCMbOv8hb4TmShRNyXsYNFPs= 2024-04-12T03:58:22.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZOQoCMbOv8hb4TmShRNyXsYNFPs=', 'Content-Length': '0'} 2024-04-12T03:58:22.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b4b549af01430a3-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-12T03:58:22.465 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.465 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.465 DEBUG:boto:path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.465 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.465 DEBUG:boto:Method: GET 2024-04-12T03:58:22.465 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.465 DEBUG:boto:Data: 2024-04-12T03:58:22.465 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.465 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.465 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.466 DEBUG:boto:Params: {} 2024-04-12T03:58:22.466 DEBUG:boto:Token: None 2024-04-12T03:58:22.466 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.466 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e9D4fAcFAzugbucMNlHGOEccDBU= 2024-04-12T03:58:22.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e9D4fAcFAzugbucMNlHGOEccDBU=', 'Content-Length': '0'} 2024-04-12T03:58:22.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050288385e704ef8d-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.468 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.468 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.468 DEBUG:boto:path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.468 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.469 DEBUG:boto:Method: GET 2024-04-12T03:58:22.469 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.469 DEBUG:boto:Data: 2024-04-12T03:58:22.469 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.469 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.469 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.469 DEBUG:boto:Params: {} 2024-04-12T03:58:22.469 DEBUG:boto:Token: None 2024-04-12T03:58:22.469 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:22.469 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e9D4fAcFAzugbucMNlHGOEccDBU= 2024-04-12T03:58:22.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e9D4fAcFAzugbucMNlHGOEccDBU=', 'Content-Length': '0'} 2024-04-12T03:58:22.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e75b3d2290b23a7-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.472 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.472 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.472 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:22.472 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:22.472 DEBUG:boto:Method: GET 2024-04-12T03:58:22.473 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:22.473 DEBUG:boto:Data: 2024-04-12T03:58:22.473 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.473 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.473 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.473 DEBUG:boto:Params: {} 2024-04-12T03:58:22.473 DEBUG:boto:Token: None 2024-04-12T03:58:22.473 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.473 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc8cb8b6c3367c9d-006618b15e-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.476 DEBUG:boto:b'dkyges-60obj000000000000000000021000falseobj000000000000000000022024-04-12T03:57:25.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.477 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.477 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.477 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:22.477 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:22.477 DEBUG:boto:Method: GET 2024-04-12T03:58:22.477 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:22.477 DEBUG:boto:Data: 2024-04-12T03:58:22.477 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.478 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.478 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.478 DEBUG:boto:Params: {} 2024-04-12T03:58:22.478 DEBUG:boto:Token: None 2024-04-12T03:58:22.478 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.478 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008974a686ecf1908f-006618b15e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.481 DEBUG:boto:b'dkyges-60obj000000000000000000021000falseobj000000000000000000022024-04-12T03:57:25.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.481 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.482 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.482 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.482 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.482 DEBUG:boto:Data: 2024-04-12T03:58:22.482 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.482 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.482 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.482 DEBUG:boto:Params: {} 2024-04-12T03:58:22.482 DEBUG:boto:Token: None 2024-04-12T03:58:22.482 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.483 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:08Z1VOZYKpnqRjN4fK0prSnNKY4= 2024-04-12T03:58:22.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:08Z1VOZYKpnqRjN4fK0prSnNKY4=', 'Content-Length': '0'} 2024-04-12T03:58:22.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d161d25194eaac4-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.485 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.485 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.485 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.485 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.485 DEBUG:boto:Data: 2024-04-12T03:58:22.485 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.485 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.485 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.486 DEBUG:boto:Params: {} 2024-04-12T03:58:22.486 DEBUG:boto:Token: None 2024-04-12T03:58:22.486 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.486 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:08Z1VOZYKpnqRjN4fK0prSnNKY4= 2024-04-12T03:58:22.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:08Z1VOZYKpnqRjN4fK0prSnNKY4=', 'Content-Length': '0'} 2024-04-12T03:58:22.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036fc49d47eb5d09f-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-12T03:58:22.489 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.489 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.489 DEBUG:boto:Method: GET 2024-04-12T03:58:22.489 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.489 DEBUG:boto:Data: 2024-04-12T03:58:22.489 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.489 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.489 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.489 DEBUG:boto:Params: {} 2024-04-12T03:58:22.489 DEBUG:boto:Token: None 2024-04-12T03:58:22.490 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.490 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rxdOg+gjLzqkj5iW6WpqpCEJank= 2024-04-12T03:58:22.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rxdOg+gjLzqkj5iW6WpqpCEJank=', 'Content-Length': '0'} 2024-04-12T03:58:22.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdb395343b46d50b-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.492 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.492 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.492 DEBUG:boto:Method: GET 2024-04-12T03:58:22.492 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.492 DEBUG:boto:Data: 2024-04-12T03:58:22.493 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.493 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.493 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.493 DEBUG:boto:Params: {} 2024-04-12T03:58:22.493 DEBUG:boto:Token: None 2024-04-12T03:58:22.493 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000002 2024-04-12T03:58:22.493 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rxdOg+gjLzqkj5iW6WpqpCEJank= 2024-04-12T03:58:22.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rxdOg+gjLzqkj5iW6WpqpCEJank=', 'Content-Length': '0'} 2024-04-12T03:58:22.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000838d93b9f987343f-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.496 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.496 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.496 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:22.496 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:22.496 DEBUG:boto:Method: GET 2024-04-12T03:58:22.496 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:22.496 DEBUG:boto:Data: 2024-04-12T03:58:22.496 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.496 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.497 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.497 DEBUG:boto:Params: {} 2024-04-12T03:58:22.497 DEBUG:boto:Token: None 2024-04-12T03:58:22.497 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.497 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019009f2a0ba0db6f-006618b15e-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.499 DEBUG:boto:b'dkyges-60obj000000000000000000021000falseobj000000000000000000022024-04-12T03:57:25.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.500 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.500 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.500 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:22.500 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:22.500 DEBUG:boto:Method: GET 2024-04-12T03:58:22.500 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:22.500 DEBUG:boto:Data: 2024-04-12T03:58:22.500 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.500 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.501 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.501 DEBUG:boto:Params: {} 2024-04-12T03:58:22.501 DEBUG:boto:Token: None 2024-04-12T03:58:22.501 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.501 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053660e55d4d78300-006618b15e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.546 DEBUG:boto:b'dkyges-60obj000000000000000000021000falseobj000000000000000000022024-04-12T03:57:25.713Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-12T03:58:22.547 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.547 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.547 DEBUG:boto:path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.547 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.547 DEBUG:boto:Method: GET 2024-04-12T03:58:22.547 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.547 DEBUG:boto:Data: 2024-04-12T03:58:22.547 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.548 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.548 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.548 DEBUG:boto:Params: {} 2024-04-12T03:58:22.548 DEBUG:boto:Token: None 2024-04-12T03:58:22.548 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.548 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xo220dH8l9VP2q9fu02O4jXcIIU= 2024-04-12T03:58:22.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xo220dH8l9VP2q9fu02O4jXcIIU=', 'Content-Length': '0'} 2024-04-12T03:58:22.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5b090569ee9682e-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.550 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.550 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.550 DEBUG:boto:path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.551 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.551 DEBUG:boto:Method: GET 2024-04-12T03:58:22.551 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.551 DEBUG:boto:Data: 2024-04-12T03:58:22.551 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.551 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.551 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.551 DEBUG:boto:Params: {} 2024-04-12T03:58:22.551 DEBUG:boto:Token: None 2024-04-12T03:58:22.552 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.552 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xo220dH8l9VP2q9fu02O4jXcIIU= 2024-04-12T03:58:22.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xo220dH8l9VP2q9fu02O4jXcIIU=', 'Content-Length': '0'} 2024-04-12T03:58:22.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0eeb9e6ae2668fc-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.554 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.554 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.554 DEBUG:boto:path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.554 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.554 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.554 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.554 DEBUG:boto:Data: 2024-04-12T03:58:22.555 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.555 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.555 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.555 DEBUG:boto:Params: {} 2024-04-12T03:58:22.555 DEBUG:boto:Token: None 2024-04-12T03:58:22.555 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.555 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:75Zvpyjm49kEDr12Oiyg4dPMRo0= 2024-04-12T03:58:22.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:75Zvpyjm49kEDr12Oiyg4dPMRo0=', 'Content-Length': '0'} 2024-04-12T03:58:22.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038a8974226237e23-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.557 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.557 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.558 DEBUG:boto:path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.558 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.558 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.558 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.558 DEBUG:boto:Data: 2024-04-12T03:58:22.558 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.558 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.558 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.558 DEBUG:boto:Params: {} 2024-04-12T03:58:22.558 DEBUG:boto:Token: None 2024-04-12T03:58:22.559 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.559 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:75Zvpyjm49kEDr12Oiyg4dPMRo0= 2024-04-12T03:58:22.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:75Zvpyjm49kEDr12Oiyg4dPMRo0=', 'Content-Length': '0'} 2024-04-12T03:58:22.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077a69e15591075b0-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-12T03:58:22.561 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.561 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.561 DEBUG:boto:path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.561 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.561 DEBUG:boto:Method: GET 2024-04-12T03:58:22.561 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.562 DEBUG:boto:Data: 2024-04-12T03:58:22.562 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.562 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.562 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.562 DEBUG:boto:Params: {} 2024-04-12T03:58:22.562 DEBUG:boto:Token: None 2024-04-12T03:58:22.562 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.562 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xo220dH8l9VP2q9fu02O4jXcIIU= 2024-04-12T03:58:22.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xo220dH8l9VP2q9fu02O4jXcIIU=', 'Content-Length': '0'} 2024-04-12T03:58:22.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa523b65506376d-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.564 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.565 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.565 DEBUG:boto:path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.565 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.565 DEBUG:boto:Method: GET 2024-04-12T03:58:22.565 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.565 DEBUG:boto:Data: 2024-04-12T03:58:22.565 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.565 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.565 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.565 DEBUG:boto:Params: {} 2024-04-12T03:58:22.565 DEBUG:boto:Token: None 2024-04-12T03:58:22.566 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:22.566 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xo220dH8l9VP2q9fu02O4jXcIIU= 2024-04-12T03:58:22.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xo220dH8l9VP2q9fu02O4jXcIIU=', 'Content-Length': '0'} 2024-04-12T03:58:22.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aebd27154db00d24-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.568 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.568 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.568 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:22.568 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:22.569 DEBUG:boto:Method: GET 2024-04-12T03:58:22.569 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:22.569 DEBUG:boto:Data: 2024-04-12T03:58:22.569 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.569 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.569 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.569 DEBUG:boto:Params: {} 2024-04-12T03:58:22.569 DEBUG:boto:Token: None 2024-04-12T03:58:22.569 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.569 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbfbf821e6d2818d-006618b15e-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.572 DEBUG:boto:b'dkyges-60obj000000000000000000041000falseobj000000000000000000042024-04-12T03:57:25.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.573 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.573 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.573 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:22.573 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:22.573 DEBUG:boto:Method: GET 2024-04-12T03:58:22.573 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:22.573 DEBUG:boto:Data: 2024-04-12T03:58:22.573 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.573 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.573 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.574 DEBUG:boto:Params: {} 2024-04-12T03:58:22.574 DEBUG:boto:Token: None 2024-04-12T03:58:22.574 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.574 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067ef8f4721927260-006618b15e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.576 DEBUG:boto:b'dkyges-60obj000000000000000000041000falseobj000000000000000000042024-04-12T03:57:25.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.577 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.577 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.577 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.577 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.577 DEBUG:boto:Data: 2024-04-12T03:58:22.577 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.577 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.577 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.578 DEBUG:boto:Params: {} 2024-04-12T03:58:22.578 DEBUG:boto:Token: None 2024-04-12T03:58:22.578 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.578 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+FGcKtSVyftCWbdvDcc1lp6MqiU= 2024-04-12T03:58:22.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+FGcKtSVyftCWbdvDcc1lp6MqiU=', 'Content-Length': '0'} 2024-04-12T03:58:22.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5966adab62c5fe1-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.580 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.580 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.581 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.581 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.581 DEBUG:boto:Data: 2024-04-12T03:58:22.581 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.581 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.581 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.581 DEBUG:boto:Params: {} 2024-04-12T03:58:22.581 DEBUG:boto:Token: None 2024-04-12T03:58:22.581 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.582 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+FGcKtSVyftCWbdvDcc1lp6MqiU= 2024-04-12T03:58:22.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+FGcKtSVyftCWbdvDcc1lp6MqiU=', 'Content-Length': '0'} 2024-04-12T03:58:22.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca83c10175dcef26-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-12T03:58:22.584 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.584 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.584 DEBUG:boto:Method: GET 2024-04-12T03:58:22.584 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.584 DEBUG:boto:Data: 2024-04-12T03:58:22.584 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.584 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.584 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.584 DEBUG:boto:Params: {} 2024-04-12T03:58:22.585 DEBUG:boto:Token: None 2024-04-12T03:58:22.585 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.585 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pJJGQlJ15sleHdVeTXUFC/XDZbA= 2024-04-12T03:58:22.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pJJGQlJ15sleHdVeTXUFC/XDZbA=', 'Content-Length': '0'} 2024-04-12T03:58:22.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026686172840a6d4a-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.630 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.630 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.631 DEBUG:boto:Method: GET 2024-04-12T03:58:22.631 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.631 DEBUG:boto:Data: 2024-04-12T03:58:22.631 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.631 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.631 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.631 DEBUG:boto:Params: {} 2024-04-12T03:58:22.631 DEBUG:boto:Token: None 2024-04-12T03:58:22.631 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000004 2024-04-12T03:58:22.631 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pJJGQlJ15sleHdVeTXUFC/XDZbA= 2024-04-12T03:58:22.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pJJGQlJ15sleHdVeTXUFC/XDZbA=', 'Content-Length': '0'} 2024-04-12T03:58:22.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099531ef8dae2c7ca-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.634 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.634 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.635 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:22.635 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:22.635 DEBUG:boto:Method: GET 2024-04-12T03:58:22.635 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:22.635 DEBUG:boto:Data: 2024-04-12T03:58:22.636 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.636 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.636 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.636 DEBUG:boto:Params: {} 2024-04-12T03:58:22.636 DEBUG:boto:Token: None 2024-04-12T03:58:22.636 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.637 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033399817c77af6c7-006618b15e-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.639 DEBUG:boto:b'dkyges-60obj000000000000000000041000falseobj000000000000000000042024-04-12T03:57:25.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.640 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.640 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.640 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:22.640 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:22.640 DEBUG:boto:Method: GET 2024-04-12T03:58:22.640 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:22.640 DEBUG:boto:Data: 2024-04-12T03:58:22.641 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.641 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.641 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.641 DEBUG:boto:Params: {} 2024-04-12T03:58:22.641 DEBUG:boto:Token: None 2024-04-12T03:58:22.641 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.641 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016d5095a9d5e5cba-006618b15e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.644 DEBUG:boto:b'dkyges-60obj000000000000000000041000falseobj000000000000000000042024-04-12T03:57:25.723Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-12T03:58:22.644 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.644 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.644 DEBUG:boto:path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.644 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.645 DEBUG:boto:Method: GET 2024-04-12T03:58:22.645 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.645 DEBUG:boto:Data: 2024-04-12T03:58:22.645 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.645 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.645 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.645 DEBUG:boto:Params: {} 2024-04-12T03:58:22.645 DEBUG:boto:Token: None 2024-04-12T03:58:22.645 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.646 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cLAw8IooB2qMCfmugIJgqo8gsnA= 2024-04-12T03:58:22.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cLAw8IooB2qMCfmugIJgqo8gsnA=', 'Content-Length': '0'} 2024-04-12T03:58:22.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dcc2c2f77a16408-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.648 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.648 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.648 DEBUG:boto:path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.648 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.648 DEBUG:boto:Method: GET 2024-04-12T03:58:22.648 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.648 DEBUG:boto:Data: 2024-04-12T03:58:22.649 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.649 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.649 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.649 DEBUG:boto:Params: {} 2024-04-12T03:58:22.649 DEBUG:boto:Token: None 2024-04-12T03:58:22.649 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.649 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cLAw8IooB2qMCfmugIJgqo8gsnA= 2024-04-12T03:58:22.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cLAw8IooB2qMCfmugIJgqo8gsnA=', 'Content-Length': '0'} 2024-04-12T03:58:22.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013c1d48be5cb6a62-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.652 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.652 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.652 DEBUG:boto:path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.652 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.652 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.652 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.652 DEBUG:boto:Data: 2024-04-12T03:58:22.652 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.653 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.653 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.653 DEBUG:boto:Params: {} 2024-04-12T03:58:22.653 DEBUG:boto:Token: None 2024-04-12T03:58:22.653 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.653 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ngwp/joXiUgBipym/UIwze+Ap8c= 2024-04-12T03:58:22.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ngwp/joXiUgBipym/UIwze+Ap8c=', 'Content-Length': '0'} 2024-04-12T03:58:22.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a655fc679506a439-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.655 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.655 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.656 DEBUG:boto:path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.656 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.656 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.656 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.656 DEBUG:boto:Data: 2024-04-12T03:58:22.656 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.656 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.656 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.656 DEBUG:boto:Params: {} 2024-04-12T03:58:22.657 DEBUG:boto:Token: None 2024-04-12T03:58:22.657 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.657 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ngwp/joXiUgBipym/UIwze+Ap8c= 2024-04-12T03:58:22.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ngwp/joXiUgBipym/UIwze+Ap8c=', 'Content-Length': '0'} 2024-04-12T03:58:22.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e00acba849598e81-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-12T03:58:22.659 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.659 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.660 DEBUG:boto:path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.660 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.660 DEBUG:boto:Method: GET 2024-04-12T03:58:22.660 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.660 DEBUG:boto:Data: 2024-04-12T03:58:22.660 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.660 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.660 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.660 DEBUG:boto:Params: {} 2024-04-12T03:58:22.660 DEBUG:boto:Token: None 2024-04-12T03:58:22.661 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.661 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cLAw8IooB2qMCfmugIJgqo8gsnA= 2024-04-12T03:58:22.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cLAw8IooB2qMCfmugIJgqo8gsnA=', 'Content-Length': '0'} 2024-04-12T03:58:22.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d60d8250c9c1850-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.706 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.706 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.707 DEBUG:boto:path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.707 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.707 DEBUG:boto:Method: GET 2024-04-12T03:58:22.707 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.707 DEBUG:boto:Data: 2024-04-12T03:58:22.707 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.707 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.707 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.707 DEBUG:boto:Params: {} 2024-04-12T03:58:22.707 DEBUG:boto:Token: None 2024-04-12T03:58:22.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:22.708 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cLAw8IooB2qMCfmugIJgqo8gsnA= 2024-04-12T03:58:22.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cLAw8IooB2qMCfmugIJgqo8gsnA=', 'Content-Length': '0'} 2024-04-12T03:58:22.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aab324a3838afe0b-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.710 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.710 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.711 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:22.711 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:22.711 DEBUG:boto:Method: GET 2024-04-12T03:58:22.711 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:22.711 DEBUG:boto:Data: 2024-04-12T03:58:22.711 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.711 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.711 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.711 DEBUG:boto:Params: {} 2024-04-12T03:58:22.711 DEBUG:boto:Token: None 2024-04-12T03:58:22.712 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.712 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a9008a1a20f2aca-006618b15e-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.714 DEBUG:boto:b'dkyges-60obj000000000000000000051000falseobj000000000000000000052024-04-12T03:57:25.796Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.715 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.715 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.715 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:22.715 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:22.715 DEBUG:boto:Method: GET 2024-04-12T03:58:22.715 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:22.715 DEBUG:boto:Data: 2024-04-12T03:58:22.715 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.715 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.716 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.716 DEBUG:boto:Params: {} 2024-04-12T03:58:22.716 DEBUG:boto:Token: None 2024-04-12T03:58:22.716 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.716 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ff113b5b8c74e07-006618b15e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.719 DEBUG:boto:b'dkyges-60obj000000000000000000051000falseobj000000000000000000052024-04-12T03:57:25.796Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.719 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.719 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.719 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.719 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.719 DEBUG:boto:Data: 2024-04-12T03:58:22.719 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.719 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.720 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.720 DEBUG:boto:Params: {} 2024-04-12T03:58:22.720 DEBUG:boto:Token: None 2024-04-12T03:58:22.720 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.720 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dQwc6hfMt0QdXK2sgNeVJ7DzIK0= 2024-04-12T03:58:22.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dQwc6hfMt0QdXK2sgNeVJ7DzIK0=', 'Content-Length': '0'} 2024-04-12T03:58:22.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0943c1d65a041ef-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.722 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.722 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.722 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.722 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.723 DEBUG:boto:Data: 2024-04-12T03:58:22.723 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.723 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.723 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.723 DEBUG:boto:Params: {} 2024-04-12T03:58:22.723 DEBUG:boto:Token: None 2024-04-12T03:58:22.723 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.723 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dQwc6hfMt0QdXK2sgNeVJ7DzIK0= 2024-04-12T03:58:22.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dQwc6hfMt0QdXK2sgNeVJ7DzIK0=', 'Content-Length': '0'} 2024-04-12T03:58:22.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004722e6506bf31455-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-12T03:58:22.725 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.725 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.726 DEBUG:boto:Method: GET 2024-04-12T03:58:22.726 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.726 DEBUG:boto:Data: 2024-04-12T03:58:22.726 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.726 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.726 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.726 DEBUG:boto:Params: {} 2024-04-12T03:58:22.726 DEBUG:boto:Token: None 2024-04-12T03:58:22.726 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.726 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nfcL+o8HEFXc5vXMIOKZ6kLi/C8= 2024-04-12T03:58:22.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nfcL+o8HEFXc5vXMIOKZ6kLi/C8=', 'Content-Length': '0'} 2024-04-12T03:58:22.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000875d057c3ab4603b-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.729 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.729 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.729 DEBUG:boto:Method: GET 2024-04-12T03:58:22.729 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.729 DEBUG:boto:Data: 2024-04-12T03:58:22.729 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.729 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.729 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.729 DEBUG:boto:Params: {} 2024-04-12T03:58:22.729 DEBUG:boto:Token: None 2024-04-12T03:58:22.730 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000005 2024-04-12T03:58:22.730 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nfcL+o8HEFXc5vXMIOKZ6kLi/C8= 2024-04-12T03:58:22.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nfcL+o8HEFXc5vXMIOKZ6kLi/C8=', 'Content-Length': '0'} 2024-04-12T03:58:22.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b16cd69bfb4aa046-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.732 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.732 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.732 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:22.732 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:22.732 DEBUG:boto:Method: GET 2024-04-12T03:58:22.732 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:22.732 DEBUG:boto:Data: 2024-04-12T03:58:22.732 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.732 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.732 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.733 DEBUG:boto:Params: {} 2024-04-12T03:58:22.733 DEBUG:boto:Token: None 2024-04-12T03:58:22.733 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.733 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dace2d6840d90aa2-006618b15e-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.735 DEBUG:boto:b'dkyges-60obj000000000000000000051000falseobj000000000000000000052024-04-12T03:57:25.796Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.736 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.736 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.736 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:22.736 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:22.736 DEBUG:boto:Method: GET 2024-04-12T03:58:22.736 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:22.736 DEBUG:boto:Data: 2024-04-12T03:58:22.737 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.737 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.737 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.737 DEBUG:boto:Params: {} 2024-04-12T03:58:22.737 DEBUG:boto:Token: None 2024-04-12T03:58:22.737 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.737 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017be2091a7a07a35-006618b15e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.740 DEBUG:boto:b'dkyges-60obj000000000000000000051000falseobj000000000000000000052024-04-12T03:57:25.796Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-12T03:58:22.740 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.741 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.741 DEBUG:boto:path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.741 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.741 DEBUG:boto:Method: GET 2024-04-12T03:58:22.741 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.741 DEBUG:boto:Data: 2024-04-12T03:58:22.741 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.741 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.741 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.741 DEBUG:boto:Params: {} 2024-04-12T03:58:22.741 DEBUG:boto:Token: None 2024-04-12T03:58:22.742 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.742 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j8h00hQQfAkjCbLCcV71zqYFd7g= 2024-04-12T03:58:22.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j8h00hQQfAkjCbLCcV71zqYFd7g=', 'Content-Length': '0'} 2024-04-12T03:58:22.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d395b833c68d448d-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.786 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.786 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.786 DEBUG:boto:path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.787 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.787 DEBUG:boto:Method: GET 2024-04-12T03:58:22.787 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.787 DEBUG:boto:Data: 2024-04-12T03:58:22.787 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.787 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.787 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.787 DEBUG:boto:Params: {} 2024-04-12T03:58:22.787 DEBUG:boto:Token: None 2024-04-12T03:58:22.787 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.788 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j8h00hQQfAkjCbLCcV71zqYFd7g= 2024-04-12T03:58:22.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j8h00hQQfAkjCbLCcV71zqYFd7g=', 'Content-Length': '0'} 2024-04-12T03:58:22.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c09aa49ecef6563-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.790 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.790 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.790 DEBUG:boto:path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.790 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.790 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.790 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.790 DEBUG:boto:Data: 2024-04-12T03:58:22.790 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.790 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.790 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.791 DEBUG:boto:Params: {} 2024-04-12T03:58:22.791 DEBUG:boto:Token: None 2024-04-12T03:58:22.791 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.791 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uEmH0qs5tdSxmqk3DB6uaik3nFg= 2024-04-12T03:58:22.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uEmH0qs5tdSxmqk3DB6uaik3nFg=', 'Content-Length': '0'} 2024-04-12T03:58:22.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045bda7abb5a08834-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.793 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.793 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.793 DEBUG:boto:path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.793 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.793 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.793 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.793 DEBUG:boto:Data: 2024-04-12T03:58:22.794 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.794 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.794 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.794 DEBUG:boto:Params: {} 2024-04-12T03:58:22.794 DEBUG:boto:Token: None 2024-04-12T03:58:22.794 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.794 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uEmH0qs5tdSxmqk3DB6uaik3nFg= 2024-04-12T03:58:22.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uEmH0qs5tdSxmqk3DB6uaik3nFg=', 'Content-Length': '0'} 2024-04-12T03:58:22.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039fa7700ed4ca80e-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-12T03:58:22.796 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.797 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.797 DEBUG:boto:path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.797 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.797 DEBUG:boto:Method: GET 2024-04-12T03:58:22.797 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.797 DEBUG:boto:Data: 2024-04-12T03:58:22.797 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.797 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.797 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.797 DEBUG:boto:Params: {} 2024-04-12T03:58:22.798 DEBUG:boto:Token: None 2024-04-12T03:58:22.798 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.798 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j8h00hQQfAkjCbLCcV71zqYFd7g= 2024-04-12T03:58:22.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j8h00hQQfAkjCbLCcV71zqYFd7g=', 'Content-Length': '0'} 2024-04-12T03:58:22.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002d51e7860327b3d-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.800 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.800 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.801 DEBUG:boto:path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.801 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.801 DEBUG:boto:Method: GET 2024-04-12T03:58:22.801 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.801 DEBUG:boto:Data: 2024-04-12T03:58:22.801 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.801 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.801 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.801 DEBUG:boto:Params: {} 2024-04-12T03:58:22.801 DEBUG:boto:Token: None 2024-04-12T03:58:22.801 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:22.802 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j8h00hQQfAkjCbLCcV71zqYFd7g= 2024-04-12T03:58:22.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j8h00hQQfAkjCbLCcV71zqYFd7g=', 'Content-Length': '0'} 2024-04-12T03:58:22.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008663feec045cc99c-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.804 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.804 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.804 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:22.804 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:22.804 DEBUG:boto:Method: GET 2024-04-12T03:58:22.804 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:22.804 DEBUG:boto:Data: 2024-04-12T03:58:22.805 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.805 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.805 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.805 DEBUG:boto:Params: {} 2024-04-12T03:58:22.805 DEBUG:boto:Token: None 2024-04-12T03:58:22.805 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.805 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008651d4ac337dd309-006618b15e-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.808 DEBUG:boto:b'dkyges-60obj000000000000000000061000falseobj000000000000000000062024-04-12T03:57:25.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.808 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.809 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.809 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:22.809 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:22.809 DEBUG:boto:Method: GET 2024-04-12T03:58:22.809 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:22.809 DEBUG:boto:Data: 2024-04-12T03:58:22.809 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.809 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.809 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.809 DEBUG:boto:Params: {} 2024-04-12T03:58:22.809 DEBUG:boto:Token: None 2024-04-12T03:58:22.809 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.810 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001955fc0452d18270-006618b15e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.812 DEBUG:boto:b'dkyges-60obj000000000000000000061000falseobj000000000000000000062024-04-12T03:57:25.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.813 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.813 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.813 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.813 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.813 DEBUG:boto:Data: 2024-04-12T03:58:22.813 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.813 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.813 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.813 DEBUG:boto:Params: {} 2024-04-12T03:58:22.814 DEBUG:boto:Token: None 2024-04-12T03:58:22.814 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.814 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qT7OyUjLl6mkYqJuj1QEKGJxwOk= 2024-04-12T03:58:22.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qT7OyUjLl6mkYqJuj1QEKGJxwOk=', 'Content-Length': '0'} 2024-04-12T03:58:22.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f1718ed0e6f812a-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.816 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.816 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.816 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.816 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.816 DEBUG:boto:Data: 2024-04-12T03:58:22.816 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.816 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.816 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.816 DEBUG:boto:Params: {} 2024-04-12T03:58:22.817 DEBUG:boto:Token: None 2024-04-12T03:58:22.817 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.817 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qT7OyUjLl6mkYqJuj1QEKGJxwOk= 2024-04-12T03:58:22.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qT7OyUjLl6mkYqJuj1QEKGJxwOk=', 'Content-Length': '0'} 2024-04-12T03:58:22.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3b24db3af511c7d-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-12T03:58:22.819 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.819 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.819 DEBUG:boto:Method: GET 2024-04-12T03:58:22.819 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.819 DEBUG:boto:Data: 2024-04-12T03:58:22.819 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.820 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.820 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.820 DEBUG:boto:Params: {} 2024-04-12T03:58:22.820 DEBUG:boto:Token: None 2024-04-12T03:58:22.820 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.820 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SOipTbNq/sX93c6AbOFityenZbw= 2024-04-12T03:58:22.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SOipTbNq/sX93c6AbOFityenZbw=', 'Content-Length': '0'} 2024-04-12T03:58:22.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073ab9871be9e25d8-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.822 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.822 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.822 DEBUG:boto:Method: GET 2024-04-12T03:58:22.823 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.823 DEBUG:boto:Data: 2024-04-12T03:58:22.823 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.823 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.823 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.823 DEBUG:boto:Params: {} 2024-04-12T03:58:22.823 DEBUG:boto:Token: None 2024-04-12T03:58:22.823 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000006 2024-04-12T03:58:22.823 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SOipTbNq/sX93c6AbOFityenZbw= 2024-04-12T03:58:22.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SOipTbNq/sX93c6AbOFityenZbw=', 'Content-Length': '0'} 2024-04-12T03:58:22.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000960d0f658ea9c94b-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.866 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.866 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.867 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:22.867 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:22.867 DEBUG:boto:Method: GET 2024-04-12T03:58:22.867 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:22.867 DEBUG:boto:Data: 2024-04-12T03:58:22.867 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.867 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.867 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.867 DEBUG:boto:Params: {} 2024-04-12T03:58:22.867 DEBUG:boto:Token: None 2024-04-12T03:58:22.868 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.868 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5765ded464b1a99-006618b15e-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.870 DEBUG:boto:b'dkyges-60obj000000000000000000061000falseobj000000000000000000062024-04-12T03:57:25.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.871 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.871 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.871 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:22.871 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:22.871 DEBUG:boto:Method: GET 2024-04-12T03:58:22.871 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:22.871 DEBUG:boto:Data: 2024-04-12T03:58:22.872 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.872 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.872 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.872 DEBUG:boto:Params: {} 2024-04-12T03:58:22.872 DEBUG:boto:Token: None 2024-04-12T03:58:22.872 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.872 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000602a8d1a1282c266-006618b15e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.875 DEBUG:boto:b'dkyges-60obj000000000000000000061000falseobj000000000000000000062024-04-12T03:57:25.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-12T03:58:22.876 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.876 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.876 DEBUG:boto:path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.876 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.876 DEBUG:boto:Method: GET 2024-04-12T03:58:22.876 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.876 DEBUG:boto:Data: 2024-04-12T03:58:22.876 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.876 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.876 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.876 DEBUG:boto:Params: {} 2024-04-12T03:58:22.876 DEBUG:boto:Token: None 2024-04-12T03:58:22.877 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.877 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kQh/b+1xSO8SENoziNAm1F1HKtM= 2024-04-12T03:58:22.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kQh/b+1xSO8SENoziNAm1F1HKtM=', 'Content-Length': '0'} 2024-04-12T03:58:22.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eeb62198b3da8692-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.922 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.923 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.923 DEBUG:boto:path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.923 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.923 DEBUG:boto:Method: GET 2024-04-12T03:58:22.923 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.923 DEBUG:boto:Data: 2024-04-12T03:58:22.923 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.923 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.923 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.923 DEBUG:boto:Params: {} 2024-04-12T03:58:22.923 DEBUG:boto:Token: None 2024-04-12T03:58:22.924 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.924 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kQh/b+1xSO8SENoziNAm1F1HKtM= 2024-04-12T03:58:22.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kQh/b+1xSO8SENoziNAm1F1HKtM=', 'Content-Length': '0'} 2024-04-12T03:58:22.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3cf09d06be253fd-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.926 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.926 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.927 DEBUG:boto:path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.929 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.930 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.930 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.930 DEBUG:boto:Data: 2024-04-12T03:58:22.930 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.930 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.930 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.930 DEBUG:boto:Params: {} 2024-04-12T03:58:22.930 DEBUG:boto:Token: None 2024-04-12T03:58:22.930 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.931 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:doDC/1EDGa4ZunnuQoTb1E+vt08= 2024-04-12T03:58:22.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:doDC/1EDGa4ZunnuQoTb1E+vt08=', 'Content-Length': '0'} 2024-04-12T03:58:22.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dd2ddfb268d5472-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.933 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.933 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.933 DEBUG:boto:path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.933 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.933 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.933 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.933 DEBUG:boto:Data: 2024-04-12T03:58:22.934 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.934 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.934 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.934 DEBUG:boto:Params: {} 2024-04-12T03:58:22.934 DEBUG:boto:Token: None 2024-04-12T03:58:22.934 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.934 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:doDC/1EDGa4ZunnuQoTb1E+vt08= 2024-04-12T03:58:22.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:doDC/1EDGa4ZunnuQoTb1E+vt08=', 'Content-Length': '0'} 2024-04-12T03:58:22.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6e4ead50259a95c-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-12T03:58:22.937 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.937 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.937 DEBUG:boto:path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.937 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.937 DEBUG:boto:Method: GET 2024-04-12T03:58:22.937 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.937 DEBUG:boto:Data: 2024-04-12T03:58:22.937 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.937 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.937 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.938 DEBUG:boto:Params: {} 2024-04-12T03:58:22.938 DEBUG:boto:Token: None 2024-04-12T03:58:22.938 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.938 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kQh/b+1xSO8SENoziNAm1F1HKtM= 2024-04-12T03:58:22.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kQh/b+1xSO8SENoziNAm1F1HKtM=', 'Content-Length': '0'} 2024-04-12T03:58:22.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000079ee918853acb2-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.940 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.941 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.941 DEBUG:boto:path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.941 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.941 DEBUG:boto:Method: GET 2024-04-12T03:58:22.941 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.941 DEBUG:boto:Data: 2024-04-12T03:58:22.941 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.941 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.941 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.941 DEBUG:boto:Params: {} 2024-04-12T03:58:22.941 DEBUG:boto:Token: None 2024-04-12T03:58:22.941 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:22.942 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kQh/b+1xSO8SENoziNAm1F1HKtM= 2024-04-12T03:58:22.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kQh/b+1xSO8SENoziNAm1F1HKtM=', 'Content-Length': '0'} 2024-04-12T03:58:22.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f68b7d0ca55c957-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.944 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.944 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.944 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:22.944 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:22.944 DEBUG:boto:Method: GET 2024-04-12T03:58:22.945 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:22.945 DEBUG:boto:Data: 2024-04-12T03:58:22.945 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.945 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.945 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.945 DEBUG:boto:Params: {} 2024-04-12T03:58:22.945 DEBUG:boto:Token: None 2024-04-12T03:58:22.945 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.945 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6b5ffa0d26dbe15-006618b15e-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.948 DEBUG:boto:b'dkyges-60obj000000000000000000101000falseobj000000000000000000102024-04-12T03:57:25.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.948 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:22.948 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:22.949 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:22.949 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:22.949 DEBUG:boto:Method: GET 2024-04-12T03:58:22.949 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:22.949 DEBUG:boto:Data: 2024-04-12T03:58:22.949 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.949 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.949 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.949 DEBUG:boto:Params: {} 2024-04-12T03:58:22.949 DEBUG:boto:Token: None 2024-04-12T03:58:22.950 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/ 2024-04-12T03:58:22.950 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w= 2024-04-12T03:58:22.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umURJm4j470ez/I5CAt4hiIj34w=', 'Content-Length': '0'} 2024-04-12T03:58:22.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bad8ff7e0d7a2144-006618b15e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.952 DEBUG:boto:b'dkyges-60obj000000000000000000101000falseobj000000000000000000102024-04-12T03:57:25.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:22.953 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.953 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.953 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.953 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.953 DEBUG:boto:Data: 2024-04-12T03:58:22.953 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.953 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.953 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.953 DEBUG:boto:Params: {} 2024-04-12T03:58:22.954 DEBUG:boto:Token: None 2024-04-12T03:58:22.954 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.954 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:enGdPAQY7oWLqK68q0pnq8t9PcA= 2024-04-12T03:58:22.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:enGdPAQY7oWLqK68q0pnq8t9PcA=', 'Content-Length': '0'} 2024-04-12T03:58:22.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001807c3f5eb85e168-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.956 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.956 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.956 DEBUG:boto:Method: HEAD 2024-04-12T03:58:22.956 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.956 DEBUG:boto:Data: 2024-04-12T03:58:22.956 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.957 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.957 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.957 DEBUG:boto:Params: {} 2024-04-12T03:58:22.957 DEBUG:boto:Token: None 2024-04-12T03:58:22.957 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.957 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:enGdPAQY7oWLqK68q0pnq8t9PcA= 2024-04-12T03:58:22.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:enGdPAQY7oWLqK68q0pnq8t9PcA=', 'Content-Length': '0'} 2024-04-12T03:58:22.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009634bdc7e85e996d-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-12T03:58:22.959 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.960 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.960 DEBUG:boto:Method: GET 2024-04-12T03:58:22.960 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.960 DEBUG:boto:Data: 2024-04-12T03:58:22.960 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.960 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:22.960 DEBUG:boto:Port: 8001 2024-04-12T03:58:22.960 DEBUG:boto:Params: {} 2024-04-12T03:58:22.960 DEBUG:boto:Token: None 2024-04-12T03:58:22.960 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.960 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r1uiAxCgnTugCTWZq05sH0EV+/k= 2024-04-12T03:58:22.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r1uiAxCgnTugCTWZq05sH0EV+/k=', 'Content-Length': '0'} 2024-04-12T03:58:22.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a2bf4f04f0da494-006618b15e-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:22.963 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.963 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.963 DEBUG:boto:Method: GET 2024-04-12T03:58:22.963 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.963 DEBUG:boto:Data: 2024-04-12T03:58:22.963 DEBUG:boto:Headers: {} 2024-04-12T03:58:22.963 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:22.963 DEBUG:boto:Port: 8000 2024-04-12T03:58:22.963 DEBUG:boto:Params: {} 2024-04-12T03:58:22.964 DEBUG:boto:Token: None 2024-04-12T03:58:22.964 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:22 GMT /dkyges-60/obj00000000000000000010 2024-04-12T03:58:22.964 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r1uiAxCgnTugCTWZq05sH0EV+/k= 2024-04-12T03:58:22.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r1uiAxCgnTugCTWZq05sH0EV+/k=', 'Content-Length': '0'} 2024-04-12T03:58:22.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045502dba1094b375-006618b15e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.006 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:23.006 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:23.007 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:23.007 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:23.007 DEBUG:boto:Method: GET 2024-04-12T03:58:23.007 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:23.007 DEBUG:boto:Data: 2024-04-12T03:58:23.007 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.007 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.007 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.007 DEBUG:boto:Params: {} 2024-04-12T03:58:23.007 DEBUG:boto:Token: None 2024-04-12T03:58:23.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/ 2024-04-12T03:58:23.008 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs= 2024-04-12T03:58:23.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs=', 'Content-Length': '0'} 2024-04-12T03:58:23.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a429c7b9d54a5db2-006618b15f-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.010 DEBUG:boto:b'dkyges-60obj000000000000000000101000falseobj000000000000000000102024-04-12T03:57:25.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:23.011 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:23.011 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:23.011 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:23.011 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:23.011 DEBUG:boto:Method: GET 2024-04-12T03:58:23.011 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:23.011 DEBUG:boto:Data: 2024-04-12T03:58:23.012 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.012 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.012 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.012 DEBUG:boto:Params: {} 2024-04-12T03:58:23.012 DEBUG:boto:Token: None 2024-04-12T03:58:23.012 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/ 2024-04-12T03:58:23.013 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs= 2024-04-12T03:58:23.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs=', 'Content-Length': '0'} 2024-04-12T03:58:23.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1fd4393ecccb67b-006618b15f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.015 DEBUG:boto:b'dkyges-60obj000000000000000000101000falseobj000000000000000000102024-04-12T03:57:25.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:23.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-12T03:58:23.016 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.016 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.016 DEBUG:boto:path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.016 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.016 DEBUG:boto:Method: GET 2024-04-12T03:58:23.017 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.017 DEBUG:boto:Data: 2024-04-12T03:58:23.017 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.017 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.017 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.017 DEBUG:boto:Params: {} 2024-04-12T03:58:23.017 DEBUG:boto:Token: None 2024-04-12T03:58:23.017 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.017 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b24QRIgOLfYf0OQY7s6p29XIr+8= 2024-04-12T03:58:23.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b24QRIgOLfYf0OQY7s6p29XIr+8=', 'Content-Length': '0'} 2024-04-12T03:58:23.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c25b5cfeef0274e9-006618b15f-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.062 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.062 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.063 DEBUG:boto:path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.063 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.063 DEBUG:boto:Method: GET 2024-04-12T03:58:23.063 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.063 DEBUG:boto:Data: 2024-04-12T03:58:23.063 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.063 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.063 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.063 DEBUG:boto:Params: {} 2024-04-12T03:58:23.063 DEBUG:boto:Token: None 2024-04-12T03:58:23.064 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.064 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b24QRIgOLfYf0OQY7s6p29XIr+8= 2024-04-12T03:58:23.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b24QRIgOLfYf0OQY7s6p29XIr+8=', 'Content-Length': '0'} 2024-04-12T03:58:23.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000002734431b13a62c-006618b15f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.066 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.066 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.066 DEBUG:boto:path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.066 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.066 DEBUG:boto:Method: HEAD 2024-04-12T03:58:23.067 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.067 DEBUG:boto:Data: 2024-04-12T03:58:23.067 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.067 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.067 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.067 DEBUG:boto:Params: {} 2024-04-12T03:58:23.067 DEBUG:boto:Token: None 2024-04-12T03:58:23.067 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.067 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xxUmo1TLDaVaSfeqYjoqIjJErrA= 2024-04-12T03:58:23.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xxUmo1TLDaVaSfeqYjoqIjJErrA=', 'Content-Length': '0'} 2024-04-12T03:58:23.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000490aaf5414cffbcc-006618b15f-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.070 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.070 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.070 DEBUG:boto:path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.070 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.070 DEBUG:boto:Method: HEAD 2024-04-12T03:58:23.070 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.070 DEBUG:boto:Data: 2024-04-12T03:58:23.070 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.071 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.071 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.071 DEBUG:boto:Params: {} 2024-04-12T03:58:23.071 DEBUG:boto:Token: None 2024-04-12T03:58:23.071 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.071 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xxUmo1TLDaVaSfeqYjoqIjJErrA= 2024-04-12T03:58:23.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xxUmo1TLDaVaSfeqYjoqIjJErrA=', 'Content-Length': '0'} 2024-04-12T03:58:23.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f8cb8d14141137b-006618b15f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-12T03:58:23.073 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.074 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.074 DEBUG:boto:path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.074 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.074 DEBUG:boto:Method: GET 2024-04-12T03:58:23.074 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.074 DEBUG:boto:Data: 2024-04-12T03:58:23.074 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.074 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.074 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.074 DEBUG:boto:Params: {} 2024-04-12T03:58:23.075 DEBUG:boto:Token: None 2024-04-12T03:58:23.075 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.075 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b24QRIgOLfYf0OQY7s6p29XIr+8= 2024-04-12T03:58:23.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b24QRIgOLfYf0OQY7s6p29XIr+8=', 'Content-Length': '0'} 2024-04-12T03:58:23.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df50df846fc4df5a-006618b15f-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.077 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.077 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.077 DEBUG:boto:path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.077 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.077 DEBUG:boto:Method: GET 2024-04-12T03:58:23.077 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.078 DEBUG:boto:Data: 2024-04-12T03:58:23.078 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.078 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.078 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.078 DEBUG:boto:Params: {} 2024-04-12T03:58:23.078 DEBUG:boto:Token: None 2024-04-12T03:58:23.078 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:23.078 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b24QRIgOLfYf0OQY7s6p29XIr+8= 2024-04-12T03:58:23.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b24QRIgOLfYf0OQY7s6p29XIr+8=', 'Content-Length': '0'} 2024-04-12T03:58:23.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073e4e08b1f11c84c-006618b15f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.081 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:23.081 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:23.081 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:23.081 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:23.081 DEBUG:boto:Method: GET 2024-04-12T03:58:23.081 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:23.081 DEBUG:boto:Data: 2024-04-12T03:58:23.081 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.081 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.081 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.081 DEBUG:boto:Params: {} 2024-04-12T03:58:23.082 DEBUG:boto:Token: None 2024-04-12T03:58:23.082 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/ 2024-04-12T03:58:23.082 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs= 2024-04-12T03:58:23.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs=', 'Content-Length': '0'} 2024-04-12T03:58:23.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008335d3abf8e285d9-006618b15f-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.084 DEBUG:boto:b'dkyges-60obj000000000000000000111000falseobj000000000000000000112024-04-12T03:57:25.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:23.085 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:23.085 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:23.085 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:23.085 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:23.085 DEBUG:boto:Method: GET 2024-04-12T03:58:23.085 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:23.085 DEBUG:boto:Data: 2024-04-12T03:58:23.085 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.086 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.086 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.086 DEBUG:boto:Params: {} 2024-04-12T03:58:23.086 DEBUG:boto:Token: None 2024-04-12T03:58:23.086 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/ 2024-04-12T03:58:23.086 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs= 2024-04-12T03:58:23.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs=', 'Content-Length': '0'} 2024-04-12T03:58:23.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2bbe031ae0bdd83-006618b15f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.089 DEBUG:boto:b'dkyges-60obj000000000000000000111000falseobj000000000000000000112024-04-12T03:57:25.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:23.089 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.089 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.089 DEBUG:boto:Method: HEAD 2024-04-12T03:58:23.090 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.090 DEBUG:boto:Data: 2024-04-12T03:58:23.090 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.090 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.090 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.090 DEBUG:boto:Params: {} 2024-04-12T03:58:23.090 DEBUG:boto:Token: None 2024-04-12T03:58:23.090 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.090 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U2eHgUUzCC3eQ0X1+ZBS61K5HOE= 2024-04-12T03:58:23.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U2eHgUUzCC3eQ0X1+ZBS61K5HOE=', 'Content-Length': '0'} 2024-04-12T03:58:23.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfa6213c290b06e2-006618b15f-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.093 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.093 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.093 DEBUG:boto:Method: HEAD 2024-04-12T03:58:23.093 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.093 DEBUG:boto:Data: 2024-04-12T03:58:23.093 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.093 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.093 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.093 DEBUG:boto:Params: {} 2024-04-12T03:58:23.093 DEBUG:boto:Token: None 2024-04-12T03:58:23.094 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.094 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U2eHgUUzCC3eQ0X1+ZBS61K5HOE= 2024-04-12T03:58:23.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U2eHgUUzCC3eQ0X1+ZBS61K5HOE=', 'Content-Length': '0'} 2024-04-12T03:58:23.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007def7881f1e524ff-006618b15f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-12T03:58:23.097 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.097 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.097 DEBUG:boto:Method: GET 2024-04-12T03:58:23.097 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.097 DEBUG:boto:Data: 2024-04-12T03:58:23.098 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.098 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.098 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.098 DEBUG:boto:Params: {} 2024-04-12T03:58:23.098 DEBUG:boto:Token: None 2024-04-12T03:58:23.098 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.098 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Sp34/+EEraOem7j6FMsyWId+/gE= 2024-04-12T03:58:23.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Sp34/+EEraOem7j6FMsyWId+/gE=', 'Content-Length': '0'} 2024-04-12T03:58:23.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e158a49b8004516-006618b15f-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.100 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.100 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.100 DEBUG:boto:Method: GET 2024-04-12T03:58:23.101 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.101 DEBUG:boto:Data: 2024-04-12T03:58:23.101 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.101 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.101 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.101 DEBUG:boto:Params: {} 2024-04-12T03:58:23.101 DEBUG:boto:Token: None 2024-04-12T03:58:23.101 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000011 2024-04-12T03:58:23.101 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Sp34/+EEraOem7j6FMsyWId+/gE= 2024-04-12T03:58:23.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Sp34/+EEraOem7j6FMsyWId+/gE=', 'Content-Length': '0'} 2024-04-12T03:58:23.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019457a11db8cdac6-006618b15f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.104 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:23.104 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:23.104 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:23.104 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:23.104 DEBUG:boto:Method: GET 2024-04-12T03:58:23.104 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:23.104 DEBUG:boto:Data: 2024-04-12T03:58:23.104 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.104 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.104 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.105 DEBUG:boto:Params: {} 2024-04-12T03:58:23.105 DEBUG:boto:Token: None 2024-04-12T03:58:23.105 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/ 2024-04-12T03:58:23.105 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs= 2024-04-12T03:58:23.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs=', 'Content-Length': '0'} 2024-04-12T03:58:23.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcd10acc4cf7379a-006618b15f-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.107 DEBUG:boto:b'dkyges-60obj000000000000000000111000falseobj000000000000000000112024-04-12T03:57:25.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:23.108 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:23.108 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:23.108 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:23.108 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:23.108 DEBUG:boto:Method: GET 2024-04-12T03:58:23.108 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:23.108 DEBUG:boto:Data: 2024-04-12T03:58:23.108 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.109 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.109 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.109 DEBUG:boto:Params: {} 2024-04-12T03:58:23.109 DEBUG:boto:Token: None 2024-04-12T03:58:23.109 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/ 2024-04-12T03:58:23.109 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs= 2024-04-12T03:58:23.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs=', 'Content-Length': '0'} 2024-04-12T03:58:23.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e08956e4af2dc400-006618b15f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.112 DEBUG:boto:b'dkyges-60obj000000000000000000111000falseobj000000000000000000112024-04-12T03:57:25.655Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:23.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-12T03:58:23.112 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.112 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.112 DEBUG:boto:path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.112 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.113 DEBUG:boto:Method: GET 2024-04-12T03:58:23.113 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.113 DEBUG:boto:Data: 2024-04-12T03:58:23.113 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.113 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.113 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.113 DEBUG:boto:Params: {} 2024-04-12T03:58:23.113 DEBUG:boto:Token: None 2024-04-12T03:58:23.113 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.113 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kDEN4tOusYfq3EzyV4l7MlNvUcM= 2024-04-12T03:58:23.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kDEN4tOusYfq3EzyV4l7MlNvUcM=', 'Content-Length': '0'} 2024-04-12T03:58:23.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019fcee1970f43c2d-006618b15f-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.116 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.116 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.116 DEBUG:boto:path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.116 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.116 DEBUG:boto:Method: GET 2024-04-12T03:58:23.116 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.116 DEBUG:boto:Data: 2024-04-12T03:58:23.116 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.116 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.116 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.116 DEBUG:boto:Params: {} 2024-04-12T03:58:23.117 DEBUG:boto:Token: None 2024-04-12T03:58:23.117 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.117 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kDEN4tOusYfq3EzyV4l7MlNvUcM= 2024-04-12T03:58:23.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kDEN4tOusYfq3EzyV4l7MlNvUcM=', 'Content-Length': '0'} 2024-04-12T03:58:23.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018dbd3a3acd8748e-006618b15f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.163 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.163 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.163 DEBUG:boto:path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.163 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.163 DEBUG:boto:Method: HEAD 2024-04-12T03:58:23.163 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.163 DEBUG:boto:Data: 2024-04-12T03:58:23.164 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.164 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.164 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.164 DEBUG:boto:Params: {} 2024-04-12T03:58:23.164 DEBUG:boto:Token: None 2024-04-12T03:58:23.164 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.164 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/HnQ/Xo4x/8+YnJfn+muLifbDDs= 2024-04-12T03:58:23.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/HnQ/Xo4x/8+YnJfn+muLifbDDs=', 'Content-Length': '0'} 2024-04-12T03:58:23.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a8cadee89adac19-006618b15f-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.166 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.166 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.166 DEBUG:boto:path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.167 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.167 DEBUG:boto:Method: HEAD 2024-04-12T03:58:23.167 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.167 DEBUG:boto:Data: 2024-04-12T03:58:23.167 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.167 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.167 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.167 DEBUG:boto:Params: {} 2024-04-12T03:58:23.167 DEBUG:boto:Token: None 2024-04-12T03:58:23.167 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.168 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/HnQ/Xo4x/8+YnJfn+muLifbDDs= 2024-04-12T03:58:23.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/HnQ/Xo4x/8+YnJfn+muLifbDDs=', 'Content-Length': '0'} 2024-04-12T03:58:23.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b857c76c3494a77-006618b15f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-12T03:58:23.170 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.170 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.170 DEBUG:boto:path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.170 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.170 DEBUG:boto:Method: GET 2024-04-12T03:58:23.170 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.170 DEBUG:boto:Data: 2024-04-12T03:58:23.170 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.170 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.170 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.171 DEBUG:boto:Params: {} 2024-04-12T03:58:23.171 DEBUG:boto:Token: None 2024-04-12T03:58:23.171 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.171 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kDEN4tOusYfq3EzyV4l7MlNvUcM= 2024-04-12T03:58:23.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kDEN4tOusYfq3EzyV4l7MlNvUcM=', 'Content-Length': '0'} 2024-04-12T03:58:23.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d547c550e0bae7ac-006618b15f-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.174 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.174 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.174 DEBUG:boto:path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.174 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.174 DEBUG:boto:Method: GET 2024-04-12T03:58:23.174 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.174 DEBUG:boto:Data: 2024-04-12T03:58:23.175 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.175 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.175 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.175 DEBUG:boto:Params: {} 2024-04-12T03:58:23.175 DEBUG:boto:Token: None 2024-04-12T03:58:23.175 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:23.175 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kDEN4tOusYfq3EzyV4l7MlNvUcM= 2024-04-12T03:58:23.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kDEN4tOusYfq3EzyV4l7MlNvUcM=', 'Content-Length': '0'} 2024-04-12T03:58:23.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1bed48fe7f299ab-006618b15f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.177 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:23.177 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:23.177 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:23.177 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:23.178 DEBUG:boto:Method: GET 2024-04-12T03:58:23.178 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:23.178 DEBUG:boto:Data: 2024-04-12T03:58:23.178 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.178 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.178 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.178 DEBUG:boto:Params: {} 2024-04-12T03:58:23.178 DEBUG:boto:Token: None 2024-04-12T03:58:23.178 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/ 2024-04-12T03:58:23.178 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs= 2024-04-12T03:58:23.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs=', 'Content-Length': '0'} 2024-04-12T03:58:23.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093619ce30d81e555-006618b15f-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.181 DEBUG:boto:b'dkyges-60obj000000000000000000121000falseobj000000000000000000122024-04-12T03:57:25.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:23.182 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:23.182 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:23.182 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:23.182 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:23.182 DEBUG:boto:Method: GET 2024-04-12T03:58:23.182 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:23.182 DEBUG:boto:Data: 2024-04-12T03:58:23.182 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.183 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.183 DEBUG:boto:Params: {} 2024-04-12T03:58:23.183 DEBUG:boto:Token: None 2024-04-12T03:58:23.183 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/ 2024-04-12T03:58:23.183 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs= 2024-04-12T03:58:23.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs=', 'Content-Length': '0'} 2024-04-12T03:58:23.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025fda063c463f186-006618b15f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.186 DEBUG:boto:b'dkyges-60obj000000000000000000121000falseobj000000000000000000122024-04-12T03:57:25.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:23.186 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.186 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.186 DEBUG:boto:Method: HEAD 2024-04-12T03:58:23.186 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.187 DEBUG:boto:Data: 2024-04-12T03:58:23.187 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.187 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.187 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.187 DEBUG:boto:Params: {} 2024-04-12T03:58:23.187 DEBUG:boto:Token: None 2024-04-12T03:58:23.187 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.187 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t1ycuO6/UgliEN63HX9SZG+goUI= 2024-04-12T03:58:23.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t1ycuO6/UgliEN63HX9SZG+goUI=', 'Content-Length': '0'} 2024-04-12T03:58:23.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070a97b054af31be5-006618b15f-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.189 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.189 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.190 DEBUG:boto:Method: HEAD 2024-04-12T03:58:23.190 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.190 DEBUG:boto:Data: 2024-04-12T03:58:23.190 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.190 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.190 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.190 DEBUG:boto:Params: {} 2024-04-12T03:58:23.190 DEBUG:boto:Token: None 2024-04-12T03:58:23.190 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.190 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t1ycuO6/UgliEN63HX9SZG+goUI= 2024-04-12T03:58:23.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t1ycuO6/UgliEN63HX9SZG+goUI=', 'Content-Length': '0'} 2024-04-12T03:58:23.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1e8c26208c31a01-006618b15f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-12T03:58:23.192 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.192 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.193 DEBUG:boto:Method: GET 2024-04-12T03:58:23.193 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.193 DEBUG:boto:Data: 2024-04-12T03:58:23.193 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.193 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.193 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.193 DEBUG:boto:Params: {} 2024-04-12T03:58:23.193 DEBUG:boto:Token: None 2024-04-12T03:58:23.193 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.193 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jmxPTF3itobW+lJF4wioipaVuFU= 2024-04-12T03:58:23.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jmxPTF3itobW+lJF4wioipaVuFU=', 'Content-Length': '0'} 2024-04-12T03:58:23.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad4bae38d20a466c-006618b15f-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.196 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.197 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.197 DEBUG:boto:Method: GET 2024-04-12T03:58:23.197 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.197 DEBUG:boto:Data: 2024-04-12T03:58:23.197 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.197 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.197 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.197 DEBUG:boto:Params: {} 2024-04-12T03:58:23.197 DEBUG:boto:Token: None 2024-04-12T03:58:23.197 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000012 2024-04-12T03:58:23.197 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jmxPTF3itobW+lJF4wioipaVuFU= 2024-04-12T03:58:23.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jmxPTF3itobW+lJF4wioipaVuFU=', 'Content-Length': '0'} 2024-04-12T03:58:23.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1b8747302ebe9d9-006618b15f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.199 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:23.200 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:23.200 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:23.200 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:23.200 DEBUG:boto:Method: GET 2024-04-12T03:58:23.200 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:23.200 DEBUG:boto:Data: 2024-04-12T03:58:23.200 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.200 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.200 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.200 DEBUG:boto:Params: {} 2024-04-12T03:58:23.201 DEBUG:boto:Token: None 2024-04-12T03:58:23.201 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/ 2024-04-12T03:58:23.201 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs= 2024-04-12T03:58:23.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs=', 'Content-Length': '0'} 2024-04-12T03:58:23.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003470ecebf1e1f94e-006618b15f-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.250 DEBUG:boto:b'dkyges-60obj000000000000000000121000falseobj000000000000000000122024-04-12T03:57:25.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:23.251 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:23.251 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:23.251 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:23.251 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:23.251 DEBUG:boto:Method: GET 2024-04-12T03:58:23.251 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:23.251 DEBUG:boto:Data: 2024-04-12T03:58:23.251 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.251 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.251 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.251 DEBUG:boto:Params: {} 2024-04-12T03:58:23.252 DEBUG:boto:Token: None 2024-04-12T03:58:23.252 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/ 2024-04-12T03:58:23.252 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs= 2024-04-12T03:58:23.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs=', 'Content-Length': '0'} 2024-04-12T03:58:23.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2f22f2c88281cb1-006618b15f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.255 DEBUG:boto:b'dkyges-60obj000000000000000000121000falseobj000000000000000000122024-04-12T03:57:25.644Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:23.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-12T03:58:23.255 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.255 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.255 DEBUG:boto:path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.256 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.256 DEBUG:boto:Method: GET 2024-04-12T03:58:23.256 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.256 DEBUG:boto:Data: 2024-04-12T03:58:23.256 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.256 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.256 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.256 DEBUG:boto:Params: {} 2024-04-12T03:58:23.256 DEBUG:boto:Token: None 2024-04-12T03:58:23.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.257 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RTiRXn8MV1qpZ1+bV+Q+ZqVKfA0= 2024-04-12T03:58:23.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RTiRXn8MV1qpZ1+bV+Q+ZqVKfA0=', 'Content-Length': '0'} 2024-04-12T03:58:23.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064b027613f13b48f-006618b15f-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.259 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.259 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.259 DEBUG:boto:path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.259 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.259 DEBUG:boto:Method: GET 2024-04-12T03:58:23.259 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.259 DEBUG:boto:Data: 2024-04-12T03:58:23.259 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.260 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.260 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.260 DEBUG:boto:Params: {} 2024-04-12T03:58:23.260 DEBUG:boto:Token: None 2024-04-12T03:58:23.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.260 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RTiRXn8MV1qpZ1+bV+Q+ZqVKfA0= 2024-04-12T03:58:23.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RTiRXn8MV1qpZ1+bV+Q+ZqVKfA0=', 'Content-Length': '0'} 2024-04-12T03:58:23.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049c5a5fef714cafa-006618b15f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.262 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.263 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.263 DEBUG:boto:path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.263 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.263 DEBUG:boto:Method: HEAD 2024-04-12T03:58:23.263 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.263 DEBUG:boto:Data: 2024-04-12T03:58:23.263 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.263 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.263 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.263 DEBUG:boto:Params: {} 2024-04-12T03:58:23.263 DEBUG:boto:Token: None 2024-04-12T03:58:23.264 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.264 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KZH+XDzjJ0M4oprT/OibIXAtbDQ= 2024-04-12T03:58:23.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KZH+XDzjJ0M4oprT/OibIXAtbDQ=', 'Content-Length': '0'} 2024-04-12T03:58:23.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ba100f3a008e996-006618b15f-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.266 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.266 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.266 DEBUG:boto:path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.266 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.266 DEBUG:boto:Method: HEAD 2024-04-12T03:58:23.266 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.266 DEBUG:boto:Data: 2024-04-12T03:58:23.266 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.266 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.266 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.266 DEBUG:boto:Params: {} 2024-04-12T03:58:23.267 DEBUG:boto:Token: None 2024-04-12T03:58:23.267 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.267 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KZH+XDzjJ0M4oprT/OibIXAtbDQ= 2024-04-12T03:58:23.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KZH+XDzjJ0M4oprT/OibIXAtbDQ=', 'Content-Length': '0'} 2024-04-12T03:58:23.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000880799bbb5dfb9c5-006618b15f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-12T03:58:23.269 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.269 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.269 DEBUG:boto:path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.269 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.269 DEBUG:boto:Method: GET 2024-04-12T03:58:23.269 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.269 DEBUG:boto:Data: 2024-04-12T03:58:23.269 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.269 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.269 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.269 DEBUG:boto:Params: {} 2024-04-12T03:58:23.269 DEBUG:boto:Token: None 2024-04-12T03:58:23.270 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.270 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RTiRXn8MV1qpZ1+bV+Q+ZqVKfA0= 2024-04-12T03:58:23.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RTiRXn8MV1qpZ1+bV+Q+ZqVKfA0=', 'Content-Length': '0'} 2024-04-12T03:58:23.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000213e5ed3068e0763-006618b15f-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.272 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.272 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.272 DEBUG:boto:path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.272 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.272 DEBUG:boto:Method: GET 2024-04-12T03:58:23.272 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.272 DEBUG:boto:Data: 2024-04-12T03:58:23.272 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.273 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.273 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.273 DEBUG:boto:Params: {} 2024-04-12T03:58:23.273 DEBUG:boto:Token: None 2024-04-12T03:58:23.273 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:23.273 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RTiRXn8MV1qpZ1+bV+Q+ZqVKfA0= 2024-04-12T03:58:23.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RTiRXn8MV1qpZ1+bV+Q+ZqVKfA0=', 'Content-Length': '0'} 2024-04-12T03:58:23.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a79652e25df1c48-006618b15f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.318 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:23.318 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:23.318 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:23.319 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:23.319 DEBUG:boto:Method: GET 2024-04-12T03:58:23.319 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:23.319 DEBUG:boto:Data: 2024-04-12T03:58:23.319 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.319 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.319 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.319 DEBUG:boto:Params: {} 2024-04-12T03:58:23.319 DEBUG:boto:Token: None 2024-04-12T03:58:23.320 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/ 2024-04-12T03:58:23.320 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs= 2024-04-12T03:58:23.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs=', 'Content-Length': '0'} 2024-04-12T03:58:23.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006487e4f599737ed3-006618b15f-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.323 DEBUG:boto:b'dkyges-60obj000000000000000000131000falseobj000000000000000000132024-04-12T03:57:25.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:23.323 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:23.323 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:23.323 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:23.323 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:23.323 DEBUG:boto:Method: GET 2024-04-12T03:58:23.323 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:23.323 DEBUG:boto:Data: 2024-04-12T03:58:23.324 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.324 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.324 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.324 DEBUG:boto:Params: {} 2024-04-12T03:58:23.324 DEBUG:boto:Token: None 2024-04-12T03:58:23.324 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/ 2024-04-12T03:58:23.324 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs= 2024-04-12T03:58:23.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs=', 'Content-Length': '0'} 2024-04-12T03:58:23.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058c78e41477c0518-006618b15f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.327 DEBUG:boto:b'dkyges-60obj000000000000000000131000falseobj000000000000000000132024-04-12T03:57:25.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:23.327 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.327 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.327 DEBUG:boto:Method: HEAD 2024-04-12T03:58:23.327 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.327 DEBUG:boto:Data: 2024-04-12T03:58:23.327 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.327 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.328 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.328 DEBUG:boto:Params: {} 2024-04-12T03:58:23.328 DEBUG:boto:Token: None 2024-04-12T03:58:23.328 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.328 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UMAVhE9DZ3DrGI2M/qGQEvnOlxQ= 2024-04-12T03:58:23.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UMAVhE9DZ3DrGI2M/qGQEvnOlxQ=', 'Content-Length': '0'} 2024-04-12T03:58:23.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ee2f1052eefedd5-006618b15f-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.330 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.330 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.330 DEBUG:boto:Method: HEAD 2024-04-12T03:58:23.330 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.331 DEBUG:boto:Data: 2024-04-12T03:58:23.331 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.331 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.331 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.331 DEBUG:boto:Params: {} 2024-04-12T03:58:23.331 DEBUG:boto:Token: None 2024-04-12T03:58:23.331 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.331 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UMAVhE9DZ3DrGI2M/qGQEvnOlxQ= 2024-04-12T03:58:23.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UMAVhE9DZ3DrGI2M/qGQEvnOlxQ=', 'Content-Length': '0'} 2024-04-12T03:58:23.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3e66e14eaf356ca-006618b15f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-12T03:58:23.334 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.334 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.334 DEBUG:boto:Method: GET 2024-04-12T03:58:23.334 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.334 DEBUG:boto:Data: 2024-04-12T03:58:23.334 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.334 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.334 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.334 DEBUG:boto:Params: {} 2024-04-12T03:58:23.335 DEBUG:boto:Token: None 2024-04-12T03:58:23.335 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.335 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8+Mz0uVrRwyCfSr27SNi7xxxybk= 2024-04-12T03:58:23.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8+Mz0uVrRwyCfSr27SNi7xxxybk=', 'Content-Length': '0'} 2024-04-12T03:58:23.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ec8d19ba94fcc52-006618b15f-1c1a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.378 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.378 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.379 DEBUG:boto:Method: GET 2024-04-12T03:58:23.379 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.379 DEBUG:boto:Data: 2024-04-12T03:58:23.379 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.379 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.379 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.379 DEBUG:boto:Params: {} 2024-04-12T03:58:23.379 DEBUG:boto:Token: None 2024-04-12T03:58:23.379 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/obj00000000000000000013 2024-04-12T03:58:23.379 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8+Mz0uVrRwyCfSr27SNi7xxxybk= 2024-04-12T03:58:23.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8+Mz0uVrRwyCfSr27SNi7xxxybk=', 'Content-Length': '0'} 2024-04-12T03:58:23.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:57:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7cde9a880198dee-006618b15f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.382 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:23.382 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:23.382 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:23.382 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:23.382 DEBUG:boto:Method: GET 2024-04-12T03:58:23.382 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:23.382 DEBUG:boto:Data: 2024-04-12T03:58:23.382 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.382 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:23.382 DEBUG:boto:Port: 8001 2024-04-12T03:58:23.383 DEBUG:boto:Params: {} 2024-04-12T03:58:23.383 DEBUG:boto:Token: None 2024-04-12T03:58:23.383 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/ 2024-04-12T03:58:23.383 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs= 2024-04-12T03:58:23.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs=', 'Content-Length': '0'} 2024-04-12T03:58:23.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdd02d8c857f73b6-006618b15f-1c1a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.385 DEBUG:boto:b'dkyges-60obj000000000000000000131000falseobj000000000000000000132024-04-12T03:57:25.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:23.386 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:23.386 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:23.386 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:23.386 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:23.386 DEBUG:boto:Method: GET 2024-04-12T03:58:23.386 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:23.386 DEBUG:boto:Data: 2024-04-12T03:58:23.387 DEBUG:boto:Headers: {} 2024-04-12T03:58:23.387 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:23.387 DEBUG:boto:Port: 8000 2024-04-12T03:58:23.387 DEBUG:boto:Params: {} 2024-04-12T03:58:23.387 DEBUG:boto:Token: None 2024-04-12T03:58:23.387 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:23 GMT /dkyges-60/ 2024-04-12T03:58:23.387 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs= 2024-04-12T03:58:23.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CDZvOWj7ceZoSWZql3bWne9KQPs=', 'Content-Length': '0'} 2024-04-12T03:58:23.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004863e170893d0cde-006618b15f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:23.390 DEBUG:boto:b'dkyges-60obj000000000000000000131000falseobj000000000000000000132024-04-12T03:57:25.633Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:23.390 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-60 zones={a2, a1} 2024-04-12T03:58:23.391 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T03:58:23.391 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T03:58:23.392 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:58:23.403+0000 7fe02b731700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 87815) UID: 0 2024-04-12T03:58:23.392 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T03:58:23.403+0000 7fe0309067c0 -1 shutting down 2024-04-12T03:58:29.115 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T03:58:29.491 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T03:58:29.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:29.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:29.492 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-60 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:58:29.556 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:29.556 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:29.606 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T03:58:29.606 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-60 2024-04-12T03:58:29.611 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 10 2024-04-12T03:58:29.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:29.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:29.656 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:29.715 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:29.715 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:29.762 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:58:29.762 INFO:teuthology.orchestra.run.smithi043.stdout: "objs": [ 2024-04-12T03:58:29.762 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000007", 2024-04-12T03:58:29.762 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000016", 2024-04-12T03:58:29.762 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000024", 2024-04-12T03:58:29.762 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000020", 2024-04-12T03:58:29.762 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000028", 2024-04-12T03:58:29.762 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000000", 2024-04-12T03:58:29.763 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000032", 2024-04-12T03:58:29.763 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000001", 2024-04-12T03:58:29.763 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000010", 2024-04-12T03:58:29.763 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000019", 2024-04-12T03:58:29.763 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000013", 2024-04-12T03:58:29.763 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000021", 2024-04-12T03:58:29.763 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000030", 2024-04-12T03:58:29.763 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000026", 2024-04-12T03:58:29.763 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000033", 2024-04-12T03:58:29.763 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000004", 2024-04-12T03:58:29.763 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000076" 2024-04-12T03:58:29.764 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:58:29.764 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:58:29.765 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:29.765 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:29.765 DEBUG:boto:Method: HEAD 2024-04-12T03:58:29.765 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:29.765 DEBUG:boto:Data: 2024-04-12T03:58:29.765 DEBUG:boto:Headers: {} 2024-04-12T03:58:29.765 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:29.765 DEBUG:boto:Port: 8000 2024-04-12T03:58:29.765 DEBUG:boto:Params: {} 2024-04-12T03:58:29.766 DEBUG:boto:Token: None 2024-04-12T03:58:29.766 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:29 GMT /dkyges-60/ 2024-04-12T03:58:29.766 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GTBquG7XgbXw1r/8XeULnQv1OBQ= 2024-04-12T03:58:29.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GTBquG7XgbXw1r/8XeULnQv1OBQ=', 'Content-Length': '0'} 2024-04-12T03:58:29.769 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e07b3047ed76a07-006618b165-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:29.769 DEBUG:boto:path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:29.769 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:29.770 DEBUG:boto:Method: PUT 2024-04-12T03:58:29.770 DEBUG:boto:Path: /dkyges-60/obj00000000000000000033 2024-04-12T03:58:29.770 DEBUG:boto:Data: 2024-04-12T03:58:29.770 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:29.770 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:29.770 DEBUG:boto:Port: 8000 2024-04-12T03:58:29.770 DEBUG:boto:Params: {} 2024-04-12T03:58:29.770 DEBUG:boto:Token: None 2024-04-12T03:58:29.770 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:29 GMT /dkyges-60/obj00000000000000000033 2024-04-12T03:58:29.770 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Vj+QHPYjnAkqOsGZMxb4B7Ws84s= 2024-04-12T03:58:29.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Vj+QHPYjnAkqOsGZMxb4B7Ws84s='} 2024-04-12T03:58:29.776 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000059fe0076fdae016-006618b165-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:29.777 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:29.777 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:29.777 DEBUG:boto:Method: HEAD 2024-04-12T03:58:29.777 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:29.777 DEBUG:boto:Data: 2024-04-12T03:58:29.777 DEBUG:boto:Headers: {} 2024-04-12T03:58:29.777 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:29.777 DEBUG:boto:Port: 8000 2024-04-12T03:58:29.777 DEBUG:boto:Params: {} 2024-04-12T03:58:29.778 DEBUG:boto:Token: None 2024-04-12T03:58:29.778 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:29 GMT /dkyges-60/ 2024-04-12T03:58:29.778 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GTBquG7XgbXw1r/8XeULnQv1OBQ= 2024-04-12T03:58:29.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GTBquG7XgbXw1r/8XeULnQv1OBQ=', 'Content-Length': '0'} 2024-04-12T03:58:29.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e4b593eb0752510-006618b165-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:29.781 DEBUG:boto:path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:29.781 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:29.781 DEBUG:boto:Method: PUT 2024-04-12T03:58:29.781 DEBUG:boto:Path: /dkyges-60/obj00000000000000000026 2024-04-12T03:58:29.781 DEBUG:boto:Data: 2024-04-12T03:58:29.781 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:29.781 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:29.781 DEBUG:boto:Port: 8000 2024-04-12T03:58:29.782 DEBUG:boto:Params: {} 2024-04-12T03:58:29.782 DEBUG:boto:Token: None 2024-04-12T03:58:29.782 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:29 GMT /dkyges-60/obj00000000000000000026 2024-04-12T03:58:29.782 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fgBMn0mUA1JvAnXU2rhySdW2pEw= 2024-04-12T03:58:29.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fgBMn0mUA1JvAnXU2rhySdW2pEw='} 2024-04-12T03:58:29.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d418a434eb2a005d-006618b165-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:29.827 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:29.827 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:29.827 DEBUG:boto:Method: HEAD 2024-04-12T03:58:29.827 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:29.827 DEBUG:boto:Data: 2024-04-12T03:58:29.827 DEBUG:boto:Headers: {} 2024-04-12T03:58:29.827 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:29.827 DEBUG:boto:Port: 8000 2024-04-12T03:58:29.827 DEBUG:boto:Params: {} 2024-04-12T03:58:29.827 DEBUG:boto:Token: None 2024-04-12T03:58:29.828 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:29 GMT /dkyges-60/ 2024-04-12T03:58:29.828 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GTBquG7XgbXw1r/8XeULnQv1OBQ= 2024-04-12T03:58:29.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GTBquG7XgbXw1r/8XeULnQv1OBQ=', 'Content-Length': '0'} 2024-04-12T03:58:29.831 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f73d4f7bd76878a-006618b165-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:29.831 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:29.831 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:29.831 DEBUG:boto:Method: PUT 2024-04-12T03:58:29.831 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000 2024-04-12T03:58:29.832 DEBUG:boto:Data: 2024-04-12T03:58:29.832 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:29.832 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:29.832 DEBUG:boto:Port: 8000 2024-04-12T03:58:29.832 DEBUG:boto:Params: {} 2024-04-12T03:58:29.832 DEBUG:boto:Token: None 2024-04-12T03:58:29.832 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:29 GMT /dkyges-60/obj00000000000000000000 2024-04-12T03:58:29.832 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZM+oGgBpN/895giCMBTw1kWy+f0= 2024-04-12T03:58:29.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZM+oGgBpN/895giCMBTw1kWy+f0='} 2024-04-12T03:58:29.839 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000676e1c53be549cf8-006618b165-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:29.840 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:29.840 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:29.840 DEBUG:boto:Method: HEAD 2024-04-12T03:58:29.840 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:29.840 DEBUG:boto:Data: 2024-04-12T03:58:29.840 DEBUG:boto:Headers: {} 2024-04-12T03:58:29.840 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:29.840 DEBUG:boto:Port: 8000 2024-04-12T03:58:29.840 DEBUG:boto:Params: {} 2024-04-12T03:58:29.841 DEBUG:boto:Token: None 2024-04-12T03:58:29.841 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:29 GMT /dkyges-60/ 2024-04-12T03:58:29.841 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GTBquG7XgbXw1r/8XeULnQv1OBQ= 2024-04-12T03:58:29.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GTBquG7XgbXw1r/8XeULnQv1OBQ=', 'Content-Length': '0'} 2024-04-12T03:58:29.844 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000756acef598c93425-006618b165-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:29.844 DEBUG:boto:path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:29.844 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:29.844 DEBUG:boto:Method: PUT 2024-04-12T03:58:29.844 DEBUG:boto:Path: /dkyges-60/obj00000000000000000016 2024-04-12T03:58:29.844 DEBUG:boto:Data: 2024-04-12T03:58:29.844 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:29.844 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:29.844 DEBUG:boto:Port: 8000 2024-04-12T03:58:29.844 DEBUG:boto:Params: {} 2024-04-12T03:58:29.845 DEBUG:boto:Token: None 2024-04-12T03:58:29.845 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:29 GMT /dkyges-60/obj00000000000000000016 2024-04-12T03:58:29.845 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mGVz8RRcrw4FDW9r1Uj9kbhO3b0= 2024-04-12T03:58:29.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mGVz8RRcrw4FDW9r1Uj9kbhO3b0='} 2024-04-12T03:58:29.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba457b8de6522d4d-006618b165-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:29.887 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:29.887 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:29.887 DEBUG:boto:Method: HEAD 2024-04-12T03:58:29.887 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:29.887 DEBUG:boto:Data: 2024-04-12T03:58:29.887 DEBUG:boto:Headers: {} 2024-04-12T03:58:29.887 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:29.887 DEBUG:boto:Port: 8000 2024-04-12T03:58:29.887 DEBUG:boto:Params: {} 2024-04-12T03:58:29.887 DEBUG:boto:Token: None 2024-04-12T03:58:29.888 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:29 GMT /dkyges-60/ 2024-04-12T03:58:29.888 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GTBquG7XgbXw1r/8XeULnQv1OBQ= 2024-04-12T03:58:29.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GTBquG7XgbXw1r/8XeULnQv1OBQ=', 'Content-Length': '0'} 2024-04-12T03:58:29.890 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000403e789089c213c3-006618b165-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:29.891 DEBUG:boto:path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:29.891 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:29.891 DEBUG:boto:Method: PUT 2024-04-12T03:58:29.891 DEBUG:boto:Path: /dkyges-60/obj00000000000000000024 2024-04-12T03:58:29.891 DEBUG:boto:Data: 2024-04-12T03:58:29.891 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:29.891 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:29.891 DEBUG:boto:Port: 8000 2024-04-12T03:58:29.891 DEBUG:boto:Params: {} 2024-04-12T03:58:29.891 DEBUG:boto:Token: None 2024-04-12T03:58:29.892 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:29 GMT /dkyges-60/obj00000000000000000024 2024-04-12T03:58:29.892 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t56erHisJv4yv53f6m4/mKmnNJo= 2024-04-12T03:58:29.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t56erHisJv4yv53f6m4/mKmnNJo='} 2024-04-12T03:58:29.898 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001a9cb2d2e0ce06f0-006618b165-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:29.898 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:29.899 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:29.899 DEBUG:boto:Method: HEAD 2024-04-12T03:58:29.899 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:29.899 DEBUG:boto:Data: 2024-04-12T03:58:29.899 DEBUG:boto:Headers: {} 2024-04-12T03:58:29.899 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:29.899 DEBUG:boto:Port: 8000 2024-04-12T03:58:29.899 DEBUG:boto:Params: {} 2024-04-12T03:58:29.899 DEBUG:boto:Token: None 2024-04-12T03:58:29.899 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:29 GMT /dkyges-60/ 2024-04-12T03:58:29.899 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GTBquG7XgbXw1r/8XeULnQv1OBQ= 2024-04-12T03:58:29.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GTBquG7XgbXw1r/8XeULnQv1OBQ=', 'Content-Length': '0'} 2024-04-12T03:58:29.902 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7691e0ccec726ac-006618b165-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:29.902 DEBUG:boto:path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:29.903 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:29.903 DEBUG:boto:Method: PUT 2024-04-12T03:58:29.903 DEBUG:boto:Path: /dkyges-60/obj00000000000000000019 2024-04-12T03:58:29.903 DEBUG:boto:Data: 2024-04-12T03:58:29.903 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:29.903 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:29.903 DEBUG:boto:Port: 8000 2024-04-12T03:58:29.903 DEBUG:boto:Params: {} 2024-04-12T03:58:29.903 DEBUG:boto:Token: None 2024-04-12T03:58:29.903 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:29 GMT /dkyges-60/obj00000000000000000019 2024-04-12T03:58:29.903 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qcOl0Po2BpFgm0e4XO4GQU9Xnls= 2024-04-12T03:58:29.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qcOl0Po2BpFgm0e4XO4GQU9Xnls='} 2024-04-12T03:58:29.946 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1e8f6ff63bf8c28-006618b165-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:29.947 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:29.947 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:29.947 DEBUG:boto:Method: HEAD 2024-04-12T03:58:29.947 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:29.947 DEBUG:boto:Data: 2024-04-12T03:58:29.947 DEBUG:boto:Headers: {} 2024-04-12T03:58:29.947 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:29.947 DEBUG:boto:Port: 8000 2024-04-12T03:58:29.947 DEBUG:boto:Params: {} 2024-04-12T03:58:29.947 DEBUG:boto:Token: None 2024-04-12T03:58:29.947 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:29 GMT /dkyges-60/ 2024-04-12T03:58:29.948 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GTBquG7XgbXw1r/8XeULnQv1OBQ= 2024-04-12T03:58:29.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GTBquG7XgbXw1r/8XeULnQv1OBQ=', 'Content-Length': '0'} 2024-04-12T03:58:29.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bdf6fcf94cdb9caa-006618b165-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:29.951 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:29.951 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:29.951 DEBUG:boto:Method: PUT 2024-04-12T03:58:29.951 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004 2024-04-12T03:58:29.951 DEBUG:boto:Data: 2024-04-12T03:58:29.952 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:29.952 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:29.952 DEBUG:boto:Port: 8000 2024-04-12T03:58:29.952 DEBUG:boto:Params: {} 2024-04-12T03:58:29.952 DEBUG:boto:Token: None 2024-04-12T03:58:29.952 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:29 GMT /dkyges-60/obj00000000000000000004 2024-04-12T03:58:29.952 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mwOCUBdVjliGBldyvDxnZWLjul0= 2024-04-12T03:58:29.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mwOCUBdVjliGBldyvDxnZWLjul0='} 2024-04-12T03:58:29.960 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b03b120eab5365a5-006618b165-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:29.960 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:29.960 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:29.961 DEBUG:boto:Method: HEAD 2024-04-12T03:58:29.961 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:29.961 DEBUG:boto:Data: 2024-04-12T03:58:29.961 DEBUG:boto:Headers: {} 2024-04-12T03:58:29.961 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:29.961 DEBUG:boto:Port: 8000 2024-04-12T03:58:29.961 DEBUG:boto:Params: {} 2024-04-12T03:58:29.961 DEBUG:boto:Token: None 2024-04-12T03:58:29.961 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:29 GMT /dkyges-60/ 2024-04-12T03:58:29.961 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GTBquG7XgbXw1r/8XeULnQv1OBQ= 2024-04-12T03:58:29.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GTBquG7XgbXw1r/8XeULnQv1OBQ=', 'Content-Length': '0'} 2024-04-12T03:58:29.964 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014dc6d2ed396f3a2-006618b165-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:29.965 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:29.965 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:29.965 DEBUG:boto:Method: PUT 2024-04-12T03:58:29.965 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010 2024-04-12T03:58:29.965 DEBUG:boto:Data: 2024-04-12T03:58:29.965 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:29.965 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:29.965 DEBUG:boto:Port: 8000 2024-04-12T03:58:29.965 DEBUG:boto:Params: {} 2024-04-12T03:58:29.965 DEBUG:boto:Token: None 2024-04-12T03:58:29.966 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:29 GMT /dkyges-60/obj00000000000000000010 2024-04-12T03:58:29.966 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:w5nQd48GH5KDg3i1cku/sKO82G0= 2024-04-12T03:58:29.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:w5nQd48GH5KDg3i1cku/sKO82G0='} 2024-04-12T03:58:30.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ca767fad299ce2f-006618b165-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.011 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.011 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.011 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.011 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.011 DEBUG:boto:Data: 2024-04-12T03:58:30.011 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.011 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.011 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.012 DEBUG:boto:Params: {} 2024-04-12T03:58:30.012 DEBUG:boto:Token: None 2024-04-12T03:58:30.012 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.012 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000367c5dc9e3f23593-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.016 DEBUG:boto:path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:30.016 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:30.016 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.016 DEBUG:boto:Path: /dkyges-60/obj00000000000000000032 2024-04-12T03:58:30.016 DEBUG:boto:Data: 2024-04-12T03:58:30.016 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.016 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.016 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.016 DEBUG:boto:Params: {} 2024-04-12T03:58:30.016 DEBUG:boto:Token: None 2024-04-12T03:58:30.016 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000032 2024-04-12T03:58:30.017 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OnH0BmRh+7tIhr5XxXv9A1Jl6nU= 2024-04-12T03:58:30.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OnH0BmRh+7tIhr5XxXv9A1Jl6nU='} 2024-04-12T03:58:30.023 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000024601ba031eb84db-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.023 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.023 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.023 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.024 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.024 DEBUG:boto:Data: 2024-04-12T03:58:30.024 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.024 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.024 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.024 DEBUG:boto:Params: {} 2024-04-12T03:58:30.024 DEBUG:boto:Token: None 2024-04-12T03:58:30.024 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.024 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.027 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059edd462b26f7075-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.028 DEBUG:boto:path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:30.028 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:30.028 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.028 DEBUG:boto:Path: /dkyges-60/obj00000000000000000021 2024-04-12T03:58:30.028 DEBUG:boto:Data: 2024-04-12T03:58:30.028 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.028 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.028 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.028 DEBUG:boto:Params: {} 2024-04-12T03:58:30.029 DEBUG:boto:Token: None 2024-04-12T03:58:30.029 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000021 2024-04-12T03:58:30.029 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UJJOWnhKp5uhYyUB9TPpKRxWJS0= 2024-04-12T03:58:30.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UJJOWnhKp5uhYyUB9TPpKRxWJS0='} 2024-04-12T03:58:30.075 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6bafa768039d260-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.075 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.075 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.075 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.075 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.076 DEBUG:boto:Data: 2024-04-12T03:58:30.076 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.076 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.076 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.076 DEBUG:boto:Params: {} 2024-04-12T03:58:30.076 DEBUG:boto:Token: None 2024-04-12T03:58:30.076 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.076 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.079 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009554264939ed1276-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.080 DEBUG:boto:path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:30.080 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:30.080 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.080 DEBUG:boto:Path: /dkyges-60/obj00000000000000000007 2024-04-12T03:58:30.080 DEBUG:boto:Data: 2024-04-12T03:58:30.080 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.080 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.080 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.080 DEBUG:boto:Params: {} 2024-04-12T03:58:30.080 DEBUG:boto:Token: None 2024-04-12T03:58:30.081 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000007 2024-04-12T03:58:30.081 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tM/A4Jn4ymzEdPLL7C2f/aqMytQ= 2024-04-12T03:58:30.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tM/A4Jn4ymzEdPLL7C2f/aqMytQ='} 2024-04-12T03:58:30.087 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000035dfaab90707d2d-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.087 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.087 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.088 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.088 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.088 DEBUG:boto:Data: 2024-04-12T03:58:30.088 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.088 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.088 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.088 DEBUG:boto:Params: {} 2024-04-12T03:58:30.088 DEBUG:boto:Token: None 2024-04-12T03:58:30.089 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.089 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.092 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfe835abcc9959c1-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.092 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:30.092 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:30.092 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.092 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013 2024-04-12T03:58:30.092 DEBUG:boto:Data: 2024-04-12T03:58:30.092 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.092 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.092 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.093 DEBUG:boto:Params: {} 2024-04-12T03:58:30.093 DEBUG:boto:Token: None 2024-04-12T03:58:30.093 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000013 2024-04-12T03:58:30.093 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MjQHIk/W/BK81wK2KaDVkIijPAg= 2024-04-12T03:58:30.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MjQHIk/W/BK81wK2KaDVkIijPAg='} 2024-04-12T03:58:30.138 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c865318c359e5482-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.139 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.139 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.139 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.139 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.139 DEBUG:boto:Data: 2024-04-12T03:58:30.139 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.139 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.139 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.139 DEBUG:boto:Params: {} 2024-04-12T03:58:30.139 DEBUG:boto:Token: None 2024-04-12T03:58:30.140 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.140 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.142 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009f0cbda97ec0881-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.143 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:30.143 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:30.143 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.143 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001 2024-04-12T03:58:30.143 DEBUG:boto:Data: 2024-04-12T03:58:30.143 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.143 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.143 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.143 DEBUG:boto:Params: {} 2024-04-12T03:58:30.143 DEBUG:boto:Token: None 2024-04-12T03:58:30.144 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000001 2024-04-12T03:58:30.144 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0r+v4mB0qgr0Bopbd8GBsf12i6o= 2024-04-12T03:58:30.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0r+v4mB0qgr0Bopbd8GBsf12i6o='} 2024-04-12T03:58:30.152 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b90cfe952415664-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.152 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.153 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.153 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.153 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.153 DEBUG:boto:Data: 2024-04-12T03:58:30.153 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.153 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.153 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.153 DEBUG:boto:Params: {} 2024-04-12T03:58:30.153 DEBUG:boto:Token: None 2024-04-12T03:58:30.153 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.154 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.156 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000abbca90e2f0dc564-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.157 DEBUG:boto:path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:30.157 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:30.157 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.157 DEBUG:boto:Path: /dkyges-60/obj00000000000000000030 2024-04-12T03:58:30.157 DEBUG:boto:Data: 2024-04-12T03:58:30.157 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.157 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.157 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.157 DEBUG:boto:Params: {} 2024-04-12T03:58:30.158 DEBUG:boto:Token: None 2024-04-12T03:58:30.158 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000030 2024-04-12T03:58:30.158 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YLLwkbVo1Mu3vlJDsZFUO6yH/BE= 2024-04-12T03:58:30.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YLLwkbVo1Mu3vlJDsZFUO6yH/BE='} 2024-04-12T03:58:30.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ea3b76e42320986-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.207 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.207 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.207 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.207 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.207 DEBUG:boto:Data: 2024-04-12T03:58:30.207 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.207 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.207 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.207 DEBUG:boto:Params: {} 2024-04-12T03:58:30.207 DEBUG:boto:Token: None 2024-04-12T03:58:30.208 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.210 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006292c08316f05c38-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.211 DEBUG:boto:path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:30.211 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:30.211 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.211 DEBUG:boto:Path: /dkyges-60/obj00000000000000000028 2024-04-12T03:58:30.211 DEBUG:boto:Data: 2024-04-12T03:58:30.211 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.211 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.211 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.211 DEBUG:boto:Params: {} 2024-04-12T03:58:30.211 DEBUG:boto:Token: None 2024-04-12T03:58:30.212 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000028 2024-04-12T03:58:30.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wWc7rQZAYR0RQLSBtGZpo1RQ1Jo= 2024-04-12T03:58:30.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wWc7rQZAYR0RQLSBtGZpo1RQ1Jo='} 2024-04-12T03:58:30.218 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003005588c9ddc32aa-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.218 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.218 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.218 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.218 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.219 DEBUG:boto:Data: 2024-04-12T03:58:30.219 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.219 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.219 DEBUG:boto:Params: {} 2024-04-12T03:58:30.219 DEBUG:boto:Token: None 2024-04-12T03:58:30.219 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.219 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.222 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009169f02fbe8a5e93-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.222 DEBUG:boto:path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:30.222 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:30.222 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.223 DEBUG:boto:Path: /dkyges-60/obj00000000000000000076 2024-04-12T03:58:30.223 DEBUG:boto:Data: 2024-04-12T03:58:30.223 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.223 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.223 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.223 DEBUG:boto:Params: {} 2024-04-12T03:58:30.223 DEBUG:boto:Token: None 2024-04-12T03:58:30.223 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000076 2024-04-12T03:58:30.223 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xMijookPGr35fBRc5lZ0TTPHGcY= 2024-04-12T03:58:30.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xMijookPGr35fBRc5lZ0TTPHGcY='} 2024-04-12T03:58:30.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002a239b0466a0139-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:30.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:30.267 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-60 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:58:30.333 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:30.334 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:30.386 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T03:58:30.386 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-60 2024-04-12T03:58:30.394 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 21 2024-04-12T03:58:30.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:30.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:30.448 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:30.504 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:30.505 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:30.553 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:58:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "objs": [ 2024-04-12T03:58:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000005", 2024-04-12T03:58:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000010", 2024-04-12T03:58:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000016", 2024-04-12T03:58:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000021", 2024-04-12T03:58:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000004", 2024-04-12T03:58:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000025", 2024-04-12T03:58:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000030", 2024-04-12T03:58:30.554 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000015", 2024-04-12T03:58:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000019", 2024-04-12T03:58:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000003", 2024-04-12T03:58:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000009", 2024-04-12T03:58:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000014", 2024-04-12T03:58:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000012", 2024-04-12T03:58:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000002", 2024-04-12T03:58:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000000", 2024-04-12T03:58:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000006", 2024-04-12T03:58:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000011", 2024-04-12T03:58:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000001", 2024-04-12T03:58:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000007" 2024-04-12T03:58:30.555 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:58:30.556 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:58:30.556 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.557 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.557 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.557 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.557 DEBUG:boto:Data: 2024-04-12T03:58:30.557 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.557 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.557 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.557 DEBUG:boto:Params: {} 2024-04-12T03:58:30.557 DEBUG:boto:Token: None 2024-04-12T03:58:30.558 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.558 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.561 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002aa1b190909af44b-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.561 DEBUG:boto:path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:30.561 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:30.562 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.562 DEBUG:boto:Path: /dkyges-60/obj00000000000000000016 2024-04-12T03:58:30.562 DEBUG:boto:Data: 2024-04-12T03:58:30.562 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.562 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.562 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.562 DEBUG:boto:Params: {} 2024-04-12T03:58:30.562 DEBUG:boto:Token: None 2024-04-12T03:58:30.562 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000016 2024-04-12T03:58:30.562 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vjEcVzX2SBkQZ8fjudzUp7FZhts= 2024-04-12T03:58:30.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vjEcVzX2SBkQZ8fjudzUp7FZhts='} 2024-04-12T03:58:30.569 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001723cd36825c8ea3-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.570 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.570 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.570 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.570 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.570 DEBUG:boto:Data: 2024-04-12T03:58:30.570 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.570 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.570 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.570 DEBUG:boto:Params: {} 2024-04-12T03:58:30.570 DEBUG:boto:Token: None 2024-04-12T03:58:30.571 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.571 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.573 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000784e2a204628928a-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.574 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:30.574 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:30.574 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.574 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006 2024-04-12T03:58:30.574 DEBUG:boto:Data: 2024-04-12T03:58:30.574 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.574 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.574 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.574 DEBUG:boto:Params: {} 2024-04-12T03:58:30.575 DEBUG:boto:Token: None 2024-04-12T03:58:30.575 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000006 2024-04-12T03:58:30.575 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IlkqE12jXnR4GjM4KOOrLR2is1E= 2024-04-12T03:58:30.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IlkqE12jXnR4GjM4KOOrLR2is1E='} 2024-04-12T03:58:30.581 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a75ab929cd090c1-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.581 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.582 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.582 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.582 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.582 DEBUG:boto:Data: 2024-04-12T03:58:30.582 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.582 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.582 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.582 DEBUG:boto:Params: {} 2024-04-12T03:58:30.582 DEBUG:boto:Token: None 2024-04-12T03:58:30.582 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.582 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba22846b77c015b0-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.585 DEBUG:boto:path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:30.585 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:30.585 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.586 DEBUG:boto:Path: /dkyges-60/obj00000000000000000014 2024-04-12T03:58:30.586 DEBUG:boto:Data: 2024-04-12T03:58:30.586 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.586 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.586 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.586 DEBUG:boto:Params: {} 2024-04-12T03:58:30.586 DEBUG:boto:Token: None 2024-04-12T03:58:30.586 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000014 2024-04-12T03:58:30.586 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LWY7svD0mEDd+ekYcW/MiOx5Qg8= 2024-04-12T03:58:30.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LWY7svD0mEDd+ekYcW/MiOx5Qg8='} 2024-04-12T03:58:30.593 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d33fdec63f79620e-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.593 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.593 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.593 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.593 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.593 DEBUG:boto:Data: 2024-04-12T03:58:30.593 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.593 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.593 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.593 DEBUG:boto:Params: {} 2024-04-12T03:58:30.594 DEBUG:boto:Token: None 2024-04-12T03:58:30.594 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.594 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.597 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000002b9023bf59f9dc9-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.597 DEBUG:boto:path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:30.597 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:30.597 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.597 DEBUG:boto:Path: /dkyges-60/obj00000000000000000003 2024-04-12T03:58:30.597 DEBUG:boto:Data: 2024-04-12T03:58:30.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.598 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.598 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.598 DEBUG:boto:Params: {} 2024-04-12T03:58:30.598 DEBUG:boto:Token: None 2024-04-12T03:58:30.598 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000003 2024-04-12T03:58:30.598 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:E7IOaXsKTOMjqnSCRc9zu6Boy4c= 2024-04-12T03:58:30.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:E7IOaXsKTOMjqnSCRc9zu6Boy4c='} 2024-04-12T03:58:30.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009cdc93ea37762dba-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.643 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.643 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.643 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.643 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.643 DEBUG:boto:Data: 2024-04-12T03:58:30.643 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.643 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.643 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.643 DEBUG:boto:Params: {} 2024-04-12T03:58:30.643 DEBUG:boto:Token: None 2024-04-12T03:58:30.643 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.644 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.647 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ad09eff61ac07b1-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.647 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:30.647 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:30.647 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.647 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000 2024-04-12T03:58:30.647 DEBUG:boto:Data: 2024-04-12T03:58:30.647 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.648 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.648 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.648 DEBUG:boto:Params: {} 2024-04-12T03:58:30.648 DEBUG:boto:Token: None 2024-04-12T03:58:30.648 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000000 2024-04-12T03:58:30.648 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZeORinvRHaSSx3PDjozr+UJNwqI= 2024-04-12T03:58:30.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZeORinvRHaSSx3PDjozr+UJNwqI='} 2024-04-12T03:58:30.656 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000091cd009b0c93dee6-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.656 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.656 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.657 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.657 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.657 DEBUG:boto:Data: 2024-04-12T03:58:30.657 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.657 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.657 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.657 DEBUG:boto:Params: {} 2024-04-12T03:58:30.657 DEBUG:boto:Token: None 2024-04-12T03:58:30.657 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.657 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.660 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004813382eab4f17d-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.660 DEBUG:boto:path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:30.661 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:30.661 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.661 DEBUG:boto:Path: /dkyges-60/obj00000000000000000015 2024-04-12T03:58:30.661 DEBUG:boto:Data: 2024-04-12T03:58:30.661 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.661 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.661 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.661 DEBUG:boto:Params: {} 2024-04-12T03:58:30.661 DEBUG:boto:Token: None 2024-04-12T03:58:30.661 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000015 2024-04-12T03:58:30.661 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T/+Oa06fI7E0DY67i8gNM+93cCo= 2024-04-12T03:58:30.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T/+Oa06fI7E0DY67i8gNM+93cCo='} 2024-04-12T03:58:30.707 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4462e1b65e4e116-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.707 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.707 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.707 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.707 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.707 DEBUG:boto:Data: 2024-04-12T03:58:30.707 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.707 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.707 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.707 DEBUG:boto:Params: {} 2024-04-12T03:58:30.708 DEBUG:boto:Token: None 2024-04-12T03:58:30.708 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.708 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075f14e5312d6de36-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.711 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:30.711 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:30.712 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.712 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012 2024-04-12T03:58:30.712 DEBUG:boto:Data: 2024-04-12T03:58:30.712 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.712 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.712 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.712 DEBUG:boto:Params: {} 2024-04-12T03:58:30.712 DEBUG:boto:Token: None 2024-04-12T03:58:30.712 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000012 2024-04-12T03:58:30.712 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V+k8Wmu4ikMiqjDkxS36kZ8+d8I= 2024-04-12T03:58:30.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V+k8Wmu4ikMiqjDkxS36kZ8+d8I='} 2024-04-12T03:58:30.720 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002d63dc0fb87a3cd5-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.720 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.720 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.720 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.720 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.720 DEBUG:boto:Data: 2024-04-12T03:58:30.721 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.721 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.721 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.721 DEBUG:boto:Params: {} 2024-04-12T03:58:30.721 DEBUG:boto:Token: None 2024-04-12T03:58:30.721 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.721 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.725 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000173f7683b1fe27a7-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.725 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:30.725 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:30.725 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.725 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002 2024-04-12T03:58:30.725 DEBUG:boto:Data: 2024-04-12T03:58:30.725 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.725 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.726 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.726 DEBUG:boto:Params: {} 2024-04-12T03:58:30.726 DEBUG:boto:Token: None 2024-04-12T03:58:30.726 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000002 2024-04-12T03:58:30.726 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f6N7RZYe1IvC1BcwNWQYVdpsiSw= 2024-04-12T03:58:30.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f6N7RZYe1IvC1BcwNWQYVdpsiSw='} 2024-04-12T03:58:30.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000070025195106c8f34-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.771 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.771 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.771 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.771 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.771 DEBUG:boto:Data: 2024-04-12T03:58:30.771 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.771 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.771 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.771 DEBUG:boto:Params: {} 2024-04-12T03:58:30.771 DEBUG:boto:Token: None 2024-04-12T03:58:30.772 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.772 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.775 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071e112191753c347-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.775 DEBUG:boto:path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:30.775 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:30.775 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.775 DEBUG:boto:Path: /dkyges-60/obj00000000000000000030 2024-04-12T03:58:30.775 DEBUG:boto:Data: 2024-04-12T03:58:30.776 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.776 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.776 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.776 DEBUG:boto:Params: {} 2024-04-12T03:58:30.776 DEBUG:boto:Token: None 2024-04-12T03:58:30.776 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000030 2024-04-12T03:58:30.776 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YLLwkbVo1Mu3vlJDsZFUO6yH/BE= 2024-04-12T03:58:30.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YLLwkbVo1Mu3vlJDsZFUO6yH/BE='} 2024-04-12T03:58:30.785 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000089cd41dede0c805a-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.785 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.785 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.785 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.785 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.785 DEBUG:boto:Data: 2024-04-12T03:58:30.785 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.785 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.785 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.785 DEBUG:boto:Params: {} 2024-04-12T03:58:30.786 DEBUG:boto:Token: None 2024-04-12T03:58:30.786 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.786 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.789 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000066d842336464ed79-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.789 DEBUG:boto:path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:30.789 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:30.789 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.789 DEBUG:boto:Path: /dkyges-60/obj00000000000000000019 2024-04-12T03:58:30.789 DEBUG:boto:Data: 2024-04-12T03:58:30.789 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.789 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.789 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.790 DEBUG:boto:Params: {} 2024-04-12T03:58:30.790 DEBUG:boto:Token: None 2024-04-12T03:58:30.790 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000019 2024-04-12T03:58:30.790 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:M8VMcs2XQ5NCZXxu3wCVp5GlC7E= 2024-04-12T03:58:30.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:M8VMcs2XQ5NCZXxu3wCVp5GlC7E='} 2024-04-12T03:58:30.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002fc4ddf20c7bb23f-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.835 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.835 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.835 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.835 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.835 DEBUG:boto:Data: 2024-04-12T03:58:30.835 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.835 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.835 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.835 DEBUG:boto:Params: {} 2024-04-12T03:58:30.835 DEBUG:boto:Token: None 2024-04-12T03:58:30.836 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.836 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.839 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020bd399a4b2d1964-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.839 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:30.839 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:30.839 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.839 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011 2024-04-12T03:58:30.839 DEBUG:boto:Data: 2024-04-12T03:58:30.840 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.840 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.840 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.840 DEBUG:boto:Params: {} 2024-04-12T03:58:30.840 DEBUG:boto:Token: None 2024-04-12T03:58:30.840 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000011 2024-04-12T03:58:30.840 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uu8Q50RzknsRCos3bbRLZaw3FMs= 2024-04-12T03:58:30.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uu8Q50RzknsRCos3bbRLZaw3FMs='} 2024-04-12T03:58:30.849 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff65cbf8ead13915-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.849 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.849 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.849 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.849 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.849 DEBUG:boto:Data: 2024-04-12T03:58:30.849 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.849 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.849 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.849 DEBUG:boto:Params: {} 2024-04-12T03:58:30.850 DEBUG:boto:Token: None 2024-04-12T03:58:30.850 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.850 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.852 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000702dd0fbdf311827-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.853 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:30.853 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:30.853 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.853 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001 2024-04-12T03:58:30.853 DEBUG:boto:Data: 2024-04-12T03:58:30.853 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.853 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.853 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.853 DEBUG:boto:Params: {} 2024-04-12T03:58:30.854 DEBUG:boto:Token: None 2024-04-12T03:58:30.854 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000001 2024-04-12T03:58:30.854 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0r+v4mB0qgr0Bopbd8GBsf12i6o= 2024-04-12T03:58:30.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0r+v4mB0qgr0Bopbd8GBsf12i6o='} 2024-04-12T03:58:30.898 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000296c97df0b1bb13f-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.899 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.899 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.899 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.899 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.899 DEBUG:boto:Data: 2024-04-12T03:58:30.899 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.899 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.899 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.899 DEBUG:boto:Params: {} 2024-04-12T03:58:30.899 DEBUG:boto:Token: None 2024-04-12T03:58:30.900 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.900 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.903 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ef1f429a1e32811-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.903 DEBUG:boto:path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:30.903 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:30.903 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.903 DEBUG:boto:Path: /dkyges-60/obj00000000000000000009 2024-04-12T03:58:30.903 DEBUG:boto:Data: 2024-04-12T03:58:30.903 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.903 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.904 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.904 DEBUG:boto:Params: {} 2024-04-12T03:58:30.904 DEBUG:boto:Token: None 2024-04-12T03:58:30.904 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000009 2024-04-12T03:58:30.904 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PzdBzwcai57daBXyhqTJYA0imBI= 2024-04-12T03:58:30.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PzdBzwcai57daBXyhqTJYA0imBI='} 2024-04-12T03:58:30.911 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000068059a286f82fc26-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.911 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.911 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.911 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.911 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.911 DEBUG:boto:Data: 2024-04-12T03:58:30.911 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.911 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.912 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.912 DEBUG:boto:Params: {} 2024-04-12T03:58:30.912 DEBUG:boto:Token: None 2024-04-12T03:58:30.912 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.912 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.915 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000020150e4b0c6726e2-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.916 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:30.916 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:30.916 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.916 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005 2024-04-12T03:58:30.916 DEBUG:boto:Data: 2024-04-12T03:58:30.916 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.916 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.916 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.916 DEBUG:boto:Params: {} 2024-04-12T03:58:30.916 DEBUG:boto:Token: None 2024-04-12T03:58:30.917 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000005 2024-04-12T03:58:30.917 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jtcXaakVx10OvBppYq780guiIa0= 2024-04-12T03:58:30.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jtcXaakVx10OvBppYq780guiIa0='} 2024-04-12T03:58:30.958 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0cfc8e2efdefd2b-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.959 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.959 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.959 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.959 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.959 DEBUG:boto:Data: 2024-04-12T03:58:30.959 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.959 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.959 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.959 DEBUG:boto:Params: {} 2024-04-12T03:58:30.959 DEBUG:boto:Token: None 2024-04-12T03:58:30.960 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.960 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.963 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7b3958b5f721462-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.963 DEBUG:boto:path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:30.963 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:30.963 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.963 DEBUG:boto:Path: /dkyges-60/obj00000000000000000007 2024-04-12T03:58:30.963 DEBUG:boto:Data: 2024-04-12T03:58:30.963 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.963 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.964 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.964 DEBUG:boto:Params: {} 2024-04-12T03:58:30.964 DEBUG:boto:Token: None 2024-04-12T03:58:30.964 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000007 2024-04-12T03:58:30.964 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tM/A4Jn4ymzEdPLL7C2f/aqMytQ= 2024-04-12T03:58:30.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tM/A4Jn4ymzEdPLL7C2f/aqMytQ='} 2024-04-12T03:58:30.973 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000039a8b5ea7fdc53b9-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.973 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:30.973 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:30.973 DEBUG:boto:Method: HEAD 2024-04-12T03:58:30.973 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:30.973 DEBUG:boto:Data: 2024-04-12T03:58:30.973 DEBUG:boto:Headers: {} 2024-04-12T03:58:30.973 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.973 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.974 DEBUG:boto:Params: {} 2024-04-12T03:58:30.974 DEBUG:boto:Token: None 2024-04-12T03:58:30.974 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/ 2024-04-12T03:58:30.974 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4= 2024-04-12T03:58:30.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBcasPz8uqLCIcQ2D8ghhsRjCK4=', 'Content-Length': '0'} 2024-04-12T03:58:30.977 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039e1f8c8dda183c5-006618b166-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:30.977 DEBUG:boto:path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:30.977 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:30.977 DEBUG:boto:Method: PUT 2024-04-12T03:58:30.977 DEBUG:boto:Path: /dkyges-60/obj00000000000000000021 2024-04-12T03:58:30.977 DEBUG:boto:Data: 2024-04-12T03:58:30.977 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:30.977 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:30.977 DEBUG:boto:Port: 8000 2024-04-12T03:58:30.978 DEBUG:boto:Params: {} 2024-04-12T03:58:30.978 DEBUG:boto:Token: None 2024-04-12T03:58:30.978 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:30 GMT /dkyges-60/obj00000000000000000021 2024-04-12T03:58:30.978 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UJJOWnhKp5uhYyUB9TPpKRxWJS0= 2024-04-12T03:58:30.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UJJOWnhKp5uhYyUB9TPpKRxWJS0='} 2024-04-12T03:58:31.022 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5e637da03240e7f-006618b166-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.023 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.023 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.023 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.023 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.023 DEBUG:boto:Data: 2024-04-12T03:58:31.023 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.023 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.023 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.023 DEBUG:boto:Params: {} 2024-04-12T03:58:31.023 DEBUG:boto:Token: None 2024-04-12T03:58:31.024 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.024 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.027 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f71ae69831ca812f-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.027 DEBUG:boto:path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:31.027 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:31.027 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.027 DEBUG:boto:Path: /dkyges-60/obj00000000000000000025 2024-04-12T03:58:31.027 DEBUG:boto:Data: 2024-04-12T03:58:31.027 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.027 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.028 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.028 DEBUG:boto:Params: {} 2024-04-12T03:58:31.028 DEBUG:boto:Token: None 2024-04-12T03:58:31.028 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000025 2024-04-12T03:58:31.028 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MV2rA6gwAPAX+uIHBMc9AN+5s/Y= 2024-04-12T03:58:31.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MV2rA6gwAPAX+uIHBMc9AN+5s/Y='} 2024-04-12T03:58:31.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd96f4cdcceba154-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.034 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.034 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.034 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.035 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.035 DEBUG:boto:Data: 2024-04-12T03:58:31.035 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.035 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.035 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.035 DEBUG:boto:Params: {} 2024-04-12T03:58:31.035 DEBUG:boto:Token: None 2024-04-12T03:58:31.035 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.035 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.038 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041049227a21dbf34-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.039 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:31.039 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:31.039 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.039 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004 2024-04-12T03:58:31.039 DEBUG:boto:Data: 2024-04-12T03:58:31.039 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.039 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.039 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.039 DEBUG:boto:Params: {} 2024-04-12T03:58:31.040 DEBUG:boto:Token: None 2024-04-12T03:58:31.040 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000004 2024-04-12T03:58:31.040 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oSsi5zGnfCKK0QRIfOLDA1sCnJ4= 2024-04-12T03:58:31.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oSsi5zGnfCKK0QRIfOLDA1sCnJ4='} 2024-04-12T03:58:31.082 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb9d7aa051486187-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:31.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:31.083 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-60 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:58:31.146 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:31.146 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:31.208 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T03:58:31.208 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-60 2024-04-12T03:58:31.217 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 26 2024-04-12T03:58:31.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:31.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:31.273 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:31.331 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:31.332 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:31.380 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:58:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: "objs": [ 2024-04-12T03:58:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000058", 2024-04-12T03:58:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000026", 2024-04-12T03:58:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000011", 2024-04-12T03:58:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000046", 2024-04-12T03:58:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000014", 2024-04-12T03:58:31.381 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000050", 2024-04-12T03:58:31.381 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000032", 2024-04-12T03:58:31.381 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000002", 2024-04-12T03:58:31.381 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000039", 2024-04-12T03:58:31.381 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000007", 2024-04-12T03:58:31.381 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000006", 2024-04-12T03:58:31.381 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000040", 2024-04-12T03:58:31.381 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000009", 2024-04-12T03:58:31.381 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000045", 2024-04-12T03:58:31.381 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000013", 2024-04-12T03:58:31.381 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000012", 2024-04-12T03:58:31.381 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000033", 2024-04-12T03:58:31.381 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000001", 2024-04-12T03:58:31.382 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000000", 2024-04-12T03:58:31.382 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000021", 2024-04-12T03:58:31.382 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000003", 2024-04-12T03:58:31.382 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000055", 2024-04-12T03:58:31.382 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000008" 2024-04-12T03:58:31.382 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:58:31.382 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:58:31.383 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.383 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.383 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.383 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.383 DEBUG:boto:Data: 2024-04-12T03:58:31.383 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.384 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.384 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.384 DEBUG:boto:Params: {} 2024-04-12T03:58:31.384 DEBUG:boto:Token: None 2024-04-12T03:58:31.384 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.384 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.388 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3069d9ae29348c0-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.388 DEBUG:boto:path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:31.388 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:31.388 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.388 DEBUG:boto:Path: /dkyges-60/obj00000000000000000050 2024-04-12T03:58:31.388 DEBUG:boto:Data: 2024-04-12T03:58:31.388 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.389 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.389 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.389 DEBUG:boto:Params: {} 2024-04-12T03:58:31.389 DEBUG:boto:Token: None 2024-04-12T03:58:31.389 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000050 2024-04-12T03:58:31.389 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ExQpFNlMZNq9YHxk0hpXSz1WVDo= 2024-04-12T03:58:31.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ExQpFNlMZNq9YHxk0hpXSz1WVDo='} 2024-04-12T03:58:31.395 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e794d4d519f2443f-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.395 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.395 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.395 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.395 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.395 DEBUG:boto:Data: 2024-04-12T03:58:31.395 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.395 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.396 DEBUG:boto:Params: {} 2024-04-12T03:58:31.396 DEBUG:boto:Token: None 2024-04-12T03:58:31.396 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.396 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.400 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f5746bd986ec341-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.400 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:31.400 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:31.400 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.400 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000 2024-04-12T03:58:31.400 DEBUG:boto:Data: 2024-04-12T03:58:31.400 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.401 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.401 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.401 DEBUG:boto:Params: {} 2024-04-12T03:58:31.401 DEBUG:boto:Token: None 2024-04-12T03:58:31.401 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000000 2024-04-12T03:58:31.401 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AY/jjDf7QNKZ/XGsvoUrpd10Wo8= 2024-04-12T03:58:31.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AY/jjDf7QNKZ/XGsvoUrpd10Wo8='} 2024-04-12T03:58:31.447 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d48c708bce3396fa-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.447 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.447 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.447 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.447 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.447 DEBUG:boto:Data: 2024-04-12T03:58:31.447 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.447 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.447 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.448 DEBUG:boto:Params: {} 2024-04-12T03:58:31.448 DEBUG:boto:Token: None 2024-04-12T03:58:31.448 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.448 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.451 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000680349253f62c6a0-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.452 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:31.452 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:31.452 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.452 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002 2024-04-12T03:58:31.452 DEBUG:boto:Data: 2024-04-12T03:58:31.452 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.452 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.452 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.452 DEBUG:boto:Params: {} 2024-04-12T03:58:31.452 DEBUG:boto:Token: None 2024-04-12T03:58:31.453 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000002 2024-04-12T03:58:31.453 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9tCuYk3PlBBZX+TDNGZvJi9xPk4= 2024-04-12T03:58:31.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9tCuYk3PlBBZX+TDNGZvJi9xPk4='} 2024-04-12T03:58:31.460 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000530004d10079b325-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.460 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.460 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.461 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.461 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.461 DEBUG:boto:Data: 2024-04-12T03:58:31.461 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.461 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.461 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.461 DEBUG:boto:Params: {} 2024-04-12T03:58:31.461 DEBUG:boto:Token: None 2024-04-12T03:58:31.461 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.461 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.464 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e15ef30b3bd49b7c-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.465 DEBUG:boto:path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:31.465 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:31.465 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.465 DEBUG:boto:Path: /dkyges-60/obj00000000000000000058 2024-04-12T03:58:31.465 DEBUG:boto:Data: 2024-04-12T03:58:31.465 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.465 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.465 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.465 DEBUG:boto:Params: {} 2024-04-12T03:58:31.466 DEBUG:boto:Token: None 2024-04-12T03:58:31.466 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000058 2024-04-12T03:58:31.466 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TcuOV2diVRU6fP1JOV7W6aYBt2Y= 2024-04-12T03:58:31.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TcuOV2diVRU6fP1JOV7W6aYBt2Y='} 2024-04-12T03:58:31.514 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0b07d4be325e4fd-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.515 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.515 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.515 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.515 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.515 DEBUG:boto:Data: 2024-04-12T03:58:31.515 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.515 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.515 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.515 DEBUG:boto:Params: {} 2024-04-12T03:58:31.516 DEBUG:boto:Token: None 2024-04-12T03:58:31.516 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.516 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.519 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000490c78ebff7d09be-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.519 DEBUG:boto:path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:31.520 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:31.520 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.520 DEBUG:boto:Path: /dkyges-60/obj00000000000000000026 2024-04-12T03:58:31.520 DEBUG:boto:Data: 2024-04-12T03:58:31.520 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.520 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.520 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.520 DEBUG:boto:Params: {} 2024-04-12T03:58:31.520 DEBUG:boto:Token: None 2024-04-12T03:58:31.521 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000026 2024-04-12T03:58:31.521 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3hRSOLe/Pmq7aZBkeNs6/V+IQwc= 2024-04-12T03:58:31.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3hRSOLe/Pmq7aZBkeNs6/V+IQwc='} 2024-04-12T03:58:31.529 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049a3de5324240f1f-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.529 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.529 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.529 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.529 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.529 DEBUG:boto:Data: 2024-04-12T03:58:31.530 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.530 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.530 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.530 DEBUG:boto:Params: {} 2024-04-12T03:58:31.530 DEBUG:boto:Token: None 2024-04-12T03:58:31.530 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.530 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7f15f4b0f8cacba-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.534 DEBUG:boto:path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:31.534 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:31.534 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.534 DEBUG:boto:Path: /dkyges-60/obj00000000000000000014 2024-04-12T03:58:31.534 DEBUG:boto:Data: 2024-04-12T03:58:31.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.535 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.535 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.535 DEBUG:boto:Params: {} 2024-04-12T03:58:31.535 DEBUG:boto:Token: None 2024-04-12T03:58:31.535 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000014 2024-04-12T03:58:31.535 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Q7Dy48s8cHTpMawywHqu6D2yEOI= 2024-04-12T03:58:31.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Q7Dy48s8cHTpMawywHqu6D2yEOI='} 2024-04-12T03:58:31.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000469e8f7a0d42364c-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.579 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.579 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.579 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.579 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.579 DEBUG:boto:Data: 2024-04-12T03:58:31.579 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.579 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.579 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.579 DEBUG:boto:Params: {} 2024-04-12T03:58:31.579 DEBUG:boto:Token: None 2024-04-12T03:58:31.579 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.580 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.583 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000208f88585c431621-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.583 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:31.583 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:31.583 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.583 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011 2024-04-12T03:58:31.583 DEBUG:boto:Data: 2024-04-12T03:58:31.584 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.584 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.584 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.584 DEBUG:boto:Params: {} 2024-04-12T03:58:31.584 DEBUG:boto:Token: None 2024-04-12T03:58:31.584 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000011 2024-04-12T03:58:31.584 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dxgZ2NyzqL6+0UyLkxqhGDw1utM= 2024-04-12T03:58:31.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dxgZ2NyzqL6+0UyLkxqhGDw1utM='} 2024-04-12T03:58:31.592 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000835b714a4369efab-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.593 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.593 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.593 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.593 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.593 DEBUG:boto:Data: 2024-04-12T03:58:31.593 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.593 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.593 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.593 DEBUG:boto:Params: {} 2024-04-12T03:58:31.593 DEBUG:boto:Token: None 2024-04-12T03:58:31.593 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.594 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.597 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071147c992bb00b75-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.597 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:31.597 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:31.597 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.597 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001 2024-04-12T03:58:31.598 DEBUG:boto:Data: 2024-04-12T03:58:31.598 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.598 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.598 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.598 DEBUG:boto:Params: {} 2024-04-12T03:58:31.598 DEBUG:boto:Token: None 2024-04-12T03:58:31.598 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000001 2024-04-12T03:58:31.598 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Aq5vam7QXfnfyLqL4gXYTAkLoM4= 2024-04-12T03:58:31.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Aq5vam7QXfnfyLqL4gXYTAkLoM4='} 2024-04-12T03:58:31.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049c3c47e8024da53-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.643 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.643 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.643 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.643 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.643 DEBUG:boto:Data: 2024-04-12T03:58:31.643 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.643 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.643 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.643 DEBUG:boto:Params: {} 2024-04-12T03:58:31.643 DEBUG:boto:Token: None 2024-04-12T03:58:31.643 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.644 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.647 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd44a85ad803113a-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.647 DEBUG:boto:path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:31.647 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:31.647 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.648 DEBUG:boto:Path: /dkyges-60/obj00000000000000000046 2024-04-12T03:58:31.648 DEBUG:boto:Data: 2024-04-12T03:58:31.648 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.648 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.648 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.648 DEBUG:boto:Params: {} 2024-04-12T03:58:31.648 DEBUG:boto:Token: None 2024-04-12T03:58:31.648 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000046 2024-04-12T03:58:31.648 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eRgZSOIedavxEp/7FEWvCMl5lAg= 2024-04-12T03:58:31.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eRgZSOIedavxEp/7FEWvCMl5lAg='} 2024-04-12T03:58:31.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1837100d2d594af-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.655 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.655 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.655 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.655 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.655 DEBUG:boto:Data: 2024-04-12T03:58:31.655 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.655 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.655 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.655 DEBUG:boto:Params: {} 2024-04-12T03:58:31.655 DEBUG:boto:Token: None 2024-04-12T03:58:31.655 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.659 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f897cab588b2441-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.659 DEBUG:boto:path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:31.659 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:31.660 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.660 DEBUG:boto:Path: /dkyges-60/obj00000000000000000008 2024-04-12T03:58:31.660 DEBUG:boto:Data: 2024-04-12T03:58:31.660 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.660 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.660 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.660 DEBUG:boto:Params: {} 2024-04-12T03:58:31.660 DEBUG:boto:Token: None 2024-04-12T03:58:31.660 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000008 2024-04-12T03:58:31.660 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UT5NsyCWI2dKyyXQMrIVqDWknTc= 2024-04-12T03:58:31.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UT5NsyCWI2dKyyXQMrIVqDWknTc='} 2024-04-12T03:58:31.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ed2f9ac5ec1dfd1-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.707 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.707 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.707 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.707 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.707 DEBUG:boto:Data: 2024-04-12T03:58:31.707 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.707 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.707 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.707 DEBUG:boto:Params: {} 2024-04-12T03:58:31.707 DEBUG:boto:Token: None 2024-04-12T03:58:31.708 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.708 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.711 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000098ecb89556e3c0d-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.711 DEBUG:boto:path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:31.711 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:31.711 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.711 DEBUG:boto:Path: /dkyges-60/obj00000000000000000003 2024-04-12T03:58:31.712 DEBUG:boto:Data: 2024-04-12T03:58:31.712 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.712 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.712 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.712 DEBUG:boto:Params: {} 2024-04-12T03:58:31.712 DEBUG:boto:Token: None 2024-04-12T03:58:31.712 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000003 2024-04-12T03:58:31.712 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZsWTZ7B4JYnkHV1lwMkIcG5SOHY= 2024-04-12T03:58:31.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZsWTZ7B4JYnkHV1lwMkIcG5SOHY='} 2024-04-12T03:58:31.720 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af0f934592dfe358-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.720 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.720 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.720 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.721 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.721 DEBUG:boto:Data: 2024-04-12T03:58:31.721 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.721 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.721 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.721 DEBUG:boto:Params: {} 2024-04-12T03:58:31.721 DEBUG:boto:Token: None 2024-04-12T03:58:31.721 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.721 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.724 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5db7648befa01c5-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.725 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:31.725 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:31.725 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.725 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006 2024-04-12T03:58:31.725 DEBUG:boto:Data: 2024-04-12T03:58:31.725 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.725 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.725 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.725 DEBUG:boto:Params: {} 2024-04-12T03:58:31.725 DEBUG:boto:Token: None 2024-04-12T03:58:31.726 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000006 2024-04-12T03:58:31.726 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IoJuiPgYc41VoS/G9cI6631R45s= 2024-04-12T03:58:31.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IoJuiPgYc41VoS/G9cI6631R45s='} 2024-04-12T03:58:31.770 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f67e10c9c31cb28-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.770 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.771 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.771 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.771 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.771 DEBUG:boto:Data: 2024-04-12T03:58:31.771 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.771 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.771 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.771 DEBUG:boto:Params: {} 2024-04-12T03:58:31.771 DEBUG:boto:Token: None 2024-04-12T03:58:31.771 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.771 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.775 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021b94b96113e413f-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.775 DEBUG:boto:path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:31.776 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:31.776 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.776 DEBUG:boto:Path: /dkyges-60/obj00000000000000000045 2024-04-12T03:58:31.776 DEBUG:boto:Data: 2024-04-12T03:58:31.776 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.776 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.776 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.776 DEBUG:boto:Params: {} 2024-04-12T03:58:31.776 DEBUG:boto:Token: None 2024-04-12T03:58:31.776 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000045 2024-04-12T03:58:31.776 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TdNnfE7wtIlkBPl0/VdeCygavKs= 2024-04-12T03:58:31.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TdNnfE7wtIlkBPl0/VdeCygavKs='} 2024-04-12T03:58:31.783 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4a9f072af2e2757-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.783 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.783 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.783 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.783 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.783 DEBUG:boto:Data: 2024-04-12T03:58:31.783 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.783 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.783 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.783 DEBUG:boto:Params: {} 2024-04-12T03:58:31.783 DEBUG:boto:Token: None 2024-04-12T03:58:31.784 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.784 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.787 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a7b8d3ac0fe1bff-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.787 DEBUG:boto:path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:31.787 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:31.787 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.787 DEBUG:boto:Path: /dkyges-60/obj00000000000000000040 2024-04-12T03:58:31.787 DEBUG:boto:Data: 2024-04-12T03:58:31.787 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.788 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.788 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.788 DEBUG:boto:Params: {} 2024-04-12T03:58:31.788 DEBUG:boto:Token: None 2024-04-12T03:58:31.788 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000040 2024-04-12T03:58:31.788 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u27nOYJbsTu3sNh7XYb+jWV9Tjg= 2024-04-12T03:58:31.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u27nOYJbsTu3sNh7XYb+jWV9Tjg='} 2024-04-12T03:58:31.830 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009634df0f796fce98-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.831 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.831 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.831 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.831 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.831 DEBUG:boto:Data: 2024-04-12T03:58:31.831 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.831 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.831 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.831 DEBUG:boto:Params: {} 2024-04-12T03:58:31.831 DEBUG:boto:Token: None 2024-04-12T03:58:31.832 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.832 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.835 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cca89a1eef9dcc60-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.835 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:31.835 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:31.836 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.836 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013 2024-04-12T03:58:31.836 DEBUG:boto:Data: 2024-04-12T03:58:31.836 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.836 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.836 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.836 DEBUG:boto:Params: {} 2024-04-12T03:58:31.836 DEBUG:boto:Token: None 2024-04-12T03:58:31.836 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000013 2024-04-12T03:58:31.836 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:shm3i5o1dXD7a9FhoEFNX7cjf6E= 2024-04-12T03:58:31.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:shm3i5o1dXD7a9FhoEFNX7cjf6E='} 2024-04-12T03:58:31.845 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000017392896f64b5a1e-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.845 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.845 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.845 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.845 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.845 DEBUG:boto:Data: 2024-04-12T03:58:31.845 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.845 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.845 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.846 DEBUG:boto:Params: {} 2024-04-12T03:58:31.846 DEBUG:boto:Token: None 2024-04-12T03:58:31.846 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.846 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.849 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f3a4f2a2d5db582b-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.849 DEBUG:boto:path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:31.849 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:31.849 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.849 DEBUG:boto:Path: /dkyges-60/obj00000000000000000039 2024-04-12T03:58:31.849 DEBUG:boto:Data: 2024-04-12T03:58:31.849 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.849 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.850 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.850 DEBUG:boto:Params: {} 2024-04-12T03:58:31.850 DEBUG:boto:Token: None 2024-04-12T03:58:31.850 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000039 2024-04-12T03:58:31.850 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RCZdY+mjXuiCLOjvo0FmUNLMMgQ= 2024-04-12T03:58:31.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RCZdY+mjXuiCLOjvo0FmUNLMMgQ='} 2024-04-12T03:58:31.894 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a24c0a8c53636421-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.895 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.895 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.895 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.895 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.895 DEBUG:boto:Data: 2024-04-12T03:58:31.895 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.895 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.895 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.895 DEBUG:boto:Params: {} 2024-04-12T03:58:31.895 DEBUG:boto:Token: None 2024-04-12T03:58:31.896 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.896 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.899 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002109f3a924908f09-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.899 DEBUG:boto:path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:31.899 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:31.899 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.899 DEBUG:boto:Path: /dkyges-60/obj00000000000000000009 2024-04-12T03:58:31.899 DEBUG:boto:Data: 2024-04-12T03:58:31.899 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.899 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.899 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.899 DEBUG:boto:Params: {} 2024-04-12T03:58:31.900 DEBUG:boto:Token: None 2024-04-12T03:58:31.900 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000009 2024-04-12T03:58:31.900 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Odv0td+DG+hGVf1UKEyiCJL2t2I= 2024-04-12T03:58:31.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Odv0td+DG+hGVf1UKEyiCJL2t2I='} 2024-04-12T03:58:31.907 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000714ebc778b355d82-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.908 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.908 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.908 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.908 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.908 DEBUG:boto:Data: 2024-04-12T03:58:31.908 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.908 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.908 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.908 DEBUG:boto:Params: {} 2024-04-12T03:58:31.909 DEBUG:boto:Token: None 2024-04-12T03:58:31.909 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.909 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.914 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b91cdaff466ad76c-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.914 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:31.914 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:31.914 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.914 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012 2024-04-12T03:58:31.915 DEBUG:boto:Data: 2024-04-12T03:58:31.915 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.915 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.915 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.915 DEBUG:boto:Params: {} 2024-04-12T03:58:31.915 DEBUG:boto:Token: None 2024-04-12T03:58:31.915 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000012 2024-04-12T03:58:31.915 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4YQIdqBS1Wrjw+NUVgFK6f9Gh0A= 2024-04-12T03:58:31.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4YQIdqBS1Wrjw+NUVgFK6f9Gh0A='} 2024-04-12T03:58:31.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000077e093e212992888-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.963 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.963 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.963 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.963 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.963 DEBUG:boto:Data: 2024-04-12T03:58:31.963 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.963 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.963 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.963 DEBUG:boto:Params: {} 2024-04-12T03:58:31.963 DEBUG:boto:Token: None 2024-04-12T03:58:31.963 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.964 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.967 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca3ec5761533d60c-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.967 DEBUG:boto:path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:31.967 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:31.967 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.967 DEBUG:boto:Path: /dkyges-60/obj00000000000000000032 2024-04-12T03:58:31.967 DEBUG:boto:Data: 2024-04-12T03:58:31.968 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.968 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.968 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.968 DEBUG:boto:Params: {} 2024-04-12T03:58:31.968 DEBUG:boto:Token: None 2024-04-12T03:58:31.968 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000032 2024-04-12T03:58:31.968 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yqeTTGMVHKzzzeeKjSPF02byVvw= 2024-04-12T03:58:31.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yqeTTGMVHKzzzeeKjSPF02byVvw='} 2024-04-12T03:58:31.976 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000061d6d4f53714dc1-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.976 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:31.976 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:31.977 DEBUG:boto:Method: HEAD 2024-04-12T03:58:31.977 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:31.977 DEBUG:boto:Data: 2024-04-12T03:58:31.977 DEBUG:boto:Headers: {} 2024-04-12T03:58:31.977 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.977 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.977 DEBUG:boto:Params: {} 2024-04-12T03:58:31.977 DEBUG:boto:Token: None 2024-04-12T03:58:31.977 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/ 2024-04-12T03:58:31.977 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw= 2024-04-12T03:58:31.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nbasQTZ5zxsI3TAEnVeyMOB9Jjw=', 'Content-Length': '0'} 2024-04-12T03:58:31.981 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000162d0b7975e4d604-006618b167-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:31.981 DEBUG:boto:path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:31.981 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:31.982 DEBUG:boto:Method: PUT 2024-04-12T03:58:31.982 DEBUG:boto:Path: /dkyges-60/obj00000000000000000033 2024-04-12T03:58:31.982 DEBUG:boto:Data: 2024-04-12T03:58:31.982 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:31.982 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:31.982 DEBUG:boto:Port: 8000 2024-04-12T03:58:31.982 DEBUG:boto:Params: {} 2024-04-12T03:58:31.982 DEBUG:boto:Token: None 2024-04-12T03:58:31.982 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:31 GMT /dkyges-60/obj00000000000000000033 2024-04-12T03:58:31.983 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lICAus3womJHELLH2+Bhvjes7Nw= 2024-04-12T03:58:31.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lICAus3womJHELLH2+Bhvjes7Nw='} 2024-04-12T03:58:32.026 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6dcff8bf2a94b3a-006618b167-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.027 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.027 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.027 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.027 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.027 DEBUG:boto:Data: 2024-04-12T03:58:32.027 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.027 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.027 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.027 DEBUG:boto:Params: {} 2024-04-12T03:58:32.028 DEBUG:boto:Token: None 2024-04-12T03:58:32.028 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.028 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.031 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb25fc2a090835e3-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.031 DEBUG:boto:path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:32.032 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:32.032 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.032 DEBUG:boto:Path: /dkyges-60/obj00000000000000000055 2024-04-12T03:58:32.032 DEBUG:boto:Data: 2024-04-12T03:58:32.032 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.032 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.032 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.032 DEBUG:boto:Params: {} 2024-04-12T03:58:32.032 DEBUG:boto:Token: None 2024-04-12T03:58:32.032 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000055 2024-04-12T03:58:32.032 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VlASumcSR72UdSXNnsL+xMIGkfs= 2024-04-12T03:58:32.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VlASumcSR72UdSXNnsL+xMIGkfs='} 2024-04-12T03:58:32.039 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004269917a37f6c5fb-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:32.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:32.039 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-60 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:58:32.102 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:32.102 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:32.158 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T03:58:32.158 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-60 2024-04-12T03:58:32.170 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 34 2024-04-12T03:58:32.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:32.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:32.232 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 29 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:32.289 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:32.289 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:32.340 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:58:32.340 INFO:teuthology.orchestra.run.smithi043.stdout: "objs": [ 2024-04-12T03:58:32.340 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000006", 2024-04-12T03:58:32.340 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000059", 2024-04-12T03:58:32.340 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000027", 2024-04-12T03:58:32.340 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000004", 2024-04-12T03:58:32.340 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000011", 2024-04-12T03:58:32.340 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000002", 2024-04-12T03:58:32.341 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000019", 2024-04-12T03:58:32.341 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000049", 2024-04-12T03:58:32.341 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000017", 2024-04-12T03:58:32.341 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000008", 2024-04-12T03:58:32.341 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000015", 2024-04-12T03:58:32.341 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000031", 2024-04-12T03:58:32.341 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000022", 2024-04-12T03:58:32.341 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000013", 2024-04-12T03:58:32.341 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000020", 2024-04-12T03:58:32.341 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000037", 2024-04-12T03:58:32.341 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000005", 2024-04-12T03:58:32.341 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000012", 2024-04-12T03:58:32.342 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000003", 2024-04-12T03:58:32.342 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000042", 2024-04-12T03:58:32.342 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000010", 2024-04-12T03:58:32.342 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000001", 2024-04-12T03:58:32.342 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000009", 2024-04-12T03:58:32.342 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000025", 2024-04-12T03:58:32.342 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000016", 2024-04-12T03:58:32.342 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000000", 2024-04-12T03:58:32.342 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000014", 2024-04-12T03:58:32.342 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000030", 2024-04-12T03:58:32.342 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000021" 2024-04-12T03:58:32.342 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:58:32.343 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:58:32.343 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.343 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.344 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.344 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.344 DEBUG:boto:Data: 2024-04-12T03:58:32.344 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.344 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.344 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.344 DEBUG:boto:Params: {} 2024-04-12T03:58:32.344 DEBUG:boto:Token: None 2024-04-12T03:58:32.344 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.344 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.348 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a732b9e149cd273-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.348 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:32.348 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:32.348 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.349 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013 2024-04-12T03:58:32.349 DEBUG:boto:Data: 2024-04-12T03:58:32.349 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.349 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.349 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.349 DEBUG:boto:Params: {} 2024-04-12T03:58:32.349 DEBUG:boto:Token: None 2024-04-12T03:58:32.349 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000013 2024-04-12T03:58:32.349 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OlXCdKXLuzfI/TmUjvncLBi2cVA= 2024-04-12T03:58:32.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OlXCdKXLuzfI/TmUjvncLBi2cVA='} 2024-04-12T03:58:32.394 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e9f9b5c86a37e797-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.395 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.395 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.395 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.395 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.395 DEBUG:boto:Data: 2024-04-12T03:58:32.395 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.395 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.395 DEBUG:boto:Params: {} 2024-04-12T03:58:32.395 DEBUG:boto:Token: None 2024-04-12T03:58:32.395 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.396 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.399 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a84199273457f40c-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.399 DEBUG:boto:path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:32.399 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:32.399 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.399 DEBUG:boto:Path: /dkyges-60/obj00000000000000000019 2024-04-12T03:58:32.400 DEBUG:boto:Data: 2024-04-12T03:58:32.400 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.400 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.400 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.400 DEBUG:boto:Params: {} 2024-04-12T03:58:32.400 DEBUG:boto:Token: None 2024-04-12T03:58:32.400 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000019 2024-04-12T03:58:32.400 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BLdec8c1Sik6hq1VBTHYQzdbiXY= 2024-04-12T03:58:32.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BLdec8c1Sik6hq1VBTHYQzdbiXY='} 2024-04-12T03:58:32.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096f5f84c812b5200-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.408 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.408 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.408 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.408 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.409 DEBUG:boto:Data: 2024-04-12T03:58:32.409 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.409 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.409 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.409 DEBUG:boto:Params: {} 2024-04-12T03:58:32.409 DEBUG:boto:Token: None 2024-04-12T03:58:32.409 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.409 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046f43dc27e35b119-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.412 DEBUG:boto:path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:32.413 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:32.413 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.413 DEBUG:boto:Path: /dkyges-60/obj00000000000000000014 2024-04-12T03:58:32.413 DEBUG:boto:Data: 2024-04-12T03:58:32.413 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.413 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.413 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.413 DEBUG:boto:Params: {} 2024-04-12T03:58:32.413 DEBUG:boto:Token: None 2024-04-12T03:58:32.413 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000014 2024-04-12T03:58:32.414 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r7VS80Ej5u88iLjcmlqtAeaedhA= 2024-04-12T03:58:32.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r7VS80Ej5u88iLjcmlqtAeaedhA='} 2024-04-12T03:58:32.458 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b130a0de1a62dd1c-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.458 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.459 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.459 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.459 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.459 DEBUG:boto:Data: 2024-04-12T03:58:32.459 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.459 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.459 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.459 DEBUG:boto:Params: {} 2024-04-12T03:58:32.459 DEBUG:boto:Token: None 2024-04-12T03:58:32.459 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.460 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.463 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e48cccb89d795366-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.463 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:32.463 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:32.464 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.464 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012 2024-04-12T03:58:32.464 DEBUG:boto:Data: 2024-04-12T03:58:32.464 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.464 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.464 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.464 DEBUG:boto:Params: {} 2024-04-12T03:58:32.464 DEBUG:boto:Token: None 2024-04-12T03:58:32.464 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000012 2024-04-12T03:58:32.464 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uKd2s3kwrgPeXi2RwS2LlP9ljhQ= 2024-04-12T03:58:32.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uKd2s3kwrgPeXi2RwS2LlP9ljhQ='} 2024-04-12T03:58:32.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a69a678b2ba0c742-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.472 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.472 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.473 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.473 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.473 DEBUG:boto:Data: 2024-04-12T03:58:32.473 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.473 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.473 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.473 DEBUG:boto:Params: {} 2024-04-12T03:58:32.473 DEBUG:boto:Token: None 2024-04-12T03:58:32.473 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.473 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.477 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000083a60bd4060f39dc-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.478 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:32.478 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:32.478 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.478 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010 2024-04-12T03:58:32.478 DEBUG:boto:Data: 2024-04-12T03:58:32.478 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.478 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.478 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.478 DEBUG:boto:Params: {} 2024-04-12T03:58:32.478 DEBUG:boto:Token: None 2024-04-12T03:58:32.479 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000010 2024-04-12T03:58:32.479 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9BCzyQHj0du2x57rdw/MPI6Qq0A= 2024-04-12T03:58:32.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9BCzyQHj0du2x57rdw/MPI6Qq0A='} 2024-04-12T03:58:32.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083aa0f69950dc397-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.527 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.527 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.527 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.527 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.527 DEBUG:boto:Data: 2024-04-12T03:58:32.527 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.527 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.527 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.527 DEBUG:boto:Params: {} 2024-04-12T03:58:32.527 DEBUG:boto:Token: None 2024-04-12T03:58:32.527 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.528 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005698a8b226757a85-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.531 DEBUG:boto:path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:32.531 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:32.531 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.531 DEBUG:boto:Path: /dkyges-60/obj00000000000000000037 2024-04-12T03:58:32.532 DEBUG:boto:Data: 2024-04-12T03:58:32.532 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.532 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.532 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.532 DEBUG:boto:Params: {} 2024-04-12T03:58:32.532 DEBUG:boto:Token: None 2024-04-12T03:58:32.532 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000037 2024-04-12T03:58:32.532 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ih6/MlGkpqAO//N/TxAs+Q06yK0= 2024-04-12T03:58:32.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ih6/MlGkpqAO//N/TxAs+Q06yK0='} 2024-04-12T03:58:32.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000febf4bb9870fe935-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.539 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.539 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.539 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.539 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.539 DEBUG:boto:Data: 2024-04-12T03:58:32.539 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.539 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.539 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.539 DEBUG:boto:Params: {} 2024-04-12T03:58:32.539 DEBUG:boto:Token: None 2024-04-12T03:58:32.539 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.540 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.543 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f6b0c0e5c1f56c1-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.543 DEBUG:boto:path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:32.543 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:32.543 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.544 DEBUG:boto:Path: /dkyges-60/obj00000000000000000030 2024-04-12T03:58:32.544 DEBUG:boto:Data: 2024-04-12T03:58:32.544 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.544 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.544 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.544 DEBUG:boto:Params: {} 2024-04-12T03:58:32.544 DEBUG:boto:Token: None 2024-04-12T03:58:32.544 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000030 2024-04-12T03:58:32.544 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CnRPkDmw4YO1Hkqe1uPL+kyYDDU= 2024-04-12T03:58:32.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CnRPkDmw4YO1Hkqe1uPL+kyYDDU='} 2024-04-12T03:58:32.586 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d8b73b84687f152-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.587 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.587 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.587 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.587 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.587 DEBUG:boto:Data: 2024-04-12T03:58:32.587 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.587 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.587 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.587 DEBUG:boto:Params: {} 2024-04-12T03:58:32.588 DEBUG:boto:Token: None 2024-04-12T03:58:32.588 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.588 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.591 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a985c33a7394197c-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.591 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:32.591 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:32.592 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.592 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005 2024-04-12T03:58:32.592 DEBUG:boto:Data: 2024-04-12T03:58:32.592 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.592 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.592 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.592 DEBUG:boto:Params: {} 2024-04-12T03:58:32.592 DEBUG:boto:Token: None 2024-04-12T03:58:32.592 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000005 2024-04-12T03:58:32.592 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vGzxp5nBA14Lm7ZoqL1lAVMPjOo= 2024-04-12T03:58:32.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vGzxp5nBA14Lm7ZoqL1lAVMPjOo='} 2024-04-12T03:58:32.600 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f571cb062dccd147-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.601 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.601 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.601 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.601 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.601 DEBUG:boto:Data: 2024-04-12T03:58:32.601 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.601 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.601 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.601 DEBUG:boto:Params: {} 2024-04-12T03:58:32.601 DEBUG:boto:Token: None 2024-04-12T03:58:32.602 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.602 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.605 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000401f75f7a2770869-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.605 DEBUG:boto:path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:32.606 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:32.606 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.606 DEBUG:boto:Path: /dkyges-60/obj00000000000000000022 2024-04-12T03:58:32.606 DEBUG:boto:Data: 2024-04-12T03:58:32.606 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.606 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.606 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.606 DEBUG:boto:Params: {} 2024-04-12T03:58:32.606 DEBUG:boto:Token: None 2024-04-12T03:58:32.606 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000022 2024-04-12T03:58:32.607 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IiPjusuiqn9zuwdHdWvgen9aH2w= 2024-04-12T03:58:32.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IiPjusuiqn9zuwdHdWvgen9aH2w='} 2024-04-12T03:58:32.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fea3cb86ec82c15a-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.651 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.651 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.651 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.651 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.651 DEBUG:boto:Data: 2024-04-12T03:58:32.651 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.651 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.651 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.651 DEBUG:boto:Params: {} 2024-04-12T03:58:32.651 DEBUG:boto:Token: None 2024-04-12T03:58:32.652 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.652 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.655 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fe13b5e187cf794-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.655 DEBUG:boto:path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:32.656 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:32.656 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.656 DEBUG:boto:Path: /dkyges-60/obj00000000000000000016 2024-04-12T03:58:32.656 DEBUG:boto:Data: 2024-04-12T03:58:32.656 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.656 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.656 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.656 DEBUG:boto:Params: {} 2024-04-12T03:58:32.656 DEBUG:boto:Token: None 2024-04-12T03:58:32.656 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000016 2024-04-12T03:58:32.657 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hh3ihy8oJt24Z+pLsyceA29Jsew= 2024-04-12T03:58:32.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hh3ihy8oJt24Z+pLsyceA29Jsew='} 2024-04-12T03:58:32.665 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd6f62e301d0323d-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.666 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.666 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.666 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.666 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.666 DEBUG:boto:Data: 2024-04-12T03:58:32.666 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.666 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.666 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.666 DEBUG:boto:Params: {} 2024-04-12T03:58:32.666 DEBUG:boto:Token: None 2024-04-12T03:58:32.667 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.667 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.670 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e06a52dac7257afc-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.670 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:32.670 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:32.671 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.671 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001 2024-04-12T03:58:32.671 DEBUG:boto:Data: 2024-04-12T03:58:32.671 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.671 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.671 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.671 DEBUG:boto:Params: {} 2024-04-12T03:58:32.671 DEBUG:boto:Token: None 2024-04-12T03:58:32.671 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000001 2024-04-12T03:58:32.671 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:udCc+R08P6UwlWrvDkl3ljowC3s= 2024-04-12T03:58:32.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:udCc+R08P6UwlWrvDkl3ljowC3s='} 2024-04-12T03:58:32.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b76f0086535dc3b9-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.715 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.715 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.715 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.715 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.715 DEBUG:boto:Data: 2024-04-12T03:58:32.715 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.715 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.715 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.715 DEBUG:boto:Params: {} 2024-04-12T03:58:32.716 DEBUG:boto:Token: None 2024-04-12T03:58:32.716 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.716 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000138f6e878d059f61-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.720 DEBUG:boto:path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:32.720 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:32.720 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.720 DEBUG:boto:Path: /dkyges-60/obj00000000000000000008 2024-04-12T03:58:32.720 DEBUG:boto:Data: 2024-04-12T03:58:32.720 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.720 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.721 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.721 DEBUG:boto:Params: {} 2024-04-12T03:58:32.721 DEBUG:boto:Token: None 2024-04-12T03:58:32.721 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000008 2024-04-12T03:58:32.721 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ydk/FyOyCshT5Xxa5XN1uTJtOAQ= 2024-04-12T03:58:32.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ydk/FyOyCshT5Xxa5XN1uTJtOAQ='} 2024-04-12T03:58:32.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058ac92aff0f3d86d-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.730 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.730 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.730 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.730 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.730 DEBUG:boto:Data: 2024-04-12T03:58:32.730 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.731 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.731 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.731 DEBUG:boto:Params: {} 2024-04-12T03:58:32.731 DEBUG:boto:Token: None 2024-04-12T03:58:32.731 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.731 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.735 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000161f3c8de1048363-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.735 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:32.735 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:32.735 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.735 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004 2024-04-12T03:58:32.735 DEBUG:boto:Data: 2024-04-12T03:58:32.735 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.736 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.736 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.736 DEBUG:boto:Params: {} 2024-04-12T03:58:32.736 DEBUG:boto:Token: None 2024-04-12T03:58:32.736 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000004 2024-04-12T03:58:32.736 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UaUQw5kB/CaJ97zvBGgPBgAMm+4= 2024-04-12T03:58:32.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UaUQw5kB/CaJ97zvBGgPBgAMm+4='} 2024-04-12T03:58:32.779 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057fd5ccf15bc21af-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.779 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.779 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.779 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.779 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.779 DEBUG:boto:Data: 2024-04-12T03:58:32.779 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.779 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.779 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.779 DEBUG:boto:Params: {} 2024-04-12T03:58:32.780 DEBUG:boto:Token: None 2024-04-12T03:58:32.780 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.780 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057424c20a62044a4-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.784 DEBUG:boto:path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:32.784 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:32.784 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.785 DEBUG:boto:Path: /dkyges-60/obj00000000000000000009 2024-04-12T03:58:32.785 DEBUG:boto:Data: 2024-04-12T03:58:32.785 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.785 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.785 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.785 DEBUG:boto:Params: {} 2024-04-12T03:58:32.785 DEBUG:boto:Token: None 2024-04-12T03:58:32.785 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000009 2024-04-12T03:58:32.785 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5rJXp/nJxkDaBbCZ3asxchHl5tM= 2024-04-12T03:58:32.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5rJXp/nJxkDaBbCZ3asxchHl5tM='} 2024-04-12T03:58:32.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d945a15c6fb66997-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.794 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.794 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.794 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.794 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.794 DEBUG:boto:Data: 2024-04-12T03:58:32.794 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.795 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.795 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.795 DEBUG:boto:Params: {} 2024-04-12T03:58:32.795 DEBUG:boto:Token: None 2024-04-12T03:58:32.795 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.795 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d18af5d2adff96f-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.799 DEBUG:boto:path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:32.799 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:32.799 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.799 DEBUG:boto:Path: /dkyges-60/obj00000000000000000003 2024-04-12T03:58:32.800 DEBUG:boto:Data: 2024-04-12T03:58:32.800 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.800 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.800 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.800 DEBUG:boto:Params: {} 2024-04-12T03:58:32.800 DEBUG:boto:Token: None 2024-04-12T03:58:32.800 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000003 2024-04-12T03:58:32.800 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pqft+kKzGPkCqVd7dg8jJSZwZ78= 2024-04-12T03:58:32.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pqft+kKzGPkCqVd7dg8jJSZwZ78='} 2024-04-12T03:58:32.842 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053b85aca89c13a81-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.843 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.843 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.843 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.843 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.843 DEBUG:boto:Data: 2024-04-12T03:58:32.843 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.843 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.843 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.843 DEBUG:boto:Params: {} 2024-04-12T03:58:32.844 DEBUG:boto:Token: None 2024-04-12T03:58:32.844 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.844 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.847 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e24af2e643f9a2bc-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.848 DEBUG:boto:path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:32.848 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:32.848 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.848 DEBUG:boto:Path: /dkyges-60/obj00000000000000000059 2024-04-12T03:58:32.848 DEBUG:boto:Data: 2024-04-12T03:58:32.848 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.848 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.848 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.848 DEBUG:boto:Params: {} 2024-04-12T03:58:32.849 DEBUG:boto:Token: None 2024-04-12T03:58:32.849 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000059 2024-04-12T03:58:32.849 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NgoRwYVlDwpSgZ9NHjT8I56sPlY= 2024-04-12T03:58:32.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NgoRwYVlDwpSgZ9NHjT8I56sPlY='} 2024-04-12T03:58:32.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c1a182399e5d54b-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.856 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.856 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.856 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.856 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.856 DEBUG:boto:Data: 2024-04-12T03:58:32.856 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.856 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.856 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.856 DEBUG:boto:Params: {} 2024-04-12T03:58:32.857 DEBUG:boto:Token: None 2024-04-12T03:58:32.857 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.857 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051ee6712ada99d7b-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.861 DEBUG:boto:path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:32.861 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:32.861 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.861 DEBUG:boto:Path: /dkyges-60/obj00000000000000000017 2024-04-12T03:58:32.861 DEBUG:boto:Data: 2024-04-12T03:58:32.862 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.862 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.862 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.862 DEBUG:boto:Params: {} 2024-04-12T03:58:32.862 DEBUG:boto:Token: None 2024-04-12T03:58:32.862 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000017 2024-04-12T03:58:32.862 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eIDtMBV4Q6myOvB/nKB7AIr9BRc= 2024-04-12T03:58:32.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eIDtMBV4Q6myOvB/nKB7AIr9BRc='} 2024-04-12T03:58:32.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000909f85506f571aa8-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.907 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.907 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.907 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.907 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.907 DEBUG:boto:Data: 2024-04-12T03:58:32.907 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.907 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.907 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.907 DEBUG:boto:Params: {} 2024-04-12T03:58:32.908 DEBUG:boto:Token: None 2024-04-12T03:58:32.908 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.908 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.912 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fe9959c61160fc6-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.912 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:32.912 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:32.912 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.912 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002 2024-04-12T03:58:32.912 DEBUG:boto:Data: 2024-04-12T03:58:32.912 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.912 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.913 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.913 DEBUG:boto:Params: {} 2024-04-12T03:58:32.913 DEBUG:boto:Token: None 2024-04-12T03:58:32.913 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000002 2024-04-12T03:58:32.913 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S1E3k+Og0rO6RTtcI+ZaM/V5U7I= 2024-04-12T03:58:32.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S1E3k+Og0rO6RTtcI+ZaM/V5U7I='} 2024-04-12T03:58:32.921 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6e560b51ece524b-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.921 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.922 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.922 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.922 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.922 DEBUG:boto:Data: 2024-04-12T03:58:32.922 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.922 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.922 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.922 DEBUG:boto:Params: {} 2024-04-12T03:58:32.922 DEBUG:boto:Token: None 2024-04-12T03:58:32.922 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.923 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c6d667e731423f0-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.926 DEBUG:boto:path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:32.926 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:32.926 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.926 DEBUG:boto:Path: /dkyges-60/obj00000000000000000042 2024-04-12T03:58:32.926 DEBUG:boto:Data: 2024-04-12T03:58:32.927 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.927 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.927 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.927 DEBUG:boto:Params: {} 2024-04-12T03:58:32.927 DEBUG:boto:Token: None 2024-04-12T03:58:32.927 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000042 2024-04-12T03:58:32.927 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OQ/kctqQRohjxmhioUc/GRFD4vc= 2024-04-12T03:58:32.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OQ/kctqQRohjxmhioUc/GRFD4vc='} 2024-04-12T03:58:32.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000674a285fce1a3c4b-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.970 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.971 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.971 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.971 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.971 DEBUG:boto:Data: 2024-04-12T03:58:32.971 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.971 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.971 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.971 DEBUG:boto:Params: {} 2024-04-12T03:58:32.971 DEBUG:boto:Token: None 2024-04-12T03:58:32.971 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.971 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000309d9ef035e57689-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.976 DEBUG:boto:path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:32.976 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:32.976 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.976 DEBUG:boto:Path: /dkyges-60/obj00000000000000000027 2024-04-12T03:58:32.976 DEBUG:boto:Data: 2024-04-12T03:58:32.976 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.976 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.976 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.976 DEBUG:boto:Params: {} 2024-04-12T03:58:32.976 DEBUG:boto:Token: None 2024-04-12T03:58:32.977 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000027 2024-04-12T03:58:32.977 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GiKPAvYIZ0GVL8JtdbytJzQ2moA= 2024-04-12T03:58:32.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GiKPAvYIZ0GVL8JtdbytJzQ2moA='} 2024-04-12T03:58:32.984 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000051ceed84895d954-006618b168-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.984 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:32.984 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:32.984 DEBUG:boto:Method: HEAD 2024-04-12T03:58:32.984 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:32.984 DEBUG:boto:Data: 2024-04-12T03:58:32.984 DEBUG:boto:Headers: {} 2024-04-12T03:58:32.985 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.985 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.985 DEBUG:boto:Params: {} 2024-04-12T03:58:32.985 DEBUG:boto:Token: None 2024-04-12T03:58:32.985 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/ 2024-04-12T03:58:32.985 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY= 2024-04-12T03:58:32.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFpTSB+4Yc0XPw88KR1rAqVxEfY=', 'Content-Length': '0'} 2024-04-12T03:58:32.988 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8a2a2aa1843d44f-006618b168-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:32.988 DEBUG:boto:path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:32.989 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:32.989 DEBUG:boto:Method: PUT 2024-04-12T03:58:32.989 DEBUG:boto:Path: /dkyges-60/obj00000000000000000021 2024-04-12T03:58:32.989 DEBUG:boto:Data: 2024-04-12T03:58:32.989 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:32.989 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:32.989 DEBUG:boto:Port: 8000 2024-04-12T03:58:32.989 DEBUG:boto:Params: {} 2024-04-12T03:58:32.989 DEBUG:boto:Token: None 2024-04-12T03:58:32.989 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:32 GMT /dkyges-60/obj00000000000000000021 2024-04-12T03:58:32.990 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hFIW1fWcLRrtIR6VJIbunkEk3MI= 2024-04-12T03:58:32.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:32 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hFIW1fWcLRrtIR6VJIbunkEk3MI='} 2024-04-12T03:58:33.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005cf5dfa630863f94-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.039 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.039 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.039 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.039 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.039 DEBUG:boto:Data: 2024-04-12T03:58:33.039 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.039 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.039 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.039 DEBUG:boto:Params: {} 2024-04-12T03:58:33.039 DEBUG:boto:Token: None 2024-04-12T03:58:33.039 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.040 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.043 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c82b7d129d95fca8-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.043 DEBUG:boto:path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:33.044 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:33.044 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.044 DEBUG:boto:Path: /dkyges-60/obj00000000000000000020 2024-04-12T03:58:33.044 DEBUG:boto:Data: 2024-04-12T03:58:33.044 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.044 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.044 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.044 DEBUG:boto:Params: {} 2024-04-12T03:58:33.044 DEBUG:boto:Token: None 2024-04-12T03:58:33.044 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000020 2024-04-12T03:58:33.045 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H64GeUgufcQiPN0LweKRkD+c8yY= 2024-04-12T03:58:33.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H64GeUgufcQiPN0LweKRkD+c8yY='} 2024-04-12T03:58:33.051 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c178cbe4bd5a009-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.051 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.051 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.052 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.052 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.052 DEBUG:boto:Data: 2024-04-12T03:58:33.052 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.052 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.052 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.052 DEBUG:boto:Params: {} 2024-04-12T03:58:33.052 DEBUG:boto:Token: None 2024-04-12T03:58:33.052 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.052 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.056 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000930a8f4ce86435a0-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.056 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:33.057 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:33.057 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.057 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006 2024-04-12T03:58:33.057 DEBUG:boto:Data: 2024-04-12T03:58:33.057 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.057 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.057 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.057 DEBUG:boto:Params: {} 2024-04-12T03:58:33.057 DEBUG:boto:Token: None 2024-04-12T03:58:33.057 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000006 2024-04-12T03:58:33.058 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:egqUWywCdYlT5mT0O3ejkemC9nw= 2024-04-12T03:58:33.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:egqUWywCdYlT5mT0O3ejkemC9nw='} 2024-04-12T03:58:33.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e47107425f669b88-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.103 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.103 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.103 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.103 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.103 DEBUG:boto:Data: 2024-04-12T03:58:33.103 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.103 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.103 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.103 DEBUG:boto:Params: {} 2024-04-12T03:58:33.103 DEBUG:boto:Token: None 2024-04-12T03:58:33.104 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.104 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.107 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d887b251077c6d01-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.107 DEBUG:boto:path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:33.108 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:33.108 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.108 DEBUG:boto:Path: /dkyges-60/obj00000000000000000031 2024-04-12T03:58:33.108 DEBUG:boto:Data: 2024-04-12T03:58:33.108 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.108 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.108 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.108 DEBUG:boto:Params: {} 2024-04-12T03:58:33.108 DEBUG:boto:Token: None 2024-04-12T03:58:33.108 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000031 2024-04-12T03:58:33.108 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tloJkzAOL7dBZRrjwtKAI/sO0Qk= 2024-04-12T03:58:33.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tloJkzAOL7dBZRrjwtKAI/sO0Qk='} 2024-04-12T03:58:33.115 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004382769046634177-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.115 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.115 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.115 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.116 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.116 DEBUG:boto:Data: 2024-04-12T03:58:33.116 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.116 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.116 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.116 DEBUG:boto:Params: {} 2024-04-12T03:58:33.116 DEBUG:boto:Token: None 2024-04-12T03:58:33.116 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.116 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.120 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d38cd766b8f36242-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.120 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:33.120 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:33.120 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.120 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011 2024-04-12T03:58:33.120 DEBUG:boto:Data: 2024-04-12T03:58:33.120 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.120 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.121 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.121 DEBUG:boto:Params: {} 2024-04-12T03:58:33.121 DEBUG:boto:Token: None 2024-04-12T03:58:33.121 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000011 2024-04-12T03:58:33.121 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VFNf/H6IgjLH93ynreFcZzrS09k= 2024-04-12T03:58:33.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VFNf/H6IgjLH93ynreFcZzrS09k='} 2024-04-12T03:58:33.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7a21b6c88e84301-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.167 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.167 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.167 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.167 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.167 DEBUG:boto:Data: 2024-04-12T03:58:33.167 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.167 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.167 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.167 DEBUG:boto:Params: {} 2024-04-12T03:58:33.167 DEBUG:boto:Token: None 2024-04-12T03:58:33.167 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.167 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009377ca186d0d5763-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.171 DEBUG:boto:path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:33.172 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:33.172 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.172 DEBUG:boto:Path: /dkyges-60/obj00000000000000000015 2024-04-12T03:58:33.172 DEBUG:boto:Data: 2024-04-12T03:58:33.172 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.172 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.172 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.172 DEBUG:boto:Params: {} 2024-04-12T03:58:33.172 DEBUG:boto:Token: None 2024-04-12T03:58:33.172 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000015 2024-04-12T03:58:33.172 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2zjFWFhhsD6Oqrlma3TNpnQvUtY= 2024-04-12T03:58:33.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2zjFWFhhsD6Oqrlma3TNpnQvUtY='} 2024-04-12T03:58:33.181 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f9b3a6d81c52d659-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.181 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.182 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.182 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.182 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.182 DEBUG:boto:Data: 2024-04-12T03:58:33.182 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.182 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.182 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.182 DEBUG:boto:Params: {} 2024-04-12T03:58:33.182 DEBUG:boto:Token: None 2024-04-12T03:58:33.182 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.182 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.186 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a927bfba3ca27c0-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.186 DEBUG:boto:path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:33.187 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:33.187 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.187 DEBUG:boto:Path: /dkyges-60/obj00000000000000000025 2024-04-12T03:58:33.187 DEBUG:boto:Data: 2024-04-12T03:58:33.187 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.187 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.187 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.187 DEBUG:boto:Params: {} 2024-04-12T03:58:33.187 DEBUG:boto:Token: None 2024-04-12T03:58:33.187 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000025 2024-04-12T03:58:33.187 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P3Am+t0DF/tXkxs5UHUSNylX3jM= 2024-04-12T03:58:33.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P3Am+t0DF/tXkxs5UHUSNylX3jM='} 2024-04-12T03:58:33.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002aed997d7e3a461b-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:33.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:33.231 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-60 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:58:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:33.294 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:33.356 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T03:58:33.356 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-60 2024-04-12T03:58:33.369 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 42 2024-04-12T03:58:33.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:33.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:33.438 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 31 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:33.496 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:33.496 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:33.546 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:58:33.546 INFO:teuthology.orchestra.run.smithi043.stdout: "objs": [ 2024-04-12T03:58:33.546 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000019", 2024-04-12T03:58:33.547 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000006", 2024-04-12T03:58:33.547 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000467", 2024-04-12T03:58:33.547 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000005", 2024-04-12T03:58:33.547 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000056", 2024-04-12T03:58:33.547 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000004", 2024-04-12T03:58:33.547 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000023", 2024-04-12T03:58:33.547 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000317", 2024-04-12T03:58:33.547 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000022", 2024-04-12T03:58:33.547 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000000", 2024-04-12T03:58:33.547 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000021", 2024-04-12T03:58:33.547 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000009", 2024-04-12T03:58:33.548 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000271", 2024-04-12T03:58:33.548 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000008", 2024-04-12T03:58:33.548 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000018", 2024-04-12T03:58:33.548 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000007", 2024-04-12T03:58:33.548 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000017", 2024-04-12T03:58:33.548 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000162", 2024-04-12T03:58:33.548 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000016", 2024-04-12T03:58:33.548 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000003", 2024-04-12T03:58:33.548 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000423", 2024-04-12T03:58:33.548 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000002", 2024-04-12T03:58:33.548 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000012", 2024-04-12T03:58:33.548 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000001", 2024-04-12T03:58:33.549 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000011", 2024-04-12T03:58:33.549 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000283", 2024-04-12T03:58:33.549 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000010", 2024-04-12T03:58:33.549 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000061", 2024-04-12T03:58:33.549 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000050", 2024-04-12T03:58:33.549 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000029", 2024-04-12T03:58:33.549 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000133" 2024-04-12T03:58:33.549 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:58:33.549 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:58:33.550 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.550 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.551 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.551 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.551 DEBUG:boto:Data: 2024-04-12T03:58:33.551 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.551 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.551 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.551 DEBUG:boto:Params: {} 2024-04-12T03:58:33.551 DEBUG:boto:Token: None 2024-04-12T03:58:33.551 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.551 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.555 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b88b62712bad53cb-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.556 DEBUG:boto:path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:33.556 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:33.556 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.556 DEBUG:boto:Path: /dkyges-60/obj00000000000000000050 2024-04-12T03:58:33.556 DEBUG:boto:Data: 2024-04-12T03:58:33.556 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.556 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.556 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.556 DEBUG:boto:Params: {} 2024-04-12T03:58:33.556 DEBUG:boto:Token: None 2024-04-12T03:58:33.557 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000050 2024-04-12T03:58:33.557 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yzCHyy5LRSthG0xr7vrJMbxxyJE= 2024-04-12T03:58:33.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yzCHyy5LRSthG0xr7vrJMbxxyJE='} 2024-04-12T03:58:33.564 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e0cf7bcd285b41b-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.564 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.564 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.564 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.564 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.564 DEBUG:boto:Data: 2024-04-12T03:58:33.564 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.564 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.565 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.565 DEBUG:boto:Params: {} 2024-04-12T03:58:33.565 DEBUG:boto:Token: None 2024-04-12T03:58:33.565 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.565 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.568 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b51b1b0738ad6da0-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.569 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:33.569 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:33.569 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.569 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000 2024-04-12T03:58:33.569 DEBUG:boto:Data: 2024-04-12T03:58:33.569 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.569 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.569 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.569 DEBUG:boto:Params: {} 2024-04-12T03:58:33.569 DEBUG:boto:Token: None 2024-04-12T03:58:33.569 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000000 2024-04-12T03:58:33.570 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I4PstUE5LcqMet9e6GlU55FFqwI= 2024-04-12T03:58:33.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I4PstUE5LcqMet9e6GlU55FFqwI='} 2024-04-12T03:58:33.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036dbf2b1e8c5f905-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.614 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.615 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.615 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.615 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.615 DEBUG:boto:Data: 2024-04-12T03:58:33.615 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.615 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.615 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.615 DEBUG:boto:Params: {} 2024-04-12T03:58:33.615 DEBUG:boto:Token: None 2024-04-12T03:58:33.615 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.616 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.619 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031a4843009337c17-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.619 DEBUG:boto:path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:33.619 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:33.619 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.620 DEBUG:boto:Path: /dkyges-60/obj00000000000000000283 2024-04-12T03:58:33.620 DEBUG:boto:Data: 2024-04-12T03:58:33.620 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.620 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.620 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.620 DEBUG:boto:Params: {} 2024-04-12T03:58:33.620 DEBUG:boto:Token: None 2024-04-12T03:58:33.620 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000283 2024-04-12T03:58:33.620 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ofSdfR52GxOgq/pGR5WP0AhS7HE= 2024-04-12T03:58:33.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ofSdfR52GxOgq/pGR5WP0AhS7HE='} 2024-04-12T03:58:33.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007ab25014b687ecd-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.627 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.627 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.627 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.627 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.627 DEBUG:boto:Data: 2024-04-12T03:58:33.627 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.627 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.627 DEBUG:boto:Params: {} 2024-04-12T03:58:33.627 DEBUG:boto:Token: None 2024-04-12T03:58:33.627 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.631 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee9d0cf7570c2cef-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.632 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:33.632 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:33.632 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.632 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010 2024-04-12T03:58:33.632 DEBUG:boto:Data: 2024-04-12T03:58:33.632 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.632 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.632 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.632 DEBUG:boto:Params: {} 2024-04-12T03:58:33.632 DEBUG:boto:Token: None 2024-04-12T03:58:33.632 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000010 2024-04-12T03:58:33.633 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/mx9wDCrxP3nQfu2jiXH+/DJK2Y= 2024-04-12T03:58:33.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/mx9wDCrxP3nQfu2jiXH+/DJK2Y='} 2024-04-12T03:58:33.674 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000434d2db572b420f2-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.675 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.675 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.675 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.675 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.675 DEBUG:boto:Data: 2024-04-12T03:58:33.675 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.675 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.675 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.675 DEBUG:boto:Params: {} 2024-04-12T03:58:33.675 DEBUG:boto:Token: None 2024-04-12T03:58:33.675 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.676 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.679 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000270e7b64b354c3e8-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.679 DEBUG:boto:path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:33.680 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:33.680 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.680 DEBUG:boto:Path: /dkyges-60/obj00000000000000000133 2024-04-12T03:58:33.680 DEBUG:boto:Data: 2024-04-12T03:58:33.680 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.680 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.680 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.680 DEBUG:boto:Params: {} 2024-04-12T03:58:33.680 DEBUG:boto:Token: None 2024-04-12T03:58:33.680 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000133 2024-04-12T03:58:33.681 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ml07+5Kgp3r/bdB1szT38OcNKck= 2024-04-12T03:58:33.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ml07+5Kgp3r/bdB1szT38OcNKck='} 2024-04-12T03:58:33.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000098c636e4976533cc-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.687 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.687 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.687 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.687 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.688 DEBUG:boto:Data: 2024-04-12T03:58:33.688 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.688 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.688 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.688 DEBUG:boto:Params: {} 2024-04-12T03:58:33.688 DEBUG:boto:Token: None 2024-04-12T03:58:33.688 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.688 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.692 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000edcf65605ed77576-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.692 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:33.692 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:33.692 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.692 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005 2024-04-12T03:58:33.693 DEBUG:boto:Data: 2024-04-12T03:58:33.693 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.693 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.693 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.693 DEBUG:boto:Params: {} 2024-04-12T03:58:33.693 DEBUG:boto:Token: None 2024-04-12T03:58:33.693 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000005 2024-04-12T03:58:33.693 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:swFVqm3346d/XkzgyOxjiT6S7WU= 2024-04-12T03:58:33.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:swFVqm3346d/XkzgyOxjiT6S7WU='} 2024-04-12T03:58:33.738 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000817af0c4d6c076a7-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.739 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.739 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.739 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.739 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.739 DEBUG:boto:Data: 2024-04-12T03:58:33.739 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.739 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.739 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.739 DEBUG:boto:Params: {} 2024-04-12T03:58:33.739 DEBUG:boto:Token: None 2024-04-12T03:58:33.739 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.739 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.743 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000243ec65c99a754b6-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.743 DEBUG:boto:path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:33.743 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:33.744 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.744 DEBUG:boto:Path: /dkyges-60/obj00000000000000000022 2024-04-12T03:58:33.744 DEBUG:boto:Data: 2024-04-12T03:58:33.744 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.744 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.744 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.744 DEBUG:boto:Params: {} 2024-04-12T03:58:33.744 DEBUG:boto:Token: None 2024-04-12T03:58:33.744 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000022 2024-04-12T03:58:33.744 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:guUy3Gn/g93kt7EaOGMSn/TTe0Y= 2024-04-12T03:58:33.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:guUy3Gn/g93kt7EaOGMSn/TTe0Y='} 2024-04-12T03:58:33.753 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000870153fa345d8a49-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.753 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.753 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.753 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.753 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.753 DEBUG:boto:Data: 2024-04-12T03:58:33.753 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.753 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.753 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.754 DEBUG:boto:Params: {} 2024-04-12T03:58:33.754 DEBUG:boto:Token: None 2024-04-12T03:58:33.754 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.754 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.758 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b1f10d8bc9a4732-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.758 DEBUG:boto:path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:33.758 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:33.758 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.758 DEBUG:boto:Path: /dkyges-60/obj00000000000000000061 2024-04-12T03:58:33.759 DEBUG:boto:Data: 2024-04-12T03:58:33.759 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.759 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.759 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.759 DEBUG:boto:Params: {} 2024-04-12T03:58:33.759 DEBUG:boto:Token: None 2024-04-12T03:58:33.759 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000061 2024-04-12T03:58:33.759 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4efyNdRsAA2y6CDDVnpeeekETc= 2024-04-12T03:58:33.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4efyNdRsAA2y6CDDVnpeeekETc='} 2024-04-12T03:58:33.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019876201a849b66a-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.803 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.803 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.803 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.803 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.803 DEBUG:boto:Data: 2024-04-12T03:58:33.803 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.803 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.803 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.803 DEBUG:boto:Params: {} 2024-04-12T03:58:33.803 DEBUG:boto:Token: None 2024-04-12T03:58:33.804 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.804 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4dd9a070de3d7ea-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.807 DEBUG:boto:path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:33.807 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:33.807 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.808 DEBUG:boto:Path: /dkyges-60/obj00000000000000000162 2024-04-12T03:58:33.808 DEBUG:boto:Data: 2024-04-12T03:58:33.808 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.808 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.808 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.808 DEBUG:boto:Params: {} 2024-04-12T03:58:33.808 DEBUG:boto:Token: None 2024-04-12T03:58:33.808 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000162 2024-04-12T03:58:33.808 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hjLGooN8sBEs/rW5jVn7NDaSOrA= 2024-04-12T03:58:33.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hjLGooN8sBEs/rW5jVn7NDaSOrA='} 2024-04-12T03:58:33.815 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ba4b8b448242cde-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.815 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.815 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.815 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.815 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.815 DEBUG:boto:Data: 2024-04-12T03:58:33.815 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.815 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.815 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.815 DEBUG:boto:Params: {} 2024-04-12T03:58:33.816 DEBUG:boto:Token: None 2024-04-12T03:58:33.816 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.816 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.819 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007127f49627b731ab-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.819 DEBUG:boto:path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:33.820 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:33.820 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.820 DEBUG:boto:Path: /dkyges-60/obj00000000000000000029 2024-04-12T03:58:33.820 DEBUG:boto:Data: 2024-04-12T03:58:33.820 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.820 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.820 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.820 DEBUG:boto:Params: {} 2024-04-12T03:58:33.820 DEBUG:boto:Token: None 2024-04-12T03:58:33.820 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000029 2024-04-12T03:58:33.821 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cbCk46AIvNBSdhuTlb8OXquvrMc= 2024-04-12T03:58:33.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cbCk46AIvNBSdhuTlb8OXquvrMc='} 2024-04-12T03:58:33.862 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000005c2d2b45c25172-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.862 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.863 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.863 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.863 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.863 DEBUG:boto:Data: 2024-04-12T03:58:33.863 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.863 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.863 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.863 DEBUG:boto:Params: {} 2024-04-12T03:58:33.863 DEBUG:boto:Token: None 2024-04-12T03:58:33.863 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.863 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.867 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db0dd850e7b56b98-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.867 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:33.867 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:33.867 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.868 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011 2024-04-12T03:58:33.868 DEBUG:boto:Data: 2024-04-12T03:58:33.868 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.868 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.868 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.868 DEBUG:boto:Params: {} 2024-04-12T03:58:33.868 DEBUG:boto:Token: None 2024-04-12T03:58:33.868 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000011 2024-04-12T03:58:33.868 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VFNf/H6IgjLH93ynreFcZzrS09k= 2024-04-12T03:58:33.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VFNf/H6IgjLH93ynreFcZzrS09k='} 2024-04-12T03:58:33.876 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000789f9b5c8aa50235-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.877 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.877 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.877 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.877 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.877 DEBUG:boto:Data: 2024-04-12T03:58:33.877 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.877 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.877 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.877 DEBUG:boto:Params: {} 2024-04-12T03:58:33.877 DEBUG:boto:Token: None 2024-04-12T03:58:33.877 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.877 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.881 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c62c4d3bf47cd3c-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.881 DEBUG:boto:path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:33.881 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:33.881 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.881 DEBUG:boto:Path: /dkyges-60/obj00000000000000000271 2024-04-12T03:58:33.881 DEBUG:boto:Data: 2024-04-12T03:58:33.881 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.882 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.882 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.882 DEBUG:boto:Params: {} 2024-04-12T03:58:33.882 DEBUG:boto:Token: None 2024-04-12T03:58:33.882 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000271 2024-04-12T03:58:33.882 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qFxrRheHH6LEU1UcwWKyIi7Y/TY= 2024-04-12T03:58:33.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qFxrRheHH6LEU1UcwWKyIi7Y/TY='} 2024-04-12T03:58:33.926 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ee41303dba8b30cc-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.927 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.927 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.927 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.927 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.927 DEBUG:boto:Data: 2024-04-12T03:58:33.927 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.927 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.927 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.927 DEBUG:boto:Params: {} 2024-04-12T03:58:33.927 DEBUG:boto:Token: None 2024-04-12T03:58:33.928 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.928 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3b889ab81cde752-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.932 DEBUG:boto:path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:33.932 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:33.932 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.932 DEBUG:boto:Path: /dkyges-60/obj00000000000000000016 2024-04-12T03:58:33.932 DEBUG:boto:Data: 2024-04-12T03:58:33.932 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.932 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.932 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.932 DEBUG:boto:Params: {} 2024-04-12T03:58:33.933 DEBUG:boto:Token: None 2024-04-12T03:58:33.933 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000016 2024-04-12T03:58:33.933 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mLO/1Tok8GtG2GnAoMFynmoiOFo= 2024-04-12T03:58:33.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mLO/1Tok8GtG2GnAoMFynmoiOFo='} 2024-04-12T03:58:33.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b872cbe18f861e11-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.943 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.943 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.943 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.943 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.943 DEBUG:boto:Data: 2024-04-12T03:58:33.943 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.943 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.943 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.943 DEBUG:boto:Params: {} 2024-04-12T03:58:33.943 DEBUG:boto:Token: None 2024-04-12T03:58:33.944 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.944 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.947 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011e4728612a400bc-006618b169-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.947 DEBUG:boto:path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:33.947 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:33.947 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.948 DEBUG:boto:Path: /dkyges-60/obj00000000000000000423 2024-04-12T03:58:33.948 DEBUG:boto:Data: 2024-04-12T03:58:33.948 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.948 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.948 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.948 DEBUG:boto:Params: {} 2024-04-12T03:58:33.948 DEBUG:boto:Token: None 2024-04-12T03:58:33.948 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000423 2024-04-12T03:58:33.948 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hwIIVvN4uXAfw0srAAEU5/QKjAs= 2024-04-12T03:58:33.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hwIIVvN4uXAfw0srAAEU5/QKjAs='} 2024-04-12T03:58:33.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006b6e364ff3a85990-006618b169-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.990 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:33.991 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:33.991 DEBUG:boto:Method: HEAD 2024-04-12T03:58:33.991 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:33.991 DEBUG:boto:Data: 2024-04-12T03:58:33.991 DEBUG:boto:Headers: {} 2024-04-12T03:58:33.991 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.991 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.991 DEBUG:boto:Params: {} 2024-04-12T03:58:33.991 DEBUG:boto:Token: None 2024-04-12T03:58:33.991 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/ 2024-04-12T03:58:33.992 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4= 2024-04-12T03:58:33.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WjWT1fYt73xTUAv2r4xLVdpKI4=', 'Content-Length': '0'} 2024-04-12T03:58:33.995 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a78fa28befb55988-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:33.996 DEBUG:boto:path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:33.996 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:33.996 DEBUG:boto:Method: PUT 2024-04-12T03:58:33.996 DEBUG:boto:Path: /dkyges-60/obj00000000000000000007 2024-04-12T03:58:33.996 DEBUG:boto:Data: 2024-04-12T03:58:33.996 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:33.996 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:33.996 DEBUG:boto:Port: 8000 2024-04-12T03:58:33.996 DEBUG:boto:Params: {} 2024-04-12T03:58:33.996 DEBUG:boto:Token: None 2024-04-12T03:58:33.997 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:33 GMT /dkyges-60/obj00000000000000000007 2024-04-12T03:58:33.997 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dD4bwg8UR3tQ08xzsw77V9RflRU= 2024-04-12T03:58:33.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:33 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dD4bwg8UR3tQ08xzsw77V9RflRU='} 2024-04-12T03:58:34.005 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000002caa995035c8895-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.005 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.005 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.005 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.005 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.006 DEBUG:boto:Data: 2024-04-12T03:58:34.006 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.006 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.006 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.006 DEBUG:boto:Params: {} 2024-04-12T03:58:34.007 DEBUG:boto:Token: None 2024-04-12T03:58:34.007 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.007 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.011 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e830a0aa00492f71-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.011 DEBUG:boto:path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:34.011 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:34.011 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.011 DEBUG:boto:Path: /dkyges-60/obj00000000000000000056 2024-04-12T03:58:34.011 DEBUG:boto:Data: 2024-04-12T03:58:34.011 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.011 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.012 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.012 DEBUG:boto:Params: {} 2024-04-12T03:58:34.012 DEBUG:boto:Token: None 2024-04-12T03:58:34.012 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000056 2024-04-12T03:58:34.012 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tX52ZBmdL2oRPLKaZAdIPQ2Sj90= 2024-04-12T03:58:34.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tX52ZBmdL2oRPLKaZAdIPQ2Sj90='} 2024-04-12T03:58:34.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000614e3940a7ea3a32-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.055 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.055 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.055 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.055 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.055 DEBUG:boto:Data: 2024-04-12T03:58:34.055 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.055 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.055 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.055 DEBUG:boto:Params: {} 2024-04-12T03:58:34.055 DEBUG:boto:Token: None 2024-04-12T03:58:34.056 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.056 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.059 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd9f3802c9b39027-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.059 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:34.059 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:34.060 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.060 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002 2024-04-12T03:58:34.060 DEBUG:boto:Data: 2024-04-12T03:58:34.060 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.060 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.060 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.060 DEBUG:boto:Params: {} 2024-04-12T03:58:34.060 DEBUG:boto:Token: None 2024-04-12T03:58:34.060 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000002 2024-04-12T03:58:34.060 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CMWAQCB4MZ/zaBSqIUjZMCxb01U= 2024-04-12T03:58:34.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CMWAQCB4MZ/zaBSqIUjZMCxb01U='} 2024-04-12T03:58:34.069 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d29aa75e845d2554-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.069 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.069 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.069 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.069 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.069 DEBUG:boto:Data: 2024-04-12T03:58:34.069 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.069 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.069 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.070 DEBUG:boto:Params: {} 2024-04-12T03:58:34.070 DEBUG:boto:Token: None 2024-04-12T03:58:34.070 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.070 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.074 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011d1a20e6c86979e-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.074 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:34.074 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:34.075 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.075 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001 2024-04-12T03:58:34.075 DEBUG:boto:Data: 2024-04-12T03:58:34.075 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.075 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.075 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.075 DEBUG:boto:Params: {} 2024-04-12T03:58:34.075 DEBUG:boto:Token: None 2024-04-12T03:58:34.075 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000001 2024-04-12T03:58:34.076 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hTskkEfoudgmHXyAVIbnesvOt3g= 2024-04-12T03:58:34.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hTskkEfoudgmHXyAVIbnesvOt3g='} 2024-04-12T03:58:34.118 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000240c5b9dbe74bc4c-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.119 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.119 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.119 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.119 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.119 DEBUG:boto:Data: 2024-04-12T03:58:34.119 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.119 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.119 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.119 DEBUG:boto:Params: {} 2024-04-12T03:58:34.119 DEBUG:boto:Token: None 2024-04-12T03:58:34.120 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.120 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.123 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e60c7f92e4e895f-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.123 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:34.124 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:34.124 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.124 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004 2024-04-12T03:58:34.124 DEBUG:boto:Data: 2024-04-12T03:58:34.124 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.124 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.124 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.124 DEBUG:boto:Params: {} 2024-04-12T03:58:34.124 DEBUG:boto:Token: None 2024-04-12T03:58:34.124 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000004 2024-04-12T03:58:34.125 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:92c4ZMOC/qBnGQCo//iDgRus4ao= 2024-04-12T03:58:34.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:92c4ZMOC/qBnGQCo//iDgRus4ao='} 2024-04-12T03:58:34.133 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000407daca073ab0efe-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.133 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.133 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.133 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.133 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.133 DEBUG:boto:Data: 2024-04-12T03:58:34.133 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.133 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.133 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.134 DEBUG:boto:Params: {} 2024-04-12T03:58:34.134 DEBUG:boto:Token: None 2024-04-12T03:58:34.134 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.134 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.138 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f0187d49ef55f27-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.138 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:34.138 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:34.138 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.138 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006 2024-04-12T03:58:34.138 DEBUG:boto:Data: 2024-04-12T03:58:34.138 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.139 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.139 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.139 DEBUG:boto:Params: {} 2024-04-12T03:58:34.139 DEBUG:boto:Token: None 2024-04-12T03:58:34.139 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000006 2024-04-12T03:58:34.139 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yHInsR9TrJ5Y3dPeIXwkxXCO3IY= 2024-04-12T03:58:34.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yHInsR9TrJ5Y3dPeIXwkxXCO3IY='} 2024-04-12T03:58:34.182 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a7a1b51f346ff118-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.183 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.183 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.183 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.183 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.183 DEBUG:boto:Data: 2024-04-12T03:58:34.183 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.183 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.183 DEBUG:boto:Params: {} 2024-04-12T03:58:34.184 DEBUG:boto:Token: None 2024-04-12T03:58:34.184 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.184 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.188 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1917ac5a5e44719-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.188 DEBUG:boto:path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:34.188 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:34.188 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.189 DEBUG:boto:Path: /dkyges-60/obj00000000000000000008 2024-04-12T03:58:34.189 DEBUG:boto:Data: 2024-04-12T03:58:34.189 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.189 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.189 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.189 DEBUG:boto:Params: {} 2024-04-12T03:58:34.189 DEBUG:boto:Token: None 2024-04-12T03:58:34.189 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000008 2024-04-12T03:58:34.189 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/1e5QSUkI5qqZ6znykHCQDJqlvI= 2024-04-12T03:58:34.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/1e5QSUkI5qqZ6znykHCQDJqlvI='} 2024-04-12T03:58:34.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000815a1bf8005e4de9-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.198 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.198 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.198 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.198 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.198 DEBUG:boto:Data: 2024-04-12T03:58:34.198 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.198 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.199 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.199 DEBUG:boto:Params: {} 2024-04-12T03:58:34.199 DEBUG:boto:Token: None 2024-04-12T03:58:34.199 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.199 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.203 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008848fd92bedd9b2e-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.203 DEBUG:boto:path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:34.204 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:34.204 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.204 DEBUG:boto:Path: /dkyges-60/obj00000000000000000023 2024-04-12T03:58:34.204 DEBUG:boto:Data: 2024-04-12T03:58:34.204 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.204 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.204 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.204 DEBUG:boto:Params: {} 2024-04-12T03:58:34.204 DEBUG:boto:Token: None 2024-04-12T03:58:34.205 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000023 2024-04-12T03:58:34.205 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T78DIxDbvNd7E/ne5yeRzHa6mz0= 2024-04-12T03:58:34.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T78DIxDbvNd7E/ne5yeRzHa6mz0='} 2024-04-12T03:58:34.246 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ec5e56d899baaf4-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.247 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.247 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.247 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.247 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.247 DEBUG:boto:Data: 2024-04-12T03:58:34.247 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.247 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.247 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.247 DEBUG:boto:Params: {} 2024-04-12T03:58:34.247 DEBUG:boto:Token: None 2024-04-12T03:58:34.247 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.248 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.251 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9263bdff5b4b009-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.251 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:34.252 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:34.252 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.252 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012 2024-04-12T03:58:34.252 DEBUG:boto:Data: 2024-04-12T03:58:34.252 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.252 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.252 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.252 DEBUG:boto:Params: {} 2024-04-12T03:58:34.252 DEBUG:boto:Token: None 2024-04-12T03:58:34.253 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000012 2024-04-12T03:58:34.253 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:COlbxzEDKbzyupY7YqCL1OftifQ= 2024-04-12T03:58:34.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:COlbxzEDKbzyupY7YqCL1OftifQ='} 2024-04-12T03:58:34.261 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025563d5d6c553f1f-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.261 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.261 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.261 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.261 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.262 DEBUG:boto:Data: 2024-04-12T03:58:34.262 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.262 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.262 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.262 DEBUG:boto:Params: {} 2024-04-12T03:58:34.262 DEBUG:boto:Token: None 2024-04-12T03:58:34.262 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.262 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.267 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b47787df553b93d-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.267 DEBUG:boto:path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:34.267 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:34.267 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.267 DEBUG:boto:Path: /dkyges-60/obj00000000000000000017 2024-04-12T03:58:34.267 DEBUG:boto:Data: 2024-04-12T03:58:34.267 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.267 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.267 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.268 DEBUG:boto:Params: {} 2024-04-12T03:58:34.268 DEBUG:boto:Token: None 2024-04-12T03:58:34.268 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000017 2024-04-12T03:58:34.268 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:36bWqWKYY4VMdRe7isbllHTzHZk= 2024-04-12T03:58:34.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:36bWqWKYY4VMdRe7isbllHTzHZk='} 2024-04-12T03:58:34.310 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015ee71dddb31dc51-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.311 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.311 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.311 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.311 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.311 DEBUG:boto:Data: 2024-04-12T03:58:34.311 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.311 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.311 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.311 DEBUG:boto:Params: {} 2024-04-12T03:58:34.311 DEBUG:boto:Token: None 2024-04-12T03:58:34.312 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.312 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.316 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ce13928695a6923f-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.316 DEBUG:boto:path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:34.316 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:34.316 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.316 DEBUG:boto:Path: /dkyges-60/obj00000000000000000467 2024-04-12T03:58:34.316 DEBUG:boto:Data: 2024-04-12T03:58:34.317 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.317 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.317 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.317 DEBUG:boto:Params: {} 2024-04-12T03:58:34.317 DEBUG:boto:Token: None 2024-04-12T03:58:34.317 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000467 2024-04-12T03:58:34.317 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:o4VedsZaY5dMm1MTrM5qDES6Ods= 2024-04-12T03:58:34.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:o4VedsZaY5dMm1MTrM5qDES6Ods='} 2024-04-12T03:58:34.325 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058f4cef4fcac93a9-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.325 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.325 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.325 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.325 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.325 DEBUG:boto:Data: 2024-04-12T03:58:34.325 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.325 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.325 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.326 DEBUG:boto:Params: {} 2024-04-12T03:58:34.326 DEBUG:boto:Token: None 2024-04-12T03:58:34.326 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.326 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.330 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008eb314ae635ba3db-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.330 DEBUG:boto:path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:34.331 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:34.331 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.331 DEBUG:boto:Path: /dkyges-60/obj00000000000000000317 2024-04-12T03:58:34.331 DEBUG:boto:Data: 2024-04-12T03:58:34.331 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.331 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.331 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.331 DEBUG:boto:Params: {} 2024-04-12T03:58:34.331 DEBUG:boto:Token: None 2024-04-12T03:58:34.331 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000317 2024-04-12T03:58:34.332 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z94YpAsGbzcPrXo+ERemqb0WeeA= 2024-04-12T03:58:34.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z94YpAsGbzcPrXo+ERemqb0WeeA='} 2024-04-12T03:58:34.374 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4e41dfba54bb7a5-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.375 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.375 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.375 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.375 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.375 DEBUG:boto:Data: 2024-04-12T03:58:34.375 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.375 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.375 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.375 DEBUG:boto:Params: {} 2024-04-12T03:58:34.376 DEBUG:boto:Token: None 2024-04-12T03:58:34.376 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.376 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.380 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007dd0a914a4d8685e-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.380 DEBUG:boto:path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:34.380 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:34.380 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.380 DEBUG:boto:Path: /dkyges-60/obj00000000000000000009 2024-04-12T03:58:34.380 DEBUG:boto:Data: 2024-04-12T03:58:34.380 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.381 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.381 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.381 DEBUG:boto:Params: {} 2024-04-12T03:58:34.381 DEBUG:boto:Token: None 2024-04-12T03:58:34.381 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000009 2024-04-12T03:58:34.381 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6hSymsTBxo7y77IIqlgshAje0cQ= 2024-04-12T03:58:34.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6hSymsTBxo7y77IIqlgshAje0cQ='} 2024-04-12T03:58:34.389 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054c4d5783a23bbe1-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.389 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.390 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.390 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.390 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.390 DEBUG:boto:Data: 2024-04-12T03:58:34.390 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.390 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.390 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.390 DEBUG:boto:Params: {} 2024-04-12T03:58:34.390 DEBUG:boto:Token: None 2024-04-12T03:58:34.390 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.390 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.394 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a9d884fd64a6e52f-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.395 DEBUG:boto:path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:34.395 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:34.395 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.395 DEBUG:boto:Path: /dkyges-60/obj00000000000000000018 2024-04-12T03:58:34.395 DEBUG:boto:Data: 2024-04-12T03:58:34.395 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.395 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.395 DEBUG:boto:Params: {} 2024-04-12T03:58:34.396 DEBUG:boto:Token: None 2024-04-12T03:58:34.396 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000018 2024-04-12T03:58:34.396 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t9cbn5J6TB/EtACIeugD2uVgeOU= 2024-04-12T03:58:34.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t9cbn5J6TB/EtACIeugD2uVgeOU='} 2024-04-12T03:58:34.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b31e3a7fbeaf2deb-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:34.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:34.439 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-60 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:58:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:34.501 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:34.561 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T03:58:34.561 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-60 2024-04-12T03:58:34.575 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 54 2024-04-12T03:58:34.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:34.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:34.649 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 37 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:34.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T03:58:34.756 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:58:34.756 INFO:teuthology.orchestra.run.smithi043.stdout: "objs": [ 2024-04-12T03:58:34.756 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000060", 2024-04-12T03:58:34.756 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000000", 2024-04-12T03:58:34.756 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000011", 2024-04-12T03:58:34.757 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000001", 2024-04-12T03:58:34.757 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000033", 2024-04-12T03:58:34.757 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000065", 2024-04-12T03:58:34.757 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000005", 2024-04-12T03:58:34.757 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000016", 2024-04-12T03:58:34.757 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000048", 2024-04-12T03:58:34.757 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000015", 2024-04-12T03:58:34.757 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000026", 2024-04-12T03:58:34.757 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000030", 2024-04-12T03:58:34.757 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000062", 2024-04-12T03:58:34.757 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000009", 2024-04-12T03:58:34.758 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000040", 2024-04-12T03:58:34.758 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000024", 2024-04-12T03:58:34.758 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000012", 2024-04-12T03:58:34.758 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000002", 2024-04-12T03:58:34.758 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000013", 2024-04-12T03:58:34.758 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000045", 2024-04-12T03:58:34.758 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000006", 2024-04-12T03:58:34.758 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000017", 2024-04-12T03:58:34.758 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000007", 2024-04-12T03:58:34.758 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000039", 2024-04-12T03:58:34.758 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000027", 2024-04-12T03:58:34.758 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000010", 2024-04-12T03:58:34.759 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000021", 2024-04-12T03:58:34.759 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000053", 2024-04-12T03:58:34.759 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000020", 2024-04-12T03:58:34.759 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000004", 2024-04-12T03:58:34.759 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000036", 2024-04-12T03:58:34.759 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000003", 2024-04-12T03:58:34.759 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000014", 2024-04-12T03:58:34.759 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000019", 2024-04-12T03:58:34.759 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000057", 2024-04-12T03:58:34.759 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000018", 2024-04-12T03:58:34.759 INFO:teuthology.orchestra.run.smithi043.stdout: "obj00000000000000000008" 2024-04-12T03:58:34.759 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:58:34.760 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:58:34.761 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.761 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.761 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.761 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.761 DEBUG:boto:Data: 2024-04-12T03:58:34.761 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.761 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.761 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.761 DEBUG:boto:Params: {} 2024-04-12T03:58:34.761 DEBUG:boto:Token: None 2024-04-12T03:58:34.762 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.762 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.766 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097a8a49322f9743e-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.767 DEBUG:boto:path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:34.767 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:34.767 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.767 DEBUG:boto:Path: /dkyges-60/obj00000000000000000030 2024-04-12T03:58:34.767 DEBUG:boto:Data: 2024-04-12T03:58:34.767 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.767 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.767 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.767 DEBUG:boto:Params: {} 2024-04-12T03:58:34.767 DEBUG:boto:Token: None 2024-04-12T03:58:34.768 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000030 2024-04-12T03:58:34.768 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7bhgSbULBd+u3eUCo/564aNNfy0= 2024-04-12T03:58:34.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7bhgSbULBd+u3eUCo/564aNNfy0='} 2024-04-12T03:58:34.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003513c0c3735a7cc0-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.775 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.775 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.775 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.775 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.776 DEBUG:boto:Data: 2024-04-12T03:58:34.776 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.776 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.776 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.776 DEBUG:boto:Params: {} 2024-04-12T03:58:34.776 DEBUG:boto:Token: None 2024-04-12T03:58:34.776 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.776 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002dc5fddc8c23e97c-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.780 DEBUG:boto:path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:34.780 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:34.781 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.781 DEBUG:boto:Path: /dkyges-60/obj00000000000000000062 2024-04-12T03:58:34.781 DEBUG:boto:Data: 2024-04-12T03:58:34.781 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.781 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.781 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.781 DEBUG:boto:Params: {} 2024-04-12T03:58:34.781 DEBUG:boto:Token: None 2024-04-12T03:58:34.781 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000062 2024-04-12T03:58:34.781 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wm8PTrtiIXdFCwYDx21BF1jlIn4= 2024-04-12T03:58:34.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wm8PTrtiIXdFCwYDx21BF1jlIn4='} 2024-04-12T03:58:34.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000341f4ac314502b5-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.827 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.827 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.827 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.827 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.827 DEBUG:boto:Data: 2024-04-12T03:58:34.827 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.827 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.827 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.827 DEBUG:boto:Params: {} 2024-04-12T03:58:34.827 DEBUG:boto:Token: None 2024-04-12T03:58:34.827 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.828 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.831 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d0edc56990c373ca-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.831 DEBUG:boto:path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:34.832 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:34.832 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.832 DEBUG:boto:Path: /dkyges-60/obj00000000000000000027 2024-04-12T03:58:34.832 DEBUG:boto:Data: 2024-04-12T03:58:34.832 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.832 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.832 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.832 DEBUG:boto:Params: {} 2024-04-12T03:58:34.832 DEBUG:boto:Token: None 2024-04-12T03:58:34.832 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000027 2024-04-12T03:58:34.832 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zWP/KH8bCeE8Pw08Hao0a0AvDVU= 2024-04-12T03:58:34.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zWP/KH8bCeE8Pw08Hao0a0AvDVU='} 2024-04-12T03:58:34.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081f4e93fe4293bb2-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.840 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.840 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.840 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.840 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.840 DEBUG:boto:Data: 2024-04-12T03:58:34.840 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.841 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.841 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.841 DEBUG:boto:Params: {} 2024-04-12T03:58:34.841 DEBUG:boto:Token: None 2024-04-12T03:58:34.841 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.841 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.845 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076f276e974f19e3f-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.845 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:34.845 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:34.845 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.845 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004 2024-04-12T03:58:34.846 DEBUG:boto:Data: 2024-04-12T03:58:34.846 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.846 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.846 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.846 DEBUG:boto:Params: {} 2024-04-12T03:58:34.846 DEBUG:boto:Token: None 2024-04-12T03:58:34.846 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000004 2024-04-12T03:58:34.846 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:92c4ZMOC/qBnGQCo//iDgRus4ao= 2024-04-12T03:58:34.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:92c4ZMOC/qBnGQCo//iDgRus4ao='} 2024-04-12T03:58:34.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007aff1a469831b3ee-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.891 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.891 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.891 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.891 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.891 DEBUG:boto:Data: 2024-04-12T03:58:34.891 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.891 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.891 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.891 DEBUG:boto:Params: {} 2024-04-12T03:58:34.891 DEBUG:boto:Token: None 2024-04-12T03:58:34.892 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.892 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.896 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e4fd1ae7b614575-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.896 DEBUG:boto:path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:34.896 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:34.896 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.896 DEBUG:boto:Path: /dkyges-60/obj00000000000000000003 2024-04-12T03:58:34.896 DEBUG:boto:Data: 2024-04-12T03:58:34.897 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.897 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.897 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.897 DEBUG:boto:Params: {} 2024-04-12T03:58:34.897 DEBUG:boto:Token: None 2024-04-12T03:58:34.897 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000003 2024-04-12T03:58:34.897 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YD4fSH5MmiFtob/R0UqnTn+2T4k= 2024-04-12T03:58:34.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YD4fSH5MmiFtob/R0UqnTn+2T4k='} 2024-04-12T03:58:34.905 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b2ea762bb267656-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.905 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.906 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.906 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.906 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.906 DEBUG:boto:Data: 2024-04-12T03:58:34.906 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.906 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.906 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.906 DEBUG:boto:Params: {} 2024-04-12T03:58:34.906 DEBUG:boto:Token: None 2024-04-12T03:58:34.906 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.906 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.910 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af16b8b73d19caae-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.910 DEBUG:boto:path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:34.911 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:34.911 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.911 DEBUG:boto:Path: /dkyges-60/obj00000000000000000015 2024-04-12T03:58:34.911 DEBUG:boto:Data: 2024-04-12T03:58:34.911 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.911 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.911 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.911 DEBUG:boto:Params: {} 2024-04-12T03:58:34.911 DEBUG:boto:Token: None 2024-04-12T03:58:34.911 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000015 2024-04-12T03:58:34.911 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OVlCDEgRPDYgU82VpaBdWPq0W7M= 2024-04-12T03:58:34.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OVlCDEgRPDYgU82VpaBdWPq0W7M='} 2024-04-12T03:58:34.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000928ee3547ba455cc-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.955 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.955 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.955 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.955 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.955 DEBUG:boto:Data: 2024-04-12T03:58:34.955 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.955 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.955 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.955 DEBUG:boto:Params: {} 2024-04-12T03:58:34.956 DEBUG:boto:Token: None 2024-04-12T03:58:34.956 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.956 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.960 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db0dc282dc86b40c-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.960 DEBUG:boto:path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:34.961 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:34.961 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.961 DEBUG:boto:Path: /dkyges-60/obj00000000000000000024 2024-04-12T03:58:34.961 DEBUG:boto:Data: 2024-04-12T03:58:34.961 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.961 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.961 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.961 DEBUG:boto:Params: {} 2024-04-12T03:58:34.961 DEBUG:boto:Token: None 2024-04-12T03:58:34.962 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000024 2024-04-12T03:58:34.962 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z7gKqlAa9iY/R+z7B8H3Zt1tzGc= 2024-04-12T03:58:34.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z7gKqlAa9iY/R+z7B8H3Zt1tzGc='} 2024-04-12T03:58:34.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000057fdb8e9c4f8927e-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.970 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:34.970 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:34.970 DEBUG:boto:Method: HEAD 2024-04-12T03:58:34.970 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:34.970 DEBUG:boto:Data: 2024-04-12T03:58:34.971 DEBUG:boto:Headers: {} 2024-04-12T03:58:34.971 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.971 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.971 DEBUG:boto:Params: {} 2024-04-12T03:58:34.971 DEBUG:boto:Token: None 2024-04-12T03:58:34.971 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/ 2024-04-12T03:58:34.971 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8= 2024-04-12T03:58:34.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i13/LDOEnwZi9pF0X/zJnGjn7e8=', 'Content-Length': '0'} 2024-04-12T03:58:34.975 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e860d53eb92373d-006618b16a-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:34.976 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:34.976 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:34.976 DEBUG:boto:Method: PUT 2024-04-12T03:58:34.976 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013 2024-04-12T03:58:34.976 DEBUG:boto:Data: 2024-04-12T03:58:34.976 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:34.976 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:34.976 DEBUG:boto:Port: 8000 2024-04-12T03:58:34.976 DEBUG:boto:Params: {} 2024-04-12T03:58:34.976 DEBUG:boto:Token: None 2024-04-12T03:58:34.977 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:34 GMT /dkyges-60/obj00000000000000000013 2024-04-12T03:58:34.977 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/ldjAMUiRY+vIfAQCnju0Vm+MfI= 2024-04-12T03:58:34.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:34 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/ldjAMUiRY+vIfAQCnju0Vm+MfI='} 2024-04-12T03:58:35.022 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c27a48994a829f32-006618b16a-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.023 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.023 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.023 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.023 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.023 DEBUG:boto:Data: 2024-04-12T03:58:35.023 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.023 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.023 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.023 DEBUG:boto:Params: {} 2024-04-12T03:58:35.023 DEBUG:boto:Token: None 2024-04-12T03:58:35.023 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.024 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.028 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016474293e7ab5348-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.028 DEBUG:boto:path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:35.028 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:35.028 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.028 DEBUG:boto:Path: /dkyges-60/obj00000000000000000008 2024-04-12T03:58:35.028 DEBUG:boto:Data: 2024-04-12T03:58:35.028 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.028 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.028 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.028 DEBUG:boto:Params: {} 2024-04-12T03:58:35.029 DEBUG:boto:Token: None 2024-04-12T03:58:35.029 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000008 2024-04-12T03:58:35.029 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jF5MUeVgSG2JvQg4srkWGcRVu1k= 2024-04-12T03:58:35.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jF5MUeVgSG2JvQg4srkWGcRVu1k='} 2024-04-12T03:58:35.038 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb59c7761908191e-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.038 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.038 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.038 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.038 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.038 DEBUG:boto:Data: 2024-04-12T03:58:35.038 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.038 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.038 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.039 DEBUG:boto:Params: {} 2024-04-12T03:58:35.039 DEBUG:boto:Token: None 2024-04-12T03:58:35.039 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.039 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.043 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ec118759355833f-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.043 DEBUG:boto:path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:35.043 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:35.043 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.043 DEBUG:boto:Path: /dkyges-60/obj00000000000000000040 2024-04-12T03:58:35.043 DEBUG:boto:Data: 2024-04-12T03:58:35.043 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.043 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.043 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.044 DEBUG:boto:Params: {} 2024-04-12T03:58:35.044 DEBUG:boto:Token: None 2024-04-12T03:58:35.044 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000040 2024-04-12T03:58:35.044 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u9lTDFXrmcM/z1riBe/4Y1WZaWI= 2024-04-12T03:58:35.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u9lTDFXrmcM/z1riBe/4Y1WZaWI='} 2024-04-12T03:58:35.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c16bf7f8a91d2e38-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.086 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.087 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.087 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.087 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.087 DEBUG:boto:Data: 2024-04-12T03:58:35.087 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.087 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.087 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.087 DEBUG:boto:Params: {} 2024-04-12T03:58:35.087 DEBUG:boto:Token: None 2024-04-12T03:58:35.088 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.088 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.092 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb3fa36bb1095512-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.092 DEBUG:boto:path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:35.092 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:35.092 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.092 DEBUG:boto:Path: /dkyges-60/obj00000000000000000009 2024-04-12T03:58:35.092 DEBUG:boto:Data: 2024-04-12T03:58:35.093 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.093 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.093 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.093 DEBUG:boto:Params: {} 2024-04-12T03:58:35.093 DEBUG:boto:Token: None 2024-04-12T03:58:35.093 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000009 2024-04-12T03:58:35.093 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DveipDc3IUgVvM7C2luyHsknqAQ= 2024-04-12T03:58:35.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DveipDc3IUgVvM7C2luyHsknqAQ='} 2024-04-12T03:58:35.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd40ef58af432e2a-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.102 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.102 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.102 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.102 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.103 DEBUG:boto:Data: 2024-04-12T03:58:35.103 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.103 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.103 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.103 DEBUG:boto:Params: {} 2024-04-12T03:58:35.103 DEBUG:boto:Token: None 2024-04-12T03:58:35.103 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.103 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.108 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001761824827db1c17-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.108 DEBUG:boto:path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:35.108 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:35.108 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.108 DEBUG:boto:Path: /dkyges-60/obj00000000000000000018 2024-04-12T03:58:35.108 DEBUG:boto:Data: 2024-04-12T03:58:35.108 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.108 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.108 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.109 DEBUG:boto:Params: {} 2024-04-12T03:58:35.109 DEBUG:boto:Token: None 2024-04-12T03:58:35.109 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000018 2024-04-12T03:58:35.109 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C4GrPuEbCe2PN6/LLgtpyo2BTok= 2024-04-12T03:58:35.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C4GrPuEbCe2PN6/LLgtpyo2BTok='} 2024-04-12T03:58:35.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000580f8cc2e987da4a-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.155 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.155 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.155 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.155 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.155 DEBUG:boto:Data: 2024-04-12T03:58:35.155 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.155 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.155 DEBUG:boto:Params: {} 2024-04-12T03:58:35.155 DEBUG:boto:Token: None 2024-04-12T03:58:35.155 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.160 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005cba4f7ea48a5d6c-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.160 DEBUG:boto:path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:35.160 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:35.160 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.160 DEBUG:boto:Path: /dkyges-60/obj00000000000000000048 2024-04-12T03:58:35.160 DEBUG:boto:Data: 2024-04-12T03:58:35.161 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.161 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.161 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.161 DEBUG:boto:Params: {} 2024-04-12T03:58:35.161 DEBUG:boto:Token: None 2024-04-12T03:58:35.161 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000048 2024-04-12T03:58:35.161 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D+cTjEjg+QUQHdr03nfKyPqAuiI= 2024-04-12T03:58:35.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D+cTjEjg+QUQHdr03nfKyPqAuiI='} 2024-04-12T03:58:35.168 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c574acddc832ab8d-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.169 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.169 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.169 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.169 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.169 DEBUG:boto:Data: 2024-04-12T03:58:35.169 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.169 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.169 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.169 DEBUG:boto:Params: {} 2024-04-12T03:58:35.169 DEBUG:boto:Token: None 2024-04-12T03:58:35.170 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.170 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.174 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068d08dde84e39192-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.174 DEBUG:boto:path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:35.179 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:35.179 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.179 DEBUG:boto:Path: /dkyges-60/obj00000000000000000036 2024-04-12T03:58:35.179 DEBUG:boto:Data: 2024-04-12T03:58:35.179 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.179 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.179 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.179 DEBUG:boto:Params: {} 2024-04-12T03:58:35.180 DEBUG:boto:Token: None 2024-04-12T03:58:35.180 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000036 2024-04-12T03:58:35.180 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i9Q5fsYlWPefvDIpJb0UCKzm1SM= 2024-04-12T03:58:35.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i9Q5fsYlWPefvDIpJb0UCKzm1SM='} 2024-04-12T03:58:35.222 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d33b0fb0c81d8552-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.222 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.223 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.223 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.223 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.223 DEBUG:boto:Data: 2024-04-12T03:58:35.223 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.223 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.223 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.223 DEBUG:boto:Params: {} 2024-04-12T03:58:35.223 DEBUG:boto:Token: None 2024-04-12T03:58:35.224 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.224 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c201fc3918bdfc0-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.228 DEBUG:boto:path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:35.228 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:35.228 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.229 DEBUG:boto:Path: /dkyges-60/obj00000000000000000045 2024-04-12T03:58:35.229 DEBUG:boto:Data: 2024-04-12T03:58:35.229 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.229 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.229 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.229 DEBUG:boto:Params: {} 2024-04-12T03:58:35.229 DEBUG:boto:Token: None 2024-04-12T03:58:35.229 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000045 2024-04-12T03:58:35.229 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YTBnGtEjlwlp+Doo4rqmFyjS+S0= 2024-04-12T03:58:35.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YTBnGtEjlwlp+Doo4rqmFyjS+S0='} 2024-04-12T03:58:35.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db7a477b83f04dd0-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.238 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.238 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.238 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.238 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.238 DEBUG:boto:Data: 2024-04-12T03:58:35.238 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.238 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.239 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.239 DEBUG:boto:Params: {} 2024-04-12T03:58:35.239 DEBUG:boto:Token: None 2024-04-12T03:58:35.239 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.239 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.243 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a4ba401eecc23da-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.243 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:35.244 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:35.244 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.244 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001 2024-04-12T03:58:35.244 DEBUG:boto:Data: 2024-04-12T03:58:35.244 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.244 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.244 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.244 DEBUG:boto:Params: {} 2024-04-12T03:58:35.244 DEBUG:boto:Token: None 2024-04-12T03:58:35.245 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000001 2024-04-12T03:58:35.245 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GSjlMWs9HaFwu4XVL4AsMEVn9cw= 2024-04-12T03:58:35.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GSjlMWs9HaFwu4XVL4AsMEVn9cw='} 2024-04-12T03:58:35.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6f5c556270a1df5-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.291 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.291 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.291 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.291 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.291 DEBUG:boto:Data: 2024-04-12T03:58:35.291 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.291 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.291 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.291 DEBUG:boto:Params: {} 2024-04-12T03:58:35.291 DEBUG:boto:Token: None 2024-04-12T03:58:35.292 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.292 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.296 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0ca65552c5e8c19-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.296 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:35.296 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:35.296 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.296 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002 2024-04-12T03:58:35.297 DEBUG:boto:Data: 2024-04-12T03:58:35.297 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.297 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.297 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.297 DEBUG:boto:Params: {} 2024-04-12T03:58:35.297 DEBUG:boto:Token: None 2024-04-12T03:58:35.297 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000002 2024-04-12T03:58:35.297 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:p3B0BRk13eVCdIYzeNrnEvMITdU= 2024-04-12T03:58:35.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p3B0BRk13eVCdIYzeNrnEvMITdU='} 2024-04-12T03:58:35.307 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097a3d1b129aad8e5-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.307 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.307 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.307 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.307 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.307 DEBUG:boto:Data: 2024-04-12T03:58:35.307 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.307 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.308 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.308 DEBUG:boto:Params: {} 2024-04-12T03:58:35.308 DEBUG:boto:Token: None 2024-04-12T03:58:35.308 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.308 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.312 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093aacf801790de02-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.312 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:35.312 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:35.312 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.313 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006 2024-04-12T03:58:35.313 DEBUG:boto:Data: 2024-04-12T03:58:35.313 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.313 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.313 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.313 DEBUG:boto:Params: {} 2024-04-12T03:58:35.313 DEBUG:boto:Token: None 2024-04-12T03:58:35.313 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000006 2024-04-12T03:58:35.313 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cyC5BgEW6YG1NjpQ7mcZqcPn/Qg= 2024-04-12T03:58:35.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cyC5BgEW6YG1NjpQ7mcZqcPn/Qg='} 2024-04-12T03:58:35.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004cdaf987c4fdcd18-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.359 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.359 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.359 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.359 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.359 DEBUG:boto:Data: 2024-04-12T03:58:35.359 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.359 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.359 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.359 DEBUG:boto:Params: {} 2024-04-12T03:58:35.360 DEBUG:boto:Token: None 2024-04-12T03:58:35.360 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.360 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.364 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d76c1c7b97d6dba7-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.364 DEBUG:boto:path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:35.364 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:35.364 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.364 DEBUG:boto:Path: /dkyges-60/obj00000000000000000039 2024-04-12T03:58:35.365 DEBUG:boto:Data: 2024-04-12T03:58:35.365 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.365 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.365 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.365 DEBUG:boto:Params: {} 2024-04-12T03:58:35.365 DEBUG:boto:Token: None 2024-04-12T03:58:35.365 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000039 2024-04-12T03:58:35.365 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KpjeqMWAUm0nekNPa3rdbK80l18= 2024-04-12T03:58:35.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KpjeqMWAUm0nekNPa3rdbK80l18='} 2024-04-12T03:58:35.373 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073530f56d6ddb2af-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.374 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.374 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.374 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.374 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.374 DEBUG:boto:Data: 2024-04-12T03:58:35.374 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.374 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.374 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.374 DEBUG:boto:Params: {} 2024-04-12T03:58:35.375 DEBUG:boto:Token: None 2024-04-12T03:58:35.375 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.375 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.379 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a19c7cadcb0b859c-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.380 DEBUG:boto:path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:35.380 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:35.380 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.380 DEBUG:boto:Path: /dkyges-60/obj00000000000000000057 2024-04-12T03:58:35.380 DEBUG:boto:Data: 2024-04-12T03:58:35.380 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.380 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.380 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.380 DEBUG:boto:Params: {} 2024-04-12T03:58:35.380 DEBUG:boto:Token: None 2024-04-12T03:58:35.381 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000057 2024-04-12T03:58:35.381 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7MMucS6kuKc3z85b7IkYO1++hUo= 2024-04-12T03:58:35.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7MMucS6kuKc3z85b7IkYO1++hUo='} 2024-04-12T03:58:35.423 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052a8e21c50874d1f-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.423 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.423 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.423 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.423 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.423 DEBUG:boto:Data: 2024-04-12T03:58:35.423 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.423 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.424 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.424 DEBUG:boto:Params: {} 2024-04-12T03:58:35.424 DEBUG:boto:Token: None 2024-04-12T03:58:35.424 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.424 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.428 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d03557847b377fdb-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.428 DEBUG:boto:path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:35.429 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:35.429 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.429 DEBUG:boto:Path: /dkyges-60/obj00000000000000000033 2024-04-12T03:58:35.429 DEBUG:boto:Data: 2024-04-12T03:58:35.429 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.429 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.429 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.429 DEBUG:boto:Params: {} 2024-04-12T03:58:35.429 DEBUG:boto:Token: None 2024-04-12T03:58:35.429 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000033 2024-04-12T03:58:35.430 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8kctLb4ED0ZusXqhuCwNqCzNLPo= 2024-04-12T03:58:35.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8kctLb4ED0ZusXqhuCwNqCzNLPo='} 2024-04-12T03:58:35.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003645ee2ef919a18d-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.438 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.438 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.438 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.438 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.438 DEBUG:boto:Data: 2024-04-12T03:58:35.438 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.438 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.439 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.439 DEBUG:boto:Params: {} 2024-04-12T03:58:35.439 DEBUG:boto:Token: None 2024-04-12T03:58:35.439 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.439 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.444 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d675515c868f46b4-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.444 DEBUG:boto:path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:35.444 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:35.444 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.444 DEBUG:boto:Path: /dkyges-60/obj00000000000000000017 2024-04-12T03:58:35.444 DEBUG:boto:Data: 2024-04-12T03:58:35.444 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.444 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.445 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.445 DEBUG:boto:Params: {} 2024-04-12T03:58:35.445 DEBUG:boto:Token: None 2024-04-12T03:58:35.445 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000017 2024-04-12T03:58:35.445 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:li/dc70xYZtdPdUmmBBTxYnA11c= 2024-04-12T03:58:35.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:li/dc70xYZtdPdUmmBBTxYnA11c='} 2024-04-12T03:58:35.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000578f0b80023d3368-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.491 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.491 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.491 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.491 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.491 DEBUG:boto:Data: 2024-04-12T03:58:35.491 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.491 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.491 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.491 DEBUG:boto:Params: {} 2024-04-12T03:58:35.492 DEBUG:boto:Token: None 2024-04-12T03:58:35.492 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.492 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.496 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6cc41895c9716f0-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.496 DEBUG:boto:path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:35.496 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:35.496 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.496 DEBUG:boto:Path: /dkyges-60/obj00000000000000000014 2024-04-12T03:58:35.496 DEBUG:boto:Data: 2024-04-12T03:58:35.496 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.497 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.497 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.497 DEBUG:boto:Params: {} 2024-04-12T03:58:35.497 DEBUG:boto:Token: None 2024-04-12T03:58:35.497 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000014 2024-04-12T03:58:35.497 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Bd1+MgMy8IAnziXTwCvqSy4sHDY= 2024-04-12T03:58:35.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bd1+MgMy8IAnziXTwCvqSy4sHDY='} 2024-04-12T03:58:35.505 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000054dc4d5b43458c97-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.506 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.506 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.506 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.506 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.506 DEBUG:boto:Data: 2024-04-12T03:58:35.506 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.506 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.506 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.506 DEBUG:boto:Params: {} 2024-04-12T03:58:35.506 DEBUG:boto:Token: None 2024-04-12T03:58:35.507 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.507 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.511 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017ddea31f4fda295-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.511 DEBUG:boto:path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:35.511 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:35.511 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.511 DEBUG:boto:Path: /dkyges-60/obj00000000000000000026 2024-04-12T03:58:35.512 DEBUG:boto:Data: 2024-04-12T03:58:35.512 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.512 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.512 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.512 DEBUG:boto:Params: {} 2024-04-12T03:58:35.512 DEBUG:boto:Token: None 2024-04-12T03:58:35.512 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000026 2024-04-12T03:58:35.512 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QEjobHTkYZ9sbduDjEeV4zOZZXM= 2024-04-12T03:58:35.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QEjobHTkYZ9sbduDjEeV4zOZZXM='} 2024-04-12T03:58:35.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3c10fc720e52046-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.559 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.559 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.559 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.559 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.559 DEBUG:boto:Data: 2024-04-12T03:58:35.559 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.559 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.559 DEBUG:boto:Params: {} 2024-04-12T03:58:35.559 DEBUG:boto:Token: None 2024-04-12T03:58:35.560 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.560 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000093e01329689629b8-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.564 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:35.565 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:35.565 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.565 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005 2024-04-12T03:58:35.565 DEBUG:boto:Data: 2024-04-12T03:58:35.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.565 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.565 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.565 DEBUG:boto:Params: {} 2024-04-12T03:58:35.565 DEBUG:boto:Token: None 2024-04-12T03:58:35.565 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000005 2024-04-12T03:58:35.566 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xVzpQz90hukBcaylqe8rtCh0Ul0= 2024-04-12T03:58:35.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xVzpQz90hukBcaylqe8rtCh0Ul0='} 2024-04-12T03:58:35.574 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2f07bb928d72964-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.574 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.574 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.574 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.575 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.575 DEBUG:boto:Data: 2024-04-12T03:58:35.575 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.575 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.575 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.575 DEBUG:boto:Params: {} 2024-04-12T03:58:35.575 DEBUG:boto:Token: None 2024-04-12T03:58:35.575 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.575 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.580 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000092b4f14f7fcbd181-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.580 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:35.580 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:35.581 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.581 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011 2024-04-12T03:58:35.581 DEBUG:boto:Data: 2024-04-12T03:58:35.581 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.581 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.581 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.581 DEBUG:boto:Params: {} 2024-04-12T03:58:35.581 DEBUG:boto:Token: None 2024-04-12T03:58:35.581 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000011 2024-04-12T03:58:35.581 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+/RH03lmPo4xzwViRiTta0BB79c= 2024-04-12T03:58:35.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+/RH03lmPo4xzwViRiTta0BB79c='} 2024-04-12T03:58:35.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000060883a509d39b807-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.627 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.627 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.627 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.627 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.627 DEBUG:boto:Data: 2024-04-12T03:58:35.627 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.627 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.627 DEBUG:boto:Params: {} 2024-04-12T03:58:35.627 DEBUG:boto:Token: None 2024-04-12T03:58:35.628 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.632 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028232a8718ee0c7d-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.632 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:35.633 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:35.633 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.633 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000 2024-04-12T03:58:35.633 DEBUG:boto:Data: 2024-04-12T03:58:35.633 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.633 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.633 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.633 DEBUG:boto:Params: {} 2024-04-12T03:58:35.633 DEBUG:boto:Token: None 2024-04-12T03:58:35.633 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000000 2024-04-12T03:58:35.634 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AgjC5E31IA4nwEVoWzFY1flcdEE= 2024-04-12T03:58:35.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AgjC5E31IA4nwEVoWzFY1flcdEE='} 2024-04-12T03:58:35.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e94f5ff90f2839d-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.642 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.642 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.642 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.642 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.642 DEBUG:boto:Data: 2024-04-12T03:58:35.643 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.643 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.643 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.643 DEBUG:boto:Params: {} 2024-04-12T03:58:35.643 DEBUG:boto:Token: None 2024-04-12T03:58:35.643 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.643 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000618d1c8be8674f20-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.648 DEBUG:boto:path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:35.648 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:35.649 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.649 DEBUG:boto:Path: /dkyges-60/obj00000000000000000060 2024-04-12T03:58:35.649 DEBUG:boto:Data: 2024-04-12T03:58:35.649 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.649 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.649 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.649 DEBUG:boto:Params: {} 2024-04-12T03:58:35.649 DEBUG:boto:Token: None 2024-04-12T03:58:35.649 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000060 2024-04-12T03:58:35.649 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HYwgZq7Kdsh6PCBpDFFtvJ6/xWY= 2024-04-12T03:58:35.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HYwgZq7Kdsh6PCBpDFFtvJ6/xWY='} 2024-04-12T03:58:35.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ef28fe371bf533e-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.695 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.695 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.695 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.695 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.695 DEBUG:boto:Data: 2024-04-12T03:58:35.695 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.695 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.695 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.695 DEBUG:boto:Params: {} 2024-04-12T03:58:35.695 DEBUG:boto:Token: None 2024-04-12T03:58:35.696 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.696 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014fe0e02129b69c2-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.701 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:35.701 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:35.701 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.701 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012 2024-04-12T03:58:35.701 DEBUG:boto:Data: 2024-04-12T03:58:35.701 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.701 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.701 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.701 DEBUG:boto:Params: {} 2024-04-12T03:58:35.701 DEBUG:boto:Token: None 2024-04-12T03:58:35.702 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000012 2024-04-12T03:58:35.702 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:N2B82QryV3SgejBy/IBS/SuJVmQ= 2024-04-12T03:58:35.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:N2B82QryV3SgejBy/IBS/SuJVmQ='} 2024-04-12T03:58:35.710 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042df7aa5305d2de2-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.711 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.711 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.711 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.711 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.711 DEBUG:boto:Data: 2024-04-12T03:58:35.711 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.711 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.711 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.711 DEBUG:boto:Params: {} 2024-04-12T03:58:35.711 DEBUG:boto:Token: None 2024-04-12T03:58:35.712 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.712 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.716 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e4bc9d40a3075a7-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.716 DEBUG:boto:path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:35.716 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:35.717 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.717 DEBUG:boto:Path: /dkyges-60/obj00000000000000000007 2024-04-12T03:58:35.717 DEBUG:boto:Data: 2024-04-12T03:58:35.717 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.717 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.717 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.717 DEBUG:boto:Params: {} 2024-04-12T03:58:35.717 DEBUG:boto:Token: None 2024-04-12T03:58:35.717 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000007 2024-04-12T03:58:35.718 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mOgulKozUZ845gJeytf2sQCW7aY= 2024-04-12T03:58:35.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mOgulKozUZ845gJeytf2sQCW7aY='} 2024-04-12T03:58:35.762 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093b5b1362c363ea9-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.763 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.763 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.763 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.763 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.763 DEBUG:boto:Data: 2024-04-12T03:58:35.763 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.763 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.763 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.763 DEBUG:boto:Params: {} 2024-04-12T03:58:35.764 DEBUG:boto:Token: None 2024-04-12T03:58:35.764 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.764 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.768 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027a69694b5363064-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.768 DEBUG:boto:path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:35.768 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:35.768 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.768 DEBUG:boto:Path: /dkyges-60/obj00000000000000000021 2024-04-12T03:58:35.768 DEBUG:boto:Data: 2024-04-12T03:58:35.769 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.769 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.769 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.769 DEBUG:boto:Params: {} 2024-04-12T03:58:35.769 DEBUG:boto:Token: None 2024-04-12T03:58:35.769 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000021 2024-04-12T03:58:35.769 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MzzDiCQYc51JIBIWqi4Xdw91KlY= 2024-04-12T03:58:35.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MzzDiCQYc51JIBIWqi4Xdw91KlY='} 2024-04-12T03:58:35.778 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055798e88e914acdf-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.778 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.778 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.778 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.778 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.778 DEBUG:boto:Data: 2024-04-12T03:58:35.778 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.779 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.779 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.779 DEBUG:boto:Params: {} 2024-04-12T03:58:35.779 DEBUG:boto:Token: None 2024-04-12T03:58:35.779 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.779 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.783 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c21e585423fbdbe2-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.784 DEBUG:boto:path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:35.784 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:35.784 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.784 DEBUG:boto:Path: /dkyges-60/obj00000000000000000016 2024-04-12T03:58:35.784 DEBUG:boto:Data: 2024-04-12T03:58:35.784 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.784 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.784 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.784 DEBUG:boto:Params: {} 2024-04-12T03:58:35.785 DEBUG:boto:Token: None 2024-04-12T03:58:35.785 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000016 2024-04-12T03:58:35.785 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iYgetPE4l2U8B9C63P9k4QXvJIY= 2024-04-12T03:58:35.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iYgetPE4l2U8B9C63P9k4QXvJIY='} 2024-04-12T03:58:35.827 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc693c785060f960-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.827 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.827 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.827 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.827 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.827 DEBUG:boto:Data: 2024-04-12T03:58:35.827 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.828 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.828 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.828 DEBUG:boto:Params: {} 2024-04-12T03:58:35.828 DEBUG:boto:Token: None 2024-04-12T03:58:35.828 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.828 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.832 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4bcc8e499626a40-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.833 DEBUG:boto:path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:35.833 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:35.833 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.833 DEBUG:boto:Path: /dkyges-60/obj00000000000000000053 2024-04-12T03:58:35.833 DEBUG:boto:Data: 2024-04-12T03:58:35.833 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.833 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.833 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.834 DEBUG:boto:Params: {} 2024-04-12T03:58:35.834 DEBUG:boto:Token: None 2024-04-12T03:58:35.834 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000053 2024-04-12T03:58:35.834 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vYvklGO7eZoiU1dO4+pjM3SVSYE= 2024-04-12T03:58:35.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vYvklGO7eZoiU1dO4+pjM3SVSYE='} 2024-04-12T03:58:35.841 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048f34452924948c5-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.841 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:35.841 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:35.841 DEBUG:boto:Method: HEAD 2024-04-12T03:58:35.841 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:35.841 DEBUG:boto:Data: 2024-04-12T03:58:35.841 DEBUG:boto:Headers: {} 2024-04-12T03:58:35.841 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.841 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.842 DEBUG:boto:Params: {} 2024-04-12T03:58:35.842 DEBUG:boto:Token: None 2024-04-12T03:58:35.842 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/ 2024-04-12T03:58:35.842 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY= 2024-04-12T03:58:35.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n4kLQa/Ob7DqeYAFtL8nZF62SSY=', 'Content-Length': '0'} 2024-04-12T03:58:35.846 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001e686709b6e847bb-006618b16b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.846 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:35.846 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:35.846 DEBUG:boto:Method: PUT 2024-04-12T03:58:35.846 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010 2024-04-12T03:58:35.847 DEBUG:boto:Data: 2024-04-12T03:58:35.847 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T03:58:35.847 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:35.847 DEBUG:boto:Port: 8000 2024-04-12T03:58:35.847 DEBUG:boto:Params: {} 2024-04-12T03:58:35.847 DEBUG:boto:Token: None 2024-04-12T03:58:35.847 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 03:58:35 GMT /dkyges-60/obj00000000000000000010 2024-04-12T03:58:35.847 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4evmFnAwFsUG9HD9T3vOW57DaF0= 2024-04-12T03:58:35.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:58:35 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4evmFnAwFsUG9HD9T3vOW57DaF0='} 2024-04-12T03:58:35.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bb246210b6b5d910-006618b16b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:35.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:35.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T03:58:35.891 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-60 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T03:58:35.953 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:35.953 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:36.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.049+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.050+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.050+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.050+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: sync status for source bucket shard: 2 2024-04-12T03:58:36.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.050+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: listing bilog for incremental sync; position=00000000001.134.6 2024-04-12T03:58:36.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.050+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: sync status for source bucket shard: 2 2024-04-12T03:58:36.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.050+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: listing bilog for incremental sync; position=00000000001.62.6 2024-04-12T03:58:36.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.050+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.050+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket shard: 2 2024-04-12T03:58:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.050+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position=00000000001.32.6 2024-04-12T03:58:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.051+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.051+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.051+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.051+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: sync status for source bucket shard: 2 2024-04-12T03:58:36.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.051+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.051+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: listing bilog for incremental sync; position=00000000001.58.6 2024-04-12T03:58:36.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.052+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: sync status for source bucket shard: 2 2024-04-12T03:58:36.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.052+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.052+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: sync status for source bucket shard: 2 2024-04-12T03:58:36.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.052+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.052+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.052+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.052+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: listing bilog for incremental sync; position=00000000001.51.6 2024-04-12T03:58:36.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.052+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: sync status for source bucket shard: 2 2024-04-12T03:58:36.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.052+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: listing bilog for incremental sync; position=00000000001.54.6 2024-04-12T03:58:36.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.053+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket shard: 2 2024-04-12T03:58:36.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.053+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position=00000000001.56.6 2024-04-12T03:58:36.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.053+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: sync status for source bucket shard: 2 2024-04-12T03:58:36.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.053+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: listing bilog for incremental sync; position=00000000001.63.6 2024-04-12T03:58:36.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.054+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: sync status for source bucket shard: 2 2024-04-12T03:58:36.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.054+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: listing bilog for incremental sync; position=00000000001.54.6 2024-04-12T03:58:36.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.054+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: sync status for source bucket shard: 2 2024-04-12T03:58:36.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.054+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: listing bilog for incremental sync; position=00000000001.56.6 2024-04-12T03:58:36.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.103+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.104+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.104+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.104+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket shard: 0 2024-04-12T03:58:36.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.104+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.104+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: sync status for source bucket shard: 0 2024-04-12T03:58:36.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.104+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.104+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.104+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: sync status for source bucket shard: 0 2024-04-12T03:58:36.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.104+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.104+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.105+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: sync status for source bucket shard: 0 2024-04-12T03:58:36.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.105+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.105+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket shard: 0 2024-04-12T03:58:36.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.105+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.105+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.105+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.106+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.106+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.106+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: sync status for source bucket shard: 0 2024-04-12T03:58:36.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.106+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.106+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: sync status for source bucket shard: 0 2024-04-12T03:58:36.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.106+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.106+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.106+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.106+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.106+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.107+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: sync status for source bucket shard: 0 2024-04-12T03:58:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.107+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: sync status for source bucket shard: 0 2024-04-12T03:58:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.107+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: sync status for source bucket shard: 0 2024-04-12T03:58:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.107+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.108+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.108+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.108+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: parsed entry: id=00000000001.34.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-12T03:58:36.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.108+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1/obj00000000000000000016 2024-04-12T03:58:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.108+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: entry->timestamp=2024-04-12T03:58:29.859912+0000 2024-04-12T03:58:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.108+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.34.6 op=0 op_state=1 2024-04-12T03:58:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.108+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: parsed entry: id=00000000001.104.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2024-04-12T03:58:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.108+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4/obj00000000000000000028 2024-04-12T03:58:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.108+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: entry->timestamp=2024-04-12T03:58:30.226917+0000 2024-04-12T03:58:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.108+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.104.6 op=0 op_state=1 2024-04-12T03:58:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.108+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000016]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000016[0] 2024-04-12T03:58:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.108+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: listing bilog for incremental sync; position=00000000001.34.6 2024-04-12T03:58:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.108+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000028]: : injecting data sync error on key=obj00000000000000000028 2024-04-12T03:58:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.108+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position=00000000001.104.6 2024-04-12T03:58:36.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.108+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000028]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.108+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000028]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4/obj00000000000000000028 2024-04-12T03:58:36.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: sync status for source bucket shard: 0 2024-04-12T03:58:36.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: sync status for source bucket shard: 0 2024-04-12T03:58:36.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: sync status for source bucket shard: 0 2024-04-12T03:58:36.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: parsed entry: id=00000000002.25.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-12T03:58:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5/obj00000000000000000000 2024-04-12T03:58:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: entry->timestamp=2024-04-12T03:58:29.847365+0000 2024-04-12T03:58:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.25.6 op=0 op_state=1 2024-04-12T03:58:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: parsed entry: id=00000000001.49.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2024-04-12T03:58:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6/obj00000000000000000032 2024-04-12T03:58:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: entry->timestamp=2024-04-12T03:58:30.031784+0000 2024-04-12T03:58:36.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.49.6 op=0 op_state=1 2024-04-12T03:58:36.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:entry[obj00000000000000000000]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000000[0] 2024-04-12T03:58:36.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: listing bilog for incremental sync; position=00000000002.25.6 2024-04-12T03:58:36.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000032[0] 2024-04-12T03:58:36.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.109+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: listing bilog for incremental sync; position=00000000001.49.6 2024-04-12T03:58:36.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.112+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket shard: 0 2024-04-12T03:58:36.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.112+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.112+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: sync status for source bucket shard: 0 2024-04-12T03:58:36.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.112+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: sync status for source bucket shard: 0 2024-04-12T03:58:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: sync status for source bucket shard: 0 2024-04-12T03:58:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-12T03:58:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7/obj00000000000000000001 2024-04-12T03:58:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: entry->timestamp=2024-04-12T03:58:30.158909+0000 2024-04-12T03:58:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.37.6 op=0 op_state=1 2024-04-12T03:58:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.113+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:entry[obj00000000000000000001]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000001[0] 2024-04-12T03:58:36.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: listing bilog for incremental sync; position=00000000002.37.6 2024-04-12T03:58:36.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: parsed entry: id=00000000001.34.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-12T03:58:36.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9/obj00000000000000000019 2024-04-12T03:58:36.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: entry->timestamp=2024-04-12T03:58:29.918581+0000 2024-04-12T03:58:36.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.34.6 op=0 op_state=1 2024-04-12T03:58:36.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.113+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:entry[obj00000000000000000019]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000019[0] 2024-04-12T03:58:36.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: listing bilog for incremental sync; position=00000000001.34.6 2024-04-12T03:58:36.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.115+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: parsed entry: id=00000000001.67.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-04-12T03:58:36.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.115+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12/obj00000000000000000030 2024-04-12T03:58:36.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.115+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: entry->timestamp=2024-04-12T03:58:30.173110+0000 2024-04-12T03:58:36.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.115+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.67.6 op=0 op_state=1 2024-04-12T03:58:36.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.115+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000030]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000030[0] 2024-04-12T03:58:36.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.115+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: listing bilog for incremental sync; position=00000000001.67.6 2024-04-12T03:58:36.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.126+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000032]: success 2024-04-12T03:58:36.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6:1 marker=00000000001.49.6 timestamp=2024-04-12T03:58:30.031784+0000 2024-04-12T03:58:36.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.126+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000016]: success 2024-04-12T03:58:36.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1:1 marker=00000000001.34.6 timestamp=2024-04-12T03:58:29.859912+0000 2024-04-12T03:58:36.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.126+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:entry[obj00000000000000000000]: success 2024-04-12T03:58:36.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5:1 marker=00000000002.25.6 timestamp=2024-04-12T03:58:29.847365+0000 2024-04-12T03:58:36.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.147+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: parsed entry: id=00000000001.83.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2024-04-12T03:58:36.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.147+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2/obj00000000000000000024 2024-04-12T03:58:36.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.147+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: entry->timestamp=2024-04-12T03:58:29.906817+0000 2024-04-12T03:58:36.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.147+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.83.6 op=0 op_state=1 2024-04-12T03:58:36.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.147+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000024]: : injecting data sync error on key=obj00000000000000000024 2024-04-12T03:58:36.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.147+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position=00000000001.83.6 2024-04-12T03:58:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.147+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: parsed entry: id=00000000001.53.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-12T03:58:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.147+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0/obj00000000000000000007 2024-04-12T03:58:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.147+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: entry->timestamp=2024-04-12T03:58:30.095828+0000 2024-04-12T03:58:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.147+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.53.6 op=0 op_state=1 2024-04-12T03:58:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.147+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000024]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.147+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000024]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2/obj00000000000000000024 2024-04-12T03:58:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.147+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:entry[obj00000000000000000007]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000007[0] 2024-04-12T03:58:36.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.147+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: listing bilog for incremental sync; position=00000000001.53.6 2024-04-12T03:58:36.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.150+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-12T03:58:36.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.150+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8/obj00000000000000000010 2024-04-12T03:58:36.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.150+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: entry->timestamp=2024-04-12T03:58:29.980794+0000 2024-04-12T03:58:36.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.150+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.36.6 op=0 op_state=1 2024-04-12T03:58:36.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.150+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000010]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000010[0] 2024-04-12T03:58:36.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.150+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: listing bilog for incremental sync; position=00000000002.36.6 2024-04-12T03:58:36.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.150+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: parsed entry: id=00000000002.24.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-04-12T03:58:36.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.150+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10/obj00000000000000000013 2024-04-12T03:58:36.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.150+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: entry->timestamp=2024-04-12T03:58:30.108270+0000 2024-04-12T03:58:36.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.150+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.24.6 op=0 op_state=1 2024-04-12T03:58:36.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.150+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:entry[obj00000000000000000013]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000013[0] 2024-04-12T03:58:36.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.150+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: listing bilog for incremental sync; position=00000000002.24.6 2024-04-12T03:58:36.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.153+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: parsed entry: id=00000000001.47.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-12T03:58:36.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.153+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11/obj00000000000000000021 2024-04-12T03:58:36.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.153+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: entry->timestamp=2024-04-12T03:58:30.043948+0000 2024-04-12T03:58:36.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.153+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.47.6 op=0 op_state=1 2024-04-12T03:58:36.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.153+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000021]: : injecting data sync error on key=obj00000000000000000021 2024-04-12T03:58:36.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.153+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: listing bilog for incremental sync; position=00000000001.47.6 2024-04-12T03:58:36.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.153+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000021]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.153+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000021]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11/obj00000000000000000021 2024-04-12T03:58:36.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.153+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: a sync operation returned error 2024-04-12T03:58:36.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.153+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: backing out with sync_status=-5 2024-04-12T03:58:36.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.153+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.153+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.154+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:entry[obj00000000000000000007]: success 2024-04-12T03:58:36.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.154+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0:1 marker=00000000001.53.6 timestamp=2024-04-12T03:58:30.095828+0000 2024-04-12T03:58:36.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.154+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.155+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket shard: 0 2024-04-12T03:58:36.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.155+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: parsed entry: id=00000000001.35.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-04-12T03:58:36.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13/obj00000000000000000026 2024-04-12T03:58:36.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: entry->timestamp=2024-04-12T03:58:29.797004+0000 2024-04-12T03:58:36.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.35.6 op=0 op_state=1 2024-04-12T03:58:36.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000026]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000026[0] 2024-04-12T03:58:36.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position=00000000001.35.6 2024-04-12T03:58:36.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-12T03:58:36.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15/obj00000000000000000004 2024-04-12T03:58:36.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: entry->timestamp=2024-04-12T03:58:29.967442+0000 2024-04-12T03:58:36.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.43.6 op=0 op_state=1 2024-04-12T03:58:36.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-04-12T03:58:36.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14/obj00000000000000000033 2024-04-12T03:58:36.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: entry->timestamp=2024-04-12T03:58:29.785588+0000 2024-04-12T03:58:36.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.45.6 op=0 op_state=1 2024-04-12T03:58:36.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000004]: : injecting data sync error on key=obj00000000000000000004 2024-04-12T03:58:36.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: listing bilog for incremental sync; position=00000000002.43.6 2024-04-12T03:58:36.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000033]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000033[0] 2024-04-12T03:58:36.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: listing bilog for incremental sync; position=00000000001.45.6 2024-04-12T03:58:36.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000004]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.156+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000004]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15/obj00000000000000000004 2024-04-12T03:58:36.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.157+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: parsed entry: id=00000000001.48.6 iter->object=obj00000000000000000076 iter->instance= name=obj00000000000000000076 instance= ns= 2024-04-12T03:58:36.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.157+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16/obj00000000000000000076 2024-04-12T03:58:36.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.157+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: entry->timestamp=2024-04-12T03:58:30.238284+0000 2024-04-12T03:58:36.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.157+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000076]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.48.6 op=0 op_state=1 2024-04-12T03:58:36.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.157+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000076]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000076[0] 2024-04-12T03:58:36.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.157+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: listing bilog for incremental sync; position=00000000001.48.6 2024-04-12T03:58:36.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.162+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:entry[obj00000000000000000013]: success 2024-04-12T03:58:36.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.163+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10:1 marker=00000000002.24.6 timestamp=2024-04-12T03:58:30.108270+0000 2024-04-12T03:58:36.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.165+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:entry[obj00000000000000000001]: success 2024-04-12T03:58:36.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.165+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7:1 marker=00000000002.37.6 timestamp=2024-04-12T03:58:30.158909+0000 2024-04-12T03:58:36.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.165+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000030]: success 2024-04-12T03:58:36.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.165+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12:1 marker=00000000001.67.6 timestamp=2024-04-12T03:58:30.173110+0000 2024-04-12T03:58:36.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.169+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000076]: success 2024-04-12T03:58:36.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.169+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16:1 marker=00000000001.48.6 timestamp=2024-04-12T03:58:30.238284+0000 2024-04-12T03:58:36.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.170+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000033]: success 2024-04-12T03:58:36.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.170+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14:1 marker=00000000001.45.6 timestamp=2024-04-12T03:58:29.785588+0000 2024-04-12T03:58:36.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.172+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:entry[obj00000000000000000019]: success 2024-04-12T03:58:36.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.172+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9:1 marker=00000000001.34.6 timestamp=2024-04-12T03:58:29.918581+0000 2024-04-12T03:58:36.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.190+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: a sync operation returned error 2024-04-12T03:58:36.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.190+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: backing out with sync_status=-5 2024-04-12T03:58:36.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.190+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.190+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.191+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.191+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket shard: 0 2024-04-12T03:58:36.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.191+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.196+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000010]: success 2024-04-12T03:58:36.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.196+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8:1 marker=00000000002.36.6 timestamp=2024-04-12T03:58:29.980794+0000 2024-04-12T03:58:36.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.197+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: a sync operation returned error 2024-04-12T03:58:36.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.197+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: parsed entry: id=00000000001.104.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2024-04-12T03:58:36.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.197+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4/obj00000000000000000028 2024-04-12T03:58:36.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.197+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: entry->timestamp=2024-04-12T03:58:30.226917+0000 2024-04-12T03:58:36.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.197+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.104.6 op=0 op_state=1 2024-04-12T03:58:36.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.197+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000028]: : injecting data sync error on key=obj00000000000000000028 2024-04-12T03:58:36.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.197+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position=00000000001.104.6 2024-04-12T03:58:36.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.197+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: backing out with sync_status=-5 2024-04-12T03:58:36.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.197+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000028]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.197+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000028]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4/obj00000000000000000028 2024-04-12T03:58:36.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.197+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.197+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.198+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.198+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: sync status for source bucket shard: 0 2024-04-12T03:58:36.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.198+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.199+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: a sync operation returned error 2024-04-12T03:58:36.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.199+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: backing out with sync_status=-5 2024-04-12T03:58:36.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.199+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.199+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.200+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.200+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: sync status for source bucket shard: 0 2024-04-12T03:58:36.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.200+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.205+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000026]: success 2024-04-12T03:58:36.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.205+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13:1 marker=00000000001.35.6 timestamp=2024-04-12T03:58:29.797004+0000 2024-04-12T03:58:36.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.233+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: parsed entry: id=00000000001.83.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2024-04-12T03:58:36.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.233+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2/obj00000000000000000024 2024-04-12T03:58:36.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.233+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: entry->timestamp=2024-04-12T03:58:29.906817+0000 2024-04-12T03:58:36.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.233+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.83.6 op=0 op_state=1 2024-04-12T03:58:36.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.233+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000024]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000024[0] 2024-04-12T03:58:36.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.233+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position=00000000001.83.6 2024-04-12T03:58:36.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.238+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000024]: success 2024-04-12T03:58:36.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.238+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2:1 marker=00000000001.83.6 timestamp=2024-04-12T03:58:29.906817+0000 2024-04-12T03:58:36.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.239+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: a sync operation returned error 2024-04-12T03:58:36.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.239+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: backing out with sync_status=-5 2024-04-12T03:58:36.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.239+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.239+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.239+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.240+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket shard: 0 2024-04-12T03:58:36.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.240+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.241+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: parsed entry: id=00000000001.47.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-12T03:58:36.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.241+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11/obj00000000000000000021 2024-04-12T03:58:36.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.241+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: entry->timestamp=2024-04-12T03:58:30.043948+0000 2024-04-12T03:58:36.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.241+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.47.6 op=0 op_state=1 2024-04-12T03:58:36.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.241+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000021]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000021[0] 2024-04-12T03:58:36.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.241+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: listing bilog for incremental sync; position=00000000001.47.6 2024-04-12T03:58:36.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.242+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-12T03:58:36.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.242+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15/obj00000000000000000004 2024-04-12T03:58:36.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.242+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: entry->timestamp=2024-04-12T03:58:29.967442+0000 2024-04-12T03:58:36.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.242+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.43.6 op=0 op_state=1 2024-04-12T03:58:36.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.242+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000004]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000004[0] 2024-04-12T03:58:36.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.242+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: listing bilog for incremental sync; position=00000000002.43.6 2024-04-12T03:58:36.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.247+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000004]: success 2024-04-12T03:58:36.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.247+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15:1 marker=00000000002.43.6 timestamp=2024-04-12T03:58:29.967442+0000 2024-04-12T03:58:36.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.282+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: parsed entry: id=00000000001.104.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2024-04-12T03:58:36.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.282+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4/obj00000000000000000028 2024-04-12T03:58:36.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.282+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: entry->timestamp=2024-04-12T03:58:30.226917+0000 2024-04-12T03:58:36.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.282+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.104.6 op=0 op_state=1 2024-04-12T03:58:36.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.282+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000028]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000028[0] 2024-04-12T03:58:36.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.282+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position=00000000001.104.6 2024-04-12T03:58:36.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.287+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000021]: success 2024-04-12T03:58:36.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.287+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11:1 marker=00000000001.47.6 timestamp=2024-04-12T03:58:30.043948+0000 2024-04-12T03:58:36.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.328+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000028]: success 2024-04-12T03:58:36.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.328+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4:1 marker=00000000001.104.6 timestamp=2024-04-12T03:58:30.226917+0000 2024-04-12T03:58:36.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.337+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.337+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.338+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: sync status for source bucket shard: 0 2024-04-12T03:58:36.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.338+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.338+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.338+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: sync status for source bucket shard: 0 2024-04-12T03:58:36.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.338+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.338+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket shard: 0 2024-04-12T03:58:36.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.338+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.338+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.338+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.339+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.339+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: sync status for source bucket shard: 0 2024-04-12T03:58:36.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.339+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.339+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket shard: 0 2024-04-12T03:58:36.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.339+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.339+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: sync status for source bucket shard: 0 2024-04-12T03:58:36.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.339+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.340+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.340+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.340+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.340+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.340+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.340+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.341+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.341+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-12T03:58:36.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.341+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4/obj00000000000000000004 2024-04-12T03:58:36.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.341+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: entry->timestamp=2024-04-12T03:58:31.054848+0000 2024-04-12T03:58:36.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.341+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.54.6 op=0 op_state=1 2024-04-12T03:58:36.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.341+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000004]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000004[0] 2024-04-12T03:58:36.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.341+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position=00000000002.54.6 2024-04-12T03:58:36.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.342+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: sync status for source bucket shard: 0 2024-04-12T03:58:36.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.342+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.342+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: sync status for source bucket shard: 0 2024-04-12T03:58:36.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.342+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.342+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: sync status for source bucket shard: 0 2024-04-12T03:58:36.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.342+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: sync status for source bucket shard: 0 2024-04-12T03:58:36.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.342+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: sync status for source bucket shard: 0 2024-04-12T03:58:36.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.342+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: sync status for source bucket shard: 0 2024-04-12T03:58:36.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.342+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: sync status for source bucket shard: 0 2024-04-12T03:58:36.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.342+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.342+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.342+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.342+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.342+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.342+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.342+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: parsed entry: id=00000000002.113.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-12T03:58:36.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3/obj00000000000000000021 2024-04-12T03:58:36.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: entry->timestamp=2024-04-12T03:58:30.993121+0000 2024-04-12T03:58:36.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.113.6 op=0 op_state=1 2024-04-12T03:58:36.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-12T03:58:36.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2/obj00000000000000000016 2024-04-12T03:58:36.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: entry->timestamp=2024-04-12T03:58:30.577561+0000 2024-04-12T03:58:36.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.42.6 op=0 op_state=1 2024-04-12T03:58:36.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: parsed entry: id=00000000002.29.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-04-12T03:58:36.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6/obj00000000000000000030 2024-04-12T03:58:36.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: entry->timestamp=2024-04-12T03:58:30.791378+0000 2024-04-12T03:58:36.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.29.6 op=0 op_state=1 2024-04-12T03:58:36.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:entry[obj00000000000000000021]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000021[0] 2024-04-12T03:58:36.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: listing bilog for incremental sync; position=00000000002.113.6 2024-04-12T03:58:36.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000016]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000016[0] 2024-04-12T03:58:36.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position=00000000002.42.6 2024-04-12T03:58:36.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000030]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000030[0] 2024-04-12T03:58:36.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.343+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: listing bilog for incremental sync; position=00000000002.29.6 2024-04-12T03:58:36.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.345+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: sync status for source bucket shard: 0 2024-04-12T03:58:36.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.345+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.346+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: sync status for source bucket shard: 0 2024-04-12T03:58:36.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.346+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.346+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket shard: 0 2024-04-12T03:58:36.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.346+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: sync status for source bucket shard: 0 2024-04-12T03:58:36.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.346+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.346+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.347+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000004]: success 2024-04-12T03:58:36.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.347+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4:2 marker=00000000002.54.6 timestamp=2024-04-12T03:58:31.054848+0000 2024-04-12T03:58:36.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.347+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: parsed entry: id=00000000001.77.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-12T03:58:36.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.347+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10/obj00000000000000000009 2024-04-12T03:58:36.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.347+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: entry->timestamp=2024-04-12T03:58:30.919294+0000 2024-04-12T03:58:36.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.347+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.77.6 op=0 op_state=1 2024-04-12T03:58:36.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.347+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:entry[obj00000000000000000009]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000009[0] 2024-04-12T03:58:36.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.347+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: listing bilog for incremental sync; position=00000000001.77.6 2024-04-12T03:58:36.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.348+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: sync status for source bucket shard: 0 2024-04-12T03:58:36.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.348+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.348+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: sync status for source bucket shard: 0 2024-04-12T03:58:36.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.348+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.348+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-12T03:58:36.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.348+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8/obj00000000000000000019 2024-04-12T03:58:36.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.348+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: entry->timestamp=2024-04-12T03:58:30.805394+0000 2024-04-12T03:58:36.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.348+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.63.6 op=0 op_state=1 2024-04-12T03:58:36.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.349+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000019]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000019[0] 2024-04-12T03:58:36.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.349+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: listing bilog for incremental sync; position=00000000002.63.6 2024-04-12T03:58:36.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.349+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: parsed entry: id=00000000002.96.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-12T03:58:36.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.349+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12/obj00000000000000000012 2024-04-12T03:58:36.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.349+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: entry->timestamp=2024-04-12T03:58:30.727488+0000 2024-04-12T03:58:36.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.349+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.96.6 op=0 op_state=1 2024-04-12T03:58:36.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.349+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000012]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000012[0] 2024-04-12T03:58:36.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.349+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: listing bilog for incremental sync; position=00000000002.96.6 2024-04-12T03:58:36.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.351+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-12T03:58:36.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.351+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17/obj00000000000000000001 2024-04-12T03:58:36.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.351+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: entry->timestamp=2024-04-12T03:58:30.869061+0000 2024-04-12T03:58:36.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.351+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.50.6 op=0 op_state=1 2024-04-12T03:58:36.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.351+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000001[0] 2024-04-12T03:58:36.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.351+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: listing bilog for incremental sync; position=00000000002.50.6 2024-04-12T03:58:36.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.351+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:entry[obj00000000000000000021]: success 2024-04-12T03:58:36.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.351+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3:2 marker=00000000002.113.6 timestamp=2024-04-12T03:58:30.993121+0000 2024-04-12T03:58:36.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.352+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000016]: success 2024-04-12T03:58:36.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.352+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2:2 marker=00000000002.42.6 timestamp=2024-04-12T03:58:30.577561+0000 2024-04-12T03:58:36.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.352+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000030]: success 2024-04-12T03:58:36.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.352+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6:2 marker=00000000002.29.6 timestamp=2024-04-12T03:58:30.791378+0000 2024-04-12T03:58:36.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.353+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-12T03:58:36.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.353+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14/obj00000000000000000000 2024-04-12T03:58:36.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.353+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: entry->timestamp=2024-04-12T03:58:30.663267+0000 2024-04-12T03:58:36.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.353+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.37.6 op=0 op_state=1 2024-04-12T03:58:36.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.353+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000000]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000000[0] 2024-04-12T03:58:36.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.353+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: listing bilog for incremental sync; position=00000000002.37.6 2024-04-12T03:58:36.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.353+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000019]: success 2024-04-12T03:58:36.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.353+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8:2 marker=00000000002.63.6 timestamp=2024-04-12T03:58:30.805394+0000 2024-04-12T03:58:36.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.357+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:entry[obj00000000000000000001]: success 2024-04-12T03:58:36.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.357+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17:2 marker=00000000002.50.6 timestamp=2024-04-12T03:58:30.869061+0000 2024-04-12T03:58:36.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.357+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000000]: success 2024-04-12T03:58:36.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.357+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14:2 marker=00000000002.37.6 timestamp=2024-04-12T03:58:30.663267+0000 2024-04-12T03:58:36.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.362+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:entry[obj00000000000000000009]: success 2024-04-12T03:58:36.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.362+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10:2 marker=00000000001.77.6 timestamp=2024-04-12T03:58:30.919294+0000 2024-04-12T03:58:36.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.380+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-04-12T03:58:36.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.380+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0/obj00000000000000000005 2024-04-12T03:58:36.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.380+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: entry->timestamp=2024-04-12T03:58:30.931756+0000 2024-04-12T03:58:36.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.380+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.30.6 op=0 op_state=1 2024-04-12T03:58:36.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.380+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:entry[obj00000000000000000005]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000005[0] 2024-04-12T03:58:36.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.380+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: listing bilog for incremental sync; position=00000000002.30.6 2024-04-12T03:58:36.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.385+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: parsed entry: id=00000000001.36.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2024-04-12T03:58:36.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.385+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5/obj00000000000000000025 2024-04-12T03:58:36.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.385+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: entry->timestamp=2024-04-12T03:58:31.043156+0000 2024-04-12T03:58:36.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.385+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.36.6 op=0 op_state=1 2024-04-12T03:58:36.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.385+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:entry[obj00000000000000000025]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000025[0] 2024-04-12T03:58:36.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.385+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: listing bilog for incremental sync; position=00000000001.36.6 2024-04-12T03:58:36.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.386+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: parsed entry: id=00000000001.42.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-04-12T03:58:36.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.386+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11/obj00000000000000000014 2024-04-12T03:58:36.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.386+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: entry->timestamp=2024-04-12T03:58:30.601717+0000 2024-04-12T03:58:36.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.386+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.42.6 op=0 op_state=1 2024-04-12T03:58:36.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.386+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000014]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000014[0] 2024-04-12T03:58:36.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.386+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: listing bilog for incremental sync; position=00000000001.42.6 2024-04-12T03:58:36.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.387+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-04-12T03:58:36.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.387+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7/obj00000000000000000015 2024-04-12T03:58:36.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.387+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: entry->timestamp=2024-04-12T03:58:30.676620+0000 2024-04-12T03:58:36.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.387+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.48.6 op=0 op_state=1 2024-04-12T03:58:36.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.387+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:entry[obj00000000000000000015]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000015[0] 2024-04-12T03:58:36.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.387+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: listing bilog for incremental sync; position=00000000002.48.6 2024-04-12T03:58:36.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.388+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: parsed entry: id=00000000002.94.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-12T03:58:36.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.388+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9/obj00000000000000000003 2024-04-12T03:58:36.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.388+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: entry->timestamp=2024-04-12T03:58:30.613126+0000 2024-04-12T03:58:36.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.388+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.94.6 op=0 op_state=1 2024-04-12T03:58:36.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.388+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:entry[obj00000000000000000003]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000003[0] 2024-04-12T03:58:36.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.388+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: listing bilog for incremental sync; position=00000000002.94.6 2024-04-12T03:58:36.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.391+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: parsed entry: id=00000000002.111.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-12T03:58:36.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.391+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16/obj00000000000000000011 2024-04-12T03:58:36.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.391+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: entry->timestamp=2024-04-12T03:58:30.855398+0000 2024-04-12T03:58:36.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.391+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.111.6 op=0 op_state=1 2024-04-12T03:58:36.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.391+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000011]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000011[0] 2024-04-12T03:58:36.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.391+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: listing bilog for incremental sync; position=00000000002.111.6 2024-04-12T03:58:36.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.391+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-12T03:58:36.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.391+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18/obj00000000000000000007 2024-04-12T03:58:36.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.391+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: entry->timestamp=2024-04-12T03:58:30.979434+0000 2024-04-12T03:58:36.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.391+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.35.6 op=0 op_state=1 2024-04-12T03:58:36.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.391+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000007]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000007[0] 2024-04-12T03:58:36.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.391+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: listing bilog for incremental sync; position=00000000002.35.6 2024-04-12T03:58:36.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.392+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: parsed entry: id=00000000002.92.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-12T03:58:36.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.392+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15/obj00000000000000000006 2024-04-12T03:58:36.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.392+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: entry->timestamp=2024-04-12T03:58:30.589761+0000 2024-04-12T03:58:36.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.392+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.92.6 op=0 op_state=1 2024-04-12T03:58:36.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.392+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000006]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000006[0] 2024-04-12T03:58:36.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.392+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: listing bilog for incremental sync; position=00000000002.92.6 2024-04-12T03:58:36.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.393+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: parsed entry: id=00000000002.72.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-12T03:58:36.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.393+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13/obj00000000000000000002 2024-04-12T03:58:36.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.393+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: entry->timestamp=2024-04-12T03:58:30.741031+0000 2024-04-12T03:58:36.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.393+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.72.6 op=0 op_state=1 2024-04-12T03:58:36.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.393+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000002]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000002[0] 2024-04-12T03:58:36.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.393+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position=00000000002.72.6 2024-04-12T03:58:36.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.394+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000007]: success 2024-04-12T03:58:36.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.394+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18:2 marker=00000000002.35.6 timestamp=2024-04-12T03:58:30.979434+0000 2024-04-12T03:58:36.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.401+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:entry[obj00000000000000000015]: success 2024-04-12T03:58:36.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.401+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7:2 marker=00000000002.48.6 timestamp=2024-04-12T03:58:30.676620+0000 2024-04-12T03:58:36.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.401+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:entry[obj00000000000000000003]: success 2024-04-12T03:58:36.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.401+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9:2 marker=00000000002.94.6 timestamp=2024-04-12T03:58:30.613126+0000 2024-04-12T03:58:36.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.402+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000012]: success 2024-04-12T03:58:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.403+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12:2 marker=00000000002.96.6 timestamp=2024-04-12T03:58:30.727488+0000 2024-04-12T03:58:36.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.406+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000006]: success 2024-04-12T03:58:36.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.406+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15:2 marker=00000000002.92.6 timestamp=2024-04-12T03:58:30.589761+0000 2024-04-12T03:58:36.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.427+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:entry[obj00000000000000000005]: success 2024-04-12T03:58:36.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.427+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0:2 marker=00000000002.30.6 timestamp=2024-04-12T03:58:30.931756+0000 2024-04-12T03:58:36.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.432+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:entry[obj00000000000000000025]: success 2024-04-12T03:58:36.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.432+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5:2 marker=00000000001.36.6 timestamp=2024-04-12T03:58:31.043156+0000 2024-04-12T03:58:36.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.433+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000014]: success 2024-04-12T03:58:36.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.433+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11:2 marker=00000000001.42.6 timestamp=2024-04-12T03:58:30.601717+0000 2024-04-12T03:58:36.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.441+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000011]: success 2024-04-12T03:58:36.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.441+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16:2 marker=00000000002.111.6 timestamp=2024-04-12T03:58:30.855398+0000 2024-04-12T03:58:36.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.446+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000002]: success 2024-04-12T03:58:36.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.446+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13:2 marker=00000000002.72.6 timestamp=2024-04-12T03:58:30.741031+0000 2024-04-12T03:58:36.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.456+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.456+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.456+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.457+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.457+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.457+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: sync status for source bucket shard: 0 2024-04-12T03:58:36.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.457+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.457+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket shard: 0 2024-04-12T03:58:36.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.457+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket shard: 0 2024-04-12T03:58:36.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.457+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.457+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.457+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: sync status for source bucket shard: 0 2024-04-12T03:58:36.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.457+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.457+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.457+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.457+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.458+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.459+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: sync status for source bucket shard: 0 2024-04-12T03:58:36.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.459+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.459+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: sync status for source bucket shard: 0 2024-04-12T03:58:36.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.459+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.459+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.459+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.459+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.459+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.459+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.459+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.459+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.460+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: sync status for source bucket shard: 0 2024-04-12T03:58:36.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.460+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.460+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: sync status for source bucket shard: 0 2024-04-12T03:58:36.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.460+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: sync status for source bucket shard: 0 2024-04-12T03:58:36.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.460+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.460+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket shard: 0 2024-04-12T03:58:36.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.460+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.460+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.460+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.460+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.461+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: sync status for source bucket shard: 0 2024-04-12T03:58:36.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.461+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: sync status for source bucket shard: 0 2024-04-12T03:58:36.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.461+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.461+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.461+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.461+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.461+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: sync status for source bucket shard: 0 2024-04-12T03:58:36.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.461+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.461+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.461+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.462+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: sync status for source bucket shard: 0 2024-04-12T03:58:36.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.462+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.462+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: sync status for source bucket shard: 0 2024-04-12T03:58:36.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.462+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: sync status for source bucket shard: 0 2024-04-12T03:58:36.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.462+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: sync status for source bucket shard: 0 2024-04-12T03:58:36.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.462+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: sync status for source bucket shard: 0 2024-04-12T03:58:36.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.462+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: sync status for source bucket shard: 0 2024-04-12T03:58:36.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.462+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.462+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.462+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.462+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.462+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.462+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.463+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: sync status for source bucket shard: 0 2024-04-12T03:58:36.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.463+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: sync status for source bucket shard: 0 2024-04-12T03:58:36.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.463+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.463+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.464+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: sync status for source bucket shard: 0 2024-04-12T03:58:36.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.464+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.464+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2024-04-12T03:58:36.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.464+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13/obj00000000000000000045 2024-04-12T03:58:36.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.464+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: entry->timestamp=2024-04-12T03:58:31.791489+0000 2024-04-12T03:58:36.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.464+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.56.6 op=0 op_state=1 2024-04-12T03:58:36.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.464+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: parsed entry: id=00000000002.25.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-04-12T03:58:36.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.464+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14/obj00000000000000000013 2024-04-12T03:58:36.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.464+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: entry->timestamp=2024-04-12T03:58:31.851398+0000 2024-04-12T03:58:36.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.464+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.25.6 op=0 op_state=1 2024-04-12T03:58:36.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.464+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-12T03:58:36.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.464+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12/obj00000000000000000009 2024-04-12T03:58:36.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.464+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: entry->timestamp=2024-04-12T03:58:31.914834+0000 2024-04-12T03:58:36.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.465+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.47.6 op=0 op_state=1 2024-04-12T03:58:36.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.465+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-12T03:58:36.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.465+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15/obj00000000000000000012 2024-04-12T03:58:36.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.465+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: entry->timestamp=2024-04-12T03:58:31.930686+0000 2024-04-12T03:58:36.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.465+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.73.6 op=0 op_state=1 2024-04-12T03:58:36.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.465+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000045]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000045[0] 2024-04-12T03:58:36.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.465+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position=00000000001.56.6 2024-04-12T03:58:36.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.465+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000013]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000013[0] 2024-04-12T03:58:36.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.465+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: listing bilog for incremental sync; position=00000000002.25.6 2024-04-12T03:58:36.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.465+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000009]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000009[0] 2024-04-12T03:58:36.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.465+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: listing bilog for incremental sync; position=00000000002.47.6 2024-04-12T03:58:36.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.465+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000012]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000012[0] 2024-04-12T03:58:36.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.465+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: listing bilog for incremental sync; position=00000000002.73.6 2024-04-12T03:58:36.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.466+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: parsed entry: id=00000000002.105.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-12T03:58:36.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.466+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18/obj00000000000000000000 2024-04-12T03:58:36.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.466+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: entry->timestamp=2024-04-12T03:58:31.416074+0000 2024-04-12T03:58:36.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.466+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.105.6 op=0 op_state=1 2024-04-12T03:58:36.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.466+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000000]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000000[0] 2024-04-12T03:58:36.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.466+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: listing bilog for incremental sync; position=00000000002.105.6 2024-04-12T03:58:36.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.467+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: sync status for source bucket shard: 0 2024-04-12T03:58:36.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.467+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.468+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000012]: success 2024-04-12T03:58:36.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.468+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15:3 marker=00000000002.73.6 timestamp=2024-04-12T03:58:31.930686+0000 2024-04-12T03:58:36.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-12T03:58:36.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17/obj00000000000000000001 2024-04-12T03:58:36.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: entry->timestamp=2024-04-12T03:58:31.613591+0000 2024-04-12T03:58:36.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.54.6 op=0 op_state=1 2024-04-12T03:58:36.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: parsed entry: id=00000000002.107.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-12T03:58:36.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20/obj00000000000000000003 2024-04-12T03:58:36.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: entry->timestamp=2024-04-12T03:58:31.727476+0000 2024-04-12T03:58:36.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.107.6 op=0 op_state=1 2024-04-12T03:58:36.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000001[0] 2024-04-12T03:58:36.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: listing bilog for incremental sync; position=00000000002.54.6 2024-04-12T03:58:36.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-12T03:58:36.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7/obj00000000000000000002 2024-04-12T03:58:36.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: entry->timestamp=2024-04-12T03:58:31.467916+0000 2024-04-12T03:58:36.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.47.6 op=0 op_state=1 2024-04-12T03:58:36.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000003]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000003[0] 2024-04-12T03:58:36.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: listing bilog for incremental sync; position=00000000002.107.6 2024-04-12T03:58:36.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:entry[obj00000000000000000002]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000002[0] 2024-04-12T03:58:36.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: listing bilog for incremental sync; position=00000000002.47.6 2024-04-12T03:58:36.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: parsed entry: id=00000000001.69.6 iter->object=obj00000000000000000058 iter->instance= name=obj00000000000000000058 instance= ns= 2024-04-12T03:58:36.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0/obj00000000000000000058 2024-04-12T03:58:36.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.469+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: entry->timestamp=2024-04-12T03:58:31.481097+0000 2024-04-12T03:58:36.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.470+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:entry[obj00000000000000000058]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.69.6 op=0 op_state=1 2024-04-12T03:58:36.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.470+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: parsed entry: id=00000000001.75.6 iter->object=obj00000000000000000055 iter->instance= name=obj00000000000000000055 instance= ns= 2024-04-12T03:58:36.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.470+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21/obj00000000000000000055 2024-04-12T03:58:36.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.470+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: entry->timestamp=2024-04-12T03:58:32.047512+0000 2024-04-12T03:58:36.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.470+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:entry[obj00000000000000000055]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.75.6 op=0 op_state=1 2024-04-12T03:58:36.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.470+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:entry[obj00000000000000000058]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000058[0] 2024-04-12T03:58:36.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.470+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: listing bilog for incremental sync; position=00000000001.69.6 2024-04-12T03:58:36.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.470+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000009]: success 2024-04-12T03:58:36.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.470+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12:3 marker=00000000002.47.6 timestamp=2024-04-12T03:58:31.914834+0000 2024-04-12T03:58:36.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.470+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:entry[obj00000000000000000055]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000055[0] 2024-04-12T03:58:36.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.470+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: listing bilog for incremental sync; position=00000000001.75.6 2024-04-12T03:58:36.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.470+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000013]: success 2024-04-12T03:58:36.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.470+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14:3 marker=00000000002.25.6 timestamp=2024-04-12T03:58:31.851398+0000 2024-04-12T03:58:36.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.471+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000000]: success 2024-04-12T03:58:36.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.471+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18:3 marker=00000000002.105.6 timestamp=2024-04-12T03:58:31.416074+0000 2024-04-12T03:58:36.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.472+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:entry[obj00000000000000000001]: success 2024-04-12T03:58:36.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.472+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17:3 marker=00000000002.54.6 timestamp=2024-04-12T03:58:31.613591+0000 2024-04-12T03:58:36.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.473+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000003]: success 2024-04-12T03:58:36.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.473+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20:3 marker=00000000002.107.6 timestamp=2024-04-12T03:58:31.727476+0000 2024-04-12T03:58:36.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.476+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:entry[obj00000000000000000002]: success 2024-04-12T03:58:36.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.476+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7:3 marker=00000000002.47.6 timestamp=2024-04-12T03:58:31.467916+0000 2024-04-12T03:58:36.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.478+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000045]: success 2024-04-12T03:58:36.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.478+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13:3 marker=00000000001.56.6 timestamp=2024-04-12T03:58:31.791489+0000 2024-04-12T03:58:36.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-04-12T03:58:36.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1/obj00000000000000000026 2024-04-12T03:58:36.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: entry->timestamp=2024-04-12T03:58:31.535826+0000 2024-04-12T03:58:36.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.35.6 op=0 op_state=1 2024-04-12T03:58:36.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000026]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000026[0] 2024-04-12T03:58:36.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: listing bilog for incremental sync; position=00000000002.35.6 2024-04-12T03:58:36.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: parsed entry: id=00000000002.71.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-04-12T03:58:36.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4/obj00000000000000000014 2024-04-12T03:58:36.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: entry->timestamp=2024-04-12T03:58:31.550334+0000 2024-04-12T03:58:36.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.71.6 op=0 op_state=1 2024-04-12T03:58:36.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: parsed entry: id=00000000002.23.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-12T03:58:36.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2/obj00000000000000000011 2024-04-12T03:58:36.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: entry->timestamp=2024-04-12T03:58:31.599462+0000 2024-04-12T03:58:36.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.23.6 op=0 op_state=1 2024-04-12T03:58:36.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000046 iter->instance= name=obj00000000000000000046 instance= ns= 2024-04-12T03:58:36.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3/obj00000000000000000046 2024-04-12T03:58:36.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: entry->timestamp=2024-04-12T03:58:31.663508+0000 2024-04-12T03:58:36.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:entry[obj00000000000000000046]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.45.6 op=0 op_state=1 2024-04-12T03:58:36.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000014]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000014[0] 2024-04-12T03:58:36.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position=00000000002.71.6 2024-04-12T03:58:36.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000011[0] 2024-04-12T03:58:36.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position=00000000002.23.6 2024-04-12T03:58:36.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:entry[obj00000000000000000046]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000046[0] 2024-04-12T03:58:36.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.500+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: listing bilog for incremental sync; position=00000000001.45.6 2024-04-12T03:58:36.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.502+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: parsed entry: id=00000000001.52.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2024-04-12T03:58:36.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.502+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8/obj00000000000000000039 2024-04-12T03:58:36.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.502+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: entry->timestamp=2024-04-12T03:58:31.865319+0000 2024-04-12T03:58:36.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.502+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.52.6 op=0 op_state=1 2024-04-12T03:58:36.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.502+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000039]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000039[0] 2024-04-12T03:58:36.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.502+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: listing bilog for incremental sync; position=00000000001.52.6 2024-04-12T03:58:36.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.502+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2024-04-12T03:58:36.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.502+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5/obj00000000000000000050 2024-04-12T03:58:36.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.502+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: entry->timestamp=2024-04-12T03:58:31.404294+0000 2024-04-12T03:58:36.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.502+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.43.6 op=0 op_state=1 2024-04-12T03:58:36.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.502+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:entry[obj00000000000000000050]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000050[0] 2024-04-12T03:58:36.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.502+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: listing bilog for incremental sync; position=00000000001.43.6 2024-04-12T03:58:36.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.503+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-12T03:58:36.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.503+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10/obj00000000000000000006 2024-04-12T03:58:36.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.503+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: entry->timestamp=2024-04-12T03:58:31.740777+0000 2024-04-12T03:58:36.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.503+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.39.6 op=0 op_state=1 2024-04-12T03:58:36.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.503+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:entry[obj00000000000000000006]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000006[0] 2024-04-12T03:58:36.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.503+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: listing bilog for incremental sync; position=00000000002.39.6 2024-04-12T03:58:36.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.504+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: parsed entry: id=00000000001.46.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2024-04-12T03:58:36.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.504+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11/obj00000000000000000040 2024-04-12T03:58:36.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.504+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: entry->timestamp=2024-04-12T03:58:31.803518+0000 2024-04-12T03:58:36.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.504+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.46.6 op=0 op_state=1 2024-04-12T03:58:36.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.504+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000040]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000040[0] 2024-04-12T03:58:36.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.504+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: listing bilog for incremental sync; position=00000000001.46.6 2024-04-12T03:58:36.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.504+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000026]: success 2024-04-12T03:58:36.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.504+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1:3 marker=00000000002.35.6 timestamp=2024-04-12T03:58:31.535826+0000 2024-04-12T03:58:36.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.504+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000011]: success 2024-04-12T03:58:36.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.504+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2:3 marker=00000000002.23.6 timestamp=2024-04-12T03:58:31.599462+0000 2024-04-12T03:58:36.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.504+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000014]: success 2024-04-12T03:58:36.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.504+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4:3 marker=00000000002.71.6 timestamp=2024-04-12T03:58:31.550334+0000 2024-04-12T03:58:36.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.506+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-04-12T03:58:36.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.506+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16/obj00000000000000000033 2024-04-12T03:58:36.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.506+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: entry->timestamp=2024-04-12T03:58:31.997656+0000 2024-04-12T03:58:36.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.506+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.47.6 op=0 op_state=1 2024-04-12T03:58:36.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.506+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000033]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000033[0] 2024-04-12T03:58:36.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.506+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: listing bilog for incremental sync; position=00000000002.47.6 2024-04-12T03:58:36.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.507+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:entry[obj00000000000000000006]: success 2024-04-12T03:58:36.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.507+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10:3 marker=00000000002.39.6 timestamp=2024-04-12T03:58:31.740777+0000 2024-04-12T03:58:36.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.508+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-12T03:58:36.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.508+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22/obj00000000000000000008 2024-04-12T03:58:36.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.508+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: entry->timestamp=2024-04-12T03:58:31.675724+0000 2024-04-12T03:58:36.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.508+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.59.6 op=0 op_state=1 2024-04-12T03:58:36.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.508+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:entry[obj00000000000000000008]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000008[0] 2024-04-12T03:58:36.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.508+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: listing bilog for incremental sync; position=00000000002.59.6 2024-04-12T03:58:36.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.511+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2024-04-12T03:58:36.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.511+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6/obj00000000000000000032 2024-04-12T03:58:36.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.511+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: entry->timestamp=2024-04-12T03:58:31.983268+0000 2024-04-12T03:58:36.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.511+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.68.6 op=0 op_state=1 2024-04-12T03:58:36.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.511+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000032[0] 2024-04-12T03:58:36.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.511+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: listing bilog for incremental sync; position=00000000002.68.6 2024-04-12T03:58:36.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.511+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000033]: success 2024-04-12T03:58:36.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.511+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16:3 marker=00000000002.47.6 timestamp=2024-04-12T03:58:31.997656+0000 2024-04-12T03:58:36.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.513+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000032]: success 2024-04-12T03:58:36.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.513+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6:3 marker=00000000002.68.6 timestamp=2024-04-12T03:58:31.983268+0000 2024-04-12T03:58:36.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.518+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:entry[obj00000000000000000055]: success 2024-04-12T03:58:36.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.518+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21:3 marker=00000000001.75.6 timestamp=2024-04-12T03:58:32.047512+0000 2024-04-12T03:58:36.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.519+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:entry[obj00000000000000000008]: success 2024-04-12T03:58:36.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.519+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22:3 marker=00000000002.59.6 timestamp=2024-04-12T03:58:31.675724+0000 2024-04-12T03:58:36.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.524+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:entry[obj00000000000000000058]: success 2024-04-12T03:58:36.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.524+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0:3 marker=00000000001.69.6 timestamp=2024-04-12T03:58:31.481097+0000 2024-04-12T03:58:36.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.550+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:entry[obj00000000000000000046]: success 2024-04-12T03:58:36.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.550+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3:3 marker=00000000001.45.6 timestamp=2024-04-12T03:58:31.663508+0000 2024-04-12T03:58:36.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.553+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:entry[obj00000000000000000050]: success 2024-04-12T03:58:36.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.553+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5:3 marker=00000000001.43.6 timestamp=2024-04-12T03:58:31.404294+0000 2024-04-12T03:58:36.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.556+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000040]: success 2024-04-12T03:58:36.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.556+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11:3 marker=00000000001.46.6 timestamp=2024-04-12T03:58:31.803518+0000 2024-04-12T03:58:36.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.556+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000039]: success 2024-04-12T03:58:36.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.556+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8:3 marker=00000000001.52.6 timestamp=2024-04-12T03:58:31.865319+0000 2024-04-12T03:58:36.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.572+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.572+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.572+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.573+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.573+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: sync status for source bucket shard: 0 2024-04-12T03:58:36.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.573+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.573+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: sync status for source bucket shard: 0 2024-04-12T03:58:36.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.573+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.573+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.573+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket shard: 0 2024-04-12T03:58:36.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.573+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.573+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.573+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: sync status for source bucket shard: 0 2024-04-12T03:58:36.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.573+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.573+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.573+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.573+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket shard: 0 2024-04-12T03:58:36.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.573+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.574+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: sync status for source bucket shard: 0 2024-04-12T03:58:36.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.574+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.574+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.574+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.574+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: sync status for source bucket shard: 0 2024-04-12T03:58:36.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.574+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.574+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.575+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: sync status for source bucket shard: 0 2024-04-12T03:58:36.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.575+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.575+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: sync status for source bucket shard: 0 2024-04-12T03:58:36.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.575+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.576+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: sync status for source bucket shard: 0 2024-04-12T03:58:36.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.576+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.576+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: sync status for source bucket shard: 0 2024-04-12T03:58:36.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.576+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.576+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.576+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.576+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.576+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.576+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.576+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.576+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.576+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.577+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.577+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.577+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.577+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: sync status for source bucket shard: 0 2024-04-12T03:58:36.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.577+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: sync status for source bucket shard: 0 2024-04-12T03:58:36.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.577+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: sync status for source bucket shard: 0 2024-04-12T03:58:36.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.577+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: sync status for source bucket shard: 0 2024-04-12T03:58:36.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.577+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket shard: 0 2024-04-12T03:58:36.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: sync status for source bucket shard: 0 2024-04-12T03:58:36.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: sync status for source bucket shard: 0 2024-04-12T03:58:36.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: sync status for source bucket shard: 0 2024-04-12T03:58:36.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: sync status for source bucket shard: 0 2024-04-12T03:58:36.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: sync status for source bucket shard: 0 2024-04-12T03:58:36.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.578+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.579+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.579+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: sync status for source bucket shard: 0 2024-04-12T03:58:36.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.579+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.579+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.579+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.581+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: sync status for source bucket shard: 0 2024-04-12T03:58:36.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.581+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.581+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: sync status for source bucket shard: 0 2024-04-12T03:58:36.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.581+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: sync status for source bucket shard: 0 2024-04-12T03:58:36.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.581+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: sync status for source bucket shard: 0 2024-04-12T03:58:36.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.581+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: sync status for source bucket shard: 0 2024-04-12T03:58:36.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.581+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.581+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.581+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.581+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.581+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.581+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: parsed entry: id=00000000001.31.6 iter->object=obj00000000000000000037 iter->instance= name=obj00000000000000000037 instance= ns= 2024-04-12T03:58:36.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15/obj00000000000000000037 2024-04-12T03:58:36.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: entry->timestamp=2024-04-12T03:58:32.547464+0000 2024-04-12T03:58:36.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000037]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.31.6 op=0 op_state=1 2024-04-12T03:58:36.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-04-12T03:58:36.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13/obj00000000000000000013 2024-04-12T03:58:36.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: entry->timestamp=2024-04-12T03:58:32.364218+0000 2024-04-12T03:58:36.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.67.6 op=0 op_state=1 2024-04-12T03:58:36.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000037]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000037[0] 2024-04-12T03:58:36.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: listing bilog for incremental sync; position=00000000001.31.6 2024-04-12T03:58:36.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: parsed entry: id=00000000001.63.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2024-04-12T03:58:36.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14/obj00000000000000000020 2024-04-12T03:58:36.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: entry->timestamp=2024-04-12T03:58:33.059878+0000 2024-04-12T03:58:36.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.63.6 op=0 op_state=1 2024-04-12T03:58:36.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000013]: : injecting data sync error on key=obj00000000000000000013 2024-04-12T03:58:36.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position=00000000002.67.6 2024-04-12T03:58:36.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-12T03:58:36.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17/obj00000000000000000012 2024-04-12T03:58:36.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.582+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: entry->timestamp=2024-04-12T03:58:32.479650+0000 2024-04-12T03:58:36.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.61.6 op=0 op_state=1 2024-04-12T03:58:36.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-12T03:58:36.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20/obj00000000000000000010 2024-04-12T03:58:36.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: entry->timestamp=2024-04-12T03:58:32.493874+0000 2024-04-12T03:58:36.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.69.6 op=0 op_state=1 2024-04-12T03:58:36.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: parsed entry: id=00000000001.71.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2024-04-12T03:58:36.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11/obj00000000000000000031 2024-04-12T03:58:36.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: entry->timestamp=2024-04-12T03:58:33.123895+0000 2024-04-12T03:58:36.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.71.6 op=0 op_state=1 2024-04-12T03:58:36.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000020]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000020[0] 2024-04-12T03:58:36.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: listing bilog for incremental sync; position=00000000001.63.6 2024-04-12T03:58:36.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-12T03:58:36.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18/obj00000000000000000003 2024-04-12T03:58:36.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: entry->timestamp=2024-04-12T03:58:32.815441+0000 2024-04-12T03:58:36.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.58.6 op=0 op_state=1 2024-04-12T03:58:36.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: parsed entry: id=00000000001.140.6 iter->object=obj00000000000000000042 iter->instance= name=obj00000000000000000042 instance= ns= 2024-04-12T03:58:36.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19/obj00000000000000000042 2024-04-12T03:58:36.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: entry->timestamp=2024-04-12T03:58:32.942465+0000 2024-04-12T03:58:36.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:entry[obj00000000000000000042]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.140.6 op=0 op_state=1 2024-04-12T03:58:36.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000013]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000013]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13/obj00000000000000000013 2024-04-12T03:58:36.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:entry[obj00000000000000000012]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000012[0] 2024-04-12T03:58:36.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: listing bilog for incremental sync; position=00000000002.61.6 2024-04-12T03:58:36.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-04-12T03:58:36.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16/obj00000000000000000005 2024-04-12T03:58:36.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: entry->timestamp=2024-04-12T03:58:32.607610+0000 2024-04-12T03:58:36.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.51.6 op=0 op_state=1 2024-04-12T03:58:36.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000010]: : injecting data sync error on key=obj00000000000000000010 2024-04-12T03:58:36.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: listing bilog for incremental sync; position=00000000002.69.6 2024-04-12T03:58:36.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000031]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000031[0] 2024-04-12T03:58:36.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: listing bilog for incremental sync; position=00000000001.71.6 2024-04-12T03:58:36.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000003]: : injecting data sync error on key=obj00000000000000000003 2024-04-12T03:58:36.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: listing bilog for incremental sync; position=00000000002.58.6 2024-04-12T03:58:36.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:entry[obj00000000000000000042]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000042[0] 2024-04-12T03:58:36.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: listing bilog for incremental sync; position=00000000001.140.6 2024-04-12T03:58:36.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000005]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000005[0] 2024-04-12T03:58:36.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: listing bilog for incremental sync; position=00000000002.51.6 2024-04-12T03:58:36.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000010]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000010]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20/obj00000000000000000010 2024-04-12T03:58:36.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000003]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.583+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000003]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18/obj00000000000000000003 2024-04-12T03:58:36.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.584+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: sync status for source bucket shard: 0 2024-04-12T03:58:36.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.584+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: sync status for source bucket shard: 0 2024-04-12T03:58:36.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.584+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.584+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.584+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-12T03:58:36.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.584+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21/obj00000000000000000001 2024-04-12T03:58:36.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.584+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: entry->timestamp=2024-04-12T03:58:32.686497+0000 2024-04-12T03:58:36.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.584+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.62.6 op=0 op_state=1 2024-04-12T03:58:36.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.584+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-12T03:58:36.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.584+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22/obj00000000000000000009 2024-04-12T03:58:36.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.584+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: entry->timestamp=2024-04-12T03:58:32.800500+0000 2024-04-12T03:58:36.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.584+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.69.6 op=0 op_state=1 2024-04-12T03:58:36.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.584+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:entry[obj00000000000000000001]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000001[0] 2024-04-12T03:58:36.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.584+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: listing bilog for incremental sync; position=00000000002.62.6 2024-04-12T03:58:36.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.585+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:entry[obj00000000000000000009]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000009[0] 2024-04-12T03:58:36.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.585+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: listing bilog for incremental sync; position=00000000002.69.6 2024-04-12T03:58:36.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.589+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-04-12T03:58:36.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.589+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27/obj00000000000000000030 2024-04-12T03:58:36.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.589+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: entry->timestamp=2024-04-12T03:58:32.559387+0000 2024-04-12T03:58:36.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.589+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.64.6 op=0 op_state=1 2024-04-12T03:58:36.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.589+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:entry[obj00000000000000000030]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000030[0] 2024-04-12T03:58:36.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.589+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: listing bilog for incremental sync; position=00000000002.64.6 2024-04-12T03:58:36.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.589+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:entry[obj00000000000000000012]: success 2024-04-12T03:58:36.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.589+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17:4 marker=00000000002.61.6 timestamp=2024-04-12T03:58:32.479650+0000 2024-04-12T03:58:36.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.591+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: parsed entry: id=00000000002.142.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-12T03:58:36.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.591+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28/obj00000000000000000021 2024-04-12T03:58:36.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.591+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: entry->timestamp=2024-04-12T03:58:33.004944+0000 2024-04-12T03:58:36.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.592+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.142.6 op=0 op_state=1 2024-04-12T03:58:36.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.592+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:entry[obj00000000000000000009]: success 2024-04-12T03:58:36.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.592+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22:4 marker=00000000002.69.6 timestamp=2024-04-12T03:58:32.800500+0000 2024-04-12T03:58:36.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.592+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:entry[obj00000000000000000021]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000021[0] 2024-04-12T03:58:36.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.592+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: listing bilog for incremental sync; position=00000000002.142.6 2024-04-12T03:58:36.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.592+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: a sync operation returned error 2024-04-12T03:58:36.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.592+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: backing out with sync_status=-5 2024-04-12T03:58:36.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.592+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.592+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.592+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000005]: success 2024-04-12T03:58:36.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.592+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16:4 marker=00000000002.51.6 timestamp=2024-04-12T03:58:32.607610+0000 2024-04-12T03:58:36.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.592+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:entry[obj00000000000000000001]: success 2024-04-12T03:58:36.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.592+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21:4 marker=00000000002.62.6 timestamp=2024-04-12T03:58:32.686497+0000 2024-04-12T03:58:36.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.593+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.595+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: sync status for source bucket shard: 0 2024-04-12T03:58:36.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.595+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.596+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:entry[obj00000000000000000030]: success 2024-04-12T03:58:36.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.596+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27:4 marker=00000000002.64.6 timestamp=2024-04-12T03:58:32.559387+0000 2024-04-12T03:58:36.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.597+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:entry[obj00000000000000000021]: success 2024-04-12T03:58:36.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.597+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28:4 marker=00000000002.142.6 timestamp=2024-04-12T03:58:33.004944+0000 2024-04-12T03:58:36.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.598+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000031]: success 2024-04-12T03:58:36.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.598+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11:4 marker=00000000001.71.6 timestamp=2024-04-12T03:58:33.123895+0000 2024-04-12T03:58:36.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.598+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000020]: success 2024-04-12T03:58:36.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.598+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14:4 marker=00000000001.63.6 timestamp=2024-04-12T03:58:33.059878+0000 2024-04-12T03:58:36.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.616+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: parsed entry: id=00000000001.51.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2024-04-12T03:58:36.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.616+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1/obj00000000000000000059 2024-04-12T03:58:36.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.616+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: entry->timestamp=2024-04-12T03:58:32.864361+0000 2024-04-12T03:58:36.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.616+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.51.6 op=0 op_state=1 2024-04-12T03:58:36.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.616+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000059]: : injecting data sync error on key=obj00000000000000000059 2024-04-12T03:58:36.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.616+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: listing bilog for incremental sync; position=00000000001.51.6 2024-04-12T03:58:36.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.616+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000059]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.616+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000059]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1/obj00000000000000000059 2024-04-12T03:58:36.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-12T03:58:36.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0/obj00000000000000000006 2024-04-12T03:58:36.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: entry->timestamp=2024-04-12T03:58:33.072597+0000 2024-04-12T03:58:36.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.54.6 op=0 op_state=1 2024-04-12T03:58:36.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:entry[obj00000000000000000006]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000006[0] 2024-04-12T03:58:36.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: listing bilog for incremental sync; position=00000000002.54.6 2024-04-12T03:58:36.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-12T03:58:36.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4/obj00000000000000000011 2024-04-12T03:58:36.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: entry->timestamp=2024-04-12T03:58:33.136322+0000 2024-04-12T03:58:36.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.38.6 op=0 op_state=1 2024-04-12T03:58:36.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: parsed entry: id=00000000002.33.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-12T03:58:36.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5/obj00000000000000000002 2024-04-12T03:58:36.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: entry->timestamp=2024-04-12T03:58:32.928442+0000 2024-04-12T03:58:36.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.33.6 op=0 op_state=1 2024-04-12T03:58:36.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000011]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000011[0] 2024-04-12T03:58:36.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position=00000000002.38.6 2024-04-12T03:58:36.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:entry[obj00000000000000000002]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000002[0] 2024-04-12T03:58:36.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.617+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: listing bilog for incremental sync; position=00000000002.33.6 2024-04-12T03:58:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2024-04-12T03:58:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2/obj00000000000000000027 2024-04-12T03:58:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: entry->timestamp=2024-04-12T03:58:32.992044+0000 2024-04-12T03:58:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.58.6 op=0 op_state=1 2024-04-12T03:58:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-12T03:58:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3/obj00000000000000000004 2024-04-12T03:58:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: entry->timestamp=2024-04-12T03:58:32.751464+0000 2024-04-12T03:58:36.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.59.6 op=0 op_state=1 2024-04-12T03:58:36.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2024-04-12T03:58:36.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6/obj00000000000000000019 2024-04-12T03:58:36.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: entry->timestamp=2024-04-12T03:58:32.415462+0000 2024-04-12T03:58:36.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.85.6 op=0 op_state=1 2024-04-12T03:58:36.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000027]: : injecting data sync error on key=obj00000000000000000027 2024-04-12T03:58:36.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position=00000000001.58.6 2024-04-12T03:58:36.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: parsed entry: id=00000000001.39.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-04-12T03:58:36.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8/obj00000000000000000017 2024-04-12T03:58:36.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: entry->timestamp=2024-04-12T03:58:32.877422+0000 2024-04-12T03:58:36.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.39.6 op=0 op_state=1 2024-04-12T03:58:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-12T03:58:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9/obj00000000000000000008 2024-04-12T03:58:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: entry->timestamp=2024-04-12T03:58:32.736261+0000 2024-04-12T03:58:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.66.6 op=0 op_state=1 2024-04-12T03:58:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:entry[obj00000000000000000004]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000004[0] 2024-04-12T03:58:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: listing bilog for incremental sync; position=00000000002.59.6 2024-04-12T03:58:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000019]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000019[0] 2024-04-12T03:58:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: listing bilog for incremental sync; position=00000000002.85.6 2024-04-12T03:58:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000027]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000027]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2/obj00000000000000000027 2024-04-12T03:58:36.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000017]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000017[0] 2024-04-12T03:58:36.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: listing bilog for incremental sync; position=00000000001.39.6 2024-04-12T03:58:36.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:entry[obj00000000000000000008]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000008[0] 2024-04-12T03:58:36.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.618+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: listing bilog for incremental sync; position=00000000002.66.6 2024-04-12T03:58:36.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.621+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:entry[obj00000000000000000002]: success 2024-04-12T03:58:36.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.621+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5:4 marker=00000000002.33.6 timestamp=2024-04-12T03:58:32.928442+0000 2024-04-12T03:58:36.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.621+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000011]: success 2024-04-12T03:58:36.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.621+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4:4 marker=00000000002.38.6 timestamp=2024-04-12T03:58:33.136322+0000 2024-04-12T03:58:36.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.622+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-04-12T03:58:36.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.622+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10/obj00000000000000000015 2024-04-12T03:58:36.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.622+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: entry->timestamp=2024-04-12T03:58:33.187630+0000 2024-04-12T03:58:36.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.622+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.65.6 op=0 op_state=1 2024-04-12T03:58:36.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.622+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:entry[obj00000000000000000015]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000015[0] 2024-04-12T03:58:36.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.622+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: listing bilog for incremental sync; position=00000000002.65.6 2024-04-12T03:58:36.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.622+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:entry[obj00000000000000000006]: success 2024-04-12T03:58:36.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.622+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0:4 marker=00000000002.54.6 timestamp=2024-04-12T03:58:33.072597+0000 2024-04-12T03:58:36.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.623+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:entry[obj00000000000000000004]: success 2024-04-12T03:58:36.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.623+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3:4 marker=00000000002.59.6 timestamp=2024-04-12T03:58:32.751464+0000 2024-04-12T03:58:36.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000019]: success 2024-04-12T03:58:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6:4 marker=00000000002.85.6 timestamp=2024-04-12T03:58:32.415462+0000 2024-04-12T03:58:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-12T03:58:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24/obj00000000000000000016 2024-04-12T03:58:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: entry->timestamp=2024-04-12T03:58:32.671802+0000 2024-04-12T03:58:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.52.6 op=0 op_state=1 2024-04-12T03:58:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2024-04-12T03:58:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23/obj00000000000000000025 2024-04-12T03:58:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: entry->timestamp=2024-04-12T03:58:33.202862+0000 2024-04-12T03:58:36.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.77.6 op=0 op_state=1 2024-04-12T03:58:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:entry[obj00000000000000000008]: success 2024-04-12T03:58:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9:4 marker=00000000002.66.6 timestamp=2024-04-12T03:58:32.736261+0000 2024-04-12T03:58:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:entry[obj00000000000000000016]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000016[0] 2024-04-12T03:58:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: listing bilog for incremental sync; position=00000000002.52.6 2024-04-12T03:58:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2024-04-12T03:58:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12/obj00000000000000000022 2024-04-12T03:58:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: entry->timestamp=2024-04-12T03:58:32.621725+0000 2024-04-12T03:58:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.56.6 op=0 op_state=1 2024-04-12T03:58:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: parsed entry: id=00000000002.36.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-04-12T03:58:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26/obj00000000000000000014 2024-04-12T03:58:36.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: entry->timestamp=2024-04-12T03:58:32.428607+0000 2024-04-12T03:58:36.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.36.6 op=0 op_state=1 2024-04-12T03:58:36.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:entry[obj00000000000000000025]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000025[0] 2024-04-12T03:58:36.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: listing bilog for incremental sync; position=00000000002.77.6 2024-04-12T03:58:36.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000022]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000022[0] 2024-04-12T03:58:36.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: listing bilog for incremental sync; position=00000000002.56.6 2024-04-12T03:58:36.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:entry[obj00000000000000000014]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000014[0] 2024-04-12T03:58:36.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.625+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: listing bilog for incremental sync; position=00000000002.36.6 2024-04-12T03:58:36.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.628+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:entry[obj00000000000000000015]: success 2024-04-12T03:58:36.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.628+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10:4 marker=00000000002.65.6 timestamp=2024-04-12T03:58:33.187630+0000 2024-04-12T03:58:36.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.628+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: a sync operation returned error 2024-04-12T03:58:36.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.628+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: backing out with sync_status=-5 2024-04-12T03:58:36.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.628+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.628+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.630+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:entry[obj00000000000000000025]: success 2024-04-12T03:58:36.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.630+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23:4 marker=00000000002.77.6 timestamp=2024-04-12T03:58:33.202862+0000 2024-04-12T03:58:36.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.630+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:entry[obj00000000000000000016]: success 2024-04-12T03:58:36.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.630+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24:4 marker=00000000002.52.6 timestamp=2024-04-12T03:58:32.671802+0000 2024-04-12T03:58:36.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.630+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:entry[obj00000000000000000014]: success 2024-04-12T03:58:36.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.630+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26:4 marker=00000000002.36.6 timestamp=2024-04-12T03:58:32.428607+0000 2024-04-12T03:58:36.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.631+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: a sync operation returned error 2024-04-12T03:58:36.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.631+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: backing out with sync_status=-5 2024-04-12T03:58:36.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.631+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.631+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.634+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.634+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket shard: 0 2024-04-12T03:58:36.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.634+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.635+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.635+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: sync status for source bucket shard: 0 2024-04-12T03:58:36.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.635+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.636+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000037]: success 2024-04-12T03:58:36.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.636+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15:4 marker=00000000001.31.6 timestamp=2024-04-12T03:58:32.547464+0000 2024-04-12T03:58:36.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.636+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:entry[obj00000000000000000042]: success 2024-04-12T03:58:36.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.636+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19:4 marker=00000000001.140.6 timestamp=2024-04-12T03:58:32.942465+0000 2024-04-12T03:58:36.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.638+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-12T03:58:36.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.638+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18/obj00000000000000000003 2024-04-12T03:58:36.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.638+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: entry->timestamp=2024-04-12T03:58:32.815441+0000 2024-04-12T03:58:36.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.638+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.58.6 op=0 op_state=1 2024-04-12T03:58:36.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.638+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000003]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000003[0] 2024-04-12T03:58:36.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.638+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: listing bilog for incremental sync; position=00000000002.58.6 2024-04-12T03:58:36.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.641+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000003]: success 2024-04-12T03:58:36.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.641+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18:4 marker=00000000002.58.6 timestamp=2024-04-12T03:58:32.815441+0000 2024-04-12T03:58:36.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.658+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: a sync operation returned error 2024-04-12T03:58:36.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.658+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: backing out with sync_status=-5 2024-04-12T03:58:36.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.658+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.658+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.658+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.659+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: sync status for source bucket shard: 0 2024-04-12T03:58:36.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.659+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.662+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: a sync operation returned error 2024-04-12T03:58:36.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.662+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: backing out with sync_status=-5 2024-04-12T03:58:36.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.662+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.662+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.663+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.664+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket shard: 0 2024-04-12T03:58:36.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.664+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.667+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000017]: success 2024-04-12T03:58:36.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.667+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8:4 marker=00000000001.39.6 timestamp=2024-04-12T03:58:32.877422+0000 2024-04-12T03:58:36.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.674+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000022]: success 2024-04-12T03:58:36.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.674+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12:4 marker=00000000002.56.6 timestamp=2024-04-12T03:58:32.621725+0000 2024-04-12T03:58:36.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.677+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-04-12T03:58:36.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.677+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13/obj00000000000000000013 2024-04-12T03:58:36.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.677+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: entry->timestamp=2024-04-12T03:58:32.364218+0000 2024-04-12T03:58:36.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.677+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.67.6 op=0 op_state=1 2024-04-12T03:58:36.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.677+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000013]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000013[0] 2024-04-12T03:58:36.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.677+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position=00000000002.67.6 2024-04-12T03:58:36.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.679+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000013]: success 2024-04-12T03:58:36.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.680+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13:4 marker=00000000002.67.6 timestamp=2024-04-12T03:58:32.364218+0000 2024-04-12T03:58:36.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.680+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-12T03:58:36.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.680+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20/obj00000000000000000010 2024-04-12T03:58:36.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.680+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: entry->timestamp=2024-04-12T03:58:32.493874+0000 2024-04-12T03:58:36.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.680+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.69.6 op=0 op_state=1 2024-04-12T03:58:36.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.680+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000010]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000010[0] 2024-04-12T03:58:36.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.680+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: listing bilog for incremental sync; position=00000000002.69.6 2024-04-12T03:58:36.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.682+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000010]: success 2024-04-12T03:58:36.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.682+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20:4 marker=00000000002.69.6 timestamp=2024-04-12T03:58:32.493874+0000 2024-04-12T03:58:36.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.701+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: parsed entry: id=00000000001.51.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2024-04-12T03:58:36.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.701+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1/obj00000000000000000059 2024-04-12T03:58:36.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.701+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: entry->timestamp=2024-04-12T03:58:32.864361+0000 2024-04-12T03:58:36.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.701+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.51.6 op=0 op_state=1 2024-04-12T03:58:36.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.701+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000059]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000059[0] 2024-04-12T03:58:36.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.701+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: listing bilog for incremental sync; position=00000000001.51.6 2024-04-12T03:58:36.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.706+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2024-04-12T03:58:36.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.706+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2/obj00000000000000000027 2024-04-12T03:58:36.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.706+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: entry->timestamp=2024-04-12T03:58:32.992044+0000 2024-04-12T03:58:36.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.706+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.58.6 op=0 op_state=1 2024-04-12T03:58:36.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.706+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000027]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000027[0] 2024-04-12T03:58:36.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.706+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position=00000000001.58.6 2024-04-12T03:58:36.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.711+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000027]: success 2024-04-12T03:58:36.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.711+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2:4 marker=00000000001.58.6 timestamp=2024-04-12T03:58:32.992044+0000 2024-04-12T03:58:36.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.748+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000059]: success 2024-04-12T03:58:36.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.748+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1:4 marker=00000000001.51.6 timestamp=2024-04-12T03:58:32.864361+0000 2024-04-12T03:58:36.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.754+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.754+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.754+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.755+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: sync status for source bucket shard: 0 2024-04-12T03:58:36.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.755+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.755+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.755+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket shard: 0 2024-04-12T03:58:36.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.755+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.755+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: sync status for source bucket shard: 0 2024-04-12T03:58:36.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.755+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.755+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.756+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.756+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.756+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.756+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: sync status for source bucket shard: 0 2024-04-12T03:58:36.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.756+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.756+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.756+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: sync status for source bucket shard: 0 2024-04-12T03:58:36.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.756+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: sync status for source bucket shard: 0 2024-04-12T03:58:36.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.756+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: sync status for source bucket shard: 0 2024-04-12T03:58:36.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.756+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.756+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.756+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.756+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.757+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket shard: 0 2024-04-12T03:58:36.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.757+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.757+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.757+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: sync status for source bucket shard: 0 2024-04-12T03:58:36.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.757+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.757+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: sync status for source bucket shard: 0 2024-04-12T03:58:36.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.757+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.757+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.758+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.758+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.758+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.758+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: sync status for source bucket shard: 0 2024-04-12T03:58:36.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.758+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: sync status for source bucket shard: 0 2024-04-12T03:58:36.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: sync status for source bucket shard: 0 2024-04-12T03:58:36.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket shard: 0 2024-04-12T03:58:36.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: sync status for source bucket shard: 0 2024-04-12T03:58:36.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: parsed entry: id=00000000002.58.6 iter->object=obj00000000000000000023 iter->instance= name=obj00000000000000000023 instance= ns= 2024-04-12T03:58:36.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6/obj00000000000000000023 2024-04-12T03:58:36.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: entry->timestamp=2024-04-12T03:58:34.220083+0000 2024-04-12T03:58:36.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000023]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.58.6 op=0 op_state=1 2024-04-12T03:58:36.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-12T03:58:36.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5/obj00000000000000000004 2024-04-12T03:58:36.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: entry->timestamp=2024-04-12T03:58:34.139872+0000 2024-04-12T03:58:36.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.47.6 op=0 op_state=1 2024-04-12T03:58:36.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2024-04-12T03:58:36.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8/obj00000000000000000022 2024-04-12T03:58:36.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: entry->timestamp=2024-04-12T03:58:33.759748+0000 2024-04-12T03:58:36.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.760+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.44.6 op=0 op_state=1 2024-04-12T03:58:36.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.761+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000023]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000023[0] 2024-04-12T03:58:36.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.761+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: listing bilog for incremental sync; position=00000000002.58.6 2024-04-12T03:58:36.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.761+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.761+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:entry[obj00000000000000000004]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000004[0] 2024-04-12T03:58:36.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.761+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: listing bilog for incremental sync; position=00000000002.47.6 2024-04-12T03:58:36.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.761+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000022]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000022[0] 2024-04-12T03:58:36.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.761+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: listing bilog for incremental sync; position=00000000002.44.6 2024-04-12T03:58:36.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.761+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: parsed entry: id=00000000001.93.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2024-04-12T03:58:36.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.761+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4/obj00000000000000000056 2024-04-12T03:58:36.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.761+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: entry->timestamp=2024-04-12T03:58:34.027311+0000 2024-04-12T03:58:36.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.761+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.93.6 op=0 op_state=1 2024-04-12T03:58:36.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.761+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.761+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000056]: : injecting data sync error on key=obj00000000000000000056 2024-04-12T03:58:36.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.761+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position=00000000001.93.6 2024-04-12T03:58:36.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: parsed entry: id=00000000002.84.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-12T03:58:36.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9/obj00000000000000000000 2024-04-12T03:58:36.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: entry->timestamp=2024-04-12T03:58:33.584685+0000 2024-04-12T03:58:36.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.84.6 op=0 op_state=1 2024-04-12T03:58:36.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: parsed entry: id=00000000001.41.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2024-04-12T03:58:36.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7/obj00000000000000000317 2024-04-12T03:58:36.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: entry->timestamp=2024-04-12T03:58:34.346593+0000 2024-04-12T03:58:36.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.41.6 op=0 op_state=1 2024-04-12T03:58:36.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000056]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000056]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4/obj00000000000000000056 2024-04-12T03:58:36.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:entry[obj00000000000000000000]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000000[0] 2024-04-12T03:58:36.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: listing bilog for incremental sync; position=00000000002.84.6 2024-04-12T03:58:36.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:entry[obj00000000000000000317]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000317[0] 2024-04-12T03:58:36.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: listing bilog for incremental sync; position=00000000001.41.6 2024-04-12T03:58:36.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: sync status for source bucket shard: 0 2024-04-12T03:58:36.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.762+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.763+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.763+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.763+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.763+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: sync status for source bucket shard: 0 2024-04-12T03:58:36.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: sync status for source bucket shard: 0 2024-04-12T03:58:36.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: sync status for source bucket shard: 0 2024-04-12T03:58:36.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: sync status for source bucket shard: 0 2024-04-12T03:58:36.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: sync status for source bucket shard: 0 2024-04-12T03:58:36.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: sync status for source bucket shard: 0 2024-04-12T03:58:36.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: sync status for source bucket shard: 0 2024-04-12T03:58:36.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: sync status for source bucket shard: 0 2024-04-12T03:58:36.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: sync status for source bucket shard: 0 2024-04-12T03:58:36.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]: sync status for source bucket shard: 0 2024-04-12T03:58:36.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: sync status for source bucket shard: 0 2024-04-12T03:58:36.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: sync status for source bucket shard: 0 2024-04-12T03:58:36.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: sync status for source bucket shard: 0 2024-04-12T03:58:36.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.766+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.767+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-04-12T03:58:36.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.767+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16/obj00000000000000000017 2024-04-12T03:58:36.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.767+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: entry->timestamp=2024-04-12T03:58:34.283006+0000 2024-04-12T03:58:36.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.767+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.83.6 op=0 op_state=1 2024-04-12T03:58:36.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.768+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000017]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000017[0] 2024-04-12T03:58:36.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.768+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: listing bilog for incremental sync; position=00000000002.83.6 2024-04-12T03:58:36.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.768+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:entry[obj00000000000000000000]: success 2024-04-12T03:58:36.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.768+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9:5 marker=00000000002.84.6 timestamp=2024-04-12T03:58:33.584685+0000 2024-04-12T03:58:36.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.768+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000022]: success 2024-04-12T03:58:36.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.768+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8:5 marker=00000000002.44.6 timestamp=2024-04-12T03:58:33.759748+0000 2024-04-12T03:58:36.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.768+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:entry[obj00000000000000000004]: success 2024-04-12T03:58:36.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.768+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5:5 marker=00000000002.47.6 timestamp=2024-04-12T03:58:34.139872+0000 2024-04-12T03:58:36.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.768+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: a sync operation returned error 2024-04-12T03:58:36.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.768+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: backing out with sync_status=-5 2024-04-12T03:58:36.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.768+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.768+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: sync status for source bucket shard: 0 2024-04-12T03:58:36.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: parsed entry: id=00000000001.74.6 iter->object=obj00000000000000000162 iter->instance= name=obj00000000000000000162 instance= ns= 2024-04-12T03:58:36.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17/obj00000000000000000162 2024-04-12T03:58:36.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: entry->timestamp=2024-04-12T03:58:33.823565+0000 2024-04-12T03:58:36.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:entry[obj00000000000000000162]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.74.6 op=0 op_state=1 2024-04-12T03:58:36.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2024-04-12T03:58:36.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28/obj00000000000000000050 2024-04-12T03:58:36.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: entry->timestamp=2024-04-12T03:58:33.571666+0000 2024-04-12T03:58:36.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.56.6 op=0 op_state=1 2024-04-12T03:58:36.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2024-04-12T03:58:36.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20/obj00000000000000000423 2024-04-12T03:58:36.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: entry->timestamp=2024-04-12T03:58:33.963441+0000 2024-04-12T03:58:36.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.45.6 op=0 op_state=1 2024-04-12T03:58:36.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:entry[obj00000000000000000162]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000162[0] 2024-04-12T03:58:36.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: listing bilog for incremental sync; position=00000000001.74.6 2024-04-12T03:58:36.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:entry[obj00000000000000000050]: : injecting data sync error on key=obj00000000000000000050 2024-04-12T03:58:36.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: listing bilog for incremental sync; position=00000000002.56.6 2024-04-12T03:58:36.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000423]: : injecting data sync error on key=obj00000000000000000423 2024-04-12T03:58:36.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: listing bilog for incremental sync; position=00000000001.45.6 2024-04-12T03:58:36.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:entry[obj00000000000000000050]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:entry[obj00000000000000000050]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28/obj00000000000000000050 2024-04-12T03:58:36.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000423]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.771+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000423]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20/obj00000000000000000423 2024-04-12T03:58:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.772+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: parsed entry: id=00000000001.43.6 iter->object=obj00000000000000000283 iter->instance= name=obj00000000000000000283 instance= ns= 2024-04-12T03:58:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.772+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25/obj00000000000000000283 2024-04-12T03:58:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.772+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: entry->timestamp=2024-04-12T03:58:33.635658+0000 2024-04-12T03:58:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.772+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:entry[obj00000000000000000283]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.43.6 op=0 op_state=1 2024-04-12T03:58:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.772+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:entry[obj00000000000000000283]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000283[0] 2024-04-12T03:58:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.772+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: listing bilog for incremental sync; position=00000000001.43.6 2024-04-12T03:58:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.772+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2024-04-12T03:58:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.772+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14/obj00000000000000000018 2024-04-12T03:58:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.772+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: entry->timestamp=2024-04-12T03:58:34.411049+0000 2024-04-12T03:58:36.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.772+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.73.6 op=0 op_state=1 2024-04-12T03:58:36.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.772+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000018]: : injecting data sync error on key=obj00000000000000000018 2024-04-12T03:58:36.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.772+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: listing bilog for incremental sync; position=00000000002.73.6 2024-04-12T03:58:36.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.772+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000018]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.772+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000018]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14/obj00000000000000000018 2024-04-12T03:58:36.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: sync status for source bucket shard: 0 2024-04-12T03:58:36.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]: parsed entry: id=00000000001.116.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2024-04-12T03:58:36.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29/obj00000000000000000029 2024-04-12T03:58:36.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]: entry->timestamp=2024-04-12T03:58:33.835684+0000 2024-04-12T03:58:36.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.116.6 op=0 op_state=1 2024-04-12T03:58:36.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-12T03:58:36.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21/obj00000000000000000002 2024-04-12T03:58:36.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: entry->timestamp=2024-04-12T03:58:34.075707+0000 2024-04-12T03:58:36.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.76.6 op=0 op_state=1 2024-04-12T03:58:36.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-12T03:58:36.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23/obj00000000000000000001 2024-04-12T03:58:36.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: entry->timestamp=2024-04-12T03:58:34.090613+0000 2024-04-12T03:58:36.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.40.6 op=0 op_state=1 2024-04-12T03:58:36.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2024-04-12T03:58:36.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27/obj00000000000000000061 2024-04-12T03:58:36.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: entry->timestamp=2024-04-12T03:58:33.774653+0000 2024-04-12T03:58:36.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.56.6 op=0 op_state=1 2024-04-12T03:58:36.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000017]: success 2024-04-12T03:58:36.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16:5 marker=00000000002.83.6 timestamp=2024-04-12T03:58:34.283006+0000 2024-04-12T03:58:36.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:entry[obj00000000000000000029]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000029[0] 2024-04-12T03:58:36.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]: listing bilog for incremental sync; position=00000000001.116.6 2024-04-12T03:58:36.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:entry[obj00000000000000000002]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000002[0] 2024-04-12T03:58:36.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: listing bilog for incremental sync; position=00000000002.76.6 2024-04-12T03:58:36.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:entry[obj00000000000000000001]: : injecting data sync error on key=obj00000000000000000001 2024-04-12T03:58:36.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.775+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: listing bilog for incremental sync; position=00000000002.40.6 2024-04-12T03:58:36.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.776+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:entry[obj00000000000000000061]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000061[0] 2024-04-12T03:58:36.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.776+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: listing bilog for incremental sync; position=00000000001.56.6 2024-04-12T03:58:36.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.776+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:entry[obj00000000000000000001]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.776+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:entry[obj00000000000000000001]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23/obj00000000000000000001 2024-04-12T03:58:36.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.778+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.780+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket shard: 0 2024-04-12T03:58:36.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.780+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.780+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:entry[obj00000000000000000002]: success 2024-04-12T03:58:36.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.780+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21:5 marker=00000000002.76.6 timestamp=2024-04-12T03:58:34.075707+0000 2024-04-12T03:58:36.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.785+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:entry[obj00000000000000000029]: success 2024-04-12T03:58:36.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.785+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29:5 marker=00000000001.116.6 timestamp=2024-04-12T03:58:33.835684+0000 2024-04-12T03:58:36.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.786+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:entry[obj00000000000000000061]: success 2024-04-12T03:58:36.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.786+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27:5 marker=00000000001.56.6 timestamp=2024-04-12T03:58:33.774653+0000 2024-04-12T03:58:36.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.798+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-12T03:58:36.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.798+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1/obj00000000000000000006 2024-04-12T03:58:36.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.798+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: entry->timestamp=2024-04-12T03:58:34.154161+0000 2024-04-12T03:58:36.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.798+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.67.6 op=0 op_state=1 2024-04-12T03:58:36.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.798+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000006]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000006[0] 2024-04-12T03:58:36.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.798+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: listing bilog for incremental sync; position=00000000002.67.6 2024-04-12T03:58:36.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.799+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: parsed entry: id=00000000001.57.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2024-04-12T03:58:36.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.799+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2/obj00000000000000000467 2024-04-12T03:58:36.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.799+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: entry->timestamp=2024-04-12T03:58:34.332389+0000 2024-04-12T03:58:36.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.799+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.57.6 op=0 op_state=1 2024-04-12T03:58:36.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.799+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000467]: : injecting data sync error on key=obj00000000000000000467 2024-04-12T03:58:36.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.799+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position=00000000001.57.6 2024-04-12T03:58:36.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.799+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000467]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.799+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000467]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2/obj00000000000000000467 2024-04-12T03:58:36.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.800+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000006]: success 2024-04-12T03:58:36.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.800+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1:5 marker=00000000002.67.6 timestamp=2024-04-12T03:58:34.154161+0000 2024-04-12T03:58:36.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.801+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: parsed entry: id=00000000002.28.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-04-12T03:58:36.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.801+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3/obj00000000000000000005 2024-04-12T03:58:36.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.801+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: entry->timestamp=2024-04-12T03:58:33.708524+0000 2024-04-12T03:58:36.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.801+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.28.6 op=0 op_state=1 2024-04-12T03:58:36.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.801+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:entry[obj00000000000000000005]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000005[0] 2024-04-12T03:58:36.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.801+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: listing bilog for incremental sync; position=00000000002.28.6 2024-04-12T03:58:36.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.803+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:entry[obj00000000000000000005]: success 2024-04-12T03:58:36.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.803+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3:5 marker=00000000002.28.6 timestamp=2024-04-12T03:58:33.708524+0000 2024-04-12T03:58:36.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.803+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: parsed entry: id=00000000001.72.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2024-04-12T03:58:36.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.803+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12/obj00000000000000000271 2024-04-12T03:58:36.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.803+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: entry->timestamp=2024-04-12T03:58:33.897372+0000 2024-04-12T03:58:36.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.803+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.72.6 op=0 op_state=1 2024-04-12T03:58:36.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.803+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000271]: : injecting data sync error on key=obj00000000000000000271 2024-04-12T03:58:36.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.803+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: listing bilog for incremental sync; position=00000000001.72.6 2024-04-12T03:58:36.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.803+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000271]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.803+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000271]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12/obj00000000000000000271 2024-04-12T03:58:36.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.804+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-12T03:58:36.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.804+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11/obj00000000000000000009 2024-04-12T03:58:36.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.804+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: entry->timestamp=2024-04-12T03:58:34.396469+0000 2024-04-12T03:58:36.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.804+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.76.6 op=0 op_state=1 2024-04-12T03:58:36.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.804+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000009]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000009[0] 2024-04-12T03:58:36.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.804+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: listing bilog for incremental sync; position=00000000002.76.6 2024-04-12T03:58:36.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.806+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000009]: success 2024-04-12T03:58:36.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.806+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11:5 marker=00000000002.76.6 timestamp=2024-04-12T03:58:34.396469+0000 2024-04-12T03:58:36.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.806+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: parsed entry: id=00000000002.118.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-12T03:58:36.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.806+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13/obj00000000000000000008 2024-04-12T03:58:36.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.806+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: entry->timestamp=2024-04-12T03:58:34.204615+0000 2024-04-12T03:58:36.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.806+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.118.6 op=0 op_state=1 2024-04-12T03:58:36.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.806+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000008]: : injecting data sync error on key=obj00000000000000000008 2024-04-12T03:58:36.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.806+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position=00000000002.118.6 2024-04-12T03:58:36.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.806+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000008]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.806+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000008]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13/obj00000000000000000008 2024-04-12T03:58:36.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.807+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: parsed entry: id=00000000002.86.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-12T03:58:36.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.807+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15/obj00000000000000000007 2024-04-12T03:58:36.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.807+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: entry->timestamp=2024-04-12T03:58:34.011884+0000 2024-04-12T03:58:36.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.807+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.86.6 op=0 op_state=1 2024-04-12T03:58:36.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.807+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000007]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000007[0] 2024-04-12T03:58:36.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.807+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: listing bilog for incremental sync; position=00000000002.86.6 2024-04-12T03:58:36.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.809+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-12T03:58:36.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.809+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22/obj00000000000000000012 2024-04-12T03:58:36.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.809+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: entry->timestamp=2024-04-12T03:58:34.267930+0000 2024-04-12T03:58:36.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.809+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.74.6 op=0 op_state=1 2024-04-12T03:58:36.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.809+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:entry[obj00000000000000000012]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000012[0] 2024-04-12T03:58:36.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.809+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: listing bilog for incremental sync; position=00000000002.74.6 2024-04-12T03:58:36.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.809+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-12T03:58:36.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.809+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24/obj00000000000000000011 2024-04-12T03:58:36.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.809+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: entry->timestamp=2024-04-12T03:58:33.883626+0000 2024-04-12T03:58:36.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.809+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.34.6 op=0 op_state=1 2024-04-12T03:58:36.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.809+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:entry[obj00000000000000000011]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000011[0] 2024-04-12T03:58:36.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.809+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: listing bilog for incremental sync; position=00000000002.34.6 2024-04-12T03:58:36.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.810+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000007]: success 2024-04-12T03:58:36.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.810+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15:5 marker=00000000002.86.6 timestamp=2024-04-12T03:58:34.011884+0000 2024-04-12T03:58:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.812+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: parsed entry: id=00000000002.91.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-12T03:58:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.812+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18/obj00000000000000000016 2024-04-12T03:58:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.812+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: entry->timestamp=2024-04-12T03:58:33.947782+0000 2024-04-12T03:58:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.812+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.91.6 op=0 op_state=1 2024-04-12T03:58:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.812+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000016]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000016[0] 2024-04-12T03:58:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.812+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: listing bilog for incremental sync; position=00000000002.91.6 2024-04-12T03:58:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.813+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:entry[obj00000000000000000011]: success 2024-04-12T03:58:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.813+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24:5 marker=00000000002.34.6 timestamp=2024-04-12T03:58:33.883626+0000 2024-04-12T03:58:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.813+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:entry[obj00000000000000000012]: success 2024-04-12T03:58:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.813+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22:5 marker=00000000002.74.6 timestamp=2024-04-12T03:58:34.267930+0000 2024-04-12T03:58:36.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.814+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000023]: success 2024-04-12T03:58:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.814+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6:5 marker=00000000002.58.6 timestamp=2024-04-12T03:58:34.220083+0000 2024-04-12T03:58:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.815+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: a sync operation returned error 2024-04-12T03:58:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.815+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: a sync operation returned error 2024-04-12T03:58:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.815+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: backing out with sync_status=-5 2024-04-12T03:58:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.815+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: backing out with sync_status=-5 2024-04-12T03:58:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.815+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.815+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.815+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.815+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.815+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000016]: success 2024-04-12T03:58:36.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.815+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18:5 marker=00000000002.91.6 timestamp=2024-04-12T03:58:33.947782+0000 2024-04-12T03:58:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.816+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: parsed entry: id=00000000001.32.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2024-04-12T03:58:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.816+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30/obj00000000000000000133 2024-04-12T03:58:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.816+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: entry->timestamp=2024-04-12T03:58:33.696022+0000 2024-04-12T03:58:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.816+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.32.6 op=0 op_state=1 2024-04-12T03:58:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.816+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:entry[obj00000000000000000133]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000133[0] 2024-04-12T03:58:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.816+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: listing bilog for incremental sync; position=00000000001.32.6 2024-04-12T03:58:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.816+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: a sync operation returned error 2024-04-12T03:58:36.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.816+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: backing out with sync_status=-5 2024-04-12T03:58:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.816+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.816+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.817+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.817+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.817+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:entry[obj00000000000000000317]: success 2024-04-12T03:58:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.817+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7:5 marker=00000000001.41.6 timestamp=2024-04-12T03:58:34.346593+0000 2024-04-12T03:58:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.818+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: parsed entry: id=00000000002.148.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-12T03:58:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.818+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26/obj00000000000000000010 2024-04-12T03:58:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.818+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: entry->timestamp=2024-04-12T03:58:33.647702+0000 2024-04-12T03:58:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.818+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.148.6 op=0 op_state=1 2024-04-12T03:58:36.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.818+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:entry[obj00000000000000000010]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000010[0] 2024-04-12T03:58:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.818+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: listing bilog for incremental sync; position=00000000002.148.6 2024-04-12T03:58:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.819+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: sync status for source bucket shard: 0 2024-04-12T03:58:36.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.819+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: sync status for source bucket shard: 0 2024-04-12T03:58:36.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.819+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.819+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.819+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.820+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: sync status for source bucket shard: 0 2024-04-12T03:58:36.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.820+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.820+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: a sync operation returned error 2024-04-12T03:58:36.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.820+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: backing out with sync_status=-5 2024-04-12T03:58:36.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.820+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.820+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.821+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:entry[obj00000000000000000010]: success 2024-04-12T03:58:36.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.821+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26:5 marker=00000000002.148.6 timestamp=2024-04-12T03:58:33.647702+0000 2024-04-12T03:58:36.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.822+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: parsed entry: id=00000000001.93.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2024-04-12T03:58:36.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.822+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4/obj00000000000000000056 2024-04-12T03:58:36.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.822+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: entry->timestamp=2024-04-12T03:58:34.027311+0000 2024-04-12T03:58:36.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.822+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.93.6 op=0 op_state=1 2024-04-12T03:58:36.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.822+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000056]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000056[0] 2024-04-12T03:58:36.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.822+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position=00000000001.93.6 2024-04-12T03:58:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.825+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:entry[obj00000000000000000162]: success 2024-04-12T03:58:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.825+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17:5 marker=00000000001.74.6 timestamp=2024-04-12T03:58:33.823565+0000 2024-04-12T03:58:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.825+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:entry[obj00000000000000000283]: success 2024-04-12T03:58:36.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.825+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25:5 marker=00000000001.43.6 timestamp=2024-04-12T03:58:33.635658+0000 2024-04-12T03:58:36.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.826+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.827+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: sync status for source bucket shard: 0 2024-04-12T03:58:36.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.827+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.842+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: a sync operation returned error 2024-04-12T03:58:36.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.842+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: backing out with sync_status=-5 2024-04-12T03:58:36.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.842+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.842+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.842+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.842+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket shard: 0 2024-04-12T03:58:36.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.842+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.845+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: a sync operation returned error 2024-04-12T03:58:36.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.845+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: backing out with sync_status=-5 2024-04-12T03:58:36.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.845+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.845+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.845+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.846+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: sync status for source bucket shard: 0 2024-04-12T03:58:36.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.846+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.849+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: a sync operation returned error 2024-04-12T03:58:36.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.849+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: backing out with sync_status=-5 2024-04-12T03:58:36.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.849+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.849+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.849+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.849+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket shard: 0 2024-04-12T03:58:36.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.849+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.862+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: parsed entry: id=00000000002.56.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2024-04-12T03:58:36.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.862+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28/obj00000000000000000050 2024-04-12T03:58:36.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.862+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: entry->timestamp=2024-04-12T03:58:33.571666+0000 2024-04-12T03:58:36.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.862+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.56.6 op=0 op_state=1 2024-04-12T03:58:36.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.862+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:entry[obj00000000000000000050]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000050[0] 2024-04-12T03:58:36.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.862+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: listing bilog for incremental sync; position=00000000002.56.6 2024-04-12T03:58:36.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.863+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: parsed entry: id=00000000001.45.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2024-04-12T03:58:36.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.863+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20/obj00000000000000000423 2024-04-12T03:58:36.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.863+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: entry->timestamp=2024-04-12T03:58:33.963441+0000 2024-04-12T03:58:36.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.863+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.45.6 op=0 op_state=1 2024-04-12T03:58:36.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.863+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000423]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000423[0] 2024-04-12T03:58:36.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.863+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: listing bilog for incremental sync; position=00000000001.45.6 2024-04-12T03:58:36.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.863+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2024-04-12T03:58:36.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.863+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14/obj00000000000000000018 2024-04-12T03:58:36.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.863+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: entry->timestamp=2024-04-12T03:58:34.411049+0000 2024-04-12T03:58:36.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.863+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.73.6 op=0 op_state=1 2024-04-12T03:58:36.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.863+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000018]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000018[0] 2024-04-12T03:58:36.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.863+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: listing bilog for incremental sync; position=00000000002.73.6 2024-04-12T03:58:36.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.865+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:entry[obj00000000000000000050]: success 2024-04-12T03:58:36.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.865+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28:5 marker=00000000002.56.6 timestamp=2024-04-12T03:58:33.571666+0000 2024-04-12T03:58:36.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.867+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:entry[obj00000000000000000133]: success 2024-04-12T03:58:36.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.867+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30:5 marker=00000000001.32.6 timestamp=2024-04-12T03:58:33.696022+0000 2024-04-12T03:58:36.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.870+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-12T03:58:36.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.870+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23/obj00000000000000000001 2024-04-12T03:58:36.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.870+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: entry->timestamp=2024-04-12T03:58:34.090613+0000 2024-04-12T03:58:36.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.870+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.40.6 op=0 op_state=1 2024-04-12T03:58:36.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.870+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:entry[obj00000000000000000001]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000001[0] 2024-04-12T03:58:36.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.870+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: listing bilog for incremental sync; position=00000000002.40.6 2024-04-12T03:58:36.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.871+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000018]: success 2024-04-12T03:58:36.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.871+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14:5 marker=00000000002.73.6 timestamp=2024-04-12T03:58:34.411049+0000 2024-04-12T03:58:36.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.872+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000056]: success 2024-04-12T03:58:36.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.872+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4:5 marker=00000000001.93.6 timestamp=2024-04-12T03:58:34.027311+0000 2024-04-12T03:58:36.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.872+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000423]: success 2024-04-12T03:58:36.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.872+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20:5 marker=00000000001.45.6 timestamp=2024-04-12T03:58:33.963441+0000 2024-04-12T03:58:36.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.873+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:entry[obj00000000000000000001]: success 2024-04-12T03:58:36.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.873+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23:5 marker=00000000002.40.6 timestamp=2024-04-12T03:58:34.090613+0000 2024-04-12T03:58:36.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.885+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: parsed entry: id=00000000001.57.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2024-04-12T03:58:36.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.885+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2/obj00000000000000000467 2024-04-12T03:58:36.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.885+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: entry->timestamp=2024-04-12T03:58:34.332389+0000 2024-04-12T03:58:36.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.885+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.57.6 op=0 op_state=1 2024-04-12T03:58:36.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.885+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000467]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000467[0] 2024-04-12T03:58:36.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.885+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position=00000000001.57.6 2024-04-12T03:58:36.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.888+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: parsed entry: id=00000000001.72.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2024-04-12T03:58:36.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.888+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12/obj00000000000000000271 2024-04-12T03:58:36.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.888+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: entry->timestamp=2024-04-12T03:58:33.897372+0000 2024-04-12T03:58:36.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.888+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.72.6 op=0 op_state=1 2024-04-12T03:58:36.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.888+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000271]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000271[0] 2024-04-12T03:58:36.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.888+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: listing bilog for incremental sync; position=00000000001.72.6 2024-04-12T03:58:36.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.892+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: parsed entry: id=00000000002.118.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-12T03:58:36.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.892+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13/obj00000000000000000008 2024-04-12T03:58:36.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.892+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: entry->timestamp=2024-04-12T03:58:34.204615+0000 2024-04-12T03:58:36.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.892+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.118.6 op=0 op_state=1 2024-04-12T03:58:36.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.892+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000008]: : injecting data sync error on key=obj00000000000000000008 2024-04-12T03:58:36.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.892+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position=00000000002.118.6 2024-04-12T03:58:36.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.892+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000008]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.892+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000008]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13/obj00000000000000000008 2024-04-12T03:58:36.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.930+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000467]: success 2024-04-12T03:58:36.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.930+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2:5 marker=00000000001.57.6 timestamp=2024-04-12T03:58:34.332389+0000 2024-04-12T03:58:36.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.934+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: a sync operation returned error 2024-04-12T03:58:36.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.934+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: backing out with sync_status=-5 2024-04-12T03:58:36.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.934+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:36.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.934+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:36.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.935+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000271]: success 2024-04-12T03:58:36.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.935+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12:5 marker=00000000001.72.6 timestamp=2024-04-12T03:58:33.897372+0000 2024-04-12T03:58:36.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.935+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:36.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.935+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket shard: 0 2024-04-12T03:58:36.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.935+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position= 2024-04-12T03:58:36.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.977+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: parsed entry: id=00000000002.118.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-12T03:58:36.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.977+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13/obj00000000000000000008 2024-04-12T03:58:36.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.977+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: entry->timestamp=2024-04-12T03:58:34.204615+0000 2024-04-12T03:58:36.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.977+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.118.6 op=0 op_state=1 2024-04-12T03:58:36.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.977+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000008]: : injecting data sync error on key=obj00000000000000000008 2024-04-12T03:58:36.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.977+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position=00000000002.118.6 2024-04-12T03:58:36.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.977+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000008]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:36.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:36.977+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000008]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13/obj00000000000000000008 2024-04-12T03:58:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.019+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: a sync operation returned error 2024-04-12T03:58:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.019+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: backing out with sync_status=-5 2024-04-12T03:58:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.019+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.019+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:37.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.019+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.020+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket shard: 0 2024-04-12T03:58:37.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.020+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.062+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: parsed entry: id=00000000002.118.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-12T03:58:37.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.062+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13/obj00000000000000000008 2024-04-12T03:58:37.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.062+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: entry->timestamp=2024-04-12T03:58:34.204615+0000 2024-04-12T03:58:37.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.062+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.118.6 op=0 op_state=1 2024-04-12T03:58:37.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.062+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000008]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000008[0] 2024-04-12T03:58:37.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.062+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position=00000000002.118.6 2024-04-12T03:58:37.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.064+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000008]: success 2024-04-12T03:58:37.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.064+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13:5 marker=00000000002.118.6 timestamp=2024-04-12T03:58:34.204615+0000 2024-04-12T03:58:37.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.110+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.110+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.110+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.110+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.111+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.111+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.111+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: sync status for source bucket shard: 0 2024-04-12T03:58:37.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.111+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.111+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: sync status for source bucket shard: 0 2024-04-12T03:58:37.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.111+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.111+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.111+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: sync status for source bucket shard: 0 2024-04-12T03:58:37.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.111+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: sync status for source bucket shard: 0 2024-04-12T03:58:37.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.111+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: sync status for source bucket shard: 0 2024-04-12T03:58:37.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.111+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.112+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:5]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.112+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.112+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.112+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: sync status for source bucket shard: 0 2024-04-12T03:58:37.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.112+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.112+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: sync status for source bucket shard: 0 2024-04-12T03:58:37.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: sync status for source bucket shard: 0 2024-04-12T03:58:37.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.113+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.115+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: sync status for source bucket shard: 0 2024-04-12T03:58:37.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.115+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: sync status for source bucket shard: 0 2024-04-12T03:58:37.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.115+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2024-04-12T03:58:37.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1/obj00000000000000000000 2024-04-12T03:58:37.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: entry->timestamp=2024-04-12T03:58:35.648708+0000 2024-04-12T03:58:37.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.76.6 op=0 op_state=1 2024-04-12T03:58:37.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000000]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000000[0] 2024-04-12T03:58:37.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: listing bilog for incremental sync; position=00000000002.76.6 2024-04-12T03:58:37.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: parsed entry: id=00000000001.59.6 iter->object=obj00000000000000000060 iter->instance= name=obj00000000000000000060 instance= ns= 2024-04-12T03:58:37.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0/obj00000000000000000060 2024-04-12T03:58:37.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: entry->timestamp=2024-04-12T03:58:35.664604+0000 2024-04-12T03:58:37.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:entry[obj00000000000000000060]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.59.6 op=0 op_state=1 2024-04-12T03:58:37.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:entry[obj00000000000000000060]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000060[0] 2024-04-12T03:58:37.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: listing bilog for incremental sync; position=00000000001.59.6 2024-04-12T03:58:37.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2024-04-12T03:58:37.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6/obj00000000000000000005 2024-04-12T03:58:37.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: entry->timestamp=2024-04-12T03:58:35.580964+0000 2024-04-12T03:58:37.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.74.6 op=0 op_state=1 2024-04-12T03:58:37.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000005]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000005[0] 2024-04-12T03:58:37.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: listing bilog for incremental sync; position=00000000002.74.6 2024-04-12T03:58:37.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2024-04-12T03:58:37.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4/obj00000000000000000033 2024-04-12T03:58:37.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: entry->timestamp=2024-04-12T03:58:35.444868+0000 2024-04-12T03:58:37.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.43.6 op=0 op_state=1 2024-04-12T03:58:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: parsed entry: id=00000000002.119.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2024-04-12T03:58:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7/obj00000000000000000016 2024-04-12T03:58:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: entry->timestamp=2024-04-12T03:58:35.800005+0000 2024-04-12T03:58:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.119.6 op=0 op_state=1 2024-04-12T03:58:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.116+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.117+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000033]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000033[0] 2024-04-12T03:58:37.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.117+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: listing bilog for incremental sync; position=00000000002.43.6 2024-04-12T03:58:37.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.117+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:entry[obj00000000000000000016]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000016[0] 2024-04-12T03:58:37.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.117+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: listing bilog for incremental sync; position=00000000002.119.6 2024-04-12T03:58:37.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.117+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.117+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2024-04-12T03:58:37.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.117+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3/obj00000000000000000001 2024-04-12T03:58:37.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.117+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: entry->timestamp=2024-04-12T03:58:35.260288+0000 2024-04-12T03:58:37.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.117+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.41.6 op=0 op_state=1 2024-04-12T03:58:37.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.117+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.117+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:entry[obj00000000000000000001]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000001[0] 2024-04-12T03:58:37.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.117+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: listing bilog for incremental sync; position=00000000002.41.6 2024-04-12T03:58:37.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.117+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.118+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.118+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: sync status for source bucket shard: 0 2024-04-12T03:58:37.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.118+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.118+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.118+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: sync status for source bucket shard: 0 2024-04-12T03:58:37.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.118+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: sync status for source bucket shard: 0 2024-04-12T03:58:37.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.118+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: sync status for source bucket shard: 0 2024-04-12T03:58:37.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.118+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.118+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.118+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.119+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: sync status for source bucket shard: 0 2024-04-12T03:58:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.119+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: sync status for source bucket shard: 0 2024-04-12T03:58:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.119+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.119+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.120+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: sync status for source bucket shard: 0 2024-04-12T03:58:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.120+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: sync status for source bucket shard: 0 2024-04-12T03:58:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.120+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: sync status for source bucket shard: 0 2024-04-12T03:58:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.120+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.120+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.120+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.120+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.120+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.120+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.120+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.120+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: sync status for source bucket shard: 0 2024-04-12T03:58:37.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: sync status for source bucket shard: 0 2024-04-12T03:58:37.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: sync status for source bucket shard: 0 2024-04-12T03:58:37.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: sync status for source bucket shard: 0 2024-04-12T03:58:37.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: sync status for source bucket shard: 0 2024-04-12T03:58:37.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: sync status for source bucket shard: 0 2024-04-12T03:58:37.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: sync status for source bucket shard: 0 2024-04-12T03:58:37.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: sync status for source bucket shard: 0 2024-04-12T03:58:37.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: sync status for source bucket shard: 0 2024-04-12T03:58:37.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: sync status for source bucket shard: 0 2024-04-12T03:58:37.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: sync status for source bucket shard: 0 2024-04-12T03:58:37.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]: sync status for source bucket shard: 0 2024-04-12T03:58:37.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:28]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.122+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.124+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]: sync status for source bucket shard: 0 2024-04-12T03:58:37.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.124+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]: sync status for source bucket shard: 0 2024-04-12T03:58:37.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.124+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.124+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.124+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:entry[obj00000000000000000016]: success 2024-04-12T03:58:37.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.124+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:7:6 marker=00000000002.119.6 timestamp=2024-04-12T03:58:35.800005+0000 2024-04-12T03:58:37.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.124+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:entry[obj00000000000000000000]: success 2024-04-12T03:58:37.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.124+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:1:6 marker=00000000002.76.6 timestamp=2024-04-12T03:58:35.648708+0000 2024-04-12T03:58:37.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.124+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: parsed entry: id=00000000001.77.6 iter->object=obj00000000000000000062 iter->instance= name=obj00000000000000000062 instance= ns= 2024-04-12T03:58:37.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.124+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12/obj00000000000000000062 2024-04-12T03:58:37.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.124+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: entry->timestamp=2024-04-12T03:58:34.796344+0000 2024-04-12T03:58:37.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.124+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000062]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.77.6 op=0 op_state=1 2024-04-12T03:58:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.125+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000062]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000062[0] 2024-04-12T03:58:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.125+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: listing bilog for incremental sync; position=00000000001.77.6 2024-04-12T03:58:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2024-04-12T03:58:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16/obj00000000000000000012 2024-04-12T03:58:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: entry->timestamp=2024-04-12T03:58:35.716905+0000 2024-04-12T03:58:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.69.6 op=0 op_state=1 2024-04-12T03:58:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2024-04-12T03:58:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21/obj00000000000000000017 2024-04-12T03:58:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: entry->timestamp=2024-04-12T03:58:35.460327+0000 2024-04-12T03:58:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.67.6 op=0 op_state=1 2024-04-12T03:58:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000012]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000012[0] 2024-04-12T03:58:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: listing bilog for incremental sync; position=00000000002.69.6 2024-04-12T03:58:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: parsed entry: id=00000000002.153.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2024-04-12T03:58:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15/obj00000000000000000024 2024-04-12T03:58:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: entry->timestamp=2024-04-12T03:58:34.976949+0000 2024-04-12T03:58:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.153.6 op=0 op_state=1 2024-04-12T03:58:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: parsed entry: id=00000000002.125.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2024-04-12T03:58:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20/obj00000000000000000006 2024-04-12T03:58:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: entry->timestamp=2024-04-12T03:58:35.328518+0000 2024-04-12T03:58:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.125.6 op=0 op_state=1 2024-04-12T03:58:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:entry[obj00000000000000000017]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000017[0] 2024-04-12T03:58:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: listing bilog for incremental sync; position=00000000002.67.6 2024-04-12T03:58:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000024]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000024[0] 2024-04-12T03:58:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: listing bilog for incremental sync; position=00000000002.153.6 2024-04-12T03:58:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000006]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000006[0] 2024-04-12T03:58:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: listing bilog for incremental sync; position=00000000002.125.6 2024-04-12T03:58:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2024-04-12T03:58:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23/obj00000000000000000039 2024-04-12T03:58:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: entry->timestamp=2024-04-12T03:58:35.380766+0000 2024-04-12T03:58:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.37.6 op=0 op_state=1 2024-04-12T03:58:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:entry[obj00000000000000000039]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000039[0] 2024-04-12T03:58:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: listing bilog for incremental sync; position=00000000002.37.6 2024-04-12T03:58:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]: sync status for source bucket shard: 0 2024-04-12T03:58:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:entry[obj00000000000000000001]: success 2024-04-12T03:58:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.126+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:3:6 marker=00000000002.41.6 timestamp=2024-04-12T03:58:35.260288+0000 2024-04-12T03:58:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.127+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:33]: sync status for source bucket shard: 0 2024-04-12T03:58:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.127+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:33]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:33]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.128+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]: sync status for source bucket shard: 0 2024-04-12T03:58:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.128+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.128+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]: sync status for source bucket shard: 0 2024-04-12T03:58:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.128+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.130+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: parsed entry: id=00000000002.35.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2024-04-12T03:58:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.130+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11/obj00000000000000000030 2024-04-12T03:58:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.130+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: entry->timestamp=2024-04-12T03:58:34.783102+0000 2024-04-12T03:58:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.130+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.35.6 op=0 op_state=1 2024-04-12T03:58:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.130+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000030]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000030[0] 2024-04-12T03:58:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.130+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: listing bilog for incremental sync; position=00000000002.35.6 2024-04-12T03:58:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.131+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:entry[obj00000000000000000033]: success 2024-04-12T03:58:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.131+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:4:6 marker=00000000002.43.6 timestamp=2024-04-12T03:58:35.444868+0000 2024-04-12T03:58:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.133+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2024-04-12T03:58:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.133+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25/obj00000000000000000010 2024-04-12T03:58:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.133+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: entry->timestamp=2024-04-12T03:58:35.862645+0000 2024-04-12T03:58:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.133+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.44.6 op=0 op_state=1 2024-04-12T03:58:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.133+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:entry[obj00000000000000000010]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000010[0] 2024-04-12T03:58:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.133+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: listing bilog for incremental sync; position=00000000002.44.6 2024-04-12T03:58:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.133+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: parsed entry: id=00000000002.92.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2024-04-12T03:58:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.133+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13/obj00000000000000000009 2024-04-12T03:58:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.133+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: entry->timestamp=2024-04-12T03:58:35.108545+0000 2024-04-12T03:58:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.133+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.92.6 op=0 op_state=1 2024-04-12T03:58:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.133+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000009]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000009[0] 2024-04-12T03:58:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.133+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: listing bilog for incremental sync; position=00000000002.92.6 2024-04-12T03:58:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.134+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2024-04-12T03:58:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.134+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24/obj00000000000000000027 2024-04-12T03:58:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.134+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: entry->timestamp=2024-04-12T03:58:34.847512+0000 2024-04-12T03:58:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.134+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.41.6 op=0 op_state=1 2024-04-12T03:58:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.134+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:entry[obj00000000000000000027]: : injecting data sync error on key=obj00000000000000000027 2024-04-12T03:58:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.134+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: listing bilog for incremental sync; position=00000000002.41.6 2024-04-12T03:58:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.135+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2024-04-12T03:58:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.135+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10/obj00000000000000000026 2024-04-12T03:58:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.135+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: entry->timestamp=2024-04-12T03:58:35.527446+0000 2024-04-12T03:58:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.135+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.57.6 op=0 op_state=1 2024-04-12T03:58:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.135+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:entry[obj00000000000000000027]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.135+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:entry[obj00000000000000000027]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24/obj00000000000000000027 2024-04-12T03:58:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.135+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:entry[obj00000000000000000026]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000026[0] 2024-04-12T03:58:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.135+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: listing bilog for incremental sync; position=00000000002.57.6 2024-04-12T03:58:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.135+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2024-04-12T03:58:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.135+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27/obj00000000000000000053 2024-04-12T03:58:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.135+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: entry->timestamp=2024-04-12T03:58:35.849488+0000 2024-04-12T03:58:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.136+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.76.6 op=0 op_state=1 2024-04-12T03:58:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.136+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:entry[obj00000000000000000053]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000053[0] 2024-04-12T03:58:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.136+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: listing bilog for incremental sync; position=00000000002.76.6 2024-04-12T03:58:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.137+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:entry[obj00000000000000000017]: success 2024-04-12T03:58:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.137+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:21:6 marker=00000000002.67.6 timestamp=2024-04-12T03:58:35.460327+0000 2024-04-12T03:58:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.137+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:entry[obj00000000000000000005]: success 2024-04-12T03:58:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.137+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:6:6 marker=00000000002.74.6 timestamp=2024-04-12T03:58:35.580964+0000 2024-04-12T03:58:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.137+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: parsed entry: id=00000000002.78.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2024-04-12T03:58:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.137+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22/obj00000000000000000007 2024-04-12T03:58:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.137+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: entry->timestamp=2024-04-12T03:58:35.732745+0000 2024-04-12T03:58:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.137+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.78.6 op=0 op_state=1 2024-04-12T03:58:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.137+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2024-04-12T03:58:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.137+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17/obj00000000000000000002 2024-04-12T03:58:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.137+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: entry->timestamp=2024-04-12T03:58:35.312641+0000 2024-04-12T03:58:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.137+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.65.6 op=0 op_state=1 2024-04-12T03:58:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.138+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:entry[obj00000000000000000007]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000007[0] 2024-04-12T03:58:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.138+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: listing bilog for incremental sync; position=00000000002.78.6 2024-04-12T03:58:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.138+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:entry[obj00000000000000000002]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000002[0] 2024-04-12T03:58:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.138+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: listing bilog for incremental sync; position=00000000002.65.6 2024-04-12T03:58:37.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.138+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:entry[obj00000000000000000060]: success 2024-04-12T03:58:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.138+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:0:6 marker=00000000001.59.6 timestamp=2024-04-12T03:58:35.664604+0000 2024-04-12T03:58:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.139+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:entry[obj00000000000000000039]: success 2024-04-12T03:58:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.139+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:23:6 marker=00000000002.37.6 timestamp=2024-04-12T03:58:35.380766+0000 2024-04-12T03:58:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.140+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:entry[obj00000000000000000006]: success 2024-04-12T03:58:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.140+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:20:6 marker=00000000002.125.6 timestamp=2024-04-12T03:58:35.328518+0000 2024-04-12T03:58:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.141+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: parsed entry: id=00000000002.67.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2024-04-12T03:58:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.141+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26/obj00000000000000000021 2024-04-12T03:58:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.141+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: entry->timestamp=2024-04-12T03:58:35.784569+0000 2024-04-12T03:58:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.141+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.67.6 op=0 op_state=1 2024-04-12T03:58:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.141+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:entry[obj00000000000000000021]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000021[0] 2024-04-12T03:58:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.141+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: listing bilog for incremental sync; position=00000000002.67.6 2024-04-12T03:58:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.141+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:entry[obj00000000000000000030]: success 2024-04-12T03:58:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.141+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:11:6 marker=00000000002.35.6 timestamp=2024-04-12T03:58:34.783102+0000 2024-04-12T03:58:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.142+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: parsed entry: id=00000000002.42.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2024-04-12T03:58:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.142+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14/obj00000000000000000040 2024-04-12T03:58:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.142+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: entry->timestamp=2024-04-12T03:58:35.059113+0000 2024-04-12T03:58:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.142+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.42.6 op=0 op_state=1 2024-04-12T03:58:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.142+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000040]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000040[0] 2024-04-12T03:58:37.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.142+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: listing bilog for incremental sync; position=00000000002.42.6 2024-04-12T03:58:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.142+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:entry[obj00000000000000000012]: success 2024-04-12T03:58:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.142+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:16:6 marker=00000000002.69.6 timestamp=2024-04-12T03:58:35.716905+0000 2024-04-12T03:58:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]: parsed entry: id=00000000001.57.6 iter->object=obj00000000000000000057 iter->instance= name=obj00000000000000000057 instance= ns= 2024-04-12T03:58:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34/obj00000000000000000057 2024-04-12T03:58:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]: entry->timestamp=2024-04-12T03:58:35.395720+0000 2024-04-12T03:58:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]:entry[obj00000000000000000057]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.57.6 op=0 op_state=1 2024-04-12T03:58:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]:entry[obj00000000000000000057]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000057[0] 2024-04-12T03:58:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]: listing bilog for incremental sync; position=00000000001.57.6 2024-04-12T03:58:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: parsed entry: id=00000000001.81.6 iter->object=obj00000000000000000036 iter->instance= name=obj00000000000000000036 instance= ns= 2024-04-12T03:58:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30/obj00000000000000000036 2024-04-12T03:58:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: entry->timestamp=2024-04-12T03:58:35.195404+0000 2024-04-12T03:58:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:entry[obj00000000000000000036]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.81.6 op=0 op_state=1 2024-04-12T03:58:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: parsed entry: id=00000000002.72.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2024-04-12T03:58:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19/obj00000000000000000045 2024-04-12T03:58:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: entry->timestamp=2024-04-12T03:58:35.244407+0000 2024-04-12T03:58:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.72.6 op=0 op_state=1 2024-04-12T03:58:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:entry[obj00000000000000000036]: : injecting data sync error on key=obj00000000000000000036 2024-04-12T03:58:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: listing bilog for incremental sync; position=00000000001.81.6 2024-04-12T03:58:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:entry[obj00000000000000000024]: success 2024-04-12T03:58:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:15:6 marker=00000000002.153.6 timestamp=2024-04-12T03:58:34.976949+0000 2024-04-12T03:58:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:entry[obj00000000000000000045]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000045[0] 2024-04-12T03:58:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: listing bilog for incremental sync; position=00000000002.72.6 2024-04-12T03:58:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:entry[obj00000000000000000036]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T03:58:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.143+0000 7fe4194f8780 0 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:entry[obj00000000000000000036]: ERROR: failed to sync object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30/obj00000000000000000036 2024-04-12T03:58:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.150+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:entry[obj00000000000000000007]: success 2024-04-12T03:58:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.150+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:22:6 marker=00000000002.78.6 timestamp=2024-04-12T03:58:35.732745+0000 2024-04-12T03:58:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.150+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:entry[obj00000000000000000021]: success 2024-04-12T03:58:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.150+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:26:6 marker=00000000002.67.6 timestamp=2024-04-12T03:58:35.784569+0000 2024-04-12T03:58:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.150+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:entry[obj00000000000000000026]: success 2024-04-12T03:58:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.150+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:10:6 marker=00000000002.57.6 timestamp=2024-04-12T03:58:35.527446+0000 2024-04-12T03:58:37.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.150+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:entry[obj00000000000000000009]: success 2024-04-12T03:58:37.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.150+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:13:6 marker=00000000002.92.6 timestamp=2024-04-12T03:58:35.108545+0000 2024-04-12T03:58:37.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.151+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:entry[obj00000000000000000002]: success 2024-04-12T03:58:37.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.151+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:17:6 marker=00000000002.65.6 timestamp=2024-04-12T03:58:35.312641+0000 2024-04-12T03:58:37.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.151+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:entry[obj00000000000000000010]: success 2024-04-12T03:58:37.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.151+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:25:6 marker=00000000002.44.6 timestamp=2024-04-12T03:58:35.862645+0000 2024-04-12T03:58:37.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.151+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:entry[obj00000000000000000045]: success 2024-04-12T03:58:37.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.151+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:19:6 marker=00000000002.72.6 timestamp=2024-04-12T03:58:35.244407+0000 2024-04-12T03:58:37.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.152+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:entry[obj00000000000000000040]: success 2024-04-12T03:58:37.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.152+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:14:6 marker=00000000002.42.6 timestamp=2024-04-12T03:58:35.059113+0000 2024-04-12T03:58:37.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.155+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2024-04-12T03:58:37.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.155+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2/obj00000000000000000011 2024-04-12T03:58:37.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.155+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: entry->timestamp=2024-04-12T03:58:35.596612+0000 2024-04-12T03:58:37.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.155+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.65.6 op=0 op_state=1 2024-04-12T03:58:37.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.155+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000011[0] 2024-04-12T03:58:37.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.155+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: listing bilog for incremental sync; position=00000000002.65.6 2024-04-12T03:58:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.158+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:entry[obj00000000000000000011]: success 2024-04-12T03:58:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.158+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:2:6 marker=00000000002.65.6 timestamp=2024-04-12T03:58:35.596612+0000 2024-04-12T03:58:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.158+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:entry[obj00000000000000000053]: success 2024-04-12T03:58:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.158+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:27:6 marker=00000000002.76.6 timestamp=2024-04-12T03:58:35.849488+0000 2024-04-12T03:58:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.161+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: parsed entry: id=00000000002.65.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2024-04-12T03:58:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.161+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9/obj00000000000000000015 2024-04-12T03:58:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.161+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: entry->timestamp=2024-04-12T03:58:34.926588+0000 2024-04-12T03:58:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.161+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.65.6 op=0 op_state=1 2024-04-12T03:58:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.161+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:entry[obj00000000000000000015]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000015[0] 2024-04-12T03:58:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.161+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: listing bilog for incremental sync; position=00000000002.65.6 2024-04-12T03:58:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.164+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: parsed entry: id=00000000001.83.6 iter->object=obj00000000000000000048 iter->instance= name=obj00000000000000000048 instance= ns= 2024-04-12T03:58:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.164+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8/obj00000000000000000048 2024-04-12T03:58:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.164+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: entry->timestamp=2024-04-12T03:58:35.176469+0000 2024-04-12T03:58:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.164+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000048]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.83.6 op=0 op_state=1 2024-04-12T03:58:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.164+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000048]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000048[0] 2024-04-12T03:58:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.164+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: listing bilog for incremental sync; position=00000000001.83.6 2024-04-12T03:58:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.165+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:entry[obj00000000000000000015]: success 2024-04-12T03:58:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.165+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:9:6 marker=00000000002.65.6 timestamp=2024-04-12T03:58:34.926588+0000 2024-04-12T03:58:37.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.175+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2024-04-12T03:58:37.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.175+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32/obj00000000000000000014 2024-04-12T03:58:37.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.175+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]: entry->timestamp=2024-04-12T03:58:35.512663+0000 2024-04-12T03:58:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.175+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.85.6 op=0 op_state=1 2024-04-12T03:58:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.175+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]:entry[obj00000000000000000014]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000014[0] 2024-04-12T03:58:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.175+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]: listing bilog for incremental sync; position=00000000002.85.6 2024-04-12T03:58:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.175+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2024-04-12T03:58:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.175+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29/obj00000000000000000004 2024-04-12T03:58:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.175+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]: entry->timestamp=2024-04-12T03:58:34.861572+0000 2024-04-12T03:58:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.175+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.55.6 op=0 op_state=1 2024-04-12T03:58:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.175+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:entry[obj00000000000000000004]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000004[0] 2024-04-12T03:58:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.175+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]: listing bilog for incremental sync; position=00000000002.55.6 2024-04-12T03:58:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.177+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:entry[obj00000000000000000004]: success 2024-04-12T03:58:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.177+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:29:6 marker=00000000002.55.6 timestamp=2024-04-12T03:58:34.861572+0000 2024-04-12T03:58:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.177+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]:entry[obj00000000000000000014]: success 2024-04-12T03:58:37.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.177+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:32:6 marker=00000000002.85.6 timestamp=2024-04-12T03:58:35.512663+0000 2024-04-12T03:58:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.179+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]: parsed entry: id=00000000002.103.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2024-04-12T03:58:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.179+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36/obj00000000000000000008 2024-04-12T03:58:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.179+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]: entry->timestamp=2024-04-12T03:58:35.044439+0000 2024-04-12T03:58:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.179+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.103.6 op=0 op_state=1 2024-04-12T03:58:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.179+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]:entry[obj00000000000000000008]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000008[0] 2024-04-12T03:58:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.179+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]: listing bilog for incremental sync; position=00000000002.103.6 2024-04-12T03:58:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.180+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]: parsed entry: id=00000000002.101.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2024-04-12T03:58:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.180+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31/obj00000000000000000003 2024-04-12T03:58:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.180+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]: entry->timestamp=2024-04-12T03:58:34.912390+0000 2024-04-12T03:58:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.180+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.101.6 op=0 op_state=1 2024-04-12T03:58:37.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.180+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]:entry[obj00000000000000000003]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000003[0] 2024-04-12T03:58:37.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.180+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]: listing bilog for incremental sync; position=00000000002.101.6 2024-04-12T03:58:37.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.181+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]:entry[obj00000000000000000008]: success 2024-04-12T03:58:37.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.181+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:36:6 marker=00000000002.103.6 timestamp=2024-04-12T03:58:35.044439+0000 2024-04-12T03:58:37.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.182+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]:entry[obj00000000000000000003]: success 2024-04-12T03:58:37.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.182+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:31:6 marker=00000000002.101.6 timestamp=2024-04-12T03:58:34.912390+0000 2024-04-12T03:58:37.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.183+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2024-04-12T03:58:37.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.183+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35/obj00000000000000000018 2024-04-12T03:58:37.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.183+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]: entry->timestamp=2024-04-12T03:58:35.124192+0000 2024-04-12T03:58:37.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.183+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.63.6 op=0 op_state=1 2024-04-12T03:58:37.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.183+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]:entry[obj00000000000000000018]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000018[0] 2024-04-12T03:58:37.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.183+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]: listing bilog for incremental sync; position=00000000002.63.6 2024-04-12T03:58:37.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.184+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2024-04-12T03:58:37.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.184+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18/obj00000000000000000013 2024-04-12T03:58:37.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.184+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: entry->timestamp=2024-04-12T03:58:34.991846+0000 2024-04-12T03:58:37.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.184+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.49.6 op=0 op_state=1 2024-04-12T03:58:37.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.184+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000013]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000013[0] 2024-04-12T03:58:37.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.184+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: listing bilog for incremental sync; position=00000000002.49.6 2024-04-12T03:58:37.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.184+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:entry[obj00000000000000000062]: success 2024-04-12T03:58:37.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.184+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:12:6 marker=00000000001.77.6 timestamp=2024-04-12T03:58:34.796344+0000 2024-04-12T03:58:37.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.185+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]:entry[obj00000000000000000018]: success 2024-04-12T03:58:37.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.185+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:35:6 marker=00000000002.63.6 timestamp=2024-04-12T03:58:35.124192+0000 2024-04-12T03:58:37.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.187+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:entry[obj00000000000000000013]: success 2024-04-12T03:58:37.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.187+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:18:6 marker=00000000002.49.6 timestamp=2024-04-12T03:58:34.991846+0000 2024-04-12T03:58:37.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.188+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: a sync operation returned error 2024-04-12T03:58:37.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.188+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: backing out with sync_status=-5 2024-04-12T03:58:37.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.188+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:37.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.188+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:37.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.188+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.188+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: sync status for source bucket shard: 0 2024-04-12T03:58:37.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.188+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.191+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: a sync operation returned error 2024-04-12T03:58:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.191+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: backing out with sync_status=-5 2024-04-12T03:58:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.191+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: incremental sync on bucket failed, retcode=-5 2024-04-12T03:58:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.191+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: ERROR: incremental sync failed. error: -5 2024-04-12T03:58:37.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.191+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T03:58:37.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.192+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: sync status for source bucket shard: 0 2024-04-12T03:58:37.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.192+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: listing bilog for incremental sync; position= 2024-04-12T03:58:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.195+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]:entry[obj00000000000000000057]: success 2024-04-12T03:58:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.195+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:34:6 marker=00000000001.57.6 timestamp=2024-04-12T03:58:35.395720+0000 2024-04-12T03:58:37.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.212+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:entry[obj00000000000000000048]: success 2024-04-12T03:58:37.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.212+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:8:6 marker=00000000001.83.6 timestamp=2024-04-12T03:58:35.176469+0000 2024-04-12T03:58:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.231+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2024-04-12T03:58:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.231+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24/obj00000000000000000027 2024-04-12T03:58:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.231+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: entry->timestamp=2024-04-12T03:58:34.847512+0000 2024-04-12T03:58:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.231+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000002.41.6 op=0 op_state=1 2024-04-12T03:58:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.231+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:entry[obj00000000000000000027]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000027[0] 2024-04-12T03:58:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.231+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: listing bilog for incremental sync; position=00000000002.41.6 2024-04-12T03:58:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.233+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:entry[obj00000000000000000027]: success 2024-04-12T03:58:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.233+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:24:6 marker=00000000002.41.6 timestamp=2024-04-12T03:58:34.847512+0000 2024-04-12T03:58:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.234+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: parsed entry: id=00000000001.81.6 iter->object=obj00000000000000000036 iter->instance= name=obj00000000000000000036 instance= ns= 2024-04-12T03:58:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.234+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: syncing object: dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30/obj00000000000000000036 2024-04-12T03:58:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.234+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: entry->timestamp=2024-04-12T03:58:35.195404+0000 2024-04-12T03:58:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.234+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:entry[obj00000000000000000036]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=00000000001.81.6 op=0 op_state=1 2024-04-12T03:58:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.234+0000 7fe4194f8780 5 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:entry[obj00000000000000000036]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-60[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23])/obj00000000000000000036[0] 2024-04-12T03:58:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.234+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: listing bilog for incremental sync; position=00000000001.81.6 2024-04-12T03:58:37.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.280+0000 7fe4194f8780 10 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:entry[obj00000000000000000036]: success 2024-04-12T03:58:37.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:37.280+0000 7fe4194f8780 20 RGW-SYNC:bucket[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23<-dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]:inc_sync[dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30]: updating marker marker_oid=bucket.sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23:30:6 marker=00000000001.81.6 timestamp=2024-04-12T03:58:35.195404+0000 2024-04-12T03:58:37.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:37.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:37.299 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-60 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:37.405 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:37.405 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:37.446 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:58:37.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.446 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000000", 2024-04-12T03:58:37.446 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.446 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.447 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.447 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 48 2024-04-12T03:58:37.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.447 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.447 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.447 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.447 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.447 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.447 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.648708Z", 2024-04-12T03:58:37.447 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.447 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.448 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.448 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.448 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.448 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.448 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.448 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.448 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.6815342614115681181", 2024-04-12T03:58:37.448 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.448 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.448 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.449 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000001", 2024-04-12T03:58:37.449 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.449 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.449 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.449 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 25 2024-04-12T03:58:37.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.449 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.449 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.449 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.449 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.450 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.450 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.260288Z", 2024-04-12T03:58:37.450 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.450 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.450 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.450 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.450 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.450 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.450 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.450 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.451 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.17795346476437216757", 2024-04-12T03:58:37.451 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.451 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.451 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.451 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000002", 2024-04-12T03:58:37.451 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.451 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.451 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.452 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12 2024-04-12T03:58:37.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.452 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.452 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.452 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.452 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.452 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.452 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.312641Z", 2024-04-12T03:58:37.452 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.452 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.452 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.452 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.453 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.453 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.453 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.453 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.453 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.10926807679769434341", 2024-04-12T03:58:37.453 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.453 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.453 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.454 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000003", 2024-04-12T03:58:37.454 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.454 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.454 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.454 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9 2024-04-12T03:58:37.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.454 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.454 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.454 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.454 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.454 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.455 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.912390Z", 2024-04-12T03:58:37.455 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.455 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.455 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.455 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.455 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.455 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.455 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.455 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.455 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.11182058967296013910", 2024-04-12T03:58:37.456 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.456 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.456 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.456 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000004", 2024-04-12T03:58:37.456 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.456 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.456 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.456 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14 2024-04-12T03:58:37.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.456 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.457 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.457 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.457 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.457 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.457 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.861572Z", 2024-04-12T03:58:37.457 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.457 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.457 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.457 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.457 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.457 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.457 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.458 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.458 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.8862831482191852526", 2024-04-12T03:58:37.458 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.458 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.458 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.458 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000005", 2024-04-12T03:58:37.458 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.458 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.459 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.459 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14 2024-04-12T03:58:37.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.459 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.459 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.459 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.459 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.459 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.459 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.580964Z", 2024-04-12T03:58:37.459 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.459 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.459 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.460 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.460 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.460 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.460 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.460 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.460 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.16352706282166626660", 2024-04-12T03:58:37.460 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.460 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.460 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.461 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000006", 2024-04-12T03:58:37.461 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.461 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.461 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.461 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 20 2024-04-12T03:58:37.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.461 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.461 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.461 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.461 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.461 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.462 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.328518Z", 2024-04-12T03:58:37.462 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.462 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.462 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.462 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.462 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.462 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.462 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.462 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.462 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.5538013053326314776", 2024-04-12T03:58:37.463 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.463 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.463 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.463 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000007", 2024-04-12T03:58:37.463 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.463 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.463 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.463 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 10 2024-04-12T03:58:37.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.464 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.464 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.464 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.464 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.464 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.464 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.732745Z", 2024-04-12T03:58:37.464 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.464 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.464 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.464 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.464 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.465 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.465 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.465 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.465 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.10643608140588138153", 2024-04-12T03:58:37.465 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.465 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.465 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.465 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000008", 2024-04-12T03:58:37.465 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.466 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.466 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.466 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T03:58:37.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.466 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.466 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.466 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.466 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.466 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.466 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.044439Z", 2024-04-12T03:58:37.466 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.467 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.467 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.467 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.467 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.467 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.467 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.467 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.467 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.13500040668059801886", 2024-04-12T03:58:37.467 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.467 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.468 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.468 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000009", 2024-04-12T03:58:37.468 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.468 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.468 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.468 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 23 2024-04-12T03:58:37.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.468 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.468 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.468 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.469 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.469 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.469 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.108545Z", 2024-04-12T03:58:37.469 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.469 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.469 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.469 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.469 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.469 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.469 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.469 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.470 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.15943005845068131882", 2024-04-12T03:58:37.470 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.470 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.470 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.470 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000010", 2024-04-12T03:58:37.470 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.470 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.470 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.471 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 30 2024-04-12T03:58:37.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.471 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.471 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.471 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.471 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.471 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.471 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.862645Z", 2024-04-12T03:58:37.471 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.471 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.471 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.472 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.472 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.472 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.472 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.472 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.472 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.13485011008178477328", 2024-04-12T03:58:37.472 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.472 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.472 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.473 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000011", 2024-04-12T03:58:37.473 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.473 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.473 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.473 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 15 2024-04-12T03:58:37.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.473 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.473 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.473 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.473 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.473 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.474 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.596612Z", 2024-04-12T03:58:37.474 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.474 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.474 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.474 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.474 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.474 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.474 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.474 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.475 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.6955873742383331335", 2024-04-12T03:58:37.475 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.475 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.475 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.475 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.475 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000012", 2024-04-12T03:58:37.475 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.475 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.475 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.475 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 28 2024-04-12T03:58:37.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.476 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.476 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.476 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.476 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.476 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.476 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.716905Z", 2024-04-12T03:58:37.476 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.476 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.476 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.476 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.477 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.477 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.477 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.477 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.477 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.4818704976209325538", 2024-04-12T03:58:37.477 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.477 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.477 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.477 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.478 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000013", 2024-04-12T03:58:37.478 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.478 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.478 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.478 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 13 2024-04-12T03:58:37.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.478 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.478 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.478 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.478 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.478 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.479 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.991846Z", 2024-04-12T03:58:37.479 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.479 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.479 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.479 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.479 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.479 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.479 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.479 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.479 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.14013593013733990194", 2024-04-12T03:58:37.480 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.480 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.480 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.480 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.480 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.480 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000014", 2024-04-12T03:58:37.480 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.480 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.480 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.480 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T03:58:37.480 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.480 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.481 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.481 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.481 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.481 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.481 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.512663Z", 2024-04-12T03:58:37.481 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.481 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.481 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.481 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.481 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.481 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.481 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.482 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.482 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.6114847448428285079", 2024-04-12T03:58:37.482 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.482 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.482 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.482 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.482 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000015", 2024-04-12T03:58:37.482 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.482 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.483 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.483 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 17 2024-04-12T03:58:37.483 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.483 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.483 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.483 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.483 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.483 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.483 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.926588Z", 2024-04-12T03:58:37.483 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.483 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.484 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.484 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.484 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.484 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.484 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.484 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.484 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.10560628128221517260", 2024-04-12T03:58:37.484 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.484 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.484 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.485 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.485 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000016", 2024-04-12T03:58:37.485 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.485 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.485 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.485 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12 2024-04-12T03:58:37.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.485 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.485 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.485 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.485 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.486 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.486 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.800005Z", 2024-04-12T03:58:37.486 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.486 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.486 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.486 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.486 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.486 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.486 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.486 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.487 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.18188135057554733408", 2024-04-12T03:58:37.487 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.487 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.487 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.487 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.487 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000017", 2024-04-12T03:58:37.487 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.487 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.487 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.487 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 21 2024-04-12T03:58:37.488 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.488 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.488 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.488 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.488 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.488 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.488 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.460327Z", 2024-04-12T03:58:37.488 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.488 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.488 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.488 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.488 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.489 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.489 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.489 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.489 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.6309274247390638952", 2024-04-12T03:58:37.489 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.489 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.489 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.489 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.489 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000018", 2024-04-12T03:58:37.490 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.490 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.490 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.490 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9 2024-04-12T03:58:37.490 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.490 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.490 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.490 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.490 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.490 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.490 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.124192Z", 2024-04-12T03:58:37.491 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.491 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.491 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.491 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.491 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.491 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.491 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.491 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.491 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.6345445168757856842", 2024-04-12T03:58:37.491 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.492 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.492 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.492 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000019", 2024-04-12T03:58:37.492 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.492 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.492 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.492 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 18 2024-04-12T03:58:37.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.492 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.492 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.493 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.493 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.493 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.493 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:32.415462Z", 2024-04-12T03:58:37.493 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.493 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.493 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.493 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.493 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.493 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.494 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.494 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.494 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.10877873482451603968", 2024-04-12T03:58:37.494 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.494 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.494 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.494 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.494 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000020", 2024-04-12T03:58:37.494 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.495 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.495 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.495 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 19 2024-04-12T03:58:37.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.495 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.495 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.495 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.495 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.495 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.495 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.059878Z", 2024-04-12T03:58:37.495 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.496 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.496 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.496 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.496 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.496 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.496 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.496 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.496 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.7788848829504528393", 2024-04-12T03:58:37.496 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.496 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.497 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.497 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000021", 2024-04-12T03:58:37.497 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.497 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.497 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.497 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 21 2024-04-12T03:58:37.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.497 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.497 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.498 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.498 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.498 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.498 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.784569Z", 2024-04-12T03:58:37.498 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.498 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.498 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.498 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.498 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.498 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.498 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.499 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.499 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.6159110684083006687", 2024-04-12T03:58:37.499 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.499 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.499 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.499 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000022", 2024-04-12T03:58:37.499 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.499 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.500 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.500 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T03:58:37.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.500 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.500 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.500 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.500 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.500 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.500 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.759748Z", 2024-04-12T03:58:37.500 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.500 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.500 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.501 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.501 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.501 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.501 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.501 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.501 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.9728149004182456905", 2024-04-12T03:58:37.501 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.501 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.501 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.502 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000023", 2024-04-12T03:58:37.502 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.502 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.502 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.502 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.502 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.502 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.502 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.502 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.502 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.503 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.220083Z", 2024-04-12T03:58:37.503 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.503 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.503 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.503 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.503 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.503 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.503 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.503 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.503 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.1064509145565866740", 2024-04-12T03:58:37.504 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.504 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.504 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.504 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.504 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000024", 2024-04-12T03:58:37.504 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.504 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.504 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.504 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9 2024-04-12T03:58:37.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.505 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.505 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.505 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.505 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.505 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.505 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.976949Z", 2024-04-12T03:58:37.505 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.505 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.505 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.505 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.505 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.505 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.506 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.506 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.506 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.6340427164579041918", 2024-04-12T03:58:37.506 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.506 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.506 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.506 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000025", 2024-04-12T03:58:37.506 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.507 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.507 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.507 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9 2024-04-12T03:58:37.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.507 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.507 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.507 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.507 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.507 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.507 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.202862Z", 2024-04-12T03:58:37.507 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.507 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.508 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.508 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.508 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.508 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.508 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.508 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.508 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.3093297283341108763", 2024-04-12T03:58:37.508 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.508 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.508 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.509 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000026", 2024-04-12T03:58:37.509 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.509 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.509 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.509 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 13 2024-04-12T03:58:37.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.509 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.509 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.509 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.510 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.510 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.510 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.527446Z", 2024-04-12T03:58:37.510 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.510 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.510 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.510 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.510 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.510 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.510 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.510 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.511 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.16411415865039593542", 2024-04-12T03:58:37.511 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.511 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.511 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.511 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000027", 2024-04-12T03:58:37.511 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.511 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.512 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.512 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 10 2024-04-12T03:58:37.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.512 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.512 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.512 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.512 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.512 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.512 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.847512Z", 2024-04-12T03:58:37.512 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.512 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.512 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.512 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.513 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.513 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.513 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.513 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.513 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.9364365985830222770", 2024-04-12T03:58:37.513 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.513 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.514 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000028", 2024-04-12T03:58:37.514 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.514 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.514 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.514 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T03:58:37.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.514 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.514 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.514 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.515 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.515 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.515 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:30.226917Z", 2024-04-12T03:58:37.515 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.515 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.515 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.515 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.515 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.515 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.516 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.516 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.516 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.3460269249671213738", 2024-04-12T03:58:37.516 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.516 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.516 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.516 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000029", 2024-04-12T03:58:37.516 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.516 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T03:58:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.835684Z", 2024-04-12T03:58:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.25945361734586738", 2024-04-12T03:58:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000030", 2024-04-12T03:58:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 18 2024-04-12T03:58:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.783102Z", 2024-04-12T03:58:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.3824612454254148800", 2024-04-12T03:58:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000031", 2024-04-12T03:58:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2 2024-04-12T03:58:37.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.522 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.522 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.522 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.522 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.522 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.522 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.123895Z", 2024-04-12T03:58:37.522 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.522 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.522 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.522 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.523 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.523 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.523 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.523 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.523 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.4864580909541835127", 2024-04-12T03:58:37.523 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.523 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.523 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.524 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000032", 2024-04-12T03:58:37.524 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.524 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.524 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.524 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.524 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.524 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.524 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.524 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.524 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.524 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.525 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:31.983268Z", 2024-04-12T03:58:37.525 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.525 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.525 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.525 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.525 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.525 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.525 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.525 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.525 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.440628526021955009", 2024-04-12T03:58:37.525 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.526 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.526 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.526 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.526 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000033", 2024-04-12T03:58:37.526 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.526 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.526 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.526 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 10 2024-04-12T03:58:37.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.527 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.527 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.527 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.527 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.527 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.527 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.444868Z", 2024-04-12T03:58:37.527 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.527 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.527 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.527 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.527 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.528 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.528 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.528 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.528 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.3910793736956256653", 2024-04-12T03:58:37.528 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.528 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.528 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.528 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000036", 2024-04-12T03:58:37.528 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.528 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.529 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.529 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.529 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.529 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.529 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.529 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.529 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.529 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.195404Z", 2024-04-12T03:58:37.529 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.529 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.530 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.530 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.530 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.530 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.530 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.530 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.530 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.15220776617574761810", 2024-04-12T03:58:37.530 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.530 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.530 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.531 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000037", 2024-04-12T03:58:37.531 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.531 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.531 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.531 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 8 2024-04-12T03:58:37.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.531 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.531 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.531 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.532 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.532 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.532 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:32.547464Z", 2024-04-12T03:58:37.532 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.532 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.532 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.532 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.532 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.532 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.532 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.532 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.533 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.18356473866392430901", 2024-04-12T03:58:37.533 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.533 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.533 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.533 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000039", 2024-04-12T03:58:37.533 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.533 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.533 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.534 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 19 2024-04-12T03:58:37.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.534 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.534 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.534 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.534 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.534 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.534 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.380766Z", 2024-04-12T03:58:37.534 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.534 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.534 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.535 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.535 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.535 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.535 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.535 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.535 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.8310002603075154607", 2024-04-12T03:58:37.535 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.535 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.535 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.536 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000040", 2024-04-12T03:58:37.536 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.536 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.536 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.536 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 17 2024-04-12T03:58:37.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.536 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.536 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.536 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.536 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.536 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.537 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.059113Z", 2024-04-12T03:58:37.537 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.537 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.537 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.537 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.537 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.537 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.537 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.537 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.537 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.13937506119189343800", 2024-04-12T03:58:37.537 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.538 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.538 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.538 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.538 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000042", 2024-04-12T03:58:37.538 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.538 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.538 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.538 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T03:58:37.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.538 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.539 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.539 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.539 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.539 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.539 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:32.942465Z", 2024-04-12T03:58:37.539 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.539 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.539 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.539 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.539 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.539 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.540 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.540 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.540 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.7442805726127995979", 2024-04-12T03:58:37.540 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.540 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.540 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.540 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000045", 2024-04-12T03:58:37.540 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.541 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.541 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.541 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 19 2024-04-12T03:58:37.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.541 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.541 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.541 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.541 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.541 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.541 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.244407Z", 2024-04-12T03:58:37.541 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.542 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.542 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.542 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.542 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.542 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.542 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.542 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.542 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.15815031637285031376", 2024-04-12T03:58:37.542 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.542 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.543 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.543 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000046", 2024-04-12T03:58:37.543 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.543 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.543 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.543 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 17 2024-04-12T03:58:37.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.543 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.543 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.544 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.544 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.544 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.544 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:31.663508Z", 2024-04-12T03:58:37.544 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.544 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.544 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.544 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.544 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.544 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.544 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.544 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.545 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.15097034624227185839", 2024-04-12T03:58:37.545 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.545 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.545 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.545 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000048", 2024-04-12T03:58:37.545 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.545 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.545 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.545 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.546 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.546 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.546 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.546 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.546 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.546 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.176469Z", 2024-04-12T03:58:37.546 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.546 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.546 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.547 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.547 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.547 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.547 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.547 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.547 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.14228187191316753293", 2024-04-12T03:58:37.547 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.547 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.547 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.548 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000050", 2024-04-12T03:58:37.548 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.548 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.548 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.548 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.548 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.548 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.548 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.548 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.548 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.548 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.571666Z", 2024-04-12T03:58:37.549 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.549 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.549 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.549 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.549 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.549 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.549 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.549 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.549 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.2165377911216256027", 2024-04-12T03:58:37.549 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.550 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.550 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.550 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000053", 2024-04-12T03:58:37.550 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.550 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.550 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.550 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.550 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.551 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.551 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.551 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.551 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.551 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.849488Z", 2024-04-12T03:58:37.551 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.551 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.551 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.551 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.551 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.551 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.552 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.552 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.552 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.5256620311503784133", 2024-04-12T03:58:37.552 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.552 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.552 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.552 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.552 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000055", 2024-04-12T03:58:37.552 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.552 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.553 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.553 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 16 2024-04-12T03:58:37.553 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.553 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.553 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.553 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.553 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.553 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.553 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:32.047512Z", 2024-04-12T03:58:37.553 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.553 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.554 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.554 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.554 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.554 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.554 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.554 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.554 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.4785516033168819707", 2024-04-12T03:58:37.554 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.554 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.554 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.555 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.555 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000056", 2024-04-12T03:58:37.555 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.555 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.555 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.555 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2 2024-04-12T03:58:37.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.555 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.555 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.555 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.555 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.556 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.556 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.027311Z", 2024-04-12T03:58:37.556 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.556 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.556 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.556 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.556 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.556 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.556 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.556 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.556 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.7011604619720276530", 2024-04-12T03:58:37.557 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.557 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.557 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.557 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000057", 2024-04-12T03:58:37.557 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.557 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.557 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.557 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1 2024-04-12T03:58:37.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.558 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.558 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.558 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.558 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.558 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.558 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.395720Z", 2024-04-12T03:58:37.558 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.558 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.558 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.558 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.558 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.559 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.559 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.559 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.559 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.5956259118435487007", 2024-04-12T03:58:37.559 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.559 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.559 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.559 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000058", 2024-04-12T03:58:37.560 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.560 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.560 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.560 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.560 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.560 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.560 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.560 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.560 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.560 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:31.481097Z", 2024-04-12T03:58:37.560 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.561 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.561 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.561 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.561 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.561 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.561 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.561 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.561 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.17343499929890710781", 2024-04-12T03:58:37.561 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.561 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.562 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.562 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000059", 2024-04-12T03:58:37.562 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.562 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.562 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.562 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T03:58:37.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.562 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.562 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.562 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.563 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.563 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.563 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:32.864361Z", 2024-04-12T03:58:37.563 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.563 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.563 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.563 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.563 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.563 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.563 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.563 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.564 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.3177879028248204619", 2024-04-12T03:58:37.564 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.564 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.564 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.564 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000060", 2024-04-12T03:58:37.564 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.564 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.564 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 11 2024-04-12T03:58:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.664604Z", 2024-04-12T03:58:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.565 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.566 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.566 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.566 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.566 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.566 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.566 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.1077081467923682110", 2024-04-12T03:58:37.566 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.566 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.566 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.567 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.567 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000061", 2024-04-12T03:58:37.567 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.567 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.567 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.567 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T03:58:37.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.567 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.567 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.567 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.567 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.567 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.568 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.774653Z", 2024-04-12T03:58:37.568 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.568 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.568 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.568 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.568 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.568 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.568 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.568 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.568 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.1839546732062029418", 2024-04-12T03:58:37.569 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.569 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.569 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.569 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000062", 2024-04-12T03:58:37.569 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.569 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.569 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.569 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T03:58:37.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.570 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.570 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.570 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.570 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.570 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.570 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.796344Z", 2024-04-12T03:58:37.570 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.570 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.570 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.570 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.570 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.571 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.571 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.571 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.571 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.234737675998134965", 2024-04-12T03:58:37.571 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.571 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.571 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.571 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000076", 2024-04-12T03:58:37.571 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.572 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.572 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.572 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 3 2024-04-12T03:58:37.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.572 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.572 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.572 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.572 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.572 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.572 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:30.238284Z", 2024-04-12T03:58:37.572 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.573 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.573 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.573 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.573 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.573 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.573 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.573 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.573 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.189777563561361721", 2024-04-12T03:58:37.573 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.573 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.574 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.574 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000133", 2024-04-12T03:58:37.574 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.574 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.574 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.574 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T03:58:37.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.574 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.574 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.575 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.575 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.575 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.575 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.696022Z", 2024-04-12T03:58:37.575 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.575 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.575 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.575 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.575 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.575 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.575 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.575 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.576 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.11008546694574191564", 2024-04-12T03:58:37.576 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.576 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.576 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.576 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000162", 2024-04-12T03:58:37.576 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.576 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.576 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.576 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 10 2024-04-12T03:58:37.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.577 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.577 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.577 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.577 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.577 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.577 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.823565Z", 2024-04-12T03:58:37.577 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.577 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.577 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.577 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.577 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.578 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.578 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.578 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.578 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.11215292056503332062", 2024-04-12T03:58:37.578 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.578 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.578 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.578 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000271", 2024-04-12T03:58:37.579 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.579 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.579 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.579 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 3 2024-04-12T03:58:37.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.579 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.579 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.579 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.579 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.579 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.579 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.897372Z", 2024-04-12T03:58:37.580 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.580 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.580 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.580 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.580 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.580 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.580 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.580 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.580 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.17168056296193863884", 2024-04-12T03:58:37.580 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.581 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.581 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.581 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000283", 2024-04-12T03:58:37.581 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.581 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.581 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.581 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T03:58:37.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.581 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.581 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.582 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.582 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.582 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.582 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.635658Z", 2024-04-12T03:58:37.582 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.582 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.582 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.582 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.582 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.582 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.582 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.582 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.583 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.552576066773352141", 2024-04-12T03:58:37.583 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.583 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.583 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.583 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000317", 2024-04-12T03:58:37.583 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.583 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.583 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.584 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T03:58:37.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.584 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.584 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.584 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.584 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.584 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.584 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.346593Z", 2024-04-12T03:58:37.584 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.584 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.584 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.584 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.585 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.585 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.585 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.585 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.585 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.14187497692771104677", 2024-04-12T03:58:37.585 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.585 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.585 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.586 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000423", 2024-04-12T03:58:37.586 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.586 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.586 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.586 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.586 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.586 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.586 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.586 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.586 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.587 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.963441Z", 2024-04-12T03:58:37.587 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.587 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.587 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.587 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.587 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.587 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.587 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.587 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.587 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.7741184526514674064", 2024-04-12T03:58:37.588 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.588 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.589 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.589 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000467", 2024-04-12T03:58:37.589 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.589 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.589 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T03:58:37.589 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 15 2024-04-12T03:58:37.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.590 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.590 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.590 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.590 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.590 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.590 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.332389Z", 2024-04-12T03:58:37.590 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.590 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.590 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.590 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.591 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.591 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.591 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.591 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.591 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.6409975721261568937", 2024-04-12T03:58:37.591 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.591 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.591 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.591 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:58:37.592 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:58:37.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:37.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-60', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:37.593 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-60 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:37.657 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:37.657 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:37.697 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:58:37.697 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.697 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000000", 2024-04-12T03:58:37.697 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.697 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.697 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.697 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 37 2024-04-12T03:58:37.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.698 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.698 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.698 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.698 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.698 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.698 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.648708Z", 2024-04-12T03:58:37.698 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.698 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.698 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.698 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.699 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.699 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.699 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.699 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.699 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_cSwHKJqr__9ygEcKWY0evqvsTdlHMe6", 2024-04-12T03:58:37.699 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.699 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.699 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.699 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.700 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000001", 2024-04-12T03:58:37.700 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.700 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.700 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.700 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T03:58:37.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.700 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.700 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.700 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.700 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.700 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.700 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.260288Z", 2024-04-12T03:58:37.701 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.701 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.701 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.701 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.701 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.701 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.701 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.701 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.701 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_-cDow-PRCnQTLT0im-VarLw5U3IDJcg", 2024-04-12T03:58:37.701 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.702 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.702 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.702 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.702 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000002", 2024-04-12T03:58:37.702 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.702 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.702 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.702 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 3 2024-04-12T03:58:37.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.702 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.702 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.703 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.703 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.703 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.703 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.312641Z", 2024-04-12T03:58:37.703 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.703 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.703 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.703 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.703 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.703 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.704 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.704 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.704 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_F89p2SfD-REq0XUOUd4C4QZ81dz7z2-", 2024-04-12T03:58:37.704 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.704 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.704 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.704 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.704 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000003", 2024-04-12T03:58:37.704 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.704 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.705 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.705 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2 2024-04-12T03:58:37.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.705 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.705 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.705 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.705 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.705 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.705 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.912390Z", 2024-04-12T03:58:37.705 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.705 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.706 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.706 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.706 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.706 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.706 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.706 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.706 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_NFJ0MOvm0Pc7sxiDeHMVfjyjMOANCrT", 2024-04-12T03:58:37.706 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.706 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.707 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.707 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.707 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000004", 2024-04-12T03:58:37.707 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.707 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.707 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.707 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T03:58:37.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.707 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.707 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.708 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.708 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.708 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.708 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.861572Z", 2024-04-12T03:58:37.708 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.708 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.708 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.708 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.708 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.708 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.708 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.708 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.709 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_ZgdtBrokI4BMImcKt1ehMVCLKbZlaBk", 2024-04-12T03:58:37.709 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.709 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.709 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.709 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000005", 2024-04-12T03:58:37.709 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.709 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.709 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.710 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9 2024-04-12T03:58:37.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.710 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.710 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.710 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.710 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.710 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.710 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.580964Z", 2024-04-12T03:58:37.710 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.710 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.711 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.711 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.711 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.711 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.711 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.711 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.711 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_12uqijY77ffzVD5d4J5gKa1o_BmHo7l", 2024-04-12T03:58:37.711 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.711 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.711 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.712 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.712 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000006", 2024-04-12T03:58:37.712 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.712 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.712 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.712 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T03:58:37.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.712 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.712 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.712 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.713 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.713 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.713 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.328518Z", 2024-04-12T03:58:37.713 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.713 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.713 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.713 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.713 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.713 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.713 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.713 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.714 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_LtbManeNxaNqYG02bUo6bVraxGp6mJG", 2024-04-12T03:58:37.714 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.714 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.714 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.714 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.714 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000007", 2024-04-12T03:58:37.714 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.714 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.714 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.715 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1 2024-04-12T03:58:37.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.715 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.715 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.715 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.715 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.715 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.715 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.732745Z", 2024-04-12T03:58:37.715 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.715 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.715 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.715 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.716 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.716 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.716 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.716 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.716 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_8PpaZwMlIrnA9mHg2RiQ57AHCkbdlph", 2024-04-12T03:58:37.716 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.716 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.716 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.717 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.717 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000008", 2024-04-12T03:58:37.717 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.717 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.717 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.717 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1 2024-04-12T03:58:37.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.717 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.717 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.717 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.717 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.717 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.718 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.044439Z", 2024-04-12T03:58:37.718 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.718 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.718 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.718 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.718 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.718 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.718 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.718 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.718 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_ZG9jlx3jFar04OSK_yfzsx7MiqYl9iO", 2024-04-12T03:58:37.719 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.719 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.719 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.719 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.719 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000009", 2024-04-12T03:58:37.719 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.719 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.719 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.719 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.720 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.720 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.720 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.720 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.720 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.720 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.108545Z", 2024-04-12T03:58:37.720 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.720 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.720 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.720 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.720 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.721 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.721 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.721 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.721 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_kxwy_3PWKW2tlxRB0Ln34FtkXejH-f7", 2024-04-12T03:58:37.721 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.721 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.721 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.721 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000010", 2024-04-12T03:58:37.722 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.722 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.722 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.722 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.722 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.722 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.722 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.722 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.722 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.722 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.862645Z", 2024-04-12T03:58:37.723 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.723 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.723 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.723 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.723 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.723 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.723 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.723 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.723 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_XIXexL_u-SjNS-T2gTmadhDCC3itiBU", 2024-04-12T03:58:37.723 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.724 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.724 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.724 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000011", 2024-04-12T03:58:37.724 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.724 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.724 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.724 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T03:58:37.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.724 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.725 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.725 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.725 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.725 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.725 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.596612Z", 2024-04-12T03:58:37.725 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.725 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.725 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.725 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.725 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.725 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.726 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.726 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.726 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_fRcXiu41r7E8X8lA4GOdO-gHFBr3I-j", 2024-04-12T03:58:37.726 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.726 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.726 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.726 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000012", 2024-04-12T03:58:37.727 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.727 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.727 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.727 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T03:58:37.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.727 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.727 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.727 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.727 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.727 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.727 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.716905Z", 2024-04-12T03:58:37.728 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.728 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.728 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.728 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.728 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.728 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.728 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.728 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.728 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.728 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_nAo9y807IkkeyptmogIIrijC4BNAJGA", 2024-04-12T03:58:37.728 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.728 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.729 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.729 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.729 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.729 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000013", 2024-04-12T03:58:37.729 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.729 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.729 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.729 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.729 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.729 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.729 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.730 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.730 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.730 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.730 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.991846Z", 2024-04-12T03:58:37.730 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.730 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.730 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.730 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.730 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.730 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.731 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.731 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.731 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_wZ0o4ClCePxlHhmd_xTn9hf7-X0707V", 2024-04-12T03:58:37.731 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.731 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.731 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.731 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.731 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000014", 2024-04-12T03:58:37.732 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.732 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.732 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.732 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1 2024-04-12T03:58:37.732 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.732 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.732 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.732 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.732 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.732 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.732 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.512663Z", 2024-04-12T03:58:37.733 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.733 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.733 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.733 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.733 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.733 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.733 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.733 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.733 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.733 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_WE9dS7b0lwRU2PlvHj5QjPYQhKdmGm0", 2024-04-12T03:58:37.733 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.734 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.734 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.734 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.734 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.734 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000015", 2024-04-12T03:58:37.734 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.734 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.734 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.734 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T03:58:37.734 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.734 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.735 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.735 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.735 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.735 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.735 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.926588Z", 2024-04-12T03:58:37.735 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.735 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.735 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.735 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.735 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.735 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.735 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.736 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.736 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_cUX7YDF_kwy3r2psz2sRJOcLqTHlFh4", 2024-04-12T03:58:37.736 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.736 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.736 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.736 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.736 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000016", 2024-04-12T03:58:37.736 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.736 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.737 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.737 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.737 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.737 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.737 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.737 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.737 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.737 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.737 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.800005Z", 2024-04-12T03:58:37.737 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.737 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.738 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.738 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.738 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.738 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.738 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.738 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.738 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_7ZLssKADHKm1JQh72fcOiY4WIYAPBAy", 2024-04-12T03:58:37.738 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.738 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.738 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.739 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.739 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000017", 2024-04-12T03:58:37.739 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.739 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.739 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.739 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12 2024-04-12T03:58:37.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.739 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.739 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.739 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.460327Z", 2024-04-12T03:58:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.740 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.741 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_S9TCqdjRJDyM1v2PbOk8-XyaNIYooMq", 2024-04-12T03:58:37.741 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.741 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.741 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.741 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.741 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000018", 2024-04-12T03:58:37.741 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.741 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.741 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.741 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.742 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.742 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.742 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.742 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.742 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.742 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.124192Z", 2024-04-12T03:58:37.742 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.742 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.742 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.742 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.743 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.743 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.743 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.743 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.743 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_MJnGlNHkyigAu3p66jCsiKNjoAKAzey", 2024-04-12T03:58:37.743 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.743 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.743 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.743 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000019", 2024-04-12T03:58:37.744 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.744 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.744 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.744 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T03:58:37.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.744 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.744 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.744 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.744 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.744 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.744 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:32.415462Z", 2024-04-12T03:58:37.745 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.745 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.745 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.745 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.745 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.745 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.745 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.745 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.745 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_mcR9vmqP6-Pm-_YVT9fw1-_asl-sWZy", 2024-04-12T03:58:37.745 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.745 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.746 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.746 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.746 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000020", 2024-04-12T03:58:37.746 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.746 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.746 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.746 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 19 2024-04-12T03:58:37.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.746 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.746 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.747 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.747 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.747 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.747 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.059878Z", 2024-04-12T03:58:37.747 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.747 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.747 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.747 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.747 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.747 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.747 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.748 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.748 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_EUCzRXJWce7fw5w4AFZFm3iGI77B-2r", 2024-04-12T03:58:37.748 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.748 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.748 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.748 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.748 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000021", 2024-04-12T03:58:37.748 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.748 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.749 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.749 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 13 2024-04-12T03:58:37.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.749 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.749 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.749 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.749 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.749 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.749 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.784569Z", 2024-04-12T03:58:37.749 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.749 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.749 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.750 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.750 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.750 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.750 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.750 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.750 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_m7LHWW8z1aHZZBEUBI-7tyy0gu1RxMD", 2024-04-12T03:58:37.750 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.750 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.750 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.751 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000022", 2024-04-12T03:58:37.751 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.751 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.751 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.751 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T03:58:37.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.751 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.751 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.751 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.751 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.752 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.752 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.759748Z", 2024-04-12T03:58:37.752 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.752 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.752 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.752 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.752 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.752 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.752 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.752 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.753 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_NgdY_Y4Tt7P1Uk0Gx2mr9jAI2G3f0Jw", 2024-04-12T03:58:37.753 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.753 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.753 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.753 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.753 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000023", 2024-04-12T03:58:37.753 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.753 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.753 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.753 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.754 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.754 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.754 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.754 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.754 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.754 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.220083Z", 2024-04-12T03:58:37.754 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.754 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.754 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.754 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.755 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.755 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.755 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.755 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.755 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_ZjNMRTlikZfZPAFu6G_BhbxQ68MaSX5", 2024-04-12T03:58:37.755 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.755 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.755 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.755 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.755 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000024", 2024-04-12T03:58:37.756 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.756 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.756 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.756 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 17 2024-04-12T03:58:37.756 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.756 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.756 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.756 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.756 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.756 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.757 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.976949Z", 2024-04-12T03:58:37.757 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.757 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.757 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.757 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.757 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.757 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.757 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.757 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.757 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_D5NnqWoHXncCFxseJrxrbyWEh-TToCe", 2024-04-12T03:58:37.757 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.758 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.758 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.758 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000025", 2024-04-12T03:58:37.758 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.758 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.758 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.758 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T03:58:37.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.758 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.759 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.759 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.759 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.759 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.759 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.202862Z", 2024-04-12T03:58:37.759 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.759 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.759 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.759 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.759 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.759 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.759 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.760 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.760 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_Hx0MvB-MKpB8QVbsRkDXPpsrMZNvykA", 2024-04-12T03:58:37.760 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.760 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.760 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.760 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000026", 2024-04-12T03:58:37.760 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.760 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.761 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.761 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T03:58:37.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.761 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.761 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.761 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.761 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.761 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.761 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.527446Z", 2024-04-12T03:58:37.761 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.761 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.761 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.762 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.762 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.762 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.762 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.762 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.762 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_-3PgKw1UAD4v71TX4Mcl4leaW2cHFIo", 2024-04-12T03:58:37.762 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.762 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.762 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.763 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000027", 2024-04-12T03:58:37.763 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.763 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.763 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.763 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.763 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.763 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.763 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.763 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.764 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.764 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.847512Z", 2024-04-12T03:58:37.764 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.764 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.764 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.764 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.764 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.764 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.764 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.764 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.764 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_Wq5-o6rA0QES-RMTmt4aYPfw_BxFkEG", 2024-04-12T03:58:37.765 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.765 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.765 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.765 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000028", 2024-04-12T03:58:37.765 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.765 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.765 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.765 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T03:58:37.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.766 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.766 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.766 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.766 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.766 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.766 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:30.226917Z", 2024-04-12T03:58:37.766 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.766 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.766 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.766 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.766 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.767 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.767 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.767 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.767 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_M3hAtEimOhNiYWFnbqheNGinxEUyYUs", 2024-04-12T03:58:37.767 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.767 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.767 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.767 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000029", 2024-04-12T03:58:37.767 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.768 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.768 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.768 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T03:58:37.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.768 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.768 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.768 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.768 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.768 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.768 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.835684Z", 2024-04-12T03:58:37.768 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.769 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.769 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.769 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.769 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.769 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.769 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.769 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.769 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_YKUjezVBrSdKV609kwUJiahbgTrxEWT", 2024-04-12T03:58:37.769 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.769 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.770 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.770 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000030", 2024-04-12T03:58:37.770 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.770 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.770 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.770 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T03:58:37.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.770 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.770 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.770 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.771 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.771 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.771 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.783102Z", 2024-04-12T03:58:37.771 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.771 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.771 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.771 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.771 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.771 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.771 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.771 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.772 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_y1kWuI2NZcgaS3Xzy3e-wB3FaF9H1sF", 2024-04-12T03:58:37.772 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.772 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.772 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.772 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000031", 2024-04-12T03:58:37.772 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.772 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.772 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.772 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2 2024-04-12T03:58:37.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.773 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.773 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.773 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.773 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.773 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.773 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.123895Z", 2024-04-12T03:58:37.773 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.773 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.773 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.773 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.773 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.774 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.774 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.774 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.774 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_DOcnWkwBYbsHsh9M6Ay9FbDR2xRQsFa", 2024-04-12T03:58:37.774 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.774 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.774 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.774 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000032", 2024-04-12T03:58:37.775 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.775 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.775 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.775 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2 2024-04-12T03:58:37.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.775 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.775 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.775 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.775 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.775 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.775 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:31.983268Z", 2024-04-12T03:58:37.775 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.775 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.776 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.776 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.776 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.776 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.776 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.776 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.776 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_IT-YGFhGkSejwHM_u2OKDZX6QBHnz4n", 2024-04-12T03:58:37.776 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.776 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.776 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.777 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000033", 2024-04-12T03:58:37.777 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.777 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.777 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.777 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 3 2024-04-12T03:58:37.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.777 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.777 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.777 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.777 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.777 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.778 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.444868Z", 2024-04-12T03:58:37.778 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.778 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.778 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.778 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.778 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.778 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.778 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.778 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.778 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_bddOxHf7jeQPNN5yMELBLBI4_Ejs2GH", 2024-04-12T03:58:37.779 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.779 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.779 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.779 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000036", 2024-04-12T03:58:37.779 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.779 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.779 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.779 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.780 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.780 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.780 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.780 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.780 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.780 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.195404Z", 2024-04-12T03:58:37.780 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.780 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.780 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.780 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.781 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.781 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.781 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.781 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.781 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "__GRVe7hzzNrGPVYXqJs-cs9-NzpuRBf", 2024-04-12T03:58:37.781 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.781 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.781 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.781 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000037", 2024-04-12T03:58:37.782 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.782 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.782 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.782 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 10 2024-04-12T03:58:37.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.782 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.782 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.782 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.782 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.782 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.782 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:32.547464Z", 2024-04-12T03:58:37.782 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.783 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.783 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.783 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.783 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.783 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.783 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.783 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.783 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_YtY8X2RejWAnD_ae4kyPP4VDEkHoQnT", 2024-04-12T03:58:37.783 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.783 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000039", 2024-04-12T03:58:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14 2024-04-12T03:58:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.784 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.785 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.785 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.785 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.380766Z", 2024-04-12T03:58:37.785 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.785 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.785 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.785 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.785 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.785 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.785 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.785 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.786 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_mnDineLKpNM8uckylnGQ5IkwWvARAHd", 2024-04-12T03:58:37.786 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.786 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.786 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.786 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.786 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000040", 2024-04-12T03:58:37.786 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.786 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.786 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.787 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 11 2024-04-12T03:58:37.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.787 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.787 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.787 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.787 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.787 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.787 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.059113Z", 2024-04-12T03:58:37.787 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.787 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.787 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.787 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.788 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.788 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.788 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.788 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.788 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_fClYAbUh2vAtJx2D831H2oFR7Ho_PKl", 2024-04-12T03:58:37.788 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.788 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.788 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.788 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000042", 2024-04-12T03:58:37.789 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.789 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.789 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.789 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T03:58:37.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.789 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.789 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.789 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.789 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.789 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.789 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:32.942465Z", 2024-04-12T03:58:37.789 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.789 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.790 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.790 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.790 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.790 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.790 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.790 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.790 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_B8h_qFuM6n6_3Um3Mb9QY3yeMv6v5Gy", 2024-04-12T03:58:37.790 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.790 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.790 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.791 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000045", 2024-04-12T03:58:37.791 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.791 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.791 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.791 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 10 2024-04-12T03:58:37.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.791 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.791 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.791 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.791 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.791 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.792 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.244407Z", 2024-04-12T03:58:37.792 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.792 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.792 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.792 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.792 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.792 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.792 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.792 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.792 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_b08cOS0q_hfH-5mBSZ5jTzhNLfVJJet", 2024-04-12T03:58:37.793 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.793 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.793 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.793 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000046", 2024-04-12T03:58:37.793 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.793 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.793 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.793 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 13 2024-04-12T03:58:37.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.793 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.793 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.794 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.794 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.794 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.794 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:31.663508Z", 2024-04-12T03:58:37.794 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.794 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.794 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.794 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.794 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.794 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.795 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.795 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.795 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_c_ZMd4AI9r9UHHF16e8CdnjVygOrrf5", 2024-04-12T03:58:37.795 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.795 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.795 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.795 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000048", 2024-04-12T03:58:37.795 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.796 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.796 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.796 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.796 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.796 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.796 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.796 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.796 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.796 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.176469Z", 2024-04-12T03:58:37.796 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.796 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.797 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.797 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.797 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.797 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.797 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.797 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.797 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_LeR0jhVz0BSg0J5qwRP4REeuX1JasFt", 2024-04-12T03:58:37.797 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.797 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.797 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.798 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000050", 2024-04-12T03:58:37.798 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.798 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.798 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.798 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2 2024-04-12T03:58:37.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.798 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.798 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.798 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.799 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.799 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.799 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.571666Z", 2024-04-12T03:58:37.799 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.799 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.799 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.799 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.799 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.799 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.799 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.799 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.800 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_FTT73uvrMudCO9toH7QwwhZai7hNwJn", 2024-04-12T03:58:37.800 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.800 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.800 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.800 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000053", 2024-04-12T03:58:37.800 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.800 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.800 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.800 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.801 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.801 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.801 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.801 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.801 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.801 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.849488Z", 2024-04-12T03:58:37.801 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.801 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.801 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.801 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.801 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.801 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.802 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.802 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.802 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_ejO7q0XLoBkQL14IZlVm757dvoFN79F", 2024-04-12T03:58:37.802 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.802 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.802 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.802 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000055", 2024-04-12T03:58:37.802 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.802 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.803 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.803 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 8 2024-04-12T03:58:37.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.803 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.803 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.803 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.803 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.803 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.803 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:32.047512Z", 2024-04-12T03:58:37.803 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.803 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.803 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.804 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.804 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.804 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.804 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.804 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.804 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_sCZ3BwTP7C22g5FQJX8fNmaRiLHtFbp", 2024-04-12T03:58:37.804 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.804 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.804 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.805 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000056", 2024-04-12T03:58:37.805 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.805 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.805 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2 2024-04-12T03:58:37.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.805 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.805 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.805 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.805 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.805 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.806 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.027311Z", 2024-04-12T03:58:37.806 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.806 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.806 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.806 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.806 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.806 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.806 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.806 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.806 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_gQ3EX3poumgr0OlWeSzPpf6jMGWDZw5", 2024-04-12T03:58:37.806 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.807 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.807 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.807 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000057", 2024-04-12T03:58:37.807 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.807 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.807 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.807 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1 2024-04-12T03:58:37.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.807 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.807 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.808 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.808 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.808 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.808 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.395720Z", 2024-04-12T03:58:37.808 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.808 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.808 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.808 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.808 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.808 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.808 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.808 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.809 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_uPccwtWVwayTAGFr9nlKsOEif4BJ2Mo", 2024-04-12T03:58:37.809 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.809 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.809 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.809 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000058", 2024-04-12T03:58:37.809 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.809 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.810 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.810 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T03:58:37.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.810 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.810 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.810 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.810 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.810 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.810 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:31.481097Z", 2024-04-12T03:58:37.810 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.810 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.811 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.811 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.811 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.811 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.811 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.811 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.811 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_iPMnuPO7FBKBFGV0SDFrwlYjql9ozzx", 2024-04-12T03:58:37.811 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.811 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.811 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.812 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000059", 2024-04-12T03:58:37.812 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.812 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.812 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.812 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T03:58:37.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.812 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.812 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.812 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.812 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.813 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.813 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:32.864361Z", 2024-04-12T03:58:37.813 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.813 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.813 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.813 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.813 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.813 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.813 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.813 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.814 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_F_RSgL6r5PicAaNd00FQ0kd-_XcI1ID", 2024-04-12T03:58:37.814 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.814 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.814 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.814 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000060", 2024-04-12T03:58:37.814 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.814 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.814 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.814 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T03:58:37.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.815 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.815 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.815 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.815 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.815 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.815 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:35.664604Z", 2024-04-12T03:58:37.815 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.815 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.815 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.815 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.815 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.816 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.816 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.816 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.816 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.816 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_bZnbjwi_0ST8-xvSNFkp8l9Ki5itm2a", 2024-04-12T03:58:37.816 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.816 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.816 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.816 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.816 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.816 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000061", 2024-04-12T03:58:37.816 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.817 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.817 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.817 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T03:58:37.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.817 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.817 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.817 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.817 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.817 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T03:58:37.817 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.774653Z", 2024-04-12T03:58:37.817 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.818 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.818 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.818 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.818 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.818 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.818 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.818 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.818 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_CnRQdexLp-MTH4W0uH1xjxadwtoXb7Q", 2024-04-12T03:58:37.818 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.818 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.819 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.819 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000062", 2024-04-12T03:58:37.819 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.819 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.819 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.819 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 3 2024-04-12T03:58:37.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.819 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.820 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.820 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.820 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.820 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.820 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.796344Z", 2024-04-12T03:58:37.820 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.820 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.820 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.820 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.820 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.820 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.821 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.821 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.821 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_KPZnsQgQFGW0NUZSzQ8RGfmDKcSSDf8", 2024-04-12T03:58:37.821 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.821 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.821 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.821 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000076", 2024-04-12T03:58:37.821 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.822 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.822 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.822 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 3 2024-04-12T03:58:37.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.822 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.822 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.822 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.822 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.822 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.822 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:30.238284Z", 2024-04-12T03:58:37.822 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.823 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.823 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.823 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.823 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.823 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.823 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.823 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.823 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_HEZiJI3P3SWDh_t5DqTzRtHFDGc9o5e", 2024-04-12T03:58:37.823 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.823 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.823 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.824 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.824 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000133", 2024-04-12T03:58:37.824 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.824 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.824 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.824 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T03:58:37.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.824 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.824 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.824 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.825 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.825 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.825 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.696022Z", 2024-04-12T03:58:37.825 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.825 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.825 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.825 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.825 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.825 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.825 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.825 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.826 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_rGeXXDuHrzlUIWXvTzOm29do1wJMDn-", 2024-04-12T03:58:37.826 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.826 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.826 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.826 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000162", 2024-04-12T03:58:37.826 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.826 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.826 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.826 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T03:58:37.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.827 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.827 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.827 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.827 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.827 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.827 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.823565Z", 2024-04-12T03:58:37.827 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.827 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.828 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.828 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.828 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.828 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.828 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.828 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.828 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_FxDoI1G2SzKq4I4wtuCzUCGbH0P33DR", 2024-04-12T03:58:37.828 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.828 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.828 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.829 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000271", 2024-04-12T03:58:37.829 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.829 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.829 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.829 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 3 2024-04-12T03:58:37.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.829 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.829 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.829 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.830 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.830 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.830 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.897372Z", 2024-04-12T03:58:37.830 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.830 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.830 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.830 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.830 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.830 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.830 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.830 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.831 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_fq8zrdcfoLIk7ayVDQlJ2PGENHd3xrD", 2024-04-12T03:58:37.831 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.831 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.831 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.832 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000283", 2024-04-12T03:58:37.832 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.832 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.832 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.832 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T03:58:37.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.833 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.833 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.833 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.833 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.833 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T03:58:37.833 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.635658Z", 2024-04-12T03:58:37.833 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.833 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.833 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.833 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.833 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.833 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.834 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.834 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.834 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_f6tLLIP5Hw4hAYiEI22JZ7kO7x6a77D", 2024-04-12T03:58:37.834 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.834 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.834 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.834 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000317", 2024-04-12T03:58:37.834 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.834 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.835 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.835 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 3 2024-04-12T03:58:37.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.835 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.835 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.835 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.835 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.835 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T03:58:37.835 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.346593Z", 2024-04-12T03:58:37.835 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.835 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.836 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.836 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.836 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.836 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.836 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.836 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.836 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_egocdgdpZ1KOpzRszUDrNvZKmHBv5QM", 2024-04-12T03:58:37.836 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.836 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.836 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.837 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000423", 2024-04-12T03:58:37.837 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.837 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.837 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.837 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T03:58:37.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.837 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.837 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.837 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.837 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.837 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:33.963441Z", 2024-04-12T03:58:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_rTzCyxeUsaAVXbBLXyV_hRSCzD3_xiF", 2024-04-12T03:58:37.838 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.839 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.839 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:37.839 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj00000000000000000467", 2024-04-12T03:58:37.839 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T03:58:37.839 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T03:58:37.839 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T03:58:37.839 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 13 2024-04-12T03:58:37.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.839 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T03:58:37.839 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T03:58:37.839 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T03:58:37.840 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T03:58:37.840 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T03:58:37.840 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T03:58:34.332389Z", 2024-04-12T03:58:37.840 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T03:58:37.840 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T03:58:37.840 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T03:58:37.840 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T03:58:37.840 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T03:58:37.840 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T03:58:37.840 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T03:58:37.840 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T03:58:37.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:37.841 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_5YbJbYF-UKwy_c92ixcOlioCAD4dzM_", 2024-04-12T03:58:37.841 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T03:58:37.841 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T03:58:37.841 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T03:58:37.841 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:58:37.841 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:58:37.842 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T03:58:37.842 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T03:58:37.844 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T03:58:37.845 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:37.871 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T03:58:37.871 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:58:37.871 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:58:37.871 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi043.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T03:58:38.873 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:38.899 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T03:58:38.899 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T03:58:38.900 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T03:58:38.901 INFO:teuthology.orchestra.run.smithi043.stdout:anonymous 2024-04-12T03:58:38.901 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-60 2024-04-12T03:58:38.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:58:38.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:58:38.902 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:58:39.008 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:39.008 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:39.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:39.069+0000 7f2c10dbf780 1 bucket sync caught up with source: 2024-04-12T03:58:39.057 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.59.6, 00000000002.76.6, 00000000002.65.6, 00000000002.41.6, 00000000002.43.6, , 00000000002.74.6, 00000000002.119.6, 00000000001.83.6, 00000000002.65.6, 00000000002.57.6, 00000000002.35.6, 00000000001.77.6, 00000000002.92.6, 00000000002.42.6, 00000000002.153.6, 00000000002.69.6, 00000000002.65.6, 00000000002.49.6, 00000000002.72.6, 00000000002.125.6, 00000000002.67.6, 00000000002.78.6, 00000000002.37.6, 00000000002.41.6, 00000000002.44.6, 00000000002.67.6, 00000000002.76.6, , 00000000002.55.6, 00000000001.81.6, 00000000002.101.6, 00000000002.85.6, , 00000000001.57.6, 00000000002.63.6, 00000000002.103.6] 2024-04-12T03:58:39.058 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.59.6, 00000000002.76.6, 00000000002.65.6, 00000000002.41.6, 00000000002.43.6, , 00000000002.74.6, 00000000002.119.6, 00000000001.83.6, 00000000002.65.6, 00000000002.57.6, 00000000002.35.6, 00000000001.77.6, 00000000002.92.6, 00000000002.42.6, 00000000002.153.6, 00000000002.69.6, 00000000002.65.6, 00000000002.49.6, 00000000002.72.6, 00000000002.125.6, 00000000002.67.6, 00000000002.78.6, 00000000002.37.6, 00000000002.41.6, 00000000002.44.6, 00000000002.67.6, 00000000002.76.6, , 00000000002.55.6, 00000000001.81.6, 00000000002.101.6, 00000000002.85.6, , 00000000001.57.6, 00000000002.63.6, 00000000002.103.6] 2024-04-12T03:58:39.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:39.069+0000 7f2c10dbf780 0 bucket checkpoint complete 2024-04-12T03:58:39.074 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-60 2024-04-12T03:58:39.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:58:39.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T03:58:39.074 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T03:58:39.141 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:39.141 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:39.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:39.200+0000 7f25c2397780 1 waiting for incremental sync to catch up: 2024-04-12T03:58:39.189 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000004.2774.6, 00000000001.2433.6, 00000000001.2532.6, 00000000010.1766.6, 00000000004.2459.6, 00000000001.2595.6, 00000000001.2445.6, 00000000005.2542.6, 00000000007.1925.6, 00000000001.2927.6, 00000000001.1744.6] 2024-04-12T03:58:39.189 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000004.2774.6, 00000000007.2445.6, 00000000006.2542.6, 00000000010.1766.6, 00000000004.2459.6, 00000000008.2609.6, 00000000008.2465.6, 00000000005.2542.6, 00000000007.1925.6, 00000000007.2939.6, 00000000004.1768.6] 2024-04-12T03:58:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:44.202+0000 7f25c2397780 1 bucket sync caught up with source: 2024-04-12T03:58:44.190 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000004.2774.6, 00000000007.2445.6, 00000000006.2542.6, 00000000010.1766.6, 00000000004.2459.6, 00000000008.2609.6, 00000000008.2465.6, 00000000005.2542.6, 00000000007.1925.6, 00000000007.2939.6, 00000000004.1768.6] 2024-04-12T03:58:44.190 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000004.2774.6, 00000000007.2445.6, 00000000006.2542.6, 00000000010.1766.6, 00000000004.2459.6, 00000000008.2609.6, 00000000008.2465.6, 00000000005.2542.6, 00000000007.1925.6, 00000000007.2939.6, 00000000004.1768.6] 2024-04-12T03:58:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:44.202+0000 7f25c2397780 0 bucket checkpoint complete 2024-04-12T03:58:44.209 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-60 zones={a2, a1} 2024-04-12T03:58:44.209 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:44.209 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:44.209 DEBUG:boto:Method: HEAD 2024-04-12T03:58:44.209 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:44.209 DEBUG:boto:Data: 2024-04-12T03:58:44.209 DEBUG:boto:Headers: {} 2024-04-12T03:58:44.209 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:44.209 DEBUG:boto:Port: 8001 2024-04-12T03:58:44.209 DEBUG:boto:Params: {} 2024-04-12T03:58:44.210 DEBUG:boto:Token: None 2024-04-12T03:58:44.210 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:44 GMT /dkyges-60/ 2024-04-12T03:58:44.210 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UdarVwbKszNJ0BZq5Tv5oDHTsio= 2024-04-12T03:58:44.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UdarVwbKszNJ0BZq5Tv5oDHTsio=', 'Content-Length': '0'} 2024-04-12T03:58:44.211 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2024-04-12T03:58:44.211 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T03:58:44.916 DEBUG:boto:Token: None 2024-04-12T03:58:44.916 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:44 GMT /dkyges-60/ 2024-04-12T03:58:44.916 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UdarVwbKszNJ0BZq5Tv5oDHTsio= 2024-04-12T03:58:44.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UdarVwbKszNJ0BZq5Tv5oDHTsio=', 'Content-Length': '0'} 2024-04-12T03:58:44.973 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d5fa8ba7100a260-006618b174-1cfd-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:44.974 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:44.974 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:44.974 DEBUG:boto:Method: HEAD 2024-04-12T03:58:44.974 DEBUG:boto:Path: /dkyges-60/ 2024-04-12T03:58:44.974 DEBUG:boto:Data: 2024-04-12T03:58:44.974 DEBUG:boto:Headers: {} 2024-04-12T03:58:44.974 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:44.974 DEBUG:boto:Port: 8000 2024-04-12T03:58:44.974 DEBUG:boto:Params: {} 2024-04-12T03:58:44.974 DEBUG:boto:Token: None 2024-04-12T03:58:44.975 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:44 GMT /dkyges-60/ 2024-04-12T03:58:44.975 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UdarVwbKszNJ0BZq5Tv5oDHTsio= 2024-04-12T03:58:44.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UdarVwbKszNJ0BZq5Tv5oDHTsio=', 'Content-Length': '0'} 2024-04-12T03:58:44.979 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000826754a912334527-006618b174-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:44.979 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T03:58:44.980 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:44.980 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:44.980 DEBUG:boto:path=/dkyges-60/?versions&max-keys=999 2024-04-12T03:58:44.980 DEBUG:boto:auth_path=/dkyges-60/?versions&max-keys=999 2024-04-12T03:58:44.980 DEBUG:boto:Method: GET 2024-04-12T03:58:44.980 DEBUG:boto:Path: /dkyges-60/?versions&max-keys=999 2024-04-12T03:58:44.980 DEBUG:boto:Data: 2024-04-12T03:58:44.980 DEBUG:boto:Headers: {} 2024-04-12T03:58:44.980 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:44.980 DEBUG:boto:Port: 8001 2024-04-12T03:58:44.980 DEBUG:boto:Params: {} 2024-04-12T03:58:44.981 DEBUG:boto:Token: None 2024-04-12T03:58:44.981 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:44 GMT /dkyges-60/?versions 2024-04-12T03:58:44.981 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OE3O9vCjnj8zQnUnNk7sDFYUzdA= 2024-04-12T03:58:44.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OE3O9vCjnj8zQnUnNk7sDFYUzdA=', 'Content-Length': '0'} 2024-04-12T03:58:44.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003ed62bf1c268de3-006618b174-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:44.984 DEBUG:boto:b'dkyges-60999falseobj00000000000000000000nulltrue2024-04-12T03:58:35.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-12T03:58:35.260Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-12T03:58:35.312Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-12T03:58:34.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-12T03:58:34.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-12T03:58:35.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-12T03:58:35.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2024-04-12T03:58:35.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2024-04-12T03:58:35.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2024-04-12T03:58:35.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-12T03:58:35.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-12T03:58:35.596Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-12T03:58:35.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-12T03:58:34.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2024-04-12T03:58:35.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2024-04-12T03:58:34.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2024-04-12T03:58:35.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2024-04-12T03:58:35.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2024-04-12T03:58:35.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2024-04-12T03:58:32.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2024-04-12T03:58:33.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2024-04-12T03:58:35.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2024-04-12T03:58:33.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2024-04-12T03:58:34.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2024-04-12T03:58:34.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2024-04-12T03:58:33.202Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2024-04-12T03:58:35.527Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2024-04-12T03:58:34.847Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2024-04-12T03:58:30.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2024-04-12T03:58:33.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2024-04-12T03:58:34.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2024-04-12T03:58:33.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2024-04-12T03:58:31.983Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2024-04-12T03:58:35.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2024-04-12T03:58:35.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2024-04-12T03:58:32.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2024-04-12T03:58:35.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2024-04-12T03:58:35.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2024-04-12T03:58:32.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2024-04-12T03:58:35.244Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2024-04-12T03:58:31.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2024-04-12T03:58:35.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2024-04-12T03:58:33.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2024-04-12T03:58:35.849Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2024-04-12T03:58:32.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2024-04-12T03:58:34.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2024-04-12T03:58:35.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2024-04-12T03:58:31.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2024-04-12T03:58:32.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2024-04-12T03:58:35.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2024-04-12T03:58:33.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2024-04-12T03:58:34.796Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2024-04-12T03:58:30.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2024-04-12T03:58:33.696Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2024-04-12T03:58:33.823Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2024-04-12T03:58:33.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2024-04-12T03:58:33.635Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2024-04-12T03:58:34.346Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2024-04-12T03:58:33.963Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2024-04-12T03:58:34.332Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:44.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-04-12T03:58:44.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-04-12T03:58:44.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-04-12T03:58:44.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-04-12T03:58:44.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-04-12T03:58:44.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-04-12T03:58:44.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-04-12T03:58:44.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2024-04-12T03:58:44.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2024-04-12T03:58:44.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2024-04-12T03:58:44.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-04-12T03:58:44.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-04-12T03:58:44.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-04-12T03:58:44.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-04-12T03:58:44.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2024-04-12T03:58:44.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2024-04-12T03:58:44.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2024-04-12T03:58:44.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2024-04-12T03:58:44.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2024-04-12T03:58:44.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2024-04-12T03:58:44.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2024-04-12T03:58:44.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2024-04-12T03:58:44.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2024-04-12T03:58:44.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2024-04-12T03:58:44.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2024-04-12T03:58:44.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2024-04-12T03:58:44.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2024-04-12T03:58:44.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2024-04-12T03:58:44.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2024-04-12T03:58:44.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2024-04-12T03:58:44.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2024-04-12T03:58:44.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2024-04-12T03:58:44.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2024-04-12T03:58:44.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2024-04-12T03:58:44.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2024-04-12T03:58:44.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2024-04-12T03:58:44.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2024-04-12T03:58:44.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2024-04-12T03:58:44.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2024-04-12T03:58:44.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2024-04-12T03:58:44.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2024-04-12T03:58:44.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2024-04-12T03:58:44.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2024-04-12T03:58:44.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2024-04-12T03:58:44.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2024-04-12T03:58:44.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2024-04-12T03:58:44.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2024-04-12T03:58:44.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2024-04-12T03:58:44.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2024-04-12T03:58:44.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2024-04-12T03:58:44.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2024-04-12T03:58:44.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2024-04-12T03:58:44.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2024-04-12T03:58:44.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2024-04-12T03:58:44.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2024-04-12T03:58:44.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2024-04-12T03:58:44.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2024-04-12T03:58:44.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2024-04-12T03:58:44.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2024-04-12T03:58:44.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2024-04-12T03:58:44.992 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T03:58:44.992 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:44.993 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:44.993 DEBUG:boto:path=/dkyges-60/?versions&max-keys=999 2024-04-12T03:58:44.993 DEBUG:boto:auth_path=/dkyges-60/?versions&max-keys=999 2024-04-12T03:58:44.993 DEBUG:boto:Method: GET 2024-04-12T03:58:44.993 DEBUG:boto:Path: /dkyges-60/?versions&max-keys=999 2024-04-12T03:58:44.993 DEBUG:boto:Data: 2024-04-12T03:58:44.993 DEBUG:boto:Headers: {} 2024-04-12T03:58:44.993 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:44.993 DEBUG:boto:Port: 8000 2024-04-12T03:58:44.993 DEBUG:boto:Params: {} 2024-04-12T03:58:44.993 DEBUG:boto:Token: None 2024-04-12T03:58:44.994 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:44 GMT /dkyges-60/?versions 2024-04-12T03:58:44.994 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OE3O9vCjnj8zQnUnNk7sDFYUzdA= 2024-04-12T03:58:44.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OE3O9vCjnj8zQnUnNk7sDFYUzdA=', 'Content-Length': '0'} 2024-04-12T03:58:44.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005d4e9a7a31c888c-006618b175-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:44.999 DEBUG:boto:b'dkyges-60999falseobj00000000000000000000nulltrue2024-04-12T03:58:35.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-12T03:58:35.260Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-12T03:58:35.312Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-12T03:58:34.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-12T03:58:34.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-12T03:58:35.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-12T03:58:35.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2024-04-12T03:58:35.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2024-04-12T03:58:35.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2024-04-12T03:58:35.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-12T03:58:35.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-12T03:58:35.596Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-12T03:58:35.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-12T03:58:34.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2024-04-12T03:58:35.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2024-04-12T03:58:34.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2024-04-12T03:58:35.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2024-04-12T03:58:35.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2024-04-12T03:58:35.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2024-04-12T03:58:32.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2024-04-12T03:58:33.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2024-04-12T03:58:35.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2024-04-12T03:58:33.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2024-04-12T03:58:34.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2024-04-12T03:58:34.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2024-04-12T03:58:33.202Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2024-04-12T03:58:35.527Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2024-04-12T03:58:34.847Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2024-04-12T03:58:30.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2024-04-12T03:58:33.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2024-04-12T03:58:34.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2024-04-12T03:58:33.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2024-04-12T03:58:31.983Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2024-04-12T03:58:35.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2024-04-12T03:58:35.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2024-04-12T03:58:32.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2024-04-12T03:58:35.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2024-04-12T03:58:35.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2024-04-12T03:58:32.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2024-04-12T03:58:35.244Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2024-04-12T03:58:31.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2024-04-12T03:58:35.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2024-04-12T03:58:33.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2024-04-12T03:58:35.849Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2024-04-12T03:58:32.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2024-04-12T03:58:34.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2024-04-12T03:58:35.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2024-04-12T03:58:31.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2024-04-12T03:58:32.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2024-04-12T03:58:35.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2024-04-12T03:58:33.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2024-04-12T03:58:34.796Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2024-04-12T03:58:30.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2024-04-12T03:58:33.696Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2024-04-12T03:58:33.823Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2024-04-12T03:58:33.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2024-04-12T03:58:33.635Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2024-04-12T03:58:34.346Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2024-04-12T03:58:33.963Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2024-04-12T03:58:34.332Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2024-04-12T03:58:45.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2024-04-12T03:58:45.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2024-04-12T03:58:45.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2024-04-12T03:58:45.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2024-04-12T03:58:45.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2024-04-12T03:58:45.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2024-04-12T03:58:45.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2024-04-12T03:58:45.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2024-04-12T03:58:45.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2024-04-12T03:58:45.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2024-04-12T03:58:45.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2024-04-12T03:58:45.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2024-04-12T03:58:45.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2024-04-12T03:58:45.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2024-04-12T03:58:45.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2024-04-12T03:58:45.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2024-04-12T03:58:45.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2024-04-12T03:58:45.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2024-04-12T03:58:45.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2024-04-12T03:58:45.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2024-04-12T03:58:45.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2024-04-12T03:58:45.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2024-04-12T03:58:45.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2024-04-12T03:58:45.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2024-04-12T03:58:45.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2024-04-12T03:58:45.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2024-04-12T03:58:45.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2024-04-12T03:58:45.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2024-04-12T03:58:45.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2024-04-12T03:58:45.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2024-04-12T03:58:45.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2024-04-12T03:58:45.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2024-04-12T03:58:45.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2024-04-12T03:58:45.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2024-04-12T03:58:45.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2024-04-12T03:58:45.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2024-04-12T03:58:45.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2024-04-12T03:58:45.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2024-04-12T03:58:45.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2024-04-12T03:58:45.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2024-04-12T03:58:45.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2024-04-12T03:58:45.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2024-04-12T03:58:45.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2024-04-12T03:58:45.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2024-04-12T03:58:45.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2024-04-12T03:58:45.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2024-04-12T03:58:45.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2024-04-12T03:58:45.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2024-04-12T03:58:45.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2024-04-12T03:58:45.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2024-04-12T03:58:45.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2024-04-12T03:58:45.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2024-04-12T03:58:45.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2024-04-12T03:58:45.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2024-04-12T03:58:45.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2024-04-12T03:58:45.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2024-04-12T03:58:45.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2024-04-12T03:58:45.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2024-04-12T03:58:45.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2024-04-12T03:58:45.007 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.007 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.007 DEBUG:boto:path=/dkyges-60/?versions&max-keys=999 2024-04-12T03:58:45.007 DEBUG:boto:auth_path=/dkyges-60/?versions&max-keys=999 2024-04-12T03:58:45.007 DEBUG:boto:Method: GET 2024-04-12T03:58:45.007 DEBUG:boto:Path: /dkyges-60/?versions&max-keys=999 2024-04-12T03:58:45.007 DEBUG:boto:Data: 2024-04-12T03:58:45.007 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.007 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.007 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.007 DEBUG:boto:Params: {} 2024-04-12T03:58:45.008 DEBUG:boto:Token: None 2024-04-12T03:58:45.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/?versions 2024-04-12T03:58:45.008 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VgxPGJNHRUrOc9J1POe2POcHc6I= 2024-04-12T03:58:45.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VgxPGJNHRUrOc9J1POe2POcHc6I=', 'Content-Length': '0'} 2024-04-12T03:58:45.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000227ea81dc8a457b4-006618b175-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.011 DEBUG:boto:b'dkyges-60999falseobj00000000000000000000nulltrue2024-04-12T03:58:35.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-12T03:58:35.260Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-12T03:58:35.312Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-12T03:58:34.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-12T03:58:34.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-12T03:58:35.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-12T03:58:35.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2024-04-12T03:58:35.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2024-04-12T03:58:35.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2024-04-12T03:58:35.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-12T03:58:35.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-12T03:58:35.596Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-12T03:58:35.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-12T03:58:34.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2024-04-12T03:58:35.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2024-04-12T03:58:34.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2024-04-12T03:58:35.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2024-04-12T03:58:35.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2024-04-12T03:58:35.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2024-04-12T03:58:32.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2024-04-12T03:58:33.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2024-04-12T03:58:35.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2024-04-12T03:58:33.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2024-04-12T03:58:34.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2024-04-12T03:58:34.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2024-04-12T03:58:33.202Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2024-04-12T03:58:35.527Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2024-04-12T03:58:34.847Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2024-04-12T03:58:30.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2024-04-12T03:58:33.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2024-04-12T03:58:34.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2024-04-12T03:58:33.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2024-04-12T03:58:31.983Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2024-04-12T03:58:35.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2024-04-12T03:58:35.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2024-04-12T03:58:32.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2024-04-12T03:58:35.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2024-04-12T03:58:35.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2024-04-12T03:58:32.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2024-04-12T03:58:35.244Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2024-04-12T03:58:31.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2024-04-12T03:58:35.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2024-04-12T03:58:33.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2024-04-12T03:58:35.849Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2024-04-12T03:58:32.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2024-04-12T03:58:34.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2024-04-12T03:58:35.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2024-04-12T03:58:31.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2024-04-12T03:58:32.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2024-04-12T03:58:35.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2024-04-12T03:58:33.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2024-04-12T03:58:34.796Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2024-04-12T03:58:30.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2024-04-12T03:58:33.696Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2024-04-12T03:58:33.823Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2024-04-12T03:58:33.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2024-04-12T03:58:33.635Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2024-04-12T03:58:34.346Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2024-04-12T03:58:33.963Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2024-04-12T03:58:34.332Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.014 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.014 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.014 DEBUG:boto:path=/dkyges-60/?versions&max-keys=999 2024-04-12T03:58:45.014 DEBUG:boto:auth_path=/dkyges-60/?versions&max-keys=999 2024-04-12T03:58:45.015 DEBUG:boto:Method: GET 2024-04-12T03:58:45.015 DEBUG:boto:Path: /dkyges-60/?versions&max-keys=999 2024-04-12T03:58:45.015 DEBUG:boto:Data: 2024-04-12T03:58:45.015 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.015 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.015 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.015 DEBUG:boto:Params: {} 2024-04-12T03:58:45.015 DEBUG:boto:Token: None 2024-04-12T03:58:45.015 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/?versions 2024-04-12T03:58:45.015 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VgxPGJNHRUrOc9J1POe2POcHc6I= 2024-04-12T03:58:45.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VgxPGJNHRUrOc9J1POe2POcHc6I=', 'Content-Length': '0'} 2024-04-12T03:58:45.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d37b5a4140e1f15-006618b175-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.021 DEBUG:boto:b'dkyges-60999falseobj00000000000000000000nulltrue2024-04-12T03:58:35.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2024-04-12T03:58:35.260Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2024-04-12T03:58:35.312Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2024-04-12T03:58:34.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2024-04-12T03:58:34.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2024-04-12T03:58:35.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2024-04-12T03:58:35.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2024-04-12T03:58:35.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2024-04-12T03:58:35.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2024-04-12T03:58:35.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2024-04-12T03:58:35.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2024-04-12T03:58:35.596Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2024-04-12T03:58:35.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2024-04-12T03:58:34.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2024-04-12T03:58:35.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2024-04-12T03:58:34.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2024-04-12T03:58:35.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2024-04-12T03:58:35.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2024-04-12T03:58:35.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2024-04-12T03:58:32.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2024-04-12T03:58:33.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2024-04-12T03:58:35.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2024-04-12T03:58:33.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2024-04-12T03:58:34.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2024-04-12T03:58:34.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2024-04-12T03:58:33.202Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2024-04-12T03:58:35.527Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2024-04-12T03:58:34.847Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2024-04-12T03:58:30.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2024-04-12T03:58:33.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2024-04-12T03:58:34.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2024-04-12T03:58:33.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2024-04-12T03:58:31.983Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2024-04-12T03:58:35.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2024-04-12T03:58:35.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2024-04-12T03:58:32.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2024-04-12T03:58:35.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2024-04-12T03:58:35.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2024-04-12T03:58:32.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2024-04-12T03:58:35.244Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2024-04-12T03:58:31.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2024-04-12T03:58:35.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2024-04-12T03:58:33.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2024-04-12T03:58:35.849Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2024-04-12T03:58:32.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2024-04-12T03:58:34.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2024-04-12T03:58:35.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2024-04-12T03:58:31.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2024-04-12T03:58:32.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2024-04-12T03:58:35.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2024-04-12T03:58:33.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2024-04-12T03:58:34.796Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2024-04-12T03:58:30.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2024-04-12T03:58:33.696Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2024-04-12T03:58:33.823Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2024-04-12T03:58:33.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2024-04-12T03:58:33.635Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2024-04-12T03:58:34.346Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2024-04-12T03:58:33.963Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2024-04-12T03:58:34.332Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-12T03:58:45.025 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.025 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.025 DEBUG:boto:path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.025 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.025 DEBUG:boto:Method: GET 2024-04-12T03:58:45.025 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.025 DEBUG:boto:Data: 2024-04-12T03:58:45.025 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.026 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.026 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.026 DEBUG:boto:Params: {} 2024-04-12T03:58:45.026 DEBUG:boto:Token: None 2024-04-12T03:58:45.026 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.026 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uOuklri6JEZk//E5s8u4DR4fsjE= 2024-04-12T03:58:45.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uOuklri6JEZk//E5s8u4DR4fsjE=', 'Content-Length': '0'} 2024-04-12T03:58:45.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d88c13e1c6add3f0-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.074 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.075 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.075 DEBUG:boto:path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.075 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.075 DEBUG:boto:Method: GET 2024-04-12T03:58:45.075 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.075 DEBUG:boto:Data: 2024-04-12T03:58:45.075 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.075 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.075 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.075 DEBUG:boto:Params: {} 2024-04-12T03:58:45.076 DEBUG:boto:Token: None 2024-04-12T03:58:45.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.076 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uOuklri6JEZk//E5s8u4DR4fsjE= 2024-04-12T03:58:45.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uOuklri6JEZk//E5s8u4DR4fsjE=', 'Content-Length': '0'} 2024-04-12T03:58:45.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007618a750c0c091cb-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.078 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.078 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.079 DEBUG:boto:path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.079 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.079 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.079 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.079 DEBUG:boto:Data: 2024-04-12T03:58:45.079 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.079 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.079 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.079 DEBUG:boto:Params: {} 2024-04-12T03:58:45.079 DEBUG:boto:Token: None 2024-04-12T03:58:45.079 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.080 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l8CnazeImQcTq+Azn72f05inU/c= 2024-04-12T03:58:45.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l8CnazeImQcTq+Azn72f05inU/c=', 'Content-Length': '0'} 2024-04-12T03:58:45.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092c085fa70384bfb-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.083 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.083 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.083 DEBUG:boto:path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.083 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.083 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.083 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.083 DEBUG:boto:Data: 2024-04-12T03:58:45.083 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.083 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.084 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.084 DEBUG:boto:Params: {} 2024-04-12T03:58:45.084 DEBUG:boto:Token: None 2024-04-12T03:58:45.084 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.084 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l8CnazeImQcTq+Azn72f05inU/c= 2024-04-12T03:58:45.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l8CnazeImQcTq+Azn72f05inU/c=', 'Content-Length': '0'} 2024-04-12T03:58:45.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000276a0f202a666af9-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-12T03:58:45.086 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.086 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.086 DEBUG:boto:path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.087 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.087 DEBUG:boto:Method: GET 2024-04-12T03:58:45.087 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.087 DEBUG:boto:Data: 2024-04-12T03:58:45.087 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.087 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.087 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.087 DEBUG:boto:Params: {} 2024-04-12T03:58:45.087 DEBUG:boto:Token: None 2024-04-12T03:58:45.088 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.088 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uOuklri6JEZk//E5s8u4DR4fsjE= 2024-04-12T03:58:45.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uOuklri6JEZk//E5s8u4DR4fsjE=', 'Content-Length': '0'} 2024-04-12T03:58:45.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d41eff052d81ebd-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.090 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.090 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.090 DEBUG:boto:path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.090 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.090 DEBUG:boto:Method: GET 2024-04-12T03:58:45.091 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.091 DEBUG:boto:Data: 2024-04-12T03:58:45.091 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.091 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.091 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.091 DEBUG:boto:Params: {} 2024-04-12T03:58:45.092 DEBUG:boto:Token: None 2024-04-12T03:58:45.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000000?versionId=null 2024-04-12T03:58:45.092 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uOuklri6JEZk//E5s8u4DR4fsjE= 2024-04-12T03:58:45.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uOuklri6JEZk//E5s8u4DR4fsjE=', 'Content-Length': '0'} 2024-04-12T03:58:45.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086ea8b0584f25605-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.094 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.094 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.094 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:45.094 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:45.095 DEBUG:boto:Method: GET 2024-04-12T03:58:45.095 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:45.095 DEBUG:boto:Data: 2024-04-12T03:58:45.095 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.095 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.095 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.095 DEBUG:boto:Params: {} 2024-04-12T03:58:45.095 DEBUG:boto:Token: None 2024-04-12T03:58:45.095 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.095 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007646762c3ed8afed-006618b175-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.098 DEBUG:boto:b'dkyges-60obj000000000000000000001000falseobj000000000000000000002024-04-12T03:58:35.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.098 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.098 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.099 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:45.099 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:45.099 DEBUG:boto:Method: GET 2024-04-12T03:58:45.099 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:45.099 DEBUG:boto:Data: 2024-04-12T03:58:45.099 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.099 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.099 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.099 DEBUG:boto:Params: {} 2024-04-12T03:58:45.099 DEBUG:boto:Token: None 2024-04-12T03:58:45.100 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.100 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ecf843438a48cef-006618b175-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.105 DEBUG:boto:b'dkyges-60obj000000000000000000001000falseobj000000000000000000002024-04-12T03:58:35.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.105 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.105 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.105 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.105 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.105 DEBUG:boto:Data: 2024-04-12T03:58:45.105 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.106 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.106 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.106 DEBUG:boto:Params: {} 2024-04-12T03:58:45.106 DEBUG:boto:Token: None 2024-04-12T03:58:45.106 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.106 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IlZzCpsrFCo9IJUGPhHLT4aftBc= 2024-04-12T03:58:45.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IlZzCpsrFCo9IJUGPhHLT4aftBc=', 'Content-Length': '0'} 2024-04-12T03:58:45.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abe4361bf7aab599-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.108 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.108 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.108 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.108 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.109 DEBUG:boto:Data: 2024-04-12T03:58:45.109 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.109 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.109 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.109 DEBUG:boto:Params: {} 2024-04-12T03:58:45.109 DEBUG:boto:Token: None 2024-04-12T03:58:45.109 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.109 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IlZzCpsrFCo9IJUGPhHLT4aftBc= 2024-04-12T03:58:45.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IlZzCpsrFCo9IJUGPhHLT4aftBc=', 'Content-Length': '0'} 2024-04-12T03:58:45.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e5702d9ab9e7cea-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2024-04-12T03:58:45.112 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.112 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.112 DEBUG:boto:Method: GET 2024-04-12T03:58:45.112 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.112 DEBUG:boto:Data: 2024-04-12T03:58:45.112 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.112 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.112 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.112 DEBUG:boto:Params: {} 2024-04-12T03:58:45.112 DEBUG:boto:Token: None 2024-04-12T03:58:45.113 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.113 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SSF2jBO0F3d+XsW5tOaBNe4mF28= 2024-04-12T03:58:45.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SSF2jBO0F3d+XsW5tOaBNe4mF28=', 'Content-Length': '0'} 2024-04-12T03:58:45.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023d4c96dbe95ea33-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.115 DEBUG:boto:path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.115 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.115 DEBUG:boto:Method: GET 2024-04-12T03:58:45.115 DEBUG:boto:Path: /dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.115 DEBUG:boto:Data: 2024-04-12T03:58:45.115 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.115 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.116 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.116 DEBUG:boto:Params: {} 2024-04-12T03:58:45.116 DEBUG:boto:Token: None 2024-04-12T03:58:45.116 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000000 2024-04-12T03:58:45.116 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SSF2jBO0F3d+XsW5tOaBNe4mF28= 2024-04-12T03:58:45.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SSF2jBO0F3d+XsW5tOaBNe4mF28=', 'Content-Length': '0'} 2024-04-12T03:58:45.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc28b5163872b0ca-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.158 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.158 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.158 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:45.159 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:45.159 DEBUG:boto:Method: GET 2024-04-12T03:58:45.159 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:45.159 DEBUG:boto:Data: 2024-04-12T03:58:45.159 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.159 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.159 DEBUG:boto:Params: {} 2024-04-12T03:58:45.159 DEBUG:boto:Token: None 2024-04-12T03:58:45.159 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.160 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ded5ba7dc843961-006618b175-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.162 DEBUG:boto:b'dkyges-60obj000000000000000000001000falseobj000000000000000000002024-04-12T03:58:35.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.163 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.163 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.163 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:45.163 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:45.163 DEBUG:boto:Method: GET 2024-04-12T03:58:45.163 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000000 2024-04-12T03:58:45.163 DEBUG:boto:Data: 2024-04-12T03:58:45.163 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.163 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.163 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.164 DEBUG:boto:Params: {} 2024-04-12T03:58:45.164 DEBUG:boto:Token: None 2024-04-12T03:58:45.164 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.164 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035c856e610618af2-006618b175-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.169 DEBUG:boto:b'dkyges-60obj000000000000000000001000falseobj000000000000000000002024-04-12T03:58:35.648Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-12T03:58:45.169 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.169 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.169 DEBUG:boto:path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.169 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.169 DEBUG:boto:Method: GET 2024-04-12T03:58:45.169 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.170 DEBUG:boto:Data: 2024-04-12T03:58:45.170 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.170 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.170 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.170 DEBUG:boto:Params: {} 2024-04-12T03:58:45.170 DEBUG:boto:Token: None 2024-04-12T03:58:45.170 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.170 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BaXN6yyWucr5lZbeNkhN7+7qPgQ= 2024-04-12T03:58:45.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BaXN6yyWucr5lZbeNkhN7+7qPgQ=', 'Content-Length': '0'} 2024-04-12T03:58:45.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6b6dc2a638932fc-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.214 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.214 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.214 DEBUG:boto:path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.214 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.215 DEBUG:boto:Method: GET 2024-04-12T03:58:45.215 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.215 DEBUG:boto:Data: 2024-04-12T03:58:45.215 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.215 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.215 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.215 DEBUG:boto:Params: {} 2024-04-12T03:58:45.215 DEBUG:boto:Token: None 2024-04-12T03:58:45.215 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.215 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BaXN6yyWucr5lZbeNkhN7+7qPgQ= 2024-04-12T03:58:45.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BaXN6yyWucr5lZbeNkhN7+7qPgQ=', 'Content-Length': '0'} 2024-04-12T03:58:45.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b9675dc57ce34c6-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.218 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.218 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.218 DEBUG:boto:path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.218 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.218 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.218 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.218 DEBUG:boto:Data: 2024-04-12T03:58:45.218 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.218 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.219 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.219 DEBUG:boto:Params: {} 2024-04-12T03:58:45.219 DEBUG:boto:Token: None 2024-04-12T03:58:45.219 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.219 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+49zS/SvQYfLTuhcrzelWOe0YWo= 2024-04-12T03:58:45.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+49zS/SvQYfLTuhcrzelWOe0YWo=', 'Content-Length': '0'} 2024-04-12T03:58:45.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daa1944b619af112-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.221 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.221 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.221 DEBUG:boto:path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.222 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.222 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.222 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.222 DEBUG:boto:Data: 2024-04-12T03:58:45.222 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.222 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.222 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.222 DEBUG:boto:Params: {} 2024-04-12T03:58:45.222 DEBUG:boto:Token: None 2024-04-12T03:58:45.222 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.222 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+49zS/SvQYfLTuhcrzelWOe0YWo= 2024-04-12T03:58:45.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+49zS/SvQYfLTuhcrzelWOe0YWo=', 'Content-Length': '0'} 2024-04-12T03:58:45.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000746744cd7e10cb8b-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-12T03:58:45.225 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.225 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.225 DEBUG:boto:path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.225 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.225 DEBUG:boto:Method: GET 2024-04-12T03:58:45.225 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.225 DEBUG:boto:Data: 2024-04-12T03:58:45.226 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.226 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.226 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.226 DEBUG:boto:Params: {} 2024-04-12T03:58:45.226 DEBUG:boto:Token: None 2024-04-12T03:58:45.226 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.226 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BaXN6yyWucr5lZbeNkhN7+7qPgQ= 2024-04-12T03:58:45.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BaXN6yyWucr5lZbeNkhN7+7qPgQ=', 'Content-Length': '0'} 2024-04-12T03:58:45.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3af153a887bd7e8-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.228 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.228 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.229 DEBUG:boto:path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.229 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.229 DEBUG:boto:Method: GET 2024-04-12T03:58:45.229 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.229 DEBUG:boto:Data: 2024-04-12T03:58:45.229 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.229 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.229 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.229 DEBUG:boto:Params: {} 2024-04-12T03:58:45.229 DEBUG:boto:Token: None 2024-04-12T03:58:45.229 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000001?versionId=null 2024-04-12T03:58:45.230 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BaXN6yyWucr5lZbeNkhN7+7qPgQ= 2024-04-12T03:58:45.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BaXN6yyWucr5lZbeNkhN7+7qPgQ=', 'Content-Length': '0'} 2024-04-12T03:58:45.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb536c1d3673dae6-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.232 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.232 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.232 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:45.232 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:45.232 DEBUG:boto:Method: GET 2024-04-12T03:58:45.232 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:45.232 DEBUG:boto:Data: 2024-04-12T03:58:45.232 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.232 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.233 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.233 DEBUG:boto:Params: {} 2024-04-12T03:58:45.233 DEBUG:boto:Token: None 2024-04-12T03:58:45.233 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.233 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df4ebe065803b6ec-006618b175-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.236 DEBUG:boto:b'dkyges-60obj000000000000000000011000falseobj000000000000000000012024-04-12T03:58:35.260Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.236 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.236 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.236 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:45.236 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:45.237 DEBUG:boto:Method: GET 2024-04-12T03:58:45.237 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:45.237 DEBUG:boto:Data: 2024-04-12T03:58:45.237 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.237 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.237 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.237 DEBUG:boto:Params: {} 2024-04-12T03:58:45.237 DEBUG:boto:Token: None 2024-04-12T03:58:45.237 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.237 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6f046d615b768d1-006618b175-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.242 DEBUG:boto:b'dkyges-60obj000000000000000000011000falseobj000000000000000000012024-04-12T03:58:35.260Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.242 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.242 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.242 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.242 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.242 DEBUG:boto:Data: 2024-04-12T03:58:45.243 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.243 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.243 DEBUG:boto:Params: {} 2024-04-12T03:58:45.243 DEBUG:boto:Token: None 2024-04-12T03:58:45.243 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.243 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1+tPP1tER/teRecZa3dwdEsMuAI= 2024-04-12T03:58:45.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1+tPP1tER/teRecZa3dwdEsMuAI=', 'Content-Length': '0'} 2024-04-12T03:58:45.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005201731472501c23-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.245 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.245 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.245 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.245 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.246 DEBUG:boto:Data: 2024-04-12T03:58:45.246 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.246 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.246 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.246 DEBUG:boto:Params: {} 2024-04-12T03:58:45.246 DEBUG:boto:Token: None 2024-04-12T03:58:45.246 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.246 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1+tPP1tER/teRecZa3dwdEsMuAI= 2024-04-12T03:58:45.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1+tPP1tER/teRecZa3dwdEsMuAI=', 'Content-Length': '0'} 2024-04-12T03:58:45.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e78fb5b779a44eb-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2024-04-12T03:58:45.249 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.249 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.249 DEBUG:boto:Method: GET 2024-04-12T03:58:45.249 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.249 DEBUG:boto:Data: 2024-04-12T03:58:45.249 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.249 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.249 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.249 DEBUG:boto:Params: {} 2024-04-12T03:58:45.249 DEBUG:boto:Token: None 2024-04-12T03:58:45.249 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.250 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nKlss3a3U27mMG4zDyKuc7uMxNc= 2024-04-12T03:58:45.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nKlss3a3U27mMG4zDyKuc7uMxNc=', 'Content-Length': '0'} 2024-04-12T03:58:45.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2996998a1655a8f-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.252 DEBUG:boto:path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.252 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.252 DEBUG:boto:Method: GET 2024-04-12T03:58:45.252 DEBUG:boto:Path: /dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.252 DEBUG:boto:Data: 2024-04-12T03:58:45.252 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.252 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.252 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.252 DEBUG:boto:Params: {} 2024-04-12T03:58:45.252 DEBUG:boto:Token: None 2024-04-12T03:58:45.253 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000001 2024-04-12T03:58:45.253 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nKlss3a3U27mMG4zDyKuc7uMxNc= 2024-04-12T03:58:45.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nKlss3a3U27mMG4zDyKuc7uMxNc=', 'Content-Length': '0'} 2024-04-12T03:58:45.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a9502d96f8982e1-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.255 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.255 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.255 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:45.255 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:45.255 DEBUG:boto:Method: GET 2024-04-12T03:58:45.256 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:45.256 DEBUG:boto:Data: 2024-04-12T03:58:45.256 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.256 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.256 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.256 DEBUG:boto:Params: {} 2024-04-12T03:58:45.256 DEBUG:boto:Token: None 2024-04-12T03:58:45.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.256 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d606211b00b9ced-006618b175-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.259 DEBUG:boto:b'dkyges-60obj000000000000000000011000falseobj000000000000000000012024-04-12T03:58:35.260Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.259 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.260 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.260 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:45.260 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:45.260 DEBUG:boto:Method: GET 2024-04-12T03:58:45.260 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000001 2024-04-12T03:58:45.260 DEBUG:boto:Data: 2024-04-12T03:58:45.260 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.260 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.260 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.260 DEBUG:boto:Params: {} 2024-04-12T03:58:45.260 DEBUG:boto:Token: None 2024-04-12T03:58:45.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.261 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002614b7fb1c2349e9-006618b175-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.310 DEBUG:boto:b'dkyges-60obj000000000000000000011000falseobj000000000000000000012024-04-12T03:58:35.260Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-12T03:58:45.311 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.311 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.311 DEBUG:boto:path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.311 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.311 DEBUG:boto:Method: GET 2024-04-12T03:58:45.311 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.311 DEBUG:boto:Data: 2024-04-12T03:58:45.312 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.312 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.312 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.312 DEBUG:boto:Params: {} 2024-04-12T03:58:45.312 DEBUG:boto:Token: None 2024-04-12T03:58:45.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.312 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:krdK/mm0LTZX/Y9s/AoO0zWhxxM= 2024-04-12T03:58:45.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:krdK/mm0LTZX/Y9s/AoO0zWhxxM=', 'Content-Length': '0'} 2024-04-12T03:58:45.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066b900a8dcd21bd0-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.315 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.315 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.315 DEBUG:boto:path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.315 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.315 DEBUG:boto:Method: GET 2024-04-12T03:58:45.315 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.315 DEBUG:boto:Data: 2024-04-12T03:58:45.315 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.315 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.316 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.316 DEBUG:boto:Params: {} 2024-04-12T03:58:45.316 DEBUG:boto:Token: None 2024-04-12T03:58:45.316 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.316 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:krdK/mm0LTZX/Y9s/AoO0zWhxxM= 2024-04-12T03:58:45.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:krdK/mm0LTZX/Y9s/AoO0zWhxxM=', 'Content-Length': '0'} 2024-04-12T03:58:45.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046338c732bc6c0d8-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.318 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.318 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.318 DEBUG:boto:path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.319 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.319 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.319 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.319 DEBUG:boto:Data: 2024-04-12T03:58:45.319 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.319 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.319 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.319 DEBUG:boto:Params: {} 2024-04-12T03:58:45.319 DEBUG:boto:Token: None 2024-04-12T03:58:45.319 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.319 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3PzKraLsflTqB7sIuVAHKRPXuTM= 2024-04-12T03:58:45.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3PzKraLsflTqB7sIuVAHKRPXuTM=', 'Content-Length': '0'} 2024-04-12T03:58:45.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086a273678c05a80f-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.321 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.322 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.322 DEBUG:boto:path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.322 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.322 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.322 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.322 DEBUG:boto:Data: 2024-04-12T03:58:45.322 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.322 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.322 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.322 DEBUG:boto:Params: {} 2024-04-12T03:58:45.323 DEBUG:boto:Token: None 2024-04-12T03:58:45.323 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.323 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3PzKraLsflTqB7sIuVAHKRPXuTM= 2024-04-12T03:58:45.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3PzKraLsflTqB7sIuVAHKRPXuTM=', 'Content-Length': '0'} 2024-04-12T03:58:45.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089eef59226bf0b3a-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-12T03:58:45.325 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.325 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.325 DEBUG:boto:path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.326 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.326 DEBUG:boto:Method: GET 2024-04-12T03:58:45.326 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.326 DEBUG:boto:Data: 2024-04-12T03:58:45.326 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.326 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.326 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.326 DEBUG:boto:Params: {} 2024-04-12T03:58:45.326 DEBUG:boto:Token: None 2024-04-12T03:58:45.326 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.327 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:krdK/mm0LTZX/Y9s/AoO0zWhxxM= 2024-04-12T03:58:45.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:krdK/mm0LTZX/Y9s/AoO0zWhxxM=', 'Content-Length': '0'} 2024-04-12T03:58:45.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001861ddae20baba9a-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.370 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.370 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.370 DEBUG:boto:path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.371 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.371 DEBUG:boto:Method: GET 2024-04-12T03:58:45.371 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.371 DEBUG:boto:Data: 2024-04-12T03:58:45.371 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.371 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.371 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.371 DEBUG:boto:Params: {} 2024-04-12T03:58:45.371 DEBUG:boto:Token: None 2024-04-12T03:58:45.371 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000002?versionId=null 2024-04-12T03:58:45.371 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:krdK/mm0LTZX/Y9s/AoO0zWhxxM= 2024-04-12T03:58:45.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:krdK/mm0LTZX/Y9s/AoO0zWhxxM=', 'Content-Length': '0'} 2024-04-12T03:58:45.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000799cd954ea8f168e-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.374 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.374 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.374 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:45.374 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:45.374 DEBUG:boto:Method: GET 2024-04-12T03:58:45.374 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:45.374 DEBUG:boto:Data: 2024-04-12T03:58:45.374 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.375 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.375 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.375 DEBUG:boto:Params: {} 2024-04-12T03:58:45.375 DEBUG:boto:Token: None 2024-04-12T03:58:45.375 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.375 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007abbccf1142a56cc-006618b175-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.378 DEBUG:boto:b'dkyges-60obj000000000000000000021000falseobj000000000000000000022024-04-12T03:58:35.312Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.379 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.379 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.379 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:45.379 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:45.379 DEBUG:boto:Method: GET 2024-04-12T03:58:45.379 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:45.379 DEBUG:boto:Data: 2024-04-12T03:58:45.379 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.380 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.380 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.380 DEBUG:boto:Params: {} 2024-04-12T03:58:45.380 DEBUG:boto:Token: None 2024-04-12T03:58:45.380 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.380 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f5e09cdd033f36e-006618b175-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.385 DEBUG:boto:b'dkyges-60obj000000000000000000021000falseobj000000000000000000022024-04-12T03:58:35.312Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.385 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.385 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.385 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.385 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.385 DEBUG:boto:Data: 2024-04-12T03:58:45.385 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.385 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.386 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.386 DEBUG:boto:Params: {} 2024-04-12T03:58:45.386 DEBUG:boto:Token: None 2024-04-12T03:58:45.386 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.386 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:c6WDA/3es0vLdNAnKrUAB7Vx0S8= 2024-04-12T03:58:45.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:c6WDA/3es0vLdNAnKrUAB7Vx0S8=', 'Content-Length': '0'} 2024-04-12T03:58:45.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026136f29738108d9-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.388 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.388 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.388 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.388 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.388 DEBUG:boto:Data: 2024-04-12T03:58:45.388 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.389 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.389 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.389 DEBUG:boto:Params: {} 2024-04-12T03:58:45.389 DEBUG:boto:Token: None 2024-04-12T03:58:45.389 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.389 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:c6WDA/3es0vLdNAnKrUAB7Vx0S8= 2024-04-12T03:58:45.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:c6WDA/3es0vLdNAnKrUAB7Vx0S8=', 'Content-Length': '0'} 2024-04-12T03:58:45.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1a9c3dd3a4fb9f4-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2024-04-12T03:58:45.391 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.391 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.391 DEBUG:boto:Method: GET 2024-04-12T03:58:45.391 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.392 DEBUG:boto:Data: 2024-04-12T03:58:45.392 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.392 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.392 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.392 DEBUG:boto:Params: {} 2024-04-12T03:58:45.392 DEBUG:boto:Token: None 2024-04-12T03:58:45.392 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.392 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oC4anxtsV5E/4Bft6vjgUeesOOU= 2024-04-12T03:58:45.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oC4anxtsV5E/4Bft6vjgUeesOOU=', 'Content-Length': '0'} 2024-04-12T03:58:45.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006efeb23cead676ba-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.394 DEBUG:boto:path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.395 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.395 DEBUG:boto:Method: GET 2024-04-12T03:58:45.395 DEBUG:boto:Path: /dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.395 DEBUG:boto:Data: 2024-04-12T03:58:45.395 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.395 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.395 DEBUG:boto:Params: {} 2024-04-12T03:58:45.395 DEBUG:boto:Token: None 2024-04-12T03:58:45.395 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000002 2024-04-12T03:58:45.396 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oC4anxtsV5E/4Bft6vjgUeesOOU= 2024-04-12T03:58:45.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oC4anxtsV5E/4Bft6vjgUeesOOU=', 'Content-Length': '0'} 2024-04-12T03:58:45.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7d3591529b13eff-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.438 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.439 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.439 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:45.439 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:45.439 DEBUG:boto:Method: GET 2024-04-12T03:58:45.439 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:45.439 DEBUG:boto:Data: 2024-04-12T03:58:45.439 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.439 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.439 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.439 DEBUG:boto:Params: {} 2024-04-12T03:58:45.440 DEBUG:boto:Token: None 2024-04-12T03:58:45.440 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.440 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0eeb359aac956a2-006618b175-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.443 DEBUG:boto:b'dkyges-60obj000000000000000000021000falseobj000000000000000000022024-04-12T03:58:35.312Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.443 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.443 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.443 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:45.443 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:45.443 DEBUG:boto:Method: GET 2024-04-12T03:58:45.443 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000002 2024-04-12T03:58:45.444 DEBUG:boto:Data: 2024-04-12T03:58:45.444 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.444 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.444 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.444 DEBUG:boto:Params: {} 2024-04-12T03:58:45.444 DEBUG:boto:Token: None 2024-04-12T03:58:45.444 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.444 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7026851a216a4f7-006618b175-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.449 DEBUG:boto:b'dkyges-60obj000000000000000000021000falseobj000000000000000000022024-04-12T03:58:35.312Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-12T03:58:45.449 DEBUG:boto:path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.450 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.450 DEBUG:boto:path=/dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.450 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.450 DEBUG:boto:Method: GET 2024-04-12T03:58:45.450 DEBUG:boto:Path: /dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.450 DEBUG:boto:Data: 2024-04-12T03:58:45.450 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.450 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.450 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.450 DEBUG:boto:Params: {} 2024-04-12T03:58:45.450 DEBUG:boto:Token: None 2024-04-12T03:58:45.451 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.451 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sL73k+TqYlDd+BJbLJaETCs4+r0= 2024-04-12T03:58:45.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sL73k+TqYlDd+BJbLJaETCs4+r0=', 'Content-Length': '0'} 2024-04-12T03:58:45.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083e416cdd79c12ba-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.494 DEBUG:boto:path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.494 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.495 DEBUG:boto:path=/dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.495 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.495 DEBUG:boto:Method: GET 2024-04-12T03:58:45.495 DEBUG:boto:Path: /dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.495 DEBUG:boto:Data: 2024-04-12T03:58:45.495 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.495 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.495 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.495 DEBUG:boto:Params: {} 2024-04-12T03:58:45.495 DEBUG:boto:Token: None 2024-04-12T03:58:45.496 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.496 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sL73k+TqYlDd+BJbLJaETCs4+r0= 2024-04-12T03:58:45.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sL73k+TqYlDd+BJbLJaETCs4+r0=', 'Content-Length': '0'} 2024-04-12T03:58:45.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e5c03f51ca18f52-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.498 DEBUG:boto:path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.498 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.499 DEBUG:boto:path=/dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.499 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.499 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.499 DEBUG:boto:Path: /dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.499 DEBUG:boto:Data: 2024-04-12T03:58:45.499 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.499 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.499 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.499 DEBUG:boto:Params: {} 2024-04-12T03:58:45.499 DEBUG:boto:Token: None 2024-04-12T03:58:45.499 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.500 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R+z7rhu0ZCD0zlFa0ILoJGw5j0I= 2024-04-12T03:58:45.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R+z7rhu0ZCD0zlFa0ILoJGw5j0I=', 'Content-Length': '0'} 2024-04-12T03:58:45.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8fcd1b83869884e-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.502 DEBUG:boto:path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.502 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.502 DEBUG:boto:path=/dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.502 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.502 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.502 DEBUG:boto:Path: /dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.502 DEBUG:boto:Data: 2024-04-12T03:58:45.503 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.503 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.503 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.503 DEBUG:boto:Params: {} 2024-04-12T03:58:45.503 DEBUG:boto:Token: None 2024-04-12T03:58:45.503 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.503 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R+z7rhu0ZCD0zlFa0ILoJGw5j0I= 2024-04-12T03:58:45.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R+z7rhu0ZCD0zlFa0ILoJGw5j0I=', 'Content-Length': '0'} 2024-04-12T03:58:45.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7d4dff3722616d2-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-12T03:58:45.505 DEBUG:boto:path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.506 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.506 DEBUG:boto:path=/dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.506 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.506 DEBUG:boto:Method: GET 2024-04-12T03:58:45.506 DEBUG:boto:Path: /dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.506 DEBUG:boto:Data: 2024-04-12T03:58:45.506 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.506 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.506 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.506 DEBUG:boto:Params: {} 2024-04-12T03:58:45.506 DEBUG:boto:Token: None 2024-04-12T03:58:45.507 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.507 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sL73k+TqYlDd+BJbLJaETCs4+r0= 2024-04-12T03:58:45.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sL73k+TqYlDd+BJbLJaETCs4+r0=', 'Content-Length': '0'} 2024-04-12T03:58:45.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008607941e9efec156-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.509 DEBUG:boto:path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.509 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.509 DEBUG:boto:path=/dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.509 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.509 DEBUG:boto:Method: GET 2024-04-12T03:58:45.510 DEBUG:boto:Path: /dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.510 DEBUG:boto:Data: 2024-04-12T03:58:45.510 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.510 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.510 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.510 DEBUG:boto:Params: {} 2024-04-12T03:58:45.510 DEBUG:boto:Token: None 2024-04-12T03:58:45.510 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000003?versionId=null 2024-04-12T03:58:45.510 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sL73k+TqYlDd+BJbLJaETCs4+r0= 2024-04-12T03:58:45.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sL73k+TqYlDd+BJbLJaETCs4+r0=', 'Content-Length': '0'} 2024-04-12T03:58:45.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b35d57ee9d117dd-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.513 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.513 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.513 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000003 2024-04-12T03:58:45.513 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000003 2024-04-12T03:58:45.513 DEBUG:boto:Method: GET 2024-04-12T03:58:45.513 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000003 2024-04-12T03:58:45.513 DEBUG:boto:Data: 2024-04-12T03:58:45.513 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.513 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.513 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.513 DEBUG:boto:Params: {} 2024-04-12T03:58:45.514 DEBUG:boto:Token: None 2024-04-12T03:58:45.514 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.514 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d48fa818f4ab855-006618b175-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.517 DEBUG:boto:b'dkyges-60obj000000000000000000031000falseobj000000000000000000032024-04-12T03:58:34.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.517 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.517 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.518 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000003 2024-04-12T03:58:45.518 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000003 2024-04-12T03:58:45.518 DEBUG:boto:Method: GET 2024-04-12T03:58:45.518 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000003 2024-04-12T03:58:45.518 DEBUG:boto:Data: 2024-04-12T03:58:45.518 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.518 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.518 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.518 DEBUG:boto:Params: {} 2024-04-12T03:58:45.518 DEBUG:boto:Token: None 2024-04-12T03:58:45.519 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.519 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d7faebffd557577-006618b175-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.523 DEBUG:boto:b'dkyges-60obj000000000000000000031000falseobj000000000000000000032024-04-12T03:58:34.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.523 DEBUG:boto:path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.524 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.524 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.524 DEBUG:boto:Path: /dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.524 DEBUG:boto:Data: 2024-04-12T03:58:45.524 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.524 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.524 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.524 DEBUG:boto:Params: {} 2024-04-12T03:58:45.524 DEBUG:boto:Token: None 2024-04-12T03:58:45.524 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.525 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m4KrPF9UqCWcWR+B//t9ViTpISw= 2024-04-12T03:58:45.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m4KrPF9UqCWcWR+B//t9ViTpISw=', 'Content-Length': '0'} 2024-04-12T03:58:45.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7bd6c100b113ce0-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.527 DEBUG:boto:path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.527 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.527 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.527 DEBUG:boto:Path: /dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.527 DEBUG:boto:Data: 2024-04-12T03:58:45.527 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.527 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.527 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.527 DEBUG:boto:Params: {} 2024-04-12T03:58:45.527 DEBUG:boto:Token: None 2024-04-12T03:58:45.528 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.528 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m4KrPF9UqCWcWR+B//t9ViTpISw= 2024-04-12T03:58:45.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m4KrPF9UqCWcWR+B//t9ViTpISw=', 'Content-Length': '0'} 2024-04-12T03:58:45.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d436db1869fc48bf-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2024-04-12T03:58:45.530 DEBUG:boto:path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.530 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.530 DEBUG:boto:Method: GET 2024-04-12T03:58:45.530 DEBUG:boto:Path: /dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.530 DEBUG:boto:Data: 2024-04-12T03:58:45.531 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.531 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.531 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.531 DEBUG:boto:Params: {} 2024-04-12T03:58:45.531 DEBUG:boto:Token: None 2024-04-12T03:58:45.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.531 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WzIVX6EjNahsZmJp9wZF66E4/fU= 2024-04-12T03:58:45.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WzIVX6EjNahsZmJp9wZF66E4/fU=', 'Content-Length': '0'} 2024-04-12T03:58:45.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000569de7a4d72fc423-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.533 DEBUG:boto:path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.534 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.534 DEBUG:boto:Method: GET 2024-04-12T03:58:45.534 DEBUG:boto:Path: /dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.534 DEBUG:boto:Data: 2024-04-12T03:58:45.534 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.534 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.534 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.534 DEBUG:boto:Params: {} 2024-04-12T03:58:45.534 DEBUG:boto:Token: None 2024-04-12T03:58:45.535 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000003 2024-04-12T03:58:45.535 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WzIVX6EjNahsZmJp9wZF66E4/fU= 2024-04-12T03:58:45.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WzIVX6EjNahsZmJp9wZF66E4/fU=', 'Content-Length': '0'} 2024-04-12T03:58:45.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bad8b01e64c55129-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.537 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.537 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.537 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000003 2024-04-12T03:58:45.537 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000003 2024-04-12T03:58:45.537 DEBUG:boto:Method: GET 2024-04-12T03:58:45.537 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000003 2024-04-12T03:58:45.537 DEBUG:boto:Data: 2024-04-12T03:58:45.537 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.538 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.538 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.538 DEBUG:boto:Params: {} 2024-04-12T03:58:45.538 DEBUG:boto:Token: None 2024-04-12T03:58:45.538 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.538 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024c499eebb9a3373-006618b175-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.541 DEBUG:boto:b'dkyges-60obj000000000000000000031000falseobj000000000000000000032024-04-12T03:58:34.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.541 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.541 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.542 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000003 2024-04-12T03:58:45.542 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000003 2024-04-12T03:58:45.542 DEBUG:boto:Method: GET 2024-04-12T03:58:45.542 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000003 2024-04-12T03:58:45.542 DEBUG:boto:Data: 2024-04-12T03:58:45.542 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.542 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.542 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.542 DEBUG:boto:Params: {} 2024-04-12T03:58:45.542 DEBUG:boto:Token: None 2024-04-12T03:58:45.543 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.543 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af959035ca0a12e7-006618b175-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.547 DEBUG:boto:b'dkyges-60obj000000000000000000031000falseobj000000000000000000032024-04-12T03:58:34.912Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-12T03:58:45.548 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.548 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.548 DEBUG:boto:path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.548 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.548 DEBUG:boto:Method: GET 2024-04-12T03:58:45.548 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.548 DEBUG:boto:Data: 2024-04-12T03:58:45.548 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.549 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.549 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.549 DEBUG:boto:Params: {} 2024-04-12T03:58:45.549 DEBUG:boto:Token: None 2024-04-12T03:58:45.549 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.549 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ST1bqsEyL+APBtMERTWMDz3w0BI= 2024-04-12T03:58:45.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ST1bqsEyL+APBtMERTWMDz3w0BI=', 'Content-Length': '0'} 2024-04-12T03:58:45.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042f146fdd1138909-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.552 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.552 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.552 DEBUG:boto:path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.552 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.552 DEBUG:boto:Method: GET 2024-04-12T03:58:45.553 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.553 DEBUG:boto:Data: 2024-04-12T03:58:45.553 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.553 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.553 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.553 DEBUG:boto:Params: {} 2024-04-12T03:58:45.553 DEBUG:boto:Token: None 2024-04-12T03:58:45.553 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.553 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ST1bqsEyL+APBtMERTWMDz3w0BI= 2024-04-12T03:58:45.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ST1bqsEyL+APBtMERTWMDz3w0BI=', 'Content-Length': '0'} 2024-04-12T03:58:45.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf6b90d08035f304-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.598 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.598 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.599 DEBUG:boto:path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.599 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.599 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.599 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.599 DEBUG:boto:Data: 2024-04-12T03:58:45.599 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.599 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.599 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.599 DEBUG:boto:Params: {} 2024-04-12T03:58:45.599 DEBUG:boto:Token: None 2024-04-12T03:58:45.599 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.600 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9kbtVSPtG7mNL6ZV08xQYqzGmIw= 2024-04-12T03:58:45.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9kbtVSPtG7mNL6ZV08xQYqzGmIw=', 'Content-Length': '0'} 2024-04-12T03:58:45.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021348ee57e516432-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.602 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.602 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.602 DEBUG:boto:path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.602 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.602 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.602 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.602 DEBUG:boto:Data: 2024-04-12T03:58:45.602 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.603 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.603 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.603 DEBUG:boto:Params: {} 2024-04-12T03:58:45.603 DEBUG:boto:Token: None 2024-04-12T03:58:45.603 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.603 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9kbtVSPtG7mNL6ZV08xQYqzGmIw= 2024-04-12T03:58:45.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9kbtVSPtG7mNL6ZV08xQYqzGmIw=', 'Content-Length': '0'} 2024-04-12T03:58:45.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000068f3c8831615fb0-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-12T03:58:45.605 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.605 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.605 DEBUG:boto:path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.605 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.605 DEBUG:boto:Method: GET 2024-04-12T03:58:45.606 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.606 DEBUG:boto:Data: 2024-04-12T03:58:45.606 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.606 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.606 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.606 DEBUG:boto:Params: {} 2024-04-12T03:58:45.606 DEBUG:boto:Token: None 2024-04-12T03:58:45.606 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.606 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ST1bqsEyL+APBtMERTWMDz3w0BI= 2024-04-12T03:58:45.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ST1bqsEyL+APBtMERTWMDz3w0BI=', 'Content-Length': '0'} 2024-04-12T03:58:45.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fe7eb31266515a8-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.609 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.609 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.609 DEBUG:boto:path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.609 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.610 DEBUG:boto:Method: GET 2024-04-12T03:58:45.610 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.610 DEBUG:boto:Data: 2024-04-12T03:58:45.610 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.610 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.610 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.610 DEBUG:boto:Params: {} 2024-04-12T03:58:45.610 DEBUG:boto:Token: None 2024-04-12T03:58:45.610 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000004?versionId=null 2024-04-12T03:58:45.610 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ST1bqsEyL+APBtMERTWMDz3w0BI= 2024-04-12T03:58:45.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ST1bqsEyL+APBtMERTWMDz3w0BI=', 'Content-Length': '0'} 2024-04-12T03:58:45.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1dd5d63b0c9ace4-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.613 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.613 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.613 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:45.613 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:45.613 DEBUG:boto:Method: GET 2024-04-12T03:58:45.613 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:45.613 DEBUG:boto:Data: 2024-04-12T03:58:45.613 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.613 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.613 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.613 DEBUG:boto:Params: {} 2024-04-12T03:58:45.614 DEBUG:boto:Token: None 2024-04-12T03:58:45.614 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.614 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cd626facbdbcfe9-006618b175-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.617 DEBUG:boto:b'dkyges-60obj000000000000000000041000falseobj000000000000000000042024-04-12T03:58:34.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.617 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.617 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.617 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:45.618 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:45.618 DEBUG:boto:Method: GET 2024-04-12T03:58:45.618 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:45.618 DEBUG:boto:Data: 2024-04-12T03:58:45.618 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.618 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.618 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.618 DEBUG:boto:Params: {} 2024-04-12T03:58:45.618 DEBUG:boto:Token: None 2024-04-12T03:58:45.618 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.619 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010a5c0d0a194c07e-006618b175-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.623 DEBUG:boto:b'dkyges-60obj000000000000000000041000falseobj000000000000000000042024-04-12T03:58:34.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.624 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.624 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.624 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.624 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.624 DEBUG:boto:Data: 2024-04-12T03:58:45.624 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.624 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.624 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.624 DEBUG:boto:Params: {} 2024-04-12T03:58:45.625 DEBUG:boto:Token: None 2024-04-12T03:58:45.625 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.625 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I4UhaO0cT2XKEFmya5hQZuNhkz8= 2024-04-12T03:58:45.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I4UhaO0cT2XKEFmya5hQZuNhkz8=', 'Content-Length': '0'} 2024-04-12T03:58:45.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da175337fa572661-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.627 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.627 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.627 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.627 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.627 DEBUG:boto:Data: 2024-04-12T03:58:45.627 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.627 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.628 DEBUG:boto:Params: {} 2024-04-12T03:58:45.628 DEBUG:boto:Token: None 2024-04-12T03:58:45.628 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I4UhaO0cT2XKEFmya5hQZuNhkz8= 2024-04-12T03:58:45.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I4UhaO0cT2XKEFmya5hQZuNhkz8=', 'Content-Length': '0'} 2024-04-12T03:58:45.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f194140a9b2ea67-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2024-04-12T03:58:45.630 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.630 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.631 DEBUG:boto:Method: GET 2024-04-12T03:58:45.631 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.631 DEBUG:boto:Data: 2024-04-12T03:58:45.631 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.631 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.631 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.631 DEBUG:boto:Params: {} 2024-04-12T03:58:45.631 DEBUG:boto:Token: None 2024-04-12T03:58:45.631 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.631 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Odz2l0vWRui8OksTs3jYCnewYjQ= 2024-04-12T03:58:45.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Odz2l0vWRui8OksTs3jYCnewYjQ=', 'Content-Length': '0'} 2024-04-12T03:58:45.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bfb367fe383086a-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.674 DEBUG:boto:path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.675 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.675 DEBUG:boto:Method: GET 2024-04-12T03:58:45.675 DEBUG:boto:Path: /dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.675 DEBUG:boto:Data: 2024-04-12T03:58:45.675 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.675 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.675 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.675 DEBUG:boto:Params: {} 2024-04-12T03:58:45.675 DEBUG:boto:Token: None 2024-04-12T03:58:45.675 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000004 2024-04-12T03:58:45.676 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Odz2l0vWRui8OksTs3jYCnewYjQ= 2024-04-12T03:58:45.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Odz2l0vWRui8OksTs3jYCnewYjQ=', 'Content-Length': '0'} 2024-04-12T03:58:45.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dee5c8b28422d038-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.678 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.678 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.678 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:45.678 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:45.679 DEBUG:boto:Method: GET 2024-04-12T03:58:45.679 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:45.679 DEBUG:boto:Data: 2024-04-12T03:58:45.679 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.679 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.679 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.679 DEBUG:boto:Params: {} 2024-04-12T03:58:45.679 DEBUG:boto:Token: None 2024-04-12T03:58:45.679 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.679 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.682 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fef7908ae28ac0d8-006618b175-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.682 DEBUG:boto:b'dkyges-60obj000000000000000000041000falseobj000000000000000000042024-04-12T03:58:34.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.682 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.682 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.682 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:45.683 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:45.683 DEBUG:boto:Method: GET 2024-04-12T03:58:45.683 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000004 2024-04-12T03:58:45.683 DEBUG:boto:Data: 2024-04-12T03:58:45.683 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.683 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.683 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.683 DEBUG:boto:Params: {} 2024-04-12T03:58:45.683 DEBUG:boto:Token: None 2024-04-12T03:58:45.683 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.684 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1111f3f51f3fe34-006618b175-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.688 DEBUG:boto:b'dkyges-60obj000000000000000000041000falseobj000000000000000000042024-04-12T03:58:34.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-12T03:58:45.689 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.689 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.689 DEBUG:boto:path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.689 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.689 DEBUG:boto:Method: GET 2024-04-12T03:58:45.689 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.689 DEBUG:boto:Data: 2024-04-12T03:58:45.689 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.689 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.689 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.690 DEBUG:boto:Params: {} 2024-04-12T03:58:45.690 DEBUG:boto:Token: None 2024-04-12T03:58:45.690 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.690 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7TZrmByBagpw+NyXazYqYxF84YU= 2024-04-12T03:58:45.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7TZrmByBagpw+NyXazYqYxF84YU=', 'Content-Length': '0'} 2024-04-12T03:58:45.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000319eee682bf0d841-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.692 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.692 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.693 DEBUG:boto:path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.693 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.693 DEBUG:boto:Method: GET 2024-04-12T03:58:45.693 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.693 DEBUG:boto:Data: 2024-04-12T03:58:45.693 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.693 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.693 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.693 DEBUG:boto:Params: {} 2024-04-12T03:58:45.693 DEBUG:boto:Token: None 2024-04-12T03:58:45.693 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.694 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7TZrmByBagpw+NyXazYqYxF84YU= 2024-04-12T03:58:45.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7TZrmByBagpw+NyXazYqYxF84YU=', 'Content-Length': '0'} 2024-04-12T03:58:45.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9fc0b4d9ca6a873-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.742 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.743 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.743 DEBUG:boto:path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.743 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.743 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.743 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.743 DEBUG:boto:Data: 2024-04-12T03:58:45.743 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.743 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.743 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.743 DEBUG:boto:Params: {} 2024-04-12T03:58:45.744 DEBUG:boto:Token: None 2024-04-12T03:58:45.744 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.744 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lo3xjT7zUNn9q5BjtmTHFIZtDjE= 2024-04-12T03:58:45.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lo3xjT7zUNn9q5BjtmTHFIZtDjE=', 'Content-Length': '0'} 2024-04-12T03:58:45.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b622aa6e634897b-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.746 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.746 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.746 DEBUG:boto:path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.746 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.746 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.747 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.747 DEBUG:boto:Data: 2024-04-12T03:58:45.747 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.747 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.747 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.747 DEBUG:boto:Params: {} 2024-04-12T03:58:45.747 DEBUG:boto:Token: None 2024-04-12T03:58:45.747 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.747 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lo3xjT7zUNn9q5BjtmTHFIZtDjE= 2024-04-12T03:58:45.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lo3xjT7zUNn9q5BjtmTHFIZtDjE=', 'Content-Length': '0'} 2024-04-12T03:58:45.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f29b79e1b06f9ff-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-12T03:58:45.750 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.750 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.750 DEBUG:boto:path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.750 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.750 DEBUG:boto:Method: GET 2024-04-12T03:58:45.750 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.750 DEBUG:boto:Data: 2024-04-12T03:58:45.750 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.751 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.751 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.751 DEBUG:boto:Params: {} 2024-04-12T03:58:45.751 DEBUG:boto:Token: None 2024-04-12T03:58:45.751 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.751 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7TZrmByBagpw+NyXazYqYxF84YU= 2024-04-12T03:58:45.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7TZrmByBagpw+NyXazYqYxF84YU=', 'Content-Length': '0'} 2024-04-12T03:58:45.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d06bc75f664942f-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.753 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.753 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.753 DEBUG:boto:path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.754 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.754 DEBUG:boto:Method: GET 2024-04-12T03:58:45.754 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.754 DEBUG:boto:Data: 2024-04-12T03:58:45.754 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.754 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.754 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.754 DEBUG:boto:Params: {} 2024-04-12T03:58:45.754 DEBUG:boto:Token: None 2024-04-12T03:58:45.754 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000005?versionId=null 2024-04-12T03:58:45.755 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7TZrmByBagpw+NyXazYqYxF84YU= 2024-04-12T03:58:45.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7TZrmByBagpw+NyXazYqYxF84YU=', 'Content-Length': '0'} 2024-04-12T03:58:45.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bf14afbcd206838-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.757 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.757 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.757 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:45.757 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:45.757 DEBUG:boto:Method: GET 2024-04-12T03:58:45.757 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:45.757 DEBUG:boto:Data: 2024-04-12T03:58:45.757 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.757 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.758 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.758 DEBUG:boto:Params: {} 2024-04-12T03:58:45.758 DEBUG:boto:Token: None 2024-04-12T03:58:45.758 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.758 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077e0ef14c15e9e47-006618b175-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.761 DEBUG:boto:b'dkyges-60obj000000000000000000051000falseobj000000000000000000052024-04-12T03:58:35.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.761 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.761 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.761 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:45.761 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:45.761 DEBUG:boto:Method: GET 2024-04-12T03:58:45.762 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:45.762 DEBUG:boto:Data: 2024-04-12T03:58:45.762 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.762 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.762 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.762 DEBUG:boto:Params: {} 2024-04-12T03:58:45.762 DEBUG:boto:Token: None 2024-04-12T03:58:45.762 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.762 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.766 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005039d054c65393c-006618b175-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.766 DEBUG:boto:b'dkyges-60obj000000000000000000051000falseobj000000000000000000052024-04-12T03:58:35.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.767 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.767 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.767 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.767 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.767 DEBUG:boto:Data: 2024-04-12T03:58:45.767 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.767 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.767 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.767 DEBUG:boto:Params: {} 2024-04-12T03:58:45.768 DEBUG:boto:Token: None 2024-04-12T03:58:45.768 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.768 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ktSZHvtXuFk3ZR5E8JGok9Q3oRQ= 2024-04-12T03:58:45.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ktSZHvtXuFk3ZR5E8JGok9Q3oRQ=', 'Content-Length': '0'} 2024-04-12T03:58:45.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afaf6fd18ef14874-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.770 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.770 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.770 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.770 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.770 DEBUG:boto:Data: 2024-04-12T03:58:45.770 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.770 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.770 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.770 DEBUG:boto:Params: {} 2024-04-12T03:58:45.771 DEBUG:boto:Token: None 2024-04-12T03:58:45.771 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.771 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ktSZHvtXuFk3ZR5E8JGok9Q3oRQ= 2024-04-12T03:58:45.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ktSZHvtXuFk3ZR5E8JGok9Q3oRQ=', 'Content-Length': '0'} 2024-04-12T03:58:45.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003977d4046e22550b-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2024-04-12T03:58:45.773 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.773 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.773 DEBUG:boto:Method: GET 2024-04-12T03:58:45.774 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.774 DEBUG:boto:Data: 2024-04-12T03:58:45.774 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.774 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.774 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.774 DEBUG:boto:Params: {} 2024-04-12T03:58:45.774 DEBUG:boto:Token: None 2024-04-12T03:58:45.774 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.774 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:871sdeicg0IAzy8Uhx/yhhXpwQM= 2024-04-12T03:58:45.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:871sdeicg0IAzy8Uhx/yhhXpwQM=', 'Content-Length': '0'} 2024-04-12T03:58:45.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7ab40d1b9b002e2-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.776 DEBUG:boto:path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.776 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.777 DEBUG:boto:Method: GET 2024-04-12T03:58:45.777 DEBUG:boto:Path: /dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.777 DEBUG:boto:Data: 2024-04-12T03:58:45.777 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.777 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.777 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.777 DEBUG:boto:Params: {} 2024-04-12T03:58:45.777 DEBUG:boto:Token: None 2024-04-12T03:58:45.777 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000005 2024-04-12T03:58:45.777 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:871sdeicg0IAzy8Uhx/yhhXpwQM= 2024-04-12T03:58:45.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:871sdeicg0IAzy8Uhx/yhhXpwQM=', 'Content-Length': '0'} 2024-04-12T03:58:45.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000231a711ead87f59e-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.780 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.780 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.780 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:45.780 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:45.780 DEBUG:boto:Method: GET 2024-04-12T03:58:45.780 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:45.780 DEBUG:boto:Data: 2024-04-12T03:58:45.780 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.780 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.780 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.780 DEBUG:boto:Params: {} 2024-04-12T03:58:45.781 DEBUG:boto:Token: None 2024-04-12T03:58:45.781 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.781 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c211854a9048fe96-006618b175-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.783 DEBUG:boto:b'dkyges-60obj000000000000000000051000falseobj000000000000000000052024-04-12T03:58:35.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.784 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.784 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.784 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:45.784 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:45.784 DEBUG:boto:Method: GET 2024-04-12T03:58:45.784 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000005 2024-04-12T03:58:45.784 DEBUG:boto:Data: 2024-04-12T03:58:45.784 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.784 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.785 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.785 DEBUG:boto:Params: {} 2024-04-12T03:58:45.785 DEBUG:boto:Token: None 2024-04-12T03:58:45.785 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.785 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f79ec32dd1cb144-006618b175-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.830 DEBUG:boto:b'dkyges-60obj000000000000000000051000falseobj000000000000000000052024-04-12T03:58:35.580Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-12T03:58:45.831 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.831 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.831 DEBUG:boto:path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.831 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.831 DEBUG:boto:Method: GET 2024-04-12T03:58:45.831 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.831 DEBUG:boto:Data: 2024-04-12T03:58:45.831 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.831 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.831 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.831 DEBUG:boto:Params: {} 2024-04-12T03:58:45.832 DEBUG:boto:Token: None 2024-04-12T03:58:45.832 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.832 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:759LxRu76Y2dB19HeyFGObWMesA= 2024-04-12T03:58:45.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:759LxRu76Y2dB19HeyFGObWMesA=', 'Content-Length': '0'} 2024-04-12T03:58:45.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a50f471e87e51c7-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.834 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.834 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.835 DEBUG:boto:path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.835 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.835 DEBUG:boto:Method: GET 2024-04-12T03:58:45.835 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.835 DEBUG:boto:Data: 2024-04-12T03:58:45.835 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.835 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.835 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.835 DEBUG:boto:Params: {} 2024-04-12T03:58:45.835 DEBUG:boto:Token: None 2024-04-12T03:58:45.835 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.836 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:759LxRu76Y2dB19HeyFGObWMesA= 2024-04-12T03:58:45.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:759LxRu76Y2dB19HeyFGObWMesA=', 'Content-Length': '0'} 2024-04-12T03:58:45.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc4ff2b358617be-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.838 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.838 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.838 DEBUG:boto:path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.838 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.838 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.838 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.838 DEBUG:boto:Data: 2024-04-12T03:58:45.839 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.839 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.839 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.839 DEBUG:boto:Params: {} 2024-04-12T03:58:45.839 DEBUG:boto:Token: None 2024-04-12T03:58:45.839 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.839 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ILcu8fo5tFCPMj66RZuAl67EVCg= 2024-04-12T03:58:45.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ILcu8fo5tFCPMj66RZuAl67EVCg=', 'Content-Length': '0'} 2024-04-12T03:58:45.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000472905360f877332-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.841 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.841 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.842 DEBUG:boto:path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.842 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.842 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.842 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.842 DEBUG:boto:Data: 2024-04-12T03:58:45.842 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.842 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.842 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.842 DEBUG:boto:Params: {} 2024-04-12T03:58:45.842 DEBUG:boto:Token: None 2024-04-12T03:58:45.843 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.843 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ILcu8fo5tFCPMj66RZuAl67EVCg= 2024-04-12T03:58:45.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ILcu8fo5tFCPMj66RZuAl67EVCg=', 'Content-Length': '0'} 2024-04-12T03:58:45.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6d5ed3eb2414e7a-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-12T03:58:45.845 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.845 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.845 DEBUG:boto:path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.845 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.845 DEBUG:boto:Method: GET 2024-04-12T03:58:45.845 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.845 DEBUG:boto:Data: 2024-04-12T03:58:45.845 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.845 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.846 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.846 DEBUG:boto:Params: {} 2024-04-12T03:58:45.846 DEBUG:boto:Token: None 2024-04-12T03:58:45.846 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.846 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:759LxRu76Y2dB19HeyFGObWMesA= 2024-04-12T03:58:45.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:759LxRu76Y2dB19HeyFGObWMesA=', 'Content-Length': '0'} 2024-04-12T03:58:45.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2edd487ec83cc09-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.848 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.849 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.849 DEBUG:boto:path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.849 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.849 DEBUG:boto:Method: GET 2024-04-12T03:58:45.849 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.849 DEBUG:boto:Data: 2024-04-12T03:58:45.849 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.849 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.849 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.849 DEBUG:boto:Params: {} 2024-04-12T03:58:45.849 DEBUG:boto:Token: None 2024-04-12T03:58:45.850 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000006?versionId=null 2024-04-12T03:58:45.850 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:759LxRu76Y2dB19HeyFGObWMesA= 2024-04-12T03:58:45.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:759LxRu76Y2dB19HeyFGObWMesA=', 'Content-Length': '0'} 2024-04-12T03:58:45.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ab55bace65653a4-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.852 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.852 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.852 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:45.852 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:45.852 DEBUG:boto:Method: GET 2024-04-12T03:58:45.852 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:45.852 DEBUG:boto:Data: 2024-04-12T03:58:45.852 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.852 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.852 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.853 DEBUG:boto:Params: {} 2024-04-12T03:58:45.853 DEBUG:boto:Token: None 2024-04-12T03:58:45.853 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.853 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009821481001816c36-006618b175-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.856 DEBUG:boto:b'dkyges-60obj000000000000000000061000falseobj000000000000000000062024-04-12T03:58:35.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.856 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.856 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.856 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:45.856 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:45.856 DEBUG:boto:Method: GET 2024-04-12T03:58:45.856 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:45.856 DEBUG:boto:Data: 2024-04-12T03:58:45.857 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.857 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.857 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.857 DEBUG:boto:Params: {} 2024-04-12T03:58:45.857 DEBUG:boto:Token: None 2024-04-12T03:58:45.857 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.857 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c81415339caad91a-006618b175-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.862 DEBUG:boto:b'dkyges-60obj000000000000000000061000falseobj000000000000000000062024-04-12T03:58:35.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.862 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.862 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.862 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.862 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.862 DEBUG:boto:Data: 2024-04-12T03:58:45.863 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.863 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.863 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.863 DEBUG:boto:Params: {} 2024-04-12T03:58:45.863 DEBUG:boto:Token: None 2024-04-12T03:58:45.863 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.863 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ipa+3vNeGbJwE+gGX2WLdPobZZA= 2024-04-12T03:58:45.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ipa+3vNeGbJwE+gGX2WLdPobZZA=', 'Content-Length': '0'} 2024-04-12T03:58:45.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a107b96a001f8901-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.865 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.865 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.865 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.866 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.866 DEBUG:boto:Data: 2024-04-12T03:58:45.866 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.866 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.866 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.866 DEBUG:boto:Params: {} 2024-04-12T03:58:45.866 DEBUG:boto:Token: None 2024-04-12T03:58:45.866 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.866 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ipa+3vNeGbJwE+gGX2WLdPobZZA= 2024-04-12T03:58:45.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ipa+3vNeGbJwE+gGX2WLdPobZZA=', 'Content-Length': '0'} 2024-04-12T03:58:45.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de75b87e0b7384be-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2024-04-12T03:58:45.868 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.868 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.868 DEBUG:boto:Method: GET 2024-04-12T03:58:45.868 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.869 DEBUG:boto:Data: 2024-04-12T03:58:45.869 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.869 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.869 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.869 DEBUG:boto:Params: {} 2024-04-12T03:58:45.869 DEBUG:boto:Token: None 2024-04-12T03:58:45.869 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.869 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WkW4FciKxpfg/gU6iJl7ADNm/y0= 2024-04-12T03:58:45.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WkW4FciKxpfg/gU6iJl7ADNm/y0=', 'Content-Length': '0'} 2024-04-12T03:58:45.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a93f92c4748ddcf-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.871 DEBUG:boto:path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.871 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.871 DEBUG:boto:Method: GET 2024-04-12T03:58:45.871 DEBUG:boto:Path: /dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.871 DEBUG:boto:Data: 2024-04-12T03:58:45.872 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.872 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.872 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.872 DEBUG:boto:Params: {} 2024-04-12T03:58:45.872 DEBUG:boto:Token: None 2024-04-12T03:58:45.872 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000006 2024-04-12T03:58:45.872 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WkW4FciKxpfg/gU6iJl7ADNm/y0= 2024-04-12T03:58:45.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WkW4FciKxpfg/gU6iJl7ADNm/y0=', 'Content-Length': '0'} 2024-04-12T03:58:45.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000531a8cfe9bd458e7-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.874 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.874 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.874 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:45.875 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:45.875 DEBUG:boto:Method: GET 2024-04-12T03:58:45.875 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:45.875 DEBUG:boto:Data: 2024-04-12T03:58:45.875 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.875 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.875 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.875 DEBUG:boto:Params: {} 2024-04-12T03:58:45.875 DEBUG:boto:Token: None 2024-04-12T03:58:45.875 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.875 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fbb926c0825e901-006618b175-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.918 DEBUG:boto:b'dkyges-60obj000000000000000000061000falseobj000000000000000000062024-04-12T03:58:35.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.919 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.919 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.919 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:45.919 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:45.919 DEBUG:boto:Method: GET 2024-04-12T03:58:45.919 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000006 2024-04-12T03:58:45.919 DEBUG:boto:Data: 2024-04-12T03:58:45.919 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.919 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.919 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.919 DEBUG:boto:Params: {} 2024-04-12T03:58:45.920 DEBUG:boto:Token: None 2024-04-12T03:58:45.920 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.920 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc9769e0bef630cc-006618b175-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.924 DEBUG:boto:b'dkyges-60obj000000000000000000061000falseobj000000000000000000062024-04-12T03:58:35.328Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-04-12T03:58:45.925 DEBUG:boto:path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:45.925 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:45.925 DEBUG:boto:path=/dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.925 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.925 DEBUG:boto:Method: GET 2024-04-12T03:58:45.925 DEBUG:boto:Path: /dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.925 DEBUG:boto:Data: 2024-04-12T03:58:45.925 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.925 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.925 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.926 DEBUG:boto:Params: {} 2024-04-12T03:58:45.926 DEBUG:boto:Token: None 2024-04-12T03:58:45.926 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.926 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8XfE4AaaeMOWdIM60B9BoC6lrdA= 2024-04-12T03:58:45.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8XfE4AaaeMOWdIM60B9BoC6lrdA=', 'Content-Length': '0'} 2024-04-12T03:58:45.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088493d435f879759-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.928 DEBUG:boto:path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:45.928 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:45.929 DEBUG:boto:path=/dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.929 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.929 DEBUG:boto:Method: GET 2024-04-12T03:58:45.929 DEBUG:boto:Path: /dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.929 DEBUG:boto:Data: 2024-04-12T03:58:45.929 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.929 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.929 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.929 DEBUG:boto:Params: {} 2024-04-12T03:58:45.929 DEBUG:boto:Token: None 2024-04-12T03:58:45.930 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.930 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8XfE4AaaeMOWdIM60B9BoC6lrdA= 2024-04-12T03:58:45.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8XfE4AaaeMOWdIM60B9BoC6lrdA=', 'Content-Length': '0'} 2024-04-12T03:58:45.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b6de0202774cf46-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.978 DEBUG:boto:path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:45.979 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:45.979 DEBUG:boto:path=/dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.979 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.979 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.979 DEBUG:boto:Path: /dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.979 DEBUG:boto:Data: 2024-04-12T03:58:45.979 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.979 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.979 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.979 DEBUG:boto:Params: {} 2024-04-12T03:58:45.979 DEBUG:boto:Token: None 2024-04-12T03:58:45.980 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.980 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xjQfp8CIIRbIrb+zmB4Y6xhO0Zo= 2024-04-12T03:58:45.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xjQfp8CIIRbIrb+zmB4Y6xhO0Zo=', 'Content-Length': '0'} 2024-04-12T03:58:45.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffaa789d23034e69-006618b175-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.982 DEBUG:boto:path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:45.982 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:45.982 DEBUG:boto:path=/dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.982 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.982 DEBUG:boto:Method: HEAD 2024-04-12T03:58:45.982 DEBUG:boto:Path: /dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.982 DEBUG:boto:Data: 2024-04-12T03:58:45.983 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.983 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.983 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.983 DEBUG:boto:Params: {} 2024-04-12T03:58:45.983 DEBUG:boto:Token: None 2024-04-12T03:58:45.983 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.983 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xjQfp8CIIRbIrb+zmB4Y6xhO0Zo= 2024-04-12T03:58:45.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xjQfp8CIIRbIrb+zmB4Y6xhO0Zo=', 'Content-Length': '0'} 2024-04-12T03:58:45.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098f94c0449f8af41-006618b175-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-04-12T03:58:45.985 DEBUG:boto:path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:45.985 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:45.985 DEBUG:boto:path=/dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.986 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.986 DEBUG:boto:Method: GET 2024-04-12T03:58:45.986 DEBUG:boto:Path: /dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.986 DEBUG:boto:Data: 2024-04-12T03:58:45.986 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.986 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.986 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.986 DEBUG:boto:Params: {} 2024-04-12T03:58:45.986 DEBUG:boto:Token: None 2024-04-12T03:58:45.986 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.987 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8XfE4AaaeMOWdIM60B9BoC6lrdA= 2024-04-12T03:58:45.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8XfE4AaaeMOWdIM60B9BoC6lrdA=', 'Content-Length': '0'} 2024-04-12T03:58:45.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000153edd304f5c18f6-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.989 DEBUG:boto:path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:45.989 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:45.989 DEBUG:boto:path=/dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.989 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.989 DEBUG:boto:Method: GET 2024-04-12T03:58:45.989 DEBUG:boto:Path: /dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.989 DEBUG:boto:Data: 2024-04-12T03:58:45.989 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.989 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.989 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.990 DEBUG:boto:Params: {} 2024-04-12T03:58:45.990 DEBUG:boto:Token: None 2024-04-12T03:58:45.990 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/obj00000000000000000007?versionId=null 2024-04-12T03:58:45.990 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8XfE4AaaeMOWdIM60B9BoC6lrdA= 2024-04-12T03:58:45.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8XfE4AaaeMOWdIM60B9BoC6lrdA=', 'Content-Length': '0'} 2024-04-12T03:58:45.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aabd2de2a92ce74b-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.993 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.993 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.993 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000007 2024-04-12T03:58:45.993 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000007 2024-04-12T03:58:45.993 DEBUG:boto:Method: GET 2024-04-12T03:58:45.993 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000007 2024-04-12T03:58:45.993 DEBUG:boto:Data: 2024-04-12T03:58:45.993 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.993 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:45.993 DEBUG:boto:Port: 8001 2024-04-12T03:58:45.993 DEBUG:boto:Params: {} 2024-04-12T03:58:45.994 DEBUG:boto:Token: None 2024-04-12T03:58:45.994 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.994 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:45.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8bb5549c2ce5011-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:45.996 DEBUG:boto:b'dkyges-60obj000000000000000000071000falseobj000000000000000000072024-04-12T03:58:35.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:45.997 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:45.997 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:45.997 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000007 2024-04-12T03:58:45.997 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000007 2024-04-12T03:58:45.997 DEBUG:boto:Method: GET 2024-04-12T03:58:45.997 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000007 2024-04-12T03:58:45.997 DEBUG:boto:Data: 2024-04-12T03:58:45.997 DEBUG:boto:Headers: {} 2024-04-12T03:58:45.997 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:45.997 DEBUG:boto:Port: 8000 2024-04-12T03:58:45.997 DEBUG:boto:Params: {} 2024-04-12T03:58:45.998 DEBUG:boto:Token: None 2024-04-12T03:58:45.998 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:45 GMT /dkyges-60/ 2024-04-12T03:58:45.998 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI= 2024-04-12T03:58:45.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0DLUDxcU7Rx8YQe7XbiuE62gXI=', 'Content-Length': '0'} 2024-04-12T03:58:46.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f7855daacb94ad4-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.003 DEBUG:boto:b'dkyges-60obj000000000000000000071000falseobj000000000000000000072024-04-12T03:58:35.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.003 DEBUG:boto:path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:46.003 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:46.003 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.003 DEBUG:boto:Path: /dkyges-60/obj00000000000000000007 2024-04-12T03:58:46.003 DEBUG:boto:Data: 2024-04-12T03:58:46.004 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.004 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.004 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.004 DEBUG:boto:Params: {} 2024-04-12T03:58:46.004 DEBUG:boto:Token: None 2024-04-12T03:58:46.004 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000007 2024-04-12T03:58:46.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5YEyBy5czcpymgHm/i2V8BY8ufM= 2024-04-12T03:58:46.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5YEyBy5czcpymgHm/i2V8BY8ufM=', 'Content-Length': '0'} 2024-04-12T03:58:46.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f359fb96e3c4ca66-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.006 DEBUG:boto:path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:46.006 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:46.007 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.007 DEBUG:boto:Path: /dkyges-60/obj00000000000000000007 2024-04-12T03:58:46.007 DEBUG:boto:Data: 2024-04-12T03:58:46.007 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.007 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.007 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.007 DEBUG:boto:Params: {} 2024-04-12T03:58:46.007 DEBUG:boto:Token: None 2024-04-12T03:58:46.007 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000007 2024-04-12T03:58:46.008 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5YEyBy5czcpymgHm/i2V8BY8ufM= 2024-04-12T03:58:46.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5YEyBy5czcpymgHm/i2V8BY8ufM=', 'Content-Length': '0'} 2024-04-12T03:58:46.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2d3a4c67e81f97f-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2024-04-12T03:58:46.010 DEBUG:boto:path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:46.010 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:46.010 DEBUG:boto:Method: GET 2024-04-12T03:58:46.010 DEBUG:boto:Path: /dkyges-60/obj00000000000000000007 2024-04-12T03:58:46.010 DEBUG:boto:Data: 2024-04-12T03:58:46.010 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.011 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.011 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.011 DEBUG:boto:Params: {} 2024-04-12T03:58:46.011 DEBUG:boto:Token: None 2024-04-12T03:58:46.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000007 2024-04-12T03:58:46.011 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MP/j90c4SdS/e2uCfo5jqVFyYe4= 2024-04-12T03:58:46.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MP/j90c4SdS/e2uCfo5jqVFyYe4=', 'Content-Length': '0'} 2024-04-12T03:58:46.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b46faa69cc7e46e9-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.013 DEBUG:boto:path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:46.013 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000007 2024-04-12T03:58:46.014 DEBUG:boto:Method: GET 2024-04-12T03:58:46.014 DEBUG:boto:Path: /dkyges-60/obj00000000000000000007 2024-04-12T03:58:46.014 DEBUG:boto:Data: 2024-04-12T03:58:46.014 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.014 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.014 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.014 DEBUG:boto:Params: {} 2024-04-12T03:58:46.015 DEBUG:boto:Token: None 2024-04-12T03:58:46.015 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000007 2024-04-12T03:58:46.016 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MP/j90c4SdS/e2uCfo5jqVFyYe4= 2024-04-12T03:58:46.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MP/j90c4SdS/e2uCfo5jqVFyYe4=', 'Content-Length': '0'} 2024-04-12T03:58:46.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2e76f32e4d0a690-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.018 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.018 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.018 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000007 2024-04-12T03:58:46.018 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000007 2024-04-12T03:58:46.018 DEBUG:boto:Method: GET 2024-04-12T03:58:46.018 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000007 2024-04-12T03:58:46.019 DEBUG:boto:Data: 2024-04-12T03:58:46.019 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.019 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.019 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.019 DEBUG:boto:Params: {} 2024-04-12T03:58:46.019 DEBUG:boto:Token: None 2024-04-12T03:58:46.019 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.019 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d11fbb17b00f20d-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.022 DEBUG:boto:b'dkyges-60obj000000000000000000071000falseobj000000000000000000072024-04-12T03:58:35.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.022 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.023 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.023 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000007 2024-04-12T03:58:46.023 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000007 2024-04-12T03:58:46.023 DEBUG:boto:Method: GET 2024-04-12T03:58:46.023 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000007 2024-04-12T03:58:46.023 DEBUG:boto:Data: 2024-04-12T03:58:46.023 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.023 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.023 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.023 DEBUG:boto:Params: {} 2024-04-12T03:58:46.024 DEBUG:boto:Token: None 2024-04-12T03:58:46.024 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.024 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e508718986a4ffa-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.028 DEBUG:boto:b'dkyges-60obj000000000000000000071000falseobj000000000000000000072024-04-12T03:58:35.732Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-04-12T03:58:46.029 DEBUG:boto:path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.029 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.029 DEBUG:boto:path=/dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.029 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.029 DEBUG:boto:Method: GET 2024-04-12T03:58:46.029 DEBUG:boto:Path: /dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.029 DEBUG:boto:Data: 2024-04-12T03:58:46.030 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.030 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.030 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.030 DEBUG:boto:Params: {} 2024-04-12T03:58:46.030 DEBUG:boto:Token: None 2024-04-12T03:58:46.030 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.030 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dh1Lzmdt/duDLqaGWXI4fHkfbos= 2024-04-12T03:58:46.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dh1Lzmdt/duDLqaGWXI4fHkfbos=', 'Content-Length': '0'} 2024-04-12T03:58:46.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006529153443bb6d09-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.033 DEBUG:boto:path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.033 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.033 DEBUG:boto:path=/dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.033 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.033 DEBUG:boto:Method: GET 2024-04-12T03:58:46.033 DEBUG:boto:Path: /dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.033 DEBUG:boto:Data: 2024-04-12T03:58:46.033 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.033 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.033 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.033 DEBUG:boto:Params: {} 2024-04-12T03:58:46.034 DEBUG:boto:Token: None 2024-04-12T03:58:46.034 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.034 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dh1Lzmdt/duDLqaGWXI4fHkfbos= 2024-04-12T03:58:46.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dh1Lzmdt/duDLqaGWXI4fHkfbos=', 'Content-Length': '0'} 2024-04-12T03:58:46.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed8606cdbe3ce5ac-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.078 DEBUG:boto:path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.078 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.079 DEBUG:boto:path=/dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.079 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.079 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.079 DEBUG:boto:Path: /dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.079 DEBUG:boto:Data: 2024-04-12T03:58:46.079 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.079 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.079 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.079 DEBUG:boto:Params: {} 2024-04-12T03:58:46.079 DEBUG:boto:Token: None 2024-04-12T03:58:46.079 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.080 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qr5YeTYG7vo81l/WvN4v+n+neG0= 2024-04-12T03:58:46.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qr5YeTYG7vo81l/WvN4v+n+neG0=', 'Content-Length': '0'} 2024-04-12T03:58:46.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f633016144934ea-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.082 DEBUG:boto:path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.082 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.082 DEBUG:boto:path=/dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.082 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.082 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.082 DEBUG:boto:Path: /dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.083 DEBUG:boto:Data: 2024-04-12T03:58:46.083 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.083 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.083 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.083 DEBUG:boto:Params: {} 2024-04-12T03:58:46.083 DEBUG:boto:Token: None 2024-04-12T03:58:46.083 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.083 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qr5YeTYG7vo81l/WvN4v+n+neG0= 2024-04-12T03:58:46.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qr5YeTYG7vo81l/WvN4v+n+neG0=', 'Content-Length': '0'} 2024-04-12T03:58:46.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dc09a6708b1b3b7-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-04-12T03:58:46.086 DEBUG:boto:path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.086 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.086 DEBUG:boto:path=/dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.086 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.086 DEBUG:boto:Method: GET 2024-04-12T03:58:46.086 DEBUG:boto:Path: /dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.086 DEBUG:boto:Data: 2024-04-12T03:58:46.086 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.086 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.087 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.087 DEBUG:boto:Params: {} 2024-04-12T03:58:46.087 DEBUG:boto:Token: None 2024-04-12T03:58:46.087 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.087 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dh1Lzmdt/duDLqaGWXI4fHkfbos= 2024-04-12T03:58:46.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dh1Lzmdt/duDLqaGWXI4fHkfbos=', 'Content-Length': '0'} 2024-04-12T03:58:46.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028ccbe6d10a17fa7-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.089 DEBUG:boto:path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.089 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.089 DEBUG:boto:path=/dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.090 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.090 DEBUG:boto:Method: GET 2024-04-12T03:58:46.090 DEBUG:boto:Path: /dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.090 DEBUG:boto:Data: 2024-04-12T03:58:46.090 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.090 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.090 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.090 DEBUG:boto:Params: {} 2024-04-12T03:58:46.090 DEBUG:boto:Token: None 2024-04-12T03:58:46.090 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000008?versionId=null 2024-04-12T03:58:46.091 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dh1Lzmdt/duDLqaGWXI4fHkfbos= 2024-04-12T03:58:46.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dh1Lzmdt/duDLqaGWXI4fHkfbos=', 'Content-Length': '0'} 2024-04-12T03:58:46.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f33730898bea652-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.093 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.093 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.093 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000008 2024-04-12T03:58:46.093 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000008 2024-04-12T03:58:46.093 DEBUG:boto:Method: GET 2024-04-12T03:58:46.093 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000008 2024-04-12T03:58:46.093 DEBUG:boto:Data: 2024-04-12T03:58:46.093 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.093 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.093 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.093 DEBUG:boto:Params: {} 2024-04-12T03:58:46.094 DEBUG:boto:Token: None 2024-04-12T03:58:46.094 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.094 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff8338ef82151d9a-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.098 DEBUG:boto:b'dkyges-60obj000000000000000000081000falseobj000000000000000000082024-04-12T03:58:35.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.098 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.098 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.099 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000008 2024-04-12T03:58:46.099 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000008 2024-04-12T03:58:46.099 DEBUG:boto:Method: GET 2024-04-12T03:58:46.099 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000008 2024-04-12T03:58:46.099 DEBUG:boto:Data: 2024-04-12T03:58:46.099 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.099 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.099 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.099 DEBUG:boto:Params: {} 2024-04-12T03:58:46.099 DEBUG:boto:Token: None 2024-04-12T03:58:46.100 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.100 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.104 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bee91c2278ca9a7c-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.104 DEBUG:boto:b'dkyges-60obj000000000000000000081000falseobj000000000000000000082024-04-12T03:58:35.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.105 DEBUG:boto:path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.105 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.105 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.105 DEBUG:boto:Path: /dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.105 DEBUG:boto:Data: 2024-04-12T03:58:46.105 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.105 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.105 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.105 DEBUG:boto:Params: {} 2024-04-12T03:58:46.106 DEBUG:boto:Token: None 2024-04-12T03:58:46.106 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.106 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7JYkqXRMAj9RneJ+mBc8gp1mEn4= 2024-04-12T03:58:46.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7JYkqXRMAj9RneJ+mBc8gp1mEn4=', 'Content-Length': '0'} 2024-04-12T03:58:46.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008241e761e07617d5-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.108 DEBUG:boto:path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.108 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.108 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.108 DEBUG:boto:Path: /dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.108 DEBUG:boto:Data: 2024-04-12T03:58:46.108 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.108 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.108 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.109 DEBUG:boto:Params: {} 2024-04-12T03:58:46.109 DEBUG:boto:Token: None 2024-04-12T03:58:46.109 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.109 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7JYkqXRMAj9RneJ+mBc8gp1mEn4= 2024-04-12T03:58:46.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7JYkqXRMAj9RneJ+mBc8gp1mEn4=', 'Content-Length': '0'} 2024-04-12T03:58:46.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f68f9be6c266716-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2024-04-12T03:58:46.111 DEBUG:boto:path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.111 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.112 DEBUG:boto:Method: GET 2024-04-12T03:58:46.112 DEBUG:boto:Path: /dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.112 DEBUG:boto:Data: 2024-04-12T03:58:46.112 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.112 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.112 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.112 DEBUG:boto:Params: {} 2024-04-12T03:58:46.112 DEBUG:boto:Token: None 2024-04-12T03:58:46.112 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.112 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GqQhhOtIkzdfnP+IyZFu1/NZnJY= 2024-04-12T03:58:46.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GqQhhOtIkzdfnP+IyZFu1/NZnJY=', 'Content-Length': '0'} 2024-04-12T03:58:46.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c37405c65e815634-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.114 DEBUG:boto:path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.115 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.115 DEBUG:boto:Method: GET 2024-04-12T03:58:46.115 DEBUG:boto:Path: /dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.115 DEBUG:boto:Data: 2024-04-12T03:58:46.115 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.115 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.115 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.115 DEBUG:boto:Params: {} 2024-04-12T03:58:46.115 DEBUG:boto:Token: None 2024-04-12T03:58:46.115 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000008 2024-04-12T03:58:46.116 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GqQhhOtIkzdfnP+IyZFu1/NZnJY= 2024-04-12T03:58:46.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GqQhhOtIkzdfnP+IyZFu1/NZnJY=', 'Content-Length': '0'} 2024-04-12T03:58:46.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ff604bda18c01f3-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.158 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.158 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.159 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000008 2024-04-12T03:58:46.159 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000008 2024-04-12T03:58:46.159 DEBUG:boto:Method: GET 2024-04-12T03:58:46.159 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000008 2024-04-12T03:58:46.159 DEBUG:boto:Data: 2024-04-12T03:58:46.159 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.159 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.159 DEBUG:boto:Params: {} 2024-04-12T03:58:46.159 DEBUG:boto:Token: None 2024-04-12T03:58:46.159 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.160 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000530a1a20ac4b8aaa-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.162 DEBUG:boto:b'dkyges-60obj000000000000000000081000falseobj000000000000000000082024-04-12T03:58:35.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.163 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.163 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.163 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000008 2024-04-12T03:58:46.163 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000008 2024-04-12T03:58:46.163 DEBUG:boto:Method: GET 2024-04-12T03:58:46.163 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000008 2024-04-12T03:58:46.163 DEBUG:boto:Data: 2024-04-12T03:58:46.163 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.164 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.164 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.164 DEBUG:boto:Params: {} 2024-04-12T03:58:46.164 DEBUG:boto:Token: None 2024-04-12T03:58:46.164 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.164 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093fa03dfb6432214-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.169 DEBUG:boto:b'dkyges-60obj000000000000000000081000falseobj000000000000000000082024-04-12T03:58:35.044Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-04-12T03:58:46.169 DEBUG:boto:path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.169 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.169 DEBUG:boto:path=/dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.169 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.170 DEBUG:boto:Method: GET 2024-04-12T03:58:46.170 DEBUG:boto:Path: /dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.170 DEBUG:boto:Data: 2024-04-12T03:58:46.170 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.170 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.170 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.170 DEBUG:boto:Params: {} 2024-04-12T03:58:46.170 DEBUG:boto:Token: None 2024-04-12T03:58:46.170 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.171 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8qlqpoUiwddWFTLFRzpkS/oba5E= 2024-04-12T03:58:46.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8qlqpoUiwddWFTLFRzpkS/oba5E=', 'Content-Length': '0'} 2024-04-12T03:58:46.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdd47b259a73e17c-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.173 DEBUG:boto:path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.173 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.173 DEBUG:boto:path=/dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.173 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.173 DEBUG:boto:Method: GET 2024-04-12T03:58:46.173 DEBUG:boto:Path: /dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.173 DEBUG:boto:Data: 2024-04-12T03:58:46.173 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.174 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.174 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.174 DEBUG:boto:Params: {} 2024-04-12T03:58:46.174 DEBUG:boto:Token: None 2024-04-12T03:58:46.174 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.174 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8qlqpoUiwddWFTLFRzpkS/oba5E= 2024-04-12T03:58:46.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8qlqpoUiwddWFTLFRzpkS/oba5E=', 'Content-Length': '0'} 2024-04-12T03:58:46.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ee69fb25264df38-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.176 DEBUG:boto:path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.177 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.177 DEBUG:boto:path=/dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.177 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.177 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.177 DEBUG:boto:Path: /dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.177 DEBUG:boto:Data: 2024-04-12T03:58:46.177 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.177 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.177 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.177 DEBUG:boto:Params: {} 2024-04-12T03:58:46.177 DEBUG:boto:Token: None 2024-04-12T03:58:46.178 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.178 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IEI9peibLbu7vAUbG4zqwDCqVz4= 2024-04-12T03:58:46.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IEI9peibLbu7vAUbG4zqwDCqVz4=', 'Content-Length': '0'} 2024-04-12T03:58:46.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dc7641823dcdf5f-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.180 DEBUG:boto:path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.180 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.180 DEBUG:boto:path=/dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.180 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.180 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.180 DEBUG:boto:Path: /dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.180 DEBUG:boto:Data: 2024-04-12T03:58:46.181 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.181 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.181 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.181 DEBUG:boto:Params: {} 2024-04-12T03:58:46.181 DEBUG:boto:Token: None 2024-04-12T03:58:46.181 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.181 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IEI9peibLbu7vAUbG4zqwDCqVz4= 2024-04-12T03:58:46.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IEI9peibLbu7vAUbG4zqwDCqVz4=', 'Content-Length': '0'} 2024-04-12T03:58:46.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6dda467a7363b5c-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-04-12T03:58:46.183 DEBUG:boto:path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.183 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.184 DEBUG:boto:path=/dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.184 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.184 DEBUG:boto:Method: GET 2024-04-12T03:58:46.184 DEBUG:boto:Path: /dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.184 DEBUG:boto:Data: 2024-04-12T03:58:46.184 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.184 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.184 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.184 DEBUG:boto:Params: {} 2024-04-12T03:58:46.184 DEBUG:boto:Token: None 2024-04-12T03:58:46.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.185 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8qlqpoUiwddWFTLFRzpkS/oba5E= 2024-04-12T03:58:46.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8qlqpoUiwddWFTLFRzpkS/oba5E=', 'Content-Length': '0'} 2024-04-12T03:58:46.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004264d6d3cb159e55-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.187 DEBUG:boto:path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.187 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.187 DEBUG:boto:path=/dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.187 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.187 DEBUG:boto:Method: GET 2024-04-12T03:58:46.187 DEBUG:boto:Path: /dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.187 DEBUG:boto:Data: 2024-04-12T03:58:46.187 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.188 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.188 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.188 DEBUG:boto:Params: {} 2024-04-12T03:58:46.188 DEBUG:boto:Token: None 2024-04-12T03:58:46.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000009?versionId=null 2024-04-12T03:58:46.188 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8qlqpoUiwddWFTLFRzpkS/oba5E= 2024-04-12T03:58:46.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8qlqpoUiwddWFTLFRzpkS/oba5E=', 'Content-Length': '0'} 2024-04-12T03:58:46.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b463d6d7bbcf68a9-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.234 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.234 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.235 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000009 2024-04-12T03:58:46.235 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000009 2024-04-12T03:58:46.235 DEBUG:boto:Method: GET 2024-04-12T03:58:46.235 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000009 2024-04-12T03:58:46.235 DEBUG:boto:Data: 2024-04-12T03:58:46.235 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.235 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.235 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.235 DEBUG:boto:Params: {} 2024-04-12T03:58:46.235 DEBUG:boto:Token: None 2024-04-12T03:58:46.236 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.236 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055bd477a244b75ce-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.238 DEBUG:boto:b'dkyges-60obj000000000000000000091000falseobj000000000000000000092024-04-12T03:58:35.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.239 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.239 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.239 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000009 2024-04-12T03:58:46.239 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000009 2024-04-12T03:58:46.239 DEBUG:boto:Method: GET 2024-04-12T03:58:46.239 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000009 2024-04-12T03:58:46.239 DEBUG:boto:Data: 2024-04-12T03:58:46.239 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.240 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.240 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.240 DEBUG:boto:Params: {} 2024-04-12T03:58:46.240 DEBUG:boto:Token: None 2024-04-12T03:58:46.240 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.240 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d991872bdc400660-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.245 DEBUG:boto:b'dkyges-60obj000000000000000000091000falseobj000000000000000000092024-04-12T03:58:35.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.245 DEBUG:boto:path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.245 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.245 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.245 DEBUG:boto:Path: /dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.245 DEBUG:boto:Data: 2024-04-12T03:58:46.245 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.246 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.246 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.246 DEBUG:boto:Params: {} 2024-04-12T03:58:46.246 DEBUG:boto:Token: None 2024-04-12T03:58:46.246 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.246 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1/3ThHA3c8cfxaXipLAXxhKzGCw= 2024-04-12T03:58:46.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1/3ThHA3c8cfxaXipLAXxhKzGCw=', 'Content-Length': '0'} 2024-04-12T03:58:46.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faf2bb97c02cd57b-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.248 DEBUG:boto:path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.248 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.248 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.249 DEBUG:boto:Path: /dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.249 DEBUG:boto:Data: 2024-04-12T03:58:46.249 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.249 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.249 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.249 DEBUG:boto:Params: {} 2024-04-12T03:58:46.249 DEBUG:boto:Token: None 2024-04-12T03:58:46.249 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.249 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1/3ThHA3c8cfxaXipLAXxhKzGCw= 2024-04-12T03:58:46.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1/3ThHA3c8cfxaXipLAXxhKzGCw=', 'Content-Length': '0'} 2024-04-12T03:58:46.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6f5566a2e26e744-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2024-04-12T03:58:46.252 DEBUG:boto:path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.252 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.252 DEBUG:boto:Method: GET 2024-04-12T03:58:46.252 DEBUG:boto:Path: /dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.252 DEBUG:boto:Data: 2024-04-12T03:58:46.252 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.252 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.252 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.252 DEBUG:boto:Params: {} 2024-04-12T03:58:46.252 DEBUG:boto:Token: None 2024-04-12T03:58:46.253 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.253 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SjRtgK5RXQaeHOvaIpLdSrMXbaU= 2024-04-12T03:58:46.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SjRtgK5RXQaeHOvaIpLdSrMXbaU=', 'Content-Length': '0'} 2024-04-12T03:58:46.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b34ff069bd2481c2-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.298 DEBUG:boto:path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.298 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.299 DEBUG:boto:Method: GET 2024-04-12T03:58:46.299 DEBUG:boto:Path: /dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.299 DEBUG:boto:Data: 2024-04-12T03:58:46.299 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.299 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.299 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.299 DEBUG:boto:Params: {} 2024-04-12T03:58:46.299 DEBUG:boto:Token: None 2024-04-12T03:58:46.299 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000009 2024-04-12T03:58:46.299 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SjRtgK5RXQaeHOvaIpLdSrMXbaU= 2024-04-12T03:58:46.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SjRtgK5RXQaeHOvaIpLdSrMXbaU=', 'Content-Length': '0'} 2024-04-12T03:58:46.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba4877f36e571327-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.302 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.302 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.303 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000009 2024-04-12T03:58:46.303 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000009 2024-04-12T03:58:46.303 DEBUG:boto:Method: GET 2024-04-12T03:58:46.303 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000009 2024-04-12T03:58:46.303 DEBUG:boto:Data: 2024-04-12T03:58:46.303 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.303 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.303 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.303 DEBUG:boto:Params: {} 2024-04-12T03:58:46.303 DEBUG:boto:Token: None 2024-04-12T03:58:46.304 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.304 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce38dfaeb6ea57f5-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.306 DEBUG:boto:b'dkyges-60obj000000000000000000091000falseobj000000000000000000092024-04-12T03:58:35.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.307 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.307 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.307 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000009 2024-04-12T03:58:46.307 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000009 2024-04-12T03:58:46.307 DEBUG:boto:Method: GET 2024-04-12T03:58:46.307 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000009 2024-04-12T03:58:46.307 DEBUG:boto:Data: 2024-04-12T03:58:46.307 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.307 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.308 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.308 DEBUG:boto:Params: {} 2024-04-12T03:58:46.308 DEBUG:boto:Token: None 2024-04-12T03:58:46.308 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.308 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.312 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d018366d0adce872-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.313 DEBUG:boto:b'dkyges-60obj000000000000000000091000falseobj000000000000000000092024-04-12T03:58:35.108Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-12T03:58:46.313 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.313 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.313 DEBUG:boto:path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.313 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.313 DEBUG:boto:Method: GET 2024-04-12T03:58:46.314 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.314 DEBUG:boto:Data: 2024-04-12T03:58:46.314 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.314 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.314 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.314 DEBUG:boto:Params: {} 2024-04-12T03:58:46.314 DEBUG:boto:Token: None 2024-04-12T03:58:46.314 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.314 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lhh7She2gR+VYzd0xmQxIlwZx8M= 2024-04-12T03:58:46.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lhh7She2gR+VYzd0xmQxIlwZx8M=', 'Content-Length': '0'} 2024-04-12T03:58:46.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b621399c26184319-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.317 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.317 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.317 DEBUG:boto:path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.317 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.317 DEBUG:boto:Method: GET 2024-04-12T03:58:46.317 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.317 DEBUG:boto:Data: 2024-04-12T03:58:46.317 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.317 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.317 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.317 DEBUG:boto:Params: {} 2024-04-12T03:58:46.317 DEBUG:boto:Token: None 2024-04-12T03:58:46.318 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.318 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lhh7She2gR+VYzd0xmQxIlwZx8M= 2024-04-12T03:58:46.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lhh7She2gR+VYzd0xmQxIlwZx8M=', 'Content-Length': '0'} 2024-04-12T03:58:46.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7abf69f16eb708c-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.366 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.366 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.367 DEBUG:boto:path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.367 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.367 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.367 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.367 DEBUG:boto:Data: 2024-04-12T03:58:46.367 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.367 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.367 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.367 DEBUG:boto:Params: {} 2024-04-12T03:58:46.367 DEBUG:boto:Token: None 2024-04-12T03:58:46.368 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.368 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PZHKQlxvPrwgXUIuQxNtenEjgZ8= 2024-04-12T03:58:46.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PZHKQlxvPrwgXUIuQxNtenEjgZ8=', 'Content-Length': '0'} 2024-04-12T03:58:46.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bbbd0297fd0bcc6-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.370 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.370 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.370 DEBUG:boto:path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.370 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.370 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.370 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.370 DEBUG:boto:Data: 2024-04-12T03:58:46.370 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.370 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.371 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.371 DEBUG:boto:Params: {} 2024-04-12T03:58:46.371 DEBUG:boto:Token: None 2024-04-12T03:58:46.371 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.371 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PZHKQlxvPrwgXUIuQxNtenEjgZ8= 2024-04-12T03:58:46.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PZHKQlxvPrwgXUIuQxNtenEjgZ8=', 'Content-Length': '0'} 2024-04-12T03:58:46.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac5ca5f2bbb5cd95-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-12T03:58:46.373 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.373 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.373 DEBUG:boto:path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.373 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.374 DEBUG:boto:Method: GET 2024-04-12T03:58:46.374 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.374 DEBUG:boto:Data: 2024-04-12T03:58:46.374 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.374 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.374 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.374 DEBUG:boto:Params: {} 2024-04-12T03:58:46.374 DEBUG:boto:Token: None 2024-04-12T03:58:46.374 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.374 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lhh7She2gR+VYzd0xmQxIlwZx8M= 2024-04-12T03:58:46.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lhh7She2gR+VYzd0xmQxIlwZx8M=', 'Content-Length': '0'} 2024-04-12T03:58:46.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086c856477a11dcf6-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.376 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.377 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.377 DEBUG:boto:path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.377 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.377 DEBUG:boto:Method: GET 2024-04-12T03:58:46.377 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.377 DEBUG:boto:Data: 2024-04-12T03:58:46.377 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.377 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.377 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.377 DEBUG:boto:Params: {} 2024-04-12T03:58:46.377 DEBUG:boto:Token: None 2024-04-12T03:58:46.378 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000010?versionId=null 2024-04-12T03:58:46.378 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lhh7She2gR+VYzd0xmQxIlwZx8M= 2024-04-12T03:58:46.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lhh7She2gR+VYzd0xmQxIlwZx8M=', 'Content-Length': '0'} 2024-04-12T03:58:46.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eeea6da706068d0-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.380 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.380 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.380 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:46.380 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:46.380 DEBUG:boto:Method: GET 2024-04-12T03:58:46.380 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:46.380 DEBUG:boto:Data: 2024-04-12T03:58:46.381 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.381 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.381 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.381 DEBUG:boto:Params: {} 2024-04-12T03:58:46.381 DEBUG:boto:Token: None 2024-04-12T03:58:46.381 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.381 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea0f3d0c74a8cd55-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.384 DEBUG:boto:b'dkyges-60obj000000000000000000101000falseobj000000000000000000102024-04-12T03:58:35.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.384 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.384 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.384 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:46.385 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:46.385 DEBUG:boto:Method: GET 2024-04-12T03:58:46.385 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:46.385 DEBUG:boto:Data: 2024-04-12T03:58:46.385 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.385 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.385 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.385 DEBUG:boto:Params: {} 2024-04-12T03:58:46.385 DEBUG:boto:Token: None 2024-04-12T03:58:46.385 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.386 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074fb91fb8c6da56f-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.390 DEBUG:boto:b'dkyges-60obj000000000000000000101000falseobj000000000000000000102024-04-12T03:58:35.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.391 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.391 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.391 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.391 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.391 DEBUG:boto:Data: 2024-04-12T03:58:46.391 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.391 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.391 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.391 DEBUG:boto:Params: {} 2024-04-12T03:58:46.391 DEBUG:boto:Token: None 2024-04-12T03:58:46.392 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.392 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rdNPGsvtFxvGMnPPffAVZOXibbE= 2024-04-12T03:58:46.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rdNPGsvtFxvGMnPPffAVZOXibbE=', 'Content-Length': '0'} 2024-04-12T03:58:46.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac5a2794f4536e0c-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.394 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.394 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.394 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.394 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.395 DEBUG:boto:Data: 2024-04-12T03:58:46.395 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.395 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.395 DEBUG:boto:Params: {} 2024-04-12T03:58:46.395 DEBUG:boto:Token: None 2024-04-12T03:58:46.395 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.395 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rdNPGsvtFxvGMnPPffAVZOXibbE= 2024-04-12T03:58:46.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rdNPGsvtFxvGMnPPffAVZOXibbE=', 'Content-Length': '0'} 2024-04-12T03:58:46.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8fcda9ffcfa70e6-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2024-04-12T03:58:46.397 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.397 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.397 DEBUG:boto:Method: GET 2024-04-12T03:58:46.397 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.398 DEBUG:boto:Data: 2024-04-12T03:58:46.398 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.398 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.398 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.398 DEBUG:boto:Params: {} 2024-04-12T03:58:46.398 DEBUG:boto:Token: None 2024-04-12T03:58:46.398 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.398 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gFXcsnXq6HSVxNMTQMMxHFutoys= 2024-04-12T03:58:46.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gFXcsnXq6HSVxNMTQMMxHFutoys=', 'Content-Length': '0'} 2024-04-12T03:58:46.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000620b3c0a16fa25b3-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.442 DEBUG:boto:path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.443 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.443 DEBUG:boto:Method: GET 2024-04-12T03:58:46.443 DEBUG:boto:Path: /dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.443 DEBUG:boto:Data: 2024-04-12T03:58:46.443 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.443 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.443 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.443 DEBUG:boto:Params: {} 2024-04-12T03:58:46.443 DEBUG:boto:Token: None 2024-04-12T03:58:46.444 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000010 2024-04-12T03:58:46.444 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gFXcsnXq6HSVxNMTQMMxHFutoys= 2024-04-12T03:58:46.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gFXcsnXq6HSVxNMTQMMxHFutoys=', 'Content-Length': '0'} 2024-04-12T03:58:46.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3b6c7f0fb29eac9-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.446 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.446 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.446 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:46.447 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:46.447 DEBUG:boto:Method: GET 2024-04-12T03:58:46.447 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:46.447 DEBUG:boto:Data: 2024-04-12T03:58:46.447 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.447 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.447 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.447 DEBUG:boto:Params: {} 2024-04-12T03:58:46.447 DEBUG:boto:Token: None 2024-04-12T03:58:46.447 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.448 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c64f863a84b01850-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.450 DEBUG:boto:b'dkyges-60obj000000000000000000101000falseobj000000000000000000102024-04-12T03:58:35.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.451 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.451 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.451 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:46.451 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:46.451 DEBUG:boto:Method: GET 2024-04-12T03:58:46.451 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000010 2024-04-12T03:58:46.451 DEBUG:boto:Data: 2024-04-12T03:58:46.451 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.451 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.452 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.452 DEBUG:boto:Params: {} 2024-04-12T03:58:46.452 DEBUG:boto:Token: None 2024-04-12T03:58:46.452 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.452 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e20375ab1a3f1db-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.456 DEBUG:boto:b'dkyges-60obj000000000000000000101000falseobj000000000000000000102024-04-12T03:58:35.862Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-12T03:58:46.457 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.457 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.457 DEBUG:boto:path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.457 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.457 DEBUG:boto:Method: GET 2024-04-12T03:58:46.457 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.458 DEBUG:boto:Data: 2024-04-12T03:58:46.458 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.458 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.458 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.458 DEBUG:boto:Params: {} 2024-04-12T03:58:46.458 DEBUG:boto:Token: None 2024-04-12T03:58:46.458 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.458 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nBjoyqR13DilxNtuhefeGsWbmx8= 2024-04-12T03:58:46.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nBjoyqR13DilxNtuhefeGsWbmx8=', 'Content-Length': '0'} 2024-04-12T03:58:46.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000869986e3f869ff83-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.461 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.461 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.461 DEBUG:boto:path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.461 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.461 DEBUG:boto:Method: GET 2024-04-12T03:58:46.461 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.461 DEBUG:boto:Data: 2024-04-12T03:58:46.462 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.462 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.462 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.462 DEBUG:boto:Params: {} 2024-04-12T03:58:46.462 DEBUG:boto:Token: None 2024-04-12T03:58:46.462 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.462 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nBjoyqR13DilxNtuhefeGsWbmx8= 2024-04-12T03:58:46.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nBjoyqR13DilxNtuhefeGsWbmx8=', 'Content-Length': '0'} 2024-04-12T03:58:46.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de71d9fc8f78eecb-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.506 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.506 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.507 DEBUG:boto:path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.507 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.507 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.507 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.507 DEBUG:boto:Data: 2024-04-12T03:58:46.507 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.507 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.507 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.507 DEBUG:boto:Params: {} 2024-04-12T03:58:46.507 DEBUG:boto:Token: None 2024-04-12T03:58:46.508 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.508 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+MdHdlj6PxBpcit4FIAj4HybTGQ= 2024-04-12T03:58:46.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+MdHdlj6PxBpcit4FIAj4HybTGQ=', 'Content-Length': '0'} 2024-04-12T03:58:46.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e588b40dcdaab1fa-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.510 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.510 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.510 DEBUG:boto:path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.510 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.510 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.510 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.511 DEBUG:boto:Data: 2024-04-12T03:58:46.511 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.511 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.511 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.511 DEBUG:boto:Params: {} 2024-04-12T03:58:46.511 DEBUG:boto:Token: None 2024-04-12T03:58:46.511 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.511 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+MdHdlj6PxBpcit4FIAj4HybTGQ= 2024-04-12T03:58:46.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+MdHdlj6PxBpcit4FIAj4HybTGQ=', 'Content-Length': '0'} 2024-04-12T03:58:46.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cde0fe5d2536818c-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-12T03:58:46.513 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.514 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.514 DEBUG:boto:path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.514 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.514 DEBUG:boto:Method: GET 2024-04-12T03:58:46.514 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.514 DEBUG:boto:Data: 2024-04-12T03:58:46.514 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.514 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.514 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.515 DEBUG:boto:Params: {} 2024-04-12T03:58:46.515 DEBUG:boto:Token: None 2024-04-12T03:58:46.515 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.515 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nBjoyqR13DilxNtuhefeGsWbmx8= 2024-04-12T03:58:46.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nBjoyqR13DilxNtuhefeGsWbmx8=', 'Content-Length': '0'} 2024-04-12T03:58:46.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066d4c3921995efec-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.518 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.518 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.518 DEBUG:boto:path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.518 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.518 DEBUG:boto:Method: GET 2024-04-12T03:58:46.518 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.518 DEBUG:boto:Data: 2024-04-12T03:58:46.518 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.518 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.518 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.519 DEBUG:boto:Params: {} 2024-04-12T03:58:46.519 DEBUG:boto:Token: None 2024-04-12T03:58:46.519 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000011?versionId=null 2024-04-12T03:58:46.519 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nBjoyqR13DilxNtuhefeGsWbmx8= 2024-04-12T03:58:46.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nBjoyqR13DilxNtuhefeGsWbmx8=', 'Content-Length': '0'} 2024-04-12T03:58:46.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058a6418c339b39a4-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.521 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.521 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.521 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:46.521 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:46.521 DEBUG:boto:Method: GET 2024-04-12T03:58:46.522 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:46.522 DEBUG:boto:Data: 2024-04-12T03:58:46.522 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.522 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.522 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.522 DEBUG:boto:Params: {} 2024-04-12T03:58:46.522 DEBUG:boto:Token: None 2024-04-12T03:58:46.522 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.522 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d20bc41c3cc4adae-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.525 DEBUG:boto:b'dkyges-60obj000000000000000000111000falseobj000000000000000000112024-04-12T03:58:35.596Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.526 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.526 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.526 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:46.526 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:46.526 DEBUG:boto:Method: GET 2024-04-12T03:58:46.526 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:46.526 DEBUG:boto:Data: 2024-04-12T03:58:46.527 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.527 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.527 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.527 DEBUG:boto:Params: {} 2024-04-12T03:58:46.527 DEBUG:boto:Token: None 2024-04-12T03:58:46.527 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.527 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e70c58261ab2faf-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.532 DEBUG:boto:b'dkyges-60obj000000000000000000111000falseobj000000000000000000112024-04-12T03:58:35.596Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.533 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.533 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.533 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.533 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.533 DEBUG:boto:Data: 2024-04-12T03:58:46.533 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.533 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.533 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.533 DEBUG:boto:Params: {} 2024-04-12T03:58:46.534 DEBUG:boto:Token: None 2024-04-12T03:58:46.534 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.534 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KYXWYo5d+9sGrQbU5HB7pn/up5g= 2024-04-12T03:58:46.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KYXWYo5d+9sGrQbU5HB7pn/up5g=', 'Content-Length': '0'} 2024-04-12T03:58:46.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066baec5ad8fc5346-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.536 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.536 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.536 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.537 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.537 DEBUG:boto:Data: 2024-04-12T03:58:46.537 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.537 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.537 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.537 DEBUG:boto:Params: {} 2024-04-12T03:58:46.537 DEBUG:boto:Token: None 2024-04-12T03:58:46.537 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.537 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KYXWYo5d+9sGrQbU5HB7pn/up5g= 2024-04-12T03:58:46.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KYXWYo5d+9sGrQbU5HB7pn/up5g=', 'Content-Length': '0'} 2024-04-12T03:58:46.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033791ac9d9b7d47c-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2024-04-12T03:58:46.540 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.541 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.541 DEBUG:boto:Method: GET 2024-04-12T03:58:46.541 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.541 DEBUG:boto:Data: 2024-04-12T03:58:46.541 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.541 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.541 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.541 DEBUG:boto:Params: {} 2024-04-12T03:58:46.541 DEBUG:boto:Token: None 2024-04-12T03:58:46.542 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.543 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:p3Gz0BG60ejN0ETptioqi79sYgc= 2024-04-12T03:58:46.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p3Gz0BG60ejN0ETptioqi79sYgc=', 'Content-Length': '0'} 2024-04-12T03:58:46.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ddbb857b0d9c8b8-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.545 DEBUG:boto:path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.546 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.546 DEBUG:boto:Method: GET 2024-04-12T03:58:46.546 DEBUG:boto:Path: /dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.546 DEBUG:boto:Data: 2024-04-12T03:58:46.546 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.546 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.546 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.546 DEBUG:boto:Params: {} 2024-04-12T03:58:46.546 DEBUG:boto:Token: None 2024-04-12T03:58:46.546 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000011 2024-04-12T03:58:46.547 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:p3Gz0BG60ejN0ETptioqi79sYgc= 2024-04-12T03:58:46.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p3Gz0BG60ejN0ETptioqi79sYgc=', 'Content-Length': '0'} 2024-04-12T03:58:46.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027d73936a67e196a-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.590 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.590 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.591 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:46.591 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:46.591 DEBUG:boto:Method: GET 2024-04-12T03:58:46.591 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:46.591 DEBUG:boto:Data: 2024-04-12T03:58:46.591 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.591 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.591 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.591 DEBUG:boto:Params: {} 2024-04-12T03:58:46.591 DEBUG:boto:Token: None 2024-04-12T03:58:46.591 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.592 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e29fde7951338c8-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.594 DEBUG:boto:b'dkyges-60obj000000000000000000111000falseobj000000000000000000112024-04-12T03:58:35.596Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.595 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.595 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.595 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:46.595 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:46.595 DEBUG:boto:Method: GET 2024-04-12T03:58:46.595 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000011 2024-04-12T03:58:46.595 DEBUG:boto:Data: 2024-04-12T03:58:46.595 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.595 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.595 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.595 DEBUG:boto:Params: {} 2024-04-12T03:58:46.596 DEBUG:boto:Token: None 2024-04-12T03:58:46.596 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.596 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060ceb5d342cd3388-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.600 DEBUG:boto:b'dkyges-60obj000000000000000000111000falseobj000000000000000000112024-04-12T03:58:35.596Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-12T03:58:46.601 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.601 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.601 DEBUG:boto:path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.601 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.601 DEBUG:boto:Method: GET 2024-04-12T03:58:46.601 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.602 DEBUG:boto:Data: 2024-04-12T03:58:46.602 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.602 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.602 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.602 DEBUG:boto:Params: {} 2024-04-12T03:58:46.602 DEBUG:boto:Token: None 2024-04-12T03:58:46.602 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.602 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7IC4r9T97EYpKcvyNT3pIT3F7EU= 2024-04-12T03:58:46.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7IC4r9T97EYpKcvyNT3pIT3F7EU=', 'Content-Length': '0'} 2024-04-12T03:58:46.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ae975273e6f712c-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.605 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.606 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.606 DEBUG:boto:path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.606 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.606 DEBUG:boto:Method: GET 2024-04-12T03:58:46.606 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.606 DEBUG:boto:Data: 2024-04-12T03:58:46.606 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.606 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.606 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.606 DEBUG:boto:Params: {} 2024-04-12T03:58:46.606 DEBUG:boto:Token: None 2024-04-12T03:58:46.607 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.607 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7IC4r9T97EYpKcvyNT3pIT3F7EU= 2024-04-12T03:58:46.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7IC4r9T97EYpKcvyNT3pIT3F7EU=', 'Content-Length': '0'} 2024-04-12T03:58:46.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008581a8b3f0d13484-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.609 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.609 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.609 DEBUG:boto:path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.609 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.609 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.609 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.609 DEBUG:boto:Data: 2024-04-12T03:58:46.610 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.610 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.610 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.610 DEBUG:boto:Params: {} 2024-04-12T03:58:46.610 DEBUG:boto:Token: None 2024-04-12T03:58:46.610 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.610 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FmlyZvgB8XW/6tAWcq/aqvaDGjE= 2024-04-12T03:58:46.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FmlyZvgB8XW/6tAWcq/aqvaDGjE=', 'Content-Length': '0'} 2024-04-12T03:58:46.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000312bb45693eb1a3c-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.613 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.613 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.613 DEBUG:boto:path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.613 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.613 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.613 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.613 DEBUG:boto:Data: 2024-04-12T03:58:46.613 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.613 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.613 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.614 DEBUG:boto:Params: {} 2024-04-12T03:58:46.614 DEBUG:boto:Token: None 2024-04-12T03:58:46.614 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.614 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FmlyZvgB8XW/6tAWcq/aqvaDGjE= 2024-04-12T03:58:46.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FmlyZvgB8XW/6tAWcq/aqvaDGjE=', 'Content-Length': '0'} 2024-04-12T03:58:46.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004235b8c321147be0-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-12T03:58:46.616 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.616 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.616 DEBUG:boto:path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.617 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.617 DEBUG:boto:Method: GET 2024-04-12T03:58:46.617 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.617 DEBUG:boto:Data: 2024-04-12T03:58:46.617 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.617 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.617 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.617 DEBUG:boto:Params: {} 2024-04-12T03:58:46.617 DEBUG:boto:Token: None 2024-04-12T03:58:46.617 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.617 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7IC4r9T97EYpKcvyNT3pIT3F7EU= 2024-04-12T03:58:46.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7IC4r9T97EYpKcvyNT3pIT3F7EU=', 'Content-Length': '0'} 2024-04-12T03:58:46.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5214bed857dd5d8-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.619 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.620 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.620 DEBUG:boto:path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.620 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.620 DEBUG:boto:Method: GET 2024-04-12T03:58:46.620 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.620 DEBUG:boto:Data: 2024-04-12T03:58:46.620 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.620 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.620 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.620 DEBUG:boto:Params: {} 2024-04-12T03:58:46.621 DEBUG:boto:Token: None 2024-04-12T03:58:46.621 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000012?versionId=null 2024-04-12T03:58:46.621 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7IC4r9T97EYpKcvyNT3pIT3F7EU= 2024-04-12T03:58:46.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7IC4r9T97EYpKcvyNT3pIT3F7EU=', 'Content-Length': '0'} 2024-04-12T03:58:46.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b175151e35c2dcb-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.623 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.624 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.624 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:46.624 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:46.624 DEBUG:boto:Method: GET 2024-04-12T03:58:46.624 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:46.624 DEBUG:boto:Data: 2024-04-12T03:58:46.624 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.624 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.624 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.624 DEBUG:boto:Params: {} 2024-04-12T03:58:46.624 DEBUG:boto:Token: None 2024-04-12T03:58:46.625 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.625 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efb3750c9ec9031e-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.674 DEBUG:boto:b'dkyges-60obj000000000000000000121000falseobj000000000000000000122024-04-12T03:58:35.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.675 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.675 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.675 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:46.675 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:46.675 DEBUG:boto:Method: GET 2024-04-12T03:58:46.675 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:46.676 DEBUG:boto:Data: 2024-04-12T03:58:46.676 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.676 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.676 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.676 DEBUG:boto:Params: {} 2024-04-12T03:58:46.676 DEBUG:boto:Token: None 2024-04-12T03:58:46.676 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.676 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017225c3b45217514-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.682 DEBUG:boto:b'dkyges-60obj000000000000000000121000falseobj000000000000000000122024-04-12T03:58:35.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.682 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.682 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.682 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.682 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.682 DEBUG:boto:Data: 2024-04-12T03:58:46.682 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.683 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.683 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.683 DEBUG:boto:Params: {} 2024-04-12T03:58:46.683 DEBUG:boto:Token: None 2024-04-12T03:58:46.683 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.683 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iVvLUHNN4S91gpY6vH+J/okkLk4= 2024-04-12T03:58:46.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iVvLUHNN4S91gpY6vH+J/okkLk4=', 'Content-Length': '0'} 2024-04-12T03:58:46.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000855ce18b9445e929-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.685 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.686 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.686 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.686 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.686 DEBUG:boto:Data: 2024-04-12T03:58:46.686 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.686 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.686 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.686 DEBUG:boto:Params: {} 2024-04-12T03:58:46.686 DEBUG:boto:Token: None 2024-04-12T03:58:46.686 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.687 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iVvLUHNN4S91gpY6vH+J/okkLk4= 2024-04-12T03:58:46.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iVvLUHNN4S91gpY6vH+J/okkLk4=', 'Content-Length': '0'} 2024-04-12T03:58:46.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036a5a7ab90aff56c-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2024-04-12T03:58:46.689 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.689 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.689 DEBUG:boto:Method: GET 2024-04-12T03:58:46.689 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.689 DEBUG:boto:Data: 2024-04-12T03:58:46.689 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.689 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.689 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.690 DEBUG:boto:Params: {} 2024-04-12T03:58:46.690 DEBUG:boto:Token: None 2024-04-12T03:58:46.690 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.690 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ehAHRZYt7Pco2KKyTkYRoYAdw3I= 2024-04-12T03:58:46.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ehAHRZYt7Pco2KKyTkYRoYAdw3I=', 'Content-Length': '0'} 2024-04-12T03:58:46.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026b5a25222918cb4-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.692 DEBUG:boto:path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.692 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.692 DEBUG:boto:Method: GET 2024-04-12T03:58:46.692 DEBUG:boto:Path: /dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.692 DEBUG:boto:Data: 2024-04-12T03:58:46.693 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.693 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.693 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.693 DEBUG:boto:Params: {} 2024-04-12T03:58:46.693 DEBUG:boto:Token: None 2024-04-12T03:58:46.693 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000012 2024-04-12T03:58:46.693 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ehAHRZYt7Pco2KKyTkYRoYAdw3I= 2024-04-12T03:58:46.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ehAHRZYt7Pco2KKyTkYRoYAdw3I=', 'Content-Length': '0'} 2024-04-12T03:58:46.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000115173992fb308a3-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.742 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.743 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.743 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:46.743 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:46.743 DEBUG:boto:Method: GET 2024-04-12T03:58:46.743 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:46.743 DEBUG:boto:Data: 2024-04-12T03:58:46.743 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.743 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.743 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.743 DEBUG:boto:Params: {} 2024-04-12T03:58:46.743 DEBUG:boto:Token: None 2024-04-12T03:58:46.744 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.744 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074fc9c0414dce565-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.747 DEBUG:boto:b'dkyges-60obj000000000000000000121000falseobj000000000000000000122024-04-12T03:58:35.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.747 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.747 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.747 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:46.748 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:46.748 DEBUG:boto:Method: GET 2024-04-12T03:58:46.748 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000012 2024-04-12T03:58:46.748 DEBUG:boto:Data: 2024-04-12T03:58:46.748 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.748 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.748 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.748 DEBUG:boto:Params: {} 2024-04-12T03:58:46.748 DEBUG:boto:Token: None 2024-04-12T03:58:46.748 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.749 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.753 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045c7ff530378952a-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.754 DEBUG:boto:b'dkyges-60obj000000000000000000121000falseobj000000000000000000122024-04-12T03:58:35.716Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-12T03:58:46.754 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.754 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.754 DEBUG:boto:path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.754 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.754 DEBUG:boto:Method: GET 2024-04-12T03:58:46.755 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.755 DEBUG:boto:Data: 2024-04-12T03:58:46.755 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.755 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.755 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.755 DEBUG:boto:Params: {} 2024-04-12T03:58:46.755 DEBUG:boto:Token: None 2024-04-12T03:58:46.755 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.755 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0FNGpJtiEoPVbk3AySZ44Ub95GE= 2024-04-12T03:58:46.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0FNGpJtiEoPVbk3AySZ44Ub95GE=', 'Content-Length': '0'} 2024-04-12T03:58:46.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006147af344d61ed0e-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.758 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.758 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.758 DEBUG:boto:path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.758 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.758 DEBUG:boto:Method: GET 2024-04-12T03:58:46.759 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.759 DEBUG:boto:Data: 2024-04-12T03:58:46.759 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.759 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.759 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.759 DEBUG:boto:Params: {} 2024-04-12T03:58:46.759 DEBUG:boto:Token: None 2024-04-12T03:58:46.759 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.759 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0FNGpJtiEoPVbk3AySZ44Ub95GE= 2024-04-12T03:58:46.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0FNGpJtiEoPVbk3AySZ44Ub95GE=', 'Content-Length': '0'} 2024-04-12T03:58:46.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000710fd72c49b1b494-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.762 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.762 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.762 DEBUG:boto:path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.762 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.762 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.762 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.762 DEBUG:boto:Data: 2024-04-12T03:58:46.762 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.763 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.763 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.763 DEBUG:boto:Params: {} 2024-04-12T03:58:46.763 DEBUG:boto:Token: None 2024-04-12T03:58:46.763 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.763 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T9RK7hT0LnyUOYq0TYTJhFsp9d0= 2024-04-12T03:58:46.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T9RK7hT0LnyUOYq0TYTJhFsp9d0=', 'Content-Length': '0'} 2024-04-12T03:58:46.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e31d482ba27de15-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.765 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.765 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.765 DEBUG:boto:path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.766 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.766 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.766 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.766 DEBUG:boto:Data: 2024-04-12T03:58:46.766 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.766 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.766 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.766 DEBUG:boto:Params: {} 2024-04-12T03:58:46.766 DEBUG:boto:Token: None 2024-04-12T03:58:46.766 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.767 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T9RK7hT0LnyUOYq0TYTJhFsp9d0= 2024-04-12T03:58:46.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T9RK7hT0LnyUOYq0TYTJhFsp9d0=', 'Content-Length': '0'} 2024-04-12T03:58:46.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9c7ec8de1daccfb-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-12T03:58:46.769 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.769 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.769 DEBUG:boto:path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.769 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.769 DEBUG:boto:Method: GET 2024-04-12T03:58:46.769 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.769 DEBUG:boto:Data: 2024-04-12T03:58:46.769 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.770 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.770 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.770 DEBUG:boto:Params: {} 2024-04-12T03:58:46.770 DEBUG:boto:Token: None 2024-04-12T03:58:46.770 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.770 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0FNGpJtiEoPVbk3AySZ44Ub95GE= 2024-04-12T03:58:46.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0FNGpJtiEoPVbk3AySZ44Ub95GE=', 'Content-Length': '0'} 2024-04-12T03:58:46.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db12107d91ac6df9-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.772 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.773 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.773 DEBUG:boto:path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.773 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.773 DEBUG:boto:Method: GET 2024-04-12T03:58:46.773 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.773 DEBUG:boto:Data: 2024-04-12T03:58:46.773 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.773 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.773 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.773 DEBUG:boto:Params: {} 2024-04-12T03:58:46.773 DEBUG:boto:Token: None 2024-04-12T03:58:46.774 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000013?versionId=null 2024-04-12T03:58:46.774 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0FNGpJtiEoPVbk3AySZ44Ub95GE= 2024-04-12T03:58:46.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0FNGpJtiEoPVbk3AySZ44Ub95GE=', 'Content-Length': '0'} 2024-04-12T03:58:46.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058c501f778f599b1-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.776 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.776 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.776 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:46.776 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:46.776 DEBUG:boto:Method: GET 2024-04-12T03:58:46.776 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:46.776 DEBUG:boto:Data: 2024-04-12T03:58:46.776 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.777 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.777 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.777 DEBUG:boto:Params: {} 2024-04-12T03:58:46.777 DEBUG:boto:Token: None 2024-04-12T03:58:46.777 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.777 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009886fb77a9b580af-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.780 DEBUG:boto:b'dkyges-60obj000000000000000000131000falseobj000000000000000000132024-04-12T03:58:34.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.780 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.781 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.781 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:46.781 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:46.781 DEBUG:boto:Method: GET 2024-04-12T03:58:46.781 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:46.781 DEBUG:boto:Data: 2024-04-12T03:58:46.781 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.781 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.781 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.781 DEBUG:boto:Params: {} 2024-04-12T03:58:46.781 DEBUG:boto:Token: None 2024-04-12T03:58:46.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.782 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078eaf248ade72860-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.786 DEBUG:boto:b'dkyges-60obj000000000000000000131000falseobj000000000000000000132024-04-12T03:58:34.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.787 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.787 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.787 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.787 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.787 DEBUG:boto:Data: 2024-04-12T03:58:46.787 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.787 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.787 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.788 DEBUG:boto:Params: {} 2024-04-12T03:58:46.788 DEBUG:boto:Token: None 2024-04-12T03:58:46.788 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.788 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SJUhv1MHzmgcaIlRpbsfohL/mRc= 2024-04-12T03:58:46.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SJUhv1MHzmgcaIlRpbsfohL/mRc=', 'Content-Length': '0'} 2024-04-12T03:58:46.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000210ae0599df34b8f-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.790 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.790 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.791 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.791 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.791 DEBUG:boto:Data: 2024-04-12T03:58:46.791 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.791 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.791 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.791 DEBUG:boto:Params: {} 2024-04-12T03:58:46.791 DEBUG:boto:Token: None 2024-04-12T03:58:46.791 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.791 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SJUhv1MHzmgcaIlRpbsfohL/mRc= 2024-04-12T03:58:46.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SJUhv1MHzmgcaIlRpbsfohL/mRc=', 'Content-Length': '0'} 2024-04-12T03:58:46.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce712a8c157f54b6-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2024-04-12T03:58:46.793 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.794 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.794 DEBUG:boto:Method: GET 2024-04-12T03:58:46.794 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.794 DEBUG:boto:Data: 2024-04-12T03:58:46.794 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.794 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.794 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.794 DEBUG:boto:Params: {} 2024-04-12T03:58:46.794 DEBUG:boto:Token: None 2024-04-12T03:58:46.794 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.795 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FTBUG98N8AsVAhIVjEK3Re6V+ok= 2024-04-12T03:58:46.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FTBUG98N8AsVAhIVjEK3Re6V+ok=', 'Content-Length': '0'} 2024-04-12T03:58:46.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9b5bc55adec4522-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.838 DEBUG:boto:path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.838 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.838 DEBUG:boto:Method: GET 2024-04-12T03:58:46.839 DEBUG:boto:Path: /dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.839 DEBUG:boto:Data: 2024-04-12T03:58:46.839 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.839 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.839 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.839 DEBUG:boto:Params: {} 2024-04-12T03:58:46.839 DEBUG:boto:Token: None 2024-04-12T03:58:46.839 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000013 2024-04-12T03:58:46.839 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FTBUG98N8AsVAhIVjEK3Re6V+ok= 2024-04-12T03:58:46.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FTBUG98N8AsVAhIVjEK3Re6V+ok=', 'Content-Length': '0'} 2024-04-12T03:58:46.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000899ea6857714cbcb-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.842 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.842 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.842 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:46.842 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:46.842 DEBUG:boto:Method: GET 2024-04-12T03:58:46.842 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:46.842 DEBUG:boto:Data: 2024-04-12T03:58:46.842 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.842 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.843 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.843 DEBUG:boto:Params: {} 2024-04-12T03:58:46.843 DEBUG:boto:Token: None 2024-04-12T03:58:46.843 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.843 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d71bbf66e0fa75e8-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.846 DEBUG:boto:b'dkyges-60obj000000000000000000131000falseobj000000000000000000132024-04-12T03:58:34.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.847 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.847 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.847 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:46.847 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:46.847 DEBUG:boto:Method: GET 2024-04-12T03:58:46.847 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000013 2024-04-12T03:58:46.847 DEBUG:boto:Data: 2024-04-12T03:58:46.847 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.847 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.847 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.848 DEBUG:boto:Params: {} 2024-04-12T03:58:46.848 DEBUG:boto:Token: None 2024-04-12T03:58:46.848 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.849 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082c71734ae6ee45c-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.854 DEBUG:boto:b'dkyges-60obj000000000000000000131000falseobj000000000000000000132024-04-12T03:58:34.991Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-04-12T03:58:46.854 DEBUG:boto:path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.854 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.854 DEBUG:boto:path=/dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.854 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.855 DEBUG:boto:Method: GET 2024-04-12T03:58:46.855 DEBUG:boto:Path: /dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.855 DEBUG:boto:Data: 2024-04-12T03:58:46.855 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.855 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.855 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.856 DEBUG:boto:Params: {} 2024-04-12T03:58:46.856 DEBUG:boto:Token: None 2024-04-12T03:58:46.856 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.856 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:deaRjEV48bCsGNXHDEZlw1is+oQ= 2024-04-12T03:58:46.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:deaRjEV48bCsGNXHDEZlw1is+oQ=', 'Content-Length': '0'} 2024-04-12T03:58:46.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba734c5ce56a7652-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.858 DEBUG:boto:path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.858 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.858 DEBUG:boto:path=/dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.859 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.859 DEBUG:boto:Method: GET 2024-04-12T03:58:46.859 DEBUG:boto:Path: /dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.859 DEBUG:boto:Data: 2024-04-12T03:58:46.859 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.859 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.859 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.859 DEBUG:boto:Params: {} 2024-04-12T03:58:46.859 DEBUG:boto:Token: None 2024-04-12T03:58:46.859 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.859 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:deaRjEV48bCsGNXHDEZlw1is+oQ= 2024-04-12T03:58:46.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:deaRjEV48bCsGNXHDEZlw1is+oQ=', 'Content-Length': '0'} 2024-04-12T03:58:46.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beeabf1fcfab7045-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.862 DEBUG:boto:path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.862 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.862 DEBUG:boto:path=/dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.862 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.862 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.862 DEBUG:boto:Path: /dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.862 DEBUG:boto:Data: 2024-04-12T03:58:46.862 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.862 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.862 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.863 DEBUG:boto:Params: {} 2024-04-12T03:58:46.863 DEBUG:boto:Token: None 2024-04-12T03:58:46.863 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.863 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sB1ytAYJqWsePw0luUBm8etuwNg= 2024-04-12T03:58:46.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sB1ytAYJqWsePw0luUBm8etuwNg=', 'Content-Length': '0'} 2024-04-12T03:58:46.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056220a5c5125b32f-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.865 DEBUG:boto:path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.865 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.865 DEBUG:boto:path=/dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.865 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.865 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.865 DEBUG:boto:Path: /dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.865 DEBUG:boto:Data: 2024-04-12T03:58:46.866 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.866 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.866 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.866 DEBUG:boto:Params: {} 2024-04-12T03:58:46.866 DEBUG:boto:Token: None 2024-04-12T03:58:46.866 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.866 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sB1ytAYJqWsePw0luUBm8etuwNg= 2024-04-12T03:58:46.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sB1ytAYJqWsePw0luUBm8etuwNg=', 'Content-Length': '0'} 2024-04-12T03:58:46.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000299ec7f7c83a2207-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-04-12T03:58:46.868 DEBUG:boto:path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.868 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.868 DEBUG:boto:path=/dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.869 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.869 DEBUG:boto:Method: GET 2024-04-12T03:58:46.869 DEBUG:boto:Path: /dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.869 DEBUG:boto:Data: 2024-04-12T03:58:46.869 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.869 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.869 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.869 DEBUG:boto:Params: {} 2024-04-12T03:58:46.869 DEBUG:boto:Token: None 2024-04-12T03:58:46.869 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.869 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:deaRjEV48bCsGNXHDEZlw1is+oQ= 2024-04-12T03:58:46.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:deaRjEV48bCsGNXHDEZlw1is+oQ=', 'Content-Length': '0'} 2024-04-12T03:58:46.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e68d9f6c1d8817d7-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.871 DEBUG:boto:path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.872 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.872 DEBUG:boto:path=/dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.872 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.872 DEBUG:boto:Method: GET 2024-04-12T03:58:46.872 DEBUG:boto:Path: /dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.872 DEBUG:boto:Data: 2024-04-12T03:58:46.872 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.872 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.872 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.872 DEBUG:boto:Params: {} 2024-04-12T03:58:46.872 DEBUG:boto:Token: None 2024-04-12T03:58:46.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000014?versionId=null 2024-04-12T03:58:46.873 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:deaRjEV48bCsGNXHDEZlw1is+oQ= 2024-04-12T03:58:46.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:deaRjEV48bCsGNXHDEZlw1is+oQ=', 'Content-Length': '0'} 2024-04-12T03:58:46.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff932119aa7bb7ce-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.875 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.875 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.875 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000014 2024-04-12T03:58:46.875 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000014 2024-04-12T03:58:46.875 DEBUG:boto:Method: GET 2024-04-12T03:58:46.875 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000014 2024-04-12T03:58:46.875 DEBUG:boto:Data: 2024-04-12T03:58:46.875 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.876 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.876 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.876 DEBUG:boto:Params: {} 2024-04-12T03:58:46.876 DEBUG:boto:Token: None 2024-04-12T03:58:46.876 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.876 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf3256ba26f1a4c8-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.881 DEBUG:boto:b'dkyges-60obj000000000000000000141000falseobj000000000000000000142024-04-12T03:58:35.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.882 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.882 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.882 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000014 2024-04-12T03:58:46.882 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000014 2024-04-12T03:58:46.882 DEBUG:boto:Method: GET 2024-04-12T03:58:46.882 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000014 2024-04-12T03:58:46.882 DEBUG:boto:Data: 2024-04-12T03:58:46.882 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.882 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.882 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.883 DEBUG:boto:Params: {} 2024-04-12T03:58:46.883 DEBUG:boto:Token: None 2024-04-12T03:58:46.883 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.883 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5ed53d828851ff2-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.887 DEBUG:boto:b'dkyges-60obj000000000000000000141000falseobj000000000000000000142024-04-12T03:58:35.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.887 DEBUG:boto:path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.888 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.888 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.888 DEBUG:boto:Path: /dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.888 DEBUG:boto:Data: 2024-04-12T03:58:46.888 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.888 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.888 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.888 DEBUG:boto:Params: {} 2024-04-12T03:58:46.888 DEBUG:boto:Token: None 2024-04-12T03:58:46.888 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.889 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UskphJx4PFF8sKI7YWfHM+zDycE= 2024-04-12T03:58:46.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UskphJx4PFF8sKI7YWfHM+zDycE=', 'Content-Length': '0'} 2024-04-12T03:58:46.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000118cd1b22203071a-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.891 DEBUG:boto:path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.891 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.891 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.891 DEBUG:boto:Path: /dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.891 DEBUG:boto:Data: 2024-04-12T03:58:46.891 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.891 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.891 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.891 DEBUG:boto:Params: {} 2024-04-12T03:58:46.891 DEBUG:boto:Token: None 2024-04-12T03:58:46.892 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.892 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UskphJx4PFF8sKI7YWfHM+zDycE= 2024-04-12T03:58:46.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UskphJx4PFF8sKI7YWfHM+zDycE=', 'Content-Length': '0'} 2024-04-12T03:58:46.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002730f0975f437c24-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2024-04-12T03:58:46.894 DEBUG:boto:path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.894 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.894 DEBUG:boto:Method: GET 2024-04-12T03:58:46.894 DEBUG:boto:Path: /dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.894 DEBUG:boto:Data: 2024-04-12T03:58:46.894 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.894 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.894 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.895 DEBUG:boto:Params: {} 2024-04-12T03:58:46.895 DEBUG:boto:Token: None 2024-04-12T03:58:46.895 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.895 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kJXmuvXeBe2sj3j3vumOfdk5Abg= 2024-04-12T03:58:46.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kJXmuvXeBe2sj3j3vumOfdk5Abg=', 'Content-Length': '0'} 2024-04-12T03:58:46.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0885e70cfc2e886-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.938 DEBUG:boto:path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.938 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.939 DEBUG:boto:Method: GET 2024-04-12T03:58:46.939 DEBUG:boto:Path: /dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.939 DEBUG:boto:Data: 2024-04-12T03:58:46.939 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.939 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.939 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.939 DEBUG:boto:Params: {} 2024-04-12T03:58:46.939 DEBUG:boto:Token: None 2024-04-12T03:58:46.939 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000014 2024-04-12T03:58:46.939 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kJXmuvXeBe2sj3j3vumOfdk5Abg= 2024-04-12T03:58:46.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kJXmuvXeBe2sj3j3vumOfdk5Abg=', 'Content-Length': '0'} 2024-04-12T03:58:46.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003745a2999c5648dc-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.942 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.942 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.942 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000014 2024-04-12T03:58:46.942 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000014 2024-04-12T03:58:46.942 DEBUG:boto:Method: GET 2024-04-12T03:58:46.942 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000014 2024-04-12T03:58:46.942 DEBUG:boto:Data: 2024-04-12T03:58:46.942 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.942 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.943 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.943 DEBUG:boto:Params: {} 2024-04-12T03:58:46.943 DEBUG:boto:Token: None 2024-04-12T03:58:46.943 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.943 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b0610840a36082d-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.946 DEBUG:boto:b'dkyges-60obj000000000000000000141000falseobj000000000000000000142024-04-12T03:58:35.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.946 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.946 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.946 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000014 2024-04-12T03:58:46.946 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000014 2024-04-12T03:58:46.946 DEBUG:boto:Method: GET 2024-04-12T03:58:46.947 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000014 2024-04-12T03:58:46.947 DEBUG:boto:Data: 2024-04-12T03:58:46.947 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.947 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.947 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.947 DEBUG:boto:Params: {} 2024-04-12T03:58:46.947 DEBUG:boto:Token: None 2024-04-12T03:58:46.947 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.947 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.952 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a677ceef6d9bd00-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.952 DEBUG:boto:b'dkyges-60obj000000000000000000141000falseobj000000000000000000142024-04-12T03:58:35.512Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-04-12T03:58:46.952 DEBUG:boto:path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.952 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.953 DEBUG:boto:path=/dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.953 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.953 DEBUG:boto:Method: GET 2024-04-12T03:58:46.953 DEBUG:boto:Path: /dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.953 DEBUG:boto:Data: 2024-04-12T03:58:46.953 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.953 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.953 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.953 DEBUG:boto:Params: {} 2024-04-12T03:58:46.953 DEBUG:boto:Token: None 2024-04-12T03:58:46.953 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.954 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wMEetj6SgR7FXQrpx9iq6nFykHA= 2024-04-12T03:58:46.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wMEetj6SgR7FXQrpx9iq6nFykHA=', 'Content-Length': '0'} 2024-04-12T03:58:46.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014d3295613314bde-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.956 DEBUG:boto:path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.956 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.956 DEBUG:boto:path=/dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.956 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.956 DEBUG:boto:Method: GET 2024-04-12T03:58:46.956 DEBUG:boto:Path: /dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.956 DEBUG:boto:Data: 2024-04-12T03:58:46.956 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.957 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.957 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.957 DEBUG:boto:Params: {} 2024-04-12T03:58:46.957 DEBUG:boto:Token: None 2024-04-12T03:58:46.957 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.957 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wMEetj6SgR7FXQrpx9iq6nFykHA= 2024-04-12T03:58:46.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wMEetj6SgR7FXQrpx9iq6nFykHA=', 'Content-Length': '0'} 2024-04-12T03:58:46.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e54c468ad02900de-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.959 DEBUG:boto:path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.960 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.960 DEBUG:boto:path=/dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.960 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.960 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.960 DEBUG:boto:Path: /dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.960 DEBUG:boto:Data: 2024-04-12T03:58:46.960 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.960 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.960 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.960 DEBUG:boto:Params: {} 2024-04-12T03:58:46.960 DEBUG:boto:Token: None 2024-04-12T03:58:46.960 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.961 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gnpW1FLRdQIbzVzHjm7Jjatz6Nc= 2024-04-12T03:58:46.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gnpW1FLRdQIbzVzHjm7Jjatz6Nc=', 'Content-Length': '0'} 2024-04-12T03:58:46.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099507f958290ca84-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.962 DEBUG:boto:path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.963 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.963 DEBUG:boto:path=/dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.963 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.963 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.963 DEBUG:boto:Path: /dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.963 DEBUG:boto:Data: 2024-04-12T03:58:46.963 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.963 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.963 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.963 DEBUG:boto:Params: {} 2024-04-12T03:58:46.963 DEBUG:boto:Token: None 2024-04-12T03:58:46.963 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.964 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gnpW1FLRdQIbzVzHjm7Jjatz6Nc= 2024-04-12T03:58:46.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gnpW1FLRdQIbzVzHjm7Jjatz6Nc=', 'Content-Length': '0'} 2024-04-12T03:58:46.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da4549098c6a699f-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-04-12T03:58:46.966 DEBUG:boto:path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.966 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.966 DEBUG:boto:path=/dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.966 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.966 DEBUG:boto:Method: GET 2024-04-12T03:58:46.966 DEBUG:boto:Path: /dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.966 DEBUG:boto:Data: 2024-04-12T03:58:46.966 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.966 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.966 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.967 DEBUG:boto:Params: {} 2024-04-12T03:58:46.967 DEBUG:boto:Token: None 2024-04-12T03:58:46.967 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.967 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wMEetj6SgR7FXQrpx9iq6nFykHA= 2024-04-12T03:58:46.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wMEetj6SgR7FXQrpx9iq6nFykHA=', 'Content-Length': '0'} 2024-04-12T03:58:46.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c33265e3b6b836f8-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.969 DEBUG:boto:path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.969 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.969 DEBUG:boto:path=/dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.969 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.969 DEBUG:boto:Method: GET 2024-04-12T03:58:46.969 DEBUG:boto:Path: /dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.969 DEBUG:boto:Data: 2024-04-12T03:58:46.969 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.970 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.970 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.970 DEBUG:boto:Params: {} 2024-04-12T03:58:46.970 DEBUG:boto:Token: None 2024-04-12T03:58:46.970 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000015?versionId=null 2024-04-12T03:58:46.970 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wMEetj6SgR7FXQrpx9iq6nFykHA= 2024-04-12T03:58:46.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wMEetj6SgR7FXQrpx9iq6nFykHA=', 'Content-Length': '0'} 2024-04-12T03:58:46.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013f9984bff15401b-006618b176-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.973 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.973 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.973 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000015 2024-04-12T03:58:46.973 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000015 2024-04-12T03:58:46.973 DEBUG:boto:Method: GET 2024-04-12T03:58:46.973 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000015 2024-04-12T03:58:46.973 DEBUG:boto:Data: 2024-04-12T03:58:46.973 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.973 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.973 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.973 DEBUG:boto:Params: {} 2024-04-12T03:58:46.974 DEBUG:boto:Token: None 2024-04-12T03:58:46.974 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.974 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074a819b3bdc049ef-006618b176-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.977 DEBUG:boto:b'dkyges-60obj000000000000000000151000falseobj000000000000000000152024-04-12T03:58:34.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.977 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:46.977 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:46.977 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000015 2024-04-12T03:58:46.977 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000015 2024-04-12T03:58:46.977 DEBUG:boto:Method: GET 2024-04-12T03:58:46.977 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000015 2024-04-12T03:58:46.977 DEBUG:boto:Data: 2024-04-12T03:58:46.978 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.978 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.978 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.978 DEBUG:boto:Params: {} 2024-04-12T03:58:46.978 DEBUG:boto:Token: None 2024-04-12T03:58:46.978 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/ 2024-04-12T03:58:46.978 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4= 2024-04-12T03:58:46.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+OwpjNrDy5CngnGm9Uj8A3l7Zx4=', 'Content-Length': '0'} 2024-04-12T03:58:46.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3046d3d8330d117-006618b176-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.983 DEBUG:boto:b'dkyges-60obj000000000000000000151000falseobj000000000000000000152024-04-12T03:58:34.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:46.983 DEBUG:boto:path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.984 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.984 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.984 DEBUG:boto:Path: /dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.984 DEBUG:boto:Data: 2024-04-12T03:58:46.984 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.984 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.984 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.984 DEBUG:boto:Params: {} 2024-04-12T03:58:46.984 DEBUG:boto:Token: None 2024-04-12T03:58:46.984 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.984 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sFXf3muhg4ziAuw8peoIx6buqXM= 2024-04-12T03:58:46.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sFXf3muhg4ziAuw8peoIx6buqXM=', 'Content-Length': '0'} 2024-04-12T03:58:46.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad475bd59aa18b24-006618b176-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.986 DEBUG:boto:path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.986 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.987 DEBUG:boto:Method: HEAD 2024-04-12T03:58:46.987 DEBUG:boto:Path: /dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.987 DEBUG:boto:Data: 2024-04-12T03:58:46.987 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.987 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.987 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.987 DEBUG:boto:Params: {} 2024-04-12T03:58:46.987 DEBUG:boto:Token: None 2024-04-12T03:58:46.987 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.987 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sFXf3muhg4ziAuw8peoIx6buqXM= 2024-04-12T03:58:46.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sFXf3muhg4ziAuw8peoIx6buqXM=', 'Content-Length': '0'} 2024-04-12T03:58:46.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a6cda15f710fb2c-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2024-04-12T03:58:46.990 DEBUG:boto:path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.990 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.990 DEBUG:boto:Method: GET 2024-04-12T03:58:46.990 DEBUG:boto:Path: /dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.990 DEBUG:boto:Data: 2024-04-12T03:58:46.990 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.990 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:46.990 DEBUG:boto:Port: 8001 2024-04-12T03:58:46.990 DEBUG:boto:Params: {} 2024-04-12T03:58:46.991 DEBUG:boto:Token: None 2024-04-12T03:58:46.991 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.991 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hSYA/Uf0Qb0fJMVqG8E7dr/q/kY= 2024-04-12T03:58:46.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hSYA/Uf0Qb0fJMVqG8E7dr/q/kY=', 'Content-Length': '0'} 2024-04-12T03:58:46.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003aa8fbaaeda1ffe5-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:46.993 DEBUG:boto:path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.993 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.993 DEBUG:boto:Method: GET 2024-04-12T03:58:46.993 DEBUG:boto:Path: /dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.993 DEBUG:boto:Data: 2024-04-12T03:58:46.993 DEBUG:boto:Headers: {} 2024-04-12T03:58:46.993 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:46.993 DEBUG:boto:Port: 8000 2024-04-12T03:58:46.993 DEBUG:boto:Params: {} 2024-04-12T03:58:46.994 DEBUG:boto:Token: None 2024-04-12T03:58:46.994 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:46 GMT /dkyges-60/obj00000000000000000015 2024-04-12T03:58:46.994 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hSYA/Uf0Qb0fJMVqG8E7dr/q/kY= 2024-04-12T03:58:46.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hSYA/Uf0Qb0fJMVqG8E7dr/q/kY=', 'Content-Length': '0'} 2024-04-12T03:58:46.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051bd41eaf971a95e-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.038 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.038 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.038 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000015 2024-04-12T03:58:47.039 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000015 2024-04-12T03:58:47.039 DEBUG:boto:Method: GET 2024-04-12T03:58:47.039 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000015 2024-04-12T03:58:47.039 DEBUG:boto:Data: 2024-04-12T03:58:47.039 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.039 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.039 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.039 DEBUG:boto:Params: {} 2024-04-12T03:58:47.039 DEBUG:boto:Token: None 2024-04-12T03:58:47.039 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.040 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028695032127b7133-006618b177-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.042 DEBUG:boto:b'dkyges-60obj000000000000000000151000falseobj000000000000000000152024-04-12T03:58:34.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.043 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.043 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.043 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000015 2024-04-12T03:58:47.043 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000015 2024-04-12T03:58:47.043 DEBUG:boto:Method: GET 2024-04-12T03:58:47.043 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000015 2024-04-12T03:58:47.043 DEBUG:boto:Data: 2024-04-12T03:58:47.043 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.043 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.043 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.044 DEBUG:boto:Params: {} 2024-04-12T03:58:47.044 DEBUG:boto:Token: None 2024-04-12T03:58:47.044 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.044 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000749e215ba0c936cf-006618b177-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.049 DEBUG:boto:b'dkyges-60obj000000000000000000151000falseobj000000000000000000152024-04-12T03:58:34.926Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-04-12T03:58:47.049 DEBUG:boto:path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.049 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.049 DEBUG:boto:path=/dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.049 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.049 DEBUG:boto:Method: GET 2024-04-12T03:58:47.049 DEBUG:boto:Path: /dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.050 DEBUG:boto:Data: 2024-04-12T03:58:47.050 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.050 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.050 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.050 DEBUG:boto:Params: {} 2024-04-12T03:58:47.050 DEBUG:boto:Token: None 2024-04-12T03:58:47.050 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.050 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5tgAt9Dx9p+3MTIF0lc4XrpNPR8= 2024-04-12T03:58:47.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5tgAt9Dx9p+3MTIF0lc4XrpNPR8=', 'Content-Length': '0'} 2024-04-12T03:58:47.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daa6d39f92873b5f-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.098 DEBUG:boto:path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.098 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.098 DEBUG:boto:path=/dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.098 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.099 DEBUG:boto:Method: GET 2024-04-12T03:58:47.099 DEBUG:boto:Path: /dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.099 DEBUG:boto:Data: 2024-04-12T03:58:47.099 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.099 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.099 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.099 DEBUG:boto:Params: {} 2024-04-12T03:58:47.099 DEBUG:boto:Token: None 2024-04-12T03:58:47.099 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.099 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5tgAt9Dx9p+3MTIF0lc4XrpNPR8= 2024-04-12T03:58:47.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5tgAt9Dx9p+3MTIF0lc4XrpNPR8=', 'Content-Length': '0'} 2024-04-12T03:58:47.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdedfa4cade397dc-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.102 DEBUG:boto:path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.102 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.102 DEBUG:boto:path=/dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.102 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.102 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.102 DEBUG:boto:Path: /dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.102 DEBUG:boto:Data: 2024-04-12T03:58:47.102 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.102 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.103 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.103 DEBUG:boto:Params: {} 2024-04-12T03:58:47.103 DEBUG:boto:Token: None 2024-04-12T03:58:47.103 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.103 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D64HvbFimKp6KLRkaA/wDn6VqOQ= 2024-04-12T03:58:47.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D64HvbFimKp6KLRkaA/wDn6VqOQ=', 'Content-Length': '0'} 2024-04-12T03:58:47.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d38bb1d3a2640b1d-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.105 DEBUG:boto:path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.105 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.105 DEBUG:boto:path=/dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.106 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.106 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.106 DEBUG:boto:Path: /dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.106 DEBUG:boto:Data: 2024-04-12T03:58:47.106 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.106 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.106 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.106 DEBUG:boto:Params: {} 2024-04-12T03:58:47.106 DEBUG:boto:Token: None 2024-04-12T03:58:47.106 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.107 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D64HvbFimKp6KLRkaA/wDn6VqOQ= 2024-04-12T03:58:47.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D64HvbFimKp6KLRkaA/wDn6VqOQ=', 'Content-Length': '0'} 2024-04-12T03:58:47.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c1e2b84f0fef81e-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-04-12T03:58:47.109 DEBUG:boto:path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.109 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.109 DEBUG:boto:path=/dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.109 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.109 DEBUG:boto:Method: GET 2024-04-12T03:58:47.109 DEBUG:boto:Path: /dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.109 DEBUG:boto:Data: 2024-04-12T03:58:47.109 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.110 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.110 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.110 DEBUG:boto:Params: {} 2024-04-12T03:58:47.110 DEBUG:boto:Token: None 2024-04-12T03:58:47.110 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.110 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5tgAt9Dx9p+3MTIF0lc4XrpNPR8= 2024-04-12T03:58:47.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5tgAt9Dx9p+3MTIF0lc4XrpNPR8=', 'Content-Length': '0'} 2024-04-12T03:58:47.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059a54cb6800af2cd-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.112 DEBUG:boto:path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.112 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.112 DEBUG:boto:path=/dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.113 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.113 DEBUG:boto:Method: GET 2024-04-12T03:58:47.113 DEBUG:boto:Path: /dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.113 DEBUG:boto:Data: 2024-04-12T03:58:47.113 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.113 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.113 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.113 DEBUG:boto:Params: {} 2024-04-12T03:58:47.113 DEBUG:boto:Token: None 2024-04-12T03:58:47.113 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000016?versionId=null 2024-04-12T03:58:47.113 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5tgAt9Dx9p+3MTIF0lc4XrpNPR8= 2024-04-12T03:58:47.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5tgAt9Dx9p+3MTIF0lc4XrpNPR8=', 'Content-Length': '0'} 2024-04-12T03:58:47.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d536f082bf1dee0-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.116 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.116 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.116 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000016 2024-04-12T03:58:47.116 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000016 2024-04-12T03:58:47.116 DEBUG:boto:Method: GET 2024-04-12T03:58:47.116 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000016 2024-04-12T03:58:47.116 DEBUG:boto:Data: 2024-04-12T03:58:47.116 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.116 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.116 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.116 DEBUG:boto:Params: {} 2024-04-12T03:58:47.117 DEBUG:boto:Token: None 2024-04-12T03:58:47.117 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.117 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000208e9c866352756c-006618b177-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.119 DEBUG:boto:b'dkyges-60obj000000000000000000161000falseobj000000000000000000162024-04-12T03:58:35.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.120 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.120 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.120 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000016 2024-04-12T03:58:47.120 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000016 2024-04-12T03:58:47.120 DEBUG:boto:Method: GET 2024-04-12T03:58:47.120 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000016 2024-04-12T03:58:47.120 DEBUG:boto:Data: 2024-04-12T03:58:47.120 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.120 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.121 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.121 DEBUG:boto:Params: {} 2024-04-12T03:58:47.121 DEBUG:boto:Token: None 2024-04-12T03:58:47.121 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.121 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfccea38ed1a451d-006618b177-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.126 DEBUG:boto:b'dkyges-60obj000000000000000000161000falseobj000000000000000000162024-04-12T03:58:35.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.126 DEBUG:boto:path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.126 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.126 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.126 DEBUG:boto:Path: /dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.127 DEBUG:boto:Data: 2024-04-12T03:58:47.127 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.127 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.127 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.127 DEBUG:boto:Params: {} 2024-04-12T03:58:47.127 DEBUG:boto:Token: None 2024-04-12T03:58:47.127 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.127 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:izq3XPZnRdFdMaOyHAGlHLWU4yM= 2024-04-12T03:58:47.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:izq3XPZnRdFdMaOyHAGlHLWU4yM=', 'Content-Length': '0'} 2024-04-12T03:58:47.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022ba798bf61a6d63-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.129 DEBUG:boto:path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.129 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.130 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.130 DEBUG:boto:Path: /dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.130 DEBUG:boto:Data: 2024-04-12T03:58:47.130 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.130 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.130 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.130 DEBUG:boto:Params: {} 2024-04-12T03:58:47.130 DEBUG:boto:Token: None 2024-04-12T03:58:47.130 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.130 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:izq3XPZnRdFdMaOyHAGlHLWU4yM= 2024-04-12T03:58:47.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:izq3XPZnRdFdMaOyHAGlHLWU4yM=', 'Content-Length': '0'} 2024-04-12T03:58:47.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064aa916ae1b937c8-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2024-04-12T03:58:47.133 DEBUG:boto:path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.133 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.133 DEBUG:boto:Method: GET 2024-04-12T03:58:47.133 DEBUG:boto:Path: /dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.133 DEBUG:boto:Data: 2024-04-12T03:58:47.133 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.133 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.133 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.133 DEBUG:boto:Params: {} 2024-04-12T03:58:47.133 DEBUG:boto:Token: None 2024-04-12T03:58:47.134 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.134 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rpZFIufnldJz9CGxLkuyEFkedaI= 2024-04-12T03:58:47.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rpZFIufnldJz9CGxLkuyEFkedaI=', 'Content-Length': '0'} 2024-04-12T03:58:47.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014fb0583846239ae-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.136 DEBUG:boto:path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.136 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.136 DEBUG:boto:Method: GET 2024-04-12T03:58:47.136 DEBUG:boto:Path: /dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.136 DEBUG:boto:Data: 2024-04-12T03:58:47.136 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.136 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.136 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.136 DEBUG:boto:Params: {} 2024-04-12T03:58:47.137 DEBUG:boto:Token: None 2024-04-12T03:58:47.137 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000016 2024-04-12T03:58:47.137 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rpZFIufnldJz9CGxLkuyEFkedaI= 2024-04-12T03:58:47.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rpZFIufnldJz9CGxLkuyEFkedaI=', 'Content-Length': '0'} 2024-04-12T03:58:47.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a44bf36b5b86d3a4-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.182 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.182 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.183 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000016 2024-04-12T03:58:47.183 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000016 2024-04-12T03:58:47.183 DEBUG:boto:Method: GET 2024-04-12T03:58:47.183 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000016 2024-04-12T03:58:47.183 DEBUG:boto:Data: 2024-04-12T03:58:47.183 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.183 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.183 DEBUG:boto:Params: {} 2024-04-12T03:58:47.183 DEBUG:boto:Token: None 2024-04-12T03:58:47.183 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.184 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019db3056231898ad-006618b177-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.186 DEBUG:boto:b'dkyges-60obj000000000000000000161000falseobj000000000000000000162024-04-12T03:58:35.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.187 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.187 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.187 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000016 2024-04-12T03:58:47.187 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000016 2024-04-12T03:58:47.187 DEBUG:boto:Method: GET 2024-04-12T03:58:47.187 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000016 2024-04-12T03:58:47.187 DEBUG:boto:Data: 2024-04-12T03:58:47.187 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.187 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.187 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.187 DEBUG:boto:Params: {} 2024-04-12T03:58:47.188 DEBUG:boto:Token: None 2024-04-12T03:58:47.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.188 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000498fbbd1d0ac1ff-006618b177-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.193 DEBUG:boto:b'dkyges-60obj000000000000000000161000falseobj000000000000000000162024-04-12T03:58:35.800Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-04-12T03:58:47.193 DEBUG:boto:path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.194 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.194 DEBUG:boto:path=/dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.194 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.194 DEBUG:boto:Method: GET 2024-04-12T03:58:47.194 DEBUG:boto:Path: /dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.194 DEBUG:boto:Data: 2024-04-12T03:58:47.194 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.194 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.194 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.194 DEBUG:boto:Params: {} 2024-04-12T03:58:47.195 DEBUG:boto:Token: None 2024-04-12T03:58:47.195 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.195 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hnFIuYRoia8BN7AGwHJm2ILZS/A= 2024-04-12T03:58:47.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hnFIuYRoia8BN7AGwHJm2ILZS/A=', 'Content-Length': '0'} 2024-04-12T03:58:47.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f82f905614b9720-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.238 DEBUG:boto:path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.238 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.239 DEBUG:boto:path=/dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.239 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.239 DEBUG:boto:Method: GET 2024-04-12T03:58:47.239 DEBUG:boto:Path: /dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.239 DEBUG:boto:Data: 2024-04-12T03:58:47.239 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.239 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.239 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.239 DEBUG:boto:Params: {} 2024-04-12T03:58:47.239 DEBUG:boto:Token: None 2024-04-12T03:58:47.239 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.240 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hnFIuYRoia8BN7AGwHJm2ILZS/A= 2024-04-12T03:58:47.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hnFIuYRoia8BN7AGwHJm2ILZS/A=', 'Content-Length': '0'} 2024-04-12T03:58:47.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e420d924c0a02e2f-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.242 DEBUG:boto:path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.242 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.243 DEBUG:boto:path=/dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.243 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.243 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.243 DEBUG:boto:Path: /dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.243 DEBUG:boto:Data: 2024-04-12T03:58:47.243 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.243 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.244 DEBUG:boto:Params: {} 2024-04-12T03:58:47.244 DEBUG:boto:Token: None 2024-04-12T03:58:47.244 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.244 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0UXhQeXPn/2DQBDhG44GrIuYaJg= 2024-04-12T03:58:47.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0UXhQeXPn/2DQBDhG44GrIuYaJg=', 'Content-Length': '0'} 2024-04-12T03:58:47.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5e4ceac69e96f40-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.246 DEBUG:boto:path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.246 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.246 DEBUG:boto:path=/dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.246 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.246 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.247 DEBUG:boto:Path: /dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.247 DEBUG:boto:Data: 2024-04-12T03:58:47.247 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.247 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.247 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.247 DEBUG:boto:Params: {} 2024-04-12T03:58:47.247 DEBUG:boto:Token: None 2024-04-12T03:58:47.247 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.247 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0UXhQeXPn/2DQBDhG44GrIuYaJg= 2024-04-12T03:58:47.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0UXhQeXPn/2DQBDhG44GrIuYaJg=', 'Content-Length': '0'} 2024-04-12T03:58:47.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b03a1310ac53b7-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-04-12T03:58:47.249 DEBUG:boto:path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.249 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.250 DEBUG:boto:path=/dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.250 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.250 DEBUG:boto:Method: GET 2024-04-12T03:58:47.250 DEBUG:boto:Path: /dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.250 DEBUG:boto:Data: 2024-04-12T03:58:47.250 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.250 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.250 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.250 DEBUG:boto:Params: {} 2024-04-12T03:58:47.251 DEBUG:boto:Token: None 2024-04-12T03:58:47.251 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.251 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hnFIuYRoia8BN7AGwHJm2ILZS/A= 2024-04-12T03:58:47.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hnFIuYRoia8BN7AGwHJm2ILZS/A=', 'Content-Length': '0'} 2024-04-12T03:58:47.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024d82c0045016116-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.253 DEBUG:boto:path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.253 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.253 DEBUG:boto:path=/dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.253 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.253 DEBUG:boto:Method: GET 2024-04-12T03:58:47.253 DEBUG:boto:Path: /dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.254 DEBUG:boto:Data: 2024-04-12T03:58:47.254 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.254 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.254 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.254 DEBUG:boto:Params: {} 2024-04-12T03:58:47.254 DEBUG:boto:Token: None 2024-04-12T03:58:47.254 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000017?versionId=null 2024-04-12T03:58:47.254 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hnFIuYRoia8BN7AGwHJm2ILZS/A= 2024-04-12T03:58:47.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hnFIuYRoia8BN7AGwHJm2ILZS/A=', 'Content-Length': '0'} 2024-04-12T03:58:47.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000069e5cbe733c9301-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.257 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.257 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.257 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000017 2024-04-12T03:58:47.257 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000017 2024-04-12T03:58:47.257 DEBUG:boto:Method: GET 2024-04-12T03:58:47.257 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000017 2024-04-12T03:58:47.257 DEBUG:boto:Data: 2024-04-12T03:58:47.257 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.258 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.258 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.258 DEBUG:boto:Params: {} 2024-04-12T03:58:47.258 DEBUG:boto:Token: None 2024-04-12T03:58:47.258 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.258 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0da9808baffb2ac-006618b177-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.261 DEBUG:boto:b'dkyges-60obj000000000000000000171000falseobj000000000000000000172024-04-12T03:58:35.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.262 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.262 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.262 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000017 2024-04-12T03:58:47.262 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000017 2024-04-12T03:58:47.262 DEBUG:boto:Method: GET 2024-04-12T03:58:47.262 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000017 2024-04-12T03:58:47.262 DEBUG:boto:Data: 2024-04-12T03:58:47.262 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.262 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.263 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.263 DEBUG:boto:Params: {} 2024-04-12T03:58:47.263 DEBUG:boto:Token: None 2024-04-12T03:58:47.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.263 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1bcc23a77f7de81-006618b177-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.268 DEBUG:boto:b'dkyges-60obj000000000000000000171000falseobj000000000000000000172024-04-12T03:58:35.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.268 DEBUG:boto:path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.268 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.268 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.268 DEBUG:boto:Path: /dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.268 DEBUG:boto:Data: 2024-04-12T03:58:47.268 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.268 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.269 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.269 DEBUG:boto:Params: {} 2024-04-12T03:58:47.269 DEBUG:boto:Token: None 2024-04-12T03:58:47.269 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.269 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:socsSEHx4JOpS5VB3nMiaSrHf/g= 2024-04-12T03:58:47.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:socsSEHx4JOpS5VB3nMiaSrHf/g=', 'Content-Length': '0'} 2024-04-12T03:58:47.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c9de66e32909d73-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.271 DEBUG:boto:path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.271 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.271 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.271 DEBUG:boto:Path: /dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.271 DEBUG:boto:Data: 2024-04-12T03:58:47.271 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.271 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.272 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.272 DEBUG:boto:Params: {} 2024-04-12T03:58:47.272 DEBUG:boto:Token: None 2024-04-12T03:58:47.272 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.272 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:socsSEHx4JOpS5VB3nMiaSrHf/g= 2024-04-12T03:58:47.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:socsSEHx4JOpS5VB3nMiaSrHf/g=', 'Content-Length': '0'} 2024-04-12T03:58:47.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c41207cce7374037-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2024-04-12T03:58:47.275 DEBUG:boto:path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.275 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.275 DEBUG:boto:Method: GET 2024-04-12T03:58:47.275 DEBUG:boto:Path: /dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.275 DEBUG:boto:Data: 2024-04-12T03:58:47.275 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.275 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.275 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.275 DEBUG:boto:Params: {} 2024-04-12T03:58:47.275 DEBUG:boto:Token: None 2024-04-12T03:58:47.275 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.276 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AWvsE9E+cOvZLn9YNboK9Fev1ds= 2024-04-12T03:58:47.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AWvsE9E+cOvZLn9YNboK9Fev1ds=', 'Content-Length': '0'} 2024-04-12T03:58:47.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007485be361f005d5f-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.278 DEBUG:boto:path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.278 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.278 DEBUG:boto:Method: GET 2024-04-12T03:58:47.279 DEBUG:boto:Path: /dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.279 DEBUG:boto:Data: 2024-04-12T03:58:47.279 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.279 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.279 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.279 DEBUG:boto:Params: {} 2024-04-12T03:58:47.279 DEBUG:boto:Token: None 2024-04-12T03:58:47.279 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000017 2024-04-12T03:58:47.279 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AWvsE9E+cOvZLn9YNboK9Fev1ds= 2024-04-12T03:58:47.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AWvsE9E+cOvZLn9YNboK9Fev1ds=', 'Content-Length': '0'} 2024-04-12T03:58:47.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029615148c2ba4632-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.282 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.282 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.282 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000017 2024-04-12T03:58:47.282 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000017 2024-04-12T03:58:47.282 DEBUG:boto:Method: GET 2024-04-12T03:58:47.283 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000017 2024-04-12T03:58:47.283 DEBUG:boto:Data: 2024-04-12T03:58:47.283 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.283 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.283 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.283 DEBUG:boto:Params: {} 2024-04-12T03:58:47.283 DEBUG:boto:Token: None 2024-04-12T03:58:47.283 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.283 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000357a68fa191b006a-006618b177-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.286 DEBUG:boto:b'dkyges-60obj000000000000000000171000falseobj000000000000000000172024-04-12T03:58:35.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.287 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.287 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.287 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000017 2024-04-12T03:58:47.287 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000017 2024-04-12T03:58:47.287 DEBUG:boto:Method: GET 2024-04-12T03:58:47.287 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000017 2024-04-12T03:58:47.287 DEBUG:boto:Data: 2024-04-12T03:58:47.287 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.287 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.287 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.287 DEBUG:boto:Params: {} 2024-04-12T03:58:47.288 DEBUG:boto:Token: None 2024-04-12T03:58:47.288 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.288 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008df6661f1f7dbf1e-006618b177-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.293 DEBUG:boto:b'dkyges-60obj000000000000000000171000falseobj000000000000000000172024-04-12T03:58:35.460Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-04-12T03:58:47.294 DEBUG:boto:path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.294 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.294 DEBUG:boto:path=/dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.294 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.294 DEBUG:boto:Method: GET 2024-04-12T03:58:47.294 DEBUG:boto:Path: /dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.294 DEBUG:boto:Data: 2024-04-12T03:58:47.295 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.295 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.295 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.295 DEBUG:boto:Params: {} 2024-04-12T03:58:47.295 DEBUG:boto:Token: None 2024-04-12T03:58:47.295 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.295 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZiumbbiYbdLmALmc23s2mKqXHnI= 2024-04-12T03:58:47.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZiumbbiYbdLmALmc23s2mKqXHnI=', 'Content-Length': '0'} 2024-04-12T03:58:47.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dfd2a2397bfaf5a-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.338 DEBUG:boto:path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.338 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.339 DEBUG:boto:path=/dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.339 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.339 DEBUG:boto:Method: GET 2024-04-12T03:58:47.339 DEBUG:boto:Path: /dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.339 DEBUG:boto:Data: 2024-04-12T03:58:47.339 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.339 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.339 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.339 DEBUG:boto:Params: {} 2024-04-12T03:58:47.339 DEBUG:boto:Token: None 2024-04-12T03:58:47.340 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.340 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZiumbbiYbdLmALmc23s2mKqXHnI= 2024-04-12T03:58:47.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZiumbbiYbdLmALmc23s2mKqXHnI=', 'Content-Length': '0'} 2024-04-12T03:58:47.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e592935a5de5574-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.342 DEBUG:boto:path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.342 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.342 DEBUG:boto:path=/dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.343 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.343 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.343 DEBUG:boto:Path: /dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.343 DEBUG:boto:Data: 2024-04-12T03:58:47.343 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.343 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.343 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.343 DEBUG:boto:Params: {} 2024-04-12T03:58:47.343 DEBUG:boto:Token: None 2024-04-12T03:58:47.343 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.344 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TLViOu8P/vX5zAw4RKX8xCalchY= 2024-04-12T03:58:47.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TLViOu8P/vX5zAw4RKX8xCalchY=', 'Content-Length': '0'} 2024-04-12T03:58:47.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be19f238ba75ffa9-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.346 DEBUG:boto:path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.346 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.346 DEBUG:boto:path=/dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.346 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.346 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.346 DEBUG:boto:Path: /dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.346 DEBUG:boto:Data: 2024-04-12T03:58:47.346 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.346 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.347 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.347 DEBUG:boto:Params: {} 2024-04-12T03:58:47.347 DEBUG:boto:Token: None 2024-04-12T03:58:47.347 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.347 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TLViOu8P/vX5zAw4RKX8xCalchY= 2024-04-12T03:58:47.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TLViOu8P/vX5zAw4RKX8xCalchY=', 'Content-Length': '0'} 2024-04-12T03:58:47.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e58cd6b16cb7f7b-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-04-12T03:58:47.349 DEBUG:boto:path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.349 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.349 DEBUG:boto:path=/dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.349 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.350 DEBUG:boto:Method: GET 2024-04-12T03:58:47.350 DEBUG:boto:Path: /dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.350 DEBUG:boto:Data: 2024-04-12T03:58:47.350 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.350 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.350 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.350 DEBUG:boto:Params: {} 2024-04-12T03:58:47.350 DEBUG:boto:Token: None 2024-04-12T03:58:47.350 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.350 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZiumbbiYbdLmALmc23s2mKqXHnI= 2024-04-12T03:58:47.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZiumbbiYbdLmALmc23s2mKqXHnI=', 'Content-Length': '0'} 2024-04-12T03:58:47.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061af0d0e50499c7c-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.353 DEBUG:boto:path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.353 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.353 DEBUG:boto:path=/dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.353 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.353 DEBUG:boto:Method: GET 2024-04-12T03:58:47.353 DEBUG:boto:Path: /dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.353 DEBUG:boto:Data: 2024-04-12T03:58:47.353 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.353 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.353 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.353 DEBUG:boto:Params: {} 2024-04-12T03:58:47.354 DEBUG:boto:Token: None 2024-04-12T03:58:47.354 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000018?versionId=null 2024-04-12T03:58:47.354 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZiumbbiYbdLmALmc23s2mKqXHnI= 2024-04-12T03:58:47.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZiumbbiYbdLmALmc23s2mKqXHnI=', 'Content-Length': '0'} 2024-04-12T03:58:47.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097a1bcbb1856a621-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.356 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.356 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.356 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000018 2024-04-12T03:58:47.356 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000018 2024-04-12T03:58:47.356 DEBUG:boto:Method: GET 2024-04-12T03:58:47.356 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000018 2024-04-12T03:58:47.356 DEBUG:boto:Data: 2024-04-12T03:58:47.356 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.357 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.357 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.357 DEBUG:boto:Params: {} 2024-04-12T03:58:47.357 DEBUG:boto:Token: None 2024-04-12T03:58:47.357 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.357 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000144e03f2ca92d3b8-006618b177-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.360 DEBUG:boto:b'dkyges-60obj000000000000000000181000falseobj000000000000000000182024-04-12T03:58:35.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.360 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.361 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.361 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000018 2024-04-12T03:58:47.361 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000018 2024-04-12T03:58:47.361 DEBUG:boto:Method: GET 2024-04-12T03:58:47.361 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000018 2024-04-12T03:58:47.361 DEBUG:boto:Data: 2024-04-12T03:58:47.361 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.361 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.361 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.361 DEBUG:boto:Params: {} 2024-04-12T03:58:47.362 DEBUG:boto:Token: None 2024-04-12T03:58:47.362 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.362 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009274e1faa56ee3ad-006618b177-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.366 DEBUG:boto:b'dkyges-60obj000000000000000000181000falseobj000000000000000000182024-04-12T03:58:35.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.366 DEBUG:boto:path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.367 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.367 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.367 DEBUG:boto:Path: /dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.367 DEBUG:boto:Data: 2024-04-12T03:58:47.367 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.367 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.367 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.367 DEBUG:boto:Params: {} 2024-04-12T03:58:47.367 DEBUG:boto:Token: None 2024-04-12T03:58:47.368 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.368 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rcOLSpSuXswRz0ziGT+FefNUGLM= 2024-04-12T03:58:47.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rcOLSpSuXswRz0ziGT+FefNUGLM=', 'Content-Length': '0'} 2024-04-12T03:58:47.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000442f0bc0489df208-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.370 DEBUG:boto:path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.370 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.370 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.370 DEBUG:boto:Path: /dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.370 DEBUG:boto:Data: 2024-04-12T03:58:47.370 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.370 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.370 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.370 DEBUG:boto:Params: {} 2024-04-12T03:58:47.371 DEBUG:boto:Token: None 2024-04-12T03:58:47.371 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.371 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rcOLSpSuXswRz0ziGT+FefNUGLM= 2024-04-12T03:58:47.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rcOLSpSuXswRz0ziGT+FefNUGLM=', 'Content-Length': '0'} 2024-04-12T03:58:47.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094ee0e025dfdb887-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2024-04-12T03:58:47.373 DEBUG:boto:path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.373 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.374 DEBUG:boto:Method: GET 2024-04-12T03:58:47.374 DEBUG:boto:Path: /dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.374 DEBUG:boto:Data: 2024-04-12T03:58:47.374 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.374 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.374 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.374 DEBUG:boto:Params: {} 2024-04-12T03:58:47.374 DEBUG:boto:Token: None 2024-04-12T03:58:47.374 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.374 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fgL537KKsgsg25RzvJe7Tejn9To= 2024-04-12T03:58:47.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fgL537KKsgsg25RzvJe7Tejn9To=', 'Content-Length': '0'} 2024-04-12T03:58:47.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008db88b9766c93d34-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.377 DEBUG:boto:path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.377 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.377 DEBUG:boto:Method: GET 2024-04-12T03:58:47.377 DEBUG:boto:Path: /dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.377 DEBUG:boto:Data: 2024-04-12T03:58:47.377 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.377 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.377 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.377 DEBUG:boto:Params: {} 2024-04-12T03:58:47.377 DEBUG:boto:Token: None 2024-04-12T03:58:47.378 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000018 2024-04-12T03:58:47.378 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fgL537KKsgsg25RzvJe7Tejn9To= 2024-04-12T03:58:47.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fgL537KKsgsg25RzvJe7Tejn9To=', 'Content-Length': '0'} 2024-04-12T03:58:47.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1d083d819c06804-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.422 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.422 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.422 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000018 2024-04-12T03:58:47.423 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000018 2024-04-12T03:58:47.423 DEBUG:boto:Method: GET 2024-04-12T03:58:47.423 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000018 2024-04-12T03:58:47.423 DEBUG:boto:Data: 2024-04-12T03:58:47.423 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.423 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.423 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.423 DEBUG:boto:Params: {} 2024-04-12T03:58:47.423 DEBUG:boto:Token: None 2024-04-12T03:58:47.423 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.424 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bf00234e4832738-006618b177-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.426 DEBUG:boto:b'dkyges-60obj000000000000000000181000falseobj000000000000000000182024-04-12T03:58:35.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.427 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.427 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.427 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000018 2024-04-12T03:58:47.427 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000018 2024-04-12T03:58:47.427 DEBUG:boto:Method: GET 2024-04-12T03:58:47.427 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000018 2024-04-12T03:58:47.427 DEBUG:boto:Data: 2024-04-12T03:58:47.427 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.427 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.427 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.428 DEBUG:boto:Params: {} 2024-04-12T03:58:47.428 DEBUG:boto:Token: None 2024-04-12T03:58:47.428 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.428 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076426b592c2395eb-006618b177-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.433 DEBUG:boto:b'dkyges-60obj000000000000000000181000falseobj000000000000000000182024-04-12T03:58:35.124Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-04-12T03:58:47.433 DEBUG:boto:path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.433 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.434 DEBUG:boto:path=/dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.434 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.434 DEBUG:boto:Method: GET 2024-04-12T03:58:47.434 DEBUG:boto:Path: /dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.434 DEBUG:boto:Data: 2024-04-12T03:58:47.434 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.434 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.434 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.434 DEBUG:boto:Params: {} 2024-04-12T03:58:47.434 DEBUG:boto:Token: None 2024-04-12T03:58:47.434 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.435 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f8D9cx2gCdMMPTmK3/Xn267LznA= 2024-04-12T03:58:47.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f8D9cx2gCdMMPTmK3/Xn267LznA=', 'Content-Length': '0'} 2024-04-12T03:58:47.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3830375cea7dbf7-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.482 DEBUG:boto:path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.483 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.483 DEBUG:boto:path=/dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.483 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.483 DEBUG:boto:Method: GET 2024-04-12T03:58:47.483 DEBUG:boto:Path: /dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.483 DEBUG:boto:Data: 2024-04-12T03:58:47.483 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.483 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.483 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.483 DEBUG:boto:Params: {} 2024-04-12T03:58:47.483 DEBUG:boto:Token: None 2024-04-12T03:58:47.484 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.484 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f8D9cx2gCdMMPTmK3/Xn267LznA= 2024-04-12T03:58:47.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f8D9cx2gCdMMPTmK3/Xn267LznA=', 'Content-Length': '0'} 2024-04-12T03:58:47.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000690a2a8f90b96ff7-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.486 DEBUG:boto:path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.486 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.486 DEBUG:boto:path=/dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.486 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.486 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.486 DEBUG:boto:Path: /dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.487 DEBUG:boto:Data: 2024-04-12T03:58:47.487 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.487 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.487 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.487 DEBUG:boto:Params: {} 2024-04-12T03:58:47.487 DEBUG:boto:Token: None 2024-04-12T03:58:47.487 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.487 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AZH2ha4qaDjmUnjU7bTIXtLfyF8= 2024-04-12T03:58:47.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AZH2ha4qaDjmUnjU7bTIXtLfyF8=', 'Content-Length': '0'} 2024-04-12T03:58:47.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007db39b32be57beec-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.489 DEBUG:boto:path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.489 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.489 DEBUG:boto:path=/dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.489 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.489 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.490 DEBUG:boto:Path: /dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.490 DEBUG:boto:Data: 2024-04-12T03:58:47.490 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.490 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.490 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.490 DEBUG:boto:Params: {} 2024-04-12T03:58:47.490 DEBUG:boto:Token: None 2024-04-12T03:58:47.490 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.490 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AZH2ha4qaDjmUnjU7bTIXtLfyF8= 2024-04-12T03:58:47.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AZH2ha4qaDjmUnjU7bTIXtLfyF8=', 'Content-Length': '0'} 2024-04-12T03:58:47.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015f3ce761b206846-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-04-12T03:58:47.493 DEBUG:boto:path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.493 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.493 DEBUG:boto:path=/dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.493 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.493 DEBUG:boto:Method: GET 2024-04-12T03:58:47.493 DEBUG:boto:Path: /dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.493 DEBUG:boto:Data: 2024-04-12T03:58:47.493 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.493 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.493 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.493 DEBUG:boto:Params: {} 2024-04-12T03:58:47.493 DEBUG:boto:Token: None 2024-04-12T03:58:47.494 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.494 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f8D9cx2gCdMMPTmK3/Xn267LznA= 2024-04-12T03:58:47.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f8D9cx2gCdMMPTmK3/Xn267LznA=', 'Content-Length': '0'} 2024-04-12T03:58:47.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d92f71540ecd34b-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.496 DEBUG:boto:path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.496 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.496 DEBUG:boto:path=/dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.496 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.496 DEBUG:boto:Method: GET 2024-04-12T03:58:47.496 DEBUG:boto:Path: /dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.496 DEBUG:boto:Data: 2024-04-12T03:58:47.496 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.496 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.497 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.497 DEBUG:boto:Params: {} 2024-04-12T03:58:47.497 DEBUG:boto:Token: None 2024-04-12T03:58:47.497 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000019?versionId=null 2024-04-12T03:58:47.497 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f8D9cx2gCdMMPTmK3/Xn267LznA= 2024-04-12T03:58:47.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f8D9cx2gCdMMPTmK3/Xn267LznA=', 'Content-Length': '0'} 2024-04-12T03:58:47.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098509fb8c90c3e41-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.499 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.499 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.499 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000019 2024-04-12T03:58:47.499 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000019 2024-04-12T03:58:47.499 DEBUG:boto:Method: GET 2024-04-12T03:58:47.500 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000019 2024-04-12T03:58:47.500 DEBUG:boto:Data: 2024-04-12T03:58:47.500 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.500 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.500 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.500 DEBUG:boto:Params: {} 2024-04-12T03:58:47.500 DEBUG:boto:Token: None 2024-04-12T03:58:47.500 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.500 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9579301c38e046a-006618b177-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.503 DEBUG:boto:b'dkyges-60obj000000000000000000191000falseobj000000000000000000192024-04-12T03:58:32.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.503 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.503 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.503 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000019 2024-04-12T03:58:47.503 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000019 2024-04-12T03:58:47.504 DEBUG:boto:Method: GET 2024-04-12T03:58:47.504 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000019 2024-04-12T03:58:47.504 DEBUG:boto:Data: 2024-04-12T03:58:47.504 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.504 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.504 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.504 DEBUG:boto:Params: {} 2024-04-12T03:58:47.504 DEBUG:boto:Token: None 2024-04-12T03:58:47.504 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.504 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000300c80644952876-006618b177-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.510 DEBUG:boto:b'dkyges-60obj000000000000000000191000falseobj000000000000000000192024-04-12T03:58:32.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.510 DEBUG:boto:path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.510 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.510 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.510 DEBUG:boto:Path: /dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.510 DEBUG:boto:Data: 2024-04-12T03:58:47.510 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.510 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.511 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.511 DEBUG:boto:Params: {} 2024-04-12T03:58:47.511 DEBUG:boto:Token: None 2024-04-12T03:58:47.511 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.511 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Vye7czqm/ih9DHcCtSQ5iLjjdp0= 2024-04-12T03:58:47.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Vye7czqm/ih9DHcCtSQ5iLjjdp0=', 'Content-Length': '0'} 2024-04-12T03:58:47.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b24b6f8a240b9a8-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.513 DEBUG:boto:path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.513 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.513 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.513 DEBUG:boto:Path: /dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.514 DEBUG:boto:Data: 2024-04-12T03:58:47.514 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.514 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.514 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.514 DEBUG:boto:Params: {} 2024-04-12T03:58:47.514 DEBUG:boto:Token: None 2024-04-12T03:58:47.514 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.514 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Vye7czqm/ih9DHcCtSQ5iLjjdp0= 2024-04-12T03:58:47.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Vye7czqm/ih9DHcCtSQ5iLjjdp0=', 'Content-Length': '0'} 2024-04-12T03:58:47.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d6c0e398c848d11-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2024-04-12T03:58:47.517 DEBUG:boto:path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.517 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.517 DEBUG:boto:Method: GET 2024-04-12T03:58:47.517 DEBUG:boto:Path: /dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.517 DEBUG:boto:Data: 2024-04-12T03:58:47.517 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.517 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.517 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.517 DEBUG:boto:Params: {} 2024-04-12T03:58:47.517 DEBUG:boto:Token: None 2024-04-12T03:58:47.517 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.518 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ML3QrRDxS283JDh+O/gxy4jryjw= 2024-04-12T03:58:47.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ML3QrRDxS283JDh+O/gxy4jryjw=', 'Content-Length': '0'} 2024-04-12T03:58:47.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fc57e573ebdae08-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.519 DEBUG:boto:path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.520 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.520 DEBUG:boto:Method: GET 2024-04-12T03:58:47.520 DEBUG:boto:Path: /dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.520 DEBUG:boto:Data: 2024-04-12T03:58:47.520 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.520 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.520 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.520 DEBUG:boto:Params: {} 2024-04-12T03:58:47.520 DEBUG:boto:Token: None 2024-04-12T03:58:47.520 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000019 2024-04-12T03:58:47.521 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ML3QrRDxS283JDh+O/gxy4jryjw= 2024-04-12T03:58:47.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ML3QrRDxS283JDh+O/gxy4jryjw=', 'Content-Length': '0'} 2024-04-12T03:58:47.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eadf518c2b5b9e1-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.566 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.567 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.567 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000019 2024-04-12T03:58:47.567 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000019 2024-04-12T03:58:47.567 DEBUG:boto:Method: GET 2024-04-12T03:58:47.567 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000019 2024-04-12T03:58:47.567 DEBUG:boto:Data: 2024-04-12T03:58:47.567 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.567 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.567 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.567 DEBUG:boto:Params: {} 2024-04-12T03:58:47.567 DEBUG:boto:Token: None 2024-04-12T03:58:47.568 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.568 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035201406c58618d0-006618b177-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.571 DEBUG:boto:b'dkyges-60obj000000000000000000191000falseobj000000000000000000192024-04-12T03:58:32.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.571 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.571 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.571 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000019 2024-04-12T03:58:47.571 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000019 2024-04-12T03:58:47.571 DEBUG:boto:Method: GET 2024-04-12T03:58:47.572 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000019 2024-04-12T03:58:47.572 DEBUG:boto:Data: 2024-04-12T03:58:47.572 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.572 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.572 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.572 DEBUG:boto:Params: {} 2024-04-12T03:58:47.572 DEBUG:boto:Token: None 2024-04-12T03:58:47.572 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.572 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000804833590727b1da-006618b177-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.577 DEBUG:boto:b'dkyges-60obj000000000000000000191000falseobj000000000000000000192024-04-12T03:58:32.415Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-04-12T03:58:47.577 DEBUG:boto:path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.577 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.577 DEBUG:boto:path=/dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.577 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.577 DEBUG:boto:Method: GET 2024-04-12T03:58:47.577 DEBUG:boto:Path: /dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.578 DEBUG:boto:Data: 2024-04-12T03:58:47.578 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.578 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.578 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.578 DEBUG:boto:Params: {} 2024-04-12T03:58:47.578 DEBUG:boto:Token: None 2024-04-12T03:58:47.578 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.578 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sjC0egYE8qftN7QjmDbEc1Km3C0= 2024-04-12T03:58:47.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sjC0egYE8qftN7QjmDbEc1Km3C0=', 'Content-Length': '0'} 2024-04-12T03:58:47.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a21bd5e89a84822d-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.622 DEBUG:boto:path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.622 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.623 DEBUG:boto:path=/dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.623 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.623 DEBUG:boto:Method: GET 2024-04-12T03:58:47.623 DEBUG:boto:Path: /dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.623 DEBUG:boto:Data: 2024-04-12T03:58:47.623 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.623 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.623 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.623 DEBUG:boto:Params: {} 2024-04-12T03:58:47.623 DEBUG:boto:Token: None 2024-04-12T03:58:47.624 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.624 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sjC0egYE8qftN7QjmDbEc1Km3C0= 2024-04-12T03:58:47.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sjC0egYE8qftN7QjmDbEc1Km3C0=', 'Content-Length': '0'} 2024-04-12T03:58:47.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a87fbc3e92431f97-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.626 DEBUG:boto:path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.626 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.626 DEBUG:boto:path=/dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.626 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.626 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.626 DEBUG:boto:Path: /dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.627 DEBUG:boto:Data: 2024-04-12T03:58:47.627 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.627 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.627 DEBUG:boto:Params: {} 2024-04-12T03:58:47.627 DEBUG:boto:Token: None 2024-04-12T03:58:47.627 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.627 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XVqH9vkUgeJS/An3Vysn2PlbFNo= 2024-04-12T03:58:47.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XVqH9vkUgeJS/An3Vysn2PlbFNo=', 'Content-Length': '0'} 2024-04-12T03:58:47.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075be2ceb59f73f97-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.629 DEBUG:boto:path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.630 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.630 DEBUG:boto:path=/dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.630 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.630 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.630 DEBUG:boto:Path: /dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.630 DEBUG:boto:Data: 2024-04-12T03:58:47.630 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.630 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.630 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.630 DEBUG:boto:Params: {} 2024-04-12T03:58:47.630 DEBUG:boto:Token: None 2024-04-12T03:58:47.631 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.631 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XVqH9vkUgeJS/An3Vysn2PlbFNo= 2024-04-12T03:58:47.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XVqH9vkUgeJS/An3Vysn2PlbFNo=', 'Content-Length': '0'} 2024-04-12T03:58:47.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088a2574a919ab2ba-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-04-12T03:58:47.633 DEBUG:boto:path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.633 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.633 DEBUG:boto:path=/dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.633 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.633 DEBUG:boto:Method: GET 2024-04-12T03:58:47.633 DEBUG:boto:Path: /dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.634 DEBUG:boto:Data: 2024-04-12T03:58:47.634 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.634 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.634 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.634 DEBUG:boto:Params: {} 2024-04-12T03:58:47.634 DEBUG:boto:Token: None 2024-04-12T03:58:47.634 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.634 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sjC0egYE8qftN7QjmDbEc1Km3C0= 2024-04-12T03:58:47.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sjC0egYE8qftN7QjmDbEc1Km3C0=', 'Content-Length': '0'} 2024-04-12T03:58:47.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072171b044d82da12-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.637 DEBUG:boto:path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.637 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.637 DEBUG:boto:path=/dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.637 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.637 DEBUG:boto:Method: GET 2024-04-12T03:58:47.637 DEBUG:boto:Path: /dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.637 DEBUG:boto:Data: 2024-04-12T03:58:47.637 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.638 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.638 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.638 DEBUG:boto:Params: {} 2024-04-12T03:58:47.638 DEBUG:boto:Token: None 2024-04-12T03:58:47.638 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000020?versionId=null 2024-04-12T03:58:47.638 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sjC0egYE8qftN7QjmDbEc1Km3C0= 2024-04-12T03:58:47.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sjC0egYE8qftN7QjmDbEc1Km3C0=', 'Content-Length': '0'} 2024-04-12T03:58:47.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002091acb6757db34f-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.640 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.641 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.641 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000020 2024-04-12T03:58:47.641 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000020 2024-04-12T03:58:47.641 DEBUG:boto:Method: GET 2024-04-12T03:58:47.641 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000020 2024-04-12T03:58:47.641 DEBUG:boto:Data: 2024-04-12T03:58:47.641 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.641 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.641 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.641 DEBUG:boto:Params: {} 2024-04-12T03:58:47.641 DEBUG:boto:Token: None 2024-04-12T03:58:47.642 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.642 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fbb30dcb61dfb8d-006618b177-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.644 DEBUG:boto:b'dkyges-60obj000000000000000000201000falseobj000000000000000000202024-04-12T03:58:33.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.645 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.645 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.645 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000020 2024-04-12T03:58:47.645 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000020 2024-04-12T03:58:47.645 DEBUG:boto:Method: GET 2024-04-12T03:58:47.645 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000020 2024-04-12T03:58:47.645 DEBUG:boto:Data: 2024-04-12T03:58:47.645 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.645 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.646 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.646 DEBUG:boto:Params: {} 2024-04-12T03:58:47.646 DEBUG:boto:Token: None 2024-04-12T03:58:47.646 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.646 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3081987056ace90-006618b177-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.650 DEBUG:boto:b'dkyges-60obj000000000000000000201000falseobj000000000000000000202024-04-12T03:58:33.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.651 DEBUG:boto:path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.651 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.651 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.651 DEBUG:boto:Path: /dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.651 DEBUG:boto:Data: 2024-04-12T03:58:47.651 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.651 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.652 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.652 DEBUG:boto:Params: {} 2024-04-12T03:58:47.652 DEBUG:boto:Token: None 2024-04-12T03:58:47.652 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.652 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fb1wbV640ToZD8RBld9Z/lAX1ec= 2024-04-12T03:58:47.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fb1wbV640ToZD8RBld9Z/lAX1ec=', 'Content-Length': '0'} 2024-04-12T03:58:47.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094c577eb8d7490b7-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.654 DEBUG:boto:path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.655 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.655 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.655 DEBUG:boto:Path: /dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.655 DEBUG:boto:Data: 2024-04-12T03:58:47.655 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.656 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.656 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.656 DEBUG:boto:Params: {} 2024-04-12T03:58:47.656 DEBUG:boto:Token: None 2024-04-12T03:58:47.656 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fb1wbV640ToZD8RBld9Z/lAX1ec= 2024-04-12T03:58:47.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fb1wbV640ToZD8RBld9Z/lAX1ec=', 'Content-Length': '0'} 2024-04-12T03:58:47.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8fce30b214bf2e3-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2024-04-12T03:58:47.658 DEBUG:boto:path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.659 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.659 DEBUG:boto:Method: GET 2024-04-12T03:58:47.659 DEBUG:boto:Path: /dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.659 DEBUG:boto:Data: 2024-04-12T03:58:47.659 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.659 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.659 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.659 DEBUG:boto:Params: {} 2024-04-12T03:58:47.659 DEBUG:boto:Token: None 2024-04-12T03:58:47.659 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.660 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Kmig41CfG6RbsFsJKccO5/Wq2j0= 2024-04-12T03:58:47.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Kmig41CfG6RbsFsJKccO5/Wq2j0=', 'Content-Length': '0'} 2024-04-12T03:58:47.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdc8a8ec0a74a9b9-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.662 DEBUG:boto:path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.662 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.662 DEBUG:boto:Method: GET 2024-04-12T03:58:47.662 DEBUG:boto:Path: /dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.662 DEBUG:boto:Data: 2024-04-12T03:58:47.662 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.662 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.662 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.663 DEBUG:boto:Params: {} 2024-04-12T03:58:47.663 DEBUG:boto:Token: None 2024-04-12T03:58:47.663 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000020 2024-04-12T03:58:47.663 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Kmig41CfG6RbsFsJKccO5/Wq2j0= 2024-04-12T03:58:47.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Kmig41CfG6RbsFsJKccO5/Wq2j0=', 'Content-Length': '0'} 2024-04-12T03:58:47.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000967f6106af7dd91-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.665 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.665 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.666 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000020 2024-04-12T03:58:47.666 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000020 2024-04-12T03:58:47.666 DEBUG:boto:Method: GET 2024-04-12T03:58:47.666 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000020 2024-04-12T03:58:47.666 DEBUG:boto:Data: 2024-04-12T03:58:47.666 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.666 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.666 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.666 DEBUG:boto:Params: {} 2024-04-12T03:58:47.666 DEBUG:boto:Token: None 2024-04-12T03:58:47.667 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.667 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2fe4d043eee0388-006618b177-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.669 DEBUG:boto:b'dkyges-60obj000000000000000000201000falseobj000000000000000000202024-04-12T03:58:33.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.670 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.670 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.670 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000020 2024-04-12T03:58:47.670 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000020 2024-04-12T03:58:47.670 DEBUG:boto:Method: GET 2024-04-12T03:58:47.670 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000020 2024-04-12T03:58:47.670 DEBUG:boto:Data: 2024-04-12T03:58:47.670 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.670 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.671 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.671 DEBUG:boto:Params: {} 2024-04-12T03:58:47.671 DEBUG:boto:Token: None 2024-04-12T03:58:47.671 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.671 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042dc5b34965f85da-006618b177-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.676 DEBUG:boto:b'dkyges-60obj000000000000000000201000falseobj000000000000000000202024-04-12T03:58:33.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-04-12T03:58:47.676 DEBUG:boto:path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.676 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.676 DEBUG:boto:path=/dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.676 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.676 DEBUG:boto:Method: GET 2024-04-12T03:58:47.677 DEBUG:boto:Path: /dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.677 DEBUG:boto:Data: 2024-04-12T03:58:47.677 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.677 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.677 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.677 DEBUG:boto:Params: {} 2024-04-12T03:58:47.677 DEBUG:boto:Token: None 2024-04-12T03:58:47.677 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.677 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:24qrohyIWjHrJId9aKIZWuEm8d8= 2024-04-12T03:58:47.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:24qrohyIWjHrJId9aKIZWuEm8d8=', 'Content-Length': '0'} 2024-04-12T03:58:47.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000777337ccfcddcb06-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.680 DEBUG:boto:path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.680 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.680 DEBUG:boto:path=/dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.680 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.680 DEBUG:boto:Method: GET 2024-04-12T03:58:47.680 DEBUG:boto:Path: /dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.680 DEBUG:boto:Data: 2024-04-12T03:58:47.680 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.680 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.680 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.680 DEBUG:boto:Params: {} 2024-04-12T03:58:47.681 DEBUG:boto:Token: None 2024-04-12T03:58:47.681 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.681 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:24qrohyIWjHrJId9aKIZWuEm8d8= 2024-04-12T03:58:47.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:24qrohyIWjHrJId9aKIZWuEm8d8=', 'Content-Length': '0'} 2024-04-12T03:58:47.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3200dddae60260b-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.726 DEBUG:boto:path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.726 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.727 DEBUG:boto:path=/dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.727 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.727 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.727 DEBUG:boto:Path: /dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.727 DEBUG:boto:Data: 2024-04-12T03:58:47.727 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.727 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.727 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.727 DEBUG:boto:Params: {} 2024-04-12T03:58:47.727 DEBUG:boto:Token: None 2024-04-12T03:58:47.728 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.728 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vs82I+Yb4CO+40i3cuvaEmlNwc4= 2024-04-12T03:58:47.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vs82I+Yb4CO+40i3cuvaEmlNwc4=', 'Content-Length': '0'} 2024-04-12T03:58:47.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f849f0f99bca39c-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.730 DEBUG:boto:path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.730 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.730 DEBUG:boto:path=/dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.730 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.730 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.730 DEBUG:boto:Path: /dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.730 DEBUG:boto:Data: 2024-04-12T03:58:47.730 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.730 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.730 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.731 DEBUG:boto:Params: {} 2024-04-12T03:58:47.731 DEBUG:boto:Token: None 2024-04-12T03:58:47.731 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.731 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vs82I+Yb4CO+40i3cuvaEmlNwc4= 2024-04-12T03:58:47.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vs82I+Yb4CO+40i3cuvaEmlNwc4=', 'Content-Length': '0'} 2024-04-12T03:58:47.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c020add04278f71-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-04-12T03:58:47.733 DEBUG:boto:path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.733 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.733 DEBUG:boto:path=/dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.733 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.733 DEBUG:boto:Method: GET 2024-04-12T03:58:47.733 DEBUG:boto:Path: /dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.734 DEBUG:boto:Data: 2024-04-12T03:58:47.734 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.734 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.734 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.734 DEBUG:boto:Params: {} 2024-04-12T03:58:47.734 DEBUG:boto:Token: None 2024-04-12T03:58:47.734 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.734 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:24qrohyIWjHrJId9aKIZWuEm8d8= 2024-04-12T03:58:47.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:24qrohyIWjHrJId9aKIZWuEm8d8=', 'Content-Length': '0'} 2024-04-12T03:58:47.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000411ef3d329b8bd62-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.736 DEBUG:boto:path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.737 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.737 DEBUG:boto:path=/dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.737 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.737 DEBUG:boto:Method: GET 2024-04-12T03:58:47.737 DEBUG:boto:Path: /dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.737 DEBUG:boto:Data: 2024-04-12T03:58:47.737 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.737 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.737 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.737 DEBUG:boto:Params: {} 2024-04-12T03:58:47.737 DEBUG:boto:Token: None 2024-04-12T03:58:47.738 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000021?versionId=null 2024-04-12T03:58:47.738 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:24qrohyIWjHrJId9aKIZWuEm8d8= 2024-04-12T03:58:47.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:24qrohyIWjHrJId9aKIZWuEm8d8=', 'Content-Length': '0'} 2024-04-12T03:58:47.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b72152627ccd0101-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.740 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.740 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.740 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000021 2024-04-12T03:58:47.740 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000021 2024-04-12T03:58:47.740 DEBUG:boto:Method: GET 2024-04-12T03:58:47.740 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000021 2024-04-12T03:58:47.740 DEBUG:boto:Data: 2024-04-12T03:58:47.740 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.741 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.741 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.741 DEBUG:boto:Params: {} 2024-04-12T03:58:47.741 DEBUG:boto:Token: None 2024-04-12T03:58:47.741 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.741 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecb0441b1dd145b8-006618b177-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.744 DEBUG:boto:b'dkyges-60obj000000000000000000211000falseobj000000000000000000212024-04-12T03:58:35.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.744 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.745 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.745 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000021 2024-04-12T03:58:47.745 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000021 2024-04-12T03:58:47.745 DEBUG:boto:Method: GET 2024-04-12T03:58:47.745 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000021 2024-04-12T03:58:47.745 DEBUG:boto:Data: 2024-04-12T03:58:47.745 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.745 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.746 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.746 DEBUG:boto:Params: {} 2024-04-12T03:58:47.746 DEBUG:boto:Token: None 2024-04-12T03:58:47.746 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.746 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040ea274595c933c5-006618b177-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.751 DEBUG:boto:b'dkyges-60obj000000000000000000211000falseobj000000000000000000212024-04-12T03:58:35.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.752 DEBUG:boto:path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.752 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.752 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.752 DEBUG:boto:Path: /dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.752 DEBUG:boto:Data: 2024-04-12T03:58:47.752 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.752 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.752 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.752 DEBUG:boto:Params: {} 2024-04-12T03:58:47.752 DEBUG:boto:Token: None 2024-04-12T03:58:47.753 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.753 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L3PwR8QT3VqpiD0+ocVK4bELloA= 2024-04-12T03:58:47.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L3PwR8QT3VqpiD0+ocVK4bELloA=', 'Content-Length': '0'} 2024-04-12T03:58:47.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000949fb2c62ea7a4df-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.755 DEBUG:boto:path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.755 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.755 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.755 DEBUG:boto:Path: /dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.755 DEBUG:boto:Data: 2024-04-12T03:58:47.755 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.755 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.756 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.756 DEBUG:boto:Params: {} 2024-04-12T03:58:47.756 DEBUG:boto:Token: None 2024-04-12T03:58:47.756 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.756 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L3PwR8QT3VqpiD0+ocVK4bELloA= 2024-04-12T03:58:47.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L3PwR8QT3VqpiD0+ocVK4bELloA=', 'Content-Length': '0'} 2024-04-12T03:58:47.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005376dd7a73b3111e-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2024-04-12T03:58:47.758 DEBUG:boto:path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.759 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.759 DEBUG:boto:Method: GET 2024-04-12T03:58:47.759 DEBUG:boto:Path: /dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.759 DEBUG:boto:Data: 2024-04-12T03:58:47.759 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.759 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.759 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.759 DEBUG:boto:Params: {} 2024-04-12T03:58:47.759 DEBUG:boto:Token: None 2024-04-12T03:58:47.760 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.760 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xao/8ZUFMU4SAmIOO5LyNlsnuQc= 2024-04-12T03:58:47.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xao/8ZUFMU4SAmIOO5LyNlsnuQc=', 'Content-Length': '0'} 2024-04-12T03:58:47.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a14ed7f0794cee05-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.802 DEBUG:boto:path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.802 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.803 DEBUG:boto:Method: GET 2024-04-12T03:58:47.803 DEBUG:boto:Path: /dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.803 DEBUG:boto:Data: 2024-04-12T03:58:47.803 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.803 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.803 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.803 DEBUG:boto:Params: {} 2024-04-12T03:58:47.803 DEBUG:boto:Token: None 2024-04-12T03:58:47.803 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000021 2024-04-12T03:58:47.803 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xao/8ZUFMU4SAmIOO5LyNlsnuQc= 2024-04-12T03:58:47.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xao/8ZUFMU4SAmIOO5LyNlsnuQc=', 'Content-Length': '0'} 2024-04-12T03:58:47.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5871b949c2ec468-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.806 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.806 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.806 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000021 2024-04-12T03:58:47.806 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000021 2024-04-12T03:58:47.806 DEBUG:boto:Method: GET 2024-04-12T03:58:47.806 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000021 2024-04-12T03:58:47.806 DEBUG:boto:Data: 2024-04-12T03:58:47.806 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.807 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.807 DEBUG:boto:Params: {} 2024-04-12T03:58:47.807 DEBUG:boto:Token: None 2024-04-12T03:58:47.807 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.807 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000462568c26b6d1e45-006618b177-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.810 DEBUG:boto:b'dkyges-60obj000000000000000000211000falseobj000000000000000000212024-04-12T03:58:35.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.810 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.810 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.810 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000021 2024-04-12T03:58:47.811 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000021 2024-04-12T03:58:47.811 DEBUG:boto:Method: GET 2024-04-12T03:58:47.811 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000021 2024-04-12T03:58:47.811 DEBUG:boto:Data: 2024-04-12T03:58:47.811 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.811 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.811 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.811 DEBUG:boto:Params: {} 2024-04-12T03:58:47.811 DEBUG:boto:Token: None 2024-04-12T03:58:47.811 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.811 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bc259a95a10fbcd-006618b177-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.816 DEBUG:boto:b'dkyges-60obj000000000000000000211000falseobj000000000000000000212024-04-12T03:58:35.784Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-04-12T03:58:47.816 DEBUG:boto:path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.817 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.817 DEBUG:boto:path=/dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.817 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.817 DEBUG:boto:Method: GET 2024-04-12T03:58:47.817 DEBUG:boto:Path: /dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.817 DEBUG:boto:Data: 2024-04-12T03:58:47.817 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.817 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.817 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.817 DEBUG:boto:Params: {} 2024-04-12T03:58:47.817 DEBUG:boto:Token: None 2024-04-12T03:58:47.817 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.818 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h55dekZ+4VzD2xfZaqDAIk/u/0k= 2024-04-12T03:58:47.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h55dekZ+4VzD2xfZaqDAIk/u/0k=', 'Content-Length': '0'} 2024-04-12T03:58:47.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfe8465a75280b68-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.820 DEBUG:boto:path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.820 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.820 DEBUG:boto:path=/dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.820 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.820 DEBUG:boto:Method: GET 2024-04-12T03:58:47.820 DEBUG:boto:Path: /dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.821 DEBUG:boto:Data: 2024-04-12T03:58:47.821 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.821 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.821 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.821 DEBUG:boto:Params: {} 2024-04-12T03:58:47.821 DEBUG:boto:Token: None 2024-04-12T03:58:47.821 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.821 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h55dekZ+4VzD2xfZaqDAIk/u/0k= 2024-04-12T03:58:47.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h55dekZ+4VzD2xfZaqDAIk/u/0k=', 'Content-Length': '0'} 2024-04-12T03:58:47.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb93016bd23c70fe-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.866 DEBUG:boto:path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.866 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.867 DEBUG:boto:path=/dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.867 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.867 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.867 DEBUG:boto:Path: /dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.867 DEBUG:boto:Data: 2024-04-12T03:58:47.867 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.867 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.867 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.867 DEBUG:boto:Params: {} 2024-04-12T03:58:47.867 DEBUG:boto:Token: None 2024-04-12T03:58:47.867 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.868 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OVQTAXa0lde5oEFOFSCki1i/9bE= 2024-04-12T03:58:47.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OVQTAXa0lde5oEFOFSCki1i/9bE=', 'Content-Length': '0'} 2024-04-12T03:58:47.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcca5a6c5ec71bd3-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.870 DEBUG:boto:path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.870 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.870 DEBUG:boto:path=/dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.870 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.870 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.870 DEBUG:boto:Path: /dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.871 DEBUG:boto:Data: 2024-04-12T03:58:47.871 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.871 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.871 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.871 DEBUG:boto:Params: {} 2024-04-12T03:58:47.871 DEBUG:boto:Token: None 2024-04-12T03:58:47.871 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.871 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OVQTAXa0lde5oEFOFSCki1i/9bE= 2024-04-12T03:58:47.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OVQTAXa0lde5oEFOFSCki1i/9bE=', 'Content-Length': '0'} 2024-04-12T03:58:47.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c22a0b4ba412cf5-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-04-12T03:58:47.874 DEBUG:boto:path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.874 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.874 DEBUG:boto:path=/dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.874 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.874 DEBUG:boto:Method: GET 2024-04-12T03:58:47.874 DEBUG:boto:Path: /dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.874 DEBUG:boto:Data: 2024-04-12T03:58:47.874 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.875 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.875 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.875 DEBUG:boto:Params: {} 2024-04-12T03:58:47.875 DEBUG:boto:Token: None 2024-04-12T03:58:47.875 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.875 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h55dekZ+4VzD2xfZaqDAIk/u/0k= 2024-04-12T03:58:47.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h55dekZ+4VzD2xfZaqDAIk/u/0k=', 'Content-Length': '0'} 2024-04-12T03:58:47.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff9ffb8641f860b8-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.877 DEBUG:boto:path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.877 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.878 DEBUG:boto:path=/dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.878 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.878 DEBUG:boto:Method: GET 2024-04-12T03:58:47.878 DEBUG:boto:Path: /dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.878 DEBUG:boto:Data: 2024-04-12T03:58:47.878 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.878 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.878 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.878 DEBUG:boto:Params: {} 2024-04-12T03:58:47.879 DEBUG:boto:Token: None 2024-04-12T03:58:47.879 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000022?versionId=null 2024-04-12T03:58:47.879 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h55dekZ+4VzD2xfZaqDAIk/u/0k= 2024-04-12T03:58:47.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h55dekZ+4VzD2xfZaqDAIk/u/0k=', 'Content-Length': '0'} 2024-04-12T03:58:47.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000654b1d8db7d7a38b-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.881 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.881 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.881 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000022 2024-04-12T03:58:47.882 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000022 2024-04-12T03:58:47.882 DEBUG:boto:Method: GET 2024-04-12T03:58:47.882 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000022 2024-04-12T03:58:47.882 DEBUG:boto:Data: 2024-04-12T03:58:47.882 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.882 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.882 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.882 DEBUG:boto:Params: {} 2024-04-12T03:58:47.883 DEBUG:boto:Token: None 2024-04-12T03:58:47.883 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.883 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2a353f8f7321986-006618b177-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.886 DEBUG:boto:b'dkyges-60obj000000000000000000221000falseobj000000000000000000222024-04-12T03:58:33.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.886 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.887 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.887 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000022 2024-04-12T03:58:47.887 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000022 2024-04-12T03:58:47.887 DEBUG:boto:Method: GET 2024-04-12T03:58:47.887 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000022 2024-04-12T03:58:47.887 DEBUG:boto:Data: 2024-04-12T03:58:47.887 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.887 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.887 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.888 DEBUG:boto:Params: {} 2024-04-12T03:58:47.888 DEBUG:boto:Token: None 2024-04-12T03:58:47.888 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.888 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fd398e8da6ac6d4-006618b177-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.893 DEBUG:boto:b'dkyges-60obj000000000000000000221000falseobj000000000000000000222024-04-12T03:58:33.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.894 DEBUG:boto:path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.894 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.894 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.894 DEBUG:boto:Path: /dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.894 DEBUG:boto:Data: 2024-04-12T03:58:47.894 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.895 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.895 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.895 DEBUG:boto:Params: {} 2024-04-12T03:58:47.895 DEBUG:boto:Token: None 2024-04-12T03:58:47.895 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.895 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xw3ohBB3HIQxnKjfkbmJbY69bZY= 2024-04-12T03:58:47.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xw3ohBB3HIQxnKjfkbmJbY69bZY=', 'Content-Length': '0'} 2024-04-12T03:58:47.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006061a06e515254e6-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.897 DEBUG:boto:path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.898 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.898 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.898 DEBUG:boto:Path: /dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.898 DEBUG:boto:Data: 2024-04-12T03:58:47.898 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.898 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.898 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.898 DEBUG:boto:Params: {} 2024-04-12T03:58:47.898 DEBUG:boto:Token: None 2024-04-12T03:58:47.899 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.899 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xw3ohBB3HIQxnKjfkbmJbY69bZY= 2024-04-12T03:58:47.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xw3ohBB3HIQxnKjfkbmJbY69bZY=', 'Content-Length': '0'} 2024-04-12T03:58:47.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000201b57e345e134d8-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2024-04-12T03:58:47.901 DEBUG:boto:path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.901 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.901 DEBUG:boto:Method: GET 2024-04-12T03:58:47.901 DEBUG:boto:Path: /dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.902 DEBUG:boto:Data: 2024-04-12T03:58:47.902 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.902 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.902 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.902 DEBUG:boto:Params: {} 2024-04-12T03:58:47.902 DEBUG:boto:Token: None 2024-04-12T03:58:47.902 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.902 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vMym2BmPT3SotL5799gidHFqKlE= 2024-04-12T03:58:47.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vMym2BmPT3SotL5799gidHFqKlE=', 'Content-Length': '0'} 2024-04-12T03:58:47.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2c6d790edcc477b-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.905 DEBUG:boto:path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.905 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.905 DEBUG:boto:Method: GET 2024-04-12T03:58:47.905 DEBUG:boto:Path: /dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.905 DEBUG:boto:Data: 2024-04-12T03:58:47.905 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.905 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.905 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.905 DEBUG:boto:Params: {} 2024-04-12T03:58:47.906 DEBUG:boto:Token: None 2024-04-12T03:58:47.906 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000022 2024-04-12T03:58:47.906 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vMym2BmPT3SotL5799gidHFqKlE= 2024-04-12T03:58:47.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vMym2BmPT3SotL5799gidHFqKlE=', 'Content-Length': '0'} 2024-04-12T03:58:47.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066cd77195b57e9ea-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.908 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.908 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.908 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000022 2024-04-12T03:58:47.909 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000022 2024-04-12T03:58:47.909 DEBUG:boto:Method: GET 2024-04-12T03:58:47.909 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000022 2024-04-12T03:58:47.909 DEBUG:boto:Data: 2024-04-12T03:58:47.909 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.909 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.909 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.909 DEBUG:boto:Params: {} 2024-04-12T03:58:47.909 DEBUG:boto:Token: None 2024-04-12T03:58:47.910 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.910 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007280fae919b1cee-006618b177-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.912 DEBUG:boto:b'dkyges-60obj000000000000000000221000falseobj000000000000000000222024-04-12T03:58:33.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.913 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.913 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.913 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000022 2024-04-12T03:58:47.913 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000022 2024-04-12T03:58:47.913 DEBUG:boto:Method: GET 2024-04-12T03:58:47.913 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000022 2024-04-12T03:58:47.913 DEBUG:boto:Data: 2024-04-12T03:58:47.913 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.914 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.914 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.914 DEBUG:boto:Params: {} 2024-04-12T03:58:47.914 DEBUG:boto:Token: None 2024-04-12T03:58:47.914 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.914 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000273cb90c6a3f8ea8-006618b177-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.919 DEBUG:boto:b'dkyges-60obj000000000000000000221000falseobj000000000000000000222024-04-12T03:58:33.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-04-12T03:58:47.919 DEBUG:boto:path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.919 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.919 DEBUG:boto:path=/dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.920 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.920 DEBUG:boto:Method: GET 2024-04-12T03:58:47.920 DEBUG:boto:Path: /dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.920 DEBUG:boto:Data: 2024-04-12T03:58:47.920 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.920 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.920 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.920 DEBUG:boto:Params: {} 2024-04-12T03:58:47.920 DEBUG:boto:Token: None 2024-04-12T03:58:47.921 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.921 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6boBzOw+5jAEYa896W3VsTw8aQk= 2024-04-12T03:58:47.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6boBzOw+5jAEYa896W3VsTw8aQk=', 'Content-Length': '0'} 2024-04-12T03:58:47.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e5614e3319282f8-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.923 DEBUG:boto:path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.923 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.923 DEBUG:boto:path=/dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.924 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.924 DEBUG:boto:Method: GET 2024-04-12T03:58:47.924 DEBUG:boto:Path: /dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.924 DEBUG:boto:Data: 2024-04-12T03:58:47.924 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.924 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.924 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.924 DEBUG:boto:Params: {} 2024-04-12T03:58:47.924 DEBUG:boto:Token: None 2024-04-12T03:58:47.924 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.925 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6boBzOw+5jAEYa896W3VsTw8aQk= 2024-04-12T03:58:47.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6boBzOw+5jAEYa896W3VsTw8aQk=', 'Content-Length': '0'} 2024-04-12T03:58:47.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ae914692cd250a9-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.970 DEBUG:boto:path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.970 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.970 DEBUG:boto:path=/dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.971 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.971 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.971 DEBUG:boto:Path: /dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.971 DEBUG:boto:Data: 2024-04-12T03:58:47.971 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.971 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.971 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.971 DEBUG:boto:Params: {} 2024-04-12T03:58:47.971 DEBUG:boto:Token: None 2024-04-12T03:58:47.971 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.971 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RqoQlOC6fZgk5FhY8AKSlfyfFzc= 2024-04-12T03:58:47.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RqoQlOC6fZgk5FhY8AKSlfyfFzc=', 'Content-Length': '0'} 2024-04-12T03:58:47.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064a28f75c66847ee-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.974 DEBUG:boto:path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.974 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.974 DEBUG:boto:path=/dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.974 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.974 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.974 DEBUG:boto:Path: /dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.974 DEBUG:boto:Data: 2024-04-12T03:58:47.974 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.974 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.975 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.975 DEBUG:boto:Params: {} 2024-04-12T03:58:47.975 DEBUG:boto:Token: None 2024-04-12T03:58:47.975 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.975 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RqoQlOC6fZgk5FhY8AKSlfyfFzc= 2024-04-12T03:58:47.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RqoQlOC6fZgk5FhY8AKSlfyfFzc=', 'Content-Length': '0'} 2024-04-12T03:58:47.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005cbd71d7c008e9f-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-04-12T03:58:47.977 DEBUG:boto:path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.977 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.977 DEBUG:boto:path=/dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.977 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.978 DEBUG:boto:Method: GET 2024-04-12T03:58:47.978 DEBUG:boto:Path: /dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.978 DEBUG:boto:Data: 2024-04-12T03:58:47.978 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.978 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.978 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.978 DEBUG:boto:Params: {} 2024-04-12T03:58:47.978 DEBUG:boto:Token: None 2024-04-12T03:58:47.978 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.978 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6boBzOw+5jAEYa896W3VsTw8aQk= 2024-04-12T03:58:47.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6boBzOw+5jAEYa896W3VsTw8aQk=', 'Content-Length': '0'} 2024-04-12T03:58:47.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f81b32ff22f94cfc-006618b177-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.981 DEBUG:boto:path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.981 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.981 DEBUG:boto:path=/dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.981 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.981 DEBUG:boto:Method: GET 2024-04-12T03:58:47.981 DEBUG:boto:Path: /dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.981 DEBUG:boto:Data: 2024-04-12T03:58:47.981 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.981 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.981 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.981 DEBUG:boto:Params: {} 2024-04-12T03:58:47.981 DEBUG:boto:Token: None 2024-04-12T03:58:47.982 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000023?versionId=null 2024-04-12T03:58:47.982 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6boBzOw+5jAEYa896W3VsTw8aQk= 2024-04-12T03:58:47.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6boBzOw+5jAEYa896W3VsTw8aQk=', 'Content-Length': '0'} 2024-04-12T03:58:47.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033c8e6954611e021-006618b177-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.984 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.984 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.984 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000023 2024-04-12T03:58:47.984 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000023 2024-04-12T03:58:47.984 DEBUG:boto:Method: GET 2024-04-12T03:58:47.985 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000023 2024-04-12T03:58:47.985 DEBUG:boto:Data: 2024-04-12T03:58:47.985 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.985 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.985 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.985 DEBUG:boto:Params: {} 2024-04-12T03:58:47.985 DEBUG:boto:Token: None 2024-04-12T03:58:47.985 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.985 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bea1f9effdbae01a-006618b177-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.988 DEBUG:boto:b'dkyges-60obj000000000000000000231000falseobj000000000000000000232024-04-12T03:58:34.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.988 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:47.988 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:47.988 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000023 2024-04-12T03:58:47.988 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000023 2024-04-12T03:58:47.989 DEBUG:boto:Method: GET 2024-04-12T03:58:47.989 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000023 2024-04-12T03:58:47.989 DEBUG:boto:Data: 2024-04-12T03:58:47.989 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.989 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.989 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.989 DEBUG:boto:Params: {} 2024-04-12T03:58:47.989 DEBUG:boto:Token: None 2024-04-12T03:58:47.989 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/ 2024-04-12T03:58:47.989 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8= 2024-04-12T03:58:47.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8oRm9uRB+s7KvnQ0jdlVEVewMd8=', 'Content-Length': '0'} 2024-04-12T03:58:47.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bb22766abab8405-006618b178-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.994 DEBUG:boto:b'dkyges-60obj000000000000000000231000falseobj000000000000000000232024-04-12T03:58:34.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:47.995 DEBUG:boto:path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.995 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.995 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.995 DEBUG:boto:Path: /dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.995 DEBUG:boto:Data: 2024-04-12T03:58:47.995 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.995 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:47.995 DEBUG:boto:Port: 8001 2024-04-12T03:58:47.995 DEBUG:boto:Params: {} 2024-04-12T03:58:47.995 DEBUG:boto:Token: None 2024-04-12T03:58:47.996 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.996 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gQAGsCRvlLtjMZBqbbKol3K/bSs= 2024-04-12T03:58:47.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gQAGsCRvlLtjMZBqbbKol3K/bSs=', 'Content-Length': '0'} 2024-04-12T03:58:47.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae536f8416318828-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:47.998 DEBUG:boto:path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.998 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.998 DEBUG:boto:Method: HEAD 2024-04-12T03:58:47.998 DEBUG:boto:Path: /dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.998 DEBUG:boto:Data: 2024-04-12T03:58:47.998 DEBUG:boto:Headers: {} 2024-04-12T03:58:47.998 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:47.998 DEBUG:boto:Port: 8000 2024-04-12T03:58:47.998 DEBUG:boto:Params: {} 2024-04-12T03:58:47.998 DEBUG:boto:Token: None 2024-04-12T03:58:47.999 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:47 GMT /dkyges-60/obj00000000000000000023 2024-04-12T03:58:47.999 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gQAGsCRvlLtjMZBqbbKol3K/bSs= 2024-04-12T03:58:47.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gQAGsCRvlLtjMZBqbbKol3K/bSs=', 'Content-Length': '0'} 2024-04-12T03:58:48.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e46c525af5781bbe-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2024-04-12T03:58:48.001 DEBUG:boto:path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:48.001 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:48.001 DEBUG:boto:Method: GET 2024-04-12T03:58:48.001 DEBUG:boto:Path: /dkyges-60/obj00000000000000000023 2024-04-12T03:58:48.001 DEBUG:boto:Data: 2024-04-12T03:58:48.001 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.002 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.002 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.002 DEBUG:boto:Params: {} 2024-04-12T03:58:48.002 DEBUG:boto:Token: None 2024-04-12T03:58:48.002 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000023 2024-04-12T03:58:48.002 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W+U+65AxGsiAvHsr3cMpiDMFmCU= 2024-04-12T03:58:48.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W+U+65AxGsiAvHsr3cMpiDMFmCU=', 'Content-Length': '0'} 2024-04-12T03:58:48.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ff6ac4d377b4267-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.004 DEBUG:boto:path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:48.004 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000023 2024-04-12T03:58:48.004 DEBUG:boto:Method: GET 2024-04-12T03:58:48.004 DEBUG:boto:Path: /dkyges-60/obj00000000000000000023 2024-04-12T03:58:48.005 DEBUG:boto:Data: 2024-04-12T03:58:48.005 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.005 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.005 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.005 DEBUG:boto:Params: {} 2024-04-12T03:58:48.005 DEBUG:boto:Token: None 2024-04-12T03:58:48.005 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000023 2024-04-12T03:58:48.005 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W+U+65AxGsiAvHsr3cMpiDMFmCU= 2024-04-12T03:58:48.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W+U+65AxGsiAvHsr3cMpiDMFmCU=', 'Content-Length': '0'} 2024-04-12T03:58:48.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032e169720e1e8870-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.008 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.008 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.008 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000023 2024-04-12T03:58:48.008 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000023 2024-04-12T03:58:48.008 DEBUG:boto:Method: GET 2024-04-12T03:58:48.008 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000023 2024-04-12T03:58:48.008 DEBUG:boto:Data: 2024-04-12T03:58:48.008 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.009 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.009 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.009 DEBUG:boto:Params: {} 2024-04-12T03:58:48.009 DEBUG:boto:Token: None 2024-04-12T03:58:48.009 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.009 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8112af3ddf3be77-006618b178-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.012 DEBUG:boto:b'dkyges-60obj000000000000000000231000falseobj000000000000000000232024-04-12T03:58:34.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.012 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.012 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.012 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000023 2024-04-12T03:58:48.012 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000023 2024-04-12T03:58:48.012 DEBUG:boto:Method: GET 2024-04-12T03:58:48.013 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000023 2024-04-12T03:58:48.013 DEBUG:boto:Data: 2024-04-12T03:58:48.013 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.013 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.013 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.013 DEBUG:boto:Params: {} 2024-04-12T03:58:48.013 DEBUG:boto:Token: None 2024-04-12T03:58:48.013 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.013 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000616a15cd71d3e06b-006618b178-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.058 DEBUG:boto:b'dkyges-60obj000000000000000000231000falseobj000000000000000000232024-04-12T03:58:34.220Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-04-12T03:58:48.059 DEBUG:boto:path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.059 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.059 DEBUG:boto:path=/dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.059 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.059 DEBUG:boto:Method: GET 2024-04-12T03:58:48.059 DEBUG:boto:Path: /dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.059 DEBUG:boto:Data: 2024-04-12T03:58:48.059 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.060 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.060 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.060 DEBUG:boto:Params: {} 2024-04-12T03:58:48.060 DEBUG:boto:Token: None 2024-04-12T03:58:48.060 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.061 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4QJdwX72JK1t/hQfmg8Tnj4j9iQ= 2024-04-12T03:58:48.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4QJdwX72JK1t/hQfmg8Tnj4j9iQ=', 'Content-Length': '0'} 2024-04-12T03:58:48.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fdc1b7497ae9a37-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.063 DEBUG:boto:path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.063 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.064 DEBUG:boto:path=/dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.064 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.064 DEBUG:boto:Method: GET 2024-04-12T03:58:48.064 DEBUG:boto:Path: /dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.064 DEBUG:boto:Data: 2024-04-12T03:58:48.064 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.064 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.064 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.064 DEBUG:boto:Params: {} 2024-04-12T03:58:48.064 DEBUG:boto:Token: None 2024-04-12T03:58:48.065 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.065 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4QJdwX72JK1t/hQfmg8Tnj4j9iQ= 2024-04-12T03:58:48.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4QJdwX72JK1t/hQfmg8Tnj4j9iQ=', 'Content-Length': '0'} 2024-04-12T03:58:48.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f45e240acfbc00d-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.067 DEBUG:boto:path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.067 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.067 DEBUG:boto:path=/dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.067 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.067 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.067 DEBUG:boto:Path: /dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.067 DEBUG:boto:Data: 2024-04-12T03:58:48.067 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.068 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.068 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.068 DEBUG:boto:Params: {} 2024-04-12T03:58:48.068 DEBUG:boto:Token: None 2024-04-12T03:58:48.068 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.068 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C6SognhaN1ixtUQcM3AuWbD32Xg= 2024-04-12T03:58:48.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C6SognhaN1ixtUQcM3AuWbD32Xg=', 'Content-Length': '0'} 2024-04-12T03:58:48.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000675e6a4031ab28c5-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.070 DEBUG:boto:path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.070 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.070 DEBUG:boto:path=/dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.070 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.070 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.070 DEBUG:boto:Path: /dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.071 DEBUG:boto:Data: 2024-04-12T03:58:48.071 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.071 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.071 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.071 DEBUG:boto:Params: {} 2024-04-12T03:58:48.071 DEBUG:boto:Token: None 2024-04-12T03:58:48.071 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.071 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C6SognhaN1ixtUQcM3AuWbD32Xg= 2024-04-12T03:58:48.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C6SognhaN1ixtUQcM3AuWbD32Xg=', 'Content-Length': '0'} 2024-04-12T03:58:48.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059f53cac7ef1323b-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-04-12T03:58:48.073 DEBUG:boto:path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.073 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.074 DEBUG:boto:path=/dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.074 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.074 DEBUG:boto:Method: GET 2024-04-12T03:58:48.074 DEBUG:boto:Path: /dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.074 DEBUG:boto:Data: 2024-04-12T03:58:48.074 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.074 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.074 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.074 DEBUG:boto:Params: {} 2024-04-12T03:58:48.074 DEBUG:boto:Token: None 2024-04-12T03:58:48.074 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.075 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4QJdwX72JK1t/hQfmg8Tnj4j9iQ= 2024-04-12T03:58:48.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4QJdwX72JK1t/hQfmg8Tnj4j9iQ=', 'Content-Length': '0'} 2024-04-12T03:58:48.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9adb89051acbb0a-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.119 DEBUG:boto:path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.119 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.119 DEBUG:boto:path=/dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.119 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.119 DEBUG:boto:Method: GET 2024-04-12T03:58:48.119 DEBUG:boto:Path: /dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.119 DEBUG:boto:Data: 2024-04-12T03:58:48.119 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.119 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.120 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.120 DEBUG:boto:Params: {} 2024-04-12T03:58:48.120 DEBUG:boto:Token: None 2024-04-12T03:58:48.120 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000024?versionId=null 2024-04-12T03:58:48.120 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4QJdwX72JK1t/hQfmg8Tnj4j9iQ= 2024-04-12T03:58:48.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4QJdwX72JK1t/hQfmg8Tnj4j9iQ=', 'Content-Length': '0'} 2024-04-12T03:58:48.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019fdb7a8d852ee29-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.122 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.122 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.122 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000024 2024-04-12T03:58:48.123 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000024 2024-04-12T03:58:48.123 DEBUG:boto:Method: GET 2024-04-12T03:58:48.123 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000024 2024-04-12T03:58:48.123 DEBUG:boto:Data: 2024-04-12T03:58:48.123 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.123 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.123 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.123 DEBUG:boto:Params: {} 2024-04-12T03:58:48.123 DEBUG:boto:Token: None 2024-04-12T03:58:48.123 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.124 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086feb8535df61700-006618b178-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.126 DEBUG:boto:b'dkyges-60obj000000000000000000241000falseobj000000000000000000242024-04-12T03:58:34.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.127 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.127 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.127 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000024 2024-04-12T03:58:48.127 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000024 2024-04-12T03:58:48.127 DEBUG:boto:Method: GET 2024-04-12T03:58:48.127 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000024 2024-04-12T03:58:48.127 DEBUG:boto:Data: 2024-04-12T03:58:48.127 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.127 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.127 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.127 DEBUG:boto:Params: {} 2024-04-12T03:58:48.128 DEBUG:boto:Token: None 2024-04-12T03:58:48.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.128 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.132 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b049c82636c20eff-006618b178-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.132 DEBUG:boto:b'dkyges-60obj000000000000000000241000falseobj000000000000000000242024-04-12T03:58:34.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.133 DEBUG:boto:path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.133 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.133 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.133 DEBUG:boto:Path: /dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.133 DEBUG:boto:Data: 2024-04-12T03:58:48.133 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.133 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.133 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.133 DEBUG:boto:Params: {} 2024-04-12T03:58:48.134 DEBUG:boto:Token: None 2024-04-12T03:58:48.134 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.134 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lINrUyK1MatWrqP31ToF45Zb2hk= 2024-04-12T03:58:48.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lINrUyK1MatWrqP31ToF45Zb2hk=', 'Content-Length': '0'} 2024-04-12T03:58:48.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000869a9ea207e11670-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.136 DEBUG:boto:path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.136 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.136 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.136 DEBUG:boto:Path: /dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.136 DEBUG:boto:Data: 2024-04-12T03:58:48.136 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.137 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.137 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.137 DEBUG:boto:Params: {} 2024-04-12T03:58:48.137 DEBUG:boto:Token: None 2024-04-12T03:58:48.137 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.137 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lINrUyK1MatWrqP31ToF45Zb2hk= 2024-04-12T03:58:48.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lINrUyK1MatWrqP31ToF45Zb2hk=', 'Content-Length': '0'} 2024-04-12T03:58:48.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006178a6a56a2846ea-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2024-04-12T03:58:48.139 DEBUG:boto:path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.140 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.140 DEBUG:boto:Method: GET 2024-04-12T03:58:48.140 DEBUG:boto:Path: /dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.140 DEBUG:boto:Data: 2024-04-12T03:58:48.140 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.140 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.140 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.140 DEBUG:boto:Params: {} 2024-04-12T03:58:48.140 DEBUG:boto:Token: None 2024-04-12T03:58:48.140 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.140 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OG4CecXjEdASKERD4vZ3pOGIljo= 2024-04-12T03:58:48.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OG4CecXjEdASKERD4vZ3pOGIljo=', 'Content-Length': '0'} 2024-04-12T03:58:48.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e40514b383139269-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.142 DEBUG:boto:path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.143 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.143 DEBUG:boto:Method: GET 2024-04-12T03:58:48.143 DEBUG:boto:Path: /dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.143 DEBUG:boto:Data: 2024-04-12T03:58:48.143 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.143 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.143 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.143 DEBUG:boto:Params: {} 2024-04-12T03:58:48.143 DEBUG:boto:Token: None 2024-04-12T03:58:48.143 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000024 2024-04-12T03:58:48.144 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OG4CecXjEdASKERD4vZ3pOGIljo= 2024-04-12T03:58:48.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OG4CecXjEdASKERD4vZ3pOGIljo=', 'Content-Length': '0'} 2024-04-12T03:58:48.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a12e22bf33207aba-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.186 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.186 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.187 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000024 2024-04-12T03:58:48.187 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000024 2024-04-12T03:58:48.187 DEBUG:boto:Method: GET 2024-04-12T03:58:48.187 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000024 2024-04-12T03:58:48.187 DEBUG:boto:Data: 2024-04-12T03:58:48.187 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.187 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.187 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.187 DEBUG:boto:Params: {} 2024-04-12T03:58:48.187 DEBUG:boto:Token: None 2024-04-12T03:58:48.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.188 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a428b9ee4af7804-006618b178-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.190 DEBUG:boto:b'dkyges-60obj000000000000000000241000falseobj000000000000000000242024-04-12T03:58:34.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.191 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.191 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.191 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000024 2024-04-12T03:58:48.191 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000024 2024-04-12T03:58:48.191 DEBUG:boto:Method: GET 2024-04-12T03:58:48.191 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000024 2024-04-12T03:58:48.191 DEBUG:boto:Data: 2024-04-12T03:58:48.191 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.192 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.192 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.192 DEBUG:boto:Params: {} 2024-04-12T03:58:48.192 DEBUG:boto:Token: None 2024-04-12T03:58:48.192 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.192 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b034f6fc89b55c6b-006618b178-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.197 DEBUG:boto:b'dkyges-60obj000000000000000000241000falseobj000000000000000000242024-04-12T03:58:34.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-04-12T03:58:48.197 DEBUG:boto:path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.197 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.197 DEBUG:boto:path=/dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.197 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.197 DEBUG:boto:Method: GET 2024-04-12T03:58:48.197 DEBUG:boto:Path: /dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.198 DEBUG:boto:Data: 2024-04-12T03:58:48.198 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.198 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.198 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.198 DEBUG:boto:Params: {} 2024-04-12T03:58:48.198 DEBUG:boto:Token: None 2024-04-12T03:58:48.198 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.198 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ssj0QLC70GUqPS07doR+pbPgHiM= 2024-04-12T03:58:48.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ssj0QLC70GUqPS07doR+pbPgHiM=', 'Content-Length': '0'} 2024-04-12T03:58:48.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000878da74eab91dec6-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.242 DEBUG:boto:path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.243 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.243 DEBUG:boto:path=/dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.243 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.243 DEBUG:boto:Method: GET 2024-04-12T03:58:48.243 DEBUG:boto:Path: /dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.243 DEBUG:boto:Data: 2024-04-12T03:58:48.243 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.243 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.243 DEBUG:boto:Params: {} 2024-04-12T03:58:48.243 DEBUG:boto:Token: None 2024-04-12T03:58:48.244 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.244 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ssj0QLC70GUqPS07doR+pbPgHiM= 2024-04-12T03:58:48.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ssj0QLC70GUqPS07doR+pbPgHiM=', 'Content-Length': '0'} 2024-04-12T03:58:48.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000097141adaf590ffc-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.246 DEBUG:boto:path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.246 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.246 DEBUG:boto:path=/dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.246 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.246 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.246 DEBUG:boto:Path: /dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.246 DEBUG:boto:Data: 2024-04-12T03:58:48.246 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.246 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.247 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.247 DEBUG:boto:Params: {} 2024-04-12T03:58:48.247 DEBUG:boto:Token: None 2024-04-12T03:58:48.247 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.247 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:riES8fLtyTjd44OJ6oJXhoy5Ei8= 2024-04-12T03:58:48.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:riES8fLtyTjd44OJ6oJXhoy5Ei8=', 'Content-Length': '0'} 2024-04-12T03:58:48.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db825ae3a834638e-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.249 DEBUG:boto:path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.249 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.249 DEBUG:boto:path=/dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.250 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.250 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.250 DEBUG:boto:Path: /dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.250 DEBUG:boto:Data: 2024-04-12T03:58:48.250 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.250 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.250 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.250 DEBUG:boto:Params: {} 2024-04-12T03:58:48.250 DEBUG:boto:Token: None 2024-04-12T03:58:48.250 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.251 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:riES8fLtyTjd44OJ6oJXhoy5Ei8= 2024-04-12T03:58:48.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:riES8fLtyTjd44OJ6oJXhoy5Ei8=', 'Content-Length': '0'} 2024-04-12T03:58:48.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e6db16fcd81ead3-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-04-12T03:58:48.253 DEBUG:boto:path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.253 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.253 DEBUG:boto:path=/dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.253 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.253 DEBUG:boto:Method: GET 2024-04-12T03:58:48.253 DEBUG:boto:Path: /dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.253 DEBUG:boto:Data: 2024-04-12T03:58:48.253 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.253 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.254 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.254 DEBUG:boto:Params: {} 2024-04-12T03:58:48.254 DEBUG:boto:Token: None 2024-04-12T03:58:48.254 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.254 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ssj0QLC70GUqPS07doR+pbPgHiM= 2024-04-12T03:58:48.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ssj0QLC70GUqPS07doR+pbPgHiM=', 'Content-Length': '0'} 2024-04-12T03:58:48.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000693a52664307b7a9-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.256 DEBUG:boto:path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.256 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.257 DEBUG:boto:path=/dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.257 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.257 DEBUG:boto:Method: GET 2024-04-12T03:58:48.257 DEBUG:boto:Path: /dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.257 DEBUG:boto:Data: 2024-04-12T03:58:48.257 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.257 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.257 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.257 DEBUG:boto:Params: {} 2024-04-12T03:58:48.257 DEBUG:boto:Token: None 2024-04-12T03:58:48.258 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000025?versionId=null 2024-04-12T03:58:48.258 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ssj0QLC70GUqPS07doR+pbPgHiM= 2024-04-12T03:58:48.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ssj0QLC70GUqPS07doR+pbPgHiM=', 'Content-Length': '0'} 2024-04-12T03:58:48.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074a27765da11755d-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.260 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.260 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.260 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000025 2024-04-12T03:58:48.260 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000025 2024-04-12T03:58:48.260 DEBUG:boto:Method: GET 2024-04-12T03:58:48.260 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000025 2024-04-12T03:58:48.260 DEBUG:boto:Data: 2024-04-12T03:58:48.260 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.260 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.261 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.261 DEBUG:boto:Params: {} 2024-04-12T03:58:48.261 DEBUG:boto:Token: None 2024-04-12T03:58:48.261 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.261 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbc01741d170f0f9-006618b178-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.263 DEBUG:boto:b'dkyges-60obj000000000000000000251000falseobj000000000000000000252024-04-12T03:58:33.202Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.264 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.264 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.264 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000025 2024-04-12T03:58:48.264 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000025 2024-04-12T03:58:48.264 DEBUG:boto:Method: GET 2024-04-12T03:58:48.264 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000025 2024-04-12T03:58:48.265 DEBUG:boto:Data: 2024-04-12T03:58:48.265 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.265 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.265 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.265 DEBUG:boto:Params: {} 2024-04-12T03:58:48.265 DEBUG:boto:Token: None 2024-04-12T03:58:48.265 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.265 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c169d05b0e38222f-006618b178-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.270 DEBUG:boto:b'dkyges-60obj000000000000000000251000falseobj000000000000000000252024-04-12T03:58:33.202Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.270 DEBUG:boto:path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.270 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.270 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.270 DEBUG:boto:Path: /dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.270 DEBUG:boto:Data: 2024-04-12T03:58:48.270 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.271 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.271 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.271 DEBUG:boto:Params: {} 2024-04-12T03:58:48.271 DEBUG:boto:Token: None 2024-04-12T03:58:48.271 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.271 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:O35gtCZG/l68Dg5cCtyzmirETek= 2024-04-12T03:58:48.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:O35gtCZG/l68Dg5cCtyzmirETek=', 'Content-Length': '0'} 2024-04-12T03:58:48.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9f117923ec3e5d4-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.273 DEBUG:boto:path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.274 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.274 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.274 DEBUG:boto:Path: /dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.274 DEBUG:boto:Data: 2024-04-12T03:58:48.274 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.274 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.274 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.274 DEBUG:boto:Params: {} 2024-04-12T03:58:48.274 DEBUG:boto:Token: None 2024-04-12T03:58:48.275 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.275 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:O35gtCZG/l68Dg5cCtyzmirETek= 2024-04-12T03:58:48.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:O35gtCZG/l68Dg5cCtyzmirETek=', 'Content-Length': '0'} 2024-04-12T03:58:48.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041802f3b073a21d6-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2024-04-12T03:58:48.277 DEBUG:boto:path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.277 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.277 DEBUG:boto:Method: GET 2024-04-12T03:58:48.277 DEBUG:boto:Path: /dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.277 DEBUG:boto:Data: 2024-04-12T03:58:48.277 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.278 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.278 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.278 DEBUG:boto:Params: {} 2024-04-12T03:58:48.278 DEBUG:boto:Token: None 2024-04-12T03:58:48.278 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.278 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n9URJS/XSuP9bk8ebS8HIc0/Zn4= 2024-04-12T03:58:48.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n9URJS/XSuP9bk8ebS8HIc0/Zn4=', 'Content-Length': '0'} 2024-04-12T03:58:48.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8523e769dc02021-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.280 DEBUG:boto:path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.280 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.281 DEBUG:boto:Method: GET 2024-04-12T03:58:48.281 DEBUG:boto:Path: /dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.281 DEBUG:boto:Data: 2024-04-12T03:58:48.281 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.281 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.281 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.281 DEBUG:boto:Params: {} 2024-04-12T03:58:48.281 DEBUG:boto:Token: None 2024-04-12T03:58:48.281 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000025 2024-04-12T03:58:48.281 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n9URJS/XSuP9bk8ebS8HIc0/Zn4= 2024-04-12T03:58:48.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n9URJS/XSuP9bk8ebS8HIc0/Zn4=', 'Content-Length': '0'} 2024-04-12T03:58:48.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5a7045f7986e99f-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.283 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.284 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.284 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000025 2024-04-12T03:58:48.284 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000025 2024-04-12T03:58:48.284 DEBUG:boto:Method: GET 2024-04-12T03:58:48.284 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000025 2024-04-12T03:58:48.284 DEBUG:boto:Data: 2024-04-12T03:58:48.284 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.284 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.284 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.284 DEBUG:boto:Params: {} 2024-04-12T03:58:48.285 DEBUG:boto:Token: None 2024-04-12T03:58:48.285 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.285 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c92fe332a91de2f3-006618b178-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.288 DEBUG:boto:b'dkyges-60obj000000000000000000251000falseobj000000000000000000252024-04-12T03:58:33.202Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.288 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.288 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.288 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000025 2024-04-12T03:58:48.288 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000025 2024-04-12T03:58:48.288 DEBUG:boto:Method: GET 2024-04-12T03:58:48.288 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000025 2024-04-12T03:58:48.289 DEBUG:boto:Data: 2024-04-12T03:58:48.289 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.289 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.289 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.289 DEBUG:boto:Params: {} 2024-04-12T03:58:48.289 DEBUG:boto:Token: None 2024-04-12T03:58:48.289 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.289 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5bb5f8caa12317b-006618b178-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.338 DEBUG:boto:b'dkyges-60obj000000000000000000251000falseobj000000000000000000252024-04-12T03:58:33.202Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-04-12T03:58:48.339 DEBUG:boto:path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.339 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.339 DEBUG:boto:path=/dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.339 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.339 DEBUG:boto:Method: GET 2024-04-12T03:58:48.339 DEBUG:boto:Path: /dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.339 DEBUG:boto:Data: 2024-04-12T03:58:48.339 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.339 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.340 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.340 DEBUG:boto:Params: {} 2024-04-12T03:58:48.340 DEBUG:boto:Token: None 2024-04-12T03:58:48.340 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.340 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mOPZ0kRBSiRWLSMuAK9gp7jkMGc= 2024-04-12T03:58:48.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mOPZ0kRBSiRWLSMuAK9gp7jkMGc=', 'Content-Length': '0'} 2024-04-12T03:58:48.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f3865214249b32-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.343 DEBUG:boto:path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.343 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.343 DEBUG:boto:path=/dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.343 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.343 DEBUG:boto:Method: GET 2024-04-12T03:58:48.343 DEBUG:boto:Path: /dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.343 DEBUG:boto:Data: 2024-04-12T03:58:48.343 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.343 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.343 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.343 DEBUG:boto:Params: {} 2024-04-12T03:58:48.344 DEBUG:boto:Token: None 2024-04-12T03:58:48.344 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.344 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mOPZ0kRBSiRWLSMuAK9gp7jkMGc= 2024-04-12T03:58:48.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mOPZ0kRBSiRWLSMuAK9gp7jkMGc=', 'Content-Length': '0'} 2024-04-12T03:58:48.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4bde2164998591f-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.346 DEBUG:boto:path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.346 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.346 DEBUG:boto:path=/dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.347 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.347 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.347 DEBUG:boto:Path: /dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.347 DEBUG:boto:Data: 2024-04-12T03:58:48.347 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.347 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.347 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.347 DEBUG:boto:Params: {} 2024-04-12T03:58:48.347 DEBUG:boto:Token: None 2024-04-12T03:58:48.347 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.347 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:26bC19hODVSH/NG2BCkFpbwjKW4= 2024-04-12T03:58:48.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:26bC19hODVSH/NG2BCkFpbwjKW4=', 'Content-Length': '0'} 2024-04-12T03:58:48.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007248f7ccdfe745ba-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.349 DEBUG:boto:path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.349 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.350 DEBUG:boto:path=/dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.350 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.350 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.350 DEBUG:boto:Path: /dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.350 DEBUG:boto:Data: 2024-04-12T03:58:48.350 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.350 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.350 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.350 DEBUG:boto:Params: {} 2024-04-12T03:58:48.350 DEBUG:boto:Token: None 2024-04-12T03:58:48.350 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.351 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:26bC19hODVSH/NG2BCkFpbwjKW4= 2024-04-12T03:58:48.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:26bC19hODVSH/NG2BCkFpbwjKW4=', 'Content-Length': '0'} 2024-04-12T03:58:48.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b401777cc89446f6-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-04-12T03:58:48.353 DEBUG:boto:path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.353 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.353 DEBUG:boto:path=/dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.353 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.353 DEBUG:boto:Method: GET 2024-04-12T03:58:48.353 DEBUG:boto:Path: /dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.354 DEBUG:boto:Data: 2024-04-12T03:58:48.354 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.354 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.354 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.354 DEBUG:boto:Params: {} 2024-04-12T03:58:48.354 DEBUG:boto:Token: None 2024-04-12T03:58:48.354 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.354 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mOPZ0kRBSiRWLSMuAK9gp7jkMGc= 2024-04-12T03:58:48.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mOPZ0kRBSiRWLSMuAK9gp7jkMGc=', 'Content-Length': '0'} 2024-04-12T03:58:48.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc432a4fca150004-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.398 DEBUG:boto:path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.398 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.399 DEBUG:boto:path=/dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.399 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.399 DEBUG:boto:Method: GET 2024-04-12T03:58:48.399 DEBUG:boto:Path: /dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.399 DEBUG:boto:Data: 2024-04-12T03:58:48.399 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.399 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.399 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.399 DEBUG:boto:Params: {} 2024-04-12T03:58:48.399 DEBUG:boto:Token: None 2024-04-12T03:58:48.400 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000026?versionId=null 2024-04-12T03:58:48.400 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mOPZ0kRBSiRWLSMuAK9gp7jkMGc= 2024-04-12T03:58:48.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mOPZ0kRBSiRWLSMuAK9gp7jkMGc=', 'Content-Length': '0'} 2024-04-12T03:58:48.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e99e59b28fb006dd-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.402 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.402 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.402 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000026 2024-04-12T03:58:48.402 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000026 2024-04-12T03:58:48.402 DEBUG:boto:Method: GET 2024-04-12T03:58:48.402 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000026 2024-04-12T03:58:48.402 DEBUG:boto:Data: 2024-04-12T03:58:48.402 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.403 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.403 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.403 DEBUG:boto:Params: {} 2024-04-12T03:58:48.403 DEBUG:boto:Token: None 2024-04-12T03:58:48.403 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.403 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d5b37b469e14080-006618b178-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.406 DEBUG:boto:b'dkyges-60obj000000000000000000261000falseobj000000000000000000262024-04-12T03:58:35.527Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.406 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.406 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.406 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000026 2024-04-12T03:58:48.407 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000026 2024-04-12T03:58:48.407 DEBUG:boto:Method: GET 2024-04-12T03:58:48.407 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000026 2024-04-12T03:58:48.407 DEBUG:boto:Data: 2024-04-12T03:58:48.407 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.407 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.407 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.407 DEBUG:boto:Params: {} 2024-04-12T03:58:48.407 DEBUG:boto:Token: None 2024-04-12T03:58:48.408 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.408 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e474fa5265ec0e2c-006618b178-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.412 DEBUG:boto:b'dkyges-60obj000000000000000000261000falseobj000000000000000000262024-04-12T03:58:35.527Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.412 DEBUG:boto:path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.412 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.412 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.413 DEBUG:boto:Path: /dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.413 DEBUG:boto:Data: 2024-04-12T03:58:48.413 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.413 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.413 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.413 DEBUG:boto:Params: {} 2024-04-12T03:58:48.413 DEBUG:boto:Token: None 2024-04-12T03:58:48.413 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.413 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cacaTUbxqXvF1u8S+hmuYiafpU8= 2024-04-12T03:58:48.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cacaTUbxqXvF1u8S+hmuYiafpU8=', 'Content-Length': '0'} 2024-04-12T03:58:48.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005921a6dbfc1e319d-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.416 DEBUG:boto:path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.416 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.416 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.416 DEBUG:boto:Path: /dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.416 DEBUG:boto:Data: 2024-04-12T03:58:48.416 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.416 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.416 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.416 DEBUG:boto:Params: {} 2024-04-12T03:58:48.416 DEBUG:boto:Token: None 2024-04-12T03:58:48.417 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.417 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cacaTUbxqXvF1u8S+hmuYiafpU8= 2024-04-12T03:58:48.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cacaTUbxqXvF1u8S+hmuYiafpU8=', 'Content-Length': '0'} 2024-04-12T03:58:48.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f13e63f829349746-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2024-04-12T03:58:48.419 DEBUG:boto:path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.419 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.419 DEBUG:boto:Method: GET 2024-04-12T03:58:48.419 DEBUG:boto:Path: /dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.419 DEBUG:boto:Data: 2024-04-12T03:58:48.419 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.420 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.420 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.420 DEBUG:boto:Params: {} 2024-04-12T03:58:48.420 DEBUG:boto:Token: None 2024-04-12T03:58:48.420 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.420 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+v7d43+Z4RdUDq7peOv+FgBiYNg= 2024-04-12T03:58:48.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+v7d43+Z4RdUDq7peOv+FgBiYNg=', 'Content-Length': '0'} 2024-04-12T03:58:48.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbe4812e5e2b8492-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.422 DEBUG:boto:path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.423 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.423 DEBUG:boto:Method: GET 2024-04-12T03:58:48.423 DEBUG:boto:Path: /dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.423 DEBUG:boto:Data: 2024-04-12T03:58:48.423 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.423 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.423 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.423 DEBUG:boto:Params: {} 2024-04-12T03:58:48.423 DEBUG:boto:Token: None 2024-04-12T03:58:48.424 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000026 2024-04-12T03:58:48.424 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+v7d43+Z4RdUDq7peOv+FgBiYNg= 2024-04-12T03:58:48.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+v7d43+Z4RdUDq7peOv+FgBiYNg=', 'Content-Length': '0'} 2024-04-12T03:58:48.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d617fc9629579b8-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.466 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.467 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.467 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000026 2024-04-12T03:58:48.467 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000026 2024-04-12T03:58:48.467 DEBUG:boto:Method: GET 2024-04-12T03:58:48.467 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000026 2024-04-12T03:58:48.467 DEBUG:boto:Data: 2024-04-12T03:58:48.467 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.467 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.467 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.467 DEBUG:boto:Params: {} 2024-04-12T03:58:48.468 DEBUG:boto:Token: None 2024-04-12T03:58:48.468 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.468 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089ad4d217a92c8f2-006618b178-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.471 DEBUG:boto:b'dkyges-60obj000000000000000000261000falseobj000000000000000000262024-04-12T03:58:35.527Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.471 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.472 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.472 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000026 2024-04-12T03:58:48.472 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000026 2024-04-12T03:58:48.472 DEBUG:boto:Method: GET 2024-04-12T03:58:48.472 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000026 2024-04-12T03:58:48.472 DEBUG:boto:Data: 2024-04-12T03:58:48.472 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.472 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.472 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.472 DEBUG:boto:Params: {} 2024-04-12T03:58:48.472 DEBUG:boto:Token: None 2024-04-12T03:58:48.473 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.473 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e0a2a3d428fc9de-006618b178-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.477 DEBUG:boto:b'dkyges-60obj000000000000000000261000falseobj000000000000000000262024-04-12T03:58:35.527Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-04-12T03:58:48.478 DEBUG:boto:path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.478 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.478 DEBUG:boto:path=/dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.478 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.478 DEBUG:boto:Method: GET 2024-04-12T03:58:48.478 DEBUG:boto:Path: /dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.478 DEBUG:boto:Data: 2024-04-12T03:58:48.478 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.478 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.478 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.478 DEBUG:boto:Params: {} 2024-04-12T03:58:48.479 DEBUG:boto:Token: None 2024-04-12T03:58:48.479 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.479 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/zKmyxB+U9WQihaTrrQoRC1DqIU= 2024-04-12T03:58:48.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/zKmyxB+U9WQihaTrrQoRC1DqIU=', 'Content-Length': '0'} 2024-04-12T03:58:48.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e95b6c7d3cf514a1-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.522 DEBUG:boto:path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.522 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.522 DEBUG:boto:path=/dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.523 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.523 DEBUG:boto:Method: GET 2024-04-12T03:58:48.523 DEBUG:boto:Path: /dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.523 DEBUG:boto:Data: 2024-04-12T03:58:48.523 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.523 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.523 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.523 DEBUG:boto:Params: {} 2024-04-12T03:58:48.523 DEBUG:boto:Token: None 2024-04-12T03:58:48.523 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.524 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/zKmyxB+U9WQihaTrrQoRC1DqIU= 2024-04-12T03:58:48.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/zKmyxB+U9WQihaTrrQoRC1DqIU=', 'Content-Length': '0'} 2024-04-12T03:58:48.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071c6e30429aa681a-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.526 DEBUG:boto:path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.526 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.527 DEBUG:boto:path=/dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.527 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.527 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.527 DEBUG:boto:Path: /dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.527 DEBUG:boto:Data: 2024-04-12T03:58:48.527 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.527 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.527 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.527 DEBUG:boto:Params: {} 2024-04-12T03:58:48.527 DEBUG:boto:Token: None 2024-04-12T03:58:48.527 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.528 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wZgnQcJcwWlbCteMFgXe2Yhf8FA= 2024-04-12T03:58:48.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wZgnQcJcwWlbCteMFgXe2Yhf8FA=', 'Content-Length': '0'} 2024-04-12T03:58:48.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000330beeb8dc9c6422-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.530 DEBUG:boto:path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.530 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.530 DEBUG:boto:path=/dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.530 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.530 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.530 DEBUG:boto:Path: /dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.530 DEBUG:boto:Data: 2024-04-12T03:58:48.530 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.530 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.530 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.530 DEBUG:boto:Params: {} 2024-04-12T03:58:48.531 DEBUG:boto:Token: None 2024-04-12T03:58:48.531 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.531 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wZgnQcJcwWlbCteMFgXe2Yhf8FA= 2024-04-12T03:58:48.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wZgnQcJcwWlbCteMFgXe2Yhf8FA=', 'Content-Length': '0'} 2024-04-12T03:58:48.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7e6261c6e0a638f-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-04-12T03:58:48.533 DEBUG:boto:path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.534 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.534 DEBUG:boto:path=/dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.534 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.534 DEBUG:boto:Method: GET 2024-04-12T03:58:48.534 DEBUG:boto:Path: /dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.534 DEBUG:boto:Data: 2024-04-12T03:58:48.534 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.534 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.534 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.534 DEBUG:boto:Params: {} 2024-04-12T03:58:48.535 DEBUG:boto:Token: None 2024-04-12T03:58:48.535 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.535 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/zKmyxB+U9WQihaTrrQoRC1DqIU= 2024-04-12T03:58:48.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/zKmyxB+U9WQihaTrrQoRC1DqIU=', 'Content-Length': '0'} 2024-04-12T03:58:48.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070b48158725f68ee-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.537 DEBUG:boto:path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.537 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.537 DEBUG:boto:path=/dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.537 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.537 DEBUG:boto:Method: GET 2024-04-12T03:58:48.537 DEBUG:boto:Path: /dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.537 DEBUG:boto:Data: 2024-04-12T03:58:48.538 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.538 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.538 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.538 DEBUG:boto:Params: {} 2024-04-12T03:58:48.538 DEBUG:boto:Token: None 2024-04-12T03:58:48.538 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000027?versionId=null 2024-04-12T03:58:48.538 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/zKmyxB+U9WQihaTrrQoRC1DqIU= 2024-04-12T03:58:48.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/zKmyxB+U9WQihaTrrQoRC1DqIU=', 'Content-Length': '0'} 2024-04-12T03:58:48.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001776337944826756-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.540 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.540 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.541 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000027 2024-04-12T03:58:48.541 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000027 2024-04-12T03:58:48.541 DEBUG:boto:Method: GET 2024-04-12T03:58:48.541 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000027 2024-04-12T03:58:48.541 DEBUG:boto:Data: 2024-04-12T03:58:48.541 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.541 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.541 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.541 DEBUG:boto:Params: {} 2024-04-12T03:58:48.541 DEBUG:boto:Token: None 2024-04-12T03:58:48.542 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.542 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.544 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2b86c51a8c05c47-006618b178-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.545 DEBUG:boto:b'dkyges-60obj000000000000000000271000falseobj000000000000000000272024-04-12T03:58:34.847Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.545 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.545 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.545 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000027 2024-04-12T03:58:48.545 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000027 2024-04-12T03:58:48.545 DEBUG:boto:Method: GET 2024-04-12T03:58:48.546 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000027 2024-04-12T03:58:48.546 DEBUG:boto:Data: 2024-04-12T03:58:48.546 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.546 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.546 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.546 DEBUG:boto:Params: {} 2024-04-12T03:58:48.546 DEBUG:boto:Token: None 2024-04-12T03:58:48.546 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.546 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000887a82762431c51-006618b178-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.551 DEBUG:boto:b'dkyges-60obj000000000000000000271000falseobj000000000000000000272024-04-12T03:58:34.847Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.551 DEBUG:boto:path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.552 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.552 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.552 DEBUG:boto:Path: /dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.552 DEBUG:boto:Data: 2024-04-12T03:58:48.552 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.552 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.552 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.552 DEBUG:boto:Params: {} 2024-04-12T03:58:48.552 DEBUG:boto:Token: None 2024-04-12T03:58:48.552 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.553 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:78p1iXJCM0cKI5uc6ZvRSaUwWjM= 2024-04-12T03:58:48.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:78p1iXJCM0cKI5uc6ZvRSaUwWjM=', 'Content-Length': '0'} 2024-04-12T03:58:48.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067e4d44ca0ca366c-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.555 DEBUG:boto:path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.555 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.555 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.555 DEBUG:boto:Path: /dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.555 DEBUG:boto:Data: 2024-04-12T03:58:48.555 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.555 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.555 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.555 DEBUG:boto:Params: {} 2024-04-12T03:58:48.555 DEBUG:boto:Token: None 2024-04-12T03:58:48.555 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.556 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:78p1iXJCM0cKI5uc6ZvRSaUwWjM= 2024-04-12T03:58:48.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:78p1iXJCM0cKI5uc6ZvRSaUwWjM=', 'Content-Length': '0'} 2024-04-12T03:58:48.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec2d5c097b393ba3-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2024-04-12T03:58:48.558 DEBUG:boto:path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.558 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.558 DEBUG:boto:Method: GET 2024-04-12T03:58:48.558 DEBUG:boto:Path: /dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.558 DEBUG:boto:Data: 2024-04-12T03:58:48.558 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.558 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.558 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.559 DEBUG:boto:Params: {} 2024-04-12T03:58:48.559 DEBUG:boto:Token: None 2024-04-12T03:58:48.559 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.559 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WNizhisV0DUSrjxpb3bcDs7AhO4= 2024-04-12T03:58:48.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WNizhisV0DUSrjxpb3bcDs7AhO4=', 'Content-Length': '0'} 2024-04-12T03:58:48.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc2a5c68d18be3ac-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.561 DEBUG:boto:path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.561 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.561 DEBUG:boto:Method: GET 2024-04-12T03:58:48.561 DEBUG:boto:Path: /dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.561 DEBUG:boto:Data: 2024-04-12T03:58:48.561 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.562 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.562 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.562 DEBUG:boto:Params: {} 2024-04-12T03:58:48.562 DEBUG:boto:Token: None 2024-04-12T03:58:48.562 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000027 2024-04-12T03:58:48.562 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WNizhisV0DUSrjxpb3bcDs7AhO4= 2024-04-12T03:58:48.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WNizhisV0DUSrjxpb3bcDs7AhO4=', 'Content-Length': '0'} 2024-04-12T03:58:48.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f08234da26b29dd9-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.610 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.610 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.611 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000027 2024-04-12T03:58:48.611 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000027 2024-04-12T03:58:48.611 DEBUG:boto:Method: GET 2024-04-12T03:58:48.611 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000027 2024-04-12T03:58:48.611 DEBUG:boto:Data: 2024-04-12T03:58:48.611 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.611 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.611 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.611 DEBUG:boto:Params: {} 2024-04-12T03:58:48.612 DEBUG:boto:Token: None 2024-04-12T03:58:48.612 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.612 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9a65060e5383e5e-006618b178-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.615 DEBUG:boto:b'dkyges-60obj000000000000000000271000falseobj000000000000000000272024-04-12T03:58:34.847Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.615 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.615 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.615 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000027 2024-04-12T03:58:48.615 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000027 2024-04-12T03:58:48.615 DEBUG:boto:Method: GET 2024-04-12T03:58:48.615 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000027 2024-04-12T03:58:48.616 DEBUG:boto:Data: 2024-04-12T03:58:48.616 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.616 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.616 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.616 DEBUG:boto:Params: {} 2024-04-12T03:58:48.616 DEBUG:boto:Token: None 2024-04-12T03:58:48.616 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.616 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000534aadae4a77dbf9-006618b178-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.621 DEBUG:boto:b'dkyges-60obj000000000000000000271000falseobj000000000000000000272024-04-12T03:58:34.847Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-04-12T03:58:48.621 DEBUG:boto:path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.621 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.622 DEBUG:boto:path=/dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.622 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.622 DEBUG:boto:Method: GET 2024-04-12T03:58:48.622 DEBUG:boto:Path: /dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.622 DEBUG:boto:Data: 2024-04-12T03:58:48.622 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.622 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.622 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.622 DEBUG:boto:Params: {} 2024-04-12T03:58:48.622 DEBUG:boto:Token: None 2024-04-12T03:58:48.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.623 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3p2UP2aNtN3pRErSMZLkN22UJ8k= 2024-04-12T03:58:48.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3p2UP2aNtN3pRErSMZLkN22UJ8k=', 'Content-Length': '0'} 2024-04-12T03:58:48.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000847a4cf0e0bba272-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.670 DEBUG:boto:path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.670 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.670 DEBUG:boto:path=/dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.671 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.671 DEBUG:boto:Method: GET 2024-04-12T03:58:48.671 DEBUG:boto:Path: /dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.671 DEBUG:boto:Data: 2024-04-12T03:58:48.671 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.671 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.671 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.671 DEBUG:boto:Params: {} 2024-04-12T03:58:48.671 DEBUG:boto:Token: None 2024-04-12T03:58:48.671 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.671 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3p2UP2aNtN3pRErSMZLkN22UJ8k= 2024-04-12T03:58:48.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3p2UP2aNtN3pRErSMZLkN22UJ8k=', 'Content-Length': '0'} 2024-04-12T03:58:48.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dbcb3430849774b-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.674 DEBUG:boto:path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.674 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.674 DEBUG:boto:path=/dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.674 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.674 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.674 DEBUG:boto:Path: /dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.674 DEBUG:boto:Data: 2024-04-12T03:58:48.674 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.674 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.675 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.675 DEBUG:boto:Params: {} 2024-04-12T03:58:48.675 DEBUG:boto:Token: None 2024-04-12T03:58:48.675 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.675 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rHuGyi0PAD7ZAZ7JomAIwOp/Pl8= 2024-04-12T03:58:48.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rHuGyi0PAD7ZAZ7JomAIwOp/Pl8=', 'Content-Length': '0'} 2024-04-12T03:58:48.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e97f9ad24b1ff858-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.677 DEBUG:boto:path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.678 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.678 DEBUG:boto:path=/dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.678 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.678 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.678 DEBUG:boto:Path: /dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.678 DEBUG:boto:Data: 2024-04-12T03:58:48.678 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.678 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.678 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.678 DEBUG:boto:Params: {} 2024-04-12T03:58:48.679 DEBUG:boto:Token: None 2024-04-12T03:58:48.679 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.679 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rHuGyi0PAD7ZAZ7JomAIwOp/Pl8= 2024-04-12T03:58:48.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rHuGyi0PAD7ZAZ7JomAIwOp/Pl8=', 'Content-Length': '0'} 2024-04-12T03:58:48.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2648555214c6174-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-04-12T03:58:48.681 DEBUG:boto:path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.681 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.681 DEBUG:boto:path=/dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.681 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.681 DEBUG:boto:Method: GET 2024-04-12T03:58:48.681 DEBUG:boto:Path: /dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.681 DEBUG:boto:Data: 2024-04-12T03:58:48.681 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.681 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.681 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.682 DEBUG:boto:Params: {} 2024-04-12T03:58:48.682 DEBUG:boto:Token: None 2024-04-12T03:58:48.682 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.682 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3p2UP2aNtN3pRErSMZLkN22UJ8k= 2024-04-12T03:58:48.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3p2UP2aNtN3pRErSMZLkN22UJ8k=', 'Content-Length': '0'} 2024-04-12T03:58:48.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f8037b1178ffdb6-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.684 DEBUG:boto:path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.684 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.684 DEBUG:boto:path=/dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.685 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.685 DEBUG:boto:Method: GET 2024-04-12T03:58:48.685 DEBUG:boto:Path: /dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.685 DEBUG:boto:Data: 2024-04-12T03:58:48.685 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.685 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.685 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.685 DEBUG:boto:Params: {} 2024-04-12T03:58:48.685 DEBUG:boto:Token: None 2024-04-12T03:58:48.685 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000028?versionId=null 2024-04-12T03:58:48.685 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3p2UP2aNtN3pRErSMZLkN22UJ8k= 2024-04-12T03:58:48.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3p2UP2aNtN3pRErSMZLkN22UJ8k=', 'Content-Length': '0'} 2024-04-12T03:58:48.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000092a4c1fcb2e6aa7-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.688 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.688 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.688 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000028 2024-04-12T03:58:48.688 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000028 2024-04-12T03:58:48.688 DEBUG:boto:Method: GET 2024-04-12T03:58:48.688 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000028 2024-04-12T03:58:48.688 DEBUG:boto:Data: 2024-04-12T03:58:48.688 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.688 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.688 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.688 DEBUG:boto:Params: {} 2024-04-12T03:58:48.689 DEBUG:boto:Token: None 2024-04-12T03:58:48.689 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.689 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000319341122eddbb59-006618b178-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.691 DEBUG:boto:b'dkyges-60obj000000000000000000281000falseobj000000000000000000282024-04-12T03:58:30.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.692 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.692 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.692 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000028 2024-04-12T03:58:48.692 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000028 2024-04-12T03:58:48.692 DEBUG:boto:Method: GET 2024-04-12T03:58:48.692 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000028 2024-04-12T03:58:48.692 DEBUG:boto:Data: 2024-04-12T03:58:48.692 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.692 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.692 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.693 DEBUG:boto:Params: {} 2024-04-12T03:58:48.693 DEBUG:boto:Token: None 2024-04-12T03:58:48.693 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.693 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058a2f33d819429ca-006618b178-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.698 DEBUG:boto:b'dkyges-60obj000000000000000000281000falseobj000000000000000000282024-04-12T03:58:30.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.698 DEBUG:boto:path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.698 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.698 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.699 DEBUG:boto:Path: /dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.699 DEBUG:boto:Data: 2024-04-12T03:58:48.699 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.699 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.699 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.699 DEBUG:boto:Params: {} 2024-04-12T03:58:48.699 DEBUG:boto:Token: None 2024-04-12T03:58:48.699 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.699 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BwmZMB8oEerE4eTdHhBj85J9StU= 2024-04-12T03:58:48.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BwmZMB8oEerE4eTdHhBj85J9StU=', 'Content-Length': '0'} 2024-04-12T03:58:48.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c698a6c4857a1e7f-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.702 DEBUG:boto:path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.702 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.702 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.702 DEBUG:boto:Path: /dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.702 DEBUG:boto:Data: 2024-04-12T03:58:48.702 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.702 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.702 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.702 DEBUG:boto:Params: {} 2024-04-12T03:58:48.702 DEBUG:boto:Token: None 2024-04-12T03:58:48.703 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.703 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BwmZMB8oEerE4eTdHhBj85J9StU= 2024-04-12T03:58:48.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BwmZMB8oEerE4eTdHhBj85J9StU=', 'Content-Length': '0'} 2024-04-12T03:58:48.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c1b2e1a4874aab4-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2024-04-12T03:58:48.705 DEBUG:boto:path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.705 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.705 DEBUG:boto:Method: GET 2024-04-12T03:58:48.705 DEBUG:boto:Path: /dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.706 DEBUG:boto:Data: 2024-04-12T03:58:48.706 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.706 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.706 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.706 DEBUG:boto:Params: {} 2024-04-12T03:58:48.706 DEBUG:boto:Token: None 2024-04-12T03:58:48.706 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.707 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pWeleKtDu3EZbygp4MJX68Ew4jw= 2024-04-12T03:58:48.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pWeleKtDu3EZbygp4MJX68Ew4jw=', 'Content-Length': '0'} 2024-04-12T03:58:48.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000196f53cfd21205dd-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.709 DEBUG:boto:path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.709 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.709 DEBUG:boto:Method: GET 2024-04-12T03:58:48.709 DEBUG:boto:Path: /dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.709 DEBUG:boto:Data: 2024-04-12T03:58:48.710 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.710 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.710 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.710 DEBUG:boto:Params: {} 2024-04-12T03:58:48.710 DEBUG:boto:Token: None 2024-04-12T03:58:48.710 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000028 2024-04-12T03:58:48.710 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pWeleKtDu3EZbygp4MJX68Ew4jw= 2024-04-12T03:58:48.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pWeleKtDu3EZbygp4MJX68Ew4jw=', 'Content-Length': '0'} 2024-04-12T03:58:48.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010b95ded30a5c04f-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.754 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.755 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.755 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000028 2024-04-12T03:58:48.755 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000028 2024-04-12T03:58:48.755 DEBUG:boto:Method: GET 2024-04-12T03:58:48.755 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000028 2024-04-12T03:58:48.755 DEBUG:boto:Data: 2024-04-12T03:58:48.755 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.755 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.755 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.755 DEBUG:boto:Params: {} 2024-04-12T03:58:48.756 DEBUG:boto:Token: None 2024-04-12T03:58:48.756 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.756 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a35556fde2319280-006618b178-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.759 DEBUG:boto:b'dkyges-60obj000000000000000000281000falseobj000000000000000000282024-04-12T03:58:30.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.759 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.759 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.759 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000028 2024-04-12T03:58:48.759 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000028 2024-04-12T03:58:48.759 DEBUG:boto:Method: GET 2024-04-12T03:58:48.759 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000028 2024-04-12T03:58:48.759 DEBUG:boto:Data: 2024-04-12T03:58:48.760 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.760 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.760 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.760 DEBUG:boto:Params: {} 2024-04-12T03:58:48.760 DEBUG:boto:Token: None 2024-04-12T03:58:48.760 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.760 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000329e3eed1bf22eb9-006618b178-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.764 DEBUG:boto:b'dkyges-60obj000000000000000000281000falseobj000000000000000000282024-04-12T03:58:30.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-04-12T03:58:48.765 DEBUG:boto:path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.765 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.765 DEBUG:boto:path=/dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.765 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.765 DEBUG:boto:Method: GET 2024-04-12T03:58:48.765 DEBUG:boto:Path: /dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.765 DEBUG:boto:Data: 2024-04-12T03:58:48.766 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.766 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.766 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.766 DEBUG:boto:Params: {} 2024-04-12T03:58:48.766 DEBUG:boto:Token: None 2024-04-12T03:58:48.766 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.766 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Asd5XUyeVislZEnGpht+L0xdV0s= 2024-04-12T03:58:48.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Asd5XUyeVislZEnGpht+L0xdV0s=', 'Content-Length': '0'} 2024-04-12T03:58:48.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087dd1043e53bd68f-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.810 DEBUG:boto:path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.811 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.811 DEBUG:boto:path=/dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.811 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.811 DEBUG:boto:Method: GET 2024-04-12T03:58:48.811 DEBUG:boto:Path: /dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.811 DEBUG:boto:Data: 2024-04-12T03:58:48.811 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.811 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.811 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.811 DEBUG:boto:Params: {} 2024-04-12T03:58:48.811 DEBUG:boto:Token: None 2024-04-12T03:58:48.812 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.812 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Asd5XUyeVislZEnGpht+L0xdV0s= 2024-04-12T03:58:48.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Asd5XUyeVislZEnGpht+L0xdV0s=', 'Content-Length': '0'} 2024-04-12T03:58:48.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000049b8197e84442aa-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.814 DEBUG:boto:path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.814 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.814 DEBUG:boto:path=/dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.815 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.815 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.815 DEBUG:boto:Path: /dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.815 DEBUG:boto:Data: 2024-04-12T03:58:48.815 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.815 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.815 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.815 DEBUG:boto:Params: {} 2024-04-12T03:58:48.815 DEBUG:boto:Token: None 2024-04-12T03:58:48.815 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.815 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:awNgTdJTdYpbliJOF5h5WJ8hBHs= 2024-04-12T03:58:48.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:awNgTdJTdYpbliJOF5h5WJ8hBHs=', 'Content-Length': '0'} 2024-04-12T03:58:48.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf4032b2c283642b-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.818 DEBUG:boto:path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.818 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.818 DEBUG:boto:path=/dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.818 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.818 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.818 DEBUG:boto:Path: /dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.818 DEBUG:boto:Data: 2024-04-12T03:58:48.818 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.818 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.818 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.819 DEBUG:boto:Params: {} 2024-04-12T03:58:48.819 DEBUG:boto:Token: None 2024-04-12T03:58:48.819 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.819 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:awNgTdJTdYpbliJOF5h5WJ8hBHs= 2024-04-12T03:58:48.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:awNgTdJTdYpbliJOF5h5WJ8hBHs=', 'Content-Length': '0'} 2024-04-12T03:58:48.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000591f65f0d43f2cad-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-04-12T03:58:48.821 DEBUG:boto:path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.821 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.821 DEBUG:boto:path=/dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.821 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.821 DEBUG:boto:Method: GET 2024-04-12T03:58:48.822 DEBUG:boto:Path: /dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.822 DEBUG:boto:Data: 2024-04-12T03:58:48.822 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.822 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.822 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.822 DEBUG:boto:Params: {} 2024-04-12T03:58:48.822 DEBUG:boto:Token: None 2024-04-12T03:58:48.822 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.822 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Asd5XUyeVislZEnGpht+L0xdV0s= 2024-04-12T03:58:48.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Asd5XUyeVislZEnGpht+L0xdV0s=', 'Content-Length': '0'} 2024-04-12T03:58:48.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092c9272f5d350456-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.825 DEBUG:boto:path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.825 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.825 DEBUG:boto:path=/dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.825 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.825 DEBUG:boto:Method: GET 2024-04-12T03:58:48.825 DEBUG:boto:Path: /dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.825 DEBUG:boto:Data: 2024-04-12T03:58:48.826 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.826 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.826 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.826 DEBUG:boto:Params: {} 2024-04-12T03:58:48.826 DEBUG:boto:Token: None 2024-04-12T03:58:48.826 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000029?versionId=null 2024-04-12T03:58:48.826 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Asd5XUyeVislZEnGpht+L0xdV0s= 2024-04-12T03:58:48.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Asd5XUyeVislZEnGpht+L0xdV0s=', 'Content-Length': '0'} 2024-04-12T03:58:48.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ce0be255a458d34-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.829 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.829 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.829 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000029 2024-04-12T03:58:48.829 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000029 2024-04-12T03:58:48.829 DEBUG:boto:Method: GET 2024-04-12T03:58:48.829 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000029 2024-04-12T03:58:48.829 DEBUG:boto:Data: 2024-04-12T03:58:48.829 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.829 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.829 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.830 DEBUG:boto:Params: {} 2024-04-12T03:58:48.830 DEBUG:boto:Token: None 2024-04-12T03:58:48.830 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.830 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041dbaa20718ed765-006618b178-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.833 DEBUG:boto:b'dkyges-60obj000000000000000000291000falseobj000000000000000000292024-04-12T03:58:33.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.833 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.834 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.834 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000029 2024-04-12T03:58:48.834 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000029 2024-04-12T03:58:48.834 DEBUG:boto:Method: GET 2024-04-12T03:58:48.834 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000029 2024-04-12T03:58:48.834 DEBUG:boto:Data: 2024-04-12T03:58:48.834 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.834 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.834 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.834 DEBUG:boto:Params: {} 2024-04-12T03:58:48.835 DEBUG:boto:Token: None 2024-04-12T03:58:48.835 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.835 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013e6e170661a1215-006618b178-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.839 DEBUG:boto:b'dkyges-60obj000000000000000000291000falseobj000000000000000000292024-04-12T03:58:33.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.840 DEBUG:boto:path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.840 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.840 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.840 DEBUG:boto:Path: /dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.840 DEBUG:boto:Data: 2024-04-12T03:58:48.840 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.840 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.840 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.840 DEBUG:boto:Params: {} 2024-04-12T03:58:48.841 DEBUG:boto:Token: None 2024-04-12T03:58:48.841 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.841 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aj+6+nTHUr60n7SOFRQytJQO4eg= 2024-04-12T03:58:48.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aj+6+nTHUr60n7SOFRQytJQO4eg=', 'Content-Length': '0'} 2024-04-12T03:58:48.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c4acd6734bd1e27-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.843 DEBUG:boto:path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.844 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.844 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.844 DEBUG:boto:Path: /dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.844 DEBUG:boto:Data: 2024-04-12T03:58:48.844 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.844 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.844 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.844 DEBUG:boto:Params: {} 2024-04-12T03:58:48.844 DEBUG:boto:Token: None 2024-04-12T03:58:48.844 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.845 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aj+6+nTHUr60n7SOFRQytJQO4eg= 2024-04-12T03:58:48.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aj+6+nTHUr60n7SOFRQytJQO4eg=', 'Content-Length': '0'} 2024-04-12T03:58:48.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e911d17e7d5e0b45-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2024-04-12T03:58:48.847 DEBUG:boto:path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.847 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.847 DEBUG:boto:Method: GET 2024-04-12T03:58:48.847 DEBUG:boto:Path: /dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.848 DEBUG:boto:Data: 2024-04-12T03:58:48.848 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.848 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.848 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.848 DEBUG:boto:Params: {} 2024-04-12T03:58:48.848 DEBUG:boto:Token: None 2024-04-12T03:58:48.848 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.849 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LVkxyD4lgV82Et6Du1btvMc84Eg= 2024-04-12T03:58:48.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LVkxyD4lgV82Et6Du1btvMc84Eg=', 'Content-Length': '0'} 2024-04-12T03:58:48.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb9be292e0047b4d-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.851 DEBUG:boto:path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.851 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.851 DEBUG:boto:Method: GET 2024-04-12T03:58:48.851 DEBUG:boto:Path: /dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.852 DEBUG:boto:Data: 2024-04-12T03:58:48.852 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.852 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.852 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.852 DEBUG:boto:Params: {} 2024-04-12T03:58:48.852 DEBUG:boto:Token: None 2024-04-12T03:58:48.853 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000029 2024-04-12T03:58:48.853 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LVkxyD4lgV82Et6Du1btvMc84Eg= 2024-04-12T03:58:48.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LVkxyD4lgV82Et6Du1btvMc84Eg=', 'Content-Length': '0'} 2024-04-12T03:58:48.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cdd712b88ed4e83-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.898 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.898 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.899 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000029 2024-04-12T03:58:48.899 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000029 2024-04-12T03:58:48.899 DEBUG:boto:Method: GET 2024-04-12T03:58:48.899 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000029 2024-04-12T03:58:48.899 DEBUG:boto:Data: 2024-04-12T03:58:48.899 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.899 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.899 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.899 DEBUG:boto:Params: {} 2024-04-12T03:58:48.899 DEBUG:boto:Token: None 2024-04-12T03:58:48.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.900 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9dffe08b9c189fc-006618b178-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.902 DEBUG:boto:b'dkyges-60obj000000000000000000291000falseobj000000000000000000292024-04-12T03:58:33.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.903 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.903 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.903 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000029 2024-04-12T03:58:48.903 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000029 2024-04-12T03:58:48.903 DEBUG:boto:Method: GET 2024-04-12T03:58:48.903 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000029 2024-04-12T03:58:48.903 DEBUG:boto:Data: 2024-04-12T03:58:48.903 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.904 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.904 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.904 DEBUG:boto:Params: {} 2024-04-12T03:58:48.904 DEBUG:boto:Token: None 2024-04-12T03:58:48.904 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.904 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f17be84fb5b59f50-006618b178-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.908 DEBUG:boto:b'dkyges-60obj000000000000000000291000falseobj000000000000000000292024-04-12T03:58:33.835Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-04-12T03:58:48.909 DEBUG:boto:path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.909 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.909 DEBUG:boto:path=/dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.909 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.909 DEBUG:boto:Method: GET 2024-04-12T03:58:48.909 DEBUG:boto:Path: /dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.909 DEBUG:boto:Data: 2024-04-12T03:58:48.909 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.909 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.910 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.910 DEBUG:boto:Params: {} 2024-04-12T03:58:48.910 DEBUG:boto:Token: None 2024-04-12T03:58:48.910 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.910 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kWBif8eGdzgn2qzgijrVDLHCPAg= 2024-04-12T03:58:48.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kWBif8eGdzgn2qzgijrVDLHCPAg=', 'Content-Length': '0'} 2024-04-12T03:58:48.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000295bf8a884d4c7ed-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.954 DEBUG:boto:path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.954 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.955 DEBUG:boto:path=/dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.955 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.955 DEBUG:boto:Method: GET 2024-04-12T03:58:48.955 DEBUG:boto:Path: /dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.955 DEBUG:boto:Data: 2024-04-12T03:58:48.955 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.955 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.955 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.955 DEBUG:boto:Params: {} 2024-04-12T03:58:48.955 DEBUG:boto:Token: None 2024-04-12T03:58:48.955 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.956 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kWBif8eGdzgn2qzgijrVDLHCPAg= 2024-04-12T03:58:48.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kWBif8eGdzgn2qzgijrVDLHCPAg=', 'Content-Length': '0'} 2024-04-12T03:58:48.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000be5721a59b6d903-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.958 DEBUG:boto:path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.958 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.958 DEBUG:boto:path=/dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.958 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.958 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.958 DEBUG:boto:Path: /dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.958 DEBUG:boto:Data: 2024-04-12T03:58:48.958 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.958 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.958 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.959 DEBUG:boto:Params: {} 2024-04-12T03:58:48.959 DEBUG:boto:Token: None 2024-04-12T03:58:48.959 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.959 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xOVG9oA0Rm+OVe5AQDX6cFqJMtU= 2024-04-12T03:58:48.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xOVG9oA0Rm+OVe5AQDX6cFqJMtU=', 'Content-Length': '0'} 2024-04-12T03:58:48.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcedab223b48ee74-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.961 DEBUG:boto:path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.961 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.961 DEBUG:boto:path=/dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.961 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.961 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.961 DEBUG:boto:Path: /dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.961 DEBUG:boto:Data: 2024-04-12T03:58:48.962 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.962 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.962 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.962 DEBUG:boto:Params: {} 2024-04-12T03:58:48.962 DEBUG:boto:Token: None 2024-04-12T03:58:48.962 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.962 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xOVG9oA0Rm+OVe5AQDX6cFqJMtU= 2024-04-12T03:58:48.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xOVG9oA0Rm+OVe5AQDX6cFqJMtU=', 'Content-Length': '0'} 2024-04-12T03:58:48.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000628f896bf0d168e9-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-04-12T03:58:48.964 DEBUG:boto:path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.965 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.965 DEBUG:boto:path=/dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.965 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.965 DEBUG:boto:Method: GET 2024-04-12T03:58:48.965 DEBUG:boto:Path: /dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.965 DEBUG:boto:Data: 2024-04-12T03:58:48.965 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.965 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.965 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.965 DEBUG:boto:Params: {} 2024-04-12T03:58:48.966 DEBUG:boto:Token: None 2024-04-12T03:58:48.966 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.966 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kWBif8eGdzgn2qzgijrVDLHCPAg= 2024-04-12T03:58:48.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kWBif8eGdzgn2qzgijrVDLHCPAg=', 'Content-Length': '0'} 2024-04-12T03:58:48.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f500333bec653dd-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.968 DEBUG:boto:path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.968 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.968 DEBUG:boto:path=/dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.968 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.968 DEBUG:boto:Method: GET 2024-04-12T03:58:48.968 DEBUG:boto:Path: /dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.968 DEBUG:boto:Data: 2024-04-12T03:58:48.968 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.969 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.969 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.969 DEBUG:boto:Params: {} 2024-04-12T03:58:48.969 DEBUG:boto:Token: None 2024-04-12T03:58:48.969 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000030?versionId=null 2024-04-12T03:58:48.969 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kWBif8eGdzgn2qzgijrVDLHCPAg= 2024-04-12T03:58:48.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kWBif8eGdzgn2qzgijrVDLHCPAg=', 'Content-Length': '0'} 2024-04-12T03:58:48.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cbaf9e67bef9c6f-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.971 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.972 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.972 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000030 2024-04-12T03:58:48.972 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000030 2024-04-12T03:58:48.972 DEBUG:boto:Method: GET 2024-04-12T03:58:48.972 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000030 2024-04-12T03:58:48.972 DEBUG:boto:Data: 2024-04-12T03:58:48.972 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.972 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.972 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.972 DEBUG:boto:Params: {} 2024-04-12T03:58:48.972 DEBUG:boto:Token: None 2024-04-12T03:58:48.973 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.973 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000401228d6d026d9d1-006618b178-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.975 DEBUG:boto:b'dkyges-60obj000000000000000000301000falseobj000000000000000000302024-04-12T03:58:34.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.976 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:48.976 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:48.976 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000030 2024-04-12T03:58:48.976 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000030 2024-04-12T03:58:48.976 DEBUG:boto:Method: GET 2024-04-12T03:58:48.976 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000030 2024-04-12T03:58:48.976 DEBUG:boto:Data: 2024-04-12T03:58:48.976 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.976 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.976 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.977 DEBUG:boto:Params: {} 2024-04-12T03:58:48.977 DEBUG:boto:Token: None 2024-04-12T03:58:48.977 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/ 2024-04-12T03:58:48.977 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI= 2024-04-12T03:58:48.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v4ph9m5mN52l37r8VSzS6NwB8tI=', 'Content-Length': '0'} 2024-04-12T03:58:48.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005572197e160836a9-006618b178-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.981 DEBUG:boto:b'dkyges-60obj000000000000000000301000falseobj000000000000000000302024-04-12T03:58:34.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:48.982 DEBUG:boto:path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.982 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.982 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.982 DEBUG:boto:Path: /dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.982 DEBUG:boto:Data: 2024-04-12T03:58:48.982 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.982 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.982 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.983 DEBUG:boto:Params: {} 2024-04-12T03:58:48.983 DEBUG:boto:Token: None 2024-04-12T03:58:48.983 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.983 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OeLla6mvDE2W3iCLEVyBufTx4oo= 2024-04-12T03:58:48.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OeLla6mvDE2W3iCLEVyBufTx4oo=', 'Content-Length': '0'} 2024-04-12T03:58:48.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cb7258ee52b6437-006618b178-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.985 DEBUG:boto:path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.985 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.985 DEBUG:boto:Method: HEAD 2024-04-12T03:58:48.985 DEBUG:boto:Path: /dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.985 DEBUG:boto:Data: 2024-04-12T03:58:48.985 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.985 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.985 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.986 DEBUG:boto:Params: {} 2024-04-12T03:58:48.986 DEBUG:boto:Token: None 2024-04-12T03:58:48.986 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.986 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OeLla6mvDE2W3iCLEVyBufTx4oo= 2024-04-12T03:58:48.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OeLla6mvDE2W3iCLEVyBufTx4oo=', 'Content-Length': '0'} 2024-04-12T03:58:48.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000610d75e7f13b3735-006618b178-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2024-04-12T03:58:48.988 DEBUG:boto:path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.988 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.988 DEBUG:boto:Method: GET 2024-04-12T03:58:48.988 DEBUG:boto:Path: /dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.988 DEBUG:boto:Data: 2024-04-12T03:58:48.988 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.989 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:48.989 DEBUG:boto:Port: 8001 2024-04-12T03:58:48.989 DEBUG:boto:Params: {} 2024-04-12T03:58:48.989 DEBUG:boto:Token: None 2024-04-12T03:58:48.989 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.989 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i3TX+8X82ELMMDiepGDaEWfwmMA= 2024-04-12T03:58:48.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i3TX+8X82ELMMDiepGDaEWfwmMA=', 'Content-Length': '0'} 2024-04-12T03:58:48.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be8cc0421b146b5e-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:48.991 DEBUG:boto:path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.991 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.991 DEBUG:boto:Method: GET 2024-04-12T03:58:48.992 DEBUG:boto:Path: /dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.992 DEBUG:boto:Data: 2024-04-12T03:58:48.992 DEBUG:boto:Headers: {} 2024-04-12T03:58:48.992 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:48.992 DEBUG:boto:Port: 8000 2024-04-12T03:58:48.992 DEBUG:boto:Params: {} 2024-04-12T03:58:48.992 DEBUG:boto:Token: None 2024-04-12T03:58:48.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:48 GMT /dkyges-60/obj00000000000000000030 2024-04-12T03:58:48.992 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i3TX+8X82ELMMDiepGDaEWfwmMA= 2024-04-12T03:58:48.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i3TX+8X82ELMMDiepGDaEWfwmMA=', 'Content-Length': '0'} 2024-04-12T03:58:48.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000932541d2997620d2-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.038 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.038 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.038 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000030 2024-04-12T03:58:49.039 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000030 2024-04-12T03:58:49.039 DEBUG:boto:Method: GET 2024-04-12T03:58:49.039 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000030 2024-04-12T03:58:49.039 DEBUG:boto:Data: 2024-04-12T03:58:49.039 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.039 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.039 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.039 DEBUG:boto:Params: {} 2024-04-12T03:58:49.039 DEBUG:boto:Token: None 2024-04-12T03:58:49.039 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.039 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028f1197c35bc7f22-006618b179-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.042 DEBUG:boto:b'dkyges-60obj000000000000000000301000falseobj000000000000000000302024-04-12T03:58:34.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.042 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.043 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.043 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000030 2024-04-12T03:58:49.043 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000030 2024-04-12T03:58:49.043 DEBUG:boto:Method: GET 2024-04-12T03:58:49.043 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000030 2024-04-12T03:58:49.043 DEBUG:boto:Data: 2024-04-12T03:58:49.043 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.043 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.043 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.043 DEBUG:boto:Params: {} 2024-04-12T03:58:49.044 DEBUG:boto:Token: None 2024-04-12T03:58:49.044 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.044 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad74da64ef8b8f4e-006618b179-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.048 DEBUG:boto:b'dkyges-60obj000000000000000000301000falseobj000000000000000000302024-04-12T03:58:34.783Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-04-12T03:58:49.049 DEBUG:boto:path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.049 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.049 DEBUG:boto:path=/dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.049 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.049 DEBUG:boto:Method: GET 2024-04-12T03:58:49.049 DEBUG:boto:Path: /dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.049 DEBUG:boto:Data: 2024-04-12T03:58:49.049 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.049 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.049 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.050 DEBUG:boto:Params: {} 2024-04-12T03:58:49.050 DEBUG:boto:Token: None 2024-04-12T03:58:49.050 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.050 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZYWth77Ngr6YI/0x5dofBoa4Ktc= 2024-04-12T03:58:49.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZYWth77Ngr6YI/0x5dofBoa4Ktc=', 'Content-Length': '0'} 2024-04-12T03:58:49.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a74ad16aa4ccc112-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.052 DEBUG:boto:path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.053 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.053 DEBUG:boto:path=/dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.053 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.053 DEBUG:boto:Method: GET 2024-04-12T03:58:49.053 DEBUG:boto:Path: /dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.053 DEBUG:boto:Data: 2024-04-12T03:58:49.053 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.053 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.053 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.053 DEBUG:boto:Params: {} 2024-04-12T03:58:49.054 DEBUG:boto:Token: None 2024-04-12T03:58:49.054 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.054 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZYWth77Ngr6YI/0x5dofBoa4Ktc= 2024-04-12T03:58:49.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZYWth77Ngr6YI/0x5dofBoa4Ktc=', 'Content-Length': '0'} 2024-04-12T03:58:49.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb14c7b7b638b6de-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.056 DEBUG:boto:path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.056 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.057 DEBUG:boto:path=/dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.057 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.057 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.057 DEBUG:boto:Path: /dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.057 DEBUG:boto:Data: 2024-04-12T03:58:49.057 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.057 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.057 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.057 DEBUG:boto:Params: {} 2024-04-12T03:58:49.057 DEBUG:boto:Token: None 2024-04-12T03:58:49.058 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.058 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iBvgedKcXBPckS/hK0N/EPuncPs= 2024-04-12T03:58:49.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iBvgedKcXBPckS/hK0N/EPuncPs=', 'Content-Length': '0'} 2024-04-12T03:58:49.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000544e4284eaaaa684-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.060 DEBUG:boto:path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.060 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.060 DEBUG:boto:path=/dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.060 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.060 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.060 DEBUG:boto:Path: /dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.060 DEBUG:boto:Data: 2024-04-12T03:58:49.060 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.060 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.060 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.060 DEBUG:boto:Params: {} 2024-04-12T03:58:49.061 DEBUG:boto:Token: None 2024-04-12T03:58:49.061 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.061 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iBvgedKcXBPckS/hK0N/EPuncPs= 2024-04-12T03:58:49.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iBvgedKcXBPckS/hK0N/EPuncPs=', 'Content-Length': '0'} 2024-04-12T03:58:49.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000371ac2df1f752d47-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-04-12T03:58:49.063 DEBUG:boto:path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.063 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.063 DEBUG:boto:path=/dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.064 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.064 DEBUG:boto:Method: GET 2024-04-12T03:58:49.064 DEBUG:boto:Path: /dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.064 DEBUG:boto:Data: 2024-04-12T03:58:49.064 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.064 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.064 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.064 DEBUG:boto:Params: {} 2024-04-12T03:58:49.064 DEBUG:boto:Token: None 2024-04-12T03:58:49.064 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.065 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZYWth77Ngr6YI/0x5dofBoa4Ktc= 2024-04-12T03:58:49.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZYWth77Ngr6YI/0x5dofBoa4Ktc=', 'Content-Length': '0'} 2024-04-12T03:58:49.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051f2aaa15d7751d6-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.067 DEBUG:boto:path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.067 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.067 DEBUG:boto:path=/dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.067 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.067 DEBUG:boto:Method: GET 2024-04-12T03:58:49.067 DEBUG:boto:Path: /dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.067 DEBUG:boto:Data: 2024-04-12T03:58:49.068 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.068 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.068 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.068 DEBUG:boto:Params: {} 2024-04-12T03:58:49.068 DEBUG:boto:Token: None 2024-04-12T03:58:49.068 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000031?versionId=null 2024-04-12T03:58:49.068 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZYWth77Ngr6YI/0x5dofBoa4Ktc= 2024-04-12T03:58:49.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZYWth77Ngr6YI/0x5dofBoa4Ktc=', 'Content-Length': '0'} 2024-04-12T03:58:49.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5b965a5469d87df-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.118 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.118 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.119 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000031 2024-04-12T03:58:49.119 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000031 2024-04-12T03:58:49.119 DEBUG:boto:Method: GET 2024-04-12T03:58:49.119 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000031 2024-04-12T03:58:49.119 DEBUG:boto:Data: 2024-04-12T03:58:49.119 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.119 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.119 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.119 DEBUG:boto:Params: {} 2024-04-12T03:58:49.119 DEBUG:boto:Token: None 2024-04-12T03:58:49.119 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.119 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e84c265d1da86e4b-006618b179-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.122 DEBUG:boto:b'dkyges-60obj000000000000000000311000falseobj000000000000000000312024-04-12T03:58:33.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.123 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.123 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.123 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000031 2024-04-12T03:58:49.123 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000031 2024-04-12T03:58:49.123 DEBUG:boto:Method: GET 2024-04-12T03:58:49.123 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000031 2024-04-12T03:58:49.123 DEBUG:boto:Data: 2024-04-12T03:58:49.123 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.123 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.123 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.124 DEBUG:boto:Params: {} 2024-04-12T03:58:49.124 DEBUG:boto:Token: None 2024-04-12T03:58:49.124 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.124 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f5737f7b0dc84ac-006618b179-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.129 DEBUG:boto:b'dkyges-60obj000000000000000000311000falseobj000000000000000000312024-04-12T03:58:33.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.129 DEBUG:boto:path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.130 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.130 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.130 DEBUG:boto:Path: /dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.130 DEBUG:boto:Data: 2024-04-12T03:58:49.130 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.130 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.130 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.130 DEBUG:boto:Params: {} 2024-04-12T03:58:49.130 DEBUG:boto:Token: None 2024-04-12T03:58:49.130 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.131 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ABlIIYMGHlPg0jrctPz3z11okHA= 2024-04-12T03:58:49.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ABlIIYMGHlPg0jrctPz3z11okHA=', 'Content-Length': '0'} 2024-04-12T03:58:49.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001588ad90aa53b6a7-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.132 DEBUG:boto:path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.133 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.133 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.133 DEBUG:boto:Path: /dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.133 DEBUG:boto:Data: 2024-04-12T03:58:49.133 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.133 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.133 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.133 DEBUG:boto:Params: {} 2024-04-12T03:58:49.133 DEBUG:boto:Token: None 2024-04-12T03:58:49.133 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.134 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ABlIIYMGHlPg0jrctPz3z11okHA= 2024-04-12T03:58:49.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ABlIIYMGHlPg0jrctPz3z11okHA=', 'Content-Length': '0'} 2024-04-12T03:58:49.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e342387894254f3-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2024-04-12T03:58:49.136 DEBUG:boto:path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.136 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.136 DEBUG:boto:Method: GET 2024-04-12T03:58:49.136 DEBUG:boto:Path: /dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.136 DEBUG:boto:Data: 2024-04-12T03:58:49.136 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.136 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.136 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.137 DEBUG:boto:Params: {} 2024-04-12T03:58:49.137 DEBUG:boto:Token: None 2024-04-12T03:58:49.137 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.137 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:76+fA6lbY2VpsPCwfKGyu9jk5Ho= 2024-04-12T03:58:49.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:76+fA6lbY2VpsPCwfKGyu9jk5Ho=', 'Content-Length': '0'} 2024-04-12T03:58:49.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d16bd644b7dee82-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.182 DEBUG:boto:path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.182 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.183 DEBUG:boto:Method: GET 2024-04-12T03:58:49.183 DEBUG:boto:Path: /dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.183 DEBUG:boto:Data: 2024-04-12T03:58:49.183 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.183 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.183 DEBUG:boto:Params: {} 2024-04-12T03:58:49.183 DEBUG:boto:Token: None 2024-04-12T03:58:49.183 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000031 2024-04-12T03:58:49.184 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:76+fA6lbY2VpsPCwfKGyu9jk5Ho= 2024-04-12T03:58:49.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:76+fA6lbY2VpsPCwfKGyu9jk5Ho=', 'Content-Length': '0'} 2024-04-12T03:58:49.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083ec5e53169a0fda-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.186 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.186 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.186 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000031 2024-04-12T03:58:49.186 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000031 2024-04-12T03:58:49.186 DEBUG:boto:Method: GET 2024-04-12T03:58:49.186 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000031 2024-04-12T03:58:49.187 DEBUG:boto:Data: 2024-04-12T03:58:49.187 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.187 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.187 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.187 DEBUG:boto:Params: {} 2024-04-12T03:58:49.187 DEBUG:boto:Token: None 2024-04-12T03:58:49.187 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.187 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.191 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009881a557ed183b69-006618b179-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.191 DEBUG:boto:b'dkyges-60obj000000000000000000311000falseobj000000000000000000312024-04-12T03:58:33.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.191 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.191 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.191 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000031 2024-04-12T03:58:49.192 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000031 2024-04-12T03:58:49.192 DEBUG:boto:Method: GET 2024-04-12T03:58:49.192 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000031 2024-04-12T03:58:49.192 DEBUG:boto:Data: 2024-04-12T03:58:49.192 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.192 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.192 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.192 DEBUG:boto:Params: {} 2024-04-12T03:58:49.192 DEBUG:boto:Token: None 2024-04-12T03:58:49.192 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.193 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccec8849c620646e-006618b179-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.197 DEBUG:boto:b'dkyges-60obj000000000000000000311000falseobj000000000000000000312024-04-12T03:58:33.123Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-04-12T03:58:49.198 DEBUG:boto:path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.198 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.198 DEBUG:boto:path=/dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.198 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.198 DEBUG:boto:Method: GET 2024-04-12T03:58:49.198 DEBUG:boto:Path: /dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.198 DEBUG:boto:Data: 2024-04-12T03:58:49.198 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.198 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.199 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.199 DEBUG:boto:Params: {} 2024-04-12T03:58:49.199 DEBUG:boto:Token: None 2024-04-12T03:58:49.199 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.199 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y4Y5GbSDYv7mNZXUtDXrreWEwrA= 2024-04-12T03:58:49.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y4Y5GbSDYv7mNZXUtDXrreWEwrA=', 'Content-Length': '0'} 2024-04-12T03:58:49.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000215da70f918fde80-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.201 DEBUG:boto:path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.202 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.202 DEBUG:boto:path=/dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.202 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.202 DEBUG:boto:Method: GET 2024-04-12T03:58:49.202 DEBUG:boto:Path: /dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.202 DEBUG:boto:Data: 2024-04-12T03:58:49.202 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.202 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.203 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.203 DEBUG:boto:Params: {} 2024-04-12T03:58:49.203 DEBUG:boto:Token: None 2024-04-12T03:58:49.203 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.203 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y4Y5GbSDYv7mNZXUtDXrreWEwrA= 2024-04-12T03:58:49.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y4Y5GbSDYv7mNZXUtDXrreWEwrA=', 'Content-Length': '0'} 2024-04-12T03:58:49.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f98d6ef502ad43ee-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.246 DEBUG:boto:path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.247 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.247 DEBUG:boto:path=/dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.247 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.247 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.247 DEBUG:boto:Path: /dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.247 DEBUG:boto:Data: 2024-04-12T03:58:49.247 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.247 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.247 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.247 DEBUG:boto:Params: {} 2024-04-12T03:58:49.247 DEBUG:boto:Token: None 2024-04-12T03:58:49.248 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.248 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cqGyojmWy025HNHFKDv/y6tBfFk= 2024-04-12T03:58:49.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cqGyojmWy025HNHFKDv/y6tBfFk=', 'Content-Length': '0'} 2024-04-12T03:58:49.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1ba298be3160e15-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.250 DEBUG:boto:path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.250 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.250 DEBUG:boto:path=/dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.250 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.250 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.250 DEBUG:boto:Path: /dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.251 DEBUG:boto:Data: 2024-04-12T03:58:49.251 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.251 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.251 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.251 DEBUG:boto:Params: {} 2024-04-12T03:58:49.251 DEBUG:boto:Token: None 2024-04-12T03:58:49.251 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.251 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cqGyojmWy025HNHFKDv/y6tBfFk= 2024-04-12T03:58:49.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cqGyojmWy025HNHFKDv/y6tBfFk=', 'Content-Length': '0'} 2024-04-12T03:58:49.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e8847c4901ff582-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-04-12T03:58:49.253 DEBUG:boto:path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.254 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.254 DEBUG:boto:path=/dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.254 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.254 DEBUG:boto:Method: GET 2024-04-12T03:58:49.254 DEBUG:boto:Path: /dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.254 DEBUG:boto:Data: 2024-04-12T03:58:49.254 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.254 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.254 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.254 DEBUG:boto:Params: {} 2024-04-12T03:58:49.255 DEBUG:boto:Token: None 2024-04-12T03:58:49.255 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.255 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y4Y5GbSDYv7mNZXUtDXrreWEwrA= 2024-04-12T03:58:49.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y4Y5GbSDYv7mNZXUtDXrreWEwrA=', 'Content-Length': '0'} 2024-04-12T03:58:49.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0017e64c41f4bfc-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.257 DEBUG:boto:path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.257 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.257 DEBUG:boto:path=/dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.258 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.258 DEBUG:boto:Method: GET 2024-04-12T03:58:49.258 DEBUG:boto:Path: /dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.258 DEBUG:boto:Data: 2024-04-12T03:58:49.258 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.258 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.258 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.258 DEBUG:boto:Params: {} 2024-04-12T03:58:49.258 DEBUG:boto:Token: None 2024-04-12T03:58:49.259 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000032?versionId=null 2024-04-12T03:58:49.259 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y4Y5GbSDYv7mNZXUtDXrreWEwrA= 2024-04-12T03:58:49.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y4Y5GbSDYv7mNZXUtDXrreWEwrA=', 'Content-Length': '0'} 2024-04-12T03:58:49.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000335dad0fc8a785b-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.261 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.261 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.261 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000032 2024-04-12T03:58:49.261 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000032 2024-04-12T03:58:49.262 DEBUG:boto:Method: GET 2024-04-12T03:58:49.262 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000032 2024-04-12T03:58:49.262 DEBUG:boto:Data: 2024-04-12T03:58:49.262 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.262 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.262 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.262 DEBUG:boto:Params: {} 2024-04-12T03:58:49.262 DEBUG:boto:Token: None 2024-04-12T03:58:49.262 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.263 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3eac13a42bfe236-006618b179-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.266 DEBUG:boto:b'dkyges-60obj000000000000000000321000falseobj000000000000000000322024-04-12T03:58:31.983Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.266 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.266 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.266 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000032 2024-04-12T03:58:49.266 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000032 2024-04-12T03:58:49.266 DEBUG:boto:Method: GET 2024-04-12T03:58:49.267 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000032 2024-04-12T03:58:49.267 DEBUG:boto:Data: 2024-04-12T03:58:49.267 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.267 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.267 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.267 DEBUG:boto:Params: {} 2024-04-12T03:58:49.267 DEBUG:boto:Token: None 2024-04-12T03:58:49.267 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.267 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4e06a9a9f6ea6e1-006618b179-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.272 DEBUG:boto:b'dkyges-60obj000000000000000000321000falseobj000000000000000000322024-04-12T03:58:31.983Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.272 DEBUG:boto:path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.272 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.272 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.273 DEBUG:boto:Path: /dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.273 DEBUG:boto:Data: 2024-04-12T03:58:49.273 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.273 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.273 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.273 DEBUG:boto:Params: {} 2024-04-12T03:58:49.273 DEBUG:boto:Token: None 2024-04-12T03:58:49.273 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.273 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BWNPHFoYr6seV6ceJ1rty8q7gdQ= 2024-04-12T03:58:49.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BWNPHFoYr6seV6ceJ1rty8q7gdQ=', 'Content-Length': '0'} 2024-04-12T03:58:49.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a9e0d24526a3979-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.276 DEBUG:boto:path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.276 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.276 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.276 DEBUG:boto:Path: /dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.276 DEBUG:boto:Data: 2024-04-12T03:58:49.276 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.276 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.276 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.276 DEBUG:boto:Params: {} 2024-04-12T03:58:49.276 DEBUG:boto:Token: None 2024-04-12T03:58:49.276 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.277 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BWNPHFoYr6seV6ceJ1rty8q7gdQ= 2024-04-12T03:58:49.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BWNPHFoYr6seV6ceJ1rty8q7gdQ=', 'Content-Length': '0'} 2024-04-12T03:58:49.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0957046e3a939c8-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2024-04-12T03:58:49.279 DEBUG:boto:path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.279 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.279 DEBUG:boto:Method: GET 2024-04-12T03:58:49.279 DEBUG:boto:Path: /dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.279 DEBUG:boto:Data: 2024-04-12T03:58:49.279 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.279 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.279 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.279 DEBUG:boto:Params: {} 2024-04-12T03:58:49.280 DEBUG:boto:Token: None 2024-04-12T03:58:49.280 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.280 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fXQlLGvLzjsO+UaLxgr3HB2C7uc= 2024-04-12T03:58:49.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fXQlLGvLzjsO+UaLxgr3HB2C7uc=', 'Content-Length': '0'} 2024-04-12T03:58:49.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d06c438d25e01408-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.322 DEBUG:boto:path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.322 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.322 DEBUG:boto:Method: GET 2024-04-12T03:58:49.323 DEBUG:boto:Path: /dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.323 DEBUG:boto:Data: 2024-04-12T03:58:49.323 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.323 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.323 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.323 DEBUG:boto:Params: {} 2024-04-12T03:58:49.323 DEBUG:boto:Token: None 2024-04-12T03:58:49.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000032 2024-04-12T03:58:49.323 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fXQlLGvLzjsO+UaLxgr3HB2C7uc= 2024-04-12T03:58:49.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fXQlLGvLzjsO+UaLxgr3HB2C7uc=', 'Content-Length': '0'} 2024-04-12T03:58:49.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007387703ab39660cd-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.326 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.326 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.326 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000032 2024-04-12T03:58:49.326 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000032 2024-04-12T03:58:49.326 DEBUG:boto:Method: GET 2024-04-12T03:58:49.326 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000032 2024-04-12T03:58:49.326 DEBUG:boto:Data: 2024-04-12T03:58:49.327 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.327 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.327 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.327 DEBUG:boto:Params: {} 2024-04-12T03:58:49.327 DEBUG:boto:Token: None 2024-04-12T03:58:49.327 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.327 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c306b455ce3b39bf-006618b179-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.330 DEBUG:boto:b'dkyges-60obj000000000000000000321000falseobj000000000000000000322024-04-12T03:58:31.983Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.330 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.330 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.330 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000032 2024-04-12T03:58:49.331 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000032 2024-04-12T03:58:49.331 DEBUG:boto:Method: GET 2024-04-12T03:58:49.331 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000032 2024-04-12T03:58:49.331 DEBUG:boto:Data: 2024-04-12T03:58:49.331 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.331 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.331 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.331 DEBUG:boto:Params: {} 2024-04-12T03:58:49.331 DEBUG:boto:Token: None 2024-04-12T03:58:49.331 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.331 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000453d8ef583f44115-006618b179-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.336 DEBUG:boto:b'dkyges-60obj000000000000000000321000falseobj000000000000000000322024-04-12T03:58:31.983Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-04-12T03:58:49.336 DEBUG:boto:path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.336 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.336 DEBUG:boto:path=/dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.336 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.336 DEBUG:boto:Method: GET 2024-04-12T03:58:49.337 DEBUG:boto:Path: /dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.337 DEBUG:boto:Data: 2024-04-12T03:58:49.337 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.337 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.337 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.337 DEBUG:boto:Params: {} 2024-04-12T03:58:49.337 DEBUG:boto:Token: None 2024-04-12T03:58:49.337 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.337 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D6/QhwmsiAHHflBA89UEJyvGiqE= 2024-04-12T03:58:49.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D6/QhwmsiAHHflBA89UEJyvGiqE=', 'Content-Length': '0'} 2024-04-12T03:58:49.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e32977f262613d18-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.340 DEBUG:boto:path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.340 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.340 DEBUG:boto:path=/dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.340 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.340 DEBUG:boto:Method: GET 2024-04-12T03:58:49.340 DEBUG:boto:Path: /dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.340 DEBUG:boto:Data: 2024-04-12T03:58:49.340 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.340 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.340 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.341 DEBUG:boto:Params: {} 2024-04-12T03:58:49.341 DEBUG:boto:Token: None 2024-04-12T03:58:49.341 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.341 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D6/QhwmsiAHHflBA89UEJyvGiqE= 2024-04-12T03:58:49.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D6/QhwmsiAHHflBA89UEJyvGiqE=', 'Content-Length': '0'} 2024-04-12T03:58:49.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000422fdf8221376c10-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.386 DEBUG:boto:path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.386 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.387 DEBUG:boto:path=/dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.387 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.387 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.387 DEBUG:boto:Path: /dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.387 DEBUG:boto:Data: 2024-04-12T03:58:49.387 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.387 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.387 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.387 DEBUG:boto:Params: {} 2024-04-12T03:58:49.388 DEBUG:boto:Token: None 2024-04-12T03:58:49.388 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.388 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rosWX5+Kz4WV+PgCmKgseJjU2G8= 2024-04-12T03:58:49.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rosWX5+Kz4WV+PgCmKgseJjU2G8=', 'Content-Length': '0'} 2024-04-12T03:58:49.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f7089476cc80989-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.390 DEBUG:boto:path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.391 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.391 DEBUG:boto:path=/dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.391 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.391 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.391 DEBUG:boto:Path: /dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.391 DEBUG:boto:Data: 2024-04-12T03:58:49.391 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.391 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.391 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.391 DEBUG:boto:Params: {} 2024-04-12T03:58:49.391 DEBUG:boto:Token: None 2024-04-12T03:58:49.392 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.392 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rosWX5+Kz4WV+PgCmKgseJjU2G8= 2024-04-12T03:58:49.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rosWX5+Kz4WV+PgCmKgseJjU2G8=', 'Content-Length': '0'} 2024-04-12T03:58:49.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000197acb2533734520-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-04-12T03:58:49.394 DEBUG:boto:path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.394 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.395 DEBUG:boto:path=/dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.395 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.395 DEBUG:boto:Method: GET 2024-04-12T03:58:49.395 DEBUG:boto:Path: /dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.395 DEBUG:boto:Data: 2024-04-12T03:58:49.395 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.395 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.395 DEBUG:boto:Params: {} 2024-04-12T03:58:49.395 DEBUG:boto:Token: None 2024-04-12T03:58:49.395 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.396 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D6/QhwmsiAHHflBA89UEJyvGiqE= 2024-04-12T03:58:49.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D6/QhwmsiAHHflBA89UEJyvGiqE=', 'Content-Length': '0'} 2024-04-12T03:58:49.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005439d2bf6f9046d8-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.398 DEBUG:boto:path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.398 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.398 DEBUG:boto:path=/dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.398 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.398 DEBUG:boto:Method: GET 2024-04-12T03:58:49.398 DEBUG:boto:Path: /dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.398 DEBUG:boto:Data: 2024-04-12T03:58:49.398 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.398 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.399 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.399 DEBUG:boto:Params: {} 2024-04-12T03:58:49.399 DEBUG:boto:Token: None 2024-04-12T03:58:49.399 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000033?versionId=null 2024-04-12T03:58:49.399 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D6/QhwmsiAHHflBA89UEJyvGiqE= 2024-04-12T03:58:49.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D6/QhwmsiAHHflBA89UEJyvGiqE=', 'Content-Length': '0'} 2024-04-12T03:58:49.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeeb752a0d071cc0-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.401 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.401 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.402 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000033 2024-04-12T03:58:49.402 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000033 2024-04-12T03:58:49.402 DEBUG:boto:Method: GET 2024-04-12T03:58:49.402 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000033 2024-04-12T03:58:49.402 DEBUG:boto:Data: 2024-04-12T03:58:49.402 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.402 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.402 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.402 DEBUG:boto:Params: {} 2024-04-12T03:58:49.402 DEBUG:boto:Token: None 2024-04-12T03:58:49.402 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.403 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bf4c6d0967f2c36-006618b179-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.405 DEBUG:boto:b'dkyges-60obj000000000000000000331000falseobj000000000000000000332024-04-12T03:58:35.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.406 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.406 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.406 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000033 2024-04-12T03:58:49.406 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000033 2024-04-12T03:58:49.406 DEBUG:boto:Method: GET 2024-04-12T03:58:49.406 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000033 2024-04-12T03:58:49.406 DEBUG:boto:Data: 2024-04-12T03:58:49.406 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.406 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.406 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.406 DEBUG:boto:Params: {} 2024-04-12T03:58:49.407 DEBUG:boto:Token: None 2024-04-12T03:58:49.407 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.407 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bf17c3e6662a659-006618b179-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.411 DEBUG:boto:b'dkyges-60obj000000000000000000331000falseobj000000000000000000332024-04-12T03:58:35.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.412 DEBUG:boto:path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.412 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.412 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.412 DEBUG:boto:Path: /dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.412 DEBUG:boto:Data: 2024-04-12T03:58:49.412 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.412 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.412 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.412 DEBUG:boto:Params: {} 2024-04-12T03:58:49.413 DEBUG:boto:Token: None 2024-04-12T03:58:49.413 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.413 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9utyEn29i7wO8ZqT7FqbsdLo7oA= 2024-04-12T03:58:49.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9utyEn29i7wO8ZqT7FqbsdLo7oA=', 'Content-Length': '0'} 2024-04-12T03:58:49.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4f56ddc462c0bfc-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.415 DEBUG:boto:path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.415 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.415 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.415 DEBUG:boto:Path: /dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.415 DEBUG:boto:Data: 2024-04-12T03:58:49.415 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.415 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.415 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.416 DEBUG:boto:Params: {} 2024-04-12T03:58:49.416 DEBUG:boto:Token: None 2024-04-12T03:58:49.416 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.416 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9utyEn29i7wO8ZqT7FqbsdLo7oA= 2024-04-12T03:58:49.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9utyEn29i7wO8ZqT7FqbsdLo7oA=', 'Content-Length': '0'} 2024-04-12T03:58:49.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bac17d8cff3b03b-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2024-04-12T03:58:49.418 DEBUG:boto:path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.419 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.419 DEBUG:boto:Method: GET 2024-04-12T03:58:49.419 DEBUG:boto:Path: /dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.419 DEBUG:boto:Data: 2024-04-12T03:58:49.419 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.419 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.419 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.419 DEBUG:boto:Params: {} 2024-04-12T03:58:49.419 DEBUG:boto:Token: None 2024-04-12T03:58:49.419 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.419 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2j1lt4NLzGq0HTIP4ukKtUwqbZU= 2024-04-12T03:58:49.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2j1lt4NLzGq0HTIP4ukKtUwqbZU=', 'Content-Length': '0'} 2024-04-12T03:58:49.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec977bdde8302cfd-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.421 DEBUG:boto:path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.422 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.422 DEBUG:boto:Method: GET 2024-04-12T03:58:49.422 DEBUG:boto:Path: /dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.422 DEBUG:boto:Data: 2024-04-12T03:58:49.422 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.422 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.422 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.422 DEBUG:boto:Params: {} 2024-04-12T03:58:49.422 DEBUG:boto:Token: None 2024-04-12T03:58:49.423 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000033 2024-04-12T03:58:49.423 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2j1lt4NLzGq0HTIP4ukKtUwqbZU= 2024-04-12T03:58:49.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2j1lt4NLzGq0HTIP4ukKtUwqbZU=', 'Content-Length': '0'} 2024-04-12T03:58:49.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009711526d883242a6-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.425 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.425 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.425 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000033 2024-04-12T03:58:49.425 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000033 2024-04-12T03:58:49.425 DEBUG:boto:Method: GET 2024-04-12T03:58:49.425 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000033 2024-04-12T03:58:49.425 DEBUG:boto:Data: 2024-04-12T03:58:49.425 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.425 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.426 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.426 DEBUG:boto:Params: {} 2024-04-12T03:58:49.426 DEBUG:boto:Token: None 2024-04-12T03:58:49.426 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.426 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000966be06d2f094dfd-006618b179-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.429 DEBUG:boto:b'dkyges-60obj000000000000000000331000falseobj000000000000000000332024-04-12T03:58:35.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.429 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.429 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.429 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000033 2024-04-12T03:58:49.429 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000033 2024-04-12T03:58:49.429 DEBUG:boto:Method: GET 2024-04-12T03:58:49.430 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000033 2024-04-12T03:58:49.430 DEBUG:boto:Data: 2024-04-12T03:58:49.430 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.430 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.430 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.430 DEBUG:boto:Params: {} 2024-04-12T03:58:49.430 DEBUG:boto:Token: None 2024-04-12T03:58:49.430 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.430 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c35a267499c0140b-006618b179-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.435 DEBUG:boto:b'dkyges-60obj000000000000000000331000falseobj000000000000000000332024-04-12T03:58:35.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-04-12T03:58:49.435 DEBUG:boto:path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.435 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.435 DEBUG:boto:path=/dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.436 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.436 DEBUG:boto:Method: GET 2024-04-12T03:58:49.436 DEBUG:boto:Path: /dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.436 DEBUG:boto:Data: 2024-04-12T03:58:49.436 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.436 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.436 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.436 DEBUG:boto:Params: {} 2024-04-12T03:58:49.436 DEBUG:boto:Token: None 2024-04-12T03:58:49.436 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.436 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VGhZlBrBajc3+heG7Bw+OGfN6+4= 2024-04-12T03:58:49.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VGhZlBrBajc3+heG7Bw+OGfN6+4=', 'Content-Length': '0'} 2024-04-12T03:58:49.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea29a84281286d0c-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.439 DEBUG:boto:path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.439 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.439 DEBUG:boto:path=/dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.439 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.439 DEBUG:boto:Method: GET 2024-04-12T03:58:49.439 DEBUG:boto:Path: /dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.439 DEBUG:boto:Data: 2024-04-12T03:58:49.439 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.440 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.440 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.440 DEBUG:boto:Params: {} 2024-04-12T03:58:49.440 DEBUG:boto:Token: None 2024-04-12T03:58:49.440 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.440 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VGhZlBrBajc3+heG7Bw+OGfN6+4= 2024-04-12T03:58:49.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VGhZlBrBajc3+heG7Bw+OGfN6+4=', 'Content-Length': '0'} 2024-04-12T03:58:49.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009d99db4349afdc8-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.486 DEBUG:boto:path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.486 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.486 DEBUG:boto:path=/dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.486 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.487 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.487 DEBUG:boto:Path: /dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.487 DEBUG:boto:Data: 2024-04-12T03:58:49.487 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.487 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.487 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.487 DEBUG:boto:Params: {} 2024-04-12T03:58:49.487 DEBUG:boto:Token: None 2024-04-12T03:58:49.487 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.488 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4i0U93e8OzpodBJhdsbK+TphO+0= 2024-04-12T03:58:49.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4i0U93e8OzpodBJhdsbK+TphO+0=', 'Content-Length': '0'} 2024-04-12T03:58:49.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfb471d5806c9e2a-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.491 DEBUG:boto:path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.491 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.491 DEBUG:boto:path=/dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.491 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.491 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.491 DEBUG:boto:Path: /dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.491 DEBUG:boto:Data: 2024-04-12T03:58:49.491 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.491 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.492 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.492 DEBUG:boto:Params: {} 2024-04-12T03:58:49.492 DEBUG:boto:Token: None 2024-04-12T03:58:49.492 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.492 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4i0U93e8OzpodBJhdsbK+TphO+0= 2024-04-12T03:58:49.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4i0U93e8OzpodBJhdsbK+TphO+0=', 'Content-Length': '0'} 2024-04-12T03:58:49.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006be0a4baa33cc44e-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-04-12T03:58:49.495 DEBUG:boto:path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.495 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.495 DEBUG:boto:path=/dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.495 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.495 DEBUG:boto:Method: GET 2024-04-12T03:58:49.495 DEBUG:boto:Path: /dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.495 DEBUG:boto:Data: 2024-04-12T03:58:49.495 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.495 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.495 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.495 DEBUG:boto:Params: {} 2024-04-12T03:58:49.496 DEBUG:boto:Token: None 2024-04-12T03:58:49.496 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.496 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VGhZlBrBajc3+heG7Bw+OGfN6+4= 2024-04-12T03:58:49.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VGhZlBrBajc3+heG7Bw+OGfN6+4=', 'Content-Length': '0'} 2024-04-12T03:58:49.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006315056065d82dc-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.542 DEBUG:boto:path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.542 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.543 DEBUG:boto:path=/dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.543 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.543 DEBUG:boto:Method: GET 2024-04-12T03:58:49.543 DEBUG:boto:Path: /dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.543 DEBUG:boto:Data: 2024-04-12T03:58:49.543 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.543 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.543 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.543 DEBUG:boto:Params: {} 2024-04-12T03:58:49.543 DEBUG:boto:Token: None 2024-04-12T03:58:49.544 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000036?versionId=null 2024-04-12T03:58:49.544 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VGhZlBrBajc3+heG7Bw+OGfN6+4= 2024-04-12T03:58:49.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VGhZlBrBajc3+heG7Bw+OGfN6+4=', 'Content-Length': '0'} 2024-04-12T03:58:49.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001abdbd9b52dfbe87-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.546 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.546 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.546 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000036 2024-04-12T03:58:49.546 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000036 2024-04-12T03:58:49.546 DEBUG:boto:Method: GET 2024-04-12T03:58:49.547 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000036 2024-04-12T03:58:49.547 DEBUG:boto:Data: 2024-04-12T03:58:49.547 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.547 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.547 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.547 DEBUG:boto:Params: {} 2024-04-12T03:58:49.547 DEBUG:boto:Token: None 2024-04-12T03:58:49.547 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.547 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b13453b70013c2f-006618b179-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.550 DEBUG:boto:b'dkyges-60obj000000000000000000361000falseobj000000000000000000362024-04-12T03:58:35.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.550 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.550 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.550 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000036 2024-04-12T03:58:49.551 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000036 2024-04-12T03:58:49.551 DEBUG:boto:Method: GET 2024-04-12T03:58:49.551 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000036 2024-04-12T03:58:49.551 DEBUG:boto:Data: 2024-04-12T03:58:49.551 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.551 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.551 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.551 DEBUG:boto:Params: {} 2024-04-12T03:58:49.551 DEBUG:boto:Token: None 2024-04-12T03:58:49.551 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.552 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b2e3d04c498f90a-006618b179-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.556 DEBUG:boto:b'dkyges-60obj000000000000000000361000falseobj000000000000000000362024-04-12T03:58:35.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.556 DEBUG:boto:path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.556 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.556 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.557 DEBUG:boto:Path: /dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.557 DEBUG:boto:Data: 2024-04-12T03:58:49.557 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.557 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.557 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.557 DEBUG:boto:Params: {} 2024-04-12T03:58:49.557 DEBUG:boto:Token: None 2024-04-12T03:58:49.557 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.557 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNALQHMs3LjgvxqZIwc4SHw8iss= 2024-04-12T03:58:49.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNALQHMs3LjgvxqZIwc4SHw8iss=', 'Content-Length': '0'} 2024-04-12T03:58:49.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b20789d951d2c27-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.559 DEBUG:boto:path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.559 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.560 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.560 DEBUG:boto:Path: /dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.560 DEBUG:boto:Data: 2024-04-12T03:58:49.560 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.560 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.560 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.560 DEBUG:boto:Params: {} 2024-04-12T03:58:49.560 DEBUG:boto:Token: None 2024-04-12T03:58:49.560 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.560 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNALQHMs3LjgvxqZIwc4SHw8iss= 2024-04-12T03:58:49.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNALQHMs3LjgvxqZIwc4SHw8iss=', 'Content-Length': '0'} 2024-04-12T03:58:49.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b25e2fc3c1c3694-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2024-04-12T03:58:49.563 DEBUG:boto:path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.563 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.563 DEBUG:boto:Method: GET 2024-04-12T03:58:49.563 DEBUG:boto:Path: /dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.563 DEBUG:boto:Data: 2024-04-12T03:58:49.563 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.563 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.563 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.563 DEBUG:boto:Params: {} 2024-04-12T03:58:49.563 DEBUG:boto:Token: None 2024-04-12T03:58:49.564 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.564 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:c5dXgwJGZnWb9D55yLR6puPpq74= 2024-04-12T03:58:49.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:c5dXgwJGZnWb9D55yLR6puPpq74=', 'Content-Length': '0'} 2024-04-12T03:58:49.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e95ff1cb8e7ac8d7-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.566 DEBUG:boto:path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.566 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.566 DEBUG:boto:Method: GET 2024-04-12T03:58:49.566 DEBUG:boto:Path: /dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.566 DEBUG:boto:Data: 2024-04-12T03:58:49.566 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.566 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.566 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.567 DEBUG:boto:Params: {} 2024-04-12T03:58:49.567 DEBUG:boto:Token: None 2024-04-12T03:58:49.567 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000036 2024-04-12T03:58:49.567 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:c5dXgwJGZnWb9D55yLR6puPpq74= 2024-04-12T03:58:49.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:c5dXgwJGZnWb9D55yLR6puPpq74=', 'Content-Length': '0'} 2024-04-12T03:58:49.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d81a2345d7b272f-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.610 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.610 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.610 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000036 2024-04-12T03:58:49.611 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000036 2024-04-12T03:58:49.611 DEBUG:boto:Method: GET 2024-04-12T03:58:49.611 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000036 2024-04-12T03:58:49.611 DEBUG:boto:Data: 2024-04-12T03:58:49.611 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.611 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.611 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.611 DEBUG:boto:Params: {} 2024-04-12T03:58:49.611 DEBUG:boto:Token: None 2024-04-12T03:58:49.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.611 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1920b527005f536-006618b179-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.614 DEBUG:boto:b'dkyges-60obj000000000000000000361000falseobj000000000000000000362024-04-12T03:58:35.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.614 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.614 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.615 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000036 2024-04-12T03:58:49.615 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000036 2024-04-12T03:58:49.615 DEBUG:boto:Method: GET 2024-04-12T03:58:49.615 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000036 2024-04-12T03:58:49.615 DEBUG:boto:Data: 2024-04-12T03:58:49.615 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.615 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.615 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.615 DEBUG:boto:Params: {} 2024-04-12T03:58:49.615 DEBUG:boto:Token: None 2024-04-12T03:58:49.615 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.616 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064161a7e3df3d153-006618b179-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.620 DEBUG:boto:b'dkyges-60obj000000000000000000361000falseobj000000000000000000362024-04-12T03:58:35.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-04-12T03:58:49.621 DEBUG:boto:path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.621 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.621 DEBUG:boto:path=/dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.621 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.622 DEBUG:boto:Method: GET 2024-04-12T03:58:49.622 DEBUG:boto:Path: /dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.622 DEBUG:boto:Data: 2024-04-12T03:58:49.622 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.622 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.622 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.622 DEBUG:boto:Params: {} 2024-04-12T03:58:49.622 DEBUG:boto:Token: None 2024-04-12T03:58:49.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.623 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7HkOIHwKLTt0LReZlaC/zfZmwbw= 2024-04-12T03:58:49.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7HkOIHwKLTt0LReZlaC/zfZmwbw=', 'Content-Length': '0'} 2024-04-12T03:58:49.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dd9163cf7236df6-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.625 DEBUG:boto:path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.625 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.626 DEBUG:boto:path=/dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.626 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.626 DEBUG:boto:Method: GET 2024-04-12T03:58:49.626 DEBUG:boto:Path: /dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.626 DEBUG:boto:Data: 2024-04-12T03:58:49.626 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.627 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.627 DEBUG:boto:Params: {} 2024-04-12T03:58:49.627 DEBUG:boto:Token: None 2024-04-12T03:58:49.627 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.627 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7HkOIHwKLTt0LReZlaC/zfZmwbw= 2024-04-12T03:58:49.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7HkOIHwKLTt0LReZlaC/zfZmwbw=', 'Content-Length': '0'} 2024-04-12T03:58:49.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080756ffdba436715-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.630 DEBUG:boto:path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.630 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.630 DEBUG:boto:path=/dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.630 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.630 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.630 DEBUG:boto:Path: /dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.630 DEBUG:boto:Data: 2024-04-12T03:58:49.630 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.631 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.631 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.631 DEBUG:boto:Params: {} 2024-04-12T03:58:49.631 DEBUG:boto:Token: None 2024-04-12T03:58:49.631 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.631 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nPpONnhHtExpBI29EdHupP8d+8k= 2024-04-12T03:58:49.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nPpONnhHtExpBI29EdHupP8d+8k=', 'Content-Length': '0'} 2024-04-12T03:58:49.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd5148fd51dca7c0-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.633 DEBUG:boto:path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.633 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.633 DEBUG:boto:path=/dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.633 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.634 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.634 DEBUG:boto:Path: /dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.634 DEBUG:boto:Data: 2024-04-12T03:58:49.634 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.634 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.634 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.634 DEBUG:boto:Params: {} 2024-04-12T03:58:49.634 DEBUG:boto:Token: None 2024-04-12T03:58:49.634 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.635 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nPpONnhHtExpBI29EdHupP8d+8k= 2024-04-12T03:58:49.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nPpONnhHtExpBI29EdHupP8d+8k=', 'Content-Length': '0'} 2024-04-12T03:58:49.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfdc316242ef13ac-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-04-12T03:58:49.637 DEBUG:boto:path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.637 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.637 DEBUG:boto:path=/dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.637 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.637 DEBUG:boto:Method: GET 2024-04-12T03:58:49.637 DEBUG:boto:Path: /dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.637 DEBUG:boto:Data: 2024-04-12T03:58:49.637 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.637 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.638 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.638 DEBUG:boto:Params: {} 2024-04-12T03:58:49.638 DEBUG:boto:Token: None 2024-04-12T03:58:49.638 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.638 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7HkOIHwKLTt0LReZlaC/zfZmwbw= 2024-04-12T03:58:49.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7HkOIHwKLTt0LReZlaC/zfZmwbw=', 'Content-Length': '0'} 2024-04-12T03:58:49.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ac82258876d83ca-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.640 DEBUG:boto:path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.641 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.641 DEBUG:boto:path=/dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.641 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.641 DEBUG:boto:Method: GET 2024-04-12T03:58:49.641 DEBUG:boto:Path: /dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.641 DEBUG:boto:Data: 2024-04-12T03:58:49.641 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.641 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.641 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.641 DEBUG:boto:Params: {} 2024-04-12T03:58:49.642 DEBUG:boto:Token: None 2024-04-12T03:58:49.642 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000037?versionId=null 2024-04-12T03:58:49.642 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7HkOIHwKLTt0LReZlaC/zfZmwbw= 2024-04-12T03:58:49.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7HkOIHwKLTt0LReZlaC/zfZmwbw=', 'Content-Length': '0'} 2024-04-12T03:58:49.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001239822bb23c376a-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.645 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.645 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.645 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000037 2024-04-12T03:58:49.645 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000037 2024-04-12T03:58:49.645 DEBUG:boto:Method: GET 2024-04-12T03:58:49.645 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000037 2024-04-12T03:58:49.645 DEBUG:boto:Data: 2024-04-12T03:58:49.645 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.645 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.645 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.645 DEBUG:boto:Params: {} 2024-04-12T03:58:49.646 DEBUG:boto:Token: None 2024-04-12T03:58:49.646 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.646 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073cb9b682962e314-006618b179-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.648 DEBUG:boto:b'dkyges-60obj000000000000000000371000falseobj000000000000000000372024-04-12T03:58:32.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.649 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.649 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.649 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000037 2024-04-12T03:58:49.649 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000037 2024-04-12T03:58:49.649 DEBUG:boto:Method: GET 2024-04-12T03:58:49.649 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000037 2024-04-12T03:58:49.649 DEBUG:boto:Data: 2024-04-12T03:58:49.649 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.649 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.649 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.650 DEBUG:boto:Params: {} 2024-04-12T03:58:49.650 DEBUG:boto:Token: None 2024-04-12T03:58:49.650 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.650 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007eae0ed12fe7473e-006618b179-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.654 DEBUG:boto:b'dkyges-60obj000000000000000000371000falseobj000000000000000000372024-04-12T03:58:32.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.655 DEBUG:boto:path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.655 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.655 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.655 DEBUG:boto:Path: /dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.655 DEBUG:boto:Data: 2024-04-12T03:58:49.655 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.655 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.655 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.655 DEBUG:boto:Params: {} 2024-04-12T03:58:49.656 DEBUG:boto:Token: None 2024-04-12T03:58:49.656 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nA9cN/DyFhXX7pUrDuW9rAbXgM8= 2024-04-12T03:58:49.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nA9cN/DyFhXX7pUrDuW9rAbXgM8=', 'Content-Length': '0'} 2024-04-12T03:58:49.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028ea6177c3a1004e-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.658 DEBUG:boto:path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.658 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.658 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.658 DEBUG:boto:Path: /dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.658 DEBUG:boto:Data: 2024-04-12T03:58:49.658 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.658 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.658 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.658 DEBUG:boto:Params: {} 2024-04-12T03:58:49.659 DEBUG:boto:Token: None 2024-04-12T03:58:49.659 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.659 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nA9cN/DyFhXX7pUrDuW9rAbXgM8= 2024-04-12T03:58:49.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nA9cN/DyFhXX7pUrDuW9rAbXgM8=', 'Content-Length': '0'} 2024-04-12T03:58:49.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054fdbcd168f926d5-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2024-04-12T03:58:49.661 DEBUG:boto:path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.661 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.661 DEBUG:boto:Method: GET 2024-04-12T03:58:49.661 DEBUG:boto:Path: /dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.661 DEBUG:boto:Data: 2024-04-12T03:58:49.662 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.662 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.662 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.662 DEBUG:boto:Params: {} 2024-04-12T03:58:49.662 DEBUG:boto:Token: None 2024-04-12T03:58:49.662 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.662 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jLdURVJYlLx4EO1nNRkzisg9KJI= 2024-04-12T03:58:49.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jLdURVJYlLx4EO1nNRkzisg9KJI=', 'Content-Length': '0'} 2024-04-12T03:58:49.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a3a84226688259d-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.664 DEBUG:boto:path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.664 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.664 DEBUG:boto:Method: GET 2024-04-12T03:58:49.665 DEBUG:boto:Path: /dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.665 DEBUG:boto:Data: 2024-04-12T03:58:49.665 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.665 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.665 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.665 DEBUG:boto:Params: {} 2024-04-12T03:58:49.665 DEBUG:boto:Token: None 2024-04-12T03:58:49.665 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000037 2024-04-12T03:58:49.665 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jLdURVJYlLx4EO1nNRkzisg9KJI= 2024-04-12T03:58:49.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jLdURVJYlLx4EO1nNRkzisg9KJI=', 'Content-Length': '0'} 2024-04-12T03:58:49.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f7c164f1949836d-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.710 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.710 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.710 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000037 2024-04-12T03:58:49.711 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000037 2024-04-12T03:58:49.711 DEBUG:boto:Method: GET 2024-04-12T03:58:49.711 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000037 2024-04-12T03:58:49.711 DEBUG:boto:Data: 2024-04-12T03:58:49.711 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.711 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.711 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.711 DEBUG:boto:Params: {} 2024-04-12T03:58:49.711 DEBUG:boto:Token: None 2024-04-12T03:58:49.711 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.711 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008db66ae2cf6f6b85-006618b179-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.714 DEBUG:boto:b'dkyges-60obj000000000000000000371000falseobj000000000000000000372024-04-12T03:58:32.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.715 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.715 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.715 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000037 2024-04-12T03:58:49.715 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000037 2024-04-12T03:58:49.715 DEBUG:boto:Method: GET 2024-04-12T03:58:49.715 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000037 2024-04-12T03:58:49.715 DEBUG:boto:Data: 2024-04-12T03:58:49.715 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.715 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.715 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.715 DEBUG:boto:Params: {} 2024-04-12T03:58:49.715 DEBUG:boto:Token: None 2024-04-12T03:58:49.716 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.716 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cf1ecaa8a0afe4e-006618b179-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.721 DEBUG:boto:b'dkyges-60obj000000000000000000371000falseobj000000000000000000372024-04-12T03:58:32.547Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-04-12T03:58:49.721 DEBUG:boto:path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.721 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.721 DEBUG:boto:path=/dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.721 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.721 DEBUG:boto:Method: GET 2024-04-12T03:58:49.721 DEBUG:boto:Path: /dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.721 DEBUG:boto:Data: 2024-04-12T03:58:49.722 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.722 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.722 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.722 DEBUG:boto:Params: {} 2024-04-12T03:58:49.722 DEBUG:boto:Token: None 2024-04-12T03:58:49.722 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.722 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dvP7FWuBYfA4YQt45f09Ge1rkDA= 2024-04-12T03:58:49.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dvP7FWuBYfA4YQt45f09Ge1rkDA=', 'Content-Length': '0'} 2024-04-12T03:58:49.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010f6ee77c368a9bd-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.766 DEBUG:boto:path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.766 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.766 DEBUG:boto:path=/dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.766 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.767 DEBUG:boto:Method: GET 2024-04-12T03:58:49.767 DEBUG:boto:Path: /dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.767 DEBUG:boto:Data: 2024-04-12T03:58:49.767 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.767 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.767 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.767 DEBUG:boto:Params: {} 2024-04-12T03:58:49.767 DEBUG:boto:Token: None 2024-04-12T03:58:49.767 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.767 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dvP7FWuBYfA4YQt45f09Ge1rkDA= 2024-04-12T03:58:49.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dvP7FWuBYfA4YQt45f09Ge1rkDA=', 'Content-Length': '0'} 2024-04-12T03:58:49.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008c325ed64d1c9ca-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.770 DEBUG:boto:path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.770 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.770 DEBUG:boto:path=/dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.770 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.770 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.770 DEBUG:boto:Path: /dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.770 DEBUG:boto:Data: 2024-04-12T03:58:49.770 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.770 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.770 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.770 DEBUG:boto:Params: {} 2024-04-12T03:58:49.771 DEBUG:boto:Token: None 2024-04-12T03:58:49.771 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.771 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zayEa8X44j22Wb0tGnDYiUXi1ns= 2024-04-12T03:58:49.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zayEa8X44j22Wb0tGnDYiUXi1ns=', 'Content-Length': '0'} 2024-04-12T03:58:49.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000499657fc347cb4eb-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.773 DEBUG:boto:path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.773 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.773 DEBUG:boto:path=/dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.773 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.773 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.773 DEBUG:boto:Path: /dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.773 DEBUG:boto:Data: 2024-04-12T03:58:49.774 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.774 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.774 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.774 DEBUG:boto:Params: {} 2024-04-12T03:58:49.774 DEBUG:boto:Token: None 2024-04-12T03:58:49.774 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.774 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zayEa8X44j22Wb0tGnDYiUXi1ns= 2024-04-12T03:58:49.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zayEa8X44j22Wb0tGnDYiUXi1ns=', 'Content-Length': '0'} 2024-04-12T03:58:49.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f686ed2e4e6329f3-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-04-12T03:58:49.776 DEBUG:boto:path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.777 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.777 DEBUG:boto:path=/dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.777 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.777 DEBUG:boto:Method: GET 2024-04-12T03:58:49.777 DEBUG:boto:Path: /dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.777 DEBUG:boto:Data: 2024-04-12T03:58:49.777 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.777 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.777 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.777 DEBUG:boto:Params: {} 2024-04-12T03:58:49.777 DEBUG:boto:Token: None 2024-04-12T03:58:49.777 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.778 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dvP7FWuBYfA4YQt45f09Ge1rkDA= 2024-04-12T03:58:49.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dvP7FWuBYfA4YQt45f09Ge1rkDA=', 'Content-Length': '0'} 2024-04-12T03:58:49.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008335f9ae0c60886d-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.779 DEBUG:boto:path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.780 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.780 DEBUG:boto:path=/dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.780 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.780 DEBUG:boto:Method: GET 2024-04-12T03:58:49.780 DEBUG:boto:Path: /dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.780 DEBUG:boto:Data: 2024-04-12T03:58:49.780 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.780 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.780 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.780 DEBUG:boto:Params: {} 2024-04-12T03:58:49.780 DEBUG:boto:Token: None 2024-04-12T03:58:49.781 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000039?versionId=null 2024-04-12T03:58:49.781 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dvP7FWuBYfA4YQt45f09Ge1rkDA= 2024-04-12T03:58:49.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dvP7FWuBYfA4YQt45f09Ge1rkDA=', 'Content-Length': '0'} 2024-04-12T03:58:49.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b3f3221b6ee4e9d-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.783 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.783 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.783 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000039 2024-04-12T03:58:49.783 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000039 2024-04-12T03:58:49.783 DEBUG:boto:Method: GET 2024-04-12T03:58:49.783 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000039 2024-04-12T03:58:49.783 DEBUG:boto:Data: 2024-04-12T03:58:49.783 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.783 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.784 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.784 DEBUG:boto:Params: {} 2024-04-12T03:58:49.784 DEBUG:boto:Token: None 2024-04-12T03:58:49.784 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.784 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf4ae1fdbd4d70e4-006618b179-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.787 DEBUG:boto:b'dkyges-60obj000000000000000000391000falseobj000000000000000000392024-04-12T03:58:35.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.787 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.787 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.788 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000039 2024-04-12T03:58:49.788 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000039 2024-04-12T03:58:49.788 DEBUG:boto:Method: GET 2024-04-12T03:58:49.788 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000039 2024-04-12T03:58:49.788 DEBUG:boto:Data: 2024-04-12T03:58:49.788 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.788 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.788 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.788 DEBUG:boto:Params: {} 2024-04-12T03:58:49.788 DEBUG:boto:Token: None 2024-04-12T03:58:49.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.788 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca06378f943dddb1-006618b179-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.793 DEBUG:boto:b'dkyges-60obj000000000000000000391000falseobj000000000000000000392024-04-12T03:58:35.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.794 DEBUG:boto:path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.794 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.794 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.794 DEBUG:boto:Path: /dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.794 DEBUG:boto:Data: 2024-04-12T03:58:49.794 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.794 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.794 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.794 DEBUG:boto:Params: {} 2024-04-12T03:58:49.794 DEBUG:boto:Token: None 2024-04-12T03:58:49.795 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.795 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SR8kOrh7sBWqezirlswhzq4yr5s= 2024-04-12T03:58:49.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SR8kOrh7sBWqezirlswhzq4yr5s=', 'Content-Length': '0'} 2024-04-12T03:58:49.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002483852bdb2a69d2-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.797 DEBUG:boto:path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.797 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.797 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.797 DEBUG:boto:Path: /dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.797 DEBUG:boto:Data: 2024-04-12T03:58:49.797 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.797 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.797 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.797 DEBUG:boto:Params: {} 2024-04-12T03:58:49.798 DEBUG:boto:Token: None 2024-04-12T03:58:49.798 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.798 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SR8kOrh7sBWqezirlswhzq4yr5s= 2024-04-12T03:58:49.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SR8kOrh7sBWqezirlswhzq4yr5s=', 'Content-Length': '0'} 2024-04-12T03:58:49.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c279449bc52cbf6a-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2024-04-12T03:58:49.800 DEBUG:boto:path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.800 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.800 DEBUG:boto:Method: GET 2024-04-12T03:58:49.800 DEBUG:boto:Path: /dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.800 DEBUG:boto:Data: 2024-04-12T03:58:49.801 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.801 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.801 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.801 DEBUG:boto:Params: {} 2024-04-12T03:58:49.801 DEBUG:boto:Token: None 2024-04-12T03:58:49.801 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.801 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Co2f21NtW6EGHVUPYcc36YvSu4E= 2024-04-12T03:58:49.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Co2f21NtW6EGHVUPYcc36YvSu4E=', 'Content-Length': '0'} 2024-04-12T03:58:49.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f5b72123bce809a-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.846 DEBUG:boto:path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.847 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.847 DEBUG:boto:Method: GET 2024-04-12T03:58:49.847 DEBUG:boto:Path: /dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.847 DEBUG:boto:Data: 2024-04-12T03:58:49.847 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.847 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.847 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.847 DEBUG:boto:Params: {} 2024-04-12T03:58:49.847 DEBUG:boto:Token: None 2024-04-12T03:58:49.847 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000039 2024-04-12T03:58:49.848 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Co2f21NtW6EGHVUPYcc36YvSu4E= 2024-04-12T03:58:49.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Co2f21NtW6EGHVUPYcc36YvSu4E=', 'Content-Length': '0'} 2024-04-12T03:58:49.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033d9210e98aeb22b-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.850 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.850 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.850 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000039 2024-04-12T03:58:49.850 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000039 2024-04-12T03:58:49.850 DEBUG:boto:Method: GET 2024-04-12T03:58:49.850 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000039 2024-04-12T03:58:49.851 DEBUG:boto:Data: 2024-04-12T03:58:49.851 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.851 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.851 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.851 DEBUG:boto:Params: {} 2024-04-12T03:58:49.851 DEBUG:boto:Token: None 2024-04-12T03:58:49.851 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.851 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000061d59404927d46b-006618b179-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.854 DEBUG:boto:b'dkyges-60obj000000000000000000391000falseobj000000000000000000392024-04-12T03:58:35.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.854 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.854 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.855 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000039 2024-04-12T03:58:49.855 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000039 2024-04-12T03:58:49.855 DEBUG:boto:Method: GET 2024-04-12T03:58:49.855 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000039 2024-04-12T03:58:49.855 DEBUG:boto:Data: 2024-04-12T03:58:49.855 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.855 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.855 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.855 DEBUG:boto:Params: {} 2024-04-12T03:58:49.855 DEBUG:boto:Token: None 2024-04-12T03:58:49.856 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.856 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd47e7f4827df984-006618b179-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.860 DEBUG:boto:b'dkyges-60obj000000000000000000391000falseobj000000000000000000392024-04-12T03:58:35.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-04-12T03:58:49.861 DEBUG:boto:path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.861 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.861 DEBUG:boto:path=/dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.861 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.861 DEBUG:boto:Method: GET 2024-04-12T03:58:49.861 DEBUG:boto:Path: /dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.861 DEBUG:boto:Data: 2024-04-12T03:58:49.861 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.861 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.861 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.861 DEBUG:boto:Params: {} 2024-04-12T03:58:49.862 DEBUG:boto:Token: None 2024-04-12T03:58:49.862 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.862 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5dPooKXta6NO04Dwd7cp2k21+sI= 2024-04-12T03:58:49.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5dPooKXta6NO04Dwd7cp2k21+sI=', 'Content-Length': '0'} 2024-04-12T03:58:49.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020b6a30eea3c328a-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.865 DEBUG:boto:path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.865 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.865 DEBUG:boto:path=/dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.865 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.865 DEBUG:boto:Method: GET 2024-04-12T03:58:49.865 DEBUG:boto:Path: /dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.865 DEBUG:boto:Data: 2024-04-12T03:58:49.865 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.865 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.865 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.866 DEBUG:boto:Params: {} 2024-04-12T03:58:49.866 DEBUG:boto:Token: None 2024-04-12T03:58:49.866 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.866 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5dPooKXta6NO04Dwd7cp2k21+sI= 2024-04-12T03:58:49.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5dPooKXta6NO04Dwd7cp2k21+sI=', 'Content-Length': '0'} 2024-04-12T03:58:49.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdcaf3eb377fdb09-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.868 DEBUG:boto:path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.869 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.869 DEBUG:boto:path=/dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.869 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.869 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.869 DEBUG:boto:Path: /dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.869 DEBUG:boto:Data: 2024-04-12T03:58:49.869 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.869 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.869 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.869 DEBUG:boto:Params: {} 2024-04-12T03:58:49.869 DEBUG:boto:Token: None 2024-04-12T03:58:49.870 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YlK15YB3k3uR3WPqiskJaPZez7A= 2024-04-12T03:58:49.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YlK15YB3k3uR3WPqiskJaPZez7A=', 'Content-Length': '0'} 2024-04-12T03:58:49.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000865f9262bf8b0d58-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.872 DEBUG:boto:path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.872 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.872 DEBUG:boto:path=/dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.872 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.872 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.872 DEBUG:boto:Path: /dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.872 DEBUG:boto:Data: 2024-04-12T03:58:49.872 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.873 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.873 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.873 DEBUG:boto:Params: {} 2024-04-12T03:58:49.873 DEBUG:boto:Token: None 2024-04-12T03:58:49.873 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.873 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YlK15YB3k3uR3WPqiskJaPZez7A= 2024-04-12T03:58:49.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YlK15YB3k3uR3WPqiskJaPZez7A=', 'Content-Length': '0'} 2024-04-12T03:58:49.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000865fd35460d967a7-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-04-12T03:58:49.875 DEBUG:boto:path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.875 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.876 DEBUG:boto:path=/dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.876 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.876 DEBUG:boto:Method: GET 2024-04-12T03:58:49.876 DEBUG:boto:Path: /dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.876 DEBUG:boto:Data: 2024-04-12T03:58:49.876 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.876 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.876 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.876 DEBUG:boto:Params: {} 2024-04-12T03:58:49.876 DEBUG:boto:Token: None 2024-04-12T03:58:49.877 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.877 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5dPooKXta6NO04Dwd7cp2k21+sI= 2024-04-12T03:58:49.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5dPooKXta6NO04Dwd7cp2k21+sI=', 'Content-Length': '0'} 2024-04-12T03:58:49.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7cc4382879a0487-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.879 DEBUG:boto:path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.879 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.879 DEBUG:boto:path=/dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.879 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.879 DEBUG:boto:Method: GET 2024-04-12T03:58:49.879 DEBUG:boto:Path: /dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.879 DEBUG:boto:Data: 2024-04-12T03:58:49.879 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.879 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.879 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.879 DEBUG:boto:Params: {} 2024-04-12T03:58:49.880 DEBUG:boto:Token: None 2024-04-12T03:58:49.880 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000040?versionId=null 2024-04-12T03:58:49.880 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5dPooKXta6NO04Dwd7cp2k21+sI= 2024-04-12T03:58:49.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5dPooKXta6NO04Dwd7cp2k21+sI=', 'Content-Length': '0'} 2024-04-12T03:58:49.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e135ead5b0ce99f-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.882 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.882 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.882 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000040 2024-04-12T03:58:49.882 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000040 2024-04-12T03:58:49.882 DEBUG:boto:Method: GET 2024-04-12T03:58:49.883 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000040 2024-04-12T03:58:49.883 DEBUG:boto:Data: 2024-04-12T03:58:49.883 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.883 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.883 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.883 DEBUG:boto:Params: {} 2024-04-12T03:58:49.883 DEBUG:boto:Token: None 2024-04-12T03:58:49.883 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.883 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b8d6ebdb68f55f6-006618b179-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.886 DEBUG:boto:b'dkyges-60obj000000000000000000401000falseobj000000000000000000402024-04-12T03:58:35.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.886 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.886 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.887 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000040 2024-04-12T03:58:49.887 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000040 2024-04-12T03:58:49.887 DEBUG:boto:Method: GET 2024-04-12T03:58:49.887 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000040 2024-04-12T03:58:49.887 DEBUG:boto:Data: 2024-04-12T03:58:49.887 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.887 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.887 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.887 DEBUG:boto:Params: {} 2024-04-12T03:58:49.887 DEBUG:boto:Token: None 2024-04-12T03:58:49.888 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.888 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000171a31c01507fad8-006618b179-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.892 DEBUG:boto:b'dkyges-60obj000000000000000000401000falseobj000000000000000000402024-04-12T03:58:35.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.892 DEBUG:boto:path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.892 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.892 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.893 DEBUG:boto:Path: /dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.893 DEBUG:boto:Data: 2024-04-12T03:58:49.893 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.893 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.893 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.893 DEBUG:boto:Params: {} 2024-04-12T03:58:49.893 DEBUG:boto:Token: None 2024-04-12T03:58:49.893 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.893 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gDd0BBU19n/htYzbTJ8uMUdE2EA= 2024-04-12T03:58:49.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gDd0BBU19n/htYzbTJ8uMUdE2EA=', 'Content-Length': '0'} 2024-04-12T03:58:49.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7f4ad51816e2ba6-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.895 DEBUG:boto:path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.896 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.896 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.896 DEBUG:boto:Path: /dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.896 DEBUG:boto:Data: 2024-04-12T03:58:49.896 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.896 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.896 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.896 DEBUG:boto:Params: {} 2024-04-12T03:58:49.896 DEBUG:boto:Token: None 2024-04-12T03:58:49.897 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.897 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gDd0BBU19n/htYzbTJ8uMUdE2EA= 2024-04-12T03:58:49.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gDd0BBU19n/htYzbTJ8uMUdE2EA=', 'Content-Length': '0'} 2024-04-12T03:58:49.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c50d3ebcbcf1f299-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2024-04-12T03:58:49.899 DEBUG:boto:path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.899 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.899 DEBUG:boto:Method: GET 2024-04-12T03:58:49.899 DEBUG:boto:Path: /dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.899 DEBUG:boto:Data: 2024-04-12T03:58:49.899 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.899 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.899 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.899 DEBUG:boto:Params: {} 2024-04-12T03:58:49.900 DEBUG:boto:Token: None 2024-04-12T03:58:49.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.900 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:spUHlQzyy1oRBgaelwZ5soynF4s= 2024-04-12T03:58:49.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:spUHlQzyy1oRBgaelwZ5soynF4s=', 'Content-Length': '0'} 2024-04-12T03:58:49.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d8b665a5fae52cd-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.942 DEBUG:boto:path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.942 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.942 DEBUG:boto:Method: GET 2024-04-12T03:58:49.943 DEBUG:boto:Path: /dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.943 DEBUG:boto:Data: 2024-04-12T03:58:49.943 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.943 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.943 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.943 DEBUG:boto:Params: {} 2024-04-12T03:58:49.943 DEBUG:boto:Token: None 2024-04-12T03:58:49.943 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000040 2024-04-12T03:58:49.944 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:spUHlQzyy1oRBgaelwZ5soynF4s= 2024-04-12T03:58:49.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:spUHlQzyy1oRBgaelwZ5soynF4s=', 'Content-Length': '0'} 2024-04-12T03:58:49.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1e9cd1c518e478b-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.946 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.946 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.946 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000040 2024-04-12T03:58:49.946 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000040 2024-04-12T03:58:49.947 DEBUG:boto:Method: GET 2024-04-12T03:58:49.947 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000040 2024-04-12T03:58:49.947 DEBUG:boto:Data: 2024-04-12T03:58:49.947 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.947 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.947 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.947 DEBUG:boto:Params: {} 2024-04-12T03:58:49.947 DEBUG:boto:Token: None 2024-04-12T03:58:49.947 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.947 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dac205e8852067d-006618b179-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.950 DEBUG:boto:b'dkyges-60obj000000000000000000401000falseobj000000000000000000402024-04-12T03:58:35.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.950 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.950 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.951 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000040 2024-04-12T03:58:49.951 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000040 2024-04-12T03:58:49.951 DEBUG:boto:Method: GET 2024-04-12T03:58:49.951 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000040 2024-04-12T03:58:49.951 DEBUG:boto:Data: 2024-04-12T03:58:49.951 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.951 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.951 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.951 DEBUG:boto:Params: {} 2024-04-12T03:58:49.951 DEBUG:boto:Token: None 2024-04-12T03:58:49.951 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.951 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc53674bddfb3274-006618b179-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.956 DEBUG:boto:b'dkyges-60obj000000000000000000401000falseobj000000000000000000402024-04-12T03:58:35.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-04-12T03:58:49.957 DEBUG:boto:path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.957 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.957 DEBUG:boto:path=/dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.957 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.957 DEBUG:boto:Method: GET 2024-04-12T03:58:49.957 DEBUG:boto:Path: /dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.957 DEBUG:boto:Data: 2024-04-12T03:58:49.957 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.957 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.957 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.957 DEBUG:boto:Params: {} 2024-04-12T03:58:49.958 DEBUG:boto:Token: None 2024-04-12T03:58:49.958 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.958 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8VD9xfk07+Tc5tCCj5rCsg0UVgc= 2024-04-12T03:58:49.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8VD9xfk07+Tc5tCCj5rCsg0UVgc=', 'Content-Length': '0'} 2024-04-12T03:58:49.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030dfd55eaae3be3e-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.960 DEBUG:boto:path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.960 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.960 DEBUG:boto:path=/dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.961 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.961 DEBUG:boto:Method: GET 2024-04-12T03:58:49.961 DEBUG:boto:Path: /dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.961 DEBUG:boto:Data: 2024-04-12T03:58:49.961 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.961 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.961 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.961 DEBUG:boto:Params: {} 2024-04-12T03:58:49.961 DEBUG:boto:Token: None 2024-04-12T03:58:49.961 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.961 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8VD9xfk07+Tc5tCCj5rCsg0UVgc= 2024-04-12T03:58:49.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8VD9xfk07+Tc5tCCj5rCsg0UVgc=', 'Content-Length': '0'} 2024-04-12T03:58:49.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004982bf23279b7b1d-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.964 DEBUG:boto:path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.964 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.964 DEBUG:boto:path=/dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.964 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.964 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.964 DEBUG:boto:Path: /dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.964 DEBUG:boto:Data: 2024-04-12T03:58:49.964 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.964 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.965 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.965 DEBUG:boto:Params: {} 2024-04-12T03:58:49.965 DEBUG:boto:Token: None 2024-04-12T03:58:49.965 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.965 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pSRwu6riZ0VsDgzNp49s8vdF7ro= 2024-04-12T03:58:49.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pSRwu6riZ0VsDgzNp49s8vdF7ro=', 'Content-Length': '0'} 2024-04-12T03:58:49.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb151f1ce30aa910-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.967 DEBUG:boto:path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.967 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.967 DEBUG:boto:path=/dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.967 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.967 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.967 DEBUG:boto:Path: /dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.968 DEBUG:boto:Data: 2024-04-12T03:58:49.968 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.968 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.968 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.968 DEBUG:boto:Params: {} 2024-04-12T03:58:49.968 DEBUG:boto:Token: None 2024-04-12T03:58:49.968 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.968 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pSRwu6riZ0VsDgzNp49s8vdF7ro= 2024-04-12T03:58:49.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pSRwu6riZ0VsDgzNp49s8vdF7ro=', 'Content-Length': '0'} 2024-04-12T03:58:49.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066082b0cac658803-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-04-12T03:58:49.970 DEBUG:boto:path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.971 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.971 DEBUG:boto:path=/dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.971 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.971 DEBUG:boto:Method: GET 2024-04-12T03:58:49.971 DEBUG:boto:Path: /dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.971 DEBUG:boto:Data: 2024-04-12T03:58:49.971 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.971 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.971 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.971 DEBUG:boto:Params: {} 2024-04-12T03:58:49.971 DEBUG:boto:Token: None 2024-04-12T03:58:49.972 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.972 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8VD9xfk07+Tc5tCCj5rCsg0UVgc= 2024-04-12T03:58:49.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8VD9xfk07+Tc5tCCj5rCsg0UVgc=', 'Content-Length': '0'} 2024-04-12T03:58:49.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b521952b8b0d280e-006618b179-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.974 DEBUG:boto:path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.974 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.974 DEBUG:boto:path=/dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.974 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.974 DEBUG:boto:Method: GET 2024-04-12T03:58:49.974 DEBUG:boto:Path: /dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.974 DEBUG:boto:Data: 2024-04-12T03:58:49.974 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.974 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.975 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.975 DEBUG:boto:Params: {} 2024-04-12T03:58:49.975 DEBUG:boto:Token: None 2024-04-12T03:58:49.975 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000042?versionId=null 2024-04-12T03:58:49.975 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8VD9xfk07+Tc5tCCj5rCsg0UVgc= 2024-04-12T03:58:49.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8VD9xfk07+Tc5tCCj5rCsg0UVgc=', 'Content-Length': '0'} 2024-04-12T03:58:49.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094d3b7bedc3c3e69-006618b179-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.977 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.977 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.977 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000042 2024-04-12T03:58:49.977 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000042 2024-04-12T03:58:49.978 DEBUG:boto:Method: GET 2024-04-12T03:58:49.978 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000042 2024-04-12T03:58:49.978 DEBUG:boto:Data: 2024-04-12T03:58:49.978 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.978 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.978 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.978 DEBUG:boto:Params: {} 2024-04-12T03:58:49.978 DEBUG:boto:Token: None 2024-04-12T03:58:49.978 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.978 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d20a0587d7dff62-006618b179-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.981 DEBUG:boto:b'dkyges-60obj000000000000000000421000falseobj000000000000000000422024-04-12T03:58:32.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.981 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:49.981 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:49.981 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000042 2024-04-12T03:58:49.982 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000042 2024-04-12T03:58:49.982 DEBUG:boto:Method: GET 2024-04-12T03:58:49.982 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000042 2024-04-12T03:58:49.982 DEBUG:boto:Data: 2024-04-12T03:58:49.982 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.982 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.982 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.982 DEBUG:boto:Params: {} 2024-04-12T03:58:49.982 DEBUG:boto:Token: None 2024-04-12T03:58:49.982 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/ 2024-04-12T03:58:49.983 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4= 2024-04-12T03:58:49.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zlawK5b+ti453vDzwzj37Qm0GQ4=', 'Content-Length': '0'} 2024-04-12T03:58:49.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000197125e441193d4c-006618b179-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.988 DEBUG:boto:b'dkyges-60obj000000000000000000421000falseobj000000000000000000422024-04-12T03:58:32.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:49.988 DEBUG:boto:path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.988 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.988 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.988 DEBUG:boto:Path: /dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.988 DEBUG:boto:Data: 2024-04-12T03:58:49.988 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.988 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.989 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.989 DEBUG:boto:Params: {} 2024-04-12T03:58:49.989 DEBUG:boto:Token: None 2024-04-12T03:58:49.989 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.989 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZjmadQcwnmnIs99ea3cDmghy77Q= 2024-04-12T03:58:49.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZjmadQcwnmnIs99ea3cDmghy77Q=', 'Content-Length': '0'} 2024-04-12T03:58:49.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fda3a7340fe51cc-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.991 DEBUG:boto:path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.991 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.991 DEBUG:boto:Method: HEAD 2024-04-12T03:58:49.991 DEBUG:boto:Path: /dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.991 DEBUG:boto:Data: 2024-04-12T03:58:49.992 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.992 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:49.992 DEBUG:boto:Port: 8000 2024-04-12T03:58:49.992 DEBUG:boto:Params: {} 2024-04-12T03:58:49.992 DEBUG:boto:Token: None 2024-04-12T03:58:49.992 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.992 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZjmadQcwnmnIs99ea3cDmghy77Q= 2024-04-12T03:58:49.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZjmadQcwnmnIs99ea3cDmghy77Q=', 'Content-Length': '0'} 2024-04-12T03:58:49.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065d4255cfe731493-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:49.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2024-04-12T03:58:49.995 DEBUG:boto:path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.995 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.995 DEBUG:boto:Method: GET 2024-04-12T03:58:49.995 DEBUG:boto:Path: /dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.995 DEBUG:boto:Data: 2024-04-12T03:58:49.995 DEBUG:boto:Headers: {} 2024-04-12T03:58:49.995 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:49.995 DEBUG:boto:Port: 8001 2024-04-12T03:58:49.995 DEBUG:boto:Params: {} 2024-04-12T03:58:49.995 DEBUG:boto:Token: None 2024-04-12T03:58:49.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:49 GMT /dkyges-60/obj00000000000000000042 2024-04-12T03:58:49.996 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kRcxiO4S+yd/L1UVHJQQnNe4wUo= 2024-04-12T03:58:49.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kRcxiO4S+yd/L1UVHJQQnNe4wUo=', 'Content-Length': '0'} 2024-04-12T03:58:49.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077b143d8cef76abe-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.038 DEBUG:boto:path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:50.038 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000042 2024-04-12T03:58:50.038 DEBUG:boto:Method: GET 2024-04-12T03:58:50.039 DEBUG:boto:Path: /dkyges-60/obj00000000000000000042 2024-04-12T03:58:50.039 DEBUG:boto:Data: 2024-04-12T03:58:50.039 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.039 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.039 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.039 DEBUG:boto:Params: {} 2024-04-12T03:58:50.039 DEBUG:boto:Token: None 2024-04-12T03:58:50.039 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000042 2024-04-12T03:58:50.039 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kH2kRzNiFM9A2dr88KL6P9A7sXM= 2024-04-12T03:58:50.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kH2kRzNiFM9A2dr88KL6P9A7sXM=', 'Content-Length': '0'} 2024-04-12T03:58:50.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7b9fe04c8c2c617-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.042 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.042 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.042 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000042 2024-04-12T03:58:50.042 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000042 2024-04-12T03:58:50.042 DEBUG:boto:Method: GET 2024-04-12T03:58:50.042 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000042 2024-04-12T03:58:50.042 DEBUG:boto:Data: 2024-04-12T03:58:50.042 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.042 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.042 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.043 DEBUG:boto:Params: {} 2024-04-12T03:58:50.043 DEBUG:boto:Token: None 2024-04-12T03:58:50.043 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.043 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c34df9505b2159f1-006618b17a-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.046 DEBUG:boto:b'dkyges-60obj000000000000000000421000falseobj000000000000000000422024-04-12T03:58:32.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.046 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.046 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.046 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000042 2024-04-12T03:58:50.046 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000042 2024-04-12T03:58:50.047 DEBUG:boto:Method: GET 2024-04-12T03:58:50.047 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000042 2024-04-12T03:58:50.047 DEBUG:boto:Data: 2024-04-12T03:58:50.047 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.047 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.047 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.047 DEBUG:boto:Params: {} 2024-04-12T03:58:50.047 DEBUG:boto:Token: None 2024-04-12T03:58:50.047 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.047 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000073b1d99d73c63ef-006618b17a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.052 DEBUG:boto:b'dkyges-60obj000000000000000000421000falseobj000000000000000000422024-04-12T03:58:32.942Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-04-12T03:58:50.052 DEBUG:boto:path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.053 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.053 DEBUG:boto:path=/dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.053 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.053 DEBUG:boto:Method: GET 2024-04-12T03:58:50.053 DEBUG:boto:Path: /dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.053 DEBUG:boto:Data: 2024-04-12T03:58:50.053 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.053 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.053 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.053 DEBUG:boto:Params: {} 2024-04-12T03:58:50.054 DEBUG:boto:Token: None 2024-04-12T03:58:50.054 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.054 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CwzU8sO3PHEzEKQBAATrWDckQn4= 2024-04-12T03:58:50.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CwzU8sO3PHEzEKQBAATrWDckQn4=', 'Content-Length': '0'} 2024-04-12T03:58:50.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1ca67c897a0f0b9-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.056 DEBUG:boto:path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.056 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.056 DEBUG:boto:path=/dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.056 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.056 DEBUG:boto:Method: GET 2024-04-12T03:58:50.057 DEBUG:boto:Path: /dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.057 DEBUG:boto:Data: 2024-04-12T03:58:50.057 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.057 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.057 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.057 DEBUG:boto:Params: {} 2024-04-12T03:58:50.057 DEBUG:boto:Token: None 2024-04-12T03:58:50.057 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.057 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CwzU8sO3PHEzEKQBAATrWDckQn4= 2024-04-12T03:58:50.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CwzU8sO3PHEzEKQBAATrWDckQn4=', 'Content-Length': '0'} 2024-04-12T03:58:50.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eba69b405faaf51-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.059 DEBUG:boto:path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.059 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.060 DEBUG:boto:path=/dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.060 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.060 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.060 DEBUG:boto:Path: /dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.060 DEBUG:boto:Data: 2024-04-12T03:58:50.060 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.060 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.060 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.060 DEBUG:boto:Params: {} 2024-04-12T03:58:50.060 DEBUG:boto:Token: None 2024-04-12T03:58:50.061 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.061 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NvEVpWaJteVuIepXU/G9RHDZhkA= 2024-04-12T03:58:50.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NvEVpWaJteVuIepXU/G9RHDZhkA=', 'Content-Length': '0'} 2024-04-12T03:58:50.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3df4e3c43995f46-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.063 DEBUG:boto:path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.063 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.063 DEBUG:boto:path=/dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.063 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.063 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.063 DEBUG:boto:Path: /dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.064 DEBUG:boto:Data: 2024-04-12T03:58:50.064 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.064 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.064 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.064 DEBUG:boto:Params: {} 2024-04-12T03:58:50.064 DEBUG:boto:Token: None 2024-04-12T03:58:50.064 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.064 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NvEVpWaJteVuIepXU/G9RHDZhkA= 2024-04-12T03:58:50.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NvEVpWaJteVuIepXU/G9RHDZhkA=', 'Content-Length': '0'} 2024-04-12T03:58:50.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a7bbe623a322f42-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-04-12T03:58:50.067 DEBUG:boto:path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.067 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.067 DEBUG:boto:path=/dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.067 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.067 DEBUG:boto:Method: GET 2024-04-12T03:58:50.067 DEBUG:boto:Path: /dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.067 DEBUG:boto:Data: 2024-04-12T03:58:50.067 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.067 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.067 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.068 DEBUG:boto:Params: {} 2024-04-12T03:58:50.068 DEBUG:boto:Token: None 2024-04-12T03:58:50.068 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.068 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CwzU8sO3PHEzEKQBAATrWDckQn4= 2024-04-12T03:58:50.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CwzU8sO3PHEzEKQBAATrWDckQn4=', 'Content-Length': '0'} 2024-04-12T03:58:50.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ff5c0803e8c2b8c-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.070 DEBUG:boto:path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.070 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.071 DEBUG:boto:path=/dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.071 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.071 DEBUG:boto:Method: GET 2024-04-12T03:58:50.071 DEBUG:boto:Path: /dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.071 DEBUG:boto:Data: 2024-04-12T03:58:50.071 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.071 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.071 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.071 DEBUG:boto:Params: {} 2024-04-12T03:58:50.071 DEBUG:boto:Token: None 2024-04-12T03:58:50.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000045?versionId=null 2024-04-12T03:58:50.072 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CwzU8sO3PHEzEKQBAATrWDckQn4= 2024-04-12T03:58:50.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CwzU8sO3PHEzEKQBAATrWDckQn4=', 'Content-Length': '0'} 2024-04-12T03:58:50.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086280697c2f3c371-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.074 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.074 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.074 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000045 2024-04-12T03:58:50.074 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000045 2024-04-12T03:58:50.074 DEBUG:boto:Method: GET 2024-04-12T03:58:50.074 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000045 2024-04-12T03:58:50.074 DEBUG:boto:Data: 2024-04-12T03:58:50.074 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.074 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.074 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.075 DEBUG:boto:Params: {} 2024-04-12T03:58:50.075 DEBUG:boto:Token: None 2024-04-12T03:58:50.075 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.075 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb6bad2146bc0436-006618b17a-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.078 DEBUG:boto:b'dkyges-60obj000000000000000000451000falseobj000000000000000000452024-04-12T03:58:35.244Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.078 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.078 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.078 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000045 2024-04-12T03:58:50.078 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000045 2024-04-12T03:58:50.078 DEBUG:boto:Method: GET 2024-04-12T03:58:50.079 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000045 2024-04-12T03:58:50.079 DEBUG:boto:Data: 2024-04-12T03:58:50.079 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.079 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.079 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.079 DEBUG:boto:Params: {} 2024-04-12T03:58:50.079 DEBUG:boto:Token: None 2024-04-12T03:58:50.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.079 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a809588f5b0b16f-006618b17a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.084 DEBUG:boto:b'dkyges-60obj000000000000000000451000falseobj000000000000000000452024-04-12T03:58:35.244Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.084 DEBUG:boto:path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.084 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.085 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.085 DEBUG:boto:Path: /dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.085 DEBUG:boto:Data: 2024-04-12T03:58:50.085 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.085 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.085 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.085 DEBUG:boto:Params: {} 2024-04-12T03:58:50.085 DEBUG:boto:Token: None 2024-04-12T03:58:50.085 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.085 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JJb/XfFXBH5tuJGxe10QDcAoVXA= 2024-04-12T03:58:50.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JJb/XfFXBH5tuJGxe10QDcAoVXA=', 'Content-Length': '0'} 2024-04-12T03:58:50.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000157d3e09ccf36495-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.087 DEBUG:boto:path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.088 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.088 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.088 DEBUG:boto:Path: /dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.088 DEBUG:boto:Data: 2024-04-12T03:58:50.088 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.088 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.088 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.088 DEBUG:boto:Params: {} 2024-04-12T03:58:50.088 DEBUG:boto:Token: None 2024-04-12T03:58:50.089 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.089 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JJb/XfFXBH5tuJGxe10QDcAoVXA= 2024-04-12T03:58:50.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JJb/XfFXBH5tuJGxe10QDcAoVXA=', 'Content-Length': '0'} 2024-04-12T03:58:50.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000454a18bef21631e1-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2024-04-12T03:58:50.091 DEBUG:boto:path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.091 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.091 DEBUG:boto:Method: GET 2024-04-12T03:58:50.091 DEBUG:boto:Path: /dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.091 DEBUG:boto:Data: 2024-04-12T03:58:50.092 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.092 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.092 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.092 DEBUG:boto:Params: {} 2024-04-12T03:58:50.092 DEBUG:boto:Token: None 2024-04-12T03:58:50.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.092 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U0qcatOZYvTJdu4nOsF8uLgYzpw= 2024-04-12T03:58:50.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U0qcatOZYvTJdu4nOsF8uLgYzpw=', 'Content-Length': '0'} 2024-04-12T03:58:50.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e0e03f95c1739b1-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.134 DEBUG:boto:path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.135 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.135 DEBUG:boto:Method: GET 2024-04-12T03:58:50.135 DEBUG:boto:Path: /dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.135 DEBUG:boto:Data: 2024-04-12T03:58:50.135 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.135 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.135 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.135 DEBUG:boto:Params: {} 2024-04-12T03:58:50.135 DEBUG:boto:Token: None 2024-04-12T03:58:50.135 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000045 2024-04-12T03:58:50.136 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U0qcatOZYvTJdu4nOsF8uLgYzpw= 2024-04-12T03:58:50.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U0qcatOZYvTJdu4nOsF8uLgYzpw=', 'Content-Length': '0'} 2024-04-12T03:58:50.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015e9968ddd90872c-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.138 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.138 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.138 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000045 2024-04-12T03:58:50.138 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000045 2024-04-12T03:58:50.138 DEBUG:boto:Method: GET 2024-04-12T03:58:50.138 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000045 2024-04-12T03:58:50.139 DEBUG:boto:Data: 2024-04-12T03:58:50.139 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.139 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.139 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.139 DEBUG:boto:Params: {} 2024-04-12T03:58:50.139 DEBUG:boto:Token: None 2024-04-12T03:58:50.139 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.139 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c77c252e30ef36e-006618b17a-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.142 DEBUG:boto:b'dkyges-60obj000000000000000000451000falseobj000000000000000000452024-04-12T03:58:35.244Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.142 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.143 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.143 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000045 2024-04-12T03:58:50.143 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000045 2024-04-12T03:58:50.143 DEBUG:boto:Method: GET 2024-04-12T03:58:50.143 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000045 2024-04-12T03:58:50.143 DEBUG:boto:Data: 2024-04-12T03:58:50.143 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.143 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.143 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.143 DEBUG:boto:Params: {} 2024-04-12T03:58:50.144 DEBUG:boto:Token: None 2024-04-12T03:58:50.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.144 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006de7ffb47ceb00a7-006618b17a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.149 DEBUG:boto:b'dkyges-60obj000000000000000000451000falseobj000000000000000000452024-04-12T03:58:35.244Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-04-12T03:58:50.149 DEBUG:boto:path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.149 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.149 DEBUG:boto:path=/dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.149 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.150 DEBUG:boto:Method: GET 2024-04-12T03:58:50.150 DEBUG:boto:Path: /dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.150 DEBUG:boto:Data: 2024-04-12T03:58:50.150 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.150 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.150 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.150 DEBUG:boto:Params: {} 2024-04-12T03:58:50.150 DEBUG:boto:Token: None 2024-04-12T03:58:50.150 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.150 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NiQfPksDTzg3FuTVIGJo09tEqSI= 2024-04-12T03:58:50.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NiQfPksDTzg3FuTVIGJo09tEqSI=', 'Content-Length': '0'} 2024-04-12T03:58:50.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000721b98b19b1eb518-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.153 DEBUG:boto:path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.153 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.153 DEBUG:boto:path=/dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.153 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.153 DEBUG:boto:Method: GET 2024-04-12T03:58:50.153 DEBUG:boto:Path: /dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.153 DEBUG:boto:Data: 2024-04-12T03:58:50.153 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.154 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.154 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.154 DEBUG:boto:Params: {} 2024-04-12T03:58:50.154 DEBUG:boto:Token: None 2024-04-12T03:58:50.154 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.154 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NiQfPksDTzg3FuTVIGJo09tEqSI= 2024-04-12T03:58:50.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NiQfPksDTzg3FuTVIGJo09tEqSI=', 'Content-Length': '0'} 2024-04-12T03:58:50.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe846ae955003b33-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.157 DEBUG:boto:path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.157 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.157 DEBUG:boto:path=/dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.157 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.157 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.157 DEBUG:boto:Path: /dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.157 DEBUG:boto:Data: 2024-04-12T03:58:50.157 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.157 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.157 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.158 DEBUG:boto:Params: {} 2024-04-12T03:58:50.158 DEBUG:boto:Token: None 2024-04-12T03:58:50.158 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.158 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XHTA6DDJ3b+y5pmWGK9F8VEJRTs= 2024-04-12T03:58:50.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XHTA6DDJ3b+y5pmWGK9F8VEJRTs=', 'Content-Length': '0'} 2024-04-12T03:58:50.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083bfdd7d43cf4fd0-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.160 DEBUG:boto:path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.160 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.160 DEBUG:boto:path=/dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.160 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.160 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.161 DEBUG:boto:Path: /dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.161 DEBUG:boto:Data: 2024-04-12T03:58:50.161 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.161 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.161 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.161 DEBUG:boto:Params: {} 2024-04-12T03:58:50.161 DEBUG:boto:Token: None 2024-04-12T03:58:50.161 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.161 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XHTA6DDJ3b+y5pmWGK9F8VEJRTs= 2024-04-12T03:58:50.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XHTA6DDJ3b+y5pmWGK9F8VEJRTs=', 'Content-Length': '0'} 2024-04-12T03:58:50.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005052cb8b1618502a-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-04-12T03:58:50.164 DEBUG:boto:path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.164 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.164 DEBUG:boto:path=/dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.164 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.164 DEBUG:boto:Method: GET 2024-04-12T03:58:50.164 DEBUG:boto:Path: /dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.164 DEBUG:boto:Data: 2024-04-12T03:58:50.164 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.164 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.165 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.165 DEBUG:boto:Params: {} 2024-04-12T03:58:50.165 DEBUG:boto:Token: None 2024-04-12T03:58:50.165 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.165 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NiQfPksDTzg3FuTVIGJo09tEqSI= 2024-04-12T03:58:50.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NiQfPksDTzg3FuTVIGJo09tEqSI=', 'Content-Length': '0'} 2024-04-12T03:58:50.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8ad27a53d5a9515-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.167 DEBUG:boto:path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.167 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.167 DEBUG:boto:path=/dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.167 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.167 DEBUG:boto:Method: GET 2024-04-12T03:58:50.168 DEBUG:boto:Path: /dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.168 DEBUG:boto:Data: 2024-04-12T03:58:50.168 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.168 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.168 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.168 DEBUG:boto:Params: {} 2024-04-12T03:58:50.168 DEBUG:boto:Token: None 2024-04-12T03:58:50.168 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000046?versionId=null 2024-04-12T03:58:50.168 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NiQfPksDTzg3FuTVIGJo09tEqSI= 2024-04-12T03:58:50.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NiQfPksDTzg3FuTVIGJo09tEqSI=', 'Content-Length': '0'} 2024-04-12T03:58:50.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe8fac189a9c617a-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.170 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.171 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.171 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000046 2024-04-12T03:58:50.171 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000046 2024-04-12T03:58:50.171 DEBUG:boto:Method: GET 2024-04-12T03:58:50.171 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000046 2024-04-12T03:58:50.171 DEBUG:boto:Data: 2024-04-12T03:58:50.171 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.171 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.171 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.171 DEBUG:boto:Params: {} 2024-04-12T03:58:50.172 DEBUG:boto:Token: None 2024-04-12T03:58:50.172 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.172 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000265348db325bcae9-006618b17a-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.174 DEBUG:boto:b'dkyges-60obj000000000000000000461000falseobj000000000000000000462024-04-12T03:58:31.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.175 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.175 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.175 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000046 2024-04-12T03:58:50.175 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000046 2024-04-12T03:58:50.175 DEBUG:boto:Method: GET 2024-04-12T03:58:50.175 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000046 2024-04-12T03:58:50.175 DEBUG:boto:Data: 2024-04-12T03:58:50.175 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.175 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.175 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.176 DEBUG:boto:Params: {} 2024-04-12T03:58:50.176 DEBUG:boto:Token: None 2024-04-12T03:58:50.176 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.176 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009675b82abb47a34e-006618b17a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.222 DEBUG:boto:b'dkyges-60obj000000000000000000461000falseobj000000000000000000462024-04-12T03:58:31.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.223 DEBUG:boto:path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.223 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.223 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.223 DEBUG:boto:Path: /dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.223 DEBUG:boto:Data: 2024-04-12T03:58:50.223 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.223 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.223 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.223 DEBUG:boto:Params: {} 2024-04-12T03:58:50.224 DEBUG:boto:Token: None 2024-04-12T03:58:50.224 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.224 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LM+U8JqeH3He/ou+HqCzhcVyNrg= 2024-04-12T03:58:50.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LM+U8JqeH3He/ou+HqCzhcVyNrg=', 'Content-Length': '0'} 2024-04-12T03:58:50.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000246c2face3abb604-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.226 DEBUG:boto:path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.226 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.226 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.226 DEBUG:boto:Path: /dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.226 DEBUG:boto:Data: 2024-04-12T03:58:50.227 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.227 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.227 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.227 DEBUG:boto:Params: {} 2024-04-12T03:58:50.227 DEBUG:boto:Token: None 2024-04-12T03:58:50.227 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.227 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LM+U8JqeH3He/ou+HqCzhcVyNrg= 2024-04-12T03:58:50.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LM+U8JqeH3He/ou+HqCzhcVyNrg=', 'Content-Length': '0'} 2024-04-12T03:58:50.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a816ddc2ca195b53-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2024-04-12T03:58:50.229 DEBUG:boto:path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.230 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.230 DEBUG:boto:Method: GET 2024-04-12T03:58:50.230 DEBUG:boto:Path: /dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.230 DEBUG:boto:Data: 2024-04-12T03:58:50.230 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.230 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.230 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.230 DEBUG:boto:Params: {} 2024-04-12T03:58:50.230 DEBUG:boto:Token: None 2024-04-12T03:58:50.230 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.231 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JnNt/MHSFTBX3Ik1DP3ZOO3d5h8= 2024-04-12T03:58:50.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JnNt/MHSFTBX3Ik1DP3ZOO3d5h8=', 'Content-Length': '0'} 2024-04-12T03:58:50.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a12648e22c5f69d0-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.233 DEBUG:boto:path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.233 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.233 DEBUG:boto:Method: GET 2024-04-12T03:58:50.233 DEBUG:boto:Path: /dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.233 DEBUG:boto:Data: 2024-04-12T03:58:50.233 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.233 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.233 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.233 DEBUG:boto:Params: {} 2024-04-12T03:58:50.234 DEBUG:boto:Token: None 2024-04-12T03:58:50.234 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000046 2024-04-12T03:58:50.234 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JnNt/MHSFTBX3Ik1DP3ZOO3d5h8= 2024-04-12T03:58:50.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JnNt/MHSFTBX3Ik1DP3ZOO3d5h8=', 'Content-Length': '0'} 2024-04-12T03:58:50.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000338ff26510a67861-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.236 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.236 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.236 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000046 2024-04-12T03:58:50.236 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000046 2024-04-12T03:58:50.237 DEBUG:boto:Method: GET 2024-04-12T03:58:50.237 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000046 2024-04-12T03:58:50.237 DEBUG:boto:Data: 2024-04-12T03:58:50.237 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.237 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.237 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.237 DEBUG:boto:Params: {} 2024-04-12T03:58:50.237 DEBUG:boto:Token: None 2024-04-12T03:58:50.237 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.237 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e591e47404172325-006618b17a-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.240 DEBUG:boto:b'dkyges-60obj000000000000000000461000falseobj000000000000000000462024-04-12T03:58:31.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.241 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.241 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.241 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000046 2024-04-12T03:58:50.241 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000046 2024-04-12T03:58:50.241 DEBUG:boto:Method: GET 2024-04-12T03:58:50.241 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000046 2024-04-12T03:58:50.241 DEBUG:boto:Data: 2024-04-12T03:58:50.241 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.241 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.241 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.242 DEBUG:boto:Params: {} 2024-04-12T03:58:50.242 DEBUG:boto:Token: None 2024-04-12T03:58:50.242 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.242 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b5fa9c41d886718-006618b17a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.247 DEBUG:boto:b'dkyges-60obj000000000000000000461000falseobj000000000000000000462024-04-12T03:58:31.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-04-12T03:58:50.247 DEBUG:boto:path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.247 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.247 DEBUG:boto:path=/dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.247 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.247 DEBUG:boto:Method: GET 2024-04-12T03:58:50.247 DEBUG:boto:Path: /dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.248 DEBUG:boto:Data: 2024-04-12T03:58:50.248 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.248 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.248 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.248 DEBUG:boto:Params: {} 2024-04-12T03:58:50.248 DEBUG:boto:Token: None 2024-04-12T03:58:50.248 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.248 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CF+dcIZNpaJwKslgP/a+KkpZA7g= 2024-04-12T03:58:50.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CF+dcIZNpaJwKslgP/a+KkpZA7g=', 'Content-Length': '0'} 2024-04-12T03:58:50.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a22a24f2711a3e1-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.294 DEBUG:boto:path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.295 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.295 DEBUG:boto:path=/dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.295 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.295 DEBUG:boto:Method: GET 2024-04-12T03:58:50.295 DEBUG:boto:Path: /dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.295 DEBUG:boto:Data: 2024-04-12T03:58:50.295 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.295 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.295 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.295 DEBUG:boto:Params: {} 2024-04-12T03:58:50.296 DEBUG:boto:Token: None 2024-04-12T03:58:50.296 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.296 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CF+dcIZNpaJwKslgP/a+KkpZA7g= 2024-04-12T03:58:50.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CF+dcIZNpaJwKslgP/a+KkpZA7g=', 'Content-Length': '0'} 2024-04-12T03:58:50.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa77660dffc07d18-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.298 DEBUG:boto:path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.299 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.299 DEBUG:boto:path=/dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.299 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.299 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.299 DEBUG:boto:Path: /dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.299 DEBUG:boto:Data: 2024-04-12T03:58:50.299 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.299 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.299 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.299 DEBUG:boto:Params: {} 2024-04-12T03:58:50.299 DEBUG:boto:Token: None 2024-04-12T03:58:50.300 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.300 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PWVGB9+c87y2qLBpAL5tBGWDpFc= 2024-04-12T03:58:50.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PWVGB9+c87y2qLBpAL5tBGWDpFc=', 'Content-Length': '0'} 2024-04-12T03:58:50.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bcb38f2a6e33dd0-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.302 DEBUG:boto:path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.302 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.302 DEBUG:boto:path=/dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.302 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.302 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.302 DEBUG:boto:Path: /dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.302 DEBUG:boto:Data: 2024-04-12T03:58:50.302 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.302 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.303 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.303 DEBUG:boto:Params: {} 2024-04-12T03:58:50.303 DEBUG:boto:Token: None 2024-04-12T03:58:50.303 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.303 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PWVGB9+c87y2qLBpAL5tBGWDpFc= 2024-04-12T03:58:50.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PWVGB9+c87y2qLBpAL5tBGWDpFc=', 'Content-Length': '0'} 2024-04-12T03:58:50.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9e77af970daa89f-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-04-12T03:58:50.305 DEBUG:boto:path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.306 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.306 DEBUG:boto:path=/dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.306 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.306 DEBUG:boto:Method: GET 2024-04-12T03:58:50.306 DEBUG:boto:Path: /dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.306 DEBUG:boto:Data: 2024-04-12T03:58:50.306 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.306 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.306 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.306 DEBUG:boto:Params: {} 2024-04-12T03:58:50.306 DEBUG:boto:Token: None 2024-04-12T03:58:50.307 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.307 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CF+dcIZNpaJwKslgP/a+KkpZA7g= 2024-04-12T03:58:50.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CF+dcIZNpaJwKslgP/a+KkpZA7g=', 'Content-Length': '0'} 2024-04-12T03:58:50.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008f1a01b3ac663a0-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.309 DEBUG:boto:path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.309 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.309 DEBUG:boto:path=/dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.309 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.309 DEBUG:boto:Method: GET 2024-04-12T03:58:50.309 DEBUG:boto:Path: /dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.309 DEBUG:boto:Data: 2024-04-12T03:58:50.310 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.310 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.310 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.310 DEBUG:boto:Params: {} 2024-04-12T03:58:50.310 DEBUG:boto:Token: None 2024-04-12T03:58:50.310 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000048?versionId=null 2024-04-12T03:58:50.310 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CF+dcIZNpaJwKslgP/a+KkpZA7g= 2024-04-12T03:58:50.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CF+dcIZNpaJwKslgP/a+KkpZA7g=', 'Content-Length': '0'} 2024-04-12T03:58:50.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdbaae827f469cbc-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.313 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.314 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.314 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000048 2024-04-12T03:58:50.314 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000048 2024-04-12T03:58:50.314 DEBUG:boto:Method: GET 2024-04-12T03:58:50.314 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000048 2024-04-12T03:58:50.314 DEBUG:boto:Data: 2024-04-12T03:58:50.314 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.314 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.314 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.314 DEBUG:boto:Params: {} 2024-04-12T03:58:50.314 DEBUG:boto:Token: None 2024-04-12T03:58:50.315 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.315 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.318 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056d77f08e04db346-006618b17a-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.318 DEBUG:boto:b'dkyges-60obj000000000000000000481000falseobj000000000000000000482024-04-12T03:58:35.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.318 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.319 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.319 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000048 2024-04-12T03:58:50.319 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000048 2024-04-12T03:58:50.319 DEBUG:boto:Method: GET 2024-04-12T03:58:50.319 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000048 2024-04-12T03:58:50.319 DEBUG:boto:Data: 2024-04-12T03:58:50.319 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.319 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.319 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.319 DEBUG:boto:Params: {} 2024-04-12T03:58:50.320 DEBUG:boto:Token: None 2024-04-12T03:58:50.320 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.320 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fe5be64e8d76c9f-006618b17a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.324 DEBUG:boto:b'dkyges-60obj000000000000000000481000falseobj000000000000000000482024-04-12T03:58:35.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.324 DEBUG:boto:path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.325 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.325 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.325 DEBUG:boto:Path: /dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.325 DEBUG:boto:Data: 2024-04-12T03:58:50.325 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.325 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.325 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.325 DEBUG:boto:Params: {} 2024-04-12T03:58:50.325 DEBUG:boto:Token: None 2024-04-12T03:58:50.326 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.326 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hIKKR6cCIUakCGj/d85lgaF158I= 2024-04-12T03:58:50.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hIKKR6cCIUakCGj/d85lgaF158I=', 'Content-Length': '0'} 2024-04-12T03:58:50.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014a2a94c33a76c6e-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.328 DEBUG:boto:path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.328 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.328 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.328 DEBUG:boto:Path: /dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.328 DEBUG:boto:Data: 2024-04-12T03:58:50.328 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.328 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.328 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.329 DEBUG:boto:Params: {} 2024-04-12T03:58:50.329 DEBUG:boto:Token: None 2024-04-12T03:58:50.329 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.329 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hIKKR6cCIUakCGj/d85lgaF158I= 2024-04-12T03:58:50.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hIKKR6cCIUakCGj/d85lgaF158I=', 'Content-Length': '0'} 2024-04-12T03:58:50.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055e6dd0577d712cc-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2024-04-12T03:58:50.331 DEBUG:boto:path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.331 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.331 DEBUG:boto:Method: GET 2024-04-12T03:58:50.331 DEBUG:boto:Path: /dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.332 DEBUG:boto:Data: 2024-04-12T03:58:50.332 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.332 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.332 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.332 DEBUG:boto:Params: {} 2024-04-12T03:58:50.332 DEBUG:boto:Token: None 2024-04-12T03:58:50.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.332 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXth+hxpsfynRxHKoYfU/uHGiZA= 2024-04-12T03:58:50.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXth+hxpsfynRxHKoYfU/uHGiZA=', 'Content-Length': '0'} 2024-04-12T03:58:50.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eff70116c301d79-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.334 DEBUG:boto:path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.334 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.335 DEBUG:boto:Method: GET 2024-04-12T03:58:50.335 DEBUG:boto:Path: /dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.335 DEBUG:boto:Data: 2024-04-12T03:58:50.335 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.335 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.335 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.335 DEBUG:boto:Params: {} 2024-04-12T03:58:50.335 DEBUG:boto:Token: None 2024-04-12T03:58:50.335 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000048 2024-04-12T03:58:50.335 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXth+hxpsfynRxHKoYfU/uHGiZA= 2024-04-12T03:58:50.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXth+hxpsfynRxHKoYfU/uHGiZA=', 'Content-Length': '0'} 2024-04-12T03:58:50.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005aad84f381326c7e-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.378 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.379 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.379 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000048 2024-04-12T03:58:50.379 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000048 2024-04-12T03:58:50.379 DEBUG:boto:Method: GET 2024-04-12T03:58:50.379 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000048 2024-04-12T03:58:50.379 DEBUG:boto:Data: 2024-04-12T03:58:50.379 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.379 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.379 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.379 DEBUG:boto:Params: {} 2024-04-12T03:58:50.379 DEBUG:boto:Token: None 2024-04-12T03:58:50.380 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.380 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9934f6286372583-006618b17a-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.382 DEBUG:boto:b'dkyges-60obj000000000000000000481000falseobj000000000000000000482024-04-12T03:58:35.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.383 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.383 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.383 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000048 2024-04-12T03:58:50.383 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000048 2024-04-12T03:58:50.383 DEBUG:boto:Method: GET 2024-04-12T03:58:50.383 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000048 2024-04-12T03:58:50.383 DEBUG:boto:Data: 2024-04-12T03:58:50.383 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.383 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.384 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.384 DEBUG:boto:Params: {} 2024-04-12T03:58:50.384 DEBUG:boto:Token: None 2024-04-12T03:58:50.384 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.384 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000015f07df067fa684-006618b17a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.388 DEBUG:boto:b'dkyges-60obj000000000000000000481000falseobj000000000000000000482024-04-12T03:58:35.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-04-12T03:58:50.389 DEBUG:boto:path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.389 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.389 DEBUG:boto:path=/dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.389 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.389 DEBUG:boto:Method: GET 2024-04-12T03:58:50.389 DEBUG:boto:Path: /dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.390 DEBUG:boto:Data: 2024-04-12T03:58:50.390 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.390 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.390 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.390 DEBUG:boto:Params: {} 2024-04-12T03:58:50.390 DEBUG:boto:Token: None 2024-04-12T03:58:50.390 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.390 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2//+hOTlVTlHFXnPO0JyeCUE1o8= 2024-04-12T03:58:50.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2//+hOTlVTlHFXnPO0JyeCUE1o8=', 'Content-Length': '0'} 2024-04-12T03:58:50.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f36db519817c3f7-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.393 DEBUG:boto:path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.393 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.393 DEBUG:boto:path=/dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.393 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.393 DEBUG:boto:Method: GET 2024-04-12T03:58:50.393 DEBUG:boto:Path: /dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.393 DEBUG:boto:Data: 2024-04-12T03:58:50.393 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.393 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.393 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.393 DEBUG:boto:Params: {} 2024-04-12T03:58:50.394 DEBUG:boto:Token: None 2024-04-12T03:58:50.394 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.394 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2//+hOTlVTlHFXnPO0JyeCUE1o8= 2024-04-12T03:58:50.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2//+hOTlVTlHFXnPO0JyeCUE1o8=', 'Content-Length': '0'} 2024-04-12T03:58:50.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a30d95cb1596e21e-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.396 DEBUG:boto:path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.396 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.396 DEBUG:boto:path=/dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.396 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.397 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.397 DEBUG:boto:Path: /dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.397 DEBUG:boto:Data: 2024-04-12T03:58:50.397 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.397 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.397 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.397 DEBUG:boto:Params: {} 2024-04-12T03:58:50.397 DEBUG:boto:Token: None 2024-04-12T03:58:50.397 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.397 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+4SDIM1Rww84GdqMw/e1FETZQBg= 2024-04-12T03:58:50.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+4SDIM1Rww84GdqMw/e1FETZQBg=', 'Content-Length': '0'} 2024-04-12T03:58:50.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cef6154bc00b2f19-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.400 DEBUG:boto:path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.400 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.400 DEBUG:boto:path=/dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.400 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.400 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.400 DEBUG:boto:Path: /dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.400 DEBUG:boto:Data: 2024-04-12T03:58:50.400 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.400 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.400 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.400 DEBUG:boto:Params: {} 2024-04-12T03:58:50.401 DEBUG:boto:Token: None 2024-04-12T03:58:50.401 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.401 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+4SDIM1Rww84GdqMw/e1FETZQBg= 2024-04-12T03:58:50.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+4SDIM1Rww84GdqMw/e1FETZQBg=', 'Content-Length': '0'} 2024-04-12T03:58:50.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b05b4f59b47ac3-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-04-12T03:58:50.403 DEBUG:boto:path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.403 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.403 DEBUG:boto:path=/dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.404 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.404 DEBUG:boto:Method: GET 2024-04-12T03:58:50.404 DEBUG:boto:Path: /dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.404 DEBUG:boto:Data: 2024-04-12T03:58:50.404 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.404 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.404 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.404 DEBUG:boto:Params: {} 2024-04-12T03:58:50.404 DEBUG:boto:Token: None 2024-04-12T03:58:50.404 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.405 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2//+hOTlVTlHFXnPO0JyeCUE1o8= 2024-04-12T03:58:50.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2//+hOTlVTlHFXnPO0JyeCUE1o8=', 'Content-Length': '0'} 2024-04-12T03:58:50.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006781266981fec968-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.450 DEBUG:boto:path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.450 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.451 DEBUG:boto:path=/dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.451 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.451 DEBUG:boto:Method: GET 2024-04-12T03:58:50.451 DEBUG:boto:Path: /dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.451 DEBUG:boto:Data: 2024-04-12T03:58:50.451 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.451 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.451 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.451 DEBUG:boto:Params: {} 2024-04-12T03:58:50.452 DEBUG:boto:Token: None 2024-04-12T03:58:50.452 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000050?versionId=null 2024-04-12T03:58:50.452 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2//+hOTlVTlHFXnPO0JyeCUE1o8= 2024-04-12T03:58:50.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2//+hOTlVTlHFXnPO0JyeCUE1o8=', 'Content-Length': '0'} 2024-04-12T03:58:50.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000849177394210f4ff-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.454 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.454 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.454 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000050 2024-04-12T03:58:50.454 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000050 2024-04-12T03:58:50.455 DEBUG:boto:Method: GET 2024-04-12T03:58:50.455 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000050 2024-04-12T03:58:50.455 DEBUG:boto:Data: 2024-04-12T03:58:50.455 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.455 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.455 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.455 DEBUG:boto:Params: {} 2024-04-12T03:58:50.455 DEBUG:boto:Token: None 2024-04-12T03:58:50.455 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.455 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.458 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a495a78027fdae3-006618b17a-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.458 DEBUG:boto:b'dkyges-60obj000000000000000000501000falseobj000000000000000000502024-04-12T03:58:33.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.458 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.458 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.459 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000050 2024-04-12T03:58:50.459 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000050 2024-04-12T03:58:50.459 DEBUG:boto:Method: GET 2024-04-12T03:58:50.459 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000050 2024-04-12T03:58:50.459 DEBUG:boto:Data: 2024-04-12T03:58:50.459 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.459 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.459 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.459 DEBUG:boto:Params: {} 2024-04-12T03:58:50.459 DEBUG:boto:Token: None 2024-04-12T03:58:50.460 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.460 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006466b4a09420071f-006618b17a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.464 DEBUG:boto:b'dkyges-60obj000000000000000000501000falseobj000000000000000000502024-04-12T03:58:33.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.464 DEBUG:boto:path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.465 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.465 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.465 DEBUG:boto:Path: /dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.465 DEBUG:boto:Data: 2024-04-12T03:58:50.465 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.465 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.465 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.465 DEBUG:boto:Params: {} 2024-04-12T03:58:50.465 DEBUG:boto:Token: None 2024-04-12T03:58:50.465 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.466 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hOSoWwxMCH37dkLXY5ph5R1roHo= 2024-04-12T03:58:50.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hOSoWwxMCH37dkLXY5ph5R1roHo=', 'Content-Length': '0'} 2024-04-12T03:58:50.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb3260d80421e75a-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.467 DEBUG:boto:path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.468 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.468 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.468 DEBUG:boto:Path: /dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.468 DEBUG:boto:Data: 2024-04-12T03:58:50.468 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.468 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.468 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.468 DEBUG:boto:Params: {} 2024-04-12T03:58:50.468 DEBUG:boto:Token: None 2024-04-12T03:58:50.469 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.469 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hOSoWwxMCH37dkLXY5ph5R1roHo= 2024-04-12T03:58:50.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hOSoWwxMCH37dkLXY5ph5R1roHo=', 'Content-Length': '0'} 2024-04-12T03:58:50.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e189257ddfd7c840-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2024-04-12T03:58:50.471 DEBUG:boto:path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.471 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.471 DEBUG:boto:Method: GET 2024-04-12T03:58:50.471 DEBUG:boto:Path: /dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.471 DEBUG:boto:Data: 2024-04-12T03:58:50.471 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.471 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.472 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.472 DEBUG:boto:Params: {} 2024-04-12T03:58:50.472 DEBUG:boto:Token: None 2024-04-12T03:58:50.472 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.472 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iZXQ735cg7KVKQphpslFDYou8wI= 2024-04-12T03:58:50.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iZXQ735cg7KVKQphpslFDYou8wI=', 'Content-Length': '0'} 2024-04-12T03:58:50.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bd6c13230f2775e-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.474 DEBUG:boto:path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.474 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.475 DEBUG:boto:Method: GET 2024-04-12T03:58:50.475 DEBUG:boto:Path: /dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.475 DEBUG:boto:Data: 2024-04-12T03:58:50.475 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.475 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.475 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.475 DEBUG:boto:Params: {} 2024-04-12T03:58:50.475 DEBUG:boto:Token: None 2024-04-12T03:58:50.475 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000050 2024-04-12T03:58:50.476 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iZXQ735cg7KVKQphpslFDYou8wI= 2024-04-12T03:58:50.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iZXQ735cg7KVKQphpslFDYou8wI=', 'Content-Length': '0'} 2024-04-12T03:58:50.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e3da16b676c64e5-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.518 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.518 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.519 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000050 2024-04-12T03:58:50.519 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000050 2024-04-12T03:58:50.519 DEBUG:boto:Method: GET 2024-04-12T03:58:50.519 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000050 2024-04-12T03:58:50.519 DEBUG:boto:Data: 2024-04-12T03:58:50.519 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.519 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.519 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.519 DEBUG:boto:Params: {} 2024-04-12T03:58:50.519 DEBUG:boto:Token: None 2024-04-12T03:58:50.520 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.520 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcfecb0d43920d93-006618b17a-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.523 DEBUG:boto:b'dkyges-60obj000000000000000000501000falseobj000000000000000000502024-04-12T03:58:33.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.523 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.523 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.523 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000050 2024-04-12T03:58:50.524 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000050 2024-04-12T03:58:50.524 DEBUG:boto:Method: GET 2024-04-12T03:58:50.524 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000050 2024-04-12T03:58:50.524 DEBUG:boto:Data: 2024-04-12T03:58:50.524 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.524 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.524 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.524 DEBUG:boto:Params: {} 2024-04-12T03:58:50.524 DEBUG:boto:Token: None 2024-04-12T03:58:50.524 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.525 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5ec0fbb824cda34-006618b17a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.529 DEBUG:boto:b'dkyges-60obj000000000000000000501000falseobj000000000000000000502024-04-12T03:58:33.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-04-12T03:58:50.529 DEBUG:boto:path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.529 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.529 DEBUG:boto:path=/dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.529 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.530 DEBUG:boto:Method: GET 2024-04-12T03:58:50.530 DEBUG:boto:Path: /dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.530 DEBUG:boto:Data: 2024-04-12T03:58:50.530 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.530 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.530 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.530 DEBUG:boto:Params: {} 2024-04-12T03:58:50.530 DEBUG:boto:Token: None 2024-04-12T03:58:50.530 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.531 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jidiP95MnW1dtNzY9Xghy+rReFs= 2024-04-12T03:58:50.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jidiP95MnW1dtNzY9Xghy+rReFs=', 'Content-Length': '0'} 2024-04-12T03:58:50.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033b44e0db6107261-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.574 DEBUG:boto:path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.575 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.575 DEBUG:boto:path=/dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.575 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.575 DEBUG:boto:Method: GET 2024-04-12T03:58:50.575 DEBUG:boto:Path: /dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.575 DEBUG:boto:Data: 2024-04-12T03:58:50.575 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.575 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.575 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.575 DEBUG:boto:Params: {} 2024-04-12T03:58:50.575 DEBUG:boto:Token: None 2024-04-12T03:58:50.576 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.576 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jidiP95MnW1dtNzY9Xghy+rReFs= 2024-04-12T03:58:50.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jidiP95MnW1dtNzY9Xghy+rReFs=', 'Content-Length': '0'} 2024-04-12T03:58:50.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055fc5e3aff2cccd9-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.578 DEBUG:boto:path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.578 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.578 DEBUG:boto:path=/dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.578 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.579 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.579 DEBUG:boto:Path: /dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.579 DEBUG:boto:Data: 2024-04-12T03:58:50.579 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.579 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.579 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.579 DEBUG:boto:Params: {} 2024-04-12T03:58:50.579 DEBUG:boto:Token: None 2024-04-12T03:58:50.579 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.579 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8QH6c+6tVmZHyqygOkeUK6COlOQ= 2024-04-12T03:58:50.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8QH6c+6tVmZHyqygOkeUK6COlOQ=', 'Content-Length': '0'} 2024-04-12T03:58:50.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000289246d314871ee7-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.581 DEBUG:boto:path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.582 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.582 DEBUG:boto:path=/dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.582 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.582 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.582 DEBUG:boto:Path: /dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.582 DEBUG:boto:Data: 2024-04-12T03:58:50.582 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.582 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.582 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.582 DEBUG:boto:Params: {} 2024-04-12T03:58:50.582 DEBUG:boto:Token: None 2024-04-12T03:58:50.583 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.583 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8QH6c+6tVmZHyqygOkeUK6COlOQ= 2024-04-12T03:58:50.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8QH6c+6tVmZHyqygOkeUK6COlOQ=', 'Content-Length': '0'} 2024-04-12T03:58:50.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba653c4a2f20596b-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-04-12T03:58:50.585 DEBUG:boto:path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.585 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.585 DEBUG:boto:path=/dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.585 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.585 DEBUG:boto:Method: GET 2024-04-12T03:58:50.585 DEBUG:boto:Path: /dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.585 DEBUG:boto:Data: 2024-04-12T03:58:50.586 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.586 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.586 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.586 DEBUG:boto:Params: {} 2024-04-12T03:58:50.586 DEBUG:boto:Token: None 2024-04-12T03:58:50.586 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.586 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jidiP95MnW1dtNzY9Xghy+rReFs= 2024-04-12T03:58:50.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jidiP95MnW1dtNzY9Xghy+rReFs=', 'Content-Length': '0'} 2024-04-12T03:58:50.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000239f0f7a30e68f19-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.588 DEBUG:boto:path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.588 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.588 DEBUG:boto:path=/dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.589 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.589 DEBUG:boto:Method: GET 2024-04-12T03:58:50.589 DEBUG:boto:Path: /dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.589 DEBUG:boto:Data: 2024-04-12T03:58:50.589 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.589 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.589 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.589 DEBUG:boto:Params: {} 2024-04-12T03:58:50.589 DEBUG:boto:Token: None 2024-04-12T03:58:50.589 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000053?versionId=null 2024-04-12T03:58:50.590 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jidiP95MnW1dtNzY9Xghy+rReFs= 2024-04-12T03:58:50.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jidiP95MnW1dtNzY9Xghy+rReFs=', 'Content-Length': '0'} 2024-04-12T03:58:50.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdfba5463054b089-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.592 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.592 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.592 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000053 2024-04-12T03:58:50.592 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000053 2024-04-12T03:58:50.592 DEBUG:boto:Method: GET 2024-04-12T03:58:50.592 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000053 2024-04-12T03:58:50.592 DEBUG:boto:Data: 2024-04-12T03:58:50.592 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.593 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.593 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.593 DEBUG:boto:Params: {} 2024-04-12T03:58:50.593 DEBUG:boto:Token: None 2024-04-12T03:58:50.593 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.593 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082717adf46aff2cd-006618b17a-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.596 DEBUG:boto:b'dkyges-60obj000000000000000000531000falseobj000000000000000000532024-04-12T03:58:35.849Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.596 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.596 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.596 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000053 2024-04-12T03:58:50.596 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000053 2024-04-12T03:58:50.596 DEBUG:boto:Method: GET 2024-04-12T03:58:50.596 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000053 2024-04-12T03:58:50.596 DEBUG:boto:Data: 2024-04-12T03:58:50.597 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.597 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.597 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.597 DEBUG:boto:Params: {} 2024-04-12T03:58:50.597 DEBUG:boto:Token: None 2024-04-12T03:58:50.597 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.597 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc9bb09f70661f61-006618b17a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.602 DEBUG:boto:b'dkyges-60obj000000000000000000531000falseobj000000000000000000532024-04-12T03:58:35.849Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.602 DEBUG:boto:path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.602 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.602 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.602 DEBUG:boto:Path: /dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.602 DEBUG:boto:Data: 2024-04-12T03:58:50.602 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.602 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.603 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.603 DEBUG:boto:Params: {} 2024-04-12T03:58:50.603 DEBUG:boto:Token: None 2024-04-12T03:58:50.603 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.603 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jSslYfIF6Hfa2dAkUzUEofxDKgw= 2024-04-12T03:58:50.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jSslYfIF6Hfa2dAkUzUEofxDKgw=', 'Content-Length': '0'} 2024-04-12T03:58:50.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ec4df87834c4b0f-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.605 DEBUG:boto:path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.605 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.605 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.605 DEBUG:boto:Path: /dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.605 DEBUG:boto:Data: 2024-04-12T03:58:50.606 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.606 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.606 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.606 DEBUG:boto:Params: {} 2024-04-12T03:58:50.606 DEBUG:boto:Token: None 2024-04-12T03:58:50.606 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.606 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jSslYfIF6Hfa2dAkUzUEofxDKgw= 2024-04-12T03:58:50.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jSslYfIF6Hfa2dAkUzUEofxDKgw=', 'Content-Length': '0'} 2024-04-12T03:58:50.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a857d09899230f72-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2024-04-12T03:58:50.608 DEBUG:boto:path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.608 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.609 DEBUG:boto:Method: GET 2024-04-12T03:58:50.609 DEBUG:boto:Path: /dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.609 DEBUG:boto:Data: 2024-04-12T03:58:50.609 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.609 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.609 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.609 DEBUG:boto:Params: {} 2024-04-12T03:58:50.609 DEBUG:boto:Token: None 2024-04-12T03:58:50.609 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.610 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:muKuSj92YOsozEEQ9AOqILhwzBo= 2024-04-12T03:58:50.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:muKuSj92YOsozEEQ9AOqILhwzBo=', 'Content-Length': '0'} 2024-04-12T03:58:50.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000127b727e0345e707-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.612 DEBUG:boto:path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.612 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.612 DEBUG:boto:Method: GET 2024-04-12T03:58:50.612 DEBUG:boto:Path: /dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.612 DEBUG:boto:Data: 2024-04-12T03:58:50.612 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.612 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.612 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.613 DEBUG:boto:Params: {} 2024-04-12T03:58:50.613 DEBUG:boto:Token: None 2024-04-12T03:58:50.613 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000053 2024-04-12T03:58:50.613 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:muKuSj92YOsozEEQ9AOqILhwzBo= 2024-04-12T03:58:50.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:muKuSj92YOsozEEQ9AOqILhwzBo=', 'Content-Length': '0'} 2024-04-12T03:58:50.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059bc51484e08fb19-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.658 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.659 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.659 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000053 2024-04-12T03:58:50.659 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000053 2024-04-12T03:58:50.659 DEBUG:boto:Method: GET 2024-04-12T03:58:50.659 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000053 2024-04-12T03:58:50.659 DEBUG:boto:Data: 2024-04-12T03:58:50.659 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.659 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.659 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.659 DEBUG:boto:Params: {} 2024-04-12T03:58:50.659 DEBUG:boto:Token: None 2024-04-12T03:58:50.660 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.660 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a97a7a882a1dbe57-006618b17a-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.662 DEBUG:boto:b'dkyges-60obj000000000000000000531000falseobj000000000000000000532024-04-12T03:58:35.849Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.663 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.663 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.663 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000053 2024-04-12T03:58:50.663 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000053 2024-04-12T03:58:50.663 DEBUG:boto:Method: GET 2024-04-12T03:58:50.663 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000053 2024-04-12T03:58:50.663 DEBUG:boto:Data: 2024-04-12T03:58:50.664 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.664 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.664 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.664 DEBUG:boto:Params: {} 2024-04-12T03:58:50.664 DEBUG:boto:Token: None 2024-04-12T03:58:50.664 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.664 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061d373f4fb779068-006618b17a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.669 DEBUG:boto:b'dkyges-60obj000000000000000000531000falseobj000000000000000000532024-04-12T03:58:35.849Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-04-12T03:58:50.670 DEBUG:boto:path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.670 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.670 DEBUG:boto:path=/dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.670 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.670 DEBUG:boto:Method: GET 2024-04-12T03:58:50.670 DEBUG:boto:Path: /dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.670 DEBUG:boto:Data: 2024-04-12T03:58:50.670 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.670 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.670 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.671 DEBUG:boto:Params: {} 2024-04-12T03:58:50.671 DEBUG:boto:Token: None 2024-04-12T03:58:50.671 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.671 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zYZ0sjZyUpgjBnsY9XEmDrmcopA= 2024-04-12T03:58:50.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zYZ0sjZyUpgjBnsY9XEmDrmcopA=', 'Content-Length': '0'} 2024-04-12T03:58:50.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029dcde6c1faa3d74-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.714 DEBUG:boto:path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.715 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.715 DEBUG:boto:path=/dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.715 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.715 DEBUG:boto:Method: GET 2024-04-12T03:58:50.715 DEBUG:boto:Path: /dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.715 DEBUG:boto:Data: 2024-04-12T03:58:50.715 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.715 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.715 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.715 DEBUG:boto:Params: {} 2024-04-12T03:58:50.716 DEBUG:boto:Token: None 2024-04-12T03:58:50.716 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.716 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zYZ0sjZyUpgjBnsY9XEmDrmcopA= 2024-04-12T03:58:50.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zYZ0sjZyUpgjBnsY9XEmDrmcopA=', 'Content-Length': '0'} 2024-04-12T03:58:50.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db565ad71775fa44-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.719 DEBUG:boto:path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.719 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.719 DEBUG:boto:path=/dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.719 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.719 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.719 DEBUG:boto:Path: /dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.719 DEBUG:boto:Data: 2024-04-12T03:58:50.719 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.720 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.720 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.720 DEBUG:boto:Params: {} 2024-04-12T03:58:50.720 DEBUG:boto:Token: None 2024-04-12T03:58:50.720 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.720 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AvMMXaGDOhocxaWZ3HgZui2WO44= 2024-04-12T03:58:50.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AvMMXaGDOhocxaWZ3HgZui2WO44=', 'Content-Length': '0'} 2024-04-12T03:58:50.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ab9bd64659ce244-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.722 DEBUG:boto:path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.722 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.722 DEBUG:boto:path=/dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.722 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.723 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.723 DEBUG:boto:Path: /dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.723 DEBUG:boto:Data: 2024-04-12T03:58:50.723 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.723 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.723 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.723 DEBUG:boto:Params: {} 2024-04-12T03:58:50.723 DEBUG:boto:Token: None 2024-04-12T03:58:50.723 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.725 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AvMMXaGDOhocxaWZ3HgZui2WO44= 2024-04-12T03:58:50.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AvMMXaGDOhocxaWZ3HgZui2WO44=', 'Content-Length': '0'} 2024-04-12T03:58:50.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e33d88102c6d3fd9-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-04-12T03:58:50.727 DEBUG:boto:path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.727 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.727 DEBUG:boto:path=/dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.727 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.728 DEBUG:boto:Method: GET 2024-04-12T03:58:50.728 DEBUG:boto:Path: /dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.728 DEBUG:boto:Data: 2024-04-12T03:58:50.728 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.728 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.728 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.728 DEBUG:boto:Params: {} 2024-04-12T03:58:50.728 DEBUG:boto:Token: None 2024-04-12T03:58:50.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.729 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zYZ0sjZyUpgjBnsY9XEmDrmcopA= 2024-04-12T03:58:50.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zYZ0sjZyUpgjBnsY9XEmDrmcopA=', 'Content-Length': '0'} 2024-04-12T03:58:50.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d1c3bcddcc4130e-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.731 DEBUG:boto:path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.731 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.731 DEBUG:boto:path=/dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.731 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.731 DEBUG:boto:Method: GET 2024-04-12T03:58:50.731 DEBUG:boto:Path: /dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.731 DEBUG:boto:Data: 2024-04-12T03:58:50.731 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.732 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.732 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.732 DEBUG:boto:Params: {} 2024-04-12T03:58:50.732 DEBUG:boto:Token: None 2024-04-12T03:58:50.732 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000055?versionId=null 2024-04-12T03:58:50.732 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zYZ0sjZyUpgjBnsY9XEmDrmcopA= 2024-04-12T03:58:50.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zYZ0sjZyUpgjBnsY9XEmDrmcopA=', 'Content-Length': '0'} 2024-04-12T03:58:50.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8382a6cddf58931-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.774 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.774 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.775 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000055 2024-04-12T03:58:50.775 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000055 2024-04-12T03:58:50.775 DEBUG:boto:Method: GET 2024-04-12T03:58:50.775 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000055 2024-04-12T03:58:50.775 DEBUG:boto:Data: 2024-04-12T03:58:50.775 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.775 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.775 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.775 DEBUG:boto:Params: {} 2024-04-12T03:58:50.775 DEBUG:boto:Token: None 2024-04-12T03:58:50.776 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.776 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.779 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000777b7e6bbb659205-006618b17a-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.779 DEBUG:boto:b'dkyges-60obj000000000000000000551000falseobj000000000000000000552024-04-12T03:58:32.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.779 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.779 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.780 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000055 2024-04-12T03:58:50.780 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000055 2024-04-12T03:58:50.780 DEBUG:boto:Method: GET 2024-04-12T03:58:50.780 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000055 2024-04-12T03:58:50.780 DEBUG:boto:Data: 2024-04-12T03:58:50.780 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.780 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.780 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.780 DEBUG:boto:Params: {} 2024-04-12T03:58:50.780 DEBUG:boto:Token: None 2024-04-12T03:58:50.781 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.781 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9e5a2039e4b5af8-006618b17a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.785 DEBUG:boto:b'dkyges-60obj000000000000000000551000falseobj000000000000000000552024-04-12T03:58:32.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.785 DEBUG:boto:path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.786 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.786 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.786 DEBUG:boto:Path: /dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.786 DEBUG:boto:Data: 2024-04-12T03:58:50.786 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.786 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.786 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.786 DEBUG:boto:Params: {} 2024-04-12T03:58:50.786 DEBUG:boto:Token: None 2024-04-12T03:58:50.787 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.787 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SEkFaE0cFXvOTlH3znETMEcGE4Y= 2024-04-12T03:58:50.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SEkFaE0cFXvOTlH3znETMEcGE4Y=', 'Content-Length': '0'} 2024-04-12T03:58:50.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000553ad2b63cd5aa46-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.789 DEBUG:boto:path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.789 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.789 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.789 DEBUG:boto:Path: /dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.789 DEBUG:boto:Data: 2024-04-12T03:58:50.789 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.789 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.789 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.789 DEBUG:boto:Params: {} 2024-04-12T03:58:50.790 DEBUG:boto:Token: None 2024-04-12T03:58:50.790 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.790 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SEkFaE0cFXvOTlH3znETMEcGE4Y= 2024-04-12T03:58:50.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SEkFaE0cFXvOTlH3znETMEcGE4Y=', 'Content-Length': '0'} 2024-04-12T03:58:50.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dec1c3bd8b433de-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2024-04-12T03:58:50.792 DEBUG:boto:path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.792 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.792 DEBUG:boto:Method: GET 2024-04-12T03:58:50.792 DEBUG:boto:Path: /dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.793 DEBUG:boto:Data: 2024-04-12T03:58:50.793 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.793 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.793 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.793 DEBUG:boto:Params: {} 2024-04-12T03:58:50.793 DEBUG:boto:Token: None 2024-04-12T03:58:50.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.793 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DqxyQJ5n3I6RCa+bMEyDMTrvrZQ= 2024-04-12T03:58:50.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DqxyQJ5n3I6RCa+bMEyDMTrvrZQ=', 'Content-Length': '0'} 2024-04-12T03:58:50.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c36a5a41e917ad4-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.795 DEBUG:boto:path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.795 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.795 DEBUG:boto:Method: GET 2024-04-12T03:58:50.796 DEBUG:boto:Path: /dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.796 DEBUG:boto:Data: 2024-04-12T03:58:50.796 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.796 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.796 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.796 DEBUG:boto:Params: {} 2024-04-12T03:58:50.796 DEBUG:boto:Token: None 2024-04-12T03:58:50.796 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000055 2024-04-12T03:58:50.796 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DqxyQJ5n3I6RCa+bMEyDMTrvrZQ= 2024-04-12T03:58:50.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DqxyQJ5n3I6RCa+bMEyDMTrvrZQ=', 'Content-Length': '0'} 2024-04-12T03:58:50.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc9aab3fca5a6136-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.799 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.799 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.799 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000055 2024-04-12T03:58:50.799 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000055 2024-04-12T03:58:50.799 DEBUG:boto:Method: GET 2024-04-12T03:58:50.799 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000055 2024-04-12T03:58:50.799 DEBUG:boto:Data: 2024-04-12T03:58:50.799 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.799 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.800 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.800 DEBUG:boto:Params: {} 2024-04-12T03:58:50.800 DEBUG:boto:Token: None 2024-04-12T03:58:50.800 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.800 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058eaa23b44d8b980-006618b17a-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.846 DEBUG:boto:b'dkyges-60obj000000000000000000551000falseobj000000000000000000552024-04-12T03:58:32.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.847 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.847 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.847 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000055 2024-04-12T03:58:50.847 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000055 2024-04-12T03:58:50.847 DEBUG:boto:Method: GET 2024-04-12T03:58:50.847 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000055 2024-04-12T03:58:50.847 DEBUG:boto:Data: 2024-04-12T03:58:50.847 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.847 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.848 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.848 DEBUG:boto:Params: {} 2024-04-12T03:58:50.848 DEBUG:boto:Token: None 2024-04-12T03:58:50.848 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.848 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096ebbdf2c25aa797-006618b17a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.853 DEBUG:boto:b'dkyges-60obj000000000000000000551000falseobj000000000000000000552024-04-12T03:58:32.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-04-12T03:58:50.853 DEBUG:boto:path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.853 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.853 DEBUG:boto:path=/dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.853 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.853 DEBUG:boto:Method: GET 2024-04-12T03:58:50.854 DEBUG:boto:Path: /dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.854 DEBUG:boto:Data: 2024-04-12T03:58:50.854 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.854 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.854 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.854 DEBUG:boto:Params: {} 2024-04-12T03:58:50.854 DEBUG:boto:Token: None 2024-04-12T03:58:50.854 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.854 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f/paQwDxcKNGjNJcFyO2RJDemBw= 2024-04-12T03:58:50.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f/paQwDxcKNGjNJcFyO2RJDemBw=', 'Content-Length': '0'} 2024-04-12T03:58:50.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000323d0c0a42cf915d-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.857 DEBUG:boto:path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.857 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.857 DEBUG:boto:path=/dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.857 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.857 DEBUG:boto:Method: GET 2024-04-12T03:58:50.857 DEBUG:boto:Path: /dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.857 DEBUG:boto:Data: 2024-04-12T03:58:50.858 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.858 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.858 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.858 DEBUG:boto:Params: {} 2024-04-12T03:58:50.858 DEBUG:boto:Token: None 2024-04-12T03:58:50.858 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.858 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f/paQwDxcKNGjNJcFyO2RJDemBw= 2024-04-12T03:58:50.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f/paQwDxcKNGjNJcFyO2RJDemBw=', 'Content-Length': '0'} 2024-04-12T03:58:50.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc2d1e263b7bdcf6-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.902 DEBUG:boto:path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.902 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.903 DEBUG:boto:path=/dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.903 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.903 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.903 DEBUG:boto:Path: /dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.903 DEBUG:boto:Data: 2024-04-12T03:58:50.903 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.903 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.903 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.903 DEBUG:boto:Params: {} 2024-04-12T03:58:50.903 DEBUG:boto:Token: None 2024-04-12T03:58:50.904 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.904 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2CRSuieJfAlRpmvSLuHZs86T54w= 2024-04-12T03:58:50.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2CRSuieJfAlRpmvSLuHZs86T54w=', 'Content-Length': '0'} 2024-04-12T03:58:50.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088a42e75033da485-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.906 DEBUG:boto:path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.906 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.906 DEBUG:boto:path=/dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.906 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.906 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.906 DEBUG:boto:Path: /dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.907 DEBUG:boto:Data: 2024-04-12T03:58:50.907 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.907 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.907 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.907 DEBUG:boto:Params: {} 2024-04-12T03:58:50.907 DEBUG:boto:Token: None 2024-04-12T03:58:50.907 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.907 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2CRSuieJfAlRpmvSLuHZs86T54w= 2024-04-12T03:58:50.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2CRSuieJfAlRpmvSLuHZs86T54w=', 'Content-Length': '0'} 2024-04-12T03:58:50.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000325506420ec84c00-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-04-12T03:58:50.909 DEBUG:boto:path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.910 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.910 DEBUG:boto:path=/dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.910 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.910 DEBUG:boto:Method: GET 2024-04-12T03:58:50.910 DEBUG:boto:Path: /dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.910 DEBUG:boto:Data: 2024-04-12T03:58:50.910 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.910 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.910 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.910 DEBUG:boto:Params: {} 2024-04-12T03:58:50.911 DEBUG:boto:Token: None 2024-04-12T03:58:50.911 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.911 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f/paQwDxcKNGjNJcFyO2RJDemBw= 2024-04-12T03:58:50.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f/paQwDxcKNGjNJcFyO2RJDemBw=', 'Content-Length': '0'} 2024-04-12T03:58:50.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018a4721cc5ee2e4f-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.913 DEBUG:boto:path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.913 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.913 DEBUG:boto:path=/dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.914 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.914 DEBUG:boto:Method: GET 2024-04-12T03:58:50.914 DEBUG:boto:Path: /dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.914 DEBUG:boto:Data: 2024-04-12T03:58:50.914 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.914 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.914 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.914 DEBUG:boto:Params: {} 2024-04-12T03:58:50.914 DEBUG:boto:Token: None 2024-04-12T03:58:50.914 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000056?versionId=null 2024-04-12T03:58:50.915 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f/paQwDxcKNGjNJcFyO2RJDemBw= 2024-04-12T03:58:50.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f/paQwDxcKNGjNJcFyO2RJDemBw=', 'Content-Length': '0'} 2024-04-12T03:58:50.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085729b7338af1caa-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.917 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.918 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.918 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000056 2024-04-12T03:58:50.918 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000056 2024-04-12T03:58:50.918 DEBUG:boto:Method: GET 2024-04-12T03:58:50.918 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000056 2024-04-12T03:58:50.918 DEBUG:boto:Data: 2024-04-12T03:58:50.918 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.918 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.918 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.918 DEBUG:boto:Params: {} 2024-04-12T03:58:50.919 DEBUG:boto:Token: None 2024-04-12T03:58:50.919 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.919 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e256e363e37d5f8f-006618b17a-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.921 DEBUG:boto:b'dkyges-60obj000000000000000000561000falseobj000000000000000000562024-04-12T03:58:34.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.922 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.922 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.922 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000056 2024-04-12T03:58:50.922 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000056 2024-04-12T03:58:50.922 DEBUG:boto:Method: GET 2024-04-12T03:58:50.922 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000056 2024-04-12T03:58:50.922 DEBUG:boto:Data: 2024-04-12T03:58:50.922 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.923 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.923 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.923 DEBUG:boto:Params: {} 2024-04-12T03:58:50.923 DEBUG:boto:Token: None 2024-04-12T03:58:50.923 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.923 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d830a77163e9a27-006618b17a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.928 DEBUG:boto:b'dkyges-60obj000000000000000000561000falseobj000000000000000000562024-04-12T03:58:34.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.928 DEBUG:boto:path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.929 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.929 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.929 DEBUG:boto:Path: /dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.929 DEBUG:boto:Data: 2024-04-12T03:58:50.929 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.929 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.929 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.929 DEBUG:boto:Params: {} 2024-04-12T03:58:50.929 DEBUG:boto:Token: None 2024-04-12T03:58:50.930 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.930 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KA7llQb/I5WFJeMFCcoh97DSLO4= 2024-04-12T03:58:50.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KA7llQb/I5WFJeMFCcoh97DSLO4=', 'Content-Length': '0'} 2024-04-12T03:58:50.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc74e8d4ccf421a9-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.932 DEBUG:boto:path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.932 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.932 DEBUG:boto:Method: HEAD 2024-04-12T03:58:50.932 DEBUG:boto:Path: /dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.932 DEBUG:boto:Data: 2024-04-12T03:58:50.932 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.932 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.932 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.932 DEBUG:boto:Params: {} 2024-04-12T03:58:50.933 DEBUG:boto:Token: None 2024-04-12T03:58:50.933 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.933 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KA7llQb/I5WFJeMFCcoh97DSLO4= 2024-04-12T03:58:50.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KA7llQb/I5WFJeMFCcoh97DSLO4=', 'Content-Length': '0'} 2024-04-12T03:58:50.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3a9ced633f000ec-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2024-04-12T03:58:50.935 DEBUG:boto:path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.935 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.935 DEBUG:boto:Method: GET 2024-04-12T03:58:50.936 DEBUG:boto:Path: /dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.936 DEBUG:boto:Data: 2024-04-12T03:58:50.936 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.936 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.936 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.936 DEBUG:boto:Params: {} 2024-04-12T03:58:50.936 DEBUG:boto:Token: None 2024-04-12T03:58:50.936 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.936 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HfIhC57oXW2KzNwq1THZgAsiTxI= 2024-04-12T03:58:50.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HfIhC57oXW2KzNwq1THZgAsiTxI=', 'Content-Length': '0'} 2024-04-12T03:58:50.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a11dfa9669e7e3cf-006618b17a-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.939 DEBUG:boto:path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.939 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.939 DEBUG:boto:Method: GET 2024-04-12T03:58:50.939 DEBUG:boto:Path: /dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.939 DEBUG:boto:Data: 2024-04-12T03:58:50.939 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.939 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.939 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.939 DEBUG:boto:Params: {} 2024-04-12T03:58:50.940 DEBUG:boto:Token: None 2024-04-12T03:58:50.940 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000056 2024-04-12T03:58:50.940 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HfIhC57oXW2KzNwq1THZgAsiTxI= 2024-04-12T03:58:50.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HfIhC57oXW2KzNwq1THZgAsiTxI=', 'Content-Length': '0'} 2024-04-12T03:58:50.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2868e2ff5ab74c7-006618b17a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.982 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.982 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.983 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000056 2024-04-12T03:58:50.983 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000056 2024-04-12T03:58:50.983 DEBUG:boto:Method: GET 2024-04-12T03:58:50.983 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000056 2024-04-12T03:58:50.983 DEBUG:boto:Data: 2024-04-12T03:58:50.983 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.983 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.983 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.983 DEBUG:boto:Params: {} 2024-04-12T03:58:50.983 DEBUG:boto:Token: None 2024-04-12T03:58:50.983 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.984 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.986 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dce86446c69962d-006618b17a-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.986 DEBUG:boto:b'dkyges-60obj000000000000000000561000falseobj000000000000000000562024-04-12T03:58:34.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.987 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:50.987 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:50.987 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000056 2024-04-12T03:58:50.987 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000056 2024-04-12T03:58:50.987 DEBUG:boto:Method: GET 2024-04-12T03:58:50.987 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000056 2024-04-12T03:58:50.987 DEBUG:boto:Data: 2024-04-12T03:58:50.988 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.988 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.988 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.988 DEBUG:boto:Params: {} 2024-04-12T03:58:50.988 DEBUG:boto:Token: None 2024-04-12T03:58:50.988 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/ 2024-04-12T03:58:50.988 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs= 2024-04-12T03:58:50.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4+EeYOBzh9hzcYTbYU5Df74mDcs=', 'Content-Length': '0'} 2024-04-12T03:58:50.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081d673fc4559cb90-006618b17b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.993 DEBUG:boto:b'dkyges-60obj000000000000000000561000falseobj000000000000000000562024-04-12T03:58:34.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:50.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-04-12T03:58:50.993 DEBUG:boto:path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:50.993 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:50.993 DEBUG:boto:path=/dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:50.993 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:50.993 DEBUG:boto:Method: GET 2024-04-12T03:58:50.993 DEBUG:boto:Path: /dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:50.993 DEBUG:boto:Data: 2024-04-12T03:58:50.994 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.994 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:50.994 DEBUG:boto:Port: 8001 2024-04-12T03:58:50.994 DEBUG:boto:Params: {} 2024-04-12T03:58:50.994 DEBUG:boto:Token: None 2024-04-12T03:58:50.994 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:50.994 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IPrcrOg0YT3O34C4hVblntAVhsk= 2024-04-12T03:58:50.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IPrcrOg0YT3O34C4hVblntAVhsk=', 'Content-Length': '0'} 2024-04-12T03:58:50.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000076df01fe7a81285-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:50.996 DEBUG:boto:path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:50.996 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:50.997 DEBUG:boto:path=/dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:50.997 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:50.997 DEBUG:boto:Method: GET 2024-04-12T03:58:50.997 DEBUG:boto:Path: /dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:50.997 DEBUG:boto:Data: 2024-04-12T03:58:50.997 DEBUG:boto:Headers: {} 2024-04-12T03:58:50.997 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:50.997 DEBUG:boto:Port: 8000 2024-04-12T03:58:50.997 DEBUG:boto:Params: {} 2024-04-12T03:58:50.997 DEBUG:boto:Token: None 2024-04-12T03:58:50.997 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:50 GMT /dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:50.998 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IPrcrOg0YT3O34C4hVblntAVhsk= 2024-04-12T03:58:50.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IPrcrOg0YT3O34C4hVblntAVhsk=', 'Content-Length': '0'} 2024-04-12T03:58:50.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033bae567976afec6-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.000 DEBUG:boto:path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.000 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.000 DEBUG:boto:path=/dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:51.000 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:51.000 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.000 DEBUG:boto:Path: /dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:51.000 DEBUG:boto:Data: 2024-04-12T03:58:51.000 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.000 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.000 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.000 DEBUG:boto:Params: {} 2024-04-12T03:58:51.000 DEBUG:boto:Token: None 2024-04-12T03:58:51.001 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:51.001 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6hu62n+z2bVS2MA2q90/sJtwa9M= 2024-04-12T03:58:51.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6hu62n+z2bVS2MA2q90/sJtwa9M=', 'Content-Length': '0'} 2024-04-12T03:58:51.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec6c6bbb527bdb58-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.003 DEBUG:boto:path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.003 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.003 DEBUG:boto:path=/dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:51.003 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:51.003 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.003 DEBUG:boto:Path: /dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:51.003 DEBUG:boto:Data: 2024-04-12T03:58:51.003 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.003 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.003 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.003 DEBUG:boto:Params: {} 2024-04-12T03:58:51.004 DEBUG:boto:Token: None 2024-04-12T03:58:51.004 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:51.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6hu62n+z2bVS2MA2q90/sJtwa9M= 2024-04-12T03:58:51.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6hu62n+z2bVS2MA2q90/sJtwa9M=', 'Content-Length': '0'} 2024-04-12T03:58:51.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc7dba55f2b5411a-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-04-12T03:58:51.006 DEBUG:boto:path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.006 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.006 DEBUG:boto:path=/dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:51.006 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:51.007 DEBUG:boto:Method: GET 2024-04-12T03:58:51.007 DEBUG:boto:Path: /dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:51.007 DEBUG:boto:Data: 2024-04-12T03:58:51.007 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.007 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.007 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.007 DEBUG:boto:Params: {} 2024-04-12T03:58:51.007 DEBUG:boto:Token: None 2024-04-12T03:58:51.007 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:51.007 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hzluwv50/9Qv8jqC+AqD/h52tIE= 2024-04-12T03:58:51.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hzluwv50/9Qv8jqC+AqD/h52tIE=', 'Content-Length': '0'} 2024-04-12T03:58:51.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdc214be4abade1c-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.050 DEBUG:boto:path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.050 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.051 DEBUG:boto:path=/dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:51.051 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:51.051 DEBUG:boto:Method: GET 2024-04-12T03:58:51.051 DEBUG:boto:Path: /dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:51.051 DEBUG:boto:Data: 2024-04-12T03:58:51.051 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.051 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.051 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.051 DEBUG:boto:Params: {} 2024-04-12T03:58:51.051 DEBUG:boto:Token: None 2024-04-12T03:58:51.051 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000057?versionId=null 2024-04-12T03:58:51.052 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hzluwv50/9Qv8jqC+AqD/h52tIE= 2024-04-12T03:58:51.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hzluwv50/9Qv8jqC+AqD/h52tIE=', 'Content-Length': '0'} 2024-04-12T03:58:51.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0cd496a87eb6a6a-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.053 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.054 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.054 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000057 2024-04-12T03:58:51.054 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000057 2024-04-12T03:58:51.054 DEBUG:boto:Method: GET 2024-04-12T03:58:51.054 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000057 2024-04-12T03:58:51.054 DEBUG:boto:Data: 2024-04-12T03:58:51.054 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.054 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.054 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.054 DEBUG:boto:Params: {} 2024-04-12T03:58:51.054 DEBUG:boto:Token: None 2024-04-12T03:58:51.055 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.055 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000782a64d7eddd71f0-006618b17b-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.058 DEBUG:boto:b'dkyges-60obj000000000000000000571000falseobj000000000000000000572024-04-12T03:58:35.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.058 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.058 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.058 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000057 2024-04-12T03:58:51.058 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000057 2024-04-12T03:58:51.059 DEBUG:boto:Method: GET 2024-04-12T03:58:51.059 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000057 2024-04-12T03:58:51.059 DEBUG:boto:Data: 2024-04-12T03:58:51.059 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.059 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.059 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.059 DEBUG:boto:Params: {} 2024-04-12T03:58:51.059 DEBUG:boto:Token: None 2024-04-12T03:58:51.059 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.060 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0631f4db5ae7704-006618b17b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.064 DEBUG:boto:b'dkyges-60obj000000000000000000571000falseobj000000000000000000572024-04-12T03:58:35.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.064 DEBUG:boto:path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.065 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.065 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.065 DEBUG:boto:Path: /dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.065 DEBUG:boto:Data: 2024-04-12T03:58:51.065 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.065 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.065 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.065 DEBUG:boto:Params: {} 2024-04-12T03:58:51.065 DEBUG:boto:Token: None 2024-04-12T03:58:51.065 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.066 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2rVUMtlym0Z/gvmJ107dZ/S11Ng= 2024-04-12T03:58:51.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2rVUMtlym0Z/gvmJ107dZ/S11Ng=', 'Content-Length': '0'} 2024-04-12T03:58:51.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7c1bac760972fe8-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.068 DEBUG:boto:path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.068 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.068 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.068 DEBUG:boto:Path: /dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.068 DEBUG:boto:Data: 2024-04-12T03:58:51.068 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.068 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.068 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.068 DEBUG:boto:Params: {} 2024-04-12T03:58:51.069 DEBUG:boto:Token: None 2024-04-12T03:58:51.069 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.069 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2rVUMtlym0Z/gvmJ107dZ/S11Ng= 2024-04-12T03:58:51.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2rVUMtlym0Z/gvmJ107dZ/S11Ng=', 'Content-Length': '0'} 2024-04-12T03:58:51.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084614095082b3d66-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2024-04-12T03:58:51.071 DEBUG:boto:path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.071 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.071 DEBUG:boto:Method: GET 2024-04-12T03:58:51.071 DEBUG:boto:Path: /dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.071 DEBUG:boto:Data: 2024-04-12T03:58:51.072 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.072 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.072 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.072 DEBUG:boto:Params: {} 2024-04-12T03:58:51.072 DEBUG:boto:Token: None 2024-04-12T03:58:51.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.072 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ipr2+y4joeNCvtPUDuKhqsPKykc= 2024-04-12T03:58:51.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ipr2+y4joeNCvtPUDuKhqsPKykc=', 'Content-Length': '0'} 2024-04-12T03:58:51.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ebb16781e948981-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.074 DEBUG:boto:path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.074 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.075 DEBUG:boto:Method: GET 2024-04-12T03:58:51.075 DEBUG:boto:Path: /dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.075 DEBUG:boto:Data: 2024-04-12T03:58:51.075 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.075 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.075 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.075 DEBUG:boto:Params: {} 2024-04-12T03:58:51.075 DEBUG:boto:Token: None 2024-04-12T03:58:51.075 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000057 2024-04-12T03:58:51.075 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ipr2+y4joeNCvtPUDuKhqsPKykc= 2024-04-12T03:58:51.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ipr2+y4joeNCvtPUDuKhqsPKykc=', 'Content-Length': '0'} 2024-04-12T03:58:51.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7e4762fb087d5c1-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.077 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.078 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.078 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000057 2024-04-12T03:58:51.078 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000057 2024-04-12T03:58:51.078 DEBUG:boto:Method: GET 2024-04-12T03:58:51.078 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000057 2024-04-12T03:58:51.078 DEBUG:boto:Data: 2024-04-12T03:58:51.078 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.078 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.078 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.078 DEBUG:boto:Params: {} 2024-04-12T03:58:51.079 DEBUG:boto:Token: None 2024-04-12T03:58:51.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.079 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000683b19d7df50b398-006618b17b-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.126 DEBUG:boto:b'dkyges-60obj000000000000000000571000falseobj000000000000000000572024-04-12T03:58:35.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.127 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.127 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.127 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000057 2024-04-12T03:58:51.127 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000057 2024-04-12T03:58:51.127 DEBUG:boto:Method: GET 2024-04-12T03:58:51.127 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000057 2024-04-12T03:58:51.128 DEBUG:boto:Data: 2024-04-12T03:58:51.128 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.128 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.128 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.128 DEBUG:boto:Params: {} 2024-04-12T03:58:51.128 DEBUG:boto:Token: None 2024-04-12T03:58:51.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.129 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f389ec95fa16132-006618b17b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.133 DEBUG:boto:b'dkyges-60obj000000000000000000571000falseobj000000000000000000572024-04-12T03:58:35.395Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-04-12T03:58:51.134 DEBUG:boto:path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.134 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.134 DEBUG:boto:path=/dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.134 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.134 DEBUG:boto:Method: GET 2024-04-12T03:58:51.134 DEBUG:boto:Path: /dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.134 DEBUG:boto:Data: 2024-04-12T03:58:51.134 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.134 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.135 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.135 DEBUG:boto:Params: {} 2024-04-12T03:58:51.135 DEBUG:boto:Token: None 2024-04-12T03:58:51.135 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.135 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP/fOQ5R6602QFWVHtZ+hn0/3rM= 2024-04-12T03:58:51.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP/fOQ5R6602QFWVHtZ+hn0/3rM=', 'Content-Length': '0'} 2024-04-12T03:58:51.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4410902e26cf08b-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.137 DEBUG:boto:path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.137 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.137 DEBUG:boto:path=/dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.138 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.138 DEBUG:boto:Method: GET 2024-04-12T03:58:51.138 DEBUG:boto:Path: /dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.138 DEBUG:boto:Data: 2024-04-12T03:58:51.138 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.138 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.138 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.138 DEBUG:boto:Params: {} 2024-04-12T03:58:51.138 DEBUG:boto:Token: None 2024-04-12T03:58:51.138 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.139 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP/fOQ5R6602QFWVHtZ+hn0/3rM= 2024-04-12T03:58:51.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP/fOQ5R6602QFWVHtZ+hn0/3rM=', 'Content-Length': '0'} 2024-04-12T03:58:51.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065d7271989350a93-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.182 DEBUG:boto:path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.182 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.183 DEBUG:boto:path=/dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.183 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.183 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.183 DEBUG:boto:Path: /dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.183 DEBUG:boto:Data: 2024-04-12T03:58:51.183 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.183 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.183 DEBUG:boto:Params: {} 2024-04-12T03:58:51.183 DEBUG:boto:Token: None 2024-04-12T03:58:51.184 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.184 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1vTq6aVwRbF4+yW0DsQIlQxzyZM= 2024-04-12T03:58:51.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1vTq6aVwRbF4+yW0DsQIlQxzyZM=', 'Content-Length': '0'} 2024-04-12T03:58:51.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a42037d9edf3b7a-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.186 DEBUG:boto:path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.186 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.186 DEBUG:boto:path=/dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.186 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.186 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.186 DEBUG:boto:Path: /dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.187 DEBUG:boto:Data: 2024-04-12T03:58:51.187 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.187 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.187 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.187 DEBUG:boto:Params: {} 2024-04-12T03:58:51.187 DEBUG:boto:Token: None 2024-04-12T03:58:51.187 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.187 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1vTq6aVwRbF4+yW0DsQIlQxzyZM= 2024-04-12T03:58:51.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1vTq6aVwRbF4+yW0DsQIlQxzyZM=', 'Content-Length': '0'} 2024-04-12T03:58:51.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b2d52538c3beac1-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-04-12T03:58:51.190 DEBUG:boto:path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.190 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.190 DEBUG:boto:path=/dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.190 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.190 DEBUG:boto:Method: GET 2024-04-12T03:58:51.190 DEBUG:boto:Path: /dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.190 DEBUG:boto:Data: 2024-04-12T03:58:51.191 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.191 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.191 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.191 DEBUG:boto:Params: {} 2024-04-12T03:58:51.191 DEBUG:boto:Token: None 2024-04-12T03:58:51.191 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.191 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP/fOQ5R6602QFWVHtZ+hn0/3rM= 2024-04-12T03:58:51.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP/fOQ5R6602QFWVHtZ+hn0/3rM=', 'Content-Length': '0'} 2024-04-12T03:58:51.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a6bf7f9e31d0634-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.194 DEBUG:boto:path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.194 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.194 DEBUG:boto:path=/dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.194 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.194 DEBUG:boto:Method: GET 2024-04-12T03:58:51.194 DEBUG:boto:Path: /dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.194 DEBUG:boto:Data: 2024-04-12T03:58:51.194 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.194 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.194 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.194 DEBUG:boto:Params: {} 2024-04-12T03:58:51.195 DEBUG:boto:Token: None 2024-04-12T03:58:51.195 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000058?versionId=null 2024-04-12T03:58:51.195 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xP/fOQ5R6602QFWVHtZ+hn0/3rM= 2024-04-12T03:58:51.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xP/fOQ5R6602QFWVHtZ+hn0/3rM=', 'Content-Length': '0'} 2024-04-12T03:58:51.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e4001415f273ee3-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.197 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.197 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.198 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000058 2024-04-12T03:58:51.198 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000058 2024-04-12T03:58:51.198 DEBUG:boto:Method: GET 2024-04-12T03:58:51.198 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000058 2024-04-12T03:58:51.198 DEBUG:boto:Data: 2024-04-12T03:58:51.198 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.198 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.198 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.198 DEBUG:boto:Params: {} 2024-04-12T03:58:51.198 DEBUG:boto:Token: None 2024-04-12T03:58:51.198 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.199 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033d1ce1bd71ea08b-006618b17b-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.201 DEBUG:boto:b'dkyges-60obj000000000000000000581000falseobj000000000000000000582024-04-12T03:58:31.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.201 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.202 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.202 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000058 2024-04-12T03:58:51.202 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000058 2024-04-12T03:58:51.202 DEBUG:boto:Method: GET 2024-04-12T03:58:51.202 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000058 2024-04-12T03:58:51.202 DEBUG:boto:Data: 2024-04-12T03:58:51.202 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.202 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.202 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.202 DEBUG:boto:Params: {} 2024-04-12T03:58:51.203 DEBUG:boto:Token: None 2024-04-12T03:58:51.203 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.203 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002759188e7752b059-006618b17b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.207 DEBUG:boto:b'dkyges-60obj000000000000000000581000falseobj000000000000000000582024-04-12T03:58:31.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.207 DEBUG:boto:path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.208 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.208 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.208 DEBUG:boto:Path: /dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.208 DEBUG:boto:Data: 2024-04-12T03:58:51.208 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.208 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.208 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.208 DEBUG:boto:Params: {} 2024-04-12T03:58:51.208 DEBUG:boto:Token: None 2024-04-12T03:58:51.208 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V9kzj+AqI7xsUMyr664DBAYzOuU= 2024-04-12T03:58:51.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V9kzj+AqI7xsUMyr664DBAYzOuU=', 'Content-Length': '0'} 2024-04-12T03:58:51.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079af433e5c1430a0-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.211 DEBUG:boto:path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.211 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.211 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.211 DEBUG:boto:Path: /dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.211 DEBUG:boto:Data: 2024-04-12T03:58:51.211 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.211 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.211 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.211 DEBUG:boto:Params: {} 2024-04-12T03:58:51.211 DEBUG:boto:Token: None 2024-04-12T03:58:51.211 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V9kzj+AqI7xsUMyr664DBAYzOuU= 2024-04-12T03:58:51.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V9kzj+AqI7xsUMyr664DBAYzOuU=', 'Content-Length': '0'} 2024-04-12T03:58:51.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087e36a5a272b22f1-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2024-04-12T03:58:51.214 DEBUG:boto:path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.214 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.214 DEBUG:boto:Method: GET 2024-04-12T03:58:51.214 DEBUG:boto:Path: /dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.214 DEBUG:boto:Data: 2024-04-12T03:58:51.215 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.215 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.215 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.215 DEBUG:boto:Params: {} 2024-04-12T03:58:51.215 DEBUG:boto:Token: None 2024-04-12T03:58:51.215 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.215 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rdEvOcIZFZDyGuT7swQ/N6w73Vw= 2024-04-12T03:58:51.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rdEvOcIZFZDyGuT7swQ/N6w73Vw=', 'Content-Length': '0'} 2024-04-12T03:58:51.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c17a26dd6f35a9d3-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.217 DEBUG:boto:path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.217 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.218 DEBUG:boto:Method: GET 2024-04-12T03:58:51.218 DEBUG:boto:Path: /dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.218 DEBUG:boto:Data: 2024-04-12T03:58:51.218 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.218 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.218 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.218 DEBUG:boto:Params: {} 2024-04-12T03:58:51.218 DEBUG:boto:Token: None 2024-04-12T03:58:51.218 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000058 2024-04-12T03:58:51.218 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rdEvOcIZFZDyGuT7swQ/N6w73Vw= 2024-04-12T03:58:51.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rdEvOcIZFZDyGuT7swQ/N6w73Vw=', 'Content-Length': '0'} 2024-04-12T03:58:51.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000869f7338820f8d76-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.221 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.221 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.221 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000058 2024-04-12T03:58:51.221 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000058 2024-04-12T03:58:51.221 DEBUG:boto:Method: GET 2024-04-12T03:58:51.221 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000058 2024-04-12T03:58:51.221 DEBUG:boto:Data: 2024-04-12T03:58:51.221 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.221 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.221 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.221 DEBUG:boto:Params: {} 2024-04-12T03:58:51.222 DEBUG:boto:Token: None 2024-04-12T03:58:51.222 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.222 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2cefaa0fa189052-006618b17b-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.224 DEBUG:boto:b'dkyges-60obj000000000000000000581000falseobj000000000000000000582024-04-12T03:58:31.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.225 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.225 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.225 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000058 2024-04-12T03:58:51.225 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000058 2024-04-12T03:58:51.225 DEBUG:boto:Method: GET 2024-04-12T03:58:51.225 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000058 2024-04-12T03:58:51.225 DEBUG:boto:Data: 2024-04-12T03:58:51.225 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.225 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.226 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.226 DEBUG:boto:Params: {} 2024-04-12T03:58:51.226 DEBUG:boto:Token: None 2024-04-12T03:58:51.226 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.226 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009be7423c67c0867-006618b17b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.231 DEBUG:boto:b'dkyges-60obj000000000000000000581000falseobj000000000000000000582024-04-12T03:58:31.481Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-04-12T03:58:51.231 DEBUG:boto:path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.231 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.231 DEBUG:boto:path=/dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.231 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.232 DEBUG:boto:Method: GET 2024-04-12T03:58:51.232 DEBUG:boto:Path: /dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.232 DEBUG:boto:Data: 2024-04-12T03:58:51.232 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.232 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.232 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.232 DEBUG:boto:Params: {} 2024-04-12T03:58:51.232 DEBUG:boto:Token: None 2024-04-12T03:58:51.232 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.232 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ks012kdfAw0YSmAldnyCdtwLsAc= 2024-04-12T03:58:51.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ks012kdfAw0YSmAldnyCdtwLsAc=', 'Content-Length': '0'} 2024-04-12T03:58:51.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e9f6a3a2d726e89-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.235 DEBUG:boto:path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.235 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.235 DEBUG:boto:path=/dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.235 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.235 DEBUG:boto:Method: GET 2024-04-12T03:58:51.235 DEBUG:boto:Path: /dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.235 DEBUG:boto:Data: 2024-04-12T03:58:51.235 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.235 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.235 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.235 DEBUG:boto:Params: {} 2024-04-12T03:58:51.236 DEBUG:boto:Token: None 2024-04-12T03:58:51.236 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.236 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ks012kdfAw0YSmAldnyCdtwLsAc= 2024-04-12T03:58:51.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ks012kdfAw0YSmAldnyCdtwLsAc=', 'Content-Length': '0'} 2024-04-12T03:58:51.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f80bd2c54049ce28-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.282 DEBUG:boto:path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.282 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.282 DEBUG:boto:path=/dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.282 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.283 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.283 DEBUG:boto:Path: /dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.283 DEBUG:boto:Data: 2024-04-12T03:58:51.283 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.283 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.283 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.283 DEBUG:boto:Params: {} 2024-04-12T03:58:51.283 DEBUG:boto:Token: None 2024-04-12T03:58:51.283 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.283 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nH7FBKAze8ItBW+w2i85MZm16ug= 2024-04-12T03:58:51.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nH7FBKAze8ItBW+w2i85MZm16ug=', 'Content-Length': '0'} 2024-04-12T03:58:51.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e437948fe91a1c34-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.285 DEBUG:boto:path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.285 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.286 DEBUG:boto:path=/dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.286 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.286 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.286 DEBUG:boto:Path: /dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.286 DEBUG:boto:Data: 2024-04-12T03:58:51.286 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.286 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.286 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.286 DEBUG:boto:Params: {} 2024-04-12T03:58:51.286 DEBUG:boto:Token: None 2024-04-12T03:58:51.286 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.286 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nH7FBKAze8ItBW+w2i85MZm16ug= 2024-04-12T03:58:51.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nH7FBKAze8ItBW+w2i85MZm16ug=', 'Content-Length': '0'} 2024-04-12T03:58:51.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0d041892dbfd5bf-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-04-12T03:58:51.289 DEBUG:boto:path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.289 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.289 DEBUG:boto:path=/dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.289 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.289 DEBUG:boto:Method: GET 2024-04-12T03:58:51.289 DEBUG:boto:Path: /dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.289 DEBUG:boto:Data: 2024-04-12T03:58:51.289 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.289 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.290 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.290 DEBUG:boto:Params: {} 2024-04-12T03:58:51.290 DEBUG:boto:Token: None 2024-04-12T03:58:51.290 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.290 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ks012kdfAw0YSmAldnyCdtwLsAc= 2024-04-12T03:58:51.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ks012kdfAw0YSmAldnyCdtwLsAc=', 'Content-Length': '0'} 2024-04-12T03:58:51.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002808d6a81866b0ec-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.293 DEBUG:boto:path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.293 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.293 DEBUG:boto:path=/dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.293 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.293 DEBUG:boto:Method: GET 2024-04-12T03:58:51.293 DEBUG:boto:Path: /dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.293 DEBUG:boto:Data: 2024-04-12T03:58:51.293 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.293 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.293 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.293 DEBUG:boto:Params: {} 2024-04-12T03:58:51.294 DEBUG:boto:Token: None 2024-04-12T03:58:51.294 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000059?versionId=null 2024-04-12T03:58:51.294 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ks012kdfAw0YSmAldnyCdtwLsAc= 2024-04-12T03:58:51.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ks012kdfAw0YSmAldnyCdtwLsAc=', 'Content-Length': '0'} 2024-04-12T03:58:51.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053cab39ef283a581-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.296 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.297 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.297 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000059 2024-04-12T03:58:51.297 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000059 2024-04-12T03:58:51.297 DEBUG:boto:Method: GET 2024-04-12T03:58:51.297 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000059 2024-04-12T03:58:51.297 DEBUG:boto:Data: 2024-04-12T03:58:51.297 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.297 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.297 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.297 DEBUG:boto:Params: {} 2024-04-12T03:58:51.298 DEBUG:boto:Token: None 2024-04-12T03:58:51.298 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.298 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063f7aea823fa2358-006618b17b-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.301 DEBUG:boto:b'dkyges-60obj000000000000000000591000falseobj000000000000000000592024-04-12T03:58:32.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.302 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.302 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.302 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000059 2024-04-12T03:58:51.302 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000059 2024-04-12T03:58:51.302 DEBUG:boto:Method: GET 2024-04-12T03:58:51.302 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000059 2024-04-12T03:58:51.302 DEBUG:boto:Data: 2024-04-12T03:58:51.303 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.303 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.303 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.303 DEBUG:boto:Params: {} 2024-04-12T03:58:51.303 DEBUG:boto:Token: None 2024-04-12T03:58:51.303 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.303 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.308 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e25eb3cbe564d5a7-006618b17b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.308 DEBUG:boto:b'dkyges-60obj000000000000000000591000falseobj000000000000000000592024-04-12T03:58:32.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.308 DEBUG:boto:path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.308 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.308 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.308 DEBUG:boto:Path: /dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.309 DEBUG:boto:Data: 2024-04-12T03:58:51.309 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.309 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.309 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.309 DEBUG:boto:Params: {} 2024-04-12T03:58:51.309 DEBUG:boto:Token: None 2024-04-12T03:58:51.309 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.309 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VjTMyrjUuiay75/4G4q/hicV24A= 2024-04-12T03:58:51.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VjTMyrjUuiay75/4G4q/hicV24A=', 'Content-Length': '0'} 2024-04-12T03:58:51.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c92ced0e2451a5f-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.311 DEBUG:boto:path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.312 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.312 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.312 DEBUG:boto:Path: /dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.312 DEBUG:boto:Data: 2024-04-12T03:58:51.312 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.312 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.312 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.312 DEBUG:boto:Params: {} 2024-04-12T03:58:51.312 DEBUG:boto:Token: None 2024-04-12T03:58:51.312 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.313 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VjTMyrjUuiay75/4G4q/hicV24A= 2024-04-12T03:58:51.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VjTMyrjUuiay75/4G4q/hicV24A=', 'Content-Length': '0'} 2024-04-12T03:58:51.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021701d420961e0ac-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2024-04-12T03:58:51.315 DEBUG:boto:path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.315 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.315 DEBUG:boto:Method: GET 2024-04-12T03:58:51.315 DEBUG:boto:Path: /dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.315 DEBUG:boto:Data: 2024-04-12T03:58:51.315 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.315 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.315 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.315 DEBUG:boto:Params: {} 2024-04-12T03:58:51.315 DEBUG:boto:Token: None 2024-04-12T03:58:51.316 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.316 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:10GyEKdUu18pZtUGfQgM6EQ/sXA= 2024-04-12T03:58:51.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:10GyEKdUu18pZtUGfQgM6EQ/sXA=', 'Content-Length': '0'} 2024-04-12T03:58:51.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012a647147a705e34-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.358 DEBUG:boto:path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.358 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.359 DEBUG:boto:Method: GET 2024-04-12T03:58:51.359 DEBUG:boto:Path: /dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.359 DEBUG:boto:Data: 2024-04-12T03:58:51.359 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.359 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.359 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.359 DEBUG:boto:Params: {} 2024-04-12T03:58:51.359 DEBUG:boto:Token: None 2024-04-12T03:58:51.359 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000059 2024-04-12T03:58:51.360 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:10GyEKdUu18pZtUGfQgM6EQ/sXA= 2024-04-12T03:58:51.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:10GyEKdUu18pZtUGfQgM6EQ/sXA=', 'Content-Length': '0'} 2024-04-12T03:58:51.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:32 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000636205c6634bab6a-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.362 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.362 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.362 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000059 2024-04-12T03:58:51.362 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000059 2024-04-12T03:58:51.362 DEBUG:boto:Method: GET 2024-04-12T03:58:51.362 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000059 2024-04-12T03:58:51.362 DEBUG:boto:Data: 2024-04-12T03:58:51.362 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.362 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.363 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.363 DEBUG:boto:Params: {} 2024-04-12T03:58:51.363 DEBUG:boto:Token: None 2024-04-12T03:58:51.363 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.363 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f1d9c0985684866-006618b17b-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.366 DEBUG:boto:b'dkyges-60obj000000000000000000591000falseobj000000000000000000592024-04-12T03:58:32.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.366 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.366 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.366 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000059 2024-04-12T03:58:51.366 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000059 2024-04-12T03:58:51.367 DEBUG:boto:Method: GET 2024-04-12T03:58:51.367 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000059 2024-04-12T03:58:51.367 DEBUG:boto:Data: 2024-04-12T03:58:51.367 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.367 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.367 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.367 DEBUG:boto:Params: {} 2024-04-12T03:58:51.367 DEBUG:boto:Token: None 2024-04-12T03:58:51.367 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.367 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045dec19a2f7712c7-006618b17b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.372 DEBUG:boto:b'dkyges-60obj000000000000000000591000falseobj000000000000000000592024-04-12T03:58:32.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-04-12T03:58:51.372 DEBUG:boto:path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.372 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.373 DEBUG:boto:path=/dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.373 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.373 DEBUG:boto:Method: GET 2024-04-12T03:58:51.373 DEBUG:boto:Path: /dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.373 DEBUG:boto:Data: 2024-04-12T03:58:51.373 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.373 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.373 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.373 DEBUG:boto:Params: {} 2024-04-12T03:58:51.373 DEBUG:boto:Token: None 2024-04-12T03:58:51.373 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.373 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2Qe2jC9OvTL32bOWm40/nhVHTAk= 2024-04-12T03:58:51.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2Qe2jC9OvTL32bOWm40/nhVHTAk=', 'Content-Length': '0'} 2024-04-12T03:58:51.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003d292e15c07dd9b-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.376 DEBUG:boto:path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.376 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.376 DEBUG:boto:path=/dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.376 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.376 DEBUG:boto:Method: GET 2024-04-12T03:58:51.376 DEBUG:boto:Path: /dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.376 DEBUG:boto:Data: 2024-04-12T03:58:51.376 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.376 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.377 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.377 DEBUG:boto:Params: {} 2024-04-12T03:58:51.377 DEBUG:boto:Token: None 2024-04-12T03:58:51.377 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.377 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2Qe2jC9OvTL32bOWm40/nhVHTAk= 2024-04-12T03:58:51.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2Qe2jC9OvTL32bOWm40/nhVHTAk=', 'Content-Length': '0'} 2024-04-12T03:58:51.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc97cafcb0c2f2c5-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.422 DEBUG:boto:path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.422 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.423 DEBUG:boto:path=/dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.423 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.423 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.423 DEBUG:boto:Path: /dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.423 DEBUG:boto:Data: 2024-04-12T03:58:51.423 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.423 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.423 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.423 DEBUG:boto:Params: {} 2024-04-12T03:58:51.423 DEBUG:boto:Token: None 2024-04-12T03:58:51.424 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.424 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:00+oaZY1cKp+OVHLPU8ZjsSaUxU= 2024-04-12T03:58:51.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:00+oaZY1cKp+OVHLPU8ZjsSaUxU=', 'Content-Length': '0'} 2024-04-12T03:58:51.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e897bc21e125888b-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.426 DEBUG:boto:path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.426 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.426 DEBUG:boto:path=/dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.426 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.426 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.426 DEBUG:boto:Path: /dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.426 DEBUG:boto:Data: 2024-04-12T03:58:51.427 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.427 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.427 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.427 DEBUG:boto:Params: {} 2024-04-12T03:58:51.427 DEBUG:boto:Token: None 2024-04-12T03:58:51.427 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.427 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:00+oaZY1cKp+OVHLPU8ZjsSaUxU= 2024-04-12T03:58:51.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:00+oaZY1cKp+OVHLPU8ZjsSaUxU=', 'Content-Length': '0'} 2024-04-12T03:58:51.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e2c694a32b958b3-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-04-12T03:58:51.430 DEBUG:boto:path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.430 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.430 DEBUG:boto:path=/dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.430 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.430 DEBUG:boto:Method: GET 2024-04-12T03:58:51.430 DEBUG:boto:Path: /dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.430 DEBUG:boto:Data: 2024-04-12T03:58:51.430 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.430 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.430 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.431 DEBUG:boto:Params: {} 2024-04-12T03:58:51.431 DEBUG:boto:Token: None 2024-04-12T03:58:51.431 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.431 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2Qe2jC9OvTL32bOWm40/nhVHTAk= 2024-04-12T03:58:51.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2Qe2jC9OvTL32bOWm40/nhVHTAk=', 'Content-Length': '0'} 2024-04-12T03:58:51.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007157c629adc840ff-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.433 DEBUG:boto:path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.433 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.433 DEBUG:boto:path=/dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.433 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.433 DEBUG:boto:Method: GET 2024-04-12T03:58:51.433 DEBUG:boto:Path: /dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.433 DEBUG:boto:Data: 2024-04-12T03:58:51.434 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.434 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.434 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.434 DEBUG:boto:Params: {} 2024-04-12T03:58:51.434 DEBUG:boto:Token: None 2024-04-12T03:58:51.434 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000060?versionId=null 2024-04-12T03:58:51.434 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2Qe2jC9OvTL32bOWm40/nhVHTAk= 2024-04-12T03:58:51.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2Qe2jC9OvTL32bOWm40/nhVHTAk=', 'Content-Length': '0'} 2024-04-12T03:58:51.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008168c22f83a8c083-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.436 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.436 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.437 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000060 2024-04-12T03:58:51.437 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000060 2024-04-12T03:58:51.437 DEBUG:boto:Method: GET 2024-04-12T03:58:51.437 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000060 2024-04-12T03:58:51.437 DEBUG:boto:Data: 2024-04-12T03:58:51.437 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.437 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.437 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.437 DEBUG:boto:Params: {} 2024-04-12T03:58:51.437 DEBUG:boto:Token: None 2024-04-12T03:58:51.438 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.438 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c4b3d218e092509-006618b17b-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.440 DEBUG:boto:b'dkyges-60obj000000000000000000601000falseobj000000000000000000602024-04-12T03:58:35.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.441 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.441 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.441 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000060 2024-04-12T03:58:51.441 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000060 2024-04-12T03:58:51.441 DEBUG:boto:Method: GET 2024-04-12T03:58:51.441 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000060 2024-04-12T03:58:51.441 DEBUG:boto:Data: 2024-04-12T03:58:51.441 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.442 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.442 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.442 DEBUG:boto:Params: {} 2024-04-12T03:58:51.442 DEBUG:boto:Token: None 2024-04-12T03:58:51.442 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.442 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af64b2d3baa0acbb-006618b17b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.447 DEBUG:boto:b'dkyges-60obj000000000000000000601000falseobj000000000000000000602024-04-12T03:58:35.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.447 DEBUG:boto:path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.447 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.448 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.448 DEBUG:boto:Path: /dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.448 DEBUG:boto:Data: 2024-04-12T03:58:51.448 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.448 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.448 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.448 DEBUG:boto:Params: {} 2024-04-12T03:58:51.448 DEBUG:boto:Token: None 2024-04-12T03:58:51.448 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.448 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0IrsgevIZaFNXOlDkqHKd/Q6Yw= 2024-04-12T03:58:51.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0IrsgevIZaFNXOlDkqHKd/Q6Yw=', 'Content-Length': '0'} 2024-04-12T03:58:51.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007e62aa5905b4e17-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.451 DEBUG:boto:path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.451 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.451 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.451 DEBUG:boto:Path: /dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.451 DEBUG:boto:Data: 2024-04-12T03:58:51.451 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.451 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.451 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.451 DEBUG:boto:Params: {} 2024-04-12T03:58:51.451 DEBUG:boto:Token: None 2024-04-12T03:58:51.451 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.452 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S0IrsgevIZaFNXOlDkqHKd/Q6Yw= 2024-04-12T03:58:51.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S0IrsgevIZaFNXOlDkqHKd/Q6Yw=', 'Content-Length': '0'} 2024-04-12T03:58:51.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dc0e9cfd16a6ce3-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2024-04-12T03:58:51.454 DEBUG:boto:path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.454 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.454 DEBUG:boto:Method: GET 2024-04-12T03:58:51.454 DEBUG:boto:Path: /dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.454 DEBUG:boto:Data: 2024-04-12T03:58:51.454 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.454 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.454 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.454 DEBUG:boto:Params: {} 2024-04-12T03:58:51.455 DEBUG:boto:Token: None 2024-04-12T03:58:51.455 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.455 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V36NBLhhBxzq9zRvPe2ahq/Qm8c= 2024-04-12T03:58:51.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V36NBLhhBxzq9zRvPe2ahq/Qm8c=', 'Content-Length': '0'} 2024-04-12T03:58:51.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f61d40489c5d7d0a-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.457 DEBUG:boto:path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.457 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.457 DEBUG:boto:Method: GET 2024-04-12T03:58:51.457 DEBUG:boto:Path: /dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.457 DEBUG:boto:Data: 2024-04-12T03:58:51.457 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.457 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.457 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.457 DEBUG:boto:Params: {} 2024-04-12T03:58:51.458 DEBUG:boto:Token: None 2024-04-12T03:58:51.458 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000060 2024-04-12T03:58:51.458 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V36NBLhhBxzq9zRvPe2ahq/Qm8c= 2024-04-12T03:58:51.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V36NBLhhBxzq9zRvPe2ahq/Qm8c=', 'Content-Length': '0'} 2024-04-12T03:58:51.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:35 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ec31e8fdd0b38e3-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.460 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.460 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.460 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000060 2024-04-12T03:58:51.460 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000060 2024-04-12T03:58:51.461 DEBUG:boto:Method: GET 2024-04-12T03:58:51.461 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000060 2024-04-12T03:58:51.461 DEBUG:boto:Data: 2024-04-12T03:58:51.461 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.461 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.461 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.461 DEBUG:boto:Params: {} 2024-04-12T03:58:51.461 DEBUG:boto:Token: None 2024-04-12T03:58:51.461 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.461 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4e38975b920eb70-006618b17b-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.464 DEBUG:boto:b'dkyges-60obj000000000000000000601000falseobj000000000000000000602024-04-12T03:58:35.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.464 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.464 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.465 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000060 2024-04-12T03:58:51.465 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000060 2024-04-12T03:58:51.465 DEBUG:boto:Method: GET 2024-04-12T03:58:51.465 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000060 2024-04-12T03:58:51.465 DEBUG:boto:Data: 2024-04-12T03:58:51.465 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.465 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.465 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.465 DEBUG:boto:Params: {} 2024-04-12T03:58:51.465 DEBUG:boto:Token: None 2024-04-12T03:58:51.465 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.466 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000459eb5a356d4381f-006618b17b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.470 DEBUG:boto:b'dkyges-60obj000000000000000000601000falseobj000000000000000000602024-04-12T03:58:35.664Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-04-12T03:58:51.470 DEBUG:boto:path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.470 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.470 DEBUG:boto:path=/dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.470 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.470 DEBUG:boto:Method: GET 2024-04-12T03:58:51.470 DEBUG:boto:Path: /dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.471 DEBUG:boto:Data: 2024-04-12T03:58:51.471 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.471 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.471 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.471 DEBUG:boto:Params: {} 2024-04-12T03:58:51.471 DEBUG:boto:Token: None 2024-04-12T03:58:51.471 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.471 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gkOAy6M/EXNc13d89W9LNuvSthA= 2024-04-12T03:58:51.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gkOAy6M/EXNc13d89W9LNuvSthA=', 'Content-Length': '0'} 2024-04-12T03:58:51.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005986f597f0b1285d-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.514 DEBUG:boto:path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.514 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.515 DEBUG:boto:path=/dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.515 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.515 DEBUG:boto:Method: GET 2024-04-12T03:58:51.515 DEBUG:boto:Path: /dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.515 DEBUG:boto:Data: 2024-04-12T03:58:51.515 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.515 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.515 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.515 DEBUG:boto:Params: {} 2024-04-12T03:58:51.515 DEBUG:boto:Token: None 2024-04-12T03:58:51.516 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.516 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gkOAy6M/EXNc13d89W9LNuvSthA= 2024-04-12T03:58:51.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gkOAy6M/EXNc13d89W9LNuvSthA=', 'Content-Length': '0'} 2024-04-12T03:58:51.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098dda00b98f88e2a-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.518 DEBUG:boto:path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.518 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.518 DEBUG:boto:path=/dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.518 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.519 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.519 DEBUG:boto:Path: /dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.519 DEBUG:boto:Data: 2024-04-12T03:58:51.519 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.519 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.519 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.519 DEBUG:boto:Params: {} 2024-04-12T03:58:51.519 DEBUG:boto:Token: None 2024-04-12T03:58:51.519 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.519 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8NKzvEauofG+iut+kEZZDcJ6HPI= 2024-04-12T03:58:51.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8NKzvEauofG+iut+kEZZDcJ6HPI=', 'Content-Length': '0'} 2024-04-12T03:58:51.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083e3a0111ab74abd-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.521 DEBUG:boto:path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.522 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.522 DEBUG:boto:path=/dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.522 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.522 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.522 DEBUG:boto:Path: /dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.522 DEBUG:boto:Data: 2024-04-12T03:58:51.522 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.522 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.522 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.522 DEBUG:boto:Params: {} 2024-04-12T03:58:51.523 DEBUG:boto:Token: None 2024-04-12T03:58:51.523 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.523 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8NKzvEauofG+iut+kEZZDcJ6HPI= 2024-04-12T03:58:51.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8NKzvEauofG+iut+kEZZDcJ6HPI=', 'Content-Length': '0'} 2024-04-12T03:58:51.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d24e201bc507044d-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-04-12T03:58:51.525 DEBUG:boto:path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.525 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.525 DEBUG:boto:path=/dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.526 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.526 DEBUG:boto:Method: GET 2024-04-12T03:58:51.526 DEBUG:boto:Path: /dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.526 DEBUG:boto:Data: 2024-04-12T03:58:51.526 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.526 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.526 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.526 DEBUG:boto:Params: {} 2024-04-12T03:58:51.526 DEBUG:boto:Token: None 2024-04-12T03:58:51.526 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.527 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gkOAy6M/EXNc13d89W9LNuvSthA= 2024-04-12T03:58:51.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gkOAy6M/EXNc13d89W9LNuvSthA=', 'Content-Length': '0'} 2024-04-12T03:58:51.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8c0caf8af9b33bd-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.529 DEBUG:boto:path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.529 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.529 DEBUG:boto:path=/dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.529 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.529 DEBUG:boto:Method: GET 2024-04-12T03:58:51.529 DEBUG:boto:Path: /dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.529 DEBUG:boto:Data: 2024-04-12T03:58:51.529 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.529 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.529 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.529 DEBUG:boto:Params: {} 2024-04-12T03:58:51.530 DEBUG:boto:Token: None 2024-04-12T03:58:51.530 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000061?versionId=null 2024-04-12T03:58:51.530 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gkOAy6M/EXNc13d89W9LNuvSthA= 2024-04-12T03:58:51.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gkOAy6M/EXNc13d89W9LNuvSthA=', 'Content-Length': '0'} 2024-04-12T03:58:51.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf457fc55a7bd0f9-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.532 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.532 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.532 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000061 2024-04-12T03:58:51.532 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000061 2024-04-12T03:58:51.532 DEBUG:boto:Method: GET 2024-04-12T03:58:51.533 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000061 2024-04-12T03:58:51.533 DEBUG:boto:Data: 2024-04-12T03:58:51.533 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.533 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.533 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.533 DEBUG:boto:Params: {} 2024-04-12T03:58:51.533 DEBUG:boto:Token: None 2024-04-12T03:58:51.533 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.533 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.536 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efc86ffe61436328-006618b17b-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.536 DEBUG:boto:b'dkyges-60obj000000000000000000611000falseobj000000000000000000612024-04-12T03:58:33.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.536 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.536 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.536 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000061 2024-04-12T03:58:51.537 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000061 2024-04-12T03:58:51.537 DEBUG:boto:Method: GET 2024-04-12T03:58:51.537 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000061 2024-04-12T03:58:51.537 DEBUG:boto:Data: 2024-04-12T03:58:51.537 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.537 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.537 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.537 DEBUG:boto:Params: {} 2024-04-12T03:58:51.537 DEBUG:boto:Token: None 2024-04-12T03:58:51.537 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.537 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043312512922e7a5d-006618b17b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.542 DEBUG:boto:b'dkyges-60obj000000000000000000611000falseobj000000000000000000612024-04-12T03:58:33.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.543 DEBUG:boto:path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.543 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.543 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.543 DEBUG:boto:Path: /dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.543 DEBUG:boto:Data: 2024-04-12T03:58:51.544 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.544 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.544 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.544 DEBUG:boto:Params: {} 2024-04-12T03:58:51.544 DEBUG:boto:Token: None 2024-04-12T03:58:51.544 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.544 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fLda1Yq5PjpPdeQdQkv/jCpWhPY= 2024-04-12T03:58:51.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fLda1Yq5PjpPdeQdQkv/jCpWhPY=', 'Content-Length': '0'} 2024-04-12T03:58:51.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d6e893c522f69e4-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.547 DEBUG:boto:path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.547 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.547 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.547 DEBUG:boto:Path: /dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.547 DEBUG:boto:Data: 2024-04-12T03:58:51.547 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.547 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.547 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.547 DEBUG:boto:Params: {} 2024-04-12T03:58:51.547 DEBUG:boto:Token: None 2024-04-12T03:58:51.548 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.548 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fLda1Yq5PjpPdeQdQkv/jCpWhPY= 2024-04-12T03:58:51.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fLda1Yq5PjpPdeQdQkv/jCpWhPY=', 'Content-Length': '0'} 2024-04-12T03:58:51.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c377d2dbd18c9eef-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2024-04-12T03:58:51.550 DEBUG:boto:path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.550 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.550 DEBUG:boto:Method: GET 2024-04-12T03:58:51.550 DEBUG:boto:Path: /dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.550 DEBUG:boto:Data: 2024-04-12T03:58:51.551 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.551 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.551 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.551 DEBUG:boto:Params: {} 2024-04-12T03:58:51.551 DEBUG:boto:Token: None 2024-04-12T03:58:51.551 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.551 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r1tX1NIuxidvydSTk7dbpShi88k= 2024-04-12T03:58:51.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r1tX1NIuxidvydSTk7dbpShi88k=', 'Content-Length': '0'} 2024-04-12T03:58:51.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cd705c45e1b9788-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.553 DEBUG:boto:path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.553 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.553 DEBUG:boto:Method: GET 2024-04-12T03:58:51.554 DEBUG:boto:Path: /dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.554 DEBUG:boto:Data: 2024-04-12T03:58:51.554 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.554 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.554 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.554 DEBUG:boto:Params: {} 2024-04-12T03:58:51.554 DEBUG:boto:Token: None 2024-04-12T03:58:51.554 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000061 2024-04-12T03:58:51.554 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r1tX1NIuxidvydSTk7dbpShi88k= 2024-04-12T03:58:51.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r1tX1NIuxidvydSTk7dbpShi88k=', 'Content-Length': '0'} 2024-04-12T03:58:51.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094bee12bac68bd30-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.602 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.602 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.602 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000061 2024-04-12T03:58:51.603 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000061 2024-04-12T03:58:51.603 DEBUG:boto:Method: GET 2024-04-12T03:58:51.603 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000061 2024-04-12T03:58:51.603 DEBUG:boto:Data: 2024-04-12T03:58:51.603 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.603 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.603 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.603 DEBUG:boto:Params: {} 2024-04-12T03:58:51.603 DEBUG:boto:Token: None 2024-04-12T03:58:51.603 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abe5fc5fe0a2d143-006618b17b-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.606 DEBUG:boto:b'dkyges-60obj000000000000000000611000falseobj000000000000000000612024-04-12T03:58:33.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.606 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.607 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.607 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000061 2024-04-12T03:58:51.607 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000061 2024-04-12T03:58:51.607 DEBUG:boto:Method: GET 2024-04-12T03:58:51.607 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000061 2024-04-12T03:58:51.607 DEBUG:boto:Data: 2024-04-12T03:58:51.607 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.607 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.607 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.607 DEBUG:boto:Params: {} 2024-04-12T03:58:51.607 DEBUG:boto:Token: None 2024-04-12T03:58:51.607 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.608 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008055655dcb9074b4-006618b17b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.611 DEBUG:boto:b'dkyges-60obj000000000000000000611000falseobj000000000000000000612024-04-12T03:58:33.774Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-04-12T03:58:51.612 DEBUG:boto:path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.612 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.612 DEBUG:boto:path=/dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.612 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.612 DEBUG:boto:Method: GET 2024-04-12T03:58:51.612 DEBUG:boto:Path: /dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.612 DEBUG:boto:Data: 2024-04-12T03:58:51.612 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.612 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.613 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.613 DEBUG:boto:Params: {} 2024-04-12T03:58:51.613 DEBUG:boto:Token: None 2024-04-12T03:58:51.613 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.613 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eu0zbAv+TKnYqSn3uqwx2PJCbTo= 2024-04-12T03:58:51.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eu0zbAv+TKnYqSn3uqwx2PJCbTo=', 'Content-Length': '0'} 2024-04-12T03:58:51.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f64653592d15c77-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.615 DEBUG:boto:path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.615 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.615 DEBUG:boto:path=/dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.615 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.615 DEBUG:boto:Method: GET 2024-04-12T03:58:51.615 DEBUG:boto:Path: /dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.616 DEBUG:boto:Data: 2024-04-12T03:58:51.616 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.616 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.616 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.616 DEBUG:boto:Params: {} 2024-04-12T03:58:51.616 DEBUG:boto:Token: None 2024-04-12T03:58:51.616 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.616 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eu0zbAv+TKnYqSn3uqwx2PJCbTo= 2024-04-12T03:58:51.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eu0zbAv+TKnYqSn3uqwx2PJCbTo=', 'Content-Length': '0'} 2024-04-12T03:58:51.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ff023307d760e2-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.618 DEBUG:boto:path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.619 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.619 DEBUG:boto:path=/dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.619 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.619 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.619 DEBUG:boto:Path: /dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.619 DEBUG:boto:Data: 2024-04-12T03:58:51.619 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.619 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.619 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.619 DEBUG:boto:Params: {} 2024-04-12T03:58:51.619 DEBUG:boto:Token: None 2024-04-12T03:58:51.620 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.620 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nVplEBydSq33vj3BH/pa5O1eseE= 2024-04-12T03:58:51.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nVplEBydSq33vj3BH/pa5O1eseE=', 'Content-Length': '0'} 2024-04-12T03:58:51.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4b0adda440c2941-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.622 DEBUG:boto:path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.622 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.622 DEBUG:boto:path=/dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.622 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.622 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.622 DEBUG:boto:Path: /dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.622 DEBUG:boto:Data: 2024-04-12T03:58:51.622 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.622 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.622 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.623 DEBUG:boto:Params: {} 2024-04-12T03:58:51.623 DEBUG:boto:Token: None 2024-04-12T03:58:51.623 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.623 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nVplEBydSq33vj3BH/pa5O1eseE= 2024-04-12T03:58:51.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nVplEBydSq33vj3BH/pa5O1eseE=', 'Content-Length': '0'} 2024-04-12T03:58:51.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f4181914e3dacbd-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-04-12T03:58:51.625 DEBUG:boto:path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.625 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.625 DEBUG:boto:path=/dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.626 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.626 DEBUG:boto:Method: GET 2024-04-12T03:58:51.626 DEBUG:boto:Path: /dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.626 DEBUG:boto:Data: 2024-04-12T03:58:51.626 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.626 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.626 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.626 DEBUG:boto:Params: {} 2024-04-12T03:58:51.626 DEBUG:boto:Token: None 2024-04-12T03:58:51.626 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.627 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eu0zbAv+TKnYqSn3uqwx2PJCbTo= 2024-04-12T03:58:51.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eu0zbAv+TKnYqSn3uqwx2PJCbTo=', 'Content-Length': '0'} 2024-04-12T03:58:51.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f947a6a439002ae2-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.670 DEBUG:boto:path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.670 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.670 DEBUG:boto:path=/dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.671 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.671 DEBUG:boto:Method: GET 2024-04-12T03:58:51.671 DEBUG:boto:Path: /dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.671 DEBUG:boto:Data: 2024-04-12T03:58:51.671 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.671 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.671 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.671 DEBUG:boto:Params: {} 2024-04-12T03:58:51.671 DEBUG:boto:Token: None 2024-04-12T03:58:51.671 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000062?versionId=null 2024-04-12T03:58:51.672 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eu0zbAv+TKnYqSn3uqwx2PJCbTo= 2024-04-12T03:58:51.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eu0zbAv+TKnYqSn3uqwx2PJCbTo=', 'Content-Length': '0'} 2024-04-12T03:58:51.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaa0a91008f7d16d-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.674 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.674 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.674 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000062 2024-04-12T03:58:51.674 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000062 2024-04-12T03:58:51.674 DEBUG:boto:Method: GET 2024-04-12T03:58:51.674 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000062 2024-04-12T03:58:51.674 DEBUG:boto:Data: 2024-04-12T03:58:51.675 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.675 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.675 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.675 DEBUG:boto:Params: {} 2024-04-12T03:58:51.675 DEBUG:boto:Token: None 2024-04-12T03:58:51.675 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.675 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081ea10e8d4522d81-006618b17b-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.678 DEBUG:boto:b'dkyges-60obj000000000000000000621000falseobj000000000000000000622024-04-12T03:58:34.796Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.678 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.679 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.679 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000062 2024-04-12T03:58:51.679 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000062 2024-04-12T03:58:51.679 DEBUG:boto:Method: GET 2024-04-12T03:58:51.679 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000062 2024-04-12T03:58:51.679 DEBUG:boto:Data: 2024-04-12T03:58:51.679 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.679 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.679 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.679 DEBUG:boto:Params: {} 2024-04-12T03:58:51.679 DEBUG:boto:Token: None 2024-04-12T03:58:51.680 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.680 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082ad98a01b4a4fd5-006618b17b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.684 DEBUG:boto:b'dkyges-60obj000000000000000000621000falseobj000000000000000000622024-04-12T03:58:34.796Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.684 DEBUG:boto:path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.685 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.685 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.685 DEBUG:boto:Path: /dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.685 DEBUG:boto:Data: 2024-04-12T03:58:51.685 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.685 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.685 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.685 DEBUG:boto:Params: {} 2024-04-12T03:58:51.685 DEBUG:boto:Token: None 2024-04-12T03:58:51.686 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.686 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9y7iLCXcuIYoLkH/oaWhKqjTgfw= 2024-04-12T03:58:51.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9y7iLCXcuIYoLkH/oaWhKqjTgfw=', 'Content-Length': '0'} 2024-04-12T03:58:51.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e77dc04c45eb50-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.688 DEBUG:boto:path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.688 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.688 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.688 DEBUG:boto:Path: /dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.688 DEBUG:boto:Data: 2024-04-12T03:58:51.688 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.688 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.688 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.688 DEBUG:boto:Params: {} 2024-04-12T03:58:51.689 DEBUG:boto:Token: None 2024-04-12T03:58:51.689 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.689 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9y7iLCXcuIYoLkH/oaWhKqjTgfw= 2024-04-12T03:58:51.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9y7iLCXcuIYoLkH/oaWhKqjTgfw=', 'Content-Length': '0'} 2024-04-12T03:58:51.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e1713d001cf8aa3-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2024-04-12T03:58:51.691 DEBUG:boto:path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.691 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.691 DEBUG:boto:Method: GET 2024-04-12T03:58:51.691 DEBUG:boto:Path: /dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.691 DEBUG:boto:Data: 2024-04-12T03:58:51.692 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.692 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.692 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.692 DEBUG:boto:Params: {} 2024-04-12T03:58:51.692 DEBUG:boto:Token: None 2024-04-12T03:58:51.692 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.692 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l9tz1DnTw30lfufVifmPy0af2lk= 2024-04-12T03:58:51.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l9tz1DnTw30lfufVifmPy0af2lk=', 'Content-Length': '0'} 2024-04-12T03:58:51.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000256403f8fcf23aa0-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.694 DEBUG:boto:path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.694 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.695 DEBUG:boto:Method: GET 2024-04-12T03:58:51.695 DEBUG:boto:Path: /dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.695 DEBUG:boto:Data: 2024-04-12T03:58:51.695 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.695 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.695 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.695 DEBUG:boto:Params: {} 2024-04-12T03:58:51.695 DEBUG:boto:Token: None 2024-04-12T03:58:51.695 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000062 2024-04-12T03:58:51.695 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l9tz1DnTw30lfufVifmPy0af2lk= 2024-04-12T03:58:51.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l9tz1DnTw30lfufVifmPy0af2lk=', 'Content-Length': '0'} 2024-04-12T03:58:51.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de22b23a773ee8c0-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.697 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.698 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.698 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000062 2024-04-12T03:58:51.698 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000062 2024-04-12T03:58:51.698 DEBUG:boto:Method: GET 2024-04-12T03:58:51.698 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000062 2024-04-12T03:58:51.698 DEBUG:boto:Data: 2024-04-12T03:58:51.698 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.698 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.698 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.698 DEBUG:boto:Params: {} 2024-04-12T03:58:51.699 DEBUG:boto:Token: None 2024-04-12T03:58:51.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.699 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001131493ab0b86db1-006618b17b-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.701 DEBUG:boto:b'dkyges-60obj000000000000000000621000falseobj000000000000000000622024-04-12T03:58:34.796Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.702 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.702 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.702 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000062 2024-04-12T03:58:51.702 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000062 2024-04-12T03:58:51.702 DEBUG:boto:Method: GET 2024-04-12T03:58:51.702 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000062 2024-04-12T03:58:51.702 DEBUG:boto:Data: 2024-04-12T03:58:51.703 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.703 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.703 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.703 DEBUG:boto:Params: {} 2024-04-12T03:58:51.703 DEBUG:boto:Token: None 2024-04-12T03:58:51.703 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.703 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.707 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2f4f175f25fa0cd-006618b17b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.708 DEBUG:boto:b'dkyges-60obj000000000000000000621000falseobj000000000000000000622024-04-12T03:58:34.796Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-04-12T03:58:51.708 DEBUG:boto:path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.708 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.708 DEBUG:boto:path=/dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.708 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.708 DEBUG:boto:Method: GET 2024-04-12T03:58:51.709 DEBUG:boto:Path: /dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.709 DEBUG:boto:Data: 2024-04-12T03:58:51.709 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.709 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.709 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.709 DEBUG:boto:Params: {} 2024-04-12T03:58:51.709 DEBUG:boto:Token: None 2024-04-12T03:58:51.709 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.709 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fTCqqy/OvamFGioTpyvZPSp34eE= 2024-04-12T03:58:51.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fTCqqy/OvamFGioTpyvZPSp34eE=', 'Content-Length': '0'} 2024-04-12T03:58:51.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bba22a48388f19c-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.712 DEBUG:boto:path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.712 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.712 DEBUG:boto:path=/dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.712 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.712 DEBUG:boto:Method: GET 2024-04-12T03:58:51.712 DEBUG:boto:Path: /dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.712 DEBUG:boto:Data: 2024-04-12T03:58:51.712 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.712 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.713 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.713 DEBUG:boto:Params: {} 2024-04-12T03:58:51.713 DEBUG:boto:Token: None 2024-04-12T03:58:51.713 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.713 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fTCqqy/OvamFGioTpyvZPSp34eE= 2024-04-12T03:58:51.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fTCqqy/OvamFGioTpyvZPSp34eE=', 'Content-Length': '0'} 2024-04-12T03:58:51.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a90e3e2527f23df-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.762 DEBUG:boto:path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.763 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.763 DEBUG:boto:path=/dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.763 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.763 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.763 DEBUG:boto:Path: /dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.763 DEBUG:boto:Data: 2024-04-12T03:58:51.763 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.763 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.763 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.763 DEBUG:boto:Params: {} 2024-04-12T03:58:51.764 DEBUG:boto:Token: None 2024-04-12T03:58:51.764 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.764 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QpafnNHj7BIwq534aF8+AY2wYOc= 2024-04-12T03:58:51.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QpafnNHj7BIwq534aF8+AY2wYOc=', 'Content-Length': '0'} 2024-04-12T03:58:51.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041a16aeb126072d8-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.766 DEBUG:boto:path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.767 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.767 DEBUG:boto:path=/dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.767 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.767 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.767 DEBUG:boto:Path: /dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.767 DEBUG:boto:Data: 2024-04-12T03:58:51.767 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.767 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.767 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.767 DEBUG:boto:Params: {} 2024-04-12T03:58:51.768 DEBUG:boto:Token: None 2024-04-12T03:58:51.768 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.768 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QpafnNHj7BIwq534aF8+AY2wYOc= 2024-04-12T03:58:51.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QpafnNHj7BIwq534aF8+AY2wYOc=', 'Content-Length': '0'} 2024-04-12T03:58:51.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b440b5bf37b84df0-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-04-12T03:58:51.770 DEBUG:boto:path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.771 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.771 DEBUG:boto:path=/dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.771 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.771 DEBUG:boto:Method: GET 2024-04-12T03:58:51.771 DEBUG:boto:Path: /dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.771 DEBUG:boto:Data: 2024-04-12T03:58:51.771 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.771 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.771 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.771 DEBUG:boto:Params: {} 2024-04-12T03:58:51.772 DEBUG:boto:Token: None 2024-04-12T03:58:51.772 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.772 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fTCqqy/OvamFGioTpyvZPSp34eE= 2024-04-12T03:58:51.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fTCqqy/OvamFGioTpyvZPSp34eE=', 'Content-Length': '0'} 2024-04-12T03:58:51.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd7b05ffd9a51f42-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.774 DEBUG:boto:path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.774 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.774 DEBUG:boto:path=/dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.775 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.775 DEBUG:boto:Method: GET 2024-04-12T03:58:51.775 DEBUG:boto:Path: /dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.775 DEBUG:boto:Data: 2024-04-12T03:58:51.775 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.775 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.775 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.775 DEBUG:boto:Params: {} 2024-04-12T03:58:51.775 DEBUG:boto:Token: None 2024-04-12T03:58:51.775 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000076?versionId=null 2024-04-12T03:58:51.776 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fTCqqy/OvamFGioTpyvZPSp34eE= 2024-04-12T03:58:51.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fTCqqy/OvamFGioTpyvZPSp34eE=', 'Content-Length': '0'} 2024-04-12T03:58:51.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1ad7b7b4258f9b4-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.778 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.778 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.778 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000076 2024-04-12T03:58:51.778 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000076 2024-04-12T03:58:51.778 DEBUG:boto:Method: GET 2024-04-12T03:58:51.779 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000076 2024-04-12T03:58:51.779 DEBUG:boto:Data: 2024-04-12T03:58:51.779 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.779 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.779 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.779 DEBUG:boto:Params: {} 2024-04-12T03:58:51.779 DEBUG:boto:Token: None 2024-04-12T03:58:51.779 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.780 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000260bd4110cf0aa3c-006618b17b-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.782 DEBUG:boto:b'dkyges-60obj000000000000000000761000falseobj000000000000000000762024-04-12T03:58:30.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.783 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.783 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.783 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000076 2024-04-12T03:58:51.783 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000076 2024-04-12T03:58:51.783 DEBUG:boto:Method: GET 2024-04-12T03:58:51.783 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000076 2024-04-12T03:58:51.783 DEBUG:boto:Data: 2024-04-12T03:58:51.783 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.784 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.784 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.784 DEBUG:boto:Params: {} 2024-04-12T03:58:51.784 DEBUG:boto:Token: None 2024-04-12T03:58:51.784 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.784 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003288d6312129b44b-006618b17b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.788 DEBUG:boto:b'dkyges-60obj000000000000000000761000falseobj000000000000000000762024-04-12T03:58:30.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.789 DEBUG:boto:path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.789 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.789 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.789 DEBUG:boto:Path: /dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.789 DEBUG:boto:Data: 2024-04-12T03:58:51.789 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.789 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.789 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.790 DEBUG:boto:Params: {} 2024-04-12T03:58:51.790 DEBUG:boto:Token: None 2024-04-12T03:58:51.790 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.790 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/dqQPwOSdANCBLksa4kkUta4/w8= 2024-04-12T03:58:51.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/dqQPwOSdANCBLksa4kkUta4/w8=', 'Content-Length': '0'} 2024-04-12T03:58:51.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea93d671f57c15b4-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.792 DEBUG:boto:path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.792 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.792 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.793 DEBUG:boto:Path: /dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.793 DEBUG:boto:Data: 2024-04-12T03:58:51.793 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.793 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.793 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.793 DEBUG:boto:Params: {} 2024-04-12T03:58:51.793 DEBUG:boto:Token: None 2024-04-12T03:58:51.793 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.793 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/dqQPwOSdANCBLksa4kkUta4/w8= 2024-04-12T03:58:51.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/dqQPwOSdANCBLksa4kkUta4/w8=', 'Content-Length': '0'} 2024-04-12T03:58:51.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000959a4175ae18b31b-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2024-04-12T03:58:51.796 DEBUG:boto:path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.796 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.796 DEBUG:boto:Method: GET 2024-04-12T03:58:51.796 DEBUG:boto:Path: /dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.796 DEBUG:boto:Data: 2024-04-12T03:58:51.796 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.796 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.796 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.797 DEBUG:boto:Params: {} 2024-04-12T03:58:51.797 DEBUG:boto:Token: None 2024-04-12T03:58:51.797 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.797 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WWTUfT4z2mVJyrTIN5DwztVqhU= 2024-04-12T03:58:51.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WWTUfT4z2mVJyrTIN5DwztVqhU=', 'Content-Length': '0'} 2024-04-12T03:58:51.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b007776934460119-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.799 DEBUG:boto:path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.799 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.800 DEBUG:boto:Method: GET 2024-04-12T03:58:51.800 DEBUG:boto:Path: /dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.800 DEBUG:boto:Data: 2024-04-12T03:58:51.800 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.800 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.800 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.800 DEBUG:boto:Params: {} 2024-04-12T03:58:51.800 DEBUG:boto:Token: None 2024-04-12T03:58:51.800 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000076 2024-04-12T03:58:51.801 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5WWTUfT4z2mVJyrTIN5DwztVqhU= 2024-04-12T03:58:51.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5WWTUfT4z2mVJyrTIN5DwztVqhU=', 'Content-Length': '0'} 2024-04-12T03:58:51.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd43a4131286db96-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.850 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.851 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.851 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000076 2024-04-12T03:58:51.851 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000076 2024-04-12T03:58:51.851 DEBUG:boto:Method: GET 2024-04-12T03:58:51.851 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000076 2024-04-12T03:58:51.851 DEBUG:boto:Data: 2024-04-12T03:58:51.851 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.851 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.851 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.851 DEBUG:boto:Params: {} 2024-04-12T03:58:51.852 DEBUG:boto:Token: None 2024-04-12T03:58:51.852 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.852 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000444a966ee88b5415-006618b17b-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.855 DEBUG:boto:b'dkyges-60obj000000000000000000761000falseobj000000000000000000762024-04-12T03:58:30.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.855 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.855 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.855 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000076 2024-04-12T03:58:51.855 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000076 2024-04-12T03:58:51.855 DEBUG:boto:Method: GET 2024-04-12T03:58:51.856 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000076 2024-04-12T03:58:51.856 DEBUG:boto:Data: 2024-04-12T03:58:51.856 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.856 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.856 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.856 DEBUG:boto:Params: {} 2024-04-12T03:58:51.856 DEBUG:boto:Token: None 2024-04-12T03:58:51.856 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.856 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a9722fe669ab2c4-006618b17b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.861 DEBUG:boto:b'dkyges-60obj000000000000000000761000falseobj000000000000000000762024-04-12T03:58:30.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-04-12T03:58:51.862 DEBUG:boto:path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.862 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.862 DEBUG:boto:path=/dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.862 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.862 DEBUG:boto:Method: GET 2024-04-12T03:58:51.862 DEBUG:boto:Path: /dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.862 DEBUG:boto:Data: 2024-04-12T03:58:51.862 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.862 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.862 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.863 DEBUG:boto:Params: {} 2024-04-12T03:58:51.863 DEBUG:boto:Token: None 2024-04-12T03:58:51.863 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.863 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Mr3yUUoz+n/RPcE75+fH3L2Ns8c= 2024-04-12T03:58:51.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Mr3yUUoz+n/RPcE75+fH3L2Ns8c=', 'Content-Length': '0'} 2024-04-12T03:58:51.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b254e6667e73d13d-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.906 DEBUG:boto:path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.906 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.907 DEBUG:boto:path=/dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.907 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.907 DEBUG:boto:Method: GET 2024-04-12T03:58:51.907 DEBUG:boto:Path: /dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.907 DEBUG:boto:Data: 2024-04-12T03:58:51.907 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.907 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.907 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.907 DEBUG:boto:Params: {} 2024-04-12T03:58:51.907 DEBUG:boto:Token: None 2024-04-12T03:58:51.908 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.908 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Mr3yUUoz+n/RPcE75+fH3L2Ns8c= 2024-04-12T03:58:51.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Mr3yUUoz+n/RPcE75+fH3L2Ns8c=', 'Content-Length': '0'} 2024-04-12T03:58:51.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b50b3f4996a5aea3-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.910 DEBUG:boto:path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.910 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.910 DEBUG:boto:path=/dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.910 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.910 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.910 DEBUG:boto:Path: /dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.910 DEBUG:boto:Data: 2024-04-12T03:58:51.911 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.911 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.911 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.911 DEBUG:boto:Params: {} 2024-04-12T03:58:51.911 DEBUG:boto:Token: None 2024-04-12T03:58:51.911 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.911 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oSvXWddTPBuMJ+CTvoLwyec3+7g= 2024-04-12T03:58:51.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oSvXWddTPBuMJ+CTvoLwyec3+7g=', 'Content-Length': '0'} 2024-04-12T03:58:51.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3e5e618253590a7-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.913 DEBUG:boto:path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.913 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.913 DEBUG:boto:path=/dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.914 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.914 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.914 DEBUG:boto:Path: /dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.914 DEBUG:boto:Data: 2024-04-12T03:58:51.914 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.914 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.914 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.914 DEBUG:boto:Params: {} 2024-04-12T03:58:51.914 DEBUG:boto:Token: None 2024-04-12T03:58:51.914 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.915 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oSvXWddTPBuMJ+CTvoLwyec3+7g= 2024-04-12T03:58:51.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oSvXWddTPBuMJ+CTvoLwyec3+7g=', 'Content-Length': '0'} 2024-04-12T03:58:51.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009144f72a486c9256-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-04-12T03:58:51.917 DEBUG:boto:path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.917 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.917 DEBUG:boto:path=/dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.917 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.917 DEBUG:boto:Method: GET 2024-04-12T03:58:51.917 DEBUG:boto:Path: /dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.917 DEBUG:boto:Data: 2024-04-12T03:58:51.917 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.917 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.917 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.917 DEBUG:boto:Params: {} 2024-04-12T03:58:51.918 DEBUG:boto:Token: None 2024-04-12T03:58:51.918 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.918 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Mr3yUUoz+n/RPcE75+fH3L2Ns8c= 2024-04-12T03:58:51.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Mr3yUUoz+n/RPcE75+fH3L2Ns8c=', 'Content-Length': '0'} 2024-04-12T03:58:51.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cf2d39680788287-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.920 DEBUG:boto:path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.920 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.920 DEBUG:boto:path=/dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.920 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.921 DEBUG:boto:Method: GET 2024-04-12T03:58:51.921 DEBUG:boto:Path: /dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.921 DEBUG:boto:Data: 2024-04-12T03:58:51.921 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.921 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.921 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.921 DEBUG:boto:Params: {} 2024-04-12T03:58:51.921 DEBUG:boto:Token: None 2024-04-12T03:58:51.921 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000133?versionId=null 2024-04-12T03:58:51.921 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Mr3yUUoz+n/RPcE75+fH3L2Ns8c= 2024-04-12T03:58:51.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Mr3yUUoz+n/RPcE75+fH3L2Ns8c=', 'Content-Length': '0'} 2024-04-12T03:58:51.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007540eba21d5b8ee-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.924 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.924 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.924 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000133 2024-04-12T03:58:51.924 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000133 2024-04-12T03:58:51.924 DEBUG:boto:Method: GET 2024-04-12T03:58:51.924 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000133 2024-04-12T03:58:51.924 DEBUG:boto:Data: 2024-04-12T03:58:51.924 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.924 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.925 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.925 DEBUG:boto:Params: {} 2024-04-12T03:58:51.925 DEBUG:boto:Token: None 2024-04-12T03:58:51.925 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.925 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066772dd59ae9318f-006618b17b-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.928 DEBUG:boto:b'dkyges-60obj000000000000000001331000falseobj000000000000000001332024-04-12T03:58:33.696Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.928 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.928 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.928 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000133 2024-04-12T03:58:51.929 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000133 2024-04-12T03:58:51.929 DEBUG:boto:Method: GET 2024-04-12T03:58:51.929 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000133 2024-04-12T03:58:51.929 DEBUG:boto:Data: 2024-04-12T03:58:51.929 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.929 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.929 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.929 DEBUG:boto:Params: {} 2024-04-12T03:58:51.929 DEBUG:boto:Token: None 2024-04-12T03:58:51.929 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.929 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000504b3542b1ffc5ad-006618b17b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.933 DEBUG:boto:b'dkyges-60obj000000000000000001331000falseobj000000000000000001332024-04-12T03:58:33.696Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.934 DEBUG:boto:path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.934 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.934 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.934 DEBUG:boto:Path: /dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.934 DEBUG:boto:Data: 2024-04-12T03:58:51.934 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.934 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.935 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.935 DEBUG:boto:Params: {} 2024-04-12T03:58:51.935 DEBUG:boto:Token: None 2024-04-12T03:58:51.935 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.935 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6+pfqLzsNjFNprlxyr3LFE5/8gQ= 2024-04-12T03:58:51.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6+pfqLzsNjFNprlxyr3LFE5/8gQ=', 'Content-Length': '0'} 2024-04-12T03:58:51.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc23293a80463c4a-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.937 DEBUG:boto:path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.937 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.937 DEBUG:boto:Method: HEAD 2024-04-12T03:58:51.938 DEBUG:boto:Path: /dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.938 DEBUG:boto:Data: 2024-04-12T03:58:51.938 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.938 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.938 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.938 DEBUG:boto:Params: {} 2024-04-12T03:58:51.938 DEBUG:boto:Token: None 2024-04-12T03:58:51.938 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.938 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6+pfqLzsNjFNprlxyr3LFE5/8gQ= 2024-04-12T03:58:51.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6+pfqLzsNjFNprlxyr3LFE5/8gQ=', 'Content-Length': '0'} 2024-04-12T03:58:51.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075530b60e8d5ada0-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2024-04-12T03:58:51.940 DEBUG:boto:path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.941 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.941 DEBUG:boto:Method: GET 2024-04-12T03:58:51.941 DEBUG:boto:Path: /dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.941 DEBUG:boto:Data: 2024-04-12T03:58:51.941 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.941 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.941 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.941 DEBUG:boto:Params: {} 2024-04-12T03:58:51.941 DEBUG:boto:Token: None 2024-04-12T03:58:51.942 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.942 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Rr56RG9CX7TWmSzU7nryrt9iRoc= 2024-04-12T03:58:51.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Rr56RG9CX7TWmSzU7nryrt9iRoc=', 'Content-Length': '0'} 2024-04-12T03:58:51.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051bf2701f238a706-006618b17b-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.944 DEBUG:boto:path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.944 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.944 DEBUG:boto:Method: GET 2024-04-12T03:58:51.944 DEBUG:boto:Path: /dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.944 DEBUG:boto:Data: 2024-04-12T03:58:51.944 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.944 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.945 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.945 DEBUG:boto:Params: {} 2024-04-12T03:58:51.945 DEBUG:boto:Token: None 2024-04-12T03:58:51.945 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/obj00000000000000000133 2024-04-12T03:58:51.945 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Rr56RG9CX7TWmSzU7nryrt9iRoc= 2024-04-12T03:58:51.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Rr56RG9CX7TWmSzU7nryrt9iRoc=', 'Content-Length': '0'} 2024-04-12T03:58:51.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000523b2aed8c134c18-006618b17b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.990 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.990 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.991 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000133 2024-04-12T03:58:51.991 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000133 2024-04-12T03:58:51.991 DEBUG:boto:Method: GET 2024-04-12T03:58:51.991 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000133 2024-04-12T03:58:51.991 DEBUG:boto:Data: 2024-04-12T03:58:51.991 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.991 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:51.991 DEBUG:boto:Port: 8001 2024-04-12T03:58:51.991 DEBUG:boto:Params: {} 2024-04-12T03:58:51.991 DEBUG:boto:Token: None 2024-04-12T03:58:51.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.992 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:51.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f8ec0770bc1df99-006618b17c-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:51.994 DEBUG:boto:b'dkyges-60obj000000000000000001331000falseobj000000000000000001332024-04-12T03:58:33.696Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:51.995 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:51.995 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:51.995 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000133 2024-04-12T03:58:51.995 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000133 2024-04-12T03:58:51.995 DEBUG:boto:Method: GET 2024-04-12T03:58:51.995 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000133 2024-04-12T03:58:51.995 DEBUG:boto:Data: 2024-04-12T03:58:51.996 DEBUG:boto:Headers: {} 2024-04-12T03:58:51.996 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:51.996 DEBUG:boto:Port: 8000 2024-04-12T03:58:51.996 DEBUG:boto:Params: {} 2024-04-12T03:58:51.996 DEBUG:boto:Token: None 2024-04-12T03:58:51.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:51 GMT /dkyges-60/ 2024-04-12T03:58:51.996 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY= 2024-04-12T03:58:51.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uMNHB6apRorKxkSKe5rRdC+dtnY=', 'Content-Length': '0'} 2024-04-12T03:58:52.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ac93e6ab91e2fa6-006618b17c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.000 DEBUG:boto:b'dkyges-60obj000000000000000001331000falseobj000000000000000001332024-04-12T03:58:33.696Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-04-12T03:58:52.001 DEBUG:boto:path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.001 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.001 DEBUG:boto:path=/dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.001 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.001 DEBUG:boto:Method: GET 2024-04-12T03:58:52.001 DEBUG:boto:Path: /dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.001 DEBUG:boto:Data: 2024-04-12T03:58:52.001 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.002 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.002 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.002 DEBUG:boto:Params: {} 2024-04-12T03:58:52.002 DEBUG:boto:Token: None 2024-04-12T03:58:52.002 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.002 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hjy+hCYIj1Wf/jtn2NwZoKrPmO4= 2024-04-12T03:58:52.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hjy+hCYIj1Wf/jtn2NwZoKrPmO4=', 'Content-Length': '0'} 2024-04-12T03:58:52.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f0f7f7668b6ea73-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.050 DEBUG:boto:path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.051 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.051 DEBUG:boto:path=/dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.051 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.051 DEBUG:boto:Method: GET 2024-04-12T03:58:52.051 DEBUG:boto:Path: /dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.051 DEBUG:boto:Data: 2024-04-12T03:58:52.051 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.051 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.051 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.051 DEBUG:boto:Params: {} 2024-04-12T03:58:52.052 DEBUG:boto:Token: None 2024-04-12T03:58:52.052 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.052 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hjy+hCYIj1Wf/jtn2NwZoKrPmO4= 2024-04-12T03:58:52.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hjy+hCYIj1Wf/jtn2NwZoKrPmO4=', 'Content-Length': '0'} 2024-04-12T03:58:52.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ac009a8c0750569-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.055 DEBUG:boto:path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.055 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.055 DEBUG:boto:path=/dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.055 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.055 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.055 DEBUG:boto:Path: /dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.055 DEBUG:boto:Data: 2024-04-12T03:58:52.055 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.055 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.056 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.056 DEBUG:boto:Params: {} 2024-04-12T03:58:52.056 DEBUG:boto:Token: None 2024-04-12T03:58:52.056 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.056 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OSQl2NpCwkbYCUlZPEES2MfxSfQ= 2024-04-12T03:58:52.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OSQl2NpCwkbYCUlZPEES2MfxSfQ=', 'Content-Length': '0'} 2024-04-12T03:58:52.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d725d2c08bcd292-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.058 DEBUG:boto:path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.058 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.059 DEBUG:boto:path=/dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.059 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.059 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.059 DEBUG:boto:Path: /dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.059 DEBUG:boto:Data: 2024-04-12T03:58:52.059 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.059 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.059 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.059 DEBUG:boto:Params: {} 2024-04-12T03:58:52.059 DEBUG:boto:Token: None 2024-04-12T03:58:52.059 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.060 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OSQl2NpCwkbYCUlZPEES2MfxSfQ= 2024-04-12T03:58:52.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OSQl2NpCwkbYCUlZPEES2MfxSfQ=', 'Content-Length': '0'} 2024-04-12T03:58:52.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc6e0b2547873e67-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-04-12T03:58:52.062 DEBUG:boto:path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.062 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.062 DEBUG:boto:path=/dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.062 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.062 DEBUG:boto:Method: GET 2024-04-12T03:58:52.062 DEBUG:boto:Path: /dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.062 DEBUG:boto:Data: 2024-04-12T03:58:52.062 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.062 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.063 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.063 DEBUG:boto:Params: {} 2024-04-12T03:58:52.063 DEBUG:boto:Token: None 2024-04-12T03:58:52.063 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.063 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hjy+hCYIj1Wf/jtn2NwZoKrPmO4= 2024-04-12T03:58:52.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hjy+hCYIj1Wf/jtn2NwZoKrPmO4=', 'Content-Length': '0'} 2024-04-12T03:58:52.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c8a95aeaaa83083-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.065 DEBUG:boto:path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.065 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.065 DEBUG:boto:path=/dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.065 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.065 DEBUG:boto:Method: GET 2024-04-12T03:58:52.066 DEBUG:boto:Path: /dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.066 DEBUG:boto:Data: 2024-04-12T03:58:52.066 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.066 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.066 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.066 DEBUG:boto:Params: {} 2024-04-12T03:58:52.066 DEBUG:boto:Token: None 2024-04-12T03:58:52.066 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000162?versionId=null 2024-04-12T03:58:52.066 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hjy+hCYIj1Wf/jtn2NwZoKrPmO4= 2024-04-12T03:58:52.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hjy+hCYIj1Wf/jtn2NwZoKrPmO4=', 'Content-Length': '0'} 2024-04-12T03:58:52.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a345101b89481a4-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.069 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.069 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.069 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000162 2024-04-12T03:58:52.069 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000162 2024-04-12T03:58:52.069 DEBUG:boto:Method: GET 2024-04-12T03:58:52.069 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000162 2024-04-12T03:58:52.069 DEBUG:boto:Data: 2024-04-12T03:58:52.069 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.069 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.070 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.070 DEBUG:boto:Params: {} 2024-04-12T03:58:52.070 DEBUG:boto:Token: None 2024-04-12T03:58:52.070 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.070 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f3db39d9ac2348e-006618b17c-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.073 DEBUG:boto:b'dkyges-60obj000000000000000001621000falseobj000000000000000001622024-04-12T03:58:33.823Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.073 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.074 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.074 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000162 2024-04-12T03:58:52.074 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000162 2024-04-12T03:58:52.074 DEBUG:boto:Method: GET 2024-04-12T03:58:52.074 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000162 2024-04-12T03:58:52.074 DEBUG:boto:Data: 2024-04-12T03:58:52.074 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.074 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.075 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.075 DEBUG:boto:Params: {} 2024-04-12T03:58:52.075 DEBUG:boto:Token: None 2024-04-12T03:58:52.075 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.075 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047ddb6d23aa59f03-006618b17c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.079 DEBUG:boto:b'dkyges-60obj000000000000000001621000falseobj000000000000000001622024-04-12T03:58:33.823Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.080 DEBUG:boto:path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.080 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.080 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.080 DEBUG:boto:Path: /dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.080 DEBUG:boto:Data: 2024-04-12T03:58:52.080 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.080 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.080 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.080 DEBUG:boto:Params: {} 2024-04-12T03:58:52.080 DEBUG:boto:Token: None 2024-04-12T03:58:52.081 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.081 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eEO+A1rgiYH8OxfJV41LbQA5Ipc= 2024-04-12T03:58:52.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eEO+A1rgiYH8OxfJV41LbQA5Ipc=', 'Content-Length': '0'} 2024-04-12T03:58:52.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000b5aca52bdc9912-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.083 DEBUG:boto:path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.083 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.083 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.083 DEBUG:boto:Path: /dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.083 DEBUG:boto:Data: 2024-04-12T03:58:52.083 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.083 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.083 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.083 DEBUG:boto:Params: {} 2024-04-12T03:58:52.084 DEBUG:boto:Token: None 2024-04-12T03:58:52.084 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.084 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eEO+A1rgiYH8OxfJV41LbQA5Ipc= 2024-04-12T03:58:52.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eEO+A1rgiYH8OxfJV41LbQA5Ipc=', 'Content-Length': '0'} 2024-04-12T03:58:52.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f30d3a6606cb974-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2024-04-12T03:58:52.086 DEBUG:boto:path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.086 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.086 DEBUG:boto:Method: GET 2024-04-12T03:58:52.086 DEBUG:boto:Path: /dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.087 DEBUG:boto:Data: 2024-04-12T03:58:52.087 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.087 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.087 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.087 DEBUG:boto:Params: {} 2024-04-12T03:58:52.087 DEBUG:boto:Token: None 2024-04-12T03:58:52.087 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.087 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q/+sd5I7fBc/cwZAVslPla2Nwio= 2024-04-12T03:58:52.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q/+sd5I7fBc/cwZAVslPla2Nwio=', 'Content-Length': '0'} 2024-04-12T03:58:52.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0e24e8b6f9ed973-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.089 DEBUG:boto:path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.089 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.090 DEBUG:boto:Method: GET 2024-04-12T03:58:52.090 DEBUG:boto:Path: /dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.090 DEBUG:boto:Data: 2024-04-12T03:58:52.090 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.090 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.090 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.090 DEBUG:boto:Params: {} 2024-04-12T03:58:52.090 DEBUG:boto:Token: None 2024-04-12T03:58:52.090 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000162 2024-04-12T03:58:52.091 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q/+sd5I7fBc/cwZAVslPla2Nwio= 2024-04-12T03:58:52.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q/+sd5I7fBc/cwZAVslPla2Nwio=', 'Content-Length': '0'} 2024-04-12T03:58:52.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f090503962eedd4-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.134 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.134 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.135 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000162 2024-04-12T03:58:52.135 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000162 2024-04-12T03:58:52.135 DEBUG:boto:Method: GET 2024-04-12T03:58:52.135 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000162 2024-04-12T03:58:52.135 DEBUG:boto:Data: 2024-04-12T03:58:52.135 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.135 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.135 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.135 DEBUG:boto:Params: {} 2024-04-12T03:58:52.135 DEBUG:boto:Token: None 2024-04-12T03:58:52.136 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.136 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005aae1d1fcf3d2edc-006618b17c-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.138 DEBUG:boto:b'dkyges-60obj000000000000000001621000falseobj000000000000000001622024-04-12T03:58:33.823Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.139 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.139 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.139 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000162 2024-04-12T03:58:52.139 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000162 2024-04-12T03:58:52.139 DEBUG:boto:Method: GET 2024-04-12T03:58:52.139 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000162 2024-04-12T03:58:52.139 DEBUG:boto:Data: 2024-04-12T03:58:52.139 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.140 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.140 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.140 DEBUG:boto:Params: {} 2024-04-12T03:58:52.140 DEBUG:boto:Token: None 2024-04-12T03:58:52.140 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.140 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9be68d720b4671a-006618b17c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.145 DEBUG:boto:b'dkyges-60obj000000000000000001621000falseobj000000000000000001622024-04-12T03:58:33.823Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-04-12T03:58:52.145 DEBUG:boto:path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.146 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.146 DEBUG:boto:path=/dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.146 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.146 DEBUG:boto:Method: GET 2024-04-12T03:58:52.146 DEBUG:boto:Path: /dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.146 DEBUG:boto:Data: 2024-04-12T03:58:52.146 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.146 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.146 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.146 DEBUG:boto:Params: {} 2024-04-12T03:58:52.147 DEBUG:boto:Token: None 2024-04-12T03:58:52.147 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.147 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3mvDwoXyeF5KHLxEmO4LRsWDnpI= 2024-04-12T03:58:52.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3mvDwoXyeF5KHLxEmO4LRsWDnpI=', 'Content-Length': '0'} 2024-04-12T03:58:52.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a740a05e6ae514b-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.190 DEBUG:boto:path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.190 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.191 DEBUG:boto:path=/dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.191 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.191 DEBUG:boto:Method: GET 2024-04-12T03:58:52.191 DEBUG:boto:Path: /dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.191 DEBUG:boto:Data: 2024-04-12T03:58:52.191 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.191 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.191 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.191 DEBUG:boto:Params: {} 2024-04-12T03:58:52.191 DEBUG:boto:Token: None 2024-04-12T03:58:52.192 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.192 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3mvDwoXyeF5KHLxEmO4LRsWDnpI= 2024-04-12T03:58:52.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3mvDwoXyeF5KHLxEmO4LRsWDnpI=', 'Content-Length': '0'} 2024-04-12T03:58:52.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6a8c06a07419b5f-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.194 DEBUG:boto:path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.194 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.195 DEBUG:boto:path=/dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.195 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.195 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.195 DEBUG:boto:Path: /dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.195 DEBUG:boto:Data: 2024-04-12T03:58:52.195 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.195 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.195 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.195 DEBUG:boto:Params: {} 2024-04-12T03:58:52.195 DEBUG:boto:Token: None 2024-04-12T03:58:52.195 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.196 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n2sFDG0pmBY72tCViO+tFFsz9wA= 2024-04-12T03:58:52.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n2sFDG0pmBY72tCViO+tFFsz9wA=', 'Content-Length': '0'} 2024-04-12T03:58:52.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbfc3eaeeb018031-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.198 DEBUG:boto:path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.198 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.198 DEBUG:boto:path=/dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.198 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.198 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.198 DEBUG:boto:Path: /dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.198 DEBUG:boto:Data: 2024-04-12T03:58:52.198 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.198 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.198 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.199 DEBUG:boto:Params: {} 2024-04-12T03:58:52.199 DEBUG:boto:Token: None 2024-04-12T03:58:52.199 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.199 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n2sFDG0pmBY72tCViO+tFFsz9wA= 2024-04-12T03:58:52.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n2sFDG0pmBY72tCViO+tFFsz9wA=', 'Content-Length': '0'} 2024-04-12T03:58:52.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a39e772c62ccabbc-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-04-12T03:58:52.201 DEBUG:boto:path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.201 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.201 DEBUG:boto:path=/dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.201 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.202 DEBUG:boto:Method: GET 2024-04-12T03:58:52.202 DEBUG:boto:Path: /dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.202 DEBUG:boto:Data: 2024-04-12T03:58:52.202 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.202 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.202 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.202 DEBUG:boto:Params: {} 2024-04-12T03:58:52.202 DEBUG:boto:Token: None 2024-04-12T03:58:52.202 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.202 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3mvDwoXyeF5KHLxEmO4LRsWDnpI= 2024-04-12T03:58:52.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3mvDwoXyeF5KHLxEmO4LRsWDnpI=', 'Content-Length': '0'} 2024-04-12T03:58:52.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c80c56bc23fc484-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.204 DEBUG:boto:path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.204 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.205 DEBUG:boto:path=/dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.205 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.205 DEBUG:boto:Method: GET 2024-04-12T03:58:52.205 DEBUG:boto:Path: /dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.205 DEBUG:boto:Data: 2024-04-12T03:58:52.205 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.205 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.205 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.205 DEBUG:boto:Params: {} 2024-04-12T03:58:52.205 DEBUG:boto:Token: None 2024-04-12T03:58:52.206 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000271?versionId=null 2024-04-12T03:58:52.206 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3mvDwoXyeF5KHLxEmO4LRsWDnpI= 2024-04-12T03:58:52.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3mvDwoXyeF5KHLxEmO4LRsWDnpI=', 'Content-Length': '0'} 2024-04-12T03:58:52.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000786ce7f18a2d6413-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.208 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.208 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.208 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000271 2024-04-12T03:58:52.208 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000271 2024-04-12T03:58:52.208 DEBUG:boto:Method: GET 2024-04-12T03:58:52.208 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000271 2024-04-12T03:58:52.208 DEBUG:boto:Data: 2024-04-12T03:58:52.208 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.209 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.209 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.209 DEBUG:boto:Params: {} 2024-04-12T03:58:52.209 DEBUG:boto:Token: None 2024-04-12T03:58:52.209 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.209 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fa634a3b2fd92b5-006618b17c-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.211 DEBUG:boto:b'dkyges-60obj000000000000000002711000falseobj000000000000000002712024-04-12T03:58:33.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.212 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.212 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.212 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000271 2024-04-12T03:58:52.212 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000271 2024-04-12T03:58:52.212 DEBUG:boto:Method: GET 2024-04-12T03:58:52.212 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000271 2024-04-12T03:58:52.212 DEBUG:boto:Data: 2024-04-12T03:58:52.212 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.212 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.213 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.213 DEBUG:boto:Params: {} 2024-04-12T03:58:52.213 DEBUG:boto:Token: None 2024-04-12T03:58:52.213 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.213 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bf46c0367edc44e-006618b17c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.218 DEBUG:boto:b'dkyges-60obj000000000000000002711000falseobj000000000000000002712024-04-12T03:58:33.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.218 DEBUG:boto:path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.218 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.218 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.219 DEBUG:boto:Path: /dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.219 DEBUG:boto:Data: 2024-04-12T03:58:52.219 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.219 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.219 DEBUG:boto:Params: {} 2024-04-12T03:58:52.219 DEBUG:boto:Token: None 2024-04-12T03:58:52.219 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.219 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+TeXg7/9bPoDz0ToRCclC9ohieM= 2024-04-12T03:58:52.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+TeXg7/9bPoDz0ToRCclC9ohieM=', 'Content-Length': '0'} 2024-04-12T03:58:52.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007b37dc893403f01-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.221 DEBUG:boto:path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.221 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.221 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.222 DEBUG:boto:Path: /dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.222 DEBUG:boto:Data: 2024-04-12T03:58:52.222 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.222 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.222 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.222 DEBUG:boto:Params: {} 2024-04-12T03:58:52.222 DEBUG:boto:Token: None 2024-04-12T03:58:52.222 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.222 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+TeXg7/9bPoDz0ToRCclC9ohieM= 2024-04-12T03:58:52.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+TeXg7/9bPoDz0ToRCclC9ohieM=', 'Content-Length': '0'} 2024-04-12T03:58:52.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004069c3d0bdc7946f-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2024-04-12T03:58:52.224 DEBUG:boto:path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.225 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.225 DEBUG:boto:Method: GET 2024-04-12T03:58:52.225 DEBUG:boto:Path: /dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.225 DEBUG:boto:Data: 2024-04-12T03:58:52.225 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.225 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.225 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.225 DEBUG:boto:Params: {} 2024-04-12T03:58:52.225 DEBUG:boto:Token: None 2024-04-12T03:58:52.225 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.226 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iZ38d3BU0J9URfLUkeb9PUQ7BcY= 2024-04-12T03:58:52.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iZ38d3BU0J9URfLUkeb9PUQ7BcY=', 'Content-Length': '0'} 2024-04-12T03:58:52.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d464f1480ca90129-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.228 DEBUG:boto:path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.228 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.228 DEBUG:boto:Method: GET 2024-04-12T03:58:52.228 DEBUG:boto:Path: /dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.228 DEBUG:boto:Data: 2024-04-12T03:58:52.228 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.228 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.228 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.228 DEBUG:boto:Params: {} 2024-04-12T03:58:52.228 DEBUG:boto:Token: None 2024-04-12T03:58:52.228 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000271 2024-04-12T03:58:52.229 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iZ38d3BU0J9URfLUkeb9PUQ7BcY= 2024-04-12T03:58:52.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iZ38d3BU0J9URfLUkeb9PUQ7BcY=', 'Content-Length': '0'} 2024-04-12T03:58:52.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000099fd2da1118cc89-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.231 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.231 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.231 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000271 2024-04-12T03:58:52.231 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000271 2024-04-12T03:58:52.231 DEBUG:boto:Method: GET 2024-04-12T03:58:52.231 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000271 2024-04-12T03:58:52.231 DEBUG:boto:Data: 2024-04-12T03:58:52.231 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.231 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.231 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.231 DEBUG:boto:Params: {} 2024-04-12T03:58:52.231 DEBUG:boto:Token: None 2024-04-12T03:58:52.232 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.232 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaea7e47e66b1be9-006618b17c-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.274 DEBUG:boto:b'dkyges-60obj000000000000000002711000falseobj000000000000000002712024-04-12T03:58:33.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.275 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.275 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.275 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000271 2024-04-12T03:58:52.275 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000271 2024-04-12T03:58:52.275 DEBUG:boto:Method: GET 2024-04-12T03:58:52.275 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000271 2024-04-12T03:58:52.275 DEBUG:boto:Data: 2024-04-12T03:58:52.275 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.275 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.275 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.275 DEBUG:boto:Params: {} 2024-04-12T03:58:52.275 DEBUG:boto:Token: None 2024-04-12T03:58:52.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.276 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e8dc6c721b44fa1-006618b17c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.280 DEBUG:boto:b'dkyges-60obj000000000000000002711000falseobj000000000000000002712024-04-12T03:58:33.897Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-04-12T03:58:52.281 DEBUG:boto:path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.281 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.281 DEBUG:boto:path=/dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.281 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.281 DEBUG:boto:Method: GET 2024-04-12T03:58:52.281 DEBUG:boto:Path: /dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.281 DEBUG:boto:Data: 2024-04-12T03:58:52.281 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.281 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.281 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.282 DEBUG:boto:Params: {} 2024-04-12T03:58:52.282 DEBUG:boto:Token: None 2024-04-12T03:58:52.282 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.282 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m+TAPwVXfnC5Puer6+khf/SNDQo= 2024-04-12T03:58:52.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m+TAPwVXfnC5Puer6+khf/SNDQo=', 'Content-Length': '0'} 2024-04-12T03:58:52.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d151c43ad9105140-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.284 DEBUG:boto:path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.285 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.285 DEBUG:boto:path=/dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.285 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.285 DEBUG:boto:Method: GET 2024-04-12T03:58:52.285 DEBUG:boto:Path: /dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.285 DEBUG:boto:Data: 2024-04-12T03:58:52.285 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.285 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.285 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.285 DEBUG:boto:Params: {} 2024-04-12T03:58:52.285 DEBUG:boto:Token: None 2024-04-12T03:58:52.285 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.286 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m+TAPwVXfnC5Puer6+khf/SNDQo= 2024-04-12T03:58:52.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m+TAPwVXfnC5Puer6+khf/SNDQo=', 'Content-Length': '0'} 2024-04-12T03:58:52.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb5de0d3b33246a9-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.334 DEBUG:boto:path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.335 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.335 DEBUG:boto:path=/dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.335 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.335 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.335 DEBUG:boto:Path: /dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.335 DEBUG:boto:Data: 2024-04-12T03:58:52.335 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.335 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.335 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.335 DEBUG:boto:Params: {} 2024-04-12T03:58:52.335 DEBUG:boto:Token: None 2024-04-12T03:58:52.335 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.336 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:twQAH40YciH6NfMTSS/ZGLRp/S0= 2024-04-12T03:58:52.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:twQAH40YciH6NfMTSS/ZGLRp/S0=', 'Content-Length': '0'} 2024-04-12T03:58:52.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e4da484fda304c5-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.338 DEBUG:boto:path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.338 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.338 DEBUG:boto:path=/dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.338 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.338 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.338 DEBUG:boto:Path: /dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.338 DEBUG:boto:Data: 2024-04-12T03:58:52.338 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.339 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.339 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.339 DEBUG:boto:Params: {} 2024-04-12T03:58:52.339 DEBUG:boto:Token: None 2024-04-12T03:58:52.339 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.339 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:twQAH40YciH6NfMTSS/ZGLRp/S0= 2024-04-12T03:58:52.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:twQAH40YciH6NfMTSS/ZGLRp/S0=', 'Content-Length': '0'} 2024-04-12T03:58:52.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c94063781f89181-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-04-12T03:58:52.342 DEBUG:boto:path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.342 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.342 DEBUG:boto:path=/dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.342 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.342 DEBUG:boto:Method: GET 2024-04-12T03:58:52.342 DEBUG:boto:Path: /dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.342 DEBUG:boto:Data: 2024-04-12T03:58:52.342 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.342 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.342 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.343 DEBUG:boto:Params: {} 2024-04-12T03:58:52.343 DEBUG:boto:Token: None 2024-04-12T03:58:52.343 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.343 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m+TAPwVXfnC5Puer6+khf/SNDQo= 2024-04-12T03:58:52.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m+TAPwVXfnC5Puer6+khf/SNDQo=', 'Content-Length': '0'} 2024-04-12T03:58:52.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d4afe8ab7efc969-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.345 DEBUG:boto:path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.345 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.346 DEBUG:boto:path=/dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.346 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.346 DEBUG:boto:Method: GET 2024-04-12T03:58:52.346 DEBUG:boto:Path: /dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.346 DEBUG:boto:Data: 2024-04-12T03:58:52.346 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.346 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.346 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.346 DEBUG:boto:Params: {} 2024-04-12T03:58:52.347 DEBUG:boto:Token: None 2024-04-12T03:58:52.347 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000283?versionId=null 2024-04-12T03:58:52.347 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m+TAPwVXfnC5Puer6+khf/SNDQo= 2024-04-12T03:58:52.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m+TAPwVXfnC5Puer6+khf/SNDQo=', 'Content-Length': '0'} 2024-04-12T03:58:52.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002984feb58ec92ed9-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.349 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.349 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.349 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000283 2024-04-12T03:58:52.350 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000283 2024-04-12T03:58:52.350 DEBUG:boto:Method: GET 2024-04-12T03:58:52.350 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000283 2024-04-12T03:58:52.350 DEBUG:boto:Data: 2024-04-12T03:58:52.350 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.350 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.350 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.350 DEBUG:boto:Params: {} 2024-04-12T03:58:52.351 DEBUG:boto:Token: None 2024-04-12T03:58:52.351 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.351 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009159e43e83ca5633-006618b17c-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.354 DEBUG:boto:b'dkyges-60obj000000000000000002831000falseobj000000000000000002832024-04-12T03:58:33.635Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.354 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.354 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.354 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000283 2024-04-12T03:58:52.354 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000283 2024-04-12T03:58:52.355 DEBUG:boto:Method: GET 2024-04-12T03:58:52.355 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000283 2024-04-12T03:58:52.355 DEBUG:boto:Data: 2024-04-12T03:58:52.355 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.355 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.355 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.355 DEBUG:boto:Params: {} 2024-04-12T03:58:52.355 DEBUG:boto:Token: None 2024-04-12T03:58:52.355 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.356 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030a12457842dfcf8-006618b17c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.360 DEBUG:boto:b'dkyges-60obj000000000000000002831000falseobj000000000000000002832024-04-12T03:58:33.635Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.361 DEBUG:boto:path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.361 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.361 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.361 DEBUG:boto:Path: /dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.361 DEBUG:boto:Data: 2024-04-12T03:58:52.361 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.361 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.361 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.361 DEBUG:boto:Params: {} 2024-04-12T03:58:52.362 DEBUG:boto:Token: None 2024-04-12T03:58:52.362 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.362 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5kNRkuWX3ApJwbXunGpmvmiwoGQ= 2024-04-12T03:58:52.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5kNRkuWX3ApJwbXunGpmvmiwoGQ=', 'Content-Length': '0'} 2024-04-12T03:58:52.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff43d5fe0020ad64-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.364 DEBUG:boto:path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.364 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.364 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.365 DEBUG:boto:Path: /dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.365 DEBUG:boto:Data: 2024-04-12T03:58:52.365 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.365 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.365 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.365 DEBUG:boto:Params: {} 2024-04-12T03:58:52.365 DEBUG:boto:Token: None 2024-04-12T03:58:52.365 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.365 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5kNRkuWX3ApJwbXunGpmvmiwoGQ= 2024-04-12T03:58:52.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5kNRkuWX3ApJwbXunGpmvmiwoGQ=', 'Content-Length': '0'} 2024-04-12T03:58:52.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b22afc08ff68398b-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2024-04-12T03:58:52.368 DEBUG:boto:path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.368 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.368 DEBUG:boto:Method: GET 2024-04-12T03:58:52.368 DEBUG:boto:Path: /dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.368 DEBUG:boto:Data: 2024-04-12T03:58:52.368 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.368 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.368 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.368 DEBUG:boto:Params: {} 2024-04-12T03:58:52.369 DEBUG:boto:Token: None 2024-04-12T03:58:52.369 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.369 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TPDtKBgkAtewmcP6p0770opbFx8= 2024-04-12T03:58:52.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TPDtKBgkAtewmcP6p0770opbFx8=', 'Content-Length': '0'} 2024-04-12T03:58:52.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000497bbf6f10bc750e-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.371 DEBUG:boto:path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.371 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.371 DEBUG:boto:Method: GET 2024-04-12T03:58:52.371 DEBUG:boto:Path: /dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.371 DEBUG:boto:Data: 2024-04-12T03:58:52.371 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.371 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.372 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.372 DEBUG:boto:Params: {} 2024-04-12T03:58:52.372 DEBUG:boto:Token: None 2024-04-12T03:58:52.372 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000283 2024-04-12T03:58:52.372 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TPDtKBgkAtewmcP6p0770opbFx8= 2024-04-12T03:58:52.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TPDtKBgkAtewmcP6p0770opbFx8=', 'Content-Length': '0'} 2024-04-12T03:58:52.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d5b7f894eb7a09a-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.414 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.414 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.414 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000283 2024-04-12T03:58:52.415 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000283 2024-04-12T03:58:52.415 DEBUG:boto:Method: GET 2024-04-12T03:58:52.415 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000283 2024-04-12T03:58:52.415 DEBUG:boto:Data: 2024-04-12T03:58:52.415 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.415 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.415 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.415 DEBUG:boto:Params: {} 2024-04-12T03:58:52.415 DEBUG:boto:Token: None 2024-04-12T03:58:52.416 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.416 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057610410e7214023-006618b17c-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.418 DEBUG:boto:b'dkyges-60obj000000000000000002831000falseobj000000000000000002832024-04-12T03:58:33.635Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.419 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.419 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.419 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000283 2024-04-12T03:58:52.419 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000283 2024-04-12T03:58:52.419 DEBUG:boto:Method: GET 2024-04-12T03:58:52.419 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000283 2024-04-12T03:58:52.419 DEBUG:boto:Data: 2024-04-12T03:58:52.420 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.420 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.420 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.420 DEBUG:boto:Params: {} 2024-04-12T03:58:52.420 DEBUG:boto:Token: None 2024-04-12T03:58:52.420 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.420 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a96a771073bf90e-006618b17c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.425 DEBUG:boto:b'dkyges-60obj000000000000000002831000falseobj000000000000000002832024-04-12T03:58:33.635Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-04-12T03:58:52.426 DEBUG:boto:path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.426 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.426 DEBUG:boto:path=/dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.426 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.426 DEBUG:boto:Method: GET 2024-04-12T03:58:52.426 DEBUG:boto:Path: /dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.426 DEBUG:boto:Data: 2024-04-12T03:58:52.426 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.426 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.426 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.426 DEBUG:boto:Params: {} 2024-04-12T03:58:52.427 DEBUG:boto:Token: None 2024-04-12T03:58:52.427 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.427 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GUEtEAURMlcVLjR9Q20Z4/jru8I= 2024-04-12T03:58:52.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GUEtEAURMlcVLjR9Q20Z4/jru8I=', 'Content-Length': '0'} 2024-04-12T03:58:52.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001877a299caca2943-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.429 DEBUG:boto:path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.430 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.430 DEBUG:boto:path=/dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.430 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.430 DEBUG:boto:Method: GET 2024-04-12T03:58:52.430 DEBUG:boto:Path: /dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.430 DEBUG:boto:Data: 2024-04-12T03:58:52.430 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.430 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.430 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.430 DEBUG:boto:Params: {} 2024-04-12T03:58:52.431 DEBUG:boto:Token: None 2024-04-12T03:58:52.431 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.431 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GUEtEAURMlcVLjR9Q20Z4/jru8I= 2024-04-12T03:58:52.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GUEtEAURMlcVLjR9Q20Z4/jru8I=', 'Content-Length': '0'} 2024-04-12T03:58:52.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000392d657d420f4cb9-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.433 DEBUG:boto:path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.433 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.433 DEBUG:boto:path=/dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.434 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.434 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.434 DEBUG:boto:Path: /dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.434 DEBUG:boto:Data: 2024-04-12T03:58:52.434 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.434 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.434 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.434 DEBUG:boto:Params: {} 2024-04-12T03:58:52.434 DEBUG:boto:Token: None 2024-04-12T03:58:52.434 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.435 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBwK3QLdgFjhwyerM5LzXG+T8SY= 2024-04-12T03:58:52.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBwK3QLdgFjhwyerM5LzXG+T8SY=', 'Content-Length': '0'} 2024-04-12T03:58:52.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cf4026a1bc53f93-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.437 DEBUG:boto:path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.437 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.437 DEBUG:boto:path=/dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.437 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.437 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.437 DEBUG:boto:Path: /dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.437 DEBUG:boto:Data: 2024-04-12T03:58:52.437 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.437 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.438 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.438 DEBUG:boto:Params: {} 2024-04-12T03:58:52.438 DEBUG:boto:Token: None 2024-04-12T03:58:52.438 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.438 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBwK3QLdgFjhwyerM5LzXG+T8SY= 2024-04-12T03:58:52.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBwK3QLdgFjhwyerM5LzXG+T8SY=', 'Content-Length': '0'} 2024-04-12T03:58:52.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f78a8581705fabfb-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-04-12T03:58:52.440 DEBUG:boto:path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.440 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.440 DEBUG:boto:path=/dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.440 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.441 DEBUG:boto:Method: GET 2024-04-12T03:58:52.441 DEBUG:boto:Path: /dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.441 DEBUG:boto:Data: 2024-04-12T03:58:52.441 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.441 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.441 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.441 DEBUG:boto:Params: {} 2024-04-12T03:58:52.441 DEBUG:boto:Token: None 2024-04-12T03:58:52.441 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.441 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GUEtEAURMlcVLjR9Q20Z4/jru8I= 2024-04-12T03:58:52.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GUEtEAURMlcVLjR9Q20Z4/jru8I=', 'Content-Length': '0'} 2024-04-12T03:58:52.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f758337da1dbea9f-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.444 DEBUG:boto:path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.444 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.444 DEBUG:boto:path=/dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.444 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.444 DEBUG:boto:Method: GET 2024-04-12T03:58:52.444 DEBUG:boto:Path: /dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.444 DEBUG:boto:Data: 2024-04-12T03:58:52.444 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.444 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.444 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.444 DEBUG:boto:Params: {} 2024-04-12T03:58:52.445 DEBUG:boto:Token: None 2024-04-12T03:58:52.445 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000317?versionId=null 2024-04-12T03:58:52.445 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GUEtEAURMlcVLjR9Q20Z4/jru8I= 2024-04-12T03:58:52.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GUEtEAURMlcVLjR9Q20Z4/jru8I=', 'Content-Length': '0'} 2024-04-12T03:58:52.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000274e778703d94da3-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.447 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.448 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.448 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000317 2024-04-12T03:58:52.448 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000317 2024-04-12T03:58:52.448 DEBUG:boto:Method: GET 2024-04-12T03:58:52.448 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000317 2024-04-12T03:58:52.448 DEBUG:boto:Data: 2024-04-12T03:58:52.448 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.448 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.448 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.448 DEBUG:boto:Params: {} 2024-04-12T03:58:52.449 DEBUG:boto:Token: None 2024-04-12T03:58:52.449 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.449 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c90e6f4881397c7-006618b17c-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.498 DEBUG:boto:b'dkyges-60obj000000000000000003171000falseobj000000000000000003172024-04-12T03:58:34.346Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.499 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.499 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.499 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000317 2024-04-12T03:58:52.499 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000317 2024-04-12T03:58:52.499 DEBUG:boto:Method: GET 2024-04-12T03:58:52.499 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000317 2024-04-12T03:58:52.499 DEBUG:boto:Data: 2024-04-12T03:58:52.499 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.500 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.500 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.500 DEBUG:boto:Params: {} 2024-04-12T03:58:52.500 DEBUG:boto:Token: None 2024-04-12T03:58:52.500 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.500 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035226c2f79504e07-006618b17c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.505 DEBUG:boto:b'dkyges-60obj000000000000000003171000falseobj000000000000000003172024-04-12T03:58:34.346Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.505 DEBUG:boto:path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.505 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.505 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.505 DEBUG:boto:Path: /dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.506 DEBUG:boto:Data: 2024-04-12T03:58:52.506 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.506 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.506 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.506 DEBUG:boto:Params: {} 2024-04-12T03:58:52.506 DEBUG:boto:Token: None 2024-04-12T03:58:52.506 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.506 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GmxuhuXhX5k4LuyHMHhjL9RiL2c= 2024-04-12T03:58:52.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GmxuhuXhX5k4LuyHMHhjL9RiL2c=', 'Content-Length': '0'} 2024-04-12T03:58:52.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004afdb96303369322-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.508 DEBUG:boto:path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.508 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.508 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.509 DEBUG:boto:Path: /dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.509 DEBUG:boto:Data: 2024-04-12T03:58:52.509 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.509 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.509 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.509 DEBUG:boto:Params: {} 2024-04-12T03:58:52.509 DEBUG:boto:Token: None 2024-04-12T03:58:52.509 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.509 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GmxuhuXhX5k4LuyHMHhjL9RiL2c= 2024-04-12T03:58:52.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GmxuhuXhX5k4LuyHMHhjL9RiL2c=', 'Content-Length': '0'} 2024-04-12T03:58:52.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ece7af7109c34ecd-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2024-04-12T03:58:52.511 DEBUG:boto:path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.512 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.512 DEBUG:boto:Method: GET 2024-04-12T03:58:52.512 DEBUG:boto:Path: /dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.512 DEBUG:boto:Data: 2024-04-12T03:58:52.512 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.512 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.512 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.512 DEBUG:boto:Params: {} 2024-04-12T03:58:52.512 DEBUG:boto:Token: None 2024-04-12T03:58:52.512 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.513 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gcng/ehnYOETV0TOayJmX73L/rw= 2024-04-12T03:58:52.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gcng/ehnYOETV0TOayJmX73L/rw=', 'Content-Length': '0'} 2024-04-12T03:58:52.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000505bee20456ecfeb-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.515 DEBUG:boto:path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.515 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.515 DEBUG:boto:Method: GET 2024-04-12T03:58:52.515 DEBUG:boto:Path: /dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.515 DEBUG:boto:Data: 2024-04-12T03:58:52.515 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.515 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.515 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.515 DEBUG:boto:Params: {} 2024-04-12T03:58:52.516 DEBUG:boto:Token: None 2024-04-12T03:58:52.516 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000317 2024-04-12T03:58:52.516 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gcng/ehnYOETV0TOayJmX73L/rw= 2024-04-12T03:58:52.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gcng/ehnYOETV0TOayJmX73L/rw=', 'Content-Length': '0'} 2024-04-12T03:58:52.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e5564bbb73d1c2c-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.562 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.563 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.563 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000317 2024-04-12T03:58:52.563 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000317 2024-04-12T03:58:52.563 DEBUG:boto:Method: GET 2024-04-12T03:58:52.563 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000317 2024-04-12T03:58:52.563 DEBUG:boto:Data: 2024-04-12T03:58:52.563 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.563 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.563 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.563 DEBUG:boto:Params: {} 2024-04-12T03:58:52.564 DEBUG:boto:Token: None 2024-04-12T03:58:52.564 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.564 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef99ef2c07e66e9f-006618b17c-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.567 DEBUG:boto:b'dkyges-60obj000000000000000003171000falseobj000000000000000003172024-04-12T03:58:34.346Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.567 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.567 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.567 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000317 2024-04-12T03:58:52.567 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000317 2024-04-12T03:58:52.567 DEBUG:boto:Method: GET 2024-04-12T03:58:52.567 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000317 2024-04-12T03:58:52.567 DEBUG:boto:Data: 2024-04-12T03:58:52.568 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.568 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.568 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.568 DEBUG:boto:Params: {} 2024-04-12T03:58:52.568 DEBUG:boto:Token: None 2024-04-12T03:58:52.568 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.568 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021cf4ea4db2b8f06-006618b17c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.573 DEBUG:boto:b'dkyges-60obj000000000000000003171000falseobj000000000000000003172024-04-12T03:58:34.346Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-04-12T03:58:52.573 DEBUG:boto:path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.573 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.573 DEBUG:boto:path=/dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.573 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.574 DEBUG:boto:Method: GET 2024-04-12T03:58:52.574 DEBUG:boto:Path: /dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.574 DEBUG:boto:Data: 2024-04-12T03:58:52.574 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.574 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.574 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.574 DEBUG:boto:Params: {} 2024-04-12T03:58:52.574 DEBUG:boto:Token: None 2024-04-12T03:58:52.574 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.574 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wMhuVo/AJ79C67a31UEuYH1bFQE= 2024-04-12T03:58:52.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wMhuVo/AJ79C67a31UEuYH1bFQE=', 'Content-Length': '0'} 2024-04-12T03:58:52.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6dcea6647d0abd4-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.577 DEBUG:boto:path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.577 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.577 DEBUG:boto:path=/dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.577 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.577 DEBUG:boto:Method: GET 2024-04-12T03:58:52.577 DEBUG:boto:Path: /dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.577 DEBUG:boto:Data: 2024-04-12T03:58:52.577 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.577 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.578 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.578 DEBUG:boto:Params: {} 2024-04-12T03:58:52.578 DEBUG:boto:Token: None 2024-04-12T03:58:52.578 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.578 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wMhuVo/AJ79C67a31UEuYH1bFQE= 2024-04-12T03:58:52.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wMhuVo/AJ79C67a31UEuYH1bFQE=', 'Content-Length': '0'} 2024-04-12T03:58:52.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008504a10851bff84b-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.581 DEBUG:boto:path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.581 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.581 DEBUG:boto:path=/dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.581 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.581 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.581 DEBUG:boto:Path: /dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.581 DEBUG:boto:Data: 2024-04-12T03:58:52.581 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.581 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.581 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.581 DEBUG:boto:Params: {} 2024-04-12T03:58:52.581 DEBUG:boto:Token: None 2024-04-12T03:58:52.582 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.582 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CxcT/tsasYUEgB0Of6nLTSfcwtY= 2024-04-12T03:58:52.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CxcT/tsasYUEgB0Of6nLTSfcwtY=', 'Content-Length': '0'} 2024-04-12T03:58:52.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b76835d157d3c89-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.584 DEBUG:boto:path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.584 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.584 DEBUG:boto:path=/dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.584 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.584 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.584 DEBUG:boto:Path: /dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.584 DEBUG:boto:Data: 2024-04-12T03:58:52.584 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.584 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.584 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.584 DEBUG:boto:Params: {} 2024-04-12T03:58:52.585 DEBUG:boto:Token: None 2024-04-12T03:58:52.585 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.585 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CxcT/tsasYUEgB0Of6nLTSfcwtY= 2024-04-12T03:58:52.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CxcT/tsasYUEgB0Of6nLTSfcwtY=', 'Content-Length': '0'} 2024-04-12T03:58:52.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002583530f0b77591c-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-04-12T03:58:52.587 DEBUG:boto:path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.587 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.587 DEBUG:boto:path=/dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.587 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.587 DEBUG:boto:Method: GET 2024-04-12T03:58:52.587 DEBUG:boto:Path: /dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.588 DEBUG:boto:Data: 2024-04-12T03:58:52.588 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.588 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.588 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.588 DEBUG:boto:Params: {} 2024-04-12T03:58:52.588 DEBUG:boto:Token: None 2024-04-12T03:58:52.588 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.588 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wMhuVo/AJ79C67a31UEuYH1bFQE= 2024-04-12T03:58:52.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wMhuVo/AJ79C67a31UEuYH1bFQE=', 'Content-Length': '0'} 2024-04-12T03:58:52.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd790550b2b7d28c-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.591 DEBUG:boto:path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.591 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.591 DEBUG:boto:path=/dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.591 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.591 DEBUG:boto:Method: GET 2024-04-12T03:58:52.591 DEBUG:boto:Path: /dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.591 DEBUG:boto:Data: 2024-04-12T03:58:52.591 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.591 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.591 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.591 DEBUG:boto:Params: {} 2024-04-12T03:58:52.592 DEBUG:boto:Token: None 2024-04-12T03:58:52.592 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000423?versionId=null 2024-04-12T03:58:52.592 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wMhuVo/AJ79C67a31UEuYH1bFQE= 2024-04-12T03:58:52.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wMhuVo/AJ79C67a31UEuYH1bFQE=', 'Content-Length': '0'} 2024-04-12T03:58:52.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092ca47677104fc87-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.634 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.634 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.635 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000423 2024-04-12T03:58:52.635 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000423 2024-04-12T03:58:52.635 DEBUG:boto:Method: GET 2024-04-12T03:58:52.635 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000423 2024-04-12T03:58:52.635 DEBUG:boto:Data: 2024-04-12T03:58:52.635 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.635 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.635 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.635 DEBUG:boto:Params: {} 2024-04-12T03:58:52.635 DEBUG:boto:Token: None 2024-04-12T03:58:52.636 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.636 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a054ad27ad31dc0-006618b17c-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.638 DEBUG:boto:b'dkyges-60obj000000000000000004231000falseobj000000000000000004232024-04-12T03:58:33.963Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.639 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.639 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.639 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000423 2024-04-12T03:58:52.639 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000423 2024-04-12T03:58:52.639 DEBUG:boto:Method: GET 2024-04-12T03:58:52.639 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000423 2024-04-12T03:58:52.639 DEBUG:boto:Data: 2024-04-12T03:58:52.639 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.639 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.640 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.640 DEBUG:boto:Params: {} 2024-04-12T03:58:52.640 DEBUG:boto:Token: None 2024-04-12T03:58:52.640 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.640 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df653b9960c63c81-006618b17c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.645 DEBUG:boto:b'dkyges-60obj000000000000000004231000falseobj000000000000000004232024-04-12T03:58:33.963Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.645 DEBUG:boto:path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.645 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.645 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.646 DEBUG:boto:Path: /dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.646 DEBUG:boto:Data: 2024-04-12T03:58:52.646 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.646 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.646 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.646 DEBUG:boto:Params: {} 2024-04-12T03:58:52.646 DEBUG:boto:Token: None 2024-04-12T03:58:52.646 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.646 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cX1qw4wD1ydiZbVrAlTABINWqEU= 2024-04-12T03:58:52.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cX1qw4wD1ydiZbVrAlTABINWqEU=', 'Content-Length': '0'} 2024-04-12T03:58:52.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036216be5bfe09d4e-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.649 DEBUG:boto:path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.649 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.649 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.649 DEBUG:boto:Path: /dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.649 DEBUG:boto:Data: 2024-04-12T03:58:52.649 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.649 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.649 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.649 DEBUG:boto:Params: {} 2024-04-12T03:58:52.649 DEBUG:boto:Token: None 2024-04-12T03:58:52.650 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.650 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cX1qw4wD1ydiZbVrAlTABINWqEU= 2024-04-12T03:58:52.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cX1qw4wD1ydiZbVrAlTABINWqEU=', 'Content-Length': '0'} 2024-04-12T03:58:52.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d3fceda7b4df716-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2024-04-12T03:58:52.652 DEBUG:boto:path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.652 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.652 DEBUG:boto:Method: GET 2024-04-12T03:58:52.652 DEBUG:boto:Path: /dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.652 DEBUG:boto:Data: 2024-04-12T03:58:52.652 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.653 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.653 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.653 DEBUG:boto:Params: {} 2024-04-12T03:58:52.653 DEBUG:boto:Token: None 2024-04-12T03:58:52.653 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.653 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Mav0geYoKB7iFc24DlkErlYqYdY= 2024-04-12T03:58:52.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Mav0geYoKB7iFc24DlkErlYqYdY=', 'Content-Length': '0'} 2024-04-12T03:58:52.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a87916bea9a99ab-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.698 DEBUG:boto:path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.698 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.699 DEBUG:boto:Method: GET 2024-04-12T03:58:52.699 DEBUG:boto:Path: /dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.699 DEBUG:boto:Data: 2024-04-12T03:58:52.699 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.699 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.699 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.699 DEBUG:boto:Params: {} 2024-04-12T03:58:52.699 DEBUG:boto:Token: None 2024-04-12T03:58:52.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000423 2024-04-12T03:58:52.699 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Mav0geYoKB7iFc24DlkErlYqYdY= 2024-04-12T03:58:52.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Mav0geYoKB7iFc24DlkErlYqYdY=', 'Content-Length': '0'} 2024-04-12T03:58:52.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b2ddf58148d9915-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.702 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.702 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.702 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000423 2024-04-12T03:58:52.702 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000423 2024-04-12T03:58:52.702 DEBUG:boto:Method: GET 2024-04-12T03:58:52.702 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000423 2024-04-12T03:58:52.703 DEBUG:boto:Data: 2024-04-12T03:58:52.703 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.703 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.703 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.703 DEBUG:boto:Params: {} 2024-04-12T03:58:52.703 DEBUG:boto:Token: None 2024-04-12T03:58:52.703 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.703 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000952e547e54a38ca1-006618b17c-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.706 DEBUG:boto:b'dkyges-60obj000000000000000004231000falseobj000000000000000004232024-04-12T03:58:33.963Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.706 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.706 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.707 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000423 2024-04-12T03:58:52.707 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000423 2024-04-12T03:58:52.707 DEBUG:boto:Method: GET 2024-04-12T03:58:52.707 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000423 2024-04-12T03:58:52.707 DEBUG:boto:Data: 2024-04-12T03:58:52.707 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.707 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.707 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.707 DEBUG:boto:Params: {} 2024-04-12T03:58:52.707 DEBUG:boto:Token: None 2024-04-12T03:58:52.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.708 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000844acb7083bdc07-006618b17c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.712 DEBUG:boto:b'dkyges-60obj000000000000000004231000falseobj000000000000000004232024-04-12T03:58:33.963Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-04-12T03:58:52.713 DEBUG:boto:path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.713 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.713 DEBUG:boto:path=/dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.713 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.713 DEBUG:boto:Method: GET 2024-04-12T03:58:52.713 DEBUG:boto:Path: /dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.713 DEBUG:boto:Data: 2024-04-12T03:58:52.713 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.713 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.713 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.713 DEBUG:boto:Params: {} 2024-04-12T03:58:52.714 DEBUG:boto:Token: None 2024-04-12T03:58:52.714 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.714 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vVn9ppTSHeus0UXlWrbkFMWWqyg= 2024-04-12T03:58:52.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vVn9ppTSHeus0UXlWrbkFMWWqyg=', 'Content-Length': '0'} 2024-04-12T03:58:52.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d900f50a37cfb84-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.716 DEBUG:boto:path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.716 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.716 DEBUG:boto:path=/dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.717 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.717 DEBUG:boto:Method: GET 2024-04-12T03:58:52.717 DEBUG:boto:Path: /dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.717 DEBUG:boto:Data: 2024-04-12T03:58:52.717 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.717 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.717 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.717 DEBUG:boto:Params: {} 2024-04-12T03:58:52.717 DEBUG:boto:Token: None 2024-04-12T03:58:52.717 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.718 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vVn9ppTSHeus0UXlWrbkFMWWqyg= 2024-04-12T03:58:52.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vVn9ppTSHeus0UXlWrbkFMWWqyg=', 'Content-Length': '0'} 2024-04-12T03:58:52.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dcf6ca3828dd475-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.766 DEBUG:boto:path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.766 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.767 DEBUG:boto:path=/dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.767 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.767 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.767 DEBUG:boto:Path: /dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.767 DEBUG:boto:Data: 2024-04-12T03:58:52.767 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.767 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.767 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.767 DEBUG:boto:Params: {} 2024-04-12T03:58:52.767 DEBUG:boto:Token: None 2024-04-12T03:58:52.768 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.768 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3ITekioitqClPh3V5Y2YqQbMZ+E= 2024-04-12T03:58:52.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3ITekioitqClPh3V5Y2YqQbMZ+E=', 'Content-Length': '0'} 2024-04-12T03:58:52.770 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c766c25f392e297b-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.770 DEBUG:boto:path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.770 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.770 DEBUG:boto:path=/dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.770 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.770 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.770 DEBUG:boto:Path: /dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.770 DEBUG:boto:Data: 2024-04-12T03:58:52.771 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.771 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.771 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.771 DEBUG:boto:Params: {} 2024-04-12T03:58:52.771 DEBUG:boto:Token: None 2024-04-12T03:58:52.771 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.771 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3ITekioitqClPh3V5Y2YqQbMZ+E= 2024-04-12T03:58:52.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3ITekioitqClPh3V5Y2YqQbMZ+E=', 'Content-Length': '0'} 2024-04-12T03:58:52.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000261ebcf3bf93e348-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-04-12T03:58:52.774 DEBUG:boto:path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.774 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.774 DEBUG:boto:path=/dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.774 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.774 DEBUG:boto:Method: GET 2024-04-12T03:58:52.774 DEBUG:boto:Path: /dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.774 DEBUG:boto:Data: 2024-04-12T03:58:52.775 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.775 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.775 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.775 DEBUG:boto:Params: {} 2024-04-12T03:58:52.775 DEBUG:boto:Token: None 2024-04-12T03:58:52.775 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.775 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vVn9ppTSHeus0UXlWrbkFMWWqyg= 2024-04-12T03:58:52.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vVn9ppTSHeus0UXlWrbkFMWWqyg=', 'Content-Length': '0'} 2024-04-12T03:58:52.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d69bce54c3a29031-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.777 DEBUG:boto:path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.777 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.777 DEBUG:boto:path=/dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.777 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.778 DEBUG:boto:Method: GET 2024-04-12T03:58:52.778 DEBUG:boto:Path: /dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.778 DEBUG:boto:Data: 2024-04-12T03:58:52.778 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.778 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.778 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.778 DEBUG:boto:Params: {} 2024-04-12T03:58:52.778 DEBUG:boto:Token: None 2024-04-12T03:58:52.778 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000467?versionId=null 2024-04-12T03:58:52.778 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vVn9ppTSHeus0UXlWrbkFMWWqyg= 2024-04-12T03:58:52.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vVn9ppTSHeus0UXlWrbkFMWWqyg=', 'Content-Length': '0'} 2024-04-12T03:58:52.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007837780322743f11-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.780 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.781 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.781 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000467 2024-04-12T03:58:52.781 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000467 2024-04-12T03:58:52.781 DEBUG:boto:Method: GET 2024-04-12T03:58:52.781 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000467 2024-04-12T03:58:52.781 DEBUG:boto:Data: 2024-04-12T03:58:52.781 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.781 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.781 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.781 DEBUG:boto:Params: {} 2024-04-12T03:58:52.781 DEBUG:boto:Token: None 2024-04-12T03:58:52.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.782 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009376a884588ae69a-006618b17c-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.784 DEBUG:boto:b'dkyges-60obj000000000000000004671000falseobj000000000000000004672024-04-12T03:58:34.332Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.785 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.785 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.785 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000467 2024-04-12T03:58:52.785 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000467 2024-04-12T03:58:52.785 DEBUG:boto:Method: GET 2024-04-12T03:58:52.785 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000467 2024-04-12T03:58:52.785 DEBUG:boto:Data: 2024-04-12T03:58:52.785 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.785 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.786 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.786 DEBUG:boto:Params: {} 2024-04-12T03:58:52.786 DEBUG:boto:Token: None 2024-04-12T03:58:52.786 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.786 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066c35f23ae2a5c69-006618b17c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.791 DEBUG:boto:b'dkyges-60obj000000000000000004671000falseobj000000000000000004672024-04-12T03:58:34.332Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.791 DEBUG:boto:path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.791 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.791 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.791 DEBUG:boto:Path: /dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.791 DEBUG:boto:Data: 2024-04-12T03:58:52.791 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.791 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.792 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.792 DEBUG:boto:Params: {} 2024-04-12T03:58:52.792 DEBUG:boto:Token: None 2024-04-12T03:58:52.792 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.792 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v8sxRj5tVK8H7nCJF31Cl9lXuaQ= 2024-04-12T03:58:52.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v8sxRj5tVK8H7nCJF31Cl9lXuaQ=', 'Content-Length': '0'} 2024-04-12T03:58:52.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000653dca4be262cc69-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.794 DEBUG:boto:path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.794 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.794 DEBUG:boto:Method: HEAD 2024-04-12T03:58:52.794 DEBUG:boto:Path: /dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.795 DEBUG:boto:Data: 2024-04-12T03:58:52.795 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.795 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.795 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.795 DEBUG:boto:Params: {} 2024-04-12T03:58:52.795 DEBUG:boto:Token: None 2024-04-12T03:58:52.795 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.795 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v8sxRj5tVK8H7nCJF31Cl9lXuaQ= 2024-04-12T03:58:52.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v8sxRj5tVK8H7nCJF31Cl9lXuaQ=', 'Content-Length': '0'} 2024-04-12T03:58:52.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ef3ad3e6417d36f-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2024-04-12T03:58:52.797 DEBUG:boto:path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.797 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.798 DEBUG:boto:Method: GET 2024-04-12T03:58:52.798 DEBUG:boto:Path: /dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.798 DEBUG:boto:Data: 2024-04-12T03:58:52.798 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.798 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.798 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.798 DEBUG:boto:Params: {} 2024-04-12T03:58:52.798 DEBUG:boto:Token: None 2024-04-12T03:58:52.798 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.798 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBn4o1b+uB/Ez0EUtXtJQ+oxkTE= 2024-04-12T03:58:52.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBn4o1b+uB/Ez0EUtXtJQ+oxkTE=', 'Content-Length': '0'} 2024-04-12T03:58:52.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-60:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043380226974b9e2c-006618b17c-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.801 DEBUG:boto:path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.801 DEBUG:boto:auth_path=/dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.801 DEBUG:boto:Method: GET 2024-04-12T03:58:52.801 DEBUG:boto:Path: /dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.801 DEBUG:boto:Data: 2024-04-12T03:58:52.801 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.801 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.801 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.801 DEBUG:boto:Params: {} 2024-04-12T03:58:52.802 DEBUG:boto:Token: None 2024-04-12T03:58:52.802 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/obj00000000000000000467 2024-04-12T03:58:52.802 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HBn4o1b+uB/Ez0EUtXtJQ+oxkTE= 2024-04-12T03:58:52.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HBn4o1b+uB/Ez0EUtXtJQ+oxkTE=', 'Content-Length': '0'} 2024-04-12T03:58:52.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:58:34 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079253be09268f4e6-006618b17c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.804 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.804 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.804 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000467 2024-04-12T03:58:52.804 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000467 2024-04-12T03:58:52.804 DEBUG:boto:Method: GET 2024-04-12T03:58:52.804 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000467 2024-04-12T03:58:52.804 DEBUG:boto:Data: 2024-04-12T03:58:52.804 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.805 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T03:58:52.805 DEBUG:boto:Port: 8001 2024-04-12T03:58:52.805 DEBUG:boto:Params: {} 2024-04-12T03:58:52.805 DEBUG:boto:Token: None 2024-04-12T03:58:52.805 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.805 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba62b7c36494f562-006618b17c-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.808 DEBUG:boto:b'dkyges-60obj000000000000000004671000falseobj000000000000000004672024-04-12T03:58:34.332Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.808 DEBUG:boto:path=/dkyges-60/ 2024-04-12T03:58:52.808 DEBUG:boto:auth_path=/dkyges-60/ 2024-04-12T03:58:52.808 DEBUG:boto:path=/dkyges-60/?prefix=obj00000000000000000467 2024-04-12T03:58:52.808 DEBUG:boto:auth_path=/dkyges-60/?prefix=obj00000000000000000467 2024-04-12T03:58:52.808 DEBUG:boto:Method: GET 2024-04-12T03:58:52.809 DEBUG:boto:Path: /dkyges-60/?prefix=obj00000000000000000467 2024-04-12T03:58:52.809 DEBUG:boto:Data: 2024-04-12T03:58:52.809 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.809 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.809 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.809 DEBUG:boto:Params: {} 2024-04-12T03:58:52.809 DEBUG:boto:Token: None 2024-04-12T03:58:52.809 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 03:58:52 GMT /dkyges-60/ 2024-04-12T03:58:52.809 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I= 2024-04-12T03:58:52.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fw4hkhv3Rlx1PdhoO610+QMc+2I=', 'Content-Length': '0'} 2024-04-12T03:58:52.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024e6f9a1dc827654-006618b17c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.814 DEBUG:boto:b'dkyges-60obj000000000000000004671000falseobj000000000000000004672024-04-12T03:58:34.332Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T03:58:52.814 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-60 zones={a2, a1} 2024-04-12T03:58:52.815 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ok 2024-04-12T03:58:52.818 DEBUG:boto:path=/dkyges-61/ 2024-04-12T03:58:52.818 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T03:58:52.818 DEBUG:boto:Method: PUT 2024-04-12T03:58:52.818 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T03:58:52.818 DEBUG:boto:Data: 2024-04-12T03:58:52.818 DEBUG:boto:Headers: {} 2024-04-12T03:58:52.818 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T03:58:52.818 DEBUG:boto:Port: 8000 2024-04-12T03:58:52.818 DEBUG:boto:Params: {} 2024-04-12T03:58:52.819 DEBUG:boto:Token: None 2024-04-12T03:58:52.819 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 03:58:52 GMT /dkyges-61/ 2024-04-12T03:58:52.819 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eFP+Ji5+NN2bDT4kXCt5qIz2nNo= 2024-04-12T03:58:52.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:58:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eFP+Ji5+NN2bDT4kXCt5qIz2nNo=', 'Content-Length': '0'} 2024-04-12T03:58:52.838 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a4dc4d5963459d04-006618b17c-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:58:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T03:58:52.838 DEBUG:rgw_multi.tests:created bucket=dkyges-61 2024-04-12T03:58:52.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:52.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:52.839 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:52.897 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:52.897 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:52.935 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T03:58:52.951 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T03:58:52.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:52.951 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:58:52.951 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T03:58:52.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:52.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:58:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "0.000000" 2024-04-12T03:58:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:58:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T03:58:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894332.849566_610.1", 2024-04-12T03:58:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:58:52.849566Z" 2024-04-12T03:58:52.952 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:58:52.953 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T03:58:52.953 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T03:58:52.954 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T03:58:52.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:52.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:52.954 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:53.008 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:53.009 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:53.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:53.055+0000 7fa525982780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:58:53.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:53.056+0000 7fa525982780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:58:53.061 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:58:53.061 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:58:53.061 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:58:53.061 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:58:53.061 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:58:53.061 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:58:53.061 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:58:53.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:53.062 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:58:53.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:53.062 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:58:53.062 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:58:53.062 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:58:53.062 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:58:53.062 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:58:53.062 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:58:53.062 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:58:53.062 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:58:53.062 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:58:53.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:58:53.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:53.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:53.063 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:58:53.063 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:58:53.063 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:58:53.063 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:58:53.063 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:58:53.063 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:58:53.063 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:58:53.063 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:58:53.063 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:58:53.064 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:58:53.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:53.064 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:53.064 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:58:53.064 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:58:53.064 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:58:53.064 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:58:53.064 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:58:53.064 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:58:53.064 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:58:53.064 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:58:53.064 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:58:53.065 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:58:53.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:53.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:53.065 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:58:53.065 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:58:53.065 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:58:53.065 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T03:58:53.065 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:58:53.065 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:58:53.065 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:58:53.066 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T03:58:53.066 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:58:53.066 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:58:53.066 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:58:53.066 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:58:53.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:53.066 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:58:53.066 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:58:53.066 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:58:53.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:53.066 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:58:53Z" 2024-04-12T03:58:53.067 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:58:53.067 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:58:53Z" } 2024-04-12T03:58:53.068 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T03:58:53.068 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T03:58:53.068 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:58:53.068 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T03:58:58.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:58.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:58:58.070 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:58:58.130 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:58:58.130 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:58:58.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:58.179+0000 7fe2f525e780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:58:58.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:58:58.180+0000 7fe2f525e780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:58:58.183 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:58:58.183 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:58:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:58:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:58:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:58:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:58:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:58:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:58:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:58:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:58:58.184 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:58:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:58:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:58:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:58:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:58:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:58:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:58:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:58:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:58:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:58:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:58:58.185 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:58:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:58:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:58:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:58:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:58:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:58:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:58:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:58:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:58:58.186 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:58:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:58:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:58:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:58:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:58:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:58:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:58:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:58:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:58:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:58:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:58:58.187 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:58:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T03:58:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:58:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:58:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:58:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T03:58:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:58:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:58:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:58:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:58:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:58.188 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:58:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:58:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:58:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:58:58.189 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:58:58Z" 2024-04-12T03:58:58.189 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:58:58.190 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:58:58Z" } 2024-04-12T03:58:58.190 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T03:58:58.190 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T03:58:58.190 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:58:58.190 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T03:59:03.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:03.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:03.192 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:59:03.254 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:59:03.254 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:59:03.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:03.305+0000 7fb7eb077780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:03.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:03.306+0000 7fb7eb077780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:03.309 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:59:03.310 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:59:03.310 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:59:03.310 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:59:03.310 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:59:03.310 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:59:03.310 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:03.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:03.310 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:59:03.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:03.311 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:59:03.311 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:03.311 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:03.311 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:59:03.311 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:03.311 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:59:03.311 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:03.311 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:59:03.311 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:03.311 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:03.311 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:03.311 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:03.312 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:59:03.312 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:03.312 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:03.312 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:59:03.312 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:03.312 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:03.312 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:03.312 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:59:03.312 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:03.312 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:03.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:03.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:03.313 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:59:03.313 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:03.313 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:03.313 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:59:03.313 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:03.313 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:03.313 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:03.313 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:59:03.313 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:03.314 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:03.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:03.314 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:03.314 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:59:03.314 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:03.314 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:03.314 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T03:59:03.314 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:03.314 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:03.314 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:03.314 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T03:59:03.315 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:03.315 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:03.315 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:03.315 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:59:03.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:03.315 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:59:03.315 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:59:03.315 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:59:03.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:03.315 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:59:03Z" 2024-04-12T03:59:03.316 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:59:03.316 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:59:03Z" } 2024-04-12T03:59:03.317 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T03:59:03.317 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T03:59:03.317 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:59:03.317 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T03:59:08.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:08.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:08.319 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:59:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:59:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:59:08.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:08.429+0000 7fa715723780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:08.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:08.430+0000 7fa715723780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:08.433 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:59:08.433 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:59:08.433 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:59:08.433 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:59:08.433 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:59:08.433 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:59:08.433 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:08.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:08.433 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:59:08.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:08.434 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:59:08.434 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:08.434 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:08.434 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:59:08.434 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:08.434 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:59:08.434 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:08.434 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:59:08.434 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:08.435 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:08.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:08.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:08.435 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:59:08.435 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:08.435 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:08.435 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:59:08.435 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:08.435 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:08.435 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:08.435 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:59:08.436 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:08.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:08.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:08.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:08.436 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:59:08.436 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:08.436 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:08.436 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:59:08.436 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:08.436 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:08.436 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:08.436 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:59:08.437 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:08.437 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:08.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:08.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:08.437 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:59:08.437 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:08.437 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:08.437 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T03:59:08.437 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:08.437 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:08.437 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:08.438 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T03:59:08.438 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:08.438 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:08.438 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:08.438 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:59:08.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:08.438 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:59:08.438 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:59:08.438 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:59:08.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:08.438 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:59:08Z" 2024-04-12T03:59:08.439 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:59:08.440 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:59:08Z" } 2024-04-12T03:59:08.440 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T03:59:08.440 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T03:59:08.440 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:59:08.440 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T03:59:13.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:13.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:13.442 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:59:13.502 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:59:13.502 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:59:13.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:13.550+0000 7ff024e04780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:13.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:13.551+0000 7ff024e04780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:13.554 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:59:13.554 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:59:13.554 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:59:13.554 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:59:13.555 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:59:13.555 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:59:13.555 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:13.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:13.555 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:59:13.555 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:13.555 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:59:13.555 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:13.555 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:13.555 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:59:13.555 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:13.556 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:59:13.556 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:13.556 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:59:13.556 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:13.556 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:13.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:13.556 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:13.556 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:59:13.556 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:13.556 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:13.557 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:59:13.557 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:13.557 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:13.557 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:13.557 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:59:13.557 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:13.557 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:13.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:13.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:13.557 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:59:13.557 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:13.558 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:13.558 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:59:13.558 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:13.558 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:13.558 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:13.558 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:59:13.558 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:13.558 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:13.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:13.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:13.559 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:59:13.559 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:13.559 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:13.559 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T03:59:13.559 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:13.559 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:13.559 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:13.559 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T03:59:13.559 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:13.559 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:13.559 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:13.559 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:59:13.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:13.560 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:59:13.560 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:59:13.560 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:59:13.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:13.560 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:59:13Z" 2024-04-12T03:59:13.560 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:59:13.561 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:59:13Z" } 2024-04-12T03:59:13.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T03:59:13.561 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T03:59:13.562 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:59:13.562 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T03:59:18.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:18.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:18.564 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:59:18.624 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:59:18.624 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:59:18.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:18.672+0000 7ff2f58da780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:18.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:18.673+0000 7ff2f58da780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:18.677 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:59:18.677 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:59:18.677 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:59:18.677 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:59:18.677 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:59:18.677 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:59:18.678 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:18.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:18.678 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:59:18.678 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:18.678 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:59:18.678 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:18.678 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:18.678 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:59:18.678 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:18.678 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:59:18.679 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:18.679 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:59:18.679 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:18.679 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:18.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:18.679 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:18.679 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:59:18.679 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:18.679 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:18.679 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:59:18.679 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:18.679 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:18.680 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:18.680 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:59:18.680 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:18.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:18.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:18.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:18.680 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:59:18.680 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:18.680 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:18.680 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:59:18.680 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:18.680 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:18.681 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:18.681 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:59:18.681 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:18.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:18.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:18.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:18.681 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:59:18.681 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:18.681 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:18.681 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T03:59:18.681 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:18.681 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:18.682 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:18.682 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T03:59:18.682 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:18.682 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:18.682 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:18.682 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:59:18.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:18.682 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:59:18.682 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:59:18.682 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:59:18.683 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:18.683 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:59:18Z" 2024-04-12T03:59:18.683 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:59:18.684 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:59:18Z" } 2024-04-12T03:59:18.684 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T03:59:18.684 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T03:59:18.684 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:59:18.684 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T03:59:23.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:23.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:23.685 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:59:23.745 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:59:23.745 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:59:23.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:23.794+0000 7f1166a51780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:23.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:23.795+0000 7f1166a51780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:23.798 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:59:23.798 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:59:23.799 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:59:23.799 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:59:23.799 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:59:23.799 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:59:23.799 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:23.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:23.799 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:59:23.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:23.799 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:59:23.799 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:23.799 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:23.800 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:59:23.800 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:23.800 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:59:23.800 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:23.800 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:59:23.800 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:23.800 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:23.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:23.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:23.800 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:59:23.801 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:23.801 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:23.801 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:59:23.801 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:23.801 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:23.801 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:23.801 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:59:23.801 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:23.801 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:23.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:23.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:23.801 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:59:23.802 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:23.802 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:23.802 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:59:23.802 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:23.802 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:23.802 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:23.802 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:59:23.802 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:23.802 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:23.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:23.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:23.803 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:59:23.803 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:23.803 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:23.803 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T03:59:23.803 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:23.803 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:23.803 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:23.803 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T03:59:23.803 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:23.803 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:23.803 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:23.804 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:59:23.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:23.804 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:59:23.804 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:59:23.804 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:59:23.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:23.804 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:59:23Z" 2024-04-12T03:59:23.804 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:59:23.805 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:59:23Z" } 2024-04-12T03:59:23.805 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T03:59:23.806 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T03:59:23.806 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:59:23.806 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T03:59:28.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:28.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:28.807 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:59:28.867 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:59:28.867 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:59:28.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:28.918+0000 7fbd482a3780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:28.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:28.920+0000 7fbd482a3780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:28.923 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:59:28.923 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:59:28.923 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:59:28.923 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:59:28.923 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:59:28.924 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:59:28.924 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:28.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:28.924 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:59:28.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:28.924 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:59:28.924 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:28.924 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:28.924 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:59:28.924 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:28.924 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:59:28.924 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:28.925 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:59:28.925 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:28.925 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:28.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:28.925 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:28.925 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:59:28.925 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:28.925 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:28.925 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:59:28.925 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:28.925 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:28.926 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:28.926 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:59:28.926 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:28.926 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:28.926 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:28.926 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:28.926 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:59:28.926 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:28.926 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:28.926 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:59:28.926 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:28.927 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:28.927 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:28.927 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:59:28.927 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:28.927 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:28.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:28.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:28.927 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:59:28.927 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:28.927 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:28.927 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T03:59:28.927 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:28.927 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:28.928 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:28.928 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T03:59:28.928 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:28.928 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:28.928 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:28.928 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:59:28.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:28.928 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:59:28.928 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:59:28.928 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:59:28.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:28.928 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:59:28Z" 2024-04-12T03:59:28.929 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:59:28.929 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:59:28Z" } 2024-04-12T03:59:28.929 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T03:59:28.930 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T03:59:28.930 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:59:28.930 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T03:59:33.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:33.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:33.932 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:59:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:59:33.990 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:59:34.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:34.038+0000 7f82dab1d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:34.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:34.039+0000 7f82dab1d780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:34.043 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:59:34.043 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:59:34.043 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:59:34.043 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:59:34.043 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:59:34.043 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:59:34.043 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:34.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:34.044 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:59:34.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:34.044 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:59:34.044 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:34.044 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:34.044 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:59:34.044 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:34.044 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:59:34.045 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:34.045 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:59:34.045 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:34.045 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:34.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:34.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:34.045 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:59:34.045 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:34.045 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:34.045 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:59:34.046 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:34.046 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:34.046 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:34.046 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:59:34.046 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:34.046 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:34.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:34.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:34.046 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:59:34.047 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:34.047 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:34.047 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:59:34.047 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:34.047 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:34.047 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:34.047 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:59:34.047 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:34.048 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:34.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:34.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:34.048 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:59:34.048 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:34.048 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:34.048 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T03:59:34.048 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:34.048 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:34.048 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:34.048 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T03:59:34.049 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:34.049 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:34.049 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:34.049 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:59:34.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:34.049 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:59:34.049 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:59:34.049 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:59:34.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:34.049 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:59:34Z" 2024-04-12T03:59:34.050 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:59:34.050 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:59:34Z" } 2024-04-12T03:59:34.051 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T03:59:34.051 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T03:59:34.051 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:59:34.051 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T03:59:39.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:39.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:39.054 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:59:39.114 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:59:39.115 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:59:39.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:39.164+0000 7f0e9bc89780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:39.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:39.165+0000 7f0e9bc89780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:39.170 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:59:39.170 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:59:39.170 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:59:39.170 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:59:39.170 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:59:39.171 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:59:39.171 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:39.171 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:39.171 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:59:39.171 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:39.171 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:59:39.171 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:39.171 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:39.171 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:59:39.171 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:39.171 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:59:39.172 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:39.172 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:59:39.172 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:39.172 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:39.172 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:39.172 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:39.172 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:59:39.172 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:39.172 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:39.173 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:59:39.173 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:39.173 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:39.173 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:39.173 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:59:39.173 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:39.173 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:39.173 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:39.173 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:39.173 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:59:39.174 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:39.174 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:39.174 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:59:39.174 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:39.174 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:39.174 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:39.174 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:59:39.174 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:39.174 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:39.174 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:39.174 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:39.175 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:59:39.175 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:39.175 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:39.175 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T03:59:39.175 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:39.175 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:39.175 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:39.175 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T03:59:39.175 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:39.175 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:39.175 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:39.176 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:59:39.176 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:39.176 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:59:39.176 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:59:39.176 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:59:39.176 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:39.176 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:59:39Z" 2024-04-12T03:59:39.176 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:59:39.177 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:59:39Z" } 2024-04-12T03:59:39.177 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T03:59:39.177 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T03:59:39.178 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:59:39.178 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T03:59:44.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:44.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:44.179 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:59:44.239 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:59:44.240 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:59:44.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:44.288+0000 7f5d90723780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:44.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:44.289+0000 7f5d90723780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:44.292 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:59:44.293 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:59:44.293 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:59:44.293 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:59:44.293 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:59:44.293 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:59:44.293 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:44.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:44.293 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:59:44.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:44.294 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:59:44.294 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:44.294 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:44.294 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:59:44.294 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:44.294 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:59:44.294 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:44.294 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:59:44.294 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:44.294 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:44.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:44.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:44.295 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:59:44.295 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:44.295 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:44.295 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:59:44.295 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:44.295 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:44.295 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:44.295 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:59:44.295 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:44.296 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:44.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:44.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:44.296 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:59:44.296 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:44.296 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:44.296 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:59:44.296 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:44.296 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:44.296 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:44.296 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:59:44.297 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:44.297 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:44.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:44.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:44.297 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:59:44.297 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:44.297 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:44.297 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T03:59:44.297 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:44.297 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:44.297 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:44.298 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T03:59:44.298 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:44.298 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:44.298 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:44.298 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:59:44.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:44.298 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:59:44.298 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:59:44.298 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:59:44.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:44.298 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:59:44Z" 2024-04-12T03:59:44.299 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:59:44.299 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:59:44Z" } 2024-04-12T03:59:44.300 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T03:59:44.300 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T03:59:44.300 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:59:44.300 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T03:59:49.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:49.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:49.302 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:59:49.361 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:59:49.361 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:59:49.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:49.411+0000 7fdd07901780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:49.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:49.412+0000 7fdd07901780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:49.415 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:59:49.415 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:59:49.415 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:59:49.415 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:59:49.416 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:59:49.416 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:59:49.416 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:49.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:49.416 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:59:49.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:49.416 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:59:49.416 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:49.417 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:49.417 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:59:49.417 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:49.417 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:59:49.417 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:49.417 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:59:49.417 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:49.417 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:49.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:49.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:49.418 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:59:49.418 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:49.418 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:49.418 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:59:49.418 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:49.418 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:49.418 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:49.418 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:59:49.419 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:49.419 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:49.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:49.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:49.419 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:59:49.419 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:49.419 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:49.419 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:59:49.419 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:49.419 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:49.419 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:49.420 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:59:49.420 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:49.420 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:49.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:49.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:49.420 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:59:49.420 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:49.420 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:49.420 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T03:59:49.420 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:49.420 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:49.421 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:49.421 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T03:59:49.421 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:49.421 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:49.421 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:49.421 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:59:49.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:49.421 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:59:49.421 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:59:49.421 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:59:49.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:49.422 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:59:49Z" 2024-04-12T03:59:49.422 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:59:49.424 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:59:49Z" } 2024-04-12T03:59:49.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T03:59:49.424 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T03:59:49.425 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:59:49.425 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T03:59:54.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:54.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:54.426 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:59:54.486 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:59:54.486 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:59:54.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:54.535+0000 7fc663f58780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:54.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:54.536+0000 7fc663f58780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:54.539 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:59:54.539 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:59:54.539 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:59:54.539 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:59:54.540 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:59:54.540 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:59:54.540 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:54.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:54.540 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:59:54.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:54.540 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:59:54.540 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:54.540 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:54.540 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:59:54.540 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:54.541 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:59:54.541 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:54.541 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:59:54.541 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:54.541 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:54.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:54.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:54.541 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:59:54.541 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:54.541 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:54.541 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:59:54.542 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:54.542 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:54.542 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:54.542 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:59:54.542 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:54.542 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:54.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:54.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:54.542 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:59:54.542 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:54.543 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:54.543 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:59:54.543 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:54.543 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:54.543 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:54.543 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:59:54.543 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:54.543 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:54.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:54.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:54.543 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:59:54.544 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:54.544 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:54.544 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T03:59:54.544 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:54.544 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:54.544 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:54.544 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T03:59:54.544 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:54.544 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:54.544 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:54.544 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:59:54.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:54.545 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:59:54.545 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:59:54.545 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:59:54.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:54.545 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:59:54Z" 2024-04-12T03:59:54.545 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:59:54.546 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:59:54Z" } 2024-04-12T03:59:54.546 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T03:59:54.546 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T03:59:54.546 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:59:54.546 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T03:59:59.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:59.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T03:59:59.548 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T03:59:59.606 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T03:59:59.607 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T03:59:59.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:59.655+0000 7f77aa0a0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:59.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T03:59:59.656+0000 7f77aa0a0780 20 RGW-SYNC:meta: read sync status 2024-04-12T03:59:59.659 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T03:59:59.659 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T03:59:59.659 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T03:59:59.660 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T03:59:59.660 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T03:59:59.660 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T03:59:59.660 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:59.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:59.660 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T03:59:59.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:59.660 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T03:59:59.660 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:59.660 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:59.660 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T03:59:59.661 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:59.661 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T03:59:59.661 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:59.661 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T03:59:59.661 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:59.661 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:59.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:59.661 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:59.661 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T03:59:59.661 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:59.661 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:59.661 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T03:59:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T03:59:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T03:59:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T03:59:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T03:59:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T03:59:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T03:59:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T03:59:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T03:59:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T03:59:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T03:59:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T03:59:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T03:59:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T03:59:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T03:59:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T03:59:59.665 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T03:59:59.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:59.665 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T03:59:59.665 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T03:59:59.665 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T03:59:59.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T03:59:59.665 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T03:59:59Z" 2024-04-12T03:59:59.665 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T03:59:59.666 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T03:59:59Z" } 2024-04-12T03:59:59.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T03:59:59.666 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T03:59:59.666 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T03:59:59.667 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T04:00:04.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:00:04.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:00:04.668 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:00:04.728 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:00:04.729 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:00:04.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:04.778+0000 7f3d8eb52780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:00:04.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:04.780+0000 7f3d8eb52780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:00:04.782 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:00:04.783 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:00:04.783 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:00:04.783 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:00:04.783 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:00:04.783 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:00:04.783 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:04.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:04.783 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:00:04.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:04.783 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:00:04.783 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:04.784 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:04.784 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:00:04.784 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:04.784 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:00:04.784 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:04.784 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:00:04.784 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:04.784 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:04.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:04.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:04.784 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:00:04.785 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:04.785 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:04.785 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:00:04.785 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:04.785 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:04.785 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:04.785 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:00:04.785 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:04.785 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:04.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:04.786 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:04.786 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:00:04.786 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:04.786 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:04.786 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:00:04.786 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:04.786 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:04.786 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:04.786 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:00:04.786 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:04.787 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:04.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:04.787 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:04.787 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:00:04.787 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:04.787 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:04.787 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T04:00:04.787 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:04.787 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:04.787 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:04.787 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T04:00:04.787 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:04.788 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:04.788 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:04.788 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:00:04.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:04.788 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:00:04.788 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:00:04.788 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:00:04.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:04.788 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:00:04Z" 2024-04-12T04:00:04.788 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:00:04.790 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:00:04Z" } 2024-04-12T04:00:04.790 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T04:00:04.790 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T04:00:04.790 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T04:00:04.790 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T04:00:09.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:00:09.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:00:09.792 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:00:09.854 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:00:09.854 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:00:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:09.902+0000 7fd6dc838780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:00:09.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:09.904+0000 7fd6dc838780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:00:09.906 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:00:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:00:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:00:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:00:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:00:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:00:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:00:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:00:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:00:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:00:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:00:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:00:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:00:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:00:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:00:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:00:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:00:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:00:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T04:00:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T04:00:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:00:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:00:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:00:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:00:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:00:09Z" 2024-04-12T04:00:09.912 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:00:09.913 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:00:09Z" } 2024-04-12T04:00:09.913 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T04:00:09.913 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T04:00:09.913 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T04:00:09.914 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T04:00:14.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:00:14.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:00:14.915 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:00:14.975 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:00:14.975 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:00:15.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:15.024+0000 7f0818b55780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:00:15.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:15.025+0000 7f0818b55780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:00:15.029 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:00:15.030 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:00:15.030 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:00:15.030 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:00:15.030 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:00:15.030 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:00:15.030 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:15.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:15.030 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:00:15.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:15.030 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:00:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:00:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:00:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:00:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:15.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:00:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:00:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:00:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:15.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:00:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:00:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:00:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:15.033 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:00:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T04:00:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T04:00:15.034 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:15.035 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:15.035 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:15.035 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:00:15.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:15.035 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:00:15.035 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:00:15.035 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:00:15.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:15.035 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:00:15Z" 2024-04-12T04:00:15.035 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:00:15.036 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:00:15Z" } 2024-04-12T04:00:15.036 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T04:00:15.037 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T04:00:15.037 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T04:00:15.037 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T04:00:20.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:00:20.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:00:20.038 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:00:20.099 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:00:20.099 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:00:20.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:20.149+0000 7f4fb51a8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:00:20.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:20.150+0000 7f4fb51a8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:00:20.154 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:00:20.154 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:00:20.154 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:00:20.154 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:00:20.154 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:00:20.154 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:00:20.154 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:20.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:20.155 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:00:20.155 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:20.155 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:00:20.155 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:20.155 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:20.155 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:00:20.155 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:20.155 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:00:20.155 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:20.155 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:00:20.155 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:20.156 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:20.156 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:20.156 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:20.156 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:00:20.156 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:20.156 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:20.156 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:00:20.156 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:20.156 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:20.156 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:20.157 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:00:20.157 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:20.157 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:20.157 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:20.157 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:20.157 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:00:20.157 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:20.157 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:20.157 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:00:20.157 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:20.157 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:20.158 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:20.158 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:00:20.158 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:20.158 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:20.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:20.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:20.159 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:00:20.159 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:20.159 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:20.159 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894240.120233_545.1", 2024-04-12T04:00:20.159 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:20.159 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:20.160 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:20.161 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:57:20.120233Z", 2024-04-12T04:00:20.161 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:20.161 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:20.161 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:20.161 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:00:20.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:20.161 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:00:20.161 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:00:20.161 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:00:20.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:20.161 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:00:20Z" 2024-04-12T04:00:20.161 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:00:20.163 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894240.120233_545.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:57:20.120233Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:00:20Z" } 2024-04-12T04:00:20.163 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894240.120233_545.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:57:20.120233Z', 'realm_epoch': 2}}] 2024-04-12T04:00:20.164 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T04:00:20.164 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894240.120233_545.1'} 2024-04-12T04:00:20.164 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894240.120233_545.1 2024-04-12T04:00:25.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:00:25.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:00:25.165 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:00:25.225 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:00:25.225 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:00:25.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:25.275+0000 7fdb9c5cf780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:00:25.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:25.276+0000 7fdb9c5cf780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:00:25.279 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:00:25.279 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:00:25.279 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:00:25.279 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:00:25.280 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:00:25.280 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:00:25.280 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:25.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:25.280 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:00:25.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:25.280 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:00:25.280 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:25.280 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:25.280 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:00:25.280 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:25.281 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:00:25.281 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:25.281 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:00:25.281 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:25.281 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:25.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:25.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:25.281 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:00:25.281 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:25.281 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:25.281 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:00:25.282 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:25.282 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:25.282 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:25.282 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:00:25.282 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:25.282 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:25.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:25.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:25.282 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:00:25.282 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:25.282 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:25.283 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:00:25.283 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:25.283 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:25.283 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:25.283 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:00:25.283 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:25.283 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:25.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:25.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:25.283 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:00:25.283 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:25.284 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:25.284 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894309.612120_549.1", 2024-04-12T04:00:25.284 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:25.284 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:25.284 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:25.284 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:58:29.612120Z", 2024-04-12T04:00:25.284 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:25.284 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:25.284 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:25.284 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:00:25.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:25.285 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:00:25.285 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:00:25.285 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:00:25.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:25.285 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:00:25Z" 2024-04-12T04:00:25.285 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:00:25.286 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894309.612120_549.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:58:29.612120Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:00:25Z" } 2024-04-12T04:00:25.286 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894309.612120_549.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:58:29.612120Z', 'realm_epoch': 2}}] 2024-04-12T04:00:25.286 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T04:00:25.287 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894309.612120_549.1'} 2024-04-12T04:00:25.287 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712894332.849566_610.1 target=1_1712894309.612120_549.1 2024-04-12T04:00:30.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:00:30.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:00:30.288 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:00:30.348 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:00:30.348 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:00:30.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:30.399+0000 7fb7e8dea780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:00:30.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:30.400+0000 7fb7e8dea780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:00:30.403 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:00:30.403 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:00:30.403 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:00:30.403 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:00:30.403 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:00:30.403 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:00:30.403 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:30.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:30.404 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:00:30.404 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:30.404 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:00:30.404 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:30.404 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:30.404 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:00:30.404 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:30.404 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:00:30.404 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:30.404 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:00:30.404 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:30.405 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:30.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:30.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:30.405 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:00:30.405 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:30.405 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:30.405 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:00:30.405 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:30.405 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:30.405 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:30.405 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:00:30.406 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:30.406 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:30.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:30.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:30.406 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:00:30.406 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:30.406 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:30.406 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:00:30.406 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:30.406 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:30.406 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:30.407 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:00:30.407 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:30.407 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:30.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:30.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:00:30.407 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:00:30.407 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:00:30.407 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:00:30.407 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894332.849566_610.1", 2024-04-12T04:00:30.407 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:00:30.407 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:00:30.408 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:00:30.408 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:58:52.849566Z", 2024-04-12T04:00:30.408 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:00:30.408 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:30.408 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:00:30.408 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:00:30.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:30.408 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:00:30.408 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:00:30.408 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:00:30.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:00:30.408 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:00:30Z" 2024-04-12T04:00:30.409 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:00:30.409 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894332.849566_610.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:58:52.849566Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:00:30Z" } 2024-04-12T04:00:30.410 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894332.849566_610.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:58:52.849566Z', 'realm_epoch': 2}}] 2024-04-12T04:00:30.410 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T04:00:30.410 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894332.849566_610.1'} 2024-04-12T04:00:30.410 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:00:30.410 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:30.410 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:30.411 DEBUG:boto:Method: HEAD 2024-04-12T04:00:30.411 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:00:30.411 DEBUG:boto:Data: 2024-04-12T04:00:30.411 DEBUG:boto:Headers: {} 2024-04-12T04:00:30.411 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:30.411 DEBUG:boto:Port: 8000 2024-04-12T04:00:30.411 DEBUG:boto:Params: {} 2024-04-12T04:00:30.412 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:00:30.412 DEBUG:boto:Token: None 2024-04-12T04:00:30.412 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:30 GMT /dkyges-61/ 2024-04-12T04:00:30.412 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W0U5A4jQMH2CQ/tM+Vi0GT9rYJk= 2024-04-12T04:00:30.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W0U5A4jQMH2CQ/tM+Vi0GT9rYJk=', 'Content-Length': '0'} 2024-04-12T04:00:30.425 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a743995a71065c6b-006618b1de-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:30.426 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:00:30.426 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:00:30.426 DEBUG:boto:Method: PUT 2024-04-12T04:00:30.426 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:00:30.427 DEBUG:boto:Data: 2024-04-12T04:00:30.427 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:00:30.427 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:30.427 DEBUG:boto:Port: 8000 2024-04-12T04:00:30.427 DEBUG:boto:Params: {} 2024-04-12T04:00:30.427 DEBUG:boto:Token: None 2024-04-12T04:00:30.428 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:00:30 GMT /dkyges-61/obj11 2024-04-12T04:00:30.428 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ncJZkXgMXdJhUpOsqXDST06r89A= 2024-04-12T04:00:30.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:00:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ncJZkXgMXdJhUpOsqXDST06r89A='} 2024-04-12T04:00:30.436 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ec56157a5dff5b7-006618b1de-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:30.436 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:30.436 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:30.436 DEBUG:boto:Method: HEAD 2024-04-12T04:00:30.436 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:00:30.436 DEBUG:boto:Data: 2024-04-12T04:00:30.436 DEBUG:boto:Headers: {} 2024-04-12T04:00:30.436 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:30.436 DEBUG:boto:Port: 8000 2024-04-12T04:00:30.437 DEBUG:boto:Params: {} 2024-04-12T04:00:30.437 DEBUG:boto:Token: None 2024-04-12T04:00:30.437 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:30 GMT /dkyges-61/ 2024-04-12T04:00:30.437 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W0U5A4jQMH2CQ/tM+Vi0GT9rYJk= 2024-04-12T04:00:30.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W0U5A4jQMH2CQ/tM+Vi0GT9rYJk=', 'Content-Length': '0'} 2024-04-12T04:00:30.439 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8fdef02aab90eef-006618b1de-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:30.439 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:00:30.440 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:00:30.440 DEBUG:boto:Method: PUT 2024-04-12T04:00:30.440 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:00:30.440 DEBUG:boto:Data: 2024-04-12T04:00:30.440 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:00:30.440 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:30.440 DEBUG:boto:Port: 8000 2024-04-12T04:00:30.440 DEBUG:boto:Params: {} 2024-04-12T04:00:30.440 DEBUG:boto:Token: None 2024-04-12T04:00:30.440 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:00:30 GMT /dkyges-61/obj22 2024-04-12T04:00:30.441 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NL4VqEl9GZJ8+ASVwSzC6pHgnJc= 2024-04-12T04:00:30.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:00:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NL4VqEl9GZJ8+ASVwSzC6pHgnJc='} 2024-04-12T04:00:30.445 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003e3b116a36ff9a4-006618b1de-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:30.445 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:30.445 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:30.445 DEBUG:boto:Method: HEAD 2024-04-12T04:00:30.445 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:00:30.445 DEBUG:boto:Data: 2024-04-12T04:00:30.445 DEBUG:boto:Headers: {} 2024-04-12T04:00:30.446 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:30.446 DEBUG:boto:Port: 8000 2024-04-12T04:00:30.446 DEBUG:boto:Params: {} 2024-04-12T04:00:30.446 DEBUG:boto:Token: None 2024-04-12T04:00:30.446 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:30 GMT /dkyges-61/ 2024-04-12T04:00:30.446 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W0U5A4jQMH2CQ/tM+Vi0GT9rYJk= 2024-04-12T04:00:30.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W0U5A4jQMH2CQ/tM+Vi0GT9rYJk=', 'Content-Length': '0'} 2024-04-12T04:00:30.448 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f74f1798f60b430-006618b1de-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:30.449 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:00:30.449 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:00:30.449 DEBUG:boto:Method: PUT 2024-04-12T04:00:30.449 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:00:30.449 DEBUG:boto:Data: 2024-04-12T04:00:30.449 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:00:30.449 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:30.450 DEBUG:boto:Port: 8000 2024-04-12T04:00:30.450 DEBUG:boto:Params: {} 2024-04-12T04:00:30.450 DEBUG:boto:Token: None 2024-04-12T04:00:30.450 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:00:30 GMT /dkyges-61/obj33 2024-04-12T04:00:30.450 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Lx70cd71WQHHiuGNyXEJYT+iM7Y= 2024-04-12T04:00:30.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:00:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Lx70cd71WQHHiuGNyXEJYT+iM7Y='} 2024-04-12T04:00:30.499 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e01c6837518dfce9-006618b1de-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:30.499 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:30.499 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:30.500 DEBUG:boto:Method: HEAD 2024-04-12T04:00:30.500 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:00:30.500 DEBUG:boto:Data: 2024-04-12T04:00:30.500 DEBUG:boto:Headers: {} 2024-04-12T04:00:30.500 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:30.501 DEBUG:boto:Port: 8000 2024-04-12T04:00:30.501 DEBUG:boto:Params: {} 2024-04-12T04:00:30.501 DEBUG:boto:Token: None 2024-04-12T04:00:30.501 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:30 GMT /dkyges-61/ 2024-04-12T04:00:30.502 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W0U5A4jQMH2CQ/tM+Vi0GT9rYJk= 2024-04-12T04:00:30.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W0U5A4jQMH2CQ/tM+Vi0GT9rYJk=', 'Content-Length': '0'} 2024-04-12T04:00:30.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005718f92ef4bc7485-006618b1de-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:30.505 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:00:30.505 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:00:30.505 DEBUG:boto:Method: PUT 2024-04-12T04:00:30.505 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:00:30.505 DEBUG:boto:Data: 2024-04-12T04:00:30.505 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:00:30.505 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:30.505 DEBUG:boto:Port: 8000 2024-04-12T04:00:30.505 DEBUG:boto:Params: {} 2024-04-12T04:00:30.505 DEBUG:boto:Token: None 2024-04-12T04:00:30.506 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:00:30 GMT /dkyges-61/obj44 2024-04-12T04:00:30.506 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j6IR1bBT8XpYClzFBuNoPHIgylE= 2024-04-12T04:00:30.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:00:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j6IR1bBT8XpYClzFBuNoPHIgylE='} 2024-04-12T04:00:30.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3a4818f3bdcf980-006618b1de-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:30.513 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:30.513 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:30.513 DEBUG:boto:Method: HEAD 2024-04-12T04:00:30.513 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:00:30.513 DEBUG:boto:Data: 2024-04-12T04:00:30.513 DEBUG:boto:Headers: {} 2024-04-12T04:00:30.513 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:30.513 DEBUG:boto:Port: 8000 2024-04-12T04:00:30.513 DEBUG:boto:Params: {} 2024-04-12T04:00:30.514 DEBUG:boto:Token: None 2024-04-12T04:00:30.514 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:30 GMT /dkyges-61/ 2024-04-12T04:00:30.514 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W0U5A4jQMH2CQ/tM+Vi0GT9rYJk= 2024-04-12T04:00:30.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W0U5A4jQMH2CQ/tM+Vi0GT9rYJk=', 'Content-Length': '0'} 2024-04-12T04:00:30.516 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e29d8cc520973a87-006618b1de-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:30.517 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:00:30.517 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:00:30.517 DEBUG:boto:Method: PUT 2024-04-12T04:00:30.517 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:00:30.517 DEBUG:boto:Data: 2024-04-12T04:00:30.517 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:00:30.517 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:30.517 DEBUG:boto:Port: 8000 2024-04-12T04:00:30.517 DEBUG:boto:Params: {} 2024-04-12T04:00:30.517 DEBUG:boto:Token: None 2024-04-12T04:00:30.518 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:00:30 GMT /dkyges-61/obj55 2024-04-12T04:00:30.518 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3j99lrswtmQgdMq+peAYoTcHoF4= 2024-04-12T04:00:30.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:00:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3j99lrswtmQgdMq+peAYoTcHoF4='} 2024-04-12T04:00:30.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000176d20ec9d64e38a-006618b1de-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:30.523 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-61 2024-04-12T04:00:30.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:00:30.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:00:30.523 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:00:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:00:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:00:30.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:30.645+0000 7f8cfdde4780 1 waiting to reach incremental sync.. 2024-04-12T04:00:35.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:35.646+0000 7f8cfdde4780 1 waiting to reach incremental sync.. 2024-04-12T04:00:40.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:40.648+0000 7f8cfdde4780 1 bucket sync caught up with source: 2024-04-12T04:00:40.635 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000001.69.6, 00000000002.130.6, 00000000002.81.6] 2024-04-12T04:00:40.635 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000001.69.6, 00000000002.130.6, 00000000002.81.6] 2024-04-12T04:00:40.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:40.648+0000 7f8cfdde4780 0 bucket checkpoint complete 2024-04-12T04:00:40.654 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-61 2024-04-12T04:00:40.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:00:40.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:00:40.655 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:00:40.722 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:00:40.722 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:00:40.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:40.778+0000 7f06fcc51780 1 bucket sync caught up with source: 2024-04-12T04:00:40.766 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000001.2553.6, 00000000002.2272.6, 00000000002.2604.6] 2024-04-12T04:00:40.766 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000001.2553.6, 00000000002.2272.6, 00000000002.2604.6] 2024-04-12T04:00:40.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:40.778+0000 7f06fcc51780 0 bucket checkpoint complete 2024-04-12T04:00:40.782 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-61 zones={a2, a1} 2024-04-12T04:00:40.782 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:40.782 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:40.782 DEBUG:boto:Method: HEAD 2024-04-12T04:00:40.783 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:00:40.783 DEBUG:boto:Data: 2024-04-12T04:00:40.783 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.783 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:40.783 DEBUG:boto:Port: 8001 2024-04-12T04:00:40.783 DEBUG:boto:Params: {} 2024-04-12T04:00:40.784 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:00:40.784 DEBUG:boto:Token: None 2024-04-12T04:00:40.784 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/ 2024-04-12T04:00:40.785 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:npI72MsPvkRx0vmbT808o8uYKcQ= 2024-04-12T04:00:40.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:npI72MsPvkRx0vmbT808o8uYKcQ=', 'Content-Length': '0'} 2024-04-12T04:00:40.801 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065c52112cae9b77a-006618b1e8-1cfd-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.802 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:40.802 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:40.802 DEBUG:boto:Method: HEAD 2024-04-12T04:00:40.802 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:00:40.802 DEBUG:boto:Data: 2024-04-12T04:00:40.802 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.802 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:40.802 DEBUG:boto:Port: 8000 2024-04-12T04:00:40.802 DEBUG:boto:Params: {} 2024-04-12T04:00:40.803 DEBUG:boto:Token: None 2024-04-12T04:00:40.803 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/ 2024-04-12T04:00:40.803 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:npI72MsPvkRx0vmbT808o8uYKcQ= 2024-04-12T04:00:40.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:npI72MsPvkRx0vmbT808o8uYKcQ=', 'Content-Length': '0'} 2024-04-12T04:00:40.805 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7af1a819331adb7-006618b1e8-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.805 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:00:40.806 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:40.806 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:40.806 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:00:40.806 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:00:40.806 DEBUG:boto:Method: GET 2024-04-12T04:00:40.807 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:00:40.807 DEBUG:boto:Data: 2024-04-12T04:00:40.807 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:40.807 DEBUG:boto:Port: 8001 2024-04-12T04:00:40.807 DEBUG:boto:Params: {} 2024-04-12T04:00:40.808 DEBUG:boto:Token: None 2024-04-12T04:00:40.808 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/?versions 2024-04-12T04:00:40.808 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JYSJCP9cYH+WWZxL9mKQbX+iWsM= 2024-04-12T04:00:40.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JYSJCP9cYH+WWZxL9mKQbX+iWsM=', 'Content-Length': '0'} 2024-04-12T04:00:40.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000376998a98da599a1-006618b1e8-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.811 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:40.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:00:40.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:00:40.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:00:40.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:00:40.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:00:40.813 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:00:40.814 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:40.814 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:40.814 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:00:40.814 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:00:40.814 DEBUG:boto:Method: GET 2024-04-12T04:00:40.814 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:00:40.815 DEBUG:boto:Data: 2024-04-12T04:00:40.815 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.815 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:40.815 DEBUG:boto:Port: 8000 2024-04-12T04:00:40.815 DEBUG:boto:Params: {} 2024-04-12T04:00:40.816 DEBUG:boto:Token: None 2024-04-12T04:00:40.816 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/?versions 2024-04-12T04:00:40.816 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JYSJCP9cYH+WWZxL9mKQbX+iWsM= 2024-04-12T04:00:40.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JYSJCP9cYH+WWZxL9mKQbX+iWsM=', 'Content-Length': '0'} 2024-04-12T04:00:40.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002226df80439bc153-006618b1e8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.819 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:40.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:00:40.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:00:40.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:00:40.820 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:00:40.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:00:40.821 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:40.821 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:40.821 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:00:40.822 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:00:40.822 DEBUG:boto:Method: GET 2024-04-12T04:00:40.822 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:00:40.822 DEBUG:boto:Data: 2024-04-12T04:00:40.822 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.823 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:40.823 DEBUG:boto:Port: 8001 2024-04-12T04:00:40.823 DEBUG:boto:Params: {} 2024-04-12T04:00:40.824 DEBUG:boto:Token: None 2024-04-12T04:00:40.824 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/?versions 2024-04-12T04:00:40.824 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JYSJCP9cYH+WWZxL9mKQbX+iWsM= 2024-04-12T04:00:40.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JYSJCP9cYH+WWZxL9mKQbX+iWsM=', 'Content-Length': '0'} 2024-04-12T04:00:40.828 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ac97bd0589be615-006618b1e8-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.828 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:40.829 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:40.829 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:40.829 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:00:40.829 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:00:40.829 DEBUG:boto:Method: GET 2024-04-12T04:00:40.830 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:00:40.830 DEBUG:boto:Data: 2024-04-12T04:00:40.830 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.830 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:40.830 DEBUG:boto:Port: 8000 2024-04-12T04:00:40.830 DEBUG:boto:Params: {} 2024-04-12T04:00:40.831 DEBUG:boto:Token: None 2024-04-12T04:00:40.831 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/?versions 2024-04-12T04:00:40.831 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JYSJCP9cYH+WWZxL9mKQbX+iWsM= 2024-04-12T04:00:40.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JYSJCP9cYH+WWZxL9mKQbX+iWsM=', 'Content-Length': '0'} 2024-04-12T04:00:40.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dcdea035c47cc82-006618b1e8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.834 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:40.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:00:40.834 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:00:40.835 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:00:40.835 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.835 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.835 DEBUG:boto:Method: GET 2024-04-12T04:00:40.835 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.835 DEBUG:boto:Data: 2024-04-12T04:00:40.835 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.835 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:40.835 DEBUG:boto:Port: 8001 2024-04-12T04:00:40.835 DEBUG:boto:Params: {} 2024-04-12T04:00:40.835 DEBUG:boto:Token: None 2024-04-12T04:00:40.836 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.836 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fvmW5IqQhDAlTLkKVE/j642Z8z8= 2024-04-12T04:00:40.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fvmW5IqQhDAlTLkKVE/j642Z8z8=', 'Content-Length': '0'} 2024-04-12T04:00:40.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000890fcb73b3af4204-006618b1e8-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.838 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:00:40.838 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:00:40.838 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.838 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.838 DEBUG:boto:Method: GET 2024-04-12T04:00:40.838 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.838 DEBUG:boto:Data: 2024-04-12T04:00:40.838 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.838 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:40.839 DEBUG:boto:Port: 8000 2024-04-12T04:00:40.839 DEBUG:boto:Params: {} 2024-04-12T04:00:40.839 DEBUG:boto:Token: None 2024-04-12T04:00:40.839 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.839 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fvmW5IqQhDAlTLkKVE/j642Z8z8= 2024-04-12T04:00:40.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fvmW5IqQhDAlTLkKVE/j642Z8z8=', 'Content-Length': '0'} 2024-04-12T04:00:40.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024f638ccae3db9d2-006618b1e8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.886 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:00:40.887 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:00:40.887 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.887 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.887 DEBUG:boto:Method: HEAD 2024-04-12T04:00:40.887 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.887 DEBUG:boto:Data: 2024-04-12T04:00:40.887 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.888 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:40.888 DEBUG:boto:Port: 8001 2024-04-12T04:00:40.888 DEBUG:boto:Params: {} 2024-04-12T04:00:40.888 DEBUG:boto:Token: None 2024-04-12T04:00:40.888 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.888 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a6/d2TIHJTsgKCo/EIFDBRPu0g4= 2024-04-12T04:00:40.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a6/d2TIHJTsgKCo/EIFDBRPu0g4=', 'Content-Length': '0'} 2024-04-12T04:00:40.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1db68974ff4d7fb-006618b1e8-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.891 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:00:40.891 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:00:40.891 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.891 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.891 DEBUG:boto:Method: HEAD 2024-04-12T04:00:40.891 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.891 DEBUG:boto:Data: 2024-04-12T04:00:40.891 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.891 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:40.891 DEBUG:boto:Port: 8000 2024-04-12T04:00:40.891 DEBUG:boto:Params: {} 2024-04-12T04:00:40.892 DEBUG:boto:Token: None 2024-04-12T04:00:40.892 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.892 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a6/d2TIHJTsgKCo/EIFDBRPu0g4= 2024-04-12T04:00:40.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a6/d2TIHJTsgKCo/EIFDBRPu0g4=', 'Content-Length': '0'} 2024-04-12T04:00:40.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003818e15cf2857f5-006618b1e8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:00:40.894 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:00:40.894 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:00:40.894 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.894 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.895 DEBUG:boto:Method: GET 2024-04-12T04:00:40.895 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.895 DEBUG:boto:Data: 2024-04-12T04:00:40.895 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.895 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:40.895 DEBUG:boto:Port: 8001 2024-04-12T04:00:40.895 DEBUG:boto:Params: {} 2024-04-12T04:00:40.895 DEBUG:boto:Token: None 2024-04-12T04:00:40.896 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.896 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fvmW5IqQhDAlTLkKVE/j642Z8z8= 2024-04-12T04:00:40.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fvmW5IqQhDAlTLkKVE/j642Z8z8=', 'Content-Length': '0'} 2024-04-12T04:00:40.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eaced759b81d04a-006618b1e8-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.898 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:00:40.898 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:00:40.898 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.899 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.899 DEBUG:boto:Method: GET 2024-04-12T04:00:40.899 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.899 DEBUG:boto:Data: 2024-04-12T04:00:40.899 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.899 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:40.899 DEBUG:boto:Port: 8000 2024-04-12T04:00:40.899 DEBUG:boto:Params: {} 2024-04-12T04:00:40.900 DEBUG:boto:Token: None 2024-04-12T04:00:40.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:00:40.900 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fvmW5IqQhDAlTLkKVE/j642Z8z8= 2024-04-12T04:00:40.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fvmW5IqQhDAlTLkKVE/j642Z8z8=', 'Content-Length': '0'} 2024-04-12T04:00:40.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000657826ced0989383-006618b1e8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.902 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:40.902 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:40.902 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:00:40.902 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:00:40.902 DEBUG:boto:Method: GET 2024-04-12T04:00:40.902 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:00:40.902 DEBUG:boto:Data: 2024-04-12T04:00:40.903 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.903 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:40.903 DEBUG:boto:Port: 8001 2024-04-12T04:00:40.903 DEBUG:boto:Params: {} 2024-04-12T04:00:40.903 DEBUG:boto:Token: None 2024-04-12T04:00:40.906 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/ 2024-04-12T04:00:40.906 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dyDjVVXZQvvdREBIXGC09CQveNU= 2024-04-12T04:00:40.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dyDjVVXZQvvdREBIXGC09CQveNU=', 'Content-Length': '0'} 2024-04-12T04:00:40.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000dd8c451ed21d7b-006618b1e8-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.909 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:40.910 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:40.910 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:40.911 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:00:40.911 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:00:40.911 DEBUG:boto:Method: GET 2024-04-12T04:00:40.911 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:00:40.911 DEBUG:boto:Data: 2024-04-12T04:00:40.911 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.911 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:40.911 DEBUG:boto:Port: 8000 2024-04-12T04:00:40.911 DEBUG:boto:Params: {} 2024-04-12T04:00:40.911 DEBUG:boto:Token: None 2024-04-12T04:00:40.912 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/ 2024-04-12T04:00:40.912 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dyDjVVXZQvvdREBIXGC09CQveNU= 2024-04-12T04:00:40.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dyDjVVXZQvvdREBIXGC09CQveNU=', 'Content-Length': '0'} 2024-04-12T04:00:40.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024e28dd91813b4db-006618b1e8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.914 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:40.915 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:00:40.915 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:00:40.915 DEBUG:boto:Method: HEAD 2024-04-12T04:00:40.915 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:00:40.915 DEBUG:boto:Data: 2024-04-12T04:00:40.915 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.915 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:40.915 DEBUG:boto:Port: 8001 2024-04-12T04:00:40.915 DEBUG:boto:Params: {} 2024-04-12T04:00:40.915 DEBUG:boto:Token: None 2024-04-12T04:00:40.915 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/obj11 2024-04-12T04:00:40.916 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fcYoSqgs/geJBkiMy92PfoPBkiM= 2024-04-12T04:00:40.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fcYoSqgs/geJBkiMy92PfoPBkiM=', 'Content-Length': '0'} 2024-04-12T04:00:40.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000296c9cd20da5787d-006618b1e8-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.919 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:00:40.919 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:00:40.919 DEBUG:boto:Method: HEAD 2024-04-12T04:00:40.919 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:00:40.919 DEBUG:boto:Data: 2024-04-12T04:00:40.919 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.920 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:40.920 DEBUG:boto:Port: 8000 2024-04-12T04:00:40.920 DEBUG:boto:Params: {} 2024-04-12T04:00:40.920 DEBUG:boto:Token: None 2024-04-12T04:00:40.920 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/obj11 2024-04-12T04:00:40.920 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fcYoSqgs/geJBkiMy92PfoPBkiM= 2024-04-12T04:00:40.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fcYoSqgs/geJBkiMy92PfoPBkiM=', 'Content-Length': '0'} 2024-04-12T04:00:40.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035896d314c9a9f05-006618b1e8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:00:40.922 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:00:40.922 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:00:40.922 DEBUG:boto:Method: GET 2024-04-12T04:00:40.922 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:00:40.922 DEBUG:boto:Data: 2024-04-12T04:00:40.923 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.923 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:40.923 DEBUG:boto:Port: 8001 2024-04-12T04:00:40.923 DEBUG:boto:Params: {} 2024-04-12T04:00:40.923 DEBUG:boto:Token: None 2024-04-12T04:00:40.923 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/obj11 2024-04-12T04:00:40.923 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Nvbd+yQ6waimZLc6gx7U7bOyoe8= 2024-04-12T04:00:40.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Nvbd+yQ6waimZLc6gx7U7bOyoe8=', 'Content-Length': '0'} 2024-04-12T04:00:40.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1c5d0419cc38190-006618b1e8-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.925 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:00:40.925 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:00:40.925 DEBUG:boto:Method: GET 2024-04-12T04:00:40.925 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:00:40.925 DEBUG:boto:Data: 2024-04-12T04:00:40.926 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.926 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:40.926 DEBUG:boto:Port: 8000 2024-04-12T04:00:40.926 DEBUG:boto:Params: {} 2024-04-12T04:00:40.926 DEBUG:boto:Token: None 2024-04-12T04:00:40.926 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/obj11 2024-04-12T04:00:40.926 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Nvbd+yQ6waimZLc6gx7U7bOyoe8= 2024-04-12T04:00:40.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Nvbd+yQ6waimZLc6gx7U7bOyoe8=', 'Content-Length': '0'} 2024-04-12T04:00:40.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009552f2329bec3ece-006618b1e8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.928 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:40.929 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:40.929 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:00:40.929 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:00:40.929 DEBUG:boto:Method: GET 2024-04-12T04:00:40.929 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:00:40.929 DEBUG:boto:Data: 2024-04-12T04:00:40.929 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.929 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:40.929 DEBUG:boto:Port: 8001 2024-04-12T04:00:40.929 DEBUG:boto:Params: {} 2024-04-12T04:00:40.929 DEBUG:boto:Token: None 2024-04-12T04:00:40.930 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/ 2024-04-12T04:00:40.930 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dyDjVVXZQvvdREBIXGC09CQveNU= 2024-04-12T04:00:40.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dyDjVVXZQvvdREBIXGC09CQveNU=', 'Content-Length': '0'} 2024-04-12T04:00:40.932 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7e193497595e923-006618b1e8-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.932 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:40.932 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:40.932 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:40.933 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:00:40.933 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:00:40.933 DEBUG:boto:Method: GET 2024-04-12T04:00:40.933 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:00:40.933 DEBUG:boto:Data: 2024-04-12T04:00:40.933 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.933 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:40.933 DEBUG:boto:Port: 8000 2024-04-12T04:00:40.933 DEBUG:boto:Params: {} 2024-04-12T04:00:40.933 DEBUG:boto:Token: None 2024-04-12T04:00:40.933 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/ 2024-04-12T04:00:40.934 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dyDjVVXZQvvdREBIXGC09CQveNU= 2024-04-12T04:00:40.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dyDjVVXZQvvdREBIXGC09CQveNU=', 'Content-Length': '0'} 2024-04-12T04:00:40.936 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e90e229d99d2ebb-006618b1e8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.936 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:40.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:00:40.937 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:00:40.937 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:00:40.937 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.937 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.937 DEBUG:boto:Method: GET 2024-04-12T04:00:40.937 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.937 DEBUG:boto:Data: 2024-04-12T04:00:40.937 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.938 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:40.938 DEBUG:boto:Port: 8001 2024-04-12T04:00:40.938 DEBUG:boto:Params: {} 2024-04-12T04:00:40.938 DEBUG:boto:Token: None 2024-04-12T04:00:40.938 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.938 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rndjKtmxntQXjp23+LIVnl7hST4= 2024-04-12T04:00:40.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rndjKtmxntQXjp23+LIVnl7hST4=', 'Content-Length': '0'} 2024-04-12T04:00:40.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005676b94f3f269ec4-006618b1e8-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.940 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:00:40.940 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:00:40.940 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.941 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.941 DEBUG:boto:Method: GET 2024-04-12T04:00:40.941 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.941 DEBUG:boto:Data: 2024-04-12T04:00:40.941 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.941 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:40.941 DEBUG:boto:Port: 8000 2024-04-12T04:00:40.941 DEBUG:boto:Params: {} 2024-04-12T04:00:40.941 DEBUG:boto:Token: None 2024-04-12T04:00:40.941 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.942 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rndjKtmxntQXjp23+LIVnl7hST4= 2024-04-12T04:00:40.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rndjKtmxntQXjp23+LIVnl7hST4=', 'Content-Length': '0'} 2024-04-12T04:00:40.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c6e99c970ab419e-006618b1e8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.944 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:00:40.944 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:00:40.944 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.944 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.944 DEBUG:boto:Method: HEAD 2024-04-12T04:00:40.944 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.944 DEBUG:boto:Data: 2024-04-12T04:00:40.944 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.944 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:40.945 DEBUG:boto:Port: 8001 2024-04-12T04:00:40.945 DEBUG:boto:Params: {} 2024-04-12T04:00:40.945 DEBUG:boto:Token: None 2024-04-12T04:00:40.945 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.945 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qB1fIzwfNwXA4nc+wEP2QTTvyF8= 2024-04-12T04:00:40.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qB1fIzwfNwXA4nc+wEP2QTTvyF8=', 'Content-Length': '0'} 2024-04-12T04:00:40.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc9c132c82f0b2ba-006618b1e8-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.947 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:00:40.947 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:00:40.947 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.947 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.947 DEBUG:boto:Method: HEAD 2024-04-12T04:00:40.947 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.947 DEBUG:boto:Data: 2024-04-12T04:00:40.947 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.948 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:40.948 DEBUG:boto:Port: 8000 2024-04-12T04:00:40.948 DEBUG:boto:Params: {} 2024-04-12T04:00:40.948 DEBUG:boto:Token: None 2024-04-12T04:00:40.948 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.948 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qB1fIzwfNwXA4nc+wEP2QTTvyF8= 2024-04-12T04:00:40.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qB1fIzwfNwXA4nc+wEP2QTTvyF8=', 'Content-Length': '0'} 2024-04-12T04:00:40.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e66d9c80b2e054c-006618b1e8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:00:40.950 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:00:40.950 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:00:40.950 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.950 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.951 DEBUG:boto:Method: GET 2024-04-12T04:00:40.951 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.951 DEBUG:boto:Data: 2024-04-12T04:00:40.951 DEBUG:boto:Headers: {} 2024-04-12T04:00:40.951 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:40.951 DEBUG:boto:Port: 8001 2024-04-12T04:00:40.951 DEBUG:boto:Params: {} 2024-04-12T04:00:40.951 DEBUG:boto:Token: None 2024-04-12T04:00:40.951 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:40 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.951 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rndjKtmxntQXjp23+LIVnl7hST4= 2024-04-12T04:00:40.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rndjKtmxntQXjp23+LIVnl7hST4=', 'Content-Length': '0'} 2024-04-12T04:00:40.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3cc0865c149130a-006618b1e8-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:40.998 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:00:40.999 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:00:40.999 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.999 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:00:40.999 DEBUG:boto:Method: GET 2024-04-12T04:00:41.000 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:00:41.000 DEBUG:boto:Data: 2024-04-12T04:00:41.000 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.000 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.000 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.001 DEBUG:boto:Params: {} 2024-04-12T04:00:41.001 DEBUG:boto:Token: None 2024-04-12T04:00:41.001 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:00:41.001 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5T0xmbdzBtoYOGGYf/Ch3eoaCD0= 2024-04-12T04:00:41.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5T0xmbdzBtoYOGGYf/Ch3eoaCD0=', 'Content-Length': '0'} 2024-04-12T04:00:41.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbb32d7af214eade-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.007 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.007 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.007 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:00:41.007 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:00:41.007 DEBUG:boto:Method: GET 2024-04-12T04:00:41.007 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:00:41.007 DEBUG:boto:Data: 2024-04-12T04:00:41.007 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.007 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.008 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.008 DEBUG:boto:Params: {} 2024-04-12T04:00:41.008 DEBUG:boto:Token: None 2024-04-12T04:00:41.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.008 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc= 2024-04-12T04:00:41.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc=', 'Content-Length': '0'} 2024-04-12T04:00:41.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000748a41946f88358-006618b1e9-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.011 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:41.011 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.011 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.011 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:00:41.011 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:00:41.011 DEBUG:boto:Method: GET 2024-04-12T04:00:41.011 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:00:41.012 DEBUG:boto:Data: 2024-04-12T04:00:41.012 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.012 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.012 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.012 DEBUG:boto:Params: {} 2024-04-12T04:00:41.012 DEBUG:boto:Token: None 2024-04-12T04:00:41.012 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.012 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc= 2024-04-12T04:00:41.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc=', 'Content-Length': '0'} 2024-04-12T04:00:41.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000922d69dafe4e0bc9-006618b1e9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.015 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:41.015 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:00:41.016 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:00:41.016 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.016 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:00:41.016 DEBUG:boto:Data: 2024-04-12T04:00:41.016 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.016 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.016 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.016 DEBUG:boto:Params: {} 2024-04-12T04:00:41.016 DEBUG:boto:Token: None 2024-04-12T04:00:41.016 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj22 2024-04-12T04:00:41.017 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C1fM1xTQaA7KuXHkL9DfRzboyfg= 2024-04-12T04:00:41.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C1fM1xTQaA7KuXHkL9DfRzboyfg=', 'Content-Length': '0'} 2024-04-12T04:00:41.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc63a177f070e9b5-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.018 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:00:41.019 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:00:41.019 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.019 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:00:41.019 DEBUG:boto:Data: 2024-04-12T04:00:41.019 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.019 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.019 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.019 DEBUG:boto:Params: {} 2024-04-12T04:00:41.019 DEBUG:boto:Token: None 2024-04-12T04:00:41.019 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj22 2024-04-12T04:00:41.019 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C1fM1xTQaA7KuXHkL9DfRzboyfg= 2024-04-12T04:00:41.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C1fM1xTQaA7KuXHkL9DfRzboyfg=', 'Content-Length': '0'} 2024-04-12T04:00:41.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d5d8ff75db9b100-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:00:41.022 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:00:41.022 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:00:41.022 DEBUG:boto:Method: GET 2024-04-12T04:00:41.022 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:00:41.022 DEBUG:boto:Data: 2024-04-12T04:00:41.022 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.022 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.022 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.022 DEBUG:boto:Params: {} 2024-04-12T04:00:41.023 DEBUG:boto:Token: None 2024-04-12T04:00:41.023 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj22 2024-04-12T04:00:41.023 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WkscIYIB82W/i+MGAkDs9KGiKKA= 2024-04-12T04:00:41.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WkscIYIB82W/i+MGAkDs9KGiKKA=', 'Content-Length': '0'} 2024-04-12T04:00:41.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cac27e7a2310930-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.070 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:00:41.071 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:00:41.071 DEBUG:boto:Method: GET 2024-04-12T04:00:41.071 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:00:41.071 DEBUG:boto:Data: 2024-04-12T04:00:41.071 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.071 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.071 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.071 DEBUG:boto:Params: {} 2024-04-12T04:00:41.072 DEBUG:boto:Token: None 2024-04-12T04:00:41.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj22 2024-04-12T04:00:41.072 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WkscIYIB82W/i+MGAkDs9KGiKKA= 2024-04-12T04:00:41.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WkscIYIB82W/i+MGAkDs9KGiKKA=', 'Content-Length': '0'} 2024-04-12T04:00:41.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c69c21cb4e806b6-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.075 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.075 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.075 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:00:41.075 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:00:41.075 DEBUG:boto:Method: GET 2024-04-12T04:00:41.076 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:00:41.076 DEBUG:boto:Data: 2024-04-12T04:00:41.076 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.076 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.076 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.077 DEBUG:boto:Params: {} 2024-04-12T04:00:41.077 DEBUG:boto:Token: None 2024-04-12T04:00:41.077 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.077 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc= 2024-04-12T04:00:41.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc=', 'Content-Length': '0'} 2024-04-12T04:00:41.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000162ee4f868b9b1bc-006618b1e9-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.080 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:41.081 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.081 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.081 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:00:41.081 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:00:41.081 DEBUG:boto:Method: GET 2024-04-12T04:00:41.081 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:00:41.081 DEBUG:boto:Data: 2024-04-12T04:00:41.081 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.082 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.082 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.082 DEBUG:boto:Params: {} 2024-04-12T04:00:41.082 DEBUG:boto:Token: None 2024-04-12T04:00:41.082 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.082 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc= 2024-04-12T04:00:41.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc=', 'Content-Length': '0'} 2024-04-12T04:00:41.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b55f900c5458d058-006618b1e9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.130 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:41.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:00:41.131 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:00:41.131 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:00:41.132 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.132 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.133 DEBUG:boto:Method: GET 2024-04-12T04:00:41.133 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.133 DEBUG:boto:Data: 2024-04-12T04:00:41.133 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.133 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.133 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.134 DEBUG:boto:Params: {} 2024-04-12T04:00:41.134 DEBUG:boto:Token: None 2024-04-12T04:00:41.134 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.134 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4G9PN4lKjT6FlzzOAvpi2lWTHxU= 2024-04-12T04:00:41.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4G9PN4lKjT6FlzzOAvpi2lWTHxU=', 'Content-Length': '0'} 2024-04-12T04:00:41.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dba04c00033d69c4-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.137 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:00:41.137 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:00:41.137 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.137 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.138 DEBUG:boto:Method: GET 2024-04-12T04:00:41.138 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.138 DEBUG:boto:Data: 2024-04-12T04:00:41.138 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.138 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.138 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.138 DEBUG:boto:Params: {} 2024-04-12T04:00:41.139 DEBUG:boto:Token: None 2024-04-12T04:00:41.139 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.139 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4G9PN4lKjT6FlzzOAvpi2lWTHxU= 2024-04-12T04:00:41.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4G9PN4lKjT6FlzzOAvpi2lWTHxU=', 'Content-Length': '0'} 2024-04-12T04:00:41.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074fc086349271780-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.142 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:00:41.142 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:00:41.142 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.142 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.142 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.142 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.142 DEBUG:boto:Data: 2024-04-12T04:00:41.142 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.142 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.143 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.143 DEBUG:boto:Params: {} 2024-04-12T04:00:41.143 DEBUG:boto:Token: None 2024-04-12T04:00:41.143 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.143 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QmvczU6LLeQ6j4XhKtGr6CkRUZw= 2024-04-12T04:00:41.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QmvczU6LLeQ6j4XhKtGr6CkRUZw=', 'Content-Length': '0'} 2024-04-12T04:00:41.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000722a98c84c8080ce-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.145 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:00:41.145 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:00:41.145 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.145 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.146 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.146 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.146 DEBUG:boto:Data: 2024-04-12T04:00:41.146 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.146 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.146 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.146 DEBUG:boto:Params: {} 2024-04-12T04:00:41.147 DEBUG:boto:Token: None 2024-04-12T04:00:41.147 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.147 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QmvczU6LLeQ6j4XhKtGr6CkRUZw= 2024-04-12T04:00:41.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QmvczU6LLeQ6j4XhKtGr6CkRUZw=', 'Content-Length': '0'} 2024-04-12T04:00:41.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000daf380d9e556c16-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:00:41.149 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:00:41.149 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:00:41.149 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.149 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.149 DEBUG:boto:Method: GET 2024-04-12T04:00:41.150 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.150 DEBUG:boto:Data: 2024-04-12T04:00:41.150 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.150 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.150 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.150 DEBUG:boto:Params: {} 2024-04-12T04:00:41.150 DEBUG:boto:Token: None 2024-04-12T04:00:41.150 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.151 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4G9PN4lKjT6FlzzOAvpi2lWTHxU= 2024-04-12T04:00:41.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4G9PN4lKjT6FlzzOAvpi2lWTHxU=', 'Content-Length': '0'} 2024-04-12T04:00:41.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080e8f1da771ac6c8-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.153 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:00:41.153 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:00:41.153 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.153 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.153 DEBUG:boto:Method: GET 2024-04-12T04:00:41.153 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.153 DEBUG:boto:Data: 2024-04-12T04:00:41.153 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.153 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.153 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.154 DEBUG:boto:Params: {} 2024-04-12T04:00:41.154 DEBUG:boto:Token: None 2024-04-12T04:00:41.154 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:00:41.154 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4G9PN4lKjT6FlzzOAvpi2lWTHxU= 2024-04-12T04:00:41.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4G9PN4lKjT6FlzzOAvpi2lWTHxU=', 'Content-Length': '0'} 2024-04-12T04:00:41.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efee5c4ed426e414-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.156 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.156 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.156 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:00:41.156 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:00:41.157 DEBUG:boto:Method: GET 2024-04-12T04:00:41.157 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:00:41.157 DEBUG:boto:Data: 2024-04-12T04:00:41.157 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.157 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.157 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.157 DEBUG:boto:Params: {} 2024-04-12T04:00:41.157 DEBUG:boto:Token: None 2024-04-12T04:00:41.158 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.158 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc= 2024-04-12T04:00:41.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc=', 'Content-Length': '0'} 2024-04-12T04:00:41.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf0ac7413d16ee95-006618b1e9-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.161 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:41.161 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.162 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.162 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:00:41.162 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:00:41.162 DEBUG:boto:Method: GET 2024-04-12T04:00:41.162 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:00:41.162 DEBUG:boto:Data: 2024-04-12T04:00:41.162 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.162 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.162 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.162 DEBUG:boto:Params: {} 2024-04-12T04:00:41.163 DEBUG:boto:Token: None 2024-04-12T04:00:41.163 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.163 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc= 2024-04-12T04:00:41.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc=', 'Content-Length': '0'} 2024-04-12T04:00:41.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038eee391b3823be6-006618b1e9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.165 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:41.166 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:00:41.166 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:00:41.166 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.166 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:00:41.166 DEBUG:boto:Data: 2024-04-12T04:00:41.166 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.166 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.166 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.166 DEBUG:boto:Params: {} 2024-04-12T04:00:41.166 DEBUG:boto:Token: None 2024-04-12T04:00:41.167 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj33 2024-04-12T04:00:41.167 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OkyDLTnaOobF87pLd7F8mecJ/wI= 2024-04-12T04:00:41.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OkyDLTnaOobF87pLd7F8mecJ/wI=', 'Content-Length': '0'} 2024-04-12T04:00:41.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1778b8e98fb99a0-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.169 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:00:41.169 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:00:41.169 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.169 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:00:41.169 DEBUG:boto:Data: 2024-04-12T04:00:41.169 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.170 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.170 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.170 DEBUG:boto:Params: {} 2024-04-12T04:00:41.170 DEBUG:boto:Token: None 2024-04-12T04:00:41.170 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj33 2024-04-12T04:00:41.170 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OkyDLTnaOobF87pLd7F8mecJ/wI= 2024-04-12T04:00:41.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OkyDLTnaOobF87pLd7F8mecJ/wI=', 'Content-Length': '0'} 2024-04-12T04:00:41.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007e2559f193d66b2-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:00:41.172 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:00:41.172 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:00:41.173 DEBUG:boto:Method: GET 2024-04-12T04:00:41.173 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:00:41.173 DEBUG:boto:Data: 2024-04-12T04:00:41.173 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.173 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.173 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.173 DEBUG:boto:Params: {} 2024-04-12T04:00:41.173 DEBUG:boto:Token: None 2024-04-12T04:00:41.173 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj33 2024-04-12T04:00:41.173 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6G79NL1HIH5vp82wUVm5uphe+BI= 2024-04-12T04:00:41.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6G79NL1HIH5vp82wUVm5uphe+BI=', 'Content-Length': '0'} 2024-04-12T04:00:41.175 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a727d839b52dee0-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.175 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:00:41.176 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:00:41.176 DEBUG:boto:Method: GET 2024-04-12T04:00:41.176 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:00:41.176 DEBUG:boto:Data: 2024-04-12T04:00:41.176 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.176 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.176 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.176 DEBUG:boto:Params: {} 2024-04-12T04:00:41.176 DEBUG:boto:Token: None 2024-04-12T04:00:41.176 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj33 2024-04-12T04:00:41.177 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6G79NL1HIH5vp82wUVm5uphe+BI= 2024-04-12T04:00:41.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6G79NL1HIH5vp82wUVm5uphe+BI=', 'Content-Length': '0'} 2024-04-12T04:00:41.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4105bf06bc96ae8-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.179 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.179 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.179 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:00:41.179 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:00:41.179 DEBUG:boto:Method: GET 2024-04-12T04:00:41.179 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:00:41.179 DEBUG:boto:Data: 2024-04-12T04:00:41.179 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.180 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.180 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.180 DEBUG:boto:Params: {} 2024-04-12T04:00:41.180 DEBUG:boto:Token: None 2024-04-12T04:00:41.180 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.180 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc= 2024-04-12T04:00:41.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc=', 'Content-Length': '0'} 2024-04-12T04:00:41.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca43ed7fba2e4550-006618b1e9-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.183 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:41.183 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.183 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.183 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:00:41.183 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:00:41.183 DEBUG:boto:Method: GET 2024-04-12T04:00:41.183 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:00:41.184 DEBUG:boto:Data: 2024-04-12T04:00:41.184 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.184 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.184 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.184 DEBUG:boto:Params: {} 2024-04-12T04:00:41.184 DEBUG:boto:Token: None 2024-04-12T04:00:41.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.184 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc= 2024-04-12T04:00:41.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc=', 'Content-Length': '0'} 2024-04-12T04:00:41.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa0ca9d0c292dc09-006618b1e9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.187 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:41.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:00:41.188 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:00:41.188 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:00:41.188 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.188 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.189 DEBUG:boto:Method: GET 2024-04-12T04:00:41.189 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.189 DEBUG:boto:Data: 2024-04-12T04:00:41.189 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.189 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.189 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.190 DEBUG:boto:Params: {} 2024-04-12T04:00:41.190 DEBUG:boto:Token: None 2024-04-12T04:00:41.190 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.190 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L2fuL9qsGEMLdQMe/jgjoI6emwc= 2024-04-12T04:00:41.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L2fuL9qsGEMLdQMe/jgjoI6emwc=', 'Content-Length': '0'} 2024-04-12T04:00:41.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe759a893c16df64-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.234 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:00:41.234 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:00:41.235 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.235 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.235 DEBUG:boto:Method: GET 2024-04-12T04:00:41.235 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.235 DEBUG:boto:Data: 2024-04-12T04:00:41.235 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.235 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.235 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.235 DEBUG:boto:Params: {} 2024-04-12T04:00:41.235 DEBUG:boto:Token: None 2024-04-12T04:00:41.236 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.236 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L2fuL9qsGEMLdQMe/jgjoI6emwc= 2024-04-12T04:00:41.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L2fuL9qsGEMLdQMe/jgjoI6emwc=', 'Content-Length': '0'} 2024-04-12T04:00:41.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046907bc255195b0d-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.238 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:00:41.238 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:00:41.238 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.238 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.239 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.239 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.239 DEBUG:boto:Data: 2024-04-12T04:00:41.239 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.239 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.239 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.239 DEBUG:boto:Params: {} 2024-04-12T04:00:41.239 DEBUG:boto:Token: None 2024-04-12T04:00:41.239 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.239 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7rmfnvqGtd5yoLA1yFpv1BnJnqo= 2024-04-12T04:00:41.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7rmfnvqGtd5yoLA1yFpv1BnJnqo=', 'Content-Length': '0'} 2024-04-12T04:00:41.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad9aa0054e97d15a-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.241 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:00:41.241 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:00:41.242 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.242 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.242 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.242 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.242 DEBUG:boto:Data: 2024-04-12T04:00:41.242 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.242 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.242 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.242 DEBUG:boto:Params: {} 2024-04-12T04:00:41.242 DEBUG:boto:Token: None 2024-04-12T04:00:41.242 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.243 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7rmfnvqGtd5yoLA1yFpv1BnJnqo= 2024-04-12T04:00:41.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7rmfnvqGtd5yoLA1yFpv1BnJnqo=', 'Content-Length': '0'} 2024-04-12T04:00:41.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009778cc8f199ab53f-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:00:41.245 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:00:41.245 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:00:41.245 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.245 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.245 DEBUG:boto:Method: GET 2024-04-12T04:00:41.245 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.245 DEBUG:boto:Data: 2024-04-12T04:00:41.245 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.246 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.246 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.246 DEBUG:boto:Params: {} 2024-04-12T04:00:41.246 DEBUG:boto:Token: None 2024-04-12T04:00:41.246 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.246 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L2fuL9qsGEMLdQMe/jgjoI6emwc= 2024-04-12T04:00:41.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L2fuL9qsGEMLdQMe/jgjoI6emwc=', 'Content-Length': '0'} 2024-04-12T04:00:41.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ba50476cefd41f0-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.248 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:00:41.248 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:00:41.248 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.249 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.249 DEBUG:boto:Method: GET 2024-04-12T04:00:41.249 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.249 DEBUG:boto:Data: 2024-04-12T04:00:41.249 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.249 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.249 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.249 DEBUG:boto:Params: {} 2024-04-12T04:00:41.249 DEBUG:boto:Token: None 2024-04-12T04:00:41.249 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:00:41.249 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L2fuL9qsGEMLdQMe/jgjoI6emwc= 2024-04-12T04:00:41.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L2fuL9qsGEMLdQMe/jgjoI6emwc=', 'Content-Length': '0'} 2024-04-12T04:00:41.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd577f74a186d84a-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.252 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.252 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.252 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:00:41.252 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:00:41.252 DEBUG:boto:Method: GET 2024-04-12T04:00:41.253 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:00:41.253 DEBUG:boto:Data: 2024-04-12T04:00:41.253 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.253 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.253 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.253 DEBUG:boto:Params: {} 2024-04-12T04:00:41.253 DEBUG:boto:Token: None 2024-04-12T04:00:41.253 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.254 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc= 2024-04-12T04:00:41.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc=', 'Content-Length': '0'} 2024-04-12T04:00:41.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060134d962649da26-006618b1e9-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.257 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:41.257 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.257 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.258 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:00:41.258 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:00:41.258 DEBUG:boto:Method: GET 2024-04-12T04:00:41.258 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:00:41.258 DEBUG:boto:Data: 2024-04-12T04:00:41.259 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.259 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.259 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.259 DEBUG:boto:Params: {} 2024-04-12T04:00:41.259 DEBUG:boto:Token: None 2024-04-12T04:00:41.259 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.260 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc= 2024-04-12T04:00:41.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc=', 'Content-Length': '0'} 2024-04-12T04:00:41.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b4c2d003f0d55d4-006618b1e9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.310 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:41.311 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:00:41.311 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:00:41.312 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.312 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:00:41.312 DEBUG:boto:Data: 2024-04-12T04:00:41.312 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.312 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.312 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.313 DEBUG:boto:Params: {} 2024-04-12T04:00:41.313 DEBUG:boto:Token: None 2024-04-12T04:00:41.313 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj44 2024-04-12T04:00:41.313 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PddZA7FTLf7TapkwY9d5DNNI0cI= 2024-04-12T04:00:41.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PddZA7FTLf7TapkwY9d5DNNI0cI=', 'Content-Length': '0'} 2024-04-12T04:00:41.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c05c1f9ebed2951d-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.316 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:00:41.317 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:00:41.317 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.317 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:00:41.318 DEBUG:boto:Data: 2024-04-12T04:00:41.318 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.318 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.318 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.319 DEBUG:boto:Params: {} 2024-04-12T04:00:41.319 DEBUG:boto:Token: None 2024-04-12T04:00:41.319 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj44 2024-04-12T04:00:41.319 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PddZA7FTLf7TapkwY9d5DNNI0cI= 2024-04-12T04:00:41.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PddZA7FTLf7TapkwY9d5DNNI0cI=', 'Content-Length': '0'} 2024-04-12T04:00:41.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076e50d8f94281c2d-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:00:41.323 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:00:41.323 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:00:41.323 DEBUG:boto:Method: GET 2024-04-12T04:00:41.324 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:00:41.324 DEBUG:boto:Data: 2024-04-12T04:00:41.324 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.324 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.324 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.324 DEBUG:boto:Params: {} 2024-04-12T04:00:41.325 DEBUG:boto:Token: None 2024-04-12T04:00:41.325 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj44 2024-04-12T04:00:41.325 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A4k2LK9aIyxPjYqCH0mhCMpnGqc= 2024-04-12T04:00:41.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A4k2LK9aIyxPjYqCH0mhCMpnGqc=', 'Content-Length': '0'} 2024-04-12T04:00:41.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006a0e470bf67e277-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.328 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:00:41.328 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:00:41.328 DEBUG:boto:Method: GET 2024-04-12T04:00:41.329 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:00:41.329 DEBUG:boto:Data: 2024-04-12T04:00:41.329 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.330 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.330 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.330 DEBUG:boto:Params: {} 2024-04-12T04:00:41.330 DEBUG:boto:Token: None 2024-04-12T04:00:41.330 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj44 2024-04-12T04:00:41.331 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A4k2LK9aIyxPjYqCH0mhCMpnGqc= 2024-04-12T04:00:41.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A4k2LK9aIyxPjYqCH0mhCMpnGqc=', 'Content-Length': '0'} 2024-04-12T04:00:41.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000605062ad79bcb80a-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.333 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.333 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.334 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:00:41.334 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:00:41.334 DEBUG:boto:Method: GET 2024-04-12T04:00:41.334 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:00:41.334 DEBUG:boto:Data: 2024-04-12T04:00:41.334 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.335 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.335 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.335 DEBUG:boto:Params: {} 2024-04-12T04:00:41.335 DEBUG:boto:Token: None 2024-04-12T04:00:41.335 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.336 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc= 2024-04-12T04:00:41.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc=', 'Content-Length': '0'} 2024-04-12T04:00:41.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000165023d9758615b-006618b1e9-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.338 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:41.339 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.339 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.339 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:00:41.339 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:00:41.339 DEBUG:boto:Method: GET 2024-04-12T04:00:41.339 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:00:41.339 DEBUG:boto:Data: 2024-04-12T04:00:41.340 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.340 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.340 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.340 DEBUG:boto:Params: {} 2024-04-12T04:00:41.340 DEBUG:boto:Token: None 2024-04-12T04:00:41.340 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.340 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc= 2024-04-12T04:00:41.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc=', 'Content-Length': '0'} 2024-04-12T04:00:41.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000224e4770f9302026-006618b1e9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.343 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:41.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:00:41.343 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:00:41.343 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:00:41.343 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.343 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.344 DEBUG:boto:Method: GET 2024-04-12T04:00:41.344 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.344 DEBUG:boto:Data: 2024-04-12T04:00:41.344 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.344 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.344 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.344 DEBUG:boto:Params: {} 2024-04-12T04:00:41.344 DEBUG:boto:Token: None 2024-04-12T04:00:41.344 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.345 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MCDhpR/AOYjViUjhcFm7pZ/XxOs= 2024-04-12T04:00:41.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MCDhpR/AOYjViUjhcFm7pZ/XxOs=', 'Content-Length': '0'} 2024-04-12T04:00:41.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e1db82969d6e639-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.390 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:00:41.391 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:00:41.391 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.391 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.391 DEBUG:boto:Method: GET 2024-04-12T04:00:41.391 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.391 DEBUG:boto:Data: 2024-04-12T04:00:41.391 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.391 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.391 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.391 DEBUG:boto:Params: {} 2024-04-12T04:00:41.392 DEBUG:boto:Token: None 2024-04-12T04:00:41.392 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.392 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MCDhpR/AOYjViUjhcFm7pZ/XxOs= 2024-04-12T04:00:41.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MCDhpR/AOYjViUjhcFm7pZ/XxOs=', 'Content-Length': '0'} 2024-04-12T04:00:41.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000567adf21dfb6525-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.394 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:00:41.394 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:00:41.394 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.395 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.395 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.395 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.395 DEBUG:boto:Data: 2024-04-12T04:00:41.395 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.395 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.395 DEBUG:boto:Params: {} 2024-04-12T04:00:41.395 DEBUG:boto:Token: None 2024-04-12T04:00:41.396 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.396 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A3HeY8jUEsvkjyYRsJXoCo76ND0= 2024-04-12T04:00:41.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A3HeY8jUEsvkjyYRsJXoCo76ND0=', 'Content-Length': '0'} 2024-04-12T04:00:41.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdcff0f3466b10c6-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.398 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:00:41.398 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:00:41.398 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.398 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.398 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.398 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.399 DEBUG:boto:Data: 2024-04-12T04:00:41.399 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.399 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.399 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.399 DEBUG:boto:Params: {} 2024-04-12T04:00:41.399 DEBUG:boto:Token: None 2024-04-12T04:00:41.399 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.399 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A3HeY8jUEsvkjyYRsJXoCo76ND0= 2024-04-12T04:00:41.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A3HeY8jUEsvkjyYRsJXoCo76ND0=', 'Content-Length': '0'} 2024-04-12T04:00:41.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069533b14094c040f-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:00:41.402 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:00:41.402 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:00:41.402 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.402 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.402 DEBUG:boto:Method: GET 2024-04-12T04:00:41.402 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.402 DEBUG:boto:Data: 2024-04-12T04:00:41.402 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.402 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.403 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.403 DEBUG:boto:Params: {} 2024-04-12T04:00:41.403 DEBUG:boto:Token: None 2024-04-12T04:00:41.403 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.403 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MCDhpR/AOYjViUjhcFm7pZ/XxOs= 2024-04-12T04:00:41.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MCDhpR/AOYjViUjhcFm7pZ/XxOs=', 'Content-Length': '0'} 2024-04-12T04:00:41.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060dcdf70063091f9-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.405 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:00:41.406 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:00:41.406 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.406 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.406 DEBUG:boto:Method: GET 2024-04-12T04:00:41.406 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.406 DEBUG:boto:Data: 2024-04-12T04:00:41.406 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.406 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.406 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.406 DEBUG:boto:Params: {} 2024-04-12T04:00:41.407 DEBUG:boto:Token: None 2024-04-12T04:00:41.407 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:00:41.407 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MCDhpR/AOYjViUjhcFm7pZ/XxOs= 2024-04-12T04:00:41.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MCDhpR/AOYjViUjhcFm7pZ/XxOs=', 'Content-Length': '0'} 2024-04-12T04:00:41.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f28c7c880059623d-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.409 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.409 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.409 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:00:41.409 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:00:41.410 DEBUG:boto:Method: GET 2024-04-12T04:00:41.410 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:00:41.410 DEBUG:boto:Data: 2024-04-12T04:00:41.410 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.410 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.410 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.410 DEBUG:boto:Params: {} 2024-04-12T04:00:41.410 DEBUG:boto:Token: None 2024-04-12T04:00:41.410 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.410 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc= 2024-04-12T04:00:41.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc=', 'Content-Length': '0'} 2024-04-12T04:00:41.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7e34d8c3866be0c-006618b1e9-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.413 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:41.413 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.413 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.414 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:00:41.414 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:00:41.414 DEBUG:boto:Method: GET 2024-04-12T04:00:41.414 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:00:41.414 DEBUG:boto:Data: 2024-04-12T04:00:41.414 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.414 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.414 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.414 DEBUG:boto:Params: {} 2024-04-12T04:00:41.414 DEBUG:boto:Token: None 2024-04-12T04:00:41.415 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.415 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc= 2024-04-12T04:00:41.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc=', 'Content-Length': '0'} 2024-04-12T04:00:41.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcf5363d59a69969-006618b1e9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.417 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:41.417 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:00:41.418 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:00:41.418 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.418 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:00:41.418 DEBUG:boto:Data: 2024-04-12T04:00:41.418 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.418 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.419 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.419 DEBUG:boto:Params: {} 2024-04-12T04:00:41.419 DEBUG:boto:Token: None 2024-04-12T04:00:41.419 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj55 2024-04-12T04:00:41.419 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wSIQCvJ3JYfnHBEGaQpTzF8vZ4k= 2024-04-12T04:00:41.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wSIQCvJ3JYfnHBEGaQpTzF8vZ4k=', 'Content-Length': '0'} 2024-04-12T04:00:41.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc1e6ab271734c5-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.422 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:00:41.422 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:00:41.423 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.423 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:00:41.423 DEBUG:boto:Data: 2024-04-12T04:00:41.423 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.423 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.423 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.424 DEBUG:boto:Params: {} 2024-04-12T04:00:41.424 DEBUG:boto:Token: None 2024-04-12T04:00:41.424 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj55 2024-04-12T04:00:41.424 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wSIQCvJ3JYfnHBEGaQpTzF8vZ4k= 2024-04-12T04:00:41.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wSIQCvJ3JYfnHBEGaQpTzF8vZ4k=', 'Content-Length': '0'} 2024-04-12T04:00:41.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012ac9072b3ccad66-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:00:41.427 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:00:41.427 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:00:41.427 DEBUG:boto:Method: GET 2024-04-12T04:00:41.428 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:00:41.428 DEBUG:boto:Data: 2024-04-12T04:00:41.428 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.428 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.428 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.429 DEBUG:boto:Params: {} 2024-04-12T04:00:41.429 DEBUG:boto:Token: None 2024-04-12T04:00:41.429 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj55 2024-04-12T04:00:41.429 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QENNB/Y9V5+vsus7GdovdbVWP6M= 2024-04-12T04:00:41.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QENNB/Y9V5+vsus7GdovdbVWP6M=', 'Content-Length': '0'} 2024-04-12T04:00:41.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b0119425e3fb8aa-006618b1e9-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.432 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:00:41.432 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:00:41.432 DEBUG:boto:Method: GET 2024-04-12T04:00:41.432 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:00:41.432 DEBUG:boto:Data: 2024-04-12T04:00:41.432 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.432 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.432 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.432 DEBUG:boto:Params: {} 2024-04-12T04:00:41.433 DEBUG:boto:Token: None 2024-04-12T04:00:41.433 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj55 2024-04-12T04:00:41.433 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QENNB/Y9V5+vsus7GdovdbVWP6M= 2024-04-12T04:00:41.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QENNB/Y9V5+vsus7GdovdbVWP6M=', 'Content-Length': '0'} 2024-04-12T04:00:41.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e20fa6ea5af3b7a7-006618b1e9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.435 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.435 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.435 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:00:41.435 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:00:41.435 DEBUG:boto:Method: GET 2024-04-12T04:00:41.435 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:00:41.436 DEBUG:boto:Data: 2024-04-12T04:00:41.436 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.436 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:00:41.436 DEBUG:boto:Port: 8001 2024-04-12T04:00:41.436 DEBUG:boto:Params: {} 2024-04-12T04:00:41.436 DEBUG:boto:Token: None 2024-04-12T04:00:41.436 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.436 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc= 2024-04-12T04:00:41.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc=', 'Content-Length': '0'} 2024-04-12T04:00:41.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000647f3f822a4b5c7-006618b1e9-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.439 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:41.439 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.439 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.440 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:00:41.440 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:00:41.440 DEBUG:boto:Method: GET 2024-04-12T04:00:41.440 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:00:41.440 DEBUG:boto:Data: 2024-04-12T04:00:41.440 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.441 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.441 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.441 DEBUG:boto:Params: {} 2024-04-12T04:00:41.441 DEBUG:boto:Token: None 2024-04-12T04:00:41.441 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.441 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc= 2024-04-12T04:00:41.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNYkC0XlJNS2mxjL3RyAUWFxMXc=', 'Content-Length': '0'} 2024-04-12T04:00:41.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053f4ecc099f44ddb-006618b1e9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.490 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:00:41.491 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-61 zones={a2, a1} 2024-04-12T04:00:41.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:00:41.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:00:41.492 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:00:41.555 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:00:41.556 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:00:41.605 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:00:41.605 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-61 2024-04-12T04:00:41.610 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 5 2024-04-12T04:00:41.654 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.655 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.655 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.655 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:00:41.655 DEBUG:boto:Data: 2024-04-12T04:00:41.655 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.655 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.655 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.655 DEBUG:boto:Params: {} 2024-04-12T04:00:41.655 DEBUG:boto:Token: None 2024-04-12T04:00:41.655 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:usAhSE+RG8UZK2lCmZRX5K7Zcok= 2024-04-12T04:00:41.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:usAhSE+RG8UZK2lCmZRX5K7Zcok=', 'Content-Length': '0'} 2024-04-12T04:00:41.658 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e9ee861aadbc093-006618b1e9-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.659 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:00:41.659 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:00:41.659 DEBUG:boto:Method: PUT 2024-04-12T04:00:41.659 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:00:41.659 DEBUG:boto:Data: 2024-04-12T04:00:41.659 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:00:41.659 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.659 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.660 DEBUG:boto:Params: {} 2024-04-12T04:00:41.660 DEBUG:boto:Token: None 2024-04-12T04:00:41.660 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj17 2024-04-12T04:00:41.660 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WPgMi5U1SA0cL5e1XGqN42WjqmU= 2024-04-12T04:00:41.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WPgMi5U1SA0cL5e1XGqN42WjqmU='} 2024-04-12T04:00:41.665 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0be2bbfe75ea7fb-006618b1e9-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.666 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.666 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.666 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.666 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:00:41.666 DEBUG:boto:Data: 2024-04-12T04:00:41.666 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.666 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.666 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.666 DEBUG:boto:Params: {} 2024-04-12T04:00:41.667 DEBUG:boto:Token: None 2024-04-12T04:00:41.667 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.667 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:usAhSE+RG8UZK2lCmZRX5K7Zcok= 2024-04-12T04:00:41.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:usAhSE+RG8UZK2lCmZRX5K7Zcok=', 'Content-Length': '0'} 2024-04-12T04:00:41.670 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e959f3a4fbb19e30-006618b1e9-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.670 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:00:41.670 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:00:41.670 DEBUG:boto:Method: PUT 2024-04-12T04:00:41.670 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:00:41.670 DEBUG:boto:Data: 2024-04-12T04:00:41.670 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:00:41.670 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.671 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.671 DEBUG:boto:Params: {} 2024-04-12T04:00:41.671 DEBUG:boto:Token: None 2024-04-12T04:00:41.671 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj34 2024-04-12T04:00:41.671 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:k0oywwmc/jMn+QaGyIfFfJwhDpk= 2024-04-12T04:00:41.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:k0oywwmc/jMn+QaGyIfFfJwhDpk='} 2024-04-12T04:00:41.677 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000809bd9176d25d51b-006618b1e9-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.677 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.677 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.677 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.677 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:00:41.678 DEBUG:boto:Data: 2024-04-12T04:00:41.678 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.678 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.678 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.678 DEBUG:boto:Params: {} 2024-04-12T04:00:41.678 DEBUG:boto:Token: None 2024-04-12T04:00:41.678 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.678 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:usAhSE+RG8UZK2lCmZRX5K7Zcok= 2024-04-12T04:00:41.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:usAhSE+RG8UZK2lCmZRX5K7Zcok=', 'Content-Length': '0'} 2024-04-12T04:00:41.681 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b959a7fa74e28938-006618b1e9-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.682 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:00:41.682 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:00:41.682 DEBUG:boto:Method: PUT 2024-04-12T04:00:41.682 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:00:41.682 DEBUG:boto:Data: 2024-04-12T04:00:41.682 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:00:41.682 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.682 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.682 DEBUG:boto:Params: {} 2024-04-12T04:00:41.683 DEBUG:boto:Token: None 2024-04-12T04:00:41.683 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj51 2024-04-12T04:00:41.683 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cH+CtTCYOaL/95XuepH67mrck/s= 2024-04-12T04:00:41.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cH+CtTCYOaL/95XuepH67mrck/s='} 2024-04-12T04:00:41.726 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bdbb67e03e56b833-006618b1e9-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.727 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.727 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.727 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.727 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:00:41.727 DEBUG:boto:Data: 2024-04-12T04:00:41.727 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.727 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.728 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.728 DEBUG:boto:Params: {} 2024-04-12T04:00:41.728 DEBUG:boto:Token: None 2024-04-12T04:00:41.728 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.728 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:usAhSE+RG8UZK2lCmZRX5K7Zcok= 2024-04-12T04:00:41.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:usAhSE+RG8UZK2lCmZRX5K7Zcok=', 'Content-Length': '0'} 2024-04-12T04:00:41.731 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000102aad7c031cec13-006618b1e9-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.731 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:00:41.731 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:00:41.732 DEBUG:boto:Method: PUT 2024-04-12T04:00:41.732 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:00:41.732 DEBUG:boto:Data: 2024-04-12T04:00:41.732 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:00:41.732 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.732 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.732 DEBUG:boto:Params: {} 2024-04-12T04:00:41.732 DEBUG:boto:Token: None 2024-04-12T04:00:41.732 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj68 2024-04-12T04:00:41.732 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TGnowzXNDLQHqaMxQB8zrDj3D6c= 2024-04-12T04:00:41.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TGnowzXNDLQHqaMxQB8zrDj3D6c='} 2024-04-12T04:00:41.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000083c7eb28e52205eb-006618b1e9-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.739 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:00:41.739 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:00:41.739 DEBUG:boto:Method: HEAD 2024-04-12T04:00:41.739 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:00:41.739 DEBUG:boto:Data: 2024-04-12T04:00:41.739 DEBUG:boto:Headers: {} 2024-04-12T04:00:41.739 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.739 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.739 DEBUG:boto:Params: {} 2024-04-12T04:00:41.740 DEBUG:boto:Token: None 2024-04-12T04:00:41.740 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/ 2024-04-12T04:00:41.740 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:usAhSE+RG8UZK2lCmZRX5K7Zcok= 2024-04-12T04:00:41.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:usAhSE+RG8UZK2lCmZRX5K7Zcok=', 'Content-Length': '0'} 2024-04-12T04:00:41.742 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081a3bbb1426224ae-006618b1e9-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.743 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:00:41.743 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:00:41.743 DEBUG:boto:Method: PUT 2024-04-12T04:00:41.743 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:00:41.743 DEBUG:boto:Data: 2024-04-12T04:00:41.743 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:00:41.743 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:00:41.743 DEBUG:boto:Port: 8000 2024-04-12T04:00:41.743 DEBUG:boto:Params: {} 2024-04-12T04:00:41.743 DEBUG:boto:Token: None 2024-04-12T04:00:41.744 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:00:41 GMT /dkyges-61/obj85 2024-04-12T04:00:41.744 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oq63Wq9sUx1MP2Tdq28DwkmmOzc= 2024-04-12T04:00:41.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:00:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oq63Wq9sUx1MP2Tdq28DwkmmOzc='} 2024-04-12T04:00:41.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000902349a738e61ea9-006618b1e9-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:00:41.787 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-61 2024-04-12T04:00:41.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:00:41.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:00:41.787 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:00:41.854 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:00:41.854 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:00:41.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:41.913+0000 7f6af4791780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T04:00:46.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:46.913+0000 7f6af4791780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T04:00:51.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:51.914+0000 7f6af4791780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T04:00:56.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:00:56.914+0000 7f6af4791780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T04:01:01.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:01:01.917+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:01:01.904 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-04-12T04:01:01.904 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:01:06.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:01:06.918+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:01:06.991 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , , , , , ] 2024-04-12T04:01:06.991 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:01:11.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:01:11.920+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:01:11.907 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , , 00000000001.84.6, , , ] 2024-04-12T04:01:11.907 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:01:16.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:01:16.921+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:01:16.908 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , , 00000000001.84.6, , , ] 2024-04-12T04:01:16.908 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:01:21.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:01:21.923+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:01:21.910 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , , 00000000001.84.6, , , ] 2024-04-12T04:01:21.910 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:01:26.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:01:26.924+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:01:26.912 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , , 00000000001.84.6, , , ] 2024-04-12T04:01:26.912 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:01:31.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:01:31.926+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:01:31.913 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , , 00000000001.84.6, , , ] 2024-04-12T04:01:31.913 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:01:36.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:01:36.929+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:01:36.915 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , , 00000000001.84.6, , , ] 2024-04-12T04:01:36.915 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:01:41.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:01:41.929+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:01:41.916 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.97.6, , , , , , , , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:01:41.916 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:01:46.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:01:46.931+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:01:46.918 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.97.6, , , , , , , , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:01:46.918 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:01:51.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:01:51.932+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:01:51.919 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.97.6, , , , , , , , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:01:51.919 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:01:56.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:01:56.934+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:01:56.921 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.97.6, , , , , , , , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:01:56.921 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:01.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:02:01.936+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:01.923 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.97.6, , , , , , , , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:01.923 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:06.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:02:06.937+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:06.924 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.97.6, , , , , , , , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:06.924 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:11.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:02:11.939+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:11.926 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.97.6, , , , , , , , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:11.926 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:16.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:02:16.941+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:16.928 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.97.6, , , , , , , , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:16.928 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:21.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:02:21.943+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:21.930 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.97.6, , , , , , , , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:21.930 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:26.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:02:26.945+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:26.932 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.97.6, , , , , , , , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:26.932 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:31.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:02:31.946+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:31.934 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.97.6, , , , , , , , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:31.934 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:36.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:02:36.948+0000 7f6af4791780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:36.935 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.97.6, , , , , , , , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:36.935 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:41.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:02:41.950+0000 7f6af4791780 1 bucket sync caught up with source: 2024-04-12T04:02:41.936 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:41.936 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000001.97.6, , , , , , 00000000001.47.6, , 00000000001.62.6, , , , 00000000001.72.6, 00000000001.84.6, , , ] 2024-04-12T04:02:41.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:02:41.950+0000 7f6af4791780 0 bucket checkpoint complete 2024-04-12T04:02:41.955 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-61 2024-04-12T04:02:41.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:02:41.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:02:41.956 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:02:42.025 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:02:42.026 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:02:42.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:02:42.081+0000 7fa6ff30e780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:42.068 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.2360.6, 00000000001.1403.6, , , 00000000001.2593.6, , , 00000000001.2553.6, 00000000003.2278.6, 00000000002.2604.6] 2024-04-12T04:02:42.068 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.2360.6, 00000000001.1403.6, , , 00000000001.2593.6, , , 00000000001.2553.6, 00000000003.2278.6, 00000000003.2610.6] 2024-04-12T04:02:47.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:02:47.083+0000 7fa6ff30e780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:47.069 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.2360.6, 00000000001.1403.6, , , 00000000001.2593.6, , , 00000000001.2553.6, 00000000003.2278.6, 00000000002.2604.6] 2024-04-12T04:02:47.069 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.2360.6, 00000000001.1403.6, , , 00000000001.2593.6, , , 00000000001.2553.6, 00000000003.2278.6, 00000000003.2610.6] 2024-04-12T04:02:52.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:02:52.084+0000 7fa6ff30e780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:52.071 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.2360.6, 00000000001.1403.6, , , 00000000001.2593.6, , , 00000000001.2553.6, 00000000003.2278.6, 00000000002.2604.6] 2024-04-12T04:02:52.071 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.2360.6, 00000000001.1403.6, , , 00000000001.2593.6, , , 00000000001.2553.6, 00000000003.2278.6, 00000000003.2610.6] 2024-04-12T04:02:57.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:02:57.086+0000 7fa6ff30e780 1 waiting for incremental sync to catch up: 2024-04-12T04:02:57.072 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.2360.6, 00000000001.1403.6, , , 00000000001.2593.6, , , 00000000001.2553.6, 00000000003.2278.6, 00000000002.2604.6] 2024-04-12T04:02:57.072 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.2360.6, 00000000001.1403.6, , , 00000000001.2593.6, , , 00000000001.2553.6, 00000000003.2278.6, 00000000003.2610.6] 2024-04-12T04:03:00.354 INFO:tasks.rgw.c1.client.0.smithi043.stdout:2024-04-12T04:03:00.368+0000 7f1e272bd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:03:02.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:03:02.087+0000 7fa6ff30e780 1 bucket sync caught up with source: 2024-04-12T04:03:02.073 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000001.2360.6, 00000000001.1403.6, , , 00000000001.2593.6, , , 00000000001.2553.6, 00000000003.2278.6, 00000000003.2610.6] 2024-04-12T04:03:02.073 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000001.2360.6, 00000000001.1403.6, , , 00000000001.2593.6, , , 00000000001.2553.6, 00000000003.2278.6, 00000000003.2610.6] 2024-04-12T04:03:02.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:03:02.087+0000 7fa6ff30e780 0 bucket checkpoint complete 2024-04-12T04:03:02.092 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-61 zones={a2, a1} 2024-04-12T04:03:02.092 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.092 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.092 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.092 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:03:02.092 DEBUG:boto:Data: 2024-04-12T04:03:02.093 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.093 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.093 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.093 DEBUG:boto:Params: {} 2024-04-12T04:03:02.093 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:03:02.094 DEBUG:boto:Token: None 2024-04-12T04:03:02.094 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.094 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H5A9SZCIf0y1TPqvsc/cqviAAnU= 2024-04-12T04:03:02.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H5A9SZCIf0y1TPqvsc/cqviAAnU=', 'Content-Length': '0'} 2024-04-12T04:03:02.109 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007cafbfbc79ecaa20-006618b276-1cfd-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.109 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.109 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.109 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.109 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:03:02.109 DEBUG:boto:Data: 2024-04-12T04:03:02.109 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.110 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.110 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.110 DEBUG:boto:Params: {} 2024-04-12T04:03:02.110 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:03:02.110 DEBUG:boto:Token: None 2024-04-12T04:03:02.110 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.110 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H5A9SZCIf0y1TPqvsc/cqviAAnU= 2024-04-12T04:03:02.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H5A9SZCIf0y1TPqvsc/cqviAAnU=', 'Content-Length': '0'} 2024-04-12T04:03:02.125 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000926ab0936b34e6dd-006618b276-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.125 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:03:02.125 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.125 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.125 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:03:02.125 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:03:02.125 DEBUG:boto:Method: GET 2024-04-12T04:03:02.125 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:03:02.126 DEBUG:boto:Data: 2024-04-12T04:03:02.126 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.126 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.126 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.126 DEBUG:boto:Params: {} 2024-04-12T04:03:02.126 DEBUG:boto:Token: None 2024-04-12T04:03:02.126 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/?versions 2024-04-12T04:03:02.126 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:w0izfev98uRjHNvEAcv96dsJD+4= 2024-04-12T04:03:02.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:w0izfev98uRjHNvEAcv96dsJD+4=', 'Content-Length': '0'} 2024-04-12T04:03:02.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006246aad591da4f43-006618b276-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.129 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:03:02.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:03:02.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:03:02.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:03:02.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:03:02.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:03:02.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:03:02.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:03:02.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:03:02.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:03:02.131 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:03:02.131 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.131 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.131 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:03:02.132 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:03:02.132 DEBUG:boto:Method: GET 2024-04-12T04:03:02.132 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:03:02.132 DEBUG:boto:Data: 2024-04-12T04:03:02.132 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.132 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.132 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.132 DEBUG:boto:Params: {} 2024-04-12T04:03:02.132 DEBUG:boto:Token: None 2024-04-12T04:03:02.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/?versions 2024-04-12T04:03:02.133 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:w0izfev98uRjHNvEAcv96dsJD+4= 2024-04-12T04:03:02.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:w0izfev98uRjHNvEAcv96dsJD+4=', 'Content-Length': '0'} 2024-04-12T04:03:02.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000111b062f45fa6b6d-006618b276-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.136 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:03:02.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:03:02.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:03:02.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:03:02.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:03:02.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:03:02.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:03:02.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:03:02.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:03:02.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:03:02.137 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.137 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.138 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:03:02.138 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:03:02.138 DEBUG:boto:Method: GET 2024-04-12T04:03:02.138 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:03:02.138 DEBUG:boto:Data: 2024-04-12T04:03:02.138 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.138 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.138 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.138 DEBUG:boto:Params: {} 2024-04-12T04:03:02.138 DEBUG:boto:Token: None 2024-04-12T04:03:02.139 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/?versions 2024-04-12T04:03:02.139 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:w0izfev98uRjHNvEAcv96dsJD+4= 2024-04-12T04:03:02.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:w0izfev98uRjHNvEAcv96dsJD+4=', 'Content-Length': '0'} 2024-04-12T04:03:02.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065255793589d397b-006618b276-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.141 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.142 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.142 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.142 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:03:02.142 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:03:02.142 DEBUG:boto:Method: GET 2024-04-12T04:03:02.142 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:03:02.143 DEBUG:boto:Data: 2024-04-12T04:03:02.143 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.143 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.143 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.143 DEBUG:boto:Params: {} 2024-04-12T04:03:02.143 DEBUG:boto:Token: None 2024-04-12T04:03:02.143 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/?versions 2024-04-12T04:03:02.143 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:w0izfev98uRjHNvEAcv96dsJD+4= 2024-04-12T04:03:02.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:w0izfev98uRjHNvEAcv96dsJD+4=', 'Content-Length': '0'} 2024-04-12T04:03:02.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f7dccb91a208c7c-006618b276-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.146 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:03:02.147 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:03:02.147 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:03:02.147 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.147 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.148 DEBUG:boto:Method: GET 2024-04-12T04:03:02.148 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.148 DEBUG:boto:Data: 2024-04-12T04:03:02.148 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.148 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.148 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.148 DEBUG:boto:Params: {} 2024-04-12T04:03:02.148 DEBUG:boto:Token: None 2024-04-12T04:03:02.148 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.148 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lRMtnRZBP178c5/bnyWl53L5GBE= 2024-04-12T04:03:02.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lRMtnRZBP178c5/bnyWl53L5GBE=', 'Content-Length': '0'} 2024-04-12T04:03:02.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000253771562106f1a-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.150 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:03:02.151 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:03:02.151 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.151 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.151 DEBUG:boto:Method: GET 2024-04-12T04:03:02.151 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.151 DEBUG:boto:Data: 2024-04-12T04:03:02.151 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.151 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.151 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.151 DEBUG:boto:Params: {} 2024-04-12T04:03:02.152 DEBUG:boto:Token: None 2024-04-12T04:03:02.152 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.152 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lRMtnRZBP178c5/bnyWl53L5GBE= 2024-04-12T04:03:02.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lRMtnRZBP178c5/bnyWl53L5GBE=', 'Content-Length': '0'} 2024-04-12T04:03:02.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea9e7b22ead643e9-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.154 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:03:02.154 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:03:02.154 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.155 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.155 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.155 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.155 DEBUG:boto:Data: 2024-04-12T04:03:02.155 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.155 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.155 DEBUG:boto:Params: {} 2024-04-12T04:03:02.155 DEBUG:boto:Token: None 2024-04-12T04:03:02.155 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V3M1tFTSUyK4l4AkxZMs6hT6tLU= 2024-04-12T04:03:02.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V3M1tFTSUyK4l4AkxZMs6hT6tLU=', 'Content-Length': '0'} 2024-04-12T04:03:02.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e1952ea88c25495-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.158 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:03:02.158 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:03:02.158 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.158 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.158 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.158 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.159 DEBUG:boto:Data: 2024-04-12T04:03:02.159 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.159 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.159 DEBUG:boto:Params: {} 2024-04-12T04:03:02.159 DEBUG:boto:Token: None 2024-04-12T04:03:02.159 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.159 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V3M1tFTSUyK4l4AkxZMs6hT6tLU= 2024-04-12T04:03:02.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V3M1tFTSUyK4l4AkxZMs6hT6tLU=', 'Content-Length': '0'} 2024-04-12T04:03:02.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000647ed39bf2f3ae6b-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:03:02.161 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:03:02.162 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:03:02.162 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.162 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.162 DEBUG:boto:Method: GET 2024-04-12T04:03:02.162 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.162 DEBUG:boto:Data: 2024-04-12T04:03:02.162 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.162 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.162 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.162 DEBUG:boto:Params: {} 2024-04-12T04:03:02.162 DEBUG:boto:Token: None 2024-04-12T04:03:02.163 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.163 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lRMtnRZBP178c5/bnyWl53L5GBE= 2024-04-12T04:03:02.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lRMtnRZBP178c5/bnyWl53L5GBE=', 'Content-Length': '0'} 2024-04-12T04:03:02.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0d28340d607fcf2-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.165 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:03:02.165 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:03:02.165 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.165 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.165 DEBUG:boto:Method: GET 2024-04-12T04:03:02.165 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.165 DEBUG:boto:Data: 2024-04-12T04:03:02.166 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.166 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.166 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.166 DEBUG:boto:Params: {} 2024-04-12T04:03:02.166 DEBUG:boto:Token: None 2024-04-12T04:03:02.166 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:03:02.166 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lRMtnRZBP178c5/bnyWl53L5GBE= 2024-04-12T04:03:02.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lRMtnRZBP178c5/bnyWl53L5GBE=', 'Content-Length': '0'} 2024-04-12T04:03:02.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc62fa741ac8d19a-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.169 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.169 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.169 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:03:02.169 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:03:02.169 DEBUG:boto:Method: GET 2024-04-12T04:03:02.169 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:03:02.169 DEBUG:boto:Data: 2024-04-12T04:03:02.169 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.169 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.169 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.169 DEBUG:boto:Params: {} 2024-04-12T04:03:02.170 DEBUG:boto:Token: None 2024-04-12T04:03:02.170 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.170 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009784f9bed83f552c-006618b276-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.172 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.173 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.173 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.173 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:03:02.173 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:03:02.173 DEBUG:boto:Method: GET 2024-04-12T04:03:02.173 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:03:02.173 DEBUG:boto:Data: 2024-04-12T04:03:02.173 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.174 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.174 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.174 DEBUG:boto:Params: {} 2024-04-12T04:03:02.174 DEBUG:boto:Token: None 2024-04-12T04:03:02.174 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.174 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006474112e6e597239-006618b276-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.177 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.178 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:03:02.178 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:03:02.178 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.178 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:03:02.178 DEBUG:boto:Data: 2024-04-12T04:03:02.178 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.178 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.178 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.178 DEBUG:boto:Params: {} 2024-04-12T04:03:02.178 DEBUG:boto:Token: None 2024-04-12T04:03:02.179 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj11 2024-04-12T04:03:02.179 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uw44qeLHGLa9Iu+a3tQbf3We8PQ= 2024-04-12T04:03:02.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uw44qeLHGLa9Iu+a3tQbf3We8PQ=', 'Content-Length': '0'} 2024-04-12T04:03:02.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000563fdaab1b5c157e-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.181 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:03:02.181 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:03:02.181 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.181 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:03:02.181 DEBUG:boto:Data: 2024-04-12T04:03:02.181 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.182 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.182 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.182 DEBUG:boto:Params: {} 2024-04-12T04:03:02.182 DEBUG:boto:Token: None 2024-04-12T04:03:02.182 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj11 2024-04-12T04:03:02.182 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uw44qeLHGLa9Iu+a3tQbf3We8PQ= 2024-04-12T04:03:02.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uw44qeLHGLa9Iu+a3tQbf3We8PQ=', 'Content-Length': '0'} 2024-04-12T04:03:02.184 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a697d8a2dbc27e8a-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:03:02.184 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:03:02.185 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:03:02.185 DEBUG:boto:Method: GET 2024-04-12T04:03:02.185 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:03:02.185 DEBUG:boto:Data: 2024-04-12T04:03:02.185 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.185 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.185 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.185 DEBUG:boto:Params: {} 2024-04-12T04:03:02.185 DEBUG:boto:Token: None 2024-04-12T04:03:02.185 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj11 2024-04-12T04:03:02.185 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R4KEgrJmcG3UHe65Kw9XsBG8dk0= 2024-04-12T04:03:02.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R4KEgrJmcG3UHe65Kw9XsBG8dk0=', 'Content-Length': '0'} 2024-04-12T04:03:02.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000099094d43ff3ccbd-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.188 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:03:02.188 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:03:02.189 DEBUG:boto:Method: GET 2024-04-12T04:03:02.189 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:03:02.189 DEBUG:boto:Data: 2024-04-12T04:03:02.189 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.189 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.189 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.189 DEBUG:boto:Params: {} 2024-04-12T04:03:02.190 DEBUG:boto:Token: None 2024-04-12T04:03:02.190 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj11 2024-04-12T04:03:02.190 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R4KEgrJmcG3UHe65Kw9XsBG8dk0= 2024-04-12T04:03:02.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R4KEgrJmcG3UHe65Kw9XsBG8dk0=', 'Content-Length': '0'} 2024-04-12T04:03:02.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000555003621add7e6e-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.192 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.192 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.192 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:03:02.192 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:03:02.193 DEBUG:boto:Method: GET 2024-04-12T04:03:02.193 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:03:02.193 DEBUG:boto:Data: 2024-04-12T04:03:02.193 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.193 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.193 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.193 DEBUG:boto:Params: {} 2024-04-12T04:03:02.193 DEBUG:boto:Token: None 2024-04-12T04:03:02.193 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.193 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000546ad49691abd430-006618b276-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.196 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.196 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.196 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.196 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:03:02.196 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:03:02.197 DEBUG:boto:Method: GET 2024-04-12T04:03:02.197 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:03:02.197 DEBUG:boto:Data: 2024-04-12T04:03:02.197 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.197 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.197 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.197 DEBUG:boto:Params: {} 2024-04-12T04:03:02.197 DEBUG:boto:Token: None 2024-04-12T04:03:02.197 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.197 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d99b78af75b620ed-006618b276-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.201 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:03:02.201 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:03:02.202 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:03:02.202 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.202 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.202 DEBUG:boto:Method: GET 2024-04-12T04:03:02.202 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.203 DEBUG:boto:Data: 2024-04-12T04:03:02.203 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.203 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.203 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.203 DEBUG:boto:Params: {} 2024-04-12T04:03:02.204 DEBUG:boto:Token: None 2024-04-12T04:03:02.204 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.204 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z4b0xLjcUU8F2KOL/hN/j952uMY= 2024-04-12T04:03:02.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z4b0xLjcUU8F2KOL/hN/j952uMY=', 'Content-Length': '0'} 2024-04-12T04:03:02.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061df0babed73ceb4-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.207 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:03:02.207 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:03:02.208 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.208 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.208 DEBUG:boto:Method: GET 2024-04-12T04:03:02.208 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.208 DEBUG:boto:Data: 2024-04-12T04:03:02.208 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.209 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.209 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.209 DEBUG:boto:Params: {} 2024-04-12T04:03:02.209 DEBUG:boto:Token: None 2024-04-12T04:03:02.209 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.209 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z4b0xLjcUU8F2KOL/hN/j952uMY= 2024-04-12T04:03:02.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z4b0xLjcUU8F2KOL/hN/j952uMY=', 'Content-Length': '0'} 2024-04-12T04:03:02.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050944ba0ad925724-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.258 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:03:02.259 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:03:02.259 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.259 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.259 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.259 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.259 DEBUG:boto:Data: 2024-04-12T04:03:02.259 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.259 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.260 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.260 DEBUG:boto:Params: {} 2024-04-12T04:03:02.260 DEBUG:boto:Token: None 2024-04-12T04:03:02.260 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.261 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z+Y9lVHpXNB63ciK80f4x9TIB/I= 2024-04-12T04:03:02.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z+Y9lVHpXNB63ciK80f4x9TIB/I=', 'Content-Length': '0'} 2024-04-12T04:03:02.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000910ae951828e8660-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.263 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:03:02.263 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:03:02.263 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.263 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.263 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.264 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.264 DEBUG:boto:Data: 2024-04-12T04:03:02.264 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.264 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.264 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.264 DEBUG:boto:Params: {} 2024-04-12T04:03:02.264 DEBUG:boto:Token: None 2024-04-12T04:03:02.264 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.264 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z+Y9lVHpXNB63ciK80f4x9TIB/I= 2024-04-12T04:03:02.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z+Y9lVHpXNB63ciK80f4x9TIB/I=', 'Content-Length': '0'} 2024-04-12T04:03:02.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e58a26440b9d655-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:03:02.267 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:03:02.267 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:03:02.267 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.267 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.267 DEBUG:boto:Method: GET 2024-04-12T04:03:02.267 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.267 DEBUG:boto:Data: 2024-04-12T04:03:02.267 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.267 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.268 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.268 DEBUG:boto:Params: {} 2024-04-12T04:03:02.268 DEBUG:boto:Token: None 2024-04-12T04:03:02.268 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.268 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z4b0xLjcUU8F2KOL/hN/j952uMY= 2024-04-12T04:03:02.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z4b0xLjcUU8F2KOL/hN/j952uMY=', 'Content-Length': '0'} 2024-04-12T04:03:02.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad952a7a01dd7481-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.310 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:03:02.311 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:03:02.311 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.311 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.311 DEBUG:boto:Method: GET 2024-04-12T04:03:02.312 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.312 DEBUG:boto:Data: 2024-04-12T04:03:02.312 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.312 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.312 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.313 DEBUG:boto:Params: {} 2024-04-12T04:03:02.313 DEBUG:boto:Token: None 2024-04-12T04:03:02.313 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:03:02.314 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z4b0xLjcUU8F2KOL/hN/j952uMY= 2024-04-12T04:03:02.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z4b0xLjcUU8F2KOL/hN/j952uMY=', 'Content-Length': '0'} 2024-04-12T04:03:02.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be0f051a1ff6d94d-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.317 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.317 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.317 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:03:02.317 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:03:02.318 DEBUG:boto:Method: GET 2024-04-12T04:03:02.318 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:03:02.318 DEBUG:boto:Data: 2024-04-12T04:03:02.318 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.318 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.318 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.318 DEBUG:boto:Params: {} 2024-04-12T04:03:02.319 DEBUG:boto:Token: None 2024-04-12T04:03:02.319 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.319 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000081dc30993688738-006618b276-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.322 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.322 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.322 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.322 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:03:02.322 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:03:02.323 DEBUG:boto:Method: GET 2024-04-12T04:03:02.323 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:03:02.323 DEBUG:boto:Data: 2024-04-12T04:03:02.323 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.323 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.323 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.323 DEBUG:boto:Params: {} 2024-04-12T04:03:02.323 DEBUG:boto:Token: None 2024-04-12T04:03:02.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.323 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065439b445e005dc4-006618b276-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.326 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.327 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:03:02.327 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:03:02.327 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.327 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:03:02.327 DEBUG:boto:Data: 2024-04-12T04:03:02.327 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.327 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.327 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.327 DEBUG:boto:Params: {} 2024-04-12T04:03:02.328 DEBUG:boto:Token: None 2024-04-12T04:03:02.328 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj17 2024-04-12T04:03:02.328 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VOr/L97B8HfkmnxFE5fG8oM5gck= 2024-04-12T04:03:02.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VOr/L97B8HfkmnxFE5fG8oM5gck=', 'Content-Length': '0'} 2024-04-12T04:03:02.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005da97ce84dea7b60-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.330 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:03:02.330 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:03:02.330 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.330 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:03:02.330 DEBUG:boto:Data: 2024-04-12T04:03:02.330 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.330 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.330 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.330 DEBUG:boto:Params: {} 2024-04-12T04:03:02.331 DEBUG:boto:Token: None 2024-04-12T04:03:02.331 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj17 2024-04-12T04:03:02.331 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VOr/L97B8HfkmnxFE5fG8oM5gck= 2024-04-12T04:03:02.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VOr/L97B8HfkmnxFE5fG8oM5gck=', 'Content-Length': '0'} 2024-04-12T04:03:02.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3595b5cf19772fa-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:03:02.333 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:03:02.333 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:03:02.333 DEBUG:boto:Method: GET 2024-04-12T04:03:02.333 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:03:02.333 DEBUG:boto:Data: 2024-04-12T04:03:02.333 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.334 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.334 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.334 DEBUG:boto:Params: {} 2024-04-12T04:03:02.334 DEBUG:boto:Token: None 2024-04-12T04:03:02.334 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj17 2024-04-12T04:03:02.334 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UtMDIU1S7gUBFnsZCSubGrJ7nwU= 2024-04-12T04:03:02.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UtMDIU1S7gUBFnsZCSubGrJ7nwU=', 'Content-Length': '0'} 2024-04-12T04:03:02.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1c2c6da4e12b24f-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.336 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:03:02.336 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:03:02.336 DEBUG:boto:Method: GET 2024-04-12T04:03:02.336 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:03:02.336 DEBUG:boto:Data: 2024-04-12T04:03:02.337 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.337 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.337 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.337 DEBUG:boto:Params: {} 2024-04-12T04:03:02.337 DEBUG:boto:Token: None 2024-04-12T04:03:02.337 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj17 2024-04-12T04:03:02.337 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UtMDIU1S7gUBFnsZCSubGrJ7nwU= 2024-04-12T04:03:02.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UtMDIU1S7gUBFnsZCSubGrJ7nwU=', 'Content-Length': '0'} 2024-04-12T04:03:02.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092559195963df75b-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.340 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.340 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.340 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:03:02.340 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:03:02.340 DEBUG:boto:Method: GET 2024-04-12T04:03:02.340 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:03:02.340 DEBUG:boto:Data: 2024-04-12T04:03:02.340 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.340 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.340 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.340 DEBUG:boto:Params: {} 2024-04-12T04:03:02.340 DEBUG:boto:Token: None 2024-04-12T04:03:02.341 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.341 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003625599bc3a775ee-006618b276-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.344 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.344 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.344 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.344 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:03:02.344 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:03:02.344 DEBUG:boto:Method: GET 2024-04-12T04:03:02.345 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:03:02.345 DEBUG:boto:Data: 2024-04-12T04:03:02.345 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.345 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.345 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.345 DEBUG:boto:Params: {} 2024-04-12T04:03:02.345 DEBUG:boto:Token: None 2024-04-12T04:03:02.345 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.345 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000506ae672f2d4cf53-006618b276-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.390 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:03:02.391 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:03:02.391 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:03:02.391 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.391 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.391 DEBUG:boto:Method: GET 2024-04-12T04:03:02.391 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.392 DEBUG:boto:Data: 2024-04-12T04:03:02.392 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.392 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.392 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.392 DEBUG:boto:Params: {} 2024-04-12T04:03:02.392 DEBUG:boto:Token: None 2024-04-12T04:03:02.392 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.392 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0fwMe02c81YwuUUPHcF2Hvrqybo= 2024-04-12T04:03:02.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0fwMe02c81YwuUUPHcF2Hvrqybo=', 'Content-Length': '0'} 2024-04-12T04:03:02.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d355e9510d3d4dad-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.395 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:03:02.395 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:03:02.395 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.395 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.395 DEBUG:boto:Method: GET 2024-04-12T04:03:02.395 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.395 DEBUG:boto:Data: 2024-04-12T04:03:02.395 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.396 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.396 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.396 DEBUG:boto:Params: {} 2024-04-12T04:03:02.396 DEBUG:boto:Token: None 2024-04-12T04:03:02.396 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.396 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0fwMe02c81YwuUUPHcF2Hvrqybo= 2024-04-12T04:03:02.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0fwMe02c81YwuUUPHcF2Hvrqybo=', 'Content-Length': '0'} 2024-04-12T04:03:02.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dae6d0cfe3963d26-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.398 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:03:02.398 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:03:02.399 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.399 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.399 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.399 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.399 DEBUG:boto:Data: 2024-04-12T04:03:02.399 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.399 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.399 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.399 DEBUG:boto:Params: {} 2024-04-12T04:03:02.399 DEBUG:boto:Token: None 2024-04-12T04:03:02.400 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.400 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KmARztdZHjg7JAMPeWNRxtNXmyo= 2024-04-12T04:03:02.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KmARztdZHjg7JAMPeWNRxtNXmyo=', 'Content-Length': '0'} 2024-04-12T04:03:02.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6c746fb4ef10677-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.402 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:03:02.402 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:03:02.402 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.402 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.402 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.402 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.402 DEBUG:boto:Data: 2024-04-12T04:03:02.402 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.402 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.403 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.403 DEBUG:boto:Params: {} 2024-04-12T04:03:02.403 DEBUG:boto:Token: None 2024-04-12T04:03:02.403 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.403 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KmARztdZHjg7JAMPeWNRxtNXmyo= 2024-04-12T04:03:02.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KmARztdZHjg7JAMPeWNRxtNXmyo=', 'Content-Length': '0'} 2024-04-12T04:03:02.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0809481b12be3c1-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:03:02.405 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:03:02.405 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:03:02.405 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.405 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.405 DEBUG:boto:Method: GET 2024-04-12T04:03:02.406 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.406 DEBUG:boto:Data: 2024-04-12T04:03:02.406 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.406 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.406 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.406 DEBUG:boto:Params: {} 2024-04-12T04:03:02.406 DEBUG:boto:Token: None 2024-04-12T04:03:02.406 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.406 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0fwMe02c81YwuUUPHcF2Hvrqybo= 2024-04-12T04:03:02.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0fwMe02c81YwuUUPHcF2Hvrqybo=', 'Content-Length': '0'} 2024-04-12T04:03:02.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092b0449a1a110485-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.409 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:03:02.409 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:03:02.409 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.409 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.409 DEBUG:boto:Method: GET 2024-04-12T04:03:02.409 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.409 DEBUG:boto:Data: 2024-04-12T04:03:02.409 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.409 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.409 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.410 DEBUG:boto:Params: {} 2024-04-12T04:03:02.410 DEBUG:boto:Token: None 2024-04-12T04:03:02.410 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:03:02.410 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0fwMe02c81YwuUUPHcF2Hvrqybo= 2024-04-12T04:03:02.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0fwMe02c81YwuUUPHcF2Hvrqybo=', 'Content-Length': '0'} 2024-04-12T04:03:02.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000005b4da953222fdd-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.412 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.412 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.412 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:03:02.412 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:03:02.413 DEBUG:boto:Method: GET 2024-04-12T04:03:02.413 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:03:02.413 DEBUG:boto:Data: 2024-04-12T04:03:02.413 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.413 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.413 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.413 DEBUG:boto:Params: {} 2024-04-12T04:03:02.413 DEBUG:boto:Token: None 2024-04-12T04:03:02.413 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.413 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093847c7e70031aec-006618b276-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.416 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.416 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.416 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.417 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:03:02.417 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:03:02.417 DEBUG:boto:Method: GET 2024-04-12T04:03:02.417 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:03:02.417 DEBUG:boto:Data: 2024-04-12T04:03:02.417 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.417 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.417 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.417 DEBUG:boto:Params: {} 2024-04-12T04:03:02.417 DEBUG:boto:Token: None 2024-04-12T04:03:02.417 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.418 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9f4e742f325e47b-006618b276-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.421 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.421 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:03:02.421 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:03:02.421 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.421 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:03:02.421 DEBUG:boto:Data: 2024-04-12T04:03:02.421 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.422 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.422 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.422 DEBUG:boto:Params: {} 2024-04-12T04:03:02.422 DEBUG:boto:Token: None 2024-04-12T04:03:02.422 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj22 2024-04-12T04:03:02.422 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Uz7gGyAA8/LUy7a1AWNgFb2wcso= 2024-04-12T04:03:02.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Uz7gGyAA8/LUy7a1AWNgFb2wcso=', 'Content-Length': '0'} 2024-04-12T04:03:02.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc49515811d4ed78-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.424 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:03:02.424 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:03:02.424 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.425 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:03:02.425 DEBUG:boto:Data: 2024-04-12T04:03:02.425 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.425 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.425 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.425 DEBUG:boto:Params: {} 2024-04-12T04:03:02.425 DEBUG:boto:Token: None 2024-04-12T04:03:02.425 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj22 2024-04-12T04:03:02.425 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Uz7gGyAA8/LUy7a1AWNgFb2wcso= 2024-04-12T04:03:02.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Uz7gGyAA8/LUy7a1AWNgFb2wcso=', 'Content-Length': '0'} 2024-04-12T04:03:02.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf18b48a6189e0bd-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:03:02.428 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:03:02.428 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:03:02.428 DEBUG:boto:Method: GET 2024-04-12T04:03:02.428 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:03:02.428 DEBUG:boto:Data: 2024-04-12T04:03:02.428 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.428 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.428 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.428 DEBUG:boto:Params: {} 2024-04-12T04:03:02.428 DEBUG:boto:Token: None 2024-04-12T04:03:02.429 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj22 2024-04-12T04:03:02.429 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vk/iG8opoJn3INuICzm/OwpnVZs= 2024-04-12T04:03:02.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vk/iG8opoJn3INuICzm/OwpnVZs=', 'Content-Length': '0'} 2024-04-12T04:03:02.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfb2670a544f8e11-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.478 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:03:02.479 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:03:02.479 DEBUG:boto:Method: GET 2024-04-12T04:03:02.479 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:03:02.479 DEBUG:boto:Data: 2024-04-12T04:03:02.479 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.479 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.479 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.479 DEBUG:boto:Params: {} 2024-04-12T04:03:02.479 DEBUG:boto:Token: None 2024-04-12T04:03:02.480 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj22 2024-04-12T04:03:02.480 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vk/iG8opoJn3INuICzm/OwpnVZs= 2024-04-12T04:03:02.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vk/iG8opoJn3INuICzm/OwpnVZs=', 'Content-Length': '0'} 2024-04-12T04:03:02.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000562fa08bc8ed6b78-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.482 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.482 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.482 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:03:02.483 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:03:02.483 DEBUG:boto:Method: GET 2024-04-12T04:03:02.483 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:03:02.483 DEBUG:boto:Data: 2024-04-12T04:03:02.483 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.483 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.483 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.483 DEBUG:boto:Params: {} 2024-04-12T04:03:02.483 DEBUG:boto:Token: None 2024-04-12T04:03:02.483 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.484 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000496b1cd788b9eeb2-006618b276-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.486 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.487 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.487 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.487 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:03:02.487 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:03:02.487 DEBUG:boto:Method: GET 2024-04-12T04:03:02.487 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:03:02.487 DEBUG:boto:Data: 2024-04-12T04:03:02.487 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.487 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.487 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.488 DEBUG:boto:Params: {} 2024-04-12T04:03:02.488 DEBUG:boto:Token: None 2024-04-12T04:03:02.488 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.488 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bcf84a4fade8def-006618b276-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.491 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:03:02.491 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:03:02.492 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:03:02.492 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.492 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.492 DEBUG:boto:Method: GET 2024-04-12T04:03:02.492 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.492 DEBUG:boto:Data: 2024-04-12T04:03:02.492 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.492 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.492 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.492 DEBUG:boto:Params: {} 2024-04-12T04:03:02.492 DEBUG:boto:Token: None 2024-04-12T04:03:02.493 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.493 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pUPD5hm2KEsDkmtE8IMoYPlCwdQ= 2024-04-12T04:03:02.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pUPD5hm2KEsDkmtE8IMoYPlCwdQ=', 'Content-Length': '0'} 2024-04-12T04:03:02.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c187bac0e5cfd424-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.495 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:03:02.495 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:03:02.495 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.495 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.495 DEBUG:boto:Method: GET 2024-04-12T04:03:02.495 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.495 DEBUG:boto:Data: 2024-04-12T04:03:02.495 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.495 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.495 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.496 DEBUG:boto:Params: {} 2024-04-12T04:03:02.496 DEBUG:boto:Token: None 2024-04-12T04:03:02.496 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.496 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pUPD5hm2KEsDkmtE8IMoYPlCwdQ= 2024-04-12T04:03:02.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pUPD5hm2KEsDkmtE8IMoYPlCwdQ=', 'Content-Length': '0'} 2024-04-12T04:03:02.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cf6cefd2b43ed23-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.498 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:03:02.498 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:03:02.498 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.498 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.499 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.499 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.499 DEBUG:boto:Data: 2024-04-12T04:03:02.499 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.499 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.499 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.499 DEBUG:boto:Params: {} 2024-04-12T04:03:02.499 DEBUG:boto:Token: None 2024-04-12T04:03:02.499 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.499 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mVfhpC983gyXhijBhvwbCQycJTI= 2024-04-12T04:03:02.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mVfhpC983gyXhijBhvwbCQycJTI=', 'Content-Length': '0'} 2024-04-12T04:03:02.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e726cd1dac34b95-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.502 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:03:02.502 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:03:02.502 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.502 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.502 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.502 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.502 DEBUG:boto:Data: 2024-04-12T04:03:02.502 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.502 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.502 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.502 DEBUG:boto:Params: {} 2024-04-12T04:03:02.503 DEBUG:boto:Token: None 2024-04-12T04:03:02.503 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.503 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mVfhpC983gyXhijBhvwbCQycJTI= 2024-04-12T04:03:02.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mVfhpC983gyXhijBhvwbCQycJTI=', 'Content-Length': '0'} 2024-04-12T04:03:02.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f210baf48884f29b-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:03:02.505 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:03:02.505 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:03:02.505 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.505 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.506 DEBUG:boto:Method: GET 2024-04-12T04:03:02.506 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.506 DEBUG:boto:Data: 2024-04-12T04:03:02.506 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.506 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.506 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.506 DEBUG:boto:Params: {} 2024-04-12T04:03:02.506 DEBUG:boto:Token: None 2024-04-12T04:03:02.506 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.506 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pUPD5hm2KEsDkmtE8IMoYPlCwdQ= 2024-04-12T04:03:02.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pUPD5hm2KEsDkmtE8IMoYPlCwdQ=', 'Content-Length': '0'} 2024-04-12T04:03:02.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d274678c66a63c79-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.554 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:03:02.555 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:03:02.555 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.555 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.555 DEBUG:boto:Method: GET 2024-04-12T04:03:02.555 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.555 DEBUG:boto:Data: 2024-04-12T04:03:02.555 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.555 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.555 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.555 DEBUG:boto:Params: {} 2024-04-12T04:03:02.556 DEBUG:boto:Token: None 2024-04-12T04:03:02.557 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:03:02.557 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pUPD5hm2KEsDkmtE8IMoYPlCwdQ= 2024-04-12T04:03:02.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pUPD5hm2KEsDkmtE8IMoYPlCwdQ=', 'Content-Length': '0'} 2024-04-12T04:03:02.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d1ccc9c87099706-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.560 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.560 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.560 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:03:02.560 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:03:02.560 DEBUG:boto:Method: GET 2024-04-12T04:03:02.560 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:03:02.560 DEBUG:boto:Data: 2024-04-12T04:03:02.560 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.560 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.560 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.561 DEBUG:boto:Params: {} 2024-04-12T04:03:02.561 DEBUG:boto:Token: None 2024-04-12T04:03:02.561 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.561 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e42387b020da204-006618b276-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.563 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.564 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.564 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.564 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:03:02.564 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:03:02.564 DEBUG:boto:Method: GET 2024-04-12T04:03:02.564 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:03:02.564 DEBUG:boto:Data: 2024-04-12T04:03:02.565 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.565 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.565 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.565 DEBUG:boto:Params: {} 2024-04-12T04:03:02.565 DEBUG:boto:Token: None 2024-04-12T04:03:02.565 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.565 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000954a266c991b0589-006618b276-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.568 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.569 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:03:02.569 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:03:02.569 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.569 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:03:02.569 DEBUG:boto:Data: 2024-04-12T04:03:02.569 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.569 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.569 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.569 DEBUG:boto:Params: {} 2024-04-12T04:03:02.570 DEBUG:boto:Token: None 2024-04-12T04:03:02.570 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj33 2024-04-12T04:03:02.570 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZvE7K2p8NAOu1gK1CjS7qIyUEZQ= 2024-04-12T04:03:02.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZvE7K2p8NAOu1gK1CjS7qIyUEZQ=', 'Content-Length': '0'} 2024-04-12T04:03:02.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab3370b14f8be2f5-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.572 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:03:02.572 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:03:02.572 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.572 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:03:02.572 DEBUG:boto:Data: 2024-04-12T04:03:02.572 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.572 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.572 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.573 DEBUG:boto:Params: {} 2024-04-12T04:03:02.573 DEBUG:boto:Token: None 2024-04-12T04:03:02.573 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj33 2024-04-12T04:03:02.573 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZvE7K2p8NAOu1gK1CjS7qIyUEZQ= 2024-04-12T04:03:02.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZvE7K2p8NAOu1gK1CjS7qIyUEZQ=', 'Content-Length': '0'} 2024-04-12T04:03:02.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2723054a3c08ddd-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:03:02.575 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:03:02.575 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:03:02.575 DEBUG:boto:Method: GET 2024-04-12T04:03:02.575 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:03:02.575 DEBUG:boto:Data: 2024-04-12T04:03:02.576 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.576 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.576 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.576 DEBUG:boto:Params: {} 2024-04-12T04:03:02.576 DEBUG:boto:Token: None 2024-04-12T04:03:02.576 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj33 2024-04-12T04:03:02.576 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jqvFaYiNC7UWIDNXR7MCentTKew= 2024-04-12T04:03:02.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jqvFaYiNC7UWIDNXR7MCentTKew=', 'Content-Length': '0'} 2024-04-12T04:03:02.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000979ff96230dde50b-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.578 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:03:02.579 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:03:02.579 DEBUG:boto:Method: GET 2024-04-12T04:03:02.579 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:03:02.579 DEBUG:boto:Data: 2024-04-12T04:03:02.579 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.579 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.579 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.579 DEBUG:boto:Params: {} 2024-04-12T04:03:02.579 DEBUG:boto:Token: None 2024-04-12T04:03:02.579 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj33 2024-04-12T04:03:02.579 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jqvFaYiNC7UWIDNXR7MCentTKew= 2024-04-12T04:03:02.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jqvFaYiNC7UWIDNXR7MCentTKew=', 'Content-Length': '0'} 2024-04-12T04:03:02.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f96225384f6e7dc5-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.622 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.623 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.623 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:03:02.623 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:03:02.623 DEBUG:boto:Method: GET 2024-04-12T04:03:02.623 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:03:02.623 DEBUG:boto:Data: 2024-04-12T04:03:02.623 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.623 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.623 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.623 DEBUG:boto:Params: {} 2024-04-12T04:03:02.623 DEBUG:boto:Token: None 2024-04-12T04:03:02.624 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.624 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000638f0646f793295-006618b276-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.626 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.627 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.627 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.627 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:03:02.627 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:03:02.627 DEBUG:boto:Method: GET 2024-04-12T04:03:02.627 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:03:02.627 DEBUG:boto:Data: 2024-04-12T04:03:02.627 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.627 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.627 DEBUG:boto:Params: {} 2024-04-12T04:03:02.628 DEBUG:boto:Token: None 2024-04-12T04:03:02.628 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de7b608672122f13-006618b276-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.631 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:03:02.631 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:03:02.631 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:03:02.631 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.632 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.632 DEBUG:boto:Method: GET 2024-04-12T04:03:02.632 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.632 DEBUG:boto:Data: 2024-04-12T04:03:02.632 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.632 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.632 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.632 DEBUG:boto:Params: {} 2024-04-12T04:03:02.632 DEBUG:boto:Token: None 2024-04-12T04:03:02.632 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.633 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vG1fiXWtUSHXxJpuBilQorZmdjQ= 2024-04-12T04:03:02.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vG1fiXWtUSHXxJpuBilQorZmdjQ=', 'Content-Length': '0'} 2024-04-12T04:03:02.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c67630e9d934fc0d-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.682 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:03:02.683 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:03:02.683 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.683 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.683 DEBUG:boto:Method: GET 2024-04-12T04:03:02.683 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.683 DEBUG:boto:Data: 2024-04-12T04:03:02.683 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.683 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.683 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.683 DEBUG:boto:Params: {} 2024-04-12T04:03:02.684 DEBUG:boto:Token: None 2024-04-12T04:03:02.684 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.684 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vG1fiXWtUSHXxJpuBilQorZmdjQ= 2024-04-12T04:03:02.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vG1fiXWtUSHXxJpuBilQorZmdjQ=', 'Content-Length': '0'} 2024-04-12T04:03:02.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a659cf0e0b07674-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.686 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:03:02.686 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:03:02.686 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.686 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.687 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.687 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.687 DEBUG:boto:Data: 2024-04-12T04:03:02.687 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.687 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.687 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.687 DEBUG:boto:Params: {} 2024-04-12T04:03:02.687 DEBUG:boto:Token: None 2024-04-12T04:03:02.687 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.687 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e2wMEAZpR6QaJDXJZ7FOPcP7A+s= 2024-04-12T04:03:02.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e2wMEAZpR6QaJDXJZ7FOPcP7A+s=', 'Content-Length': '0'} 2024-04-12T04:03:02.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b46034d8dedd17ef-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.690 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:03:02.690 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:03:02.690 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.690 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.690 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.690 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.690 DEBUG:boto:Data: 2024-04-12T04:03:02.690 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.690 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.690 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.691 DEBUG:boto:Params: {} 2024-04-12T04:03:02.691 DEBUG:boto:Token: None 2024-04-12T04:03:02.691 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.691 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e2wMEAZpR6QaJDXJZ7FOPcP7A+s= 2024-04-12T04:03:02.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e2wMEAZpR6QaJDXJZ7FOPcP7A+s=', 'Content-Length': '0'} 2024-04-12T04:03:02.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000508d63d7a2fad17-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:03:02.693 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:03:02.693 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:03:02.693 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.693 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.694 DEBUG:boto:Method: GET 2024-04-12T04:03:02.694 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.694 DEBUG:boto:Data: 2024-04-12T04:03:02.694 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.694 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.694 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.694 DEBUG:boto:Params: {} 2024-04-12T04:03:02.694 DEBUG:boto:Token: None 2024-04-12T04:03:02.694 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.694 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vG1fiXWtUSHXxJpuBilQorZmdjQ= 2024-04-12T04:03:02.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vG1fiXWtUSHXxJpuBilQorZmdjQ=', 'Content-Length': '0'} 2024-04-12T04:03:02.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b59d0f9dde672526-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.697 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:03:02.697 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:03:02.697 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.697 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.697 DEBUG:boto:Method: GET 2024-04-12T04:03:02.697 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.697 DEBUG:boto:Data: 2024-04-12T04:03:02.697 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.697 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.697 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.697 DEBUG:boto:Params: {} 2024-04-12T04:03:02.698 DEBUG:boto:Token: None 2024-04-12T04:03:02.698 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:03:02.698 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vG1fiXWtUSHXxJpuBilQorZmdjQ= 2024-04-12T04:03:02.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vG1fiXWtUSHXxJpuBilQorZmdjQ=', 'Content-Length': '0'} 2024-04-12T04:03:02.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7943de2ec87d61c-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.700 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.700 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.700 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:03:02.700 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:03:02.700 DEBUG:boto:Method: GET 2024-04-12T04:03:02.700 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:03:02.700 DEBUG:boto:Data: 2024-04-12T04:03:02.701 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.701 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.701 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.701 DEBUG:boto:Params: {} 2024-04-12T04:03:02.701 DEBUG:boto:Token: None 2024-04-12T04:03:02.701 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.701 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.704 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcba44cbdf101c56-006618b276-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.704 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.705 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.705 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.705 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:03:02.705 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:03:02.705 DEBUG:boto:Method: GET 2024-04-12T04:03:02.705 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:03:02.705 DEBUG:boto:Data: 2024-04-12T04:03:02.705 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.705 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.705 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.705 DEBUG:boto:Params: {} 2024-04-12T04:03:02.706 DEBUG:boto:Token: None 2024-04-12T04:03:02.706 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.706 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bafb76fcbaa68f5-006618b276-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.709 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.709 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:03:02.709 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:03:02.710 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.710 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:03:02.710 DEBUG:boto:Data: 2024-04-12T04:03:02.710 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.710 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.710 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.710 DEBUG:boto:Params: {} 2024-04-12T04:03:02.710 DEBUG:boto:Token: None 2024-04-12T04:03:02.710 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj34 2024-04-12T04:03:02.710 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jN5yPyJpiLU42Yq7YwypOUCExBc= 2024-04-12T04:03:02.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jN5yPyJpiLU42Yq7YwypOUCExBc=', 'Content-Length': '0'} 2024-04-12T04:03:02.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038f8f0ebdb3f25f4-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.713 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:03:02.713 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:03:02.713 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.713 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:03:02.713 DEBUG:boto:Data: 2024-04-12T04:03:02.713 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.713 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.713 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.713 DEBUG:boto:Params: {} 2024-04-12T04:03:02.714 DEBUG:boto:Token: None 2024-04-12T04:03:02.714 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj34 2024-04-12T04:03:02.714 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jN5yPyJpiLU42Yq7YwypOUCExBc= 2024-04-12T04:03:02.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jN5yPyJpiLU42Yq7YwypOUCExBc=', 'Content-Length': '0'} 2024-04-12T04:03:02.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cf1c5b5560fc8c6-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:03:02.716 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:03:02.716 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:03:02.716 DEBUG:boto:Method: GET 2024-04-12T04:03:02.716 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:03:02.716 DEBUG:boto:Data: 2024-04-12T04:03:02.716 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.716 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.717 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.717 DEBUG:boto:Params: {} 2024-04-12T04:03:02.717 DEBUG:boto:Token: None 2024-04-12T04:03:02.717 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj34 2024-04-12T04:03:02.717 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lTaJ6bVvegL/pGU+HlQnGgmsS6w= 2024-04-12T04:03:02.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lTaJ6bVvegL/pGU+HlQnGgmsS6w=', 'Content-Length': '0'} 2024-04-12T04:03:02.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff1e4957be4d1342-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.719 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:03:02.719 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:03:02.720 DEBUG:boto:Method: GET 2024-04-12T04:03:02.720 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:03:02.720 DEBUG:boto:Data: 2024-04-12T04:03:02.720 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.720 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.720 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.720 DEBUG:boto:Params: {} 2024-04-12T04:03:02.720 DEBUG:boto:Token: None 2024-04-12T04:03:02.720 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj34 2024-04-12T04:03:02.720 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lTaJ6bVvegL/pGU+HlQnGgmsS6w= 2024-04-12T04:03:02.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lTaJ6bVvegL/pGU+HlQnGgmsS6w=', 'Content-Length': '0'} 2024-04-12T04:03:02.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096ba100347fcd1a9-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.723 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.723 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.723 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:03:02.723 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:03:02.723 DEBUG:boto:Method: GET 2024-04-12T04:03:02.723 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:03:02.723 DEBUG:boto:Data: 2024-04-12T04:03:02.723 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.723 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.724 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.724 DEBUG:boto:Params: {} 2024-04-12T04:03:02.724 DEBUG:boto:Token: None 2024-04-12T04:03:02.724 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.724 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002df9b3d891367889-006618b276-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.770 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.771 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.771 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.771 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:03:02.771 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:03:02.771 DEBUG:boto:Method: GET 2024-04-12T04:03:02.771 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:03:02.772 DEBUG:boto:Data: 2024-04-12T04:03:02.772 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.772 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.772 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.772 DEBUG:boto:Params: {} 2024-04-12T04:03:02.772 DEBUG:boto:Token: None 2024-04-12T04:03:02.772 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.772 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037c773482fa628f8-006618b276-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.776 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:03:02.776 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:03:02.777 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:03:02.777 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.777 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.777 DEBUG:boto:Method: GET 2024-04-12T04:03:02.777 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.777 DEBUG:boto:Data: 2024-04-12T04:03:02.777 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.777 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.777 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.777 DEBUG:boto:Params: {} 2024-04-12T04:03:02.777 DEBUG:boto:Token: None 2024-04-12T04:03:02.778 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.778 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YY+LU7TBYRv9rsZHXhD2p2kw/jY= 2024-04-12T04:03:02.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YY+LU7TBYRv9rsZHXhD2p2kw/jY=', 'Content-Length': '0'} 2024-04-12T04:03:02.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007481640de39bd5cb-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.780 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:03:02.780 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:03:02.780 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.780 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.781 DEBUG:boto:Method: GET 2024-04-12T04:03:02.781 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.781 DEBUG:boto:Data: 2024-04-12T04:03:02.781 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.781 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.781 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.781 DEBUG:boto:Params: {} 2024-04-12T04:03:02.781 DEBUG:boto:Token: None 2024-04-12T04:03:02.781 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.781 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YY+LU7TBYRv9rsZHXhD2p2kw/jY= 2024-04-12T04:03:02.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YY+LU7TBYRv9rsZHXhD2p2kw/jY=', 'Content-Length': '0'} 2024-04-12T04:03:02.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d36c9fbfb5075a-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.784 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:03:02.784 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:03:02.784 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.784 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.784 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.784 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.785 DEBUG:boto:Data: 2024-04-12T04:03:02.785 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.785 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.785 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.785 DEBUG:boto:Params: {} 2024-04-12T04:03:02.785 DEBUG:boto:Token: None 2024-04-12T04:03:02.785 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.785 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZA/ms+Dfz0aWqRCRG8Ov+2qUawk= 2024-04-12T04:03:02.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZA/ms+Dfz0aWqRCRG8Ov+2qUawk=', 'Content-Length': '0'} 2024-04-12T04:03:02.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eea003a45c2201f4-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.787 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:03:02.787 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:03:02.787 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.787 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.788 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.788 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.788 DEBUG:boto:Data: 2024-04-12T04:03:02.788 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.788 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.788 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.788 DEBUG:boto:Params: {} 2024-04-12T04:03:02.788 DEBUG:boto:Token: None 2024-04-12T04:03:02.788 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.788 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZA/ms+Dfz0aWqRCRG8Ov+2qUawk= 2024-04-12T04:03:02.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZA/ms+Dfz0aWqRCRG8Ov+2qUawk=', 'Content-Length': '0'} 2024-04-12T04:03:02.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012538c3f0ceb1ba8-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:03:02.791 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:03:02.791 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:03:02.791 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.791 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.791 DEBUG:boto:Method: GET 2024-04-12T04:03:02.791 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.791 DEBUG:boto:Data: 2024-04-12T04:03:02.791 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.791 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.791 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.792 DEBUG:boto:Params: {} 2024-04-12T04:03:02.792 DEBUG:boto:Token: None 2024-04-12T04:03:02.792 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.792 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YY+LU7TBYRv9rsZHXhD2p2kw/jY= 2024-04-12T04:03:02.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YY+LU7TBYRv9rsZHXhD2p2kw/jY=', 'Content-Length': '0'} 2024-04-12T04:03:02.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c20b83037b62010a-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.794 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:03:02.794 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:03:02.794 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.795 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.795 DEBUG:boto:Method: GET 2024-04-12T04:03:02.795 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.795 DEBUG:boto:Data: 2024-04-12T04:03:02.795 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.795 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.795 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.795 DEBUG:boto:Params: {} 2024-04-12T04:03:02.795 DEBUG:boto:Token: None 2024-04-12T04:03:02.795 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:03:02.795 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YY+LU7TBYRv9rsZHXhD2p2kw/jY= 2024-04-12T04:03:02.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YY+LU7TBYRv9rsZHXhD2p2kw/jY=', 'Content-Length': '0'} 2024-04-12T04:03:02.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dc8a5e7a8e6c1cc-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.839 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.839 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.839 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:03:02.839 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:03:02.839 DEBUG:boto:Method: GET 2024-04-12T04:03:02.839 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:03:02.839 DEBUG:boto:Data: 2024-04-12T04:03:02.839 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.839 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.840 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.840 DEBUG:boto:Params: {} 2024-04-12T04:03:02.840 DEBUG:boto:Token: None 2024-04-12T04:03:02.840 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.840 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065ad9de33bfc7d27-006618b276-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.843 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.843 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.843 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.844 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:03:02.844 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:03:02.844 DEBUG:boto:Method: GET 2024-04-12T04:03:02.844 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:03:02.844 DEBUG:boto:Data: 2024-04-12T04:03:02.844 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.844 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.844 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.844 DEBUG:boto:Params: {} 2024-04-12T04:03:02.844 DEBUG:boto:Token: None 2024-04-12T04:03:02.844 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.845 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0c9f2d131a48a10-006618b276-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.848 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.848 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:03:02.848 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:03:02.848 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.848 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:03:02.848 DEBUG:boto:Data: 2024-04-12T04:03:02.848 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.848 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.849 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.849 DEBUG:boto:Params: {} 2024-04-12T04:03:02.849 DEBUG:boto:Token: None 2024-04-12T04:03:02.849 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj44 2024-04-12T04:03:02.849 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X5HR9lTN+3JmkAsGDhthxs/LVnI= 2024-04-12T04:03:02.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X5HR9lTN+3JmkAsGDhthxs/LVnI=', 'Content-Length': '0'} 2024-04-12T04:03:02.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6c2e24a6e2547df-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.851 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:03:02.851 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:03:02.852 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.852 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:03:02.852 DEBUG:boto:Data: 2024-04-12T04:03:02.852 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.852 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.852 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.852 DEBUG:boto:Params: {} 2024-04-12T04:03:02.852 DEBUG:boto:Token: None 2024-04-12T04:03:02.852 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj44 2024-04-12T04:03:02.852 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X5HR9lTN+3JmkAsGDhthxs/LVnI= 2024-04-12T04:03:02.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X5HR9lTN+3JmkAsGDhthxs/LVnI=', 'Content-Length': '0'} 2024-04-12T04:03:02.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a46cc80d31d3309a-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:03:02.854 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:03:02.855 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:03:02.855 DEBUG:boto:Method: GET 2024-04-12T04:03:02.855 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:03:02.855 DEBUG:boto:Data: 2024-04-12T04:03:02.855 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.855 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.855 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.855 DEBUG:boto:Params: {} 2024-04-12T04:03:02.855 DEBUG:boto:Token: None 2024-04-12T04:03:02.855 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj44 2024-04-12T04:03:02.856 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:y4gZ2U4h9cjsLwmOZxKCVFHEwt4= 2024-04-12T04:03:02.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y4gZ2U4h9cjsLwmOZxKCVFHEwt4=', 'Content-Length': '0'} 2024-04-12T04:03:02.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d59234ac0d3aa946-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.898 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:03:02.899 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:03:02.899 DEBUG:boto:Method: GET 2024-04-12T04:03:02.899 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:03:02.899 DEBUG:boto:Data: 2024-04-12T04:03:02.899 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.899 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.899 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.899 DEBUG:boto:Params: {} 2024-04-12T04:03:02.899 DEBUG:boto:Token: None 2024-04-12T04:03:02.899 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj44 2024-04-12T04:03:02.900 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:y4gZ2U4h9cjsLwmOZxKCVFHEwt4= 2024-04-12T04:03:02.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y4gZ2U4h9cjsLwmOZxKCVFHEwt4=', 'Content-Length': '0'} 2024-04-12T04:03:02.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000708811d628776587-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.902 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.902 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.902 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:03:02.902 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:03:02.903 DEBUG:boto:Method: GET 2024-04-12T04:03:02.903 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:03:02.903 DEBUG:boto:Data: 2024-04-12T04:03:02.903 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.903 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.903 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.903 DEBUG:boto:Params: {} 2024-04-12T04:03:02.903 DEBUG:boto:Token: None 2024-04-12T04:03:02.903 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.903 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.906 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000924bbe270c354de0-006618b276-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.906 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.906 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.906 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.906 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:03:02.907 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:03:02.907 DEBUG:boto:Method: GET 2024-04-12T04:03:02.907 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:03:02.907 DEBUG:boto:Data: 2024-04-12T04:03:02.907 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.907 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.907 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.907 DEBUG:boto:Params: {} 2024-04-12T04:03:02.907 DEBUG:boto:Token: None 2024-04-12T04:03:02.907 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.907 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e94461636bb2d9d-006618b276-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.913 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:03:02.914 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:03:02.914 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:03:02.914 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.914 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.914 DEBUG:boto:Method: GET 2024-04-12T04:03:02.914 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.914 DEBUG:boto:Data: 2024-04-12T04:03:02.914 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.915 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.915 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.915 DEBUG:boto:Params: {} 2024-04-12T04:03:02.915 DEBUG:boto:Token: None 2024-04-12T04:03:02.915 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.915 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ThthKkj7o1IDvJIhJNF8cPUsk8o= 2024-04-12T04:03:02.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ThthKkj7o1IDvJIhJNF8cPUsk8o=', 'Content-Length': '0'} 2024-04-12T04:03:02.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006991bdad9832512e-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.918 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:03:02.918 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:03:02.918 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.918 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.918 DEBUG:boto:Method: GET 2024-04-12T04:03:02.918 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.918 DEBUG:boto:Data: 2024-04-12T04:03:02.918 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.918 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.918 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.918 DEBUG:boto:Params: {} 2024-04-12T04:03:02.919 DEBUG:boto:Token: None 2024-04-12T04:03:02.919 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.919 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ThthKkj7o1IDvJIhJNF8cPUsk8o= 2024-04-12T04:03:02.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ThthKkj7o1IDvJIhJNF8cPUsk8o=', 'Content-Length': '0'} 2024-04-12T04:03:02.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d24137a32655e645-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.966 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:03:02.967 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:03:02.967 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.967 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.967 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.967 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.967 DEBUG:boto:Data: 2024-04-12T04:03:02.967 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.967 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.967 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.967 DEBUG:boto:Params: {} 2024-04-12T04:03:02.967 DEBUG:boto:Token: None 2024-04-12T04:03:02.968 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.968 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W2kva61A/jvT9H1CdJagK8lcCoM= 2024-04-12T04:03:02.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W2kva61A/jvT9H1CdJagK8lcCoM=', 'Content-Length': '0'} 2024-04-12T04:03:02.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000337da3bb2d1abd2a-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.970 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:03:02.970 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:03:02.970 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.970 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.970 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.970 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.971 DEBUG:boto:Data: 2024-04-12T04:03:02.971 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.971 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.971 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.971 DEBUG:boto:Params: {} 2024-04-12T04:03:02.971 DEBUG:boto:Token: None 2024-04-12T04:03:02.971 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.971 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W2kva61A/jvT9H1CdJagK8lcCoM= 2024-04-12T04:03:02.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W2kva61A/jvT9H1CdJagK8lcCoM=', 'Content-Length': '0'} 2024-04-12T04:03:02.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004530e62005e632d1-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:03:02.974 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:03:02.974 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:03:02.974 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.974 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.974 DEBUG:boto:Method: GET 2024-04-12T04:03:02.974 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.974 DEBUG:boto:Data: 2024-04-12T04:03:02.974 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.974 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.975 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.975 DEBUG:boto:Params: {} 2024-04-12T04:03:02.975 DEBUG:boto:Token: None 2024-04-12T04:03:02.975 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.975 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ThthKkj7o1IDvJIhJNF8cPUsk8o= 2024-04-12T04:03:02.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ThthKkj7o1IDvJIhJNF8cPUsk8o=', 'Content-Length': '0'} 2024-04-12T04:03:02.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bcb5562757863b8-006618b276-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.977 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:03:02.977 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:03:02.977 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.978 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.978 DEBUG:boto:Method: GET 2024-04-12T04:03:02.978 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.978 DEBUG:boto:Data: 2024-04-12T04:03:02.978 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.978 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.978 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.978 DEBUG:boto:Params: {} 2024-04-12T04:03:02.978 DEBUG:boto:Token: None 2024-04-12T04:03:02.978 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:03:02.979 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ThthKkj7o1IDvJIhJNF8cPUsk8o= 2024-04-12T04:03:02.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ThthKkj7o1IDvJIhJNF8cPUsk8o=', 'Content-Length': '0'} 2024-04-12T04:03:02.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000669a0f59fe652f80-006618b276-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.981 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.981 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.981 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:03:02.981 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:03:02.981 DEBUG:boto:Method: GET 2024-04-12T04:03:02.981 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:03:02.981 DEBUG:boto:Data: 2024-04-12T04:03:02.982 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.982 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.982 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.982 DEBUG:boto:Params: {} 2024-04-12T04:03:02.982 DEBUG:boto:Token: None 2024-04-12T04:03:02.982 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.982 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1ac6f2538bb66d7-006618b276-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.985 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.986 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:02.986 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:02.986 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:03:02.986 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:03:02.986 DEBUG:boto:Method: GET 2024-04-12T04:03:02.986 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:03:02.986 DEBUG:boto:Data: 2024-04-12T04:03:02.986 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.986 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.986 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.986 DEBUG:boto:Params: {} 2024-04-12T04:03:02.987 DEBUG:boto:Token: None 2024-04-12T04:03:02.987 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/ 2024-04-12T04:03:02.987 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4= 2024-04-12T04:03:02.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sizsICn8aTQ2dPEqr+rzN0five4=', 'Content-Length': '0'} 2024-04-12T04:03:02.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083289ce032064ad3-006618b277-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.990 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:02.990 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:03:02.990 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:03:02.990 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.991 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:03:02.991 DEBUG:boto:Data: 2024-04-12T04:03:02.991 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.991 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.991 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.991 DEBUG:boto:Params: {} 2024-04-12T04:03:02.991 DEBUG:boto:Token: None 2024-04-12T04:03:02.991 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj51 2024-04-12T04:03:02.991 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ujpqss3SGD/AuhUjmdtGY2YMx4A= 2024-04-12T04:03:02.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ujpqss3SGD/AuhUjmdtGY2YMx4A=', 'Content-Length': '0'} 2024-04-12T04:03:02.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b83914a376126bea-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.993 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:03:02.993 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:03:02.993 DEBUG:boto:Method: HEAD 2024-04-12T04:03:02.994 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:03:02.994 DEBUG:boto:Data: 2024-04-12T04:03:02.994 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.994 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:02.994 DEBUG:boto:Port: 8000 2024-04-12T04:03:02.994 DEBUG:boto:Params: {} 2024-04-12T04:03:02.994 DEBUG:boto:Token: None 2024-04-12T04:03:02.994 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj51 2024-04-12T04:03:02.994 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ujpqss3SGD/AuhUjmdtGY2YMx4A= 2024-04-12T04:03:02.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ujpqss3SGD/AuhUjmdtGY2YMx4A=', 'Content-Length': '0'} 2024-04-12T04:03:02.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afa05ac052cd7e7b-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:02.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:03:02.997 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:03:02.997 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:03:02.997 DEBUG:boto:Method: GET 2024-04-12T04:03:02.997 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:03:02.997 DEBUG:boto:Data: 2024-04-12T04:03:02.997 DEBUG:boto:Headers: {} 2024-04-12T04:03:02.997 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:02.997 DEBUG:boto:Port: 8001 2024-04-12T04:03:02.997 DEBUG:boto:Params: {} 2024-04-12T04:03:02.997 DEBUG:boto:Token: None 2024-04-12T04:03:02.997 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:02 GMT /dkyges-61/obj51 2024-04-12T04:03:02.998 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VRl7Bpj66qyTmRK4AxHJn+60s6c= 2024-04-12T04:03:02.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VRl7Bpj66qyTmRK4AxHJn+60s6c=', 'Content-Length': '0'} 2024-04-12T04:03:02.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c29eb63b793ac73-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.000 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:03:03.000 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:03:03.000 DEBUG:boto:Method: GET 2024-04-12T04:03:03.000 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:03:03.000 DEBUG:boto:Data: 2024-04-12T04:03:03.000 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.000 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.000 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.001 DEBUG:boto:Params: {} 2024-04-12T04:03:03.001 DEBUG:boto:Token: None 2024-04-12T04:03:03.001 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj51 2024-04-12T04:03:03.001 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pfpmCk2RnxsEdCfEQ2WBw0BKiqo= 2024-04-12T04:03:03.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pfpmCk2RnxsEdCfEQ2WBw0BKiqo=', 'Content-Length': '0'} 2024-04-12T04:03:03.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095a842ac5cc3f06a-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.003 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.003 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.003 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:03:03.003 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:03:03.003 DEBUG:boto:Method: GET 2024-04-12T04:03:03.003 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:03:03.004 DEBUG:boto:Data: 2024-04-12T04:03:03.004 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.004 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.004 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.004 DEBUG:boto:Params: {} 2024-04-12T04:03:03.004 DEBUG:boto:Token: None 2024-04-12T04:03:03.004 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88= 2024-04-12T04:03:03.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88=', 'Content-Length': '0'} 2024-04-12T04:03:03.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000246367496296deb8-006618b277-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.007 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:03.007 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.007 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.007 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:03:03.007 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:03:03.007 DEBUG:boto:Method: GET 2024-04-12T04:03:03.008 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:03:03.008 DEBUG:boto:Data: 2024-04-12T04:03:03.008 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.008 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.008 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.008 DEBUG:boto:Params: {} 2024-04-12T04:03:03.008 DEBUG:boto:Token: None 2024-04-12T04:03:03.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.008 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88= 2024-04-12T04:03:03.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88=', 'Content-Length': '0'} 2024-04-12T04:03:03.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017331ba5dbec496e-006618b277-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.011 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:03.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:03:03.012 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:03:03.012 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:03:03.012 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.012 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.012 DEBUG:boto:Method: GET 2024-04-12T04:03:03.012 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.012 DEBUG:boto:Data: 2024-04-12T04:03:03.012 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.012 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.012 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.013 DEBUG:boto:Params: {} 2024-04-12T04:03:03.013 DEBUG:boto:Token: None 2024-04-12T04:03:03.013 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.013 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NJuIAJMEPulsaoZqEBxu3ay/+68= 2024-04-12T04:03:03.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NJuIAJMEPulsaoZqEBxu3ay/+68=', 'Content-Length': '0'} 2024-04-12T04:03:03.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005db73ef7dc328269-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.015 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:03:03.015 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:03:03.015 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.015 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.015 DEBUG:boto:Method: GET 2024-04-12T04:03:03.015 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.016 DEBUG:boto:Data: 2024-04-12T04:03:03.016 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.016 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.016 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.016 DEBUG:boto:Params: {} 2024-04-12T04:03:03.016 DEBUG:boto:Token: None 2024-04-12T04:03:03.016 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.016 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NJuIAJMEPulsaoZqEBxu3ay/+68= 2024-04-12T04:03:03.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NJuIAJMEPulsaoZqEBxu3ay/+68=', 'Content-Length': '0'} 2024-04-12T04:03:03.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089254469b6ca1b36-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.058 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:03:03.059 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:03:03.059 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.059 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.059 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.059 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.059 DEBUG:boto:Data: 2024-04-12T04:03:03.059 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.059 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.059 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.059 DEBUG:boto:Params: {} 2024-04-12T04:03:03.060 DEBUG:boto:Token: None 2024-04-12T04:03:03.060 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.060 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:neeHe8o976vv0R6tN54bEGjHhLY= 2024-04-12T04:03:03.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:neeHe8o976vv0R6tN54bEGjHhLY=', 'Content-Length': '0'} 2024-04-12T04:03:03.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b92df2480a47b92-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.062 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:03:03.062 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:03:03.062 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.062 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.062 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.062 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.063 DEBUG:boto:Data: 2024-04-12T04:03:03.063 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.063 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.063 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.063 DEBUG:boto:Params: {} 2024-04-12T04:03:03.063 DEBUG:boto:Token: None 2024-04-12T04:03:03.063 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.063 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:neeHe8o976vv0R6tN54bEGjHhLY= 2024-04-12T04:03:03.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:neeHe8o976vv0R6tN54bEGjHhLY=', 'Content-Length': '0'} 2024-04-12T04:03:03.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf997170676a2dbd-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:03:03.066 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:03:03.066 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:03:03.066 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.066 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.066 DEBUG:boto:Method: GET 2024-04-12T04:03:03.066 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.066 DEBUG:boto:Data: 2024-04-12T04:03:03.066 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.066 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.066 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.067 DEBUG:boto:Params: {} 2024-04-12T04:03:03.067 DEBUG:boto:Token: None 2024-04-12T04:03:03.067 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.067 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NJuIAJMEPulsaoZqEBxu3ay/+68= 2024-04-12T04:03:03.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NJuIAJMEPulsaoZqEBxu3ay/+68=', 'Content-Length': '0'} 2024-04-12T04:03:03.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094171d5d22b77753-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.069 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:03:03.069 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:03:03.069 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.069 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.069 DEBUG:boto:Method: GET 2024-04-12T04:03:03.070 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.070 DEBUG:boto:Data: 2024-04-12T04:03:03.070 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.070 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.070 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.070 DEBUG:boto:Params: {} 2024-04-12T04:03:03.070 DEBUG:boto:Token: None 2024-04-12T04:03:03.070 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:03:03.070 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NJuIAJMEPulsaoZqEBxu3ay/+68= 2024-04-12T04:03:03.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NJuIAJMEPulsaoZqEBxu3ay/+68=', 'Content-Length': '0'} 2024-04-12T04:03:03.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f3e845205c8d90c-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.072 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.073 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.073 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:03:03.073 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:03:03.073 DEBUG:boto:Method: GET 2024-04-12T04:03:03.073 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:03:03.073 DEBUG:boto:Data: 2024-04-12T04:03:03.073 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.073 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.073 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.073 DEBUG:boto:Params: {} 2024-04-12T04:03:03.074 DEBUG:boto:Token: None 2024-04-12T04:03:03.074 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.074 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88= 2024-04-12T04:03:03.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88=', 'Content-Length': '0'} 2024-04-12T04:03:03.076 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f295268671d7c157-006618b277-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.076 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:03.077 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.077 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.077 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:03:03.077 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:03:03.077 DEBUG:boto:Method: GET 2024-04-12T04:03:03.077 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:03:03.077 DEBUG:boto:Data: 2024-04-12T04:03:03.077 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.077 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.078 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.078 DEBUG:boto:Params: {} 2024-04-12T04:03:03.078 DEBUG:boto:Token: None 2024-04-12T04:03:03.078 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.078 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88= 2024-04-12T04:03:03.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88=', 'Content-Length': '0'} 2024-04-12T04:03:03.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a00643ed76949ad6-006618b277-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.081 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:03.081 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:03:03.082 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:03:03.082 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.082 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:03:03.082 DEBUG:boto:Data: 2024-04-12T04:03:03.082 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.082 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.082 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.082 DEBUG:boto:Params: {} 2024-04-12T04:03:03.082 DEBUG:boto:Token: None 2024-04-12T04:03:03.083 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj55 2024-04-12T04:03:03.083 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67/AmpiQ22LwMoMqsudHxL77kz0= 2024-04-12T04:03:03.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67/AmpiQ22LwMoMqsudHxL77kz0=', 'Content-Length': '0'} 2024-04-12T04:03:03.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee41dd0d132b5348-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.085 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:03:03.085 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:03:03.085 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.085 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:03:03.085 DEBUG:boto:Data: 2024-04-12T04:03:03.085 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.085 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.085 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.085 DEBUG:boto:Params: {} 2024-04-12T04:03:03.085 DEBUG:boto:Token: None 2024-04-12T04:03:03.085 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj55 2024-04-12T04:03:03.086 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:67/AmpiQ22LwMoMqsudHxL77kz0= 2024-04-12T04:03:03.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:67/AmpiQ22LwMoMqsudHxL77kz0=', 'Content-Length': '0'} 2024-04-12T04:03:03.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081597fa6b03369d9-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:03:03.088 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:03:03.088 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:03:03.088 DEBUG:boto:Method: GET 2024-04-12T04:03:03.088 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:03:03.088 DEBUG:boto:Data: 2024-04-12T04:03:03.088 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.088 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.088 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.088 DEBUG:boto:Params: {} 2024-04-12T04:03:03.089 DEBUG:boto:Token: None 2024-04-12T04:03:03.089 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj55 2024-04-12T04:03:03.089 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BH1PJpXq7byKHfpOaSha/HoOnoc= 2024-04-12T04:03:03.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BH1PJpXq7byKHfpOaSha/HoOnoc=', 'Content-Length': '0'} 2024-04-12T04:03:03.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037fe1099b1f0573f-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.091 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:03:03.091 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:03:03.091 DEBUG:boto:Method: GET 2024-04-12T04:03:03.091 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:03:03.091 DEBUG:boto:Data: 2024-04-12T04:03:03.091 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.091 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.092 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.092 DEBUG:boto:Params: {} 2024-04-12T04:03:03.092 DEBUG:boto:Token: None 2024-04-12T04:03:03.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj55 2024-04-12T04:03:03.092 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BH1PJpXq7byKHfpOaSha/HoOnoc= 2024-04-12T04:03:03.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BH1PJpXq7byKHfpOaSha/HoOnoc=', 'Content-Length': '0'} 2024-04-12T04:03:03.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000542d547b0756b97e-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.094 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.094 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.094 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:03:03.094 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:03:03.095 DEBUG:boto:Method: GET 2024-04-12T04:03:03.095 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:03:03.095 DEBUG:boto:Data: 2024-04-12T04:03:03.095 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.095 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.095 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.095 DEBUG:boto:Params: {} 2024-04-12T04:03:03.095 DEBUG:boto:Token: None 2024-04-12T04:03:03.095 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.095 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88= 2024-04-12T04:03:03.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88=', 'Content-Length': '0'} 2024-04-12T04:03:03.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c600fc6a3d76903a-006618b277-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.138 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:03.139 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.139 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.139 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:03:03.139 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:03:03.139 DEBUG:boto:Method: GET 2024-04-12T04:03:03.139 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:03:03.140 DEBUG:boto:Data: 2024-04-12T04:03:03.140 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.140 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.140 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.140 DEBUG:boto:Params: {} 2024-04-12T04:03:03.140 DEBUG:boto:Token: None 2024-04-12T04:03:03.140 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.140 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88= 2024-04-12T04:03:03.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88=', 'Content-Length': '0'} 2024-04-12T04:03:03.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000151c06e571a1c624-006618b277-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.143 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:03.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:03:03.144 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:03:03.144 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:03:03.144 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.144 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.144 DEBUG:boto:Method: GET 2024-04-12T04:03:03.144 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.145 DEBUG:boto:Data: 2024-04-12T04:03:03.145 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.145 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.145 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.145 DEBUG:boto:Params: {} 2024-04-12T04:03:03.145 DEBUG:boto:Token: None 2024-04-12T04:03:03.145 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.145 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FNW9LLDzyJjix16HdjVt/4f2B5k= 2024-04-12T04:03:03.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FNW9LLDzyJjix16HdjVt/4f2B5k=', 'Content-Length': '0'} 2024-04-12T04:03:03.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c65efed3f3df813f-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.148 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:03:03.148 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:03:03.148 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.148 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.148 DEBUG:boto:Method: GET 2024-04-12T04:03:03.148 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.148 DEBUG:boto:Data: 2024-04-12T04:03:03.148 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.148 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.148 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.148 DEBUG:boto:Params: {} 2024-04-12T04:03:03.149 DEBUG:boto:Token: None 2024-04-12T04:03:03.149 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.149 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FNW9LLDzyJjix16HdjVt/4f2B5k= 2024-04-12T04:03:03.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FNW9LLDzyJjix16HdjVt/4f2B5k=', 'Content-Length': '0'} 2024-04-12T04:03:03.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000617f69a561e5a4f1-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.151 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:03:03.151 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:03:03.151 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.151 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.151 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.152 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.152 DEBUG:boto:Data: 2024-04-12T04:03:03.152 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.152 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.152 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.152 DEBUG:boto:Params: {} 2024-04-12T04:03:03.152 DEBUG:boto:Token: None 2024-04-12T04:03:03.152 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.152 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jKJFjXk0Zw8vWKkOhC2nxE5uMzA= 2024-04-12T04:03:03.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jKJFjXk0Zw8vWKkOhC2nxE5uMzA=', 'Content-Length': '0'} 2024-04-12T04:03:03.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec152f2a1545e46f-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.154 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:03:03.155 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:03:03.155 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.155 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.155 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.155 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.155 DEBUG:boto:Data: 2024-04-12T04:03:03.155 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.155 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.155 DEBUG:boto:Params: {} 2024-04-12T04:03:03.156 DEBUG:boto:Token: None 2024-04-12T04:03:03.156 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jKJFjXk0Zw8vWKkOhC2nxE5uMzA= 2024-04-12T04:03:03.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jKJFjXk0Zw8vWKkOhC2nxE5uMzA=', 'Content-Length': '0'} 2024-04-12T04:03:03.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050eb311fbe20ecd9-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:03:03.158 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:03:03.158 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:03:03.158 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.159 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.159 DEBUG:boto:Method: GET 2024-04-12T04:03:03.159 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.159 DEBUG:boto:Data: 2024-04-12T04:03:03.159 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.159 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.159 DEBUG:boto:Params: {} 2024-04-12T04:03:03.159 DEBUG:boto:Token: None 2024-04-12T04:03:03.159 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.159 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FNW9LLDzyJjix16HdjVt/4f2B5k= 2024-04-12T04:03:03.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FNW9LLDzyJjix16HdjVt/4f2B5k=', 'Content-Length': '0'} 2024-04-12T04:03:03.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037399b02f0fa13cf-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.202 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:03:03.203 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:03:03.203 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.203 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.203 DEBUG:boto:Method: GET 2024-04-12T04:03:03.203 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.204 DEBUG:boto:Data: 2024-04-12T04:03:03.204 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.204 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.204 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.204 DEBUG:boto:Params: {} 2024-04-12T04:03:03.204 DEBUG:boto:Token: None 2024-04-12T04:03:03.204 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:03:03.204 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FNW9LLDzyJjix16HdjVt/4f2B5k= 2024-04-12T04:03:03.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FNW9LLDzyJjix16HdjVt/4f2B5k=', 'Content-Length': '0'} 2024-04-12T04:03:03.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba4233eb37b750d1-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.207 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.207 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.207 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:03:03.207 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:03:03.207 DEBUG:boto:Method: GET 2024-04-12T04:03:03.207 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:03:03.207 DEBUG:boto:Data: 2024-04-12T04:03:03.207 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.207 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.208 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.208 DEBUG:boto:Params: {} 2024-04-12T04:03:03.208 DEBUG:boto:Token: None 2024-04-12T04:03:03.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88= 2024-04-12T04:03:03.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88=', 'Content-Length': '0'} 2024-04-12T04:03:03.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008efff46b501207cc-006618b277-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.211 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:03.211 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.211 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.211 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:03:03.211 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:03:03.211 DEBUG:boto:Method: GET 2024-04-12T04:03:03.211 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:03:03.212 DEBUG:boto:Data: 2024-04-12T04:03:03.212 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.212 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.212 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.212 DEBUG:boto:Params: {} 2024-04-12T04:03:03.212 DEBUG:boto:Token: None 2024-04-12T04:03:03.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88= 2024-04-12T04:03:03.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88=', 'Content-Length': '0'} 2024-04-12T04:03:03.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0e4d862bce4fed9-006618b277-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.215 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:03.216 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:03:03.216 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:03:03.216 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.216 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:03:03.216 DEBUG:boto:Data: 2024-04-12T04:03:03.216 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.216 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.216 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.216 DEBUG:boto:Params: {} 2024-04-12T04:03:03.217 DEBUG:boto:Token: None 2024-04-12T04:03:03.217 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj68 2024-04-12T04:03:03.217 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r49GmT5vWE6NIBAWzMFYnpA7tmk= 2024-04-12T04:03:03.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r49GmT5vWE6NIBAWzMFYnpA7tmk=', 'Content-Length': '0'} 2024-04-12T04:03:03.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0e26b3efe58066e-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.219 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:03:03.219 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:03:03.219 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.219 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:03:03.219 DEBUG:boto:Data: 2024-04-12T04:03:03.219 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.219 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.219 DEBUG:boto:Params: {} 2024-04-12T04:03:03.220 DEBUG:boto:Token: None 2024-04-12T04:03:03.220 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj68 2024-04-12T04:03:03.220 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r49GmT5vWE6NIBAWzMFYnpA7tmk= 2024-04-12T04:03:03.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r49GmT5vWE6NIBAWzMFYnpA7tmk=', 'Content-Length': '0'} 2024-04-12T04:03:03.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000517279409f246848-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:03:03.222 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:03:03.222 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:03:03.222 DEBUG:boto:Method: GET 2024-04-12T04:03:03.222 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:03:03.222 DEBUG:boto:Data: 2024-04-12T04:03:03.222 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.223 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.223 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.223 DEBUG:boto:Params: {} 2024-04-12T04:03:03.223 DEBUG:boto:Token: None 2024-04-12T04:03:03.223 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj68 2024-04-12T04:03:03.223 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vOssCfpoUqEn/I2qTzVGTt7uq5s= 2024-04-12T04:03:03.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vOssCfpoUqEn/I2qTzVGTt7uq5s=', 'Content-Length': '0'} 2024-04-12T04:03:03.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005de4fe47d251bfb8-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.266 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:03:03.266 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:03:03.267 DEBUG:boto:Method: GET 2024-04-12T04:03:03.267 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:03:03.267 DEBUG:boto:Data: 2024-04-12T04:03:03.267 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.267 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.267 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.267 DEBUG:boto:Params: {} 2024-04-12T04:03:03.267 DEBUG:boto:Token: None 2024-04-12T04:03:03.267 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj68 2024-04-12T04:03:03.268 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vOssCfpoUqEn/I2qTzVGTt7uq5s= 2024-04-12T04:03:03.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vOssCfpoUqEn/I2qTzVGTt7uq5s=', 'Content-Length': '0'} 2024-04-12T04:03:03.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e71fb16a071c7b7-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.270 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.270 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.270 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:03:03.270 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:03:03.270 DEBUG:boto:Method: GET 2024-04-12T04:03:03.271 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:03:03.271 DEBUG:boto:Data: 2024-04-12T04:03:03.271 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.271 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.271 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.271 DEBUG:boto:Params: {} 2024-04-12T04:03:03.271 DEBUG:boto:Token: None 2024-04-12T04:03:03.271 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.271 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88= 2024-04-12T04:03:03.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88=', 'Content-Length': '0'} 2024-04-12T04:03:03.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095d652257bff5493-006618b277-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.274 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:03.275 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.275 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.275 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:03:03.275 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:03:03.275 DEBUG:boto:Method: GET 2024-04-12T04:03:03.275 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:03:03.275 DEBUG:boto:Data: 2024-04-12T04:03:03.275 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.275 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.275 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.275 DEBUG:boto:Params: {} 2024-04-12T04:03:03.276 DEBUG:boto:Token: None 2024-04-12T04:03:03.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.276 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88= 2024-04-12T04:03:03.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88=', 'Content-Length': '0'} 2024-04-12T04:03:03.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054dfe5401b096227-006618b277-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.279 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:03.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:03:03.280 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:03:03.280 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:03:03.280 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.280 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.280 DEBUG:boto:Method: GET 2024-04-12T04:03:03.280 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.280 DEBUG:boto:Data: 2024-04-12T04:03:03.280 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.280 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.280 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.280 DEBUG:boto:Params: {} 2024-04-12T04:03:03.281 DEBUG:boto:Token: None 2024-04-12T04:03:03.281 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.281 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:y5tEKrAL/4FzrUyZrybeMvC6I1I= 2024-04-12T04:03:03.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y5tEKrAL/4FzrUyZrybeMvC6I1I=', 'Content-Length': '0'} 2024-04-12T04:03:03.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c81c664c6092155d-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.283 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:03:03.283 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:03:03.283 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.283 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.284 DEBUG:boto:Method: GET 2024-04-12T04:03:03.284 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.284 DEBUG:boto:Data: 2024-04-12T04:03:03.284 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.284 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.284 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.284 DEBUG:boto:Params: {} 2024-04-12T04:03:03.284 DEBUG:boto:Token: None 2024-04-12T04:03:03.284 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.284 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:y5tEKrAL/4FzrUyZrybeMvC6I1I= 2024-04-12T04:03:03.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y5tEKrAL/4FzrUyZrybeMvC6I1I=', 'Content-Length': '0'} 2024-04-12T04:03:03.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d8c35a989fe4392-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.330 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:03:03.331 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:03:03.331 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.331 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.331 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.331 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.331 DEBUG:boto:Data: 2024-04-12T04:03:03.331 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.331 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.331 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.331 DEBUG:boto:Params: {} 2024-04-12T04:03:03.332 DEBUG:boto:Token: None 2024-04-12T04:03:03.332 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.332 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KbhP2KjlsVfhl/e4u3SOSxwcZ4U= 2024-04-12T04:03:03.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KbhP2KjlsVfhl/e4u3SOSxwcZ4U=', 'Content-Length': '0'} 2024-04-12T04:03:03.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd0a996b31957cbc-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.334 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:03:03.334 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:03:03.334 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.334 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.335 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.335 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.335 DEBUG:boto:Data: 2024-04-12T04:03:03.335 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.335 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.335 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.335 DEBUG:boto:Params: {} 2024-04-12T04:03:03.335 DEBUG:boto:Token: None 2024-04-12T04:03:03.335 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.335 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KbhP2KjlsVfhl/e4u3SOSxwcZ4U= 2024-04-12T04:03:03.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KbhP2KjlsVfhl/e4u3SOSxwcZ4U=', 'Content-Length': '0'} 2024-04-12T04:03:03.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085da4079e0172475-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:03:03.338 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:03:03.338 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:03:03.338 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.338 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.338 DEBUG:boto:Method: GET 2024-04-12T04:03:03.338 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.338 DEBUG:boto:Data: 2024-04-12T04:03:03.339 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.339 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.339 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.339 DEBUG:boto:Params: {} 2024-04-12T04:03:03.339 DEBUG:boto:Token: None 2024-04-12T04:03:03.339 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.339 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:y5tEKrAL/4FzrUyZrybeMvC6I1I= 2024-04-12T04:03:03.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y5tEKrAL/4FzrUyZrybeMvC6I1I=', 'Content-Length': '0'} 2024-04-12T04:03:03.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059bd758f63edcca0-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.341 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:03:03.341 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:03:03.341 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.342 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.342 DEBUG:boto:Method: GET 2024-04-12T04:03:03.342 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.342 DEBUG:boto:Data: 2024-04-12T04:03:03.342 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.342 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.342 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.342 DEBUG:boto:Params: {} 2024-04-12T04:03:03.342 DEBUG:boto:Token: None 2024-04-12T04:03:03.342 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:03:03.343 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:y5tEKrAL/4FzrUyZrybeMvC6I1I= 2024-04-12T04:03:03.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y5tEKrAL/4FzrUyZrybeMvC6I1I=', 'Content-Length': '0'} 2024-04-12T04:03:03.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b8752d874848308-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.345 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.345 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.345 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:03:03.345 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:03:03.345 DEBUG:boto:Method: GET 2024-04-12T04:03:03.345 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:03:03.345 DEBUG:boto:Data: 2024-04-12T04:03:03.345 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.345 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.345 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.346 DEBUG:boto:Params: {} 2024-04-12T04:03:03.346 DEBUG:boto:Token: None 2024-04-12T04:03:03.346 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.346 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88= 2024-04-12T04:03:03.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88=', 'Content-Length': '0'} 2024-04-12T04:03:03.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018a38f1fd1851a2c-006618b277-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.349 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:03.349 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.349 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.349 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:03:03.350 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:03:03.350 DEBUG:boto:Method: GET 2024-04-12T04:03:03.350 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:03:03.350 DEBUG:boto:Data: 2024-04-12T04:03:03.350 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.350 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.350 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.350 DEBUG:boto:Params: {} 2024-04-12T04:03:03.350 DEBUG:boto:Token: None 2024-04-12T04:03:03.350 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.351 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88= 2024-04-12T04:03:03.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88=', 'Content-Length': '0'} 2024-04-12T04:03:03.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0c9e2d0c96ef3e2-006618b277-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.354 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:03.354 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:03:03.354 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:03:03.355 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.355 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:03:03.355 DEBUG:boto:Data: 2024-04-12T04:03:03.355 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.355 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.355 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.355 DEBUG:boto:Params: {} 2024-04-12T04:03:03.355 DEBUG:boto:Token: None 2024-04-12T04:03:03.355 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj85 2024-04-12T04:03:03.355 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UaTnC4ZzVKsFFZnZz0dMMtKN8Bc= 2024-04-12T04:03:03.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UaTnC4ZzVKsFFZnZz0dMMtKN8Bc=', 'Content-Length': '0'} 2024-04-12T04:03:03.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000188150f741d068d4-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.357 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:03:03.358 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:03:03.358 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.358 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:03:03.358 DEBUG:boto:Data: 2024-04-12T04:03:03.358 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.358 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.358 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.358 DEBUG:boto:Params: {} 2024-04-12T04:03:03.358 DEBUG:boto:Token: None 2024-04-12T04:03:03.358 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj85 2024-04-12T04:03:03.359 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UaTnC4ZzVKsFFZnZz0dMMtKN8Bc= 2024-04-12T04:03:03.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UaTnC4ZzVKsFFZnZz0dMMtKN8Bc=', 'Content-Length': '0'} 2024-04-12T04:03:03.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bfeed77eb355aa4-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:03:03.361 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:03:03.361 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:03:03.361 DEBUG:boto:Method: GET 2024-04-12T04:03:03.361 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:03:03.361 DEBUG:boto:Data: 2024-04-12T04:03:03.361 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.361 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.361 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.361 DEBUG:boto:Params: {} 2024-04-12T04:03:03.362 DEBUG:boto:Token: None 2024-04-12T04:03:03.362 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj85 2024-04-12T04:03:03.362 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:clGBnH1iaDNVvHPJWSkJUPmx19c= 2024-04-12T04:03:03.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:clGBnH1iaDNVvHPJWSkJUPmx19c=', 'Content-Length': '0'} 2024-04-12T04:03:03.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ef7927c43867f50-006618b277-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.364 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:03:03.364 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:03:03.364 DEBUG:boto:Method: GET 2024-04-12T04:03:03.364 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:03:03.364 DEBUG:boto:Data: 2024-04-12T04:03:03.364 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.364 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.364 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.365 DEBUG:boto:Params: {} 2024-04-12T04:03:03.365 DEBUG:boto:Token: None 2024-04-12T04:03:03.365 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj85 2024-04-12T04:03:03.365 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:clGBnH1iaDNVvHPJWSkJUPmx19c= 2024-04-12T04:03:03.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:clGBnH1iaDNVvHPJWSkJUPmx19c=', 'Content-Length': '0'} 2024-04-12T04:03:03.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008057ce3107e6fc5a-006618b277-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.367 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.367 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.367 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:03:03.367 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:03:03.367 DEBUG:boto:Method: GET 2024-04-12T04:03:03.368 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:03:03.368 DEBUG:boto:Data: 2024-04-12T04:03:03.368 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.368 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:03:03.368 DEBUG:boto:Port: 8001 2024-04-12T04:03:03.368 DEBUG:boto:Params: {} 2024-04-12T04:03:03.368 DEBUG:boto:Token: None 2024-04-12T04:03:03.368 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.368 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88= 2024-04-12T04:03:03.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88=', 'Content-Length': '0'} 2024-04-12T04:03:03.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097c4752966c081b6-006618b277-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.371 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:03.371 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.371 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.371 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:03:03.371 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:03:03.371 DEBUG:boto:Method: GET 2024-04-12T04:03:03.372 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:03:03.372 DEBUG:boto:Data: 2024-04-12T04:03:03.372 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.372 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.372 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.372 DEBUG:boto:Params: {} 2024-04-12T04:03:03.372 DEBUG:boto:Token: None 2024-04-12T04:03:03.372 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.372 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88= 2024-04-12T04:03:03.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aOYbTOfNKHdF+vk8+Pwktd+YT88=', 'Content-Length': '0'} 2024-04-12T04:03:03.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aab2deb8dfafc8c2-006618b277-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.377 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:03:03.378 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-61 zones={a2, a1} 2024-04-12T04:03:03.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:03:03.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:03:03.379 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:03:03.442 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:03:03.442 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:03:03.496 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:03:03.496 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-61 2024-04-12T04:03:03.503 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 10 2024-04-12T04:03:03.558 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.559 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.559 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.559 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:03:03.559 DEBUG:boto:Data: 2024-04-12T04:03:03.559 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.559 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.559 DEBUG:boto:Params: {} 2024-04-12T04:03:03.560 DEBUG:boto:Token: None 2024-04-12T04:03:03.560 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.560 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QlRwk2MAAMt+PYhEuTm0bM0tzhE= 2024-04-12T04:03:03.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QlRwk2MAAMt+PYhEuTm0bM0tzhE=', 'Content-Length': '0'} 2024-04-12T04:03:03.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008bba9354dd557070-006618b277-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.563 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:03:03.564 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:03:03.564 DEBUG:boto:Method: PUT 2024-04-12T04:03:03.564 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:03:03.564 DEBUG:boto:Data: 2024-04-12T04:03:03.564 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:03:03.564 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.564 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.564 DEBUG:boto:Params: {} 2024-04-12T04:03:03.564 DEBUG:boto:Token: None 2024-04-12T04:03:03.564 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj19 2024-04-12T04:03:03.564 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Yi3U5+UkmZ0PXxlSvbqe09Sgd00= 2024-04-12T04:03:03.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Yi3U5+UkmZ0PXxlSvbqe09Sgd00='} 2024-04-12T04:03:03.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0467573e27abf79-006618b277-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.607 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.607 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.607 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.607 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:03:03.607 DEBUG:boto:Data: 2024-04-12T04:03:03.607 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.607 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.607 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.608 DEBUG:boto:Params: {} 2024-04-12T04:03:03.608 DEBUG:boto:Token: None 2024-04-12T04:03:03.608 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.608 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QlRwk2MAAMt+PYhEuTm0bM0tzhE= 2024-04-12T04:03:03.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QlRwk2MAAMt+PYhEuTm0bM0tzhE=', 'Content-Length': '0'} 2024-04-12T04:03:03.611 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007475886bceca72eb-006618b277-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.611 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:03:03.611 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:03:03.611 DEBUG:boto:Method: PUT 2024-04-12T04:03:03.611 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:03:03.611 DEBUG:boto:Data: 2024-04-12T04:03:03.611 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:03:03.612 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.612 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.612 DEBUG:boto:Params: {} 2024-04-12T04:03:03.612 DEBUG:boto:Token: None 2024-04-12T04:03:03.612 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj38 2024-04-12T04:03:03.612 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yqINATcURYf4ACx0Yhelre1b+zg= 2024-04-12T04:03:03.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yqINATcURYf4ACx0Yhelre1b+zg='} 2024-04-12T04:03:03.616 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002da1583ec0cc2faa-006618b277-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.617 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.617 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.617 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.617 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:03:03.617 DEBUG:boto:Data: 2024-04-12T04:03:03.617 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.617 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.617 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.617 DEBUG:boto:Params: {} 2024-04-12T04:03:03.617 DEBUG:boto:Token: None 2024-04-12T04:03:03.618 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.618 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QlRwk2MAAMt+PYhEuTm0bM0tzhE= 2024-04-12T04:03:03.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QlRwk2MAAMt+PYhEuTm0bM0tzhE=', 'Content-Length': '0'} 2024-04-12T04:03:03.621 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008dd0bf005016d4a7-006618b277-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.621 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:03:03.621 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:03:03.621 DEBUG:boto:Method: PUT 2024-04-12T04:03:03.621 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:03:03.621 DEBUG:boto:Data: 2024-04-12T04:03:03.621 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:03:03.621 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.621 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.622 DEBUG:boto:Params: {} 2024-04-12T04:03:03.622 DEBUG:boto:Token: None 2024-04-12T04:03:03.622 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj57 2024-04-12T04:03:03.622 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7uhh3LlQMN8hc2VX76jGwrXikl8= 2024-04-12T04:03:03.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7uhh3LlQMN8hc2VX76jGwrXikl8='} 2024-04-12T04:03:03.666 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe8b8096f72447e1-006618b277-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.667 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.667 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.667 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.667 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:03:03.667 DEBUG:boto:Data: 2024-04-12T04:03:03.667 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.667 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.667 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.667 DEBUG:boto:Params: {} 2024-04-12T04:03:03.668 DEBUG:boto:Token: None 2024-04-12T04:03:03.668 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.668 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QlRwk2MAAMt+PYhEuTm0bM0tzhE= 2024-04-12T04:03:03.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QlRwk2MAAMt+PYhEuTm0bM0tzhE=', 'Content-Length': '0'} 2024-04-12T04:03:03.671 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007eedafff51c668cb-006618b277-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.671 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:03:03.671 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:03:03.671 DEBUG:boto:Method: PUT 2024-04-12T04:03:03.672 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:03:03.672 DEBUG:boto:Data: 2024-04-12T04:03:03.672 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:03:03.672 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.672 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.672 DEBUG:boto:Params: {} 2024-04-12T04:03:03.672 DEBUG:boto:Token: None 2024-04-12T04:03:03.672 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj76 2024-04-12T04:03:03.672 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:p8VrBUoGf1nYnfQO2jz4ZIR23IM= 2024-04-12T04:03:03.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p8VrBUoGf1nYnfQO2jz4ZIR23IM='} 2024-04-12T04:03:03.679 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e2e47f42833ad41-006618b277-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.679 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:03:03.679 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:03:03.679 DEBUG:boto:Method: HEAD 2024-04-12T04:03:03.680 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:03:03.680 DEBUG:boto:Data: 2024-04-12T04:03:03.680 DEBUG:boto:Headers: {} 2024-04-12T04:03:03.680 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.680 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.680 DEBUG:boto:Params: {} 2024-04-12T04:03:03.680 DEBUG:boto:Token: None 2024-04-12T04:03:03.680 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/ 2024-04-12T04:03:03.680 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QlRwk2MAAMt+PYhEuTm0bM0tzhE= 2024-04-12T04:03:03.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QlRwk2MAAMt+PYhEuTm0bM0tzhE=', 'Content-Length': '0'} 2024-04-12T04:03:03.683 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d5ad54e176a4af5-006618b277-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.684 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:03:03.684 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:03:03.684 DEBUG:boto:Method: PUT 2024-04-12T04:03:03.684 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:03:03.684 DEBUG:boto:Data: 2024-04-12T04:03:03.684 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:03:03.684 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:03:03.684 DEBUG:boto:Port: 8000 2024-04-12T04:03:03.684 DEBUG:boto:Params: {} 2024-04-12T04:03:03.685 DEBUG:boto:Token: None 2024-04-12T04:03:03.685 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:03:03 GMT /dkyges-61/obj95 2024-04-12T04:03:03.685 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XfANpiO0uQits8m8R7JQ1wMSb2o= 2024-04-12T04:03:03.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:03:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XfANpiO0uQits8m8R7JQ1wMSb2o='} 2024-04-12T04:03:03.730 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c2f2f0462b309eb0-006618b277-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:03:03.731 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-61 2024-04-12T04:03:03.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:03:03.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:03:03.731 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:03:03.799 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:03:03.799 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:03:03.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:03:03.856+0000 7f84afc23780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T04:03:08.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:03:08.857+0000 7f84afc23780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T04:03:13.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:03:13.858+0000 7f84afc23780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T04:03:18.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:03:18.858+0000 7f84afc23780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T04:03:23.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:03:23.860+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:23.850 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-04-12T04:03:23.850 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:03:28.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:03:28.862+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:28.849 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-04-12T04:03:28.849 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:03:33.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:03:33.864+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:33.850 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-04-12T04:03:33.850 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:03:38.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:03:38.866+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:38.852 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2024-04-12T04:03:38.852 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:03:43.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:03:43.867+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:43.853 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.76.6, , , , , , , , , , , , ] 2024-04-12T04:03:43.854 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:03:48.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:03:48.869+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:48.855 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.76.6, , , , , , , , , , , , ] 2024-04-12T04:03:48.855 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:03:53.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:03:53.871+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:53.857 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.76.6, 00000000002.81.6, , , , , , , , , , , ] 2024-04-12T04:03:53.857 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:03:58.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:03:58.873+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:03:58.859 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.76.6, 00000000002.81.6, , , , , , , , , , , ] 2024-04-12T04:03:58.859 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:04:03.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:04:03.874+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:03.860 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.76.6, 00000000002.81.6, , , , , , , , , , , ] 2024-04-12T04:04:03.861 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:04:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:04:08.877+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:08.863 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.76.6, 00000000002.81.6, , , , , , , , , , , ] 2024-04-12T04:04:08.863 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:04:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:04:13.878+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:13.865 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.76.6, 00000000002.81.6, , , , , , , , , , , ] 2024-04-12T04:04:13.865 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:04:18.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:04:18.879+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:18.866 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.76.6, 00000000002.81.6, , , , , , , , , , , ] 2024-04-12T04:04:18.866 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:04:23.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:04:23.881+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:23.867 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.76.6, 00000000002.81.6, , , , , , , , , , , ] 2024-04-12T04:04:23.867 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:04:28.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:04:28.883+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:28.869 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.76.6, 00000000002.81.6, , , , , , , , , , , ] 2024-04-12T04:04:28.869 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:04:33.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:04:33.884+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:33.870 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.76.6, 00000000002.81.6, , , , , , , , , , , ] 2024-04-12T04:04:33.870 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:04:38.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:04:38.886+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:38.872 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.76.6, 00000000002.81.6, , , , , , , , , , , ] 2024-04-12T04:04:38.872 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:04:43.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:04:43.888+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:43.874 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.76.6, 00000000002.81.6, , , , , , , , , , , ] 2024-04-12T04:04:43.874 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:04:48.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:04:48.890+0000 7f84afc23780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:48.876 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.76.6, 00000000002.81.6, , , , , , , , , , , ] 2024-04-12T04:04:48.876 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:04:53.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:04:53.891+0000 7f84afc23780 1 bucket sync caught up with source: 2024-04-12T04:04:53.877 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:04:53.877 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.76.6, 00000000004.85.6, , , , , , , , , , , ] 2024-04-12T04:04:53.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:04:53.891+0000 7f84afc23780 0 bucket checkpoint complete 2024-04-12T04:04:53.896 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-61 2024-04-12T04:04:53.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:04:53.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:04:53.897 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:04:53.964 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:04:53.964 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:04:54.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:04:54.020+0000 7fb24619e780 1 waiting for incremental sync to catch up: 2024-04-12T04:04:54.006 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.2368.6, 00000000001.1403.6, , 00000000001.2366.6, 00000000001.2593.6, , , 00000000002.2561.6, 00000000004.2282.6, 00000000003.2610.6] 2024-04-12T04:04:54.006 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.2368.6, 00000000001.1403.6, 00000000001.2992.6, 00000000001.2366.6, 00000000001.2593.6, , , 00000000002.2561.6, 00000000004.2282.6, 00000000003.2610.6] 2024-04-12T04:04:59.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:04:59.022+0000 7fb24619e780 1 bucket sync caught up with source: 2024-04-12T04:04:59.008 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.2368.6, 00000000001.1403.6, 00000000001.2992.6, 00000000001.2366.6, 00000000001.2593.6, , , 00000000002.2561.6, 00000000004.2282.6, 00000000003.2610.6] 2024-04-12T04:04:59.008 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.2368.6, 00000000001.1403.6, 00000000001.2992.6, 00000000001.2366.6, 00000000001.2593.6, , , 00000000002.2561.6, 00000000004.2282.6, 00000000003.2610.6] 2024-04-12T04:04:59.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:04:59.022+0000 7fb24619e780 0 bucket checkpoint complete 2024-04-12T04:04:59.026 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-61 zones={a2, a1} 2024-04-12T04:04:59.026 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.027 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.027 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.027 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:04:59.027 DEBUG:boto:Data: 2024-04-12T04:04:59.027 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.027 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.027 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.027 DEBUG:boto:Params: {} 2024-04-12T04:04:59.028 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:04:59.028 DEBUG:boto:Token: None 2024-04-12T04:04:59.028 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.029 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:s5r3g4rDmi0sjk6eZ/ddRptWqEk= 2024-04-12T04:04:59.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:s5r3g4rDmi0sjk6eZ/ddRptWqEk=', 'Content-Length': '0'} 2024-04-12T04:04:59.044 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085deaffea6706fec-006618b2eb-1cfd-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.044 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.044 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.044 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.044 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:04:59.044 DEBUG:boto:Data: 2024-04-12T04:04:59.044 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.044 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.045 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.045 DEBUG:boto:Params: {} 2024-04-12T04:04:59.045 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:04:59.045 DEBUG:boto:Token: None 2024-04-12T04:04:59.045 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.045 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:s5r3g4rDmi0sjk6eZ/ddRptWqEk= 2024-04-12T04:04:59.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:s5r3g4rDmi0sjk6eZ/ddRptWqEk=', 'Content-Length': '0'} 2024-04-12T04:04:59.075 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000070b02c9c39c274ec-006618b2eb-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.076 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:04:59.076 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.076 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.076 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:04:59.076 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:04:59.076 DEBUG:boto:Method: GET 2024-04-12T04:04:59.076 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:04:59.076 DEBUG:boto:Data: 2024-04-12T04:04:59.077 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.077 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.077 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.077 DEBUG:boto:Params: {} 2024-04-12T04:04:59.077 DEBUG:boto:Token: None 2024-04-12T04:04:59.077 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/?versions 2024-04-12T04:04:59.077 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ez/Wrn6HVCcKhPH3kccbRq6PQ14= 2024-04-12T04:04:59.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ez/Wrn6HVCcKhPH3kccbRq6PQ14=', 'Content-Length': '0'} 2024-04-12T04:04:59.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6f20f12ffb28175-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.080 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:04:59.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:04:59.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T04:04:59.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:04:59.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:04:59.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:04:59.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T04:04:59.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:04:59.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:04:59.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:04:59.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T04:04:59.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:04:59.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T04:04:59.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:04:59.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T04:04:59.083 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:04:59.083 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.083 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.083 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:04:59.084 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:04:59.084 DEBUG:boto:Method: GET 2024-04-12T04:04:59.084 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:04:59.084 DEBUG:boto:Data: 2024-04-12T04:04:59.084 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.084 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.084 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.084 DEBUG:boto:Params: {} 2024-04-12T04:04:59.084 DEBUG:boto:Token: None 2024-04-12T04:04:59.085 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/?versions 2024-04-12T04:04:59.085 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ez/Wrn6HVCcKhPH3kccbRq6PQ14= 2024-04-12T04:04:59.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ez/Wrn6HVCcKhPH3kccbRq6PQ14=', 'Content-Length': '0'} 2024-04-12T04:04:59.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdb77afb3afb0efd-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.088 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:04:59.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:04:59.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T04:04:59.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:04:59.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:04:59.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:04:59.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T04:04:59.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:04:59.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:04:59.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:04:59.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T04:04:59.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:04:59.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T04:04:59.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:04:59.090 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T04:04:59.090 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.090 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.091 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:04:59.091 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:04:59.091 DEBUG:boto:Method: GET 2024-04-12T04:04:59.091 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:04:59.091 DEBUG:boto:Data: 2024-04-12T04:04:59.091 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.091 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.091 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.091 DEBUG:boto:Params: {} 2024-04-12T04:04:59.091 DEBUG:boto:Token: None 2024-04-12T04:04:59.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/?versions 2024-04-12T04:04:59.092 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ez/Wrn6HVCcKhPH3kccbRq6PQ14= 2024-04-12T04:04:59.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ez/Wrn6HVCcKhPH3kccbRq6PQ14=', 'Content-Length': '0'} 2024-04-12T04:04:59.094 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092a01caad0f62c5f-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.094 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.095 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.096 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.096 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:04:59.096 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:04:59.096 DEBUG:boto:Method: GET 2024-04-12T04:04:59.096 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:04:59.096 DEBUG:boto:Data: 2024-04-12T04:04:59.096 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.096 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.096 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.096 DEBUG:boto:Params: {} 2024-04-12T04:04:59.097 DEBUG:boto:Token: None 2024-04-12T04:04:59.097 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/?versions 2024-04-12T04:04:59.097 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ez/Wrn6HVCcKhPH3kccbRq6PQ14= 2024-04-12T04:04:59.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ez/Wrn6HVCcKhPH3kccbRq6PQ14=', 'Content-Length': '0'} 2024-04-12T04:04:59.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb1bb0e141bfff1-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.100 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:04:59.101 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:04:59.101 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:04:59.101 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.101 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.101 DEBUG:boto:Method: GET 2024-04-12T04:04:59.102 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.102 DEBUG:boto:Data: 2024-04-12T04:04:59.102 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.102 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.102 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.102 DEBUG:boto:Params: {} 2024-04-12T04:04:59.102 DEBUG:boto:Token: None 2024-04-12T04:04:59.102 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.102 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pLTprKkU/7NJNa10AgGrullsByM= 2024-04-12T04:04:59.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pLTprKkU/7NJNa10AgGrullsByM=', 'Content-Length': '0'} 2024-04-12T04:04:59.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081213d5bd603def1-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.146 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:04:59.147 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:04:59.147 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.147 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.147 DEBUG:boto:Method: GET 2024-04-12T04:04:59.147 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.147 DEBUG:boto:Data: 2024-04-12T04:04:59.147 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.147 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.147 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.147 DEBUG:boto:Params: {} 2024-04-12T04:04:59.148 DEBUG:boto:Token: None 2024-04-12T04:04:59.148 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.148 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pLTprKkU/7NJNa10AgGrullsByM= 2024-04-12T04:04:59.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pLTprKkU/7NJNa10AgGrullsByM=', 'Content-Length': '0'} 2024-04-12T04:04:59.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7c67c9a3125dddb-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.150 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:04:59.151 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:04:59.151 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.151 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.151 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.151 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.151 DEBUG:boto:Data: 2024-04-12T04:04:59.151 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.151 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.151 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.151 DEBUG:boto:Params: {} 2024-04-12T04:04:59.151 DEBUG:boto:Token: None 2024-04-12T04:04:59.152 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.152 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BnAwqOkTiDvbQNYSFWijkkom4h4= 2024-04-12T04:04:59.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BnAwqOkTiDvbQNYSFWijkkom4h4=', 'Content-Length': '0'} 2024-04-12T04:04:59.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b44040fefbe1cb6-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.154 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:04:59.154 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:04:59.154 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.154 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.154 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.154 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.154 DEBUG:boto:Data: 2024-04-12T04:04:59.154 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.155 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.155 DEBUG:boto:Params: {} 2024-04-12T04:04:59.155 DEBUG:boto:Token: None 2024-04-12T04:04:59.155 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.155 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BnAwqOkTiDvbQNYSFWijkkom4h4= 2024-04-12T04:04:59.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BnAwqOkTiDvbQNYSFWijkkom4h4=', 'Content-Length': '0'} 2024-04-12T04:04:59.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dde802a15e6cdb4a-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:04:59.157 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:04:59.157 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:04:59.157 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.157 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.157 DEBUG:boto:Method: GET 2024-04-12T04:04:59.158 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.158 DEBUG:boto:Data: 2024-04-12T04:04:59.158 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.158 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.158 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.158 DEBUG:boto:Params: {} 2024-04-12T04:04:59.158 DEBUG:boto:Token: None 2024-04-12T04:04:59.158 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.158 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pLTprKkU/7NJNa10AgGrullsByM= 2024-04-12T04:04:59.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pLTprKkU/7NJNa10AgGrullsByM=', 'Content-Length': '0'} 2024-04-12T04:04:59.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d7ab17fee4c80f2-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.160 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:04:59.160 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:04:59.160 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.161 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.161 DEBUG:boto:Method: GET 2024-04-12T04:04:59.161 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.161 DEBUG:boto:Data: 2024-04-12T04:04:59.161 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.161 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.161 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.161 DEBUG:boto:Params: {} 2024-04-12T04:04:59.161 DEBUG:boto:Token: None 2024-04-12T04:04:59.161 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:04:59.162 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pLTprKkU/7NJNa10AgGrullsByM= 2024-04-12T04:04:59.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pLTprKkU/7NJNa10AgGrullsByM=', 'Content-Length': '0'} 2024-04-12T04:04:59.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c636f8526cf95b27-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.164 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.164 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.164 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:04:59.164 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:04:59.164 DEBUG:boto:Method: GET 2024-04-12T04:04:59.164 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:04:59.164 DEBUG:boto:Data: 2024-04-12T04:04:59.164 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.164 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.165 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.165 DEBUG:boto:Params: {} 2024-04-12T04:04:59.165 DEBUG:boto:Token: None 2024-04-12T04:04:59.165 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.165 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008e650866dc86528-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.168 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.168 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.168 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.169 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:04:59.169 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:04:59.169 DEBUG:boto:Method: GET 2024-04-12T04:04:59.169 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:04:59.169 DEBUG:boto:Data: 2024-04-12T04:04:59.169 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.169 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.169 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.169 DEBUG:boto:Params: {} 2024-04-12T04:04:59.169 DEBUG:boto:Token: None 2024-04-12T04:04:59.170 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.170 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fb67e0fed44ff94-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.214 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.215 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:04:59.215 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:04:59.215 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.215 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:04:59.215 DEBUG:boto:Data: 2024-04-12T04:04:59.215 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.215 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.215 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.215 DEBUG:boto:Params: {} 2024-04-12T04:04:59.216 DEBUG:boto:Token: None 2024-04-12T04:04:59.216 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj11 2024-04-12T04:04:59.216 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iyWNwKMuLWjUH6PP/ydFS7Lw4N0= 2024-04-12T04:04:59.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iyWNwKMuLWjUH6PP/ydFS7Lw4N0=', 'Content-Length': '0'} 2024-04-12T04:04:59.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f83fcb9967a437a5-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.218 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:04:59.218 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:04:59.218 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.219 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:04:59.219 DEBUG:boto:Data: 2024-04-12T04:04:59.219 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.219 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.219 DEBUG:boto:Params: {} 2024-04-12T04:04:59.219 DEBUG:boto:Token: None 2024-04-12T04:04:59.219 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj11 2024-04-12T04:04:59.220 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iyWNwKMuLWjUH6PP/ydFS7Lw4N0= 2024-04-12T04:04:59.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iyWNwKMuLWjUH6PP/ydFS7Lw4N0=', 'Content-Length': '0'} 2024-04-12T04:04:59.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014f443d00d4f6144-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:04:59.222 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:04:59.222 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:04:59.222 DEBUG:boto:Method: GET 2024-04-12T04:04:59.222 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:04:59.222 DEBUG:boto:Data: 2024-04-12T04:04:59.222 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.222 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.222 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.222 DEBUG:boto:Params: {} 2024-04-12T04:04:59.223 DEBUG:boto:Token: None 2024-04-12T04:04:59.223 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj11 2024-04-12T04:04:59.223 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BbcWdx9N8P1z6DUDW0ZD2ioDN6U= 2024-04-12T04:04:59.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BbcWdx9N8P1z6DUDW0ZD2ioDN6U=', 'Content-Length': '0'} 2024-04-12T04:04:59.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff59ee2ad4e2b17d-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.225 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:04:59.225 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:04:59.225 DEBUG:boto:Method: GET 2024-04-12T04:04:59.225 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:04:59.225 DEBUG:boto:Data: 2024-04-12T04:04:59.225 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.226 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.226 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.226 DEBUG:boto:Params: {} 2024-04-12T04:04:59.226 DEBUG:boto:Token: None 2024-04-12T04:04:59.226 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj11 2024-04-12T04:04:59.226 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BbcWdx9N8P1z6DUDW0ZD2ioDN6U= 2024-04-12T04:04:59.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BbcWdx9N8P1z6DUDW0ZD2ioDN6U=', 'Content-Length': '0'} 2024-04-12T04:04:59.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d28b8bb544d7ab6b-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.228 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.228 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.228 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:04:59.228 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:04:59.228 DEBUG:boto:Method: GET 2024-04-12T04:04:59.228 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:04:59.229 DEBUG:boto:Data: 2024-04-12T04:04:59.229 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.229 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.229 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.229 DEBUG:boto:Params: {} 2024-04-12T04:04:59.229 DEBUG:boto:Token: None 2024-04-12T04:04:59.229 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.229 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd876bb795482161-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.232 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.232 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.233 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.233 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:04:59.233 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:04:59.233 DEBUG:boto:Method: GET 2024-04-12T04:04:59.233 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:04:59.233 DEBUG:boto:Data: 2024-04-12T04:04:59.233 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.233 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.233 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.233 DEBUG:boto:Params: {} 2024-04-12T04:04:59.233 DEBUG:boto:Token: None 2024-04-12T04:04:59.234 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.234 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000650c7c469935afc5-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.237 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:04:59.238 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:04:59.238 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:04:59.238 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.238 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.238 DEBUG:boto:Method: GET 2024-04-12T04:04:59.238 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.238 DEBUG:boto:Data: 2024-04-12T04:04:59.238 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.238 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.239 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.239 DEBUG:boto:Params: {} 2024-04-12T04:04:59.239 DEBUG:boto:Token: None 2024-04-12T04:04:59.239 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.239 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tcvgT7UAlPs/Lno6HwyqOnB9tcg= 2024-04-12T04:04:59.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tcvgT7UAlPs/Lno6HwyqOnB9tcg=', 'Content-Length': '0'} 2024-04-12T04:04:59.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063d75a780f8d3d57-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.241 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:04:59.241 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:04:59.241 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.241 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.241 DEBUG:boto:Method: GET 2024-04-12T04:04:59.242 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.242 DEBUG:boto:Data: 2024-04-12T04:04:59.242 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.242 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.242 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.242 DEBUG:boto:Params: {} 2024-04-12T04:04:59.242 DEBUG:boto:Token: None 2024-04-12T04:04:59.242 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.243 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tcvgT7UAlPs/Lno6HwyqOnB9tcg= 2024-04-12T04:04:59.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tcvgT7UAlPs/Lno6HwyqOnB9tcg=', 'Content-Length': '0'} 2024-04-12T04:04:59.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bf4cbb2f90131b8-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.245 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:04:59.245 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:04:59.245 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.245 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.245 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.246 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.246 DEBUG:boto:Data: 2024-04-12T04:04:59.246 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.246 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.246 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.246 DEBUG:boto:Params: {} 2024-04-12T04:04:59.246 DEBUG:boto:Token: None 2024-04-12T04:04:59.246 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.246 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Igw4GfLdETkEtpa0fJOAiEOFX+g= 2024-04-12T04:04:59.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Igw4GfLdETkEtpa0fJOAiEOFX+g=', 'Content-Length': '0'} 2024-04-12T04:04:59.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c12bf9f5d7a23818-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.249 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:04:59.249 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:04:59.249 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.249 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.249 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.249 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.249 DEBUG:boto:Data: 2024-04-12T04:04:59.249 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.249 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.249 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.249 DEBUG:boto:Params: {} 2024-04-12T04:04:59.249 DEBUG:boto:Token: None 2024-04-12T04:04:59.250 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.250 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Igw4GfLdETkEtpa0fJOAiEOFX+g= 2024-04-12T04:04:59.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Igw4GfLdETkEtpa0fJOAiEOFX+g=', 'Content-Length': '0'} 2024-04-12T04:04:59.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6aa13fbd5668366-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:04:59.252 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:04:59.252 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:04:59.252 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.252 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.252 DEBUG:boto:Method: GET 2024-04-12T04:04:59.252 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.252 DEBUG:boto:Data: 2024-04-12T04:04:59.253 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.253 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.253 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.253 DEBUG:boto:Params: {} 2024-04-12T04:04:59.253 DEBUG:boto:Token: None 2024-04-12T04:04:59.253 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.253 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tcvgT7UAlPs/Lno6HwyqOnB9tcg= 2024-04-12T04:04:59.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tcvgT7UAlPs/Lno6HwyqOnB9tcg=', 'Content-Length': '0'} 2024-04-12T04:04:59.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000387604eab6a26a24-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.255 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:04:59.255 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:04:59.255 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.256 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.256 DEBUG:boto:Method: GET 2024-04-12T04:04:59.256 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.256 DEBUG:boto:Data: 2024-04-12T04:04:59.256 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.256 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.256 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.256 DEBUG:boto:Params: {} 2024-04-12T04:04:59.256 DEBUG:boto:Token: None 2024-04-12T04:04:59.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:04:59.257 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tcvgT7UAlPs/Lno6HwyqOnB9tcg= 2024-04-12T04:04:59.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tcvgT7UAlPs/Lno6HwyqOnB9tcg=', 'Content-Length': '0'} 2024-04-12T04:04:59.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000109e4635cbaf054-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.259 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.259 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.259 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:04:59.259 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:04:59.259 DEBUG:boto:Method: GET 2024-04-12T04:04:59.259 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:04:59.259 DEBUG:boto:Data: 2024-04-12T04:04:59.259 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.259 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.259 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.260 DEBUG:boto:Params: {} 2024-04-12T04:04:59.260 DEBUG:boto:Token: None 2024-04-12T04:04:59.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.260 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee3cf40d8e853ae8-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.306 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.307 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.307 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.307 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:04:59.307 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:04:59.307 DEBUG:boto:Method: GET 2024-04-12T04:04:59.307 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:04:59.307 DEBUG:boto:Data: 2024-04-12T04:04:59.307 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.307 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.307 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.307 DEBUG:boto:Params: {} 2024-04-12T04:04:59.308 DEBUG:boto:Token: None 2024-04-12T04:04:59.308 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.308 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9247685cd33f1e5-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.311 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.312 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:04:59.312 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:04:59.312 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.312 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:04:59.312 DEBUG:boto:Data: 2024-04-12T04:04:59.312 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.312 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.312 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.313 DEBUG:boto:Params: {} 2024-04-12T04:04:59.313 DEBUG:boto:Token: None 2024-04-12T04:04:59.313 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj17 2024-04-12T04:04:59.313 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e16rCLubnl8amV/4GsBlrVAwtNE= 2024-04-12T04:04:59.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e16rCLubnl8amV/4GsBlrVAwtNE=', 'Content-Length': '0'} 2024-04-12T04:04:59.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049ab00c94cd66742-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.315 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:04:59.315 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:04:59.315 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.315 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:04:59.315 DEBUG:boto:Data: 2024-04-12T04:04:59.315 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.316 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.316 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.316 DEBUG:boto:Params: {} 2024-04-12T04:04:59.316 DEBUG:boto:Token: None 2024-04-12T04:04:59.316 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj17 2024-04-12T04:04:59.316 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e16rCLubnl8amV/4GsBlrVAwtNE= 2024-04-12T04:04:59.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e16rCLubnl8amV/4GsBlrVAwtNE=', 'Content-Length': '0'} 2024-04-12T04:04:59.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069e35e637507e81e-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:04:59.318 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:04:59.318 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:04:59.319 DEBUG:boto:Method: GET 2024-04-12T04:04:59.319 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:04:59.319 DEBUG:boto:Data: 2024-04-12T04:04:59.319 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.319 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.319 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.319 DEBUG:boto:Params: {} 2024-04-12T04:04:59.319 DEBUG:boto:Token: None 2024-04-12T04:04:59.319 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj17 2024-04-12T04:04:59.319 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LvmbagpwFp76UTK9h5g/fbfkbHY= 2024-04-12T04:04:59.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LvmbagpwFp76UTK9h5g/fbfkbHY=', 'Content-Length': '0'} 2024-04-12T04:04:59.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e999daf9418a78b9-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.322 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:04:59.322 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:04:59.322 DEBUG:boto:Method: GET 2024-04-12T04:04:59.322 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:04:59.322 DEBUG:boto:Data: 2024-04-12T04:04:59.322 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.322 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.322 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.322 DEBUG:boto:Params: {} 2024-04-12T04:04:59.323 DEBUG:boto:Token: None 2024-04-12T04:04:59.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj17 2024-04-12T04:04:59.323 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LvmbagpwFp76UTK9h5g/fbfkbHY= 2024-04-12T04:04:59.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LvmbagpwFp76UTK9h5g/fbfkbHY=', 'Content-Length': '0'} 2024-04-12T04:04:59.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088fefc1394b53033-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.325 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.325 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.325 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:04:59.325 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:04:59.325 DEBUG:boto:Method: GET 2024-04-12T04:04:59.325 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:04:59.325 DEBUG:boto:Data: 2024-04-12T04:04:59.326 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.326 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.326 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.326 DEBUG:boto:Params: {} 2024-04-12T04:04:59.326 DEBUG:boto:Token: None 2024-04-12T04:04:59.326 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.326 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092022e5c525d1d8c-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.329 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.329 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.329 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.329 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:04:59.330 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:04:59.330 DEBUG:boto:Method: GET 2024-04-12T04:04:59.330 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:04:59.330 DEBUG:boto:Data: 2024-04-12T04:04:59.330 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.330 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.330 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.330 DEBUG:boto:Params: {} 2024-04-12T04:04:59.331 DEBUG:boto:Token: None 2024-04-12T04:04:59.331 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.331 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfdc7a19e839ff37-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.334 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:04:59.335 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:04:59.335 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:04:59.335 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.335 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.335 DEBUG:boto:Method: GET 2024-04-12T04:04:59.335 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.336 DEBUG:boto:Data: 2024-04-12T04:04:59.336 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.336 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.336 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.336 DEBUG:boto:Params: {} 2024-04-12T04:04:59.336 DEBUG:boto:Token: None 2024-04-12T04:04:59.336 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.336 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ci8tK0/tje34ulBi4W/74PYfyMA= 2024-04-12T04:04:59.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ci8tK0/tje34ulBi4W/74PYfyMA=', 'Content-Length': '0'} 2024-04-12T04:04:59.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc537f69aa59e799-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.338 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:04:59.339 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:04:59.339 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.339 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.339 DEBUG:boto:Method: GET 2024-04-12T04:04:59.339 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.339 DEBUG:boto:Data: 2024-04-12T04:04:59.339 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.339 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.339 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.339 DEBUG:boto:Params: {} 2024-04-12T04:04:59.340 DEBUG:boto:Token: None 2024-04-12T04:04:59.340 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.340 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ci8tK0/tje34ulBi4W/74PYfyMA= 2024-04-12T04:04:59.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ci8tK0/tje34ulBi4W/74PYfyMA=', 'Content-Length': '0'} 2024-04-12T04:04:59.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7f33dd764f894ca-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.382 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:04:59.383 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:04:59.383 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.383 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.383 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.383 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.383 DEBUG:boto:Data: 2024-04-12T04:04:59.383 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.383 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.383 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.383 DEBUG:boto:Params: {} 2024-04-12T04:04:59.383 DEBUG:boto:Token: None 2024-04-12T04:04:59.384 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.384 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wz5NB6yE10AHLh5t/SLt+E+I2eU= 2024-04-12T04:04:59.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wz5NB6yE10AHLh5t/SLt+E+I2eU=', 'Content-Length': '0'} 2024-04-12T04:04:59.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc8d02c0692ebfdc-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.386 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:04:59.386 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:04:59.386 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.386 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.386 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.386 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.386 DEBUG:boto:Data: 2024-04-12T04:04:59.386 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.386 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.387 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.387 DEBUG:boto:Params: {} 2024-04-12T04:04:59.387 DEBUG:boto:Token: None 2024-04-12T04:04:59.387 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.387 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wz5NB6yE10AHLh5t/SLt+E+I2eU= 2024-04-12T04:04:59.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wz5NB6yE10AHLh5t/SLt+E+I2eU=', 'Content-Length': '0'} 2024-04-12T04:04:59.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f14e2b8240e024e0-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:04:59.389 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:04:59.389 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:04:59.389 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.389 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.389 DEBUG:boto:Method: GET 2024-04-12T04:04:59.389 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.389 DEBUG:boto:Data: 2024-04-12T04:04:59.390 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.390 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.390 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.390 DEBUG:boto:Params: {} 2024-04-12T04:04:59.390 DEBUG:boto:Token: None 2024-04-12T04:04:59.390 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.390 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ci8tK0/tje34ulBi4W/74PYfyMA= 2024-04-12T04:04:59.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ci8tK0/tje34ulBi4W/74PYfyMA=', 'Content-Length': '0'} 2024-04-12T04:04:59.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000754d25b6e29b4c76-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.393 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:04:59.393 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:04:59.393 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.393 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.393 DEBUG:boto:Method: GET 2024-04-12T04:04:59.393 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.393 DEBUG:boto:Data: 2024-04-12T04:04:59.393 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.394 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.394 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.394 DEBUG:boto:Params: {} 2024-04-12T04:04:59.394 DEBUG:boto:Token: None 2024-04-12T04:04:59.394 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:04:59.394 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ci8tK0/tje34ulBi4W/74PYfyMA= 2024-04-12T04:04:59.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ci8tK0/tje34ulBi4W/74PYfyMA=', 'Content-Length': '0'} 2024-04-12T04:04:59.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000567509e461e628d4-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.396 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.396 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.397 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:04:59.397 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:04:59.397 DEBUG:boto:Method: GET 2024-04-12T04:04:59.397 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:04:59.397 DEBUG:boto:Data: 2024-04-12T04:04:59.397 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.397 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.397 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.397 DEBUG:boto:Params: {} 2024-04-12T04:04:59.397 DEBUG:boto:Token: None 2024-04-12T04:04:59.398 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.398 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c388a0856fa5ce7f-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.401 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.401 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.401 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.401 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:04:59.402 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:04:59.402 DEBUG:boto:Method: GET 2024-04-12T04:04:59.402 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:04:59.402 DEBUG:boto:Data: 2024-04-12T04:04:59.402 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.402 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.402 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.402 DEBUG:boto:Params: {} 2024-04-12T04:04:59.402 DEBUG:boto:Token: None 2024-04-12T04:04:59.402 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.403 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f28a9e430b1191a-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.406 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.406 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:04:59.407 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:04:59.407 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.407 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:04:59.407 DEBUG:boto:Data: 2024-04-12T04:04:59.407 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.407 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.407 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.407 DEBUG:boto:Params: {} 2024-04-12T04:04:59.407 DEBUG:boto:Token: None 2024-04-12T04:04:59.408 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj19 2024-04-12T04:04:59.408 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:32/MSF+1arrpcc/sOsA0Mj3x0iI= 2024-04-12T04:04:59.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:32/MSF+1arrpcc/sOsA0Mj3x0iI=', 'Content-Length': '0'} 2024-04-12T04:04:59.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb574812d1206270-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.410 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:04:59.410 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:04:59.410 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.410 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:04:59.411 DEBUG:boto:Data: 2024-04-12T04:04:59.411 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.411 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.411 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.411 DEBUG:boto:Params: {} 2024-04-12T04:04:59.411 DEBUG:boto:Token: None 2024-04-12T04:04:59.411 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj19 2024-04-12T04:04:59.411 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:32/MSF+1arrpcc/sOsA0Mj3x0iI= 2024-04-12T04:04:59.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:32/MSF+1arrpcc/sOsA0Mj3x0iI=', 'Content-Length': '0'} 2024-04-12T04:04:59.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb15b3364efdd068-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:04:59.414 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:04:59.414 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:04:59.414 DEBUG:boto:Method: GET 2024-04-12T04:04:59.414 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:04:59.414 DEBUG:boto:Data: 2024-04-12T04:04:59.415 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.415 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.415 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.415 DEBUG:boto:Params: {} 2024-04-12T04:04:59.415 DEBUG:boto:Token: None 2024-04-12T04:04:59.415 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj19 2024-04-12T04:04:59.415 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RSVtNLalF3xn41rXXadI7Vadf90= 2024-04-12T04:04:59.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RSVtNLalF3xn41rXXadI7Vadf90=', 'Content-Length': '0'} 2024-04-12T04:04:59.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f729b7cf4b222024-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.418 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:04:59.418 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:04:59.418 DEBUG:boto:Method: GET 2024-04-12T04:04:59.418 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:04:59.418 DEBUG:boto:Data: 2024-04-12T04:04:59.418 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.418 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.418 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.418 DEBUG:boto:Params: {} 2024-04-12T04:04:59.419 DEBUG:boto:Token: None 2024-04-12T04:04:59.419 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj19 2024-04-12T04:04:59.419 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RSVtNLalF3xn41rXXadI7Vadf90= 2024-04-12T04:04:59.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RSVtNLalF3xn41rXXadI7Vadf90=', 'Content-Length': '0'} 2024-04-12T04:04:59.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baef5b5e04f4501b-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.421 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.421 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.421 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:04:59.422 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:04:59.422 DEBUG:boto:Method: GET 2024-04-12T04:04:59.422 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:04:59.422 DEBUG:boto:Data: 2024-04-12T04:04:59.422 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.422 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.422 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.422 DEBUG:boto:Params: {} 2024-04-12T04:04:59.422 DEBUG:boto:Token: None 2024-04-12T04:04:59.422 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.423 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000892389e7d3d9aba0-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.425 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.426 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.426 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.426 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:04:59.426 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:04:59.426 DEBUG:boto:Method: GET 2024-04-12T04:04:59.426 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:04:59.426 DEBUG:boto:Data: 2024-04-12T04:04:59.426 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.426 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.426 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.427 DEBUG:boto:Params: {} 2024-04-12T04:04:59.427 DEBUG:boto:Token: None 2024-04-12T04:04:59.429 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.429 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bb3694230b980ba-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.432 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:04:59.433 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:04:59.433 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:04:59.433 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.433 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.433 DEBUG:boto:Method: GET 2024-04-12T04:04:59.433 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.433 DEBUG:boto:Data: 2024-04-12T04:04:59.434 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.434 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.434 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.434 DEBUG:boto:Params: {} 2024-04-12T04:04:59.434 DEBUG:boto:Token: None 2024-04-12T04:04:59.434 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.434 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHj9/vaY2WBpD1dPMHcELStuDH0= 2024-04-12T04:04:59.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHj9/vaY2WBpD1dPMHcELStuDH0=', 'Content-Length': '0'} 2024-04-12T04:04:59.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f883186aeb67a0f9-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.437 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:04:59.437 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:04:59.437 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.437 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.437 DEBUG:boto:Method: GET 2024-04-12T04:04:59.437 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.437 DEBUG:boto:Data: 2024-04-12T04:04:59.437 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.437 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.437 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.437 DEBUG:boto:Params: {} 2024-04-12T04:04:59.438 DEBUG:boto:Token: None 2024-04-12T04:04:59.438 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.438 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHj9/vaY2WBpD1dPMHcELStuDH0= 2024-04-12T04:04:59.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHj9/vaY2WBpD1dPMHcELStuDH0=', 'Content-Length': '0'} 2024-04-12T04:04:59.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bf208c9ae76a92d-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.440 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:04:59.440 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:04:59.440 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.440 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.440 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.440 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.441 DEBUG:boto:Data: 2024-04-12T04:04:59.441 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.441 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.441 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.441 DEBUG:boto:Params: {} 2024-04-12T04:04:59.441 DEBUG:boto:Token: None 2024-04-12T04:04:59.441 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.441 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:M5WX5jO/craqjPzSuN+NtotTLVY= 2024-04-12T04:04:59.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:M5WX5jO/craqjPzSuN+NtotTLVY=', 'Content-Length': '0'} 2024-04-12T04:04:59.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000115aea6e0a3e613f-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.443 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:04:59.444 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:04:59.444 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.444 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.444 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.444 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.444 DEBUG:boto:Data: 2024-04-12T04:04:59.444 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.444 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.444 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.444 DEBUG:boto:Params: {} 2024-04-12T04:04:59.445 DEBUG:boto:Token: None 2024-04-12T04:04:59.445 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.445 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:M5WX5jO/craqjPzSuN+NtotTLVY= 2024-04-12T04:04:59.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:M5WX5jO/craqjPzSuN+NtotTLVY=', 'Content-Length': '0'} 2024-04-12T04:04:59.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d3d501eb3ba1a23-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:04:59.447 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:04:59.447 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:04:59.447 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.448 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.448 DEBUG:boto:Method: GET 2024-04-12T04:04:59.448 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.448 DEBUG:boto:Data: 2024-04-12T04:04:59.448 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.448 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.448 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.448 DEBUG:boto:Params: {} 2024-04-12T04:04:59.448 DEBUG:boto:Token: None 2024-04-12T04:04:59.449 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.449 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHj9/vaY2WBpD1dPMHcELStuDH0= 2024-04-12T04:04:59.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHj9/vaY2WBpD1dPMHcELStuDH0=', 'Content-Length': '0'} 2024-04-12T04:04:59.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aae91818aebb0f3-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.494 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:04:59.494 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:04:59.495 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.495 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.495 DEBUG:boto:Method: GET 2024-04-12T04:04:59.495 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.495 DEBUG:boto:Data: 2024-04-12T04:04:59.495 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.495 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.495 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.495 DEBUG:boto:Params: {} 2024-04-12T04:04:59.495 DEBUG:boto:Token: None 2024-04-12T04:04:59.496 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:04:59.496 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oHj9/vaY2WBpD1dPMHcELStuDH0= 2024-04-12T04:04:59.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oHj9/vaY2WBpD1dPMHcELStuDH0=', 'Content-Length': '0'} 2024-04-12T04:04:59.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061f5a76085aa701e-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.498 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.498 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.498 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:04:59.498 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:04:59.498 DEBUG:boto:Method: GET 2024-04-12T04:04:59.498 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:04:59.499 DEBUG:boto:Data: 2024-04-12T04:04:59.499 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.499 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.499 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.499 DEBUG:boto:Params: {} 2024-04-12T04:04:59.499 DEBUG:boto:Token: None 2024-04-12T04:04:59.499 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.499 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8fc19ca7a15e590-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.502 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.503 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.503 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.503 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:04:59.503 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:04:59.503 DEBUG:boto:Method: GET 2024-04-12T04:04:59.503 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:04:59.503 DEBUG:boto:Data: 2024-04-12T04:04:59.503 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.503 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.503 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.503 DEBUG:boto:Params: {} 2024-04-12T04:04:59.504 DEBUG:boto:Token: None 2024-04-12T04:04:59.504 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.504 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dd64215c3c2c9cd-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.507 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.508 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:04:59.508 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:04:59.508 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.508 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:04:59.508 DEBUG:boto:Data: 2024-04-12T04:04:59.508 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.508 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.508 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.508 DEBUG:boto:Params: {} 2024-04-12T04:04:59.508 DEBUG:boto:Token: None 2024-04-12T04:04:59.509 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj22 2024-04-12T04:04:59.509 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:um7l/cSX1Lu/fKGK0oPdZHMurSw= 2024-04-12T04:04:59.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:um7l/cSX1Lu/fKGK0oPdZHMurSw=', 'Content-Length': '0'} 2024-04-12T04:04:59.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a743bcdb033b6110-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.511 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:04:59.511 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:04:59.511 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.511 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:04:59.511 DEBUG:boto:Data: 2024-04-12T04:04:59.511 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.511 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.511 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.511 DEBUG:boto:Params: {} 2024-04-12T04:04:59.511 DEBUG:boto:Token: None 2024-04-12T04:04:59.511 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj22 2024-04-12T04:04:59.512 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:um7l/cSX1Lu/fKGK0oPdZHMurSw= 2024-04-12T04:04:59.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:um7l/cSX1Lu/fKGK0oPdZHMurSw=', 'Content-Length': '0'} 2024-04-12T04:04:59.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007f62c8445f0b3f8-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:04:59.514 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:04:59.514 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:04:59.514 DEBUG:boto:Method: GET 2024-04-12T04:04:59.514 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:04:59.514 DEBUG:boto:Data: 2024-04-12T04:04:59.514 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.514 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.514 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.514 DEBUG:boto:Params: {} 2024-04-12T04:04:59.515 DEBUG:boto:Token: None 2024-04-12T04:04:59.515 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj22 2024-04-12T04:04:59.515 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i90N6YCOzdWels2pEZBNJrNKHOI= 2024-04-12T04:04:59.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i90N6YCOzdWels2pEZBNJrNKHOI=', 'Content-Length': '0'} 2024-04-12T04:04:59.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093c4da377293dda8-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.517 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:04:59.517 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:04:59.517 DEBUG:boto:Method: GET 2024-04-12T04:04:59.517 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:04:59.518 DEBUG:boto:Data: 2024-04-12T04:04:59.518 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.518 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.518 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.518 DEBUG:boto:Params: {} 2024-04-12T04:04:59.518 DEBUG:boto:Token: None 2024-04-12T04:04:59.518 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj22 2024-04-12T04:04:59.518 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i90N6YCOzdWels2pEZBNJrNKHOI= 2024-04-12T04:04:59.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i90N6YCOzdWels2pEZBNJrNKHOI=', 'Content-Length': '0'} 2024-04-12T04:04:59.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009638b6def1fb5f17-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.562 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.563 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.563 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:04:59.563 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:04:59.563 DEBUG:boto:Method: GET 2024-04-12T04:04:59.563 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:04:59.563 DEBUG:boto:Data: 2024-04-12T04:04:59.563 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.563 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.563 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.563 DEBUG:boto:Params: {} 2024-04-12T04:04:59.564 DEBUG:boto:Token: None 2024-04-12T04:04:59.564 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.564 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042fa6f97037fd1f2-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.566 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.567 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.567 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.567 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:04:59.567 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:04:59.567 DEBUG:boto:Method: GET 2024-04-12T04:04:59.567 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:04:59.568 DEBUG:boto:Data: 2024-04-12T04:04:59.568 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.568 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.568 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.568 DEBUG:boto:Params: {} 2024-04-12T04:04:59.568 DEBUG:boto:Token: None 2024-04-12T04:04:59.568 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.568 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e357bcc79cfbed6-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.571 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:04:59.572 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:04:59.572 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:04:59.572 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.572 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.572 DEBUG:boto:Method: GET 2024-04-12T04:04:59.572 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.572 DEBUG:boto:Data: 2024-04-12T04:04:59.573 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.573 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.573 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.573 DEBUG:boto:Params: {} 2024-04-12T04:04:59.573 DEBUG:boto:Token: None 2024-04-12T04:04:59.573 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.573 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DIX4LbSa5RF6p4n+kDt2gfz+QIs= 2024-04-12T04:04:59.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DIX4LbSa5RF6p4n+kDt2gfz+QIs=', 'Content-Length': '0'} 2024-04-12T04:04:59.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5454e3cdd15a25a-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.623 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:04:59.623 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:04:59.623 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.623 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.624 DEBUG:boto:Method: GET 2024-04-12T04:04:59.624 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.624 DEBUG:boto:Data: 2024-04-12T04:04:59.624 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.624 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.624 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.624 DEBUG:boto:Params: {} 2024-04-12T04:04:59.624 DEBUG:boto:Token: None 2024-04-12T04:04:59.624 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.625 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DIX4LbSa5RF6p4n+kDt2gfz+QIs= 2024-04-12T04:04:59.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DIX4LbSa5RF6p4n+kDt2gfz+QIs=', 'Content-Length': '0'} 2024-04-12T04:04:59.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087c9d5cd88c24cba-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.627 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:04:59.627 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:04:59.627 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.627 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.627 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.628 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.628 DEBUG:boto:Data: 2024-04-12T04:04:59.628 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.628 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.628 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.628 DEBUG:boto:Params: {} 2024-04-12T04:04:59.628 DEBUG:boto:Token: None 2024-04-12T04:04:59.628 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EGsQ2uVrEggy0htPf+hGWIZLGsU= 2024-04-12T04:04:59.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EGsQ2uVrEggy0htPf+hGWIZLGsU=', 'Content-Length': '0'} 2024-04-12T04:04:59.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033ff73a74006dd8d-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.630 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:04:59.631 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:04:59.631 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.631 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.631 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.631 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.631 DEBUG:boto:Data: 2024-04-12T04:04:59.631 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.631 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.631 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.631 DEBUG:boto:Params: {} 2024-04-12T04:04:59.631 DEBUG:boto:Token: None 2024-04-12T04:04:59.632 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.632 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EGsQ2uVrEggy0htPf+hGWIZLGsU= 2024-04-12T04:04:59.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EGsQ2uVrEggy0htPf+hGWIZLGsU=', 'Content-Length': '0'} 2024-04-12T04:04:59.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000183fea1937451265-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:04:59.634 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:04:59.634 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:04:59.634 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.634 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.634 DEBUG:boto:Method: GET 2024-04-12T04:04:59.634 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.634 DEBUG:boto:Data: 2024-04-12T04:04:59.634 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.635 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.635 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.635 DEBUG:boto:Params: {} 2024-04-12T04:04:59.635 DEBUG:boto:Token: None 2024-04-12T04:04:59.635 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.635 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DIX4LbSa5RF6p4n+kDt2gfz+QIs= 2024-04-12T04:04:59.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DIX4LbSa5RF6p4n+kDt2gfz+QIs=', 'Content-Length': '0'} 2024-04-12T04:04:59.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074e4fc7b7765ee58-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.637 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:04:59.637 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:04:59.637 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.637 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.637 DEBUG:boto:Method: GET 2024-04-12T04:04:59.638 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.638 DEBUG:boto:Data: 2024-04-12T04:04:59.638 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.638 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.638 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.638 DEBUG:boto:Params: {} 2024-04-12T04:04:59.638 DEBUG:boto:Token: None 2024-04-12T04:04:59.638 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:04:59.638 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DIX4LbSa5RF6p4n+kDt2gfz+QIs= 2024-04-12T04:04:59.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DIX4LbSa5RF6p4n+kDt2gfz+QIs=', 'Content-Length': '0'} 2024-04-12T04:04:59.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baf5b843c854dc9e-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.641 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.641 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.641 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:04:59.641 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:04:59.641 DEBUG:boto:Method: GET 2024-04-12T04:04:59.641 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:04:59.641 DEBUG:boto:Data: 2024-04-12T04:04:59.641 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.641 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.641 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.642 DEBUG:boto:Params: {} 2024-04-12T04:04:59.642 DEBUG:boto:Token: None 2024-04-12T04:04:59.642 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.642 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097f83403705f3aef-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.644 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.645 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.645 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.645 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:04:59.645 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:04:59.645 DEBUG:boto:Method: GET 2024-04-12T04:04:59.645 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:04:59.645 DEBUG:boto:Data: 2024-04-12T04:04:59.645 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.645 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.645 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.645 DEBUG:boto:Params: {} 2024-04-12T04:04:59.646 DEBUG:boto:Token: None 2024-04-12T04:04:59.646 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.646 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cbee3f45dd450e6-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.649 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.649 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:04:59.649 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:04:59.649 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.649 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:04:59.649 DEBUG:boto:Data: 2024-04-12T04:04:59.649 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.649 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.650 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.650 DEBUG:boto:Params: {} 2024-04-12T04:04:59.650 DEBUG:boto:Token: None 2024-04-12T04:04:59.650 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj33 2024-04-12T04:04:59.650 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IcQsZLIZJb4K1Vjmd/TGwmtxVKs= 2024-04-12T04:04:59.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IcQsZLIZJb4K1Vjmd/TGwmtxVKs=', 'Content-Length': '0'} 2024-04-12T04:04:59.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000186bb61e2d551e5-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.652 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:04:59.652 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:04:59.652 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.652 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:04:59.652 DEBUG:boto:Data: 2024-04-12T04:04:59.652 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.653 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.653 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.653 DEBUG:boto:Params: {} 2024-04-12T04:04:59.653 DEBUG:boto:Token: None 2024-04-12T04:04:59.653 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj33 2024-04-12T04:04:59.653 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IcQsZLIZJb4K1Vjmd/TGwmtxVKs= 2024-04-12T04:04:59.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IcQsZLIZJb4K1Vjmd/TGwmtxVKs=', 'Content-Length': '0'} 2024-04-12T04:04:59.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eecc44e80e526a24-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:04:59.655 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:04:59.655 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:04:59.656 DEBUG:boto:Method: GET 2024-04-12T04:04:59.656 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:04:59.656 DEBUG:boto:Data: 2024-04-12T04:04:59.656 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.656 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.656 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.656 DEBUG:boto:Params: {} 2024-04-12T04:04:59.656 DEBUG:boto:Token: None 2024-04-12T04:04:59.656 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj33 2024-04-12T04:04:59.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tR1ODlzqh0leMhlVIuH8PFwoKFU= 2024-04-12T04:04:59.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tR1ODlzqh0leMhlVIuH8PFwoKFU=', 'Content-Length': '0'} 2024-04-12T04:04:59.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057d4d887ce559000-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.659 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:04:59.659 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:04:59.659 DEBUG:boto:Method: GET 2024-04-12T04:04:59.659 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:04:59.659 DEBUG:boto:Data: 2024-04-12T04:04:59.659 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.659 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.659 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.659 DEBUG:boto:Params: {} 2024-04-12T04:04:59.660 DEBUG:boto:Token: None 2024-04-12T04:04:59.660 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj33 2024-04-12T04:04:59.660 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tR1ODlzqh0leMhlVIuH8PFwoKFU= 2024-04-12T04:04:59.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tR1ODlzqh0leMhlVIuH8PFwoKFU=', 'Content-Length': '0'} 2024-04-12T04:04:59.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fcdd97c67ccebf1-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.702 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.703 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.703 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:04:59.703 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:04:59.703 DEBUG:boto:Method: GET 2024-04-12T04:04:59.703 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:04:59.703 DEBUG:boto:Data: 2024-04-12T04:04:59.703 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.703 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.703 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.703 DEBUG:boto:Params: {} 2024-04-12T04:04:59.704 DEBUG:boto:Token: None 2024-04-12T04:04:59.704 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.704 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007faab60cc28311f4-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.706 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.707 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.707 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.707 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:04:59.707 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:04:59.707 DEBUG:boto:Method: GET 2024-04-12T04:04:59.707 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:04:59.707 DEBUG:boto:Data: 2024-04-12T04:04:59.708 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.708 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.708 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.708 DEBUG:boto:Params: {} 2024-04-12T04:04:59.708 DEBUG:boto:Token: None 2024-04-12T04:04:59.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.708 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ccb10fc9c6573da-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.711 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:04:59.712 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:04:59.712 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:04:59.712 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.712 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.712 DEBUG:boto:Method: GET 2024-04-12T04:04:59.712 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.712 DEBUG:boto:Data: 2024-04-12T04:04:59.712 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.712 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.713 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.713 DEBUG:boto:Params: {} 2024-04-12T04:04:59.713 DEBUG:boto:Token: None 2024-04-12T04:04:59.713 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.713 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iGGh5TmXXOJ2hC4gkC2A6GaQz9s= 2024-04-12T04:04:59.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iGGh5TmXXOJ2hC4gkC2A6GaQz9s=', 'Content-Length': '0'} 2024-04-12T04:04:59.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb7d944e153429c3-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.715 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:04:59.715 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:04:59.715 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.716 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.716 DEBUG:boto:Method: GET 2024-04-12T04:04:59.716 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.716 DEBUG:boto:Data: 2024-04-12T04:04:59.716 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.716 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.716 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.716 DEBUG:boto:Params: {} 2024-04-12T04:04:59.716 DEBUG:boto:Token: None 2024-04-12T04:04:59.716 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.717 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iGGh5TmXXOJ2hC4gkC2A6GaQz9s= 2024-04-12T04:04:59.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iGGh5TmXXOJ2hC4gkC2A6GaQz9s=', 'Content-Length': '0'} 2024-04-12T04:04:59.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bef02bb217d7d79-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.719 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:04:59.719 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:04:59.719 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.719 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.719 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.719 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.720 DEBUG:boto:Data: 2024-04-12T04:04:59.720 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.720 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.720 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.720 DEBUG:boto:Params: {} 2024-04-12T04:04:59.720 DEBUG:boto:Token: None 2024-04-12T04:04:59.720 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.720 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jNgLrElLTl1J5M7/Rw6kjV38pIM= 2024-04-12T04:04:59.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jNgLrElLTl1J5M7/Rw6kjV38pIM=', 'Content-Length': '0'} 2024-04-12T04:04:59.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e9b325a63a2975b-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.722 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:04:59.722 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:04:59.722 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.722 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.723 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.723 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.723 DEBUG:boto:Data: 2024-04-12T04:04:59.723 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.723 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.723 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.723 DEBUG:boto:Params: {} 2024-04-12T04:04:59.723 DEBUG:boto:Token: None 2024-04-12T04:04:59.723 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.723 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jNgLrElLTl1J5M7/Rw6kjV38pIM= 2024-04-12T04:04:59.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jNgLrElLTl1J5M7/Rw6kjV38pIM=', 'Content-Length': '0'} 2024-04-12T04:04:59.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f55db5c1e123071-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:04:59.726 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:04:59.726 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:04:59.726 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.726 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.726 DEBUG:boto:Method: GET 2024-04-12T04:04:59.726 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.726 DEBUG:boto:Data: 2024-04-12T04:04:59.727 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.727 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.727 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.727 DEBUG:boto:Params: {} 2024-04-12T04:04:59.727 DEBUG:boto:Token: None 2024-04-12T04:04:59.727 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.727 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iGGh5TmXXOJ2hC4gkC2A6GaQz9s= 2024-04-12T04:04:59.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iGGh5TmXXOJ2hC4gkC2A6GaQz9s=', 'Content-Length': '0'} 2024-04-12T04:04:59.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6c1f1ee12e0880a-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.770 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:04:59.770 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:04:59.770 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.771 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.771 DEBUG:boto:Method: GET 2024-04-12T04:04:59.771 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.771 DEBUG:boto:Data: 2024-04-12T04:04:59.771 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.771 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.771 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.771 DEBUG:boto:Params: {} 2024-04-12T04:04:59.771 DEBUG:boto:Token: None 2024-04-12T04:04:59.771 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:04:59.771 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iGGh5TmXXOJ2hC4gkC2A6GaQz9s= 2024-04-12T04:04:59.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iGGh5TmXXOJ2hC4gkC2A6GaQz9s=', 'Content-Length': '0'} 2024-04-12T04:04:59.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019ce6e381812e04e-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.774 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.774 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.774 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:04:59.774 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:04:59.774 DEBUG:boto:Method: GET 2024-04-12T04:04:59.774 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:04:59.774 DEBUG:boto:Data: 2024-04-12T04:04:59.774 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.774 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.775 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.775 DEBUG:boto:Params: {} 2024-04-12T04:04:59.775 DEBUG:boto:Token: None 2024-04-12T04:04:59.775 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.775 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e97ee9a4f042ba56-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.777 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.778 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.778 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.778 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:04:59.778 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:04:59.778 DEBUG:boto:Method: GET 2024-04-12T04:04:59.778 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:04:59.778 DEBUG:boto:Data: 2024-04-12T04:04:59.778 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.779 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.779 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.779 DEBUG:boto:Params: {} 2024-04-12T04:04:59.779 DEBUG:boto:Token: None 2024-04-12T04:04:59.779 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.779 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007520685b70c6bd55-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.782 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.782 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:04:59.782 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:04:59.782 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.783 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:04:59.783 DEBUG:boto:Data: 2024-04-12T04:04:59.783 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.783 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.783 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.783 DEBUG:boto:Params: {} 2024-04-12T04:04:59.783 DEBUG:boto:Token: None 2024-04-12T04:04:59.783 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj34 2024-04-12T04:04:59.783 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0h7kzFqHqM5Thd9k8HFCCg850zQ= 2024-04-12T04:04:59.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0h7kzFqHqM5Thd9k8HFCCg850zQ=', 'Content-Length': '0'} 2024-04-12T04:04:59.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec054fdf716c3956-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.786 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:04:59.786 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:04:59.786 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.786 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:04:59.786 DEBUG:boto:Data: 2024-04-12T04:04:59.786 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.786 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.786 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.786 DEBUG:boto:Params: {} 2024-04-12T04:04:59.786 DEBUG:boto:Token: None 2024-04-12T04:04:59.787 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj34 2024-04-12T04:04:59.787 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0h7kzFqHqM5Thd9k8HFCCg850zQ= 2024-04-12T04:04:59.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0h7kzFqHqM5Thd9k8HFCCg850zQ=', 'Content-Length': '0'} 2024-04-12T04:04:59.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea9920062baa1681-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:04:59.789 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:04:59.789 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:04:59.789 DEBUG:boto:Method: GET 2024-04-12T04:04:59.789 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:04:59.789 DEBUG:boto:Data: 2024-04-12T04:04:59.789 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.789 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.790 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.790 DEBUG:boto:Params: {} 2024-04-12T04:04:59.790 DEBUG:boto:Token: None 2024-04-12T04:04:59.790 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj34 2024-04-12T04:04:59.790 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZScqj4g0MxfAOw1cWdBGWp7R43Y= 2024-04-12T04:04:59.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZScqj4g0MxfAOw1cWdBGWp7R43Y=', 'Content-Length': '0'} 2024-04-12T04:04:59.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f46777d2183ce588-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.792 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:04:59.792 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:04:59.792 DEBUG:boto:Method: GET 2024-04-12T04:04:59.792 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:04:59.792 DEBUG:boto:Data: 2024-04-12T04:04:59.792 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.792 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.793 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.793 DEBUG:boto:Params: {} 2024-04-12T04:04:59.793 DEBUG:boto:Token: None 2024-04-12T04:04:59.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj34 2024-04-12T04:04:59.793 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZScqj4g0MxfAOw1cWdBGWp7R43Y= 2024-04-12T04:04:59.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZScqj4g0MxfAOw1cWdBGWp7R43Y=', 'Content-Length': '0'} 2024-04-12T04:04:59.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056136fafd3c6b7c3-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.795 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.795 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.796 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:04:59.796 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:04:59.796 DEBUG:boto:Method: GET 2024-04-12T04:04:59.796 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:04:59.796 DEBUG:boto:Data: 2024-04-12T04:04:59.796 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.796 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.796 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.796 DEBUG:boto:Params: {} 2024-04-12T04:04:59.796 DEBUG:boto:Token: None 2024-04-12T04:04:59.797 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.797 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006422d06ac2b86d3d-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.799 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.799 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.800 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.800 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:04:59.800 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:04:59.800 DEBUG:boto:Method: GET 2024-04-12T04:04:59.800 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:04:59.800 DEBUG:boto:Data: 2024-04-12T04:04:59.800 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.800 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.800 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.800 DEBUG:boto:Params: {} 2024-04-12T04:04:59.800 DEBUG:boto:Token: None 2024-04-12T04:04:59.800 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.801 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ccdba50f9cf9d32-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.804 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:04:59.804 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:04:59.804 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:04:59.805 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.805 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.805 DEBUG:boto:Method: GET 2024-04-12T04:04:59.805 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.805 DEBUG:boto:Data: 2024-04-12T04:04:59.805 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.805 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.805 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.805 DEBUG:boto:Params: {} 2024-04-12T04:04:59.805 DEBUG:boto:Token: None 2024-04-12T04:04:59.805 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.806 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A1Y/s/eRBCPoD9UPycmK7Cds+5I= 2024-04-12T04:04:59.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A1Y/s/eRBCPoD9UPycmK7Cds+5I=', 'Content-Length': '0'} 2024-04-12T04:04:59.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9666c2de0fa90d7-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.808 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:04:59.808 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:04:59.808 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.808 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.808 DEBUG:boto:Method: GET 2024-04-12T04:04:59.808 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.808 DEBUG:boto:Data: 2024-04-12T04:04:59.808 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.808 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.808 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.809 DEBUG:boto:Params: {} 2024-04-12T04:04:59.809 DEBUG:boto:Token: None 2024-04-12T04:04:59.809 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.809 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A1Y/s/eRBCPoD9UPycmK7Cds+5I= 2024-04-12T04:04:59.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A1Y/s/eRBCPoD9UPycmK7Cds+5I=', 'Content-Length': '0'} 2024-04-12T04:04:59.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a40ed1b5019af2ca-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.811 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:04:59.811 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:04:59.811 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.811 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.811 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.811 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.812 DEBUG:boto:Data: 2024-04-12T04:04:59.812 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.812 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.812 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.812 DEBUG:boto:Params: {} 2024-04-12T04:04:59.812 DEBUG:boto:Token: None 2024-04-12T04:04:59.812 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.812 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h4OunaQ7JONIVYFm2TYq2SwVo9o= 2024-04-12T04:04:59.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h4OunaQ7JONIVYFm2TYq2SwVo9o=', 'Content-Length': '0'} 2024-04-12T04:04:59.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d49c92e594715169-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.814 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:04:59.814 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:04:59.814 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.814 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.815 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.815 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.815 DEBUG:boto:Data: 2024-04-12T04:04:59.815 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.815 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.815 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.815 DEBUG:boto:Params: {} 2024-04-12T04:04:59.815 DEBUG:boto:Token: None 2024-04-12T04:04:59.815 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.815 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h4OunaQ7JONIVYFm2TYq2SwVo9o= 2024-04-12T04:04:59.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h4OunaQ7JONIVYFm2TYq2SwVo9o=', 'Content-Length': '0'} 2024-04-12T04:04:59.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc0fdb72f9f023c-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:04:59.817 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:04:59.818 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:04:59.818 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.818 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.818 DEBUG:boto:Method: GET 2024-04-12T04:04:59.818 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.818 DEBUG:boto:Data: 2024-04-12T04:04:59.818 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.818 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.818 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.818 DEBUG:boto:Params: {} 2024-04-12T04:04:59.818 DEBUG:boto:Token: None 2024-04-12T04:04:59.819 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.819 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A1Y/s/eRBCPoD9UPycmK7Cds+5I= 2024-04-12T04:04:59.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A1Y/s/eRBCPoD9UPycmK7Cds+5I=', 'Content-Length': '0'} 2024-04-12T04:04:59.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056ad1c7ba39ecdb3-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.862 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:04:59.862 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:04:59.863 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.863 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.863 DEBUG:boto:Method: GET 2024-04-12T04:04:59.863 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.863 DEBUG:boto:Data: 2024-04-12T04:04:59.863 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.863 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.863 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.863 DEBUG:boto:Params: {} 2024-04-12T04:04:59.863 DEBUG:boto:Token: None 2024-04-12T04:04:59.863 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:04:59.864 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A1Y/s/eRBCPoD9UPycmK7Cds+5I= 2024-04-12T04:04:59.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A1Y/s/eRBCPoD9UPycmK7Cds+5I=', 'Content-Length': '0'} 2024-04-12T04:04:59.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb515225c28dfe50-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.866 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.866 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.866 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:04:59.866 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:04:59.866 DEBUG:boto:Method: GET 2024-04-12T04:04:59.866 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:04:59.866 DEBUG:boto:Data: 2024-04-12T04:04:59.866 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.866 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.866 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.867 DEBUG:boto:Params: {} 2024-04-12T04:04:59.867 DEBUG:boto:Token: None 2024-04-12T04:04:59.867 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.867 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eae4136d1be22794-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.869 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.870 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.870 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.870 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:04:59.870 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:04:59.870 DEBUG:boto:Method: GET 2024-04-12T04:04:59.870 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:04:59.870 DEBUG:boto:Data: 2024-04-12T04:04:59.871 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.871 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.871 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.871 DEBUG:boto:Params: {} 2024-04-12T04:04:59.871 DEBUG:boto:Token: None 2024-04-12T04:04:59.871 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.871 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000907e70fa785f37aa-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.874 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.875 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:04:59.875 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:04:59.875 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.875 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:04:59.875 DEBUG:boto:Data: 2024-04-12T04:04:59.875 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.875 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.875 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.875 DEBUG:boto:Params: {} 2024-04-12T04:04:59.875 DEBUG:boto:Token: None 2024-04-12T04:04:59.876 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj38 2024-04-12T04:04:59.876 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eJbsXrMhKFF/g0yZ67J6465589w= 2024-04-12T04:04:59.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eJbsXrMhKFF/g0yZ67J6465589w=', 'Content-Length': '0'} 2024-04-12T04:04:59.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000344342908681e82c-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.878 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:04:59.878 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:04:59.878 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.878 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:04:59.878 DEBUG:boto:Data: 2024-04-12T04:04:59.878 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.878 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.878 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.878 DEBUG:boto:Params: {} 2024-04-12T04:04:59.878 DEBUG:boto:Token: None 2024-04-12T04:04:59.879 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj38 2024-04-12T04:04:59.879 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eJbsXrMhKFF/g0yZ67J6465589w= 2024-04-12T04:04:59.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eJbsXrMhKFF/g0yZ67J6465589w=', 'Content-Length': '0'} 2024-04-12T04:04:59.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbedff551c7863c4-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:04:59.881 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:04:59.881 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:04:59.881 DEBUG:boto:Method: GET 2024-04-12T04:04:59.881 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:04:59.881 DEBUG:boto:Data: 2024-04-12T04:04:59.881 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.881 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.882 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.882 DEBUG:boto:Params: {} 2024-04-12T04:04:59.882 DEBUG:boto:Token: None 2024-04-12T04:04:59.882 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj38 2024-04-12T04:04:59.882 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MoGbZ7jlvwx5swRktRkvGoIwWdA= 2024-04-12T04:04:59.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MoGbZ7jlvwx5swRktRkvGoIwWdA=', 'Content-Length': '0'} 2024-04-12T04:04:59.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ece6f3faa9a5a29-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.884 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:04:59.884 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:04:59.884 DEBUG:boto:Method: GET 2024-04-12T04:04:59.884 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:04:59.885 DEBUG:boto:Data: 2024-04-12T04:04:59.885 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.885 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.885 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.885 DEBUG:boto:Params: {} 2024-04-12T04:04:59.885 DEBUG:boto:Token: None 2024-04-12T04:04:59.885 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj38 2024-04-12T04:04:59.885 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MoGbZ7jlvwx5swRktRkvGoIwWdA= 2024-04-12T04:04:59.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MoGbZ7jlvwx5swRktRkvGoIwWdA=', 'Content-Length': '0'} 2024-04-12T04:04:59.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a5db0f190a32430-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.887 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.887 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.887 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:04:59.887 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:04:59.887 DEBUG:boto:Method: GET 2024-04-12T04:04:59.888 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:04:59.888 DEBUG:boto:Data: 2024-04-12T04:04:59.888 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.888 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.888 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.888 DEBUG:boto:Params: {} 2024-04-12T04:04:59.888 DEBUG:boto:Token: None 2024-04-12T04:04:59.888 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.888 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6c0e8548aa92e92-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.891 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.891 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.891 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.891 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:04:59.891 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:04:59.891 DEBUG:boto:Method: GET 2024-04-12T04:04:59.892 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:04:59.892 DEBUG:boto:Data: 2024-04-12T04:04:59.892 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.892 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.892 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.892 DEBUG:boto:Params: {} 2024-04-12T04:04:59.892 DEBUG:boto:Token: None 2024-04-12T04:04:59.892 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.892 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd54fcdad7f88b69-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.896 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:04:59.896 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:04:59.896 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:04:59.896 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.896 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.897 DEBUG:boto:Method: GET 2024-04-12T04:04:59.897 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.897 DEBUG:boto:Data: 2024-04-12T04:04:59.897 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.897 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.897 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.897 DEBUG:boto:Params: {} 2024-04-12T04:04:59.897 DEBUG:boto:Token: None 2024-04-12T04:04:59.897 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.897 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PCQnSB0Mq/13AfGNtauQroJor4M= 2024-04-12T04:04:59.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PCQnSB0Mq/13AfGNtauQroJor4M=', 'Content-Length': '0'} 2024-04-12T04:04:59.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c29cde68872d6aa-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.900 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:04:59.900 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:04:59.900 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.900 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.900 DEBUG:boto:Method: GET 2024-04-12T04:04:59.900 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.900 DEBUG:boto:Data: 2024-04-12T04:04:59.900 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.900 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.900 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.900 DEBUG:boto:Params: {} 2024-04-12T04:04:59.900 DEBUG:boto:Token: None 2024-04-12T04:04:59.901 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.901 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PCQnSB0Mq/13AfGNtauQroJor4M= 2024-04-12T04:04:59.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PCQnSB0Mq/13AfGNtauQroJor4M=', 'Content-Length': '0'} 2024-04-12T04:04:59.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd4d099624309ef6-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.946 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:04:59.946 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:04:59.947 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.947 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.947 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.947 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.947 DEBUG:boto:Data: 2024-04-12T04:04:59.947 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.947 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.947 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.947 DEBUG:boto:Params: {} 2024-04-12T04:04:59.947 DEBUG:boto:Token: None 2024-04-12T04:04:59.947 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.947 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r0ouFEi5fZCF8MshTqBRMxUJhmE= 2024-04-12T04:04:59.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r0ouFEi5fZCF8MshTqBRMxUJhmE=', 'Content-Length': '0'} 2024-04-12T04:04:59.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000688dc90ec6e3d44-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.950 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:04:59.950 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:04:59.950 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.950 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.950 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.950 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.950 DEBUG:boto:Data: 2024-04-12T04:04:59.950 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.950 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.950 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.951 DEBUG:boto:Params: {} 2024-04-12T04:04:59.951 DEBUG:boto:Token: None 2024-04-12T04:04:59.951 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.951 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r0ouFEi5fZCF8MshTqBRMxUJhmE= 2024-04-12T04:04:59.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r0ouFEi5fZCF8MshTqBRMxUJhmE=', 'Content-Length': '0'} 2024-04-12T04:04:59.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000df330c77a42802-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:04:59.953 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:04:59.953 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:04:59.953 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.953 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.953 DEBUG:boto:Method: GET 2024-04-12T04:04:59.953 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.953 DEBUG:boto:Data: 2024-04-12T04:04:59.953 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.953 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.954 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.954 DEBUG:boto:Params: {} 2024-04-12T04:04:59.954 DEBUG:boto:Token: None 2024-04-12T04:04:59.954 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.954 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PCQnSB0Mq/13AfGNtauQroJor4M= 2024-04-12T04:04:59.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PCQnSB0Mq/13AfGNtauQroJor4M=', 'Content-Length': '0'} 2024-04-12T04:04:59.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000da7a23dba7aadee-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.956 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:04:59.956 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:04:59.956 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.956 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.956 DEBUG:boto:Method: GET 2024-04-12T04:04:59.956 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.956 DEBUG:boto:Data: 2024-04-12T04:04:59.957 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.957 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.957 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.957 DEBUG:boto:Params: {} 2024-04-12T04:04:59.957 DEBUG:boto:Token: None 2024-04-12T04:04:59.957 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:04:59.957 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PCQnSB0Mq/13AfGNtauQroJor4M= 2024-04-12T04:04:59.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PCQnSB0Mq/13AfGNtauQroJor4M=', 'Content-Length': '0'} 2024-04-12T04:04:59.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b82ab4bd96d8015f-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.959 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.959 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.959 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:04:59.959 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:04:59.959 DEBUG:boto:Method: GET 2024-04-12T04:04:59.959 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:04:59.960 DEBUG:boto:Data: 2024-04-12T04:04:59.960 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.960 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.960 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.960 DEBUG:boto:Params: {} 2024-04-12T04:04:59.960 DEBUG:boto:Token: None 2024-04-12T04:04:59.960 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.960 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba6d6fe963e9a269-006618b2eb-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.963 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.963 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:04:59.963 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:04:59.963 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:04:59.963 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:04:59.964 DEBUG:boto:Method: GET 2024-04-12T04:04:59.964 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:04:59.964 DEBUG:boto:Data: 2024-04-12T04:04:59.964 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.964 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.964 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.964 DEBUG:boto:Params: {} 2024-04-12T04:04:59.964 DEBUG:boto:Token: None 2024-04-12T04:04:59.964 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/ 2024-04-12T04:04:59.964 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg= 2024-04-12T04:04:59.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuOoCqHJIvZc/kHzI7Y2T/cM4pg=', 'Content-Length': '0'} 2024-04-12T04:04:59.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e69b22180b65bae-006618b2eb-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.968 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:04:59.968 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:04:59.968 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:04:59.968 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.968 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:04:59.968 DEBUG:boto:Data: 2024-04-12T04:04:59.968 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.968 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.968 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.969 DEBUG:boto:Params: {} 2024-04-12T04:04:59.969 DEBUG:boto:Token: None 2024-04-12T04:04:59.969 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj44 2024-04-12T04:04:59.969 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4pI0wgAEpBezGTWa/lE84zyGqdk= 2024-04-12T04:04:59.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4pI0wgAEpBezGTWa/lE84zyGqdk=', 'Content-Length': '0'} 2024-04-12T04:04:59.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022f404e0f9093330-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.971 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:04:59.971 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:04:59.971 DEBUG:boto:Method: HEAD 2024-04-12T04:04:59.971 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:04:59.971 DEBUG:boto:Data: 2024-04-12T04:04:59.971 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.971 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:04:59.971 DEBUG:boto:Port: 8000 2024-04-12T04:04:59.971 DEBUG:boto:Params: {} 2024-04-12T04:04:59.971 DEBUG:boto:Token: None 2024-04-12T04:04:59.972 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj44 2024-04-12T04:04:59.972 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4pI0wgAEpBezGTWa/lE84zyGqdk= 2024-04-12T04:04:59.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4pI0wgAEpBezGTWa/lE84zyGqdk=', 'Content-Length': '0'} 2024-04-12T04:04:59.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab9307b0557247b3-006618b2eb-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:04:59.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:04:59.974 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:04:59.974 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:04:59.974 DEBUG:boto:Method: GET 2024-04-12T04:04:59.974 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:04:59.974 DEBUG:boto:Data: 2024-04-12T04:04:59.974 DEBUG:boto:Headers: {} 2024-04-12T04:04:59.974 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:04:59.974 DEBUG:boto:Port: 8001 2024-04-12T04:04:59.974 DEBUG:boto:Params: {} 2024-04-12T04:04:59.975 DEBUG:boto:Token: None 2024-04-12T04:04:59.975 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:04:59 GMT /dkyges-61/obj44 2024-04-12T04:04:59.975 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Usu46ihRK/WIsLgIfpN+vAUDA1Q= 2024-04-12T04:04:59.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:04:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Usu46ihRK/WIsLgIfpN+vAUDA1Q=', 'Content-Length': '0'} 2024-04-12T04:04:59.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000124958a0518e144d-006618b2eb-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:04:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.018 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:05:00.018 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:05:00.019 DEBUG:boto:Method: GET 2024-04-12T04:05:00.019 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:05:00.019 DEBUG:boto:Data: 2024-04-12T04:05:00.019 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.019 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.019 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.019 DEBUG:boto:Params: {} 2024-04-12T04:05:00.019 DEBUG:boto:Token: None 2024-04-12T04:05:00.020 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj44 2024-04-12T04:05:00.020 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Aj4R6+k4KHwD61ImIOT+oIR9gGI= 2024-04-12T04:05:00.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Aj4R6+k4KHwD61ImIOT+oIR9gGI=', 'Content-Length': '0'} 2024-04-12T04:05:00.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0bd09df92b59fb4-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.022 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.022 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.022 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:05:00.022 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:05:00.022 DEBUG:boto:Method: GET 2024-04-12T04:05:00.022 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:05:00.023 DEBUG:boto:Data: 2024-04-12T04:05:00.023 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.023 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.023 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.023 DEBUG:boto:Params: {} 2024-04-12T04:05:00.023 DEBUG:boto:Token: None 2024-04-12T04:05:00.023 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.023 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6dd4f5e1b98d3ee-006618b2ec-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.026 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.026 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.027 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.027 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:05:00.027 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:05:00.027 DEBUG:boto:Method: GET 2024-04-12T04:05:00.027 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:05:00.027 DEBUG:boto:Data: 2024-04-12T04:05:00.027 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.027 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.027 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.027 DEBUG:boto:Params: {} 2024-04-12T04:05:00.027 DEBUG:boto:Token: None 2024-04-12T04:05:00.028 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.028 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009e7a27e651bffe0-006618b2ec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.031 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:05:00.031 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:05:00.031 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:05:00.031 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.031 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.031 DEBUG:boto:Method: GET 2024-04-12T04:05:00.032 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.032 DEBUG:boto:Data: 2024-04-12T04:05:00.032 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.032 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.032 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.032 DEBUG:boto:Params: {} 2024-04-12T04:05:00.032 DEBUG:boto:Token: None 2024-04-12T04:05:00.032 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.032 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qyeCeSG12+sPtAXRRH5Pv+Fo/O8= 2024-04-12T04:05:00.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qyeCeSG12+sPtAXRRH5Pv+Fo/O8=', 'Content-Length': '0'} 2024-04-12T04:05:00.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002081f994e849c710-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.035 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:05:00.035 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:05:00.035 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.035 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.035 DEBUG:boto:Method: GET 2024-04-12T04:05:00.035 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.035 DEBUG:boto:Data: 2024-04-12T04:05:00.035 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.035 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.035 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.035 DEBUG:boto:Params: {} 2024-04-12T04:05:00.036 DEBUG:boto:Token: None 2024-04-12T04:05:00.036 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.036 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qyeCeSG12+sPtAXRRH5Pv+Fo/O8= 2024-04-12T04:05:00.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qyeCeSG12+sPtAXRRH5Pv+Fo/O8=', 'Content-Length': '0'} 2024-04-12T04:05:00.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dab7f8195d42aba-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.038 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:05:00.038 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:05:00.038 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.039 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.039 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.039 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.039 DEBUG:boto:Data: 2024-04-12T04:05:00.039 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.039 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.039 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.039 DEBUG:boto:Params: {} 2024-04-12T04:05:00.039 DEBUG:boto:Token: None 2024-04-12T04:05:00.039 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.039 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3dag2UI1u+m8nMHJG9I1Il3sVsM= 2024-04-12T04:05:00.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3dag2UI1u+m8nMHJG9I1Il3sVsM=', 'Content-Length': '0'} 2024-04-12T04:05:00.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000050606b3892c1130-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.041 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:05:00.042 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:05:00.042 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.042 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.042 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.042 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.042 DEBUG:boto:Data: 2024-04-12T04:05:00.042 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.042 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.042 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.042 DEBUG:boto:Params: {} 2024-04-12T04:05:00.042 DEBUG:boto:Token: None 2024-04-12T04:05:00.043 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.043 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3dag2UI1u+m8nMHJG9I1Il3sVsM= 2024-04-12T04:05:00.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3dag2UI1u+m8nMHJG9I1Il3sVsM=', 'Content-Length': '0'} 2024-04-12T04:05:00.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008db4f67e907458f6-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:05:00.045 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:05:00.045 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:05:00.045 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.045 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.045 DEBUG:boto:Method: GET 2024-04-12T04:05:00.045 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.045 DEBUG:boto:Data: 2024-04-12T04:05:00.045 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.045 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.046 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.046 DEBUG:boto:Params: {} 2024-04-12T04:05:00.046 DEBUG:boto:Token: None 2024-04-12T04:05:00.046 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.046 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qyeCeSG12+sPtAXRRH5Pv+Fo/O8= 2024-04-12T04:05:00.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qyeCeSG12+sPtAXRRH5Pv+Fo/O8=', 'Content-Length': '0'} 2024-04-12T04:05:00.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006961076b1cd99a43-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.048 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:05:00.048 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:05:00.048 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.048 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.048 DEBUG:boto:Method: GET 2024-04-12T04:05:00.049 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.049 DEBUG:boto:Data: 2024-04-12T04:05:00.049 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.049 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.049 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.049 DEBUG:boto:Params: {} 2024-04-12T04:05:00.049 DEBUG:boto:Token: None 2024-04-12T04:05:00.049 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:05:00.049 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qyeCeSG12+sPtAXRRH5Pv+Fo/O8= 2024-04-12T04:05:00.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qyeCeSG12+sPtAXRRH5Pv+Fo/O8=', 'Content-Length': '0'} 2024-04-12T04:05:00.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a50f2a867927d4e0-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.051 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.052 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.052 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:05:00.052 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:05:00.052 DEBUG:boto:Method: GET 2024-04-12T04:05:00.052 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:05:00.052 DEBUG:boto:Data: 2024-04-12T04:05:00.052 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.052 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.052 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.052 DEBUG:boto:Params: {} 2024-04-12T04:05:00.052 DEBUG:boto:Token: None 2024-04-12T04:05:00.053 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.053 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063c32331b16c2f61-006618b2ec-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.055 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.056 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.056 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.056 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:05:00.056 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:05:00.056 DEBUG:boto:Method: GET 2024-04-12T04:05:00.056 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:05:00.056 DEBUG:boto:Data: 2024-04-12T04:05:00.056 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.056 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.056 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.056 DEBUG:boto:Params: {} 2024-04-12T04:05:00.056 DEBUG:boto:Token: None 2024-04-12T04:05:00.057 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.057 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005110f8a39e24e382-006618b2ec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.060 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.060 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:05:00.060 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:05:00.060 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.060 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:05:00.060 DEBUG:boto:Data: 2024-04-12T04:05:00.060 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.060 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.060 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.061 DEBUG:boto:Params: {} 2024-04-12T04:05:00.061 DEBUG:boto:Token: None 2024-04-12T04:05:00.061 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj51 2024-04-12T04:05:00.061 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9t38lvSu0xpto1z2SftJODIt6n8= 2024-04-12T04:05:00.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9t38lvSu0xpto1z2SftJODIt6n8=', 'Content-Length': '0'} 2024-04-12T04:05:00.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006338c8e93d3afc5e-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.063 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:05:00.063 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:05:00.063 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.063 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:05:00.063 DEBUG:boto:Data: 2024-04-12T04:05:00.063 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.063 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.063 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.063 DEBUG:boto:Params: {} 2024-04-12T04:05:00.063 DEBUG:boto:Token: None 2024-04-12T04:05:00.064 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj51 2024-04-12T04:05:00.064 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9t38lvSu0xpto1z2SftJODIt6n8= 2024-04-12T04:05:00.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9t38lvSu0xpto1z2SftJODIt6n8=', 'Content-Length': '0'} 2024-04-12T04:05:00.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000361ca2974b7b4118-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:05:00.065 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:05:00.066 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:05:00.066 DEBUG:boto:Method: GET 2024-04-12T04:05:00.066 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:05:00.066 DEBUG:boto:Data: 2024-04-12T04:05:00.066 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.066 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.066 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.066 DEBUG:boto:Params: {} 2024-04-12T04:05:00.066 DEBUG:boto:Token: None 2024-04-12T04:05:00.066 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj51 2024-04-12T04:05:00.067 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HrgCXWZcvLT7MbJhvqzk6EHFbAo= 2024-04-12T04:05:00.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HrgCXWZcvLT7MbJhvqzk6EHFbAo=', 'Content-Length': '0'} 2024-04-12T04:05:00.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000950ff596c14ba717-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.110 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:05:00.110 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:05:00.111 DEBUG:boto:Method: GET 2024-04-12T04:05:00.111 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:05:00.111 DEBUG:boto:Data: 2024-04-12T04:05:00.111 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.111 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.111 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.111 DEBUG:boto:Params: {} 2024-04-12T04:05:00.111 DEBUG:boto:Token: None 2024-04-12T04:05:00.111 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj51 2024-04-12T04:05:00.111 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HrgCXWZcvLT7MbJhvqzk6EHFbAo= 2024-04-12T04:05:00.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HrgCXWZcvLT7MbJhvqzk6EHFbAo=', 'Content-Length': '0'} 2024-04-12T04:05:00.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c14440344a9881c2-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.114 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.114 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.114 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:05:00.114 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:05:00.114 DEBUG:boto:Method: GET 2024-04-12T04:05:00.114 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:05:00.114 DEBUG:boto:Data: 2024-04-12T04:05:00.114 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.115 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.115 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.115 DEBUG:boto:Params: {} 2024-04-12T04:05:00.115 DEBUG:boto:Token: None 2024-04-12T04:05:00.115 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.115 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8bfad253d05ac28-006618b2ec-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.118 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.118 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.118 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.118 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:05:00.118 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:05:00.118 DEBUG:boto:Method: GET 2024-04-12T04:05:00.119 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:05:00.119 DEBUG:boto:Data: 2024-04-12T04:05:00.119 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.119 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.119 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.119 DEBUG:boto:Params: {} 2024-04-12T04:05:00.119 DEBUG:boto:Token: None 2024-04-12T04:05:00.119 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.119 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f6035688d25f640-006618b2ec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.123 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:05:00.123 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:05:00.123 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:05:00.123 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.123 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.123 DEBUG:boto:Method: GET 2024-04-12T04:05:00.123 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.123 DEBUG:boto:Data: 2024-04-12T04:05:00.124 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.124 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.124 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.124 DEBUG:boto:Params: {} 2024-04-12T04:05:00.124 DEBUG:boto:Token: None 2024-04-12T04:05:00.124 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.124 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1ry6nyeCiHSmLnw2WglEsHVEVgw= 2024-04-12T04:05:00.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1ry6nyeCiHSmLnw2WglEsHVEVgw=', 'Content-Length': '0'} 2024-04-12T04:05:00.126 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d5deec80e736f9-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.126 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:05:00.126 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:05:00.126 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.127 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.127 DEBUG:boto:Method: GET 2024-04-12T04:05:00.127 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.127 DEBUG:boto:Data: 2024-04-12T04:05:00.127 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.127 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.127 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.127 DEBUG:boto:Params: {} 2024-04-12T04:05:00.127 DEBUG:boto:Token: None 2024-04-12T04:05:00.127 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.127 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1ry6nyeCiHSmLnw2WglEsHVEVgw= 2024-04-12T04:05:00.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1ry6nyeCiHSmLnw2WglEsHVEVgw=', 'Content-Length': '0'} 2024-04-12T04:05:00.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005590af13c56b3625-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.130 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:05:00.130 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:05:00.130 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.130 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.130 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.130 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.130 DEBUG:boto:Data: 2024-04-12T04:05:00.130 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.130 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.131 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.131 DEBUG:boto:Params: {} 2024-04-12T04:05:00.131 DEBUG:boto:Token: None 2024-04-12T04:05:00.131 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.131 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MzOgo7HFZCbYQSFW0oRuMyD8TFg= 2024-04-12T04:05:00.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MzOgo7HFZCbYQSFW0oRuMyD8TFg=', 'Content-Length': '0'} 2024-04-12T04:05:00.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e573512e55e3bec9-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.133 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:05:00.133 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:05:00.133 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.133 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.134 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.134 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.134 DEBUG:boto:Data: 2024-04-12T04:05:00.134 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.134 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.134 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.134 DEBUG:boto:Params: {} 2024-04-12T04:05:00.134 DEBUG:boto:Token: None 2024-04-12T04:05:00.134 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.134 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MzOgo7HFZCbYQSFW0oRuMyD8TFg= 2024-04-12T04:05:00.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MzOgo7HFZCbYQSFW0oRuMyD8TFg=', 'Content-Length': '0'} 2024-04-12T04:05:00.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbfca71ee0909e19-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:05:00.137 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:05:00.137 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:05:00.137 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.137 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.137 DEBUG:boto:Method: GET 2024-04-12T04:05:00.138 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.138 DEBUG:boto:Data: 2024-04-12T04:05:00.138 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.138 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.138 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.138 DEBUG:boto:Params: {} 2024-04-12T04:05:00.138 DEBUG:boto:Token: None 2024-04-12T04:05:00.138 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.138 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1ry6nyeCiHSmLnw2WglEsHVEVgw= 2024-04-12T04:05:00.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1ry6nyeCiHSmLnw2WglEsHVEVgw=', 'Content-Length': '0'} 2024-04-12T04:05:00.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d220c8638407fda-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.141 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:05:00.141 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:05:00.141 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.141 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.141 DEBUG:boto:Method: GET 2024-04-12T04:05:00.141 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.141 DEBUG:boto:Data: 2024-04-12T04:05:00.141 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.141 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.142 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.142 DEBUG:boto:Params: {} 2024-04-12T04:05:00.142 DEBUG:boto:Token: None 2024-04-12T04:05:00.142 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:05:00.142 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1ry6nyeCiHSmLnw2WglEsHVEVgw= 2024-04-12T04:05:00.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1ry6nyeCiHSmLnw2WglEsHVEVgw=', 'Content-Length': '0'} 2024-04-12T04:05:00.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000972ebd201e4c71dd-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.145 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.145 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.145 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:05:00.145 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:05:00.145 DEBUG:boto:Method: GET 2024-04-12T04:05:00.145 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:05:00.145 DEBUG:boto:Data: 2024-04-12T04:05:00.145 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.145 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.145 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.145 DEBUG:boto:Params: {} 2024-04-12T04:05:00.146 DEBUG:boto:Token: None 2024-04-12T04:05:00.146 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.146 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024eb1695f883f286-006618b2ec-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.148 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.149 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.149 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.149 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:05:00.149 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:05:00.149 DEBUG:boto:Method: GET 2024-04-12T04:05:00.149 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:05:00.149 DEBUG:boto:Data: 2024-04-12T04:05:00.149 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.149 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.149 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.150 DEBUG:boto:Params: {} 2024-04-12T04:05:00.150 DEBUG:boto:Token: None 2024-04-12T04:05:00.150 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.150 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d54a10768dd8d105-006618b2ec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.153 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.153 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:05:00.154 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:05:00.154 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.154 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:05:00.154 DEBUG:boto:Data: 2024-04-12T04:05:00.154 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.154 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.154 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.154 DEBUG:boto:Params: {} 2024-04-12T04:05:00.154 DEBUG:boto:Token: None 2024-04-12T04:05:00.155 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj55 2024-04-12T04:05:00.155 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hct8klvvkNAD8NZ+hlOf/meU8Z0= 2024-04-12T04:05:00.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hct8klvvkNAD8NZ+hlOf/meU8Z0=', 'Content-Length': '0'} 2024-04-12T04:05:00.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1fa1b376e3d0ee2-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.157 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:05:00.157 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:05:00.157 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.157 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:05:00.157 DEBUG:boto:Data: 2024-04-12T04:05:00.157 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.157 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.157 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.157 DEBUG:boto:Params: {} 2024-04-12T04:05:00.158 DEBUG:boto:Token: None 2024-04-12T04:05:00.158 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj55 2024-04-12T04:05:00.158 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hct8klvvkNAD8NZ+hlOf/meU8Z0= 2024-04-12T04:05:00.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hct8klvvkNAD8NZ+hlOf/meU8Z0=', 'Content-Length': '0'} 2024-04-12T04:05:00.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000525e860d2decc1d7-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:05:00.160 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:05:00.160 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:05:00.160 DEBUG:boto:Method: GET 2024-04-12T04:05:00.160 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:05:00.160 DEBUG:boto:Data: 2024-04-12T04:05:00.161 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.161 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.161 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.161 DEBUG:boto:Params: {} 2024-04-12T04:05:00.161 DEBUG:boto:Token: None 2024-04-12T04:05:00.161 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj55 2024-04-12T04:05:00.161 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G9YBnQKsvHxOYkwwOM7gaS5Dd2k= 2024-04-12T04:05:00.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G9YBnQKsvHxOYkwwOM7gaS5Dd2k=', 'Content-Length': '0'} 2024-04-12T04:05:00.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2250589d08ba9c0-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.163 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:05:00.163 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:05:00.164 DEBUG:boto:Method: GET 2024-04-12T04:05:00.164 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:05:00.164 DEBUG:boto:Data: 2024-04-12T04:05:00.164 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.164 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.164 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.164 DEBUG:boto:Params: {} 2024-04-12T04:05:00.164 DEBUG:boto:Token: None 2024-04-12T04:05:00.164 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj55 2024-04-12T04:05:00.164 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G9YBnQKsvHxOYkwwOM7gaS5Dd2k= 2024-04-12T04:05:00.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G9YBnQKsvHxOYkwwOM7gaS5Dd2k=', 'Content-Length': '0'} 2024-04-12T04:05:00.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000081a7aa46469eb03-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.167 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.167 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.167 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:05:00.167 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:05:00.167 DEBUG:boto:Method: GET 2024-04-12T04:05:00.167 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:05:00.167 DEBUG:boto:Data: 2024-04-12T04:05:00.167 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.167 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.167 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.167 DEBUG:boto:Params: {} 2024-04-12T04:05:00.168 DEBUG:boto:Token: None 2024-04-12T04:05:00.168 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.168 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9f081bd11eca7ab-006618b2ec-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.170 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.171 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.171 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.171 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:05:00.171 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:05:00.171 DEBUG:boto:Method: GET 2024-04-12T04:05:00.171 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:05:00.171 DEBUG:boto:Data: 2024-04-12T04:05:00.172 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.172 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.172 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.172 DEBUG:boto:Params: {} 2024-04-12T04:05:00.172 DEBUG:boto:Token: None 2024-04-12T04:05:00.172 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.172 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ac4759d64fb4dc9-006618b2ec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.176 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:05:00.176 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:05:00.176 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:05:00.176 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.176 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.177 DEBUG:boto:Method: GET 2024-04-12T04:05:00.177 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.177 DEBUG:boto:Data: 2024-04-12T04:05:00.177 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.177 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.177 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.177 DEBUG:boto:Params: {} 2024-04-12T04:05:00.177 DEBUG:boto:Token: None 2024-04-12T04:05:00.177 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.177 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t3JUy03y5epinGjslHaoK2AeQlA= 2024-04-12T04:05:00.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t3JUy03y5epinGjslHaoK2AeQlA=', 'Content-Length': '0'} 2024-04-12T04:05:00.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef811295c7d5e92a-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.226 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:05:00.226 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:05:00.227 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.227 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.227 DEBUG:boto:Method: GET 2024-04-12T04:05:00.227 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.227 DEBUG:boto:Data: 2024-04-12T04:05:00.227 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.227 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.227 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.227 DEBUG:boto:Params: {} 2024-04-12T04:05:00.227 DEBUG:boto:Token: None 2024-04-12T04:05:00.227 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.227 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t3JUy03y5epinGjslHaoK2AeQlA= 2024-04-12T04:05:00.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t3JUy03y5epinGjslHaoK2AeQlA=', 'Content-Length': '0'} 2024-04-12T04:05:00.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fa61208413d0913-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.230 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:05:00.230 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:05:00.230 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.230 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.230 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.230 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.231 DEBUG:boto:Data: 2024-04-12T04:05:00.231 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.231 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.231 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.231 DEBUG:boto:Params: {} 2024-04-12T04:05:00.231 DEBUG:boto:Token: None 2024-04-12T04:05:00.231 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.231 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUnDpM9Xph3BmrCJzA59NtkXpc8= 2024-04-12T04:05:00.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUnDpM9Xph3BmrCJzA59NtkXpc8=', 'Content-Length': '0'} 2024-04-12T04:05:00.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dae5bb76d57224a-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.233 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:05:00.233 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:05:00.234 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.234 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.234 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.234 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.234 DEBUG:boto:Data: 2024-04-12T04:05:00.234 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.234 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.234 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.234 DEBUG:boto:Params: {} 2024-04-12T04:05:00.234 DEBUG:boto:Token: None 2024-04-12T04:05:00.235 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.235 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUnDpM9Xph3BmrCJzA59NtkXpc8= 2024-04-12T04:05:00.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUnDpM9Xph3BmrCJzA59NtkXpc8=', 'Content-Length': '0'} 2024-04-12T04:05:00.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e0e84dc1e2f9db8-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:05:00.237 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:05:00.237 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:05:00.237 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.237 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.238 DEBUG:boto:Method: GET 2024-04-12T04:05:00.238 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.238 DEBUG:boto:Data: 2024-04-12T04:05:00.238 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.238 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.238 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.238 DEBUG:boto:Params: {} 2024-04-12T04:05:00.238 DEBUG:boto:Token: None 2024-04-12T04:05:00.238 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.238 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t3JUy03y5epinGjslHaoK2AeQlA= 2024-04-12T04:05:00.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t3JUy03y5epinGjslHaoK2AeQlA=', 'Content-Length': '0'} 2024-04-12T04:05:00.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053317359da53c1f2-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.241 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:05:00.241 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:05:00.241 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.241 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.241 DEBUG:boto:Method: GET 2024-04-12T04:05:00.241 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.241 DEBUG:boto:Data: 2024-04-12T04:05:00.241 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.242 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.242 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.242 DEBUG:boto:Params: {} 2024-04-12T04:05:00.242 DEBUG:boto:Token: None 2024-04-12T04:05:00.242 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:05:00.242 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t3JUy03y5epinGjslHaoK2AeQlA= 2024-04-12T04:05:00.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t3JUy03y5epinGjslHaoK2AeQlA=', 'Content-Length': '0'} 2024-04-12T04:05:00.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b69e0251a4e591c-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.246 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.246 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.246 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:05:00.246 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:05:00.246 DEBUG:boto:Method: GET 2024-04-12T04:05:00.246 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:05:00.247 DEBUG:boto:Data: 2024-04-12T04:05:00.247 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.247 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.247 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.247 DEBUG:boto:Params: {} 2024-04-12T04:05:00.247 DEBUG:boto:Token: None 2024-04-12T04:05:00.247 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.247 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000855a76c7d6f0c07e-006618b2ec-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.250 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.250 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.250 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.251 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:05:00.251 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:05:00.251 DEBUG:boto:Method: GET 2024-04-12T04:05:00.251 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:05:00.251 DEBUG:boto:Data: 2024-04-12T04:05:00.251 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.251 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.251 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.251 DEBUG:boto:Params: {} 2024-04-12T04:05:00.251 DEBUG:boto:Token: None 2024-04-12T04:05:00.251 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.252 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086c974010c9a6116-006618b2ec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.255 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.255 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:05:00.255 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:05:00.255 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.255 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:05:00.255 DEBUG:boto:Data: 2024-04-12T04:05:00.255 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.256 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.256 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.256 DEBUG:boto:Params: {} 2024-04-12T04:05:00.256 DEBUG:boto:Token: None 2024-04-12T04:05:00.256 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj57 2024-04-12T04:05:00.256 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:inF5P7O3W4v966GVKUR/IwfbfCA= 2024-04-12T04:05:00.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:inF5P7O3W4v966GVKUR/IwfbfCA=', 'Content-Length': '0'} 2024-04-12T04:05:00.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b3e1d7a490ba982-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.258 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:05:00.258 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:05:00.258 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.258 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:05:00.259 DEBUG:boto:Data: 2024-04-12T04:05:00.259 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.259 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.259 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.259 DEBUG:boto:Params: {} 2024-04-12T04:05:00.259 DEBUG:boto:Token: None 2024-04-12T04:05:00.259 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj57 2024-04-12T04:05:00.259 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:inF5P7O3W4v966GVKUR/IwfbfCA= 2024-04-12T04:05:00.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:inF5P7O3W4v966GVKUR/IwfbfCA=', 'Content-Length': '0'} 2024-04-12T04:05:00.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0ecd1b2f3c64b79-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:05:00.261 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:05:00.261 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:05:00.261 DEBUG:boto:Method: GET 2024-04-12T04:05:00.261 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:05:00.261 DEBUG:boto:Data: 2024-04-12T04:05:00.262 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.262 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.262 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.262 DEBUG:boto:Params: {} 2024-04-12T04:05:00.262 DEBUG:boto:Token: None 2024-04-12T04:05:00.262 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj57 2024-04-12T04:05:00.262 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XItlL53Y5B4wdafUWMkjeM8TRhM= 2024-04-12T04:05:00.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XItlL53Y5B4wdafUWMkjeM8TRhM=', 'Content-Length': '0'} 2024-04-12T04:05:00.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067dd0d8c23d6413f-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.264 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:05:00.264 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:05:00.265 DEBUG:boto:Method: GET 2024-04-12T04:05:00.265 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:05:00.265 DEBUG:boto:Data: 2024-04-12T04:05:00.265 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.265 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.265 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.265 DEBUG:boto:Params: {} 2024-04-12T04:05:00.265 DEBUG:boto:Token: None 2024-04-12T04:05:00.265 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj57 2024-04-12T04:05:00.265 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XItlL53Y5B4wdafUWMkjeM8TRhM= 2024-04-12T04:05:00.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XItlL53Y5B4wdafUWMkjeM8TRhM=', 'Content-Length': '0'} 2024-04-12T04:05:00.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba5ad00556b5be6e-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.310 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.311 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.311 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:05:00.311 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:05:00.311 DEBUG:boto:Method: GET 2024-04-12T04:05:00.311 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:05:00.311 DEBUG:boto:Data: 2024-04-12T04:05:00.311 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.311 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.311 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.311 DEBUG:boto:Params: {} 2024-04-12T04:05:00.311 DEBUG:boto:Token: None 2024-04-12T04:05:00.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.312 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4340bbcb910e842-006618b2ec-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.315 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.315 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.315 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.315 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:05:00.315 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:05:00.316 DEBUG:boto:Method: GET 2024-04-12T04:05:00.316 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:05:00.316 DEBUG:boto:Data: 2024-04-12T04:05:00.316 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.316 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.316 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.316 DEBUG:boto:Params: {} 2024-04-12T04:05:00.316 DEBUG:boto:Token: None 2024-04-12T04:05:00.316 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.316 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.320 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f43d33f5eb6493a-006618b2ec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.320 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:05:00.321 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:05:00.321 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:05:00.321 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.321 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.321 DEBUG:boto:Method: GET 2024-04-12T04:05:00.322 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.322 DEBUG:boto:Data: 2024-04-12T04:05:00.322 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.322 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.322 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.322 DEBUG:boto:Params: {} 2024-04-12T04:05:00.322 DEBUG:boto:Token: None 2024-04-12T04:05:00.322 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.322 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RJHhbF/XVxpsYpwZ9ISrI4rcf+Y= 2024-04-12T04:05:00.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RJHhbF/XVxpsYpwZ9ISrI4rcf+Y=', 'Content-Length': '0'} 2024-04-12T04:05:00.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098ccb9756e1724bd-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.324 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:05:00.325 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:05:00.325 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.325 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.325 DEBUG:boto:Method: GET 2024-04-12T04:05:00.325 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.325 DEBUG:boto:Data: 2024-04-12T04:05:00.325 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.325 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.325 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.325 DEBUG:boto:Params: {} 2024-04-12T04:05:00.325 DEBUG:boto:Token: None 2024-04-12T04:05:00.326 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.326 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RJHhbF/XVxpsYpwZ9ISrI4rcf+Y= 2024-04-12T04:05:00.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RJHhbF/XVxpsYpwZ9ISrI4rcf+Y=', 'Content-Length': '0'} 2024-04-12T04:05:00.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbed47953e3affab-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.328 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:05:00.328 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:05:00.328 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.328 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.328 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.328 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.329 DEBUG:boto:Data: 2024-04-12T04:05:00.329 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.329 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.329 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.329 DEBUG:boto:Params: {} 2024-04-12T04:05:00.329 DEBUG:boto:Token: None 2024-04-12T04:05:00.329 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.329 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NGOyEBHtk4QoxqRyvNhHPpnFKT8= 2024-04-12T04:05:00.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NGOyEBHtk4QoxqRyvNhHPpnFKT8=', 'Content-Length': '0'} 2024-04-12T04:05:00.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016ba7f9f8137db91-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.331 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:05:00.332 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:05:00.332 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.332 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.332 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.332 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.332 DEBUG:boto:Data: 2024-04-12T04:05:00.332 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.332 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.332 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.332 DEBUG:boto:Params: {} 2024-04-12T04:05:00.332 DEBUG:boto:Token: None 2024-04-12T04:05:00.332 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.333 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NGOyEBHtk4QoxqRyvNhHPpnFKT8= 2024-04-12T04:05:00.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NGOyEBHtk4QoxqRyvNhHPpnFKT8=', 'Content-Length': '0'} 2024-04-12T04:05:00.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060033eadea049160-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:05:00.335 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:05:00.335 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:05:00.335 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.335 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.335 DEBUG:boto:Method: GET 2024-04-12T04:05:00.335 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.335 DEBUG:boto:Data: 2024-04-12T04:05:00.335 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.335 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.336 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.336 DEBUG:boto:Params: {} 2024-04-12T04:05:00.336 DEBUG:boto:Token: None 2024-04-12T04:05:00.336 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.336 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RJHhbF/XVxpsYpwZ9ISrI4rcf+Y= 2024-04-12T04:05:00.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RJHhbF/XVxpsYpwZ9ISrI4rcf+Y=', 'Content-Length': '0'} 2024-04-12T04:05:00.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000130d0a722e1db024-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.378 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:05:00.379 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:05:00.379 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.379 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.379 DEBUG:boto:Method: GET 2024-04-12T04:05:00.379 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.379 DEBUG:boto:Data: 2024-04-12T04:05:00.379 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.379 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.379 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.379 DEBUG:boto:Params: {} 2024-04-12T04:05:00.380 DEBUG:boto:Token: None 2024-04-12T04:05:00.380 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:05:00.380 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RJHhbF/XVxpsYpwZ9ISrI4rcf+Y= 2024-04-12T04:05:00.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RJHhbF/XVxpsYpwZ9ISrI4rcf+Y=', 'Content-Length': '0'} 2024-04-12T04:05:00.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000858e85098948a5c8-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.382 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.382 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.383 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:05:00.383 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:05:00.383 DEBUG:boto:Method: GET 2024-04-12T04:05:00.383 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:05:00.383 DEBUG:boto:Data: 2024-04-12T04:05:00.383 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.383 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.383 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.383 DEBUG:boto:Params: {} 2024-04-12T04:05:00.383 DEBUG:boto:Token: None 2024-04-12T04:05:00.383 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.384 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.386 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a2a43ac30da3872-006618b2ec-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.386 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.386 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.387 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.387 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:05:00.387 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:05:00.387 DEBUG:boto:Method: GET 2024-04-12T04:05:00.387 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:05:00.387 DEBUG:boto:Data: 2024-04-12T04:05:00.387 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.387 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.387 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.387 DEBUG:boto:Params: {} 2024-04-12T04:05:00.387 DEBUG:boto:Token: None 2024-04-12T04:05:00.388 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.388 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045553bd828442226-006618b2ec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.391 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.391 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:05:00.391 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:05:00.391 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.391 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:05:00.392 DEBUG:boto:Data: 2024-04-12T04:05:00.392 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.392 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.392 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.392 DEBUG:boto:Params: {} 2024-04-12T04:05:00.392 DEBUG:boto:Token: None 2024-04-12T04:05:00.392 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj68 2024-04-12T04:05:00.392 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d8Ptdy7zawb2qC7Zu4kH2AxaWUA= 2024-04-12T04:05:00.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d8Ptdy7zawb2qC7Zu4kH2AxaWUA=', 'Content-Length': '0'} 2024-04-12T04:05:00.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009abf4f2f20d49c81-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.394 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:05:00.395 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:05:00.395 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.395 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:05:00.395 DEBUG:boto:Data: 2024-04-12T04:05:00.395 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.395 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.395 DEBUG:boto:Params: {} 2024-04-12T04:05:00.395 DEBUG:boto:Token: None 2024-04-12T04:05:00.395 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj68 2024-04-12T04:05:00.395 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d8Ptdy7zawb2qC7Zu4kH2AxaWUA= 2024-04-12T04:05:00.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d8Ptdy7zawb2qC7Zu4kH2AxaWUA=', 'Content-Length': '0'} 2024-04-12T04:05:00.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0a1649e652797ce-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:05:00.398 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:05:00.398 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:05:00.398 DEBUG:boto:Method: GET 2024-04-12T04:05:00.398 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:05:00.398 DEBUG:boto:Data: 2024-04-12T04:05:00.398 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.398 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.398 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.399 DEBUG:boto:Params: {} 2024-04-12T04:05:00.399 DEBUG:boto:Token: None 2024-04-12T04:05:00.399 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj68 2024-04-12T04:05:00.399 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zHxL4f5B9UMHhKgKIZQnokKBdqw= 2024-04-12T04:05:00.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zHxL4f5B9UMHhKgKIZQnokKBdqw=', 'Content-Length': '0'} 2024-04-12T04:05:00.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005862eac2aabb06a0-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.401 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:05:00.401 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:05:00.401 DEBUG:boto:Method: GET 2024-04-12T04:05:00.401 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:05:00.402 DEBUG:boto:Data: 2024-04-12T04:05:00.402 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.402 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.402 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.402 DEBUG:boto:Params: {} 2024-04-12T04:05:00.402 DEBUG:boto:Token: None 2024-04-12T04:05:00.402 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj68 2024-04-12T04:05:00.402 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zHxL4f5B9UMHhKgKIZQnokKBdqw= 2024-04-12T04:05:00.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zHxL4f5B9UMHhKgKIZQnokKBdqw=', 'Content-Length': '0'} 2024-04-12T04:05:00.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000124de5bda714b5b3-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.446 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.446 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.447 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:05:00.447 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:05:00.447 DEBUG:boto:Method: GET 2024-04-12T04:05:00.447 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:05:00.447 DEBUG:boto:Data: 2024-04-12T04:05:00.447 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.447 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.447 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.447 DEBUG:boto:Params: {} 2024-04-12T04:05:00.447 DEBUG:boto:Token: None 2024-04-12T04:05:00.447 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.447 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f3613d851cb4066-006618b2ec-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.451 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.451 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.451 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.451 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:05:00.451 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:05:00.451 DEBUG:boto:Method: GET 2024-04-12T04:05:00.451 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:05:00.451 DEBUG:boto:Data: 2024-04-12T04:05:00.452 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.452 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.452 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.452 DEBUG:boto:Params: {} 2024-04-12T04:05:00.452 DEBUG:boto:Token: None 2024-04-12T04:05:00.452 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.452 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000548ec1517e31f363-006618b2ec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.455 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:05:00.455 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:05:00.456 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:05:00.456 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.456 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.456 DEBUG:boto:Method: GET 2024-04-12T04:05:00.456 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.456 DEBUG:boto:Data: 2024-04-12T04:05:00.456 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.456 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.456 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.456 DEBUG:boto:Params: {} 2024-04-12T04:05:00.456 DEBUG:boto:Token: None 2024-04-12T04:05:00.457 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.457 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kjUnr08NqGukN8WVrJznVcVKXAg= 2024-04-12T04:05:00.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kjUnr08NqGukN8WVrJznVcVKXAg=', 'Content-Length': '0'} 2024-04-12T04:05:00.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c06a6f3815c41e0-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.459 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:05:00.459 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:05:00.459 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.459 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.459 DEBUG:boto:Method: GET 2024-04-12T04:05:00.459 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.460 DEBUG:boto:Data: 2024-04-12T04:05:00.460 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.460 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.460 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.460 DEBUG:boto:Params: {} 2024-04-12T04:05:00.460 DEBUG:boto:Token: None 2024-04-12T04:05:00.460 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.460 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kjUnr08NqGukN8WVrJznVcVKXAg= 2024-04-12T04:05:00.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kjUnr08NqGukN8WVrJznVcVKXAg=', 'Content-Length': '0'} 2024-04-12T04:05:00.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc88f1b8934a590e-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.462 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:05:00.463 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:05:00.463 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.463 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.463 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.463 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.463 DEBUG:boto:Data: 2024-04-12T04:05:00.463 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.463 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.463 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.463 DEBUG:boto:Params: {} 2024-04-12T04:05:00.463 DEBUG:boto:Token: None 2024-04-12T04:05:00.463 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.464 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WpiyHlD3AIT0ayUQlYtMHobyc00= 2024-04-12T04:05:00.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WpiyHlD3AIT0ayUQlYtMHobyc00=', 'Content-Length': '0'} 2024-04-12T04:05:00.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa3b47db0aea921b-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.465 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:05:00.466 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:05:00.466 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.466 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.466 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.466 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.466 DEBUG:boto:Data: 2024-04-12T04:05:00.466 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.466 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.466 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.466 DEBUG:boto:Params: {} 2024-04-12T04:05:00.466 DEBUG:boto:Token: None 2024-04-12T04:05:00.466 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.467 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WpiyHlD3AIT0ayUQlYtMHobyc00= 2024-04-12T04:05:00.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WpiyHlD3AIT0ayUQlYtMHobyc00=', 'Content-Length': '0'} 2024-04-12T04:05:00.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7ba81bf6c513917-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:05:00.469 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:05:00.469 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:05:00.469 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.469 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.469 DEBUG:boto:Method: GET 2024-04-12T04:05:00.469 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.469 DEBUG:boto:Data: 2024-04-12T04:05:00.469 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.469 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.469 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.469 DEBUG:boto:Params: {} 2024-04-12T04:05:00.470 DEBUG:boto:Token: None 2024-04-12T04:05:00.470 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.470 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kjUnr08NqGukN8WVrJznVcVKXAg= 2024-04-12T04:05:00.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kjUnr08NqGukN8WVrJznVcVKXAg=', 'Content-Length': '0'} 2024-04-12T04:05:00.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c425c52206d45fc-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.514 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:05:00.514 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:05:00.514 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.515 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.515 DEBUG:boto:Method: GET 2024-04-12T04:05:00.515 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.515 DEBUG:boto:Data: 2024-04-12T04:05:00.515 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.515 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.515 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.515 DEBUG:boto:Params: {} 2024-04-12T04:05:00.515 DEBUG:boto:Token: None 2024-04-12T04:05:00.515 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:05:00.515 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kjUnr08NqGukN8WVrJznVcVKXAg= 2024-04-12T04:05:00.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kjUnr08NqGukN8WVrJznVcVKXAg=', 'Content-Length': '0'} 2024-04-12T04:05:00.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039ca61a5c478ea8f-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.517 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.518 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.518 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:05:00.518 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:05:00.518 DEBUG:boto:Method: GET 2024-04-12T04:05:00.518 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:05:00.518 DEBUG:boto:Data: 2024-04-12T04:05:00.518 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.518 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.518 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.518 DEBUG:boto:Params: {} 2024-04-12T04:05:00.518 DEBUG:boto:Token: None 2024-04-12T04:05:00.519 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.519 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095910545f1a0bd31-006618b2ec-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.521 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.522 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.522 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.522 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:05:00.522 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:05:00.522 DEBUG:boto:Method: GET 2024-04-12T04:05:00.522 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:05:00.522 DEBUG:boto:Data: 2024-04-12T04:05:00.522 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.522 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.522 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.523 DEBUG:boto:Params: {} 2024-04-12T04:05:00.523 DEBUG:boto:Token: None 2024-04-12T04:05:00.523 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.523 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f578ba733c52cdd0-006618b2ec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.526 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.527 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:05:00.527 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:05:00.527 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.527 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:05:00.527 DEBUG:boto:Data: 2024-04-12T04:05:00.527 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.527 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.527 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.527 DEBUG:boto:Params: {} 2024-04-12T04:05:00.528 DEBUG:boto:Token: None 2024-04-12T04:05:00.528 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj76 2024-04-12T04:05:00.528 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FFP8KHCzg04gDXaMpiSVT252YfE= 2024-04-12T04:05:00.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FFP8KHCzg04gDXaMpiSVT252YfE=', 'Content-Length': '0'} 2024-04-12T04:05:00.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aba055fe285c92a7-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.530 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:05:00.530 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:05:00.530 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.530 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:05:00.530 DEBUG:boto:Data: 2024-04-12T04:05:00.530 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.530 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.530 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.530 DEBUG:boto:Params: {} 2024-04-12T04:05:00.530 DEBUG:boto:Token: None 2024-04-12T04:05:00.531 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj76 2024-04-12T04:05:00.531 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FFP8KHCzg04gDXaMpiSVT252YfE= 2024-04-12T04:05:00.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FFP8KHCzg04gDXaMpiSVT252YfE=', 'Content-Length': '0'} 2024-04-12T04:05:00.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009968a23ee3478c1c-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:05:00.533 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:05:00.533 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:05:00.533 DEBUG:boto:Method: GET 2024-04-12T04:05:00.533 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:05:00.533 DEBUG:boto:Data: 2024-04-12T04:05:00.533 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.534 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.534 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.534 DEBUG:boto:Params: {} 2024-04-12T04:05:00.534 DEBUG:boto:Token: None 2024-04-12T04:05:00.534 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj76 2024-04-12T04:05:00.534 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNNgnj6KUpLXmSn4XoRUzvoWIGI= 2024-04-12T04:05:00.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNNgnj6KUpLXmSn4XoRUzvoWIGI=', 'Content-Length': '0'} 2024-04-12T04:05:00.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e1d0a2630b869e-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.536 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:05:00.536 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:05:00.537 DEBUG:boto:Method: GET 2024-04-12T04:05:00.537 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:05:00.537 DEBUG:boto:Data: 2024-04-12T04:05:00.537 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.537 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.537 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.537 DEBUG:boto:Params: {} 2024-04-12T04:05:00.537 DEBUG:boto:Token: None 2024-04-12T04:05:00.537 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj76 2024-04-12T04:05:00.537 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gNNgnj6KUpLXmSn4XoRUzvoWIGI= 2024-04-12T04:05:00.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gNNgnj6KUpLXmSn4XoRUzvoWIGI=', 'Content-Length': '0'} 2024-04-12T04:05:00.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdb67bc9b81af62d-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.582 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.582 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.583 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:05:00.583 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:05:00.583 DEBUG:boto:Method: GET 2024-04-12T04:05:00.583 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:05:00.583 DEBUG:boto:Data: 2024-04-12T04:05:00.583 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.583 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.583 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.583 DEBUG:boto:Params: {} 2024-04-12T04:05:00.583 DEBUG:boto:Token: None 2024-04-12T04:05:00.583 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.583 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc15123fa671db35-006618b2ec-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.586 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.587 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.587 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.587 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:05:00.587 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:05:00.587 DEBUG:boto:Method: GET 2024-04-12T04:05:00.587 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:05:00.587 DEBUG:boto:Data: 2024-04-12T04:05:00.587 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.587 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.587 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.588 DEBUG:boto:Params: {} 2024-04-12T04:05:00.588 DEBUG:boto:Token: None 2024-04-12T04:05:00.588 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.588 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000994bf8493005ca88-006618b2ec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.591 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:05:00.591 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:05:00.591 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:05:00.592 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.592 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.592 DEBUG:boto:Method: GET 2024-04-12T04:05:00.592 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.592 DEBUG:boto:Data: 2024-04-12T04:05:00.592 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.592 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.592 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.592 DEBUG:boto:Params: {} 2024-04-12T04:05:00.592 DEBUG:boto:Token: None 2024-04-12T04:05:00.592 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.592 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a6D1ZWMl60pYgZR6Hy18d+HjlhU= 2024-04-12T04:05:00.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a6D1ZWMl60pYgZR6Hy18d+HjlhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e528b0b210384fd-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.595 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:05:00.595 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:05:00.595 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.595 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.595 DEBUG:boto:Method: GET 2024-04-12T04:05:00.595 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.595 DEBUG:boto:Data: 2024-04-12T04:05:00.595 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.595 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.595 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.595 DEBUG:boto:Params: {} 2024-04-12T04:05:00.596 DEBUG:boto:Token: None 2024-04-12T04:05:00.596 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.596 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a6D1ZWMl60pYgZR6Hy18d+HjlhU= 2024-04-12T04:05:00.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a6D1ZWMl60pYgZR6Hy18d+HjlhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6cfdf7997758482-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.598 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:05:00.598 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:05:00.598 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.598 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.599 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.599 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.599 DEBUG:boto:Data: 2024-04-12T04:05:00.599 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.599 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.599 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.599 DEBUG:boto:Params: {} 2024-04-12T04:05:00.599 DEBUG:boto:Token: None 2024-04-12T04:05:00.599 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.599 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PhHiyDxYKnr7tC03rRqSqQoxlns= 2024-04-12T04:05:00.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PhHiyDxYKnr7tC03rRqSqQoxlns=', 'Content-Length': '0'} 2024-04-12T04:05:00.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c47c1cf14eb30ab1-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.601 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:05:00.602 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:05:00.602 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.602 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.602 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.602 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.602 DEBUG:boto:Data: 2024-04-12T04:05:00.602 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.602 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.602 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.602 DEBUG:boto:Params: {} 2024-04-12T04:05:00.602 DEBUG:boto:Token: None 2024-04-12T04:05:00.603 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.603 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PhHiyDxYKnr7tC03rRqSqQoxlns= 2024-04-12T04:05:00.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PhHiyDxYKnr7tC03rRqSqQoxlns=', 'Content-Length': '0'} 2024-04-12T04:05:00.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006af9e294cb2fb5b-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:05:00.605 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:05:00.605 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:05:00.605 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.605 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.605 DEBUG:boto:Method: GET 2024-04-12T04:05:00.605 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.605 DEBUG:boto:Data: 2024-04-12T04:05:00.605 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.605 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.605 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.605 DEBUG:boto:Params: {} 2024-04-12T04:05:00.606 DEBUG:boto:Token: None 2024-04-12T04:05:00.606 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.606 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a6D1ZWMl60pYgZR6Hy18d+HjlhU= 2024-04-12T04:05:00.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a6D1ZWMl60pYgZR6Hy18d+HjlhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000622c69399ab08410-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.650 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:05:00.650 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:05:00.650 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.651 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.651 DEBUG:boto:Method: GET 2024-04-12T04:05:00.651 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.651 DEBUG:boto:Data: 2024-04-12T04:05:00.651 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.651 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.651 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.651 DEBUG:boto:Params: {} 2024-04-12T04:05:00.651 DEBUG:boto:Token: None 2024-04-12T04:05:00.651 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:05:00.652 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a6D1ZWMl60pYgZR6Hy18d+HjlhU= 2024-04-12T04:05:00.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a6D1ZWMl60pYgZR6Hy18d+HjlhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003822c29b83a29053-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.654 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.654 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.654 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:05:00.654 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:05:00.654 DEBUG:boto:Method: GET 2024-04-12T04:05:00.654 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:05:00.654 DEBUG:boto:Data: 2024-04-12T04:05:00.654 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.654 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.654 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.655 DEBUG:boto:Params: {} 2024-04-12T04:05:00.655 DEBUG:boto:Token: None 2024-04-12T04:05:00.655 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.655 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb5848635492a49e-006618b2ec-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.657 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.658 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.658 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.658 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:05:00.658 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:05:00.658 DEBUG:boto:Method: GET 2024-04-12T04:05:00.658 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:05:00.658 DEBUG:boto:Data: 2024-04-12T04:05:00.658 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.659 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.659 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.659 DEBUG:boto:Params: {} 2024-04-12T04:05:00.659 DEBUG:boto:Token: None 2024-04-12T04:05:00.659 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.659 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6a07935c86d28a9-006618b2ec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.662 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.663 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:05:00.663 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:05:00.663 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.663 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:05:00.663 DEBUG:boto:Data: 2024-04-12T04:05:00.663 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.663 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.663 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.663 DEBUG:boto:Params: {} 2024-04-12T04:05:00.663 DEBUG:boto:Token: None 2024-04-12T04:05:00.663 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj85 2024-04-12T04:05:00.664 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:E7ywM//v+AkOhKUn//GMsbVSSV8= 2024-04-12T04:05:00.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:E7ywM//v+AkOhKUn//GMsbVSSV8=', 'Content-Length': '0'} 2024-04-12T04:05:00.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065583e19f34af1b6-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.666 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:05:00.666 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:05:00.666 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.666 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:05:00.666 DEBUG:boto:Data: 2024-04-12T04:05:00.666 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.666 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.666 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.666 DEBUG:boto:Params: {} 2024-04-12T04:05:00.666 DEBUG:boto:Token: None 2024-04-12T04:05:00.666 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj85 2024-04-12T04:05:00.667 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:E7ywM//v+AkOhKUn//GMsbVSSV8= 2024-04-12T04:05:00.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:E7ywM//v+AkOhKUn//GMsbVSSV8=', 'Content-Length': '0'} 2024-04-12T04:05:00.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d7c361c959ae877-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:05:00.669 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:05:00.669 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:05:00.669 DEBUG:boto:Method: GET 2024-04-12T04:05:00.669 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:05:00.669 DEBUG:boto:Data: 2024-04-12T04:05:00.669 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.669 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.669 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.669 DEBUG:boto:Params: {} 2024-04-12T04:05:00.669 DEBUG:boto:Token: None 2024-04-12T04:05:00.669 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj85 2024-04-12T04:05:00.669 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VhJ770ECnT5UhJHWaOcylKKmg8s= 2024-04-12T04:05:00.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VhJ770ECnT5UhJHWaOcylKKmg8s=', 'Content-Length': '0'} 2024-04-12T04:05:00.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000289c47fb3602ec81-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.672 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:05:00.672 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:05:00.672 DEBUG:boto:Method: GET 2024-04-12T04:05:00.672 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:05:00.672 DEBUG:boto:Data: 2024-04-12T04:05:00.672 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.672 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.672 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.672 DEBUG:boto:Params: {} 2024-04-12T04:05:00.672 DEBUG:boto:Token: None 2024-04-12T04:05:00.673 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj85 2024-04-12T04:05:00.673 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VhJ770ECnT5UhJHWaOcylKKmg8s= 2024-04-12T04:05:00.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VhJ770ECnT5UhJHWaOcylKKmg8s=', 'Content-Length': '0'} 2024-04-12T04:05:00.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5cc4d4ed8a131a0-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.718 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.719 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.719 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:05:00.719 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:05:00.719 DEBUG:boto:Method: GET 2024-04-12T04:05:00.719 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:05:00.719 DEBUG:boto:Data: 2024-04-12T04:05:00.719 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.719 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.719 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.719 DEBUG:boto:Params: {} 2024-04-12T04:05:00.719 DEBUG:boto:Token: None 2024-04-12T04:05:00.720 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.720 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016b644b5d5d9578e-006618b2ec-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.722 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.722 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.723 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.723 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:05:00.723 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:05:00.723 DEBUG:boto:Method: GET 2024-04-12T04:05:00.723 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:05:00.723 DEBUG:boto:Data: 2024-04-12T04:05:00.723 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.723 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.723 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.723 DEBUG:boto:Params: {} 2024-04-12T04:05:00.723 DEBUG:boto:Token: None 2024-04-12T04:05:00.724 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.724 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.727 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aee38044f8692386-006618b2ec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.727 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:05:00.728 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:05:00.728 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:05:00.728 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.728 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.728 DEBUG:boto:Method: GET 2024-04-12T04:05:00.728 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.728 DEBUG:boto:Data: 2024-04-12T04:05:00.728 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.728 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.728 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.728 DEBUG:boto:Params: {} 2024-04-12T04:05:00.728 DEBUG:boto:Token: None 2024-04-12T04:05:00.729 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.729 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZwhVOXHdpHu6Vv5uL6K/z9HQdDs= 2024-04-12T04:05:00.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZwhVOXHdpHu6Vv5uL6K/z9HQdDs=', 'Content-Length': '0'} 2024-04-12T04:05:00.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000107d7be37aab8cd3-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.731 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:05:00.731 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:05:00.731 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.732 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.732 DEBUG:boto:Method: GET 2024-04-12T04:05:00.732 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.732 DEBUG:boto:Data: 2024-04-12T04:05:00.732 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.732 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.732 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.732 DEBUG:boto:Params: {} 2024-04-12T04:05:00.732 DEBUG:boto:Token: None 2024-04-12T04:05:00.732 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.732 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZwhVOXHdpHu6Vv5uL6K/z9HQdDs= 2024-04-12T04:05:00.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZwhVOXHdpHu6Vv5uL6K/z9HQdDs=', 'Content-Length': '0'} 2024-04-12T04:05:00.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce9d9ab55fbcfd63-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.735 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:05:00.735 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:05:00.735 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.735 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.735 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.735 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.735 DEBUG:boto:Data: 2024-04-12T04:05:00.735 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.735 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.735 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.736 DEBUG:boto:Params: {} 2024-04-12T04:05:00.736 DEBUG:boto:Token: None 2024-04-12T04:05:00.736 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.736 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5xuIeZ1PMpCcbvZKcLYFurtrz5g= 2024-04-12T04:05:00.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5xuIeZ1PMpCcbvZKcLYFurtrz5g=', 'Content-Length': '0'} 2024-04-12T04:05:00.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b100ec44a32e1c0c-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.738 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:05:00.738 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:05:00.738 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.738 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.738 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.738 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.738 DEBUG:boto:Data: 2024-04-12T04:05:00.738 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.738 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.738 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.738 DEBUG:boto:Params: {} 2024-04-12T04:05:00.739 DEBUG:boto:Token: None 2024-04-12T04:05:00.739 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.739 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5xuIeZ1PMpCcbvZKcLYFurtrz5g= 2024-04-12T04:05:00.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5xuIeZ1PMpCcbvZKcLYFurtrz5g=', 'Content-Length': '0'} 2024-04-12T04:05:00.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd5af13f65c09dfc-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:05:00.741 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:05:00.741 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:05:00.741 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.742 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.742 DEBUG:boto:Method: GET 2024-04-12T04:05:00.742 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.742 DEBUG:boto:Data: 2024-04-12T04:05:00.742 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.742 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.742 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.742 DEBUG:boto:Params: {} 2024-04-12T04:05:00.743 DEBUG:boto:Token: None 2024-04-12T04:05:00.743 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.743 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZwhVOXHdpHu6Vv5uL6K/z9HQdDs= 2024-04-12T04:05:00.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZwhVOXHdpHu6Vv5uL6K/z9HQdDs=', 'Content-Length': '0'} 2024-04-12T04:05:00.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006058f7bee07fc3a2-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.786 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:05:00.787 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:05:00.787 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.787 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.787 DEBUG:boto:Method: GET 2024-04-12T04:05:00.787 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.787 DEBUG:boto:Data: 2024-04-12T04:05:00.787 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.787 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.787 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.788 DEBUG:boto:Params: {} 2024-04-12T04:05:00.788 DEBUG:boto:Token: None 2024-04-12T04:05:00.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:05:00.788 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZwhVOXHdpHu6Vv5uL6K/z9HQdDs= 2024-04-12T04:05:00.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZwhVOXHdpHu6Vv5uL6K/z9HQdDs=', 'Content-Length': '0'} 2024-04-12T04:05:00.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c73df11a4a52073-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.791 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.791 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.791 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:05:00.791 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:05:00.791 DEBUG:boto:Method: GET 2024-04-12T04:05:00.791 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:05:00.791 DEBUG:boto:Data: 2024-04-12T04:05:00.791 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.791 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.792 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.792 DEBUG:boto:Params: {} 2024-04-12T04:05:00.792 DEBUG:boto:Token: None 2024-04-12T04:05:00.792 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.792 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccf8ccd110bd8a26-006618b2ec-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.795 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.795 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.795 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.795 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:05:00.796 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:05:00.796 DEBUG:boto:Method: GET 2024-04-12T04:05:00.796 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:05:00.796 DEBUG:boto:Data: 2024-04-12T04:05:00.796 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.796 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.796 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.796 DEBUG:boto:Params: {} 2024-04-12T04:05:00.796 DEBUG:boto:Token: None 2024-04-12T04:05:00.796 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.797 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049a83fdf63532741-006618b2ec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.800 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.800 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:05:00.801 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:05:00.801 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.801 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:05:00.801 DEBUG:boto:Data: 2024-04-12T04:05:00.801 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.801 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.801 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.801 DEBUG:boto:Params: {} 2024-04-12T04:05:00.802 DEBUG:boto:Token: None 2024-04-12T04:05:00.802 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj95 2024-04-12T04:05:00.802 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JdT1tAJBl6Loklvk/UP6NBtKszE= 2024-04-12T04:05:00.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JdT1tAJBl6Loklvk/UP6NBtKszE=', 'Content-Length': '0'} 2024-04-12T04:05:00.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090b286efaea9efb1-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.804 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:05:00.804 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:05:00.804 DEBUG:boto:Method: HEAD 2024-04-12T04:05:00.805 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:05:00.805 DEBUG:boto:Data: 2024-04-12T04:05:00.805 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.805 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.805 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.805 DEBUG:boto:Params: {} 2024-04-12T04:05:00.805 DEBUG:boto:Token: None 2024-04-12T04:05:00.805 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj95 2024-04-12T04:05:00.805 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JdT1tAJBl6Loklvk/UP6NBtKszE= 2024-04-12T04:05:00.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JdT1tAJBl6Loklvk/UP6NBtKszE=', 'Content-Length': '0'} 2024-04-12T04:05:00.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011b4a1420cbbb6e3-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:05:00.808 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:05:00.808 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:05:00.808 DEBUG:boto:Method: GET 2024-04-12T04:05:00.808 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:05:00.808 DEBUG:boto:Data: 2024-04-12T04:05:00.808 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.808 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.808 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.808 DEBUG:boto:Params: {} 2024-04-12T04:05:00.808 DEBUG:boto:Token: None 2024-04-12T04:05:00.808 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj95 2024-04-12T04:05:00.809 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VIran6UJCMvLQppiAjd8dSGd5d0= 2024-04-12T04:05:00.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VIran6UJCMvLQppiAjd8dSGd5d0=', 'Content-Length': '0'} 2024-04-12T04:05:00.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b41030ae559a23d4-006618b2ec-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.811 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:05:00.811 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:05:00.811 DEBUG:boto:Method: GET 2024-04-12T04:05:00.811 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:05:00.811 DEBUG:boto:Data: 2024-04-12T04:05:00.811 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.811 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.811 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.811 DEBUG:boto:Params: {} 2024-04-12T04:05:00.811 DEBUG:boto:Token: None 2024-04-12T04:05:00.812 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/obj95 2024-04-12T04:05:00.812 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VIran6UJCMvLQppiAjd8dSGd5d0= 2024-04-12T04:05:00.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VIran6UJCMvLQppiAjd8dSGd5d0=', 'Content-Length': '0'} 2024-04-12T04:05:00.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddb633ab531f25fb-006618b2ec-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.854 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.854 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.855 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:05:00.855 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:05:00.855 DEBUG:boto:Method: GET 2024-04-12T04:05:00.855 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:05:00.855 DEBUG:boto:Data: 2024-04-12T04:05:00.855 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.855 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:05:00.855 DEBUG:boto:Port: 8001 2024-04-12T04:05:00.855 DEBUG:boto:Params: {} 2024-04-12T04:05:00.855 DEBUG:boto:Token: None 2024-04-12T04:05:00.855 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.856 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.858 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041bb989aff52b7d3-006618b2ec-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.858 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.859 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:00.859 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:00.859 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:05:00.859 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:05:00.859 DEBUG:boto:Method: GET 2024-04-12T04:05:00.859 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:05:00.859 DEBUG:boto:Data: 2024-04-12T04:05:00.859 DEBUG:boto:Headers: {} 2024-04-12T04:05:00.859 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:00.859 DEBUG:boto:Port: 8000 2024-04-12T04:05:00.859 DEBUG:boto:Params: {} 2024-04-12T04:05:00.860 DEBUG:boto:Token: None 2024-04-12T04:05:00.860 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:05:00 GMT /dkyges-61/ 2024-04-12T04:05:00.860 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU= 2024-04-12T04:05:00.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CLiis4wu3f4sq7K9pLXlbce7OhU=', 'Content-Length': '0'} 2024-04-12T04:05:00.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098a2d77ef35aafd1-006618b2ec-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:05:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:00.863 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:05:00.864 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-61 zones={a2, a1} 2024-04-12T04:05:00.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:05:00.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:05:00.864 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:05:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:05:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:05:00.983 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:05:00.983 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-61 2024-04-12T04:05:00.991 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 15 2024-04-12T04:05:01.046 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:01.046 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:01.047 DEBUG:boto:Method: HEAD 2024-04-12T04:05:01.047 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:05:01.047 DEBUG:boto:Data: 2024-04-12T04:05:01.047 DEBUG:boto:Headers: {} 2024-04-12T04:05:01.047 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:01.047 DEBUG:boto:Port: 8000 2024-04-12T04:05:01.047 DEBUG:boto:Params: {} 2024-04-12T04:05:01.047 DEBUG:boto:Token: None 2024-04-12T04:05:01.048 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:01 GMT /dkyges-61/ 2024-04-12T04:05:01.048 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mPtDMvDP7H0v6vNcTsd2iD4LkeE= 2024-04-12T04:05:01.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mPtDMvDP7H0v6vNcTsd2iD4LkeE=', 'Content-Length': '0'} 2024-04-12T04:05:01.051 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000670c258a00ba29d5-006618b2ed-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:01.052 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:05:01.052 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:05:01.052 DEBUG:boto:Method: PUT 2024-04-12T04:05:01.052 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:05:01.052 DEBUG:boto:Data: 2024-04-12T04:05:01.052 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:05:01.052 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:01.052 DEBUG:boto:Port: 8000 2024-04-12T04:05:01.052 DEBUG:boto:Params: {} 2024-04-12T04:05:01.052 DEBUG:boto:Token: None 2024-04-12T04:05:01.053 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:05:01 GMT /dkyges-61/obj23 2024-04-12T04:05:01.053 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KZJnMcXd2aCsM5k+eGkhNxBGKXQ= 2024-04-12T04:05:01.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:05:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KZJnMcXd2aCsM5k+eGkhNxBGKXQ='} 2024-04-12T04:05:01.059 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f29534d4792dcde3-006618b2ed-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:01.059 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:01.059 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:01.059 DEBUG:boto:Method: HEAD 2024-04-12T04:05:01.059 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:05:01.059 DEBUG:boto:Data: 2024-04-12T04:05:01.059 DEBUG:boto:Headers: {} 2024-04-12T04:05:01.060 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:01.060 DEBUG:boto:Port: 8000 2024-04-12T04:05:01.060 DEBUG:boto:Params: {} 2024-04-12T04:05:01.060 DEBUG:boto:Token: None 2024-04-12T04:05:01.060 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:01 GMT /dkyges-61/ 2024-04-12T04:05:01.060 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mPtDMvDP7H0v6vNcTsd2iD4LkeE= 2024-04-12T04:05:01.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mPtDMvDP7H0v6vNcTsd2iD4LkeE=', 'Content-Length': '0'} 2024-04-12T04:05:01.063 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a7f9b48b559fd67-006618b2ed-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:01.063 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:05:01.063 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:05:01.063 DEBUG:boto:Method: PUT 2024-04-12T04:05:01.063 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:05:01.064 DEBUG:boto:Data: 2024-04-12T04:05:01.064 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:05:01.064 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:01.064 DEBUG:boto:Port: 8000 2024-04-12T04:05:01.064 DEBUG:boto:Params: {} 2024-04-12T04:05:01.064 DEBUG:boto:Token: None 2024-04-12T04:05:01.064 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:05:01 GMT /dkyges-61/obj46 2024-04-12T04:05:01.064 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2Lbc50Srqdh+XEtM+SXUxrxxIx0= 2024-04-12T04:05:01.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:05:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2Lbc50Srqdh+XEtM+SXUxrxxIx0='} 2024-04-12T04:05:01.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b51bd7df99a0728-006618b2ed-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:01.107 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:01.107 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:01.107 DEBUG:boto:Method: HEAD 2024-04-12T04:05:01.107 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:05:01.107 DEBUG:boto:Data: 2024-04-12T04:05:01.107 DEBUG:boto:Headers: {} 2024-04-12T04:05:01.107 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:01.107 DEBUG:boto:Port: 8000 2024-04-12T04:05:01.107 DEBUG:boto:Params: {} 2024-04-12T04:05:01.107 DEBUG:boto:Token: None 2024-04-12T04:05:01.108 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:01 GMT /dkyges-61/ 2024-04-12T04:05:01.108 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mPtDMvDP7H0v6vNcTsd2iD4LkeE= 2024-04-12T04:05:01.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mPtDMvDP7H0v6vNcTsd2iD4LkeE=', 'Content-Length': '0'} 2024-04-12T04:05:01.111 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b60ef99a4a17ea9-006618b2ed-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:01.111 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:05:01.111 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:05:01.111 DEBUG:boto:Method: PUT 2024-04-12T04:05:01.112 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:05:01.112 DEBUG:boto:Data: 2024-04-12T04:05:01.112 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:05:01.112 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:01.112 DEBUG:boto:Port: 8000 2024-04-12T04:05:01.112 DEBUG:boto:Params: {} 2024-04-12T04:05:01.112 DEBUG:boto:Token: None 2024-04-12T04:05:01.112 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:05:01 GMT /dkyges-61/obj69 2024-04-12T04:05:01.112 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1HXdC/i95+Gp3ErmlQM22REypEk= 2024-04-12T04:05:01.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:05:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1HXdC/i95+Gp3ErmlQM22REypEk='} 2024-04-12T04:05:01.119 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ddb5df4027ebd75-006618b2ed-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:01.119 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:01.119 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:01.119 DEBUG:boto:Method: HEAD 2024-04-12T04:05:01.119 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:05:01.119 DEBUG:boto:Data: 2024-04-12T04:05:01.119 DEBUG:boto:Headers: {} 2024-04-12T04:05:01.119 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:01.119 DEBUG:boto:Port: 8000 2024-04-12T04:05:01.119 DEBUG:boto:Params: {} 2024-04-12T04:05:01.120 DEBUG:boto:Token: None 2024-04-12T04:05:01.120 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:01 GMT /dkyges-61/ 2024-04-12T04:05:01.120 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mPtDMvDP7H0v6vNcTsd2iD4LkeE= 2024-04-12T04:05:01.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mPtDMvDP7H0v6vNcTsd2iD4LkeE=', 'Content-Length': '0'} 2024-04-12T04:05:01.123 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc428e56fa177f09-006618b2ed-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:01.123 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:05:01.123 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:05:01.123 DEBUG:boto:Method: PUT 2024-04-12T04:05:01.123 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:05:01.123 DEBUG:boto:Data: 2024-04-12T04:05:01.123 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:05:01.123 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:01.123 DEBUG:boto:Port: 8000 2024-04-12T04:05:01.123 DEBUG:boto:Params: {} 2024-04-12T04:05:01.124 DEBUG:boto:Token: None 2024-04-12T04:05:01.124 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:05:01 GMT /dkyges-61/obj92 2024-04-12T04:05:01.124 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:K55HnjahPYlSUFPS0ZSqj5Wa2gY= 2024-04-12T04:05:01.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:05:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:K55HnjahPYlSUFPS0ZSqj5Wa2gY='} 2024-04-12T04:05:01.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008afbc009000c77b4-006618b2ed-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:01.167 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:05:01.167 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:05:01.167 DEBUG:boto:Method: HEAD 2024-04-12T04:05:01.167 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:05:01.167 DEBUG:boto:Data: 2024-04-12T04:05:01.167 DEBUG:boto:Headers: {} 2024-04-12T04:05:01.167 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:01.167 DEBUG:boto:Port: 8000 2024-04-12T04:05:01.167 DEBUG:boto:Params: {} 2024-04-12T04:05:01.167 DEBUG:boto:Token: None 2024-04-12T04:05:01.167 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:05:01 GMT /dkyges-61/ 2024-04-12T04:05:01.168 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mPtDMvDP7H0v6vNcTsd2iD4LkeE= 2024-04-12T04:05:01.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:05:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mPtDMvDP7H0v6vNcTsd2iD4LkeE=', 'Content-Length': '0'} 2024-04-12T04:05:01.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec9a829ef584e3ca-006618b2ed-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:01.172 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:05:01.172 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:05:01.172 DEBUG:boto:Method: PUT 2024-04-12T04:05:01.172 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:05:01.172 DEBUG:boto:Data: 2024-04-12T04:05:01.172 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:05:01.172 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:05:01.172 DEBUG:boto:Port: 8000 2024-04-12T04:05:01.172 DEBUG:boto:Params: {} 2024-04-12T04:05:01.173 DEBUG:boto:Token: None 2024-04-12T04:05:01.173 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:05:01 GMT /dkyges-61/obj115 2024-04-12T04:05:01.173 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:edXSZHfDq0q589hlKYAFVVYJ6RE= 2024-04-12T04:05:01.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:05:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:edXSZHfDq0q589hlKYAFVVYJ6RE='} 2024-04-12T04:05:01.179 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008af2253f93109ceb-006618b2ed-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:05:01.179 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-61 2024-04-12T04:05:01.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:05:01.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:05:01.180 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:05:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:05:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:05:01.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:05:01.314+0000 7f161e98c780 1 waiting to reach latest gen 3, on 2.. 2024-04-12T04:05:06.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:05:06.315+0000 7f161e98c780 1 waiting to reach latest gen 3, on 2.. 2024-04-12T04:05:11.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:05:11.315+0000 7f161e98c780 1 waiting to reach latest gen 3, on 2.. 2024-04-12T04:05:16.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:05:16.317+0000 7f161e98c780 1 waiting to reach latest gen 3, on 2.. 2024-04-12T04:05:21.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:05:21.319+0000 7f161e98c780 1 waiting for incremental sync to catch up: 2024-04-12T04:05:21.304 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , , , , , 00000000001.88.6, , , , , , ] 2024-04-12T04:05:21.305 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:05:26.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:05:26.322+0000 7f161e98c780 1 waiting for incremental sync to catch up: 2024-04-12T04:05:26.307 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:05:26.307 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:05:31.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:05:31.323+0000 7f161e98c780 1 waiting for incremental sync to catch up: 2024-04-12T04:05:31.308 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:05:31.308 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:05:36.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:05:36.325+0000 7f161e98c780 1 waiting for incremental sync to catch up: 2024-04-12T04:05:36.310 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:05:36.311 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:05:41.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:05:41.327+0000 7f161e98c780 1 waiting for incremental sync to catch up: 2024-04-12T04:05:41.312 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:05:41.312 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:05:46.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:05:46.329+0000 7f161e98c780 1 waiting for incremental sync to catch up: 2024-04-12T04:05:46.315 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:05:46.315 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:05:51.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:05:51.331+0000 7f161e98c780 1 waiting for incremental sync to catch up: 2024-04-12T04:05:51.316 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:05:51.317 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:05:56.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:05:56.333+0000 7f161e98c780 1 waiting for incremental sync to catch up: 2024-04-12T04:05:56.319 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:05:56.319 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:01.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:06:01.335+0000 7f161e98c780 1 waiting for incremental sync to catch up: 2024-04-12T04:06:01.321 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:01.321 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:06.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:06:06.337+0000 7f161e98c780 1 waiting for incremental sync to catch up: 2024-04-12T04:06:06.323 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:06.323 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:06:11.339+0000 7f161e98c780 1 waiting for incremental sync to catch up: 2024-04-12T04:06:11.325 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:11.325 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:16.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:06:16.341+0000 7f161e98c780 1 waiting for incremental sync to catch up: 2024-04-12T04:06:16.327 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:16.327 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:21.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:06:21.344+0000 7f161e98c780 1 waiting for incremental sync to catch up: 2024-04-12T04:06:21.329 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:21.329 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:26.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:06:26.346+0000 7f161e98c780 1 waiting for incremental sync to catch up: 2024-04-12T04:06:26.331 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:26.331 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:31.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:06:31.348+0000 7f161e98c780 1 waiting for incremental sync to catch up: 2024-04-12T04:06:31.333 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:31.333 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:36.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:06:36.350+0000 7f161e98c780 1 bucket sync caught up with source: 2024-04-12T04:06:36.335 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:36.335 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.67.6, , , 00000000002.160.6, , 00000000002.51.6, , , , , , 00000000001.88.6, , 00000000001.89.6, , , , ] 2024-04-12T04:06:36.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:06:36.350+0000 7f161e98c780 0 bucket checkpoint complete 2024-04-12T04:06:36.354 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-61 2024-04-12T04:06:36.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:06:36.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:06:36.355 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:06:36.420 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:06:36.420 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:06:36.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:06:36.477+0000 7f4b71f5f780 1 waiting for incremental sync to catch up: 2024-04-12T04:06:36.461 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.2368.6, 00000000002.1411.6, 00000000001.2992.6, 00000000002.2378.6, 00000000002.2601.6, , , 00000000002.2561.6, 00000000004.2282.6, 00000000004.2618.6] 2024-04-12T04:06:36.461 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.2368.6, 00000000002.1411.6, 00000000001.2992.6, 00000000002.2378.6, 00000000002.2601.6, , , 00000000002.2561.6, 00000000005.2288.6, 00000000004.2618.6] 2024-04-12T04:06:41.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:06:41.479+0000 7f4b71f5f780 1 waiting for incremental sync to catch up: 2024-04-12T04:06:41.463 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.2368.6, 00000000002.1411.6, 00000000001.2992.6, 00000000002.2378.6, 00000000002.2601.6, , , 00000000002.2561.6, 00000000004.2282.6, 00000000004.2618.6] 2024-04-12T04:06:41.463 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.2368.6, 00000000002.1411.6, 00000000001.2992.6, 00000000002.2378.6, 00000000002.2601.6, , , 00000000002.2561.6, 00000000005.2288.6, 00000000004.2618.6] 2024-04-12T04:06:46.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:06:46.479+0000 7f4b71f5f780 1 bucket sync caught up with source: 2024-04-12T04:06:46.464 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.2368.6, 00000000002.1411.6, 00000000001.2992.6, 00000000002.2378.6, 00000000002.2601.6, , , 00000000002.2561.6, 00000000005.2288.6, 00000000004.2618.6] 2024-04-12T04:06:46.464 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.2368.6, 00000000002.1411.6, 00000000001.2992.6, 00000000002.2378.6, 00000000002.2601.6, , , 00000000002.2561.6, 00000000005.2288.6, 00000000004.2618.6] 2024-04-12T04:06:46.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:06:46.479+0000 7f4b71f5f780 0 bucket checkpoint complete 2024-04-12T04:06:46.483 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-61 zones={a2, a1} 2024-04-12T04:06:46.483 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.483 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.483 DEBUG:boto:Method: HEAD 2024-04-12T04:06:46.483 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:06:46.484 DEBUG:boto:Data: 2024-04-12T04:06:46.484 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.484 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.484 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.484 DEBUG:boto:Params: {} 2024-04-12T04:06:46.484 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:06:46.485 DEBUG:boto:Token: None 2024-04-12T04:06:46.485 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/ 2024-04-12T04:06:46.485 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UcyI7RrgoPTrzeH15LwsuM6T5ig= 2024-04-12T04:06:46.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UcyI7RrgoPTrzeH15LwsuM6T5ig=', 'Content-Length': '0'} 2024-04-12T04:06:46.498 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e451ecb4ebd28601-006618b356-1cfd-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.498 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.498 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.499 DEBUG:boto:Method: HEAD 2024-04-12T04:06:46.499 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:06:46.499 DEBUG:boto:Data: 2024-04-12T04:06:46.499 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.499 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.499 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.499 DEBUG:boto:Params: {} 2024-04-12T04:06:46.500 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:06:46.500 DEBUG:boto:Token: None 2024-04-12T04:06:46.500 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/ 2024-04-12T04:06:46.500 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UcyI7RrgoPTrzeH15LwsuM6T5ig= 2024-04-12T04:06:46.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UcyI7RrgoPTrzeH15LwsuM6T5ig=', 'Content-Length': '0'} 2024-04-12T04:06:46.530 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084628cfbb2a3c688-006618b356-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.530 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:06:46.530 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.530 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.530 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:06:46.531 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:06:46.531 DEBUG:boto:Method: GET 2024-04-12T04:06:46.531 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:06:46.531 DEBUG:boto:Data: 2024-04-12T04:06:46.531 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.531 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.531 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.531 DEBUG:boto:Params: {} 2024-04-12T04:06:46.531 DEBUG:boto:Token: None 2024-04-12T04:06:46.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/?versions 2024-04-12T04:06:46.532 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:25cqXwE11y+E51dT1THXIFnDOEI= 2024-04-12T04:06:46.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:25cqXwE11y+E51dT1THXIFnDOEI=', 'Content-Length': '0'} 2024-04-12T04:06:46.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060bb07e72b1d293c-006618b356-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.535 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.536 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:06:46.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T04:06:46.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:06:46.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T04:06:46.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:06:46.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T04:06:46.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:06:46.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:06:46.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T04:06:46.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:06:46.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T04:06:46.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:06:46.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:06:46.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T04:06:46.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:06:46.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T04:06:46.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T04:06:46.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:06:46.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T04:06:46.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T04:06:46.538 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:06:46.538 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.538 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.538 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:06:46.539 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:06:46.539 DEBUG:boto:Method: GET 2024-04-12T04:06:46.539 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:06:46.539 DEBUG:boto:Data: 2024-04-12T04:06:46.539 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.539 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.539 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.539 DEBUG:boto:Params: {} 2024-04-12T04:06:46.539 DEBUG:boto:Token: None 2024-04-12T04:06:46.539 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/?versions 2024-04-12T04:06:46.540 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:25cqXwE11y+E51dT1THXIFnDOEI= 2024-04-12T04:06:46.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:25cqXwE11y+E51dT1THXIFnDOEI=', 'Content-Length': '0'} 2024-04-12T04:06:46.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005694cba12b37e097-006618b356-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.543 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:06:46.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T04:06:46.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:06:46.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T04:06:46.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:06:46.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T04:06:46.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:06:46.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:06:46.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T04:06:46.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:06:46.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T04:06:46.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:06:46.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:06:46.545 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T04:06:46.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:06:46.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T04:06:46.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T04:06:46.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:06:46.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T04:06:46.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T04:06:46.546 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.546 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.546 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:06:46.546 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:06:46.546 DEBUG:boto:Method: GET 2024-04-12T04:06:46.547 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:06:46.547 DEBUG:boto:Data: 2024-04-12T04:06:46.547 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.547 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.547 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.547 DEBUG:boto:Params: {} 2024-04-12T04:06:46.547 DEBUG:boto:Token: None 2024-04-12T04:06:46.547 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/?versions 2024-04-12T04:06:46.547 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:25cqXwE11y+E51dT1THXIFnDOEI= 2024-04-12T04:06:46.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:25cqXwE11y+E51dT1THXIFnDOEI=', 'Content-Length': '0'} 2024-04-12T04:06:46.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd91898e457984ec-006618b356-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.550 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.551 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.551 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.551 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:06:46.551 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:06:46.551 DEBUG:boto:Method: GET 2024-04-12T04:06:46.552 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:06:46.552 DEBUG:boto:Data: 2024-04-12T04:06:46.552 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.552 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.552 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.552 DEBUG:boto:Params: {} 2024-04-12T04:06:46.552 DEBUG:boto:Token: None 2024-04-12T04:06:46.552 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/?versions 2024-04-12T04:06:46.552 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:25cqXwE11y+E51dT1THXIFnDOEI= 2024-04-12T04:06:46.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:25cqXwE11y+E51dT1THXIFnDOEI=', 'Content-Length': '0'} 2024-04-12T04:06:46.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080e3ca34d253a06c-006618b356-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.556 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:06:46.557 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:06:46.557 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:06:46.558 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.558 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.558 DEBUG:boto:Method: GET 2024-04-12T04:06:46.558 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.558 DEBUG:boto:Data: 2024-04-12T04:06:46.558 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.558 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.558 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.558 DEBUG:boto:Params: {} 2024-04-12T04:06:46.558 DEBUG:boto:Token: None 2024-04-12T04:06:46.559 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.559 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HrDe7uCgskbacOwLh3ZU730tXUc= 2024-04-12T04:06:46.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HrDe7uCgskbacOwLh3ZU730tXUc=', 'Content-Length': '0'} 2024-04-12T04:06:46.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000488f507b4f9d8593-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.561 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:06:46.561 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:06:46.561 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.561 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.561 DEBUG:boto:Method: GET 2024-04-12T04:06:46.562 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.562 DEBUG:boto:Data: 2024-04-12T04:06:46.562 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.562 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.562 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.562 DEBUG:boto:Params: {} 2024-04-12T04:06:46.562 DEBUG:boto:Token: None 2024-04-12T04:06:46.562 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.562 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HrDe7uCgskbacOwLh3ZU730tXUc= 2024-04-12T04:06:46.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HrDe7uCgskbacOwLh3ZU730tXUc=', 'Content-Length': '0'} 2024-04-12T04:06:46.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049cc6c6e709535bc-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.564 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:06:46.564 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:06:46.564 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.565 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.565 DEBUG:boto:Method: HEAD 2024-04-12T04:06:46.565 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.565 DEBUG:boto:Data: 2024-04-12T04:06:46.565 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.565 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.565 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.565 DEBUG:boto:Params: {} 2024-04-12T04:06:46.565 DEBUG:boto:Token: None 2024-04-12T04:06:46.565 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.566 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sdwVstv5ATRguamNVCPMwY8TleE= 2024-04-12T04:06:46.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sdwVstv5ATRguamNVCPMwY8TleE=', 'Content-Length': '0'} 2024-04-12T04:06:46.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f51f80e7fa15c8a1-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.568 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:06:46.568 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:06:46.568 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.568 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.568 DEBUG:boto:Method: HEAD 2024-04-12T04:06:46.568 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.568 DEBUG:boto:Data: 2024-04-12T04:06:46.568 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.568 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.569 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.569 DEBUG:boto:Params: {} 2024-04-12T04:06:46.569 DEBUG:boto:Token: None 2024-04-12T04:06:46.569 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.569 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sdwVstv5ATRguamNVCPMwY8TleE= 2024-04-12T04:06:46.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sdwVstv5ATRguamNVCPMwY8TleE=', 'Content-Length': '0'} 2024-04-12T04:06:46.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b054f05c6fab02a-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:06:46.571 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:06:46.571 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:06:46.571 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.571 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.572 DEBUG:boto:Method: GET 2024-04-12T04:06:46.572 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.572 DEBUG:boto:Data: 2024-04-12T04:06:46.572 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.572 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.572 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.572 DEBUG:boto:Params: {} 2024-04-12T04:06:46.572 DEBUG:boto:Token: None 2024-04-12T04:06:46.572 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.572 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HrDe7uCgskbacOwLh3ZU730tXUc= 2024-04-12T04:06:46.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HrDe7uCgskbacOwLh3ZU730tXUc=', 'Content-Length': '0'} 2024-04-12T04:06:46.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019ae4e8c1e94f61b-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.575 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:06:46.575 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:06:46.575 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.575 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.575 DEBUG:boto:Method: GET 2024-04-12T04:06:46.575 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.575 DEBUG:boto:Data: 2024-04-12T04:06:46.575 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.575 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.575 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.575 DEBUG:boto:Params: {} 2024-04-12T04:06:46.576 DEBUG:boto:Token: None 2024-04-12T04:06:46.576 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:06:46.576 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HrDe7uCgskbacOwLh3ZU730tXUc= 2024-04-12T04:06:46.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HrDe7uCgskbacOwLh3ZU730tXUc=', 'Content-Length': '0'} 2024-04-12T04:06:46.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3663407a82da5df-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.618 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.618 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.618 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:06:46.618 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:06:46.619 DEBUG:boto:Method: GET 2024-04-12T04:06:46.619 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:06:46.619 DEBUG:boto:Data: 2024-04-12T04:06:46.619 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.619 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.619 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.619 DEBUG:boto:Params: {} 2024-04-12T04:06:46.619 DEBUG:boto:Token: None 2024-04-12T04:06:46.619 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/ 2024-04-12T04:06:46.619 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0= 2024-04-12T04:06:46.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0=', 'Content-Length': '0'} 2024-04-12T04:06:46.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000004870a726964736-006618b356-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.623 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.623 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.623 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.623 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:06:46.623 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:06:46.623 DEBUG:boto:Method: GET 2024-04-12T04:06:46.623 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:06:46.624 DEBUG:boto:Data: 2024-04-12T04:06:46.624 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.624 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.624 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.624 DEBUG:boto:Params: {} 2024-04-12T04:06:46.624 DEBUG:boto:Token: None 2024-04-12T04:06:46.624 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/ 2024-04-12T04:06:46.624 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0= 2024-04-12T04:06:46.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0=', 'Content-Length': '0'} 2024-04-12T04:06:46.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018624efadad5a037-006618b356-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.627 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.628 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:06:46.628 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:06:46.628 DEBUG:boto:Method: HEAD 2024-04-12T04:06:46.628 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:06:46.628 DEBUG:boto:Data: 2024-04-12T04:06:46.628 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.628 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.628 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.628 DEBUG:boto:Params: {} 2024-04-12T04:06:46.629 DEBUG:boto:Token: None 2024-04-12T04:06:46.629 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj11 2024-04-12T04:06:46.629 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LsXzVPecQ5zVS2njsKxbXOzouh0= 2024-04-12T04:06:46.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LsXzVPecQ5zVS2njsKxbXOzouh0=', 'Content-Length': '0'} 2024-04-12T04:06:46.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a681f140a00e693-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.631 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:06:46.631 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:06:46.631 DEBUG:boto:Method: HEAD 2024-04-12T04:06:46.631 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:06:46.631 DEBUG:boto:Data: 2024-04-12T04:06:46.631 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.631 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.631 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.631 DEBUG:boto:Params: {} 2024-04-12T04:06:46.632 DEBUG:boto:Token: None 2024-04-12T04:06:46.632 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj11 2024-04-12T04:06:46.632 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LsXzVPecQ5zVS2njsKxbXOzouh0= 2024-04-12T04:06:46.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LsXzVPecQ5zVS2njsKxbXOzouh0=', 'Content-Length': '0'} 2024-04-12T04:06:46.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e61a34dfecbe86c-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:06:46.634 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:06:46.634 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:06:46.635 DEBUG:boto:Method: GET 2024-04-12T04:06:46.635 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:06:46.635 DEBUG:boto:Data: 2024-04-12T04:06:46.635 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.635 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.635 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.635 DEBUG:boto:Params: {} 2024-04-12T04:06:46.635 DEBUG:boto:Token: None 2024-04-12T04:06:46.635 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj11 2024-04-12T04:06:46.635 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FAqfmNiCOQUBIrOTZbC5rdZyPgU= 2024-04-12T04:06:46.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FAqfmNiCOQUBIrOTZbC5rdZyPgU=', 'Content-Length': '0'} 2024-04-12T04:06:46.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d05b1420d5a41abf-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.678 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:06:46.679 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:06:46.679 DEBUG:boto:Method: GET 2024-04-12T04:06:46.679 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:06:46.679 DEBUG:boto:Data: 2024-04-12T04:06:46.679 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.679 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.679 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.679 DEBUG:boto:Params: {} 2024-04-12T04:06:46.679 DEBUG:boto:Token: None 2024-04-12T04:06:46.680 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj11 2024-04-12T04:06:46.680 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FAqfmNiCOQUBIrOTZbC5rdZyPgU= 2024-04-12T04:06:46.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FAqfmNiCOQUBIrOTZbC5rdZyPgU=', 'Content-Length': '0'} 2024-04-12T04:06:46.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab2c6fccb810b361-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.682 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.682 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.682 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:06:46.682 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:06:46.682 DEBUG:boto:Method: GET 2024-04-12T04:06:46.683 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:06:46.683 DEBUG:boto:Data: 2024-04-12T04:06:46.683 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.683 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.683 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.683 DEBUG:boto:Params: {} 2024-04-12T04:06:46.683 DEBUG:boto:Token: None 2024-04-12T04:06:46.683 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/ 2024-04-12T04:06:46.683 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0= 2024-04-12T04:06:46.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0=', 'Content-Length': '0'} 2024-04-12T04:06:46.686 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d47ae7bad01f177e-006618b356-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.686 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.686 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.687 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.687 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:06:46.687 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:06:46.687 DEBUG:boto:Method: GET 2024-04-12T04:06:46.687 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:06:46.687 DEBUG:boto:Data: 2024-04-12T04:06:46.687 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.687 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.687 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.687 DEBUG:boto:Params: {} 2024-04-12T04:06:46.688 DEBUG:boto:Token: None 2024-04-12T04:06:46.688 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/ 2024-04-12T04:06:46.688 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0= 2024-04-12T04:06:46.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0=', 'Content-Length': '0'} 2024-04-12T04:06:46.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000008cebdec56f4c07-006618b356-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.691 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:06:46.692 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:06:46.692 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:06:46.692 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.692 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.692 DEBUG:boto:Method: GET 2024-04-12T04:06:46.692 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.692 DEBUG:boto:Data: 2024-04-12T04:06:46.692 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.692 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.692 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.693 DEBUG:boto:Params: {} 2024-04-12T04:06:46.693 DEBUG:boto:Token: None 2024-04-12T04:06:46.693 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.693 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ai9+2ufCrdElbuEheGJI6Xqx5u8= 2024-04-12T04:06:46.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ai9+2ufCrdElbuEheGJI6Xqx5u8=', 'Content-Length': '0'} 2024-04-12T04:06:46.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d99d0c5244b778b-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.695 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:06:46.695 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:06:46.695 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.695 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.695 DEBUG:boto:Method: GET 2024-04-12T04:06:46.696 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.696 DEBUG:boto:Data: 2024-04-12T04:06:46.696 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.696 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.696 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.696 DEBUG:boto:Params: {} 2024-04-12T04:06:46.696 DEBUG:boto:Token: None 2024-04-12T04:06:46.696 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.696 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ai9+2ufCrdElbuEheGJI6Xqx5u8= 2024-04-12T04:06:46.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ai9+2ufCrdElbuEheGJI6Xqx5u8=', 'Content-Length': '0'} 2024-04-12T04:06:46.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8e18607d8caa7bc-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.739 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:06:46.739 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:06:46.739 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.740 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.740 DEBUG:boto:Method: HEAD 2024-04-12T04:06:46.740 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.740 DEBUG:boto:Data: 2024-04-12T04:06:46.740 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.740 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.740 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.740 DEBUG:boto:Params: {} 2024-04-12T04:06:46.740 DEBUG:boto:Token: None 2024-04-12T04:06:46.740 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.741 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EnFwKXnrr14wCpJDUSPPDiXjYOo= 2024-04-12T04:06:46.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EnFwKXnrr14wCpJDUSPPDiXjYOo=', 'Content-Length': '0'} 2024-04-12T04:06:46.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000611f3b2f8e521618-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.743 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:06:46.743 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:06:46.743 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.743 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.743 DEBUG:boto:Method: HEAD 2024-04-12T04:06:46.743 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.743 DEBUG:boto:Data: 2024-04-12T04:06:46.743 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.744 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.744 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.744 DEBUG:boto:Params: {} 2024-04-12T04:06:46.744 DEBUG:boto:Token: None 2024-04-12T04:06:46.744 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.744 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EnFwKXnrr14wCpJDUSPPDiXjYOo= 2024-04-12T04:06:46.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EnFwKXnrr14wCpJDUSPPDiXjYOo=', 'Content-Length': '0'} 2024-04-12T04:06:46.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050568e7e83f14511-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:06:46.746 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:06:46.747 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:06:46.747 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.747 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.747 DEBUG:boto:Method: GET 2024-04-12T04:06:46.747 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.747 DEBUG:boto:Data: 2024-04-12T04:06:46.747 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.747 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.747 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.747 DEBUG:boto:Params: {} 2024-04-12T04:06:46.747 DEBUG:boto:Token: None 2024-04-12T04:06:46.748 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.748 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ai9+2ufCrdElbuEheGJI6Xqx5u8= 2024-04-12T04:06:46.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ai9+2ufCrdElbuEheGJI6Xqx5u8=', 'Content-Length': '0'} 2024-04-12T04:06:46.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e11364a8fc214cdb-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.750 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:06:46.750 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:06:46.750 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.750 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.750 DEBUG:boto:Method: GET 2024-04-12T04:06:46.750 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.750 DEBUG:boto:Data: 2024-04-12T04:06:46.750 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.750 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.750 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.751 DEBUG:boto:Params: {} 2024-04-12T04:06:46.751 DEBUG:boto:Token: None 2024-04-12T04:06:46.751 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:06:46.751 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ai9+2ufCrdElbuEheGJI6Xqx5u8= 2024-04-12T04:06:46.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ai9+2ufCrdElbuEheGJI6Xqx5u8=', 'Content-Length': '0'} 2024-04-12T04:06:46.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d9320cc7651d940-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.753 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.753 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.753 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:06:46.753 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:06:46.753 DEBUG:boto:Method: GET 2024-04-12T04:06:46.754 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:06:46.754 DEBUG:boto:Data: 2024-04-12T04:06:46.754 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.754 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.754 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.754 DEBUG:boto:Params: {} 2024-04-12T04:06:46.754 DEBUG:boto:Token: None 2024-04-12T04:06:46.754 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/ 2024-04-12T04:06:46.754 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0= 2024-04-12T04:06:46.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0=', 'Content-Length': '0'} 2024-04-12T04:06:46.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002382a5d69ea15b39-006618b356-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.757 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.757 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.758 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.758 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:06:46.758 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:06:46.758 DEBUG:boto:Method: GET 2024-04-12T04:06:46.758 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:06:46.758 DEBUG:boto:Data: 2024-04-12T04:06:46.758 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.758 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.758 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.758 DEBUG:boto:Params: {} 2024-04-12T04:06:46.758 DEBUG:boto:Token: None 2024-04-12T04:06:46.759 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/ 2024-04-12T04:06:46.759 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0= 2024-04-12T04:06:46.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0=', 'Content-Length': '0'} 2024-04-12T04:06:46.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ea1163648f0bd82-006618b356-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.762 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.763 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:06:46.763 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:06:46.763 DEBUG:boto:Method: HEAD 2024-04-12T04:06:46.763 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:06:46.763 DEBUG:boto:Data: 2024-04-12T04:06:46.763 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.763 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.763 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.764 DEBUG:boto:Params: {} 2024-04-12T04:06:46.764 DEBUG:boto:Token: None 2024-04-12T04:06:46.764 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj115 2024-04-12T04:06:46.764 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:va2OD1ceS4pvCTVZlxThuor5tuk= 2024-04-12T04:06:46.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:va2OD1ceS4pvCTVZlxThuor5tuk=', 'Content-Length': '0'} 2024-04-12T04:06:46.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b7a2d00b6594598-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.766 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:06:46.766 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:06:46.766 DEBUG:boto:Method: HEAD 2024-04-12T04:06:46.766 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:06:46.766 DEBUG:boto:Data: 2024-04-12T04:06:46.766 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.766 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.766 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.767 DEBUG:boto:Params: {} 2024-04-12T04:06:46.767 DEBUG:boto:Token: None 2024-04-12T04:06:46.767 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj115 2024-04-12T04:06:46.767 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:va2OD1ceS4pvCTVZlxThuor5tuk= 2024-04-12T04:06:46.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:va2OD1ceS4pvCTVZlxThuor5tuk=', 'Content-Length': '0'} 2024-04-12T04:06:46.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d63ddac294d62ff7-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:06:46.769 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:06:46.769 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:06:46.769 DEBUG:boto:Method: GET 2024-04-12T04:06:46.770 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:06:46.770 DEBUG:boto:Data: 2024-04-12T04:06:46.770 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.770 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.770 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.770 DEBUG:boto:Params: {} 2024-04-12T04:06:46.771 DEBUG:boto:Token: None 2024-04-12T04:06:46.771 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj115 2024-04-12T04:06:46.771 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vRrL+armNbnYRRnwsbpjwKxi8Sg= 2024-04-12T04:06:46.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vRrL+armNbnYRRnwsbpjwKxi8Sg=', 'Content-Length': '0'} 2024-04-12T04:06:46.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000921bcbae75458d5a-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.773 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:06:46.773 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:06:46.773 DEBUG:boto:Method: GET 2024-04-12T04:06:46.773 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:06:46.773 DEBUG:boto:Data: 2024-04-12T04:06:46.774 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.774 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.774 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.774 DEBUG:boto:Params: {} 2024-04-12T04:06:46.774 DEBUG:boto:Token: None 2024-04-12T04:06:46.774 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj115 2024-04-12T04:06:46.774 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vRrL+armNbnYRRnwsbpjwKxi8Sg= 2024-04-12T04:06:46.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vRrL+armNbnYRRnwsbpjwKxi8Sg=', 'Content-Length': '0'} 2024-04-12T04:06:46.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc977b82a800dd91-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.776 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.776 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.777 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:06:46.777 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:06:46.777 DEBUG:boto:Method: GET 2024-04-12T04:06:46.777 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:06:46.777 DEBUG:boto:Data: 2024-04-12T04:06:46.777 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.777 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.777 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.777 DEBUG:boto:Params: {} 2024-04-12T04:06:46.777 DEBUG:boto:Token: None 2024-04-12T04:06:46.778 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/ 2024-04-12T04:06:46.778 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0= 2024-04-12T04:06:46.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0=', 'Content-Length': '0'} 2024-04-12T04:06:46.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020d0d2eed84da965-006618b356-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.780 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.781 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.781 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.781 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:06:46.781 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:06:46.781 DEBUG:boto:Method: GET 2024-04-12T04:06:46.781 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:06:46.781 DEBUG:boto:Data: 2024-04-12T04:06:46.782 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.782 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.782 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.782 DEBUG:boto:Params: {} 2024-04-12T04:06:46.782 DEBUG:boto:Token: None 2024-04-12T04:06:46.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/ 2024-04-12T04:06:46.782 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0= 2024-04-12T04:06:46.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0=', 'Content-Length': '0'} 2024-04-12T04:06:46.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c4a8d2fb3cd7314-006618b356-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.785 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:06:46.786 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:06:46.786 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:06:46.786 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.786 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.786 DEBUG:boto:Method: GET 2024-04-12T04:06:46.786 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.786 DEBUG:boto:Data: 2024-04-12T04:06:46.786 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.787 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.787 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.787 DEBUG:boto:Params: {} 2024-04-12T04:06:46.787 DEBUG:boto:Token: None 2024-04-12T04:06:46.787 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.787 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lKva3a2TeUUKKjCdrzUDH38Os+o= 2024-04-12T04:06:46.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lKva3a2TeUUKKjCdrzUDH38Os+o=', 'Content-Length': '0'} 2024-04-12T04:06:46.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000248933e2c5368fbe-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.789 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:06:46.790 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:06:46.790 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.790 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.790 DEBUG:boto:Method: GET 2024-04-12T04:06:46.790 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.790 DEBUG:boto:Data: 2024-04-12T04:06:46.790 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.790 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.790 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.790 DEBUG:boto:Params: {} 2024-04-12T04:06:46.790 DEBUG:boto:Token: None 2024-04-12T04:06:46.791 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.791 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lKva3a2TeUUKKjCdrzUDH38Os+o= 2024-04-12T04:06:46.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lKva3a2TeUUKKjCdrzUDH38Os+o=', 'Content-Length': '0'} 2024-04-12T04:06:46.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000216ea3cf47ceec6c-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.793 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:06:46.793 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:06:46.793 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.793 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.794 DEBUG:boto:Method: HEAD 2024-04-12T04:06:46.794 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.794 DEBUG:boto:Data: 2024-04-12T04:06:46.794 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.794 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.794 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.794 DEBUG:boto:Params: {} 2024-04-12T04:06:46.794 DEBUG:boto:Token: None 2024-04-12T04:06:46.794 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.794 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IJyv2rEIJjZRgOqNE+tUKMP9uJI= 2024-04-12T04:06:46.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IJyv2rEIJjZRgOqNE+tUKMP9uJI=', 'Content-Length': '0'} 2024-04-12T04:06:46.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024e432b3efe96d2b-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.797 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:06:46.797 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:06:46.797 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.797 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.797 DEBUG:boto:Method: HEAD 2024-04-12T04:06:46.797 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.797 DEBUG:boto:Data: 2024-04-12T04:06:46.797 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.797 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.797 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.797 DEBUG:boto:Params: {} 2024-04-12T04:06:46.798 DEBUG:boto:Token: None 2024-04-12T04:06:46.798 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.798 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IJyv2rEIJjZRgOqNE+tUKMP9uJI= 2024-04-12T04:06:46.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IJyv2rEIJjZRgOqNE+tUKMP9uJI=', 'Content-Length': '0'} 2024-04-12T04:06:46.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9a89fce34884cf1-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:06:46.800 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:06:46.800 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:06:46.800 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.800 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.800 DEBUG:boto:Method: GET 2024-04-12T04:06:46.800 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.800 DEBUG:boto:Data: 2024-04-12T04:06:46.801 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.801 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.801 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.801 DEBUG:boto:Params: {} 2024-04-12T04:06:46.801 DEBUG:boto:Token: None 2024-04-12T04:06:46.801 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.802 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lKva3a2TeUUKKjCdrzUDH38Os+o= 2024-04-12T04:06:46.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lKva3a2TeUUKKjCdrzUDH38Os+o=', 'Content-Length': '0'} 2024-04-12T04:06:46.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000898db52746abaae8-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.804 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:06:46.804 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:06:46.804 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.804 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.804 DEBUG:boto:Method: GET 2024-04-12T04:06:46.804 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.804 DEBUG:boto:Data: 2024-04-12T04:06:46.804 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.804 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.804 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.804 DEBUG:boto:Params: {} 2024-04-12T04:06:46.805 DEBUG:boto:Token: None 2024-04-12T04:06:46.805 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:06:46.805 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lKva3a2TeUUKKjCdrzUDH38Os+o= 2024-04-12T04:06:46.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lKva3a2TeUUKKjCdrzUDH38Os+o=', 'Content-Length': '0'} 2024-04-12T04:06:46.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4df1e80f51d9b01-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.807 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.807 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.807 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:06:46.808 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:06:46.808 DEBUG:boto:Method: GET 2024-04-12T04:06:46.808 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:06:46.808 DEBUG:boto:Data: 2024-04-12T04:06:46.808 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.808 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.808 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.808 DEBUG:boto:Params: {} 2024-04-12T04:06:46.808 DEBUG:boto:Token: None 2024-04-12T04:06:46.808 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/ 2024-04-12T04:06:46.809 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0= 2024-04-12T04:06:46.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0=', 'Content-Length': '0'} 2024-04-12T04:06:46.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dba49036b3852d7-006618b356-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.811 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.812 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.812 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.812 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:06:46.812 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:06:46.812 DEBUG:boto:Method: GET 2024-04-12T04:06:46.812 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:06:46.812 DEBUG:boto:Data: 2024-04-12T04:06:46.812 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.813 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.813 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.813 DEBUG:boto:Params: {} 2024-04-12T04:06:46.813 DEBUG:boto:Token: None 2024-04-12T04:06:46.813 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/ 2024-04-12T04:06:46.813 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0= 2024-04-12T04:06:46.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0=', 'Content-Length': '0'} 2024-04-12T04:06:46.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a512a3e24142bbd6-006618b356-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.858 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.859 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:06:46.859 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:06:46.859 DEBUG:boto:Method: HEAD 2024-04-12T04:06:46.859 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:06:46.859 DEBUG:boto:Data: 2024-04-12T04:06:46.859 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.859 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.859 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.860 DEBUG:boto:Params: {} 2024-04-12T04:06:46.860 DEBUG:boto:Token: None 2024-04-12T04:06:46.860 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj17 2024-04-12T04:06:46.860 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NLWrnQUYRNCDwwyjsM2AKz8sCyU= 2024-04-12T04:06:46.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NLWrnQUYRNCDwwyjsM2AKz8sCyU=', 'Content-Length': '0'} 2024-04-12T04:06:46.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a8a0f3531b2ddf7-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.863 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:06:46.863 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:06:46.863 DEBUG:boto:Method: HEAD 2024-04-12T04:06:46.863 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:06:46.863 DEBUG:boto:Data: 2024-04-12T04:06:46.863 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.863 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.863 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.863 DEBUG:boto:Params: {} 2024-04-12T04:06:46.863 DEBUG:boto:Token: None 2024-04-12T04:06:46.864 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj17 2024-04-12T04:06:46.864 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NLWrnQUYRNCDwwyjsM2AKz8sCyU= 2024-04-12T04:06:46.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NLWrnQUYRNCDwwyjsM2AKz8sCyU=', 'Content-Length': '0'} 2024-04-12T04:06:46.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a28ef8002651e13-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:06:46.866 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:06:46.866 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:06:46.867 DEBUG:boto:Method: GET 2024-04-12T04:06:46.867 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:06:46.867 DEBUG:boto:Data: 2024-04-12T04:06:46.867 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.867 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.867 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.867 DEBUG:boto:Params: {} 2024-04-12T04:06:46.867 DEBUG:boto:Token: None 2024-04-12T04:06:46.867 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj17 2024-04-12T04:06:46.867 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AuZGnvb0HiHKc7ZSLTtcU/iN/kc= 2024-04-12T04:06:46.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AuZGnvb0HiHKc7ZSLTtcU/iN/kc=', 'Content-Length': '0'} 2024-04-12T04:06:46.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c58f9f8e1a2d242e-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.869 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:06:46.869 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:06:46.870 DEBUG:boto:Method: GET 2024-04-12T04:06:46.870 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:06:46.870 DEBUG:boto:Data: 2024-04-12T04:06:46.870 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.870 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.870 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.870 DEBUG:boto:Params: {} 2024-04-12T04:06:46.870 DEBUG:boto:Token: None 2024-04-12T04:06:46.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj17 2024-04-12T04:06:46.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AuZGnvb0HiHKc7ZSLTtcU/iN/kc= 2024-04-12T04:06:46.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AuZGnvb0HiHKc7ZSLTtcU/iN/kc=', 'Content-Length': '0'} 2024-04-12T04:06:46.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ab27500a976c954-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.873 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.873 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.873 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:06:46.873 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:06:46.873 DEBUG:boto:Method: GET 2024-04-12T04:06:46.874 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:06:46.874 DEBUG:boto:Data: 2024-04-12T04:06:46.874 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.874 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.874 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.874 DEBUG:boto:Params: {} 2024-04-12T04:06:46.874 DEBUG:boto:Token: None 2024-04-12T04:06:46.874 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/ 2024-04-12T04:06:46.874 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0= 2024-04-12T04:06:46.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0=', 'Content-Length': '0'} 2024-04-12T04:06:46.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0465be57560c96f-006618b356-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.877 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.878 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.878 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.878 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:06:46.878 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:06:46.878 DEBUG:boto:Method: GET 2024-04-12T04:06:46.878 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:06:46.878 DEBUG:boto:Data: 2024-04-12T04:06:46.878 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.879 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.879 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.879 DEBUG:boto:Params: {} 2024-04-12T04:06:46.879 DEBUG:boto:Token: None 2024-04-12T04:06:46.879 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/ 2024-04-12T04:06:46.879 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0= 2024-04-12T04:06:46.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0=', 'Content-Length': '0'} 2024-04-12T04:06:46.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4f81985c4c9b98d-006618b356-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.883 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:06:46.883 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:06:46.883 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:06:46.883 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.883 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.884 DEBUG:boto:Method: GET 2024-04-12T04:06:46.884 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.884 DEBUG:boto:Data: 2024-04-12T04:06:46.884 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.884 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.884 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.884 DEBUG:boto:Params: {} 2024-04-12T04:06:46.884 DEBUG:boto:Token: None 2024-04-12T04:06:46.884 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.884 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eBy+Ji/1N2E+r6B6R7Bfr95dSn8= 2024-04-12T04:06:46.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eBy+Ji/1N2E+r6B6R7Bfr95dSn8=', 'Content-Length': '0'} 2024-04-12T04:06:46.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093c7b6fa6a5f7709-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.887 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:06:46.887 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:06:46.887 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.887 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.887 DEBUG:boto:Method: GET 2024-04-12T04:06:46.887 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.887 DEBUG:boto:Data: 2024-04-12T04:06:46.887 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.887 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.887 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.887 DEBUG:boto:Params: {} 2024-04-12T04:06:46.888 DEBUG:boto:Token: None 2024-04-12T04:06:46.888 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.888 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eBy+Ji/1N2E+r6B6R7Bfr95dSn8= 2024-04-12T04:06:46.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eBy+Ji/1N2E+r6B6R7Bfr95dSn8=', 'Content-Length': '0'} 2024-04-12T04:06:46.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000262aec7f7f6e45cd-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.890 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:06:46.890 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:06:46.890 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.890 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.890 DEBUG:boto:Method: HEAD 2024-04-12T04:06:46.891 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.891 DEBUG:boto:Data: 2024-04-12T04:06:46.891 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.891 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.891 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.891 DEBUG:boto:Params: {} 2024-04-12T04:06:46.891 DEBUG:boto:Token: None 2024-04-12T04:06:46.891 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.891 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vFPKlxlJP5MIrNrLnCHb01U+Aro= 2024-04-12T04:06:46.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vFPKlxlJP5MIrNrLnCHb01U+Aro=', 'Content-Length': '0'} 2024-04-12T04:06:46.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ab7c390e2326a07-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.893 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:06:46.893 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:06:46.894 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.894 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.894 DEBUG:boto:Method: HEAD 2024-04-12T04:06:46.894 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.894 DEBUG:boto:Data: 2024-04-12T04:06:46.894 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.894 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.894 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.894 DEBUG:boto:Params: {} 2024-04-12T04:06:46.894 DEBUG:boto:Token: None 2024-04-12T04:06:46.894 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.895 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vFPKlxlJP5MIrNrLnCHb01U+Aro= 2024-04-12T04:06:46.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vFPKlxlJP5MIrNrLnCHb01U+Aro=', 'Content-Length': '0'} 2024-04-12T04:06:46.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071141495f6587197-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:06:46.897 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:06:46.897 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:06:46.897 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.897 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.897 DEBUG:boto:Method: GET 2024-04-12T04:06:46.897 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.898 DEBUG:boto:Data: 2024-04-12T04:06:46.898 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.898 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.898 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.898 DEBUG:boto:Params: {} 2024-04-12T04:06:46.898 DEBUG:boto:Token: None 2024-04-12T04:06:46.898 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.898 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eBy+Ji/1N2E+r6B6R7Bfr95dSn8= 2024-04-12T04:06:46.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eBy+Ji/1N2E+r6B6R7Bfr95dSn8=', 'Content-Length': '0'} 2024-04-12T04:06:46.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f0a3bb515853f95-006618b356-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.942 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:06:46.942 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:06:46.943 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.943 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.943 DEBUG:boto:Method: GET 2024-04-12T04:06:46.943 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.943 DEBUG:boto:Data: 2024-04-12T04:06:46.943 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.943 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.943 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.943 DEBUG:boto:Params: {} 2024-04-12T04:06:46.944 DEBUG:boto:Token: None 2024-04-12T04:06:46.944 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:06:46.944 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eBy+Ji/1N2E+r6B6R7Bfr95dSn8= 2024-04-12T04:06:46.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eBy+Ji/1N2E+r6B6R7Bfr95dSn8=', 'Content-Length': '0'} 2024-04-12T04:06:46.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026c5157d0ef7d832-006618b356-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.946 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.947 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.947 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:06:46.947 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:06:46.947 DEBUG:boto:Method: GET 2024-04-12T04:06:46.947 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:06:46.947 DEBUG:boto:Data: 2024-04-12T04:06:46.947 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.947 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:46.947 DEBUG:boto:Port: 8001 2024-04-12T04:06:46.947 DEBUG:boto:Params: {} 2024-04-12T04:06:46.948 DEBUG:boto:Token: None 2024-04-12T04:06:46.948 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/ 2024-04-12T04:06:46.948 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0= 2024-04-12T04:06:46.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0=', 'Content-Length': '0'} 2024-04-12T04:06:46.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076b58dd6b811c728-006618b356-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:46.950 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:46.951 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:46.951 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:46.951 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:06:46.951 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:06:46.951 DEBUG:boto:Method: GET 2024-04-12T04:06:46.951 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:06:46.951 DEBUG:boto:Data: 2024-04-12T04:06:46.952 DEBUG:boto:Headers: {} 2024-04-12T04:06:46.952 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:46.952 DEBUG:boto:Port: 8000 2024-04-12T04:06:46.952 DEBUG:boto:Params: {} 2024-04-12T04:06:46.952 DEBUG:boto:Token: None 2024-04-12T04:06:46.952 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:46 GMT /dkyges-61/ 2024-04-12T04:06:46.952 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0= 2024-04-12T04:06:46.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMmSMTMNsEQZ63sYwwh2mSVPkC0=', 'Content-Length': '0'} 2024-04-12T04:06:46.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d70ae66ebd14d627-006618b356-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.002 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.003 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:06:47.003 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:06:47.003 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.003 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:06:47.003 DEBUG:boto:Data: 2024-04-12T04:06:47.003 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.003 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.003 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.004 DEBUG:boto:Params: {} 2024-04-12T04:06:47.004 DEBUG:boto:Token: None 2024-04-12T04:06:47.004 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj19 2024-04-12T04:06:47.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bzf0cdW0MsVmgRoET+nuO3i8oFE= 2024-04-12T04:06:47.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bzf0cdW0MsVmgRoET+nuO3i8oFE=', 'Content-Length': '0'} 2024-04-12T04:06:47.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090a17a4a7043d8ee-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.006 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:06:47.006 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:06:47.006 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.007 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:06:47.007 DEBUG:boto:Data: 2024-04-12T04:06:47.007 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.007 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.007 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.007 DEBUG:boto:Params: {} 2024-04-12T04:06:47.007 DEBUG:boto:Token: None 2024-04-12T04:06:47.007 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj19 2024-04-12T04:06:47.007 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bzf0cdW0MsVmgRoET+nuO3i8oFE= 2024-04-12T04:06:47.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bzf0cdW0MsVmgRoET+nuO3i8oFE=', 'Content-Length': '0'} 2024-04-12T04:06:47.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d010068457fb15b5-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:06:47.009 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:06:47.010 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:06:47.010 DEBUG:boto:Method: GET 2024-04-12T04:06:47.010 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:06:47.010 DEBUG:boto:Data: 2024-04-12T04:06:47.010 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.010 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.010 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.010 DEBUG:boto:Params: {} 2024-04-12T04:06:47.010 DEBUG:boto:Token: None 2024-04-12T04:06:47.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj19 2024-04-12T04:06:47.011 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:US83B95rA1RS+rLpBU6OFrvXW5E= 2024-04-12T04:06:47.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:US83B95rA1RS+rLpBU6OFrvXW5E=', 'Content-Length': '0'} 2024-04-12T04:06:47.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007770f3cc5369e15f-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.013 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:06:47.013 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:06:47.013 DEBUG:boto:Method: GET 2024-04-12T04:06:47.013 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:06:47.013 DEBUG:boto:Data: 2024-04-12T04:06:47.013 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.013 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.014 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.014 DEBUG:boto:Params: {} 2024-04-12T04:06:47.014 DEBUG:boto:Token: None 2024-04-12T04:06:47.014 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj19 2024-04-12T04:06:47.014 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:US83B95rA1RS+rLpBU6OFrvXW5E= 2024-04-12T04:06:47.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:US83B95rA1RS+rLpBU6OFrvXW5E=', 'Content-Length': '0'} 2024-04-12T04:06:47.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003682e55dbd362262-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.016 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.016 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.017 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:06:47.017 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:06:47.017 DEBUG:boto:Method: GET 2024-04-12T04:06:47.017 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:06:47.017 DEBUG:boto:Data: 2024-04-12T04:06:47.017 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.017 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.017 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.017 DEBUG:boto:Params: {} 2024-04-12T04:06:47.017 DEBUG:boto:Token: None 2024-04-12T04:06:47.018 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.018 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007844125f9915f1b5-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.020 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.021 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.021 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.021 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:06:47.021 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:06:47.021 DEBUG:boto:Method: GET 2024-04-12T04:06:47.021 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:06:47.021 DEBUG:boto:Data: 2024-04-12T04:06:47.021 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.022 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.022 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.022 DEBUG:boto:Params: {} 2024-04-12T04:06:47.022 DEBUG:boto:Token: None 2024-04-12T04:06:47.022 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.022 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009913e3906d3521d4-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.025 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:06:47.026 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:06:47.026 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:06:47.026 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.026 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.026 DEBUG:boto:Method: GET 2024-04-12T04:06:47.026 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.026 DEBUG:boto:Data: 2024-04-12T04:06:47.027 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.027 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.027 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.027 DEBUG:boto:Params: {} 2024-04-12T04:06:47.027 DEBUG:boto:Token: None 2024-04-12T04:06:47.027 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.027 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nW6m9LnZ3xiDYGN5ZhbJtc9Ut14= 2024-04-12T04:06:47.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nW6m9LnZ3xiDYGN5ZhbJtc9Ut14=', 'Content-Length': '0'} 2024-04-12T04:06:47.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032612279c4b91c3d-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.029 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:06:47.029 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:06:47.030 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.030 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.030 DEBUG:boto:Method: GET 2024-04-12T04:06:47.030 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.030 DEBUG:boto:Data: 2024-04-12T04:06:47.030 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.030 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.030 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.030 DEBUG:boto:Params: {} 2024-04-12T04:06:47.030 DEBUG:boto:Token: None 2024-04-12T04:06:47.031 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.031 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nW6m9LnZ3xiDYGN5ZhbJtc9Ut14= 2024-04-12T04:06:47.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nW6m9LnZ3xiDYGN5ZhbJtc9Ut14=', 'Content-Length': '0'} 2024-04-12T04:06:47.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa4ca57172d3a32e-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.033 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:06:47.033 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:06:47.033 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.033 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.034 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.034 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.034 DEBUG:boto:Data: 2024-04-12T04:06:47.034 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.034 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.034 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.034 DEBUG:boto:Params: {} 2024-04-12T04:06:47.034 DEBUG:boto:Token: None 2024-04-12T04:06:47.034 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.034 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wdkrM+TrZseGTCX4gEJxEEP33NM= 2024-04-12T04:06:47.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wdkrM+TrZseGTCX4gEJxEEP33NM=', 'Content-Length': '0'} 2024-04-12T04:06:47.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005293a5696c54284b-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.037 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:06:47.037 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:06:47.037 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.037 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.037 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.037 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.037 DEBUG:boto:Data: 2024-04-12T04:06:47.037 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.037 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.038 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.038 DEBUG:boto:Params: {} 2024-04-12T04:06:47.038 DEBUG:boto:Token: None 2024-04-12T04:06:47.038 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.038 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wdkrM+TrZseGTCX4gEJxEEP33NM= 2024-04-12T04:06:47.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wdkrM+TrZseGTCX4gEJxEEP33NM=', 'Content-Length': '0'} 2024-04-12T04:06:47.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d8bb4ef07f8fa0a-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:06:47.040 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:06:47.040 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:06:47.041 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.041 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.041 DEBUG:boto:Method: GET 2024-04-12T04:06:47.041 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.041 DEBUG:boto:Data: 2024-04-12T04:06:47.041 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.041 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.041 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.041 DEBUG:boto:Params: {} 2024-04-12T04:06:47.041 DEBUG:boto:Token: None 2024-04-12T04:06:47.041 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.042 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nW6m9LnZ3xiDYGN5ZhbJtc9Ut14= 2024-04-12T04:06:47.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nW6m9LnZ3xiDYGN5ZhbJtc9Ut14=', 'Content-Length': '0'} 2024-04-12T04:06:47.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4ab55f8a6886df6-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.044 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:06:47.044 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:06:47.044 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.044 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.044 DEBUG:boto:Method: GET 2024-04-12T04:06:47.044 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.044 DEBUG:boto:Data: 2024-04-12T04:06:47.044 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.044 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.044 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.045 DEBUG:boto:Params: {} 2024-04-12T04:06:47.045 DEBUG:boto:Token: None 2024-04-12T04:06:47.045 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:06:47.045 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nW6m9LnZ3xiDYGN5ZhbJtc9Ut14= 2024-04-12T04:06:47.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nW6m9LnZ3xiDYGN5ZhbJtc9Ut14=', 'Content-Length': '0'} 2024-04-12T04:06:47.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be861719b097cc22-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.047 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.047 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.047 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:06:47.048 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:06:47.048 DEBUG:boto:Method: GET 2024-04-12T04:06:47.048 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:06:47.048 DEBUG:boto:Data: 2024-04-12T04:06:47.048 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.048 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.048 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.048 DEBUG:boto:Params: {} 2024-04-12T04:06:47.048 DEBUG:boto:Token: None 2024-04-12T04:06:47.048 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.048 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f4e9056a0924dda-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.051 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.051 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.052 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.052 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:06:47.052 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:06:47.053 DEBUG:boto:Method: GET 2024-04-12T04:06:47.053 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:06:47.053 DEBUG:boto:Data: 2024-04-12T04:06:47.053 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.053 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.053 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.053 DEBUG:boto:Params: {} 2024-04-12T04:06:47.054 DEBUG:boto:Token: None 2024-04-12T04:06:47.054 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.054 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed75a9a059d3ef9a-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.058 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.058 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:06:47.058 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:06:47.058 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.058 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:06:47.058 DEBUG:boto:Data: 2024-04-12T04:06:47.058 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.058 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.059 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.059 DEBUG:boto:Params: {} 2024-04-12T04:06:47.059 DEBUG:boto:Token: None 2024-04-12T04:06:47.059 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj22 2024-04-12T04:06:47.059 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:E5zKpvzFZ0sNBsV3wzQbZ2Si2D4= 2024-04-12T04:06:47.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:E5zKpvzFZ0sNBsV3wzQbZ2Si2D4=', 'Content-Length': '0'} 2024-04-12T04:06:47.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e31adeb265d2fca-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.061 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:06:47.061 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:06:47.061 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.061 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:06:47.061 DEBUG:boto:Data: 2024-04-12T04:06:47.062 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.062 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.062 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.062 DEBUG:boto:Params: {} 2024-04-12T04:06:47.062 DEBUG:boto:Token: None 2024-04-12T04:06:47.062 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj22 2024-04-12T04:06:47.062 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:E5zKpvzFZ0sNBsV3wzQbZ2Si2D4= 2024-04-12T04:06:47.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:E5zKpvzFZ0sNBsV3wzQbZ2Si2D4=', 'Content-Length': '0'} 2024-04-12T04:06:47.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043eb2ae678efa664-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:06:47.064 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:06:47.064 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:06:47.064 DEBUG:boto:Method: GET 2024-04-12T04:06:47.065 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:06:47.065 DEBUG:boto:Data: 2024-04-12T04:06:47.065 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.065 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.065 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.065 DEBUG:boto:Params: {} 2024-04-12T04:06:47.065 DEBUG:boto:Token: None 2024-04-12T04:06:47.065 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj22 2024-04-12T04:06:47.065 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ToaGg2XJO8YXGQr1CdifB5uNH0c= 2024-04-12T04:06:47.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ToaGg2XJO8YXGQr1CdifB5uNH0c=', 'Content-Length': '0'} 2024-04-12T04:06:47.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089feb06837e68f26-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.114 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:06:47.114 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:06:47.114 DEBUG:boto:Method: GET 2024-04-12T04:06:47.115 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:06:47.115 DEBUG:boto:Data: 2024-04-12T04:06:47.115 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.115 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.115 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.115 DEBUG:boto:Params: {} 2024-04-12T04:06:47.115 DEBUG:boto:Token: None 2024-04-12T04:06:47.115 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj22 2024-04-12T04:06:47.115 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ToaGg2XJO8YXGQr1CdifB5uNH0c= 2024-04-12T04:06:47.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ToaGg2XJO8YXGQr1CdifB5uNH0c=', 'Content-Length': '0'} 2024-04-12T04:06:47.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d23f27b697974666-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.118 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.118 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.118 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:06:47.118 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:06:47.118 DEBUG:boto:Method: GET 2024-04-12T04:06:47.118 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:06:47.118 DEBUG:boto:Data: 2024-04-12T04:06:47.118 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.119 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.119 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.119 DEBUG:boto:Params: {} 2024-04-12T04:06:47.119 DEBUG:boto:Token: None 2024-04-12T04:06:47.119 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.119 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005371650b257d325d-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.122 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.122 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.122 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.122 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:06:47.123 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:06:47.123 DEBUG:boto:Method: GET 2024-04-12T04:06:47.123 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:06:47.123 DEBUG:boto:Data: 2024-04-12T04:06:47.123 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.123 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.123 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.123 DEBUG:boto:Params: {} 2024-04-12T04:06:47.123 DEBUG:boto:Token: None 2024-04-12T04:06:47.123 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.123 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c7df870128b6ed6-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.127 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:06:47.127 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:06:47.127 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:06:47.128 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.128 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.128 DEBUG:boto:Method: GET 2024-04-12T04:06:47.128 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.128 DEBUG:boto:Data: 2024-04-12T04:06:47.128 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.128 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.128 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.128 DEBUG:boto:Params: {} 2024-04-12T04:06:47.128 DEBUG:boto:Token: None 2024-04-12T04:06:47.129 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.129 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VYmzvZqViw1J+azqu/ttNTFZVC4= 2024-04-12T04:06:47.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VYmzvZqViw1J+azqu/ttNTFZVC4=', 'Content-Length': '0'} 2024-04-12T04:06:47.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a32323bfa3760861-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.131 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:06:47.131 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:06:47.131 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.131 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.131 DEBUG:boto:Method: GET 2024-04-12T04:06:47.131 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.131 DEBUG:boto:Data: 2024-04-12T04:06:47.132 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.132 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.132 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.132 DEBUG:boto:Params: {} 2024-04-12T04:06:47.132 DEBUG:boto:Token: None 2024-04-12T04:06:47.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.132 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VYmzvZqViw1J+azqu/ttNTFZVC4= 2024-04-12T04:06:47.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VYmzvZqViw1J+azqu/ttNTFZVC4=', 'Content-Length': '0'} 2024-04-12T04:06:47.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0f8654a23a5de49-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.178 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:06:47.179 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:06:47.179 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.179 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.179 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.179 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.179 DEBUG:boto:Data: 2024-04-12T04:06:47.179 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.179 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.179 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.179 DEBUG:boto:Params: {} 2024-04-12T04:06:47.179 DEBUG:boto:Token: None 2024-04-12T04:06:47.180 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.180 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CKadWKT5KY+sEB3KDjzzBf0LKCo= 2024-04-12T04:06:47.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CKadWKT5KY+sEB3KDjzzBf0LKCo=', 'Content-Length': '0'} 2024-04-12T04:06:47.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ff7ae849be7fae4-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.182 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:06:47.183 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:06:47.183 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.183 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.183 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.183 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.183 DEBUG:boto:Data: 2024-04-12T04:06:47.183 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.183 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.183 DEBUG:boto:Params: {} 2024-04-12T04:06:47.183 DEBUG:boto:Token: None 2024-04-12T04:06:47.184 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.184 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CKadWKT5KY+sEB3KDjzzBf0LKCo= 2024-04-12T04:06:47.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CKadWKT5KY+sEB3KDjzzBf0LKCo=', 'Content-Length': '0'} 2024-04-12T04:06:47.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb3007decfec82b8-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:06:47.186 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:06:47.186 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:06:47.186 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.187 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.187 DEBUG:boto:Method: GET 2024-04-12T04:06:47.187 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.187 DEBUG:boto:Data: 2024-04-12T04:06:47.187 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.187 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.187 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.187 DEBUG:boto:Params: {} 2024-04-12T04:06:47.187 DEBUG:boto:Token: None 2024-04-12T04:06:47.188 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.188 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VYmzvZqViw1J+azqu/ttNTFZVC4= 2024-04-12T04:06:47.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VYmzvZqViw1J+azqu/ttNTFZVC4=', 'Content-Length': '0'} 2024-04-12T04:06:47.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063f22f825252f056-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.190 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:06:47.190 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:06:47.191 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.191 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.191 DEBUG:boto:Method: GET 2024-04-12T04:06:47.191 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.191 DEBUG:boto:Data: 2024-04-12T04:06:47.191 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.191 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.191 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.191 DEBUG:boto:Params: {} 2024-04-12T04:06:47.191 DEBUG:boto:Token: None 2024-04-12T04:06:47.192 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:06:47.192 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VYmzvZqViw1J+azqu/ttNTFZVC4= 2024-04-12T04:06:47.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VYmzvZqViw1J+azqu/ttNTFZVC4=', 'Content-Length': '0'} 2024-04-12T04:06:47.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e48ff9903e569034-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.194 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.194 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.194 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:06:47.194 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:06:47.194 DEBUG:boto:Method: GET 2024-04-12T04:06:47.194 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:06:47.194 DEBUG:boto:Data: 2024-04-12T04:06:47.194 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.194 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.195 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.195 DEBUG:boto:Params: {} 2024-04-12T04:06:47.195 DEBUG:boto:Token: None 2024-04-12T04:06:47.195 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.195 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.197 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d92a629676b6f23f-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.198 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.198 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.198 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.198 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:06:47.198 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:06:47.199 DEBUG:boto:Method: GET 2024-04-12T04:06:47.199 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:06:47.199 DEBUG:boto:Data: 2024-04-12T04:06:47.199 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.199 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.199 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.199 DEBUG:boto:Params: {} 2024-04-12T04:06:47.199 DEBUG:boto:Token: None 2024-04-12T04:06:47.199 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.199 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006aaaa5f7a84355b2-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.203 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.203 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:06:47.203 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:06:47.203 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.203 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:06:47.203 DEBUG:boto:Data: 2024-04-12T04:06:47.203 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.204 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.204 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.204 DEBUG:boto:Params: {} 2024-04-12T04:06:47.204 DEBUG:boto:Token: None 2024-04-12T04:06:47.204 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj23 2024-04-12T04:06:47.204 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hpCtnZnVatMEYGE0vnTkjTQkFHo= 2024-04-12T04:06:47.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hpCtnZnVatMEYGE0vnTkjTQkFHo=', 'Content-Length': '0'} 2024-04-12T04:06:47.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1e4e6f82e5f030f-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.206 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:06:47.206 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:06:47.206 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.206 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:06:47.207 DEBUG:boto:Data: 2024-04-12T04:06:47.207 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.207 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.207 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.207 DEBUG:boto:Params: {} 2024-04-12T04:06:47.207 DEBUG:boto:Token: None 2024-04-12T04:06:47.207 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj23 2024-04-12T04:06:47.207 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hpCtnZnVatMEYGE0vnTkjTQkFHo= 2024-04-12T04:06:47.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hpCtnZnVatMEYGE0vnTkjTQkFHo=', 'Content-Length': '0'} 2024-04-12T04:06:47.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b999da576138fe3-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:06:47.210 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:06:47.210 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:06:47.211 DEBUG:boto:Method: GET 2024-04-12T04:06:47.211 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:06:47.211 DEBUG:boto:Data: 2024-04-12T04:06:47.211 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.211 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.211 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.211 DEBUG:boto:Params: {} 2024-04-12T04:06:47.211 DEBUG:boto:Token: None 2024-04-12T04:06:47.211 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj23 2024-04-12T04:06:47.211 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a0dF9CSd3ANNCUuQPgYrbYiEC0A= 2024-04-12T04:06:47.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a0dF9CSd3ANNCUuQPgYrbYiEC0A=', 'Content-Length': '0'} 2024-04-12T04:06:47.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beb07f5cf03438ab-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.214 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:06:47.214 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:06:47.214 DEBUG:boto:Method: GET 2024-04-12T04:06:47.214 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:06:47.214 DEBUG:boto:Data: 2024-04-12T04:06:47.214 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.214 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.214 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.214 DEBUG:boto:Params: {} 2024-04-12T04:06:47.215 DEBUG:boto:Token: None 2024-04-12T04:06:47.215 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj23 2024-04-12T04:06:47.215 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a0dF9CSd3ANNCUuQPgYrbYiEC0A= 2024-04-12T04:06:47.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a0dF9CSd3ANNCUuQPgYrbYiEC0A=', 'Content-Length': '0'} 2024-04-12T04:06:47.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d56e47afa092f2a4-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.217 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.217 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.217 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:06:47.217 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:06:47.218 DEBUG:boto:Method: GET 2024-04-12T04:06:47.218 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:06:47.218 DEBUG:boto:Data: 2024-04-12T04:06:47.218 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.218 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.218 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.218 DEBUG:boto:Params: {} 2024-04-12T04:06:47.218 DEBUG:boto:Token: None 2024-04-12T04:06:47.218 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.219 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f1a9ffd9d93a82a-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.221 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.222 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.222 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.222 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:06:47.222 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:06:47.222 DEBUG:boto:Method: GET 2024-04-12T04:06:47.222 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:06:47.222 DEBUG:boto:Data: 2024-04-12T04:06:47.222 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.223 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.223 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.223 DEBUG:boto:Params: {} 2024-04-12T04:06:47.223 DEBUG:boto:Token: None 2024-04-12T04:06:47.223 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.223 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008146ab0d07f768e7-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.227 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:06:47.227 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:06:47.227 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:06:47.227 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.227 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.227 DEBUG:boto:Method: GET 2024-04-12T04:06:47.228 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.228 DEBUG:boto:Data: 2024-04-12T04:06:47.228 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.228 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.228 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.228 DEBUG:boto:Params: {} 2024-04-12T04:06:47.228 DEBUG:boto:Token: None 2024-04-12T04:06:47.228 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.228 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7v+3uJiqZCTPqbZ8Xr12XbU3qHo= 2024-04-12T04:06:47.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7v+3uJiqZCTPqbZ8Xr12XbU3qHo=', 'Content-Length': '0'} 2024-04-12T04:06:47.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ab005ba89d0d6e7-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.231 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:06:47.231 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:06:47.231 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.231 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.231 DEBUG:boto:Method: GET 2024-04-12T04:06:47.231 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.231 DEBUG:boto:Data: 2024-04-12T04:06:47.231 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.231 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.231 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.231 DEBUG:boto:Params: {} 2024-04-12T04:06:47.232 DEBUG:boto:Token: None 2024-04-12T04:06:47.232 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.232 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7v+3uJiqZCTPqbZ8Xr12XbU3qHo= 2024-04-12T04:06:47.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7v+3uJiqZCTPqbZ8Xr12XbU3qHo=', 'Content-Length': '0'} 2024-04-12T04:06:47.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061a9e4c7ab2fe496-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.274 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:06:47.275 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:06:47.275 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.275 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.275 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.275 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.275 DEBUG:boto:Data: 2024-04-12T04:06:47.275 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.275 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.275 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.275 DEBUG:boto:Params: {} 2024-04-12T04:06:47.276 DEBUG:boto:Token: None 2024-04-12T04:06:47.276 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.276 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d4LQIk7vSo2I+PruCdEPCbDHG60= 2024-04-12T04:06:47.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d4LQIk7vSo2I+PruCdEPCbDHG60=', 'Content-Length': '0'} 2024-04-12T04:06:47.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eb3cb4300fcbd91-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.278 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:06:47.279 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:06:47.279 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.279 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.279 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.279 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.279 DEBUG:boto:Data: 2024-04-12T04:06:47.279 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.279 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.279 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.279 DEBUG:boto:Params: {} 2024-04-12T04:06:47.279 DEBUG:boto:Token: None 2024-04-12T04:06:47.280 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.280 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d4LQIk7vSo2I+PruCdEPCbDHG60= 2024-04-12T04:06:47.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d4LQIk7vSo2I+PruCdEPCbDHG60=', 'Content-Length': '0'} 2024-04-12T04:06:47.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ee7c7e5ea4cd89c-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:06:47.282 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:06:47.282 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:06:47.282 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.282 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.282 DEBUG:boto:Method: GET 2024-04-12T04:06:47.283 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.283 DEBUG:boto:Data: 2024-04-12T04:06:47.283 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.283 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.283 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.283 DEBUG:boto:Params: {} 2024-04-12T04:06:47.283 DEBUG:boto:Token: None 2024-04-12T04:06:47.283 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.283 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7v+3uJiqZCTPqbZ8Xr12XbU3qHo= 2024-04-12T04:06:47.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7v+3uJiqZCTPqbZ8Xr12XbU3qHo=', 'Content-Length': '0'} 2024-04-12T04:06:47.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b664379e0170b5f2-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.286 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:06:47.286 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:06:47.286 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.286 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.286 DEBUG:boto:Method: GET 2024-04-12T04:06:47.286 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.286 DEBUG:boto:Data: 2024-04-12T04:06:47.286 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.286 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.287 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.287 DEBUG:boto:Params: {} 2024-04-12T04:06:47.287 DEBUG:boto:Token: None 2024-04-12T04:06:47.287 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:06:47.287 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7v+3uJiqZCTPqbZ8Xr12XbU3qHo= 2024-04-12T04:06:47.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7v+3uJiqZCTPqbZ8Xr12XbU3qHo=', 'Content-Length': '0'} 2024-04-12T04:06:47.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4245c736dd53dfa-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.289 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.289 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.290 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:06:47.290 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:06:47.290 DEBUG:boto:Method: GET 2024-04-12T04:06:47.290 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:06:47.290 DEBUG:boto:Data: 2024-04-12T04:06:47.290 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.290 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.290 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.290 DEBUG:boto:Params: {} 2024-04-12T04:06:47.290 DEBUG:boto:Token: None 2024-04-12T04:06:47.291 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.291 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096fad5d87ff2007d-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.294 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.294 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.295 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.295 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:06:47.295 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:06:47.295 DEBUG:boto:Method: GET 2024-04-12T04:06:47.295 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:06:47.295 DEBUG:boto:Data: 2024-04-12T04:06:47.295 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.295 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.295 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.295 DEBUG:boto:Params: {} 2024-04-12T04:06:47.296 DEBUG:boto:Token: None 2024-04-12T04:06:47.296 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.296 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.299 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6b3605efa2e373c-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.299 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.300 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:06:47.300 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:06:47.300 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.300 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:06:47.300 DEBUG:boto:Data: 2024-04-12T04:06:47.300 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.300 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.300 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.300 DEBUG:boto:Params: {} 2024-04-12T04:06:47.301 DEBUG:boto:Token: None 2024-04-12T04:06:47.301 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj33 2024-04-12T04:06:47.301 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZJDr8u17iD7uNRo32ORynOw/uPg= 2024-04-12T04:06:47.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZJDr8u17iD7uNRo32ORynOw/uPg=', 'Content-Length': '0'} 2024-04-12T04:06:47.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ae604513ab3b3c5-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.303 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:06:47.303 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:06:47.303 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.303 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:06:47.303 DEBUG:boto:Data: 2024-04-12T04:06:47.303 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.303 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.303 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.304 DEBUG:boto:Params: {} 2024-04-12T04:06:47.304 DEBUG:boto:Token: None 2024-04-12T04:06:47.304 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj33 2024-04-12T04:06:47.304 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZJDr8u17iD7uNRo32ORynOw/uPg= 2024-04-12T04:06:47.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZJDr8u17iD7uNRo32ORynOw/uPg=', 'Content-Length': '0'} 2024-04-12T04:06:47.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082868c6794bfddd8-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:06:47.306 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:06:47.306 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:06:47.306 DEBUG:boto:Method: GET 2024-04-12T04:06:47.306 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:06:47.307 DEBUG:boto:Data: 2024-04-12T04:06:47.307 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.307 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.307 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.307 DEBUG:boto:Params: {} 2024-04-12T04:06:47.307 DEBUG:boto:Token: None 2024-04-12T04:06:47.307 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj33 2024-04-12T04:06:47.307 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z71cnWeDIbcbGKJXFvrSH4Z3S+4= 2024-04-12T04:06:47.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z71cnWeDIbcbGKJXFvrSH4Z3S+4=', 'Content-Length': '0'} 2024-04-12T04:06:47.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d625387d72adf8ec-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.354 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:06:47.355 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:06:47.355 DEBUG:boto:Method: GET 2024-04-12T04:06:47.355 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:06:47.355 DEBUG:boto:Data: 2024-04-12T04:06:47.355 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.355 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.355 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.355 DEBUG:boto:Params: {} 2024-04-12T04:06:47.355 DEBUG:boto:Token: None 2024-04-12T04:06:47.356 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj33 2024-04-12T04:06:47.356 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z71cnWeDIbcbGKJXFvrSH4Z3S+4= 2024-04-12T04:06:47.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z71cnWeDIbcbGKJXFvrSH4Z3S+4=', 'Content-Length': '0'} 2024-04-12T04:06:47.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c268a6d8bfa45bfb-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.358 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.358 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.358 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:06:47.358 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:06:47.359 DEBUG:boto:Method: GET 2024-04-12T04:06:47.359 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:06:47.359 DEBUG:boto:Data: 2024-04-12T04:06:47.359 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.359 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.359 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.359 DEBUG:boto:Params: {} 2024-04-12T04:06:47.359 DEBUG:boto:Token: None 2024-04-12T04:06:47.359 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.359 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac4813fd2f12fc09-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.362 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.363 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.363 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.363 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:06:47.363 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:06:47.363 DEBUG:boto:Method: GET 2024-04-12T04:06:47.363 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:06:47.363 DEBUG:boto:Data: 2024-04-12T04:06:47.363 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.363 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.363 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.364 DEBUG:boto:Params: {} 2024-04-12T04:06:47.364 DEBUG:boto:Token: None 2024-04-12T04:06:47.364 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.364 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000997285a2f57f730e-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.367 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:06:47.368 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:06:47.368 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:06:47.368 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.368 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.368 DEBUG:boto:Method: GET 2024-04-12T04:06:47.369 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.369 DEBUG:boto:Data: 2024-04-12T04:06:47.369 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.369 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.369 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.369 DEBUG:boto:Params: {} 2024-04-12T04:06:47.369 DEBUG:boto:Token: None 2024-04-12T04:06:47.369 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.369 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ln5tgw23ZqZLs0CR+hBcVuBMOcc= 2024-04-12T04:06:47.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ln5tgw23ZqZLs0CR+hBcVuBMOcc=', 'Content-Length': '0'} 2024-04-12T04:06:47.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f32663a4ee2e4a7-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.372 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:06:47.372 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:06:47.372 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.372 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.372 DEBUG:boto:Method: GET 2024-04-12T04:06:47.372 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.372 DEBUG:boto:Data: 2024-04-12T04:06:47.372 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.372 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.372 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.372 DEBUG:boto:Params: {} 2024-04-12T04:06:47.373 DEBUG:boto:Token: None 2024-04-12T04:06:47.373 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.373 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ln5tgw23ZqZLs0CR+hBcVuBMOcc= 2024-04-12T04:06:47.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ln5tgw23ZqZLs0CR+hBcVuBMOcc=', 'Content-Length': '0'} 2024-04-12T04:06:47.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba655456bb75b288-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.418 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:06:47.419 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:06:47.419 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.419 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.419 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.419 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.419 DEBUG:boto:Data: 2024-04-12T04:06:47.419 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.419 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.419 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.419 DEBUG:boto:Params: {} 2024-04-12T04:06:47.420 DEBUG:boto:Token: None 2024-04-12T04:06:47.420 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.420 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Gz147w/BnSuaPJJA7NT0nYzKusQ= 2024-04-12T04:06:47.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Gz147w/BnSuaPJJA7NT0nYzKusQ=', 'Content-Length': '0'} 2024-04-12T04:06:47.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec17274697ce322e-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.422 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:06:47.422 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:06:47.422 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.422 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.422 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.423 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.423 DEBUG:boto:Data: 2024-04-12T04:06:47.423 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.423 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.423 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.423 DEBUG:boto:Params: {} 2024-04-12T04:06:47.423 DEBUG:boto:Token: None 2024-04-12T04:06:47.423 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.423 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Gz147w/BnSuaPJJA7NT0nYzKusQ= 2024-04-12T04:06:47.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Gz147w/BnSuaPJJA7NT0nYzKusQ=', 'Content-Length': '0'} 2024-04-12T04:06:47.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ceafc8cbb2a9e13-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:06:47.426 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:06:47.426 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:06:47.426 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.426 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.426 DEBUG:boto:Method: GET 2024-04-12T04:06:47.426 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.426 DEBUG:boto:Data: 2024-04-12T04:06:47.426 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.426 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.426 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.426 DEBUG:boto:Params: {} 2024-04-12T04:06:47.427 DEBUG:boto:Token: None 2024-04-12T04:06:47.427 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.427 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ln5tgw23ZqZLs0CR+hBcVuBMOcc= 2024-04-12T04:06:47.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ln5tgw23ZqZLs0CR+hBcVuBMOcc=', 'Content-Length': '0'} 2024-04-12T04:06:47.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005952001547c068ee-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.429 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:06:47.429 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:06:47.429 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.429 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.429 DEBUG:boto:Method: GET 2024-04-12T04:06:47.429 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.430 DEBUG:boto:Data: 2024-04-12T04:06:47.430 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.430 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.430 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.430 DEBUG:boto:Params: {} 2024-04-12T04:06:47.430 DEBUG:boto:Token: None 2024-04-12T04:06:47.430 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:06:47.430 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ln5tgw23ZqZLs0CR+hBcVuBMOcc= 2024-04-12T04:06:47.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ln5tgw23ZqZLs0CR+hBcVuBMOcc=', 'Content-Length': '0'} 2024-04-12T04:06:47.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0dbd696be5e6150-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.432 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.433 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.433 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:06:47.433 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:06:47.433 DEBUG:boto:Method: GET 2024-04-12T04:06:47.433 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:06:47.433 DEBUG:boto:Data: 2024-04-12T04:06:47.433 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.433 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.433 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.433 DEBUG:boto:Params: {} 2024-04-12T04:06:47.434 DEBUG:boto:Token: None 2024-04-12T04:06:47.434 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.434 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000efcdb9c3978a01d-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.437 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.437 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.437 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.437 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:06:47.437 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:06:47.437 DEBUG:boto:Method: GET 2024-04-12T04:06:47.438 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:06:47.438 DEBUG:boto:Data: 2024-04-12T04:06:47.438 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.438 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.438 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.438 DEBUG:boto:Params: {} 2024-04-12T04:06:47.438 DEBUG:boto:Token: None 2024-04-12T04:06:47.438 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.438 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4846d6eb24ad990-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.442 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.442 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:06:47.443 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:06:47.443 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.443 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:06:47.443 DEBUG:boto:Data: 2024-04-12T04:06:47.443 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.443 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.443 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.443 DEBUG:boto:Params: {} 2024-04-12T04:06:47.443 DEBUG:boto:Token: None 2024-04-12T04:06:47.444 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj34 2024-04-12T04:06:47.444 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mSkr6STz9dsVcj0phTMPykcA61A= 2024-04-12T04:06:47.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mSkr6STz9dsVcj0phTMPykcA61A=', 'Content-Length': '0'} 2024-04-12T04:06:47.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e95f8f8ddc18ea90-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.446 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:06:47.446 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:06:47.446 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.446 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:06:47.446 DEBUG:boto:Data: 2024-04-12T04:06:47.446 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.446 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.446 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.447 DEBUG:boto:Params: {} 2024-04-12T04:06:47.447 DEBUG:boto:Token: None 2024-04-12T04:06:47.447 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj34 2024-04-12T04:06:47.447 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mSkr6STz9dsVcj0phTMPykcA61A= 2024-04-12T04:06:47.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mSkr6STz9dsVcj0phTMPykcA61A=', 'Content-Length': '0'} 2024-04-12T04:06:47.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c67b5496fbf314ba-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:06:47.449 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:06:47.449 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:06:47.449 DEBUG:boto:Method: GET 2024-04-12T04:06:47.449 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:06:47.450 DEBUG:boto:Data: 2024-04-12T04:06:47.450 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.450 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.450 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.450 DEBUG:boto:Params: {} 2024-04-12T04:06:47.450 DEBUG:boto:Token: None 2024-04-12T04:06:47.450 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj34 2024-04-12T04:06:47.450 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:myauUpmKFqaZKtd0peSda1/XQPI= 2024-04-12T04:06:47.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:myauUpmKFqaZKtd0peSda1/XQPI=', 'Content-Length': '0'} 2024-04-12T04:06:47.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6fb3a74738b4c0e-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.452 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:06:47.452 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:06:47.453 DEBUG:boto:Method: GET 2024-04-12T04:06:47.453 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:06:47.453 DEBUG:boto:Data: 2024-04-12T04:06:47.453 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.453 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.453 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.453 DEBUG:boto:Params: {} 2024-04-12T04:06:47.453 DEBUG:boto:Token: None 2024-04-12T04:06:47.453 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj34 2024-04-12T04:06:47.453 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:myauUpmKFqaZKtd0peSda1/XQPI= 2024-04-12T04:06:47.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:myauUpmKFqaZKtd0peSda1/XQPI=', 'Content-Length': '0'} 2024-04-12T04:06:47.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e76220821914563-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.456 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.456 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.456 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:06:47.456 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:06:47.456 DEBUG:boto:Method: GET 2024-04-12T04:06:47.456 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:06:47.456 DEBUG:boto:Data: 2024-04-12T04:06:47.456 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.456 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.456 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.456 DEBUG:boto:Params: {} 2024-04-12T04:06:47.457 DEBUG:boto:Token: None 2024-04-12T04:06:47.457 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.457 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d9a67dbb1c9109d-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.460 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.460 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.460 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.460 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:06:47.460 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:06:47.460 DEBUG:boto:Method: GET 2024-04-12T04:06:47.460 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:06:47.460 DEBUG:boto:Data: 2024-04-12T04:06:47.461 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.461 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.461 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.461 DEBUG:boto:Params: {} 2024-04-12T04:06:47.461 DEBUG:boto:Token: None 2024-04-12T04:06:47.461 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.461 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4be58ab46687cb9-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.465 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:06:47.466 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:06:47.466 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:06:47.466 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.466 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.466 DEBUG:boto:Method: GET 2024-04-12T04:06:47.466 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.466 DEBUG:boto:Data: 2024-04-12T04:06:47.467 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.467 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.467 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.467 DEBUG:boto:Params: {} 2024-04-12T04:06:47.467 DEBUG:boto:Token: None 2024-04-12T04:06:47.467 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.467 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZncvVOJzJ4vUsV0ve2YmJhPSJX4= 2024-04-12T04:06:47.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZncvVOJzJ4vUsV0ve2YmJhPSJX4=', 'Content-Length': '0'} 2024-04-12T04:06:47.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046ba47df44a1329a-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.510 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:06:47.511 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:06:47.511 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.511 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.511 DEBUG:boto:Method: GET 2024-04-12T04:06:47.511 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.511 DEBUG:boto:Data: 2024-04-12T04:06:47.511 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.511 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.511 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.511 DEBUG:boto:Params: {} 2024-04-12T04:06:47.512 DEBUG:boto:Token: None 2024-04-12T04:06:47.512 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.512 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZncvVOJzJ4vUsV0ve2YmJhPSJX4= 2024-04-12T04:06:47.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZncvVOJzJ4vUsV0ve2YmJhPSJX4=', 'Content-Length': '0'} 2024-04-12T04:06:47.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cb8ff86583e0c44-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.514 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:06:47.514 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:06:47.514 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.515 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.515 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.515 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.515 DEBUG:boto:Data: 2024-04-12T04:06:47.515 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.515 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.515 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.515 DEBUG:boto:Params: {} 2024-04-12T04:06:47.515 DEBUG:boto:Token: None 2024-04-12T04:06:47.515 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.516 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6M5Cibc/4gcJhn0JISxk5b5Gcb4= 2024-04-12T04:06:47.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6M5Cibc/4gcJhn0JISxk5b5Gcb4=', 'Content-Length': '0'} 2024-04-12T04:06:47.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021848884822be97a-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.518 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:06:47.518 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:06:47.518 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.518 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.518 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.518 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.518 DEBUG:boto:Data: 2024-04-12T04:06:47.518 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.518 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.519 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.519 DEBUG:boto:Params: {} 2024-04-12T04:06:47.519 DEBUG:boto:Token: None 2024-04-12T04:06:47.519 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.519 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6M5Cibc/4gcJhn0JISxk5b5Gcb4= 2024-04-12T04:06:47.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6M5Cibc/4gcJhn0JISxk5b5Gcb4=', 'Content-Length': '0'} 2024-04-12T04:06:47.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb772ab0cb4e09b5-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:06:47.521 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:06:47.521 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:06:47.521 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.521 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.522 DEBUG:boto:Method: GET 2024-04-12T04:06:47.522 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.522 DEBUG:boto:Data: 2024-04-12T04:06:47.522 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.522 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.522 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.522 DEBUG:boto:Params: {} 2024-04-12T04:06:47.522 DEBUG:boto:Token: None 2024-04-12T04:06:47.522 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.523 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZncvVOJzJ4vUsV0ve2YmJhPSJX4= 2024-04-12T04:06:47.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZncvVOJzJ4vUsV0ve2YmJhPSJX4=', 'Content-Length': '0'} 2024-04-12T04:06:47.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd95fc9b0da26db3-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.525 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:06:47.525 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:06:47.525 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.525 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.525 DEBUG:boto:Method: GET 2024-04-12T04:06:47.525 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.526 DEBUG:boto:Data: 2024-04-12T04:06:47.526 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.526 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.526 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.526 DEBUG:boto:Params: {} 2024-04-12T04:06:47.526 DEBUG:boto:Token: None 2024-04-12T04:06:47.526 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:06:47.526 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZncvVOJzJ4vUsV0ve2YmJhPSJX4= 2024-04-12T04:06:47.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZncvVOJzJ4vUsV0ve2YmJhPSJX4=', 'Content-Length': '0'} 2024-04-12T04:06:47.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc3fc5634c746bab-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.528 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.529 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.529 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:06:47.529 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:06:47.529 DEBUG:boto:Method: GET 2024-04-12T04:06:47.529 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:06:47.529 DEBUG:boto:Data: 2024-04-12T04:06:47.529 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.529 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.529 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.529 DEBUG:boto:Params: {} 2024-04-12T04:06:47.530 DEBUG:boto:Token: None 2024-04-12T04:06:47.530 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.530 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046cc6d76131c5553-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.533 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.533 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.533 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.533 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:06:47.533 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:06:47.533 DEBUG:boto:Method: GET 2024-04-12T04:06:47.533 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:06:47.534 DEBUG:boto:Data: 2024-04-12T04:06:47.534 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.534 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.534 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.534 DEBUG:boto:Params: {} 2024-04-12T04:06:47.534 DEBUG:boto:Token: None 2024-04-12T04:06:47.534 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.534 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbee98f82b42d36a-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.538 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.538 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:06:47.538 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:06:47.538 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.538 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:06:47.539 DEBUG:boto:Data: 2024-04-12T04:06:47.539 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.539 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.539 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.539 DEBUG:boto:Params: {} 2024-04-12T04:06:47.539 DEBUG:boto:Token: None 2024-04-12T04:06:47.539 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj38 2024-04-12T04:06:47.539 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:INbyySkC07h/YJ+m3Iv7Nn6O3/0= 2024-04-12T04:06:47.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:INbyySkC07h/YJ+m3Iv7Nn6O3/0=', 'Content-Length': '0'} 2024-04-12T04:06:47.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002828493eb920d40d-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.541 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:06:47.541 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:06:47.541 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.541 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:06:47.542 DEBUG:boto:Data: 2024-04-12T04:06:47.542 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.542 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.542 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.542 DEBUG:boto:Params: {} 2024-04-12T04:06:47.542 DEBUG:boto:Token: None 2024-04-12T04:06:47.542 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj38 2024-04-12T04:06:47.542 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:INbyySkC07h/YJ+m3Iv7Nn6O3/0= 2024-04-12T04:06:47.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:INbyySkC07h/YJ+m3Iv7Nn6O3/0=', 'Content-Length': '0'} 2024-04-12T04:06:47.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de3d3abe92c62ca9-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:06:47.545 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:06:47.545 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:06:47.545 DEBUG:boto:Method: GET 2024-04-12T04:06:47.545 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:06:47.545 DEBUG:boto:Data: 2024-04-12T04:06:47.545 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.545 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.545 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.545 DEBUG:boto:Params: {} 2024-04-12T04:06:47.545 DEBUG:boto:Token: None 2024-04-12T04:06:47.546 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj38 2024-04-12T04:06:47.546 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:K6NsRw/0MNsibUdcraavnlgVoh8= 2024-04-12T04:06:47.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:K6NsRw/0MNsibUdcraavnlgVoh8=', 'Content-Length': '0'} 2024-04-12T04:06:47.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000427eac430ac55f2b-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.548 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:06:47.548 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:06:47.548 DEBUG:boto:Method: GET 2024-04-12T04:06:47.548 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:06:47.548 DEBUG:boto:Data: 2024-04-12T04:06:47.548 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.548 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.548 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.549 DEBUG:boto:Params: {} 2024-04-12T04:06:47.549 DEBUG:boto:Token: None 2024-04-12T04:06:47.549 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj38 2024-04-12T04:06:47.549 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:K6NsRw/0MNsibUdcraavnlgVoh8= 2024-04-12T04:06:47.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:K6NsRw/0MNsibUdcraavnlgVoh8=', 'Content-Length': '0'} 2024-04-12T04:06:47.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a07a69a287736245-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.551 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.551 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.551 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:06:47.551 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:06:47.552 DEBUG:boto:Method: GET 2024-04-12T04:06:47.552 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:06:47.552 DEBUG:boto:Data: 2024-04-12T04:06:47.552 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.552 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.552 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.552 DEBUG:boto:Params: {} 2024-04-12T04:06:47.552 DEBUG:boto:Token: None 2024-04-12T04:06:47.552 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.552 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffd3e5e7e1ea059c-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.555 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.555 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.555 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.555 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:06:47.555 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:06:47.556 DEBUG:boto:Method: GET 2024-04-12T04:06:47.556 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:06:47.556 DEBUG:boto:Data: 2024-04-12T04:06:47.556 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.556 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.556 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.556 DEBUG:boto:Params: {} 2024-04-12T04:06:47.556 DEBUG:boto:Token: None 2024-04-12T04:06:47.556 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.557 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4f07c045cfced96-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.602 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:06:47.603 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:06:47.603 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:06:47.603 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.603 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.603 DEBUG:boto:Method: GET 2024-04-12T04:06:47.603 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.603 DEBUG:boto:Data: 2024-04-12T04:06:47.603 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.603 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.603 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.604 DEBUG:boto:Params: {} 2024-04-12T04:06:47.604 DEBUG:boto:Token: None 2024-04-12T04:06:47.604 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EglQ1GoBbpHXnEAhmuzxSX8355E= 2024-04-12T04:06:47.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EglQ1GoBbpHXnEAhmuzxSX8355E=', 'Content-Length': '0'} 2024-04-12T04:06:47.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078eb385bd1d722de-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.606 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:06:47.606 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:06:47.606 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.606 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.607 DEBUG:boto:Method: GET 2024-04-12T04:06:47.607 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.607 DEBUG:boto:Data: 2024-04-12T04:06:47.607 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.607 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.607 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.607 DEBUG:boto:Params: {} 2024-04-12T04:06:47.607 DEBUG:boto:Token: None 2024-04-12T04:06:47.607 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.607 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EglQ1GoBbpHXnEAhmuzxSX8355E= 2024-04-12T04:06:47.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EglQ1GoBbpHXnEAhmuzxSX8355E=', 'Content-Length': '0'} 2024-04-12T04:06:47.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000d7c7da29a49587-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.609 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:06:47.610 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:06:47.610 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.610 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.610 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.610 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.610 DEBUG:boto:Data: 2024-04-12T04:06:47.610 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.610 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.610 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.610 DEBUG:boto:Params: {} 2024-04-12T04:06:47.611 DEBUG:boto:Token: None 2024-04-12T04:06:47.611 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.611 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XfKAxYmte737tl8AKs6ZmYN0V/4= 2024-04-12T04:06:47.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XfKAxYmte737tl8AKs6ZmYN0V/4=', 'Content-Length': '0'} 2024-04-12T04:06:47.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b196bbc977ad2818-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.613 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:06:47.613 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:06:47.613 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.613 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.613 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.613 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.613 DEBUG:boto:Data: 2024-04-12T04:06:47.614 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.614 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.614 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.614 DEBUG:boto:Params: {} 2024-04-12T04:06:47.614 DEBUG:boto:Token: None 2024-04-12T04:06:47.614 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.614 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XfKAxYmte737tl8AKs6ZmYN0V/4= 2024-04-12T04:06:47.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XfKAxYmte737tl8AKs6ZmYN0V/4=', 'Content-Length': '0'} 2024-04-12T04:06:47.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065d5aac3c62717b5-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:06:47.617 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:06:47.617 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:06:47.617 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.617 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.617 DEBUG:boto:Method: GET 2024-04-12T04:06:47.617 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.617 DEBUG:boto:Data: 2024-04-12T04:06:47.617 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.617 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.617 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.618 DEBUG:boto:Params: {} 2024-04-12T04:06:47.618 DEBUG:boto:Token: None 2024-04-12T04:06:47.618 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.618 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EglQ1GoBbpHXnEAhmuzxSX8355E= 2024-04-12T04:06:47.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EglQ1GoBbpHXnEAhmuzxSX8355E=', 'Content-Length': '0'} 2024-04-12T04:06:47.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e9001a06676f113-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.620 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:06:47.620 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:06:47.620 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.620 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.620 DEBUG:boto:Method: GET 2024-04-12T04:06:47.621 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.621 DEBUG:boto:Data: 2024-04-12T04:06:47.621 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.621 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.621 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.621 DEBUG:boto:Params: {} 2024-04-12T04:06:47.621 DEBUG:boto:Token: None 2024-04-12T04:06:47.621 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:06:47.621 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EglQ1GoBbpHXnEAhmuzxSX8355E= 2024-04-12T04:06:47.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EglQ1GoBbpHXnEAhmuzxSX8355E=', 'Content-Length': '0'} 2024-04-12T04:06:47.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fafd53d42f2d07d-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.623 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.624 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.624 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:06:47.624 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:06:47.624 DEBUG:boto:Method: GET 2024-04-12T04:06:47.624 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:06:47.624 DEBUG:boto:Data: 2024-04-12T04:06:47.624 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.624 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.624 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.624 DEBUG:boto:Params: {} 2024-04-12T04:06:47.624 DEBUG:boto:Token: None 2024-04-12T04:06:47.625 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.625 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f03cb4f58229b805-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.627 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.628 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.628 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.628 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:06:47.628 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:06:47.628 DEBUG:boto:Method: GET 2024-04-12T04:06:47.628 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:06:47.628 DEBUG:boto:Data: 2024-04-12T04:06:47.628 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.628 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.628 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.628 DEBUG:boto:Params: {} 2024-04-12T04:06:47.628 DEBUG:boto:Token: None 2024-04-12T04:06:47.629 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.629 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.632 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008252877967306531-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.633 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.633 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:06:47.633 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:06:47.633 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.633 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:06:47.633 DEBUG:boto:Data: 2024-04-12T04:06:47.633 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.633 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.634 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.634 DEBUG:boto:Params: {} 2024-04-12T04:06:47.634 DEBUG:boto:Token: None 2024-04-12T04:06:47.634 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj44 2024-04-12T04:06:47.634 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KjKtjAoNYXtJj/Nj/HyEfv+x7Wk= 2024-04-12T04:06:47.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KjKtjAoNYXtJj/Nj/HyEfv+x7Wk=', 'Content-Length': '0'} 2024-04-12T04:06:47.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000006a33ea774401b6-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.636 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:06:47.636 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:06:47.637 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.637 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:06:47.637 DEBUG:boto:Data: 2024-04-12T04:06:47.637 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.637 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.637 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.637 DEBUG:boto:Params: {} 2024-04-12T04:06:47.637 DEBUG:boto:Token: None 2024-04-12T04:06:47.637 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj44 2024-04-12T04:06:47.638 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KjKtjAoNYXtJj/Nj/HyEfv+x7Wk= 2024-04-12T04:06:47.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KjKtjAoNYXtJj/Nj/HyEfv+x7Wk=', 'Content-Length': '0'} 2024-04-12T04:06:47.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3f05b72e249c289-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:06:47.640 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:06:47.640 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:06:47.640 DEBUG:boto:Method: GET 2024-04-12T04:06:47.640 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:06:47.640 DEBUG:boto:Data: 2024-04-12T04:06:47.640 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.640 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.640 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.640 DEBUG:boto:Params: {} 2024-04-12T04:06:47.641 DEBUG:boto:Token: None 2024-04-12T04:06:47.641 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj44 2024-04-12T04:06:47.641 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ziz3xnJmj+9hyDREkPENYGdKKPA= 2024-04-12T04:06:47.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ziz3xnJmj+9hyDREkPENYGdKKPA=', 'Content-Length': '0'} 2024-04-12T04:06:47.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7805080944cc63d-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.643 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:06:47.643 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:06:47.643 DEBUG:boto:Method: GET 2024-04-12T04:06:47.643 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:06:47.643 DEBUG:boto:Data: 2024-04-12T04:06:47.643 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.644 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.644 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.644 DEBUG:boto:Params: {} 2024-04-12T04:06:47.644 DEBUG:boto:Token: None 2024-04-12T04:06:47.644 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj44 2024-04-12T04:06:47.644 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ziz3xnJmj+9hyDREkPENYGdKKPA= 2024-04-12T04:06:47.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ziz3xnJmj+9hyDREkPENYGdKKPA=', 'Content-Length': '0'} 2024-04-12T04:06:47.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a057da6ece2e6119-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.687 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.687 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.687 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:06:47.687 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:06:47.687 DEBUG:boto:Method: GET 2024-04-12T04:06:47.688 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:06:47.688 DEBUG:boto:Data: 2024-04-12T04:06:47.688 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.688 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.688 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.688 DEBUG:boto:Params: {} 2024-04-12T04:06:47.688 DEBUG:boto:Token: None 2024-04-12T04:06:47.688 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.689 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.691 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b13df4116fd0448d-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.691 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.692 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.692 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.692 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:06:47.692 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:06:47.692 DEBUG:boto:Method: GET 2024-04-12T04:06:47.692 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:06:47.693 DEBUG:boto:Data: 2024-04-12T04:06:47.693 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.693 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.693 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.693 DEBUG:boto:Params: {} 2024-04-12T04:06:47.693 DEBUG:boto:Token: None 2024-04-12T04:06:47.693 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.693 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.696 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e7ef4d967fec65c-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.696 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:06:47.697 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:06:47.697 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:06:47.697 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.697 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.697 DEBUG:boto:Method: GET 2024-04-12T04:06:47.697 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.698 DEBUG:boto:Data: 2024-04-12T04:06:47.698 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.698 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.698 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.698 DEBUG:boto:Params: {} 2024-04-12T04:06:47.698 DEBUG:boto:Token: None 2024-04-12T04:06:47.698 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.698 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r79i1Qczu51iqvh4icFQKpb2btk= 2024-04-12T04:06:47.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r79i1Qczu51iqvh4icFQKpb2btk=', 'Content-Length': '0'} 2024-04-12T04:06:47.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ab01a07743a0bc0-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.746 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:06:47.746 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:06:47.746 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.747 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.747 DEBUG:boto:Method: GET 2024-04-12T04:06:47.747 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.747 DEBUG:boto:Data: 2024-04-12T04:06:47.747 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.747 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.747 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.747 DEBUG:boto:Params: {} 2024-04-12T04:06:47.747 DEBUG:boto:Token: None 2024-04-12T04:06:47.747 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.748 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r79i1Qczu51iqvh4icFQKpb2btk= 2024-04-12T04:06:47.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r79i1Qczu51iqvh4icFQKpb2btk=', 'Content-Length': '0'} 2024-04-12T04:06:47.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000305c06d21218239a-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.750 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:06:47.750 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:06:47.750 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.750 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.750 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.750 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.751 DEBUG:boto:Data: 2024-04-12T04:06:47.751 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.751 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.751 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.751 DEBUG:boto:Params: {} 2024-04-12T04:06:47.751 DEBUG:boto:Token: None 2024-04-12T04:06:47.751 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.751 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IxHb18b51xWorZ8htzVDLRK/egE= 2024-04-12T04:06:47.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IxHb18b51xWorZ8htzVDLRK/egE=', 'Content-Length': '0'} 2024-04-12T04:06:47.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000387e9e441f978a8c-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.753 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:06:47.753 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:06:47.753 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.754 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.754 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.754 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.754 DEBUG:boto:Data: 2024-04-12T04:06:47.754 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.754 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.754 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.754 DEBUG:boto:Params: {} 2024-04-12T04:06:47.754 DEBUG:boto:Token: None 2024-04-12T04:06:47.754 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.755 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IxHb18b51xWorZ8htzVDLRK/egE= 2024-04-12T04:06:47.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IxHb18b51xWorZ8htzVDLRK/egE=', 'Content-Length': '0'} 2024-04-12T04:06:47.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7826f034a86b187-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:06:47.757 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:06:47.757 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:06:47.757 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.757 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.757 DEBUG:boto:Method: GET 2024-04-12T04:06:47.757 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.757 DEBUG:boto:Data: 2024-04-12T04:06:47.757 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.757 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.758 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.758 DEBUG:boto:Params: {} 2024-04-12T04:06:47.758 DEBUG:boto:Token: None 2024-04-12T04:06:47.758 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.758 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r79i1Qczu51iqvh4icFQKpb2btk= 2024-04-12T04:06:47.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r79i1Qczu51iqvh4icFQKpb2btk=', 'Content-Length': '0'} 2024-04-12T04:06:47.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ffc23dfa5401c56-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.760 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:06:47.760 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:06:47.760 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.760 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.761 DEBUG:boto:Method: GET 2024-04-12T04:06:47.761 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.761 DEBUG:boto:Data: 2024-04-12T04:06:47.761 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.761 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.761 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.761 DEBUG:boto:Params: {} 2024-04-12T04:06:47.761 DEBUG:boto:Token: None 2024-04-12T04:06:47.761 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:06:47.761 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r79i1Qczu51iqvh4icFQKpb2btk= 2024-04-12T04:06:47.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r79i1Qczu51iqvh4icFQKpb2btk=', 'Content-Length': '0'} 2024-04-12T04:06:47.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca869fbbeeaac092-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.764 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.764 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.764 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:06:47.764 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:06:47.764 DEBUG:boto:Method: GET 2024-04-12T04:06:47.764 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:06:47.764 DEBUG:boto:Data: 2024-04-12T04:06:47.764 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.764 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.764 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.765 DEBUG:boto:Params: {} 2024-04-12T04:06:47.765 DEBUG:boto:Token: None 2024-04-12T04:06:47.765 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.765 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.767 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041ebf0484a299706-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.767 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.768 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.768 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.768 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:06:47.768 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:06:47.768 DEBUG:boto:Method: GET 2024-04-12T04:06:47.768 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:06:47.768 DEBUG:boto:Data: 2024-04-12T04:06:47.768 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.768 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.769 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.769 DEBUG:boto:Params: {} 2024-04-12T04:06:47.769 DEBUG:boto:Token: None 2024-04-12T04:06:47.769 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.769 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c34f0137e2927c8-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.772 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.772 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:06:47.772 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:06:47.773 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.773 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:06:47.773 DEBUG:boto:Data: 2024-04-12T04:06:47.773 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.773 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.773 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.773 DEBUG:boto:Params: {} 2024-04-12T04:06:47.773 DEBUG:boto:Token: None 2024-04-12T04:06:47.773 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj46 2024-04-12T04:06:47.773 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2vHLC0H2OazrkMxM2QoRs1h4uDI= 2024-04-12T04:06:47.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2vHLC0H2OazrkMxM2QoRs1h4uDI=', 'Content-Length': '0'} 2024-04-12T04:06:47.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040a549dc4494367f-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.775 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:06:47.776 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:06:47.776 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.776 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:06:47.776 DEBUG:boto:Data: 2024-04-12T04:06:47.776 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.776 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.776 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.776 DEBUG:boto:Params: {} 2024-04-12T04:06:47.776 DEBUG:boto:Token: None 2024-04-12T04:06:47.776 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj46 2024-04-12T04:06:47.776 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2vHLC0H2OazrkMxM2QoRs1h4uDI= 2024-04-12T04:06:47.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2vHLC0H2OazrkMxM2QoRs1h4uDI=', 'Content-Length': '0'} 2024-04-12T04:06:47.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006caea1bac73ee721-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:06:47.778 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:06:47.779 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:06:47.779 DEBUG:boto:Method: GET 2024-04-12T04:06:47.779 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:06:47.779 DEBUG:boto:Data: 2024-04-12T04:06:47.779 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.779 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.779 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.779 DEBUG:boto:Params: {} 2024-04-12T04:06:47.779 DEBUG:boto:Token: None 2024-04-12T04:06:47.779 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj46 2024-04-12T04:06:47.780 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lbVcv45fJNs0EhRcp6zqcD5otxs= 2024-04-12T04:06:47.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lbVcv45fJNs0EhRcp6zqcD5otxs=', 'Content-Length': '0'} 2024-04-12T04:06:47.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbba297e270ed9e1-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.822 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:06:47.822 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:06:47.823 DEBUG:boto:Method: GET 2024-04-12T04:06:47.823 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:06:47.823 DEBUG:boto:Data: 2024-04-12T04:06:47.823 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.823 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.823 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.823 DEBUG:boto:Params: {} 2024-04-12T04:06:47.823 DEBUG:boto:Token: None 2024-04-12T04:06:47.823 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj46 2024-04-12T04:06:47.824 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lbVcv45fJNs0EhRcp6zqcD5otxs= 2024-04-12T04:06:47.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lbVcv45fJNs0EhRcp6zqcD5otxs=', 'Content-Length': '0'} 2024-04-12T04:06:47.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000867f09d7e2c4744a-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.826 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.826 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.826 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:06:47.826 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:06:47.827 DEBUG:boto:Method: GET 2024-04-12T04:06:47.827 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:06:47.827 DEBUG:boto:Data: 2024-04-12T04:06:47.827 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.827 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.827 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.827 DEBUG:boto:Params: {} 2024-04-12T04:06:47.827 DEBUG:boto:Token: None 2024-04-12T04:06:47.827 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.827 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000471b30e4f7e2a0ba-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.830 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.830 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.831 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.831 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:06:47.831 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:06:47.831 DEBUG:boto:Method: GET 2024-04-12T04:06:47.831 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:06:47.831 DEBUG:boto:Data: 2024-04-12T04:06:47.831 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.831 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.831 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.831 DEBUG:boto:Params: {} 2024-04-12T04:06:47.831 DEBUG:boto:Token: None 2024-04-12T04:06:47.832 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.832 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005e460ca918be1ba-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.836 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:06:47.836 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:06:47.836 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:06:47.836 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.836 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.836 DEBUG:boto:Method: GET 2024-04-12T04:06:47.837 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.837 DEBUG:boto:Data: 2024-04-12T04:06:47.837 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.837 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.837 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.837 DEBUG:boto:Params: {} 2024-04-12T04:06:47.837 DEBUG:boto:Token: None 2024-04-12T04:06:47.837 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.837 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cTLTGVKGuNMWXQCaB/PX4NZlBX8= 2024-04-12T04:06:47.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cTLTGVKGuNMWXQCaB/PX4NZlBX8=', 'Content-Length': '0'} 2024-04-12T04:06:47.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d9cf69f8194a148-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.839 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:06:47.840 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:06:47.840 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.840 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.840 DEBUG:boto:Method: GET 2024-04-12T04:06:47.840 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.840 DEBUG:boto:Data: 2024-04-12T04:06:47.840 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.840 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.840 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.840 DEBUG:boto:Params: {} 2024-04-12T04:06:47.840 DEBUG:boto:Token: None 2024-04-12T04:06:47.841 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.841 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cTLTGVKGuNMWXQCaB/PX4NZlBX8= 2024-04-12T04:06:47.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cTLTGVKGuNMWXQCaB/PX4NZlBX8=', 'Content-Length': '0'} 2024-04-12T04:06:47.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007049d837befee39c-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.843 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:06:47.843 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:06:47.843 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.844 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.844 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.844 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.844 DEBUG:boto:Data: 2024-04-12T04:06:47.844 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.844 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.844 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.844 DEBUG:boto:Params: {} 2024-04-12T04:06:47.844 DEBUG:boto:Token: None 2024-04-12T04:06:47.845 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.845 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NtXHypAXssTh6wRpVVkPW1UT7sk= 2024-04-12T04:06:47.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NtXHypAXssTh6wRpVVkPW1UT7sk=', 'Content-Length': '0'} 2024-04-12T04:06:47.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e48158063d8c26b-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.847 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:06:47.847 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:06:47.847 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.848 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.848 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.848 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.848 DEBUG:boto:Data: 2024-04-12T04:06:47.848 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.848 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.848 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.848 DEBUG:boto:Params: {} 2024-04-12T04:06:47.848 DEBUG:boto:Token: None 2024-04-12T04:06:47.848 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.848 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NtXHypAXssTh6wRpVVkPW1UT7sk= 2024-04-12T04:06:47.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NtXHypAXssTh6wRpVVkPW1UT7sk=', 'Content-Length': '0'} 2024-04-12T04:06:47.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000968cb37b04863885-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:06:47.851 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:06:47.851 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:06:47.851 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.851 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.851 DEBUG:boto:Method: GET 2024-04-12T04:06:47.851 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.851 DEBUG:boto:Data: 2024-04-12T04:06:47.851 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.851 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.852 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.852 DEBUG:boto:Params: {} 2024-04-12T04:06:47.852 DEBUG:boto:Token: None 2024-04-12T04:06:47.852 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.852 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cTLTGVKGuNMWXQCaB/PX4NZlBX8= 2024-04-12T04:06:47.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cTLTGVKGuNMWXQCaB/PX4NZlBX8=', 'Content-Length': '0'} 2024-04-12T04:06:47.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db76b9c83166bf0f-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.894 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:06:47.894 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:06:47.895 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.895 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.895 DEBUG:boto:Method: GET 2024-04-12T04:06:47.895 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.895 DEBUG:boto:Data: 2024-04-12T04:06:47.895 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.895 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.895 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.895 DEBUG:boto:Params: {} 2024-04-12T04:06:47.895 DEBUG:boto:Token: None 2024-04-12T04:06:47.895 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:06:47.896 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cTLTGVKGuNMWXQCaB/PX4NZlBX8= 2024-04-12T04:06:47.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cTLTGVKGuNMWXQCaB/PX4NZlBX8=', 'Content-Length': '0'} 2024-04-12T04:06:47.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c63ea676dd4e1d96-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.898 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.898 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.898 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:06:47.898 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:06:47.899 DEBUG:boto:Method: GET 2024-04-12T04:06:47.899 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:06:47.899 DEBUG:boto:Data: 2024-04-12T04:06:47.899 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.899 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.899 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.899 DEBUG:boto:Params: {} 2024-04-12T04:06:47.899 DEBUG:boto:Token: None 2024-04-12T04:06:47.899 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.899 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023e8609c4a9c6938-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.902 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.903 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.903 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.903 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:06:47.903 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:06:47.903 DEBUG:boto:Method: GET 2024-04-12T04:06:47.903 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:06:47.903 DEBUG:boto:Data: 2024-04-12T04:06:47.903 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.903 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.903 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.904 DEBUG:boto:Params: {} 2024-04-12T04:06:47.904 DEBUG:boto:Token: None 2024-04-12T04:06:47.904 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.904 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c139ec42fea7eb0e-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.908 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.908 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:06:47.908 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:06:47.908 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.908 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:06:47.908 DEBUG:boto:Data: 2024-04-12T04:06:47.908 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.909 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.909 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.909 DEBUG:boto:Params: {} 2024-04-12T04:06:47.909 DEBUG:boto:Token: None 2024-04-12T04:06:47.909 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj51 2024-04-12T04:06:47.909 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OcdUC8AA/zu9PAoOc9zDFUnuIIw= 2024-04-12T04:06:47.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OcdUC8AA/zu9PAoOc9zDFUnuIIw=', 'Content-Length': '0'} 2024-04-12T04:06:47.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068d80316d8071bc9-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.911 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:06:47.911 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:06:47.912 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.912 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:06:47.912 DEBUG:boto:Data: 2024-04-12T04:06:47.912 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.912 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.912 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.912 DEBUG:boto:Params: {} 2024-04-12T04:06:47.912 DEBUG:boto:Token: None 2024-04-12T04:06:47.912 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj51 2024-04-12T04:06:47.912 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OcdUC8AA/zu9PAoOc9zDFUnuIIw= 2024-04-12T04:06:47.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OcdUC8AA/zu9PAoOc9zDFUnuIIw=', 'Content-Length': '0'} 2024-04-12T04:06:47.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082fa09b3cf3ffa46-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:06:47.914 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:06:47.915 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:06:47.915 DEBUG:boto:Method: GET 2024-04-12T04:06:47.915 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:06:47.915 DEBUG:boto:Data: 2024-04-12T04:06:47.915 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.915 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.915 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.915 DEBUG:boto:Params: {} 2024-04-12T04:06:47.915 DEBUG:boto:Token: None 2024-04-12T04:06:47.915 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj51 2024-04-12T04:06:47.916 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BhC2XB+ZIhhG/D+jOsA6uYLC8po= 2024-04-12T04:06:47.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BhC2XB+ZIhhG/D+jOsA6uYLC8po=', 'Content-Length': '0'} 2024-04-12T04:06:47.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b87a1a17ceb0d21-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.958 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:06:47.958 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:06:47.958 DEBUG:boto:Method: GET 2024-04-12T04:06:47.959 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:06:47.959 DEBUG:boto:Data: 2024-04-12T04:06:47.959 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.959 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.959 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.959 DEBUG:boto:Params: {} 2024-04-12T04:06:47.959 DEBUG:boto:Token: None 2024-04-12T04:06:47.959 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj51 2024-04-12T04:06:47.959 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BhC2XB+ZIhhG/D+jOsA6uYLC8po= 2024-04-12T04:06:47.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BhC2XB+ZIhhG/D+jOsA6uYLC8po=', 'Content-Length': '0'} 2024-04-12T04:06:47.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1cf8e379bc94cb6-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.962 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.962 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.962 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:06:47.962 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:06:47.962 DEBUG:boto:Method: GET 2024-04-12T04:06:47.962 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:06:47.963 DEBUG:boto:Data: 2024-04-12T04:06:47.963 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.963 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.963 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.963 DEBUG:boto:Params: {} 2024-04-12T04:06:47.963 DEBUG:boto:Token: None 2024-04-12T04:06:47.963 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.963 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037162e48c61977e7-006618b357-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.966 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.966 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:47.966 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:47.966 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:06:47.967 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:06:47.967 DEBUG:boto:Method: GET 2024-04-12T04:06:47.967 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:06:47.967 DEBUG:boto:Data: 2024-04-12T04:06:47.967 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.967 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.967 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.967 DEBUG:boto:Params: {} 2024-04-12T04:06:47.967 DEBUG:boto:Token: None 2024-04-12T04:06:47.967 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/ 2024-04-12T04:06:47.968 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8= 2024-04-12T04:06:47.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2fm5Lp3cYuaOW646/rY0A+imRy8=', 'Content-Length': '0'} 2024-04-12T04:06:47.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d999023c32630aad-006618b357-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.971 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:47.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:06:47.972 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:06:47.972 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:06:47.972 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.972 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.972 DEBUG:boto:Method: GET 2024-04-12T04:06:47.972 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.972 DEBUG:boto:Data: 2024-04-12T04:06:47.973 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.973 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.973 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.973 DEBUG:boto:Params: {} 2024-04-12T04:06:47.973 DEBUG:boto:Token: None 2024-04-12T04:06:47.973 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.973 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PBg7Z2RJqTwLw2BtjG52elM/g10= 2024-04-12T04:06:47.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PBg7Z2RJqTwLw2BtjG52elM/g10=', 'Content-Length': '0'} 2024-04-12T04:06:47.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b796b0c4ae06b173-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.975 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:06:47.976 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:06:47.976 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.976 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.976 DEBUG:boto:Method: GET 2024-04-12T04:06:47.976 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.976 DEBUG:boto:Data: 2024-04-12T04:06:47.976 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.976 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.976 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.977 DEBUG:boto:Params: {} 2024-04-12T04:06:47.977 DEBUG:boto:Token: None 2024-04-12T04:06:47.977 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.977 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PBg7Z2RJqTwLw2BtjG52elM/g10= 2024-04-12T04:06:47.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PBg7Z2RJqTwLw2BtjG52elM/g10=', 'Content-Length': '0'} 2024-04-12T04:06:47.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba77330b65967047-006618b357-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.979 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:06:47.979 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:06:47.979 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.979 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.980 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.980 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.980 DEBUG:boto:Data: 2024-04-12T04:06:47.980 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.980 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.980 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.980 DEBUG:boto:Params: {} 2024-04-12T04:06:47.980 DEBUG:boto:Token: None 2024-04-12T04:06:47.980 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.980 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B3KK7Ne9kQy7qKR+uR7/H/A55f4= 2024-04-12T04:06:47.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B3KK7Ne9kQy7qKR+uR7/H/A55f4=', 'Content-Length': '0'} 2024-04-12T04:06:47.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050f12d0d4fc64806-006618b357-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.983 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:06:47.983 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:06:47.983 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.983 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.983 DEBUG:boto:Method: HEAD 2024-04-12T04:06:47.983 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.983 DEBUG:boto:Data: 2024-04-12T04:06:47.983 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.983 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:47.983 DEBUG:boto:Port: 8000 2024-04-12T04:06:47.983 DEBUG:boto:Params: {} 2024-04-12T04:06:47.983 DEBUG:boto:Token: None 2024-04-12T04:06:47.984 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.984 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B3KK7Ne9kQy7qKR+uR7/H/A55f4= 2024-04-12T04:06:47.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B3KK7Ne9kQy7qKR+uR7/H/A55f4=', 'Content-Length': '0'} 2024-04-12T04:06:47.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c9e4919c52ad51d-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:47.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:06:47.986 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:06:47.986 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:06:47.986 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.986 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.986 DEBUG:boto:Method: GET 2024-04-12T04:06:47.986 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.987 DEBUG:boto:Data: 2024-04-12T04:06:47.987 DEBUG:boto:Headers: {} 2024-04-12T04:06:47.987 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:47.987 DEBUG:boto:Port: 8001 2024-04-12T04:06:47.987 DEBUG:boto:Params: {} 2024-04-12T04:06:47.987 DEBUG:boto:Token: None 2024-04-12T04:06:47.987 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:47 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:06:47.987 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PBg7Z2RJqTwLw2BtjG52elM/g10= 2024-04-12T04:06:47.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PBg7Z2RJqTwLw2BtjG52elM/g10=', 'Content-Length': '0'} 2024-04-12T04:06:47.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e6f5868466f75c3-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.030 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:06:48.030 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:06:48.030 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:06:48.031 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:06:48.031 DEBUG:boto:Method: GET 2024-04-12T04:06:48.031 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:06:48.031 DEBUG:boto:Data: 2024-04-12T04:06:48.031 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.031 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.031 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.031 DEBUG:boto:Params: {} 2024-04-12T04:06:48.031 DEBUG:boto:Token: None 2024-04-12T04:06:48.031 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:06:48.032 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gcybEfLiYAzudSwxaq+Mhhc7CAA= 2024-04-12T04:06:48.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gcybEfLiYAzudSwxaq+Mhhc7CAA=', 'Content-Length': '0'} 2024-04-12T04:06:48.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025de52e736b0d5e9-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.034 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.034 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.034 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:06:48.034 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:06:48.034 DEBUG:boto:Method: GET 2024-04-12T04:06:48.034 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:06:48.034 DEBUG:boto:Data: 2024-04-12T04:06:48.034 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.035 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.035 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.035 DEBUG:boto:Params: {} 2024-04-12T04:06:48.035 DEBUG:boto:Token: None 2024-04-12T04:06:48.035 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.035 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d86f911da785bb2b-006618b358-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.038 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.038 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.038 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.038 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:06:48.039 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:06:48.039 DEBUG:boto:Method: GET 2024-04-12T04:06:48.039 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:06:48.039 DEBUG:boto:Data: 2024-04-12T04:06:48.039 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.039 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.039 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.039 DEBUG:boto:Params: {} 2024-04-12T04:06:48.039 DEBUG:boto:Token: None 2024-04-12T04:06:48.039 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.039 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b8f14a80eee3e11-006618b358-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.043 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.043 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:06:48.044 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:06:48.044 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.044 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:06:48.044 DEBUG:boto:Data: 2024-04-12T04:06:48.044 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.044 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.044 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.044 DEBUG:boto:Params: {} 2024-04-12T04:06:48.044 DEBUG:boto:Token: None 2024-04-12T04:06:48.044 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj55 2024-04-12T04:06:48.045 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U7Pm4hxUF1514gf6J3EFoBCC8MM= 2024-04-12T04:06:48.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U7Pm4hxUF1514gf6J3EFoBCC8MM=', 'Content-Length': '0'} 2024-04-12T04:06:48.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014e0db261924220c-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.047 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:06:48.047 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:06:48.047 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.047 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:06:48.047 DEBUG:boto:Data: 2024-04-12T04:06:48.047 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.047 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.048 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.048 DEBUG:boto:Params: {} 2024-04-12T04:06:48.048 DEBUG:boto:Token: None 2024-04-12T04:06:48.048 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj55 2024-04-12T04:06:48.048 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U7Pm4hxUF1514gf6J3EFoBCC8MM= 2024-04-12T04:06:48.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U7Pm4hxUF1514gf6J3EFoBCC8MM=', 'Content-Length': '0'} 2024-04-12T04:06:48.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000199c80d77d2be0b6-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:06:48.050 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:06:48.050 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:06:48.050 DEBUG:boto:Method: GET 2024-04-12T04:06:48.051 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:06:48.051 DEBUG:boto:Data: 2024-04-12T04:06:48.051 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.051 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.051 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.051 DEBUG:boto:Params: {} 2024-04-12T04:06:48.051 DEBUG:boto:Token: None 2024-04-12T04:06:48.051 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj55 2024-04-12T04:06:48.051 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Rrk9pzXxTWgyfxoQuEg4UtHZHX8= 2024-04-12T04:06:48.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Rrk9pzXxTWgyfxoQuEg4UtHZHX8=', 'Content-Length': '0'} 2024-04-12T04:06:48.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b2f8105aac2c1dd-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.054 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:06:48.054 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:06:48.054 DEBUG:boto:Method: GET 2024-04-12T04:06:48.054 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:06:48.054 DEBUG:boto:Data: 2024-04-12T04:06:48.054 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.054 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.054 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.054 DEBUG:boto:Params: {} 2024-04-12T04:06:48.055 DEBUG:boto:Token: None 2024-04-12T04:06:48.055 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj55 2024-04-12T04:06:48.055 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Rrk9pzXxTWgyfxoQuEg4UtHZHX8= 2024-04-12T04:06:48.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Rrk9pzXxTWgyfxoQuEg4UtHZHX8=', 'Content-Length': '0'} 2024-04-12T04:06:48.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e585618b9997811-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.098 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.099 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.099 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:06:48.099 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:06:48.099 DEBUG:boto:Method: GET 2024-04-12T04:06:48.099 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:06:48.099 DEBUG:boto:Data: 2024-04-12T04:06:48.099 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.099 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.099 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.099 DEBUG:boto:Params: {} 2024-04-12T04:06:48.099 DEBUG:boto:Token: None 2024-04-12T04:06:48.100 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.100 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000126bb2d96fa753ea-006618b358-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.102 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.103 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.103 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.103 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:06:48.103 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:06:48.103 DEBUG:boto:Method: GET 2024-04-12T04:06:48.103 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:06:48.103 DEBUG:boto:Data: 2024-04-12T04:06:48.103 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.103 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.103 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.103 DEBUG:boto:Params: {} 2024-04-12T04:06:48.104 DEBUG:boto:Token: None 2024-04-12T04:06:48.104 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.104 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095b9f5def0949495-006618b358-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.107 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:06:48.108 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:06:48.108 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:06:48.108 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.108 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.108 DEBUG:boto:Method: GET 2024-04-12T04:06:48.108 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.108 DEBUG:boto:Data: 2024-04-12T04:06:48.108 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.108 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.108 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.108 DEBUG:boto:Params: {} 2024-04-12T04:06:48.109 DEBUG:boto:Token: None 2024-04-12T04:06:48.109 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.109 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fUJBO30TZMOtBqS+Sq6fXowaZOw= 2024-04-12T04:06:48.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fUJBO30TZMOtBqS+Sq6fXowaZOw=', 'Content-Length': '0'} 2024-04-12T04:06:48.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078bd80535f7396ba-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.154 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:06:48.154 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:06:48.155 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.155 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.155 DEBUG:boto:Method: GET 2024-04-12T04:06:48.155 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.155 DEBUG:boto:Data: 2024-04-12T04:06:48.155 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.155 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.155 DEBUG:boto:Params: {} 2024-04-12T04:06:48.155 DEBUG:boto:Token: None 2024-04-12T04:06:48.155 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fUJBO30TZMOtBqS+Sq6fXowaZOw= 2024-04-12T04:06:48.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fUJBO30TZMOtBqS+Sq6fXowaZOw=', 'Content-Length': '0'} 2024-04-12T04:06:48.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9e7f76cd0adccec-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.158 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:06:48.158 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:06:48.158 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.158 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.159 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.159 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.159 DEBUG:boto:Data: 2024-04-12T04:06:48.159 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.159 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.159 DEBUG:boto:Params: {} 2024-04-12T04:06:48.159 DEBUG:boto:Token: None 2024-04-12T04:06:48.159 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.159 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Z6NjrYhtgJD9WG1d3LevG8etCs8= 2024-04-12T04:06:48.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Z6NjrYhtgJD9WG1d3LevG8etCs8=', 'Content-Length': '0'} 2024-04-12T04:06:48.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000584a1b4e71667159-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.162 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:06:48.162 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:06:48.162 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.162 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.162 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.162 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.162 DEBUG:boto:Data: 2024-04-12T04:06:48.162 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.162 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.162 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.162 DEBUG:boto:Params: {} 2024-04-12T04:06:48.163 DEBUG:boto:Token: None 2024-04-12T04:06:48.163 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.163 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Z6NjrYhtgJD9WG1d3LevG8etCs8= 2024-04-12T04:06:48.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Z6NjrYhtgJD9WG1d3LevG8etCs8=', 'Content-Length': '0'} 2024-04-12T04:06:48.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5c4c4a8fb86bf9c-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:06:48.165 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:06:48.165 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:06:48.165 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.166 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.166 DEBUG:boto:Method: GET 2024-04-12T04:06:48.166 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.166 DEBUG:boto:Data: 2024-04-12T04:06:48.166 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.166 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.166 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.166 DEBUG:boto:Params: {} 2024-04-12T04:06:48.166 DEBUG:boto:Token: None 2024-04-12T04:06:48.166 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.167 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fUJBO30TZMOtBqS+Sq6fXowaZOw= 2024-04-12T04:06:48.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fUJBO30TZMOtBqS+Sq6fXowaZOw=', 'Content-Length': '0'} 2024-04-12T04:06:48.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090aea2c8afca0767-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.169 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:06:48.169 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:06:48.169 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.169 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.169 DEBUG:boto:Method: GET 2024-04-12T04:06:48.169 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.169 DEBUG:boto:Data: 2024-04-12T04:06:48.169 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.170 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.170 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.170 DEBUG:boto:Params: {} 2024-04-12T04:06:48.170 DEBUG:boto:Token: None 2024-04-12T04:06:48.170 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:06:48.170 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fUJBO30TZMOtBqS+Sq6fXowaZOw= 2024-04-12T04:06:48.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fUJBO30TZMOtBqS+Sq6fXowaZOw=', 'Content-Length': '0'} 2024-04-12T04:06:48.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000664c4de99deeb32b-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.172 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.173 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.173 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:06:48.173 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:06:48.173 DEBUG:boto:Method: GET 2024-04-12T04:06:48.173 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:06:48.173 DEBUG:boto:Data: 2024-04-12T04:06:48.173 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.173 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.173 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.173 DEBUG:boto:Params: {} 2024-04-12T04:06:48.173 DEBUG:boto:Token: None 2024-04-12T04:06:48.174 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.174 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005043486ba1834a7f-006618b358-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.176 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.177 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.177 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.177 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:06:48.177 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:06:48.177 DEBUG:boto:Method: GET 2024-04-12T04:06:48.177 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:06:48.177 DEBUG:boto:Data: 2024-04-12T04:06:48.177 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.178 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.178 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.178 DEBUG:boto:Params: {} 2024-04-12T04:06:48.178 DEBUG:boto:Token: None 2024-04-12T04:06:48.178 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.178 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a5018e137e40954-006618b358-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.182 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.182 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:06:48.182 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:06:48.182 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.182 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:06:48.182 DEBUG:boto:Data: 2024-04-12T04:06:48.182 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.183 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.183 DEBUG:boto:Params: {} 2024-04-12T04:06:48.183 DEBUG:boto:Token: None 2024-04-12T04:06:48.183 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj57 2024-04-12T04:06:48.183 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZuS58GtNm79kXy6cpsQWs1hCE0o= 2024-04-12T04:06:48.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZuS58GtNm79kXy6cpsQWs1hCE0o=', 'Content-Length': '0'} 2024-04-12T04:06:48.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000081ff055b1378332-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.185 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:06:48.185 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:06:48.185 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.185 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:06:48.185 DEBUG:boto:Data: 2024-04-12T04:06:48.185 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.185 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.186 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.186 DEBUG:boto:Params: {} 2024-04-12T04:06:48.186 DEBUG:boto:Token: None 2024-04-12T04:06:48.186 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj57 2024-04-12T04:06:48.186 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZuS58GtNm79kXy6cpsQWs1hCE0o= 2024-04-12T04:06:48.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZuS58GtNm79kXy6cpsQWs1hCE0o=', 'Content-Length': '0'} 2024-04-12T04:06:48.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085e33502f7d030be-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:06:48.188 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:06:48.189 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:06:48.189 DEBUG:boto:Method: GET 2024-04-12T04:06:48.189 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:06:48.189 DEBUG:boto:Data: 2024-04-12T04:06:48.189 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.189 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.189 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.189 DEBUG:boto:Params: {} 2024-04-12T04:06:48.189 DEBUG:boto:Token: None 2024-04-12T04:06:48.189 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj57 2024-04-12T04:06:48.190 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kbb/V8o/Ev7CdwcYA71yMEVwZqY= 2024-04-12T04:06:48.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kbb/V8o/Ev7CdwcYA71yMEVwZqY=', 'Content-Length': '0'} 2024-04-12T04:06:48.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001921163d31f241b4-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.192 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:06:48.192 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:06:48.192 DEBUG:boto:Method: GET 2024-04-12T04:06:48.192 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:06:48.192 DEBUG:boto:Data: 2024-04-12T04:06:48.192 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.192 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.193 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.193 DEBUG:boto:Params: {} 2024-04-12T04:06:48.193 DEBUG:boto:Token: None 2024-04-12T04:06:48.193 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj57 2024-04-12T04:06:48.193 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kbb/V8o/Ev7CdwcYA71yMEVwZqY= 2024-04-12T04:06:48.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kbb/V8o/Ev7CdwcYA71yMEVwZqY=', 'Content-Length': '0'} 2024-04-12T04:06:48.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f63380d085d1bc71-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.195 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.195 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.195 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:06:48.195 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:06:48.195 DEBUG:boto:Method: GET 2024-04-12T04:06:48.196 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:06:48.196 DEBUG:boto:Data: 2024-04-12T04:06:48.196 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.196 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.196 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.196 DEBUG:boto:Params: {} 2024-04-12T04:06:48.196 DEBUG:boto:Token: None 2024-04-12T04:06:48.196 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.196 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a0d6e2ce7433c61-006618b358-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.199 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.199 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.200 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.200 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:06:48.200 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:06:48.200 DEBUG:boto:Method: GET 2024-04-12T04:06:48.200 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:06:48.200 DEBUG:boto:Data: 2024-04-12T04:06:48.200 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.200 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.200 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.200 DEBUG:boto:Params: {} 2024-04-12T04:06:48.201 DEBUG:boto:Token: None 2024-04-12T04:06:48.201 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.201 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a1cc6d826145ba5-006618b358-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.205 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:06:48.205 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:06:48.205 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:06:48.205 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.205 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.206 DEBUG:boto:Method: GET 2024-04-12T04:06:48.206 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.206 DEBUG:boto:Data: 2024-04-12T04:06:48.206 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.206 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.206 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.206 DEBUG:boto:Params: {} 2024-04-12T04:06:48.206 DEBUG:boto:Token: None 2024-04-12T04:06:48.207 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.207 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Oi08hg00IQvzFxO8xtO4FJlNXKU= 2024-04-12T04:06:48.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Oi08hg00IQvzFxO8xtO4FJlNXKU=', 'Content-Length': '0'} 2024-04-12T04:06:48.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1e89bb394ed33c6-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.209 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:06:48.209 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:06:48.209 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.209 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.209 DEBUG:boto:Method: GET 2024-04-12T04:06:48.209 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.209 DEBUG:boto:Data: 2024-04-12T04:06:48.209 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.210 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.210 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.210 DEBUG:boto:Params: {} 2024-04-12T04:06:48.210 DEBUG:boto:Token: None 2024-04-12T04:06:48.210 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.210 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Oi08hg00IQvzFxO8xtO4FJlNXKU= 2024-04-12T04:06:48.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Oi08hg00IQvzFxO8xtO4FJlNXKU=', 'Content-Length': '0'} 2024-04-12T04:06:48.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003857df6340b73a5a-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.254 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:06:48.254 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:06:48.254 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.255 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.255 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.255 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.255 DEBUG:boto:Data: 2024-04-12T04:06:48.255 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.255 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.255 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.255 DEBUG:boto:Params: {} 2024-04-12T04:06:48.255 DEBUG:boto:Token: None 2024-04-12T04:06:48.255 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.256 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6r8vtzauy14Ilmk2dK7dqjJUcZM= 2024-04-12T04:06:48.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6r8vtzauy14Ilmk2dK7dqjJUcZM=', 'Content-Length': '0'} 2024-04-12T04:06:48.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a193c1dd510d238-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.258 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:06:48.258 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:06:48.258 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.258 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.258 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.258 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.258 DEBUG:boto:Data: 2024-04-12T04:06:48.258 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.258 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.259 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.259 DEBUG:boto:Params: {} 2024-04-12T04:06:48.259 DEBUG:boto:Token: None 2024-04-12T04:06:48.259 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.259 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6r8vtzauy14Ilmk2dK7dqjJUcZM= 2024-04-12T04:06:48.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6r8vtzauy14Ilmk2dK7dqjJUcZM=', 'Content-Length': '0'} 2024-04-12T04:06:48.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8a745df591f103c-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:06:48.261 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:06:48.261 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:06:48.261 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.261 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.262 DEBUG:boto:Method: GET 2024-04-12T04:06:48.262 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.262 DEBUG:boto:Data: 2024-04-12T04:06:48.262 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.262 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.262 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.262 DEBUG:boto:Params: {} 2024-04-12T04:06:48.262 DEBUG:boto:Token: None 2024-04-12T04:06:48.262 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.262 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Oi08hg00IQvzFxO8xtO4FJlNXKU= 2024-04-12T04:06:48.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Oi08hg00IQvzFxO8xtO4FJlNXKU=', 'Content-Length': '0'} 2024-04-12T04:06:48.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc978ad08fe6f61-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.264 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:06:48.264 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:06:48.265 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.265 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.265 DEBUG:boto:Method: GET 2024-04-12T04:06:48.265 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.265 DEBUG:boto:Data: 2024-04-12T04:06:48.265 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.265 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.265 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.265 DEBUG:boto:Params: {} 2024-04-12T04:06:48.265 DEBUG:boto:Token: None 2024-04-12T04:06:48.265 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:06:48.266 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Oi08hg00IQvzFxO8xtO4FJlNXKU= 2024-04-12T04:06:48.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Oi08hg00IQvzFxO8xtO4FJlNXKU=', 'Content-Length': '0'} 2024-04-12T04:06:48.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000455a08a91e78da1-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.268 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.268 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.268 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:06:48.268 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:06:48.268 DEBUG:boto:Method: GET 2024-04-12T04:06:48.268 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:06:48.268 DEBUG:boto:Data: 2024-04-12T04:06:48.268 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.269 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.269 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.269 DEBUG:boto:Params: {} 2024-04-12T04:06:48.269 DEBUG:boto:Token: None 2024-04-12T04:06:48.269 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.269 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6a90a3a4d1bacf6-006618b358-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.271 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.272 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.272 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.272 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:06:48.272 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:06:48.272 DEBUG:boto:Method: GET 2024-04-12T04:06:48.272 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:06:48.272 DEBUG:boto:Data: 2024-04-12T04:06:48.272 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.272 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.272 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.273 DEBUG:boto:Params: {} 2024-04-12T04:06:48.273 DEBUG:boto:Token: None 2024-04-12T04:06:48.273 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.273 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000878d649e095e4641-006618b358-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.276 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.276 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:06:48.277 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:06:48.277 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.277 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:06:48.277 DEBUG:boto:Data: 2024-04-12T04:06:48.277 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.277 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.277 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.277 DEBUG:boto:Params: {} 2024-04-12T04:06:48.277 DEBUG:boto:Token: None 2024-04-12T04:06:48.277 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj68 2024-04-12T04:06:48.277 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l+j4Rdo+7q2hgeXDLgYFttwSjNQ= 2024-04-12T04:06:48.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l+j4Rdo+7q2hgeXDLgYFttwSjNQ=', 'Content-Length': '0'} 2024-04-12T04:06:48.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f4a187b75f3ae89-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.280 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:06:48.280 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:06:48.280 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.280 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:06:48.280 DEBUG:boto:Data: 2024-04-12T04:06:48.280 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.280 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.280 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.280 DEBUG:boto:Params: {} 2024-04-12T04:06:48.280 DEBUG:boto:Token: None 2024-04-12T04:06:48.280 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj68 2024-04-12T04:06:48.281 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l+j4Rdo+7q2hgeXDLgYFttwSjNQ= 2024-04-12T04:06:48.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l+j4Rdo+7q2hgeXDLgYFttwSjNQ=', 'Content-Length': '0'} 2024-04-12T04:06:48.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca92e3f839212df6-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:06:48.283 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:06:48.283 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:06:48.283 DEBUG:boto:Method: GET 2024-04-12T04:06:48.283 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:06:48.283 DEBUG:boto:Data: 2024-04-12T04:06:48.283 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.283 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.283 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.283 DEBUG:boto:Params: {} 2024-04-12T04:06:48.284 DEBUG:boto:Token: None 2024-04-12T04:06:48.284 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj68 2024-04-12T04:06:48.284 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zEQilJGRHS9YQuHJGALVJUTA1qw= 2024-04-12T04:06:48.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zEQilJGRHS9YQuHJGALVJUTA1qw=', 'Content-Length': '0'} 2024-04-12T04:06:48.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028ea42c8b3e494b8-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.326 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:06:48.326 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:06:48.326 DEBUG:boto:Method: GET 2024-04-12T04:06:48.326 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:06:48.327 DEBUG:boto:Data: 2024-04-12T04:06:48.327 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.327 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.327 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.327 DEBUG:boto:Params: {} 2024-04-12T04:06:48.327 DEBUG:boto:Token: None 2024-04-12T04:06:48.327 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj68 2024-04-12T04:06:48.327 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zEQilJGRHS9YQuHJGALVJUTA1qw= 2024-04-12T04:06:48.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zEQilJGRHS9YQuHJGALVJUTA1qw=', 'Content-Length': '0'} 2024-04-12T04:06:48.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007195505c3d4a4cfd-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.329 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.330 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.330 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:06:48.330 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:06:48.330 DEBUG:boto:Method: GET 2024-04-12T04:06:48.330 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:06:48.330 DEBUG:boto:Data: 2024-04-12T04:06:48.330 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.330 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.330 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.330 DEBUG:boto:Params: {} 2024-04-12T04:06:48.330 DEBUG:boto:Token: None 2024-04-12T04:06:48.331 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.331 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008016ead0c0ccede-006618b358-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.334 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.334 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.334 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.334 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:06:48.335 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:06:48.335 DEBUG:boto:Method: GET 2024-04-12T04:06:48.335 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:06:48.335 DEBUG:boto:Data: 2024-04-12T04:06:48.335 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.335 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.335 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.335 DEBUG:boto:Params: {} 2024-04-12T04:06:48.335 DEBUG:boto:Token: None 2024-04-12T04:06:48.335 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.335 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008b8947a7955e581-006618b358-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.339 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:06:48.339 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:06:48.339 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:06:48.339 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.339 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.339 DEBUG:boto:Method: GET 2024-04-12T04:06:48.340 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.340 DEBUG:boto:Data: 2024-04-12T04:06:48.340 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.340 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.340 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.340 DEBUG:boto:Params: {} 2024-04-12T04:06:48.340 DEBUG:boto:Token: None 2024-04-12T04:06:48.340 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.340 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ObwQxnx7h8ulbU/nRcmUsz3NOcA= 2024-04-12T04:06:48.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ObwQxnx7h8ulbU/nRcmUsz3NOcA=', 'Content-Length': '0'} 2024-04-12T04:06:48.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0afce8f0988ab5a-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.342 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:06:48.343 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:06:48.343 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.343 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.343 DEBUG:boto:Method: GET 2024-04-12T04:06:48.343 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.343 DEBUG:boto:Data: 2024-04-12T04:06:48.343 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.343 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.343 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.343 DEBUG:boto:Params: {} 2024-04-12T04:06:48.343 DEBUG:boto:Token: None 2024-04-12T04:06:48.344 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.344 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ObwQxnx7h8ulbU/nRcmUsz3NOcA= 2024-04-12T04:06:48.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ObwQxnx7h8ulbU/nRcmUsz3NOcA=', 'Content-Length': '0'} 2024-04-12T04:06:48.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f74706d4530a86b4-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.386 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:06:48.387 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:06:48.387 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.387 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.387 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.387 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.387 DEBUG:boto:Data: 2024-04-12T04:06:48.387 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.387 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.387 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.387 DEBUG:boto:Params: {} 2024-04-12T04:06:48.387 DEBUG:boto:Token: None 2024-04-12T04:06:48.388 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.388 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zIXqeEaZXzEG+ytf5QrDWwKGQSo= 2024-04-12T04:06:48.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zIXqeEaZXzEG+ytf5QrDWwKGQSo=', 'Content-Length': '0'} 2024-04-12T04:06:48.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005675ed0e566650f9-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.390 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:06:48.390 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:06:48.390 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.390 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.390 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.390 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.390 DEBUG:boto:Data: 2024-04-12T04:06:48.391 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.391 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.391 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.391 DEBUG:boto:Params: {} 2024-04-12T04:06:48.391 DEBUG:boto:Token: None 2024-04-12T04:06:48.391 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.391 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zIXqeEaZXzEG+ytf5QrDWwKGQSo= 2024-04-12T04:06:48.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zIXqeEaZXzEG+ytf5QrDWwKGQSo=', 'Content-Length': '0'} 2024-04-12T04:06:48.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef29946168f69023-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:06:48.393 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:06:48.394 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:06:48.394 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.394 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.394 DEBUG:boto:Method: GET 2024-04-12T04:06:48.394 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.394 DEBUG:boto:Data: 2024-04-12T04:06:48.394 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.394 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.394 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.394 DEBUG:boto:Params: {} 2024-04-12T04:06:48.394 DEBUG:boto:Token: None 2024-04-12T04:06:48.395 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.395 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ObwQxnx7h8ulbU/nRcmUsz3NOcA= 2024-04-12T04:06:48.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ObwQxnx7h8ulbU/nRcmUsz3NOcA=', 'Content-Length': '0'} 2024-04-12T04:06:48.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000423142e1d778042c-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.397 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:06:48.397 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:06:48.397 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.397 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.397 DEBUG:boto:Method: GET 2024-04-12T04:06:48.397 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.397 DEBUG:boto:Data: 2024-04-12T04:06:48.397 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.397 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.398 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.398 DEBUG:boto:Params: {} 2024-04-12T04:06:48.398 DEBUG:boto:Token: None 2024-04-12T04:06:48.398 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:06:48.398 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ObwQxnx7h8ulbU/nRcmUsz3NOcA= 2024-04-12T04:06:48.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ObwQxnx7h8ulbU/nRcmUsz3NOcA=', 'Content-Length': '0'} 2024-04-12T04:06:48.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f88ef96182f4cbc6-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.400 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.400 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.400 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:06:48.400 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:06:48.401 DEBUG:boto:Method: GET 2024-04-12T04:06:48.401 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:06:48.401 DEBUG:boto:Data: 2024-04-12T04:06:48.401 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.401 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.401 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.401 DEBUG:boto:Params: {} 2024-04-12T04:06:48.401 DEBUG:boto:Token: None 2024-04-12T04:06:48.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.401 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052c8fc2547216eda-006618b358-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.404 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.404 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.405 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.405 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:06:48.405 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:06:48.405 DEBUG:boto:Method: GET 2024-04-12T04:06:48.405 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:06:48.405 DEBUG:boto:Data: 2024-04-12T04:06:48.405 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.405 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.405 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.405 DEBUG:boto:Params: {} 2024-04-12T04:06:48.405 DEBUG:boto:Token: None 2024-04-12T04:06:48.405 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.406 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf944cfc6446dbe6-006618b358-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.409 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.409 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:06:48.409 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:06:48.410 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.410 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:06:48.410 DEBUG:boto:Data: 2024-04-12T04:06:48.410 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.410 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.410 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.410 DEBUG:boto:Params: {} 2024-04-12T04:06:48.410 DEBUG:boto:Token: None 2024-04-12T04:06:48.410 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj69 2024-04-12T04:06:48.410 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FBK+3Oja+TXS3Qgucf3V+xr0B64= 2024-04-12T04:06:48.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FBK+3Oja+TXS3Qgucf3V+xr0B64=', 'Content-Length': '0'} 2024-04-12T04:06:48.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002509a55ba468483c-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.412 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:06:48.412 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:06:48.413 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.413 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:06:48.413 DEBUG:boto:Data: 2024-04-12T04:06:48.413 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.413 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.413 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.413 DEBUG:boto:Params: {} 2024-04-12T04:06:48.413 DEBUG:boto:Token: None 2024-04-12T04:06:48.413 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj69 2024-04-12T04:06:48.413 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FBK+3Oja+TXS3Qgucf3V+xr0B64= 2024-04-12T04:06:48.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FBK+3Oja+TXS3Qgucf3V+xr0B64=', 'Content-Length': '0'} 2024-04-12T04:06:48.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035b60094a8e8e2d0-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:06:48.416 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:06:48.416 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:06:48.416 DEBUG:boto:Method: GET 2024-04-12T04:06:48.416 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:06:48.416 DEBUG:boto:Data: 2024-04-12T04:06:48.416 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.416 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.416 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.416 DEBUG:boto:Params: {} 2024-04-12T04:06:48.416 DEBUG:boto:Token: None 2024-04-12T04:06:48.417 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj69 2024-04-12T04:06:48.417 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9OlJQ0ArT/PqvqThJ/TBzWppp6s= 2024-04-12T04:06:48.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9OlJQ0ArT/PqvqThJ/TBzWppp6s=', 'Content-Length': '0'} 2024-04-12T04:06:48.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec1f97f4ebe81028-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.419 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:06:48.419 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:06:48.419 DEBUG:boto:Method: GET 2024-04-12T04:06:48.419 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:06:48.419 DEBUG:boto:Data: 2024-04-12T04:06:48.419 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.419 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.419 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.420 DEBUG:boto:Params: {} 2024-04-12T04:06:48.420 DEBUG:boto:Token: None 2024-04-12T04:06:48.420 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj69 2024-04-12T04:06:48.420 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9OlJQ0ArT/PqvqThJ/TBzWppp6s= 2024-04-12T04:06:48.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9OlJQ0ArT/PqvqThJ/TBzWppp6s=', 'Content-Length': '0'} 2024-04-12T04:06:48.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000157e280e44f75763-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.462 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.463 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.463 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:06:48.463 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:06:48.463 DEBUG:boto:Method: GET 2024-04-12T04:06:48.463 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:06:48.463 DEBUG:boto:Data: 2024-04-12T04:06:48.463 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.463 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.463 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.463 DEBUG:boto:Params: {} 2024-04-12T04:06:48.464 DEBUG:boto:Token: None 2024-04-12T04:06:48.464 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.464 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065920e10196bf10d-006618b358-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.467 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.467 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.467 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.467 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:06:48.467 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:06:48.467 DEBUG:boto:Method: GET 2024-04-12T04:06:48.467 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:06:48.468 DEBUG:boto:Data: 2024-04-12T04:06:48.468 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.468 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.468 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.468 DEBUG:boto:Params: {} 2024-04-12T04:06:48.468 DEBUG:boto:Token: None 2024-04-12T04:06:48.468 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.468 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9c10681b823d76d-006618b358-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.472 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:06:48.472 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:06:48.472 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:06:48.472 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.473 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.473 DEBUG:boto:Method: GET 2024-04-12T04:06:48.473 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.473 DEBUG:boto:Data: 2024-04-12T04:06:48.473 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.473 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.473 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.473 DEBUG:boto:Params: {} 2024-04-12T04:06:48.473 DEBUG:boto:Token: None 2024-04-12T04:06:48.473 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.474 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ny6o30PyB7XLbKKfFIETUzF0RPU= 2024-04-12T04:06:48.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ny6o30PyB7XLbKKfFIETUzF0RPU=', 'Content-Length': '0'} 2024-04-12T04:06:48.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001d94f1bbc4c33fe-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.518 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:06:48.519 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:06:48.519 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.519 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.519 DEBUG:boto:Method: GET 2024-04-12T04:06:48.519 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.519 DEBUG:boto:Data: 2024-04-12T04:06:48.519 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.519 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.519 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.519 DEBUG:boto:Params: {} 2024-04-12T04:06:48.519 DEBUG:boto:Token: None 2024-04-12T04:06:48.520 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.520 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ny6o30PyB7XLbKKfFIETUzF0RPU= 2024-04-12T04:06:48.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ny6o30PyB7XLbKKfFIETUzF0RPU=', 'Content-Length': '0'} 2024-04-12T04:06:48.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6e1d65bedfcd9f7-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.522 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:06:48.523 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:06:48.523 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.523 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.523 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.523 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.523 DEBUG:boto:Data: 2024-04-12T04:06:48.523 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.523 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.523 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.523 DEBUG:boto:Params: {} 2024-04-12T04:06:48.523 DEBUG:boto:Token: None 2024-04-12T04:06:48.524 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.524 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:c5nfKvXRI9/oVbm5z1ozq74p92w= 2024-04-12T04:06:48.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:c5nfKvXRI9/oVbm5z1ozq74p92w=', 'Content-Length': '0'} 2024-04-12T04:06:48.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c66179eb399f236-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.526 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:06:48.526 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:06:48.526 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.526 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.526 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.526 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.526 DEBUG:boto:Data: 2024-04-12T04:06:48.527 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.527 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.527 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.527 DEBUG:boto:Params: {} 2024-04-12T04:06:48.527 DEBUG:boto:Token: None 2024-04-12T04:06:48.527 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.527 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:c5nfKvXRI9/oVbm5z1ozq74p92w= 2024-04-12T04:06:48.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:c5nfKvXRI9/oVbm5z1ozq74p92w=', 'Content-Length': '0'} 2024-04-12T04:06:48.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cc467aef1d536cd-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:06:48.529 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:06:48.530 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:06:48.530 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.530 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.530 DEBUG:boto:Method: GET 2024-04-12T04:06:48.530 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.530 DEBUG:boto:Data: 2024-04-12T04:06:48.530 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.530 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.530 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.530 DEBUG:boto:Params: {} 2024-04-12T04:06:48.531 DEBUG:boto:Token: None 2024-04-12T04:06:48.531 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.531 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ny6o30PyB7XLbKKfFIETUzF0RPU= 2024-04-12T04:06:48.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ny6o30PyB7XLbKKfFIETUzF0RPU=', 'Content-Length': '0'} 2024-04-12T04:06:48.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fd27481e4759c76-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.533 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:06:48.533 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:06:48.533 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.533 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.533 DEBUG:boto:Method: GET 2024-04-12T04:06:48.534 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.534 DEBUG:boto:Data: 2024-04-12T04:06:48.534 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.534 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.534 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.534 DEBUG:boto:Params: {} 2024-04-12T04:06:48.534 DEBUG:boto:Token: None 2024-04-12T04:06:48.534 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:06:48.534 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ny6o30PyB7XLbKKfFIETUzF0RPU= 2024-04-12T04:06:48.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ny6o30PyB7XLbKKfFIETUzF0RPU=', 'Content-Length': '0'} 2024-04-12T04:06:48.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059decdf3fdccc74f-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.537 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.537 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.537 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:06:48.537 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:06:48.537 DEBUG:boto:Method: GET 2024-04-12T04:06:48.537 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:06:48.537 DEBUG:boto:Data: 2024-04-12T04:06:48.537 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.537 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.537 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.538 DEBUG:boto:Params: {} 2024-04-12T04:06:48.538 DEBUG:boto:Token: None 2024-04-12T04:06:48.538 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.538 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000694440945ac81aff-006618b358-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.541 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.541 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.541 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.541 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:06:48.541 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:06:48.541 DEBUG:boto:Method: GET 2024-04-12T04:06:48.542 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:06:48.542 DEBUG:boto:Data: 2024-04-12T04:06:48.542 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.542 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.542 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.542 DEBUG:boto:Params: {} 2024-04-12T04:06:48.542 DEBUG:boto:Token: None 2024-04-12T04:06:48.542 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.542 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad347fa2ee800cf8-006618b358-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.546 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.546 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:06:48.546 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:06:48.546 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.546 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:06:48.547 DEBUG:boto:Data: 2024-04-12T04:06:48.547 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.547 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.547 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.547 DEBUG:boto:Params: {} 2024-04-12T04:06:48.547 DEBUG:boto:Token: None 2024-04-12T04:06:48.547 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj76 2024-04-12T04:06:48.547 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GtvbruDDsP0e5iEs62y+Krm4iBg= 2024-04-12T04:06:48.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GtvbruDDsP0e5iEs62y+Krm4iBg=', 'Content-Length': '0'} 2024-04-12T04:06:48.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e55b9b29ace4d3e4-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.549 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:06:48.550 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:06:48.550 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.550 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:06:48.550 DEBUG:boto:Data: 2024-04-12T04:06:48.550 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.550 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.550 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.550 DEBUG:boto:Params: {} 2024-04-12T04:06:48.550 DEBUG:boto:Token: None 2024-04-12T04:06:48.550 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj76 2024-04-12T04:06:48.551 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GtvbruDDsP0e5iEs62y+Krm4iBg= 2024-04-12T04:06:48.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GtvbruDDsP0e5iEs62y+Krm4iBg=', 'Content-Length': '0'} 2024-04-12T04:06:48.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013c85dd8bff181ab-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:06:48.553 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:06:48.553 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:06:48.553 DEBUG:boto:Method: GET 2024-04-12T04:06:48.554 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:06:48.554 DEBUG:boto:Data: 2024-04-12T04:06:48.554 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.554 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.554 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.554 DEBUG:boto:Params: {} 2024-04-12T04:06:48.554 DEBUG:boto:Token: None 2024-04-12T04:06:48.554 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj76 2024-04-12T04:06:48.554 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cyaiJeoWyELrhhzqizcpuvuAyP0= 2024-04-12T04:06:48.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cyaiJeoWyELrhhzqizcpuvuAyP0=', 'Content-Length': '0'} 2024-04-12T04:06:48.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ef44aeff4f7e15e-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.556 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:06:48.557 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:06:48.557 DEBUG:boto:Method: GET 2024-04-12T04:06:48.557 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:06:48.557 DEBUG:boto:Data: 2024-04-12T04:06:48.557 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.557 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.557 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.557 DEBUG:boto:Params: {} 2024-04-12T04:06:48.557 DEBUG:boto:Token: None 2024-04-12T04:06:48.557 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj76 2024-04-12T04:06:48.557 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cyaiJeoWyELrhhzqizcpuvuAyP0= 2024-04-12T04:06:48.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cyaiJeoWyELrhhzqizcpuvuAyP0=', 'Content-Length': '0'} 2024-04-12T04:06:48.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f53b389bb119fec6-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.559 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.560 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.560 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:06:48.560 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:06:48.560 DEBUG:boto:Method: GET 2024-04-12T04:06:48.560 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:06:48.560 DEBUG:boto:Data: 2024-04-12T04:06:48.560 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.560 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.560 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.560 DEBUG:boto:Params: {} 2024-04-12T04:06:48.560 DEBUG:boto:Token: None 2024-04-12T04:06:48.561 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.561 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c6303c95f91a0b3-006618b358-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.563 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.564 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.564 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.564 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:06:48.564 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:06:48.564 DEBUG:boto:Method: GET 2024-04-12T04:06:48.564 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:06:48.564 DEBUG:boto:Data: 2024-04-12T04:06:48.564 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.564 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.564 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.564 DEBUG:boto:Params: {} 2024-04-12T04:06:48.565 DEBUG:boto:Token: None 2024-04-12T04:06:48.565 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.565 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c95b98da13d1515-006618b358-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.568 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:06:48.569 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:06:48.569 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:06:48.569 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.569 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.569 DEBUG:boto:Method: GET 2024-04-12T04:06:48.569 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.569 DEBUG:boto:Data: 2024-04-12T04:06:48.569 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.569 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.569 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.570 DEBUG:boto:Params: {} 2024-04-12T04:06:48.570 DEBUG:boto:Token: None 2024-04-12T04:06:48.570 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.570 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gEHwPD5vgWg2uB58scFa09zOPzs= 2024-04-12T04:06:48.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gEHwPD5vgWg2uB58scFa09zOPzs=', 'Content-Length': '0'} 2024-04-12T04:06:48.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c54b0fb870d4080-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.572 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:06:48.572 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:06:48.572 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.572 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.573 DEBUG:boto:Method: GET 2024-04-12T04:06:48.573 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.573 DEBUG:boto:Data: 2024-04-12T04:06:48.573 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.573 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.573 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.573 DEBUG:boto:Params: {} 2024-04-12T04:06:48.573 DEBUG:boto:Token: None 2024-04-12T04:06:48.573 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.573 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gEHwPD5vgWg2uB58scFa09zOPzs= 2024-04-12T04:06:48.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gEHwPD5vgWg2uB58scFa09zOPzs=', 'Content-Length': '0'} 2024-04-12T04:06:48.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae4c98ef1469420f-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.618 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:06:48.618 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:06:48.618 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.619 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.619 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.619 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.619 DEBUG:boto:Data: 2024-04-12T04:06:48.619 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.619 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.619 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.619 DEBUG:boto:Params: {} 2024-04-12T04:06:48.619 DEBUG:boto:Token: None 2024-04-12T04:06:48.619 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.620 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xNlSspmA7q1X4zxXOL+/S2/mS3U= 2024-04-12T04:06:48.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xNlSspmA7q1X4zxXOL+/S2/mS3U=', 'Content-Length': '0'} 2024-04-12T04:06:48.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090fb9313a4fdde64-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.621 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:06:48.622 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:06:48.622 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.622 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.622 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.622 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.622 DEBUG:boto:Data: 2024-04-12T04:06:48.622 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.622 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.622 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.622 DEBUG:boto:Params: {} 2024-04-12T04:06:48.623 DEBUG:boto:Token: None 2024-04-12T04:06:48.623 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.623 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xNlSspmA7q1X4zxXOL+/S2/mS3U= 2024-04-12T04:06:48.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xNlSspmA7q1X4zxXOL+/S2/mS3U=', 'Content-Length': '0'} 2024-04-12T04:06:48.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4f1d1c04c3f8e0f-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:06:48.625 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:06:48.625 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:06:48.625 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.625 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.626 DEBUG:boto:Method: GET 2024-04-12T04:06:48.626 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.626 DEBUG:boto:Data: 2024-04-12T04:06:48.626 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.626 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.626 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.626 DEBUG:boto:Params: {} 2024-04-12T04:06:48.626 DEBUG:boto:Token: None 2024-04-12T04:06:48.626 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.626 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gEHwPD5vgWg2uB58scFa09zOPzs= 2024-04-12T04:06:48.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gEHwPD5vgWg2uB58scFa09zOPzs=', 'Content-Length': '0'} 2024-04-12T04:06:48.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000465b2c98fe2e7dfd-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.629 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:06:48.629 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:06:48.629 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.629 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.629 DEBUG:boto:Method: GET 2024-04-12T04:06:48.629 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.629 DEBUG:boto:Data: 2024-04-12T04:06:48.629 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.630 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.630 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.630 DEBUG:boto:Params: {} 2024-04-12T04:06:48.630 DEBUG:boto:Token: None 2024-04-12T04:06:48.630 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:06:48.630 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gEHwPD5vgWg2uB58scFa09zOPzs= 2024-04-12T04:06:48.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gEHwPD5vgWg2uB58scFa09zOPzs=', 'Content-Length': '0'} 2024-04-12T04:06:48.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cf5693373c622bd-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.632 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.632 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.632 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:06:48.632 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:06:48.632 DEBUG:boto:Method: GET 2024-04-12T04:06:48.632 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:06:48.633 DEBUG:boto:Data: 2024-04-12T04:06:48.633 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.633 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.633 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.633 DEBUG:boto:Params: {} 2024-04-12T04:06:48.633 DEBUG:boto:Token: None 2024-04-12T04:06:48.633 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.633 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb90373da3f7d191-006618b358-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.636 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.636 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.636 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.636 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:06:48.636 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:06:48.637 DEBUG:boto:Method: GET 2024-04-12T04:06:48.637 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:06:48.637 DEBUG:boto:Data: 2024-04-12T04:06:48.637 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.637 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.637 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.637 DEBUG:boto:Params: {} 2024-04-12T04:06:48.637 DEBUG:boto:Token: None 2024-04-12T04:06:48.637 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.637 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a11f57221ca84849-006618b358-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.642 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.642 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:06:48.642 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:06:48.643 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.643 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:06:48.643 DEBUG:boto:Data: 2024-04-12T04:06:48.643 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.643 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.643 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.643 DEBUG:boto:Params: {} 2024-04-12T04:06:48.643 DEBUG:boto:Token: None 2024-04-12T04:06:48.643 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj85 2024-04-12T04:06:48.643 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iK+bbNIPxmK+xkaohc/+elQjA3g= 2024-04-12T04:06:48.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iK+bbNIPxmK+xkaohc/+elQjA3g=', 'Content-Length': '0'} 2024-04-12T04:06:48.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e9d0cd1d6f6e468-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.646 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:06:48.646 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:06:48.646 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.646 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:06:48.646 DEBUG:boto:Data: 2024-04-12T04:06:48.646 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.646 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.646 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.646 DEBUG:boto:Params: {} 2024-04-12T04:06:48.647 DEBUG:boto:Token: None 2024-04-12T04:06:48.647 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj85 2024-04-12T04:06:48.647 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iK+bbNIPxmK+xkaohc/+elQjA3g= 2024-04-12T04:06:48.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iK+bbNIPxmK+xkaohc/+elQjA3g=', 'Content-Length': '0'} 2024-04-12T04:06:48.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000deffedde6cb7130b-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:06:48.649 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:06:48.649 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:06:48.649 DEBUG:boto:Method: GET 2024-04-12T04:06:48.649 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:06:48.649 DEBUG:boto:Data: 2024-04-12T04:06:48.649 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.649 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.650 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.650 DEBUG:boto:Params: {} 2024-04-12T04:06:48.650 DEBUG:boto:Token: None 2024-04-12T04:06:48.650 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj85 2024-04-12T04:06:48.650 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:abglvXGDKh2+3kinFTNtjb/GLBA= 2024-04-12T04:06:48.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:abglvXGDKh2+3kinFTNtjb/GLBA=', 'Content-Length': '0'} 2024-04-12T04:06:48.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002373dc2a0927c9cd-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.694 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:06:48.694 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:06:48.695 DEBUG:boto:Method: GET 2024-04-12T04:06:48.695 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:06:48.695 DEBUG:boto:Data: 2024-04-12T04:06:48.695 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.695 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.695 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.695 DEBUG:boto:Params: {} 2024-04-12T04:06:48.695 DEBUG:boto:Token: None 2024-04-12T04:06:48.695 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj85 2024-04-12T04:06:48.695 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:abglvXGDKh2+3kinFTNtjb/GLBA= 2024-04-12T04:06:48.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:abglvXGDKh2+3kinFTNtjb/GLBA=', 'Content-Length': '0'} 2024-04-12T04:06:48.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057c23a1d41e345a4-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.697 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.698 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.698 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:06:48.698 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:06:48.698 DEBUG:boto:Method: GET 2024-04-12T04:06:48.698 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:06:48.698 DEBUG:boto:Data: 2024-04-12T04:06:48.698 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.698 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.698 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.698 DEBUG:boto:Params: {} 2024-04-12T04:06:48.698 DEBUG:boto:Token: None 2024-04-12T04:06:48.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.699 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b300893d0464242-006618b358-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.702 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.702 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.702 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.702 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:06:48.705 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:06:48.705 DEBUG:boto:Method: GET 2024-04-12T04:06:48.705 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:06:48.705 DEBUG:boto:Data: 2024-04-12T04:06:48.705 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.705 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.705 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.705 DEBUG:boto:Params: {} 2024-04-12T04:06:48.706 DEBUG:boto:Token: None 2024-04-12T04:06:48.706 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.706 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6de6ccf0dfd1fc8-006618b358-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.709 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:06:48.710 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:06:48.710 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:06:48.710 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.710 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.710 DEBUG:boto:Method: GET 2024-04-12T04:06:48.710 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.711 DEBUG:boto:Data: 2024-04-12T04:06:48.711 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.711 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.711 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.711 DEBUG:boto:Params: {} 2024-04-12T04:06:48.711 DEBUG:boto:Token: None 2024-04-12T04:06:48.711 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.711 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qPtuRIR0Y4tznrK+mmrEim8Y5TA= 2024-04-12T04:06:48.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qPtuRIR0Y4tznrK+mmrEim8Y5TA=', 'Content-Length': '0'} 2024-04-12T04:06:48.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9f9b4a309a06c11-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.713 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:06:48.714 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:06:48.714 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.714 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.714 DEBUG:boto:Method: GET 2024-04-12T04:06:48.714 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.714 DEBUG:boto:Data: 2024-04-12T04:06:48.714 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.714 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.714 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.714 DEBUG:boto:Params: {} 2024-04-12T04:06:48.715 DEBUG:boto:Token: None 2024-04-12T04:06:48.715 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.715 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qPtuRIR0Y4tznrK+mmrEim8Y5TA= 2024-04-12T04:06:48.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qPtuRIR0Y4tznrK+mmrEim8Y5TA=', 'Content-Length': '0'} 2024-04-12T04:06:48.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad7bc5417bba4892-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.717 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:06:48.717 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:06:48.717 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.717 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.717 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.718 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.718 DEBUG:boto:Data: 2024-04-12T04:06:48.718 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.718 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.718 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.718 DEBUG:boto:Params: {} 2024-04-12T04:06:48.718 DEBUG:boto:Token: None 2024-04-12T04:06:48.718 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.718 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D4sOM0Wv/LcZAo08FOWcOJ49zok= 2024-04-12T04:06:48.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D4sOM0Wv/LcZAo08FOWcOJ49zok=', 'Content-Length': '0'} 2024-04-12T04:06:48.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc061399b5cceab1-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.720 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:06:48.721 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:06:48.721 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.721 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.721 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.721 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.721 DEBUG:boto:Data: 2024-04-12T04:06:48.721 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.721 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.721 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.721 DEBUG:boto:Params: {} 2024-04-12T04:06:48.721 DEBUG:boto:Token: None 2024-04-12T04:06:48.722 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.722 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D4sOM0Wv/LcZAo08FOWcOJ49zok= 2024-04-12T04:06:48.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D4sOM0Wv/LcZAo08FOWcOJ49zok=', 'Content-Length': '0'} 2024-04-12T04:06:48.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000683d0666dbec004a-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:06:48.724 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:06:48.724 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:06:48.724 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.724 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.724 DEBUG:boto:Method: GET 2024-04-12T04:06:48.725 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.725 DEBUG:boto:Data: 2024-04-12T04:06:48.725 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.725 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.725 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.725 DEBUG:boto:Params: {} 2024-04-12T04:06:48.725 DEBUG:boto:Token: None 2024-04-12T04:06:48.725 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.725 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qPtuRIR0Y4tznrK+mmrEim8Y5TA= 2024-04-12T04:06:48.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qPtuRIR0Y4tznrK+mmrEim8Y5TA=', 'Content-Length': '0'} 2024-04-12T04:06:48.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d249ad847d302a1-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.727 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:06:48.727 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:06:48.728 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.728 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.728 DEBUG:boto:Method: GET 2024-04-12T04:06:48.728 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.728 DEBUG:boto:Data: 2024-04-12T04:06:48.728 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.728 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.728 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.728 DEBUG:boto:Params: {} 2024-04-12T04:06:48.728 DEBUG:boto:Token: None 2024-04-12T04:06:48.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:06:48.729 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qPtuRIR0Y4tznrK+mmrEim8Y5TA= 2024-04-12T04:06:48.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qPtuRIR0Y4tznrK+mmrEim8Y5TA=', 'Content-Length': '0'} 2024-04-12T04:06:48.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b2a2c53cee0de54-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.731 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.731 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.731 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:06:48.731 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:06:48.731 DEBUG:boto:Method: GET 2024-04-12T04:06:48.731 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:06:48.731 DEBUG:boto:Data: 2024-04-12T04:06:48.731 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.732 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.732 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.732 DEBUG:boto:Params: {} 2024-04-12T04:06:48.732 DEBUG:boto:Token: None 2024-04-12T04:06:48.732 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.732 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e5aff80570cfd6e-006618b358-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.735 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.735 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.735 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.735 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:06:48.735 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:06:48.735 DEBUG:boto:Method: GET 2024-04-12T04:06:48.736 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:06:48.736 DEBUG:boto:Data: 2024-04-12T04:06:48.736 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.736 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.736 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.736 DEBUG:boto:Params: {} 2024-04-12T04:06:48.736 DEBUG:boto:Token: None 2024-04-12T04:06:48.736 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.736 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd76f71f64dc0142-006618b358-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.782 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.783 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:06:48.783 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:06:48.783 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.783 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:06:48.783 DEBUG:boto:Data: 2024-04-12T04:06:48.783 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.784 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.784 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.784 DEBUG:boto:Params: {} 2024-04-12T04:06:48.784 DEBUG:boto:Token: None 2024-04-12T04:06:48.784 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj92 2024-04-12T04:06:48.784 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ry53OTJ05oKFYykWBcpvA/3SkQE= 2024-04-12T04:06:48.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ry53OTJ05oKFYykWBcpvA/3SkQE=', 'Content-Length': '0'} 2024-04-12T04:06:48.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090a7958691f76c4c-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.786 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:06:48.786 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:06:48.787 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.787 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:06:48.787 DEBUG:boto:Data: 2024-04-12T04:06:48.787 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.787 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.787 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.787 DEBUG:boto:Params: {} 2024-04-12T04:06:48.787 DEBUG:boto:Token: None 2024-04-12T04:06:48.787 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj92 2024-04-12T04:06:48.787 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ry53OTJ05oKFYykWBcpvA/3SkQE= 2024-04-12T04:06:48.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ry53OTJ05oKFYykWBcpvA/3SkQE=', 'Content-Length': '0'} 2024-04-12T04:06:48.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071036016e52747d0-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:06:48.790 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:06:48.790 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:06:48.790 DEBUG:boto:Method: GET 2024-04-12T04:06:48.790 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:06:48.790 DEBUG:boto:Data: 2024-04-12T04:06:48.790 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.790 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.790 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.791 DEBUG:boto:Params: {} 2024-04-12T04:06:48.791 DEBUG:boto:Token: None 2024-04-12T04:06:48.791 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj92 2024-04-12T04:06:48.791 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TS3Yaw9Phxv9sKJbOEoO9XyL7qU= 2024-04-12T04:06:48.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TS3Yaw9Phxv9sKJbOEoO9XyL7qU=', 'Content-Length': '0'} 2024-04-12T04:06:48.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a551ba0c1bd6a23b-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.793 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:06:48.793 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:06:48.793 DEBUG:boto:Method: GET 2024-04-12T04:06:48.793 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:06:48.793 DEBUG:boto:Data: 2024-04-12T04:06:48.794 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.794 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.794 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.794 DEBUG:boto:Params: {} 2024-04-12T04:06:48.794 DEBUG:boto:Token: None 2024-04-12T04:06:48.794 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj92 2024-04-12T04:06:48.794 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TS3Yaw9Phxv9sKJbOEoO9XyL7qU= 2024-04-12T04:06:48.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TS3Yaw9Phxv9sKJbOEoO9XyL7qU=', 'Content-Length': '0'} 2024-04-12T04:06:48.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b73c946df779faf0-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.796 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.797 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.797 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:06:48.797 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:06:48.797 DEBUG:boto:Method: GET 2024-04-12T04:06:48.797 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:06:48.797 DEBUG:boto:Data: 2024-04-12T04:06:48.797 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.797 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.797 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.797 DEBUG:boto:Params: {} 2024-04-12T04:06:48.797 DEBUG:boto:Token: None 2024-04-12T04:06:48.798 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.798 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbb9e64964980216-006618b358-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.800 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.801 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.801 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.801 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:06:48.801 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:06:48.801 DEBUG:boto:Method: GET 2024-04-12T04:06:48.801 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:06:48.801 DEBUG:boto:Data: 2024-04-12T04:06:48.801 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.801 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.802 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.802 DEBUG:boto:Params: {} 2024-04-12T04:06:48.802 DEBUG:boto:Token: None 2024-04-12T04:06:48.802 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.802 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.805 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007941a9bd26d8cde6-006618b358-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.806 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:06:48.806 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:06:48.806 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:06:48.806 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.806 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.807 DEBUG:boto:Method: GET 2024-04-12T04:06:48.807 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.807 DEBUG:boto:Data: 2024-04-12T04:06:48.807 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.807 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.807 DEBUG:boto:Params: {} 2024-04-12T04:06:48.807 DEBUG:boto:Token: None 2024-04-12T04:06:48.807 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.807 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BVaTW1D5wve+B/TylPXiVqPXZMk= 2024-04-12T04:06:48.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BVaTW1D5wve+B/TylPXiVqPXZMk=', 'Content-Length': '0'} 2024-04-12T04:06:48.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf273f3b56b68af6-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.810 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:06:48.810 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:06:48.810 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.810 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.810 DEBUG:boto:Method: GET 2024-04-12T04:06:48.810 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.810 DEBUG:boto:Data: 2024-04-12T04:06:48.810 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.811 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.811 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.811 DEBUG:boto:Params: {} 2024-04-12T04:06:48.811 DEBUG:boto:Token: None 2024-04-12T04:06:48.811 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.811 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BVaTW1D5wve+B/TylPXiVqPXZMk= 2024-04-12T04:06:48.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BVaTW1D5wve+B/TylPXiVqPXZMk=', 'Content-Length': '0'} 2024-04-12T04:06:48.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093acb2410f252cce-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.813 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:06:48.814 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:06:48.814 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.814 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.814 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.814 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.814 DEBUG:boto:Data: 2024-04-12T04:06:48.814 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.814 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.814 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.814 DEBUG:boto:Params: {} 2024-04-12T04:06:48.815 DEBUG:boto:Token: None 2024-04-12T04:06:48.815 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.815 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kYcHcksRgh5U7sI5p2o6Tcjwj4k= 2024-04-12T04:06:48.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kYcHcksRgh5U7sI5p2o6Tcjwj4k=', 'Content-Length': '0'} 2024-04-12T04:06:48.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9e193b1e9cc376e-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.817 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:06:48.817 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:06:48.817 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.817 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.817 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.817 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.817 DEBUG:boto:Data: 2024-04-12T04:06:48.818 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.818 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.818 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.818 DEBUG:boto:Params: {} 2024-04-12T04:06:48.818 DEBUG:boto:Token: None 2024-04-12T04:06:48.818 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.818 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kYcHcksRgh5U7sI5p2o6Tcjwj4k= 2024-04-12T04:06:48.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kYcHcksRgh5U7sI5p2o6Tcjwj4k=', 'Content-Length': '0'} 2024-04-12T04:06:48.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b165b1a6d150926d-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:06:48.820 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:06:48.821 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:06:48.821 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.821 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.821 DEBUG:boto:Method: GET 2024-04-12T04:06:48.821 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.821 DEBUG:boto:Data: 2024-04-12T04:06:48.821 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.821 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.821 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.821 DEBUG:boto:Params: {} 2024-04-12T04:06:48.822 DEBUG:boto:Token: None 2024-04-12T04:06:48.822 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.822 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BVaTW1D5wve+B/TylPXiVqPXZMk= 2024-04-12T04:06:48.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BVaTW1D5wve+B/TylPXiVqPXZMk=', 'Content-Length': '0'} 2024-04-12T04:06:48.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3d90f976fbc827e-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.824 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:06:48.824 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:06:48.824 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.824 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.824 DEBUG:boto:Method: GET 2024-04-12T04:06:48.824 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.824 DEBUG:boto:Data: 2024-04-12T04:06:48.824 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.825 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.825 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.825 DEBUG:boto:Params: {} 2024-04-12T04:06:48.825 DEBUG:boto:Token: None 2024-04-12T04:06:48.825 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:06:48.825 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BVaTW1D5wve+B/TylPXiVqPXZMk= 2024-04-12T04:06:48.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BVaTW1D5wve+B/TylPXiVqPXZMk=', 'Content-Length': '0'} 2024-04-12T04:06:48.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fcf53d4a9d44aca-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.827 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.827 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.827 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:06:48.828 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:06:48.828 DEBUG:boto:Method: GET 2024-04-12T04:06:48.828 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:06:48.828 DEBUG:boto:Data: 2024-04-12T04:06:48.828 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.828 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.828 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.828 DEBUG:boto:Params: {} 2024-04-12T04:06:48.828 DEBUG:boto:Token: None 2024-04-12T04:06:48.828 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.829 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bd068e82feeb17b-006618b358-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.831 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.832 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.832 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.832 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:06:48.832 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:06:48.832 DEBUG:boto:Method: GET 2024-04-12T04:06:48.832 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:06:48.832 DEBUG:boto:Data: 2024-04-12T04:06:48.832 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.832 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.832 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.832 DEBUG:boto:Params: {} 2024-04-12T04:06:48.833 DEBUG:boto:Token: None 2024-04-12T04:06:48.833 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.833 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.836 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000858b0abc40b1a2dd-006618b358-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.836 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.837 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:06:48.837 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:06:48.837 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.837 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:06:48.837 DEBUG:boto:Data: 2024-04-12T04:06:48.837 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.837 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.837 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.837 DEBUG:boto:Params: {} 2024-04-12T04:06:48.838 DEBUG:boto:Token: None 2024-04-12T04:06:48.838 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj95 2024-04-12T04:06:48.838 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UZ+YGtmOPcziqsJBDVzQR7XeTbw= 2024-04-12T04:06:48.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UZ+YGtmOPcziqsJBDVzQR7XeTbw=', 'Content-Length': '0'} 2024-04-12T04:06:48.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000344ba2a0d2161e7a-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.840 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:06:48.840 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:06:48.840 DEBUG:boto:Method: HEAD 2024-04-12T04:06:48.840 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:06:48.840 DEBUG:boto:Data: 2024-04-12T04:06:48.840 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.840 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.840 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.840 DEBUG:boto:Params: {} 2024-04-12T04:06:48.841 DEBUG:boto:Token: None 2024-04-12T04:06:48.841 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj95 2024-04-12T04:06:48.841 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UZ+YGtmOPcziqsJBDVzQR7XeTbw= 2024-04-12T04:06:48.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UZ+YGtmOPcziqsJBDVzQR7XeTbw=', 'Content-Length': '0'} 2024-04-12T04:06:48.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018cd17669e42d3dd-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:06:48.843 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:06:48.843 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:06:48.843 DEBUG:boto:Method: GET 2024-04-12T04:06:48.843 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:06:48.843 DEBUG:boto:Data: 2024-04-12T04:06:48.843 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.843 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.844 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.844 DEBUG:boto:Params: {} 2024-04-12T04:06:48.844 DEBUG:boto:Token: None 2024-04-12T04:06:48.844 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj95 2024-04-12T04:06:48.844 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u9QfnoAxPI+/b/hNjuR+ejQZ6fQ= 2024-04-12T04:06:48.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u9QfnoAxPI+/b/hNjuR+ejQZ6fQ=', 'Content-Length': '0'} 2024-04-12T04:06:48.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002871c182e6e8a5a6-006618b358-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.886 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:06:48.887 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:06:48.887 DEBUG:boto:Method: GET 2024-04-12T04:06:48.887 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:06:48.887 DEBUG:boto:Data: 2024-04-12T04:06:48.887 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.887 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.887 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.887 DEBUG:boto:Params: {} 2024-04-12T04:06:48.887 DEBUG:boto:Token: None 2024-04-12T04:06:48.888 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/obj95 2024-04-12T04:06:48.888 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u9QfnoAxPI+/b/hNjuR+ejQZ6fQ= 2024-04-12T04:06:48.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u9QfnoAxPI+/b/hNjuR+ejQZ6fQ=', 'Content-Length': '0'} 2024-04-12T04:06:48.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e2ba0ccebe6b909-006618b358-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.890 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.890 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.890 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:06:48.890 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:06:48.891 DEBUG:boto:Method: GET 2024-04-12T04:06:48.891 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:06:48.891 DEBUG:boto:Data: 2024-04-12T04:06:48.891 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.891 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:06:48.891 DEBUG:boto:Port: 8001 2024-04-12T04:06:48.891 DEBUG:boto:Params: {} 2024-04-12T04:06:48.891 DEBUG:boto:Token: None 2024-04-12T04:06:48.891 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.891 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f17fcf9a3652af0-006618b358-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.894 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.895 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:48.895 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:48.895 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:06:48.895 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:06:48.895 DEBUG:boto:Method: GET 2024-04-12T04:06:48.895 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:06:48.895 DEBUG:boto:Data: 2024-04-12T04:06:48.895 DEBUG:boto:Headers: {} 2024-04-12T04:06:48.895 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:48.895 DEBUG:boto:Port: 8000 2024-04-12T04:06:48.896 DEBUG:boto:Params: {} 2024-04-12T04:06:48.896 DEBUG:boto:Token: None 2024-04-12T04:06:48.897 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:06:48 GMT /dkyges-61/ 2024-04-12T04:06:48.897 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc= 2024-04-12T04:06:48.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6OTqYBatt2KgVYb1ARlNP0VGtNc=', 'Content-Length': '0'} 2024-04-12T04:06:48.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dc3e40d3852e286-006618b358-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:06:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:48.901 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:06:48.901 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-61 zones={a2, a1} 2024-04-12T04:06:48.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:06:48.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:06:48.902 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:06:48.964 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:06:48.964 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:06:49.020 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:06:49.020 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-61 2024-04-12T04:06:49.032 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 20 2024-04-12T04:06:49.095 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:49.096 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:49.096 DEBUG:boto:Method: HEAD 2024-04-12T04:06:49.096 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:06:49.096 DEBUG:boto:Data: 2024-04-12T04:06:49.096 DEBUG:boto:Headers: {} 2024-04-12T04:06:49.096 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:49.096 DEBUG:boto:Port: 8000 2024-04-12T04:06:49.096 DEBUG:boto:Params: {} 2024-04-12T04:06:49.096 DEBUG:boto:Token: None 2024-04-12T04:06:49.097 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:49 GMT /dkyges-61/ 2024-04-12T04:06:49.097 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:++QCz4bf2vtlkfWmPtr7wnY5pL4= 2024-04-12T04:06:49.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:++QCz4bf2vtlkfWmPtr7wnY5pL4=', 'Content-Length': '0'} 2024-04-12T04:06:49.101 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001a0c30b3c47457f-006618b359-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:49.101 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:06:49.102 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:06:49.102 DEBUG:boto:Method: PUT 2024-04-12T04:06:49.102 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:06:49.102 DEBUG:boto:Data: 2024-04-12T04:06:49.102 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:06:49.102 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:49.102 DEBUG:boto:Port: 8000 2024-04-12T04:06:49.102 DEBUG:boto:Params: {} 2024-04-12T04:06:49.102 DEBUG:boto:Token: None 2024-04-12T04:06:49.103 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:06:49 GMT /dkyges-61/obj29 2024-04-12T04:06:49.103 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G2QmBW1/BCWvdnpsdp0CeIqAzzo= 2024-04-12T04:06:49.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:06:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G2QmBW1/BCWvdnpsdp0CeIqAzzo='} 2024-04-12T04:06:49.109 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004a9d5c15179cb033-006618b359-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:49.109 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:49.109 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:49.109 DEBUG:boto:Method: HEAD 2024-04-12T04:06:49.109 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:06:49.109 DEBUG:boto:Data: 2024-04-12T04:06:49.109 DEBUG:boto:Headers: {} 2024-04-12T04:06:49.109 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:49.109 DEBUG:boto:Port: 8000 2024-04-12T04:06:49.110 DEBUG:boto:Params: {} 2024-04-12T04:06:49.110 DEBUG:boto:Token: None 2024-04-12T04:06:49.110 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:49 GMT /dkyges-61/ 2024-04-12T04:06:49.110 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:++QCz4bf2vtlkfWmPtr7wnY5pL4= 2024-04-12T04:06:49.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:++QCz4bf2vtlkfWmPtr7wnY5pL4=', 'Content-Length': '0'} 2024-04-12T04:06:49.113 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009bbac5db973f30e0-006618b359-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:49.114 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:06:49.114 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:06:49.114 DEBUG:boto:Method: PUT 2024-04-12T04:06:49.114 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:06:49.114 DEBUG:boto:Data: 2024-04-12T04:06:49.114 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:06:49.114 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:49.114 DEBUG:boto:Port: 8000 2024-04-12T04:06:49.114 DEBUG:boto:Params: {} 2024-04-12T04:06:49.115 DEBUG:boto:Token: None 2024-04-12T04:06:49.115 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:06:49 GMT /dkyges-61/obj58 2024-04-12T04:06:49.115 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ON8ajoWOVKBs4oCRbZ/qZOjCUTk= 2024-04-12T04:06:49.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:06:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ON8ajoWOVKBs4oCRbZ/qZOjCUTk='} 2024-04-12T04:06:49.120 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e805f2ee1d0bc2ed-006618b359-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:49.120 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:49.120 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:49.120 DEBUG:boto:Method: HEAD 2024-04-12T04:06:49.121 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:06:49.121 DEBUG:boto:Data: 2024-04-12T04:06:49.121 DEBUG:boto:Headers: {} 2024-04-12T04:06:49.121 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:49.121 DEBUG:boto:Port: 8000 2024-04-12T04:06:49.121 DEBUG:boto:Params: {} 2024-04-12T04:06:49.121 DEBUG:boto:Token: None 2024-04-12T04:06:49.121 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:49 GMT /dkyges-61/ 2024-04-12T04:06:49.121 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:++QCz4bf2vtlkfWmPtr7wnY5pL4= 2024-04-12T04:06:49.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:++QCz4bf2vtlkfWmPtr7wnY5pL4=', 'Content-Length': '0'} 2024-04-12T04:06:49.125 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5b5d29eb159589a-006618b359-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:49.125 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:06:49.125 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:06:49.125 DEBUG:boto:Method: PUT 2024-04-12T04:06:49.125 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:06:49.125 DEBUG:boto:Data: 2024-04-12T04:06:49.125 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:06:49.126 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:49.126 DEBUG:boto:Port: 8000 2024-04-12T04:06:49.126 DEBUG:boto:Params: {} 2024-04-12T04:06:49.126 DEBUG:boto:Token: None 2024-04-12T04:06:49.126 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:06:49 GMT /dkyges-61/obj87 2024-04-12T04:06:49.126 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sEmkn1IVrwwwOGpFbI/35FGOZIs= 2024-04-12T04:06:49.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:06:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sEmkn1IVrwwwOGpFbI/35FGOZIs='} 2024-04-12T04:06:49.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de48369c34d5533b-006618b359-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:49.175 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:49.175 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:49.175 DEBUG:boto:Method: HEAD 2024-04-12T04:06:49.175 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:06:49.175 DEBUG:boto:Data: 2024-04-12T04:06:49.175 DEBUG:boto:Headers: {} 2024-04-12T04:06:49.175 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:49.175 DEBUG:boto:Port: 8000 2024-04-12T04:06:49.175 DEBUG:boto:Params: {} 2024-04-12T04:06:49.175 DEBUG:boto:Token: None 2024-04-12T04:06:49.176 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:49 GMT /dkyges-61/ 2024-04-12T04:06:49.176 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:++QCz4bf2vtlkfWmPtr7wnY5pL4= 2024-04-12T04:06:49.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:++QCz4bf2vtlkfWmPtr7wnY5pL4=', 'Content-Length': '0'} 2024-04-12T04:06:49.179 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ddcc9fc7d2ee85b-006618b359-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:49.180 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:06:49.180 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:06:49.180 DEBUG:boto:Method: PUT 2024-04-12T04:06:49.180 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:06:49.180 DEBUG:boto:Data: 2024-04-12T04:06:49.180 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:06:49.180 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:49.180 DEBUG:boto:Port: 8000 2024-04-12T04:06:49.180 DEBUG:boto:Params: {} 2024-04-12T04:06:49.181 DEBUG:boto:Token: None 2024-04-12T04:06:49.181 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:06:49 GMT /dkyges-61/obj116 2024-04-12T04:06:49.181 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iBDzIfXSB9MqvVSMkZ2kDN+2eDI= 2024-04-12T04:06:49.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:06:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iBDzIfXSB9MqvVSMkZ2kDN+2eDI='} 2024-04-12T04:06:49.187 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a99d8c1da6405432-006618b359-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:49.187 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:06:49.187 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:06:49.187 DEBUG:boto:Method: HEAD 2024-04-12T04:06:49.187 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:06:49.187 DEBUG:boto:Data: 2024-04-12T04:06:49.188 DEBUG:boto:Headers: {} 2024-04-12T04:06:49.188 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:49.188 DEBUG:boto:Port: 8000 2024-04-12T04:06:49.188 DEBUG:boto:Params: {} 2024-04-12T04:06:49.188 DEBUG:boto:Token: None 2024-04-12T04:06:49.188 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:06:49 GMT /dkyges-61/ 2024-04-12T04:06:49.188 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:++QCz4bf2vtlkfWmPtr7wnY5pL4= 2024-04-12T04:06:49.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:06:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:++QCz4bf2vtlkfWmPtr7wnY5pL4=', 'Content-Length': '0'} 2024-04-12T04:06:49.191 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003a94a8af814b509d-006618b359-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:49.191 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:06:49.192 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:06:49.192 DEBUG:boto:Method: PUT 2024-04-12T04:06:49.192 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:06:49.192 DEBUG:boto:Data: 2024-04-12T04:06:49.192 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:06:49.192 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:06:49.192 DEBUG:boto:Port: 8000 2024-04-12T04:06:49.192 DEBUG:boto:Params: {} 2024-04-12T04:06:49.192 DEBUG:boto:Token: None 2024-04-12T04:06:49.192 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:06:49 GMT /dkyges-61/obj145 2024-04-12T04:06:49.193 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OEOt2YjUbwRnxQshBSrJ8xd8cGE= 2024-04-12T04:06:49.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:06:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OEOt2YjUbwRnxQshBSrJ8xd8cGE='} 2024-04-12T04:06:49.235 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000008cdf728710375e3-006618b359-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:06:49.235 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-61 2024-04-12T04:06:49.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:06:49.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:06:49.235 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:06:49.303 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:06:49.303 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:06:49.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:06:49.365+0000 7f5e4701a780 1 waiting to reach latest gen 4, on 3.. 2024-04-12T04:06:54.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:06:54.366+0000 7f5e4701a780 1 waiting to reach latest gen 4, on 3.. 2024-04-12T04:06:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:06:59.366+0000 7f5e4701a780 1 waiting to reach latest gen 4, on 3.. 2024-04-12T04:07:04.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:07:04.369+0000 7f5e4701a780 1 waiting for incremental sync to catch up: 2024-04-12T04:07:04.354 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000002.55.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T04:07:04.355 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000002.55.6, 00000000001.56.6, , , , , 00000000001.97.6, , , , , , , , , , , 00000000002.125.6, 00000000001.95.6, , , , , , , , , , ] 2024-04-12T04:07:09.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:07:09.372+0000 7f5e4701a780 1 waiting for incremental sync to catch up: 2024-04-12T04:07:09.357 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000002.55.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T04:07:09.357 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000002.55.6, 00000000001.56.6, , , , , 00000000001.97.6, , , , , , , , , , , 00000000002.125.6, 00000000001.95.6, , , , , , , , , , ] 2024-04-12T04:07:14.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:07:14.374+0000 7f5e4701a780 1 waiting for incremental sync to catch up: 2024-04-12T04:07:14.359 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000002.55.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T04:07:14.359 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000002.55.6, 00000000001.56.6, , , , , 00000000001.97.6, , , , , , , , , , , 00000000002.125.6, 00000000001.95.6, , , , , , , , , , ] 2024-04-12T04:07:19.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:07:19.377+0000 7f5e4701a780 1 waiting for incremental sync to catch up: 2024-04-12T04:07:19.362 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000002.55.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T04:07:19.362 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000002.55.6, 00000000001.56.6, , , , , 00000000001.97.6, , , , , , , , , , , 00000000002.125.6, 00000000001.95.6, , , , , , , , , , ] 2024-04-12T04:07:24.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:07:24.379+0000 7f5e4701a780 1 waiting for incremental sync to catch up: 2024-04-12T04:07:24.364 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000002.55.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T04:07:24.364 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000002.55.6, 00000000001.56.6, , , , , 00000000001.97.6, , , , , , , , , , , 00000000002.125.6, 00000000001.95.6, , , , , , , , , , ] 2024-04-12T04:07:29.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:07:29.381+0000 7f5e4701a780 1 waiting for incremental sync to catch up: 2024-04-12T04:07:29.366 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000002.55.6, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2024-04-12T04:07:29.366 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000002.55.6, 00000000001.56.6, , , , , 00000000001.97.6, , , , , , , , , , , 00000000002.125.6, 00000000001.95.6, , , , , , , , , , ] 2024-04-12T04:07:34.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:07:34.383+0000 7f5e4701a780 1 waiting for incremental sync to catch up: 2024-04-12T04:07:34.368 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000002.55.6, 00000000001.56.6, , , , , , , , , , , , , , , , 00000000002.125.6, , , , , , , , , , , ] 2024-04-12T04:07:34.368 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000002.55.6, 00000000001.56.6, , , , , 00000000001.97.6, , , , , , , , , , , 00000000002.125.6, 00000000001.95.6, , , , , , , , , , ] 2024-04-12T04:07:39.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:07:39.385+0000 7f5e4701a780 1 waiting for incremental sync to catch up: 2024-04-12T04:07:39.370 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000002.55.6, 00000000001.56.6, , , , , , , , , , , , , , , , 00000000002.125.6, , , , , , , , , , , ] 2024-04-12T04:07:39.370 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000002.55.6, 00000000001.56.6, , , , , 00000000001.97.6, , , , , , , , , , , 00000000002.125.6, 00000000001.95.6, , , , , , , , , , ] 2024-04-12T04:07:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:07:44.388+0000 7f5e4701a780 1 bucket sync caught up with source: 2024-04-12T04:07:44.373 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [00000000002.55.6, 00000000001.56.6, , , , , 00000000001.97.6, , , , , , , , , , , 00000000002.125.6, 00000000001.95.6, , , , , , , , , , ] 2024-04-12T04:07:44.373 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [00000000002.55.6, 00000000001.56.6, , , , , 00000000001.97.6, , , , , , , , , , , 00000000002.125.6, 00000000001.95.6, , , , , , , , , , ] 2024-04-12T04:07:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:07:44.388+0000 7f5e4701a780 0 bucket checkpoint complete 2024-04-12T04:07:44.398 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-61 2024-04-12T04:07:44.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:07:44.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:07:44.399 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:07:44.465 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:07:44.466 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:07:44.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:07:44.523+0000 7fb4787cc780 1 waiting for incremental sync to catch up: 2024-04-12T04:07:44.507 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.2368.6, 00000000002.1411.6, 00000000001.2992.6, 00000000003.2384.6, 00000000004.2607.6, , 00000000001.3636.6, 00000000002.2561.6, 00000000005.2288.6, 00000000004.2618.6] 2024-04-12T04:07:44.508 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.2368.6, 00000000002.1411.6, 00000000001.2992.6, 00000000003.2384.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000002.2561.6, 00000000005.2288.6, 00000000004.2618.6] 2024-04-12T04:07:49.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:07:49.524+0000 7fb4787cc780 1 waiting for incremental sync to catch up: 2024-04-12T04:07:49.509 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.2368.6, 00000000002.1411.6, 00000000001.2992.6, 00000000003.2384.6, 00000000004.2607.6, , 00000000001.3636.6, 00000000002.2561.6, 00000000005.2288.6, 00000000004.2618.6] 2024-04-12T04:07:49.509 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.2368.6, 00000000002.1411.6, 00000000001.2992.6, 00000000003.2384.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000002.2561.6, 00000000005.2288.6, 00000000004.2618.6] 2024-04-12T04:07:54.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:07:54.525+0000 7fb4787cc780 1 waiting for incremental sync to catch up: 2024-04-12T04:07:54.510 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.2368.6, 00000000002.1411.6, 00000000001.2992.6, 00000000003.2384.6, 00000000004.2607.6, , 00000000001.3636.6, 00000000002.2561.6, 00000000005.2288.6, 00000000004.2618.6] 2024-04-12T04:07:54.510 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.2368.6, 00000000002.1411.6, 00000000001.2992.6, 00000000003.2384.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000002.2561.6, 00000000005.2288.6, 00000000004.2618.6] 2024-04-12T04:07:59.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:07:59.528+0000 7fb4787cc780 1 waiting for incremental sync to catch up: 2024-04-12T04:07:59.512 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.2368.6, 00000000002.1411.6, 00000000001.2992.6, 00000000003.2384.6, 00000000004.2607.6, , 00000000001.3636.6, 00000000002.2561.6, 00000000005.2288.6, 00000000004.2618.6] 2024-04-12T04:07:59.512 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.2368.6, 00000000002.1411.6, 00000000001.2992.6, 00000000003.2384.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000002.2561.6, 00000000005.2288.6, 00000000004.2618.6] 2024-04-12T04:08:00.355 INFO:tasks.rgw.c1.client.0.smithi043.stdout:2024-04-12T04:08:00.372+0000 7f1e272bd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:08:04.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:08:04.528+0000 7fb4787cc780 1 bucket sync caught up with source: 2024-04-12T04:08:04.513 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000002.2368.6, 00000000002.1411.6, 00000000001.2992.6, 00000000003.2384.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000002.2561.6, 00000000005.2288.6, 00000000004.2618.6] 2024-04-12T04:08:04.513 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000002.2368.6, 00000000002.1411.6, 00000000001.2992.6, 00000000003.2384.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000002.2561.6, 00000000005.2288.6, 00000000004.2618.6] 2024-04-12T04:08:04.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:08:04.528+0000 7fb4787cc780 0 bucket checkpoint complete 2024-04-12T04:08:04.532 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-61 zones={a2, a1} 2024-04-12T04:08:04.532 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:04.532 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:04.532 DEBUG:boto:Method: HEAD 2024-04-12T04:08:04.532 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:08:04.533 DEBUG:boto:Data: 2024-04-12T04:08:04.533 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.533 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.533 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.533 DEBUG:boto:Params: {} 2024-04-12T04:08:04.533 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:08:04.534 DEBUG:boto:Token: None 2024-04-12T04:08:04.534 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/ 2024-04-12T04:08:04.534 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NYlXVds0tk40jABS5SBhRwVSR7s= 2024-04-12T04:08:04.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NYlXVds0tk40jABS5SBhRwVSR7s=', 'Content-Length': '0'} 2024-04-12T04:08:04.547 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c0dda9b52e07a87-006618b3a4-1cfd-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.548 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:04.548 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:04.548 DEBUG:boto:Method: HEAD 2024-04-12T04:08:04.548 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:08:04.548 DEBUG:boto:Data: 2024-04-12T04:08:04.548 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.548 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.548 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.548 DEBUG:boto:Params: {} 2024-04-12T04:08:04.549 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:08:04.549 DEBUG:boto:Token: None 2024-04-12T04:08:04.549 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/ 2024-04-12T04:08:04.549 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NYlXVds0tk40jABS5SBhRwVSR7s= 2024-04-12T04:08:04.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NYlXVds0tk40jABS5SBhRwVSR7s=', 'Content-Length': '0'} 2024-04-12T04:08:04.563 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d627e420515fd85-006618b3a4-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.564 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:08:04.564 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:04.564 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:04.564 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:08:04.564 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:08:04.564 DEBUG:boto:Method: GET 2024-04-12T04:08:04.565 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:08:04.565 DEBUG:boto:Data: 2024-04-12T04:08:04.565 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.565 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.565 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.565 DEBUG:boto:Params: {} 2024-04-12T04:08:04.565 DEBUG:boto:Token: None 2024-04-12T04:08:04.565 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/?versions 2024-04-12T04:08:04.566 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:byqGKVyPJdAgLm3QFdgWX+62vHY= 2024-04-12T04:08:04.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:byqGKVyPJdAgLm3QFdgWX+62vHY=', 'Content-Length': '0'} 2024-04-12T04:08:04.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4d559b1b2956e41-006618b3a4-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.569 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:04.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:08:04.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T04:08:04.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T04:08:04.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T04:08:04.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:08:04.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T04:08:04.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:08:04.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T04:08:04.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T04:08:04.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:08:04.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:08:04.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T04:08:04.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:08:04.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T04:08:04.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:08:04.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:08:04.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T04:08:04.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T04:08:04.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:08:04.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T04:08:04.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T04:08:04.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:08:04.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T04:08:04.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T04:08:04.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T04:08:04.573 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:08:04.573 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:04.573 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:04.573 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:08:04.573 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:08:04.573 DEBUG:boto:Method: GET 2024-04-12T04:08:04.574 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:08:04.574 DEBUG:boto:Data: 2024-04-12T04:08:04.574 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.574 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.574 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.574 DEBUG:boto:Params: {} 2024-04-12T04:08:04.574 DEBUG:boto:Token: None 2024-04-12T04:08:04.574 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/?versions 2024-04-12T04:08:04.574 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:byqGKVyPJdAgLm3QFdgWX+62vHY= 2024-04-12T04:08:04.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:byqGKVyPJdAgLm3QFdgWX+62vHY=', 'Content-Length': '0'} 2024-04-12T04:08:04.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045124ea155886afb-006618b3a4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.579 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:04.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:08:04.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T04:08:04.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T04:08:04.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T04:08:04.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:08:04.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T04:08:04.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:08:04.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T04:08:04.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T04:08:04.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:08:04.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:08:04.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T04:08:04.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:08:04.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T04:08:04.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:08:04.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:08:04.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T04:08:04.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T04:08:04.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:08:04.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T04:08:04.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T04:08:04.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:08:04.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T04:08:04.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T04:08:04.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T04:08:04.583 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:04.583 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:04.583 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:08:04.583 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:08:04.583 DEBUG:boto:Method: GET 2024-04-12T04:08:04.583 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:08:04.583 DEBUG:boto:Data: 2024-04-12T04:08:04.583 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.583 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.584 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.584 DEBUG:boto:Params: {} 2024-04-12T04:08:04.584 DEBUG:boto:Token: None 2024-04-12T04:08:04.584 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/?versions 2024-04-12T04:08:04.584 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:byqGKVyPJdAgLm3QFdgWX+62vHY= 2024-04-12T04:08:04.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:byqGKVyPJdAgLm3QFdgWX+62vHY=', 'Content-Length': '0'} 2024-04-12T04:08:04.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caa226240c66d2be-006618b3a4-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.587 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:04.589 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:04.589 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:04.589 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:08:04.589 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:08:04.589 DEBUG:boto:Method: GET 2024-04-12T04:08:04.589 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:08:04.589 DEBUG:boto:Data: 2024-04-12T04:08:04.589 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.589 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.590 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.590 DEBUG:boto:Params: {} 2024-04-12T04:08:04.590 DEBUG:boto:Token: None 2024-04-12T04:08:04.590 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/?versions 2024-04-12T04:08:04.590 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:byqGKVyPJdAgLm3QFdgWX+62vHY= 2024-04-12T04:08:04.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:byqGKVyPJdAgLm3QFdgWX+62vHY=', 'Content-Length': '0'} 2024-04-12T04:08:04.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b49a63f235c8c20a-006618b3a4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.595 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:04.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:08:04.596 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:08:04.596 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:08:04.596 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.597 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.597 DEBUG:boto:Method: GET 2024-04-12T04:08:04.597 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.597 DEBUG:boto:Data: 2024-04-12T04:08:04.597 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.597 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.597 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.597 DEBUG:boto:Params: {} 2024-04-12T04:08:04.597 DEBUG:boto:Token: None 2024-04-12T04:08:04.597 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.598 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P/kutUFHUwEHJLQ+cS5QO8szDh8= 2024-04-12T04:08:04.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P/kutUFHUwEHJLQ+cS5QO8szDh8=', 'Content-Length': '0'} 2024-04-12T04:08:04.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddf86a09c8153910-006618b3a4-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.646 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:08:04.647 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:08:04.647 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.647 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.647 DEBUG:boto:Method: GET 2024-04-12T04:08:04.647 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.647 DEBUG:boto:Data: 2024-04-12T04:08:04.647 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.647 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.647 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.647 DEBUG:boto:Params: {} 2024-04-12T04:08:04.647 DEBUG:boto:Token: None 2024-04-12T04:08:04.648 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.648 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P/kutUFHUwEHJLQ+cS5QO8szDh8= 2024-04-12T04:08:04.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P/kutUFHUwEHJLQ+cS5QO8szDh8=', 'Content-Length': '0'} 2024-04-12T04:08:04.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000283c7ec6165ca3cd-006618b3a4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.650 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:08:04.650 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:08:04.650 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.651 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.651 DEBUG:boto:Method: HEAD 2024-04-12T04:08:04.651 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.651 DEBUG:boto:Data: 2024-04-12T04:08:04.651 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.651 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.651 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.651 DEBUG:boto:Params: {} 2024-04-12T04:08:04.651 DEBUG:boto:Token: None 2024-04-12T04:08:04.651 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.651 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZjK4mZGI6U5ey1/7kyHMrT+KUxo= 2024-04-12T04:08:04.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZjK4mZGI6U5ey1/7kyHMrT+KUxo=', 'Content-Length': '0'} 2024-04-12T04:08:04.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000629d9c612896846f-006618b3a4-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.653 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:08:04.653 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:08:04.653 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.654 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.654 DEBUG:boto:Method: HEAD 2024-04-12T04:08:04.654 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.654 DEBUG:boto:Data: 2024-04-12T04:08:04.654 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.654 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.654 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.654 DEBUG:boto:Params: {} 2024-04-12T04:08:04.654 DEBUG:boto:Token: None 2024-04-12T04:08:04.654 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.655 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZjK4mZGI6U5ey1/7kyHMrT+KUxo= 2024-04-12T04:08:04.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZjK4mZGI6U5ey1/7kyHMrT+KUxo=', 'Content-Length': '0'} 2024-04-12T04:08:04.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030732e1371e00121-006618b3a4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:08:04.657 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:08:04.657 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:08:04.657 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.657 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.657 DEBUG:boto:Method: GET 2024-04-12T04:08:04.657 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.657 DEBUG:boto:Data: 2024-04-12T04:08:04.657 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.658 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.658 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.658 DEBUG:boto:Params: {} 2024-04-12T04:08:04.658 DEBUG:boto:Token: None 2024-04-12T04:08:04.658 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.658 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P/kutUFHUwEHJLQ+cS5QO8szDh8= 2024-04-12T04:08:04.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P/kutUFHUwEHJLQ+cS5QO8szDh8=', 'Content-Length': '0'} 2024-04-12T04:08:04.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e99a6d758cbd803-006618b3a4-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.660 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:08:04.660 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:08:04.660 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.660 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.661 DEBUG:boto:Method: GET 2024-04-12T04:08:04.661 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.661 DEBUG:boto:Data: 2024-04-12T04:08:04.661 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.661 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.661 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.661 DEBUG:boto:Params: {} 2024-04-12T04:08:04.661 DEBUG:boto:Token: None 2024-04-12T04:08:04.661 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:08:04.661 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P/kutUFHUwEHJLQ+cS5QO8szDh8= 2024-04-12T04:08:04.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P/kutUFHUwEHJLQ+cS5QO8szDh8=', 'Content-Length': '0'} 2024-04-12T04:08:04.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b22b6eee3a54e063-006618b3a4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.664 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:04.664 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:04.664 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:08:04.664 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:08:04.664 DEBUG:boto:Method: GET 2024-04-12T04:08:04.664 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:08:04.664 DEBUG:boto:Data: 2024-04-12T04:08:04.664 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.664 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.664 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.664 DEBUG:boto:Params: {} 2024-04-12T04:08:04.665 DEBUG:boto:Token: None 2024-04-12T04:08:04.665 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/ 2024-04-12T04:08:04.665 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ= 2024-04-12T04:08:04.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ=', 'Content-Length': '0'} 2024-04-12T04:08:04.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021937c8f2d5fd412-006618b3a4-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.668 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:04.668 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:04.668 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:04.668 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:08:04.668 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:08:04.669 DEBUG:boto:Method: GET 2024-04-12T04:08:04.669 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:08:04.669 DEBUG:boto:Data: 2024-04-12T04:08:04.669 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.669 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.669 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.669 DEBUG:boto:Params: {} 2024-04-12T04:08:04.669 DEBUG:boto:Token: None 2024-04-12T04:08:04.669 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/ 2024-04-12T04:08:04.669 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ= 2024-04-12T04:08:04.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ=', 'Content-Length': '0'} 2024-04-12T04:08:04.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000421650c805cafc32-006618b3a4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.719 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:04.719 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:08:04.719 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:08:04.720 DEBUG:boto:Method: HEAD 2024-04-12T04:08:04.720 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:08:04.720 DEBUG:boto:Data: 2024-04-12T04:08:04.720 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.720 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.720 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.720 DEBUG:boto:Params: {} 2024-04-12T04:08:04.720 DEBUG:boto:Token: None 2024-04-12T04:08:04.720 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj11 2024-04-12T04:08:04.720 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0csWcQDklOg4KSq5TaLtW0BJGOk= 2024-04-12T04:08:04.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0csWcQDklOg4KSq5TaLtW0BJGOk=', 'Content-Length': '0'} 2024-04-12T04:08:04.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000229ee0a8cee3d103-006618b3a4-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.723 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:08:04.724 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:08:04.724 DEBUG:boto:Method: HEAD 2024-04-12T04:08:04.724 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:08:04.724 DEBUG:boto:Data: 2024-04-12T04:08:04.724 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.724 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.724 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.724 DEBUG:boto:Params: {} 2024-04-12T04:08:04.724 DEBUG:boto:Token: None 2024-04-12T04:08:04.724 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj11 2024-04-12T04:08:04.724 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0csWcQDklOg4KSq5TaLtW0BJGOk= 2024-04-12T04:08:04.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0csWcQDklOg4KSq5TaLtW0BJGOk=', 'Content-Length': '0'} 2024-04-12T04:08:04.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074cea1f2009b6602-006618b3a4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:08:04.727 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:08:04.727 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:08:04.727 DEBUG:boto:Method: GET 2024-04-12T04:08:04.727 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:08:04.727 DEBUG:boto:Data: 2024-04-12T04:08:04.727 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.727 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.728 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.728 DEBUG:boto:Params: {} 2024-04-12T04:08:04.728 DEBUG:boto:Token: None 2024-04-12T04:08:04.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj11 2024-04-12T04:08:04.729 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zGlT+Hvy7nDeOxQXVKpEnwVS398= 2024-04-12T04:08:04.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zGlT+Hvy7nDeOxQXVKpEnwVS398=', 'Content-Length': '0'} 2024-04-12T04:08:04.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d60327c87a488577-006618b3a4-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.731 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:08:04.731 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:08:04.731 DEBUG:boto:Method: GET 2024-04-12T04:08:04.731 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:08:04.731 DEBUG:boto:Data: 2024-04-12T04:08:04.731 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.731 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.731 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.731 DEBUG:boto:Params: {} 2024-04-12T04:08:04.731 DEBUG:boto:Token: None 2024-04-12T04:08:04.732 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj11 2024-04-12T04:08:04.732 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zGlT+Hvy7nDeOxQXVKpEnwVS398= 2024-04-12T04:08:04.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zGlT+Hvy7nDeOxQXVKpEnwVS398=', 'Content-Length': '0'} 2024-04-12T04:08:04.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c7410ee9e389784-006618b3a4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.734 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:04.734 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:04.734 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:08:04.734 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:08:04.734 DEBUG:boto:Method: GET 2024-04-12T04:08:04.734 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:08:04.735 DEBUG:boto:Data: 2024-04-12T04:08:04.735 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.735 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.735 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.735 DEBUG:boto:Params: {} 2024-04-12T04:08:04.735 DEBUG:boto:Token: None 2024-04-12T04:08:04.735 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/ 2024-04-12T04:08:04.735 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ= 2024-04-12T04:08:04.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ=', 'Content-Length': '0'} 2024-04-12T04:08:04.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a446c801b4e81d5-006618b3a4-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.778 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:04.779 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:04.779 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:04.779 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:08:04.779 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:08:04.779 DEBUG:boto:Method: GET 2024-04-12T04:08:04.779 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:08:04.779 DEBUG:boto:Data: 2024-04-12T04:08:04.780 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.780 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.780 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.780 DEBUG:boto:Params: {} 2024-04-12T04:08:04.780 DEBUG:boto:Token: None 2024-04-12T04:08:04.780 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/ 2024-04-12T04:08:04.780 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ= 2024-04-12T04:08:04.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ=', 'Content-Length': '0'} 2024-04-12T04:08:04.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083ef009ef069ddd5-006618b3a4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.785 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:04.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:08:04.785 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:08:04.785 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:08:04.786 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.786 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.786 DEBUG:boto:Method: GET 2024-04-12T04:08:04.786 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.786 DEBUG:boto:Data: 2024-04-12T04:08:04.786 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.786 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.786 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.786 DEBUG:boto:Params: {} 2024-04-12T04:08:04.786 DEBUG:boto:Token: None 2024-04-12T04:08:04.787 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.787 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yaNPBF7yDn1AcMpnfKoxTF19R3Y= 2024-04-12T04:08:04.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yaNPBF7yDn1AcMpnfKoxTF19R3Y=', 'Content-Length': '0'} 2024-04-12T04:08:04.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5c44fe89fa66e0d-006618b3a4-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.789 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:08:04.789 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:08:04.789 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.789 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.789 DEBUG:boto:Method: GET 2024-04-12T04:08:04.790 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.790 DEBUG:boto:Data: 2024-04-12T04:08:04.790 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.790 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.790 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.790 DEBUG:boto:Params: {} 2024-04-12T04:08:04.790 DEBUG:boto:Token: None 2024-04-12T04:08:04.790 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.790 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yaNPBF7yDn1AcMpnfKoxTF19R3Y= 2024-04-12T04:08:04.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yaNPBF7yDn1AcMpnfKoxTF19R3Y=', 'Content-Length': '0'} 2024-04-12T04:08:04.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ce3ecdc5c19a94c-006618b3a4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.838 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:08:04.839 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:08:04.839 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.839 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.839 DEBUG:boto:Method: HEAD 2024-04-12T04:08:04.839 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.839 DEBUG:boto:Data: 2024-04-12T04:08:04.839 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.839 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.839 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.839 DEBUG:boto:Params: {} 2024-04-12T04:08:04.840 DEBUG:boto:Token: None 2024-04-12T04:08:04.840 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.840 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PvJyOsf4WPSA2U4WIrqNAFGWZqM= 2024-04-12T04:08:04.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PvJyOsf4WPSA2U4WIrqNAFGWZqM=', 'Content-Length': '0'} 2024-04-12T04:08:04.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001048e3ae81946e4b-006618b3a4-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.842 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:08:04.842 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:08:04.842 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.842 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.842 DEBUG:boto:Method: HEAD 2024-04-12T04:08:04.843 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.843 DEBUG:boto:Data: 2024-04-12T04:08:04.843 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.843 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.843 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.843 DEBUG:boto:Params: {} 2024-04-12T04:08:04.843 DEBUG:boto:Token: None 2024-04-12T04:08:04.843 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.843 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PvJyOsf4WPSA2U4WIrqNAFGWZqM= 2024-04-12T04:08:04.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PvJyOsf4WPSA2U4WIrqNAFGWZqM=', 'Content-Length': '0'} 2024-04-12T04:08:04.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d19c3813891b52-006618b3a4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:08:04.846 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:08:04.846 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:08:04.846 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.846 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.846 DEBUG:boto:Method: GET 2024-04-12T04:08:04.846 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.846 DEBUG:boto:Data: 2024-04-12T04:08:04.846 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.846 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.846 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.847 DEBUG:boto:Params: {} 2024-04-12T04:08:04.847 DEBUG:boto:Token: None 2024-04-12T04:08:04.847 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.847 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yaNPBF7yDn1AcMpnfKoxTF19R3Y= 2024-04-12T04:08:04.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yaNPBF7yDn1AcMpnfKoxTF19R3Y=', 'Content-Length': '0'} 2024-04-12T04:08:04.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088b55efc540ac09c-006618b3a4-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.849 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:08:04.849 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:08:04.850 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.850 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.850 DEBUG:boto:Method: GET 2024-04-12T04:08:04.850 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.850 DEBUG:boto:Data: 2024-04-12T04:08:04.850 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.850 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.850 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.850 DEBUG:boto:Params: {} 2024-04-12T04:08:04.850 DEBUG:boto:Token: None 2024-04-12T04:08:04.850 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:08:04.851 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yaNPBF7yDn1AcMpnfKoxTF19R3Y= 2024-04-12T04:08:04.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yaNPBF7yDn1AcMpnfKoxTF19R3Y=', 'Content-Length': '0'} 2024-04-12T04:08:04.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066979bee3c9f8e45-006618b3a4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.853 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:04.853 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:04.853 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:08:04.853 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:08:04.853 DEBUG:boto:Method: GET 2024-04-12T04:08:04.853 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:08:04.854 DEBUG:boto:Data: 2024-04-12T04:08:04.854 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.854 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.854 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.854 DEBUG:boto:Params: {} 2024-04-12T04:08:04.854 DEBUG:boto:Token: None 2024-04-12T04:08:04.854 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/ 2024-04-12T04:08:04.854 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ= 2024-04-12T04:08:04.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ=', 'Content-Length': '0'} 2024-04-12T04:08:04.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009a3fd32b3f853a2-006618b3a4-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.857 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:04.857 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:04.857 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:04.857 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:08:04.858 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:08:04.858 DEBUG:boto:Method: GET 2024-04-12T04:08:04.858 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:08:04.858 DEBUG:boto:Data: 2024-04-12T04:08:04.858 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.858 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.858 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.858 DEBUG:boto:Params: {} 2024-04-12T04:08:04.858 DEBUG:boto:Token: None 2024-04-12T04:08:04.858 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/ 2024-04-12T04:08:04.859 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ= 2024-04-12T04:08:04.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ=', 'Content-Length': '0'} 2024-04-12T04:08:04.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004097f94bf080fb2c-006618b3a4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.863 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:04.863 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:08:04.863 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:08:04.863 DEBUG:boto:Method: HEAD 2024-04-12T04:08:04.863 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:08:04.863 DEBUG:boto:Data: 2024-04-12T04:08:04.863 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.863 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.864 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.864 DEBUG:boto:Params: {} 2024-04-12T04:08:04.864 DEBUG:boto:Token: None 2024-04-12T04:08:04.864 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj115 2024-04-12T04:08:04.864 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0EIiIr8oz7cfg7zfoSJ32pJcYPg= 2024-04-12T04:08:04.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0EIiIr8oz7cfg7zfoSJ32pJcYPg=', 'Content-Length': '0'} 2024-04-12T04:08:04.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054a7de5ddf51be9a-006618b3a4-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.866 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:08:04.866 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:08:04.866 DEBUG:boto:Method: HEAD 2024-04-12T04:08:04.866 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:08:04.866 DEBUG:boto:Data: 2024-04-12T04:08:04.866 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.866 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.866 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.867 DEBUG:boto:Params: {} 2024-04-12T04:08:04.867 DEBUG:boto:Token: None 2024-04-12T04:08:04.867 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj115 2024-04-12T04:08:04.867 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0EIiIr8oz7cfg7zfoSJ32pJcYPg= 2024-04-12T04:08:04.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0EIiIr8oz7cfg7zfoSJ32pJcYPg=', 'Content-Length': '0'} 2024-04-12T04:08:04.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055d1294bd6993297-006618b3a4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:08:04.869 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:08:04.869 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:08:04.869 DEBUG:boto:Method: GET 2024-04-12T04:08:04.869 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:08:04.869 DEBUG:boto:Data: 2024-04-12T04:08:04.869 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.869 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.870 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.870 DEBUG:boto:Params: {} 2024-04-12T04:08:04.870 DEBUG:boto:Token: None 2024-04-12T04:08:04.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj115 2024-04-12T04:08:04.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nhWzXsSL5suWJy9sewBlxoHdCy8= 2024-04-12T04:08:04.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nhWzXsSL5suWJy9sewBlxoHdCy8=', 'Content-Length': '0'} 2024-04-12T04:08:04.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7fef1614539c6ab-006618b3a4-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.872 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:08:04.872 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:08:04.873 DEBUG:boto:Method: GET 2024-04-12T04:08:04.873 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:08:04.873 DEBUG:boto:Data: 2024-04-12T04:08:04.873 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.873 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.873 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.873 DEBUG:boto:Params: {} 2024-04-12T04:08:04.873 DEBUG:boto:Token: None 2024-04-12T04:08:04.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj115 2024-04-12T04:08:04.873 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nhWzXsSL5suWJy9sewBlxoHdCy8= 2024-04-12T04:08:04.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nhWzXsSL5suWJy9sewBlxoHdCy8=', 'Content-Length': '0'} 2024-04-12T04:08:04.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab8d217bcef09d5a-006618b3a4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.875 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:04.876 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:04.876 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:08:04.876 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:08:04.876 DEBUG:boto:Method: GET 2024-04-12T04:08:04.876 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:08:04.876 DEBUG:boto:Data: 2024-04-12T04:08:04.876 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.876 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.876 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.876 DEBUG:boto:Params: {} 2024-04-12T04:08:04.877 DEBUG:boto:Token: None 2024-04-12T04:08:04.877 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/ 2024-04-12T04:08:04.877 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ= 2024-04-12T04:08:04.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ=', 'Content-Length': '0'} 2024-04-12T04:08:04.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b00cb95966ab3d6-006618b3a4-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.879 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:04.880 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:04.880 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:04.880 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:08:04.880 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:08:04.880 DEBUG:boto:Method: GET 2024-04-12T04:08:04.880 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:08:04.880 DEBUG:boto:Data: 2024-04-12T04:08:04.880 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.881 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.881 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.881 DEBUG:boto:Params: {} 2024-04-12T04:08:04.881 DEBUG:boto:Token: None 2024-04-12T04:08:04.881 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/ 2024-04-12T04:08:04.881 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ= 2024-04-12T04:08:04.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ=', 'Content-Length': '0'} 2024-04-12T04:08:04.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b925190f008eb66f-006618b3a4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.926 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:04.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:08:04.927 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:08:04.927 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:08:04.927 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.927 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.927 DEBUG:boto:Method: GET 2024-04-12T04:08:04.927 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.928 DEBUG:boto:Data: 2024-04-12T04:08:04.928 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.928 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.928 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.928 DEBUG:boto:Params: {} 2024-04-12T04:08:04.928 DEBUG:boto:Token: None 2024-04-12T04:08:04.928 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.928 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Xgahb4+vzgp1ai0S4/bVdt+eFqs= 2024-04-12T04:08:04.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Xgahb4+vzgp1ai0S4/bVdt+eFqs=', 'Content-Length': '0'} 2024-04-12T04:08:04.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009356e6e18f46845b-006618b3a4-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.930 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:08:04.931 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:08:04.931 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.931 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.931 DEBUG:boto:Method: GET 2024-04-12T04:08:04.931 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.931 DEBUG:boto:Data: 2024-04-12T04:08:04.931 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.931 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.931 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.931 DEBUG:boto:Params: {} 2024-04-12T04:08:04.931 DEBUG:boto:Token: None 2024-04-12T04:08:04.932 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.932 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Xgahb4+vzgp1ai0S4/bVdt+eFqs= 2024-04-12T04:08:04.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Xgahb4+vzgp1ai0S4/bVdt+eFqs=', 'Content-Length': '0'} 2024-04-12T04:08:04.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d578894d631dc45b-006618b3a4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.934 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:08:04.934 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:08:04.934 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.934 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.934 DEBUG:boto:Method: HEAD 2024-04-12T04:08:04.934 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.934 DEBUG:boto:Data: 2024-04-12T04:08:04.935 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.935 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.935 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.935 DEBUG:boto:Params: {} 2024-04-12T04:08:04.935 DEBUG:boto:Token: None 2024-04-12T04:08:04.935 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.935 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8pNfI262xO7R0uvfhtVCmqglVdo= 2024-04-12T04:08:04.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8pNfI262xO7R0uvfhtVCmqglVdo=', 'Content-Length': '0'} 2024-04-12T04:08:04.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be84a15bf0eb8934-006618b3a4-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.937 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:08:04.937 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:08:04.937 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.938 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.938 DEBUG:boto:Method: HEAD 2024-04-12T04:08:04.938 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.938 DEBUG:boto:Data: 2024-04-12T04:08:04.938 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.938 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.938 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.938 DEBUG:boto:Params: {} 2024-04-12T04:08:04.938 DEBUG:boto:Token: None 2024-04-12T04:08:04.938 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.938 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8pNfI262xO7R0uvfhtVCmqglVdo= 2024-04-12T04:08:04.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8pNfI262xO7R0uvfhtVCmqglVdo=', 'Content-Length': '0'} 2024-04-12T04:08:04.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d188d40b16125398-006618b3a4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:08:04.941 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:08:04.941 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:08:04.941 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.941 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.941 DEBUG:boto:Method: GET 2024-04-12T04:08:04.941 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.941 DEBUG:boto:Data: 2024-04-12T04:08:04.942 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.942 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.942 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.942 DEBUG:boto:Params: {} 2024-04-12T04:08:04.942 DEBUG:boto:Token: None 2024-04-12T04:08:04.942 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.942 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Xgahb4+vzgp1ai0S4/bVdt+eFqs= 2024-04-12T04:08:04.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Xgahb4+vzgp1ai0S4/bVdt+eFqs=', 'Content-Length': '0'} 2024-04-12T04:08:04.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000260cfad9a41e79b2-006618b3a4-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.944 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:08:04.945 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:08:04.945 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.945 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.945 DEBUG:boto:Method: GET 2024-04-12T04:08:04.945 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.945 DEBUG:boto:Data: 2024-04-12T04:08:04.945 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.945 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.945 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.945 DEBUG:boto:Params: {} 2024-04-12T04:08:04.945 DEBUG:boto:Token: None 2024-04-12T04:08:04.946 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:08:04.946 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Xgahb4+vzgp1ai0S4/bVdt+eFqs= 2024-04-12T04:08:04.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Xgahb4+vzgp1ai0S4/bVdt+eFqs=', 'Content-Length': '0'} 2024-04-12T04:08:04.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000753ebd76ff1869e7-006618b3a4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.948 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:04.948 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:04.948 DEBUG:boto:path=/dkyges-61/?prefix=obj116 2024-04-12T04:08:04.948 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj116 2024-04-12T04:08:04.948 DEBUG:boto:Method: GET 2024-04-12T04:08:04.948 DEBUG:boto:Path: /dkyges-61/?prefix=obj116 2024-04-12T04:08:04.949 DEBUG:boto:Data: 2024-04-12T04:08:04.949 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.949 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.949 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.949 DEBUG:boto:Params: {} 2024-04-12T04:08:04.949 DEBUG:boto:Token: None 2024-04-12T04:08:04.949 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/ 2024-04-12T04:08:04.949 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ= 2024-04-12T04:08:04.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ=', 'Content-Length': '0'} 2024-04-12T04:08:04.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062eca6543186bdfe-006618b3a4-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.952 DEBUG:boto:b'dkyges-61obj1161000falseobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:04.952 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:04.952 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:04.952 DEBUG:boto:path=/dkyges-61/?prefix=obj116 2024-04-12T04:08:04.952 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj116 2024-04-12T04:08:04.952 DEBUG:boto:Method: GET 2024-04-12T04:08:04.952 DEBUG:boto:Path: /dkyges-61/?prefix=obj116 2024-04-12T04:08:04.952 DEBUG:boto:Data: 2024-04-12T04:08:04.953 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.953 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.953 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.953 DEBUG:boto:Params: {} 2024-04-12T04:08:04.953 DEBUG:boto:Token: None 2024-04-12T04:08:04.953 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/ 2024-04-12T04:08:04.953 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ= 2024-04-12T04:08:04.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9VCtFTOCcwuXt1xy77YUpQX2dcQ=', 'Content-Length': '0'} 2024-04-12T04:08:04.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0d740f224774e1f-006618b3a4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.957 DEBUG:boto:b'dkyges-61obj1161000falseobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:04.958 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:08:04.958 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:08:04.958 DEBUG:boto:Method: HEAD 2024-04-12T04:08:04.958 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:08:04.958 DEBUG:boto:Data: 2024-04-12T04:08:04.958 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.958 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.958 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.958 DEBUG:boto:Params: {} 2024-04-12T04:08:04.959 DEBUG:boto:Token: None 2024-04-12T04:08:04.959 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj116 2024-04-12T04:08:04.959 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aiiOPXqjq8dJegDAQ+V3O3llLT8= 2024-04-12T04:08:04.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aiiOPXqjq8dJegDAQ+V3O3llLT8=', 'Content-Length': '0'} 2024-04-12T04:08:04.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eefe4947b5048c72-006618b3a4-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.961 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:08:04.961 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:08:04.961 DEBUG:boto:Method: HEAD 2024-04-12T04:08:04.961 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:08:04.961 DEBUG:boto:Data: 2024-04-12T04:08:04.961 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.961 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:04.962 DEBUG:boto:Port: 8000 2024-04-12T04:08:04.962 DEBUG:boto:Params: {} 2024-04-12T04:08:04.962 DEBUG:boto:Token: None 2024-04-12T04:08:04.962 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj116 2024-04-12T04:08:04.962 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aiiOPXqjq8dJegDAQ+V3O3llLT8= 2024-04-12T04:08:04.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aiiOPXqjq8dJegDAQ+V3O3llLT8=', 'Content-Length': '0'} 2024-04-12T04:08:04.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027740ecd4b1fda05-006618b3a4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:04.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:08:04.964 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:08:04.964 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:08:04.965 DEBUG:boto:Method: GET 2024-04-12T04:08:04.965 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:08:04.965 DEBUG:boto:Data: 2024-04-12T04:08:04.965 DEBUG:boto:Headers: {} 2024-04-12T04:08:04.965 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:04.965 DEBUG:boto:Port: 8001 2024-04-12T04:08:04.965 DEBUG:boto:Params: {} 2024-04-12T04:08:04.965 DEBUG:boto:Token: None 2024-04-12T04:08:04.965 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:04 GMT /dkyges-61/obj116 2024-04-12T04:08:04.965 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eSAWMDtO+n8fQZF7g56mLniMbhk= 2024-04-12T04:08:04.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eSAWMDtO+n8fQZF7g56mLniMbhk=', 'Content-Length': '0'} 2024-04-12T04:08:04.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009105b8d7d942a59-006618b3a4-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.010 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:08:05.011 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:08:05.011 DEBUG:boto:Method: GET 2024-04-12T04:08:05.011 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:08:05.011 DEBUG:boto:Data: 2024-04-12T04:08:05.011 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.011 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.011 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.011 DEBUG:boto:Params: {} 2024-04-12T04:08:05.011 DEBUG:boto:Token: None 2024-04-12T04:08:05.012 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj116 2024-04-12T04:08:05.012 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MwTWyG11miJDs2V9FuocSJnwTuk= 2024-04-12T04:08:05.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MwTWyG11miJDs2V9FuocSJnwTuk=', 'Content-Length': '0'} 2024-04-12T04:08:05.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fc8175fd3cda678-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.014 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.014 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.014 DEBUG:boto:path=/dkyges-61/?prefix=obj116 2024-04-12T04:08:05.014 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj116 2024-04-12T04:08:05.014 DEBUG:boto:Method: GET 2024-04-12T04:08:05.014 DEBUG:boto:Path: /dkyges-61/?prefix=obj116 2024-04-12T04:08:05.015 DEBUG:boto:Data: 2024-04-12T04:08:05.015 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.015 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.015 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.015 DEBUG:boto:Params: {} 2024-04-12T04:08:05.015 DEBUG:boto:Token: None 2024-04-12T04:08:05.015 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.015 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e49c1b95f743c6d-006618b3a5-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.018 DEBUG:boto:b'dkyges-61obj1161000falseobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.018 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.018 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.018 DEBUG:boto:path=/dkyges-61/?prefix=obj116 2024-04-12T04:08:05.019 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj116 2024-04-12T04:08:05.019 DEBUG:boto:Method: GET 2024-04-12T04:08:05.019 DEBUG:boto:Path: /dkyges-61/?prefix=obj116 2024-04-12T04:08:05.019 DEBUG:boto:Data: 2024-04-12T04:08:05.019 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.019 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.019 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.019 DEBUG:boto:Params: {} 2024-04-12T04:08:05.019 DEBUG:boto:Token: None 2024-04-12T04:08:05.019 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.020 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009983a8dfbf7bd9c2-006618b3a5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.024 DEBUG:boto:b'dkyges-61obj1161000falseobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:08:05.024 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:08:05.024 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:08:05.024 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.024 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.024 DEBUG:boto:Method: GET 2024-04-12T04:08:05.025 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.025 DEBUG:boto:Data: 2024-04-12T04:08:05.025 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.025 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.025 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.025 DEBUG:boto:Params: {} 2024-04-12T04:08:05.025 DEBUG:boto:Token: None 2024-04-12T04:08:05.025 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.025 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pTf/3lRhkhXg/RTTR2irLcKkhYA= 2024-04-12T04:08:05.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pTf/3lRhkhXg/RTTR2irLcKkhYA=', 'Content-Length': '0'} 2024-04-12T04:08:05.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dfda74a97ef3419-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.030 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:08:05.030 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:08:05.030 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.030 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.030 DEBUG:boto:Method: GET 2024-04-12T04:08:05.030 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.030 DEBUG:boto:Data: 2024-04-12T04:08:05.030 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.030 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.030 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.031 DEBUG:boto:Params: {} 2024-04-12T04:08:05.031 DEBUG:boto:Token: None 2024-04-12T04:08:05.031 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.031 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pTf/3lRhkhXg/RTTR2irLcKkhYA= 2024-04-12T04:08:05.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pTf/3lRhkhXg/RTTR2irLcKkhYA=', 'Content-Length': '0'} 2024-04-12T04:08:05.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee3604467fd1165b-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.078 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:08:05.078 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:08:05.079 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.079 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.079 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.079 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.079 DEBUG:boto:Data: 2024-04-12T04:08:05.079 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.079 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.079 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.079 DEBUG:boto:Params: {} 2024-04-12T04:08:05.079 DEBUG:boto:Token: None 2024-04-12T04:08:05.080 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.080 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Go3tTAGLzetI63OIz5hRMUQmDdw= 2024-04-12T04:08:05.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Go3tTAGLzetI63OIz5hRMUQmDdw=', 'Content-Length': '0'} 2024-04-12T04:08:05.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db146c4f47576eb0-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.082 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:08:05.082 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:08:05.082 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.082 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.082 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.083 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.083 DEBUG:boto:Data: 2024-04-12T04:08:05.083 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.083 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.083 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.083 DEBUG:boto:Params: {} 2024-04-12T04:08:05.083 DEBUG:boto:Token: None 2024-04-12T04:08:05.083 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.083 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Go3tTAGLzetI63OIz5hRMUQmDdw= 2024-04-12T04:08:05.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Go3tTAGLzetI63OIz5hRMUQmDdw=', 'Content-Length': '0'} 2024-04-12T04:08:05.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bf59bcaa3b8fc8b-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:08:05.086 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:08:05.086 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:08:05.086 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.086 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.086 DEBUG:boto:Method: GET 2024-04-12T04:08:05.086 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.086 DEBUG:boto:Data: 2024-04-12T04:08:05.086 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.086 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.087 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.087 DEBUG:boto:Params: {} 2024-04-12T04:08:05.087 DEBUG:boto:Token: None 2024-04-12T04:08:05.087 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.087 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pTf/3lRhkhXg/RTTR2irLcKkhYA= 2024-04-12T04:08:05.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pTf/3lRhkhXg/RTTR2irLcKkhYA=', 'Content-Length': '0'} 2024-04-12T04:08:05.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056687df372a9621d-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.089 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:08:05.089 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:08:05.089 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.089 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.090 DEBUG:boto:Method: GET 2024-04-12T04:08:05.090 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.090 DEBUG:boto:Data: 2024-04-12T04:08:05.090 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.090 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.090 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.090 DEBUG:boto:Params: {} 2024-04-12T04:08:05.090 DEBUG:boto:Token: None 2024-04-12T04:08:05.090 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:08:05.090 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pTf/3lRhkhXg/RTTR2irLcKkhYA= 2024-04-12T04:08:05.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pTf/3lRhkhXg/RTTR2irLcKkhYA=', 'Content-Length': '0'} 2024-04-12T04:08:05.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023fbd71e45e130e0-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.093 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.093 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.093 DEBUG:boto:path=/dkyges-61/?prefix=obj145 2024-04-12T04:08:05.093 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj145 2024-04-12T04:08:05.093 DEBUG:boto:Method: GET 2024-04-12T04:08:05.093 DEBUG:boto:Path: /dkyges-61/?prefix=obj145 2024-04-12T04:08:05.093 DEBUG:boto:Data: 2024-04-12T04:08:05.093 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.093 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.093 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.094 DEBUG:boto:Params: {} 2024-04-12T04:08:05.094 DEBUG:boto:Token: None 2024-04-12T04:08:05.094 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.094 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074de10fc6ae3de9b-006618b3a5-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.097 DEBUG:boto:b'dkyges-61obj1451000falseobj1452024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.097 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.097 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.097 DEBUG:boto:path=/dkyges-61/?prefix=obj145 2024-04-12T04:08:05.097 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj145 2024-04-12T04:08:05.097 DEBUG:boto:Method: GET 2024-04-12T04:08:05.098 DEBUG:boto:Path: /dkyges-61/?prefix=obj145 2024-04-12T04:08:05.098 DEBUG:boto:Data: 2024-04-12T04:08:05.098 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.098 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.098 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.098 DEBUG:boto:Params: {} 2024-04-12T04:08:05.098 DEBUG:boto:Token: None 2024-04-12T04:08:05.098 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.098 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088eabbfdb20f67c2-006618b3a5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.104 DEBUG:boto:b'dkyges-61obj1451000falseobj1452024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.104 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:08:05.104 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:08:05.104 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.104 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:08:05.104 DEBUG:boto:Data: 2024-04-12T04:08:05.105 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.105 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.105 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.105 DEBUG:boto:Params: {} 2024-04-12T04:08:05.105 DEBUG:boto:Token: None 2024-04-12T04:08:05.105 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj145 2024-04-12T04:08:05.105 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Srze6HCP25DqkyhyD3VDg03Ybgk= 2024-04-12T04:08:05.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Srze6HCP25DqkyhyD3VDg03Ybgk=', 'Content-Length': '0'} 2024-04-12T04:08:05.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7f9e356ed93f5a3-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.107 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:08:05.107 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:08:05.108 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.108 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:08:05.108 DEBUG:boto:Data: 2024-04-12T04:08:05.108 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.108 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.108 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.108 DEBUG:boto:Params: {} 2024-04-12T04:08:05.108 DEBUG:boto:Token: None 2024-04-12T04:08:05.108 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj145 2024-04-12T04:08:05.108 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Srze6HCP25DqkyhyD3VDg03Ybgk= 2024-04-12T04:08:05.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Srze6HCP25DqkyhyD3VDg03Ybgk=', 'Content-Length': '0'} 2024-04-12T04:08:05.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035809ca847583c62-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:08:05.111 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:08:05.111 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:08:05.111 DEBUG:boto:Method: GET 2024-04-12T04:08:05.111 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:08:05.111 DEBUG:boto:Data: 2024-04-12T04:08:05.111 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.111 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.111 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.111 DEBUG:boto:Params: {} 2024-04-12T04:08:05.111 DEBUG:boto:Token: None 2024-04-12T04:08:05.112 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj145 2024-04-12T04:08:05.112 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fZT1ZTFWLV98DO98h7PRTnB1gGU= 2024-04-12T04:08:05.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fZT1ZTFWLV98DO98h7PRTnB1gGU=', 'Content-Length': '0'} 2024-04-12T04:08:05.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002428a2f03f2091c2-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.114 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:08:05.114 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:08:05.114 DEBUG:boto:Method: GET 2024-04-12T04:08:05.114 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:08:05.114 DEBUG:boto:Data: 2024-04-12T04:08:05.114 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.114 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.114 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.114 DEBUG:boto:Params: {} 2024-04-12T04:08:05.115 DEBUG:boto:Token: None 2024-04-12T04:08:05.115 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj145 2024-04-12T04:08:05.115 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fZT1ZTFWLV98DO98h7PRTnB1gGU= 2024-04-12T04:08:05.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fZT1ZTFWLV98DO98h7PRTnB1gGU=', 'Content-Length': '0'} 2024-04-12T04:08:05.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000801038dad1ca8d8f-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.162 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.163 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.163 DEBUG:boto:path=/dkyges-61/?prefix=obj145 2024-04-12T04:08:05.163 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj145 2024-04-12T04:08:05.163 DEBUG:boto:Method: GET 2024-04-12T04:08:05.163 DEBUG:boto:Path: /dkyges-61/?prefix=obj145 2024-04-12T04:08:05.163 DEBUG:boto:Data: 2024-04-12T04:08:05.163 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.163 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.163 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.163 DEBUG:boto:Params: {} 2024-04-12T04:08:05.163 DEBUG:boto:Token: None 2024-04-12T04:08:05.164 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.164 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000530d1086b13cf629-006618b3a5-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.168 DEBUG:boto:b'dkyges-61obj1451000falseobj1452024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.168 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.168 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.169 DEBUG:boto:path=/dkyges-61/?prefix=obj145 2024-04-12T04:08:05.169 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj145 2024-04-12T04:08:05.169 DEBUG:boto:Method: GET 2024-04-12T04:08:05.169 DEBUG:boto:Path: /dkyges-61/?prefix=obj145 2024-04-12T04:08:05.169 DEBUG:boto:Data: 2024-04-12T04:08:05.169 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.169 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.169 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.169 DEBUG:boto:Params: {} 2024-04-12T04:08:05.169 DEBUG:boto:Token: None 2024-04-12T04:08:05.169 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.170 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a734c07f53c35a68-006618b3a5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.174 DEBUG:boto:b'dkyges-61obj1451000falseobj1452024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:08:05.174 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:08:05.174 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:08:05.174 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.174 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.174 DEBUG:boto:Method: GET 2024-04-12T04:08:05.175 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.175 DEBUG:boto:Data: 2024-04-12T04:08:05.175 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.175 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.175 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.175 DEBUG:boto:Params: {} 2024-04-12T04:08:05.175 DEBUG:boto:Token: None 2024-04-12T04:08:05.175 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.175 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MXulRFO83vtN7Y1j7zmAnnpIMz8= 2024-04-12T04:08:05.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MXulRFO83vtN7Y1j7zmAnnpIMz8=', 'Content-Length': '0'} 2024-04-12T04:08:05.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f91d75301587b7e-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.218 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:08:05.219 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:08:05.219 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.219 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.219 DEBUG:boto:Method: GET 2024-04-12T04:08:05.219 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.219 DEBUG:boto:Data: 2024-04-12T04:08:05.219 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.219 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.219 DEBUG:boto:Params: {} 2024-04-12T04:08:05.219 DEBUG:boto:Token: None 2024-04-12T04:08:05.220 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.220 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MXulRFO83vtN7Y1j7zmAnnpIMz8= 2024-04-12T04:08:05.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MXulRFO83vtN7Y1j7zmAnnpIMz8=', 'Content-Length': '0'} 2024-04-12T04:08:05.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000558e05f7681cbe65-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.223 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:08:05.223 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:08:05.223 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.223 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.223 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.223 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.223 DEBUG:boto:Data: 2024-04-12T04:08:05.223 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.223 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.223 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.223 DEBUG:boto:Params: {} 2024-04-12T04:08:05.224 DEBUG:boto:Token: None 2024-04-12T04:08:05.224 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.224 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+snDCjCdT3HGrbJdCrDo5R9CUu8= 2024-04-12T04:08:05.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+snDCjCdT3HGrbJdCrDo5R9CUu8=', 'Content-Length': '0'} 2024-04-12T04:08:05.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e57e5cf2801ab322-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.226 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:08:05.226 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:08:05.226 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.226 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.226 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.226 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.226 DEBUG:boto:Data: 2024-04-12T04:08:05.226 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.226 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.226 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.227 DEBUG:boto:Params: {} 2024-04-12T04:08:05.227 DEBUG:boto:Token: None 2024-04-12T04:08:05.227 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.227 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+snDCjCdT3HGrbJdCrDo5R9CUu8= 2024-04-12T04:08:05.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+snDCjCdT3HGrbJdCrDo5R9CUu8=', 'Content-Length': '0'} 2024-04-12T04:08:05.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000933ed664b7e20ac6-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:08:05.229 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:08:05.229 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:08:05.229 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.229 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.230 DEBUG:boto:Method: GET 2024-04-12T04:08:05.230 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.230 DEBUG:boto:Data: 2024-04-12T04:08:05.230 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.230 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.230 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.230 DEBUG:boto:Params: {} 2024-04-12T04:08:05.231 DEBUG:boto:Token: None 2024-04-12T04:08:05.231 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.231 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MXulRFO83vtN7Y1j7zmAnnpIMz8= 2024-04-12T04:08:05.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MXulRFO83vtN7Y1j7zmAnnpIMz8=', 'Content-Length': '0'} 2024-04-12T04:08:05.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032789da24f886f8a-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.233 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:08:05.233 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:08:05.233 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.233 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.233 DEBUG:boto:Method: GET 2024-04-12T04:08:05.233 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.233 DEBUG:boto:Data: 2024-04-12T04:08:05.233 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.233 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.234 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.234 DEBUG:boto:Params: {} 2024-04-12T04:08:05.234 DEBUG:boto:Token: None 2024-04-12T04:08:05.234 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:08:05.234 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MXulRFO83vtN7Y1j7zmAnnpIMz8= 2024-04-12T04:08:05.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MXulRFO83vtN7Y1j7zmAnnpIMz8=', 'Content-Length': '0'} 2024-04-12T04:08:05.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d4a8c77fd705f28-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.236 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.236 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.237 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:08:05.237 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:08:05.237 DEBUG:boto:Method: GET 2024-04-12T04:08:05.237 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:08:05.237 DEBUG:boto:Data: 2024-04-12T04:08:05.237 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.237 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.237 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.237 DEBUG:boto:Params: {} 2024-04-12T04:08:05.237 DEBUG:boto:Token: None 2024-04-12T04:08:05.237 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.238 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021e10d09f4ba474a-006618b3a5-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.241 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.241 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.242 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.242 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:08:05.242 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:08:05.242 DEBUG:boto:Method: GET 2024-04-12T04:08:05.242 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:08:05.242 DEBUG:boto:Data: 2024-04-12T04:08:05.242 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.242 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.242 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.242 DEBUG:boto:Params: {} 2024-04-12T04:08:05.243 DEBUG:boto:Token: None 2024-04-12T04:08:05.243 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.243 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3df4d9e6347a312-006618b3a5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.247 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.247 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:08:05.247 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:08:05.247 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.247 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:08:05.247 DEBUG:boto:Data: 2024-04-12T04:08:05.248 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.248 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.248 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.248 DEBUG:boto:Params: {} 2024-04-12T04:08:05.248 DEBUG:boto:Token: None 2024-04-12T04:08:05.248 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj17 2024-04-12T04:08:05.248 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Q7NP1A2QwA10swgS/AX1nj3+L+s= 2024-04-12T04:08:05.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Q7NP1A2QwA10swgS/AX1nj3+L+s=', 'Content-Length': '0'} 2024-04-12T04:08:05.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feefd8226cf1f91b-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.251 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:08:05.251 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:08:05.252 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.252 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:08:05.252 DEBUG:boto:Data: 2024-04-12T04:08:05.252 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.252 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.252 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.252 DEBUG:boto:Params: {} 2024-04-12T04:08:05.252 DEBUG:boto:Token: None 2024-04-12T04:08:05.252 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj17 2024-04-12T04:08:05.252 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Q7NP1A2QwA10swgS/AX1nj3+L+s= 2024-04-12T04:08:05.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Q7NP1A2QwA10swgS/AX1nj3+L+s=', 'Content-Length': '0'} 2024-04-12T04:08:05.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026d1b384077b1b45-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:08:05.255 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:08:05.255 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:08:05.255 DEBUG:boto:Method: GET 2024-04-12T04:08:05.255 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:08:05.255 DEBUG:boto:Data: 2024-04-12T04:08:05.255 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.255 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.255 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.255 DEBUG:boto:Params: {} 2024-04-12T04:08:05.255 DEBUG:boto:Token: None 2024-04-12T04:08:05.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj17 2024-04-12T04:08:05.256 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kCq9YtdoCvO6EGRbdXZlNJqaV68= 2024-04-12T04:08:05.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kCq9YtdoCvO6EGRbdXZlNJqaV68=', 'Content-Length': '0'} 2024-04-12T04:08:05.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d4a89fb7c28eb13-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.258 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:08:05.259 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:08:05.259 DEBUG:boto:Method: GET 2024-04-12T04:08:05.259 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:08:05.259 DEBUG:boto:Data: 2024-04-12T04:08:05.259 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.259 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.259 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.259 DEBUG:boto:Params: {} 2024-04-12T04:08:05.259 DEBUG:boto:Token: None 2024-04-12T04:08:05.259 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj17 2024-04-12T04:08:05.260 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kCq9YtdoCvO6EGRbdXZlNJqaV68= 2024-04-12T04:08:05.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kCq9YtdoCvO6EGRbdXZlNJqaV68=', 'Content-Length': '0'} 2024-04-12T04:08:05.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021ce5a55dde0832f-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.302 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.303 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.303 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:08:05.303 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:08:05.303 DEBUG:boto:Method: GET 2024-04-12T04:08:05.303 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:08:05.303 DEBUG:boto:Data: 2024-04-12T04:08:05.303 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.303 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.303 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.303 DEBUG:boto:Params: {} 2024-04-12T04:08:05.304 DEBUG:boto:Token: None 2024-04-12T04:08:05.304 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.304 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000981ee343c73c26e-006618b3a5-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.307 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.307 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.307 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.307 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:08:05.307 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:08:05.308 DEBUG:boto:Method: GET 2024-04-12T04:08:05.308 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:08:05.308 DEBUG:boto:Data: 2024-04-12T04:08:05.308 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.308 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.308 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.308 DEBUG:boto:Params: {} 2024-04-12T04:08:05.308 DEBUG:boto:Token: None 2024-04-12T04:08:05.308 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.308 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9fb6606e7e3defb-006618b3a5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.314 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:08:05.314 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:08:05.314 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:08:05.314 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.314 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.315 DEBUG:boto:Method: GET 2024-04-12T04:08:05.315 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.315 DEBUG:boto:Data: 2024-04-12T04:08:05.315 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.315 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.315 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.315 DEBUG:boto:Params: {} 2024-04-12T04:08:05.315 DEBUG:boto:Token: None 2024-04-12T04:08:05.315 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.315 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CX6Cx44gX2eZ87/Nsan/RX2tbck= 2024-04-12T04:08:05.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CX6Cx44gX2eZ87/Nsan/RX2tbck=', 'Content-Length': '0'} 2024-04-12T04:08:05.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6242961c7a43669-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.362 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:08:05.363 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:08:05.363 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.363 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.363 DEBUG:boto:Method: GET 2024-04-12T04:08:05.363 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.363 DEBUG:boto:Data: 2024-04-12T04:08:05.363 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.363 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.363 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.363 DEBUG:boto:Params: {} 2024-04-12T04:08:05.364 DEBUG:boto:Token: None 2024-04-12T04:08:05.364 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.364 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CX6Cx44gX2eZ87/Nsan/RX2tbck= 2024-04-12T04:08:05.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CX6Cx44gX2eZ87/Nsan/RX2tbck=', 'Content-Length': '0'} 2024-04-12T04:08:05.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084d0898e1aadc0d2-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.366 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:08:05.366 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:08:05.367 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.367 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.367 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.367 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.367 DEBUG:boto:Data: 2024-04-12T04:08:05.367 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.367 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.367 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.367 DEBUG:boto:Params: {} 2024-04-12T04:08:05.367 DEBUG:boto:Token: None 2024-04-12T04:08:05.368 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.368 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ihHrdVLJzKXH32E2iJyyPhnG4b0= 2024-04-12T04:08:05.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ihHrdVLJzKXH32E2iJyyPhnG4b0=', 'Content-Length': '0'} 2024-04-12T04:08:05.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f37b008e52dcedc-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.370 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:08:05.370 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:08:05.370 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.370 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.370 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.370 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.370 DEBUG:boto:Data: 2024-04-12T04:08:05.371 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.371 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.371 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.371 DEBUG:boto:Params: {} 2024-04-12T04:08:05.371 DEBUG:boto:Token: None 2024-04-12T04:08:05.371 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.371 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ihHrdVLJzKXH32E2iJyyPhnG4b0= 2024-04-12T04:08:05.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ihHrdVLJzKXH32E2iJyyPhnG4b0=', 'Content-Length': '0'} 2024-04-12T04:08:05.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0ece600de2fc1f5-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:08:05.373 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:08:05.373 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:08:05.373 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.373 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.374 DEBUG:boto:Method: GET 2024-04-12T04:08:05.374 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.374 DEBUG:boto:Data: 2024-04-12T04:08:05.374 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.374 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.374 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.374 DEBUG:boto:Params: {} 2024-04-12T04:08:05.374 DEBUG:boto:Token: None 2024-04-12T04:08:05.374 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.374 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CX6Cx44gX2eZ87/Nsan/RX2tbck= 2024-04-12T04:08:05.375 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CX6Cx44gX2eZ87/Nsan/RX2tbck=', 'Content-Length': '0'} 2024-04-12T04:08:05.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5a35a74f884bbe9-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.376 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:08:05.377 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:08:05.377 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.377 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.377 DEBUG:boto:Method: GET 2024-04-12T04:08:05.377 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.377 DEBUG:boto:Data: 2024-04-12T04:08:05.377 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.377 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.377 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.377 DEBUG:boto:Params: {} 2024-04-12T04:08:05.378 DEBUG:boto:Token: None 2024-04-12T04:08:05.378 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:08:05.378 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CX6Cx44gX2eZ87/Nsan/RX2tbck= 2024-04-12T04:08:05.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CX6Cx44gX2eZ87/Nsan/RX2tbck=', 'Content-Length': '0'} 2024-04-12T04:08:05.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc1d396d54d976e4-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.380 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.380 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.380 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:08:05.380 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:08:05.380 DEBUG:boto:Method: GET 2024-04-12T04:08:05.380 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:08:05.380 DEBUG:boto:Data: 2024-04-12T04:08:05.381 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.381 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.381 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.381 DEBUG:boto:Params: {} 2024-04-12T04:08:05.381 DEBUG:boto:Token: None 2024-04-12T04:08:05.381 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.381 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d858ecd40b08af6-006618b3a5-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.384 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.384 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.384 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.384 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:08:05.385 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:08:05.385 DEBUG:boto:Method: GET 2024-04-12T04:08:05.385 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:08:05.385 DEBUG:boto:Data: 2024-04-12T04:08:05.385 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.385 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.385 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.385 DEBUG:boto:Params: {} 2024-04-12T04:08:05.385 DEBUG:boto:Token: None 2024-04-12T04:08:05.385 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.386 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db279cc99eb221ad-006618b3a5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.389 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.390 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:08:05.390 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:08:05.390 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.390 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:08:05.390 DEBUG:boto:Data: 2024-04-12T04:08:05.390 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.390 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.390 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.390 DEBUG:boto:Params: {} 2024-04-12T04:08:05.391 DEBUG:boto:Token: None 2024-04-12T04:08:05.391 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj19 2024-04-12T04:08:05.391 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nJKthroDskRV9LCyESni8OBHuac= 2024-04-12T04:08:05.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nJKthroDskRV9LCyESni8OBHuac=', 'Content-Length': '0'} 2024-04-12T04:08:05.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddcbdbd9ee718f02-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.393 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:08:05.393 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:08:05.394 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.394 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:08:05.394 DEBUG:boto:Data: 2024-04-12T04:08:05.394 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.394 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.394 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.394 DEBUG:boto:Params: {} 2024-04-12T04:08:05.394 DEBUG:boto:Token: None 2024-04-12T04:08:05.394 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj19 2024-04-12T04:08:05.394 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nJKthroDskRV9LCyESni8OBHuac= 2024-04-12T04:08:05.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nJKthroDskRV9LCyESni8OBHuac=', 'Content-Length': '0'} 2024-04-12T04:08:05.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcf945d62ecec920-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:08:05.396 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:08:05.396 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:08:05.397 DEBUG:boto:Method: GET 2024-04-12T04:08:05.397 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:08:05.397 DEBUG:boto:Data: 2024-04-12T04:08:05.397 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.397 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.397 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.397 DEBUG:boto:Params: {} 2024-04-12T04:08:05.397 DEBUG:boto:Token: None 2024-04-12T04:08:05.397 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj19 2024-04-12T04:08:05.397 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wkpxfYdspvrQeOXHmb++TJN6QrM= 2024-04-12T04:08:05.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wkpxfYdspvrQeOXHmb++TJN6QrM=', 'Content-Length': '0'} 2024-04-12T04:08:05.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf1b8bd67688946e-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.400 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:08:05.400 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:08:05.400 DEBUG:boto:Method: GET 2024-04-12T04:08:05.400 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:08:05.400 DEBUG:boto:Data: 2024-04-12T04:08:05.400 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.400 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.400 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.400 DEBUG:boto:Params: {} 2024-04-12T04:08:05.400 DEBUG:boto:Token: None 2024-04-12T04:08:05.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj19 2024-04-12T04:08:05.401 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wkpxfYdspvrQeOXHmb++TJN6QrM= 2024-04-12T04:08:05.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wkpxfYdspvrQeOXHmb++TJN6QrM=', 'Content-Length': '0'} 2024-04-12T04:08:05.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed54f0dc11d1f92b-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.403 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.403 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.403 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:08:05.403 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:08:05.403 DEBUG:boto:Method: GET 2024-04-12T04:08:05.403 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:08:05.404 DEBUG:boto:Data: 2024-04-12T04:08:05.404 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.404 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.404 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.404 DEBUG:boto:Params: {} 2024-04-12T04:08:05.404 DEBUG:boto:Token: None 2024-04-12T04:08:05.404 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.404 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f82a565566817fe-006618b3a5-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.450 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.451 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.451 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.451 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:08:05.451 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:08:05.451 DEBUG:boto:Method: GET 2024-04-12T04:08:05.451 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:08:05.451 DEBUG:boto:Data: 2024-04-12T04:08:05.452 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.452 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.452 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.452 DEBUG:boto:Params: {} 2024-04-12T04:08:05.452 DEBUG:boto:Token: None 2024-04-12T04:08:05.452 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.452 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2fa8f1cd3acaeb1-006618b3a5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.456 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:08:05.457 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:08:05.457 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:08:05.457 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.457 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.457 DEBUG:boto:Method: GET 2024-04-12T04:08:05.457 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.458 DEBUG:boto:Data: 2024-04-12T04:08:05.458 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.458 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.458 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.458 DEBUG:boto:Params: {} 2024-04-12T04:08:05.458 DEBUG:boto:Token: None 2024-04-12T04:08:05.458 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.458 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vSvV4C75sMuhT1vbSwrrcPgm+JM= 2024-04-12T04:08:05.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vSvV4C75sMuhT1vbSwrrcPgm+JM=', 'Content-Length': '0'} 2024-04-12T04:08:05.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d3b7a3902be37c3-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.461 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:08:05.461 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:08:05.461 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.461 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.461 DEBUG:boto:Method: GET 2024-04-12T04:08:05.462 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.462 DEBUG:boto:Data: 2024-04-12T04:08:05.462 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.462 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.462 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.462 DEBUG:boto:Params: {} 2024-04-12T04:08:05.462 DEBUG:boto:Token: None 2024-04-12T04:08:05.462 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.462 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vSvV4C75sMuhT1vbSwrrcPgm+JM= 2024-04-12T04:08:05.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vSvV4C75sMuhT1vbSwrrcPgm+JM=', 'Content-Length': '0'} 2024-04-12T04:08:05.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000493b4bd502f33c48-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.510 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:08:05.511 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:08:05.511 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.511 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.511 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.511 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.511 DEBUG:boto:Data: 2024-04-12T04:08:05.511 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.511 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.511 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.511 DEBUG:boto:Params: {} 2024-04-12T04:08:05.512 DEBUG:boto:Token: None 2024-04-12T04:08:05.512 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.512 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jEPn484o9XsT+Q94pm1XvNx+r3U= 2024-04-12T04:08:05.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jEPn484o9XsT+Q94pm1XvNx+r3U=', 'Content-Length': '0'} 2024-04-12T04:08:05.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bb3b266763c6331-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.515 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:08:05.515 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:08:05.515 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.515 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.515 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.515 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.515 DEBUG:boto:Data: 2024-04-12T04:08:05.515 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.515 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.515 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.516 DEBUG:boto:Params: {} 2024-04-12T04:08:05.516 DEBUG:boto:Token: None 2024-04-12T04:08:05.516 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.516 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jEPn484o9XsT+Q94pm1XvNx+r3U= 2024-04-12T04:08:05.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jEPn484o9XsT+Q94pm1XvNx+r3U=', 'Content-Length': '0'} 2024-04-12T04:08:05.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4db82a07b4fa0c7-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:08:05.518 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:08:05.519 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:08:05.519 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.519 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.519 DEBUG:boto:Method: GET 2024-04-12T04:08:05.519 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.519 DEBUG:boto:Data: 2024-04-12T04:08:05.519 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.519 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.519 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.519 DEBUG:boto:Params: {} 2024-04-12T04:08:05.519 DEBUG:boto:Token: None 2024-04-12T04:08:05.520 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.520 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vSvV4C75sMuhT1vbSwrrcPgm+JM= 2024-04-12T04:08:05.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vSvV4C75sMuhT1vbSwrrcPgm+JM=', 'Content-Length': '0'} 2024-04-12T04:08:05.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000143408a5777786b6-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.522 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:08:05.522 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:08:05.523 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.523 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.523 DEBUG:boto:Method: GET 2024-04-12T04:08:05.523 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.523 DEBUG:boto:Data: 2024-04-12T04:08:05.523 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.524 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.524 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.524 DEBUG:boto:Params: {} 2024-04-12T04:08:05.524 DEBUG:boto:Token: None 2024-04-12T04:08:05.524 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:08:05.525 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vSvV4C75sMuhT1vbSwrrcPgm+JM= 2024-04-12T04:08:05.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vSvV4C75sMuhT1vbSwrrcPgm+JM=', 'Content-Length': '0'} 2024-04-12T04:08:05.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d107c84768e2f1-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.527 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.527 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.527 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:08:05.527 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:08:05.527 DEBUG:boto:Method: GET 2024-04-12T04:08:05.527 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:08:05.528 DEBUG:boto:Data: 2024-04-12T04:08:05.528 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.528 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.528 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.528 DEBUG:boto:Params: {} 2024-04-12T04:08:05.528 DEBUG:boto:Token: None 2024-04-12T04:08:05.528 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.528 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cad9ec51a6ef649e-006618b3a5-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.531 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.531 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.532 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.532 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:08:05.532 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:08:05.532 DEBUG:boto:Method: GET 2024-04-12T04:08:05.532 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:08:05.532 DEBUG:boto:Data: 2024-04-12T04:08:05.532 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.532 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.532 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.532 DEBUG:boto:Params: {} 2024-04-12T04:08:05.532 DEBUG:boto:Token: None 2024-04-12T04:08:05.533 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.533 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc23eddd051e4b2d-006618b3a5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.537 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.537 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:08:05.537 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:08:05.537 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.538 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:08:05.538 DEBUG:boto:Data: 2024-04-12T04:08:05.538 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.538 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.538 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.538 DEBUG:boto:Params: {} 2024-04-12T04:08:05.538 DEBUG:boto:Token: None 2024-04-12T04:08:05.538 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj22 2024-04-12T04:08:05.538 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZaXR287ZgwsHq4arHnG+xVCQlZQ= 2024-04-12T04:08:05.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZaXR287ZgwsHq4arHnG+xVCQlZQ=', 'Content-Length': '0'} 2024-04-12T04:08:05.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000052aa1894efc138c-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.541 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:08:05.541 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:08:05.541 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.541 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:08:05.541 DEBUG:boto:Data: 2024-04-12T04:08:05.541 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.541 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.541 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.541 DEBUG:boto:Params: {} 2024-04-12T04:08:05.541 DEBUG:boto:Token: None 2024-04-12T04:08:05.541 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj22 2024-04-12T04:08:05.542 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZaXR287ZgwsHq4arHnG+xVCQlZQ= 2024-04-12T04:08:05.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZaXR287ZgwsHq4arHnG+xVCQlZQ=', 'Content-Length': '0'} 2024-04-12T04:08:05.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054cae0da53b62c3b-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:08:05.544 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:08:05.544 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:08:05.544 DEBUG:boto:Method: GET 2024-04-12T04:08:05.544 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:08:05.544 DEBUG:boto:Data: 2024-04-12T04:08:05.544 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.544 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.544 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.544 DEBUG:boto:Params: {} 2024-04-12T04:08:05.545 DEBUG:boto:Token: None 2024-04-12T04:08:05.545 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj22 2024-04-12T04:08:05.545 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aqYsyls1tah60+PN55vJR3Y6L+o= 2024-04-12T04:08:05.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aqYsyls1tah60+PN55vJR3Y6L+o=', 'Content-Length': '0'} 2024-04-12T04:08:05.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ee479cdb860f71-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.547 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:08:05.547 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:08:05.547 DEBUG:boto:Method: GET 2024-04-12T04:08:05.547 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:08:05.547 DEBUG:boto:Data: 2024-04-12T04:08:05.548 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.548 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.548 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.548 DEBUG:boto:Params: {} 2024-04-12T04:08:05.548 DEBUG:boto:Token: None 2024-04-12T04:08:05.548 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj22 2024-04-12T04:08:05.548 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aqYsyls1tah60+PN55vJR3Y6L+o= 2024-04-12T04:08:05.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aqYsyls1tah60+PN55vJR3Y6L+o=', 'Content-Length': '0'} 2024-04-12T04:08:05.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebe24001f62b4aab-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.590 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.591 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.591 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:08:05.591 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:08:05.591 DEBUG:boto:Method: GET 2024-04-12T04:08:05.591 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:08:05.591 DEBUG:boto:Data: 2024-04-12T04:08:05.591 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.591 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.591 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.592 DEBUG:boto:Params: {} 2024-04-12T04:08:05.592 DEBUG:boto:Token: None 2024-04-12T04:08:05.592 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.592 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000840b9f2c9dd21b59-006618b3a5-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.595 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.595 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.595 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.596 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:08:05.596 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:08:05.596 DEBUG:boto:Method: GET 2024-04-12T04:08:05.596 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:08:05.596 DEBUG:boto:Data: 2024-04-12T04:08:05.596 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.596 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.596 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.596 DEBUG:boto:Params: {} 2024-04-12T04:08:05.596 DEBUG:boto:Token: None 2024-04-12T04:08:05.597 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.597 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b4b776461e93dc9-006618b3a5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.601 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:08:05.601 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:08:05.601 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:08:05.602 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.602 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.602 DEBUG:boto:Method: GET 2024-04-12T04:08:05.602 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.602 DEBUG:boto:Data: 2024-04-12T04:08:05.602 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.602 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.602 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.602 DEBUG:boto:Params: {} 2024-04-12T04:08:05.602 DEBUG:boto:Token: None 2024-04-12T04:08:05.603 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.603 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tE2xHVORMoXwZMUS7XxQqb374Jk= 2024-04-12T04:08:05.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tE2xHVORMoXwZMUS7XxQqb374Jk=', 'Content-Length': '0'} 2024-04-12T04:08:05.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdc405d129a6b2e6-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.650 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:08:05.651 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:08:05.651 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.651 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.651 DEBUG:boto:Method: GET 2024-04-12T04:08:05.651 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.651 DEBUG:boto:Data: 2024-04-12T04:08:05.651 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.651 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.651 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.651 DEBUG:boto:Params: {} 2024-04-12T04:08:05.652 DEBUG:boto:Token: None 2024-04-12T04:08:05.652 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.652 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tE2xHVORMoXwZMUS7XxQqb374Jk= 2024-04-12T04:08:05.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tE2xHVORMoXwZMUS7XxQqb374Jk=', 'Content-Length': '0'} 2024-04-12T04:08:05.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd24baac18275ca1-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.654 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:08:05.655 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:08:05.655 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.655 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.655 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.655 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.655 DEBUG:boto:Data: 2024-04-12T04:08:05.655 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.655 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.655 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.655 DEBUG:boto:Params: {} 2024-04-12T04:08:05.655 DEBUG:boto:Token: None 2024-04-12T04:08:05.656 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ICrbrP/riGzxdY25t0mbg69L/C8= 2024-04-12T04:08:05.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ICrbrP/riGzxdY25t0mbg69L/C8=', 'Content-Length': '0'} 2024-04-12T04:08:05.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e723fe313b7e4c63-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.658 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:08:05.658 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:08:05.658 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.658 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.658 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.658 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.658 DEBUG:boto:Data: 2024-04-12T04:08:05.658 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.659 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.659 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.659 DEBUG:boto:Params: {} 2024-04-12T04:08:05.659 DEBUG:boto:Token: None 2024-04-12T04:08:05.659 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.659 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ICrbrP/riGzxdY25t0mbg69L/C8= 2024-04-12T04:08:05.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ICrbrP/riGzxdY25t0mbg69L/C8=', 'Content-Length': '0'} 2024-04-12T04:08:05.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f110eb41d74a907c-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:08:05.661 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:08:05.661 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:08:05.662 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.662 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.662 DEBUG:boto:Method: GET 2024-04-12T04:08:05.662 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.662 DEBUG:boto:Data: 2024-04-12T04:08:05.662 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.662 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.662 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.662 DEBUG:boto:Params: {} 2024-04-12T04:08:05.662 DEBUG:boto:Token: None 2024-04-12T04:08:05.663 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.663 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tE2xHVORMoXwZMUS7XxQqb374Jk= 2024-04-12T04:08:05.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tE2xHVORMoXwZMUS7XxQqb374Jk=', 'Content-Length': '0'} 2024-04-12T04:08:05.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000663f74d80726d1b-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.665 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:08:05.665 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:08:05.665 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.665 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.665 DEBUG:boto:Method: GET 2024-04-12T04:08:05.665 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.665 DEBUG:boto:Data: 2024-04-12T04:08:05.665 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.665 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.666 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.666 DEBUG:boto:Params: {} 2024-04-12T04:08:05.666 DEBUG:boto:Token: None 2024-04-12T04:08:05.666 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:08:05.666 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tE2xHVORMoXwZMUS7XxQqb374Jk= 2024-04-12T04:08:05.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tE2xHVORMoXwZMUS7XxQqb374Jk=', 'Content-Length': '0'} 2024-04-12T04:08:05.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019c0b3ca0db4b274-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.668 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.668 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.668 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:08:05.669 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:08:05.669 DEBUG:boto:Method: GET 2024-04-12T04:08:05.669 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:08:05.669 DEBUG:boto:Data: 2024-04-12T04:08:05.669 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.669 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.669 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.669 DEBUG:boto:Params: {} 2024-04-12T04:08:05.669 DEBUG:boto:Token: None 2024-04-12T04:08:05.669 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.669 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d150d7f75fe57b3c-006618b3a5-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.672 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.672 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.673 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.673 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:08:05.673 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:08:05.673 DEBUG:boto:Method: GET 2024-04-12T04:08:05.673 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:08:05.673 DEBUG:boto:Data: 2024-04-12T04:08:05.673 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.673 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.673 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.673 DEBUG:boto:Params: {} 2024-04-12T04:08:05.674 DEBUG:boto:Token: None 2024-04-12T04:08:05.674 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.674 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fac28cae0a56678-006618b3a5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.677 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.678 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:08:05.678 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:08:05.678 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.678 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:08:05.678 DEBUG:boto:Data: 2024-04-12T04:08:05.678 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.678 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.679 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.679 DEBUG:boto:Params: {} 2024-04-12T04:08:05.679 DEBUG:boto:Token: None 2024-04-12T04:08:05.679 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj23 2024-04-12T04:08:05.679 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e7tbVCBtm6ZZUYtpFfoOYD7tAIw= 2024-04-12T04:08:05.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e7tbVCBtm6ZZUYtpFfoOYD7tAIw=', 'Content-Length': '0'} 2024-04-12T04:08:05.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5299c482902cbd3-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.681 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:08:05.681 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:08:05.681 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.681 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:08:05.681 DEBUG:boto:Data: 2024-04-12T04:08:05.681 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.682 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.682 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.682 DEBUG:boto:Params: {} 2024-04-12T04:08:05.682 DEBUG:boto:Token: None 2024-04-12T04:08:05.682 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj23 2024-04-12T04:08:05.682 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e7tbVCBtm6ZZUYtpFfoOYD7tAIw= 2024-04-12T04:08:05.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e7tbVCBtm6ZZUYtpFfoOYD7tAIw=', 'Content-Length': '0'} 2024-04-12T04:08:05.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b042b251948a020b-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:08:05.684 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:08:05.685 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:08:05.685 DEBUG:boto:Method: GET 2024-04-12T04:08:05.685 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:08:05.685 DEBUG:boto:Data: 2024-04-12T04:08:05.685 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.685 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.685 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.685 DEBUG:boto:Params: {} 2024-04-12T04:08:05.685 DEBUG:boto:Token: None 2024-04-12T04:08:05.685 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj23 2024-04-12T04:08:05.686 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XDrbyqD9YLaDDvG80tppZsiOGRw= 2024-04-12T04:08:05.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XDrbyqD9YLaDDvG80tppZsiOGRw=', 'Content-Length': '0'} 2024-04-12T04:08:05.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab5b9f8a6b22ba7e-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.688 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:08:05.688 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:08:05.688 DEBUG:boto:Method: GET 2024-04-12T04:08:05.688 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:08:05.688 DEBUG:boto:Data: 2024-04-12T04:08:05.688 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.689 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.689 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.689 DEBUG:boto:Params: {} 2024-04-12T04:08:05.689 DEBUG:boto:Token: None 2024-04-12T04:08:05.689 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj23 2024-04-12T04:08:05.689 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XDrbyqD9YLaDDvG80tppZsiOGRw= 2024-04-12T04:08:05.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XDrbyqD9YLaDDvG80tppZsiOGRw=', 'Content-Length': '0'} 2024-04-12T04:08:05.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f00b72ce3f1261e7-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.691 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.691 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.692 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:08:05.692 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:08:05.692 DEBUG:boto:Method: GET 2024-04-12T04:08:05.692 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:08:05.692 DEBUG:boto:Data: 2024-04-12T04:08:05.692 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.692 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.692 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.692 DEBUG:boto:Params: {} 2024-04-12T04:08:05.692 DEBUG:boto:Token: None 2024-04-12T04:08:05.692 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.693 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e174d46e747d6b30-006618b3a5-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.738 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.739 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.739 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.739 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:08:05.739 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:08:05.739 DEBUG:boto:Method: GET 2024-04-12T04:08:05.739 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:08:05.739 DEBUG:boto:Data: 2024-04-12T04:08:05.740 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.740 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.740 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.740 DEBUG:boto:Params: {} 2024-04-12T04:08:05.740 DEBUG:boto:Token: None 2024-04-12T04:08:05.740 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.740 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a66f134ef30871c-006618b3a5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.745 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:08:05.745 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:08:05.745 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:08:05.746 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.746 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.746 DEBUG:boto:Method: GET 2024-04-12T04:08:05.746 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.746 DEBUG:boto:Data: 2024-04-12T04:08:05.746 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.746 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.746 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.746 DEBUG:boto:Params: {} 2024-04-12T04:08:05.746 DEBUG:boto:Token: None 2024-04-12T04:08:05.747 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.747 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OlZthD8xIS87wAekWvM8W4/5kDk= 2024-04-12T04:08:05.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OlZthD8xIS87wAekWvM8W4/5kDk=', 'Content-Length': '0'} 2024-04-12T04:08:05.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a13647c2606bb6a0-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.749 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:08:05.749 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:08:05.749 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.749 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.749 DEBUG:boto:Method: GET 2024-04-12T04:08:05.749 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.750 DEBUG:boto:Data: 2024-04-12T04:08:05.750 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.750 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.750 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.750 DEBUG:boto:Params: {} 2024-04-12T04:08:05.750 DEBUG:boto:Token: None 2024-04-12T04:08:05.750 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.750 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OlZthD8xIS87wAekWvM8W4/5kDk= 2024-04-12T04:08:05.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OlZthD8xIS87wAekWvM8W4/5kDk=', 'Content-Length': '0'} 2024-04-12T04:08:05.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004574e642659fa2ea-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.794 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:08:05.795 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:08:05.795 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.795 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.795 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.795 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.795 DEBUG:boto:Data: 2024-04-12T04:08:05.795 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.795 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.795 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.795 DEBUG:boto:Params: {} 2024-04-12T04:08:05.796 DEBUG:boto:Token: None 2024-04-12T04:08:05.796 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.796 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hEEmmRtVBskCnxepcQIVZsXqYZs= 2024-04-12T04:08:05.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hEEmmRtVBskCnxepcQIVZsXqYZs=', 'Content-Length': '0'} 2024-04-12T04:08:05.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001be7ffa248e06abf-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.798 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:08:05.798 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:08:05.798 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.798 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.798 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.799 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.799 DEBUG:boto:Data: 2024-04-12T04:08:05.799 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.799 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.799 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.799 DEBUG:boto:Params: {} 2024-04-12T04:08:05.799 DEBUG:boto:Token: None 2024-04-12T04:08:05.799 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.799 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hEEmmRtVBskCnxepcQIVZsXqYZs= 2024-04-12T04:08:05.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hEEmmRtVBskCnxepcQIVZsXqYZs=', 'Content-Length': '0'} 2024-04-12T04:08:05.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a07fc16dc4f6278-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:08:05.801 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:08:05.802 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:08:05.802 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.802 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.802 DEBUG:boto:Method: GET 2024-04-12T04:08:05.802 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.802 DEBUG:boto:Data: 2024-04-12T04:08:05.802 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.802 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.802 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.802 DEBUG:boto:Params: {} 2024-04-12T04:08:05.802 DEBUG:boto:Token: None 2024-04-12T04:08:05.803 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.803 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OlZthD8xIS87wAekWvM8W4/5kDk= 2024-04-12T04:08:05.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OlZthD8xIS87wAekWvM8W4/5kDk=', 'Content-Length': '0'} 2024-04-12T04:08:05.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078101c2e7315b456-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.805 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:08:05.805 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:08:05.805 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.805 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.805 DEBUG:boto:Method: GET 2024-04-12T04:08:05.805 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.805 DEBUG:boto:Data: 2024-04-12T04:08:05.806 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.806 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.806 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.806 DEBUG:boto:Params: {} 2024-04-12T04:08:05.806 DEBUG:boto:Token: None 2024-04-12T04:08:05.806 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:08:05.806 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OlZthD8xIS87wAekWvM8W4/5kDk= 2024-04-12T04:08:05.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OlZthD8xIS87wAekWvM8W4/5kDk=', 'Content-Length': '0'} 2024-04-12T04:08:05.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000658a7bdcb76b53a2-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.809 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.809 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.809 DEBUG:boto:path=/dkyges-61/?prefix=obj29 2024-04-12T04:08:05.809 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj29 2024-04-12T04:08:05.809 DEBUG:boto:Method: GET 2024-04-12T04:08:05.809 DEBUG:boto:Path: /dkyges-61/?prefix=obj29 2024-04-12T04:08:05.810 DEBUG:boto:Data: 2024-04-12T04:08:05.810 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.810 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.810 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.810 DEBUG:boto:Params: {} 2024-04-12T04:08:05.810 DEBUG:boto:Token: None 2024-04-12T04:08:05.810 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.810 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b97368edaa256e1-006618b3a5-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.813 DEBUG:boto:b'dkyges-61obj291000falseobj292024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.813 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.813 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.814 DEBUG:boto:path=/dkyges-61/?prefix=obj29 2024-04-12T04:08:05.814 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj29 2024-04-12T04:08:05.814 DEBUG:boto:Method: GET 2024-04-12T04:08:05.814 DEBUG:boto:Path: /dkyges-61/?prefix=obj29 2024-04-12T04:08:05.814 DEBUG:boto:Data: 2024-04-12T04:08:05.814 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.814 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.814 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.814 DEBUG:boto:Params: {} 2024-04-12T04:08:05.814 DEBUG:boto:Token: None 2024-04-12T04:08:05.814 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.815 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa38e6385243effa-006618b3a5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.819 DEBUG:boto:b'dkyges-61obj291000falseobj292024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.819 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:08:05.819 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:08:05.819 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.819 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:08:05.819 DEBUG:boto:Data: 2024-04-12T04:08:05.820 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.820 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.820 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.820 DEBUG:boto:Params: {} 2024-04-12T04:08:05.820 DEBUG:boto:Token: None 2024-04-12T04:08:05.820 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj29 2024-04-12T04:08:05.820 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2pPreCRlDzUS/gJin7WKg7/TLR0= 2024-04-12T04:08:05.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2pPreCRlDzUS/gJin7WKg7/TLR0=', 'Content-Length': '0'} 2024-04-12T04:08:05.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042a5e7fea9ad9606-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.822 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:08:05.822 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:08:05.822 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.823 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:08:05.823 DEBUG:boto:Data: 2024-04-12T04:08:05.823 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.823 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.823 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.823 DEBUG:boto:Params: {} 2024-04-12T04:08:05.823 DEBUG:boto:Token: None 2024-04-12T04:08:05.823 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj29 2024-04-12T04:08:05.823 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2pPreCRlDzUS/gJin7WKg7/TLR0= 2024-04-12T04:08:05.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2pPreCRlDzUS/gJin7WKg7/TLR0=', 'Content-Length': '0'} 2024-04-12T04:08:05.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1f7c172199df447-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:08:05.825 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:08:05.826 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:08:05.826 DEBUG:boto:Method: GET 2024-04-12T04:08:05.826 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:08:05.826 DEBUG:boto:Data: 2024-04-12T04:08:05.826 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.826 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.826 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.826 DEBUG:boto:Params: {} 2024-04-12T04:08:05.826 DEBUG:boto:Token: None 2024-04-12T04:08:05.826 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj29 2024-04-12T04:08:05.827 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Xzt03zNj9VgZm5Id8lST0v6YRI4= 2024-04-12T04:08:05.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Xzt03zNj9VgZm5Id8lST0v6YRI4=', 'Content-Length': '0'} 2024-04-12T04:08:05.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fa39aa49d6de745-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.829 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:08:05.829 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:08:05.829 DEBUG:boto:Method: GET 2024-04-12T04:08:05.829 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:08:05.829 DEBUG:boto:Data: 2024-04-12T04:08:05.829 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.829 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.830 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.830 DEBUG:boto:Params: {} 2024-04-12T04:08:05.830 DEBUG:boto:Token: None 2024-04-12T04:08:05.830 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj29 2024-04-12T04:08:05.830 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Xzt03zNj9VgZm5Id8lST0v6YRI4= 2024-04-12T04:08:05.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Xzt03zNj9VgZm5Id8lST0v6YRI4=', 'Content-Length': '0'} 2024-04-12T04:08:05.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bbb3c0713d7f13c-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.832 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.832 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.832 DEBUG:boto:path=/dkyges-61/?prefix=obj29 2024-04-12T04:08:05.832 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj29 2024-04-12T04:08:05.832 DEBUG:boto:Method: GET 2024-04-12T04:08:05.832 DEBUG:boto:Path: /dkyges-61/?prefix=obj29 2024-04-12T04:08:05.832 DEBUG:boto:Data: 2024-04-12T04:08:05.833 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.833 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.833 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.833 DEBUG:boto:Params: {} 2024-04-12T04:08:05.833 DEBUG:boto:Token: None 2024-04-12T04:08:05.833 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.833 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3fefcdd031dfb4b-006618b3a5-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.836 DEBUG:boto:b'dkyges-61obj291000falseobj292024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.836 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.836 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.836 DEBUG:boto:path=/dkyges-61/?prefix=obj29 2024-04-12T04:08:05.836 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj29 2024-04-12T04:08:05.837 DEBUG:boto:Method: GET 2024-04-12T04:08:05.837 DEBUG:boto:Path: /dkyges-61/?prefix=obj29 2024-04-12T04:08:05.837 DEBUG:boto:Data: 2024-04-12T04:08:05.837 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.837 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.837 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.837 DEBUG:boto:Params: {} 2024-04-12T04:08:05.837 DEBUG:boto:Token: None 2024-04-12T04:08:05.837 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.837 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae5d9610cadad3f5-006618b3a5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.842 DEBUG:boto:b'dkyges-61obj291000falseobj292024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:08:05.842 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:08:05.842 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:08:05.842 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.842 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.842 DEBUG:boto:Method: GET 2024-04-12T04:08:05.843 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.843 DEBUG:boto:Data: 2024-04-12T04:08:05.843 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.843 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.843 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.843 DEBUG:boto:Params: {} 2024-04-12T04:08:05.843 DEBUG:boto:Token: None 2024-04-12T04:08:05.846 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.846 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eJzL9C9bDm4q1HeAjiVBg8AMA+8= 2024-04-12T04:08:05.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eJzL9C9bDm4q1HeAjiVBg8AMA+8=', 'Content-Length': '0'} 2024-04-12T04:08:05.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eab7fd4bffe6aaf-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.848 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:08:05.848 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:08:05.848 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.848 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.848 DEBUG:boto:Method: GET 2024-04-12T04:08:05.848 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.848 DEBUG:boto:Data: 2024-04-12T04:08:05.849 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.849 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.849 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.849 DEBUG:boto:Params: {} 2024-04-12T04:08:05.849 DEBUG:boto:Token: None 2024-04-12T04:08:05.849 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.849 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eJzL9C9bDm4q1HeAjiVBg8AMA+8= 2024-04-12T04:08:05.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eJzL9C9bDm4q1HeAjiVBg8AMA+8=', 'Content-Length': '0'} 2024-04-12T04:08:05.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009339f1ae356e8a80-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.898 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:08:05.899 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:08:05.899 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.899 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.899 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.899 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.899 DEBUG:boto:Data: 2024-04-12T04:08:05.899 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.899 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.899 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.899 DEBUG:boto:Params: {} 2024-04-12T04:08:05.900 DEBUG:boto:Token: None 2024-04-12T04:08:05.900 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.900 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B31U9N3ANgCQ6gX8+1PFfiCFR+w= 2024-04-12T04:08:05.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B31U9N3ANgCQ6gX8+1PFfiCFR+w=', 'Content-Length': '0'} 2024-04-12T04:08:05.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004558ee408da12ddf-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.902 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:08:05.902 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:08:05.902 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.902 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.902 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.903 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.903 DEBUG:boto:Data: 2024-04-12T04:08:05.903 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.903 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.903 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.903 DEBUG:boto:Params: {} 2024-04-12T04:08:05.903 DEBUG:boto:Token: None 2024-04-12T04:08:05.903 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.903 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B31U9N3ANgCQ6gX8+1PFfiCFR+w= 2024-04-12T04:08:05.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B31U9N3ANgCQ6gX8+1PFfiCFR+w=', 'Content-Length': '0'} 2024-04-12T04:08:05.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0486ff2263c87ac-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:08:05.906 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:08:05.906 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:08:05.906 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.906 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.906 DEBUG:boto:Method: GET 2024-04-12T04:08:05.906 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.906 DEBUG:boto:Data: 2024-04-12T04:08:05.906 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.906 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.907 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.907 DEBUG:boto:Params: {} 2024-04-12T04:08:05.907 DEBUG:boto:Token: None 2024-04-12T04:08:05.907 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.907 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eJzL9C9bDm4q1HeAjiVBg8AMA+8= 2024-04-12T04:08:05.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eJzL9C9bDm4q1HeAjiVBg8AMA+8=', 'Content-Length': '0'} 2024-04-12T04:08:05.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f99b1dce071ad9b6-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.909 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:08:05.909 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:08:05.910 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.910 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.910 DEBUG:boto:Method: GET 2024-04-12T04:08:05.910 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.910 DEBUG:boto:Data: 2024-04-12T04:08:05.910 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.910 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.910 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.910 DEBUG:boto:Params: {} 2024-04-12T04:08:05.910 DEBUG:boto:Token: None 2024-04-12T04:08:05.911 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:08:05.911 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eJzL9C9bDm4q1HeAjiVBg8AMA+8= 2024-04-12T04:08:05.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eJzL9C9bDm4q1HeAjiVBg8AMA+8=', 'Content-Length': '0'} 2024-04-12T04:08:05.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daed043d11b1e1ae-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.913 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.913 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.913 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:08:05.913 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:08:05.913 DEBUG:boto:Method: GET 2024-04-12T04:08:05.913 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:08:05.913 DEBUG:boto:Data: 2024-04-12T04:08:05.913 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.914 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.914 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.914 DEBUG:boto:Params: {} 2024-04-12T04:08:05.914 DEBUG:boto:Token: None 2024-04-12T04:08:05.914 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.914 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000326694d00fce2b1e-006618b3a5-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.917 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.917 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.917 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.917 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:08:05.918 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:08:05.918 DEBUG:boto:Method: GET 2024-04-12T04:08:05.918 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:08:05.918 DEBUG:boto:Data: 2024-04-12T04:08:05.918 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.918 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.918 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.918 DEBUG:boto:Params: {} 2024-04-12T04:08:05.918 DEBUG:boto:Token: None 2024-04-12T04:08:05.918 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.919 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008de1c1188913b7a8-006618b3a5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.922 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.923 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:08:05.923 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:08:05.923 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.923 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:08:05.923 DEBUG:boto:Data: 2024-04-12T04:08:05.923 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.923 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.923 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.923 DEBUG:boto:Params: {} 2024-04-12T04:08:05.924 DEBUG:boto:Token: None 2024-04-12T04:08:05.924 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj33 2024-04-12T04:08:05.924 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QAU7xAqUKE7z7dMTf7TbT03/19E= 2024-04-12T04:08:05.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QAU7xAqUKE7z7dMTf7TbT03/19E=', 'Content-Length': '0'} 2024-04-12T04:08:05.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e4cf464034f27d1-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.926 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:08:05.926 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:08:05.926 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.926 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:08:05.926 DEBUG:boto:Data: 2024-04-12T04:08:05.927 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.927 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.927 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.927 DEBUG:boto:Params: {} 2024-04-12T04:08:05.927 DEBUG:boto:Token: None 2024-04-12T04:08:05.927 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj33 2024-04-12T04:08:05.927 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QAU7xAqUKE7z7dMTf7TbT03/19E= 2024-04-12T04:08:05.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QAU7xAqUKE7z7dMTf7TbT03/19E=', 'Content-Length': '0'} 2024-04-12T04:08:05.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1758e5173d19b46-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:08:05.929 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:08:05.929 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:08:05.930 DEBUG:boto:Method: GET 2024-04-12T04:08:05.930 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:08:05.930 DEBUG:boto:Data: 2024-04-12T04:08:05.930 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.930 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.930 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.930 DEBUG:boto:Params: {} 2024-04-12T04:08:05.930 DEBUG:boto:Token: None 2024-04-12T04:08:05.930 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj33 2024-04-12T04:08:05.931 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ix6DN0GzgvSDI18cgjns8VcGC0A= 2024-04-12T04:08:05.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ix6DN0GzgvSDI18cgjns8VcGC0A=', 'Content-Length': '0'} 2024-04-12T04:08:05.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f7e71994eafcb6b-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.932 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:08:05.933 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:08:05.933 DEBUG:boto:Method: GET 2024-04-12T04:08:05.933 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:08:05.933 DEBUG:boto:Data: 2024-04-12T04:08:05.933 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.933 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.933 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.933 DEBUG:boto:Params: {} 2024-04-12T04:08:05.933 DEBUG:boto:Token: None 2024-04-12T04:08:05.933 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj33 2024-04-12T04:08:05.934 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ix6DN0GzgvSDI18cgjns8VcGC0A= 2024-04-12T04:08:05.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ix6DN0GzgvSDI18cgjns8VcGC0A=', 'Content-Length': '0'} 2024-04-12T04:08:05.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000592d7baa8d6f60da-006618b3a5-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.936 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.936 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.936 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:08:05.936 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:08:05.936 DEBUG:boto:Method: GET 2024-04-12T04:08:05.936 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:08:05.936 DEBUG:boto:Data: 2024-04-12T04:08:05.936 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.937 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.937 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.937 DEBUG:boto:Params: {} 2024-04-12T04:08:05.937 DEBUG:boto:Token: None 2024-04-12T04:08:05.937 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.937 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e11db1854473f13-006618b3a5-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.940 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.940 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:05.940 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:05.940 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:08:05.940 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:08:05.940 DEBUG:boto:Method: GET 2024-04-12T04:08:05.940 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:08:05.941 DEBUG:boto:Data: 2024-04-12T04:08:05.941 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.941 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.941 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.941 DEBUG:boto:Params: {} 2024-04-12T04:08:05.941 DEBUG:boto:Token: None 2024-04-12T04:08:05.941 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/ 2024-04-12T04:08:05.941 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM= 2024-04-12T04:08:05.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jVtstKw8KPEN6b2zH5XoTLdTqWM=', 'Content-Length': '0'} 2024-04-12T04:08:05.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1ec861d1702f2c9-006618b3a5-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.946 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:05.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:08:05.946 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:08:05.946 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:08:05.946 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:08:05.946 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:08:05.946 DEBUG:boto:Method: GET 2024-04-12T04:08:05.946 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:08:05.947 DEBUG:boto:Data: 2024-04-12T04:08:05.947 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.947 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.947 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.947 DEBUG:boto:Params: {} 2024-04-12T04:08:05.947 DEBUG:boto:Token: None 2024-04-12T04:08:05.947 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:08:05.947 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XiVqSLVUNWryCf1qMp2xafmAdJM= 2024-04-12T04:08:05.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XiVqSLVUNWryCf1qMp2xafmAdJM=', 'Content-Length': '0'} 2024-04-12T04:08:05.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed73164087ab35d7-006618b3a5-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.990 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:08:05.990 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:08:05.991 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:08:05.991 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:08:05.991 DEBUG:boto:Method: GET 2024-04-12T04:08:05.991 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:08:05.991 DEBUG:boto:Data: 2024-04-12T04:08:05.991 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.991 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.991 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.991 DEBUG:boto:Params: {} 2024-04-12T04:08:05.991 DEBUG:boto:Token: None 2024-04-12T04:08:05.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:08:05.992 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XiVqSLVUNWryCf1qMp2xafmAdJM= 2024-04-12T04:08:05.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XiVqSLVUNWryCf1qMp2xafmAdJM=', 'Content-Length': '0'} 2024-04-12T04:08:05.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bc87404fb52735e-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.994 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:08:05.995 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:08:05.995 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:08:05.995 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:08:05.995 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.995 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:08:05.995 DEBUG:boto:Data: 2024-04-12T04:08:05.995 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.995 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:05.995 DEBUG:boto:Port: 8001 2024-04-12T04:08:05.995 DEBUG:boto:Params: {} 2024-04-12T04:08:05.996 DEBUG:boto:Token: None 2024-04-12T04:08:05.996 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:08:05.996 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gLS9/lTfoagG8j8/5DV7lzP6kwY= 2024-04-12T04:08:05.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gLS9/lTfoagG8j8/5DV7lzP6kwY=', 'Content-Length': '0'} 2024-04-12T04:08:05.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e944fd51bba69c4-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:05.998 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:08:05.998 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:08:05.998 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:08:05.998 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:08:05.999 DEBUG:boto:Method: HEAD 2024-04-12T04:08:05.999 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:08:05.999 DEBUG:boto:Data: 2024-04-12T04:08:05.999 DEBUG:boto:Headers: {} 2024-04-12T04:08:05.999 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:05.999 DEBUG:boto:Port: 8000 2024-04-12T04:08:05.999 DEBUG:boto:Params: {} 2024-04-12T04:08:05.999 DEBUG:boto:Token: None 2024-04-12T04:08:05.999 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:05 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:08:05.999 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gLS9/lTfoagG8j8/5DV7lzP6kwY= 2024-04-12T04:08:05.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gLS9/lTfoagG8j8/5DV7lzP6kwY=', 'Content-Length': '0'} 2024-04-12T04:08:06.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a2900e7ab526e8d-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:08:06.002 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:08:06.002 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:08:06.002 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:08:06.002 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:08:06.002 DEBUG:boto:Method: GET 2024-04-12T04:08:06.002 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:08:06.002 DEBUG:boto:Data: 2024-04-12T04:08:06.002 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.002 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.002 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.003 DEBUG:boto:Params: {} 2024-04-12T04:08:06.003 DEBUG:boto:Token: None 2024-04-12T04:08:06.003 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:08:06.003 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I41v34sfo8WkPumIuq+B1rvolh8= 2024-04-12T04:08:06.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I41v34sfo8WkPumIuq+B1rvolh8=', 'Content-Length': '0'} 2024-04-12T04:08:06.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc6090055da04719-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.005 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:08:06.005 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:08:06.006 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:08:06.006 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:08:06.006 DEBUG:boto:Method: GET 2024-04-12T04:08:06.006 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:08:06.006 DEBUG:boto:Data: 2024-04-12T04:08:06.006 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.006 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.006 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.006 DEBUG:boto:Params: {} 2024-04-12T04:08:06.006 DEBUG:boto:Token: None 2024-04-12T04:08:06.007 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:08:06.007 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I41v34sfo8WkPumIuq+B1rvolh8= 2024-04-12T04:08:06.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I41v34sfo8WkPumIuq+B1rvolh8=', 'Content-Length': '0'} 2024-04-12T04:08:06.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034f42000baf27ce7-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.050 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.050 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.051 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:08:06.051 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:08:06.051 DEBUG:boto:Method: GET 2024-04-12T04:08:06.051 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:08:06.051 DEBUG:boto:Data: 2024-04-12T04:08:06.051 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.051 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.051 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.051 DEBUG:boto:Params: {} 2024-04-12T04:08:06.051 DEBUG:boto:Token: None 2024-04-12T04:08:06.051 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.052 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b82adc65ab2eb195-006618b3a6-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.054 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.055 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.055 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.055 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:08:06.055 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:08:06.055 DEBUG:boto:Method: GET 2024-04-12T04:08:06.055 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:08:06.056 DEBUG:boto:Data: 2024-04-12T04:08:06.056 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.056 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.056 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.056 DEBUG:boto:Params: {} 2024-04-12T04:08:06.056 DEBUG:boto:Token: None 2024-04-12T04:08:06.056 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.057 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055252a3e215d4dda-006618b3a6-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.061 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.061 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:08:06.062 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:08:06.062 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.062 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:08:06.062 DEBUG:boto:Data: 2024-04-12T04:08:06.062 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.062 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.062 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.062 DEBUG:boto:Params: {} 2024-04-12T04:08:06.062 DEBUG:boto:Token: None 2024-04-12T04:08:06.062 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj34 2024-04-12T04:08:06.063 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:J4D9c77Hpn27vdLMHqEJqjonwbs= 2024-04-12T04:08:06.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:J4D9c77Hpn27vdLMHqEJqjonwbs=', 'Content-Length': '0'} 2024-04-12T04:08:06.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004924fbc311886ae-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.064 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:08:06.065 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:08:06.065 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.065 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:08:06.065 DEBUG:boto:Data: 2024-04-12T04:08:06.065 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.065 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.065 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.065 DEBUG:boto:Params: {} 2024-04-12T04:08:06.065 DEBUG:boto:Token: None 2024-04-12T04:08:06.066 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj34 2024-04-12T04:08:06.066 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:J4D9c77Hpn27vdLMHqEJqjonwbs= 2024-04-12T04:08:06.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:J4D9c77Hpn27vdLMHqEJqjonwbs=', 'Content-Length': '0'} 2024-04-12T04:08:06.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcb9ab2d5047368c-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:08:06.068 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:08:06.068 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:08:06.068 DEBUG:boto:Method: GET 2024-04-12T04:08:06.068 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:08:06.068 DEBUG:boto:Data: 2024-04-12T04:08:06.068 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.069 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.069 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.069 DEBUG:boto:Params: {} 2024-04-12T04:08:06.069 DEBUG:boto:Token: None 2024-04-12T04:08:06.069 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj34 2024-04-12T04:08:06.069 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b6REUtwVrlQZmBCI62nOi+YDjJc= 2024-04-12T04:08:06.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b6REUtwVrlQZmBCI62nOi+YDjJc=', 'Content-Length': '0'} 2024-04-12T04:08:06.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031c84a67e1f9706c-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.071 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:08:06.071 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:08:06.072 DEBUG:boto:Method: GET 2024-04-12T04:08:06.072 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:08:06.072 DEBUG:boto:Data: 2024-04-12T04:08:06.072 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.072 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.072 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.072 DEBUG:boto:Params: {} 2024-04-12T04:08:06.072 DEBUG:boto:Token: None 2024-04-12T04:08:06.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj34 2024-04-12T04:08:06.072 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b6REUtwVrlQZmBCI62nOi+YDjJc= 2024-04-12T04:08:06.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b6REUtwVrlQZmBCI62nOi+YDjJc=', 'Content-Length': '0'} 2024-04-12T04:08:06.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063f80e92d52adca1-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.075 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.075 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.075 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:08:06.075 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:08:06.075 DEBUG:boto:Method: GET 2024-04-12T04:08:06.075 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:08:06.075 DEBUG:boto:Data: 2024-04-12T04:08:06.075 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.075 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.075 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.075 DEBUG:boto:Params: {} 2024-04-12T04:08:06.076 DEBUG:boto:Token: None 2024-04-12T04:08:06.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.076 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3c2cf2ac655fb12-006618b3a6-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.078 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.079 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.079 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.079 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:08:06.079 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:08:06.079 DEBUG:boto:Method: GET 2024-04-12T04:08:06.079 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:08:06.079 DEBUG:boto:Data: 2024-04-12T04:08:06.079 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.079 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.080 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.080 DEBUG:boto:Params: {} 2024-04-12T04:08:06.080 DEBUG:boto:Token: None 2024-04-12T04:08:06.080 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.080 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099e1fd80200e7208-006618b3a6-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.126 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:08:06.127 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:08:06.127 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:08:06.127 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.127 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.127 DEBUG:boto:Method: GET 2024-04-12T04:08:06.127 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.127 DEBUG:boto:Data: 2024-04-12T04:08:06.128 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.128 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.128 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.128 DEBUG:boto:Params: {} 2024-04-12T04:08:06.128 DEBUG:boto:Token: None 2024-04-12T04:08:06.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.128 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rKKf7cYW/ftfqGmU7PLG/PiGdwU= 2024-04-12T04:08:06.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rKKf7cYW/ftfqGmU7PLG/PiGdwU=', 'Content-Length': '0'} 2024-04-12T04:08:06.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f217b9fd1f0c0497-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.131 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:08:06.132 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:08:06.132 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.132 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.132 DEBUG:boto:Method: GET 2024-04-12T04:08:06.132 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.133 DEBUG:boto:Data: 2024-04-12T04:08:06.133 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.133 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.133 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.133 DEBUG:boto:Params: {} 2024-04-12T04:08:06.133 DEBUG:boto:Token: None 2024-04-12T04:08:06.133 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.133 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rKKf7cYW/ftfqGmU7PLG/PiGdwU= 2024-04-12T04:08:06.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rKKf7cYW/ftfqGmU7PLG/PiGdwU=', 'Content-Length': '0'} 2024-04-12T04:08:06.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000195a99a530d075ff-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.136 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:08:06.136 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:08:06.136 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.136 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.136 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.137 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.137 DEBUG:boto:Data: 2024-04-12T04:08:06.137 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.137 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.137 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.137 DEBUG:boto:Params: {} 2024-04-12T04:08:06.137 DEBUG:boto:Token: None 2024-04-12T04:08:06.137 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.137 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LLr3Z2Q8DmauEb5H3nYX7d0eGE8= 2024-04-12T04:08:06.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LLr3Z2Q8DmauEb5H3nYX7d0eGE8=', 'Content-Length': '0'} 2024-04-12T04:08:06.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c621ff8cab9b5ae-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.139 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:08:06.139 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:08:06.139 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.140 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.140 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.140 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.140 DEBUG:boto:Data: 2024-04-12T04:08:06.140 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.140 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.140 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.140 DEBUG:boto:Params: {} 2024-04-12T04:08:06.140 DEBUG:boto:Token: None 2024-04-12T04:08:06.140 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.140 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LLr3Z2Q8DmauEb5H3nYX7d0eGE8= 2024-04-12T04:08:06.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LLr3Z2Q8DmauEb5H3nYX7d0eGE8=', 'Content-Length': '0'} 2024-04-12T04:08:06.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000434327e74ed6095c-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:08:06.142 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:08:06.143 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:08:06.143 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.143 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.143 DEBUG:boto:Method: GET 2024-04-12T04:08:06.143 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.143 DEBUG:boto:Data: 2024-04-12T04:08:06.143 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.143 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.143 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.143 DEBUG:boto:Params: {} 2024-04-12T04:08:06.143 DEBUG:boto:Token: None 2024-04-12T04:08:06.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.144 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rKKf7cYW/ftfqGmU7PLG/PiGdwU= 2024-04-12T04:08:06.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rKKf7cYW/ftfqGmU7PLG/PiGdwU=', 'Content-Length': '0'} 2024-04-12T04:08:06.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000140962ed9856775a-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.186 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:08:06.186 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:08:06.186 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.187 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.187 DEBUG:boto:Method: GET 2024-04-12T04:08:06.187 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.187 DEBUG:boto:Data: 2024-04-12T04:08:06.187 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.187 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.187 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.187 DEBUG:boto:Params: {} 2024-04-12T04:08:06.187 DEBUG:boto:Token: None 2024-04-12T04:08:06.187 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:08:06.187 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rKKf7cYW/ftfqGmU7PLG/PiGdwU= 2024-04-12T04:08:06.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rKKf7cYW/ftfqGmU7PLG/PiGdwU=', 'Content-Length': '0'} 2024-04-12T04:08:06.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a6eea2fe5e7a53b-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.190 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.190 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.190 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:08:06.190 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:08:06.190 DEBUG:boto:Method: GET 2024-04-12T04:08:06.190 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:08:06.190 DEBUG:boto:Data: 2024-04-12T04:08:06.190 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.191 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.191 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.191 DEBUG:boto:Params: {} 2024-04-12T04:08:06.191 DEBUG:boto:Token: None 2024-04-12T04:08:06.191 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.191 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028e548fe26a1c0aa-006618b3a6-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.194 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.194 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.194 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.194 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:08:06.194 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:08:06.194 DEBUG:boto:Method: GET 2024-04-12T04:08:06.195 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:08:06.195 DEBUG:boto:Data: 2024-04-12T04:08:06.195 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.195 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.195 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.195 DEBUG:boto:Params: {} 2024-04-12T04:08:06.195 DEBUG:boto:Token: None 2024-04-12T04:08:06.195 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.195 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000440a556f74b60d1e-006618b3a6-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.200 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.200 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:08:06.200 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:08:06.200 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.200 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:08:06.200 DEBUG:boto:Data: 2024-04-12T04:08:06.200 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.200 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.201 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.201 DEBUG:boto:Params: {} 2024-04-12T04:08:06.201 DEBUG:boto:Token: None 2024-04-12T04:08:06.201 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj38 2024-04-12T04:08:06.201 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EpOmeHd2nbEuojQgbOYM231dG10= 2024-04-12T04:08:06.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EpOmeHd2nbEuojQgbOYM231dG10=', 'Content-Length': '0'} 2024-04-12T04:08:06.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb1a51ac8d6ab510-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.203 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:08:06.203 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:08:06.203 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.203 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:08:06.203 DEBUG:boto:Data: 2024-04-12T04:08:06.203 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.203 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.203 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.204 DEBUG:boto:Params: {} 2024-04-12T04:08:06.204 DEBUG:boto:Token: None 2024-04-12T04:08:06.204 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj38 2024-04-12T04:08:06.204 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EpOmeHd2nbEuojQgbOYM231dG10= 2024-04-12T04:08:06.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EpOmeHd2nbEuojQgbOYM231dG10=', 'Content-Length': '0'} 2024-04-12T04:08:06.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089204905a8b54449-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:08:06.206 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:08:06.206 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:08:06.207 DEBUG:boto:Method: GET 2024-04-12T04:08:06.207 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:08:06.207 DEBUG:boto:Data: 2024-04-12T04:08:06.207 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.207 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.207 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.207 DEBUG:boto:Params: {} 2024-04-12T04:08:06.207 DEBUG:boto:Token: None 2024-04-12T04:08:06.207 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj38 2024-04-12T04:08:06.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Uz78qS2V0rm0P+KHmRMMWrmqkWk= 2024-04-12T04:08:06.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Uz78qS2V0rm0P+KHmRMMWrmqkWk=', 'Content-Length': '0'} 2024-04-12T04:08:06.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032d2c0ea09a43911-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.209 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:08:06.210 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:08:06.210 DEBUG:boto:Method: GET 2024-04-12T04:08:06.210 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:08:06.210 DEBUG:boto:Data: 2024-04-12T04:08:06.210 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.210 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.210 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.210 DEBUG:boto:Params: {} 2024-04-12T04:08:06.211 DEBUG:boto:Token: None 2024-04-12T04:08:06.211 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj38 2024-04-12T04:08:06.211 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Uz78qS2V0rm0P+KHmRMMWrmqkWk= 2024-04-12T04:08:06.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Uz78qS2V0rm0P+KHmRMMWrmqkWk=', 'Content-Length': '0'} 2024-04-12T04:08:06.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e42a918b3ccbef5a-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.213 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.213 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.213 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:08:06.213 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:08:06.213 DEBUG:boto:Method: GET 2024-04-12T04:08:06.213 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:08:06.214 DEBUG:boto:Data: 2024-04-12T04:08:06.214 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.214 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.214 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.214 DEBUG:boto:Params: {} 2024-04-12T04:08:06.214 DEBUG:boto:Token: None 2024-04-12T04:08:06.215 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.215 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a44f3153052470b8-006618b3a6-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.218 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.218 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.219 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.219 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:08:06.219 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:08:06.219 DEBUG:boto:Method: GET 2024-04-12T04:08:06.219 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:08:06.219 DEBUG:boto:Data: 2024-04-12T04:08:06.219 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.219 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.219 DEBUG:boto:Params: {} 2024-04-12T04:08:06.219 DEBUG:boto:Token: None 2024-04-12T04:08:06.220 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.220 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e76704649b1def5-006618b3a6-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.224 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:08:06.224 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:08:06.224 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:08:06.224 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.224 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.224 DEBUG:boto:Method: GET 2024-04-12T04:08:06.224 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.225 DEBUG:boto:Data: 2024-04-12T04:08:06.225 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.225 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.225 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.225 DEBUG:boto:Params: {} 2024-04-12T04:08:06.225 DEBUG:boto:Token: None 2024-04-12T04:08:06.225 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.225 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D2sHfyJQmSc+E8nJLo4c7lTsKrk= 2024-04-12T04:08:06.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D2sHfyJQmSc+E8nJLo4c7lTsKrk=', 'Content-Length': '0'} 2024-04-12T04:08:06.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac894de4228eee12-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.228 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:08:06.228 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:08:06.228 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.228 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.228 DEBUG:boto:Method: GET 2024-04-12T04:08:06.228 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.228 DEBUG:boto:Data: 2024-04-12T04:08:06.228 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.228 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.228 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.229 DEBUG:boto:Params: {} 2024-04-12T04:08:06.229 DEBUG:boto:Token: None 2024-04-12T04:08:06.229 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.229 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D2sHfyJQmSc+E8nJLo4c7lTsKrk= 2024-04-12T04:08:06.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D2sHfyJQmSc+E8nJLo4c7lTsKrk=', 'Content-Length': '0'} 2024-04-12T04:08:06.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dc84c225add72bc-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.278 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:08:06.279 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:08:06.279 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.279 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.279 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.280 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.280 DEBUG:boto:Data: 2024-04-12T04:08:06.280 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.280 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.280 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.280 DEBUG:boto:Params: {} 2024-04-12T04:08:06.280 DEBUG:boto:Token: None 2024-04-12T04:08:06.280 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.281 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FxyVm/llChwCt0j/m6f040fLzsE= 2024-04-12T04:08:06.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FxyVm/llChwCt0j/m6f040fLzsE=', 'Content-Length': '0'} 2024-04-12T04:08:06.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a8472eb91197013-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.283 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:08:06.283 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:08:06.283 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.283 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.283 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.283 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.283 DEBUG:boto:Data: 2024-04-12T04:08:06.284 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.284 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.284 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.284 DEBUG:boto:Params: {} 2024-04-12T04:08:06.284 DEBUG:boto:Token: None 2024-04-12T04:08:06.284 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.284 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FxyVm/llChwCt0j/m6f040fLzsE= 2024-04-12T04:08:06.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FxyVm/llChwCt0j/m6f040fLzsE=', 'Content-Length': '0'} 2024-04-12T04:08:06.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec9c745074c2959b-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:08:06.287 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:08:06.287 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:08:06.287 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.287 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.287 DEBUG:boto:Method: GET 2024-04-12T04:08:06.287 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.287 DEBUG:boto:Data: 2024-04-12T04:08:06.287 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.287 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.288 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.288 DEBUG:boto:Params: {} 2024-04-12T04:08:06.288 DEBUG:boto:Token: None 2024-04-12T04:08:06.288 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.288 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D2sHfyJQmSc+E8nJLo4c7lTsKrk= 2024-04-12T04:08:06.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D2sHfyJQmSc+E8nJLo4c7lTsKrk=', 'Content-Length': '0'} 2024-04-12T04:08:06.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038f6db1e497cb00b-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.334 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:08:06.335 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:08:06.335 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.335 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.335 DEBUG:boto:Method: GET 2024-04-12T04:08:06.335 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.335 DEBUG:boto:Data: 2024-04-12T04:08:06.335 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.335 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.335 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.335 DEBUG:boto:Params: {} 2024-04-12T04:08:06.336 DEBUG:boto:Token: None 2024-04-12T04:08:06.336 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:08:06.336 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D2sHfyJQmSc+E8nJLo4c7lTsKrk= 2024-04-12T04:08:06.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D2sHfyJQmSc+E8nJLo4c7lTsKrk=', 'Content-Length': '0'} 2024-04-12T04:08:06.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000652a974109b7abcd-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.339 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.339 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.339 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:08:06.339 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:08:06.339 DEBUG:boto:Method: GET 2024-04-12T04:08:06.339 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:08:06.339 DEBUG:boto:Data: 2024-04-12T04:08:06.339 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.339 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.340 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.340 DEBUG:boto:Params: {} 2024-04-12T04:08:06.340 DEBUG:boto:Token: None 2024-04-12T04:08:06.340 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.340 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.343 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000760f2c2563fbb4ae-006618b3a6-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.343 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.343 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.343 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.343 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:08:06.344 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:08:06.344 DEBUG:boto:Method: GET 2024-04-12T04:08:06.344 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:08:06.344 DEBUG:boto:Data: 2024-04-12T04:08:06.344 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.344 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.344 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.344 DEBUG:boto:Params: {} 2024-04-12T04:08:06.344 DEBUG:boto:Token: None 2024-04-12T04:08:06.344 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.345 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ae2fb530c38ce47-006618b3a6-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.349 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.349 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:08:06.349 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:08:06.349 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.349 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:08:06.349 DEBUG:boto:Data: 2024-04-12T04:08:06.349 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.350 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.350 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.350 DEBUG:boto:Params: {} 2024-04-12T04:08:06.350 DEBUG:boto:Token: None 2024-04-12T04:08:06.350 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj44 2024-04-12T04:08:06.350 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cyEEtLfHN2kfGWJ0UkEXxx0OyEY= 2024-04-12T04:08:06.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cyEEtLfHN2kfGWJ0UkEXxx0OyEY=', 'Content-Length': '0'} 2024-04-12T04:08:06.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008081c25d8bfa56e2-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.352 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:08:06.352 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:08:06.352 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.352 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:08:06.352 DEBUG:boto:Data: 2024-04-12T04:08:06.352 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.353 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.353 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.353 DEBUG:boto:Params: {} 2024-04-12T04:08:06.353 DEBUG:boto:Token: None 2024-04-12T04:08:06.353 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj44 2024-04-12T04:08:06.353 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cyEEtLfHN2kfGWJ0UkEXxx0OyEY= 2024-04-12T04:08:06.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cyEEtLfHN2kfGWJ0UkEXxx0OyEY=', 'Content-Length': '0'} 2024-04-12T04:08:06.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000787d6ec003f00755-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:08:06.355 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:08:06.355 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:08:06.355 DEBUG:boto:Method: GET 2024-04-12T04:08:06.355 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:08:06.356 DEBUG:boto:Data: 2024-04-12T04:08:06.356 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.356 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.356 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.356 DEBUG:boto:Params: {} 2024-04-12T04:08:06.356 DEBUG:boto:Token: None 2024-04-12T04:08:06.356 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj44 2024-04-12T04:08:06.356 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuFT3cK2KgReXzzAhyICrHQY72w= 2024-04-12T04:08:06.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuFT3cK2KgReXzzAhyICrHQY72w=', 'Content-Length': '0'} 2024-04-12T04:08:06.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070623e98c2701ebf-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.359 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:08:06.359 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:08:06.359 DEBUG:boto:Method: GET 2024-04-12T04:08:06.359 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:08:06.359 DEBUG:boto:Data: 2024-04-12T04:08:06.359 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.360 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.360 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.360 DEBUG:boto:Params: {} 2024-04-12T04:08:06.360 DEBUG:boto:Token: None 2024-04-12T04:08:06.360 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj44 2024-04-12T04:08:06.360 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QuFT3cK2KgReXzzAhyICrHQY72w= 2024-04-12T04:08:06.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QuFT3cK2KgReXzzAhyICrHQY72w=', 'Content-Length': '0'} 2024-04-12T04:08:06.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011bbd90c405e76f4-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.406 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.407 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.407 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:08:06.407 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:08:06.407 DEBUG:boto:Method: GET 2024-04-12T04:08:06.407 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:08:06.407 DEBUG:boto:Data: 2024-04-12T04:08:06.407 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.407 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.407 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.407 DEBUG:boto:Params: {} 2024-04-12T04:08:06.408 DEBUG:boto:Token: None 2024-04-12T04:08:06.408 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.408 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e789811a4ad35f25-006618b3a6-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.411 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.411 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.411 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.411 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:08:06.411 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:08:06.411 DEBUG:boto:Method: GET 2024-04-12T04:08:06.411 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:08:06.411 DEBUG:boto:Data: 2024-04-12T04:08:06.412 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.412 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.412 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.412 DEBUG:boto:Params: {} 2024-04-12T04:08:06.412 DEBUG:boto:Token: None 2024-04-12T04:08:06.412 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.412 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c539990f7bff59e-006618b3a6-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.416 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:08:06.417 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:08:06.417 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:08:06.417 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.417 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.417 DEBUG:boto:Method: GET 2024-04-12T04:08:06.417 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.417 DEBUG:boto:Data: 2024-04-12T04:08:06.417 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.417 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.417 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.418 DEBUG:boto:Params: {} 2024-04-12T04:08:06.418 DEBUG:boto:Token: None 2024-04-12T04:08:06.418 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.418 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OMvWZTbZ4bZWpzNgOsAk+ROdWpo= 2024-04-12T04:08:06.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OMvWZTbZ4bZWpzNgOsAk+ROdWpo=', 'Content-Length': '0'} 2024-04-12T04:08:06.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099e895ab01d42dd7-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.462 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:08:06.462 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:08:06.463 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.463 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.463 DEBUG:boto:Method: GET 2024-04-12T04:08:06.463 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.463 DEBUG:boto:Data: 2024-04-12T04:08:06.463 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.463 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.463 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.463 DEBUG:boto:Params: {} 2024-04-12T04:08:06.463 DEBUG:boto:Token: None 2024-04-12T04:08:06.464 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.464 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OMvWZTbZ4bZWpzNgOsAk+ROdWpo= 2024-04-12T04:08:06.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OMvWZTbZ4bZWpzNgOsAk+ROdWpo=', 'Content-Length': '0'} 2024-04-12T04:08:06.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000821d1e9703146da7-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.466 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:08:06.466 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:08:06.466 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.466 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.467 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.467 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.467 DEBUG:boto:Data: 2024-04-12T04:08:06.467 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.467 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.467 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.467 DEBUG:boto:Params: {} 2024-04-12T04:08:06.467 DEBUG:boto:Token: None 2024-04-12T04:08:06.467 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.468 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RlybWHDdCvSnZh+bSkK4QL0OxJs= 2024-04-12T04:08:06.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RlybWHDdCvSnZh+bSkK4QL0OxJs=', 'Content-Length': '0'} 2024-04-12T04:08:06.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033eb4670e70ee837-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.470 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:08:06.470 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:08:06.470 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.470 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.470 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.470 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.470 DEBUG:boto:Data: 2024-04-12T04:08:06.470 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.471 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.471 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.471 DEBUG:boto:Params: {} 2024-04-12T04:08:06.471 DEBUG:boto:Token: None 2024-04-12T04:08:06.471 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.471 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RlybWHDdCvSnZh+bSkK4QL0OxJs= 2024-04-12T04:08:06.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RlybWHDdCvSnZh+bSkK4QL0OxJs=', 'Content-Length': '0'} 2024-04-12T04:08:06.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002804ae387681f22d-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:08:06.473 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:08:06.474 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:08:06.474 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.474 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.474 DEBUG:boto:Method: GET 2024-04-12T04:08:06.474 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.474 DEBUG:boto:Data: 2024-04-12T04:08:06.474 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.474 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.474 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.474 DEBUG:boto:Params: {} 2024-04-12T04:08:06.475 DEBUG:boto:Token: None 2024-04-12T04:08:06.475 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.475 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OMvWZTbZ4bZWpzNgOsAk+ROdWpo= 2024-04-12T04:08:06.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OMvWZTbZ4bZWpzNgOsAk+ROdWpo=', 'Content-Length': '0'} 2024-04-12T04:08:06.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f09099aa887273ad-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.477 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:08:06.477 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:08:06.477 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.477 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.477 DEBUG:boto:Method: GET 2024-04-12T04:08:06.477 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.477 DEBUG:boto:Data: 2024-04-12T04:08:06.478 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.478 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.478 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.478 DEBUG:boto:Params: {} 2024-04-12T04:08:06.478 DEBUG:boto:Token: None 2024-04-12T04:08:06.478 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:08:06.478 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OMvWZTbZ4bZWpzNgOsAk+ROdWpo= 2024-04-12T04:08:06.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OMvWZTbZ4bZWpzNgOsAk+ROdWpo=', 'Content-Length': '0'} 2024-04-12T04:08:06.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e449fc45d992397-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.481 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.481 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.481 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:08:06.481 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:08:06.481 DEBUG:boto:Method: GET 2024-04-12T04:08:06.481 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:08:06.481 DEBUG:boto:Data: 2024-04-12T04:08:06.481 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.481 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.481 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.481 DEBUG:boto:Params: {} 2024-04-12T04:08:06.482 DEBUG:boto:Token: None 2024-04-12T04:08:06.482 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.482 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3f9ec7e4a458bce-006618b3a6-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.484 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.485 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.485 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.485 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:08:06.485 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:08:06.485 DEBUG:boto:Method: GET 2024-04-12T04:08:06.485 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:08:06.485 DEBUG:boto:Data: 2024-04-12T04:08:06.485 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.486 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.486 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.486 DEBUG:boto:Params: {} 2024-04-12T04:08:06.486 DEBUG:boto:Token: None 2024-04-12T04:08:06.486 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.486 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055b90a82f7df7164-006618b3a6-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.490 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.491 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:08:06.491 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:08:06.491 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.491 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:08:06.491 DEBUG:boto:Data: 2024-04-12T04:08:06.491 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.491 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.491 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.492 DEBUG:boto:Params: {} 2024-04-12T04:08:06.492 DEBUG:boto:Token: None 2024-04-12T04:08:06.492 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj46 2024-04-12T04:08:06.492 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5cT9ycDhteawzN4FBPalMx5ZsMg= 2024-04-12T04:08:06.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5cT9ycDhteawzN4FBPalMx5ZsMg=', 'Content-Length': '0'} 2024-04-12T04:08:06.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ae3d32afaebc150-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.494 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:08:06.494 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:08:06.494 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.494 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:08:06.494 DEBUG:boto:Data: 2024-04-12T04:08:06.494 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.494 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.495 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.495 DEBUG:boto:Params: {} 2024-04-12T04:08:06.495 DEBUG:boto:Token: None 2024-04-12T04:08:06.495 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj46 2024-04-12T04:08:06.495 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5cT9ycDhteawzN4FBPalMx5ZsMg= 2024-04-12T04:08:06.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5cT9ycDhteawzN4FBPalMx5ZsMg=', 'Content-Length': '0'} 2024-04-12T04:08:06.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a95b30f21e6ab6db-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:08:06.497 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:08:06.497 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:08:06.497 DEBUG:boto:Method: GET 2024-04-12T04:08:06.498 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:08:06.498 DEBUG:boto:Data: 2024-04-12T04:08:06.498 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.498 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.498 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.498 DEBUG:boto:Params: {} 2024-04-12T04:08:06.498 DEBUG:boto:Token: None 2024-04-12T04:08:06.498 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj46 2024-04-12T04:08:06.498 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0ozdCCXNd87E8iDBab3ZipTepQQ= 2024-04-12T04:08:06.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0ozdCCXNd87E8iDBab3ZipTepQQ=', 'Content-Length': '0'} 2024-04-12T04:08:06.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df055838b3b420ba-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.501 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:08:06.501 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:08:06.501 DEBUG:boto:Method: GET 2024-04-12T04:08:06.501 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:08:06.501 DEBUG:boto:Data: 2024-04-12T04:08:06.501 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.501 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.501 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.502 DEBUG:boto:Params: {} 2024-04-12T04:08:06.502 DEBUG:boto:Token: None 2024-04-12T04:08:06.502 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj46 2024-04-12T04:08:06.502 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0ozdCCXNd87E8iDBab3ZipTepQQ= 2024-04-12T04:08:06.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0ozdCCXNd87E8iDBab3ZipTepQQ=', 'Content-Length': '0'} 2024-04-12T04:08:06.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006626e1006c61fcab-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.504 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.504 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.504 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:08:06.505 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:08:06.505 DEBUG:boto:Method: GET 2024-04-12T04:08:06.505 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:08:06.505 DEBUG:boto:Data: 2024-04-12T04:08:06.505 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.505 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.505 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.505 DEBUG:boto:Params: {} 2024-04-12T04:08:06.505 DEBUG:boto:Token: None 2024-04-12T04:08:06.505 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.506 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c78eba4bb899d981-006618b3a6-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.508 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.508 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.508 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.509 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:08:06.509 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:08:06.509 DEBUG:boto:Method: GET 2024-04-12T04:08:06.509 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:08:06.509 DEBUG:boto:Data: 2024-04-12T04:08:06.509 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.509 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.509 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.509 DEBUG:boto:Params: {} 2024-04-12T04:08:06.509 DEBUG:boto:Token: None 2024-04-12T04:08:06.509 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.510 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088ebdf49c7737fbb-006618b3a6-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.514 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:08:06.514 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:08:06.514 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:08:06.514 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.514 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.514 DEBUG:boto:Method: GET 2024-04-12T04:08:06.515 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.515 DEBUG:boto:Data: 2024-04-12T04:08:06.515 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.515 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.515 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.515 DEBUG:boto:Params: {} 2024-04-12T04:08:06.515 DEBUG:boto:Token: None 2024-04-12T04:08:06.515 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.515 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gxrrVtoAHNAnouZCnfwbjcGLn4M= 2024-04-12T04:08:06.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gxrrVtoAHNAnouZCnfwbjcGLn4M=', 'Content-Length': '0'} 2024-04-12T04:08:06.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d9b8a76c6d2626c-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.517 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:08:06.517 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:08:06.518 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.518 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.518 DEBUG:boto:Method: GET 2024-04-12T04:08:06.518 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.518 DEBUG:boto:Data: 2024-04-12T04:08:06.518 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.518 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.518 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.518 DEBUG:boto:Params: {} 2024-04-12T04:08:06.518 DEBUG:boto:Token: None 2024-04-12T04:08:06.519 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.519 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gxrrVtoAHNAnouZCnfwbjcGLn4M= 2024-04-12T04:08:06.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gxrrVtoAHNAnouZCnfwbjcGLn4M=', 'Content-Length': '0'} 2024-04-12T04:08:06.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf1a6ec644566a17-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.521 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:08:06.521 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:08:06.521 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.521 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.521 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.521 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.521 DEBUG:boto:Data: 2024-04-12T04:08:06.522 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.522 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.522 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.522 DEBUG:boto:Params: {} 2024-04-12T04:08:06.522 DEBUG:boto:Token: None 2024-04-12T04:08:06.522 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.522 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fGJcEXDjdCkiPgtkVBNSVuZa2MU= 2024-04-12T04:08:06.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fGJcEXDjdCkiPgtkVBNSVuZa2MU=', 'Content-Length': '0'} 2024-04-12T04:08:06.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f07c24650aa9a22-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.524 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:08:06.524 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:08:06.524 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.524 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.524 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.524 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.525 DEBUG:boto:Data: 2024-04-12T04:08:06.525 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.525 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.525 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.525 DEBUG:boto:Params: {} 2024-04-12T04:08:06.525 DEBUG:boto:Token: None 2024-04-12T04:08:06.525 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.525 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fGJcEXDjdCkiPgtkVBNSVuZa2MU= 2024-04-12T04:08:06.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fGJcEXDjdCkiPgtkVBNSVuZa2MU=', 'Content-Length': '0'} 2024-04-12T04:08:06.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000200761b689234667-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:08:06.527 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:08:06.527 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:08:06.528 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.528 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.528 DEBUG:boto:Method: GET 2024-04-12T04:08:06.528 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.528 DEBUG:boto:Data: 2024-04-12T04:08:06.528 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.528 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.528 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.528 DEBUG:boto:Params: {} 2024-04-12T04:08:06.529 DEBUG:boto:Token: None 2024-04-12T04:08:06.529 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.529 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gxrrVtoAHNAnouZCnfwbjcGLn4M= 2024-04-12T04:08:06.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gxrrVtoAHNAnouZCnfwbjcGLn4M=', 'Content-Length': '0'} 2024-04-12T04:08:06.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dadf6d4d9987f38b-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.531 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:08:06.531 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:08:06.531 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.531 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.531 DEBUG:boto:Method: GET 2024-04-12T04:08:06.531 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.531 DEBUG:boto:Data: 2024-04-12T04:08:06.531 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.532 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.532 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.532 DEBUG:boto:Params: {} 2024-04-12T04:08:06.532 DEBUG:boto:Token: None 2024-04-12T04:08:06.532 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:08:06.532 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gxrrVtoAHNAnouZCnfwbjcGLn4M= 2024-04-12T04:08:06.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gxrrVtoAHNAnouZCnfwbjcGLn4M=', 'Content-Length': '0'} 2024-04-12T04:08:06.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aebf3de65ffa6ab-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.578 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.578 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.579 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:08:06.579 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:08:06.579 DEBUG:boto:Method: GET 2024-04-12T04:08:06.579 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:08:06.579 DEBUG:boto:Data: 2024-04-12T04:08:06.579 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.579 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.579 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.579 DEBUG:boto:Params: {} 2024-04-12T04:08:06.579 DEBUG:boto:Token: None 2024-04-12T04:08:06.580 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.580 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000371c0a42475bba8c-006618b3a6-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.583 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.583 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.583 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.583 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:08:06.583 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:08:06.583 DEBUG:boto:Method: GET 2024-04-12T04:08:06.583 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:08:06.584 DEBUG:boto:Data: 2024-04-12T04:08:06.584 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.584 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.584 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.584 DEBUG:boto:Params: {} 2024-04-12T04:08:06.584 DEBUG:boto:Token: None 2024-04-12T04:08:06.584 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.584 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f1d46967995cc91-006618b3a6-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.588 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.589 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:08:06.589 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:08:06.589 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.589 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:08:06.589 DEBUG:boto:Data: 2024-04-12T04:08:06.589 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.589 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.589 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.589 DEBUG:boto:Params: {} 2024-04-12T04:08:06.589 DEBUG:boto:Token: None 2024-04-12T04:08:06.589 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj51 2024-04-12T04:08:06.590 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a+0Z/kn+2fwLpbIFuNvGGCOPfJY= 2024-04-12T04:08:06.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a+0Z/kn+2fwLpbIFuNvGGCOPfJY=', 'Content-Length': '0'} 2024-04-12T04:08:06.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004390f31db44faaed-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.592 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:08:06.592 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:08:06.592 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.592 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:08:06.592 DEBUG:boto:Data: 2024-04-12T04:08:06.592 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.592 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.592 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.592 DEBUG:boto:Params: {} 2024-04-12T04:08:06.593 DEBUG:boto:Token: None 2024-04-12T04:08:06.593 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj51 2024-04-12T04:08:06.593 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a+0Z/kn+2fwLpbIFuNvGGCOPfJY= 2024-04-12T04:08:06.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a+0Z/kn+2fwLpbIFuNvGGCOPfJY=', 'Content-Length': '0'} 2024-04-12T04:08:06.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b55b77303a8a6fe-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:08:06.595 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:08:06.595 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:08:06.595 DEBUG:boto:Method: GET 2024-04-12T04:08:06.595 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:08:06.595 DEBUG:boto:Data: 2024-04-12T04:08:06.596 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.596 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.596 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.596 DEBUG:boto:Params: {} 2024-04-12T04:08:06.596 DEBUG:boto:Token: None 2024-04-12T04:08:06.596 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj51 2024-04-12T04:08:06.596 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lccDSo1Y2ylnMpBrAPqpH1Dltj4= 2024-04-12T04:08:06.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lccDSo1Y2ylnMpBrAPqpH1Dltj4=', 'Content-Length': '0'} 2024-04-12T04:08:06.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000256c9089df58cded-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.638 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:08:06.639 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:08:06.639 DEBUG:boto:Method: GET 2024-04-12T04:08:06.639 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:08:06.639 DEBUG:boto:Data: 2024-04-12T04:08:06.639 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.639 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.639 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.639 DEBUG:boto:Params: {} 2024-04-12T04:08:06.639 DEBUG:boto:Token: None 2024-04-12T04:08:06.640 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj51 2024-04-12T04:08:06.640 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lccDSo1Y2ylnMpBrAPqpH1Dltj4= 2024-04-12T04:08:06.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lccDSo1Y2ylnMpBrAPqpH1Dltj4=', 'Content-Length': '0'} 2024-04-12T04:08:06.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c56bf9e5d21a07d4-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.642 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.642 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.642 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:08:06.642 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:08:06.643 DEBUG:boto:Method: GET 2024-04-12T04:08:06.643 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:08:06.643 DEBUG:boto:Data: 2024-04-12T04:08:06.643 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.643 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.643 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.643 DEBUG:boto:Params: {} 2024-04-12T04:08:06.643 DEBUG:boto:Token: None 2024-04-12T04:08:06.643 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.643 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7146fc4211d07c2-006618b3a6-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.646 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.647 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.647 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.647 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:08:06.647 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:08:06.647 DEBUG:boto:Method: GET 2024-04-12T04:08:06.647 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:08:06.647 DEBUG:boto:Data: 2024-04-12T04:08:06.648 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.648 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.648 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.648 DEBUG:boto:Params: {} 2024-04-12T04:08:06.648 DEBUG:boto:Token: None 2024-04-12T04:08:06.648 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.648 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a95918a00b5db925-006618b3a6-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.653 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:08:06.653 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:08:06.653 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:08:06.654 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.654 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.654 DEBUG:boto:Method: GET 2024-04-12T04:08:06.654 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.654 DEBUG:boto:Data: 2024-04-12T04:08:06.654 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.654 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.654 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.654 DEBUG:boto:Params: {} 2024-04-12T04:08:06.654 DEBUG:boto:Token: None 2024-04-12T04:08:06.655 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.655 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OZ4A8VdkFs0p2Ki9IVOiv5cYKS4= 2024-04-12T04:08:06.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OZ4A8VdkFs0p2Ki9IVOiv5cYKS4=', 'Content-Length': '0'} 2024-04-12T04:08:06.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001015f41c624377e3-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.657 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:08:06.657 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:08:06.657 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.657 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.657 DEBUG:boto:Method: GET 2024-04-12T04:08:06.657 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.657 DEBUG:boto:Data: 2024-04-12T04:08:06.657 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.657 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.658 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.658 DEBUG:boto:Params: {} 2024-04-12T04:08:06.658 DEBUG:boto:Token: None 2024-04-12T04:08:06.658 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.658 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OZ4A8VdkFs0p2Ki9IVOiv5cYKS4= 2024-04-12T04:08:06.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OZ4A8VdkFs0p2Ki9IVOiv5cYKS4=', 'Content-Length': '0'} 2024-04-12T04:08:06.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bbcd7bb26fdb9fd-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.702 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:08:06.703 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:08:06.703 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.703 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.703 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.703 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.703 DEBUG:boto:Data: 2024-04-12T04:08:06.703 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.703 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.704 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.704 DEBUG:boto:Params: {} 2024-04-12T04:08:06.704 DEBUG:boto:Token: None 2024-04-12T04:08:06.704 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.704 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NHkgOZnVvKTemHBvfGfkGrdmQf0= 2024-04-12T04:08:06.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NHkgOZnVvKTemHBvfGfkGrdmQf0=', 'Content-Length': '0'} 2024-04-12T04:08:06.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010bbec1480210e3d-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.706 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:08:06.707 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:08:06.707 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.707 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.707 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.707 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.707 DEBUG:boto:Data: 2024-04-12T04:08:06.707 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.707 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.707 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.707 DEBUG:boto:Params: {} 2024-04-12T04:08:06.708 DEBUG:boto:Token: None 2024-04-12T04:08:06.708 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.708 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NHkgOZnVvKTemHBvfGfkGrdmQf0= 2024-04-12T04:08:06.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NHkgOZnVvKTemHBvfGfkGrdmQf0=', 'Content-Length': '0'} 2024-04-12T04:08:06.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b20e1f8e323d8cdf-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:08:06.710 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:08:06.710 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:08:06.711 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.711 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.711 DEBUG:boto:Method: GET 2024-04-12T04:08:06.711 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.711 DEBUG:boto:Data: 2024-04-12T04:08:06.711 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.711 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.711 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.711 DEBUG:boto:Params: {} 2024-04-12T04:08:06.711 DEBUG:boto:Token: None 2024-04-12T04:08:06.712 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.712 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OZ4A8VdkFs0p2Ki9IVOiv5cYKS4= 2024-04-12T04:08:06.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OZ4A8VdkFs0p2Ki9IVOiv5cYKS4=', 'Content-Length': '0'} 2024-04-12T04:08:06.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5bc49a1c21eaaf7-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.714 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:08:06.714 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:08:06.714 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.714 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.714 DEBUG:boto:Method: GET 2024-04-12T04:08:06.714 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.714 DEBUG:boto:Data: 2024-04-12T04:08:06.714 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.715 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.715 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.715 DEBUG:boto:Params: {} 2024-04-12T04:08:06.715 DEBUG:boto:Token: None 2024-04-12T04:08:06.715 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:08:06.715 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OZ4A8VdkFs0p2Ki9IVOiv5cYKS4= 2024-04-12T04:08:06.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OZ4A8VdkFs0p2Ki9IVOiv5cYKS4=', 'Content-Length': '0'} 2024-04-12T04:08:06.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6676161e0bab8bd-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.717 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.717 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.717 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:08:06.717 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:08:06.718 DEBUG:boto:Method: GET 2024-04-12T04:08:06.718 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:08:06.718 DEBUG:boto:Data: 2024-04-12T04:08:06.718 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.718 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.718 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.718 DEBUG:boto:Params: {} 2024-04-12T04:08:06.718 DEBUG:boto:Token: None 2024-04-12T04:08:06.718 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.719 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2c3ad800e5c0fad-006618b3a6-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.721 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.721 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.722 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.722 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:08:06.722 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:08:06.722 DEBUG:boto:Method: GET 2024-04-12T04:08:06.722 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:08:06.722 DEBUG:boto:Data: 2024-04-12T04:08:06.722 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.722 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.722 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.722 DEBUG:boto:Params: {} 2024-04-12T04:08:06.722 DEBUG:boto:Token: None 2024-04-12T04:08:06.723 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.723 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000465276e986b12ab9-006618b3a6-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.727 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.727 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:08:06.727 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:08:06.727 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.727 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:08:06.727 DEBUG:boto:Data: 2024-04-12T04:08:06.728 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.728 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.728 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.728 DEBUG:boto:Params: {} 2024-04-12T04:08:06.728 DEBUG:boto:Token: None 2024-04-12T04:08:06.728 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj55 2024-04-12T04:08:06.728 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2LPca3z3u5MSYfRNQrGkY/51lg0= 2024-04-12T04:08:06.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2LPca3z3u5MSYfRNQrGkY/51lg0=', 'Content-Length': '0'} 2024-04-12T04:08:06.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc6cd377c7aea937-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.730 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:08:06.730 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:08:06.730 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.730 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:08:06.730 DEBUG:boto:Data: 2024-04-12T04:08:06.730 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.731 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.731 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.731 DEBUG:boto:Params: {} 2024-04-12T04:08:06.731 DEBUG:boto:Token: None 2024-04-12T04:08:06.731 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj55 2024-04-12T04:08:06.731 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2LPca3z3u5MSYfRNQrGkY/51lg0= 2024-04-12T04:08:06.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2LPca3z3u5MSYfRNQrGkY/51lg0=', 'Content-Length': '0'} 2024-04-12T04:08:06.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006ba810d61f33d4e-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:08:06.733 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:08:06.733 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:08:06.733 DEBUG:boto:Method: GET 2024-04-12T04:08:06.734 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:08:06.734 DEBUG:boto:Data: 2024-04-12T04:08:06.734 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.734 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.734 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.734 DEBUG:boto:Params: {} 2024-04-12T04:08:06.734 DEBUG:boto:Token: None 2024-04-12T04:08:06.734 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj55 2024-04-12T04:08:06.734 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/aKY0j8zqJxjJqndq/GsZVuZY0U= 2024-04-12T04:08:06.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/aKY0j8zqJxjJqndq/GsZVuZY0U=', 'Content-Length': '0'} 2024-04-12T04:08:06.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000066e71045fc3caf-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.736 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:08:06.737 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:08:06.737 DEBUG:boto:Method: GET 2024-04-12T04:08:06.737 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:08:06.737 DEBUG:boto:Data: 2024-04-12T04:08:06.737 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.737 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.737 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.737 DEBUG:boto:Params: {} 2024-04-12T04:08:06.737 DEBUG:boto:Token: None 2024-04-12T04:08:06.737 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj55 2024-04-12T04:08:06.738 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/aKY0j8zqJxjJqndq/GsZVuZY0U= 2024-04-12T04:08:06.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/aKY0j8zqJxjJqndq/GsZVuZY0U=', 'Content-Length': '0'} 2024-04-12T04:08:06.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000380541bae2286dd0-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.740 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.740 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.740 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:08:06.741 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:08:06.741 DEBUG:boto:Method: GET 2024-04-12T04:08:06.741 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:08:06.741 DEBUG:boto:Data: 2024-04-12T04:08:06.741 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.741 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.741 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.741 DEBUG:boto:Params: {} 2024-04-12T04:08:06.741 DEBUG:boto:Token: None 2024-04-12T04:08:06.742 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.742 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029856769c47b8ee1-006618b3a6-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.744 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.745 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.745 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.745 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:08:06.746 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:08:06.746 DEBUG:boto:Method: GET 2024-04-12T04:08:06.746 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:08:06.746 DEBUG:boto:Data: 2024-04-12T04:08:06.746 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.746 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.746 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.746 DEBUG:boto:Params: {} 2024-04-12T04:08:06.747 DEBUG:boto:Token: None 2024-04-12T04:08:06.747 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.747 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006d994f4521e3334-006618b3a6-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.794 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:08:06.795 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:08:06.795 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:08:06.795 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.795 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.795 DEBUG:boto:Method: GET 2024-04-12T04:08:06.796 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.796 DEBUG:boto:Data: 2024-04-12T04:08:06.796 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.796 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.796 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.796 DEBUG:boto:Params: {} 2024-04-12T04:08:06.796 DEBUG:boto:Token: None 2024-04-12T04:08:06.796 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.796 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7OJ1GG6f/83YoKALUra2BcvTxGE= 2024-04-12T04:08:06.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7OJ1GG6f/83YoKALUra2BcvTxGE=', 'Content-Length': '0'} 2024-04-12T04:08:06.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087391bcb4fbd3fec-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.799 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:08:06.799 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:08:06.799 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.799 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.799 DEBUG:boto:Method: GET 2024-04-12T04:08:06.799 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.799 DEBUG:boto:Data: 2024-04-12T04:08:06.799 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.800 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.800 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.800 DEBUG:boto:Params: {} 2024-04-12T04:08:06.800 DEBUG:boto:Token: None 2024-04-12T04:08:06.800 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.800 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7OJ1GG6f/83YoKALUra2BcvTxGE= 2024-04-12T04:08:06.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7OJ1GG6f/83YoKALUra2BcvTxGE=', 'Content-Length': '0'} 2024-04-12T04:08:06.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef9470b7369a01c7-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.802 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:08:06.803 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:08:06.803 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.803 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.803 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.803 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.803 DEBUG:boto:Data: 2024-04-12T04:08:06.803 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.803 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.803 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.803 DEBUG:boto:Params: {} 2024-04-12T04:08:06.803 DEBUG:boto:Token: None 2024-04-12T04:08:06.804 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.804 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:184t76zUGCjKafUbcUep6qujoQM= 2024-04-12T04:08:06.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:184t76zUGCjKafUbcUep6qujoQM=', 'Content-Length': '0'} 2024-04-12T04:08:06.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b344b7d6fbf2ee38-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.806 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:08:06.806 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:08:06.806 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.806 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.806 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.807 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.807 DEBUG:boto:Data: 2024-04-12T04:08:06.807 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.807 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.807 DEBUG:boto:Params: {} 2024-04-12T04:08:06.807 DEBUG:boto:Token: None 2024-04-12T04:08:06.807 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.807 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:184t76zUGCjKafUbcUep6qujoQM= 2024-04-12T04:08:06.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:184t76zUGCjKafUbcUep6qujoQM=', 'Content-Length': '0'} 2024-04-12T04:08:06.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048b4f2099c2a246b-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:08:06.810 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:08:06.810 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:08:06.810 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.810 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.811 DEBUG:boto:Method: GET 2024-04-12T04:08:06.811 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.811 DEBUG:boto:Data: 2024-04-12T04:08:06.811 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.811 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.812 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.812 DEBUG:boto:Params: {} 2024-04-12T04:08:06.812 DEBUG:boto:Token: None 2024-04-12T04:08:06.812 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.812 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7OJ1GG6f/83YoKALUra2BcvTxGE= 2024-04-12T04:08:06.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7OJ1GG6f/83YoKALUra2BcvTxGE=', 'Content-Length': '0'} 2024-04-12T04:08:06.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adcec3a00be22bb8-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.815 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:08:06.815 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:08:06.816 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.816 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.816 DEBUG:boto:Method: GET 2024-04-12T04:08:06.816 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.816 DEBUG:boto:Data: 2024-04-12T04:08:06.816 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.816 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.817 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.817 DEBUG:boto:Params: {} 2024-04-12T04:08:06.817 DEBUG:boto:Token: None 2024-04-12T04:08:06.817 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:08:06.817 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7OJ1GG6f/83YoKALUra2BcvTxGE= 2024-04-12T04:08:06.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7OJ1GG6f/83YoKALUra2BcvTxGE=', 'Content-Length': '0'} 2024-04-12T04:08:06.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b77c75c3ba40200e-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.820 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.820 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.820 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:08:06.820 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:08:06.820 DEBUG:boto:Method: GET 2024-04-12T04:08:06.820 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:08:06.820 DEBUG:boto:Data: 2024-04-12T04:08:06.820 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.820 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.820 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.821 DEBUG:boto:Params: {} 2024-04-12T04:08:06.821 DEBUG:boto:Token: None 2024-04-12T04:08:06.821 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.821 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002229d4d548012ae8-006618b3a6-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.870 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.874 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.874 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.874 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:08:06.874 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:08:06.874 DEBUG:boto:Method: GET 2024-04-12T04:08:06.874 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:08:06.874 DEBUG:boto:Data: 2024-04-12T04:08:06.874 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.874 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.874 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.875 DEBUG:boto:Params: {} 2024-04-12T04:08:06.875 DEBUG:boto:Token: None 2024-04-12T04:08:06.875 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.875 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000397e11f5a1a2d161-006618b3a6-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.879 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.880 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:08:06.880 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:08:06.880 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.880 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:08:06.880 DEBUG:boto:Data: 2024-04-12T04:08:06.880 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.880 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.880 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.881 DEBUG:boto:Params: {} 2024-04-12T04:08:06.881 DEBUG:boto:Token: None 2024-04-12T04:08:06.881 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj57 2024-04-12T04:08:06.881 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VyNOTji9MMRJ6aaRzj9x6pPzxgU= 2024-04-12T04:08:06.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VyNOTji9MMRJ6aaRzj9x6pPzxgU=', 'Content-Length': '0'} 2024-04-12T04:08:06.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008342e0d52a644ec0-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.883 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:08:06.883 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:08:06.883 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.883 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:08:06.883 DEBUG:boto:Data: 2024-04-12T04:08:06.883 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.884 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.884 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.884 DEBUG:boto:Params: {} 2024-04-12T04:08:06.884 DEBUG:boto:Token: None 2024-04-12T04:08:06.884 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj57 2024-04-12T04:08:06.884 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VyNOTji9MMRJ6aaRzj9x6pPzxgU= 2024-04-12T04:08:06.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VyNOTji9MMRJ6aaRzj9x6pPzxgU=', 'Content-Length': '0'} 2024-04-12T04:08:06.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046d77dab3f9a23b0-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:08:06.886 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:08:06.887 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:08:06.887 DEBUG:boto:Method: GET 2024-04-12T04:08:06.887 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:08:06.887 DEBUG:boto:Data: 2024-04-12T04:08:06.887 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.887 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.887 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.887 DEBUG:boto:Params: {} 2024-04-12T04:08:06.887 DEBUG:boto:Token: None 2024-04-12T04:08:06.887 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj57 2024-04-12T04:08:06.888 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9WzVXNUw6rAdN2CfSQuGqSCi6BY= 2024-04-12T04:08:06.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9WzVXNUw6rAdN2CfSQuGqSCi6BY=', 'Content-Length': '0'} 2024-04-12T04:08:06.890 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b724662290d803f-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.890 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:08:06.890 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:08:06.890 DEBUG:boto:Method: GET 2024-04-12T04:08:06.890 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:08:06.890 DEBUG:boto:Data: 2024-04-12T04:08:06.890 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.890 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.890 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.891 DEBUG:boto:Params: {} 2024-04-12T04:08:06.891 DEBUG:boto:Token: None 2024-04-12T04:08:06.891 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj57 2024-04-12T04:08:06.891 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9WzVXNUw6rAdN2CfSQuGqSCi6BY= 2024-04-12T04:08:06.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9WzVXNUw6rAdN2CfSQuGqSCi6BY=', 'Content-Length': '0'} 2024-04-12T04:08:06.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed70e62be86ed268-006618b3a6-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.938 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.939 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.939 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:08:06.939 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:08:06.939 DEBUG:boto:Method: GET 2024-04-12T04:08:06.939 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:08:06.939 DEBUG:boto:Data: 2024-04-12T04:08:06.939 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.939 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.939 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.939 DEBUG:boto:Params: {} 2024-04-12T04:08:06.939 DEBUG:boto:Token: None 2024-04-12T04:08:06.940 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.940 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1c9df28318fc3b5-006618b3a6-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.942 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.943 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:06.943 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:06.943 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:08:06.943 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:08:06.943 DEBUG:boto:Method: GET 2024-04-12T04:08:06.943 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:08:06.943 DEBUG:boto:Data: 2024-04-12T04:08:06.943 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.944 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.944 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.944 DEBUG:boto:Params: {} 2024-04-12T04:08:06.944 DEBUG:boto:Token: None 2024-04-12T04:08:06.944 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/ 2024-04-12T04:08:06.944 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw= 2024-04-12T04:08:06.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OuOhgtfZoeYxUe/7D8NLnS9K1lw=', 'Content-Length': '0'} 2024-04-12T04:08:06.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000690bb2a0d9cb77e2-006618b3a6-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.948 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:06.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:08:06.949 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:08:06.949 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:08:06.949 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:08:06.949 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:08:06.949 DEBUG:boto:Method: GET 2024-04-12T04:08:06.949 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:08:06.949 DEBUG:boto:Data: 2024-04-12T04:08:06.949 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.949 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.950 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.950 DEBUG:boto:Params: {} 2024-04-12T04:08:06.950 DEBUG:boto:Token: None 2024-04-12T04:08:06.950 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:08:06.950 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NHQjmV+tY2JCQcebIqBZwZU/aAY= 2024-04-12T04:08:06.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NHQjmV+tY2JCQcebIqBZwZU/aAY=', 'Content-Length': '0'} 2024-04-12T04:08:06.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000021d0e55fabc6aad-006618b3a6-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.994 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:08:06.995 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:08:06.995 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:08:06.995 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:08:06.995 DEBUG:boto:Method: GET 2024-04-12T04:08:06.995 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:08:06.995 DEBUG:boto:Data: 2024-04-12T04:08:06.995 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.995 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:06.995 DEBUG:boto:Port: 8000 2024-04-12T04:08:06.995 DEBUG:boto:Params: {} 2024-04-12T04:08:06.996 DEBUG:boto:Token: None 2024-04-12T04:08:06.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:08:06.996 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NHQjmV+tY2JCQcebIqBZwZU/aAY= 2024-04-12T04:08:06.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NHQjmV+tY2JCQcebIqBZwZU/aAY=', 'Content-Length': '0'} 2024-04-12T04:08:06.998 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000045e89422390b935-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:06.998 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:08:06.998 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:08:06.999 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:08:06.999 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:08:06.999 DEBUG:boto:Method: HEAD 2024-04-12T04:08:06.999 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:08:06.999 DEBUG:boto:Data: 2024-04-12T04:08:06.999 DEBUG:boto:Headers: {} 2024-04-12T04:08:06.999 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:06.999 DEBUG:boto:Port: 8001 2024-04-12T04:08:06.999 DEBUG:boto:Params: {} 2024-04-12T04:08:06.999 DEBUG:boto:Token: None 2024-04-12T04:08:06.999 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:06 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:08:07.000 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CNnn0WDYbIfz2sTGBqZFsXSvXw4= 2024-04-12T04:08:07.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CNnn0WDYbIfz2sTGBqZFsXSvXw4=', 'Content-Length': '0'} 2024-04-12T04:08:07.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d0f4b4f99f6c91e-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.002 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:08:07.002 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:08:07.002 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:08:07.002 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:08:07.002 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.002 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:08:07.003 DEBUG:boto:Data: 2024-04-12T04:08:07.003 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.003 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.003 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.003 DEBUG:boto:Params: {} 2024-04-12T04:08:07.004 DEBUG:boto:Token: None 2024-04-12T04:08:07.004 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:08:07.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a9JLE0vK3wAQquXX//emgutypMs= 2024-04-12T04:08:07.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a9JLE0vK3wAQquXX//emgutypMs=', 'Content-Length': '0'} 2024-04-12T04:08:07.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b59d50ee000ae3a-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:08:07.007 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:08:07.007 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:08:07.007 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:08:07.007 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:08:07.007 DEBUG:boto:Method: GET 2024-04-12T04:08:07.007 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:08:07.007 DEBUG:boto:Data: 2024-04-12T04:08:07.007 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.007 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.007 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.008 DEBUG:boto:Params: {} 2024-04-12T04:08:07.008 DEBUG:boto:Token: None 2024-04-12T04:08:07.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:08:07.008 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uSJ+Z/5V+Gqt+lc01KPck4uj8qA= 2024-04-12T04:08:07.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uSJ+Z/5V+Gqt+lc01KPck4uj8qA=', 'Content-Length': '0'} 2024-04-12T04:08:07.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9580b3e21380d77-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.010 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:08:07.010 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:08:07.010 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:08:07.010 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:08:07.010 DEBUG:boto:Method: GET 2024-04-12T04:08:07.011 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:08:07.011 DEBUG:boto:Data: 2024-04-12T04:08:07.011 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.011 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.011 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.011 DEBUG:boto:Params: {} 2024-04-12T04:08:07.011 DEBUG:boto:Token: None 2024-04-12T04:08:07.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:08:07.011 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uSJ+Z/5V+Gqt+lc01KPck4uj8qA= 2024-04-12T04:08:07.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uSJ+Z/5V+Gqt+lc01KPck4uj8qA=', 'Content-Length': '0'} 2024-04-12T04:08:07.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002601a4ded3d82e18-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.014 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.014 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.014 DEBUG:boto:path=/dkyges-61/?prefix=obj58 2024-04-12T04:08:07.014 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj58 2024-04-12T04:08:07.014 DEBUG:boto:Method: GET 2024-04-12T04:08:07.014 DEBUG:boto:Path: /dkyges-61/?prefix=obj58 2024-04-12T04:08:07.014 DEBUG:boto:Data: 2024-04-12T04:08:07.014 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.014 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.015 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.015 DEBUG:boto:Params: {} 2024-04-12T04:08:07.015 DEBUG:boto:Token: None 2024-04-12T04:08:07.015 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.015 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000015f9bbe525581fa-006618b3a7-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.017 DEBUG:boto:b'dkyges-61obj581000falseobj582024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.018 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.018 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.018 DEBUG:boto:path=/dkyges-61/?prefix=obj58 2024-04-12T04:08:07.018 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj58 2024-04-12T04:08:07.018 DEBUG:boto:Method: GET 2024-04-12T04:08:07.018 DEBUG:boto:Path: /dkyges-61/?prefix=obj58 2024-04-12T04:08:07.018 DEBUG:boto:Data: 2024-04-12T04:08:07.019 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.019 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.019 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.019 DEBUG:boto:Params: {} 2024-04-12T04:08:07.019 DEBUG:boto:Token: None 2024-04-12T04:08:07.019 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.019 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f9596742d3b7d39-006618b3a7-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.023 DEBUG:boto:b'dkyges-61obj581000falseobj582024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.023 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:08:07.023 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:08:07.023 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.023 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:08:07.024 DEBUG:boto:Data: 2024-04-12T04:08:07.024 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.024 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.024 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.024 DEBUG:boto:Params: {} 2024-04-12T04:08:07.024 DEBUG:boto:Token: None 2024-04-12T04:08:07.024 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj58 2024-04-12T04:08:07.024 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SWsQVNJqhyugDp7F5stTX4rAvD4= 2024-04-12T04:08:07.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SWsQVNJqhyugDp7F5stTX4rAvD4=', 'Content-Length': '0'} 2024-04-12T04:08:07.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac2ce7d9bfa20f2a-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.026 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:08:07.026 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:08:07.027 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.027 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:08:07.027 DEBUG:boto:Data: 2024-04-12T04:08:07.027 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.027 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.027 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.027 DEBUG:boto:Params: {} 2024-04-12T04:08:07.027 DEBUG:boto:Token: None 2024-04-12T04:08:07.027 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj58 2024-04-12T04:08:07.027 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SWsQVNJqhyugDp7F5stTX4rAvD4= 2024-04-12T04:08:07.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SWsQVNJqhyugDp7F5stTX4rAvD4=', 'Content-Length': '0'} 2024-04-12T04:08:07.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f22c85a949482a0a-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:08:07.030 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:08:07.030 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:08:07.030 DEBUG:boto:Method: GET 2024-04-12T04:08:07.030 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:08:07.030 DEBUG:boto:Data: 2024-04-12T04:08:07.030 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.030 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.031 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.031 DEBUG:boto:Params: {} 2024-04-12T04:08:07.031 DEBUG:boto:Token: None 2024-04-12T04:08:07.031 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj58 2024-04-12T04:08:07.031 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dGmSd1N5Q0D2w2ctfPNSXX4uKPo= 2024-04-12T04:08:07.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dGmSd1N5Q0D2w2ctfPNSXX4uKPo=', 'Content-Length': '0'} 2024-04-12T04:08:07.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007df98aae9c1b6d0c-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.033 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:08:07.033 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:08:07.033 DEBUG:boto:Method: GET 2024-04-12T04:08:07.034 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:08:07.034 DEBUG:boto:Data: 2024-04-12T04:08:07.034 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.034 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.034 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.034 DEBUG:boto:Params: {} 2024-04-12T04:08:07.034 DEBUG:boto:Token: None 2024-04-12T04:08:07.034 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj58 2024-04-12T04:08:07.034 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dGmSd1N5Q0D2w2ctfPNSXX4uKPo= 2024-04-12T04:08:07.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dGmSd1N5Q0D2w2ctfPNSXX4uKPo=', 'Content-Length': '0'} 2024-04-12T04:08:07.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050595139ee6381c7-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.037 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.037 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.037 DEBUG:boto:path=/dkyges-61/?prefix=obj58 2024-04-12T04:08:07.037 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj58 2024-04-12T04:08:07.037 DEBUG:boto:Method: GET 2024-04-12T04:08:07.037 DEBUG:boto:Path: /dkyges-61/?prefix=obj58 2024-04-12T04:08:07.037 DEBUG:boto:Data: 2024-04-12T04:08:07.038 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.038 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.038 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.038 DEBUG:boto:Params: {} 2024-04-12T04:08:07.038 DEBUG:boto:Token: None 2024-04-12T04:08:07.038 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.038 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c5a59322dc3614d-006618b3a7-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.041 DEBUG:boto:b'dkyges-61obj581000falseobj582024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.041 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.041 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.041 DEBUG:boto:path=/dkyges-61/?prefix=obj58 2024-04-12T04:08:07.041 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj58 2024-04-12T04:08:07.042 DEBUG:boto:Method: GET 2024-04-12T04:08:07.042 DEBUG:boto:Path: /dkyges-61/?prefix=obj58 2024-04-12T04:08:07.042 DEBUG:boto:Data: 2024-04-12T04:08:07.042 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.042 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.042 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.042 DEBUG:boto:Params: {} 2024-04-12T04:08:07.042 DEBUG:boto:Token: None 2024-04-12T04:08:07.042 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.042 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.046 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e08df0308af3f7fd-006618b3a7-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.046 DEBUG:boto:b'dkyges-61obj581000falseobj582024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:08:07.047 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:08:07.047 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:08:07.047 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.047 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.047 DEBUG:boto:Method: GET 2024-04-12T04:08:07.047 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.047 DEBUG:boto:Data: 2024-04-12T04:08:07.048 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.048 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.048 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.048 DEBUG:boto:Params: {} 2024-04-12T04:08:07.048 DEBUG:boto:Token: None 2024-04-12T04:08:07.048 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.048 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7CoOvcjkKR7m7HoM7vHIP0SGH8A= 2024-04-12T04:08:07.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7CoOvcjkKR7m7HoM7vHIP0SGH8A=', 'Content-Length': '0'} 2024-04-12T04:08:07.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000953d7208151f72ef-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.050 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:08:07.050 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:08:07.051 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.051 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.051 DEBUG:boto:Method: GET 2024-04-12T04:08:07.051 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.051 DEBUG:boto:Data: 2024-04-12T04:08:07.051 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.051 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.051 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.051 DEBUG:boto:Params: {} 2024-04-12T04:08:07.051 DEBUG:boto:Token: None 2024-04-12T04:08:07.052 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.052 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7CoOvcjkKR7m7HoM7vHIP0SGH8A= 2024-04-12T04:08:07.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7CoOvcjkKR7m7HoM7vHIP0SGH8A=', 'Content-Length': '0'} 2024-04-12T04:08:07.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000772eff878059f629-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.094 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:08:07.095 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:08:07.095 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.095 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.095 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.095 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.095 DEBUG:boto:Data: 2024-04-12T04:08:07.095 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.095 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.096 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.096 DEBUG:boto:Params: {} 2024-04-12T04:08:07.096 DEBUG:boto:Token: None 2024-04-12T04:08:07.096 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.096 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A64ZvOBw+tYu/Qi7DtOUNJRPmP8= 2024-04-12T04:08:07.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A64ZvOBw+tYu/Qi7DtOUNJRPmP8=', 'Content-Length': '0'} 2024-04-12T04:08:07.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9be5d17faadf2d5-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.099 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:08:07.099 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:08:07.099 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.099 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.099 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.099 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.099 DEBUG:boto:Data: 2024-04-12T04:08:07.099 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.099 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.099 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.099 DEBUG:boto:Params: {} 2024-04-12T04:08:07.100 DEBUG:boto:Token: None 2024-04-12T04:08:07.100 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.100 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A64ZvOBw+tYu/Qi7DtOUNJRPmP8= 2024-04-12T04:08:07.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A64ZvOBw+tYu/Qi7DtOUNJRPmP8=', 'Content-Length': '0'} 2024-04-12T04:08:07.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d0fd39d1fdd33e6-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:08:07.102 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:08:07.103 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:08:07.103 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.103 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.103 DEBUG:boto:Method: GET 2024-04-12T04:08:07.103 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.103 DEBUG:boto:Data: 2024-04-12T04:08:07.103 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.103 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.103 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.103 DEBUG:boto:Params: {} 2024-04-12T04:08:07.104 DEBUG:boto:Token: None 2024-04-12T04:08:07.104 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.104 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7CoOvcjkKR7m7HoM7vHIP0SGH8A= 2024-04-12T04:08:07.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7CoOvcjkKR7m7HoM7vHIP0SGH8A=', 'Content-Length': '0'} 2024-04-12T04:08:07.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd25fc145cfccec1-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.106 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:08:07.106 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:08:07.106 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.107 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.107 DEBUG:boto:Method: GET 2024-04-12T04:08:07.107 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.107 DEBUG:boto:Data: 2024-04-12T04:08:07.107 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.107 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.107 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.107 DEBUG:boto:Params: {} 2024-04-12T04:08:07.108 DEBUG:boto:Token: None 2024-04-12T04:08:07.108 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:08:07.108 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7CoOvcjkKR7m7HoM7vHIP0SGH8A= 2024-04-12T04:08:07.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7CoOvcjkKR7m7HoM7vHIP0SGH8A=', 'Content-Length': '0'} 2024-04-12T04:08:07.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2647f6fde469f15-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.110 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.110 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.110 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:08:07.110 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:08:07.110 DEBUG:boto:Method: GET 2024-04-12T04:08:07.111 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:08:07.111 DEBUG:boto:Data: 2024-04-12T04:08:07.111 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.111 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.111 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.111 DEBUG:boto:Params: {} 2024-04-12T04:08:07.111 DEBUG:boto:Token: None 2024-04-12T04:08:07.111 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.111 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034875cf273b1aec6-006618b3a7-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.114 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.115 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.115 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.115 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:08:07.115 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:08:07.115 DEBUG:boto:Method: GET 2024-04-12T04:08:07.115 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:08:07.115 DEBUG:boto:Data: 2024-04-12T04:08:07.115 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.116 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.116 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.116 DEBUG:boto:Params: {} 2024-04-12T04:08:07.116 DEBUG:boto:Token: None 2024-04-12T04:08:07.116 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.116 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000376ffe4ed582abff-006618b3a7-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.120 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.121 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:08:07.121 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:08:07.121 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.121 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:08:07.121 DEBUG:boto:Data: 2024-04-12T04:08:07.121 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.121 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.121 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.121 DEBUG:boto:Params: {} 2024-04-12T04:08:07.122 DEBUG:boto:Token: None 2024-04-12T04:08:07.122 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj68 2024-04-12T04:08:07.122 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:73kN98JPL1Ya0rdzVlT3xja2X1o= 2024-04-12T04:08:07.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:73kN98JPL1Ya0rdzVlT3xja2X1o=', 'Content-Length': '0'} 2024-04-12T04:08:07.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c5f546b9bc320fa-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.124 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:08:07.124 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:08:07.124 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.124 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:08:07.124 DEBUG:boto:Data: 2024-04-12T04:08:07.124 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.124 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.125 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.125 DEBUG:boto:Params: {} 2024-04-12T04:08:07.125 DEBUG:boto:Token: None 2024-04-12T04:08:07.125 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj68 2024-04-12T04:08:07.125 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:73kN98JPL1Ya0rdzVlT3xja2X1o= 2024-04-12T04:08:07.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:73kN98JPL1Ya0rdzVlT3xja2X1o=', 'Content-Length': '0'} 2024-04-12T04:08:07.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000640c0166903189fe-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:08:07.127 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:08:07.127 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:08:07.127 DEBUG:boto:Method: GET 2024-04-12T04:08:07.127 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:08:07.128 DEBUG:boto:Data: 2024-04-12T04:08:07.128 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.128 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.128 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.128 DEBUG:boto:Params: {} 2024-04-12T04:08:07.128 DEBUG:boto:Token: None 2024-04-12T04:08:07.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj68 2024-04-12T04:08:07.128 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JVn1n1MVxMf5d86NHJ+plIaD1K0= 2024-04-12T04:08:07.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JVn1n1MVxMf5d86NHJ+plIaD1K0=', 'Content-Length': '0'} 2024-04-12T04:08:07.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047cbc0980474c42d-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.130 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:08:07.130 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:08:07.131 DEBUG:boto:Method: GET 2024-04-12T04:08:07.131 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:08:07.131 DEBUG:boto:Data: 2024-04-12T04:08:07.131 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.131 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.131 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.131 DEBUG:boto:Params: {} 2024-04-12T04:08:07.131 DEBUG:boto:Token: None 2024-04-12T04:08:07.131 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj68 2024-04-12T04:08:07.131 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JVn1n1MVxMf5d86NHJ+plIaD1K0= 2024-04-12T04:08:07.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JVn1n1MVxMf5d86NHJ+plIaD1K0=', 'Content-Length': '0'} 2024-04-12T04:08:07.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad54f264929a3d88-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.174 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.175 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.175 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:08:07.175 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:08:07.175 DEBUG:boto:Method: GET 2024-04-12T04:08:07.175 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:08:07.175 DEBUG:boto:Data: 2024-04-12T04:08:07.175 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.175 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.175 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.175 DEBUG:boto:Params: {} 2024-04-12T04:08:07.175 DEBUG:boto:Token: None 2024-04-12T04:08:07.176 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.176 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bbac9a7b930b7a9-006618b3a7-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.178 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.179 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.179 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.179 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:08:07.179 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:08:07.179 DEBUG:boto:Method: GET 2024-04-12T04:08:07.179 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:08:07.179 DEBUG:boto:Data: 2024-04-12T04:08:07.179 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.180 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.180 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.180 DEBUG:boto:Params: {} 2024-04-12T04:08:07.180 DEBUG:boto:Token: None 2024-04-12T04:08:07.180 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.180 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025b47ea1c2f9e927-006618b3a7-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.184 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:08:07.185 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:08:07.185 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:08:07.185 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.185 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.185 DEBUG:boto:Method: GET 2024-04-12T04:08:07.185 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.185 DEBUG:boto:Data: 2024-04-12T04:08:07.185 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.186 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.186 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.186 DEBUG:boto:Params: {} 2024-04-12T04:08:07.186 DEBUG:boto:Token: None 2024-04-12T04:08:07.186 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.186 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sePmB+g7szjJnQ0V0nU4hgEkERw= 2024-04-12T04:08:07.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sePmB+g7szjJnQ0V0nU4hgEkERw=', 'Content-Length': '0'} 2024-04-12T04:08:07.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031cec730b5b41be7-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.230 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:08:07.231 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:08:07.231 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.231 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.231 DEBUG:boto:Method: GET 2024-04-12T04:08:07.231 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.231 DEBUG:boto:Data: 2024-04-12T04:08:07.231 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.231 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.231 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.231 DEBUG:boto:Params: {} 2024-04-12T04:08:07.232 DEBUG:boto:Token: None 2024-04-12T04:08:07.232 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.232 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sePmB+g7szjJnQ0V0nU4hgEkERw= 2024-04-12T04:08:07.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sePmB+g7szjJnQ0V0nU4hgEkERw=', 'Content-Length': '0'} 2024-04-12T04:08:07.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab88e591230e4c0d-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.234 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:08:07.235 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:08:07.235 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.235 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.235 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.235 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.235 DEBUG:boto:Data: 2024-04-12T04:08:07.235 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.235 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.235 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.235 DEBUG:boto:Params: {} 2024-04-12T04:08:07.235 DEBUG:boto:Token: None 2024-04-12T04:08:07.236 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.236 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IKH0hnrWmu1jBOUoPsiw4BqOCMk= 2024-04-12T04:08:07.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IKH0hnrWmu1jBOUoPsiw4BqOCMk=', 'Content-Length': '0'} 2024-04-12T04:08:07.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dec0e9829b44fdd8-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.238 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:08:07.238 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:08:07.238 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.238 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.238 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.238 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.238 DEBUG:boto:Data: 2024-04-12T04:08:07.239 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.239 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.239 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.239 DEBUG:boto:Params: {} 2024-04-12T04:08:07.239 DEBUG:boto:Token: None 2024-04-12T04:08:07.239 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.239 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IKH0hnrWmu1jBOUoPsiw4BqOCMk= 2024-04-12T04:08:07.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IKH0hnrWmu1jBOUoPsiw4BqOCMk=', 'Content-Length': '0'} 2024-04-12T04:08:07.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c38bb44b03cdcbb-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:08:07.241 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:08:07.241 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:08:07.241 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.242 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.242 DEBUG:boto:Method: GET 2024-04-12T04:08:07.242 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.242 DEBUG:boto:Data: 2024-04-12T04:08:07.242 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.242 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.242 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.242 DEBUG:boto:Params: {} 2024-04-12T04:08:07.242 DEBUG:boto:Token: None 2024-04-12T04:08:07.242 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.243 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sePmB+g7szjJnQ0V0nU4hgEkERw= 2024-04-12T04:08:07.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sePmB+g7szjJnQ0V0nU4hgEkERw=', 'Content-Length': '0'} 2024-04-12T04:08:07.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000646a2d085bbdcb8d-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.244 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:08:07.245 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:08:07.245 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.245 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.245 DEBUG:boto:Method: GET 2024-04-12T04:08:07.245 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.245 DEBUG:boto:Data: 2024-04-12T04:08:07.245 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.245 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.245 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.245 DEBUG:boto:Params: {} 2024-04-12T04:08:07.245 DEBUG:boto:Token: None 2024-04-12T04:08:07.246 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:08:07.246 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sePmB+g7szjJnQ0V0nU4hgEkERw= 2024-04-12T04:08:07.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sePmB+g7szjJnQ0V0nU4hgEkERw=', 'Content-Length': '0'} 2024-04-12T04:08:07.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029ca8fac3c1339e1-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.248 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.248 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.248 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:08:07.248 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:08:07.249 DEBUG:boto:Method: GET 2024-04-12T04:08:07.249 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:08:07.249 DEBUG:boto:Data: 2024-04-12T04:08:07.249 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.249 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.249 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.249 DEBUG:boto:Params: {} 2024-04-12T04:08:07.249 DEBUG:boto:Token: None 2024-04-12T04:08:07.249 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.249 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fcdbf9eba8885b4-006618b3a7-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.252 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.253 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.253 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.253 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:08:07.253 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:08:07.253 DEBUG:boto:Method: GET 2024-04-12T04:08:07.253 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:08:07.253 DEBUG:boto:Data: 2024-04-12T04:08:07.253 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.253 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.253 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.254 DEBUG:boto:Params: {} 2024-04-12T04:08:07.254 DEBUG:boto:Token: None 2024-04-12T04:08:07.254 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.254 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087667b8914d92707-006618b3a7-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.258 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.258 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:08:07.258 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:08:07.259 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.259 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:08:07.259 DEBUG:boto:Data: 2024-04-12T04:08:07.259 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.259 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.259 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.259 DEBUG:boto:Params: {} 2024-04-12T04:08:07.259 DEBUG:boto:Token: None 2024-04-12T04:08:07.259 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj69 2024-04-12T04:08:07.259 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ESm3mYC+fYq6zUlbbuHWZCuvCCQ= 2024-04-12T04:08:07.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ESm3mYC+fYq6zUlbbuHWZCuvCCQ=', 'Content-Length': '0'} 2024-04-12T04:08:07.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9bb8cfd495f978b-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.262 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:08:07.262 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:08:07.262 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.262 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:08:07.262 DEBUG:boto:Data: 2024-04-12T04:08:07.262 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.262 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.262 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.262 DEBUG:boto:Params: {} 2024-04-12T04:08:07.262 DEBUG:boto:Token: None 2024-04-12T04:08:07.263 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj69 2024-04-12T04:08:07.263 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ESm3mYC+fYq6zUlbbuHWZCuvCCQ= 2024-04-12T04:08:07.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ESm3mYC+fYq6zUlbbuHWZCuvCCQ=', 'Content-Length': '0'} 2024-04-12T04:08:07.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f21faaed31a279ff-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:08:07.265 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:08:07.265 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:08:07.265 DEBUG:boto:Method: GET 2024-04-12T04:08:07.265 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:08:07.265 DEBUG:boto:Data: 2024-04-12T04:08:07.265 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.265 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.265 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.265 DEBUG:boto:Params: {} 2024-04-12T04:08:07.266 DEBUG:boto:Token: None 2024-04-12T04:08:07.266 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj69 2024-04-12T04:08:07.266 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FOVMD/sJBYb5G4Wf4GGIkcd19JE= 2024-04-12T04:08:07.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FOVMD/sJBYb5G4Wf4GGIkcd19JE=', 'Content-Length': '0'} 2024-04-12T04:08:07.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfa77d845ba88bb3-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.268 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:08:07.268 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:08:07.268 DEBUG:boto:Method: GET 2024-04-12T04:08:07.268 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:08:07.268 DEBUG:boto:Data: 2024-04-12T04:08:07.268 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.268 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.268 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.268 DEBUG:boto:Params: {} 2024-04-12T04:08:07.269 DEBUG:boto:Token: None 2024-04-12T04:08:07.269 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj69 2024-04-12T04:08:07.269 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FOVMD/sJBYb5G4Wf4GGIkcd19JE= 2024-04-12T04:08:07.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FOVMD/sJBYb5G4Wf4GGIkcd19JE=', 'Content-Length': '0'} 2024-04-12T04:08:07.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a23b13229fe0e311-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.271 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.271 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.271 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:08:07.271 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:08:07.271 DEBUG:boto:Method: GET 2024-04-12T04:08:07.271 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:08:07.271 DEBUG:boto:Data: 2024-04-12T04:08:07.271 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.271 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.272 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.272 DEBUG:boto:Params: {} 2024-04-12T04:08:07.272 DEBUG:boto:Token: None 2024-04-12T04:08:07.272 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.272 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce55df75962f1c99-006618b3a7-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.274 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.275 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.275 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.275 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:08:07.275 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:08:07.275 DEBUG:boto:Method: GET 2024-04-12T04:08:07.275 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:08:07.276 DEBUG:boto:Data: 2024-04-12T04:08:07.276 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.276 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.276 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.276 DEBUG:boto:Params: {} 2024-04-12T04:08:07.276 DEBUG:boto:Token: None 2024-04-12T04:08:07.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.276 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4ca21c611b9d20c-006618b3a7-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.280 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:08:07.280 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:08:07.281 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:08:07.281 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.281 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.281 DEBUG:boto:Method: GET 2024-04-12T04:08:07.281 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.281 DEBUG:boto:Data: 2024-04-12T04:08:07.281 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.281 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.281 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.281 DEBUG:boto:Params: {} 2024-04-12T04:08:07.281 DEBUG:boto:Token: None 2024-04-12T04:08:07.282 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.282 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G8IproW/jImh4IQlrmsrFYCyyZU= 2024-04-12T04:08:07.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G8IproW/jImh4IQlrmsrFYCyyZU=', 'Content-Length': '0'} 2024-04-12T04:08:07.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e9d49639606266e-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.330 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:08:07.331 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:08:07.331 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.331 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.331 DEBUG:boto:Method: GET 2024-04-12T04:08:07.331 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.331 DEBUG:boto:Data: 2024-04-12T04:08:07.331 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.331 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.331 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.331 DEBUG:boto:Params: {} 2024-04-12T04:08:07.332 DEBUG:boto:Token: None 2024-04-12T04:08:07.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.332 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G8IproW/jImh4IQlrmsrFYCyyZU= 2024-04-12T04:08:07.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G8IproW/jImh4IQlrmsrFYCyyZU=', 'Content-Length': '0'} 2024-04-12T04:08:07.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b4914b368b88615-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.334 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:08:07.334 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:08:07.335 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.335 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.335 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.335 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.335 DEBUG:boto:Data: 2024-04-12T04:08:07.335 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.335 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.335 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.335 DEBUG:boto:Params: {} 2024-04-12T04:08:07.335 DEBUG:boto:Token: None 2024-04-12T04:08:07.335 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.336 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SuCTUqqFvEZujryxAjx5DXPHSBM= 2024-04-12T04:08:07.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SuCTUqqFvEZujryxAjx5DXPHSBM=', 'Content-Length': '0'} 2024-04-12T04:08:07.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b628bd6900b64363-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.338 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:08:07.338 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:08:07.338 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.338 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.338 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.338 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.338 DEBUG:boto:Data: 2024-04-12T04:08:07.338 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.339 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.339 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.339 DEBUG:boto:Params: {} 2024-04-12T04:08:07.339 DEBUG:boto:Token: None 2024-04-12T04:08:07.339 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.339 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SuCTUqqFvEZujryxAjx5DXPHSBM= 2024-04-12T04:08:07.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SuCTUqqFvEZujryxAjx5DXPHSBM=', 'Content-Length': '0'} 2024-04-12T04:08:07.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005807054409bab28b-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:08:07.341 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:08:07.341 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:08:07.342 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.342 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.342 DEBUG:boto:Method: GET 2024-04-12T04:08:07.342 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.342 DEBUG:boto:Data: 2024-04-12T04:08:07.342 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.342 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.342 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.342 DEBUG:boto:Params: {} 2024-04-12T04:08:07.342 DEBUG:boto:Token: None 2024-04-12T04:08:07.342 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.343 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G8IproW/jImh4IQlrmsrFYCyyZU= 2024-04-12T04:08:07.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G8IproW/jImh4IQlrmsrFYCyyZU=', 'Content-Length': '0'} 2024-04-12T04:08:07.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001649a77ca1ff77ff-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.345 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:08:07.345 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:08:07.345 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.345 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.345 DEBUG:boto:Method: GET 2024-04-12T04:08:07.345 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.346 DEBUG:boto:Data: 2024-04-12T04:08:07.346 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.346 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.346 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.346 DEBUG:boto:Params: {} 2024-04-12T04:08:07.346 DEBUG:boto:Token: None 2024-04-12T04:08:07.346 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:08:07.346 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G8IproW/jImh4IQlrmsrFYCyyZU= 2024-04-12T04:08:07.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G8IproW/jImh4IQlrmsrFYCyyZU=', 'Content-Length': '0'} 2024-04-12T04:08:07.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002376ce5b0bf0defa-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.348 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.348 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.349 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:08:07.349 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:08:07.349 DEBUG:boto:Method: GET 2024-04-12T04:08:07.349 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:08:07.349 DEBUG:boto:Data: 2024-04-12T04:08:07.349 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.349 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.349 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.349 DEBUG:boto:Params: {} 2024-04-12T04:08:07.349 DEBUG:boto:Token: None 2024-04-12T04:08:07.349 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.350 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.352 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000507711263032c01d-006618b3a7-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.352 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.353 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.353 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.353 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:08:07.353 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:08:07.353 DEBUG:boto:Method: GET 2024-04-12T04:08:07.353 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:08:07.353 DEBUG:boto:Data: 2024-04-12T04:08:07.353 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.353 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.353 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.353 DEBUG:boto:Params: {} 2024-04-12T04:08:07.354 DEBUG:boto:Token: None 2024-04-12T04:08:07.354 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.354 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004803ec1bd2f5a2e9-006618b3a7-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.358 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.358 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:08:07.358 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:08:07.358 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.358 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:08:07.358 DEBUG:boto:Data: 2024-04-12T04:08:07.359 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.359 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.359 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.359 DEBUG:boto:Params: {} 2024-04-12T04:08:07.359 DEBUG:boto:Token: None 2024-04-12T04:08:07.359 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj76 2024-04-12T04:08:07.359 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wr1kgvbxx6iZXHM0URQEmkG1qHs= 2024-04-12T04:08:07.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wr1kgvbxx6iZXHM0URQEmkG1qHs=', 'Content-Length': '0'} 2024-04-12T04:08:07.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7c96a7cb1ba8f6f-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.361 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:08:07.361 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:08:07.361 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.362 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:08:07.362 DEBUG:boto:Data: 2024-04-12T04:08:07.362 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.362 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.362 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.362 DEBUG:boto:Params: {} 2024-04-12T04:08:07.362 DEBUG:boto:Token: None 2024-04-12T04:08:07.362 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj76 2024-04-12T04:08:07.362 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wr1kgvbxx6iZXHM0URQEmkG1qHs= 2024-04-12T04:08:07.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wr1kgvbxx6iZXHM0URQEmkG1qHs=', 'Content-Length': '0'} 2024-04-12T04:08:07.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b8517e0afb17854-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:08:07.364 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:08:07.365 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:08:07.365 DEBUG:boto:Method: GET 2024-04-12T04:08:07.365 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:08:07.365 DEBUG:boto:Data: 2024-04-12T04:08:07.365 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.365 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.365 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.365 DEBUG:boto:Params: {} 2024-04-12T04:08:07.365 DEBUG:boto:Token: None 2024-04-12T04:08:07.365 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj76 2024-04-12T04:08:07.365 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZasAhTFw6i/MkO/W1TCzVDdTlyo= 2024-04-12T04:08:07.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZasAhTFw6i/MkO/W1TCzVDdTlyo=', 'Content-Length': '0'} 2024-04-12T04:08:07.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038526c40f5394adb-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.368 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:08:07.368 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:08:07.368 DEBUG:boto:Method: GET 2024-04-12T04:08:07.368 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:08:07.368 DEBUG:boto:Data: 2024-04-12T04:08:07.368 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.368 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.368 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.368 DEBUG:boto:Params: {} 2024-04-12T04:08:07.368 DEBUG:boto:Token: None 2024-04-12T04:08:07.369 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj76 2024-04-12T04:08:07.369 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZasAhTFw6i/MkO/W1TCzVDdTlyo= 2024-04-12T04:08:07.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZasAhTFw6i/MkO/W1TCzVDdTlyo=', 'Content-Length': '0'} 2024-04-12T04:08:07.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044a4ba01a2cdd5f0-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.371 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.371 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.371 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:08:07.371 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:08:07.371 DEBUG:boto:Method: GET 2024-04-12T04:08:07.371 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:08:07.371 DEBUG:boto:Data: 2024-04-12T04:08:07.371 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.371 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.372 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.372 DEBUG:boto:Params: {} 2024-04-12T04:08:07.372 DEBUG:boto:Token: None 2024-04-12T04:08:07.372 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.372 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b5e412fa49ac599-006618b3a7-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.375 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.375 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.375 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.375 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:08:07.375 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:08:07.375 DEBUG:boto:Method: GET 2024-04-12T04:08:07.375 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:08:07.375 DEBUG:boto:Data: 2024-04-12T04:08:07.376 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.376 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.376 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.376 DEBUG:boto:Params: {} 2024-04-12T04:08:07.376 DEBUG:boto:Token: None 2024-04-12T04:08:07.376 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.376 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.380 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f3f1b68d3460fa5-006618b3a7-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.422 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:08:07.423 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:08:07.423 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:08:07.423 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.423 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.423 DEBUG:boto:Method: GET 2024-04-12T04:08:07.423 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.423 DEBUG:boto:Data: 2024-04-12T04:08:07.424 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.424 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.424 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.424 DEBUG:boto:Params: {} 2024-04-12T04:08:07.424 DEBUG:boto:Token: None 2024-04-12T04:08:07.424 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.424 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9TuptSavi6qeAKY754sMK8XTW/U= 2024-04-12T04:08:07.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9TuptSavi6qeAKY754sMK8XTW/U=', 'Content-Length': '0'} 2024-04-12T04:08:07.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000990ccec54b853e1f-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.427 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:08:07.427 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:08:07.427 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.427 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.427 DEBUG:boto:Method: GET 2024-04-12T04:08:07.427 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.427 DEBUG:boto:Data: 2024-04-12T04:08:07.428 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.428 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.428 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.428 DEBUG:boto:Params: {} 2024-04-12T04:08:07.428 DEBUG:boto:Token: None 2024-04-12T04:08:07.428 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.428 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9TuptSavi6qeAKY754sMK8XTW/U= 2024-04-12T04:08:07.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9TuptSavi6qeAKY754sMK8XTW/U=', 'Content-Length': '0'} 2024-04-12T04:08:07.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079707c3c2a18a3cc-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.430 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:08:07.430 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:08:07.431 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.431 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.431 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.431 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.431 DEBUG:boto:Data: 2024-04-12T04:08:07.431 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.431 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.431 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.431 DEBUG:boto:Params: {} 2024-04-12T04:08:07.431 DEBUG:boto:Token: None 2024-04-12T04:08:07.431 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.432 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KlWR1/yZLd1EDxwFsnjYzvmmuSA= 2024-04-12T04:08:07.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KlWR1/yZLd1EDxwFsnjYzvmmuSA=', 'Content-Length': '0'} 2024-04-12T04:08:07.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a342a9484e01204-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.434 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:08:07.434 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:08:07.434 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.434 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.434 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.434 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.434 DEBUG:boto:Data: 2024-04-12T04:08:07.434 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.434 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.434 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.435 DEBUG:boto:Params: {} 2024-04-12T04:08:07.435 DEBUG:boto:Token: None 2024-04-12T04:08:07.435 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.435 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KlWR1/yZLd1EDxwFsnjYzvmmuSA= 2024-04-12T04:08:07.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KlWR1/yZLd1EDxwFsnjYzvmmuSA=', 'Content-Length': '0'} 2024-04-12T04:08:07.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003843f2c088dd6a27-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:08:07.437 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:08:07.437 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:08:07.437 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.437 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.437 DEBUG:boto:Method: GET 2024-04-12T04:08:07.438 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.438 DEBUG:boto:Data: 2024-04-12T04:08:07.438 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.438 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.438 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.438 DEBUG:boto:Params: {} 2024-04-12T04:08:07.438 DEBUG:boto:Token: None 2024-04-12T04:08:07.438 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.438 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9TuptSavi6qeAKY754sMK8XTW/U= 2024-04-12T04:08:07.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9TuptSavi6qeAKY754sMK8XTW/U=', 'Content-Length': '0'} 2024-04-12T04:08:07.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d36a483e81938f-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.482 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:08:07.483 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:08:07.483 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.483 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.483 DEBUG:boto:Method: GET 2024-04-12T04:08:07.483 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.483 DEBUG:boto:Data: 2024-04-12T04:08:07.483 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.483 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.483 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.483 DEBUG:boto:Params: {} 2024-04-12T04:08:07.483 DEBUG:boto:Token: None 2024-04-12T04:08:07.484 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:08:07.484 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9TuptSavi6qeAKY754sMK8XTW/U= 2024-04-12T04:08:07.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9TuptSavi6qeAKY754sMK8XTW/U=', 'Content-Length': '0'} 2024-04-12T04:08:07.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dae723758e1d7d3-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.486 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.486 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.486 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:08:07.487 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:08:07.487 DEBUG:boto:Method: GET 2024-04-12T04:08:07.487 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:08:07.487 DEBUG:boto:Data: 2024-04-12T04:08:07.487 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.487 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.487 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.487 DEBUG:boto:Params: {} 2024-04-12T04:08:07.487 DEBUG:boto:Token: None 2024-04-12T04:08:07.487 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.488 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e4af1629b2fd057-006618b3a7-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.490 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.493 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.493 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.493 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:08:07.494 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:08:07.494 DEBUG:boto:Method: GET 2024-04-12T04:08:07.494 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:08:07.494 DEBUG:boto:Data: 2024-04-12T04:08:07.494 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.494 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.494 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.494 DEBUG:boto:Params: {} 2024-04-12T04:08:07.494 DEBUG:boto:Token: None 2024-04-12T04:08:07.494 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.495 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cb495f2b4f11d86-006618b3a7-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.498 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.499 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:08:07.499 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:08:07.499 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.499 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:08:07.499 DEBUG:boto:Data: 2024-04-12T04:08:07.499 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.499 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.499 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.499 DEBUG:boto:Params: {} 2024-04-12T04:08:07.500 DEBUG:boto:Token: None 2024-04-12T04:08:07.500 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj85 2024-04-12T04:08:07.500 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R0Fry6tnlp6m1hR8OM6QxPQZozw= 2024-04-12T04:08:07.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R0Fry6tnlp6m1hR8OM6QxPQZozw=', 'Content-Length': '0'} 2024-04-12T04:08:07.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ad90ef69567780f-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.502 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:08:07.502 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:08:07.502 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.502 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:08:07.502 DEBUG:boto:Data: 2024-04-12T04:08:07.502 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.502 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.503 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.503 DEBUG:boto:Params: {} 2024-04-12T04:08:07.503 DEBUG:boto:Token: None 2024-04-12T04:08:07.503 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj85 2024-04-12T04:08:07.503 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R0Fry6tnlp6m1hR8OM6QxPQZozw= 2024-04-12T04:08:07.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R0Fry6tnlp6m1hR8OM6QxPQZozw=', 'Content-Length': '0'} 2024-04-12T04:08:07.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dc8a6b7e9510b72-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:08:07.505 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:08:07.505 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:08:07.505 DEBUG:boto:Method: GET 2024-04-12T04:08:07.505 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:08:07.506 DEBUG:boto:Data: 2024-04-12T04:08:07.506 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.506 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.506 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.506 DEBUG:boto:Params: {} 2024-04-12T04:08:07.506 DEBUG:boto:Token: None 2024-04-12T04:08:07.506 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj85 2024-04-12T04:08:07.506 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lbvRjK//DvYWI2y0MqdOwROJHhQ= 2024-04-12T04:08:07.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lbvRjK//DvYWI2y0MqdOwROJHhQ=', 'Content-Length': '0'} 2024-04-12T04:08:07.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040b7555305a96bf8-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.508 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:08:07.509 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:08:07.509 DEBUG:boto:Method: GET 2024-04-12T04:08:07.509 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:08:07.509 DEBUG:boto:Data: 2024-04-12T04:08:07.509 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.509 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.509 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.509 DEBUG:boto:Params: {} 2024-04-12T04:08:07.509 DEBUG:boto:Token: None 2024-04-12T04:08:07.509 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj85 2024-04-12T04:08:07.509 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lbvRjK//DvYWI2y0MqdOwROJHhQ= 2024-04-12T04:08:07.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lbvRjK//DvYWI2y0MqdOwROJHhQ=', 'Content-Length': '0'} 2024-04-12T04:08:07.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089b9311793a9901a-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.554 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.555 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.555 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:08:07.555 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:08:07.555 DEBUG:boto:Method: GET 2024-04-12T04:08:07.555 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:08:07.555 DEBUG:boto:Data: 2024-04-12T04:08:07.555 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.555 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.555 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.555 DEBUG:boto:Params: {} 2024-04-12T04:08:07.556 DEBUG:boto:Token: None 2024-04-12T04:08:07.556 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.556 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000873fd0d1404fa5af-006618b3a7-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.558 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.559 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.559 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.559 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:08:07.559 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:08:07.559 DEBUG:boto:Method: GET 2024-04-12T04:08:07.559 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:08:07.559 DEBUG:boto:Data: 2024-04-12T04:08:07.559 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.560 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.560 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.560 DEBUG:boto:Params: {} 2024-04-12T04:08:07.560 DEBUG:boto:Token: None 2024-04-12T04:08:07.560 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.560 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071e18e37d6fb813e-006618b3a7-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.564 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:08:07.564 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:08:07.564 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:08:07.564 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.565 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.565 DEBUG:boto:Method: GET 2024-04-12T04:08:07.565 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.565 DEBUG:boto:Data: 2024-04-12T04:08:07.565 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.565 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.565 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.565 DEBUG:boto:Params: {} 2024-04-12T04:08:07.565 DEBUG:boto:Token: None 2024-04-12T04:08:07.565 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.566 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5ZkTg0O+okvg4BMx+dx7xUP7CmA= 2024-04-12T04:08:07.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5ZkTg0O+okvg4BMx+dx7xUP7CmA=', 'Content-Length': '0'} 2024-04-12T04:08:07.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6bfe17203d85030-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.610 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:08:07.610 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:08:07.611 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.611 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.611 DEBUG:boto:Method: GET 2024-04-12T04:08:07.611 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.611 DEBUG:boto:Data: 2024-04-12T04:08:07.611 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.611 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.611 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.611 DEBUG:boto:Params: {} 2024-04-12T04:08:07.611 DEBUG:boto:Token: None 2024-04-12T04:08:07.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.612 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5ZkTg0O+okvg4BMx+dx7xUP7CmA= 2024-04-12T04:08:07.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5ZkTg0O+okvg4BMx+dx7xUP7CmA=', 'Content-Length': '0'} 2024-04-12T04:08:07.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007021e424c18c9cf5-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.614 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:08:07.614 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:08:07.614 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.614 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.614 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.614 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.614 DEBUG:boto:Data: 2024-04-12T04:08:07.615 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.615 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.615 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.615 DEBUG:boto:Params: {} 2024-04-12T04:08:07.615 DEBUG:boto:Token: None 2024-04-12T04:08:07.615 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.615 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VmVExSuk4A0i6mGQ5F7yTfd4yic= 2024-04-12T04:08:07.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VmVExSuk4A0i6mGQ5F7yTfd4yic=', 'Content-Length': '0'} 2024-04-12T04:08:07.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d30be05e1566c83-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.617 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:08:07.618 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:08:07.618 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.618 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.618 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.618 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.618 DEBUG:boto:Data: 2024-04-12T04:08:07.618 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.618 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.618 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.618 DEBUG:boto:Params: {} 2024-04-12T04:08:07.619 DEBUG:boto:Token: None 2024-04-12T04:08:07.619 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.619 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VmVExSuk4A0i6mGQ5F7yTfd4yic= 2024-04-12T04:08:07.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VmVExSuk4A0i6mGQ5F7yTfd4yic=', 'Content-Length': '0'} 2024-04-12T04:08:07.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b6fa8a61195165e-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:08:07.621 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:08:07.621 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:08:07.621 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.621 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.621 DEBUG:boto:Method: GET 2024-04-12T04:08:07.621 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.621 DEBUG:boto:Data: 2024-04-12T04:08:07.622 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.622 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.622 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.622 DEBUG:boto:Params: {} 2024-04-12T04:08:07.622 DEBUG:boto:Token: None 2024-04-12T04:08:07.622 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.622 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5ZkTg0O+okvg4BMx+dx7xUP7CmA= 2024-04-12T04:08:07.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5ZkTg0O+okvg4BMx+dx7xUP7CmA=', 'Content-Length': '0'} 2024-04-12T04:08:07.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002727a7061bef0531-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.624 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:08:07.624 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:08:07.624 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.625 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.625 DEBUG:boto:Method: GET 2024-04-12T04:08:07.625 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.625 DEBUG:boto:Data: 2024-04-12T04:08:07.625 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.625 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.625 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.625 DEBUG:boto:Params: {} 2024-04-12T04:08:07.625 DEBUG:boto:Token: None 2024-04-12T04:08:07.625 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:08:07.625 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5ZkTg0O+okvg4BMx+dx7xUP7CmA= 2024-04-12T04:08:07.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5ZkTg0O+okvg4BMx+dx7xUP7CmA=', 'Content-Length': '0'} 2024-04-12T04:08:07.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009ea185c28226a69-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.628 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.628 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.628 DEBUG:boto:path=/dkyges-61/?prefix=obj87 2024-04-12T04:08:07.628 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj87 2024-04-12T04:08:07.628 DEBUG:boto:Method: GET 2024-04-12T04:08:07.628 DEBUG:boto:Path: /dkyges-61/?prefix=obj87 2024-04-12T04:08:07.628 DEBUG:boto:Data: 2024-04-12T04:08:07.628 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.629 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.629 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.629 DEBUG:boto:Params: {} 2024-04-12T04:08:07.629 DEBUG:boto:Token: None 2024-04-12T04:08:07.629 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.629 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef543d0349963523-006618b3a7-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.631 DEBUG:boto:b'dkyges-61obj871000falseobj872024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.632 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.632 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.632 DEBUG:boto:path=/dkyges-61/?prefix=obj87 2024-04-12T04:08:07.632 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj87 2024-04-12T04:08:07.632 DEBUG:boto:Method: GET 2024-04-12T04:08:07.632 DEBUG:boto:Path: /dkyges-61/?prefix=obj87 2024-04-12T04:08:07.632 DEBUG:boto:Data: 2024-04-12T04:08:07.632 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.633 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.633 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.633 DEBUG:boto:Params: {} 2024-04-12T04:08:07.633 DEBUG:boto:Token: None 2024-04-12T04:08:07.633 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.633 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f1b4ceef644f8d4-006618b3a7-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.637 DEBUG:boto:b'dkyges-61obj871000falseobj872024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.638 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:08:07.638 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:08:07.638 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.638 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:08:07.638 DEBUG:boto:Data: 2024-04-12T04:08:07.638 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.638 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.638 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.638 DEBUG:boto:Params: {} 2024-04-12T04:08:07.639 DEBUG:boto:Token: None 2024-04-12T04:08:07.639 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj87 2024-04-12T04:08:07.639 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a8MnaFlZ/tMBBk6dk0/j/fpTk0s= 2024-04-12T04:08:07.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a8MnaFlZ/tMBBk6dk0/j/fpTk0s=', 'Content-Length': '0'} 2024-04-12T04:08:07.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d4e92326d86dc47-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.641 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:08:07.641 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:08:07.641 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.641 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:08:07.641 DEBUG:boto:Data: 2024-04-12T04:08:07.641 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.641 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.641 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.641 DEBUG:boto:Params: {} 2024-04-12T04:08:07.641 DEBUG:boto:Token: None 2024-04-12T04:08:07.642 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj87 2024-04-12T04:08:07.642 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a8MnaFlZ/tMBBk6dk0/j/fpTk0s= 2024-04-12T04:08:07.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a8MnaFlZ/tMBBk6dk0/j/fpTk0s=', 'Content-Length': '0'} 2024-04-12T04:08:07.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000333d5100302d21da-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:08:07.644 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:08:07.644 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:08:07.644 DEBUG:boto:Method: GET 2024-04-12T04:08:07.644 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:08:07.644 DEBUG:boto:Data: 2024-04-12T04:08:07.645 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.645 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.645 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.645 DEBUG:boto:Params: {} 2024-04-12T04:08:07.645 DEBUG:boto:Token: None 2024-04-12T04:08:07.645 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj87 2024-04-12T04:08:07.645 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y925wWFvmU/UGvPPdoWhI13YFdk= 2024-04-12T04:08:07.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y925wWFvmU/UGvPPdoWhI13YFdk=', 'Content-Length': '0'} 2024-04-12T04:08:07.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000b4110558c2bcd0-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.647 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:08:07.647 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:08:07.647 DEBUG:boto:Method: GET 2024-04-12T04:08:07.648 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:08:07.648 DEBUG:boto:Data: 2024-04-12T04:08:07.648 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.648 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.648 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.648 DEBUG:boto:Params: {} 2024-04-12T04:08:07.648 DEBUG:boto:Token: None 2024-04-12T04:08:07.648 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj87 2024-04-12T04:08:07.648 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y925wWFvmU/UGvPPdoWhI13YFdk= 2024-04-12T04:08:07.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y925wWFvmU/UGvPPdoWhI13YFdk=', 'Content-Length': '0'} 2024-04-12T04:08:07.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005775908270f1b91-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.690 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.690 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.690 DEBUG:boto:path=/dkyges-61/?prefix=obj87 2024-04-12T04:08:07.691 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj87 2024-04-12T04:08:07.691 DEBUG:boto:Method: GET 2024-04-12T04:08:07.691 DEBUG:boto:Path: /dkyges-61/?prefix=obj87 2024-04-12T04:08:07.691 DEBUG:boto:Data: 2024-04-12T04:08:07.691 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.691 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.691 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.691 DEBUG:boto:Params: {} 2024-04-12T04:08:07.691 DEBUG:boto:Token: None 2024-04-12T04:08:07.691 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.692 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce9d32172a15a411-006618b3a7-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.697 DEBUG:boto:b'dkyges-61obj871000falseobj872024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.698 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.698 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.698 DEBUG:boto:path=/dkyges-61/?prefix=obj87 2024-04-12T04:08:07.698 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj87 2024-04-12T04:08:07.698 DEBUG:boto:Method: GET 2024-04-12T04:08:07.698 DEBUG:boto:Path: /dkyges-61/?prefix=obj87 2024-04-12T04:08:07.698 DEBUG:boto:Data: 2024-04-12T04:08:07.698 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.698 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.698 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.699 DEBUG:boto:Params: {} 2024-04-12T04:08:07.699 DEBUG:boto:Token: None 2024-04-12T04:08:07.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.699 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013da568bf766fe7b-006618b3a7-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.703 DEBUG:boto:b'dkyges-61obj871000falseobj872024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:08:07.703 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:08:07.703 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:08:07.704 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.704 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.704 DEBUG:boto:Method: GET 2024-04-12T04:08:07.704 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.704 DEBUG:boto:Data: 2024-04-12T04:08:07.704 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.704 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.704 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.704 DEBUG:boto:Params: {} 2024-04-12T04:08:07.704 DEBUG:boto:Token: None 2024-04-12T04:08:07.704 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.705 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hXmbyjk/RjUlTLMoeNWs5wgAJOU= 2024-04-12T04:08:07.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hXmbyjk/RjUlTLMoeNWs5wgAJOU=', 'Content-Length': '0'} 2024-04-12T04:08:07.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb913217049d0d3b-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.750 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:08:07.751 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:08:07.751 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.751 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.751 DEBUG:boto:Method: GET 2024-04-12T04:08:07.751 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.751 DEBUG:boto:Data: 2024-04-12T04:08:07.751 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.751 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.751 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.751 DEBUG:boto:Params: {} 2024-04-12T04:08:07.751 DEBUG:boto:Token: None 2024-04-12T04:08:07.752 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.752 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hXmbyjk/RjUlTLMoeNWs5wgAJOU= 2024-04-12T04:08:07.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hXmbyjk/RjUlTLMoeNWs5wgAJOU=', 'Content-Length': '0'} 2024-04-12T04:08:07.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037899f007a25f3e8-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.754 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:08:07.754 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:08:07.754 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.754 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.755 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.755 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.755 DEBUG:boto:Data: 2024-04-12T04:08:07.755 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.755 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.755 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.755 DEBUG:boto:Params: {} 2024-04-12T04:08:07.755 DEBUG:boto:Token: None 2024-04-12T04:08:07.755 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.755 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8IU/QY7DjgAvuzgeqkA8t74mzX4= 2024-04-12T04:08:07.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8IU/QY7DjgAvuzgeqkA8t74mzX4=', 'Content-Length': '0'} 2024-04-12T04:08:07.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087829a4c00b7d482-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.757 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:08:07.757 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:08:07.758 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.758 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.758 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.758 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.758 DEBUG:boto:Data: 2024-04-12T04:08:07.758 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.758 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.758 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.758 DEBUG:boto:Params: {} 2024-04-12T04:08:07.758 DEBUG:boto:Token: None 2024-04-12T04:08:07.759 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.759 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8IU/QY7DjgAvuzgeqkA8t74mzX4= 2024-04-12T04:08:07.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8IU/QY7DjgAvuzgeqkA8t74mzX4=', 'Content-Length': '0'} 2024-04-12T04:08:07.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e04e4181e227ec25-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:08:07.761 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:08:07.761 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:08:07.761 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.761 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.761 DEBUG:boto:Method: GET 2024-04-12T04:08:07.761 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.762 DEBUG:boto:Data: 2024-04-12T04:08:07.762 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.762 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.762 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.762 DEBUG:boto:Params: {} 2024-04-12T04:08:07.762 DEBUG:boto:Token: None 2024-04-12T04:08:07.762 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.762 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hXmbyjk/RjUlTLMoeNWs5wgAJOU= 2024-04-12T04:08:07.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hXmbyjk/RjUlTLMoeNWs5wgAJOU=', 'Content-Length': '0'} 2024-04-12T04:08:07.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000090b2dfa55c28330-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.764 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:08:07.765 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:08:07.765 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.765 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.765 DEBUG:boto:Method: GET 2024-04-12T04:08:07.765 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.765 DEBUG:boto:Data: 2024-04-12T04:08:07.765 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.765 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.765 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.765 DEBUG:boto:Params: {} 2024-04-12T04:08:07.765 DEBUG:boto:Token: None 2024-04-12T04:08:07.766 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:08:07.766 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hXmbyjk/RjUlTLMoeNWs5wgAJOU= 2024-04-12T04:08:07.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hXmbyjk/RjUlTLMoeNWs5wgAJOU=', 'Content-Length': '0'} 2024-04-12T04:08:07.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9c6570c11e40f65-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.768 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.768 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.769 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:08:07.769 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:08:07.769 DEBUG:boto:Method: GET 2024-04-12T04:08:07.769 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:08:07.769 DEBUG:boto:Data: 2024-04-12T04:08:07.769 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.769 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.769 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.769 DEBUG:boto:Params: {} 2024-04-12T04:08:07.769 DEBUG:boto:Token: None 2024-04-12T04:08:07.769 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.770 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b6cc3dc95ab12aa-006618b3a7-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.772 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.772 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.772 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.773 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:08:07.773 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:08:07.773 DEBUG:boto:Method: GET 2024-04-12T04:08:07.773 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:08:07.773 DEBUG:boto:Data: 2024-04-12T04:08:07.773 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.773 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.773 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.773 DEBUG:boto:Params: {} 2024-04-12T04:08:07.773 DEBUG:boto:Token: None 2024-04-12T04:08:07.774 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.774 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b283280ab7862dc-006618b3a7-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.778 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.778 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:08:07.778 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:08:07.778 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.778 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:08:07.779 DEBUG:boto:Data: 2024-04-12T04:08:07.779 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.779 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.779 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.779 DEBUG:boto:Params: {} 2024-04-12T04:08:07.779 DEBUG:boto:Token: None 2024-04-12T04:08:07.779 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj92 2024-04-12T04:08:07.779 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HjxOjqJH+XyoLQumpJpI4wWlDfo= 2024-04-12T04:08:07.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HjxOjqJH+XyoLQumpJpI4wWlDfo=', 'Content-Length': '0'} 2024-04-12T04:08:07.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f92782253d22c92-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.781 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:08:07.781 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:08:07.782 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.782 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:08:07.782 DEBUG:boto:Data: 2024-04-12T04:08:07.782 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.782 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.782 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.782 DEBUG:boto:Params: {} 2024-04-12T04:08:07.782 DEBUG:boto:Token: None 2024-04-12T04:08:07.782 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj92 2024-04-12T04:08:07.783 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HjxOjqJH+XyoLQumpJpI4wWlDfo= 2024-04-12T04:08:07.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HjxOjqJH+XyoLQumpJpI4wWlDfo=', 'Content-Length': '0'} 2024-04-12T04:08:07.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017f941bffab29bbd-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:08:07.785 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:08:07.785 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:08:07.785 DEBUG:boto:Method: GET 2024-04-12T04:08:07.785 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:08:07.785 DEBUG:boto:Data: 2024-04-12T04:08:07.785 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.785 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.785 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.786 DEBUG:boto:Params: {} 2024-04-12T04:08:07.786 DEBUG:boto:Token: None 2024-04-12T04:08:07.786 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj92 2024-04-12T04:08:07.786 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WFUMk7mSaV4X6BNoVspod89koTo= 2024-04-12T04:08:07.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WFUMk7mSaV4X6BNoVspod89koTo=', 'Content-Length': '0'} 2024-04-12T04:08:07.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003155b8001fefe5e6-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.788 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:08:07.788 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:08:07.788 DEBUG:boto:Method: GET 2024-04-12T04:08:07.788 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:08:07.788 DEBUG:boto:Data: 2024-04-12T04:08:07.788 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.789 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.789 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.789 DEBUG:boto:Params: {} 2024-04-12T04:08:07.789 DEBUG:boto:Token: None 2024-04-12T04:08:07.789 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj92 2024-04-12T04:08:07.789 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WFUMk7mSaV4X6BNoVspod89koTo= 2024-04-12T04:08:07.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WFUMk7mSaV4X6BNoVspod89koTo=', 'Content-Length': '0'} 2024-04-12T04:08:07.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3991a032293c2dd-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.791 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.791 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.792 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:08:07.792 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:08:07.792 DEBUG:boto:Method: GET 2024-04-12T04:08:07.792 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:08:07.792 DEBUG:boto:Data: 2024-04-12T04:08:07.792 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.792 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.792 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.792 DEBUG:boto:Params: {} 2024-04-12T04:08:07.792 DEBUG:boto:Token: None 2024-04-12T04:08:07.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.793 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a2b98d09fcee4f9-006618b3a7-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.795 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.796 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.796 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.796 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:08:07.796 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:08:07.796 DEBUG:boto:Method: GET 2024-04-12T04:08:07.796 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:08:07.796 DEBUG:boto:Data: 2024-04-12T04:08:07.796 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.796 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.796 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.797 DEBUG:boto:Params: {} 2024-04-12T04:08:07.797 DEBUG:boto:Token: None 2024-04-12T04:08:07.797 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.797 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf1cf6db10b47f2c-006618b3a7-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.801 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:08:07.801 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:08:07.801 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:08:07.802 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.802 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.802 DEBUG:boto:Method: GET 2024-04-12T04:08:07.802 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.802 DEBUG:boto:Data: 2024-04-12T04:08:07.802 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.802 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.802 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.802 DEBUG:boto:Params: {} 2024-04-12T04:08:07.802 DEBUG:boto:Token: None 2024-04-12T04:08:07.803 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.803 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CKUvYBr/yAjNowCJ5cNc+eCpJ0s= 2024-04-12T04:08:07.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CKUvYBr/yAjNowCJ5cNc+eCpJ0s=', 'Content-Length': '0'} 2024-04-12T04:08:07.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f37e99be2b51f3c-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.846 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:08:07.846 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:08:07.846 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.847 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.847 DEBUG:boto:Method: GET 2024-04-12T04:08:07.847 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.847 DEBUG:boto:Data: 2024-04-12T04:08:07.847 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.847 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.847 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.847 DEBUG:boto:Params: {} 2024-04-12T04:08:07.847 DEBUG:boto:Token: None 2024-04-12T04:08:07.847 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.848 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CKUvYBr/yAjNowCJ5cNc+eCpJ0s= 2024-04-12T04:08:07.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CKUvYBr/yAjNowCJ5cNc+eCpJ0s=', 'Content-Length': '0'} 2024-04-12T04:08:07.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe758db33581a788-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.850 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:08:07.850 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:08:07.850 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.850 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.850 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.850 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.851 DEBUG:boto:Data: 2024-04-12T04:08:07.851 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.851 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.851 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.851 DEBUG:boto:Params: {} 2024-04-12T04:08:07.851 DEBUG:boto:Token: None 2024-04-12T04:08:07.851 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.851 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ufvdx93xcipbO7eflyKN6ISI4cc= 2024-04-12T04:08:07.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ufvdx93xcipbO7eflyKN6ISI4cc=', 'Content-Length': '0'} 2024-04-12T04:08:07.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009c39ba8eb32c584-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.853 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:08:07.854 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:08:07.854 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.854 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.854 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.854 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.854 DEBUG:boto:Data: 2024-04-12T04:08:07.854 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.854 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.854 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.854 DEBUG:boto:Params: {} 2024-04-12T04:08:07.855 DEBUG:boto:Token: None 2024-04-12T04:08:07.855 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.855 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ufvdx93xcipbO7eflyKN6ISI4cc= 2024-04-12T04:08:07.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ufvdx93xcipbO7eflyKN6ISI4cc=', 'Content-Length': '0'} 2024-04-12T04:08:07.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa086a0894b4d28f-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:08:07.857 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:08:07.857 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:08:07.857 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.857 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.858 DEBUG:boto:Method: GET 2024-04-12T04:08:07.858 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.858 DEBUG:boto:Data: 2024-04-12T04:08:07.858 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.858 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.858 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.858 DEBUG:boto:Params: {} 2024-04-12T04:08:07.858 DEBUG:boto:Token: None 2024-04-12T04:08:07.858 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.858 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CKUvYBr/yAjNowCJ5cNc+eCpJ0s= 2024-04-12T04:08:07.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CKUvYBr/yAjNowCJ5cNc+eCpJ0s=', 'Content-Length': '0'} 2024-04-12T04:08:07.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3bef368abc39b44-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.861 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:08:07.861 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:08:07.861 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.861 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.861 DEBUG:boto:Method: GET 2024-04-12T04:08:07.861 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.861 DEBUG:boto:Data: 2024-04-12T04:08:07.861 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.861 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.861 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.862 DEBUG:boto:Params: {} 2024-04-12T04:08:07.862 DEBUG:boto:Token: None 2024-04-12T04:08:07.862 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:08:07.862 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CKUvYBr/yAjNowCJ5cNc+eCpJ0s= 2024-04-12T04:08:07.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CKUvYBr/yAjNowCJ5cNc+eCpJ0s=', 'Content-Length': '0'} 2024-04-12T04:08:07.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d714aed7c199e30-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.864 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.864 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.864 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:08:07.864 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:08:07.865 DEBUG:boto:Method: GET 2024-04-12T04:08:07.865 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:08:07.865 DEBUG:boto:Data: 2024-04-12T04:08:07.865 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.865 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.865 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.865 DEBUG:boto:Params: {} 2024-04-12T04:08:07.865 DEBUG:boto:Token: None 2024-04-12T04:08:07.865 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.865 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db59d5b7e05860ac-006618b3a7-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.868 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.869 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.869 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.869 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:08:07.869 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:08:07.869 DEBUG:boto:Method: GET 2024-04-12T04:08:07.869 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:08:07.869 DEBUG:boto:Data: 2024-04-12T04:08:07.869 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.870 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.870 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.870 DEBUG:boto:Params: {} 2024-04-12T04:08:07.870 DEBUG:boto:Token: None 2024-04-12T04:08:07.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acdb22b89a1f3153-006618b3a7-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.874 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.875 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:08:07.875 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:08:07.875 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.875 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:08:07.875 DEBUG:boto:Data: 2024-04-12T04:08:07.875 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.875 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.875 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.875 DEBUG:boto:Params: {} 2024-04-12T04:08:07.876 DEBUG:boto:Token: None 2024-04-12T04:08:07.876 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj95 2024-04-12T04:08:07.876 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f9YmpuXkVKiIn7+okMLPXrXgaes= 2024-04-12T04:08:07.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f9YmpuXkVKiIn7+okMLPXrXgaes=', 'Content-Length': '0'} 2024-04-12T04:08:07.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062f2ca9157d86ea6-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.878 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:08:07.878 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:08:07.878 DEBUG:boto:Method: HEAD 2024-04-12T04:08:07.878 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:08:07.878 DEBUG:boto:Data: 2024-04-12T04:08:07.878 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.879 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.879 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.879 DEBUG:boto:Params: {} 2024-04-12T04:08:07.879 DEBUG:boto:Token: None 2024-04-12T04:08:07.879 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj95 2024-04-12T04:08:07.879 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f9YmpuXkVKiIn7+okMLPXrXgaes= 2024-04-12T04:08:07.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f9YmpuXkVKiIn7+okMLPXrXgaes=', 'Content-Length': '0'} 2024-04-12T04:08:07.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000214d228198f4e363-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:08:07.881 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:08:07.881 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:08:07.881 DEBUG:boto:Method: GET 2024-04-12T04:08:07.881 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:08:07.882 DEBUG:boto:Data: 2024-04-12T04:08:07.882 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.882 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.882 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.882 DEBUG:boto:Params: {} 2024-04-12T04:08:07.882 DEBUG:boto:Token: None 2024-04-12T04:08:07.882 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj95 2024-04-12T04:08:07.882 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7JPbzpjyGmX7DBGNHydpvScv5iI= 2024-04-12T04:08:07.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7JPbzpjyGmX7DBGNHydpvScv5iI=', 'Content-Length': '0'} 2024-04-12T04:08:07.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2dd329b24d95a95-006618b3a7-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.884 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:08:07.884 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:08:07.885 DEBUG:boto:Method: GET 2024-04-12T04:08:07.885 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:08:07.885 DEBUG:boto:Data: 2024-04-12T04:08:07.885 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.885 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.885 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.885 DEBUG:boto:Params: {} 2024-04-12T04:08:07.885 DEBUG:boto:Token: None 2024-04-12T04:08:07.885 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/obj95 2024-04-12T04:08:07.885 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7JPbzpjyGmX7DBGNHydpvScv5iI= 2024-04-12T04:08:07.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7JPbzpjyGmX7DBGNHydpvScv5iI=', 'Content-Length': '0'} 2024-04-12T04:08:07.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a1f3a83e7f8c230-006618b3a7-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.930 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.931 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.931 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:08:07.931 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:08:07.931 DEBUG:boto:Method: GET 2024-04-12T04:08:07.931 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:08:07.931 DEBUG:boto:Data: 2024-04-12T04:08:07.931 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.931 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:08:07.931 DEBUG:boto:Port: 8001 2024-04-12T04:08:07.931 DEBUG:boto:Params: {} 2024-04-12T04:08:07.931 DEBUG:boto:Token: None 2024-04-12T04:08:07.932 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.932 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dc984c41b32030a-006618b3a7-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.934 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.935 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:07.935 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:07.935 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:08:07.935 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:08:07.935 DEBUG:boto:Method: GET 2024-04-12T04:08:07.935 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:08:07.935 DEBUG:boto:Data: 2024-04-12T04:08:07.936 DEBUG:boto:Headers: {} 2024-04-12T04:08:07.936 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:07.936 DEBUG:boto:Port: 8000 2024-04-12T04:08:07.936 DEBUG:boto:Params: {} 2024-04-12T04:08:07.936 DEBUG:boto:Token: None 2024-04-12T04:08:07.936 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:08:07 GMT /dkyges-61/ 2024-04-12T04:08:07.936 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0= 2024-04-12T04:08:07.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:umjGV8iILdPM2XmZ4EANdLIcUG0=', 'Content-Length': '0'} 2024-04-12T04:08:07.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000817cb7e6c02a6c7d-006618b3a7-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:08:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:07.941 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:08:07.941 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-61 zones={a2, a1} 2024-04-12T04:08:07.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:08:07.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:08:07.942 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:08:08.007 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:08:08.007 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:08:08.067 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:08:08.068 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-61 2024-04-12T04:08:08.080 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 25 2024-04-12T04:08:08.149 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:08.149 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:08.149 DEBUG:boto:Method: HEAD 2024-04-12T04:08:08.149 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:08:08.149 DEBUG:boto:Data: 2024-04-12T04:08:08.150 DEBUG:boto:Headers: {} 2024-04-12T04:08:08.150 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:08.150 DEBUG:boto:Port: 8000 2024-04-12T04:08:08.150 DEBUG:boto:Params: {} 2024-04-12T04:08:08.150 DEBUG:boto:Token: None 2024-04-12T04:08:08.150 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:08 GMT /dkyges-61/ 2024-04-12T04:08:08.150 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wM38Wf2KDF8n5ENLKXgssEpSEMY= 2024-04-12T04:08:08.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wM38Wf2KDF8n5ENLKXgssEpSEMY=', 'Content-Length': '0'} 2024-04-12T04:08:08.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c5dc5df4f5e4881-006618b3a8-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:08.154 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:08:08.154 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:08:08.154 DEBUG:boto:Method: PUT 2024-04-12T04:08:08.154 DEBUG:boto:Path: /dkyges-61/obj31 2024-04-12T04:08:08.154 DEBUG:boto:Data: 2024-04-12T04:08:08.155 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:08:08.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:08.155 DEBUG:boto:Port: 8000 2024-04-12T04:08:08.155 DEBUG:boto:Params: {} 2024-04-12T04:08:08.155 DEBUG:boto:Token: None 2024-04-12T04:08:08.155 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:08:08 GMT /dkyges-61/obj31 2024-04-12T04:08:08.155 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nVp0u1ggRXedzrrqdeCNiBtGYdQ= 2024-04-12T04:08:08.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:08:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nVp0u1ggRXedzrrqdeCNiBtGYdQ='} 2024-04-12T04:08:08.161 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065e321d1adf7abcf-006618b3a8-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:08.162 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:08.162 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:08.162 DEBUG:boto:Method: HEAD 2024-04-12T04:08:08.162 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:08:08.162 DEBUG:boto:Data: 2024-04-12T04:08:08.162 DEBUG:boto:Headers: {} 2024-04-12T04:08:08.162 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:08.162 DEBUG:boto:Port: 8000 2024-04-12T04:08:08.162 DEBUG:boto:Params: {} 2024-04-12T04:08:08.162 DEBUG:boto:Token: None 2024-04-12T04:08:08.163 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:08 GMT /dkyges-61/ 2024-04-12T04:08:08.163 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wM38Wf2KDF8n5ENLKXgssEpSEMY= 2024-04-12T04:08:08.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wM38Wf2KDF8n5ENLKXgssEpSEMY=', 'Content-Length': '0'} 2024-04-12T04:08:08.166 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055fb941617816f26-006618b3a8-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:08.167 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:08:08.167 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:08:08.167 DEBUG:boto:Method: PUT 2024-04-12T04:08:08.167 DEBUG:boto:Path: /dkyges-61/obj62 2024-04-12T04:08:08.167 DEBUG:boto:Data: 2024-04-12T04:08:08.167 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:08:08.167 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:08.167 DEBUG:boto:Port: 8000 2024-04-12T04:08:08.167 DEBUG:boto:Params: {} 2024-04-12T04:08:08.168 DEBUG:boto:Token: None 2024-04-12T04:08:08.168 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:08:08 GMT /dkyges-61/obj62 2024-04-12T04:08:08.168 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:16QOJPxwAmDwTIVm7iZOaHC1paA= 2024-04-12T04:08:08.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:08:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:16QOJPxwAmDwTIVm7iZOaHC1paA='} 2024-04-12T04:08:08.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aea3df8ad6c57f81-006618b3a8-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:08.211 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:08.211 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:08.211 DEBUG:boto:Method: HEAD 2024-04-12T04:08:08.211 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:08:08.211 DEBUG:boto:Data: 2024-04-12T04:08:08.211 DEBUG:boto:Headers: {} 2024-04-12T04:08:08.211 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:08.211 DEBUG:boto:Port: 8000 2024-04-12T04:08:08.211 DEBUG:boto:Params: {} 2024-04-12T04:08:08.212 DEBUG:boto:Token: None 2024-04-12T04:08:08.212 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:08 GMT /dkyges-61/ 2024-04-12T04:08:08.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wM38Wf2KDF8n5ENLKXgssEpSEMY= 2024-04-12T04:08:08.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wM38Wf2KDF8n5ENLKXgssEpSEMY=', 'Content-Length': '0'} 2024-04-12T04:08:08.215 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000108256adf8086ae1-006618b3a8-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:08.216 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:08:08.216 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:08:08.216 DEBUG:boto:Method: PUT 2024-04-12T04:08:08.216 DEBUG:boto:Path: /dkyges-61/obj93 2024-04-12T04:08:08.216 DEBUG:boto:Data: 2024-04-12T04:08:08.216 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:08:08.216 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:08.216 DEBUG:boto:Port: 8000 2024-04-12T04:08:08.216 DEBUG:boto:Params: {} 2024-04-12T04:08:08.216 DEBUG:boto:Token: None 2024-04-12T04:08:08.217 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:08:08 GMT /dkyges-61/obj93 2024-04-12T04:08:08.217 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kK6eLD5M+0cs9SPiIA9MtNuyfYU= 2024-04-12T04:08:08.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:08:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kK6eLD5M+0cs9SPiIA9MtNuyfYU='} 2024-04-12T04:08:08.223 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004713a7cb1a8cb12e-006618b3a8-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:08.223 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:08.223 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:08.223 DEBUG:boto:Method: HEAD 2024-04-12T04:08:08.224 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:08:08.224 DEBUG:boto:Data: 2024-04-12T04:08:08.224 DEBUG:boto:Headers: {} 2024-04-12T04:08:08.224 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:08.224 DEBUG:boto:Port: 8000 2024-04-12T04:08:08.224 DEBUG:boto:Params: {} 2024-04-12T04:08:08.224 DEBUG:boto:Token: None 2024-04-12T04:08:08.224 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:08 GMT /dkyges-61/ 2024-04-12T04:08:08.224 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wM38Wf2KDF8n5ENLKXgssEpSEMY= 2024-04-12T04:08:08.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wM38Wf2KDF8n5ENLKXgssEpSEMY=', 'Content-Length': '0'} 2024-04-12T04:08:08.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbe1c45b56c16973-006618b3a8-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:08.228 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:08:08.228 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:08:08.228 DEBUG:boto:Method: PUT 2024-04-12T04:08:08.228 DEBUG:boto:Path: /dkyges-61/obj124 2024-04-12T04:08:08.228 DEBUG:boto:Data: 2024-04-12T04:08:08.229 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:08:08.229 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:08.229 DEBUG:boto:Port: 8000 2024-04-12T04:08:08.229 DEBUG:boto:Params: {} 2024-04-12T04:08:08.229 DEBUG:boto:Token: None 2024-04-12T04:08:08.229 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:08:08 GMT /dkyges-61/obj124 2024-04-12T04:08:08.229 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qEV7inEZDf9zzUKWN2z8d+U/aiQ= 2024-04-12T04:08:08.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:08:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qEV7inEZDf9zzUKWN2z8d+U/aiQ='} 2024-04-12T04:08:08.274 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009485e32671f2c084-006618b3a8-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:08.274 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:08:08.275 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:08:08.275 DEBUG:boto:Method: HEAD 2024-04-12T04:08:08.275 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:08:08.275 DEBUG:boto:Data: 2024-04-12T04:08:08.275 DEBUG:boto:Headers: {} 2024-04-12T04:08:08.275 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:08.275 DEBUG:boto:Port: 8000 2024-04-12T04:08:08.275 DEBUG:boto:Params: {} 2024-04-12T04:08:08.275 DEBUG:boto:Token: None 2024-04-12T04:08:08.275 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:08:08 GMT /dkyges-61/ 2024-04-12T04:08:08.275 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wM38Wf2KDF8n5ENLKXgssEpSEMY= 2024-04-12T04:08:08.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:08:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wM38Wf2KDF8n5ENLKXgssEpSEMY=', 'Content-Length': '0'} 2024-04-12T04:08:08.279 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5d5c85d55573fc4-006618b3a8-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:08.279 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:08:08.279 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:08:08.280 DEBUG:boto:Method: PUT 2024-04-12T04:08:08.280 DEBUG:boto:Path: /dkyges-61/obj155 2024-04-12T04:08:08.280 DEBUG:boto:Data: 2024-04-12T04:08:08.280 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:08:08.280 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:08:08.280 DEBUG:boto:Port: 8000 2024-04-12T04:08:08.280 DEBUG:boto:Params: {} 2024-04-12T04:08:08.281 DEBUG:boto:Token: None 2024-04-12T04:08:08.281 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:08:08 GMT /dkyges-61/obj155 2024-04-12T04:08:08.281 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HqlF085yfsxHfYB3BEZJacxD/40= 2024-04-12T04:08:08.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:08:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HqlF085yfsxHfYB3BEZJacxD/40='} 2024-04-12T04:08:08.288 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e325f334147a27b-006618b3a8-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:08:08.288 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-61 2024-04-12T04:08:08.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:08:08.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:08:08.288 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:08:08.356 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:08:08.357 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:08:09.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:08:09.417+0000 7fbe8e551780 1 waiting to reach latest gen 5, on 4.. 2024-04-12T04:08:14.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:08:14.418+0000 7fbe8e551780 1 waiting to reach latest gen 5, on 4.. 2024-04-12T04:08:19.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:08:19.418+0000 7fbe8e551780 1 waiting to reach latest gen 5, on 4.. 2024-04-12T04:08:24.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:08:24.421+0000 7fbe8e551780 1 waiting for incremental sync to catch up: 2024-04-12T04:08:24.406 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , 00000000002.132.6, , , , 00000000001.61.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:08:24.406 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , 00000000002.132.6, , 00000000001.50.6, , 00000000001.61.6, , , , , , , , , , 00000000001.69.6, , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:08:29.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:08:29.424+0000 7fbe8e551780 1 waiting for incremental sync to catch up: 2024-04-12T04:08:29.408 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , 00000000002.132.6, , , , 00000000001.61.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:08:29.408 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , 00000000002.132.6, , 00000000001.50.6, , 00000000001.61.6, , , , , , , , , , 00000000001.69.6, , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:08:34.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:08:34.426+0000 7fbe8e551780 1 waiting for incremental sync to catch up: 2024-04-12T04:08:34.411 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , 00000000002.132.6, , , , 00000000001.61.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:08:34.411 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , 00000000002.132.6, , 00000000001.50.6, , 00000000001.61.6, , , , , , , , , , 00000000001.69.6, , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:08:39.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:08:39.428+0000 7fbe8e551780 1 waiting for incremental sync to catch up: 2024-04-12T04:08:39.413 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , 00000000002.132.6, , , , 00000000001.61.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:08:39.413 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , 00000000002.132.6, , 00000000001.50.6, , 00000000001.61.6, , , , , , , , , , 00000000001.69.6, , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:08:44.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:08:44.431+0000 7fbe8e551780 1 waiting for incremental sync to catch up: 2024-04-12T04:08:44.415 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , 00000000002.132.6, , , , 00000000001.61.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:08:44.415 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , 00000000002.132.6, , 00000000001.50.6, , 00000000001.61.6, , , , , , , , , , 00000000001.69.6, , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:08:49.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:08:49.433+0000 7fbe8e551780 1 waiting for incremental sync to catch up: 2024-04-12T04:08:49.418 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , 00000000002.132.6, , , , 00000000001.61.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:08:49.418 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , 00000000002.132.6, , 00000000001.50.6, , 00000000001.61.6, , , , , , , , , , 00000000001.69.6, , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:08:54.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:08:54.436+0000 7fbe8e551780 1 waiting for incremental sync to catch up: 2024-04-12T04:08:54.420 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , 00000000002.132.6, , , , 00000000001.61.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:08:54.420 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , 00000000002.132.6, , 00000000001.50.6, , 00000000001.61.6, , , , , , , , , , 00000000001.69.6, , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:08:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:08:59.438+0000 7fbe8e551780 1 waiting for incremental sync to catch up: 2024-04-12T04:08:59.423 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , 00000000002.132.6, , , , 00000000001.61.6, , , , , , , , , , , , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:08:59.423 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , 00000000002.132.6, , 00000000001.50.6, , 00000000001.61.6, , , , , , , , , , 00000000001.69.6, , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:09:04.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:09:04.441+0000 7fbe8e551780 1 bucket sync caught up with source: 2024-04-12T04:09:04.425 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , 00000000002.132.6, , 00000000001.50.6, , 00000000001.61.6, , , , , , , , , , 00000000001.69.6, , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:09:04.425 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , 00000000002.132.6, , 00000000001.50.6, , 00000000001.61.6, , , , , , , , , , 00000000001.69.6, , , , , , , , , , , , , , 00000000001.51.6] 2024-04-12T04:09:04.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:09:04.441+0000 7fbe8e551780 0 bucket checkpoint complete 2024-04-12T04:09:04.444 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-61 2024-04-12T04:09:04.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:09:04.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:09:04.445 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:09:04.512 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:09:04.512 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:09:04.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:09:04.571+0000 7f7a3d416780 1 waiting for incremental sync to catch up: 2024-04-12T04:09:04.555 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000003.2390.6, 00000000002.1411.6, 00000000001.2992.6, 00000000004.2392.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000002.2561.6, 00000000006.2294.6, 00000000004.2618.6] 2024-04-12T04:09:04.555 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000003.2390.6, 00000000002.1411.6, 00000000002.3002.6, 00000000004.2392.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000003.2571.6, 00000000006.2294.6, 00000000004.2618.6] 2024-04-12T04:09:09.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:09:09.572+0000 7f7a3d416780 1 waiting for incremental sync to catch up: 2024-04-12T04:09:09.557 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000003.2390.6, 00000000002.1411.6, 00000000001.2992.6, 00000000004.2392.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000002.2561.6, 00000000006.2294.6, 00000000004.2618.6] 2024-04-12T04:09:09.557 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000003.2390.6, 00000000002.1411.6, 00000000002.3002.6, 00000000004.2392.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000003.2571.6, 00000000006.2294.6, 00000000004.2618.6] 2024-04-12T04:09:14.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:09:14.575+0000 7f7a3d416780 1 waiting for incremental sync to catch up: 2024-04-12T04:09:14.558 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000003.2390.6, 00000000002.1411.6, 00000000001.2992.6, 00000000004.2392.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000002.2561.6, 00000000006.2294.6, 00000000004.2618.6] 2024-04-12T04:09:14.558 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000003.2390.6, 00000000002.1411.6, 00000000002.3002.6, 00000000004.2392.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000003.2571.6, 00000000006.2294.6, 00000000004.2618.6] 2024-04-12T04:09:19.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:09:19.575+0000 7f7a3d416780 1 bucket sync caught up with source: 2024-04-12T04:09:19.560 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000003.2390.6, 00000000002.1411.6, 00000000002.3002.6, 00000000004.2392.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000003.2571.6, 00000000006.2294.6, 00000000004.2618.6] 2024-04-12T04:09:19.560 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000003.2390.6, 00000000002.1411.6, 00000000002.3002.6, 00000000004.2392.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000003.2571.6, 00000000006.2294.6, 00000000004.2618.6] 2024-04-12T04:09:19.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:09:19.575+0000 7f7a3d416780 0 bucket checkpoint complete 2024-04-12T04:09:19.581 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-61 zones={a2, a1} 2024-04-12T04:09:19.582 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:19.583 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:19.583 DEBUG:boto:Method: HEAD 2024-04-12T04:09:19.583 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:09:19.583 DEBUG:boto:Data: 2024-04-12T04:09:19.583 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.583 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:19.584 DEBUG:boto:Port: 8001 2024-04-12T04:09:19.584 DEBUG:boto:Params: {} 2024-04-12T04:09:19.584 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:09:19.584 DEBUG:boto:Token: None 2024-04-12T04:09:19.585 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/ 2024-04-12T04:09:19.585 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3/+gLIYzVsp/iLeOQC9vZK1I/zQ= 2024-04-12T04:09:19.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3/+gLIYzVsp/iLeOQC9vZK1I/zQ=', 'Content-Length': '0'} 2024-04-12T04:09:19.602 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ba16aae36409058-006618b3ef-1cfd-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.602 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:19.602 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:19.602 DEBUG:boto:Method: HEAD 2024-04-12T04:09:19.602 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:09:19.603 DEBUG:boto:Data: 2024-04-12T04:09:19.603 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.603 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:19.603 DEBUG:boto:Port: 8000 2024-04-12T04:09:19.603 DEBUG:boto:Params: {} 2024-04-12T04:09:19.603 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:09:19.604 DEBUG:boto:Token: None 2024-04-12T04:09:19.604 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/ 2024-04-12T04:09:19.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3/+gLIYzVsp/iLeOQC9vZK1I/zQ= 2024-04-12T04:09:19.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3/+gLIYzVsp/iLeOQC9vZK1I/zQ=', 'Content-Length': '0'} 2024-04-12T04:09:19.637 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9079d9d21de9722-006618b3ef-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.638 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:09:19.638 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:19.638 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:19.638 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:09:19.638 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:09:19.638 DEBUG:boto:Method: GET 2024-04-12T04:09:19.638 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:09:19.638 DEBUG:boto:Data: 2024-04-12T04:09:19.638 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.639 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:19.639 DEBUG:boto:Port: 8001 2024-04-12T04:09:19.639 DEBUG:boto:Params: {} 2024-04-12T04:09:19.639 DEBUG:boto:Token: None 2024-04-12T04:09:19.639 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/?versions 2024-04-12T04:09:19.639 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5oamWg4t4digABppZnV1SCWD3kw= 2024-04-12T04:09:19.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5oamWg4t4digABppZnV1SCWD3kw=', 'Content-Length': '0'} 2024-04-12T04:09:19.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8670725e6744381-006618b3ef-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.642 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:19.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:09:19.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T04:09:19.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T04:09:19.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-12T04:09:19.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T04:09:19.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-12T04:09:19.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:09:19.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T04:09:19.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:09:19.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T04:09:19.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T04:09:19.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-12T04:09:19.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:09:19.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:09:19.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T04:09:19.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:09:19.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T04:09:19.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:09:19.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:09:19.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T04:09:19.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T04:09:19.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-12T04:09:19.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:09:19.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T04:09:19.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T04:09:19.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:09:19.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T04:09:19.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T04:09:19.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-12T04:09:19.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T04:09:19.646 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:09:19.647 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:19.647 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:19.647 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:09:19.647 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:09:19.647 DEBUG:boto:Method: GET 2024-04-12T04:09:19.647 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:09:19.647 DEBUG:boto:Data: 2024-04-12T04:09:19.647 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.647 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:19.647 DEBUG:boto:Port: 8000 2024-04-12T04:09:19.647 DEBUG:boto:Params: {} 2024-04-12T04:09:19.647 DEBUG:boto:Token: None 2024-04-12T04:09:19.648 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/?versions 2024-04-12T04:09:19.648 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5oamWg4t4digABppZnV1SCWD3kw= 2024-04-12T04:09:19.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5oamWg4t4digABppZnV1SCWD3kw=', 'Content-Length': '0'} 2024-04-12T04:09:19.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffce7ff21d2dec20-006618b3ef-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.652 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:19.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:09:19.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T04:09:19.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T04:09:19.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-12T04:09:19.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T04:09:19.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-12T04:09:19.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:09:19.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T04:09:19.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:09:19.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T04:09:19.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T04:09:19.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-12T04:09:19.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:09:19.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:09:19.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T04:09:19.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:09:19.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T04:09:19.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:09:19.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:09:19.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T04:09:19.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T04:09:19.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-12T04:09:19.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:09:19.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T04:09:19.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T04:09:19.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:09:19.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T04:09:19.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T04:09:19.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-12T04:09:19.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T04:09:19.656 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:19.656 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:19.656 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:09:19.656 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:09:19.656 DEBUG:boto:Method: GET 2024-04-12T04:09:19.656 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:09:19.656 DEBUG:boto:Data: 2024-04-12T04:09:19.656 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.656 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:19.657 DEBUG:boto:Port: 8001 2024-04-12T04:09:19.657 DEBUG:boto:Params: {} 2024-04-12T04:09:19.657 DEBUG:boto:Token: None 2024-04-12T04:09:19.657 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/?versions 2024-04-12T04:09:19.657 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5oamWg4t4digABppZnV1SCWD3kw= 2024-04-12T04:09:19.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5oamWg4t4digABppZnV1SCWD3kw=', 'Content-Length': '0'} 2024-04-12T04:09:19.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009965ce88701ff5d8-006618b3ef-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.662 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:19.663 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:19.664 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:19.664 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:09:19.664 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:09:19.664 DEBUG:boto:Method: GET 2024-04-12T04:09:19.664 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:09:19.664 DEBUG:boto:Data: 2024-04-12T04:09:19.664 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.664 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:19.664 DEBUG:boto:Port: 8000 2024-04-12T04:09:19.664 DEBUG:boto:Params: {} 2024-04-12T04:09:19.664 DEBUG:boto:Token: None 2024-04-12T04:09:19.665 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/?versions 2024-04-12T04:09:19.665 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5oamWg4t4digABppZnV1SCWD3kw= 2024-04-12T04:09:19.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5oamWg4t4digABppZnV1SCWD3kw=', 'Content-Length': '0'} 2024-04-12T04:09:19.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005da2f55e74abaf75-006618b3ef-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.670 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:19.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:09:19.673 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:09:19.673 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:09:19.673 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.673 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.673 DEBUG:boto:Method: GET 2024-04-12T04:09:19.673 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.673 DEBUG:boto:Data: 2024-04-12T04:09:19.673 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.673 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:19.674 DEBUG:boto:Port: 8001 2024-04-12T04:09:19.674 DEBUG:boto:Params: {} 2024-04-12T04:09:19.674 DEBUG:boto:Token: None 2024-04-12T04:09:19.674 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.674 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+5TJBVxLeMmukutnSVhUc4n1/hE= 2024-04-12T04:09:19.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+5TJBVxLeMmukutnSVhUc4n1/hE=', 'Content-Length': '0'} 2024-04-12T04:09:19.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a24962b276bb57b-006618b3ef-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.678 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:09:19.678 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:09:19.679 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.679 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.679 DEBUG:boto:Method: GET 2024-04-12T04:09:19.679 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.679 DEBUG:boto:Data: 2024-04-12T04:09:19.679 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.679 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:19.679 DEBUG:boto:Port: 8000 2024-04-12T04:09:19.679 DEBUG:boto:Params: {} 2024-04-12T04:09:19.679 DEBUG:boto:Token: None 2024-04-12T04:09:19.679 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.680 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+5TJBVxLeMmukutnSVhUc4n1/hE= 2024-04-12T04:09:19.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+5TJBVxLeMmukutnSVhUc4n1/hE=', 'Content-Length': '0'} 2024-04-12T04:09:19.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001766f3e07292512-006618b3ef-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.727 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:09:19.727 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:09:19.727 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.727 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.728 DEBUG:boto:Method: HEAD 2024-04-12T04:09:19.728 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.728 DEBUG:boto:Data: 2024-04-12T04:09:19.728 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.728 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:19.728 DEBUG:boto:Port: 8001 2024-04-12T04:09:19.728 DEBUG:boto:Params: {} 2024-04-12T04:09:19.728 DEBUG:boto:Token: None 2024-04-12T04:09:19.728 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.728 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tdPMTSN3ASxaRygsW+4d9A90T3A= 2024-04-12T04:09:19.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tdPMTSN3ASxaRygsW+4d9A90T3A=', 'Content-Length': '0'} 2024-04-12T04:09:19.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d46810b8a6c9cb0-006618b3ef-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.731 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:09:19.731 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:09:19.732 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.732 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.732 DEBUG:boto:Method: HEAD 2024-04-12T04:09:19.732 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.732 DEBUG:boto:Data: 2024-04-12T04:09:19.732 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.732 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:19.733 DEBUG:boto:Port: 8000 2024-04-12T04:09:19.733 DEBUG:boto:Params: {} 2024-04-12T04:09:19.733 DEBUG:boto:Token: None 2024-04-12T04:09:19.733 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.734 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tdPMTSN3ASxaRygsW+4d9A90T3A= 2024-04-12T04:09:19.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tdPMTSN3ASxaRygsW+4d9A90T3A=', 'Content-Length': '0'} 2024-04-12T04:09:19.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006301d3578dae8b97-006618b3ef-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:09:19.737 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:09:19.737 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:09:19.737 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.737 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.737 DEBUG:boto:Method: GET 2024-04-12T04:09:19.737 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.738 DEBUG:boto:Data: 2024-04-12T04:09:19.738 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.738 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:19.738 DEBUG:boto:Port: 8001 2024-04-12T04:09:19.738 DEBUG:boto:Params: {} 2024-04-12T04:09:19.738 DEBUG:boto:Token: None 2024-04-12T04:09:19.738 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.738 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+5TJBVxLeMmukutnSVhUc4n1/hE= 2024-04-12T04:09:19.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+5TJBVxLeMmukutnSVhUc4n1/hE=', 'Content-Length': '0'} 2024-04-12T04:09:19.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094e09db0f9a18c6a-006618b3ef-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.787 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:09:19.787 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:09:19.787 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.787 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.787 DEBUG:boto:Method: GET 2024-04-12T04:09:19.787 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.787 DEBUG:boto:Data: 2024-04-12T04:09:19.788 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.788 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:19.788 DEBUG:boto:Port: 8000 2024-04-12T04:09:19.788 DEBUG:boto:Params: {} 2024-04-12T04:09:19.788 DEBUG:boto:Token: None 2024-04-12T04:09:19.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:09:19.788 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+5TJBVxLeMmukutnSVhUc4n1/hE= 2024-04-12T04:09:19.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+5TJBVxLeMmukutnSVhUc4n1/hE=', 'Content-Length': '0'} 2024-04-12T04:09:19.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb683341de5d5aae-006618b3ef-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.792 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:19.792 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:19.792 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:09:19.792 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:09:19.792 DEBUG:boto:Method: GET 2024-04-12T04:09:19.792 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:09:19.792 DEBUG:boto:Data: 2024-04-12T04:09:19.792 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.793 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:19.793 DEBUG:boto:Port: 8001 2024-04-12T04:09:19.793 DEBUG:boto:Params: {} 2024-04-12T04:09:19.793 DEBUG:boto:Token: None 2024-04-12T04:09:19.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/ 2024-04-12T04:09:19.793 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mN23NCfnXoIdruehqYEnMR/9G/w= 2024-04-12T04:09:19.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mN23NCfnXoIdruehqYEnMR/9G/w=', 'Content-Length': '0'} 2024-04-12T04:09:19.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022014fb3002a48eb-006618b3ef-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.796 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:19.796 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:19.797 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:19.797 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:09:19.797 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:09:19.797 DEBUG:boto:Method: GET 2024-04-12T04:09:19.797 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:09:19.797 DEBUG:boto:Data: 2024-04-12T04:09:19.797 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.797 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:19.797 DEBUG:boto:Port: 8000 2024-04-12T04:09:19.797 DEBUG:boto:Params: {} 2024-04-12T04:09:19.797 DEBUG:boto:Token: None 2024-04-12T04:09:19.798 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/ 2024-04-12T04:09:19.798 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mN23NCfnXoIdruehqYEnMR/9G/w= 2024-04-12T04:09:19.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mN23NCfnXoIdruehqYEnMR/9G/w=', 'Content-Length': '0'} 2024-04-12T04:09:19.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a34550fa2a7c0aed-006618b3ef-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.806 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:19.807 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:09:19.807 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:09:19.807 DEBUG:boto:Method: HEAD 2024-04-12T04:09:19.807 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:09:19.807 DEBUG:boto:Data: 2024-04-12T04:09:19.808 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.808 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:19.808 DEBUG:boto:Port: 8001 2024-04-12T04:09:19.808 DEBUG:boto:Params: {} 2024-04-12T04:09:19.808 DEBUG:boto:Token: None 2024-04-12T04:09:19.808 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/obj11 2024-04-12T04:09:19.808 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3jKc8Nyn+xvQt0sfaTT2fYp51R4= 2024-04-12T04:09:19.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3jKc8Nyn+xvQt0sfaTT2fYp51R4=', 'Content-Length': '0'} 2024-04-12T04:09:19.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cc503f841fbbee7-006618b3ef-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.813 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:09:19.813 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:09:19.813 DEBUG:boto:Method: HEAD 2024-04-12T04:09:19.813 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:09:19.813 DEBUG:boto:Data: 2024-04-12T04:09:19.813 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.813 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:19.813 DEBUG:boto:Port: 8000 2024-04-12T04:09:19.814 DEBUG:boto:Params: {} 2024-04-12T04:09:19.814 DEBUG:boto:Token: None 2024-04-12T04:09:19.814 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/obj11 2024-04-12T04:09:19.814 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3jKc8Nyn+xvQt0sfaTT2fYp51R4= 2024-04-12T04:09:19.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3jKc8Nyn+xvQt0sfaTT2fYp51R4=', 'Content-Length': '0'} 2024-04-12T04:09:19.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040289bd2d37b0ac0-006618b3ef-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:09:19.817 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:09:19.817 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:09:19.817 DEBUG:boto:Method: GET 2024-04-12T04:09:19.817 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:09:19.817 DEBUG:boto:Data: 2024-04-12T04:09:19.817 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.817 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:19.818 DEBUG:boto:Port: 8001 2024-04-12T04:09:19.818 DEBUG:boto:Params: {} 2024-04-12T04:09:19.818 DEBUG:boto:Token: None 2024-04-12T04:09:19.818 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/obj11 2024-04-12T04:09:19.818 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SjiFeapBBKR4ngF5tftb9KOoxaM= 2024-04-12T04:09:19.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SjiFeapBBKR4ngF5tftb9KOoxaM=', 'Content-Length': '0'} 2024-04-12T04:09:19.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa6879b4387d0603-006618b3ef-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.862 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:09:19.863 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:09:19.863 DEBUG:boto:Method: GET 2024-04-12T04:09:19.863 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:09:19.863 DEBUG:boto:Data: 2024-04-12T04:09:19.863 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.863 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:19.863 DEBUG:boto:Port: 8000 2024-04-12T04:09:19.863 DEBUG:boto:Params: {} 2024-04-12T04:09:19.863 DEBUG:boto:Token: None 2024-04-12T04:09:19.864 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/obj11 2024-04-12T04:09:19.864 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SjiFeapBBKR4ngF5tftb9KOoxaM= 2024-04-12T04:09:19.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SjiFeapBBKR4ngF5tftb9KOoxaM=', 'Content-Length': '0'} 2024-04-12T04:09:19.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d11e904216aae011-006618b3ef-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.867 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:19.867 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:19.867 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:09:19.867 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:09:19.867 DEBUG:boto:Method: GET 2024-04-12T04:09:19.867 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:09:19.867 DEBUG:boto:Data: 2024-04-12T04:09:19.867 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.868 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:19.868 DEBUG:boto:Port: 8001 2024-04-12T04:09:19.868 DEBUG:boto:Params: {} 2024-04-12T04:09:19.868 DEBUG:boto:Token: None 2024-04-12T04:09:19.868 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/ 2024-04-12T04:09:19.868 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mN23NCfnXoIdruehqYEnMR/9G/w= 2024-04-12T04:09:19.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mN23NCfnXoIdruehqYEnMR/9G/w=', 'Content-Length': '0'} 2024-04-12T04:09:19.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2d1412444d546a9-006618b3ef-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.871 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:19.871 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:19.871 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:19.872 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:09:19.872 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:09:19.872 DEBUG:boto:Method: GET 2024-04-12T04:09:19.872 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:09:19.872 DEBUG:boto:Data: 2024-04-12T04:09:19.872 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.872 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:19.872 DEBUG:boto:Port: 8000 2024-04-12T04:09:19.872 DEBUG:boto:Params: {} 2024-04-12T04:09:19.872 DEBUG:boto:Token: None 2024-04-12T04:09:19.872 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/ 2024-04-12T04:09:19.873 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mN23NCfnXoIdruehqYEnMR/9G/w= 2024-04-12T04:09:19.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mN23NCfnXoIdruehqYEnMR/9G/w=', 'Content-Length': '0'} 2024-04-12T04:09:19.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2617f6f54685d6d-006618b3ef-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.878 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:19.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:09:19.879 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:09:19.879 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:09:19.879 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.879 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.879 DEBUG:boto:Method: GET 2024-04-12T04:09:19.879 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.879 DEBUG:boto:Data: 2024-04-12T04:09:19.879 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.879 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:19.879 DEBUG:boto:Port: 8001 2024-04-12T04:09:19.879 DEBUG:boto:Params: {} 2024-04-12T04:09:19.880 DEBUG:boto:Token: None 2024-04-12T04:09:19.880 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.880 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Pq8Mkv0guvfq9xPV+Z9VAZAjEjM= 2024-04-12T04:09:19.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Pq8Mkv0guvfq9xPV+Z9VAZAjEjM=', 'Content-Length': '0'} 2024-04-12T04:09:19.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cf061121f2948a7-006618b3ef-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.882 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:09:19.882 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:09:19.882 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.883 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.883 DEBUG:boto:Method: GET 2024-04-12T04:09:19.883 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.883 DEBUG:boto:Data: 2024-04-12T04:09:19.883 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.883 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:19.883 DEBUG:boto:Port: 8000 2024-04-12T04:09:19.883 DEBUG:boto:Params: {} 2024-04-12T04:09:19.883 DEBUG:boto:Token: None 2024-04-12T04:09:19.883 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.883 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Pq8Mkv0guvfq9xPV+Z9VAZAjEjM= 2024-04-12T04:09:19.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Pq8Mkv0guvfq9xPV+Z9VAZAjEjM=', 'Content-Length': '0'} 2024-04-12T04:09:19.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c59940208a43a7e-006618b3ef-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.886 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:09:19.887 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:09:19.887 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.887 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.887 DEBUG:boto:Method: HEAD 2024-04-12T04:09:19.887 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.887 DEBUG:boto:Data: 2024-04-12T04:09:19.887 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.888 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:19.888 DEBUG:boto:Port: 8001 2024-04-12T04:09:19.888 DEBUG:boto:Params: {} 2024-04-12T04:09:19.888 DEBUG:boto:Token: None 2024-04-12T04:09:19.888 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.888 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CRV59x3gFcE9hzQbPVzI3+YTZMU= 2024-04-12T04:09:19.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CRV59x3gFcE9hzQbPVzI3+YTZMU=', 'Content-Length': '0'} 2024-04-12T04:09:19.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000143bdeeffc6397c7-006618b3ef-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.891 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:09:19.892 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:09:19.892 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.892 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.892 DEBUG:boto:Method: HEAD 2024-04-12T04:09:19.892 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.892 DEBUG:boto:Data: 2024-04-12T04:09:19.893 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.893 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:19.893 DEBUG:boto:Port: 8000 2024-04-12T04:09:19.893 DEBUG:boto:Params: {} 2024-04-12T04:09:19.893 DEBUG:boto:Token: None 2024-04-12T04:09:19.893 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.894 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CRV59x3gFcE9hzQbPVzI3+YTZMU= 2024-04-12T04:09:19.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CRV59x3gFcE9hzQbPVzI3+YTZMU=', 'Content-Length': '0'} 2024-04-12T04:09:19.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ec3e64783798526-006618b3ef-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:09:19.899 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:09:19.899 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:09:19.899 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.899 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.900 DEBUG:boto:Method: GET 2024-04-12T04:09:19.900 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.900 DEBUG:boto:Data: 2024-04-12T04:09:19.900 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.900 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:19.900 DEBUG:boto:Port: 8001 2024-04-12T04:09:19.900 DEBUG:boto:Params: {} 2024-04-12T04:09:19.901 DEBUG:boto:Token: None 2024-04-12T04:09:19.901 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.901 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Pq8Mkv0guvfq9xPV+Z9VAZAjEjM= 2024-04-12T04:09:19.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Pq8Mkv0guvfq9xPV+Z9VAZAjEjM=', 'Content-Length': '0'} 2024-04-12T04:09:19.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006450c949565c393d-006618b3ef-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.946 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:09:19.947 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:09:19.947 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.947 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.947 DEBUG:boto:Method: GET 2024-04-12T04:09:19.947 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.947 DEBUG:boto:Data: 2024-04-12T04:09:19.947 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.947 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:19.948 DEBUG:boto:Port: 8000 2024-04-12T04:09:19.948 DEBUG:boto:Params: {} 2024-04-12T04:09:19.948 DEBUG:boto:Token: None 2024-04-12T04:09:19.948 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:09:19.948 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Pq8Mkv0guvfq9xPV+Z9VAZAjEjM= 2024-04-12T04:09:19.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Pq8Mkv0guvfq9xPV+Z9VAZAjEjM=', 'Content-Length': '0'} 2024-04-12T04:09:19.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a789fd470a814490-006618b3ef-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.951 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:19.951 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:19.951 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:09:19.951 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:09:19.951 DEBUG:boto:Method: GET 2024-04-12T04:09:19.952 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:09:19.952 DEBUG:boto:Data: 2024-04-12T04:09:19.952 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.952 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:19.952 DEBUG:boto:Port: 8001 2024-04-12T04:09:19.952 DEBUG:boto:Params: {} 2024-04-12T04:09:19.952 DEBUG:boto:Token: None 2024-04-12T04:09:19.952 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/ 2024-04-12T04:09:19.952 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mN23NCfnXoIdruehqYEnMR/9G/w= 2024-04-12T04:09:19.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mN23NCfnXoIdruehqYEnMR/9G/w=', 'Content-Length': '0'} 2024-04-12T04:09:19.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051b28e67fc7dbc75-006618b3ef-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:19.956 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:19.957 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:19.957 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:19.957 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:09:19.957 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:09:19.957 DEBUG:boto:Method: GET 2024-04-12T04:09:19.957 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:09:19.957 DEBUG:boto:Data: 2024-04-12T04:09:19.957 DEBUG:boto:Headers: {} 2024-04-12T04:09:19.958 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:19.958 DEBUG:boto:Port: 8000 2024-04-12T04:09:19.958 DEBUG:boto:Params: {} 2024-04-12T04:09:19.958 DEBUG:boto:Token: None 2024-04-12T04:09:19.958 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:19 GMT /dkyges-61/ 2024-04-12T04:09:19.958 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mN23NCfnXoIdruehqYEnMR/9G/w= 2024-04-12T04:09:19.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mN23NCfnXoIdruehqYEnMR/9G/w=', 'Content-Length': '0'} 2024-04-12T04:09:19.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082013f67b7d09b4f-006618b3ef-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.010 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.011 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:09:20.011 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:09:20.011 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.011 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:09:20.011 DEBUG:boto:Data: 2024-04-12T04:09:20.011 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.012 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.012 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.012 DEBUG:boto:Params: {} 2024-04-12T04:09:20.012 DEBUG:boto:Token: None 2024-04-12T04:09:20.012 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj115 2024-04-12T04:09:20.012 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y5HQP4mDKDX05nm4P3IcacBfRPA= 2024-04-12T04:09:20.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y5HQP4mDKDX05nm4P3IcacBfRPA=', 'Content-Length': '0'} 2024-04-12T04:09:20.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5ecc1953d83441b-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.015 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:09:20.015 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:09:20.015 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.015 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:09:20.015 DEBUG:boto:Data: 2024-04-12T04:09:20.016 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.016 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.016 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.016 DEBUG:boto:Params: {} 2024-04-12T04:09:20.016 DEBUG:boto:Token: None 2024-04-12T04:09:20.016 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj115 2024-04-12T04:09:20.016 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Y5HQP4mDKDX05nm4P3IcacBfRPA= 2024-04-12T04:09:20.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y5HQP4mDKDX05nm4P3IcacBfRPA=', 'Content-Length': '0'} 2024-04-12T04:09:20.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001d212e68665ada5-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:09:20.019 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:09:20.019 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:09:20.019 DEBUG:boto:Method: GET 2024-04-12T04:09:20.019 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:09:20.019 DEBUG:boto:Data: 2024-04-12T04:09:20.019 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.020 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.020 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.020 DEBUG:boto:Params: {} 2024-04-12T04:09:20.020 DEBUG:boto:Token: None 2024-04-12T04:09:20.020 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj115 2024-04-12T04:09:20.020 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QYuMd5YrbU0o2vqSyGx5Yp9+GkU= 2024-04-12T04:09:20.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QYuMd5YrbU0o2vqSyGx5Yp9+GkU=', 'Content-Length': '0'} 2024-04-12T04:09:20.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbf6536e01e8c97f-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.022 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:09:20.023 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:09:20.023 DEBUG:boto:Method: GET 2024-04-12T04:09:20.023 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:09:20.023 DEBUG:boto:Data: 2024-04-12T04:09:20.023 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.023 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.023 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.023 DEBUG:boto:Params: {} 2024-04-12T04:09:20.023 DEBUG:boto:Token: None 2024-04-12T04:09:20.023 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj115 2024-04-12T04:09:20.024 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QYuMd5YrbU0o2vqSyGx5Yp9+GkU= 2024-04-12T04:09:20.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QYuMd5YrbU0o2vqSyGx5Yp9+GkU=', 'Content-Length': '0'} 2024-04-12T04:09:20.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cf3b2ccf2f1ebc9-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.026 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.026 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.026 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:09:20.026 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:09:20.026 DEBUG:boto:Method: GET 2024-04-12T04:09:20.026 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:09:20.026 DEBUG:boto:Data: 2024-04-12T04:09:20.026 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.026 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.027 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.027 DEBUG:boto:Params: {} 2024-04-12T04:09:20.027 DEBUG:boto:Token: None 2024-04-12T04:09:20.027 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.027 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000376689997c10e12-006618b3f0-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.074 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.075 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.075 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.075 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:09:20.075 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:09:20.075 DEBUG:boto:Method: GET 2024-04-12T04:09:20.076 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:09:20.076 DEBUG:boto:Data: 2024-04-12T04:09:20.076 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.076 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.076 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.076 DEBUG:boto:Params: {} 2024-04-12T04:09:20.076 DEBUG:boto:Token: None 2024-04-12T04:09:20.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.076 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a233ea08f3c6b385-006618b3f0-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.080 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:09:20.081 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:09:20.081 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:09:20.081 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.081 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.081 DEBUG:boto:Method: GET 2024-04-12T04:09:20.081 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.081 DEBUG:boto:Data: 2024-04-12T04:09:20.082 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.082 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.082 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.082 DEBUG:boto:Params: {} 2024-04-12T04:09:20.082 DEBUG:boto:Token: None 2024-04-12T04:09:20.082 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.082 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dCSFFhKYkLfy2zLwl2D6iFpR4lg= 2024-04-12T04:09:20.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dCSFFhKYkLfy2zLwl2D6iFpR4lg=', 'Content-Length': '0'} 2024-04-12T04:09:20.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000812873d508f9ca98-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.085 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:09:20.085 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:09:20.085 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.085 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.085 DEBUG:boto:Method: GET 2024-04-12T04:09:20.085 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.085 DEBUG:boto:Data: 2024-04-12T04:09:20.085 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.085 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.086 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.086 DEBUG:boto:Params: {} 2024-04-12T04:09:20.086 DEBUG:boto:Token: None 2024-04-12T04:09:20.086 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.086 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dCSFFhKYkLfy2zLwl2D6iFpR4lg= 2024-04-12T04:09:20.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dCSFFhKYkLfy2zLwl2D6iFpR4lg=', 'Content-Length': '0'} 2024-04-12T04:09:20.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2fbfb538e751501-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.130 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:09:20.131 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:09:20.131 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.131 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.131 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.131 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.131 DEBUG:boto:Data: 2024-04-12T04:09:20.131 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.131 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.131 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.131 DEBUG:boto:Params: {} 2024-04-12T04:09:20.132 DEBUG:boto:Token: None 2024-04-12T04:09:20.132 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.132 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/rZWn5uolyaPR0nb7Lj0FbDfpvY= 2024-04-12T04:09:20.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/rZWn5uolyaPR0nb7Lj0FbDfpvY=', 'Content-Length': '0'} 2024-04-12T04:09:20.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f92b428b29b7e13-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.134 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:09:20.134 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:09:20.134 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.135 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.135 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.135 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.135 DEBUG:boto:Data: 2024-04-12T04:09:20.135 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.135 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.135 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.135 DEBUG:boto:Params: {} 2024-04-12T04:09:20.135 DEBUG:boto:Token: None 2024-04-12T04:09:20.135 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.136 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/rZWn5uolyaPR0nb7Lj0FbDfpvY= 2024-04-12T04:09:20.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/rZWn5uolyaPR0nb7Lj0FbDfpvY=', 'Content-Length': '0'} 2024-04-12T04:09:20.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b61714fc7270edc-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:09:20.138 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:09:20.138 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:09:20.138 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.138 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.138 DEBUG:boto:Method: GET 2024-04-12T04:09:20.138 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.138 DEBUG:boto:Data: 2024-04-12T04:09:20.139 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.139 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.139 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.139 DEBUG:boto:Params: {} 2024-04-12T04:09:20.139 DEBUG:boto:Token: None 2024-04-12T04:09:20.139 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.139 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dCSFFhKYkLfy2zLwl2D6iFpR4lg= 2024-04-12T04:09:20.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dCSFFhKYkLfy2zLwl2D6iFpR4lg=', 'Content-Length': '0'} 2024-04-12T04:09:20.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b696e97c811cfbf7-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.141 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:09:20.142 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:09:20.142 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.142 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.142 DEBUG:boto:Method: GET 2024-04-12T04:09:20.142 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.142 DEBUG:boto:Data: 2024-04-12T04:09:20.142 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.142 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.142 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.143 DEBUG:boto:Params: {} 2024-04-12T04:09:20.143 DEBUG:boto:Token: None 2024-04-12T04:09:20.143 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:09:20.143 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dCSFFhKYkLfy2zLwl2D6iFpR4lg= 2024-04-12T04:09:20.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dCSFFhKYkLfy2zLwl2D6iFpR4lg=', 'Content-Length': '0'} 2024-04-12T04:09:20.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005646cfea2cde7861-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.145 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.146 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.146 DEBUG:boto:path=/dkyges-61/?prefix=obj116 2024-04-12T04:09:20.146 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj116 2024-04-12T04:09:20.146 DEBUG:boto:Method: GET 2024-04-12T04:09:20.146 DEBUG:boto:Path: /dkyges-61/?prefix=obj116 2024-04-12T04:09:20.146 DEBUG:boto:Data: 2024-04-12T04:09:20.146 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.146 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.146 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.146 DEBUG:boto:Params: {} 2024-04-12T04:09:20.147 DEBUG:boto:Token: None 2024-04-12T04:09:20.147 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.147 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018d461c55b6caeb5-006618b3f0-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.150 DEBUG:boto:b'dkyges-61obj1161000falseobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.150 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.150 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.150 DEBUG:boto:path=/dkyges-61/?prefix=obj116 2024-04-12T04:09:20.150 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj116 2024-04-12T04:09:20.150 DEBUG:boto:Method: GET 2024-04-12T04:09:20.151 DEBUG:boto:Path: /dkyges-61/?prefix=obj116 2024-04-12T04:09:20.151 DEBUG:boto:Data: 2024-04-12T04:09:20.151 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.151 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.151 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.151 DEBUG:boto:Params: {} 2024-04-12T04:09:20.151 DEBUG:boto:Token: None 2024-04-12T04:09:20.151 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.151 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a01523e4977cfef1-006618b3f0-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.155 DEBUG:boto:b'dkyges-61obj1161000falseobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.155 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:09:20.156 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:09:20.156 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.156 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:09:20.156 DEBUG:boto:Data: 2024-04-12T04:09:20.156 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.156 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.156 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.156 DEBUG:boto:Params: {} 2024-04-12T04:09:20.156 DEBUG:boto:Token: None 2024-04-12T04:09:20.156 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj116 2024-04-12T04:09:20.157 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QeHuhyNTu7Di2K2r1+SIW5gVKpY= 2024-04-12T04:09:20.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QeHuhyNTu7Di2K2r1+SIW5gVKpY=', 'Content-Length': '0'} 2024-04-12T04:09:20.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000174338289f2b4eb7-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.159 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:09:20.159 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:09:20.159 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.159 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:09:20.159 DEBUG:boto:Data: 2024-04-12T04:09:20.159 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.159 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.159 DEBUG:boto:Params: {} 2024-04-12T04:09:20.159 DEBUG:boto:Token: None 2024-04-12T04:09:20.159 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj116 2024-04-12T04:09:20.160 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QeHuhyNTu7Di2K2r1+SIW5gVKpY= 2024-04-12T04:09:20.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QeHuhyNTu7Di2K2r1+SIW5gVKpY=', 'Content-Length': '0'} 2024-04-12T04:09:20.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000889e3d8ff740d34d-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:09:20.162 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:09:20.162 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:09:20.162 DEBUG:boto:Method: GET 2024-04-12T04:09:20.162 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:09:20.162 DEBUG:boto:Data: 2024-04-12T04:09:20.162 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.162 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.162 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.162 DEBUG:boto:Params: {} 2024-04-12T04:09:20.163 DEBUG:boto:Token: None 2024-04-12T04:09:20.163 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj116 2024-04-12T04:09:20.163 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6HJjo7GzkkBi/ndQ+eIpOL7/WMA= 2024-04-12T04:09:20.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6HJjo7GzkkBi/ndQ+eIpOL7/WMA=', 'Content-Length': '0'} 2024-04-12T04:09:20.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9f17b28d1d7675d-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.165 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:09:20.165 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:09:20.165 DEBUG:boto:Method: GET 2024-04-12T04:09:20.165 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:09:20.165 DEBUG:boto:Data: 2024-04-12T04:09:20.165 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.166 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.166 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.166 DEBUG:boto:Params: {} 2024-04-12T04:09:20.166 DEBUG:boto:Token: None 2024-04-12T04:09:20.166 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj116 2024-04-12T04:09:20.166 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6HJjo7GzkkBi/ndQ+eIpOL7/WMA= 2024-04-12T04:09:20.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6HJjo7GzkkBi/ndQ+eIpOL7/WMA=', 'Content-Length': '0'} 2024-04-12T04:09:20.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8cfa64aae213d70-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.168 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.168 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.168 DEBUG:boto:path=/dkyges-61/?prefix=obj116 2024-04-12T04:09:20.168 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj116 2024-04-12T04:09:20.168 DEBUG:boto:Method: GET 2024-04-12T04:09:20.169 DEBUG:boto:Path: /dkyges-61/?prefix=obj116 2024-04-12T04:09:20.169 DEBUG:boto:Data: 2024-04-12T04:09:20.169 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.169 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.169 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.169 DEBUG:boto:Params: {} 2024-04-12T04:09:20.169 DEBUG:boto:Token: None 2024-04-12T04:09:20.169 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.169 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f2d04d7ac7e607a-006618b3f0-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.172 DEBUG:boto:b'dkyges-61obj1161000falseobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.172 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.172 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.172 DEBUG:boto:path=/dkyges-61/?prefix=obj116 2024-04-12T04:09:20.172 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj116 2024-04-12T04:09:20.173 DEBUG:boto:Method: GET 2024-04-12T04:09:20.173 DEBUG:boto:Path: /dkyges-61/?prefix=obj116 2024-04-12T04:09:20.173 DEBUG:boto:Data: 2024-04-12T04:09:20.173 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.173 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.173 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.173 DEBUG:boto:Params: {} 2024-04-12T04:09:20.173 DEBUG:boto:Token: None 2024-04-12T04:09:20.173 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.173 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae8c64924c0eca14-006618b3f0-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.177 DEBUG:boto:b'dkyges-61obj1161000falseobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T04:09:20.178 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:09:20.178 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:09:20.178 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.178 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.178 DEBUG:boto:Method: GET 2024-04-12T04:09:20.178 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.178 DEBUG:boto:Data: 2024-04-12T04:09:20.178 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.178 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.179 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.179 DEBUG:boto:Params: {} 2024-04-12T04:09:20.179 DEBUG:boto:Token: None 2024-04-12T04:09:20.179 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.179 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P+kb40blRl4XZldrIqFAG/sNVsg= 2024-04-12T04:09:20.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P+kb40blRl4XZldrIqFAG/sNVsg=', 'Content-Length': '0'} 2024-04-12T04:09:20.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be7ea95361457d39-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.226 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:09:20.227 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:09:20.227 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.227 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.227 DEBUG:boto:Method: GET 2024-04-12T04:09:20.227 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.227 DEBUG:boto:Data: 2024-04-12T04:09:20.227 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.227 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.227 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.227 DEBUG:boto:Params: {} 2024-04-12T04:09:20.228 DEBUG:boto:Token: None 2024-04-12T04:09:20.228 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.228 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P+kb40blRl4XZldrIqFAG/sNVsg= 2024-04-12T04:09:20.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P+kb40blRl4XZldrIqFAG/sNVsg=', 'Content-Length': '0'} 2024-04-12T04:09:20.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000196b0b6c1fca2e96-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.231 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:09:20.231 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:09:20.231 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.231 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.231 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.231 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.231 DEBUG:boto:Data: 2024-04-12T04:09:20.231 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.231 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.231 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.231 DEBUG:boto:Params: {} 2024-04-12T04:09:20.232 DEBUG:boto:Token: None 2024-04-12T04:09:20.232 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.232 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DjbVY9KqbSZamQMLtUKWVzBLqEo= 2024-04-12T04:09:20.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DjbVY9KqbSZamQMLtUKWVzBLqEo=', 'Content-Length': '0'} 2024-04-12T04:09:20.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2c77dc4c309f51c-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.234 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:09:20.234 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:09:20.234 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.234 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.234 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.234 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.234 DEBUG:boto:Data: 2024-04-12T04:09:20.234 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.234 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.235 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.235 DEBUG:boto:Params: {} 2024-04-12T04:09:20.235 DEBUG:boto:Token: None 2024-04-12T04:09:20.235 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.235 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DjbVY9KqbSZamQMLtUKWVzBLqEo= 2024-04-12T04:09:20.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DjbVY9KqbSZamQMLtUKWVzBLqEo=', 'Content-Length': '0'} 2024-04-12T04:09:20.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0c7d6007c49323a-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T04:09:20.237 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:09:20.237 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:09:20.237 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.238 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.238 DEBUG:boto:Method: GET 2024-04-12T04:09:20.238 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.238 DEBUG:boto:Data: 2024-04-12T04:09:20.238 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.238 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.238 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.238 DEBUG:boto:Params: {} 2024-04-12T04:09:20.238 DEBUG:boto:Token: None 2024-04-12T04:09:20.238 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.239 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P+kb40blRl4XZldrIqFAG/sNVsg= 2024-04-12T04:09:20.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P+kb40blRl4XZldrIqFAG/sNVsg=', 'Content-Length': '0'} 2024-04-12T04:09:20.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baf40f0b6e4208c7-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.241 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:09:20.241 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:09:20.241 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.241 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.241 DEBUG:boto:Method: GET 2024-04-12T04:09:20.241 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.241 DEBUG:boto:Data: 2024-04-12T04:09:20.241 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.242 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.242 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.242 DEBUG:boto:Params: {} 2024-04-12T04:09:20.242 DEBUG:boto:Token: None 2024-04-12T04:09:20.242 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:09:20.242 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P+kb40blRl4XZldrIqFAG/sNVsg= 2024-04-12T04:09:20.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P+kb40blRl4XZldrIqFAG/sNVsg=', 'Content-Length': '0'} 2024-04-12T04:09:20.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f16660122ac4c4f3-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.286 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.286 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.287 DEBUG:boto:path=/dkyges-61/?prefix=obj124 2024-04-12T04:09:20.287 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj124 2024-04-12T04:09:20.287 DEBUG:boto:Method: GET 2024-04-12T04:09:20.287 DEBUG:boto:Path: /dkyges-61/?prefix=obj124 2024-04-12T04:09:20.287 DEBUG:boto:Data: 2024-04-12T04:09:20.287 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.287 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.287 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.287 DEBUG:boto:Params: {} 2024-04-12T04:09:20.287 DEBUG:boto:Token: None 2024-04-12T04:09:20.288 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.288 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003129a75886ad5ece-006618b3f0-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.290 DEBUG:boto:b'dkyges-61obj1241000falseobj1242024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.291 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.291 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.291 DEBUG:boto:path=/dkyges-61/?prefix=obj124 2024-04-12T04:09:20.291 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj124 2024-04-12T04:09:20.291 DEBUG:boto:Method: GET 2024-04-12T04:09:20.291 DEBUG:boto:Path: /dkyges-61/?prefix=obj124 2024-04-12T04:09:20.292 DEBUG:boto:Data: 2024-04-12T04:09:20.292 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.292 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.292 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.292 DEBUG:boto:Params: {} 2024-04-12T04:09:20.292 DEBUG:boto:Token: None 2024-04-12T04:09:20.292 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.292 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acc0d7e61e3bd8df-006618b3f0-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.296 DEBUG:boto:b'dkyges-61obj1241000falseobj1242024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.297 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:09:20.297 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:09:20.297 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.297 DEBUG:boto:Path: /dkyges-61/obj124 2024-04-12T04:09:20.297 DEBUG:boto:Data: 2024-04-12T04:09:20.297 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.297 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.297 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.298 DEBUG:boto:Params: {} 2024-04-12T04:09:20.298 DEBUG:boto:Token: None 2024-04-12T04:09:20.298 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj124 2024-04-12T04:09:20.298 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6e4P0joyW6HtYeBH6xtf+CfqYao= 2024-04-12T04:09:20.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6e4P0joyW6HtYeBH6xtf+CfqYao=', 'Content-Length': '0'} 2024-04-12T04:09:20.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000410b446991450c4f-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.300 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:09:20.300 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:09:20.300 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.300 DEBUG:boto:Path: /dkyges-61/obj124 2024-04-12T04:09:20.301 DEBUG:boto:Data: 2024-04-12T04:09:20.301 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.301 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.301 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.301 DEBUG:boto:Params: {} 2024-04-12T04:09:20.301 DEBUG:boto:Token: None 2024-04-12T04:09:20.301 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj124 2024-04-12T04:09:20.301 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6e4P0joyW6HtYeBH6xtf+CfqYao= 2024-04-12T04:09:20.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6e4P0joyW6HtYeBH6xtf+CfqYao=', 'Content-Length': '0'} 2024-04-12T04:09:20.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f88e8566c7830f8-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T04:09:20.304 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:09:20.304 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:09:20.304 DEBUG:boto:Method: GET 2024-04-12T04:09:20.304 DEBUG:boto:Path: /dkyges-61/obj124 2024-04-12T04:09:20.304 DEBUG:boto:Data: 2024-04-12T04:09:20.304 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.304 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.304 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.304 DEBUG:boto:Params: {} 2024-04-12T04:09:20.305 DEBUG:boto:Token: None 2024-04-12T04:09:20.305 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj124 2024-04-12T04:09:20.305 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4URYDDRjBQuFMUbZ1lmDGkgYbbM= 2024-04-12T04:09:20.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4URYDDRjBQuFMUbZ1lmDGkgYbbM=', 'Content-Length': '0'} 2024-04-12T04:09:20.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040a76739da378c08-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.307 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:09:20.307 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:09:20.307 DEBUG:boto:Method: GET 2024-04-12T04:09:20.307 DEBUG:boto:Path: /dkyges-61/obj124 2024-04-12T04:09:20.307 DEBUG:boto:Data: 2024-04-12T04:09:20.307 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.308 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.308 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.308 DEBUG:boto:Params: {} 2024-04-12T04:09:20.308 DEBUG:boto:Token: None 2024-04-12T04:09:20.308 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj124 2024-04-12T04:09:20.308 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4URYDDRjBQuFMUbZ1lmDGkgYbbM= 2024-04-12T04:09:20.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4URYDDRjBQuFMUbZ1lmDGkgYbbM=', 'Content-Length': '0'} 2024-04-12T04:09:20.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf4d857b77b9999d-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.350 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.351 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.351 DEBUG:boto:path=/dkyges-61/?prefix=obj124 2024-04-12T04:09:20.351 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj124 2024-04-12T04:09:20.351 DEBUG:boto:Method: GET 2024-04-12T04:09:20.351 DEBUG:boto:Path: /dkyges-61/?prefix=obj124 2024-04-12T04:09:20.351 DEBUG:boto:Data: 2024-04-12T04:09:20.351 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.351 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.351 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.351 DEBUG:boto:Params: {} 2024-04-12T04:09:20.351 DEBUG:boto:Token: None 2024-04-12T04:09:20.352 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.352 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8e379d0d91afad8-006618b3f0-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.355 DEBUG:boto:b'dkyges-61obj1241000falseobj1242024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.355 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.355 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.355 DEBUG:boto:path=/dkyges-61/?prefix=obj124 2024-04-12T04:09:20.356 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj124 2024-04-12T04:09:20.356 DEBUG:boto:Method: GET 2024-04-12T04:09:20.356 DEBUG:boto:Path: /dkyges-61/?prefix=obj124 2024-04-12T04:09:20.356 DEBUG:boto:Data: 2024-04-12T04:09:20.356 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.356 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.356 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.356 DEBUG:boto:Params: {} 2024-04-12T04:09:20.356 DEBUG:boto:Token: None 2024-04-12T04:09:20.356 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.356 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f75903cf0cb1d6af-006618b3f0-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.361 DEBUG:boto:b'dkyges-61obj1241000falseobj1242024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:09:20.361 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:09:20.361 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:09:20.361 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.361 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.362 DEBUG:boto:Method: GET 2024-04-12T04:09:20.362 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.362 DEBUG:boto:Data: 2024-04-12T04:09:20.362 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.362 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.362 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.362 DEBUG:boto:Params: {} 2024-04-12T04:09:20.362 DEBUG:boto:Token: None 2024-04-12T04:09:20.362 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.363 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UmOg1PMrHyC9yJ3ZY6yIVMankZA= 2024-04-12T04:09:20.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UmOg1PMrHyC9yJ3ZY6yIVMankZA=', 'Content-Length': '0'} 2024-04-12T04:09:20.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008efdb9602b3613f6-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.406 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:09:20.406 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:09:20.407 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.407 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.407 DEBUG:boto:Method: GET 2024-04-12T04:09:20.407 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.407 DEBUG:boto:Data: 2024-04-12T04:09:20.407 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.407 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.407 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.407 DEBUG:boto:Params: {} 2024-04-12T04:09:20.407 DEBUG:boto:Token: None 2024-04-12T04:09:20.408 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.408 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UmOg1PMrHyC9yJ3ZY6yIVMankZA= 2024-04-12T04:09:20.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UmOg1PMrHyC9yJ3ZY6yIVMankZA=', 'Content-Length': '0'} 2024-04-12T04:09:20.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cffba85665e65254-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.410 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:09:20.410 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:09:20.410 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.410 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.411 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.411 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.411 DEBUG:boto:Data: 2024-04-12T04:09:20.411 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.411 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.411 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.411 DEBUG:boto:Params: {} 2024-04-12T04:09:20.411 DEBUG:boto:Token: None 2024-04-12T04:09:20.411 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.411 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P0ImGQSjxrYA45LHccC31kxSlvU= 2024-04-12T04:09:20.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P0ImGQSjxrYA45LHccC31kxSlvU=', 'Content-Length': '0'} 2024-04-12T04:09:20.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4d92ea3e9a82e19-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.414 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:09:20.414 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:09:20.414 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.414 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.414 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.414 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.414 DEBUG:boto:Data: 2024-04-12T04:09:20.414 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.415 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.415 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.415 DEBUG:boto:Params: {} 2024-04-12T04:09:20.415 DEBUG:boto:Token: None 2024-04-12T04:09:20.415 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.415 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P0ImGQSjxrYA45LHccC31kxSlvU= 2024-04-12T04:09:20.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P0ImGQSjxrYA45LHccC31kxSlvU=', 'Content-Length': '0'} 2024-04-12T04:09:20.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072f3274da767c30d-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:09:20.417 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:09:20.417 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:09:20.417 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.418 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.418 DEBUG:boto:Method: GET 2024-04-12T04:09:20.418 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.418 DEBUG:boto:Data: 2024-04-12T04:09:20.418 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.418 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.418 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.418 DEBUG:boto:Params: {} 2024-04-12T04:09:20.418 DEBUG:boto:Token: None 2024-04-12T04:09:20.418 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.418 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UmOg1PMrHyC9yJ3ZY6yIVMankZA= 2024-04-12T04:09:20.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UmOg1PMrHyC9yJ3ZY6yIVMankZA=', 'Content-Length': '0'} 2024-04-12T04:09:20.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c292df8966dc174-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.421 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:09:20.421 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:09:20.421 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.421 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.421 DEBUG:boto:Method: GET 2024-04-12T04:09:20.421 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.421 DEBUG:boto:Data: 2024-04-12T04:09:20.422 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.422 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.422 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.422 DEBUG:boto:Params: {} 2024-04-12T04:09:20.422 DEBUG:boto:Token: None 2024-04-12T04:09:20.422 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:09:20.422 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UmOg1PMrHyC9yJ3ZY6yIVMankZA= 2024-04-12T04:09:20.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UmOg1PMrHyC9yJ3ZY6yIVMankZA=', 'Content-Length': '0'} 2024-04-12T04:09:20.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004513294145a77d3c-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.424 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.425 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.425 DEBUG:boto:path=/dkyges-61/?prefix=obj145 2024-04-12T04:09:20.425 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj145 2024-04-12T04:09:20.425 DEBUG:boto:Method: GET 2024-04-12T04:09:20.425 DEBUG:boto:Path: /dkyges-61/?prefix=obj145 2024-04-12T04:09:20.425 DEBUG:boto:Data: 2024-04-12T04:09:20.425 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.425 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.425 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.425 DEBUG:boto:Params: {} 2024-04-12T04:09:20.425 DEBUG:boto:Token: None 2024-04-12T04:09:20.426 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.426 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004de5395f509fca9a-006618b3f0-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.428 DEBUG:boto:b'dkyges-61obj1451000falseobj1452024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.429 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.429 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.429 DEBUG:boto:path=/dkyges-61/?prefix=obj145 2024-04-12T04:09:20.429 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj145 2024-04-12T04:09:20.429 DEBUG:boto:Method: GET 2024-04-12T04:09:20.429 DEBUG:boto:Path: /dkyges-61/?prefix=obj145 2024-04-12T04:09:20.429 DEBUG:boto:Data: 2024-04-12T04:09:20.429 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.430 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.430 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.430 DEBUG:boto:Params: {} 2024-04-12T04:09:20.430 DEBUG:boto:Token: None 2024-04-12T04:09:20.430 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.430 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000490991550224566-006618b3f0-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.434 DEBUG:boto:b'dkyges-61obj1451000falseobj1452024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.435 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:09:20.435 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:09:20.435 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.435 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:09:20.435 DEBUG:boto:Data: 2024-04-12T04:09:20.435 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.435 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.435 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.435 DEBUG:boto:Params: {} 2024-04-12T04:09:20.436 DEBUG:boto:Token: None 2024-04-12T04:09:20.436 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj145 2024-04-12T04:09:20.436 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IO+TnmMayOWiiwqEdv17/WYGzys= 2024-04-12T04:09:20.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IO+TnmMayOWiiwqEdv17/WYGzys=', 'Content-Length': '0'} 2024-04-12T04:09:20.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f20dab09f67d594f-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.438 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:09:20.438 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:09:20.438 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.438 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:09:20.438 DEBUG:boto:Data: 2024-04-12T04:09:20.438 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.438 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.439 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.439 DEBUG:boto:Params: {} 2024-04-12T04:09:20.439 DEBUG:boto:Token: None 2024-04-12T04:09:20.439 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj145 2024-04-12T04:09:20.439 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IO+TnmMayOWiiwqEdv17/WYGzys= 2024-04-12T04:09:20.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IO+TnmMayOWiiwqEdv17/WYGzys=', 'Content-Length': '0'} 2024-04-12T04:09:20.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eada15277be9cef-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:09:20.441 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:09:20.441 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:09:20.441 DEBUG:boto:Method: GET 2024-04-12T04:09:20.442 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:09:20.442 DEBUG:boto:Data: 2024-04-12T04:09:20.442 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.442 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.442 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.442 DEBUG:boto:Params: {} 2024-04-12T04:09:20.442 DEBUG:boto:Token: None 2024-04-12T04:09:20.442 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj145 2024-04-12T04:09:20.442 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U8qvXr+PI7O2d3w5JHaWya0n/TE= 2024-04-12T04:09:20.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U8qvXr+PI7O2d3w5JHaWya0n/TE=', 'Content-Length': '0'} 2024-04-12T04:09:20.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000092d6f506718a9d5-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.444 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:09:20.445 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:09:20.445 DEBUG:boto:Method: GET 2024-04-12T04:09:20.445 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:09:20.445 DEBUG:boto:Data: 2024-04-12T04:09:20.445 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.445 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.445 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.445 DEBUG:boto:Params: {} 2024-04-12T04:09:20.445 DEBUG:boto:Token: None 2024-04-12T04:09:20.445 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj145 2024-04-12T04:09:20.445 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U8qvXr+PI7O2d3w5JHaWya0n/TE= 2024-04-12T04:09:20.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U8qvXr+PI7O2d3w5JHaWya0n/TE=', 'Content-Length': '0'} 2024-04-12T04:09:20.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bed9ac8c8e407cf-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.448 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.448 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.448 DEBUG:boto:path=/dkyges-61/?prefix=obj145 2024-04-12T04:09:20.448 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj145 2024-04-12T04:09:20.448 DEBUG:boto:Method: GET 2024-04-12T04:09:20.448 DEBUG:boto:Path: /dkyges-61/?prefix=obj145 2024-04-12T04:09:20.448 DEBUG:boto:Data: 2024-04-12T04:09:20.448 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.448 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.448 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.448 DEBUG:boto:Params: {} 2024-04-12T04:09:20.449 DEBUG:boto:Token: None 2024-04-12T04:09:20.449 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.449 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000935367aec42ce83f-006618b3f0-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.451 DEBUG:boto:b'dkyges-61obj1451000falseobj1452024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.452 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.452 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.452 DEBUG:boto:path=/dkyges-61/?prefix=obj145 2024-04-12T04:09:20.452 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj145 2024-04-12T04:09:20.452 DEBUG:boto:Method: GET 2024-04-12T04:09:20.452 DEBUG:boto:Path: /dkyges-61/?prefix=obj145 2024-04-12T04:09:20.452 DEBUG:boto:Data: 2024-04-12T04:09:20.452 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.452 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.453 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.453 DEBUG:boto:Params: {} 2024-04-12T04:09:20.453 DEBUG:boto:Token: None 2024-04-12T04:09:20.453 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.453 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009129fe04f3225ce3-006618b3f0-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.502 DEBUG:boto:b'dkyges-61obj1451000falseobj1452024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T04:09:20.503 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:09:20.503 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:09:20.504 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.504 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.504 DEBUG:boto:Method: GET 2024-04-12T04:09:20.504 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.504 DEBUG:boto:Data: 2024-04-12T04:09:20.505 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.505 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.505 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.505 DEBUG:boto:Params: {} 2024-04-12T04:09:20.506 DEBUG:boto:Token: None 2024-04-12T04:09:20.506 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.506 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GORTwuYweKDb8Fd6str1p+cHp0E= 2024-04-12T04:09:20.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GORTwuYweKDb8Fd6str1p+cHp0E=', 'Content-Length': '0'} 2024-04-12T04:09:20.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009612dfc289f79b33-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.509 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:09:20.510 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:09:20.510 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.510 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.510 DEBUG:boto:Method: GET 2024-04-12T04:09:20.511 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.511 DEBUG:boto:Data: 2024-04-12T04:09:20.511 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.511 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.511 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.512 DEBUG:boto:Params: {} 2024-04-12T04:09:20.512 DEBUG:boto:Token: None 2024-04-12T04:09:20.512 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.512 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GORTwuYweKDb8Fd6str1p+cHp0E= 2024-04-12T04:09:20.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GORTwuYweKDb8Fd6str1p+cHp0E=', 'Content-Length': '0'} 2024-04-12T04:09:20.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9b9b6763811340b-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.515 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:09:20.516 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:09:20.516 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.516 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.516 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.516 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.517 DEBUG:boto:Data: 2024-04-12T04:09:20.517 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.517 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.517 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.517 DEBUG:boto:Params: {} 2024-04-12T04:09:20.518 DEBUG:boto:Token: None 2024-04-12T04:09:20.518 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.518 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cVOyptsmjP39aaXQlYAUH8lfVfA= 2024-04-12T04:09:20.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cVOyptsmjP39aaXQlYAUH8lfVfA=', 'Content-Length': '0'} 2024-04-12T04:09:20.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bbacce9badb6825-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.521 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:09:20.521 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:09:20.522 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.522 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.522 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.522 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.522 DEBUG:boto:Data: 2024-04-12T04:09:20.523 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.523 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.523 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.523 DEBUG:boto:Params: {} 2024-04-12T04:09:20.524 DEBUG:boto:Token: None 2024-04-12T04:09:20.524 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.524 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cVOyptsmjP39aaXQlYAUH8lfVfA= 2024-04-12T04:09:20.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cVOyptsmjP39aaXQlYAUH8lfVfA=', 'Content-Length': '0'} 2024-04-12T04:09:20.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a55755187bccd95e-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T04:09:20.527 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:09:20.527 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:09:20.527 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.528 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.528 DEBUG:boto:Method: GET 2024-04-12T04:09:20.528 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.528 DEBUG:boto:Data: 2024-04-12T04:09:20.529 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.529 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.529 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.529 DEBUG:boto:Params: {} 2024-04-12T04:09:20.529 DEBUG:boto:Token: None 2024-04-12T04:09:20.530 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.530 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GORTwuYweKDb8Fd6str1p+cHp0E= 2024-04-12T04:09:20.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GORTwuYweKDb8Fd6str1p+cHp0E=', 'Content-Length': '0'} 2024-04-12T04:09:20.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d32107f06e9fc612-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.533 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:09:20.533 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:09:20.533 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.533 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.534 DEBUG:boto:Method: GET 2024-04-12T04:09:20.534 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.534 DEBUG:boto:Data: 2024-04-12T04:09:20.534 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.534 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.534 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.535 DEBUG:boto:Params: {} 2024-04-12T04:09:20.535 DEBUG:boto:Token: None 2024-04-12T04:09:20.535 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:09:20.535 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GORTwuYweKDb8Fd6str1p+cHp0E= 2024-04-12T04:09:20.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GORTwuYweKDb8Fd6str1p+cHp0E=', 'Content-Length': '0'} 2024-04-12T04:09:20.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b950e9b9f900890a-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.582 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.583 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.583 DEBUG:boto:path=/dkyges-61/?prefix=obj155 2024-04-12T04:09:20.583 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj155 2024-04-12T04:09:20.583 DEBUG:boto:Method: GET 2024-04-12T04:09:20.583 DEBUG:boto:Path: /dkyges-61/?prefix=obj155 2024-04-12T04:09:20.583 DEBUG:boto:Data: 2024-04-12T04:09:20.583 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.583 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.583 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.583 DEBUG:boto:Params: {} 2024-04-12T04:09:20.584 DEBUG:boto:Token: None 2024-04-12T04:09:20.584 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.584 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e49a15ccedb5f0e1-006618b3f0-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.587 DEBUG:boto:b'dkyges-61obj1551000falseobj1552024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.587 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.587 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.587 DEBUG:boto:path=/dkyges-61/?prefix=obj155 2024-04-12T04:09:20.587 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj155 2024-04-12T04:09:20.587 DEBUG:boto:Method: GET 2024-04-12T04:09:20.587 DEBUG:boto:Path: /dkyges-61/?prefix=obj155 2024-04-12T04:09:20.588 DEBUG:boto:Data: 2024-04-12T04:09:20.588 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.588 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.588 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.588 DEBUG:boto:Params: {} 2024-04-12T04:09:20.588 DEBUG:boto:Token: None 2024-04-12T04:09:20.588 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.588 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.592 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de14f074e91cd51b-006618b3f0-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.592 DEBUG:boto:b'dkyges-61obj1551000falseobj1552024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.593 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:09:20.593 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:09:20.593 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.593 DEBUG:boto:Path: /dkyges-61/obj155 2024-04-12T04:09:20.593 DEBUG:boto:Data: 2024-04-12T04:09:20.593 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.593 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.593 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.593 DEBUG:boto:Params: {} 2024-04-12T04:09:20.593 DEBUG:boto:Token: None 2024-04-12T04:09:20.593 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj155 2024-04-12T04:09:20.594 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MjytJRuW1kyl7L8u8Jrglw8haEI= 2024-04-12T04:09:20.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MjytJRuW1kyl7L8u8Jrglw8haEI=', 'Content-Length': '0'} 2024-04-12T04:09:20.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009603249df8a7d516-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.596 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:09:20.596 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:09:20.596 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.596 DEBUG:boto:Path: /dkyges-61/obj155 2024-04-12T04:09:20.596 DEBUG:boto:Data: 2024-04-12T04:09:20.596 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.596 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.596 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.596 DEBUG:boto:Params: {} 2024-04-12T04:09:20.596 DEBUG:boto:Token: None 2024-04-12T04:09:20.597 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj155 2024-04-12T04:09:20.597 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MjytJRuW1kyl7L8u8Jrglw8haEI= 2024-04-12T04:09:20.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MjytJRuW1kyl7L8u8Jrglw8haEI=', 'Content-Length': '0'} 2024-04-12T04:09:20.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006515df56e99d826e-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T04:09:20.599 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:09:20.599 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:09:20.599 DEBUG:boto:Method: GET 2024-04-12T04:09:20.599 DEBUG:boto:Path: /dkyges-61/obj155 2024-04-12T04:09:20.599 DEBUG:boto:Data: 2024-04-12T04:09:20.599 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.599 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.599 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.599 DEBUG:boto:Params: {} 2024-04-12T04:09:20.599 DEBUG:boto:Token: None 2024-04-12T04:09:20.600 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj155 2024-04-12T04:09:20.600 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tH5AnphbyKMqKyYC6H0G4L6MoqA= 2024-04-12T04:09:20.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tH5AnphbyKMqKyYC6H0G4L6MoqA=', 'Content-Length': '0'} 2024-04-12T04:09:20.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bc866bab480d835-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.602 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:09:20.602 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:09:20.602 DEBUG:boto:Method: GET 2024-04-12T04:09:20.602 DEBUG:boto:Path: /dkyges-61/obj155 2024-04-12T04:09:20.602 DEBUG:boto:Data: 2024-04-12T04:09:20.602 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.603 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.603 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.603 DEBUG:boto:Params: {} 2024-04-12T04:09:20.603 DEBUG:boto:Token: None 2024-04-12T04:09:20.603 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj155 2024-04-12T04:09:20.603 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tH5AnphbyKMqKyYC6H0G4L6MoqA= 2024-04-12T04:09:20.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tH5AnphbyKMqKyYC6H0G4L6MoqA=', 'Content-Length': '0'} 2024-04-12T04:09:20.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dc4bcd567808875-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.646 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.646 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.647 DEBUG:boto:path=/dkyges-61/?prefix=obj155 2024-04-12T04:09:20.647 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj155 2024-04-12T04:09:20.647 DEBUG:boto:Method: GET 2024-04-12T04:09:20.647 DEBUG:boto:Path: /dkyges-61/?prefix=obj155 2024-04-12T04:09:20.647 DEBUG:boto:Data: 2024-04-12T04:09:20.647 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.647 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.647 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.647 DEBUG:boto:Params: {} 2024-04-12T04:09:20.647 DEBUG:boto:Token: None 2024-04-12T04:09:20.647 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.648 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bf05c3052a8384c-006618b3f0-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.650 DEBUG:boto:b'dkyges-61obj1551000falseobj1552024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.651 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.651 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.651 DEBUG:boto:path=/dkyges-61/?prefix=obj155 2024-04-12T04:09:20.651 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj155 2024-04-12T04:09:20.651 DEBUG:boto:Method: GET 2024-04-12T04:09:20.651 DEBUG:boto:Path: /dkyges-61/?prefix=obj155 2024-04-12T04:09:20.651 DEBUG:boto:Data: 2024-04-12T04:09:20.651 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.651 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.651 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.651 DEBUG:boto:Params: {} 2024-04-12T04:09:20.652 DEBUG:boto:Token: None 2024-04-12T04:09:20.652 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.652 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000556e2791901842c8-006618b3f0-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.656 DEBUG:boto:b'dkyges-61obj1551000falseobj1552024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:09:20.657 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:09:20.657 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:09:20.657 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.657 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.657 DEBUG:boto:Method: GET 2024-04-12T04:09:20.657 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.657 DEBUG:boto:Data: 2024-04-12T04:09:20.657 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.657 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.657 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.657 DEBUG:boto:Params: {} 2024-04-12T04:09:20.658 DEBUG:boto:Token: None 2024-04-12T04:09:20.658 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.658 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vNEiWCpUG8JflmgN/5dcrC/ddFg= 2024-04-12T04:09:20.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vNEiWCpUG8JflmgN/5dcrC/ddFg=', 'Content-Length': '0'} 2024-04-12T04:09:20.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d8332edcafd34c7-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.660 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:09:20.660 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:09:20.660 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.660 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.660 DEBUG:boto:Method: GET 2024-04-12T04:09:20.660 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.660 DEBUG:boto:Data: 2024-04-12T04:09:20.661 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.661 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.661 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.661 DEBUG:boto:Params: {} 2024-04-12T04:09:20.661 DEBUG:boto:Token: None 2024-04-12T04:09:20.661 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.661 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vNEiWCpUG8JflmgN/5dcrC/ddFg= 2024-04-12T04:09:20.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vNEiWCpUG8JflmgN/5dcrC/ddFg=', 'Content-Length': '0'} 2024-04-12T04:09:20.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f668ca46d9d09c69-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.663 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:09:20.663 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:09:20.663 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.663 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.663 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.664 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.664 DEBUG:boto:Data: 2024-04-12T04:09:20.664 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.664 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.664 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.664 DEBUG:boto:Params: {} 2024-04-12T04:09:20.664 DEBUG:boto:Token: None 2024-04-12T04:09:20.664 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.664 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rK/XuO48I6U2KBpYqPmuAl2H62k= 2024-04-12T04:09:20.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rK/XuO48I6U2KBpYqPmuAl2H62k=', 'Content-Length': '0'} 2024-04-12T04:09:20.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d223a774fd02aedb-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.666 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:09:20.667 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:09:20.667 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.667 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.667 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.667 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.667 DEBUG:boto:Data: 2024-04-12T04:09:20.667 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.667 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.667 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.667 DEBUG:boto:Params: {} 2024-04-12T04:09:20.667 DEBUG:boto:Token: None 2024-04-12T04:09:20.668 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.668 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rK/XuO48I6U2KBpYqPmuAl2H62k= 2024-04-12T04:09:20.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rK/XuO48I6U2KBpYqPmuAl2H62k=', 'Content-Length': '0'} 2024-04-12T04:09:20.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c23359cfe9d07c2a-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:09:20.670 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:09:20.670 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:09:20.670 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.670 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.670 DEBUG:boto:Method: GET 2024-04-12T04:09:20.670 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.670 DEBUG:boto:Data: 2024-04-12T04:09:20.670 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.670 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.671 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.671 DEBUG:boto:Params: {} 2024-04-12T04:09:20.671 DEBUG:boto:Token: None 2024-04-12T04:09:20.671 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.671 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vNEiWCpUG8JflmgN/5dcrC/ddFg= 2024-04-12T04:09:20.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vNEiWCpUG8JflmgN/5dcrC/ddFg=', 'Content-Length': '0'} 2024-04-12T04:09:20.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002faa587a7284f719-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.673 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:09:20.673 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:09:20.673 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.673 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.673 DEBUG:boto:Method: GET 2024-04-12T04:09:20.674 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.674 DEBUG:boto:Data: 2024-04-12T04:09:20.674 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.674 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.674 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.674 DEBUG:boto:Params: {} 2024-04-12T04:09:20.674 DEBUG:boto:Token: None 2024-04-12T04:09:20.674 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:09:20.674 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vNEiWCpUG8JflmgN/5dcrC/ddFg= 2024-04-12T04:09:20.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vNEiWCpUG8JflmgN/5dcrC/ddFg=', 'Content-Length': '0'} 2024-04-12T04:09:20.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078a4700604dd37b9-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.718 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.718 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.719 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:09:20.719 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:09:20.719 DEBUG:boto:Method: GET 2024-04-12T04:09:20.719 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:09:20.719 DEBUG:boto:Data: 2024-04-12T04:09:20.719 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.719 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.719 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.719 DEBUG:boto:Params: {} 2024-04-12T04:09:20.719 DEBUG:boto:Token: None 2024-04-12T04:09:20.720 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.720 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095f7cb35269f41b0-006618b3f0-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.722 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.723 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.723 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.723 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:09:20.723 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:09:20.723 DEBUG:boto:Method: GET 2024-04-12T04:09:20.723 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:09:20.723 DEBUG:boto:Data: 2024-04-12T04:09:20.723 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.724 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.724 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.724 DEBUG:boto:Params: {} 2024-04-12T04:09:20.724 DEBUG:boto:Token: None 2024-04-12T04:09:20.724 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.724 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000183273ec4fe7e255-006618b3f0-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.728 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.729 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:09:20.729 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:09:20.729 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.729 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:09:20.729 DEBUG:boto:Data: 2024-04-12T04:09:20.729 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.729 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.729 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.729 DEBUG:boto:Params: {} 2024-04-12T04:09:20.729 DEBUG:boto:Token: None 2024-04-12T04:09:20.730 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj17 2024-04-12T04:09:20.730 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RaTJzMvkgdnM0mGmDuN+/EgNUro= 2024-04-12T04:09:20.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RaTJzMvkgdnM0mGmDuN+/EgNUro=', 'Content-Length': '0'} 2024-04-12T04:09:20.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009add95f2a7cb1ba3-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.732 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:09:20.732 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:09:20.732 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.732 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:09:20.732 DEBUG:boto:Data: 2024-04-12T04:09:20.732 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.732 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.733 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.733 DEBUG:boto:Params: {} 2024-04-12T04:09:20.733 DEBUG:boto:Token: None 2024-04-12T04:09:20.733 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj17 2024-04-12T04:09:20.733 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RaTJzMvkgdnM0mGmDuN+/EgNUro= 2024-04-12T04:09:20.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RaTJzMvkgdnM0mGmDuN+/EgNUro=', 'Content-Length': '0'} 2024-04-12T04:09:20.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000547b59478a58c26b-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:09:20.735 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:09:20.735 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:09:20.736 DEBUG:boto:Method: GET 2024-04-12T04:09:20.736 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:09:20.736 DEBUG:boto:Data: 2024-04-12T04:09:20.736 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.736 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.736 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.736 DEBUG:boto:Params: {} 2024-04-12T04:09:20.736 DEBUG:boto:Token: None 2024-04-12T04:09:20.736 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj17 2024-04-12T04:09:20.736 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:F9eZE25HM7Z20bhNWWwpcxw/OAo= 2024-04-12T04:09:20.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:F9eZE25HM7Z20bhNWWwpcxw/OAo=', 'Content-Length': '0'} 2024-04-12T04:09:20.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001090d55fe70e2064-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.778 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:09:20.778 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:09:20.779 DEBUG:boto:Method: GET 2024-04-12T04:09:20.779 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:09:20.779 DEBUG:boto:Data: 2024-04-12T04:09:20.779 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.779 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.779 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.779 DEBUG:boto:Params: {} 2024-04-12T04:09:20.779 DEBUG:boto:Token: None 2024-04-12T04:09:20.779 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj17 2024-04-12T04:09:20.779 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:F9eZE25HM7Z20bhNWWwpcxw/OAo= 2024-04-12T04:09:20.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:F9eZE25HM7Z20bhNWWwpcxw/OAo=', 'Content-Length': '0'} 2024-04-12T04:09:20.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003abaea401fe777af-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.782 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.782 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.782 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:09:20.782 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:09:20.782 DEBUG:boto:Method: GET 2024-04-12T04:09:20.783 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:09:20.783 DEBUG:boto:Data: 2024-04-12T04:09:20.783 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.783 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.783 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.783 DEBUG:boto:Params: {} 2024-04-12T04:09:20.783 DEBUG:boto:Token: None 2024-04-12T04:09:20.783 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.783 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000240e6f324f1362bb-006618b3f0-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.786 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.786 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.786 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.786 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:09:20.787 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:09:20.787 DEBUG:boto:Method: GET 2024-04-12T04:09:20.787 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:09:20.787 DEBUG:boto:Data: 2024-04-12T04:09:20.787 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.787 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.787 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.787 DEBUG:boto:Params: {} 2024-04-12T04:09:20.787 DEBUG:boto:Token: None 2024-04-12T04:09:20.787 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.787 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a4ed58c5210aab0-006618b3f0-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.791 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:09:20.792 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:09:20.792 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:09:20.792 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.792 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.792 DEBUG:boto:Method: GET 2024-04-12T04:09:20.792 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.792 DEBUG:boto:Data: 2024-04-12T04:09:20.792 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.792 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.793 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.793 DEBUG:boto:Params: {} 2024-04-12T04:09:20.793 DEBUG:boto:Token: None 2024-04-12T04:09:20.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.793 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZWS4ku5Ryj7IhMSM+OUONDXAK/E= 2024-04-12T04:09:20.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZWS4ku5Ryj7IhMSM+OUONDXAK/E=', 'Content-Length': '0'} 2024-04-12T04:09:20.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079fbf3fe30b1886f-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.795 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:09:20.795 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:09:20.795 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.795 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.795 DEBUG:boto:Method: GET 2024-04-12T04:09:20.795 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.796 DEBUG:boto:Data: 2024-04-12T04:09:20.796 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.796 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.796 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.796 DEBUG:boto:Params: {} 2024-04-12T04:09:20.796 DEBUG:boto:Token: None 2024-04-12T04:09:20.796 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.796 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZWS4ku5Ryj7IhMSM+OUONDXAK/E= 2024-04-12T04:09:20.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZWS4ku5Ryj7IhMSM+OUONDXAK/E=', 'Content-Length': '0'} 2024-04-12T04:09:20.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027da3ce3b23eb06f-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.838 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:09:20.838 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:09:20.838 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.839 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.839 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.839 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.839 DEBUG:boto:Data: 2024-04-12T04:09:20.839 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.839 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.839 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.839 DEBUG:boto:Params: {} 2024-04-12T04:09:20.839 DEBUG:boto:Token: None 2024-04-12T04:09:20.839 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.839 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lD1D/L1k1KqQgqiHNQolA4Twp/4= 2024-04-12T04:09:20.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lD1D/L1k1KqQgqiHNQolA4Twp/4=', 'Content-Length': '0'} 2024-04-12T04:09:20.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e29b1ecee19e4dc3-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.842 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:09:20.842 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:09:20.842 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.842 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.842 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.842 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.842 DEBUG:boto:Data: 2024-04-12T04:09:20.842 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.842 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.842 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.843 DEBUG:boto:Params: {} 2024-04-12T04:09:20.843 DEBUG:boto:Token: None 2024-04-12T04:09:20.843 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.843 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lD1D/L1k1KqQgqiHNQolA4Twp/4= 2024-04-12T04:09:20.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lD1D/L1k1KqQgqiHNQolA4Twp/4=', 'Content-Length': '0'} 2024-04-12T04:09:20.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0838363b2439db0-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:09:20.845 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:09:20.845 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:09:20.845 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.845 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.845 DEBUG:boto:Method: GET 2024-04-12T04:09:20.846 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.846 DEBUG:boto:Data: 2024-04-12T04:09:20.846 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.846 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.846 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.846 DEBUG:boto:Params: {} 2024-04-12T04:09:20.846 DEBUG:boto:Token: None 2024-04-12T04:09:20.846 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.846 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZWS4ku5Ryj7IhMSM+OUONDXAK/E= 2024-04-12T04:09:20.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZWS4ku5Ryj7IhMSM+OUONDXAK/E=', 'Content-Length': '0'} 2024-04-12T04:09:20.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bb4c5b6036ead45-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.849 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:09:20.849 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:09:20.849 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.849 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.849 DEBUG:boto:Method: GET 2024-04-12T04:09:20.849 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.849 DEBUG:boto:Data: 2024-04-12T04:09:20.849 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.849 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.849 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.849 DEBUG:boto:Params: {} 2024-04-12T04:09:20.849 DEBUG:boto:Token: None 2024-04-12T04:09:20.850 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:09:20.850 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZWS4ku5Ryj7IhMSM+OUONDXAK/E= 2024-04-12T04:09:20.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZWS4ku5Ryj7IhMSM+OUONDXAK/E=', 'Content-Length': '0'} 2024-04-12T04:09:20.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4061b30a9b0735d-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.852 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.852 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.852 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:09:20.852 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:09:20.852 DEBUG:boto:Method: GET 2024-04-12T04:09:20.852 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:09:20.852 DEBUG:boto:Data: 2024-04-12T04:09:20.852 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.852 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.853 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.853 DEBUG:boto:Params: {} 2024-04-12T04:09:20.853 DEBUG:boto:Token: None 2024-04-12T04:09:20.853 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.853 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c945f01a7818561a-006618b3f0-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.856 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.856 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.856 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.856 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:09:20.856 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:09:20.856 DEBUG:boto:Method: GET 2024-04-12T04:09:20.856 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:09:20.857 DEBUG:boto:Data: 2024-04-12T04:09:20.857 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.857 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.857 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.857 DEBUG:boto:Params: {} 2024-04-12T04:09:20.857 DEBUG:boto:Token: None 2024-04-12T04:09:20.857 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.857 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb0c464859873e1a-006618b3f0-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.861 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.862 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:09:20.862 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:09:20.862 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.862 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:09:20.862 DEBUG:boto:Data: 2024-04-12T04:09:20.862 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.862 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.862 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.862 DEBUG:boto:Params: {} 2024-04-12T04:09:20.862 DEBUG:boto:Token: None 2024-04-12T04:09:20.863 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj19 2024-04-12T04:09:20.863 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PYisZJXJuAR+3ftCRgtf+iczmys= 2024-04-12T04:09:20.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PYisZJXJuAR+3ftCRgtf+iczmys=', 'Content-Length': '0'} 2024-04-12T04:09:20.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008803598507c94680-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.864 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:09:20.865 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:09:20.865 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.865 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:09:20.865 DEBUG:boto:Data: 2024-04-12T04:09:20.865 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.865 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.865 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.865 DEBUG:boto:Params: {} 2024-04-12T04:09:20.865 DEBUG:boto:Token: None 2024-04-12T04:09:20.865 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj19 2024-04-12T04:09:20.866 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PYisZJXJuAR+3ftCRgtf+iczmys= 2024-04-12T04:09:20.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PYisZJXJuAR+3ftCRgtf+iczmys=', 'Content-Length': '0'} 2024-04-12T04:09:20.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ece6b2ae0822756e-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:09:20.868 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:09:20.868 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:09:20.868 DEBUG:boto:Method: GET 2024-04-12T04:09:20.868 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:09:20.868 DEBUG:boto:Data: 2024-04-12T04:09:20.868 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.868 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.868 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.868 DEBUG:boto:Params: {} 2024-04-12T04:09:20.869 DEBUG:boto:Token: None 2024-04-12T04:09:20.869 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj19 2024-04-12T04:09:20.869 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ARzwiSdz1NZMq9dksjiVZn61GUQ= 2024-04-12T04:09:20.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ARzwiSdz1NZMq9dksjiVZn61GUQ=', 'Content-Length': '0'} 2024-04-12T04:09:20.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c523272e793979a4-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.871 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:09:20.871 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:09:20.871 DEBUG:boto:Method: GET 2024-04-12T04:09:20.871 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:09:20.871 DEBUG:boto:Data: 2024-04-12T04:09:20.871 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.871 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.871 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.871 DEBUG:boto:Params: {} 2024-04-12T04:09:20.872 DEBUG:boto:Token: None 2024-04-12T04:09:20.872 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj19 2024-04-12T04:09:20.872 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ARzwiSdz1NZMq9dksjiVZn61GUQ= 2024-04-12T04:09:20.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ARzwiSdz1NZMq9dksjiVZn61GUQ=', 'Content-Length': '0'} 2024-04-12T04:09:20.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d7434dcdaa6bb85-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.914 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.915 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.915 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:09:20.915 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:09:20.915 DEBUG:boto:Method: GET 2024-04-12T04:09:20.915 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:09:20.915 DEBUG:boto:Data: 2024-04-12T04:09:20.915 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.915 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.915 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.915 DEBUG:boto:Params: {} 2024-04-12T04:09:20.915 DEBUG:boto:Token: None 2024-04-12T04:09:20.916 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.916 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a3ebfd5124ec552-006618b3f0-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.918 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.919 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.919 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.919 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:09:20.919 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:09:20.919 DEBUG:boto:Method: GET 2024-04-12T04:09:20.919 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:09:20.919 DEBUG:boto:Data: 2024-04-12T04:09:20.919 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.919 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.919 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.920 DEBUG:boto:Params: {} 2024-04-12T04:09:20.920 DEBUG:boto:Token: None 2024-04-12T04:09:20.920 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.920 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f187cafe4e3729a-006618b3f0-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.924 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:09:20.925 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:09:20.925 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:09:20.925 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.925 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.925 DEBUG:boto:Method: GET 2024-04-12T04:09:20.925 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.925 DEBUG:boto:Data: 2024-04-12T04:09:20.925 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.925 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.925 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.925 DEBUG:boto:Params: {} 2024-04-12T04:09:20.926 DEBUG:boto:Token: None 2024-04-12T04:09:20.926 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.926 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v7KNov+aixXUkJlST1zXP/nZcZ0= 2024-04-12T04:09:20.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v7KNov+aixXUkJlST1zXP/nZcZ0=', 'Content-Length': '0'} 2024-04-12T04:09:20.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062c9e1415ca10d5c-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.974 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:09:20.974 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:09:20.974 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.975 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.975 DEBUG:boto:Method: GET 2024-04-12T04:09:20.975 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.975 DEBUG:boto:Data: 2024-04-12T04:09:20.975 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.975 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.975 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.975 DEBUG:boto:Params: {} 2024-04-12T04:09:20.975 DEBUG:boto:Token: None 2024-04-12T04:09:20.975 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.976 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v7KNov+aixXUkJlST1zXP/nZcZ0= 2024-04-12T04:09:20.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v7KNov+aixXUkJlST1zXP/nZcZ0=', 'Content-Length': '0'} 2024-04-12T04:09:20.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e45632621cc71dd-006618b3f0-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.978 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:09:20.978 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:09:20.978 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.978 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.978 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.979 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.979 DEBUG:boto:Data: 2024-04-12T04:09:20.979 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.979 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.979 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.979 DEBUG:boto:Params: {} 2024-04-12T04:09:20.979 DEBUG:boto:Token: None 2024-04-12T04:09:20.979 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.979 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JBirC/9JqSlslS7zwUahmLQYWOY= 2024-04-12T04:09:20.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JBirC/9JqSlslS7zwUahmLQYWOY=', 'Content-Length': '0'} 2024-04-12T04:09:20.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010b222a8fb3ad2a0-006618b3f0-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.982 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:09:20.982 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:09:20.982 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.982 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.982 DEBUG:boto:Method: HEAD 2024-04-12T04:09:20.982 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.982 DEBUG:boto:Data: 2024-04-12T04:09:20.982 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.983 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.983 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.983 DEBUG:boto:Params: {} 2024-04-12T04:09:20.983 DEBUG:boto:Token: None 2024-04-12T04:09:20.983 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.983 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JBirC/9JqSlslS7zwUahmLQYWOY= 2024-04-12T04:09:20.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JBirC/9JqSlslS7zwUahmLQYWOY=', 'Content-Length': '0'} 2024-04-12T04:09:20.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000147039b455f8624a-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:09:20.986 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:09:20.986 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:09:20.986 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.986 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.986 DEBUG:boto:Method: GET 2024-04-12T04:09:20.986 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.986 DEBUG:boto:Data: 2024-04-12T04:09:20.986 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.986 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.987 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.987 DEBUG:boto:Params: {} 2024-04-12T04:09:20.987 DEBUG:boto:Token: None 2024-04-12T04:09:20.987 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.987 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v7KNov+aixXUkJlST1zXP/nZcZ0= 2024-04-12T04:09:20.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v7KNov+aixXUkJlST1zXP/nZcZ0=', 'Content-Length': '0'} 2024-04-12T04:09:20.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd20e56500e7cd61-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.990 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:09:20.990 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:09:20.990 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.990 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.990 DEBUG:boto:Method: GET 2024-04-12T04:09:20.990 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.990 DEBUG:boto:Data: 2024-04-12T04:09:20.990 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.990 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.990 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.990 DEBUG:boto:Params: {} 2024-04-12T04:09:20.991 DEBUG:boto:Token: None 2024-04-12T04:09:20.991 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:09:20.991 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v7KNov+aixXUkJlST1zXP/nZcZ0= 2024-04-12T04:09:20.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v7KNov+aixXUkJlST1zXP/nZcZ0=', 'Content-Length': '0'} 2024-04-12T04:09:20.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a35071595f458d5-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.993 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.993 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.993 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:09:20.994 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:09:20.994 DEBUG:boto:Method: GET 2024-04-12T04:09:20.994 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:09:20.994 DEBUG:boto:Data: 2024-04-12T04:09:20.994 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.994 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:20.994 DEBUG:boto:Port: 8001 2024-04-12T04:09:20.994 DEBUG:boto:Params: {} 2024-04-12T04:09:20.994 DEBUG:boto:Token: None 2024-04-12T04:09:20.994 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.995 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:20.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082a94913e5566bb3-006618b3f1-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:20.997 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:20.997 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:20.998 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:20.998 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:09:20.998 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:09:20.998 DEBUG:boto:Method: GET 2024-04-12T04:09:20.998 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:09:20.998 DEBUG:boto:Data: 2024-04-12T04:09:20.998 DEBUG:boto:Headers: {} 2024-04-12T04:09:20.998 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:20.998 DEBUG:boto:Port: 8000 2024-04-12T04:09:20.998 DEBUG:boto:Params: {} 2024-04-12T04:09:20.998 DEBUG:boto:Token: None 2024-04-12T04:09:20.999 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:20 GMT /dkyges-61/ 2024-04-12T04:09:20.999 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE= 2024-04-12T04:09:20.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FiBD3u6JKyezmarpBBht7QniNoE=', 'Content-Length': '0'} 2024-04-12T04:09:21.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038d3edda5ffa4e3f-006618b3f1-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.003 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.003 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:09:21.004 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:09:21.004 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.004 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:09:21.004 DEBUG:boto:Data: 2024-04-12T04:09:21.004 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.004 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.004 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.004 DEBUG:boto:Params: {} 2024-04-12T04:09:21.004 DEBUG:boto:Token: None 2024-04-12T04:09:21.004 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj22 2024-04-12T04:09:21.005 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EMQX5XTn3sNcE9LI7GA74moMohI= 2024-04-12T04:09:21.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EMQX5XTn3sNcE9LI7GA74moMohI=', 'Content-Length': '0'} 2024-04-12T04:09:21.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4d38928245877f1-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.006 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:09:21.007 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:09:21.007 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.007 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:09:21.007 DEBUG:boto:Data: 2024-04-12T04:09:21.007 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.007 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.007 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.007 DEBUG:boto:Params: {} 2024-04-12T04:09:21.007 DEBUG:boto:Token: None 2024-04-12T04:09:21.008 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj22 2024-04-12T04:09:21.008 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EMQX5XTn3sNcE9LI7GA74moMohI= 2024-04-12T04:09:21.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EMQX5XTn3sNcE9LI7GA74moMohI=', 'Content-Length': '0'} 2024-04-12T04:09:21.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c971b6496b2f92c-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:09:21.010 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:09:21.010 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:09:21.010 DEBUG:boto:Method: GET 2024-04-12T04:09:21.010 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:09:21.010 DEBUG:boto:Data: 2024-04-12T04:09:21.010 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.011 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.011 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.011 DEBUG:boto:Params: {} 2024-04-12T04:09:21.011 DEBUG:boto:Token: None 2024-04-12T04:09:21.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj22 2024-04-12T04:09:21.011 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:df33zcT4v7oUz8AoVmYnktf/1Nw= 2024-04-12T04:09:21.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:df33zcT4v7oUz8AoVmYnktf/1Nw=', 'Content-Length': '0'} 2024-04-12T04:09:21.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006be461fe8e2b45fa-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.013 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:09:21.013 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:09:21.013 DEBUG:boto:Method: GET 2024-04-12T04:09:21.014 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:09:21.014 DEBUG:boto:Data: 2024-04-12T04:09:21.014 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.014 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.014 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.014 DEBUG:boto:Params: {} 2024-04-12T04:09:21.017 DEBUG:boto:Token: None 2024-04-12T04:09:21.017 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj22 2024-04-12T04:09:21.017 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:df33zcT4v7oUz8AoVmYnktf/1Nw= 2024-04-12T04:09:21.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:df33zcT4v7oUz8AoVmYnktf/1Nw=', 'Content-Length': '0'} 2024-04-12T04:09:21.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006561169a7ca34864-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.019 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.019 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.019 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:09:21.020 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:09:21.020 DEBUG:boto:Method: GET 2024-04-12T04:09:21.020 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:09:21.020 DEBUG:boto:Data: 2024-04-12T04:09:21.020 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.020 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.020 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.020 DEBUG:boto:Params: {} 2024-04-12T04:09:21.020 DEBUG:boto:Token: None 2024-04-12T04:09:21.020 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.021 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aef3c489d00c2e89-006618b3f1-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.023 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.024 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.024 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.024 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:09:21.024 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:09:21.024 DEBUG:boto:Method: GET 2024-04-12T04:09:21.024 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:09:21.024 DEBUG:boto:Data: 2024-04-12T04:09:21.024 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.024 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.025 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.025 DEBUG:boto:Params: {} 2024-04-12T04:09:21.025 DEBUG:boto:Token: None 2024-04-12T04:09:21.025 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.025 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000226c82cafa4c5649-006618b3f1-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.074 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:09:21.075 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:09:21.075 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:09:21.075 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.075 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.076 DEBUG:boto:Method: GET 2024-04-12T04:09:21.076 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.076 DEBUG:boto:Data: 2024-04-12T04:09:21.076 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.076 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.076 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.076 DEBUG:boto:Params: {} 2024-04-12T04:09:21.076 DEBUG:boto:Token: None 2024-04-12T04:09:21.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.077 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QPhdxDdx+J/mc79dhqWF9hPc6uQ= 2024-04-12T04:09:21.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QPhdxDdx+J/mc79dhqWF9hPc6uQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8dc2ada7995f3a6-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.079 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:09:21.079 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:09:21.079 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.080 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.080 DEBUG:boto:Method: GET 2024-04-12T04:09:21.080 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.080 DEBUG:boto:Data: 2024-04-12T04:09:21.080 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.080 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.080 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.080 DEBUG:boto:Params: {} 2024-04-12T04:09:21.080 DEBUG:boto:Token: None 2024-04-12T04:09:21.080 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.081 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QPhdxDdx+J/mc79dhqWF9hPc6uQ= 2024-04-12T04:09:21.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QPhdxDdx+J/mc79dhqWF9hPc6uQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbf1a4a69b1e2d4f-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.083 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:09:21.083 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:09:21.083 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.083 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.083 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.084 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.084 DEBUG:boto:Data: 2024-04-12T04:09:21.084 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.084 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.084 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.084 DEBUG:boto:Params: {} 2024-04-12T04:09:21.084 DEBUG:boto:Token: None 2024-04-12T04:09:21.084 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.085 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zL55n4PAyVa4tjKrLmDHFIaHAGc= 2024-04-12T04:09:21.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zL55n4PAyVa4tjKrLmDHFIaHAGc=', 'Content-Length': '0'} 2024-04-12T04:09:21.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c506bcf766b093b-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.087 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:09:21.087 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:09:21.087 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.087 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.087 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.087 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.087 DEBUG:boto:Data: 2024-04-12T04:09:21.087 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.087 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.087 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.087 DEBUG:boto:Params: {} 2024-04-12T04:09:21.088 DEBUG:boto:Token: None 2024-04-12T04:09:21.088 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.088 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zL55n4PAyVa4tjKrLmDHFIaHAGc= 2024-04-12T04:09:21.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zL55n4PAyVa4tjKrLmDHFIaHAGc=', 'Content-Length': '0'} 2024-04-12T04:09:21.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5b6f9ba28b22bf0-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:09:21.091 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:09:21.091 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:09:21.091 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.091 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.091 DEBUG:boto:Method: GET 2024-04-12T04:09:21.091 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.091 DEBUG:boto:Data: 2024-04-12T04:09:21.091 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.091 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.092 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.092 DEBUG:boto:Params: {} 2024-04-12T04:09:21.092 DEBUG:boto:Token: None 2024-04-12T04:09:21.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.092 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QPhdxDdx+J/mc79dhqWF9hPc6uQ= 2024-04-12T04:09:21.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QPhdxDdx+J/mc79dhqWF9hPc6uQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdbccca3754da192-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.134 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:09:21.134 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:09:21.135 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.135 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.135 DEBUG:boto:Method: GET 2024-04-12T04:09:21.135 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.135 DEBUG:boto:Data: 2024-04-12T04:09:21.135 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.135 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.135 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.135 DEBUG:boto:Params: {} 2024-04-12T04:09:21.135 DEBUG:boto:Token: None 2024-04-12T04:09:21.136 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:09:21.136 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QPhdxDdx+J/mc79dhqWF9hPc6uQ= 2024-04-12T04:09:21.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QPhdxDdx+J/mc79dhqWF9hPc6uQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f1daebaeefef717-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.138 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.138 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.139 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:09:21.139 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:09:21.139 DEBUG:boto:Method: GET 2024-04-12T04:09:21.139 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:09:21.139 DEBUG:boto:Data: 2024-04-12T04:09:21.139 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.139 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.139 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.139 DEBUG:boto:Params: {} 2024-04-12T04:09:21.139 DEBUG:boto:Token: None 2024-04-12T04:09:21.140 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.140 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae7da4eabbb86b74-006618b3f1-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.142 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.143 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.143 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.143 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:09:21.143 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:09:21.143 DEBUG:boto:Method: GET 2024-04-12T04:09:21.143 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:09:21.143 DEBUG:boto:Data: 2024-04-12T04:09:21.143 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.143 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.144 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.144 DEBUG:boto:Params: {} 2024-04-12T04:09:21.144 DEBUG:boto:Token: None 2024-04-12T04:09:21.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.144 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba34b7634d17b973-006618b3f1-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.148 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.148 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:09:21.149 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:09:21.149 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.149 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:09:21.149 DEBUG:boto:Data: 2024-04-12T04:09:21.149 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.149 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.149 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.149 DEBUG:boto:Params: {} 2024-04-12T04:09:21.149 DEBUG:boto:Token: None 2024-04-12T04:09:21.149 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj23 2024-04-12T04:09:21.150 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NxTUM5juEVBpXyLVoDx/pzMD1Qk= 2024-04-12T04:09:21.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NxTUM5juEVBpXyLVoDx/pzMD1Qk=', 'Content-Length': '0'} 2024-04-12T04:09:21.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6086ed46f298ed5-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.152 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:09:21.152 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:09:21.152 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.152 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:09:21.152 DEBUG:boto:Data: 2024-04-12T04:09:21.152 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.152 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.152 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.153 DEBUG:boto:Params: {} 2024-04-12T04:09:21.153 DEBUG:boto:Token: None 2024-04-12T04:09:21.153 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj23 2024-04-12T04:09:21.153 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NxTUM5juEVBpXyLVoDx/pzMD1Qk= 2024-04-12T04:09:21.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NxTUM5juEVBpXyLVoDx/pzMD1Qk=', 'Content-Length': '0'} 2024-04-12T04:09:21.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de0ed6d9483edba2-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:09:21.155 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:09:21.155 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:09:21.155 DEBUG:boto:Method: GET 2024-04-12T04:09:21.155 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:09:21.155 DEBUG:boto:Data: 2024-04-12T04:09:21.155 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.156 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.156 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.156 DEBUG:boto:Params: {} 2024-04-12T04:09:21.156 DEBUG:boto:Token: None 2024-04-12T04:09:21.156 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj23 2024-04-12T04:09:21.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2H182481GOmBIJuJtq/rv6fd1gs= 2024-04-12T04:09:21.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2H182481GOmBIJuJtq/rv6fd1gs=', 'Content-Length': '0'} 2024-04-12T04:09:21.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053a54f7dc577d0d1-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.158 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:09:21.158 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:09:21.158 DEBUG:boto:Method: GET 2024-04-12T04:09:21.159 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:09:21.159 DEBUG:boto:Data: 2024-04-12T04:09:21.159 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.159 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.159 DEBUG:boto:Params: {} 2024-04-12T04:09:21.159 DEBUG:boto:Token: None 2024-04-12T04:09:21.159 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj23 2024-04-12T04:09:21.159 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2H182481GOmBIJuJtq/rv6fd1gs= 2024-04-12T04:09:21.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2H182481GOmBIJuJtq/rv6fd1gs=', 'Content-Length': '0'} 2024-04-12T04:09:21.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098bbb5a534389a8c-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.202 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.203 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.203 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:09:21.203 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:09:21.203 DEBUG:boto:Method: GET 2024-04-12T04:09:21.203 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:09:21.203 DEBUG:boto:Data: 2024-04-12T04:09:21.203 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.203 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.203 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.203 DEBUG:boto:Params: {} 2024-04-12T04:09:21.204 DEBUG:boto:Token: None 2024-04-12T04:09:21.204 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.204 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001edc56ffac36874e-006618b3f1-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.207 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.207 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.207 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.207 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:09:21.207 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:09:21.208 DEBUG:boto:Method: GET 2024-04-12T04:09:21.208 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:09:21.208 DEBUG:boto:Data: 2024-04-12T04:09:21.208 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.208 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.208 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.208 DEBUG:boto:Params: {} 2024-04-12T04:09:21.208 DEBUG:boto:Token: None 2024-04-12T04:09:21.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010577c21fdcf9a3d-006618b3f1-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.212 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:09:21.213 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:09:21.213 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:09:21.213 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.213 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.213 DEBUG:boto:Method: GET 2024-04-12T04:09:21.213 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.213 DEBUG:boto:Data: 2024-04-12T04:09:21.213 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.213 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.213 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.213 DEBUG:boto:Params: {} 2024-04-12T04:09:21.214 DEBUG:boto:Token: None 2024-04-12T04:09:21.214 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.214 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bZy3DRaWGsa8l1iChDMJu9iPQxQ= 2024-04-12T04:09:21.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bZy3DRaWGsa8l1iChDMJu9iPQxQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019329cce2edf5ab0-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.258 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:09:21.259 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:09:21.259 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.259 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.259 DEBUG:boto:Method: GET 2024-04-12T04:09:21.259 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.259 DEBUG:boto:Data: 2024-04-12T04:09:21.259 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.259 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.259 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.260 DEBUG:boto:Params: {} 2024-04-12T04:09:21.260 DEBUG:boto:Token: None 2024-04-12T04:09:21.260 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.260 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bZy3DRaWGsa8l1iChDMJu9iPQxQ= 2024-04-12T04:09:21.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bZy3DRaWGsa8l1iChDMJu9iPQxQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000454d1af87d184303-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.263 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:09:21.263 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:09:21.263 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.263 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.263 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.263 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.263 DEBUG:boto:Data: 2024-04-12T04:09:21.264 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.264 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.264 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.264 DEBUG:boto:Params: {} 2024-04-12T04:09:21.264 DEBUG:boto:Token: None 2024-04-12T04:09:21.264 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.264 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6gWKwuCCk80iE798ibauP+VsvtE= 2024-04-12T04:09:21.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6gWKwuCCk80iE798ibauP+VsvtE=', 'Content-Length': '0'} 2024-04-12T04:09:21.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000908e14537d7afa74-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.267 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:09:21.267 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:09:21.267 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.267 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.268 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.268 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.268 DEBUG:boto:Data: 2024-04-12T04:09:21.268 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.268 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.268 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.268 DEBUG:boto:Params: {} 2024-04-12T04:09:21.268 DEBUG:boto:Token: None 2024-04-12T04:09:21.268 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.269 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6gWKwuCCk80iE798ibauP+VsvtE= 2024-04-12T04:09:21.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6gWKwuCCk80iE798ibauP+VsvtE=', 'Content-Length': '0'} 2024-04-12T04:09:21.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f8aec51b518b4e5-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:09:21.271 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:09:21.271 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:09:21.271 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.271 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.271 DEBUG:boto:Method: GET 2024-04-12T04:09:21.271 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.271 DEBUG:boto:Data: 2024-04-12T04:09:21.272 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.272 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.272 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.272 DEBUG:boto:Params: {} 2024-04-12T04:09:21.272 DEBUG:boto:Token: None 2024-04-12T04:09:21.272 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.272 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bZy3DRaWGsa8l1iChDMJu9iPQxQ= 2024-04-12T04:09:21.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bZy3DRaWGsa8l1iChDMJu9iPQxQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba582fd27df956ad-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.275 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:09:21.275 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:09:21.275 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.275 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.275 DEBUG:boto:Method: GET 2024-04-12T04:09:21.276 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.276 DEBUG:boto:Data: 2024-04-12T04:09:21.276 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.276 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.276 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.276 DEBUG:boto:Params: {} 2024-04-12T04:09:21.277 DEBUG:boto:Token: None 2024-04-12T04:09:21.277 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:09:21.277 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bZy3DRaWGsa8l1iChDMJu9iPQxQ= 2024-04-12T04:09:21.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bZy3DRaWGsa8l1iChDMJu9iPQxQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000343c839010d17080-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.322 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.322 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.323 DEBUG:boto:path=/dkyges-61/?prefix=obj29 2024-04-12T04:09:21.323 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj29 2024-04-12T04:09:21.323 DEBUG:boto:Method: GET 2024-04-12T04:09:21.323 DEBUG:boto:Path: /dkyges-61/?prefix=obj29 2024-04-12T04:09:21.323 DEBUG:boto:Data: 2024-04-12T04:09:21.323 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.323 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.323 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.323 DEBUG:boto:Params: {} 2024-04-12T04:09:21.323 DEBUG:boto:Token: None 2024-04-12T04:09:21.324 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.324 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071398f9baf21d1d4-006618b3f1-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.326 DEBUG:boto:b'dkyges-61obj291000falseobj292024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.327 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.327 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.327 DEBUG:boto:path=/dkyges-61/?prefix=obj29 2024-04-12T04:09:21.327 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj29 2024-04-12T04:09:21.327 DEBUG:boto:Method: GET 2024-04-12T04:09:21.327 DEBUG:boto:Path: /dkyges-61/?prefix=obj29 2024-04-12T04:09:21.328 DEBUG:boto:Data: 2024-04-12T04:09:21.328 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.328 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.328 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.328 DEBUG:boto:Params: {} 2024-04-12T04:09:21.328 DEBUG:boto:Token: None 2024-04-12T04:09:21.328 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.328 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038d2066065f9f6f9-006618b3f1-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.332 DEBUG:boto:b'dkyges-61obj291000falseobj292024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.333 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:09:21.333 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:09:21.333 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.333 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:09:21.333 DEBUG:boto:Data: 2024-04-12T04:09:21.333 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.333 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.333 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.333 DEBUG:boto:Params: {} 2024-04-12T04:09:21.333 DEBUG:boto:Token: None 2024-04-12T04:09:21.334 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj29 2024-04-12T04:09:21.334 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OyKi6I2Wb+swPDa3grDrw2jzwcA= 2024-04-12T04:09:21.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OyKi6I2Wb+swPDa3grDrw2jzwcA=', 'Content-Length': '0'} 2024-04-12T04:09:21.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032dc09d5cca41153-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.336 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:09:21.336 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:09:21.336 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.336 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:09:21.336 DEBUG:boto:Data: 2024-04-12T04:09:21.336 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.336 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.336 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.336 DEBUG:boto:Params: {} 2024-04-12T04:09:21.337 DEBUG:boto:Token: None 2024-04-12T04:09:21.337 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj29 2024-04-12T04:09:21.337 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OyKi6I2Wb+swPDa3grDrw2jzwcA= 2024-04-12T04:09:21.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OyKi6I2Wb+swPDa3grDrw2jzwcA=', 'Content-Length': '0'} 2024-04-12T04:09:21.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000900704e7b98c40b1-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:09:21.339 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:09:21.339 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:09:21.339 DEBUG:boto:Method: GET 2024-04-12T04:09:21.339 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:09:21.339 DEBUG:boto:Data: 2024-04-12T04:09:21.340 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.340 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.340 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.340 DEBUG:boto:Params: {} 2024-04-12T04:09:21.340 DEBUG:boto:Token: None 2024-04-12T04:09:21.340 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj29 2024-04-12T04:09:21.340 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xC9k/c5mv4IDxXWVGozyylrslZs= 2024-04-12T04:09:21.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xC9k/c5mv4IDxXWVGozyylrslZs=', 'Content-Length': '0'} 2024-04-12T04:09:21.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004ae43b707067e59-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.342 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:09:21.342 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:09:21.342 DEBUG:boto:Method: GET 2024-04-12T04:09:21.343 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:09:21.343 DEBUG:boto:Data: 2024-04-12T04:09:21.343 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.343 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.343 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.343 DEBUG:boto:Params: {} 2024-04-12T04:09:21.343 DEBUG:boto:Token: None 2024-04-12T04:09:21.343 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj29 2024-04-12T04:09:21.343 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xC9k/c5mv4IDxXWVGozyylrslZs= 2024-04-12T04:09:21.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xC9k/c5mv4IDxXWVGozyylrslZs=', 'Content-Length': '0'} 2024-04-12T04:09:21.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8e149f1b9fa0760-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.386 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.387 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.387 DEBUG:boto:path=/dkyges-61/?prefix=obj29 2024-04-12T04:09:21.387 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj29 2024-04-12T04:09:21.387 DEBUG:boto:Method: GET 2024-04-12T04:09:21.387 DEBUG:boto:Path: /dkyges-61/?prefix=obj29 2024-04-12T04:09:21.387 DEBUG:boto:Data: 2024-04-12T04:09:21.387 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.387 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.387 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.387 DEBUG:boto:Params: {} 2024-04-12T04:09:21.388 DEBUG:boto:Token: None 2024-04-12T04:09:21.388 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.388 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d43820ac11100e9-006618b3f1-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.391 DEBUG:boto:b'dkyges-61obj291000falseobj292024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.391 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.391 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.391 DEBUG:boto:path=/dkyges-61/?prefix=obj29 2024-04-12T04:09:21.391 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj29 2024-04-12T04:09:21.391 DEBUG:boto:Method: GET 2024-04-12T04:09:21.391 DEBUG:boto:Path: /dkyges-61/?prefix=obj29 2024-04-12T04:09:21.392 DEBUG:boto:Data: 2024-04-12T04:09:21.392 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.392 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.392 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.392 DEBUG:boto:Params: {} 2024-04-12T04:09:21.392 DEBUG:boto:Token: None 2024-04-12T04:09:21.392 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.392 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.396 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d746fcf70dfaf1b4-006618b3f1-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.397 DEBUG:boto:b'dkyges-61obj291000falseobj292024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T04:09:21.397 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:09:21.397 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:09:21.397 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.397 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.397 DEBUG:boto:Method: GET 2024-04-12T04:09:21.397 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.398 DEBUG:boto:Data: 2024-04-12T04:09:21.398 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.398 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.398 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.398 DEBUG:boto:Params: {} 2024-04-12T04:09:21.398 DEBUG:boto:Token: None 2024-04-12T04:09:21.398 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.398 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UHtVGcTDtwLhUlE4Fy7XBuK7LJM= 2024-04-12T04:09:21.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UHtVGcTDtwLhUlE4Fy7XBuK7LJM=', 'Content-Length': '0'} 2024-04-12T04:09:21.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f81f4c5fbd264ca-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.442 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:09:21.443 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:09:21.443 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.443 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.443 DEBUG:boto:Method: GET 2024-04-12T04:09:21.443 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.443 DEBUG:boto:Data: 2024-04-12T04:09:21.443 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.443 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.443 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.443 DEBUG:boto:Params: {} 2024-04-12T04:09:21.443 DEBUG:boto:Token: None 2024-04-12T04:09:21.444 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.444 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UHtVGcTDtwLhUlE4Fy7XBuK7LJM= 2024-04-12T04:09:21.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UHtVGcTDtwLhUlE4Fy7XBuK7LJM=', 'Content-Length': '0'} 2024-04-12T04:09:21.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c987809417fa45fc-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.447 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:09:21.447 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:09:21.447 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.447 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.447 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.447 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.447 DEBUG:boto:Data: 2024-04-12T04:09:21.447 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.447 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.447 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.448 DEBUG:boto:Params: {} 2024-04-12T04:09:21.448 DEBUG:boto:Token: None 2024-04-12T04:09:21.448 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.448 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:58oe1BFekMNZIoEmvz2rouJVn3M= 2024-04-12T04:09:21.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:58oe1BFekMNZIoEmvz2rouJVn3M=', 'Content-Length': '0'} 2024-04-12T04:09:21.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068da3305773043e2-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.450 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:09:21.450 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:09:21.450 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.450 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.450 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.451 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.451 DEBUG:boto:Data: 2024-04-12T04:09:21.451 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.451 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.451 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.451 DEBUG:boto:Params: {} 2024-04-12T04:09:21.451 DEBUG:boto:Token: None 2024-04-12T04:09:21.451 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.451 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:58oe1BFekMNZIoEmvz2rouJVn3M= 2024-04-12T04:09:21.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:58oe1BFekMNZIoEmvz2rouJVn3M=', 'Content-Length': '0'} 2024-04-12T04:09:21.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e98a96156ce02467-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T04:09:21.454 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:09:21.454 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:09:21.454 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.454 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.454 DEBUG:boto:Method: GET 2024-04-12T04:09:21.454 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.454 DEBUG:boto:Data: 2024-04-12T04:09:21.454 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.454 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.454 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.454 DEBUG:boto:Params: {} 2024-04-12T04:09:21.455 DEBUG:boto:Token: None 2024-04-12T04:09:21.455 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.455 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UHtVGcTDtwLhUlE4Fy7XBuK7LJM= 2024-04-12T04:09:21.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UHtVGcTDtwLhUlE4Fy7XBuK7LJM=', 'Content-Length': '0'} 2024-04-12T04:09:21.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6ec467a4860de53-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.457 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:09:21.457 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:09:21.457 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.457 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.457 DEBUG:boto:Method: GET 2024-04-12T04:09:21.457 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.458 DEBUG:boto:Data: 2024-04-12T04:09:21.458 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.458 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.458 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.458 DEBUG:boto:Params: {} 2024-04-12T04:09:21.458 DEBUG:boto:Token: None 2024-04-12T04:09:21.458 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:09:21.458 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UHtVGcTDtwLhUlE4Fy7XBuK7LJM= 2024-04-12T04:09:21.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UHtVGcTDtwLhUlE4Fy7XBuK7LJM=', 'Content-Length': '0'} 2024-04-12T04:09:21.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1f240c9e730dc1b-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.460 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.460 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.461 DEBUG:boto:path=/dkyges-61/?prefix=obj31 2024-04-12T04:09:21.461 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj31 2024-04-12T04:09:21.461 DEBUG:boto:Method: GET 2024-04-12T04:09:21.461 DEBUG:boto:Path: /dkyges-61/?prefix=obj31 2024-04-12T04:09:21.461 DEBUG:boto:Data: 2024-04-12T04:09:21.461 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.461 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.461 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.461 DEBUG:boto:Params: {} 2024-04-12T04:09:21.461 DEBUG:boto:Token: None 2024-04-12T04:09:21.461 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.462 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000670ca0473fac5b77-006618b3f1-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.464 DEBUG:boto:b'dkyges-61obj311000falseobj312024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.464 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.464 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.465 DEBUG:boto:path=/dkyges-61/?prefix=obj31 2024-04-12T04:09:21.465 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj31 2024-04-12T04:09:21.465 DEBUG:boto:Method: GET 2024-04-12T04:09:21.465 DEBUG:boto:Path: /dkyges-61/?prefix=obj31 2024-04-12T04:09:21.465 DEBUG:boto:Data: 2024-04-12T04:09:21.465 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.465 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.465 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.465 DEBUG:boto:Params: {} 2024-04-12T04:09:21.465 DEBUG:boto:Token: None 2024-04-12T04:09:21.465 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.466 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfada443e343ad2b-006618b3f1-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.470 DEBUG:boto:b'dkyges-61obj311000falseobj312024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.470 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:09:21.470 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:09:21.470 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.470 DEBUG:boto:Path: /dkyges-61/obj31 2024-04-12T04:09:21.470 DEBUG:boto:Data: 2024-04-12T04:09:21.470 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.470 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.471 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.471 DEBUG:boto:Params: {} 2024-04-12T04:09:21.471 DEBUG:boto:Token: None 2024-04-12T04:09:21.471 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj31 2024-04-12T04:09:21.471 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9X48749eyry+kpHGULRUxIyD2v8= 2024-04-12T04:09:21.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9X48749eyry+kpHGULRUxIyD2v8=', 'Content-Length': '0'} 2024-04-12T04:09:21.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000577ca814940c73c2-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.473 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:09:21.473 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:09:21.473 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.473 DEBUG:boto:Path: /dkyges-61/obj31 2024-04-12T04:09:21.473 DEBUG:boto:Data: 2024-04-12T04:09:21.474 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.474 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.474 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.474 DEBUG:boto:Params: {} 2024-04-12T04:09:21.474 DEBUG:boto:Token: None 2024-04-12T04:09:21.474 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj31 2024-04-12T04:09:21.474 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9X48749eyry+kpHGULRUxIyD2v8= 2024-04-12T04:09:21.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9X48749eyry+kpHGULRUxIyD2v8=', 'Content-Length': '0'} 2024-04-12T04:09:21.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e149169985b4ce17-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T04:09:21.476 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:09:21.477 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:09:21.477 DEBUG:boto:Method: GET 2024-04-12T04:09:21.477 DEBUG:boto:Path: /dkyges-61/obj31 2024-04-12T04:09:21.477 DEBUG:boto:Data: 2024-04-12T04:09:21.477 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.477 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.477 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.477 DEBUG:boto:Params: {} 2024-04-12T04:09:21.477 DEBUG:boto:Token: None 2024-04-12T04:09:21.477 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj31 2024-04-12T04:09:21.477 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6+VnpOkhzi63/XeA3i1vh1CoUxY= 2024-04-12T04:09:21.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6+VnpOkhzi63/XeA3i1vh1CoUxY=', 'Content-Length': '0'} 2024-04-12T04:09:21.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be65269e4c0315cd-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.480 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:09:21.480 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:09:21.480 DEBUG:boto:Method: GET 2024-04-12T04:09:21.480 DEBUG:boto:Path: /dkyges-61/obj31 2024-04-12T04:09:21.480 DEBUG:boto:Data: 2024-04-12T04:09:21.480 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.480 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.480 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.480 DEBUG:boto:Params: {} 2024-04-12T04:09:21.480 DEBUG:boto:Token: None 2024-04-12T04:09:21.480 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj31 2024-04-12T04:09:21.481 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6+VnpOkhzi63/XeA3i1vh1CoUxY= 2024-04-12T04:09:21.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6+VnpOkhzi63/XeA3i1vh1CoUxY=', 'Content-Length': '0'} 2024-04-12T04:09:21.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c20f0c594bae7444-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.483 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.483 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.483 DEBUG:boto:path=/dkyges-61/?prefix=obj31 2024-04-12T04:09:21.483 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj31 2024-04-12T04:09:21.483 DEBUG:boto:Method: GET 2024-04-12T04:09:21.483 DEBUG:boto:Path: /dkyges-61/?prefix=obj31 2024-04-12T04:09:21.483 DEBUG:boto:Data: 2024-04-12T04:09:21.483 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.483 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.484 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.484 DEBUG:boto:Params: {} 2024-04-12T04:09:21.484 DEBUG:boto:Token: None 2024-04-12T04:09:21.484 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.484 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d011599b85e1df5c-006618b3f1-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.534 DEBUG:boto:b'dkyges-61obj311000falseobj312024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.535 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.535 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.535 DEBUG:boto:path=/dkyges-61/?prefix=obj31 2024-04-12T04:09:21.535 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj31 2024-04-12T04:09:21.535 DEBUG:boto:Method: GET 2024-04-12T04:09:21.535 DEBUG:boto:Path: /dkyges-61/?prefix=obj31 2024-04-12T04:09:21.535 DEBUG:boto:Data: 2024-04-12T04:09:21.536 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.536 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.536 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.536 DEBUG:boto:Params: {} 2024-04-12T04:09:21.536 DEBUG:boto:Token: None 2024-04-12T04:09:21.536 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.536 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f01b3a2d56bbe2a-006618b3f1-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.541 DEBUG:boto:b'dkyges-61obj311000falseobj312024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:09:21.541 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:09:21.541 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:09:21.541 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.541 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.541 DEBUG:boto:Method: GET 2024-04-12T04:09:21.541 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.542 DEBUG:boto:Data: 2024-04-12T04:09:21.542 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.542 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.542 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.542 DEBUG:boto:Params: {} 2024-04-12T04:09:21.542 DEBUG:boto:Token: None 2024-04-12T04:09:21.542 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.542 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+CNvjKegIzfRmf6vPMB/iuRBM70= 2024-04-12T04:09:21.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+CNvjKegIzfRmf6vPMB/iuRBM70=', 'Content-Length': '0'} 2024-04-12T04:09:21.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071409ad2cbe87667-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.544 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:09:21.545 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:09:21.545 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.545 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.545 DEBUG:boto:Method: GET 2024-04-12T04:09:21.545 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.545 DEBUG:boto:Data: 2024-04-12T04:09:21.545 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.545 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.545 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.545 DEBUG:boto:Params: {} 2024-04-12T04:09:21.545 DEBUG:boto:Token: None 2024-04-12T04:09:21.546 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.546 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+CNvjKegIzfRmf6vPMB/iuRBM70= 2024-04-12T04:09:21.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+CNvjKegIzfRmf6vPMB/iuRBM70=', 'Content-Length': '0'} 2024-04-12T04:09:21.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fd519503f8de512-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.590 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:09:21.591 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:09:21.591 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.591 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.591 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.591 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.592 DEBUG:boto:Data: 2024-04-12T04:09:21.592 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.592 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.592 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.592 DEBUG:boto:Params: {} 2024-04-12T04:09:21.592 DEBUG:boto:Token: None 2024-04-12T04:09:21.592 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.592 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mffP/C9LQ2hlhXn9NT3uk6tysXY= 2024-04-12T04:09:21.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mffP/C9LQ2hlhXn9NT3uk6tysXY=', 'Content-Length': '0'} 2024-04-12T04:09:21.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5ab75cda0daed06-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.595 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:09:21.595 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:09:21.595 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.595 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.595 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.596 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.596 DEBUG:boto:Data: 2024-04-12T04:09:21.596 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.596 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.596 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.596 DEBUG:boto:Params: {} 2024-04-12T04:09:21.596 DEBUG:boto:Token: None 2024-04-12T04:09:21.596 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.596 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mffP/C9LQ2hlhXn9NT3uk6tysXY= 2024-04-12T04:09:21.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mffP/C9LQ2hlhXn9NT3uk6tysXY=', 'Content-Length': '0'} 2024-04-12T04:09:21.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000630a306157b54cde-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:09:21.599 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:09:21.599 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:09:21.599 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.599 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.599 DEBUG:boto:Method: GET 2024-04-12T04:09:21.599 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.599 DEBUG:boto:Data: 2024-04-12T04:09:21.599 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.599 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.600 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.600 DEBUG:boto:Params: {} 2024-04-12T04:09:21.600 DEBUG:boto:Token: None 2024-04-12T04:09:21.600 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.600 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+CNvjKegIzfRmf6vPMB/iuRBM70= 2024-04-12T04:09:21.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+CNvjKegIzfRmf6vPMB/iuRBM70=', 'Content-Length': '0'} 2024-04-12T04:09:21.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bd2e59df6779e74-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.602 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:09:21.602 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:09:21.603 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.603 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.603 DEBUG:boto:Method: GET 2024-04-12T04:09:21.603 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.603 DEBUG:boto:Data: 2024-04-12T04:09:21.603 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.603 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.603 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.603 DEBUG:boto:Params: {} 2024-04-12T04:09:21.603 DEBUG:boto:Token: None 2024-04-12T04:09:21.603 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:09:21.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+CNvjKegIzfRmf6vPMB/iuRBM70= 2024-04-12T04:09:21.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+CNvjKegIzfRmf6vPMB/iuRBM70=', 'Content-Length': '0'} 2024-04-12T04:09:21.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef1cde6ae42cd355-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.606 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.606 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.606 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:09:21.606 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:09:21.606 DEBUG:boto:Method: GET 2024-04-12T04:09:21.606 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:09:21.606 DEBUG:boto:Data: 2024-04-12T04:09:21.607 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.607 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.607 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.607 DEBUG:boto:Params: {} 2024-04-12T04:09:21.607 DEBUG:boto:Token: None 2024-04-12T04:09:21.607 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.607 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf24a2ee1ffa47c2-006618b3f1-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.610 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.610 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.610 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.610 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:09:21.610 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:09:21.611 DEBUG:boto:Method: GET 2024-04-12T04:09:21.611 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:09:21.611 DEBUG:boto:Data: 2024-04-12T04:09:21.611 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.611 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.611 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.611 DEBUG:boto:Params: {} 2024-04-12T04:09:21.611 DEBUG:boto:Token: None 2024-04-12T04:09:21.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.611 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c1ba09b232c03aa7-006618b3f1-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.615 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.615 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:09:21.615 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:09:21.616 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.616 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:09:21.616 DEBUG:boto:Data: 2024-04-12T04:09:21.616 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.616 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.616 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.616 DEBUG:boto:Params: {} 2024-04-12T04:09:21.616 DEBUG:boto:Token: None 2024-04-12T04:09:21.616 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj33 2024-04-12T04:09:21.616 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pisvmhW4dZArsuzAbR6jxWxuTqU= 2024-04-12T04:09:21.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pisvmhW4dZArsuzAbR6jxWxuTqU=', 'Content-Length': '0'} 2024-04-12T04:09:21.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043e237bce1eade36-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.618 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:09:21.618 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:09:21.619 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.619 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:09:21.619 DEBUG:boto:Data: 2024-04-12T04:09:21.619 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.619 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.619 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.619 DEBUG:boto:Params: {} 2024-04-12T04:09:21.619 DEBUG:boto:Token: None 2024-04-12T04:09:21.619 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj33 2024-04-12T04:09:21.620 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pisvmhW4dZArsuzAbR6jxWxuTqU= 2024-04-12T04:09:21.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pisvmhW4dZArsuzAbR6jxWxuTqU=', 'Content-Length': '0'} 2024-04-12T04:09:21.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ff5193168bf684d-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:09:21.622 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:09:21.622 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:09:21.622 DEBUG:boto:Method: GET 2024-04-12T04:09:21.622 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:09:21.622 DEBUG:boto:Data: 2024-04-12T04:09:21.622 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.622 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.622 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.622 DEBUG:boto:Params: {} 2024-04-12T04:09:21.622 DEBUG:boto:Token: None 2024-04-12T04:09:21.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj33 2024-04-12T04:09:21.623 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qby9/+S1Osv8Z8Rqx1q0p32cdkU= 2024-04-12T04:09:21.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qby9/+S1Osv8Z8Rqx1q0p32cdkU=', 'Content-Length': '0'} 2024-04-12T04:09:21.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adb241516cf6556a-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.625 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:09:21.625 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:09:21.625 DEBUG:boto:Method: GET 2024-04-12T04:09:21.625 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:09:21.625 DEBUG:boto:Data: 2024-04-12T04:09:21.625 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.625 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.625 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.626 DEBUG:boto:Params: {} 2024-04-12T04:09:21.626 DEBUG:boto:Token: None 2024-04-12T04:09:21.626 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj33 2024-04-12T04:09:21.626 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qby9/+S1Osv8Z8Rqx1q0p32cdkU= 2024-04-12T04:09:21.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qby9/+S1Osv8Z8Rqx1q0p32cdkU=', 'Content-Length': '0'} 2024-04-12T04:09:21.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034213be79343802f-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.629 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.629 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.630 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:09:21.630 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:09:21.630 DEBUG:boto:Method: GET 2024-04-12T04:09:21.630 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:09:21.630 DEBUG:boto:Data: 2024-04-12T04:09:21.630 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.630 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.630 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.630 DEBUG:boto:Params: {} 2024-04-12T04:09:21.630 DEBUG:boto:Token: None 2024-04-12T04:09:21.631 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.631 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000415f4a9328b61124-006618b3f1-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.634 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.634 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.634 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.634 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:09:21.634 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:09:21.634 DEBUG:boto:Method: GET 2024-04-12T04:09:21.635 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:09:21.635 DEBUG:boto:Data: 2024-04-12T04:09:21.635 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.635 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.635 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.635 DEBUG:boto:Params: {} 2024-04-12T04:09:21.635 DEBUG:boto:Token: None 2024-04-12T04:09:21.635 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.635 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004482cf903e07fc7e-006618b3f1-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.640 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:09:21.640 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:09:21.640 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:09:21.640 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.641 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.641 DEBUG:boto:Method: GET 2024-04-12T04:09:21.641 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.641 DEBUG:boto:Data: 2024-04-12T04:09:21.641 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.641 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.641 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.641 DEBUG:boto:Params: {} 2024-04-12T04:09:21.641 DEBUG:boto:Token: None 2024-04-12T04:09:21.641 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.641 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XEvKsUhrCUo1KzVukq4dKdz25Yw= 2024-04-12T04:09:21.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XEvKsUhrCUo1KzVukq4dKdz25Yw=', 'Content-Length': '0'} 2024-04-12T04:09:21.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecfb0e8e7908e5e8-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.644 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:09:21.644 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:09:21.644 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.644 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.644 DEBUG:boto:Method: GET 2024-04-12T04:09:21.644 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.644 DEBUG:boto:Data: 2024-04-12T04:09:21.644 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.644 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.644 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.644 DEBUG:boto:Params: {} 2024-04-12T04:09:21.645 DEBUG:boto:Token: None 2024-04-12T04:09:21.645 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.645 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XEvKsUhrCUo1KzVukq4dKdz25Yw= 2024-04-12T04:09:21.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XEvKsUhrCUo1KzVukq4dKdz25Yw=', 'Content-Length': '0'} 2024-04-12T04:09:21.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000918b07ebb7402d67-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.690 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:09:21.691 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:09:21.691 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.691 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.691 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.691 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.691 DEBUG:boto:Data: 2024-04-12T04:09:21.691 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.691 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.691 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.691 DEBUG:boto:Params: {} 2024-04-12T04:09:21.691 DEBUG:boto:Token: None 2024-04-12T04:09:21.692 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.692 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AaS9rZlXK5f9/IbPs3c4hBb2xmw= 2024-04-12T04:09:21.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AaS9rZlXK5f9/IbPs3c4hBb2xmw=', 'Content-Length': '0'} 2024-04-12T04:09:21.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d25f347587648b7-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.694 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:09:21.694 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:09:21.694 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.694 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.695 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.695 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.695 DEBUG:boto:Data: 2024-04-12T04:09:21.695 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.695 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.695 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.695 DEBUG:boto:Params: {} 2024-04-12T04:09:21.695 DEBUG:boto:Token: None 2024-04-12T04:09:21.695 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.695 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AaS9rZlXK5f9/IbPs3c4hBb2xmw= 2024-04-12T04:09:21.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AaS9rZlXK5f9/IbPs3c4hBb2xmw=', 'Content-Length': '0'} 2024-04-12T04:09:21.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045627abaf34678d2-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:09:21.698 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:09:21.698 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:09:21.698 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.698 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.698 DEBUG:boto:Method: GET 2024-04-12T04:09:21.698 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.698 DEBUG:boto:Data: 2024-04-12T04:09:21.698 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.698 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.699 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.699 DEBUG:boto:Params: {} 2024-04-12T04:09:21.699 DEBUG:boto:Token: None 2024-04-12T04:09:21.699 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.699 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XEvKsUhrCUo1KzVukq4dKdz25Yw= 2024-04-12T04:09:21.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XEvKsUhrCUo1KzVukq4dKdz25Yw=', 'Content-Length': '0'} 2024-04-12T04:09:21.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082452400096ab92a-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.701 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:09:21.701 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:09:21.701 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.702 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.702 DEBUG:boto:Method: GET 2024-04-12T04:09:21.702 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.702 DEBUG:boto:Data: 2024-04-12T04:09:21.702 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.702 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.702 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.702 DEBUG:boto:Params: {} 2024-04-12T04:09:21.702 DEBUG:boto:Token: None 2024-04-12T04:09:21.702 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:09:21.703 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XEvKsUhrCUo1KzVukq4dKdz25Yw= 2024-04-12T04:09:21.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XEvKsUhrCUo1KzVukq4dKdz25Yw=', 'Content-Length': '0'} 2024-04-12T04:09:21.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b58ee02e980b391-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.706 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.706 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.706 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:09:21.706 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:09:21.706 DEBUG:boto:Method: GET 2024-04-12T04:09:21.706 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:09:21.706 DEBUG:boto:Data: 2024-04-12T04:09:21.706 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.706 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.706 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.707 DEBUG:boto:Params: {} 2024-04-12T04:09:21.707 DEBUG:boto:Token: None 2024-04-12T04:09:21.707 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.707 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ef5bcff72c0243b-006618b3f1-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.709 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.710 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.710 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.710 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:09:21.710 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:09:21.710 DEBUG:boto:Method: GET 2024-04-12T04:09:21.710 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:09:21.710 DEBUG:boto:Data: 2024-04-12T04:09:21.710 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.710 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.711 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.711 DEBUG:boto:Params: {} 2024-04-12T04:09:21.711 DEBUG:boto:Token: None 2024-04-12T04:09:21.711 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.711 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000293d55c14e9d3a62-006618b3f1-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.715 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.716 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:09:21.716 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:09:21.716 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.716 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:09:21.716 DEBUG:boto:Data: 2024-04-12T04:09:21.716 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.716 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.716 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.716 DEBUG:boto:Params: {} 2024-04-12T04:09:21.716 DEBUG:boto:Token: None 2024-04-12T04:09:21.717 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj34 2024-04-12T04:09:21.717 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RyXUHu+3vMjlUVMpDA9Z8Iizv6c= 2024-04-12T04:09:21.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RyXUHu+3vMjlUVMpDA9Z8Iizv6c=', 'Content-Length': '0'} 2024-04-12T04:09:21.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029a01e36f7bf8ee4-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.719 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:09:21.719 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:09:21.719 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.719 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:09:21.719 DEBUG:boto:Data: 2024-04-12T04:09:21.719 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.719 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.719 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.719 DEBUG:boto:Params: {} 2024-04-12T04:09:21.719 DEBUG:boto:Token: None 2024-04-12T04:09:21.720 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj34 2024-04-12T04:09:21.720 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RyXUHu+3vMjlUVMpDA9Z8Iizv6c= 2024-04-12T04:09:21.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RyXUHu+3vMjlUVMpDA9Z8Iizv6c=', 'Content-Length': '0'} 2024-04-12T04:09:21.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cd381c8908fcb82-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:09:21.722 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:09:21.722 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:09:21.722 DEBUG:boto:Method: GET 2024-04-12T04:09:21.722 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:09:21.722 DEBUG:boto:Data: 2024-04-12T04:09:21.722 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.722 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.722 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.723 DEBUG:boto:Params: {} 2024-04-12T04:09:21.723 DEBUG:boto:Token: None 2024-04-12T04:09:21.723 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj34 2024-04-12T04:09:21.723 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:udUm6ehHN5R+FHKR4CRYRMEGC3k= 2024-04-12T04:09:21.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:udUm6ehHN5R+FHKR4CRYRMEGC3k=', 'Content-Length': '0'} 2024-04-12T04:09:21.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000299dc553c395729e-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.770 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:09:21.770 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:09:21.771 DEBUG:boto:Method: GET 2024-04-12T04:09:21.771 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:09:21.771 DEBUG:boto:Data: 2024-04-12T04:09:21.771 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.771 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.771 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.771 DEBUG:boto:Params: {} 2024-04-12T04:09:21.771 DEBUG:boto:Token: None 2024-04-12T04:09:21.771 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj34 2024-04-12T04:09:21.772 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:udUm6ehHN5R+FHKR4CRYRMEGC3k= 2024-04-12T04:09:21.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:udUm6ehHN5R+FHKR4CRYRMEGC3k=', 'Content-Length': '0'} 2024-04-12T04:09:21.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000593d11bb0bbb332f-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.774 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.774 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.774 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:09:21.774 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:09:21.774 DEBUG:boto:Method: GET 2024-04-12T04:09:21.774 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:09:21.774 DEBUG:boto:Data: 2024-04-12T04:09:21.774 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.774 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.774 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.775 DEBUG:boto:Params: {} 2024-04-12T04:09:21.775 DEBUG:boto:Token: None 2024-04-12T04:09:21.775 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.775 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007879bd5a9f181120-006618b3f1-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.778 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.778 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.778 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.778 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:09:21.778 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:09:21.779 DEBUG:boto:Method: GET 2024-04-12T04:09:21.779 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:09:21.779 DEBUG:boto:Data: 2024-04-12T04:09:21.779 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.779 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.779 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.779 DEBUG:boto:Params: {} 2024-04-12T04:09:21.779 DEBUG:boto:Token: None 2024-04-12T04:09:21.779 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.779 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fe49f3a35925f5e-006618b3f1-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.783 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:09:21.784 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:09:21.784 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:09:21.784 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.784 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.784 DEBUG:boto:Method: GET 2024-04-12T04:09:21.784 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.784 DEBUG:boto:Data: 2024-04-12T04:09:21.784 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.785 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.785 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.785 DEBUG:boto:Params: {} 2024-04-12T04:09:21.785 DEBUG:boto:Token: None 2024-04-12T04:09:21.785 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.785 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U7vj+QwGpF+fQgK1XlAP9ToFsoE= 2024-04-12T04:09:21.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U7vj+QwGpF+fQgK1XlAP9ToFsoE=', 'Content-Length': '0'} 2024-04-12T04:09:21.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9be0be4f1086ae8-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.787 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:09:21.787 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:09:21.787 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.787 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.788 DEBUG:boto:Method: GET 2024-04-12T04:09:21.788 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.788 DEBUG:boto:Data: 2024-04-12T04:09:21.788 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.788 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.788 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.788 DEBUG:boto:Params: {} 2024-04-12T04:09:21.788 DEBUG:boto:Token: None 2024-04-12T04:09:21.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.788 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U7vj+QwGpF+fQgK1XlAP9ToFsoE= 2024-04-12T04:09:21.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U7vj+QwGpF+fQgK1XlAP9ToFsoE=', 'Content-Length': '0'} 2024-04-12T04:09:21.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc20d4982e4a4bbe-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.838 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:09:21.839 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:09:21.839 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.839 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.839 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.839 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.839 DEBUG:boto:Data: 2024-04-12T04:09:21.839 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.839 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.839 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.839 DEBUG:boto:Params: {} 2024-04-12T04:09:21.840 DEBUG:boto:Token: None 2024-04-12T04:09:21.840 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.840 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:McgLzT91cgtaPhp8yaTLXwOiBWY= 2024-04-12T04:09:21.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:McgLzT91cgtaPhp8yaTLXwOiBWY=', 'Content-Length': '0'} 2024-04-12T04:09:21.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad0395cfd8b57cc8-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.842 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:09:21.842 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:09:21.842 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.843 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.843 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.843 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.843 DEBUG:boto:Data: 2024-04-12T04:09:21.843 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.843 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.843 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.843 DEBUG:boto:Params: {} 2024-04-12T04:09:21.843 DEBUG:boto:Token: None 2024-04-12T04:09:21.843 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.844 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:McgLzT91cgtaPhp8yaTLXwOiBWY= 2024-04-12T04:09:21.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:McgLzT91cgtaPhp8yaTLXwOiBWY=', 'Content-Length': '0'} 2024-04-12T04:09:21.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2006cd3cbd3071e-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:09:21.846 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:09:21.846 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:09:21.846 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.846 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.846 DEBUG:boto:Method: GET 2024-04-12T04:09:21.846 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.846 DEBUG:boto:Data: 2024-04-12T04:09:21.846 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.846 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.846 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.846 DEBUG:boto:Params: {} 2024-04-12T04:09:21.847 DEBUG:boto:Token: None 2024-04-12T04:09:21.847 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.847 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U7vj+QwGpF+fQgK1XlAP9ToFsoE= 2024-04-12T04:09:21.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U7vj+QwGpF+fQgK1XlAP9ToFsoE=', 'Content-Length': '0'} 2024-04-12T04:09:21.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8a9e4ce71352d82-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.849 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:09:21.849 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:09:21.849 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.849 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.849 DEBUG:boto:Method: GET 2024-04-12T04:09:21.850 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.850 DEBUG:boto:Data: 2024-04-12T04:09:21.850 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.850 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.850 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.850 DEBUG:boto:Params: {} 2024-04-12T04:09:21.850 DEBUG:boto:Token: None 2024-04-12T04:09:21.850 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:09:21.850 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U7vj+QwGpF+fQgK1XlAP9ToFsoE= 2024-04-12T04:09:21.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U7vj+QwGpF+fQgK1XlAP9ToFsoE=', 'Content-Length': '0'} 2024-04-12T04:09:21.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069d836403c44234f-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.853 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.853 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.853 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:09:21.853 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:09:21.853 DEBUG:boto:Method: GET 2024-04-12T04:09:21.853 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:09:21.853 DEBUG:boto:Data: 2024-04-12T04:09:21.853 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.853 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.853 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.854 DEBUG:boto:Params: {} 2024-04-12T04:09:21.854 DEBUG:boto:Token: None 2024-04-12T04:09:21.854 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.854 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e541820eca995f3-006618b3f1-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.856 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.857 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.857 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.857 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:09:21.857 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:09:21.857 DEBUG:boto:Method: GET 2024-04-12T04:09:21.857 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:09:21.857 DEBUG:boto:Data: 2024-04-12T04:09:21.857 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.857 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.858 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.858 DEBUG:boto:Params: {} 2024-04-12T04:09:21.858 DEBUG:boto:Token: None 2024-04-12T04:09:21.858 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.858 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027283bece84c44f8-006618b3f1-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.862 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.863 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:09:21.863 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:09:21.863 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.863 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:09:21.863 DEBUG:boto:Data: 2024-04-12T04:09:21.863 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.863 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.863 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.863 DEBUG:boto:Params: {} 2024-04-12T04:09:21.863 DEBUG:boto:Token: None 2024-04-12T04:09:21.864 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj38 2024-04-12T04:09:21.864 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I5FycS3gHemCPlueRMouS3YsUnI= 2024-04-12T04:09:21.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I5FycS3gHemCPlueRMouS3YsUnI=', 'Content-Length': '0'} 2024-04-12T04:09:21.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfa0f55962171dd5-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.866 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:09:21.866 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:09:21.866 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.866 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:09:21.866 DEBUG:boto:Data: 2024-04-12T04:09:21.866 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.866 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.866 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.866 DEBUG:boto:Params: {} 2024-04-12T04:09:21.867 DEBUG:boto:Token: None 2024-04-12T04:09:21.867 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj38 2024-04-12T04:09:21.867 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I5FycS3gHemCPlueRMouS3YsUnI= 2024-04-12T04:09:21.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I5FycS3gHemCPlueRMouS3YsUnI=', 'Content-Length': '0'} 2024-04-12T04:09:21.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de4d2e50c1064aed-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:09:21.869 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:09:21.869 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:09:21.869 DEBUG:boto:Method: GET 2024-04-12T04:09:21.869 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:09:21.869 DEBUG:boto:Data: 2024-04-12T04:09:21.869 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.869 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.870 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.870 DEBUG:boto:Params: {} 2024-04-12T04:09:21.870 DEBUG:boto:Token: None 2024-04-12T04:09:21.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj38 2024-04-12T04:09:21.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kbktWHt8JQSuJlIUz6RfNwlfD0A= 2024-04-12T04:09:21.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kbktWHt8JQSuJlIUz6RfNwlfD0A=', 'Content-Length': '0'} 2024-04-12T04:09:21.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f20421a8126e7cf1-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.872 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:09:21.872 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:09:21.872 DEBUG:boto:Method: GET 2024-04-12T04:09:21.873 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:09:21.873 DEBUG:boto:Data: 2024-04-12T04:09:21.873 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.873 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.873 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.873 DEBUG:boto:Params: {} 2024-04-12T04:09:21.873 DEBUG:boto:Token: None 2024-04-12T04:09:21.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj38 2024-04-12T04:09:21.873 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kbktWHt8JQSuJlIUz6RfNwlfD0A= 2024-04-12T04:09:21.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kbktWHt8JQSuJlIUz6RfNwlfD0A=', 'Content-Length': '0'} 2024-04-12T04:09:21.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000483091d738ac7dca-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.875 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.876 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.876 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:09:21.876 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:09:21.876 DEBUG:boto:Method: GET 2024-04-12T04:09:21.876 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:09:21.876 DEBUG:boto:Data: 2024-04-12T04:09:21.876 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.876 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.876 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.876 DEBUG:boto:Params: {} 2024-04-12T04:09:21.876 DEBUG:boto:Token: None 2024-04-12T04:09:21.877 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.877 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c71f9ca33c4e0cda-006618b3f1-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.879 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.879 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.880 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.880 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:09:21.880 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:09:21.880 DEBUG:boto:Method: GET 2024-04-12T04:09:21.880 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:09:21.880 DEBUG:boto:Data: 2024-04-12T04:09:21.880 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.880 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.880 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.880 DEBUG:boto:Params: {} 2024-04-12T04:09:21.881 DEBUG:boto:Token: None 2024-04-12T04:09:21.881 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.881 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f2b2d089a890e69-006618b3f1-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.885 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:09:21.885 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:09:21.886 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:09:21.886 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.886 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.886 DEBUG:boto:Method: GET 2024-04-12T04:09:21.886 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.886 DEBUG:boto:Data: 2024-04-12T04:09:21.886 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.886 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.886 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.886 DEBUG:boto:Params: {} 2024-04-12T04:09:21.886 DEBUG:boto:Token: None 2024-04-12T04:09:21.887 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.887 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiSYBPpAD2QCWFf4PxrNkxVRM/0= 2024-04-12T04:09:21.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiSYBPpAD2QCWFf4PxrNkxVRM/0=', 'Content-Length': '0'} 2024-04-12T04:09:21.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000570011bbaf19b42e-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.889 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:09:21.889 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:09:21.889 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.890 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.890 DEBUG:boto:Method: GET 2024-04-12T04:09:21.890 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.890 DEBUG:boto:Data: 2024-04-12T04:09:21.890 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.890 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.890 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.890 DEBUG:boto:Params: {} 2024-04-12T04:09:21.890 DEBUG:boto:Token: None 2024-04-12T04:09:21.890 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.891 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiSYBPpAD2QCWFf4PxrNkxVRM/0= 2024-04-12T04:09:21.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiSYBPpAD2QCWFf4PxrNkxVRM/0=', 'Content-Length': '0'} 2024-04-12T04:09:21.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000045775908271335f-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.939 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:09:21.939 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:09:21.939 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.939 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.939 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.939 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.939 DEBUG:boto:Data: 2024-04-12T04:09:21.939 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.939 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.940 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.940 DEBUG:boto:Params: {} 2024-04-12T04:09:21.940 DEBUG:boto:Token: None 2024-04-12T04:09:21.940 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.940 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mRqTq5AYj7FSHTHIWVbcHo3mqeE= 2024-04-12T04:09:21.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mRqTq5AYj7FSHTHIWVbcHo3mqeE=', 'Content-Length': '0'} 2024-04-12T04:09:21.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e29961d7fd6dd9d-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.943 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:09:21.943 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:09:21.943 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.943 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.943 DEBUG:boto:Method: HEAD 2024-04-12T04:09:21.943 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.943 DEBUG:boto:Data: 2024-04-12T04:09:21.943 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.943 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.943 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.944 DEBUG:boto:Params: {} 2024-04-12T04:09:21.944 DEBUG:boto:Token: None 2024-04-12T04:09:21.944 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.944 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mRqTq5AYj7FSHTHIWVbcHo3mqeE= 2024-04-12T04:09:21.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mRqTq5AYj7FSHTHIWVbcHo3mqeE=', 'Content-Length': '0'} 2024-04-12T04:09:21.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000170b87f48fe7c376-006618b3f1-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:09:21.946 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:09:21.947 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:09:21.947 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.947 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.947 DEBUG:boto:Method: GET 2024-04-12T04:09:21.947 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.947 DEBUG:boto:Data: 2024-04-12T04:09:21.947 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.947 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.947 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.947 DEBUG:boto:Params: {} 2024-04-12T04:09:21.947 DEBUG:boto:Token: None 2024-04-12T04:09:21.948 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.948 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiSYBPpAD2QCWFf4PxrNkxVRM/0= 2024-04-12T04:09:21.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiSYBPpAD2QCWFf4PxrNkxVRM/0=', 'Content-Length': '0'} 2024-04-12T04:09:21.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6d12e0643081ea0-006618b3f1-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.990 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:09:21.990 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:09:21.991 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.991 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.991 DEBUG:boto:Method: GET 2024-04-12T04:09:21.991 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.991 DEBUG:boto:Data: 2024-04-12T04:09:21.991 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.991 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.991 DEBUG:boto:Port: 8000 2024-04-12T04:09:21.991 DEBUG:boto:Params: {} 2024-04-12T04:09:21.992 DEBUG:boto:Token: None 2024-04-12T04:09:21.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:09:21.992 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiSYBPpAD2QCWFf4PxrNkxVRM/0= 2024-04-12T04:09:21.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiSYBPpAD2QCWFf4PxrNkxVRM/0=', 'Content-Length': '0'} 2024-04-12T04:09:21.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000542c48297dbb756e-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.994 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.994 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.994 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:09:21.994 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:09:21.994 DEBUG:boto:Method: GET 2024-04-12T04:09:21.995 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:09:21.995 DEBUG:boto:Data: 2024-04-12T04:09:21.995 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.995 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:21.995 DEBUG:boto:Port: 8001 2024-04-12T04:09:21.995 DEBUG:boto:Params: {} 2024-04-12T04:09:21.995 DEBUG:boto:Token: None 2024-04-12T04:09:21.995 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:21 GMT /dkyges-61/ 2024-04-12T04:09:21.995 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ= 2024-04-12T04:09:21.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IIUwqZl2n5CmQoIq3RMTRXw1HXQ=', 'Content-Length': '0'} 2024-04-12T04:09:21.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3c607ef2d504d43-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:21.998 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:21.999 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:21.999 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:21.999 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:09:21.999 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:09:21.999 DEBUG:boto:Method: GET 2024-04-12T04:09:21.999 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:09:21.999 DEBUG:boto:Data: 2024-04-12T04:09:21.999 DEBUG:boto:Headers: {} 2024-04-12T04:09:21.999 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:21.999 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.000 DEBUG:boto:Params: {} 2024-04-12T04:09:22.000 DEBUG:boto:Token: None 2024-04-12T04:09:22.000 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.000 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d232f878d7e52613-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.004 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.004 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:09:22.004 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:09:22.004 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.004 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:09:22.004 DEBUG:boto:Data: 2024-04-12T04:09:22.005 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.005 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.005 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.005 DEBUG:boto:Params: {} 2024-04-12T04:09:22.005 DEBUG:boto:Token: None 2024-04-12T04:09:22.005 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj44 2024-04-12T04:09:22.005 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ObMd74/1+/4/uJglJVKjh3QHLD4= 2024-04-12T04:09:22.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ObMd74/1+/4/uJglJVKjh3QHLD4=', 'Content-Length': '0'} 2024-04-12T04:09:22.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d58b075fdef9ad47-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.007 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:09:22.008 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:09:22.008 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.008 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:09:22.008 DEBUG:boto:Data: 2024-04-12T04:09:22.008 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.008 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.008 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.008 DEBUG:boto:Params: {} 2024-04-12T04:09:22.008 DEBUG:boto:Token: None 2024-04-12T04:09:22.009 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj44 2024-04-12T04:09:22.009 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ObMd74/1+/4/uJglJVKjh3QHLD4= 2024-04-12T04:09:22.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ObMd74/1+/4/uJglJVKjh3QHLD4=', 'Content-Length': '0'} 2024-04-12T04:09:22.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003290d67b3b9fe95d-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:09:22.011 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:09:22.011 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:09:22.011 DEBUG:boto:Method: GET 2024-04-12T04:09:22.011 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:09:22.011 DEBUG:boto:Data: 2024-04-12T04:09:22.012 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.012 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.012 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.012 DEBUG:boto:Params: {} 2024-04-12T04:09:22.012 DEBUG:boto:Token: None 2024-04-12T04:09:22.012 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj44 2024-04-12T04:09:22.012 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8FSflnorfTOfcHZb1b0o3Ybd+1c= 2024-04-12T04:09:22.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8FSflnorfTOfcHZb1b0o3Ybd+1c=', 'Content-Length': '0'} 2024-04-12T04:09:22.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000712cbf9d9cfeb85b-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.014 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:09:22.015 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:09:22.015 DEBUG:boto:Method: GET 2024-04-12T04:09:22.015 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:09:22.015 DEBUG:boto:Data: 2024-04-12T04:09:22.015 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.015 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.015 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.015 DEBUG:boto:Params: {} 2024-04-12T04:09:22.015 DEBUG:boto:Token: None 2024-04-12T04:09:22.015 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj44 2024-04-12T04:09:22.016 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8FSflnorfTOfcHZb1b0o3Ybd+1c= 2024-04-12T04:09:22.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8FSflnorfTOfcHZb1b0o3Ybd+1c=', 'Content-Length': '0'} 2024-04-12T04:09:22.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d3d9072b4b75c5b-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.058 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.059 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.059 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:09:22.059 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:09:22.059 DEBUG:boto:Method: GET 2024-04-12T04:09:22.059 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:09:22.059 DEBUG:boto:Data: 2024-04-12T04:09:22.059 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.060 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.060 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.060 DEBUG:boto:Params: {} 2024-04-12T04:09:22.060 DEBUG:boto:Token: None 2024-04-12T04:09:22.060 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.060 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d03619d35ede17b2-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.063 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.064 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.064 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.064 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:09:22.064 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:09:22.064 DEBUG:boto:Method: GET 2024-04-12T04:09:22.064 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:09:22.064 DEBUG:boto:Data: 2024-04-12T04:09:22.064 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.064 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.065 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.065 DEBUG:boto:Params: {} 2024-04-12T04:09:22.065 DEBUG:boto:Token: None 2024-04-12T04:09:22.065 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.065 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e241cdbd93512b7e-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.070 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:09:22.071 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:09:22.071 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:09:22.071 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.071 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.071 DEBUG:boto:Method: GET 2024-04-12T04:09:22.071 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.071 DEBUG:boto:Data: 2024-04-12T04:09:22.071 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.071 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.071 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.072 DEBUG:boto:Params: {} 2024-04-12T04:09:22.072 DEBUG:boto:Token: None 2024-04-12T04:09:22.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.072 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T/28ycEjIe3h8lqg1iLekJoBciQ= 2024-04-12T04:09:22.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T/28ycEjIe3h8lqg1iLekJoBciQ=', 'Content-Length': '0'} 2024-04-12T04:09:22.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a56de7870f41b8e6-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.114 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:09:22.115 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:09:22.115 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.115 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.115 DEBUG:boto:Method: GET 2024-04-12T04:09:22.115 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.115 DEBUG:boto:Data: 2024-04-12T04:09:22.116 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.116 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.116 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.116 DEBUG:boto:Params: {} 2024-04-12T04:09:22.116 DEBUG:boto:Token: None 2024-04-12T04:09:22.116 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.116 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T/28ycEjIe3h8lqg1iLekJoBciQ= 2024-04-12T04:09:22.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T/28ycEjIe3h8lqg1iLekJoBciQ=', 'Content-Length': '0'} 2024-04-12T04:09:22.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8c4263ec8491c84-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.119 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:09:22.119 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:09:22.119 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.119 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.119 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.119 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.119 DEBUG:boto:Data: 2024-04-12T04:09:22.119 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.119 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.119 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.119 DEBUG:boto:Params: {} 2024-04-12T04:09:22.120 DEBUG:boto:Token: None 2024-04-12T04:09:22.120 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.120 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FGFdv5eEvzUhADRoiBhDbJYkiTs= 2024-04-12T04:09:22.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FGFdv5eEvzUhADRoiBhDbJYkiTs=', 'Content-Length': '0'} 2024-04-12T04:09:22.122 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c66e2646b9707fe7-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.122 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:09:22.122 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:09:22.122 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.122 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.122 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.123 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.123 DEBUG:boto:Data: 2024-04-12T04:09:22.123 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.123 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.123 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.123 DEBUG:boto:Params: {} 2024-04-12T04:09:22.123 DEBUG:boto:Token: None 2024-04-12T04:09:22.123 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.123 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FGFdv5eEvzUhADRoiBhDbJYkiTs= 2024-04-12T04:09:22.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FGFdv5eEvzUhADRoiBhDbJYkiTs=', 'Content-Length': '0'} 2024-04-12T04:09:22.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e866d027ca40226f-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:09:22.126 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:09:22.126 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:09:22.126 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.126 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.126 DEBUG:boto:Method: GET 2024-04-12T04:09:22.126 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.126 DEBUG:boto:Data: 2024-04-12T04:09:22.126 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.126 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.127 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.127 DEBUG:boto:Params: {} 2024-04-12T04:09:22.127 DEBUG:boto:Token: None 2024-04-12T04:09:22.127 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.127 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T/28ycEjIe3h8lqg1iLekJoBciQ= 2024-04-12T04:09:22.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T/28ycEjIe3h8lqg1iLekJoBciQ=', 'Content-Length': '0'} 2024-04-12T04:09:22.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000479f148e9b822cc0-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.129 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:09:22.129 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:09:22.129 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.130 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.130 DEBUG:boto:Method: GET 2024-04-12T04:09:22.130 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.130 DEBUG:boto:Data: 2024-04-12T04:09:22.130 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.130 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.130 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.130 DEBUG:boto:Params: {} 2024-04-12T04:09:22.130 DEBUG:boto:Token: None 2024-04-12T04:09:22.131 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:09:22.131 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T/28ycEjIe3h8lqg1iLekJoBciQ= 2024-04-12T04:09:22.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T/28ycEjIe3h8lqg1iLekJoBciQ=', 'Content-Length': '0'} 2024-04-12T04:09:22.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091bb73e90c821d60-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.133 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.134 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.134 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:09:22.134 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:09:22.134 DEBUG:boto:Method: GET 2024-04-12T04:09:22.134 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:09:22.134 DEBUG:boto:Data: 2024-04-12T04:09:22.134 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.135 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.135 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.135 DEBUG:boto:Params: {} 2024-04-12T04:09:22.135 DEBUG:boto:Token: None 2024-04-12T04:09:22.135 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.135 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022fe89045db14e58-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.138 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.139 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.139 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.140 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:09:22.140 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:09:22.140 DEBUG:boto:Method: GET 2024-04-12T04:09:22.140 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:09:22.140 DEBUG:boto:Data: 2024-04-12T04:09:22.141 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.141 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.141 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.141 DEBUG:boto:Params: {} 2024-04-12T04:09:22.141 DEBUG:boto:Token: None 2024-04-12T04:09:22.141 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.141 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061cec34193b0a200-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.145 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.145 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:09:22.145 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:09:22.146 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.146 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:09:22.146 DEBUG:boto:Data: 2024-04-12T04:09:22.146 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.146 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.146 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.146 DEBUG:boto:Params: {} 2024-04-12T04:09:22.146 DEBUG:boto:Token: None 2024-04-12T04:09:22.146 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj46 2024-04-12T04:09:22.146 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gEZ2jHJ7UqhaOjn4M9qUoZM9ORM= 2024-04-12T04:09:22.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gEZ2jHJ7UqhaOjn4M9qUoZM9ORM=', 'Content-Length': '0'} 2024-04-12T04:09:22.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006401502b1468dc40-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.148 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:09:22.148 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:09:22.149 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.149 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:09:22.149 DEBUG:boto:Data: 2024-04-12T04:09:22.149 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.149 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.149 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.149 DEBUG:boto:Params: {} 2024-04-12T04:09:22.149 DEBUG:boto:Token: None 2024-04-12T04:09:22.149 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj46 2024-04-12T04:09:22.149 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gEZ2jHJ7UqhaOjn4M9qUoZM9ORM= 2024-04-12T04:09:22.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gEZ2jHJ7UqhaOjn4M9qUoZM9ORM=', 'Content-Length': '0'} 2024-04-12T04:09:22.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb7176355e4ced22-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:09:22.152 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:09:22.152 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:09:22.152 DEBUG:boto:Method: GET 2024-04-12T04:09:22.152 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:09:22.152 DEBUG:boto:Data: 2024-04-12T04:09:22.152 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.152 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.152 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.153 DEBUG:boto:Params: {} 2024-04-12T04:09:22.153 DEBUG:boto:Token: None 2024-04-12T04:09:22.153 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj46 2024-04-12T04:09:22.153 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:O3EKLtRi+A6Xu+VgSgNNCEu0bqQ= 2024-04-12T04:09:22.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:O3EKLtRi+A6Xu+VgSgNNCEu0bqQ=', 'Content-Length': '0'} 2024-04-12T04:09:22.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000008d55147b6415fe-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.155 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:09:22.155 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:09:22.155 DEBUG:boto:Method: GET 2024-04-12T04:09:22.156 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:09:22.156 DEBUG:boto:Data: 2024-04-12T04:09:22.156 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.156 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.156 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.156 DEBUG:boto:Params: {} 2024-04-12T04:09:22.156 DEBUG:boto:Token: None 2024-04-12T04:09:22.156 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj46 2024-04-12T04:09:22.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:O3EKLtRi+A6Xu+VgSgNNCEu0bqQ= 2024-04-12T04:09:22.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:O3EKLtRi+A6Xu+VgSgNNCEu0bqQ=', 'Content-Length': '0'} 2024-04-12T04:09:22.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f003d06d54c6107b-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.198 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.198 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.199 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:09:22.199 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:09:22.199 DEBUG:boto:Method: GET 2024-04-12T04:09:22.199 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:09:22.199 DEBUG:boto:Data: 2024-04-12T04:09:22.199 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.199 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.199 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.199 DEBUG:boto:Params: {} 2024-04-12T04:09:22.199 DEBUG:boto:Token: None 2024-04-12T04:09:22.200 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.200 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a984750f48333e67-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.202 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.203 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.203 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.203 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:09:22.203 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:09:22.203 DEBUG:boto:Method: GET 2024-04-12T04:09:22.203 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:09:22.203 DEBUG:boto:Data: 2024-04-12T04:09:22.203 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.203 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.204 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.204 DEBUG:boto:Params: {} 2024-04-12T04:09:22.204 DEBUG:boto:Token: None 2024-04-12T04:09:22.204 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.204 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef1d4fc49e4e63b4-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.208 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:09:22.208 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:09:22.209 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:09:22.209 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.209 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.209 DEBUG:boto:Method: GET 2024-04-12T04:09:22.209 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.209 DEBUG:boto:Data: 2024-04-12T04:09:22.209 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.209 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.209 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.209 DEBUG:boto:Params: {} 2024-04-12T04:09:22.209 DEBUG:boto:Token: None 2024-04-12T04:09:22.210 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.210 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LhGYPPniEv7mc3Qa/N/xc78LZ+g= 2024-04-12T04:09:22.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LhGYPPniEv7mc3Qa/N/xc78LZ+g=', 'Content-Length': '0'} 2024-04-12T04:09:22.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5c889d08a803f02-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.254 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:09:22.254 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:09:22.255 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.255 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.255 DEBUG:boto:Method: GET 2024-04-12T04:09:22.255 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.255 DEBUG:boto:Data: 2024-04-12T04:09:22.255 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.255 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.255 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.255 DEBUG:boto:Params: {} 2024-04-12T04:09:22.255 DEBUG:boto:Token: None 2024-04-12T04:09:22.255 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.256 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LhGYPPniEv7mc3Qa/N/xc78LZ+g= 2024-04-12T04:09:22.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LhGYPPniEv7mc3Qa/N/xc78LZ+g=', 'Content-Length': '0'} 2024-04-12T04:09:22.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c02465222a48f767-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.258 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:09:22.258 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:09:22.258 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.258 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.258 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.259 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.259 DEBUG:boto:Data: 2024-04-12T04:09:22.259 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.259 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.259 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.259 DEBUG:boto:Params: {} 2024-04-12T04:09:22.259 DEBUG:boto:Token: None 2024-04-12T04:09:22.259 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.259 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wZSAEFOOLG7gQUXDzrVyauOMy8g= 2024-04-12T04:09:22.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wZSAEFOOLG7gQUXDzrVyauOMy8g=', 'Content-Length': '0'} 2024-04-12T04:09:22.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6ea1a03cafeac08-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.261 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:09:22.261 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:09:22.262 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.262 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.262 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.262 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.262 DEBUG:boto:Data: 2024-04-12T04:09:22.262 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.262 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.263 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.263 DEBUG:boto:Params: {} 2024-04-12T04:09:22.263 DEBUG:boto:Token: None 2024-04-12T04:09:22.263 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.263 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wZSAEFOOLG7gQUXDzrVyauOMy8g= 2024-04-12T04:09:22.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wZSAEFOOLG7gQUXDzrVyauOMy8g=', 'Content-Length': '0'} 2024-04-12T04:09:22.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb7d65a05d434f66-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:09:22.265 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:09:22.265 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:09:22.266 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.266 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.266 DEBUG:boto:Method: GET 2024-04-12T04:09:22.266 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.266 DEBUG:boto:Data: 2024-04-12T04:09:22.266 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.266 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.266 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.266 DEBUG:boto:Params: {} 2024-04-12T04:09:22.267 DEBUG:boto:Token: None 2024-04-12T04:09:22.267 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.267 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LhGYPPniEv7mc3Qa/N/xc78LZ+g= 2024-04-12T04:09:22.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LhGYPPniEv7mc3Qa/N/xc78LZ+g=', 'Content-Length': '0'} 2024-04-12T04:09:22.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c48384c81b0c074-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.269 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:09:22.269 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:09:22.269 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.269 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.269 DEBUG:boto:Method: GET 2024-04-12T04:09:22.269 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.269 DEBUG:boto:Data: 2024-04-12T04:09:22.269 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.270 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.270 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.270 DEBUG:boto:Params: {} 2024-04-12T04:09:22.270 DEBUG:boto:Token: None 2024-04-12T04:09:22.270 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:09:22.270 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LhGYPPniEv7mc3Qa/N/xc78LZ+g= 2024-04-12T04:09:22.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LhGYPPniEv7mc3Qa/N/xc78LZ+g=', 'Content-Length': '0'} 2024-04-12T04:09:22.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b3751330c8adee5-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.272 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.272 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.272 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:09:22.273 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:09:22.273 DEBUG:boto:Method: GET 2024-04-12T04:09:22.273 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:09:22.273 DEBUG:boto:Data: 2024-04-12T04:09:22.273 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.273 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.273 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.273 DEBUG:boto:Params: {} 2024-04-12T04:09:22.274 DEBUG:boto:Token: None 2024-04-12T04:09:22.274 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.274 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9677aeccbd43eff-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.276 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.277 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.277 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.277 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:09:22.277 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:09:22.277 DEBUG:boto:Method: GET 2024-04-12T04:09:22.277 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:09:22.277 DEBUG:boto:Data: 2024-04-12T04:09:22.277 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.278 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.278 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.278 DEBUG:boto:Params: {} 2024-04-12T04:09:22.278 DEBUG:boto:Token: None 2024-04-12T04:09:22.278 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.278 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057733721bc85b620-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.282 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.283 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:09:22.283 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:09:22.283 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.283 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:09:22.283 DEBUG:boto:Data: 2024-04-12T04:09:22.284 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.284 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.284 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.284 DEBUG:boto:Params: {} 2024-04-12T04:09:22.284 DEBUG:boto:Token: None 2024-04-12T04:09:22.284 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj51 2024-04-12T04:09:22.284 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WMAglnHpYafl+/9s3Dg7S4fkgr4= 2024-04-12T04:09:22.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WMAglnHpYafl+/9s3Dg7S4fkgr4=', 'Content-Length': '0'} 2024-04-12T04:09:22.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002db94a9d08be9a94-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.286 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:09:22.286 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:09:22.287 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.287 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:09:22.287 DEBUG:boto:Data: 2024-04-12T04:09:22.287 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.287 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.287 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.287 DEBUG:boto:Params: {} 2024-04-12T04:09:22.287 DEBUG:boto:Token: None 2024-04-12T04:09:22.287 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj51 2024-04-12T04:09:22.287 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WMAglnHpYafl+/9s3Dg7S4fkgr4= 2024-04-12T04:09:22.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WMAglnHpYafl+/9s3Dg7S4fkgr4=', 'Content-Length': '0'} 2024-04-12T04:09:22.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031cc032b4a98e373-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:09:22.290 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:09:22.290 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:09:22.290 DEBUG:boto:Method: GET 2024-04-12T04:09:22.290 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:09:22.290 DEBUG:boto:Data: 2024-04-12T04:09:22.290 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.290 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.290 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.291 DEBUG:boto:Params: {} 2024-04-12T04:09:22.291 DEBUG:boto:Token: None 2024-04-12T04:09:22.291 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj51 2024-04-12T04:09:22.291 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:85iyNqeWwVTiwpZCCRJbffSqlsg= 2024-04-12T04:09:22.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:85iyNqeWwVTiwpZCCRJbffSqlsg=', 'Content-Length': '0'} 2024-04-12T04:09:22.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068c92a0bbaabbd8c-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.293 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:09:22.293 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:09:22.293 DEBUG:boto:Method: GET 2024-04-12T04:09:22.293 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:09:22.294 DEBUG:boto:Data: 2024-04-12T04:09:22.294 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.294 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.294 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.294 DEBUG:boto:Params: {} 2024-04-12T04:09:22.294 DEBUG:boto:Token: None 2024-04-12T04:09:22.294 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj51 2024-04-12T04:09:22.294 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:85iyNqeWwVTiwpZCCRJbffSqlsg= 2024-04-12T04:09:22.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:85iyNqeWwVTiwpZCCRJbffSqlsg=', 'Content-Length': '0'} 2024-04-12T04:09:22.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000600ffc3ef2b49057-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.297 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.297 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.297 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:09:22.297 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:09:22.297 DEBUG:boto:Method: GET 2024-04-12T04:09:22.297 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:09:22.297 DEBUG:boto:Data: 2024-04-12T04:09:22.297 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.297 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.298 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.298 DEBUG:boto:Params: {} 2024-04-12T04:09:22.298 DEBUG:boto:Token: None 2024-04-12T04:09:22.298 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.298 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8fc324be6a6287e-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.301 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.301 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.301 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.301 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:09:22.301 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:09:22.301 DEBUG:boto:Method: GET 2024-04-12T04:09:22.301 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:09:22.301 DEBUG:boto:Data: 2024-04-12T04:09:22.302 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.302 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.302 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.302 DEBUG:boto:Params: {} 2024-04-12T04:09:22.302 DEBUG:boto:Token: None 2024-04-12T04:09:22.302 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.302 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062b1d1b405b76782-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.306 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:09:22.307 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:09:22.307 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:09:22.307 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.307 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.307 DEBUG:boto:Method: GET 2024-04-12T04:09:22.307 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.307 DEBUG:boto:Data: 2024-04-12T04:09:22.307 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.308 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.308 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.308 DEBUG:boto:Params: {} 2024-04-12T04:09:22.308 DEBUG:boto:Token: None 2024-04-12T04:09:22.308 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.308 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:N3gX3FdXV+FpF5KuFr91hys3aXA= 2024-04-12T04:09:22.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:N3gX3FdXV+FpF5KuFr91hys3aXA=', 'Content-Length': '0'} 2024-04-12T04:09:22.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a779873b9967edd-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.310 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:09:22.311 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:09:22.311 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.311 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.311 DEBUG:boto:Method: GET 2024-04-12T04:09:22.311 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.311 DEBUG:boto:Data: 2024-04-12T04:09:22.311 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.311 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.311 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.311 DEBUG:boto:Params: {} 2024-04-12T04:09:22.311 DEBUG:boto:Token: None 2024-04-12T04:09:22.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.312 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:N3gX3FdXV+FpF5KuFr91hys3aXA= 2024-04-12T04:09:22.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:N3gX3FdXV+FpF5KuFr91hys3aXA=', 'Content-Length': '0'} 2024-04-12T04:09:22.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e54c45ee4fe23b1-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.314 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:09:22.314 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:09:22.314 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.314 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.314 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.314 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.315 DEBUG:boto:Data: 2024-04-12T04:09:22.315 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.315 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.315 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.315 DEBUG:boto:Params: {} 2024-04-12T04:09:22.315 DEBUG:boto:Token: None 2024-04-12T04:09:22.315 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.315 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2wlRPqEnoZ/3OharJ+ro9i3XIgs= 2024-04-12T04:09:22.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2wlRPqEnoZ/3OharJ+ro9i3XIgs=', 'Content-Length': '0'} 2024-04-12T04:09:22.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008756face217e3fe9-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.317 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:09:22.317 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:09:22.318 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.318 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.318 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.318 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.318 DEBUG:boto:Data: 2024-04-12T04:09:22.318 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.318 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.318 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.318 DEBUG:boto:Params: {} 2024-04-12T04:09:22.318 DEBUG:boto:Token: None 2024-04-12T04:09:22.319 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.319 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2wlRPqEnoZ/3OharJ+ro9i3XIgs= 2024-04-12T04:09:22.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2wlRPqEnoZ/3OharJ+ro9i3XIgs=', 'Content-Length': '0'} 2024-04-12T04:09:22.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b421f95ef470786b-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:09:22.321 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:09:22.321 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:09:22.321 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.321 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.321 DEBUG:boto:Method: GET 2024-04-12T04:09:22.321 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.321 DEBUG:boto:Data: 2024-04-12T04:09:22.321 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.321 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.322 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.322 DEBUG:boto:Params: {} 2024-04-12T04:09:22.322 DEBUG:boto:Token: None 2024-04-12T04:09:22.322 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.322 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:N3gX3FdXV+FpF5KuFr91hys3aXA= 2024-04-12T04:09:22.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:N3gX3FdXV+FpF5KuFr91hys3aXA=', 'Content-Length': '0'} 2024-04-12T04:09:22.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e53110315b7b7e63-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.324 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:09:22.324 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:09:22.324 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.325 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.325 DEBUG:boto:Method: GET 2024-04-12T04:09:22.325 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.325 DEBUG:boto:Data: 2024-04-12T04:09:22.325 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.325 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.325 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.325 DEBUG:boto:Params: {} 2024-04-12T04:09:22.325 DEBUG:boto:Token: None 2024-04-12T04:09:22.325 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:09:22.326 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:N3gX3FdXV+FpF5KuFr91hys3aXA= 2024-04-12T04:09:22.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:N3gX3FdXV+FpF5KuFr91hys3aXA=', 'Content-Length': '0'} 2024-04-12T04:09:22.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000873db4ea2b6c5fe7-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.328 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.328 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.328 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:09:22.328 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:09:22.328 DEBUG:boto:Method: GET 2024-04-12T04:09:22.328 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:09:22.328 DEBUG:boto:Data: 2024-04-12T04:09:22.328 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.329 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.329 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.329 DEBUG:boto:Params: {} 2024-04-12T04:09:22.329 DEBUG:boto:Token: None 2024-04-12T04:09:22.329 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.329 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcddc87015b15788-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.332 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.332 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.332 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.332 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:09:22.333 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:09:22.333 DEBUG:boto:Method: GET 2024-04-12T04:09:22.333 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:09:22.333 DEBUG:boto:Data: 2024-04-12T04:09:22.333 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.333 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.333 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.333 DEBUG:boto:Params: {} 2024-04-12T04:09:22.333 DEBUG:boto:Token: None 2024-04-12T04:09:22.333 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.334 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c3c0ba743522178-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.338 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.338 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:09:22.338 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:09:22.338 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.338 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:09:22.339 DEBUG:boto:Data: 2024-04-12T04:09:22.339 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.339 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.339 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.339 DEBUG:boto:Params: {} 2024-04-12T04:09:22.339 DEBUG:boto:Token: None 2024-04-12T04:09:22.339 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj55 2024-04-12T04:09:22.339 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X1Gg8JyVmf/crHIbQStxTg/RLl4= 2024-04-12T04:09:22.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X1Gg8JyVmf/crHIbQStxTg/RLl4=', 'Content-Length': '0'} 2024-04-12T04:09:22.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f88aed06ef78b5a-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.341 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:09:22.341 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:09:22.341 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.342 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:09:22.342 DEBUG:boto:Data: 2024-04-12T04:09:22.342 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.342 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.342 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.342 DEBUG:boto:Params: {} 2024-04-12T04:09:22.342 DEBUG:boto:Token: None 2024-04-12T04:09:22.342 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj55 2024-04-12T04:09:22.342 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X1Gg8JyVmf/crHIbQStxTg/RLl4= 2024-04-12T04:09:22.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X1Gg8JyVmf/crHIbQStxTg/RLl4=', 'Content-Length': '0'} 2024-04-12T04:09:22.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1ac8bc607696c42-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:09:22.345 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:09:22.345 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:09:22.345 DEBUG:boto:Method: GET 2024-04-12T04:09:22.345 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:09:22.345 DEBUG:boto:Data: 2024-04-12T04:09:22.345 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.345 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.345 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.345 DEBUG:boto:Params: {} 2024-04-12T04:09:22.346 DEBUG:boto:Token: None 2024-04-12T04:09:22.346 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj55 2024-04-12T04:09:22.346 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0ta7odHGTE9r2NtXf545dvrPDNg= 2024-04-12T04:09:22.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0ta7odHGTE9r2NtXf545dvrPDNg=', 'Content-Length': '0'} 2024-04-12T04:09:22.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f93406c1a8830bd5-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.348 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:09:22.348 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:09:22.349 DEBUG:boto:Method: GET 2024-04-12T04:09:22.349 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:09:22.349 DEBUG:boto:Data: 2024-04-12T04:09:22.349 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.349 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.349 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.349 DEBUG:boto:Params: {} 2024-04-12T04:09:22.349 DEBUG:boto:Token: None 2024-04-12T04:09:22.349 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj55 2024-04-12T04:09:22.349 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0ta7odHGTE9r2NtXf545dvrPDNg= 2024-04-12T04:09:22.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0ta7odHGTE9r2NtXf545dvrPDNg=', 'Content-Length': '0'} 2024-04-12T04:09:22.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2ea90cd1fe9768e-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.394 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.395 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.395 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:09:22.395 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:09:22.395 DEBUG:boto:Method: GET 2024-04-12T04:09:22.395 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:09:22.395 DEBUG:boto:Data: 2024-04-12T04:09:22.395 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.395 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.395 DEBUG:boto:Params: {} 2024-04-12T04:09:22.396 DEBUG:boto:Token: None 2024-04-12T04:09:22.396 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.396 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4add15a4c62ec3e-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.399 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.399 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.399 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.399 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:09:22.399 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:09:22.399 DEBUG:boto:Method: GET 2024-04-12T04:09:22.399 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:09:22.400 DEBUG:boto:Data: 2024-04-12T04:09:22.400 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.400 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.400 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.400 DEBUG:boto:Params: {} 2024-04-12T04:09:22.400 DEBUG:boto:Token: None 2024-04-12T04:09:22.400 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.400 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fb37000dc303ba3-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.404 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:09:22.405 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:09:22.405 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:09:22.405 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.405 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.405 DEBUG:boto:Method: GET 2024-04-12T04:09:22.405 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.405 DEBUG:boto:Data: 2024-04-12T04:09:22.405 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.405 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.405 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.405 DEBUG:boto:Params: {} 2024-04-12T04:09:22.406 DEBUG:boto:Token: None 2024-04-12T04:09:22.406 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.406 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DLkVcCiLZJSmcz2+PDPg8UNhPRI= 2024-04-12T04:09:22.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DLkVcCiLZJSmcz2+PDPg8UNhPRI=', 'Content-Length': '0'} 2024-04-12T04:09:22.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045cd0a7599a074a8-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.450 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:09:22.451 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:09:22.451 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.451 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.451 DEBUG:boto:Method: GET 2024-04-12T04:09:22.451 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.451 DEBUG:boto:Data: 2024-04-12T04:09:22.451 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.451 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.451 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.451 DEBUG:boto:Params: {} 2024-04-12T04:09:22.451 DEBUG:boto:Token: None 2024-04-12T04:09:22.452 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.452 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DLkVcCiLZJSmcz2+PDPg8UNhPRI= 2024-04-12T04:09:22.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DLkVcCiLZJSmcz2+PDPg8UNhPRI=', 'Content-Length': '0'} 2024-04-12T04:09:22.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a74989c6fcc5614e-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.454 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:09:22.454 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:09:22.454 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.454 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.455 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.455 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.455 DEBUG:boto:Data: 2024-04-12T04:09:22.455 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.455 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.455 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.455 DEBUG:boto:Params: {} 2024-04-12T04:09:22.455 DEBUG:boto:Token: None 2024-04-12T04:09:22.455 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.455 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2Ktr9HGUPaKjDQJtZiOhl655zMU= 2024-04-12T04:09:22.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2Ktr9HGUPaKjDQJtZiOhl655zMU=', 'Content-Length': '0'} 2024-04-12T04:09:22.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ecfa6556a259f88-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.457 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:09:22.458 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:09:22.458 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.458 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.458 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.458 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.458 DEBUG:boto:Data: 2024-04-12T04:09:22.458 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.458 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.458 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.458 DEBUG:boto:Params: {} 2024-04-12T04:09:22.459 DEBUG:boto:Token: None 2024-04-12T04:09:22.459 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.459 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2Ktr9HGUPaKjDQJtZiOhl655zMU= 2024-04-12T04:09:22.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2Ktr9HGUPaKjDQJtZiOhl655zMU=', 'Content-Length': '0'} 2024-04-12T04:09:22.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e1c20efd8fd1861-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:09:22.461 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:09:22.461 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:09:22.462 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.462 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.462 DEBUG:boto:Method: GET 2024-04-12T04:09:22.462 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.462 DEBUG:boto:Data: 2024-04-12T04:09:22.462 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.462 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.462 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.462 DEBUG:boto:Params: {} 2024-04-12T04:09:22.462 DEBUG:boto:Token: None 2024-04-12T04:09:22.462 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.463 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DLkVcCiLZJSmcz2+PDPg8UNhPRI= 2024-04-12T04:09:22.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DLkVcCiLZJSmcz2+PDPg8UNhPRI=', 'Content-Length': '0'} 2024-04-12T04:09:22.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc8cec2bbd99e960-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.465 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:09:22.465 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:09:22.465 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.465 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.465 DEBUG:boto:Method: GET 2024-04-12T04:09:22.465 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.465 DEBUG:boto:Data: 2024-04-12T04:09:22.465 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.466 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.466 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.466 DEBUG:boto:Params: {} 2024-04-12T04:09:22.466 DEBUG:boto:Token: None 2024-04-12T04:09:22.466 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:09:22.466 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DLkVcCiLZJSmcz2+PDPg8UNhPRI= 2024-04-12T04:09:22.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DLkVcCiLZJSmcz2+PDPg8UNhPRI=', 'Content-Length': '0'} 2024-04-12T04:09:22.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000028472c68105051c-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.468 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.468 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.468 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:09:22.468 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:09:22.469 DEBUG:boto:Method: GET 2024-04-12T04:09:22.469 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:09:22.469 DEBUG:boto:Data: 2024-04-12T04:09:22.469 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.469 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.469 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.469 DEBUG:boto:Params: {} 2024-04-12T04:09:22.469 DEBUG:boto:Token: None 2024-04-12T04:09:22.469 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.470 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000176582c9b545860a-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.472 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.473 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.473 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.473 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:09:22.473 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:09:22.473 DEBUG:boto:Method: GET 2024-04-12T04:09:22.473 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:09:22.473 DEBUG:boto:Data: 2024-04-12T04:09:22.473 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.473 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.474 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.474 DEBUG:boto:Params: {} 2024-04-12T04:09:22.474 DEBUG:boto:Token: None 2024-04-12T04:09:22.474 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.474 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.478 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da826745648ddb36-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.478 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.479 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:09:22.479 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:09:22.479 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.479 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:09:22.479 DEBUG:boto:Data: 2024-04-12T04:09:22.479 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.479 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.479 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.479 DEBUG:boto:Params: {} 2024-04-12T04:09:22.480 DEBUG:boto:Token: None 2024-04-12T04:09:22.480 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj57 2024-04-12T04:09:22.480 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GsJ4MJ/7ozcxJeIv6lk6TsqjUpo= 2024-04-12T04:09:22.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GsJ4MJ/7ozcxJeIv6lk6TsqjUpo=', 'Content-Length': '0'} 2024-04-12T04:09:22.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ce66587208fca17-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.482 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:09:22.482 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:09:22.482 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.482 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:09:22.482 DEBUG:boto:Data: 2024-04-12T04:09:22.482 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.482 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.483 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.483 DEBUG:boto:Params: {} 2024-04-12T04:09:22.483 DEBUG:boto:Token: None 2024-04-12T04:09:22.483 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj57 2024-04-12T04:09:22.483 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GsJ4MJ/7ozcxJeIv6lk6TsqjUpo= 2024-04-12T04:09:22.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GsJ4MJ/7ozcxJeIv6lk6TsqjUpo=', 'Content-Length': '0'} 2024-04-12T04:09:22.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f0789be33ab6294-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:09:22.485 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:09:22.485 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:09:22.485 DEBUG:boto:Method: GET 2024-04-12T04:09:22.486 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:09:22.486 DEBUG:boto:Data: 2024-04-12T04:09:22.486 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.486 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.486 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.486 DEBUG:boto:Params: {} 2024-04-12T04:09:22.486 DEBUG:boto:Token: None 2024-04-12T04:09:22.486 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj57 2024-04-12T04:09:22.486 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:s/ejCb8X2pbRJaURzg4fKj8tvz0= 2024-04-12T04:09:22.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:s/ejCb8X2pbRJaURzg4fKj8tvz0=', 'Content-Length': '0'} 2024-04-12T04:09:22.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0e4625ec05e7338-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.488 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:09:22.489 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:09:22.489 DEBUG:boto:Method: GET 2024-04-12T04:09:22.489 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:09:22.489 DEBUG:boto:Data: 2024-04-12T04:09:22.489 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.489 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.489 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.489 DEBUG:boto:Params: {} 2024-04-12T04:09:22.489 DEBUG:boto:Token: None 2024-04-12T04:09:22.489 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj57 2024-04-12T04:09:22.489 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:s/ejCb8X2pbRJaURzg4fKj8tvz0= 2024-04-12T04:09:22.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:s/ejCb8X2pbRJaURzg4fKj8tvz0=', 'Content-Length': '0'} 2024-04-12T04:09:22.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0b1e54a7bc4de00-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.491 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.492 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.492 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:09:22.492 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:09:22.492 DEBUG:boto:Method: GET 2024-04-12T04:09:22.492 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:09:22.492 DEBUG:boto:Data: 2024-04-12T04:09:22.492 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.492 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.492 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.492 DEBUG:boto:Params: {} 2024-04-12T04:09:22.493 DEBUG:boto:Token: None 2024-04-12T04:09:22.493 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.493 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5e2da488948e9cd-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.496 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.496 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.496 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.496 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:09:22.496 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:09:22.496 DEBUG:boto:Method: GET 2024-04-12T04:09:22.496 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:09:22.496 DEBUG:boto:Data: 2024-04-12T04:09:22.497 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.497 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.497 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.497 DEBUG:boto:Params: {} 2024-04-12T04:09:22.497 DEBUG:boto:Token: None 2024-04-12T04:09:22.497 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.497 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000670ab668c471e5a4-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.546 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:09:22.547 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:09:22.547 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:09:22.547 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.547 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.547 DEBUG:boto:Method: GET 2024-04-12T04:09:22.547 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.547 DEBUG:boto:Data: 2024-04-12T04:09:22.548 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.548 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.548 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.548 DEBUG:boto:Params: {} 2024-04-12T04:09:22.548 DEBUG:boto:Token: None 2024-04-12T04:09:22.548 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.548 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H3VhiVr8ui/rQ/RAbh4KPVj9e6I= 2024-04-12T04:09:22.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H3VhiVr8ui/rQ/RAbh4KPVj9e6I=', 'Content-Length': '0'} 2024-04-12T04:09:22.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f5423eabcb51bb8-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.550 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:09:22.551 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:09:22.551 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.551 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.551 DEBUG:boto:Method: GET 2024-04-12T04:09:22.551 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.551 DEBUG:boto:Data: 2024-04-12T04:09:22.551 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.551 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.551 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.551 DEBUG:boto:Params: {} 2024-04-12T04:09:22.551 DEBUG:boto:Token: None 2024-04-12T04:09:22.552 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.552 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H3VhiVr8ui/rQ/RAbh4KPVj9e6I= 2024-04-12T04:09:22.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H3VhiVr8ui/rQ/RAbh4KPVj9e6I=', 'Content-Length': '0'} 2024-04-12T04:09:22.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098bc2b5f7098d050-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.554 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:09:22.554 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:09:22.555 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.555 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.555 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.555 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.555 DEBUG:boto:Data: 2024-04-12T04:09:22.555 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.555 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.555 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.555 DEBUG:boto:Params: {} 2024-04-12T04:09:22.555 DEBUG:boto:Token: None 2024-04-12T04:09:22.556 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.556 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gFxlSFiCd/FZRSXyGXaM13ijUWo= 2024-04-12T04:09:22.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gFxlSFiCd/FZRSXyGXaM13ijUWo=', 'Content-Length': '0'} 2024-04-12T04:09:22.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ab5488a5c88315b-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.558 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:09:22.558 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:09:22.558 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.558 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.558 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.558 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.558 DEBUG:boto:Data: 2024-04-12T04:09:22.559 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.559 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.559 DEBUG:boto:Params: {} 2024-04-12T04:09:22.559 DEBUG:boto:Token: None 2024-04-12T04:09:22.559 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.559 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gFxlSFiCd/FZRSXyGXaM13ijUWo= 2024-04-12T04:09:22.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gFxlSFiCd/FZRSXyGXaM13ijUWo=', 'Content-Length': '0'} 2024-04-12T04:09:22.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001197d552480d350b-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:09:22.561 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:09:22.562 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:09:22.562 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.562 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.562 DEBUG:boto:Method: GET 2024-04-12T04:09:22.562 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.562 DEBUG:boto:Data: 2024-04-12T04:09:22.562 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.562 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.562 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.562 DEBUG:boto:Params: {} 2024-04-12T04:09:22.562 DEBUG:boto:Token: None 2024-04-12T04:09:22.563 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.563 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H3VhiVr8ui/rQ/RAbh4KPVj9e6I= 2024-04-12T04:09:22.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H3VhiVr8ui/rQ/RAbh4KPVj9e6I=', 'Content-Length': '0'} 2024-04-12T04:09:22.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d90b4aa58483b3b-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.606 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:09:22.606 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:09:22.607 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.607 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.607 DEBUG:boto:Method: GET 2024-04-12T04:09:22.607 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.607 DEBUG:boto:Data: 2024-04-12T04:09:22.607 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.607 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.607 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.607 DEBUG:boto:Params: {} 2024-04-12T04:09:22.607 DEBUG:boto:Token: None 2024-04-12T04:09:22.608 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:09:22.608 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H3VhiVr8ui/rQ/RAbh4KPVj9e6I= 2024-04-12T04:09:22.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H3VhiVr8ui/rQ/RAbh4KPVj9e6I=', 'Content-Length': '0'} 2024-04-12T04:09:22.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000128e7eb9c80a2473-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.610 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.611 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.611 DEBUG:boto:path=/dkyges-61/?prefix=obj58 2024-04-12T04:09:22.611 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj58 2024-04-12T04:09:22.611 DEBUG:boto:Method: GET 2024-04-12T04:09:22.611 DEBUG:boto:Path: /dkyges-61/?prefix=obj58 2024-04-12T04:09:22.611 DEBUG:boto:Data: 2024-04-12T04:09:22.611 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.611 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.611 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.611 DEBUG:boto:Params: {} 2024-04-12T04:09:22.611 DEBUG:boto:Token: None 2024-04-12T04:09:22.612 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.612 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7e0967493eceb91-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.614 DEBUG:boto:b'dkyges-61obj581000falseobj582024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.614 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.615 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.615 DEBUG:boto:path=/dkyges-61/?prefix=obj58 2024-04-12T04:09:22.615 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj58 2024-04-12T04:09:22.615 DEBUG:boto:Method: GET 2024-04-12T04:09:22.615 DEBUG:boto:Path: /dkyges-61/?prefix=obj58 2024-04-12T04:09:22.615 DEBUG:boto:Data: 2024-04-12T04:09:22.615 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.615 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.615 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.615 DEBUG:boto:Params: {} 2024-04-12T04:09:22.616 DEBUG:boto:Token: None 2024-04-12T04:09:22.616 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.616 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d9cc86b49e35ec9-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.620 DEBUG:boto:b'dkyges-61obj581000falseobj582024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.620 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:09:22.621 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:09:22.621 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.621 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:09:22.621 DEBUG:boto:Data: 2024-04-12T04:09:22.621 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.621 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.621 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.621 DEBUG:boto:Params: {} 2024-04-12T04:09:22.621 DEBUG:boto:Token: None 2024-04-12T04:09:22.621 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj58 2024-04-12T04:09:22.622 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jlJAO7TYvRK+6W7eJPu3F15MoIo= 2024-04-12T04:09:22.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jlJAO7TYvRK+6W7eJPu3F15MoIo=', 'Content-Length': '0'} 2024-04-12T04:09:22.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000515f8f0febd5e4b0-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.623 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:09:22.624 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:09:22.624 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.624 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:09:22.624 DEBUG:boto:Data: 2024-04-12T04:09:22.624 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.624 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.624 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.624 DEBUG:boto:Params: {} 2024-04-12T04:09:22.624 DEBUG:boto:Token: None 2024-04-12T04:09:22.624 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj58 2024-04-12T04:09:22.624 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jlJAO7TYvRK+6W7eJPu3F15MoIo= 2024-04-12T04:09:22.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jlJAO7TYvRK+6W7eJPu3F15MoIo=', 'Content-Length': '0'} 2024-04-12T04:09:22.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000559e8e03628235ee-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:09:22.626 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:09:22.627 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:09:22.627 DEBUG:boto:Method: GET 2024-04-12T04:09:22.627 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:09:22.627 DEBUG:boto:Data: 2024-04-12T04:09:22.627 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.627 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.627 DEBUG:boto:Params: {} 2024-04-12T04:09:22.627 DEBUG:boto:Token: None 2024-04-12T04:09:22.627 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj58 2024-04-12T04:09:22.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FxKexnAOmnbG74l7gDogPiQytFs= 2024-04-12T04:09:22.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FxKexnAOmnbG74l7gDogPiQytFs=', 'Content-Length': '0'} 2024-04-12T04:09:22.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b504d3504b8a0da3-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.629 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:09:22.630 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:09:22.630 DEBUG:boto:Method: GET 2024-04-12T04:09:22.630 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:09:22.630 DEBUG:boto:Data: 2024-04-12T04:09:22.630 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.630 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.630 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.630 DEBUG:boto:Params: {} 2024-04-12T04:09:22.630 DEBUG:boto:Token: None 2024-04-12T04:09:22.630 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj58 2024-04-12T04:09:22.631 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FxKexnAOmnbG74l7gDogPiQytFs= 2024-04-12T04:09:22.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FxKexnAOmnbG74l7gDogPiQytFs=', 'Content-Length': '0'} 2024-04-12T04:09:22.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d4fb890c47deca1-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.632 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.633 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.633 DEBUG:boto:path=/dkyges-61/?prefix=obj58 2024-04-12T04:09:22.633 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj58 2024-04-12T04:09:22.633 DEBUG:boto:Method: GET 2024-04-12T04:09:22.633 DEBUG:boto:Path: /dkyges-61/?prefix=obj58 2024-04-12T04:09:22.633 DEBUG:boto:Data: 2024-04-12T04:09:22.633 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.633 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.633 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.633 DEBUG:boto:Params: {} 2024-04-12T04:09:22.634 DEBUG:boto:Token: None 2024-04-12T04:09:22.634 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.634 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b79e26940177ac50-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.636 DEBUG:boto:b'dkyges-61obj581000falseobj582024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.637 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.637 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.637 DEBUG:boto:path=/dkyges-61/?prefix=obj58 2024-04-12T04:09:22.637 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj58 2024-04-12T04:09:22.637 DEBUG:boto:Method: GET 2024-04-12T04:09:22.637 DEBUG:boto:Path: /dkyges-61/?prefix=obj58 2024-04-12T04:09:22.637 DEBUG:boto:Data: 2024-04-12T04:09:22.637 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.637 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.637 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.637 DEBUG:boto:Params: {} 2024-04-12T04:09:22.638 DEBUG:boto:Token: None 2024-04-12T04:09:22.638 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.638 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077d1ab936d69d648-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.642 DEBUG:boto:b'dkyges-61obj581000falseobj582024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T04:09:22.643 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:09:22.643 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:09:22.643 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.643 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.643 DEBUG:boto:Method: GET 2024-04-12T04:09:22.643 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.643 DEBUG:boto:Data: 2024-04-12T04:09:22.643 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.643 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.644 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.644 DEBUG:boto:Params: {} 2024-04-12T04:09:22.644 DEBUG:boto:Token: None 2024-04-12T04:09:22.644 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.644 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9wRr641KUq/CaAHELFtYgQ1S8Ag= 2024-04-12T04:09:22.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9wRr641KUq/CaAHELFtYgQ1S8Ag=', 'Content-Length': '0'} 2024-04-12T04:09:22.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7ea5c4bf6fd51fd-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.649 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:09:22.649 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:09:22.649 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.649 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.650 DEBUG:boto:Method: GET 2024-04-12T04:09:22.650 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.650 DEBUG:boto:Data: 2024-04-12T04:09:22.650 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.650 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.650 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.650 DEBUG:boto:Params: {} 2024-04-12T04:09:22.650 DEBUG:boto:Token: None 2024-04-12T04:09:22.650 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.651 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9wRr641KUq/CaAHELFtYgQ1S8Ag= 2024-04-12T04:09:22.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9wRr641KUq/CaAHELFtYgQ1S8Ag=', 'Content-Length': '0'} 2024-04-12T04:09:22.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c8921438baadf95-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.653 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:09:22.653 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:09:22.653 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.653 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.653 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.653 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.654 DEBUG:boto:Data: 2024-04-12T04:09:22.654 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.654 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.654 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.654 DEBUG:boto:Params: {} 2024-04-12T04:09:22.654 DEBUG:boto:Token: None 2024-04-12T04:09:22.654 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.654 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:91ZsuGvaKuvtc/IxeBL6/R5xdJ8= 2024-04-12T04:09:22.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:91ZsuGvaKuvtc/IxeBL6/R5xdJ8=', 'Content-Length': '0'} 2024-04-12T04:09:22.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce9bd04df4a584f6-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.656 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:09:22.657 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:09:22.657 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.657 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.657 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.657 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.657 DEBUG:boto:Data: 2024-04-12T04:09:22.657 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.657 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.657 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.657 DEBUG:boto:Params: {} 2024-04-12T04:09:22.657 DEBUG:boto:Token: None 2024-04-12T04:09:22.658 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.658 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:91ZsuGvaKuvtc/IxeBL6/R5xdJ8= 2024-04-12T04:09:22.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:91ZsuGvaKuvtc/IxeBL6/R5xdJ8=', 'Content-Length': '0'} 2024-04-12T04:09:22.660 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1632293e15d3275-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T04:09:22.660 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:09:22.660 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:09:22.660 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.660 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.660 DEBUG:boto:Method: GET 2024-04-12T04:09:22.661 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.661 DEBUG:boto:Data: 2024-04-12T04:09:22.661 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.661 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.661 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.661 DEBUG:boto:Params: {} 2024-04-12T04:09:22.661 DEBUG:boto:Token: None 2024-04-12T04:09:22.661 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.661 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9wRr641KUq/CaAHELFtYgQ1S8Ag= 2024-04-12T04:09:22.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9wRr641KUq/CaAHELFtYgQ1S8Ag=', 'Content-Length': '0'} 2024-04-12T04:09:22.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000408f781d93f119d8-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.664 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:09:22.664 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:09:22.664 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.664 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.664 DEBUG:boto:Method: GET 2024-04-12T04:09:22.664 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.664 DEBUG:boto:Data: 2024-04-12T04:09:22.665 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.665 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.665 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.665 DEBUG:boto:Params: {} 2024-04-12T04:09:22.665 DEBUG:boto:Token: None 2024-04-12T04:09:22.665 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:09:22.665 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9wRr641KUq/CaAHELFtYgQ1S8Ag= 2024-04-12T04:09:22.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9wRr641KUq/CaAHELFtYgQ1S8Ag=', 'Content-Length': '0'} 2024-04-12T04:09:22.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007355b614b2f51f9e-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.667 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.668 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.668 DEBUG:boto:path=/dkyges-61/?prefix=obj62 2024-04-12T04:09:22.668 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj62 2024-04-12T04:09:22.668 DEBUG:boto:Method: GET 2024-04-12T04:09:22.668 DEBUG:boto:Path: /dkyges-61/?prefix=obj62 2024-04-12T04:09:22.668 DEBUG:boto:Data: 2024-04-12T04:09:22.668 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.668 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.668 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.668 DEBUG:boto:Params: {} 2024-04-12T04:09:22.668 DEBUG:boto:Token: None 2024-04-12T04:09:22.669 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.669 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003903811f46172f62-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.671 DEBUG:boto:b'dkyges-61obj621000falseobj622024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.672 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.672 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.672 DEBUG:boto:path=/dkyges-61/?prefix=obj62 2024-04-12T04:09:22.672 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj62 2024-04-12T04:09:22.672 DEBUG:boto:Method: GET 2024-04-12T04:09:22.672 DEBUG:boto:Path: /dkyges-61/?prefix=obj62 2024-04-12T04:09:22.672 DEBUG:boto:Data: 2024-04-12T04:09:22.672 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.673 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.673 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.673 DEBUG:boto:Params: {} 2024-04-12T04:09:22.673 DEBUG:boto:Token: None 2024-04-12T04:09:22.673 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.673 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000095a2eb9d9fab95c-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.718 DEBUG:boto:b'dkyges-61obj621000falseobj622024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.719 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:09:22.719 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:09:22.719 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.719 DEBUG:boto:Path: /dkyges-61/obj62 2024-04-12T04:09:22.719 DEBUG:boto:Data: 2024-04-12T04:09:22.719 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.719 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.719 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.719 DEBUG:boto:Params: {} 2024-04-12T04:09:22.720 DEBUG:boto:Token: None 2024-04-12T04:09:22.720 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj62 2024-04-12T04:09:22.720 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d36nUBbCcSz6OzTq8n6ZUfKRs0I= 2024-04-12T04:09:22.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d36nUBbCcSz6OzTq8n6ZUfKRs0I=', 'Content-Length': '0'} 2024-04-12T04:09:22.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba35216f837a84eb-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.722 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:09:22.722 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:09:22.723 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.723 DEBUG:boto:Path: /dkyges-61/obj62 2024-04-12T04:09:22.723 DEBUG:boto:Data: 2024-04-12T04:09:22.723 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.723 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.723 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.723 DEBUG:boto:Params: {} 2024-04-12T04:09:22.723 DEBUG:boto:Token: None 2024-04-12T04:09:22.723 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj62 2024-04-12T04:09:22.723 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d36nUBbCcSz6OzTq8n6ZUfKRs0I= 2024-04-12T04:09:22.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d36nUBbCcSz6OzTq8n6ZUfKRs0I=', 'Content-Length': '0'} 2024-04-12T04:09:22.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030b1d4961330ea9a-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T04:09:22.726 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:09:22.726 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:09:22.726 DEBUG:boto:Method: GET 2024-04-12T04:09:22.726 DEBUG:boto:Path: /dkyges-61/obj62 2024-04-12T04:09:22.726 DEBUG:boto:Data: 2024-04-12T04:09:22.726 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.726 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.726 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.727 DEBUG:boto:Params: {} 2024-04-12T04:09:22.727 DEBUG:boto:Token: None 2024-04-12T04:09:22.727 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj62 2024-04-12T04:09:22.727 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A3ZRO/H0gpXqozgNeVJaD6Y6YwU= 2024-04-12T04:09:22.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A3ZRO/H0gpXqozgNeVJaD6Y6YwU=', 'Content-Length': '0'} 2024-04-12T04:09:22.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fdf9d494420c826-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.729 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:09:22.729 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:09:22.729 DEBUG:boto:Method: GET 2024-04-12T04:09:22.729 DEBUG:boto:Path: /dkyges-61/obj62 2024-04-12T04:09:22.730 DEBUG:boto:Data: 2024-04-12T04:09:22.730 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.730 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.730 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.730 DEBUG:boto:Params: {} 2024-04-12T04:09:22.730 DEBUG:boto:Token: None 2024-04-12T04:09:22.730 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj62 2024-04-12T04:09:22.730 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A3ZRO/H0gpXqozgNeVJaD6Y6YwU= 2024-04-12T04:09:22.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A3ZRO/H0gpXqozgNeVJaD6Y6YwU=', 'Content-Length': '0'} 2024-04-12T04:09:22.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000862ae4213b2529b3-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.732 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.733 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.733 DEBUG:boto:path=/dkyges-61/?prefix=obj62 2024-04-12T04:09:22.733 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj62 2024-04-12T04:09:22.733 DEBUG:boto:Method: GET 2024-04-12T04:09:22.733 DEBUG:boto:Path: /dkyges-61/?prefix=obj62 2024-04-12T04:09:22.733 DEBUG:boto:Data: 2024-04-12T04:09:22.733 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.733 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.733 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.733 DEBUG:boto:Params: {} 2024-04-12T04:09:22.733 DEBUG:boto:Token: None 2024-04-12T04:09:22.734 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.734 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047f2c8008887620a-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.736 DEBUG:boto:b'dkyges-61obj621000falseobj622024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.737 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.737 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.737 DEBUG:boto:path=/dkyges-61/?prefix=obj62 2024-04-12T04:09:22.737 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj62 2024-04-12T04:09:22.737 DEBUG:boto:Method: GET 2024-04-12T04:09:22.737 DEBUG:boto:Path: /dkyges-61/?prefix=obj62 2024-04-12T04:09:22.737 DEBUG:boto:Data: 2024-04-12T04:09:22.737 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.737 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.738 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.738 DEBUG:boto:Params: {} 2024-04-12T04:09:22.738 DEBUG:boto:Token: None 2024-04-12T04:09:22.738 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.738 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a93fb70d07194b8d-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.742 DEBUG:boto:b'dkyges-61obj621000falseobj622024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:09:22.743 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:09:22.743 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:09:22.743 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.743 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.743 DEBUG:boto:Method: GET 2024-04-12T04:09:22.743 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.743 DEBUG:boto:Data: 2024-04-12T04:09:22.743 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.743 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.744 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.744 DEBUG:boto:Params: {} 2024-04-12T04:09:22.744 DEBUG:boto:Token: None 2024-04-12T04:09:22.744 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.744 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I9TQN/zpCTFc2719hwytoBsPgvQ= 2024-04-12T04:09:22.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I9TQN/zpCTFc2719hwytoBsPgvQ=', 'Content-Length': '0'} 2024-04-12T04:09:22.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040a9d0dc96f87e29-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.746 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:09:22.747 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:09:22.747 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.747 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.747 DEBUG:boto:Method: GET 2024-04-12T04:09:22.747 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.747 DEBUG:boto:Data: 2024-04-12T04:09:22.747 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.747 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.747 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.747 DEBUG:boto:Params: {} 2024-04-12T04:09:22.748 DEBUG:boto:Token: None 2024-04-12T04:09:22.748 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.748 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I9TQN/zpCTFc2719hwytoBsPgvQ= 2024-04-12T04:09:22.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I9TQN/zpCTFc2719hwytoBsPgvQ=', 'Content-Length': '0'} 2024-04-12T04:09:22.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7b40ef8bdb88899-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.750 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:09:22.750 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:09:22.750 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.750 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.750 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.750 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.751 DEBUG:boto:Data: 2024-04-12T04:09:22.751 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.751 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.751 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.751 DEBUG:boto:Params: {} 2024-04-12T04:09:22.751 DEBUG:boto:Token: None 2024-04-12T04:09:22.751 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.751 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dBlEwAJIIoxC5eBehe2ykkW4Xwc= 2024-04-12T04:09:22.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dBlEwAJIIoxC5eBehe2ykkW4Xwc=', 'Content-Length': '0'} 2024-04-12T04:09:22.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006604f7ae882f94d4-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.753 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:09:22.753 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:09:22.753 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.754 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.754 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.754 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.754 DEBUG:boto:Data: 2024-04-12T04:09:22.754 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.754 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.754 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.754 DEBUG:boto:Params: {} 2024-04-12T04:09:22.754 DEBUG:boto:Token: None 2024-04-12T04:09:22.754 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.755 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dBlEwAJIIoxC5eBehe2ykkW4Xwc= 2024-04-12T04:09:22.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dBlEwAJIIoxC5eBehe2ykkW4Xwc=', 'Content-Length': '0'} 2024-04-12T04:09:22.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c38842d73fbd906-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:09:22.757 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:09:22.757 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:09:22.757 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.757 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.757 DEBUG:boto:Method: GET 2024-04-12T04:09:22.757 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.757 DEBUG:boto:Data: 2024-04-12T04:09:22.757 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.757 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.758 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.758 DEBUG:boto:Params: {} 2024-04-12T04:09:22.758 DEBUG:boto:Token: None 2024-04-12T04:09:22.758 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.758 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I9TQN/zpCTFc2719hwytoBsPgvQ= 2024-04-12T04:09:22.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I9TQN/zpCTFc2719hwytoBsPgvQ=', 'Content-Length': '0'} 2024-04-12T04:09:22.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0180bdab5080511-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.802 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:09:22.802 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:09:22.803 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.803 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.803 DEBUG:boto:Method: GET 2024-04-12T04:09:22.803 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.803 DEBUG:boto:Data: 2024-04-12T04:09:22.803 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.803 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.803 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.803 DEBUG:boto:Params: {} 2024-04-12T04:09:22.803 DEBUG:boto:Token: None 2024-04-12T04:09:22.804 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:09:22.804 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I9TQN/zpCTFc2719hwytoBsPgvQ= 2024-04-12T04:09:22.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I9TQN/zpCTFc2719hwytoBsPgvQ=', 'Content-Length': '0'} 2024-04-12T04:09:22.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da80dc0499d99910-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.806 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.806 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.806 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:09:22.806 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:09:22.806 DEBUG:boto:Method: GET 2024-04-12T04:09:22.806 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:09:22.807 DEBUG:boto:Data: 2024-04-12T04:09:22.807 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.807 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.807 DEBUG:boto:Params: {} 2024-04-12T04:09:22.807 DEBUG:boto:Token: None 2024-04-12T04:09:22.807 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.807 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002736fd6df9953751-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.810 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.810 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.810 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.811 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:09:22.811 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:09:22.811 DEBUG:boto:Method: GET 2024-04-12T04:09:22.811 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:09:22.811 DEBUG:boto:Data: 2024-04-12T04:09:22.811 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.811 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.811 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.811 DEBUG:boto:Params: {} 2024-04-12T04:09:22.811 DEBUG:boto:Token: None 2024-04-12T04:09:22.811 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.812 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079defc1fd57b0680-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.816 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.816 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:09:22.816 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:09:22.816 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.816 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:09:22.816 DEBUG:boto:Data: 2024-04-12T04:09:22.816 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.817 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.817 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.817 DEBUG:boto:Params: {} 2024-04-12T04:09:22.817 DEBUG:boto:Token: None 2024-04-12T04:09:22.817 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj68 2024-04-12T04:09:22.817 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9Bcmkv32LMe82PujVbSSdD4OUe8= 2024-04-12T04:09:22.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9Bcmkv32LMe82PujVbSSdD4OUe8=', 'Content-Length': '0'} 2024-04-12T04:09:22.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033235a685f7c5375-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.820 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:09:22.820 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:09:22.820 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.820 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:09:22.820 DEBUG:boto:Data: 2024-04-12T04:09:22.820 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.820 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.820 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.820 DEBUG:boto:Params: {} 2024-04-12T04:09:22.821 DEBUG:boto:Token: None 2024-04-12T04:09:22.821 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj68 2024-04-12T04:09:22.821 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9Bcmkv32LMe82PujVbSSdD4OUe8= 2024-04-12T04:09:22.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9Bcmkv32LMe82PujVbSSdD4OUe8=', 'Content-Length': '0'} 2024-04-12T04:09:22.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000645ca0b36903bbed-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:09:22.823 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:09:22.823 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:09:22.824 DEBUG:boto:Method: GET 2024-04-12T04:09:22.824 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:09:22.824 DEBUG:boto:Data: 2024-04-12T04:09:22.824 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.824 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.824 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.824 DEBUG:boto:Params: {} 2024-04-12T04:09:22.824 DEBUG:boto:Token: None 2024-04-12T04:09:22.824 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj68 2024-04-12T04:09:22.824 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xWmgPYtEYZoG8tajGcc8fYkjyEY= 2024-04-12T04:09:22.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xWmgPYtEYZoG8tajGcc8fYkjyEY=', 'Content-Length': '0'} 2024-04-12T04:09:22.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8a45dbcfb7ea0cf-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.826 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:09:22.827 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:09:22.827 DEBUG:boto:Method: GET 2024-04-12T04:09:22.827 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:09:22.827 DEBUG:boto:Data: 2024-04-12T04:09:22.827 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.827 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.827 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.827 DEBUG:boto:Params: {} 2024-04-12T04:09:22.827 DEBUG:boto:Token: None 2024-04-12T04:09:22.827 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj68 2024-04-12T04:09:22.828 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xWmgPYtEYZoG8tajGcc8fYkjyEY= 2024-04-12T04:09:22.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xWmgPYtEYZoG8tajGcc8fYkjyEY=', 'Content-Length': '0'} 2024-04-12T04:09:22.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f5a7b21fd2bb84a-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.870 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.871 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.871 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:09:22.871 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:09:22.871 DEBUG:boto:Method: GET 2024-04-12T04:09:22.871 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:09:22.871 DEBUG:boto:Data: 2024-04-12T04:09:22.871 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.871 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.871 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.872 DEBUG:boto:Params: {} 2024-04-12T04:09:22.872 DEBUG:boto:Token: None 2024-04-12T04:09:22.872 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.872 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ac71d64019b04cc-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.875 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.875 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.876 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.876 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:09:22.876 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:09:22.876 DEBUG:boto:Method: GET 2024-04-12T04:09:22.876 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:09:22.876 DEBUG:boto:Data: 2024-04-12T04:09:22.876 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.876 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.876 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.876 DEBUG:boto:Params: {} 2024-04-12T04:09:22.876 DEBUG:boto:Token: None 2024-04-12T04:09:22.876 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.877 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b31e2a01b639ef7f-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.881 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:09:22.882 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:09:22.882 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:09:22.882 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.882 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.882 DEBUG:boto:Method: GET 2024-04-12T04:09:22.882 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.882 DEBUG:boto:Data: 2024-04-12T04:09:22.882 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.882 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.883 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.883 DEBUG:boto:Params: {} 2024-04-12T04:09:22.883 DEBUG:boto:Token: None 2024-04-12T04:09:22.883 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.883 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WmorWkmO6E/KdqWDd657BUgvn6s= 2024-04-12T04:09:22.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WmorWkmO6E/KdqWDd657BUgvn6s=', 'Content-Length': '0'} 2024-04-12T04:09:22.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cf1bf7892dca351-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.926 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:09:22.926 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:09:22.927 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.927 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.927 DEBUG:boto:Method: GET 2024-04-12T04:09:22.927 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.927 DEBUG:boto:Data: 2024-04-12T04:09:22.927 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.927 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.927 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.927 DEBUG:boto:Params: {} 2024-04-12T04:09:22.927 DEBUG:boto:Token: None 2024-04-12T04:09:22.928 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.928 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WmorWkmO6E/KdqWDd657BUgvn6s= 2024-04-12T04:09:22.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WmorWkmO6E/KdqWDd657BUgvn6s=', 'Content-Length': '0'} 2024-04-12T04:09:22.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074feff8d4e7d07a7-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.930 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:09:22.930 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:09:22.930 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.930 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.930 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.930 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.931 DEBUG:boto:Data: 2024-04-12T04:09:22.931 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.931 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.931 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.931 DEBUG:boto:Params: {} 2024-04-12T04:09:22.931 DEBUG:boto:Token: None 2024-04-12T04:09:22.931 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.931 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8VQmZUpH707d6GiejnaS+6w+V34= 2024-04-12T04:09:22.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8VQmZUpH707d6GiejnaS+6w+V34=', 'Content-Length': '0'} 2024-04-12T04:09:22.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2bca706f40445ac-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.933 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:09:22.933 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:09:22.933 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.934 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.934 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.934 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.934 DEBUG:boto:Data: 2024-04-12T04:09:22.934 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.934 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.934 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.934 DEBUG:boto:Params: {} 2024-04-12T04:09:22.934 DEBUG:boto:Token: None 2024-04-12T04:09:22.934 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.935 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8VQmZUpH707d6GiejnaS+6w+V34= 2024-04-12T04:09:22.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8VQmZUpH707d6GiejnaS+6w+V34=', 'Content-Length': '0'} 2024-04-12T04:09:22.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076a32249c5b58953-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:09:22.937 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:09:22.937 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:09:22.937 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.937 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.937 DEBUG:boto:Method: GET 2024-04-12T04:09:22.937 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.938 DEBUG:boto:Data: 2024-04-12T04:09:22.938 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.938 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.938 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.938 DEBUG:boto:Params: {} 2024-04-12T04:09:22.938 DEBUG:boto:Token: None 2024-04-12T04:09:22.938 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.938 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WmorWkmO6E/KdqWDd657BUgvn6s= 2024-04-12T04:09:22.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WmorWkmO6E/KdqWDd657BUgvn6s=', 'Content-Length': '0'} 2024-04-12T04:09:22.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007309c9e7432791a9-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.940 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:09:22.940 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:09:22.940 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.940 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.941 DEBUG:boto:Method: GET 2024-04-12T04:09:22.941 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.941 DEBUG:boto:Data: 2024-04-12T04:09:22.941 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.941 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.941 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.941 DEBUG:boto:Params: {} 2024-04-12T04:09:22.941 DEBUG:boto:Token: None 2024-04-12T04:09:22.941 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:09:22.941 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WmorWkmO6E/KdqWDd657BUgvn6s= 2024-04-12T04:09:22.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WmorWkmO6E/KdqWDd657BUgvn6s=', 'Content-Length': '0'} 2024-04-12T04:09:22.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6d48daf9df48bf4-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.944 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.944 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.944 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:09:22.944 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:09:22.944 DEBUG:boto:Method: GET 2024-04-12T04:09:22.944 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:09:22.944 DEBUG:boto:Data: 2024-04-12T04:09:22.944 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.944 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.944 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.944 DEBUG:boto:Params: {} 2024-04-12T04:09:22.945 DEBUG:boto:Token: None 2024-04-12T04:09:22.945 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.945 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008383c57ff229e971-006618b3f2-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.947 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.948 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:22.948 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:22.948 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:09:22.948 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:09:22.948 DEBUG:boto:Method: GET 2024-04-12T04:09:22.948 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:09:22.948 DEBUG:boto:Data: 2024-04-12T04:09:22.948 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.948 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.948 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.949 DEBUG:boto:Params: {} 2024-04-12T04:09:22.949 DEBUG:boto:Token: None 2024-04-12T04:09:22.949 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/ 2024-04-12T04:09:22.949 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0= 2024-04-12T04:09:22.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3kzbECGkOZT21isNmHPwfmqYDN0=', 'Content-Length': '0'} 2024-04-12T04:09:22.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a06605e2928411a-006618b3f2-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.953 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:22.954 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:09:22.954 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:09:22.954 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.954 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:09:22.954 DEBUG:boto:Data: 2024-04-12T04:09:22.954 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.954 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.954 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.954 DEBUG:boto:Params: {} 2024-04-12T04:09:22.954 DEBUG:boto:Token: None 2024-04-12T04:09:22.955 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj69 2024-04-12T04:09:22.955 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2MPP1dCCKVYdCb0e5Nu8Laked40= 2024-04-12T04:09:22.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2MPP1dCCKVYdCb0e5Nu8Laked40=', 'Content-Length': '0'} 2024-04-12T04:09:22.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000223efc32a1436485-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.957 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:09:22.957 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:09:22.957 DEBUG:boto:Method: HEAD 2024-04-12T04:09:22.957 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:09:22.957 DEBUG:boto:Data: 2024-04-12T04:09:22.957 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.957 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.957 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.958 DEBUG:boto:Params: {} 2024-04-12T04:09:22.958 DEBUG:boto:Token: None 2024-04-12T04:09:22.958 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj69 2024-04-12T04:09:22.958 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2MPP1dCCKVYdCb0e5Nu8Laked40= 2024-04-12T04:09:22.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2MPP1dCCKVYdCb0e5Nu8Laked40=', 'Content-Length': '0'} 2024-04-12T04:09:22.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3c41a1e500fede6-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:09:22.960 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:09:22.960 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:09:22.960 DEBUG:boto:Method: GET 2024-04-12T04:09:22.960 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:09:22.961 DEBUG:boto:Data: 2024-04-12T04:09:22.961 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.961 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:22.961 DEBUG:boto:Port: 8001 2024-04-12T04:09:22.961 DEBUG:boto:Params: {} 2024-04-12T04:09:22.961 DEBUG:boto:Token: None 2024-04-12T04:09:22.961 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj69 2024-04-12T04:09:22.962 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fFyOwumL7VQh4PY4e+2tbf+jJQs= 2024-04-12T04:09:22.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fFyOwumL7VQh4PY4e+2tbf+jJQs=', 'Content-Length': '0'} 2024-04-12T04:09:22.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f121dfd78959c444-006618b3f2-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:22.964 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:09:22.964 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:09:22.964 DEBUG:boto:Method: GET 2024-04-12T04:09:22.964 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:09:22.964 DEBUG:boto:Data: 2024-04-12T04:09:22.964 DEBUG:boto:Headers: {} 2024-04-12T04:09:22.964 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:22.964 DEBUG:boto:Port: 8000 2024-04-12T04:09:22.964 DEBUG:boto:Params: {} 2024-04-12T04:09:22.964 DEBUG:boto:Token: None 2024-04-12T04:09:22.965 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:22 GMT /dkyges-61/obj69 2024-04-12T04:09:22.965 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fFyOwumL7VQh4PY4e+2tbf+jJQs= 2024-04-12T04:09:22.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:22 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fFyOwumL7VQh4PY4e+2tbf+jJQs=', 'Content-Length': '0'} 2024-04-12T04:09:22.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2dda23e1e1a6b2c-006618b3f2-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:22 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.014 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.015 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.015 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:09:23.015 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:09:23.015 DEBUG:boto:Method: GET 2024-04-12T04:09:23.015 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:09:23.015 DEBUG:boto:Data: 2024-04-12T04:09:23.015 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.015 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.015 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.015 DEBUG:boto:Params: {} 2024-04-12T04:09:23.016 DEBUG:boto:Token: None 2024-04-12T04:09:23.016 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.016 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000979233eda0d7f22a-006618b3f3-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.019 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.019 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.019 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.019 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:09:23.019 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:09:23.019 DEBUG:boto:Method: GET 2024-04-12T04:09:23.020 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:09:23.020 DEBUG:boto:Data: 2024-04-12T04:09:23.020 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.020 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.020 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.020 DEBUG:boto:Params: {} 2024-04-12T04:09:23.020 DEBUG:boto:Token: None 2024-04-12T04:09:23.020 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.020 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010f2786df9b76cd4-006618b3f3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.024 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:09:23.025 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:09:23.025 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:09:23.025 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.025 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.025 DEBUG:boto:Method: GET 2024-04-12T04:09:23.025 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.025 DEBUG:boto:Data: 2024-04-12T04:09:23.025 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.025 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.026 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.026 DEBUG:boto:Params: {} 2024-04-12T04:09:23.026 DEBUG:boto:Token: None 2024-04-12T04:09:23.026 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.026 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GHCs01221f/esYDDOrM7OYTKKFk= 2024-04-12T04:09:23.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GHCs01221f/esYDDOrM7OYTKKFk=', 'Content-Length': '0'} 2024-04-12T04:09:23.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017cdc869af0a86b5-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.070 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:09:23.071 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:09:23.071 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.071 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.071 DEBUG:boto:Method: GET 2024-04-12T04:09:23.071 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.071 DEBUG:boto:Data: 2024-04-12T04:09:23.071 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.072 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.072 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.072 DEBUG:boto:Params: {} 2024-04-12T04:09:23.072 DEBUG:boto:Token: None 2024-04-12T04:09:23.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.072 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GHCs01221f/esYDDOrM7OYTKKFk= 2024-04-12T04:09:23.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GHCs01221f/esYDDOrM7OYTKKFk=', 'Content-Length': '0'} 2024-04-12T04:09:23.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004db94124c3e3d85e-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.075 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:09:23.075 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:09:23.075 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.075 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.075 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.075 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.075 DEBUG:boto:Data: 2024-04-12T04:09:23.075 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.076 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.076 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.076 DEBUG:boto:Params: {} 2024-04-12T04:09:23.076 DEBUG:boto:Token: None 2024-04-12T04:09:23.076 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.076 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AVbcDtZe9DqiUS4eNoYLFYDtdgw= 2024-04-12T04:09:23.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AVbcDtZe9DqiUS4eNoYLFYDtdgw=', 'Content-Length': '0'} 2024-04-12T04:09:23.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f86c35bd2ecb523b-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.078 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:09:23.079 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:09:23.079 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.079 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.079 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.079 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.079 DEBUG:boto:Data: 2024-04-12T04:09:23.079 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.079 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.079 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.079 DEBUG:boto:Params: {} 2024-04-12T04:09:23.080 DEBUG:boto:Token: None 2024-04-12T04:09:23.080 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.080 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AVbcDtZe9DqiUS4eNoYLFYDtdgw= 2024-04-12T04:09:23.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AVbcDtZe9DqiUS4eNoYLFYDtdgw=', 'Content-Length': '0'} 2024-04-12T04:09:23.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b55e046d89741760-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:09:23.082 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:09:23.082 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:09:23.082 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.082 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.083 DEBUG:boto:Method: GET 2024-04-12T04:09:23.083 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.083 DEBUG:boto:Data: 2024-04-12T04:09:23.083 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.083 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.083 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.083 DEBUG:boto:Params: {} 2024-04-12T04:09:23.083 DEBUG:boto:Token: None 2024-04-12T04:09:23.083 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.084 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GHCs01221f/esYDDOrM7OYTKKFk= 2024-04-12T04:09:23.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GHCs01221f/esYDDOrM7OYTKKFk=', 'Content-Length': '0'} 2024-04-12T04:09:23.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e942623e6593c7c-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.086 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:09:23.086 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:09:23.086 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.086 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.086 DEBUG:boto:Method: GET 2024-04-12T04:09:23.086 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.086 DEBUG:boto:Data: 2024-04-12T04:09:23.086 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.086 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.087 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.087 DEBUG:boto:Params: {} 2024-04-12T04:09:23.087 DEBUG:boto:Token: None 2024-04-12T04:09:23.087 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:09:23.087 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GHCs01221f/esYDDOrM7OYTKKFk= 2024-04-12T04:09:23.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GHCs01221f/esYDDOrM7OYTKKFk=', 'Content-Length': '0'} 2024-04-12T04:09:23.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f48dfdbf59efd64-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.089 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.090 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.090 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:09:23.090 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:09:23.090 DEBUG:boto:Method: GET 2024-04-12T04:09:23.090 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:09:23.090 DEBUG:boto:Data: 2024-04-12T04:09:23.090 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.090 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.090 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.090 DEBUG:boto:Params: {} 2024-04-12T04:09:23.091 DEBUG:boto:Token: None 2024-04-12T04:09:23.091 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.091 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d92d8f8b17a30e6f-006618b3f3-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.094 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.094 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.094 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.094 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:09:23.094 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:09:23.094 DEBUG:boto:Method: GET 2024-04-12T04:09:23.095 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:09:23.095 DEBUG:boto:Data: 2024-04-12T04:09:23.095 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.095 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.095 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.095 DEBUG:boto:Params: {} 2024-04-12T04:09:23.095 DEBUG:boto:Token: None 2024-04-12T04:09:23.095 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.096 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005983fa01e916acfe-006618b3f3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.100 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.100 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:09:23.101 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:09:23.101 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.101 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:09:23.101 DEBUG:boto:Data: 2024-04-12T04:09:23.101 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.101 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.101 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.101 DEBUG:boto:Params: {} 2024-04-12T04:09:23.101 DEBUG:boto:Token: None 2024-04-12T04:09:23.101 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj76 2024-04-12T04:09:23.102 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XYteHlidPo/FRk7zctcXkoBktRA= 2024-04-12T04:09:23.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XYteHlidPo/FRk7zctcXkoBktRA=', 'Content-Length': '0'} 2024-04-12T04:09:23.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e2efd96255b6cbe-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.104 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:09:23.104 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:09:23.104 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.104 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:09:23.104 DEBUG:boto:Data: 2024-04-12T04:09:23.104 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.104 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.104 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.105 DEBUG:boto:Params: {} 2024-04-12T04:09:23.105 DEBUG:boto:Token: None 2024-04-12T04:09:23.105 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj76 2024-04-12T04:09:23.105 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XYteHlidPo/FRk7zctcXkoBktRA= 2024-04-12T04:09:23.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XYteHlidPo/FRk7zctcXkoBktRA=', 'Content-Length': '0'} 2024-04-12T04:09:23.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5988aa84d0fb9ca-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:09:23.107 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:09:23.107 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:09:23.107 DEBUG:boto:Method: GET 2024-04-12T04:09:23.107 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:09:23.108 DEBUG:boto:Data: 2024-04-12T04:09:23.108 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.108 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.108 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.108 DEBUG:boto:Params: {} 2024-04-12T04:09:23.108 DEBUG:boto:Token: None 2024-04-12T04:09:23.108 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj76 2024-04-12T04:09:23.108 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gEjaTsM8x/LyPTIIq67pp1ewBxs= 2024-04-12T04:09:23.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gEjaTsM8x/LyPTIIq67pp1ewBxs=', 'Content-Length': '0'} 2024-04-12T04:09:23.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea7eab251a352a5c-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.110 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:09:23.110 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:09:23.111 DEBUG:boto:Method: GET 2024-04-12T04:09:23.111 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:09:23.111 DEBUG:boto:Data: 2024-04-12T04:09:23.111 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.111 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.111 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.111 DEBUG:boto:Params: {} 2024-04-12T04:09:23.111 DEBUG:boto:Token: None 2024-04-12T04:09:23.111 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj76 2024-04-12T04:09:23.111 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gEjaTsM8x/LyPTIIq67pp1ewBxs= 2024-04-12T04:09:23.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gEjaTsM8x/LyPTIIq67pp1ewBxs=', 'Content-Length': '0'} 2024-04-12T04:09:23.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bac00bce4193b41-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.155 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.155 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.155 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:09:23.155 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:09:23.155 DEBUG:boto:Method: GET 2024-04-12T04:09:23.155 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:09:23.155 DEBUG:boto:Data: 2024-04-12T04:09:23.155 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.155 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.156 DEBUG:boto:Params: {} 2024-04-12T04:09:23.156 DEBUG:boto:Token: None 2024-04-12T04:09:23.156 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000237468a8d13521de-006618b3f3-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.159 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.159 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.159 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.159 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:09:23.160 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:09:23.160 DEBUG:boto:Method: GET 2024-04-12T04:09:23.160 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:09:23.160 DEBUG:boto:Data: 2024-04-12T04:09:23.160 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.160 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.160 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.160 DEBUG:boto:Params: {} 2024-04-12T04:09:23.160 DEBUG:boto:Token: None 2024-04-12T04:09:23.160 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.161 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e14c21b1154f16d-006618b3f3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.166 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:09:23.166 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:09:23.166 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:09:23.166 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.167 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.167 DEBUG:boto:Method: GET 2024-04-12T04:09:23.167 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.167 DEBUG:boto:Data: 2024-04-12T04:09:23.167 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.167 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.167 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.167 DEBUG:boto:Params: {} 2024-04-12T04:09:23.167 DEBUG:boto:Token: None 2024-04-12T04:09:23.167 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.168 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TmsI0msV8U6Xoy/9rC+UZUFKu1c= 2024-04-12T04:09:23.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TmsI0msV8U6Xoy/9rC+UZUFKu1c=', 'Content-Length': '0'} 2024-04-12T04:09:23.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af0cab0c15335636-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.210 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:09:23.210 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:09:23.211 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.211 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.211 DEBUG:boto:Method: GET 2024-04-12T04:09:23.211 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.211 DEBUG:boto:Data: 2024-04-12T04:09:23.211 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.211 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.211 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.211 DEBUG:boto:Params: {} 2024-04-12T04:09:23.211 DEBUG:boto:Token: None 2024-04-12T04:09:23.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TmsI0msV8U6Xoy/9rC+UZUFKu1c= 2024-04-12T04:09:23.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TmsI0msV8U6Xoy/9rC+UZUFKu1c=', 'Content-Length': '0'} 2024-04-12T04:09:23.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e04a30dacc75a6db-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.214 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:09:23.214 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:09:23.214 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.214 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.214 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.215 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.215 DEBUG:boto:Data: 2024-04-12T04:09:23.215 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.215 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.215 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.215 DEBUG:boto:Params: {} 2024-04-12T04:09:23.215 DEBUG:boto:Token: None 2024-04-12T04:09:23.215 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.215 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YhgHo6zC/DMAe2Gl2D1Aih1esDc= 2024-04-12T04:09:23.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YhgHo6zC/DMAe2Gl2D1Aih1esDc=', 'Content-Length': '0'} 2024-04-12T04:09:23.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0ca6b3d8f6730fd-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.218 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:09:23.218 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:09:23.218 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.218 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.218 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.218 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.218 DEBUG:boto:Data: 2024-04-12T04:09:23.218 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.218 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.218 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.219 DEBUG:boto:Params: {} 2024-04-12T04:09:23.219 DEBUG:boto:Token: None 2024-04-12T04:09:23.219 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.219 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YhgHo6zC/DMAe2Gl2D1Aih1esDc= 2024-04-12T04:09:23.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YhgHo6zC/DMAe2Gl2D1Aih1esDc=', 'Content-Length': '0'} 2024-04-12T04:09:23.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2a4b638ecdfa280-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:09:23.221 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:09:23.221 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:09:23.221 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.221 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.221 DEBUG:boto:Method: GET 2024-04-12T04:09:23.221 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.221 DEBUG:boto:Data: 2024-04-12T04:09:23.222 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.222 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.222 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.222 DEBUG:boto:Params: {} 2024-04-12T04:09:23.222 DEBUG:boto:Token: None 2024-04-12T04:09:23.222 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.222 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TmsI0msV8U6Xoy/9rC+UZUFKu1c= 2024-04-12T04:09:23.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TmsI0msV8U6Xoy/9rC+UZUFKu1c=', 'Content-Length': '0'} 2024-04-12T04:09:23.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c64b3df2d2765b7b-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.224 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:09:23.224 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:09:23.225 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.225 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.225 DEBUG:boto:Method: GET 2024-04-12T04:09:23.225 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.225 DEBUG:boto:Data: 2024-04-12T04:09:23.225 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.225 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.225 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.225 DEBUG:boto:Params: {} 2024-04-12T04:09:23.225 DEBUG:boto:Token: None 2024-04-12T04:09:23.225 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:09:23.226 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TmsI0msV8U6Xoy/9rC+UZUFKu1c= 2024-04-12T04:09:23.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TmsI0msV8U6Xoy/9rC+UZUFKu1c=', 'Content-Length': '0'} 2024-04-12T04:09:23.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070c2c607c3a51238-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.229 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.229 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.229 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:09:23.229 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:09:23.229 DEBUG:boto:Method: GET 2024-04-12T04:09:23.229 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:09:23.229 DEBUG:boto:Data: 2024-04-12T04:09:23.229 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.229 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.230 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.230 DEBUG:boto:Params: {} 2024-04-12T04:09:23.230 DEBUG:boto:Token: None 2024-04-12T04:09:23.230 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.230 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaa91b54dde22868-006618b3f3-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.233 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.233 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.233 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.233 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:09:23.233 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:09:23.234 DEBUG:boto:Method: GET 2024-04-12T04:09:23.234 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:09:23.234 DEBUG:boto:Data: 2024-04-12T04:09:23.234 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.234 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.234 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.234 DEBUG:boto:Params: {} 2024-04-12T04:09:23.234 DEBUG:boto:Token: None 2024-04-12T04:09:23.234 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.235 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb3d82449974caca-006618b3f3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.239 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.239 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:09:23.240 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:09:23.240 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.240 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:09:23.240 DEBUG:boto:Data: 2024-04-12T04:09:23.240 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.240 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.240 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.240 DEBUG:boto:Params: {} 2024-04-12T04:09:23.240 DEBUG:boto:Token: None 2024-04-12T04:09:23.241 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj85 2024-04-12T04:09:23.241 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7929m/i5LhFNz3Tertj2/d1y8rs= 2024-04-12T04:09:23.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7929m/i5LhFNz3Tertj2/d1y8rs=', 'Content-Length': '0'} 2024-04-12T04:09:23.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075e340665da6d24b-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.243 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:09:23.243 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:09:23.243 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.243 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:09:23.243 DEBUG:boto:Data: 2024-04-12T04:09:23.243 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.243 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.243 DEBUG:boto:Params: {} 2024-04-12T04:09:23.244 DEBUG:boto:Token: None 2024-04-12T04:09:23.244 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj85 2024-04-12T04:09:23.244 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7929m/i5LhFNz3Tertj2/d1y8rs= 2024-04-12T04:09:23.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7929m/i5LhFNz3Tertj2/d1y8rs=', 'Content-Length': '0'} 2024-04-12T04:09:23.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5493de481db2c67-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:09:23.246 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:09:23.246 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:09:23.246 DEBUG:boto:Method: GET 2024-04-12T04:09:23.247 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:09:23.247 DEBUG:boto:Data: 2024-04-12T04:09:23.247 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.247 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.247 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.247 DEBUG:boto:Params: {} 2024-04-12T04:09:23.247 DEBUG:boto:Token: None 2024-04-12T04:09:23.247 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj85 2024-04-12T04:09:23.247 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:79MScw0SzT7DAjqD+OlAjPbs1ts= 2024-04-12T04:09:23.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:79MScw0SzT7DAjqD+OlAjPbs1ts=', 'Content-Length': '0'} 2024-04-12T04:09:23.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f516742355f427f-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.250 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:09:23.250 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:09:23.250 DEBUG:boto:Method: GET 2024-04-12T04:09:23.250 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:09:23.250 DEBUG:boto:Data: 2024-04-12T04:09:23.250 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.250 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.250 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.250 DEBUG:boto:Params: {} 2024-04-12T04:09:23.250 DEBUG:boto:Token: None 2024-04-12T04:09:23.251 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj85 2024-04-12T04:09:23.251 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:79MScw0SzT7DAjqD+OlAjPbs1ts= 2024-04-12T04:09:23.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:79MScw0SzT7DAjqD+OlAjPbs1ts=', 'Content-Length': '0'} 2024-04-12T04:09:23.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000749550ee1b94ee9f-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.253 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.253 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.253 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:09:23.253 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:09:23.253 DEBUG:boto:Method: GET 2024-04-12T04:09:23.253 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:09:23.254 DEBUG:boto:Data: 2024-04-12T04:09:23.254 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.254 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.254 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.254 DEBUG:boto:Params: {} 2024-04-12T04:09:23.254 DEBUG:boto:Token: None 2024-04-12T04:09:23.254 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.254 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dae25d811b4bbcf1-006618b3f3-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.257 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.257 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.257 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.258 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:09:23.258 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:09:23.258 DEBUG:boto:Method: GET 2024-04-12T04:09:23.258 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:09:23.258 DEBUG:boto:Data: 2024-04-12T04:09:23.258 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.258 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.258 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.258 DEBUG:boto:Params: {} 2024-04-12T04:09:23.258 DEBUG:boto:Token: None 2024-04-12T04:09:23.259 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.259 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcd5946420afc58f-006618b3f3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.265 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:09:23.266 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:09:23.266 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:09:23.266 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.266 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.266 DEBUG:boto:Method: GET 2024-04-12T04:09:23.266 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.266 DEBUG:boto:Data: 2024-04-12T04:09:23.266 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.266 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.266 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.266 DEBUG:boto:Params: {} 2024-04-12T04:09:23.267 DEBUG:boto:Token: None 2024-04-12T04:09:23.267 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.267 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:huksc/h5Q+5d50rZyZSUBlX0ZtE= 2024-04-12T04:09:23.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:huksc/h5Q+5d50rZyZSUBlX0ZtE=', 'Content-Length': '0'} 2024-04-12T04:09:23.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009def1434fe7563dc-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.269 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:09:23.269 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:09:23.269 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.269 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.270 DEBUG:boto:Method: GET 2024-04-12T04:09:23.270 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.270 DEBUG:boto:Data: 2024-04-12T04:09:23.270 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.270 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.270 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.270 DEBUG:boto:Params: {} 2024-04-12T04:09:23.270 DEBUG:boto:Token: None 2024-04-12T04:09:23.270 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.271 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:huksc/h5Q+5d50rZyZSUBlX0ZtE= 2024-04-12T04:09:23.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:huksc/h5Q+5d50rZyZSUBlX0ZtE=', 'Content-Length': '0'} 2024-04-12T04:09:23.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc24d145b9be1e69-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.314 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:09:23.314 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:09:23.315 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.315 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.315 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.315 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.315 DEBUG:boto:Data: 2024-04-12T04:09:23.315 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.315 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.315 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.315 DEBUG:boto:Params: {} 2024-04-12T04:09:23.315 DEBUG:boto:Token: None 2024-04-12T04:09:23.316 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.316 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PF8Gu+acssaeOkdu7mFQSs/E27g= 2024-04-12T04:09:23.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PF8Gu+acssaeOkdu7mFQSs/E27g=', 'Content-Length': '0'} 2024-04-12T04:09:23.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb56fa67a94a676a-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.318 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:09:23.318 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:09:23.318 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.318 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.318 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.318 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.318 DEBUG:boto:Data: 2024-04-12T04:09:23.319 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.319 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.319 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.319 DEBUG:boto:Params: {} 2024-04-12T04:09:23.319 DEBUG:boto:Token: None 2024-04-12T04:09:23.319 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.319 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PF8Gu+acssaeOkdu7mFQSs/E27g= 2024-04-12T04:09:23.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PF8Gu+acssaeOkdu7mFQSs/E27g=', 'Content-Length': '0'} 2024-04-12T04:09:23.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ae1e7515eadfc78-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:09:23.322 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:09:23.322 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:09:23.322 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.322 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.322 DEBUG:boto:Method: GET 2024-04-12T04:09:23.322 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.322 DEBUG:boto:Data: 2024-04-12T04:09:23.322 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.322 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.322 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.323 DEBUG:boto:Params: {} 2024-04-12T04:09:23.323 DEBUG:boto:Token: None 2024-04-12T04:09:23.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.323 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:huksc/h5Q+5d50rZyZSUBlX0ZtE= 2024-04-12T04:09:23.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:huksc/h5Q+5d50rZyZSUBlX0ZtE=', 'Content-Length': '0'} 2024-04-12T04:09:23.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039c9bc9aefd49460-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.325 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:09:23.325 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:09:23.325 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.325 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.325 DEBUG:boto:Method: GET 2024-04-12T04:09:23.325 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.326 DEBUG:boto:Data: 2024-04-12T04:09:23.326 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.326 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.326 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.326 DEBUG:boto:Params: {} 2024-04-12T04:09:23.326 DEBUG:boto:Token: None 2024-04-12T04:09:23.326 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:09:23.326 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:huksc/h5Q+5d50rZyZSUBlX0ZtE= 2024-04-12T04:09:23.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:huksc/h5Q+5d50rZyZSUBlX0ZtE=', 'Content-Length': '0'} 2024-04-12T04:09:23.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002789330ca2c8d2f6-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.329 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.329 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.329 DEBUG:boto:path=/dkyges-61/?prefix=obj87 2024-04-12T04:09:23.329 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj87 2024-04-12T04:09:23.329 DEBUG:boto:Method: GET 2024-04-12T04:09:23.329 DEBUG:boto:Path: /dkyges-61/?prefix=obj87 2024-04-12T04:09:23.329 DEBUG:boto:Data: 2024-04-12T04:09:23.329 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.329 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.329 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.329 DEBUG:boto:Params: {} 2024-04-12T04:09:23.330 DEBUG:boto:Token: None 2024-04-12T04:09:23.330 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.330 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3d8e0d837c331df-006618b3f3-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.332 DEBUG:boto:b'dkyges-61obj871000falseobj872024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.333 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.333 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.333 DEBUG:boto:path=/dkyges-61/?prefix=obj87 2024-04-12T04:09:23.333 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj87 2024-04-12T04:09:23.333 DEBUG:boto:Method: GET 2024-04-12T04:09:23.333 DEBUG:boto:Path: /dkyges-61/?prefix=obj87 2024-04-12T04:09:23.334 DEBUG:boto:Data: 2024-04-12T04:09:23.334 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.334 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.334 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.334 DEBUG:boto:Params: {} 2024-04-12T04:09:23.334 DEBUG:boto:Token: None 2024-04-12T04:09:23.334 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.334 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccf88d35f01ed658-006618b3f3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.338 DEBUG:boto:b'dkyges-61obj871000falseobj872024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.339 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:09:23.339 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:09:23.339 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.339 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:09:23.339 DEBUG:boto:Data: 2024-04-12T04:09:23.339 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.339 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.339 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.340 DEBUG:boto:Params: {} 2024-04-12T04:09:23.340 DEBUG:boto:Token: None 2024-04-12T04:09:23.340 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj87 2024-04-12T04:09:23.340 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pPq3Vbc/O4OLUmuNML5Iqd89k9Q= 2024-04-12T04:09:23.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pPq3Vbc/O4OLUmuNML5Iqd89k9Q=', 'Content-Length': '0'} 2024-04-12T04:09:23.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040dfe56c531fde5b-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.342 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:09:23.342 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:09:23.343 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.343 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:09:23.343 DEBUG:boto:Data: 2024-04-12T04:09:23.343 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.343 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.343 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.343 DEBUG:boto:Params: {} 2024-04-12T04:09:23.343 DEBUG:boto:Token: None 2024-04-12T04:09:23.343 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj87 2024-04-12T04:09:23.344 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pPq3Vbc/O4OLUmuNML5Iqd89k9Q= 2024-04-12T04:09:23.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pPq3Vbc/O4OLUmuNML5Iqd89k9Q=', 'Content-Length': '0'} 2024-04-12T04:09:23.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e78bfdd591691e7-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:09:23.346 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:09:23.346 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:09:23.346 DEBUG:boto:Method: GET 2024-04-12T04:09:23.346 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:09:23.346 DEBUG:boto:Data: 2024-04-12T04:09:23.346 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.346 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.347 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.347 DEBUG:boto:Params: {} 2024-04-12T04:09:23.347 DEBUG:boto:Token: None 2024-04-12T04:09:23.347 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj87 2024-04-12T04:09:23.347 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VbWbgR/tDwL9E9jBpG6WjXaAxL0= 2024-04-12T04:09:23.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VbWbgR/tDwL9E9jBpG6WjXaAxL0=', 'Content-Length': '0'} 2024-04-12T04:09:23.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c858a5554647e97-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.349 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:09:23.350 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:09:23.350 DEBUG:boto:Method: GET 2024-04-12T04:09:23.350 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:09:23.350 DEBUG:boto:Data: 2024-04-12T04:09:23.350 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.350 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.350 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.350 DEBUG:boto:Params: {} 2024-04-12T04:09:23.350 DEBUG:boto:Token: None 2024-04-12T04:09:23.350 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj87 2024-04-12T04:09:23.351 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VbWbgR/tDwL9E9jBpG6WjXaAxL0= 2024-04-12T04:09:23.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VbWbgR/tDwL9E9jBpG6WjXaAxL0=', 'Content-Length': '0'} 2024-04-12T04:09:23.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088328346b7c95d1f-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.353 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.353 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.353 DEBUG:boto:path=/dkyges-61/?prefix=obj87 2024-04-12T04:09:23.353 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj87 2024-04-12T04:09:23.353 DEBUG:boto:Method: GET 2024-04-12T04:09:23.353 DEBUG:boto:Path: /dkyges-61/?prefix=obj87 2024-04-12T04:09:23.353 DEBUG:boto:Data: 2024-04-12T04:09:23.353 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.353 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.354 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.354 DEBUG:boto:Params: {} 2024-04-12T04:09:23.354 DEBUG:boto:Token: None 2024-04-12T04:09:23.354 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.354 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000317ace0fa4f23507-006618b3f3-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.357 DEBUG:boto:b'dkyges-61obj871000falseobj872024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.357 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.357 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.357 DEBUG:boto:path=/dkyges-61/?prefix=obj87 2024-04-12T04:09:23.357 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj87 2024-04-12T04:09:23.357 DEBUG:boto:Method: GET 2024-04-12T04:09:23.358 DEBUG:boto:Path: /dkyges-61/?prefix=obj87 2024-04-12T04:09:23.358 DEBUG:boto:Data: 2024-04-12T04:09:23.358 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.358 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.358 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.358 DEBUG:boto:Params: {} 2024-04-12T04:09:23.358 DEBUG:boto:Token: None 2024-04-12T04:09:23.358 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.358 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b8feac47cbec4ff-006618b3f3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.362 DEBUG:boto:b'dkyges-61obj871000falseobj872024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:09:23.363 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:09:23.363 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:09:23.363 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.363 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.363 DEBUG:boto:Method: GET 2024-04-12T04:09:23.364 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.364 DEBUG:boto:Data: 2024-04-12T04:09:23.364 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.364 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.364 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.364 DEBUG:boto:Params: {} 2024-04-12T04:09:23.364 DEBUG:boto:Token: None 2024-04-12T04:09:23.364 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.364 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YM2eVuUDjefJuw4D40BM40QctiM= 2024-04-12T04:09:23.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YM2eVuUDjefJuw4D40BM40QctiM=', 'Content-Length': '0'} 2024-04-12T04:09:23.366 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb6afef5cd691713-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.367 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:09:23.367 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:09:23.367 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.367 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.367 DEBUG:boto:Method: GET 2024-04-12T04:09:23.367 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.367 DEBUG:boto:Data: 2024-04-12T04:09:23.367 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.367 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.367 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.368 DEBUG:boto:Params: {} 2024-04-12T04:09:23.368 DEBUG:boto:Token: None 2024-04-12T04:09:23.368 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.368 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YM2eVuUDjefJuw4D40BM40QctiM= 2024-04-12T04:09:23.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YM2eVuUDjefJuw4D40BM40QctiM=', 'Content-Length': '0'} 2024-04-12T04:09:23.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070ae9b2eebe3a469-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.414 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:09:23.415 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:09:23.415 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.415 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.415 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.415 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.415 DEBUG:boto:Data: 2024-04-12T04:09:23.415 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.415 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.415 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.415 DEBUG:boto:Params: {} 2024-04-12T04:09:23.416 DEBUG:boto:Token: None 2024-04-12T04:09:23.416 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.416 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6EN6QMreOq2oindxlIVFju6QH1w= 2024-04-12T04:09:23.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6EN6QMreOq2oindxlIVFju6QH1w=', 'Content-Length': '0'} 2024-04-12T04:09:23.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000744b37239af6da4a-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.418 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:09:23.419 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:09:23.419 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.419 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.419 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.419 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.419 DEBUG:boto:Data: 2024-04-12T04:09:23.419 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.420 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.420 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.420 DEBUG:boto:Params: {} 2024-04-12T04:09:23.420 DEBUG:boto:Token: None 2024-04-12T04:09:23.420 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.420 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6EN6QMreOq2oindxlIVFju6QH1w= 2024-04-12T04:09:23.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6EN6QMreOq2oindxlIVFju6QH1w=', 'Content-Length': '0'} 2024-04-12T04:09:23.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015eb06c39a665724-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:09:23.423 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:09:23.423 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:09:23.423 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.423 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.423 DEBUG:boto:Method: GET 2024-04-12T04:09:23.423 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.423 DEBUG:boto:Data: 2024-04-12T04:09:23.424 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.424 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.424 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.424 DEBUG:boto:Params: {} 2024-04-12T04:09:23.424 DEBUG:boto:Token: None 2024-04-12T04:09:23.424 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.424 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YM2eVuUDjefJuw4D40BM40QctiM= 2024-04-12T04:09:23.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YM2eVuUDjefJuw4D40BM40QctiM=', 'Content-Length': '0'} 2024-04-12T04:09:23.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac6080023568c494-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.427 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:09:23.427 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:09:23.427 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.427 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.427 DEBUG:boto:Method: GET 2024-04-12T04:09:23.427 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.428 DEBUG:boto:Data: 2024-04-12T04:09:23.428 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.428 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.428 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.428 DEBUG:boto:Params: {} 2024-04-12T04:09:23.428 DEBUG:boto:Token: None 2024-04-12T04:09:23.428 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:09:23.428 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YM2eVuUDjefJuw4D40BM40QctiM= 2024-04-12T04:09:23.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YM2eVuUDjefJuw4D40BM40QctiM=', 'Content-Length': '0'} 2024-04-12T04:09:23.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010ef8be9f9d782a9-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.431 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.431 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.431 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:09:23.431 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:09:23.431 DEBUG:boto:Method: GET 2024-04-12T04:09:23.431 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:09:23.431 DEBUG:boto:Data: 2024-04-12T04:09:23.432 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.432 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.432 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.432 DEBUG:boto:Params: {} 2024-04-12T04:09:23.432 DEBUG:boto:Token: None 2024-04-12T04:09:23.432 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.432 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.435 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e70125a65c9c9311-006618b3f3-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.478 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.479 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.479 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.479 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:09:23.479 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:09:23.479 DEBUG:boto:Method: GET 2024-04-12T04:09:23.479 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:09:23.480 DEBUG:boto:Data: 2024-04-12T04:09:23.480 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.480 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.480 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.480 DEBUG:boto:Params: {} 2024-04-12T04:09:23.480 DEBUG:boto:Token: None 2024-04-12T04:09:23.480 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.480 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a11de088dcae11f-006618b3f3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.485 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.485 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:09:23.485 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:09:23.485 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.485 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:09:23.485 DEBUG:boto:Data: 2024-04-12T04:09:23.486 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.486 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.486 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.486 DEBUG:boto:Params: {} 2024-04-12T04:09:23.486 DEBUG:boto:Token: None 2024-04-12T04:09:23.486 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj92 2024-04-12T04:09:23.486 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e+Mhk5ZXJfjkr/uRt+GSYj7V/NM= 2024-04-12T04:09:23.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e+Mhk5ZXJfjkr/uRt+GSYj7V/NM=', 'Content-Length': '0'} 2024-04-12T04:09:23.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000724d39400ea009fc-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.488 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:09:23.489 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:09:23.489 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.489 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:09:23.489 DEBUG:boto:Data: 2024-04-12T04:09:23.489 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.489 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.489 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.489 DEBUG:boto:Params: {} 2024-04-12T04:09:23.489 DEBUG:boto:Token: None 2024-04-12T04:09:23.490 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj92 2024-04-12T04:09:23.490 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e+Mhk5ZXJfjkr/uRt+GSYj7V/NM= 2024-04-12T04:09:23.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e+Mhk5ZXJfjkr/uRt+GSYj7V/NM=', 'Content-Length': '0'} 2024-04-12T04:09:23.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008de9819c82d4e9d8-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:09:23.492 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:09:23.492 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:09:23.493 DEBUG:boto:Method: GET 2024-04-12T04:09:23.493 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:09:23.493 DEBUG:boto:Data: 2024-04-12T04:09:23.493 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.493 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.493 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.493 DEBUG:boto:Params: {} 2024-04-12T04:09:23.493 DEBUG:boto:Token: None 2024-04-12T04:09:23.494 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj92 2024-04-12T04:09:23.494 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R7AkQPMHGLa48LKo9Qz/JVGchDM= 2024-04-12T04:09:23.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R7AkQPMHGLa48LKo9Qz/JVGchDM=', 'Content-Length': '0'} 2024-04-12T04:09:23.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a21fbf211bfdd606-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.496 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:09:23.496 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:09:23.496 DEBUG:boto:Method: GET 2024-04-12T04:09:23.496 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:09:23.497 DEBUG:boto:Data: 2024-04-12T04:09:23.497 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.497 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.497 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.497 DEBUG:boto:Params: {} 2024-04-12T04:09:23.497 DEBUG:boto:Token: None 2024-04-12T04:09:23.497 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj92 2024-04-12T04:09:23.497 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R7AkQPMHGLa48LKo9Qz/JVGchDM= 2024-04-12T04:09:23.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R7AkQPMHGLa48LKo9Qz/JVGchDM=', 'Content-Length': '0'} 2024-04-12T04:09:23.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000008580cb777f5ddc-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.542 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.542 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.543 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:09:23.543 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:09:23.543 DEBUG:boto:Method: GET 2024-04-12T04:09:23.543 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:09:23.543 DEBUG:boto:Data: 2024-04-12T04:09:23.543 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.543 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.543 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.543 DEBUG:boto:Params: {} 2024-04-12T04:09:23.544 DEBUG:boto:Token: None 2024-04-12T04:09:23.544 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.544 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005c1e2b7610f7573-006618b3f3-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.547 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.547 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.547 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.547 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:09:23.548 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:09:23.548 DEBUG:boto:Method: GET 2024-04-12T04:09:23.548 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:09:23.548 DEBUG:boto:Data: 2024-04-12T04:09:23.548 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.548 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.548 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.548 DEBUG:boto:Params: {} 2024-04-12T04:09:23.548 DEBUG:boto:Token: None 2024-04-12T04:09:23.549 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.549 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd702621c5f18d65-006618b3f3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.553 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T04:09:23.554 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:09:23.554 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:09:23.554 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.554 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.554 DEBUG:boto:Method: GET 2024-04-12T04:09:23.554 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.554 DEBUG:boto:Data: 2024-04-12T04:09:23.554 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.554 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.554 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.555 DEBUG:boto:Params: {} 2024-04-12T04:09:23.555 DEBUG:boto:Token: None 2024-04-12T04:09:23.555 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.555 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jSRSR61QAxWyQp7Lw69bSOIUDPE= 2024-04-12T04:09:23.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jSRSR61QAxWyQp7Lw69bSOIUDPE=', 'Content-Length': '0'} 2024-04-12T04:09:23.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cdd2ab984cdb290-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.558 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:09:23.558 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:09:23.558 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.558 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.558 DEBUG:boto:Method: GET 2024-04-12T04:09:23.559 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.559 DEBUG:boto:Data: 2024-04-12T04:09:23.559 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.559 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.559 DEBUG:boto:Params: {} 2024-04-12T04:09:23.559 DEBUG:boto:Token: None 2024-04-12T04:09:23.559 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.559 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jSRSR61QAxWyQp7Lw69bSOIUDPE= 2024-04-12T04:09:23.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jSRSR61QAxWyQp7Lw69bSOIUDPE=', 'Content-Length': '0'} 2024-04-12T04:09:23.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062b9a78d8a6984e9-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.562 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:09:23.562 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:09:23.562 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.562 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.562 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.563 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.563 DEBUG:boto:Data: 2024-04-12T04:09:23.563 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.563 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.563 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.563 DEBUG:boto:Params: {} 2024-04-12T04:09:23.563 DEBUG:boto:Token: None 2024-04-12T04:09:23.563 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.563 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QFSTwuZSYOK9KAcpXE0yern9PXo= 2024-04-12T04:09:23.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QFSTwuZSYOK9KAcpXE0yern9PXo=', 'Content-Length': '0'} 2024-04-12T04:09:23.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb540c5d28a238de-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.565 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:09:23.565 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:09:23.565 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.566 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.566 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.566 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.566 DEBUG:boto:Data: 2024-04-12T04:09:23.566 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.566 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.566 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.566 DEBUG:boto:Params: {} 2024-04-12T04:09:23.566 DEBUG:boto:Token: None 2024-04-12T04:09:23.566 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.567 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QFSTwuZSYOK9KAcpXE0yern9PXo= 2024-04-12T04:09:23.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QFSTwuZSYOK9KAcpXE0yern9PXo=', 'Content-Length': '0'} 2024-04-12T04:09:23.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000279f619e8306ff41-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T04:09:23.569 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:09:23.569 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:09:23.569 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.569 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.569 DEBUG:boto:Method: GET 2024-04-12T04:09:23.569 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.569 DEBUG:boto:Data: 2024-04-12T04:09:23.569 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.569 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.569 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.569 DEBUG:boto:Params: {} 2024-04-12T04:09:23.570 DEBUG:boto:Token: None 2024-04-12T04:09:23.570 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.570 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jSRSR61QAxWyQp7Lw69bSOIUDPE= 2024-04-12T04:09:23.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jSRSR61QAxWyQp7Lw69bSOIUDPE=', 'Content-Length': '0'} 2024-04-12T04:09:23.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc292a1d74761338-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.572 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:09:23.572 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:09:23.572 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.573 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.573 DEBUG:boto:Method: GET 2024-04-12T04:09:23.573 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.573 DEBUG:boto:Data: 2024-04-12T04:09:23.573 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.573 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.573 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.573 DEBUG:boto:Params: {} 2024-04-12T04:09:23.573 DEBUG:boto:Token: None 2024-04-12T04:09:23.573 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:09:23.574 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jSRSR61QAxWyQp7Lw69bSOIUDPE= 2024-04-12T04:09:23.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jSRSR61QAxWyQp7Lw69bSOIUDPE=', 'Content-Length': '0'} 2024-04-12T04:09:23.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004245f526f4acdb33-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.576 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.576 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.576 DEBUG:boto:path=/dkyges-61/?prefix=obj93 2024-04-12T04:09:23.576 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj93 2024-04-12T04:09:23.576 DEBUG:boto:Method: GET 2024-04-12T04:09:23.576 DEBUG:boto:Path: /dkyges-61/?prefix=obj93 2024-04-12T04:09:23.576 DEBUG:boto:Data: 2024-04-12T04:09:23.576 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.576 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.576 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.577 DEBUG:boto:Params: {} 2024-04-12T04:09:23.577 DEBUG:boto:Token: None 2024-04-12T04:09:23.577 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.577 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e9029231aba765b-006618b3f3-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.580 DEBUG:boto:b'dkyges-61obj931000falseobj932024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.580 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.580 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.580 DEBUG:boto:path=/dkyges-61/?prefix=obj93 2024-04-12T04:09:23.580 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj93 2024-04-12T04:09:23.580 DEBUG:boto:Method: GET 2024-04-12T04:09:23.580 DEBUG:boto:Path: /dkyges-61/?prefix=obj93 2024-04-12T04:09:23.580 DEBUG:boto:Data: 2024-04-12T04:09:23.581 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.581 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.581 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.581 DEBUG:boto:Params: {} 2024-04-12T04:09:23.581 DEBUG:boto:Token: None 2024-04-12T04:09:23.581 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.581 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b0a3828ebb39103-006618b3f3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.626 DEBUG:boto:b'dkyges-61obj931000falseobj932024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.627 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:09:23.627 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:09:23.627 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.627 DEBUG:boto:Path: /dkyges-61/obj93 2024-04-12T04:09:23.627 DEBUG:boto:Data: 2024-04-12T04:09:23.627 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.627 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.628 DEBUG:boto:Params: {} 2024-04-12T04:09:23.628 DEBUG:boto:Token: None 2024-04-12T04:09:23.628 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj93 2024-04-12T04:09:23.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xRaxzpkhfFT1l82oX0nO1cvlG+0= 2024-04-12T04:09:23.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xRaxzpkhfFT1l82oX0nO1cvlG+0=', 'Content-Length': '0'} 2024-04-12T04:09:23.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a407b4665b8d0cec-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.630 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:09:23.630 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:09:23.631 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.631 DEBUG:boto:Path: /dkyges-61/obj93 2024-04-12T04:09:23.631 DEBUG:boto:Data: 2024-04-12T04:09:23.631 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.631 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.631 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.631 DEBUG:boto:Params: {} 2024-04-12T04:09:23.631 DEBUG:boto:Token: None 2024-04-12T04:09:23.631 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj93 2024-04-12T04:09:23.631 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xRaxzpkhfFT1l82oX0nO1cvlG+0= 2024-04-12T04:09:23.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xRaxzpkhfFT1l82oX0nO1cvlG+0=', 'Content-Length': '0'} 2024-04-12T04:09:23.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb2a33ff12d24810-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T04:09:23.634 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:09:23.634 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:09:23.634 DEBUG:boto:Method: GET 2024-04-12T04:09:23.634 DEBUG:boto:Path: /dkyges-61/obj93 2024-04-12T04:09:23.634 DEBUG:boto:Data: 2024-04-12T04:09:23.634 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.634 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.634 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.634 DEBUG:boto:Params: {} 2024-04-12T04:09:23.635 DEBUG:boto:Token: None 2024-04-12T04:09:23.635 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj93 2024-04-12T04:09:23.635 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZZhGoM2gX64NgCOG8rOdObeLLEw= 2024-04-12T04:09:23.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZZhGoM2gX64NgCOG8rOdObeLLEw=', 'Content-Length': '0'} 2024-04-12T04:09:23.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000996957924045b0c0-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.637 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:09:23.637 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:09:23.637 DEBUG:boto:Method: GET 2024-04-12T04:09:23.638 DEBUG:boto:Path: /dkyges-61/obj93 2024-04-12T04:09:23.638 DEBUG:boto:Data: 2024-04-12T04:09:23.638 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.638 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.638 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.638 DEBUG:boto:Params: {} 2024-04-12T04:09:23.638 DEBUG:boto:Token: None 2024-04-12T04:09:23.638 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj93 2024-04-12T04:09:23.638 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZZhGoM2gX64NgCOG8rOdObeLLEw= 2024-04-12T04:09:23.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZZhGoM2gX64NgCOG8rOdObeLLEw=', 'Content-Length': '0'} 2024-04-12T04:09:23.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002206d65c55abb0ca-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.640 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.641 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.641 DEBUG:boto:path=/dkyges-61/?prefix=obj93 2024-04-12T04:09:23.641 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj93 2024-04-12T04:09:23.641 DEBUG:boto:Method: GET 2024-04-12T04:09:23.641 DEBUG:boto:Path: /dkyges-61/?prefix=obj93 2024-04-12T04:09:23.641 DEBUG:boto:Data: 2024-04-12T04:09:23.641 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.641 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.641 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.641 DEBUG:boto:Params: {} 2024-04-12T04:09:23.641 DEBUG:boto:Token: None 2024-04-12T04:09:23.642 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.642 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000814c62757678a4fc-006618b3f3-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.644 DEBUG:boto:b'dkyges-61obj931000falseobj932024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.645 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.645 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.645 DEBUG:boto:path=/dkyges-61/?prefix=obj93 2024-04-12T04:09:23.645 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj93 2024-04-12T04:09:23.645 DEBUG:boto:Method: GET 2024-04-12T04:09:23.645 DEBUG:boto:Path: /dkyges-61/?prefix=obj93 2024-04-12T04:09:23.645 DEBUG:boto:Data: 2024-04-12T04:09:23.645 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.645 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.646 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.646 DEBUG:boto:Params: {} 2024-04-12T04:09:23.646 DEBUG:boto:Token: None 2024-04-12T04:09:23.646 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.646 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.650 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bdd5400cff72985-006618b3f3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.650 DEBUG:boto:b'dkyges-61obj931000falseobj932024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:09:23.651 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:09:23.651 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:09:23.651 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.651 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.651 DEBUG:boto:Method: GET 2024-04-12T04:09:23.652 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.652 DEBUG:boto:Data: 2024-04-12T04:09:23.652 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.652 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.652 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.652 DEBUG:boto:Params: {} 2024-04-12T04:09:23.652 DEBUG:boto:Token: None 2024-04-12T04:09:23.652 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.652 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RukBq+olvnP3/agcGJnWkOrfFNo= 2024-04-12T04:09:23.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RukBq+olvnP3/agcGJnWkOrfFNo=', 'Content-Length': '0'} 2024-04-12T04:09:23.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081579ddc391d85ce-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.655 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:09:23.655 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:09:23.655 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.655 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.655 DEBUG:boto:Method: GET 2024-04-12T04:09:23.655 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.655 DEBUG:boto:Data: 2024-04-12T04:09:23.655 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.655 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.655 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.656 DEBUG:boto:Params: {} 2024-04-12T04:09:23.656 DEBUG:boto:Token: None 2024-04-12T04:09:23.656 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RukBq+olvnP3/agcGJnWkOrfFNo= 2024-04-12T04:09:23.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RukBq+olvnP3/agcGJnWkOrfFNo=', 'Content-Length': '0'} 2024-04-12T04:09:23.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddb392ce7a0b8f16-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.658 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:09:23.658 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:09:23.658 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.659 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.659 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.659 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.659 DEBUG:boto:Data: 2024-04-12T04:09:23.659 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.659 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.659 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.659 DEBUG:boto:Params: {} 2024-04-12T04:09:23.660 DEBUG:boto:Token: None 2024-04-12T04:09:23.660 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.660 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5jxw2JUsYzjkSXXF4LLVQ5+ykfQ= 2024-04-12T04:09:23.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5jxw2JUsYzjkSXXF4LLVQ5+ykfQ=', 'Content-Length': '0'} 2024-04-12T04:09:23.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdb8529929904342-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.662 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:09:23.663 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:09:23.663 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.663 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.663 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.663 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.663 DEBUG:boto:Data: 2024-04-12T04:09:23.663 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.663 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.663 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.663 DEBUG:boto:Params: {} 2024-04-12T04:09:23.664 DEBUG:boto:Token: None 2024-04-12T04:09:23.664 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.664 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5jxw2JUsYzjkSXXF4LLVQ5+ykfQ= 2024-04-12T04:09:23.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5jxw2JUsYzjkSXXF4LLVQ5+ykfQ=', 'Content-Length': '0'} 2024-04-12T04:09:23.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac6702f40c4c748f-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:09:23.666 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:09:23.666 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:09:23.666 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.666 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.666 DEBUG:boto:Method: GET 2024-04-12T04:09:23.667 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.667 DEBUG:boto:Data: 2024-04-12T04:09:23.667 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.667 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.667 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.667 DEBUG:boto:Params: {} 2024-04-12T04:09:23.667 DEBUG:boto:Token: None 2024-04-12T04:09:23.667 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.667 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RukBq+olvnP3/agcGJnWkOrfFNo= 2024-04-12T04:09:23.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RukBq+olvnP3/agcGJnWkOrfFNo=', 'Content-Length': '0'} 2024-04-12T04:09:23.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c9e6334b28ee674-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.710 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:09:23.711 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:09:23.711 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.711 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.711 DEBUG:boto:Method: GET 2024-04-12T04:09:23.711 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.711 DEBUG:boto:Data: 2024-04-12T04:09:23.711 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.711 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.711 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.711 DEBUG:boto:Params: {} 2024-04-12T04:09:23.712 DEBUG:boto:Token: None 2024-04-12T04:09:23.712 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:09:23.712 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RukBq+olvnP3/agcGJnWkOrfFNo= 2024-04-12T04:09:23.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RukBq+olvnP3/agcGJnWkOrfFNo=', 'Content-Length': '0'} 2024-04-12T04:09:23.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077768b88b1ad3362-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.714 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.714 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.715 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:09:23.715 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:09:23.715 DEBUG:boto:Method: GET 2024-04-12T04:09:23.715 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:09:23.715 DEBUG:boto:Data: 2024-04-12T04:09:23.715 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.715 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.715 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.715 DEBUG:boto:Params: {} 2024-04-12T04:09:23.715 DEBUG:boto:Token: None 2024-04-12T04:09:23.716 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.716 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b3b129139954725-006618b3f3-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.718 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.719 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.719 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.719 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:09:23.719 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:09:23.719 DEBUG:boto:Method: GET 2024-04-12T04:09:23.719 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:09:23.719 DEBUG:boto:Data: 2024-04-12T04:09:23.719 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.719 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.719 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.719 DEBUG:boto:Params: {} 2024-04-12T04:09:23.720 DEBUG:boto:Token: None 2024-04-12T04:09:23.720 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.720 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9aa0e0130559c91-006618b3f3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.724 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.724 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:09:23.724 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:09:23.724 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.725 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:09:23.725 DEBUG:boto:Data: 2024-04-12T04:09:23.725 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.725 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.725 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.725 DEBUG:boto:Params: {} 2024-04-12T04:09:23.725 DEBUG:boto:Token: None 2024-04-12T04:09:23.725 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj95 2024-04-12T04:09:23.725 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U96CyBpDwgssDQwB0gA04CmjPCo= 2024-04-12T04:09:23.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U96CyBpDwgssDQwB0gA04CmjPCo=', 'Content-Length': '0'} 2024-04-12T04:09:23.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000178669a514ea5279-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.727 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:09:23.728 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:09:23.728 DEBUG:boto:Method: HEAD 2024-04-12T04:09:23.728 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:09:23.728 DEBUG:boto:Data: 2024-04-12T04:09:23.728 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.728 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.728 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.728 DEBUG:boto:Params: {} 2024-04-12T04:09:23.728 DEBUG:boto:Token: None 2024-04-12T04:09:23.728 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj95 2024-04-12T04:09:23.729 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U96CyBpDwgssDQwB0gA04CmjPCo= 2024-04-12T04:09:23.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U96CyBpDwgssDQwB0gA04CmjPCo=', 'Content-Length': '0'} 2024-04-12T04:09:23.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eb810d071d1ebb4-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:09:23.731 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:09:23.731 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:09:23.731 DEBUG:boto:Method: GET 2024-04-12T04:09:23.731 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:09:23.731 DEBUG:boto:Data: 2024-04-12T04:09:23.731 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.731 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.731 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.731 DEBUG:boto:Params: {} 2024-04-12T04:09:23.732 DEBUG:boto:Token: None 2024-04-12T04:09:23.732 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj95 2024-04-12T04:09:23.732 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eHH9vkTQTfR7ctIYPXpU/5lVuck= 2024-04-12T04:09:23.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eHH9vkTQTfR7ctIYPXpU/5lVuck=', 'Content-Length': '0'} 2024-04-12T04:09:23.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007cd17b9b46d11c5-006618b3f3-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.734 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:09:23.734 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:09:23.734 DEBUG:boto:Method: GET 2024-04-12T04:09:23.734 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:09:23.734 DEBUG:boto:Data: 2024-04-12T04:09:23.734 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.734 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.735 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.735 DEBUG:boto:Params: {} 2024-04-12T04:09:23.735 DEBUG:boto:Token: None 2024-04-12T04:09:23.735 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/obj95 2024-04-12T04:09:23.735 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eHH9vkTQTfR7ctIYPXpU/5lVuck= 2024-04-12T04:09:23.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eHH9vkTQTfR7ctIYPXpU/5lVuck=', 'Content-Length': '0'} 2024-04-12T04:09:23.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071e6fc9ea00bde64-006618b3f3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.782 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.783 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.783 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:09:23.783 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:09:23.783 DEBUG:boto:Method: GET 2024-04-12T04:09:23.783 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:09:23.784 DEBUG:boto:Data: 2024-04-12T04:09:23.784 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.784 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:09:23.784 DEBUG:boto:Port: 8001 2024-04-12T04:09:23.784 DEBUG:boto:Params: {} 2024-04-12T04:09:23.784 DEBUG:boto:Token: None 2024-04-12T04:09:23.785 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.785 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051376be7563e0152-006618b3f3-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.788 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.788 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:23.789 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:23.789 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:09:23.789 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:09:23.789 DEBUG:boto:Method: GET 2024-04-12T04:09:23.789 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:09:23.789 DEBUG:boto:Data: 2024-04-12T04:09:23.789 DEBUG:boto:Headers: {} 2024-04-12T04:09:23.789 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:23.790 DEBUG:boto:Port: 8000 2024-04-12T04:09:23.790 DEBUG:boto:Params: {} 2024-04-12T04:09:23.790 DEBUG:boto:Token: None 2024-04-12T04:09:23.790 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:09:23 GMT /dkyges-61/ 2024-04-12T04:09:23.790 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8= 2024-04-12T04:09:23.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:23 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RglIXSJ6NO4ZsDyyaM+emEAqPB8=', 'Content-Length': '0'} 2024-04-12T04:09:23.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c059ec191dd26004-006618b3f3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:09:23 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:23.795 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:09:23.795 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-61 zones={a2, a1} 2024-04-12T04:09:23.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:09:23.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:09:23.796 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:09:23.859 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:09:23.860 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:09:23.920 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:09:23.920 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-61 2024-04-12T04:09:23.934 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 30 2024-04-12T04:09:24.008 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:24.008 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:24.008 DEBUG:boto:Method: HEAD 2024-04-12T04:09:24.008 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:09:24.008 DEBUG:boto:Data: 2024-04-12T04:09:24.008 DEBUG:boto:Headers: {} 2024-04-12T04:09:24.008 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:24.008 DEBUG:boto:Port: 8000 2024-04-12T04:09:24.009 DEBUG:boto:Params: {} 2024-04-12T04:09:24.009 DEBUG:boto:Token: None 2024-04-12T04:09:24.009 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:24 GMT /dkyges-61/ 2024-04-12T04:09:24.009 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mLccCJ0xGs0ORxBDllMvv35BSF8= 2024-04-12T04:09:24.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:24 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mLccCJ0xGs0ORxBDllMvv35BSF8=', 'Content-Length': '0'} 2024-04-12T04:09:24.013 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000975ce6904ad8fe12-006618b3f4-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:24.014 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:09:24.014 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:09:24.014 DEBUG:boto:Method: PUT 2024-04-12T04:09:24.014 DEBUG:boto:Path: /dkyges-61/obj37 2024-04-12T04:09:24.014 DEBUG:boto:Data: 2024-04-12T04:09:24.014 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:09:24.014 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:24.015 DEBUG:boto:Port: 8000 2024-04-12T04:09:24.015 DEBUG:boto:Params: {} 2024-04-12T04:09:24.015 DEBUG:boto:Token: None 2024-04-12T04:09:24.015 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:09:24 GMT /dkyges-61/obj37 2024-04-12T04:09:24.015 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JTnAS6JIdJf47AM9ULAMs8h42c8= 2024-04-12T04:09:24.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:09:24 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JTnAS6JIdJf47AM9ULAMs8h42c8='} 2024-04-12T04:09:24.021 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001cc812c8b372a846-006618b3f4-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:24.021 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:24.021 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:24.021 DEBUG:boto:Method: HEAD 2024-04-12T04:09:24.021 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:09:24.022 DEBUG:boto:Data: 2024-04-12T04:09:24.022 DEBUG:boto:Headers: {} 2024-04-12T04:09:24.022 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:24.022 DEBUG:boto:Port: 8000 2024-04-12T04:09:24.022 DEBUG:boto:Params: {} 2024-04-12T04:09:24.022 DEBUG:boto:Token: None 2024-04-12T04:09:24.022 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:24 GMT /dkyges-61/ 2024-04-12T04:09:24.022 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mLccCJ0xGs0ORxBDllMvv35BSF8= 2024-04-12T04:09:24.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:24 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mLccCJ0xGs0ORxBDllMvv35BSF8=', 'Content-Length': '0'} 2024-04-12T04:09:24.026 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004d64d6b0f90688a-006618b3f4-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:24.027 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:09:24.027 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:09:24.027 DEBUG:boto:Method: PUT 2024-04-12T04:09:24.027 DEBUG:boto:Path: /dkyges-61/obj74 2024-04-12T04:09:24.027 DEBUG:boto:Data: 2024-04-12T04:09:24.027 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:09:24.027 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:24.028 DEBUG:boto:Port: 8000 2024-04-12T04:09:24.028 DEBUG:boto:Params: {} 2024-04-12T04:09:24.028 DEBUG:boto:Token: None 2024-04-12T04:09:24.028 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:09:24 GMT /dkyges-61/obj74 2024-04-12T04:09:24.028 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t7h6SgatLdeRcCUoumcYz3sdNYg= 2024-04-12T04:09:24.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:09:24 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t7h6SgatLdeRcCUoumcYz3sdNYg='} 2024-04-12T04:09:24.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bfca31206798a6cf-006618b3f4-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:24.075 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:24.075 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:24.075 DEBUG:boto:Method: HEAD 2024-04-12T04:09:24.075 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:09:24.075 DEBUG:boto:Data: 2024-04-12T04:09:24.075 DEBUG:boto:Headers: {} 2024-04-12T04:09:24.075 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:24.075 DEBUG:boto:Port: 8000 2024-04-12T04:09:24.075 DEBUG:boto:Params: {} 2024-04-12T04:09:24.076 DEBUG:boto:Token: None 2024-04-12T04:09:24.076 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:24 GMT /dkyges-61/ 2024-04-12T04:09:24.076 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mLccCJ0xGs0ORxBDllMvv35BSF8= 2024-04-12T04:09:24.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:24 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mLccCJ0xGs0ORxBDllMvv35BSF8=', 'Content-Length': '0'} 2024-04-12T04:09:24.080 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000220530b47d6d17de-006618b3f4-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:24.080 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:09:24.080 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:09:24.080 DEBUG:boto:Method: PUT 2024-04-12T04:09:24.080 DEBUG:boto:Path: /dkyges-61/obj111 2024-04-12T04:09:24.081 DEBUG:boto:Data: 2024-04-12T04:09:24.081 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:09:24.081 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:24.081 DEBUG:boto:Port: 8000 2024-04-12T04:09:24.081 DEBUG:boto:Params: {} 2024-04-12T04:09:24.081 DEBUG:boto:Token: None 2024-04-12T04:09:24.081 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:09:24 GMT /dkyges-61/obj111 2024-04-12T04:09:24.081 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2glJH0WkydD2PqcE5zuWaJCP5Xg= 2024-04-12T04:09:24.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:09:24 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2glJH0WkydD2PqcE5zuWaJCP5Xg='} 2024-04-12T04:09:24.088 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e24831f05a77696-006618b3f4-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:24.088 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:24.088 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:24.088 DEBUG:boto:Method: HEAD 2024-04-12T04:09:24.088 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:09:24.088 DEBUG:boto:Data: 2024-04-12T04:09:24.088 DEBUG:boto:Headers: {} 2024-04-12T04:09:24.089 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:24.089 DEBUG:boto:Port: 8000 2024-04-12T04:09:24.089 DEBUG:boto:Params: {} 2024-04-12T04:09:24.089 DEBUG:boto:Token: None 2024-04-12T04:09:24.089 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:24 GMT /dkyges-61/ 2024-04-12T04:09:24.089 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mLccCJ0xGs0ORxBDllMvv35BSF8= 2024-04-12T04:09:24.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:24 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mLccCJ0xGs0ORxBDllMvv35BSF8=', 'Content-Length': '0'} 2024-04-12T04:09:24.093 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006465d9af23d728cf-006618b3f4-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:24.093 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:09:24.094 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:09:24.094 DEBUG:boto:Method: PUT 2024-04-12T04:09:24.094 DEBUG:boto:Path: /dkyges-61/obj148 2024-04-12T04:09:24.094 DEBUG:boto:Data: 2024-04-12T04:09:24.094 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:09:24.094 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:24.094 DEBUG:boto:Port: 8000 2024-04-12T04:09:24.095 DEBUG:boto:Params: {} 2024-04-12T04:09:24.095 DEBUG:boto:Token: None 2024-04-12T04:09:24.095 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:09:24 GMT /dkyges-61/obj148 2024-04-12T04:09:24.095 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Fx78Bh6ozRaDYyZi3xSRs+X2PG0= 2024-04-12T04:09:24.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:09:24 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Fx78Bh6ozRaDYyZi3xSRs+X2PG0='} 2024-04-12T04:09:24.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000261554dbec97c5f4-006618b3f4-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:24.143 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:09:24.143 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:09:24.143 DEBUG:boto:Method: HEAD 2024-04-12T04:09:24.143 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:09:24.143 DEBUG:boto:Data: 2024-04-12T04:09:24.143 DEBUG:boto:Headers: {} 2024-04-12T04:09:24.143 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:24.143 DEBUG:boto:Port: 8000 2024-04-12T04:09:24.143 DEBUG:boto:Params: {} 2024-04-12T04:09:24.144 DEBUG:boto:Token: None 2024-04-12T04:09:24.144 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:09:24 GMT /dkyges-61/ 2024-04-12T04:09:24.144 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mLccCJ0xGs0ORxBDllMvv35BSF8= 2024-04-12T04:09:24.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:09:24 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mLccCJ0xGs0ORxBDllMvv35BSF8=', 'Content-Length': '0'} 2024-04-12T04:09:24.148 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3fadbfe651998de-006618b3f4-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:24.149 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:09:24.149 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:09:24.149 DEBUG:boto:Method: PUT 2024-04-12T04:09:24.149 DEBUG:boto:Path: /dkyges-61/obj185 2024-04-12T04:09:24.149 DEBUG:boto:Data: 2024-04-12T04:09:24.149 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:09:24.149 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:09:24.149 DEBUG:boto:Port: 8000 2024-04-12T04:09:24.149 DEBUG:boto:Params: {} 2024-04-12T04:09:24.150 DEBUG:boto:Token: None 2024-04-12T04:09:24.150 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:09:24 GMT /dkyges-61/obj185 2024-04-12T04:09:24.150 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VyqfsIv37Omw0urT377ZqAL0kho= 2024-04-12T04:09:24.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:09:24 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VyqfsIv37Omw0urT377ZqAL0kho='} 2024-04-12T04:09:24.157 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a279241d7481fb39-006618b3f4-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:09:24.157 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-61 2024-04-12T04:09:24.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:09:24.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:09:24.158 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:09:24.225 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:09:24.225 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:09:24.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:09:24.284+0000 7f9077d23780 1 waiting to reach latest gen 6, on 5.. 2024-04-12T04:09:29.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:09:29.284+0000 7f9077d23780 1 waiting to reach latest gen 6, on 5.. 2024-04-12T04:09:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:09:34.285+0000 7f9077d23780 1 waiting to reach latest gen 6, on 5.. 2024-04-12T04:09:39.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:09:39.285+0000 7f9077d23780 1 waiting to reach latest gen 6, on 5.. 2024-04-12T04:09:44.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:09:44.289+0000 7f9077d23780 1 waiting for incremental sync to catch up: 2024-04-12T04:09:44.273 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , 00000000002.170.6, , , , , , , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:09:44.273 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.140.6, , , , , , 00000000002.170.6, , , , , 00000000002.110.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:09:49.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:09:49.291+0000 7f9077d23780 1 waiting for incremental sync to catch up: 2024-04-12T04:09:49.276 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , 00000000002.170.6, , , , , , , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:09:49.276 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.140.6, , , , , , 00000000002.170.6, , , , , 00000000002.110.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:09:54.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:09:54.294+0000 7f9077d23780 1 waiting for incremental sync to catch up: 2024-04-12T04:09:54.278 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , 00000000002.170.6, , , , , , , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:09:54.278 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.140.6, , , , , , 00000000002.170.6, , , , , 00000000002.110.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:09:59.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:09:59.297+0000 7f9077d23780 1 waiting for incremental sync to catch up: 2024-04-12T04:09:59.281 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , 00000000002.170.6, , , , , , , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:09:59.281 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.140.6, , , , , , 00000000002.170.6, , , , , 00000000002.110.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:10:04.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:10:04.300+0000 7f9077d23780 1 waiting for incremental sync to catch up: 2024-04-12T04:10:04.284 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , 00000000002.170.6, , , , , , , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:10:04.284 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.140.6, , , , , , 00000000002.170.6, , , , , 00000000002.110.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:10:09.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:10:09.303+0000 7f9077d23780 1 waiting for incremental sync to catch up: 2024-04-12T04:10:09.287 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , 00000000002.170.6, , , , , , , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:10:09.287 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.140.6, , , , , , 00000000002.170.6, , , , , 00000000002.110.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:10:14.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:10:14.306+0000 7f9077d23780 1 waiting for incremental sync to catch up: 2024-04-12T04:10:14.290 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , 00000000002.170.6, , , , , , , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:10:14.290 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.140.6, , , , , , 00000000002.170.6, , , , , 00000000002.110.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:10:19.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:10:19.309+0000 7f9077d23780 1 waiting for incremental sync to catch up: 2024-04-12T04:10:19.293 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , , , , , 00000000002.170.6, , , , , , , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:10:19.293 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.140.6, , , , , , 00000000002.170.6, , , , , 00000000002.110.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:10:24.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:10:24.312+0000 7f9077d23780 1 waiting for incremental sync to catch up: 2024-04-12T04:10:24.296 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.140.6, , , , , , 00000000002.170.6, , , , , , , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:10:24.296 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.140.6, , , , , , 00000000002.170.6, , , , , 00000000002.110.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:10:29.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:10:29.315+0000 7f9077d23780 1 waiting for incremental sync to catch up: 2024-04-12T04:10:29.299 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.140.6, , , , , , 00000000002.170.6, , , , , , , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:10:29.299 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.140.6, , , , , , 00000000002.170.6, , , , , 00000000002.110.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:10:34.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:10:34.317+0000 7f9077d23780 1 bucket sync caught up with source: 2024-04-12T04:10:34.301 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.140.6, , , , , , 00000000002.170.6, , , , , 00000000002.110.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:10:34.301 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.140.6, , , , , , 00000000002.170.6, , , , , 00000000002.110.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:10:34.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:10:34.317+0000 7f9077d23780 0 bucket checkpoint complete 2024-04-12T04:10:34.320 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-61 2024-04-12T04:10:34.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:10:34.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:10:34.321 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:10:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:34.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:10:34.447+0000 7fbfb9b61780 1 waiting for incremental sync to catch up: 2024-04-12T04:10:34.431 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000003.2390.6, 00000000002.1411.6, 00000000002.3002.6, 00000000005.2398.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000004.2575.6, 00000000006.2294.6, 00000000005.2626.6] 2024-04-12T04:10:34.431 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000003.2390.6, 00000000002.1411.6, 00000000002.3002.6, 00000000005.2398.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000004.2575.6, 00000000007.2298.6, 00000000006.2628.6] 2024-04-12T04:10:39.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:10:39.448+0000 7fbfb9b61780 1 waiting for incremental sync to catch up: 2024-04-12T04:10:39.433 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000003.2390.6, 00000000002.1411.6, 00000000002.3002.6, 00000000005.2398.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000004.2575.6, 00000000006.2294.6, 00000000005.2626.6] 2024-04-12T04:10:39.433 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000003.2390.6, 00000000002.1411.6, 00000000002.3002.6, 00000000005.2398.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000004.2575.6, 00000000007.2298.6, 00000000006.2628.6] 2024-04-12T04:10:44.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:10:44.449+0000 7fbfb9b61780 1 bucket sync caught up with source: 2024-04-12T04:10:44.433 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000003.2390.6, 00000000002.1411.6, 00000000002.3002.6, 00000000005.2398.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000004.2575.6, 00000000007.2298.6, 00000000006.2628.6] 2024-04-12T04:10:44.433 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000003.2390.6, 00000000002.1411.6, 00000000002.3002.6, 00000000005.2398.6, 00000000004.2607.6, 00000000001.3050.6, 00000000001.3636.6, 00000000004.2575.6, 00000000007.2298.6, 00000000006.2628.6] 2024-04-12T04:10:44.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:10:44.449+0000 7fbfb9b61780 0 bucket checkpoint complete 2024-04-12T04:10:44.452 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-61 zones={a2, a1} 2024-04-12T04:10:44.452 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.452 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.452 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.452 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:10:44.452 DEBUG:boto:Data: 2024-04-12T04:10:44.452 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.453 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.453 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.453 DEBUG:boto:Params: {} 2024-04-12T04:10:44.453 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:10:44.453 DEBUG:boto:Token: None 2024-04-12T04:10:44.454 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.454 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8H2/+sKR9rAayJk9UHaGU4f6AYk= 2024-04-12T04:10:44.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8H2/+sKR9rAayJk9UHaGU4f6AYk=', 'Content-Length': '0'} 2024-04-12T04:10:44.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004851d6ddc89cc54d-006618b444-1cfd-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.468 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.468 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.468 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.468 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:10:44.468 DEBUG:boto:Data: 2024-04-12T04:10:44.468 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.468 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.468 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.469 DEBUG:boto:Params: {} 2024-04-12T04:10:44.469 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:10:44.469 DEBUG:boto:Token: None 2024-04-12T04:10:44.469 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.469 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8H2/+sKR9rAayJk9UHaGU4f6AYk= 2024-04-12T04:10:44.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8H2/+sKR9rAayJk9UHaGU4f6AYk=', 'Content-Length': '0'} 2024-04-12T04:10:44.484 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d82dfd0385c75bbe-006618b444-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.485 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:10:44.485 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.485 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.485 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:10:44.485 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:10:44.485 DEBUG:boto:Method: GET 2024-04-12T04:10:44.485 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:10:44.485 DEBUG:boto:Data: 2024-04-12T04:10:44.485 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.485 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.486 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.486 DEBUG:boto:Params: {} 2024-04-12T04:10:44.486 DEBUG:boto:Token: None 2024-04-12T04:10:44.486 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/?versions 2024-04-12T04:10:44.486 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iXgbRQiGDU5btzYMVDeKxa9TAzI= 2024-04-12T04:10:44.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iXgbRQiGDU5btzYMVDeKxa9TAzI=', 'Content-Length': '0'} 2024-04-12T04:10:44.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0c649bb710eb327-006618b444-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.489 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T04:09:24.114Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T04:09:24.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T04:09:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T04:09:24.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:10:44.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-12T04:10:44.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T04:10:44.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T04:10:44.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-12T04:10:44.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T04:10:44.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-12T04:10:44.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-12T04:10:44.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:10:44.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-12T04:10:44.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T04:10:44.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:10:44.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T04:10:44.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T04:10:44.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-12T04:10:44.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:10:44.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:10:44.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-12T04:10:44.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T04:10:44.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:10:44.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T04:10:44.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:10:44.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:10:44.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T04:10:44.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T04:10:44.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-12T04:10:44.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:10:44.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T04:10:44.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-12T04:10:44.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T04:10:44.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:10:44.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T04:10:44.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T04:10:44.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-12T04:10:44.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T04:10:44.494 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:10:44.494 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.494 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.494 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:10:44.495 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:10:44.495 DEBUG:boto:Method: GET 2024-04-12T04:10:44.495 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:10:44.495 DEBUG:boto:Data: 2024-04-12T04:10:44.495 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.495 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.495 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.495 DEBUG:boto:Params: {} 2024-04-12T04:10:44.495 DEBUG:boto:Token: None 2024-04-12T04:10:44.495 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/?versions 2024-04-12T04:10:44.496 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iXgbRQiGDU5btzYMVDeKxa9TAzI= 2024-04-12T04:10:44.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iXgbRQiGDU5btzYMVDeKxa9TAzI=', 'Content-Length': '0'} 2024-04-12T04:10:44.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bde8baacfccf4cfe-006618b444-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.501 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T04:09:24.114Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T04:09:24.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T04:09:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T04:09:24.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:10:44.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-12T04:10:44.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T04:10:44.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T04:10:44.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-12T04:10:44.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T04:10:44.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-12T04:10:44.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-12T04:10:44.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:10:44.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-12T04:10:44.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T04:10:44.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:10:44.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T04:10:44.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T04:10:44.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-12T04:10:44.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:10:44.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:10:44.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-12T04:10:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T04:10:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:10:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T04:10:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:10:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:10:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T04:10:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T04:10:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-12T04:10:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:10:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T04:10:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-12T04:10:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T04:10:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:10:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T04:10:44.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T04:10:44.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-12T04:10:44.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T04:10:44.505 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.505 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.505 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:10:44.505 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:10:44.505 DEBUG:boto:Method: GET 2024-04-12T04:10:44.505 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:10:44.505 DEBUG:boto:Data: 2024-04-12T04:10:44.505 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.506 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.506 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.506 DEBUG:boto:Params: {} 2024-04-12T04:10:44.506 DEBUG:boto:Token: None 2024-04-12T04:10:44.506 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/?versions 2024-04-12T04:10:44.506 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iXgbRQiGDU5btzYMVDeKxa9TAzI= 2024-04-12T04:10:44.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iXgbRQiGDU5btzYMVDeKxa9TAzI=', 'Content-Length': '0'} 2024-04-12T04:10:44.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e858e02d246b9478-006618b444-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.509 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T04:09:24.114Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T04:09:24.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T04:09:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T04:09:24.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.511 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.511 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.511 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:10:44.511 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:10:44.511 DEBUG:boto:Method: GET 2024-04-12T04:10:44.511 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:10:44.511 DEBUG:boto:Data: 2024-04-12T04:10:44.511 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.511 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.511 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.512 DEBUG:boto:Params: {} 2024-04-12T04:10:44.512 DEBUG:boto:Token: None 2024-04-12T04:10:44.512 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/?versions 2024-04-12T04:10:44.512 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iXgbRQiGDU5btzYMVDeKxa9TAzI= 2024-04-12T04:10:44.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iXgbRQiGDU5btzYMVDeKxa9TAzI=', 'Content-Length': '0'} 2024-04-12T04:10:44.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbbe92a2ca2299e2-006618b444-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.517 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T04:09:24.114Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T04:09:24.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T04:09:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T04:09:24.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:10:44.519 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:10:44.519 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:10:44.519 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.519 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.519 DEBUG:boto:Method: GET 2024-04-12T04:10:44.519 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.520 DEBUG:boto:Data: 2024-04-12T04:10:44.520 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.520 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.520 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.520 DEBUG:boto:Params: {} 2024-04-12T04:10:44.520 DEBUG:boto:Token: None 2024-04-12T04:10:44.520 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.520 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cmLCpKXN+FSU31tiayuCGVjrREQ= 2024-04-12T04:10:44.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cmLCpKXN+FSU31tiayuCGVjrREQ=', 'Content-Length': '0'} 2024-04-12T04:10:44.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a21c643d63a854c-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.523 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:10:44.523 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:10:44.523 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.523 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.523 DEBUG:boto:Method: GET 2024-04-12T04:10:44.523 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.523 DEBUG:boto:Data: 2024-04-12T04:10:44.524 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.524 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.524 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.524 DEBUG:boto:Params: {} 2024-04-12T04:10:44.524 DEBUG:boto:Token: None 2024-04-12T04:10:44.524 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.524 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cmLCpKXN+FSU31tiayuCGVjrREQ= 2024-04-12T04:10:44.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cmLCpKXN+FSU31tiayuCGVjrREQ=', 'Content-Length': '0'} 2024-04-12T04:10:44.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b6c70844aec5f85-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.570 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:10:44.570 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:10:44.571 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.571 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.571 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.571 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.571 DEBUG:boto:Data: 2024-04-12T04:10:44.571 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.571 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.571 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.571 DEBUG:boto:Params: {} 2024-04-12T04:10:44.571 DEBUG:boto:Token: None 2024-04-12T04:10:44.572 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.572 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t0qVlH0KDZeRm/SnU+U/d/F28V4= 2024-04-12T04:10:44.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t0qVlH0KDZeRm/SnU+U/d/F28V4=', 'Content-Length': '0'} 2024-04-12T04:10:44.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fc5c4da82e8718d-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.574 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:10:44.574 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:10:44.574 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.574 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.575 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.575 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.575 DEBUG:boto:Data: 2024-04-12T04:10:44.575 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.575 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.575 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.575 DEBUG:boto:Params: {} 2024-04-12T04:10:44.575 DEBUG:boto:Token: None 2024-04-12T04:10:44.575 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.575 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t0qVlH0KDZeRm/SnU+U/d/F28V4= 2024-04-12T04:10:44.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t0qVlH0KDZeRm/SnU+U/d/F28V4=', 'Content-Length': '0'} 2024-04-12T04:10:44.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000584fb83a52545d2-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:10:44.578 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:10:44.578 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:10:44.578 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.578 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.578 DEBUG:boto:Method: GET 2024-04-12T04:10:44.578 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.578 DEBUG:boto:Data: 2024-04-12T04:10:44.578 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.578 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.578 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.578 DEBUG:boto:Params: {} 2024-04-12T04:10:44.579 DEBUG:boto:Token: None 2024-04-12T04:10:44.579 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.579 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cmLCpKXN+FSU31tiayuCGVjrREQ= 2024-04-12T04:10:44.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cmLCpKXN+FSU31tiayuCGVjrREQ=', 'Content-Length': '0'} 2024-04-12T04:10:44.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000437eeb7ff533ded4-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.581 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:10:44.581 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:10:44.581 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.581 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.581 DEBUG:boto:Method: GET 2024-04-12T04:10:44.581 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.581 DEBUG:boto:Data: 2024-04-12T04:10:44.581 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.581 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.582 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.582 DEBUG:boto:Params: {} 2024-04-12T04:10:44.582 DEBUG:boto:Token: None 2024-04-12T04:10:44.582 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:10:44.582 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cmLCpKXN+FSU31tiayuCGVjrREQ= 2024-04-12T04:10:44.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cmLCpKXN+FSU31tiayuCGVjrREQ=', 'Content-Length': '0'} 2024-04-12T04:10:44.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013b5f89e50d34f2f-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.584 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.584 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.584 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:10:44.585 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:10:44.585 DEBUG:boto:Method: GET 2024-04-12T04:10:44.585 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:10:44.585 DEBUG:boto:Data: 2024-04-12T04:10:44.585 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.585 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.585 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.585 DEBUG:boto:Params: {} 2024-04-12T04:10:44.585 DEBUG:boto:Token: None 2024-04-12T04:10:44.585 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.585 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8= 2024-04-12T04:10:44.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8=', 'Content-Length': '0'} 2024-04-12T04:10:44.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc3e66657872ebd4-006618b444-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.588 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.589 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.589 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.589 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:10:44.589 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:10:44.589 DEBUG:boto:Method: GET 2024-04-12T04:10:44.589 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:10:44.590 DEBUG:boto:Data: 2024-04-12T04:10:44.590 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.590 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.590 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.590 DEBUG:boto:Params: {} 2024-04-12T04:10:44.590 DEBUG:boto:Token: None 2024-04-12T04:10:44.590 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.590 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8= 2024-04-12T04:10:44.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8=', 'Content-Length': '0'} 2024-04-12T04:10:44.595 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fefde0999f6c64e-006618b444-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.595 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.595 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:10:44.595 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:10:44.596 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.596 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:10:44.596 DEBUG:boto:Data: 2024-04-12T04:10:44.596 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.596 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.596 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.596 DEBUG:boto:Params: {} 2024-04-12T04:10:44.596 DEBUG:boto:Token: None 2024-04-12T04:10:44.596 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj11 2024-04-12T04:10:44.597 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bjgbrVh1e31NtwnY+Pmy0ro+Xa4= 2024-04-12T04:10:44.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bjgbrVh1e31NtwnY+Pmy0ro+Xa4=', 'Content-Length': '0'} 2024-04-12T04:10:44.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e503869d00eed62-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.599 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:10:44.599 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:10:44.599 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.599 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:10:44.599 DEBUG:boto:Data: 2024-04-12T04:10:44.599 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.600 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.600 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.600 DEBUG:boto:Params: {} 2024-04-12T04:10:44.600 DEBUG:boto:Token: None 2024-04-12T04:10:44.600 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj11 2024-04-12T04:10:44.600 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bjgbrVh1e31NtwnY+Pmy0ro+Xa4= 2024-04-12T04:10:44.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bjgbrVh1e31NtwnY+Pmy0ro+Xa4=', 'Content-Length': '0'} 2024-04-12T04:10:44.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059edf03340b32c3a-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:10:44.602 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:10:44.602 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:10:44.603 DEBUG:boto:Method: GET 2024-04-12T04:10:44.603 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:10:44.603 DEBUG:boto:Data: 2024-04-12T04:10:44.603 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.603 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.603 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.603 DEBUG:boto:Params: {} 2024-04-12T04:10:44.603 DEBUG:boto:Token: None 2024-04-12T04:10:44.603 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj11 2024-04-12T04:10:44.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d5exPzukCmxklchMXG9ec489v9M= 2024-04-12T04:10:44.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d5exPzukCmxklchMXG9ec489v9M=', 'Content-Length': '0'} 2024-04-12T04:10:44.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e42e08bb83d4fa1-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.646 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:10:44.646 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:10:44.647 DEBUG:boto:Method: GET 2024-04-12T04:10:44.647 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:10:44.647 DEBUG:boto:Data: 2024-04-12T04:10:44.647 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.647 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.647 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.647 DEBUG:boto:Params: {} 2024-04-12T04:10:44.647 DEBUG:boto:Token: None 2024-04-12T04:10:44.647 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj11 2024-04-12T04:10:44.647 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d5exPzukCmxklchMXG9ec489v9M= 2024-04-12T04:10:44.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d5exPzukCmxklchMXG9ec489v9M=', 'Content-Length': '0'} 2024-04-12T04:10:44.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e36835220f58dfd-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.650 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.650 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.650 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:10:44.650 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:10:44.650 DEBUG:boto:Method: GET 2024-04-12T04:10:44.650 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:10:44.650 DEBUG:boto:Data: 2024-04-12T04:10:44.650 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.650 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.651 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.651 DEBUG:boto:Params: {} 2024-04-12T04:10:44.651 DEBUG:boto:Token: None 2024-04-12T04:10:44.651 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.651 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8= 2024-04-12T04:10:44.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8=', 'Content-Length': '0'} 2024-04-12T04:10:44.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000987d47462fb215d6-006618b444-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.654 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.654 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.655 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.655 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:10:44.655 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:10:44.655 DEBUG:boto:Method: GET 2024-04-12T04:10:44.655 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:10:44.655 DEBUG:boto:Data: 2024-04-12T04:10:44.655 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.655 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.655 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.655 DEBUG:boto:Params: {} 2024-04-12T04:10:44.655 DEBUG:boto:Token: None 2024-04-12T04:10:44.656 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8= 2024-04-12T04:10:44.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8=', 'Content-Length': '0'} 2024-04-12T04:10:44.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccd544a357c8e45b-006618b444-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.662 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T04:10:44.662 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:10:44.662 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:10:44.663 DEBUG:boto:path=/dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.663 DEBUG:boto:auth_path=/dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.663 DEBUG:boto:Method: GET 2024-04-12T04:10:44.663 DEBUG:boto:Path: /dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.663 DEBUG:boto:Data: 2024-04-12T04:10:44.663 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.663 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.663 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.663 DEBUG:boto:Params: {} 2024-04-12T04:10:44.663 DEBUG:boto:Token: None 2024-04-12T04:10:44.663 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.663 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NZhORQGziRx0s5twDn+yC4kjjBE= 2024-04-12T04:10:44.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NZhORQGziRx0s5twDn+yC4kjjBE=', 'Content-Length': '0'} 2024-04-12T04:10:44.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dba5c0d44f99ecb7-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.666 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:10:44.666 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:10:44.666 DEBUG:boto:path=/dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.666 DEBUG:boto:auth_path=/dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.666 DEBUG:boto:Method: GET 2024-04-12T04:10:44.666 DEBUG:boto:Path: /dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.666 DEBUG:boto:Data: 2024-04-12T04:10:44.666 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.666 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.667 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.667 DEBUG:boto:Params: {} 2024-04-12T04:10:44.667 DEBUG:boto:Token: None 2024-04-12T04:10:44.667 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.667 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NZhORQGziRx0s5twDn+yC4kjjBE= 2024-04-12T04:10:44.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NZhORQGziRx0s5twDn+yC4kjjBE=', 'Content-Length': '0'} 2024-04-12T04:10:44.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000857270002269034-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.714 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:10:44.714 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:10:44.714 DEBUG:boto:path=/dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.715 DEBUG:boto:auth_path=/dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.715 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.715 DEBUG:boto:Path: /dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.715 DEBUG:boto:Data: 2024-04-12T04:10:44.715 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.715 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.715 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.715 DEBUG:boto:Params: {} 2024-04-12T04:10:44.715 DEBUG:boto:Token: None 2024-04-12T04:10:44.715 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.716 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zDXlv2n//9o3Br+697h+guO9XMc= 2024-04-12T04:10:44.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zDXlv2n//9o3Br+697h+guO9XMc=', 'Content-Length': '0'} 2024-04-12T04:10:44.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0691eaca918dbaf-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.718 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:10:44.718 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:10:44.718 DEBUG:boto:path=/dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.718 DEBUG:boto:auth_path=/dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.718 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.718 DEBUG:boto:Path: /dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.718 DEBUG:boto:Data: 2024-04-12T04:10:44.718 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.719 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.719 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.719 DEBUG:boto:Params: {} 2024-04-12T04:10:44.719 DEBUG:boto:Token: None 2024-04-12T04:10:44.719 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.719 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zDXlv2n//9o3Br+697h+guO9XMc= 2024-04-12T04:10:44.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zDXlv2n//9o3Br+697h+guO9XMc=', 'Content-Length': '0'} 2024-04-12T04:10:44.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000054b3d528821386d-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T04:10:44.721 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:10:44.721 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:10:44.722 DEBUG:boto:path=/dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.722 DEBUG:boto:auth_path=/dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.722 DEBUG:boto:Method: GET 2024-04-12T04:10:44.722 DEBUG:boto:Path: /dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.722 DEBUG:boto:Data: 2024-04-12T04:10:44.722 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.722 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.722 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.722 DEBUG:boto:Params: {} 2024-04-12T04:10:44.722 DEBUG:boto:Token: None 2024-04-12T04:10:44.723 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.723 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NZhORQGziRx0s5twDn+yC4kjjBE= 2024-04-12T04:10:44.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NZhORQGziRx0s5twDn+yC4kjjBE=', 'Content-Length': '0'} 2024-04-12T04:10:44.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef092ef6cbb3bfb9-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.725 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:10:44.725 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:10:44.725 DEBUG:boto:path=/dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.725 DEBUG:boto:auth_path=/dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.725 DEBUG:boto:Method: GET 2024-04-12T04:10:44.725 DEBUG:boto:Path: /dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.725 DEBUG:boto:Data: 2024-04-12T04:10:44.725 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.726 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.726 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.726 DEBUG:boto:Params: {} 2024-04-12T04:10:44.726 DEBUG:boto:Token: None 2024-04-12T04:10:44.726 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj111?versionId=null 2024-04-12T04:10:44.726 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NZhORQGziRx0s5twDn+yC4kjjBE= 2024-04-12T04:10:44.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NZhORQGziRx0s5twDn+yC4kjjBE=', 'Content-Length': '0'} 2024-04-12T04:10:44.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d7992d7db4ab2f9-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.728 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.728 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.729 DEBUG:boto:path=/dkyges-61/?prefix=obj111 2024-04-12T04:10:44.729 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj111 2024-04-12T04:10:44.729 DEBUG:boto:Method: GET 2024-04-12T04:10:44.729 DEBUG:boto:Path: /dkyges-61/?prefix=obj111 2024-04-12T04:10:44.729 DEBUG:boto:Data: 2024-04-12T04:10:44.729 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.729 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.729 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.729 DEBUG:boto:Params: {} 2024-04-12T04:10:44.729 DEBUG:boto:Token: None 2024-04-12T04:10:44.730 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.730 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8= 2024-04-12T04:10:44.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8=', 'Content-Length': '0'} 2024-04-12T04:10:44.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000696c878482b9ae2e-006618b444-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.733 DEBUG:boto:b'dkyges-61obj1111000falseobj1112024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.734 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.734 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.734 DEBUG:boto:path=/dkyges-61/?prefix=obj111 2024-04-12T04:10:44.734 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj111 2024-04-12T04:10:44.734 DEBUG:boto:Method: GET 2024-04-12T04:10:44.734 DEBUG:boto:Path: /dkyges-61/?prefix=obj111 2024-04-12T04:10:44.735 DEBUG:boto:Data: 2024-04-12T04:10:44.735 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.735 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.735 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.735 DEBUG:boto:Params: {} 2024-04-12T04:10:44.735 DEBUG:boto:Token: None 2024-04-12T04:10:44.735 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.735 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8= 2024-04-12T04:10:44.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8=', 'Content-Length': '0'} 2024-04-12T04:10:44.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005a8471ebf0971c4-006618b444-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.740 DEBUG:boto:b'dkyges-61obj1111000falseobj1112024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.740 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:10:44.740 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:10:44.740 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.741 DEBUG:boto:Path: /dkyges-61/obj111 2024-04-12T04:10:44.741 DEBUG:boto:Data: 2024-04-12T04:10:44.741 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.741 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.741 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.741 DEBUG:boto:Params: {} 2024-04-12T04:10:44.741 DEBUG:boto:Token: None 2024-04-12T04:10:44.741 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj111 2024-04-12T04:10:44.742 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GQKQWPO3V8hA4oeptcbfgzUndtQ= 2024-04-12T04:10:44.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GQKQWPO3V8hA4oeptcbfgzUndtQ=', 'Content-Length': '0'} 2024-04-12T04:10:44.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a65aa3dc78263a7-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.744 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:10:44.744 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:10:44.744 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.744 DEBUG:boto:Path: /dkyges-61/obj111 2024-04-12T04:10:44.744 DEBUG:boto:Data: 2024-04-12T04:10:44.744 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.744 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.745 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.745 DEBUG:boto:Params: {} 2024-04-12T04:10:44.745 DEBUG:boto:Token: None 2024-04-12T04:10:44.745 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj111 2024-04-12T04:10:44.745 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GQKQWPO3V8hA4oeptcbfgzUndtQ= 2024-04-12T04:10:44.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GQKQWPO3V8hA4oeptcbfgzUndtQ=', 'Content-Length': '0'} 2024-04-12T04:10:44.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b85df5895fd8905-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T04:10:44.747 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:10:44.747 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:10:44.747 DEBUG:boto:Method: GET 2024-04-12T04:10:44.747 DEBUG:boto:Path: /dkyges-61/obj111 2024-04-12T04:10:44.747 DEBUG:boto:Data: 2024-04-12T04:10:44.748 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.748 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.748 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.748 DEBUG:boto:Params: {} 2024-04-12T04:10:44.748 DEBUG:boto:Token: None 2024-04-12T04:10:44.748 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj111 2024-04-12T04:10:44.748 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/pVj06NChXyf2gsnpjVR7nz4umM= 2024-04-12T04:10:44.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/pVj06NChXyf2gsnpjVR7nz4umM=', 'Content-Length': '0'} 2024-04-12T04:10:44.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4a2368fd8d28e29-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.751 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:10:44.751 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:10:44.751 DEBUG:boto:Method: GET 2024-04-12T04:10:44.751 DEBUG:boto:Path: /dkyges-61/obj111 2024-04-12T04:10:44.751 DEBUG:boto:Data: 2024-04-12T04:10:44.751 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.751 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.751 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.751 DEBUG:boto:Params: {} 2024-04-12T04:10:44.751 DEBUG:boto:Token: None 2024-04-12T04:10:44.752 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj111 2024-04-12T04:10:44.752 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/pVj06NChXyf2gsnpjVR7nz4umM= 2024-04-12T04:10:44.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/pVj06NChXyf2gsnpjVR7nz4umM=', 'Content-Length': '0'} 2024-04-12T04:10:44.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a654663d44b97c6e-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.794 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.794 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.794 DEBUG:boto:path=/dkyges-61/?prefix=obj111 2024-04-12T04:10:44.795 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj111 2024-04-12T04:10:44.795 DEBUG:boto:Method: GET 2024-04-12T04:10:44.795 DEBUG:boto:Path: /dkyges-61/?prefix=obj111 2024-04-12T04:10:44.795 DEBUG:boto:Data: 2024-04-12T04:10:44.795 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.795 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.795 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.795 DEBUG:boto:Params: {} 2024-04-12T04:10:44.795 DEBUG:boto:Token: None 2024-04-12T04:10:44.796 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.796 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8= 2024-04-12T04:10:44.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8=', 'Content-Length': '0'} 2024-04-12T04:10:44.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005846929da16e9cd-006618b444-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.798 DEBUG:boto:b'dkyges-61obj1111000falseobj1112024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.799 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.799 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.799 DEBUG:boto:path=/dkyges-61/?prefix=obj111 2024-04-12T04:10:44.799 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj111 2024-04-12T04:10:44.799 DEBUG:boto:Method: GET 2024-04-12T04:10:44.799 DEBUG:boto:Path: /dkyges-61/?prefix=obj111 2024-04-12T04:10:44.799 DEBUG:boto:Data: 2024-04-12T04:10:44.799 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.799 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.799 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.800 DEBUG:boto:Params: {} 2024-04-12T04:10:44.800 DEBUG:boto:Token: None 2024-04-12T04:10:44.800 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.800 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8= 2024-04-12T04:10:44.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8=', 'Content-Length': '0'} 2024-04-12T04:10:44.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef998dff56109242-006618b444-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.805 DEBUG:boto:b'dkyges-61obj1111000falseobj1112024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:10:44.805 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:10:44.805 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:10:44.805 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.805 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.805 DEBUG:boto:Method: GET 2024-04-12T04:10:44.805 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.806 DEBUG:boto:Data: 2024-04-12T04:10:44.806 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.806 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.806 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.806 DEBUG:boto:Params: {} 2024-04-12T04:10:44.806 DEBUG:boto:Token: None 2024-04-12T04:10:44.806 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.806 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aVA7ymdaY0JhKGL/L+GXj1aDSTI= 2024-04-12T04:10:44.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aVA7ymdaY0JhKGL/L+GXj1aDSTI=', 'Content-Length': '0'} 2024-04-12T04:10:44.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008ded8831935e99d-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.850 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:10:44.850 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:10:44.850 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.851 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.851 DEBUG:boto:Method: GET 2024-04-12T04:10:44.851 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.851 DEBUG:boto:Data: 2024-04-12T04:10:44.851 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.851 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.851 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.851 DEBUG:boto:Params: {} 2024-04-12T04:10:44.851 DEBUG:boto:Token: None 2024-04-12T04:10:44.851 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.852 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aVA7ymdaY0JhKGL/L+GXj1aDSTI= 2024-04-12T04:10:44.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aVA7ymdaY0JhKGL/L+GXj1aDSTI=', 'Content-Length': '0'} 2024-04-12T04:10:44.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af24343fa73659b1-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.854 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:10:44.854 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:10:44.854 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.854 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.854 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.855 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.855 DEBUG:boto:Data: 2024-04-12T04:10:44.855 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.855 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.855 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.855 DEBUG:boto:Params: {} 2024-04-12T04:10:44.855 DEBUG:boto:Token: None 2024-04-12T04:10:44.855 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.855 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GBO7J/JRYJo2T+/iFYOf1PjZgn4= 2024-04-12T04:10:44.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBO7J/JRYJo2T+/iFYOf1PjZgn4=', 'Content-Length': '0'} 2024-04-12T04:10:44.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e619178f5a52ba7d-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.858 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:10:44.858 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:10:44.858 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.858 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.858 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.858 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.858 DEBUG:boto:Data: 2024-04-12T04:10:44.858 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.858 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.858 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.858 DEBUG:boto:Params: {} 2024-04-12T04:10:44.859 DEBUG:boto:Token: None 2024-04-12T04:10:44.859 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.859 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GBO7J/JRYJo2T+/iFYOf1PjZgn4= 2024-04-12T04:10:44.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBO7J/JRYJo2T+/iFYOf1PjZgn4=', 'Content-Length': '0'} 2024-04-12T04:10:44.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1955dfbe4483440-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:10:44.861 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:10:44.861 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:10:44.861 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.861 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.861 DEBUG:boto:Method: GET 2024-04-12T04:10:44.862 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.862 DEBUG:boto:Data: 2024-04-12T04:10:44.862 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.862 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.862 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.862 DEBUG:boto:Params: {} 2024-04-12T04:10:44.862 DEBUG:boto:Token: None 2024-04-12T04:10:44.862 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.863 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aVA7ymdaY0JhKGL/L+GXj1aDSTI= 2024-04-12T04:10:44.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aVA7ymdaY0JhKGL/L+GXj1aDSTI=', 'Content-Length': '0'} 2024-04-12T04:10:44.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bac41ef158895312-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.865 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:10:44.865 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:10:44.865 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.865 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.865 DEBUG:boto:Method: GET 2024-04-12T04:10:44.865 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.865 DEBUG:boto:Data: 2024-04-12T04:10:44.866 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.866 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.866 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.866 DEBUG:boto:Params: {} 2024-04-12T04:10:44.866 DEBUG:boto:Token: None 2024-04-12T04:10:44.866 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:10:44.866 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aVA7ymdaY0JhKGL/L+GXj1aDSTI= 2024-04-12T04:10:44.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aVA7ymdaY0JhKGL/L+GXj1aDSTI=', 'Content-Length': '0'} 2024-04-12T04:10:44.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f3c6cdb932c7db2-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.869 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.869 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.869 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:10:44.869 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:10:44.869 DEBUG:boto:Method: GET 2024-04-12T04:10:44.869 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:10:44.869 DEBUG:boto:Data: 2024-04-12T04:10:44.869 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.869 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.869 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.870 DEBUG:boto:Params: {} 2024-04-12T04:10:44.870 DEBUG:boto:Token: None 2024-04-12T04:10:44.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8= 2024-04-12T04:10:44.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8=', 'Content-Length': '0'} 2024-04-12T04:10:44.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085c3f6c0f1ec6b8a-006618b444-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.873 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.873 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.873 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.873 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:10:44.873 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:10:44.874 DEBUG:boto:Method: GET 2024-04-12T04:10:44.874 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:10:44.874 DEBUG:boto:Data: 2024-04-12T04:10:44.874 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.874 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.874 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.874 DEBUG:boto:Params: {} 2024-04-12T04:10:44.874 DEBUG:boto:Token: None 2024-04-12T04:10:44.874 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.874 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8= 2024-04-12T04:10:44.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8=', 'Content-Length': '0'} 2024-04-12T04:10:44.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ab46801095ef054-006618b444-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.879 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.880 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:10:44.880 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:10:44.880 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.880 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:10:44.880 DEBUG:boto:Data: 2024-04-12T04:10:44.880 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.880 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.880 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.880 DEBUG:boto:Params: {} 2024-04-12T04:10:44.880 DEBUG:boto:Token: None 2024-04-12T04:10:44.880 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj115 2024-04-12T04:10:44.881 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eVXuPyaNLPv/in9J+3Lxy+m8Dqo= 2024-04-12T04:10:44.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eVXuPyaNLPv/in9J+3Lxy+m8Dqo=', 'Content-Length': '0'} 2024-04-12T04:10:44.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082525104cb430a04-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.882 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:10:44.883 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:10:44.883 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.883 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:10:44.883 DEBUG:boto:Data: 2024-04-12T04:10:44.883 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.883 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.883 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.883 DEBUG:boto:Params: {} 2024-04-12T04:10:44.883 DEBUG:boto:Token: None 2024-04-12T04:10:44.883 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj115 2024-04-12T04:10:44.883 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eVXuPyaNLPv/in9J+3Lxy+m8Dqo= 2024-04-12T04:10:44.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eVXuPyaNLPv/in9J+3Lxy+m8Dqo=', 'Content-Length': '0'} 2024-04-12T04:10:44.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003460eaf63c40c7c7-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:10:44.886 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:10:44.886 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:10:44.886 DEBUG:boto:Method: GET 2024-04-12T04:10:44.886 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:10:44.886 DEBUG:boto:Data: 2024-04-12T04:10:44.886 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.886 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.886 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.887 DEBUG:boto:Params: {} 2024-04-12T04:10:44.887 DEBUG:boto:Token: None 2024-04-12T04:10:44.887 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj115 2024-04-12T04:10:44.887 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:twHjLKr5aAwvqxUTqrd1SmoXbBg= 2024-04-12T04:10:44.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:twHjLKr5aAwvqxUTqrd1SmoXbBg=', 'Content-Length': '0'} 2024-04-12T04:10:44.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afb0cdec427e4d44-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.889 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:10:44.889 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:10:44.889 DEBUG:boto:Method: GET 2024-04-12T04:10:44.889 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:10:44.889 DEBUG:boto:Data: 2024-04-12T04:10:44.889 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.889 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.890 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.890 DEBUG:boto:Params: {} 2024-04-12T04:10:44.890 DEBUG:boto:Token: None 2024-04-12T04:10:44.890 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj115 2024-04-12T04:10:44.890 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:twHjLKr5aAwvqxUTqrd1SmoXbBg= 2024-04-12T04:10:44.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:twHjLKr5aAwvqxUTqrd1SmoXbBg=', 'Content-Length': '0'} 2024-04-12T04:10:44.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a5bce7df9357d4a-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.892 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.893 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.893 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:10:44.893 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:10:44.893 DEBUG:boto:Method: GET 2024-04-12T04:10:44.893 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:10:44.893 DEBUG:boto:Data: 2024-04-12T04:10:44.893 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.893 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.893 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.893 DEBUG:boto:Params: {} 2024-04-12T04:10:44.893 DEBUG:boto:Token: None 2024-04-12T04:10:44.893 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.894 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8= 2024-04-12T04:10:44.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8=', 'Content-Length': '0'} 2024-04-12T04:10:44.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000055db5a101082ab1-006618b444-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.896 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.896 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.897 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.897 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:10:44.897 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:10:44.897 DEBUG:boto:Method: GET 2024-04-12T04:10:44.897 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:10:44.897 DEBUG:boto:Data: 2024-04-12T04:10:44.897 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.897 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.897 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.897 DEBUG:boto:Params: {} 2024-04-12T04:10:44.897 DEBUG:boto:Token: None 2024-04-12T04:10:44.897 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.898 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8= 2024-04-12T04:10:44.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8=', 'Content-Length': '0'} 2024-04-12T04:10:44.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a56d0be91ea06339-006618b444-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.902 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:10:44.902 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:10:44.903 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:10:44.903 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.903 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.903 DEBUG:boto:Method: GET 2024-04-12T04:10:44.903 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.903 DEBUG:boto:Data: 2024-04-12T04:10:44.903 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.903 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.903 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.903 DEBUG:boto:Params: {} 2024-04-12T04:10:44.903 DEBUG:boto:Token: None 2024-04-12T04:10:44.904 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.904 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7f5Q7Wjg7IiLPoMFtRf/E6ywPWU= 2024-04-12T04:10:44.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7f5Q7Wjg7IiLPoMFtRf/E6ywPWU=', 'Content-Length': '0'} 2024-04-12T04:10:44.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab606199dbd0c17e-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.906 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:10:44.906 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:10:44.906 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.906 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.906 DEBUG:boto:Method: GET 2024-04-12T04:10:44.906 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.906 DEBUG:boto:Data: 2024-04-12T04:10:44.906 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.906 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.907 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.907 DEBUG:boto:Params: {} 2024-04-12T04:10:44.907 DEBUG:boto:Token: None 2024-04-12T04:10:44.907 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.907 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7f5Q7Wjg7IiLPoMFtRf/E6ywPWU= 2024-04-12T04:10:44.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7f5Q7Wjg7IiLPoMFtRf/E6ywPWU=', 'Content-Length': '0'} 2024-04-12T04:10:44.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065df3bceae23a5ed-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.909 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:10:44.909 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:10:44.910 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.910 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.910 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.910 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.910 DEBUG:boto:Data: 2024-04-12T04:10:44.910 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.910 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.910 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.910 DEBUG:boto:Params: {} 2024-04-12T04:10:44.910 DEBUG:boto:Token: None 2024-04-12T04:10:44.911 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.911 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VzzhQmSvuAewgk/ocUrWL1nue6I= 2024-04-12T04:10:44.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VzzhQmSvuAewgk/ocUrWL1nue6I=', 'Content-Length': '0'} 2024-04-12T04:10:44.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006daf6403f4e2ade8-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.913 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:10:44.913 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:10:44.913 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.913 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.913 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.913 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.913 DEBUG:boto:Data: 2024-04-12T04:10:44.913 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.914 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.914 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.914 DEBUG:boto:Params: {} 2024-04-12T04:10:44.914 DEBUG:boto:Token: None 2024-04-12T04:10:44.914 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.914 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VzzhQmSvuAewgk/ocUrWL1nue6I= 2024-04-12T04:10:44.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VzzhQmSvuAewgk/ocUrWL1nue6I=', 'Content-Length': '0'} 2024-04-12T04:10:44.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da335c985b3e3de3-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:10:44.917 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:10:44.917 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:10:44.917 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.917 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.917 DEBUG:boto:Method: GET 2024-04-12T04:10:44.917 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.917 DEBUG:boto:Data: 2024-04-12T04:10:44.917 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.917 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.917 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.917 DEBUG:boto:Params: {} 2024-04-12T04:10:44.918 DEBUG:boto:Token: None 2024-04-12T04:10:44.918 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.918 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7f5Q7Wjg7IiLPoMFtRf/E6ywPWU= 2024-04-12T04:10:44.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7f5Q7Wjg7IiLPoMFtRf/E6ywPWU=', 'Content-Length': '0'} 2024-04-12T04:10:44.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac5f69c026b48f7f-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.920 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:10:44.920 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:10:44.920 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.920 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.920 DEBUG:boto:Method: GET 2024-04-12T04:10:44.920 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.920 DEBUG:boto:Data: 2024-04-12T04:10:44.921 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.921 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.921 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.921 DEBUG:boto:Params: {} 2024-04-12T04:10:44.921 DEBUG:boto:Token: None 2024-04-12T04:10:44.921 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:10:44.921 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7f5Q7Wjg7IiLPoMFtRf/E6ywPWU= 2024-04-12T04:10:44.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7f5Q7Wjg7IiLPoMFtRf/E6ywPWU=', 'Content-Length': '0'} 2024-04-12T04:10:44.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5bd451803149c41-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.923 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.923 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.923 DEBUG:boto:path=/dkyges-61/?prefix=obj116 2024-04-12T04:10:44.924 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj116 2024-04-12T04:10:44.924 DEBUG:boto:Method: GET 2024-04-12T04:10:44.924 DEBUG:boto:Path: /dkyges-61/?prefix=obj116 2024-04-12T04:10:44.924 DEBUG:boto:Data: 2024-04-12T04:10:44.924 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.924 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.924 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.924 DEBUG:boto:Params: {} 2024-04-12T04:10:44.924 DEBUG:boto:Token: None 2024-04-12T04:10:44.924 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.925 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8= 2024-04-12T04:10:44.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8=', 'Content-Length': '0'} 2024-04-12T04:10:44.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080d721a6d46a3d53-006618b444-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.927 DEBUG:boto:b'dkyges-61obj1161000falseobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.928 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.928 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.928 DEBUG:boto:path=/dkyges-61/?prefix=obj116 2024-04-12T04:10:44.928 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj116 2024-04-12T04:10:44.928 DEBUG:boto:Method: GET 2024-04-12T04:10:44.928 DEBUG:boto:Path: /dkyges-61/?prefix=obj116 2024-04-12T04:10:44.928 DEBUG:boto:Data: 2024-04-12T04:10:44.928 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.928 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.928 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.928 DEBUG:boto:Params: {} 2024-04-12T04:10:44.929 DEBUG:boto:Token: None 2024-04-12T04:10:44.929 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.929 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8= 2024-04-12T04:10:44.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8=', 'Content-Length': '0'} 2024-04-12T04:10:44.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000933b0dd14726959f-006618b444-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.974 DEBUG:boto:b'dkyges-61obj1161000falseobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.975 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:10:44.975 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:10:44.975 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.975 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:10:44.975 DEBUG:boto:Data: 2024-04-12T04:10:44.975 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.975 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.975 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.975 DEBUG:boto:Params: {} 2024-04-12T04:10:44.975 DEBUG:boto:Token: None 2024-04-12T04:10:44.976 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj116 2024-04-12T04:10:44.976 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BsLpPlccfkoKNuw1ZkKw4AWCMbo= 2024-04-12T04:10:44.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BsLpPlccfkoKNuw1ZkKw4AWCMbo=', 'Content-Length': '0'} 2024-04-12T04:10:44.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba0e267a85ca9af1-006618b444-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.978 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:10:44.978 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:10:44.978 DEBUG:boto:Method: HEAD 2024-04-12T04:10:44.978 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:10:44.978 DEBUG:boto:Data: 2024-04-12T04:10:44.978 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.978 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.978 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.979 DEBUG:boto:Params: {} 2024-04-12T04:10:44.979 DEBUG:boto:Token: None 2024-04-12T04:10:44.979 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj116 2024-04-12T04:10:44.979 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BsLpPlccfkoKNuw1ZkKw4AWCMbo= 2024-04-12T04:10:44.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BsLpPlccfkoKNuw1ZkKw4AWCMbo=', 'Content-Length': '0'} 2024-04-12T04:10:44.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007281e480117b4ab2-006618b444-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:10:44.982 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:10:44.982 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:10:44.982 DEBUG:boto:Method: GET 2024-04-12T04:10:44.982 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:10:44.982 DEBUG:boto:Data: 2024-04-12T04:10:44.982 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.982 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.982 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.982 DEBUG:boto:Params: {} 2024-04-12T04:10:44.982 DEBUG:boto:Token: None 2024-04-12T04:10:44.983 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj116 2024-04-12T04:10:44.983 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X0xRWID3XbSujtKjYmwWLAH6aVg= 2024-04-12T04:10:44.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X0xRWID3XbSujtKjYmwWLAH6aVg=', 'Content-Length': '0'} 2024-04-12T04:10:44.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf50268870e66218-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.985 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:10:44.985 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:10:44.985 DEBUG:boto:Method: GET 2024-04-12T04:10:44.985 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:10:44.985 DEBUG:boto:Data: 2024-04-12T04:10:44.985 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.985 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.985 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.986 DEBUG:boto:Params: {} 2024-04-12T04:10:44.986 DEBUG:boto:Token: None 2024-04-12T04:10:44.986 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/obj116 2024-04-12T04:10:44.986 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X0xRWID3XbSujtKjYmwWLAH6aVg= 2024-04-12T04:10:44.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X0xRWID3XbSujtKjYmwWLAH6aVg=', 'Content-Length': '0'} 2024-04-12T04:10:44.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceacdfbeb47a8d0c-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.988 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.988 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.988 DEBUG:boto:path=/dkyges-61/?prefix=obj116 2024-04-12T04:10:44.989 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj116 2024-04-12T04:10:44.989 DEBUG:boto:Method: GET 2024-04-12T04:10:44.989 DEBUG:boto:Path: /dkyges-61/?prefix=obj116 2024-04-12T04:10:44.989 DEBUG:boto:Data: 2024-04-12T04:10:44.989 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.989 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:44.989 DEBUG:boto:Port: 8001 2024-04-12T04:10:44.989 DEBUG:boto:Params: {} 2024-04-12T04:10:44.989 DEBUG:boto:Token: None 2024-04-12T04:10:44.989 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.990 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8= 2024-04-12T04:10:44.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8=', 'Content-Length': '0'} 2024-04-12T04:10:44.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ac322f9374f3cba-006618b445-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.992 DEBUG:boto:b'dkyges-61obj1161000falseobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.993 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:44.993 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:44.993 DEBUG:boto:path=/dkyges-61/?prefix=obj116 2024-04-12T04:10:44.993 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj116 2024-04-12T04:10:44.993 DEBUG:boto:Method: GET 2024-04-12T04:10:44.993 DEBUG:boto:Path: /dkyges-61/?prefix=obj116 2024-04-12T04:10:44.993 DEBUG:boto:Data: 2024-04-12T04:10:44.993 DEBUG:boto:Headers: {} 2024-04-12T04:10:44.993 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:44.993 DEBUG:boto:Port: 8000 2024-04-12T04:10:44.994 DEBUG:boto:Params: {} 2024-04-12T04:10:44.994 DEBUG:boto:Token: None 2024-04-12T04:10:44.994 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:44 GMT /dkyges-61/ 2024-04-12T04:10:44.994 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8= 2024-04-12T04:10:44.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FXbKf8fCavVQgkPYQSrFUZGWxY8=', 'Content-Length': '0'} 2024-04-12T04:10:44.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055a6088b385f3946-006618b445-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:44.998 DEBUG:boto:b'dkyges-61obj1161000falseobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:44.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T04:10:44.999 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:10:44.999 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:10:44.999 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:10:44.999 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:10:44.999 DEBUG:boto:Method: GET 2024-04-12T04:10:44.999 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:10:44.999 DEBUG:boto:Data: 2024-04-12T04:10:44.999 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.000 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.000 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.000 DEBUG:boto:Params: {} 2024-04-12T04:10:45.000 DEBUG:boto:Token: None 2024-04-12T04:10:45.000 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.000 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hf4DoIJUhLSKSPXevnNPZ4GixCA= 2024-04-12T04:10:45.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hf4DoIJUhLSKSPXevnNPZ4GixCA=', 'Content-Length': '0'} 2024-04-12T04:10:45.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047b51b9c677c60e3-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.002 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:10:45.002 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:10:45.002 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.002 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.002 DEBUG:boto:Method: GET 2024-04-12T04:10:45.002 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.003 DEBUG:boto:Data: 2024-04-12T04:10:45.003 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.003 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.003 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.003 DEBUG:boto:Params: {} 2024-04-12T04:10:45.003 DEBUG:boto:Token: None 2024-04-12T04:10:45.003 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.003 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hf4DoIJUhLSKSPXevnNPZ4GixCA= 2024-04-12T04:10:45.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hf4DoIJUhLSKSPXevnNPZ4GixCA=', 'Content-Length': '0'} 2024-04-12T04:10:45.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6ad33c44f920167-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.005 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:10:45.006 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:10:45.006 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.006 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.006 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.006 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.006 DEBUG:boto:Data: 2024-04-12T04:10:45.006 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.006 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.006 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.006 DEBUG:boto:Params: {} 2024-04-12T04:10:45.006 DEBUG:boto:Token: None 2024-04-12T04:10:45.007 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.007 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OjGbHDFZ8UxqZlqF03yCjwyTubA= 2024-04-12T04:10:45.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OjGbHDFZ8UxqZlqF03yCjwyTubA=', 'Content-Length': '0'} 2024-04-12T04:10:45.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e10abf608d0d6c5d-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.009 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:10:45.009 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:10:45.009 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.009 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.009 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.009 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.009 DEBUG:boto:Data: 2024-04-12T04:10:45.009 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.009 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.009 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.009 DEBUG:boto:Params: {} 2024-04-12T04:10:45.009 DEBUG:boto:Token: None 2024-04-12T04:10:45.010 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.010 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OjGbHDFZ8UxqZlqF03yCjwyTubA= 2024-04-12T04:10:45.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OjGbHDFZ8UxqZlqF03yCjwyTubA=', 'Content-Length': '0'} 2024-04-12T04:10:45.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcb93e3f58694ba0-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T04:10:45.012 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:10:45.012 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:10:45.012 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.012 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.012 DEBUG:boto:Method: GET 2024-04-12T04:10:45.012 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.012 DEBUG:boto:Data: 2024-04-12T04:10:45.012 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.012 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.012 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.013 DEBUG:boto:Params: {} 2024-04-12T04:10:45.013 DEBUG:boto:Token: None 2024-04-12T04:10:45.013 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.013 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hf4DoIJUhLSKSPXevnNPZ4GixCA= 2024-04-12T04:10:45.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hf4DoIJUhLSKSPXevnNPZ4GixCA=', 'Content-Length': '0'} 2024-04-12T04:10:45.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f09c7b0c20367e68-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.015 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:10:45.015 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:10:45.015 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.015 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.015 DEBUG:boto:Method: GET 2024-04-12T04:10:45.015 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.015 DEBUG:boto:Data: 2024-04-12T04:10:45.015 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.015 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.015 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.016 DEBUG:boto:Params: {} 2024-04-12T04:10:45.016 DEBUG:boto:Token: None 2024-04-12T04:10:45.016 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:10:45.016 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hf4DoIJUhLSKSPXevnNPZ4GixCA= 2024-04-12T04:10:45.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hf4DoIJUhLSKSPXevnNPZ4GixCA=', 'Content-Length': '0'} 2024-04-12T04:10:45.018 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060d10a3df4ac19d3-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.018 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.018 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.019 DEBUG:boto:path=/dkyges-61/?prefix=obj124 2024-04-12T04:10:45.019 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj124 2024-04-12T04:10:45.019 DEBUG:boto:Method: GET 2024-04-12T04:10:45.019 DEBUG:boto:Path: /dkyges-61/?prefix=obj124 2024-04-12T04:10:45.019 DEBUG:boto:Data: 2024-04-12T04:10:45.019 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.019 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.019 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.019 DEBUG:boto:Params: {} 2024-04-12T04:10:45.019 DEBUG:boto:Token: None 2024-04-12T04:10:45.019 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.020 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a7207ad5ee65a34-006618b445-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.062 DEBUG:boto:b'dkyges-61obj1241000falseobj1242024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.063 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.063 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.063 DEBUG:boto:path=/dkyges-61/?prefix=obj124 2024-04-12T04:10:45.063 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj124 2024-04-12T04:10:45.063 DEBUG:boto:Method: GET 2024-04-12T04:10:45.063 DEBUG:boto:Path: /dkyges-61/?prefix=obj124 2024-04-12T04:10:45.063 DEBUG:boto:Data: 2024-04-12T04:10:45.063 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.063 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.063 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.063 DEBUG:boto:Params: {} 2024-04-12T04:10:45.063 DEBUG:boto:Token: None 2024-04-12T04:10:45.064 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.064 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8e936824a515415-006618b445-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.068 DEBUG:boto:b'dkyges-61obj1241000falseobj1242024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.068 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:10:45.068 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:10:45.068 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.068 DEBUG:boto:Path: /dkyges-61/obj124 2024-04-12T04:10:45.069 DEBUG:boto:Data: 2024-04-12T04:10:45.069 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.069 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.069 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.069 DEBUG:boto:Params: {} 2024-04-12T04:10:45.069 DEBUG:boto:Token: None 2024-04-12T04:10:45.069 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj124 2024-04-12T04:10:45.069 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IYp8qkE8ef6AP+YnALNqxGdoxl4= 2024-04-12T04:10:45.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IYp8qkE8ef6AP+YnALNqxGdoxl4=', 'Content-Length': '0'} 2024-04-12T04:10:45.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000068b18fb91fe338f-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.071 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:10:45.071 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:10:45.071 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.072 DEBUG:boto:Path: /dkyges-61/obj124 2024-04-12T04:10:45.072 DEBUG:boto:Data: 2024-04-12T04:10:45.072 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.072 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.072 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.072 DEBUG:boto:Params: {} 2024-04-12T04:10:45.072 DEBUG:boto:Token: None 2024-04-12T04:10:45.072 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj124 2024-04-12T04:10:45.072 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IYp8qkE8ef6AP+YnALNqxGdoxl4= 2024-04-12T04:10:45.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IYp8qkE8ef6AP+YnALNqxGdoxl4=', 'Content-Length': '0'} 2024-04-12T04:10:45.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4d60cc12e6a2fa6-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T04:10:45.075 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:10:45.075 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:10:45.075 DEBUG:boto:Method: GET 2024-04-12T04:10:45.075 DEBUG:boto:Path: /dkyges-61/obj124 2024-04-12T04:10:45.075 DEBUG:boto:Data: 2024-04-12T04:10:45.075 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.075 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.075 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.075 DEBUG:boto:Params: {} 2024-04-12T04:10:45.075 DEBUG:boto:Token: None 2024-04-12T04:10:45.075 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj124 2024-04-12T04:10:45.076 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zgL1XysVp36yMixB8+1+oVmHv0s= 2024-04-12T04:10:45.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zgL1XysVp36yMixB8+1+oVmHv0s=', 'Content-Length': '0'} 2024-04-12T04:10:45.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000879ec7f8743c4dfc-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.077 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:10:45.077 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:10:45.078 DEBUG:boto:Method: GET 2024-04-12T04:10:45.078 DEBUG:boto:Path: /dkyges-61/obj124 2024-04-12T04:10:45.078 DEBUG:boto:Data: 2024-04-12T04:10:45.078 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.078 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.078 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.078 DEBUG:boto:Params: {} 2024-04-12T04:10:45.078 DEBUG:boto:Token: None 2024-04-12T04:10:45.078 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj124 2024-04-12T04:10:45.078 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zgL1XysVp36yMixB8+1+oVmHv0s= 2024-04-12T04:10:45.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zgL1XysVp36yMixB8+1+oVmHv0s=', 'Content-Length': '0'} 2024-04-12T04:10:45.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eca4a8ccaa10b2c-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.122 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.122 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.123 DEBUG:boto:path=/dkyges-61/?prefix=obj124 2024-04-12T04:10:45.123 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj124 2024-04-12T04:10:45.123 DEBUG:boto:Method: GET 2024-04-12T04:10:45.123 DEBUG:boto:Path: /dkyges-61/?prefix=obj124 2024-04-12T04:10:45.123 DEBUG:boto:Data: 2024-04-12T04:10:45.123 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.123 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.123 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.123 DEBUG:boto:Params: {} 2024-04-12T04:10:45.123 DEBUG:boto:Token: None 2024-04-12T04:10:45.123 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.123 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e29bee4f4c0ad5f-006618b445-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.126 DEBUG:boto:b'dkyges-61obj1241000falseobj1242024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.127 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.127 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.127 DEBUG:boto:path=/dkyges-61/?prefix=obj124 2024-04-12T04:10:45.127 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj124 2024-04-12T04:10:45.127 DEBUG:boto:Method: GET 2024-04-12T04:10:45.127 DEBUG:boto:Path: /dkyges-61/?prefix=obj124 2024-04-12T04:10:45.127 DEBUG:boto:Data: 2024-04-12T04:10:45.127 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.128 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.128 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.128 DEBUG:boto:Params: {} 2024-04-12T04:10:45.128 DEBUG:boto:Token: None 2024-04-12T04:10:45.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.128 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee627e404f1746cd-006618b445-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.133 DEBUG:boto:b'dkyges-61obj1241000falseobj1242024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:10:45.133 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:10:45.134 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:10:45.134 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.134 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.134 DEBUG:boto:Method: GET 2024-04-12T04:10:45.134 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.134 DEBUG:boto:Data: 2024-04-12T04:10:45.134 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.134 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.134 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.134 DEBUG:boto:Params: {} 2024-04-12T04:10:45.135 DEBUG:boto:Token: None 2024-04-12T04:10:45.135 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.135 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Cc4EYbIMei/VDCdQr16rljJzYd0= 2024-04-12T04:10:45.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Cc4EYbIMei/VDCdQr16rljJzYd0=', 'Content-Length': '0'} 2024-04-12T04:10:45.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c659e8edd8323918-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.182 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:10:45.182 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:10:45.183 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.183 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.183 DEBUG:boto:Method: GET 2024-04-12T04:10:45.183 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.183 DEBUG:boto:Data: 2024-04-12T04:10:45.183 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.183 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.183 DEBUG:boto:Params: {} 2024-04-12T04:10:45.183 DEBUG:boto:Token: None 2024-04-12T04:10:45.184 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.184 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Cc4EYbIMei/VDCdQr16rljJzYd0= 2024-04-12T04:10:45.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Cc4EYbIMei/VDCdQr16rljJzYd0=', 'Content-Length': '0'} 2024-04-12T04:10:45.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b20caa8872efe8c7-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.186 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:10:45.187 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:10:45.187 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.187 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.187 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.187 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.187 DEBUG:boto:Data: 2024-04-12T04:10:45.187 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.187 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.187 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.187 DEBUG:boto:Params: {} 2024-04-12T04:10:45.187 DEBUG:boto:Token: None 2024-04-12T04:10:45.188 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.188 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dZSiPzGi9z003wLJaEBPDYmMHkE= 2024-04-12T04:10:45.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dZSiPzGi9z003wLJaEBPDYmMHkE=', 'Content-Length': '0'} 2024-04-12T04:10:45.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057bba39ee0f9098c-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.190 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:10:45.190 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:10:45.190 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.190 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.190 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.190 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.190 DEBUG:boto:Data: 2024-04-12T04:10:45.190 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.191 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.191 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.191 DEBUG:boto:Params: {} 2024-04-12T04:10:45.191 DEBUG:boto:Token: None 2024-04-12T04:10:45.191 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.191 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dZSiPzGi9z003wLJaEBPDYmMHkE= 2024-04-12T04:10:45.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dZSiPzGi9z003wLJaEBPDYmMHkE=', 'Content-Length': '0'} 2024-04-12T04:10:45.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fffe4ae11fe7cf3-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:10:45.193 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:10:45.194 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:10:45.194 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.194 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.194 DEBUG:boto:Method: GET 2024-04-12T04:10:45.194 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.194 DEBUG:boto:Data: 2024-04-12T04:10:45.194 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.194 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.194 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.194 DEBUG:boto:Params: {} 2024-04-12T04:10:45.194 DEBUG:boto:Token: None 2024-04-12T04:10:45.195 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.195 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Cc4EYbIMei/VDCdQr16rljJzYd0= 2024-04-12T04:10:45.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Cc4EYbIMei/VDCdQr16rljJzYd0=', 'Content-Length': '0'} 2024-04-12T04:10:45.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7fc1276ec53a96e-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.197 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:10:45.197 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:10:45.197 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.197 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.197 DEBUG:boto:Method: GET 2024-04-12T04:10:45.198 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.198 DEBUG:boto:Data: 2024-04-12T04:10:45.198 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.198 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.198 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.198 DEBUG:boto:Params: {} 2024-04-12T04:10:45.198 DEBUG:boto:Token: None 2024-04-12T04:10:45.198 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:10:45.198 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Cc4EYbIMei/VDCdQr16rljJzYd0= 2024-04-12T04:10:45.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Cc4EYbIMei/VDCdQr16rljJzYd0=', 'Content-Length': '0'} 2024-04-12T04:10:45.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c57ad922f2b9993-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.201 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.201 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.201 DEBUG:boto:path=/dkyges-61/?prefix=obj145 2024-04-12T04:10:45.201 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj145 2024-04-12T04:10:45.201 DEBUG:boto:Method: GET 2024-04-12T04:10:45.201 DEBUG:boto:Path: /dkyges-61/?prefix=obj145 2024-04-12T04:10:45.202 DEBUG:boto:Data: 2024-04-12T04:10:45.202 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.202 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.202 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.202 DEBUG:boto:Params: {} 2024-04-12T04:10:45.202 DEBUG:boto:Token: None 2024-04-12T04:10:45.202 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.202 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000376580e6744df23f-006618b445-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.205 DEBUG:boto:b'dkyges-61obj1451000falseobj1452024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.206 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.206 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.206 DEBUG:boto:path=/dkyges-61/?prefix=obj145 2024-04-12T04:10:45.206 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj145 2024-04-12T04:10:45.206 DEBUG:boto:Method: GET 2024-04-12T04:10:45.206 DEBUG:boto:Path: /dkyges-61/?prefix=obj145 2024-04-12T04:10:45.206 DEBUG:boto:Data: 2024-04-12T04:10:45.206 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.206 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.206 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.206 DEBUG:boto:Params: {} 2024-04-12T04:10:45.207 DEBUG:boto:Token: None 2024-04-12T04:10:45.207 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.207 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000821aa993407d8e7c-006618b445-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.211 DEBUG:boto:b'dkyges-61obj1451000falseobj1452024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.211 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:10:45.212 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:10:45.212 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.212 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:10:45.212 DEBUG:boto:Data: 2024-04-12T04:10:45.212 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.212 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.212 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.212 DEBUG:boto:Params: {} 2024-04-12T04:10:45.212 DEBUG:boto:Token: None 2024-04-12T04:10:45.212 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj145 2024-04-12T04:10:45.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NlLuipPVfNendJFMlj9B35Rz4yE= 2024-04-12T04:10:45.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NlLuipPVfNendJFMlj9B35Rz4yE=', 'Content-Length': '0'} 2024-04-12T04:10:45.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b5d9a3c8efd51c5-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.215 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:10:45.215 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:10:45.215 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.215 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:10:45.215 DEBUG:boto:Data: 2024-04-12T04:10:45.215 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.215 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.215 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.215 DEBUG:boto:Params: {} 2024-04-12T04:10:45.215 DEBUG:boto:Token: None 2024-04-12T04:10:45.216 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj145 2024-04-12T04:10:45.216 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NlLuipPVfNendJFMlj9B35Rz4yE= 2024-04-12T04:10:45.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NlLuipPVfNendJFMlj9B35Rz4yE=', 'Content-Length': '0'} 2024-04-12T04:10:45.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f008a84bd0c8815-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:10:45.218 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:10:45.218 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:10:45.218 DEBUG:boto:Method: GET 2024-04-12T04:10:45.218 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:10:45.218 DEBUG:boto:Data: 2024-04-12T04:10:45.218 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.218 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.218 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.218 DEBUG:boto:Params: {} 2024-04-12T04:10:45.219 DEBUG:boto:Token: None 2024-04-12T04:10:45.219 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj145 2024-04-12T04:10:45.219 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OpCvZlhB6Q7ABvnOaO0ziWyj0pU= 2024-04-12T04:10:45.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OpCvZlhB6Q7ABvnOaO0ziWyj0pU=', 'Content-Length': '0'} 2024-04-12T04:10:45.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f412b3221f31edf-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.221 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:10:45.221 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:10:45.221 DEBUG:boto:Method: GET 2024-04-12T04:10:45.221 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:10:45.221 DEBUG:boto:Data: 2024-04-12T04:10:45.221 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.221 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.221 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.221 DEBUG:boto:Params: {} 2024-04-12T04:10:45.221 DEBUG:boto:Token: None 2024-04-12T04:10:45.221 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj145 2024-04-12T04:10:45.222 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OpCvZlhB6Q7ABvnOaO0ziWyj0pU= 2024-04-12T04:10:45.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OpCvZlhB6Q7ABvnOaO0ziWyj0pU=', 'Content-Length': '0'} 2024-04-12T04:10:45.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077a20efc82854d71-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.266 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.267 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.267 DEBUG:boto:path=/dkyges-61/?prefix=obj145 2024-04-12T04:10:45.267 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj145 2024-04-12T04:10:45.267 DEBUG:boto:Method: GET 2024-04-12T04:10:45.267 DEBUG:boto:Path: /dkyges-61/?prefix=obj145 2024-04-12T04:10:45.267 DEBUG:boto:Data: 2024-04-12T04:10:45.267 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.267 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.267 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.267 DEBUG:boto:Params: {} 2024-04-12T04:10:45.267 DEBUG:boto:Token: None 2024-04-12T04:10:45.268 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.268 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000227b2e2400386912-006618b445-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.270 DEBUG:boto:b'dkyges-61obj1451000falseobj1452024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.271 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.271 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.271 DEBUG:boto:path=/dkyges-61/?prefix=obj145 2024-04-12T04:10:45.271 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj145 2024-04-12T04:10:45.271 DEBUG:boto:Method: GET 2024-04-12T04:10:45.271 DEBUG:boto:Path: /dkyges-61/?prefix=obj145 2024-04-12T04:10:45.271 DEBUG:boto:Data: 2024-04-12T04:10:45.271 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.272 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.272 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.272 DEBUG:boto:Params: {} 2024-04-12T04:10:45.272 DEBUG:boto:Token: None 2024-04-12T04:10:45.272 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.272 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea1c5f2b30ea45b3-006618b445-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.277 DEBUG:boto:b'dkyges-61obj1451000falseobj1452024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T04:10:45.277 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:10:45.277 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:10:45.277 DEBUG:boto:path=/dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.278 DEBUG:boto:auth_path=/dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.278 DEBUG:boto:Method: GET 2024-04-12T04:10:45.278 DEBUG:boto:Path: /dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.278 DEBUG:boto:Data: 2024-04-12T04:10:45.278 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.278 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.278 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.278 DEBUG:boto:Params: {} 2024-04-12T04:10:45.278 DEBUG:boto:Token: None 2024-04-12T04:10:45.278 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.279 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U//iPnwg7ZVzoSJNBhKSwKOBQgk= 2024-04-12T04:10:45.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U//iPnwg7ZVzoSJNBhKSwKOBQgk=', 'Content-Length': '0'} 2024-04-12T04:10:45.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002789bba8e8d865b4-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.326 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:10:45.326 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:10:45.327 DEBUG:boto:path=/dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.327 DEBUG:boto:auth_path=/dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.327 DEBUG:boto:Method: GET 2024-04-12T04:10:45.327 DEBUG:boto:Path: /dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.327 DEBUG:boto:Data: 2024-04-12T04:10:45.327 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.327 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.327 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.327 DEBUG:boto:Params: {} 2024-04-12T04:10:45.327 DEBUG:boto:Token: None 2024-04-12T04:10:45.328 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.328 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U//iPnwg7ZVzoSJNBhKSwKOBQgk= 2024-04-12T04:10:45.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U//iPnwg7ZVzoSJNBhKSwKOBQgk=', 'Content-Length': '0'} 2024-04-12T04:10:45.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005105eb3b7316cb67-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.330 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:10:45.330 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:10:45.330 DEBUG:boto:path=/dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.331 DEBUG:boto:auth_path=/dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.331 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.331 DEBUG:boto:Path: /dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.331 DEBUG:boto:Data: 2024-04-12T04:10:45.331 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.331 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.331 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.331 DEBUG:boto:Params: {} 2024-04-12T04:10:45.331 DEBUG:boto:Token: None 2024-04-12T04:10:45.331 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.332 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Mfio51HkTNSF2Nk1xxRmxZPp6HY= 2024-04-12T04:10:45.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Mfio51HkTNSF2Nk1xxRmxZPp6HY=', 'Content-Length': '0'} 2024-04-12T04:10:45.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041f84e0321b36021-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.334 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:10:45.334 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:10:45.334 DEBUG:boto:path=/dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.334 DEBUG:boto:auth_path=/dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.334 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.334 DEBUG:boto:Path: /dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.334 DEBUG:boto:Data: 2024-04-12T04:10:45.334 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.334 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.335 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.335 DEBUG:boto:Params: {} 2024-04-12T04:10:45.335 DEBUG:boto:Token: None 2024-04-12T04:10:45.335 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.335 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Mfio51HkTNSF2Nk1xxRmxZPp6HY= 2024-04-12T04:10:45.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Mfio51HkTNSF2Nk1xxRmxZPp6HY=', 'Content-Length': '0'} 2024-04-12T04:10:45.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000358729b3a369a296-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T04:10:45.337 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:10:45.337 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:10:45.337 DEBUG:boto:path=/dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.337 DEBUG:boto:auth_path=/dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.337 DEBUG:boto:Method: GET 2024-04-12T04:10:45.337 DEBUG:boto:Path: /dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.338 DEBUG:boto:Data: 2024-04-12T04:10:45.338 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.338 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.338 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.338 DEBUG:boto:Params: {} 2024-04-12T04:10:45.338 DEBUG:boto:Token: None 2024-04-12T04:10:45.338 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.338 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U//iPnwg7ZVzoSJNBhKSwKOBQgk= 2024-04-12T04:10:45.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U//iPnwg7ZVzoSJNBhKSwKOBQgk=', 'Content-Length': '0'} 2024-04-12T04:10:45.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fad5a70ad640bd9b-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.340 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:10:45.340 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:10:45.340 DEBUG:boto:path=/dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.340 DEBUG:boto:auth_path=/dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.341 DEBUG:boto:Method: GET 2024-04-12T04:10:45.341 DEBUG:boto:Path: /dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.341 DEBUG:boto:Data: 2024-04-12T04:10:45.341 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.341 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.341 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.341 DEBUG:boto:Params: {} 2024-04-12T04:10:45.341 DEBUG:boto:Token: None 2024-04-12T04:10:45.341 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj148?versionId=null 2024-04-12T04:10:45.341 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U//iPnwg7ZVzoSJNBhKSwKOBQgk= 2024-04-12T04:10:45.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U//iPnwg7ZVzoSJNBhKSwKOBQgk=', 'Content-Length': '0'} 2024-04-12T04:10:45.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c1ba48e25338fb5-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.344 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.344 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.344 DEBUG:boto:path=/dkyges-61/?prefix=obj148 2024-04-12T04:10:45.344 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj148 2024-04-12T04:10:45.344 DEBUG:boto:Method: GET 2024-04-12T04:10:45.344 DEBUG:boto:Path: /dkyges-61/?prefix=obj148 2024-04-12T04:10:45.344 DEBUG:boto:Data: 2024-04-12T04:10:45.345 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.345 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.345 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.345 DEBUG:boto:Params: {} 2024-04-12T04:10:45.345 DEBUG:boto:Token: None 2024-04-12T04:10:45.345 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.345 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dcfd93cda40590b-006618b445-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.349 DEBUG:boto:b'dkyges-61obj1481000falseobj1482024-04-12T04:09:24.114Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.350 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.350 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.350 DEBUG:boto:path=/dkyges-61/?prefix=obj148 2024-04-12T04:10:45.350 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj148 2024-04-12T04:10:45.350 DEBUG:boto:Method: GET 2024-04-12T04:10:45.350 DEBUG:boto:Path: /dkyges-61/?prefix=obj148 2024-04-12T04:10:45.350 DEBUG:boto:Data: 2024-04-12T04:10:45.351 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.351 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.351 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.351 DEBUG:boto:Params: {} 2024-04-12T04:10:45.351 DEBUG:boto:Token: None 2024-04-12T04:10:45.351 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.351 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000324fd987e7663037-006618b445-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.356 DEBUG:boto:b'dkyges-61obj1481000falseobj1482024-04-12T04:09:24.114Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.356 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:10:45.356 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:10:45.356 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.356 DEBUG:boto:Path: /dkyges-61/obj148 2024-04-12T04:10:45.356 DEBUG:boto:Data: 2024-04-12T04:10:45.356 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.357 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.357 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.357 DEBUG:boto:Params: {} 2024-04-12T04:10:45.357 DEBUG:boto:Token: None 2024-04-12T04:10:45.357 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj148 2024-04-12T04:10:45.357 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tWLk7uOQvbC6N0Q/YH4gHkd+4vs= 2024-04-12T04:10:45.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tWLk7uOQvbC6N0Q/YH4gHkd+4vs=', 'Content-Length': '0'} 2024-04-12T04:10:45.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b65ed345bd88ac7d-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.359 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:10:45.359 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:10:45.359 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.359 DEBUG:boto:Path: /dkyges-61/obj148 2024-04-12T04:10:45.359 DEBUG:boto:Data: 2024-04-12T04:10:45.359 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.360 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.360 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.360 DEBUG:boto:Params: {} 2024-04-12T04:10:45.360 DEBUG:boto:Token: None 2024-04-12T04:10:45.360 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj148 2024-04-12T04:10:45.360 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tWLk7uOQvbC6N0Q/YH4gHkd+4vs= 2024-04-12T04:10:45.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tWLk7uOQvbC6N0Q/YH4gHkd+4vs=', 'Content-Length': '0'} 2024-04-12T04:10:45.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000296ea1a796b41eb6-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T04:10:45.362 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:10:45.363 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:10:45.363 DEBUG:boto:Method: GET 2024-04-12T04:10:45.363 DEBUG:boto:Path: /dkyges-61/obj148 2024-04-12T04:10:45.363 DEBUG:boto:Data: 2024-04-12T04:10:45.363 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.363 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.363 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.363 DEBUG:boto:Params: {} 2024-04-12T04:10:45.363 DEBUG:boto:Token: None 2024-04-12T04:10:45.363 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj148 2024-04-12T04:10:45.364 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xMLhPIH2mXOXmcPYG74XgZXWzrY= 2024-04-12T04:10:45.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xMLhPIH2mXOXmcPYG74XgZXWzrY=', 'Content-Length': '0'} 2024-04-12T04:10:45.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000725e314f427ac0d-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.406 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:10:45.406 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:10:45.407 DEBUG:boto:Method: GET 2024-04-12T04:10:45.407 DEBUG:boto:Path: /dkyges-61/obj148 2024-04-12T04:10:45.407 DEBUG:boto:Data: 2024-04-12T04:10:45.407 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.407 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.407 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.407 DEBUG:boto:Params: {} 2024-04-12T04:10:45.407 DEBUG:boto:Token: None 2024-04-12T04:10:45.407 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj148 2024-04-12T04:10:45.408 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xMLhPIH2mXOXmcPYG74XgZXWzrY= 2024-04-12T04:10:45.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xMLhPIH2mXOXmcPYG74XgZXWzrY=', 'Content-Length': '0'} 2024-04-12T04:10:45.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005e2a8b62fda2e4b-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.410 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.410 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.410 DEBUG:boto:path=/dkyges-61/?prefix=obj148 2024-04-12T04:10:45.410 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj148 2024-04-12T04:10:45.410 DEBUG:boto:Method: GET 2024-04-12T04:10:45.410 DEBUG:boto:Path: /dkyges-61/?prefix=obj148 2024-04-12T04:10:45.410 DEBUG:boto:Data: 2024-04-12T04:10:45.410 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.410 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.411 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.411 DEBUG:boto:Params: {} 2024-04-12T04:10:45.411 DEBUG:boto:Token: None 2024-04-12T04:10:45.411 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.411 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a6d02e2ebfe61f7-006618b445-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.414 DEBUG:boto:b'dkyges-61obj1481000falseobj1482024-04-12T04:09:24.114Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.414 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.414 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.414 DEBUG:boto:path=/dkyges-61/?prefix=obj148 2024-04-12T04:10:45.414 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj148 2024-04-12T04:10:45.415 DEBUG:boto:Method: GET 2024-04-12T04:10:45.415 DEBUG:boto:Path: /dkyges-61/?prefix=obj148 2024-04-12T04:10:45.415 DEBUG:boto:Data: 2024-04-12T04:10:45.415 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.415 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.415 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.415 DEBUG:boto:Params: {} 2024-04-12T04:10:45.415 DEBUG:boto:Token: None 2024-04-12T04:10:45.415 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.415 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a39fbf160b016eb-006618b445-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.420 DEBUG:boto:b'dkyges-61obj1481000falseobj1482024-04-12T04:09:24.114Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T04:10:45.420 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:10:45.420 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:10:45.420 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.421 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.421 DEBUG:boto:Method: GET 2024-04-12T04:10:45.421 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.421 DEBUG:boto:Data: 2024-04-12T04:10:45.421 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.421 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.421 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.421 DEBUG:boto:Params: {} 2024-04-12T04:10:45.421 DEBUG:boto:Token: None 2024-04-12T04:10:45.421 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.421 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pFlJP/C/cMx4ew5l73UUTqAEu6s= 2024-04-12T04:10:45.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pFlJP/C/cMx4ew5l73UUTqAEu6s=', 'Content-Length': '0'} 2024-04-12T04:10:45.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ffa99d5bb29bec8-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.424 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:10:45.424 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:10:45.424 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.424 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.424 DEBUG:boto:Method: GET 2024-04-12T04:10:45.424 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.424 DEBUG:boto:Data: 2024-04-12T04:10:45.424 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.424 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.424 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.424 DEBUG:boto:Params: {} 2024-04-12T04:10:45.425 DEBUG:boto:Token: None 2024-04-12T04:10:45.425 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.425 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pFlJP/C/cMx4ew5l73UUTqAEu6s= 2024-04-12T04:10:45.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pFlJP/C/cMx4ew5l73UUTqAEu6s=', 'Content-Length': '0'} 2024-04-12T04:10:45.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ce7876555388053-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.427 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:10:45.427 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:10:45.427 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.427 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.427 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.428 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.428 DEBUG:boto:Data: 2024-04-12T04:10:45.428 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.428 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.428 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.428 DEBUG:boto:Params: {} 2024-04-12T04:10:45.428 DEBUG:boto:Token: None 2024-04-12T04:10:45.428 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.428 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ck2uzBTxt8L7n3TcNrXjOkZIxP0= 2024-04-12T04:10:45.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ck2uzBTxt8L7n3TcNrXjOkZIxP0=', 'Content-Length': '0'} 2024-04-12T04:10:45.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bfbe2047b7907bf-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.430 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:10:45.430 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:10:45.430 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.431 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.431 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.431 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.431 DEBUG:boto:Data: 2024-04-12T04:10:45.431 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.431 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.431 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.431 DEBUG:boto:Params: {} 2024-04-12T04:10:45.431 DEBUG:boto:Token: None 2024-04-12T04:10:45.431 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.431 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ck2uzBTxt8L7n3TcNrXjOkZIxP0= 2024-04-12T04:10:45.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ck2uzBTxt8L7n3TcNrXjOkZIxP0=', 'Content-Length': '0'} 2024-04-12T04:10:45.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0816993be73099d-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T04:10:45.434 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:10:45.434 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:10:45.434 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.434 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.434 DEBUG:boto:Method: GET 2024-04-12T04:10:45.434 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.434 DEBUG:boto:Data: 2024-04-12T04:10:45.435 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.435 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.435 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.435 DEBUG:boto:Params: {} 2024-04-12T04:10:45.435 DEBUG:boto:Token: None 2024-04-12T04:10:45.435 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.435 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pFlJP/C/cMx4ew5l73UUTqAEu6s= 2024-04-12T04:10:45.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pFlJP/C/cMx4ew5l73UUTqAEu6s=', 'Content-Length': '0'} 2024-04-12T04:10:45.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7ddeab67544aa2a-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.437 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:10:45.437 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:10:45.437 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.437 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.437 DEBUG:boto:Method: GET 2024-04-12T04:10:45.438 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.438 DEBUG:boto:Data: 2024-04-12T04:10:45.438 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.438 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.438 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.438 DEBUG:boto:Params: {} 2024-04-12T04:10:45.438 DEBUG:boto:Token: None 2024-04-12T04:10:45.438 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:10:45.438 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pFlJP/C/cMx4ew5l73UUTqAEu6s= 2024-04-12T04:10:45.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pFlJP/C/cMx4ew5l73UUTqAEu6s=', 'Content-Length': '0'} 2024-04-12T04:10:45.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d00069f35659a0bb-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.440 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.440 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.441 DEBUG:boto:path=/dkyges-61/?prefix=obj155 2024-04-12T04:10:45.441 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj155 2024-04-12T04:10:45.441 DEBUG:boto:Method: GET 2024-04-12T04:10:45.441 DEBUG:boto:Path: /dkyges-61/?prefix=obj155 2024-04-12T04:10:45.441 DEBUG:boto:Data: 2024-04-12T04:10:45.441 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.441 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.441 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.441 DEBUG:boto:Params: {} 2024-04-12T04:10:45.441 DEBUG:boto:Token: None 2024-04-12T04:10:45.441 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.442 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e7e24ec9d5dbd80-006618b445-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.444 DEBUG:boto:b'dkyges-61obj1551000falseobj1552024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.445 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.445 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.445 DEBUG:boto:path=/dkyges-61/?prefix=obj155 2024-04-12T04:10:45.445 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj155 2024-04-12T04:10:45.445 DEBUG:boto:Method: GET 2024-04-12T04:10:45.445 DEBUG:boto:Path: /dkyges-61/?prefix=obj155 2024-04-12T04:10:45.445 DEBUG:boto:Data: 2024-04-12T04:10:45.445 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.445 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.445 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.445 DEBUG:boto:Params: {} 2024-04-12T04:10:45.446 DEBUG:boto:Token: None 2024-04-12T04:10:45.446 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.446 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d53e1140e6c8ab02-006618b445-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.450 DEBUG:boto:b'dkyges-61obj1551000falseobj1552024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.451 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:10:45.451 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:10:45.451 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.451 DEBUG:boto:Path: /dkyges-61/obj155 2024-04-12T04:10:45.451 DEBUG:boto:Data: 2024-04-12T04:10:45.451 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.451 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.451 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.451 DEBUG:boto:Params: {} 2024-04-12T04:10:45.451 DEBUG:boto:Token: None 2024-04-12T04:10:45.452 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj155 2024-04-12T04:10:45.452 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j6/J7V8O/uLRxclyFBq2G9OBjz4= 2024-04-12T04:10:45.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j6/J7V8O/uLRxclyFBq2G9OBjz4=', 'Content-Length': '0'} 2024-04-12T04:10:45.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099374296910eabb4-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.454 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:10:45.454 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:10:45.454 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.454 DEBUG:boto:Path: /dkyges-61/obj155 2024-04-12T04:10:45.454 DEBUG:boto:Data: 2024-04-12T04:10:45.454 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.454 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.454 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.454 DEBUG:boto:Params: {} 2024-04-12T04:10:45.455 DEBUG:boto:Token: None 2024-04-12T04:10:45.455 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj155 2024-04-12T04:10:45.455 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j6/J7V8O/uLRxclyFBq2G9OBjz4= 2024-04-12T04:10:45.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j6/J7V8O/uLRxclyFBq2G9OBjz4=', 'Content-Length': '0'} 2024-04-12T04:10:45.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000783c21f086c1a497-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T04:10:45.457 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:10:45.457 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:10:45.457 DEBUG:boto:Method: GET 2024-04-12T04:10:45.457 DEBUG:boto:Path: /dkyges-61/obj155 2024-04-12T04:10:45.457 DEBUG:boto:Data: 2024-04-12T04:10:45.457 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.457 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.457 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.457 DEBUG:boto:Params: {} 2024-04-12T04:10:45.457 DEBUG:boto:Token: None 2024-04-12T04:10:45.458 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj155 2024-04-12T04:10:45.458 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ag63QtZGXx7djUdxampPovUUcz8= 2024-04-12T04:10:45.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ag63QtZGXx7djUdxampPovUUcz8=', 'Content-Length': '0'} 2024-04-12T04:10:45.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1bd79c0b615663c-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.460 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:10:45.460 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:10:45.460 DEBUG:boto:Method: GET 2024-04-12T04:10:45.460 DEBUG:boto:Path: /dkyges-61/obj155 2024-04-12T04:10:45.460 DEBUG:boto:Data: 2024-04-12T04:10:45.460 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.460 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.460 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.460 DEBUG:boto:Params: {} 2024-04-12T04:10:45.460 DEBUG:boto:Token: None 2024-04-12T04:10:45.461 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj155 2024-04-12T04:10:45.461 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ag63QtZGXx7djUdxampPovUUcz8= 2024-04-12T04:10:45.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ag63QtZGXx7djUdxampPovUUcz8=', 'Content-Length': '0'} 2024-04-12T04:10:45.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b698a5ea6509bdc5-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.506 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.506 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.507 DEBUG:boto:path=/dkyges-61/?prefix=obj155 2024-04-12T04:10:45.507 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj155 2024-04-12T04:10:45.507 DEBUG:boto:Method: GET 2024-04-12T04:10:45.507 DEBUG:boto:Path: /dkyges-61/?prefix=obj155 2024-04-12T04:10:45.507 DEBUG:boto:Data: 2024-04-12T04:10:45.507 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.507 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.507 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.507 DEBUG:boto:Params: {} 2024-04-12T04:10:45.507 DEBUG:boto:Token: None 2024-04-12T04:10:45.507 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.508 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3e73e8e95de6a89-006618b445-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.510 DEBUG:boto:b'dkyges-61obj1551000falseobj1552024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.511 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.511 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.511 DEBUG:boto:path=/dkyges-61/?prefix=obj155 2024-04-12T04:10:45.511 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj155 2024-04-12T04:10:45.511 DEBUG:boto:Method: GET 2024-04-12T04:10:45.511 DEBUG:boto:Path: /dkyges-61/?prefix=obj155 2024-04-12T04:10:45.511 DEBUG:boto:Data: 2024-04-12T04:10:45.511 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.511 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.511 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.512 DEBUG:boto:Params: {} 2024-04-12T04:10:45.512 DEBUG:boto:Token: None 2024-04-12T04:10:45.512 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.512 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009467205cab611400-006618b445-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.517 DEBUG:boto:b'dkyges-61obj1551000falseobj1552024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:10:45.517 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:10:45.517 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:10:45.517 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.517 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.518 DEBUG:boto:Method: GET 2024-04-12T04:10:45.518 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.518 DEBUG:boto:Data: 2024-04-12T04:10:45.518 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.518 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.518 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.518 DEBUG:boto:Params: {} 2024-04-12T04:10:45.518 DEBUG:boto:Token: None 2024-04-12T04:10:45.518 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.518 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+8WsKICA029Gwip6PyD2Rh8uKhw= 2024-04-12T04:10:45.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+8WsKICA029Gwip6PyD2Rh8uKhw=', 'Content-Length': '0'} 2024-04-12T04:10:45.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce34afca9132d409-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.566 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:10:45.567 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:10:45.567 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.567 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.567 DEBUG:boto:Method: GET 2024-04-12T04:10:45.567 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.567 DEBUG:boto:Data: 2024-04-12T04:10:45.567 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.567 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.567 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.567 DEBUG:boto:Params: {} 2024-04-12T04:10:45.567 DEBUG:boto:Token: None 2024-04-12T04:10:45.568 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.568 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+8WsKICA029Gwip6PyD2Rh8uKhw= 2024-04-12T04:10:45.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+8WsKICA029Gwip6PyD2Rh8uKhw=', 'Content-Length': '0'} 2024-04-12T04:10:45.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000903c94bcb1a1d37a-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.570 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:10:45.571 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:10:45.571 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.571 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.571 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.571 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.571 DEBUG:boto:Data: 2024-04-12T04:10:45.571 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.571 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.571 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.571 DEBUG:boto:Params: {} 2024-04-12T04:10:45.571 DEBUG:boto:Token: None 2024-04-12T04:10:45.572 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.572 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RECH9FjL0agvmORAS67+Eby1oCk= 2024-04-12T04:10:45.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RECH9FjL0agvmORAS67+Eby1oCk=', 'Content-Length': '0'} 2024-04-12T04:10:45.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a9c30993e060cd2-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.574 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:10:45.574 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:10:45.574 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.574 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.574 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.574 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.575 DEBUG:boto:Data: 2024-04-12T04:10:45.575 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.575 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.575 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.575 DEBUG:boto:Params: {} 2024-04-12T04:10:45.575 DEBUG:boto:Token: None 2024-04-12T04:10:45.575 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.575 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RECH9FjL0agvmORAS67+Eby1oCk= 2024-04-12T04:10:45.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RECH9FjL0agvmORAS67+Eby1oCk=', 'Content-Length': '0'} 2024-04-12T04:10:45.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b693fc6280f0e96-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:10:45.577 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:10:45.578 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:10:45.578 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.578 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.578 DEBUG:boto:Method: GET 2024-04-12T04:10:45.578 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.578 DEBUG:boto:Data: 2024-04-12T04:10:45.578 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.578 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.578 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.578 DEBUG:boto:Params: {} 2024-04-12T04:10:45.579 DEBUG:boto:Token: None 2024-04-12T04:10:45.579 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.579 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+8WsKICA029Gwip6PyD2Rh8uKhw= 2024-04-12T04:10:45.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+8WsKICA029Gwip6PyD2Rh8uKhw=', 'Content-Length': '0'} 2024-04-12T04:10:45.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d4f271379e2b309-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.581 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:10:45.581 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:10:45.581 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.582 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.582 DEBUG:boto:Method: GET 2024-04-12T04:10:45.582 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.582 DEBUG:boto:Data: 2024-04-12T04:10:45.582 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.582 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.582 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.582 DEBUG:boto:Params: {} 2024-04-12T04:10:45.582 DEBUG:boto:Token: None 2024-04-12T04:10:45.582 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:10:45.582 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+8WsKICA029Gwip6PyD2Rh8uKhw= 2024-04-12T04:10:45.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+8WsKICA029Gwip6PyD2Rh8uKhw=', 'Content-Length': '0'} 2024-04-12T04:10:45.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044ff4ca51605e410-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.585 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.585 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.585 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:10:45.585 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:10:45.585 DEBUG:boto:Method: GET 2024-04-12T04:10:45.585 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:10:45.585 DEBUG:boto:Data: 2024-04-12T04:10:45.585 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.585 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.585 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.585 DEBUG:boto:Params: {} 2024-04-12T04:10:45.586 DEBUG:boto:Token: None 2024-04-12T04:10:45.586 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.586 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039b0fc8a3e649e02-006618b445-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.588 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.589 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.589 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.589 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:10:45.589 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:10:45.589 DEBUG:boto:Method: GET 2024-04-12T04:10:45.589 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:10:45.589 DEBUG:boto:Data: 2024-04-12T04:10:45.589 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.589 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.589 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.589 DEBUG:boto:Params: {} 2024-04-12T04:10:45.590 DEBUG:boto:Token: None 2024-04-12T04:10:45.590 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.590 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d518a67f36702d9-006618b445-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.594 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.595 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:10:45.595 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:10:45.595 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.595 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:10:45.595 DEBUG:boto:Data: 2024-04-12T04:10:45.595 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.595 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.595 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.595 DEBUG:boto:Params: {} 2024-04-12T04:10:45.595 DEBUG:boto:Token: None 2024-04-12T04:10:45.596 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj17 2024-04-12T04:10:45.596 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:05bk8fSvkeejNx+2apmrJVhjT/0= 2024-04-12T04:10:45.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:05bk8fSvkeejNx+2apmrJVhjT/0=', 'Content-Length': '0'} 2024-04-12T04:10:45.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059f54a808608ad83-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.598 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:10:45.598 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:10:45.598 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.598 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:10:45.598 DEBUG:boto:Data: 2024-04-12T04:10:45.598 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.599 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.599 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.599 DEBUG:boto:Params: {} 2024-04-12T04:10:45.599 DEBUG:boto:Token: None 2024-04-12T04:10:45.599 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj17 2024-04-12T04:10:45.599 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:05bk8fSvkeejNx+2apmrJVhjT/0= 2024-04-12T04:10:45.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:05bk8fSvkeejNx+2apmrJVhjT/0=', 'Content-Length': '0'} 2024-04-12T04:10:45.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e6ac6876e76211c-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:10:45.601 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:10:45.601 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:10:45.601 DEBUG:boto:Method: GET 2024-04-12T04:10:45.601 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:10:45.602 DEBUG:boto:Data: 2024-04-12T04:10:45.602 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.602 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.602 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.602 DEBUG:boto:Params: {} 2024-04-12T04:10:45.602 DEBUG:boto:Token: None 2024-04-12T04:10:45.602 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj17 2024-04-12T04:10:45.602 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WqXuvrZ/3Jf+p/2CfvnNouV5iHQ= 2024-04-12T04:10:45.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WqXuvrZ/3Jf+p/2CfvnNouV5iHQ=', 'Content-Length': '0'} 2024-04-12T04:10:45.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfcd0086988ab213-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.604 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:10:45.605 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:10:45.605 DEBUG:boto:Method: GET 2024-04-12T04:10:45.605 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:10:45.605 DEBUG:boto:Data: 2024-04-12T04:10:45.605 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.605 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.605 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.605 DEBUG:boto:Params: {} 2024-04-12T04:10:45.605 DEBUG:boto:Token: None 2024-04-12T04:10:45.605 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj17 2024-04-12T04:10:45.606 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WqXuvrZ/3Jf+p/2CfvnNouV5iHQ= 2024-04-12T04:10:45.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WqXuvrZ/3Jf+p/2CfvnNouV5iHQ=', 'Content-Length': '0'} 2024-04-12T04:10:45.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014bdef24304724fa-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.654 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.654 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.655 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:10:45.655 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:10:45.655 DEBUG:boto:Method: GET 2024-04-12T04:10:45.655 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:10:45.655 DEBUG:boto:Data: 2024-04-12T04:10:45.655 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.655 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.655 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.655 DEBUG:boto:Params: {} 2024-04-12T04:10:45.655 DEBUG:boto:Token: None 2024-04-12T04:10:45.655 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3d4f8edd6f538fb-006618b445-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.658 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.659 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.659 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.659 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:10:45.659 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:10:45.659 DEBUG:boto:Method: GET 2024-04-12T04:10:45.659 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:10:45.659 DEBUG:boto:Data: 2024-04-12T04:10:45.659 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.659 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.660 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.660 DEBUG:boto:Params: {} 2024-04-12T04:10:45.660 DEBUG:boto:Token: None 2024-04-12T04:10:45.660 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.660 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a23575d03ec031b2-006618b445-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.664 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T04:10:45.664 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:10:45.665 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:10:45.665 DEBUG:boto:path=/dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.665 DEBUG:boto:auth_path=/dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.665 DEBUG:boto:Method: GET 2024-04-12T04:10:45.665 DEBUG:boto:Path: /dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.665 DEBUG:boto:Data: 2024-04-12T04:10:45.665 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.665 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.665 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.665 DEBUG:boto:Params: {} 2024-04-12T04:10:45.665 DEBUG:boto:Token: None 2024-04-12T04:10:45.665 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.666 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4VvDGI/MavjBz4tdX3Y65J7WJqY= 2024-04-12T04:10:45.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4VvDGI/MavjBz4tdX3Y65J7WJqY=', 'Content-Length': '0'} 2024-04-12T04:10:45.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6ed220f18fc3dbf-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.668 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:10:45.668 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:10:45.668 DEBUG:boto:path=/dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.668 DEBUG:boto:auth_path=/dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.668 DEBUG:boto:Method: GET 2024-04-12T04:10:45.668 DEBUG:boto:Path: /dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.668 DEBUG:boto:Data: 2024-04-12T04:10:45.668 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.669 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.669 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.669 DEBUG:boto:Params: {} 2024-04-12T04:10:45.669 DEBUG:boto:Token: None 2024-04-12T04:10:45.669 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.669 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4VvDGI/MavjBz4tdX3Y65J7WJqY= 2024-04-12T04:10:45.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4VvDGI/MavjBz4tdX3Y65J7WJqY=', 'Content-Length': '0'} 2024-04-12T04:10:45.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000565b63403a361152-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.671 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:10:45.671 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:10:45.671 DEBUG:boto:path=/dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.671 DEBUG:boto:auth_path=/dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.671 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.672 DEBUG:boto:Path: /dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.672 DEBUG:boto:Data: 2024-04-12T04:10:45.672 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.672 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.672 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.672 DEBUG:boto:Params: {} 2024-04-12T04:10:45.672 DEBUG:boto:Token: None 2024-04-12T04:10:45.672 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.672 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jDtlDTbxIGJjFFOipJ6I8jHVrp0= 2024-04-12T04:10:45.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jDtlDTbxIGJjFFOipJ6I8jHVrp0=', 'Content-Length': '0'} 2024-04-12T04:10:45.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d82d8c794c1035da-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.674 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:10:45.674 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:10:45.674 DEBUG:boto:path=/dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.675 DEBUG:boto:auth_path=/dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.675 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.675 DEBUG:boto:Path: /dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.675 DEBUG:boto:Data: 2024-04-12T04:10:45.675 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.675 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.675 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.675 DEBUG:boto:Params: {} 2024-04-12T04:10:45.675 DEBUG:boto:Token: None 2024-04-12T04:10:45.675 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.675 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jDtlDTbxIGJjFFOipJ6I8jHVrp0= 2024-04-12T04:10:45.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jDtlDTbxIGJjFFOipJ6I8jHVrp0=', 'Content-Length': '0'} 2024-04-12T04:10:45.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036eeb1d205fe9858-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T04:10:45.678 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:10:45.678 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:10:45.678 DEBUG:boto:path=/dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.678 DEBUG:boto:auth_path=/dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.678 DEBUG:boto:Method: GET 2024-04-12T04:10:45.678 DEBUG:boto:Path: /dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.678 DEBUG:boto:Data: 2024-04-12T04:10:45.678 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.678 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.678 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.679 DEBUG:boto:Params: {} 2024-04-12T04:10:45.679 DEBUG:boto:Token: None 2024-04-12T04:10:45.679 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.679 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4VvDGI/MavjBz4tdX3Y65J7WJqY= 2024-04-12T04:10:45.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4VvDGI/MavjBz4tdX3Y65J7WJqY=', 'Content-Length': '0'} 2024-04-12T04:10:45.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f4548ae9e3901b9-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.722 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:10:45.722 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:10:45.722 DEBUG:boto:path=/dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.723 DEBUG:boto:auth_path=/dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.723 DEBUG:boto:Method: GET 2024-04-12T04:10:45.723 DEBUG:boto:Path: /dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.723 DEBUG:boto:Data: 2024-04-12T04:10:45.723 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.723 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.723 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.723 DEBUG:boto:Params: {} 2024-04-12T04:10:45.723 DEBUG:boto:Token: None 2024-04-12T04:10:45.723 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj185?versionId=null 2024-04-12T04:10:45.723 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4VvDGI/MavjBz4tdX3Y65J7WJqY= 2024-04-12T04:10:45.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4VvDGI/MavjBz4tdX3Y65J7WJqY=', 'Content-Length': '0'} 2024-04-12T04:10:45.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df4ffb2dfaaa9ad8-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.726 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.726 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.726 DEBUG:boto:path=/dkyges-61/?prefix=obj185 2024-04-12T04:10:45.726 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj185 2024-04-12T04:10:45.726 DEBUG:boto:Method: GET 2024-04-12T04:10:45.726 DEBUG:boto:Path: /dkyges-61/?prefix=obj185 2024-04-12T04:10:45.726 DEBUG:boto:Data: 2024-04-12T04:10:45.726 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.726 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.726 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.727 DEBUG:boto:Params: {} 2024-04-12T04:10:45.727 DEBUG:boto:Token: None 2024-04-12T04:10:45.727 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.727 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.729 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bc31c487de0fd3c-006618b445-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.729 DEBUG:boto:b'dkyges-61obj1851000falseobj1852024-04-12T04:09:24.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.730 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.730 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.730 DEBUG:boto:path=/dkyges-61/?prefix=obj185 2024-04-12T04:10:45.730 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj185 2024-04-12T04:10:45.730 DEBUG:boto:Method: GET 2024-04-12T04:10:45.730 DEBUG:boto:Path: /dkyges-61/?prefix=obj185 2024-04-12T04:10:45.730 DEBUG:boto:Data: 2024-04-12T04:10:45.730 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.730 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.731 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.731 DEBUG:boto:Params: {} 2024-04-12T04:10:45.731 DEBUG:boto:Token: None 2024-04-12T04:10:45.731 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.731 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000987fbc88f2690291-006618b445-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.735 DEBUG:boto:b'dkyges-61obj1851000falseobj1852024-04-12T04:09:24.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.736 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:10:45.736 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:10:45.736 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.736 DEBUG:boto:Path: /dkyges-61/obj185 2024-04-12T04:10:45.736 DEBUG:boto:Data: 2024-04-12T04:10:45.736 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.736 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.736 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.736 DEBUG:boto:Params: {} 2024-04-12T04:10:45.736 DEBUG:boto:Token: None 2024-04-12T04:10:45.737 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj185 2024-04-12T04:10:45.737 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7x7eWuKRkKqk/xYWZEFVcGyGKgg= 2024-04-12T04:10:45.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7x7eWuKRkKqk/xYWZEFVcGyGKgg=', 'Content-Length': '0'} 2024-04-12T04:10:45.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7a1afb731ab2fa0-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.739 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:10:45.739 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:10:45.739 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.739 DEBUG:boto:Path: /dkyges-61/obj185 2024-04-12T04:10:45.739 DEBUG:boto:Data: 2024-04-12T04:10:45.739 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.739 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.739 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.739 DEBUG:boto:Params: {} 2024-04-12T04:10:45.739 DEBUG:boto:Token: None 2024-04-12T04:10:45.739 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj185 2024-04-12T04:10:45.740 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7x7eWuKRkKqk/xYWZEFVcGyGKgg= 2024-04-12T04:10:45.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7x7eWuKRkKqk/xYWZEFVcGyGKgg=', 'Content-Length': '0'} 2024-04-12T04:10:45.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001857f33b8ae41c51-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T04:10:45.742 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:10:45.742 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:10:45.742 DEBUG:boto:Method: GET 2024-04-12T04:10:45.742 DEBUG:boto:Path: /dkyges-61/obj185 2024-04-12T04:10:45.742 DEBUG:boto:Data: 2024-04-12T04:10:45.742 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.742 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.742 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.742 DEBUG:boto:Params: {} 2024-04-12T04:10:45.743 DEBUG:boto:Token: None 2024-04-12T04:10:45.743 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj185 2024-04-12T04:10:45.743 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uPHbF5b0q1U37VYX/9hI+HNxySk= 2024-04-12T04:10:45.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uPHbF5b0q1U37VYX/9hI+HNxySk=', 'Content-Length': '0'} 2024-04-12T04:10:45.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bff348757a69e48b-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.745 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:10:45.745 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:10:45.745 DEBUG:boto:Method: GET 2024-04-12T04:10:45.745 DEBUG:boto:Path: /dkyges-61/obj185 2024-04-12T04:10:45.745 DEBUG:boto:Data: 2024-04-12T04:10:45.746 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.746 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.746 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.746 DEBUG:boto:Params: {} 2024-04-12T04:10:45.746 DEBUG:boto:Token: None 2024-04-12T04:10:45.746 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj185 2024-04-12T04:10:45.746 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uPHbF5b0q1U37VYX/9hI+HNxySk= 2024-04-12T04:10:45.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uPHbF5b0q1U37VYX/9hI+HNxySk=', 'Content-Length': '0'} 2024-04-12T04:10:45.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d14d68df1d0dd4a3-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.748 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.748 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.748 DEBUG:boto:path=/dkyges-61/?prefix=obj185 2024-04-12T04:10:45.748 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj185 2024-04-12T04:10:45.749 DEBUG:boto:Method: GET 2024-04-12T04:10:45.749 DEBUG:boto:Path: /dkyges-61/?prefix=obj185 2024-04-12T04:10:45.749 DEBUG:boto:Data: 2024-04-12T04:10:45.749 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.749 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.749 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.749 DEBUG:boto:Params: {} 2024-04-12T04:10:45.749 DEBUG:boto:Token: None 2024-04-12T04:10:45.749 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.749 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.752 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8e7a738dd8c4283-006618b445-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.752 DEBUG:boto:b'dkyges-61obj1851000falseobj1852024-04-12T04:09:24.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.752 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.752 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.752 DEBUG:boto:path=/dkyges-61/?prefix=obj185 2024-04-12T04:10:45.752 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj185 2024-04-12T04:10:45.752 DEBUG:boto:Method: GET 2024-04-12T04:10:45.753 DEBUG:boto:Path: /dkyges-61/?prefix=obj185 2024-04-12T04:10:45.753 DEBUG:boto:Data: 2024-04-12T04:10:45.753 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.753 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.753 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.753 DEBUG:boto:Params: {} 2024-04-12T04:10:45.753 DEBUG:boto:Token: None 2024-04-12T04:10:45.753 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.753 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b207609d3c989f4-006618b445-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.758 DEBUG:boto:b'dkyges-61obj1851000falseobj1852024-04-12T04:09:24.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:10:45.758 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:10:45.758 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:10:45.759 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.759 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.759 DEBUG:boto:Method: GET 2024-04-12T04:10:45.759 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.759 DEBUG:boto:Data: 2024-04-12T04:10:45.759 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.759 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.759 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.759 DEBUG:boto:Params: {} 2024-04-12T04:10:45.759 DEBUG:boto:Token: None 2024-04-12T04:10:45.759 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.760 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MH2iH0h9I5B41eOvEErwFzTF4yg= 2024-04-12T04:10:45.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MH2iH0h9I5B41eOvEErwFzTF4yg=', 'Content-Length': '0'} 2024-04-12T04:10:45.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b9ef3ae32a6d50d-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.802 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:10:45.802 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:10:45.803 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.803 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.803 DEBUG:boto:Method: GET 2024-04-12T04:10:45.803 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.803 DEBUG:boto:Data: 2024-04-12T04:10:45.803 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.803 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.803 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.803 DEBUG:boto:Params: {} 2024-04-12T04:10:45.803 DEBUG:boto:Token: None 2024-04-12T04:10:45.803 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.804 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MH2iH0h9I5B41eOvEErwFzTF4yg= 2024-04-12T04:10:45.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MH2iH0h9I5B41eOvEErwFzTF4yg=', 'Content-Length': '0'} 2024-04-12T04:10:45.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003548c0ad21a0f2f5-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.806 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:10:45.806 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:10:45.806 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.807 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.807 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.807 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.807 DEBUG:boto:Data: 2024-04-12T04:10:45.807 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.807 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.807 DEBUG:boto:Params: {} 2024-04-12T04:10:45.807 DEBUG:boto:Token: None 2024-04-12T04:10:45.807 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.808 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zT4YyFaWjiLMPCyW/GJOeUd3ppw= 2024-04-12T04:10:45.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zT4YyFaWjiLMPCyW/GJOeUd3ppw=', 'Content-Length': '0'} 2024-04-12T04:10:45.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000107e64ee722f8b02-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.810 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:10:45.810 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:10:45.810 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.810 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.810 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.810 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.810 DEBUG:boto:Data: 2024-04-12T04:10:45.810 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.811 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.811 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.811 DEBUG:boto:Params: {} 2024-04-12T04:10:45.811 DEBUG:boto:Token: None 2024-04-12T04:10:45.811 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.811 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zT4YyFaWjiLMPCyW/GJOeUd3ppw= 2024-04-12T04:10:45.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zT4YyFaWjiLMPCyW/GJOeUd3ppw=', 'Content-Length': '0'} 2024-04-12T04:10:45.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065b2d191e6efb91d-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:10:45.813 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:10:45.813 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:10:45.813 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.813 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.814 DEBUG:boto:Method: GET 2024-04-12T04:10:45.814 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.814 DEBUG:boto:Data: 2024-04-12T04:10:45.814 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.814 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.814 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.814 DEBUG:boto:Params: {} 2024-04-12T04:10:45.814 DEBUG:boto:Token: None 2024-04-12T04:10:45.814 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.815 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MH2iH0h9I5B41eOvEErwFzTF4yg= 2024-04-12T04:10:45.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MH2iH0h9I5B41eOvEErwFzTF4yg=', 'Content-Length': '0'} 2024-04-12T04:10:45.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eec9fa5f6185052d-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.817 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:10:45.817 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:10:45.817 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.817 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.817 DEBUG:boto:Method: GET 2024-04-12T04:10:45.817 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.817 DEBUG:boto:Data: 2024-04-12T04:10:45.818 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.818 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.818 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.818 DEBUG:boto:Params: {} 2024-04-12T04:10:45.818 DEBUG:boto:Token: None 2024-04-12T04:10:45.818 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:10:45.818 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MH2iH0h9I5B41eOvEErwFzTF4yg= 2024-04-12T04:10:45.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MH2iH0h9I5B41eOvEErwFzTF4yg=', 'Content-Length': '0'} 2024-04-12T04:10:45.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001438418921cc62fd-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.820 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.820 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.820 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:10:45.821 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:10:45.821 DEBUG:boto:Method: GET 2024-04-12T04:10:45.821 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:10:45.821 DEBUG:boto:Data: 2024-04-12T04:10:45.821 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.821 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.821 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.821 DEBUG:boto:Params: {} 2024-04-12T04:10:45.821 DEBUG:boto:Token: None 2024-04-12T04:10:45.821 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.822 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e732eae9e2f1a69-006618b445-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.824 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.825 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.825 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.825 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:10:45.825 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:10:45.825 DEBUG:boto:Method: GET 2024-04-12T04:10:45.825 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:10:45.825 DEBUG:boto:Data: 2024-04-12T04:10:45.825 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.825 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.825 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.826 DEBUG:boto:Params: {} 2024-04-12T04:10:45.826 DEBUG:boto:Token: None 2024-04-12T04:10:45.826 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.826 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c8d1d6036d4a765-006618b445-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.830 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.830 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:10:45.831 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:10:45.831 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.831 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:10:45.831 DEBUG:boto:Data: 2024-04-12T04:10:45.831 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.831 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.831 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.831 DEBUG:boto:Params: {} 2024-04-12T04:10:45.831 DEBUG:boto:Token: None 2024-04-12T04:10:45.831 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj19 2024-04-12T04:10:45.832 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2z1ve0FzuumDyEewYb2p9OubSUg= 2024-04-12T04:10:45.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2z1ve0FzuumDyEewYb2p9OubSUg=', 'Content-Length': '0'} 2024-04-12T04:10:45.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e44b26a1f281cd22-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.834 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:10:45.834 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:10:45.834 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.834 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:10:45.834 DEBUG:boto:Data: 2024-04-12T04:10:45.834 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.834 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.835 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.835 DEBUG:boto:Params: {} 2024-04-12T04:10:45.835 DEBUG:boto:Token: None 2024-04-12T04:10:45.835 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj19 2024-04-12T04:10:45.835 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2z1ve0FzuumDyEewYb2p9OubSUg= 2024-04-12T04:10:45.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2z1ve0FzuumDyEewYb2p9OubSUg=', 'Content-Length': '0'} 2024-04-12T04:10:45.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e969deeaa5cbe2d-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:10:45.838 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:10:45.838 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:10:45.838 DEBUG:boto:Method: GET 2024-04-12T04:10:45.838 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:10:45.838 DEBUG:boto:Data: 2024-04-12T04:10:45.838 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.838 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.838 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.838 DEBUG:boto:Params: {} 2024-04-12T04:10:45.838 DEBUG:boto:Token: None 2024-04-12T04:10:45.839 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj19 2024-04-12T04:10:45.839 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hix/IB8mdP/ash4fCoxG9XLUPuI= 2024-04-12T04:10:45.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hix/IB8mdP/ash4fCoxG9XLUPuI=', 'Content-Length': '0'} 2024-04-12T04:10:45.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f3d916a6936e029-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.841 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:10:45.841 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:10:45.841 DEBUG:boto:Method: GET 2024-04-12T04:10:45.841 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:10:45.841 DEBUG:boto:Data: 2024-04-12T04:10:45.841 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.841 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.841 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.841 DEBUG:boto:Params: {} 2024-04-12T04:10:45.842 DEBUG:boto:Token: None 2024-04-12T04:10:45.842 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj19 2024-04-12T04:10:45.842 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hix/IB8mdP/ash4fCoxG9XLUPuI= 2024-04-12T04:10:45.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hix/IB8mdP/ash4fCoxG9XLUPuI=', 'Content-Length': '0'} 2024-04-12T04:10:45.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b015bad084e37393-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.844 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.844 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.844 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:10:45.844 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:10:45.845 DEBUG:boto:Method: GET 2024-04-12T04:10:45.845 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:10:45.845 DEBUG:boto:Data: 2024-04-12T04:10:45.845 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.845 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.845 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.845 DEBUG:boto:Params: {} 2024-04-12T04:10:45.845 DEBUG:boto:Token: None 2024-04-12T04:10:45.845 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.845 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000765a2895d8d63f00-006618b445-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.848 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.848 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.849 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.849 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:10:45.849 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:10:45.849 DEBUG:boto:Method: GET 2024-04-12T04:10:45.849 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:10:45.849 DEBUG:boto:Data: 2024-04-12T04:10:45.849 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.849 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.849 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.849 DEBUG:boto:Params: {} 2024-04-12T04:10:45.849 DEBUG:boto:Token: None 2024-04-12T04:10:45.850 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.850 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ece40cfcac94bb9c-006618b445-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.854 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:10:45.855 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:10:45.855 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:10:45.855 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.855 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.855 DEBUG:boto:Method: GET 2024-04-12T04:10:45.855 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.855 DEBUG:boto:Data: 2024-04-12T04:10:45.855 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.855 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.855 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.855 DEBUG:boto:Params: {} 2024-04-12T04:10:45.856 DEBUG:boto:Token: None 2024-04-12T04:10:45.856 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.856 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3A1PWX0lD3rt5rqcW1vzMgbcKE0= 2024-04-12T04:10:45.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3A1PWX0lD3rt5rqcW1vzMgbcKE0=', 'Content-Length': '0'} 2024-04-12T04:10:45.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003890eec38413cc54-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.898 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:10:45.898 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:10:45.899 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.899 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.899 DEBUG:boto:Method: GET 2024-04-12T04:10:45.899 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.899 DEBUG:boto:Data: 2024-04-12T04:10:45.899 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.899 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.899 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.899 DEBUG:boto:Params: {} 2024-04-12T04:10:45.899 DEBUG:boto:Token: None 2024-04-12T04:10:45.899 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.899 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3A1PWX0lD3rt5rqcW1vzMgbcKE0= 2024-04-12T04:10:45.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3A1PWX0lD3rt5rqcW1vzMgbcKE0=', 'Content-Length': '0'} 2024-04-12T04:10:45.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067694b78460feb00-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.902 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:10:45.902 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:10:45.902 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.902 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.902 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.902 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.902 DEBUG:boto:Data: 2024-04-12T04:10:45.902 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.902 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.903 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.903 DEBUG:boto:Params: {} 2024-04-12T04:10:45.903 DEBUG:boto:Token: None 2024-04-12T04:10:45.903 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.903 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4a2yvpHt1nSzyaMXRlza1HjOLmk= 2024-04-12T04:10:45.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4a2yvpHt1nSzyaMXRlza1HjOLmk=', 'Content-Length': '0'} 2024-04-12T04:10:45.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009591af4848d1296b-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.905 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:10:45.905 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:10:45.905 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.905 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.906 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.906 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.906 DEBUG:boto:Data: 2024-04-12T04:10:45.906 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.906 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.906 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.906 DEBUG:boto:Params: {} 2024-04-12T04:10:45.906 DEBUG:boto:Token: None 2024-04-12T04:10:45.906 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.907 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4a2yvpHt1nSzyaMXRlza1HjOLmk= 2024-04-12T04:10:45.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4a2yvpHt1nSzyaMXRlza1HjOLmk=', 'Content-Length': '0'} 2024-04-12T04:10:45.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a4818261d331ba6-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:10:45.909 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:10:45.909 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:10:45.909 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.909 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.909 DEBUG:boto:Method: GET 2024-04-12T04:10:45.909 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.909 DEBUG:boto:Data: 2024-04-12T04:10:45.909 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.909 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.909 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.909 DEBUG:boto:Params: {} 2024-04-12T04:10:45.910 DEBUG:boto:Token: None 2024-04-12T04:10:45.910 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.910 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3A1PWX0lD3rt5rqcW1vzMgbcKE0= 2024-04-12T04:10:45.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3A1PWX0lD3rt5rqcW1vzMgbcKE0=', 'Content-Length': '0'} 2024-04-12T04:10:45.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002250a2222e46ea52-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.912 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:10:45.912 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:10:45.912 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.912 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.912 DEBUG:boto:Method: GET 2024-04-12T04:10:45.912 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.912 DEBUG:boto:Data: 2024-04-12T04:10:45.913 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.913 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.913 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.913 DEBUG:boto:Params: {} 2024-04-12T04:10:45.913 DEBUG:boto:Token: None 2024-04-12T04:10:45.913 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:10:45.913 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3A1PWX0lD3rt5rqcW1vzMgbcKE0= 2024-04-12T04:10:45.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3A1PWX0lD3rt5rqcW1vzMgbcKE0=', 'Content-Length': '0'} 2024-04-12T04:10:45.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f32b9d6f9c31ecc-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.915 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.915 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.916 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:10:45.916 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:10:45.916 DEBUG:boto:Method: GET 2024-04-12T04:10:45.916 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:10:45.916 DEBUG:boto:Data: 2024-04-12T04:10:45.916 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.916 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.916 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.916 DEBUG:boto:Params: {} 2024-04-12T04:10:45.916 DEBUG:boto:Token: None 2024-04-12T04:10:45.916 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.917 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018a2bb6464eef378-006618b445-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.920 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.920 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.920 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.920 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:10:45.920 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:10:45.920 DEBUG:boto:Method: GET 2024-04-12T04:10:45.920 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:10:45.920 DEBUG:boto:Data: 2024-04-12T04:10:45.921 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.921 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.921 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.921 DEBUG:boto:Params: {} 2024-04-12T04:10:45.921 DEBUG:boto:Token: None 2024-04-12T04:10:45.921 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.921 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.926 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072953cf5fd573ee2-006618b445-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.926 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.926 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:10:45.926 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:10:45.927 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.927 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:10:45.927 DEBUG:boto:Data: 2024-04-12T04:10:45.927 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.927 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.927 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.927 DEBUG:boto:Params: {} 2024-04-12T04:10:45.927 DEBUG:boto:Token: None 2024-04-12T04:10:45.927 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj22 2024-04-12T04:10:45.927 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gTpbB/RO6URZXV63l8yf9bPU+P4= 2024-04-12T04:10:45.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gTpbB/RO6URZXV63l8yf9bPU+P4=', 'Content-Length': '0'} 2024-04-12T04:10:45.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003942c876b9e3956e-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.929 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:10:45.930 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:10:45.930 DEBUG:boto:Method: HEAD 2024-04-12T04:10:45.930 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:10:45.930 DEBUG:boto:Data: 2024-04-12T04:10:45.930 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.930 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.930 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.930 DEBUG:boto:Params: {} 2024-04-12T04:10:45.930 DEBUG:boto:Token: None 2024-04-12T04:10:45.930 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj22 2024-04-12T04:10:45.931 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gTpbB/RO6URZXV63l8yf9bPU+P4= 2024-04-12T04:10:45.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gTpbB/RO6URZXV63l8yf9bPU+P4=', 'Content-Length': '0'} 2024-04-12T04:10:45.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000429a5b67bd65ad54-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:10:45.933 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:10:45.933 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:10:45.933 DEBUG:boto:Method: GET 2024-04-12T04:10:45.933 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:10:45.933 DEBUG:boto:Data: 2024-04-12T04:10:45.933 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.933 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.933 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.933 DEBUG:boto:Params: {} 2024-04-12T04:10:45.933 DEBUG:boto:Token: None 2024-04-12T04:10:45.934 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj22 2024-04-12T04:10:45.934 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XyUDiEwr+tdfikSCAQ/3nMRw+M8= 2024-04-12T04:10:45.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XyUDiEwr+tdfikSCAQ/3nMRw+M8=', 'Content-Length': '0'} 2024-04-12T04:10:45.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc2ac26fc5df592b-006618b445-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.936 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:10:45.936 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:10:45.936 DEBUG:boto:Method: GET 2024-04-12T04:10:45.936 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:10:45.936 DEBUG:boto:Data: 2024-04-12T04:10:45.936 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.936 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.936 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.936 DEBUG:boto:Params: {} 2024-04-12T04:10:45.936 DEBUG:boto:Token: None 2024-04-12T04:10:45.937 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj22 2024-04-12T04:10:45.937 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XyUDiEwr+tdfikSCAQ/3nMRw+M8= 2024-04-12T04:10:45.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XyUDiEwr+tdfikSCAQ/3nMRw+M8=', 'Content-Length': '0'} 2024-04-12T04:10:45.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d72558a2cd2a754a-006618b445-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.982 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.983 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.983 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:10:45.983 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:10:45.983 DEBUG:boto:Method: GET 2024-04-12T04:10:45.983 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:10:45.983 DEBUG:boto:Data: 2024-04-12T04:10:45.983 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.983 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.983 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.983 DEBUG:boto:Params: {} 2024-04-12T04:10:45.983 DEBUG:boto:Token: None 2024-04-12T04:10:45.984 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.984 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b06bc9aa277ee4e-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.987 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.987 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:45.987 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:45.987 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:10:45.987 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:10:45.988 DEBUG:boto:Method: GET 2024-04-12T04:10:45.988 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:10:45.988 DEBUG:boto:Data: 2024-04-12T04:10:45.988 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.988 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:45.988 DEBUG:boto:Port: 8000 2024-04-12T04:10:45.988 DEBUG:boto:Params: {} 2024-04-12T04:10:45.988 DEBUG:boto:Token: None 2024-04-12T04:10:45.988 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/ 2024-04-12T04:10:45.988 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg= 2024-04-12T04:10:45.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3GzYl4+sVG0wNNrRJKmLTCoplg=', 'Content-Length': '0'} 2024-04-12T04:10:45.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038aa5e3c0f78aec3-006618b446-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:45.993 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:45.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:10:45.993 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:10:45.993 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:10:45.994 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:10:45.994 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:10:45.994 DEBUG:boto:Method: GET 2024-04-12T04:10:45.994 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:10:45.994 DEBUG:boto:Data: 2024-04-12T04:10:45.994 DEBUG:boto:Headers: {} 2024-04-12T04:10:45.994 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:45.994 DEBUG:boto:Port: 8001 2024-04-12T04:10:45.994 DEBUG:boto:Params: {} 2024-04-12T04:10:45.994 DEBUG:boto:Token: None 2024-04-12T04:10:45.994 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:45 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:10:45.995 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aPgpGL0aCVf0ToZcPZukvOGhKYw= 2024-04-12T04:10:45.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aPgpGL0aCVf0ToZcPZukvOGhKYw=', 'Content-Length': '0'} 2024-04-12T04:10:45.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba5a47d310dc8854-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.038 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:10:46.038 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:10:46.038 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.039 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.039 DEBUG:boto:Method: GET 2024-04-12T04:10:46.039 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.039 DEBUG:boto:Data: 2024-04-12T04:10:46.039 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.039 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.039 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.039 DEBUG:boto:Params: {} 2024-04-12T04:10:46.039 DEBUG:boto:Token: None 2024-04-12T04:10:46.039 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.040 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SJQhXH7K+bJXIcuO6a9E1vy1aCo= 2024-04-12T04:10:46.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SJQhXH7K+bJXIcuO6a9E1vy1aCo=', 'Content-Length': '0'} 2024-04-12T04:10:46.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b02d4320167ac47-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.042 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:10:46.042 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:10:46.042 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.042 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.042 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.042 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.043 DEBUG:boto:Data: 2024-04-12T04:10:46.043 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.043 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.043 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.043 DEBUG:boto:Params: {} 2024-04-12T04:10:46.043 DEBUG:boto:Token: None 2024-04-12T04:10:46.043 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.043 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fTp/gZ+ojanTjJwYI2A7AwhA1fQ= 2024-04-12T04:10:46.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fTp/gZ+ojanTjJwYI2A7AwhA1fQ=', 'Content-Length': '0'} 2024-04-12T04:10:46.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037943999ea5330d2-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.045 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:10:46.045 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:10:46.046 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.046 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.046 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.046 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.046 DEBUG:boto:Data: 2024-04-12T04:10:46.046 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.046 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.046 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.046 DEBUG:boto:Params: {} 2024-04-12T04:10:46.046 DEBUG:boto:Token: None 2024-04-12T04:10:46.046 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.047 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fTp/gZ+ojanTjJwYI2A7AwhA1fQ= 2024-04-12T04:10:46.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fTp/gZ+ojanTjJwYI2A7AwhA1fQ=', 'Content-Length': '0'} 2024-04-12T04:10:46.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8421fa2194fd504-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:10:46.049 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:10:46.049 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:10:46.049 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.049 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.049 DEBUG:boto:Method: GET 2024-04-12T04:10:46.049 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.049 DEBUG:boto:Data: 2024-04-12T04:10:46.049 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.049 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.049 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.049 DEBUG:boto:Params: {} 2024-04-12T04:10:46.049 DEBUG:boto:Token: None 2024-04-12T04:10:46.050 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.050 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SJQhXH7K+bJXIcuO6a9E1vy1aCo= 2024-04-12T04:10:46.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SJQhXH7K+bJXIcuO6a9E1vy1aCo=', 'Content-Length': '0'} 2024-04-12T04:10:46.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d23c5421a480de51-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.052 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:10:46.052 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:10:46.052 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.052 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.052 DEBUG:boto:Method: GET 2024-04-12T04:10:46.052 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.053 DEBUG:boto:Data: 2024-04-12T04:10:46.053 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.053 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.053 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.053 DEBUG:boto:Params: {} 2024-04-12T04:10:46.053 DEBUG:boto:Token: None 2024-04-12T04:10:46.053 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:10:46.053 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SJQhXH7K+bJXIcuO6a9E1vy1aCo= 2024-04-12T04:10:46.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SJQhXH7K+bJXIcuO6a9E1vy1aCo=', 'Content-Length': '0'} 2024-04-12T04:10:46.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa171ebd9f6def96-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.056 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.056 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.056 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:10:46.056 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:10:46.056 DEBUG:boto:Method: GET 2024-04-12T04:10:46.056 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:10:46.056 DEBUG:boto:Data: 2024-04-12T04:10:46.056 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.056 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.056 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.056 DEBUG:boto:Params: {} 2024-04-12T04:10:46.057 DEBUG:boto:Token: None 2024-04-12T04:10:46.057 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.057 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fc68163a399b047-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.059 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.060 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.060 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.060 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:10:46.060 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:10:46.060 DEBUG:boto:Method: GET 2024-04-12T04:10:46.060 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:10:46.060 DEBUG:boto:Data: 2024-04-12T04:10:46.060 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.061 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.061 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.061 DEBUG:boto:Params: {} 2024-04-12T04:10:46.061 DEBUG:boto:Token: None 2024-04-12T04:10:46.061 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.061 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b534764c39b61eac-006618b446-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.065 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.066 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:10:46.066 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:10:46.066 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.066 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:10:46.066 DEBUG:boto:Data: 2024-04-12T04:10:46.066 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.066 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.066 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.066 DEBUG:boto:Params: {} 2024-04-12T04:10:46.066 DEBUG:boto:Token: None 2024-04-12T04:10:46.067 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj23 2024-04-12T04:10:46.067 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Sbd9LqebZwKU6X6+DOeOCNjNHs8= 2024-04-12T04:10:46.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Sbd9LqebZwKU6X6+DOeOCNjNHs8=', 'Content-Length': '0'} 2024-04-12T04:10:46.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057c7f60132b14192-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.069 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:10:46.069 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:10:46.069 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.069 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:10:46.069 DEBUG:boto:Data: 2024-04-12T04:10:46.069 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.069 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.069 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.069 DEBUG:boto:Params: {} 2024-04-12T04:10:46.070 DEBUG:boto:Token: None 2024-04-12T04:10:46.070 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj23 2024-04-12T04:10:46.070 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Sbd9LqebZwKU6X6+DOeOCNjNHs8= 2024-04-12T04:10:46.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Sbd9LqebZwKU6X6+DOeOCNjNHs8=', 'Content-Length': '0'} 2024-04-12T04:10:46.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e4155648d96736a-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:10:46.072 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:10:46.072 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:10:46.072 DEBUG:boto:Method: GET 2024-04-12T04:10:46.072 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:10:46.072 DEBUG:boto:Data: 2024-04-12T04:10:46.072 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.073 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.073 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.073 DEBUG:boto:Params: {} 2024-04-12T04:10:46.073 DEBUG:boto:Token: None 2024-04-12T04:10:46.073 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj23 2024-04-12T04:10:46.073 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kmUgOczNw98Qcxx9mcyZOgXwyDQ= 2024-04-12T04:10:46.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kmUgOczNw98Qcxx9mcyZOgXwyDQ=', 'Content-Length': '0'} 2024-04-12T04:10:46.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d04439081ceafa6d-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.075 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:10:46.075 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:10:46.075 DEBUG:boto:Method: GET 2024-04-12T04:10:46.075 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:10:46.075 DEBUG:boto:Data: 2024-04-12T04:10:46.076 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.076 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.076 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.076 DEBUG:boto:Params: {} 2024-04-12T04:10:46.076 DEBUG:boto:Token: None 2024-04-12T04:10:46.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj23 2024-04-12T04:10:46.076 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kmUgOczNw98Qcxx9mcyZOgXwyDQ= 2024-04-12T04:10:46.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kmUgOczNw98Qcxx9mcyZOgXwyDQ=', 'Content-Length': '0'} 2024-04-12T04:10:46.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013032e441f9880ce-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.118 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.118 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.118 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:10:46.119 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:10:46.119 DEBUG:boto:Method: GET 2024-04-12T04:10:46.119 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:10:46.119 DEBUG:boto:Data: 2024-04-12T04:10:46.119 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.119 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.119 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.119 DEBUG:boto:Params: {} 2024-04-12T04:10:46.119 DEBUG:boto:Token: None 2024-04-12T04:10:46.119 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.120 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009df84e6739e80dff-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.123 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.123 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.123 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.123 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:10:46.123 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:10:46.123 DEBUG:boto:Method: GET 2024-04-12T04:10:46.123 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:10:46.123 DEBUG:boto:Data: 2024-04-12T04:10:46.124 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.124 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.124 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.124 DEBUG:boto:Params: {} 2024-04-12T04:10:46.124 DEBUG:boto:Token: None 2024-04-12T04:10:46.124 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.124 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c2478920c3e5130-006618b446-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.129 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:10:46.130 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:10:46.130 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:10:46.130 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.130 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.130 DEBUG:boto:Method: GET 2024-04-12T04:10:46.131 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.131 DEBUG:boto:Data: 2024-04-12T04:10:46.131 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.131 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.131 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.131 DEBUG:boto:Params: {} 2024-04-12T04:10:46.131 DEBUG:boto:Token: None 2024-04-12T04:10:46.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.132 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KjT0CVJnMdkeKFI+SRcjrAMdums= 2024-04-12T04:10:46.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KjT0CVJnMdkeKFI+SRcjrAMdums=', 'Content-Length': '0'} 2024-04-12T04:10:46.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e488e2909b852bd2-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.174 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:10:46.174 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:10:46.175 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.175 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.175 DEBUG:boto:Method: GET 2024-04-12T04:10:46.175 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.175 DEBUG:boto:Data: 2024-04-12T04:10:46.175 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.175 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.175 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.175 DEBUG:boto:Params: {} 2024-04-12T04:10:46.175 DEBUG:boto:Token: None 2024-04-12T04:10:46.175 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.176 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KjT0CVJnMdkeKFI+SRcjrAMdums= 2024-04-12T04:10:46.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KjT0CVJnMdkeKFI+SRcjrAMdums=', 'Content-Length': '0'} 2024-04-12T04:10:46.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2e2cf6085b1bc32-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.178 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:10:46.178 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:10:46.178 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.179 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.179 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.179 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.179 DEBUG:boto:Data: 2024-04-12T04:10:46.179 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.179 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.179 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.179 DEBUG:boto:Params: {} 2024-04-12T04:10:46.179 DEBUG:boto:Token: None 2024-04-12T04:10:46.179 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.179 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:E0T0btaqD5x8O61oJ3Qu6CfcTNc= 2024-04-12T04:10:46.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:E0T0btaqD5x8O61oJ3Qu6CfcTNc=', 'Content-Length': '0'} 2024-04-12T04:10:46.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001584af559e13c6ef-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.181 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:10:46.182 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:10:46.182 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.182 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.182 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.182 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.182 DEBUG:boto:Data: 2024-04-12T04:10:46.182 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.182 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.182 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.182 DEBUG:boto:Params: {} 2024-04-12T04:10:46.183 DEBUG:boto:Token: None 2024-04-12T04:10:46.183 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.183 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:E0T0btaqD5x8O61oJ3Qu6CfcTNc= 2024-04-12T04:10:46.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:E0T0btaqD5x8O61oJ3Qu6CfcTNc=', 'Content-Length': '0'} 2024-04-12T04:10:46.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea4ce74614976eb3-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:10:46.185 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:10:46.185 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:10:46.185 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.185 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.185 DEBUG:boto:Method: GET 2024-04-12T04:10:46.186 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.186 DEBUG:boto:Data: 2024-04-12T04:10:46.186 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.186 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.186 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.186 DEBUG:boto:Params: {} 2024-04-12T04:10:46.186 DEBUG:boto:Token: None 2024-04-12T04:10:46.186 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.186 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KjT0CVJnMdkeKFI+SRcjrAMdums= 2024-04-12T04:10:46.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KjT0CVJnMdkeKFI+SRcjrAMdums=', 'Content-Length': '0'} 2024-04-12T04:10:46.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009312859b7dbb93d9-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.188 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:10:46.189 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:10:46.189 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.189 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.189 DEBUG:boto:Method: GET 2024-04-12T04:10:46.189 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.189 DEBUG:boto:Data: 2024-04-12T04:10:46.189 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.189 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.189 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.189 DEBUG:boto:Params: {} 2024-04-12T04:10:46.189 DEBUG:boto:Token: None 2024-04-12T04:10:46.190 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:10:46.190 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KjT0CVJnMdkeKFI+SRcjrAMdums= 2024-04-12T04:10:46.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KjT0CVJnMdkeKFI+SRcjrAMdums=', 'Content-Length': '0'} 2024-04-12T04:10:46.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc4e4791bd43d8ab-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.192 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.192 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.192 DEBUG:boto:path=/dkyges-61/?prefix=obj29 2024-04-12T04:10:46.192 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj29 2024-04-12T04:10:46.192 DEBUG:boto:Method: GET 2024-04-12T04:10:46.192 DEBUG:boto:Path: /dkyges-61/?prefix=obj29 2024-04-12T04:10:46.192 DEBUG:boto:Data: 2024-04-12T04:10:46.193 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.193 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.193 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.193 DEBUG:boto:Params: {} 2024-04-12T04:10:46.193 DEBUG:boto:Token: None 2024-04-12T04:10:46.193 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.193 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7ba42d16c0b4eb4-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.196 DEBUG:boto:b'dkyges-61obj291000falseobj292024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.196 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.196 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.196 DEBUG:boto:path=/dkyges-61/?prefix=obj29 2024-04-12T04:10:46.197 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj29 2024-04-12T04:10:46.197 DEBUG:boto:Method: GET 2024-04-12T04:10:46.197 DEBUG:boto:Path: /dkyges-61/?prefix=obj29 2024-04-12T04:10:46.197 DEBUG:boto:Data: 2024-04-12T04:10:46.197 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.197 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.197 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.197 DEBUG:boto:Params: {} 2024-04-12T04:10:46.197 DEBUG:boto:Token: None 2024-04-12T04:10:46.197 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.197 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000530fadb5886d72c2-006618b446-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.202 DEBUG:boto:b'dkyges-61obj291000falseobj292024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.202 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:10:46.203 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:10:46.203 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.203 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:10:46.203 DEBUG:boto:Data: 2024-04-12T04:10:46.203 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.203 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.203 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.203 DEBUG:boto:Params: {} 2024-04-12T04:10:46.203 DEBUG:boto:Token: None 2024-04-12T04:10:46.203 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj29 2024-04-12T04:10:46.204 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eT2GJrJrotIOwIbJTALo8U9jCGw= 2024-04-12T04:10:46.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eT2GJrJrotIOwIbJTALo8U9jCGw=', 'Content-Length': '0'} 2024-04-12T04:10:46.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b89c6e0fe98ccb2d-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.206 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:10:46.206 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:10:46.206 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.206 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:10:46.206 DEBUG:boto:Data: 2024-04-12T04:10:46.206 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.206 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.206 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.206 DEBUG:boto:Params: {} 2024-04-12T04:10:46.207 DEBUG:boto:Token: None 2024-04-12T04:10:46.207 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj29 2024-04-12T04:10:46.207 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eT2GJrJrotIOwIbJTALo8U9jCGw= 2024-04-12T04:10:46.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eT2GJrJrotIOwIbJTALo8U9jCGw=', 'Content-Length': '0'} 2024-04-12T04:10:46.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b3caa09ea8eafaa-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:10:46.209 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:10:46.209 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:10:46.209 DEBUG:boto:Method: GET 2024-04-12T04:10:46.209 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:10:46.209 DEBUG:boto:Data: 2024-04-12T04:10:46.210 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.210 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.210 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.210 DEBUG:boto:Params: {} 2024-04-12T04:10:46.210 DEBUG:boto:Token: None 2024-04-12T04:10:46.210 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj29 2024-04-12T04:10:46.210 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W9U58kAtHbSRSt0OzI8C4FGcR1U= 2024-04-12T04:10:46.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W9U58kAtHbSRSt0OzI8C4FGcR1U=', 'Content-Length': '0'} 2024-04-12T04:10:46.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1bec1c75e509236-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.212 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:10:46.212 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:10:46.212 DEBUG:boto:Method: GET 2024-04-12T04:10:46.213 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:10:46.213 DEBUG:boto:Data: 2024-04-12T04:10:46.213 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.213 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.213 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.213 DEBUG:boto:Params: {} 2024-04-12T04:10:46.213 DEBUG:boto:Token: None 2024-04-12T04:10:46.213 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj29 2024-04-12T04:10:46.213 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W9U58kAtHbSRSt0OzI8C4FGcR1U= 2024-04-12T04:10:46.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W9U58kAtHbSRSt0OzI8C4FGcR1U=', 'Content-Length': '0'} 2024-04-12T04:10:46.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004258c65c1b2bb53e-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.215 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.215 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.216 DEBUG:boto:path=/dkyges-61/?prefix=obj29 2024-04-12T04:10:46.216 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj29 2024-04-12T04:10:46.216 DEBUG:boto:Method: GET 2024-04-12T04:10:46.216 DEBUG:boto:Path: /dkyges-61/?prefix=obj29 2024-04-12T04:10:46.216 DEBUG:boto:Data: 2024-04-12T04:10:46.216 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.216 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.216 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.216 DEBUG:boto:Params: {} 2024-04-12T04:10:46.216 DEBUG:boto:Token: None 2024-04-12T04:10:46.216 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.217 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a86270d78c359eb3-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.219 DEBUG:boto:b'dkyges-61obj291000falseobj292024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.219 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.220 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.220 DEBUG:boto:path=/dkyges-61/?prefix=obj29 2024-04-12T04:10:46.220 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj29 2024-04-12T04:10:46.220 DEBUG:boto:Method: GET 2024-04-12T04:10:46.220 DEBUG:boto:Path: /dkyges-61/?prefix=obj29 2024-04-12T04:10:46.220 DEBUG:boto:Data: 2024-04-12T04:10:46.220 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.220 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.220 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.220 DEBUG:boto:Params: {} 2024-04-12T04:10:46.220 DEBUG:boto:Token: None 2024-04-12T04:10:46.221 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.221 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7089c18f06146df-006618b446-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.266 DEBUG:boto:b'dkyges-61obj291000falseobj292024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T04:10:46.267 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:10:46.267 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:10:46.267 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.267 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.267 DEBUG:boto:Method: GET 2024-04-12T04:10:46.267 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.267 DEBUG:boto:Data: 2024-04-12T04:10:46.267 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.268 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.268 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.268 DEBUG:boto:Params: {} 2024-04-12T04:10:46.268 DEBUG:boto:Token: None 2024-04-12T04:10:46.268 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.268 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5KSg2yzwbDn2S7vhjON8sDoc7GM= 2024-04-12T04:10:46.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5KSg2yzwbDn2S7vhjON8sDoc7GM=', 'Content-Length': '0'} 2024-04-12T04:10:46.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e6659d8e990c707-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.271 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:10:46.271 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:10:46.271 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.271 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.271 DEBUG:boto:Method: GET 2024-04-12T04:10:46.271 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.271 DEBUG:boto:Data: 2024-04-12T04:10:46.271 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.271 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.271 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.271 DEBUG:boto:Params: {} 2024-04-12T04:10:46.272 DEBUG:boto:Token: None 2024-04-12T04:10:46.272 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.272 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5KSg2yzwbDn2S7vhjON8sDoc7GM= 2024-04-12T04:10:46.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5KSg2yzwbDn2S7vhjON8sDoc7GM=', 'Content-Length': '0'} 2024-04-12T04:10:46.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046601dc58705fa5f-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.274 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:10:46.274 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:10:46.275 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.275 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.275 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.275 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.275 DEBUG:boto:Data: 2024-04-12T04:10:46.275 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.275 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.275 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.275 DEBUG:boto:Params: {} 2024-04-12T04:10:46.275 DEBUG:boto:Token: None 2024-04-12T04:10:46.275 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.276 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1W70xmLFM4wIucRXO2gg0S3GMuU= 2024-04-12T04:10:46.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1W70xmLFM4wIucRXO2gg0S3GMuU=', 'Content-Length': '0'} 2024-04-12T04:10:46.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000131c54b10587636a-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.278 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:10:46.278 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:10:46.278 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.278 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.278 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.278 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.278 DEBUG:boto:Data: 2024-04-12T04:10:46.278 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.278 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.278 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.278 DEBUG:boto:Params: {} 2024-04-12T04:10:46.279 DEBUG:boto:Token: None 2024-04-12T04:10:46.279 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.279 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1W70xmLFM4wIucRXO2gg0S3GMuU= 2024-04-12T04:10:46.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1W70xmLFM4wIucRXO2gg0S3GMuU=', 'Content-Length': '0'} 2024-04-12T04:10:46.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000970ba3be9e2d4ad-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T04:10:46.281 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:10:46.281 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:10:46.281 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.281 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.282 DEBUG:boto:Method: GET 2024-04-12T04:10:46.282 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.282 DEBUG:boto:Data: 2024-04-12T04:10:46.282 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.282 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.282 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.282 DEBUG:boto:Params: {} 2024-04-12T04:10:46.282 DEBUG:boto:Token: None 2024-04-12T04:10:46.282 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.282 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5KSg2yzwbDn2S7vhjON8sDoc7GM= 2024-04-12T04:10:46.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5KSg2yzwbDn2S7vhjON8sDoc7GM=', 'Content-Length': '0'} 2024-04-12T04:10:46.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be048632bdfdb3d7-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.330 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:10:46.331 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:10:46.331 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.331 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.331 DEBUG:boto:Method: GET 2024-04-12T04:10:46.331 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.331 DEBUG:boto:Data: 2024-04-12T04:10:46.331 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.331 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.331 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.331 DEBUG:boto:Params: {} 2024-04-12T04:10:46.331 DEBUG:boto:Token: None 2024-04-12T04:10:46.332 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:10:46.332 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5KSg2yzwbDn2S7vhjON8sDoc7GM= 2024-04-12T04:10:46.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5KSg2yzwbDn2S7vhjON8sDoc7GM=', 'Content-Length': '0'} 2024-04-12T04:10:46.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059b05e17ad22dd88-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.335 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.335 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.335 DEBUG:boto:path=/dkyges-61/?prefix=obj31 2024-04-12T04:10:46.335 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj31 2024-04-12T04:10:46.335 DEBUG:boto:Method: GET 2024-04-12T04:10:46.335 DEBUG:boto:Path: /dkyges-61/?prefix=obj31 2024-04-12T04:10:46.335 DEBUG:boto:Data: 2024-04-12T04:10:46.335 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.335 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.335 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.335 DEBUG:boto:Params: {} 2024-04-12T04:10:46.336 DEBUG:boto:Token: None 2024-04-12T04:10:46.336 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.336 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.338 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097cb8d388a1d2313-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.339 DEBUG:boto:b'dkyges-61obj311000falseobj312024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.339 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.339 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.339 DEBUG:boto:path=/dkyges-61/?prefix=obj31 2024-04-12T04:10:46.339 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj31 2024-04-12T04:10:46.339 DEBUG:boto:Method: GET 2024-04-12T04:10:46.339 DEBUG:boto:Path: /dkyges-61/?prefix=obj31 2024-04-12T04:10:46.339 DEBUG:boto:Data: 2024-04-12T04:10:46.339 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.340 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.340 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.340 DEBUG:boto:Params: {} 2024-04-12T04:10:46.340 DEBUG:boto:Token: None 2024-04-12T04:10:46.340 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.340 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae9ace933077afc7-006618b446-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.386 DEBUG:boto:b'dkyges-61obj311000falseobj312024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.387 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:10:46.387 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:10:46.387 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.387 DEBUG:boto:Path: /dkyges-61/obj31 2024-04-12T04:10:46.387 DEBUG:boto:Data: 2024-04-12T04:10:46.387 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.387 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.387 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.387 DEBUG:boto:Params: {} 2024-04-12T04:10:46.388 DEBUG:boto:Token: None 2024-04-12T04:10:46.388 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj31 2024-04-12T04:10:46.388 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5V2bBarNukcy5Ks1RPBURD4mE/4= 2024-04-12T04:10:46.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5V2bBarNukcy5Ks1RPBURD4mE/4=', 'Content-Length': '0'} 2024-04-12T04:10:46.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2597141b2d1f526-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.390 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:10:46.390 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:10:46.390 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.391 DEBUG:boto:Path: /dkyges-61/obj31 2024-04-12T04:10:46.391 DEBUG:boto:Data: 2024-04-12T04:10:46.391 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.391 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.391 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.391 DEBUG:boto:Params: {} 2024-04-12T04:10:46.391 DEBUG:boto:Token: None 2024-04-12T04:10:46.391 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj31 2024-04-12T04:10:46.391 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5V2bBarNukcy5Ks1RPBURD4mE/4= 2024-04-12T04:10:46.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5V2bBarNukcy5Ks1RPBURD4mE/4=', 'Content-Length': '0'} 2024-04-12T04:10:46.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a55b1293e6d78fef-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T04:10:46.394 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:10:46.394 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:10:46.394 DEBUG:boto:Method: GET 2024-04-12T04:10:46.394 DEBUG:boto:Path: /dkyges-61/obj31 2024-04-12T04:10:46.394 DEBUG:boto:Data: 2024-04-12T04:10:46.394 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.394 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.394 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.395 DEBUG:boto:Params: {} 2024-04-12T04:10:46.395 DEBUG:boto:Token: None 2024-04-12T04:10:46.395 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj31 2024-04-12T04:10:46.395 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xDni1HV0hXCFosrkYUYS1LkC5ok= 2024-04-12T04:10:46.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xDni1HV0hXCFosrkYUYS1LkC5ok=', 'Content-Length': '0'} 2024-04-12T04:10:46.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d715b99a4a452345-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.397 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:10:46.397 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:10:46.397 DEBUG:boto:Method: GET 2024-04-12T04:10:46.397 DEBUG:boto:Path: /dkyges-61/obj31 2024-04-12T04:10:46.397 DEBUG:boto:Data: 2024-04-12T04:10:46.397 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.398 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.398 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.398 DEBUG:boto:Params: {} 2024-04-12T04:10:46.398 DEBUG:boto:Token: None 2024-04-12T04:10:46.398 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj31 2024-04-12T04:10:46.398 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xDni1HV0hXCFosrkYUYS1LkC5ok= 2024-04-12T04:10:46.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xDni1HV0hXCFosrkYUYS1LkC5ok=', 'Content-Length': '0'} 2024-04-12T04:10:46.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021815b5be124e09c-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.400 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.400 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.400 DEBUG:boto:path=/dkyges-61/?prefix=obj31 2024-04-12T04:10:46.400 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj31 2024-04-12T04:10:46.401 DEBUG:boto:Method: GET 2024-04-12T04:10:46.401 DEBUG:boto:Path: /dkyges-61/?prefix=obj31 2024-04-12T04:10:46.401 DEBUG:boto:Data: 2024-04-12T04:10:46.401 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.401 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.401 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.401 DEBUG:boto:Params: {} 2024-04-12T04:10:46.401 DEBUG:boto:Token: None 2024-04-12T04:10:46.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.401 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000738630b48fb8306a-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.404 DEBUG:boto:b'dkyges-61obj311000falseobj312024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.405 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.405 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.405 DEBUG:boto:path=/dkyges-61/?prefix=obj31 2024-04-12T04:10:46.405 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj31 2024-04-12T04:10:46.405 DEBUG:boto:Method: GET 2024-04-12T04:10:46.405 DEBUG:boto:Path: /dkyges-61/?prefix=obj31 2024-04-12T04:10:46.405 DEBUG:boto:Data: 2024-04-12T04:10:46.405 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.405 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.405 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.405 DEBUG:boto:Params: {} 2024-04-12T04:10:46.406 DEBUG:boto:Token: None 2024-04-12T04:10:46.406 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.406 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e1724a86a7428e7-006618b446-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.410 DEBUG:boto:b'dkyges-61obj311000falseobj312024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:10:46.411 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:10:46.411 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:10:46.411 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.411 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.411 DEBUG:boto:Method: GET 2024-04-12T04:10:46.411 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.412 DEBUG:boto:Data: 2024-04-12T04:10:46.412 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.412 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.412 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.412 DEBUG:boto:Params: {} 2024-04-12T04:10:46.412 DEBUG:boto:Token: None 2024-04-12T04:10:46.412 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.412 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NTRy2V0JnxAqLbA+GDurFJoCkPY= 2024-04-12T04:10:46.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NTRy2V0JnxAqLbA+GDurFJoCkPY=', 'Content-Length': '0'} 2024-04-12T04:10:46.414 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000475d898efe6f4022-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.414 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:10:46.415 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:10:46.415 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.415 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.415 DEBUG:boto:Method: GET 2024-04-12T04:10:46.415 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.415 DEBUG:boto:Data: 2024-04-12T04:10:46.415 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.415 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.415 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.415 DEBUG:boto:Params: {} 2024-04-12T04:10:46.415 DEBUG:boto:Token: None 2024-04-12T04:10:46.415 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.416 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NTRy2V0JnxAqLbA+GDurFJoCkPY= 2024-04-12T04:10:46.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NTRy2V0JnxAqLbA+GDurFJoCkPY=', 'Content-Length': '0'} 2024-04-12T04:10:46.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f84331b225de85e3-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.418 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:10:46.418 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:10:46.418 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.418 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.418 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.418 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.418 DEBUG:boto:Data: 2024-04-12T04:10:46.419 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.419 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.419 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.419 DEBUG:boto:Params: {} 2024-04-12T04:10:46.419 DEBUG:boto:Token: None 2024-04-12T04:10:46.419 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.419 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/BL2Wk0pAMVQn/mVSMti/Gekzgw= 2024-04-12T04:10:46.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/BL2Wk0pAMVQn/mVSMti/Gekzgw=', 'Content-Length': '0'} 2024-04-12T04:10:46.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b6baaf9e59eae39-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.421 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:10:46.421 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:10:46.421 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.422 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.422 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.422 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.422 DEBUG:boto:Data: 2024-04-12T04:10:46.422 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.422 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.422 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.422 DEBUG:boto:Params: {} 2024-04-12T04:10:46.422 DEBUG:boto:Token: None 2024-04-12T04:10:46.422 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.422 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/BL2Wk0pAMVQn/mVSMti/Gekzgw= 2024-04-12T04:10:46.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/BL2Wk0pAMVQn/mVSMti/Gekzgw=', 'Content-Length': '0'} 2024-04-12T04:10:46.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac8a026c7f24f708-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:10:46.425 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:10:46.425 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:10:46.425 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.425 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.425 DEBUG:boto:Method: GET 2024-04-12T04:10:46.425 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.425 DEBUG:boto:Data: 2024-04-12T04:10:46.425 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.425 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.425 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.426 DEBUG:boto:Params: {} 2024-04-12T04:10:46.426 DEBUG:boto:Token: None 2024-04-12T04:10:46.426 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.426 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NTRy2V0JnxAqLbA+GDurFJoCkPY= 2024-04-12T04:10:46.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NTRy2V0JnxAqLbA+GDurFJoCkPY=', 'Content-Length': '0'} 2024-04-12T04:10:46.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f6d1f460c8c902d-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.470 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:10:46.471 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:10:46.471 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.471 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.471 DEBUG:boto:Method: GET 2024-04-12T04:10:46.471 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.472 DEBUG:boto:Data: 2024-04-12T04:10:46.472 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.472 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.472 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.472 DEBUG:boto:Params: {} 2024-04-12T04:10:46.472 DEBUG:boto:Token: None 2024-04-12T04:10:46.473 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:10:46.473 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NTRy2V0JnxAqLbA+GDurFJoCkPY= 2024-04-12T04:10:46.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NTRy2V0JnxAqLbA+GDurFJoCkPY=', 'Content-Length': '0'} 2024-04-12T04:10:46.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a95e109c17d42f0-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.476 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.476 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.476 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:10:46.476 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:10:46.476 DEBUG:boto:Method: GET 2024-04-12T04:10:46.477 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:10:46.477 DEBUG:boto:Data: 2024-04-12T04:10:46.477 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.477 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.477 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.477 DEBUG:boto:Params: {} 2024-04-12T04:10:46.477 DEBUG:boto:Token: None 2024-04-12T04:10:46.478 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.478 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d690e711ac246dc8-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.481 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.481 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.481 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.481 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:10:46.481 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:10:46.482 DEBUG:boto:Method: GET 2024-04-12T04:10:46.482 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:10:46.482 DEBUG:boto:Data: 2024-04-12T04:10:46.482 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.482 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.482 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.482 DEBUG:boto:Params: {} 2024-04-12T04:10:46.482 DEBUG:boto:Token: None 2024-04-12T04:10:46.482 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.482 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b77a182183a6e792-006618b446-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.530 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.531 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:10:46.531 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:10:46.531 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.531 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:10:46.531 DEBUG:boto:Data: 2024-04-12T04:10:46.531 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.531 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.531 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.531 DEBUG:boto:Params: {} 2024-04-12T04:10:46.532 DEBUG:boto:Token: None 2024-04-12T04:10:46.532 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj33 2024-04-12T04:10:46.532 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TDkzpysoP0XS+jzUQ0SoPy7LIWY= 2024-04-12T04:10:46.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TDkzpysoP0XS+jzUQ0SoPy7LIWY=', 'Content-Length': '0'} 2024-04-12T04:10:46.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2350d7f69418d76-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.534 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:10:46.534 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:10:46.534 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.534 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:10:46.534 DEBUG:boto:Data: 2024-04-12T04:10:46.534 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.534 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.534 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.535 DEBUG:boto:Params: {} 2024-04-12T04:10:46.535 DEBUG:boto:Token: None 2024-04-12T04:10:46.535 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj33 2024-04-12T04:10:46.535 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TDkzpysoP0XS+jzUQ0SoPy7LIWY= 2024-04-12T04:10:46.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TDkzpysoP0XS+jzUQ0SoPy7LIWY=', 'Content-Length': '0'} 2024-04-12T04:10:46.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca21c9cfb80ce44d-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:10:46.537 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:10:46.537 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:10:46.538 DEBUG:boto:Method: GET 2024-04-12T04:10:46.538 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:10:46.538 DEBUG:boto:Data: 2024-04-12T04:10:46.538 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.538 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.538 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.538 DEBUG:boto:Params: {} 2024-04-12T04:10:46.538 DEBUG:boto:Token: None 2024-04-12T04:10:46.538 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj33 2024-04-12T04:10:46.539 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oyBxKBFhMeYV6r2W4xCN2PwYJsM= 2024-04-12T04:10:46.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oyBxKBFhMeYV6r2W4xCN2PwYJsM=', 'Content-Length': '0'} 2024-04-12T04:10:46.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076387ad4132fbec9-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.540 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:10:46.541 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:10:46.541 DEBUG:boto:Method: GET 2024-04-12T04:10:46.541 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:10:46.541 DEBUG:boto:Data: 2024-04-12T04:10:46.541 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.541 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.541 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.541 DEBUG:boto:Params: {} 2024-04-12T04:10:46.541 DEBUG:boto:Token: None 2024-04-12T04:10:46.542 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj33 2024-04-12T04:10:46.542 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oyBxKBFhMeYV6r2W4xCN2PwYJsM= 2024-04-12T04:10:46.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oyBxKBFhMeYV6r2W4xCN2PwYJsM=', 'Content-Length': '0'} 2024-04-12T04:10:46.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1656ddd6e8d81cd-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.544 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.544 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.544 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:10:46.544 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:10:46.545 DEBUG:boto:Method: GET 2024-04-12T04:10:46.545 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:10:46.545 DEBUG:boto:Data: 2024-04-12T04:10:46.545 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.545 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.545 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.545 DEBUG:boto:Params: {} 2024-04-12T04:10:46.545 DEBUG:boto:Token: None 2024-04-12T04:10:46.545 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.546 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa33488170eff5b4-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.548 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.549 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.549 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.549 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:10:46.549 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:10:46.549 DEBUG:boto:Method: GET 2024-04-12T04:10:46.549 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:10:46.549 DEBUG:boto:Data: 2024-04-12T04:10:46.549 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.549 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.549 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.549 DEBUG:boto:Params: {} 2024-04-12T04:10:46.550 DEBUG:boto:Token: None 2024-04-12T04:10:46.550 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.550 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023cef2543fd3b04e-006618b446-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.554 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:10:46.555 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:10:46.555 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:10:46.555 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.555 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.555 DEBUG:boto:Method: GET 2024-04-12T04:10:46.555 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.555 DEBUG:boto:Data: 2024-04-12T04:10:46.556 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.556 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.556 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.556 DEBUG:boto:Params: {} 2024-04-12T04:10:46.556 DEBUG:boto:Token: None 2024-04-12T04:10:46.556 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.556 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fvqw2tTrmpH5SIb7zREfCXmqlGQ= 2024-04-12T04:10:46.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fvqw2tTrmpH5SIb7zREfCXmqlGQ=', 'Content-Length': '0'} 2024-04-12T04:10:46.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee009a851b427fb5-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.558 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:10:46.558 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:10:46.558 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.559 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.559 DEBUG:boto:Method: GET 2024-04-12T04:10:46.559 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.559 DEBUG:boto:Data: 2024-04-12T04:10:46.559 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.559 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.559 DEBUG:boto:Params: {} 2024-04-12T04:10:46.559 DEBUG:boto:Token: None 2024-04-12T04:10:46.559 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.559 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fvqw2tTrmpH5SIb7zREfCXmqlGQ= 2024-04-12T04:10:46.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fvqw2tTrmpH5SIb7zREfCXmqlGQ=', 'Content-Length': '0'} 2024-04-12T04:10:46.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006203d3ec4a8eb2ff-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.562 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:10:46.562 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:10:46.562 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.562 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.562 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.562 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.562 DEBUG:boto:Data: 2024-04-12T04:10:46.562 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.562 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.562 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.563 DEBUG:boto:Params: {} 2024-04-12T04:10:46.563 DEBUG:boto:Token: None 2024-04-12T04:10:46.563 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.563 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:et8YZJIcZo3v0DanaNcnpgbavUE= 2024-04-12T04:10:46.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:et8YZJIcZo3v0DanaNcnpgbavUE=', 'Content-Length': '0'} 2024-04-12T04:10:46.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d875de26e2dc390-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.565 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:10:46.565 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:10:46.565 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.565 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.565 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.565 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.565 DEBUG:boto:Data: 2024-04-12T04:10:46.566 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.566 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.566 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.566 DEBUG:boto:Params: {} 2024-04-12T04:10:46.566 DEBUG:boto:Token: None 2024-04-12T04:10:46.566 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.566 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:et8YZJIcZo3v0DanaNcnpgbavUE= 2024-04-12T04:10:46.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:et8YZJIcZo3v0DanaNcnpgbavUE=', 'Content-Length': '0'} 2024-04-12T04:10:46.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000915a046f944cb75d-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:10:46.568 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:10:46.569 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:10:46.569 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.569 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.569 DEBUG:boto:Method: GET 2024-04-12T04:10:46.569 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.569 DEBUG:boto:Data: 2024-04-12T04:10:46.569 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.569 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.569 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.569 DEBUG:boto:Params: {} 2024-04-12T04:10:46.569 DEBUG:boto:Token: None 2024-04-12T04:10:46.570 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.570 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fvqw2tTrmpH5SIb7zREfCXmqlGQ= 2024-04-12T04:10:46.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fvqw2tTrmpH5SIb7zREfCXmqlGQ=', 'Content-Length': '0'} 2024-04-12T04:10:46.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c214427077de1d95-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.572 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:10:46.572 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:10:46.572 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.572 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.572 DEBUG:boto:Method: GET 2024-04-12T04:10:46.572 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.572 DEBUG:boto:Data: 2024-04-12T04:10:46.573 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.573 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.573 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.573 DEBUG:boto:Params: {} 2024-04-12T04:10:46.573 DEBUG:boto:Token: None 2024-04-12T04:10:46.573 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:10:46.573 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fvqw2tTrmpH5SIb7zREfCXmqlGQ= 2024-04-12T04:10:46.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fvqw2tTrmpH5SIb7zREfCXmqlGQ=', 'Content-Length': '0'} 2024-04-12T04:10:46.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9485121706e5303-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.575 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.575 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.575 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:10:46.576 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:10:46.576 DEBUG:boto:Method: GET 2024-04-12T04:10:46.576 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:10:46.576 DEBUG:boto:Data: 2024-04-12T04:10:46.576 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.576 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.576 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.576 DEBUG:boto:Params: {} 2024-04-12T04:10:46.576 DEBUG:boto:Token: None 2024-04-12T04:10:46.577 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.577 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dac1067202421b1-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.579 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.580 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.580 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.580 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:10:46.580 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:10:46.580 DEBUG:boto:Method: GET 2024-04-12T04:10:46.580 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:10:46.580 DEBUG:boto:Data: 2024-04-12T04:10:46.580 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.581 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.581 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.581 DEBUG:boto:Params: {} 2024-04-12T04:10:46.581 DEBUG:boto:Token: None 2024-04-12T04:10:46.581 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.582 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fcb865eb6988b42-006618b446-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.634 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.635 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:10:46.635 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:10:46.635 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.635 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:10:46.635 DEBUG:boto:Data: 2024-04-12T04:10:46.636 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.636 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.636 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.636 DEBUG:boto:Params: {} 2024-04-12T04:10:46.636 DEBUG:boto:Token: None 2024-04-12T04:10:46.636 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj34 2024-04-12T04:10:46.636 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kRjays8dyCfzsUpNc6hCG9dctfc= 2024-04-12T04:10:46.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kRjays8dyCfzsUpNc6hCG9dctfc=', 'Content-Length': '0'} 2024-04-12T04:10:46.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008062eab53ab4c68d-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.639 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:10:46.639 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:10:46.639 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.639 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:10:46.639 DEBUG:boto:Data: 2024-04-12T04:10:46.639 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.639 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.639 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.639 DEBUG:boto:Params: {} 2024-04-12T04:10:46.639 DEBUG:boto:Token: None 2024-04-12T04:10:46.639 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj34 2024-04-12T04:10:46.640 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kRjays8dyCfzsUpNc6hCG9dctfc= 2024-04-12T04:10:46.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kRjays8dyCfzsUpNc6hCG9dctfc=', 'Content-Length': '0'} 2024-04-12T04:10:46.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004558b58a708ba743-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:10:46.642 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:10:46.642 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:10:46.642 DEBUG:boto:Method: GET 2024-04-12T04:10:46.642 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:10:46.643 DEBUG:boto:Data: 2024-04-12T04:10:46.643 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.643 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.643 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.643 DEBUG:boto:Params: {} 2024-04-12T04:10:46.643 DEBUG:boto:Token: None 2024-04-12T04:10:46.643 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj34 2024-04-12T04:10:46.643 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uAdRfy2t1JkDIWDsP8qjs7YVcfQ= 2024-04-12T04:10:46.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uAdRfy2t1JkDIWDsP8qjs7YVcfQ=', 'Content-Length': '0'} 2024-04-12T04:10:46.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff38a24f41717168-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.645 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:10:46.645 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:10:46.645 DEBUG:boto:Method: GET 2024-04-12T04:10:46.645 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:10:46.645 DEBUG:boto:Data: 2024-04-12T04:10:46.645 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.646 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.646 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.646 DEBUG:boto:Params: {} 2024-04-12T04:10:46.646 DEBUG:boto:Token: None 2024-04-12T04:10:46.646 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj34 2024-04-12T04:10:46.646 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uAdRfy2t1JkDIWDsP8qjs7YVcfQ= 2024-04-12T04:10:46.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uAdRfy2t1JkDIWDsP8qjs7YVcfQ=', 'Content-Length': '0'} 2024-04-12T04:10:46.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000008010672d69b4ef-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.648 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.648 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.648 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:10:46.649 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:10:46.649 DEBUG:boto:Method: GET 2024-04-12T04:10:46.649 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:10:46.649 DEBUG:boto:Data: 2024-04-12T04:10:46.649 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.649 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.649 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.649 DEBUG:boto:Params: {} 2024-04-12T04:10:46.649 DEBUG:boto:Token: None 2024-04-12T04:10:46.650 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.650 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085306ca6479aae24-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.652 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.653 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.653 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.653 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:10:46.653 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:10:46.653 DEBUG:boto:Method: GET 2024-04-12T04:10:46.653 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:10:46.653 DEBUG:boto:Data: 2024-04-12T04:10:46.653 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.653 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.654 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.654 DEBUG:boto:Params: {} 2024-04-12T04:10:46.654 DEBUG:boto:Token: None 2024-04-12T04:10:46.654 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.654 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000148763ffb74f3eb5-006618b446-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.659 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T04:10:46.659 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:10:46.659 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:10:46.660 DEBUG:boto:path=/dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.660 DEBUG:boto:auth_path=/dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.660 DEBUG:boto:Method: GET 2024-04-12T04:10:46.660 DEBUG:boto:Path: /dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.660 DEBUG:boto:Data: 2024-04-12T04:10:46.660 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.660 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.660 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.660 DEBUG:boto:Params: {} 2024-04-12T04:10:46.660 DEBUG:boto:Token: None 2024-04-12T04:10:46.660 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.660 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7pnwbcznBkppwlnMX/zCTyo/RRs= 2024-04-12T04:10:46.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7pnwbcznBkppwlnMX/zCTyo/RRs=', 'Content-Length': '0'} 2024-04-12T04:10:46.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045e7ea23e1db270f-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.663 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:10:46.663 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:10:46.663 DEBUG:boto:path=/dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.663 DEBUG:boto:auth_path=/dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.663 DEBUG:boto:Method: GET 2024-04-12T04:10:46.663 DEBUG:boto:Path: /dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.664 DEBUG:boto:Data: 2024-04-12T04:10:46.664 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.664 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.664 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.664 DEBUG:boto:Params: {} 2024-04-12T04:10:46.664 DEBUG:boto:Token: None 2024-04-12T04:10:46.664 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.664 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7pnwbcznBkppwlnMX/zCTyo/RRs= 2024-04-12T04:10:46.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7pnwbcznBkppwlnMX/zCTyo/RRs=', 'Content-Length': '0'} 2024-04-12T04:10:46.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e935ce57f2286ec9-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.666 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:10:46.666 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:10:46.666 DEBUG:boto:path=/dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.667 DEBUG:boto:auth_path=/dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.667 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.667 DEBUG:boto:Path: /dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.667 DEBUG:boto:Data: 2024-04-12T04:10:46.667 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.667 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.667 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.667 DEBUG:boto:Params: {} 2024-04-12T04:10:46.667 DEBUG:boto:Token: None 2024-04-12T04:10:46.667 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.667 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CHYiQe+OdT+imj1kz4qsbjkFRPY= 2024-04-12T04:10:46.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CHYiQe+OdT+imj1kz4qsbjkFRPY=', 'Content-Length': '0'} 2024-04-12T04:10:46.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3e89481e53be73f-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.669 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:10:46.670 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:10:46.670 DEBUG:boto:path=/dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.670 DEBUG:boto:auth_path=/dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.670 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.670 DEBUG:boto:Path: /dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.670 DEBUG:boto:Data: 2024-04-12T04:10:46.670 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.670 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.670 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.671 DEBUG:boto:Params: {} 2024-04-12T04:10:46.671 DEBUG:boto:Token: None 2024-04-12T04:10:46.671 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.671 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CHYiQe+OdT+imj1kz4qsbjkFRPY= 2024-04-12T04:10:46.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CHYiQe+OdT+imj1kz4qsbjkFRPY=', 'Content-Length': '0'} 2024-04-12T04:10:46.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040a5f46a1c0c2d8c-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T04:10:46.673 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:10:46.673 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:10:46.673 DEBUG:boto:path=/dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.674 DEBUG:boto:auth_path=/dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.674 DEBUG:boto:Method: GET 2024-04-12T04:10:46.674 DEBUG:boto:Path: /dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.674 DEBUG:boto:Data: 2024-04-12T04:10:46.674 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.674 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.674 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.674 DEBUG:boto:Params: {} 2024-04-12T04:10:46.674 DEBUG:boto:Token: None 2024-04-12T04:10:46.674 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.675 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7pnwbcznBkppwlnMX/zCTyo/RRs= 2024-04-12T04:10:46.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7pnwbcznBkppwlnMX/zCTyo/RRs=', 'Content-Length': '0'} 2024-04-12T04:10:46.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013be721ca971e7a3-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.718 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:10:46.718 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:10:46.718 DEBUG:boto:path=/dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.719 DEBUG:boto:auth_path=/dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.719 DEBUG:boto:Method: GET 2024-04-12T04:10:46.719 DEBUG:boto:Path: /dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.719 DEBUG:boto:Data: 2024-04-12T04:10:46.719 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.719 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.719 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.719 DEBUG:boto:Params: {} 2024-04-12T04:10:46.719 DEBUG:boto:Token: None 2024-04-12T04:10:46.719 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj37?versionId=null 2024-04-12T04:10:46.719 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7pnwbcznBkppwlnMX/zCTyo/RRs= 2024-04-12T04:10:46.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7pnwbcznBkppwlnMX/zCTyo/RRs=', 'Content-Length': '0'} 2024-04-12T04:10:46.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d95dc359d0364bc-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.722 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.722 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.722 DEBUG:boto:path=/dkyges-61/?prefix=obj37 2024-04-12T04:10:46.722 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj37 2024-04-12T04:10:46.722 DEBUG:boto:Method: GET 2024-04-12T04:10:46.722 DEBUG:boto:Path: /dkyges-61/?prefix=obj37 2024-04-12T04:10:46.723 DEBUG:boto:Data: 2024-04-12T04:10:46.723 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.723 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.723 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.723 DEBUG:boto:Params: {} 2024-04-12T04:10:46.723 DEBUG:boto:Token: None 2024-04-12T04:10:46.723 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.723 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055f2f2a6aa66f808-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.726 DEBUG:boto:b'dkyges-61obj371000falseobj372024-04-12T04:09:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.726 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.726 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.726 DEBUG:boto:path=/dkyges-61/?prefix=obj37 2024-04-12T04:10:46.726 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj37 2024-04-12T04:10:46.727 DEBUG:boto:Method: GET 2024-04-12T04:10:46.727 DEBUG:boto:Path: /dkyges-61/?prefix=obj37 2024-04-12T04:10:46.727 DEBUG:boto:Data: 2024-04-12T04:10:46.727 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.727 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.727 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.727 DEBUG:boto:Params: {} 2024-04-12T04:10:46.727 DEBUG:boto:Token: None 2024-04-12T04:10:46.727 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.727 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc59de6a2ece3ccc-006618b446-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.732 DEBUG:boto:b'dkyges-61obj371000falseobj372024-04-12T04:09:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.732 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:10:46.732 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:10:46.733 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.733 DEBUG:boto:Path: /dkyges-61/obj37 2024-04-12T04:10:46.733 DEBUG:boto:Data: 2024-04-12T04:10:46.733 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.733 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.733 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.733 DEBUG:boto:Params: {} 2024-04-12T04:10:46.733 DEBUG:boto:Token: None 2024-04-12T04:10:46.733 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj37 2024-04-12T04:10:46.733 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:76xcof4G6D9/Mr21KhsWAU+h5Y0= 2024-04-12T04:10:46.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:76xcof4G6D9/Mr21KhsWAU+h5Y0=', 'Content-Length': '0'} 2024-04-12T04:10:46.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d99a1bf642dea872-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.736 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:10:46.736 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:10:46.736 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.736 DEBUG:boto:Path: /dkyges-61/obj37 2024-04-12T04:10:46.736 DEBUG:boto:Data: 2024-04-12T04:10:46.736 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.736 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.736 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.737 DEBUG:boto:Params: {} 2024-04-12T04:10:46.737 DEBUG:boto:Token: None 2024-04-12T04:10:46.737 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj37 2024-04-12T04:10:46.737 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:76xcof4G6D9/Mr21KhsWAU+h5Y0= 2024-04-12T04:10:46.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:76xcof4G6D9/Mr21KhsWAU+h5Y0=', 'Content-Length': '0'} 2024-04-12T04:10:46.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f1c38b4ef2e6b79-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T04:10:46.739 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:10:46.739 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:10:46.740 DEBUG:boto:Method: GET 2024-04-12T04:10:46.740 DEBUG:boto:Path: /dkyges-61/obj37 2024-04-12T04:10:46.740 DEBUG:boto:Data: 2024-04-12T04:10:46.740 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.740 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.740 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.740 DEBUG:boto:Params: {} 2024-04-12T04:10:46.740 DEBUG:boto:Token: None 2024-04-12T04:10:46.740 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj37 2024-04-12T04:10:46.740 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/z3MYCMzt+qEFLFu3fRnfrHI4bc= 2024-04-12T04:10:46.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/z3MYCMzt+qEFLFu3fRnfrHI4bc=', 'Content-Length': '0'} 2024-04-12T04:10:46.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2d9e263b1d71fe7-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.743 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:10:46.743 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:10:46.743 DEBUG:boto:Method: GET 2024-04-12T04:10:46.743 DEBUG:boto:Path: /dkyges-61/obj37 2024-04-12T04:10:46.743 DEBUG:boto:Data: 2024-04-12T04:10:46.743 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.743 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.743 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.743 DEBUG:boto:Params: {} 2024-04-12T04:10:46.743 DEBUG:boto:Token: None 2024-04-12T04:10:46.743 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj37 2024-04-12T04:10:46.744 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/z3MYCMzt+qEFLFu3fRnfrHI4bc= 2024-04-12T04:10:46.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/z3MYCMzt+qEFLFu3fRnfrHI4bc=', 'Content-Length': '0'} 2024-04-12T04:10:46.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003181ca77ba20a4c4-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.786 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.786 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.787 DEBUG:boto:path=/dkyges-61/?prefix=obj37 2024-04-12T04:10:46.787 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj37 2024-04-12T04:10:46.787 DEBUG:boto:Method: GET 2024-04-12T04:10:46.787 DEBUG:boto:Path: /dkyges-61/?prefix=obj37 2024-04-12T04:10:46.787 DEBUG:boto:Data: 2024-04-12T04:10:46.787 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.787 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.787 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.787 DEBUG:boto:Params: {} 2024-04-12T04:10:46.787 DEBUG:boto:Token: None 2024-04-12T04:10:46.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.788 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fa91b142fbdb38d-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.790 DEBUG:boto:b'dkyges-61obj371000falseobj372024-04-12T04:09:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.791 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.791 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.791 DEBUG:boto:path=/dkyges-61/?prefix=obj37 2024-04-12T04:10:46.791 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj37 2024-04-12T04:10:46.791 DEBUG:boto:Method: GET 2024-04-12T04:10:46.791 DEBUG:boto:Path: /dkyges-61/?prefix=obj37 2024-04-12T04:10:46.791 DEBUG:boto:Data: 2024-04-12T04:10:46.791 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.791 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.791 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.792 DEBUG:boto:Params: {} 2024-04-12T04:10:46.792 DEBUG:boto:Token: None 2024-04-12T04:10:46.792 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.792 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab04746536dc10cd-006618b446-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.797 DEBUG:boto:b'dkyges-61obj371000falseobj372024-04-12T04:09:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:10:46.797 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:10:46.797 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:10:46.797 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.797 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.797 DEBUG:boto:Method: GET 2024-04-12T04:10:46.797 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.797 DEBUG:boto:Data: 2024-04-12T04:10:46.798 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.798 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.798 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.798 DEBUG:boto:Params: {} 2024-04-12T04:10:46.798 DEBUG:boto:Token: None 2024-04-12T04:10:46.798 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.798 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P17iaGok3oja4qAuw5jFzWOZkSk= 2024-04-12T04:10:46.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P17iaGok3oja4qAuw5jFzWOZkSk=', 'Content-Length': '0'} 2024-04-12T04:10:46.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b6e3315b2011f46-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.842 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:10:46.842 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:10:46.842 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.843 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.843 DEBUG:boto:Method: GET 2024-04-12T04:10:46.843 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.843 DEBUG:boto:Data: 2024-04-12T04:10:46.843 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.843 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.843 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.843 DEBUG:boto:Params: {} 2024-04-12T04:10:46.843 DEBUG:boto:Token: None 2024-04-12T04:10:46.844 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.844 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P17iaGok3oja4qAuw5jFzWOZkSk= 2024-04-12T04:10:46.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P17iaGok3oja4qAuw5jFzWOZkSk=', 'Content-Length': '0'} 2024-04-12T04:10:46.846 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033acbe3bd54869a5-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.846 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:10:46.846 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:10:46.846 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.846 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.847 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.847 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.847 DEBUG:boto:Data: 2024-04-12T04:10:46.847 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.847 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.847 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.847 DEBUG:boto:Params: {} 2024-04-12T04:10:46.847 DEBUG:boto:Token: None 2024-04-12T04:10:46.847 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.847 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ywx2thlitdgb9RXKOd47rqqDLGo= 2024-04-12T04:10:46.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ywx2thlitdgb9RXKOd47rqqDLGo=', 'Content-Length': '0'} 2024-04-12T04:10:46.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093ded92fc790bb18-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.850 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:10:46.850 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:10:46.850 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.850 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.850 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.850 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.850 DEBUG:boto:Data: 2024-04-12T04:10:46.850 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.850 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.850 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.851 DEBUG:boto:Params: {} 2024-04-12T04:10:46.851 DEBUG:boto:Token: None 2024-04-12T04:10:46.851 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.851 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ywx2thlitdgb9RXKOd47rqqDLGo= 2024-04-12T04:10:46.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ywx2thlitdgb9RXKOd47rqqDLGo=', 'Content-Length': '0'} 2024-04-12T04:10:46.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007de9c05482fe9f52-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:10:46.853 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:10:46.854 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:10:46.854 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.854 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.854 DEBUG:boto:Method: GET 2024-04-12T04:10:46.854 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.854 DEBUG:boto:Data: 2024-04-12T04:10:46.854 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.854 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.854 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.854 DEBUG:boto:Params: {} 2024-04-12T04:10:46.854 DEBUG:boto:Token: None 2024-04-12T04:10:46.855 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.855 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P17iaGok3oja4qAuw5jFzWOZkSk= 2024-04-12T04:10:46.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P17iaGok3oja4qAuw5jFzWOZkSk=', 'Content-Length': '0'} 2024-04-12T04:10:46.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000520d8baea66606b3-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.857 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:10:46.857 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:10:46.857 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.857 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.857 DEBUG:boto:Method: GET 2024-04-12T04:10:46.857 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.858 DEBUG:boto:Data: 2024-04-12T04:10:46.858 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.858 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.858 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.858 DEBUG:boto:Params: {} 2024-04-12T04:10:46.858 DEBUG:boto:Token: None 2024-04-12T04:10:46.858 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:10:46.858 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P17iaGok3oja4qAuw5jFzWOZkSk= 2024-04-12T04:10:46.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P17iaGok3oja4qAuw5jFzWOZkSk=', 'Content-Length': '0'} 2024-04-12T04:10:46.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d1562bf39b79ddf-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.861 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.861 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.861 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:10:46.861 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:10:46.861 DEBUG:boto:Method: GET 2024-04-12T04:10:46.861 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:10:46.861 DEBUG:boto:Data: 2024-04-12T04:10:46.861 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.861 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.861 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.861 DEBUG:boto:Params: {} 2024-04-12T04:10:46.862 DEBUG:boto:Token: None 2024-04-12T04:10:46.862 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.862 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efa4cb4599ab1a89-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.865 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.865 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.865 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.865 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:10:46.865 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:10:46.865 DEBUG:boto:Method: GET 2024-04-12T04:10:46.865 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:10:46.866 DEBUG:boto:Data: 2024-04-12T04:10:46.866 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.866 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.866 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.866 DEBUG:boto:Params: {} 2024-04-12T04:10:46.866 DEBUG:boto:Token: None 2024-04-12T04:10:46.866 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.866 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000367e02c33bd79234-006618b446-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.871 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.871 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:10:46.871 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:10:46.871 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.872 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:10:46.872 DEBUG:boto:Data: 2024-04-12T04:10:46.872 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.872 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.872 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.872 DEBUG:boto:Params: {} 2024-04-12T04:10:46.872 DEBUG:boto:Token: None 2024-04-12T04:10:46.872 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj38 2024-04-12T04:10:46.872 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:klVy6fAYD2nJIgQ39U1UDfvh80I= 2024-04-12T04:10:46.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:klVy6fAYD2nJIgQ39U1UDfvh80I=', 'Content-Length': '0'} 2024-04-12T04:10:46.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096a84a722ac082ad-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.874 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:10:46.874 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:10:46.875 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.875 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:10:46.875 DEBUG:boto:Data: 2024-04-12T04:10:46.875 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.875 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.875 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.875 DEBUG:boto:Params: {} 2024-04-12T04:10:46.875 DEBUG:boto:Token: None 2024-04-12T04:10:46.875 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj38 2024-04-12T04:10:46.875 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:klVy6fAYD2nJIgQ39U1UDfvh80I= 2024-04-12T04:10:46.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:klVy6fAYD2nJIgQ39U1UDfvh80I=', 'Content-Length': '0'} 2024-04-12T04:10:46.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000182c97e5ce9fb9e-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:10:46.878 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:10:46.878 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:10:46.878 DEBUG:boto:Method: GET 2024-04-12T04:10:46.878 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:10:46.878 DEBUG:boto:Data: 2024-04-12T04:10:46.878 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.878 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.879 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.879 DEBUG:boto:Params: {} 2024-04-12T04:10:46.879 DEBUG:boto:Token: None 2024-04-12T04:10:46.879 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj38 2024-04-12T04:10:46.879 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pKX8jjlxI/zplHsJ315QldM32Zs= 2024-04-12T04:10:46.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pKX8jjlxI/zplHsJ315QldM32Zs=', 'Content-Length': '0'} 2024-04-12T04:10:46.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc35ebe37f4e6881-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.881 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:10:46.881 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:10:46.881 DEBUG:boto:Method: GET 2024-04-12T04:10:46.881 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:10:46.881 DEBUG:boto:Data: 2024-04-12T04:10:46.881 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.881 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.882 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.882 DEBUG:boto:Params: {} 2024-04-12T04:10:46.882 DEBUG:boto:Token: None 2024-04-12T04:10:46.882 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj38 2024-04-12T04:10:46.882 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pKX8jjlxI/zplHsJ315QldM32Zs= 2024-04-12T04:10:46.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pKX8jjlxI/zplHsJ315QldM32Zs=', 'Content-Length': '0'} 2024-04-12T04:10:46.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4d1c2961b38ce9d-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.884 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.884 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.884 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:10:46.884 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:10:46.884 DEBUG:boto:Method: GET 2024-04-12T04:10:46.884 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:10:46.885 DEBUG:boto:Data: 2024-04-12T04:10:46.885 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.885 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.885 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.885 DEBUG:boto:Params: {} 2024-04-12T04:10:46.885 DEBUG:boto:Token: None 2024-04-12T04:10:46.885 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.885 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc9e6c603743d61f-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.888 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.888 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.888 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.888 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:10:46.888 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:10:46.889 DEBUG:boto:Method: GET 2024-04-12T04:10:46.889 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:10:46.889 DEBUG:boto:Data: 2024-04-12T04:10:46.889 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.889 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.889 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.889 DEBUG:boto:Params: {} 2024-04-12T04:10:46.889 DEBUG:boto:Token: None 2024-04-12T04:10:46.889 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.889 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d3e3bbdcc90e529-006618b446-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.894 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:10:46.894 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:10:46.894 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:10:46.894 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.894 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.895 DEBUG:boto:Method: GET 2024-04-12T04:10:46.895 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.895 DEBUG:boto:Data: 2024-04-12T04:10:46.895 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.895 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.895 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.895 DEBUG:boto:Params: {} 2024-04-12T04:10:46.895 DEBUG:boto:Token: None 2024-04-12T04:10:46.895 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.895 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hBkZnI1Tjk8a+dDi001n6z8Lb/w= 2024-04-12T04:10:46.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hBkZnI1Tjk8a+dDi001n6z8Lb/w=', 'Content-Length': '0'} 2024-04-12T04:10:46.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b010a24967fa41c-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.898 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:10:46.898 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:10:46.898 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.898 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.898 DEBUG:boto:Method: GET 2024-04-12T04:10:46.898 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.898 DEBUG:boto:Data: 2024-04-12T04:10:46.898 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.898 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.899 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.899 DEBUG:boto:Params: {} 2024-04-12T04:10:46.899 DEBUG:boto:Token: None 2024-04-12T04:10:46.899 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.899 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hBkZnI1Tjk8a+dDi001n6z8Lb/w= 2024-04-12T04:10:46.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hBkZnI1Tjk8a+dDi001n6z8Lb/w=', 'Content-Length': '0'} 2024-04-12T04:10:46.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f27711ba0481189b-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.942 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:10:46.942 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:10:46.942 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.943 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.943 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.943 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.943 DEBUG:boto:Data: 2024-04-12T04:10:46.943 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.943 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.943 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.943 DEBUG:boto:Params: {} 2024-04-12T04:10:46.943 DEBUG:boto:Token: None 2024-04-12T04:10:46.944 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.944 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JGLpOe9rAeh1XksMTZupFYxzpqI= 2024-04-12T04:10:46.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JGLpOe9rAeh1XksMTZupFYxzpqI=', 'Content-Length': '0'} 2024-04-12T04:10:46.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c68fa60be7817fa1-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.946 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:10:46.946 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:10:46.946 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.946 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.946 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.946 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.946 DEBUG:boto:Data: 2024-04-12T04:10:46.946 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.947 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.947 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.947 DEBUG:boto:Params: {} 2024-04-12T04:10:46.947 DEBUG:boto:Token: None 2024-04-12T04:10:46.947 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.947 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JGLpOe9rAeh1XksMTZupFYxzpqI= 2024-04-12T04:10:46.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JGLpOe9rAeh1XksMTZupFYxzpqI=', 'Content-Length': '0'} 2024-04-12T04:10:46.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e058c96cc8c0771e-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:10:46.949 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:10:46.949 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:10:46.949 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.950 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.950 DEBUG:boto:Method: GET 2024-04-12T04:10:46.950 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.950 DEBUG:boto:Data: 2024-04-12T04:10:46.950 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.950 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.950 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.950 DEBUG:boto:Params: {} 2024-04-12T04:10:46.950 DEBUG:boto:Token: None 2024-04-12T04:10:46.950 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.951 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hBkZnI1Tjk8a+dDi001n6z8Lb/w= 2024-04-12T04:10:46.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hBkZnI1Tjk8a+dDi001n6z8Lb/w=', 'Content-Length': '0'} 2024-04-12T04:10:46.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000306a0ca2c31fc513-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.953 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:10:46.953 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:10:46.953 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.953 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.953 DEBUG:boto:Method: GET 2024-04-12T04:10:46.953 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.953 DEBUG:boto:Data: 2024-04-12T04:10:46.954 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.954 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.954 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.954 DEBUG:boto:Params: {} 2024-04-12T04:10:46.954 DEBUG:boto:Token: None 2024-04-12T04:10:46.954 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:10:46.954 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hBkZnI1Tjk8a+dDi001n6z8Lb/w= 2024-04-12T04:10:46.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hBkZnI1Tjk8a+dDi001n6z8Lb/w=', 'Content-Length': '0'} 2024-04-12T04:10:46.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028eefe8ea1592df9-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.956 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.956 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.957 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:10:46.957 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:10:46.957 DEBUG:boto:Method: GET 2024-04-12T04:10:46.957 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:10:46.957 DEBUG:boto:Data: 2024-04-12T04:10:46.957 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.957 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.957 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.957 DEBUG:boto:Params: {} 2024-04-12T04:10:46.957 DEBUG:boto:Token: None 2024-04-12T04:10:46.957 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.958 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a58d8df0ca8fd005-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.960 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.961 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.961 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.961 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:10:46.961 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:10:46.961 DEBUG:boto:Method: GET 2024-04-12T04:10:46.961 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:10:46.961 DEBUG:boto:Data: 2024-04-12T04:10:46.961 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.961 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.961 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.962 DEBUG:boto:Params: {} 2024-04-12T04:10:46.962 DEBUG:boto:Token: None 2024-04-12T04:10:46.962 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.962 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001311795e7428c606-006618b446-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.966 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.967 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:10:46.967 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:10:46.967 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.967 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:10:46.967 DEBUG:boto:Data: 2024-04-12T04:10:46.967 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.967 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.967 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.967 DEBUG:boto:Params: {} 2024-04-12T04:10:46.968 DEBUG:boto:Token: None 2024-04-12T04:10:46.968 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj44 2024-04-12T04:10:46.968 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A24762DKkjmNXZn7uWOSWLsqdMw= 2024-04-12T04:10:46.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A24762DKkjmNXZn7uWOSWLsqdMw=', 'Content-Length': '0'} 2024-04-12T04:10:46.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000969cb0e137d0ebb7-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.970 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:10:46.970 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:10:46.970 DEBUG:boto:Method: HEAD 2024-04-12T04:10:46.970 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:10:46.970 DEBUG:boto:Data: 2024-04-12T04:10:46.970 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.971 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.971 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.971 DEBUG:boto:Params: {} 2024-04-12T04:10:46.971 DEBUG:boto:Token: None 2024-04-12T04:10:46.971 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj44 2024-04-12T04:10:46.971 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A24762DKkjmNXZn7uWOSWLsqdMw= 2024-04-12T04:10:46.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A24762DKkjmNXZn7uWOSWLsqdMw=', 'Content-Length': '0'} 2024-04-12T04:10:46.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009834b090f597a494-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:10:46.973 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:10:46.973 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:10:46.973 DEBUG:boto:Method: GET 2024-04-12T04:10:46.973 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:10:46.973 DEBUG:boto:Data: 2024-04-12T04:10:46.974 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.974 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.974 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.974 DEBUG:boto:Params: {} 2024-04-12T04:10:46.974 DEBUG:boto:Token: None 2024-04-12T04:10:46.974 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj44 2024-04-12T04:10:46.974 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rpfkd/2YCF/zS8jCpu1k9l08GNQ= 2024-04-12T04:10:46.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rpfkd/2YCF/zS8jCpu1k9l08GNQ=', 'Content-Length': '0'} 2024-04-12T04:10:46.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063300b75c1c21e72-006618b446-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.977 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:10:46.977 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:10:46.977 DEBUG:boto:Method: GET 2024-04-12T04:10:46.977 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:10:46.977 DEBUG:boto:Data: 2024-04-12T04:10:46.977 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.977 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.977 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.977 DEBUG:boto:Params: {} 2024-04-12T04:10:46.978 DEBUG:boto:Token: None 2024-04-12T04:10:46.978 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj44 2024-04-12T04:10:46.978 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rpfkd/2YCF/zS8jCpu1k9l08GNQ= 2024-04-12T04:10:46.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rpfkd/2YCF/zS8jCpu1k9l08GNQ=', 'Content-Length': '0'} 2024-04-12T04:10:46.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4e41cb0c4a24906-006618b446-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.980 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.980 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.980 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:10:46.980 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:10:46.980 DEBUG:boto:Method: GET 2024-04-12T04:10:46.981 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:10:46.981 DEBUG:boto:Data: 2024-04-12T04:10:46.981 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.981 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.981 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.981 DEBUG:boto:Params: {} 2024-04-12T04:10:46.981 DEBUG:boto:Token: None 2024-04-12T04:10:46.981 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.981 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f1627cbd2c64d19-006618b446-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.984 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.984 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:46.985 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:46.985 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:10:46.985 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:10:46.985 DEBUG:boto:Method: GET 2024-04-12T04:10:46.985 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:10:46.985 DEBUG:boto:Data: 2024-04-12T04:10:46.985 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.985 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:46.985 DEBUG:boto:Port: 8000 2024-04-12T04:10:46.985 DEBUG:boto:Params: {} 2024-04-12T04:10:46.986 DEBUG:boto:Token: None 2024-04-12T04:10:46.986 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/ 2024-04-12T04:10:46.986 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk= 2024-04-12T04:10:46.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gXdHQzGanONcRoT5m9gbE1kSqUk=', 'Content-Length': '0'} 2024-04-12T04:10:46.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca1713b2b9c368e0-006618b447-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:46.990 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:46.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:10:46.991 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:10:46.991 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:10:46.991 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:10:46.991 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:10:46.991 DEBUG:boto:Method: GET 2024-04-12T04:10:46.991 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:10:46.991 DEBUG:boto:Data: 2024-04-12T04:10:46.992 DEBUG:boto:Headers: {} 2024-04-12T04:10:46.992 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:46.992 DEBUG:boto:Port: 8001 2024-04-12T04:10:46.992 DEBUG:boto:Params: {} 2024-04-12T04:10:46.992 DEBUG:boto:Token: None 2024-04-12T04:10:46.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:46 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:10:46.992 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JQfgKalq5dypq3/nh/1cDXafczI= 2024-04-12T04:10:46.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JQfgKalq5dypq3/nh/1cDXafczI=', 'Content-Length': '0'} 2024-04-12T04:10:46.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f57dcff5d7cccd0e-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.038 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:10:47.038 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:10:47.039 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.039 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.039 DEBUG:boto:Method: GET 2024-04-12T04:10:47.039 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.039 DEBUG:boto:Data: 2024-04-12T04:10:47.039 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.039 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.039 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.039 DEBUG:boto:Params: {} 2024-04-12T04:10:47.039 DEBUG:boto:Token: None 2024-04-12T04:10:47.040 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.040 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WzyvHd/disVZzlpYjlFHA27Kr3c= 2024-04-12T04:10:47.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WzyvHd/disVZzlpYjlFHA27Kr3c=', 'Content-Length': '0'} 2024-04-12T04:10:47.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c058d4198e39b1a8-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.042 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:10:47.042 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:10:47.042 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.042 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.042 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.042 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.042 DEBUG:boto:Data: 2024-04-12T04:10:47.043 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.043 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.043 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.043 DEBUG:boto:Params: {} 2024-04-12T04:10:47.043 DEBUG:boto:Token: None 2024-04-12T04:10:47.043 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.043 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YMwNOmaFZlvRwhJ3bg8q5DU8ghI= 2024-04-12T04:10:47.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YMwNOmaFZlvRwhJ3bg8q5DU8ghI=', 'Content-Length': '0'} 2024-04-12T04:10:47.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007562b0c16efbdd15-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.045 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:10:47.045 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:10:47.045 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.045 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.046 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.046 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.046 DEBUG:boto:Data: 2024-04-12T04:10:47.046 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.046 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.046 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.046 DEBUG:boto:Params: {} 2024-04-12T04:10:47.046 DEBUG:boto:Token: None 2024-04-12T04:10:47.046 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.047 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YMwNOmaFZlvRwhJ3bg8q5DU8ghI= 2024-04-12T04:10:47.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YMwNOmaFZlvRwhJ3bg8q5DU8ghI=', 'Content-Length': '0'} 2024-04-12T04:10:47.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab32da468267e8ac-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:10:47.049 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:10:47.049 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:10:47.049 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.049 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.049 DEBUG:boto:Method: GET 2024-04-12T04:10:47.049 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.049 DEBUG:boto:Data: 2024-04-12T04:10:47.049 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.049 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.050 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.050 DEBUG:boto:Params: {} 2024-04-12T04:10:47.050 DEBUG:boto:Token: None 2024-04-12T04:10:47.050 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.050 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WzyvHd/disVZzlpYjlFHA27Kr3c= 2024-04-12T04:10:47.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WzyvHd/disVZzlpYjlFHA27Kr3c=', 'Content-Length': '0'} 2024-04-12T04:10:47.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d93fa4265d710c5-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.052 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:10:47.052 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:10:47.053 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.053 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.053 DEBUG:boto:Method: GET 2024-04-12T04:10:47.053 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.053 DEBUG:boto:Data: 2024-04-12T04:10:47.053 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.053 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.053 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.053 DEBUG:boto:Params: {} 2024-04-12T04:10:47.053 DEBUG:boto:Token: None 2024-04-12T04:10:47.054 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:10:47.054 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WzyvHd/disVZzlpYjlFHA27Kr3c= 2024-04-12T04:10:47.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WzyvHd/disVZzlpYjlFHA27Kr3c=', 'Content-Length': '0'} 2024-04-12T04:10:47.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046296f98fc16dbca-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.056 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.056 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.056 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:10:47.056 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:10:47.057 DEBUG:boto:Method: GET 2024-04-12T04:10:47.057 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:10:47.057 DEBUG:boto:Data: 2024-04-12T04:10:47.057 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.057 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.057 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.057 DEBUG:boto:Params: {} 2024-04-12T04:10:47.057 DEBUG:boto:Token: None 2024-04-12T04:10:47.058 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.058 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000effc2f356b11ce43-006618b447-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.060 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.061 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.061 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.061 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:10:47.061 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:10:47.061 DEBUG:boto:Method: GET 2024-04-12T04:10:47.061 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:10:47.061 DEBUG:boto:Data: 2024-04-12T04:10:47.061 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.061 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.061 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.062 DEBUG:boto:Params: {} 2024-04-12T04:10:47.062 DEBUG:boto:Token: None 2024-04-12T04:10:47.062 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.062 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f318a8b7e37ab8ad-006618b447-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.066 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.067 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:10:47.067 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:10:47.067 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.067 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:10:47.067 DEBUG:boto:Data: 2024-04-12T04:10:47.067 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.067 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.067 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.067 DEBUG:boto:Params: {} 2024-04-12T04:10:47.067 DEBUG:boto:Token: None 2024-04-12T04:10:47.068 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj46 2024-04-12T04:10:47.068 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rKTnxcX/HU+IEfIEMDzY2ZIn0X0= 2024-04-12T04:10:47.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rKTnxcX/HU+IEfIEMDzY2ZIn0X0=', 'Content-Length': '0'} 2024-04-12T04:10:47.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000540c1ca5fe2a96ca-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.070 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:10:47.070 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:10:47.070 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.070 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:10:47.070 DEBUG:boto:Data: 2024-04-12T04:10:47.070 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.070 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.070 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.071 DEBUG:boto:Params: {} 2024-04-12T04:10:47.071 DEBUG:boto:Token: None 2024-04-12T04:10:47.071 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj46 2024-04-12T04:10:47.071 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rKTnxcX/HU+IEfIEMDzY2ZIn0X0= 2024-04-12T04:10:47.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rKTnxcX/HU+IEfIEMDzY2ZIn0X0=', 'Content-Length': '0'} 2024-04-12T04:10:47.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a6c56d833c2513f-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:10:47.073 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:10:47.073 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:10:47.074 DEBUG:boto:Method: GET 2024-04-12T04:10:47.074 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:10:47.074 DEBUG:boto:Data: 2024-04-12T04:10:47.074 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.074 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.074 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.074 DEBUG:boto:Params: {} 2024-04-12T04:10:47.074 DEBUG:boto:Token: None 2024-04-12T04:10:47.074 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj46 2024-04-12T04:10:47.075 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t4qVunO4DavcuPkITtJ1geJa4BA= 2024-04-12T04:10:47.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t4qVunO4DavcuPkITtJ1geJa4BA=', 'Content-Length': '0'} 2024-04-12T04:10:47.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018a5b36f6ab847a9-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.077 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:10:47.077 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:10:47.077 DEBUG:boto:Method: GET 2024-04-12T04:10:47.077 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:10:47.077 DEBUG:boto:Data: 2024-04-12T04:10:47.077 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.077 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.077 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.077 DEBUG:boto:Params: {} 2024-04-12T04:10:47.077 DEBUG:boto:Token: None 2024-04-12T04:10:47.078 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj46 2024-04-12T04:10:47.078 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t4qVunO4DavcuPkITtJ1geJa4BA= 2024-04-12T04:10:47.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t4qVunO4DavcuPkITtJ1geJa4BA=', 'Content-Length': '0'} 2024-04-12T04:10:47.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cccd042b9c68d9a2-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.122 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.123 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.123 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:10:47.123 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:10:47.123 DEBUG:boto:Method: GET 2024-04-12T04:10:47.123 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:10:47.123 DEBUG:boto:Data: 2024-04-12T04:10:47.123 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.123 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.123 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.123 DEBUG:boto:Params: {} 2024-04-12T04:10:47.124 DEBUG:boto:Token: None 2024-04-12T04:10:47.124 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.124 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6740848fdca7807-006618b447-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.127 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.128 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.128 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.128 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:10:47.128 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:10:47.128 DEBUG:boto:Method: GET 2024-04-12T04:10:47.128 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:10:47.128 DEBUG:boto:Data: 2024-04-12T04:10:47.128 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.128 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.128 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.129 DEBUG:boto:Params: {} 2024-04-12T04:10:47.129 DEBUG:boto:Token: None 2024-04-12T04:10:47.129 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.129 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cd4452754c18fea-006618b447-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.134 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:10:47.134 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:10:47.134 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:10:47.134 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.135 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.135 DEBUG:boto:Method: GET 2024-04-12T04:10:47.135 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.135 DEBUG:boto:Data: 2024-04-12T04:10:47.135 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.135 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.135 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.135 DEBUG:boto:Params: {} 2024-04-12T04:10:47.135 DEBUG:boto:Token: None 2024-04-12T04:10:47.135 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.136 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6ZSa22LSdaomvn5UzjmLPaOKMJU= 2024-04-12T04:10:47.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6ZSa22LSdaomvn5UzjmLPaOKMJU=', 'Content-Length': '0'} 2024-04-12T04:10:47.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e65b6b935e8d37c-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.138 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:10:47.138 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:10:47.138 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.138 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.139 DEBUG:boto:Method: GET 2024-04-12T04:10:47.139 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.139 DEBUG:boto:Data: 2024-04-12T04:10:47.139 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.139 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.139 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.139 DEBUG:boto:Params: {} 2024-04-12T04:10:47.139 DEBUG:boto:Token: None 2024-04-12T04:10:47.139 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.139 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6ZSa22LSdaomvn5UzjmLPaOKMJU= 2024-04-12T04:10:47.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6ZSa22LSdaomvn5UzjmLPaOKMJU=', 'Content-Length': '0'} 2024-04-12T04:10:47.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084cbf540c17739e6-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.142 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:10:47.142 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:10:47.142 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.142 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.142 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.142 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.142 DEBUG:boto:Data: 2024-04-12T04:10:47.142 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.142 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.143 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.143 DEBUG:boto:Params: {} 2024-04-12T04:10:47.143 DEBUG:boto:Token: None 2024-04-12T04:10:47.143 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.143 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l7t8ppXrD/wKFrHp3QLVizCusEU= 2024-04-12T04:10:47.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l7t8ppXrD/wKFrHp3QLVizCusEU=', 'Content-Length': '0'} 2024-04-12T04:10:47.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c4e7557f4baa42e-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.145 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:10:47.145 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:10:47.145 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.146 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.146 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.146 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.146 DEBUG:boto:Data: 2024-04-12T04:10:47.146 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.146 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.146 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.146 DEBUG:boto:Params: {} 2024-04-12T04:10:47.146 DEBUG:boto:Token: None 2024-04-12T04:10:47.146 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.147 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l7t8ppXrD/wKFrHp3QLVizCusEU= 2024-04-12T04:10:47.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l7t8ppXrD/wKFrHp3QLVizCusEU=', 'Content-Length': '0'} 2024-04-12T04:10:47.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f6f0f912b12b8ec-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:10:47.149 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:10:47.149 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:10:47.149 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.149 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.149 DEBUG:boto:Method: GET 2024-04-12T04:10:47.149 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.149 DEBUG:boto:Data: 2024-04-12T04:10:47.150 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.150 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.150 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.150 DEBUG:boto:Params: {} 2024-04-12T04:10:47.150 DEBUG:boto:Token: None 2024-04-12T04:10:47.150 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.150 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6ZSa22LSdaomvn5UzjmLPaOKMJU= 2024-04-12T04:10:47.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6ZSa22LSdaomvn5UzjmLPaOKMJU=', 'Content-Length': '0'} 2024-04-12T04:10:47.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abbd36bef7b84d8f-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.152 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:10:47.153 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:10:47.153 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.153 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.153 DEBUG:boto:Method: GET 2024-04-12T04:10:47.153 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.153 DEBUG:boto:Data: 2024-04-12T04:10:47.153 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.153 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.153 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.153 DEBUG:boto:Params: {} 2024-04-12T04:10:47.154 DEBUG:boto:Token: None 2024-04-12T04:10:47.154 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:10:47.154 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6ZSa22LSdaomvn5UzjmLPaOKMJU= 2024-04-12T04:10:47.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6ZSa22LSdaomvn5UzjmLPaOKMJU=', 'Content-Length': '0'} 2024-04-12T04:10:47.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d155f614ffc94848-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.156 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.156 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.156 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:10:47.156 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:10:47.156 DEBUG:boto:Method: GET 2024-04-12T04:10:47.156 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:10:47.156 DEBUG:boto:Data: 2024-04-12T04:10:47.157 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.157 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.157 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.157 DEBUG:boto:Params: {} 2024-04-12T04:10:47.157 DEBUG:boto:Token: None 2024-04-12T04:10:47.157 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.157 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b9d08163689f31b-006618b447-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.160 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.160 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.160 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.161 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:10:47.161 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:10:47.161 DEBUG:boto:Method: GET 2024-04-12T04:10:47.161 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:10:47.161 DEBUG:boto:Data: 2024-04-12T04:10:47.161 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.161 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.161 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.161 DEBUG:boto:Params: {} 2024-04-12T04:10:47.161 DEBUG:boto:Token: None 2024-04-12T04:10:47.161 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.162 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005674be5ebb2e9ad-006618b447-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.166 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.167 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:10:47.167 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:10:47.167 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.167 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:10:47.167 DEBUG:boto:Data: 2024-04-12T04:10:47.167 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.167 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.167 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.167 DEBUG:boto:Params: {} 2024-04-12T04:10:47.167 DEBUG:boto:Token: None 2024-04-12T04:10:47.168 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj51 2024-04-12T04:10:47.168 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GKCZ3vvgJzcSxGxM33/xoc80yso= 2024-04-12T04:10:47.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GKCZ3vvgJzcSxGxM33/xoc80yso=', 'Content-Length': '0'} 2024-04-12T04:10:47.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000954b3fa129522c37-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.170 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:10:47.170 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:10:47.170 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.170 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:10:47.170 DEBUG:boto:Data: 2024-04-12T04:10:47.170 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.170 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.170 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.171 DEBUG:boto:Params: {} 2024-04-12T04:10:47.171 DEBUG:boto:Token: None 2024-04-12T04:10:47.171 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj51 2024-04-12T04:10:47.171 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GKCZ3vvgJzcSxGxM33/xoc80yso= 2024-04-12T04:10:47.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GKCZ3vvgJzcSxGxM33/xoc80yso=', 'Content-Length': '0'} 2024-04-12T04:10:47.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009aabcec8207cc9a7-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:10:47.173 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:10:47.173 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:10:47.173 DEBUG:boto:Method: GET 2024-04-12T04:10:47.174 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:10:47.174 DEBUG:boto:Data: 2024-04-12T04:10:47.174 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.174 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.174 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.174 DEBUG:boto:Params: {} 2024-04-12T04:10:47.174 DEBUG:boto:Token: None 2024-04-12T04:10:47.174 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj51 2024-04-12T04:10:47.174 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SQ3IcgTEiUxpn4BmKUg26dPonJ4= 2024-04-12T04:10:47.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SQ3IcgTEiUxpn4BmKUg26dPonJ4=', 'Content-Length': '0'} 2024-04-12T04:10:47.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c37061fd00c9107-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.218 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:10:47.219 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:10:47.219 DEBUG:boto:Method: GET 2024-04-12T04:10:47.219 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:10:47.219 DEBUG:boto:Data: 2024-04-12T04:10:47.219 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.219 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.219 DEBUG:boto:Params: {} 2024-04-12T04:10:47.219 DEBUG:boto:Token: None 2024-04-12T04:10:47.220 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj51 2024-04-12T04:10:47.220 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SQ3IcgTEiUxpn4BmKUg26dPonJ4= 2024-04-12T04:10:47.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SQ3IcgTEiUxpn4BmKUg26dPonJ4=', 'Content-Length': '0'} 2024-04-12T04:10:47.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f200bab053d9aa8-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.222 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.222 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.223 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:10:47.223 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:10:47.223 DEBUG:boto:Method: GET 2024-04-12T04:10:47.223 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:10:47.223 DEBUG:boto:Data: 2024-04-12T04:10:47.223 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.223 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.223 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.223 DEBUG:boto:Params: {} 2024-04-12T04:10:47.223 DEBUG:boto:Token: None 2024-04-12T04:10:47.223 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.224 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.227 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2d9fbe23eea7b9e-006618b447-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.227 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.227 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.227 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.227 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:10:47.228 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:10:47.228 DEBUG:boto:Method: GET 2024-04-12T04:10:47.228 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:10:47.228 DEBUG:boto:Data: 2024-04-12T04:10:47.228 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.228 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.228 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.228 DEBUG:boto:Params: {} 2024-04-12T04:10:47.228 DEBUG:boto:Token: None 2024-04-12T04:10:47.228 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.229 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007996e747ddfa77c6-006618b447-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.233 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:10:47.233 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:10:47.234 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:10:47.234 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.234 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.234 DEBUG:boto:Method: GET 2024-04-12T04:10:47.234 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.234 DEBUG:boto:Data: 2024-04-12T04:10:47.234 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.234 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.234 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.234 DEBUG:boto:Params: {} 2024-04-12T04:10:47.234 DEBUG:boto:Token: None 2024-04-12T04:10:47.235 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.235 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:byBXytSjiZ8RuqdSEzjlQNCqEo8= 2024-04-12T04:10:47.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:byBXytSjiZ8RuqdSEzjlQNCqEo8=', 'Content-Length': '0'} 2024-04-12T04:10:47.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed0f2e5ccb208188-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.237 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:10:47.237 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:10:47.237 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.237 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.238 DEBUG:boto:Method: GET 2024-04-12T04:10:47.238 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.238 DEBUG:boto:Data: 2024-04-12T04:10:47.238 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.238 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.238 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.238 DEBUG:boto:Params: {} 2024-04-12T04:10:47.238 DEBUG:boto:Token: None 2024-04-12T04:10:47.238 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.239 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:byBXytSjiZ8RuqdSEzjlQNCqEo8= 2024-04-12T04:10:47.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:byBXytSjiZ8RuqdSEzjlQNCqEo8=', 'Content-Length': '0'} 2024-04-12T04:10:47.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bed92ed84dd55a1-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.241 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:10:47.241 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:10:47.241 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.242 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.242 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.242 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.242 DEBUG:boto:Data: 2024-04-12T04:10:47.242 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.242 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.242 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.242 DEBUG:boto:Params: {} 2024-04-12T04:10:47.242 DEBUG:boto:Token: None 2024-04-12T04:10:47.242 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.243 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fqFsN/j0O+99SXmWuwfbJ1nE4Kk= 2024-04-12T04:10:47.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fqFsN/j0O+99SXmWuwfbJ1nE4Kk=', 'Content-Length': '0'} 2024-04-12T04:10:47.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059ce785c9d47c499-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.245 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:10:47.245 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:10:47.245 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.245 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.245 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.245 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.245 DEBUG:boto:Data: 2024-04-12T04:10:47.245 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.245 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.245 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.245 DEBUG:boto:Params: {} 2024-04-12T04:10:47.246 DEBUG:boto:Token: None 2024-04-12T04:10:47.246 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.246 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fqFsN/j0O+99SXmWuwfbJ1nE4Kk= 2024-04-12T04:10:47.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fqFsN/j0O+99SXmWuwfbJ1nE4Kk=', 'Content-Length': '0'} 2024-04-12T04:10:47.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006489040b2cae7b8b-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:10:47.248 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:10:47.248 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:10:47.248 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.248 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.249 DEBUG:boto:Method: GET 2024-04-12T04:10:47.249 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.249 DEBUG:boto:Data: 2024-04-12T04:10:47.249 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.249 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.249 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.249 DEBUG:boto:Params: {} 2024-04-12T04:10:47.249 DEBUG:boto:Token: None 2024-04-12T04:10:47.249 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.249 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:byBXytSjiZ8RuqdSEzjlQNCqEo8= 2024-04-12T04:10:47.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:byBXytSjiZ8RuqdSEzjlQNCqEo8=', 'Content-Length': '0'} 2024-04-12T04:10:47.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1bd2ab76d8da72b-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.252 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:10:47.252 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:10:47.252 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.252 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.252 DEBUG:boto:Method: GET 2024-04-12T04:10:47.252 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.252 DEBUG:boto:Data: 2024-04-12T04:10:47.252 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.252 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.253 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.253 DEBUG:boto:Params: {} 2024-04-12T04:10:47.253 DEBUG:boto:Token: None 2024-04-12T04:10:47.253 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:10:47.253 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:byBXytSjiZ8RuqdSEzjlQNCqEo8= 2024-04-12T04:10:47.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:byBXytSjiZ8RuqdSEzjlQNCqEo8=', 'Content-Length': '0'} 2024-04-12T04:10:47.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063a42f64d1b7ceff-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.255 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.256 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.256 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:10:47.256 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:10:47.256 DEBUG:boto:Method: GET 2024-04-12T04:10:47.256 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:10:47.256 DEBUG:boto:Data: 2024-04-12T04:10:47.256 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.256 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.256 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.256 DEBUG:boto:Params: {} 2024-04-12T04:10:47.256 DEBUG:boto:Token: None 2024-04-12T04:10:47.257 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.257 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e060b268d742c3fe-006618b447-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.302 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.303 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.303 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.303 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:10:47.303 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:10:47.303 DEBUG:boto:Method: GET 2024-04-12T04:10:47.303 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:10:47.303 DEBUG:boto:Data: 2024-04-12T04:10:47.304 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.304 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.304 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.304 DEBUG:boto:Params: {} 2024-04-12T04:10:47.304 DEBUG:boto:Token: None 2024-04-12T04:10:47.304 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.304 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.309 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d1a982c858f21dd-006618b447-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.309 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.309 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:10:47.310 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:10:47.310 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.310 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:10:47.310 DEBUG:boto:Data: 2024-04-12T04:10:47.310 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.310 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.310 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.310 DEBUG:boto:Params: {} 2024-04-12T04:10:47.310 DEBUG:boto:Token: None 2024-04-12T04:10:47.310 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj55 2024-04-12T04:10:47.311 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7/ukLddSqXnos6jjGpyOyZTMF6I= 2024-04-12T04:10:47.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7/ukLddSqXnos6jjGpyOyZTMF6I=', 'Content-Length': '0'} 2024-04-12T04:10:47.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000179a4f6a493ece25-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.313 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:10:47.313 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:10:47.313 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.313 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:10:47.313 DEBUG:boto:Data: 2024-04-12T04:10:47.313 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.313 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.313 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.314 DEBUG:boto:Params: {} 2024-04-12T04:10:47.314 DEBUG:boto:Token: None 2024-04-12T04:10:47.314 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj55 2024-04-12T04:10:47.314 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7/ukLddSqXnos6jjGpyOyZTMF6I= 2024-04-12T04:10:47.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7/ukLddSqXnos6jjGpyOyZTMF6I=', 'Content-Length': '0'} 2024-04-12T04:10:47.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b39c5d9c6f1cea29-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:10:47.316 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:10:47.316 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:10:47.316 DEBUG:boto:Method: GET 2024-04-12T04:10:47.316 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:10:47.317 DEBUG:boto:Data: 2024-04-12T04:10:47.317 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.317 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.317 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.317 DEBUG:boto:Params: {} 2024-04-12T04:10:47.317 DEBUG:boto:Token: None 2024-04-12T04:10:47.317 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj55 2024-04-12T04:10:47.317 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Xr7Gmbt5z+xWo9Ukrnuv2Clqi0w= 2024-04-12T04:10:47.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Xr7Gmbt5z+xWo9Ukrnuv2Clqi0w=', 'Content-Length': '0'} 2024-04-12T04:10:47.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d9f72d5a7dd2115-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.319 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:10:47.320 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:10:47.320 DEBUG:boto:Method: GET 2024-04-12T04:10:47.320 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:10:47.320 DEBUG:boto:Data: 2024-04-12T04:10:47.320 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.320 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.320 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.320 DEBUG:boto:Params: {} 2024-04-12T04:10:47.320 DEBUG:boto:Token: None 2024-04-12T04:10:47.320 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj55 2024-04-12T04:10:47.320 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Xr7Gmbt5z+xWo9Ukrnuv2Clqi0w= 2024-04-12T04:10:47.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Xr7Gmbt5z+xWo9Ukrnuv2Clqi0w=', 'Content-Length': '0'} 2024-04-12T04:10:47.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd19f6d232d283a1-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.323 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.323 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.323 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:10:47.323 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:10:47.323 DEBUG:boto:Method: GET 2024-04-12T04:10:47.323 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:10:47.323 DEBUG:boto:Data: 2024-04-12T04:10:47.323 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.323 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.324 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.324 DEBUG:boto:Params: {} 2024-04-12T04:10:47.324 DEBUG:boto:Token: None 2024-04-12T04:10:47.324 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.324 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001141025cc100993b-006618b447-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.326 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.327 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.327 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.327 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:10:47.327 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:10:47.327 DEBUG:boto:Method: GET 2024-04-12T04:10:47.327 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:10:47.327 DEBUG:boto:Data: 2024-04-12T04:10:47.327 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.328 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.328 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.328 DEBUG:boto:Params: {} 2024-04-12T04:10:47.328 DEBUG:boto:Token: None 2024-04-12T04:10:47.328 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.328 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092783de4de7074fa-006618b447-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.374 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:10:47.375 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:10:47.375 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:10:47.375 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.375 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.375 DEBUG:boto:Method: GET 2024-04-12T04:10:47.375 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.375 DEBUG:boto:Data: 2024-04-12T04:10:47.375 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.376 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.376 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.376 DEBUG:boto:Params: {} 2024-04-12T04:10:47.376 DEBUG:boto:Token: None 2024-04-12T04:10:47.376 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.376 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V/ae5VQ7bZDy+HjbpoTJbIQB8is= 2024-04-12T04:10:47.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V/ae5VQ7bZDy+HjbpoTJbIQB8is=', 'Content-Length': '0'} 2024-04-12T04:10:47.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cf78a1f0b3e3682-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.378 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:10:47.379 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:10:47.379 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.379 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.379 DEBUG:boto:Method: GET 2024-04-12T04:10:47.379 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.379 DEBUG:boto:Data: 2024-04-12T04:10:47.379 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.379 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.379 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.379 DEBUG:boto:Params: {} 2024-04-12T04:10:47.379 DEBUG:boto:Token: None 2024-04-12T04:10:47.380 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.380 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V/ae5VQ7bZDy+HjbpoTJbIQB8is= 2024-04-12T04:10:47.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V/ae5VQ7bZDy+HjbpoTJbIQB8is=', 'Content-Length': '0'} 2024-04-12T04:10:47.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca320ec8900c3f58-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.382 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:10:47.382 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:10:47.382 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.382 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.382 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.383 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.383 DEBUG:boto:Data: 2024-04-12T04:10:47.383 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.383 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.383 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.383 DEBUG:boto:Params: {} 2024-04-12T04:10:47.383 DEBUG:boto:Token: None 2024-04-12T04:10:47.383 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.383 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sJ/F9MoQYws3PR9GLdQ/m5zTFxc= 2024-04-12T04:10:47.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sJ/F9MoQYws3PR9GLdQ/m5zTFxc=', 'Content-Length': '0'} 2024-04-12T04:10:47.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000343fff49da236248-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.385 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:10:47.385 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:10:47.385 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.386 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.386 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.386 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.386 DEBUG:boto:Data: 2024-04-12T04:10:47.386 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.386 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.386 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.386 DEBUG:boto:Params: {} 2024-04-12T04:10:47.386 DEBUG:boto:Token: None 2024-04-12T04:10:47.387 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.387 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sJ/F9MoQYws3PR9GLdQ/m5zTFxc= 2024-04-12T04:10:47.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sJ/F9MoQYws3PR9GLdQ/m5zTFxc=', 'Content-Length': '0'} 2024-04-12T04:10:47.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082cc6e5ceb938df1-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:10:47.389 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:10:47.389 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:10:47.389 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.389 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.389 DEBUG:boto:Method: GET 2024-04-12T04:10:47.389 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.390 DEBUG:boto:Data: 2024-04-12T04:10:47.390 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.390 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.390 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.390 DEBUG:boto:Params: {} 2024-04-12T04:10:47.390 DEBUG:boto:Token: None 2024-04-12T04:10:47.390 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.390 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V/ae5VQ7bZDy+HjbpoTJbIQB8is= 2024-04-12T04:10:47.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V/ae5VQ7bZDy+HjbpoTJbIQB8is=', 'Content-Length': '0'} 2024-04-12T04:10:47.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026bae944d7662652-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.434 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:10:47.435 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:10:47.435 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.435 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.435 DEBUG:boto:Method: GET 2024-04-12T04:10:47.435 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.435 DEBUG:boto:Data: 2024-04-12T04:10:47.435 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.435 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.435 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.435 DEBUG:boto:Params: {} 2024-04-12T04:10:47.436 DEBUG:boto:Token: None 2024-04-12T04:10:47.436 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:10:47.436 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V/ae5VQ7bZDy+HjbpoTJbIQB8is= 2024-04-12T04:10:47.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V/ae5VQ7bZDy+HjbpoTJbIQB8is=', 'Content-Length': '0'} 2024-04-12T04:10:47.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f508c10b4f547d4d-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.439 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.439 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.439 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:10:47.439 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:10:47.439 DEBUG:boto:Method: GET 2024-04-12T04:10:47.439 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:10:47.439 DEBUG:boto:Data: 2024-04-12T04:10:47.439 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.439 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.439 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.440 DEBUG:boto:Params: {} 2024-04-12T04:10:47.440 DEBUG:boto:Token: None 2024-04-12T04:10:47.440 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.440 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010222cbda6c63979-006618b447-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.443 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.443 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.443 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.443 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:10:47.443 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:10:47.443 DEBUG:boto:Method: GET 2024-04-12T04:10:47.444 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:10:47.444 DEBUG:boto:Data: 2024-04-12T04:10:47.444 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.444 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.444 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.444 DEBUG:boto:Params: {} 2024-04-12T04:10:47.444 DEBUG:boto:Token: None 2024-04-12T04:10:47.444 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.444 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fea8b305f063a915-006618b447-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.449 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.449 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:10:47.450 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:10:47.450 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.450 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:10:47.450 DEBUG:boto:Data: 2024-04-12T04:10:47.450 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.450 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.450 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.450 DEBUG:boto:Params: {} 2024-04-12T04:10:47.450 DEBUG:boto:Token: None 2024-04-12T04:10:47.450 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj57 2024-04-12T04:10:47.451 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4jawb2aYLOb2z70okpPBH871iRg= 2024-04-12T04:10:47.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4jawb2aYLOb2z70okpPBH871iRg=', 'Content-Length': '0'} 2024-04-12T04:10:47.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa3e33640a49c2b8-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.453 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:10:47.453 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:10:47.453 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.453 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:10:47.453 DEBUG:boto:Data: 2024-04-12T04:10:47.453 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.453 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.453 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.454 DEBUG:boto:Params: {} 2024-04-12T04:10:47.454 DEBUG:boto:Token: None 2024-04-12T04:10:47.454 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj57 2024-04-12T04:10:47.454 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4jawb2aYLOb2z70okpPBH871iRg= 2024-04-12T04:10:47.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4jawb2aYLOb2z70okpPBH871iRg=', 'Content-Length': '0'} 2024-04-12T04:10:47.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000855536732580ea8c-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:10:47.456 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:10:47.456 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:10:47.456 DEBUG:boto:Method: GET 2024-04-12T04:10:47.456 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:10:47.456 DEBUG:boto:Data: 2024-04-12T04:10:47.457 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.457 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.457 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.457 DEBUG:boto:Params: {} 2024-04-12T04:10:47.457 DEBUG:boto:Token: None 2024-04-12T04:10:47.457 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj57 2024-04-12T04:10:47.457 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FVSbQqqQSH8yjAkOcwTV0yrRelI= 2024-04-12T04:10:47.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FVSbQqqQSH8yjAkOcwTV0yrRelI=', 'Content-Length': '0'} 2024-04-12T04:10:47.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001af1a81172dbe6d8-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.459 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:10:47.459 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:10:47.459 DEBUG:boto:Method: GET 2024-04-12T04:10:47.459 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:10:47.460 DEBUG:boto:Data: 2024-04-12T04:10:47.460 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.460 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.460 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.460 DEBUG:boto:Params: {} 2024-04-12T04:10:47.460 DEBUG:boto:Token: None 2024-04-12T04:10:47.460 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj57 2024-04-12T04:10:47.460 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FVSbQqqQSH8yjAkOcwTV0yrRelI= 2024-04-12T04:10:47.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FVSbQqqQSH8yjAkOcwTV0yrRelI=', 'Content-Length': '0'} 2024-04-12T04:10:47.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f89375b34fdeb17f-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.502 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.503 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.503 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:10:47.503 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:10:47.503 DEBUG:boto:Method: GET 2024-04-12T04:10:47.503 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:10:47.503 DEBUG:boto:Data: 2024-04-12T04:10:47.503 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.503 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.503 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.503 DEBUG:boto:Params: {} 2024-04-12T04:10:47.504 DEBUG:boto:Token: None 2024-04-12T04:10:47.504 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.504 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000826601b4e45e2cf8-006618b447-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.507 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.507 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.507 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.507 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:10:47.508 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:10:47.508 DEBUG:boto:Method: GET 2024-04-12T04:10:47.508 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:10:47.508 DEBUG:boto:Data: 2024-04-12T04:10:47.508 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.508 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.508 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.508 DEBUG:boto:Params: {} 2024-04-12T04:10:47.508 DEBUG:boto:Token: None 2024-04-12T04:10:47.508 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.508 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.513 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f55e09500bf66ef1-006618b447-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.514 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:10:47.515 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:10:47.515 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:10:47.515 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.515 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.515 DEBUG:boto:Method: GET 2024-04-12T04:10:47.515 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.515 DEBUG:boto:Data: 2024-04-12T04:10:47.515 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.515 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.516 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.516 DEBUG:boto:Params: {} 2024-04-12T04:10:47.516 DEBUG:boto:Token: None 2024-04-12T04:10:47.516 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.516 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wAsdIGVw6oklTvT9py4m338UsZ4= 2024-04-12T04:10:47.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wAsdIGVw6oklTvT9py4m338UsZ4=', 'Content-Length': '0'} 2024-04-12T04:10:47.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003957568fc48b73bd-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.558 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:10:47.559 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:10:47.559 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.559 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.559 DEBUG:boto:Method: GET 2024-04-12T04:10:47.559 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.559 DEBUG:boto:Data: 2024-04-12T04:10:47.559 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.559 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.559 DEBUG:boto:Params: {} 2024-04-12T04:10:47.559 DEBUG:boto:Token: None 2024-04-12T04:10:47.560 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.560 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wAsdIGVw6oklTvT9py4m338UsZ4= 2024-04-12T04:10:47.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wAsdIGVw6oklTvT9py4m338UsZ4=', 'Content-Length': '0'} 2024-04-12T04:10:47.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003293a9cf2e94ba02-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.562 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:10:47.562 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:10:47.563 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.563 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.563 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.563 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.563 DEBUG:boto:Data: 2024-04-12T04:10:47.563 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.563 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.563 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.563 DEBUG:boto:Params: {} 2024-04-12T04:10:47.563 DEBUG:boto:Token: None 2024-04-12T04:10:47.564 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.564 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1L9rSH9MxbxOz6WbUtM9zOcZMcA= 2024-04-12T04:10:47.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1L9rSH9MxbxOz6WbUtM9zOcZMcA=', 'Content-Length': '0'} 2024-04-12T04:10:47.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a4fb75645dd53ad-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.566 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:10:47.566 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:10:47.566 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.566 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.566 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.567 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.567 DEBUG:boto:Data: 2024-04-12T04:10:47.567 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.567 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.567 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.567 DEBUG:boto:Params: {} 2024-04-12T04:10:47.567 DEBUG:boto:Token: None 2024-04-12T04:10:47.567 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.567 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1L9rSH9MxbxOz6WbUtM9zOcZMcA= 2024-04-12T04:10:47.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1L9rSH9MxbxOz6WbUtM9zOcZMcA=', 'Content-Length': '0'} 2024-04-12T04:10:47.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efdd1c20fb52ccb6-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:10:47.570 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:10:47.570 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:10:47.570 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.570 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.570 DEBUG:boto:Method: GET 2024-04-12T04:10:47.570 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.570 DEBUG:boto:Data: 2024-04-12T04:10:47.571 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.571 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.571 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.571 DEBUG:boto:Params: {} 2024-04-12T04:10:47.571 DEBUG:boto:Token: None 2024-04-12T04:10:47.571 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.571 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wAsdIGVw6oklTvT9py4m338UsZ4= 2024-04-12T04:10:47.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wAsdIGVw6oklTvT9py4m338UsZ4=', 'Content-Length': '0'} 2024-04-12T04:10:47.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005132121e17bff08d-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.573 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:10:47.573 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:10:47.573 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.574 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.574 DEBUG:boto:Method: GET 2024-04-12T04:10:47.574 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.574 DEBUG:boto:Data: 2024-04-12T04:10:47.574 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.574 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.574 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.574 DEBUG:boto:Params: {} 2024-04-12T04:10:47.574 DEBUG:boto:Token: None 2024-04-12T04:10:47.574 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:10:47.575 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wAsdIGVw6oklTvT9py4m338UsZ4= 2024-04-12T04:10:47.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wAsdIGVw6oklTvT9py4m338UsZ4=', 'Content-Length': '0'} 2024-04-12T04:10:47.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed7ab36aef4b63a8-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.577 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.577 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.577 DEBUG:boto:path=/dkyges-61/?prefix=obj58 2024-04-12T04:10:47.577 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj58 2024-04-12T04:10:47.577 DEBUG:boto:Method: GET 2024-04-12T04:10:47.577 DEBUG:boto:Path: /dkyges-61/?prefix=obj58 2024-04-12T04:10:47.577 DEBUG:boto:Data: 2024-04-12T04:10:47.577 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.578 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.578 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.578 DEBUG:boto:Params: {} 2024-04-12T04:10:47.578 DEBUG:boto:Token: None 2024-04-12T04:10:47.578 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.578 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb1246708d574ece-006618b447-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.581 DEBUG:boto:b'dkyges-61obj581000falseobj582024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.581 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.581 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.581 DEBUG:boto:path=/dkyges-61/?prefix=obj58 2024-04-12T04:10:47.581 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj58 2024-04-12T04:10:47.581 DEBUG:boto:Method: GET 2024-04-12T04:10:47.581 DEBUG:boto:Path: /dkyges-61/?prefix=obj58 2024-04-12T04:10:47.582 DEBUG:boto:Data: 2024-04-12T04:10:47.582 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.582 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.582 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.582 DEBUG:boto:Params: {} 2024-04-12T04:10:47.582 DEBUG:boto:Token: None 2024-04-12T04:10:47.582 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.582 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000860d1f8d76f2dc72-006618b447-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.587 DEBUG:boto:b'dkyges-61obj581000falseobj582024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.587 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:10:47.587 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:10:47.587 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.588 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:10:47.588 DEBUG:boto:Data: 2024-04-12T04:10:47.588 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.588 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.588 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.588 DEBUG:boto:Params: {} 2024-04-12T04:10:47.588 DEBUG:boto:Token: None 2024-04-12T04:10:47.588 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj58 2024-04-12T04:10:47.588 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q8/mMnPF6VccA9cV6SdHGii2Y30= 2024-04-12T04:10:47.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q8/mMnPF6VccA9cV6SdHGii2Y30=', 'Content-Length': '0'} 2024-04-12T04:10:47.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f695904ad015e5bf-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.591 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:10:47.591 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:10:47.591 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.591 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:10:47.591 DEBUG:boto:Data: 2024-04-12T04:10:47.591 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.591 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.591 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.591 DEBUG:boto:Params: {} 2024-04-12T04:10:47.591 DEBUG:boto:Token: None 2024-04-12T04:10:47.592 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj58 2024-04-12T04:10:47.592 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q8/mMnPF6VccA9cV6SdHGii2Y30= 2024-04-12T04:10:47.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q8/mMnPF6VccA9cV6SdHGii2Y30=', 'Content-Length': '0'} 2024-04-12T04:10:47.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000002e85b0dbdc385a-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:10:47.594 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:10:47.594 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:10:47.594 DEBUG:boto:Method: GET 2024-04-12T04:10:47.594 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:10:47.594 DEBUG:boto:Data: 2024-04-12T04:10:47.594 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.594 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.594 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.595 DEBUG:boto:Params: {} 2024-04-12T04:10:47.595 DEBUG:boto:Token: None 2024-04-12T04:10:47.595 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj58 2024-04-12T04:10:47.595 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JAhxmkmUQiVOvc8VcvK+ot8zpZs= 2024-04-12T04:10:47.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JAhxmkmUQiVOvc8VcvK+ot8zpZs=', 'Content-Length': '0'} 2024-04-12T04:10:47.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016e6d0ca3b90532d-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.597 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:10:47.597 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:10:47.597 DEBUG:boto:Method: GET 2024-04-12T04:10:47.597 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:10:47.597 DEBUG:boto:Data: 2024-04-12T04:10:47.598 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.598 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.598 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.598 DEBUG:boto:Params: {} 2024-04-12T04:10:47.598 DEBUG:boto:Token: None 2024-04-12T04:10:47.598 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj58 2024-04-12T04:10:47.598 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JAhxmkmUQiVOvc8VcvK+ot8zpZs= 2024-04-12T04:10:47.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JAhxmkmUQiVOvc8VcvK+ot8zpZs=', 'Content-Length': '0'} 2024-04-12T04:10:47.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098c55e31a84338b3-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.600 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.600 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.600 DEBUG:boto:path=/dkyges-61/?prefix=obj58 2024-04-12T04:10:47.601 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj58 2024-04-12T04:10:47.601 DEBUG:boto:Method: GET 2024-04-12T04:10:47.601 DEBUG:boto:Path: /dkyges-61/?prefix=obj58 2024-04-12T04:10:47.601 DEBUG:boto:Data: 2024-04-12T04:10:47.601 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.601 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.601 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.601 DEBUG:boto:Params: {} 2024-04-12T04:10:47.601 DEBUG:boto:Token: None 2024-04-12T04:10:47.601 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.601 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.604 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046bcac307ea7e6ac-006618b447-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.604 DEBUG:boto:b'dkyges-61obj581000falseobj582024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.605 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.605 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.605 DEBUG:boto:path=/dkyges-61/?prefix=obj58 2024-04-12T04:10:47.605 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj58 2024-04-12T04:10:47.605 DEBUG:boto:Method: GET 2024-04-12T04:10:47.605 DEBUG:boto:Path: /dkyges-61/?prefix=obj58 2024-04-12T04:10:47.605 DEBUG:boto:Data: 2024-04-12T04:10:47.605 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.605 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.605 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.605 DEBUG:boto:Params: {} 2024-04-12T04:10:47.606 DEBUG:boto:Token: None 2024-04-12T04:10:47.606 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.606 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d80e862962dd729a-006618b447-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.611 DEBUG:boto:b'dkyges-61obj581000falseobj582024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T04:10:47.611 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:10:47.611 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:10:47.611 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.611 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.611 DEBUG:boto:Method: GET 2024-04-12T04:10:47.612 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.612 DEBUG:boto:Data: 2024-04-12T04:10:47.612 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.612 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.612 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.612 DEBUG:boto:Params: {} 2024-04-12T04:10:47.612 DEBUG:boto:Token: None 2024-04-12T04:10:47.612 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.612 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wlr1FTscUjy2SAeP/N6QpcHyzRw= 2024-04-12T04:10:47.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wlr1FTscUjy2SAeP/N6QpcHyzRw=', 'Content-Length': '0'} 2024-04-12T04:10:47.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002795e9f7280fb24c-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.654 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:10:47.655 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:10:47.655 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.655 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.655 DEBUG:boto:Method: GET 2024-04-12T04:10:47.655 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.655 DEBUG:boto:Data: 2024-04-12T04:10:47.655 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.655 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.655 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.655 DEBUG:boto:Params: {} 2024-04-12T04:10:47.655 DEBUG:boto:Token: None 2024-04-12T04:10:47.656 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wlr1FTscUjy2SAeP/N6QpcHyzRw= 2024-04-12T04:10:47.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wlr1FTscUjy2SAeP/N6QpcHyzRw=', 'Content-Length': '0'} 2024-04-12T04:10:47.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004966ac6641e7595e-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.658 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:10:47.659 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:10:47.659 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.659 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.659 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.659 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.659 DEBUG:boto:Data: 2024-04-12T04:10:47.659 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.659 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.659 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.659 DEBUG:boto:Params: {} 2024-04-12T04:10:47.660 DEBUG:boto:Token: None 2024-04-12T04:10:47.660 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.660 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xlBpDdp3uXpkx87vHNB64z4GfzA= 2024-04-12T04:10:47.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xlBpDdp3uXpkx87vHNB64z4GfzA=', 'Content-Length': '0'} 2024-04-12T04:10:47.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc2884ed9fbf04fd-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.662 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:10:47.662 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:10:47.662 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.662 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.662 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.662 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.662 DEBUG:boto:Data: 2024-04-12T04:10:47.663 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.663 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.663 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.663 DEBUG:boto:Params: {} 2024-04-12T04:10:47.663 DEBUG:boto:Token: None 2024-04-12T04:10:47.663 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.663 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xlBpDdp3uXpkx87vHNB64z4GfzA= 2024-04-12T04:10:47.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xlBpDdp3uXpkx87vHNB64z4GfzA=', 'Content-Length': '0'} 2024-04-12T04:10:47.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e188877a0ece50bd-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T04:10:47.665 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:10:47.666 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:10:47.666 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.666 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.666 DEBUG:boto:Method: GET 2024-04-12T04:10:47.666 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.666 DEBUG:boto:Data: 2024-04-12T04:10:47.666 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.666 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.666 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.666 DEBUG:boto:Params: {} 2024-04-12T04:10:47.667 DEBUG:boto:Token: None 2024-04-12T04:10:47.667 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.667 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wlr1FTscUjy2SAeP/N6QpcHyzRw= 2024-04-12T04:10:47.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wlr1FTscUjy2SAeP/N6QpcHyzRw=', 'Content-Length': '0'} 2024-04-12T04:10:47.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059ed2db5dcf4bb72-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.669 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:10:47.669 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:10:47.669 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.669 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.669 DEBUG:boto:Method: GET 2024-04-12T04:10:47.669 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.669 DEBUG:boto:Data: 2024-04-12T04:10:47.670 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.670 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.670 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.670 DEBUG:boto:Params: {} 2024-04-12T04:10:47.670 DEBUG:boto:Token: None 2024-04-12T04:10:47.670 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:10:47.670 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wlr1FTscUjy2SAeP/N6QpcHyzRw= 2024-04-12T04:10:47.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wlr1FTscUjy2SAeP/N6QpcHyzRw=', 'Content-Length': '0'} 2024-04-12T04:10:47.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bc9fa76238ed0f9-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.714 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.714 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.715 DEBUG:boto:path=/dkyges-61/?prefix=obj62 2024-04-12T04:10:47.715 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj62 2024-04-12T04:10:47.715 DEBUG:boto:Method: GET 2024-04-12T04:10:47.715 DEBUG:boto:Path: /dkyges-61/?prefix=obj62 2024-04-12T04:10:47.715 DEBUG:boto:Data: 2024-04-12T04:10:47.715 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.715 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.715 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.715 DEBUG:boto:Params: {} 2024-04-12T04:10:47.715 DEBUG:boto:Token: None 2024-04-12T04:10:47.716 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.716 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000220445d484f8931d-006618b447-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.718 DEBUG:boto:b'dkyges-61obj621000falseobj622024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.719 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.719 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.719 DEBUG:boto:path=/dkyges-61/?prefix=obj62 2024-04-12T04:10:47.719 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj62 2024-04-12T04:10:47.719 DEBUG:boto:Method: GET 2024-04-12T04:10:47.719 DEBUG:boto:Path: /dkyges-61/?prefix=obj62 2024-04-12T04:10:47.719 DEBUG:boto:Data: 2024-04-12T04:10:47.720 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.720 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.720 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.720 DEBUG:boto:Params: {} 2024-04-12T04:10:47.720 DEBUG:boto:Token: None 2024-04-12T04:10:47.720 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.720 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000daa95a6bde3c54b-006618b447-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.725 DEBUG:boto:b'dkyges-61obj621000falseobj622024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.725 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:10:47.725 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:10:47.725 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.725 DEBUG:boto:Path: /dkyges-61/obj62 2024-04-12T04:10:47.725 DEBUG:boto:Data: 2024-04-12T04:10:47.725 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.725 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.726 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.726 DEBUG:boto:Params: {} 2024-04-12T04:10:47.726 DEBUG:boto:Token: None 2024-04-12T04:10:47.726 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj62 2024-04-12T04:10:47.726 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mJ5fPHnLdSG9YoX7LZziHzMYXsE= 2024-04-12T04:10:47.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mJ5fPHnLdSG9YoX7LZziHzMYXsE=', 'Content-Length': '0'} 2024-04-12T04:10:47.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a4b957c4758481c-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.728 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:10:47.728 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:10:47.728 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.728 DEBUG:boto:Path: /dkyges-61/obj62 2024-04-12T04:10:47.728 DEBUG:boto:Data: 2024-04-12T04:10:47.728 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.728 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.729 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.729 DEBUG:boto:Params: {} 2024-04-12T04:10:47.729 DEBUG:boto:Token: None 2024-04-12T04:10:47.729 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj62 2024-04-12T04:10:47.729 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mJ5fPHnLdSG9YoX7LZziHzMYXsE= 2024-04-12T04:10:47.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mJ5fPHnLdSG9YoX7LZziHzMYXsE=', 'Content-Length': '0'} 2024-04-12T04:10:47.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dbc2cb14345db16-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T04:10:47.731 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:10:47.732 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:10:47.732 DEBUG:boto:Method: GET 2024-04-12T04:10:47.732 DEBUG:boto:Path: /dkyges-61/obj62 2024-04-12T04:10:47.732 DEBUG:boto:Data: 2024-04-12T04:10:47.732 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.732 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.732 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.732 DEBUG:boto:Params: {} 2024-04-12T04:10:47.732 DEBUG:boto:Token: None 2024-04-12T04:10:47.732 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj62 2024-04-12T04:10:47.732 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:x3+Ia0MCkCem4qypUc0bpY85iuQ= 2024-04-12T04:10:47.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:x3+Ia0MCkCem4qypUc0bpY85iuQ=', 'Content-Length': '0'} 2024-04-12T04:10:47.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c56bf05b4d4bf75-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.734 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:10:47.734 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:10:47.735 DEBUG:boto:Method: GET 2024-04-12T04:10:47.735 DEBUG:boto:Path: /dkyges-61/obj62 2024-04-12T04:10:47.735 DEBUG:boto:Data: 2024-04-12T04:10:47.735 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.735 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.735 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.735 DEBUG:boto:Params: {} 2024-04-12T04:10:47.735 DEBUG:boto:Token: None 2024-04-12T04:10:47.735 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj62 2024-04-12T04:10:47.735 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:x3+Ia0MCkCem4qypUc0bpY85iuQ= 2024-04-12T04:10:47.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:x3+Ia0MCkCem4qypUc0bpY85iuQ=', 'Content-Length': '0'} 2024-04-12T04:10:47.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc928d3780a40dc0-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.778 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.778 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.779 DEBUG:boto:path=/dkyges-61/?prefix=obj62 2024-04-12T04:10:47.779 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj62 2024-04-12T04:10:47.779 DEBUG:boto:Method: GET 2024-04-12T04:10:47.779 DEBUG:boto:Path: /dkyges-61/?prefix=obj62 2024-04-12T04:10:47.779 DEBUG:boto:Data: 2024-04-12T04:10:47.779 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.779 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.779 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.779 DEBUG:boto:Params: {} 2024-04-12T04:10:47.779 DEBUG:boto:Token: None 2024-04-12T04:10:47.780 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.780 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f494af2ec8c61c2-006618b447-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.782 DEBUG:boto:b'dkyges-61obj621000falseobj622024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.783 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.783 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.783 DEBUG:boto:path=/dkyges-61/?prefix=obj62 2024-04-12T04:10:47.783 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj62 2024-04-12T04:10:47.783 DEBUG:boto:Method: GET 2024-04-12T04:10:47.783 DEBUG:boto:Path: /dkyges-61/?prefix=obj62 2024-04-12T04:10:47.783 DEBUG:boto:Data: 2024-04-12T04:10:47.783 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.783 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.784 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.784 DEBUG:boto:Params: {} 2024-04-12T04:10:47.784 DEBUG:boto:Token: None 2024-04-12T04:10:47.784 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.784 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005357d41c88ef41d2-006618b447-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.788 DEBUG:boto:b'dkyges-61obj621000falseobj622024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:10:47.789 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:10:47.789 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:10:47.789 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.789 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.789 DEBUG:boto:Method: GET 2024-04-12T04:10:47.789 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.789 DEBUG:boto:Data: 2024-04-12T04:10:47.789 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.790 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.790 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.790 DEBUG:boto:Params: {} 2024-04-12T04:10:47.790 DEBUG:boto:Token: None 2024-04-12T04:10:47.790 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.790 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8pD1pqQI+Vloo0NXcRgrBXrNucU= 2024-04-12T04:10:47.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8pD1pqQI+Vloo0NXcRgrBXrNucU=', 'Content-Length': '0'} 2024-04-12T04:10:47.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb4a4846a6790d6b-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.834 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:10:47.835 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:10:47.835 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.835 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.835 DEBUG:boto:Method: GET 2024-04-12T04:10:47.835 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.835 DEBUG:boto:Data: 2024-04-12T04:10:47.835 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.835 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.835 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.835 DEBUG:boto:Params: {} 2024-04-12T04:10:47.835 DEBUG:boto:Token: None 2024-04-12T04:10:47.836 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.836 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8pD1pqQI+Vloo0NXcRgrBXrNucU= 2024-04-12T04:10:47.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8pD1pqQI+Vloo0NXcRgrBXrNucU=', 'Content-Length': '0'} 2024-04-12T04:10:47.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db57fb096ce7deb9-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.838 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:10:47.838 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:10:47.838 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.838 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.839 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.839 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.839 DEBUG:boto:Data: 2024-04-12T04:10:47.839 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.839 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.839 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.839 DEBUG:boto:Params: {} 2024-04-12T04:10:47.839 DEBUG:boto:Token: None 2024-04-12T04:10:47.839 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.840 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6a2KE/YzBkWKnox2okCG8LBGiKU= 2024-04-12T04:10:47.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6a2KE/YzBkWKnox2okCG8LBGiKU=', 'Content-Length': '0'} 2024-04-12T04:10:47.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005086d7ddc772595e-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.842 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:10:47.842 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:10:47.842 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.842 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.842 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.842 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.842 DEBUG:boto:Data: 2024-04-12T04:10:47.842 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.842 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.842 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.843 DEBUG:boto:Params: {} 2024-04-12T04:10:47.843 DEBUG:boto:Token: None 2024-04-12T04:10:47.843 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.843 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6a2KE/YzBkWKnox2okCG8LBGiKU= 2024-04-12T04:10:47.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6a2KE/YzBkWKnox2okCG8LBGiKU=', 'Content-Length': '0'} 2024-04-12T04:10:47.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd612e65f8d97185-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:10:47.845 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:10:47.845 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:10:47.845 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.845 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.846 DEBUG:boto:Method: GET 2024-04-12T04:10:47.846 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.846 DEBUG:boto:Data: 2024-04-12T04:10:47.846 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.846 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.846 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.846 DEBUG:boto:Params: {} 2024-04-12T04:10:47.846 DEBUG:boto:Token: None 2024-04-12T04:10:47.846 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.846 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8pD1pqQI+Vloo0NXcRgrBXrNucU= 2024-04-12T04:10:47.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8pD1pqQI+Vloo0NXcRgrBXrNucU=', 'Content-Length': '0'} 2024-04-12T04:10:47.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055a9678b72f5af49-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.849 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:10:47.849 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:10:47.849 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.849 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.849 DEBUG:boto:Method: GET 2024-04-12T04:10:47.849 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.849 DEBUG:boto:Data: 2024-04-12T04:10:47.849 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.849 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.849 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.850 DEBUG:boto:Params: {} 2024-04-12T04:10:47.850 DEBUG:boto:Token: None 2024-04-12T04:10:47.850 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:10:47.850 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8pD1pqQI+Vloo0NXcRgrBXrNucU= 2024-04-12T04:10:47.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8pD1pqQI+Vloo0NXcRgrBXrNucU=', 'Content-Length': '0'} 2024-04-12T04:10:47.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8e4fb2bfebad926-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.852 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.852 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.852 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:10:47.852 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:10:47.852 DEBUG:boto:Method: GET 2024-04-12T04:10:47.853 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:10:47.853 DEBUG:boto:Data: 2024-04-12T04:10:47.853 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.853 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.853 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.853 DEBUG:boto:Params: {} 2024-04-12T04:10:47.853 DEBUG:boto:Token: None 2024-04-12T04:10:47.853 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.853 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072be116e6dde5361-006618b447-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.856 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.856 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.857 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.857 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:10:47.857 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:10:47.857 DEBUG:boto:Method: GET 2024-04-12T04:10:47.857 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:10:47.857 DEBUG:boto:Data: 2024-04-12T04:10:47.857 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.857 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.857 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.857 DEBUG:boto:Params: {} 2024-04-12T04:10:47.858 DEBUG:boto:Token: None 2024-04-12T04:10:47.858 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.858 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b04d3ba8fbdd7be1-006618b447-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.862 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.862 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:10:47.863 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:10:47.863 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.863 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:10:47.863 DEBUG:boto:Data: 2024-04-12T04:10:47.863 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.863 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.863 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.863 DEBUG:boto:Params: {} 2024-04-12T04:10:47.863 DEBUG:boto:Token: None 2024-04-12T04:10:47.864 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj68 2024-04-12T04:10:47.864 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yc3cJO74/h1EtSqGXvHqLvafbPU= 2024-04-12T04:10:47.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yc3cJO74/h1EtSqGXvHqLvafbPU=', 'Content-Length': '0'} 2024-04-12T04:10:47.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000837b06c6f60a1cee-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.866 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:10:47.866 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:10:47.866 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.866 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:10:47.866 DEBUG:boto:Data: 2024-04-12T04:10:47.866 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.866 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.866 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.866 DEBUG:boto:Params: {} 2024-04-12T04:10:47.867 DEBUG:boto:Token: None 2024-04-12T04:10:47.867 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj68 2024-04-12T04:10:47.867 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yc3cJO74/h1EtSqGXvHqLvafbPU= 2024-04-12T04:10:47.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yc3cJO74/h1EtSqGXvHqLvafbPU=', 'Content-Length': '0'} 2024-04-12T04:10:47.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffc6478765eedc29-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:10:47.869 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:10:47.869 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:10:47.869 DEBUG:boto:Method: GET 2024-04-12T04:10:47.869 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:10:47.869 DEBUG:boto:Data: 2024-04-12T04:10:47.869 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.870 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.870 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.870 DEBUG:boto:Params: {} 2024-04-12T04:10:47.870 DEBUG:boto:Token: None 2024-04-12T04:10:47.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj68 2024-04-12T04:10:47.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iqhRioiP82AwES9AnIxVUiCLRSE= 2024-04-12T04:10:47.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iqhRioiP82AwES9AnIxVUiCLRSE=', 'Content-Length': '0'} 2024-04-12T04:10:47.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034d323bd505306c2-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.872 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:10:47.872 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:10:47.872 DEBUG:boto:Method: GET 2024-04-12T04:10:47.873 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:10:47.873 DEBUG:boto:Data: 2024-04-12T04:10:47.873 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.873 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.873 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.873 DEBUG:boto:Params: {} 2024-04-12T04:10:47.873 DEBUG:boto:Token: None 2024-04-12T04:10:47.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj68 2024-04-12T04:10:47.873 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iqhRioiP82AwES9AnIxVUiCLRSE= 2024-04-12T04:10:47.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iqhRioiP82AwES9AnIxVUiCLRSE=', 'Content-Length': '0'} 2024-04-12T04:10:47.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057301ce552b32ade-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.876 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.876 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.876 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:10:47.876 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:10:47.876 DEBUG:boto:Method: GET 2024-04-12T04:10:47.876 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:10:47.876 DEBUG:boto:Data: 2024-04-12T04:10:47.876 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.876 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.876 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.877 DEBUG:boto:Params: {} 2024-04-12T04:10:47.877 DEBUG:boto:Token: None 2024-04-12T04:10:47.877 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.877 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0214321cefb4437-006618b447-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.879 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.880 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.880 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.880 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:10:47.880 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:10:47.880 DEBUG:boto:Method: GET 2024-04-12T04:10:47.880 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:10:47.880 DEBUG:boto:Data: 2024-04-12T04:10:47.880 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.880 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.880 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.881 DEBUG:boto:Params: {} 2024-04-12T04:10:47.881 DEBUG:boto:Token: None 2024-04-12T04:10:47.881 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.881 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095cb1e673d48259c-006618b447-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.885 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:10:47.886 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:10:47.886 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:10:47.886 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.886 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.886 DEBUG:boto:Method: GET 2024-04-12T04:10:47.886 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.886 DEBUG:boto:Data: 2024-04-12T04:10:47.886 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.887 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.887 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.887 DEBUG:boto:Params: {} 2024-04-12T04:10:47.887 DEBUG:boto:Token: None 2024-04-12T04:10:47.887 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.887 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tIo1FzomlEzjXGJhuhfryDKkCSY= 2024-04-12T04:10:47.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tIo1FzomlEzjXGJhuhfryDKkCSY=', 'Content-Length': '0'} 2024-04-12T04:10:47.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005abfa626cd6d5ff4-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.930 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:10:47.930 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:10:47.931 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.931 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.931 DEBUG:boto:Method: GET 2024-04-12T04:10:47.931 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.931 DEBUG:boto:Data: 2024-04-12T04:10:47.931 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.931 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.931 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.931 DEBUG:boto:Params: {} 2024-04-12T04:10:47.931 DEBUG:boto:Token: None 2024-04-12T04:10:47.932 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.932 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tIo1FzomlEzjXGJhuhfryDKkCSY= 2024-04-12T04:10:47.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tIo1FzomlEzjXGJhuhfryDKkCSY=', 'Content-Length': '0'} 2024-04-12T04:10:47.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7a80774f265bffb-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.934 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:10:47.934 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:10:47.935 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.935 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.935 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.935 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.935 DEBUG:boto:Data: 2024-04-12T04:10:47.935 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.935 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.935 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.935 DEBUG:boto:Params: {} 2024-04-12T04:10:47.935 DEBUG:boto:Token: None 2024-04-12T04:10:47.936 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.936 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EBmI8vqHQC5XWgeTWtCHUxlMPCM= 2024-04-12T04:10:47.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EBmI8vqHQC5XWgeTWtCHUxlMPCM=', 'Content-Length': '0'} 2024-04-12T04:10:47.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b69776be49a2d94-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.938 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:10:47.938 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:10:47.938 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.938 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.938 DEBUG:boto:Method: HEAD 2024-04-12T04:10:47.938 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.938 DEBUG:boto:Data: 2024-04-12T04:10:47.938 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.939 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.939 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.939 DEBUG:boto:Params: {} 2024-04-12T04:10:47.939 DEBUG:boto:Token: None 2024-04-12T04:10:47.939 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.939 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EBmI8vqHQC5XWgeTWtCHUxlMPCM= 2024-04-12T04:10:47.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EBmI8vqHQC5XWgeTWtCHUxlMPCM=', 'Content-Length': '0'} 2024-04-12T04:10:47.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095b9ad6e4b6e1290-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:10:47.941 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:10:47.941 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:10:47.942 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.942 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.942 DEBUG:boto:Method: GET 2024-04-12T04:10:47.942 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.942 DEBUG:boto:Data: 2024-04-12T04:10:47.942 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.942 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.942 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.942 DEBUG:boto:Params: {} 2024-04-12T04:10:47.942 DEBUG:boto:Token: None 2024-04-12T04:10:47.943 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.943 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tIo1FzomlEzjXGJhuhfryDKkCSY= 2024-04-12T04:10:47.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tIo1FzomlEzjXGJhuhfryDKkCSY=', 'Content-Length': '0'} 2024-04-12T04:10:47.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000715c2fabf10c883d-006618b447-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.945 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:10:47.945 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:10:47.945 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.945 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.945 DEBUG:boto:Method: GET 2024-04-12T04:10:47.945 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.946 DEBUG:boto:Data: 2024-04-12T04:10:47.946 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.946 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.946 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.946 DEBUG:boto:Params: {} 2024-04-12T04:10:47.946 DEBUG:boto:Token: None 2024-04-12T04:10:47.946 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:10:47.946 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tIo1FzomlEzjXGJhuhfryDKkCSY= 2024-04-12T04:10:47.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tIo1FzomlEzjXGJhuhfryDKkCSY=', 'Content-Length': '0'} 2024-04-12T04:10:47.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c56264bbe8006ba6-006618b447-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.990 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.990 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.991 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:10:47.991 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:10:47.991 DEBUG:boto:Method: GET 2024-04-12T04:10:47.991 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:10:47.991 DEBUG:boto:Data: 2024-04-12T04:10:47.991 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.991 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:47.991 DEBUG:boto:Port: 8001 2024-04-12T04:10:47.991 DEBUG:boto:Params: {} 2024-04-12T04:10:47.991 DEBUG:boto:Token: None 2024-04-12T04:10:47.991 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.992 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:47.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb454ec8f95d6490-006618b448-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:47.994 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:47.995 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:47.995 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:47.995 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:10:47.995 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:10:47.995 DEBUG:boto:Method: GET 2024-04-12T04:10:47.995 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:10:47.995 DEBUG:boto:Data: 2024-04-12T04:10:47.995 DEBUG:boto:Headers: {} 2024-04-12T04:10:47.995 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:47.996 DEBUG:boto:Port: 8000 2024-04-12T04:10:47.996 DEBUG:boto:Params: {} 2024-04-12T04:10:47.996 DEBUG:boto:Token: None 2024-04-12T04:10:47.996 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:47 GMT /dkyges-61/ 2024-04-12T04:10:47.996 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc= 2024-04-12T04:10:47.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qG+vo6ZTbcKRWFv6TXebFvbDemc=', 'Content-Length': '0'} 2024-04-12T04:10:48.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c6992893c81170a-006618b448-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.001 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.001 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:10:48.001 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:10:48.001 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.001 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:10:48.001 DEBUG:boto:Data: 2024-04-12T04:10:48.001 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.001 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.002 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.002 DEBUG:boto:Params: {} 2024-04-12T04:10:48.002 DEBUG:boto:Token: None 2024-04-12T04:10:48.002 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj69 2024-04-12T04:10:48.002 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BmVGl3GYApbNlJDZef0aqpd4fxk= 2024-04-12T04:10:48.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BmVGl3GYApbNlJDZef0aqpd4fxk=', 'Content-Length': '0'} 2024-04-12T04:10:48.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000114384828a536894-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.004 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:10:48.004 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:10:48.004 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.004 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:10:48.004 DEBUG:boto:Data: 2024-04-12T04:10:48.005 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.005 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.005 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.005 DEBUG:boto:Params: {} 2024-04-12T04:10:48.005 DEBUG:boto:Token: None 2024-04-12T04:10:48.005 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj69 2024-04-12T04:10:48.005 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BmVGl3GYApbNlJDZef0aqpd4fxk= 2024-04-12T04:10:48.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BmVGl3GYApbNlJDZef0aqpd4fxk=', 'Content-Length': '0'} 2024-04-12T04:10:48.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000528ab30a55b77bcc-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:10:48.007 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:10:48.007 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:10:48.007 DEBUG:boto:Method: GET 2024-04-12T04:10:48.007 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:10:48.007 DEBUG:boto:Data: 2024-04-12T04:10:48.008 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.008 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.008 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.008 DEBUG:boto:Params: {} 2024-04-12T04:10:48.008 DEBUG:boto:Token: None 2024-04-12T04:10:48.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj69 2024-04-12T04:10:48.008 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:x9DU+xMJPvVybPiAxANhA3uRon0= 2024-04-12T04:10:48.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:x9DU+xMJPvVybPiAxANhA3uRon0=', 'Content-Length': '0'} 2024-04-12T04:10:48.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd459cb39552db7d-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.010 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:10:48.010 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:10:48.010 DEBUG:boto:Method: GET 2024-04-12T04:10:48.010 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:10:48.011 DEBUG:boto:Data: 2024-04-12T04:10:48.011 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.011 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.011 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.011 DEBUG:boto:Params: {} 2024-04-12T04:10:48.011 DEBUG:boto:Token: None 2024-04-12T04:10:48.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj69 2024-04-12T04:10:48.011 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:x9DU+xMJPvVybPiAxANhA3uRon0= 2024-04-12T04:10:48.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:x9DU+xMJPvVybPiAxANhA3uRon0=', 'Content-Length': '0'} 2024-04-12T04:10:48.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6cb7a74967e5f93-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.054 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.054 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.055 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:10:48.055 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:10:48.055 DEBUG:boto:Method: GET 2024-04-12T04:10:48.055 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:10:48.055 DEBUG:boto:Data: 2024-04-12T04:10:48.055 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.055 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.055 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.055 DEBUG:boto:Params: {} 2024-04-12T04:10:48.055 DEBUG:boto:Token: None 2024-04-12T04:10:48.055 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.056 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.058 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058360cfda8471c80-006618b448-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.058 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.059 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.059 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.059 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:10:48.059 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:10:48.059 DEBUG:boto:Method: GET 2024-04-12T04:10:48.059 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:10:48.059 DEBUG:boto:Data: 2024-04-12T04:10:48.059 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.060 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.060 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.060 DEBUG:boto:Params: {} 2024-04-12T04:10:48.060 DEBUG:boto:Token: None 2024-04-12T04:10:48.060 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.060 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fdb956e12f2265d-006618b448-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.065 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T04:10:48.065 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:10:48.065 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:10:48.065 DEBUG:boto:path=/dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.065 DEBUG:boto:auth_path=/dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.065 DEBUG:boto:Method: GET 2024-04-12T04:10:48.065 DEBUG:boto:Path: /dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.065 DEBUG:boto:Data: 2024-04-12T04:10:48.066 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.066 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.066 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.066 DEBUG:boto:Params: {} 2024-04-12T04:10:48.066 DEBUG:boto:Token: None 2024-04-12T04:10:48.066 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.066 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PVIu6Kt9d80G6VP25jXP/Zgluig= 2024-04-12T04:10:48.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PVIu6Kt9d80G6VP25jXP/Zgluig=', 'Content-Length': '0'} 2024-04-12T04:10:48.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1afdf2b9c0d2ca5-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.110 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:10:48.110 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:10:48.111 DEBUG:boto:path=/dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.111 DEBUG:boto:auth_path=/dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.111 DEBUG:boto:Method: GET 2024-04-12T04:10:48.111 DEBUG:boto:Path: /dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.111 DEBUG:boto:Data: 2024-04-12T04:10:48.111 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.111 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.111 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.111 DEBUG:boto:Params: {} 2024-04-12T04:10:48.111 DEBUG:boto:Token: None 2024-04-12T04:10:48.111 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.112 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PVIu6Kt9d80G6VP25jXP/Zgluig= 2024-04-12T04:10:48.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PVIu6Kt9d80G6VP25jXP/Zgluig=', 'Content-Length': '0'} 2024-04-12T04:10:48.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5e63a06244143a0-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.114 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:10:48.114 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:10:48.114 DEBUG:boto:path=/dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.114 DEBUG:boto:auth_path=/dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.114 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.114 DEBUG:boto:Path: /dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.115 DEBUG:boto:Data: 2024-04-12T04:10:48.115 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.115 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.115 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.115 DEBUG:boto:Params: {} 2024-04-12T04:10:48.115 DEBUG:boto:Token: None 2024-04-12T04:10:48.115 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.115 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qr58ppsGyrzOedVXYGyi8NL8U88= 2024-04-12T04:10:48.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qr58ppsGyrzOedVXYGyi8NL8U88=', 'Content-Length': '0'} 2024-04-12T04:10:48.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008035b7ab0c85e77b-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.117 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:10:48.117 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:10:48.117 DEBUG:boto:path=/dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.117 DEBUG:boto:auth_path=/dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.118 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.118 DEBUG:boto:Path: /dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.118 DEBUG:boto:Data: 2024-04-12T04:10:48.118 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.118 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.118 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.118 DEBUG:boto:Params: {} 2024-04-12T04:10:48.118 DEBUG:boto:Token: None 2024-04-12T04:10:48.118 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.119 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qr58ppsGyrzOedVXYGyi8NL8U88= 2024-04-12T04:10:48.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qr58ppsGyrzOedVXYGyi8NL8U88=', 'Content-Length': '0'} 2024-04-12T04:10:48.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae019a459e6ae843-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T04:10:48.121 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:10:48.121 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:10:48.121 DEBUG:boto:path=/dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.121 DEBUG:boto:auth_path=/dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.121 DEBUG:boto:Method: GET 2024-04-12T04:10:48.121 DEBUG:boto:Path: /dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.121 DEBUG:boto:Data: 2024-04-12T04:10:48.121 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.122 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.122 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.122 DEBUG:boto:Params: {} 2024-04-12T04:10:48.122 DEBUG:boto:Token: None 2024-04-12T04:10:48.122 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.122 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PVIu6Kt9d80G6VP25jXP/Zgluig= 2024-04-12T04:10:48.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PVIu6Kt9d80G6VP25jXP/Zgluig=', 'Content-Length': '0'} 2024-04-12T04:10:48.124 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003160715a1fe287f5-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.124 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:10:48.124 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:10:48.124 DEBUG:boto:path=/dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.125 DEBUG:boto:auth_path=/dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.125 DEBUG:boto:Method: GET 2024-04-12T04:10:48.125 DEBUG:boto:Path: /dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.125 DEBUG:boto:Data: 2024-04-12T04:10:48.125 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.125 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.125 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.125 DEBUG:boto:Params: {} 2024-04-12T04:10:48.125 DEBUG:boto:Token: None 2024-04-12T04:10:48.125 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj74?versionId=null 2024-04-12T04:10:48.126 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PVIu6Kt9d80G6VP25jXP/Zgluig= 2024-04-12T04:10:48.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PVIu6Kt9d80G6VP25jXP/Zgluig=', 'Content-Length': '0'} 2024-04-12T04:10:48.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056b70d13bc9c6353-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.128 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.128 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.128 DEBUG:boto:path=/dkyges-61/?prefix=obj74 2024-04-12T04:10:48.128 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj74 2024-04-12T04:10:48.128 DEBUG:boto:Method: GET 2024-04-12T04:10:48.128 DEBUG:boto:Path: /dkyges-61/?prefix=obj74 2024-04-12T04:10:48.128 DEBUG:boto:Data: 2024-04-12T04:10:48.128 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.128 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.128 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.129 DEBUG:boto:Params: {} 2024-04-12T04:10:48.129 DEBUG:boto:Token: None 2024-04-12T04:10:48.129 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.129 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e23b5caec599d021-006618b448-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.132 DEBUG:boto:b'dkyges-61obj741000falseobj742024-04-12T04:09:24.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.132 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.132 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.132 DEBUG:boto:path=/dkyges-61/?prefix=obj74 2024-04-12T04:10:48.132 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj74 2024-04-12T04:10:48.132 DEBUG:boto:Method: GET 2024-04-12T04:10:48.133 DEBUG:boto:Path: /dkyges-61/?prefix=obj74 2024-04-12T04:10:48.133 DEBUG:boto:Data: 2024-04-12T04:10:48.133 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.133 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.133 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.133 DEBUG:boto:Params: {} 2024-04-12T04:10:48.133 DEBUG:boto:Token: None 2024-04-12T04:10:48.133 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.133 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b2a6b56ef30d940-006618b448-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.138 DEBUG:boto:b'dkyges-61obj741000falseobj742024-04-12T04:09:24.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.138 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:10:48.138 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:10:48.139 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.139 DEBUG:boto:Path: /dkyges-61/obj74 2024-04-12T04:10:48.139 DEBUG:boto:Data: 2024-04-12T04:10:48.139 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.139 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.139 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.139 DEBUG:boto:Params: {} 2024-04-12T04:10:48.139 DEBUG:boto:Token: None 2024-04-12T04:10:48.139 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj74 2024-04-12T04:10:48.139 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FsF+2ieTSvU4P5A+2qxcwgO++J0= 2024-04-12T04:10:48.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FsF+2ieTSvU4P5A+2qxcwgO++J0=', 'Content-Length': '0'} 2024-04-12T04:10:48.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ada462eb5031640-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.142 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:10:48.142 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:10:48.142 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.142 DEBUG:boto:Path: /dkyges-61/obj74 2024-04-12T04:10:48.142 DEBUG:boto:Data: 2024-04-12T04:10:48.142 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.142 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.142 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.142 DEBUG:boto:Params: {} 2024-04-12T04:10:48.142 DEBUG:boto:Token: None 2024-04-12T04:10:48.143 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj74 2024-04-12T04:10:48.143 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FsF+2ieTSvU4P5A+2qxcwgO++J0= 2024-04-12T04:10:48.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FsF+2ieTSvU4P5A+2qxcwgO++J0=', 'Content-Length': '0'} 2024-04-12T04:10:48.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fb92ab52c61fda0-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T04:10:48.145 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:10:48.145 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:10:48.145 DEBUG:boto:Method: GET 2024-04-12T04:10:48.145 DEBUG:boto:Path: /dkyges-61/obj74 2024-04-12T04:10:48.146 DEBUG:boto:Data: 2024-04-12T04:10:48.146 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.146 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.146 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.146 DEBUG:boto:Params: {} 2024-04-12T04:10:48.146 DEBUG:boto:Token: None 2024-04-12T04:10:48.146 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj74 2024-04-12T04:10:48.146 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2tMfJaxgBII176Eufid3nfEB/sY= 2024-04-12T04:10:48.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2tMfJaxgBII176Eufid3nfEB/sY=', 'Content-Length': '0'} 2024-04-12T04:10:48.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c66f2e844104406-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.148 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:10:48.148 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:10:48.148 DEBUG:boto:Method: GET 2024-04-12T04:10:48.149 DEBUG:boto:Path: /dkyges-61/obj74 2024-04-12T04:10:48.149 DEBUG:boto:Data: 2024-04-12T04:10:48.149 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.149 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.149 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.149 DEBUG:boto:Params: {} 2024-04-12T04:10:48.149 DEBUG:boto:Token: None 2024-04-12T04:10:48.149 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj74 2024-04-12T04:10:48.149 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2tMfJaxgBII176Eufid3nfEB/sY= 2024-04-12T04:10:48.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2tMfJaxgBII176Eufid3nfEB/sY=', 'Content-Length': '0'} 2024-04-12T04:10:48.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007964a0ac15503a7a-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.194 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.195 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.195 DEBUG:boto:path=/dkyges-61/?prefix=obj74 2024-04-12T04:10:48.195 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj74 2024-04-12T04:10:48.195 DEBUG:boto:Method: GET 2024-04-12T04:10:48.195 DEBUG:boto:Path: /dkyges-61/?prefix=obj74 2024-04-12T04:10:48.195 DEBUG:boto:Data: 2024-04-12T04:10:48.195 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.195 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.195 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.195 DEBUG:boto:Params: {} 2024-04-12T04:10:48.196 DEBUG:boto:Token: None 2024-04-12T04:10:48.196 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.196 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000095b207755f492a4-006618b448-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.200 DEBUG:boto:b'dkyges-61obj741000falseobj742024-04-12T04:09:24.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.200 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.200 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.200 DEBUG:boto:path=/dkyges-61/?prefix=obj74 2024-04-12T04:10:48.200 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj74 2024-04-12T04:10:48.201 DEBUG:boto:Method: GET 2024-04-12T04:10:48.201 DEBUG:boto:Path: /dkyges-61/?prefix=obj74 2024-04-12T04:10:48.201 DEBUG:boto:Data: 2024-04-12T04:10:48.201 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.201 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.201 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.201 DEBUG:boto:Params: {} 2024-04-12T04:10:48.201 DEBUG:boto:Token: None 2024-04-12T04:10:48.201 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.201 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000636ba93554b6b68a-006618b448-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.206 DEBUG:boto:b'dkyges-61obj741000falseobj742024-04-12T04:09:24.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:10:48.206 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:10:48.207 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:10:48.207 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.207 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.207 DEBUG:boto:Method: GET 2024-04-12T04:10:48.207 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.207 DEBUG:boto:Data: 2024-04-12T04:10:48.207 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.207 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.207 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.207 DEBUG:boto:Params: {} 2024-04-12T04:10:48.208 DEBUG:boto:Token: None 2024-04-12T04:10:48.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G0pi4YisHvJbWJLXGn4UmiGyFN8= 2024-04-12T04:10:48.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G0pi4YisHvJbWJLXGn4UmiGyFN8=', 'Content-Length': '0'} 2024-04-12T04:10:48.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004763b62a290b75f6-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.250 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:10:48.251 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:10:48.251 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.251 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.251 DEBUG:boto:Method: GET 2024-04-12T04:10:48.251 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.251 DEBUG:boto:Data: 2024-04-12T04:10:48.251 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.251 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.251 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.251 DEBUG:boto:Params: {} 2024-04-12T04:10:48.252 DEBUG:boto:Token: None 2024-04-12T04:10:48.252 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.252 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G0pi4YisHvJbWJLXGn4UmiGyFN8= 2024-04-12T04:10:48.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G0pi4YisHvJbWJLXGn4UmiGyFN8=', 'Content-Length': '0'} 2024-04-12T04:10:48.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e3e365c93ceb526-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.254 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:10:48.254 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:10:48.254 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.255 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.255 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.255 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.255 DEBUG:boto:Data: 2024-04-12T04:10:48.255 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.255 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.255 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.255 DEBUG:boto:Params: {} 2024-04-12T04:10:48.255 DEBUG:boto:Token: None 2024-04-12T04:10:48.255 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.256 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jvrtkcj0GI3x5vGiZa9qHqizirA= 2024-04-12T04:10:48.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jvrtkcj0GI3x5vGiZa9qHqizirA=', 'Content-Length': '0'} 2024-04-12T04:10:48.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d99e85bd0645cf57-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.258 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:10:48.258 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:10:48.258 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.258 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.258 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.258 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.258 DEBUG:boto:Data: 2024-04-12T04:10:48.258 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.258 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.258 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.259 DEBUG:boto:Params: {} 2024-04-12T04:10:48.259 DEBUG:boto:Token: None 2024-04-12T04:10:48.259 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.259 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jvrtkcj0GI3x5vGiZa9qHqizirA= 2024-04-12T04:10:48.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jvrtkcj0GI3x5vGiZa9qHqizirA=', 'Content-Length': '0'} 2024-04-12T04:10:48.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2f83e6be3b6660c-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:10:48.261 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:10:48.262 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:10:48.262 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.262 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.262 DEBUG:boto:Method: GET 2024-04-12T04:10:48.262 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.262 DEBUG:boto:Data: 2024-04-12T04:10:48.262 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.262 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.262 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.262 DEBUG:boto:Params: {} 2024-04-12T04:10:48.263 DEBUG:boto:Token: None 2024-04-12T04:10:48.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.263 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G0pi4YisHvJbWJLXGn4UmiGyFN8= 2024-04-12T04:10:48.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G0pi4YisHvJbWJLXGn4UmiGyFN8=', 'Content-Length': '0'} 2024-04-12T04:10:48.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6d52ccdcf8ed437-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.265 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:10:48.265 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:10:48.265 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.265 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.265 DEBUG:boto:Method: GET 2024-04-12T04:10:48.265 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.265 DEBUG:boto:Data: 2024-04-12T04:10:48.266 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.266 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.266 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.266 DEBUG:boto:Params: {} 2024-04-12T04:10:48.266 DEBUG:boto:Token: None 2024-04-12T04:10:48.266 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:10:48.266 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G0pi4YisHvJbWJLXGn4UmiGyFN8= 2024-04-12T04:10:48.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G0pi4YisHvJbWJLXGn4UmiGyFN8=', 'Content-Length': '0'} 2024-04-12T04:10:48.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088345ac9a999b1ff-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.268 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.268 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.268 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:10:48.269 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:10:48.269 DEBUG:boto:Method: GET 2024-04-12T04:10:48.269 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:10:48.269 DEBUG:boto:Data: 2024-04-12T04:10:48.269 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.269 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.269 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.269 DEBUG:boto:Params: {} 2024-04-12T04:10:48.269 DEBUG:boto:Token: None 2024-04-12T04:10:48.269 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.269 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.272 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da9a4c21259296e7-006618b448-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.272 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.272 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.273 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.273 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:10:48.273 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:10:48.273 DEBUG:boto:Method: GET 2024-04-12T04:10:48.273 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:10:48.273 DEBUG:boto:Data: 2024-04-12T04:10:48.273 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.273 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.274 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.274 DEBUG:boto:Params: {} 2024-04-12T04:10:48.274 DEBUG:boto:Token: None 2024-04-12T04:10:48.274 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.274 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d18ae088230de6bc-006618b448-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.279 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.279 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:10:48.279 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:10:48.279 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.280 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:10:48.280 DEBUG:boto:Data: 2024-04-12T04:10:48.280 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.280 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.280 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.280 DEBUG:boto:Params: {} 2024-04-12T04:10:48.280 DEBUG:boto:Token: None 2024-04-12T04:10:48.280 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj76 2024-04-12T04:10:48.280 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oIejvutGIGxAw9chgJSGrcTw2dU= 2024-04-12T04:10:48.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oIejvutGIGxAw9chgJSGrcTw2dU=', 'Content-Length': '0'} 2024-04-12T04:10:48.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcbbda6a2f147111-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.282 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:10:48.282 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:10:48.282 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.282 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:10:48.283 DEBUG:boto:Data: 2024-04-12T04:10:48.283 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.283 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.283 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.283 DEBUG:boto:Params: {} 2024-04-12T04:10:48.283 DEBUG:boto:Token: None 2024-04-12T04:10:48.283 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj76 2024-04-12T04:10:48.283 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oIejvutGIGxAw9chgJSGrcTw2dU= 2024-04-12T04:10:48.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oIejvutGIGxAw9chgJSGrcTw2dU=', 'Content-Length': '0'} 2024-04-12T04:10:48.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000debb4903b8839c1c-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:10:48.286 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:10:48.286 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:10:48.286 DEBUG:boto:Method: GET 2024-04-12T04:10:48.286 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:10:48.286 DEBUG:boto:Data: 2024-04-12T04:10:48.286 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.286 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.286 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.286 DEBUG:boto:Params: {} 2024-04-12T04:10:48.287 DEBUG:boto:Token: None 2024-04-12T04:10:48.287 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj76 2024-04-12T04:10:48.287 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ac1DtlVgiS3ab5ohGkitwmV/GDc= 2024-04-12T04:10:48.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ac1DtlVgiS3ab5ohGkitwmV/GDc=', 'Content-Length': '0'} 2024-04-12T04:10:48.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3fc524fc511d2ff-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.289 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:10:48.289 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:10:48.289 DEBUG:boto:Method: GET 2024-04-12T04:10:48.289 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:10:48.289 DEBUG:boto:Data: 2024-04-12T04:10:48.289 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.289 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.290 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.290 DEBUG:boto:Params: {} 2024-04-12T04:10:48.290 DEBUG:boto:Token: None 2024-04-12T04:10:48.290 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj76 2024-04-12T04:10:48.290 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ac1DtlVgiS3ab5ohGkitwmV/GDc= 2024-04-12T04:10:48.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ac1DtlVgiS3ab5ohGkitwmV/GDc=', 'Content-Length': '0'} 2024-04-12T04:10:48.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b36f1883b7ab6766-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.292 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.292 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.292 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:10:48.292 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:10:48.292 DEBUG:boto:Method: GET 2024-04-12T04:10:48.293 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:10:48.293 DEBUG:boto:Data: 2024-04-12T04:10:48.293 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.293 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.293 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.293 DEBUG:boto:Params: {} 2024-04-12T04:10:48.293 DEBUG:boto:Token: None 2024-04-12T04:10:48.293 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.293 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000178ded906f97f714-006618b448-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.296 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.297 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.297 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.297 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:10:48.297 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:10:48.297 DEBUG:boto:Method: GET 2024-04-12T04:10:48.297 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:10:48.297 DEBUG:boto:Data: 2024-04-12T04:10:48.297 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.297 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.297 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.298 DEBUG:boto:Params: {} 2024-04-12T04:10:48.298 DEBUG:boto:Token: None 2024-04-12T04:10:48.298 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.298 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f691bc67d98d7d1d-006618b448-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.303 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:10:48.304 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:10:48.304 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:10:48.304 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.304 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.304 DEBUG:boto:Method: GET 2024-04-12T04:10:48.304 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.304 DEBUG:boto:Data: 2024-04-12T04:10:48.304 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.304 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.304 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.304 DEBUG:boto:Params: {} 2024-04-12T04:10:48.304 DEBUG:boto:Token: None 2024-04-12T04:10:48.305 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.305 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XERjxiFQusdNeW7o0hTaDYynx9w= 2024-04-12T04:10:48.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XERjxiFQusdNeW7o0hTaDYynx9w=', 'Content-Length': '0'} 2024-04-12T04:10:48.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cfd772f35423207-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.350 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:10:48.351 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:10:48.351 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.351 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.351 DEBUG:boto:Method: GET 2024-04-12T04:10:48.351 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.351 DEBUG:boto:Data: 2024-04-12T04:10:48.351 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.351 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.351 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.351 DEBUG:boto:Params: {} 2024-04-12T04:10:48.351 DEBUG:boto:Token: None 2024-04-12T04:10:48.352 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.352 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XERjxiFQusdNeW7o0hTaDYynx9w= 2024-04-12T04:10:48.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XERjxiFQusdNeW7o0hTaDYynx9w=', 'Content-Length': '0'} 2024-04-12T04:10:48.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eab84c7da86e17ac-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.354 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:10:48.354 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:10:48.355 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.355 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.355 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.355 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.355 DEBUG:boto:Data: 2024-04-12T04:10:48.355 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.355 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.355 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.355 DEBUG:boto:Params: {} 2024-04-12T04:10:48.355 DEBUG:boto:Token: None 2024-04-12T04:10:48.356 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.356 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OZxaAbNwsHBHxsH66ErEvtYwp9E= 2024-04-12T04:10:48.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OZxaAbNwsHBHxsH66ErEvtYwp9E=', 'Content-Length': '0'} 2024-04-12T04:10:48.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c189483d31a96fa3-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.358 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:10:48.358 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:10:48.358 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.358 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.358 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.358 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.358 DEBUG:boto:Data: 2024-04-12T04:10:48.358 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.358 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.359 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.359 DEBUG:boto:Params: {} 2024-04-12T04:10:48.359 DEBUG:boto:Token: None 2024-04-12T04:10:48.359 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.359 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OZxaAbNwsHBHxsH66ErEvtYwp9E= 2024-04-12T04:10:48.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OZxaAbNwsHBHxsH66ErEvtYwp9E=', 'Content-Length': '0'} 2024-04-12T04:10:48.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3e5b5570c35bc75-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:10:48.362 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:10:48.362 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:10:48.362 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.362 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.362 DEBUG:boto:Method: GET 2024-04-12T04:10:48.362 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.362 DEBUG:boto:Data: 2024-04-12T04:10:48.362 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.362 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.362 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.362 DEBUG:boto:Params: {} 2024-04-12T04:10:48.363 DEBUG:boto:Token: None 2024-04-12T04:10:48.363 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.363 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XERjxiFQusdNeW7o0hTaDYynx9w= 2024-04-12T04:10:48.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XERjxiFQusdNeW7o0hTaDYynx9w=', 'Content-Length': '0'} 2024-04-12T04:10:48.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061d76a4a83aa85a3-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.365 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:10:48.365 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:10:48.365 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.365 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.365 DEBUG:boto:Method: GET 2024-04-12T04:10:48.365 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.366 DEBUG:boto:Data: 2024-04-12T04:10:48.366 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.366 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.366 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.366 DEBUG:boto:Params: {} 2024-04-12T04:10:48.366 DEBUG:boto:Token: None 2024-04-12T04:10:48.366 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:10:48.366 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XERjxiFQusdNeW7o0hTaDYynx9w= 2024-04-12T04:10:48.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XERjxiFQusdNeW7o0hTaDYynx9w=', 'Content-Length': '0'} 2024-04-12T04:10:48.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000560d4b402d5b2102-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.369 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.369 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.369 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:10:48.369 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:10:48.369 DEBUG:boto:Method: GET 2024-04-12T04:10:48.369 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:10:48.369 DEBUG:boto:Data: 2024-04-12T04:10:48.369 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.369 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.369 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.369 DEBUG:boto:Params: {} 2024-04-12T04:10:48.370 DEBUG:boto:Token: None 2024-04-12T04:10:48.370 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.370 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029f3240a0bba3737-006618b448-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.372 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.373 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.373 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.373 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:10:48.373 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:10:48.373 DEBUG:boto:Method: GET 2024-04-12T04:10:48.373 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:10:48.373 DEBUG:boto:Data: 2024-04-12T04:10:48.374 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.374 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.374 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.374 DEBUG:boto:Params: {} 2024-04-12T04:10:48.374 DEBUG:boto:Token: None 2024-04-12T04:10:48.374 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.374 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a16515bb6159abca-006618b448-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.378 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.379 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:10:48.379 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:10:48.379 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.379 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:10:48.379 DEBUG:boto:Data: 2024-04-12T04:10:48.379 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.379 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.379 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.379 DEBUG:boto:Params: {} 2024-04-12T04:10:48.380 DEBUG:boto:Token: None 2024-04-12T04:10:48.380 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj85 2024-04-12T04:10:48.380 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rcyg9QUQA1W1q0yPfIDaNhmv4YE= 2024-04-12T04:10:48.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rcyg9QUQA1W1q0yPfIDaNhmv4YE=', 'Content-Length': '0'} 2024-04-12T04:10:48.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3028f356e0838b6-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.382 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:10:48.382 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:10:48.382 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.382 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:10:48.382 DEBUG:boto:Data: 2024-04-12T04:10:48.383 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.383 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.383 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.383 DEBUG:boto:Params: {} 2024-04-12T04:10:48.383 DEBUG:boto:Token: None 2024-04-12T04:10:48.383 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj85 2024-04-12T04:10:48.383 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rcyg9QUQA1W1q0yPfIDaNhmv4YE= 2024-04-12T04:10:48.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rcyg9QUQA1W1q0yPfIDaNhmv4YE=', 'Content-Length': '0'} 2024-04-12T04:10:48.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d14634b56b6c2fd-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:10:48.385 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:10:48.385 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:10:48.386 DEBUG:boto:Method: GET 2024-04-12T04:10:48.386 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:10:48.386 DEBUG:boto:Data: 2024-04-12T04:10:48.386 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.386 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.386 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.386 DEBUG:boto:Params: {} 2024-04-12T04:10:48.386 DEBUG:boto:Token: None 2024-04-12T04:10:48.386 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj85 2024-04-12T04:10:48.386 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8f5kzroF299PcXJmh4v0eUAkPuo= 2024-04-12T04:10:48.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8f5kzroF299PcXJmh4v0eUAkPuo=', 'Content-Length': '0'} 2024-04-12T04:10:48.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cae4fad9d2981c7-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.389 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:10:48.389 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:10:48.389 DEBUG:boto:Method: GET 2024-04-12T04:10:48.389 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:10:48.389 DEBUG:boto:Data: 2024-04-12T04:10:48.389 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.389 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.389 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.389 DEBUG:boto:Params: {} 2024-04-12T04:10:48.389 DEBUG:boto:Token: None 2024-04-12T04:10:48.390 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj85 2024-04-12T04:10:48.390 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8f5kzroF299PcXJmh4v0eUAkPuo= 2024-04-12T04:10:48.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8f5kzroF299PcXJmh4v0eUAkPuo=', 'Content-Length': '0'} 2024-04-12T04:10:48.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fbed4fab42611b5-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.438 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.439 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.439 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:10:48.439 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:10:48.439 DEBUG:boto:Method: GET 2024-04-12T04:10:48.439 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:10:48.439 DEBUG:boto:Data: 2024-04-12T04:10:48.439 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.439 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.439 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.439 DEBUG:boto:Params: {} 2024-04-12T04:10:48.439 DEBUG:boto:Token: None 2024-04-12T04:10:48.440 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.440 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2a542c8f4f591a8-006618b448-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.442 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.443 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.443 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.443 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:10:48.443 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:10:48.443 DEBUG:boto:Method: GET 2024-04-12T04:10:48.443 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:10:48.443 DEBUG:boto:Data: 2024-04-12T04:10:48.443 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.444 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.444 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.444 DEBUG:boto:Params: {} 2024-04-12T04:10:48.444 DEBUG:boto:Token: None 2024-04-12T04:10:48.444 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.444 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f47aeb8d12150aaa-006618b448-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.449 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:10:48.449 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:10:48.449 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:10:48.449 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.450 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.450 DEBUG:boto:Method: GET 2024-04-12T04:10:48.450 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.450 DEBUG:boto:Data: 2024-04-12T04:10:48.450 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.450 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.450 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.450 DEBUG:boto:Params: {} 2024-04-12T04:10:48.450 DEBUG:boto:Token: None 2024-04-12T04:10:48.451 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.451 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UIDJGmDDwMhHRYutiaSSNJ0KhjI= 2024-04-12T04:10:48.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UIDJGmDDwMhHRYutiaSSNJ0KhjI=', 'Content-Length': '0'} 2024-04-12T04:10:48.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049bf046b107b5a2b-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.453 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:10:48.453 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:10:48.453 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.453 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.454 DEBUG:boto:Method: GET 2024-04-12T04:10:48.454 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.454 DEBUG:boto:Data: 2024-04-12T04:10:48.454 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.454 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.454 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.454 DEBUG:boto:Params: {} 2024-04-12T04:10:48.454 DEBUG:boto:Token: None 2024-04-12T04:10:48.454 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.455 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UIDJGmDDwMhHRYutiaSSNJ0KhjI= 2024-04-12T04:10:48.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UIDJGmDDwMhHRYutiaSSNJ0KhjI=', 'Content-Length': '0'} 2024-04-12T04:10:48.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5b93e92d2951dad-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.457 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:10:48.457 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:10:48.457 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.457 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.457 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.457 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.457 DEBUG:boto:Data: 2024-04-12T04:10:48.457 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.457 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.457 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.458 DEBUG:boto:Params: {} 2024-04-12T04:10:48.458 DEBUG:boto:Token: None 2024-04-12T04:10:48.458 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.458 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:y1Q6KJzNyi+qzpYTrdv5mS7C6ag= 2024-04-12T04:10:48.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y1Q6KJzNyi+qzpYTrdv5mS7C6ag=', 'Content-Length': '0'} 2024-04-12T04:10:48.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002009a20830b39278-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.460 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:10:48.460 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:10:48.460 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.460 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.460 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.461 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.461 DEBUG:boto:Data: 2024-04-12T04:10:48.461 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.461 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.461 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.461 DEBUG:boto:Params: {} 2024-04-12T04:10:48.461 DEBUG:boto:Token: None 2024-04-12T04:10:48.461 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.461 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:y1Q6KJzNyi+qzpYTrdv5mS7C6ag= 2024-04-12T04:10:48.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y1Q6KJzNyi+qzpYTrdv5mS7C6ag=', 'Content-Length': '0'} 2024-04-12T04:10:48.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000155a3f63da86d675-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:10:48.464 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:10:48.464 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:10:48.464 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.464 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.464 DEBUG:boto:Method: GET 2024-04-12T04:10:48.464 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.464 DEBUG:boto:Data: 2024-04-12T04:10:48.464 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.464 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.465 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.465 DEBUG:boto:Params: {} 2024-04-12T04:10:48.465 DEBUG:boto:Token: None 2024-04-12T04:10:48.465 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.465 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UIDJGmDDwMhHRYutiaSSNJ0KhjI= 2024-04-12T04:10:48.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UIDJGmDDwMhHRYutiaSSNJ0KhjI=', 'Content-Length': '0'} 2024-04-12T04:10:48.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006577307852431b01-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.467 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:10:48.468 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:10:48.468 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.468 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.468 DEBUG:boto:Method: GET 2024-04-12T04:10:48.468 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.468 DEBUG:boto:Data: 2024-04-12T04:10:48.468 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.469 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.469 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.469 DEBUG:boto:Params: {} 2024-04-12T04:10:48.469 DEBUG:boto:Token: None 2024-04-12T04:10:48.470 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:10:48.470 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UIDJGmDDwMhHRYutiaSSNJ0KhjI= 2024-04-12T04:10:48.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UIDJGmDDwMhHRYutiaSSNJ0KhjI=', 'Content-Length': '0'} 2024-04-12T04:10:48.472 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e849a469546f01c0-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.514 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.515 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.515 DEBUG:boto:path=/dkyges-61/?prefix=obj87 2024-04-12T04:10:48.515 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj87 2024-04-12T04:10:48.515 DEBUG:boto:Method: GET 2024-04-12T04:10:48.515 DEBUG:boto:Path: /dkyges-61/?prefix=obj87 2024-04-12T04:10:48.515 DEBUG:boto:Data: 2024-04-12T04:10:48.515 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.515 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.515 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.515 DEBUG:boto:Params: {} 2024-04-12T04:10:48.516 DEBUG:boto:Token: None 2024-04-12T04:10:48.516 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.516 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000983131572a7a7172-006618b448-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.518 DEBUG:boto:b'dkyges-61obj871000falseobj872024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.519 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.519 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.519 DEBUG:boto:path=/dkyges-61/?prefix=obj87 2024-04-12T04:10:48.519 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj87 2024-04-12T04:10:48.519 DEBUG:boto:Method: GET 2024-04-12T04:10:48.519 DEBUG:boto:Path: /dkyges-61/?prefix=obj87 2024-04-12T04:10:48.519 DEBUG:boto:Data: 2024-04-12T04:10:48.519 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.520 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.520 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.520 DEBUG:boto:Params: {} 2024-04-12T04:10:48.520 DEBUG:boto:Token: None 2024-04-12T04:10:48.520 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.520 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc411b3e5bc63d91-006618b448-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.524 DEBUG:boto:b'dkyges-61obj871000falseobj872024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.525 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:10:48.525 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:10:48.525 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.525 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:10:48.525 DEBUG:boto:Data: 2024-04-12T04:10:48.525 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.525 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.525 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.526 DEBUG:boto:Params: {} 2024-04-12T04:10:48.526 DEBUG:boto:Token: None 2024-04-12T04:10:48.526 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj87 2024-04-12T04:10:48.526 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oS/fC0L5rAiAsIp3yy8DFDajQkc= 2024-04-12T04:10:48.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oS/fC0L5rAiAsIp3yy8DFDajQkc=', 'Content-Length': '0'} 2024-04-12T04:10:48.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa5d1341dfc51304-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.528 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:10:48.528 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:10:48.528 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.528 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:10:48.528 DEBUG:boto:Data: 2024-04-12T04:10:48.528 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.529 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.529 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.529 DEBUG:boto:Params: {} 2024-04-12T04:10:48.529 DEBUG:boto:Token: None 2024-04-12T04:10:48.529 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj87 2024-04-12T04:10:48.529 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oS/fC0L5rAiAsIp3yy8DFDajQkc= 2024-04-12T04:10:48.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oS/fC0L5rAiAsIp3yy8DFDajQkc=', 'Content-Length': '0'} 2024-04-12T04:10:48.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058dccd432cde41c2-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:10:48.531 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:10:48.532 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:10:48.532 DEBUG:boto:Method: GET 2024-04-12T04:10:48.532 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:10:48.532 DEBUG:boto:Data: 2024-04-12T04:10:48.532 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.532 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.532 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.532 DEBUG:boto:Params: {} 2024-04-12T04:10:48.532 DEBUG:boto:Token: None 2024-04-12T04:10:48.532 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj87 2024-04-12T04:10:48.533 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Nrp4RCxe5VOpzj/fhsPo+56AY7g= 2024-04-12T04:10:48.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Nrp4RCxe5VOpzj/fhsPo+56AY7g=', 'Content-Length': '0'} 2024-04-12T04:10:48.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afccfa34354bb08b-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.578 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:10:48.578 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:10:48.579 DEBUG:boto:Method: GET 2024-04-12T04:10:48.579 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:10:48.579 DEBUG:boto:Data: 2024-04-12T04:10:48.579 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.579 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.579 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.579 DEBUG:boto:Params: {} 2024-04-12T04:10:48.579 DEBUG:boto:Token: None 2024-04-12T04:10:48.579 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj87 2024-04-12T04:10:48.580 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Nrp4RCxe5VOpzj/fhsPo+56AY7g= 2024-04-12T04:10:48.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Nrp4RCxe5VOpzj/fhsPo+56AY7g=', 'Content-Length': '0'} 2024-04-12T04:10:48.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbc7c3c820f15153-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.583 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.583 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.583 DEBUG:boto:path=/dkyges-61/?prefix=obj87 2024-04-12T04:10:48.584 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj87 2024-04-12T04:10:48.584 DEBUG:boto:Method: GET 2024-04-12T04:10:48.584 DEBUG:boto:Path: /dkyges-61/?prefix=obj87 2024-04-12T04:10:48.584 DEBUG:boto:Data: 2024-04-12T04:10:48.584 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.584 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.584 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.584 DEBUG:boto:Params: {} 2024-04-12T04:10:48.584 DEBUG:boto:Token: None 2024-04-12T04:10:48.584 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.585 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bae47be4f18ad9d-006618b448-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.587 DEBUG:boto:b'dkyges-61obj871000falseobj872024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.588 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.588 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.588 DEBUG:boto:path=/dkyges-61/?prefix=obj87 2024-04-12T04:10:48.588 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj87 2024-04-12T04:10:48.588 DEBUG:boto:Method: GET 2024-04-12T04:10:48.588 DEBUG:boto:Path: /dkyges-61/?prefix=obj87 2024-04-12T04:10:48.588 DEBUG:boto:Data: 2024-04-12T04:10:48.589 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.589 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.589 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.589 DEBUG:boto:Params: {} 2024-04-12T04:10:48.589 DEBUG:boto:Token: None 2024-04-12T04:10:48.589 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.589 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b3b0c221649943e-006618b448-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.638 DEBUG:boto:b'dkyges-61obj871000falseobj872024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:10:48.639 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:10:48.639 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:10:48.639 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.639 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.640 DEBUG:boto:Method: GET 2024-04-12T04:10:48.640 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.640 DEBUG:boto:Data: 2024-04-12T04:10:48.640 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.640 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.641 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.641 DEBUG:boto:Params: {} 2024-04-12T04:10:48.641 DEBUG:boto:Token: None 2024-04-12T04:10:48.641 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.641 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZrxoaP/uf43a8qDH6hS06SUUbkw= 2024-04-12T04:10:48.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZrxoaP/uf43a8qDH6hS06SUUbkw=', 'Content-Length': '0'} 2024-04-12T04:10:48.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c23fcdde4ccd73e8-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.643 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:10:48.644 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:10:48.644 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.644 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.644 DEBUG:boto:Method: GET 2024-04-12T04:10:48.644 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.644 DEBUG:boto:Data: 2024-04-12T04:10:48.644 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.645 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.645 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.645 DEBUG:boto:Params: {} 2024-04-12T04:10:48.645 DEBUG:boto:Token: None 2024-04-12T04:10:48.645 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.645 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZrxoaP/uf43a8qDH6hS06SUUbkw= 2024-04-12T04:10:48.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZrxoaP/uf43a8qDH6hS06SUUbkw=', 'Content-Length': '0'} 2024-04-12T04:10:48.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021f0ccee0879ace8-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.648 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:10:48.648 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:10:48.648 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.648 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.648 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.648 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.648 DEBUG:boto:Data: 2024-04-12T04:10:48.648 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.649 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.649 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.649 DEBUG:boto:Params: {} 2024-04-12T04:10:48.649 DEBUG:boto:Token: None 2024-04-12T04:10:48.649 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.649 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JAmOwQeQYa6mNKOD+z7ftjSz09k= 2024-04-12T04:10:48.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JAmOwQeQYa6mNKOD+z7ftjSz09k=', 'Content-Length': '0'} 2024-04-12T04:10:48.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dd6ccc3eae1d9b0-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.651 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:10:48.651 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:10:48.651 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.651 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.651 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.652 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.652 DEBUG:boto:Data: 2024-04-12T04:10:48.652 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.652 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.652 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.653 DEBUG:boto:Params: {} 2024-04-12T04:10:48.653 DEBUG:boto:Token: None 2024-04-12T04:10:48.653 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.653 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JAmOwQeQYa6mNKOD+z7ftjSz09k= 2024-04-12T04:10:48.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JAmOwQeQYa6mNKOD+z7ftjSz09k=', 'Content-Length': '0'} 2024-04-12T04:10:48.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a33c647d16fee67-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:10:48.655 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:10:48.655 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:10:48.655 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.655 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.655 DEBUG:boto:Method: GET 2024-04-12T04:10:48.656 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.656 DEBUG:boto:Data: 2024-04-12T04:10:48.656 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.656 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.656 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.656 DEBUG:boto:Params: {} 2024-04-12T04:10:48.656 DEBUG:boto:Token: None 2024-04-12T04:10:48.656 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZrxoaP/uf43a8qDH6hS06SUUbkw= 2024-04-12T04:10:48.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZrxoaP/uf43a8qDH6hS06SUUbkw=', 'Content-Length': '0'} 2024-04-12T04:10:48.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b7e80d4ce7fb16d-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.698 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:10:48.698 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:10:48.699 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.699 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.699 DEBUG:boto:Method: GET 2024-04-12T04:10:48.699 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.699 DEBUG:boto:Data: 2024-04-12T04:10:48.699 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.699 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.699 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.699 DEBUG:boto:Params: {} 2024-04-12T04:10:48.699 DEBUG:boto:Token: None 2024-04-12T04:10:48.700 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:10:48.700 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZrxoaP/uf43a8qDH6hS06SUUbkw= 2024-04-12T04:10:48.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZrxoaP/uf43a8qDH6hS06SUUbkw=', 'Content-Length': '0'} 2024-04-12T04:10:48.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000107517c6ad03c568-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.702 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.702 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.702 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:10:48.702 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:10:48.702 DEBUG:boto:Method: GET 2024-04-12T04:10:48.702 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:10:48.702 DEBUG:boto:Data: 2024-04-12T04:10:48.702 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.703 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.703 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.703 DEBUG:boto:Params: {} 2024-04-12T04:10:48.703 DEBUG:boto:Token: None 2024-04-12T04:10:48.703 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.703 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011ce13109953482e-006618b448-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.706 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.706 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.707 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.707 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:10:48.707 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:10:48.707 DEBUG:boto:Method: GET 2024-04-12T04:10:48.707 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:10:48.707 DEBUG:boto:Data: 2024-04-12T04:10:48.707 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.707 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.707 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.707 DEBUG:boto:Params: {} 2024-04-12T04:10:48.708 DEBUG:boto:Token: None 2024-04-12T04:10:48.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.708 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d69c7b179c3c07ea-006618b448-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.713 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.713 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:10:48.713 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:10:48.713 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.713 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:10:48.713 DEBUG:boto:Data: 2024-04-12T04:10:48.713 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.713 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.714 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.714 DEBUG:boto:Params: {} 2024-04-12T04:10:48.714 DEBUG:boto:Token: None 2024-04-12T04:10:48.714 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj92 2024-04-12T04:10:48.714 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:akqjb0ucNhMBh5PjIU1i5lEyFIg= 2024-04-12T04:10:48.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:akqjb0ucNhMBh5PjIU1i5lEyFIg=', 'Content-Length': '0'} 2024-04-12T04:10:48.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eec44d81786b72fb-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.716 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:10:48.717 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:10:48.717 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.717 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:10:48.717 DEBUG:boto:Data: 2024-04-12T04:10:48.717 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.717 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.718 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.718 DEBUG:boto:Params: {} 2024-04-12T04:10:48.718 DEBUG:boto:Token: None 2024-04-12T04:10:48.718 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj92 2024-04-12T04:10:48.718 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:akqjb0ucNhMBh5PjIU1i5lEyFIg= 2024-04-12T04:10:48.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:akqjb0ucNhMBh5PjIU1i5lEyFIg=', 'Content-Length': '0'} 2024-04-12T04:10:48.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051a01d810762e49c-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:10:48.721 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:10:48.722 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:10:48.722 DEBUG:boto:Method: GET 2024-04-12T04:10:48.722 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:10:48.722 DEBUG:boto:Data: 2024-04-12T04:10:48.723 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.723 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.723 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.723 DEBUG:boto:Params: {} 2024-04-12T04:10:48.723 DEBUG:boto:Token: None 2024-04-12T04:10:48.723 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj92 2024-04-12T04:10:48.724 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BBVvSSMfcby/GPXF7vv+zKlhhZ0= 2024-04-12T04:10:48.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BBVvSSMfcby/GPXF7vv+zKlhhZ0=', 'Content-Length': '0'} 2024-04-12T04:10:48.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a2e2dc5b2c6f055-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.726 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:10:48.726 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:10:48.727 DEBUG:boto:Method: GET 2024-04-12T04:10:48.727 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:10:48.727 DEBUG:boto:Data: 2024-04-12T04:10:48.727 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.727 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.727 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.727 DEBUG:boto:Params: {} 2024-04-12T04:10:48.727 DEBUG:boto:Token: None 2024-04-12T04:10:48.727 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj92 2024-04-12T04:10:48.728 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BBVvSSMfcby/GPXF7vv+zKlhhZ0= 2024-04-12T04:10:48.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BBVvSSMfcby/GPXF7vv+zKlhhZ0=', 'Content-Length': '0'} 2024-04-12T04:10:48.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0d76f4b3c3c3ed9-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.770 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.771 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.771 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:10:48.771 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:10:48.771 DEBUG:boto:Method: GET 2024-04-12T04:10:48.771 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:10:48.771 DEBUG:boto:Data: 2024-04-12T04:10:48.771 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.771 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.771 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.771 DEBUG:boto:Params: {} 2024-04-12T04:10:48.772 DEBUG:boto:Token: None 2024-04-12T04:10:48.772 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.772 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.774 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af771994aa872398-006618b448-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.774 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.775 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.775 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.775 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:10:48.775 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:10:48.775 DEBUG:boto:Method: GET 2024-04-12T04:10:48.775 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:10:48.776 DEBUG:boto:Data: 2024-04-12T04:10:48.776 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.776 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.776 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.776 DEBUG:boto:Params: {} 2024-04-12T04:10:48.776 DEBUG:boto:Token: None 2024-04-12T04:10:48.776 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.776 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c3e5c913366ac2f-006618b448-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.781 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T04:10:48.782 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:10:48.782 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:10:48.782 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.782 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.782 DEBUG:boto:Method: GET 2024-04-12T04:10:48.782 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.782 DEBUG:boto:Data: 2024-04-12T04:10:48.782 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.782 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.782 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.783 DEBUG:boto:Params: {} 2024-04-12T04:10:48.783 DEBUG:boto:Token: None 2024-04-12T04:10:48.783 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.783 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5PmyOJ9hQPA1TSdlE9/pYVa48UY= 2024-04-12T04:10:48.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5PmyOJ9hQPA1TSdlE9/pYVa48UY=', 'Content-Length': '0'} 2024-04-12T04:10:48.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003391ae0a742c66ba-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.785 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:10:48.785 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:10:48.785 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.786 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.786 DEBUG:boto:Method: GET 2024-04-12T04:10:48.786 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.786 DEBUG:boto:Data: 2024-04-12T04:10:48.786 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.786 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.786 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.786 DEBUG:boto:Params: {} 2024-04-12T04:10:48.786 DEBUG:boto:Token: None 2024-04-12T04:10:48.786 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.787 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5PmyOJ9hQPA1TSdlE9/pYVa48UY= 2024-04-12T04:10:48.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5PmyOJ9hQPA1TSdlE9/pYVa48UY=', 'Content-Length': '0'} 2024-04-12T04:10:48.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd20409f2c671bca-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.789 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:10:48.789 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:10:48.790 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.790 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.790 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.790 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.790 DEBUG:boto:Data: 2024-04-12T04:10:48.790 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.791 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.791 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.791 DEBUG:boto:Params: {} 2024-04-12T04:10:48.791 DEBUG:boto:Token: None 2024-04-12T04:10:48.792 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.792 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6EHkSitzl9IqXTQiI7HMzUv+2xo= 2024-04-12T04:10:48.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6EHkSitzl9IqXTQiI7HMzUv+2xo=', 'Content-Length': '0'} 2024-04-12T04:10:48.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019316f82945a5eb6-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.794 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:10:48.794 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:10:48.794 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.794 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.794 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.794 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.795 DEBUG:boto:Data: 2024-04-12T04:10:48.795 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.795 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.795 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.795 DEBUG:boto:Params: {} 2024-04-12T04:10:48.795 DEBUG:boto:Token: None 2024-04-12T04:10:48.795 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.795 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6EHkSitzl9IqXTQiI7HMzUv+2xo= 2024-04-12T04:10:48.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6EHkSitzl9IqXTQiI7HMzUv+2xo=', 'Content-Length': '0'} 2024-04-12T04:10:48.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003df9f031502985ad-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T04:10:48.798 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:10:48.798 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:10:48.798 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.798 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.798 DEBUG:boto:Method: GET 2024-04-12T04:10:48.798 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.798 DEBUG:boto:Data: 2024-04-12T04:10:48.798 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.798 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.798 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.798 DEBUG:boto:Params: {} 2024-04-12T04:10:48.799 DEBUG:boto:Token: None 2024-04-12T04:10:48.799 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.799 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5PmyOJ9hQPA1TSdlE9/pYVa48UY= 2024-04-12T04:10:48.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5PmyOJ9hQPA1TSdlE9/pYVa48UY=', 'Content-Length': '0'} 2024-04-12T04:10:48.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bf859e62bcd5ba8-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.801 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:10:48.801 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:10:48.801 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.801 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.801 DEBUG:boto:Method: GET 2024-04-12T04:10:48.801 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.801 DEBUG:boto:Data: 2024-04-12T04:10:48.801 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.802 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.802 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.802 DEBUG:boto:Params: {} 2024-04-12T04:10:48.802 DEBUG:boto:Token: None 2024-04-12T04:10:48.802 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:10:48.802 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5PmyOJ9hQPA1TSdlE9/pYVa48UY= 2024-04-12T04:10:48.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5PmyOJ9hQPA1TSdlE9/pYVa48UY=', 'Content-Length': '0'} 2024-04-12T04:10:48.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003650948eef50d958-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.805 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.805 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.805 DEBUG:boto:path=/dkyges-61/?prefix=obj93 2024-04-12T04:10:48.805 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj93 2024-04-12T04:10:48.805 DEBUG:boto:Method: GET 2024-04-12T04:10:48.805 DEBUG:boto:Path: /dkyges-61/?prefix=obj93 2024-04-12T04:10:48.805 DEBUG:boto:Data: 2024-04-12T04:10:48.805 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.805 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.805 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.805 DEBUG:boto:Params: {} 2024-04-12T04:10:48.805 DEBUG:boto:Token: None 2024-04-12T04:10:48.806 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.806 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e972b06d2b728819-006618b448-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.809 DEBUG:boto:b'dkyges-61obj931000falseobj932024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.809 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.809 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.809 DEBUG:boto:path=/dkyges-61/?prefix=obj93 2024-04-12T04:10:48.809 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj93 2024-04-12T04:10:48.809 DEBUG:boto:Method: GET 2024-04-12T04:10:48.810 DEBUG:boto:Path: /dkyges-61/?prefix=obj93 2024-04-12T04:10:48.810 DEBUG:boto:Data: 2024-04-12T04:10:48.810 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.810 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.810 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.810 DEBUG:boto:Params: {} 2024-04-12T04:10:48.810 DEBUG:boto:Token: None 2024-04-12T04:10:48.810 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.811 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec1463dbad01f43d-006618b448-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.815 DEBUG:boto:b'dkyges-61obj931000falseobj932024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.816 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:10:48.816 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:10:48.816 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.816 DEBUG:boto:Path: /dkyges-61/obj93 2024-04-12T04:10:48.816 DEBUG:boto:Data: 2024-04-12T04:10:48.816 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.816 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.817 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.817 DEBUG:boto:Params: {} 2024-04-12T04:10:48.817 DEBUG:boto:Token: None 2024-04-12T04:10:48.817 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj93 2024-04-12T04:10:48.817 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uuDnf3lYBYozrYXrliIomoD0y+k= 2024-04-12T04:10:48.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uuDnf3lYBYozrYXrliIomoD0y+k=', 'Content-Length': '0'} 2024-04-12T04:10:48.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d901686472ebcf2-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.819 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:10:48.819 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:10:48.819 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.820 DEBUG:boto:Path: /dkyges-61/obj93 2024-04-12T04:10:48.820 DEBUG:boto:Data: 2024-04-12T04:10:48.820 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.820 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.820 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.820 DEBUG:boto:Params: {} 2024-04-12T04:10:48.820 DEBUG:boto:Token: None 2024-04-12T04:10:48.820 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj93 2024-04-12T04:10:48.820 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uuDnf3lYBYozrYXrliIomoD0y+k= 2024-04-12T04:10:48.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uuDnf3lYBYozrYXrliIomoD0y+k=', 'Content-Length': '0'} 2024-04-12T04:10:48.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acefc466aed1f7b4-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T04:10:48.823 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:10:48.823 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:10:48.823 DEBUG:boto:Method: GET 2024-04-12T04:10:48.823 DEBUG:boto:Path: /dkyges-61/obj93 2024-04-12T04:10:48.823 DEBUG:boto:Data: 2024-04-12T04:10:48.823 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.823 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.823 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.823 DEBUG:boto:Params: {} 2024-04-12T04:10:48.823 DEBUG:boto:Token: None 2024-04-12T04:10:48.823 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj93 2024-04-12T04:10:48.824 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U0/3dnrROzPCpTSYD3secrAJlVU= 2024-04-12T04:10:48.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U0/3dnrROzPCpTSYD3secrAJlVU=', 'Content-Length': '0'} 2024-04-12T04:10:48.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010b3a1aebfa99c5e-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.870 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:10:48.871 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:10:48.871 DEBUG:boto:Method: GET 2024-04-12T04:10:48.871 DEBUG:boto:Path: /dkyges-61/obj93 2024-04-12T04:10:48.871 DEBUG:boto:Data: 2024-04-12T04:10:48.871 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.871 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.871 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.872 DEBUG:boto:Params: {} 2024-04-12T04:10:48.872 DEBUG:boto:Token: None 2024-04-12T04:10:48.872 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj93 2024-04-12T04:10:48.872 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U0/3dnrROzPCpTSYD3secrAJlVU= 2024-04-12T04:10:48.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U0/3dnrROzPCpTSYD3secrAJlVU=', 'Content-Length': '0'} 2024-04-12T04:10:48.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aa9c805966f40bc-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.875 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.875 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.875 DEBUG:boto:path=/dkyges-61/?prefix=obj93 2024-04-12T04:10:48.875 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj93 2024-04-12T04:10:48.875 DEBUG:boto:Method: GET 2024-04-12T04:10:48.875 DEBUG:boto:Path: /dkyges-61/?prefix=obj93 2024-04-12T04:10:48.875 DEBUG:boto:Data: 2024-04-12T04:10:48.875 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.875 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.875 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.876 DEBUG:boto:Params: {} 2024-04-12T04:10:48.876 DEBUG:boto:Token: None 2024-04-12T04:10:48.876 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.876 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faf1a35430714777-006618b448-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.879 DEBUG:boto:b'dkyges-61obj931000falseobj932024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.879 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.879 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.879 DEBUG:boto:path=/dkyges-61/?prefix=obj93 2024-04-12T04:10:48.879 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj93 2024-04-12T04:10:48.879 DEBUG:boto:Method: GET 2024-04-12T04:10:48.879 DEBUG:boto:Path: /dkyges-61/?prefix=obj93 2024-04-12T04:10:48.880 DEBUG:boto:Data: 2024-04-12T04:10:48.880 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.880 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.880 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.880 DEBUG:boto:Params: {} 2024-04-12T04:10:48.880 DEBUG:boto:Token: None 2024-04-12T04:10:48.880 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.880 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0da05178fa43811-006618b448-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.884 DEBUG:boto:b'dkyges-61obj931000falseobj932024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:10:48.885 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:10:48.885 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:10:48.885 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.885 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.885 DEBUG:boto:Method: GET 2024-04-12T04:10:48.885 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.885 DEBUG:boto:Data: 2024-04-12T04:10:48.885 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.886 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.886 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.886 DEBUG:boto:Params: {} 2024-04-12T04:10:48.886 DEBUG:boto:Token: None 2024-04-12T04:10:48.886 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.886 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G8XxNtcjTQqOGNvPm8f1M3VbbfE= 2024-04-12T04:10:48.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G8XxNtcjTQqOGNvPm8f1M3VbbfE=', 'Content-Length': '0'} 2024-04-12T04:10:48.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccc7b531e37658fe-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.888 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:10:48.888 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:10:48.889 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.889 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.889 DEBUG:boto:Method: GET 2024-04-12T04:10:48.889 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.889 DEBUG:boto:Data: 2024-04-12T04:10:48.889 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.889 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.889 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.889 DEBUG:boto:Params: {} 2024-04-12T04:10:48.889 DEBUG:boto:Token: None 2024-04-12T04:10:48.889 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.890 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G8XxNtcjTQqOGNvPm8f1M3VbbfE= 2024-04-12T04:10:48.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G8XxNtcjTQqOGNvPm8f1M3VbbfE=', 'Content-Length': '0'} 2024-04-12T04:10:48.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007daf6f2f1a9d3b65-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.892 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:10:48.892 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:10:48.892 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.892 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.892 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.892 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.892 DEBUG:boto:Data: 2024-04-12T04:10:48.892 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.893 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.893 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.893 DEBUG:boto:Params: {} 2024-04-12T04:10:48.893 DEBUG:boto:Token: None 2024-04-12T04:10:48.893 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.893 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tibE3oIEeYh9k0zMaI/n0goy62I= 2024-04-12T04:10:48.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tibE3oIEeYh9k0zMaI/n0goy62I=', 'Content-Length': '0'} 2024-04-12T04:10:48.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ac4a952bf80833a-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.895 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:10:48.895 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:10:48.895 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.895 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.895 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.896 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.896 DEBUG:boto:Data: 2024-04-12T04:10:48.896 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.896 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.896 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.896 DEBUG:boto:Params: {} 2024-04-12T04:10:48.896 DEBUG:boto:Token: None 2024-04-12T04:10:48.896 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.896 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tibE3oIEeYh9k0zMaI/n0goy62I= 2024-04-12T04:10:48.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tibE3oIEeYh9k0zMaI/n0goy62I=', 'Content-Length': '0'} 2024-04-12T04:10:48.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034300fad2deac7d5-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:10:48.899 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:10:48.899 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:10:48.899 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.899 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.899 DEBUG:boto:Method: GET 2024-04-12T04:10:48.899 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.899 DEBUG:boto:Data: 2024-04-12T04:10:48.899 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.899 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.899 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.899 DEBUG:boto:Params: {} 2024-04-12T04:10:48.900 DEBUG:boto:Token: None 2024-04-12T04:10:48.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.900 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G8XxNtcjTQqOGNvPm8f1M3VbbfE= 2024-04-12T04:10:48.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G8XxNtcjTQqOGNvPm8f1M3VbbfE=', 'Content-Length': '0'} 2024-04-12T04:10:48.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000868ae7dc78cf30c9-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.942 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:10:48.942 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:10:48.943 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.943 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.943 DEBUG:boto:Method: GET 2024-04-12T04:10:48.943 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.943 DEBUG:boto:Data: 2024-04-12T04:10:48.943 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.943 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.943 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.943 DEBUG:boto:Params: {} 2024-04-12T04:10:48.943 DEBUG:boto:Token: None 2024-04-12T04:10:48.944 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:10:48.944 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G8XxNtcjTQqOGNvPm8f1M3VbbfE= 2024-04-12T04:10:48.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G8XxNtcjTQqOGNvPm8f1M3VbbfE=', 'Content-Length': '0'} 2024-04-12T04:10:48.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000addebd6ebd550cf9-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.946 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.946 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.946 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:10:48.946 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:10:48.947 DEBUG:boto:Method: GET 2024-04-12T04:10:48.947 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:10:48.947 DEBUG:boto:Data: 2024-04-12T04:10:48.947 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.947 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.947 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.947 DEBUG:boto:Params: {} 2024-04-12T04:10:48.947 DEBUG:boto:Token: None 2024-04-12T04:10:48.947 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.947 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c265512a89d58dac-006618b448-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.950 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.951 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:48.951 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:48.951 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:10:48.951 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:10:48.951 DEBUG:boto:Method: GET 2024-04-12T04:10:48.951 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:10:48.951 DEBUG:boto:Data: 2024-04-12T04:10:48.951 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.951 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.951 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.952 DEBUG:boto:Params: {} 2024-04-12T04:10:48.952 DEBUG:boto:Token: None 2024-04-12T04:10:48.952 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/ 2024-04-12T04:10:48.952 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM= 2024-04-12T04:10:48.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RI7yu+WbWHaFtmfwITEeYSyP4RM=', 'Content-Length': '0'} 2024-04-12T04:10:48.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007514fec116702720-006618b448-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.957 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:48.957 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:10:48.957 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:10:48.957 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.957 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:10:48.957 DEBUG:boto:Data: 2024-04-12T04:10:48.957 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.957 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.957 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.958 DEBUG:boto:Params: {} 2024-04-12T04:10:48.958 DEBUG:boto:Token: None 2024-04-12T04:10:48.958 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj95 2024-04-12T04:10:48.958 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:47gUzb6Vy+CfAM40EKgGJZi+rY4= 2024-04-12T04:10:48.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:47gUzb6Vy+CfAM40EKgGJZi+rY4=', 'Content-Length': '0'} 2024-04-12T04:10:48.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062d0055c82e18fd6-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.960 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:10:48.960 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:10:48.960 DEBUG:boto:Method: HEAD 2024-04-12T04:10:48.960 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:10:48.960 DEBUG:boto:Data: 2024-04-12T04:10:48.960 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.961 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.961 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.961 DEBUG:boto:Params: {} 2024-04-12T04:10:48.961 DEBUG:boto:Token: None 2024-04-12T04:10:48.961 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj95 2024-04-12T04:10:48.961 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:47gUzb6Vy+CfAM40EKgGJZi+rY4= 2024-04-12T04:10:48.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:47gUzb6Vy+CfAM40EKgGJZi+rY4=', 'Content-Length': '0'} 2024-04-12T04:10:48.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008501d2a661577f6-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:10:48.963 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:10:48.963 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:10:48.963 DEBUG:boto:Method: GET 2024-04-12T04:10:48.963 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:10:48.964 DEBUG:boto:Data: 2024-04-12T04:10:48.964 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.964 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:48.964 DEBUG:boto:Port: 8001 2024-04-12T04:10:48.964 DEBUG:boto:Params: {} 2024-04-12T04:10:48.964 DEBUG:boto:Token: None 2024-04-12T04:10:48.964 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj95 2024-04-12T04:10:48.964 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PjmcL8mX5BMi+NZpKDnBuIYeOqM= 2024-04-12T04:10:48.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PjmcL8mX5BMi+NZpKDnBuIYeOqM=', 'Content-Length': '0'} 2024-04-12T04:10:48.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000735f81f0391b5f9b-006618b448-1cfd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:48.966 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:10:48.966 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:10:48.966 DEBUG:boto:Method: GET 2024-04-12T04:10:48.967 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:10:48.967 DEBUG:boto:Data: 2024-04-12T04:10:48.967 DEBUG:boto:Headers: {} 2024-04-12T04:10:48.967 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:48.967 DEBUG:boto:Port: 8000 2024-04-12T04:10:48.967 DEBUG:boto:Params: {} 2024-04-12T04:10:48.967 DEBUG:boto:Token: None 2024-04-12T04:10:48.967 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:48 GMT /dkyges-61/obj95 2024-04-12T04:10:48.967 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PjmcL8mX5BMi+NZpKDnBuIYeOqM= 2024-04-12T04:10:48.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PjmcL8mX5BMi+NZpKDnBuIYeOqM=', 'Content-Length': '0'} 2024-04-12T04:10:48.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf401b5d168b92e4-006618b448-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:10:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:49.010 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:49.010 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:49.011 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:10:49.011 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:10:49.011 DEBUG:boto:Method: GET 2024-04-12T04:10:49.011 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:10:49.011 DEBUG:boto:Data: 2024-04-12T04:10:49.011 DEBUG:boto:Headers: {} 2024-04-12T04:10:49.011 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:10:49.011 DEBUG:boto:Port: 8001 2024-04-12T04:10:49.011 DEBUG:boto:Params: {} 2024-04-12T04:10:49.011 DEBUG:boto:Token: None 2024-04-12T04:10:49.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:49 GMT /dkyges-61/ 2024-04-12T04:10:49.012 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:x9xBaZKKscmHfF+lsxYppa2htOo= 2024-04-12T04:10:49.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:x9xBaZKKscmHfF+lsxYppa2htOo=', 'Content-Length': '0'} 2024-04-12T04:10:49.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e5181baf8f291cd-006618b449-1cfd-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:49.014 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:49.014 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:10:49.014 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:10:49.015 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:10:49.015 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:10:49.015 DEBUG:boto:Method: GET 2024-04-12T04:10:49.015 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:10:49.015 DEBUG:boto:Data: 2024-04-12T04:10:49.015 DEBUG:boto:Headers: {} 2024-04-12T04:10:49.015 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:10:49.015 DEBUG:boto:Port: 8000 2024-04-12T04:10:49.015 DEBUG:boto:Params: {} 2024-04-12T04:10:49.015 DEBUG:boto:Token: None 2024-04-12T04:10:49.015 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:10:49 GMT /dkyges-61/ 2024-04-12T04:10:49.015 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:x9xBaZKKscmHfF+lsxYppa2htOo= 2024-04-12T04:10:49.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:10:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:x9xBaZKKscmHfF+lsxYppa2htOo=', 'Content-Length': '0'} 2024-04-12T04:10:49.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048afa29762f9d772-006618b449-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:10:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:10:49.020 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:10:49.020 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-61 zones={a2, a1} 2024-04-12T04:10:49.021 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T04:10:49.021 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:10:49.022 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T04:10:49.038+0000 7ff77ccbf700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 91532) UID: 0 2024-04-12T04:10:49.022 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T04:10:49.038+0000 7ff781e947c0 -1 shutting down 2024-04-12T04:10:50.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T04:10:55.121 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T04:10:55.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:10:55.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:10:55.122 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:10:55.187 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:55.187 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:55.232 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:10:55.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.232 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj11", 2024-04-12T04:10:55.233 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.233 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.233 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.233 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T04:10:55.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.233 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.233 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.233 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.233 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.233 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:10:55.234 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:30.444861Z", 2024-04-12T04:10:55.234 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.234 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.234 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.234 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.234 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.234 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.234 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.234 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.235 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_duPWA7IoyEBU1N4JM8DBgrrv55Z40Bo", 2024-04-12T04:10:55.235 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.235 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.235 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.235 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj111", 2024-04-12T04:10:55.235 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.236 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.236 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.236 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 8 2024-04-12T04:10:55.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.236 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.236 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.236 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.236 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.236 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:10:55.237 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:09:24.100799Z", 2024-04-12T04:10:55.237 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.237 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.237 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.237 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.237 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.237 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.237 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.237 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.238 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_7UME4hrSO6hqpJbe_FAxnZ0csVHGGqX", 2024-04-12T04:10:55.238 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.238 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.238 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.238 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj115", 2024-04-12T04:10:55.238 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.238 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.238 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2 2024-04-12T04:10:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:10:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:05:01.190659Z", 2024-04-12T04:10:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_rqreZm4ChNuDljgohvQd0zrT4uNNa70", 2024-04-12T04:10:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj116", 2024-04-12T04:10:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2 2024-04-12T04:10:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:10:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:06:49.199561Z", 2024-04-12T04:10:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_k0QHvdFwXn4vErEfy87SUvHi0bdOVtE", 2024-04-12T04:10:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj124", 2024-04-12T04:10:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T04:10:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:10:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:08:08.248308Z", 2024-04-12T04:10:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.245 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.245 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.245 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_R-UMGzTv6VmkPgt7lbx30m86WBPpFoN", 2024-04-12T04:10:55.245 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.245 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.245 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.245 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj145", 2024-04-12T04:10:55.245 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.245 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.246 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.246 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 8 2024-04-12T04:10:55.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.246 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.246 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.246 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.246 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.246 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:10:55.246 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:06:49.211076Z", 2024-04-12T04:10:55.246 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.246 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.247 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.247 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.247 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.247 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.247 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.247 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.247 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_Q011hwdc3NIgbrq7Pzt3z6iz9H5iGcp", 2024-04-12T04:10:55.247 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.247 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.247 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.248 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj148", 2024-04-12T04:10:55.248 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.248 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.248 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.248 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 38 2024-04-12T04:10:55.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.249 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.249 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.249 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.249 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.249 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:10:55.249 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:09:24.114675Z", 2024-04-12T04:10:55.249 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.249 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.249 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.250 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.250 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.250 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.250 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.250 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.250 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_l5gTK1O6H0VgGMYzJjzPx3nX9d2QZcw", 2024-04-12T04:10:55.250 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.250 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.251 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.251 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj155", 2024-04-12T04:10:55.251 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.251 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.251 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.251 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T04:10:55.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.251 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.252 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.252 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.252 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.252 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:10:55.252 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:08:08.299737Z", 2024-04-12T04:10:55.252 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.252 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.252 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.252 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.252 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.252 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.252 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.253 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.253 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_POH149JtT3_huIrLrutv9NOZCPL1XKM", 2024-04-12T04:10:55.253 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.253 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.253 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.253 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj17", 2024-04-12T04:10:55.253 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.253 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.253 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.254 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T04:10:55.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.254 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.254 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.254 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.254 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.254 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:10:55.254 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:41.675980Z", 2024-04-12T04:10:55.254 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.254 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.255 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.255 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.255 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.255 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.255 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.255 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.255 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_1AtRV8hfX89VK_mzeCHP5zvvp7WURnB", 2024-04-12T04:10:55.255 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.255 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.255 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.256 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj185", 2024-04-12T04:10:55.256 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.256 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.256 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.256 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 21 2024-04-12T04:10:55.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.256 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.256 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.256 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.256 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.257 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T04:10:55.257 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:09:24.169265Z", 2024-04-12T04:10:55.257 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.257 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.257 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.257 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.257 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.257 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.257 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.257 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.258 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_CHmhlSJuQgE6EY13nO0fC8u4jDx-T2s", 2024-04-12T04:10:55.258 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.258 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.258 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.258 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj19", 2024-04-12T04:10:55.258 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.258 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.258 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.259 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T04:10:55.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.259 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.259 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.259 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.259 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.259 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:10:55.259 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:03:03.582137Z", 2024-04-12T04:10:55.259 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.259 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.260 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.260 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.260 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.260 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.260 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.260 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.260 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_jHFQyS2XIHXhyAOdnZb6N3O45OJfckT", 2024-04-12T04:10:55.260 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.260 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.260 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.261 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj22", 2024-04-12T04:10:55.261 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.261 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.261 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.261 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T04:10:55.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.261 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.261 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.261 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.261 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.262 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:10:55.262 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:30.456641Z", 2024-04-12T04:10:55.262 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.262 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.262 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.262 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.262 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.262 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.262 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.262 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.263 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_VerlsIFA1q6JkkLBYVtw0zq5tABLcwL", 2024-04-12T04:10:55.263 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.263 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.263 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.263 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj23", 2024-04-12T04:10:55.264 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.264 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.264 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.264 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 15 2024-04-12T04:10:55.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.264 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.264 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.264 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.264 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.264 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:10:55.264 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:05:01.070582Z", 2024-04-12T04:10:55.265 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.265 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.265 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.265 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.265 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.265 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.265 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.265 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.265 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_SeggoeXWqIPVHlkm1fzt_E5TUuvDcOK", 2024-04-12T04:10:55.266 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.266 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.266 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.266 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj29", 2024-04-12T04:10:55.266 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.266 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.266 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.266 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9 2024-04-12T04:10:55.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.266 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.267 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.267 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.267 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.267 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:10:55.267 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:06:49.121381Z", 2024-04-12T04:10:55.267 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.267 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.267 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.267 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.267 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.267 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.268 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.268 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.268 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_PJJGDsq_AGWLeUC6bjN8jM8ixP1ua67", 2024-04-12T04:10:55.268 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.268 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.268 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.268 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj31", 2024-04-12T04:10:55.269 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.269 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.269 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.269 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T04:10:55.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.269 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.269 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.269 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.269 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.269 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:10:55.269 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:08:08.174157Z", 2024-04-12T04:10:55.270 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.270 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.270 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.270 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.270 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.270 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.270 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.270 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.270 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_ARIiwh5iEWc6XbiYMHsX3bVHjBDI75B", 2024-04-12T04:10:55.270 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.271 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.271 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.271 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj33", 2024-04-12T04:10:55.271 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.271 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.271 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.271 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 8 2024-04-12T04:10:55.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.271 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.271 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.272 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.272 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.272 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T04:10:55.272 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:30.466236Z", 2024-04-12T04:10:55.272 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.272 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.272 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.272 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.272 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.272 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.272 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.273 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.273 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_oRm14uvS1fP7NwW5STOWwiq02UxiJl9", 2024-04-12T04:10:55.273 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.273 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.273 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.273 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj34", 2024-04-12T04:10:55.273 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.273 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.274 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.274 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T04:10:55.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.274 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.274 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.274 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.274 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.274 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T04:10:55.274 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:41.687199Z", 2024-04-12T04:10:55.274 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.274 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.275 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.275 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.275 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.275 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.275 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.275 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.275 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_VACmH3xyc-si07EOGQobwYLlKXyWU_6", 2024-04-12T04:10:55.275 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.275 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj37", 2024-04-12T04:10:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 18 2024-04-12T04:10:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.276 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.277 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.277 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.277 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.277 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:10:55.277 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:09:24.034250Z", 2024-04-12T04:10:55.277 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.277 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.277 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.277 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.277 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.278 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.278 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.278 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.278 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_fyfT0TO-g1ZniUkTAqFu8RhISeXE8P9", 2024-04-12T04:10:55.278 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.278 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.278 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.278 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj38", 2024-04-12T04:10:55.279 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.279 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.279 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.279 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T04:10:55.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.279 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.279 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.279 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.279 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.279 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:10:55.279 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:03:03.629194Z", 2024-04-12T04:10:55.279 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.280 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.280 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.280 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.280 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.280 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.280 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.280 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.280 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_QdgGZwvQ7oe54EA9zqqlEPTw2FYI9tO", 2024-04-12T04:10:55.280 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.280 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.280 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.281 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj44", 2024-04-12T04:10:55.281 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.281 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.281 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.281 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T04:10:55.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.281 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.281 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.281 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.281 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.282 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:10:55.282 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:30.522149Z", 2024-04-12T04:10:55.282 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.282 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.282 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.282 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.282 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.282 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.282 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.282 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.283 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_AOb7l4svCJeB4g6f0T451_aNgCD6HXN", 2024-04-12T04:10:55.283 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.283 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.283 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.283 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj46", 2024-04-12T04:10:55.283 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.283 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.283 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.284 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14 2024-04-12T04:10:55.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.284 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.284 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.284 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.284 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.284 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:10:55.284 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:05:01.082150Z", 2024-04-12T04:10:55.284 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.284 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.284 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.285 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.285 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.285 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.285 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.285 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.285 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_PeR6T7XfqzfcPxCH8mCUs0mWP99jo_S", 2024-04-12T04:10:55.285 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.285 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.285 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.286 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj51", 2024-04-12T04:10:55.286 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.286 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.286 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.286 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9 2024-04-12T04:10:55.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.286 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.286 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.286 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.286 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.287 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:10:55.287 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:41.699090Z", 2024-04-12T04:10:55.287 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.287 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.287 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.287 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.287 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.287 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.287 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.287 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.288 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_50SosPCcx____aNL6YXyKFMyILE86Pn", 2024-04-12T04:10:55.288 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.288 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.288 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.288 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj55", 2024-04-12T04:10:55.288 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.288 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.288 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.288 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T04:10:55.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.289 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.289 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.289 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.289 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.289 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:10:55.289 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:30.533855Z", 2024-04-12T04:10:55.289 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.289 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.289 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.289 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.290 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.290 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.290 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.290 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.290 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_IIMQb2O7MNjZizI4ePUFqk5nO5PBS9N", 2024-04-12T04:10:55.290 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.290 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.290 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.291 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj57", 2024-04-12T04:10:55.291 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.291 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.291 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.291 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T04:10:55.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.291 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.291 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.292 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.292 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.292 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:10:55.292 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:03:03.638996Z", 2024-04-12T04:10:55.292 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.292 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.292 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.292 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.292 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.292 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.292 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.293 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.293 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_GiLDMq7NHrWFkMJdDsdEbfuI2DUKACN", 2024-04-12T04:10:55.293 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.293 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.293 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.293 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj58", 2024-04-12T04:10:55.293 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.294 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.294 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.294 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14 2024-04-12T04:10:55.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.294 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.294 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.294 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.294 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.294 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T04:10:55.295 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:06:49.133205Z", 2024-04-12T04:10:55.295 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.295 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.295 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.295 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.295 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.295 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.295 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.295 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.296 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_LPROeQzxHOd-1ylMK6OG64FnI5bVsrE", 2024-04-12T04:10:55.296 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.296 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.296 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.296 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj62", 2024-04-12T04:10:55.296 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.296 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.297 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.297 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T04:10:55.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.297 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.297 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.297 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.297 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.297 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:10:55.297 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:08:08.186687Z", 2024-04-12T04:10:55.297 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.297 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.298 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.298 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.298 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.298 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.298 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.298 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.298 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_Yv_pkbX11NsUar2RgOuVvIXEx3iyufP", 2024-04-12T04:10:55.298 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.298 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.298 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.299 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj68", 2024-04-12T04:10:55.299 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.299 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.299 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.299 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9 2024-04-12T04:10:55.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.299 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.299 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.299 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.300 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.300 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:10:55.300 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:41.748656Z", 2024-04-12T04:10:55.300 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.300 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.300 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.300 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.300 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.300 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.300 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.300 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.301 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_ohY3NRhc8oYW2bGi9efUTVYpLSeZoYW", 2024-04-12T04:10:55.301 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.301 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.301 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.301 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj69", 2024-04-12T04:10:55.301 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.301 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.302 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.302 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T04:10:55.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.302 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.302 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.302 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.302 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.302 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T04:10:55.302 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:05:01.130607Z", 2024-04-12T04:10:55.302 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.303 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.303 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.303 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.303 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.303 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.303 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.303 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.303 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_2zxDRAhQk1afMoARJwBULw93MnljQG8", 2024-04-12T04:10:55.303 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.304 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.304 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.304 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.304 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj74", 2024-04-12T04:10:55.304 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.304 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.304 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.304 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 8 2024-04-12T04:10:55.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.304 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.305 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.305 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.305 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.305 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:10:55.305 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:09:24.047621Z", 2024-04-12T04:10:55.305 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.305 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.305 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.305 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.305 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.305 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.306 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.306 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.306 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_C1YL03ydi7jKZ4TpupjrL1opIK_H5DV", 2024-04-12T04:10:55.306 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.306 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.306 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.306 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.306 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj76", 2024-04-12T04:10:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 20 2024-04-12T04:10:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:10:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:03:03.689940Z", 2024-04-12T04:10:55.307 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_FFpjDyfn9Gsqpjhid7Z_gsiFX-EIZQ4", 2024-04-12T04:10:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.308 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.309 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.309 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj85", 2024-04-12T04:10:55.309 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.309 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.309 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.309 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 3 2024-04-12T04:10:55.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.309 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.310 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.310 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.310 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.310 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:10:55.310 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:41.759706Z", 2024-04-12T04:10:55.310 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.310 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.310 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.310 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.310 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.310 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.311 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.311 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.311 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.311 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_SQshkFKPhxncoX9Ff4Z8HCP3b3UipUS", 2024-04-12T04:10:55.311 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.311 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.312 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.312 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj87", 2024-04-12T04:10:55.312 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.312 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.312 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.312 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9 2024-04-12T04:10:55.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.312 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.312 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.314 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.314 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.314 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:10:55.314 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:06:49.144674Z", 2024-04-12T04:10:55.314 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.314 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.314 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.314 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.314 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.315 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.315 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.315 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.315 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_N4_Xl_RNvR-Jk_6D6IPbfslX8Xrztrp", 2024-04-12T04:10:55.315 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.315 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.315 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.315 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.315 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj92", 2024-04-12T04:10:55.316 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.316 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.316 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.316 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 8 2024-04-12T04:10:55.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.316 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.316 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.316 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.316 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.316 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:10:55.317 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:05:01.141706Z", 2024-04-12T04:10:55.317 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.317 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.317 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.317 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.317 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.317 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.317 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.317 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.317 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_bb5o9rNBA2wNJOrFMF_kojiJ1l0ZjMb", 2024-04-12T04:10:55.318 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.318 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.318 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.318 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj93", 2024-04-12T04:10:55.318 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.318 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.318 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.319 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2 2024-04-12T04:10:55.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.319 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.319 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.319 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.319 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.319 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:10:55.319 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:08:08.235885Z", 2024-04-12T04:10:55.319 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.319 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.319 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.320 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.320 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.320 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.320 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.320 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.320 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_BC0KSyMj6VcQW2-a7x_JO4_3rLvW839", 2024-04-12T04:10:55.320 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.320 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.320 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.320 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:10:55.321 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj95", 2024-04-12T04:10:55.321 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:10:55.321 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:10:55.321 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:10:55.321 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2 2024-04-12T04:10:55.321 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.321 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:10:55.321 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:10:55.321 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:10:55.321 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:10:55.321 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:10:55.322 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:03:03.702005Z", 2024-04-12T04:10:55.322 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:10:55.322 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:10:55.322 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:10:55.322 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:10:55.322 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:10:55.322 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:10:55.322 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:10:55.322 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:10:55.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:10:55.323 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_ulxasFhXvcW_OhfbhO1eJW7Yw0wcBZm", 2024-04-12T04:10:55.323 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:10:55.323 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:10:55.323 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:10:55.323 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:10:55.323 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:10:55.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:55.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:55.324 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj11 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:55.385 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:55.385 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:55.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:55.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:55.493 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj111 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:55.552 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:55.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:55.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:55.659 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj115 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:55.717 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:55.717 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:55.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:55.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:55.822 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj116 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:55.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:55.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:55.983 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj124 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:56.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:56.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:56.148 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj145 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:56.208 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:56.208 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:56.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:56.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:56.314 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj148 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:56.330 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T04:10:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:56.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:56.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:56.479 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj155 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:56.536 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:56.536 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:56.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:56.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:56.641 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj17 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:56.698 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:56.699 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:56.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:56.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:56.807 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj185 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:56.865 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:56.865 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:56.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:56.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:56.972 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj19 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:57.030 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:57.030 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:57.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:57.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:57.136 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj22 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:57.194 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:57.194 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:57.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:57.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:57.303 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj23 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:57.362 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:57.362 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:57.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:57.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:57.469 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj29 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:57.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:57.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:57.634 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj31 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:57.692 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:57.692 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:57.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:57.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:57.803 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj33 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:57.860 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:57.860 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:57.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:57.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:57.967 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj34 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:58.024 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:58.024 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:58.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:58.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:58.137 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj37 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:58.195 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:58.195 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:58.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:58.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:58.307 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj38 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:58.365 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:58.365 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:58.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:58.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:58.471 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj44 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:58.529 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:58.529 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:58.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:58.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:58.635 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj46 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:58.696 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:58.696 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:58.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:58.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:58.802 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj51 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:58.859 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:58.859 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:58.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:58.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:58.966 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj55 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:59.023 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:59.024 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:59.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:59.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:59.132 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:59.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:59.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:59.304 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj58 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:59.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:59.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:59.470 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj62 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:59.528 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:59.528 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:59.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:59.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:59.635 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj68 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:59.692 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:59.692 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:59.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:59.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:59.801 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj69 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:10:59.858 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:10:59.859 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:10:59.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:59.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:10:59.965 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:11:00.023 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:00.023 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:00.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:00.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:00.129 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:11:00.186 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:00.186 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:00.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:00.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:00.298 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj85 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:11:00.354 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:00.355 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:00.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:00.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:00.464 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj87 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:11:00.522 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:00.523 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:00.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:00.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:00.628 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj92 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:11:00.685 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:00.686 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:00.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:00.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:00.795 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj93 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:11:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:00.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:00.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:00.958 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --object obj95 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:11:01.016 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:01.016 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:01.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:01.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:01.124 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog trim --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:11:01.183 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:01.183 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:01.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:01.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:01.245 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:11:01.302 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:01.302 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:01.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:01.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:11:01.356 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:11:01.415 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.515+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2024-04-12T04:11:01.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.515+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: waiting on bucket lease 2024-04-12T04:11:01.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.522+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: status after acquiring the lock is: init 2024-04-12T04:11:01.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.538+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: listing bucket for full sync 2024-04-12T04:11:01.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.584+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj11 2024-04-12T04:11:01.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.584+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj11 op=0 op_state=1 2024-04-12T04:11:01.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.584+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj11[0] 2024-04-12T04:11:01.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.584+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj111 2024-04-12T04:11:01.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.584+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj111 op=0 op_state=1 2024-04-12T04:11:01.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.584+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2024-04-12T04:11:01.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.584+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.584+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj111 2024-04-12T04:11:01.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.584+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj115 2024-04-12T04:11:01.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.584+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj115 op=0 op_state=1 2024-04-12T04:11:01.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.584+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj115[0] 2024-04-12T04:11:01.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.584+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj116 2024-04-12T04:11:01.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.584+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj116 op=0 op_state=1 2024-04-12T04:11:01.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.584+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj116[0] 2024-04-12T04:11:01.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj124 2024-04-12T04:11:01.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj124 op=0 op_state=1 2024-04-12T04:11:01.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj124[0] 2024-04-12T04:11:01.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj145 2024-04-12T04:11:01.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj145 op=0 op_state=1 2024-04-12T04:11:01.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj145[0] 2024-04-12T04:11:01.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj148 2024-04-12T04:11:01.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj148 op=0 op_state=1 2024-04-12T04:11:01.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj148[0] 2024-04-12T04:11:01.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj155 2024-04-12T04:11:01.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj155 op=0 op_state=1 2024-04-12T04:11:01.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2024-04-12T04:11:01.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj155 2024-04-12T04:11:01.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj17 2024-04-12T04:11:01.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj17 op=0 op_state=1 2024-04-12T04:11:01.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj17[0] 2024-04-12T04:11:01.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj185 2024-04-12T04:11:01.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj185 op=0 op_state=1 2024-04-12T04:11:01.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj185[0] 2024-04-12T04:11:01.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj19 2024-04-12T04:11:01.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj19 op=0 op_state=1 2024-04-12T04:11:01.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj19[0] 2024-04-12T04:11:01.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj22 2024-04-12T04:11:01.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj22 op=0 op_state=1 2024-04-12T04:11:01.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: : injecting data sync error on key=obj22 2024-04-12T04:11:01.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj22 2024-04-12T04:11:01.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj23 2024-04-12T04:11:01.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj23 op=0 op_state=1 2024-04-12T04:11:01.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj23[0] 2024-04-12T04:11:01.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj29 2024-04-12T04:11:01.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj29 op=0 op_state=1 2024-04-12T04:11:01.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2024-04-12T04:11:01.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj29 2024-04-12T04:11:01.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj31 2024-04-12T04:11:01.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj31 op=0 op_state=1 2024-04-12T04:11:01.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj31[0] 2024-04-12T04:11:01.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj33 2024-04-12T04:11:01.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj33 op=0 op_state=1 2024-04-12T04:11:01.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj33[0] 2024-04-12T04:11:01.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj34 2024-04-12T04:11:01.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj34 op=0 op_state=1 2024-04-12T04:11:01.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: : injecting data sync error on key=obj34 2024-04-12T04:11:01.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj34 2024-04-12T04:11:01.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj37 2024-04-12T04:11:01.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj37 op=0 op_state=1 2024-04-12T04:11:01.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj37[0] 2024-04-12T04:11:01.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj38 2024-04-12T04:11:01.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj38 op=0 op_state=1 2024-04-12T04:11:01.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj38[0] 2024-04-12T04:11:01.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj44 2024-04-12T04:11:01.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj44 op=0 op_state=1 2024-04-12T04:11:01.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj44[0] 2024-04-12T04:11:01.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.585+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj46 2024-04-12T04:11:01.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.586+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj46 op=0 op_state=1 2024-04-12T04:11:01.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.586+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj46[0] 2024-04-12T04:11:01.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.591+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.591+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj51 2024-04-12T04:11:01.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.591+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj51 op=0 op_state=1 2024-04-12T04:11:01.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.591+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj51[0] 2024-04-12T04:11:01.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.591+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.591+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj55 2024-04-12T04:11:01.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.591+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj55 op=0 op_state=1 2024-04-12T04:11:01.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.591+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj55[0] 2024-04-12T04:11:01.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.591+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.591+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj57 2024-04-12T04:11:01.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.592+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj57 op=0 op_state=1 2024-04-12T04:11:01.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.592+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj57[0] 2024-04-12T04:11:01.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.592+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.592+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj58 2024-04-12T04:11:01.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.592+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj58 op=0 op_state=1 2024-04-12T04:11:01.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.592+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj58[0] 2024-04-12T04:11:01.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.593+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.593+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj62 2024-04-12T04:11:01.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.593+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj62 op=0 op_state=1 2024-04-12T04:11:01.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.593+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj62[0] 2024-04-12T04:11:01.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.613+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: success 2024-04-12T04:11:01.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.613+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj68 2024-04-12T04:11:01.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.613+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj68 op=0 op_state=1 2024-04-12T04:11:01.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.613+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj68[0] 2024-04-12T04:11:01.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.613+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: success 2024-04-12T04:11:01.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.613+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj69 2024-04-12T04:11:01.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.613+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj69 op=0 op_state=1 2024-04-12T04:11:01.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.613+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj69[0] 2024-04-12T04:11:01.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.613+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: success 2024-04-12T04:11:01.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.613+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: success 2024-04-12T04:11:01.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj74 2024-04-12T04:11:01.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj74 op=0 op_state=1 2024-04-12T04:11:01.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj74[0] 2024-04-12T04:11:01.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj76 2024-04-12T04:11:01.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj76 op=0 op_state=1 2024-04-12T04:11:01.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj76[0] 2024-04-12T04:11:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: success 2024-04-12T04:11:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: success 2024-04-12T04:11:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj85 2024-04-12T04:11:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj85 op=0 op_state=1 2024-04-12T04:11:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj85[0] 2024-04-12T04:11:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: success 2024-04-12T04:11:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj87 2024-04-12T04:11:01.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj87 op=0 op_state=1 2024-04-12T04:11:01.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj87[0] 2024-04-12T04:11:01.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj92 2024-04-12T04:11:01.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj92 op=0 op_state=1 2024-04-12T04:11:01.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: success 2024-04-12T04:11:01.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj92[0] 2024-04-12T04:11:01.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj93 2024-04-12T04:11:01.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj93 op=0 op_state=1 2024-04-12T04:11:01.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.614+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj93[0] 2024-04-12T04:11:01.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.616+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: success 2024-04-12T04:11:01.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.616+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj95 2024-04-12T04:11:01.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.616+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj95 op=0 op_state=1 2024-04-12T04:11:01.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.616+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj95[0] 2024-04-12T04:11:01.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.618+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: success 2024-04-12T04:11:01.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.619+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: success 2024-04-12T04:11:01.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.619+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: success 2024-04-12T04:11:01.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.619+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: success 2024-04-12T04:11:01.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.624+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: success 2024-04-12T04:11:01.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.624+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: success 2024-04-12T04:11:01.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.624+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: success 2024-04-12T04:11:01.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.624+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: success 2024-04-12T04:11:01.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.643+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: success 2024-04-12T04:11:01.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.646+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: success 2024-04-12T04:11:01.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.648+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: success 2024-04-12T04:11:01.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.648+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: success 2024-04-12T04:11:01.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.648+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: success 2024-04-12T04:11:01.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.648+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: success 2024-04-12T04:11:01.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.651+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: success 2024-04-12T04:11:01.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.651+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: success 2024-04-12T04:11:01.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.663+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: success 2024-04-12T04:11:01.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.664+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: success 2024-04-12T04:11:01.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.664+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: success 2024-04-12T04:11:01.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.664+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: success 2024-04-12T04:11:01.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.665+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: success 2024-04-12T04:11:01.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.665+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: updating marker oid=bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24 marker=obj11 2024-04-12T04:11:01.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.667+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: backing out with sync_status=-5 2024-04-12T04:11:01.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.667+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: ERROR: full sync failed. error: -5 2024-04-12T04:11:01.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.669+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-12T04:11:01.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.669+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: waiting on bucket lease 2024-04-12T04:11:01.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.673+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: status after acquiring the lock is: full 2024-04-12T04:11:01.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.687+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: listing bucket for full sync 2024-04-12T04:11:01.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj11 2024-04-12T04:11:01.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj11 op=0 op_state=1 2024-04-12T04:11:01.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj11[0] 2024-04-12T04:11:01.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj111 2024-04-12T04:11:01.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj111 op=0 op_state=1 2024-04-12T04:11:01.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj111[0] 2024-04-12T04:11:01.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj115 2024-04-12T04:11:01.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj115 op=0 op_state=1 2024-04-12T04:11:01.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj115[0] 2024-04-12T04:11:01.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj116 2024-04-12T04:11:01.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj116 op=0 op_state=1 2024-04-12T04:11:01.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj116[0] 2024-04-12T04:11:01.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj124 2024-04-12T04:11:01.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj124 op=0 op_state=1 2024-04-12T04:11:01.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj124[0] 2024-04-12T04:11:01.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj145 2024-04-12T04:11:01.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj145 op=0 op_state=1 2024-04-12T04:11:01.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj145[0] 2024-04-12T04:11:01.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj148 2024-04-12T04:11:01.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj148 op=0 op_state=1 2024-04-12T04:11:01.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj148[0] 2024-04-12T04:11:01.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj155 2024-04-12T04:11:01.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj155 op=0 op_state=1 2024-04-12T04:11:01.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj155[0] 2024-04-12T04:11:01.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj17 2024-04-12T04:11:01.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj17 op=0 op_state=1 2024-04-12T04:11:01.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj17[0] 2024-04-12T04:11:01.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj185 2024-04-12T04:11:01.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj185 op=0 op_state=1 2024-04-12T04:11:01.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj185[0] 2024-04-12T04:11:01.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj19 2024-04-12T04:11:01.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj19 op=0 op_state=1 2024-04-12T04:11:01.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj19[0] 2024-04-12T04:11:01.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj22 2024-04-12T04:11:01.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj22 op=0 op_state=1 2024-04-12T04:11:01.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj22[0] 2024-04-12T04:11:01.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj23 2024-04-12T04:11:01.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj23 op=0 op_state=1 2024-04-12T04:11:01.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj23[0] 2024-04-12T04:11:01.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj29 2024-04-12T04:11:01.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj29 op=0 op_state=1 2024-04-12T04:11:01.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj29[0] 2024-04-12T04:11:01.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj31 2024-04-12T04:11:01.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj31 op=0 op_state=1 2024-04-12T04:11:01.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj31[0] 2024-04-12T04:11:01.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj33 2024-04-12T04:11:01.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj33 op=0 op_state=1 2024-04-12T04:11:01.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj33[0] 2024-04-12T04:11:01.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj34 2024-04-12T04:11:01.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj34 op=0 op_state=1 2024-04-12T04:11:01.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj34[0] 2024-04-12T04:11:01.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj37 2024-04-12T04:11:01.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj37 op=0 op_state=1 2024-04-12T04:11:01.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj37[0] 2024-04-12T04:11:01.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj38 2024-04-12T04:11:01.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj38 op=0 op_state=1 2024-04-12T04:11:01.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj38[0] 2024-04-12T04:11:01.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj44 2024-04-12T04:11:01.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj44 op=0 op_state=1 2024-04-12T04:11:01.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.734+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj44[0] 2024-04-12T04:11:01.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.735+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj46 2024-04-12T04:11:01.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.735+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj46 op=0 op_state=1 2024-04-12T04:11:01.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.735+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj46[0] 2024-04-12T04:11:01.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: success 2024-04-12T04:11:01.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: success 2024-04-12T04:11:01.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: success 2024-04-12T04:11:01.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: success 2024-04-12T04:11:01.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: success 2024-04-12T04:11:01.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: success 2024-04-12T04:11:01.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj51 2024-04-12T04:11:01.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj51 op=0 op_state=1 2024-04-12T04:11:01.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj51[0] 2024-04-12T04:11:01.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj55 2024-04-12T04:11:01.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj55 op=0 op_state=1 2024-04-12T04:11:01.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj55[0] 2024-04-12T04:11:01.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj57 2024-04-12T04:11:01.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj57 op=0 op_state=1 2024-04-12T04:11:01.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj57[0] 2024-04-12T04:11:01.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj58 2024-04-12T04:11:01.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj58 op=0 op_state=1 2024-04-12T04:11:01.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj58[0] 2024-04-12T04:11:01.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj62 2024-04-12T04:11:01.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj62 op=0 op_state=1 2024-04-12T04:11:01.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj62[0] 2024-04-12T04:11:01.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj68 2024-04-12T04:11:01.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj68 op=0 op_state=1 2024-04-12T04:11:01.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: : injecting data sync error on key=obj68 2024-04-12T04:11:01.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj68 2024-04-12T04:11:01.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: success 2024-04-12T04:11:01.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj69 2024-04-12T04:11:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj69 op=0 op_state=1 2024-04-12T04:11:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.741+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj69[0] 2024-04-12T04:11:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.742+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: success 2024-04-12T04:11:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.742+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj74 2024-04-12T04:11:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.742+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj74 op=0 op_state=1 2024-04-12T04:11:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.742+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj74[0] 2024-04-12T04:11:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.742+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: success 2024-04-12T04:11:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.742+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj76 2024-04-12T04:11:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.742+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj76 op=0 op_state=1 2024-04-12T04:11:01.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.742+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: : injecting data sync error on key=obj76 2024-04-12T04:11:01.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.742+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.742+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj76 2024-04-12T04:11:01.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.742+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: success 2024-04-12T04:11:01.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.742+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj85 2024-04-12T04:11:01.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.742+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj85 op=0 op_state=1 2024-04-12T04:11:01.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.742+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj85[0] 2024-04-12T04:11:01.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.743+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: success 2024-04-12T04:11:01.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.743+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj87 2024-04-12T04:11:01.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.743+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj87 op=0 op_state=1 2024-04-12T04:11:01.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.743+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2024-04-12T04:11:01.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.743+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.743+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj87 2024-04-12T04:11:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.743+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: success 2024-04-12T04:11:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.743+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj92 2024-04-12T04:11:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.743+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj92 op=0 op_state=1 2024-04-12T04:11:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.743+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj92[0] 2024-04-12T04:11:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.744+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: success 2024-04-12T04:11:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.744+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj93 2024-04-12T04:11:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.744+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj93 op=0 op_state=1 2024-04-12T04:11:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.744+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj93[0] 2024-04-12T04:11:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.746+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: success 2024-04-12T04:11:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.746+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: success 2024-04-12T04:11:01.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.746+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj95 2024-04-12T04:11:01.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.746+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj95 op=0 op_state=1 2024-04-12T04:11:01.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.746+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj95[0] 2024-04-12T04:11:01.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.746+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: success 2024-04-12T04:11:01.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.747+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: success 2024-04-12T04:11:01.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.748+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: success 2024-04-12T04:11:01.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.748+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.748+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: success 2024-04-12T04:11:01.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.748+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: success 2024-04-12T04:11:01.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.749+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: success 2024-04-12T04:11:01.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.749+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: success 2024-04-12T04:11:01.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.749+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.750+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.750+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: success 2024-04-12T04:11:01.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.750+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: success 2024-04-12T04:11:01.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.750+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: success 2024-04-12T04:11:01.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.750+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: success 2024-04-12T04:11:01.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.752+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: success 2024-04-12T04:11:01.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.752+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: success 2024-04-12T04:11:01.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.753+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: success 2024-04-12T04:11:01.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.753+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: success 2024-04-12T04:11:01.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.753+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: updating marker oid=bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24 marker=obj19 2024-04-12T04:11:01.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.753+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: success 2024-04-12T04:11:01.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.787+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: success 2024-04-12T04:11:01.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.787+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: updating marker oid=bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24 marker=obj62 2024-04-12T04:11:01.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.788+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: backing out with sync_status=-5 2024-04-12T04:11:01.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.788+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: ERROR: full sync failed. error: -5 2024-04-12T04:11:01.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.790+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-12T04:11:01.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.790+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: waiting on bucket lease 2024-04-12T04:11:01.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.795+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: status after acquiring the lock is: full 2024-04-12T04:11:01.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.810+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: listing bucket for full sync 2024-04-12T04:11:01.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj11 2024-04-12T04:11:01.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj11 op=0 op_state=1 2024-04-12T04:11:01.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj11[0] 2024-04-12T04:11:01.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj111 2024-04-12T04:11:01.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj111 op=0 op_state=1 2024-04-12T04:11:01.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj111[0] 2024-04-12T04:11:01.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj115 2024-04-12T04:11:01.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj115 op=0 op_state=1 2024-04-12T04:11:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj115[0] 2024-04-12T04:11:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj116 2024-04-12T04:11:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj116 op=0 op_state=1 2024-04-12T04:11:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: : injecting data sync error on key=obj116 2024-04-12T04:11:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj116 2024-04-12T04:11:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj124 2024-04-12T04:11:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj124 op=0 op_state=1 2024-04-12T04:11:01.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj124[0] 2024-04-12T04:11:01.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj145 2024-04-12T04:11:01.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj145 op=0 op_state=1 2024-04-12T04:11:01.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj145[0] 2024-04-12T04:11:01.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.857+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj148 2024-04-12T04:11:01.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj148 op=0 op_state=1 2024-04-12T04:11:01.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj148[0] 2024-04-12T04:11:01.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj155 2024-04-12T04:11:01.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj155 op=0 op_state=1 2024-04-12T04:11:01.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2024-04-12T04:11:01.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj155 2024-04-12T04:11:01.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj17 2024-04-12T04:11:01.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj17 op=0 op_state=1 2024-04-12T04:11:01.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj17[0] 2024-04-12T04:11:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj185 2024-04-12T04:11:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj185 op=0 op_state=1 2024-04-12T04:11:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj185[0] 2024-04-12T04:11:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj19 2024-04-12T04:11:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj19 op=0 op_state=1 2024-04-12T04:11:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj19[0] 2024-04-12T04:11:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj22 2024-04-12T04:11:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj22 op=0 op_state=1 2024-04-12T04:11:01.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj22[0] 2024-04-12T04:11:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj23 2024-04-12T04:11:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj23 op=0 op_state=1 2024-04-12T04:11:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj23[0] 2024-04-12T04:11:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj29 2024-04-12T04:11:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj29 op=0 op_state=1 2024-04-12T04:11:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj29[0] 2024-04-12T04:11:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj31 2024-04-12T04:11:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj31 op=0 op_state=1 2024-04-12T04:11:01.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj31[0] 2024-04-12T04:11:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj33 2024-04-12T04:11:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj33 op=0 op_state=1 2024-04-12T04:11:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj33[0] 2024-04-12T04:11:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj34 2024-04-12T04:11:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj34 op=0 op_state=1 2024-04-12T04:11:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj34[0] 2024-04-12T04:11:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj37 2024-04-12T04:11:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj37 op=0 op_state=1 2024-04-12T04:11:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj37[0] 2024-04-12T04:11:01.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj38 2024-04-12T04:11:01.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj38 op=0 op_state=1 2024-04-12T04:11:01.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj38[0] 2024-04-12T04:11:01.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj44 2024-04-12T04:11:01.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj44 op=0 op_state=1 2024-04-12T04:11:01.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj44[0] 2024-04-12T04:11:01.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj46 2024-04-12T04:11:01.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj46 op=0 op_state=1 2024-04-12T04:11:01.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.858+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj46[0] 2024-04-12T04:11:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.863+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: success 2024-04-12T04:11:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.863+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: success 2024-04-12T04:11:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.863+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: success 2024-04-12T04:11:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.863+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj51 2024-04-12T04:11:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.863+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj51 op=0 op_state=1 2024-04-12T04:11:01.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.863+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj51[0] 2024-04-12T04:11:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.863+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj55 2024-04-12T04:11:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.863+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj55 op=0 op_state=1 2024-04-12T04:11:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.863+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj55[0] 2024-04-12T04:11:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.863+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj57 2024-04-12T04:11:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.863+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj57 op=0 op_state=1 2024-04-12T04:11:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.863+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj57[0] 2024-04-12T04:11:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.864+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: success 2024-04-12T04:11:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.864+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj58 2024-04-12T04:11:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.864+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj58 op=0 op_state=1 2024-04-12T04:11:01.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.864+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj58[0] 2024-04-12T04:11:01.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj62 2024-04-12T04:11:01.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj62 op=0 op_state=1 2024-04-12T04:11:01.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: : injecting data sync error on key=obj62 2024-04-12T04:11:01.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj62 2024-04-12T04:11:01.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: success 2024-04-12T04:11:01.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: success 2024-04-12T04:11:01.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: success 2024-04-12T04:11:01.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: success 2024-04-12T04:11:01.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: success 2024-04-12T04:11:01.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: success 2024-04-12T04:11:01.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj68 2024-04-12T04:11:01.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj68 op=0 op_state=1 2024-04-12T04:11:01.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj68[0] 2024-04-12T04:11:01.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: success 2024-04-12T04:11:01.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: updating marker oid=bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24 marker=obj11 2024-04-12T04:11:01.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj69 2024-04-12T04:11:01.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj69 op=0 op_state=1 2024-04-12T04:11:01.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj69[0] 2024-04-12T04:11:01.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj74 2024-04-12T04:11:01.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj74 op=0 op_state=1 2024-04-12T04:11:01.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj74[0] 2024-04-12T04:11:01.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj76 2024-04-12T04:11:01.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj76 op=0 op_state=1 2024-04-12T04:11:01.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj76[0] 2024-04-12T04:11:01.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: success 2024-04-12T04:11:01.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj85 2024-04-12T04:11:01.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj85 op=0 op_state=1 2024-04-12T04:11:01.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: success 2024-04-12T04:11:01.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.865+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2024-04-12T04:11:01.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj85 2024-04-12T04:11:01.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj87 2024-04-12T04:11:01.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj87 op=0 op_state=1 2024-04-12T04:11:01.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2024-04-12T04:11:01.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj87 2024-04-12T04:11:01.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj92 2024-04-12T04:11:01.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj92 op=0 op_state=1 2024-04-12T04:11:01.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj92[0] 2024-04-12T04:11:01.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj93 2024-04-12T04:11:01.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj93 op=0 op_state=1 2024-04-12T04:11:01.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: : injecting data sync error on key=obj93 2024-04-12T04:11:01.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj93 2024-04-12T04:11:01.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj95 2024-04-12T04:11:01.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj95 op=0 op_state=1 2024-04-12T04:11:01.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj95[0] 2024-04-12T04:11:01.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: success 2024-04-12T04:11:01.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.866+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: success 2024-04-12T04:11:01.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.869+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: success 2024-04-12T04:11:01.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.869+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: success 2024-04-12T04:11:01.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.869+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: success 2024-04-12T04:11:01.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.870+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: success 2024-04-12T04:11:01.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.870+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: success 2024-04-12T04:11:01.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.871+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: success 2024-04-12T04:11:01.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.871+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: success 2024-04-12T04:11:01.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.871+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: success 2024-04-12T04:11:01.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.871+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.871+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: success 2024-04-12T04:11:01.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.871+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: success 2024-04-12T04:11:01.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.871+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: success 2024-04-12T04:11:01.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.871+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: success 2024-04-12T04:11:01.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.871+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.871+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.871+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.872+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: success 2024-04-12T04:11:01.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.872+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: success 2024-04-12T04:11:01.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.872+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: updating marker oid=bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24 marker=obj115 2024-04-12T04:11:01.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.874+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: backing out with sync_status=-5 2024-04-12T04:11:01.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.874+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: ERROR: full sync failed. error: -5 2024-04-12T04:11:01.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.875+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-12T04:11:01.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.875+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: waiting on bucket lease 2024-04-12T04:11:01.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.921+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: status after acquiring the lock is: full 2024-04-12T04:11:01.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.936+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: listing bucket for full sync 2024-04-12T04:11:01.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.982+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj11 2024-04-12T04:11:01.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.982+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj11 op=0 op_state=1 2024-04-12T04:11:01.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.982+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj11[0] 2024-04-12T04:11:01.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj111 2024-04-12T04:11:01.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj111 op=0 op_state=1 2024-04-12T04:11:01.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2024-04-12T04:11:01.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj111 2024-04-12T04:11:01.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj115 2024-04-12T04:11:01.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj115 op=0 op_state=1 2024-04-12T04:11:01.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj115[0] 2024-04-12T04:11:01.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj116 2024-04-12T04:11:01.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj116 op=0 op_state=1 2024-04-12T04:11:01.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj116[0] 2024-04-12T04:11:01.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj124 2024-04-12T04:11:01.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj124 op=0 op_state=1 2024-04-12T04:11:01.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: : injecting data sync error on key=obj124 2024-04-12T04:11:01.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj124 2024-04-12T04:11:01.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj145 2024-04-12T04:11:01.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj145 op=0 op_state=1 2024-04-12T04:11:01.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj145[0] 2024-04-12T04:11:01.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj148 2024-04-12T04:11:01.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj148 op=0 op_state=1 2024-04-12T04:11:01.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: : injecting data sync error on key=obj148 2024-04-12T04:11:01.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj148 2024-04-12T04:11:01.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj155 2024-04-12T04:11:01.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj155 op=0 op_state=1 2024-04-12T04:11:01.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj155[0] 2024-04-12T04:11:01.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj17 2024-04-12T04:11:01.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj17 op=0 op_state=1 2024-04-12T04:11:01.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: : injecting data sync error on key=obj17 2024-04-12T04:11:01.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj17 2024-04-12T04:11:01.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj185 2024-04-12T04:11:01.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj185 op=0 op_state=1 2024-04-12T04:11:01.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj185[0] 2024-04-12T04:11:01.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj19 2024-04-12T04:11:01.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj19 op=0 op_state=1 2024-04-12T04:11:01.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj19[0] 2024-04-12T04:11:01.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj22 2024-04-12T04:11:01.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj22 op=0 op_state=1 2024-04-12T04:11:01.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj22[0] 2024-04-12T04:11:01.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj23 2024-04-12T04:11:01.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj23 op=0 op_state=1 2024-04-12T04:11:01.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj23[0] 2024-04-12T04:11:01.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj29 2024-04-12T04:11:01.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj29 op=0 op_state=1 2024-04-12T04:11:01.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj29[0] 2024-04-12T04:11:01.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj31 2024-04-12T04:11:01.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj31 op=0 op_state=1 2024-04-12T04:11:01.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj31[0] 2024-04-12T04:11:01.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj33 2024-04-12T04:11:01.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj33 op=0 op_state=1 2024-04-12T04:11:01.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj33[0] 2024-04-12T04:11:01.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj34 2024-04-12T04:11:01.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj34 op=0 op_state=1 2024-04-12T04:11:01.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj34[0] 2024-04-12T04:11:01.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj37 2024-04-12T04:11:01.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj37 op=0 op_state=1 2024-04-12T04:11:01.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.983+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj37[0] 2024-04-12T04:11:01.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.984+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj38 2024-04-12T04:11:01.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.984+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj38 op=0 op_state=1 2024-04-12T04:11:01.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.984+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj38[0] 2024-04-12T04:11:01.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.984+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj44 2024-04-12T04:11:01.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.984+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj44 op=0 op_state=1 2024-04-12T04:11:01.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.984+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj44[0] 2024-04-12T04:11:01.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.984+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj46 2024-04-12T04:11:01.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.984+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj46 op=0 op_state=1 2024-04-12T04:11:01.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.984+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj46[0] 2024-04-12T04:11:01.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.988+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: success 2024-04-12T04:11:01.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.988+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: success 2024-04-12T04:11:01.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.988+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: success 2024-04-12T04:11:01.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.988+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj51 2024-04-12T04:11:01.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.988+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj51 op=0 op_state=1 2024-04-12T04:11:01.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.988+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj51[0] 2024-04-12T04:11:01.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.988+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj55 2024-04-12T04:11:01.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.988+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj55 op=0 op_state=1 2024-04-12T04:11:01.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.988+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: : injecting data sync error on key=obj55 2024-04-12T04:11:01.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.988+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.988+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj55 2024-04-12T04:11:01.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.988+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj57 2024-04-12T04:11:01.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.988+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj57 op=0 op_state=1 2024-04-12T04:11:01.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.988+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: : injecting data sync error on key=obj57 2024-04-12T04:11:01.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.988+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:01.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.988+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj57 2024-04-12T04:11:01.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.990+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: success 2024-04-12T04:11:01.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.990+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: success 2024-04-12T04:11:01.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.990+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj58 2024-04-12T04:11:01.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.990+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj58 op=0 op_state=1 2024-04-12T04:11:01.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.990+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj58[0] 2024-04-12T04:11:01.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.990+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj62 2024-04-12T04:11:01.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.990+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj62 op=0 op_state=1 2024-04-12T04:11:01.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.990+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: success 2024-04-12T04:11:01.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.991+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj62[0] 2024-04-12T04:11:01.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.991+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: success 2024-04-12T04:11:01.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.991+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: success 2024-04-12T04:11:01.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.991+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.991+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj68 2024-04-12T04:11:01.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.991+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj68 op=0 op_state=1 2024-04-12T04:11:01.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.991+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj68[0] 2024-04-12T04:11:01.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.991+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj69 2024-04-12T04:11:01.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.991+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj69 op=0 op_state=1 2024-04-12T04:11:01.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.991+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj69[0] 2024-04-12T04:11:01.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.991+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj74 2024-04-12T04:11:01.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.991+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj74 op=0 op_state=1 2024-04-12T04:11:01.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.991+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj74[0] 2024-04-12T04:11:01.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.991+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj76 2024-04-12T04:11:01.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.991+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj76 op=0 op_state=1 2024-04-12T04:11:01.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.991+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj76[0] 2024-04-12T04:11:01.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj85 2024-04-12T04:11:01.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj85 op=0 op_state=1 2024-04-12T04:11:01.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: success 2024-04-12T04:11:01.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: success 2024-04-12T04:11:01.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: success 2024-04-12T04:11:01.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: success 2024-04-12T04:11:01.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: success 2024-04-12T04:11:01.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj85[0] 2024-04-12T04:11:01.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj87 2024-04-12T04:11:01.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj87 op=0 op_state=1 2024-04-12T04:11:01.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj87[0] 2024-04-12T04:11:01.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj92 2024-04-12T04:11:01.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj92 op=0 op_state=1 2024-04-12T04:11:01.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj92[0] 2024-04-12T04:11:01.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj93 2024-04-12T04:11:01.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj93 op=0 op_state=1 2024-04-12T04:11:01.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj93[0] 2024-04-12T04:11:01.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj95 2024-04-12T04:11:01.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj95 op=0 op_state=1 2024-04-12T04:11:01.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj95[0] 2024-04-12T04:11:01.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.992+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: success 2024-04-12T04:11:01.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.993+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.993+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.994+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: success 2024-04-12T04:11:01.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.994+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.994+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: success 2024-04-12T04:11:01.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.994+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: success 2024-04-12T04:11:01.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.994+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:01.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.996+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: success 2024-04-12T04:11:01.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.996+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: success 2024-04-12T04:11:01.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.996+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: success 2024-04-12T04:11:01.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.996+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: success 2024-04-12T04:11:01.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.996+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: success 2024-04-12T04:11:01.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.997+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: success 2024-04-12T04:11:01.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.997+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: success 2024-04-12T04:11:01.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.997+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: success 2024-04-12T04:11:01.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.997+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: success 2024-04-12T04:11:01.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.997+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: success 2024-04-12T04:11:01.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.997+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: success 2024-04-12T04:11:01.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.997+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: success 2024-04-12T04:11:01.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.997+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: updating marker oid=bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24 marker=obj11 2024-04-12T04:11:01.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.999+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: backing out with sync_status=-5 2024-04-12T04:11:01.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:01.999+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: ERROR: full sync failed. error: -5 2024-04-12T04:11:01.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.001+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-12T04:11:01.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.001+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: waiting on bucket lease 2024-04-12T04:11:02.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.046+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: status after acquiring the lock is: full 2024-04-12T04:11:02.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.061+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: listing bucket for full sync 2024-04-12T04:11:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj11 2024-04-12T04:11:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj11 op=0 op_state=1 2024-04-12T04:11:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj11[0] 2024-04-12T04:11:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj111 2024-04-12T04:11:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj111 op=0 op_state=1 2024-04-12T04:11:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj111[0] 2024-04-12T04:11:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj115 2024-04-12T04:11:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj115 op=0 op_state=1 2024-04-12T04:11:02.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj115[0] 2024-04-12T04:11:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj116 2024-04-12T04:11:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj116 op=0 op_state=1 2024-04-12T04:11:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj116[0] 2024-04-12T04:11:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj124 2024-04-12T04:11:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj124 op=0 op_state=1 2024-04-12T04:11:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj124[0] 2024-04-12T04:11:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj145 2024-04-12T04:11:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj145 op=0 op_state=1 2024-04-12T04:11:02.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: : injecting data sync error on key=obj145 2024-04-12T04:11:02.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:02.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj145 2024-04-12T04:11:02.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj148 2024-04-12T04:11:02.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj148 op=0 op_state=1 2024-04-12T04:11:02.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj148[0] 2024-04-12T04:11:02.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj155 2024-04-12T04:11:02.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj155 op=0 op_state=1 2024-04-12T04:11:02.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj155[0] 2024-04-12T04:11:02.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj17 2024-04-12T04:11:02.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj17 op=0 op_state=1 2024-04-12T04:11:02.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj17[0] 2024-04-12T04:11:02.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj185 2024-04-12T04:11:02.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj185 op=0 op_state=1 2024-04-12T04:11:02.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj185[0] 2024-04-12T04:11:02.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj19 2024-04-12T04:11:02.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj19 op=0 op_state=1 2024-04-12T04:11:02.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj19[0] 2024-04-12T04:11:02.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj22 2024-04-12T04:11:02.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj22 op=0 op_state=1 2024-04-12T04:11:02.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj22[0] 2024-04-12T04:11:02.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj23 2024-04-12T04:11:02.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj23 op=0 op_state=1 2024-04-12T04:11:02.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj23[0] 2024-04-12T04:11:02.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj29 2024-04-12T04:11:02.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj29 op=0 op_state=1 2024-04-12T04:11:02.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2024-04-12T04:11:02.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:02.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj29 2024-04-12T04:11:02.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj31 2024-04-12T04:11:02.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj31 op=0 op_state=1 2024-04-12T04:11:02.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj31[0] 2024-04-12T04:11:02.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj33 2024-04-12T04:11:02.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj33 op=0 op_state=1 2024-04-12T04:11:02.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj33[0] 2024-04-12T04:11:02.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj34 2024-04-12T04:11:02.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj34 op=0 op_state=1 2024-04-12T04:11:02.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj34[0] 2024-04-12T04:11:02.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj37 2024-04-12T04:11:02.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj37 op=0 op_state=1 2024-04-12T04:11:02.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj37[0] 2024-04-12T04:11:02.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj38 2024-04-12T04:11:02.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj38 op=0 op_state=1 2024-04-12T04:11:02.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj38[0] 2024-04-12T04:11:02.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj44 2024-04-12T04:11:02.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj44 op=0 op_state=1 2024-04-12T04:11:02.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.109+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj44[0] 2024-04-12T04:11:02.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.110+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj46 2024-04-12T04:11:02.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.110+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj46 op=0 op_state=1 2024-04-12T04:11:02.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.110+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj46[0] 2024-04-12T04:11:02.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.113+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: success 2024-04-12T04:11:02.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.113+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj51 2024-04-12T04:11:02.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.113+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj51 op=0 op_state=1 2024-04-12T04:11:02.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.113+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj51[0] 2024-04-12T04:11:02.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.113+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: success 2024-04-12T04:11:02.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.113+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: success 2024-04-12T04:11:02.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.113+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj55 2024-04-12T04:11:02.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.113+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj55 op=0 op_state=1 2024-04-12T04:11:02.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.113+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj55[0] 2024-04-12T04:11:02.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.113+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj57 2024-04-12T04:11:02.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.113+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj57 op=0 op_state=1 2024-04-12T04:11:02.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.114+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj57[0] 2024-04-12T04:11:02.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.115+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: success 2024-04-12T04:11:02.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.115+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj58 2024-04-12T04:11:02.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.115+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj58 op=0 op_state=1 2024-04-12T04:11:02.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.115+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj58[0] 2024-04-12T04:11:02.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.116+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:02.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.116+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj62 2024-04-12T04:11:02.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.116+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj62 op=0 op_state=1 2024-04-12T04:11:02.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.116+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj62[0] 2024-04-12T04:11:02.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.116+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: success 2024-04-12T04:11:02.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.116+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj68 2024-04-12T04:11:02.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.116+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj68 op=0 op_state=1 2024-04-12T04:11:02.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.116+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj68[0] 2024-04-12T04:11:02.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.117+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:02.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.117+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj69 2024-04-12T04:11:02.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.117+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj69 op=0 op_state=1 2024-04-12T04:11:02.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.117+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj69[0] 2024-04-12T04:11:02.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: success 2024-04-12T04:11:02.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: success 2024-04-12T04:11:02.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: success 2024-04-12T04:11:02.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: success 2024-04-12T04:11:02.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: success 2024-04-12T04:11:02.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: success 2024-04-12T04:11:02.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj74 2024-04-12T04:11:02.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj74 op=0 op_state=1 2024-04-12T04:11:02.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj74[0] 2024-04-12T04:11:02.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj76 2024-04-12T04:11:02.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj76 op=0 op_state=1 2024-04-12T04:11:02.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj76[0] 2024-04-12T04:11:02.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj85 2024-04-12T04:11:02.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj85 op=0 op_state=1 2024-04-12T04:11:02.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2024-04-12T04:11:02.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2024-04-12T04:11:02.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 0 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: ERROR: failed to sync object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj85 2024-04-12T04:11:02.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj87 2024-04-12T04:11:02.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj87 op=0 op_state=1 2024-04-12T04:11:02.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj87[0] 2024-04-12T04:11:02.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj92 2024-04-12T04:11:02.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj92 op=0 op_state=1 2024-04-12T04:11:02.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj92[0] 2024-04-12T04:11:02.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj93 2024-04-12T04:11:02.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj93 op=0 op_state=1 2024-04-12T04:11:02.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.118+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj93[0] 2024-04-12T04:11:02.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.119+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: success 2024-04-12T04:11:02.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.119+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: updating marker oid=bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24 marker=obj124 2024-04-12T04:11:02.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.119+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: success 2024-04-12T04:11:02.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.119+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: success 2024-04-12T04:11:02.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.119+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: success 2024-04-12T04:11:02.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.119+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: success 2024-04-12T04:11:02.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.119+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: success 2024-04-12T04:11:02.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.119+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: success 2024-04-12T04:11:02.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.119+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: success 2024-04-12T04:11:02.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.119+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj95 2024-04-12T04:11:02.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.119+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj95 op=0 op_state=1 2024-04-12T04:11:02.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.120+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj95[0] 2024-04-12T04:11:02.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.120+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: success 2024-04-12T04:11:02.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.121+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: success 2024-04-12T04:11:02.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.122+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: success 2024-04-12T04:11:02.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.122+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: success 2024-04-12T04:11:02.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.122+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: success 2024-04-12T04:11:02.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.122+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: success 2024-04-12T04:11:02.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.123+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: success 2024-04-12T04:11:02.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.123+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: success 2024-04-12T04:11:02.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.123+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: success 2024-04-12T04:11:02.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.124+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: success 2024-04-12T04:11:02.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.124+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: success 2024-04-12T04:11:02.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.124+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: success 2024-04-12T04:11:02.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.124+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: a sync operation returned error 2024-04-12T04:11:02.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.124+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: success 2024-04-12T04:11:02.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.124+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: backing out with sync_status=-5 2024-04-12T04:11:02.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.124+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: ERROR: full sync failed. error: -5 2024-04-12T04:11:02.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.126+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.126+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: waiting on bucket lease 2024-04-12T04:11:02.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.172+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: status after acquiring the lock is: full 2024-04-12T04:11:02.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.188+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: listing bucket for full sync 2024-04-12T04:11:02.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj11 2024-04-12T04:11:02.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj11 op=0 op_state=1 2024-04-12T04:11:02.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj11[0] 2024-04-12T04:11:02.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj111 2024-04-12T04:11:02.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj111 op=0 op_state=1 2024-04-12T04:11:02.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj111[0] 2024-04-12T04:11:02.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj115 2024-04-12T04:11:02.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj115 op=0 op_state=1 2024-04-12T04:11:02.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj115[0] 2024-04-12T04:11:02.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj116 2024-04-12T04:11:02.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj116 op=0 op_state=1 2024-04-12T04:11:02.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj116[0] 2024-04-12T04:11:02.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj124 2024-04-12T04:11:02.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj124 op=0 op_state=1 2024-04-12T04:11:02.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj124[0] 2024-04-12T04:11:02.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj145 2024-04-12T04:11:02.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj145 op=0 op_state=1 2024-04-12T04:11:02.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj145[0] 2024-04-12T04:11:02.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj148 2024-04-12T04:11:02.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj148 op=0 op_state=1 2024-04-12T04:11:02.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj148[0] 2024-04-12T04:11:02.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj155 2024-04-12T04:11:02.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj155 op=0 op_state=1 2024-04-12T04:11:02.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj155[0] 2024-04-12T04:11:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj17 2024-04-12T04:11:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj17 op=0 op_state=1 2024-04-12T04:11:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj17[0] 2024-04-12T04:11:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj185 2024-04-12T04:11:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj185 op=0 op_state=1 2024-04-12T04:11:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj185[0] 2024-04-12T04:11:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj19 2024-04-12T04:11:02.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj19 op=0 op_state=1 2024-04-12T04:11:02.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj19[0] 2024-04-12T04:11:02.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj22 2024-04-12T04:11:02.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj22 op=0 op_state=1 2024-04-12T04:11:02.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj22[0] 2024-04-12T04:11:02.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj23 2024-04-12T04:11:02.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj23 op=0 op_state=1 2024-04-12T04:11:02.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj23[0] 2024-04-12T04:11:02.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj29 2024-04-12T04:11:02.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj29 op=0 op_state=1 2024-04-12T04:11:02.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj29[0] 2024-04-12T04:11:02.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj31 2024-04-12T04:11:02.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj31 op=0 op_state=1 2024-04-12T04:11:02.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj31[0] 2024-04-12T04:11:02.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj33 2024-04-12T04:11:02.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj33 op=0 op_state=1 2024-04-12T04:11:02.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj33[0] 2024-04-12T04:11:02.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj34 2024-04-12T04:11:02.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj34 op=0 op_state=1 2024-04-12T04:11:02.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.236+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj34[0] 2024-04-12T04:11:02.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.237+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj37 2024-04-12T04:11:02.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.237+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj37 op=0 op_state=1 2024-04-12T04:11:02.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.237+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj37[0] 2024-04-12T04:11:02.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.237+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj38 2024-04-12T04:11:02.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.237+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj38 op=0 op_state=1 2024-04-12T04:11:02.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.237+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj38[0] 2024-04-12T04:11:02.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.237+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj44 2024-04-12T04:11:02.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.237+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj44 op=0 op_state=1 2024-04-12T04:11:02.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.237+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj44[0] 2024-04-12T04:11:02.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.237+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj46 2024-04-12T04:11:02.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.237+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj46 op=0 op_state=1 2024-04-12T04:11:02.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.237+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj46[0] 2024-04-12T04:11:02.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.241+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj22]: success 2024-04-12T04:11:02.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.241+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj115]: success 2024-04-12T04:11:02.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.241+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj33]: success 2024-04-12T04:11:02.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.241+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj51 2024-04-12T04:11:02.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.241+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj51 op=0 op_state=1 2024-04-12T04:11:02.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.241+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj51[0] 2024-04-12T04:11:02.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.241+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj55 2024-04-12T04:11:02.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.241+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj55 op=0 op_state=1 2024-04-12T04:11:02.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.241+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj55[0] 2024-04-12T04:11:02.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.241+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj57 2024-04-12T04:11:02.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.241+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj57 op=0 op_state=1 2024-04-12T04:11:02.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.241+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj116]: success 2024-04-12T04:11:02.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.241+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj57[0] 2024-04-12T04:11:02.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.241+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj58 2024-04-12T04:11:02.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.241+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj58 op=0 op_state=1 2024-04-12T04:11:02.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.241+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj58[0] 2024-04-12T04:11:02.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj185]: success 2024-04-12T04:11:02.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj29]: success 2024-04-12T04:11:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj19]: success 2024-04-12T04:11:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj62 2024-04-12T04:11:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj62 op=0 op_state=1 2024-04-12T04:11:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj62[0] 2024-04-12T04:11:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj68 2024-04-12T04:11:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj68 op=0 op_state=1 2024-04-12T04:11:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj68[0] 2024-04-12T04:11:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj69 2024-04-12T04:11:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj69 op=0 op_state=1 2024-04-12T04:11:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj69[0] 2024-04-12T04:11:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj111]: success 2024-04-12T04:11:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj11]: success 2024-04-12T04:11:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj74 2024-04-12T04:11:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj74 op=0 op_state=1 2024-04-12T04:11:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj74[0] 2024-04-12T04:11:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj17]: success 2024-04-12T04:11:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj76 2024-04-12T04:11:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj76 op=0 op_state=1 2024-04-12T04:11:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj76[0] 2024-04-12T04:11:02.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj85 2024-04-12T04:11:02.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj85 op=0 op_state=1 2024-04-12T04:11:02.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj44]: success 2024-04-12T04:11:02.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj38]: success 2024-04-12T04:11:02.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj85[0] 2024-04-12T04:11:02.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj155]: success 2024-04-12T04:11:02.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj87 2024-04-12T04:11:02.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj87 op=0 op_state=1 2024-04-12T04:11:02.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj87[0] 2024-04-12T04:11:02.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj92 2024-04-12T04:11:02.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj92 op=0 op_state=1 2024-04-12T04:11:02.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj92[0] 2024-04-12T04:11:02.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj93 2024-04-12T04:11:02.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.244+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj93 op=0 op_state=1 2024-04-12T04:11:02.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.245+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj93[0] 2024-04-12T04:11:02.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.245+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj145]: success 2024-04-12T04:11:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.245+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: [full sync] syncing object: dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0/obj95 2024-04-12T04:11:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.245+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b=bucket sync single entry (source_zone=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61) b= log_entry=obj95 op=0 op_state=1 2024-04-12T04:11:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.245+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj124]: success 2024-04-12T04:11:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.245+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: updating marker oid=bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24 marker=obj145 2024-04-12T04:11:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.245+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj31]: success 2024-04-12T04:11:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.245+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj148]: success 2024-04-12T04:11:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.245+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: bucket sync: sync obj: 3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61/:dkyges-61[3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24])/obj95[0] 2024-04-12T04:11:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.245+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj23]: success 2024-04-12T04:11:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.246+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj46]: success 2024-04-12T04:11:02.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.247+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj34]: success 2024-04-12T04:11:02.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.249+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj37]: success 2024-04-12T04:11:02.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.249+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj57]: success 2024-04-12T04:11:02.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.249+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj58]: success 2024-04-12T04:11:02.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.250+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj85]: success 2024-04-12T04:11:02.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.250+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj55]: success 2024-04-12T04:11:02.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.250+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj93]: success 2024-04-12T04:11:02.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.250+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj62]: success 2024-04-12T04:11:02.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.250+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj95]: success 2024-04-12T04:11:02.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.250+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj76]: success 2024-04-12T04:11:02.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.251+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj51]: success 2024-04-12T04:11:02.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.251+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: updating marker oid=bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24 marker=obj62 2024-04-12T04:11:02.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.251+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj68]: success 2024-04-12T04:11:02.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.251+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj87]: success 2024-04-12T04:11:02.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.251+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj92]: success 2024-04-12T04:11:02.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.251+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj74]: success 2024-04-12T04:11:02.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.251+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:entry[obj69]: success 2024-04-12T04:11:02.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.251+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: updating marker oid=bucket.full-sync-status.3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24 marker=obj95 2024-04-12T04:11:02.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.254+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: set bucket state=incremental 2024-04-12T04:11:02.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.255+0000 7f500152c780 5 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:full_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: bucket status objv={r=_cWFCNC5PSY1yiPBR6fKly-I:151,w=:0} 2024-04-12T04:11:02.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.257+0000 7f500152c780 10 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: WARNING: requested sync of past generation 0 < 6, returning success 2024-04-12T04:11:02.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.260+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.260+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.260+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.260+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.261+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.261+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: sync status for source bucket shard: 2 2024-04-12T04:11:02.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.261+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:1]: sync status for source bucket shard: 2 2024-04-12T04:11:02.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.261+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:3]: sync status for source bucket shard: 2 2024-04-12T04:11:02.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.261+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:0]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.261+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:1]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:1]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.261+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:3]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:3]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.261+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:2]: sync status for source bucket shard: 2 2024-04-12T04:11:02.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.261+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:2]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:2]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.261+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.262+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.262+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.262+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.262+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:4]: sync status for source bucket shard: 2 2024-04-12T04:11:02.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.262+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:4]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:4]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.263+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:5]: sync status for source bucket shard: 2 2024-04-12T04:11:02.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.263+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:5]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:5]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.263+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.263+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.263+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.263+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.263+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:7]: sync status for source bucket shard: 2 2024-04-12T04:11:02.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.263+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:6]: sync status for source bucket shard: 2 2024-04-12T04:11:02.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.263+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:8]: sync status for source bucket shard: 2 2024-04-12T04:11:02.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.263+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:7]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:7]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.263+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:6]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:6]: listing bilog for incremental sync; position=00000000002.140.6 2024-04-12T04:11:02.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.263+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:8]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:8]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.264+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.264+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.264+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.264+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.264+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:9]: sync status for source bucket shard: 2 2024-04-12T04:11:02.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.264+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:12]: sync status for source bucket shard: 2 2024-04-12T04:11:02.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.264+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:11]: sync status for source bucket shard: 2 2024-04-12T04:11:02.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.264+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:9]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:9]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.264+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:12]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:12]: listing bilog for incremental sync; position=00000000002.170.6 2024-04-12T04:11:02.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.264+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:11]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:11]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.265+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.265+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.265+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.265+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:16]: sync status for source bucket shard: 2 2024-04-12T04:11:02.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.265+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:16]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:16]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.265+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.265+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:14]: sync status for source bucket shard: 2 2024-04-12T04:11:02.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.265+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:15]: sync status for source bucket shard: 2 2024-04-12T04:11:02.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.265+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:13]: sync status for source bucket shard: 2 2024-04-12T04:11:02.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.265+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:14]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:14]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.265+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:15]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:15]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.265+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:13]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:13]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.266+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:18]: sync status for source bucket shard: 2 2024-04-12T04:11:02.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.266+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:18]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:18]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.266+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:17]: sync status for source bucket shard: 2 2024-04-12T04:11:02.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.266+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:17]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:17]: listing bilog for incremental sync; position=00000000002.110.6 2024-04-12T04:11:02.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.267+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.267+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.267+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.267+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.267+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.267+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.267+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.267+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:19]: sync status for source bucket shard: 2 2024-04-12T04:11:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.267+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:19]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:19]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.269+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.269+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.269+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.269+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.269+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.269+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.269+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.269+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:23]: sync status for source bucket shard: 2 2024-04-12T04:11:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.269+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:20]: sync status for source bucket shard: 2 2024-04-12T04:11:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.269+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:23]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:23]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.269+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:20]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:20]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.269+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.269+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2024-04-12T04:11:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:24]: sync status for source bucket shard: 2 2024-04-12T04:11:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:24]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:24]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:22]: sync status for source bucket shard: 2 2024-04-12T04:11:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:34]: sync status for source bucket shard: 2 2024-04-12T04:11:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:25]: sync status for source bucket shard: 2 2024-04-12T04:11:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:21]: sync status for source bucket shard: 2 2024-04-12T04:11:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:26]: sync status for source bucket shard: 2 2024-04-12T04:11:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:27]: sync status for source bucket shard: 2 2024-04-12T04:11:02.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:10]: sync status for source bucket shard: 2 2024-04-12T04:11:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:29]: sync status for source bucket shard: 2 2024-04-12T04:11:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:30]: sync status for source bucket shard: 2 2024-04-12T04:11:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:28]: sync status for source bucket shard: 2 2024-04-12T04:11:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:36]: sync status for source bucket shard: 2 2024-04-12T04:11:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:22]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:22]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:34]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:34]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:25]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:25]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:21]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:21]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:26]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:26]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:27]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:27]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:10]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:10]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:29]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:29]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:30]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:30]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:28]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:28]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.271+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:36]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:36]: listing bilog for incremental sync; position=00000000001.56.6 2024-04-12T04:11:02.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.273+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:33]: sync status for source bucket shard: 2 2024-04-12T04:11:02.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.273+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:32]: sync status for source bucket shard: 2 2024-04-12T04:11:02.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.273+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:31]: sync status for source bucket shard: 2 2024-04-12T04:11:02.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.273+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:33]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:33]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.273+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:32]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:32]: listing bilog for incremental sync; position=00000000001.103.6 2024-04-12T04:11:02.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.273+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:31]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:31]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.273+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:35]: sync status for source bucket shard: 2 2024-04-12T04:11:02.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:02.273+0000 7f500152c780 20 RGW-SYNC:bucket[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24<-dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:35]:inc_sync[dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24:35]: listing bilog for incremental sync; position= 2024-04-12T04:11:02.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:02.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:02.329 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:11:02.433 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:02.433 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:02.473 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:11:02.474 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.474 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj11", 2024-04-12T04:11:02.474 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.474 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.474 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.474 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T04:11:02.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.474 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.474 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.475 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.475 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.475 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.475 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:30.444861Z", 2024-04-12T04:11:02.475 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.475 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.475 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.475 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.475 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.475 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.475 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.476 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.476 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.9134814448263886263", 2024-04-12T04:11:02.476 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.476 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.476 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.476 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.476 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj111", 2024-04-12T04:11:02.476 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.476 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.477 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.477 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 8 2024-04-12T04:11:02.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.477 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.477 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.477 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.477 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.477 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:11:02.477 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:09:24.100799Z", 2024-04-12T04:11:02.477 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.478 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.478 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.478 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.478 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.478 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.478 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.478 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.478 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.3324926594168354454", 2024-04-12T04:11:02.478 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.479 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.479 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.479 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.479 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj115", 2024-04-12T04:11:02.479 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.479 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.479 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.479 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2 2024-04-12T04:11:02.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.479 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.479 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.480 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.480 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.480 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:11:02.480 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:05:01.190659Z", 2024-04-12T04:11:02.480 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.480 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.480 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.480 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.480 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.480 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.481 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.481 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.481 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.481 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.10012105876578540779", 2024-04-12T04:11:02.481 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.481 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.481 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.481 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.481 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.481 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj116", 2024-04-12T04:11:02.481 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.482 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.482 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.482 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 8 2024-04-12T04:11:02.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.482 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.482 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.482 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.482 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.482 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.482 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:06:49.199561Z", 2024-04-12T04:11:02.483 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.483 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.483 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.483 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.483 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.483 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.483 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.483 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.483 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.483 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.12222079022724568114", 2024-04-12T04:11:02.484 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.484 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.484 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.484 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.484 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj124", 2024-04-12T04:11:02.484 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.484 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.484 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.484 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T04:11:02.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.484 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.484 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.485 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.485 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.485 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T04:11:02.485 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:08:08.248308Z", 2024-04-12T04:11:02.485 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.485 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.485 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.485 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.485 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.485 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.485 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.485 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.486 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.10702209843775848580", 2024-04-12T04:11:02.486 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.486 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.486 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.486 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.486 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj145", 2024-04-12T04:11:02.486 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.486 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.486 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.487 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 8 2024-04-12T04:11:02.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.487 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.487 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.487 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.487 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.487 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.487 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:06:49.211076Z", 2024-04-12T04:11:02.487 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.487 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.487 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.487 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.488 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.488 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.488 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.488 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.488 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.488 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.634434875595912675", 2024-04-12T04:11:02.488 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.488 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.488 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.488 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.488 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.488 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj148", 2024-04-12T04:11:02.489 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.489 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.489 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.489 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 49 2024-04-12T04:11:02.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.489 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.489 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.489 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.489 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.489 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.489 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:09:24.114675Z", 2024-04-12T04:11:02.489 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.490 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.490 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.490 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.490 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.490 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.490 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.490 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.490 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.490 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.2744192851496125940", 2024-04-12T04:11:02.490 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.490 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.491 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.491 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.491 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj155", 2024-04-12T04:11:02.491 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.491 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.491 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.491 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 8 2024-04-12T04:11:02.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.491 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.491 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.491 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.491 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.492 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.492 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:08:08.299737Z", 2024-04-12T04:11:02.492 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.492 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.492 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.492 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.492 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.492 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.492 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.492 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.492 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.4481749252930249339", 2024-04-12T04:11:02.493 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.493 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.493 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.493 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj17", 2024-04-12T04:11:02.493 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.493 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.493 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.493 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T04:11:02.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.493 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.494 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.494 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.494 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.494 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.494 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:41.675980Z", 2024-04-12T04:11:02.494 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.494 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.494 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.494 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.494 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.494 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.495 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.495 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.495 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.12735664899470829563", 2024-04-12T04:11:02.495 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.495 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.495 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.495 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.495 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj185", 2024-04-12T04:11:02.495 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.495 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.496 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.496 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 21 2024-04-12T04:11:02.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.496 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.496 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.496 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.496 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.496 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.496 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:09:24.169265Z", 2024-04-12T04:11:02.496 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.496 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.496 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.496 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.497 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.497 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.497 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.497 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.497 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.11707428415253642041", 2024-04-12T04:11:02.497 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.497 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.497 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.497 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj19", 2024-04-12T04:11:02.498 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.498 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.498 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.498 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 10 2024-04-12T04:11:02.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.498 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.498 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.498 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.498 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.498 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:11:02.498 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:03:03.582137Z", 2024-04-12T04:11:02.499 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.499 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.499 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.499 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.499 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.499 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.499 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.499 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.499 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.11549047435019599737", 2024-04-12T04:11:02.499 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.499 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.500 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.500 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.500 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj22", 2024-04-12T04:11:02.500 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.500 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.500 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.500 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T04:11:02.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.500 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.500 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.501 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.501 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.501 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.501 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:30.456641Z", 2024-04-12T04:11:02.501 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.501 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.501 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.501 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.501 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.501 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.501 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.501 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.502 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.280262312616524196", 2024-04-12T04:11:02.502 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.502 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.502 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.502 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.502 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj23", 2024-04-12T04:11:02.502 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.502 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.502 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.503 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 22 2024-04-12T04:11:02.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.503 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.503 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.503 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.503 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.503 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.503 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:05:01.070582Z", 2024-04-12T04:11:02.503 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.503 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.503 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.503 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.503 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.504 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.504 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.504 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.504 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.17479935615879204323", 2024-04-12T04:11:02.504 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.504 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.504 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.504 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.504 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj29", 2024-04-12T04:11:02.504 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.505 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.505 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.505 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9 2024-04-12T04:11:02.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.505 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.505 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.505 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.505 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.505 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.505 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:06:49.121381Z", 2024-04-12T04:11:02.505 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.506 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.506 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.506 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.506 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.506 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.506 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.506 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.506 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.5376554775810453555", 2024-04-12T04:11:02.506 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.506 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.507 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.507 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj31", 2024-04-12T04:11:02.507 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.507 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.507 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.507 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 11 2024-04-12T04:11:02.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.507 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.507 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.507 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.507 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.508 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.508 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:08:08.174157Z", 2024-04-12T04:11:02.508 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.508 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.508 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.508 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.508 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.508 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.508 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.508 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.508 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.7341749001994611663", 2024-04-12T04:11:02.509 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.509 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.509 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.509 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj33", 2024-04-12T04:11:02.509 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.509 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.509 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.509 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T04:11:02.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.510 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.510 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.510 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.510 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.510 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.510 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:30.466236Z", 2024-04-12T04:11:02.510 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.510 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.510 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.510 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.510 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.510 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.511 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.511 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.511 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.16148896950644505833", 2024-04-12T04:11:02.511 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.511 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.511 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.511 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj34", 2024-04-12T04:11:02.511 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.511 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.512 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.512 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T04:11:02.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.512 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.512 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.512 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.512 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.512 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:11:02.512 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:41.687199Z", 2024-04-12T04:11:02.512 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.512 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.512 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.513 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.513 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.513 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.513 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.513 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.513 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.9267239352883598619", 2024-04-12T04:11:02.513 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.513 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.513 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj37", 2024-04-12T04:11:02.514 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.514 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.514 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.514 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 10 2024-04-12T04:11:02.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.514 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.514 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.514 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.514 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.514 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.515 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:09:24.034250Z", 2024-04-12T04:11:02.515 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.515 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.515 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.515 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.515 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.515 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.515 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.515 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.515 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.2073928281617508422", 2024-04-12T04:11:02.516 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.516 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.516 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.516 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj38", 2024-04-12T04:11:02.516 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.516 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.516 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.516 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T04:11:02.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.516 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.517 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.517 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.517 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.517 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T04:11:02.517 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:03:03.629194Z", 2024-04-12T04:11:02.517 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.517 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.517 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.517 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.517 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.517 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.518 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.518 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.518 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.3288006229502996394", 2024-04-12T04:11:02.518 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.518 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.518 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.518 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj44", 2024-04-12T04:11:02.518 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.519 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.519 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T04:11:02.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.519 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.519 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.519 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.519 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.519 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.519 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:30.522149Z", 2024-04-12T04:11:02.519 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.520 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.520 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.520 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.520 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.520 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.520 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.520 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.520 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.14097535185761139072", 2024-04-12T04:11:02.520 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.520 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.521 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.521 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj46", 2024-04-12T04:11:02.521 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.521 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.521 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.521 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 18 2024-04-12T04:11:02.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.521 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.521 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.521 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.522 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.522 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.522 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:05:01.082150Z", 2024-04-12T04:11:02.522 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.522 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.522 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.522 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.522 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.522 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.522 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.522 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.523 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.8886091888534882088", 2024-04-12T04:11:02.523 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.523 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.523 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.523 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj51", 2024-04-12T04:11:02.523 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.523 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.523 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.523 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 31 2024-04-12T04:11:02.524 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.524 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.524 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.524 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.524 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.524 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:11:02.524 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:41.699090Z", 2024-04-12T04:11:02.524 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.524 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.524 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.524 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.524 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.525 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.525 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.525 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.525 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.13671635306629478451", 2024-04-12T04:11:02.525 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.525 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.525 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.525 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj55", 2024-04-12T04:11:02.525 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.526 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.526 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.526 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T04:11:02.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.526 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.526 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.526 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.526 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.526 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.526 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:30.533855Z", 2024-04-12T04:11:02.526 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.527 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.527 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.527 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.527 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.527 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.527 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.527 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.527 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.1688041635958809482", 2024-04-12T04:11:02.527 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.527 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.527 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.528 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj57", 2024-04-12T04:11:02.528 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.528 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.528 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.528 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14 2024-04-12T04:11:02.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.528 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.528 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.528 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.528 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.529 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.529 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:03:03.638996Z", 2024-04-12T04:11:02.529 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.529 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.529 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.529 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.529 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.529 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.529 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.529 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.529 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.18341895293276276705", 2024-04-12T04:11:02.530 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.530 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.530 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.530 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj58", 2024-04-12T04:11:02.530 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.530 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.530 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.530 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 22 2024-04-12T04:11:02.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.531 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.531 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.531 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.531 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.531 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.531 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:06:49.133205Z", 2024-04-12T04:11:02.531 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.531 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.531 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.531 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.531 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.532 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.532 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.532 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.532 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.16719036296186282733", 2024-04-12T04:11:02.532 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.532 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.532 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.532 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj62", 2024-04-12T04:11:02.533 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.533 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.533 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.533 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 15 2024-04-12T04:11:02.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.533 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.533 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.533 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.533 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.533 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.533 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:08:08.186687Z", 2024-04-12T04:11:02.533 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.534 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.534 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.534 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.534 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.534 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.534 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.534 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.534 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.12584147571205046145", 2024-04-12T04:11:02.534 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.535 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.535 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.535 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj68", 2024-04-12T04:11:02.535 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.535 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.535 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.535 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 11 2024-04-12T04:11:02.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.535 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.536 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.536 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.536 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.536 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T04:11:02.536 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:41.748656Z", 2024-04-12T04:11:02.536 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.536 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.536 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.536 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.536 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.536 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.537 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.537 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.537 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.9495816900209411563", 2024-04-12T04:11:02.537 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.537 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.537 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.537 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.537 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj69", 2024-04-12T04:11:02.537 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.538 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.538 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.538 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T04:11:02.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.538 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.538 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.538 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.538 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.538 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.538 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:05:01.130607Z", 2024-04-12T04:11:02.538 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.539 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.539 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.539 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.539 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.539 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.539 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.539 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.539 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.7916024072629042549", 2024-04-12T04:11:02.539 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.539 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.539 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.540 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj74", 2024-04-12T04:11:02.540 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.540 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.540 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.540 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14 2024-04-12T04:11:02.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.540 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.540 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.541 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.541 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.541 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:11:02.541 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:09:24.047621Z", 2024-04-12T04:11:02.541 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.541 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.541 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.541 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.541 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.541 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.541 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.542 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.542 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.13819912421786560207", 2024-04-12T04:11:02.542 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.542 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.542 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.542 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj76", 2024-04-12T04:11:02.542 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.543 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.543 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.543 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 20 2024-04-12T04:11:02.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.543 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.543 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.543 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.543 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.543 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:11:02.543 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:03:03.689940Z", 2024-04-12T04:11:02.543 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.543 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.544 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.544 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.544 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.544 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.544 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.544 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.544 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.4480597793252289857", 2024-04-12T04:11:02.544 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.544 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.544 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.545 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj85", 2024-04-12T04:11:02.545 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.545 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.545 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.546 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 11 2024-04-12T04:11:02.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.546 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.546 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.546 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.546 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.546 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.546 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:41.759706Z", 2024-04-12T04:11:02.547 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.547 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.547 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.547 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.547 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.547 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.547 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.547 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.547 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.10386226148209467049", 2024-04-12T04:11:02.547 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.547 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.548 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.548 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj87", 2024-04-12T04:11:02.548 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.548 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.548 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.548 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 8 2024-04-12T04:11:02.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.548 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.548 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.548 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.548 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.549 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.549 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:06:49.144674Z", 2024-04-12T04:11:02.549 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.549 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.549 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.549 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.549 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.549 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.549 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.549 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.549 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.16017112119272362811", 2024-04-12T04:11:02.550 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.550 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.550 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.550 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj92", 2024-04-12T04:11:02.550 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.550 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.550 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.550 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T04:11:02.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.551 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.551 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.551 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.551 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.551 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T04:11:02.551 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:05:01.141706Z", 2024-04-12T04:11:02.551 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.551 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.551 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.551 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.551 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.551 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.552 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.552 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.552 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.10014809341276485556", 2024-04-12T04:11:02.552 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.552 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.552 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.552 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.552 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj93", 2024-04-12T04:11:02.552 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.552 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.553 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.553 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2 2024-04-12T04:11:02.553 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.553 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.553 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.553 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.553 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.553 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.553 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:08:08.235885Z", 2024-04-12T04:11:02.553 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.553 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.554 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.554 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.554 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.554 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.554 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.554 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.554 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.5121621692016013614", 2024-04-12T04:11:02.554 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.554 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.555 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.555 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.555 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj95", 2024-04-12T04:11:02.555 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.555 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.555 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 9, 2024-04-12T04:11:02.555 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 2 2024-04-12T04:11:02.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.555 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.555 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.555 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.556 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.556 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.556 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:03:03.702005Z", 2024-04-12T04:11:02.556 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.556 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.556 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.556 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.556 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.556 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.556 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.556 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.557 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.14047554371884981936", 2024-04-12T04:11:02.557 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.557 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.557 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.557 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:02.557 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:11:02.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:02.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-61', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:02.558 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-61 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:11:02.619 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:02.619 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:02.659 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:11:02.659 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.659 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj11", 2024-04-12T04:11:02.659 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.659 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.659 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.659 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T04:11:02.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.659 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.659 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.660 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.660 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.660 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T04:11:02.660 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:30.444861Z", 2024-04-12T04:11:02.660 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.660 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.660 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.660 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.660 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.660 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.660 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.661 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.661 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_Fg6pcLnpMt3bCS75eGz-ksAgnyI7H8g", 2024-04-12T04:11:02.661 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.661 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.661 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.661 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.661 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj111", 2024-04-12T04:11:02.661 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.662 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.662 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.662 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12 2024-04-12T04:11:02.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.662 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.662 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.662 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.662 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.662 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:11:02.662 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:09:24.100799Z", 2024-04-12T04:11:02.662 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.663 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.663 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.663 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.663 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.663 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.663 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.663 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.663 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_A8wfY46UpQjUvLvCeYufnVZgMbOBDUY", 2024-04-12T04:11:02.663 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.664 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.664 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.664 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.664 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj115", 2024-04-12T04:11:02.664 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.664 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.664 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.664 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T04:11:02.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.664 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.665 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.665 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.665 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.665 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:11:02.665 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:05:01.190659Z", 2024-04-12T04:11:02.665 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.665 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.665 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.665 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.665 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.665 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.665 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.666 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.666 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_brn0Zl2omBPmd1xceKelYwm3D0gyZMN", 2024-04-12T04:11:02.666 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.666 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.666 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.666 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.666 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj116", 2024-04-12T04:11:02.666 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.666 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.667 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.667 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T04:11:02.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.667 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.667 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.667 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.667 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.667 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.667 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:06:49.199561Z", 2024-04-12T04:11:02.667 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.667 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.668 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.668 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.668 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.668 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.668 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.668 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.668 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_vA1jCYJKoyqKnJmaE6ZZ9nZRF2yefBq", 2024-04-12T04:11:02.668 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.668 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.668 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.669 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.669 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj124", 2024-04-12T04:11:02.669 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.669 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.669 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.669 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 11 2024-04-12T04:11:02.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.669 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.669 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.669 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.670 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.670 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.670 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:08:08.248308Z", 2024-04-12T04:11:02.670 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.670 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.670 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.670 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.670 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.670 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.670 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.670 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.671 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_JxD7esxuElNNiUFTFeADVSWVYNOIiIg", 2024-04-12T04:11:02.671 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.671 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.671 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.671 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj145", 2024-04-12T04:11:02.671 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.671 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.671 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.671 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 10 2024-04-12T04:11:02.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.672 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.672 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.672 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.672 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.672 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.672 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:06:49.211076Z", 2024-04-12T04:11:02.672 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.672 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.672 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.672 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.672 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.673 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.673 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.673 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.673 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_DxPINzgaePg7Ua4qeEABXuTOtZV62nw", 2024-04-12T04:11:02.673 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.673 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.673 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.673 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.673 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj148", 2024-04-12T04:11:02.673 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.674 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.674 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.674 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 40 2024-04-12T04:11:02.674 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.674 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.674 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.674 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.674 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.674 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.674 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:09:24.114675Z", 2024-04-12T04:11:02.674 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.675 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.675 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.675 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.675 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.675 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.675 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.675 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.675 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_L4zB0bTxsczRvSxG42px_21cJlC8E9M", 2024-04-12T04:11:02.675 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.675 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.675 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.676 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.676 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj155", 2024-04-12T04:11:02.676 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.676 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.676 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.676 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T04:11:02.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.676 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.676 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.676 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.677 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.677 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.677 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:08:08.299737Z", 2024-04-12T04:11:02.677 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.677 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.677 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.677 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.677 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.677 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.677 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.677 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.678 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_smci8q9Rjke-W-iD4TWynjLERckdzY0", 2024-04-12T04:11:02.678 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.678 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.678 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.678 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.678 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj17", 2024-04-12T04:11:02.678 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.678 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.678 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.679 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 8 2024-04-12T04:11:02.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.679 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.679 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.679 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.679 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.679 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T04:11:02.679 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:41.675980Z", 2024-04-12T04:11:02.679 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.679 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.679 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.679 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.679 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.680 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.680 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.680 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.680 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_k_GU9VKeKDIue6pSKCUMsNSowV65Yew", 2024-04-12T04:11:02.680 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.680 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.680 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.681 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj185", 2024-04-12T04:11:02.681 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.681 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.681 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.681 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 24 2024-04-12T04:11:02.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.681 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.681 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.681 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.681 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.681 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T04:11:02.682 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:09:24.169265Z", 2024-04-12T04:11:02.682 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.682 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.682 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.682 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.682 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.682 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.682 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.682 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.683 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_ZAuCWffM9xG0KyTHKSOXlIIDUPiTXjE", 2024-04-12T04:11:02.683 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.683 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.683 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.683 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.683 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.683 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj19", 2024-04-12T04:11:02.683 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.683 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.683 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.683 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9 2024-04-12T04:11:02.683 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.684 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.684 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.684 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.684 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.684 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.684 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:03:03.582137Z", 2024-04-12T04:11:02.684 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.684 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.684 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.684 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.684 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.684 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.685 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.685 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.685 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.685 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_NCGcXCgh6Xx7z_r0qy7YGMEvqx4k-M6", 2024-04-12T04:11:02.685 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.685 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.685 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.685 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.685 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.685 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj22", 2024-04-12T04:11:02.685 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.685 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.686 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.686 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12 2024-04-12T04:11:02.686 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.686 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.686 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.686 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.686 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.686 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.686 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:30.456641Z", 2024-04-12T04:11:02.686 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.686 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.687 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.687 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.687 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.687 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.687 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.687 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.687 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.687 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_Cbzr8egBK--Wiw0fNhl_Od2V5cvHMUa", 2024-04-12T04:11:02.687 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.687 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.687 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.687 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.687 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.688 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj23", 2024-04-12T04:11:02.688 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.688 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.688 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.688 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 17 2024-04-12T04:11:02.688 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.688 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.688 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.688 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.688 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.688 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:11:02.688 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:05:01.070582Z", 2024-04-12T04:11:02.689 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.689 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.689 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.689 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.689 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.689 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.689 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.689 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.689 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.689 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_TIFoDCmzbesTcySFfgUYD87anyZvOQ2", 2024-04-12T04:11:02.689 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.689 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.690 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.690 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.690 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.690 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj29", 2024-04-12T04:11:02.690 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.690 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.690 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.690 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 11 2024-04-12T04:11:02.690 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.690 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.690 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.691 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.691 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.691 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.691 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:06:49.121381Z", 2024-04-12T04:11:02.691 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.691 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.691 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.691 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.691 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.691 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.692 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.692 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.692 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.692 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_uxK0f8Gcr0ZzvVr2JMrah3Vo4eB-BFl", 2024-04-12T04:11:02.692 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.692 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.692 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.692 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.692 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.692 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj31", 2024-04-12T04:11:02.692 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.692 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.693 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.693 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9 2024-04-12T04:11:02.693 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.693 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.693 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.693 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.693 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.693 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.693 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:08:08.174157Z", 2024-04-12T04:11:02.693 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.693 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.694 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.694 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.694 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.694 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.694 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.694 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.694 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.694 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_h7SlPQEUCA9iXNf9pGCGtX_24hV8tWL", 2024-04-12T04:11:02.694 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.694 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.695 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.695 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.695 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj33", 2024-04-12T04:11:02.695 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.695 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.695 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.695 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 12 2024-04-12T04:11:02.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.695 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.695 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.695 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.696 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.696 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:11:02.696 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:30.466236Z", 2024-04-12T04:11:02.696 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.696 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.696 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.696 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.696 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.696 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.696 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.696 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.697 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.697 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_zOFfkO-JQaUCmCZ2rte2iTzfJkqPA0K", 2024-04-12T04:11:02.697 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.697 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.697 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.697 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.697 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.697 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj34", 2024-04-12T04:11:02.697 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.697 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.697 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.698 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T04:11:02.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.698 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.698 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.698 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.698 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.698 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T04:11:02.698 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:41.687199Z", 2024-04-12T04:11:02.698 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.698 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.698 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.699 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.699 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.699 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.699 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.699 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.699 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_atMQcODHRcSeFDwxx56s8rPfVo81OI6", 2024-04-12T04:11:02.699 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.699 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.699 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.699 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj37", 2024-04-12T04:11:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 20 2024-04-12T04:11:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.700 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:09:24.034250Z", 2024-04-12T04:11:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "__lyDM0f8zaUC_H_iO_KO3TyTioJ_Ow4", 2024-04-12T04:11:02.701 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj38", 2024-04-12T04:11:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9 2024-04-12T04:11:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.702 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 3, 2024-04-12T04:11:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:03:03.629194Z", 2024-04-12T04:11:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.703 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_WqVZrc25ffP-ac2nTSjnDuaTSYWUeBs", 2024-04-12T04:11:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj44", 2024-04-12T04:11:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.704 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 7 2024-04-12T04:11:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:30.522149Z", 2024-04-12T04:11:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.705 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.706 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.706 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.706 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.706 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.706 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_pNx0xIo2-0R4iz5GYZJbO1ypkBobivB", 2024-04-12T04:11:02.706 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.706 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.706 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.706 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.706 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj46", 2024-04-12T04:11:02.707 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.707 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.707 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.707 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 16 2024-04-12T04:11:02.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.707 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.707 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.707 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.707 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.707 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.707 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:05:01.082150Z", 2024-04-12T04:11:02.707 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.708 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.708 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.708 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.708 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.708 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.708 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.708 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.708 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_lhzyINbYFDKAYH4rxI_HE2GbadtFQER", 2024-04-12T04:11:02.708 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.708 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.708 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.709 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj51", 2024-04-12T04:11:02.709 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.709 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.709 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.709 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 11 2024-04-12T04:11:02.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.709 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.709 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.709 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.709 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.710 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.710 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:41.699090Z", 2024-04-12T04:11:02.710 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.710 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.710 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.710 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.710 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.710 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.710 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.710 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.711 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_BM0k15CeWjaG995M0FezLpTYZUFB0V2", 2024-04-12T04:11:02.711 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.711 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.711 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.711 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj55", 2024-04-12T04:11:02.711 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.711 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.711 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.711 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 6 2024-04-12T04:11:02.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.712 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.712 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.712 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.712 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.712 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.712 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:30.533855Z", 2024-04-12T04:11:02.712 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.712 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.712 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.712 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.712 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.712 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.713 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.713 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.713 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_x9y8aX1Y1fXxXTe-g2hgsf-uqIZ0wDK", 2024-04-12T04:11:02.713 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.713 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.713 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.713 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.713 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj57", 2024-04-12T04:11:02.713 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.713 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.713 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.714 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9 2024-04-12T04:11:02.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.714 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.714 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.714 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.714 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.714 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.714 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:03:03.638996Z", 2024-04-12T04:11:02.714 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.714 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.714 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.714 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.715 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.715 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.715 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.715 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.715 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_Wb6YWvnm8sh1BeZriE2dgm7WnPGwKyJ", 2024-04-12T04:11:02.715 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.715 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.715 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.715 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.715 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj58", 2024-04-12T04:11:02.716 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.716 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.716 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.716 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 16 2024-04-12T04:11:02.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.716 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.716 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.716 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.716 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.716 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.716 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:06:49.133205Z", 2024-04-12T04:11:02.716 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.716 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.717 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.717 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.717 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.717 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.717 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.717 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.717 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_seZd0Z3UWS3tsFnKOijqBQWSGfHRAch", 2024-04-12T04:11:02.717 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.717 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.717 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.718 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.718 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj62", 2024-04-12T04:11:02.718 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.718 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.718 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.718 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 8 2024-04-12T04:11:02.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.718 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.718 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.718 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.719 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.719 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.719 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:08:08.186687Z", 2024-04-12T04:11:02.719 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.719 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.719 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.719 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.719 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.719 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.719 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.719 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.720 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_hIkqvklF1ILzAWvqIFO9Bb_Q2a7gceE", 2024-04-12T04:11:02.720 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.720 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.720 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.720 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.720 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj68", 2024-04-12T04:11:02.720 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.720 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.720 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.720 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 11 2024-04-12T04:11:02.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.721 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.721 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.721 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.721 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.721 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.721 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:41.748656Z", 2024-04-12T04:11:02.721 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.721 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.721 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.721 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.721 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.721 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.722 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.722 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.722 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_1iB7mL0rrSkQRfLhwg14iLb7KI8B52A", 2024-04-12T04:11:02.722 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.722 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.722 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.722 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.722 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj69", 2024-04-12T04:11:02.722 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.723 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.723 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.723 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 9 2024-04-12T04:11:02.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.723 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.723 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.723 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.723 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.723 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.723 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:05:01.130607Z", 2024-04-12T04:11:02.723 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.723 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.724 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.724 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.724 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.724 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.724 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.724 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.724 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_H_GfHJy4LBX_UOQ9HwezS62CZruRLD7", 2024-04-12T04:11:02.724 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.724 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.724 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.725 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.725 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj74", 2024-04-12T04:11:02.725 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.725 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.725 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.725 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 10 2024-04-12T04:11:02.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.725 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.725 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.725 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.725 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.726 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:11:02.726 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:09:24.047621Z", 2024-04-12T04:11:02.726 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.726 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.726 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.726 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.726 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.726 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.726 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.726 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.726 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_R3ImnhhFDNSvEFMLQ3DS-e9uyq_q1-M", 2024-04-12T04:11:02.727 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.727 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.727 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.727 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.727 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj76", 2024-04-12T04:11:02.727 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.727 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.727 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.727 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 25 2024-04-12T04:11:02.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.727 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.728 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.728 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.728 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.728 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.728 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:03:03.689940Z", 2024-04-12T04:11:02.728 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.728 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.728 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.728 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.728 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.728 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.728 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.728 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.729 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.729 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_DxWh4EBx2oEJy8tBsxOMpYypMMfFBwx", 2024-04-12T04:11:02.729 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.729 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.729 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.729 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.729 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.729 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj85", 2024-04-12T04:11:02.729 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.729 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.729 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.730 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 5 2024-04-12T04:11:02.730 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.730 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.730 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.730 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.730 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.730 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.730 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:00:41.759706Z", 2024-04-12T04:11:02.730 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.730 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.730 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.731 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.731 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.731 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.731 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.731 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.731 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_9e8CZL_Y6WJ3AmVshtR9-80-anpqETL", 2024-04-12T04:11:02.731 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.731 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.731 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.731 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.732 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj87", 2024-04-12T04:11:02.732 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.732 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.732 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.732 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 13 2024-04-12T04:11:02.732 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.732 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.733 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.733 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.733 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.733 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 6, 2024-04-12T04:11:02.733 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:06:49.144674Z", 2024-04-12T04:11:02.733 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.734 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.734 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.734 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.734 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.734 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.734 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.734 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.734 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.734 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_kvf9VwA3Z5JVNXVSJDBX0sHaG9RVeoe", 2024-04-12T04:11:02.734 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.734 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.735 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.735 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.735 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.735 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj92", 2024-04-12T04:11:02.735 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.735 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.735 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.735 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 11 2024-04-12T04:11:02.735 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.735 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.735 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.735 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.736 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.736 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.736 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:05:01.141706Z", 2024-04-12T04:11:02.736 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.736 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.736 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.736 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.736 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.736 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.736 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.736 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.737 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_JS26BNF6ZB0eTFpvFytsWxOSTPEEEjL", 2024-04-12T04:11:02.737 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.737 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.737 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.737 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.737 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.737 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj93", 2024-04-12T04:11:02.737 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.737 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.737 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.737 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T04:11:02.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.738 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.738 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.738 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.738 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.738 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 16, 2024-04-12T04:11:02.738 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:08:08.235885Z", 2024-04-12T04:11:02.738 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.738 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.738 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.739 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.739 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.739 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.739 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.739 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.739 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_XPR8QRtc6cO2kaJre35rKdUTyHDReAw", 2024-04-12T04:11:02.739 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.739 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.739 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.740 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:02.740 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "obj95", 2024-04-12T04:11:02.740 INFO:teuthology.orchestra.run.smithi043.stdout: "instance": "", 2024-04-12T04:11:02.740 INFO:teuthology.orchestra.run.smithi043.stdout: "ver": { 2024-04-12T04:11:02.740 INFO:teuthology.orchestra.run.smithi043.stdout: "pool": 13, 2024-04-12T04:11:02.740 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 4 2024-04-12T04:11:02.740 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.740 INFO:teuthology.orchestra.run.smithi043.stdout: "locator": "", 2024-04-12T04:11:02.740 INFO:teuthology.orchestra.run.smithi043.stdout: "exists": true, 2024-04-12T04:11:02.740 INFO:teuthology.orchestra.run.smithi043.stdout: "meta": { 2024-04-12T04:11:02.740 INFO:teuthology.orchestra.run.smithi043.stdout: "category": 1, 2024-04-12T04:11:02.740 INFO:teuthology.orchestra.run.smithi043.stdout: "size": 5, 2024-04-12T04:11:02.740 INFO:teuthology.orchestra.run.smithi043.stdout: "mtime": "2024-04-12T04:03:03.702005Z", 2024-04-12T04:11:02.741 INFO:teuthology.orchestra.run.smithi043.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2024-04-12T04:11:02.741 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_class": "", 2024-04-12T04:11:02.741 INFO:teuthology.orchestra.run.smithi043.stdout: "owner": "rgw-multisite-test-user", 2024-04-12T04:11:02.741 INFO:teuthology.orchestra.run.smithi043.stdout: "owner_display_name": "Multisite Test User", 2024-04-12T04:11:02.741 INFO:teuthology.orchestra.run.smithi043.stdout: "content_type": "application/octet-stream", 2024-04-12T04:11:02.741 INFO:teuthology.orchestra.run.smithi043.stdout: "accounted_size": 3, 2024-04-12T04:11:02.741 INFO:teuthology.orchestra.run.smithi043.stdout: "user_data": "", 2024-04-12T04:11:02.741 INFO:teuthology.orchestra.run.smithi043.stdout: "appendable": false 2024-04-12T04:11:02.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:02.741 INFO:teuthology.orchestra.run.smithi043.stdout: "tag": "_aPIbBVLGf3VEWetrJoTcnY33tEi1UWX", 2024-04-12T04:11:02.741 INFO:teuthology.orchestra.run.smithi043.stdout: "flags": 0, 2024-04-12T04:11:02.742 INFO:teuthology.orchestra.run.smithi043.stdout: "pending_map": [], 2024-04-12T04:11:02.742 INFO:teuthology.orchestra.run.smithi043.stdout: "versioned_epoch": 0 2024-04-12T04:11:02.742 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:02.742 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:11:02.742 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T04:11:02.746 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T04:11:02.748 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T04:11:02.749 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T04:11:02.775 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:11:02.776 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T04:11:02.776 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi043.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T04:11:03.777 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:03.803 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:11:03.803 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T04:11:03.805 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T04:11:03.805 INFO:teuthology.orchestra.run.smithi043.stdout:anonymous 2024-04-12T04:11:03.806 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-61 2024-04-12T04:11:03.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:11:03.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:11:03.806 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:11:03.914 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:03.914 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:03.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:03.978+0000 7fcf7009a780 1 bucket sync caught up with source: 2024-04-12T04:11:03.962 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.140.6, , , , , , 00000000002.170.6, , , , , 00000000002.110.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:11:03.962 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.140.6, , , , , , 00000000002.170.6, , , , , 00000000002.110.6, , , , , , , , , , , , , , , 00000000001.103.6, , , , 00000000001.56.6] 2024-04-12T04:11:03.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:03.978+0000 7fcf7009a780 0 bucket checkpoint complete 2024-04-12T04:11:03.979 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-61 2024-04-12T04:11:03.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:11:03.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:11:03.979 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:11:04.043 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:04.043 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:04.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:04.102+0000 7f80094da780 1 bucket sync caught up with source: 2024-04-12T04:11:04.085 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000009.2434.6, 00000000006.1427.6, 00000000006.3014.6, 00000000015.2431.6, 00000000012.2629.6, 00000000003.3060.6, 00000000003.3646.6, 00000000012.2593.6, 00000000021.2328.6, 00000000018.2654.6] 2024-04-12T04:11:04.085 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000009.2434.6, 00000000006.1427.6, 00000000006.3014.6, 00000000015.2431.6, 00000000012.2629.6, 00000000003.3060.6, 00000000003.3646.6, 00000000012.2593.6, 00000000021.2328.6, 00000000018.2654.6] 2024-04-12T04:11:04.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:04.102+0000 7f80094da780 0 bucket checkpoint complete 2024-04-12T04:11:04.102 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-61 zones={a2, a1} 2024-04-12T04:11:04.102 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.102 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.102 DEBUG:boto:Method: HEAD 2024-04-12T04:11:04.102 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:11:04.102 DEBUG:boto:Data: 2024-04-12T04:11:04.102 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.103 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.103 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.103 DEBUG:boto:Params: {} 2024-04-12T04:11:04.103 DEBUG:boto:Token: None 2024-04-12T04:11:04.103 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/ 2024-04-12T04:11:04.103 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rPb2bo32r/179kGji3GwY0wFwHg= 2024-04-12T04:11:04.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rPb2bo32r/179kGji3GwY0wFwHg=', 'Content-Length': '0'} 2024-04-12T04:11:04.104 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2024-04-12T04:11:04.104 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:11:04.554 DEBUG:boto:Token: None 2024-04-12T04:11:04.554 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/ 2024-04-12T04:11:04.555 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rPb2bo32r/179kGji3GwY0wFwHg= 2024-04-12T04:11:04.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rPb2bo32r/179kGji3GwY0wFwHg=', 'Content-Length': '0'} 2024-04-12T04:11:04.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001755ea84b5d83308-006618b458-1fa3-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.570 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.570 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.571 DEBUG:boto:Method: HEAD 2024-04-12T04:11:04.571 DEBUG:boto:Path: /dkyges-61/ 2024-04-12T04:11:04.571 DEBUG:boto:Data: 2024-04-12T04:11:04.571 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.571 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.571 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.571 DEBUG:boto:Params: {} 2024-04-12T04:11:04.571 DEBUG:boto:Token: None 2024-04-12T04:11:04.572 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/ 2024-04-12T04:11:04.572 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rPb2bo32r/179kGji3GwY0wFwHg= 2024-04-12T04:11:04.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rPb2bo32r/179kGji3GwY0wFwHg=', 'Content-Length': '0'} 2024-04-12T04:11:04.576 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c073b2f2d617d2a-006618b458-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.576 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:11:04.576 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.576 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.576 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:11:04.577 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:11:04.577 DEBUG:boto:Method: GET 2024-04-12T04:11:04.577 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:11:04.577 DEBUG:boto:Data: 2024-04-12T04:11:04.577 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.577 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.577 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.577 DEBUG:boto:Params: {} 2024-04-12T04:11:04.577 DEBUG:boto:Token: None 2024-04-12T04:11:04.577 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/?versions 2024-04-12T04:11:04.578 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zALnf4JU5P8wyduoNVMHGAd31l0= 2024-04-12T04:11:04.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zALnf4JU5P8wyduoNVMHGAd31l0=', 'Content-Length': '0'} 2024-04-12T04:11:04.580 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045f83fe04273651d-006618b458-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.580 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T04:09:24.114Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T04:09:24.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T04:09:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T04:09:24.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:04.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:11:04.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-12T04:11:04.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T04:11:04.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T04:11:04.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-12T04:11:04.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T04:11:04.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-12T04:11:04.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-12T04:11:04.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:11:04.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-12T04:11:04.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T04:11:04.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:11:04.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T04:11:04.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T04:11:04.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-12T04:11:04.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:11:04.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:11:04.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-12T04:11:04.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T04:11:04.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:11:04.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T04:11:04.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:11:04.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:11:04.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T04:11:04.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T04:11:04.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-12T04:11:04.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:11:04.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T04:11:04.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-12T04:11:04.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T04:11:04.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:11:04.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T04:11:04.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T04:11:04.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-12T04:11:04.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T04:11:04.588 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:11:04.588 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.588 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.589 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:11:04.589 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:11:04.589 DEBUG:boto:Method: GET 2024-04-12T04:11:04.589 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:11:04.590 DEBUG:boto:Data: 2024-04-12T04:11:04.590 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.590 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.590 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.590 DEBUG:boto:Params: {} 2024-04-12T04:11:04.591 DEBUG:boto:Token: None 2024-04-12T04:11:04.591 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/?versions 2024-04-12T04:11:04.591 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zALnf4JU5P8wyduoNVMHGAd31l0= 2024-04-12T04:11:04.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zALnf4JU5P8wyduoNVMHGAd31l0=', 'Content-Length': '0'} 2024-04-12T04:11:04.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa7d2234f0b0d564-006618b458-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.597 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T04:09:24.114Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T04:09:24.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T04:09:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T04:09:24.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:04.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:11:04.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-12T04:11:04.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T04:11:04.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T04:11:04.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-12T04:11:04.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T04:11:04.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-12T04:11:04.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-12T04:11:04.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:11:04.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-12T04:11:04.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T04:11:04.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:11:04.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T04:11:04.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T04:11:04.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-12T04:11:04.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:11:04.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:11:04.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-12T04:11:04.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T04:11:04.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:11:04.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T04:11:04.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:11:04.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:11:04.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T04:11:04.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T04:11:04.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-12T04:11:04.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:11:04.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T04:11:04.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-12T04:11:04.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T04:11:04.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:11:04.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T04:11:04.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T04:11:04.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-12T04:11:04.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T04:11:04.604 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.604 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.604 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:11:04.604 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:11:04.604 DEBUG:boto:Method: GET 2024-04-12T04:11:04.604 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:11:04.604 DEBUG:boto:Data: 2024-04-12T04:11:04.604 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.605 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.605 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.605 DEBUG:boto:Params: {} 2024-04-12T04:11:04.605 DEBUG:boto:Token: None 2024-04-12T04:11:04.605 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/?versions 2024-04-12T04:11:04.605 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zALnf4JU5P8wyduoNVMHGAd31l0= 2024-04-12T04:11:04.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zALnf4JU5P8wyduoNVMHGAd31l0=', 'Content-Length': '0'} 2024-04-12T04:11:04.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaad109565870e91-006618b458-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.608 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T04:09:24.114Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T04:09:24.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T04:09:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T04:09:24.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:04.610 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.610 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.610 DEBUG:boto:path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:11:04.610 DEBUG:boto:auth_path=/dkyges-61/?versions&max-keys=999 2024-04-12T04:11:04.610 DEBUG:boto:Method: GET 2024-04-12T04:11:04.611 DEBUG:boto:Path: /dkyges-61/?versions&max-keys=999 2024-04-12T04:11:04.611 DEBUG:boto:Data: 2024-04-12T04:11:04.611 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.611 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.611 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.611 DEBUG:boto:Params: {} 2024-04-12T04:11:04.611 DEBUG:boto:Token: None 2024-04-12T04:11:04.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/?versions 2024-04-12T04:11:04.611 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zALnf4JU5P8wyduoNVMHGAd31l0= 2024-04-12T04:11:04.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zALnf4JU5P8wyduoNVMHGAd31l0=', 'Content-Length': '0'} 2024-04-12T04:11:04.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e95e23ada17bda29-006618b458-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.616 DEBUG:boto:b'dkyges-61999falseobj11nulltrue2024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T04:09:24.114Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T04:09:24.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T04:09:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T04:09:24.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:04.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:11:04.618 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:11:04.618 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:11:04.618 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.618 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.618 DEBUG:boto:Method: GET 2024-04-12T04:11:04.619 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.619 DEBUG:boto:Data: 2024-04-12T04:11:04.619 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.619 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.619 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.619 DEBUG:boto:Params: {} 2024-04-12T04:11:04.619 DEBUG:boto:Token: None 2024-04-12T04:11:04.619 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.619 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JA39iWTQX451Z/5lyBkBFe6oRB4= 2024-04-12T04:11:04.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JA39iWTQX451Z/5lyBkBFe6oRB4=', 'Content-Length': '0'} 2024-04-12T04:11:04.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6c95a7c5f84a62b-006618b458-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.666 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:11:04.667 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:11:04.667 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.667 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.667 DEBUG:boto:Method: GET 2024-04-12T04:11:04.667 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.667 DEBUG:boto:Data: 2024-04-12T04:11:04.667 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.667 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.667 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.667 DEBUG:boto:Params: {} 2024-04-12T04:11:04.667 DEBUG:boto:Token: None 2024-04-12T04:11:04.668 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.668 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JA39iWTQX451Z/5lyBkBFe6oRB4= 2024-04-12T04:11:04.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JA39iWTQX451Z/5lyBkBFe6oRB4=', 'Content-Length': '0'} 2024-04-12T04:11:04.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b9dee88acb71a85-006618b458-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.670 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:11:04.670 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:11:04.670 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.670 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.671 DEBUG:boto:Method: HEAD 2024-04-12T04:11:04.671 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.671 DEBUG:boto:Data: 2024-04-12T04:11:04.671 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.671 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.671 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.671 DEBUG:boto:Params: {} 2024-04-12T04:11:04.671 DEBUG:boto:Token: None 2024-04-12T04:11:04.671 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.671 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mWjcV/5PzfXd5GujMsqCb/zgjEg= 2024-04-12T04:11:04.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mWjcV/5PzfXd5GujMsqCb/zgjEg=', 'Content-Length': '0'} 2024-04-12T04:11:04.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b2c2f29d6e973d8-006618b458-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.673 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:11:04.673 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:11:04.673 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.674 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.674 DEBUG:boto:Method: HEAD 2024-04-12T04:11:04.674 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.674 DEBUG:boto:Data: 2024-04-12T04:11:04.674 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.674 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.674 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.674 DEBUG:boto:Params: {} 2024-04-12T04:11:04.674 DEBUG:boto:Token: None 2024-04-12T04:11:04.674 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.675 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mWjcV/5PzfXd5GujMsqCb/zgjEg= 2024-04-12T04:11:04.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mWjcV/5PzfXd5GujMsqCb/zgjEg=', 'Content-Length': '0'} 2024-04-12T04:11:04.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079b14546e3b32097-006618b458-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:11:04.677 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:11:04.677 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:11:04.677 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.677 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.677 DEBUG:boto:Method: GET 2024-04-12T04:11:04.678 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.678 DEBUG:boto:Data: 2024-04-12T04:11:04.678 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.678 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.678 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.678 DEBUG:boto:Params: {} 2024-04-12T04:11:04.678 DEBUG:boto:Token: None 2024-04-12T04:11:04.678 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.678 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JA39iWTQX451Z/5lyBkBFe6oRB4= 2024-04-12T04:11:04.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JA39iWTQX451Z/5lyBkBFe6oRB4=', 'Content-Length': '0'} 2024-04-12T04:11:04.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035d7f3fe12483bf2-006618b458-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.681 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:11:04.681 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:11:04.681 DEBUG:boto:path=/dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.681 DEBUG:boto:auth_path=/dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.681 DEBUG:boto:Method: GET 2024-04-12T04:11:04.681 DEBUG:boto:Path: /dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.681 DEBUG:boto:Data: 2024-04-12T04:11:04.681 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.681 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.682 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.682 DEBUG:boto:Params: {} 2024-04-12T04:11:04.682 DEBUG:boto:Token: None 2024-04-12T04:11:04.682 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj11?versionId=null 2024-04-12T04:11:04.682 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JA39iWTQX451Z/5lyBkBFe6oRB4= 2024-04-12T04:11:04.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JA39iWTQX451Z/5lyBkBFe6oRB4=', 'Content-Length': '0'} 2024-04-12T04:11:04.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000000140f8249bf157-006618b458-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.684 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.684 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.684 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:11:04.684 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:11:04.685 DEBUG:boto:Method: GET 2024-04-12T04:11:04.685 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:11:04.685 DEBUG:boto:Data: 2024-04-12T04:11:04.685 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.685 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.685 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.685 DEBUG:boto:Params: {} 2024-04-12T04:11:04.685 DEBUG:boto:Token: None 2024-04-12T04:11:04.685 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/ 2024-04-12T04:11:04.685 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts= 2024-04-12T04:11:04.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts=', 'Content-Length': '0'} 2024-04-12T04:11:04.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092f3127ddcbc34b2-006618b458-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.688 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:04.688 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.689 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.689 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:11:04.689 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:11:04.689 DEBUG:boto:Method: GET 2024-04-12T04:11:04.689 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:11:04.689 DEBUG:boto:Data: 2024-04-12T04:11:04.689 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.689 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.689 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.689 DEBUG:boto:Params: {} 2024-04-12T04:11:04.690 DEBUG:boto:Token: None 2024-04-12T04:11:04.690 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/ 2024-04-12T04:11:04.690 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts= 2024-04-12T04:11:04.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts=', 'Content-Length': '0'} 2024-04-12T04:11:04.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083c8b8864e743b73-006618b458-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.694 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:04.695 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:11:04.695 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:11:04.695 DEBUG:boto:Method: HEAD 2024-04-12T04:11:04.695 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:11:04.695 DEBUG:boto:Data: 2024-04-12T04:11:04.695 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.695 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.695 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.696 DEBUG:boto:Params: {} 2024-04-12T04:11:04.696 DEBUG:boto:Token: None 2024-04-12T04:11:04.696 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj11 2024-04-12T04:11:04.696 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gIFxr/3Xx2atwKBfyeQSqDqymAc= 2024-04-12T04:11:04.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gIFxr/3Xx2atwKBfyeQSqDqymAc=', 'Content-Length': '0'} 2024-04-12T04:11:04.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051463ae0ee6f8200-006618b458-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.698 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:11:04.698 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:11:04.698 DEBUG:boto:Method: HEAD 2024-04-12T04:11:04.698 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:11:04.698 DEBUG:boto:Data: 2024-04-12T04:11:04.698 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.699 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.699 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.699 DEBUG:boto:Params: {} 2024-04-12T04:11:04.699 DEBUG:boto:Token: None 2024-04-12T04:11:04.699 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj11 2024-04-12T04:11:04.699 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gIFxr/3Xx2atwKBfyeQSqDqymAc= 2024-04-12T04:11:04.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gIFxr/3Xx2atwKBfyeQSqDqymAc=', 'Content-Length': '0'} 2024-04-12T04:11:04.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027a73147048c3f7e-006618b458-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:11:04.701 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:11:04.701 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:11:04.701 DEBUG:boto:Method: GET 2024-04-12T04:11:04.701 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:11:04.701 DEBUG:boto:Data: 2024-04-12T04:11:04.701 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.701 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.702 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.702 DEBUG:boto:Params: {} 2024-04-12T04:11:04.702 DEBUG:boto:Token: None 2024-04-12T04:11:04.702 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj11 2024-04-12T04:11:04.702 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+nIZoIUSnWNgfcvqTH68q8SIa3A= 2024-04-12T04:11:04.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+nIZoIUSnWNgfcvqTH68q8SIa3A=', 'Content-Length': '0'} 2024-04-12T04:11:04.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b0e2b1dd229908b-006618b458-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.704 DEBUG:boto:path=/dkyges-61/obj11 2024-04-12T04:11:04.704 DEBUG:boto:auth_path=/dkyges-61/obj11 2024-04-12T04:11:04.704 DEBUG:boto:Method: GET 2024-04-12T04:11:04.704 DEBUG:boto:Path: /dkyges-61/obj11 2024-04-12T04:11:04.704 DEBUG:boto:Data: 2024-04-12T04:11:04.704 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.705 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.705 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.705 DEBUG:boto:Params: {} 2024-04-12T04:11:04.705 DEBUG:boto:Token: None 2024-04-12T04:11:04.705 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj11 2024-04-12T04:11:04.705 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+nIZoIUSnWNgfcvqTH68q8SIa3A= 2024-04-12T04:11:04.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+nIZoIUSnWNgfcvqTH68q8SIa3A=', 'Content-Length': '0'} 2024-04-12T04:11:04.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000497428ffac8fea4b-006618b458-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.707 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.707 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.707 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:11:04.707 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:11:04.707 DEBUG:boto:Method: GET 2024-04-12T04:11:04.708 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:11:04.708 DEBUG:boto:Data: 2024-04-12T04:11:04.708 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.708 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.708 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.708 DEBUG:boto:Params: {} 2024-04-12T04:11:04.708 DEBUG:boto:Token: None 2024-04-12T04:11:04.708 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/ 2024-04-12T04:11:04.708 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts= 2024-04-12T04:11:04.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts=', 'Content-Length': '0'} 2024-04-12T04:11:04.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003812613c9d12cdcc-006618b458-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.711 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:04.711 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.711 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.712 DEBUG:boto:path=/dkyges-61/?prefix=obj11 2024-04-12T04:11:04.712 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj11 2024-04-12T04:11:04.712 DEBUG:boto:Method: GET 2024-04-12T04:11:04.712 DEBUG:boto:Path: /dkyges-61/?prefix=obj11 2024-04-12T04:11:04.712 DEBUG:boto:Data: 2024-04-12T04:11:04.712 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.712 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.712 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.712 DEBUG:boto:Params: {} 2024-04-12T04:11:04.712 DEBUG:boto:Token: None 2024-04-12T04:11:04.712 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/ 2024-04-12T04:11:04.713 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts= 2024-04-12T04:11:04.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts=', 'Content-Length': '0'} 2024-04-12T04:11:04.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa3f9b946303c940-006618b458-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.717 DEBUG:boto:b'dkyges-61obj111000falseobj112024-04-12T04:00:30.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:04.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T04:11:04.718 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:11:04.718 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:11:04.718 DEBUG:boto:path=/dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.718 DEBUG:boto:auth_path=/dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.718 DEBUG:boto:Method: GET 2024-04-12T04:11:04.718 DEBUG:boto:Path: /dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.718 DEBUG:boto:Data: 2024-04-12T04:11:04.718 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.719 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.719 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.719 DEBUG:boto:Params: {} 2024-04-12T04:11:04.719 DEBUG:boto:Token: None 2024-04-12T04:11:04.719 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.719 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f9A9YYzXrkIYkdQaPc0EOfEsUmw= 2024-04-12T04:11:04.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f9A9YYzXrkIYkdQaPc0EOfEsUmw=', 'Content-Length': '0'} 2024-04-12T04:11:04.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4d2986e88e468b3-006618b458-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.762 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:11:04.763 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:11:04.763 DEBUG:boto:path=/dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.763 DEBUG:boto:auth_path=/dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.763 DEBUG:boto:Method: GET 2024-04-12T04:11:04.763 DEBUG:boto:Path: /dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.763 DEBUG:boto:Data: 2024-04-12T04:11:04.763 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.763 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.763 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.763 DEBUG:boto:Params: {} 2024-04-12T04:11:04.764 DEBUG:boto:Token: None 2024-04-12T04:11:04.764 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.764 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f9A9YYzXrkIYkdQaPc0EOfEsUmw= 2024-04-12T04:11:04.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f9A9YYzXrkIYkdQaPc0EOfEsUmw=', 'Content-Length': '0'} 2024-04-12T04:11:04.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cb23e1be14e96c4-006618b458-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.766 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:11:04.766 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:11:04.766 DEBUG:boto:path=/dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.766 DEBUG:boto:auth_path=/dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.767 DEBUG:boto:Method: HEAD 2024-04-12T04:11:04.767 DEBUG:boto:Path: /dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.767 DEBUG:boto:Data: 2024-04-12T04:11:04.767 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.767 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.767 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.767 DEBUG:boto:Params: {} 2024-04-12T04:11:04.767 DEBUG:boto:Token: None 2024-04-12T04:11:04.767 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.767 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BSZWgRisMthkNOvWVQ9Oiu5hUWE= 2024-04-12T04:11:04.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BSZWgRisMthkNOvWVQ9Oiu5hUWE=', 'Content-Length': '0'} 2024-04-12T04:11:04.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff3be6a56e5b3447-006618b458-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.769 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:11:04.769 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:11:04.770 DEBUG:boto:path=/dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.770 DEBUG:boto:auth_path=/dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.770 DEBUG:boto:Method: HEAD 2024-04-12T04:11:04.770 DEBUG:boto:Path: /dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.770 DEBUG:boto:Data: 2024-04-12T04:11:04.770 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.770 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.770 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.770 DEBUG:boto:Params: {} 2024-04-12T04:11:04.770 DEBUG:boto:Token: None 2024-04-12T04:11:04.770 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.771 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BSZWgRisMthkNOvWVQ9Oiu5hUWE= 2024-04-12T04:11:04.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BSZWgRisMthkNOvWVQ9Oiu5hUWE=', 'Content-Length': '0'} 2024-04-12T04:11:04.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f5596e29af59179-006618b458-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T04:11:04.773 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:11:04.773 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:11:04.773 DEBUG:boto:path=/dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.773 DEBUG:boto:auth_path=/dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.773 DEBUG:boto:Method: GET 2024-04-12T04:11:04.773 DEBUG:boto:Path: /dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.773 DEBUG:boto:Data: 2024-04-12T04:11:04.773 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.773 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.773 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.773 DEBUG:boto:Params: {} 2024-04-12T04:11:04.774 DEBUG:boto:Token: None 2024-04-12T04:11:04.774 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.774 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f9A9YYzXrkIYkdQaPc0EOfEsUmw= 2024-04-12T04:11:04.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f9A9YYzXrkIYkdQaPc0EOfEsUmw=', 'Content-Length': '0'} 2024-04-12T04:11:04.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ac98bbb60c4ba6b-006618b458-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.776 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:11:04.776 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:11:04.776 DEBUG:boto:path=/dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.777 DEBUG:boto:auth_path=/dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.777 DEBUG:boto:Method: GET 2024-04-12T04:11:04.777 DEBUG:boto:Path: /dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.777 DEBUG:boto:Data: 2024-04-12T04:11:04.777 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.777 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.777 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.777 DEBUG:boto:Params: {} 2024-04-12T04:11:04.777 DEBUG:boto:Token: None 2024-04-12T04:11:04.777 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj111?versionId=null 2024-04-12T04:11:04.777 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f9A9YYzXrkIYkdQaPc0EOfEsUmw= 2024-04-12T04:11:04.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f9A9YYzXrkIYkdQaPc0EOfEsUmw=', 'Content-Length': '0'} 2024-04-12T04:11:04.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce0ead6a92ff0422-006618b458-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.780 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.780 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.780 DEBUG:boto:path=/dkyges-61/?prefix=obj111 2024-04-12T04:11:04.780 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj111 2024-04-12T04:11:04.781 DEBUG:boto:Method: GET 2024-04-12T04:11:04.781 DEBUG:boto:Path: /dkyges-61/?prefix=obj111 2024-04-12T04:11:04.781 DEBUG:boto:Data: 2024-04-12T04:11:04.781 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.781 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.781 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.782 DEBUG:boto:Params: {} 2024-04-12T04:11:04.782 DEBUG:boto:Token: None 2024-04-12T04:11:04.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/ 2024-04-12T04:11:04.782 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts= 2024-04-12T04:11:04.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts=', 'Content-Length': '0'} 2024-04-12T04:11:04.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fc279ce128cf7f4-006618b458-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.785 DEBUG:boto:b'dkyges-61obj1111000falseobj1112024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:04.785 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.785 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.785 DEBUG:boto:path=/dkyges-61/?prefix=obj111 2024-04-12T04:11:04.785 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj111 2024-04-12T04:11:04.786 DEBUG:boto:Method: GET 2024-04-12T04:11:04.786 DEBUG:boto:Path: /dkyges-61/?prefix=obj111 2024-04-12T04:11:04.786 DEBUG:boto:Data: 2024-04-12T04:11:04.786 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.786 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.786 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.786 DEBUG:boto:Params: {} 2024-04-12T04:11:04.786 DEBUG:boto:Token: None 2024-04-12T04:11:04.786 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/ 2024-04-12T04:11:04.786 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts= 2024-04-12T04:11:04.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts=', 'Content-Length': '0'} 2024-04-12T04:11:04.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea2607bea4f87ae0-006618b458-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.834 DEBUG:boto:b'dkyges-61obj1111000falseobj1112024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:04.835 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:11:04.835 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:11:04.835 DEBUG:boto:Method: HEAD 2024-04-12T04:11:04.835 DEBUG:boto:Path: /dkyges-61/obj111 2024-04-12T04:11:04.835 DEBUG:boto:Data: 2024-04-12T04:11:04.835 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.835 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.835 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.836 DEBUG:boto:Params: {} 2024-04-12T04:11:04.836 DEBUG:boto:Token: None 2024-04-12T04:11:04.836 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj111 2024-04-12T04:11:04.836 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xDmgZ220GWl3IWHmQw8UKaR5rgg= 2024-04-12T04:11:04.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xDmgZ220GWl3IWHmQw8UKaR5rgg=', 'Content-Length': '0'} 2024-04-12T04:11:04.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014d41c300fbd0ce3-006618b458-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.838 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:11:04.839 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:11:04.839 DEBUG:boto:Method: HEAD 2024-04-12T04:11:04.839 DEBUG:boto:Path: /dkyges-61/obj111 2024-04-12T04:11:04.839 DEBUG:boto:Data: 2024-04-12T04:11:04.839 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.839 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.839 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.839 DEBUG:boto:Params: {} 2024-04-12T04:11:04.839 DEBUG:boto:Token: None 2024-04-12T04:11:04.840 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj111 2024-04-12T04:11:04.840 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xDmgZ220GWl3IWHmQw8UKaR5rgg= 2024-04-12T04:11:04.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xDmgZ220GWl3IWHmQw8UKaR5rgg=', 'Content-Length': '0'} 2024-04-12T04:11:04.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0e04aca671f073f-006618b458-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T04:11:04.842 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:11:04.842 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:11:04.843 DEBUG:boto:Method: GET 2024-04-12T04:11:04.843 DEBUG:boto:Path: /dkyges-61/obj111 2024-04-12T04:11:04.843 DEBUG:boto:Data: 2024-04-12T04:11:04.843 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.843 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.843 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.843 DEBUG:boto:Params: {} 2024-04-12T04:11:04.843 DEBUG:boto:Token: None 2024-04-12T04:11:04.843 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj111 2024-04-12T04:11:04.843 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WW24vXIaeJ0J6cbd6osNwFbGRq8= 2024-04-12T04:11:04.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WW24vXIaeJ0J6cbd6osNwFbGRq8=', 'Content-Length': '0'} 2024-04-12T04:11:04.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a37d6bf870a6b2c-006618b458-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.846 DEBUG:boto:path=/dkyges-61/obj111 2024-04-12T04:11:04.846 DEBUG:boto:auth_path=/dkyges-61/obj111 2024-04-12T04:11:04.846 DEBUG:boto:Method: GET 2024-04-12T04:11:04.846 DEBUG:boto:Path: /dkyges-61/obj111 2024-04-12T04:11:04.846 DEBUG:boto:Data: 2024-04-12T04:11:04.846 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.846 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.847 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.847 DEBUG:boto:Params: {} 2024-04-12T04:11:04.847 DEBUG:boto:Token: None 2024-04-12T04:11:04.847 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj111 2024-04-12T04:11:04.847 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WW24vXIaeJ0J6cbd6osNwFbGRq8= 2024-04-12T04:11:04.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WW24vXIaeJ0J6cbd6osNwFbGRq8=', 'Content-Length': '0'} 2024-04-12T04:11:04.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000387ce298bc0e8877-006618b458-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.850 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.850 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.850 DEBUG:boto:path=/dkyges-61/?prefix=obj111 2024-04-12T04:11:04.851 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj111 2024-04-12T04:11:04.851 DEBUG:boto:Method: GET 2024-04-12T04:11:04.851 DEBUG:boto:Path: /dkyges-61/?prefix=obj111 2024-04-12T04:11:04.851 DEBUG:boto:Data: 2024-04-12T04:11:04.851 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.851 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.851 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.851 DEBUG:boto:Params: {} 2024-04-12T04:11:04.851 DEBUG:boto:Token: None 2024-04-12T04:11:04.851 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/ 2024-04-12T04:11:04.852 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts= 2024-04-12T04:11:04.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts=', 'Content-Length': '0'} 2024-04-12T04:11:04.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea86a257b001767d-006618b458-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.854 DEBUG:boto:b'dkyges-61obj1111000falseobj1112024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:04.855 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.855 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.855 DEBUG:boto:path=/dkyges-61/?prefix=obj111 2024-04-12T04:11:04.855 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj111 2024-04-12T04:11:04.855 DEBUG:boto:Method: GET 2024-04-12T04:11:04.855 DEBUG:boto:Path: /dkyges-61/?prefix=obj111 2024-04-12T04:11:04.855 DEBUG:boto:Data: 2024-04-12T04:11:04.856 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.856 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.856 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.856 DEBUG:boto:Params: {} 2024-04-12T04:11:04.856 DEBUG:boto:Token: None 2024-04-12T04:11:04.856 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/ 2024-04-12T04:11:04.856 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts= 2024-04-12T04:11:04.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts=', 'Content-Length': '0'} 2024-04-12T04:11:04.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000922d05fccb16939d-006618b458-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.861 DEBUG:boto:b'dkyges-61obj1111000falseobj1112024-04-12T04:09:24.100Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:04.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:11:04.861 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:11:04.861 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:11:04.861 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.861 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.861 DEBUG:boto:Method: GET 2024-04-12T04:11:04.861 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.862 DEBUG:boto:Data: 2024-04-12T04:11:04.862 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.862 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.862 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.862 DEBUG:boto:Params: {} 2024-04-12T04:11:04.862 DEBUG:boto:Token: None 2024-04-12T04:11:04.862 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.862 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CCSbnXEMrP/odIs5XhvyLgMfU/4= 2024-04-12T04:11:04.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CCSbnXEMrP/odIs5XhvyLgMfU/4=', 'Content-Length': '0'} 2024-04-12T04:11:04.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019dd518a3ab3cdb7-006618b458-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.865 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:11:04.865 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:11:04.865 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.865 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.865 DEBUG:boto:Method: GET 2024-04-12T04:11:04.865 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.865 DEBUG:boto:Data: 2024-04-12T04:11:04.865 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.865 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.865 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.866 DEBUG:boto:Params: {} 2024-04-12T04:11:04.866 DEBUG:boto:Token: None 2024-04-12T04:11:04.866 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.866 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CCSbnXEMrP/odIs5XhvyLgMfU/4= 2024-04-12T04:11:04.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CCSbnXEMrP/odIs5XhvyLgMfU/4=', 'Content-Length': '0'} 2024-04-12T04:11:04.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072290d01018702a1-006618b458-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.868 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:11:04.868 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:11:04.869 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.869 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.869 DEBUG:boto:Method: HEAD 2024-04-12T04:11:04.869 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.869 DEBUG:boto:Data: 2024-04-12T04:11:04.869 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.869 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.869 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.869 DEBUG:boto:Params: {} 2024-04-12T04:11:04.869 DEBUG:boto:Token: None 2024-04-12T04:11:04.870 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:actJbpvuwzDTZPhA67JCSXWQyrI= 2024-04-12T04:11:04.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:actJbpvuwzDTZPhA67JCSXWQyrI=', 'Content-Length': '0'} 2024-04-12T04:11:04.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc7bc73692bc2d8c-006618b458-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.872 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:11:04.872 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:11:04.872 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.872 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.872 DEBUG:boto:Method: HEAD 2024-04-12T04:11:04.872 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.872 DEBUG:boto:Data: 2024-04-12T04:11:04.872 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.872 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.873 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.873 DEBUG:boto:Params: {} 2024-04-12T04:11:04.873 DEBUG:boto:Token: None 2024-04-12T04:11:04.873 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.873 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:actJbpvuwzDTZPhA67JCSXWQyrI= 2024-04-12T04:11:04.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:actJbpvuwzDTZPhA67JCSXWQyrI=', 'Content-Length': '0'} 2024-04-12T04:11:04.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca44dade33ea3e22-006618b458-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:11:04.875 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:11:04.875 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:11:04.875 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.875 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.876 DEBUG:boto:Method: GET 2024-04-12T04:11:04.876 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.876 DEBUG:boto:Data: 2024-04-12T04:11:04.876 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.876 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.876 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.876 DEBUG:boto:Params: {} 2024-04-12T04:11:04.876 DEBUG:boto:Token: None 2024-04-12T04:11:04.876 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.877 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CCSbnXEMrP/odIs5XhvyLgMfU/4= 2024-04-12T04:11:04.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CCSbnXEMrP/odIs5XhvyLgMfU/4=', 'Content-Length': '0'} 2024-04-12T04:11:04.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a89bb21b69284896-006618b458-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.879 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:11:04.879 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:11:04.879 DEBUG:boto:path=/dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.879 DEBUG:boto:auth_path=/dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.879 DEBUG:boto:Method: GET 2024-04-12T04:11:04.879 DEBUG:boto:Path: /dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.880 DEBUG:boto:Data: 2024-04-12T04:11:04.880 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.880 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.880 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.880 DEBUG:boto:Params: {} 2024-04-12T04:11:04.880 DEBUG:boto:Token: None 2024-04-12T04:11:04.880 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj115?versionId=null 2024-04-12T04:11:04.880 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CCSbnXEMrP/odIs5XhvyLgMfU/4= 2024-04-12T04:11:04.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CCSbnXEMrP/odIs5XhvyLgMfU/4=', 'Content-Length': '0'} 2024-04-12T04:11:04.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf564760f57a167d-006618b458-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.926 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.927 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.927 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:11:04.927 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:11:04.927 DEBUG:boto:Method: GET 2024-04-12T04:11:04.927 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:11:04.927 DEBUG:boto:Data: 2024-04-12T04:11:04.927 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.928 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.928 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.928 DEBUG:boto:Params: {} 2024-04-12T04:11:04.928 DEBUG:boto:Token: None 2024-04-12T04:11:04.928 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/ 2024-04-12T04:11:04.928 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts= 2024-04-12T04:11:04.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts=', 'Content-Length': '0'} 2024-04-12T04:11:04.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005d310bcb093939c-006618b458-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.931 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:04.932 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.932 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.932 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:11:04.932 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:11:04.932 DEBUG:boto:Method: GET 2024-04-12T04:11:04.933 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:11:04.933 DEBUG:boto:Data: 2024-04-12T04:11:04.933 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.933 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.933 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.933 DEBUG:boto:Params: {} 2024-04-12T04:11:04.933 DEBUG:boto:Token: None 2024-04-12T04:11:04.933 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/ 2024-04-12T04:11:04.933 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts= 2024-04-12T04:11:04.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts=', 'Content-Length': '0'} 2024-04-12T04:11:04.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000676830fbb5308d82-006618b458-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.938 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:04.939 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:11:04.939 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:11:04.939 DEBUG:boto:Method: HEAD 2024-04-12T04:11:04.939 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:11:04.939 DEBUG:boto:Data: 2024-04-12T04:11:04.939 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.939 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.939 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.939 DEBUG:boto:Params: {} 2024-04-12T04:11:04.939 DEBUG:boto:Token: None 2024-04-12T04:11:04.939 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj115 2024-04-12T04:11:04.940 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AUqqUmv6CfSHGBQtG8V8f7ZzPU8= 2024-04-12T04:11:04.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AUqqUmv6CfSHGBQtG8V8f7ZzPU8=', 'Content-Length': '0'} 2024-04-12T04:11:04.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f5f3a9e0c86d19e-006618b458-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.942 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:11:04.942 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:11:04.942 DEBUG:boto:Method: HEAD 2024-04-12T04:11:04.942 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:11:04.942 DEBUG:boto:Data: 2024-04-12T04:11:04.942 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.942 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.942 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.942 DEBUG:boto:Params: {} 2024-04-12T04:11:04.942 DEBUG:boto:Token: None 2024-04-12T04:11:04.943 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj115 2024-04-12T04:11:04.943 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AUqqUmv6CfSHGBQtG8V8f7ZzPU8= 2024-04-12T04:11:04.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AUqqUmv6CfSHGBQtG8V8f7ZzPU8=', 'Content-Length': '0'} 2024-04-12T04:11:04.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0ba10184c08f9a4-006618b458-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:11:04.945 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:11:04.945 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:11:04.945 DEBUG:boto:Method: GET 2024-04-12T04:11:04.945 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:11:04.945 DEBUG:boto:Data: 2024-04-12T04:11:04.945 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.945 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.946 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.946 DEBUG:boto:Params: {} 2024-04-12T04:11:04.946 DEBUG:boto:Token: None 2024-04-12T04:11:04.946 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj115 2024-04-12T04:11:04.946 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cvuDLYLBF1PUsgyDkJxxIoiJnwI= 2024-04-12T04:11:04.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cvuDLYLBF1PUsgyDkJxxIoiJnwI=', 'Content-Length': '0'} 2024-04-12T04:11:04.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c5b816e65af43c4-006618b458-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.990 DEBUG:boto:path=/dkyges-61/obj115 2024-04-12T04:11:04.991 DEBUG:boto:auth_path=/dkyges-61/obj115 2024-04-12T04:11:04.991 DEBUG:boto:Method: GET 2024-04-12T04:11:04.991 DEBUG:boto:Path: /dkyges-61/obj115 2024-04-12T04:11:04.991 DEBUG:boto:Data: 2024-04-12T04:11:04.991 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.991 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.991 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.991 DEBUG:boto:Params: {} 2024-04-12T04:11:04.991 DEBUG:boto:Token: None 2024-04-12T04:11:04.991 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/obj115 2024-04-12T04:11:04.992 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cvuDLYLBF1PUsgyDkJxxIoiJnwI= 2024-04-12T04:11:04.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cvuDLYLBF1PUsgyDkJxxIoiJnwI=', 'Content-Length': '0'} 2024-04-12T04:11:04.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bd4d23213160f4d-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.994 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.994 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.994 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:11:04.994 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:11:04.994 DEBUG:boto:Method: GET 2024-04-12T04:11:04.994 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:11:04.995 DEBUG:boto:Data: 2024-04-12T04:11:04.995 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.995 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:04.995 DEBUG:boto:Port: 8001 2024-04-12T04:11:04.995 DEBUG:boto:Params: {} 2024-04-12T04:11:04.995 DEBUG:boto:Token: None 2024-04-12T04:11:04.995 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/ 2024-04-12T04:11:04.995 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts= 2024-04-12T04:11:04.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts=', 'Content-Length': '0'} 2024-04-12T04:11:04.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d4efc25c3dc62a3-006618b459-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:04.998 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:04.998 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:04.998 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:04.999 DEBUG:boto:path=/dkyges-61/?prefix=obj115 2024-04-12T04:11:04.999 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj115 2024-04-12T04:11:04.999 DEBUG:boto:Method: GET 2024-04-12T04:11:04.999 DEBUG:boto:Path: /dkyges-61/?prefix=obj115 2024-04-12T04:11:04.999 DEBUG:boto:Data: 2024-04-12T04:11:04.999 DEBUG:boto:Headers: {} 2024-04-12T04:11:04.999 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:04.999 DEBUG:boto:Port: 8000 2024-04-12T04:11:04.999 DEBUG:boto:Params: {} 2024-04-12T04:11:04.999 DEBUG:boto:Token: None 2024-04-12T04:11:04.999 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:04 GMT /dkyges-61/ 2024-04-12T04:11:04.999 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts= 2024-04-12T04:11:05.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tPaK4TDnNsU2wWwIp5G6HV1hUts=', 'Content-Length': '0'} 2024-04-12T04:11:05.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b3060ad5d768141-006618b459-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.004 DEBUG:boto:b'dkyges-61obj1151000falseobj1152024-04-12T04:05:01.190Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:11:05.005 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:11:05.005 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:11:05.005 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.005 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.005 DEBUG:boto:Method: GET 2024-04-12T04:11:05.005 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.005 DEBUG:boto:Data: 2024-04-12T04:11:05.005 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.005 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.005 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.005 DEBUG:boto:Params: {} 2024-04-12T04:11:05.005 DEBUG:boto:Token: None 2024-04-12T04:11:05.006 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.006 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:E4h+yv7xW+Cfugit3Ccsdq29GZc= 2024-04-12T04:11:05.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:E4h+yv7xW+Cfugit3Ccsdq29GZc=', 'Content-Length': '0'} 2024-04-12T04:11:05.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae82e5c06d6aab56-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.008 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:11:05.008 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:11:05.009 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.009 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.009 DEBUG:boto:Method: GET 2024-04-12T04:11:05.009 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.009 DEBUG:boto:Data: 2024-04-12T04:11:05.009 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.009 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.009 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.009 DEBUG:boto:Params: {} 2024-04-12T04:11:05.009 DEBUG:boto:Token: None 2024-04-12T04:11:05.009 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.010 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:E4h+yv7xW+Cfugit3Ccsdq29GZc= 2024-04-12T04:11:05.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:E4h+yv7xW+Cfugit3Ccsdq29GZc=', 'Content-Length': '0'} 2024-04-12T04:11:05.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd847b07b343c9f6-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.054 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:11:05.055 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:11:05.055 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.055 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.055 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.055 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.055 DEBUG:boto:Data: 2024-04-12T04:11:05.055 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.055 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.055 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.055 DEBUG:boto:Params: {} 2024-04-12T04:11:05.056 DEBUG:boto:Token: None 2024-04-12T04:11:05.056 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.056 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v1ji1qfmy77y+3aG30VhkE1vKaU= 2024-04-12T04:11:05.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v1ji1qfmy77y+3aG30VhkE1vKaU=', 'Content-Length': '0'} 2024-04-12T04:11:05.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082936fadb0c66221-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.058 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:11:05.058 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:11:05.058 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.058 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.058 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.059 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.059 DEBUG:boto:Data: 2024-04-12T04:11:05.059 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.059 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.059 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.059 DEBUG:boto:Params: {} 2024-04-12T04:11:05.059 DEBUG:boto:Token: None 2024-04-12T04:11:05.059 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.059 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v1ji1qfmy77y+3aG30VhkE1vKaU= 2024-04-12T04:11:05.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v1ji1qfmy77y+3aG30VhkE1vKaU=', 'Content-Length': '0'} 2024-04-12T04:11:05.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049046e215fa44bad-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:11:05.061 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:11:05.062 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:11:05.062 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.062 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.062 DEBUG:boto:Method: GET 2024-04-12T04:11:05.062 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.062 DEBUG:boto:Data: 2024-04-12T04:11:05.062 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.062 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.062 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.062 DEBUG:boto:Params: {} 2024-04-12T04:11:05.062 DEBUG:boto:Token: None 2024-04-12T04:11:05.063 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.063 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:E4h+yv7xW+Cfugit3Ccsdq29GZc= 2024-04-12T04:11:05.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:E4h+yv7xW+Cfugit3Ccsdq29GZc=', 'Content-Length': '0'} 2024-04-12T04:11:05.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ef535df4d7448b1-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.065 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:11:05.065 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:11:05.065 DEBUG:boto:path=/dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.065 DEBUG:boto:auth_path=/dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.066 DEBUG:boto:Method: GET 2024-04-12T04:11:05.066 DEBUG:boto:Path: /dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.066 DEBUG:boto:Data: 2024-04-12T04:11:05.066 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.066 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.066 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.066 DEBUG:boto:Params: {} 2024-04-12T04:11:05.066 DEBUG:boto:Token: None 2024-04-12T04:11:05.066 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj116?versionId=null 2024-04-12T04:11:05.067 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:E4h+yv7xW+Cfugit3Ccsdq29GZc= 2024-04-12T04:11:05.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:E4h+yv7xW+Cfugit3Ccsdq29GZc=', 'Content-Length': '0'} 2024-04-12T04:11:05.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad4d24e512309825-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.069 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.069 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.069 DEBUG:boto:path=/dkyges-61/?prefix=obj116 2024-04-12T04:11:05.069 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj116 2024-04-12T04:11:05.069 DEBUG:boto:Method: GET 2024-04-12T04:11:05.069 DEBUG:boto:Path: /dkyges-61/?prefix=obj116 2024-04-12T04:11:05.069 DEBUG:boto:Data: 2024-04-12T04:11:05.069 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.069 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.069 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.070 DEBUG:boto:Params: {} 2024-04-12T04:11:05.070 DEBUG:boto:Token: None 2024-04-12T04:11:05.070 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.070 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033d7b34a5d19ab82-006618b459-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.072 DEBUG:boto:b'dkyges-61obj1161000falseobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.073 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.073 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.073 DEBUG:boto:path=/dkyges-61/?prefix=obj116 2024-04-12T04:11:05.073 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj116 2024-04-12T04:11:05.073 DEBUG:boto:Method: GET 2024-04-12T04:11:05.073 DEBUG:boto:Path: /dkyges-61/?prefix=obj116 2024-04-12T04:11:05.073 DEBUG:boto:Data: 2024-04-12T04:11:05.073 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.074 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.074 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.074 DEBUG:boto:Params: {} 2024-04-12T04:11:05.074 DEBUG:boto:Token: None 2024-04-12T04:11:05.074 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.074 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e05b86ffb09f700-006618b459-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.079 DEBUG:boto:b'dkyges-61obj1161000falseobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.079 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:11:05.079 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:11:05.079 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.079 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:11:05.080 DEBUG:boto:Data: 2024-04-12T04:11:05.080 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.080 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.080 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.080 DEBUG:boto:Params: {} 2024-04-12T04:11:05.080 DEBUG:boto:Token: None 2024-04-12T04:11:05.080 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj116 2024-04-12T04:11:05.080 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aLBNZFOu247xjdxi5rLCbB18QrE= 2024-04-12T04:11:05.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aLBNZFOu247xjdxi5rLCbB18QrE=', 'Content-Length': '0'} 2024-04-12T04:11:05.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff6bb90ff4da86c1-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.082 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:11:05.083 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:11:05.083 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.083 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:11:05.083 DEBUG:boto:Data: 2024-04-12T04:11:05.083 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.083 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.083 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.083 DEBUG:boto:Params: {} 2024-04-12T04:11:05.083 DEBUG:boto:Token: None 2024-04-12T04:11:05.083 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj116 2024-04-12T04:11:05.083 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aLBNZFOu247xjdxi5rLCbB18QrE= 2024-04-12T04:11:05.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aLBNZFOu247xjdxi5rLCbB18QrE=', 'Content-Length': '0'} 2024-04-12T04:11:05.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065a4fc32cec06e99-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:11:05.086 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:11:05.086 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:11:05.086 DEBUG:boto:Method: GET 2024-04-12T04:11:05.086 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:11:05.086 DEBUG:boto:Data: 2024-04-12T04:11:05.086 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.086 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.086 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.086 DEBUG:boto:Params: {} 2024-04-12T04:11:05.087 DEBUG:boto:Token: None 2024-04-12T04:11:05.087 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj116 2024-04-12T04:11:05.087 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2KBpssiH+hSL5Q6kVMCS8D4kG5Q= 2024-04-12T04:11:05.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2KBpssiH+hSL5Q6kVMCS8D4kG5Q=', 'Content-Length': '0'} 2024-04-12T04:11:05.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b57aca0acae4cf85-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.089 DEBUG:boto:path=/dkyges-61/obj116 2024-04-12T04:11:05.089 DEBUG:boto:auth_path=/dkyges-61/obj116 2024-04-12T04:11:05.089 DEBUG:boto:Method: GET 2024-04-12T04:11:05.089 DEBUG:boto:Path: /dkyges-61/obj116 2024-04-12T04:11:05.089 DEBUG:boto:Data: 2024-04-12T04:11:05.089 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.089 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.089 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.090 DEBUG:boto:Params: {} 2024-04-12T04:11:05.090 DEBUG:boto:Token: None 2024-04-12T04:11:05.090 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj116 2024-04-12T04:11:05.090 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2KBpssiH+hSL5Q6kVMCS8D4kG5Q= 2024-04-12T04:11:05.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2KBpssiH+hSL5Q6kVMCS8D4kG5Q=', 'Content-Length': '0'} 2024-04-12T04:11:05.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062843ec8a65c6cc0-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.092 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.092 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.092 DEBUG:boto:path=/dkyges-61/?prefix=obj116 2024-04-12T04:11:05.092 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj116 2024-04-12T04:11:05.092 DEBUG:boto:Method: GET 2024-04-12T04:11:05.092 DEBUG:boto:Path: /dkyges-61/?prefix=obj116 2024-04-12T04:11:05.092 DEBUG:boto:Data: 2024-04-12T04:11:05.093 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.093 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.093 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.093 DEBUG:boto:Params: {} 2024-04-12T04:11:05.093 DEBUG:boto:Token: None 2024-04-12T04:11:05.093 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.093 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fac93675750e425-006618b459-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.097 DEBUG:boto:b'dkyges-61obj1161000falseobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.097 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.097 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.097 DEBUG:boto:path=/dkyges-61/?prefix=obj116 2024-04-12T04:11:05.097 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj116 2024-04-12T04:11:05.097 DEBUG:boto:Method: GET 2024-04-12T04:11:05.098 DEBUG:boto:Path: /dkyges-61/?prefix=obj116 2024-04-12T04:11:05.098 DEBUG:boto:Data: 2024-04-12T04:11:05.098 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.098 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.098 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.098 DEBUG:boto:Params: {} 2024-04-12T04:11:05.098 DEBUG:boto:Token: None 2024-04-12T04:11:05.098 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.098 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.103 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055e2ab62513a50aa-006618b459-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.103 DEBUG:boto:b'dkyges-61obj1161000falseobj1162024-04-12T04:06:49.199Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T04:11:05.103 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:11:05.103 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:11:05.103 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.104 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.104 DEBUG:boto:Method: GET 2024-04-12T04:11:05.104 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.104 DEBUG:boto:Data: 2024-04-12T04:11:05.104 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.104 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.104 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.104 DEBUG:boto:Params: {} 2024-04-12T04:11:05.104 DEBUG:boto:Token: None 2024-04-12T04:11:05.104 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.104 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YpAwbiLGMauJzoxMbIEhk+eMuWk= 2024-04-12T04:11:05.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YpAwbiLGMauJzoxMbIEhk+eMuWk=', 'Content-Length': '0'} 2024-04-12T04:11:05.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab7eee6386b0905c-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.107 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:11:05.107 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:11:05.107 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.108 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.108 DEBUG:boto:Method: GET 2024-04-12T04:11:05.108 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.108 DEBUG:boto:Data: 2024-04-12T04:11:05.108 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.108 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.108 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.108 DEBUG:boto:Params: {} 2024-04-12T04:11:05.108 DEBUG:boto:Token: None 2024-04-12T04:11:05.108 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.108 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YpAwbiLGMauJzoxMbIEhk+eMuWk= 2024-04-12T04:11:05.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YpAwbiLGMauJzoxMbIEhk+eMuWk=', 'Content-Length': '0'} 2024-04-12T04:11:05.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f5ce2c278e57251-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.154 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:11:05.155 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:11:05.155 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.155 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.155 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.155 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.155 DEBUG:boto:Data: 2024-04-12T04:11:05.156 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.156 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.156 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.156 DEBUG:boto:Params: {} 2024-04-12T04:11:05.156 DEBUG:boto:Token: None 2024-04-12T04:11:05.157 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.157 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j+p2a+Uoe/L9JMpO8pBpFNHJ3+s= 2024-04-12T04:11:05.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j+p2a+Uoe/L9JMpO8pBpFNHJ3+s=', 'Content-Length': '0'} 2024-04-12T04:11:05.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000babaeac2a02c7218-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.159 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:11:05.159 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:11:05.159 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.159 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.159 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.160 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.160 DEBUG:boto:Data: 2024-04-12T04:11:05.160 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.160 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.160 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.160 DEBUG:boto:Params: {} 2024-04-12T04:11:05.160 DEBUG:boto:Token: None 2024-04-12T04:11:05.160 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.160 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j+p2a+Uoe/L9JMpO8pBpFNHJ3+s= 2024-04-12T04:11:05.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j+p2a+Uoe/L9JMpO8pBpFNHJ3+s=', 'Content-Length': '0'} 2024-04-12T04:11:05.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af088dbef90476f0-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T04:11:05.163 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:11:05.163 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:11:05.163 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.163 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.163 DEBUG:boto:Method: GET 2024-04-12T04:11:05.163 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.163 DEBUG:boto:Data: 2024-04-12T04:11:05.163 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.163 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.164 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.164 DEBUG:boto:Params: {} 2024-04-12T04:11:05.164 DEBUG:boto:Token: None 2024-04-12T04:11:05.164 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.164 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YpAwbiLGMauJzoxMbIEhk+eMuWk= 2024-04-12T04:11:05.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YpAwbiLGMauJzoxMbIEhk+eMuWk=', 'Content-Length': '0'} 2024-04-12T04:11:05.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009888ea2da76ba34d-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.206 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:11:05.207 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:11:05.207 DEBUG:boto:path=/dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.207 DEBUG:boto:auth_path=/dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.207 DEBUG:boto:Method: GET 2024-04-12T04:11:05.207 DEBUG:boto:Path: /dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.207 DEBUG:boto:Data: 2024-04-12T04:11:05.207 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.207 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.208 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.208 DEBUG:boto:Params: {} 2024-04-12T04:11:05.208 DEBUG:boto:Token: None 2024-04-12T04:11:05.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj124?versionId=null 2024-04-12T04:11:05.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YpAwbiLGMauJzoxMbIEhk+eMuWk= 2024-04-12T04:11:05.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YpAwbiLGMauJzoxMbIEhk+eMuWk=', 'Content-Length': '0'} 2024-04-12T04:11:05.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d488206b180a1b7-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.211 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.211 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.211 DEBUG:boto:path=/dkyges-61/?prefix=obj124 2024-04-12T04:11:05.212 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj124 2024-04-12T04:11:05.212 DEBUG:boto:Method: GET 2024-04-12T04:11:05.212 DEBUG:boto:Path: /dkyges-61/?prefix=obj124 2024-04-12T04:11:05.212 DEBUG:boto:Data: 2024-04-12T04:11:05.212 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.212 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.212 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.212 DEBUG:boto:Params: {} 2024-04-12T04:11:05.212 DEBUG:boto:Token: None 2024-04-12T04:11:05.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e0427aa8d80bb57-006618b459-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.215 DEBUG:boto:b'dkyges-61obj1241000falseobj1242024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.216 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.216 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.216 DEBUG:boto:path=/dkyges-61/?prefix=obj124 2024-04-12T04:11:05.216 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj124 2024-04-12T04:11:05.216 DEBUG:boto:Method: GET 2024-04-12T04:11:05.216 DEBUG:boto:Path: /dkyges-61/?prefix=obj124 2024-04-12T04:11:05.216 DEBUG:boto:Data: 2024-04-12T04:11:05.216 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.216 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.216 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.216 DEBUG:boto:Params: {} 2024-04-12T04:11:05.216 DEBUG:boto:Token: None 2024-04-12T04:11:05.217 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.217 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000198600e873339889-006618b459-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.221 DEBUG:boto:b'dkyges-61obj1241000falseobj1242024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.222 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:11:05.222 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:11:05.222 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.222 DEBUG:boto:Path: /dkyges-61/obj124 2024-04-12T04:11:05.222 DEBUG:boto:Data: 2024-04-12T04:11:05.223 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.223 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.223 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.223 DEBUG:boto:Params: {} 2024-04-12T04:11:05.224 DEBUG:boto:Token: None 2024-04-12T04:11:05.224 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj124 2024-04-12T04:11:05.224 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7MZNkIYYDOb6MhBwjbqOCb+VM+E= 2024-04-12T04:11:05.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7MZNkIYYDOb6MhBwjbqOCb+VM+E=', 'Content-Length': '0'} 2024-04-12T04:11:05.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5e62c1efd1ff971-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.227 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:11:05.227 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:11:05.227 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.228 DEBUG:boto:Path: /dkyges-61/obj124 2024-04-12T04:11:05.228 DEBUG:boto:Data: 2024-04-12T04:11:05.228 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.228 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.228 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.228 DEBUG:boto:Params: {} 2024-04-12T04:11:05.229 DEBUG:boto:Token: None 2024-04-12T04:11:05.229 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj124 2024-04-12T04:11:05.229 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7MZNkIYYDOb6MhBwjbqOCb+VM+E= 2024-04-12T04:11:05.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7MZNkIYYDOb6MhBwjbqOCb+VM+E=', 'Content-Length': '0'} 2024-04-12T04:11:05.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000670240486d8d605c-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T04:11:05.232 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:11:05.232 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:11:05.232 DEBUG:boto:Method: GET 2024-04-12T04:11:05.232 DEBUG:boto:Path: /dkyges-61/obj124 2024-04-12T04:11:05.232 DEBUG:boto:Data: 2024-04-12T04:11:05.232 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.233 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.233 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.233 DEBUG:boto:Params: {} 2024-04-12T04:11:05.233 DEBUG:boto:Token: None 2024-04-12T04:11:05.233 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj124 2024-04-12T04:11:05.233 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yyx7PaEljsHPNLHSvpcSQIRLMS0= 2024-04-12T04:11:05.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yyx7PaEljsHPNLHSvpcSQIRLMS0=', 'Content-Length': '0'} 2024-04-12T04:11:05.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0fbb0b0742f2dd7-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.278 DEBUG:boto:path=/dkyges-61/obj124 2024-04-12T04:11:05.278 DEBUG:boto:auth_path=/dkyges-61/obj124 2024-04-12T04:11:05.279 DEBUG:boto:Method: GET 2024-04-12T04:11:05.279 DEBUG:boto:Path: /dkyges-61/obj124 2024-04-12T04:11:05.279 DEBUG:boto:Data: 2024-04-12T04:11:05.279 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.279 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.279 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.279 DEBUG:boto:Params: {} 2024-04-12T04:11:05.279 DEBUG:boto:Token: None 2024-04-12T04:11:05.279 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj124 2024-04-12T04:11:05.280 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yyx7PaEljsHPNLHSvpcSQIRLMS0= 2024-04-12T04:11:05.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yyx7PaEljsHPNLHSvpcSQIRLMS0=', 'Content-Length': '0'} 2024-04-12T04:11:05.281 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed94c2d74cb85ea0-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.282 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.282 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.282 DEBUG:boto:path=/dkyges-61/?prefix=obj124 2024-04-12T04:11:05.282 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj124 2024-04-12T04:11:05.282 DEBUG:boto:Method: GET 2024-04-12T04:11:05.282 DEBUG:boto:Path: /dkyges-61/?prefix=obj124 2024-04-12T04:11:05.283 DEBUG:boto:Data: 2024-04-12T04:11:05.283 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.283 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.283 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.283 DEBUG:boto:Params: {} 2024-04-12T04:11:05.283 DEBUG:boto:Token: None 2024-04-12T04:11:05.283 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.283 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098ad341ac3bd48f4-006618b459-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.287 DEBUG:boto:b'dkyges-61obj1241000falseobj1242024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.287 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.288 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.288 DEBUG:boto:path=/dkyges-61/?prefix=obj124 2024-04-12T04:11:05.288 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj124 2024-04-12T04:11:05.288 DEBUG:boto:Method: GET 2024-04-12T04:11:05.288 DEBUG:boto:Path: /dkyges-61/?prefix=obj124 2024-04-12T04:11:05.289 DEBUG:boto:Data: 2024-04-12T04:11:05.289 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.289 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.289 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.289 DEBUG:boto:Params: {} 2024-04-12T04:11:05.289 DEBUG:boto:Token: None 2024-04-12T04:11:05.290 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.290 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027809ec93c6f6cea-006618b459-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.294 DEBUG:boto:b'dkyges-61obj1241000falseobj1242024-04-12T04:08:08.248Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:11:05.295 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:11:05.295 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:11:05.295 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.295 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.295 DEBUG:boto:Method: GET 2024-04-12T04:11:05.295 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.296 DEBUG:boto:Data: 2024-04-12T04:11:05.296 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.296 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.296 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.296 DEBUG:boto:Params: {} 2024-04-12T04:11:05.296 DEBUG:boto:Token: None 2024-04-12T04:11:05.296 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.296 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JLpu1Mz7cgMBnY2DMnIjgvY13Rg= 2024-04-12T04:11:05.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JLpu1Mz7cgMBnY2DMnIjgvY13Rg=', 'Content-Length': '0'} 2024-04-12T04:11:05.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e02389c9f9cba8c9-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.299 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:11:05.299 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:11:05.299 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.299 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.299 DEBUG:boto:Method: GET 2024-04-12T04:11:05.299 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.299 DEBUG:boto:Data: 2024-04-12T04:11:05.300 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.300 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.300 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.300 DEBUG:boto:Params: {} 2024-04-12T04:11:05.300 DEBUG:boto:Token: None 2024-04-12T04:11:05.300 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.300 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JLpu1Mz7cgMBnY2DMnIjgvY13Rg= 2024-04-12T04:11:05.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JLpu1Mz7cgMBnY2DMnIjgvY13Rg=', 'Content-Length': '0'} 2024-04-12T04:11:05.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9bac6cb23ba12f7-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.302 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:11:05.303 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:11:05.303 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.303 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.303 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.303 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.303 DEBUG:boto:Data: 2024-04-12T04:11:05.303 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.303 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.303 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.303 DEBUG:boto:Params: {} 2024-04-12T04:11:05.304 DEBUG:boto:Token: None 2024-04-12T04:11:05.304 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.304 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cyftW91qaNYxoL79iWtVkn0C+ws= 2024-04-12T04:11:05.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cyftW91qaNYxoL79iWtVkn0C+ws=', 'Content-Length': '0'} 2024-04-12T04:11:05.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d82e65483b63f08-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.306 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:11:05.306 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:11:05.306 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.306 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.306 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.306 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.307 DEBUG:boto:Data: 2024-04-12T04:11:05.307 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.307 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.307 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.307 DEBUG:boto:Params: {} 2024-04-12T04:11:05.307 DEBUG:boto:Token: None 2024-04-12T04:11:05.307 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.307 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cyftW91qaNYxoL79iWtVkn0C+ws= 2024-04-12T04:11:05.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cyftW91qaNYxoL79iWtVkn0C+ws=', 'Content-Length': '0'} 2024-04-12T04:11:05.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042a3190738f51bf8-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:11:05.310 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:11:05.310 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:11:05.310 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.310 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.310 DEBUG:boto:Method: GET 2024-04-12T04:11:05.310 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.310 DEBUG:boto:Data: 2024-04-12T04:11:05.310 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.310 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.310 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.311 DEBUG:boto:Params: {} 2024-04-12T04:11:05.311 DEBUG:boto:Token: None 2024-04-12T04:11:05.311 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.311 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JLpu1Mz7cgMBnY2DMnIjgvY13Rg= 2024-04-12T04:11:05.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JLpu1Mz7cgMBnY2DMnIjgvY13Rg=', 'Content-Length': '0'} 2024-04-12T04:11:05.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005298bb8a081f2f35-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.354 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:11:05.355 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:11:05.355 DEBUG:boto:path=/dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.355 DEBUG:boto:auth_path=/dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.355 DEBUG:boto:Method: GET 2024-04-12T04:11:05.355 DEBUG:boto:Path: /dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.355 DEBUG:boto:Data: 2024-04-12T04:11:05.355 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.355 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.355 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.355 DEBUG:boto:Params: {} 2024-04-12T04:11:05.355 DEBUG:boto:Token: None 2024-04-12T04:11:05.356 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj145?versionId=null 2024-04-12T04:11:05.356 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JLpu1Mz7cgMBnY2DMnIjgvY13Rg= 2024-04-12T04:11:05.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JLpu1Mz7cgMBnY2DMnIjgvY13Rg=', 'Content-Length': '0'} 2024-04-12T04:11:05.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050003d76bf86c068-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.358 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.358 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.358 DEBUG:boto:path=/dkyges-61/?prefix=obj145 2024-04-12T04:11:05.358 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj145 2024-04-12T04:11:05.359 DEBUG:boto:Method: GET 2024-04-12T04:11:05.359 DEBUG:boto:Path: /dkyges-61/?prefix=obj145 2024-04-12T04:11:05.359 DEBUG:boto:Data: 2024-04-12T04:11:05.359 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.359 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.359 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.359 DEBUG:boto:Params: {} 2024-04-12T04:11:05.359 DEBUG:boto:Token: None 2024-04-12T04:11:05.359 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.360 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005555589dd705d5cc-006618b459-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.362 DEBUG:boto:b'dkyges-61obj1451000falseobj1452024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.363 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.363 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.363 DEBUG:boto:path=/dkyges-61/?prefix=obj145 2024-04-12T04:11:05.363 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj145 2024-04-12T04:11:05.363 DEBUG:boto:Method: GET 2024-04-12T04:11:05.363 DEBUG:boto:Path: /dkyges-61/?prefix=obj145 2024-04-12T04:11:05.363 DEBUG:boto:Data: 2024-04-12T04:11:05.363 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.363 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.364 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.364 DEBUG:boto:Params: {} 2024-04-12T04:11:05.364 DEBUG:boto:Token: None 2024-04-12T04:11:05.364 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.364 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.368 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009289c6aa5c7f0ea3-006618b459-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.369 DEBUG:boto:b'dkyges-61obj1451000falseobj1452024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.369 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:11:05.369 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:11:05.370 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.370 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:11:05.370 DEBUG:boto:Data: 2024-04-12T04:11:05.370 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.370 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.370 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.370 DEBUG:boto:Params: {} 2024-04-12T04:11:05.370 DEBUG:boto:Token: None 2024-04-12T04:11:05.371 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj145 2024-04-12T04:11:05.371 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IooLgTQTVO6Vsxid4BvXXJXqZJg= 2024-04-12T04:11:05.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IooLgTQTVO6Vsxid4BvXXJXqZJg=', 'Content-Length': '0'} 2024-04-12T04:11:05.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000444eaf2c8263b506-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.373 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:11:05.373 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:11:05.373 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.373 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:11:05.373 DEBUG:boto:Data: 2024-04-12T04:11:05.373 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.373 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.373 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.373 DEBUG:boto:Params: {} 2024-04-12T04:11:05.374 DEBUG:boto:Token: None 2024-04-12T04:11:05.374 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj145 2024-04-12T04:11:05.374 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IooLgTQTVO6Vsxid4BvXXJXqZJg= 2024-04-12T04:11:05.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IooLgTQTVO6Vsxid4BvXXJXqZJg=', 'Content-Length': '0'} 2024-04-12T04:11:05.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054ec3d5f6fb2f9cf-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:11:05.376 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:11:05.376 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:11:05.376 DEBUG:boto:Method: GET 2024-04-12T04:11:05.376 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:11:05.377 DEBUG:boto:Data: 2024-04-12T04:11:05.377 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.377 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.377 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.377 DEBUG:boto:Params: {} 2024-04-12T04:11:05.377 DEBUG:boto:Token: None 2024-04-12T04:11:05.377 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj145 2024-04-12T04:11:05.377 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jm8ONQzBe2JMw62ftcs5lSVV+xM= 2024-04-12T04:11:05.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jm8ONQzBe2JMw62ftcs5lSVV+xM=', 'Content-Length': '0'} 2024-04-12T04:11:05.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a43b860a6ece9289-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.380 DEBUG:boto:path=/dkyges-61/obj145 2024-04-12T04:11:05.380 DEBUG:boto:auth_path=/dkyges-61/obj145 2024-04-12T04:11:05.380 DEBUG:boto:Method: GET 2024-04-12T04:11:05.380 DEBUG:boto:Path: /dkyges-61/obj145 2024-04-12T04:11:05.380 DEBUG:boto:Data: 2024-04-12T04:11:05.380 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.380 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.380 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.380 DEBUG:boto:Params: {} 2024-04-12T04:11:05.380 DEBUG:boto:Token: None 2024-04-12T04:11:05.381 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj145 2024-04-12T04:11:05.381 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jm8ONQzBe2JMw62ftcs5lSVV+xM= 2024-04-12T04:11:05.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jm8ONQzBe2JMw62ftcs5lSVV+xM=', 'Content-Length': '0'} 2024-04-12T04:11:05.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030fc696a52e8add4-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.426 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.427 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.427 DEBUG:boto:path=/dkyges-61/?prefix=obj145 2024-04-12T04:11:05.427 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj145 2024-04-12T04:11:05.427 DEBUG:boto:Method: GET 2024-04-12T04:11:05.428 DEBUG:boto:Path: /dkyges-61/?prefix=obj145 2024-04-12T04:11:05.428 DEBUG:boto:Data: 2024-04-12T04:11:05.428 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.428 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.428 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.428 DEBUG:boto:Params: {} 2024-04-12T04:11:05.428 DEBUG:boto:Token: None 2024-04-12T04:11:05.429 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.429 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a41e4abef10cc5c-006618b459-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.432 DEBUG:boto:b'dkyges-61obj1451000falseobj1452024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.432 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.433 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.433 DEBUG:boto:path=/dkyges-61/?prefix=obj145 2024-04-12T04:11:05.433 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj145 2024-04-12T04:11:05.433 DEBUG:boto:Method: GET 2024-04-12T04:11:05.433 DEBUG:boto:Path: /dkyges-61/?prefix=obj145 2024-04-12T04:11:05.433 DEBUG:boto:Data: 2024-04-12T04:11:05.433 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.433 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.433 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.433 DEBUG:boto:Params: {} 2024-04-12T04:11:05.433 DEBUG:boto:Token: None 2024-04-12T04:11:05.434 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.434 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006036bb6b51bb98dc-006618b459-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.438 DEBUG:boto:b'dkyges-61obj1451000falseobj1452024-04-12T04:06:49.211Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T04:11:05.439 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:11:05.439 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:11:05.439 DEBUG:boto:path=/dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.439 DEBUG:boto:auth_path=/dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.439 DEBUG:boto:Method: GET 2024-04-12T04:11:05.439 DEBUG:boto:Path: /dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.439 DEBUG:boto:Data: 2024-04-12T04:11:05.440 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.440 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.440 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.440 DEBUG:boto:Params: {} 2024-04-12T04:11:05.440 DEBUG:boto:Token: None 2024-04-12T04:11:05.440 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.440 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/jCkachxCotJmAr1wLBgV3Xk6as= 2024-04-12T04:11:05.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/jCkachxCotJmAr1wLBgV3Xk6as=', 'Content-Length': '0'} 2024-04-12T04:11:05.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab7600da728ecbb9-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.486 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:11:05.486 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:11:05.486 DEBUG:boto:path=/dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.487 DEBUG:boto:auth_path=/dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.487 DEBUG:boto:Method: GET 2024-04-12T04:11:05.487 DEBUG:boto:Path: /dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.487 DEBUG:boto:Data: 2024-04-12T04:11:05.487 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.487 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.487 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.487 DEBUG:boto:Params: {} 2024-04-12T04:11:05.487 DEBUG:boto:Token: None 2024-04-12T04:11:05.488 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.488 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/jCkachxCotJmAr1wLBgV3Xk6as= 2024-04-12T04:11:05.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/jCkachxCotJmAr1wLBgV3Xk6as=', 'Content-Length': '0'} 2024-04-12T04:11:05.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a42093b01d1914a-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.491 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:11:05.491 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:11:05.491 DEBUG:boto:path=/dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.491 DEBUG:boto:auth_path=/dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.491 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.491 DEBUG:boto:Path: /dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.491 DEBUG:boto:Data: 2024-04-12T04:11:05.491 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.491 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.492 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.492 DEBUG:boto:Params: {} 2024-04-12T04:11:05.492 DEBUG:boto:Token: None 2024-04-12T04:11:05.492 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.492 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iNCR20o/I7M8/OkQuhFuTJomraM= 2024-04-12T04:11:05.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iNCR20o/I7M8/OkQuhFuTJomraM=', 'Content-Length': '0'} 2024-04-12T04:11:05.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015cd452b57bb9916-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.494 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:11:05.495 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:11:05.495 DEBUG:boto:path=/dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.495 DEBUG:boto:auth_path=/dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.495 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.495 DEBUG:boto:Path: /dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.495 DEBUG:boto:Data: 2024-04-12T04:11:05.495 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.495 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.495 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.495 DEBUG:boto:Params: {} 2024-04-12T04:11:05.496 DEBUG:boto:Token: None 2024-04-12T04:11:05.496 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.496 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iNCR20o/I7M8/OkQuhFuTJomraM= 2024-04-12T04:11:05.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iNCR20o/I7M8/OkQuhFuTJomraM=', 'Content-Length': '0'} 2024-04-12T04:11:05.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064b3a7518a8ec198-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T04:11:05.498 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:11:05.498 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:11:05.498 DEBUG:boto:path=/dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.498 DEBUG:boto:auth_path=/dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.499 DEBUG:boto:Method: GET 2024-04-12T04:11:05.499 DEBUG:boto:Path: /dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.499 DEBUG:boto:Data: 2024-04-12T04:11:05.499 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.499 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.499 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.499 DEBUG:boto:Params: {} 2024-04-12T04:11:05.499 DEBUG:boto:Token: None 2024-04-12T04:11:05.499 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.499 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/jCkachxCotJmAr1wLBgV3Xk6as= 2024-04-12T04:11:05.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/jCkachxCotJmAr1wLBgV3Xk6as=', 'Content-Length': '0'} 2024-04-12T04:11:05.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087e6af95d48a98a7-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.502 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:11:05.502 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:11:05.502 DEBUG:boto:path=/dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.502 DEBUG:boto:auth_path=/dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.502 DEBUG:boto:Method: GET 2024-04-12T04:11:05.502 DEBUG:boto:Path: /dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.502 DEBUG:boto:Data: 2024-04-12T04:11:05.502 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.502 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.503 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.503 DEBUG:boto:Params: {} 2024-04-12T04:11:05.503 DEBUG:boto:Token: None 2024-04-12T04:11:05.503 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj148?versionId=null 2024-04-12T04:11:05.503 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/jCkachxCotJmAr1wLBgV3Xk6as= 2024-04-12T04:11:05.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/jCkachxCotJmAr1wLBgV3Xk6as=', 'Content-Length': '0'} 2024-04-12T04:11:05.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022dc6c27df1020f8-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.505 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.505 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.505 DEBUG:boto:path=/dkyges-61/?prefix=obj148 2024-04-12T04:11:05.506 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj148 2024-04-12T04:11:05.506 DEBUG:boto:Method: GET 2024-04-12T04:11:05.506 DEBUG:boto:Path: /dkyges-61/?prefix=obj148 2024-04-12T04:11:05.506 DEBUG:boto:Data: 2024-04-12T04:11:05.506 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.506 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.507 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.507 DEBUG:boto:Params: {} 2024-04-12T04:11:05.507 DEBUG:boto:Token: None 2024-04-12T04:11:05.507 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.507 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1a1141225d7b62e-006618b459-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.510 DEBUG:boto:b'dkyges-61obj1481000falseobj1482024-04-12T04:09:24.114Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.510 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.510 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.510 DEBUG:boto:path=/dkyges-61/?prefix=obj148 2024-04-12T04:11:05.510 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj148 2024-04-12T04:11:05.510 DEBUG:boto:Method: GET 2024-04-12T04:11:05.511 DEBUG:boto:Path: /dkyges-61/?prefix=obj148 2024-04-12T04:11:05.511 DEBUG:boto:Data: 2024-04-12T04:11:05.511 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.511 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.511 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.511 DEBUG:boto:Params: {} 2024-04-12T04:11:05.511 DEBUG:boto:Token: None 2024-04-12T04:11:05.511 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.511 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb56000708e35a8d-006618b459-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.516 DEBUG:boto:b'dkyges-61obj1481000falseobj1482024-04-12T04:09:24.114Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.516 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:11:05.516 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:11:05.516 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.516 DEBUG:boto:Path: /dkyges-61/obj148 2024-04-12T04:11:05.516 DEBUG:boto:Data: 2024-04-12T04:11:05.516 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.517 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.517 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.517 DEBUG:boto:Params: {} 2024-04-12T04:11:05.517 DEBUG:boto:Token: None 2024-04-12T04:11:05.517 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj148 2024-04-12T04:11:05.517 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zbtTl4qoclYtTzoA+rcqzXZ8MWE= 2024-04-12T04:11:05.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zbtTl4qoclYtTzoA+rcqzXZ8MWE=', 'Content-Length': '0'} 2024-04-12T04:11:05.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065dbf88201c20c4d-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.519 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:11:05.519 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:11:05.519 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.520 DEBUG:boto:Path: /dkyges-61/obj148 2024-04-12T04:11:05.520 DEBUG:boto:Data: 2024-04-12T04:11:05.520 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.520 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.520 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.520 DEBUG:boto:Params: {} 2024-04-12T04:11:05.520 DEBUG:boto:Token: None 2024-04-12T04:11:05.520 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj148 2024-04-12T04:11:05.520 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zbtTl4qoclYtTzoA+rcqzXZ8MWE= 2024-04-12T04:11:05.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zbtTl4qoclYtTzoA+rcqzXZ8MWE=', 'Content-Length': '0'} 2024-04-12T04:11:05.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035d4d94dfaba750f-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T04:11:05.523 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:11:05.523 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:11:05.523 DEBUG:boto:Method: GET 2024-04-12T04:11:05.524 DEBUG:boto:Path: /dkyges-61/obj148 2024-04-12T04:11:05.524 DEBUG:boto:Data: 2024-04-12T04:11:05.524 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.524 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.524 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.525 DEBUG:boto:Params: {} 2024-04-12T04:11:05.525 DEBUG:boto:Token: None 2024-04-12T04:11:05.525 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj148 2024-04-12T04:11:05.525 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lN+2Bja/KvTF11pzO6M9AgXNEtA= 2024-04-12T04:11:05.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lN+2Bja/KvTF11pzO6M9AgXNEtA=', 'Content-Length': '0'} 2024-04-12T04:11:05.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3a1171e5b69165a-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.528 DEBUG:boto:path=/dkyges-61/obj148 2024-04-12T04:11:05.528 DEBUG:boto:auth_path=/dkyges-61/obj148 2024-04-12T04:11:05.528 DEBUG:boto:Method: GET 2024-04-12T04:11:05.528 DEBUG:boto:Path: /dkyges-61/obj148 2024-04-12T04:11:05.529 DEBUG:boto:Data: 2024-04-12T04:11:05.529 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.529 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.529 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.530 DEBUG:boto:Params: {} 2024-04-12T04:11:05.530 DEBUG:boto:Token: None 2024-04-12T04:11:05.530 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj148 2024-04-12T04:11:05.530 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lN+2Bja/KvTF11pzO6M9AgXNEtA= 2024-04-12T04:11:05.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lN+2Bja/KvTF11pzO6M9AgXNEtA=', 'Content-Length': '0'} 2024-04-12T04:11:05.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbb3b8ba797e4741-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.574 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.574 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.575 DEBUG:boto:path=/dkyges-61/?prefix=obj148 2024-04-12T04:11:05.575 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj148 2024-04-12T04:11:05.575 DEBUG:boto:Method: GET 2024-04-12T04:11:05.575 DEBUG:boto:Path: /dkyges-61/?prefix=obj148 2024-04-12T04:11:05.575 DEBUG:boto:Data: 2024-04-12T04:11:05.575 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.575 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.575 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.575 DEBUG:boto:Params: {} 2024-04-12T04:11:05.575 DEBUG:boto:Token: None 2024-04-12T04:11:05.575 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.576 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080d838a0e37ccf98-006618b459-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.578 DEBUG:boto:b'dkyges-61obj1481000falseobj1482024-04-12T04:09:24.114Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.579 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.579 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.579 DEBUG:boto:path=/dkyges-61/?prefix=obj148 2024-04-12T04:11:05.579 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj148 2024-04-12T04:11:05.579 DEBUG:boto:Method: GET 2024-04-12T04:11:05.579 DEBUG:boto:Path: /dkyges-61/?prefix=obj148 2024-04-12T04:11:05.579 DEBUG:boto:Data: 2024-04-12T04:11:05.579 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.580 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.580 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.580 DEBUG:boto:Params: {} 2024-04-12T04:11:05.580 DEBUG:boto:Token: None 2024-04-12T04:11:05.580 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.580 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e001148fcdd917e2-006618b459-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.584 DEBUG:boto:b'dkyges-61obj1481000falseobj1482024-04-12T04:09:24.114Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T04:11:05.585 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:11:05.585 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:11:05.585 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.585 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.585 DEBUG:boto:Method: GET 2024-04-12T04:11:05.585 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.585 DEBUG:boto:Data: 2024-04-12T04:11:05.586 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.586 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.586 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.586 DEBUG:boto:Params: {} 2024-04-12T04:11:05.586 DEBUG:boto:Token: None 2024-04-12T04:11:05.586 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.586 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:36AjWnoT3BXDNQah/zze0a4bxo8= 2024-04-12T04:11:05.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:36AjWnoT3BXDNQah/zze0a4bxo8=', 'Content-Length': '0'} 2024-04-12T04:11:05.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f4ebad4765d72bf-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.630 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:11:05.631 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:11:05.631 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.631 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.631 DEBUG:boto:Method: GET 2024-04-12T04:11:05.632 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.632 DEBUG:boto:Data: 2024-04-12T04:11:05.632 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.632 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.632 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.632 DEBUG:boto:Params: {} 2024-04-12T04:11:05.632 DEBUG:boto:Token: None 2024-04-12T04:11:05.632 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.632 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:36AjWnoT3BXDNQah/zze0a4bxo8= 2024-04-12T04:11:05.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:36AjWnoT3BXDNQah/zze0a4bxo8=', 'Content-Length': '0'} 2024-04-12T04:11:05.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000274b372dd8fae740-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.635 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:11:05.635 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:11:05.635 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.635 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.635 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.635 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.635 DEBUG:boto:Data: 2024-04-12T04:11:05.636 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.636 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.636 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.636 DEBUG:boto:Params: {} 2024-04-12T04:11:05.636 DEBUG:boto:Token: None 2024-04-12T04:11:05.636 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.636 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u7xkUTN0qeLDFB4uo6MfvfLDN5c= 2024-04-12T04:11:05.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u7xkUTN0qeLDFB4uo6MfvfLDN5c=', 'Content-Length': '0'} 2024-04-12T04:11:05.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037b83e7ee33dca29-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.638 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:11:05.638 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:11:05.638 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.639 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.639 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.639 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.639 DEBUG:boto:Data: 2024-04-12T04:11:05.639 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.639 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.639 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.639 DEBUG:boto:Params: {} 2024-04-12T04:11:05.639 DEBUG:boto:Token: None 2024-04-12T04:11:05.639 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.640 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u7xkUTN0qeLDFB4uo6MfvfLDN5c= 2024-04-12T04:11:05.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u7xkUTN0qeLDFB4uo6MfvfLDN5c=', 'Content-Length': '0'} 2024-04-12T04:11:05.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063fa74178fcf4416-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T04:11:05.642 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:11:05.642 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:11:05.642 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.642 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.642 DEBUG:boto:Method: GET 2024-04-12T04:11:05.642 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.642 DEBUG:boto:Data: 2024-04-12T04:11:05.642 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.642 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.643 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.643 DEBUG:boto:Params: {} 2024-04-12T04:11:05.643 DEBUG:boto:Token: None 2024-04-12T04:11:05.643 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.643 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:36AjWnoT3BXDNQah/zze0a4bxo8= 2024-04-12T04:11:05.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:36AjWnoT3BXDNQah/zze0a4bxo8=', 'Content-Length': '0'} 2024-04-12T04:11:05.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038da6c5ab496a0eb-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.645 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:11:05.645 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:11:05.645 DEBUG:boto:path=/dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.646 DEBUG:boto:auth_path=/dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.646 DEBUG:boto:Method: GET 2024-04-12T04:11:05.646 DEBUG:boto:Path: /dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.646 DEBUG:boto:Data: 2024-04-12T04:11:05.646 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.646 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.646 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.646 DEBUG:boto:Params: {} 2024-04-12T04:11:05.646 DEBUG:boto:Token: None 2024-04-12T04:11:05.646 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj155?versionId=null 2024-04-12T04:11:05.647 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:36AjWnoT3BXDNQah/zze0a4bxo8= 2024-04-12T04:11:05.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:36AjWnoT3BXDNQah/zze0a4bxo8=', 'Content-Length': '0'} 2024-04-12T04:11:05.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f79b3f53844bd8c0-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.649 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.649 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.649 DEBUG:boto:path=/dkyges-61/?prefix=obj155 2024-04-12T04:11:05.649 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj155 2024-04-12T04:11:05.649 DEBUG:boto:Method: GET 2024-04-12T04:11:05.649 DEBUG:boto:Path: /dkyges-61/?prefix=obj155 2024-04-12T04:11:05.649 DEBUG:boto:Data: 2024-04-12T04:11:05.649 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.650 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.650 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.650 DEBUG:boto:Params: {} 2024-04-12T04:11:05.650 DEBUG:boto:Token: None 2024-04-12T04:11:05.650 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.650 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073b2e210fb65cad5-006618b459-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.653 DEBUG:boto:b'dkyges-61obj1551000falseobj1552024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.653 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.653 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.653 DEBUG:boto:path=/dkyges-61/?prefix=obj155 2024-04-12T04:11:05.653 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj155 2024-04-12T04:11:05.654 DEBUG:boto:Method: GET 2024-04-12T04:11:05.654 DEBUG:boto:Path: /dkyges-61/?prefix=obj155 2024-04-12T04:11:05.654 DEBUG:boto:Data: 2024-04-12T04:11:05.654 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.654 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.654 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.654 DEBUG:boto:Params: {} 2024-04-12T04:11:05.654 DEBUG:boto:Token: None 2024-04-12T04:11:05.654 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.654 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073f5e099d70379b4-006618b459-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.659 DEBUG:boto:b'dkyges-61obj1551000falseobj1552024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.659 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:11:05.659 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:11:05.660 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.660 DEBUG:boto:Path: /dkyges-61/obj155 2024-04-12T04:11:05.660 DEBUG:boto:Data: 2024-04-12T04:11:05.660 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.660 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.660 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.660 DEBUG:boto:Params: {} 2024-04-12T04:11:05.660 DEBUG:boto:Token: None 2024-04-12T04:11:05.660 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj155 2024-04-12T04:11:05.660 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xy84N68Omw5UYurTXgoU9qIF0vI= 2024-04-12T04:11:05.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xy84N68Omw5UYurTXgoU9qIF0vI=', 'Content-Length': '0'} 2024-04-12T04:11:05.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdd8f00e638f9f9d-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.663 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:11:05.663 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:11:05.663 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.663 DEBUG:boto:Path: /dkyges-61/obj155 2024-04-12T04:11:05.663 DEBUG:boto:Data: 2024-04-12T04:11:05.663 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.663 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.663 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.663 DEBUG:boto:Params: {} 2024-04-12T04:11:05.663 DEBUG:boto:Token: None 2024-04-12T04:11:05.664 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj155 2024-04-12T04:11:05.664 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xy84N68Omw5UYurTXgoU9qIF0vI= 2024-04-12T04:11:05.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xy84N68Omw5UYurTXgoU9qIF0vI=', 'Content-Length': '0'} 2024-04-12T04:11:05.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e954fb4aef2bc627-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T04:11:05.666 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:11:05.666 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:11:05.666 DEBUG:boto:Method: GET 2024-04-12T04:11:05.666 DEBUG:boto:Path: /dkyges-61/obj155 2024-04-12T04:11:05.666 DEBUG:boto:Data: 2024-04-12T04:11:05.666 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.667 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.667 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.667 DEBUG:boto:Params: {} 2024-04-12T04:11:05.667 DEBUG:boto:Token: None 2024-04-12T04:11:05.667 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj155 2024-04-12T04:11:05.667 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:46PUn43zUyfVKI+IF514b9a/cNg= 2024-04-12T04:11:05.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:46PUn43zUyfVKI+IF514b9a/cNg=', 'Content-Length': '0'} 2024-04-12T04:11:05.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e657d51f878f998-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.669 DEBUG:boto:path=/dkyges-61/obj155 2024-04-12T04:11:05.669 DEBUG:boto:auth_path=/dkyges-61/obj155 2024-04-12T04:11:05.670 DEBUG:boto:Method: GET 2024-04-12T04:11:05.670 DEBUG:boto:Path: /dkyges-61/obj155 2024-04-12T04:11:05.670 DEBUG:boto:Data: 2024-04-12T04:11:05.670 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.670 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.670 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.670 DEBUG:boto:Params: {} 2024-04-12T04:11:05.670 DEBUG:boto:Token: None 2024-04-12T04:11:05.670 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj155 2024-04-12T04:11:05.671 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:46PUn43zUyfVKI+IF514b9a/cNg= 2024-04-12T04:11:05.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:46PUn43zUyfVKI+IF514b9a/cNg=', 'Content-Length': '0'} 2024-04-12T04:11:05.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c72fdf1613021b29-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.714 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.715 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.715 DEBUG:boto:path=/dkyges-61/?prefix=obj155 2024-04-12T04:11:05.715 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj155 2024-04-12T04:11:05.715 DEBUG:boto:Method: GET 2024-04-12T04:11:05.715 DEBUG:boto:Path: /dkyges-61/?prefix=obj155 2024-04-12T04:11:05.715 DEBUG:boto:Data: 2024-04-12T04:11:05.715 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.715 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.715 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.715 DEBUG:boto:Params: {} 2024-04-12T04:11:05.716 DEBUG:boto:Token: None 2024-04-12T04:11:05.716 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.716 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e5dbf0c9fddf617-006618b459-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.719 DEBUG:boto:b'dkyges-61obj1551000falseobj1552024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.719 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.719 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.719 DEBUG:boto:path=/dkyges-61/?prefix=obj155 2024-04-12T04:11:05.719 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj155 2024-04-12T04:11:05.719 DEBUG:boto:Method: GET 2024-04-12T04:11:05.719 DEBUG:boto:Path: /dkyges-61/?prefix=obj155 2024-04-12T04:11:05.720 DEBUG:boto:Data: 2024-04-12T04:11:05.720 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.720 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.720 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.720 DEBUG:boto:Params: {} 2024-04-12T04:11:05.720 DEBUG:boto:Token: None 2024-04-12T04:11:05.720 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.720 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab703f67fee7491a-006618b459-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.725 DEBUG:boto:b'dkyges-61obj1551000falseobj1552024-04-12T04:08:08.299Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:11:05.725 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:11:05.726 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:11:05.726 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.726 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.726 DEBUG:boto:Method: GET 2024-04-12T04:11:05.726 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.726 DEBUG:boto:Data: 2024-04-12T04:11:05.726 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.726 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.726 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.727 DEBUG:boto:Params: {} 2024-04-12T04:11:05.727 DEBUG:boto:Token: None 2024-04-12T04:11:05.727 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.727 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1VgmK1422vXqObrIbjO5AX6asMs= 2024-04-12T04:11:05.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1VgmK1422vXqObrIbjO5AX6asMs=', 'Content-Length': '0'} 2024-04-12T04:11:05.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ded29bc72e9b9337-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.770 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:11:05.770 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:11:05.771 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.771 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.771 DEBUG:boto:Method: GET 2024-04-12T04:11:05.771 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.771 DEBUG:boto:Data: 2024-04-12T04:11:05.771 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.771 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.771 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.771 DEBUG:boto:Params: {} 2024-04-12T04:11:05.771 DEBUG:boto:Token: None 2024-04-12T04:11:05.771 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.772 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1VgmK1422vXqObrIbjO5AX6asMs= 2024-04-12T04:11:05.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1VgmK1422vXqObrIbjO5AX6asMs=', 'Content-Length': '0'} 2024-04-12T04:11:05.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011a116d987423b1a-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.774 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:11:05.774 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:11:05.774 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.774 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.774 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.774 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.774 DEBUG:boto:Data: 2024-04-12T04:11:05.775 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.775 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.775 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.775 DEBUG:boto:Params: {} 2024-04-12T04:11:05.775 DEBUG:boto:Token: None 2024-04-12T04:11:05.775 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.775 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:s9bwDFHUEuVeScN3aabOUlq1M7A= 2024-04-12T04:11:05.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:s9bwDFHUEuVeScN3aabOUlq1M7A=', 'Content-Length': '0'} 2024-04-12T04:11:05.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3a30ce65cbf7a2c-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.777 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:11:05.777 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:11:05.777 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.778 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.778 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.778 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.778 DEBUG:boto:Data: 2024-04-12T04:11:05.778 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.778 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.778 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.778 DEBUG:boto:Params: {} 2024-04-12T04:11:05.778 DEBUG:boto:Token: None 2024-04-12T04:11:05.778 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.779 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:s9bwDFHUEuVeScN3aabOUlq1M7A= 2024-04-12T04:11:05.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:s9bwDFHUEuVeScN3aabOUlq1M7A=', 'Content-Length': '0'} 2024-04-12T04:11:05.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db1cb01a046ba6c1-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:11:05.781 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:11:05.781 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:11:05.781 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.781 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.781 DEBUG:boto:Method: GET 2024-04-12T04:11:05.781 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.781 DEBUG:boto:Data: 2024-04-12T04:11:05.782 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.782 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.782 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.782 DEBUG:boto:Params: {} 2024-04-12T04:11:05.782 DEBUG:boto:Token: None 2024-04-12T04:11:05.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.782 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1VgmK1422vXqObrIbjO5AX6asMs= 2024-04-12T04:11:05.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1VgmK1422vXqObrIbjO5AX6asMs=', 'Content-Length': '0'} 2024-04-12T04:11:05.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba85b43ed5e523b9-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.784 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:11:05.784 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:11:05.784 DEBUG:boto:path=/dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.784 DEBUG:boto:auth_path=/dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.785 DEBUG:boto:Method: GET 2024-04-12T04:11:05.785 DEBUG:boto:Path: /dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.785 DEBUG:boto:Data: 2024-04-12T04:11:05.785 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.785 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.785 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.785 DEBUG:boto:Params: {} 2024-04-12T04:11:05.785 DEBUG:boto:Token: None 2024-04-12T04:11:05.785 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj17?versionId=null 2024-04-12T04:11:05.785 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1VgmK1422vXqObrIbjO5AX6asMs= 2024-04-12T04:11:05.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1VgmK1422vXqObrIbjO5AX6asMs=', 'Content-Length': '0'} 2024-04-12T04:11:05.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037f7ff38954a6c89-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.788 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.788 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.788 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:11:05.788 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:11:05.788 DEBUG:boto:Method: GET 2024-04-12T04:11:05.788 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:11:05.788 DEBUG:boto:Data: 2024-04-12T04:11:05.788 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.788 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.789 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.789 DEBUG:boto:Params: {} 2024-04-12T04:11:05.789 DEBUG:boto:Token: None 2024-04-12T04:11:05.789 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.789 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.792 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000961a7e4bf24bcb7a-006618b459-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.792 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.792 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.792 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.792 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:11:05.792 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:11:05.793 DEBUG:boto:Method: GET 2024-04-12T04:11:05.793 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:11:05.793 DEBUG:boto:Data: 2024-04-12T04:11:05.793 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.793 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.793 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.793 DEBUG:boto:Params: {} 2024-04-12T04:11:05.793 DEBUG:boto:Token: None 2024-04-12T04:11:05.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.793 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000328932b5d09c81be-006618b459-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.798 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.798 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:11:05.798 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:11:05.798 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.798 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:11:05.799 DEBUG:boto:Data: 2024-04-12T04:11:05.799 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.799 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.799 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.799 DEBUG:boto:Params: {} 2024-04-12T04:11:05.799 DEBUG:boto:Token: None 2024-04-12T04:11:05.799 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj17 2024-04-12T04:11:05.799 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4tH+5Vt1dTekXLnekkoCU9BIDgk= 2024-04-12T04:11:05.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4tH+5Vt1dTekXLnekkoCU9BIDgk=', 'Content-Length': '0'} 2024-04-12T04:11:05.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b40ac70b8f17255-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.801 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:11:05.801 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:11:05.801 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.801 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:11:05.801 DEBUG:boto:Data: 2024-04-12T04:11:05.802 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.802 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.802 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.802 DEBUG:boto:Params: {} 2024-04-12T04:11:05.802 DEBUG:boto:Token: None 2024-04-12T04:11:05.802 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj17 2024-04-12T04:11:05.802 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4tH+5Vt1dTekXLnekkoCU9BIDgk= 2024-04-12T04:11:05.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4tH+5Vt1dTekXLnekkoCU9BIDgk=', 'Content-Length': '0'} 2024-04-12T04:11:05.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c189a5366157ba9d-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:11:05.804 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:11:05.804 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:11:05.804 DEBUG:boto:Method: GET 2024-04-12T04:11:05.804 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:11:05.805 DEBUG:boto:Data: 2024-04-12T04:11:05.805 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.805 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.805 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.805 DEBUG:boto:Params: {} 2024-04-12T04:11:05.805 DEBUG:boto:Token: None 2024-04-12T04:11:05.805 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj17 2024-04-12T04:11:05.805 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:empuj8ykRHe/dYAkYT7xjwnK/hA= 2024-04-12T04:11:05.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:empuj8ykRHe/dYAkYT7xjwnK/hA=', 'Content-Length': '0'} 2024-04-12T04:11:05.807 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0d7e2f5178dbc67-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.807 DEBUG:boto:path=/dkyges-61/obj17 2024-04-12T04:11:05.807 DEBUG:boto:auth_path=/dkyges-61/obj17 2024-04-12T04:11:05.807 DEBUG:boto:Method: GET 2024-04-12T04:11:05.807 DEBUG:boto:Path: /dkyges-61/obj17 2024-04-12T04:11:05.807 DEBUG:boto:Data: 2024-04-12T04:11:05.808 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.808 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.808 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.808 DEBUG:boto:Params: {} 2024-04-12T04:11:05.808 DEBUG:boto:Token: None 2024-04-12T04:11:05.808 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj17 2024-04-12T04:11:05.808 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:empuj8ykRHe/dYAkYT7xjwnK/hA= 2024-04-12T04:11:05.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:empuj8ykRHe/dYAkYT7xjwnK/hA=', 'Content-Length': '0'} 2024-04-12T04:11:05.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000221fe6cfb0558cc7-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.850 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.850 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.851 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:11:05.851 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:11:05.851 DEBUG:boto:Method: GET 2024-04-12T04:11:05.851 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:11:05.851 DEBUG:boto:Data: 2024-04-12T04:11:05.851 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.851 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.851 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.851 DEBUG:boto:Params: {} 2024-04-12T04:11:05.851 DEBUG:boto:Token: None 2024-04-12T04:11:05.851 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.852 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bd15ddb4d8de923-006618b459-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.854 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.855 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.855 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.855 DEBUG:boto:path=/dkyges-61/?prefix=obj17 2024-04-12T04:11:05.855 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj17 2024-04-12T04:11:05.855 DEBUG:boto:Method: GET 2024-04-12T04:11:05.855 DEBUG:boto:Path: /dkyges-61/?prefix=obj17 2024-04-12T04:11:05.855 DEBUG:boto:Data: 2024-04-12T04:11:05.855 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.855 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.856 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.856 DEBUG:boto:Params: {} 2024-04-12T04:11:05.856 DEBUG:boto:Token: None 2024-04-12T04:11:05.856 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.856 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009f60588c5f2bed2-006618b459-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.860 DEBUG:boto:b'dkyges-61obj171000falseobj172024-04-12T04:00:41.675Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T04:11:05.861 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:11:05.861 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:11:05.861 DEBUG:boto:path=/dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.861 DEBUG:boto:auth_path=/dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.861 DEBUG:boto:Method: GET 2024-04-12T04:11:05.861 DEBUG:boto:Path: /dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.861 DEBUG:boto:Data: 2024-04-12T04:11:05.862 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.862 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.862 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.862 DEBUG:boto:Params: {} 2024-04-12T04:11:05.862 DEBUG:boto:Token: None 2024-04-12T04:11:05.862 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.862 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/DdkBTQmytC6mVYou9zKsAPGYCo= 2024-04-12T04:11:05.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/DdkBTQmytC6mVYou9zKsAPGYCo=', 'Content-Length': '0'} 2024-04-12T04:11:05.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5e5a54d37fb8abc-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.906 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:11:05.907 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:11:05.907 DEBUG:boto:path=/dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.907 DEBUG:boto:auth_path=/dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.907 DEBUG:boto:Method: GET 2024-04-12T04:11:05.907 DEBUG:boto:Path: /dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.907 DEBUG:boto:Data: 2024-04-12T04:11:05.907 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.907 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.907 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.907 DEBUG:boto:Params: {} 2024-04-12T04:11:05.907 DEBUG:boto:Token: None 2024-04-12T04:11:05.908 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.908 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/DdkBTQmytC6mVYou9zKsAPGYCo= 2024-04-12T04:11:05.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/DdkBTQmytC6mVYou9zKsAPGYCo=', 'Content-Length': '0'} 2024-04-12T04:11:05.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0ef2b7188616c94-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.910 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:11:05.910 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:11:05.910 DEBUG:boto:path=/dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.910 DEBUG:boto:auth_path=/dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.910 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.910 DEBUG:boto:Path: /dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.910 DEBUG:boto:Data: 2024-04-12T04:11:05.911 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.911 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.911 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.911 DEBUG:boto:Params: {} 2024-04-12T04:11:05.911 DEBUG:boto:Token: None 2024-04-12T04:11:05.911 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.911 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TWvyDb2GT6mJ8TmiRxDj4bVEsn0= 2024-04-12T04:11:05.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TWvyDb2GT6mJ8TmiRxDj4bVEsn0=', 'Content-Length': '0'} 2024-04-12T04:11:05.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5a7706990f038b2-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.913 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:11:05.913 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:11:05.914 DEBUG:boto:path=/dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.914 DEBUG:boto:auth_path=/dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.914 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.914 DEBUG:boto:Path: /dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.914 DEBUG:boto:Data: 2024-04-12T04:11:05.914 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.914 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.914 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.914 DEBUG:boto:Params: {} 2024-04-12T04:11:05.914 DEBUG:boto:Token: None 2024-04-12T04:11:05.915 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.915 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TWvyDb2GT6mJ8TmiRxDj4bVEsn0= 2024-04-12T04:11:05.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TWvyDb2GT6mJ8TmiRxDj4bVEsn0=', 'Content-Length': '0'} 2024-04-12T04:11:05.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e182255748be0998-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T04:11:05.917 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:11:05.917 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:11:05.917 DEBUG:boto:path=/dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.917 DEBUG:boto:auth_path=/dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.917 DEBUG:boto:Method: GET 2024-04-12T04:11:05.918 DEBUG:boto:Path: /dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.918 DEBUG:boto:Data: 2024-04-12T04:11:05.918 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.918 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.918 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.918 DEBUG:boto:Params: {} 2024-04-12T04:11:05.918 DEBUG:boto:Token: None 2024-04-12T04:11:05.918 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.918 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/DdkBTQmytC6mVYou9zKsAPGYCo= 2024-04-12T04:11:05.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/DdkBTQmytC6mVYou9zKsAPGYCo=', 'Content-Length': '0'} 2024-04-12T04:11:05.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004da999413c73e60b-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.920 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:11:05.921 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:11:05.921 DEBUG:boto:path=/dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.921 DEBUG:boto:auth_path=/dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.921 DEBUG:boto:Method: GET 2024-04-12T04:11:05.921 DEBUG:boto:Path: /dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.921 DEBUG:boto:Data: 2024-04-12T04:11:05.921 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.921 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.921 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.921 DEBUG:boto:Params: {} 2024-04-12T04:11:05.921 DEBUG:boto:Token: None 2024-04-12T04:11:05.922 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj185?versionId=null 2024-04-12T04:11:05.922 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/DdkBTQmytC6mVYou9zKsAPGYCo= 2024-04-12T04:11:05.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/DdkBTQmytC6mVYou9zKsAPGYCo=', 'Content-Length': '0'} 2024-04-12T04:11:05.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd72cb88c159643d-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.924 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.924 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.924 DEBUG:boto:path=/dkyges-61/?prefix=obj185 2024-04-12T04:11:05.924 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj185 2024-04-12T04:11:05.924 DEBUG:boto:Method: GET 2024-04-12T04:11:05.924 DEBUG:boto:Path: /dkyges-61/?prefix=obj185 2024-04-12T04:11:05.924 DEBUG:boto:Data: 2024-04-12T04:11:05.925 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.925 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.925 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.925 DEBUG:boto:Params: {} 2024-04-12T04:11:05.925 DEBUG:boto:Token: None 2024-04-12T04:11:05.925 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.925 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000783883faabe15b1d-006618b459-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.928 DEBUG:boto:b'dkyges-61obj1851000falseobj1852024-04-12T04:09:24.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.928 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.928 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.928 DEBUG:boto:path=/dkyges-61/?prefix=obj185 2024-04-12T04:11:05.928 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj185 2024-04-12T04:11:05.928 DEBUG:boto:Method: GET 2024-04-12T04:11:05.928 DEBUG:boto:Path: /dkyges-61/?prefix=obj185 2024-04-12T04:11:05.928 DEBUG:boto:Data: 2024-04-12T04:11:05.929 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.929 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.929 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.929 DEBUG:boto:Params: {} 2024-04-12T04:11:05.929 DEBUG:boto:Token: None 2024-04-12T04:11:05.929 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.929 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2029283a5bb5e79-006618b459-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.934 DEBUG:boto:b'dkyges-61obj1851000falseobj1852024-04-12T04:09:24.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.934 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:11:05.934 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:11:05.934 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.934 DEBUG:boto:Path: /dkyges-61/obj185 2024-04-12T04:11:05.934 DEBUG:boto:Data: 2024-04-12T04:11:05.934 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.935 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.935 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.935 DEBUG:boto:Params: {} 2024-04-12T04:11:05.935 DEBUG:boto:Token: None 2024-04-12T04:11:05.935 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj185 2024-04-12T04:11:05.935 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dEKrk4PMrKHIbOr81RneXRTPf8M= 2024-04-12T04:11:05.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dEKrk4PMrKHIbOr81RneXRTPf8M=', 'Content-Length': '0'} 2024-04-12T04:11:05.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f70ca760e36f4b8-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.937 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:11:05.937 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:11:05.937 DEBUG:boto:Method: HEAD 2024-04-12T04:11:05.937 DEBUG:boto:Path: /dkyges-61/obj185 2024-04-12T04:11:05.937 DEBUG:boto:Data: 2024-04-12T04:11:05.937 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.937 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.938 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.938 DEBUG:boto:Params: {} 2024-04-12T04:11:05.938 DEBUG:boto:Token: None 2024-04-12T04:11:05.938 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj185 2024-04-12T04:11:05.938 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dEKrk4PMrKHIbOr81RneXRTPf8M= 2024-04-12T04:11:05.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dEKrk4PMrKHIbOr81RneXRTPf8M=', 'Content-Length': '0'} 2024-04-12T04:11:05.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb9d9b8d022b7f1a-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T04:11:05.940 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:11:05.940 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:11:05.940 DEBUG:boto:Method: GET 2024-04-12T04:11:05.940 DEBUG:boto:Path: /dkyges-61/obj185 2024-04-12T04:11:05.941 DEBUG:boto:Data: 2024-04-12T04:11:05.941 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.941 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.941 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.941 DEBUG:boto:Params: {} 2024-04-12T04:11:05.941 DEBUG:boto:Token: None 2024-04-12T04:11:05.941 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj185 2024-04-12T04:11:05.941 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qrl1qzFMTIgV8pOEihdQBpE7CQE= 2024-04-12T04:11:05.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qrl1qzFMTIgV8pOEihdQBpE7CQE=', 'Content-Length': '0'} 2024-04-12T04:11:05.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036d32d486937ebfd-006618b459-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.943 DEBUG:boto:path=/dkyges-61/obj185 2024-04-12T04:11:05.943 DEBUG:boto:auth_path=/dkyges-61/obj185 2024-04-12T04:11:05.943 DEBUG:boto:Method: GET 2024-04-12T04:11:05.943 DEBUG:boto:Path: /dkyges-61/obj185 2024-04-12T04:11:05.943 DEBUG:boto:Data: 2024-04-12T04:11:05.943 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.944 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.944 DEBUG:boto:Port: 8000 2024-04-12T04:11:05.944 DEBUG:boto:Params: {} 2024-04-12T04:11:05.944 DEBUG:boto:Token: None 2024-04-12T04:11:05.944 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/obj185 2024-04-12T04:11:05.944 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qrl1qzFMTIgV8pOEihdQBpE7CQE= 2024-04-12T04:11:05.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qrl1qzFMTIgV8pOEihdQBpE7CQE=', 'Content-Length': '0'} 2024-04-12T04:11:05.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d1928a92916981f-006618b459-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.946 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.946 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.947 DEBUG:boto:path=/dkyges-61/?prefix=obj185 2024-04-12T04:11:05.947 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj185 2024-04-12T04:11:05.947 DEBUG:boto:Method: GET 2024-04-12T04:11:05.947 DEBUG:boto:Path: /dkyges-61/?prefix=obj185 2024-04-12T04:11:05.947 DEBUG:boto:Data: 2024-04-12T04:11:05.947 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.947 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:05.947 DEBUG:boto:Port: 8001 2024-04-12T04:11:05.947 DEBUG:boto:Params: {} 2024-04-12T04:11:05.947 DEBUG:boto:Token: None 2024-04-12T04:11:05.947 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:05 GMT /dkyges-61/ 2024-04-12T04:11:05.948 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98= 2024-04-12T04:11:05.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:05 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xPpl5JhNUyRJKfztZzMYHxYWu98=', 'Content-Length': '0'} 2024-04-12T04:11:05.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008eba1b4ffdc83311-006618b459-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:05 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:05.998 DEBUG:boto:b'dkyges-61obj1851000falseobj1852024-04-12T04:09:24.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:05.999 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:05.999 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:05.999 DEBUG:boto:path=/dkyges-61/?prefix=obj185 2024-04-12T04:11:05.999 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj185 2024-04-12T04:11:05.999 DEBUG:boto:Method: GET 2024-04-12T04:11:05.999 DEBUG:boto:Path: /dkyges-61/?prefix=obj185 2024-04-12T04:11:05.999 DEBUG:boto:Data: 2024-04-12T04:11:05.999 DEBUG:boto:Headers: {} 2024-04-12T04:11:05.999 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:05.999 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.000 DEBUG:boto:Params: {} 2024-04-12T04:11:06.000 DEBUG:boto:Token: None 2024-04-12T04:11:06.000 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.000 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f7e7c33741ffbfe-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.005 DEBUG:boto:b'dkyges-61obj1851000falseobj1852024-04-12T04:09:24.169Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:11:06.005 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:11:06.005 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:11:06.005 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.005 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.006 DEBUG:boto:Method: GET 2024-04-12T04:11:06.006 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.006 DEBUG:boto:Data: 2024-04-12T04:11:06.006 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.006 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.006 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.007 DEBUG:boto:Params: {} 2024-04-12T04:11:06.007 DEBUG:boto:Token: None 2024-04-12T04:11:06.007 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.007 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CqFZtjUzjFBNkTEoXPWCIN81rqk= 2024-04-12T04:11:06.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CqFZtjUzjFBNkTEoXPWCIN81rqk=', 'Content-Length': '0'} 2024-04-12T04:11:06.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000614ae020f05716ee-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.010 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:11:06.010 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:11:06.010 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.010 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.010 DEBUG:boto:Method: GET 2024-04-12T04:11:06.010 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.011 DEBUG:boto:Data: 2024-04-12T04:11:06.011 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.011 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.011 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.011 DEBUG:boto:Params: {} 2024-04-12T04:11:06.011 DEBUG:boto:Token: None 2024-04-12T04:11:06.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.011 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CqFZtjUzjFBNkTEoXPWCIN81rqk= 2024-04-12T04:11:06.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CqFZtjUzjFBNkTEoXPWCIN81rqk=', 'Content-Length': '0'} 2024-04-12T04:11:06.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0b804e01fb2bacb-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.054 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:11:06.054 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:11:06.055 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.055 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.055 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.055 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.055 DEBUG:boto:Data: 2024-04-12T04:11:06.055 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.055 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.055 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.055 DEBUG:boto:Params: {} 2024-04-12T04:11:06.055 DEBUG:boto:Token: None 2024-04-12T04:11:06.055 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.056 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7BRGI16S1eJMGWb0gPb8NUI0SLw= 2024-04-12T04:11:06.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7BRGI16S1eJMGWb0gPb8NUI0SLw=', 'Content-Length': '0'} 2024-04-12T04:11:06.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ce73166a91e6942-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.058 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:11:06.058 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:11:06.058 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.058 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.058 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.059 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.059 DEBUG:boto:Data: 2024-04-12T04:11:06.059 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.059 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.059 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.059 DEBUG:boto:Params: {} 2024-04-12T04:11:06.059 DEBUG:boto:Token: None 2024-04-12T04:11:06.059 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.059 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7BRGI16S1eJMGWb0gPb8NUI0SLw= 2024-04-12T04:11:06.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7BRGI16S1eJMGWb0gPb8NUI0SLw=', 'Content-Length': '0'} 2024-04-12T04:11:06.061 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b0003072f8fcf24-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:11:06.061 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:11:06.062 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:11:06.062 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.062 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.062 DEBUG:boto:Method: GET 2024-04-12T04:11:06.062 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.062 DEBUG:boto:Data: 2024-04-12T04:11:06.062 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.062 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.062 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.062 DEBUG:boto:Params: {} 2024-04-12T04:11:06.063 DEBUG:boto:Token: None 2024-04-12T04:11:06.063 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.063 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CqFZtjUzjFBNkTEoXPWCIN81rqk= 2024-04-12T04:11:06.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CqFZtjUzjFBNkTEoXPWCIN81rqk=', 'Content-Length': '0'} 2024-04-12T04:11:06.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6c4b9799fca25fe-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.065 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:11:06.065 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:11:06.065 DEBUG:boto:path=/dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.065 DEBUG:boto:auth_path=/dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.065 DEBUG:boto:Method: GET 2024-04-12T04:11:06.066 DEBUG:boto:Path: /dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.066 DEBUG:boto:Data: 2024-04-12T04:11:06.066 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.066 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.066 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.066 DEBUG:boto:Params: {} 2024-04-12T04:11:06.066 DEBUG:boto:Token: None 2024-04-12T04:11:06.066 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj19?versionId=null 2024-04-12T04:11:06.066 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CqFZtjUzjFBNkTEoXPWCIN81rqk= 2024-04-12T04:11:06.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CqFZtjUzjFBNkTEoXPWCIN81rqk=', 'Content-Length': '0'} 2024-04-12T04:11:06.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db13474e31133965-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.069 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.069 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.069 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:11:06.069 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:11:06.069 DEBUG:boto:Method: GET 2024-04-12T04:11:06.069 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:11:06.069 DEBUG:boto:Data: 2024-04-12T04:11:06.069 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.069 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.069 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.069 DEBUG:boto:Params: {} 2024-04-12T04:11:06.070 DEBUG:boto:Token: None 2024-04-12T04:11:06.070 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.070 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc1f880083b85fcc-006618b45a-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.074 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.074 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.074 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.075 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:11:06.075 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:11:06.075 DEBUG:boto:Method: GET 2024-04-12T04:11:06.075 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:11:06.075 DEBUG:boto:Data: 2024-04-12T04:11:06.075 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.076 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.076 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.076 DEBUG:boto:Params: {} 2024-04-12T04:11:06.076 DEBUG:boto:Token: None 2024-04-12T04:11:06.076 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.077 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2b4e52e59942180-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.081 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.082 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:11:06.082 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:11:06.082 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.082 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:11:06.082 DEBUG:boto:Data: 2024-04-12T04:11:06.082 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.082 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.082 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.082 DEBUG:boto:Params: {} 2024-04-12T04:11:06.082 DEBUG:boto:Token: None 2024-04-12T04:11:06.083 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj19 2024-04-12T04:11:06.083 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5b0mX7Uo9bhPkzlnYv1mmmFjIxE= 2024-04-12T04:11:06.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5b0mX7Uo9bhPkzlnYv1mmmFjIxE=', 'Content-Length': '0'} 2024-04-12T04:11:06.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004660e193eb713f91-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.085 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:11:06.085 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:11:06.085 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.085 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:11:06.085 DEBUG:boto:Data: 2024-04-12T04:11:06.085 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.085 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.085 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.085 DEBUG:boto:Params: {} 2024-04-12T04:11:06.085 DEBUG:boto:Token: None 2024-04-12T04:11:06.086 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj19 2024-04-12T04:11:06.086 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5b0mX7Uo9bhPkzlnYv1mmmFjIxE= 2024-04-12T04:11:06.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5b0mX7Uo9bhPkzlnYv1mmmFjIxE=', 'Content-Length': '0'} 2024-04-12T04:11:06.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b1bb0b4816ed5b9-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:11:06.088 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:11:06.088 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:11:06.088 DEBUG:boto:Method: GET 2024-04-12T04:11:06.088 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:11:06.088 DEBUG:boto:Data: 2024-04-12T04:11:06.088 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.088 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.088 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.088 DEBUG:boto:Params: {} 2024-04-12T04:11:06.089 DEBUG:boto:Token: None 2024-04-12T04:11:06.089 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj19 2024-04-12T04:11:06.089 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/0P+gewPk+A0oJF2nuPdCfA9a6E= 2024-04-12T04:11:06.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/0P+gewPk+A0oJF2nuPdCfA9a6E=', 'Content-Length': '0'} 2024-04-12T04:11:06.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040fe390b60ae6770-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.091 DEBUG:boto:path=/dkyges-61/obj19 2024-04-12T04:11:06.091 DEBUG:boto:auth_path=/dkyges-61/obj19 2024-04-12T04:11:06.091 DEBUG:boto:Method: GET 2024-04-12T04:11:06.091 DEBUG:boto:Path: /dkyges-61/obj19 2024-04-12T04:11:06.091 DEBUG:boto:Data: 2024-04-12T04:11:06.091 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.091 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.091 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.092 DEBUG:boto:Params: {} 2024-04-12T04:11:06.092 DEBUG:boto:Token: None 2024-04-12T04:11:06.092 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj19 2024-04-12T04:11:06.092 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/0P+gewPk+A0oJF2nuPdCfA9a6E= 2024-04-12T04:11:06.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/0P+gewPk+A0oJF2nuPdCfA9a6E=', 'Content-Length': '0'} 2024-04-12T04:11:06.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4994a56055d3b7a-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.094 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.094 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.095 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:11:06.095 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:11:06.095 DEBUG:boto:Method: GET 2024-04-12T04:11:06.095 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:11:06.095 DEBUG:boto:Data: 2024-04-12T04:11:06.095 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.095 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.095 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.095 DEBUG:boto:Params: {} 2024-04-12T04:11:06.095 DEBUG:boto:Token: None 2024-04-12T04:11:06.095 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.096 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000813ce51f84de3a3-006618b45a-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.098 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.099 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.099 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.099 DEBUG:boto:path=/dkyges-61/?prefix=obj19 2024-04-12T04:11:06.099 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj19 2024-04-12T04:11:06.099 DEBUG:boto:Method: GET 2024-04-12T04:11:06.099 DEBUG:boto:Path: /dkyges-61/?prefix=obj19 2024-04-12T04:11:06.099 DEBUG:boto:Data: 2024-04-12T04:11:06.099 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.099 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.099 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.100 DEBUG:boto:Params: {} 2024-04-12T04:11:06.100 DEBUG:boto:Token: None 2024-04-12T04:11:06.100 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.100 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028612a4ae4bb4765-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.105 DEBUG:boto:b'dkyges-61obj191000falseobj192024-04-12T04:03:03.582Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:11:06.105 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:11:06.106 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:11:06.106 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.106 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.106 DEBUG:boto:Method: GET 2024-04-12T04:11:06.106 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.106 DEBUG:boto:Data: 2024-04-12T04:11:06.106 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.106 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.106 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.106 DEBUG:boto:Params: {} 2024-04-12T04:11:06.106 DEBUG:boto:Token: None 2024-04-12T04:11:06.107 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.107 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BPoFA909fHhxNKPAf6S6nNBpGeU= 2024-04-12T04:11:06.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BPoFA909fHhxNKPAf6S6nNBpGeU=', 'Content-Length': '0'} 2024-04-12T04:11:06.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eecf5f45a5b5e1ae-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.150 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:11:06.151 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:11:06.151 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.151 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.151 DEBUG:boto:Method: GET 2024-04-12T04:11:06.151 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.151 DEBUG:boto:Data: 2024-04-12T04:11:06.151 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.151 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.151 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.151 DEBUG:boto:Params: {} 2024-04-12T04:11:06.152 DEBUG:boto:Token: None 2024-04-12T04:11:06.152 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.152 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BPoFA909fHhxNKPAf6S6nNBpGeU= 2024-04-12T04:11:06.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BPoFA909fHhxNKPAf6S6nNBpGeU=', 'Content-Length': '0'} 2024-04-12T04:11:06.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000646049ccc99aa9d0-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.154 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:11:06.154 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:11:06.155 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.155 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.155 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.155 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.155 DEBUG:boto:Data: 2024-04-12T04:11:06.155 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.155 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.155 DEBUG:boto:Params: {} 2024-04-12T04:11:06.155 DEBUG:boto:Token: None 2024-04-12T04:11:06.155 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SD/pScFrIWb5oRHBtWv5f+XBLUk= 2024-04-12T04:11:06.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SD/pScFrIWb5oRHBtWv5f+XBLUk=', 'Content-Length': '0'} 2024-04-12T04:11:06.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054ce8ab1b4ae8b0a-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.158 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:11:06.158 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:11:06.158 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.158 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.158 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.158 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.158 DEBUG:boto:Data: 2024-04-12T04:11:06.158 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.158 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.158 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.159 DEBUG:boto:Params: {} 2024-04-12T04:11:06.159 DEBUG:boto:Token: None 2024-04-12T04:11:06.159 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.159 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SD/pScFrIWb5oRHBtWv5f+XBLUk= 2024-04-12T04:11:06.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SD/pScFrIWb5oRHBtWv5f+XBLUk=', 'Content-Length': '0'} 2024-04-12T04:11:06.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000757e314189b28f6-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:11:06.161 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:11:06.161 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:11:06.161 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.161 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.161 DEBUG:boto:Method: GET 2024-04-12T04:11:06.161 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.161 DEBUG:boto:Data: 2024-04-12T04:11:06.161 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.162 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.162 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.162 DEBUG:boto:Params: {} 2024-04-12T04:11:06.162 DEBUG:boto:Token: None 2024-04-12T04:11:06.162 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.162 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BPoFA909fHhxNKPAf6S6nNBpGeU= 2024-04-12T04:11:06.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BPoFA909fHhxNKPAf6S6nNBpGeU=', 'Content-Length': '0'} 2024-04-12T04:11:06.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e162fbc4cd804f8-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.164 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:11:06.164 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:11:06.164 DEBUG:boto:path=/dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.164 DEBUG:boto:auth_path=/dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.165 DEBUG:boto:Method: GET 2024-04-12T04:11:06.165 DEBUG:boto:Path: /dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.165 DEBUG:boto:Data: 2024-04-12T04:11:06.165 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.165 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.165 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.165 DEBUG:boto:Params: {} 2024-04-12T04:11:06.165 DEBUG:boto:Token: None 2024-04-12T04:11:06.165 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj22?versionId=null 2024-04-12T04:11:06.165 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BPoFA909fHhxNKPAf6S6nNBpGeU= 2024-04-12T04:11:06.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BPoFA909fHhxNKPAf6S6nNBpGeU=', 'Content-Length': '0'} 2024-04-12T04:11:06.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066337113ddbd5b35-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.210 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.210 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.211 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:11:06.211 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:11:06.211 DEBUG:boto:Method: GET 2024-04-12T04:11:06.211 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:11:06.211 DEBUG:boto:Data: 2024-04-12T04:11:06.211 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.211 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.211 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.211 DEBUG:boto:Params: {} 2024-04-12T04:11:06.211 DEBUG:boto:Token: None 2024-04-12T04:11:06.211 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056a4933042e5bcfd-006618b45a-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.214 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.214 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.217 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.218 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:11:06.218 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:11:06.218 DEBUG:boto:Method: GET 2024-04-12T04:11:06.218 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:11:06.218 DEBUG:boto:Data: 2024-04-12T04:11:06.218 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.218 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.218 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.218 DEBUG:boto:Params: {} 2024-04-12T04:11:06.218 DEBUG:boto:Token: None 2024-04-12T04:11:06.219 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.219 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c3168d2eeedab27-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.223 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.224 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:11:06.224 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:11:06.224 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.224 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:11:06.224 DEBUG:boto:Data: 2024-04-12T04:11:06.224 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.224 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.224 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.224 DEBUG:boto:Params: {} 2024-04-12T04:11:06.224 DEBUG:boto:Token: None 2024-04-12T04:11:06.224 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj22 2024-04-12T04:11:06.225 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/geeKs2+uQ0T8tvPceoV4EVcwIE= 2024-04-12T04:11:06.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/geeKs2+uQ0T8tvPceoV4EVcwIE=', 'Content-Length': '0'} 2024-04-12T04:11:06.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cdddda94090e34a-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.226 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:11:06.227 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:11:06.227 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.227 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:11:06.227 DEBUG:boto:Data: 2024-04-12T04:11:06.227 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.227 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.227 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.227 DEBUG:boto:Params: {} 2024-04-12T04:11:06.227 DEBUG:boto:Token: None 2024-04-12T04:11:06.227 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj22 2024-04-12T04:11:06.227 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/geeKs2+uQ0T8tvPceoV4EVcwIE= 2024-04-12T04:11:06.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/geeKs2+uQ0T8tvPceoV4EVcwIE=', 'Content-Length': '0'} 2024-04-12T04:11:06.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000583d6787edbbe66b-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:11:06.229 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:11:06.230 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:11:06.230 DEBUG:boto:Method: GET 2024-04-12T04:11:06.230 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:11:06.230 DEBUG:boto:Data: 2024-04-12T04:11:06.230 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.230 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.230 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.230 DEBUG:boto:Params: {} 2024-04-12T04:11:06.230 DEBUG:boto:Token: None 2024-04-12T04:11:06.230 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj22 2024-04-12T04:11:06.231 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RP5wDRcCpU4LlB8qJ0jKgGjnhb4= 2024-04-12T04:11:06.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RP5wDRcCpU4LlB8qJ0jKgGjnhb4=', 'Content-Length': '0'} 2024-04-12T04:11:06.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b99353ce087f897-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.233 DEBUG:boto:path=/dkyges-61/obj22 2024-04-12T04:11:06.233 DEBUG:boto:auth_path=/dkyges-61/obj22 2024-04-12T04:11:06.233 DEBUG:boto:Method: GET 2024-04-12T04:11:06.233 DEBUG:boto:Path: /dkyges-61/obj22 2024-04-12T04:11:06.233 DEBUG:boto:Data: 2024-04-12T04:11:06.233 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.233 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.233 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.233 DEBUG:boto:Params: {} 2024-04-12T04:11:06.233 DEBUG:boto:Token: None 2024-04-12T04:11:06.234 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj22 2024-04-12T04:11:06.234 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RP5wDRcCpU4LlB8qJ0jKgGjnhb4= 2024-04-12T04:11:06.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RP5wDRcCpU4LlB8qJ0jKgGjnhb4=', 'Content-Length': '0'} 2024-04-12T04:11:06.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009289d19762197e25-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.236 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.236 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.236 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:11:06.236 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:11:06.236 DEBUG:boto:Method: GET 2024-04-12T04:11:06.236 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:11:06.237 DEBUG:boto:Data: 2024-04-12T04:11:06.237 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.237 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.237 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.237 DEBUG:boto:Params: {} 2024-04-12T04:11:06.237 DEBUG:boto:Token: None 2024-04-12T04:11:06.237 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.237 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000530c85a3ea02e930-006618b45a-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.240 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.240 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.240 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.240 DEBUG:boto:path=/dkyges-61/?prefix=obj22 2024-04-12T04:11:06.240 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj22 2024-04-12T04:11:06.240 DEBUG:boto:Method: GET 2024-04-12T04:11:06.241 DEBUG:boto:Path: /dkyges-61/?prefix=obj22 2024-04-12T04:11:06.241 DEBUG:boto:Data: 2024-04-12T04:11:06.241 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.241 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.241 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.241 DEBUG:boto:Params: {} 2024-04-12T04:11:06.241 DEBUG:boto:Token: None 2024-04-12T04:11:06.241 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.241 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a7aa7876b302712-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.246 DEBUG:boto:b'dkyges-61obj221000falseobj222024-04-12T04:00:30.456Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:11:06.246 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:11:06.246 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:11:06.246 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.246 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.247 DEBUG:boto:Method: GET 2024-04-12T04:11:06.247 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.247 DEBUG:boto:Data: 2024-04-12T04:11:06.247 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.247 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.247 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.247 DEBUG:boto:Params: {} 2024-04-12T04:11:06.247 DEBUG:boto:Token: None 2024-04-12T04:11:06.247 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.247 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:K7b09u5j5bjqYPU0JpA6xB3r3Vw= 2024-04-12T04:11:06.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:K7b09u5j5bjqYPU0JpA6xB3r3Vw=', 'Content-Length': '0'} 2024-04-12T04:11:06.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a24d838e14bc64b7-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.290 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:11:06.290 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:11:06.291 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.291 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.291 DEBUG:boto:Method: GET 2024-04-12T04:11:06.291 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.291 DEBUG:boto:Data: 2024-04-12T04:11:06.291 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.291 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.291 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.291 DEBUG:boto:Params: {} 2024-04-12T04:11:06.291 DEBUG:boto:Token: None 2024-04-12T04:11:06.291 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.292 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:K7b09u5j5bjqYPU0JpA6xB3r3Vw= 2024-04-12T04:11:06.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:K7b09u5j5bjqYPU0JpA6xB3r3Vw=', 'Content-Length': '0'} 2024-04-12T04:11:06.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1ef32ed9637a8fa-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.294 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:11:06.294 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:11:06.294 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.294 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.294 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.294 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.294 DEBUG:boto:Data: 2024-04-12T04:11:06.295 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.295 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.295 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.295 DEBUG:boto:Params: {} 2024-04-12T04:11:06.295 DEBUG:boto:Token: None 2024-04-12T04:11:06.295 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.295 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Uv5RWHLEOIMNr1bNJWNKDqh8qYM= 2024-04-12T04:11:06.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Uv5RWHLEOIMNr1bNJWNKDqh8qYM=', 'Content-Length': '0'} 2024-04-12T04:11:06.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e23d570fc94bd32d-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.297 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:11:06.297 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:11:06.297 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.297 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.297 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.297 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.298 DEBUG:boto:Data: 2024-04-12T04:11:06.298 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.298 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.298 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.298 DEBUG:boto:Params: {} 2024-04-12T04:11:06.298 DEBUG:boto:Token: None 2024-04-12T04:11:06.298 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.298 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Uv5RWHLEOIMNr1bNJWNKDqh8qYM= 2024-04-12T04:11:06.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Uv5RWHLEOIMNr1bNJWNKDqh8qYM=', 'Content-Length': '0'} 2024-04-12T04:11:06.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcf77acdc364bf29-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:11:06.300 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:11:06.300 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:11:06.301 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.301 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.301 DEBUG:boto:Method: GET 2024-04-12T04:11:06.301 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.301 DEBUG:boto:Data: 2024-04-12T04:11:06.301 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.301 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.301 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.301 DEBUG:boto:Params: {} 2024-04-12T04:11:06.301 DEBUG:boto:Token: None 2024-04-12T04:11:06.301 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.302 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:K7b09u5j5bjqYPU0JpA6xB3r3Vw= 2024-04-12T04:11:06.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:K7b09u5j5bjqYPU0JpA6xB3r3Vw=', 'Content-Length': '0'} 2024-04-12T04:11:06.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063fac429b8df9f4a-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.303 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:11:06.304 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:11:06.304 DEBUG:boto:path=/dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.304 DEBUG:boto:auth_path=/dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.304 DEBUG:boto:Method: GET 2024-04-12T04:11:06.304 DEBUG:boto:Path: /dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.304 DEBUG:boto:Data: 2024-04-12T04:11:06.304 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.304 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.304 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.304 DEBUG:boto:Params: {} 2024-04-12T04:11:06.304 DEBUG:boto:Token: None 2024-04-12T04:11:06.305 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj23?versionId=null 2024-04-12T04:11:06.305 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:K7b09u5j5bjqYPU0JpA6xB3r3Vw= 2024-04-12T04:11:06.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:K7b09u5j5bjqYPU0JpA6xB3r3Vw=', 'Content-Length': '0'} 2024-04-12T04:11:06.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b324980d82808e6-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.307 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.307 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.307 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:11:06.307 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:11:06.307 DEBUG:boto:Method: GET 2024-04-12T04:11:06.307 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:11:06.307 DEBUG:boto:Data: 2024-04-12T04:11:06.307 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.307 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.307 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.308 DEBUG:boto:Params: {} 2024-04-12T04:11:06.308 DEBUG:boto:Token: None 2024-04-12T04:11:06.308 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.308 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b4ebef1d0797606-006618b45a-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.310 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.311 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.311 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.311 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:11:06.311 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:11:06.311 DEBUG:boto:Method: GET 2024-04-12T04:11:06.311 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:11:06.311 DEBUG:boto:Data: 2024-04-12T04:11:06.311 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.311 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.311 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.311 DEBUG:boto:Params: {} 2024-04-12T04:11:06.312 DEBUG:boto:Token: None 2024-04-12T04:11:06.312 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.312 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d293c10f7a842577-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.358 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.359 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:11:06.359 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:11:06.359 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.359 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:11:06.359 DEBUG:boto:Data: 2024-04-12T04:11:06.359 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.359 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.359 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.359 DEBUG:boto:Params: {} 2024-04-12T04:11:06.359 DEBUG:boto:Token: None 2024-04-12T04:11:06.360 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj23 2024-04-12T04:11:06.360 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AoAHnWjbEEQdBDH1OZppxwZpiJk= 2024-04-12T04:11:06.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AoAHnWjbEEQdBDH1OZppxwZpiJk=', 'Content-Length': '0'} 2024-04-12T04:11:06.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004960790193d25fa-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.362 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:11:06.362 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:11:06.362 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.362 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:11:06.363 DEBUG:boto:Data: 2024-04-12T04:11:06.363 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.363 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.363 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.363 DEBUG:boto:Params: {} 2024-04-12T04:11:06.363 DEBUG:boto:Token: None 2024-04-12T04:11:06.363 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj23 2024-04-12T04:11:06.363 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AoAHnWjbEEQdBDH1OZppxwZpiJk= 2024-04-12T04:11:06.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AoAHnWjbEEQdBDH1OZppxwZpiJk=', 'Content-Length': '0'} 2024-04-12T04:11:06.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f45d62b56c8535c1-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:11:06.365 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:11:06.365 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:11:06.365 DEBUG:boto:Method: GET 2024-04-12T04:11:06.366 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:11:06.366 DEBUG:boto:Data: 2024-04-12T04:11:06.366 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.366 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.366 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.366 DEBUG:boto:Params: {} 2024-04-12T04:11:06.366 DEBUG:boto:Token: None 2024-04-12T04:11:06.366 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj23 2024-04-12T04:11:06.366 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MvXIlPY3Wt1k8JplV9OG7RJcCWY= 2024-04-12T04:11:06.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MvXIlPY3Wt1k8JplV9OG7RJcCWY=', 'Content-Length': '0'} 2024-04-12T04:11:06.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf0aa343aaa1cece-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.369 DEBUG:boto:path=/dkyges-61/obj23 2024-04-12T04:11:06.369 DEBUG:boto:auth_path=/dkyges-61/obj23 2024-04-12T04:11:06.369 DEBUG:boto:Method: GET 2024-04-12T04:11:06.369 DEBUG:boto:Path: /dkyges-61/obj23 2024-04-12T04:11:06.369 DEBUG:boto:Data: 2024-04-12T04:11:06.369 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.369 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.369 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.369 DEBUG:boto:Params: {} 2024-04-12T04:11:06.369 DEBUG:boto:Token: None 2024-04-12T04:11:06.370 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj23 2024-04-12T04:11:06.370 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MvXIlPY3Wt1k8JplV9OG7RJcCWY= 2024-04-12T04:11:06.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MvXIlPY3Wt1k8JplV9OG7RJcCWY=', 'Content-Length': '0'} 2024-04-12T04:11:06.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0800a024a259294-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.372 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.372 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.372 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:11:06.372 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:11:06.372 DEBUG:boto:Method: GET 2024-04-12T04:11:06.372 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:11:06.372 DEBUG:boto:Data: 2024-04-12T04:11:06.372 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.373 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.373 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.373 DEBUG:boto:Params: {} 2024-04-12T04:11:06.373 DEBUG:boto:Token: None 2024-04-12T04:11:06.373 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.373 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c5864d56d26c4f0-006618b45a-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.376 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.376 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.376 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.376 DEBUG:boto:path=/dkyges-61/?prefix=obj23 2024-04-12T04:11:06.376 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj23 2024-04-12T04:11:06.376 DEBUG:boto:Method: GET 2024-04-12T04:11:06.376 DEBUG:boto:Path: /dkyges-61/?prefix=obj23 2024-04-12T04:11:06.377 DEBUG:boto:Data: 2024-04-12T04:11:06.377 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.377 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.377 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.377 DEBUG:boto:Params: {} 2024-04-12T04:11:06.377 DEBUG:boto:Token: None 2024-04-12T04:11:06.377 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.377 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009342a016a9302c98-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.382 DEBUG:boto:b'dkyges-61obj231000falseobj232024-04-12T04:05:01.070Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:11:06.383 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:11:06.383 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:11:06.383 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.383 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.383 DEBUG:boto:Method: GET 2024-04-12T04:11:06.383 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.383 DEBUG:boto:Data: 2024-04-12T04:11:06.383 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.383 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.383 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.383 DEBUG:boto:Params: {} 2024-04-12T04:11:06.383 DEBUG:boto:Token: None 2024-04-12T04:11:06.384 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.384 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iIdWeZsd4HFjro+xGpwSYmrmrJ8= 2024-04-12T04:11:06.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iIdWeZsd4HFjro+xGpwSYmrmrJ8=', 'Content-Length': '0'} 2024-04-12T04:11:06.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceba60fb2183de17-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.426 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:11:06.426 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:11:06.427 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.427 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.427 DEBUG:boto:Method: GET 2024-04-12T04:11:06.427 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.427 DEBUG:boto:Data: 2024-04-12T04:11:06.427 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.427 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.427 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.427 DEBUG:boto:Params: {} 2024-04-12T04:11:06.427 DEBUG:boto:Token: None 2024-04-12T04:11:06.428 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.428 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iIdWeZsd4HFjro+xGpwSYmrmrJ8= 2024-04-12T04:11:06.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iIdWeZsd4HFjro+xGpwSYmrmrJ8=', 'Content-Length': '0'} 2024-04-12T04:11:06.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5bccb11e48054ee-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.430 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:11:06.430 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:11:06.430 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.430 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.430 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.430 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.431 DEBUG:boto:Data: 2024-04-12T04:11:06.431 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.431 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.431 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.431 DEBUG:boto:Params: {} 2024-04-12T04:11:06.431 DEBUG:boto:Token: None 2024-04-12T04:11:06.431 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.431 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KfUxOapwlpNy4kdhOhc2q1sCDjw= 2024-04-12T04:11:06.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KfUxOapwlpNy4kdhOhc2q1sCDjw=', 'Content-Length': '0'} 2024-04-12T04:11:06.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbed4096faa562b9-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.433 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:11:06.434 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:11:06.434 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.434 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.434 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.434 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.434 DEBUG:boto:Data: 2024-04-12T04:11:06.434 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.434 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.434 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.434 DEBUG:boto:Params: {} 2024-04-12T04:11:06.434 DEBUG:boto:Token: None 2024-04-12T04:11:06.435 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.435 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KfUxOapwlpNy4kdhOhc2q1sCDjw= 2024-04-12T04:11:06.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KfUxOapwlpNy4kdhOhc2q1sCDjw=', 'Content-Length': '0'} 2024-04-12T04:11:06.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a83cdc59c662a15-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:11:06.437 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:11:06.437 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:11:06.437 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.437 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.437 DEBUG:boto:Method: GET 2024-04-12T04:11:06.438 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.438 DEBUG:boto:Data: 2024-04-12T04:11:06.438 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.438 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.438 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.438 DEBUG:boto:Params: {} 2024-04-12T04:11:06.438 DEBUG:boto:Token: None 2024-04-12T04:11:06.438 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.438 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iIdWeZsd4HFjro+xGpwSYmrmrJ8= 2024-04-12T04:11:06.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iIdWeZsd4HFjro+xGpwSYmrmrJ8=', 'Content-Length': '0'} 2024-04-12T04:11:06.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ca39548fcade120-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.441 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:11:06.441 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:11:06.441 DEBUG:boto:path=/dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.441 DEBUG:boto:auth_path=/dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.441 DEBUG:boto:Method: GET 2024-04-12T04:11:06.441 DEBUG:boto:Path: /dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.441 DEBUG:boto:Data: 2024-04-12T04:11:06.441 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.441 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.441 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.441 DEBUG:boto:Params: {} 2024-04-12T04:11:06.441 DEBUG:boto:Token: None 2024-04-12T04:11:06.442 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj29?versionId=null 2024-04-12T04:11:06.442 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iIdWeZsd4HFjro+xGpwSYmrmrJ8= 2024-04-12T04:11:06.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iIdWeZsd4HFjro+xGpwSYmrmrJ8=', 'Content-Length': '0'} 2024-04-12T04:11:06.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036da315a02540d93-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.444 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.444 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.444 DEBUG:boto:path=/dkyges-61/?prefix=obj29 2024-04-12T04:11:06.444 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj29 2024-04-12T04:11:06.445 DEBUG:boto:Method: GET 2024-04-12T04:11:06.445 DEBUG:boto:Path: /dkyges-61/?prefix=obj29 2024-04-12T04:11:06.445 DEBUG:boto:Data: 2024-04-12T04:11:06.445 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.445 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.445 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.445 DEBUG:boto:Params: {} 2024-04-12T04:11:06.445 DEBUG:boto:Token: None 2024-04-12T04:11:06.445 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.445 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f00e6b3b450e3a12-006618b45a-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.448 DEBUG:boto:b'dkyges-61obj291000falseobj292024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.448 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.449 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.449 DEBUG:boto:path=/dkyges-61/?prefix=obj29 2024-04-12T04:11:06.449 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj29 2024-04-12T04:11:06.449 DEBUG:boto:Method: GET 2024-04-12T04:11:06.449 DEBUG:boto:Path: /dkyges-61/?prefix=obj29 2024-04-12T04:11:06.449 DEBUG:boto:Data: 2024-04-12T04:11:06.449 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.449 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.449 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.449 DEBUG:boto:Params: {} 2024-04-12T04:11:06.450 DEBUG:boto:Token: None 2024-04-12T04:11:06.450 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.450 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b6732851ab103fa-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.454 DEBUG:boto:b'dkyges-61obj291000falseobj292024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.455 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:11:06.455 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:11:06.455 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.455 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:11:06.455 DEBUG:boto:Data: 2024-04-12T04:11:06.455 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.455 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.455 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.455 DEBUG:boto:Params: {} 2024-04-12T04:11:06.455 DEBUG:boto:Token: None 2024-04-12T04:11:06.456 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj29 2024-04-12T04:11:06.456 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zv2gqZ2NQgxbrDvZGiGCc2oj9t8= 2024-04-12T04:11:06.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zv2gqZ2NQgxbrDvZGiGCc2oj9t8=', 'Content-Length': '0'} 2024-04-12T04:11:06.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e3e7e09b891fac0-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.458 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:11:06.458 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:11:06.458 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.458 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:11:06.458 DEBUG:boto:Data: 2024-04-12T04:11:06.458 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.458 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.458 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.458 DEBUG:boto:Params: {} 2024-04-12T04:11:06.459 DEBUG:boto:Token: None 2024-04-12T04:11:06.459 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj29 2024-04-12T04:11:06.459 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zv2gqZ2NQgxbrDvZGiGCc2oj9t8= 2024-04-12T04:11:06.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zv2gqZ2NQgxbrDvZGiGCc2oj9t8=', 'Content-Length': '0'} 2024-04-12T04:11:06.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041ddd47f0dddb3de-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:11:06.461 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:11:06.461 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:11:06.461 DEBUG:boto:Method: GET 2024-04-12T04:11:06.461 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:11:06.461 DEBUG:boto:Data: 2024-04-12T04:11:06.461 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.462 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.462 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.462 DEBUG:boto:Params: {} 2024-04-12T04:11:06.462 DEBUG:boto:Token: None 2024-04-12T04:11:06.462 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj29 2024-04-12T04:11:06.462 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L6D8KQxXtQVEE2keUKHZtspBeDU= 2024-04-12T04:11:06.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L6D8KQxXtQVEE2keUKHZtspBeDU=', 'Content-Length': '0'} 2024-04-12T04:11:06.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057cab52c1b4f435d-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.464 DEBUG:boto:path=/dkyges-61/obj29 2024-04-12T04:11:06.464 DEBUG:boto:auth_path=/dkyges-61/obj29 2024-04-12T04:11:06.464 DEBUG:boto:Method: GET 2024-04-12T04:11:06.464 DEBUG:boto:Path: /dkyges-61/obj29 2024-04-12T04:11:06.465 DEBUG:boto:Data: 2024-04-12T04:11:06.465 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.465 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.465 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.465 DEBUG:boto:Params: {} 2024-04-12T04:11:06.465 DEBUG:boto:Token: None 2024-04-12T04:11:06.465 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj29 2024-04-12T04:11:06.465 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L6D8KQxXtQVEE2keUKHZtspBeDU= 2024-04-12T04:11:06.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L6D8KQxXtQVEE2keUKHZtspBeDU=', 'Content-Length': '0'} 2024-04-12T04:11:06.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d16af0ce4226ae7d-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.467 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.468 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.468 DEBUG:boto:path=/dkyges-61/?prefix=obj29 2024-04-12T04:11:06.468 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj29 2024-04-12T04:11:06.468 DEBUG:boto:Method: GET 2024-04-12T04:11:06.468 DEBUG:boto:Path: /dkyges-61/?prefix=obj29 2024-04-12T04:11:06.468 DEBUG:boto:Data: 2024-04-12T04:11:06.468 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.468 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.468 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.468 DEBUG:boto:Params: {} 2024-04-12T04:11:06.468 DEBUG:boto:Token: None 2024-04-12T04:11:06.468 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.469 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a6bf7aa42f1e17a-006618b45a-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.471 DEBUG:boto:b'dkyges-61obj291000falseobj292024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.472 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.472 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.472 DEBUG:boto:path=/dkyges-61/?prefix=obj29 2024-04-12T04:11:06.472 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj29 2024-04-12T04:11:06.472 DEBUG:boto:Method: GET 2024-04-12T04:11:06.472 DEBUG:boto:Path: /dkyges-61/?prefix=obj29 2024-04-12T04:11:06.472 DEBUG:boto:Data: 2024-04-12T04:11:06.472 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.472 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.472 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.472 DEBUG:boto:Params: {} 2024-04-12T04:11:06.472 DEBUG:boto:Token: None 2024-04-12T04:11:06.473 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.473 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8ca4006493d54a6-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.518 DEBUG:boto:b'dkyges-61obj291000falseobj292024-04-12T04:06:49.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T04:11:06.519 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:11:06.519 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:11:06.519 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.519 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.519 DEBUG:boto:Method: GET 2024-04-12T04:11:06.519 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.519 DEBUG:boto:Data: 2024-04-12T04:11:06.519 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.519 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.519 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.520 DEBUG:boto:Params: {} 2024-04-12T04:11:06.520 DEBUG:boto:Token: None 2024-04-12T04:11:06.520 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.520 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/bsPTbEe77H2aiPXcF2RxP+VesM= 2024-04-12T04:11:06.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/bsPTbEe77H2aiPXcF2RxP+VesM=', 'Content-Length': '0'} 2024-04-12T04:11:06.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000141a300a362798ce-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.522 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:11:06.522 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:11:06.523 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.523 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.523 DEBUG:boto:Method: GET 2024-04-12T04:11:06.523 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.523 DEBUG:boto:Data: 2024-04-12T04:11:06.523 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.523 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.523 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.523 DEBUG:boto:Params: {} 2024-04-12T04:11:06.523 DEBUG:boto:Token: None 2024-04-12T04:11:06.523 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.524 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/bsPTbEe77H2aiPXcF2RxP+VesM= 2024-04-12T04:11:06.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/bsPTbEe77H2aiPXcF2RxP+VesM=', 'Content-Length': '0'} 2024-04-12T04:11:06.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b68adf1dd4671e6-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.526 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:11:06.526 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:11:06.526 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.526 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.526 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.526 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.526 DEBUG:boto:Data: 2024-04-12T04:11:06.526 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.527 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.527 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.527 DEBUG:boto:Params: {} 2024-04-12T04:11:06.527 DEBUG:boto:Token: None 2024-04-12T04:11:06.527 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.527 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j/gVdECpRncqVE3e2t4da196uNE= 2024-04-12T04:11:06.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j/gVdECpRncqVE3e2t4da196uNE=', 'Content-Length': '0'} 2024-04-12T04:11:06.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000848fe568f3ff5ede-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.529 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:11:06.529 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:11:06.529 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.529 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.529 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.529 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.530 DEBUG:boto:Data: 2024-04-12T04:11:06.530 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.530 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.530 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.530 DEBUG:boto:Params: {} 2024-04-12T04:11:06.530 DEBUG:boto:Token: None 2024-04-12T04:11:06.530 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.530 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j/gVdECpRncqVE3e2t4da196uNE= 2024-04-12T04:11:06.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j/gVdECpRncqVE3e2t4da196uNE=', 'Content-Length': '0'} 2024-04-12T04:11:06.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbf2a4e4be884b4b-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T04:11:06.533 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:11:06.533 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:11:06.533 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.533 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.533 DEBUG:boto:Method: GET 2024-04-12T04:11:06.533 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.533 DEBUG:boto:Data: 2024-04-12T04:11:06.533 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.533 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.534 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.534 DEBUG:boto:Params: {} 2024-04-12T04:11:06.534 DEBUG:boto:Token: None 2024-04-12T04:11:06.534 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.534 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/bsPTbEe77H2aiPXcF2RxP+VesM= 2024-04-12T04:11:06.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/bsPTbEe77H2aiPXcF2RxP+VesM=', 'Content-Length': '0'} 2024-04-12T04:11:06.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004173015d44f2459c-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.536 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:11:06.536 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:11:06.537 DEBUG:boto:path=/dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.537 DEBUG:boto:auth_path=/dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.537 DEBUG:boto:Method: GET 2024-04-12T04:11:06.537 DEBUG:boto:Path: /dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.537 DEBUG:boto:Data: 2024-04-12T04:11:06.537 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.537 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.537 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.537 DEBUG:boto:Params: {} 2024-04-12T04:11:06.537 DEBUG:boto:Token: None 2024-04-12T04:11:06.537 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj31?versionId=null 2024-04-12T04:11:06.537 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/bsPTbEe77H2aiPXcF2RxP+VesM= 2024-04-12T04:11:06.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/bsPTbEe77H2aiPXcF2RxP+VesM=', 'Content-Length': '0'} 2024-04-12T04:11:06.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1586bdb82c449a4-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.540 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.540 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.540 DEBUG:boto:path=/dkyges-61/?prefix=obj31 2024-04-12T04:11:06.540 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj31 2024-04-12T04:11:06.540 DEBUG:boto:Method: GET 2024-04-12T04:11:06.540 DEBUG:boto:Path: /dkyges-61/?prefix=obj31 2024-04-12T04:11:06.540 DEBUG:boto:Data: 2024-04-12T04:11:06.541 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.541 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.541 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.541 DEBUG:boto:Params: {} 2024-04-12T04:11:06.541 DEBUG:boto:Token: None 2024-04-12T04:11:06.541 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.541 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f79ed4aadac0ca8f-006618b45a-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.544 DEBUG:boto:b'dkyges-61obj311000falseobj312024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.544 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.544 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.544 DEBUG:boto:path=/dkyges-61/?prefix=obj31 2024-04-12T04:11:06.544 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj31 2024-04-12T04:11:06.544 DEBUG:boto:Method: GET 2024-04-12T04:11:06.544 DEBUG:boto:Path: /dkyges-61/?prefix=obj31 2024-04-12T04:11:06.544 DEBUG:boto:Data: 2024-04-12T04:11:06.545 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.545 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.545 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.545 DEBUG:boto:Params: {} 2024-04-12T04:11:06.545 DEBUG:boto:Token: None 2024-04-12T04:11:06.545 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.545 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076944d84aa8b3952-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.550 DEBUG:boto:b'dkyges-61obj311000falseobj312024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.550 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:11:06.550 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:11:06.550 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.550 DEBUG:boto:Path: /dkyges-61/obj31 2024-04-12T04:11:06.550 DEBUG:boto:Data: 2024-04-12T04:11:06.550 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.550 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.551 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.551 DEBUG:boto:Params: {} 2024-04-12T04:11:06.551 DEBUG:boto:Token: None 2024-04-12T04:11:06.551 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj31 2024-04-12T04:11:06.551 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ey5rQzNfHF1T6P2/uv1G8FUs2bw= 2024-04-12T04:11:06.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ey5rQzNfHF1T6P2/uv1G8FUs2bw=', 'Content-Length': '0'} 2024-04-12T04:11:06.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d24b372446f9063f-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.553 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:11:06.554 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:11:06.554 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.554 DEBUG:boto:Path: /dkyges-61/obj31 2024-04-12T04:11:06.554 DEBUG:boto:Data: 2024-04-12T04:11:06.554 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.554 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.554 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.554 DEBUG:boto:Params: {} 2024-04-12T04:11:06.554 DEBUG:boto:Token: None 2024-04-12T04:11:06.554 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj31 2024-04-12T04:11:06.555 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ey5rQzNfHF1T6P2/uv1G8FUs2bw= 2024-04-12T04:11:06.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ey5rQzNfHF1T6P2/uv1G8FUs2bw=', 'Content-Length': '0'} 2024-04-12T04:11:06.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b02bb800cbce752f-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T04:11:06.557 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:11:06.557 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:11:06.557 DEBUG:boto:Method: GET 2024-04-12T04:11:06.557 DEBUG:boto:Path: /dkyges-61/obj31 2024-04-12T04:11:06.558 DEBUG:boto:Data: 2024-04-12T04:11:06.558 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.558 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.558 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.558 DEBUG:boto:Params: {} 2024-04-12T04:11:06.558 DEBUG:boto:Token: None 2024-04-12T04:11:06.558 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj31 2024-04-12T04:11:06.558 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GRMAo2z982AePwpvQH5elKDFrOs= 2024-04-12T04:11:06.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GRMAo2z982AePwpvQH5elKDFrOs=', 'Content-Length': '0'} 2024-04-12T04:11:06.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebd23bcdbf6335ed-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.602 DEBUG:boto:path=/dkyges-61/obj31 2024-04-12T04:11:06.602 DEBUG:boto:auth_path=/dkyges-61/obj31 2024-04-12T04:11:06.603 DEBUG:boto:Method: GET 2024-04-12T04:11:06.603 DEBUG:boto:Path: /dkyges-61/obj31 2024-04-12T04:11:06.603 DEBUG:boto:Data: 2024-04-12T04:11:06.603 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.603 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.603 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.603 DEBUG:boto:Params: {} 2024-04-12T04:11:06.603 DEBUG:boto:Token: None 2024-04-12T04:11:06.603 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj31 2024-04-12T04:11:06.603 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GRMAo2z982AePwpvQH5elKDFrOs= 2024-04-12T04:11:06.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GRMAo2z982AePwpvQH5elKDFrOs=', 'Content-Length': '0'} 2024-04-12T04:11:06.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be1d4344ede49124-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.605 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.606 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.606 DEBUG:boto:path=/dkyges-61/?prefix=obj31 2024-04-12T04:11:06.606 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj31 2024-04-12T04:11:06.606 DEBUG:boto:Method: GET 2024-04-12T04:11:06.606 DEBUG:boto:Path: /dkyges-61/?prefix=obj31 2024-04-12T04:11:06.606 DEBUG:boto:Data: 2024-04-12T04:11:06.606 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.606 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.606 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.606 DEBUG:boto:Params: {} 2024-04-12T04:11:06.606 DEBUG:boto:Token: None 2024-04-12T04:11:06.607 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.607 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086adb41d51633d11-006618b45a-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.609 DEBUG:boto:b'dkyges-61obj311000falseobj312024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.609 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.610 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.610 DEBUG:boto:path=/dkyges-61/?prefix=obj31 2024-04-12T04:11:06.610 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj31 2024-04-12T04:11:06.610 DEBUG:boto:Method: GET 2024-04-12T04:11:06.610 DEBUG:boto:Path: /dkyges-61/?prefix=obj31 2024-04-12T04:11:06.610 DEBUG:boto:Data: 2024-04-12T04:11:06.610 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.610 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.610 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.610 DEBUG:boto:Params: {} 2024-04-12T04:11:06.610 DEBUG:boto:Token: None 2024-04-12T04:11:06.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.611 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6b4aaee03b49f84-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.615 DEBUG:boto:b'dkyges-61obj311000falseobj312024-04-12T04:08:08.174Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:11:06.616 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:11:06.616 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:11:06.616 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.616 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.616 DEBUG:boto:Method: GET 2024-04-12T04:11:06.616 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.616 DEBUG:boto:Data: 2024-04-12T04:11:06.616 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.616 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.616 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.617 DEBUG:boto:Params: {} 2024-04-12T04:11:06.617 DEBUG:boto:Token: None 2024-04-12T04:11:06.617 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.617 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Dk0BleAcp01cdelQNKDV2SJZzAw= 2024-04-12T04:11:06.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Dk0BleAcp01cdelQNKDV2SJZzAw=', 'Content-Length': '0'} 2024-04-12T04:11:06.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3bcf90ae81407db-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.619 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:11:06.619 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:11:06.619 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.619 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.620 DEBUG:boto:Method: GET 2024-04-12T04:11:06.620 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.620 DEBUG:boto:Data: 2024-04-12T04:11:06.620 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.620 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.620 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.620 DEBUG:boto:Params: {} 2024-04-12T04:11:06.620 DEBUG:boto:Token: None 2024-04-12T04:11:06.620 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.620 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Dk0BleAcp01cdelQNKDV2SJZzAw= 2024-04-12T04:11:06.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Dk0BleAcp01cdelQNKDV2SJZzAw=', 'Content-Length': '0'} 2024-04-12T04:11:06.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f64fd6680dcf00d9-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.626 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:11:06.626 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:11:06.626 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.627 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.627 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.627 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.627 DEBUG:boto:Data: 2024-04-12T04:11:06.627 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.627 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.627 DEBUG:boto:Params: {} 2024-04-12T04:11:06.627 DEBUG:boto:Token: None 2024-04-12T04:11:06.627 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:p9UdVj/Eo3zohyqYhB4E7u8V39E= 2024-04-12T04:11:06.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p9UdVj/Eo3zohyqYhB4E7u8V39E=', 'Content-Length': '0'} 2024-04-12T04:11:06.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b49d3ebeac9c654-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.630 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:11:06.630 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:11:06.630 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.630 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.630 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.630 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.630 DEBUG:boto:Data: 2024-04-12T04:11:06.630 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.631 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.631 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.631 DEBUG:boto:Params: {} 2024-04-12T04:11:06.631 DEBUG:boto:Token: None 2024-04-12T04:11:06.631 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.631 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:p9UdVj/Eo3zohyqYhB4E7u8V39E= 2024-04-12T04:11:06.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p9UdVj/Eo3zohyqYhB4E7u8V39E=', 'Content-Length': '0'} 2024-04-12T04:11:06.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001acb9389eb58a106-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:11:06.633 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:11:06.633 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:11:06.633 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.633 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.634 DEBUG:boto:Method: GET 2024-04-12T04:11:06.634 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.634 DEBUG:boto:Data: 2024-04-12T04:11:06.634 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.634 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.634 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.634 DEBUG:boto:Params: {} 2024-04-12T04:11:06.634 DEBUG:boto:Token: None 2024-04-12T04:11:06.634 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.634 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Dk0BleAcp01cdelQNKDV2SJZzAw= 2024-04-12T04:11:06.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Dk0BleAcp01cdelQNKDV2SJZzAw=', 'Content-Length': '0'} 2024-04-12T04:11:06.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d14edbcfdb4a614f-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.637 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:11:06.637 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:11:06.637 DEBUG:boto:path=/dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.637 DEBUG:boto:auth_path=/dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.637 DEBUG:boto:Method: GET 2024-04-12T04:11:06.637 DEBUG:boto:Path: /dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.637 DEBUG:boto:Data: 2024-04-12T04:11:06.637 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.637 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.637 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.637 DEBUG:boto:Params: {} 2024-04-12T04:11:06.637 DEBUG:boto:Token: None 2024-04-12T04:11:06.638 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj33?versionId=null 2024-04-12T04:11:06.638 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Dk0BleAcp01cdelQNKDV2SJZzAw= 2024-04-12T04:11:06.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Dk0BleAcp01cdelQNKDV2SJZzAw=', 'Content-Length': '0'} 2024-04-12T04:11:06.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1b51098e5d88f3b-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.640 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.640 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.640 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:11:06.640 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:11:06.640 DEBUG:boto:Method: GET 2024-04-12T04:11:06.640 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:11:06.640 DEBUG:boto:Data: 2024-04-12T04:11:06.640 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.640 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.641 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.641 DEBUG:boto:Params: {} 2024-04-12T04:11:06.641 DEBUG:boto:Token: None 2024-04-12T04:11:06.641 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.641 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088ece0f84b9a47ad-006618b45a-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.644 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.644 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.644 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.644 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:11:06.644 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:11:06.644 DEBUG:boto:Method: GET 2024-04-12T04:11:06.644 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:11:06.644 DEBUG:boto:Data: 2024-04-12T04:11:06.644 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.645 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.645 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.645 DEBUG:boto:Params: {} 2024-04-12T04:11:06.645 DEBUG:boto:Token: None 2024-04-12T04:11:06.645 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.645 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b41176247c8399f1-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.650 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.650 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:11:06.650 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:11:06.650 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.650 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:11:06.650 DEBUG:boto:Data: 2024-04-12T04:11:06.650 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.650 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.651 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.651 DEBUG:boto:Params: {} 2024-04-12T04:11:06.651 DEBUG:boto:Token: None 2024-04-12T04:11:06.651 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj33 2024-04-12T04:11:06.651 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hAbeZwskb60XWXMwZwzct7bVNwY= 2024-04-12T04:11:06.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hAbeZwskb60XWXMwZwzct7bVNwY=', 'Content-Length': '0'} 2024-04-12T04:11:06.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a79f5b56ffd335c-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.653 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:11:06.653 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:11:06.653 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.653 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:11:06.653 DEBUG:boto:Data: 2024-04-12T04:11:06.653 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.653 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.653 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.653 DEBUG:boto:Params: {} 2024-04-12T04:11:06.654 DEBUG:boto:Token: None 2024-04-12T04:11:06.654 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj33 2024-04-12T04:11:06.654 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hAbeZwskb60XWXMwZwzct7bVNwY= 2024-04-12T04:11:06.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hAbeZwskb60XWXMwZwzct7bVNwY=', 'Content-Length': '0'} 2024-04-12T04:11:06.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001581da3caa36a458-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:11:06.656 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:11:06.656 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:11:06.656 DEBUG:boto:Method: GET 2024-04-12T04:11:06.656 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:11:06.656 DEBUG:boto:Data: 2024-04-12T04:11:06.656 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.656 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.657 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.657 DEBUG:boto:Params: {} 2024-04-12T04:11:06.657 DEBUG:boto:Token: None 2024-04-12T04:11:06.657 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj33 2024-04-12T04:11:06.657 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EIQVcbXSIkLzUflT5VjnSsTl0bQ= 2024-04-12T04:11:06.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EIQVcbXSIkLzUflT5VjnSsTl0bQ=', 'Content-Length': '0'} 2024-04-12T04:11:06.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d1013416081ddf5-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.702 DEBUG:boto:path=/dkyges-61/obj33 2024-04-12T04:11:06.702 DEBUG:boto:auth_path=/dkyges-61/obj33 2024-04-12T04:11:06.702 DEBUG:boto:Method: GET 2024-04-12T04:11:06.703 DEBUG:boto:Path: /dkyges-61/obj33 2024-04-12T04:11:06.703 DEBUG:boto:Data: 2024-04-12T04:11:06.703 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.703 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.703 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.703 DEBUG:boto:Params: {} 2024-04-12T04:11:06.703 DEBUG:boto:Token: None 2024-04-12T04:11:06.703 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj33 2024-04-12T04:11:06.703 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EIQVcbXSIkLzUflT5VjnSsTl0bQ= 2024-04-12T04:11:06.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EIQVcbXSIkLzUflT5VjnSsTl0bQ=', 'Content-Length': '0'} 2024-04-12T04:11:06.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063bda8a662216088-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.706 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.706 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.706 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:11:06.706 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:11:06.706 DEBUG:boto:Method: GET 2024-04-12T04:11:06.706 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:11:06.706 DEBUG:boto:Data: 2024-04-12T04:11:06.706 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.706 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.706 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.706 DEBUG:boto:Params: {} 2024-04-12T04:11:06.707 DEBUG:boto:Token: None 2024-04-12T04:11:06.707 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.707 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002530a84f2631bd47-006618b45a-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.710 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.710 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.710 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.710 DEBUG:boto:path=/dkyges-61/?prefix=obj33 2024-04-12T04:11:06.710 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj33 2024-04-12T04:11:06.710 DEBUG:boto:Method: GET 2024-04-12T04:11:06.710 DEBUG:boto:Path: /dkyges-61/?prefix=obj33 2024-04-12T04:11:06.711 DEBUG:boto:Data: 2024-04-12T04:11:06.711 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.711 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.711 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.711 DEBUG:boto:Params: {} 2024-04-12T04:11:06.711 DEBUG:boto:Token: None 2024-04-12T04:11:06.711 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.711 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003a58bebc47439f6-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.716 DEBUG:boto:b'dkyges-61obj331000falseobj332024-04-12T04:00:30.466Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:11:06.716 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:11:06.716 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:11:06.716 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.717 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.717 DEBUG:boto:Method: GET 2024-04-12T04:11:06.717 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.717 DEBUG:boto:Data: 2024-04-12T04:11:06.717 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.717 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.717 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.717 DEBUG:boto:Params: {} 2024-04-12T04:11:06.717 DEBUG:boto:Token: None 2024-04-12T04:11:06.717 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.717 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BciBL8f+d8VLOG9YrgGyYmUHjDk= 2024-04-12T04:11:06.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BciBL8f+d8VLOG9YrgGyYmUHjDk=', 'Content-Length': '0'} 2024-04-12T04:11:06.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000685258504b02fc8-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.720 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:11:06.720 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:11:06.720 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.720 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.720 DEBUG:boto:Method: GET 2024-04-12T04:11:06.720 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.720 DEBUG:boto:Data: 2024-04-12T04:11:06.720 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.720 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.720 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.720 DEBUG:boto:Params: {} 2024-04-12T04:11:06.720 DEBUG:boto:Token: None 2024-04-12T04:11:06.721 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.721 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BciBL8f+d8VLOG9YrgGyYmUHjDk= 2024-04-12T04:11:06.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BciBL8f+d8VLOG9YrgGyYmUHjDk=', 'Content-Length': '0'} 2024-04-12T04:11:06.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f8e07edcd1e7d68-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.723 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:11:06.723 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:11:06.723 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.723 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.723 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.723 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.723 DEBUG:boto:Data: 2024-04-12T04:11:06.724 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.724 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.724 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.724 DEBUG:boto:Params: {} 2024-04-12T04:11:06.724 DEBUG:boto:Token: None 2024-04-12T04:11:06.724 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.724 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZjW5oLjma+N4xpHw0ix1zVT9KiQ= 2024-04-12T04:11:06.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZjW5oLjma+N4xpHw0ix1zVT9KiQ=', 'Content-Length': '0'} 2024-04-12T04:11:06.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079f0c3a873b98d87-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.726 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:11:06.726 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:11:06.726 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.726 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.726 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.727 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.727 DEBUG:boto:Data: 2024-04-12T04:11:06.727 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.727 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.727 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.727 DEBUG:boto:Params: {} 2024-04-12T04:11:06.727 DEBUG:boto:Token: None 2024-04-12T04:11:06.727 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.727 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZjW5oLjma+N4xpHw0ix1zVT9KiQ= 2024-04-12T04:11:06.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZjW5oLjma+N4xpHw0ix1zVT9KiQ=', 'Content-Length': '0'} 2024-04-12T04:11:06.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017f0c46d201ce8b6-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:11:06.729 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:11:06.730 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:11:06.730 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.730 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.730 DEBUG:boto:Method: GET 2024-04-12T04:11:06.730 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.730 DEBUG:boto:Data: 2024-04-12T04:11:06.730 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.730 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.730 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.730 DEBUG:boto:Params: {} 2024-04-12T04:11:06.730 DEBUG:boto:Token: None 2024-04-12T04:11:06.731 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.731 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BciBL8f+d8VLOG9YrgGyYmUHjDk= 2024-04-12T04:11:06.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BciBL8f+d8VLOG9YrgGyYmUHjDk=', 'Content-Length': '0'} 2024-04-12T04:11:06.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000000e089f32781caa-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.733 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:11:06.733 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:11:06.733 DEBUG:boto:path=/dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.733 DEBUG:boto:auth_path=/dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.733 DEBUG:boto:Method: GET 2024-04-12T04:11:06.733 DEBUG:boto:Path: /dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.733 DEBUG:boto:Data: 2024-04-12T04:11:06.733 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.733 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.733 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.733 DEBUG:boto:Params: {} 2024-04-12T04:11:06.733 DEBUG:boto:Token: None 2024-04-12T04:11:06.734 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj34?versionId=null 2024-04-12T04:11:06.734 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BciBL8f+d8VLOG9YrgGyYmUHjDk= 2024-04-12T04:11:06.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BciBL8f+d8VLOG9YrgGyYmUHjDk=', 'Content-Length': '0'} 2024-04-12T04:11:06.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b0ee7d1d2f1913c-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.736 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.736 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.736 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:11:06.736 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:11:06.736 DEBUG:boto:Method: GET 2024-04-12T04:11:06.736 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:11:06.737 DEBUG:boto:Data: 2024-04-12T04:11:06.737 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.737 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.737 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.737 DEBUG:boto:Params: {} 2024-04-12T04:11:06.737 DEBUG:boto:Token: None 2024-04-12T04:11:06.737 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.737 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caa6270f9a079955-006618b45a-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.740 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.740 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.740 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.740 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:11:06.740 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:11:06.740 DEBUG:boto:Method: GET 2024-04-12T04:11:06.741 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:11:06.741 DEBUG:boto:Data: 2024-04-12T04:11:06.741 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.741 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.741 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.741 DEBUG:boto:Params: {} 2024-04-12T04:11:06.741 DEBUG:boto:Token: None 2024-04-12T04:11:06.741 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.741 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a5716440d9187e9-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.745 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.746 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:11:06.746 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:11:06.746 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.746 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:11:06.746 DEBUG:boto:Data: 2024-04-12T04:11:06.746 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.746 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.746 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.747 DEBUG:boto:Params: {} 2024-04-12T04:11:06.747 DEBUG:boto:Token: None 2024-04-12T04:11:06.747 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj34 2024-04-12T04:11:06.747 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:03lNo5QjL/i+Tz7JrequazeyA8c= 2024-04-12T04:11:06.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:03lNo5QjL/i+Tz7JrequazeyA8c=', 'Content-Length': '0'} 2024-04-12T04:11:06.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006782bf9853d7dc5a-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.749 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:11:06.749 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:11:06.749 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.749 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:11:06.749 DEBUG:boto:Data: 2024-04-12T04:11:06.749 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.750 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.750 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.750 DEBUG:boto:Params: {} 2024-04-12T04:11:06.750 DEBUG:boto:Token: None 2024-04-12T04:11:06.750 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj34 2024-04-12T04:11:06.750 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:03lNo5QjL/i+Tz7JrequazeyA8c= 2024-04-12T04:11:06.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:03lNo5QjL/i+Tz7JrequazeyA8c=', 'Content-Length': '0'} 2024-04-12T04:11:06.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d673a7732c21acb-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:11:06.752 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:11:06.752 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:11:06.753 DEBUG:boto:Method: GET 2024-04-12T04:11:06.753 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:11:06.753 DEBUG:boto:Data: 2024-04-12T04:11:06.753 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.753 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.753 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.753 DEBUG:boto:Params: {} 2024-04-12T04:11:06.753 DEBUG:boto:Token: None 2024-04-12T04:11:06.753 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj34 2024-04-12T04:11:06.753 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4aGBecVOfZ2Tdld1AnSewW4uVqk= 2024-04-12T04:11:06.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4aGBecVOfZ2Tdld1AnSewW4uVqk=', 'Content-Length': '0'} 2024-04-12T04:11:06.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ecdacc026dad264-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.798 DEBUG:boto:path=/dkyges-61/obj34 2024-04-12T04:11:06.798 DEBUG:boto:auth_path=/dkyges-61/obj34 2024-04-12T04:11:06.799 DEBUG:boto:Method: GET 2024-04-12T04:11:06.799 DEBUG:boto:Path: /dkyges-61/obj34 2024-04-12T04:11:06.799 DEBUG:boto:Data: 2024-04-12T04:11:06.799 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.799 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.800 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.800 DEBUG:boto:Params: {} 2024-04-12T04:11:06.800 DEBUG:boto:Token: None 2024-04-12T04:11:06.800 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj34 2024-04-12T04:11:06.800 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4aGBecVOfZ2Tdld1AnSewW4uVqk= 2024-04-12T04:11:06.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4aGBecVOfZ2Tdld1AnSewW4uVqk=', 'Content-Length': '0'} 2024-04-12T04:11:06.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001689d3378aeccd4f-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.803 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.803 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.804 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:11:06.804 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:11:06.804 DEBUG:boto:Method: GET 2024-04-12T04:11:06.804 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:11:06.804 DEBUG:boto:Data: 2024-04-12T04:11:06.804 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.805 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.805 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.805 DEBUG:boto:Params: {} 2024-04-12T04:11:06.805 DEBUG:boto:Token: None 2024-04-12T04:11:06.805 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.806 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f7a6aefbe1d21e8-006618b45a-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.809 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.809 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.809 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.809 DEBUG:boto:path=/dkyges-61/?prefix=obj34 2024-04-12T04:11:06.809 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj34 2024-04-12T04:11:06.809 DEBUG:boto:Method: GET 2024-04-12T04:11:06.809 DEBUG:boto:Path: /dkyges-61/?prefix=obj34 2024-04-12T04:11:06.809 DEBUG:boto:Data: 2024-04-12T04:11:06.810 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.810 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.810 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.810 DEBUG:boto:Params: {} 2024-04-12T04:11:06.810 DEBUG:boto:Token: None 2024-04-12T04:11:06.810 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.810 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d592458f95e3f589-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.815 DEBUG:boto:b'dkyges-61obj341000falseobj342024-04-12T04:00:41.687Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T04:11:06.816 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:11:06.816 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:11:06.816 DEBUG:boto:path=/dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.816 DEBUG:boto:auth_path=/dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.816 DEBUG:boto:Method: GET 2024-04-12T04:11:06.816 DEBUG:boto:Path: /dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.816 DEBUG:boto:Data: 2024-04-12T04:11:06.816 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.816 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.816 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.816 DEBUG:boto:Params: {} 2024-04-12T04:11:06.816 DEBUG:boto:Token: None 2024-04-12T04:11:06.817 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.817 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UawNcA1xH3Q/0rn3JpeG4S0MxRw= 2024-04-12T04:11:06.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UawNcA1xH3Q/0rn3JpeG4S0MxRw=', 'Content-Length': '0'} 2024-04-12T04:11:06.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000412026d5472b81f7-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.819 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:11:06.819 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:11:06.819 DEBUG:boto:path=/dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.820 DEBUG:boto:auth_path=/dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.820 DEBUG:boto:Method: GET 2024-04-12T04:11:06.820 DEBUG:boto:Path: /dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.820 DEBUG:boto:Data: 2024-04-12T04:11:06.820 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.820 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.820 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.820 DEBUG:boto:Params: {} 2024-04-12T04:11:06.820 DEBUG:boto:Token: None 2024-04-12T04:11:06.820 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.820 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UawNcA1xH3Q/0rn3JpeG4S0MxRw= 2024-04-12T04:11:06.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UawNcA1xH3Q/0rn3JpeG4S0MxRw=', 'Content-Length': '0'} 2024-04-12T04:11:06.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2f1bc144d214e1b-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.823 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:11:06.823 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:11:06.823 DEBUG:boto:path=/dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.823 DEBUG:boto:auth_path=/dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.823 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.823 DEBUG:boto:Path: /dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.823 DEBUG:boto:Data: 2024-04-12T04:11:06.824 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.824 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.824 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.824 DEBUG:boto:Params: {} 2024-04-12T04:11:06.824 DEBUG:boto:Token: None 2024-04-12T04:11:06.824 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.824 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uZ1qCyUj0CgicJPWwYRPmkr4a7s= 2024-04-12T04:11:06.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uZ1qCyUj0CgicJPWwYRPmkr4a7s=', 'Content-Length': '0'} 2024-04-12T04:11:06.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f54c72a086c511a4-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.826 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:11:06.826 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:11:06.826 DEBUG:boto:path=/dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.827 DEBUG:boto:auth_path=/dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.827 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.827 DEBUG:boto:Path: /dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.827 DEBUG:boto:Data: 2024-04-12T04:11:06.827 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.827 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.827 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.827 DEBUG:boto:Params: {} 2024-04-12T04:11:06.827 DEBUG:boto:Token: None 2024-04-12T04:11:06.827 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.827 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uZ1qCyUj0CgicJPWwYRPmkr4a7s= 2024-04-12T04:11:06.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uZ1qCyUj0CgicJPWwYRPmkr4a7s=', 'Content-Length': '0'} 2024-04-12T04:11:06.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aea6f4316b3aab25-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T04:11:06.830 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:11:06.830 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:11:06.830 DEBUG:boto:path=/dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.830 DEBUG:boto:auth_path=/dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.830 DEBUG:boto:Method: GET 2024-04-12T04:11:06.830 DEBUG:boto:Path: /dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.830 DEBUG:boto:Data: 2024-04-12T04:11:06.830 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.830 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.831 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.831 DEBUG:boto:Params: {} 2024-04-12T04:11:06.831 DEBUG:boto:Token: None 2024-04-12T04:11:06.831 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.831 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UawNcA1xH3Q/0rn3JpeG4S0MxRw= 2024-04-12T04:11:06.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UawNcA1xH3Q/0rn3JpeG4S0MxRw=', 'Content-Length': '0'} 2024-04-12T04:11:06.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004691424dbaec21e1-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.833 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:11:06.833 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:11:06.833 DEBUG:boto:path=/dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.833 DEBUG:boto:auth_path=/dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.834 DEBUG:boto:Method: GET 2024-04-12T04:11:06.834 DEBUG:boto:Path: /dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.834 DEBUG:boto:Data: 2024-04-12T04:11:06.834 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.834 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.834 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.834 DEBUG:boto:Params: {} 2024-04-12T04:11:06.834 DEBUG:boto:Token: None 2024-04-12T04:11:06.834 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj37?versionId=null 2024-04-12T04:11:06.834 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UawNcA1xH3Q/0rn3JpeG4S0MxRw= 2024-04-12T04:11:06.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UawNcA1xH3Q/0rn3JpeG4S0MxRw=', 'Content-Length': '0'} 2024-04-12T04:11:06.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a296e5e23765a6e-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.878 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.878 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.879 DEBUG:boto:path=/dkyges-61/?prefix=obj37 2024-04-12T04:11:06.879 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj37 2024-04-12T04:11:06.879 DEBUG:boto:Method: GET 2024-04-12T04:11:06.879 DEBUG:boto:Path: /dkyges-61/?prefix=obj37 2024-04-12T04:11:06.879 DEBUG:boto:Data: 2024-04-12T04:11:06.879 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.879 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.879 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.879 DEBUG:boto:Params: {} 2024-04-12T04:11:06.879 DEBUG:boto:Token: None 2024-04-12T04:11:06.879 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.880 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069ae8cf85e999dfa-006618b45a-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.882 DEBUG:boto:b'dkyges-61obj371000falseobj372024-04-12T04:09:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.882 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.883 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.883 DEBUG:boto:path=/dkyges-61/?prefix=obj37 2024-04-12T04:11:06.883 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj37 2024-04-12T04:11:06.883 DEBUG:boto:Method: GET 2024-04-12T04:11:06.883 DEBUG:boto:Path: /dkyges-61/?prefix=obj37 2024-04-12T04:11:06.883 DEBUG:boto:Data: 2024-04-12T04:11:06.883 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.883 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.883 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.883 DEBUG:boto:Params: {} 2024-04-12T04:11:06.883 DEBUG:boto:Token: None 2024-04-12T04:11:06.884 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.884 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.888 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080b25f7cfa6e9f09-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.888 DEBUG:boto:b'dkyges-61obj371000falseobj372024-04-12T04:09:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.888 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:11:06.888 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:11:06.888 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.889 DEBUG:boto:Path: /dkyges-61/obj37 2024-04-12T04:11:06.889 DEBUG:boto:Data: 2024-04-12T04:11:06.889 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.889 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.889 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.889 DEBUG:boto:Params: {} 2024-04-12T04:11:06.889 DEBUG:boto:Token: None 2024-04-12T04:11:06.889 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj37 2024-04-12T04:11:06.889 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hRX30Ks+XfBiZS8c5ckKCbtjV08= 2024-04-12T04:11:06.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hRX30Ks+XfBiZS8c5ckKCbtjV08=', 'Content-Length': '0'} 2024-04-12T04:11:06.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a8c2e822853ff2b-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.892 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:11:06.892 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:11:06.892 DEBUG:boto:Method: HEAD 2024-04-12T04:11:06.892 DEBUG:boto:Path: /dkyges-61/obj37 2024-04-12T04:11:06.892 DEBUG:boto:Data: 2024-04-12T04:11:06.892 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.892 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.892 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.892 DEBUG:boto:Params: {} 2024-04-12T04:11:06.892 DEBUG:boto:Token: None 2024-04-12T04:11:06.893 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj37 2024-04-12T04:11:06.893 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hRX30Ks+XfBiZS8c5ckKCbtjV08= 2024-04-12T04:11:06.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hRX30Ks+XfBiZS8c5ckKCbtjV08=', 'Content-Length': '0'} 2024-04-12T04:11:06.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acc14b55596f459b-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T04:11:06.895 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:11:06.895 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:11:06.895 DEBUG:boto:Method: GET 2024-04-12T04:11:06.895 DEBUG:boto:Path: /dkyges-61/obj37 2024-04-12T04:11:06.896 DEBUG:boto:Data: 2024-04-12T04:11:06.896 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.896 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.896 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.896 DEBUG:boto:Params: {} 2024-04-12T04:11:06.896 DEBUG:boto:Token: None 2024-04-12T04:11:06.896 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj37 2024-04-12T04:11:06.896 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pfrME2GMFoCyp9njx+4CxbP1b70= 2024-04-12T04:11:06.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pfrME2GMFoCyp9njx+4CxbP1b70=', 'Content-Length': '0'} 2024-04-12T04:11:06.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b0c2d4b2cb42eb9-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.938 DEBUG:boto:path=/dkyges-61/obj37 2024-04-12T04:11:06.938 DEBUG:boto:auth_path=/dkyges-61/obj37 2024-04-12T04:11:06.938 DEBUG:boto:Method: GET 2024-04-12T04:11:06.939 DEBUG:boto:Path: /dkyges-61/obj37 2024-04-12T04:11:06.939 DEBUG:boto:Data: 2024-04-12T04:11:06.939 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.939 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.939 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.939 DEBUG:boto:Params: {} 2024-04-12T04:11:06.939 DEBUG:boto:Token: None 2024-04-12T04:11:06.939 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj37 2024-04-12T04:11:06.939 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pfrME2GMFoCyp9njx+4CxbP1b70= 2024-04-12T04:11:06.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pfrME2GMFoCyp9njx+4CxbP1b70=', 'Content-Length': '0'} 2024-04-12T04:11:06.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c83607cf05a08cb0-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.941 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.942 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.942 DEBUG:boto:path=/dkyges-61/?prefix=obj37 2024-04-12T04:11:06.942 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj37 2024-04-12T04:11:06.942 DEBUG:boto:Method: GET 2024-04-12T04:11:06.942 DEBUG:boto:Path: /dkyges-61/?prefix=obj37 2024-04-12T04:11:06.942 DEBUG:boto:Data: 2024-04-12T04:11:06.942 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.942 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.942 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.942 DEBUG:boto:Params: {} 2024-04-12T04:11:06.942 DEBUG:boto:Token: None 2024-04-12T04:11:06.943 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.943 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0d39911a4819a3f-006618b45a-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.945 DEBUG:boto:b'dkyges-61obj371000falseobj372024-04-12T04:09:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.946 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:06.946 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:06.946 DEBUG:boto:path=/dkyges-61/?prefix=obj37 2024-04-12T04:11:06.946 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj37 2024-04-12T04:11:06.946 DEBUG:boto:Method: GET 2024-04-12T04:11:06.946 DEBUG:boto:Path: /dkyges-61/?prefix=obj37 2024-04-12T04:11:06.946 DEBUG:boto:Data: 2024-04-12T04:11:06.946 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.946 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.946 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.946 DEBUG:boto:Params: {} 2024-04-12T04:11:06.947 DEBUG:boto:Token: None 2024-04-12T04:11:06.947 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/ 2024-04-12T04:11:06.947 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4= 2024-04-12T04:11:06.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m2HAr9C6YvL3pmPWV69eot1Suh4=', 'Content-Length': '0'} 2024-04-12T04:11:06.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e535a10db2ee33a5-006618b45a-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.951 DEBUG:boto:b'dkyges-61obj371000falseobj372024-04-12T04:09:24.034Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:06.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:11:06.952 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:11:06.952 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:11:06.952 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:11:06.952 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:11:06.952 DEBUG:boto:Method: GET 2024-04-12T04:11:06.952 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:11:06.952 DEBUG:boto:Data: 2024-04-12T04:11:06.952 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.952 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:06.952 DEBUG:boto:Port: 8001 2024-04-12T04:11:06.952 DEBUG:boto:Params: {} 2024-04-12T04:11:06.952 DEBUG:boto:Token: None 2024-04-12T04:11:06.953 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:11:06.953 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D7KDwFPNZi3ZtGEu1reKxaOEExY= 2024-04-12T04:11:06.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D7KDwFPNZi3ZtGEu1reKxaOEExY=', 'Content-Length': '0'} 2024-04-12T04:11:06.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000198f58403d98e003-006618b45a-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:06.955 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:11:06.955 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:11:06.955 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:11:06.955 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:11:06.955 DEBUG:boto:Method: GET 2024-04-12T04:11:06.955 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:11:06.956 DEBUG:boto:Data: 2024-04-12T04:11:06.956 DEBUG:boto:Headers: {} 2024-04-12T04:11:06.956 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:06.956 DEBUG:boto:Port: 8000 2024-04-12T04:11:06.956 DEBUG:boto:Params: {} 2024-04-12T04:11:06.956 DEBUG:boto:Token: None 2024-04-12T04:11:06.956 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:06 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:11:06.956 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D7KDwFPNZi3ZtGEu1reKxaOEExY= 2024-04-12T04:11:06.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:06 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D7KDwFPNZi3ZtGEu1reKxaOEExY=', 'Content-Length': '0'} 2024-04-12T04:11:06.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9f2f7df7515c09e-006618b45a-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:06 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.002 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:11:07.002 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:11:07.003 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:11:07.003 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:11:07.003 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.003 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:11:07.003 DEBUG:boto:Data: 2024-04-12T04:11:07.003 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.003 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.003 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.003 DEBUG:boto:Params: {} 2024-04-12T04:11:07.003 DEBUG:boto:Token: None 2024-04-12T04:11:07.004 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:11:07.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UQyfqK+Fic5OyumNmWbYP5m0qw8= 2024-04-12T04:11:07.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UQyfqK+Fic5OyumNmWbYP5m0qw8=', 'Content-Length': '0'} 2024-04-12T04:11:07.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000283a293963ebd5d-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.006 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:11:07.006 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:11:07.006 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:11:07.006 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:11:07.006 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.006 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:11:07.006 DEBUG:boto:Data: 2024-04-12T04:11:07.006 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.006 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.007 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.007 DEBUG:boto:Params: {} 2024-04-12T04:11:07.007 DEBUG:boto:Token: None 2024-04-12T04:11:07.007 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:11:07.007 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UQyfqK+Fic5OyumNmWbYP5m0qw8= 2024-04-12T04:11:07.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UQyfqK+Fic5OyumNmWbYP5m0qw8=', 'Content-Length': '0'} 2024-04-12T04:11:07.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004292e62e6a415e2d-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:11:07.009 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:11:07.009 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:11:07.009 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:11:07.010 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:11:07.010 DEBUG:boto:Method: GET 2024-04-12T04:11:07.010 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:11:07.010 DEBUG:boto:Data: 2024-04-12T04:11:07.010 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.010 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.010 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.010 DEBUG:boto:Params: {} 2024-04-12T04:11:07.010 DEBUG:boto:Token: None 2024-04-12T04:11:07.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:11:07.011 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XxmchXLYj9VgadqHfszM1Pu9DZ4= 2024-04-12T04:11:07.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XxmchXLYj9VgadqHfszM1Pu9DZ4=', 'Content-Length': '0'} 2024-04-12T04:11:07.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7d6069929424539-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.013 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:11:07.013 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:11:07.013 DEBUG:boto:path=/dkyges-61/obj38?versionId=null 2024-04-12T04:11:07.013 DEBUG:boto:auth_path=/dkyges-61/obj38?versionId=null 2024-04-12T04:11:07.013 DEBUG:boto:Method: GET 2024-04-12T04:11:07.013 DEBUG:boto:Path: /dkyges-61/obj38?versionId=null 2024-04-12T04:11:07.014 DEBUG:boto:Data: 2024-04-12T04:11:07.014 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.014 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.014 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.014 DEBUG:boto:Params: {} 2024-04-12T04:11:07.014 DEBUG:boto:Token: None 2024-04-12T04:11:07.014 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj38?versionId=null 2024-04-12T04:11:07.014 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XxmchXLYj9VgadqHfszM1Pu9DZ4= 2024-04-12T04:11:07.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XxmchXLYj9VgadqHfszM1Pu9DZ4=', 'Content-Length': '0'} 2024-04-12T04:11:07.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9f1c044f547aedd-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.017 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.017 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.017 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:11:07.017 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:11:07.017 DEBUG:boto:Method: GET 2024-04-12T04:11:07.017 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:11:07.017 DEBUG:boto:Data: 2024-04-12T04:11:07.017 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.017 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.017 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.018 DEBUG:boto:Params: {} 2024-04-12T04:11:07.018 DEBUG:boto:Token: None 2024-04-12T04:11:07.018 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.018 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084a05945640a7086-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.020 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.021 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.021 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.021 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:11:07.021 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:11:07.021 DEBUG:boto:Method: GET 2024-04-12T04:11:07.021 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:11:07.021 DEBUG:boto:Data: 2024-04-12T04:11:07.021 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.022 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.022 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.022 DEBUG:boto:Params: {} 2024-04-12T04:11:07.022 DEBUG:boto:Token: None 2024-04-12T04:11:07.022 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.022 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c5677c3f04a568f-006618b45b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.027 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.027 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:11:07.027 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:11:07.027 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.027 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:11:07.028 DEBUG:boto:Data: 2024-04-12T04:11:07.028 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.028 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.028 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.028 DEBUG:boto:Params: {} 2024-04-12T04:11:07.028 DEBUG:boto:Token: None 2024-04-12T04:11:07.028 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj38 2024-04-12T04:11:07.028 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7BI6QvwX10x/bUdawtZKwvOI57E= 2024-04-12T04:11:07.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7BI6QvwX10x/bUdawtZKwvOI57E=', 'Content-Length': '0'} 2024-04-12T04:11:07.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1ac6253f228e040-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.030 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:11:07.030 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:11:07.030 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.031 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:11:07.031 DEBUG:boto:Data: 2024-04-12T04:11:07.031 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.031 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.031 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.031 DEBUG:boto:Params: {} 2024-04-12T04:11:07.031 DEBUG:boto:Token: None 2024-04-12T04:11:07.031 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj38 2024-04-12T04:11:07.031 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7BI6QvwX10x/bUdawtZKwvOI57E= 2024-04-12T04:11:07.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7BI6QvwX10x/bUdawtZKwvOI57E=', 'Content-Length': '0'} 2024-04-12T04:11:07.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052683534255a5827-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:11:07.033 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:11:07.033 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:11:07.034 DEBUG:boto:Method: GET 2024-04-12T04:11:07.034 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:11:07.034 DEBUG:boto:Data: 2024-04-12T04:11:07.034 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.034 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.034 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.034 DEBUG:boto:Params: {} 2024-04-12T04:11:07.034 DEBUG:boto:Token: None 2024-04-12T04:11:07.034 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj38 2024-04-12T04:11:07.034 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:raaobp1IY3eSWfiyEGjVBVkH4L8= 2024-04-12T04:11:07.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:raaobp1IY3eSWfiyEGjVBVkH4L8=', 'Content-Length': '0'} 2024-04-12T04:11:07.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc7a5000924f7867-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.037 DEBUG:boto:path=/dkyges-61/obj38 2024-04-12T04:11:07.037 DEBUG:boto:auth_path=/dkyges-61/obj38 2024-04-12T04:11:07.037 DEBUG:boto:Method: GET 2024-04-12T04:11:07.037 DEBUG:boto:Path: /dkyges-61/obj38 2024-04-12T04:11:07.037 DEBUG:boto:Data: 2024-04-12T04:11:07.037 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.037 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.037 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.037 DEBUG:boto:Params: {} 2024-04-12T04:11:07.037 DEBUG:boto:Token: None 2024-04-12T04:11:07.038 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj38 2024-04-12T04:11:07.038 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:raaobp1IY3eSWfiyEGjVBVkH4L8= 2024-04-12T04:11:07.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:raaobp1IY3eSWfiyEGjVBVkH4L8=', 'Content-Length': '0'} 2024-04-12T04:11:07.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028723900a430686d-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.040 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.040 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.040 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:11:07.040 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:11:07.040 DEBUG:boto:Method: GET 2024-04-12T04:11:07.040 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:11:07.040 DEBUG:boto:Data: 2024-04-12T04:11:07.040 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.041 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.041 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.041 DEBUG:boto:Params: {} 2024-04-12T04:11:07.041 DEBUG:boto:Token: None 2024-04-12T04:11:07.041 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.041 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba33a51c406e1bec-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.044 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.044 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.044 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.044 DEBUG:boto:path=/dkyges-61/?prefix=obj38 2024-04-12T04:11:07.044 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj38 2024-04-12T04:11:07.044 DEBUG:boto:Method: GET 2024-04-12T04:11:07.044 DEBUG:boto:Path: /dkyges-61/?prefix=obj38 2024-04-12T04:11:07.044 DEBUG:boto:Data: 2024-04-12T04:11:07.044 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.045 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.045 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.045 DEBUG:boto:Params: {} 2024-04-12T04:11:07.045 DEBUG:boto:Token: None 2024-04-12T04:11:07.045 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.045 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f361d9be6cb93859-006618b45b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.050 DEBUG:boto:b'dkyges-61obj381000falseobj382024-04-12T04:03:03.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:11:07.051 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:11:07.051 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:11:07.052 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.052 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.052 DEBUG:boto:Method: GET 2024-04-12T04:11:07.052 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.052 DEBUG:boto:Data: 2024-04-12T04:11:07.053 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.053 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.053 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.053 DEBUG:boto:Params: {} 2024-04-12T04:11:07.053 DEBUG:boto:Token: None 2024-04-12T04:11:07.053 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.053 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:M0Wua5ChHA759+NyAMx62BTml7s= 2024-04-12T04:11:07.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:M0Wua5ChHA759+NyAMx62BTml7s=', 'Content-Length': '0'} 2024-04-12T04:11:07.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc0419d29269901a-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.056 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:11:07.056 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:11:07.056 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.056 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.056 DEBUG:boto:Method: GET 2024-04-12T04:11:07.057 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.057 DEBUG:boto:Data: 2024-04-12T04:11:07.057 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.057 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.057 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.057 DEBUG:boto:Params: {} 2024-04-12T04:11:07.057 DEBUG:boto:Token: None 2024-04-12T04:11:07.057 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.057 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:M0Wua5ChHA759+NyAMx62BTml7s= 2024-04-12T04:11:07.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:M0Wua5ChHA759+NyAMx62BTml7s=', 'Content-Length': '0'} 2024-04-12T04:11:07.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b1e3ccb4fdb1962-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.102 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:11:07.102 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:11:07.103 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.103 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.103 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.103 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.103 DEBUG:boto:Data: 2024-04-12T04:11:07.103 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.103 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.103 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.103 DEBUG:boto:Params: {} 2024-04-12T04:11:07.103 DEBUG:boto:Token: None 2024-04-12T04:11:07.103 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.104 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZaeCK0xLqvS6OJI1xMqgeOVR+iA= 2024-04-12T04:11:07.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZaeCK0xLqvS6OJI1xMqgeOVR+iA=', 'Content-Length': '0'} 2024-04-12T04:11:07.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ecb028948fc8013-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.106 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:11:07.106 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:11:07.106 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.106 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.106 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.106 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.106 DEBUG:boto:Data: 2024-04-12T04:11:07.106 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.106 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.106 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.107 DEBUG:boto:Params: {} 2024-04-12T04:11:07.107 DEBUG:boto:Token: None 2024-04-12T04:11:07.107 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.107 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZaeCK0xLqvS6OJI1xMqgeOVR+iA= 2024-04-12T04:11:07.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZaeCK0xLqvS6OJI1xMqgeOVR+iA=', 'Content-Length': '0'} 2024-04-12T04:11:07.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000228ed03b11d708cc-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:11:07.110 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:11:07.110 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:11:07.110 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.110 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.110 DEBUG:boto:Method: GET 2024-04-12T04:11:07.110 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.110 DEBUG:boto:Data: 2024-04-12T04:11:07.110 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.111 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.111 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.111 DEBUG:boto:Params: {} 2024-04-12T04:11:07.111 DEBUG:boto:Token: None 2024-04-12T04:11:07.111 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.111 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:M0Wua5ChHA759+NyAMx62BTml7s= 2024-04-12T04:11:07.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:M0Wua5ChHA759+NyAMx62BTml7s=', 'Content-Length': '0'} 2024-04-12T04:11:07.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041eb382aebbd45fe-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.113 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:11:07.113 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:11:07.113 DEBUG:boto:path=/dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.113 DEBUG:boto:auth_path=/dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.113 DEBUG:boto:Method: GET 2024-04-12T04:11:07.113 DEBUG:boto:Path: /dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.113 DEBUG:boto:Data: 2024-04-12T04:11:07.114 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.114 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.114 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.114 DEBUG:boto:Params: {} 2024-04-12T04:11:07.114 DEBUG:boto:Token: None 2024-04-12T04:11:07.114 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj44?versionId=null 2024-04-12T04:11:07.114 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:M0Wua5ChHA759+NyAMx62BTml7s= 2024-04-12T04:11:07.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:M0Wua5ChHA759+NyAMx62BTml7s=', 'Content-Length': '0'} 2024-04-12T04:11:07.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac0645eb6f9a7637-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.116 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.116 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.116 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:11:07.117 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:11:07.117 DEBUG:boto:Method: GET 2024-04-12T04:11:07.117 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:11:07.117 DEBUG:boto:Data: 2024-04-12T04:11:07.117 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.117 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.117 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.117 DEBUG:boto:Params: {} 2024-04-12T04:11:07.117 DEBUG:boto:Token: None 2024-04-12T04:11:07.117 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.117 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.120 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000423a15ab5de20ef4-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.120 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.120 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.120 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.120 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:11:07.121 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:11:07.121 DEBUG:boto:Method: GET 2024-04-12T04:11:07.121 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:11:07.121 DEBUG:boto:Data: 2024-04-12T04:11:07.121 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.121 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.121 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.121 DEBUG:boto:Params: {} 2024-04-12T04:11:07.121 DEBUG:boto:Token: None 2024-04-12T04:11:07.121 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.121 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000246c54e1f6c0d7fc-006618b45b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.126 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.126 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:11:07.126 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:11:07.126 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.126 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:11:07.127 DEBUG:boto:Data: 2024-04-12T04:11:07.127 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.127 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.127 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.127 DEBUG:boto:Params: {} 2024-04-12T04:11:07.127 DEBUG:boto:Token: None 2024-04-12T04:11:07.127 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj44 2024-04-12T04:11:07.127 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Umf8tQsyZ4qJ6oYeD4MAAumIQBk= 2024-04-12T04:11:07.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Umf8tQsyZ4qJ6oYeD4MAAumIQBk=', 'Content-Length': '0'} 2024-04-12T04:11:07.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044447d0a47c98f82-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.129 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:11:07.130 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:11:07.130 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.130 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:11:07.130 DEBUG:boto:Data: 2024-04-12T04:11:07.130 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.130 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.130 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.130 DEBUG:boto:Params: {} 2024-04-12T04:11:07.130 DEBUG:boto:Token: None 2024-04-12T04:11:07.130 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj44 2024-04-12T04:11:07.130 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Umf8tQsyZ4qJ6oYeD4MAAumIQBk= 2024-04-12T04:11:07.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Umf8tQsyZ4qJ6oYeD4MAAumIQBk=', 'Content-Length': '0'} 2024-04-12T04:11:07.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e003f936c9b0a52b-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:11:07.133 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:11:07.133 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:11:07.133 DEBUG:boto:Method: GET 2024-04-12T04:11:07.133 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:11:07.133 DEBUG:boto:Data: 2024-04-12T04:11:07.133 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.133 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.133 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.133 DEBUG:boto:Params: {} 2024-04-12T04:11:07.133 DEBUG:boto:Token: None 2024-04-12T04:11:07.133 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj44 2024-04-12T04:11:07.134 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G6Wpr8DfFOt4fz1sDe5h7mg6Juo= 2024-04-12T04:11:07.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G6Wpr8DfFOt4fz1sDe5h7mg6Juo=', 'Content-Length': '0'} 2024-04-12T04:11:07.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000285ca75f946eeccd-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.178 DEBUG:boto:path=/dkyges-61/obj44 2024-04-12T04:11:07.178 DEBUG:boto:auth_path=/dkyges-61/obj44 2024-04-12T04:11:07.178 DEBUG:boto:Method: GET 2024-04-12T04:11:07.179 DEBUG:boto:Path: /dkyges-61/obj44 2024-04-12T04:11:07.179 DEBUG:boto:Data: 2024-04-12T04:11:07.179 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.179 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.179 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.179 DEBUG:boto:Params: {} 2024-04-12T04:11:07.179 DEBUG:boto:Token: None 2024-04-12T04:11:07.179 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj44 2024-04-12T04:11:07.179 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G6Wpr8DfFOt4fz1sDe5h7mg6Juo= 2024-04-12T04:11:07.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G6Wpr8DfFOt4fz1sDe5h7mg6Juo=', 'Content-Length': '0'} 2024-04-12T04:11:07.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000978814ebf00d3731-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.182 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.182 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.182 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:11:07.182 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:11:07.182 DEBUG:boto:Method: GET 2024-04-12T04:11:07.182 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:11:07.182 DEBUG:boto:Data: 2024-04-12T04:11:07.182 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.182 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.182 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.182 DEBUG:boto:Params: {} 2024-04-12T04:11:07.183 DEBUG:boto:Token: None 2024-04-12T04:11:07.183 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.183 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000151ee42669adfdf1-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.185 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.186 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.186 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.186 DEBUG:boto:path=/dkyges-61/?prefix=obj44 2024-04-12T04:11:07.186 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj44 2024-04-12T04:11:07.186 DEBUG:boto:Method: GET 2024-04-12T04:11:07.186 DEBUG:boto:Path: /dkyges-61/?prefix=obj44 2024-04-12T04:11:07.186 DEBUG:boto:Data: 2024-04-12T04:11:07.187 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.187 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.187 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.187 DEBUG:boto:Params: {} 2024-04-12T04:11:07.187 DEBUG:boto:Token: None 2024-04-12T04:11:07.187 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.187 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fadb0548971e04f-006618b45b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.192 DEBUG:boto:b'dkyges-61obj441000falseobj442024-04-12T04:00:30.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:11:07.192 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:11:07.192 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:11:07.192 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.192 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.193 DEBUG:boto:Method: GET 2024-04-12T04:11:07.193 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.193 DEBUG:boto:Data: 2024-04-12T04:11:07.193 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.193 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.193 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.193 DEBUG:boto:Params: {} 2024-04-12T04:11:07.193 DEBUG:boto:Token: None 2024-04-12T04:11:07.193 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.194 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:orlQ2tkawLJDlEANu6qT4bOdats= 2024-04-12T04:11:07.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:orlQ2tkawLJDlEANu6qT4bOdats=', 'Content-Length': '0'} 2024-04-12T04:11:07.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076d3a64127435590-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.196 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:11:07.196 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:11:07.196 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.196 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.196 DEBUG:boto:Method: GET 2024-04-12T04:11:07.196 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.197 DEBUG:boto:Data: 2024-04-12T04:11:07.197 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.197 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.197 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.197 DEBUG:boto:Params: {} 2024-04-12T04:11:07.197 DEBUG:boto:Token: None 2024-04-12T04:11:07.197 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.197 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:orlQ2tkawLJDlEANu6qT4bOdats= 2024-04-12T04:11:07.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:orlQ2tkawLJDlEANu6qT4bOdats=', 'Content-Length': '0'} 2024-04-12T04:11:07.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6130134bb1ff4ce-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.242 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:11:07.242 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:11:07.243 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.243 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.243 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.243 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.243 DEBUG:boto:Data: 2024-04-12T04:11:07.243 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.243 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.243 DEBUG:boto:Params: {} 2024-04-12T04:11:07.243 DEBUG:boto:Token: None 2024-04-12T04:11:07.243 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.244 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fuBcMofn4+W+ABLGnm0IY2ltD/4= 2024-04-12T04:11:07.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fuBcMofn4+W+ABLGnm0IY2ltD/4=', 'Content-Length': '0'} 2024-04-12T04:11:07.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000622cb8194325fe0-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.246 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:11:07.246 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:11:07.246 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.246 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.246 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.246 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.247 DEBUG:boto:Data: 2024-04-12T04:11:07.247 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.247 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.247 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.247 DEBUG:boto:Params: {} 2024-04-12T04:11:07.247 DEBUG:boto:Token: None 2024-04-12T04:11:07.247 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.247 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fuBcMofn4+W+ABLGnm0IY2ltD/4= 2024-04-12T04:11:07.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fuBcMofn4+W+ABLGnm0IY2ltD/4=', 'Content-Length': '0'} 2024-04-12T04:11:07.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054022ca4657a71d1-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:11:07.250 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:11:07.250 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:11:07.250 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.250 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.250 DEBUG:boto:Method: GET 2024-04-12T04:11:07.250 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.250 DEBUG:boto:Data: 2024-04-12T04:11:07.250 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.250 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.250 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.250 DEBUG:boto:Params: {} 2024-04-12T04:11:07.251 DEBUG:boto:Token: None 2024-04-12T04:11:07.251 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.251 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:orlQ2tkawLJDlEANu6qT4bOdats= 2024-04-12T04:11:07.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:orlQ2tkawLJDlEANu6qT4bOdats=', 'Content-Length': '0'} 2024-04-12T04:11:07.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb1b3cb505f3ff06-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.253 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:11:07.253 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:11:07.253 DEBUG:boto:path=/dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.253 DEBUG:boto:auth_path=/dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.253 DEBUG:boto:Method: GET 2024-04-12T04:11:07.253 DEBUG:boto:Path: /dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.253 DEBUG:boto:Data: 2024-04-12T04:11:07.254 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.254 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.254 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.254 DEBUG:boto:Params: {} 2024-04-12T04:11:07.254 DEBUG:boto:Token: None 2024-04-12T04:11:07.254 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj46?versionId=null 2024-04-12T04:11:07.254 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:orlQ2tkawLJDlEANu6qT4bOdats= 2024-04-12T04:11:07.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:orlQ2tkawLJDlEANu6qT4bOdats=', 'Content-Length': '0'} 2024-04-12T04:11:07.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba5b7b25ae78d754-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.256 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.256 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.257 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:11:07.257 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:11:07.257 DEBUG:boto:Method: GET 2024-04-12T04:11:07.257 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:11:07.257 DEBUG:boto:Data: 2024-04-12T04:11:07.257 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.257 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.257 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.257 DEBUG:boto:Params: {} 2024-04-12T04:11:07.257 DEBUG:boto:Token: None 2024-04-12T04:11:07.258 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.259 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d728ba87093c6d35-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.261 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.262 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.262 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.262 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:11:07.262 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:11:07.262 DEBUG:boto:Method: GET 2024-04-12T04:11:07.262 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:11:07.262 DEBUG:boto:Data: 2024-04-12T04:11:07.262 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.263 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.263 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.263 DEBUG:boto:Params: {} 2024-04-12T04:11:07.263 DEBUG:boto:Token: None 2024-04-12T04:11:07.263 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.263 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000865f521d9e952a66-006618b45b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.267 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.267 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:11:07.267 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:11:07.268 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.268 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:11:07.268 DEBUG:boto:Data: 2024-04-12T04:11:07.268 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.268 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.268 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.268 DEBUG:boto:Params: {} 2024-04-12T04:11:07.268 DEBUG:boto:Token: None 2024-04-12T04:11:07.268 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj46 2024-04-12T04:11:07.269 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NAFETwGwOTjDIfnsw9su9PEp0F4= 2024-04-12T04:11:07.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NAFETwGwOTjDIfnsw9su9PEp0F4=', 'Content-Length': '0'} 2024-04-12T04:11:07.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043b899a009fa944f-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.270 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:11:07.271 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:11:07.271 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.271 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:11:07.271 DEBUG:boto:Data: 2024-04-12T04:11:07.271 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.271 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.271 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.271 DEBUG:boto:Params: {} 2024-04-12T04:11:07.271 DEBUG:boto:Token: None 2024-04-12T04:11:07.271 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj46 2024-04-12T04:11:07.271 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NAFETwGwOTjDIfnsw9su9PEp0F4= 2024-04-12T04:11:07.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NAFETwGwOTjDIfnsw9su9PEp0F4=', 'Content-Length': '0'} 2024-04-12T04:11:07.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a696ca45511a7f02-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:11:07.274 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:11:07.274 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:11:07.274 DEBUG:boto:Method: GET 2024-04-12T04:11:07.274 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:11:07.274 DEBUG:boto:Data: 2024-04-12T04:11:07.274 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.274 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.274 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.274 DEBUG:boto:Params: {} 2024-04-12T04:11:07.275 DEBUG:boto:Token: None 2024-04-12T04:11:07.275 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj46 2024-04-12T04:11:07.275 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dYmLzmNf2G8/mx8JP5ZkLZs7VtU= 2024-04-12T04:11:07.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dYmLzmNf2G8/mx8JP5ZkLZs7VtU=', 'Content-Length': '0'} 2024-04-12T04:11:07.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020d803c5885fcc93-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.277 DEBUG:boto:path=/dkyges-61/obj46 2024-04-12T04:11:07.277 DEBUG:boto:auth_path=/dkyges-61/obj46 2024-04-12T04:11:07.277 DEBUG:boto:Method: GET 2024-04-12T04:11:07.277 DEBUG:boto:Path: /dkyges-61/obj46 2024-04-12T04:11:07.277 DEBUG:boto:Data: 2024-04-12T04:11:07.277 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.277 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.277 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.278 DEBUG:boto:Params: {} 2024-04-12T04:11:07.278 DEBUG:boto:Token: None 2024-04-12T04:11:07.278 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj46 2024-04-12T04:11:07.278 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dYmLzmNf2G8/mx8JP5ZkLZs7VtU= 2024-04-12T04:11:07.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dYmLzmNf2G8/mx8JP5ZkLZs7VtU=', 'Content-Length': '0'} 2024-04-12T04:11:07.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0b6eb6daf5bd89a-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.280 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.280 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.280 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:11:07.280 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:11:07.280 DEBUG:boto:Method: GET 2024-04-12T04:11:07.281 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:11:07.281 DEBUG:boto:Data: 2024-04-12T04:11:07.281 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.281 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.281 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.281 DEBUG:boto:Params: {} 2024-04-12T04:11:07.281 DEBUG:boto:Token: None 2024-04-12T04:11:07.281 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.281 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bc5295bcb2814ab-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.284 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.284 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.284 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.285 DEBUG:boto:path=/dkyges-61/?prefix=obj46 2024-04-12T04:11:07.285 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj46 2024-04-12T04:11:07.285 DEBUG:boto:Method: GET 2024-04-12T04:11:07.285 DEBUG:boto:Path: /dkyges-61/?prefix=obj46 2024-04-12T04:11:07.285 DEBUG:boto:Data: 2024-04-12T04:11:07.285 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.285 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.285 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.285 DEBUG:boto:Params: {} 2024-04-12T04:11:07.285 DEBUG:boto:Token: None 2024-04-12T04:11:07.286 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.286 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fd7fc117a6c9657-006618b45b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.290 DEBUG:boto:b'dkyges-61obj461000falseobj462024-04-12T04:05:01.082Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:11:07.291 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:11:07.291 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:11:07.291 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.291 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.291 DEBUG:boto:Method: GET 2024-04-12T04:11:07.291 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.291 DEBUG:boto:Data: 2024-04-12T04:11:07.291 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.292 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.292 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.292 DEBUG:boto:Params: {} 2024-04-12T04:11:07.292 DEBUG:boto:Token: None 2024-04-12T04:11:07.292 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.292 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sGzoebD7bSB0Kjx+SSk9dQhkUJc= 2024-04-12T04:11:07.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sGzoebD7bSB0Kjx+SSk9dQhkUJc=', 'Content-Length': '0'} 2024-04-12T04:11:07.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d688632dce30a1d7-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.334 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:11:07.334 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:11:07.334 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.334 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.335 DEBUG:boto:Method: GET 2024-04-12T04:11:07.335 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.335 DEBUG:boto:Data: 2024-04-12T04:11:07.335 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.335 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.335 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.335 DEBUG:boto:Params: {} 2024-04-12T04:11:07.335 DEBUG:boto:Token: None 2024-04-12T04:11:07.335 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.335 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sGzoebD7bSB0Kjx+SSk9dQhkUJc= 2024-04-12T04:11:07.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sGzoebD7bSB0Kjx+SSk9dQhkUJc=', 'Content-Length': '0'} 2024-04-12T04:11:07.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054a32b1a763a6292-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.337 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:11:07.338 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:11:07.338 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.338 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.338 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.338 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.338 DEBUG:boto:Data: 2024-04-12T04:11:07.338 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.338 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.338 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.338 DEBUG:boto:Params: {} 2024-04-12T04:11:07.339 DEBUG:boto:Token: None 2024-04-12T04:11:07.339 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.339 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Lv0TvCNTuHcaWVKo8Vj/rAnpy1w= 2024-04-12T04:11:07.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Lv0TvCNTuHcaWVKo8Vj/rAnpy1w=', 'Content-Length': '0'} 2024-04-12T04:11:07.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d7b0f2e1e58a5d6-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.341 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:11:07.341 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:11:07.341 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.341 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.341 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.341 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.341 DEBUG:boto:Data: 2024-04-12T04:11:07.341 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.341 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.341 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.342 DEBUG:boto:Params: {} 2024-04-12T04:11:07.342 DEBUG:boto:Token: None 2024-04-12T04:11:07.342 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.342 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Lv0TvCNTuHcaWVKo8Vj/rAnpy1w= 2024-04-12T04:11:07.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Lv0TvCNTuHcaWVKo8Vj/rAnpy1w=', 'Content-Length': '0'} 2024-04-12T04:11:07.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aae3b0ea695bdfc3-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:11:07.344 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:11:07.345 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:11:07.345 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.345 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.345 DEBUG:boto:Method: GET 2024-04-12T04:11:07.345 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.345 DEBUG:boto:Data: 2024-04-12T04:11:07.345 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.345 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.345 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.345 DEBUG:boto:Params: {} 2024-04-12T04:11:07.345 DEBUG:boto:Token: None 2024-04-12T04:11:07.346 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.346 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sGzoebD7bSB0Kjx+SSk9dQhkUJc= 2024-04-12T04:11:07.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sGzoebD7bSB0Kjx+SSk9dQhkUJc=', 'Content-Length': '0'} 2024-04-12T04:11:07.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004422a457ec1d9446-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.348 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:11:07.348 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:11:07.348 DEBUG:boto:path=/dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.348 DEBUG:boto:auth_path=/dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.348 DEBUG:boto:Method: GET 2024-04-12T04:11:07.348 DEBUG:boto:Path: /dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.348 DEBUG:boto:Data: 2024-04-12T04:11:07.348 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.349 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.349 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.349 DEBUG:boto:Params: {} 2024-04-12T04:11:07.349 DEBUG:boto:Token: None 2024-04-12T04:11:07.349 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj51?versionId=null 2024-04-12T04:11:07.349 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sGzoebD7bSB0Kjx+SSk9dQhkUJc= 2024-04-12T04:11:07.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sGzoebD7bSB0Kjx+SSk9dQhkUJc=', 'Content-Length': '0'} 2024-04-12T04:11:07.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000148953d03599d99-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.351 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.352 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.352 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:11:07.352 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:11:07.352 DEBUG:boto:Method: GET 2024-04-12T04:11:07.352 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:11:07.352 DEBUG:boto:Data: 2024-04-12T04:11:07.352 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.352 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.352 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.352 DEBUG:boto:Params: {} 2024-04-12T04:11:07.352 DEBUG:boto:Token: None 2024-04-12T04:11:07.353 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.353 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aa9d45d1cc5a5b7-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.355 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.356 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.356 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.356 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:11:07.356 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:11:07.356 DEBUG:boto:Method: GET 2024-04-12T04:11:07.356 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:11:07.356 DEBUG:boto:Data: 2024-04-12T04:11:07.356 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.356 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.356 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.356 DEBUG:boto:Params: {} 2024-04-12T04:11:07.356 DEBUG:boto:Token: None 2024-04-12T04:11:07.357 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.357 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c57cc980f1329d59-006618b45b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.402 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.403 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:11:07.403 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:11:07.403 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.403 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:11:07.403 DEBUG:boto:Data: 2024-04-12T04:11:07.403 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.403 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.403 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.403 DEBUG:boto:Params: {} 2024-04-12T04:11:07.403 DEBUG:boto:Token: None 2024-04-12T04:11:07.404 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj51 2024-04-12T04:11:07.404 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iVfJYWS98rw0K5ARm1LiefZRRM0= 2024-04-12T04:11:07.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iVfJYWS98rw0K5ARm1LiefZRRM0=', 'Content-Length': '0'} 2024-04-12T04:11:07.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000191d5ec049fa408e-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.406 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:11:07.406 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:11:07.406 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.406 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:11:07.406 DEBUG:boto:Data: 2024-04-12T04:11:07.406 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.406 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.407 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.407 DEBUG:boto:Params: {} 2024-04-12T04:11:07.407 DEBUG:boto:Token: None 2024-04-12T04:11:07.407 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj51 2024-04-12T04:11:07.407 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iVfJYWS98rw0K5ARm1LiefZRRM0= 2024-04-12T04:11:07.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iVfJYWS98rw0K5ARm1LiefZRRM0=', 'Content-Length': '0'} 2024-04-12T04:11:07.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d06fd41779e18db-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:11:07.409 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:11:07.409 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:11:07.410 DEBUG:boto:Method: GET 2024-04-12T04:11:07.410 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:11:07.410 DEBUG:boto:Data: 2024-04-12T04:11:07.410 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.410 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.410 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.410 DEBUG:boto:Params: {} 2024-04-12T04:11:07.410 DEBUG:boto:Token: None 2024-04-12T04:11:07.410 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj51 2024-04-12T04:11:07.410 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pjsroV59Cb8/cUe8JrFZYRUF2kI= 2024-04-12T04:11:07.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pjsroV59Cb8/cUe8JrFZYRUF2kI=', 'Content-Length': '0'} 2024-04-12T04:11:07.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000693144dd9e7cf4ab-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.413 DEBUG:boto:path=/dkyges-61/obj51 2024-04-12T04:11:07.413 DEBUG:boto:auth_path=/dkyges-61/obj51 2024-04-12T04:11:07.413 DEBUG:boto:Method: GET 2024-04-12T04:11:07.413 DEBUG:boto:Path: /dkyges-61/obj51 2024-04-12T04:11:07.413 DEBUG:boto:Data: 2024-04-12T04:11:07.413 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.413 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.413 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.413 DEBUG:boto:Params: {} 2024-04-12T04:11:07.414 DEBUG:boto:Token: None 2024-04-12T04:11:07.414 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj51 2024-04-12T04:11:07.414 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pjsroV59Cb8/cUe8JrFZYRUF2kI= 2024-04-12T04:11:07.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pjsroV59Cb8/cUe8JrFZYRUF2kI=', 'Content-Length': '0'} 2024-04-12T04:11:07.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a83a34c269d58178-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.416 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.416 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.416 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:11:07.416 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:11:07.416 DEBUG:boto:Method: GET 2024-04-12T04:11:07.417 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:11:07.417 DEBUG:boto:Data: 2024-04-12T04:11:07.417 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.417 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.417 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.417 DEBUG:boto:Params: {} 2024-04-12T04:11:07.417 DEBUG:boto:Token: None 2024-04-12T04:11:07.417 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.417 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004af5f1ce85d5e4ad-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.420 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.420 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.421 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.421 DEBUG:boto:path=/dkyges-61/?prefix=obj51 2024-04-12T04:11:07.421 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj51 2024-04-12T04:11:07.421 DEBUG:boto:Method: GET 2024-04-12T04:11:07.421 DEBUG:boto:Path: /dkyges-61/?prefix=obj51 2024-04-12T04:11:07.421 DEBUG:boto:Data: 2024-04-12T04:11:07.421 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.421 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.421 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.421 DEBUG:boto:Params: {} 2024-04-12T04:11:07.421 DEBUG:boto:Token: None 2024-04-12T04:11:07.422 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.422 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a0287e73047d048-006618b45b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.426 DEBUG:boto:b'dkyges-61obj511000falseobj512024-04-12T04:00:41.699Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:11:07.427 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:11:07.427 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:11:07.427 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.427 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.427 DEBUG:boto:Method: GET 2024-04-12T04:11:07.427 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.427 DEBUG:boto:Data: 2024-04-12T04:11:07.427 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.427 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.428 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.428 DEBUG:boto:Params: {} 2024-04-12T04:11:07.428 DEBUG:boto:Token: None 2024-04-12T04:11:07.428 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.428 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+B4B+ziqFgNaqgNs3IqP1M2yAdU= 2024-04-12T04:11:07.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+B4B+ziqFgNaqgNs3IqP1M2yAdU=', 'Content-Length': '0'} 2024-04-12T04:11:07.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af8e4cd69076bac4-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.430 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:11:07.431 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:11:07.431 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.431 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.431 DEBUG:boto:Method: GET 2024-04-12T04:11:07.431 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.431 DEBUG:boto:Data: 2024-04-12T04:11:07.431 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.431 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.431 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.431 DEBUG:boto:Params: {} 2024-04-12T04:11:07.431 DEBUG:boto:Token: None 2024-04-12T04:11:07.432 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.432 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+B4B+ziqFgNaqgNs3IqP1M2yAdU= 2024-04-12T04:11:07.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+B4B+ziqFgNaqgNs3IqP1M2yAdU=', 'Content-Length': '0'} 2024-04-12T04:11:07.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059b88c745f3999cd-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.434 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:11:07.434 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:11:07.434 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.434 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.434 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.434 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.434 DEBUG:boto:Data: 2024-04-12T04:11:07.435 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.435 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.435 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.435 DEBUG:boto:Params: {} 2024-04-12T04:11:07.435 DEBUG:boto:Token: None 2024-04-12T04:11:07.435 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.435 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CcYeguWD8/6dqHlqFpEMAnG7gq8= 2024-04-12T04:11:07.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CcYeguWD8/6dqHlqFpEMAnG7gq8=', 'Content-Length': '0'} 2024-04-12T04:11:07.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8bc68e4f026df96-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.437 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:11:07.438 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:11:07.438 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.438 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.438 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.438 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.438 DEBUG:boto:Data: 2024-04-12T04:11:07.438 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.438 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.438 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.438 DEBUG:boto:Params: {} 2024-04-12T04:11:07.439 DEBUG:boto:Token: None 2024-04-12T04:11:07.439 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.439 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CcYeguWD8/6dqHlqFpEMAnG7gq8= 2024-04-12T04:11:07.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CcYeguWD8/6dqHlqFpEMAnG7gq8=', 'Content-Length': '0'} 2024-04-12T04:11:07.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6ac1c6de04f4a5c-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:11:07.441 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:11:07.441 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:11:07.441 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.441 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.441 DEBUG:boto:Method: GET 2024-04-12T04:11:07.441 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.442 DEBUG:boto:Data: 2024-04-12T04:11:07.442 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.442 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.442 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.442 DEBUG:boto:Params: {} 2024-04-12T04:11:07.442 DEBUG:boto:Token: None 2024-04-12T04:11:07.442 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.442 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+B4B+ziqFgNaqgNs3IqP1M2yAdU= 2024-04-12T04:11:07.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+B4B+ziqFgNaqgNs3IqP1M2yAdU=', 'Content-Length': '0'} 2024-04-12T04:11:07.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8ede6ec71f334dd-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.486 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:11:07.486 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:11:07.487 DEBUG:boto:path=/dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.487 DEBUG:boto:auth_path=/dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.487 DEBUG:boto:Method: GET 2024-04-12T04:11:07.487 DEBUG:boto:Path: /dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.487 DEBUG:boto:Data: 2024-04-12T04:11:07.487 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.487 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.487 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.487 DEBUG:boto:Params: {} 2024-04-12T04:11:07.487 DEBUG:boto:Token: None 2024-04-12T04:11:07.488 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj55?versionId=null 2024-04-12T04:11:07.488 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+B4B+ziqFgNaqgNs3IqP1M2yAdU= 2024-04-12T04:11:07.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+B4B+ziqFgNaqgNs3IqP1M2yAdU=', 'Content-Length': '0'} 2024-04-12T04:11:07.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043e6287ce740b5ba-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.490 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.490 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.490 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:11:07.490 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:11:07.490 DEBUG:boto:Method: GET 2024-04-12T04:11:07.490 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:11:07.490 DEBUG:boto:Data: 2024-04-12T04:11:07.491 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.491 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.491 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.491 DEBUG:boto:Params: {} 2024-04-12T04:11:07.491 DEBUG:boto:Token: None 2024-04-12T04:11:07.491 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.491 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008cfee2872f9dbe5-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.494 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.494 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.494 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.494 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:11:07.494 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:11:07.495 DEBUG:boto:Method: GET 2024-04-12T04:11:07.495 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:11:07.495 DEBUG:boto:Data: 2024-04-12T04:11:07.495 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.495 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.495 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.495 DEBUG:boto:Params: {} 2024-04-12T04:11:07.495 DEBUG:boto:Token: None 2024-04-12T04:11:07.495 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.495 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.499 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000750fd857f24b51c1-006618b45b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.500 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.500 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:11:07.500 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:11:07.500 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.500 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:11:07.500 DEBUG:boto:Data: 2024-04-12T04:11:07.501 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.501 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.501 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.501 DEBUG:boto:Params: {} 2024-04-12T04:11:07.501 DEBUG:boto:Token: None 2024-04-12T04:11:07.501 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj55 2024-04-12T04:11:07.501 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EnnZi+mRoj+KoOvnkZ3BLp14aDM= 2024-04-12T04:11:07.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EnnZi+mRoj+KoOvnkZ3BLp14aDM=', 'Content-Length': '0'} 2024-04-12T04:11:07.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000236e3776243e11bf-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.503 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:11:07.503 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:11:07.503 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.503 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:11:07.504 DEBUG:boto:Data: 2024-04-12T04:11:07.504 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.504 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.504 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.504 DEBUG:boto:Params: {} 2024-04-12T04:11:07.504 DEBUG:boto:Token: None 2024-04-12T04:11:07.504 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj55 2024-04-12T04:11:07.504 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EnnZi+mRoj+KoOvnkZ3BLp14aDM= 2024-04-12T04:11:07.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EnnZi+mRoj+KoOvnkZ3BLp14aDM=', 'Content-Length': '0'} 2024-04-12T04:11:07.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c846c4f01b627ec-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:11:07.506 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:11:07.507 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:11:07.507 DEBUG:boto:Method: GET 2024-04-12T04:11:07.507 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:11:07.507 DEBUG:boto:Data: 2024-04-12T04:11:07.507 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.507 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.507 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.507 DEBUG:boto:Params: {} 2024-04-12T04:11:07.507 DEBUG:boto:Token: None 2024-04-12T04:11:07.507 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj55 2024-04-12T04:11:07.508 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j3woUpfeCgMJlwqhkJXHNUbGmGQ= 2024-04-12T04:11:07.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j3woUpfeCgMJlwqhkJXHNUbGmGQ=', 'Content-Length': '0'} 2024-04-12T04:11:07.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041deb789024d72c3-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.510 DEBUG:boto:path=/dkyges-61/obj55 2024-04-12T04:11:07.510 DEBUG:boto:auth_path=/dkyges-61/obj55 2024-04-12T04:11:07.510 DEBUG:boto:Method: GET 2024-04-12T04:11:07.510 DEBUG:boto:Path: /dkyges-61/obj55 2024-04-12T04:11:07.510 DEBUG:boto:Data: 2024-04-12T04:11:07.510 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.510 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.510 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.511 DEBUG:boto:Params: {} 2024-04-12T04:11:07.511 DEBUG:boto:Token: None 2024-04-12T04:11:07.511 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj55 2024-04-12T04:11:07.511 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j3woUpfeCgMJlwqhkJXHNUbGmGQ= 2024-04-12T04:11:07.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j3woUpfeCgMJlwqhkJXHNUbGmGQ=', 'Content-Length': '0'} 2024-04-12T04:11:07.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adeb77bd44591337-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.554 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.554 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.554 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:11:07.555 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:11:07.555 DEBUG:boto:Method: GET 2024-04-12T04:11:07.555 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:11:07.555 DEBUG:boto:Data: 2024-04-12T04:11:07.555 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.555 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.555 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.555 DEBUG:boto:Params: {} 2024-04-12T04:11:07.555 DEBUG:boto:Token: None 2024-04-12T04:11:07.555 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.555 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003228488e1cc659ca-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.558 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.558 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.559 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.559 DEBUG:boto:path=/dkyges-61/?prefix=obj55 2024-04-12T04:11:07.559 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj55 2024-04-12T04:11:07.559 DEBUG:boto:Method: GET 2024-04-12T04:11:07.559 DEBUG:boto:Path: /dkyges-61/?prefix=obj55 2024-04-12T04:11:07.559 DEBUG:boto:Data: 2024-04-12T04:11:07.559 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.559 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.559 DEBUG:boto:Params: {} 2024-04-12T04:11:07.560 DEBUG:boto:Token: None 2024-04-12T04:11:07.560 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.560 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e063ed4bff536f5-006618b45b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.564 DEBUG:boto:b'dkyges-61obj551000falseobj552024-04-12T04:00:30.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:11:07.565 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:11:07.565 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:11:07.565 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.565 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.565 DEBUG:boto:Method: GET 2024-04-12T04:11:07.565 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.565 DEBUG:boto:Data: 2024-04-12T04:11:07.565 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.565 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.565 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.566 DEBUG:boto:Params: {} 2024-04-12T04:11:07.566 DEBUG:boto:Token: None 2024-04-12T04:11:07.566 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.566 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:98bFh5X2KecyL8JupwR4HDp0iV0= 2024-04-12T04:11:07.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:98bFh5X2KecyL8JupwR4HDp0iV0=', 'Content-Length': '0'} 2024-04-12T04:11:07.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002abdf6538ff8695-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.569 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:11:07.569 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:11:07.569 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.569 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.569 DEBUG:boto:Method: GET 2024-04-12T04:11:07.569 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.569 DEBUG:boto:Data: 2024-04-12T04:11:07.569 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.570 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.570 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.570 DEBUG:boto:Params: {} 2024-04-12T04:11:07.570 DEBUG:boto:Token: None 2024-04-12T04:11:07.570 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.570 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:98bFh5X2KecyL8JupwR4HDp0iV0= 2024-04-12T04:11:07.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:98bFh5X2KecyL8JupwR4HDp0iV0=', 'Content-Length': '0'} 2024-04-12T04:11:07.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a9699e24049cd15-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.572 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:11:07.572 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:11:07.572 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.573 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.573 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.573 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.573 DEBUG:boto:Data: 2024-04-12T04:11:07.573 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.573 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.573 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.573 DEBUG:boto:Params: {} 2024-04-12T04:11:07.573 DEBUG:boto:Token: None 2024-04-12T04:11:07.573 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.573 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kNzO7aMK24e9goejOBxGUZZWSF4= 2024-04-12T04:11:07.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kNzO7aMK24e9goejOBxGUZZWSF4=', 'Content-Length': '0'} 2024-04-12T04:11:07.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009007f04db244fae-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.575 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:11:07.576 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:11:07.576 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.576 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.576 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.576 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.576 DEBUG:boto:Data: 2024-04-12T04:11:07.576 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.576 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.576 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.576 DEBUG:boto:Params: {} 2024-04-12T04:11:07.576 DEBUG:boto:Token: None 2024-04-12T04:11:07.577 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.577 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kNzO7aMK24e9goejOBxGUZZWSF4= 2024-04-12T04:11:07.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kNzO7aMK24e9goejOBxGUZZWSF4=', 'Content-Length': '0'} 2024-04-12T04:11:07.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005de90b96664cc3ed-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:11:07.579 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:11:07.579 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:11:07.579 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.579 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.579 DEBUG:boto:Method: GET 2024-04-12T04:11:07.579 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.579 DEBUG:boto:Data: 2024-04-12T04:11:07.579 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.579 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.579 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.580 DEBUG:boto:Params: {} 2024-04-12T04:11:07.580 DEBUG:boto:Token: None 2024-04-12T04:11:07.580 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.580 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:98bFh5X2KecyL8JupwR4HDp0iV0= 2024-04-12T04:11:07.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:98bFh5X2KecyL8JupwR4HDp0iV0=', 'Content-Length': '0'} 2024-04-12T04:11:07.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de8ddb2c6c2c9522-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.622 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:11:07.622 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:11:07.622 DEBUG:boto:path=/dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.623 DEBUG:boto:auth_path=/dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.623 DEBUG:boto:Method: GET 2024-04-12T04:11:07.623 DEBUG:boto:Path: /dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.623 DEBUG:boto:Data: 2024-04-12T04:11:07.623 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.623 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.623 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.623 DEBUG:boto:Params: {} 2024-04-12T04:11:07.623 DEBUG:boto:Token: None 2024-04-12T04:11:07.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj57?versionId=null 2024-04-12T04:11:07.623 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:98bFh5X2KecyL8JupwR4HDp0iV0= 2024-04-12T04:11:07.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:98bFh5X2KecyL8JupwR4HDp0iV0=', 'Content-Length': '0'} 2024-04-12T04:11:07.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1126a805bd5f196-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.626 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.626 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.626 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:11:07.626 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:11:07.626 DEBUG:boto:Method: GET 2024-04-12T04:11:07.626 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:11:07.626 DEBUG:boto:Data: 2024-04-12T04:11:07.626 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.626 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.627 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.627 DEBUG:boto:Params: {} 2024-04-12T04:11:07.627 DEBUG:boto:Token: None 2024-04-12T04:11:07.627 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.627 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.629 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d586d9afba6c206-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.630 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.630 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.630 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.630 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:11:07.630 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:11:07.630 DEBUG:boto:Method: GET 2024-04-12T04:11:07.630 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:11:07.630 DEBUG:boto:Data: 2024-04-12T04:11:07.631 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.631 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.631 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.631 DEBUG:boto:Params: {} 2024-04-12T04:11:07.631 DEBUG:boto:Token: None 2024-04-12T04:11:07.631 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.631 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4e9e7b7dd3297e3-006618b45b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.635 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.636 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:11:07.636 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:11:07.636 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.636 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:11:07.636 DEBUG:boto:Data: 2024-04-12T04:11:07.636 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.636 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.636 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.636 DEBUG:boto:Params: {} 2024-04-12T04:11:07.637 DEBUG:boto:Token: None 2024-04-12T04:11:07.637 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj57 2024-04-12T04:11:07.637 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:O9k95+P/G9t1PmOqi7dz1GE8/Tw= 2024-04-12T04:11:07.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:O9k95+P/G9t1PmOqi7dz1GE8/Tw=', 'Content-Length': '0'} 2024-04-12T04:11:07.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bec8e349f279f35-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.639 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:11:07.639 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:11:07.639 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.639 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:11:07.640 DEBUG:boto:Data: 2024-04-12T04:11:07.640 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.640 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.640 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.640 DEBUG:boto:Params: {} 2024-04-12T04:11:07.640 DEBUG:boto:Token: None 2024-04-12T04:11:07.640 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj57 2024-04-12T04:11:07.640 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:O9k95+P/G9t1PmOqi7dz1GE8/Tw= 2024-04-12T04:11:07.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:O9k95+P/G9t1PmOqi7dz1GE8/Tw=', 'Content-Length': '0'} 2024-04-12T04:11:07.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b04c0ddce116dae8-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:11:07.646 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:11:07.646 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:11:07.646 DEBUG:boto:Method: GET 2024-04-12T04:11:07.646 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:11:07.647 DEBUG:boto:Data: 2024-04-12T04:11:07.647 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.647 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.647 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.647 DEBUG:boto:Params: {} 2024-04-12T04:11:07.647 DEBUG:boto:Token: None 2024-04-12T04:11:07.647 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj57 2024-04-12T04:11:07.647 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:waDhYI0NsYHIwFWbQhYngto/gDA= 2024-04-12T04:11:07.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:waDhYI0NsYHIwFWbQhYngto/gDA=', 'Content-Length': '0'} 2024-04-12T04:11:07.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ea3e826db2eca01-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.650 DEBUG:boto:path=/dkyges-61/obj57 2024-04-12T04:11:07.650 DEBUG:boto:auth_path=/dkyges-61/obj57 2024-04-12T04:11:07.650 DEBUG:boto:Method: GET 2024-04-12T04:11:07.650 DEBUG:boto:Path: /dkyges-61/obj57 2024-04-12T04:11:07.650 DEBUG:boto:Data: 2024-04-12T04:11:07.650 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.650 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.650 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.651 DEBUG:boto:Params: {} 2024-04-12T04:11:07.651 DEBUG:boto:Token: None 2024-04-12T04:11:07.651 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj57 2024-04-12T04:11:07.651 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:waDhYI0NsYHIwFWbQhYngto/gDA= 2024-04-12T04:11:07.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:waDhYI0NsYHIwFWbQhYngto/gDA=', 'Content-Length': '0'} 2024-04-12T04:11:07.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c013700477f3c2a9-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.694 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.694 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.695 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:11:07.695 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:11:07.695 DEBUG:boto:Method: GET 2024-04-12T04:11:07.695 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:11:07.695 DEBUG:boto:Data: 2024-04-12T04:11:07.695 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.695 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.695 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.695 DEBUG:boto:Params: {} 2024-04-12T04:11:07.696 DEBUG:boto:Token: None 2024-04-12T04:11:07.696 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.696 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000046a17fc10f12905-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.699 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.699 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.700 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.700 DEBUG:boto:path=/dkyges-61/?prefix=obj57 2024-04-12T04:11:07.700 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj57 2024-04-12T04:11:07.700 DEBUG:boto:Method: GET 2024-04-12T04:11:07.700 DEBUG:boto:Path: /dkyges-61/?prefix=obj57 2024-04-12T04:11:07.700 DEBUG:boto:Data: 2024-04-12T04:11:07.700 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.700 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.700 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.700 DEBUG:boto:Params: {} 2024-04-12T04:11:07.701 DEBUG:boto:Token: None 2024-04-12T04:11:07.701 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.701 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b329b406212fe0b2-006618b45b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.706 DEBUG:boto:b'dkyges-61obj571000falseobj572024-04-12T04:03:03.638Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:11:07.706 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:11:07.706 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:11:07.706 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.707 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.707 DEBUG:boto:Method: GET 2024-04-12T04:11:07.707 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.707 DEBUG:boto:Data: 2024-04-12T04:11:07.707 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.707 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.707 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.707 DEBUG:boto:Params: {} 2024-04-12T04:11:07.707 DEBUG:boto:Token: None 2024-04-12T04:11:07.707 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.707 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pGt+rRC6E6fN/0wuipjLUfGk8xk= 2024-04-12T04:11:07.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pGt+rRC6E6fN/0wuipjLUfGk8xk=', 'Content-Length': '0'} 2024-04-12T04:11:07.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038de5ccc39c4363e-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.754 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:11:07.754 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:11:07.755 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.755 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.755 DEBUG:boto:Method: GET 2024-04-12T04:11:07.755 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.755 DEBUG:boto:Data: 2024-04-12T04:11:07.755 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.755 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.755 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.755 DEBUG:boto:Params: {} 2024-04-12T04:11:07.755 DEBUG:boto:Token: None 2024-04-12T04:11:07.756 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.756 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pGt+rRC6E6fN/0wuipjLUfGk8xk= 2024-04-12T04:11:07.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pGt+rRC6E6fN/0wuipjLUfGk8xk=', 'Content-Length': '0'} 2024-04-12T04:11:07.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084746e175dffeb0d-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.758 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:11:07.758 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:11:07.758 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.758 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.759 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.759 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.759 DEBUG:boto:Data: 2024-04-12T04:11:07.759 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.759 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.759 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.759 DEBUG:boto:Params: {} 2024-04-12T04:11:07.759 DEBUG:boto:Token: None 2024-04-12T04:11:07.759 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.759 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iCDjkeG+ndafH4GknwcuKG07tco= 2024-04-12T04:11:07.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iCDjkeG+ndafH4GknwcuKG07tco=', 'Content-Length': '0'} 2024-04-12T04:11:07.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000245cacb2ef2a2d56-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.762 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:11:07.762 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:11:07.762 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.762 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.762 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.762 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.762 DEBUG:boto:Data: 2024-04-12T04:11:07.762 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.762 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.762 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.763 DEBUG:boto:Params: {} 2024-04-12T04:11:07.763 DEBUG:boto:Token: None 2024-04-12T04:11:07.763 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.763 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iCDjkeG+ndafH4GknwcuKG07tco= 2024-04-12T04:11:07.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iCDjkeG+ndafH4GknwcuKG07tco=', 'Content-Length': '0'} 2024-04-12T04:11:07.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b41e989fb9073ae-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:11:07.765 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:11:07.765 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:11:07.765 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.765 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.765 DEBUG:boto:Method: GET 2024-04-12T04:11:07.766 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.766 DEBUG:boto:Data: 2024-04-12T04:11:07.766 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.766 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.766 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.766 DEBUG:boto:Params: {} 2024-04-12T04:11:07.766 DEBUG:boto:Token: None 2024-04-12T04:11:07.766 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.766 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pGt+rRC6E6fN/0wuipjLUfGk8xk= 2024-04-12T04:11:07.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pGt+rRC6E6fN/0wuipjLUfGk8xk=', 'Content-Length': '0'} 2024-04-12T04:11:07.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071b579983bafec76-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.768 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:11:07.769 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:11:07.769 DEBUG:boto:path=/dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.769 DEBUG:boto:auth_path=/dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.769 DEBUG:boto:Method: GET 2024-04-12T04:11:07.769 DEBUG:boto:Path: /dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.769 DEBUG:boto:Data: 2024-04-12T04:11:07.769 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.769 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.769 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.769 DEBUG:boto:Params: {} 2024-04-12T04:11:07.769 DEBUG:boto:Token: None 2024-04-12T04:11:07.770 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj58?versionId=null 2024-04-12T04:11:07.770 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pGt+rRC6E6fN/0wuipjLUfGk8xk= 2024-04-12T04:11:07.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pGt+rRC6E6fN/0wuipjLUfGk8xk=', 'Content-Length': '0'} 2024-04-12T04:11:07.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080d678607d9a7d17-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.772 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.772 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.772 DEBUG:boto:path=/dkyges-61/?prefix=obj58 2024-04-12T04:11:07.773 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj58 2024-04-12T04:11:07.773 DEBUG:boto:Method: GET 2024-04-12T04:11:07.773 DEBUG:boto:Path: /dkyges-61/?prefix=obj58 2024-04-12T04:11:07.773 DEBUG:boto:Data: 2024-04-12T04:11:07.773 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.773 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.773 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.773 DEBUG:boto:Params: {} 2024-04-12T04:11:07.773 DEBUG:boto:Token: None 2024-04-12T04:11:07.773 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.774 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032f9cc108fa225da-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.776 DEBUG:boto:b'dkyges-61obj581000falseobj582024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.777 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.777 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.777 DEBUG:boto:path=/dkyges-61/?prefix=obj58 2024-04-12T04:11:07.777 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj58 2024-04-12T04:11:07.777 DEBUG:boto:Method: GET 2024-04-12T04:11:07.777 DEBUG:boto:Path: /dkyges-61/?prefix=obj58 2024-04-12T04:11:07.777 DEBUG:boto:Data: 2024-04-12T04:11:07.777 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.777 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.777 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.777 DEBUG:boto:Params: {} 2024-04-12T04:11:07.778 DEBUG:boto:Token: None 2024-04-12T04:11:07.778 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.778 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000202f96b2a7df8b68-006618b45b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.782 DEBUG:boto:b'dkyges-61obj581000falseobj582024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.783 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:11:07.783 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:11:07.783 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.783 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:11:07.783 DEBUG:boto:Data: 2024-04-12T04:11:07.783 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.783 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.783 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.783 DEBUG:boto:Params: {} 2024-04-12T04:11:07.784 DEBUG:boto:Token: None 2024-04-12T04:11:07.784 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj58 2024-04-12T04:11:07.784 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5tpPMpB9X1vTg2WbJ6UG1zdMY14= 2024-04-12T04:11:07.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5tpPMpB9X1vTg2WbJ6UG1zdMY14=', 'Content-Length': '0'} 2024-04-12T04:11:07.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5144dc97bb003cc-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.786 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:11:07.786 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:11:07.786 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.786 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:11:07.786 DEBUG:boto:Data: 2024-04-12T04:11:07.786 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.787 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.787 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.787 DEBUG:boto:Params: {} 2024-04-12T04:11:07.787 DEBUG:boto:Token: None 2024-04-12T04:11:07.787 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj58 2024-04-12T04:11:07.787 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5tpPMpB9X1vTg2WbJ6UG1zdMY14= 2024-04-12T04:11:07.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5tpPMpB9X1vTg2WbJ6UG1zdMY14=', 'Content-Length': '0'} 2024-04-12T04:11:07.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f230b8b7709f161a-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:11:07.789 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:11:07.789 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:11:07.790 DEBUG:boto:Method: GET 2024-04-12T04:11:07.790 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:11:07.790 DEBUG:boto:Data: 2024-04-12T04:11:07.790 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.790 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.790 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.790 DEBUG:boto:Params: {} 2024-04-12T04:11:07.790 DEBUG:boto:Token: None 2024-04-12T04:11:07.790 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj58 2024-04-12T04:11:07.791 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+suGipqrqiFr4iLa0KWE/t6YZS0= 2024-04-12T04:11:07.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+suGipqrqiFr4iLa0KWE/t6YZS0=', 'Content-Length': '0'} 2024-04-12T04:11:07.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d3820ed6b14d5d6-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.793 DEBUG:boto:path=/dkyges-61/obj58 2024-04-12T04:11:07.793 DEBUG:boto:auth_path=/dkyges-61/obj58 2024-04-12T04:11:07.793 DEBUG:boto:Method: GET 2024-04-12T04:11:07.793 DEBUG:boto:Path: /dkyges-61/obj58 2024-04-12T04:11:07.793 DEBUG:boto:Data: 2024-04-12T04:11:07.793 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.793 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.793 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.793 DEBUG:boto:Params: {} 2024-04-12T04:11:07.794 DEBUG:boto:Token: None 2024-04-12T04:11:07.794 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj58 2024-04-12T04:11:07.794 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+suGipqrqiFr4iLa0KWE/t6YZS0= 2024-04-12T04:11:07.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+suGipqrqiFr4iLa0KWE/t6YZS0=', 'Content-Length': '0'} 2024-04-12T04:11:07.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b54da881a5afc117-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.796 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.796 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.796 DEBUG:boto:path=/dkyges-61/?prefix=obj58 2024-04-12T04:11:07.796 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj58 2024-04-12T04:11:07.796 DEBUG:boto:Method: GET 2024-04-12T04:11:07.797 DEBUG:boto:Path: /dkyges-61/?prefix=obj58 2024-04-12T04:11:07.797 DEBUG:boto:Data: 2024-04-12T04:11:07.797 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.797 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.797 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.797 DEBUG:boto:Params: {} 2024-04-12T04:11:07.797 DEBUG:boto:Token: None 2024-04-12T04:11:07.797 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.797 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa4899177f571a27-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.800 DEBUG:boto:b'dkyges-61obj581000falseobj582024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.800 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.800 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.800 DEBUG:boto:path=/dkyges-61/?prefix=obj58 2024-04-12T04:11:07.801 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj58 2024-04-12T04:11:07.801 DEBUG:boto:Method: GET 2024-04-12T04:11:07.801 DEBUG:boto:Path: /dkyges-61/?prefix=obj58 2024-04-12T04:11:07.801 DEBUG:boto:Data: 2024-04-12T04:11:07.801 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.801 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.801 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.801 DEBUG:boto:Params: {} 2024-04-12T04:11:07.801 DEBUG:boto:Token: None 2024-04-12T04:11:07.801 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.802 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc6257c3e75a0cca-006618b45b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.806 DEBUG:boto:b'dkyges-61obj581000falseobj582024-04-12T04:06:49.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T04:11:07.807 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:11:07.807 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:11:07.807 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.808 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.808 DEBUG:boto:Method: GET 2024-04-12T04:11:07.808 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.808 DEBUG:boto:Data: 2024-04-12T04:11:07.808 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.808 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.808 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.808 DEBUG:boto:Params: {} 2024-04-12T04:11:07.809 DEBUG:boto:Token: None 2024-04-12T04:11:07.809 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.809 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IqHQP5EO9i2fOAL7bP/x0/Jn36k= 2024-04-12T04:11:07.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IqHQP5EO9i2fOAL7bP/x0/Jn36k=', 'Content-Length': '0'} 2024-04-12T04:11:07.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054ddbb54fef60337-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.854 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:11:07.854 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:11:07.855 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.855 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.855 DEBUG:boto:Method: GET 2024-04-12T04:11:07.855 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.855 DEBUG:boto:Data: 2024-04-12T04:11:07.855 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.855 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.855 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.855 DEBUG:boto:Params: {} 2024-04-12T04:11:07.855 DEBUG:boto:Token: None 2024-04-12T04:11:07.856 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.856 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IqHQP5EO9i2fOAL7bP/x0/Jn36k= 2024-04-12T04:11:07.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IqHQP5EO9i2fOAL7bP/x0/Jn36k=', 'Content-Length': '0'} 2024-04-12T04:11:07.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3f74336ece20d38-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.858 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:11:07.858 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:11:07.858 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.858 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.858 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.858 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.859 DEBUG:boto:Data: 2024-04-12T04:11:07.859 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.859 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.859 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.859 DEBUG:boto:Params: {} 2024-04-12T04:11:07.859 DEBUG:boto:Token: None 2024-04-12T04:11:07.859 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.859 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9Y+0Gk8uvB+p2+dT1fEhzar5xEU= 2024-04-12T04:11:07.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9Y+0Gk8uvB+p2+dT1fEhzar5xEU=', 'Content-Length': '0'} 2024-04-12T04:11:07.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039431fe366b9604b-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.862 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:11:07.862 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:11:07.862 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.862 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.862 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.862 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.862 DEBUG:boto:Data: 2024-04-12T04:11:07.862 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.863 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.863 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.863 DEBUG:boto:Params: {} 2024-04-12T04:11:07.863 DEBUG:boto:Token: None 2024-04-12T04:11:07.863 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.863 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9Y+0Gk8uvB+p2+dT1fEhzar5xEU= 2024-04-12T04:11:07.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9Y+0Gk8uvB+p2+dT1fEhzar5xEU=', 'Content-Length': '0'} 2024-04-12T04:11:07.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010686c4bad9a42cd-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T04:11:07.865 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:11:07.865 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:11:07.865 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.865 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.866 DEBUG:boto:Method: GET 2024-04-12T04:11:07.866 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.866 DEBUG:boto:Data: 2024-04-12T04:11:07.866 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.866 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.866 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.866 DEBUG:boto:Params: {} 2024-04-12T04:11:07.866 DEBUG:boto:Token: None 2024-04-12T04:11:07.866 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.866 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IqHQP5EO9i2fOAL7bP/x0/Jn36k= 2024-04-12T04:11:07.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IqHQP5EO9i2fOAL7bP/x0/Jn36k=', 'Content-Length': '0'} 2024-04-12T04:11:07.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6298fd14c9f43a2-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.869 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:11:07.869 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:11:07.869 DEBUG:boto:path=/dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.869 DEBUG:boto:auth_path=/dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.869 DEBUG:boto:Method: GET 2024-04-12T04:11:07.869 DEBUG:boto:Path: /dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.869 DEBUG:boto:Data: 2024-04-12T04:11:07.869 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.869 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.869 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.869 DEBUG:boto:Params: {} 2024-04-12T04:11:07.870 DEBUG:boto:Token: None 2024-04-12T04:11:07.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj62?versionId=null 2024-04-12T04:11:07.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IqHQP5EO9i2fOAL7bP/x0/Jn36k= 2024-04-12T04:11:07.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IqHQP5EO9i2fOAL7bP/x0/Jn36k=', 'Content-Length': '0'} 2024-04-12T04:11:07.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6d90a89eb2472a2-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.872 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.872 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.872 DEBUG:boto:path=/dkyges-61/?prefix=obj62 2024-04-12T04:11:07.872 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj62 2024-04-12T04:11:07.873 DEBUG:boto:Method: GET 2024-04-12T04:11:07.873 DEBUG:boto:Path: /dkyges-61/?prefix=obj62 2024-04-12T04:11:07.873 DEBUG:boto:Data: 2024-04-12T04:11:07.873 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.873 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.873 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.873 DEBUG:boto:Params: {} 2024-04-12T04:11:07.873 DEBUG:boto:Token: None 2024-04-12T04:11:07.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.873 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baf6a732c2de3ab5-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.876 DEBUG:boto:b'dkyges-61obj621000falseobj622024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.876 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.877 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.877 DEBUG:boto:path=/dkyges-61/?prefix=obj62 2024-04-12T04:11:07.877 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj62 2024-04-12T04:11:07.877 DEBUG:boto:Method: GET 2024-04-12T04:11:07.877 DEBUG:boto:Path: /dkyges-61/?prefix=obj62 2024-04-12T04:11:07.877 DEBUG:boto:Data: 2024-04-12T04:11:07.877 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.877 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.877 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.877 DEBUG:boto:Params: {} 2024-04-12T04:11:07.877 DEBUG:boto:Token: None 2024-04-12T04:11:07.878 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.878 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061cdad0284fdf6c9-006618b45b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.884 DEBUG:boto:b'dkyges-61obj621000falseobj622024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.884 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:11:07.884 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:11:07.885 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.885 DEBUG:boto:Path: /dkyges-61/obj62 2024-04-12T04:11:07.885 DEBUG:boto:Data: 2024-04-12T04:11:07.885 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.885 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.885 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.885 DEBUG:boto:Params: {} 2024-04-12T04:11:07.885 DEBUG:boto:Token: None 2024-04-12T04:11:07.885 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj62 2024-04-12T04:11:07.885 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9gOxefvA9lgnr2RozfBgi1RTLUM= 2024-04-12T04:11:07.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9gOxefvA9lgnr2RozfBgi1RTLUM=', 'Content-Length': '0'} 2024-04-12T04:11:07.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097ec47f35320ce43-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.888 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:11:07.888 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:11:07.888 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.888 DEBUG:boto:Path: /dkyges-61/obj62 2024-04-12T04:11:07.888 DEBUG:boto:Data: 2024-04-12T04:11:07.889 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.889 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.889 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.889 DEBUG:boto:Params: {} 2024-04-12T04:11:07.889 DEBUG:boto:Token: None 2024-04-12T04:11:07.889 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj62 2024-04-12T04:11:07.889 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9gOxefvA9lgnr2RozfBgi1RTLUM= 2024-04-12T04:11:07.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9gOxefvA9lgnr2RozfBgi1RTLUM=', 'Content-Length': '0'} 2024-04-12T04:11:07.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002417216f44cc1f66-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T04:11:07.891 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:11:07.891 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:11:07.891 DEBUG:boto:Method: GET 2024-04-12T04:11:07.892 DEBUG:boto:Path: /dkyges-61/obj62 2024-04-12T04:11:07.892 DEBUG:boto:Data: 2024-04-12T04:11:07.892 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.892 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.892 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.892 DEBUG:boto:Params: {} 2024-04-12T04:11:07.892 DEBUG:boto:Token: None 2024-04-12T04:11:07.892 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj62 2024-04-12T04:11:07.892 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3aRoOuDPxbvJ+L1oSv672F7odt8= 2024-04-12T04:11:07.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3aRoOuDPxbvJ+L1oSv672F7odt8=', 'Content-Length': '0'} 2024-04-12T04:11:07.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089c6bab985013efc-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.942 DEBUG:boto:path=/dkyges-61/obj62 2024-04-12T04:11:07.942 DEBUG:boto:auth_path=/dkyges-61/obj62 2024-04-12T04:11:07.942 DEBUG:boto:Method: GET 2024-04-12T04:11:07.943 DEBUG:boto:Path: /dkyges-61/obj62 2024-04-12T04:11:07.943 DEBUG:boto:Data: 2024-04-12T04:11:07.943 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.943 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.943 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.943 DEBUG:boto:Params: {} 2024-04-12T04:11:07.943 DEBUG:boto:Token: None 2024-04-12T04:11:07.943 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj62 2024-04-12T04:11:07.943 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3aRoOuDPxbvJ+L1oSv672F7odt8= 2024-04-12T04:11:07.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3aRoOuDPxbvJ+L1oSv672F7odt8=', 'Content-Length': '0'} 2024-04-12T04:11:07.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f38307f27105990-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.946 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.946 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.946 DEBUG:boto:path=/dkyges-61/?prefix=obj62 2024-04-12T04:11:07.946 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj62 2024-04-12T04:11:07.946 DEBUG:boto:Method: GET 2024-04-12T04:11:07.946 DEBUG:boto:Path: /dkyges-61/?prefix=obj62 2024-04-12T04:11:07.946 DEBUG:boto:Data: 2024-04-12T04:11:07.947 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.947 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.947 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.947 DEBUG:boto:Params: {} 2024-04-12T04:11:07.947 DEBUG:boto:Token: None 2024-04-12T04:11:07.947 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.947 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.950 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db660be5a3999405-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.950 DEBUG:boto:b'dkyges-61obj621000falseobj622024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.950 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.950 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.950 DEBUG:boto:path=/dkyges-61/?prefix=obj62 2024-04-12T04:11:07.950 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj62 2024-04-12T04:11:07.950 DEBUG:boto:Method: GET 2024-04-12T04:11:07.951 DEBUG:boto:Path: /dkyges-61/?prefix=obj62 2024-04-12T04:11:07.951 DEBUG:boto:Data: 2024-04-12T04:11:07.951 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.951 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.951 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.951 DEBUG:boto:Params: {} 2024-04-12T04:11:07.951 DEBUG:boto:Token: None 2024-04-12T04:11:07.951 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.951 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e078cc4eb38d694-006618b45b-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.956 DEBUG:boto:b'dkyges-61obj621000falseobj622024-04-12T04:08:08.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:11:07.956 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:11:07.957 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:11:07.957 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.957 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.957 DEBUG:boto:Method: GET 2024-04-12T04:11:07.957 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.957 DEBUG:boto:Data: 2024-04-12T04:11:07.957 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.957 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.957 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.957 DEBUG:boto:Params: {} 2024-04-12T04:11:07.957 DEBUG:boto:Token: None 2024-04-12T04:11:07.958 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.958 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2csFXCVylhZR0AbN66JPUvZKOpg= 2024-04-12T04:11:07.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2csFXCVylhZR0AbN66JPUvZKOpg=', 'Content-Length': '0'} 2024-04-12T04:11:07.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b17bf8513dea3f4d-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.960 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:11:07.960 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:11:07.961 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.961 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.961 DEBUG:boto:Method: GET 2024-04-12T04:11:07.961 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.961 DEBUG:boto:Data: 2024-04-12T04:11:07.961 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.961 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.961 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.961 DEBUG:boto:Params: {} 2024-04-12T04:11:07.961 DEBUG:boto:Token: None 2024-04-12T04:11:07.961 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.962 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2csFXCVylhZR0AbN66JPUvZKOpg= 2024-04-12T04:11:07.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2csFXCVylhZR0AbN66JPUvZKOpg=', 'Content-Length': '0'} 2024-04-12T04:11:07.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000518b929d7e605444-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.964 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:11:07.964 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:11:07.964 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.964 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.965 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.965 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.965 DEBUG:boto:Data: 2024-04-12T04:11:07.965 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.965 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.965 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.965 DEBUG:boto:Params: {} 2024-04-12T04:11:07.965 DEBUG:boto:Token: None 2024-04-12T04:11:07.965 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.965 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cNa1TLYBHfVHJ+LrmsKQ2DUsW8U= 2024-04-12T04:11:07.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cNa1TLYBHfVHJ+LrmsKQ2DUsW8U=', 'Content-Length': '0'} 2024-04-12T04:11:07.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fabe8912fb3c2bd3-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.967 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:11:07.967 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:11:07.968 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.968 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.968 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.968 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.968 DEBUG:boto:Data: 2024-04-12T04:11:07.968 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.968 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.968 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.968 DEBUG:boto:Params: {} 2024-04-12T04:11:07.968 DEBUG:boto:Token: None 2024-04-12T04:11:07.968 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.969 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cNa1TLYBHfVHJ+LrmsKQ2DUsW8U= 2024-04-12T04:11:07.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cNa1TLYBHfVHJ+LrmsKQ2DUsW8U=', 'Content-Length': '0'} 2024-04-12T04:11:07.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dee51faaad731d8d-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:11:07.971 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:11:07.971 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:11:07.971 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.971 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.971 DEBUG:boto:Method: GET 2024-04-12T04:11:07.971 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.971 DEBUG:boto:Data: 2024-04-12T04:11:07.971 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.971 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.971 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.972 DEBUG:boto:Params: {} 2024-04-12T04:11:07.972 DEBUG:boto:Token: None 2024-04-12T04:11:07.972 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.972 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2csFXCVylhZR0AbN66JPUvZKOpg= 2024-04-12T04:11:07.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2csFXCVylhZR0AbN66JPUvZKOpg=', 'Content-Length': '0'} 2024-04-12T04:11:07.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000369572dc40fac486-006618b45b-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.974 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:11:07.974 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:11:07.974 DEBUG:boto:path=/dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.974 DEBUG:boto:auth_path=/dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.974 DEBUG:boto:Method: GET 2024-04-12T04:11:07.974 DEBUG:boto:Path: /dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.974 DEBUG:boto:Data: 2024-04-12T04:11:07.975 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.975 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.975 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.975 DEBUG:boto:Params: {} 2024-04-12T04:11:07.975 DEBUG:boto:Token: None 2024-04-12T04:11:07.975 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj68?versionId=null 2024-04-12T04:11:07.975 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2csFXCVylhZR0AbN66JPUvZKOpg= 2024-04-12T04:11:07.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2csFXCVylhZR0AbN66JPUvZKOpg=', 'Content-Length': '0'} 2024-04-12T04:11:07.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044a91c56e263c186-006618b45b-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.977 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.977 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.977 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:11:07.977 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:11:07.978 DEBUG:boto:Method: GET 2024-04-12T04:11:07.978 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:11:07.978 DEBUG:boto:Data: 2024-04-12T04:11:07.978 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.978 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.978 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.978 DEBUG:boto:Params: {} 2024-04-12T04:11:07.978 DEBUG:boto:Token: None 2024-04-12T04:11:07.978 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.979 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e31014d75d53c630-006618b45b-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:07 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.982 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.982 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:07.982 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:07.982 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:11:07.982 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:11:07.982 DEBUG:boto:Method: GET 2024-04-12T04:11:07.982 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:11:07.982 DEBUG:boto:Data: 2024-04-12T04:11:07.982 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.983 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.983 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.983 DEBUG:boto:Params: {} 2024-04-12T04:11:07.983 DEBUG:boto:Token: None 2024-04-12T04:11:07.983 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/ 2024-04-12T04:11:07.983 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk= 2024-04-12T04:11:07.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycjv3IBNIWgTIbS7zTZYXMHirGk=', 'Content-Length': '0'} 2024-04-12T04:11:07.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd06c10058ba7ed9-006618b45c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.987 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:07.988 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:11:07.988 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:11:07.988 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.988 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:11:07.988 DEBUG:boto:Data: 2024-04-12T04:11:07.988 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.988 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.988 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.988 DEBUG:boto:Params: {} 2024-04-12T04:11:07.988 DEBUG:boto:Token: None 2024-04-12T04:11:07.989 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj68 2024-04-12T04:11:07.989 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uJ1gqo38byhwZWX3fnOTz+LS28w= 2024-04-12T04:11:07.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uJ1gqo38byhwZWX3fnOTz+LS28w=', 'Content-Length': '0'} 2024-04-12T04:11:07.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b97f923fe6d376c-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.991 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:11:07.991 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:11:07.991 DEBUG:boto:Method: HEAD 2024-04-12T04:11:07.991 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:11:07.991 DEBUG:boto:Data: 2024-04-12T04:11:07.991 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.991 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:07.991 DEBUG:boto:Port: 8000 2024-04-12T04:11:07.991 DEBUG:boto:Params: {} 2024-04-12T04:11:07.992 DEBUG:boto:Token: None 2024-04-12T04:11:07.992 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj68 2024-04-12T04:11:07.992 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uJ1gqo38byhwZWX3fnOTz+LS28w= 2024-04-12T04:11:07.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uJ1gqo38byhwZWX3fnOTz+LS28w=', 'Content-Length': '0'} 2024-04-12T04:11:07.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fb456c0f2884625-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:07.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:11:07.994 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:11:07.994 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:11:07.994 DEBUG:boto:Method: GET 2024-04-12T04:11:07.994 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:11:07.995 DEBUG:boto:Data: 2024-04-12T04:11:07.995 DEBUG:boto:Headers: {} 2024-04-12T04:11:07.995 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:07.995 DEBUG:boto:Port: 8001 2024-04-12T04:11:07.995 DEBUG:boto:Params: {} 2024-04-12T04:11:07.995 DEBUG:boto:Token: None 2024-04-12T04:11:07.995 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:07 GMT /dkyges-61/obj68 2024-04-12T04:11:07.995 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3E2plTZR9eFTY5v4eXDM06IuS/M= 2024-04-12T04:11:07.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:07 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3E2plTZR9eFTY5v4eXDM06IuS/M=', 'Content-Length': '0'} 2024-04-12T04:11:07.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053c53e5825ec69f5-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.042 DEBUG:boto:path=/dkyges-61/obj68 2024-04-12T04:11:08.042 DEBUG:boto:auth_path=/dkyges-61/obj68 2024-04-12T04:11:08.043 DEBUG:boto:Method: GET 2024-04-12T04:11:08.043 DEBUG:boto:Path: /dkyges-61/obj68 2024-04-12T04:11:08.043 DEBUG:boto:Data: 2024-04-12T04:11:08.043 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.043 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.043 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.043 DEBUG:boto:Params: {} 2024-04-12T04:11:08.043 DEBUG:boto:Token: None 2024-04-12T04:11:08.043 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj68 2024-04-12T04:11:08.044 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jdt9WMWQ3/pjxwdkvzwt4yNUwvU= 2024-04-12T04:11:08.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jdt9WMWQ3/pjxwdkvzwt4yNUwvU=', 'Content-Length': '0'} 2024-04-12T04:11:08.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052f2fa4c5052db4b-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.046 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.046 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.046 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:11:08.046 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:11:08.046 DEBUG:boto:Method: GET 2024-04-12T04:11:08.046 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:11:08.046 DEBUG:boto:Data: 2024-04-12T04:11:08.046 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.047 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.047 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.047 DEBUG:boto:Params: {} 2024-04-12T04:11:08.047 DEBUG:boto:Token: None 2024-04-12T04:11:08.047 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.047 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b25006507e792c0-006618b45c-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.050 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.050 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.050 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.050 DEBUG:boto:path=/dkyges-61/?prefix=obj68 2024-04-12T04:11:08.050 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj68 2024-04-12T04:11:08.050 DEBUG:boto:Method: GET 2024-04-12T04:11:08.051 DEBUG:boto:Path: /dkyges-61/?prefix=obj68 2024-04-12T04:11:08.051 DEBUG:boto:Data: 2024-04-12T04:11:08.051 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.051 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.051 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.051 DEBUG:boto:Params: {} 2024-04-12T04:11:08.051 DEBUG:boto:Token: None 2024-04-12T04:11:08.051 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.051 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4e60b242235fedc-006618b45c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.056 DEBUG:boto:b'dkyges-61obj681000falseobj682024-04-12T04:00:41.748Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:11:08.057 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:11:08.057 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:11:08.057 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.057 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.057 DEBUG:boto:Method: GET 2024-04-12T04:11:08.057 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.057 DEBUG:boto:Data: 2024-04-12T04:11:08.057 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.057 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.057 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.057 DEBUG:boto:Params: {} 2024-04-12T04:11:08.057 DEBUG:boto:Token: None 2024-04-12T04:11:08.058 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.058 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mDQAlSQBhfoNL8qaMpaZVc2p8tA= 2024-04-12T04:11:08.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mDQAlSQBhfoNL8qaMpaZVc2p8tA=', 'Content-Length': '0'} 2024-04-12T04:11:08.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d18bf145aeedb604-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.060 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:11:08.060 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:11:08.060 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.061 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.061 DEBUG:boto:Method: GET 2024-04-12T04:11:08.061 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.061 DEBUG:boto:Data: 2024-04-12T04:11:08.061 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.061 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.061 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.061 DEBUG:boto:Params: {} 2024-04-12T04:11:08.061 DEBUG:boto:Token: None 2024-04-12T04:11:08.061 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.062 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mDQAlSQBhfoNL8qaMpaZVc2p8tA= 2024-04-12T04:11:08.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mDQAlSQBhfoNL8qaMpaZVc2p8tA=', 'Content-Length': '0'} 2024-04-12T04:11:08.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008430bb72a60419bb-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.064 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:11:08.064 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:11:08.064 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.064 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.064 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.064 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.064 DEBUG:boto:Data: 2024-04-12T04:11:08.065 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.065 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.065 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.065 DEBUG:boto:Params: {} 2024-04-12T04:11:08.065 DEBUG:boto:Token: None 2024-04-12T04:11:08.065 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.065 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5qMZsdU/JSQ10NzjIAc1sXRXU/o= 2024-04-12T04:11:08.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5qMZsdU/JSQ10NzjIAc1sXRXU/o=', 'Content-Length': '0'} 2024-04-12T04:11:08.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005878cc7e63913969-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.067 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:11:08.067 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:11:08.067 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.067 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.068 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.068 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.068 DEBUG:boto:Data: 2024-04-12T04:11:08.068 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.068 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.068 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.068 DEBUG:boto:Params: {} 2024-04-12T04:11:08.068 DEBUG:boto:Token: None 2024-04-12T04:11:08.068 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.068 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5qMZsdU/JSQ10NzjIAc1sXRXU/o= 2024-04-12T04:11:08.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5qMZsdU/JSQ10NzjIAc1sXRXU/o=', 'Content-Length': '0'} 2024-04-12T04:11:08.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000173c9e42bc8261da-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:11:08.071 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:11:08.071 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:11:08.071 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.071 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.071 DEBUG:boto:Method: GET 2024-04-12T04:11:08.071 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.071 DEBUG:boto:Data: 2024-04-12T04:11:08.071 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.071 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.072 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.072 DEBUG:boto:Params: {} 2024-04-12T04:11:08.072 DEBUG:boto:Token: None 2024-04-12T04:11:08.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.072 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mDQAlSQBhfoNL8qaMpaZVc2p8tA= 2024-04-12T04:11:08.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mDQAlSQBhfoNL8qaMpaZVc2p8tA=', 'Content-Length': '0'} 2024-04-12T04:11:08.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048fd0207912dbe21-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.074 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:11:08.074 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:11:08.074 DEBUG:boto:path=/dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.074 DEBUG:boto:auth_path=/dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.074 DEBUG:boto:Method: GET 2024-04-12T04:11:08.075 DEBUG:boto:Path: /dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.075 DEBUG:boto:Data: 2024-04-12T04:11:08.075 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.075 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.075 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.075 DEBUG:boto:Params: {} 2024-04-12T04:11:08.075 DEBUG:boto:Token: None 2024-04-12T04:11:08.075 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj69?versionId=null 2024-04-12T04:11:08.075 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mDQAlSQBhfoNL8qaMpaZVc2p8tA= 2024-04-12T04:11:08.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mDQAlSQBhfoNL8qaMpaZVc2p8tA=', 'Content-Length': '0'} 2024-04-12T04:11:08.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036ed1090300176e8-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.077 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.078 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.078 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:11:08.078 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:11:08.078 DEBUG:boto:Method: GET 2024-04-12T04:11:08.078 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:11:08.078 DEBUG:boto:Data: 2024-04-12T04:11:08.078 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.078 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.078 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.079 DEBUG:boto:Params: {} 2024-04-12T04:11:08.079 DEBUG:boto:Token: None 2024-04-12T04:11:08.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.079 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5923eac125b8d46-006618b45c-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.082 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.082 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.082 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.082 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:11:08.082 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:11:08.082 DEBUG:boto:Method: GET 2024-04-12T04:11:08.083 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:11:08.083 DEBUG:boto:Data: 2024-04-12T04:11:08.083 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.083 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.083 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.083 DEBUG:boto:Params: {} 2024-04-12T04:11:08.083 DEBUG:boto:Token: None 2024-04-12T04:11:08.083 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.083 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005310d44e1f82a458-006618b45c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.088 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.089 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:11:08.089 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:11:08.089 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.089 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:11:08.089 DEBUG:boto:Data: 2024-04-12T04:11:08.089 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.089 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.089 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.089 DEBUG:boto:Params: {} 2024-04-12T04:11:08.090 DEBUG:boto:Token: None 2024-04-12T04:11:08.090 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj69 2024-04-12T04:11:08.090 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zy2ha4bWEtuIdmocM27mVjlJDrY= 2024-04-12T04:11:08.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zy2ha4bWEtuIdmocM27mVjlJDrY=', 'Content-Length': '0'} 2024-04-12T04:11:08.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053f9885576e685f2-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.092 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:11:08.092 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:11:08.092 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.092 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:11:08.092 DEBUG:boto:Data: 2024-04-12T04:11:08.093 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.093 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.093 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.093 DEBUG:boto:Params: {} 2024-04-12T04:11:08.093 DEBUG:boto:Token: None 2024-04-12T04:11:08.093 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj69 2024-04-12T04:11:08.093 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zy2ha4bWEtuIdmocM27mVjlJDrY= 2024-04-12T04:11:08.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zy2ha4bWEtuIdmocM27mVjlJDrY=', 'Content-Length': '0'} 2024-04-12T04:11:08.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b07885ccaf67643-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:11:08.095 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:11:08.096 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:11:08.096 DEBUG:boto:Method: GET 2024-04-12T04:11:08.096 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:11:08.096 DEBUG:boto:Data: 2024-04-12T04:11:08.096 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.096 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.096 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.096 DEBUG:boto:Params: {} 2024-04-12T04:11:08.096 DEBUG:boto:Token: None 2024-04-12T04:11:08.096 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj69 2024-04-12T04:11:08.096 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n5jl60HaxNh0yKQ7fMtY11n9njk= 2024-04-12T04:11:08.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n5jl60HaxNh0yKQ7fMtY11n9njk=', 'Content-Length': '0'} 2024-04-12T04:11:08.098 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000849be88a1d1371b-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.099 DEBUG:boto:path=/dkyges-61/obj69 2024-04-12T04:11:08.099 DEBUG:boto:auth_path=/dkyges-61/obj69 2024-04-12T04:11:08.099 DEBUG:boto:Method: GET 2024-04-12T04:11:08.099 DEBUG:boto:Path: /dkyges-61/obj69 2024-04-12T04:11:08.099 DEBUG:boto:Data: 2024-04-12T04:11:08.099 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.099 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.099 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.099 DEBUG:boto:Params: {} 2024-04-12T04:11:08.099 DEBUG:boto:Token: None 2024-04-12T04:11:08.100 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj69 2024-04-12T04:11:08.100 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n5jl60HaxNh0yKQ7fMtY11n9njk= 2024-04-12T04:11:08.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n5jl60HaxNh0yKQ7fMtY11n9njk=', 'Content-Length': '0'} 2024-04-12T04:11:08.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8adc4ac4de2f094-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.142 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.142 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.143 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:11:08.143 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:11:08.143 DEBUG:boto:Method: GET 2024-04-12T04:11:08.143 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:11:08.143 DEBUG:boto:Data: 2024-04-12T04:11:08.143 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.143 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.143 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.143 DEBUG:boto:Params: {} 2024-04-12T04:11:08.143 DEBUG:boto:Token: None 2024-04-12T04:11:08.143 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.144 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087d171376f7c7766-006618b45c-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.146 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.147 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.147 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.147 DEBUG:boto:path=/dkyges-61/?prefix=obj69 2024-04-12T04:11:08.147 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj69 2024-04-12T04:11:08.147 DEBUG:boto:Method: GET 2024-04-12T04:11:08.147 DEBUG:boto:Path: /dkyges-61/?prefix=obj69 2024-04-12T04:11:08.147 DEBUG:boto:Data: 2024-04-12T04:11:08.147 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.148 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.148 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.148 DEBUG:boto:Params: {} 2024-04-12T04:11:08.148 DEBUG:boto:Token: None 2024-04-12T04:11:08.148 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.148 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6e426a67dbd1801-006618b45c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.152 DEBUG:boto:b'dkyges-61obj691000falseobj692024-04-12T04:05:01.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T04:11:08.153 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:11:08.153 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:11:08.153 DEBUG:boto:path=/dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.154 DEBUG:boto:auth_path=/dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.154 DEBUG:boto:Method: GET 2024-04-12T04:11:08.154 DEBUG:boto:Path: /dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.154 DEBUG:boto:Data: 2024-04-12T04:11:08.154 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.154 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.154 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.154 DEBUG:boto:Params: {} 2024-04-12T04:11:08.154 DEBUG:boto:Token: None 2024-04-12T04:11:08.154 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.155 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lG+6SlNXjD867v9+gSVspM9P+ic= 2024-04-12T04:11:08.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lG+6SlNXjD867v9+gSVspM9P+ic=', 'Content-Length': '0'} 2024-04-12T04:11:08.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cde438694cdf949-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.202 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:11:08.202 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:11:08.203 DEBUG:boto:path=/dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.203 DEBUG:boto:auth_path=/dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.203 DEBUG:boto:Method: GET 2024-04-12T04:11:08.203 DEBUG:boto:Path: /dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.203 DEBUG:boto:Data: 2024-04-12T04:11:08.203 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.203 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.203 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.203 DEBUG:boto:Params: {} 2024-04-12T04:11:08.203 DEBUG:boto:Token: None 2024-04-12T04:11:08.203 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.204 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lG+6SlNXjD867v9+gSVspM9P+ic= 2024-04-12T04:11:08.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lG+6SlNXjD867v9+gSVspM9P+ic=', 'Content-Length': '0'} 2024-04-12T04:11:08.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002b23b987c0487ff-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.206 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:11:08.206 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:11:08.206 DEBUG:boto:path=/dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.206 DEBUG:boto:auth_path=/dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.206 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.206 DEBUG:boto:Path: /dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.206 DEBUG:boto:Data: 2024-04-12T04:11:08.207 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.207 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.207 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.207 DEBUG:boto:Params: {} 2024-04-12T04:11:08.207 DEBUG:boto:Token: None 2024-04-12T04:11:08.207 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.207 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hqyw2WE4HOo0JnqcpwhOPX4TFow= 2024-04-12T04:11:08.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hqyw2WE4HOo0JnqcpwhOPX4TFow=', 'Content-Length': '0'} 2024-04-12T04:11:08.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000de2700e62cc1f95-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.209 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:11:08.209 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:11:08.210 DEBUG:boto:path=/dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.210 DEBUG:boto:auth_path=/dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.210 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.210 DEBUG:boto:Path: /dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.210 DEBUG:boto:Data: 2024-04-12T04:11:08.210 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.210 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.210 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.210 DEBUG:boto:Params: {} 2024-04-12T04:11:08.210 DEBUG:boto:Token: None 2024-04-12T04:11:08.211 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.211 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hqyw2WE4HOo0JnqcpwhOPX4TFow= 2024-04-12T04:11:08.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hqyw2WE4HOo0JnqcpwhOPX4TFow=', 'Content-Length': '0'} 2024-04-12T04:11:08.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4f4037fcc569bdd-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T04:11:08.213 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:11:08.213 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:11:08.213 DEBUG:boto:path=/dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.213 DEBUG:boto:auth_path=/dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.213 DEBUG:boto:Method: GET 2024-04-12T04:11:08.213 DEBUG:boto:Path: /dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.213 DEBUG:boto:Data: 2024-04-12T04:11:08.213 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.213 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.213 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.214 DEBUG:boto:Params: {} 2024-04-12T04:11:08.214 DEBUG:boto:Token: None 2024-04-12T04:11:08.214 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.214 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lG+6SlNXjD867v9+gSVspM9P+ic= 2024-04-12T04:11:08.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lG+6SlNXjD867v9+gSVspM9P+ic=', 'Content-Length': '0'} 2024-04-12T04:11:08.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4cea750235da77c-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.216 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:11:08.216 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:11:08.216 DEBUG:boto:path=/dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.216 DEBUG:boto:auth_path=/dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.216 DEBUG:boto:Method: GET 2024-04-12T04:11:08.216 DEBUG:boto:Path: /dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.217 DEBUG:boto:Data: 2024-04-12T04:11:08.217 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.217 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.217 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.217 DEBUG:boto:Params: {} 2024-04-12T04:11:08.217 DEBUG:boto:Token: None 2024-04-12T04:11:08.217 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj74?versionId=null 2024-04-12T04:11:08.217 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lG+6SlNXjD867v9+gSVspM9P+ic= 2024-04-12T04:11:08.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lG+6SlNXjD867v9+gSVspM9P+ic=', 'Content-Length': '0'} 2024-04-12T04:11:08.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008823788f80f49f2a-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.219 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.220 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.220 DEBUG:boto:path=/dkyges-61/?prefix=obj74 2024-04-12T04:11:08.220 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj74 2024-04-12T04:11:08.220 DEBUG:boto:Method: GET 2024-04-12T04:11:08.220 DEBUG:boto:Path: /dkyges-61/?prefix=obj74 2024-04-12T04:11:08.220 DEBUG:boto:Data: 2024-04-12T04:11:08.220 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.220 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.220 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.220 DEBUG:boto:Params: {} 2024-04-12T04:11:08.221 DEBUG:boto:Token: None 2024-04-12T04:11:08.221 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.221 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f7c100fd615990a-006618b45c-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.223 DEBUG:boto:b'dkyges-61obj741000falseobj742024-04-12T04:09:24.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.224 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.224 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.224 DEBUG:boto:path=/dkyges-61/?prefix=obj74 2024-04-12T04:11:08.224 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj74 2024-04-12T04:11:08.224 DEBUG:boto:Method: GET 2024-04-12T04:11:08.224 DEBUG:boto:Path: /dkyges-61/?prefix=obj74 2024-04-12T04:11:08.224 DEBUG:boto:Data: 2024-04-12T04:11:08.224 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.224 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.224 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.225 DEBUG:boto:Params: {} 2024-04-12T04:11:08.225 DEBUG:boto:Token: None 2024-04-12T04:11:08.225 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.225 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035fbdec77dcf36d6-006618b45c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.230 DEBUG:boto:b'dkyges-61obj741000falseobj742024-04-12T04:09:24.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.230 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:11:08.230 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:11:08.230 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.230 DEBUG:boto:Path: /dkyges-61/obj74 2024-04-12T04:11:08.230 DEBUG:boto:Data: 2024-04-12T04:11:08.230 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.231 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.231 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.231 DEBUG:boto:Params: {} 2024-04-12T04:11:08.231 DEBUG:boto:Token: None 2024-04-12T04:11:08.231 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj74 2024-04-12T04:11:08.231 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rMdIMYN9coZeEBE3/jJfbxteoyw= 2024-04-12T04:11:08.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rMdIMYN9coZeEBE3/jJfbxteoyw=', 'Content-Length': '0'} 2024-04-12T04:11:08.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2f81998b0d33458-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.233 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:11:08.233 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:11:08.233 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.233 DEBUG:boto:Path: /dkyges-61/obj74 2024-04-12T04:11:08.234 DEBUG:boto:Data: 2024-04-12T04:11:08.234 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.234 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.234 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.234 DEBUG:boto:Params: {} 2024-04-12T04:11:08.234 DEBUG:boto:Token: None 2024-04-12T04:11:08.234 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj74 2024-04-12T04:11:08.234 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rMdIMYN9coZeEBE3/jJfbxteoyw= 2024-04-12T04:11:08.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rMdIMYN9coZeEBE3/jJfbxteoyw=', 'Content-Length': '0'} 2024-04-12T04:11:08.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c42701e229408b04-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T04:11:08.237 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:11:08.237 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:11:08.237 DEBUG:boto:Method: GET 2024-04-12T04:11:08.237 DEBUG:boto:Path: /dkyges-61/obj74 2024-04-12T04:11:08.237 DEBUG:boto:Data: 2024-04-12T04:11:08.237 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.237 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.237 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.237 DEBUG:boto:Params: {} 2024-04-12T04:11:08.238 DEBUG:boto:Token: None 2024-04-12T04:11:08.238 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj74 2024-04-12T04:11:08.238 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Kx9AEeELsvQGjqOx5I6of4P4koA= 2024-04-12T04:11:08.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Kx9AEeELsvQGjqOx5I6of4P4koA=', 'Content-Length': '0'} 2024-04-12T04:11:08.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000835f3af734b5c2b1-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.240 DEBUG:boto:path=/dkyges-61/obj74 2024-04-12T04:11:08.240 DEBUG:boto:auth_path=/dkyges-61/obj74 2024-04-12T04:11:08.240 DEBUG:boto:Method: GET 2024-04-12T04:11:08.240 DEBUG:boto:Path: /dkyges-61/obj74 2024-04-12T04:11:08.240 DEBUG:boto:Data: 2024-04-12T04:11:08.240 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.240 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.241 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.241 DEBUG:boto:Params: {} 2024-04-12T04:11:08.241 DEBUG:boto:Token: None 2024-04-12T04:11:08.241 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj74 2024-04-12T04:11:08.241 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Kx9AEeELsvQGjqOx5I6of4P4koA= 2024-04-12T04:11:08.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Kx9AEeELsvQGjqOx5I6of4P4koA=', 'Content-Length': '0'} 2024-04-12T04:11:08.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:09:24 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf2c53a1719ea277-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.286 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.286 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.287 DEBUG:boto:path=/dkyges-61/?prefix=obj74 2024-04-12T04:11:08.287 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj74 2024-04-12T04:11:08.287 DEBUG:boto:Method: GET 2024-04-12T04:11:08.287 DEBUG:boto:Path: /dkyges-61/?prefix=obj74 2024-04-12T04:11:08.287 DEBUG:boto:Data: 2024-04-12T04:11:08.287 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.287 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.287 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.287 DEBUG:boto:Params: {} 2024-04-12T04:11:08.287 DEBUG:boto:Token: None 2024-04-12T04:11:08.288 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.288 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbed4d8e414c2b3d-006618b45c-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.290 DEBUG:boto:b'dkyges-61obj741000falseobj742024-04-12T04:09:24.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.291 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.291 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.291 DEBUG:boto:path=/dkyges-61/?prefix=obj74 2024-04-12T04:11:08.291 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj74 2024-04-12T04:11:08.291 DEBUG:boto:Method: GET 2024-04-12T04:11:08.291 DEBUG:boto:Path: /dkyges-61/?prefix=obj74 2024-04-12T04:11:08.291 DEBUG:boto:Data: 2024-04-12T04:11:08.291 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.292 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.292 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.292 DEBUG:boto:Params: {} 2024-04-12T04:11:08.292 DEBUG:boto:Token: None 2024-04-12T04:11:08.292 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.292 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015f1db969b47d529-006618b45c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.297 DEBUG:boto:b'dkyges-61obj741000falseobj742024-04-12T04:09:24.047Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:11:08.298 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:11:08.298 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:11:08.298 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.298 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.298 DEBUG:boto:Method: GET 2024-04-12T04:11:08.298 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.298 DEBUG:boto:Data: 2024-04-12T04:11:08.298 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.298 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.298 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.298 DEBUG:boto:Params: {} 2024-04-12T04:11:08.299 DEBUG:boto:Token: None 2024-04-12T04:11:08.299 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.299 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X0mF47VoOrQImXL/gFXzyPg3cZI= 2024-04-12T04:11:08.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X0mF47VoOrQImXL/gFXzyPg3cZI=', 'Content-Length': '0'} 2024-04-12T04:11:08.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000369740ad2dd235f2-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.343 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:11:08.343 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:11:08.343 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.343 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.343 DEBUG:boto:Method: GET 2024-04-12T04:11:08.343 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.343 DEBUG:boto:Data: 2024-04-12T04:11:08.343 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.343 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.344 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.344 DEBUG:boto:Params: {} 2024-04-12T04:11:08.344 DEBUG:boto:Token: None 2024-04-12T04:11:08.344 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.344 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X0mF47VoOrQImXL/gFXzyPg3cZI= 2024-04-12T04:11:08.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X0mF47VoOrQImXL/gFXzyPg3cZI=', 'Content-Length': '0'} 2024-04-12T04:11:08.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000195780e89a429859-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.346 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:11:08.346 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:11:08.347 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.347 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.347 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.347 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.347 DEBUG:boto:Data: 2024-04-12T04:11:08.347 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.347 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.347 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.347 DEBUG:boto:Params: {} 2024-04-12T04:11:08.347 DEBUG:boto:Token: None 2024-04-12T04:11:08.348 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.348 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Nx7pochyEBRKqrXurf5wR1G5SXw= 2024-04-12T04:11:08.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Nx7pochyEBRKqrXurf5wR1G5SXw=', 'Content-Length': '0'} 2024-04-12T04:11:08.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf3d87d3444d4efd-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.350 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:11:08.350 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:11:08.350 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.350 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.350 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.350 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.350 DEBUG:boto:Data: 2024-04-12T04:11:08.351 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.351 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.351 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.351 DEBUG:boto:Params: {} 2024-04-12T04:11:08.351 DEBUG:boto:Token: None 2024-04-12T04:11:08.351 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.351 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Nx7pochyEBRKqrXurf5wR1G5SXw= 2024-04-12T04:11:08.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Nx7pochyEBRKqrXurf5wR1G5SXw=', 'Content-Length': '0'} 2024-04-12T04:11:08.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001436d24e65774c93-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:11:08.353 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:11:08.353 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:11:08.353 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.353 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.354 DEBUG:boto:Method: GET 2024-04-12T04:11:08.354 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.354 DEBUG:boto:Data: 2024-04-12T04:11:08.354 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.354 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.354 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.354 DEBUG:boto:Params: {} 2024-04-12T04:11:08.354 DEBUG:boto:Token: None 2024-04-12T04:11:08.354 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.355 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X0mF47VoOrQImXL/gFXzyPg3cZI= 2024-04-12T04:11:08.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X0mF47VoOrQImXL/gFXzyPg3cZI=', 'Content-Length': '0'} 2024-04-12T04:11:08.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c732fac8ab147445-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.357 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:11:08.357 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:11:08.357 DEBUG:boto:path=/dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.357 DEBUG:boto:auth_path=/dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.357 DEBUG:boto:Method: GET 2024-04-12T04:11:08.357 DEBUG:boto:Path: /dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.357 DEBUG:boto:Data: 2024-04-12T04:11:08.357 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.358 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.358 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.358 DEBUG:boto:Params: {} 2024-04-12T04:11:08.358 DEBUG:boto:Token: None 2024-04-12T04:11:08.358 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj76?versionId=null 2024-04-12T04:11:08.358 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X0mF47VoOrQImXL/gFXzyPg3cZI= 2024-04-12T04:11:08.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X0mF47VoOrQImXL/gFXzyPg3cZI=', 'Content-Length': '0'} 2024-04-12T04:11:08.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e291808e72db8d94-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.361 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.361 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.361 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:11:08.361 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:11:08.361 DEBUG:boto:Method: GET 2024-04-12T04:11:08.361 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:11:08.361 DEBUG:boto:Data: 2024-04-12T04:11:08.361 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.361 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.361 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.361 DEBUG:boto:Params: {} 2024-04-12T04:11:08.362 DEBUG:boto:Token: None 2024-04-12T04:11:08.362 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.362 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2e6285add31b488-006618b45c-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.365 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.365 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.365 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.365 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:11:08.365 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:11:08.365 DEBUG:boto:Method: GET 2024-04-12T04:11:08.366 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:11:08.366 DEBUG:boto:Data: 2024-04-12T04:11:08.366 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.366 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.366 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.366 DEBUG:boto:Params: {} 2024-04-12T04:11:08.366 DEBUG:boto:Token: None 2024-04-12T04:11:08.366 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.366 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002cd605d67adb667-006618b45c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.371 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.372 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:11:08.372 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:11:08.372 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.372 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:11:08.372 DEBUG:boto:Data: 2024-04-12T04:11:08.372 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.372 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.372 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.372 DEBUG:boto:Params: {} 2024-04-12T04:11:08.372 DEBUG:boto:Token: None 2024-04-12T04:11:08.373 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj76 2024-04-12T04:11:08.373 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vCqfgej5sAwTDRR/GyyTt79VnSc= 2024-04-12T04:11:08.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vCqfgej5sAwTDRR/GyyTt79VnSc=', 'Content-Length': '0'} 2024-04-12T04:11:08.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e104eb008fc779a-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.375 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:11:08.375 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:11:08.375 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.375 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:11:08.375 DEBUG:boto:Data: 2024-04-12T04:11:08.375 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.375 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.375 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.376 DEBUG:boto:Params: {} 2024-04-12T04:11:08.376 DEBUG:boto:Token: None 2024-04-12T04:11:08.376 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj76 2024-04-12T04:11:08.376 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vCqfgej5sAwTDRR/GyyTt79VnSc= 2024-04-12T04:11:08.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vCqfgej5sAwTDRR/GyyTt79VnSc=', 'Content-Length': '0'} 2024-04-12T04:11:08.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec3541cb8813ebbf-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:11:08.378 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:11:08.378 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:11:08.378 DEBUG:boto:Method: GET 2024-04-12T04:11:08.378 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:11:08.379 DEBUG:boto:Data: 2024-04-12T04:11:08.379 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.379 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.379 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.379 DEBUG:boto:Params: {} 2024-04-12T04:11:08.379 DEBUG:boto:Token: None 2024-04-12T04:11:08.379 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj76 2024-04-12T04:11:08.379 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1MtGRDvSgbBnPKeJw4MFH6ktdeo= 2024-04-12T04:11:08.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1MtGRDvSgbBnPKeJw4MFH6ktdeo=', 'Content-Length': '0'} 2024-04-12T04:11:08.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edfb7631f82f86b2-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.381 DEBUG:boto:path=/dkyges-61/obj76 2024-04-12T04:11:08.382 DEBUG:boto:auth_path=/dkyges-61/obj76 2024-04-12T04:11:08.382 DEBUG:boto:Method: GET 2024-04-12T04:11:08.382 DEBUG:boto:Path: /dkyges-61/obj76 2024-04-12T04:11:08.382 DEBUG:boto:Data: 2024-04-12T04:11:08.382 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.382 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.382 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.382 DEBUG:boto:Params: {} 2024-04-12T04:11:08.383 DEBUG:boto:Token: None 2024-04-12T04:11:08.383 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj76 2024-04-12T04:11:08.383 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1MtGRDvSgbBnPKeJw4MFH6ktdeo= 2024-04-12T04:11:08.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1MtGRDvSgbBnPKeJw4MFH6ktdeo=', 'Content-Length': '0'} 2024-04-12T04:11:08.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a278c89ec595cd28-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.426 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.426 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.427 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:11:08.427 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:11:08.427 DEBUG:boto:Method: GET 2024-04-12T04:11:08.427 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:11:08.427 DEBUG:boto:Data: 2024-04-12T04:11:08.427 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.427 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.427 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.427 DEBUG:boto:Params: {} 2024-04-12T04:11:08.428 DEBUG:boto:Token: None 2024-04-12T04:11:08.428 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.428 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eec5086758452b78-006618b45c-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.430 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.431 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.431 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.431 DEBUG:boto:path=/dkyges-61/?prefix=obj76 2024-04-12T04:11:08.431 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj76 2024-04-12T04:11:08.431 DEBUG:boto:Method: GET 2024-04-12T04:11:08.431 DEBUG:boto:Path: /dkyges-61/?prefix=obj76 2024-04-12T04:11:08.432 DEBUG:boto:Data: 2024-04-12T04:11:08.432 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.432 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.432 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.432 DEBUG:boto:Params: {} 2024-04-12T04:11:08.432 DEBUG:boto:Token: None 2024-04-12T04:11:08.432 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.432 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097497fe257b245b2-006618b45c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.437 DEBUG:boto:b'dkyges-61obj761000falseobj762024-04-12T04:03:03.689Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:11:08.437 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:11:08.437 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:11:08.437 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.438 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.438 DEBUG:boto:Method: GET 2024-04-12T04:11:08.438 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.438 DEBUG:boto:Data: 2024-04-12T04:11:08.438 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.438 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.438 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.438 DEBUG:boto:Params: {} 2024-04-12T04:11:08.439 DEBUG:boto:Token: None 2024-04-12T04:11:08.439 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.439 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:M5ii4nIYdiOM9k8HTapEf1qCmGY= 2024-04-12T04:11:08.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:M5ii4nIYdiOM9k8HTapEf1qCmGY=', 'Content-Length': '0'} 2024-04-12T04:11:08.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000327e6b0549503c2-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.482 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:11:08.482 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:11:08.483 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.483 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.483 DEBUG:boto:Method: GET 2024-04-12T04:11:08.483 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.483 DEBUG:boto:Data: 2024-04-12T04:11:08.483 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.483 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.483 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.483 DEBUG:boto:Params: {} 2024-04-12T04:11:08.484 DEBUG:boto:Token: None 2024-04-12T04:11:08.484 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.484 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:M5ii4nIYdiOM9k8HTapEf1qCmGY= 2024-04-12T04:11:08.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:M5ii4nIYdiOM9k8HTapEf1qCmGY=', 'Content-Length': '0'} 2024-04-12T04:11:08.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3997b0174521858-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.486 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:11:08.487 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:11:08.487 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.487 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.487 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.487 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.487 DEBUG:boto:Data: 2024-04-12T04:11:08.487 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.487 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.487 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.487 DEBUG:boto:Params: {} 2024-04-12T04:11:08.488 DEBUG:boto:Token: None 2024-04-12T04:11:08.488 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.488 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9V1Cb+7FqgkQdyd2iewvKEr14Lg= 2024-04-12T04:11:08.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9V1Cb+7FqgkQdyd2iewvKEr14Lg=', 'Content-Length': '0'} 2024-04-12T04:11:08.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034dd610110a246cf-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.490 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:11:08.490 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:11:08.490 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.490 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.490 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.491 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.491 DEBUG:boto:Data: 2024-04-12T04:11:08.491 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.491 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.491 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.491 DEBUG:boto:Params: {} 2024-04-12T04:11:08.491 DEBUG:boto:Token: None 2024-04-12T04:11:08.491 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.492 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9V1Cb+7FqgkQdyd2iewvKEr14Lg= 2024-04-12T04:11:08.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9V1Cb+7FqgkQdyd2iewvKEr14Lg=', 'Content-Length': '0'} 2024-04-12T04:11:08.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd270c8728b5c4e1-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:11:08.494 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:11:08.494 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:11:08.494 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.494 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.494 DEBUG:boto:Method: GET 2024-04-12T04:11:08.494 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.495 DEBUG:boto:Data: 2024-04-12T04:11:08.495 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.495 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.495 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.495 DEBUG:boto:Params: {} 2024-04-12T04:11:08.495 DEBUG:boto:Token: None 2024-04-12T04:11:08.495 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.495 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:M5ii4nIYdiOM9k8HTapEf1qCmGY= 2024-04-12T04:11:08.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:M5ii4nIYdiOM9k8HTapEf1qCmGY=', 'Content-Length': '0'} 2024-04-12T04:11:08.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f051c667505075f-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.498 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:11:08.498 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:11:08.498 DEBUG:boto:path=/dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.498 DEBUG:boto:auth_path=/dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.498 DEBUG:boto:Method: GET 2024-04-12T04:11:08.498 DEBUG:boto:Path: /dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.499 DEBUG:boto:Data: 2024-04-12T04:11:08.499 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.499 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.499 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.499 DEBUG:boto:Params: {} 2024-04-12T04:11:08.499 DEBUG:boto:Token: None 2024-04-12T04:11:08.499 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj85?versionId=null 2024-04-12T04:11:08.499 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:M5ii4nIYdiOM9k8HTapEf1qCmGY= 2024-04-12T04:11:08.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:M5ii4nIYdiOM9k8HTapEf1qCmGY=', 'Content-Length': '0'} 2024-04-12T04:11:08.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfe89da841ee425e-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.502 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.502 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.502 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:11:08.502 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:11:08.502 DEBUG:boto:Method: GET 2024-04-12T04:11:08.502 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:11:08.502 DEBUG:boto:Data: 2024-04-12T04:11:08.502 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.503 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.503 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.503 DEBUG:boto:Params: {} 2024-04-12T04:11:08.503 DEBUG:boto:Token: None 2024-04-12T04:11:08.503 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.503 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dc0c7839fd45374-006618b45c-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.506 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.506 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.507 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.507 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:11:08.507 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:11:08.507 DEBUG:boto:Method: GET 2024-04-12T04:11:08.507 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:11:08.507 DEBUG:boto:Data: 2024-04-12T04:11:08.507 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.507 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.507 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.507 DEBUG:boto:Params: {} 2024-04-12T04:11:08.508 DEBUG:boto:Token: None 2024-04-12T04:11:08.508 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.508 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e71fbde40d61b1b-006618b45c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.512 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.513 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:11:08.513 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:11:08.513 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.513 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:11:08.513 DEBUG:boto:Data: 2024-04-12T04:11:08.513 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.513 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.514 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.514 DEBUG:boto:Params: {} 2024-04-12T04:11:08.514 DEBUG:boto:Token: None 2024-04-12T04:11:08.514 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj85 2024-04-12T04:11:08.514 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UgK8bPj+yHBpDYgdqi1IyjE/Bv4= 2024-04-12T04:11:08.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UgK8bPj+yHBpDYgdqi1IyjE/Bv4=', 'Content-Length': '0'} 2024-04-12T04:11:08.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acb59c7a5b94a941-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.516 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:11:08.516 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:11:08.516 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.516 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:11:08.517 DEBUG:boto:Data: 2024-04-12T04:11:08.517 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.517 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.517 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.517 DEBUG:boto:Params: {} 2024-04-12T04:11:08.517 DEBUG:boto:Token: None 2024-04-12T04:11:08.517 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj85 2024-04-12T04:11:08.517 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UgK8bPj+yHBpDYgdqi1IyjE/Bv4= 2024-04-12T04:11:08.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UgK8bPj+yHBpDYgdqi1IyjE/Bv4=', 'Content-Length': '0'} 2024-04-12T04:11:08.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2e2e429d05a36d2-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:11:08.519 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:11:08.520 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:11:08.520 DEBUG:boto:Method: GET 2024-04-12T04:11:08.520 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:11:08.520 DEBUG:boto:Data: 2024-04-12T04:11:08.520 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.520 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.520 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.520 DEBUG:boto:Params: {} 2024-04-12T04:11:08.520 DEBUG:boto:Token: None 2024-04-12T04:11:08.521 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj85 2024-04-12T04:11:08.521 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VZYDnKLxcAt8jsFFzEigLWxYMpo= 2024-04-12T04:11:08.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VZYDnKLxcAt8jsFFzEigLWxYMpo=', 'Content-Length': '0'} 2024-04-12T04:11:08.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007be155801de7d07-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.523 DEBUG:boto:path=/dkyges-61/obj85 2024-04-12T04:11:08.523 DEBUG:boto:auth_path=/dkyges-61/obj85 2024-04-12T04:11:08.523 DEBUG:boto:Method: GET 2024-04-12T04:11:08.523 DEBUG:boto:Path: /dkyges-61/obj85 2024-04-12T04:11:08.523 DEBUG:boto:Data: 2024-04-12T04:11:08.523 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.523 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.523 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.523 DEBUG:boto:Params: {} 2024-04-12T04:11:08.524 DEBUG:boto:Token: None 2024-04-12T04:11:08.524 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj85 2024-04-12T04:11:08.524 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VZYDnKLxcAt8jsFFzEigLWxYMpo= 2024-04-12T04:11:08.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VZYDnKLxcAt8jsFFzEigLWxYMpo=', 'Content-Length': '0'} 2024-04-12T04:11:08.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:00:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000710b3fe66ef52853-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.566 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.567 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.567 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:11:08.567 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:11:08.567 DEBUG:boto:Method: GET 2024-04-12T04:11:08.567 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:11:08.567 DEBUG:boto:Data: 2024-04-12T04:11:08.567 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.567 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.567 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.568 DEBUG:boto:Params: {} 2024-04-12T04:11:08.568 DEBUG:boto:Token: None 2024-04-12T04:11:08.568 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.568 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057beaa8e6e65994a-006618b45c-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.571 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.571 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.571 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.571 DEBUG:boto:path=/dkyges-61/?prefix=obj85 2024-04-12T04:11:08.571 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj85 2024-04-12T04:11:08.572 DEBUG:boto:Method: GET 2024-04-12T04:11:08.572 DEBUG:boto:Path: /dkyges-61/?prefix=obj85 2024-04-12T04:11:08.572 DEBUG:boto:Data: 2024-04-12T04:11:08.572 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.572 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.572 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.572 DEBUG:boto:Params: {} 2024-04-12T04:11:08.572 DEBUG:boto:Token: None 2024-04-12T04:11:08.572 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.572 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d9d9be53657ee0e-006618b45c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.577 DEBUG:boto:b'dkyges-61obj851000falseobj852024-04-12T04:00:41.759Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:11:08.577 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:11:08.577 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:11:08.577 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.578 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.578 DEBUG:boto:Method: GET 2024-04-12T04:11:08.578 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.578 DEBUG:boto:Data: 2024-04-12T04:11:08.578 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.578 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.578 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.578 DEBUG:boto:Params: {} 2024-04-12T04:11:08.579 DEBUG:boto:Token: None 2024-04-12T04:11:08.579 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.579 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OWfqndWHNlhIt0z4+HdocbF3tOg= 2024-04-12T04:11:08.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OWfqndWHNlhIt0z4+HdocbF3tOg=', 'Content-Length': '0'} 2024-04-12T04:11:08.581 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca6e36286995c156-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.581 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:11:08.582 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:11:08.582 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.582 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.582 DEBUG:boto:Method: GET 2024-04-12T04:11:08.582 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.582 DEBUG:boto:Data: 2024-04-12T04:11:08.582 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.582 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.582 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.583 DEBUG:boto:Params: {} 2024-04-12T04:11:08.583 DEBUG:boto:Token: None 2024-04-12T04:11:08.583 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.583 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OWfqndWHNlhIt0z4+HdocbF3tOg= 2024-04-12T04:11:08.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OWfqndWHNlhIt0z4+HdocbF3tOg=', 'Content-Length': '0'} 2024-04-12T04:11:08.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000decd58e57bdc97dd-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.586 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:11:08.586 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:11:08.586 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.586 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.586 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.586 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.586 DEBUG:boto:Data: 2024-04-12T04:11:08.586 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.586 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.586 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.587 DEBUG:boto:Params: {} 2024-04-12T04:11:08.587 DEBUG:boto:Token: None 2024-04-12T04:11:08.587 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.587 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OymeAUkmz386RS8ujElDu/x/ZUM= 2024-04-12T04:11:08.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OymeAUkmz386RS8ujElDu/x/ZUM=', 'Content-Length': '0'} 2024-04-12T04:11:08.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fb85679df8bcc3a-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.589 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:11:08.589 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:11:08.590 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.590 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.590 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.590 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.590 DEBUG:boto:Data: 2024-04-12T04:11:08.590 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.590 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.590 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.590 DEBUG:boto:Params: {} 2024-04-12T04:11:08.591 DEBUG:boto:Token: None 2024-04-12T04:11:08.591 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.591 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OymeAUkmz386RS8ujElDu/x/ZUM= 2024-04-12T04:11:08.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OymeAUkmz386RS8ujElDu/x/ZUM=', 'Content-Length': '0'} 2024-04-12T04:11:08.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070bdcc833685e2c0-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:11:08.593 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:11:08.593 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:11:08.593 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.594 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.594 DEBUG:boto:Method: GET 2024-04-12T04:11:08.594 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.594 DEBUG:boto:Data: 2024-04-12T04:11:08.594 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.594 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.594 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.594 DEBUG:boto:Params: {} 2024-04-12T04:11:08.595 DEBUG:boto:Token: None 2024-04-12T04:11:08.595 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.595 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OWfqndWHNlhIt0z4+HdocbF3tOg= 2024-04-12T04:11:08.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OWfqndWHNlhIt0z4+HdocbF3tOg=', 'Content-Length': '0'} 2024-04-12T04:11:08.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008776934e55a9d509-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.597 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:11:08.597 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:11:08.598 DEBUG:boto:path=/dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.598 DEBUG:boto:auth_path=/dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.598 DEBUG:boto:Method: GET 2024-04-12T04:11:08.598 DEBUG:boto:Path: /dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.598 DEBUG:boto:Data: 2024-04-12T04:11:08.598 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.598 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.598 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.598 DEBUG:boto:Params: {} 2024-04-12T04:11:08.598 DEBUG:boto:Token: None 2024-04-12T04:11:08.599 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj87?versionId=null 2024-04-12T04:11:08.599 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OWfqndWHNlhIt0z4+HdocbF3tOg= 2024-04-12T04:11:08.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OWfqndWHNlhIt0z4+HdocbF3tOg=', 'Content-Length': '0'} 2024-04-12T04:11:08.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c8f02ef6fd7c6ac-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.601 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.601 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.601 DEBUG:boto:path=/dkyges-61/?prefix=obj87 2024-04-12T04:11:08.601 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj87 2024-04-12T04:11:08.601 DEBUG:boto:Method: GET 2024-04-12T04:11:08.602 DEBUG:boto:Path: /dkyges-61/?prefix=obj87 2024-04-12T04:11:08.602 DEBUG:boto:Data: 2024-04-12T04:11:08.602 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.602 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.602 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.602 DEBUG:boto:Params: {} 2024-04-12T04:11:08.602 DEBUG:boto:Token: None 2024-04-12T04:11:08.602 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.602 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000784f78be71c42637-006618b45c-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.605 DEBUG:boto:b'dkyges-61obj871000falseobj872024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.606 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.606 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.606 DEBUG:boto:path=/dkyges-61/?prefix=obj87 2024-04-12T04:11:08.606 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj87 2024-04-12T04:11:08.606 DEBUG:boto:Method: GET 2024-04-12T04:11:08.606 DEBUG:boto:Path: /dkyges-61/?prefix=obj87 2024-04-12T04:11:08.606 DEBUG:boto:Data: 2024-04-12T04:11:08.606 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.606 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.606 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.607 DEBUG:boto:Params: {} 2024-04-12T04:11:08.607 DEBUG:boto:Token: None 2024-04-12T04:11:08.607 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.607 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd561f49cceab8ca-006618b45c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.612 DEBUG:boto:b'dkyges-61obj871000falseobj872024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.612 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:11:08.612 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:11:08.612 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.613 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:11:08.613 DEBUG:boto:Data: 2024-04-12T04:11:08.613 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.613 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.613 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.613 DEBUG:boto:Params: {} 2024-04-12T04:11:08.613 DEBUG:boto:Token: None 2024-04-12T04:11:08.613 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj87 2024-04-12T04:11:08.614 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T0T7MEAdnzADwlL/6M0CgcMhGiw= 2024-04-12T04:11:08.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T0T7MEAdnzADwlL/6M0CgcMhGiw=', 'Content-Length': '0'} 2024-04-12T04:11:08.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c845e59ba8c1563-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.616 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:11:08.616 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:11:08.616 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.616 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:11:08.616 DEBUG:boto:Data: 2024-04-12T04:11:08.616 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.616 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.616 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.616 DEBUG:boto:Params: {} 2024-04-12T04:11:08.617 DEBUG:boto:Token: None 2024-04-12T04:11:08.617 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj87 2024-04-12T04:11:08.617 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T0T7MEAdnzADwlL/6M0CgcMhGiw= 2024-04-12T04:11:08.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T0T7MEAdnzADwlL/6M0CgcMhGiw=', 'Content-Length': '0'} 2024-04-12T04:11:08.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e5c889d20aa8402-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:11:08.619 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:11:08.619 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:11:08.619 DEBUG:boto:Method: GET 2024-04-12T04:11:08.619 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:11:08.619 DEBUG:boto:Data: 2024-04-12T04:11:08.619 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.619 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.620 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.620 DEBUG:boto:Params: {} 2024-04-12T04:11:08.620 DEBUG:boto:Token: None 2024-04-12T04:11:08.620 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj87 2024-04-12T04:11:08.620 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HKhc1x2S+sSDCMBKFfL3Izp7UwY= 2024-04-12T04:11:08.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HKhc1x2S+sSDCMBKFfL3Izp7UwY=', 'Content-Length': '0'} 2024-04-12T04:11:08.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c963236d71cca536-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.622 DEBUG:boto:path=/dkyges-61/obj87 2024-04-12T04:11:08.622 DEBUG:boto:auth_path=/dkyges-61/obj87 2024-04-12T04:11:08.622 DEBUG:boto:Method: GET 2024-04-12T04:11:08.623 DEBUG:boto:Path: /dkyges-61/obj87 2024-04-12T04:11:08.623 DEBUG:boto:Data: 2024-04-12T04:11:08.623 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.623 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.623 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.623 DEBUG:boto:Params: {} 2024-04-12T04:11:08.623 DEBUG:boto:Token: None 2024-04-12T04:11:08.623 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj87 2024-04-12T04:11:08.623 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HKhc1x2S+sSDCMBKFfL3Izp7UwY= 2024-04-12T04:11:08.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HKhc1x2S+sSDCMBKFfL3Izp7UwY=', 'Content-Length': '0'} 2024-04-12T04:11:08.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:06:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067e18ac1f13514e0-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.666 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.666 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.667 DEBUG:boto:path=/dkyges-61/?prefix=obj87 2024-04-12T04:11:08.667 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj87 2024-04-12T04:11:08.667 DEBUG:boto:Method: GET 2024-04-12T04:11:08.667 DEBUG:boto:Path: /dkyges-61/?prefix=obj87 2024-04-12T04:11:08.667 DEBUG:boto:Data: 2024-04-12T04:11:08.667 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.667 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.667 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.667 DEBUG:boto:Params: {} 2024-04-12T04:11:08.668 DEBUG:boto:Token: None 2024-04-12T04:11:08.668 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.668 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c6e534ecc86f7a7-006618b45c-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.671 DEBUG:boto:b'dkyges-61obj871000falseobj872024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.671 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.671 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.671 DEBUG:boto:path=/dkyges-61/?prefix=obj87 2024-04-12T04:11:08.671 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj87 2024-04-12T04:11:08.671 DEBUG:boto:Method: GET 2024-04-12T04:11:08.671 DEBUG:boto:Path: /dkyges-61/?prefix=obj87 2024-04-12T04:11:08.671 DEBUG:boto:Data: 2024-04-12T04:11:08.671 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.672 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.672 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.672 DEBUG:boto:Params: {} 2024-04-12T04:11:08.672 DEBUG:boto:Token: None 2024-04-12T04:11:08.672 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.672 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.676 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb62495615f395f7-006618b45c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.677 DEBUG:boto:b'dkyges-61obj871000falseobj872024-04-12T04:06:49.144Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:11:08.677 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:11:08.677 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:11:08.677 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.678 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.678 DEBUG:boto:Method: GET 2024-04-12T04:11:08.678 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.678 DEBUG:boto:Data: 2024-04-12T04:11:08.678 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.678 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.678 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.678 DEBUG:boto:Params: {} 2024-04-12T04:11:08.678 DEBUG:boto:Token: None 2024-04-12T04:11:08.678 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.679 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9iMhN1AuVTZmn13EsOwEa5mYmIc= 2024-04-12T04:11:08.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9iMhN1AuVTZmn13EsOwEa5mYmIc=', 'Content-Length': '0'} 2024-04-12T04:11:08.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000951239a8eac88a4d-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.722 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:11:08.722 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:11:08.723 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.723 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.723 DEBUG:boto:Method: GET 2024-04-12T04:11:08.723 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.723 DEBUG:boto:Data: 2024-04-12T04:11:08.723 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.723 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.723 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.723 DEBUG:boto:Params: {} 2024-04-12T04:11:08.723 DEBUG:boto:Token: None 2024-04-12T04:11:08.723 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.724 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9iMhN1AuVTZmn13EsOwEa5mYmIc= 2024-04-12T04:11:08.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9iMhN1AuVTZmn13EsOwEa5mYmIc=', 'Content-Length': '0'} 2024-04-12T04:11:08.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000830bdb0572833067-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.726 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:11:08.726 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:11:08.726 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.726 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.726 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.726 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.726 DEBUG:boto:Data: 2024-04-12T04:11:08.726 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.726 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.726 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.727 DEBUG:boto:Params: {} 2024-04-12T04:11:08.727 DEBUG:boto:Token: None 2024-04-12T04:11:08.727 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.727 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+uQnTiHQms5m0PLByE96Uo7RMsU= 2024-04-12T04:11:08.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+uQnTiHQms5m0PLByE96Uo7RMsU=', 'Content-Length': '0'} 2024-04-12T04:11:08.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001fc05c809cf477f-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.729 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:11:08.729 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:11:08.729 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.729 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.730 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.730 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.730 DEBUG:boto:Data: 2024-04-12T04:11:08.730 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.730 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.730 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.730 DEBUG:boto:Params: {} 2024-04-12T04:11:08.730 DEBUG:boto:Token: None 2024-04-12T04:11:08.730 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.731 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+uQnTiHQms5m0PLByE96Uo7RMsU= 2024-04-12T04:11:08.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+uQnTiHQms5m0PLByE96Uo7RMsU=', 'Content-Length': '0'} 2024-04-12T04:11:08.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4afe65b1dbce403-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:11:08.733 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:11:08.733 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:11:08.733 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.733 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.733 DEBUG:boto:Method: GET 2024-04-12T04:11:08.733 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.733 DEBUG:boto:Data: 2024-04-12T04:11:08.734 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.734 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.734 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.734 DEBUG:boto:Params: {} 2024-04-12T04:11:08.734 DEBUG:boto:Token: None 2024-04-12T04:11:08.734 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.734 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9iMhN1AuVTZmn13EsOwEa5mYmIc= 2024-04-12T04:11:08.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9iMhN1AuVTZmn13EsOwEa5mYmIc=', 'Content-Length': '0'} 2024-04-12T04:11:08.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7f80d085c628bf9-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.736 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:11:08.736 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:11:08.736 DEBUG:boto:path=/dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.737 DEBUG:boto:auth_path=/dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.737 DEBUG:boto:Method: GET 2024-04-12T04:11:08.737 DEBUG:boto:Path: /dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.737 DEBUG:boto:Data: 2024-04-12T04:11:08.737 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.737 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.737 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.737 DEBUG:boto:Params: {} 2024-04-12T04:11:08.737 DEBUG:boto:Token: None 2024-04-12T04:11:08.737 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj92?versionId=null 2024-04-12T04:11:08.737 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9iMhN1AuVTZmn13EsOwEa5mYmIc= 2024-04-12T04:11:08.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9iMhN1AuVTZmn13EsOwEa5mYmIc=', 'Content-Length': '0'} 2024-04-12T04:11:08.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaca07d5c06c472e-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.740 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.740 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.740 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:11:08.740 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:11:08.740 DEBUG:boto:Method: GET 2024-04-12T04:11:08.740 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:11:08.740 DEBUG:boto:Data: 2024-04-12T04:11:08.740 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.740 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.740 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.740 DEBUG:boto:Params: {} 2024-04-12T04:11:08.741 DEBUG:boto:Token: None 2024-04-12T04:11:08.741 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.741 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e61b07fc590f300-006618b45c-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.743 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.744 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.744 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.744 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:11:08.744 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:11:08.744 DEBUG:boto:Method: GET 2024-04-12T04:11:08.744 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:11:08.744 DEBUG:boto:Data: 2024-04-12T04:11:08.744 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.744 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.744 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.745 DEBUG:boto:Params: {} 2024-04-12T04:11:08.745 DEBUG:boto:Token: None 2024-04-12T04:11:08.745 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.745 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ec16078984491a5-006618b45c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.749 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.749 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:11:08.749 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:11:08.750 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.750 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:11:08.750 DEBUG:boto:Data: 2024-04-12T04:11:08.750 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.750 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.750 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.750 DEBUG:boto:Params: {} 2024-04-12T04:11:08.750 DEBUG:boto:Token: None 2024-04-12T04:11:08.750 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj92 2024-04-12T04:11:08.750 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WqjkNA8hJy60LcLK6ErERvtjwBQ= 2024-04-12T04:11:08.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WqjkNA8hJy60LcLK6ErERvtjwBQ=', 'Content-Length': '0'} 2024-04-12T04:11:08.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d328f0697cdee414-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.752 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:11:08.753 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:11:08.753 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.753 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:11:08.753 DEBUG:boto:Data: 2024-04-12T04:11:08.753 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.753 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.753 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.753 DEBUG:boto:Params: {} 2024-04-12T04:11:08.753 DEBUG:boto:Token: None 2024-04-12T04:11:08.753 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj92 2024-04-12T04:11:08.753 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WqjkNA8hJy60LcLK6ErERvtjwBQ= 2024-04-12T04:11:08.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WqjkNA8hJy60LcLK6ErERvtjwBQ=', 'Content-Length': '0'} 2024-04-12T04:11:08.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a769d39c8be2f9a6-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:11:08.755 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:11:08.756 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:11:08.756 DEBUG:boto:Method: GET 2024-04-12T04:11:08.756 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:11:08.756 DEBUG:boto:Data: 2024-04-12T04:11:08.756 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.756 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.756 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.756 DEBUG:boto:Params: {} 2024-04-12T04:11:08.756 DEBUG:boto:Token: None 2024-04-12T04:11:08.756 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj92 2024-04-12T04:11:08.756 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YTqsMb5C6WThDRc/kRxYuHTvLJw= 2024-04-12T04:11:08.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YTqsMb5C6WThDRc/kRxYuHTvLJw=', 'Content-Length': '0'} 2024-04-12T04:11:08.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000989ea5c74cb36a42-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.758 DEBUG:boto:path=/dkyges-61/obj92 2024-04-12T04:11:08.759 DEBUG:boto:auth_path=/dkyges-61/obj92 2024-04-12T04:11:08.759 DEBUG:boto:Method: GET 2024-04-12T04:11:08.759 DEBUG:boto:Path: /dkyges-61/obj92 2024-04-12T04:11:08.759 DEBUG:boto:Data: 2024-04-12T04:11:08.759 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.759 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.759 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.759 DEBUG:boto:Params: {} 2024-04-12T04:11:08.759 DEBUG:boto:Token: None 2024-04-12T04:11:08.759 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj92 2024-04-12T04:11:08.760 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YTqsMb5C6WThDRc/kRxYuHTvLJw= 2024-04-12T04:11:08.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YTqsMb5C6WThDRc/kRxYuHTvLJw=', 'Content-Length': '0'} 2024-04-12T04:11:08.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:05:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001de22fec61e8402a-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.802 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.802 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.803 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:11:08.803 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:11:08.803 DEBUG:boto:Method: GET 2024-04-12T04:11:08.803 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:11:08.803 DEBUG:boto:Data: 2024-04-12T04:11:08.803 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.803 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.803 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.803 DEBUG:boto:Params: {} 2024-04-12T04:11:08.803 DEBUG:boto:Token: None 2024-04-12T04:11:08.804 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.804 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000091693ddae31617f-006618b45c-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.806 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.807 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.807 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.807 DEBUG:boto:path=/dkyges-61/?prefix=obj92 2024-04-12T04:11:08.807 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj92 2024-04-12T04:11:08.807 DEBUG:boto:Method: GET 2024-04-12T04:11:08.807 DEBUG:boto:Path: /dkyges-61/?prefix=obj92 2024-04-12T04:11:08.807 DEBUG:boto:Data: 2024-04-12T04:11:08.807 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.807 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.807 DEBUG:boto:Params: {} 2024-04-12T04:11:08.807 DEBUG:boto:Token: None 2024-04-12T04:11:08.808 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.808 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f111301cfa9a88ca-006618b45c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.812 DEBUG:boto:b'dkyges-61obj921000falseobj922024-04-12T04:05:01.141Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T04:11:08.812 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:11:08.812 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:11:08.813 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.813 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.813 DEBUG:boto:Method: GET 2024-04-12T04:11:08.813 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.813 DEBUG:boto:Data: 2024-04-12T04:11:08.813 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.813 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.813 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.813 DEBUG:boto:Params: {} 2024-04-12T04:11:08.813 DEBUG:boto:Token: None 2024-04-12T04:11:08.813 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.814 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nPglq90DJ12LbhLI7ZmGq19A2Ek= 2024-04-12T04:11:08.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nPglq90DJ12LbhLI7ZmGq19A2Ek=', 'Content-Length': '0'} 2024-04-12T04:11:08.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eec5ac482c57dba-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.858 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:11:08.858 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:11:08.859 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.859 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.859 DEBUG:boto:Method: GET 2024-04-12T04:11:08.859 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.859 DEBUG:boto:Data: 2024-04-12T04:11:08.859 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.859 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.859 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.859 DEBUG:boto:Params: {} 2024-04-12T04:11:08.859 DEBUG:boto:Token: None 2024-04-12T04:11:08.860 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.860 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nPglq90DJ12LbhLI7ZmGq19A2Ek= 2024-04-12T04:11:08.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nPglq90DJ12LbhLI7ZmGq19A2Ek=', 'Content-Length': '0'} 2024-04-12T04:11:08.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000bd8e08367968b8-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.862 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:11:08.862 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:11:08.862 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.862 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.862 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.862 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.862 DEBUG:boto:Data: 2024-04-12T04:11:08.863 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.863 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.863 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.863 DEBUG:boto:Params: {} 2024-04-12T04:11:08.863 DEBUG:boto:Token: None 2024-04-12T04:11:08.863 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.863 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YDVfBzt5AGuZwYshxdXbBqK620k= 2024-04-12T04:11:08.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YDVfBzt5AGuZwYshxdXbBqK620k=', 'Content-Length': '0'} 2024-04-12T04:11:08.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004adc8e3f808cf03d-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.865 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:11:08.865 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:11:08.865 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.865 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.866 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.866 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.866 DEBUG:boto:Data: 2024-04-12T04:11:08.866 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.866 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.866 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.866 DEBUG:boto:Params: {} 2024-04-12T04:11:08.866 DEBUG:boto:Token: None 2024-04-12T04:11:08.866 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.866 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YDVfBzt5AGuZwYshxdXbBqK620k= 2024-04-12T04:11:08.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YDVfBzt5AGuZwYshxdXbBqK620k=', 'Content-Length': '0'} 2024-04-12T04:11:08.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000567e1169035b22aa-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T04:11:08.869 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:11:08.869 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:11:08.869 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.869 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.869 DEBUG:boto:Method: GET 2024-04-12T04:11:08.869 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.869 DEBUG:boto:Data: 2024-04-12T04:11:08.869 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.870 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.870 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.870 DEBUG:boto:Params: {} 2024-04-12T04:11:08.870 DEBUG:boto:Token: None 2024-04-12T04:11:08.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nPglq90DJ12LbhLI7ZmGq19A2Ek= 2024-04-12T04:11:08.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nPglq90DJ12LbhLI7ZmGq19A2Ek=', 'Content-Length': '0'} 2024-04-12T04:11:08.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f7c2826a26ff238-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.873 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:11:08.873 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:11:08.873 DEBUG:boto:path=/dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.873 DEBUG:boto:auth_path=/dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.873 DEBUG:boto:Method: GET 2024-04-12T04:11:08.873 DEBUG:boto:Path: /dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.873 DEBUG:boto:Data: 2024-04-12T04:11:08.873 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.873 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.873 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.874 DEBUG:boto:Params: {} 2024-04-12T04:11:08.874 DEBUG:boto:Token: None 2024-04-12T04:11:08.874 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj93?versionId=null 2024-04-12T04:11:08.874 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nPglq90DJ12LbhLI7ZmGq19A2Ek= 2024-04-12T04:11:08.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nPglq90DJ12LbhLI7ZmGq19A2Ek=', 'Content-Length': '0'} 2024-04-12T04:11:08.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000deec2d89cb963d0a-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.876 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.876 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.876 DEBUG:boto:path=/dkyges-61/?prefix=obj93 2024-04-12T04:11:08.876 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj93 2024-04-12T04:11:08.876 DEBUG:boto:Method: GET 2024-04-12T04:11:08.876 DEBUG:boto:Path: /dkyges-61/?prefix=obj93 2024-04-12T04:11:08.877 DEBUG:boto:Data: 2024-04-12T04:11:08.877 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.877 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.877 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.877 DEBUG:boto:Params: {} 2024-04-12T04:11:08.877 DEBUG:boto:Token: None 2024-04-12T04:11:08.877 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.877 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000724918d8f0d9f732-006618b45c-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.880 DEBUG:boto:b'dkyges-61obj931000falseobj932024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.880 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.880 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.880 DEBUG:boto:path=/dkyges-61/?prefix=obj93 2024-04-12T04:11:08.880 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj93 2024-04-12T04:11:08.881 DEBUG:boto:Method: GET 2024-04-12T04:11:08.881 DEBUG:boto:Path: /dkyges-61/?prefix=obj93 2024-04-12T04:11:08.881 DEBUG:boto:Data: 2024-04-12T04:11:08.881 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.881 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.881 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.881 DEBUG:boto:Params: {} 2024-04-12T04:11:08.881 DEBUG:boto:Token: None 2024-04-12T04:11:08.881 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.881 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3c6476f0de1cc63-006618b45c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.886 DEBUG:boto:b'dkyges-61obj931000falseobj932024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.886 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:11:08.886 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:11:08.887 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.887 DEBUG:boto:Path: /dkyges-61/obj93 2024-04-12T04:11:08.887 DEBUG:boto:Data: 2024-04-12T04:11:08.887 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.887 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.887 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.887 DEBUG:boto:Params: {} 2024-04-12T04:11:08.887 DEBUG:boto:Token: None 2024-04-12T04:11:08.887 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj93 2024-04-12T04:11:08.887 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5HksknibIw8NXGrQW1PkA1GrsCQ= 2024-04-12T04:11:08.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5HksknibIw8NXGrQW1PkA1GrsCQ=', 'Content-Length': '0'} 2024-04-12T04:11:08.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fa209dfefe6d99f-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.889 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:11:08.890 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:11:08.890 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.890 DEBUG:boto:Path: /dkyges-61/obj93 2024-04-12T04:11:08.890 DEBUG:boto:Data: 2024-04-12T04:11:08.890 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.890 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.890 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.890 DEBUG:boto:Params: {} 2024-04-12T04:11:08.890 DEBUG:boto:Token: None 2024-04-12T04:11:08.890 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj93 2024-04-12T04:11:08.891 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5HksknibIw8NXGrQW1PkA1GrsCQ= 2024-04-12T04:11:08.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5HksknibIw8NXGrQW1PkA1GrsCQ=', 'Content-Length': '0'} 2024-04-12T04:11:08.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000245ef071187bfc44-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T04:11:08.893 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:11:08.893 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:11:08.893 DEBUG:boto:Method: GET 2024-04-12T04:11:08.893 DEBUG:boto:Path: /dkyges-61/obj93 2024-04-12T04:11:08.893 DEBUG:boto:Data: 2024-04-12T04:11:08.893 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.893 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.893 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.893 DEBUG:boto:Params: {} 2024-04-12T04:11:08.894 DEBUG:boto:Token: None 2024-04-12T04:11:08.894 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj93 2024-04-12T04:11:08.894 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gBHvM5LRn9JNA+kww0iE6UEI81A= 2024-04-12T04:11:08.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gBHvM5LRn9JNA+kww0iE6UEI81A=', 'Content-Length': '0'} 2024-04-12T04:11:08.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000836e249c5a94bb72-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.896 DEBUG:boto:path=/dkyges-61/obj93 2024-04-12T04:11:08.896 DEBUG:boto:auth_path=/dkyges-61/obj93 2024-04-12T04:11:08.896 DEBUG:boto:Method: GET 2024-04-12T04:11:08.896 DEBUG:boto:Path: /dkyges-61/obj93 2024-04-12T04:11:08.896 DEBUG:boto:Data: 2024-04-12T04:11:08.896 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.897 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.897 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.897 DEBUG:boto:Params: {} 2024-04-12T04:11:08.897 DEBUG:boto:Token: None 2024-04-12T04:11:08.897 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj93 2024-04-12T04:11:08.897 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gBHvM5LRn9JNA+kww0iE6UEI81A= 2024-04-12T04:11:08.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gBHvM5LRn9JNA+kww0iE6UEI81A=', 'Content-Length': '0'} 2024-04-12T04:11:08.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:08:08 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000617d9e79f669a7dc-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.899 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.899 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.899 DEBUG:boto:path=/dkyges-61/?prefix=obj93 2024-04-12T04:11:08.899 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj93 2024-04-12T04:11:08.899 DEBUG:boto:Method: GET 2024-04-12T04:11:08.900 DEBUG:boto:Path: /dkyges-61/?prefix=obj93 2024-04-12T04:11:08.900 DEBUG:boto:Data: 2024-04-12T04:11:08.900 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.900 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.900 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.900 DEBUG:boto:Params: {} 2024-04-12T04:11:08.900 DEBUG:boto:Token: None 2024-04-12T04:11:08.900 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.900 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067d17d5035afdbdb-006618b45c-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.903 DEBUG:boto:b'dkyges-61obj931000falseobj932024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.903 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.903 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.903 DEBUG:boto:path=/dkyges-61/?prefix=obj93 2024-04-12T04:11:08.903 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj93 2024-04-12T04:11:08.904 DEBUG:boto:Method: GET 2024-04-12T04:11:08.904 DEBUG:boto:Path: /dkyges-61/?prefix=obj93 2024-04-12T04:11:08.904 DEBUG:boto:Data: 2024-04-12T04:11:08.904 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.904 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.904 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.904 DEBUG:boto:Params: {} 2024-04-12T04:11:08.904 DEBUG:boto:Token: None 2024-04-12T04:11:08.904 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.904 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090e298f1d5c66c1c-006618b45c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.950 DEBUG:boto:b'dkyges-61obj931000falseobj932024-04-12T04:08:08.235Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:08.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:11:08.951 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:11:08.951 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:11:08.951 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.951 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.951 DEBUG:boto:Method: GET 2024-04-12T04:11:08.951 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.951 DEBUG:boto:Data: 2024-04-12T04:11:08.951 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.951 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.951 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.951 DEBUG:boto:Params: {} 2024-04-12T04:11:08.952 DEBUG:boto:Token: None 2024-04-12T04:11:08.952 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.952 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zrveP4+U2hTSmmLW2AD7x1Qz1Ro= 2024-04-12T04:11:08.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zrveP4+U2hTSmmLW2AD7x1Qz1Ro=', 'Content-Length': '0'} 2024-04-12T04:11:08.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000776c2ccf6034e538-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.954 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:11:08.954 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:11:08.954 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.955 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.955 DEBUG:boto:Method: GET 2024-04-12T04:11:08.955 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.955 DEBUG:boto:Data: 2024-04-12T04:11:08.955 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.955 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.955 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.955 DEBUG:boto:Params: {} 2024-04-12T04:11:08.955 DEBUG:boto:Token: None 2024-04-12T04:11:08.955 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.955 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zrveP4+U2hTSmmLW2AD7x1Qz1Ro= 2024-04-12T04:11:08.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zrveP4+U2hTSmmLW2AD7x1Qz1Ro=', 'Content-Length': '0'} 2024-04-12T04:11:08.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054e719eecd1429c5-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.957 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:11:08.958 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:11:08.958 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.958 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.958 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.958 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.958 DEBUG:boto:Data: 2024-04-12T04:11:08.958 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.958 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.958 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.958 DEBUG:boto:Params: {} 2024-04-12T04:11:08.959 DEBUG:boto:Token: None 2024-04-12T04:11:08.959 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.959 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uAHC2ioKvTXTMXB/iI+qZulB3d4= 2024-04-12T04:11:08.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uAHC2ioKvTXTMXB/iI+qZulB3d4=', 'Content-Length': '0'} 2024-04-12T04:11:08.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e733e761646b85f9-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.961 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:11:08.961 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:11:08.961 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.961 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.961 DEBUG:boto:Method: HEAD 2024-04-12T04:11:08.961 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.961 DEBUG:boto:Data: 2024-04-12T04:11:08.961 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.961 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.962 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.962 DEBUG:boto:Params: {} 2024-04-12T04:11:08.962 DEBUG:boto:Token: None 2024-04-12T04:11:08.962 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.962 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uAHC2ioKvTXTMXB/iI+qZulB3d4= 2024-04-12T04:11:08.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uAHC2ioKvTXTMXB/iI+qZulB3d4=', 'Content-Length': '0'} 2024-04-12T04:11:08.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a47a73f0b4bc4aeb-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:11:08.964 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:11:08.964 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:11:08.964 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.964 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.965 DEBUG:boto:Method: GET 2024-04-12T04:11:08.965 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.965 DEBUG:boto:Data: 2024-04-12T04:11:08.965 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.965 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.965 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.965 DEBUG:boto:Params: {} 2024-04-12T04:11:08.965 DEBUG:boto:Token: None 2024-04-12T04:11:08.965 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.965 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zrveP4+U2hTSmmLW2AD7x1Qz1Ro= 2024-04-12T04:11:08.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zrveP4+U2hTSmmLW2AD7x1Qz1Ro=', 'Content-Length': '0'} 2024-04-12T04:11:08.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050a5c8032546b32f-006618b45c-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.967 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:11:08.968 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:11:08.968 DEBUG:boto:path=/dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.968 DEBUG:boto:auth_path=/dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.968 DEBUG:boto:Method: GET 2024-04-12T04:11:08.968 DEBUG:boto:Path: /dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.968 DEBUG:boto:Data: 2024-04-12T04:11:08.968 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.968 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:08.968 DEBUG:boto:Port: 8000 2024-04-12T04:11:08.968 DEBUG:boto:Params: {} 2024-04-12T04:11:08.968 DEBUG:boto:Token: None 2024-04-12T04:11:08.969 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/obj95?versionId=null 2024-04-12T04:11:08.969 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zrveP4+U2hTSmmLW2AD7x1Qz1Ro= 2024-04-12T04:11:08.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zrveP4+U2hTSmmLW2AD7x1Qz1Ro=', 'Content-Length': '0'} 2024-04-12T04:11:08.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066128ef300855c2f-006618b45c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:08.971 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:08.971 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:08.971 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:11:08.971 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:11:08.971 DEBUG:boto:Method: GET 2024-04-12T04:11:08.971 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:11:08.971 DEBUG:boto:Data: 2024-04-12T04:11:08.972 DEBUG:boto:Headers: {} 2024-04-12T04:11:08.972 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:08.972 DEBUG:boto:Port: 8001 2024-04-12T04:11:08.972 DEBUG:boto:Params: {} 2024-04-12T04:11:08.972 DEBUG:boto:Token: None 2024-04-12T04:11:08.972 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:08 GMT /dkyges-61/ 2024-04-12T04:11:08.972 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8= 2024-04-12T04:11:08.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tEAlFRrwxNdRmnjBbpyWwoZr2q8=', 'Content-Length': '0'} 2024-04-12T04:11:08.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000304c2ab5efe02f72-006618b45c-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:09.018 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:09.019 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:09.019 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:09.019 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:11:09.019 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:11:09.019 DEBUG:boto:Method: GET 2024-04-12T04:11:09.019 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:11:09.019 DEBUG:boto:Data: 2024-04-12T04:11:09.019 DEBUG:boto:Headers: {} 2024-04-12T04:11:09.019 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:09.019 DEBUG:boto:Port: 8000 2024-04-12T04:11:09.019 DEBUG:boto:Params: {} 2024-04-12T04:11:09.020 DEBUG:boto:Token: None 2024-04-12T04:11:09.020 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:09 GMT /dkyges-61/ 2024-04-12T04:11:09.020 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QfcqaD65TwDH+5JOVdBno5ZhtbA= 2024-04-12T04:11:09.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QfcqaD65TwDH+5JOVdBno5ZhtbA=', 'Content-Length': '0'} 2024-04-12T04:11:09.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dee0ebe485bdd44d-006618b45d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:09.024 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:09.025 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:11:09.025 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:11:09.025 DEBUG:boto:Method: HEAD 2024-04-12T04:11:09.025 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:11:09.025 DEBUG:boto:Data: 2024-04-12T04:11:09.025 DEBUG:boto:Headers: {} 2024-04-12T04:11:09.025 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:09.025 DEBUG:boto:Port: 8001 2024-04-12T04:11:09.025 DEBUG:boto:Params: {} 2024-04-12T04:11:09.025 DEBUG:boto:Token: None 2024-04-12T04:11:09.026 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:09 GMT /dkyges-61/obj95 2024-04-12T04:11:09.026 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:y5IgBdwVop/TDrPmnrPi4sQgiGg= 2024-04-12T04:11:09.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y5IgBdwVop/TDrPmnrPi4sQgiGg=', 'Content-Length': '0'} 2024-04-12T04:11:09.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000101a175d817febb1-006618b45d-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:09.028 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:11:09.028 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:11:09.028 DEBUG:boto:Method: HEAD 2024-04-12T04:11:09.028 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:11:09.028 DEBUG:boto:Data: 2024-04-12T04:11:09.028 DEBUG:boto:Headers: {} 2024-04-12T04:11:09.029 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:09.029 DEBUG:boto:Port: 8000 2024-04-12T04:11:09.029 DEBUG:boto:Params: {} 2024-04-12T04:11:09.029 DEBUG:boto:Token: None 2024-04-12T04:11:09.029 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:11:09 GMT /dkyges-61/obj95 2024-04-12T04:11:09.029 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:y5IgBdwVop/TDrPmnrPi4sQgiGg= 2024-04-12T04:11:09.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y5IgBdwVop/TDrPmnrPi4sQgiGg=', 'Content-Length': '0'} 2024-04-12T04:11:09.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab6092cb750b4e4d-006618b45d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:09.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:11:09.031 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:11:09.031 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:11:09.031 DEBUG:boto:Method: GET 2024-04-12T04:11:09.032 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:11:09.032 DEBUG:boto:Data: 2024-04-12T04:11:09.032 DEBUG:boto:Headers: {} 2024-04-12T04:11:09.032 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:09.032 DEBUG:boto:Port: 8001 2024-04-12T04:11:09.032 DEBUG:boto:Params: {} 2024-04-12T04:11:09.032 DEBUG:boto:Token: None 2024-04-12T04:11:09.032 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:09 GMT /dkyges-61/obj95 2024-04-12T04:11:09.032 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tyE97FmFIv1fC6bBnmEzyU5iiDo= 2024-04-12T04:11:09.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tyE97FmFIv1fC6bBnmEzyU5iiDo=', 'Content-Length': '0'} 2024-04-12T04:11:09.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-61:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007de9e3bd9527d852-006618b45d-1fa3-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:09.035 DEBUG:boto:path=/dkyges-61/obj95 2024-04-12T04:11:09.035 DEBUG:boto:auth_path=/dkyges-61/obj95 2024-04-12T04:11:09.035 DEBUG:boto:Method: GET 2024-04-12T04:11:09.035 DEBUG:boto:Path: /dkyges-61/obj95 2024-04-12T04:11:09.035 DEBUG:boto:Data: 2024-04-12T04:11:09.035 DEBUG:boto:Headers: {} 2024-04-12T04:11:09.035 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:09.035 DEBUG:boto:Port: 8000 2024-04-12T04:11:09.035 DEBUG:boto:Params: {} 2024-04-12T04:11:09.035 DEBUG:boto:Token: None 2024-04-12T04:11:09.036 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:09 GMT /dkyges-61/obj95 2024-04-12T04:11:09.036 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tyE97FmFIv1fC6bBnmEzyU5iiDo= 2024-04-12T04:11:09.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tyE97FmFIv1fC6bBnmEzyU5iiDo=', 'Content-Length': '0'} 2024-04-12T04:11:09.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:03:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d64c6c030d602e2b-006618b45d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:11:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:09.078 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:09.078 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:09.078 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:11:09.079 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:11:09.079 DEBUG:boto:Method: GET 2024-04-12T04:11:09.079 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:11:09.079 DEBUG:boto:Data: 2024-04-12T04:11:09.079 DEBUG:boto:Headers: {} 2024-04-12T04:11:09.079 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:11:09.079 DEBUG:boto:Port: 8001 2024-04-12T04:11:09.079 DEBUG:boto:Params: {} 2024-04-12T04:11:09.079 DEBUG:boto:Token: None 2024-04-12T04:11:09.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:09 GMT /dkyges-61/ 2024-04-12T04:11:09.080 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QfcqaD65TwDH+5JOVdBno5ZhtbA= 2024-04-12T04:11:09.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QfcqaD65TwDH+5JOVdBno5ZhtbA=', 'Content-Length': '0'} 2024-04-12T04:11:09.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009499b21e9a957c48-006618b45d-1fa3-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:09.083 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:09.083 DEBUG:boto:path=/dkyges-61/ 2024-04-12T04:11:09.083 DEBUG:boto:auth_path=/dkyges-61/ 2024-04-12T04:11:09.083 DEBUG:boto:path=/dkyges-61/?prefix=obj95 2024-04-12T04:11:09.083 DEBUG:boto:auth_path=/dkyges-61/?prefix=obj95 2024-04-12T04:11:09.083 DEBUG:boto:Method: GET 2024-04-12T04:11:09.083 DEBUG:boto:Path: /dkyges-61/?prefix=obj95 2024-04-12T04:11:09.083 DEBUG:boto:Data: 2024-04-12T04:11:09.084 DEBUG:boto:Headers: {} 2024-04-12T04:11:09.084 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:11:09.084 DEBUG:boto:Port: 8000 2024-04-12T04:11:09.084 DEBUG:boto:Params: {} 2024-04-12T04:11:09.084 DEBUG:boto:Token: None 2024-04-12T04:11:09.084 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:11:09 GMT /dkyges-61/ 2024-04-12T04:11:09.084 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QfcqaD65TwDH+5JOVdBno5ZhtbA= 2024-04-12T04:11:09.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:11:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QfcqaD65TwDH+5JOVdBno5ZhtbA=', 'Content-Length': '0'} 2024-04-12T04:11:09.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbc925e4b68bf101-006618b45d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:11:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:09.089 DEBUG:boto:b'dkyges-61obj951000falseobj952024-04-12T04:03:03.702Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:11:09.090 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-61 zones={a2, a1} 2024-04-12T04:11:09.090 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ok 2024-04-12T04:11:09.095 INFO:rgw_multi.tests:create role zone=a1 name=roles-dkyges-1 2024-04-12T04:11:09.095 DEBUG:boto:Method: POST 2024-04-12T04:11:09.095 DEBUG:boto:Path: / 2024-04-12T04:11:09.095 DEBUG:boto:Data: 2024-04-12T04:11:09.095 DEBUG:boto:Headers: {} 2024-04-12T04:11:09.095 DEBUG:boto:Host: smithi043.front.sepia.ceph.com 2024-04-12T04:11:09.095 DEBUG:boto:Port: 8000 2024-04-12T04:11:09.096 DEBUG:boto:Params: {'RoleName': 'roles-dkyges-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-04-12T04:11:09.096 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:11:09.096 DEBUG:boto:Token: None 2024-04-12T04:11:09.096 DEBUG:boto:CanonicalRequest: POST / host:smithi043.front.sepia.ceph.com:8000 x-amz-date:20240412T041109Z host;x-amz-date a3c0114b2c2a2e960ded7e09e9acbc4c0bb96093e766b5ba2c4dbfcf60c1f2fa 2024-04-12T04:11:09.096 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240412T041109Z 20240412/front/smithi043/aws4_request 74d30b68537489822404ba7613c2b84b9d05ca945b87f836366824a8487678db 2024-04-12T04:11:09.096 DEBUG:boto:Signature: d972d3dd40a05ef40122950266e74af977a0e00a3fcdd0f7f6c0cdf9f6df3a36 2024-04-12T04:11:09.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240412T041109Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=KUXF4P9H15V1YOGZBQF6/20240412/front/smithi043/aws4_request,SignedHeaders=host;x-amz-date,Signature=d972d3dd40a05ef40122950266e74af977a0e00a3fcdd0f7f6c0cdf9f6df3a36', 'Host': 'smithi043.front.sepia.ceph.com:8000'} 2024-04-12T04:11:09.155 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000884e14d5db3d9d91-006618b45d-16d5-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Fri, 12 Apr 2024 04:11:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:09.156 DEBUG:boto:b'780605d1-c318-4389-a003-a1a8764b84beroles-dkyges-1/arn:aws:iam:::role/roles-dkyges-12024-04-12T04:11:09.163Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000884e14d5db3d9d91-006618b45d-16d5-a1' 2024-04-12T04:11:09.156 INFO:rgw_multi.tests:create role zone=a2 name=roles-dkyges-2 2024-04-12T04:11:09.156 DEBUG:boto:Method: POST 2024-04-12T04:11:09.157 DEBUG:boto:Path: / 2024-04-12T04:11:09.157 DEBUG:boto:Data: 2024-04-12T04:11:09.157 DEBUG:boto:Headers: {} 2024-04-12T04:11:09.157 DEBUG:boto:Host: smithi043.front.sepia.ceph.com 2024-04-12T04:11:09.157 DEBUG:boto:Port: 8001 2024-04-12T04:11:09.157 DEBUG:boto:Params: {'RoleName': 'roles-dkyges-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2024-04-12T04:11:09.157 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:11:09.157 DEBUG:boto:Token: None 2024-04-12T04:11:09.157 DEBUG:boto:CanonicalRequest: POST / host:smithi043.front.sepia.ceph.com:8001 x-amz-date:20240412T041109Z host;x-amz-date 931e0d61ae56da934fc91c304ffecc11abb53529a2b0e53c04b57e37367f8850 2024-04-12T04:11:09.158 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240412T041109Z 20240412/front/smithi043/aws4_request 045c96ad9100338b5c963ef17a4bc614bbb962112a5a374c10e4cb7a316d2690 2024-04-12T04:11:09.158 DEBUG:boto:Signature: 241d50c2d9a6f623273134f07d8bdc89485167d59b38042b827a843021a38a77 2024-04-12T04:11:09.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240412T041109Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=KUXF4P9H15V1YOGZBQF6/20240412/front/smithi043/aws4_request,SignedHeaders=host;x-amz-date,Signature=241d50c2d9a6f623273134f07d8bdc89485167d59b38042b827a843021a38a77', 'Host': 'smithi043.front.sepia.ceph.com:8001'} 2024-04-12T04:11:09.191 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004c9f75553f23bec2-006618b45d-1fa3-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Fri, 12 Apr 2024 04:11:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:11:09.191 DEBUG:boto:b'e00f8139-fc24-4085-bd42-b3d54299d876roles-dkyges-2/arn:aws:iam:::role/roles-dkyges-22024-04-12T04:11:09.202Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx000004c9f75553f23bec2-006618b45d-1fa3-a2' 2024-04-12T04:11:09.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:09.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:09.192 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:11:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:09.288 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:11:09.303 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:11:09.304 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:09.304 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:11:09.304 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:11:09.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:09.304 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:09.304 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:11:09.304 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:11:09.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:09.304 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:09.304 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:11:09.304 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:11:09.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:09.305 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:09.305 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:11:09.305 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:09:23.960546Z" 2024-04-12T04:11:09.305 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:09.305 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:11:09.306 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:09.306 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:11:09.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:09.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:09.306 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:11:09.361 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:09.361 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:09.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:09.412+0000 7f7b8e33b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:09.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:09.413+0000 7f7b8e33b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:09.413 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:11:09.413 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:11:09.413 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:11:09.413 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:11:09.413 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:11:09.414 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:11:09.414 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:09.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:09.414 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:11:09.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:09.414 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:11:09.414 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:09.414 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:09.414 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:11:09.414 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:09.414 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:11:09.415 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:09.415 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:11:09.415 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:09.415 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:09.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:09.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:09.415 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:11:09.415 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:09.415 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:09.415 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:11:09.415 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:11:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:11:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:09.416 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:11:09.417 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:09.417 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:09.417 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:09.417 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:11:09.417 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:09.417 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:09.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:09.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:09.417 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:11:09.417 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:09.417 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:09.417 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:11:09.418 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:09.418 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:09.418 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:09.418 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:11:09.418 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:09.418 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:09.418 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:09.418 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:11:09.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:09.418 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:11:09.418 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:11:09.419 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:11:09.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:09.419 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:11:09Z" 2024-04-12T04:11:09.419 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:11:09.420 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:11:09Z" } 2024-04-12T04:11:09.420 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:11:09.420 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:09.420 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:09.420 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:11:14.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:14.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:14.422 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:11:14.481 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:14.481 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:14.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:14.534+0000 7f76ae00a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:14.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:14.535+0000 7f76ae00a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:14.534 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:11:14.535 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:11:14.535 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:11:14.535 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:11:14.535 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:11:14.535 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:11:14.535 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:14.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:14.535 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:11:14.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:14.535 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:11:14.536 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:14.536 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:14.536 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:11:14.536 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:14.536 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:11:14.536 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:14.536 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:11:14.536 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:14.536 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:14.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:14.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:14.537 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:11:14.537 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:14.537 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:14.537 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:11:14.537 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:14.537 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:14.537 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:14.537 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:11:14.537 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:14.537 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:14.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:14.538 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:14.538 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:11:14.538 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:14.538 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:14.538 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:11:14.538 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:14.538 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:14.538 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:14.538 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:11:14.538 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:14.539 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:14.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:14.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:14.539 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:11:14.539 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:14.539 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:14.539 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:11:14.539 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:14.539 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:14.539 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:14.539 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:11:14.540 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:14.540 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:14.540 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:14.540 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:11:14.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:14.540 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:11:14.540 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:11:14.540 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:11:14.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:14.540 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:11:14Z" 2024-04-12T04:11:14.540 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:11:14.541 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:11:14Z" } 2024-04-12T04:11:14.541 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:11:14.542 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:14.542 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:14.542 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:11:19.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:19.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:19.544 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:11:19.603 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:19.603 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:19.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:19.655+0000 7f8c703f4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:19.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:19.656+0000 7f8c703f4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:19.655 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:11:19.655 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:11:19.655 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:11:19.655 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:11:19.655 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:11:19.656 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:11:19.656 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:19.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:19.656 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:11:19.656 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:19.656 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:11:19.656 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:19.656 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:19.656 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:11:19.656 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:19.656 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:11:19.656 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:19.657 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:11:19.657 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:19.657 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:19.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:19.657 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:19.657 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:11:19.657 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:19.657 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:19.657 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:11:19.657 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:19.658 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:19.658 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:19.658 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:11:19.658 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:19.658 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:19.658 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:19.658 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:19.658 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:11:19.658 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:19.658 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:19.658 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:11:19.659 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:19.659 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:19.659 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:19.659 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:11:19.659 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:19.659 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:19.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:19.659 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:19.659 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:11:19.659 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:19.659 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:19.660 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:11:19.660 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:19.660 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:19.660 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:19.660 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:11:19.660 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:19.660 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:19.660 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:19.660 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:11:19.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:19.660 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:11:19.660 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:11:19.661 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:11:19.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:19.661 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:11:19Z" 2024-04-12T04:11:19.661 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:11:19.662 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:11:19Z" } 2024-04-12T04:11:19.662 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:11:19.662 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:19.662 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:19.662 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:11:24.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:24.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:24.664 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:11:24.723 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:24.724 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:24.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:24.777+0000 7f3771a3f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:24.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:24.778+0000 7f3771a3f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:24.778 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:11:24.778 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:11:24.778 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:11:24.778 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:11:24.778 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:11:24.779 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:11:24.779 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:24.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:24.779 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:11:24.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:24.779 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:11:24.779 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:24.779 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:24.780 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:11:24.780 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:24.780 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:11:24.780 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:24.780 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:11:24.780 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:24.780 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:24.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:24.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:24.780 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:11:24.780 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:24.781 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:24.781 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:11:24.781 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:24.781 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:24.781 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:24.781 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:11:24.781 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:24.781 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:24.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:24.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:24.782 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:11:24.782 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:24.782 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:24.782 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:11:24.782 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:24.782 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:24.782 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:24.782 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:11:24.782 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:24.782 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:24.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:24.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:24.783 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:11:24.783 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:24.783 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:24.783 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:11:24.783 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:24.783 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:24.783 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:24.783 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:11:24.783 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:24.783 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:24.783 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:24.784 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:11:24.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:24.784 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:11:24.784 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:11:24.784 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:11:24.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:24.784 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:11:24Z" 2024-04-12T04:11:24.784 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:11:24.785 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:11:24Z" } 2024-04-12T04:11:24.786 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:11:24.786 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:24.786 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:24.786 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:11:29.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:29.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:29.787 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:11:29.847 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:29.847 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:29.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:29.900+0000 7fe787d35780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:29.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:29.901+0000 7fe787d35780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:29.901 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:11:29.901 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:11:29.901 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:11:29.901 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:11:29.901 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:11:29.901 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:11:29.901 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:29.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:29.901 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:11:29.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:29.902 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:11:29.902 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:29.902 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:29.902 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:11:29.902 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:29.902 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:11:29.902 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:29.902 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:11:29.902 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:29.902 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:29.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:29.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:29.903 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:11:29.903 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:29.903 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:29.903 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:11:29.903 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:29.903 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:29.903 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:29.903 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:11:29.903 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:29.903 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:29.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:29.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:29.904 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:11:29.904 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:29.904 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:29.904 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:11:29.904 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:29.904 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:29.904 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:29.904 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:11:29.904 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:29.904 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:29.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:29.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:29.905 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:11:29.905 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:29.905 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:29.905 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:11:29.905 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:29.905 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:29.905 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:29.905 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:11:29.905 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:29.905 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:29.905 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:29.906 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:11:29.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:29.906 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:11:29.906 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:11:29.906 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:11:29.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:29.906 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:11:29Z" 2024-04-12T04:11:29.906 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:11:29.907 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:11:29Z" } 2024-04-12T04:11:29.907 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:11:29.908 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:29.908 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:29.908 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:11:34.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:34.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:34.909 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:11:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:35.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:35.023+0000 7f2abf683780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:35.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:35.024+0000 7f2abf683780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:35.023 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:11:35.024 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:11:35.024 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:11:35.024 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:11:35.024 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:11:35.024 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:11:35.024 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:35.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:35.024 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:11:35.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:35.024 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:11:35.024 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:35.025 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:35.025 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:11:35.025 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:35.025 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:11:35.025 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:35.025 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:11:35.025 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:35.025 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:35.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:35.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:35.025 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:11:35.026 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:35.026 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:35.026 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:11:35.026 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:35.026 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:35.026 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:35.026 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:11:35.026 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:35.026 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:35.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:35.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:35.027 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:11:35.027 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:35.027 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:35.027 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:11:35.027 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:35.027 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:35.027 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:35.027 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:11:35.027 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:35.027 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:35.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:35.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:35.028 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:11:35.028 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:35.028 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:35.028 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:11:35.028 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:35.028 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:35.028 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:35.028 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:11:35.028 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:35.028 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:35.028 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:35.028 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:11:35.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:35.029 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:11:35.029 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:11:35.029 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:11:35.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:35.029 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:11:35Z" 2024-04-12T04:11:35.029 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:11:35.030 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:11:35Z" } 2024-04-12T04:11:35.030 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:11:35.031 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:35.031 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:35.031 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:11:40.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:40.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:40.032 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:11:40.092 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:40.092 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:40.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:40.145+0000 7fb832271780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:40.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:40.146+0000 7fb832271780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:40.145 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:11:40.145 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:11:40.145 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:11:40.145 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:11:40.145 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:11:40.145 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:11:40.145 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:40.145 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:40.146 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:11:40.146 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:40.146 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:11:40.146 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:40.146 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:40.146 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:11:40.146 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:40.146 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:11:40.147 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:40.147 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:11:40.147 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:40.147 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:40.147 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:40.147 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:40.147 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:11:40.147 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:40.147 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:40.147 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:11:40.148 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:40.148 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:40.148 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:40.148 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:11:40.148 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:40.148 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:40.148 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:40.148 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:40.148 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:11:40.149 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:40.149 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:40.149 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:11:40.149 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:40.149 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:40.149 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:40.149 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:11:40.149 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:40.149 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:40.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:40.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:40.150 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:11:40.150 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:40.150 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:40.150 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:11:40.150 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:40.150 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:40.151 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:40.151 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:11:40.151 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:40.151 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:40.151 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:40.151 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:11:40.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:40.151 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:11:40.151 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:11:40.151 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:11:40.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:40.152 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:11:40Z" 2024-04-12T04:11:40.152 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:11:40.153 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:11:40Z" } 2024-04-12T04:11:40.153 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:11:40.153 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:40.153 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:40.153 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:11:45.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:45.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:45.155 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:11:45.216 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:45.216 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:45.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:45.271+0000 7fe39fe16780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:45.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:45.272+0000 7fe39fe16780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:45.269 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:11:45.270 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:11:45.270 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:11:45.270 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:11:45.270 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:11:45.270 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:11:45.270 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:45.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:45.270 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:11:45.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:45.271 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:11:45.271 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:45.271 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:45.271 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:11:45.271 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:45.271 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:11:45.271 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:45.271 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:11:45.271 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:45.271 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:45.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:45.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:45.272 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:11:45.272 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:45.272 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:45.272 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:11:45.272 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:45.272 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:45.272 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:45.272 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:11:45.272 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:45.273 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:45.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:45.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:45.273 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:11:45.273 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:45.273 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:45.273 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:11:45.273 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:45.273 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:45.273 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:45.273 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:11:45.274 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:45.274 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:45.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:45.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:45.274 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:11:45.274 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:45.274 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:45.274 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:11:45.274 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:45.274 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:45.275 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:45.275 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:11:45.275 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:45.275 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:45.275 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:45.275 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:11:45.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:45.275 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:11:45.275 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:11:45.275 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:11:45.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:45.275 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:11:45Z" 2024-04-12T04:11:45.276 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:11:45.276 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:11:45Z" } 2024-04-12T04:11:45.277 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:11:45.277 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:45.277 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:45.277 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:11:50.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:50.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:50.279 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:11:50.339 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:50.339 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:50.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:50.394+0000 7f4548249780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:50.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:50.395+0000 7f4548249780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:50.396 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:11:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:11:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:11:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:11:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:11:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:11:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:11:50.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:50.397 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:11:50.397 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:50.397 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:50.397 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:11:50.397 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:50.397 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:11:50.397 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:50.397 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:11:50.397 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:50.397 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:50.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:50.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:50.398 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:11:50.398 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:50.398 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:50.398 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:11:50.398 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:50.398 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:50.398 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:50.398 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:11:50.398 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:50.398 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:50.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:50.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:50.399 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:11:50.399 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:50.399 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:50.399 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:11:50.399 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:50.399 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:50.399 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:50.399 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:11:50.399 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:50.399 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:50.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:50.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:50.400 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:11:50.400 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:50.400 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:50.400 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:11:50.400 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:50.400 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:50.400 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:50.400 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:11:50.400 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:50.400 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:50.401 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:50.401 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:11:50.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:50.401 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:11:50.401 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:11:50.401 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:11:50.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:50.401 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:11:50Z" 2024-04-12T04:11:50.401 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:11:50.402 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:11:50Z" } 2024-04-12T04:11:50.402 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:11:50.402 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:50.402 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:50.403 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:11:55.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:55.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:11:55.404 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:11:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:11:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:11:55.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:55.515+0000 7f66906fe780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:11:55.516+0000 7f66906fe780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:11:55.515 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:11:55.515 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:11:55.515 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:11:55.515 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:11:55.515 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:11:55.516 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:11:55.516 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:55.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:55.516 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:11:55.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:55.516 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:11:55.516 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:55.516 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:55.516 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:11:55.517 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:55.517 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:11:55.517 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:55.517 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:11:55.517 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:55.517 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:55.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:55.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:55.517 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:11:55.518 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:55.518 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:55.518 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:11:55.518 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:55.518 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:55.518 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:55.518 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:11:55.518 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:55.518 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:11:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:11:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:55.519 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:11:55.520 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:55.520 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:55.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:55.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:11:55.520 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:11:55.520 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:11:55.520 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:11:55.520 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:11:55.520 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:11:55.520 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:11:55.521 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:11:55.521 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:11:55.521 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:11:55.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:55.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:11:55.521 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:11:55.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:55.521 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:11:55.521 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:11:55.521 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:11:55.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:11:55.522 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:11:55Z" 2024-04-12T04:11:55.522 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:11:55.523 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:11:55Z" } 2024-04-12T04:11:55.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:11:55.523 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:55.523 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:11:55.523 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:12:00.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:00.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:00.525 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:12:00.584 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:12:00.584 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:12:00.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:00.636+0000 7f53b9ca8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:00.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:00.637+0000 7f53b9ca8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:00.636 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:12:00.636 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:12:00.636 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:12:00.637 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:12:00.637 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:12:00.637 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:12:00.637 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:00.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:00.637 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:12:00.637 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:00.637 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:12:00.637 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:00.637 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:00.637 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:12:00.637 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:00.638 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:12:00.638 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:00.638 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:12:00.638 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:00.638 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:00.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:00.638 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:00.638 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:12:00.638 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:00.638 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:00.639 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:12:00.639 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:00.639 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:00.639 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:00.639 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:12:00.639 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:00.639 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:00.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:00.639 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:00.639 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:12:00.639 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:00.639 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:00.640 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:12:00.640 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:00.640 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:00.640 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:00.640 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:12:00.640 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:00.640 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:00.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:00.640 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:00.640 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:12:00.640 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:00.641 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:00.641 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:12:00.641 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:00.641 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:00.641 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:00.641 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:12:00.641 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:00.641 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:00.641 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:00.641 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:12:00.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:00.641 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:12:00.642 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:12:00.642 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:12:00.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:00.642 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:12:00Z" 2024-04-12T04:12:00.642 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:12:00.643 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:12:00Z" } 2024-04-12T04:12:00.643 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:12:00.643 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:00.643 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:00.643 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:12:05.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:05.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:05.645 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:12:05.704 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:12:05.704 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:12:05.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:05.757+0000 7f6917da3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:05.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:05.758+0000 7f6917da3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:05.757 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:12:05.757 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:12:05.757 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:12:05.757 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:12:05.757 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:12:05.758 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:12:05.758 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:05.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:05.758 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:12:05.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:05.758 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:12:05.758 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:05.758 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:05.758 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:12:05.758 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:05.759 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:12:05.759 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:05.759 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:12:05.759 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:05.759 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:05.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:05.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:05.759 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:12:05.759 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:05.759 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:05.759 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:12:05.759 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:05.760 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:05.760 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:05.760 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:12:05.760 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:05.760 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:05.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:05.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:05.760 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:12:05.760 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:05.760 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:05.760 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:12:05.760 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:05.761 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:05.761 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:05.761 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:12:05.761 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:05.761 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:05.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:05.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:05.761 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:12:05.761 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:05.761 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:05.761 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:12:05.761 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:05.762 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:05.762 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:05.762 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:12:05.762 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:05.762 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:05.762 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:05.762 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:12:05.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:05.762 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:12:05.762 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:12:05.762 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:12:05.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:05.763 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:12:05Z" 2024-04-12T04:12:05.763 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:12:05.763 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:12:05Z" } 2024-04-12T04:12:05.764 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:12:05.764 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:05.764 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:05.764 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:12:10.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:10.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:10.765 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:12:10.824 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:12:10.825 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:12:10.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:10.877+0000 7f5ef84af780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:10.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:10.878+0000 7f5ef84af780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:10.877 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:12:10.877 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:12:10.877 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:12:10.877 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:12:10.877 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:12:10.878 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:12:10.878 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:10.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:10.878 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:12:10.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:10.878 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:12:10.878 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:10.878 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:10.878 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:12:10.878 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:10.879 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:12:10.879 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:10.879 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:12:10.879 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:10.879 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:10.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:10.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:10.879 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:12:10.879 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:10.879 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:10.879 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:12:10.879 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:10.879 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:10.880 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:10.880 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:12:10.880 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:10.880 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:10.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:10.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:10.880 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:12:10.880 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:10.880 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:10.880 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:12:10.880 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:10.880 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:10.881 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:10.881 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:12:10.881 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:10.881 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:10.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:10.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:10.881 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:12:10.881 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:10.881 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:10.881 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:12:10.881 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:10.882 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:10.882 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:10.882 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:12:10.882 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:10.882 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:10.882 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:10.882 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:12:10.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:10.882 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:12:10.882 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:12:10.883 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:12:10.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:10.883 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:12:10Z" 2024-04-12T04:12:10.883 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:12:10.884 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:12:10Z" } 2024-04-12T04:12:10.884 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:12:10.884 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:10.884 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:10.884 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:12:15.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:15.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:15.886 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:12:15.947 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:12:15.947 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:12:15.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:16.001+0000 7f17f4100780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:15.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:16.002+0000 7f17f4100780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:16.001 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:12:16.001 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:12:16.001 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:12:16.001 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:12:16.001 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:12:16.001 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:12:16.001 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:16.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:16.002 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:12:16.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:16.002 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:12:16.002 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:16.002 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:16.002 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:12:16.002 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:16.002 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:12:16.002 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:16.002 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:12:16.002 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:16.002 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:16.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:16.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:16.003 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:12:16.003 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:16.003 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:16.003 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:12:16.003 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:16.003 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:16.003 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:16.003 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:12:16.003 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:16.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:16.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:16.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:16.004 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:12:16.004 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:16.004 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:16.004 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:12:16.004 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:16.004 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:16.004 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:16.004 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:12:16.004 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:16.004 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:16.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:16.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:16.005 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:12:16.005 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:16.005 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:16.005 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:12:16.005 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:16.005 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:16.005 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:16.005 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:12:16.005 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:16.005 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:16.005 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:16.006 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:12:16.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:16.006 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:12:16.006 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:12:16.006 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:12:16.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:16.006 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:12:16Z" 2024-04-12T04:12:16.006 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:12:16.007 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:12:16Z" } 2024-04-12T04:12:16.007 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:12:16.007 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:16.007 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:16.007 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:12:21.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:21.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:21.009 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:12:21.070 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:12:21.070 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:12:21.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:21.125+0000 7f0ab1160780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:21.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:21.128+0000 7f0ab1160780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:21.128 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:12:21.128 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:12:21.128 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:12:21.129 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:12:21.129 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:12:21.129 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:12:21.129 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:21.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:21.129 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:12:21.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:21.129 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:12:21.129 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:21.129 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:21.129 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:12:21.130 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:21.130 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:12:21.130 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:21.130 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:12:21.130 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:21.130 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:21.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:21.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:21.130 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:12:21.130 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:21.131 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:21.131 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:12:21.131 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:21.131 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:21.131 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:21.131 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:12:21.131 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:21.131 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:21.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:21.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:21.131 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:12:21.131 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:21.132 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:21.132 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:12:21.132 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:21.132 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:21.132 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:21.132 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:12:21.132 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:21.132 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:21.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:21.132 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:21.132 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:12:21.132 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:21.133 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:21.133 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:12:21.133 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:21.133 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:21.133 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:21.133 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:12:21.133 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:21.133 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:21.133 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:21.133 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:12:21.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:21.134 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:12:21.134 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:12:21.134 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:12:21.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:21.134 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:12:21Z" 2024-04-12T04:12:21.134 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:12:21.135 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:12:21Z" } 2024-04-12T04:12:21.135 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:12:21.135 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:21.135 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:21.136 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:12:26.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:26.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:26.137 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:12:26.196 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:12:26.196 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:12:26.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:26.253+0000 7fb209448780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:26.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:26.254+0000 7fb209448780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:26.253 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:12:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:12:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:12:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:12:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:12:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:12:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:26.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:12:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:12:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:12:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:12:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:12:26.254 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:12:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:12:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:12:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:12:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:12:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:12:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:12:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:12:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:12:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:12:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:12:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:12:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:12:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:12:26Z" 2024-04-12T04:12:26.258 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:12:26.259 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:12:26Z" } 2024-04-12T04:12:26.259 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:12:26.260 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:26.260 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:26.260 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:12:31.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:31.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:31.262 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:12:31.322 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:12:31.322 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:12:31.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:31.375+0000 7f014fd8b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:31.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:31.376+0000 7f014fd8b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:31.375 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:12:31.375 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:12:31.375 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:12:31.375 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:12:31.375 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:12:31.375 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:12:31.375 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:31.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:31.376 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:12:31.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:31.376 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:12:31.376 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:31.376 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:31.376 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:12:31.376 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:31.376 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:12:31.376 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:31.376 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:12:31.376 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:31.376 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:31.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:31.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:31.377 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:12:31.377 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:31.377 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:31.377 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:12:31.377 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:31.377 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:31.377 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:31.377 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:12:31.377 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:31.377 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:12:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:12:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:31.378 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:12:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:12:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:12:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:31.379 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:12:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:12:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:12:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:12:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:12:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:31.380 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:12:31Z" 2024-04-12T04:12:31.381 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:12:31.381 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:12:31Z" } 2024-04-12T04:12:31.382 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:12:31.382 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:31.382 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:31.382 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:12:36.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:36.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:36.384 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:12:36.444 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:12:36.444 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:12:36.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:36.497+0000 7fb821b2b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:36.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:36.498+0000 7fb821b2b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:36.497 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:12:36.497 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:12:36.497 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:12:36.497 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:12:36.497 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:12:36.497 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:12:36.497 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:36.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:36.497 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:12:36.498 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:36.498 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:12:36.498 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:36.498 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:36.498 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:12:36.498 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:36.498 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:12:36.498 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:36.498 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:12:36.498 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:36.498 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:36.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:36.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:36.499 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:12:36.499 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:36.499 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:36.499 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:12:36.499 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:36.499 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:36.499 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:36.499 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:12:36.500 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:36.500 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:36.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:36.500 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:36.500 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:12:36.500 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:36.500 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:36.500 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:12:36.500 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:36.500 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:36.500 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:36.500 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:12:36.501 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:36.501 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:36.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:36.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:36.501 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:12:36.501 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:36.501 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:36.501 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:12:36.501 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:36.501 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:36.502 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:36.502 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:12:36.502 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:36.502 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:36.502 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:36.502 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:12:36.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:36.502 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:12:36.502 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:12:36.502 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:12:36.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:36.503 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:12:36Z" 2024-04-12T04:12:36.503 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:12:36.504 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:12:36Z" } 2024-04-12T04:12:36.504 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:12:36.504 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:36.504 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:36.504 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:12:41.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:41.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:41.506 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:12:41.566 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:12:41.566 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:12:41.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:41.619+0000 7fa719d50780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:41.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:41.620+0000 7fa719d50780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:41.619 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:12:41.619 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:12:41.619 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:12:41.619 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:12:41.620 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:12:41.620 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:12:41.620 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:41.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:41.620 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:12:41.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:41.620 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:12:41.620 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:41.620 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:41.620 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:12:41.620 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:41.621 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:12:41.621 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:41.621 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:12:41.621 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:41.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:41.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:41.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:41.621 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:12:41.621 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:41.621 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:41.621 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:12:41.622 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:41.622 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:41.622 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:41.622 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:12:41.622 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:41.622 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:41.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:41.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:41.622 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:12:41.622 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:41.622 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:41.623 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:12:41.623 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:41.623 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:41.623 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:41.623 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:12:41.623 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:41.623 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:41.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:41.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:41.623 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:12:41.623 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:41.623 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:41.624 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:12:41.624 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:41.624 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:41.624 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:41.624 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:12:41.624 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:41.624 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:41.624 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:41.624 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:12:41.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:41.624 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:12:41.625 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:12:41.625 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:12:41.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:41.625 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:12:41Z" 2024-04-12T04:12:41.625 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:12:41.626 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:12:41Z" } 2024-04-12T04:12:41.626 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:12:41.626 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:41.626 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:41.626 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:12:46.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:46.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:46.628 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:12:46.687 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:12:46.687 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:12:46.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:46.740+0000 7f6a7eeb3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:46.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:46.741+0000 7f6a7eeb3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:46.740 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:12:46.740 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:12:46.740 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:12:46.740 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:12:46.740 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:12:46.740 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:12:46.740 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:46.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:46.741 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:12:46.741 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:46.741 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:12:46.741 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:46.741 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:46.741 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:12:46.741 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:46.741 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:12:46.741 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:46.741 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:12:46.742 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:46.742 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:46.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:46.742 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:46.742 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:12:46.742 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:46.742 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:46.742 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "", 2024-04-12T04:12:46.742 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:46.742 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:46.742 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:46.743 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "0.000000", 2024-04-12T04:12:46.743 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:46.743 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:46.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:46.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:46.743 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:12:46.743 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:46.743 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:46.743 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:12:46.743 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:46.743 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:46.743 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:46.744 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:12:46.744 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:46.744 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:46.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:46.744 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:46.744 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:12:46.744 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:46.744 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:46.744 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:12:46.744 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:46.744 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:46.745 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:46.745 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:12:46.745 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:46.745 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:46.745 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:46.745 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:12:46.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:46.745 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:12:46.745 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:12:46.745 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:12:46.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:46.746 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:12:46Z" 2024-04-12T04:12:46.746 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:12:46.746 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712891925.655973_112.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T03:18:45.655973Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:12:46Z" } 2024-04-12T04:12:46.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:12:46.747 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:46.747 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:46.747 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1712895069.167783_650.1 target=, shard=2 master=1_1712895069.196453_290.1 target=1_1712891925.655973_112.1 2024-04-12T04:12:51.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:51.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:51.749 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:12:51.807 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:12:51.807 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:12:51.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:51.861+0000 7fd07e2e8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:51.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:51.862+0000 7fd07e2e8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:51.860 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:12:51.860 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:12:51.860 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:12:51.860 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:12:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:12:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:12:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:12:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:12:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:51.861 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:12:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:12:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:12:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:12:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:51.862 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:12:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:12:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:12:51.863 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:12:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:12:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:51.864 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:12:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712894963.960546_645.1", 2024-04-12T04:12:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:23.960546Z", 2024-04-12T04:12:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:12:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:51.865 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:12:51.866 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:12:51.866 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:12:51.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:51.866 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:12:51Z" 2024-04-12T04:12:51.866 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:12:51.867 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712894963.960546_645.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:23.960546Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:12:51Z" } 2024-04-12T04:12:51.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712894963.960546_645.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:23.960546Z', 'realm_epoch': 2}}] 2024-04-12T04:12:51.867 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:51.867 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712894963.960546_645.1'} 2024-04-12T04:12:51.867 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:12:51.868 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-dkyges-1 zones={a2, a1} 2024-04-12T04:12:51.868 DEBUG:boto:Method: POST 2024-04-12T04:12:51.868 DEBUG:boto:Path: / 2024-04-12T04:12:51.868 DEBUG:boto:Data: 2024-04-12T04:12:51.868 DEBUG:boto:Headers: {} 2024-04-12T04:12:51.868 DEBUG:boto:Host: smithi043.front.sepia.ceph.com 2024-04-12T04:12:51.868 DEBUG:boto:Port: 8001 2024-04-12T04:12:51.868 DEBUG:boto:Params: {'RoleName': 'roles-dkyges-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-12T04:12:51.869 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:12:51.869 DEBUG:boto:Token: None 2024-04-12T04:12:51.869 DEBUG:boto:CanonicalRequest: POST / host:smithi043.front.sepia.ceph.com:8001 x-amz-date:20240412T041251Z host;x-amz-date c100f692e9fc88c95212a053dd8a2c72a687cc38c933604f3a6a28088324a8df 2024-04-12T04:12:51.870 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240412T041251Z 20240412/front/smithi043/aws4_request 07d4d1900dc0be21bfcbcb0502f91eeacc2793807a87715bbe36354e9a9701a4 2024-04-12T04:12:51.870 DEBUG:boto:Signature: 573cb5540f61776bfcd2358d656457cf37cb76a546fd94d195b2f530a8b4306f 2024-04-12T04:12:51.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240412T041251Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=KUXF4P9H15V1YOGZBQF6/20240412/front/smithi043/aws4_request,SignedHeaders=host;x-amz-date,Signature=573cb5540f61776bfcd2358d656457cf37cb76a546fd94d195b2f530a8b4306f', 'Host': 'smithi043.front.sepia.ceph.com:8001'} 2024-04-12T04:12:51.882 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000005dd54dbf4676342-006618b4c3-1fa3-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Fri, 12 Apr 2024 04:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:12:51.883 DEBUG:boto:b'tx0000005dd54dbf4676342-006618b4c3-1fa3-a2780605d1-c318-4389-a003-a1a8764b84beroles-dkyges-1/arn:aws:iam:::role/roles-dkyges-12024-04-12T04:12:48.294Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-12T04:12:51.883 DEBUG:boto:Method: POST 2024-04-12T04:12:51.884 DEBUG:boto:Path: / 2024-04-12T04:12:51.884 DEBUG:boto:Data: 2024-04-12T04:12:51.884 DEBUG:boto:Headers: {} 2024-04-12T04:12:51.884 DEBUG:boto:Host: smithi043.front.sepia.ceph.com 2024-04-12T04:12:51.884 DEBUG:boto:Port: 8000 2024-04-12T04:12:51.884 DEBUG:boto:Params: {'RoleName': 'roles-dkyges-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-12T04:12:51.884 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:12:51.884 DEBUG:boto:Token: None 2024-04-12T04:12:51.885 DEBUG:boto:CanonicalRequest: POST / host:smithi043.front.sepia.ceph.com:8000 x-amz-date:20240412T041251Z host;x-amz-date c100f692e9fc88c95212a053dd8a2c72a687cc38c933604f3a6a28088324a8df 2024-04-12T04:12:51.885 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240412T041251Z 20240412/front/smithi043/aws4_request 374793e5a8a3a8fa099be6af69440179b2c9a854014203926b765c5481b1af8f 2024-04-12T04:12:51.885 DEBUG:boto:Signature: ec0edb849d62484e82aa74e2c1edaa46fe919bac0e9abe244ca783cf69ad0019 2024-04-12T04:12:51.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240412T041251Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=KUXF4P9H15V1YOGZBQF6/20240412/front/smithi043/aws4_request,SignedHeaders=host;x-amz-date,Signature=ec0edb849d62484e82aa74e2c1edaa46fe919bac0e9abe244ca783cf69ad0019', 'Host': 'smithi043.front.sepia.ceph.com:8000'} 2024-04-12T04:12:51.897 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000034c30bf70f8193d0-006618b4c3-16d5-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Fri, 12 Apr 2024 04:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:12:51.898 DEBUG:boto:b'tx0000034c30bf70f8193d0-006618b4c3-16d5-a1780605d1-c318-4389-a003-a1a8764b84beroles-dkyges-1/arn:aws:iam:::role/roles-dkyges-12024-04-12T04:11:09.163Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-12T04:12:51.899 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-dkyges-1 2024-04-12T04:12:51.899 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-dkyges-1 zones={a2, a1} 2024-04-12T04:12:51.899 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-dkyges-2 zones={a1, a2} 2024-04-12T04:12:51.899 DEBUG:boto:Method: POST 2024-04-12T04:12:51.900 DEBUG:boto:Path: / 2024-04-12T04:12:51.900 DEBUG:boto:Data: 2024-04-12T04:12:51.900 DEBUG:boto:Headers: {} 2024-04-12T04:12:51.900 DEBUG:boto:Host: smithi043.front.sepia.ceph.com 2024-04-12T04:12:51.900 DEBUG:boto:Port: 8000 2024-04-12T04:12:51.900 DEBUG:boto:Params: {'RoleName': 'roles-dkyges-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-12T04:12:51.901 DEBUG:boto:Token: None 2024-04-12T04:12:51.901 DEBUG:boto:CanonicalRequest: POST / host:smithi043.front.sepia.ceph.com:8000 x-amz-date:20240412T041251Z host;x-amz-date fa02618127acd3858bd99254c1cf4683257843e50fbeb734edcbc32e8101543a 2024-04-12T04:12:51.901 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240412T041251Z 20240412/front/smithi043/aws4_request b85cd92575e39f94953219e18ef41d832cbd676c73a52dcef19672fceccc1732 2024-04-12T04:12:51.902 DEBUG:boto:Signature: 9b44175a6918922d27352c11446a121f27034f27e461f21a02a20b9c87da2540 2024-04-12T04:12:51.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240412T041251Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=KUXF4P9H15V1YOGZBQF6/20240412/front/smithi043/aws4_request,SignedHeaders=host;x-amz-date,Signature=9b44175a6918922d27352c11446a121f27034f27e461f21a02a20b9c87da2540', 'Host': 'smithi043.front.sepia.ceph.com:8000'} 2024-04-12T04:12:51.903 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cd67446ab7dfa976-006618b4c3-16d5-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Fri, 12 Apr 2024 04:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:12:51.903 DEBUG:boto:b'tx00000cd67446ab7dfa976-006618b4c3-16d5-a1e00f8139-fc24-4085-bd42-b3d54299d876roles-dkyges-2/arn:aws:iam:::role/roles-dkyges-22024-04-12T04:11:09.190Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-12T04:12:51.903 DEBUG:boto:Method: POST 2024-04-12T04:12:51.904 DEBUG:boto:Path: / 2024-04-12T04:12:51.904 DEBUG:boto:Data: 2024-04-12T04:12:51.904 DEBUG:boto:Headers: {} 2024-04-12T04:12:51.904 DEBUG:boto:Host: smithi043.front.sepia.ceph.com 2024-04-12T04:12:51.904 DEBUG:boto:Port: 8001 2024-04-12T04:12:51.904 DEBUG:boto:Params: {'RoleName': 'roles-dkyges-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2024-04-12T04:12:51.904 DEBUG:boto:Token: None 2024-04-12T04:12:51.904 DEBUG:boto:CanonicalRequest: POST / host:smithi043.front.sepia.ceph.com:8001 x-amz-date:20240412T041251Z host;x-amz-date fa02618127acd3858bd99254c1cf4683257843e50fbeb734edcbc32e8101543a 2024-04-12T04:12:51.904 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20240412T041251Z 20240412/front/smithi043/aws4_request 86ebe78b1727a01f5c5ed1d725467422924d65eb81b1d08399a1786f7972c6bd 2024-04-12T04:12:51.905 DEBUG:boto:Signature: 9b870172bdd759f33dc572a084172041b65ed8e7a36be10215995e1c4077ad43 2024-04-12T04:12:51.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'X-Amz-Date': '20240412T041251Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=KUXF4P9H15V1YOGZBQF6/20240412/front/smithi043/aws4_request,SignedHeaders=host;x-amz-date,Signature=9b870172bdd759f33dc572a084172041b65ed8e7a36be10215995e1c4077ad43', 'Host': 'smithi043.front.sepia.ceph.com:8001'} 2024-04-12T04:12:51.906 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f6df50b1a2dbf72b-006618b4c3-1fa3-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Fri, 12 Apr 2024 04:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:12:51.906 DEBUG:boto:b'tx00000f6df50b1a2dbf72b-006618b4c3-1fa3-a2e00f8139-fc24-4085-bd42-b3d54299d876roles-dkyges-2/arn:aws:iam:::role/roles-dkyges-22024-04-12T04:11:09.190Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2024-04-12T04:12:51.907 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-dkyges-2 2024-04-12T04:12:51.907 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-dkyges-2 zones={a1, a2} 2024-04-12T04:12:51.908 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2024-04-12T04:12:51.909 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:12:51.909 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:12:51.909 DEBUG:boto:Method: PUT 2024-04-12T04:12:51.909 DEBUG:boto:Path: /dkyges-62/ 2024-04-12T04:12:51.909 DEBUG:boto:Data: 2024-04-12T04:12:51.910 DEBUG:boto:Headers: {} 2024-04-12T04:12:51.910 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:12:51.910 DEBUG:boto:Port: 8000 2024-04-12T04:12:51.910 DEBUG:boto:Params: {} 2024-04-12T04:12:51.910 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:12:51.910 DEBUG:boto:Token: None 2024-04-12T04:12:51.910 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:12:51 GMT /dkyges-62/ 2024-04-12T04:12:51.911 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2UMTUqEiQNEXSsALOgKOkNz/N48= 2024-04-12T04:12:51.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:12:51 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2UMTUqEiQNEXSsALOgKOkNz/N48=', 'Content-Length': '0'} 2024-04-12T04:12:51.941 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000093be03d0f0e37080-006618b4c3-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:12:51 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:12:51.941 DEBUG:rgw_multi.tests:created bucket=dkyges-62 2024-04-12T04:12:51.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:51.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:51.942 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:12:51.998 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:12:51.998 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:12:52.034 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:12:52.050 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:12:52.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:52.050 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:12:52.050 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:12:52.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:52.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:52.050 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:12:52.050 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:12:52.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:12:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:12:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895171.957216_660.1", 2024-04-12T04:12:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:12:51.957216Z" 2024-04-12T04:12:52.051 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:52.051 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:12:52.052 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895171.957216_660.1'} 2024-04-12T04:12:52.052 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:12:52.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:52.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:52.053 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:12:52.108 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:12:52.108 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:12:52.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:52.160+0000 7f92b7e2e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:52.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:52.161+0000 7f92b7e2e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:52.160 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:12:52.160 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:12:52.160 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:12:52.160 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:12:52.160 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:12:52.160 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:12:52.160 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:52.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:52.160 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:12:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:12:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:12:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:12:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:12:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:52.161 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:12:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:12:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:12:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:52.162 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:12:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:12:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:12:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:52.163 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:12:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895171.948572_657.1", 2024-04-12T04:12:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:12:51.948572Z", 2024-04-12T04:12:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:52.164 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:52.165 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:52.165 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:12:52.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:52.165 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:12:52.165 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:12:52.165 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:12:52.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:52.165 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:12:52Z" 2024-04-12T04:12:52.165 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:12:52.166 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895171.948572_657.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:12:51.948572Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:12:52Z" } 2024-04-12T04:12:52.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895171.948572_657.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:12:51.948572Z', 'realm_epoch': 2}}] 2024-04-12T04:12:52.166 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895171.957216_660.1'} 2024-04-12T04:12:52.167 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895171.948572_657.1'} 2024-04-12T04:12:52.167 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895171.957216_660.1 target=1_1712895171.948572_657.1 2024-04-12T04:12:57.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:57.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:12:57.168 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:12:57.229 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:12:57.229 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:12:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:57.285+0000 7fe5b1445780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:12:57.286+0000 7fe5b1445780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:12:57.284 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:12:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:12:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:12:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:12:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:12:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:12:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:12:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:12:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:12:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:12:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:12:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:12:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:12:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:12:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:12:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:12:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:12:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:12:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:12:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:12:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:12:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895171.957216_660.1", 2024-04-12T04:12:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:12:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:12:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:12:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:12:51.957216Z", 2024-04-12T04:12:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:12:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:12:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:12:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:12:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:12:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:12:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:12:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:12:57Z" 2024-04-12T04:12:57.290 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:12:57.291 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895171.957216_660.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:12:51.957216Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:12:57Z" } 2024-04-12T04:12:57.291 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895171.957216_660.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:12:51.957216Z', 'realm_epoch': 2}}] 2024-04-12T04:12:57.291 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895171.957216_660.1'} 2024-04-12T04:12:57.291 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895171.957216_660.1'} 2024-04-12T04:12:57.291 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:12:57.292 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T04:12:57.292 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:12:57.293 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T04:12:57.310+0000 7fe7705ac700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 101038) UID: 0 2024-04-12T04:12:57.293 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T04:12:57.310+0000 7fe7757817c0 -1 shutting down 2024-04-12T04:13:01.661 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T04:13:03.393 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T04:13:03.394 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:13:03.394 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:13:03.394 DEBUG:boto:Method: HEAD 2024-04-12T04:13:03.394 DEBUG:boto:Path: /dkyges-62/ 2024-04-12T04:13:03.394 DEBUG:boto:Data: 2024-04-12T04:13:03.394 DEBUG:boto:Headers: {} 2024-04-12T04:13:03.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:13:03.395 DEBUG:boto:Port: 8000 2024-04-12T04:13:03.395 DEBUG:boto:Params: {} 2024-04-12T04:13:03.395 DEBUG:boto:Token: None 2024-04-12T04:13:03.395 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:13:03 GMT /dkyges-62/ 2024-04-12T04:13:03.395 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:N/1dxNtRf/wpTpAcXkRIDA7mCMM= 2024-04-12T04:13:03.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:13:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:N/1dxNtRf/wpTpAcXkRIDA7mCMM=', 'Content-Length': '0'} 2024-04-12T04:13:03.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f94548a2824c895-006618b4cf-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:13:03.399 DEBUG:boto:path=/dkyges-62/obj11 2024-04-12T04:13:03.399 DEBUG:boto:auth_path=/dkyges-62/obj11 2024-04-12T04:13:03.399 DEBUG:boto:Method: PUT 2024-04-12T04:13:03.399 DEBUG:boto:Path: /dkyges-62/obj11 2024-04-12T04:13:03.399 DEBUG:boto:Data: 2024-04-12T04:13:03.399 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:13:03.399 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:13:03.399 DEBUG:boto:Port: 8000 2024-04-12T04:13:03.399 DEBUG:boto:Params: {} 2024-04-12T04:13:03.400 DEBUG:boto:Token: None 2024-04-12T04:13:03.400 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:13:03 GMT /dkyges-62/obj11 2024-04-12T04:13:03.400 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Fmc97C+HT4rSF42E3Q6tBFKUlXE= 2024-04-12T04:13:03.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:13:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Fmc97C+HT4rSF42E3Q6tBFKUlXE='} 2024-04-12T04:13:03.408 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000034e73bd89e77cfa4-006618b4cf-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:13:03.408 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:13:03.408 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:13:03.408 DEBUG:boto:Method: HEAD 2024-04-12T04:13:03.408 DEBUG:boto:Path: /dkyges-62/ 2024-04-12T04:13:03.409 DEBUG:boto:Data: 2024-04-12T04:13:03.409 DEBUG:boto:Headers: {} 2024-04-12T04:13:03.409 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:13:03.409 DEBUG:boto:Port: 8000 2024-04-12T04:13:03.409 DEBUG:boto:Params: {} 2024-04-12T04:13:03.409 DEBUG:boto:Token: None 2024-04-12T04:13:03.409 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:13:03 GMT /dkyges-62/ 2024-04-12T04:13:03.409 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:N/1dxNtRf/wpTpAcXkRIDA7mCMM= 2024-04-12T04:13:03.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:13:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:N/1dxNtRf/wpTpAcXkRIDA7mCMM=', 'Content-Length': '0'} 2024-04-12T04:13:03.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053a1237821bc0510-006618b4cf-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:13:03.412 DEBUG:boto:path=/dkyges-62/obj22 2024-04-12T04:13:03.412 DEBUG:boto:auth_path=/dkyges-62/obj22 2024-04-12T04:13:03.412 DEBUG:boto:Method: PUT 2024-04-12T04:13:03.412 DEBUG:boto:Path: /dkyges-62/obj22 2024-04-12T04:13:03.413 DEBUG:boto:Data: 2024-04-12T04:13:03.413 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:13:03.413 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:13:03.413 DEBUG:boto:Port: 8000 2024-04-12T04:13:03.413 DEBUG:boto:Params: {} 2024-04-12T04:13:03.413 DEBUG:boto:Token: None 2024-04-12T04:13:03.413 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:13:03 GMT /dkyges-62/obj22 2024-04-12T04:13:03.413 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DirsWmVpTc5n29WRUZflQ2mM8Ks= 2024-04-12T04:13:03.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:13:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DirsWmVpTc5n29WRUZflQ2mM8Ks='} 2024-04-12T04:13:03.418 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a88f78ff246630cc-006618b4cf-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:13:03.418 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:13:03.418 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:13:03.419 DEBUG:boto:Method: HEAD 2024-04-12T04:13:03.419 DEBUG:boto:Path: /dkyges-62/ 2024-04-12T04:13:03.419 DEBUG:boto:Data: 2024-04-12T04:13:03.419 DEBUG:boto:Headers: {} 2024-04-12T04:13:03.419 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:13:03.419 DEBUG:boto:Port: 8000 2024-04-12T04:13:03.419 DEBUG:boto:Params: {} 2024-04-12T04:13:03.419 DEBUG:boto:Token: None 2024-04-12T04:13:03.419 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:13:03 GMT /dkyges-62/ 2024-04-12T04:13:03.419 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:N/1dxNtRf/wpTpAcXkRIDA7mCMM= 2024-04-12T04:13:03.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:13:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:N/1dxNtRf/wpTpAcXkRIDA7mCMM=', 'Content-Length': '0'} 2024-04-12T04:13:03.423 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000752f4831919df340-006618b4cf-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:13:03.423 DEBUG:boto:path=/dkyges-62/obj33 2024-04-12T04:13:03.423 DEBUG:boto:auth_path=/dkyges-62/obj33 2024-04-12T04:13:03.423 DEBUG:boto:Method: PUT 2024-04-12T04:13:03.423 DEBUG:boto:Path: /dkyges-62/obj33 2024-04-12T04:13:03.423 DEBUG:boto:Data: 2024-04-12T04:13:03.424 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:13:03.424 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:13:03.424 DEBUG:boto:Port: 8000 2024-04-12T04:13:03.424 DEBUG:boto:Params: {} 2024-04-12T04:13:03.424 DEBUG:boto:Token: None 2024-04-12T04:13:03.424 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:13:03 GMT /dkyges-62/obj33 2024-04-12T04:13:03.424 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aQteLsL6s2dkwclpwzp/SujFhOY= 2024-04-12T04:13:03.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:13:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aQteLsL6s2dkwclpwzp/SujFhOY='} 2024-04-12T04:13:03.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb2b47dae04b3731-006618b4cf-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:13:03.467 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:13:03.467 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:13:03.467 DEBUG:boto:Method: HEAD 2024-04-12T04:13:03.467 DEBUG:boto:Path: /dkyges-62/ 2024-04-12T04:13:03.467 DEBUG:boto:Data: 2024-04-12T04:13:03.467 DEBUG:boto:Headers: {} 2024-04-12T04:13:03.467 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:13:03.467 DEBUG:boto:Port: 8000 2024-04-12T04:13:03.467 DEBUG:boto:Params: {} 2024-04-12T04:13:03.468 DEBUG:boto:Token: None 2024-04-12T04:13:03.468 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:13:03 GMT /dkyges-62/ 2024-04-12T04:13:03.468 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:N/1dxNtRf/wpTpAcXkRIDA7mCMM= 2024-04-12T04:13:03.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:13:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:N/1dxNtRf/wpTpAcXkRIDA7mCMM=', 'Content-Length': '0'} 2024-04-12T04:13:03.470 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000019301b6765ee3de7-006618b4cf-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:13:03.471 DEBUG:boto:path=/dkyges-62/obj44 2024-04-12T04:13:03.471 DEBUG:boto:auth_path=/dkyges-62/obj44 2024-04-12T04:13:03.471 DEBUG:boto:Method: PUT 2024-04-12T04:13:03.471 DEBUG:boto:Path: /dkyges-62/obj44 2024-04-12T04:13:03.471 DEBUG:boto:Data: 2024-04-12T04:13:03.471 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:13:03.471 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:13:03.471 DEBUG:boto:Port: 8000 2024-04-12T04:13:03.471 DEBUG:boto:Params: {} 2024-04-12T04:13:03.472 DEBUG:boto:Token: None 2024-04-12T04:13:03.472 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:13:03 GMT /dkyges-62/obj44 2024-04-12T04:13:03.472 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kxC28G6O/BAc8mCIvsbfo7eWkxw= 2024-04-12T04:13:03.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:13:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kxC28G6O/BAc8mCIvsbfo7eWkxw='} 2024-04-12T04:13:03.480 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003781c972759a8ed4-006618b4cf-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:13:03.480 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:13:03.481 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:13:03.481 DEBUG:boto:Method: HEAD 2024-04-12T04:13:03.481 DEBUG:boto:Path: /dkyges-62/ 2024-04-12T04:13:03.481 DEBUG:boto:Data: 2024-04-12T04:13:03.481 DEBUG:boto:Headers: {} 2024-04-12T04:13:03.481 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:13:03.481 DEBUG:boto:Port: 8000 2024-04-12T04:13:03.481 DEBUG:boto:Params: {} 2024-04-12T04:13:03.481 DEBUG:boto:Token: None 2024-04-12T04:13:03.481 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:13:03 GMT /dkyges-62/ 2024-04-12T04:13:03.481 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:N/1dxNtRf/wpTpAcXkRIDA7mCMM= 2024-04-12T04:13:03.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:13:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:N/1dxNtRf/wpTpAcXkRIDA7mCMM=', 'Content-Length': '0'} 2024-04-12T04:13:03.484 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000768e75ebab879c12-006618b4cf-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:13:03.484 DEBUG:boto:path=/dkyges-62/obj55 2024-04-12T04:13:03.484 DEBUG:boto:auth_path=/dkyges-62/obj55 2024-04-12T04:13:03.484 DEBUG:boto:Method: PUT 2024-04-12T04:13:03.485 DEBUG:boto:Path: /dkyges-62/obj55 2024-04-12T04:13:03.485 DEBUG:boto:Data: 2024-04-12T04:13:03.485 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:13:03.485 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:13:03.485 DEBUG:boto:Port: 8000 2024-04-12T04:13:03.485 DEBUG:boto:Params: {} 2024-04-12T04:13:03.485 DEBUG:boto:Token: None 2024-04-12T04:13:03.485 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:13:03 GMT /dkyges-62/obj55 2024-04-12T04:13:03.485 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NQh2EhkY5pMFzR1J5VTU9vmLLiw= 2024-04-12T04:13:03.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:13:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NQh2EhkY5pMFzR1J5VTU9vmLLiw='} 2024-04-12T04:13:03.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000064c1ea0ddad43ffe-006618b4cf-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:13:03.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:13:03.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:13:03.531 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:13:03.592 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:13:03.592 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:13:03.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:13:03.648+0000 7f4fc0596780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-12T04:13:03.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:13:03.650+0000 7f4fc0596780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-12T04:13:03.657 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T04:13:03.658 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T04:13:03.660 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T04:13:03.660 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T04:13:03.683 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T04:13:03.684 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:13:03.684 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T04:13:03.684 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi043.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T04:13:04.685 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T04:13:04.712 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:13:04.712 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T04:13:04.712 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T04:13:04.713 INFO:teuthology.orchestra.run.smithi043.stdout:anonymous 2024-04-12T04:13:04.713 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-62 2024-04-12T04:13:04.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:13:04.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:13:04.714 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:13:04.823 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:13:04.823 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:13:04.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:13:04.886+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:13:07.368 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T04:13:09.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:13:09.885+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:13:14.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:13:14.885+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:13:19.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:13:19.886+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:13:24.384 INFO:tasks.rgw.c1.client.0.smithi043.stdout:2024-04-12T04:13:24.401+0000 7f1e272bd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:13:24.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:13:24.887+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:13:29.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:13:29.887+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:13:34.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:13:34.888+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:13:39.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:13:39.889+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:13:44.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:13:44.889+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:13:49.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:13:49.890+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:13:54.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:13:54.891+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:13:59.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:13:59.891+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:14:04.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:14:04.892+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:14:09.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:14:09.893+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:14:14.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:14:14.894+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:14:19.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:14:19.895+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:14:24.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:14:24.894+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:14:29.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:14:29.895+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:14:34.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:14:34.896+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:14:39.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:14:39.896+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:14:44.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:14:44.897+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:14:49.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:14:49.898+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:14:54.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:14:54.898+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:14:59.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:14:59.899+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:15:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:04.900+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:15:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:09.900+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:15:14.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:14.901+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:15:19.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:19.901+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:15:24.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:24.902+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:15:29.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:29.903+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:15:34.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:34.904+0000 7f448cb15780 1 waiting to reach incremental sync.. 2024-04-12T04:15:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:39.905+0000 7f448cb15780 1 bucket sync caught up with source: 2024-04-12T04:15:39.888 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000001.65.6, 00000000002.105.6, 00000000002.83.6] 2024-04-12T04:15:39.888 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000001.65.6, 00000000002.105.6, 00000000002.83.6] 2024-04-12T04:15:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:39.905+0000 7f448cb15780 0 bucket checkpoint complete 2024-04-12T04:15:39.907 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-62 2024-04-12T04:15:39.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:15:39.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:15:39.908 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:15:39.974 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:39.974 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:40.035+0000 7f8a3f33e780 1 bucket sync caught up with source: 2024-04-12T04:15:40.017 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000001.2217.6, 00000000002.3724.6, 00000000002.2077.6] 2024-04-12T04:15:40.017 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000001.2217.6, 00000000002.3724.6, 00000000002.2077.6] 2024-04-12T04:15:40.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:40.035+0000 7f8a3f33e780 0 bucket checkpoint complete 2024-04-12T04:15:40.033 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-62 zones={a2, a1} 2024-04-12T04:15:40.034 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.034 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.034 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.034 DEBUG:boto:Path: /dkyges-62/ 2024-04-12T04:15:40.034 DEBUG:boto:Data: 2024-04-12T04:15:40.034 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.034 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.034 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.035 DEBUG:boto:Params: {} 2024-04-12T04:15:40.035 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:15:40.035 DEBUG:boto:Token: None 2024-04-12T04:15:40.036 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.036 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4QjCzcNG+rjMAgKGvWbnBTFXhM0= 2024-04-12T04:15:40.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4QjCzcNG+rjMAgKGvWbnBTFXhM0=', 'Content-Length': '0'} 2024-04-12T04:15:40.049 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000728b9f0e0ec1e1c8-006618b56c-208b-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.050 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.050 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.050 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.050 DEBUG:boto:Path: /dkyges-62/ 2024-04-12T04:15:40.050 DEBUG:boto:Data: 2024-04-12T04:15:40.050 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.051 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.051 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.051 DEBUG:boto:Params: {} 2024-04-12T04:15:40.051 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:15:40.051 DEBUG:boto:Token: None 2024-04-12T04:15:40.051 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.052 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4QjCzcNG+rjMAgKGvWbnBTFXhM0= 2024-04-12T04:15:40.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4QjCzcNG+rjMAgKGvWbnBTFXhM0=', 'Content-Length': '0'} 2024-04-12T04:15:40.064 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca644662d104200f-006618b56c-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.065 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:15:40.065 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.065 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.065 DEBUG:boto:path=/dkyges-62/?versions&max-keys=999 2024-04-12T04:15:40.065 DEBUG:boto:auth_path=/dkyges-62/?versions&max-keys=999 2024-04-12T04:15:40.065 DEBUG:boto:Method: GET 2024-04-12T04:15:40.065 DEBUG:boto:Path: /dkyges-62/?versions&max-keys=999 2024-04-12T04:15:40.065 DEBUG:boto:Data: 2024-04-12T04:15:40.065 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.066 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.066 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.066 DEBUG:boto:Params: {} 2024-04-12T04:15:40.066 DEBUG:boto:Token: None 2024-04-12T04:15:40.066 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/?versions 2024-04-12T04:15:40.066 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QbMF2GWbG4fceMT1+DnviRRrhJo= 2024-04-12T04:15:40.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QbMF2GWbG4fceMT1+DnviRRrhJo=', 'Content-Length': '0'} 2024-04-12T04:15:40.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c8a866de2d6edc9-006618b56c-208b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.069 DEBUG:boto:b'dkyges-62999falseobj11nulltrue2024-04-12T04:13:03.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:13:03.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:13:03.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:13:03.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:13:03.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:15:40.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:15:40.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:15:40.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:15:40.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:15:40.070 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:15:40.071 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.071 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.071 DEBUG:boto:path=/dkyges-62/?versions&max-keys=999 2024-04-12T04:15:40.071 DEBUG:boto:auth_path=/dkyges-62/?versions&max-keys=999 2024-04-12T04:15:40.071 DEBUG:boto:Method: GET 2024-04-12T04:15:40.071 DEBUG:boto:Path: /dkyges-62/?versions&max-keys=999 2024-04-12T04:15:40.071 DEBUG:boto:Data: 2024-04-12T04:15:40.071 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.071 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.071 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.071 DEBUG:boto:Params: {} 2024-04-12T04:15:40.072 DEBUG:boto:Token: None 2024-04-12T04:15:40.072 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/?versions 2024-04-12T04:15:40.072 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QbMF2GWbG4fceMT1+DnviRRrhJo= 2024-04-12T04:15:40.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QbMF2GWbG4fceMT1+DnviRRrhJo=', 'Content-Length': '0'} 2024-04-12T04:15:40.075 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d10e85b6dca2b2d3-006618b56c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.075 DEBUG:boto:b'dkyges-62999falseobj11nulltrue2024-04-12T04:13:03.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:13:03.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:13:03.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:13:03.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:13:03.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:15:40.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:15:40.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:15:40.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:15:40.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:15:40.076 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.076 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.076 DEBUG:boto:path=/dkyges-62/?versions&max-keys=999 2024-04-12T04:15:40.076 DEBUG:boto:auth_path=/dkyges-62/?versions&max-keys=999 2024-04-12T04:15:40.076 DEBUG:boto:Method: GET 2024-04-12T04:15:40.076 DEBUG:boto:Path: /dkyges-62/?versions&max-keys=999 2024-04-12T04:15:40.076 DEBUG:boto:Data: 2024-04-12T04:15:40.076 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.076 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.076 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.077 DEBUG:boto:Params: {} 2024-04-12T04:15:40.077 DEBUG:boto:Token: None 2024-04-12T04:15:40.077 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/?versions 2024-04-12T04:15:40.077 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QbMF2GWbG4fceMT1+DnviRRrhJo= 2024-04-12T04:15:40.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QbMF2GWbG4fceMT1+DnviRRrhJo=', 'Content-Length': '0'} 2024-04-12T04:15:40.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023433232141cbfeb-006618b56c-208b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.079 DEBUG:boto:b'dkyges-62999falseobj11nulltrue2024-04-12T04:13:03.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:13:03.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:13:03.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:13:03.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:13:03.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.080 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.080 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.080 DEBUG:boto:path=/dkyges-62/?versions&max-keys=999 2024-04-12T04:15:40.080 DEBUG:boto:auth_path=/dkyges-62/?versions&max-keys=999 2024-04-12T04:15:40.080 DEBUG:boto:Method: GET 2024-04-12T04:15:40.080 DEBUG:boto:Path: /dkyges-62/?versions&max-keys=999 2024-04-12T04:15:40.080 DEBUG:boto:Data: 2024-04-12T04:15:40.081 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.081 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.081 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.081 DEBUG:boto:Params: {} 2024-04-12T04:15:40.081 DEBUG:boto:Token: None 2024-04-12T04:15:40.081 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/?versions 2024-04-12T04:15:40.081 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QbMF2GWbG4fceMT1+DnviRRrhJo= 2024-04-12T04:15:40.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QbMF2GWbG4fceMT1+DnviRRrhJo=', 'Content-Length': '0'} 2024-04-12T04:15:40.084 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fec05a4f91d4e292-006618b56c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.084 DEBUG:boto:b'dkyges-62999falseobj11nulltrue2024-04-12T04:13:03.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:13:03.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:13:03.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:13:03.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:13:03.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:15:40.085 DEBUG:boto:path=/dkyges-62/obj11 2024-04-12T04:15:40.085 DEBUG:boto:auth_path=/dkyges-62/obj11 2024-04-12T04:15:40.085 DEBUG:boto:path=/dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.085 DEBUG:boto:auth_path=/dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.085 DEBUG:boto:Method: GET 2024-04-12T04:15:40.085 DEBUG:boto:Path: /dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.085 DEBUG:boto:Data: 2024-04-12T04:15:40.086 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.086 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.086 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.086 DEBUG:boto:Params: {} 2024-04-12T04:15:40.086 DEBUG:boto:Token: None 2024-04-12T04:15:40.086 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.086 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rNjjv6+4mv6V9gd07Xv2QSXGfqQ= 2024-04-12T04:15:40.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rNjjv6+4mv6V9gd07Xv2QSXGfqQ=', 'Content-Length': '0'} 2024-04-12T04:15:40.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb84f0a4bb94cc3b-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.089 DEBUG:boto:path=/dkyges-62/obj11 2024-04-12T04:15:40.089 DEBUG:boto:auth_path=/dkyges-62/obj11 2024-04-12T04:15:40.089 DEBUG:boto:path=/dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.089 DEBUG:boto:auth_path=/dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.089 DEBUG:boto:Method: GET 2024-04-12T04:15:40.089 DEBUG:boto:Path: /dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.089 DEBUG:boto:Data: 2024-04-12T04:15:40.090 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.090 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.090 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.090 DEBUG:boto:Params: {} 2024-04-12T04:15:40.090 DEBUG:boto:Token: None 2024-04-12T04:15:40.090 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.090 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rNjjv6+4mv6V9gd07Xv2QSXGfqQ= 2024-04-12T04:15:40.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rNjjv6+4mv6V9gd07Xv2QSXGfqQ=', 'Content-Length': '0'} 2024-04-12T04:15:40.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f4f1941e6cb6b06-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.092 DEBUG:boto:path=/dkyges-62/obj11 2024-04-12T04:15:40.092 DEBUG:boto:auth_path=/dkyges-62/obj11 2024-04-12T04:15:40.092 DEBUG:boto:path=/dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.092 DEBUG:boto:auth_path=/dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.093 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.093 DEBUG:boto:Path: /dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.093 DEBUG:boto:Data: 2024-04-12T04:15:40.093 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.093 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.093 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.093 DEBUG:boto:Params: {} 2024-04-12T04:15:40.093 DEBUG:boto:Token: None 2024-04-12T04:15:40.093 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.093 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2i1G5hgMCsGZctXJJ32uS4U32D8= 2024-04-12T04:15:40.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2i1G5hgMCsGZctXJJ32uS4U32D8=', 'Content-Length': '0'} 2024-04-12T04:15:40.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5dfe8c926daecd8-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.096 DEBUG:boto:path=/dkyges-62/obj11 2024-04-12T04:15:40.096 DEBUG:boto:auth_path=/dkyges-62/obj11 2024-04-12T04:15:40.096 DEBUG:boto:path=/dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.096 DEBUG:boto:auth_path=/dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.096 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.096 DEBUG:boto:Path: /dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.096 DEBUG:boto:Data: 2024-04-12T04:15:40.096 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.096 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.096 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.097 DEBUG:boto:Params: {} 2024-04-12T04:15:40.097 DEBUG:boto:Token: None 2024-04-12T04:15:40.097 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.097 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2i1G5hgMCsGZctXJJ32uS4U32D8= 2024-04-12T04:15:40.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2i1G5hgMCsGZctXJJ32uS4U32D8=', 'Content-Length': '0'} 2024-04-12T04:15:40.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009711d089c8df4c64-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:15:40.099 DEBUG:boto:path=/dkyges-62/obj11 2024-04-12T04:15:40.099 DEBUG:boto:auth_path=/dkyges-62/obj11 2024-04-12T04:15:40.099 DEBUG:boto:path=/dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.099 DEBUG:boto:auth_path=/dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.099 DEBUG:boto:Method: GET 2024-04-12T04:15:40.100 DEBUG:boto:Path: /dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.100 DEBUG:boto:Data: 2024-04-12T04:15:40.100 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.100 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.100 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.100 DEBUG:boto:Params: {} 2024-04-12T04:15:40.100 DEBUG:boto:Token: None 2024-04-12T04:15:40.100 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.100 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rNjjv6+4mv6V9gd07Xv2QSXGfqQ= 2024-04-12T04:15:40.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rNjjv6+4mv6V9gd07Xv2QSXGfqQ=', 'Content-Length': '0'} 2024-04-12T04:15:40.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003247a22e30b3222a-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.102 DEBUG:boto:path=/dkyges-62/obj11 2024-04-12T04:15:40.103 DEBUG:boto:auth_path=/dkyges-62/obj11 2024-04-12T04:15:40.103 DEBUG:boto:path=/dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.103 DEBUG:boto:auth_path=/dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.103 DEBUG:boto:Method: GET 2024-04-12T04:15:40.103 DEBUG:boto:Path: /dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.103 DEBUG:boto:Data: 2024-04-12T04:15:40.103 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.103 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.103 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.103 DEBUG:boto:Params: {} 2024-04-12T04:15:40.103 DEBUG:boto:Token: None 2024-04-12T04:15:40.104 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj11?versionId=null 2024-04-12T04:15:40.104 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rNjjv6+4mv6V9gd07Xv2QSXGfqQ= 2024-04-12T04:15:40.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rNjjv6+4mv6V9gd07Xv2QSXGfqQ=', 'Content-Length': '0'} 2024-04-12T04:15:40.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000302bb37249762b4e-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.146 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.146 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.146 DEBUG:boto:path=/dkyges-62/?prefix=obj11 2024-04-12T04:15:40.147 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj11 2024-04-12T04:15:40.147 DEBUG:boto:Method: GET 2024-04-12T04:15:40.147 DEBUG:boto:Path: /dkyges-62/?prefix=obj11 2024-04-12T04:15:40.147 DEBUG:boto:Data: 2024-04-12T04:15:40.147 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.147 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.147 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.147 DEBUG:boto:Params: {} 2024-04-12T04:15:40.147 DEBUG:boto:Token: None 2024-04-12T04:15:40.147 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.148 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000408ea6072832dbd6-006618b56c-208b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.150 DEBUG:boto:b'dkyges-62obj111000falseobj112024-04-12T04:13:03.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.151 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.151 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.151 DEBUG:boto:path=/dkyges-62/?prefix=obj11 2024-04-12T04:15:40.151 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj11 2024-04-12T04:15:40.151 DEBUG:boto:Method: GET 2024-04-12T04:15:40.151 DEBUG:boto:Path: /dkyges-62/?prefix=obj11 2024-04-12T04:15:40.151 DEBUG:boto:Data: 2024-04-12T04:15:40.151 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.151 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.152 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.152 DEBUG:boto:Params: {} 2024-04-12T04:15:40.152 DEBUG:boto:Token: None 2024-04-12T04:15:40.152 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.152 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc040fe38d516404-006618b56c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.155 DEBUG:boto:b'dkyges-62obj111000falseobj112024-04-12T04:13:03.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.155 DEBUG:boto:path=/dkyges-62/obj11 2024-04-12T04:15:40.155 DEBUG:boto:auth_path=/dkyges-62/obj11 2024-04-12T04:15:40.155 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.155 DEBUG:boto:Path: /dkyges-62/obj11 2024-04-12T04:15:40.155 DEBUG:boto:Data: 2024-04-12T04:15:40.155 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.156 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.156 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.156 DEBUG:boto:Params: {} 2024-04-12T04:15:40.156 DEBUG:boto:Token: None 2024-04-12T04:15:40.156 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj11 2024-04-12T04:15:40.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:26Kp2js4wTioAizgdzUL1rqiexc= 2024-04-12T04:15:40.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:26Kp2js4wTioAizgdzUL1rqiexc=', 'Content-Length': '0'} 2024-04-12T04:15:40.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001850f3c07425e8d-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.158 DEBUG:boto:path=/dkyges-62/obj11 2024-04-12T04:15:40.158 DEBUG:boto:auth_path=/dkyges-62/obj11 2024-04-12T04:15:40.158 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.158 DEBUG:boto:Path: /dkyges-62/obj11 2024-04-12T04:15:40.159 DEBUG:boto:Data: 2024-04-12T04:15:40.159 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.159 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.159 DEBUG:boto:Params: {} 2024-04-12T04:15:40.159 DEBUG:boto:Token: None 2024-04-12T04:15:40.159 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj11 2024-04-12T04:15:40.159 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:26Kp2js4wTioAizgdzUL1rqiexc= 2024-04-12T04:15:40.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:26Kp2js4wTioAizgdzUL1rqiexc=', 'Content-Length': '0'} 2024-04-12T04:15:40.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b18f0e4807abbc21-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:15:40.162 DEBUG:boto:path=/dkyges-62/obj11 2024-04-12T04:15:40.162 DEBUG:boto:auth_path=/dkyges-62/obj11 2024-04-12T04:15:40.162 DEBUG:boto:Method: GET 2024-04-12T04:15:40.162 DEBUG:boto:Path: /dkyges-62/obj11 2024-04-12T04:15:40.162 DEBUG:boto:Data: 2024-04-12T04:15:40.162 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.162 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.162 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.162 DEBUG:boto:Params: {} 2024-04-12T04:15:40.162 DEBUG:boto:Token: None 2024-04-12T04:15:40.163 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj11 2024-04-12T04:15:40.163 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d+Een0RBxNUW8EnxBCK3nf9kJHQ= 2024-04-12T04:15:40.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d+Een0RBxNUW8EnxBCK3nf9kJHQ=', 'Content-Length': '0'} 2024-04-12T04:15:40.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082237fd348ae3d05-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.210 DEBUG:boto:path=/dkyges-62/obj11 2024-04-12T04:15:40.210 DEBUG:boto:auth_path=/dkyges-62/obj11 2024-04-12T04:15:40.211 DEBUG:boto:Method: GET 2024-04-12T04:15:40.211 DEBUG:boto:Path: /dkyges-62/obj11 2024-04-12T04:15:40.211 DEBUG:boto:Data: 2024-04-12T04:15:40.211 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.211 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.211 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.211 DEBUG:boto:Params: {} 2024-04-12T04:15:40.211 DEBUG:boto:Token: None 2024-04-12T04:15:40.211 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj11 2024-04-12T04:15:40.211 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d+Een0RBxNUW8EnxBCK3nf9kJHQ= 2024-04-12T04:15:40.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d+Een0RBxNUW8EnxBCK3nf9kJHQ=', 'Content-Length': '0'} 2024-04-12T04:15:40.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022e9ef564d18d7f7-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.213 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.214 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.214 DEBUG:boto:path=/dkyges-62/?prefix=obj11 2024-04-12T04:15:40.214 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj11 2024-04-12T04:15:40.214 DEBUG:boto:Method: GET 2024-04-12T04:15:40.214 DEBUG:boto:Path: /dkyges-62/?prefix=obj11 2024-04-12T04:15:40.214 DEBUG:boto:Data: 2024-04-12T04:15:40.214 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.214 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.214 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.214 DEBUG:boto:Params: {} 2024-04-12T04:15:40.215 DEBUG:boto:Token: None 2024-04-12T04:15:40.215 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.215 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a44f1a95f9a0147-006618b56c-208b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.217 DEBUG:boto:b'dkyges-62obj111000falseobj112024-04-12T04:13:03.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.218 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.218 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.218 DEBUG:boto:path=/dkyges-62/?prefix=obj11 2024-04-12T04:15:40.218 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj11 2024-04-12T04:15:40.218 DEBUG:boto:Method: GET 2024-04-12T04:15:40.218 DEBUG:boto:Path: /dkyges-62/?prefix=obj11 2024-04-12T04:15:40.218 DEBUG:boto:Data: 2024-04-12T04:15:40.219 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.219 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.219 DEBUG:boto:Params: {} 2024-04-12T04:15:40.219 DEBUG:boto:Token: None 2024-04-12T04:15:40.219 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.219 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085cf488d400d748d-006618b56c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.222 DEBUG:boto:b'dkyges-62obj111000falseobj112024-04-12T04:13:03.420Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:15:40.223 DEBUG:boto:path=/dkyges-62/obj22 2024-04-12T04:15:40.223 DEBUG:boto:auth_path=/dkyges-62/obj22 2024-04-12T04:15:40.223 DEBUG:boto:path=/dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.223 DEBUG:boto:auth_path=/dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.223 DEBUG:boto:Method: GET 2024-04-12T04:15:40.223 DEBUG:boto:Path: /dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.223 DEBUG:boto:Data: 2024-04-12T04:15:40.223 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.223 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.223 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.223 DEBUG:boto:Params: {} 2024-04-12T04:15:40.224 DEBUG:boto:Token: None 2024-04-12T04:15:40.224 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.224 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Rs5PASrrtVQ0cGoo+Zq+nWHkyF8= 2024-04-12T04:15:40.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Rs5PASrrtVQ0cGoo+Zq+nWHkyF8=', 'Content-Length': '0'} 2024-04-12T04:15:40.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c80ed8fd6d7132a-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.226 DEBUG:boto:path=/dkyges-62/obj22 2024-04-12T04:15:40.226 DEBUG:boto:auth_path=/dkyges-62/obj22 2024-04-12T04:15:40.226 DEBUG:boto:path=/dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.226 DEBUG:boto:auth_path=/dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.227 DEBUG:boto:Method: GET 2024-04-12T04:15:40.227 DEBUG:boto:Path: /dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.227 DEBUG:boto:Data: 2024-04-12T04:15:40.227 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.227 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.227 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.227 DEBUG:boto:Params: {} 2024-04-12T04:15:40.227 DEBUG:boto:Token: None 2024-04-12T04:15:40.227 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.227 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Rs5PASrrtVQ0cGoo+Zq+nWHkyF8= 2024-04-12T04:15:40.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Rs5PASrrtVQ0cGoo+Zq+nWHkyF8=', 'Content-Length': '0'} 2024-04-12T04:15:40.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fffa97cbba211dac-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.230 DEBUG:boto:path=/dkyges-62/obj22 2024-04-12T04:15:40.230 DEBUG:boto:auth_path=/dkyges-62/obj22 2024-04-12T04:15:40.230 DEBUG:boto:path=/dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.230 DEBUG:boto:auth_path=/dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.230 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.230 DEBUG:boto:Path: /dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.230 DEBUG:boto:Data: 2024-04-12T04:15:40.230 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.230 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.231 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.231 DEBUG:boto:Params: {} 2024-04-12T04:15:40.231 DEBUG:boto:Token: None 2024-04-12T04:15:40.231 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.231 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KEYqND5RJCvaulbNMeNkIUuSYdY= 2024-04-12T04:15:40.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KEYqND5RJCvaulbNMeNkIUuSYdY=', 'Content-Length': '0'} 2024-04-12T04:15:40.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b2e963212555ac2-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.233 DEBUG:boto:path=/dkyges-62/obj22 2024-04-12T04:15:40.233 DEBUG:boto:auth_path=/dkyges-62/obj22 2024-04-12T04:15:40.233 DEBUG:boto:path=/dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.233 DEBUG:boto:auth_path=/dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.233 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.234 DEBUG:boto:Path: /dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.234 DEBUG:boto:Data: 2024-04-12T04:15:40.234 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.234 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.234 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.234 DEBUG:boto:Params: {} 2024-04-12T04:15:40.234 DEBUG:boto:Token: None 2024-04-12T04:15:40.234 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.234 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KEYqND5RJCvaulbNMeNkIUuSYdY= 2024-04-12T04:15:40.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KEYqND5RJCvaulbNMeNkIUuSYdY=', 'Content-Length': '0'} 2024-04-12T04:15:40.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c1e26e70ea1df73-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:15:40.237 DEBUG:boto:path=/dkyges-62/obj22 2024-04-12T04:15:40.237 DEBUG:boto:auth_path=/dkyges-62/obj22 2024-04-12T04:15:40.237 DEBUG:boto:path=/dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.237 DEBUG:boto:auth_path=/dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.237 DEBUG:boto:Method: GET 2024-04-12T04:15:40.237 DEBUG:boto:Path: /dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.237 DEBUG:boto:Data: 2024-04-12T04:15:40.237 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.237 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.237 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.237 DEBUG:boto:Params: {} 2024-04-12T04:15:40.238 DEBUG:boto:Token: None 2024-04-12T04:15:40.238 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.238 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Rs5PASrrtVQ0cGoo+Zq+nWHkyF8= 2024-04-12T04:15:40.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Rs5PASrrtVQ0cGoo+Zq+nWHkyF8=', 'Content-Length': '0'} 2024-04-12T04:15:40.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039dd02c6524f1f87-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.240 DEBUG:boto:path=/dkyges-62/obj22 2024-04-12T04:15:40.240 DEBUG:boto:auth_path=/dkyges-62/obj22 2024-04-12T04:15:40.240 DEBUG:boto:path=/dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.240 DEBUG:boto:auth_path=/dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.240 DEBUG:boto:Method: GET 2024-04-12T04:15:40.240 DEBUG:boto:Path: /dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.241 DEBUG:boto:Data: 2024-04-12T04:15:40.241 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.241 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.241 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.241 DEBUG:boto:Params: {} 2024-04-12T04:15:40.241 DEBUG:boto:Token: None 2024-04-12T04:15:40.241 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj22?versionId=null 2024-04-12T04:15:40.241 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Rs5PASrrtVQ0cGoo+Zq+nWHkyF8= 2024-04-12T04:15:40.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Rs5PASrrtVQ0cGoo+Zq+nWHkyF8=', 'Content-Length': '0'} 2024-04-12T04:15:40.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043a39a6ac5506e42-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.244 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.244 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.244 DEBUG:boto:path=/dkyges-62/?prefix=obj22 2024-04-12T04:15:40.244 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj22 2024-04-12T04:15:40.244 DEBUG:boto:Method: GET 2024-04-12T04:15:40.244 DEBUG:boto:Path: /dkyges-62/?prefix=obj22 2024-04-12T04:15:40.244 DEBUG:boto:Data: 2024-04-12T04:15:40.244 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.244 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.244 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.244 DEBUG:boto:Params: {} 2024-04-12T04:15:40.245 DEBUG:boto:Token: None 2024-04-12T04:15:40.245 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.245 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c73cfbc5cc9dd99-006618b56c-208b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.248 DEBUG:boto:b'dkyges-62obj221000falseobj222024-04-12T04:13:03.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.248 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.248 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.248 DEBUG:boto:path=/dkyges-62/?prefix=obj22 2024-04-12T04:15:40.248 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj22 2024-04-12T04:15:40.248 DEBUG:boto:Method: GET 2024-04-12T04:15:40.248 DEBUG:boto:Path: /dkyges-62/?prefix=obj22 2024-04-12T04:15:40.249 DEBUG:boto:Data: 2024-04-12T04:15:40.249 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.249 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.249 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.249 DEBUG:boto:Params: {} 2024-04-12T04:15:40.249 DEBUG:boto:Token: None 2024-04-12T04:15:40.249 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.249 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e449b4a5b4e70eb-006618b56c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.298 DEBUG:boto:b'dkyges-62obj221000falseobj222024-04-12T04:13:03.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.299 DEBUG:boto:path=/dkyges-62/obj22 2024-04-12T04:15:40.299 DEBUG:boto:auth_path=/dkyges-62/obj22 2024-04-12T04:15:40.299 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.299 DEBUG:boto:Path: /dkyges-62/obj22 2024-04-12T04:15:40.299 DEBUG:boto:Data: 2024-04-12T04:15:40.299 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.299 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.299 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.299 DEBUG:boto:Params: {} 2024-04-12T04:15:40.300 DEBUG:boto:Token: None 2024-04-12T04:15:40.300 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj22 2024-04-12T04:15:40.300 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f0GlMJdOJ83qkZTj2wzXAURG3fQ= 2024-04-12T04:15:40.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f0GlMJdOJ83qkZTj2wzXAURG3fQ=', 'Content-Length': '0'} 2024-04-12T04:15:40.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7d47d107f6ef5bb-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.302 DEBUG:boto:path=/dkyges-62/obj22 2024-04-12T04:15:40.302 DEBUG:boto:auth_path=/dkyges-62/obj22 2024-04-12T04:15:40.302 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.302 DEBUG:boto:Path: /dkyges-62/obj22 2024-04-12T04:15:40.302 DEBUG:boto:Data: 2024-04-12T04:15:40.302 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.302 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.303 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.303 DEBUG:boto:Params: {} 2024-04-12T04:15:40.303 DEBUG:boto:Token: None 2024-04-12T04:15:40.303 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj22 2024-04-12T04:15:40.303 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f0GlMJdOJ83qkZTj2wzXAURG3fQ= 2024-04-12T04:15:40.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f0GlMJdOJ83qkZTj2wzXAURG3fQ=', 'Content-Length': '0'} 2024-04-12T04:15:40.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004768376d787e5dfe-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:15:40.306 DEBUG:boto:path=/dkyges-62/obj22 2024-04-12T04:15:40.306 DEBUG:boto:auth_path=/dkyges-62/obj22 2024-04-12T04:15:40.306 DEBUG:boto:Method: GET 2024-04-12T04:15:40.306 DEBUG:boto:Path: /dkyges-62/obj22 2024-04-12T04:15:40.306 DEBUG:boto:Data: 2024-04-12T04:15:40.306 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.306 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.306 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.306 DEBUG:boto:Params: {} 2024-04-12T04:15:40.307 DEBUG:boto:Token: None 2024-04-12T04:15:40.307 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj22 2024-04-12T04:15:40.307 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bG0vprjyf/ApStK0tukcXhZOGAU= 2024-04-12T04:15:40.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bG0vprjyf/ApStK0tukcXhZOGAU=', 'Content-Length': '0'} 2024-04-12T04:15:40.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e5c5a0848c629d0-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.309 DEBUG:boto:path=/dkyges-62/obj22 2024-04-12T04:15:40.309 DEBUG:boto:auth_path=/dkyges-62/obj22 2024-04-12T04:15:40.309 DEBUG:boto:Method: GET 2024-04-12T04:15:40.309 DEBUG:boto:Path: /dkyges-62/obj22 2024-04-12T04:15:40.309 DEBUG:boto:Data: 2024-04-12T04:15:40.310 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.310 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.310 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.310 DEBUG:boto:Params: {} 2024-04-12T04:15:40.310 DEBUG:boto:Token: None 2024-04-12T04:15:40.310 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj22 2024-04-12T04:15:40.310 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bG0vprjyf/ApStK0tukcXhZOGAU= 2024-04-12T04:15:40.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bG0vprjyf/ApStK0tukcXhZOGAU=', 'Content-Length': '0'} 2024-04-12T04:15:40.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c88a1996a320ada-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.312 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.312 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.313 DEBUG:boto:path=/dkyges-62/?prefix=obj22 2024-04-12T04:15:40.313 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj22 2024-04-12T04:15:40.313 DEBUG:boto:Method: GET 2024-04-12T04:15:40.313 DEBUG:boto:Path: /dkyges-62/?prefix=obj22 2024-04-12T04:15:40.313 DEBUG:boto:Data: 2024-04-12T04:15:40.313 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.313 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.313 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.313 DEBUG:boto:Params: {} 2024-04-12T04:15:40.314 DEBUG:boto:Token: None 2024-04-12T04:15:40.314 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.314 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025927ad8d134f5ea-006618b56c-208b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.317 DEBUG:boto:b'dkyges-62obj221000falseobj222024-04-12T04:13:03.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.317 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.317 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.317 DEBUG:boto:path=/dkyges-62/?prefix=obj22 2024-04-12T04:15:40.317 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj22 2024-04-12T04:15:40.318 DEBUG:boto:Method: GET 2024-04-12T04:15:40.318 DEBUG:boto:Path: /dkyges-62/?prefix=obj22 2024-04-12T04:15:40.318 DEBUG:boto:Data: 2024-04-12T04:15:40.318 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.318 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.318 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.318 DEBUG:boto:Params: {} 2024-04-12T04:15:40.318 DEBUG:boto:Token: None 2024-04-12T04:15:40.318 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.319 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016f1ff1ff781786f-006618b56c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.322 DEBUG:boto:b'dkyges-62obj221000falseobj222024-04-12T04:13:03.433Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:15:40.322 DEBUG:boto:path=/dkyges-62/obj33 2024-04-12T04:15:40.322 DEBUG:boto:auth_path=/dkyges-62/obj33 2024-04-12T04:15:40.322 DEBUG:boto:path=/dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.322 DEBUG:boto:auth_path=/dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.322 DEBUG:boto:Method: GET 2024-04-12T04:15:40.323 DEBUG:boto:Path: /dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.323 DEBUG:boto:Data: 2024-04-12T04:15:40.323 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.323 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.323 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.323 DEBUG:boto:Params: {} 2024-04-12T04:15:40.323 DEBUG:boto:Token: None 2024-04-12T04:15:40.323 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.323 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xDAjOsZueugiwhtHPp1pCpWtn9Y= 2024-04-12T04:15:40.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xDAjOsZueugiwhtHPp1pCpWtn9Y=', 'Content-Length': '0'} 2024-04-12T04:15:40.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d42a3badf0c2b469-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.326 DEBUG:boto:path=/dkyges-62/obj33 2024-04-12T04:15:40.326 DEBUG:boto:auth_path=/dkyges-62/obj33 2024-04-12T04:15:40.326 DEBUG:boto:path=/dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.326 DEBUG:boto:auth_path=/dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.326 DEBUG:boto:Method: GET 2024-04-12T04:15:40.326 DEBUG:boto:Path: /dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.326 DEBUG:boto:Data: 2024-04-12T04:15:40.326 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.326 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.327 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.327 DEBUG:boto:Params: {} 2024-04-12T04:15:40.327 DEBUG:boto:Token: None 2024-04-12T04:15:40.327 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.327 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xDAjOsZueugiwhtHPp1pCpWtn9Y= 2024-04-12T04:15:40.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xDAjOsZueugiwhtHPp1pCpWtn9Y=', 'Content-Length': '0'} 2024-04-12T04:15:40.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1a672a306c94f87-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.329 DEBUG:boto:path=/dkyges-62/obj33 2024-04-12T04:15:40.329 DEBUG:boto:auth_path=/dkyges-62/obj33 2024-04-12T04:15:40.329 DEBUG:boto:path=/dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.329 DEBUG:boto:auth_path=/dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.329 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.330 DEBUG:boto:Path: /dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.330 DEBUG:boto:Data: 2024-04-12T04:15:40.330 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.330 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.330 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.330 DEBUG:boto:Params: {} 2024-04-12T04:15:40.330 DEBUG:boto:Token: None 2024-04-12T04:15:40.330 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.330 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I9P+JvKeix0elbOQYf1XRi0TqhE= 2024-04-12T04:15:40.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I9P+JvKeix0elbOQYf1XRi0TqhE=', 'Content-Length': '0'} 2024-04-12T04:15:40.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc0c16c70b293899-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.332 DEBUG:boto:path=/dkyges-62/obj33 2024-04-12T04:15:40.333 DEBUG:boto:auth_path=/dkyges-62/obj33 2024-04-12T04:15:40.333 DEBUG:boto:path=/dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.333 DEBUG:boto:auth_path=/dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.333 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.333 DEBUG:boto:Path: /dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.333 DEBUG:boto:Data: 2024-04-12T04:15:40.333 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.333 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.333 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.333 DEBUG:boto:Params: {} 2024-04-12T04:15:40.334 DEBUG:boto:Token: None 2024-04-12T04:15:40.334 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.334 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I9P+JvKeix0elbOQYf1XRi0TqhE= 2024-04-12T04:15:40.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I9P+JvKeix0elbOQYf1XRi0TqhE=', 'Content-Length': '0'} 2024-04-12T04:15:40.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c61e5df4cedea1e1-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:15:40.336 DEBUG:boto:path=/dkyges-62/obj33 2024-04-12T04:15:40.336 DEBUG:boto:auth_path=/dkyges-62/obj33 2024-04-12T04:15:40.336 DEBUG:boto:path=/dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.336 DEBUG:boto:auth_path=/dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.336 DEBUG:boto:Method: GET 2024-04-12T04:15:40.336 DEBUG:boto:Path: /dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.336 DEBUG:boto:Data: 2024-04-12T04:15:40.336 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.337 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.337 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.337 DEBUG:boto:Params: {} 2024-04-12T04:15:40.337 DEBUG:boto:Token: None 2024-04-12T04:15:40.337 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.337 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xDAjOsZueugiwhtHPp1pCpWtn9Y= 2024-04-12T04:15:40.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xDAjOsZueugiwhtHPp1pCpWtn9Y=', 'Content-Length': '0'} 2024-04-12T04:15:40.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa5746d3867c868e-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.339 DEBUG:boto:path=/dkyges-62/obj33 2024-04-12T04:15:40.339 DEBUG:boto:auth_path=/dkyges-62/obj33 2024-04-12T04:15:40.340 DEBUG:boto:path=/dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.340 DEBUG:boto:auth_path=/dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.340 DEBUG:boto:Method: GET 2024-04-12T04:15:40.340 DEBUG:boto:Path: /dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.340 DEBUG:boto:Data: 2024-04-12T04:15:40.340 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.340 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.340 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.340 DEBUG:boto:Params: {} 2024-04-12T04:15:40.340 DEBUG:boto:Token: None 2024-04-12T04:15:40.341 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj33?versionId=null 2024-04-12T04:15:40.341 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xDAjOsZueugiwhtHPp1pCpWtn9Y= 2024-04-12T04:15:40.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xDAjOsZueugiwhtHPp1pCpWtn9Y=', 'Content-Length': '0'} 2024-04-12T04:15:40.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6c8e3242250e3ad-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.343 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.343 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.343 DEBUG:boto:path=/dkyges-62/?prefix=obj33 2024-04-12T04:15:40.343 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj33 2024-04-12T04:15:40.343 DEBUG:boto:Method: GET 2024-04-12T04:15:40.343 DEBUG:boto:Path: /dkyges-62/?prefix=obj33 2024-04-12T04:15:40.343 DEBUG:boto:Data: 2024-04-12T04:15:40.344 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.344 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.344 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.344 DEBUG:boto:Params: {} 2024-04-12T04:15:40.344 DEBUG:boto:Token: None 2024-04-12T04:15:40.344 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.344 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a548bb2bfee0bef-006618b56c-208b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.347 DEBUG:boto:b'dkyges-62obj331000falseobj332024-04-12T04:13:03.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.348 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.348 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.348 DEBUG:boto:path=/dkyges-62/?prefix=obj33 2024-04-12T04:15:40.348 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj33 2024-04-12T04:15:40.348 DEBUG:boto:Method: GET 2024-04-12T04:15:40.348 DEBUG:boto:Path: /dkyges-62/?prefix=obj33 2024-04-12T04:15:40.348 DEBUG:boto:Data: 2024-04-12T04:15:40.348 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.348 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.348 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.348 DEBUG:boto:Params: {} 2024-04-12T04:15:40.349 DEBUG:boto:Token: None 2024-04-12T04:15:40.349 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.349 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2307a65b0a90bfc-006618b56c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.351 DEBUG:boto:b'dkyges-62obj331000falseobj332024-04-12T04:13:03.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.352 DEBUG:boto:path=/dkyges-62/obj33 2024-04-12T04:15:40.352 DEBUG:boto:auth_path=/dkyges-62/obj33 2024-04-12T04:15:40.352 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.352 DEBUG:boto:Path: /dkyges-62/obj33 2024-04-12T04:15:40.352 DEBUG:boto:Data: 2024-04-12T04:15:40.352 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.352 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.352 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.353 DEBUG:boto:Params: {} 2024-04-12T04:15:40.353 DEBUG:boto:Token: None 2024-04-12T04:15:40.353 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj33 2024-04-12T04:15:40.353 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fcEENjeexVLCn/VdqrywnxJo/Cw= 2024-04-12T04:15:40.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fcEENjeexVLCn/VdqrywnxJo/Cw=', 'Content-Length': '0'} 2024-04-12T04:15:40.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b62c463450bbf0ba-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.355 DEBUG:boto:path=/dkyges-62/obj33 2024-04-12T04:15:40.355 DEBUG:boto:auth_path=/dkyges-62/obj33 2024-04-12T04:15:40.355 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.355 DEBUG:boto:Path: /dkyges-62/obj33 2024-04-12T04:15:40.355 DEBUG:boto:Data: 2024-04-12T04:15:40.356 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.356 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.356 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.356 DEBUG:boto:Params: {} 2024-04-12T04:15:40.356 DEBUG:boto:Token: None 2024-04-12T04:15:40.356 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj33 2024-04-12T04:15:40.356 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fcEENjeexVLCn/VdqrywnxJo/Cw= 2024-04-12T04:15:40.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fcEENjeexVLCn/VdqrywnxJo/Cw=', 'Content-Length': '0'} 2024-04-12T04:15:40.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047103200d4460521-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:15:40.358 DEBUG:boto:path=/dkyges-62/obj33 2024-04-12T04:15:40.358 DEBUG:boto:auth_path=/dkyges-62/obj33 2024-04-12T04:15:40.359 DEBUG:boto:Method: GET 2024-04-12T04:15:40.359 DEBUG:boto:Path: /dkyges-62/obj33 2024-04-12T04:15:40.359 DEBUG:boto:Data: 2024-04-12T04:15:40.359 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.359 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.359 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.359 DEBUG:boto:Params: {} 2024-04-12T04:15:40.359 DEBUG:boto:Token: None 2024-04-12T04:15:40.359 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj33 2024-04-12T04:15:40.359 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zPeGrSCYnOQM+FgzMcvitWfxUlQ= 2024-04-12T04:15:40.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zPeGrSCYnOQM+FgzMcvitWfxUlQ=', 'Content-Length': '0'} 2024-04-12T04:15:40.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004741b75e39d56adb-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.402 DEBUG:boto:path=/dkyges-62/obj33 2024-04-12T04:15:40.403 DEBUG:boto:auth_path=/dkyges-62/obj33 2024-04-12T04:15:40.403 DEBUG:boto:Method: GET 2024-04-12T04:15:40.403 DEBUG:boto:Path: /dkyges-62/obj33 2024-04-12T04:15:40.403 DEBUG:boto:Data: 2024-04-12T04:15:40.403 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.403 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.403 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.403 DEBUG:boto:Params: {} 2024-04-12T04:15:40.403 DEBUG:boto:Token: None 2024-04-12T04:15:40.404 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj33 2024-04-12T04:15:40.404 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zPeGrSCYnOQM+FgzMcvitWfxUlQ= 2024-04-12T04:15:40.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zPeGrSCYnOQM+FgzMcvitWfxUlQ=', 'Content-Length': '0'} 2024-04-12T04:15:40.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000851069f35cf84d45-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.406 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.406 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.406 DEBUG:boto:path=/dkyges-62/?prefix=obj33 2024-04-12T04:15:40.406 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj33 2024-04-12T04:15:40.407 DEBUG:boto:Method: GET 2024-04-12T04:15:40.407 DEBUG:boto:Path: /dkyges-62/?prefix=obj33 2024-04-12T04:15:40.407 DEBUG:boto:Data: 2024-04-12T04:15:40.407 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.407 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.407 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.407 DEBUG:boto:Params: {} 2024-04-12T04:15:40.407 DEBUG:boto:Token: None 2024-04-12T04:15:40.407 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.407 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e819af9abc4c925e-006618b56c-208b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.410 DEBUG:boto:b'dkyges-62obj331000falseobj332024-04-12T04:13:03.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.411 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.411 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.411 DEBUG:boto:path=/dkyges-62/?prefix=obj33 2024-04-12T04:15:40.411 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj33 2024-04-12T04:15:40.411 DEBUG:boto:Method: GET 2024-04-12T04:15:40.411 DEBUG:boto:Path: /dkyges-62/?prefix=obj33 2024-04-12T04:15:40.411 DEBUG:boto:Data: 2024-04-12T04:15:40.411 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.411 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.411 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.412 DEBUG:boto:Params: {} 2024-04-12T04:15:40.412 DEBUG:boto:Token: None 2024-04-12T04:15:40.412 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.412 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd72505442294570-006618b56c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.415 DEBUG:boto:b'dkyges-62obj331000falseobj332024-04-12T04:13:03.444Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:15:40.415 DEBUG:boto:path=/dkyges-62/obj44 2024-04-12T04:15:40.415 DEBUG:boto:auth_path=/dkyges-62/obj44 2024-04-12T04:15:40.415 DEBUG:boto:path=/dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.415 DEBUG:boto:auth_path=/dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.415 DEBUG:boto:Method: GET 2024-04-12T04:15:40.416 DEBUG:boto:Path: /dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.416 DEBUG:boto:Data: 2024-04-12T04:15:40.416 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.416 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.416 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.416 DEBUG:boto:Params: {} 2024-04-12T04:15:40.416 DEBUG:boto:Token: None 2024-04-12T04:15:40.416 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.416 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B5w4aATca2HMuo4NK33edyXqyNY= 2024-04-12T04:15:40.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B5w4aATca2HMuo4NK33edyXqyNY=', 'Content-Length': '0'} 2024-04-12T04:15:40.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5002795c64fa826-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.419 DEBUG:boto:path=/dkyges-62/obj44 2024-04-12T04:15:40.419 DEBUG:boto:auth_path=/dkyges-62/obj44 2024-04-12T04:15:40.419 DEBUG:boto:path=/dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.419 DEBUG:boto:auth_path=/dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.419 DEBUG:boto:Method: GET 2024-04-12T04:15:40.419 DEBUG:boto:Path: /dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.419 DEBUG:boto:Data: 2024-04-12T04:15:40.419 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.419 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.419 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.420 DEBUG:boto:Params: {} 2024-04-12T04:15:40.420 DEBUG:boto:Token: None 2024-04-12T04:15:40.420 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.420 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B5w4aATca2HMuo4NK33edyXqyNY= 2024-04-12T04:15:40.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B5w4aATca2HMuo4NK33edyXqyNY=', 'Content-Length': '0'} 2024-04-12T04:15:40.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2584c5f3c9db63b-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.422 DEBUG:boto:path=/dkyges-62/obj44 2024-04-12T04:15:40.422 DEBUG:boto:auth_path=/dkyges-62/obj44 2024-04-12T04:15:40.422 DEBUG:boto:path=/dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.422 DEBUG:boto:auth_path=/dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.422 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.423 DEBUG:boto:Path: /dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.423 DEBUG:boto:Data: 2024-04-12T04:15:40.423 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.423 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.423 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.423 DEBUG:boto:Params: {} 2024-04-12T04:15:40.423 DEBUG:boto:Token: None 2024-04-12T04:15:40.423 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.423 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hoaywbw9/qKZNdiGHvpmeCRF3xU= 2024-04-12T04:15:40.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hoaywbw9/qKZNdiGHvpmeCRF3xU=', 'Content-Length': '0'} 2024-04-12T04:15:40.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be6d87fb1841d1b8-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.425 DEBUG:boto:path=/dkyges-62/obj44 2024-04-12T04:15:40.425 DEBUG:boto:auth_path=/dkyges-62/obj44 2024-04-12T04:15:40.426 DEBUG:boto:path=/dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.426 DEBUG:boto:auth_path=/dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.426 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.426 DEBUG:boto:Path: /dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.426 DEBUG:boto:Data: 2024-04-12T04:15:40.426 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.426 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.426 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.426 DEBUG:boto:Params: {} 2024-04-12T04:15:40.426 DEBUG:boto:Token: None 2024-04-12T04:15:40.427 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.427 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Hoaywbw9/qKZNdiGHvpmeCRF3xU= 2024-04-12T04:15:40.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Hoaywbw9/qKZNdiGHvpmeCRF3xU=', 'Content-Length': '0'} 2024-04-12T04:15:40.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b4b26295840a224-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:15:40.429 DEBUG:boto:path=/dkyges-62/obj44 2024-04-12T04:15:40.429 DEBUG:boto:auth_path=/dkyges-62/obj44 2024-04-12T04:15:40.429 DEBUG:boto:path=/dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.429 DEBUG:boto:auth_path=/dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.429 DEBUG:boto:Method: GET 2024-04-12T04:15:40.429 DEBUG:boto:Path: /dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.430 DEBUG:boto:Data: 2024-04-12T04:15:40.430 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.430 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.430 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.430 DEBUG:boto:Params: {} 2024-04-12T04:15:40.430 DEBUG:boto:Token: None 2024-04-12T04:15:40.430 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.430 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B5w4aATca2HMuo4NK33edyXqyNY= 2024-04-12T04:15:40.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B5w4aATca2HMuo4NK33edyXqyNY=', 'Content-Length': '0'} 2024-04-12T04:15:40.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bf98c1988d55347-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.433 DEBUG:boto:path=/dkyges-62/obj44 2024-04-12T04:15:40.433 DEBUG:boto:auth_path=/dkyges-62/obj44 2024-04-12T04:15:40.433 DEBUG:boto:path=/dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.433 DEBUG:boto:auth_path=/dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.433 DEBUG:boto:Method: GET 2024-04-12T04:15:40.433 DEBUG:boto:Path: /dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.433 DEBUG:boto:Data: 2024-04-12T04:15:40.433 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.433 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.434 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.434 DEBUG:boto:Params: {} 2024-04-12T04:15:40.434 DEBUG:boto:Token: None 2024-04-12T04:15:40.434 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj44?versionId=null 2024-04-12T04:15:40.434 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:B5w4aATca2HMuo4NK33edyXqyNY= 2024-04-12T04:15:40.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:B5w4aATca2HMuo4NK33edyXqyNY=', 'Content-Length': '0'} 2024-04-12T04:15:40.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec21a8f7fb358408-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.436 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.436 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.436 DEBUG:boto:path=/dkyges-62/?prefix=obj44 2024-04-12T04:15:40.437 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj44 2024-04-12T04:15:40.437 DEBUG:boto:Method: GET 2024-04-12T04:15:40.437 DEBUG:boto:Path: /dkyges-62/?prefix=obj44 2024-04-12T04:15:40.437 DEBUG:boto:Data: 2024-04-12T04:15:40.437 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.437 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.437 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.437 DEBUG:boto:Params: {} 2024-04-12T04:15:40.438 DEBUG:boto:Token: None 2024-04-12T04:15:40.438 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.438 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fedef930c5fbf2bb-006618b56c-208b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.441 DEBUG:boto:b'dkyges-62obj441000falseobj442024-04-12T04:13:03.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.441 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.441 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.441 DEBUG:boto:path=/dkyges-62/?prefix=obj44 2024-04-12T04:15:40.441 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj44 2024-04-12T04:15:40.441 DEBUG:boto:Method: GET 2024-04-12T04:15:40.441 DEBUG:boto:Path: /dkyges-62/?prefix=obj44 2024-04-12T04:15:40.442 DEBUG:boto:Data: 2024-04-12T04:15:40.442 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.442 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.442 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.442 DEBUG:boto:Params: {} 2024-04-12T04:15:40.442 DEBUG:boto:Token: None 2024-04-12T04:15:40.442 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.442 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063ebbb5303014dba-006618b56c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.445 DEBUG:boto:b'dkyges-62obj441000falseobj442024-04-12T04:13:03.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.445 DEBUG:boto:path=/dkyges-62/obj44 2024-04-12T04:15:40.445 DEBUG:boto:auth_path=/dkyges-62/obj44 2024-04-12T04:15:40.446 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.446 DEBUG:boto:Path: /dkyges-62/obj44 2024-04-12T04:15:40.446 DEBUG:boto:Data: 2024-04-12T04:15:40.446 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.446 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.446 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.446 DEBUG:boto:Params: {} 2024-04-12T04:15:40.446 DEBUG:boto:Token: None 2024-04-12T04:15:40.446 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj44 2024-04-12T04:15:40.447 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jSRzSmWGiPgXNIC95lrIGMx5FdU= 2024-04-12T04:15:40.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jSRzSmWGiPgXNIC95lrIGMx5FdU=', 'Content-Length': '0'} 2024-04-12T04:15:40.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008363eb9c1c754168-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.448 DEBUG:boto:path=/dkyges-62/obj44 2024-04-12T04:15:40.449 DEBUG:boto:auth_path=/dkyges-62/obj44 2024-04-12T04:15:40.449 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.449 DEBUG:boto:Path: /dkyges-62/obj44 2024-04-12T04:15:40.449 DEBUG:boto:Data: 2024-04-12T04:15:40.449 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.449 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.449 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.449 DEBUG:boto:Params: {} 2024-04-12T04:15:40.449 DEBUG:boto:Token: None 2024-04-12T04:15:40.449 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj44 2024-04-12T04:15:40.450 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jSRzSmWGiPgXNIC95lrIGMx5FdU= 2024-04-12T04:15:40.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jSRzSmWGiPgXNIC95lrIGMx5FdU=', 'Content-Length': '0'} 2024-04-12T04:15:40.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000de149414b9c0152-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:15:40.452 DEBUG:boto:path=/dkyges-62/obj44 2024-04-12T04:15:40.452 DEBUG:boto:auth_path=/dkyges-62/obj44 2024-04-12T04:15:40.452 DEBUG:boto:Method: GET 2024-04-12T04:15:40.452 DEBUG:boto:Path: /dkyges-62/obj44 2024-04-12T04:15:40.452 DEBUG:boto:Data: 2024-04-12T04:15:40.452 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.452 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.452 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.453 DEBUG:boto:Params: {} 2024-04-12T04:15:40.453 DEBUG:boto:Token: None 2024-04-12T04:15:40.453 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj44 2024-04-12T04:15:40.453 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CYGt/ABQ+UXzpAfvqfsv5DdoRNA= 2024-04-12T04:15:40.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CYGt/ABQ+UXzpAfvqfsv5DdoRNA=', 'Content-Length': '0'} 2024-04-12T04:15:40.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3bfd3eb7ee88052-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.455 DEBUG:boto:path=/dkyges-62/obj44 2024-04-12T04:15:40.455 DEBUG:boto:auth_path=/dkyges-62/obj44 2024-04-12T04:15:40.455 DEBUG:boto:Method: GET 2024-04-12T04:15:40.455 DEBUG:boto:Path: /dkyges-62/obj44 2024-04-12T04:15:40.455 DEBUG:boto:Data: 2024-04-12T04:15:40.456 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.456 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.456 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.456 DEBUG:boto:Params: {} 2024-04-12T04:15:40.456 DEBUG:boto:Token: None 2024-04-12T04:15:40.456 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj44 2024-04-12T04:15:40.456 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CYGt/ABQ+UXzpAfvqfsv5DdoRNA= 2024-04-12T04:15:40.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CYGt/ABQ+UXzpAfvqfsv5DdoRNA=', 'Content-Length': '0'} 2024-04-12T04:15:40.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097a4212c526dc373-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.458 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.458 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.459 DEBUG:boto:path=/dkyges-62/?prefix=obj44 2024-04-12T04:15:40.459 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj44 2024-04-12T04:15:40.459 DEBUG:boto:Method: GET 2024-04-12T04:15:40.459 DEBUG:boto:Path: /dkyges-62/?prefix=obj44 2024-04-12T04:15:40.459 DEBUG:boto:Data: 2024-04-12T04:15:40.459 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.459 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.459 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.459 DEBUG:boto:Params: {} 2024-04-12T04:15:40.459 DEBUG:boto:Token: None 2024-04-12T04:15:40.460 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.460 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000987314ccf4eb522c-006618b56c-208b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.502 DEBUG:boto:b'dkyges-62obj441000falseobj442024-04-12T04:13:03.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.503 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.503 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.503 DEBUG:boto:path=/dkyges-62/?prefix=obj44 2024-04-12T04:15:40.503 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj44 2024-04-12T04:15:40.503 DEBUG:boto:Method: GET 2024-04-12T04:15:40.504 DEBUG:boto:Path: /dkyges-62/?prefix=obj44 2024-04-12T04:15:40.504 DEBUG:boto:Data: 2024-04-12T04:15:40.504 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.504 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.504 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.504 DEBUG:boto:Params: {} 2024-04-12T04:15:40.504 DEBUG:boto:Token: None 2024-04-12T04:15:40.504 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.505 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.507 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e6367c386d5596b-006618b56c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.508 DEBUG:boto:b'dkyges-62obj441000falseobj442024-04-12T04:13:03.492Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:15:40.508 DEBUG:boto:path=/dkyges-62/obj55 2024-04-12T04:15:40.508 DEBUG:boto:auth_path=/dkyges-62/obj55 2024-04-12T04:15:40.508 DEBUG:boto:path=/dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.508 DEBUG:boto:auth_path=/dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.508 DEBUG:boto:Method: GET 2024-04-12T04:15:40.509 DEBUG:boto:Path: /dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.509 DEBUG:boto:Data: 2024-04-12T04:15:40.509 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.509 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.509 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.509 DEBUG:boto:Params: {} 2024-04-12T04:15:40.509 DEBUG:boto:Token: None 2024-04-12T04:15:40.509 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.509 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tMfxh/C8r9HNPHUFp1DfouG5I6k= 2024-04-12T04:15:40.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tMfxh/C8r9HNPHUFp1DfouG5I6k=', 'Content-Length': '0'} 2024-04-12T04:15:40.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009341ca433b87649a-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.512 DEBUG:boto:path=/dkyges-62/obj55 2024-04-12T04:15:40.512 DEBUG:boto:auth_path=/dkyges-62/obj55 2024-04-12T04:15:40.512 DEBUG:boto:path=/dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.512 DEBUG:boto:auth_path=/dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.512 DEBUG:boto:Method: GET 2024-04-12T04:15:40.512 DEBUG:boto:Path: /dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.512 DEBUG:boto:Data: 2024-04-12T04:15:40.512 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.512 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.512 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.512 DEBUG:boto:Params: {} 2024-04-12T04:15:40.513 DEBUG:boto:Token: None 2024-04-12T04:15:40.513 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.513 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tMfxh/C8r9HNPHUFp1DfouG5I6k= 2024-04-12T04:15:40.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tMfxh/C8r9HNPHUFp1DfouG5I6k=', 'Content-Length': '0'} 2024-04-12T04:15:40.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007212444ed23de916-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.558 DEBUG:boto:path=/dkyges-62/obj55 2024-04-12T04:15:40.559 DEBUG:boto:auth_path=/dkyges-62/obj55 2024-04-12T04:15:40.559 DEBUG:boto:path=/dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.559 DEBUG:boto:auth_path=/dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.559 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.559 DEBUG:boto:Path: /dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.559 DEBUG:boto:Data: 2024-04-12T04:15:40.559 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.559 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.559 DEBUG:boto:Params: {} 2024-04-12T04:15:40.560 DEBUG:boto:Token: None 2024-04-12T04:15:40.560 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.560 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:77zNUxkQe3WVRqsrh4JMFLAdbeE= 2024-04-12T04:15:40.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:77zNUxkQe3WVRqsrh4JMFLAdbeE=', 'Content-Length': '0'} 2024-04-12T04:15:40.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000232e3e1ad5262a8a-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.562 DEBUG:boto:path=/dkyges-62/obj55 2024-04-12T04:15:40.562 DEBUG:boto:auth_path=/dkyges-62/obj55 2024-04-12T04:15:40.562 DEBUG:boto:path=/dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.562 DEBUG:boto:auth_path=/dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.562 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.563 DEBUG:boto:Path: /dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.563 DEBUG:boto:Data: 2024-04-12T04:15:40.563 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.563 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.563 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.563 DEBUG:boto:Params: {} 2024-04-12T04:15:40.563 DEBUG:boto:Token: None 2024-04-12T04:15:40.563 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.563 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:77zNUxkQe3WVRqsrh4JMFLAdbeE= 2024-04-12T04:15:40.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:77zNUxkQe3WVRqsrh4JMFLAdbeE=', 'Content-Length': '0'} 2024-04-12T04:15:40.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001377584532ab6a35-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:15:40.566 DEBUG:boto:path=/dkyges-62/obj55 2024-04-12T04:15:40.566 DEBUG:boto:auth_path=/dkyges-62/obj55 2024-04-12T04:15:40.566 DEBUG:boto:path=/dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.566 DEBUG:boto:auth_path=/dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.566 DEBUG:boto:Method: GET 2024-04-12T04:15:40.566 DEBUG:boto:Path: /dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.566 DEBUG:boto:Data: 2024-04-12T04:15:40.566 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.566 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.566 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.567 DEBUG:boto:Params: {} 2024-04-12T04:15:40.567 DEBUG:boto:Token: None 2024-04-12T04:15:40.567 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.567 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tMfxh/C8r9HNPHUFp1DfouG5I6k= 2024-04-12T04:15:40.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tMfxh/C8r9HNPHUFp1DfouG5I6k=', 'Content-Length': '0'} 2024-04-12T04:15:40.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af4f2c76b0a63ec3-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.569 DEBUG:boto:path=/dkyges-62/obj55 2024-04-12T04:15:40.569 DEBUG:boto:auth_path=/dkyges-62/obj55 2024-04-12T04:15:40.569 DEBUG:boto:path=/dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.569 DEBUG:boto:auth_path=/dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.569 DEBUG:boto:Method: GET 2024-04-12T04:15:40.570 DEBUG:boto:Path: /dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.570 DEBUG:boto:Data: 2024-04-12T04:15:40.570 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.570 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.570 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.570 DEBUG:boto:Params: {} 2024-04-12T04:15:40.570 DEBUG:boto:Token: None 2024-04-12T04:15:40.570 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj55?versionId=null 2024-04-12T04:15:40.570 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tMfxh/C8r9HNPHUFp1DfouG5I6k= 2024-04-12T04:15:40.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tMfxh/C8r9HNPHUFp1DfouG5I6k=', 'Content-Length': '0'} 2024-04-12T04:15:40.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b107291356199ce9-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.572 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.573 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.573 DEBUG:boto:path=/dkyges-62/?prefix=obj55 2024-04-12T04:15:40.573 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj55 2024-04-12T04:15:40.573 DEBUG:boto:Method: GET 2024-04-12T04:15:40.573 DEBUG:boto:Path: /dkyges-62/?prefix=obj55 2024-04-12T04:15:40.573 DEBUG:boto:Data: 2024-04-12T04:15:40.573 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.573 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.573 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.573 DEBUG:boto:Params: {} 2024-04-12T04:15:40.574 DEBUG:boto:Token: None 2024-04-12T04:15:40.574 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.574 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000625fda847c88a4f3-006618b56c-208b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.576 DEBUG:boto:b'dkyges-62obj551000falseobj552024-04-12T04:13:03.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.577 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.577 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.577 DEBUG:boto:path=/dkyges-62/?prefix=obj55 2024-04-12T04:15:40.577 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj55 2024-04-12T04:15:40.577 DEBUG:boto:Method: GET 2024-04-12T04:15:40.577 DEBUG:boto:Path: /dkyges-62/?prefix=obj55 2024-04-12T04:15:40.577 DEBUG:boto:Data: 2024-04-12T04:15:40.577 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.577 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.578 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.578 DEBUG:boto:Params: {} 2024-04-12T04:15:40.578 DEBUG:boto:Token: None 2024-04-12T04:15:40.578 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.578 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4718a8542b7215e-006618b56c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.581 DEBUG:boto:b'dkyges-62obj551000falseobj552024-04-12T04:13:03.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.581 DEBUG:boto:path=/dkyges-62/obj55 2024-04-12T04:15:40.581 DEBUG:boto:auth_path=/dkyges-62/obj55 2024-04-12T04:15:40.581 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.582 DEBUG:boto:Path: /dkyges-62/obj55 2024-04-12T04:15:40.582 DEBUG:boto:Data: 2024-04-12T04:15:40.582 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.582 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.582 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.582 DEBUG:boto:Params: {} 2024-04-12T04:15:40.582 DEBUG:boto:Token: None 2024-04-12T04:15:40.582 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj55 2024-04-12T04:15:40.582 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3nD8x1wHc5igZnXwfV3n4XbiLRQ= 2024-04-12T04:15:40.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3nD8x1wHc5igZnXwfV3n4XbiLRQ=', 'Content-Length': '0'} 2024-04-12T04:15:40.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a2c6a384b31fa54-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.584 DEBUG:boto:path=/dkyges-62/obj55 2024-04-12T04:15:40.584 DEBUG:boto:auth_path=/dkyges-62/obj55 2024-04-12T04:15:40.585 DEBUG:boto:Method: HEAD 2024-04-12T04:15:40.585 DEBUG:boto:Path: /dkyges-62/obj55 2024-04-12T04:15:40.585 DEBUG:boto:Data: 2024-04-12T04:15:40.585 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.585 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.585 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.585 DEBUG:boto:Params: {} 2024-04-12T04:15:40.585 DEBUG:boto:Token: None 2024-04-12T04:15:40.585 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj55 2024-04-12T04:15:40.586 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3nD8x1wHc5igZnXwfV3n4XbiLRQ= 2024-04-12T04:15:40.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3nD8x1wHc5igZnXwfV3n4XbiLRQ=', 'Content-Length': '0'} 2024-04-12T04:15:40.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cc23ed0b64b7a90-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:15:40.588 DEBUG:boto:path=/dkyges-62/obj55 2024-04-12T04:15:40.588 DEBUG:boto:auth_path=/dkyges-62/obj55 2024-04-12T04:15:40.588 DEBUG:boto:Method: GET 2024-04-12T04:15:40.588 DEBUG:boto:Path: /dkyges-62/obj55 2024-04-12T04:15:40.588 DEBUG:boto:Data: 2024-04-12T04:15:40.588 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.588 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.588 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.588 DEBUG:boto:Params: {} 2024-04-12T04:15:40.588 DEBUG:boto:Token: None 2024-04-12T04:15:40.589 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj55 2024-04-12T04:15:40.589 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/8r6EdPEj0CNthQQldCVO7Qsy/c= 2024-04-12T04:15:40.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/8r6EdPEj0CNthQQldCVO7Qsy/c=', 'Content-Length': '0'} 2024-04-12T04:15:40.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-62:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da3fd1bebecace91-006618b56c-208b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.591 DEBUG:boto:path=/dkyges-62/obj55 2024-04-12T04:15:40.591 DEBUG:boto:auth_path=/dkyges-62/obj55 2024-04-12T04:15:40.591 DEBUG:boto:Method: GET 2024-04-12T04:15:40.591 DEBUG:boto:Path: /dkyges-62/obj55 2024-04-12T04:15:40.591 DEBUG:boto:Data: 2024-04-12T04:15:40.591 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.591 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.591 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.591 DEBUG:boto:Params: {} 2024-04-12T04:15:40.592 DEBUG:boto:Token: None 2024-04-12T04:15:40.592 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/obj55 2024-04-12T04:15:40.592 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/8r6EdPEj0CNthQQldCVO7Qsy/c= 2024-04-12T04:15:40.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/8r6EdPEj0CNthQQldCVO7Qsy/c=', 'Content-Length': '0'} 2024-04-12T04:15:40.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:13:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1cdf1bcc83e5b34-006618b56c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.594 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.594 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.594 DEBUG:boto:path=/dkyges-62/?prefix=obj55 2024-04-12T04:15:40.594 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj55 2024-04-12T04:15:40.594 DEBUG:boto:Method: GET 2024-04-12T04:15:40.594 DEBUG:boto:Path: /dkyges-62/?prefix=obj55 2024-04-12T04:15:40.595 DEBUG:boto:Data: 2024-04-12T04:15:40.595 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.595 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:40.595 DEBUG:boto:Port: 8001 2024-04-12T04:15:40.595 DEBUG:boto:Params: {} 2024-04-12T04:15:40.595 DEBUG:boto:Token: None 2024-04-12T04:15:40.595 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.595 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd934da1cd51ba2f-006618b56c-208b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.598 DEBUG:boto:b'dkyges-62obj551000falseobj552024-04-12T04:13:03.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.598 DEBUG:boto:path=/dkyges-62/ 2024-04-12T04:15:40.598 DEBUG:boto:auth_path=/dkyges-62/ 2024-04-12T04:15:40.598 DEBUG:boto:path=/dkyges-62/?prefix=obj55 2024-04-12T04:15:40.599 DEBUG:boto:auth_path=/dkyges-62/?prefix=obj55 2024-04-12T04:15:40.599 DEBUG:boto:Method: GET 2024-04-12T04:15:40.599 DEBUG:boto:Path: /dkyges-62/?prefix=obj55 2024-04-12T04:15:40.599 DEBUG:boto:Data: 2024-04-12T04:15:40.599 DEBUG:boto:Headers: {} 2024-04-12T04:15:40.599 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:40.599 DEBUG:boto:Port: 8000 2024-04-12T04:15:40.599 DEBUG:boto:Params: {} 2024-04-12T04:15:40.599 DEBUG:boto:Token: None 2024-04-12T04:15:40.599 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:15:40 GMT /dkyges-62/ 2024-04-12T04:15:40.600 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc= 2024-04-12T04:15:40.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+sIYAmP2ZSWfWeoAahD+jyO0Yfc=', 'Content-Length': '0'} 2024-04-12T04:15:40.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006445a665a049b8ef-006618b56c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:15:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:40.602 DEBUG:boto:b'dkyges-62obj551000falseobj552024-04-12T04:13:03.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:15:40.603 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-62 zones={a2, a1} 2024-04-12T04:15:40.603 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-12T04:15:40.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:40.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:40.603 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:15:40.663 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:40.664 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:40.725 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:15:40.725 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:40.725 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000028096", 2024-04-12T04:15:40.726 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:13:04.022952Z" 2024-04-12T04:15:40.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:40.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:40.726 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000019459", 2024-04-12T04:15:40.726 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:09:24.002925Z" 2024-04-12T04:15:40.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:40.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:40.726 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033558", 2024-04-12T04:15:40.726 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:13:03.497545Z" 2024-04-12T04:15:40.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:40.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:40.727 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000029036", 2024-04-12T04:15:40.727 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:13:04.025700Z" 2024-04-12T04:15:40.727 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:40.727 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:15:40.728 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000028096', 1: '00000000000000000000:00000000000000019459', 2: '00000000000000000000:00000000000000033558', 3: '00000000000000000000:00000000000000029036'} 2024-04-12T04:15:40.728 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:15:40.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:40.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:40.729 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:15:40.786 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:40.786 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:40.837 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:15:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:15:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:15:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:15:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:15:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 3607311848422815784 2024-04-12T04:15:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:15:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:15:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:15:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:15:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000028096", 2024-04-12T04:15:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:15:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:15:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:15:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:13:04.022952Z" 2024-04-12T04:15:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:15:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:15:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:15:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000019459", 2024-04-12T04:15:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:15:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:15:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:15:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:09:24.002925Z" 2024-04-12T04:15:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:15:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:15:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:15:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033558", 2024-04-12T04:15:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:15:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 53, 2024-04-12T04:15:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:15:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:13:03.497545Z" 2024-04-12T04:15:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:15:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:15:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:15:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000029036", 2024-04-12T04:15:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:15:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:15:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:15:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:13:04.025700Z" 2024-04-12T04:15:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:15:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:15:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 62, 2024-04-12T04:15:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 62 2024-04-12T04:15:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:15:40Z" 2024-04-12T04:15:40.842 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:15:40.843 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 3607311848422815784 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000028096", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:13:04.022952Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000019459", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:09:24.002925Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033558", "next_step_marker": "", "total_entries": 53, "pos": 0, "timestamp": "2024-04-12T04:13:03.497545Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000029036", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:13:04.025700Z" } } ] }, "full_sync": { "total": 62, "complete": 62 }, "current_time": "2024-04-12T04:15:40Z" } 2024-04-12T04:15:40.843 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000028096', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:13:04.022952Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000019459', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:09:24.002925Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033558', 'next_step_marker': '', 'total_entries': 53, 'pos': 0, 'timestamp': '2024-04-12T04:13:03.497545Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000029036', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:13:04.025700Z'}}] 2024-04-12T04:15:40.844 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000028096', 1: '00000000000000000000:00000000000000019459', 2: '00000000000000000000:00000000000000033558', 3: '00000000000000000000:00000000000000029036'} 2024-04-12T04:15:40.844 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000028096', 1: '00000000000000000000:00000000000000019459', 2: '00000000000000000000:00000000000000033558', 3: '00000000000000000000:00000000000000029036'} 2024-04-12T04:15:40.844 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:15:40.844 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-12T04:15:40.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:40.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:40.844 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:15:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:40.956 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:15:40.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:40.956 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000012383", 2024-04-12T04:15:40.957 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:14:37.732849Z" 2024-04-12T04:15:40.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:40.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:40.957 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006150", 2024-04-12T04:15:40.957 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:02.322957Z" 2024-04-12T04:15:40.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:40.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:40.957 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000017280", 2024-04-12T04:15:40.957 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:13:07.269791Z" 2024-04-12T04:15:40.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:40.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:40.958 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000015450", 2024-04-12T04:15:40.958 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:13:25.799532Z" 2024-04-12T04:15:40.958 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:40.958 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:15:40.959 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000012383', 1: '00000000000000000000:00000000000000006150', 2: '00000000000000000000:00000000000000017280', 3: '00000000000000000000:00000000000000015450'} 2024-04-12T04:15:40.959 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T04:15:40.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:40.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:40.960 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:15:41.016 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:41.016 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:41.067 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:15:41.068 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:15:41.068 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:15:41.068 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:15:41.068 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:15:41.068 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5944041172307441597 2024-04-12T04:15:41.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:41.068 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:15:41.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:41.068 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:15:41.069 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:15:41.069 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:15:41.069 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000012383", 2024-04-12T04:15:41.069 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:15:41.069 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:15:41.069 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:15:41.069 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:14:37.732849Z" 2024-04-12T04:15:41.069 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:41.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:41.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:41.069 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:15:41.069 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:15:41.069 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:15:41.070 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006150", 2024-04-12T04:15:41.070 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:15:41.070 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:15:41.070 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:15:41.070 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:02.322957Z" 2024-04-12T04:15:41.070 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:41.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:41.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:41.070 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:15:41.070 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:15:41.070 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:15:41.071 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000017280", 2024-04-12T04:15:41.071 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:15:41.071 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:15:41.071 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:15:41.071 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:13:07.269791Z" 2024-04-12T04:15:41.071 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:41.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:41.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:41.071 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:15:41.071 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:15:41.071 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:15:41.072 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000015450", 2024-04-12T04:15:41.072 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:15:41.072 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:15:41.072 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:15:41.072 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:13:25.799532Z" 2024-04-12T04:15:41.072 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:41.072 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:41.072 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:15:41.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:41.072 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:15:41.072 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T04:15:41.072 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T04:15:41.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:41.073 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:15:41Z" 2024-04-12T04:15:41.073 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:15:41.074 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5944041172307441597 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000012383", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:14:37.732849Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006150", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:02.322957Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000017280", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:13:07.269791Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015450", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:13:25.799532Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T04:15:41Z" } 2024-04-12T04:15:41.074 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000012383', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:14:37.732849Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006150', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:02.322957Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000017280', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:13:07.269791Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015450', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:13:25.799532Z'}}] 2024-04-12T04:15:41.074 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000012383', 1: '00000000000000000000:00000000000000006150', 2: '00000000000000000000:00000000000000017280', 3: '00000000000000000000:00000000000000015450'} 2024-04-12T04:15:41.074 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000012383', 1: '00000000000000000000:00000000000000006150', 2: '00000000000000000000:00000000000000017280', 3: '00000000000000000000:00000000000000015450'} 2024-04-12T04:15:41.074 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T04:15:41.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ok 2024-04-12T04:15:41.076 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:41.076 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:41.076 DEBUG:boto:Method: PUT 2024-04-12T04:15:41.077 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:41.077 DEBUG:boto:Data: 2024-04-12T04:15:41.077 DEBUG:boto:Headers: {} 2024-04-12T04:15:41.077 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:41.077 DEBUG:boto:Port: 8000 2024-04-12T04:15:41.077 DEBUG:boto:Params: {} 2024-04-12T04:15:41.077 DEBUG:boto:Token: None 2024-04-12T04:15:41.077 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:15:41 GMT /dkyges-63/ 2024-04-12T04:15:41.077 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AmP6Dp3Nhvdi+q3WRlbczBy29iA= 2024-04-12T04:15:41.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AmP6Dp3Nhvdi+q3WRlbczBy29iA=', 'Content-Length': '0'} 2024-04-12T04:15:41.095 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007323ba7810c10d08-006618b56d-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:41.095 DEBUG:rgw_multi.tests:created bucket=dkyges-63 2024-04-12T04:15:41.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:41.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:41.096 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:15:41.152 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:41.152 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:41.195 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:15:41.213 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:15:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:15:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:15:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:15:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:15:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:15:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:15:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895341.112453_666.1", 2024-04-12T04:15:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:15:41.112453Z" 2024-04-12T04:15:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:41.215 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:15:41.216 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895341.112453_666.1'} 2024-04-12T04:15:41.216 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:15:41.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:41.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:15:41.217 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:15:41.272 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:41.272 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:41.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:41.323+0000 7fa57b758780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:15:41.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:41.324+0000 7fa57b758780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:15:41.322 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:15:41.322 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:15:41.322 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:15:41.322 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:15:41.322 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:15:41.323 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:15:41.323 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:15:41.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:41.323 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:15:41.323 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:41.323 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:15:41.323 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:15:41.323 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:15:41.323 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:15:41.323 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:15:41.323 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:15:41.324 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:15:41.324 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:15:41.324 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:15:41.324 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:41.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:41.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:41.324 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:15:41.324 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:15:41.324 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:15:41.325 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:15:41.325 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:15:41.325 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:15:41.325 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:15:41.325 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:15:41.325 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:15:41.326 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:41.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:41.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:41.326 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:15:41.326 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:15:41.326 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:15:41.326 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:15:41.326 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:15:41.326 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:15:41.326 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:15:41.326 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:15:41.327 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:15:41.327 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:41.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:41.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:15:41.327 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:15:41.327 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:15:41.327 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:15:41.327 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895341.112453_666.1", 2024-04-12T04:15:41.327 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:15:41.327 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:15:41.327 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:15:41.328 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:15:41.112453Z", 2024-04-12T04:15:41.328 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:15:41.328 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:41.328 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:15:41.328 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:15:41.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:41.328 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:15:41.328 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:15:41.328 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:15:41.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:15:41.328 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:15:41Z" 2024-04-12T04:15:41.329 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:15:41.330 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895341.112453_666.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:15:41.112453Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:15:41Z" } 2024-04-12T04:15:41.330 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895341.112453_666.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:15:41.112453Z', 'realm_epoch': 2}}] 2024-04-12T04:15:41.330 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895341.112453_666.1'} 2024-04-12T04:15:41.330 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895341.112453_666.1'} 2024-04-12T04:15:41.330 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:15:41.331 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T04:15:41.331 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:15:41.331 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T04:15:41.349+0000 7f6a11722700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 104689) UID: 0 2024-04-12T04:15:41.332 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T04:15:41.349+0000 7f6a168f77c0 -1 shutting down 2024-04-12T04:15:47.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T04:15:47.432 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T04:15:47.432 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:47.432 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:47.433 DEBUG:boto:Method: HEAD 2024-04-12T04:15:47.433 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:47.433 DEBUG:boto:Data: 2024-04-12T04:15:47.433 DEBUG:boto:Headers: {} 2024-04-12T04:15:47.433 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.433 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.433 DEBUG:boto:Params: {} 2024-04-12T04:15:47.433 DEBUG:boto:Token: None 2024-04-12T04:15:47.433 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/ 2024-04-12T04:15:47.434 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU= 2024-04-12T04:15:47.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU=', 'Content-Length': '0'} 2024-04-12T04:15:47.437 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000996a4665bac0f464-006618b573-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.437 DEBUG:boto:path=/dkyges-63/obj11 2024-04-12T04:15:47.437 DEBUG:boto:auth_path=/dkyges-63/obj11 2024-04-12T04:15:47.437 DEBUG:boto:Method: PUT 2024-04-12T04:15:47.437 DEBUG:boto:Path: /dkyges-63/obj11 2024-04-12T04:15:47.437 DEBUG:boto:Data: 2024-04-12T04:15:47.438 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:47.438 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.438 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.438 DEBUG:boto:Params: {} 2024-04-12T04:15:47.438 DEBUG:boto:Token: None 2024-04-12T04:15:47.438 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/obj11 2024-04-12T04:15:47.438 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aMgxrwoALKGWPjr/NymdoP5wgTQ= 2024-04-12T04:15:47.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aMgxrwoALKGWPjr/NymdoP5wgTQ='} 2024-04-12T04:15:47.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001f3c50d864ce89c5-006618b573-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.446 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:47.447 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:47.447 DEBUG:boto:Method: HEAD 2024-04-12T04:15:47.447 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:47.447 DEBUG:boto:Data: 2024-04-12T04:15:47.447 DEBUG:boto:Headers: {} 2024-04-12T04:15:47.447 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.447 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.447 DEBUG:boto:Params: {} 2024-04-12T04:15:47.447 DEBUG:boto:Token: None 2024-04-12T04:15:47.447 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/ 2024-04-12T04:15:47.447 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU= 2024-04-12T04:15:47.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU=', 'Content-Length': '0'} 2024-04-12T04:15:47.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f015e9d11cff51d0-006618b573-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.450 DEBUG:boto:path=/dkyges-63/obj22 2024-04-12T04:15:47.450 DEBUG:boto:auth_path=/dkyges-63/obj22 2024-04-12T04:15:47.450 DEBUG:boto:Method: PUT 2024-04-12T04:15:47.450 DEBUG:boto:Path: /dkyges-63/obj22 2024-04-12T04:15:47.451 DEBUG:boto:Data: 2024-04-12T04:15:47.451 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:47.451 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.451 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.451 DEBUG:boto:Params: {} 2024-04-12T04:15:47.451 DEBUG:boto:Token: None 2024-04-12T04:15:47.451 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/obj22 2024-04-12T04:15:47.451 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mk3TeKG/+AbCXW0AryZJf0hLR/o= 2024-04-12T04:15:47.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mk3TeKG/+AbCXW0AryZJf0hLR/o='} 2024-04-12T04:15:47.495 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3267c525ac1a0d5-006618b573-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.495 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:47.495 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:47.496 DEBUG:boto:Method: HEAD 2024-04-12T04:15:47.496 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:47.496 DEBUG:boto:Data: 2024-04-12T04:15:47.496 DEBUG:boto:Headers: {} 2024-04-12T04:15:47.496 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.496 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.496 DEBUG:boto:Params: {} 2024-04-12T04:15:47.497 DEBUG:boto:Token: None 2024-04-12T04:15:47.497 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/ 2024-04-12T04:15:47.497 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU= 2024-04-12T04:15:47.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU=', 'Content-Length': '0'} 2024-04-12T04:15:47.500 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008cb33c77e3c33e8d-006618b573-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.500 DEBUG:boto:path=/dkyges-63/obj33 2024-04-12T04:15:47.500 DEBUG:boto:auth_path=/dkyges-63/obj33 2024-04-12T04:15:47.500 DEBUG:boto:Method: PUT 2024-04-12T04:15:47.501 DEBUG:boto:Path: /dkyges-63/obj33 2024-04-12T04:15:47.501 DEBUG:boto:Data: 2024-04-12T04:15:47.501 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:47.501 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.501 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.501 DEBUG:boto:Params: {} 2024-04-12T04:15:47.501 DEBUG:boto:Token: None 2024-04-12T04:15:47.501 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/obj33 2024-04-12T04:15:47.501 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i8CcM7eQY0TK3xprNaLuycCc7KM= 2024-04-12T04:15:47.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i8CcM7eQY0TK3xprNaLuycCc7KM='} 2024-04-12T04:15:47.508 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000bab42d402b41ae6-006618b573-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.509 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:47.509 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:47.509 DEBUG:boto:Method: HEAD 2024-04-12T04:15:47.509 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:47.509 DEBUG:boto:Data: 2024-04-12T04:15:47.509 DEBUG:boto:Headers: {} 2024-04-12T04:15:47.509 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.509 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.509 DEBUG:boto:Params: {} 2024-04-12T04:15:47.510 DEBUG:boto:Token: None 2024-04-12T04:15:47.510 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/ 2024-04-12T04:15:47.510 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU= 2024-04-12T04:15:47.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU=', 'Content-Length': '0'} 2024-04-12T04:15:47.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002af4386ce378c5dd-006618b573-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.513 DEBUG:boto:path=/dkyges-63/obj44 2024-04-12T04:15:47.513 DEBUG:boto:auth_path=/dkyges-63/obj44 2024-04-12T04:15:47.513 DEBUG:boto:Method: PUT 2024-04-12T04:15:47.513 DEBUG:boto:Path: /dkyges-63/obj44 2024-04-12T04:15:47.513 DEBUG:boto:Data: 2024-04-12T04:15:47.513 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:47.513 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.514 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.514 DEBUG:boto:Params: {} 2024-04-12T04:15:47.514 DEBUG:boto:Token: None 2024-04-12T04:15:47.514 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/obj44 2024-04-12T04:15:47.514 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cTdzdD92acbQGyjiGQWZBU4CK+w= 2024-04-12T04:15:47.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cTdzdD92acbQGyjiGQWZBU4CK+w='} 2024-04-12T04:15:47.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073f5d87fd2dbd430-006618b573-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.559 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:47.559 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:47.559 DEBUG:boto:Method: HEAD 2024-04-12T04:15:47.559 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:47.559 DEBUG:boto:Data: 2024-04-12T04:15:47.559 DEBUG:boto:Headers: {} 2024-04-12T04:15:47.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.559 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.559 DEBUG:boto:Params: {} 2024-04-12T04:15:47.560 DEBUG:boto:Token: None 2024-04-12T04:15:47.560 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/ 2024-04-12T04:15:47.560 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU= 2024-04-12T04:15:47.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU=', 'Content-Length': '0'} 2024-04-12T04:15:47.562 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fd124a22033ca98c-006618b573-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.563 DEBUG:boto:path=/dkyges-63/obj55 2024-04-12T04:15:47.563 DEBUG:boto:auth_path=/dkyges-63/obj55 2024-04-12T04:15:47.563 DEBUG:boto:Method: PUT 2024-04-12T04:15:47.563 DEBUG:boto:Path: /dkyges-63/obj55 2024-04-12T04:15:47.563 DEBUG:boto:Data: 2024-04-12T04:15:47.563 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:47.563 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.563 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.563 DEBUG:boto:Params: {} 2024-04-12T04:15:47.564 DEBUG:boto:Token: None 2024-04-12T04:15:47.564 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/obj55 2024-04-12T04:15:47.564 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:sLh0hnug0mKIncTZ/I3t3AC+5js= 2024-04-12T04:15:47.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sLh0hnug0mKIncTZ/I3t3AC+5js='} 2024-04-12T04:15:47.569 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e8fafe61c8c8f99-006618b573-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:47.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:47.569 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:47.632 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:47.632 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:47.684 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:15:47.684 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-63 2024-04-12T04:15:47.688 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 5 2024-04-12T04:15:47.735 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:47.735 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:47.735 DEBUG:boto:Method: HEAD 2024-04-12T04:15:47.735 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:47.735 DEBUG:boto:Data: 2024-04-12T04:15:47.736 DEBUG:boto:Headers: {} 2024-04-12T04:15:47.736 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.736 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.736 DEBUG:boto:Params: {} 2024-04-12T04:15:47.736 DEBUG:boto:Token: None 2024-04-12T04:15:47.736 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/ 2024-04-12T04:15:47.736 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU= 2024-04-12T04:15:47.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU=', 'Content-Length': '0'} 2024-04-12T04:15:47.739 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fa0c7570a7bd84c-006618b573-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.739 DEBUG:boto:path=/dkyges-63/obj17 2024-04-12T04:15:47.739 DEBUG:boto:auth_path=/dkyges-63/obj17 2024-04-12T04:15:47.739 DEBUG:boto:Method: PUT 2024-04-12T04:15:47.740 DEBUG:boto:Path: /dkyges-63/obj17 2024-04-12T04:15:47.740 DEBUG:boto:Data: 2024-04-12T04:15:47.740 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:47.740 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.740 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.740 DEBUG:boto:Params: {} 2024-04-12T04:15:47.740 DEBUG:boto:Token: None 2024-04-12T04:15:47.740 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/obj17 2024-04-12T04:15:47.740 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Gvys+ohWDdZ7PqkHMNI1bIEQ868= 2024-04-12T04:15:47.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Gvys+ohWDdZ7PqkHMNI1bIEQ868='} 2024-04-12T04:15:47.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000018384c734f658e61-006618b573-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.783 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:47.783 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:47.783 DEBUG:boto:Method: HEAD 2024-04-12T04:15:47.783 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:47.783 DEBUG:boto:Data: 2024-04-12T04:15:47.783 DEBUG:boto:Headers: {} 2024-04-12T04:15:47.783 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.783 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.783 DEBUG:boto:Params: {} 2024-04-12T04:15:47.783 DEBUG:boto:Token: None 2024-04-12T04:15:47.784 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/ 2024-04-12T04:15:47.784 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU= 2024-04-12T04:15:47.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU=', 'Content-Length': '0'} 2024-04-12T04:15:47.787 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005df566713c5d1848-006618b573-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.787 DEBUG:boto:path=/dkyges-63/obj34 2024-04-12T04:15:47.787 DEBUG:boto:auth_path=/dkyges-63/obj34 2024-04-12T04:15:47.787 DEBUG:boto:Method: PUT 2024-04-12T04:15:47.787 DEBUG:boto:Path: /dkyges-63/obj34 2024-04-12T04:15:47.787 DEBUG:boto:Data: 2024-04-12T04:15:47.788 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:47.788 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.788 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.788 DEBUG:boto:Params: {} 2024-04-12T04:15:47.788 DEBUG:boto:Token: None 2024-04-12T04:15:47.788 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/obj34 2024-04-12T04:15:47.788 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:K0Sv+bdz3pIAvKJjbWMq6d6nNkE= 2024-04-12T04:15:47.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:K0Sv+bdz3pIAvKJjbWMq6d6nNkE='} 2024-04-12T04:15:47.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000888c596aca55afe3-006618b573-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.794 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:47.794 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:47.795 DEBUG:boto:Method: HEAD 2024-04-12T04:15:47.795 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:47.795 DEBUG:boto:Data: 2024-04-12T04:15:47.795 DEBUG:boto:Headers: {} 2024-04-12T04:15:47.795 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.795 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.795 DEBUG:boto:Params: {} 2024-04-12T04:15:47.795 DEBUG:boto:Token: None 2024-04-12T04:15:47.795 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/ 2024-04-12T04:15:47.795 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU= 2024-04-12T04:15:47.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU=', 'Content-Length': '0'} 2024-04-12T04:15:47.798 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aff907a2cce75fcf-006618b573-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.798 DEBUG:boto:path=/dkyges-63/obj51 2024-04-12T04:15:47.798 DEBUG:boto:auth_path=/dkyges-63/obj51 2024-04-12T04:15:47.798 DEBUG:boto:Method: PUT 2024-04-12T04:15:47.799 DEBUG:boto:Path: /dkyges-63/obj51 2024-04-12T04:15:47.799 DEBUG:boto:Data: 2024-04-12T04:15:47.799 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:47.799 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.799 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.799 DEBUG:boto:Params: {} 2024-04-12T04:15:47.799 DEBUG:boto:Token: None 2024-04-12T04:15:47.799 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/obj51 2024-04-12T04:15:47.799 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LbiC608nQ3oEqIn/xeBQE7ypGuA= 2024-04-12T04:15:47.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LbiC608nQ3oEqIn/xeBQE7ypGuA='} 2024-04-12T04:15:47.842 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000041c96ee697fb4c34-006618b573-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.843 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:47.843 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:47.843 DEBUG:boto:Method: HEAD 2024-04-12T04:15:47.843 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:47.843 DEBUG:boto:Data: 2024-04-12T04:15:47.843 DEBUG:boto:Headers: {} 2024-04-12T04:15:47.843 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.843 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.843 DEBUG:boto:Params: {} 2024-04-12T04:15:47.844 DEBUG:boto:Token: None 2024-04-12T04:15:47.844 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/ 2024-04-12T04:15:47.844 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU= 2024-04-12T04:15:47.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU=', 'Content-Length': '0'} 2024-04-12T04:15:47.847 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004554bfcee8a76ae6-006618b573-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.847 DEBUG:boto:path=/dkyges-63/obj68 2024-04-12T04:15:47.847 DEBUG:boto:auth_path=/dkyges-63/obj68 2024-04-12T04:15:47.847 DEBUG:boto:Method: PUT 2024-04-12T04:15:47.848 DEBUG:boto:Path: /dkyges-63/obj68 2024-04-12T04:15:47.848 DEBUG:boto:Data: 2024-04-12T04:15:47.848 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:47.848 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.848 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.848 DEBUG:boto:Params: {} 2024-04-12T04:15:47.848 DEBUG:boto:Token: None 2024-04-12T04:15:47.848 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/obj68 2024-04-12T04:15:47.848 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ovxSTGwfKgezSl5QzFDLhhDF/iM= 2024-04-12T04:15:47.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ovxSTGwfKgezSl5QzFDLhhDF/iM='} 2024-04-12T04:15:47.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc679d773b8b1d9c-006618b573-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.856 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:47.856 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:47.856 DEBUG:boto:Method: HEAD 2024-04-12T04:15:47.856 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:47.856 DEBUG:boto:Data: 2024-04-12T04:15:47.856 DEBUG:boto:Headers: {} 2024-04-12T04:15:47.856 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.856 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.856 DEBUG:boto:Params: {} 2024-04-12T04:15:47.856 DEBUG:boto:Token: None 2024-04-12T04:15:47.856 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/ 2024-04-12T04:15:47.857 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU= 2024-04-12T04:15:47.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AFTh/UJaZqUeo7p2y5xIJIKi9PU=', 'Content-Length': '0'} 2024-04-12T04:15:47.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a1f10569258b199-006618b573-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.860 DEBUG:boto:path=/dkyges-63/obj85 2024-04-12T04:15:47.860 DEBUG:boto:auth_path=/dkyges-63/obj85 2024-04-12T04:15:47.860 DEBUG:boto:Method: PUT 2024-04-12T04:15:47.860 DEBUG:boto:Path: /dkyges-63/obj85 2024-04-12T04:15:47.860 DEBUG:boto:Data: 2024-04-12T04:15:47.860 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:47.860 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:47.861 DEBUG:boto:Port: 8000 2024-04-12T04:15:47.861 DEBUG:boto:Params: {} 2024-04-12T04:15:47.861 DEBUG:boto:Token: None 2024-04-12T04:15:47.861 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:47 GMT /dkyges-63/obj85 2024-04-12T04:15:47.861 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C69IRR+a/eZMQsdmW1rMdS+HLAE= 2024-04-12T04:15:47.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C69IRR+a/eZMQsdmW1rMdS+HLAE='} 2024-04-12T04:15:47.907 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d6b178dae0286d0-006618b573-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:47.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:47.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:47.907 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:47.970 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:47.970 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:48.024 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:15:48.024 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-63 2024-04-12T04:15:48.032 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 10 2024-04-12T04:15:48.091 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:48.091 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:48.091 DEBUG:boto:Method: HEAD 2024-04-12T04:15:48.091 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:48.091 DEBUG:boto:Data: 2024-04-12T04:15:48.091 DEBUG:boto:Headers: {} 2024-04-12T04:15:48.091 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.091 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.092 DEBUG:boto:Params: {} 2024-04-12T04:15:48.092 DEBUG:boto:Token: None 2024-04-12T04:15:48.092 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/ 2024-04-12T04:15:48.092 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s= 2024-04-12T04:15:48.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s=', 'Content-Length': '0'} 2024-04-12T04:15:48.095 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e85e6f217e2e04ab-006618b574-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.095 DEBUG:boto:path=/dkyges-63/obj19 2024-04-12T04:15:48.096 DEBUG:boto:auth_path=/dkyges-63/obj19 2024-04-12T04:15:48.096 DEBUG:boto:Method: PUT 2024-04-12T04:15:48.096 DEBUG:boto:Path: /dkyges-63/obj19 2024-04-12T04:15:48.096 DEBUG:boto:Data: 2024-04-12T04:15:48.096 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:48.096 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.096 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.096 DEBUG:boto:Params: {} 2024-04-12T04:15:48.096 DEBUG:boto:Token: None 2024-04-12T04:15:48.096 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/obj19 2024-04-12T04:15:48.097 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Sviasw7eHgxvZ+gwTMjbD6SfqoM= 2024-04-12T04:15:48.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Sviasw7eHgxvZ+gwTMjbD6SfqoM='} 2024-04-12T04:15:48.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4737c8ee22246af-006618b574-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.103 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:48.103 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:48.103 DEBUG:boto:Method: HEAD 2024-04-12T04:15:48.103 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:48.103 DEBUG:boto:Data: 2024-04-12T04:15:48.103 DEBUG:boto:Headers: {} 2024-04-12T04:15:48.103 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.103 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.103 DEBUG:boto:Params: {} 2024-04-12T04:15:48.104 DEBUG:boto:Token: None 2024-04-12T04:15:48.104 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/ 2024-04-12T04:15:48.104 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s= 2024-04-12T04:15:48.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s=', 'Content-Length': '0'} 2024-04-12T04:15:48.106 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004559812ed8fb70ab-006618b574-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.107 DEBUG:boto:path=/dkyges-63/obj38 2024-04-12T04:15:48.107 DEBUG:boto:auth_path=/dkyges-63/obj38 2024-04-12T04:15:48.107 DEBUG:boto:Method: PUT 2024-04-12T04:15:48.107 DEBUG:boto:Path: /dkyges-63/obj38 2024-04-12T04:15:48.107 DEBUG:boto:Data: 2024-04-12T04:15:48.107 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:48.107 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.107 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.107 DEBUG:boto:Params: {} 2024-04-12T04:15:48.107 DEBUG:boto:Token: None 2024-04-12T04:15:48.108 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/obj38 2024-04-12T04:15:48.108 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n8RPDIEX+nU1RiZr5JG52c7GZP4= 2024-04-12T04:15:48.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n8RPDIEX+nU1RiZr5JG52c7GZP4='} 2024-04-12T04:15:48.150 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2ce006899c562e5-006618b574-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.151 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:48.151 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:48.151 DEBUG:boto:Method: HEAD 2024-04-12T04:15:48.151 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:48.151 DEBUG:boto:Data: 2024-04-12T04:15:48.151 DEBUG:boto:Headers: {} 2024-04-12T04:15:48.151 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.151 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.151 DEBUG:boto:Params: {} 2024-04-12T04:15:48.151 DEBUG:boto:Token: None 2024-04-12T04:15:48.151 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/ 2024-04-12T04:15:48.152 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s= 2024-04-12T04:15:48.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s=', 'Content-Length': '0'} 2024-04-12T04:15:48.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a90696b6bb09aed1-006618b574-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.154 DEBUG:boto:path=/dkyges-63/obj57 2024-04-12T04:15:48.155 DEBUG:boto:auth_path=/dkyges-63/obj57 2024-04-12T04:15:48.155 DEBUG:boto:Method: PUT 2024-04-12T04:15:48.155 DEBUG:boto:Path: /dkyges-63/obj57 2024-04-12T04:15:48.155 DEBUG:boto:Data: 2024-04-12T04:15:48.155 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:48.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.155 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.155 DEBUG:boto:Params: {} 2024-04-12T04:15:48.155 DEBUG:boto:Token: None 2024-04-12T04:15:48.156 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/obj57 2024-04-12T04:15:48.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qTa31flQfDNe2XMF11F1K7kXSxA= 2024-04-12T04:15:48.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qTa31flQfDNe2XMF11F1K7kXSxA='} 2024-04-12T04:15:48.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000005905aa97a1f38da-006618b574-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.160 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:48.160 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:48.160 DEBUG:boto:Method: HEAD 2024-04-12T04:15:48.161 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:48.161 DEBUG:boto:Data: 2024-04-12T04:15:48.161 DEBUG:boto:Headers: {} 2024-04-12T04:15:48.161 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.161 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.161 DEBUG:boto:Params: {} 2024-04-12T04:15:48.161 DEBUG:boto:Token: None 2024-04-12T04:15:48.161 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/ 2024-04-12T04:15:48.161 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s= 2024-04-12T04:15:48.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s=', 'Content-Length': '0'} 2024-04-12T04:15:48.164 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ae04fd1cdb04864-006618b574-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.164 DEBUG:boto:path=/dkyges-63/obj76 2024-04-12T04:15:48.164 DEBUG:boto:auth_path=/dkyges-63/obj76 2024-04-12T04:15:48.165 DEBUG:boto:Method: PUT 2024-04-12T04:15:48.165 DEBUG:boto:Path: /dkyges-63/obj76 2024-04-12T04:15:48.165 DEBUG:boto:Data: 2024-04-12T04:15:48.165 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:48.165 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.165 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.165 DEBUG:boto:Params: {} 2024-04-12T04:15:48.165 DEBUG:boto:Token: None 2024-04-12T04:15:48.165 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/obj76 2024-04-12T04:15:48.165 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:O0umQSg4YUNvuWiLPYJzdhBWbCs= 2024-04-12T04:15:48.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:O0umQSg4YUNvuWiLPYJzdhBWbCs='} 2024-04-12T04:15:48.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000038e892b6f64ecb91-006618b574-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.211 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:48.211 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:48.211 DEBUG:boto:Method: HEAD 2024-04-12T04:15:48.211 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:48.211 DEBUG:boto:Data: 2024-04-12T04:15:48.211 DEBUG:boto:Headers: {} 2024-04-12T04:15:48.211 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.211 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.211 DEBUG:boto:Params: {} 2024-04-12T04:15:48.211 DEBUG:boto:Token: None 2024-04-12T04:15:48.212 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/ 2024-04-12T04:15:48.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s= 2024-04-12T04:15:48.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s=', 'Content-Length': '0'} 2024-04-12T04:15:48.215 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc2fe66a08c43a92-006618b574-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.215 DEBUG:boto:path=/dkyges-63/obj95 2024-04-12T04:15:48.215 DEBUG:boto:auth_path=/dkyges-63/obj95 2024-04-12T04:15:48.215 DEBUG:boto:Method: PUT 2024-04-12T04:15:48.215 DEBUG:boto:Path: /dkyges-63/obj95 2024-04-12T04:15:48.215 DEBUG:boto:Data: 2024-04-12T04:15:48.215 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:48.216 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.216 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.216 DEBUG:boto:Params: {} 2024-04-12T04:15:48.216 DEBUG:boto:Token: None 2024-04-12T04:15:48.216 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/obj95 2024-04-12T04:15:48.216 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bxZm0IBLRdSEIz1jkt2y/EKkcl0= 2024-04-12T04:15:48.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bxZm0IBLRdSEIz1jkt2y/EKkcl0='} 2024-04-12T04:15:48.221 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0e7aa476f1d35f0-006618b574-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:48.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:48.222 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:48.284 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:48.284 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:48.337 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:15:48.337 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-63 2024-04-12T04:15:48.345 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 15 2024-04-12T04:15:48.401 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:48.401 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:48.402 DEBUG:boto:Method: HEAD 2024-04-12T04:15:48.402 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:48.402 DEBUG:boto:Data: 2024-04-12T04:15:48.402 DEBUG:boto:Headers: {} 2024-04-12T04:15:48.402 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.402 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.403 DEBUG:boto:Params: {} 2024-04-12T04:15:48.403 DEBUG:boto:Token: None 2024-04-12T04:15:48.403 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/ 2024-04-12T04:15:48.403 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s= 2024-04-12T04:15:48.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s=', 'Content-Length': '0'} 2024-04-12T04:15:48.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078f8e07152ccb9a3-006618b574-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.408 DEBUG:boto:path=/dkyges-63/obj23 2024-04-12T04:15:48.408 DEBUG:boto:auth_path=/dkyges-63/obj23 2024-04-12T04:15:48.408 DEBUG:boto:Method: PUT 2024-04-12T04:15:48.408 DEBUG:boto:Path: /dkyges-63/obj23 2024-04-12T04:15:48.409 DEBUG:boto:Data: 2024-04-12T04:15:48.409 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:48.409 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.409 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.409 DEBUG:boto:Params: {} 2024-04-12T04:15:48.409 DEBUG:boto:Token: None 2024-04-12T04:15:48.410 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/obj23 2024-04-12T04:15:48.410 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:y0nZLvgVGkEN7IB3axtyHFIk8Ck= 2024-04-12T04:15:48.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y0nZLvgVGkEN7IB3axtyHFIk8Ck='} 2024-04-12T04:15:48.457 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a304332a4589d62-006618b574-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.458 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:48.458 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:48.458 DEBUG:boto:Method: HEAD 2024-04-12T04:15:48.458 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:48.459 DEBUG:boto:Data: 2024-04-12T04:15:48.459 DEBUG:boto:Headers: {} 2024-04-12T04:15:48.459 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.459 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.459 DEBUG:boto:Params: {} 2024-04-12T04:15:48.459 DEBUG:boto:Token: None 2024-04-12T04:15:48.459 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/ 2024-04-12T04:15:48.459 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s= 2024-04-12T04:15:48.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s=', 'Content-Length': '0'} 2024-04-12T04:15:48.466 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8b493aaabc5a905-006618b574-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.466 DEBUG:boto:path=/dkyges-63/obj46 2024-04-12T04:15:48.467 DEBUG:boto:auth_path=/dkyges-63/obj46 2024-04-12T04:15:48.467 DEBUG:boto:Method: PUT 2024-04-12T04:15:48.467 DEBUG:boto:Path: /dkyges-63/obj46 2024-04-12T04:15:48.467 DEBUG:boto:Data: 2024-04-12T04:15:48.467 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:48.467 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.467 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.467 DEBUG:boto:Params: {} 2024-04-12T04:15:48.467 DEBUG:boto:Token: None 2024-04-12T04:15:48.467 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/obj46 2024-04-12T04:15:48.468 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HO7Q6e6Z8ryAP6mx18noTWwV64M= 2024-04-12T04:15:48.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HO7Q6e6Z8ryAP6mx18noTWwV64M='} 2024-04-12T04:15:48.475 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d35e4d3c5ae69f5-006618b574-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.475 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:48.475 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:48.475 DEBUG:boto:Method: HEAD 2024-04-12T04:15:48.475 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:48.475 DEBUG:boto:Data: 2024-04-12T04:15:48.475 DEBUG:boto:Headers: {} 2024-04-12T04:15:48.476 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.476 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.476 DEBUG:boto:Params: {} 2024-04-12T04:15:48.476 DEBUG:boto:Token: None 2024-04-12T04:15:48.476 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/ 2024-04-12T04:15:48.476 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s= 2024-04-12T04:15:48.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s=', 'Content-Length': '0'} 2024-04-12T04:15:48.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067ebf40bc81820a9-006618b574-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.480 DEBUG:boto:path=/dkyges-63/obj69 2024-04-12T04:15:48.480 DEBUG:boto:auth_path=/dkyges-63/obj69 2024-04-12T04:15:48.480 DEBUG:boto:Method: PUT 2024-04-12T04:15:48.480 DEBUG:boto:Path: /dkyges-63/obj69 2024-04-12T04:15:48.480 DEBUG:boto:Data: 2024-04-12T04:15:48.480 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:48.480 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.480 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.481 DEBUG:boto:Params: {} 2024-04-12T04:15:48.481 DEBUG:boto:Token: None 2024-04-12T04:15:48.481 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/obj69 2024-04-12T04:15:48.481 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xkBTQiLE42nz7rdSAoAPBqNDI0s= 2024-04-12T04:15:48.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xkBTQiLE42nz7rdSAoAPBqNDI0s='} 2024-04-12T04:15:48.527 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c89202bfb1a02813-006618b574-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.527 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:48.527 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:48.527 DEBUG:boto:Method: HEAD 2024-04-12T04:15:48.527 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:48.528 DEBUG:boto:Data: 2024-04-12T04:15:48.528 DEBUG:boto:Headers: {} 2024-04-12T04:15:48.528 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.528 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.528 DEBUG:boto:Params: {} 2024-04-12T04:15:48.528 DEBUG:boto:Token: None 2024-04-12T04:15:48.528 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/ 2024-04-12T04:15:48.528 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s= 2024-04-12T04:15:48.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s=', 'Content-Length': '0'} 2024-04-12T04:15:48.532 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c9b31966f3e210f-006618b574-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.533 DEBUG:boto:path=/dkyges-63/obj92 2024-04-12T04:15:48.533 DEBUG:boto:auth_path=/dkyges-63/obj92 2024-04-12T04:15:48.533 DEBUG:boto:Method: PUT 2024-04-12T04:15:48.533 DEBUG:boto:Path: /dkyges-63/obj92 2024-04-12T04:15:48.533 DEBUG:boto:Data: 2024-04-12T04:15:48.533 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:48.534 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.534 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.534 DEBUG:boto:Params: {} 2024-04-12T04:15:48.534 DEBUG:boto:Token: None 2024-04-12T04:15:48.535 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/obj92 2024-04-12T04:15:48.535 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gEnOBK4Ahebfw1x9hD2KFAhgWwQ= 2024-04-12T04:15:48.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gEnOBK4Ahebfw1x9hD2KFAhgWwQ='} 2024-04-12T04:15:48.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075646ac051a91dcd-006618b574-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.542 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:48.543 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:48.543 DEBUG:boto:Method: HEAD 2024-04-12T04:15:48.543 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:48.543 DEBUG:boto:Data: 2024-04-12T04:15:48.543 DEBUG:boto:Headers: {} 2024-04-12T04:15:48.543 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.543 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.544 DEBUG:boto:Params: {} 2024-04-12T04:15:48.544 DEBUG:boto:Token: None 2024-04-12T04:15:48.544 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/ 2024-04-12T04:15:48.544 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s= 2024-04-12T04:15:48.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s=', 'Content-Length': '0'} 2024-04-12T04:15:48.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d03d64f79c6d3c21-006618b574-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.549 DEBUG:boto:path=/dkyges-63/obj115 2024-04-12T04:15:48.549 DEBUG:boto:auth_path=/dkyges-63/obj115 2024-04-12T04:15:48.549 DEBUG:boto:Method: PUT 2024-04-12T04:15:48.549 DEBUG:boto:Path: /dkyges-63/obj115 2024-04-12T04:15:48.549 DEBUG:boto:Data: 2024-04-12T04:15:48.549 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:48.549 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.550 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.550 DEBUG:boto:Params: {} 2024-04-12T04:15:48.550 DEBUG:boto:Token: None 2024-04-12T04:15:48.550 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/obj115 2024-04-12T04:15:48.550 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hs00MCNPwoXg/KZ+VVspi9Xgeuo= 2024-04-12T04:15:48.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hs00MCNPwoXg/KZ+VVspi9Xgeuo='} 2024-04-12T04:15:48.595 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4cc40d8483d5551-006618b574-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:48.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:48.596 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:48.659 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:48.660 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:48.717 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:15:48.717 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-63 2024-04-12T04:15:48.728 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 20 2024-04-12T04:15:48.790 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:48.790 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:48.790 DEBUG:boto:Method: HEAD 2024-04-12T04:15:48.790 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:48.790 DEBUG:boto:Data: 2024-04-12T04:15:48.791 DEBUG:boto:Headers: {} 2024-04-12T04:15:48.791 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.791 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.791 DEBUG:boto:Params: {} 2024-04-12T04:15:48.792 DEBUG:boto:Token: None 2024-04-12T04:15:48.792 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/ 2024-04-12T04:15:48.792 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s= 2024-04-12T04:15:48.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s=', 'Content-Length': '0'} 2024-04-12T04:15:48.797 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082fc19fcde2e0a62-006618b574-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.798 DEBUG:boto:path=/dkyges-63/obj29 2024-04-12T04:15:48.798 DEBUG:boto:auth_path=/dkyges-63/obj29 2024-04-12T04:15:48.798 DEBUG:boto:Method: PUT 2024-04-12T04:15:48.798 DEBUG:boto:Path: /dkyges-63/obj29 2024-04-12T04:15:48.798 DEBUG:boto:Data: 2024-04-12T04:15:48.799 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:48.799 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.799 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.799 DEBUG:boto:Params: {} 2024-04-12T04:15:48.799 DEBUG:boto:Token: None 2024-04-12T04:15:48.799 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/obj29 2024-04-12T04:15:48.800 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Mc0MSef8rbufaHWpaUUewlG6M2k= 2024-04-12T04:15:48.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Mc0MSef8rbufaHWpaUUewlG6M2k='} 2024-04-12T04:15:48.806 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f44d5c4c349ba057-006618b574-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.806 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:48.806 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:48.807 DEBUG:boto:Method: HEAD 2024-04-12T04:15:48.807 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:48.807 DEBUG:boto:Data: 2024-04-12T04:15:48.807 DEBUG:boto:Headers: {} 2024-04-12T04:15:48.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.807 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.807 DEBUG:boto:Params: {} 2024-04-12T04:15:48.808 DEBUG:boto:Token: None 2024-04-12T04:15:48.808 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/ 2024-04-12T04:15:48.808 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s= 2024-04-12T04:15:48.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s=', 'Content-Length': '0'} 2024-04-12T04:15:48.813 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bba2a722789ef39-006618b574-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.814 DEBUG:boto:path=/dkyges-63/obj58 2024-04-12T04:15:48.814 DEBUG:boto:auth_path=/dkyges-63/obj58 2024-04-12T04:15:48.814 DEBUG:boto:Method: PUT 2024-04-12T04:15:48.815 DEBUG:boto:Path: /dkyges-63/obj58 2024-04-12T04:15:48.815 DEBUG:boto:Data: 2024-04-12T04:15:48.815 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:48.815 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.815 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.816 DEBUG:boto:Params: {} 2024-04-12T04:15:48.816 DEBUG:boto:Token: None 2024-04-12T04:15:48.816 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/obj58 2024-04-12T04:15:48.817 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:40SuI4ZNS3HiLat79edlKIsj9Gc= 2024-04-12T04:15:48.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:40SuI4ZNS3HiLat79edlKIsj9Gc='} 2024-04-12T04:15:48.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005639e6a90dc686e4-006618b574-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.863 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:48.863 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:48.864 DEBUG:boto:Method: HEAD 2024-04-12T04:15:48.864 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:48.864 DEBUG:boto:Data: 2024-04-12T04:15:48.864 DEBUG:boto:Headers: {} 2024-04-12T04:15:48.864 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.864 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.864 DEBUG:boto:Params: {} 2024-04-12T04:15:48.865 DEBUG:boto:Token: None 2024-04-12T04:15:48.865 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/ 2024-04-12T04:15:48.865 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s= 2024-04-12T04:15:48.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s=', 'Content-Length': '0'} 2024-04-12T04:15:48.871 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099b56ec7c624c705-006618b574-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.871 DEBUG:boto:path=/dkyges-63/obj87 2024-04-12T04:15:48.872 DEBUG:boto:auth_path=/dkyges-63/obj87 2024-04-12T04:15:48.872 DEBUG:boto:Method: PUT 2024-04-12T04:15:48.872 DEBUG:boto:Path: /dkyges-63/obj87 2024-04-12T04:15:48.872 DEBUG:boto:Data: 2024-04-12T04:15:48.872 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:48.872 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.872 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.872 DEBUG:boto:Params: {} 2024-04-12T04:15:48.873 DEBUG:boto:Token: None 2024-04-12T04:15:48.873 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/obj87 2024-04-12T04:15:48.873 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U1RZvS/Arvh4Toxs0glTTgL+Fp0= 2024-04-12T04:15:48.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U1RZvS/Arvh4Toxs0glTTgL+Fp0='} 2024-04-12T04:15:48.880 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe7a5dddad6b9356-006618b574-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.880 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:48.880 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:48.881 DEBUG:boto:Method: HEAD 2024-04-12T04:15:48.881 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:48.881 DEBUG:boto:Data: 2024-04-12T04:15:48.881 DEBUG:boto:Headers: {} 2024-04-12T04:15:48.881 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.881 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.881 DEBUG:boto:Params: {} 2024-04-12T04:15:48.881 DEBUG:boto:Token: None 2024-04-12T04:15:48.881 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/ 2024-04-12T04:15:48.882 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s= 2024-04-12T04:15:48.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s=', 'Content-Length': '0'} 2024-04-12T04:15:48.885 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3adf6267541a01f-006618b574-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.886 DEBUG:boto:path=/dkyges-63/obj116 2024-04-12T04:15:48.886 DEBUG:boto:auth_path=/dkyges-63/obj116 2024-04-12T04:15:48.886 DEBUG:boto:Method: PUT 2024-04-12T04:15:48.886 DEBUG:boto:Path: /dkyges-63/obj116 2024-04-12T04:15:48.886 DEBUG:boto:Data: 2024-04-12T04:15:48.886 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:48.886 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.887 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.887 DEBUG:boto:Params: {} 2024-04-12T04:15:48.887 DEBUG:boto:Token: None 2024-04-12T04:15:48.887 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/obj116 2024-04-12T04:15:48.887 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xmc8LXWEWgGo5BRdPMwpDsDJ1JE= 2024-04-12T04:15:48.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xmc8LXWEWgGo5BRdPMwpDsDJ1JE='} 2024-04-12T04:15:48.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000fdc8c8671a9c44c-006618b574-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.931 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:48.931 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:48.931 DEBUG:boto:Method: HEAD 2024-04-12T04:15:48.931 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:48.931 DEBUG:boto:Data: 2024-04-12T04:15:48.931 DEBUG:boto:Headers: {} 2024-04-12T04:15:48.932 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.932 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.932 DEBUG:boto:Params: {} 2024-04-12T04:15:48.932 DEBUG:boto:Token: None 2024-04-12T04:15:48.932 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/ 2024-04-12T04:15:48.932 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s= 2024-04-12T04:15:48.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TgFxRpCE0JD9vGNY06JSEbKA9/s=', 'Content-Length': '0'} 2024-04-12T04:15:48.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009dcb01207e44624a-006618b574-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.936 DEBUG:boto:path=/dkyges-63/obj145 2024-04-12T04:15:48.936 DEBUG:boto:auth_path=/dkyges-63/obj145 2024-04-12T04:15:48.937 DEBUG:boto:Method: PUT 2024-04-12T04:15:48.937 DEBUG:boto:Path: /dkyges-63/obj145 2024-04-12T04:15:48.937 DEBUG:boto:Data: 2024-04-12T04:15:48.937 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:48.937 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:48.937 DEBUG:boto:Port: 8000 2024-04-12T04:15:48.937 DEBUG:boto:Params: {} 2024-04-12T04:15:48.937 DEBUG:boto:Token: None 2024-04-12T04:15:48.938 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:48 GMT /dkyges-63/obj145 2024-04-12T04:15:48.938 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uXOLlleWz4wmmZrhH2tk1rNKv3A= 2024-04-12T04:15:48.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uXOLlleWz4wmmZrhH2tk1rNKv3A='} 2024-04-12T04:15:48.945 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001748bea5f4966784-006618b574-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:48.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:48.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:48.945 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:49.010 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:49.010 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:49.068 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:15:49.068 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-63 2024-04-12T04:15:49.081 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 25 2024-04-12T04:15:49.153 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:49.153 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:49.154 DEBUG:boto:Method: HEAD 2024-04-12T04:15:49.154 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:49.154 DEBUG:boto:Data: 2024-04-12T04:15:49.154 DEBUG:boto:Headers: {} 2024-04-12T04:15:49.154 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.155 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.155 DEBUG:boto:Params: {} 2024-04-12T04:15:49.155 DEBUG:boto:Token: None 2024-04-12T04:15:49.155 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/ 2024-04-12T04:15:49.155 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA= 2024-04-12T04:15:49.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA=', 'Content-Length': '0'} 2024-04-12T04:15:49.160 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f33f087c9193f521-006618b575-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.161 DEBUG:boto:path=/dkyges-63/obj31 2024-04-12T04:15:49.161 DEBUG:boto:auth_path=/dkyges-63/obj31 2024-04-12T04:15:49.161 DEBUG:boto:Method: PUT 2024-04-12T04:15:49.162 DEBUG:boto:Path: /dkyges-63/obj31 2024-04-12T04:15:49.162 DEBUG:boto:Data: 2024-04-12T04:15:49.162 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:49.162 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.162 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.163 DEBUG:boto:Params: {} 2024-04-12T04:15:49.163 DEBUG:boto:Token: None 2024-04-12T04:15:49.163 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/obj31 2024-04-12T04:15:49.163 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VnvxrSiCoNQrNv7R623+WDlN7Nc= 2024-04-12T04:15:49.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VnvxrSiCoNQrNv7R623+WDlN7Nc='} 2024-04-12T04:15:49.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf8c39873ffc1b66-006618b575-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.207 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:49.207 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:49.207 DEBUG:boto:Method: HEAD 2024-04-12T04:15:49.208 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:49.208 DEBUG:boto:Data: 2024-04-12T04:15:49.208 DEBUG:boto:Headers: {} 2024-04-12T04:15:49.208 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.208 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.208 DEBUG:boto:Params: {} 2024-04-12T04:15:49.208 DEBUG:boto:Token: None 2024-04-12T04:15:49.208 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/ 2024-04-12T04:15:49.209 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA= 2024-04-12T04:15:49.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA=', 'Content-Length': '0'} 2024-04-12T04:15:49.214 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017aa1cf5e3fe9b08-006618b575-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.215 DEBUG:boto:path=/dkyges-63/obj62 2024-04-12T04:15:49.215 DEBUG:boto:auth_path=/dkyges-63/obj62 2024-04-12T04:15:49.215 DEBUG:boto:Method: PUT 2024-04-12T04:15:49.215 DEBUG:boto:Path: /dkyges-63/obj62 2024-04-12T04:15:49.215 DEBUG:boto:Data: 2024-04-12T04:15:49.216 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:49.216 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.216 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.216 DEBUG:boto:Params: {} 2024-04-12T04:15:49.216 DEBUG:boto:Token: None 2024-04-12T04:15:49.216 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/obj62 2024-04-12T04:15:49.217 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:dEyh8TpHtir/JW9eTpt7cvT6MEE= 2024-04-12T04:15:49.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dEyh8TpHtir/JW9eTpt7cvT6MEE='} 2024-04-12T04:15:49.224 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000377cb15e58cf5d07-006618b575-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.224 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:49.224 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:49.224 DEBUG:boto:Method: HEAD 2024-04-12T04:15:49.224 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:49.225 DEBUG:boto:Data: 2024-04-12T04:15:49.225 DEBUG:boto:Headers: {} 2024-04-12T04:15:49.225 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.225 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.225 DEBUG:boto:Params: {} 2024-04-12T04:15:49.225 DEBUG:boto:Token: None 2024-04-12T04:15:49.225 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/ 2024-04-12T04:15:49.225 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA= 2024-04-12T04:15:49.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA=', 'Content-Length': '0'} 2024-04-12T04:15:49.230 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075f5e84beef5cb35-006618b575-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.230 DEBUG:boto:path=/dkyges-63/obj93 2024-04-12T04:15:49.230 DEBUG:boto:auth_path=/dkyges-63/obj93 2024-04-12T04:15:49.230 DEBUG:boto:Method: PUT 2024-04-12T04:15:49.230 DEBUG:boto:Path: /dkyges-63/obj93 2024-04-12T04:15:49.230 DEBUG:boto:Data: 2024-04-12T04:15:49.231 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:49.231 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.231 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.231 DEBUG:boto:Params: {} 2024-04-12T04:15:49.231 DEBUG:boto:Token: None 2024-04-12T04:15:49.231 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/obj93 2024-04-12T04:15:49.231 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BC9AySGlkLyyd8zI3QUVq5L2fC0= 2024-04-12T04:15:49.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BC9AySGlkLyyd8zI3QUVq5L2fC0='} 2024-04-12T04:15:49.275 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000133b2c4656c33541-006618b575-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.276 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:49.276 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:49.276 DEBUG:boto:Method: HEAD 2024-04-12T04:15:49.276 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:49.276 DEBUG:boto:Data: 2024-04-12T04:15:49.276 DEBUG:boto:Headers: {} 2024-04-12T04:15:49.276 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.276 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.277 DEBUG:boto:Params: {} 2024-04-12T04:15:49.277 DEBUG:boto:Token: None 2024-04-12T04:15:49.277 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/ 2024-04-12T04:15:49.277 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA= 2024-04-12T04:15:49.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA=', 'Content-Length': '0'} 2024-04-12T04:15:49.281 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e215385eb5fe60da-006618b575-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.281 DEBUG:boto:path=/dkyges-63/obj124 2024-04-12T04:15:49.282 DEBUG:boto:auth_path=/dkyges-63/obj124 2024-04-12T04:15:49.282 DEBUG:boto:Method: PUT 2024-04-12T04:15:49.282 DEBUG:boto:Path: /dkyges-63/obj124 2024-04-12T04:15:49.283 DEBUG:boto:Data: 2024-04-12T04:15:49.283 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:49.283 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.283 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.283 DEBUG:boto:Params: {} 2024-04-12T04:15:49.283 DEBUG:boto:Token: None 2024-04-12T04:15:49.284 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/obj124 2024-04-12T04:15:49.284 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yH2wK/XpbrU2LoTWmSajM01209M= 2024-04-12T04:15:49.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yH2wK/XpbrU2LoTWmSajM01209M='} 2024-04-12T04:15:49.291 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d1d630c7c2e3d98a-006618b575-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.291 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:49.291 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:49.291 DEBUG:boto:Method: HEAD 2024-04-12T04:15:49.291 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:49.291 DEBUG:boto:Data: 2024-04-12T04:15:49.291 DEBUG:boto:Headers: {} 2024-04-12T04:15:49.292 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.292 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.292 DEBUG:boto:Params: {} 2024-04-12T04:15:49.292 DEBUG:boto:Token: None 2024-04-12T04:15:49.292 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/ 2024-04-12T04:15:49.292 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA= 2024-04-12T04:15:49.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA=', 'Content-Length': '0'} 2024-04-12T04:15:49.297 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008541ae3c21276592-006618b575-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.297 DEBUG:boto:path=/dkyges-63/obj155 2024-04-12T04:15:49.297 DEBUG:boto:auth_path=/dkyges-63/obj155 2024-04-12T04:15:49.298 DEBUG:boto:Method: PUT 2024-04-12T04:15:49.298 DEBUG:boto:Path: /dkyges-63/obj155 2024-04-12T04:15:49.298 DEBUG:boto:Data: 2024-04-12T04:15:49.298 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:49.298 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.298 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.299 DEBUG:boto:Params: {} 2024-04-12T04:15:49.299 DEBUG:boto:Token: None 2024-04-12T04:15:49.299 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/obj155 2024-04-12T04:15:49.299 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xl1ItUt/trzsnZvGOub/AFlWGG4= 2024-04-12T04:15:49.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xl1ItUt/trzsnZvGOub/AFlWGG4='} 2024-04-12T04:15:49.347 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2141ed4459b715f-006618b575-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:49.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:49.348 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:49.414 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:49.414 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:49.476 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:15:49.477 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-63 2024-04-12T04:15:49.490 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 30 2024-04-12T04:15:49.567 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:49.567 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:49.568 DEBUG:boto:Method: HEAD 2024-04-12T04:15:49.568 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:49.568 DEBUG:boto:Data: 2024-04-12T04:15:49.568 DEBUG:boto:Headers: {} 2024-04-12T04:15:49.568 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.568 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.568 DEBUG:boto:Params: {} 2024-04-12T04:15:49.568 DEBUG:boto:Token: None 2024-04-12T04:15:49.568 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/ 2024-04-12T04:15:49.569 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA= 2024-04-12T04:15:49.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA=', 'Content-Length': '0'} 2024-04-12T04:15:49.573 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ff376e2d236e90a-006618b575-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.573 DEBUG:boto:path=/dkyges-63/obj37 2024-04-12T04:15:49.573 DEBUG:boto:auth_path=/dkyges-63/obj37 2024-04-12T04:15:49.573 DEBUG:boto:Method: PUT 2024-04-12T04:15:49.573 DEBUG:boto:Path: /dkyges-63/obj37 2024-04-12T04:15:49.573 DEBUG:boto:Data: 2024-04-12T04:15:49.574 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:49.574 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.574 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.574 DEBUG:boto:Params: {} 2024-04-12T04:15:49.574 DEBUG:boto:Token: None 2024-04-12T04:15:49.574 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/obj37 2024-04-12T04:15:49.574 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JLTj54ZYu66hC08R3AvYMl6RfGM= 2024-04-12T04:15:49.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JLTj54ZYu66hC08R3AvYMl6RfGM='} 2024-04-12T04:15:49.580 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000499d22c7b0397645-006618b575-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.580 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:49.581 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:49.581 DEBUG:boto:Method: HEAD 2024-04-12T04:15:49.581 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:49.581 DEBUG:boto:Data: 2024-04-12T04:15:49.581 DEBUG:boto:Headers: {} 2024-04-12T04:15:49.581 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.581 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.581 DEBUG:boto:Params: {} 2024-04-12T04:15:49.581 DEBUG:boto:Token: None 2024-04-12T04:15:49.581 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/ 2024-04-12T04:15:49.581 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA= 2024-04-12T04:15:49.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA=', 'Content-Length': '0'} 2024-04-12T04:15:49.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008453d1e2a4a3e63f-006618b575-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.586 DEBUG:boto:path=/dkyges-63/obj74 2024-04-12T04:15:49.586 DEBUG:boto:auth_path=/dkyges-63/obj74 2024-04-12T04:15:49.586 DEBUG:boto:Method: PUT 2024-04-12T04:15:49.586 DEBUG:boto:Path: /dkyges-63/obj74 2024-04-12T04:15:49.586 DEBUG:boto:Data: 2024-04-12T04:15:49.586 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:49.586 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.586 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.587 DEBUG:boto:Params: {} 2024-04-12T04:15:49.587 DEBUG:boto:Token: None 2024-04-12T04:15:49.587 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/obj74 2024-04-12T04:15:49.587 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PgsdVwYTyqwJSzSTJxjAAkMmCfY= 2024-04-12T04:15:49.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PgsdVwYTyqwJSzSTJxjAAkMmCfY='} 2024-04-12T04:15:49.635 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f21262a7e1d04700-006618b575-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.635 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:49.635 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:49.635 DEBUG:boto:Method: HEAD 2024-04-12T04:15:49.635 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:49.635 DEBUG:boto:Data: 2024-04-12T04:15:49.635 DEBUG:boto:Headers: {} 2024-04-12T04:15:49.635 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.635 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.635 DEBUG:boto:Params: {} 2024-04-12T04:15:49.636 DEBUG:boto:Token: None 2024-04-12T04:15:49.636 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/ 2024-04-12T04:15:49.636 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA= 2024-04-12T04:15:49.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA=', 'Content-Length': '0'} 2024-04-12T04:15:49.640 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000764d010019c70b96-006618b575-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.641 DEBUG:boto:path=/dkyges-63/obj111 2024-04-12T04:15:49.641 DEBUG:boto:auth_path=/dkyges-63/obj111 2024-04-12T04:15:49.641 DEBUG:boto:Method: PUT 2024-04-12T04:15:49.641 DEBUG:boto:Path: /dkyges-63/obj111 2024-04-12T04:15:49.641 DEBUG:boto:Data: 2024-04-12T04:15:49.641 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:49.641 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.641 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.641 DEBUG:boto:Params: {} 2024-04-12T04:15:49.641 DEBUG:boto:Token: None 2024-04-12T04:15:49.642 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/obj111 2024-04-12T04:15:49.642 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:brkbBRHejZWI8lAq5B96sPSZGQM= 2024-04-12T04:15:49.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:brkbBRHejZWI8lAq5B96sPSZGQM='} 2024-04-12T04:15:49.649 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d412eabdfb9a7ef1-006618b575-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.650 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:49.650 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:49.650 DEBUG:boto:Method: HEAD 2024-04-12T04:15:49.650 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:49.650 DEBUG:boto:Data: 2024-04-12T04:15:49.650 DEBUG:boto:Headers: {} 2024-04-12T04:15:49.650 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.650 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.650 DEBUG:boto:Params: {} 2024-04-12T04:15:49.650 DEBUG:boto:Token: None 2024-04-12T04:15:49.651 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/ 2024-04-12T04:15:49.651 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA= 2024-04-12T04:15:49.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA=', 'Content-Length': '0'} 2024-04-12T04:15:49.655 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1a5af050277e009-006618b575-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.655 DEBUG:boto:path=/dkyges-63/obj148 2024-04-12T04:15:49.655 DEBUG:boto:auth_path=/dkyges-63/obj148 2024-04-12T04:15:49.655 DEBUG:boto:Method: PUT 2024-04-12T04:15:49.655 DEBUG:boto:Path: /dkyges-63/obj148 2024-04-12T04:15:49.655 DEBUG:boto:Data: 2024-04-12T04:15:49.655 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:49.656 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.656 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.656 DEBUG:boto:Params: {} 2024-04-12T04:15:49.656 DEBUG:boto:Token: None 2024-04-12T04:15:49.656 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/obj148 2024-04-12T04:15:49.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ysmjcF52AOyKTVxJgnjDnXGa/DE= 2024-04-12T04:15:49.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ysmjcF52AOyKTVxJgnjDnXGa/DE='} 2024-04-12T04:15:49.699 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009960a02247e25a57-006618b575-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.699 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:15:49.699 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:15:49.700 DEBUG:boto:Method: HEAD 2024-04-12T04:15:49.700 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:15:49.700 DEBUG:boto:Data: 2024-04-12T04:15:49.700 DEBUG:boto:Headers: {} 2024-04-12T04:15:49.700 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.701 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.701 DEBUG:boto:Params: {} 2024-04-12T04:15:49.701 DEBUG:boto:Token: None 2024-04-12T04:15:49.701 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/ 2024-04-12T04:15:49.702 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA= 2024-04-12T04:15:49.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mFdmYlYlieiZ9CR0K3MHyclj4bA=', 'Content-Length': '0'} 2024-04-12T04:15:49.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000934c8f5dc180edce-006618b575-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.707 DEBUG:boto:path=/dkyges-63/obj185 2024-04-12T04:15:49.707 DEBUG:boto:auth_path=/dkyges-63/obj185 2024-04-12T04:15:49.707 DEBUG:boto:Method: PUT 2024-04-12T04:15:49.707 DEBUG:boto:Path: /dkyges-63/obj185 2024-04-12T04:15:49.708 DEBUG:boto:Data: 2024-04-12T04:15:49.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:15:49.708 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:15:49.708 DEBUG:boto:Port: 8000 2024-04-12T04:15:49.708 DEBUG:boto:Params: {} 2024-04-12T04:15:49.709 DEBUG:boto:Token: None 2024-04-12T04:15:49.709 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:15:49 GMT /dkyges-63/obj185 2024-04-12T04:15:49.709 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NefFzuWCPMeKAypWVqNZLRQj1rs= 2024-04-12T04:15:49.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:15:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NefFzuWCPMeKAypWVqNZLRQj1rs='} 2024-04-12T04:15:49.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009456d9f5d2a646cf-006618b575-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:15:49.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:49.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:15:49.718 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:15:49.780 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:49.780 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:49.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:49.834+0000 7fe20c28a780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-12T04:15:49.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:49.836+0000 7fe20c28a780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-12T04:15:49.844 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T04:15:49.844 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T04:15:49.848 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T04:15:49.848 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:49.873 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T04:15:49.874 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:15:49.874 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T04:15:49.875 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi043.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T04:15:50.876 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T04:15:50.903 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:15:50.903 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T04:15:50.904 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-04-12T04:15:50.905 INFO:teuthology.orchestra.run.smithi043.stdout:anonymous 2024-04-12T04:15:50.905 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-63 2024-04-12T04:15:50.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:15:50.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:15:50.905 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:15:51.012 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:15:51.012 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:15:51.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:51.076+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:15:52.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T04:15:56.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:15:56.076+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:16:01.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:16:01.077+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:16:06.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:16:06.078+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:16:11.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:16:11.078+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:16:16.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:16:16.079+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:16:21.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:16:21.080+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:16:26.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:16:26.080+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:16:31.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:16:31.081+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:16:36.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:16:36.082+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:16:41.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:16:41.083+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:16:46.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:16:46.083+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:16:51.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:16:51.084+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:16:56.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:16:56.084+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:17:01.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:17:01.085+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:17:06.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:17:06.085+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:17:11.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:17:11.086+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:17:16.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:17:16.087+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:17:21.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:17:21.087+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:17:26.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:17:26.088+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:17:31.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:17:31.089+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:17:36.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:17:36.089+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:17:41.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:17:41.090+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:17:46.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:17:46.091+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:17:51.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:17:51.091+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:17:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:17:56.092+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:18:01.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:18:01.093+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:18:06.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:18:06.095+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:18:11.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:18:11.094+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:18:16.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:18:16.095+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:18:21.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:18:21.095+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:18:26.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:18:26.096+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:18:31.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:18:31.097+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:18:36.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:18:36.097+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:18:41.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:18:41.098+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:18:46.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:18:46.099+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:18:51.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:18:51.099+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:18:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:18:56.100+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:19:01.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:19:01.100+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:19:06.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:19:06.101+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:19:11.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:19:11.102+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:19:16.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:19:16.103+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:19:21.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:19:21.104+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:19:26.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:19:26.103+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:19:31.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:19:31.104+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:19:36.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:19:36.105+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:19:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:19:41.105+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:19:46.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:19:46.105+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:19:51.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:19:51.105+0000 7f563a896780 1 waiting to reach incremental sync.. 2024-04-12T04:19:56.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:19:56.109+0000 7f563a896780 1 bucket sync caught up with source: 2024-04-12T04:19:56.093 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.73.6, , , , , , 00000000002.90.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.101.6, , , , 00000000001.174.6] 2024-04-12T04:19:56.093 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.73.6, , , , , , 00000000002.90.6, , , , , 00000000002.63.6, , , , , , , , , , , , , , , 00000000001.101.6, , , , 00000000001.174.6] 2024-04-12T04:19:56.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:19:56.109+0000 7f563a896780 0 bucket checkpoint complete 2024-04-12T04:19:56.111 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-63 2024-04-12T04:19:56.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:19:56.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:19:56.112 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:19:56.179 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:19:56.179 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:19:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:19:56.239+0000 7f9e473c6780 1 bucket sync caught up with source: 2024-04-12T04:19:56.222 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, 00000000003.1936.6, 00000000002.3074.6, 00000000002.603.6, 00000000005.605.6, 00000000004.1640.6, 00000000001.2100.6, 00000000001.3669.6, 00000000004.2655.6, 00000000007.2850.6, 00000000006.2653.6] 2024-04-12T04:19:56.222 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, 00000000003.1936.6, 00000000002.3074.6, 00000000002.603.6, 00000000005.605.6, 00000000004.1640.6, 00000000001.2100.6, 00000000001.3669.6, 00000000004.2655.6, 00000000007.2850.6, 00000000006.2653.6] 2024-04-12T04:19:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:19:56.239+0000 7f9e473c6780 0 bucket checkpoint complete 2024-04-12T04:19:56.238 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-63 zones={a2, a1} 2024-04-12T04:19:56.238 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.239 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.239 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.239 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:19:56.239 DEBUG:boto:Data: 2024-04-12T04:19:56.239 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.239 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.239 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.239 DEBUG:boto:Params: {} 2024-04-12T04:19:56.240 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:19:56.240 DEBUG:boto:Token: None 2024-04-12T04:19:56.240 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.240 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CzCuJUWuq5AQ3V2obJVKsFc5IoU= 2024-04-12T04:19:56.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CzCuJUWuq5AQ3V2obJVKsFc5IoU=', 'Content-Length': '0'} 2024-04-12T04:19:56.255 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000069cf85e247810e1a-006618b66c-210b-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.255 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.256 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.256 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.256 DEBUG:boto:Path: /dkyges-63/ 2024-04-12T04:19:56.256 DEBUG:boto:Data: 2024-04-12T04:19:56.256 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.256 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.256 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.256 DEBUG:boto:Params: {} 2024-04-12T04:19:56.256 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:19:56.257 DEBUG:boto:Token: None 2024-04-12T04:19:56.257 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.257 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CzCuJUWuq5AQ3V2obJVKsFc5IoU= 2024-04-12T04:19:56.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CzCuJUWuq5AQ3V2obJVKsFc5IoU=', 'Content-Length': '0'} 2024-04-12T04:19:56.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085317df0d3c2cd2e-006618b66c-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.271 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:19:56.271 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.271 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.271 DEBUG:boto:path=/dkyges-63/?versions&max-keys=999 2024-04-12T04:19:56.271 DEBUG:boto:auth_path=/dkyges-63/?versions&max-keys=999 2024-04-12T04:19:56.271 DEBUG:boto:Method: GET 2024-04-12T04:19:56.272 DEBUG:boto:Path: /dkyges-63/?versions&max-keys=999 2024-04-12T04:19:56.272 DEBUG:boto:Data: 2024-04-12T04:19:56.272 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.272 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.272 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.272 DEBUG:boto:Params: {} 2024-04-12T04:19:56.272 DEBUG:boto:Token: None 2024-04-12T04:19:56.272 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/?versions 2024-04-12T04:19:56.273 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AayzYMaopjzNBKvWZnlBZFGc3IA= 2024-04-12T04:19:56.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AayzYMaopjzNBKvWZnlBZFGc3IA=', 'Content-Length': '0'} 2024-04-12T04:19:56.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da5a61e35e3b26b9-006618b66c-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.275 DEBUG:boto:b'dkyges-63999falseobj11nulltrue2024-04-12T04:15:47.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T04:15:49.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:15:48.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:15:48.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T04:15:49.304Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:15:48.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T04:15:49.677Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T04:15:49.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:15:47.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T04:15:49.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:15:48.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:15:47.472Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:15:48.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:15:48.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T04:15:49.184Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:15:47.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:15:47.809Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T04:15:49.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:15:48.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:15:47.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:15:48.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:15:47.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:15:47.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:15:48.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:15:48.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T04:15:49.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:15:47.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:15:48.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T04:15:49.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:15:48.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:15:47.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:15:48.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:15:48.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T04:15:49.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:15:48.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:19:56.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-12T04:19:56.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T04:19:56.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T04:19:56.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-12T04:19:56.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T04:19:56.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-12T04:19:56.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-12T04:19:56.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:19:56.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-12T04:19:56.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T04:19:56.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:19:56.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T04:19:56.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T04:19:56.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-12T04:19:56.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:19:56.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:19:56.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-12T04:19:56.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T04:19:56.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:19:56.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T04:19:56.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:19:56.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:19:56.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T04:19:56.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T04:19:56.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-12T04:19:56.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:19:56.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T04:19:56.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-12T04:19:56.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T04:19:56.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:19:56.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T04:19:56.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T04:19:56.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-12T04:19:56.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T04:19:56.281 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:19:56.281 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.281 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.281 DEBUG:boto:path=/dkyges-63/?versions&max-keys=999 2024-04-12T04:19:56.281 DEBUG:boto:auth_path=/dkyges-63/?versions&max-keys=999 2024-04-12T04:19:56.281 DEBUG:boto:Method: GET 2024-04-12T04:19:56.281 DEBUG:boto:Path: /dkyges-63/?versions&max-keys=999 2024-04-12T04:19:56.281 DEBUG:boto:Data: 2024-04-12T04:19:56.281 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.281 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.281 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.282 DEBUG:boto:Params: {} 2024-04-12T04:19:56.282 DEBUG:boto:Token: None 2024-04-12T04:19:56.282 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/?versions 2024-04-12T04:19:56.282 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AayzYMaopjzNBKvWZnlBZFGc3IA= 2024-04-12T04:19:56.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AayzYMaopjzNBKvWZnlBZFGc3IA=', 'Content-Length': '0'} 2024-04-12T04:19:56.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e4fd99c5f71c68b-006618b66c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.286 DEBUG:boto:b'dkyges-63999falseobj11nulltrue2024-04-12T04:15:47.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T04:15:49.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:15:48.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:15:48.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T04:15:49.304Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:15:48.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T04:15:49.677Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T04:15:49.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:15:47.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T04:15:49.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:15:48.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:15:47.472Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:15:48.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:15:48.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T04:15:49.184Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:15:47.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:15:47.809Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T04:15:49.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:15:48.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:15:47.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:15:48.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:15:47.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:15:47.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:15:48.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:15:48.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T04:15:49.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:15:47.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:15:48.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T04:15:49.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:15:48.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:15:47.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:15:48.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:15:48.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T04:15:49.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:15:48.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:19:56.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2024-04-12T04:19:56.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2024-04-12T04:19:56.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2024-04-12T04:19:56.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2024-04-12T04:19:56.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2024-04-12T04:19:56.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2024-04-12T04:19:56.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2024-04-12T04:19:56.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2024-04-12T04:19:56.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2024-04-12T04:19:56.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2024-04-12T04:19:56.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:19:56.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2024-04-12T04:19:56.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2024-04-12T04:19:56.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2024-04-12T04:19:56.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:19:56.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2024-04-12T04:19:56.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2024-04-12T04:19:56.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2024-04-12T04:19:56.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:19:56.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2024-04-12T04:19:56.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2024-04-12T04:19:56.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:19:56.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2024-04-12T04:19:56.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2024-04-12T04:19:56.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2024-04-12T04:19:56.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2024-04-12T04:19:56.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2024-04-12T04:19:56.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2024-04-12T04:19:56.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2024-04-12T04:19:56.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2024-04-12T04:19:56.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2024-04-12T04:19:56.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2024-04-12T04:19:56.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2024-04-12T04:19:56.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2024-04-12T04:19:56.291 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.291 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.291 DEBUG:boto:path=/dkyges-63/?versions&max-keys=999 2024-04-12T04:19:56.291 DEBUG:boto:auth_path=/dkyges-63/?versions&max-keys=999 2024-04-12T04:19:56.291 DEBUG:boto:Method: GET 2024-04-12T04:19:56.291 DEBUG:boto:Path: /dkyges-63/?versions&max-keys=999 2024-04-12T04:19:56.291 DEBUG:boto:Data: 2024-04-12T04:19:56.291 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.291 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.291 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.292 DEBUG:boto:Params: {} 2024-04-12T04:19:56.292 DEBUG:boto:Token: None 2024-04-12T04:19:56.292 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/?versions 2024-04-12T04:19:56.292 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AayzYMaopjzNBKvWZnlBZFGc3IA= 2024-04-12T04:19:56.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AayzYMaopjzNBKvWZnlBZFGc3IA=', 'Content-Length': '0'} 2024-04-12T04:19:56.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046d24c6695e3d85e-006618b66c-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.295 DEBUG:boto:b'dkyges-63999falseobj11nulltrue2024-04-12T04:15:47.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T04:15:49.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:15:48.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:15:48.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T04:15:49.304Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:15:48.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T04:15:49.677Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T04:15:49.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:15:47.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T04:15:49.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:15:48.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:15:47.472Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:15:48.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:15:48.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T04:15:49.184Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:15:47.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:15:47.809Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T04:15:49.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:15:48.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:15:47.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:15:48.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:15:47.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:15:47.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:15:48.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:15:48.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T04:15:49.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:15:47.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:15:48.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T04:15:49.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:15:48.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:15:47.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:15:48.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:15:48.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T04:15:49.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:15:48.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.296 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.297 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.297 DEBUG:boto:path=/dkyges-63/?versions&max-keys=999 2024-04-12T04:19:56.297 DEBUG:boto:auth_path=/dkyges-63/?versions&max-keys=999 2024-04-12T04:19:56.297 DEBUG:boto:Method: GET 2024-04-12T04:19:56.297 DEBUG:boto:Path: /dkyges-63/?versions&max-keys=999 2024-04-12T04:19:56.297 DEBUG:boto:Data: 2024-04-12T04:19:56.297 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.297 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.297 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.297 DEBUG:boto:Params: {} 2024-04-12T04:19:56.297 DEBUG:boto:Token: None 2024-04-12T04:19:56.298 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/?versions 2024-04-12T04:19:56.298 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AayzYMaopjzNBKvWZnlBZFGc3IA= 2024-04-12T04:19:56.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AayzYMaopjzNBKvWZnlBZFGc3IA=', 'Content-Length': '0'} 2024-04-12T04:19:56.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acf81f0e77a0244d-006618b66c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.302 DEBUG:boto:b'dkyges-63999falseobj11nulltrue2024-04-12T04:15:47.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2024-04-12T04:15:49.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2024-04-12T04:15:48.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2024-04-12T04:15:48.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2024-04-12T04:15:49.304Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2024-04-12T04:15:48.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2024-04-12T04:15:49.677Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2024-04-12T04:15:49.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2024-04-12T04:15:47.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2024-04-12T04:15:49.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2024-04-12T04:15:48.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:15:47.472Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2024-04-12T04:15:48.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2024-04-12T04:15:48.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2024-04-12T04:15:49.184Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:15:47.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2024-04-12T04:15:47.809Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2024-04-12T04:15:49.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2024-04-12T04:15:48.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:15:47.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2024-04-12T04:15:48.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2024-04-12T04:15:47.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:15:47.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2024-04-12T04:15:48.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2024-04-12T04:15:48.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2024-04-12T04:15:49.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2024-04-12T04:15:47.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2024-04-12T04:15:48.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2024-04-12T04:15:49.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2024-04-12T04:15:48.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2024-04-12T04:15:47.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2024-04-12T04:15:48.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2024-04-12T04:15:48.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2024-04-12T04:15:49.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2024-04-12T04:15:48.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:19:56.305 DEBUG:boto:path=/dkyges-63/obj11 2024-04-12T04:19:56.305 DEBUG:boto:auth_path=/dkyges-63/obj11 2024-04-12T04:19:56.305 DEBUG:boto:path=/dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.305 DEBUG:boto:auth_path=/dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.305 DEBUG:boto:Method: GET 2024-04-12T04:19:56.305 DEBUG:boto:Path: /dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.305 DEBUG:boto:Data: 2024-04-12T04:19:56.305 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.305 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.306 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.306 DEBUG:boto:Params: {} 2024-04-12T04:19:56.306 DEBUG:boto:Token: None 2024-04-12T04:19:56.306 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.306 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wt04ATXOl2ovc3bagzK7b8OAEbU= 2024-04-12T04:19:56.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wt04ATXOl2ovc3bagzK7b8OAEbU=', 'Content-Length': '0'} 2024-04-12T04:19:56.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df52c611e94becc3-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.350 DEBUG:boto:path=/dkyges-63/obj11 2024-04-12T04:19:56.350 DEBUG:boto:auth_path=/dkyges-63/obj11 2024-04-12T04:19:56.350 DEBUG:boto:path=/dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.351 DEBUG:boto:auth_path=/dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.351 DEBUG:boto:Method: GET 2024-04-12T04:19:56.351 DEBUG:boto:Path: /dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.351 DEBUG:boto:Data: 2024-04-12T04:19:56.351 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.351 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.351 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.351 DEBUG:boto:Params: {} 2024-04-12T04:19:56.351 DEBUG:boto:Token: None 2024-04-12T04:19:56.351 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.352 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wt04ATXOl2ovc3bagzK7b8OAEbU= 2024-04-12T04:19:56.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wt04ATXOl2ovc3bagzK7b8OAEbU=', 'Content-Length': '0'} 2024-04-12T04:19:56.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c81fa7eb8c176a56-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.354 DEBUG:boto:path=/dkyges-63/obj11 2024-04-12T04:19:56.354 DEBUG:boto:auth_path=/dkyges-63/obj11 2024-04-12T04:19:56.354 DEBUG:boto:path=/dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.354 DEBUG:boto:auth_path=/dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.354 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.354 DEBUG:boto:Path: /dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.354 DEBUG:boto:Data: 2024-04-12T04:19:56.354 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.355 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.355 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.355 DEBUG:boto:Params: {} 2024-04-12T04:19:56.355 DEBUG:boto:Token: None 2024-04-12T04:19:56.355 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.355 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:527Yg+d69HTvrrl7Y6rmhxgTRog= 2024-04-12T04:19:56.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:527Yg+d69HTvrrl7Y6rmhxgTRog=', 'Content-Length': '0'} 2024-04-12T04:19:56.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000119ee5e9e795875d-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.357 DEBUG:boto:path=/dkyges-63/obj11 2024-04-12T04:19:56.357 DEBUG:boto:auth_path=/dkyges-63/obj11 2024-04-12T04:19:56.357 DEBUG:boto:path=/dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.357 DEBUG:boto:auth_path=/dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.357 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.357 DEBUG:boto:Path: /dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.358 DEBUG:boto:Data: 2024-04-12T04:19:56.358 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.358 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.358 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.358 DEBUG:boto:Params: {} 2024-04-12T04:19:56.358 DEBUG:boto:Token: None 2024-04-12T04:19:56.358 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.358 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:527Yg+d69HTvrrl7Y6rmhxgTRog= 2024-04-12T04:19:56.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:527Yg+d69HTvrrl7Y6rmhxgTRog=', 'Content-Length': '0'} 2024-04-12T04:19:56.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db92b5d68b78c537-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:19:56.361 DEBUG:boto:path=/dkyges-63/obj11 2024-04-12T04:19:56.361 DEBUG:boto:auth_path=/dkyges-63/obj11 2024-04-12T04:19:56.361 DEBUG:boto:path=/dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.361 DEBUG:boto:auth_path=/dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.361 DEBUG:boto:Method: GET 2024-04-12T04:19:56.361 DEBUG:boto:Path: /dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.361 DEBUG:boto:Data: 2024-04-12T04:19:56.361 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.361 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.361 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.362 DEBUG:boto:Params: {} 2024-04-12T04:19:56.362 DEBUG:boto:Token: None 2024-04-12T04:19:56.362 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.362 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wt04ATXOl2ovc3bagzK7b8OAEbU= 2024-04-12T04:19:56.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wt04ATXOl2ovc3bagzK7b8OAEbU=', 'Content-Length': '0'} 2024-04-12T04:19:56.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d69aaf5630735f1f-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.364 DEBUG:boto:path=/dkyges-63/obj11 2024-04-12T04:19:56.364 DEBUG:boto:auth_path=/dkyges-63/obj11 2024-04-12T04:19:56.364 DEBUG:boto:path=/dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.365 DEBUG:boto:auth_path=/dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.365 DEBUG:boto:Method: GET 2024-04-12T04:19:56.365 DEBUG:boto:Path: /dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.365 DEBUG:boto:Data: 2024-04-12T04:19:56.365 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.365 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.365 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.365 DEBUG:boto:Params: {} 2024-04-12T04:19:56.365 DEBUG:boto:Token: None 2024-04-12T04:19:56.365 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj11?versionId=null 2024-04-12T04:19:56.365 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wt04ATXOl2ovc3bagzK7b8OAEbU= 2024-04-12T04:19:56.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wt04ATXOl2ovc3bagzK7b8OAEbU=', 'Content-Length': '0'} 2024-04-12T04:19:56.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ba91d4028dbeb2c-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.368 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.368 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.368 DEBUG:boto:path=/dkyges-63/?prefix=obj11 2024-04-12T04:19:56.368 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj11 2024-04-12T04:19:56.368 DEBUG:boto:Method: GET 2024-04-12T04:19:56.368 DEBUG:boto:Path: /dkyges-63/?prefix=obj11 2024-04-12T04:19:56.368 DEBUG:boto:Data: 2024-04-12T04:19:56.368 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.368 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.368 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.368 DEBUG:boto:Params: {} 2024-04-12T04:19:56.369 DEBUG:boto:Token: None 2024-04-12T04:19:56.369 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.369 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000284608067a5a51fc-006618b66c-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.372 DEBUG:boto:b'dkyges-63obj111000falseobj112024-04-12T04:15:47.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T04:15:49.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:15:48.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:15:48.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.372 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.372 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.373 DEBUG:boto:path=/dkyges-63/?prefix=obj11 2024-04-12T04:19:56.373 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj11 2024-04-12T04:19:56.373 DEBUG:boto:Method: GET 2024-04-12T04:19:56.373 DEBUG:boto:Path: /dkyges-63/?prefix=obj11 2024-04-12T04:19:56.373 DEBUG:boto:Data: 2024-04-12T04:19:56.373 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.373 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.373 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.373 DEBUG:boto:Params: {} 2024-04-12T04:19:56.373 DEBUG:boto:Token: None 2024-04-12T04:19:56.374 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.374 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000088310ac076b2019-006618b66c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.378 DEBUG:boto:b'dkyges-63obj111000falseobj112024-04-12T04:15:47.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T04:15:49.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:15:48.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:15:48.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.379 DEBUG:boto:path=/dkyges-63/obj11 2024-04-12T04:19:56.379 DEBUG:boto:auth_path=/dkyges-63/obj11 2024-04-12T04:19:56.379 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.379 DEBUG:boto:Path: /dkyges-63/obj11 2024-04-12T04:19:56.379 DEBUG:boto:Data: 2024-04-12T04:19:56.379 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.379 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.379 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.379 DEBUG:boto:Params: {} 2024-04-12T04:19:56.380 DEBUG:boto:Token: None 2024-04-12T04:19:56.380 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj11 2024-04-12T04:19:56.380 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Za6iNN8qwRimQZHVo5FXb8Fm1aA= 2024-04-12T04:19:56.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Za6iNN8qwRimQZHVo5FXb8Fm1aA=', 'Content-Length': '0'} 2024-04-12T04:19:56.382 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003826f66fab5f4945-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.382 DEBUG:boto:path=/dkyges-63/obj11 2024-04-12T04:19:56.382 DEBUG:boto:auth_path=/dkyges-63/obj11 2024-04-12T04:19:56.382 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.382 DEBUG:boto:Path: /dkyges-63/obj11 2024-04-12T04:19:56.382 DEBUG:boto:Data: 2024-04-12T04:19:56.382 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.383 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.383 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.383 DEBUG:boto:Params: {} 2024-04-12T04:19:56.383 DEBUG:boto:Token: None 2024-04-12T04:19:56.383 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj11 2024-04-12T04:19:56.383 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Za6iNN8qwRimQZHVo5FXb8Fm1aA= 2024-04-12T04:19:56.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Za6iNN8qwRimQZHVo5FXb8Fm1aA=', 'Content-Length': '0'} 2024-04-12T04:19:56.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f2da412fe27da2e-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:19:56.385 DEBUG:boto:path=/dkyges-63/obj11 2024-04-12T04:19:56.385 DEBUG:boto:auth_path=/dkyges-63/obj11 2024-04-12T04:19:56.386 DEBUG:boto:Method: GET 2024-04-12T04:19:56.386 DEBUG:boto:Path: /dkyges-63/obj11 2024-04-12T04:19:56.386 DEBUG:boto:Data: 2024-04-12T04:19:56.386 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.386 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.386 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.386 DEBUG:boto:Params: {} 2024-04-12T04:19:56.386 DEBUG:boto:Token: None 2024-04-12T04:19:56.386 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj11 2024-04-12T04:19:56.386 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wNMbywGoqbEWT0ZJ2v3jI3J2I4g= 2024-04-12T04:19:56.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wNMbywGoqbEWT0ZJ2v3jI3J2I4g=', 'Content-Length': '0'} 2024-04-12T04:19:56.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c89642dfdfab52fd-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.389 DEBUG:boto:path=/dkyges-63/obj11 2024-04-12T04:19:56.389 DEBUG:boto:auth_path=/dkyges-63/obj11 2024-04-12T04:19:56.389 DEBUG:boto:Method: GET 2024-04-12T04:19:56.389 DEBUG:boto:Path: /dkyges-63/obj11 2024-04-12T04:19:56.389 DEBUG:boto:Data: 2024-04-12T04:19:56.389 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.389 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.389 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.389 DEBUG:boto:Params: {} 2024-04-12T04:19:56.389 DEBUG:boto:Token: None 2024-04-12T04:19:56.390 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj11 2024-04-12T04:19:56.390 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wNMbywGoqbEWT0ZJ2v3jI3J2I4g= 2024-04-12T04:19:56.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wNMbywGoqbEWT0ZJ2v3jI3J2I4g=', 'Content-Length': '0'} 2024-04-12T04:19:56.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023104ebefb032779-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.434 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.434 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.434 DEBUG:boto:path=/dkyges-63/?prefix=obj11 2024-04-12T04:19:56.434 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj11 2024-04-12T04:19:56.435 DEBUG:boto:Method: GET 2024-04-12T04:19:56.435 DEBUG:boto:Path: /dkyges-63/?prefix=obj11 2024-04-12T04:19:56.435 DEBUG:boto:Data: 2024-04-12T04:19:56.435 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.435 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.435 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.435 DEBUG:boto:Params: {} 2024-04-12T04:19:56.435 DEBUG:boto:Token: None 2024-04-12T04:19:56.435 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.435 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbf471e38af3b7f4-006618b66c-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.438 DEBUG:boto:b'dkyges-63obj111000falseobj112024-04-12T04:15:47.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T04:15:49.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:15:48.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:15:48.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.439 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.439 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.439 DEBUG:boto:path=/dkyges-63/?prefix=obj11 2024-04-12T04:19:56.439 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj11 2024-04-12T04:19:56.439 DEBUG:boto:Method: GET 2024-04-12T04:19:56.439 DEBUG:boto:Path: /dkyges-63/?prefix=obj11 2024-04-12T04:19:56.439 DEBUG:boto:Data: 2024-04-12T04:19:56.439 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.439 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.440 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.440 DEBUG:boto:Params: {} 2024-04-12T04:19:56.440 DEBUG:boto:Token: None 2024-04-12T04:19:56.440 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.440 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.444 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000818a5ba2bee4f4e3-006618b66c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.444 DEBUG:boto:b'dkyges-63obj111000falseobj112024-04-12T04:15:47.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112024-04-12T04:15:49.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152024-04-12T04:15:48.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162024-04-12T04:15:48.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T04:19:56.445 DEBUG:boto:path=/dkyges-63/obj111 2024-04-12T04:19:56.445 DEBUG:boto:auth_path=/dkyges-63/obj111 2024-04-12T04:19:56.445 DEBUG:boto:path=/dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.445 DEBUG:boto:auth_path=/dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.445 DEBUG:boto:Method: GET 2024-04-12T04:19:56.445 DEBUG:boto:Path: /dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.445 DEBUG:boto:Data: 2024-04-12T04:19:56.446 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.446 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.446 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.446 DEBUG:boto:Params: {} 2024-04-12T04:19:56.446 DEBUG:boto:Token: None 2024-04-12T04:19:56.446 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.446 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zkQJ01hCLPBvpEDd1mzeZyS4zUw= 2024-04-12T04:19:56.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zkQJ01hCLPBvpEDd1mzeZyS4zUw=', 'Content-Length': '0'} 2024-04-12T04:19:56.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009889e22926990481-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.490 DEBUG:boto:path=/dkyges-63/obj111 2024-04-12T04:19:56.490 DEBUG:boto:auth_path=/dkyges-63/obj111 2024-04-12T04:19:56.490 DEBUG:boto:path=/dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.491 DEBUG:boto:auth_path=/dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.491 DEBUG:boto:Method: GET 2024-04-12T04:19:56.491 DEBUG:boto:Path: /dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.491 DEBUG:boto:Data: 2024-04-12T04:19:56.491 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.491 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.491 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.491 DEBUG:boto:Params: {} 2024-04-12T04:19:56.491 DEBUG:boto:Token: None 2024-04-12T04:19:56.491 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.492 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zkQJ01hCLPBvpEDd1mzeZyS4zUw= 2024-04-12T04:19:56.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zkQJ01hCLPBvpEDd1mzeZyS4zUw=', 'Content-Length': '0'} 2024-04-12T04:19:56.494 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008089b74120978b2d-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.494 DEBUG:boto:path=/dkyges-63/obj111 2024-04-12T04:19:56.494 DEBUG:boto:auth_path=/dkyges-63/obj111 2024-04-12T04:19:56.494 DEBUG:boto:path=/dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.494 DEBUG:boto:auth_path=/dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.494 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.494 DEBUG:boto:Path: /dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.494 DEBUG:boto:Data: 2024-04-12T04:19:56.494 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.495 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.495 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.495 DEBUG:boto:Params: {} 2024-04-12T04:19:56.495 DEBUG:boto:Token: None 2024-04-12T04:19:56.495 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.495 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3D4RkexOf8g9ychW5esgy2zaSZ8= 2024-04-12T04:19:56.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3D4RkexOf8g9ychW5esgy2zaSZ8=', 'Content-Length': '0'} 2024-04-12T04:19:56.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000db20b4e1141433-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.497 DEBUG:boto:path=/dkyges-63/obj111 2024-04-12T04:19:56.497 DEBUG:boto:auth_path=/dkyges-63/obj111 2024-04-12T04:19:56.497 DEBUG:boto:path=/dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.497 DEBUG:boto:auth_path=/dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.498 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.498 DEBUG:boto:Path: /dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.498 DEBUG:boto:Data: 2024-04-12T04:19:56.498 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.498 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.498 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.498 DEBUG:boto:Params: {} 2024-04-12T04:19:56.498 DEBUG:boto:Token: None 2024-04-12T04:19:56.498 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.498 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3D4RkexOf8g9ychW5esgy2zaSZ8= 2024-04-12T04:19:56.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3D4RkexOf8g9ychW5esgy2zaSZ8=', 'Content-Length': '0'} 2024-04-12T04:19:56.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000d45982e72fe9b8-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T04:19:56.501 DEBUG:boto:path=/dkyges-63/obj111 2024-04-12T04:19:56.501 DEBUG:boto:auth_path=/dkyges-63/obj111 2024-04-12T04:19:56.501 DEBUG:boto:path=/dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.501 DEBUG:boto:auth_path=/dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.501 DEBUG:boto:Method: GET 2024-04-12T04:19:56.501 DEBUG:boto:Path: /dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.501 DEBUG:boto:Data: 2024-04-12T04:19:56.501 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.501 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.501 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.501 DEBUG:boto:Params: {} 2024-04-12T04:19:56.502 DEBUG:boto:Token: None 2024-04-12T04:19:56.502 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.502 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zkQJ01hCLPBvpEDd1mzeZyS4zUw= 2024-04-12T04:19:56.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zkQJ01hCLPBvpEDd1mzeZyS4zUw=', 'Content-Length': '0'} 2024-04-12T04:19:56.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c59d3e8716768336-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.504 DEBUG:boto:path=/dkyges-63/obj111 2024-04-12T04:19:56.504 DEBUG:boto:auth_path=/dkyges-63/obj111 2024-04-12T04:19:56.504 DEBUG:boto:path=/dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.504 DEBUG:boto:auth_path=/dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.504 DEBUG:boto:Method: GET 2024-04-12T04:19:56.504 DEBUG:boto:Path: /dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.505 DEBUG:boto:Data: 2024-04-12T04:19:56.505 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.505 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.505 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.505 DEBUG:boto:Params: {} 2024-04-12T04:19:56.505 DEBUG:boto:Token: None 2024-04-12T04:19:56.505 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj111?versionId=null 2024-04-12T04:19:56.505 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zkQJ01hCLPBvpEDd1mzeZyS4zUw= 2024-04-12T04:19:56.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zkQJ01hCLPBvpEDd1mzeZyS4zUw=', 'Content-Length': '0'} 2024-04-12T04:19:56.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009331295c3462acc3-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.507 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.507 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.507 DEBUG:boto:path=/dkyges-63/?prefix=obj111 2024-04-12T04:19:56.508 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj111 2024-04-12T04:19:56.508 DEBUG:boto:Method: GET 2024-04-12T04:19:56.508 DEBUG:boto:Path: /dkyges-63/?prefix=obj111 2024-04-12T04:19:56.508 DEBUG:boto:Data: 2024-04-12T04:19:56.508 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.508 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.508 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.508 DEBUG:boto:Params: {} 2024-04-12T04:19:56.508 DEBUG:boto:Token: None 2024-04-12T04:19:56.508 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.509 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.511 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a40c325ea94fa046-006618b66c-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.511 DEBUG:boto:b'dkyges-63obj1111000falseobj1112024-04-12T04:15:49.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.512 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.512 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.512 DEBUG:boto:path=/dkyges-63/?prefix=obj111 2024-04-12T04:19:56.512 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj111 2024-04-12T04:19:56.512 DEBUG:boto:Method: GET 2024-04-12T04:19:56.512 DEBUG:boto:Path: /dkyges-63/?prefix=obj111 2024-04-12T04:19:56.512 DEBUG:boto:Data: 2024-04-12T04:19:56.512 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.512 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.512 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.512 DEBUG:boto:Params: {} 2024-04-12T04:19:56.513 DEBUG:boto:Token: None 2024-04-12T04:19:56.513 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.513 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b8c92bc45b9f26e-006618b66c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.517 DEBUG:boto:b'dkyges-63obj1111000falseobj1112024-04-12T04:15:49.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.518 DEBUG:boto:path=/dkyges-63/obj111 2024-04-12T04:19:56.518 DEBUG:boto:auth_path=/dkyges-63/obj111 2024-04-12T04:19:56.518 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.518 DEBUG:boto:Path: /dkyges-63/obj111 2024-04-12T04:19:56.518 DEBUG:boto:Data: 2024-04-12T04:19:56.518 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.518 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.518 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.518 DEBUG:boto:Params: {} 2024-04-12T04:19:56.518 DEBUG:boto:Token: None 2024-04-12T04:19:56.519 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj111 2024-04-12T04:19:56.519 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hzhw7xv965Z2ILIluOcrpsYd9eU= 2024-04-12T04:19:56.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hzhw7xv965Z2ILIluOcrpsYd9eU=', 'Content-Length': '0'} 2024-04-12T04:19:56.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f158ee73c5be133a-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.521 DEBUG:boto:path=/dkyges-63/obj111 2024-04-12T04:19:56.521 DEBUG:boto:auth_path=/dkyges-63/obj111 2024-04-12T04:19:56.521 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.521 DEBUG:boto:Path: /dkyges-63/obj111 2024-04-12T04:19:56.521 DEBUG:boto:Data: 2024-04-12T04:19:56.521 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.521 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.522 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.522 DEBUG:boto:Params: {} 2024-04-12T04:19:56.522 DEBUG:boto:Token: None 2024-04-12T04:19:56.522 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj111 2024-04-12T04:19:56.522 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hzhw7xv965Z2ILIluOcrpsYd9eU= 2024-04-12T04:19:56.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hzhw7xv965Z2ILIluOcrpsYd9eU=', 'Content-Length': '0'} 2024-04-12T04:19:56.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8283f75d41f8aca-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2024-04-12T04:19:56.524 DEBUG:boto:path=/dkyges-63/obj111 2024-04-12T04:19:56.525 DEBUG:boto:auth_path=/dkyges-63/obj111 2024-04-12T04:19:56.525 DEBUG:boto:Method: GET 2024-04-12T04:19:56.525 DEBUG:boto:Path: /dkyges-63/obj111 2024-04-12T04:19:56.525 DEBUG:boto:Data: 2024-04-12T04:19:56.525 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.525 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.525 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.525 DEBUG:boto:Params: {} 2024-04-12T04:19:56.525 DEBUG:boto:Token: None 2024-04-12T04:19:56.525 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj111 2024-04-12T04:19:56.526 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3gdxTkGCSuSf/BA6tW3VBeK2rd4= 2024-04-12T04:19:56.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3gdxTkGCSuSf/BA6tW3VBeK2rd4=', 'Content-Length': '0'} 2024-04-12T04:19:56.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000825fef28471eead5-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.528 DEBUG:boto:path=/dkyges-63/obj111 2024-04-12T04:19:56.528 DEBUG:boto:auth_path=/dkyges-63/obj111 2024-04-12T04:19:56.528 DEBUG:boto:Method: GET 2024-04-12T04:19:56.528 DEBUG:boto:Path: /dkyges-63/obj111 2024-04-12T04:19:56.528 DEBUG:boto:Data: 2024-04-12T04:19:56.528 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.528 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.528 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.528 DEBUG:boto:Params: {} 2024-04-12T04:19:56.529 DEBUG:boto:Token: None 2024-04-12T04:19:56.529 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj111 2024-04-12T04:19:56.529 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3gdxTkGCSuSf/BA6tW3VBeK2rd4= 2024-04-12T04:19:56.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3gdxTkGCSuSf/BA6tW3VBeK2rd4=', 'Content-Length': '0'} 2024-04-12T04:19:56.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003190ad595c879e82-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.578 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.578 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.578 DEBUG:boto:path=/dkyges-63/?prefix=obj111 2024-04-12T04:19:56.579 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj111 2024-04-12T04:19:56.579 DEBUG:boto:Method: GET 2024-04-12T04:19:56.579 DEBUG:boto:Path: /dkyges-63/?prefix=obj111 2024-04-12T04:19:56.579 DEBUG:boto:Data: 2024-04-12T04:19:56.579 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.579 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.579 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.579 DEBUG:boto:Params: {} 2024-04-12T04:19:56.579 DEBUG:boto:Token: None 2024-04-12T04:19:56.580 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.580 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d4bde280a9e7853-006618b66c-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.582 DEBUG:boto:b'dkyges-63obj1111000falseobj1112024-04-12T04:15:49.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.583 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.583 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.583 DEBUG:boto:path=/dkyges-63/?prefix=obj111 2024-04-12T04:19:56.583 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj111 2024-04-12T04:19:56.583 DEBUG:boto:Method: GET 2024-04-12T04:19:56.583 DEBUG:boto:Path: /dkyges-63/?prefix=obj111 2024-04-12T04:19:56.583 DEBUG:boto:Data: 2024-04-12T04:19:56.583 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.583 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.584 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.584 DEBUG:boto:Params: {} 2024-04-12T04:19:56.584 DEBUG:boto:Token: None 2024-04-12T04:19:56.584 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.584 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a873e93501da424c-006618b66c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.588 DEBUG:boto:b'dkyges-63obj1111000falseobj1112024-04-12T04:15:49.663Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:19:56.589 DEBUG:boto:path=/dkyges-63/obj115 2024-04-12T04:19:56.589 DEBUG:boto:auth_path=/dkyges-63/obj115 2024-04-12T04:19:56.589 DEBUG:boto:path=/dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.589 DEBUG:boto:auth_path=/dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.589 DEBUG:boto:Method: GET 2024-04-12T04:19:56.589 DEBUG:boto:Path: /dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.589 DEBUG:boto:Data: 2024-04-12T04:19:56.589 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.590 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.590 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.590 DEBUG:boto:Params: {} 2024-04-12T04:19:56.590 DEBUG:boto:Token: None 2024-04-12T04:19:56.590 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.590 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AZj0W+fDUB8bWyeUIemND0PQLQw= 2024-04-12T04:19:56.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AZj0W+fDUB8bWyeUIemND0PQLQw=', 'Content-Length': '0'} 2024-04-12T04:19:56.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097883404ffffff6b-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.638 DEBUG:boto:path=/dkyges-63/obj115 2024-04-12T04:19:56.638 DEBUG:boto:auth_path=/dkyges-63/obj115 2024-04-12T04:19:56.639 DEBUG:boto:path=/dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.639 DEBUG:boto:auth_path=/dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.639 DEBUG:boto:Method: GET 2024-04-12T04:19:56.639 DEBUG:boto:Path: /dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.639 DEBUG:boto:Data: 2024-04-12T04:19:56.639 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.639 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.639 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.639 DEBUG:boto:Params: {} 2024-04-12T04:19:56.639 DEBUG:boto:Token: None 2024-04-12T04:19:56.639 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.640 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AZj0W+fDUB8bWyeUIemND0PQLQw= 2024-04-12T04:19:56.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AZj0W+fDUB8bWyeUIemND0PQLQw=', 'Content-Length': '0'} 2024-04-12T04:19:56.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b7b678f37abd2d3-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.642 DEBUG:boto:path=/dkyges-63/obj115 2024-04-12T04:19:56.642 DEBUG:boto:auth_path=/dkyges-63/obj115 2024-04-12T04:19:56.642 DEBUG:boto:path=/dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.642 DEBUG:boto:auth_path=/dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.642 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.643 DEBUG:boto:Path: /dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.643 DEBUG:boto:Data: 2024-04-12T04:19:56.643 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.643 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.643 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.643 DEBUG:boto:Params: {} 2024-04-12T04:19:56.643 DEBUG:boto:Token: None 2024-04-12T04:19:56.643 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.643 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NIo3S2HMeTf+YDU965m6oNbe9q4= 2024-04-12T04:19:56.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NIo3S2HMeTf+YDU965m6oNbe9q4=', 'Content-Length': '0'} 2024-04-12T04:19:56.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df396cc16943e9e4-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.645 DEBUG:boto:path=/dkyges-63/obj115 2024-04-12T04:19:56.645 DEBUG:boto:auth_path=/dkyges-63/obj115 2024-04-12T04:19:56.645 DEBUG:boto:path=/dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.646 DEBUG:boto:auth_path=/dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.646 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.646 DEBUG:boto:Path: /dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.646 DEBUG:boto:Data: 2024-04-12T04:19:56.646 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.646 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.646 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.646 DEBUG:boto:Params: {} 2024-04-12T04:19:56.646 DEBUG:boto:Token: None 2024-04-12T04:19:56.646 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.647 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NIo3S2HMeTf+YDU965m6oNbe9q4= 2024-04-12T04:19:56.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NIo3S2HMeTf+YDU965m6oNbe9q4=', 'Content-Length': '0'} 2024-04-12T04:19:56.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd542b79ec97625e-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:19:56.649 DEBUG:boto:path=/dkyges-63/obj115 2024-04-12T04:19:56.649 DEBUG:boto:auth_path=/dkyges-63/obj115 2024-04-12T04:19:56.649 DEBUG:boto:path=/dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.650 DEBUG:boto:auth_path=/dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.650 DEBUG:boto:Method: GET 2024-04-12T04:19:56.650 DEBUG:boto:Path: /dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.650 DEBUG:boto:Data: 2024-04-12T04:19:56.650 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.650 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.650 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.650 DEBUG:boto:Params: {} 2024-04-12T04:19:56.650 DEBUG:boto:Token: None 2024-04-12T04:19:56.650 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.651 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AZj0W+fDUB8bWyeUIemND0PQLQw= 2024-04-12T04:19:56.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AZj0W+fDUB8bWyeUIemND0PQLQw=', 'Content-Length': '0'} 2024-04-12T04:19:56.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c1be2a8c50d12de-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.653 DEBUG:boto:path=/dkyges-63/obj115 2024-04-12T04:19:56.653 DEBUG:boto:auth_path=/dkyges-63/obj115 2024-04-12T04:19:56.653 DEBUG:boto:path=/dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.653 DEBUG:boto:auth_path=/dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.653 DEBUG:boto:Method: GET 2024-04-12T04:19:56.653 DEBUG:boto:Path: /dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.653 DEBUG:boto:Data: 2024-04-12T04:19:56.654 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.654 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.654 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.654 DEBUG:boto:Params: {} 2024-04-12T04:19:56.654 DEBUG:boto:Token: None 2024-04-12T04:19:56.654 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj115?versionId=null 2024-04-12T04:19:56.654 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AZj0W+fDUB8bWyeUIemND0PQLQw= 2024-04-12T04:19:56.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AZj0W+fDUB8bWyeUIemND0PQLQw=', 'Content-Length': '0'} 2024-04-12T04:19:56.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfcc11991e0218eb-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.656 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.657 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.657 DEBUG:boto:path=/dkyges-63/?prefix=obj115 2024-04-12T04:19:56.657 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj115 2024-04-12T04:19:56.657 DEBUG:boto:Method: GET 2024-04-12T04:19:56.657 DEBUG:boto:Path: /dkyges-63/?prefix=obj115 2024-04-12T04:19:56.657 DEBUG:boto:Data: 2024-04-12T04:19:56.657 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.657 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.657 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.657 DEBUG:boto:Params: {} 2024-04-12T04:19:56.657 DEBUG:boto:Token: None 2024-04-12T04:19:56.658 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.658 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3b95c34cdc79274-006618b66c-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.661 DEBUG:boto:b'dkyges-63obj1151000falseobj1152024-04-12T04:15:48.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.661 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.661 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.661 DEBUG:boto:path=/dkyges-63/?prefix=obj115 2024-04-12T04:19:56.661 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj115 2024-04-12T04:19:56.661 DEBUG:boto:Method: GET 2024-04-12T04:19:56.661 DEBUG:boto:Path: /dkyges-63/?prefix=obj115 2024-04-12T04:19:56.662 DEBUG:boto:Data: 2024-04-12T04:19:56.662 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.662 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.662 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.662 DEBUG:boto:Params: {} 2024-04-12T04:19:56.662 DEBUG:boto:Token: None 2024-04-12T04:19:56.662 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.662 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009181cf197c824aaa-006618b66c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.667 DEBUG:boto:b'dkyges-63obj1151000falseobj1152024-04-12T04:15:48.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.667 DEBUG:boto:path=/dkyges-63/obj115 2024-04-12T04:19:56.667 DEBUG:boto:auth_path=/dkyges-63/obj115 2024-04-12T04:19:56.667 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.667 DEBUG:boto:Path: /dkyges-63/obj115 2024-04-12T04:19:56.668 DEBUG:boto:Data: 2024-04-12T04:19:56.668 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.668 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.668 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.668 DEBUG:boto:Params: {} 2024-04-12T04:19:56.668 DEBUG:boto:Token: None 2024-04-12T04:19:56.668 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj115 2024-04-12T04:19:56.668 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r7IiiUneBM8hHLavIkzwm4texU8= 2024-04-12T04:19:56.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r7IiiUneBM8hHLavIkzwm4texU8=', 'Content-Length': '0'} 2024-04-12T04:19:56.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc711e327b29f44a-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.670 DEBUG:boto:path=/dkyges-63/obj115 2024-04-12T04:19:56.670 DEBUG:boto:auth_path=/dkyges-63/obj115 2024-04-12T04:19:56.671 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.671 DEBUG:boto:Path: /dkyges-63/obj115 2024-04-12T04:19:56.671 DEBUG:boto:Data: 2024-04-12T04:19:56.671 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.671 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.671 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.671 DEBUG:boto:Params: {} 2024-04-12T04:19:56.671 DEBUG:boto:Token: None 2024-04-12T04:19:56.671 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj115 2024-04-12T04:19:56.671 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r7IiiUneBM8hHLavIkzwm4texU8= 2024-04-12T04:19:56.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r7IiiUneBM8hHLavIkzwm4texU8=', 'Content-Length': '0'} 2024-04-12T04:19:56.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d7098e600cb3a83-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2024-04-12T04:19:56.674 DEBUG:boto:path=/dkyges-63/obj115 2024-04-12T04:19:56.674 DEBUG:boto:auth_path=/dkyges-63/obj115 2024-04-12T04:19:56.674 DEBUG:boto:Method: GET 2024-04-12T04:19:56.674 DEBUG:boto:Path: /dkyges-63/obj115 2024-04-12T04:19:56.674 DEBUG:boto:Data: 2024-04-12T04:19:56.674 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.674 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.674 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.674 DEBUG:boto:Params: {} 2024-04-12T04:19:56.674 DEBUG:boto:Token: None 2024-04-12T04:19:56.675 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj115 2024-04-12T04:19:56.675 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m7Xf2nWGCeyfz00LPGTxt0M+qZQ= 2024-04-12T04:19:56.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m7Xf2nWGCeyfz00LPGTxt0M+qZQ=', 'Content-Length': '0'} 2024-04-12T04:19:56.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ac3cd8f0eca477f-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.677 DEBUG:boto:path=/dkyges-63/obj115 2024-04-12T04:19:56.677 DEBUG:boto:auth_path=/dkyges-63/obj115 2024-04-12T04:19:56.677 DEBUG:boto:Method: GET 2024-04-12T04:19:56.677 DEBUG:boto:Path: /dkyges-63/obj115 2024-04-12T04:19:56.677 DEBUG:boto:Data: 2024-04-12T04:19:56.677 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.677 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.678 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.678 DEBUG:boto:Params: {} 2024-04-12T04:19:56.678 DEBUG:boto:Token: None 2024-04-12T04:19:56.678 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj115 2024-04-12T04:19:56.678 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m7Xf2nWGCeyfz00LPGTxt0M+qZQ= 2024-04-12T04:19:56.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m7Xf2nWGCeyfz00LPGTxt0M+qZQ=', 'Content-Length': '0'} 2024-04-12T04:19:56.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed6870fa12045beb-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.680 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.681 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.681 DEBUG:boto:path=/dkyges-63/?prefix=obj115 2024-04-12T04:19:56.681 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj115 2024-04-12T04:19:56.681 DEBUG:boto:Method: GET 2024-04-12T04:19:56.681 DEBUG:boto:Path: /dkyges-63/?prefix=obj115 2024-04-12T04:19:56.681 DEBUG:boto:Data: 2024-04-12T04:19:56.681 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.681 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.681 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.681 DEBUG:boto:Params: {} 2024-04-12T04:19:56.681 DEBUG:boto:Token: None 2024-04-12T04:19:56.682 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.682 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056b461e9de406b2c-006618b66c-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.684 DEBUG:boto:b'dkyges-63obj1151000falseobj1152024-04-12T04:15:48.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.685 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.685 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.685 DEBUG:boto:path=/dkyges-63/?prefix=obj115 2024-04-12T04:19:56.685 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj115 2024-04-12T04:19:56.685 DEBUG:boto:Method: GET 2024-04-12T04:19:56.685 DEBUG:boto:Path: /dkyges-63/?prefix=obj115 2024-04-12T04:19:56.685 DEBUG:boto:Data: 2024-04-12T04:19:56.685 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.685 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.685 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.685 DEBUG:boto:Params: {} 2024-04-12T04:19:56.686 DEBUG:boto:Token: None 2024-04-12T04:19:56.686 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.686 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.690 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b0b3d4c999e1612f-006618b66c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.691 DEBUG:boto:b'dkyges-63obj1151000falseobj1152024-04-12T04:15:48.571Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:19:56.691 DEBUG:boto:path=/dkyges-63/obj116 2024-04-12T04:19:56.691 DEBUG:boto:auth_path=/dkyges-63/obj116 2024-04-12T04:19:56.691 DEBUG:boto:path=/dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.691 DEBUG:boto:auth_path=/dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.691 DEBUG:boto:Method: GET 2024-04-12T04:19:56.691 DEBUG:boto:Path: /dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.692 DEBUG:boto:Data: 2024-04-12T04:19:56.692 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.692 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.692 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.692 DEBUG:boto:Params: {} 2024-04-12T04:19:56.692 DEBUG:boto:Token: None 2024-04-12T04:19:56.692 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.692 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MA4E0ga2he/3xJtnq1Rejvxg20w= 2024-04-12T04:19:56.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MA4E0ga2he/3xJtnq1Rejvxg20w=', 'Content-Length': '0'} 2024-04-12T04:19:56.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000033846f5ba94686a-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.695 DEBUG:boto:path=/dkyges-63/obj116 2024-04-12T04:19:56.695 DEBUG:boto:auth_path=/dkyges-63/obj116 2024-04-12T04:19:56.695 DEBUG:boto:path=/dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.695 DEBUG:boto:auth_path=/dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.695 DEBUG:boto:Method: GET 2024-04-12T04:19:56.695 DEBUG:boto:Path: /dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.695 DEBUG:boto:Data: 2024-04-12T04:19:56.695 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.695 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.695 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.695 DEBUG:boto:Params: {} 2024-04-12T04:19:56.696 DEBUG:boto:Token: None 2024-04-12T04:19:56.696 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.696 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MA4E0ga2he/3xJtnq1Rejvxg20w= 2024-04-12T04:19:56.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MA4E0ga2he/3xJtnq1Rejvxg20w=', 'Content-Length': '0'} 2024-04-12T04:19:56.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edf16643cf2021dc-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.738 DEBUG:boto:path=/dkyges-63/obj116 2024-04-12T04:19:56.738 DEBUG:boto:auth_path=/dkyges-63/obj116 2024-04-12T04:19:56.738 DEBUG:boto:path=/dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.739 DEBUG:boto:auth_path=/dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.739 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.739 DEBUG:boto:Path: /dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.739 DEBUG:boto:Data: 2024-04-12T04:19:56.739 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.739 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.739 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.739 DEBUG:boto:Params: {} 2024-04-12T04:19:56.739 DEBUG:boto:Token: None 2024-04-12T04:19:56.739 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.740 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:afmC6tfQRMdeTNNRTf/SJnhdkmg= 2024-04-12T04:19:56.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:afmC6tfQRMdeTNNRTf/SJnhdkmg=', 'Content-Length': '0'} 2024-04-12T04:19:56.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000777ab05d51b89653-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.742 DEBUG:boto:path=/dkyges-63/obj116 2024-04-12T04:19:56.742 DEBUG:boto:auth_path=/dkyges-63/obj116 2024-04-12T04:19:56.742 DEBUG:boto:path=/dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.742 DEBUG:boto:auth_path=/dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.742 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.742 DEBUG:boto:Path: /dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.742 DEBUG:boto:Data: 2024-04-12T04:19:56.742 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.742 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.743 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.743 DEBUG:boto:Params: {} 2024-04-12T04:19:56.743 DEBUG:boto:Token: None 2024-04-12T04:19:56.743 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.743 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:afmC6tfQRMdeTNNRTf/SJnhdkmg= 2024-04-12T04:19:56.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:afmC6tfQRMdeTNNRTf/SJnhdkmg=', 'Content-Length': '0'} 2024-04-12T04:19:56.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bdcd6990fa7b80c-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:19:56.745 DEBUG:boto:path=/dkyges-63/obj116 2024-04-12T04:19:56.745 DEBUG:boto:auth_path=/dkyges-63/obj116 2024-04-12T04:19:56.746 DEBUG:boto:path=/dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.746 DEBUG:boto:auth_path=/dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.746 DEBUG:boto:Method: GET 2024-04-12T04:19:56.746 DEBUG:boto:Path: /dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.746 DEBUG:boto:Data: 2024-04-12T04:19:56.746 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.746 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.746 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.746 DEBUG:boto:Params: {} 2024-04-12T04:19:56.746 DEBUG:boto:Token: None 2024-04-12T04:19:56.746 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.747 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MA4E0ga2he/3xJtnq1Rejvxg20w= 2024-04-12T04:19:56.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MA4E0ga2he/3xJtnq1Rejvxg20w=', 'Content-Length': '0'} 2024-04-12T04:19:56.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ab108572e713ac3-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.749 DEBUG:boto:path=/dkyges-63/obj116 2024-04-12T04:19:56.749 DEBUG:boto:auth_path=/dkyges-63/obj116 2024-04-12T04:19:56.749 DEBUG:boto:path=/dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.749 DEBUG:boto:auth_path=/dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.749 DEBUG:boto:Method: GET 2024-04-12T04:19:56.749 DEBUG:boto:Path: /dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.749 DEBUG:boto:Data: 2024-04-12T04:19:56.749 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.749 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.749 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.750 DEBUG:boto:Params: {} 2024-04-12T04:19:56.750 DEBUG:boto:Token: None 2024-04-12T04:19:56.750 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj116?versionId=null 2024-04-12T04:19:56.750 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MA4E0ga2he/3xJtnq1Rejvxg20w= 2024-04-12T04:19:56.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MA4E0ga2he/3xJtnq1Rejvxg20w=', 'Content-Length': '0'} 2024-04-12T04:19:56.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004209f2daae274cfe-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.752 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.752 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.752 DEBUG:boto:path=/dkyges-63/?prefix=obj116 2024-04-12T04:19:56.753 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj116 2024-04-12T04:19:56.753 DEBUG:boto:Method: GET 2024-04-12T04:19:56.753 DEBUG:boto:Path: /dkyges-63/?prefix=obj116 2024-04-12T04:19:56.753 DEBUG:boto:Data: 2024-04-12T04:19:56.753 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.753 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.753 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.753 DEBUG:boto:Params: {} 2024-04-12T04:19:56.753 DEBUG:boto:Token: None 2024-04-12T04:19:56.753 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.753 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006679a1abdce52f6d-006618b66c-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.756 DEBUG:boto:b'dkyges-63obj1161000falseobj1162024-04-12T04:15:48.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.756 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.757 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.757 DEBUG:boto:path=/dkyges-63/?prefix=obj116 2024-04-12T04:19:56.757 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj116 2024-04-12T04:19:56.757 DEBUG:boto:Method: GET 2024-04-12T04:19:56.757 DEBUG:boto:Path: /dkyges-63/?prefix=obj116 2024-04-12T04:19:56.757 DEBUG:boto:Data: 2024-04-12T04:19:56.757 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.757 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.757 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.757 DEBUG:boto:Params: {} 2024-04-12T04:19:56.757 DEBUG:boto:Token: None 2024-04-12T04:19:56.758 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.758 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.762 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ec7ec4333b1373c-006618b66c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.762 DEBUG:boto:b'dkyges-63obj1161000falseobj1162024-04-12T04:15:48.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.763 DEBUG:boto:path=/dkyges-63/obj116 2024-04-12T04:19:56.763 DEBUG:boto:auth_path=/dkyges-63/obj116 2024-04-12T04:19:56.763 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.763 DEBUG:boto:Path: /dkyges-63/obj116 2024-04-12T04:19:56.763 DEBUG:boto:Data: 2024-04-12T04:19:56.763 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.763 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.763 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.764 DEBUG:boto:Params: {} 2024-04-12T04:19:56.764 DEBUG:boto:Token: None 2024-04-12T04:19:56.764 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj116 2024-04-12T04:19:56.764 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D4AkuFXAi+8ldcbT1sXCblRqrsI= 2024-04-12T04:19:56.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D4AkuFXAi+8ldcbT1sXCblRqrsI=', 'Content-Length': '0'} 2024-04-12T04:19:56.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005aa81b5fba9b6d56-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.766 DEBUG:boto:path=/dkyges-63/obj116 2024-04-12T04:19:56.766 DEBUG:boto:auth_path=/dkyges-63/obj116 2024-04-12T04:19:56.766 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.767 DEBUG:boto:Path: /dkyges-63/obj116 2024-04-12T04:19:56.767 DEBUG:boto:Data: 2024-04-12T04:19:56.767 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.767 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.767 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.767 DEBUG:boto:Params: {} 2024-04-12T04:19:56.767 DEBUG:boto:Token: None 2024-04-12T04:19:56.767 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj116 2024-04-12T04:19:56.767 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D4AkuFXAi+8ldcbT1sXCblRqrsI= 2024-04-12T04:19:56.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D4AkuFXAi+8ldcbT1sXCblRqrsI=', 'Content-Length': '0'} 2024-04-12T04:19:56.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa5374a410d3764f-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2024-04-12T04:19:56.770 DEBUG:boto:path=/dkyges-63/obj116 2024-04-12T04:19:56.770 DEBUG:boto:auth_path=/dkyges-63/obj116 2024-04-12T04:19:56.770 DEBUG:boto:Method: GET 2024-04-12T04:19:56.770 DEBUG:boto:Path: /dkyges-63/obj116 2024-04-12T04:19:56.770 DEBUG:boto:Data: 2024-04-12T04:19:56.770 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.770 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.770 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.770 DEBUG:boto:Params: {} 2024-04-12T04:19:56.770 DEBUG:boto:Token: None 2024-04-12T04:19:56.771 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj116 2024-04-12T04:19:56.771 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zCLLHUmujfNIgQzNxeh/ozi9sjg= 2024-04-12T04:19:56.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zCLLHUmujfNIgQzNxeh/ozi9sjg=', 'Content-Length': '0'} 2024-04-12T04:19:56.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083e53b236d8853f2-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.773 DEBUG:boto:path=/dkyges-63/obj116 2024-04-12T04:19:56.773 DEBUG:boto:auth_path=/dkyges-63/obj116 2024-04-12T04:19:56.773 DEBUG:boto:Method: GET 2024-04-12T04:19:56.773 DEBUG:boto:Path: /dkyges-63/obj116 2024-04-12T04:19:56.773 DEBUG:boto:Data: 2024-04-12T04:19:56.773 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.773 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.773 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.774 DEBUG:boto:Params: {} 2024-04-12T04:19:56.774 DEBUG:boto:Token: None 2024-04-12T04:19:56.774 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj116 2024-04-12T04:19:56.774 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zCLLHUmujfNIgQzNxeh/ozi9sjg= 2024-04-12T04:19:56.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zCLLHUmujfNIgQzNxeh/ozi9sjg=', 'Content-Length': '0'} 2024-04-12T04:19:56.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005502bcbb795fde7e-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.776 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.776 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.776 DEBUG:boto:path=/dkyges-63/?prefix=obj116 2024-04-12T04:19:56.776 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj116 2024-04-12T04:19:56.777 DEBUG:boto:Method: GET 2024-04-12T04:19:56.777 DEBUG:boto:Path: /dkyges-63/?prefix=obj116 2024-04-12T04:19:56.777 DEBUG:boto:Data: 2024-04-12T04:19:56.777 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.777 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.777 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.777 DEBUG:boto:Params: {} 2024-04-12T04:19:56.777 DEBUG:boto:Token: None 2024-04-12T04:19:56.777 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.777 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000081542bb374b44fe-006618b66c-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.780 DEBUG:boto:b'dkyges-63obj1161000falseobj1162024-04-12T04:15:48.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.781 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.781 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.781 DEBUG:boto:path=/dkyges-63/?prefix=obj116 2024-04-12T04:19:56.781 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj116 2024-04-12T04:19:56.781 DEBUG:boto:Method: GET 2024-04-12T04:19:56.781 DEBUG:boto:Path: /dkyges-63/?prefix=obj116 2024-04-12T04:19:56.781 DEBUG:boto:Data: 2024-04-12T04:19:56.781 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.781 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.781 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.781 DEBUG:boto:Params: {} 2024-04-12T04:19:56.782 DEBUG:boto:Token: None 2024-04-12T04:19:56.782 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.782 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c525ac0fc3f305b0-006618b66c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.787 DEBUG:boto:b'dkyges-63obj1161000falseobj1162024-04-12T04:15:48.908Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T04:19:56.787 DEBUG:boto:path=/dkyges-63/obj124 2024-04-12T04:19:56.787 DEBUG:boto:auth_path=/dkyges-63/obj124 2024-04-12T04:19:56.787 DEBUG:boto:path=/dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.787 DEBUG:boto:auth_path=/dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.788 DEBUG:boto:Method: GET 2024-04-12T04:19:56.788 DEBUG:boto:Path: /dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.788 DEBUG:boto:Data: 2024-04-12T04:19:56.788 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.788 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.788 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.788 DEBUG:boto:Params: {} 2024-04-12T04:19:56.788 DEBUG:boto:Token: None 2024-04-12T04:19:56.788 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.788 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vJ0W0cN3hlX/f0ZF7Cmg3zNJl34= 2024-04-12T04:19:56.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vJ0W0cN3hlX/f0ZF7Cmg3zNJl34=', 'Content-Length': '0'} 2024-04-12T04:19:56.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057a9dbbac28b9592-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.834 DEBUG:boto:path=/dkyges-63/obj124 2024-04-12T04:19:56.835 DEBUG:boto:auth_path=/dkyges-63/obj124 2024-04-12T04:19:56.835 DEBUG:boto:path=/dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.835 DEBUG:boto:auth_path=/dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.835 DEBUG:boto:Method: GET 2024-04-12T04:19:56.835 DEBUG:boto:Path: /dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.835 DEBUG:boto:Data: 2024-04-12T04:19:56.835 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.835 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.835 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.835 DEBUG:boto:Params: {} 2024-04-12T04:19:56.836 DEBUG:boto:Token: None 2024-04-12T04:19:56.836 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.836 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vJ0W0cN3hlX/f0ZF7Cmg3zNJl34= 2024-04-12T04:19:56.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vJ0W0cN3hlX/f0ZF7Cmg3zNJl34=', 'Content-Length': '0'} 2024-04-12T04:19:56.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053e5b4434ba602aa-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.838 DEBUG:boto:path=/dkyges-63/obj124 2024-04-12T04:19:56.838 DEBUG:boto:auth_path=/dkyges-63/obj124 2024-04-12T04:19:56.838 DEBUG:boto:path=/dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.838 DEBUG:boto:auth_path=/dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.838 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.838 DEBUG:boto:Path: /dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.839 DEBUG:boto:Data: 2024-04-12T04:19:56.839 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.839 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.839 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.839 DEBUG:boto:Params: {} 2024-04-12T04:19:56.839 DEBUG:boto:Token: None 2024-04-12T04:19:56.839 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.839 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FmCMJs7NOsaZSG8h6w4MEg+gBck= 2024-04-12T04:19:56.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FmCMJs7NOsaZSG8h6w4MEg+gBck=', 'Content-Length': '0'} 2024-04-12T04:19:56.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e68b9237102e2783-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.841 DEBUG:boto:path=/dkyges-63/obj124 2024-04-12T04:19:56.842 DEBUG:boto:auth_path=/dkyges-63/obj124 2024-04-12T04:19:56.842 DEBUG:boto:path=/dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.842 DEBUG:boto:auth_path=/dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.842 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.842 DEBUG:boto:Path: /dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.842 DEBUG:boto:Data: 2024-04-12T04:19:56.842 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.842 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.842 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.842 DEBUG:boto:Params: {} 2024-04-12T04:19:56.842 DEBUG:boto:Token: None 2024-04-12T04:19:56.843 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.843 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FmCMJs7NOsaZSG8h6w4MEg+gBck= 2024-04-12T04:19:56.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FmCMJs7NOsaZSG8h6w4MEg+gBck=', 'Content-Length': '0'} 2024-04-12T04:19:56.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008523aa2c3c6608d3-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T04:19:56.845 DEBUG:boto:path=/dkyges-63/obj124 2024-04-12T04:19:56.845 DEBUG:boto:auth_path=/dkyges-63/obj124 2024-04-12T04:19:56.845 DEBUG:boto:path=/dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.845 DEBUG:boto:auth_path=/dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.845 DEBUG:boto:Method: GET 2024-04-12T04:19:56.845 DEBUG:boto:Path: /dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.846 DEBUG:boto:Data: 2024-04-12T04:19:56.846 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.846 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.846 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.846 DEBUG:boto:Params: {} 2024-04-12T04:19:56.846 DEBUG:boto:Token: None 2024-04-12T04:19:56.846 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.846 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vJ0W0cN3hlX/f0ZF7Cmg3zNJl34= 2024-04-12T04:19:56.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vJ0W0cN3hlX/f0ZF7Cmg3zNJl34=', 'Content-Length': '0'} 2024-04-12T04:19:56.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c5a403360f1481e-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.848 DEBUG:boto:path=/dkyges-63/obj124 2024-04-12T04:19:56.848 DEBUG:boto:auth_path=/dkyges-63/obj124 2024-04-12T04:19:56.848 DEBUG:boto:path=/dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.849 DEBUG:boto:auth_path=/dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.849 DEBUG:boto:Method: GET 2024-04-12T04:19:56.849 DEBUG:boto:Path: /dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.849 DEBUG:boto:Data: 2024-04-12T04:19:56.849 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.849 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.849 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.849 DEBUG:boto:Params: {} 2024-04-12T04:19:56.849 DEBUG:boto:Token: None 2024-04-12T04:19:56.849 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj124?versionId=null 2024-04-12T04:19:56.850 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vJ0W0cN3hlX/f0ZF7Cmg3zNJl34= 2024-04-12T04:19:56.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vJ0W0cN3hlX/f0ZF7Cmg3zNJl34=', 'Content-Length': '0'} 2024-04-12T04:19:56.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de9bc7e290546572-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.852 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.852 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.852 DEBUG:boto:path=/dkyges-63/?prefix=obj124 2024-04-12T04:19:56.852 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj124 2024-04-12T04:19:56.852 DEBUG:boto:Method: GET 2024-04-12T04:19:56.852 DEBUG:boto:Path: /dkyges-63/?prefix=obj124 2024-04-12T04:19:56.852 DEBUG:boto:Data: 2024-04-12T04:19:56.853 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.853 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.853 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.853 DEBUG:boto:Params: {} 2024-04-12T04:19:56.853 DEBUG:boto:Token: None 2024-04-12T04:19:56.853 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.853 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c30363eb1b404e7-006618b66c-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.856 DEBUG:boto:b'dkyges-63obj1241000falseobj1242024-04-12T04:15:49.304Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.856 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.856 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.856 DEBUG:boto:path=/dkyges-63/?prefix=obj124 2024-04-12T04:19:56.856 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj124 2024-04-12T04:19:56.857 DEBUG:boto:Method: GET 2024-04-12T04:19:56.857 DEBUG:boto:Path: /dkyges-63/?prefix=obj124 2024-04-12T04:19:56.857 DEBUG:boto:Data: 2024-04-12T04:19:56.857 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.857 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.857 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.857 DEBUG:boto:Params: {} 2024-04-12T04:19:56.857 DEBUG:boto:Token: None 2024-04-12T04:19:56.857 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.857 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007167afedd701615d-006618b66c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.862 DEBUG:boto:b'dkyges-63obj1241000falseobj1242024-04-12T04:15:49.304Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.862 DEBUG:boto:path=/dkyges-63/obj124 2024-04-12T04:19:56.862 DEBUG:boto:auth_path=/dkyges-63/obj124 2024-04-12T04:19:56.863 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.863 DEBUG:boto:Path: /dkyges-63/obj124 2024-04-12T04:19:56.863 DEBUG:boto:Data: 2024-04-12T04:19:56.863 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.863 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.863 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.863 DEBUG:boto:Params: {} 2024-04-12T04:19:56.863 DEBUG:boto:Token: None 2024-04-12T04:19:56.863 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj124 2024-04-12T04:19:56.863 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NhyrJjprOZMm+HRcxvFuylFyxlA= 2024-04-12T04:19:56.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NhyrJjprOZMm+HRcxvFuylFyxlA=', 'Content-Length': '0'} 2024-04-12T04:19:56.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f3684daebbfe090-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.865 DEBUG:boto:path=/dkyges-63/obj124 2024-04-12T04:19:56.865 DEBUG:boto:auth_path=/dkyges-63/obj124 2024-04-12T04:19:56.865 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.866 DEBUG:boto:Path: /dkyges-63/obj124 2024-04-12T04:19:56.866 DEBUG:boto:Data: 2024-04-12T04:19:56.866 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.866 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.866 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.866 DEBUG:boto:Params: {} 2024-04-12T04:19:56.866 DEBUG:boto:Token: None 2024-04-12T04:19:56.866 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj124 2024-04-12T04:19:56.866 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NhyrJjprOZMm+HRcxvFuylFyxlA= 2024-04-12T04:19:56.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NhyrJjprOZMm+HRcxvFuylFyxlA=', 'Content-Length': '0'} 2024-04-12T04:19:56.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df67ea5440f66f92-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2024-04-12T04:19:56.868 DEBUG:boto:path=/dkyges-63/obj124 2024-04-12T04:19:56.869 DEBUG:boto:auth_path=/dkyges-63/obj124 2024-04-12T04:19:56.869 DEBUG:boto:Method: GET 2024-04-12T04:19:56.869 DEBUG:boto:Path: /dkyges-63/obj124 2024-04-12T04:19:56.869 DEBUG:boto:Data: 2024-04-12T04:19:56.869 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.869 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.869 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.869 DEBUG:boto:Params: {} 2024-04-12T04:19:56.869 DEBUG:boto:Token: None 2024-04-12T04:19:56.869 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj124 2024-04-12T04:19:56.869 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lYGcmusWBYYTVux72W7RctJYSOU= 2024-04-12T04:19:56.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lYGcmusWBYYTVux72W7RctJYSOU=', 'Content-Length': '0'} 2024-04-12T04:19:56.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f26884a48e1a4fd5-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.872 DEBUG:boto:path=/dkyges-63/obj124 2024-04-12T04:19:56.872 DEBUG:boto:auth_path=/dkyges-63/obj124 2024-04-12T04:19:56.872 DEBUG:boto:Method: GET 2024-04-12T04:19:56.872 DEBUG:boto:Path: /dkyges-63/obj124 2024-04-12T04:19:56.872 DEBUG:boto:Data: 2024-04-12T04:19:56.872 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.872 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.872 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.872 DEBUG:boto:Params: {} 2024-04-12T04:19:56.873 DEBUG:boto:Token: None 2024-04-12T04:19:56.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj124 2024-04-12T04:19:56.873 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lYGcmusWBYYTVux72W7RctJYSOU= 2024-04-12T04:19:56.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lYGcmusWBYYTVux72W7RctJYSOU=', 'Content-Length': '0'} 2024-04-12T04:19:56.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8750e37f82032b4-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.918 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.918 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.919 DEBUG:boto:path=/dkyges-63/?prefix=obj124 2024-04-12T04:19:56.919 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj124 2024-04-12T04:19:56.919 DEBUG:boto:Method: GET 2024-04-12T04:19:56.919 DEBUG:boto:Path: /dkyges-63/?prefix=obj124 2024-04-12T04:19:56.919 DEBUG:boto:Data: 2024-04-12T04:19:56.919 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.919 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.919 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.919 DEBUG:boto:Params: {} 2024-04-12T04:19:56.919 DEBUG:boto:Token: None 2024-04-12T04:19:56.919 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.920 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000794c03f4f601487b-006618b66c-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.922 DEBUG:boto:b'dkyges-63obj1241000falseobj1242024-04-12T04:15:49.304Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.923 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.923 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.923 DEBUG:boto:path=/dkyges-63/?prefix=obj124 2024-04-12T04:19:56.923 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj124 2024-04-12T04:19:56.923 DEBUG:boto:Method: GET 2024-04-12T04:19:56.923 DEBUG:boto:Path: /dkyges-63/?prefix=obj124 2024-04-12T04:19:56.923 DEBUG:boto:Data: 2024-04-12T04:19:56.923 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.923 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.923 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.924 DEBUG:boto:Params: {} 2024-04-12T04:19:56.924 DEBUG:boto:Token: None 2024-04-12T04:19:56.924 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.924 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c5a4a80c63998f6-006618b66c-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.929 DEBUG:boto:b'dkyges-63obj1241000falseobj1242024-04-12T04:15:49.304Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:19:56.929 DEBUG:boto:path=/dkyges-63/obj145 2024-04-12T04:19:56.929 DEBUG:boto:auth_path=/dkyges-63/obj145 2024-04-12T04:19:56.929 DEBUG:boto:path=/dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.930 DEBUG:boto:auth_path=/dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.930 DEBUG:boto:Method: GET 2024-04-12T04:19:56.930 DEBUG:boto:Path: /dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.930 DEBUG:boto:Data: 2024-04-12T04:19:56.930 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.930 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.930 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.930 DEBUG:boto:Params: {} 2024-04-12T04:19:56.930 DEBUG:boto:Token: None 2024-04-12T04:19:56.930 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.931 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HzZOrU6i3E8BAiPWu0i8+YjSd0c= 2024-04-12T04:19:56.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HzZOrU6i3E8BAiPWu0i8+YjSd0c=', 'Content-Length': '0'} 2024-04-12T04:19:56.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a90cc9dae90024e5-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.974 DEBUG:boto:path=/dkyges-63/obj145 2024-04-12T04:19:56.975 DEBUG:boto:auth_path=/dkyges-63/obj145 2024-04-12T04:19:56.975 DEBUG:boto:path=/dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.975 DEBUG:boto:auth_path=/dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.975 DEBUG:boto:Method: GET 2024-04-12T04:19:56.975 DEBUG:boto:Path: /dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.975 DEBUG:boto:Data: 2024-04-12T04:19:56.975 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.975 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.975 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.975 DEBUG:boto:Params: {} 2024-04-12T04:19:56.975 DEBUG:boto:Token: None 2024-04-12T04:19:56.976 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.976 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HzZOrU6i3E8BAiPWu0i8+YjSd0c= 2024-04-12T04:19:56.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HzZOrU6i3E8BAiPWu0i8+YjSd0c=', 'Content-Length': '0'} 2024-04-12T04:19:56.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e930cae85ff36209-006618b66c-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.978 DEBUG:boto:path=/dkyges-63/obj145 2024-04-12T04:19:56.978 DEBUG:boto:auth_path=/dkyges-63/obj145 2024-04-12T04:19:56.979 DEBUG:boto:path=/dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.979 DEBUG:boto:auth_path=/dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.979 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.979 DEBUG:boto:Path: /dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.979 DEBUG:boto:Data: 2024-04-12T04:19:56.979 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.979 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.979 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.979 DEBUG:boto:Params: {} 2024-04-12T04:19:56.979 DEBUG:boto:Token: None 2024-04-12T04:19:56.980 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.980 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u/EQ/4e5YxkYKY9ilXCFmJuQh6E= 2024-04-12T04:19:56.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u/EQ/4e5YxkYKY9ilXCFmJuQh6E=', 'Content-Length': '0'} 2024-04-12T04:19:56.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000328e2ecadf0c1ed5-006618b66c-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.982 DEBUG:boto:path=/dkyges-63/obj145 2024-04-12T04:19:56.982 DEBUG:boto:auth_path=/dkyges-63/obj145 2024-04-12T04:19:56.982 DEBUG:boto:path=/dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.982 DEBUG:boto:auth_path=/dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.982 DEBUG:boto:Method: HEAD 2024-04-12T04:19:56.982 DEBUG:boto:Path: /dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.982 DEBUG:boto:Data: 2024-04-12T04:19:56.983 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.983 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.983 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.983 DEBUG:boto:Params: {} 2024-04-12T04:19:56.983 DEBUG:boto:Token: None 2024-04-12T04:19:56.983 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.983 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u/EQ/4e5YxkYKY9ilXCFmJuQh6E= 2024-04-12T04:19:56.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u/EQ/4e5YxkYKY9ilXCFmJuQh6E=', 'Content-Length': '0'} 2024-04-12T04:19:56.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055a1f8d8a32a1683-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:19:56.985 DEBUG:boto:path=/dkyges-63/obj145 2024-04-12T04:19:56.985 DEBUG:boto:auth_path=/dkyges-63/obj145 2024-04-12T04:19:56.985 DEBUG:boto:path=/dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.986 DEBUG:boto:auth_path=/dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.986 DEBUG:boto:Method: GET 2024-04-12T04:19:56.986 DEBUG:boto:Path: /dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.986 DEBUG:boto:Data: 2024-04-12T04:19:56.986 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.986 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.986 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.986 DEBUG:boto:Params: {} 2024-04-12T04:19:56.986 DEBUG:boto:Token: None 2024-04-12T04:19:56.986 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.987 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HzZOrU6i3E8BAiPWu0i8+YjSd0c= 2024-04-12T04:19:56.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HzZOrU6i3E8BAiPWu0i8+YjSd0c=', 'Content-Length': '0'} 2024-04-12T04:19:56.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42bb7a4f39f4ab1-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.989 DEBUG:boto:path=/dkyges-63/obj145 2024-04-12T04:19:56.989 DEBUG:boto:auth_path=/dkyges-63/obj145 2024-04-12T04:19:56.989 DEBUG:boto:path=/dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.989 DEBUG:boto:auth_path=/dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.989 DEBUG:boto:Method: GET 2024-04-12T04:19:56.989 DEBUG:boto:Path: /dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.989 DEBUG:boto:Data: 2024-04-12T04:19:56.989 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.989 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.989 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.990 DEBUG:boto:Params: {} 2024-04-12T04:19:56.990 DEBUG:boto:Token: None 2024-04-12T04:19:56.990 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/obj145?versionId=null 2024-04-12T04:19:56.990 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HzZOrU6i3E8BAiPWu0i8+YjSd0c= 2024-04-12T04:19:56.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HzZOrU6i3E8BAiPWu0i8+YjSd0c=', 'Content-Length': '0'} 2024-04-12T04:19:56.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027ee023155ec616a-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.992 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.992 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.992 DEBUG:boto:path=/dkyges-63/?prefix=obj145 2024-04-12T04:19:56.992 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj145 2024-04-12T04:19:56.993 DEBUG:boto:Method: GET 2024-04-12T04:19:56.993 DEBUG:boto:Path: /dkyges-63/?prefix=obj145 2024-04-12T04:19:56.993 DEBUG:boto:Data: 2024-04-12T04:19:56.993 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.993 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:56.993 DEBUG:boto:Port: 8001 2024-04-12T04:19:56.993 DEBUG:boto:Params: {} 2024-04-12T04:19:56.993 DEBUG:boto:Token: None 2024-04-12T04:19:56.993 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.993 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:56.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c55394f9cf681783-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:56.996 DEBUG:boto:b'dkyges-63obj1451000falseobj1452024-04-12T04:15:48.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:56.996 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:56.997 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:56.997 DEBUG:boto:path=/dkyges-63/?prefix=obj145 2024-04-12T04:19:56.997 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj145 2024-04-12T04:19:56.997 DEBUG:boto:Method: GET 2024-04-12T04:19:56.997 DEBUG:boto:Path: /dkyges-63/?prefix=obj145 2024-04-12T04:19:56.997 DEBUG:boto:Data: 2024-04-12T04:19:56.997 DEBUG:boto:Headers: {} 2024-04-12T04:19:56.997 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:56.997 DEBUG:boto:Port: 8000 2024-04-12T04:19:56.997 DEBUG:boto:Params: {} 2024-04-12T04:19:56.997 DEBUG:boto:Token: None 2024-04-12T04:19:56.998 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:56 GMT /dkyges-63/ 2024-04-12T04:19:56.998 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70= 2024-04-12T04:19:56.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6qqvv8Iw8Mi1kGVbnUnu+Va2A70=', 'Content-Length': '0'} 2024-04-12T04:19:57.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b55b30a316423f54-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.002 DEBUG:boto:b'dkyges-63obj1451000falseobj1452024-04-12T04:15:48.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.003 DEBUG:boto:path=/dkyges-63/obj145 2024-04-12T04:19:57.003 DEBUG:boto:auth_path=/dkyges-63/obj145 2024-04-12T04:19:57.003 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.003 DEBUG:boto:Path: /dkyges-63/obj145 2024-04-12T04:19:57.003 DEBUG:boto:Data: 2024-04-12T04:19:57.003 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.003 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.003 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.003 DEBUG:boto:Params: {} 2024-04-12T04:19:57.004 DEBUG:boto:Token: None 2024-04-12T04:19:57.004 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj145 2024-04-12T04:19:57.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MtqetxBCgx4gaR0Wo+CiPkMCIAs= 2024-04-12T04:19:57.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MtqetxBCgx4gaR0Wo+CiPkMCIAs=', 'Content-Length': '0'} 2024-04-12T04:19:57.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078205809466812b5-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.006 DEBUG:boto:path=/dkyges-63/obj145 2024-04-12T04:19:57.006 DEBUG:boto:auth_path=/dkyges-63/obj145 2024-04-12T04:19:57.006 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.006 DEBUG:boto:Path: /dkyges-63/obj145 2024-04-12T04:19:57.007 DEBUG:boto:Data: 2024-04-12T04:19:57.007 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.007 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.007 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.007 DEBUG:boto:Params: {} 2024-04-12T04:19:57.007 DEBUG:boto:Token: None 2024-04-12T04:19:57.007 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj145 2024-04-12T04:19:57.007 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MtqetxBCgx4gaR0Wo+CiPkMCIAs= 2024-04-12T04:19:57.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MtqetxBCgx4gaR0Wo+CiPkMCIAs=', 'Content-Length': '0'} 2024-04-12T04:19:57.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000036276e55d4dd250-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2024-04-12T04:19:57.010 DEBUG:boto:path=/dkyges-63/obj145 2024-04-12T04:19:57.010 DEBUG:boto:auth_path=/dkyges-63/obj145 2024-04-12T04:19:57.010 DEBUG:boto:Method: GET 2024-04-12T04:19:57.010 DEBUG:boto:Path: /dkyges-63/obj145 2024-04-12T04:19:57.010 DEBUG:boto:Data: 2024-04-12T04:19:57.010 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.010 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.010 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.010 DEBUG:boto:Params: {} 2024-04-12T04:19:57.010 DEBUG:boto:Token: None 2024-04-12T04:19:57.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj145 2024-04-12T04:19:57.011 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X/tFNK6RRYlnpuXWQmMNNHtWfXA= 2024-04-12T04:19:57.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X/tFNK6RRYlnpuXWQmMNNHtWfXA=', 'Content-Length': '0'} 2024-04-12T04:19:57.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f36177e7ba8c2cd8-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.013 DEBUG:boto:path=/dkyges-63/obj145 2024-04-12T04:19:57.013 DEBUG:boto:auth_path=/dkyges-63/obj145 2024-04-12T04:19:57.013 DEBUG:boto:Method: GET 2024-04-12T04:19:57.013 DEBUG:boto:Path: /dkyges-63/obj145 2024-04-12T04:19:57.013 DEBUG:boto:Data: 2024-04-12T04:19:57.013 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.013 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.013 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.013 DEBUG:boto:Params: {} 2024-04-12T04:19:57.013 DEBUG:boto:Token: None 2024-04-12T04:19:57.014 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj145 2024-04-12T04:19:57.014 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:X/tFNK6RRYlnpuXWQmMNNHtWfXA= 2024-04-12T04:19:57.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:X/tFNK6RRYlnpuXWQmMNNHtWfXA=', 'Content-Length': '0'} 2024-04-12T04:19:57.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f1256709d8993a8-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.016 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.016 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.016 DEBUG:boto:path=/dkyges-63/?prefix=obj145 2024-04-12T04:19:57.016 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj145 2024-04-12T04:19:57.016 DEBUG:boto:Method: GET 2024-04-12T04:19:57.016 DEBUG:boto:Path: /dkyges-63/?prefix=obj145 2024-04-12T04:19:57.016 DEBUG:boto:Data: 2024-04-12T04:19:57.017 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.017 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.017 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.017 DEBUG:boto:Params: {} 2024-04-12T04:19:57.017 DEBUG:boto:Token: None 2024-04-12T04:19:57.017 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.017 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.020 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be80a128e3a96172-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.020 DEBUG:boto:b'dkyges-63obj1451000falseobj1452024-04-12T04:15:48.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.020 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.020 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.020 DEBUG:boto:path=/dkyges-63/?prefix=obj145 2024-04-12T04:19:57.020 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj145 2024-04-12T04:19:57.020 DEBUG:boto:Method: GET 2024-04-12T04:19:57.021 DEBUG:boto:Path: /dkyges-63/?prefix=obj145 2024-04-12T04:19:57.021 DEBUG:boto:Data: 2024-04-12T04:19:57.021 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.021 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.021 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.021 DEBUG:boto:Params: {} 2024-04-12T04:19:57.021 DEBUG:boto:Token: None 2024-04-12T04:19:57.021 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.021 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.026 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094245cf3ee161e07-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.026 DEBUG:boto:b'dkyges-63obj1451000falseobj1452024-04-12T04:15:48.959Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T04:19:57.026 DEBUG:boto:path=/dkyges-63/obj148 2024-04-12T04:19:57.026 DEBUG:boto:auth_path=/dkyges-63/obj148 2024-04-12T04:19:57.027 DEBUG:boto:path=/dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.027 DEBUG:boto:auth_path=/dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.027 DEBUG:boto:Method: GET 2024-04-12T04:19:57.027 DEBUG:boto:Path: /dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.027 DEBUG:boto:Data: 2024-04-12T04:19:57.027 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.027 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.027 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.027 DEBUG:boto:Params: {} 2024-04-12T04:19:57.027 DEBUG:boto:Token: None 2024-04-12T04:19:57.028 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.028 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EnwoyNADQYy7T5MmfqMAr2/ECUE= 2024-04-12T04:19:57.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EnwoyNADQYy7T5MmfqMAr2/ECUE=', 'Content-Length': '0'} 2024-04-12T04:19:57.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f87bd7a1332d6e7-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.070 DEBUG:boto:path=/dkyges-63/obj148 2024-04-12T04:19:57.070 DEBUG:boto:auth_path=/dkyges-63/obj148 2024-04-12T04:19:57.070 DEBUG:boto:path=/dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.071 DEBUG:boto:auth_path=/dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.071 DEBUG:boto:Method: GET 2024-04-12T04:19:57.071 DEBUG:boto:Path: /dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.071 DEBUG:boto:Data: 2024-04-12T04:19:57.071 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.071 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.071 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.071 DEBUG:boto:Params: {} 2024-04-12T04:19:57.071 DEBUG:boto:Token: None 2024-04-12T04:19:57.071 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.072 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EnwoyNADQYy7T5MmfqMAr2/ECUE= 2024-04-12T04:19:57.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EnwoyNADQYy7T5MmfqMAr2/ECUE=', 'Content-Length': '0'} 2024-04-12T04:19:57.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068adb2f224c94391-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.074 DEBUG:boto:path=/dkyges-63/obj148 2024-04-12T04:19:57.074 DEBUG:boto:auth_path=/dkyges-63/obj148 2024-04-12T04:19:57.074 DEBUG:boto:path=/dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.074 DEBUG:boto:auth_path=/dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.074 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.074 DEBUG:boto:Path: /dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.074 DEBUG:boto:Data: 2024-04-12T04:19:57.075 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.075 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.075 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.075 DEBUG:boto:Params: {} 2024-04-12T04:19:57.075 DEBUG:boto:Token: None 2024-04-12T04:19:57.075 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.075 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FMfFQcjBNUInbu/vv7vdJ/w+S6k= 2024-04-12T04:19:57.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FMfFQcjBNUInbu/vv7vdJ/w+S6k=', 'Content-Length': '0'} 2024-04-12T04:19:57.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff0db6e5be92683d-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.077 DEBUG:boto:path=/dkyges-63/obj148 2024-04-12T04:19:57.077 DEBUG:boto:auth_path=/dkyges-63/obj148 2024-04-12T04:19:57.077 DEBUG:boto:path=/dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.077 DEBUG:boto:auth_path=/dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.077 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.078 DEBUG:boto:Path: /dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.078 DEBUG:boto:Data: 2024-04-12T04:19:57.078 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.078 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.078 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.078 DEBUG:boto:Params: {} 2024-04-12T04:19:57.078 DEBUG:boto:Token: None 2024-04-12T04:19:57.078 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.078 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FMfFQcjBNUInbu/vv7vdJ/w+S6k= 2024-04-12T04:19:57.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FMfFQcjBNUInbu/vv7vdJ/w+S6k=', 'Content-Length': '0'} 2024-04-12T04:19:57.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000301edb80333ea7a7-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T04:19:57.081 DEBUG:boto:path=/dkyges-63/obj148 2024-04-12T04:19:57.081 DEBUG:boto:auth_path=/dkyges-63/obj148 2024-04-12T04:19:57.081 DEBUG:boto:path=/dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.081 DEBUG:boto:auth_path=/dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.081 DEBUG:boto:Method: GET 2024-04-12T04:19:57.081 DEBUG:boto:Path: /dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.081 DEBUG:boto:Data: 2024-04-12T04:19:57.081 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.081 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.081 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.082 DEBUG:boto:Params: {} 2024-04-12T04:19:57.082 DEBUG:boto:Token: None 2024-04-12T04:19:57.082 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.082 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EnwoyNADQYy7T5MmfqMAr2/ECUE= 2024-04-12T04:19:57.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EnwoyNADQYy7T5MmfqMAr2/ECUE=', 'Content-Length': '0'} 2024-04-12T04:19:57.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ccbb45bf02f928e-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.084 DEBUG:boto:path=/dkyges-63/obj148 2024-04-12T04:19:57.084 DEBUG:boto:auth_path=/dkyges-63/obj148 2024-04-12T04:19:57.084 DEBUG:boto:path=/dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.084 DEBUG:boto:auth_path=/dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.084 DEBUG:boto:Method: GET 2024-04-12T04:19:57.085 DEBUG:boto:Path: /dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.085 DEBUG:boto:Data: 2024-04-12T04:19:57.085 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.085 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.085 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.085 DEBUG:boto:Params: {} 2024-04-12T04:19:57.085 DEBUG:boto:Token: None 2024-04-12T04:19:57.085 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj148?versionId=null 2024-04-12T04:19:57.085 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EnwoyNADQYy7T5MmfqMAr2/ECUE= 2024-04-12T04:19:57.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EnwoyNADQYy7T5MmfqMAr2/ECUE=', 'Content-Length': '0'} 2024-04-12T04:19:57.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bb87708883c7219-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.088 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.088 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.088 DEBUG:boto:path=/dkyges-63/?prefix=obj148 2024-04-12T04:19:57.088 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj148 2024-04-12T04:19:57.088 DEBUG:boto:Method: GET 2024-04-12T04:19:57.088 DEBUG:boto:Path: /dkyges-63/?prefix=obj148 2024-04-12T04:19:57.088 DEBUG:boto:Data: 2024-04-12T04:19:57.088 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.088 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.088 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.088 DEBUG:boto:Params: {} 2024-04-12T04:19:57.089 DEBUG:boto:Token: None 2024-04-12T04:19:57.089 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.089 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040aec0001ec4c8f1-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.092 DEBUG:boto:b'dkyges-63obj1481000falseobj1482024-04-12T04:15:49.677Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.092 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.092 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.092 DEBUG:boto:path=/dkyges-63/?prefix=obj148 2024-04-12T04:19:57.092 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj148 2024-04-12T04:19:57.092 DEBUG:boto:Method: GET 2024-04-12T04:19:57.092 DEBUG:boto:Path: /dkyges-63/?prefix=obj148 2024-04-12T04:19:57.093 DEBUG:boto:Data: 2024-04-12T04:19:57.093 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.093 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.093 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.093 DEBUG:boto:Params: {} 2024-04-12T04:19:57.093 DEBUG:boto:Token: None 2024-04-12T04:19:57.093 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.093 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d42516d04d7023a-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.138 DEBUG:boto:b'dkyges-63obj1481000falseobj1482024-04-12T04:15:49.677Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.139 DEBUG:boto:path=/dkyges-63/obj148 2024-04-12T04:19:57.139 DEBUG:boto:auth_path=/dkyges-63/obj148 2024-04-12T04:19:57.139 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.139 DEBUG:boto:Path: /dkyges-63/obj148 2024-04-12T04:19:57.139 DEBUG:boto:Data: 2024-04-12T04:19:57.139 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.139 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.139 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.139 DEBUG:boto:Params: {} 2024-04-12T04:19:57.139 DEBUG:boto:Token: None 2024-04-12T04:19:57.140 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj148 2024-04-12T04:19:57.140 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:65RMldeGLVN5XhxVC6vDOt+jhqM= 2024-04-12T04:19:57.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:65RMldeGLVN5XhxVC6vDOt+jhqM=', 'Content-Length': '0'} 2024-04-12T04:19:57.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f03b35843d83ccb4-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.142 DEBUG:boto:path=/dkyges-63/obj148 2024-04-12T04:19:57.142 DEBUG:boto:auth_path=/dkyges-63/obj148 2024-04-12T04:19:57.142 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.142 DEBUG:boto:Path: /dkyges-63/obj148 2024-04-12T04:19:57.142 DEBUG:boto:Data: 2024-04-12T04:19:57.143 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.143 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.143 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.143 DEBUG:boto:Params: {} 2024-04-12T04:19:57.143 DEBUG:boto:Token: None 2024-04-12T04:19:57.143 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj148 2024-04-12T04:19:57.143 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:65RMldeGLVN5XhxVC6vDOt+jhqM= 2024-04-12T04:19:57.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:65RMldeGLVN5XhxVC6vDOt+jhqM=', 'Content-Length': '0'} 2024-04-12T04:19:57.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009881b52af535c767-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2024-04-12T04:19:57.145 DEBUG:boto:path=/dkyges-63/obj148 2024-04-12T04:19:57.145 DEBUG:boto:auth_path=/dkyges-63/obj148 2024-04-12T04:19:57.145 DEBUG:boto:Method: GET 2024-04-12T04:19:57.146 DEBUG:boto:Path: /dkyges-63/obj148 2024-04-12T04:19:57.146 DEBUG:boto:Data: 2024-04-12T04:19:57.146 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.146 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.146 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.146 DEBUG:boto:Params: {} 2024-04-12T04:19:57.146 DEBUG:boto:Token: None 2024-04-12T04:19:57.146 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj148 2024-04-12T04:19:57.146 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QvdDwzBPnmtE93/B3ZmCjOCz82U= 2024-04-12T04:19:57.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QvdDwzBPnmtE93/B3ZmCjOCz82U=', 'Content-Length': '0'} 2024-04-12T04:19:57.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2980c88ae60e175-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.148 DEBUG:boto:path=/dkyges-63/obj148 2024-04-12T04:19:57.149 DEBUG:boto:auth_path=/dkyges-63/obj148 2024-04-12T04:19:57.149 DEBUG:boto:Method: GET 2024-04-12T04:19:57.149 DEBUG:boto:Path: /dkyges-63/obj148 2024-04-12T04:19:57.149 DEBUG:boto:Data: 2024-04-12T04:19:57.149 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.149 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.149 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.149 DEBUG:boto:Params: {} 2024-04-12T04:19:57.149 DEBUG:boto:Token: None 2024-04-12T04:19:57.149 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj148 2024-04-12T04:19:57.149 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QvdDwzBPnmtE93/B3ZmCjOCz82U= 2024-04-12T04:19:57.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QvdDwzBPnmtE93/B3ZmCjOCz82U=', 'Content-Length': '0'} 2024-04-12T04:19:57.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002981419023ea517e-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.152 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.152 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.152 DEBUG:boto:path=/dkyges-63/?prefix=obj148 2024-04-12T04:19:57.152 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj148 2024-04-12T04:19:57.152 DEBUG:boto:Method: GET 2024-04-12T04:19:57.152 DEBUG:boto:Path: /dkyges-63/?prefix=obj148 2024-04-12T04:19:57.152 DEBUG:boto:Data: 2024-04-12T04:19:57.152 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.152 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.152 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.152 DEBUG:boto:Params: {} 2024-04-12T04:19:57.152 DEBUG:boto:Token: None 2024-04-12T04:19:57.153 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.153 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7c7313f0dcb4980-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.156 DEBUG:boto:b'dkyges-63obj1481000falseobj1482024-04-12T04:15:49.677Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.156 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.156 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.156 DEBUG:boto:path=/dkyges-63/?prefix=obj148 2024-04-12T04:19:57.156 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj148 2024-04-12T04:19:57.156 DEBUG:boto:Method: GET 2024-04-12T04:19:57.156 DEBUG:boto:Path: /dkyges-63/?prefix=obj148 2024-04-12T04:19:57.157 DEBUG:boto:Data: 2024-04-12T04:19:57.157 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.157 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.157 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.157 DEBUG:boto:Params: {} 2024-04-12T04:19:57.157 DEBUG:boto:Token: None 2024-04-12T04:19:57.157 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.157 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e143e0bed9c70bf-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.162 DEBUG:boto:b'dkyges-63obj1481000falseobj1482024-04-12T04:15:49.677Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T04:19:57.162 DEBUG:boto:path=/dkyges-63/obj155 2024-04-12T04:19:57.162 DEBUG:boto:auth_path=/dkyges-63/obj155 2024-04-12T04:19:57.162 DEBUG:boto:path=/dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.163 DEBUG:boto:auth_path=/dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.163 DEBUG:boto:Method: GET 2024-04-12T04:19:57.163 DEBUG:boto:Path: /dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.163 DEBUG:boto:Data: 2024-04-12T04:19:57.163 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.163 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.163 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.163 DEBUG:boto:Params: {} 2024-04-12T04:19:57.163 DEBUG:boto:Token: None 2024-04-12T04:19:57.163 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.164 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H/mWFg52H6uo+yLvjBd9S939ALo= 2024-04-12T04:19:57.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H/mWFg52H6uo+yLvjBd9S939ALo=', 'Content-Length': '0'} 2024-04-12T04:19:57.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025c94c19ee21f792-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.166 DEBUG:boto:path=/dkyges-63/obj155 2024-04-12T04:19:57.166 DEBUG:boto:auth_path=/dkyges-63/obj155 2024-04-12T04:19:57.166 DEBUG:boto:path=/dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.166 DEBUG:boto:auth_path=/dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.167 DEBUG:boto:Method: GET 2024-04-12T04:19:57.167 DEBUG:boto:Path: /dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.167 DEBUG:boto:Data: 2024-04-12T04:19:57.167 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.167 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.167 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.167 DEBUG:boto:Params: {} 2024-04-12T04:19:57.167 DEBUG:boto:Token: None 2024-04-12T04:19:57.167 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.167 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H/mWFg52H6uo+yLvjBd9S939ALo= 2024-04-12T04:19:57.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H/mWFg52H6uo+yLvjBd9S939ALo=', 'Content-Length': '0'} 2024-04-12T04:19:57.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfe0208ec775fc0d-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.214 DEBUG:boto:path=/dkyges-63/obj155 2024-04-12T04:19:57.215 DEBUG:boto:auth_path=/dkyges-63/obj155 2024-04-12T04:19:57.215 DEBUG:boto:path=/dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.215 DEBUG:boto:auth_path=/dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.215 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.215 DEBUG:boto:Path: /dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.215 DEBUG:boto:Data: 2024-04-12T04:19:57.215 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.215 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.215 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.215 DEBUG:boto:Params: {} 2024-04-12T04:19:57.215 DEBUG:boto:Token: None 2024-04-12T04:19:57.216 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.216 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:npgBQfRafZiThK5n2xdQ4YZNNWA= 2024-04-12T04:19:57.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:npgBQfRafZiThK5n2xdQ4YZNNWA=', 'Content-Length': '0'} 2024-04-12T04:19:57.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b54f226be79067e-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.218 DEBUG:boto:path=/dkyges-63/obj155 2024-04-12T04:19:57.218 DEBUG:boto:auth_path=/dkyges-63/obj155 2024-04-12T04:19:57.218 DEBUG:boto:path=/dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.218 DEBUG:boto:auth_path=/dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.218 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.218 DEBUG:boto:Path: /dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.218 DEBUG:boto:Data: 2024-04-12T04:19:57.218 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.219 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.219 DEBUG:boto:Params: {} 2024-04-12T04:19:57.219 DEBUG:boto:Token: None 2024-04-12T04:19:57.219 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.219 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:npgBQfRafZiThK5n2xdQ4YZNNWA= 2024-04-12T04:19:57.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:npgBQfRafZiThK5n2xdQ4YZNNWA=', 'Content-Length': '0'} 2024-04-12T04:19:57.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2f3cb1ebb6cfcd5-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T04:19:57.221 DEBUG:boto:path=/dkyges-63/obj155 2024-04-12T04:19:57.221 DEBUG:boto:auth_path=/dkyges-63/obj155 2024-04-12T04:19:57.221 DEBUG:boto:path=/dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.221 DEBUG:boto:auth_path=/dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.222 DEBUG:boto:Method: GET 2024-04-12T04:19:57.222 DEBUG:boto:Path: /dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.222 DEBUG:boto:Data: 2024-04-12T04:19:57.222 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.222 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.222 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.222 DEBUG:boto:Params: {} 2024-04-12T04:19:57.222 DEBUG:boto:Token: None 2024-04-12T04:19:57.222 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.222 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H/mWFg52H6uo+yLvjBd9S939ALo= 2024-04-12T04:19:57.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H/mWFg52H6uo+yLvjBd9S939ALo=', 'Content-Length': '0'} 2024-04-12T04:19:57.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b61b9622c1ec4c6-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.225 DEBUG:boto:path=/dkyges-63/obj155 2024-04-12T04:19:57.225 DEBUG:boto:auth_path=/dkyges-63/obj155 2024-04-12T04:19:57.225 DEBUG:boto:path=/dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.225 DEBUG:boto:auth_path=/dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.225 DEBUG:boto:Method: GET 2024-04-12T04:19:57.225 DEBUG:boto:Path: /dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.225 DEBUG:boto:Data: 2024-04-12T04:19:57.225 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.225 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.225 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.226 DEBUG:boto:Params: {} 2024-04-12T04:19:57.226 DEBUG:boto:Token: None 2024-04-12T04:19:57.226 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj155?versionId=null 2024-04-12T04:19:57.226 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H/mWFg52H6uo+yLvjBd9S939ALo= 2024-04-12T04:19:57.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H/mWFg52H6uo+yLvjBd9S939ALo=', 'Content-Length': '0'} 2024-04-12T04:19:57.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000683490fc05648ff9-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.228 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.228 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.228 DEBUG:boto:path=/dkyges-63/?prefix=obj155 2024-04-12T04:19:57.229 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj155 2024-04-12T04:19:57.229 DEBUG:boto:Method: GET 2024-04-12T04:19:57.229 DEBUG:boto:Path: /dkyges-63/?prefix=obj155 2024-04-12T04:19:57.229 DEBUG:boto:Data: 2024-04-12T04:19:57.229 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.229 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.229 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.229 DEBUG:boto:Params: {} 2024-04-12T04:19:57.229 DEBUG:boto:Token: None 2024-04-12T04:19:57.229 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.229 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.232 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008935a3661c241b3-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.232 DEBUG:boto:b'dkyges-63obj1551000falseobj1552024-04-12T04:15:49.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.232 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.233 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.233 DEBUG:boto:path=/dkyges-63/?prefix=obj155 2024-04-12T04:19:57.233 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj155 2024-04-12T04:19:57.233 DEBUG:boto:Method: GET 2024-04-12T04:19:57.233 DEBUG:boto:Path: /dkyges-63/?prefix=obj155 2024-04-12T04:19:57.233 DEBUG:boto:Data: 2024-04-12T04:19:57.233 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.233 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.233 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.233 DEBUG:boto:Params: {} 2024-04-12T04:19:57.233 DEBUG:boto:Token: None 2024-04-12T04:19:57.234 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.234 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7cc7b55643bc4a5-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.238 DEBUG:boto:b'dkyges-63obj1551000falseobj1552024-04-12T04:15:49.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.239 DEBUG:boto:path=/dkyges-63/obj155 2024-04-12T04:19:57.239 DEBUG:boto:auth_path=/dkyges-63/obj155 2024-04-12T04:19:57.239 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.239 DEBUG:boto:Path: /dkyges-63/obj155 2024-04-12T04:19:57.239 DEBUG:boto:Data: 2024-04-12T04:19:57.239 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.239 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.239 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.239 DEBUG:boto:Params: {} 2024-04-12T04:19:57.240 DEBUG:boto:Token: None 2024-04-12T04:19:57.240 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj155 2024-04-12T04:19:57.240 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4aZdw+nXz/40FTvQTVs1so7tpfc= 2024-04-12T04:19:57.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4aZdw+nXz/40FTvQTVs1so7tpfc=', 'Content-Length': '0'} 2024-04-12T04:19:57.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000946fe2dda9715053-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.242 DEBUG:boto:path=/dkyges-63/obj155 2024-04-12T04:19:57.242 DEBUG:boto:auth_path=/dkyges-63/obj155 2024-04-12T04:19:57.242 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.242 DEBUG:boto:Path: /dkyges-63/obj155 2024-04-12T04:19:57.242 DEBUG:boto:Data: 2024-04-12T04:19:57.242 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.242 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.242 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.242 DEBUG:boto:Params: {} 2024-04-12T04:19:57.243 DEBUG:boto:Token: None 2024-04-12T04:19:57.243 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj155 2024-04-12T04:19:57.243 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4aZdw+nXz/40FTvQTVs1so7tpfc= 2024-04-12T04:19:57.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4aZdw+nXz/40FTvQTVs1so7tpfc=', 'Content-Length': '0'} 2024-04-12T04:19:57.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000725d3d99c57b5a11-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2024-04-12T04:19:57.245 DEBUG:boto:path=/dkyges-63/obj155 2024-04-12T04:19:57.245 DEBUG:boto:auth_path=/dkyges-63/obj155 2024-04-12T04:19:57.245 DEBUG:boto:Method: GET 2024-04-12T04:19:57.245 DEBUG:boto:Path: /dkyges-63/obj155 2024-04-12T04:19:57.245 DEBUG:boto:Data: 2024-04-12T04:19:57.245 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.246 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.246 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.246 DEBUG:boto:Params: {} 2024-04-12T04:19:57.246 DEBUG:boto:Token: None 2024-04-12T04:19:57.246 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj155 2024-04-12T04:19:57.246 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qEApLLH3iJ2qxu4ib67cyJcgj7c= 2024-04-12T04:19:57.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qEApLLH3iJ2qxu4ib67cyJcgj7c=', 'Content-Length': '0'} 2024-04-12T04:19:57.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f08ae2e69a6939c-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.294 DEBUG:boto:path=/dkyges-63/obj155 2024-04-12T04:19:57.294 DEBUG:boto:auth_path=/dkyges-63/obj155 2024-04-12T04:19:57.294 DEBUG:boto:Method: GET 2024-04-12T04:19:57.295 DEBUG:boto:Path: /dkyges-63/obj155 2024-04-12T04:19:57.295 DEBUG:boto:Data: 2024-04-12T04:19:57.295 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.295 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.295 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.295 DEBUG:boto:Params: {} 2024-04-12T04:19:57.295 DEBUG:boto:Token: None 2024-04-12T04:19:57.295 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj155 2024-04-12T04:19:57.295 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qEApLLH3iJ2qxu4ib67cyJcgj7c= 2024-04-12T04:19:57.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qEApLLH3iJ2qxu4ib67cyJcgj7c=', 'Content-Length': '0'} 2024-04-12T04:19:57.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eccd9d49aaf05c1-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.297 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.298 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.298 DEBUG:boto:path=/dkyges-63/?prefix=obj155 2024-04-12T04:19:57.298 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj155 2024-04-12T04:19:57.298 DEBUG:boto:Method: GET 2024-04-12T04:19:57.298 DEBUG:boto:Path: /dkyges-63/?prefix=obj155 2024-04-12T04:19:57.298 DEBUG:boto:Data: 2024-04-12T04:19:57.298 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.298 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.298 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.298 DEBUG:boto:Params: {} 2024-04-12T04:19:57.298 DEBUG:boto:Token: None 2024-04-12T04:19:57.299 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.299 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.301 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000788f19ee13a40cb3-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.301 DEBUG:boto:b'dkyges-63obj1551000falseobj1552024-04-12T04:15:49.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.302 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.302 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.302 DEBUG:boto:path=/dkyges-63/?prefix=obj155 2024-04-12T04:19:57.302 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj155 2024-04-12T04:19:57.302 DEBUG:boto:Method: GET 2024-04-12T04:19:57.302 DEBUG:boto:Path: /dkyges-63/?prefix=obj155 2024-04-12T04:19:57.302 DEBUG:boto:Data: 2024-04-12T04:19:57.302 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.302 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.302 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.303 DEBUG:boto:Params: {} 2024-04-12T04:19:57.303 DEBUG:boto:Token: None 2024-04-12T04:19:57.303 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.303 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003573049fce9df05f-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.307 DEBUG:boto:b'dkyges-63obj1551000falseobj1552024-04-12T04:15:49.321Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:19:57.308 DEBUG:boto:path=/dkyges-63/obj17 2024-04-12T04:19:57.308 DEBUG:boto:auth_path=/dkyges-63/obj17 2024-04-12T04:19:57.308 DEBUG:boto:path=/dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.308 DEBUG:boto:auth_path=/dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.308 DEBUG:boto:Method: GET 2024-04-12T04:19:57.308 DEBUG:boto:Path: /dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.308 DEBUG:boto:Data: 2024-04-12T04:19:57.309 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.309 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.309 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.309 DEBUG:boto:Params: {} 2024-04-12T04:19:57.309 DEBUG:boto:Token: None 2024-04-12T04:19:57.309 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.309 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BEQQB+mpPxse5qj8po6cB+GZSEw= 2024-04-12T04:19:57.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BEQQB+mpPxse5qj8po6cB+GZSEw=', 'Content-Length': '0'} 2024-04-12T04:19:57.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e622a7e8c1f440e6-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.311 DEBUG:boto:path=/dkyges-63/obj17 2024-04-12T04:19:57.312 DEBUG:boto:auth_path=/dkyges-63/obj17 2024-04-12T04:19:57.312 DEBUG:boto:path=/dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.312 DEBUG:boto:auth_path=/dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.312 DEBUG:boto:Method: GET 2024-04-12T04:19:57.312 DEBUG:boto:Path: /dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.312 DEBUG:boto:Data: 2024-04-12T04:19:57.312 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.312 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.312 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.312 DEBUG:boto:Params: {} 2024-04-12T04:19:57.312 DEBUG:boto:Token: None 2024-04-12T04:19:57.313 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.313 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BEQQB+mpPxse5qj8po6cB+GZSEw= 2024-04-12T04:19:57.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BEQQB+mpPxse5qj8po6cB+GZSEw=', 'Content-Length': '0'} 2024-04-12T04:19:57.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002dde6dbe1475289-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.358 DEBUG:boto:path=/dkyges-63/obj17 2024-04-12T04:19:57.358 DEBUG:boto:auth_path=/dkyges-63/obj17 2024-04-12T04:19:57.358 DEBUG:boto:path=/dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.359 DEBUG:boto:auth_path=/dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.359 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.359 DEBUG:boto:Path: /dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.359 DEBUG:boto:Data: 2024-04-12T04:19:57.359 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.359 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.359 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.359 DEBUG:boto:Params: {} 2024-04-12T04:19:57.359 DEBUG:boto:Token: None 2024-04-12T04:19:57.359 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.360 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UtMRS6ShqMj8Bcy+56t9meUpcSw= 2024-04-12T04:19:57.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UtMRS6ShqMj8Bcy+56t9meUpcSw=', 'Content-Length': '0'} 2024-04-12T04:19:57.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d341f4d1d161482-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.362 DEBUG:boto:path=/dkyges-63/obj17 2024-04-12T04:19:57.362 DEBUG:boto:auth_path=/dkyges-63/obj17 2024-04-12T04:19:57.362 DEBUG:boto:path=/dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.362 DEBUG:boto:auth_path=/dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.362 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.362 DEBUG:boto:Path: /dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.362 DEBUG:boto:Data: 2024-04-12T04:19:57.362 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.362 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.362 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.362 DEBUG:boto:Params: {} 2024-04-12T04:19:57.363 DEBUG:boto:Token: None 2024-04-12T04:19:57.363 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.363 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UtMRS6ShqMj8Bcy+56t9meUpcSw= 2024-04-12T04:19:57.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UtMRS6ShqMj8Bcy+56t9meUpcSw=', 'Content-Length': '0'} 2024-04-12T04:19:57.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b25e35dfc4cff6e6-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:19:57.365 DEBUG:boto:path=/dkyges-63/obj17 2024-04-12T04:19:57.365 DEBUG:boto:auth_path=/dkyges-63/obj17 2024-04-12T04:19:57.365 DEBUG:boto:path=/dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.365 DEBUG:boto:auth_path=/dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.365 DEBUG:boto:Method: GET 2024-04-12T04:19:57.365 DEBUG:boto:Path: /dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.365 DEBUG:boto:Data: 2024-04-12T04:19:57.366 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.366 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.366 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.366 DEBUG:boto:Params: {} 2024-04-12T04:19:57.366 DEBUG:boto:Token: None 2024-04-12T04:19:57.366 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.366 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BEQQB+mpPxse5qj8po6cB+GZSEw= 2024-04-12T04:19:57.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BEQQB+mpPxse5qj8po6cB+GZSEw=', 'Content-Length': '0'} 2024-04-12T04:19:57.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dacd4410492d8f31-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.368 DEBUG:boto:path=/dkyges-63/obj17 2024-04-12T04:19:57.369 DEBUG:boto:auth_path=/dkyges-63/obj17 2024-04-12T04:19:57.369 DEBUG:boto:path=/dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.369 DEBUG:boto:auth_path=/dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.369 DEBUG:boto:Method: GET 2024-04-12T04:19:57.369 DEBUG:boto:Path: /dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.369 DEBUG:boto:Data: 2024-04-12T04:19:57.369 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.369 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.369 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.369 DEBUG:boto:Params: {} 2024-04-12T04:19:57.369 DEBUG:boto:Token: None 2024-04-12T04:19:57.370 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj17?versionId=null 2024-04-12T04:19:57.370 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BEQQB+mpPxse5qj8po6cB+GZSEw= 2024-04-12T04:19:57.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BEQQB+mpPxse5qj8po6cB+GZSEw=', 'Content-Length': '0'} 2024-04-12T04:19:57.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e91ed23ba880f8f4-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.372 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.372 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.372 DEBUG:boto:path=/dkyges-63/?prefix=obj17 2024-04-12T04:19:57.372 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj17 2024-04-12T04:19:57.372 DEBUG:boto:Method: GET 2024-04-12T04:19:57.372 DEBUG:boto:Path: /dkyges-63/?prefix=obj17 2024-04-12T04:19:57.372 DEBUG:boto:Data: 2024-04-12T04:19:57.373 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.373 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.373 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.373 DEBUG:boto:Params: {} 2024-04-12T04:19:57.373 DEBUG:boto:Token: None 2024-04-12T04:19:57.373 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.373 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089fce991cea351a3-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.376 DEBUG:boto:b'dkyges-63obj171000falseobj172024-04-12T04:15:47.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.376 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.376 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.376 DEBUG:boto:path=/dkyges-63/?prefix=obj17 2024-04-12T04:19:57.377 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj17 2024-04-12T04:19:57.377 DEBUG:boto:Method: GET 2024-04-12T04:19:57.377 DEBUG:boto:Path: /dkyges-63/?prefix=obj17 2024-04-12T04:19:57.377 DEBUG:boto:Data: 2024-04-12T04:19:57.377 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.377 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.377 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.377 DEBUG:boto:Params: {} 2024-04-12T04:19:57.377 DEBUG:boto:Token: None 2024-04-12T04:19:57.377 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.377 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e43067623ae8c486-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.382 DEBUG:boto:b'dkyges-63obj171000falseobj172024-04-12T04:15:47.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.382 DEBUG:boto:path=/dkyges-63/obj17 2024-04-12T04:19:57.382 DEBUG:boto:auth_path=/dkyges-63/obj17 2024-04-12T04:19:57.382 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.383 DEBUG:boto:Path: /dkyges-63/obj17 2024-04-12T04:19:57.383 DEBUG:boto:Data: 2024-04-12T04:19:57.383 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.383 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.383 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.383 DEBUG:boto:Params: {} 2024-04-12T04:19:57.383 DEBUG:boto:Token: None 2024-04-12T04:19:57.383 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj17 2024-04-12T04:19:57.383 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e+wb5dUWCfqu3mCLoAw+d2Xk5rE= 2024-04-12T04:19:57.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e+wb5dUWCfqu3mCLoAw+d2Xk5rE=', 'Content-Length': '0'} 2024-04-12T04:19:57.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083c6c9aed0aceaa6-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.385 DEBUG:boto:path=/dkyges-63/obj17 2024-04-12T04:19:57.385 DEBUG:boto:auth_path=/dkyges-63/obj17 2024-04-12T04:19:57.386 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.386 DEBUG:boto:Path: /dkyges-63/obj17 2024-04-12T04:19:57.386 DEBUG:boto:Data: 2024-04-12T04:19:57.386 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.386 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.386 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.386 DEBUG:boto:Params: {} 2024-04-12T04:19:57.386 DEBUG:boto:Token: None 2024-04-12T04:19:57.386 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj17 2024-04-12T04:19:57.386 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e+wb5dUWCfqu3mCLoAw+d2Xk5rE= 2024-04-12T04:19:57.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e+wb5dUWCfqu3mCLoAw+d2Xk5rE=', 'Content-Length': '0'} 2024-04-12T04:19:57.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d7761183b320ade-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2024-04-12T04:19:57.389 DEBUG:boto:path=/dkyges-63/obj17 2024-04-12T04:19:57.389 DEBUG:boto:auth_path=/dkyges-63/obj17 2024-04-12T04:19:57.389 DEBUG:boto:Method: GET 2024-04-12T04:19:57.389 DEBUG:boto:Path: /dkyges-63/obj17 2024-04-12T04:19:57.389 DEBUG:boto:Data: 2024-04-12T04:19:57.389 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.389 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.389 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.389 DEBUG:boto:Params: {} 2024-04-12T04:19:57.389 DEBUG:boto:Token: None 2024-04-12T04:19:57.390 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj17 2024-04-12T04:19:57.390 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qu7PosfbbEK3D65mxtIwlLVUmz0= 2024-04-12T04:19:57.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qu7PosfbbEK3D65mxtIwlLVUmz0=', 'Content-Length': '0'} 2024-04-12T04:19:57.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005e9acf0490b1516-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.392 DEBUG:boto:path=/dkyges-63/obj17 2024-04-12T04:19:57.392 DEBUG:boto:auth_path=/dkyges-63/obj17 2024-04-12T04:19:57.392 DEBUG:boto:Method: GET 2024-04-12T04:19:57.392 DEBUG:boto:Path: /dkyges-63/obj17 2024-04-12T04:19:57.392 DEBUG:boto:Data: 2024-04-12T04:19:57.392 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.392 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.393 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.393 DEBUG:boto:Params: {} 2024-04-12T04:19:57.393 DEBUG:boto:Token: None 2024-04-12T04:19:57.393 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj17 2024-04-12T04:19:57.393 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qu7PosfbbEK3D65mxtIwlLVUmz0= 2024-04-12T04:19:57.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qu7PosfbbEK3D65mxtIwlLVUmz0=', 'Content-Length': '0'} 2024-04-12T04:19:57.395 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eadf6ff23ae2f501-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.395 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.395 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.395 DEBUG:boto:path=/dkyges-63/?prefix=obj17 2024-04-12T04:19:57.395 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj17 2024-04-12T04:19:57.395 DEBUG:boto:Method: GET 2024-04-12T04:19:57.396 DEBUG:boto:Path: /dkyges-63/?prefix=obj17 2024-04-12T04:19:57.396 DEBUG:boto:Data: 2024-04-12T04:19:57.396 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.396 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.396 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.396 DEBUG:boto:Params: {} 2024-04-12T04:19:57.396 DEBUG:boto:Token: None 2024-04-12T04:19:57.396 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.396 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b506fde7e7f5e3f0-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.399 DEBUG:boto:b'dkyges-63obj171000falseobj172024-04-12T04:15:47.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.399 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.400 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.400 DEBUG:boto:path=/dkyges-63/?prefix=obj17 2024-04-12T04:19:57.400 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj17 2024-04-12T04:19:57.400 DEBUG:boto:Method: GET 2024-04-12T04:19:57.400 DEBUG:boto:Path: /dkyges-63/?prefix=obj17 2024-04-12T04:19:57.400 DEBUG:boto:Data: 2024-04-12T04:19:57.400 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.400 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.400 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.400 DEBUG:boto:Params: {} 2024-04-12T04:19:57.400 DEBUG:boto:Token: None 2024-04-12T04:19:57.401 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.401 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2223dc923870eb8-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.446 DEBUG:boto:b'dkyges-63obj171000falseobj172024-04-12T04:15:47.761Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T04:19:57.447 DEBUG:boto:path=/dkyges-63/obj185 2024-04-12T04:19:57.447 DEBUG:boto:auth_path=/dkyges-63/obj185 2024-04-12T04:19:57.447 DEBUG:boto:path=/dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.447 DEBUG:boto:auth_path=/dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.447 DEBUG:boto:Method: GET 2024-04-12T04:19:57.447 DEBUG:boto:Path: /dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.447 DEBUG:boto:Data: 2024-04-12T04:19:57.447 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.448 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.448 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.448 DEBUG:boto:Params: {} 2024-04-12T04:19:57.448 DEBUG:boto:Token: None 2024-04-12T04:19:57.448 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.448 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:anPYPCJNsn6cY/3qp4fSK+i5Yfs= 2024-04-12T04:19:57.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:anPYPCJNsn6cY/3qp4fSK+i5Yfs=', 'Content-Length': '0'} 2024-04-12T04:19:57.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b79b05cf66e58c7-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.450 DEBUG:boto:path=/dkyges-63/obj185 2024-04-12T04:19:57.450 DEBUG:boto:auth_path=/dkyges-63/obj185 2024-04-12T04:19:57.451 DEBUG:boto:path=/dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.451 DEBUG:boto:auth_path=/dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.451 DEBUG:boto:Method: GET 2024-04-12T04:19:57.451 DEBUG:boto:Path: /dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.451 DEBUG:boto:Data: 2024-04-12T04:19:57.451 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.451 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.451 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.451 DEBUG:boto:Params: {} 2024-04-12T04:19:57.451 DEBUG:boto:Token: None 2024-04-12T04:19:57.451 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.452 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:anPYPCJNsn6cY/3qp4fSK+i5Yfs= 2024-04-12T04:19:57.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:anPYPCJNsn6cY/3qp4fSK+i5Yfs=', 'Content-Length': '0'} 2024-04-12T04:19:57.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1290883353b485e-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.454 DEBUG:boto:path=/dkyges-63/obj185 2024-04-12T04:19:57.454 DEBUG:boto:auth_path=/dkyges-63/obj185 2024-04-12T04:19:57.454 DEBUG:boto:path=/dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.454 DEBUG:boto:auth_path=/dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.454 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.454 DEBUG:boto:Path: /dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.454 DEBUG:boto:Data: 2024-04-12T04:19:57.454 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.455 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.455 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.455 DEBUG:boto:Params: {} 2024-04-12T04:19:57.455 DEBUG:boto:Token: None 2024-04-12T04:19:57.455 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.455 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:koHYwGq479tOWysnM2ocR5Fr340= 2024-04-12T04:19:57.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:koHYwGq479tOWysnM2ocR5Fr340=', 'Content-Length': '0'} 2024-04-12T04:19:57.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082f4dc6ba7950b71-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.457 DEBUG:boto:path=/dkyges-63/obj185 2024-04-12T04:19:57.457 DEBUG:boto:auth_path=/dkyges-63/obj185 2024-04-12T04:19:57.457 DEBUG:boto:path=/dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.457 DEBUG:boto:auth_path=/dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.457 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.457 DEBUG:boto:Path: /dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.458 DEBUG:boto:Data: 2024-04-12T04:19:57.458 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.458 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.458 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.458 DEBUG:boto:Params: {} 2024-04-12T04:19:57.458 DEBUG:boto:Token: None 2024-04-12T04:19:57.458 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.458 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:koHYwGq479tOWysnM2ocR5Fr340= 2024-04-12T04:19:57.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:koHYwGq479tOWysnM2ocR5Fr340=', 'Content-Length': '0'} 2024-04-12T04:19:57.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094e459e864065af9-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T04:19:57.460 DEBUG:boto:path=/dkyges-63/obj185 2024-04-12T04:19:57.461 DEBUG:boto:auth_path=/dkyges-63/obj185 2024-04-12T04:19:57.461 DEBUG:boto:path=/dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.461 DEBUG:boto:auth_path=/dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.461 DEBUG:boto:Method: GET 2024-04-12T04:19:57.461 DEBUG:boto:Path: /dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.461 DEBUG:boto:Data: 2024-04-12T04:19:57.461 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.461 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.461 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.461 DEBUG:boto:Params: {} 2024-04-12T04:19:57.461 DEBUG:boto:Token: None 2024-04-12T04:19:57.462 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.462 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:anPYPCJNsn6cY/3qp4fSK+i5Yfs= 2024-04-12T04:19:57.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:anPYPCJNsn6cY/3qp4fSK+i5Yfs=', 'Content-Length': '0'} 2024-04-12T04:19:57.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5324999bc362da1-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.464 DEBUG:boto:path=/dkyges-63/obj185 2024-04-12T04:19:57.464 DEBUG:boto:auth_path=/dkyges-63/obj185 2024-04-12T04:19:57.464 DEBUG:boto:path=/dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.464 DEBUG:boto:auth_path=/dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.464 DEBUG:boto:Method: GET 2024-04-12T04:19:57.464 DEBUG:boto:Path: /dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.464 DEBUG:boto:Data: 2024-04-12T04:19:57.464 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.464 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.464 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.464 DEBUG:boto:Params: {} 2024-04-12T04:19:57.465 DEBUG:boto:Token: None 2024-04-12T04:19:57.465 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj185?versionId=null 2024-04-12T04:19:57.465 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:anPYPCJNsn6cY/3qp4fSK+i5Yfs= 2024-04-12T04:19:57.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:anPYPCJNsn6cY/3qp4fSK+i5Yfs=', 'Content-Length': '0'} 2024-04-12T04:19:57.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e966e2591da0c7b-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.467 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.467 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.467 DEBUG:boto:path=/dkyges-63/?prefix=obj185 2024-04-12T04:19:57.467 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj185 2024-04-12T04:19:57.468 DEBUG:boto:Method: GET 2024-04-12T04:19:57.468 DEBUG:boto:Path: /dkyges-63/?prefix=obj185 2024-04-12T04:19:57.468 DEBUG:boto:Data: 2024-04-12T04:19:57.468 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.468 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.468 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.468 DEBUG:boto:Params: {} 2024-04-12T04:19:57.468 DEBUG:boto:Token: None 2024-04-12T04:19:57.468 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.468 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d1cdf52d0a1bf28-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.471 DEBUG:boto:b'dkyges-63obj1851000falseobj1852024-04-12T04:15:49.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.471 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.472 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.472 DEBUG:boto:path=/dkyges-63/?prefix=obj185 2024-04-12T04:19:57.472 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj185 2024-04-12T04:19:57.472 DEBUG:boto:Method: GET 2024-04-12T04:19:57.472 DEBUG:boto:Path: /dkyges-63/?prefix=obj185 2024-04-12T04:19:57.472 DEBUG:boto:Data: 2024-04-12T04:19:57.472 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.472 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.472 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.472 DEBUG:boto:Params: {} 2024-04-12T04:19:57.473 DEBUG:boto:Token: None 2024-04-12T04:19:57.473 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.473 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3e88bd100c72324-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.477 DEBUG:boto:b'dkyges-63obj1851000falseobj1852024-04-12T04:15:49.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.477 DEBUG:boto:path=/dkyges-63/obj185 2024-04-12T04:19:57.478 DEBUG:boto:auth_path=/dkyges-63/obj185 2024-04-12T04:19:57.478 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.478 DEBUG:boto:Path: /dkyges-63/obj185 2024-04-12T04:19:57.478 DEBUG:boto:Data: 2024-04-12T04:19:57.478 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.478 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.478 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.478 DEBUG:boto:Params: {} 2024-04-12T04:19:57.478 DEBUG:boto:Token: None 2024-04-12T04:19:57.479 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj185 2024-04-12T04:19:57.479 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CTpIf54L/3RBS+tPdqxvecHpJpM= 2024-04-12T04:19:57.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CTpIf54L/3RBS+tPdqxvecHpJpM=', 'Content-Length': '0'} 2024-04-12T04:19:57.480 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb33f4f5b3617e52-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.481 DEBUG:boto:path=/dkyges-63/obj185 2024-04-12T04:19:57.481 DEBUG:boto:auth_path=/dkyges-63/obj185 2024-04-12T04:19:57.481 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.481 DEBUG:boto:Path: /dkyges-63/obj185 2024-04-12T04:19:57.481 DEBUG:boto:Data: 2024-04-12T04:19:57.481 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.481 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.481 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.481 DEBUG:boto:Params: {} 2024-04-12T04:19:57.482 DEBUG:boto:Token: None 2024-04-12T04:19:57.482 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj185 2024-04-12T04:19:57.482 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CTpIf54L/3RBS+tPdqxvecHpJpM= 2024-04-12T04:19:57.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CTpIf54L/3RBS+tPdqxvecHpJpM=', 'Content-Length': '0'} 2024-04-12T04:19:57.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3df4d1a6303f0ea-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2024-04-12T04:19:57.484 DEBUG:boto:path=/dkyges-63/obj185 2024-04-12T04:19:57.484 DEBUG:boto:auth_path=/dkyges-63/obj185 2024-04-12T04:19:57.484 DEBUG:boto:Method: GET 2024-04-12T04:19:57.484 DEBUG:boto:Path: /dkyges-63/obj185 2024-04-12T04:19:57.484 DEBUG:boto:Data: 2024-04-12T04:19:57.485 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.485 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.485 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.485 DEBUG:boto:Params: {} 2024-04-12T04:19:57.485 DEBUG:boto:Token: None 2024-04-12T04:19:57.485 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj185 2024-04-12T04:19:57.485 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AvFNpbc/gB2+NPLO4YEgTkMHFQg= 2024-04-12T04:19:57.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AvFNpbc/gB2+NPLO4YEgTkMHFQg=', 'Content-Length': '0'} 2024-04-12T04:19:57.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000904b79f2553ab726-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.534 DEBUG:boto:path=/dkyges-63/obj185 2024-04-12T04:19:57.535 DEBUG:boto:auth_path=/dkyges-63/obj185 2024-04-12T04:19:57.535 DEBUG:boto:Method: GET 2024-04-12T04:19:57.535 DEBUG:boto:Path: /dkyges-63/obj185 2024-04-12T04:19:57.535 DEBUG:boto:Data: 2024-04-12T04:19:57.535 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.535 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.535 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.535 DEBUG:boto:Params: {} 2024-04-12T04:19:57.535 DEBUG:boto:Token: None 2024-04-12T04:19:57.535 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj185 2024-04-12T04:19:57.536 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AvFNpbc/gB2+NPLO4YEgTkMHFQg= 2024-04-12T04:19:57.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AvFNpbc/gB2+NPLO4YEgTkMHFQg=', 'Content-Length': '0'} 2024-04-12T04:19:57.538 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b091ec63b6338bb-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.538 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.538 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.538 DEBUG:boto:path=/dkyges-63/?prefix=obj185 2024-04-12T04:19:57.538 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj185 2024-04-12T04:19:57.538 DEBUG:boto:Method: GET 2024-04-12T04:19:57.538 DEBUG:boto:Path: /dkyges-63/?prefix=obj185 2024-04-12T04:19:57.539 DEBUG:boto:Data: 2024-04-12T04:19:57.539 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.539 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.539 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.539 DEBUG:boto:Params: {} 2024-04-12T04:19:57.539 DEBUG:boto:Token: None 2024-04-12T04:19:57.539 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.539 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b32c02e650378f1a-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.542 DEBUG:boto:b'dkyges-63obj1851000falseobj1852024-04-12T04:15:49.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.542 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.542 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.542 DEBUG:boto:path=/dkyges-63/?prefix=obj185 2024-04-12T04:19:57.542 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj185 2024-04-12T04:19:57.542 DEBUG:boto:Method: GET 2024-04-12T04:19:57.543 DEBUG:boto:Path: /dkyges-63/?prefix=obj185 2024-04-12T04:19:57.543 DEBUG:boto:Data: 2024-04-12T04:19:57.543 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.543 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.543 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.543 DEBUG:boto:Params: {} 2024-04-12T04:19:57.543 DEBUG:boto:Token: None 2024-04-12T04:19:57.543 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.543 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088d9b0b4745bc997-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.548 DEBUG:boto:b'dkyges-63obj1851000falseobj1852024-04-12T04:15:49.730Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:19:57.548 DEBUG:boto:path=/dkyges-63/obj19 2024-04-12T04:19:57.548 DEBUG:boto:auth_path=/dkyges-63/obj19 2024-04-12T04:19:57.548 DEBUG:boto:path=/dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.548 DEBUG:boto:auth_path=/dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.549 DEBUG:boto:Method: GET 2024-04-12T04:19:57.549 DEBUG:boto:Path: /dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.549 DEBUG:boto:Data: 2024-04-12T04:19:57.549 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.549 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.549 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.549 DEBUG:boto:Params: {} 2024-04-12T04:19:57.549 DEBUG:boto:Token: None 2024-04-12T04:19:57.549 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.549 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YsWr5PWvID6RNY1zzoez5SC7v/o= 2024-04-12T04:19:57.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YsWr5PWvID6RNY1zzoez5SC7v/o=', 'Content-Length': '0'} 2024-04-12T04:19:57.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bbcaea311e98a75-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.551 DEBUG:boto:path=/dkyges-63/obj19 2024-04-12T04:19:57.552 DEBUG:boto:auth_path=/dkyges-63/obj19 2024-04-12T04:19:57.552 DEBUG:boto:path=/dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.552 DEBUG:boto:auth_path=/dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.552 DEBUG:boto:Method: GET 2024-04-12T04:19:57.552 DEBUG:boto:Path: /dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.552 DEBUG:boto:Data: 2024-04-12T04:19:57.552 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.552 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.552 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.552 DEBUG:boto:Params: {} 2024-04-12T04:19:57.552 DEBUG:boto:Token: None 2024-04-12T04:19:57.553 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.553 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YsWr5PWvID6RNY1zzoez5SC7v/o= 2024-04-12T04:19:57.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YsWr5PWvID6RNY1zzoez5SC7v/o=', 'Content-Length': '0'} 2024-04-12T04:19:57.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062cf27af9131f4d3-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.555 DEBUG:boto:path=/dkyges-63/obj19 2024-04-12T04:19:57.555 DEBUG:boto:auth_path=/dkyges-63/obj19 2024-04-12T04:19:57.555 DEBUG:boto:path=/dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.555 DEBUG:boto:auth_path=/dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.555 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.555 DEBUG:boto:Path: /dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.555 DEBUG:boto:Data: 2024-04-12T04:19:57.555 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.556 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.556 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.556 DEBUG:boto:Params: {} 2024-04-12T04:19:57.556 DEBUG:boto:Token: None 2024-04-12T04:19:57.556 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.556 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9TKlmQmG0Yd1FG5kyWQ+LD64j20= 2024-04-12T04:19:57.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9TKlmQmG0Yd1FG5kyWQ+LD64j20=', 'Content-Length': '0'} 2024-04-12T04:19:57.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022a789993fd7b811-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.558 DEBUG:boto:path=/dkyges-63/obj19 2024-04-12T04:19:57.558 DEBUG:boto:auth_path=/dkyges-63/obj19 2024-04-12T04:19:57.558 DEBUG:boto:path=/dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.558 DEBUG:boto:auth_path=/dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.558 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.559 DEBUG:boto:Path: /dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.559 DEBUG:boto:Data: 2024-04-12T04:19:57.559 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.559 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.559 DEBUG:boto:Params: {} 2024-04-12T04:19:57.559 DEBUG:boto:Token: None 2024-04-12T04:19:57.559 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.559 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9TKlmQmG0Yd1FG5kyWQ+LD64j20= 2024-04-12T04:19:57.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9TKlmQmG0Yd1FG5kyWQ+LD64j20=', 'Content-Length': '0'} 2024-04-12T04:19:57.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b18d10748235227c-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:19:57.562 DEBUG:boto:path=/dkyges-63/obj19 2024-04-12T04:19:57.562 DEBUG:boto:auth_path=/dkyges-63/obj19 2024-04-12T04:19:57.562 DEBUG:boto:path=/dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.562 DEBUG:boto:auth_path=/dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.562 DEBUG:boto:Method: GET 2024-04-12T04:19:57.562 DEBUG:boto:Path: /dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.562 DEBUG:boto:Data: 2024-04-12T04:19:57.562 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.562 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.562 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.563 DEBUG:boto:Params: {} 2024-04-12T04:19:57.563 DEBUG:boto:Token: None 2024-04-12T04:19:57.563 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.563 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YsWr5PWvID6RNY1zzoez5SC7v/o= 2024-04-12T04:19:57.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YsWr5PWvID6RNY1zzoez5SC7v/o=', 'Content-Length': '0'} 2024-04-12T04:19:57.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000516eabaa2bd6f743-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.565 DEBUG:boto:path=/dkyges-63/obj19 2024-04-12T04:19:57.565 DEBUG:boto:auth_path=/dkyges-63/obj19 2024-04-12T04:19:57.565 DEBUG:boto:path=/dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.565 DEBUG:boto:auth_path=/dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.566 DEBUG:boto:Method: GET 2024-04-12T04:19:57.566 DEBUG:boto:Path: /dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.566 DEBUG:boto:Data: 2024-04-12T04:19:57.566 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.566 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.566 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.566 DEBUG:boto:Params: {} 2024-04-12T04:19:57.566 DEBUG:boto:Token: None 2024-04-12T04:19:57.566 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj19?versionId=null 2024-04-12T04:19:57.566 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YsWr5PWvID6RNY1zzoez5SC7v/o= 2024-04-12T04:19:57.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YsWr5PWvID6RNY1zzoez5SC7v/o=', 'Content-Length': '0'} 2024-04-12T04:19:57.568 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fb3757cdf014c13-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.569 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.569 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.569 DEBUG:boto:path=/dkyges-63/?prefix=obj19 2024-04-12T04:19:57.569 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj19 2024-04-12T04:19:57.569 DEBUG:boto:Method: GET 2024-04-12T04:19:57.569 DEBUG:boto:Path: /dkyges-63/?prefix=obj19 2024-04-12T04:19:57.569 DEBUG:boto:Data: 2024-04-12T04:19:57.569 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.569 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.569 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.570 DEBUG:boto:Params: {} 2024-04-12T04:19:57.570 DEBUG:boto:Token: None 2024-04-12T04:19:57.570 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.570 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae100aa8861a0df0-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.572 DEBUG:boto:b'dkyges-63obj191000falseobj192024-04-12T04:15:48.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.573 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.573 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.573 DEBUG:boto:path=/dkyges-63/?prefix=obj19 2024-04-12T04:19:57.573 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj19 2024-04-12T04:19:57.573 DEBUG:boto:Method: GET 2024-04-12T04:19:57.573 DEBUG:boto:Path: /dkyges-63/?prefix=obj19 2024-04-12T04:19:57.573 DEBUG:boto:Data: 2024-04-12T04:19:57.573 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.574 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.574 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.574 DEBUG:boto:Params: {} 2024-04-12T04:19:57.574 DEBUG:boto:Token: None 2024-04-12T04:19:57.574 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.574 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc1a22f8bf76551c-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.579 DEBUG:boto:b'dkyges-63obj191000falseobj192024-04-12T04:15:48.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.579 DEBUG:boto:path=/dkyges-63/obj19 2024-04-12T04:19:57.579 DEBUG:boto:auth_path=/dkyges-63/obj19 2024-04-12T04:19:57.579 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.579 DEBUG:boto:Path: /dkyges-63/obj19 2024-04-12T04:19:57.579 DEBUG:boto:Data: 2024-04-12T04:19:57.580 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.580 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.580 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.580 DEBUG:boto:Params: {} 2024-04-12T04:19:57.580 DEBUG:boto:Token: None 2024-04-12T04:19:57.580 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj19 2024-04-12T04:19:57.580 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6of8wRbOu4ekkA3jLRFLnpJrJw4= 2024-04-12T04:19:57.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6of8wRbOu4ekkA3jLRFLnpJrJw4=', 'Content-Length': '0'} 2024-04-12T04:19:57.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fa98e4d4ad62336-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.582 DEBUG:boto:path=/dkyges-63/obj19 2024-04-12T04:19:57.582 DEBUG:boto:auth_path=/dkyges-63/obj19 2024-04-12T04:19:57.582 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.582 DEBUG:boto:Path: /dkyges-63/obj19 2024-04-12T04:19:57.582 DEBUG:boto:Data: 2024-04-12T04:19:57.583 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.583 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.583 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.583 DEBUG:boto:Params: {} 2024-04-12T04:19:57.583 DEBUG:boto:Token: None 2024-04-12T04:19:57.583 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj19 2024-04-12T04:19:57.583 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6of8wRbOu4ekkA3jLRFLnpJrJw4= 2024-04-12T04:19:57.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6of8wRbOu4ekkA3jLRFLnpJrJw4=', 'Content-Length': '0'} 2024-04-12T04:19:57.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fcd559c242aeb8b-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2024-04-12T04:19:57.585 DEBUG:boto:path=/dkyges-63/obj19 2024-04-12T04:19:57.586 DEBUG:boto:auth_path=/dkyges-63/obj19 2024-04-12T04:19:57.586 DEBUG:boto:Method: GET 2024-04-12T04:19:57.586 DEBUG:boto:Path: /dkyges-63/obj19 2024-04-12T04:19:57.586 DEBUG:boto:Data: 2024-04-12T04:19:57.586 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.586 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.586 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.586 DEBUG:boto:Params: {} 2024-04-12T04:19:57.586 DEBUG:boto:Token: None 2024-04-12T04:19:57.586 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj19 2024-04-12T04:19:57.587 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JXulYOdi8U6WwzkjkXq16utYRA4= 2024-04-12T04:19:57.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JXulYOdi8U6WwzkjkXq16utYRA4=', 'Content-Length': '0'} 2024-04-12T04:19:57.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037c48d57b3259ff0-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.589 DEBUG:boto:path=/dkyges-63/obj19 2024-04-12T04:19:57.589 DEBUG:boto:auth_path=/dkyges-63/obj19 2024-04-12T04:19:57.589 DEBUG:boto:Method: GET 2024-04-12T04:19:57.589 DEBUG:boto:Path: /dkyges-63/obj19 2024-04-12T04:19:57.589 DEBUG:boto:Data: 2024-04-12T04:19:57.589 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.589 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.589 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.589 DEBUG:boto:Params: {} 2024-04-12T04:19:57.589 DEBUG:boto:Token: None 2024-04-12T04:19:57.590 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj19 2024-04-12T04:19:57.590 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JXulYOdi8U6WwzkjkXq16utYRA4= 2024-04-12T04:19:57.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JXulYOdi8U6WwzkjkXq16utYRA4=', 'Content-Length': '0'} 2024-04-12T04:19:57.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ef2d9b8b79cfbdf-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.634 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.634 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.635 DEBUG:boto:path=/dkyges-63/?prefix=obj19 2024-04-12T04:19:57.635 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj19 2024-04-12T04:19:57.635 DEBUG:boto:Method: GET 2024-04-12T04:19:57.635 DEBUG:boto:Path: /dkyges-63/?prefix=obj19 2024-04-12T04:19:57.635 DEBUG:boto:Data: 2024-04-12T04:19:57.635 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.635 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.635 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.635 DEBUG:boto:Params: {} 2024-04-12T04:19:57.635 DEBUG:boto:Token: None 2024-04-12T04:19:57.636 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.636 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.639 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aea3f761e2ff68a7-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.639 DEBUG:boto:b'dkyges-63obj191000falseobj192024-04-12T04:15:48.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.639 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.639 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.640 DEBUG:boto:path=/dkyges-63/?prefix=obj19 2024-04-12T04:19:57.640 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj19 2024-04-12T04:19:57.640 DEBUG:boto:Method: GET 2024-04-12T04:19:57.640 DEBUG:boto:Path: /dkyges-63/?prefix=obj19 2024-04-12T04:19:57.640 DEBUG:boto:Data: 2024-04-12T04:19:57.640 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.640 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.640 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.640 DEBUG:boto:Params: {} 2024-04-12T04:19:57.640 DEBUG:boto:Token: None 2024-04-12T04:19:57.640 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.641 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.645 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000482adf16cc8e509f-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.645 DEBUG:boto:b'dkyges-63obj191000falseobj192024-04-12T04:15:48.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:19:57.646 DEBUG:boto:path=/dkyges-63/obj22 2024-04-12T04:19:57.646 DEBUG:boto:auth_path=/dkyges-63/obj22 2024-04-12T04:19:57.646 DEBUG:boto:path=/dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.646 DEBUG:boto:auth_path=/dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.646 DEBUG:boto:Method: GET 2024-04-12T04:19:57.646 DEBUG:boto:Path: /dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.646 DEBUG:boto:Data: 2024-04-12T04:19:57.646 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.646 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.646 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.646 DEBUG:boto:Params: {} 2024-04-12T04:19:57.647 DEBUG:boto:Token: None 2024-04-12T04:19:57.647 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.647 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zARRXmf/W52rVWqIiOyDQApLAJg= 2024-04-12T04:19:57.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zARRXmf/W52rVWqIiOyDQApLAJg=', 'Content-Length': '0'} 2024-04-12T04:19:57.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b93d98fb9119b665-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.694 DEBUG:boto:path=/dkyges-63/obj22 2024-04-12T04:19:57.695 DEBUG:boto:auth_path=/dkyges-63/obj22 2024-04-12T04:19:57.695 DEBUG:boto:path=/dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.695 DEBUG:boto:auth_path=/dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.695 DEBUG:boto:Method: GET 2024-04-12T04:19:57.695 DEBUG:boto:Path: /dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.695 DEBUG:boto:Data: 2024-04-12T04:19:57.695 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.695 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.695 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.695 DEBUG:boto:Params: {} 2024-04-12T04:19:57.695 DEBUG:boto:Token: None 2024-04-12T04:19:57.696 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.696 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zARRXmf/W52rVWqIiOyDQApLAJg= 2024-04-12T04:19:57.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zARRXmf/W52rVWqIiOyDQApLAJg=', 'Content-Length': '0'} 2024-04-12T04:19:57.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ee914c195005bf8-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.698 DEBUG:boto:path=/dkyges-63/obj22 2024-04-12T04:19:57.698 DEBUG:boto:auth_path=/dkyges-63/obj22 2024-04-12T04:19:57.698 DEBUG:boto:path=/dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.698 DEBUG:boto:auth_path=/dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.698 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.698 DEBUG:boto:Path: /dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.698 DEBUG:boto:Data: 2024-04-12T04:19:57.698 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.699 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.699 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.699 DEBUG:boto:Params: {} 2024-04-12T04:19:57.699 DEBUG:boto:Token: None 2024-04-12T04:19:57.699 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.699 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qShhqjAJpnEdIxgoyVM+KGGD5dE= 2024-04-12T04:19:57.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qShhqjAJpnEdIxgoyVM+KGGD5dE=', 'Content-Length': '0'} 2024-04-12T04:19:57.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a66c7ab59e42f75-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.701 DEBUG:boto:path=/dkyges-63/obj22 2024-04-12T04:19:57.701 DEBUG:boto:auth_path=/dkyges-63/obj22 2024-04-12T04:19:57.701 DEBUG:boto:path=/dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.701 DEBUG:boto:auth_path=/dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.701 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.701 DEBUG:boto:Path: /dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.701 DEBUG:boto:Data: 2024-04-12T04:19:57.702 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.702 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.702 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.702 DEBUG:boto:Params: {} 2024-04-12T04:19:57.702 DEBUG:boto:Token: None 2024-04-12T04:19:57.702 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.702 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qShhqjAJpnEdIxgoyVM+KGGD5dE= 2024-04-12T04:19:57.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qShhqjAJpnEdIxgoyVM+KGGD5dE=', 'Content-Length': '0'} 2024-04-12T04:19:57.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbba8faafb757a4d-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:19:57.704 DEBUG:boto:path=/dkyges-63/obj22 2024-04-12T04:19:57.705 DEBUG:boto:auth_path=/dkyges-63/obj22 2024-04-12T04:19:57.705 DEBUG:boto:path=/dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.705 DEBUG:boto:auth_path=/dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.705 DEBUG:boto:Method: GET 2024-04-12T04:19:57.705 DEBUG:boto:Path: /dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.705 DEBUG:boto:Data: 2024-04-12T04:19:57.705 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.705 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.705 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.705 DEBUG:boto:Params: {} 2024-04-12T04:19:57.705 DEBUG:boto:Token: None 2024-04-12T04:19:57.706 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.706 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zARRXmf/W52rVWqIiOyDQApLAJg= 2024-04-12T04:19:57.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zARRXmf/W52rVWqIiOyDQApLAJg=', 'Content-Length': '0'} 2024-04-12T04:19:57.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002de7839dbd465668-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.708 DEBUG:boto:path=/dkyges-63/obj22 2024-04-12T04:19:57.708 DEBUG:boto:auth_path=/dkyges-63/obj22 2024-04-12T04:19:57.708 DEBUG:boto:path=/dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.708 DEBUG:boto:auth_path=/dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.708 DEBUG:boto:Method: GET 2024-04-12T04:19:57.708 DEBUG:boto:Path: /dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.708 DEBUG:boto:Data: 2024-04-12T04:19:57.708 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.708 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.709 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.709 DEBUG:boto:Params: {} 2024-04-12T04:19:57.709 DEBUG:boto:Token: None 2024-04-12T04:19:57.709 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj22?versionId=null 2024-04-12T04:19:57.709 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zARRXmf/W52rVWqIiOyDQApLAJg= 2024-04-12T04:19:57.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zARRXmf/W52rVWqIiOyDQApLAJg=', 'Content-Length': '0'} 2024-04-12T04:19:57.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f631a6b092aa4f2f-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.711 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.712 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.712 DEBUG:boto:path=/dkyges-63/?prefix=obj22 2024-04-12T04:19:57.712 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj22 2024-04-12T04:19:57.712 DEBUG:boto:Method: GET 2024-04-12T04:19:57.712 DEBUG:boto:Path: /dkyges-63/?prefix=obj22 2024-04-12T04:19:57.712 DEBUG:boto:Data: 2024-04-12T04:19:57.712 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.712 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.712 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.712 DEBUG:boto:Params: {} 2024-04-12T04:19:57.712 DEBUG:boto:Token: None 2024-04-12T04:19:57.713 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.713 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.715 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b4bdcb795e20df7-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.715 DEBUG:boto:b'dkyges-63obj221000falseobj222024-04-12T04:15:47.472Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.716 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.716 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.716 DEBUG:boto:path=/dkyges-63/?prefix=obj22 2024-04-12T04:19:57.716 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj22 2024-04-12T04:19:57.716 DEBUG:boto:Method: GET 2024-04-12T04:19:57.716 DEBUG:boto:Path: /dkyges-63/?prefix=obj22 2024-04-12T04:19:57.716 DEBUG:boto:Data: 2024-04-12T04:19:57.716 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.716 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.716 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.716 DEBUG:boto:Params: {} 2024-04-12T04:19:57.717 DEBUG:boto:Token: None 2024-04-12T04:19:57.717 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.717 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.721 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f546441bdb322dd-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.722 DEBUG:boto:b'dkyges-63obj221000falseobj222024-04-12T04:15:47.472Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.722 DEBUG:boto:path=/dkyges-63/obj22 2024-04-12T04:19:57.722 DEBUG:boto:auth_path=/dkyges-63/obj22 2024-04-12T04:19:57.722 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.722 DEBUG:boto:Path: /dkyges-63/obj22 2024-04-12T04:19:57.722 DEBUG:boto:Data: 2024-04-12T04:19:57.722 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.722 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.723 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.723 DEBUG:boto:Params: {} 2024-04-12T04:19:57.723 DEBUG:boto:Token: None 2024-04-12T04:19:57.723 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj22 2024-04-12T04:19:57.723 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UBk2Q7Dm5448ZHqWD4dpqZci8mo= 2024-04-12T04:19:57.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UBk2Q7Dm5448ZHqWD4dpqZci8mo=', 'Content-Length': '0'} 2024-04-12T04:19:57.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001188f29ca5bc8a5f-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.725 DEBUG:boto:path=/dkyges-63/obj22 2024-04-12T04:19:57.725 DEBUG:boto:auth_path=/dkyges-63/obj22 2024-04-12T04:19:57.725 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.725 DEBUG:boto:Path: /dkyges-63/obj22 2024-04-12T04:19:57.725 DEBUG:boto:Data: 2024-04-12T04:19:57.725 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.725 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.725 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.726 DEBUG:boto:Params: {} 2024-04-12T04:19:57.726 DEBUG:boto:Token: None 2024-04-12T04:19:57.726 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj22 2024-04-12T04:19:57.726 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UBk2Q7Dm5448ZHqWD4dpqZci8mo= 2024-04-12T04:19:57.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UBk2Q7Dm5448ZHqWD4dpqZci8mo=', 'Content-Length': '0'} 2024-04-12T04:19:57.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062c0ed094511cd23-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:19:57.728 DEBUG:boto:path=/dkyges-63/obj22 2024-04-12T04:19:57.729 DEBUG:boto:auth_path=/dkyges-63/obj22 2024-04-12T04:19:57.729 DEBUG:boto:Method: GET 2024-04-12T04:19:57.729 DEBUG:boto:Path: /dkyges-63/obj22 2024-04-12T04:19:57.729 DEBUG:boto:Data: 2024-04-12T04:19:57.729 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.729 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.729 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.729 DEBUG:boto:Params: {} 2024-04-12T04:19:57.729 DEBUG:boto:Token: None 2024-04-12T04:19:57.729 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj22 2024-04-12T04:19:57.729 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Vsr04rhL68YKdwQ9M2TWCEJ/ckI= 2024-04-12T04:19:57.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Vsr04rhL68YKdwQ9M2TWCEJ/ckI=', 'Content-Length': '0'} 2024-04-12T04:19:57.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000248f72c37931dfcf-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.731 DEBUG:boto:path=/dkyges-63/obj22 2024-04-12T04:19:57.732 DEBUG:boto:auth_path=/dkyges-63/obj22 2024-04-12T04:19:57.732 DEBUG:boto:Method: GET 2024-04-12T04:19:57.732 DEBUG:boto:Path: /dkyges-63/obj22 2024-04-12T04:19:57.732 DEBUG:boto:Data: 2024-04-12T04:19:57.732 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.732 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.732 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.732 DEBUG:boto:Params: {} 2024-04-12T04:19:57.732 DEBUG:boto:Token: None 2024-04-12T04:19:57.732 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj22 2024-04-12T04:19:57.733 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Vsr04rhL68YKdwQ9M2TWCEJ/ckI= 2024-04-12T04:19:57.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Vsr04rhL68YKdwQ9M2TWCEJ/ckI=', 'Content-Length': '0'} 2024-04-12T04:19:57.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0add5474666a3b5-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.778 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.778 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.778 DEBUG:boto:path=/dkyges-63/?prefix=obj22 2024-04-12T04:19:57.779 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj22 2024-04-12T04:19:57.779 DEBUG:boto:Method: GET 2024-04-12T04:19:57.779 DEBUG:boto:Path: /dkyges-63/?prefix=obj22 2024-04-12T04:19:57.779 DEBUG:boto:Data: 2024-04-12T04:19:57.779 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.779 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.779 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.779 DEBUG:boto:Params: {} 2024-04-12T04:19:57.779 DEBUG:boto:Token: None 2024-04-12T04:19:57.779 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.779 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039974c7911dec458-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.782 DEBUG:boto:b'dkyges-63obj221000falseobj222024-04-12T04:15:47.472Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.782 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.783 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.783 DEBUG:boto:path=/dkyges-63/?prefix=obj22 2024-04-12T04:19:57.783 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj22 2024-04-12T04:19:57.783 DEBUG:boto:Method: GET 2024-04-12T04:19:57.783 DEBUG:boto:Path: /dkyges-63/?prefix=obj22 2024-04-12T04:19:57.783 DEBUG:boto:Data: 2024-04-12T04:19:57.783 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.783 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.783 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.783 DEBUG:boto:Params: {} 2024-04-12T04:19:57.783 DEBUG:boto:Token: None 2024-04-12T04:19:57.784 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.784 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002efff4919023da10-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.788 DEBUG:boto:b'dkyges-63obj221000falseobj222024-04-12T04:15:47.472Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:19:57.789 DEBUG:boto:path=/dkyges-63/obj23 2024-04-12T04:19:57.789 DEBUG:boto:auth_path=/dkyges-63/obj23 2024-04-12T04:19:57.789 DEBUG:boto:path=/dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.789 DEBUG:boto:auth_path=/dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.789 DEBUG:boto:Method: GET 2024-04-12T04:19:57.789 DEBUG:boto:Path: /dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.789 DEBUG:boto:Data: 2024-04-12T04:19:57.789 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.789 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.789 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.790 DEBUG:boto:Params: {} 2024-04-12T04:19:57.790 DEBUG:boto:Token: None 2024-04-12T04:19:57.790 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.790 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cKB6fLRsr9fmuN7utCw60o3vG5k= 2024-04-12T04:19:57.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cKB6fLRsr9fmuN7utCw60o3vG5k=', 'Content-Length': '0'} 2024-04-12T04:19:57.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3ea04f66232aa8d-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.834 DEBUG:boto:path=/dkyges-63/obj23 2024-04-12T04:19:57.834 DEBUG:boto:auth_path=/dkyges-63/obj23 2024-04-12T04:19:57.835 DEBUG:boto:path=/dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.835 DEBUG:boto:auth_path=/dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.835 DEBUG:boto:Method: GET 2024-04-12T04:19:57.835 DEBUG:boto:Path: /dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.835 DEBUG:boto:Data: 2024-04-12T04:19:57.835 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.835 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.835 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.835 DEBUG:boto:Params: {} 2024-04-12T04:19:57.835 DEBUG:boto:Token: None 2024-04-12T04:19:57.836 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.836 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cKB6fLRsr9fmuN7utCw60o3vG5k= 2024-04-12T04:19:57.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cKB6fLRsr9fmuN7utCw60o3vG5k=', 'Content-Length': '0'} 2024-04-12T04:19:57.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3c6ede2433f7729-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.838 DEBUG:boto:path=/dkyges-63/obj23 2024-04-12T04:19:57.838 DEBUG:boto:auth_path=/dkyges-63/obj23 2024-04-12T04:19:57.838 DEBUG:boto:path=/dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.838 DEBUG:boto:auth_path=/dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.839 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.839 DEBUG:boto:Path: /dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.839 DEBUG:boto:Data: 2024-04-12T04:19:57.839 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.839 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.839 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.839 DEBUG:boto:Params: {} 2024-04-12T04:19:57.839 DEBUG:boto:Token: None 2024-04-12T04:19:57.839 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.839 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W+UEZnwoA6P5ad2a8ESBuAxq3ks= 2024-04-12T04:19:57.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W+UEZnwoA6P5ad2a8ESBuAxq3ks=', 'Content-Length': '0'} 2024-04-12T04:19:57.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccdf123f5ed5277d-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.842 DEBUG:boto:path=/dkyges-63/obj23 2024-04-12T04:19:57.842 DEBUG:boto:auth_path=/dkyges-63/obj23 2024-04-12T04:19:57.842 DEBUG:boto:path=/dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.842 DEBUG:boto:auth_path=/dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.843 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.843 DEBUG:boto:Path: /dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.843 DEBUG:boto:Data: 2024-04-12T04:19:57.843 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.843 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.843 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.843 DEBUG:boto:Params: {} 2024-04-12T04:19:57.843 DEBUG:boto:Token: None 2024-04-12T04:19:57.843 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.843 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W+UEZnwoA6P5ad2a8ESBuAxq3ks= 2024-04-12T04:19:57.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W+UEZnwoA6P5ad2a8ESBuAxq3ks=', 'Content-Length': '0'} 2024-04-12T04:19:57.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff4c3f3650670142-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:19:57.846 DEBUG:boto:path=/dkyges-63/obj23 2024-04-12T04:19:57.846 DEBUG:boto:auth_path=/dkyges-63/obj23 2024-04-12T04:19:57.846 DEBUG:boto:path=/dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.846 DEBUG:boto:auth_path=/dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.846 DEBUG:boto:Method: GET 2024-04-12T04:19:57.846 DEBUG:boto:Path: /dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.846 DEBUG:boto:Data: 2024-04-12T04:19:57.846 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.846 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.846 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.846 DEBUG:boto:Params: {} 2024-04-12T04:19:57.847 DEBUG:boto:Token: None 2024-04-12T04:19:57.847 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.847 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cKB6fLRsr9fmuN7utCw60o3vG5k= 2024-04-12T04:19:57.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cKB6fLRsr9fmuN7utCw60o3vG5k=', 'Content-Length': '0'} 2024-04-12T04:19:57.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bde64ec9bef3437-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.849 DEBUG:boto:path=/dkyges-63/obj23 2024-04-12T04:19:57.849 DEBUG:boto:auth_path=/dkyges-63/obj23 2024-04-12T04:19:57.849 DEBUG:boto:path=/dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.849 DEBUG:boto:auth_path=/dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.849 DEBUG:boto:Method: GET 2024-04-12T04:19:57.850 DEBUG:boto:Path: /dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.850 DEBUG:boto:Data: 2024-04-12T04:19:57.850 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.850 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.850 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.850 DEBUG:boto:Params: {} 2024-04-12T04:19:57.850 DEBUG:boto:Token: None 2024-04-12T04:19:57.850 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj23?versionId=null 2024-04-12T04:19:57.850 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cKB6fLRsr9fmuN7utCw60o3vG5k= 2024-04-12T04:19:57.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cKB6fLRsr9fmuN7utCw60o3vG5k=', 'Content-Length': '0'} 2024-04-12T04:19:57.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d504685254abca56-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.853 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.853 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.853 DEBUG:boto:path=/dkyges-63/?prefix=obj23 2024-04-12T04:19:57.853 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj23 2024-04-12T04:19:57.853 DEBUG:boto:Method: GET 2024-04-12T04:19:57.853 DEBUG:boto:Path: /dkyges-63/?prefix=obj23 2024-04-12T04:19:57.853 DEBUG:boto:Data: 2024-04-12T04:19:57.853 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.853 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.853 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.853 DEBUG:boto:Params: {} 2024-04-12T04:19:57.854 DEBUG:boto:Token: None 2024-04-12T04:19:57.854 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.854 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000773b3f45c882dec7-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.857 DEBUG:boto:b'dkyges-63obj231000falseobj232024-04-12T04:15:48.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.857 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.857 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.857 DEBUG:boto:path=/dkyges-63/?prefix=obj23 2024-04-12T04:19:57.857 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj23 2024-04-12T04:19:57.858 DEBUG:boto:Method: GET 2024-04-12T04:19:57.858 DEBUG:boto:Path: /dkyges-63/?prefix=obj23 2024-04-12T04:19:57.858 DEBUG:boto:Data: 2024-04-12T04:19:57.858 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.858 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.858 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.858 DEBUG:boto:Params: {} 2024-04-12T04:19:57.858 DEBUG:boto:Token: None 2024-04-12T04:19:57.858 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.858 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000681b71842307d42-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.863 DEBUG:boto:b'dkyges-63obj231000falseobj232024-04-12T04:15:48.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.863 DEBUG:boto:path=/dkyges-63/obj23 2024-04-12T04:19:57.863 DEBUG:boto:auth_path=/dkyges-63/obj23 2024-04-12T04:19:57.863 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.863 DEBUG:boto:Path: /dkyges-63/obj23 2024-04-12T04:19:57.863 DEBUG:boto:Data: 2024-04-12T04:19:57.864 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.864 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.864 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.864 DEBUG:boto:Params: {} 2024-04-12T04:19:57.864 DEBUG:boto:Token: None 2024-04-12T04:19:57.864 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj23 2024-04-12T04:19:57.864 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oVSkOfoToh/jk3FA7tblYSz82mc= 2024-04-12T04:19:57.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oVSkOfoToh/jk3FA7tblYSz82mc=', 'Content-Length': '0'} 2024-04-12T04:19:57.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006718865c59d99c95-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.866 DEBUG:boto:path=/dkyges-63/obj23 2024-04-12T04:19:57.866 DEBUG:boto:auth_path=/dkyges-63/obj23 2024-04-12T04:19:57.867 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.867 DEBUG:boto:Path: /dkyges-63/obj23 2024-04-12T04:19:57.867 DEBUG:boto:Data: 2024-04-12T04:19:57.867 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.867 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.867 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.867 DEBUG:boto:Params: {} 2024-04-12T04:19:57.867 DEBUG:boto:Token: None 2024-04-12T04:19:57.867 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj23 2024-04-12T04:19:57.867 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oVSkOfoToh/jk3FA7tblYSz82mc= 2024-04-12T04:19:57.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oVSkOfoToh/jk3FA7tblYSz82mc=', 'Content-Length': '0'} 2024-04-12T04:19:57.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c2dab7b83f422c0-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2024-04-12T04:19:57.869 DEBUG:boto:path=/dkyges-63/obj23 2024-04-12T04:19:57.870 DEBUG:boto:auth_path=/dkyges-63/obj23 2024-04-12T04:19:57.870 DEBUG:boto:Method: GET 2024-04-12T04:19:57.870 DEBUG:boto:Path: /dkyges-63/obj23 2024-04-12T04:19:57.870 DEBUG:boto:Data: 2024-04-12T04:19:57.870 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.870 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.870 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.870 DEBUG:boto:Params: {} 2024-04-12T04:19:57.870 DEBUG:boto:Token: None 2024-04-12T04:19:57.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj23 2024-04-12T04:19:57.871 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ph/BPJY4TmG52uBmLdIw1OXRPPI= 2024-04-12T04:19:57.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ph/BPJY4TmG52uBmLdIw1OXRPPI=', 'Content-Length': '0'} 2024-04-12T04:19:57.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015a50599b391c45e-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.873 DEBUG:boto:path=/dkyges-63/obj23 2024-04-12T04:19:57.873 DEBUG:boto:auth_path=/dkyges-63/obj23 2024-04-12T04:19:57.873 DEBUG:boto:Method: GET 2024-04-12T04:19:57.873 DEBUG:boto:Path: /dkyges-63/obj23 2024-04-12T04:19:57.873 DEBUG:boto:Data: 2024-04-12T04:19:57.873 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.873 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.873 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.873 DEBUG:boto:Params: {} 2024-04-12T04:19:57.873 DEBUG:boto:Token: None 2024-04-12T04:19:57.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj23 2024-04-12T04:19:57.874 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ph/BPJY4TmG52uBmLdIw1OXRPPI= 2024-04-12T04:19:57.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ph/BPJY4TmG52uBmLdIw1OXRPPI=', 'Content-Length': '0'} 2024-04-12T04:19:57.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005af4ee73a0d3ad6d-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.876 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.876 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.876 DEBUG:boto:path=/dkyges-63/?prefix=obj23 2024-04-12T04:19:57.876 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj23 2024-04-12T04:19:57.876 DEBUG:boto:Method: GET 2024-04-12T04:19:57.876 DEBUG:boto:Path: /dkyges-63/?prefix=obj23 2024-04-12T04:19:57.876 DEBUG:boto:Data: 2024-04-12T04:19:57.877 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.877 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.877 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.877 DEBUG:boto:Params: {} 2024-04-12T04:19:57.877 DEBUG:boto:Token: None 2024-04-12T04:19:57.877 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.877 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ec07cffe39c89c9-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.880 DEBUG:boto:b'dkyges-63obj231000falseobj232024-04-12T04:15:48.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.880 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.880 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.880 DEBUG:boto:path=/dkyges-63/?prefix=obj23 2024-04-12T04:19:57.881 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj23 2024-04-12T04:19:57.881 DEBUG:boto:Method: GET 2024-04-12T04:19:57.881 DEBUG:boto:Path: /dkyges-63/?prefix=obj23 2024-04-12T04:19:57.881 DEBUG:boto:Data: 2024-04-12T04:19:57.881 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.881 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.881 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.881 DEBUG:boto:Params: {} 2024-04-12T04:19:57.881 DEBUG:boto:Token: None 2024-04-12T04:19:57.881 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.881 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f573e4db41b54f0-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.886 DEBUG:boto:b'dkyges-63obj231000falseobj232024-04-12T04:15:48.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:19:57.886 DEBUG:boto:path=/dkyges-63/obj29 2024-04-12T04:19:57.886 DEBUG:boto:auth_path=/dkyges-63/obj29 2024-04-12T04:19:57.886 DEBUG:boto:path=/dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.886 DEBUG:boto:auth_path=/dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.887 DEBUG:boto:Method: GET 2024-04-12T04:19:57.887 DEBUG:boto:Path: /dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.887 DEBUG:boto:Data: 2024-04-12T04:19:57.887 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.887 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.887 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.887 DEBUG:boto:Params: {} 2024-04-12T04:19:57.887 DEBUG:boto:Token: None 2024-04-12T04:19:57.887 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.887 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qSWNdLICmIuFVQZIKER3wUlkyj4= 2024-04-12T04:19:57.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qSWNdLICmIuFVQZIKER3wUlkyj4=', 'Content-Length': '0'} 2024-04-12T04:19:57.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034f864c1eb5d1fb5-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.890 DEBUG:boto:path=/dkyges-63/obj29 2024-04-12T04:19:57.890 DEBUG:boto:auth_path=/dkyges-63/obj29 2024-04-12T04:19:57.890 DEBUG:boto:path=/dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.890 DEBUG:boto:auth_path=/dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.890 DEBUG:boto:Method: GET 2024-04-12T04:19:57.890 DEBUG:boto:Path: /dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.890 DEBUG:boto:Data: 2024-04-12T04:19:57.890 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.890 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.891 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.891 DEBUG:boto:Params: {} 2024-04-12T04:19:57.891 DEBUG:boto:Token: None 2024-04-12T04:19:57.891 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.891 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qSWNdLICmIuFVQZIKER3wUlkyj4= 2024-04-12T04:19:57.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qSWNdLICmIuFVQZIKER3wUlkyj4=', 'Content-Length': '0'} 2024-04-12T04:19:57.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002972096155c406d8-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.934 DEBUG:boto:path=/dkyges-63/obj29 2024-04-12T04:19:57.935 DEBUG:boto:auth_path=/dkyges-63/obj29 2024-04-12T04:19:57.935 DEBUG:boto:path=/dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.935 DEBUG:boto:auth_path=/dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.935 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.935 DEBUG:boto:Path: /dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.935 DEBUG:boto:Data: 2024-04-12T04:19:57.935 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.935 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.935 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.935 DEBUG:boto:Params: {} 2024-04-12T04:19:57.936 DEBUG:boto:Token: None 2024-04-12T04:19:57.936 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.936 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DYWboOcImBng2NK+Vo1mWHBoU3E= 2024-04-12T04:19:57.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DYWboOcImBng2NK+Vo1mWHBoU3E=', 'Content-Length': '0'} 2024-04-12T04:19:57.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ffb133db96cac60-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.938 DEBUG:boto:path=/dkyges-63/obj29 2024-04-12T04:19:57.938 DEBUG:boto:auth_path=/dkyges-63/obj29 2024-04-12T04:19:57.938 DEBUG:boto:path=/dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.939 DEBUG:boto:auth_path=/dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.939 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.939 DEBUG:boto:Path: /dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.939 DEBUG:boto:Data: 2024-04-12T04:19:57.939 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.939 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.939 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.939 DEBUG:boto:Params: {} 2024-04-12T04:19:57.939 DEBUG:boto:Token: None 2024-04-12T04:19:57.939 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.939 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DYWboOcImBng2NK+Vo1mWHBoU3E= 2024-04-12T04:19:57.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DYWboOcImBng2NK+Vo1mWHBoU3E=', 'Content-Length': '0'} 2024-04-12T04:19:57.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b729d14d583e758-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:19:57.942 DEBUG:boto:path=/dkyges-63/obj29 2024-04-12T04:19:57.942 DEBUG:boto:auth_path=/dkyges-63/obj29 2024-04-12T04:19:57.942 DEBUG:boto:path=/dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.942 DEBUG:boto:auth_path=/dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.942 DEBUG:boto:Method: GET 2024-04-12T04:19:57.943 DEBUG:boto:Path: /dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.943 DEBUG:boto:Data: 2024-04-12T04:19:57.943 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.943 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.943 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.943 DEBUG:boto:Params: {} 2024-04-12T04:19:57.943 DEBUG:boto:Token: None 2024-04-12T04:19:57.943 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.943 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qSWNdLICmIuFVQZIKER3wUlkyj4= 2024-04-12T04:19:57.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qSWNdLICmIuFVQZIKER3wUlkyj4=', 'Content-Length': '0'} 2024-04-12T04:19:57.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4934a207185cb7d-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.945 DEBUG:boto:path=/dkyges-63/obj29 2024-04-12T04:19:57.946 DEBUG:boto:auth_path=/dkyges-63/obj29 2024-04-12T04:19:57.946 DEBUG:boto:path=/dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.946 DEBUG:boto:auth_path=/dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.946 DEBUG:boto:Method: GET 2024-04-12T04:19:57.946 DEBUG:boto:Path: /dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.946 DEBUG:boto:Data: 2024-04-12T04:19:57.946 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.946 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.946 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.946 DEBUG:boto:Params: {} 2024-04-12T04:19:57.947 DEBUG:boto:Token: None 2024-04-12T04:19:57.947 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj29?versionId=null 2024-04-12T04:19:57.947 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qSWNdLICmIuFVQZIKER3wUlkyj4= 2024-04-12T04:19:57.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qSWNdLICmIuFVQZIKER3wUlkyj4=', 'Content-Length': '0'} 2024-04-12T04:19:57.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006adfaec95b1b589d-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.949 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.949 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.949 DEBUG:boto:path=/dkyges-63/?prefix=obj29 2024-04-12T04:19:57.949 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj29 2024-04-12T04:19:57.949 DEBUG:boto:Method: GET 2024-04-12T04:19:57.950 DEBUG:boto:Path: /dkyges-63/?prefix=obj29 2024-04-12T04:19:57.950 DEBUG:boto:Data: 2024-04-12T04:19:57.950 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.950 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.950 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.950 DEBUG:boto:Params: {} 2024-04-12T04:19:57.950 DEBUG:boto:Token: None 2024-04-12T04:19:57.950 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.950 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.953 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e9974845efbdcd6-006618b66d-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.953 DEBUG:boto:b'dkyges-63obj291000falseobj292024-04-12T04:15:48.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.953 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:57.953 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:57.954 DEBUG:boto:path=/dkyges-63/?prefix=obj29 2024-04-12T04:19:57.954 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj29 2024-04-12T04:19:57.954 DEBUG:boto:Method: GET 2024-04-12T04:19:57.954 DEBUG:boto:Path: /dkyges-63/?prefix=obj29 2024-04-12T04:19:57.954 DEBUG:boto:Data: 2024-04-12T04:19:57.954 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.954 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.954 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.954 DEBUG:boto:Params: {} 2024-04-12T04:19:57.954 DEBUG:boto:Token: None 2024-04-12T04:19:57.955 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/ 2024-04-12T04:19:57.955 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8= 2024-04-12T04:19:57.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rTWeMNJkKgUj/G3Ujqz3eI/z+i8=', 'Content-Length': '0'} 2024-04-12T04:19:57.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e09333a6bdfeda1e-006618b66d-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.959 DEBUG:boto:b'dkyges-63obj291000falseobj292024-04-12T04:15:48.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:57.960 DEBUG:boto:path=/dkyges-63/obj29 2024-04-12T04:19:57.960 DEBUG:boto:auth_path=/dkyges-63/obj29 2024-04-12T04:19:57.960 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.960 DEBUG:boto:Path: /dkyges-63/obj29 2024-04-12T04:19:57.960 DEBUG:boto:Data: 2024-04-12T04:19:57.960 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.960 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.960 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.960 DEBUG:boto:Params: {} 2024-04-12T04:19:57.960 DEBUG:boto:Token: None 2024-04-12T04:19:57.961 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj29 2024-04-12T04:19:57.961 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fyvsFzfhA0NxhpMzP1dI+ORWEOo= 2024-04-12T04:19:57.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fyvsFzfhA0NxhpMzP1dI+ORWEOo=', 'Content-Length': '0'} 2024-04-12T04:19:57.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd52c4c2549fdb3b-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.963 DEBUG:boto:path=/dkyges-63/obj29 2024-04-12T04:19:57.963 DEBUG:boto:auth_path=/dkyges-63/obj29 2024-04-12T04:19:57.963 DEBUG:boto:Method: HEAD 2024-04-12T04:19:57.963 DEBUG:boto:Path: /dkyges-63/obj29 2024-04-12T04:19:57.963 DEBUG:boto:Data: 2024-04-12T04:19:57.963 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.964 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:57.964 DEBUG:boto:Port: 8000 2024-04-12T04:19:57.964 DEBUG:boto:Params: {} 2024-04-12T04:19:57.964 DEBUG:boto:Token: None 2024-04-12T04:19:57.964 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj29 2024-04-12T04:19:57.964 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fyvsFzfhA0NxhpMzP1dI+ORWEOo= 2024-04-12T04:19:57.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fyvsFzfhA0NxhpMzP1dI+ORWEOo=', 'Content-Length': '0'} 2024-04-12T04:19:57.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cc3decb0ba1ad6d-006618b66d-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:57.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2024-04-12T04:19:57.966 DEBUG:boto:path=/dkyges-63/obj29 2024-04-12T04:19:57.966 DEBUG:boto:auth_path=/dkyges-63/obj29 2024-04-12T04:19:57.966 DEBUG:boto:Method: GET 2024-04-12T04:19:57.967 DEBUG:boto:Path: /dkyges-63/obj29 2024-04-12T04:19:57.967 DEBUG:boto:Data: 2024-04-12T04:19:57.967 DEBUG:boto:Headers: {} 2024-04-12T04:19:57.967 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:57.967 DEBUG:boto:Port: 8001 2024-04-12T04:19:57.967 DEBUG:boto:Params: {} 2024-04-12T04:19:57.967 DEBUG:boto:Token: None 2024-04-12T04:19:57.967 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:57 GMT /dkyges-63/obj29 2024-04-12T04:19:57.967 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r1mEQeRojNFtX/rNedNOKnyu+Ac= 2024-04-12T04:19:57.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r1mEQeRojNFtX/rNedNOKnyu+Ac=', 'Content-Length': '0'} 2024-04-12T04:19:57.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f666f1ade7846d8-006618b66d-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.010 DEBUG:boto:path=/dkyges-63/obj29 2024-04-12T04:19:58.010 DEBUG:boto:auth_path=/dkyges-63/obj29 2024-04-12T04:19:58.011 DEBUG:boto:Method: GET 2024-04-12T04:19:58.011 DEBUG:boto:Path: /dkyges-63/obj29 2024-04-12T04:19:58.011 DEBUG:boto:Data: 2024-04-12T04:19:58.011 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.011 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.011 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.011 DEBUG:boto:Params: {} 2024-04-12T04:19:58.011 DEBUG:boto:Token: None 2024-04-12T04:19:58.011 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj29 2024-04-12T04:19:58.011 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TbwCtGQIOKkBNw6UBWUWXuNF1f8= 2024-04-12T04:19:58.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TbwCtGQIOKkBNw6UBWUWXuNF1f8=', 'Content-Length': '0'} 2024-04-12T04:19:58.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a67ac281f1077788-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.014 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.014 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.014 DEBUG:boto:path=/dkyges-63/?prefix=obj29 2024-04-12T04:19:58.014 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj29 2024-04-12T04:19:58.014 DEBUG:boto:Method: GET 2024-04-12T04:19:58.014 DEBUG:boto:Path: /dkyges-63/?prefix=obj29 2024-04-12T04:19:58.014 DEBUG:boto:Data: 2024-04-12T04:19:58.014 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.015 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.015 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.015 DEBUG:boto:Params: {} 2024-04-12T04:19:58.015 DEBUG:boto:Token: None 2024-04-12T04:19:58.015 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.015 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082d359f878247d11-006618b66e-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.017 DEBUG:boto:b'dkyges-63obj291000falseobj292024-04-12T04:15:48.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.018 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.018 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.018 DEBUG:boto:path=/dkyges-63/?prefix=obj29 2024-04-12T04:19:58.018 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj29 2024-04-12T04:19:58.018 DEBUG:boto:Method: GET 2024-04-12T04:19:58.018 DEBUG:boto:Path: /dkyges-63/?prefix=obj29 2024-04-12T04:19:58.018 DEBUG:boto:Data: 2024-04-12T04:19:58.018 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.018 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.019 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.019 DEBUG:boto:Params: {} 2024-04-12T04:19:58.019 DEBUG:boto:Token: None 2024-04-12T04:19:58.019 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.019 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080cb3f22bc11371f-006618b66e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.024 DEBUG:boto:b'dkyges-63obj291000falseobj292024-04-12T04:15:48.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T04:19:58.024 DEBUG:boto:path=/dkyges-63/obj31 2024-04-12T04:19:58.024 DEBUG:boto:auth_path=/dkyges-63/obj31 2024-04-12T04:19:58.024 DEBUG:boto:path=/dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.024 DEBUG:boto:auth_path=/dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.024 DEBUG:boto:Method: GET 2024-04-12T04:19:58.025 DEBUG:boto:Path: /dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.025 DEBUG:boto:Data: 2024-04-12T04:19:58.025 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.025 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.025 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.025 DEBUG:boto:Params: {} 2024-04-12T04:19:58.025 DEBUG:boto:Token: None 2024-04-12T04:19:58.025 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.025 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ywEVbaKQhIJjj8ZMrVz/i02AgVI= 2024-04-12T04:19:58.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ywEVbaKQhIJjj8ZMrVz/i02AgVI=', 'Content-Length': '0'} 2024-04-12T04:19:58.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c830a3f441f89e5c-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.027 DEBUG:boto:path=/dkyges-63/obj31 2024-04-12T04:19:58.027 DEBUG:boto:auth_path=/dkyges-63/obj31 2024-04-12T04:19:58.028 DEBUG:boto:path=/dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.028 DEBUG:boto:auth_path=/dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.028 DEBUG:boto:Method: GET 2024-04-12T04:19:58.028 DEBUG:boto:Path: /dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.028 DEBUG:boto:Data: 2024-04-12T04:19:58.028 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.028 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.028 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.028 DEBUG:boto:Params: {} 2024-04-12T04:19:58.028 DEBUG:boto:Token: None 2024-04-12T04:19:58.028 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.029 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ywEVbaKQhIJjj8ZMrVz/i02AgVI= 2024-04-12T04:19:58.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ywEVbaKQhIJjj8ZMrVz/i02AgVI=', 'Content-Length': '0'} 2024-04-12T04:19:58.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000591eb7867acc4263-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.074 DEBUG:boto:path=/dkyges-63/obj31 2024-04-12T04:19:58.074 DEBUG:boto:auth_path=/dkyges-63/obj31 2024-04-12T04:19:58.074 DEBUG:boto:path=/dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.074 DEBUG:boto:auth_path=/dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.075 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.075 DEBUG:boto:Path: /dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.075 DEBUG:boto:Data: 2024-04-12T04:19:58.075 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.075 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.075 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.075 DEBUG:boto:Params: {} 2024-04-12T04:19:58.075 DEBUG:boto:Token: None 2024-04-12T04:19:58.075 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.075 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SATofHVYgT7MK/hO2wenM9qGBps= 2024-04-12T04:19:58.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SATofHVYgT7MK/hO2wenM9qGBps=', 'Content-Length': '0'} 2024-04-12T04:19:58.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000469edf13b614c52b-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.077 DEBUG:boto:path=/dkyges-63/obj31 2024-04-12T04:19:58.077 DEBUG:boto:auth_path=/dkyges-63/obj31 2024-04-12T04:19:58.078 DEBUG:boto:path=/dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.078 DEBUG:boto:auth_path=/dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.078 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.078 DEBUG:boto:Path: /dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.078 DEBUG:boto:Data: 2024-04-12T04:19:58.078 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.078 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.078 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.078 DEBUG:boto:Params: {} 2024-04-12T04:19:58.078 DEBUG:boto:Token: None 2024-04-12T04:19:58.079 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.079 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SATofHVYgT7MK/hO2wenM9qGBps= 2024-04-12T04:19:58.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SATofHVYgT7MK/hO2wenM9qGBps=', 'Content-Length': '0'} 2024-04-12T04:19:58.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e0f219c6e932d38-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T04:19:58.081 DEBUG:boto:path=/dkyges-63/obj31 2024-04-12T04:19:58.081 DEBUG:boto:auth_path=/dkyges-63/obj31 2024-04-12T04:19:58.081 DEBUG:boto:path=/dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.081 DEBUG:boto:auth_path=/dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.081 DEBUG:boto:Method: GET 2024-04-12T04:19:58.081 DEBUG:boto:Path: /dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.081 DEBUG:boto:Data: 2024-04-12T04:19:58.082 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.082 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.082 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.082 DEBUG:boto:Params: {} 2024-04-12T04:19:58.082 DEBUG:boto:Token: None 2024-04-12T04:19:58.082 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.082 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ywEVbaKQhIJjj8ZMrVz/i02AgVI= 2024-04-12T04:19:58.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ywEVbaKQhIJjj8ZMrVz/i02AgVI=', 'Content-Length': '0'} 2024-04-12T04:19:58.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3d34ce0db6ec5ce-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.084 DEBUG:boto:path=/dkyges-63/obj31 2024-04-12T04:19:58.084 DEBUG:boto:auth_path=/dkyges-63/obj31 2024-04-12T04:19:58.084 DEBUG:boto:path=/dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.084 DEBUG:boto:auth_path=/dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.085 DEBUG:boto:Method: GET 2024-04-12T04:19:58.085 DEBUG:boto:Path: /dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.085 DEBUG:boto:Data: 2024-04-12T04:19:58.085 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.085 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.085 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.085 DEBUG:boto:Params: {} 2024-04-12T04:19:58.085 DEBUG:boto:Token: None 2024-04-12T04:19:58.085 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj31?versionId=null 2024-04-12T04:19:58.085 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ywEVbaKQhIJjj8ZMrVz/i02AgVI= 2024-04-12T04:19:58.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ywEVbaKQhIJjj8ZMrVz/i02AgVI=', 'Content-Length': '0'} 2024-04-12T04:19:58.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006508107e54c6ee4b-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.088 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.088 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.088 DEBUG:boto:path=/dkyges-63/?prefix=obj31 2024-04-12T04:19:58.088 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj31 2024-04-12T04:19:58.088 DEBUG:boto:Method: GET 2024-04-12T04:19:58.088 DEBUG:boto:Path: /dkyges-63/?prefix=obj31 2024-04-12T04:19:58.088 DEBUG:boto:Data: 2024-04-12T04:19:58.088 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.088 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.089 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.089 DEBUG:boto:Params: {} 2024-04-12T04:19:58.089 DEBUG:boto:Token: None 2024-04-12T04:19:58.089 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.089 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.092 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050c01c74a6193099-006618b66e-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.092 DEBUG:boto:b'dkyges-63obj311000falseobj312024-04-12T04:15:49.184Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.092 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.092 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.092 DEBUG:boto:path=/dkyges-63/?prefix=obj31 2024-04-12T04:19:58.092 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj31 2024-04-12T04:19:58.092 DEBUG:boto:Method: GET 2024-04-12T04:19:58.093 DEBUG:boto:Path: /dkyges-63/?prefix=obj31 2024-04-12T04:19:58.093 DEBUG:boto:Data: 2024-04-12T04:19:58.093 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.093 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.093 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.093 DEBUG:boto:Params: {} 2024-04-12T04:19:58.093 DEBUG:boto:Token: None 2024-04-12T04:19:58.093 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.093 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099e004f99d4fc7a5-006618b66e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.098 DEBUG:boto:b'dkyges-63obj311000falseobj312024-04-12T04:15:49.184Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.098 DEBUG:boto:path=/dkyges-63/obj31 2024-04-12T04:19:58.098 DEBUG:boto:auth_path=/dkyges-63/obj31 2024-04-12T04:19:58.099 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.099 DEBUG:boto:Path: /dkyges-63/obj31 2024-04-12T04:19:58.099 DEBUG:boto:Data: 2024-04-12T04:19:58.099 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.099 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.099 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.099 DEBUG:boto:Params: {} 2024-04-12T04:19:58.099 DEBUG:boto:Token: None 2024-04-12T04:19:58.099 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj31 2024-04-12T04:19:58.099 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FyrnlVcUlijGZGJ3iFwBh4Fol3k= 2024-04-12T04:19:58.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FyrnlVcUlijGZGJ3iFwBh4Fol3k=', 'Content-Length': '0'} 2024-04-12T04:19:58.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd09f0873af7acc3-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.101 DEBUG:boto:path=/dkyges-63/obj31 2024-04-12T04:19:58.101 DEBUG:boto:auth_path=/dkyges-63/obj31 2024-04-12T04:19:58.102 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.102 DEBUG:boto:Path: /dkyges-63/obj31 2024-04-12T04:19:58.102 DEBUG:boto:Data: 2024-04-12T04:19:58.102 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.102 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.102 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.102 DEBUG:boto:Params: {} 2024-04-12T04:19:58.102 DEBUG:boto:Token: None 2024-04-12T04:19:58.102 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj31 2024-04-12T04:19:58.102 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FyrnlVcUlijGZGJ3iFwBh4Fol3k= 2024-04-12T04:19:58.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FyrnlVcUlijGZGJ3iFwBh4Fol3k=', 'Content-Length': '0'} 2024-04-12T04:19:58.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2d83140d3b5446d-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2024-04-12T04:19:58.105 DEBUG:boto:path=/dkyges-63/obj31 2024-04-12T04:19:58.105 DEBUG:boto:auth_path=/dkyges-63/obj31 2024-04-12T04:19:58.105 DEBUG:boto:Method: GET 2024-04-12T04:19:58.105 DEBUG:boto:Path: /dkyges-63/obj31 2024-04-12T04:19:58.105 DEBUG:boto:Data: 2024-04-12T04:19:58.105 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.105 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.105 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.105 DEBUG:boto:Params: {} 2024-04-12T04:19:58.106 DEBUG:boto:Token: None 2024-04-12T04:19:58.106 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj31 2024-04-12T04:19:58.106 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JuYkpmW5VKeEFeKXeyX535pGCtw= 2024-04-12T04:19:58.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JuYkpmW5VKeEFeKXeyX535pGCtw=', 'Content-Length': '0'} 2024-04-12T04:19:58.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f6e0264c1067006-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.108 DEBUG:boto:path=/dkyges-63/obj31 2024-04-12T04:19:58.108 DEBUG:boto:auth_path=/dkyges-63/obj31 2024-04-12T04:19:58.108 DEBUG:boto:Method: GET 2024-04-12T04:19:58.108 DEBUG:boto:Path: /dkyges-63/obj31 2024-04-12T04:19:58.108 DEBUG:boto:Data: 2024-04-12T04:19:58.108 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.108 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.108 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.108 DEBUG:boto:Params: {} 2024-04-12T04:19:58.109 DEBUG:boto:Token: None 2024-04-12T04:19:58.109 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj31 2024-04-12T04:19:58.109 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JuYkpmW5VKeEFeKXeyX535pGCtw= 2024-04-12T04:19:58.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JuYkpmW5VKeEFeKXeyX535pGCtw=', 'Content-Length': '0'} 2024-04-12T04:19:58.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cec7d53f07e645cd-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.154 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.154 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.155 DEBUG:boto:path=/dkyges-63/?prefix=obj31 2024-04-12T04:19:58.155 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj31 2024-04-12T04:19:58.155 DEBUG:boto:Method: GET 2024-04-12T04:19:58.155 DEBUG:boto:Path: /dkyges-63/?prefix=obj31 2024-04-12T04:19:58.155 DEBUG:boto:Data: 2024-04-12T04:19:58.155 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.155 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.155 DEBUG:boto:Params: {} 2024-04-12T04:19:58.155 DEBUG:boto:Token: None 2024-04-12T04:19:58.155 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.158 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f904640e3a7c2941-006618b66e-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.158 DEBUG:boto:b'dkyges-63obj311000falseobj312024-04-12T04:15:49.184Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.159 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.159 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.159 DEBUG:boto:path=/dkyges-63/?prefix=obj31 2024-04-12T04:19:58.159 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj31 2024-04-12T04:19:58.159 DEBUG:boto:Method: GET 2024-04-12T04:19:58.159 DEBUG:boto:Path: /dkyges-63/?prefix=obj31 2024-04-12T04:19:58.159 DEBUG:boto:Data: 2024-04-12T04:19:58.159 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.159 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.160 DEBUG:boto:Params: {} 2024-04-12T04:19:58.160 DEBUG:boto:Token: None 2024-04-12T04:19:58.160 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.160 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000772fdda1724177ae-006618b66e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.165 DEBUG:boto:b'dkyges-63obj311000falseobj312024-04-12T04:15:49.184Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:19:58.166 DEBUG:boto:path=/dkyges-63/obj33 2024-04-12T04:19:58.166 DEBUG:boto:auth_path=/dkyges-63/obj33 2024-04-12T04:19:58.166 DEBUG:boto:path=/dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.166 DEBUG:boto:auth_path=/dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.166 DEBUG:boto:Method: GET 2024-04-12T04:19:58.166 DEBUG:boto:Path: /dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.166 DEBUG:boto:Data: 2024-04-12T04:19:58.167 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.167 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.167 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.167 DEBUG:boto:Params: {} 2024-04-12T04:19:58.167 DEBUG:boto:Token: None 2024-04-12T04:19:58.168 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.168 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:24FGIbeBuaJtV2EBw7324RouQkk= 2024-04-12T04:19:58.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:24FGIbeBuaJtV2EBw7324RouQkk=', 'Content-Length': '0'} 2024-04-12T04:19:58.170 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000945d5bd028fa5dd9-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.210 DEBUG:boto:path=/dkyges-63/obj33 2024-04-12T04:19:58.210 DEBUG:boto:auth_path=/dkyges-63/obj33 2024-04-12T04:19:58.210 DEBUG:boto:path=/dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.211 DEBUG:boto:auth_path=/dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.211 DEBUG:boto:Method: GET 2024-04-12T04:19:58.211 DEBUG:boto:Path: /dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.211 DEBUG:boto:Data: 2024-04-12T04:19:58.211 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.211 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.211 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.211 DEBUG:boto:Params: {} 2024-04-12T04:19:58.211 DEBUG:boto:Token: None 2024-04-12T04:19:58.211 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.211 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:24FGIbeBuaJtV2EBw7324RouQkk= 2024-04-12T04:19:58.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:24FGIbeBuaJtV2EBw7324RouQkk=', 'Content-Length': '0'} 2024-04-12T04:19:58.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4e4a8c6cc7a6bdb-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.214 DEBUG:boto:path=/dkyges-63/obj33 2024-04-12T04:19:58.214 DEBUG:boto:auth_path=/dkyges-63/obj33 2024-04-12T04:19:58.214 DEBUG:boto:path=/dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.215 DEBUG:boto:auth_path=/dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.215 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.215 DEBUG:boto:Path: /dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.215 DEBUG:boto:Data: 2024-04-12T04:19:58.215 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.216 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.216 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.216 DEBUG:boto:Params: {} 2024-04-12T04:19:58.216 DEBUG:boto:Token: None 2024-04-12T04:19:58.216 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.216 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4VyaxzQ+95vhhRz6o/tK4OSI1ZE= 2024-04-12T04:19:58.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4VyaxzQ+95vhhRz6o/tK4OSI1ZE=', 'Content-Length': '0'} 2024-04-12T04:19:58.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d055b00791dc1eda-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.218 DEBUG:boto:path=/dkyges-63/obj33 2024-04-12T04:19:58.219 DEBUG:boto:auth_path=/dkyges-63/obj33 2024-04-12T04:19:58.219 DEBUG:boto:path=/dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.219 DEBUG:boto:auth_path=/dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.219 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.219 DEBUG:boto:Path: /dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.219 DEBUG:boto:Data: 2024-04-12T04:19:58.219 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.219 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.220 DEBUG:boto:Params: {} 2024-04-12T04:19:58.220 DEBUG:boto:Token: None 2024-04-12T04:19:58.220 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.220 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4VyaxzQ+95vhhRz6o/tK4OSI1ZE= 2024-04-12T04:19:58.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4VyaxzQ+95vhhRz6o/tK4OSI1ZE=', 'Content-Length': '0'} 2024-04-12T04:19:58.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005294fca41c5a4eb5-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:19:58.222 DEBUG:boto:path=/dkyges-63/obj33 2024-04-12T04:19:58.222 DEBUG:boto:auth_path=/dkyges-63/obj33 2024-04-12T04:19:58.222 DEBUG:boto:path=/dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.222 DEBUG:boto:auth_path=/dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.222 DEBUG:boto:Method: GET 2024-04-12T04:19:58.223 DEBUG:boto:Path: /dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.223 DEBUG:boto:Data: 2024-04-12T04:19:58.223 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.223 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.223 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.223 DEBUG:boto:Params: {} 2024-04-12T04:19:58.223 DEBUG:boto:Token: None 2024-04-12T04:19:58.223 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.223 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:24FGIbeBuaJtV2EBw7324RouQkk= 2024-04-12T04:19:58.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:24FGIbeBuaJtV2EBw7324RouQkk=', 'Content-Length': '0'} 2024-04-12T04:19:58.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007db8aa51d68cc67-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.226 DEBUG:boto:path=/dkyges-63/obj33 2024-04-12T04:19:58.226 DEBUG:boto:auth_path=/dkyges-63/obj33 2024-04-12T04:19:58.226 DEBUG:boto:path=/dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.226 DEBUG:boto:auth_path=/dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.226 DEBUG:boto:Method: GET 2024-04-12T04:19:58.226 DEBUG:boto:Path: /dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.226 DEBUG:boto:Data: 2024-04-12T04:19:58.226 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.226 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.227 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.227 DEBUG:boto:Params: {} 2024-04-12T04:19:58.227 DEBUG:boto:Token: None 2024-04-12T04:19:58.227 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj33?versionId=null 2024-04-12T04:19:58.227 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:24FGIbeBuaJtV2EBw7324RouQkk= 2024-04-12T04:19:58.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:24FGIbeBuaJtV2EBw7324RouQkk=', 'Content-Length': '0'} 2024-04-12T04:19:58.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d82565d39e13ca2-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.229 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.229 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.229 DEBUG:boto:path=/dkyges-63/?prefix=obj33 2024-04-12T04:19:58.230 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj33 2024-04-12T04:19:58.230 DEBUG:boto:Method: GET 2024-04-12T04:19:58.230 DEBUG:boto:Path: /dkyges-63/?prefix=obj33 2024-04-12T04:19:58.230 DEBUG:boto:Data: 2024-04-12T04:19:58.230 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.230 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.230 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.230 DEBUG:boto:Params: {} 2024-04-12T04:19:58.230 DEBUG:boto:Token: None 2024-04-12T04:19:58.233 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.233 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac7ed6fb44af53ad-006618b66e-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.237 DEBUG:boto:b'dkyges-63obj331000falseobj332024-04-12T04:15:47.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.237 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.237 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.237 DEBUG:boto:path=/dkyges-63/?prefix=obj33 2024-04-12T04:19:58.238 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj33 2024-04-12T04:19:58.238 DEBUG:boto:Method: GET 2024-04-12T04:19:58.238 DEBUG:boto:Path: /dkyges-63/?prefix=obj33 2024-04-12T04:19:58.238 DEBUG:boto:Data: 2024-04-12T04:19:58.238 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.238 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.238 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.238 DEBUG:boto:Params: {} 2024-04-12T04:19:58.238 DEBUG:boto:Token: None 2024-04-12T04:19:58.238 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.238 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a883caf826f84dba-006618b66e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.243 DEBUG:boto:b'dkyges-63obj331000falseobj332024-04-12T04:15:47.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.243 DEBUG:boto:path=/dkyges-63/obj33 2024-04-12T04:19:58.244 DEBUG:boto:auth_path=/dkyges-63/obj33 2024-04-12T04:19:58.244 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.244 DEBUG:boto:Path: /dkyges-63/obj33 2024-04-12T04:19:58.244 DEBUG:boto:Data: 2024-04-12T04:19:58.244 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.244 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.244 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.244 DEBUG:boto:Params: {} 2024-04-12T04:19:58.244 DEBUG:boto:Token: None 2024-04-12T04:19:58.244 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj33 2024-04-12T04:19:58.244 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7RKOVJdCG/DQJ+MnO8YeifvhkL0= 2024-04-12T04:19:58.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7RKOVJdCG/DQJ+MnO8YeifvhkL0=', 'Content-Length': '0'} 2024-04-12T04:19:58.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e379df151e6ef03-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.247 DEBUG:boto:path=/dkyges-63/obj33 2024-04-12T04:19:58.247 DEBUG:boto:auth_path=/dkyges-63/obj33 2024-04-12T04:19:58.247 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.247 DEBUG:boto:Path: /dkyges-63/obj33 2024-04-12T04:19:58.247 DEBUG:boto:Data: 2024-04-12T04:19:58.247 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.247 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.247 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.247 DEBUG:boto:Params: {} 2024-04-12T04:19:58.248 DEBUG:boto:Token: None 2024-04-12T04:19:58.248 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj33 2024-04-12T04:19:58.248 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7RKOVJdCG/DQJ+MnO8YeifvhkL0= 2024-04-12T04:19:58.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7RKOVJdCG/DQJ+MnO8YeifvhkL0=', 'Content-Length': '0'} 2024-04-12T04:19:58.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c3c0af462f8b3f9-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:19:58.250 DEBUG:boto:path=/dkyges-63/obj33 2024-04-12T04:19:58.250 DEBUG:boto:auth_path=/dkyges-63/obj33 2024-04-12T04:19:58.250 DEBUG:boto:Method: GET 2024-04-12T04:19:58.250 DEBUG:boto:Path: /dkyges-63/obj33 2024-04-12T04:19:58.250 DEBUG:boto:Data: 2024-04-12T04:19:58.250 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.251 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.251 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.251 DEBUG:boto:Params: {} 2024-04-12T04:19:58.251 DEBUG:boto:Token: None 2024-04-12T04:19:58.251 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj33 2024-04-12T04:19:58.251 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7bwoOwQ132qyx1Mt3du1PQN0FU4= 2024-04-12T04:19:58.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7bwoOwQ132qyx1Mt3du1PQN0FU4=', 'Content-Length': '0'} 2024-04-12T04:19:58.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8ffcd17ffe7ef99-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.253 DEBUG:boto:path=/dkyges-63/obj33 2024-04-12T04:19:58.253 DEBUG:boto:auth_path=/dkyges-63/obj33 2024-04-12T04:19:58.253 DEBUG:boto:Method: GET 2024-04-12T04:19:58.253 DEBUG:boto:Path: /dkyges-63/obj33 2024-04-12T04:19:58.254 DEBUG:boto:Data: 2024-04-12T04:19:58.254 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.254 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.254 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.254 DEBUG:boto:Params: {} 2024-04-12T04:19:58.254 DEBUG:boto:Token: None 2024-04-12T04:19:58.254 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj33 2024-04-12T04:19:58.255 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7bwoOwQ132qyx1Mt3du1PQN0FU4= 2024-04-12T04:19:58.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7bwoOwQ132qyx1Mt3du1PQN0FU4=', 'Content-Length': '0'} 2024-04-12T04:19:58.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9dd32f7cb09240f-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.257 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.257 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.257 DEBUG:boto:path=/dkyges-63/?prefix=obj33 2024-04-12T04:19:58.257 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj33 2024-04-12T04:19:58.257 DEBUG:boto:Method: GET 2024-04-12T04:19:58.257 DEBUG:boto:Path: /dkyges-63/?prefix=obj33 2024-04-12T04:19:58.258 DEBUG:boto:Data: 2024-04-12T04:19:58.258 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.258 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.258 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.258 DEBUG:boto:Params: {} 2024-04-12T04:19:58.258 DEBUG:boto:Token: None 2024-04-12T04:19:58.258 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.258 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7250fd2c7d30993-006618b66e-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.261 DEBUG:boto:b'dkyges-63obj331000falseobj332024-04-12T04:15:47.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.261 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.261 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.262 DEBUG:boto:path=/dkyges-63/?prefix=obj33 2024-04-12T04:19:58.262 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj33 2024-04-12T04:19:58.262 DEBUG:boto:Method: GET 2024-04-12T04:19:58.262 DEBUG:boto:Path: /dkyges-63/?prefix=obj33 2024-04-12T04:19:58.262 DEBUG:boto:Data: 2024-04-12T04:19:58.262 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.262 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.262 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.262 DEBUG:boto:Params: {} 2024-04-12T04:19:58.262 DEBUG:boto:Token: None 2024-04-12T04:19:58.262 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.263 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb131a0b59dc36e7-006618b66e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.267 DEBUG:boto:b'dkyges-63obj331000falseobj332024-04-12T04:15:47.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:19:58.268 DEBUG:boto:path=/dkyges-63/obj34 2024-04-12T04:19:58.268 DEBUG:boto:auth_path=/dkyges-63/obj34 2024-04-12T04:19:58.268 DEBUG:boto:path=/dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.268 DEBUG:boto:auth_path=/dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.268 DEBUG:boto:Method: GET 2024-04-12T04:19:58.268 DEBUG:boto:Path: /dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.268 DEBUG:boto:Data: 2024-04-12T04:19:58.268 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.268 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.268 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.269 DEBUG:boto:Params: {} 2024-04-12T04:19:58.269 DEBUG:boto:Token: None 2024-04-12T04:19:58.269 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.269 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:upsce8TTAtg3Qw0WEzGie8j2R14= 2024-04-12T04:19:58.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:upsce8TTAtg3Qw0WEzGie8j2R14=', 'Content-Length': '0'} 2024-04-12T04:19:58.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce8a82d89502006d-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.271 DEBUG:boto:path=/dkyges-63/obj34 2024-04-12T04:19:58.271 DEBUG:boto:auth_path=/dkyges-63/obj34 2024-04-12T04:19:58.271 DEBUG:boto:path=/dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.271 DEBUG:boto:auth_path=/dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.272 DEBUG:boto:Method: GET 2024-04-12T04:19:58.272 DEBUG:boto:Path: /dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.272 DEBUG:boto:Data: 2024-04-12T04:19:58.272 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.272 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.272 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.272 DEBUG:boto:Params: {} 2024-04-12T04:19:58.272 DEBUG:boto:Token: None 2024-04-12T04:19:58.272 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.272 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:upsce8TTAtg3Qw0WEzGie8j2R14= 2024-04-12T04:19:58.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:upsce8TTAtg3Qw0WEzGie8j2R14=', 'Content-Length': '0'} 2024-04-12T04:19:58.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfd4637d42516072-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.318 DEBUG:boto:path=/dkyges-63/obj34 2024-04-12T04:19:58.318 DEBUG:boto:auth_path=/dkyges-63/obj34 2024-04-12T04:19:58.319 DEBUG:boto:path=/dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.319 DEBUG:boto:auth_path=/dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.319 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.319 DEBUG:boto:Path: /dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.319 DEBUG:boto:Data: 2024-04-12T04:19:58.319 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.319 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.319 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.319 DEBUG:boto:Params: {} 2024-04-12T04:19:58.319 DEBUG:boto:Token: None 2024-04-12T04:19:58.319 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.320 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+CHZZonM4w3cAUVa2r95txihOOQ= 2024-04-12T04:19:58.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+CHZZonM4w3cAUVa2r95txihOOQ=', 'Content-Length': '0'} 2024-04-12T04:19:58.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5d663eba51d1bde-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.322 DEBUG:boto:path=/dkyges-63/obj34 2024-04-12T04:19:58.322 DEBUG:boto:auth_path=/dkyges-63/obj34 2024-04-12T04:19:58.322 DEBUG:boto:path=/dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.322 DEBUG:boto:auth_path=/dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.322 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.322 DEBUG:boto:Path: /dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.323 DEBUG:boto:Data: 2024-04-12T04:19:58.323 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.323 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.323 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.323 DEBUG:boto:Params: {} 2024-04-12T04:19:58.323 DEBUG:boto:Token: None 2024-04-12T04:19:58.324 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.324 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+CHZZonM4w3cAUVa2r95txihOOQ= 2024-04-12T04:19:58.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+CHZZonM4w3cAUVa2r95txihOOQ=', 'Content-Length': '0'} 2024-04-12T04:19:58.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024a17fecbb25a8bb-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:19:58.326 DEBUG:boto:path=/dkyges-63/obj34 2024-04-12T04:19:58.327 DEBUG:boto:auth_path=/dkyges-63/obj34 2024-04-12T04:19:58.327 DEBUG:boto:path=/dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.327 DEBUG:boto:auth_path=/dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.327 DEBUG:boto:Method: GET 2024-04-12T04:19:58.327 DEBUG:boto:Path: /dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.327 DEBUG:boto:Data: 2024-04-12T04:19:58.327 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.327 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.327 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.327 DEBUG:boto:Params: {} 2024-04-12T04:19:58.327 DEBUG:boto:Token: None 2024-04-12T04:19:58.328 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.328 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:upsce8TTAtg3Qw0WEzGie8j2R14= 2024-04-12T04:19:58.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:upsce8TTAtg3Qw0WEzGie8j2R14=', 'Content-Length': '0'} 2024-04-12T04:19:58.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fdf75aeafd20aef-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.330 DEBUG:boto:path=/dkyges-63/obj34 2024-04-12T04:19:58.330 DEBUG:boto:auth_path=/dkyges-63/obj34 2024-04-12T04:19:58.330 DEBUG:boto:path=/dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.330 DEBUG:boto:auth_path=/dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.330 DEBUG:boto:Method: GET 2024-04-12T04:19:58.331 DEBUG:boto:Path: /dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.331 DEBUG:boto:Data: 2024-04-12T04:19:58.331 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.331 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.331 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.331 DEBUG:boto:Params: {} 2024-04-12T04:19:58.331 DEBUG:boto:Token: None 2024-04-12T04:19:58.331 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj34?versionId=null 2024-04-12T04:19:58.331 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:upsce8TTAtg3Qw0WEzGie8j2R14= 2024-04-12T04:19:58.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:upsce8TTAtg3Qw0WEzGie8j2R14=', 'Content-Length': '0'} 2024-04-12T04:19:58.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acf52751f1ff124d-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.333 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.333 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.334 DEBUG:boto:path=/dkyges-63/?prefix=obj34 2024-04-12T04:19:58.334 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj34 2024-04-12T04:19:58.334 DEBUG:boto:Method: GET 2024-04-12T04:19:58.334 DEBUG:boto:Path: /dkyges-63/?prefix=obj34 2024-04-12T04:19:58.334 DEBUG:boto:Data: 2024-04-12T04:19:58.334 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.334 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.334 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.334 DEBUG:boto:Params: {} 2024-04-12T04:19:58.334 DEBUG:boto:Token: None 2024-04-12T04:19:58.334 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.335 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.337 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f47a3d2f38c1eca-006618b66e-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.337 DEBUG:boto:b'dkyges-63obj341000falseobj342024-04-12T04:15:47.809Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.337 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.337 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.338 DEBUG:boto:path=/dkyges-63/?prefix=obj34 2024-04-12T04:19:58.338 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj34 2024-04-12T04:19:58.338 DEBUG:boto:Method: GET 2024-04-12T04:19:58.338 DEBUG:boto:Path: /dkyges-63/?prefix=obj34 2024-04-12T04:19:58.338 DEBUG:boto:Data: 2024-04-12T04:19:58.338 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.338 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.338 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.338 DEBUG:boto:Params: {} 2024-04-12T04:19:58.339 DEBUG:boto:Token: None 2024-04-12T04:19:58.339 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.339 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c2ae26a7642406a-006618b66e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.344 DEBUG:boto:b'dkyges-63obj341000falseobj342024-04-12T04:15:47.809Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.344 DEBUG:boto:path=/dkyges-63/obj34 2024-04-12T04:19:58.345 DEBUG:boto:auth_path=/dkyges-63/obj34 2024-04-12T04:19:58.345 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.345 DEBUG:boto:Path: /dkyges-63/obj34 2024-04-12T04:19:58.345 DEBUG:boto:Data: 2024-04-12T04:19:58.345 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.345 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.346 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.346 DEBUG:boto:Params: {} 2024-04-12T04:19:58.346 DEBUG:boto:Token: None 2024-04-12T04:19:58.346 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj34 2024-04-12T04:19:58.346 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/Q7NyoN9wFB20oB/eLrId4AbAOg= 2024-04-12T04:19:58.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/Q7NyoN9wFB20oB/eLrId4AbAOg=', 'Content-Length': '0'} 2024-04-12T04:19:58.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d3bb1d51f421843-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.349 DEBUG:boto:path=/dkyges-63/obj34 2024-04-12T04:19:58.349 DEBUG:boto:auth_path=/dkyges-63/obj34 2024-04-12T04:19:58.349 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.350 DEBUG:boto:Path: /dkyges-63/obj34 2024-04-12T04:19:58.350 DEBUG:boto:Data: 2024-04-12T04:19:58.350 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.350 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.350 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.351 DEBUG:boto:Params: {} 2024-04-12T04:19:58.351 DEBUG:boto:Token: None 2024-04-12T04:19:58.351 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj34 2024-04-12T04:19:58.351 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/Q7NyoN9wFB20oB/eLrId4AbAOg= 2024-04-12T04:19:58.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/Q7NyoN9wFB20oB/eLrId4AbAOg=', 'Content-Length': '0'} 2024-04-12T04:19:58.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c05f26ae4c4f8e1b-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2024-04-12T04:19:58.354 DEBUG:boto:path=/dkyges-63/obj34 2024-04-12T04:19:58.354 DEBUG:boto:auth_path=/dkyges-63/obj34 2024-04-12T04:19:58.355 DEBUG:boto:Method: GET 2024-04-12T04:19:58.355 DEBUG:boto:Path: /dkyges-63/obj34 2024-04-12T04:19:58.355 DEBUG:boto:Data: 2024-04-12T04:19:58.355 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.355 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.356 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.356 DEBUG:boto:Params: {} 2024-04-12T04:19:58.356 DEBUG:boto:Token: None 2024-04-12T04:19:58.356 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj34 2024-04-12T04:19:58.357 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Mc1JRCXMp/rJ7806GBuVHTR3gTU= 2024-04-12T04:19:58.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Mc1JRCXMp/rJ7806GBuVHTR3gTU=', 'Content-Length': '0'} 2024-04-12T04:19:58.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e7aaf0a5709bbeb-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.359 DEBUG:boto:path=/dkyges-63/obj34 2024-04-12T04:19:58.359 DEBUG:boto:auth_path=/dkyges-63/obj34 2024-04-12T04:19:58.360 DEBUG:boto:Method: GET 2024-04-12T04:19:58.360 DEBUG:boto:Path: /dkyges-63/obj34 2024-04-12T04:19:58.360 DEBUG:boto:Data: 2024-04-12T04:19:58.360 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.360 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.361 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.361 DEBUG:boto:Params: {} 2024-04-12T04:19:58.361 DEBUG:boto:Token: None 2024-04-12T04:19:58.361 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj34 2024-04-12T04:19:58.361 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Mc1JRCXMp/rJ7806GBuVHTR3gTU= 2024-04-12T04:19:58.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Mc1JRCXMp/rJ7806GBuVHTR3gTU=', 'Content-Length': '0'} 2024-04-12T04:19:58.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000743712b0c67db99d-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.364 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.364 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.364 DEBUG:boto:path=/dkyges-63/?prefix=obj34 2024-04-12T04:19:58.365 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj34 2024-04-12T04:19:58.365 DEBUG:boto:Method: GET 2024-04-12T04:19:58.365 DEBUG:boto:Path: /dkyges-63/?prefix=obj34 2024-04-12T04:19:58.365 DEBUG:boto:Data: 2024-04-12T04:19:58.365 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.366 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.366 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.366 DEBUG:boto:Params: {} 2024-04-12T04:19:58.366 DEBUG:boto:Token: None 2024-04-12T04:19:58.367 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.367 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.370 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000360e6361fd049b12-006618b66e-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.370 DEBUG:boto:b'dkyges-63obj341000falseobj342024-04-12T04:15:47.809Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.371 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.371 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.371 DEBUG:boto:path=/dkyges-63/?prefix=obj34 2024-04-12T04:19:58.371 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj34 2024-04-12T04:19:58.371 DEBUG:boto:Method: GET 2024-04-12T04:19:58.372 DEBUG:boto:Path: /dkyges-63/?prefix=obj34 2024-04-12T04:19:58.372 DEBUG:boto:Data: 2024-04-12T04:19:58.372 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.372 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.372 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.372 DEBUG:boto:Params: {} 2024-04-12T04:19:58.373 DEBUG:boto:Token: None 2024-04-12T04:19:58.373 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.373 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.378 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b369e7710e0e7ee2-006618b66e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.379 DEBUG:boto:b'dkyges-63obj341000falseobj342024-04-12T04:15:47.809Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T04:19:58.379 DEBUG:boto:path=/dkyges-63/obj37 2024-04-12T04:19:58.379 DEBUG:boto:auth_path=/dkyges-63/obj37 2024-04-12T04:19:58.380 DEBUG:boto:path=/dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.380 DEBUG:boto:auth_path=/dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.380 DEBUG:boto:Method: GET 2024-04-12T04:19:58.380 DEBUG:boto:Path: /dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.380 DEBUG:boto:Data: 2024-04-12T04:19:58.381 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.381 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.381 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.381 DEBUG:boto:Params: {} 2024-04-12T04:19:58.381 DEBUG:boto:Token: None 2024-04-12T04:19:58.382 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.382 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oaV1ygD5IPx8+aETij/wgGgF3qI= 2024-04-12T04:19:58.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oaV1ygD5IPx8+aETij/wgGgF3qI=', 'Content-Length': '0'} 2024-04-12T04:19:58.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000279a4035e1729be1-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.385 DEBUG:boto:path=/dkyges-63/obj37 2024-04-12T04:19:58.385 DEBUG:boto:auth_path=/dkyges-63/obj37 2024-04-12T04:19:58.385 DEBUG:boto:path=/dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.385 DEBUG:boto:auth_path=/dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.386 DEBUG:boto:Method: GET 2024-04-12T04:19:58.386 DEBUG:boto:Path: /dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.386 DEBUG:boto:Data: 2024-04-12T04:19:58.386 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.386 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.386 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.387 DEBUG:boto:Params: {} 2024-04-12T04:19:58.387 DEBUG:boto:Token: None 2024-04-12T04:19:58.387 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.387 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oaV1ygD5IPx8+aETij/wgGgF3qI= 2024-04-12T04:19:58.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oaV1ygD5IPx8+aETij/wgGgF3qI=', 'Content-Length': '0'} 2024-04-12T04:19:58.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab8571d25dd45b79-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.430 DEBUG:boto:path=/dkyges-63/obj37 2024-04-12T04:19:58.431 DEBUG:boto:auth_path=/dkyges-63/obj37 2024-04-12T04:19:58.431 DEBUG:boto:path=/dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.431 DEBUG:boto:auth_path=/dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.431 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.431 DEBUG:boto:Path: /dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.431 DEBUG:boto:Data: 2024-04-12T04:19:58.432 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.432 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.432 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.432 DEBUG:boto:Params: {} 2024-04-12T04:19:58.432 DEBUG:boto:Token: None 2024-04-12T04:19:58.433 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.433 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eXrXBaCksQYPu/Bn0ZzeWejhnxw= 2024-04-12T04:19:58.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eXrXBaCksQYPu/Bn0ZzeWejhnxw=', 'Content-Length': '0'} 2024-04-12T04:19:58.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a500b1f58d46c7e7-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.435 DEBUG:boto:path=/dkyges-63/obj37 2024-04-12T04:19:58.436 DEBUG:boto:auth_path=/dkyges-63/obj37 2024-04-12T04:19:58.436 DEBUG:boto:path=/dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.436 DEBUG:boto:auth_path=/dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.436 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.436 DEBUG:boto:Path: /dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.437 DEBUG:boto:Data: 2024-04-12T04:19:58.437 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.437 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.437 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.437 DEBUG:boto:Params: {} 2024-04-12T04:19:58.438 DEBUG:boto:Token: None 2024-04-12T04:19:58.438 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.438 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eXrXBaCksQYPu/Bn0ZzeWejhnxw= 2024-04-12T04:19:58.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eXrXBaCksQYPu/Bn0ZzeWejhnxw=', 'Content-Length': '0'} 2024-04-12T04:19:58.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000443dcf0d5ad2cd59-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T04:19:58.441 DEBUG:boto:path=/dkyges-63/obj37 2024-04-12T04:19:58.441 DEBUG:boto:auth_path=/dkyges-63/obj37 2024-04-12T04:19:58.441 DEBUG:boto:path=/dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.441 DEBUG:boto:auth_path=/dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.441 DEBUG:boto:Method: GET 2024-04-12T04:19:58.442 DEBUG:boto:Path: /dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.442 DEBUG:boto:Data: 2024-04-12T04:19:58.442 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.442 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.442 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.443 DEBUG:boto:Params: {} 2024-04-12T04:19:58.443 DEBUG:boto:Token: None 2024-04-12T04:19:58.443 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.443 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oaV1ygD5IPx8+aETij/wgGgF3qI= 2024-04-12T04:19:58.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oaV1ygD5IPx8+aETij/wgGgF3qI=', 'Content-Length': '0'} 2024-04-12T04:19:58.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0353d850de188f7-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.486 DEBUG:boto:path=/dkyges-63/obj37 2024-04-12T04:19:58.487 DEBUG:boto:auth_path=/dkyges-63/obj37 2024-04-12T04:19:58.487 DEBUG:boto:path=/dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.487 DEBUG:boto:auth_path=/dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.487 DEBUG:boto:Method: GET 2024-04-12T04:19:58.487 DEBUG:boto:Path: /dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.487 DEBUG:boto:Data: 2024-04-12T04:19:58.488 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.488 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.488 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.488 DEBUG:boto:Params: {} 2024-04-12T04:19:58.488 DEBUG:boto:Token: None 2024-04-12T04:19:58.489 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj37?versionId=null 2024-04-12T04:19:58.489 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oaV1ygD5IPx8+aETij/wgGgF3qI= 2024-04-12T04:19:58.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oaV1ygD5IPx8+aETij/wgGgF3qI=', 'Content-Length': '0'} 2024-04-12T04:19:58.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb1064c37650feac-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.491 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.492 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.492 DEBUG:boto:path=/dkyges-63/?prefix=obj37 2024-04-12T04:19:58.492 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj37 2024-04-12T04:19:58.492 DEBUG:boto:Method: GET 2024-04-12T04:19:58.493 DEBUG:boto:Path: /dkyges-63/?prefix=obj37 2024-04-12T04:19:58.493 DEBUG:boto:Data: 2024-04-12T04:19:58.493 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.493 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.493 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.493 DEBUG:boto:Params: {} 2024-04-12T04:19:58.494 DEBUG:boto:Token: None 2024-04-12T04:19:58.494 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.494 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a8eae84b08eeb86-006618b66e-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.497 DEBUG:boto:b'dkyges-63obj371000falseobj372024-04-12T04:15:49.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.498 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.498 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.498 DEBUG:boto:path=/dkyges-63/?prefix=obj37 2024-04-12T04:19:58.498 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj37 2024-04-12T04:19:58.498 DEBUG:boto:Method: GET 2024-04-12T04:19:58.498 DEBUG:boto:Path: /dkyges-63/?prefix=obj37 2024-04-12T04:19:58.498 DEBUG:boto:Data: 2024-04-12T04:19:58.499 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.499 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.499 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.499 DEBUG:boto:Params: {} 2024-04-12T04:19:58.499 DEBUG:boto:Token: None 2024-04-12T04:19:58.499 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.499 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031203c85b24a84c6-006618b66e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.504 DEBUG:boto:b'dkyges-63obj371000falseobj372024-04-12T04:15:49.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.504 DEBUG:boto:path=/dkyges-63/obj37 2024-04-12T04:19:58.504 DEBUG:boto:auth_path=/dkyges-63/obj37 2024-04-12T04:19:58.504 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.504 DEBUG:boto:Path: /dkyges-63/obj37 2024-04-12T04:19:58.504 DEBUG:boto:Data: 2024-04-12T04:19:58.504 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.504 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.504 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.505 DEBUG:boto:Params: {} 2024-04-12T04:19:58.505 DEBUG:boto:Token: None 2024-04-12T04:19:58.505 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj37 2024-04-12T04:19:58.505 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ALZO7ml+OOfWCKh9Wgf/HVOA4MY= 2024-04-12T04:19:58.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ALZO7ml+OOfWCKh9Wgf/HVOA4MY=', 'Content-Length': '0'} 2024-04-12T04:19:58.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000006ae85fd3d04c2e-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.507 DEBUG:boto:path=/dkyges-63/obj37 2024-04-12T04:19:58.507 DEBUG:boto:auth_path=/dkyges-63/obj37 2024-04-12T04:19:58.507 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.507 DEBUG:boto:Path: /dkyges-63/obj37 2024-04-12T04:19:58.507 DEBUG:boto:Data: 2024-04-12T04:19:58.507 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.508 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.508 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.508 DEBUG:boto:Params: {} 2024-04-12T04:19:58.508 DEBUG:boto:Token: None 2024-04-12T04:19:58.508 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj37 2024-04-12T04:19:58.508 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ALZO7ml+OOfWCKh9Wgf/HVOA4MY= 2024-04-12T04:19:58.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ALZO7ml+OOfWCKh9Wgf/HVOA4MY=', 'Content-Length': '0'} 2024-04-12T04:19:58.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005eb5d014ea0a4cd0-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2024-04-12T04:19:58.510 DEBUG:boto:path=/dkyges-63/obj37 2024-04-12T04:19:58.510 DEBUG:boto:auth_path=/dkyges-63/obj37 2024-04-12T04:19:58.510 DEBUG:boto:Method: GET 2024-04-12T04:19:58.510 DEBUG:boto:Path: /dkyges-63/obj37 2024-04-12T04:19:58.511 DEBUG:boto:Data: 2024-04-12T04:19:58.511 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.511 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.511 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.511 DEBUG:boto:Params: {} 2024-04-12T04:19:58.511 DEBUG:boto:Token: None 2024-04-12T04:19:58.511 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj37 2024-04-12T04:19:58.511 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ye61ORbpaRPLpDxk/nakWLHMXFg= 2024-04-12T04:19:58.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ye61ORbpaRPLpDxk/nakWLHMXFg=', 'Content-Length': '0'} 2024-04-12T04:19:58.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000065b3bfd0612164d-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.513 DEBUG:boto:path=/dkyges-63/obj37 2024-04-12T04:19:58.513 DEBUG:boto:auth_path=/dkyges-63/obj37 2024-04-12T04:19:58.514 DEBUG:boto:Method: GET 2024-04-12T04:19:58.514 DEBUG:boto:Path: /dkyges-63/obj37 2024-04-12T04:19:58.514 DEBUG:boto:Data: 2024-04-12T04:19:58.514 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.514 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.514 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.514 DEBUG:boto:Params: {} 2024-04-12T04:19:58.514 DEBUG:boto:Token: None 2024-04-12T04:19:58.515 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj37 2024-04-12T04:19:58.515 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ye61ORbpaRPLpDxk/nakWLHMXFg= 2024-04-12T04:19:58.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ye61ORbpaRPLpDxk/nakWLHMXFg=', 'Content-Length': '0'} 2024-04-12T04:19:58.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000990d54d990483be7-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.558 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.558 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.559 DEBUG:boto:path=/dkyges-63/?prefix=obj37 2024-04-12T04:19:58.559 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj37 2024-04-12T04:19:58.559 DEBUG:boto:Method: GET 2024-04-12T04:19:58.559 DEBUG:boto:Path: /dkyges-63/?prefix=obj37 2024-04-12T04:19:58.559 DEBUG:boto:Data: 2024-04-12T04:19:58.559 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.559 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.559 DEBUG:boto:Params: {} 2024-04-12T04:19:58.559 DEBUG:boto:Token: None 2024-04-12T04:19:58.560 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.560 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f39312fe2b3f4366-006618b66e-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.562 DEBUG:boto:b'dkyges-63obj371000falseobj372024-04-12T04:15:49.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.563 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.563 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.563 DEBUG:boto:path=/dkyges-63/?prefix=obj37 2024-04-12T04:19:58.563 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj37 2024-04-12T04:19:58.563 DEBUG:boto:Method: GET 2024-04-12T04:19:58.563 DEBUG:boto:Path: /dkyges-63/?prefix=obj37 2024-04-12T04:19:58.563 DEBUG:boto:Data: 2024-04-12T04:19:58.564 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.564 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.564 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.564 DEBUG:boto:Params: {} 2024-04-12T04:19:58.564 DEBUG:boto:Token: None 2024-04-12T04:19:58.564 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.564 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027631c551a265b44-006618b66e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.569 DEBUG:boto:b'dkyges-63obj371000falseobj372024-04-12T04:15:49.595Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:19:58.569 DEBUG:boto:path=/dkyges-63/obj38 2024-04-12T04:19:58.569 DEBUG:boto:auth_path=/dkyges-63/obj38 2024-04-12T04:19:58.569 DEBUG:boto:path=/dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.570 DEBUG:boto:auth_path=/dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.570 DEBUG:boto:Method: GET 2024-04-12T04:19:58.570 DEBUG:boto:Path: /dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.570 DEBUG:boto:Data: 2024-04-12T04:19:58.570 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.570 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.570 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.570 DEBUG:boto:Params: {} 2024-04-12T04:19:58.570 DEBUG:boto:Token: None 2024-04-12T04:19:58.570 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.571 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q7OvRWf8Cj8QlGI7xbRb7n5HebI= 2024-04-12T04:19:58.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q7OvRWf8Cj8QlGI7xbRb7n5HebI=', 'Content-Length': '0'} 2024-04-12T04:19:58.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000784e9727fe12ed35-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.614 DEBUG:boto:path=/dkyges-63/obj38 2024-04-12T04:19:58.614 DEBUG:boto:auth_path=/dkyges-63/obj38 2024-04-12T04:19:58.615 DEBUG:boto:path=/dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.615 DEBUG:boto:auth_path=/dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.615 DEBUG:boto:Method: GET 2024-04-12T04:19:58.615 DEBUG:boto:Path: /dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.615 DEBUG:boto:Data: 2024-04-12T04:19:58.615 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.615 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.615 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.615 DEBUG:boto:Params: {} 2024-04-12T04:19:58.615 DEBUG:boto:Token: None 2024-04-12T04:19:58.616 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.616 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q7OvRWf8Cj8QlGI7xbRb7n5HebI= 2024-04-12T04:19:58.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q7OvRWf8Cj8QlGI7xbRb7n5HebI=', 'Content-Length': '0'} 2024-04-12T04:19:58.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000af3aec85ce16c57-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.618 DEBUG:boto:path=/dkyges-63/obj38 2024-04-12T04:19:58.618 DEBUG:boto:auth_path=/dkyges-63/obj38 2024-04-12T04:19:58.618 DEBUG:boto:path=/dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.618 DEBUG:boto:auth_path=/dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.618 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.618 DEBUG:boto:Path: /dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.619 DEBUG:boto:Data: 2024-04-12T04:19:58.619 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.619 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.619 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.619 DEBUG:boto:Params: {} 2024-04-12T04:19:58.619 DEBUG:boto:Token: None 2024-04-12T04:19:58.619 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.619 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gr6Cwt4fFmtB4zWpda6vpIn2L6g= 2024-04-12T04:19:58.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gr6Cwt4fFmtB4zWpda6vpIn2L6g=', 'Content-Length': '0'} 2024-04-12T04:19:58.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032fba16f00592ead-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.621 DEBUG:boto:path=/dkyges-63/obj38 2024-04-12T04:19:58.622 DEBUG:boto:auth_path=/dkyges-63/obj38 2024-04-12T04:19:58.622 DEBUG:boto:path=/dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.622 DEBUG:boto:auth_path=/dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.622 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.622 DEBUG:boto:Path: /dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.622 DEBUG:boto:Data: 2024-04-12T04:19:58.622 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.622 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.622 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.622 DEBUG:boto:Params: {} 2024-04-12T04:19:58.622 DEBUG:boto:Token: None 2024-04-12T04:19:58.623 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.623 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gr6Cwt4fFmtB4zWpda6vpIn2L6g= 2024-04-12T04:19:58.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gr6Cwt4fFmtB4zWpda6vpIn2L6g=', 'Content-Length': '0'} 2024-04-12T04:19:58.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffb3619a34d01e94-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:19:58.625 DEBUG:boto:path=/dkyges-63/obj38 2024-04-12T04:19:58.625 DEBUG:boto:auth_path=/dkyges-63/obj38 2024-04-12T04:19:58.625 DEBUG:boto:path=/dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.626 DEBUG:boto:auth_path=/dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.626 DEBUG:boto:Method: GET 2024-04-12T04:19:58.626 DEBUG:boto:Path: /dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.626 DEBUG:boto:Data: 2024-04-12T04:19:58.626 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.626 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.626 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.626 DEBUG:boto:Params: {} 2024-04-12T04:19:58.626 DEBUG:boto:Token: None 2024-04-12T04:19:58.627 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.627 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q7OvRWf8Cj8QlGI7xbRb7n5HebI= 2024-04-12T04:19:58.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q7OvRWf8Cj8QlGI7xbRb7n5HebI=', 'Content-Length': '0'} 2024-04-12T04:19:58.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082c85345d086f447-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.629 DEBUG:boto:path=/dkyges-63/obj38 2024-04-12T04:19:58.629 DEBUG:boto:auth_path=/dkyges-63/obj38 2024-04-12T04:19:58.629 DEBUG:boto:path=/dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.629 DEBUG:boto:auth_path=/dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.629 DEBUG:boto:Method: GET 2024-04-12T04:19:58.629 DEBUG:boto:Path: /dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.629 DEBUG:boto:Data: 2024-04-12T04:19:58.630 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.630 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.630 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.630 DEBUG:boto:Params: {} 2024-04-12T04:19:58.630 DEBUG:boto:Token: None 2024-04-12T04:19:58.630 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj38?versionId=null 2024-04-12T04:19:58.630 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q7OvRWf8Cj8QlGI7xbRb7n5HebI= 2024-04-12T04:19:58.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q7OvRWf8Cj8QlGI7xbRb7n5HebI=', 'Content-Length': '0'} 2024-04-12T04:19:58.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce2444f825ca3b9f-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.633 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.633 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.633 DEBUG:boto:path=/dkyges-63/?prefix=obj38 2024-04-12T04:19:58.633 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj38 2024-04-12T04:19:58.633 DEBUG:boto:Method: GET 2024-04-12T04:19:58.633 DEBUG:boto:Path: /dkyges-63/?prefix=obj38 2024-04-12T04:19:58.633 DEBUG:boto:Data: 2024-04-12T04:19:58.633 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.633 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.633 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.634 DEBUG:boto:Params: {} 2024-04-12T04:19:58.634 DEBUG:boto:Token: None 2024-04-12T04:19:58.634 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.634 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e7d9165b1b3aad2-006618b66e-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.637 DEBUG:boto:b'dkyges-63obj381000falseobj382024-04-12T04:15:48.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.637 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.637 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.637 DEBUG:boto:path=/dkyges-63/?prefix=obj38 2024-04-12T04:19:58.637 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj38 2024-04-12T04:19:58.638 DEBUG:boto:Method: GET 2024-04-12T04:19:58.638 DEBUG:boto:Path: /dkyges-63/?prefix=obj38 2024-04-12T04:19:58.638 DEBUG:boto:Data: 2024-04-12T04:19:58.638 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.638 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.638 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.638 DEBUG:boto:Params: {} 2024-04-12T04:19:58.638 DEBUG:boto:Token: None 2024-04-12T04:19:58.638 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.638 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097a1ada5109a07c5-006618b66e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.644 DEBUG:boto:b'dkyges-63obj381000falseobj382024-04-12T04:15:48.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.644 DEBUG:boto:path=/dkyges-63/obj38 2024-04-12T04:19:58.644 DEBUG:boto:auth_path=/dkyges-63/obj38 2024-04-12T04:19:58.645 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.645 DEBUG:boto:Path: /dkyges-63/obj38 2024-04-12T04:19:58.645 DEBUG:boto:Data: 2024-04-12T04:19:58.645 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.645 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.645 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.645 DEBUG:boto:Params: {} 2024-04-12T04:19:58.645 DEBUG:boto:Token: None 2024-04-12T04:19:58.645 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj38 2024-04-12T04:19:58.645 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z0MKSom4koCHZP4ORmraPEN7JfE= 2024-04-12T04:19:58.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z0MKSom4koCHZP4ORmraPEN7JfE=', 'Content-Length': '0'} 2024-04-12T04:19:58.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000093e05aee8d67cb7-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.648 DEBUG:boto:path=/dkyges-63/obj38 2024-04-12T04:19:58.648 DEBUG:boto:auth_path=/dkyges-63/obj38 2024-04-12T04:19:58.648 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.648 DEBUG:boto:Path: /dkyges-63/obj38 2024-04-12T04:19:58.648 DEBUG:boto:Data: 2024-04-12T04:19:58.648 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.648 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.648 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.648 DEBUG:boto:Params: {} 2024-04-12T04:19:58.648 DEBUG:boto:Token: None 2024-04-12T04:19:58.649 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj38 2024-04-12T04:19:58.649 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:z0MKSom4koCHZP4ORmraPEN7JfE= 2024-04-12T04:19:58.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z0MKSom4koCHZP4ORmraPEN7JfE=', 'Content-Length': '0'} 2024-04-12T04:19:58.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b785a6fe71194a80-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2024-04-12T04:19:58.651 DEBUG:boto:path=/dkyges-63/obj38 2024-04-12T04:19:58.651 DEBUG:boto:auth_path=/dkyges-63/obj38 2024-04-12T04:19:58.651 DEBUG:boto:Method: GET 2024-04-12T04:19:58.652 DEBUG:boto:Path: /dkyges-63/obj38 2024-04-12T04:19:58.652 DEBUG:boto:Data: 2024-04-12T04:19:58.652 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.652 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.652 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.652 DEBUG:boto:Params: {} 2024-04-12T04:19:58.652 DEBUG:boto:Token: None 2024-04-12T04:19:58.652 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj38 2024-04-12T04:19:58.652 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bGd/lhiju7Cl4poYVIfP3Vpm5hA= 2024-04-12T04:19:58.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bGd/lhiju7Cl4poYVIfP3Vpm5hA=', 'Content-Length': '0'} 2024-04-12T04:19:58.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017cfd42e3ba77b04-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.654 DEBUG:boto:path=/dkyges-63/obj38 2024-04-12T04:19:58.655 DEBUG:boto:auth_path=/dkyges-63/obj38 2024-04-12T04:19:58.655 DEBUG:boto:Method: GET 2024-04-12T04:19:58.655 DEBUG:boto:Path: /dkyges-63/obj38 2024-04-12T04:19:58.655 DEBUG:boto:Data: 2024-04-12T04:19:58.655 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.655 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.655 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.655 DEBUG:boto:Params: {} 2024-04-12T04:19:58.655 DEBUG:boto:Token: None 2024-04-12T04:19:58.656 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj38 2024-04-12T04:19:58.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bGd/lhiju7Cl4poYVIfP3Vpm5hA= 2024-04-12T04:19:58.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bGd/lhiju7Cl4poYVIfP3Vpm5hA=', 'Content-Length': '0'} 2024-04-12T04:19:58.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e4c0c668c545971-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.698 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.699 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.699 DEBUG:boto:path=/dkyges-63/?prefix=obj38 2024-04-12T04:19:58.699 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj38 2024-04-12T04:19:58.699 DEBUG:boto:Method: GET 2024-04-12T04:19:58.699 DEBUG:boto:Path: /dkyges-63/?prefix=obj38 2024-04-12T04:19:58.699 DEBUG:boto:Data: 2024-04-12T04:19:58.699 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.699 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.699 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.700 DEBUG:boto:Params: {} 2024-04-12T04:19:58.700 DEBUG:boto:Token: None 2024-04-12T04:19:58.700 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.700 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.703 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057e24f5736061689-006618b66e-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.703 DEBUG:boto:b'dkyges-63obj381000falseobj382024-04-12T04:15:48.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.703 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.703 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.703 DEBUG:boto:path=/dkyges-63/?prefix=obj38 2024-04-12T04:19:58.704 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj38 2024-04-12T04:19:58.704 DEBUG:boto:Method: GET 2024-04-12T04:19:58.704 DEBUG:boto:Path: /dkyges-63/?prefix=obj38 2024-04-12T04:19:58.704 DEBUG:boto:Data: 2024-04-12T04:19:58.704 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.704 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.704 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.704 DEBUG:boto:Params: {} 2024-04-12T04:19:58.704 DEBUG:boto:Token: None 2024-04-12T04:19:58.704 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.704 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000901ff43c0339acb8-006618b66e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.709 DEBUG:boto:b'dkyges-63obj381000falseobj382024-04-12T04:15:48.128Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:19:58.709 DEBUG:boto:path=/dkyges-63/obj44 2024-04-12T04:19:58.710 DEBUG:boto:auth_path=/dkyges-63/obj44 2024-04-12T04:19:58.710 DEBUG:boto:path=/dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.710 DEBUG:boto:auth_path=/dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.710 DEBUG:boto:Method: GET 2024-04-12T04:19:58.710 DEBUG:boto:Path: /dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.710 DEBUG:boto:Data: 2024-04-12T04:19:58.710 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.710 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.710 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.710 DEBUG:boto:Params: {} 2024-04-12T04:19:58.710 DEBUG:boto:Token: None 2024-04-12T04:19:58.711 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.711 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aF4OrACKZS7uMbIiwCyEeZvKgQ0= 2024-04-12T04:19:58.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aF4OrACKZS7uMbIiwCyEeZvKgQ0=', 'Content-Length': '0'} 2024-04-12T04:19:58.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d20101bf383adad-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.754 DEBUG:boto:path=/dkyges-63/obj44 2024-04-12T04:19:58.755 DEBUG:boto:auth_path=/dkyges-63/obj44 2024-04-12T04:19:58.755 DEBUG:boto:path=/dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.755 DEBUG:boto:auth_path=/dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.755 DEBUG:boto:Method: GET 2024-04-12T04:19:58.755 DEBUG:boto:Path: /dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.755 DEBUG:boto:Data: 2024-04-12T04:19:58.755 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.755 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.755 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.755 DEBUG:boto:Params: {} 2024-04-12T04:19:58.756 DEBUG:boto:Token: None 2024-04-12T04:19:58.756 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.756 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aF4OrACKZS7uMbIiwCyEeZvKgQ0= 2024-04-12T04:19:58.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aF4OrACKZS7uMbIiwCyEeZvKgQ0=', 'Content-Length': '0'} 2024-04-12T04:19:58.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002be2c1f638cda36b-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.758 DEBUG:boto:path=/dkyges-63/obj44 2024-04-12T04:19:58.758 DEBUG:boto:auth_path=/dkyges-63/obj44 2024-04-12T04:19:58.758 DEBUG:boto:path=/dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.758 DEBUG:boto:auth_path=/dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.759 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.759 DEBUG:boto:Path: /dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.759 DEBUG:boto:Data: 2024-04-12T04:19:58.759 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.759 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.759 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.759 DEBUG:boto:Params: {} 2024-04-12T04:19:58.759 DEBUG:boto:Token: None 2024-04-12T04:19:58.759 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.759 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UrNUHNmBM7cjYabv9pFlwjn/T2A= 2024-04-12T04:19:58.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UrNUHNmBM7cjYabv9pFlwjn/T2A=', 'Content-Length': '0'} 2024-04-12T04:19:58.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af6ffadd83c09dab-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.761 DEBUG:boto:path=/dkyges-63/obj44 2024-04-12T04:19:58.762 DEBUG:boto:auth_path=/dkyges-63/obj44 2024-04-12T04:19:58.762 DEBUG:boto:path=/dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.762 DEBUG:boto:auth_path=/dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.762 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.762 DEBUG:boto:Path: /dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.762 DEBUG:boto:Data: 2024-04-12T04:19:58.762 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.762 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.762 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.762 DEBUG:boto:Params: {} 2024-04-12T04:19:58.762 DEBUG:boto:Token: None 2024-04-12T04:19:58.763 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.763 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UrNUHNmBM7cjYabv9pFlwjn/T2A= 2024-04-12T04:19:58.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UrNUHNmBM7cjYabv9pFlwjn/T2A=', 'Content-Length': '0'} 2024-04-12T04:19:58.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c685edeceb7a69a1-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:19:58.765 DEBUG:boto:path=/dkyges-63/obj44 2024-04-12T04:19:58.765 DEBUG:boto:auth_path=/dkyges-63/obj44 2024-04-12T04:19:58.765 DEBUG:boto:path=/dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.765 DEBUG:boto:auth_path=/dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.765 DEBUG:boto:Method: GET 2024-04-12T04:19:58.765 DEBUG:boto:Path: /dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.766 DEBUG:boto:Data: 2024-04-12T04:19:58.766 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.766 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.766 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.766 DEBUG:boto:Params: {} 2024-04-12T04:19:58.766 DEBUG:boto:Token: None 2024-04-12T04:19:58.766 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.766 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aF4OrACKZS7uMbIiwCyEeZvKgQ0= 2024-04-12T04:19:58.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aF4OrACKZS7uMbIiwCyEeZvKgQ0=', 'Content-Length': '0'} 2024-04-12T04:19:58.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed7ee72a16fd15e2-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.768 DEBUG:boto:path=/dkyges-63/obj44 2024-04-12T04:19:58.768 DEBUG:boto:auth_path=/dkyges-63/obj44 2024-04-12T04:19:58.768 DEBUG:boto:path=/dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.769 DEBUG:boto:auth_path=/dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.769 DEBUG:boto:Method: GET 2024-04-12T04:19:58.769 DEBUG:boto:Path: /dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.769 DEBUG:boto:Data: 2024-04-12T04:19:58.769 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.769 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.769 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.769 DEBUG:boto:Params: {} 2024-04-12T04:19:58.769 DEBUG:boto:Token: None 2024-04-12T04:19:58.769 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj44?versionId=null 2024-04-12T04:19:58.770 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aF4OrACKZS7uMbIiwCyEeZvKgQ0= 2024-04-12T04:19:58.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aF4OrACKZS7uMbIiwCyEeZvKgQ0=', 'Content-Length': '0'} 2024-04-12T04:19:58.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054d1bdc458bdc5e6-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.772 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.772 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.772 DEBUG:boto:path=/dkyges-63/?prefix=obj44 2024-04-12T04:19:58.772 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj44 2024-04-12T04:19:58.772 DEBUG:boto:Method: GET 2024-04-12T04:19:58.772 DEBUG:boto:Path: /dkyges-63/?prefix=obj44 2024-04-12T04:19:58.772 DEBUG:boto:Data: 2024-04-12T04:19:58.772 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.772 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.772 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.772 DEBUG:boto:Params: {} 2024-04-12T04:19:58.773 DEBUG:boto:Token: None 2024-04-12T04:19:58.773 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.773 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.775 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001039c6164111184c-006618b66e-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.776 DEBUG:boto:b'dkyges-63obj441000falseobj442024-04-12T04:15:47.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.776 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.776 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.776 DEBUG:boto:path=/dkyges-63/?prefix=obj44 2024-04-12T04:19:58.776 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj44 2024-04-12T04:19:58.776 DEBUG:boto:Method: GET 2024-04-12T04:19:58.776 DEBUG:boto:Path: /dkyges-63/?prefix=obj44 2024-04-12T04:19:58.777 DEBUG:boto:Data: 2024-04-12T04:19:58.777 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.777 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.777 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.777 DEBUG:boto:Params: {} 2024-04-12T04:19:58.777 DEBUG:boto:Token: None 2024-04-12T04:19:58.777 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.777 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e560c831d46ef039-006618b66e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.781 DEBUG:boto:b'dkyges-63obj441000falseobj442024-04-12T04:15:47.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.782 DEBUG:boto:path=/dkyges-63/obj44 2024-04-12T04:19:58.782 DEBUG:boto:auth_path=/dkyges-63/obj44 2024-04-12T04:19:58.782 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.782 DEBUG:boto:Path: /dkyges-63/obj44 2024-04-12T04:19:58.782 DEBUG:boto:Data: 2024-04-12T04:19:58.782 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.782 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.782 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.782 DEBUG:boto:Params: {} 2024-04-12T04:19:58.783 DEBUG:boto:Token: None 2024-04-12T04:19:58.783 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj44 2024-04-12T04:19:58.783 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IP+C0o8u/FMj0npGNhOs3OTw1eA= 2024-04-12T04:19:58.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IP+C0o8u/FMj0npGNhOs3OTw1eA=', 'Content-Length': '0'} 2024-04-12T04:19:58.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006266553ccf344629-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.785 DEBUG:boto:path=/dkyges-63/obj44 2024-04-12T04:19:58.785 DEBUG:boto:auth_path=/dkyges-63/obj44 2024-04-12T04:19:58.785 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.785 DEBUG:boto:Path: /dkyges-63/obj44 2024-04-12T04:19:58.785 DEBUG:boto:Data: 2024-04-12T04:19:58.785 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.785 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.785 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.786 DEBUG:boto:Params: {} 2024-04-12T04:19:58.786 DEBUG:boto:Token: None 2024-04-12T04:19:58.786 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj44 2024-04-12T04:19:58.786 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:IP+C0o8u/FMj0npGNhOs3OTw1eA= 2024-04-12T04:19:58.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:IP+C0o8u/FMj0npGNhOs3OTw1eA=', 'Content-Length': '0'} 2024-04-12T04:19:58.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070c835488d253c62-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:19:58.788 DEBUG:boto:path=/dkyges-63/obj44 2024-04-12T04:19:58.789 DEBUG:boto:auth_path=/dkyges-63/obj44 2024-04-12T04:19:58.789 DEBUG:boto:Method: GET 2024-04-12T04:19:58.789 DEBUG:boto:Path: /dkyges-63/obj44 2024-04-12T04:19:58.789 DEBUG:boto:Data: 2024-04-12T04:19:58.789 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.789 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.789 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.789 DEBUG:boto:Params: {} 2024-04-12T04:19:58.789 DEBUG:boto:Token: None 2024-04-12T04:19:58.789 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj44 2024-04-12T04:19:58.790 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bieuLL4B67QYCInyasaClX+7/Jc= 2024-04-12T04:19:58.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bieuLL4B67QYCInyasaClX+7/Jc=', 'Content-Length': '0'} 2024-04-12T04:19:58.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c64416a0a74607c0-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.792 DEBUG:boto:path=/dkyges-63/obj44 2024-04-12T04:19:58.792 DEBUG:boto:auth_path=/dkyges-63/obj44 2024-04-12T04:19:58.792 DEBUG:boto:Method: GET 2024-04-12T04:19:58.792 DEBUG:boto:Path: /dkyges-63/obj44 2024-04-12T04:19:58.792 DEBUG:boto:Data: 2024-04-12T04:19:58.792 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.792 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.792 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.792 DEBUG:boto:Params: {} 2024-04-12T04:19:58.793 DEBUG:boto:Token: None 2024-04-12T04:19:58.793 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj44 2024-04-12T04:19:58.793 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bieuLL4B67QYCInyasaClX+7/Jc= 2024-04-12T04:19:58.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bieuLL4B67QYCInyasaClX+7/Jc=', 'Content-Length': '0'} 2024-04-12T04:19:58.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b733b4b8ea6fb195-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.838 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.839 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.839 DEBUG:boto:path=/dkyges-63/?prefix=obj44 2024-04-12T04:19:58.839 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj44 2024-04-12T04:19:58.839 DEBUG:boto:Method: GET 2024-04-12T04:19:58.839 DEBUG:boto:Path: /dkyges-63/?prefix=obj44 2024-04-12T04:19:58.839 DEBUG:boto:Data: 2024-04-12T04:19:58.839 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.839 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.839 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.839 DEBUG:boto:Params: {} 2024-04-12T04:19:58.840 DEBUG:boto:Token: None 2024-04-12T04:19:58.840 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.840 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ff6bed320fadd14-006618b66e-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.843 DEBUG:boto:b'dkyges-63obj441000falseobj442024-04-12T04:15:47.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.843 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.843 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.843 DEBUG:boto:path=/dkyges-63/?prefix=obj44 2024-04-12T04:19:58.844 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj44 2024-04-12T04:19:58.844 DEBUG:boto:Method: GET 2024-04-12T04:19:58.844 DEBUG:boto:Path: /dkyges-63/?prefix=obj44 2024-04-12T04:19:58.844 DEBUG:boto:Data: 2024-04-12T04:19:58.844 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.844 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.844 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.844 DEBUG:boto:Params: {} 2024-04-12T04:19:58.844 DEBUG:boto:Token: None 2024-04-12T04:19:58.844 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.845 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034acb7161b9de7af-006618b66e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.849 DEBUG:boto:b'dkyges-63obj441000falseobj442024-04-12T04:15:47.535Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:19:58.850 DEBUG:boto:path=/dkyges-63/obj46 2024-04-12T04:19:58.850 DEBUG:boto:auth_path=/dkyges-63/obj46 2024-04-12T04:19:58.850 DEBUG:boto:path=/dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.850 DEBUG:boto:auth_path=/dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.850 DEBUG:boto:Method: GET 2024-04-12T04:19:58.850 DEBUG:boto:Path: /dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.850 DEBUG:boto:Data: 2024-04-12T04:19:58.850 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.850 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.850 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.850 DEBUG:boto:Params: {} 2024-04-12T04:19:58.851 DEBUG:boto:Token: None 2024-04-12T04:19:58.851 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.851 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kWj7+5AHn2SGlmpNCJitwzTXnig= 2024-04-12T04:19:58.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kWj7+5AHn2SGlmpNCJitwzTXnig=', 'Content-Length': '0'} 2024-04-12T04:19:58.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067ae7e64c31fa924-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.894 DEBUG:boto:path=/dkyges-63/obj46 2024-04-12T04:19:58.894 DEBUG:boto:auth_path=/dkyges-63/obj46 2024-04-12T04:19:58.894 DEBUG:boto:path=/dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.895 DEBUG:boto:auth_path=/dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.895 DEBUG:boto:Method: GET 2024-04-12T04:19:58.895 DEBUG:boto:Path: /dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.895 DEBUG:boto:Data: 2024-04-12T04:19:58.895 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.895 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.895 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.895 DEBUG:boto:Params: {} 2024-04-12T04:19:58.895 DEBUG:boto:Token: None 2024-04-12T04:19:58.895 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.896 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kWj7+5AHn2SGlmpNCJitwzTXnig= 2024-04-12T04:19:58.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kWj7+5AHn2SGlmpNCJitwzTXnig=', 'Content-Length': '0'} 2024-04-12T04:19:58.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c84b57eff3ca9950-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.898 DEBUG:boto:path=/dkyges-63/obj46 2024-04-12T04:19:58.898 DEBUG:boto:auth_path=/dkyges-63/obj46 2024-04-12T04:19:58.898 DEBUG:boto:path=/dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.898 DEBUG:boto:auth_path=/dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.898 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.898 DEBUG:boto:Path: /dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.898 DEBUG:boto:Data: 2024-04-12T04:19:58.898 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.898 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.899 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.899 DEBUG:boto:Params: {} 2024-04-12T04:19:58.899 DEBUG:boto:Token: None 2024-04-12T04:19:58.899 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.899 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aYEFUF91MqRcid+j/5U0M/NRENs= 2024-04-12T04:19:58.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aYEFUF91MqRcid+j/5U0M/NRENs=', 'Content-Length': '0'} 2024-04-12T04:19:58.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000620ca71e512e557c-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.901 DEBUG:boto:path=/dkyges-63/obj46 2024-04-12T04:19:58.901 DEBUG:boto:auth_path=/dkyges-63/obj46 2024-04-12T04:19:58.901 DEBUG:boto:path=/dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.901 DEBUG:boto:auth_path=/dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.902 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.902 DEBUG:boto:Path: /dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.902 DEBUG:boto:Data: 2024-04-12T04:19:58.902 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.902 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.902 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.902 DEBUG:boto:Params: {} 2024-04-12T04:19:58.902 DEBUG:boto:Token: None 2024-04-12T04:19:58.902 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.902 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aYEFUF91MqRcid+j/5U0M/NRENs= 2024-04-12T04:19:58.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aYEFUF91MqRcid+j/5U0M/NRENs=', 'Content-Length': '0'} 2024-04-12T04:19:58.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b856f93dcabacb8-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:19:58.905 DEBUG:boto:path=/dkyges-63/obj46 2024-04-12T04:19:58.905 DEBUG:boto:auth_path=/dkyges-63/obj46 2024-04-12T04:19:58.905 DEBUG:boto:path=/dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.905 DEBUG:boto:auth_path=/dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.905 DEBUG:boto:Method: GET 2024-04-12T04:19:58.905 DEBUG:boto:Path: /dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.905 DEBUG:boto:Data: 2024-04-12T04:19:58.905 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.905 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.905 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.906 DEBUG:boto:Params: {} 2024-04-12T04:19:58.906 DEBUG:boto:Token: None 2024-04-12T04:19:58.906 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.906 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kWj7+5AHn2SGlmpNCJitwzTXnig= 2024-04-12T04:19:58.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kWj7+5AHn2SGlmpNCJitwzTXnig=', 'Content-Length': '0'} 2024-04-12T04:19:58.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e9f13ede12f3fea-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.908 DEBUG:boto:path=/dkyges-63/obj46 2024-04-12T04:19:58.908 DEBUG:boto:auth_path=/dkyges-63/obj46 2024-04-12T04:19:58.908 DEBUG:boto:path=/dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.908 DEBUG:boto:auth_path=/dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.909 DEBUG:boto:Method: GET 2024-04-12T04:19:58.909 DEBUG:boto:Path: /dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.909 DEBUG:boto:Data: 2024-04-12T04:19:58.909 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.909 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.909 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.909 DEBUG:boto:Params: {} 2024-04-12T04:19:58.909 DEBUG:boto:Token: None 2024-04-12T04:19:58.909 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj46?versionId=null 2024-04-12T04:19:58.909 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kWj7+5AHn2SGlmpNCJitwzTXnig= 2024-04-12T04:19:58.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kWj7+5AHn2SGlmpNCJitwzTXnig=', 'Content-Length': '0'} 2024-04-12T04:19:58.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b26e6e04963adf03-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.911 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.912 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.912 DEBUG:boto:path=/dkyges-63/?prefix=obj46 2024-04-12T04:19:58.912 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj46 2024-04-12T04:19:58.912 DEBUG:boto:Method: GET 2024-04-12T04:19:58.912 DEBUG:boto:Path: /dkyges-63/?prefix=obj46 2024-04-12T04:19:58.912 DEBUG:boto:Data: 2024-04-12T04:19:58.912 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.912 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.912 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.912 DEBUG:boto:Params: {} 2024-04-12T04:19:58.912 DEBUG:boto:Token: None 2024-04-12T04:19:58.912 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.913 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3c16decd12d879d-006618b66e-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.915 DEBUG:boto:b'dkyges-63obj461000falseobj462024-04-12T04:15:48.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.916 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.916 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.916 DEBUG:boto:path=/dkyges-63/?prefix=obj46 2024-04-12T04:19:58.916 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj46 2024-04-12T04:19:58.916 DEBUG:boto:Method: GET 2024-04-12T04:19:58.916 DEBUG:boto:Path: /dkyges-63/?prefix=obj46 2024-04-12T04:19:58.916 DEBUG:boto:Data: 2024-04-12T04:19:58.916 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.916 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.917 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.917 DEBUG:boto:Params: {} 2024-04-12T04:19:58.917 DEBUG:boto:Token: None 2024-04-12T04:19:58.917 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.917 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcab554d7630c3bd-006618b66e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.922 DEBUG:boto:b'dkyges-63obj461000falseobj462024-04-12T04:15:48.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.922 DEBUG:boto:path=/dkyges-63/obj46 2024-04-12T04:19:58.922 DEBUG:boto:auth_path=/dkyges-63/obj46 2024-04-12T04:19:58.922 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.922 DEBUG:boto:Path: /dkyges-63/obj46 2024-04-12T04:19:58.922 DEBUG:boto:Data: 2024-04-12T04:19:58.923 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.923 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.923 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.923 DEBUG:boto:Params: {} 2024-04-12T04:19:58.923 DEBUG:boto:Token: None 2024-04-12T04:19:58.923 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj46 2024-04-12T04:19:58.923 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FY5jA48uulA+0AIc69yvVfDcVek= 2024-04-12T04:19:58.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FY5jA48uulA+0AIc69yvVfDcVek=', 'Content-Length': '0'} 2024-04-12T04:19:58.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc4a9e311c88e3bb-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.925 DEBUG:boto:path=/dkyges-63/obj46 2024-04-12T04:19:58.925 DEBUG:boto:auth_path=/dkyges-63/obj46 2024-04-12T04:19:58.925 DEBUG:boto:Method: HEAD 2024-04-12T04:19:58.925 DEBUG:boto:Path: /dkyges-63/obj46 2024-04-12T04:19:58.926 DEBUG:boto:Data: 2024-04-12T04:19:58.926 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.926 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.926 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.926 DEBUG:boto:Params: {} 2024-04-12T04:19:58.926 DEBUG:boto:Token: None 2024-04-12T04:19:58.926 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj46 2024-04-12T04:19:58.926 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FY5jA48uulA+0AIc69yvVfDcVek= 2024-04-12T04:19:58.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FY5jA48uulA+0AIc69yvVfDcVek=', 'Content-Length': '0'} 2024-04-12T04:19:58.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fcd9c6de87df141-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2024-04-12T04:19:58.929 DEBUG:boto:path=/dkyges-63/obj46 2024-04-12T04:19:58.929 DEBUG:boto:auth_path=/dkyges-63/obj46 2024-04-12T04:19:58.929 DEBUG:boto:Method: GET 2024-04-12T04:19:58.929 DEBUG:boto:Path: /dkyges-63/obj46 2024-04-12T04:19:58.929 DEBUG:boto:Data: 2024-04-12T04:19:58.929 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.929 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.930 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.930 DEBUG:boto:Params: {} 2024-04-12T04:19:58.930 DEBUG:boto:Token: None 2024-04-12T04:19:58.930 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj46 2024-04-12T04:19:58.930 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gnQCy4/g5c1afnhVl/cNgA0zV2Q= 2024-04-12T04:19:58.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gnQCy4/g5c1afnhVl/cNgA0zV2Q=', 'Content-Length': '0'} 2024-04-12T04:19:58.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d017bef24dbce52-006618b66e-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.932 DEBUG:boto:path=/dkyges-63/obj46 2024-04-12T04:19:58.932 DEBUG:boto:auth_path=/dkyges-63/obj46 2024-04-12T04:19:58.932 DEBUG:boto:Method: GET 2024-04-12T04:19:58.933 DEBUG:boto:Path: /dkyges-63/obj46 2024-04-12T04:19:58.933 DEBUG:boto:Data: 2024-04-12T04:19:58.933 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.933 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.933 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.933 DEBUG:boto:Params: {} 2024-04-12T04:19:58.933 DEBUG:boto:Token: None 2024-04-12T04:19:58.933 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj46 2024-04-12T04:19:58.933 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gnQCy4/g5c1afnhVl/cNgA0zV2Q= 2024-04-12T04:19:58.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gnQCy4/g5c1afnhVl/cNgA0zV2Q=', 'Content-Length': '0'} 2024-04-12T04:19:58.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ba161e751c083e1-006618b66e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.978 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.979 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.979 DEBUG:boto:path=/dkyges-63/?prefix=obj46 2024-04-12T04:19:58.979 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj46 2024-04-12T04:19:58.979 DEBUG:boto:Method: GET 2024-04-12T04:19:58.980 DEBUG:boto:Path: /dkyges-63/?prefix=obj46 2024-04-12T04:19:58.980 DEBUG:boto:Data: 2024-04-12T04:19:58.980 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.980 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.980 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.980 DEBUG:boto:Params: {} 2024-04-12T04:19:58.980 DEBUG:boto:Token: None 2024-04-12T04:19:58.980 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.980 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd99d81f88136746-006618b66e-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.983 DEBUG:boto:b'dkyges-63obj461000falseobj462024-04-12T04:15:48.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.984 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:58.984 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:58.984 DEBUG:boto:path=/dkyges-63/?prefix=obj46 2024-04-12T04:19:58.984 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj46 2024-04-12T04:19:58.984 DEBUG:boto:Method: GET 2024-04-12T04:19:58.984 DEBUG:boto:Path: /dkyges-63/?prefix=obj46 2024-04-12T04:19:58.984 DEBUG:boto:Data: 2024-04-12T04:19:58.984 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.984 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:58.985 DEBUG:boto:Port: 8000 2024-04-12T04:19:58.985 DEBUG:boto:Params: {} 2024-04-12T04:19:58.985 DEBUG:boto:Token: None 2024-04-12T04:19:58.985 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/ 2024-04-12T04:19:58.985 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno= 2024-04-12T04:19:58.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UiD28xapRP0ye8/6zxCOwqdWMno=', 'Content-Length': '0'} 2024-04-12T04:19:58.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d7569b8590abb2b-006618b66f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:58.990 DEBUG:boto:b'dkyges-63obj461000falseobj462024-04-12T04:15:48.489Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:58.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:19:58.990 DEBUG:boto:path=/dkyges-63/obj51 2024-04-12T04:19:58.991 DEBUG:boto:auth_path=/dkyges-63/obj51 2024-04-12T04:19:58.991 DEBUG:boto:path=/dkyges-63/obj51?versionId=null 2024-04-12T04:19:58.991 DEBUG:boto:auth_path=/dkyges-63/obj51?versionId=null 2024-04-12T04:19:58.991 DEBUG:boto:Method: GET 2024-04-12T04:19:58.991 DEBUG:boto:Path: /dkyges-63/obj51?versionId=null 2024-04-12T04:19:58.991 DEBUG:boto:Data: 2024-04-12T04:19:58.991 DEBUG:boto:Headers: {} 2024-04-12T04:19:58.991 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:58.991 DEBUG:boto:Port: 8001 2024-04-12T04:19:58.991 DEBUG:boto:Params: {} 2024-04-12T04:19:58.991 DEBUG:boto:Token: None 2024-04-12T04:19:58.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:58 GMT /dkyges-63/obj51?versionId=null 2024-04-12T04:19:58.992 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I8SL60RXH4fm9QeKpf80ChMjuYg= 2024-04-12T04:19:58.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I8SL60RXH4fm9QeKpf80ChMjuYg=', 'Content-Length': '0'} 2024-04-12T04:19:58.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffe7903477be654d-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.034 DEBUG:boto:path=/dkyges-63/obj51 2024-04-12T04:19:59.034 DEBUG:boto:auth_path=/dkyges-63/obj51 2024-04-12T04:19:59.035 DEBUG:boto:path=/dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.035 DEBUG:boto:auth_path=/dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.035 DEBUG:boto:Method: GET 2024-04-12T04:19:59.035 DEBUG:boto:Path: /dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.035 DEBUG:boto:Data: 2024-04-12T04:19:59.035 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.035 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.035 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.035 DEBUG:boto:Params: {} 2024-04-12T04:19:59.035 DEBUG:boto:Token: None 2024-04-12T04:19:59.036 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.036 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4I0s7SK1hPcwZWn3tLrFU/PBm34= 2024-04-12T04:19:59.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4I0s7SK1hPcwZWn3tLrFU/PBm34=', 'Content-Length': '0'} 2024-04-12T04:19:59.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b12bd88ca68f68e-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.038 DEBUG:boto:path=/dkyges-63/obj51 2024-04-12T04:19:59.038 DEBUG:boto:auth_path=/dkyges-63/obj51 2024-04-12T04:19:59.038 DEBUG:boto:path=/dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.038 DEBUG:boto:auth_path=/dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.038 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.039 DEBUG:boto:Path: /dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.039 DEBUG:boto:Data: 2024-04-12T04:19:59.039 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.039 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.039 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.039 DEBUG:boto:Params: {} 2024-04-12T04:19:59.039 DEBUG:boto:Token: None 2024-04-12T04:19:59.039 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.039 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iXsP8UzWX5tTnmjBXmvgvDVeg24= 2024-04-12T04:19:59.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iXsP8UzWX5tTnmjBXmvgvDVeg24=', 'Content-Length': '0'} 2024-04-12T04:19:59.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043e10703c5a41dbb-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.041 DEBUG:boto:path=/dkyges-63/obj51 2024-04-12T04:19:59.042 DEBUG:boto:auth_path=/dkyges-63/obj51 2024-04-12T04:19:59.042 DEBUG:boto:path=/dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.042 DEBUG:boto:auth_path=/dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.042 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.042 DEBUG:boto:Path: /dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.042 DEBUG:boto:Data: 2024-04-12T04:19:59.042 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.042 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.042 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.042 DEBUG:boto:Params: {} 2024-04-12T04:19:59.042 DEBUG:boto:Token: None 2024-04-12T04:19:59.043 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.043 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iXsP8UzWX5tTnmjBXmvgvDVeg24= 2024-04-12T04:19:59.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iXsP8UzWX5tTnmjBXmvgvDVeg24=', 'Content-Length': '0'} 2024-04-12T04:19:59.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d13302e4933d2a9f-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:19:59.045 DEBUG:boto:path=/dkyges-63/obj51 2024-04-12T04:19:59.045 DEBUG:boto:auth_path=/dkyges-63/obj51 2024-04-12T04:19:59.045 DEBUG:boto:path=/dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.045 DEBUG:boto:auth_path=/dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.045 DEBUG:boto:Method: GET 2024-04-12T04:19:59.045 DEBUG:boto:Path: /dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.045 DEBUG:boto:Data: 2024-04-12T04:19:59.046 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.046 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.046 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.046 DEBUG:boto:Params: {} 2024-04-12T04:19:59.046 DEBUG:boto:Token: None 2024-04-12T04:19:59.046 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.046 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4I0s7SK1hPcwZWn3tLrFU/PBm34= 2024-04-12T04:19:59.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4I0s7SK1hPcwZWn3tLrFU/PBm34=', 'Content-Length': '0'} 2024-04-12T04:19:59.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5b1a63b2d6deb62-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.048 DEBUG:boto:path=/dkyges-63/obj51 2024-04-12T04:19:59.048 DEBUG:boto:auth_path=/dkyges-63/obj51 2024-04-12T04:19:59.048 DEBUG:boto:path=/dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.049 DEBUG:boto:auth_path=/dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.049 DEBUG:boto:Method: GET 2024-04-12T04:19:59.049 DEBUG:boto:Path: /dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.049 DEBUG:boto:Data: 2024-04-12T04:19:59.049 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.049 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.049 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.049 DEBUG:boto:Params: {} 2024-04-12T04:19:59.049 DEBUG:boto:Token: None 2024-04-12T04:19:59.049 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj51?versionId=null 2024-04-12T04:19:59.050 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4I0s7SK1hPcwZWn3tLrFU/PBm34= 2024-04-12T04:19:59.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4I0s7SK1hPcwZWn3tLrFU/PBm34=', 'Content-Length': '0'} 2024-04-12T04:19:59.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f49af2c4ff08c8fb-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.052 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.052 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.052 DEBUG:boto:path=/dkyges-63/?prefix=obj51 2024-04-12T04:19:59.052 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj51 2024-04-12T04:19:59.052 DEBUG:boto:Method: GET 2024-04-12T04:19:59.052 DEBUG:boto:Path: /dkyges-63/?prefix=obj51 2024-04-12T04:19:59.052 DEBUG:boto:Data: 2024-04-12T04:19:59.052 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.052 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.053 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.053 DEBUG:boto:Params: {} 2024-04-12T04:19:59.053 DEBUG:boto:Token: None 2024-04-12T04:19:59.053 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.053 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d5a74e1a5f78a37-006618b66f-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.056 DEBUG:boto:b'dkyges-63obj511000falseobj512024-04-12T04:15:47.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.056 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.056 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.056 DEBUG:boto:path=/dkyges-63/?prefix=obj51 2024-04-12T04:19:59.056 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj51 2024-04-12T04:19:59.056 DEBUG:boto:Method: GET 2024-04-12T04:19:59.057 DEBUG:boto:Path: /dkyges-63/?prefix=obj51 2024-04-12T04:19:59.057 DEBUG:boto:Data: 2024-04-12T04:19:59.057 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.057 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.057 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.057 DEBUG:boto:Params: {} 2024-04-12T04:19:59.057 DEBUG:boto:Token: None 2024-04-12T04:19:59.057 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.057 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b489d97eba27a216-006618b66f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.062 DEBUG:boto:b'dkyges-63obj511000falseobj512024-04-12T04:15:47.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.062 DEBUG:boto:path=/dkyges-63/obj51 2024-04-12T04:19:59.062 DEBUG:boto:auth_path=/dkyges-63/obj51 2024-04-12T04:19:59.063 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.063 DEBUG:boto:Path: /dkyges-63/obj51 2024-04-12T04:19:59.063 DEBUG:boto:Data: 2024-04-12T04:19:59.063 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.063 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.063 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.063 DEBUG:boto:Params: {} 2024-04-12T04:19:59.063 DEBUG:boto:Token: None 2024-04-12T04:19:59.063 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj51 2024-04-12T04:19:59.063 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ybQh8oZPDMlkyY9t2QwGYoYbgxs= 2024-04-12T04:19:59.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ybQh8oZPDMlkyY9t2QwGYoYbgxs=', 'Content-Length': '0'} 2024-04-12T04:19:59.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000514558767a518fb6-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.066 DEBUG:boto:path=/dkyges-63/obj51 2024-04-12T04:19:59.066 DEBUG:boto:auth_path=/dkyges-63/obj51 2024-04-12T04:19:59.066 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.066 DEBUG:boto:Path: /dkyges-63/obj51 2024-04-12T04:19:59.066 DEBUG:boto:Data: 2024-04-12T04:19:59.066 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.066 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.066 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.066 DEBUG:boto:Params: {} 2024-04-12T04:19:59.066 DEBUG:boto:Token: None 2024-04-12T04:19:59.067 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj51 2024-04-12T04:19:59.067 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ybQh8oZPDMlkyY9t2QwGYoYbgxs= 2024-04-12T04:19:59.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ybQh8oZPDMlkyY9t2QwGYoYbgxs=', 'Content-Length': '0'} 2024-04-12T04:19:59.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc245d8cbe65a40e-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2024-04-12T04:19:59.069 DEBUG:boto:path=/dkyges-63/obj51 2024-04-12T04:19:59.069 DEBUG:boto:auth_path=/dkyges-63/obj51 2024-04-12T04:19:59.069 DEBUG:boto:Method: GET 2024-04-12T04:19:59.069 DEBUG:boto:Path: /dkyges-63/obj51 2024-04-12T04:19:59.069 DEBUG:boto:Data: 2024-04-12T04:19:59.070 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.070 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.070 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.070 DEBUG:boto:Params: {} 2024-04-12T04:19:59.070 DEBUG:boto:Token: None 2024-04-12T04:19:59.070 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj51 2024-04-12T04:19:59.070 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YcuYi+Ykh/G0rgcQFVTY1BzQ37A= 2024-04-12T04:19:59.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YcuYi+Ykh/G0rgcQFVTY1BzQ37A=', 'Content-Length': '0'} 2024-04-12T04:19:59.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004b1cae2563691ca-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.072 DEBUG:boto:path=/dkyges-63/obj51 2024-04-12T04:19:59.073 DEBUG:boto:auth_path=/dkyges-63/obj51 2024-04-12T04:19:59.073 DEBUG:boto:Method: GET 2024-04-12T04:19:59.073 DEBUG:boto:Path: /dkyges-63/obj51 2024-04-12T04:19:59.073 DEBUG:boto:Data: 2024-04-12T04:19:59.073 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.073 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.073 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.073 DEBUG:boto:Params: {} 2024-04-12T04:19:59.073 DEBUG:boto:Token: None 2024-04-12T04:19:59.073 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj51 2024-04-12T04:19:59.074 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:YcuYi+Ykh/G0rgcQFVTY1BzQ37A= 2024-04-12T04:19:59.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YcuYi+Ykh/G0rgcQFVTY1BzQ37A=', 'Content-Length': '0'} 2024-04-12T04:19:59.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d139a8e47c1b26a6-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.076 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.076 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.076 DEBUG:boto:path=/dkyges-63/?prefix=obj51 2024-04-12T04:19:59.076 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj51 2024-04-12T04:19:59.076 DEBUG:boto:Method: GET 2024-04-12T04:19:59.076 DEBUG:boto:Path: /dkyges-63/?prefix=obj51 2024-04-12T04:19:59.076 DEBUG:boto:Data: 2024-04-12T04:19:59.076 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.076 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.076 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.076 DEBUG:boto:Params: {} 2024-04-12T04:19:59.077 DEBUG:boto:Token: None 2024-04-12T04:19:59.077 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.077 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003680bd5222186ae7-006618b66f-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.079 DEBUG:boto:b'dkyges-63obj511000falseobj512024-04-12T04:15:47.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.080 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.080 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.080 DEBUG:boto:path=/dkyges-63/?prefix=obj51 2024-04-12T04:19:59.080 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj51 2024-04-12T04:19:59.080 DEBUG:boto:Method: GET 2024-04-12T04:19:59.080 DEBUG:boto:Path: /dkyges-63/?prefix=obj51 2024-04-12T04:19:59.080 DEBUG:boto:Data: 2024-04-12T04:19:59.080 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.080 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.081 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.081 DEBUG:boto:Params: {} 2024-04-12T04:19:59.081 DEBUG:boto:Token: None 2024-04-12T04:19:59.081 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.081 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000660e0876ae64c6ad-006618b66f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.126 DEBUG:boto:b'dkyges-63obj511000falseobj512024-04-12T04:15:47.820Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:19:59.127 DEBUG:boto:path=/dkyges-63/obj55 2024-04-12T04:19:59.127 DEBUG:boto:auth_path=/dkyges-63/obj55 2024-04-12T04:19:59.127 DEBUG:boto:path=/dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.127 DEBUG:boto:auth_path=/dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.127 DEBUG:boto:Method: GET 2024-04-12T04:19:59.127 DEBUG:boto:Path: /dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.127 DEBUG:boto:Data: 2024-04-12T04:19:59.127 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.127 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.128 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.128 DEBUG:boto:Params: {} 2024-04-12T04:19:59.128 DEBUG:boto:Token: None 2024-04-12T04:19:59.128 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.128 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VvJm5aLeONG7Q2FYTvKyih308Vk= 2024-04-12T04:19:59.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VvJm5aLeONG7Q2FYTvKyih308Vk=', 'Content-Length': '0'} 2024-04-12T04:19:59.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016ff7e84e398faaf-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.130 DEBUG:boto:path=/dkyges-63/obj55 2024-04-12T04:19:59.130 DEBUG:boto:auth_path=/dkyges-63/obj55 2024-04-12T04:19:59.131 DEBUG:boto:path=/dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.131 DEBUG:boto:auth_path=/dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.131 DEBUG:boto:Method: GET 2024-04-12T04:19:59.131 DEBUG:boto:Path: /dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.131 DEBUG:boto:Data: 2024-04-12T04:19:59.131 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.131 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.131 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.131 DEBUG:boto:Params: {} 2024-04-12T04:19:59.131 DEBUG:boto:Token: None 2024-04-12T04:19:59.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.132 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VvJm5aLeONG7Q2FYTvKyih308Vk= 2024-04-12T04:19:59.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VvJm5aLeONG7Q2FYTvKyih308Vk=', 'Content-Length': '0'} 2024-04-12T04:19:59.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7da968c603ce86c-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.134 DEBUG:boto:path=/dkyges-63/obj55 2024-04-12T04:19:59.134 DEBUG:boto:auth_path=/dkyges-63/obj55 2024-04-12T04:19:59.134 DEBUG:boto:path=/dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.135 DEBUG:boto:auth_path=/dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.135 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.135 DEBUG:boto:Path: /dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.135 DEBUG:boto:Data: 2024-04-12T04:19:59.135 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.135 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.135 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.135 DEBUG:boto:Params: {} 2024-04-12T04:19:59.135 DEBUG:boto:Token: None 2024-04-12T04:19:59.135 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.135 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5FUPZCGyW4e+Ra0af9Y3rXLZTjY= 2024-04-12T04:19:59.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5FUPZCGyW4e+Ra0af9Y3rXLZTjY=', 'Content-Length': '0'} 2024-04-12T04:19:59.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3f6997a95d4c248-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.138 DEBUG:boto:path=/dkyges-63/obj55 2024-04-12T04:19:59.138 DEBUG:boto:auth_path=/dkyges-63/obj55 2024-04-12T04:19:59.138 DEBUG:boto:path=/dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.138 DEBUG:boto:auth_path=/dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.138 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.138 DEBUG:boto:Path: /dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.138 DEBUG:boto:Data: 2024-04-12T04:19:59.138 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.138 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.138 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.139 DEBUG:boto:Params: {} 2024-04-12T04:19:59.139 DEBUG:boto:Token: None 2024-04-12T04:19:59.139 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.139 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5FUPZCGyW4e+Ra0af9Y3rXLZTjY= 2024-04-12T04:19:59.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5FUPZCGyW4e+Ra0af9Y3rXLZTjY=', 'Content-Length': '0'} 2024-04-12T04:19:59.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000037af0c86ac03707-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:19:59.142 DEBUG:boto:path=/dkyges-63/obj55 2024-04-12T04:19:59.142 DEBUG:boto:auth_path=/dkyges-63/obj55 2024-04-12T04:19:59.142 DEBUG:boto:path=/dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.142 DEBUG:boto:auth_path=/dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.142 DEBUG:boto:Method: GET 2024-04-12T04:19:59.142 DEBUG:boto:Path: /dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.142 DEBUG:boto:Data: 2024-04-12T04:19:59.142 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.142 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.143 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.143 DEBUG:boto:Params: {} 2024-04-12T04:19:59.143 DEBUG:boto:Token: None 2024-04-12T04:19:59.143 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.143 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VvJm5aLeONG7Q2FYTvKyih308Vk= 2024-04-12T04:19:59.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VvJm5aLeONG7Q2FYTvKyih308Vk=', 'Content-Length': '0'} 2024-04-12T04:19:59.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d62bb994c89e503e-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.145 DEBUG:boto:path=/dkyges-63/obj55 2024-04-12T04:19:59.145 DEBUG:boto:auth_path=/dkyges-63/obj55 2024-04-12T04:19:59.145 DEBUG:boto:path=/dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.145 DEBUG:boto:auth_path=/dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.145 DEBUG:boto:Method: GET 2024-04-12T04:19:59.145 DEBUG:boto:Path: /dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.146 DEBUG:boto:Data: 2024-04-12T04:19:59.146 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.146 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.146 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.146 DEBUG:boto:Params: {} 2024-04-12T04:19:59.146 DEBUG:boto:Token: None 2024-04-12T04:19:59.146 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj55?versionId=null 2024-04-12T04:19:59.146 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VvJm5aLeONG7Q2FYTvKyih308Vk= 2024-04-12T04:19:59.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VvJm5aLeONG7Q2FYTvKyih308Vk=', 'Content-Length': '0'} 2024-04-12T04:19:59.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b319eeb30aa75da4-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.148 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.148 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.148 DEBUG:boto:path=/dkyges-63/?prefix=obj55 2024-04-12T04:19:59.149 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj55 2024-04-12T04:19:59.149 DEBUG:boto:Method: GET 2024-04-12T04:19:59.149 DEBUG:boto:Path: /dkyges-63/?prefix=obj55 2024-04-12T04:19:59.149 DEBUG:boto:Data: 2024-04-12T04:19:59.149 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.149 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.149 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.149 DEBUG:boto:Params: {} 2024-04-12T04:19:59.149 DEBUG:boto:Token: None 2024-04-12T04:19:59.149 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.150 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006abc14dd190cd49b-006618b66f-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.152 DEBUG:boto:b'dkyges-63obj551000falseobj552024-04-12T04:15:47.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.153 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.153 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.153 DEBUG:boto:path=/dkyges-63/?prefix=obj55 2024-04-12T04:19:59.153 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj55 2024-04-12T04:19:59.153 DEBUG:boto:Method: GET 2024-04-12T04:19:59.153 DEBUG:boto:Path: /dkyges-63/?prefix=obj55 2024-04-12T04:19:59.153 DEBUG:boto:Data: 2024-04-12T04:19:59.153 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.154 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.154 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.154 DEBUG:boto:Params: {} 2024-04-12T04:19:59.154 DEBUG:boto:Token: None 2024-04-12T04:19:59.154 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.154 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ed008011c3bd8a9-006618b66f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.159 DEBUG:boto:b'dkyges-63obj551000falseobj552024-04-12T04:15:47.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.159 DEBUG:boto:path=/dkyges-63/obj55 2024-04-12T04:19:59.159 DEBUG:boto:auth_path=/dkyges-63/obj55 2024-04-12T04:19:59.159 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.159 DEBUG:boto:Path: /dkyges-63/obj55 2024-04-12T04:19:59.159 DEBUG:boto:Data: 2024-04-12T04:19:59.160 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.160 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.160 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.160 DEBUG:boto:Params: {} 2024-04-12T04:19:59.160 DEBUG:boto:Token: None 2024-04-12T04:19:59.160 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj55 2024-04-12T04:19:59.160 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vxdNPzeC+fDi6Cl6hQTrs5KhMX8= 2024-04-12T04:19:59.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vxdNPzeC+fDi6Cl6hQTrs5KhMX8=', 'Content-Length': '0'} 2024-04-12T04:19:59.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc89fadf9b46f9ed-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.162 DEBUG:boto:path=/dkyges-63/obj55 2024-04-12T04:19:59.162 DEBUG:boto:auth_path=/dkyges-63/obj55 2024-04-12T04:19:59.162 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.162 DEBUG:boto:Path: /dkyges-63/obj55 2024-04-12T04:19:59.163 DEBUG:boto:Data: 2024-04-12T04:19:59.163 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.163 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.163 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.163 DEBUG:boto:Params: {} 2024-04-12T04:19:59.163 DEBUG:boto:Token: None 2024-04-12T04:19:59.163 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj55 2024-04-12T04:19:59.163 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vxdNPzeC+fDi6Cl6hQTrs5KhMX8= 2024-04-12T04:19:59.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vxdNPzeC+fDi6Cl6hQTrs5KhMX8=', 'Content-Length': '0'} 2024-04-12T04:19:59.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005063d34551f8fba5-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:19:59.165 DEBUG:boto:path=/dkyges-63/obj55 2024-04-12T04:19:59.166 DEBUG:boto:auth_path=/dkyges-63/obj55 2024-04-12T04:19:59.166 DEBUG:boto:Method: GET 2024-04-12T04:19:59.166 DEBUG:boto:Path: /dkyges-63/obj55 2024-04-12T04:19:59.166 DEBUG:boto:Data: 2024-04-12T04:19:59.166 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.166 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.166 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.166 DEBUG:boto:Params: {} 2024-04-12T04:19:59.166 DEBUG:boto:Token: None 2024-04-12T04:19:59.166 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj55 2024-04-12T04:19:59.167 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C70+AG3Raq9onPwOSjZK6O4+N3s= 2024-04-12T04:19:59.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C70+AG3Raq9onPwOSjZK6O4+N3s=', 'Content-Length': '0'} 2024-04-12T04:19:59.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000693c22eeb143d8a0-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.169 DEBUG:boto:path=/dkyges-63/obj55 2024-04-12T04:19:59.169 DEBUG:boto:auth_path=/dkyges-63/obj55 2024-04-12T04:19:59.169 DEBUG:boto:Method: GET 2024-04-12T04:19:59.169 DEBUG:boto:Path: /dkyges-63/obj55 2024-04-12T04:19:59.169 DEBUG:boto:Data: 2024-04-12T04:19:59.169 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.169 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.169 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.169 DEBUG:boto:Params: {} 2024-04-12T04:19:59.169 DEBUG:boto:Token: None 2024-04-12T04:19:59.170 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj55 2024-04-12T04:19:59.170 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:C70+AG3Raq9onPwOSjZK6O4+N3s= 2024-04-12T04:19:59.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C70+AG3Raq9onPwOSjZK6O4+N3s=', 'Content-Length': '0'} 2024-04-12T04:19:59.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4b97183a2b4e7ab-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.172 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.172 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.172 DEBUG:boto:path=/dkyges-63/?prefix=obj55 2024-04-12T04:19:59.172 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj55 2024-04-12T04:19:59.172 DEBUG:boto:Method: GET 2024-04-12T04:19:59.172 DEBUG:boto:Path: /dkyges-63/?prefix=obj55 2024-04-12T04:19:59.172 DEBUG:boto:Data: 2024-04-12T04:19:59.172 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.173 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.173 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.173 DEBUG:boto:Params: {} 2024-04-12T04:19:59.173 DEBUG:boto:Token: None 2024-04-12T04:19:59.173 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.173 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8813cedf78c313d-006618b66f-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.176 DEBUG:boto:b'dkyges-63obj551000falseobj552024-04-12T04:15:47.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.176 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.176 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.176 DEBUG:boto:path=/dkyges-63/?prefix=obj55 2024-04-12T04:19:59.176 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj55 2024-04-12T04:19:59.176 DEBUG:boto:Method: GET 2024-04-12T04:19:59.176 DEBUG:boto:Path: /dkyges-63/?prefix=obj55 2024-04-12T04:19:59.176 DEBUG:boto:Data: 2024-04-12T04:19:59.177 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.177 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.177 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.177 DEBUG:boto:Params: {} 2024-04-12T04:19:59.177 DEBUG:boto:Token: None 2024-04-12T04:19:59.177 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.177 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009146054252f0ed47-006618b66f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.182 DEBUG:boto:b'dkyges-63obj551000falseobj552024-04-12T04:15:47.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:19:59.182 DEBUG:boto:path=/dkyges-63/obj57 2024-04-12T04:19:59.182 DEBUG:boto:auth_path=/dkyges-63/obj57 2024-04-12T04:19:59.182 DEBUG:boto:path=/dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.182 DEBUG:boto:auth_path=/dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.182 DEBUG:boto:Method: GET 2024-04-12T04:19:59.182 DEBUG:boto:Path: /dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.183 DEBUG:boto:Data: 2024-04-12T04:19:59.183 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.183 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.183 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.183 DEBUG:boto:Params: {} 2024-04-12T04:19:59.183 DEBUG:boto:Token: None 2024-04-12T04:19:59.183 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.183 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zubhJAG3w8D2UOJdw84R7e/+mJQ= 2024-04-12T04:19:59.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zubhJAG3w8D2UOJdw84R7e/+mJQ=', 'Content-Length': '0'} 2024-04-12T04:19:59.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a029bd1da7ee3a0-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.226 DEBUG:boto:path=/dkyges-63/obj57 2024-04-12T04:19:59.226 DEBUG:boto:auth_path=/dkyges-63/obj57 2024-04-12T04:19:59.226 DEBUG:boto:path=/dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.227 DEBUG:boto:auth_path=/dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.227 DEBUG:boto:Method: GET 2024-04-12T04:19:59.227 DEBUG:boto:Path: /dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.227 DEBUG:boto:Data: 2024-04-12T04:19:59.227 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.227 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.227 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.227 DEBUG:boto:Params: {} 2024-04-12T04:19:59.227 DEBUG:boto:Token: None 2024-04-12T04:19:59.227 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.228 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zubhJAG3w8D2UOJdw84R7e/+mJQ= 2024-04-12T04:19:59.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zubhJAG3w8D2UOJdw84R7e/+mJQ=', 'Content-Length': '0'} 2024-04-12T04:19:59.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d2280f213f5bd22-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.230 DEBUG:boto:path=/dkyges-63/obj57 2024-04-12T04:19:59.230 DEBUG:boto:auth_path=/dkyges-63/obj57 2024-04-12T04:19:59.230 DEBUG:boto:path=/dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.230 DEBUG:boto:auth_path=/dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.230 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.231 DEBUG:boto:Path: /dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.231 DEBUG:boto:Data: 2024-04-12T04:19:59.231 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.231 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.231 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.231 DEBUG:boto:Params: {} 2024-04-12T04:19:59.231 DEBUG:boto:Token: None 2024-04-12T04:19:59.231 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.231 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wo81oRrr9l5DDuB4TZKRIo2yU/4= 2024-04-12T04:19:59.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wo81oRrr9l5DDuB4TZKRIo2yU/4=', 'Content-Length': '0'} 2024-04-12T04:19:59.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069c7105a70eabe67-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.234 DEBUG:boto:path=/dkyges-63/obj57 2024-04-12T04:19:59.234 DEBUG:boto:auth_path=/dkyges-63/obj57 2024-04-12T04:19:59.234 DEBUG:boto:path=/dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.234 DEBUG:boto:auth_path=/dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.234 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.234 DEBUG:boto:Path: /dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.234 DEBUG:boto:Data: 2024-04-12T04:19:59.234 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.234 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.234 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.234 DEBUG:boto:Params: {} 2024-04-12T04:19:59.235 DEBUG:boto:Token: None 2024-04-12T04:19:59.235 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.235 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Wo81oRrr9l5DDuB4TZKRIo2yU/4= 2024-04-12T04:19:59.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wo81oRrr9l5DDuB4TZKRIo2yU/4=', 'Content-Length': '0'} 2024-04-12T04:19:59.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b335ee0d666a7d6a-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:19:59.238 DEBUG:boto:path=/dkyges-63/obj57 2024-04-12T04:19:59.238 DEBUG:boto:auth_path=/dkyges-63/obj57 2024-04-12T04:19:59.238 DEBUG:boto:path=/dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.238 DEBUG:boto:auth_path=/dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.238 DEBUG:boto:Method: GET 2024-04-12T04:19:59.238 DEBUG:boto:Path: /dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.238 DEBUG:boto:Data: 2024-04-12T04:19:59.238 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.238 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.238 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.239 DEBUG:boto:Params: {} 2024-04-12T04:19:59.239 DEBUG:boto:Token: None 2024-04-12T04:19:59.239 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.239 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zubhJAG3w8D2UOJdw84R7e/+mJQ= 2024-04-12T04:19:59.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zubhJAG3w8D2UOJdw84R7e/+mJQ=', 'Content-Length': '0'} 2024-04-12T04:19:59.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c40845c7661c5a16-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.241 DEBUG:boto:path=/dkyges-63/obj57 2024-04-12T04:19:59.241 DEBUG:boto:auth_path=/dkyges-63/obj57 2024-04-12T04:19:59.241 DEBUG:boto:path=/dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.241 DEBUG:boto:auth_path=/dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.241 DEBUG:boto:Method: GET 2024-04-12T04:19:59.241 DEBUG:boto:Path: /dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.242 DEBUG:boto:Data: 2024-04-12T04:19:59.242 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.242 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.242 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.242 DEBUG:boto:Params: {} 2024-04-12T04:19:59.242 DEBUG:boto:Token: None 2024-04-12T04:19:59.242 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj57?versionId=null 2024-04-12T04:19:59.242 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zubhJAG3w8D2UOJdw84R7e/+mJQ= 2024-04-12T04:19:59.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zubhJAG3w8D2UOJdw84R7e/+mJQ=', 'Content-Length': '0'} 2024-04-12T04:19:59.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000326ace2a3a600ecc-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.244 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.245 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.245 DEBUG:boto:path=/dkyges-63/?prefix=obj57 2024-04-12T04:19:59.245 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj57 2024-04-12T04:19:59.245 DEBUG:boto:Method: GET 2024-04-12T04:19:59.245 DEBUG:boto:Path: /dkyges-63/?prefix=obj57 2024-04-12T04:19:59.245 DEBUG:boto:Data: 2024-04-12T04:19:59.245 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.245 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.245 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.245 DEBUG:boto:Params: {} 2024-04-12T04:19:59.245 DEBUG:boto:Token: None 2024-04-12T04:19:59.246 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.246 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8092d0e44b8f030-006618b66f-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.249 DEBUG:boto:b'dkyges-63obj571000falseobj572024-04-12T04:15:48.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.249 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.249 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.249 DEBUG:boto:path=/dkyges-63/?prefix=obj57 2024-04-12T04:19:59.249 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj57 2024-04-12T04:19:59.249 DEBUG:boto:Method: GET 2024-04-12T04:19:59.249 DEBUG:boto:Path: /dkyges-63/?prefix=obj57 2024-04-12T04:19:59.250 DEBUG:boto:Data: 2024-04-12T04:19:59.250 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.250 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.250 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.250 DEBUG:boto:Params: {} 2024-04-12T04:19:59.250 DEBUG:boto:Token: None 2024-04-12T04:19:59.253 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.254 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075701283cbe0a6dc-006618b66f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.258 DEBUG:boto:b'dkyges-63obj571000falseobj572024-04-12T04:15:48.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.259 DEBUG:boto:path=/dkyges-63/obj57 2024-04-12T04:19:59.259 DEBUG:boto:auth_path=/dkyges-63/obj57 2024-04-12T04:19:59.259 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.259 DEBUG:boto:Path: /dkyges-63/obj57 2024-04-12T04:19:59.259 DEBUG:boto:Data: 2024-04-12T04:19:59.259 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.259 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.259 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.259 DEBUG:boto:Params: {} 2024-04-12T04:19:59.260 DEBUG:boto:Token: None 2024-04-12T04:19:59.260 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj57 2024-04-12T04:19:59.260 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wb3IbscsJL61mM+DfSOu6xLEjII= 2024-04-12T04:19:59.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wb3IbscsJL61mM+DfSOu6xLEjII=', 'Content-Length': '0'} 2024-04-12T04:19:59.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d0feede6dd83d75-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.262 DEBUG:boto:path=/dkyges-63/obj57 2024-04-12T04:19:59.262 DEBUG:boto:auth_path=/dkyges-63/obj57 2024-04-12T04:19:59.262 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.262 DEBUG:boto:Path: /dkyges-63/obj57 2024-04-12T04:19:59.262 DEBUG:boto:Data: 2024-04-12T04:19:59.262 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.262 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.262 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.262 DEBUG:boto:Params: {} 2024-04-12T04:19:59.263 DEBUG:boto:Token: None 2024-04-12T04:19:59.263 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj57 2024-04-12T04:19:59.263 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wb3IbscsJL61mM+DfSOu6xLEjII= 2024-04-12T04:19:59.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wb3IbscsJL61mM+DfSOu6xLEjII=', 'Content-Length': '0'} 2024-04-12T04:19:59.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42c2d0eafd67956-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2024-04-12T04:19:59.265 DEBUG:boto:path=/dkyges-63/obj57 2024-04-12T04:19:59.265 DEBUG:boto:auth_path=/dkyges-63/obj57 2024-04-12T04:19:59.265 DEBUG:boto:Method: GET 2024-04-12T04:19:59.265 DEBUG:boto:Path: /dkyges-63/obj57 2024-04-12T04:19:59.266 DEBUG:boto:Data: 2024-04-12T04:19:59.266 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.266 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.266 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.266 DEBUG:boto:Params: {} 2024-04-12T04:19:59.266 DEBUG:boto:Token: None 2024-04-12T04:19:59.266 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj57 2024-04-12T04:19:59.266 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NrUNWxG6kRIW5u65NADA/h9Kz5o= 2024-04-12T04:19:59.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NrUNWxG6kRIW5u65NADA/h9Kz5o=', 'Content-Length': '0'} 2024-04-12T04:19:59.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ff523908e119879-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.268 DEBUG:boto:path=/dkyges-63/obj57 2024-04-12T04:19:59.269 DEBUG:boto:auth_path=/dkyges-63/obj57 2024-04-12T04:19:59.269 DEBUG:boto:Method: GET 2024-04-12T04:19:59.269 DEBUG:boto:Path: /dkyges-63/obj57 2024-04-12T04:19:59.269 DEBUG:boto:Data: 2024-04-12T04:19:59.269 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.269 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.269 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.269 DEBUG:boto:Params: {} 2024-04-12T04:19:59.269 DEBUG:boto:Token: None 2024-04-12T04:19:59.269 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj57 2024-04-12T04:19:59.270 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NrUNWxG6kRIW5u65NADA/h9Kz5o= 2024-04-12T04:19:59.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NrUNWxG6kRIW5u65NADA/h9Kz5o=', 'Content-Length': '0'} 2024-04-12T04:19:59.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ada8579c5781280e-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.314 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.314 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.315 DEBUG:boto:path=/dkyges-63/?prefix=obj57 2024-04-12T04:19:59.315 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj57 2024-04-12T04:19:59.315 DEBUG:boto:Method: GET 2024-04-12T04:19:59.315 DEBUG:boto:Path: /dkyges-63/?prefix=obj57 2024-04-12T04:19:59.315 DEBUG:boto:Data: 2024-04-12T04:19:59.315 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.315 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.315 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.315 DEBUG:boto:Params: {} 2024-04-12T04:19:59.315 DEBUG:boto:Token: None 2024-04-12T04:19:59.315 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.316 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eaa1107e3e2729a-006618b66f-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.320 DEBUG:boto:b'dkyges-63obj571000falseobj572024-04-12T04:15:48.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.320 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.320 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.320 DEBUG:boto:path=/dkyges-63/?prefix=obj57 2024-04-12T04:19:59.320 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj57 2024-04-12T04:19:59.320 DEBUG:boto:Method: GET 2024-04-12T04:19:59.321 DEBUG:boto:Path: /dkyges-63/?prefix=obj57 2024-04-12T04:19:59.321 DEBUG:boto:Data: 2024-04-12T04:19:59.321 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.321 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.321 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.321 DEBUG:boto:Params: {} 2024-04-12T04:19:59.321 DEBUG:boto:Token: None 2024-04-12T04:19:59.321 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.321 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017ee1b6442d8b2bf-006618b66f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.326 DEBUG:boto:b'dkyges-63obj571000falseobj572024-04-12T04:15:48.176Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:19:59.326 DEBUG:boto:path=/dkyges-63/obj58 2024-04-12T04:19:59.326 DEBUG:boto:auth_path=/dkyges-63/obj58 2024-04-12T04:19:59.326 DEBUG:boto:path=/dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.326 DEBUG:boto:auth_path=/dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.327 DEBUG:boto:Method: GET 2024-04-12T04:19:59.327 DEBUG:boto:Path: /dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.327 DEBUG:boto:Data: 2024-04-12T04:19:59.327 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.327 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.327 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.327 DEBUG:boto:Params: {} 2024-04-12T04:19:59.327 DEBUG:boto:Token: None 2024-04-12T04:19:59.327 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.327 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l348wWGydTy1uFx8pJI+P5InHT8= 2024-04-12T04:19:59.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l348wWGydTy1uFx8pJI+P5InHT8=', 'Content-Length': '0'} 2024-04-12T04:19:59.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009af1a232efdb6637-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.330 DEBUG:boto:path=/dkyges-63/obj58 2024-04-12T04:19:59.330 DEBUG:boto:auth_path=/dkyges-63/obj58 2024-04-12T04:19:59.330 DEBUG:boto:path=/dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.330 DEBUG:boto:auth_path=/dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.330 DEBUG:boto:Method: GET 2024-04-12T04:19:59.330 DEBUG:boto:Path: /dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.330 DEBUG:boto:Data: 2024-04-12T04:19:59.330 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.330 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.330 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.330 DEBUG:boto:Params: {} 2024-04-12T04:19:59.331 DEBUG:boto:Token: None 2024-04-12T04:19:59.331 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.331 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l348wWGydTy1uFx8pJI+P5InHT8= 2024-04-12T04:19:59.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l348wWGydTy1uFx8pJI+P5InHT8=', 'Content-Length': '0'} 2024-04-12T04:19:59.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052ca898d92f086eb-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.333 DEBUG:boto:path=/dkyges-63/obj58 2024-04-12T04:19:59.333 DEBUG:boto:auth_path=/dkyges-63/obj58 2024-04-12T04:19:59.333 DEBUG:boto:path=/dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.333 DEBUG:boto:auth_path=/dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.334 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.334 DEBUG:boto:Path: /dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.334 DEBUG:boto:Data: 2024-04-12T04:19:59.334 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.334 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.334 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.334 DEBUG:boto:Params: {} 2024-04-12T04:19:59.334 DEBUG:boto:Token: None 2024-04-12T04:19:59.334 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.334 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W3+ab3rimKpF5tu2D1kKsZ0s0Co= 2024-04-12T04:19:59.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W3+ab3rimKpF5tu2D1kKsZ0s0Co=', 'Content-Length': '0'} 2024-04-12T04:19:59.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009077cc6f90408675-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.337 DEBUG:boto:path=/dkyges-63/obj58 2024-04-12T04:19:59.337 DEBUG:boto:auth_path=/dkyges-63/obj58 2024-04-12T04:19:59.337 DEBUG:boto:path=/dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.337 DEBUG:boto:auth_path=/dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.337 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.337 DEBUG:boto:Path: /dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.337 DEBUG:boto:Data: 2024-04-12T04:19:59.337 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.337 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.337 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.337 DEBUG:boto:Params: {} 2024-04-12T04:19:59.338 DEBUG:boto:Token: None 2024-04-12T04:19:59.338 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.338 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W3+ab3rimKpF5tu2D1kKsZ0s0Co= 2024-04-12T04:19:59.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W3+ab3rimKpF5tu2D1kKsZ0s0Co=', 'Content-Length': '0'} 2024-04-12T04:19:59.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d34b7670b27d3f-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:19:59.340 DEBUG:boto:path=/dkyges-63/obj58 2024-04-12T04:19:59.340 DEBUG:boto:auth_path=/dkyges-63/obj58 2024-04-12T04:19:59.340 DEBUG:boto:path=/dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.340 DEBUG:boto:auth_path=/dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.341 DEBUG:boto:Method: GET 2024-04-12T04:19:59.341 DEBUG:boto:Path: /dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.341 DEBUG:boto:Data: 2024-04-12T04:19:59.341 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.341 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.341 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.341 DEBUG:boto:Params: {} 2024-04-12T04:19:59.341 DEBUG:boto:Token: None 2024-04-12T04:19:59.341 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.341 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l348wWGydTy1uFx8pJI+P5InHT8= 2024-04-12T04:19:59.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l348wWGydTy1uFx8pJI+P5InHT8=', 'Content-Length': '0'} 2024-04-12T04:19:59.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6d1320cc5189c01-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.344 DEBUG:boto:path=/dkyges-63/obj58 2024-04-12T04:19:59.344 DEBUG:boto:auth_path=/dkyges-63/obj58 2024-04-12T04:19:59.344 DEBUG:boto:path=/dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.344 DEBUG:boto:auth_path=/dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.344 DEBUG:boto:Method: GET 2024-04-12T04:19:59.344 DEBUG:boto:Path: /dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.345 DEBUG:boto:Data: 2024-04-12T04:19:59.345 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.345 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.345 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.345 DEBUG:boto:Params: {} 2024-04-12T04:19:59.345 DEBUG:boto:Token: None 2024-04-12T04:19:59.345 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj58?versionId=null 2024-04-12T04:19:59.345 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l348wWGydTy1uFx8pJI+P5InHT8= 2024-04-12T04:19:59.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l348wWGydTy1uFx8pJI+P5InHT8=', 'Content-Length': '0'} 2024-04-12T04:19:59.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbf3ae9ec6cf4910-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.347 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.347 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.347 DEBUG:boto:path=/dkyges-63/?prefix=obj58 2024-04-12T04:19:59.348 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj58 2024-04-12T04:19:59.348 DEBUG:boto:Method: GET 2024-04-12T04:19:59.348 DEBUG:boto:Path: /dkyges-63/?prefix=obj58 2024-04-12T04:19:59.348 DEBUG:boto:Data: 2024-04-12T04:19:59.348 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.348 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.348 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.348 DEBUG:boto:Params: {} 2024-04-12T04:19:59.348 DEBUG:boto:Token: None 2024-04-12T04:19:59.348 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.349 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aecb904f31b2e451-006618b66f-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.394 DEBUG:boto:b'dkyges-63obj581000falseobj582024-04-12T04:15:48.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.394 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.395 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.395 DEBUG:boto:path=/dkyges-63/?prefix=obj58 2024-04-12T04:19:59.395 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj58 2024-04-12T04:19:59.395 DEBUG:boto:Method: GET 2024-04-12T04:19:59.395 DEBUG:boto:Path: /dkyges-63/?prefix=obj58 2024-04-12T04:19:59.395 DEBUG:boto:Data: 2024-04-12T04:19:59.395 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.395 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.395 DEBUG:boto:Params: {} 2024-04-12T04:19:59.396 DEBUG:boto:Token: None 2024-04-12T04:19:59.396 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.396 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a1b51ad86b2c7ac-006618b66f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.400 DEBUG:boto:b'dkyges-63obj581000falseobj582024-04-12T04:15:48.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.400 DEBUG:boto:path=/dkyges-63/obj58 2024-04-12T04:19:59.400 DEBUG:boto:auth_path=/dkyges-63/obj58 2024-04-12T04:19:59.400 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.401 DEBUG:boto:Path: /dkyges-63/obj58 2024-04-12T04:19:59.401 DEBUG:boto:Data: 2024-04-12T04:19:59.401 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.401 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.401 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.401 DEBUG:boto:Params: {} 2024-04-12T04:19:59.401 DEBUG:boto:Token: None 2024-04-12T04:19:59.401 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj58 2024-04-12T04:19:59.401 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HfOkgLxU2MIFIMFRwpNJ51MO3LM= 2024-04-12T04:19:59.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HfOkgLxU2MIFIMFRwpNJ51MO3LM=', 'Content-Length': '0'} 2024-04-12T04:19:59.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000463ceccffacc32c4-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.403 DEBUG:boto:path=/dkyges-63/obj58 2024-04-12T04:19:59.403 DEBUG:boto:auth_path=/dkyges-63/obj58 2024-04-12T04:19:59.404 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.404 DEBUG:boto:Path: /dkyges-63/obj58 2024-04-12T04:19:59.404 DEBUG:boto:Data: 2024-04-12T04:19:59.404 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.404 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.404 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.404 DEBUG:boto:Params: {} 2024-04-12T04:19:59.404 DEBUG:boto:Token: None 2024-04-12T04:19:59.404 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj58 2024-04-12T04:19:59.404 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HfOkgLxU2MIFIMFRwpNJ51MO3LM= 2024-04-12T04:19:59.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HfOkgLxU2MIFIMFRwpNJ51MO3LM=', 'Content-Length': '0'} 2024-04-12T04:19:59.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eebfc1f523b236d9-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2024-04-12T04:19:59.407 DEBUG:boto:path=/dkyges-63/obj58 2024-04-12T04:19:59.407 DEBUG:boto:auth_path=/dkyges-63/obj58 2024-04-12T04:19:59.407 DEBUG:boto:Method: GET 2024-04-12T04:19:59.407 DEBUG:boto:Path: /dkyges-63/obj58 2024-04-12T04:19:59.407 DEBUG:boto:Data: 2024-04-12T04:19:59.407 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.407 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.407 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.407 DEBUG:boto:Params: {} 2024-04-12T04:19:59.408 DEBUG:boto:Token: None 2024-04-12T04:19:59.408 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj58 2024-04-12T04:19:59.408 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:shqM97DgEEPQOZHYcwiTNwSRSFU= 2024-04-12T04:19:59.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:shqM97DgEEPQOZHYcwiTNwSRSFU=', 'Content-Length': '0'} 2024-04-12T04:19:59.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000647d9b8052e32251-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.410 DEBUG:boto:path=/dkyges-63/obj58 2024-04-12T04:19:59.410 DEBUG:boto:auth_path=/dkyges-63/obj58 2024-04-12T04:19:59.410 DEBUG:boto:Method: GET 2024-04-12T04:19:59.410 DEBUG:boto:Path: /dkyges-63/obj58 2024-04-12T04:19:59.410 DEBUG:boto:Data: 2024-04-12T04:19:59.410 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.411 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.411 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.411 DEBUG:boto:Params: {} 2024-04-12T04:19:59.411 DEBUG:boto:Token: None 2024-04-12T04:19:59.411 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj58 2024-04-12T04:19:59.411 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:shqM97DgEEPQOZHYcwiTNwSRSFU= 2024-04-12T04:19:59.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:shqM97DgEEPQOZHYcwiTNwSRSFU=', 'Content-Length': '0'} 2024-04-12T04:19:59.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e141cf65ca3d970-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.454 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.454 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.458 DEBUG:boto:path=/dkyges-63/?prefix=obj58 2024-04-12T04:19:59.459 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj58 2024-04-12T04:19:59.460 DEBUG:boto:Method: GET 2024-04-12T04:19:59.460 DEBUG:boto:Path: /dkyges-63/?prefix=obj58 2024-04-12T04:19:59.460 DEBUG:boto:Data: 2024-04-12T04:19:59.460 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.460 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.460 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.461 DEBUG:boto:Params: {} 2024-04-12T04:19:59.461 DEBUG:boto:Token: None 2024-04-12T04:19:59.461 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.461 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e846b10d510c9525-006618b66f-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.465 DEBUG:boto:b'dkyges-63obj581000falseobj582024-04-12T04:15:48.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.465 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.465 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.466 DEBUG:boto:path=/dkyges-63/?prefix=obj58 2024-04-12T04:19:59.466 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj58 2024-04-12T04:19:59.466 DEBUG:boto:Method: GET 2024-04-12T04:19:59.466 DEBUG:boto:Path: /dkyges-63/?prefix=obj58 2024-04-12T04:19:59.466 DEBUG:boto:Data: 2024-04-12T04:19:59.467 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.467 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.467 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.467 DEBUG:boto:Params: {} 2024-04-12T04:19:59.467 DEBUG:boto:Token: None 2024-04-12T04:19:59.468 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.468 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e58b6600af92eb5-006618b66f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.473 DEBUG:boto:b'dkyges-63obj581000falseobj582024-04-12T04:15:48.838Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T04:19:59.474 DEBUG:boto:path=/dkyges-63/obj62 2024-04-12T04:19:59.474 DEBUG:boto:auth_path=/dkyges-63/obj62 2024-04-12T04:19:59.474 DEBUG:boto:path=/dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.474 DEBUG:boto:auth_path=/dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.475 DEBUG:boto:Method: GET 2024-04-12T04:19:59.475 DEBUG:boto:Path: /dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.475 DEBUG:boto:Data: 2024-04-12T04:19:59.475 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.475 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.476 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.476 DEBUG:boto:Params: {} 2024-04-12T04:19:59.476 DEBUG:boto:Token: None 2024-04-12T04:19:59.476 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.476 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3VNVknVXwWcKiFzuHMcuU+IDZqU= 2024-04-12T04:19:59.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3VNVknVXwWcKiFzuHMcuU+IDZqU=', 'Content-Length': '0'} 2024-04-12T04:19:59.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9853924acc2ed1f-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.479 DEBUG:boto:path=/dkyges-63/obj62 2024-04-12T04:19:59.480 DEBUG:boto:auth_path=/dkyges-63/obj62 2024-04-12T04:19:59.480 DEBUG:boto:path=/dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.480 DEBUG:boto:auth_path=/dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.480 DEBUG:boto:Method: GET 2024-04-12T04:19:59.480 DEBUG:boto:Path: /dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.481 DEBUG:boto:Data: 2024-04-12T04:19:59.481 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.481 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.481 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.481 DEBUG:boto:Params: {} 2024-04-12T04:19:59.482 DEBUG:boto:Token: None 2024-04-12T04:19:59.482 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.482 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3VNVknVXwWcKiFzuHMcuU+IDZqU= 2024-04-12T04:19:59.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3VNVknVXwWcKiFzuHMcuU+IDZqU=', 'Content-Length': '0'} 2024-04-12T04:19:59.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091edf28cf1e8a4fc-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.485 DEBUG:boto:path=/dkyges-63/obj62 2024-04-12T04:19:59.485 DEBUG:boto:auth_path=/dkyges-63/obj62 2024-04-12T04:19:59.485 DEBUG:boto:path=/dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.485 DEBUG:boto:auth_path=/dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.486 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.486 DEBUG:boto:Path: /dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.486 DEBUG:boto:Data: 2024-04-12T04:19:59.486 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.486 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.487 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.487 DEBUG:boto:Params: {} 2024-04-12T04:19:59.487 DEBUG:boto:Token: None 2024-04-12T04:19:59.487 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.487 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uW0vCBzsqpxYXSb2BJTO691fL7w= 2024-04-12T04:19:59.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uW0vCBzsqpxYXSb2BJTO691fL7w=', 'Content-Length': '0'} 2024-04-12T04:19:59.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098c9e5c07af05368-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.490 DEBUG:boto:path=/dkyges-63/obj62 2024-04-12T04:19:59.490 DEBUG:boto:auth_path=/dkyges-63/obj62 2024-04-12T04:19:59.491 DEBUG:boto:path=/dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.491 DEBUG:boto:auth_path=/dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.491 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.491 DEBUG:boto:Path: /dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.491 DEBUG:boto:Data: 2024-04-12T04:19:59.491 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.492 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.492 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.492 DEBUG:boto:Params: {} 2024-04-12T04:19:59.492 DEBUG:boto:Token: None 2024-04-12T04:19:59.493 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.493 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uW0vCBzsqpxYXSb2BJTO691fL7w= 2024-04-12T04:19:59.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uW0vCBzsqpxYXSb2BJTO691fL7w=', 'Content-Length': '0'} 2024-04-12T04:19:59.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d23c55bca553a61f-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T04:19:59.496 DEBUG:boto:path=/dkyges-63/obj62 2024-04-12T04:19:59.496 DEBUG:boto:auth_path=/dkyges-63/obj62 2024-04-12T04:19:59.496 DEBUG:boto:path=/dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.496 DEBUG:boto:auth_path=/dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.496 DEBUG:boto:Method: GET 2024-04-12T04:19:59.497 DEBUG:boto:Path: /dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.497 DEBUG:boto:Data: 2024-04-12T04:19:59.497 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.497 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.497 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.498 DEBUG:boto:Params: {} 2024-04-12T04:19:59.498 DEBUG:boto:Token: None 2024-04-12T04:19:59.498 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.498 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3VNVknVXwWcKiFzuHMcuU+IDZqU= 2024-04-12T04:19:59.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3VNVknVXwWcKiFzuHMcuU+IDZqU=', 'Content-Length': '0'} 2024-04-12T04:19:59.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d751f99530804eb-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.542 DEBUG:boto:path=/dkyges-63/obj62 2024-04-12T04:19:59.543 DEBUG:boto:auth_path=/dkyges-63/obj62 2024-04-12T04:19:59.543 DEBUG:boto:path=/dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.543 DEBUG:boto:auth_path=/dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.543 DEBUG:boto:Method: GET 2024-04-12T04:19:59.544 DEBUG:boto:Path: /dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.544 DEBUG:boto:Data: 2024-04-12T04:19:59.544 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.544 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.544 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.544 DEBUG:boto:Params: {} 2024-04-12T04:19:59.545 DEBUG:boto:Token: None 2024-04-12T04:19:59.545 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj62?versionId=null 2024-04-12T04:19:59.545 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3VNVknVXwWcKiFzuHMcuU+IDZqU= 2024-04-12T04:19:59.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3VNVknVXwWcKiFzuHMcuU+IDZqU=', 'Content-Length': '0'} 2024-04-12T04:19:59.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d67979a5a8608878-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.548 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.548 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.548 DEBUG:boto:path=/dkyges-63/?prefix=obj62 2024-04-12T04:19:59.549 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj62 2024-04-12T04:19:59.549 DEBUG:boto:Method: GET 2024-04-12T04:19:59.549 DEBUG:boto:Path: /dkyges-63/?prefix=obj62 2024-04-12T04:19:59.549 DEBUG:boto:Data: 2024-04-12T04:19:59.549 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.549 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.550 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.550 DEBUG:boto:Params: {} 2024-04-12T04:19:59.550 DEBUG:boto:Token: None 2024-04-12T04:19:59.550 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.551 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088382e6e20020d10-006618b66f-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.554 DEBUG:boto:b'dkyges-63obj621000falseobj622024-04-12T04:15:49.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.554 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.555 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.555 DEBUG:boto:path=/dkyges-63/?prefix=obj62 2024-04-12T04:19:59.555 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj62 2024-04-12T04:19:59.555 DEBUG:boto:Method: GET 2024-04-12T04:19:59.555 DEBUG:boto:Path: /dkyges-63/?prefix=obj62 2024-04-12T04:19:59.556 DEBUG:boto:Data: 2024-04-12T04:19:59.556 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.556 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.556 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.556 DEBUG:boto:Params: {} 2024-04-12T04:19:59.556 DEBUG:boto:Token: None 2024-04-12T04:19:59.557 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.557 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b896deae8f927478-006618b66f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.602 DEBUG:boto:b'dkyges-63obj621000falseobj622024-04-12T04:15:49.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.603 DEBUG:boto:path=/dkyges-63/obj62 2024-04-12T04:19:59.603 DEBUG:boto:auth_path=/dkyges-63/obj62 2024-04-12T04:19:59.603 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.603 DEBUG:boto:Path: /dkyges-63/obj62 2024-04-12T04:19:59.603 DEBUG:boto:Data: 2024-04-12T04:19:59.603 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.603 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.603 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.604 DEBUG:boto:Params: {} 2024-04-12T04:19:59.604 DEBUG:boto:Token: None 2024-04-12T04:19:59.604 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj62 2024-04-12T04:19:59.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ba1YAVHiNrr8vc6q1EHFSxrw4EA= 2024-04-12T04:19:59.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ba1YAVHiNrr8vc6q1EHFSxrw4EA=', 'Content-Length': '0'} 2024-04-12T04:19:59.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009398711babf42762-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.606 DEBUG:boto:path=/dkyges-63/obj62 2024-04-12T04:19:59.606 DEBUG:boto:auth_path=/dkyges-63/obj62 2024-04-12T04:19:59.606 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.607 DEBUG:boto:Path: /dkyges-63/obj62 2024-04-12T04:19:59.607 DEBUG:boto:Data: 2024-04-12T04:19:59.607 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.607 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.607 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.607 DEBUG:boto:Params: {} 2024-04-12T04:19:59.607 DEBUG:boto:Token: None 2024-04-12T04:19:59.607 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj62 2024-04-12T04:19:59.607 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ba1YAVHiNrr8vc6q1EHFSxrw4EA= 2024-04-12T04:19:59.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ba1YAVHiNrr8vc6q1EHFSxrw4EA=', 'Content-Length': '0'} 2024-04-12T04:19:59.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eec5bef7b5e5a118-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2024-04-12T04:19:59.610 DEBUG:boto:path=/dkyges-63/obj62 2024-04-12T04:19:59.610 DEBUG:boto:auth_path=/dkyges-63/obj62 2024-04-12T04:19:59.610 DEBUG:boto:Method: GET 2024-04-12T04:19:59.610 DEBUG:boto:Path: /dkyges-63/obj62 2024-04-12T04:19:59.610 DEBUG:boto:Data: 2024-04-12T04:19:59.610 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.610 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.610 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.610 DEBUG:boto:Params: {} 2024-04-12T04:19:59.610 DEBUG:boto:Token: None 2024-04-12T04:19:59.611 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj62 2024-04-12T04:19:59.611 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wlRaXtHK2ABI9kktIBjjwTgf024= 2024-04-12T04:19:59.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wlRaXtHK2ABI9kktIBjjwTgf024=', 'Content-Length': '0'} 2024-04-12T04:19:59.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000248bd91020809768-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.613 DEBUG:boto:path=/dkyges-63/obj62 2024-04-12T04:19:59.613 DEBUG:boto:auth_path=/dkyges-63/obj62 2024-04-12T04:19:59.613 DEBUG:boto:Method: GET 2024-04-12T04:19:59.613 DEBUG:boto:Path: /dkyges-63/obj62 2024-04-12T04:19:59.613 DEBUG:boto:Data: 2024-04-12T04:19:59.613 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.613 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.613 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.613 DEBUG:boto:Params: {} 2024-04-12T04:19:59.614 DEBUG:boto:Token: None 2024-04-12T04:19:59.614 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj62 2024-04-12T04:19:59.614 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wlRaXtHK2ABI9kktIBjjwTgf024= 2024-04-12T04:19:59.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wlRaXtHK2ABI9kktIBjjwTgf024=', 'Content-Length': '0'} 2024-04-12T04:19:59.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c57fc8b07aa5af97-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.616 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.616 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.616 DEBUG:boto:path=/dkyges-63/?prefix=obj62 2024-04-12T04:19:59.616 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj62 2024-04-12T04:19:59.616 DEBUG:boto:Method: GET 2024-04-12T04:19:59.616 DEBUG:boto:Path: /dkyges-63/?prefix=obj62 2024-04-12T04:19:59.616 DEBUG:boto:Data: 2024-04-12T04:19:59.617 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.617 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.617 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.617 DEBUG:boto:Params: {} 2024-04-12T04:19:59.617 DEBUG:boto:Token: None 2024-04-12T04:19:59.617 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.617 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e0088bdb2228a69-006618b66f-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.620 DEBUG:boto:b'dkyges-63obj621000falseobj622024-04-12T04:15:49.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.620 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.620 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.620 DEBUG:boto:path=/dkyges-63/?prefix=obj62 2024-04-12T04:19:59.620 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj62 2024-04-12T04:19:59.620 DEBUG:boto:Method: GET 2024-04-12T04:19:59.621 DEBUG:boto:Path: /dkyges-63/?prefix=obj62 2024-04-12T04:19:59.621 DEBUG:boto:Data: 2024-04-12T04:19:59.621 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.621 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.621 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.621 DEBUG:boto:Params: {} 2024-04-12T04:19:59.621 DEBUG:boto:Token: None 2024-04-12T04:19:59.621 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.621 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e000a6f715ad59f-006618b66f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.626 DEBUG:boto:b'dkyges-63obj621000falseobj622024-04-12T04:15:49.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:19:59.627 DEBUG:boto:path=/dkyges-63/obj68 2024-04-12T04:19:59.627 DEBUG:boto:auth_path=/dkyges-63/obj68 2024-04-12T04:19:59.627 DEBUG:boto:path=/dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.627 DEBUG:boto:auth_path=/dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.627 DEBUG:boto:Method: GET 2024-04-12T04:19:59.627 DEBUG:boto:Path: /dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.627 DEBUG:boto:Data: 2024-04-12T04:19:59.627 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.627 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.627 DEBUG:boto:Params: {} 2024-04-12T04:19:59.628 DEBUG:boto:Token: None 2024-04-12T04:19:59.628 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MgqC7LEyli/RSlg6B1W+wQ2Ovu4= 2024-04-12T04:19:59.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MgqC7LEyli/RSlg6B1W+wQ2Ovu4=', 'Content-Length': '0'} 2024-04-12T04:19:59.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079f2f73f94c2bc2f-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.630 DEBUG:boto:path=/dkyges-63/obj68 2024-04-12T04:19:59.630 DEBUG:boto:auth_path=/dkyges-63/obj68 2024-04-12T04:19:59.630 DEBUG:boto:path=/dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.630 DEBUG:boto:auth_path=/dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.630 DEBUG:boto:Method: GET 2024-04-12T04:19:59.631 DEBUG:boto:Path: /dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.631 DEBUG:boto:Data: 2024-04-12T04:19:59.631 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.631 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.631 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.631 DEBUG:boto:Params: {} 2024-04-12T04:19:59.631 DEBUG:boto:Token: None 2024-04-12T04:19:59.631 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.631 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MgqC7LEyli/RSlg6B1W+wQ2Ovu4= 2024-04-12T04:19:59.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MgqC7LEyli/RSlg6B1W+wQ2Ovu4=', 'Content-Length': '0'} 2024-04-12T04:19:59.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000163ef047a32663a2-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.634 DEBUG:boto:path=/dkyges-63/obj68 2024-04-12T04:19:59.634 DEBUG:boto:auth_path=/dkyges-63/obj68 2024-04-12T04:19:59.634 DEBUG:boto:path=/dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.634 DEBUG:boto:auth_path=/dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.634 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.634 DEBUG:boto:Path: /dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.634 DEBUG:boto:Data: 2024-04-12T04:19:59.634 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.634 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.634 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.634 DEBUG:boto:Params: {} 2024-04-12T04:19:59.635 DEBUG:boto:Token: None 2024-04-12T04:19:59.635 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.635 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W9msBgxSdB5TaSUOy6iENMrMm+o= 2024-04-12T04:19:59.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W9msBgxSdB5TaSUOy6iENMrMm+o=', 'Content-Length': '0'} 2024-04-12T04:19:59.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e024538b6154271-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.637 DEBUG:boto:path=/dkyges-63/obj68 2024-04-12T04:19:59.637 DEBUG:boto:auth_path=/dkyges-63/obj68 2024-04-12T04:19:59.637 DEBUG:boto:path=/dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.637 DEBUG:boto:auth_path=/dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.637 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.637 DEBUG:boto:Path: /dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.637 DEBUG:boto:Data: 2024-04-12T04:19:59.637 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.637 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.638 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.638 DEBUG:boto:Params: {} 2024-04-12T04:19:59.638 DEBUG:boto:Token: None 2024-04-12T04:19:59.638 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.638 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:W9msBgxSdB5TaSUOy6iENMrMm+o= 2024-04-12T04:19:59.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:W9msBgxSdB5TaSUOy6iENMrMm+o=', 'Content-Length': '0'} 2024-04-12T04:19:59.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f720f8b24ecf0ab9-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:19:59.640 DEBUG:boto:path=/dkyges-63/obj68 2024-04-12T04:19:59.640 DEBUG:boto:auth_path=/dkyges-63/obj68 2024-04-12T04:19:59.640 DEBUG:boto:path=/dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.640 DEBUG:boto:auth_path=/dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.641 DEBUG:boto:Method: GET 2024-04-12T04:19:59.641 DEBUG:boto:Path: /dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.641 DEBUG:boto:Data: 2024-04-12T04:19:59.641 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.641 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.641 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.641 DEBUG:boto:Params: {} 2024-04-12T04:19:59.641 DEBUG:boto:Token: None 2024-04-12T04:19:59.641 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.641 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MgqC7LEyli/RSlg6B1W+wQ2Ovu4= 2024-04-12T04:19:59.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MgqC7LEyli/RSlg6B1W+wQ2Ovu4=', 'Content-Length': '0'} 2024-04-12T04:19:59.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd45177c11004ed2-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.643 DEBUG:boto:path=/dkyges-63/obj68 2024-04-12T04:19:59.643 DEBUG:boto:auth_path=/dkyges-63/obj68 2024-04-12T04:19:59.644 DEBUG:boto:path=/dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.644 DEBUG:boto:auth_path=/dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.644 DEBUG:boto:Method: GET 2024-04-12T04:19:59.644 DEBUG:boto:Path: /dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.644 DEBUG:boto:Data: 2024-04-12T04:19:59.644 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.644 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.644 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.644 DEBUG:boto:Params: {} 2024-04-12T04:19:59.644 DEBUG:boto:Token: None 2024-04-12T04:19:59.644 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj68?versionId=null 2024-04-12T04:19:59.645 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MgqC7LEyli/RSlg6B1W+wQ2Ovu4= 2024-04-12T04:19:59.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MgqC7LEyli/RSlg6B1W+wQ2Ovu4=', 'Content-Length': '0'} 2024-04-12T04:19:59.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b00fd07efc46037e-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.647 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.647 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.647 DEBUG:boto:path=/dkyges-63/?prefix=obj68 2024-04-12T04:19:59.647 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj68 2024-04-12T04:19:59.647 DEBUG:boto:Method: GET 2024-04-12T04:19:59.647 DEBUG:boto:Path: /dkyges-63/?prefix=obj68 2024-04-12T04:19:59.647 DEBUG:boto:Data: 2024-04-12T04:19:59.647 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.647 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.648 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.648 DEBUG:boto:Params: {} 2024-04-12T04:19:59.648 DEBUG:boto:Token: None 2024-04-12T04:19:59.648 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.648 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.651 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a67710608bced555-006618b66f-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.694 DEBUG:boto:b'dkyges-63obj681000falseobj682024-04-12T04:15:47.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.695 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.695 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.695 DEBUG:boto:path=/dkyges-63/?prefix=obj68 2024-04-12T04:19:59.695 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj68 2024-04-12T04:19:59.695 DEBUG:boto:Method: GET 2024-04-12T04:19:59.695 DEBUG:boto:Path: /dkyges-63/?prefix=obj68 2024-04-12T04:19:59.695 DEBUG:boto:Data: 2024-04-12T04:19:59.695 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.695 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.695 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.696 DEBUG:boto:Params: {} 2024-04-12T04:19:59.696 DEBUG:boto:Token: None 2024-04-12T04:19:59.696 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.696 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075e91e98e00ce46a-006618b66f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.701 DEBUG:boto:b'dkyges-63obj681000falseobj682024-04-12T04:15:47.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.701 DEBUG:boto:path=/dkyges-63/obj68 2024-04-12T04:19:59.701 DEBUG:boto:auth_path=/dkyges-63/obj68 2024-04-12T04:19:59.701 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.701 DEBUG:boto:Path: /dkyges-63/obj68 2024-04-12T04:19:59.701 DEBUG:boto:Data: 2024-04-12T04:19:59.701 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.701 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.702 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.702 DEBUG:boto:Params: {} 2024-04-12T04:19:59.702 DEBUG:boto:Token: None 2024-04-12T04:19:59.702 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj68 2024-04-12T04:19:59.702 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jrr+zP/Lbg95uQ0C7wZ3UzZQjVk= 2024-04-12T04:19:59.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jrr+zP/Lbg95uQ0C7wZ3UzZQjVk=', 'Content-Length': '0'} 2024-04-12T04:19:59.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000708ef03cae4222af-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.704 DEBUG:boto:path=/dkyges-63/obj68 2024-04-12T04:19:59.704 DEBUG:boto:auth_path=/dkyges-63/obj68 2024-04-12T04:19:59.705 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.705 DEBUG:boto:Path: /dkyges-63/obj68 2024-04-12T04:19:59.705 DEBUG:boto:Data: 2024-04-12T04:19:59.705 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.705 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.705 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.705 DEBUG:boto:Params: {} 2024-04-12T04:19:59.705 DEBUG:boto:Token: None 2024-04-12T04:19:59.705 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj68 2024-04-12T04:19:59.705 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jrr+zP/Lbg95uQ0C7wZ3UzZQjVk= 2024-04-12T04:19:59.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jrr+zP/Lbg95uQ0C7wZ3UzZQjVk=', 'Content-Length': '0'} 2024-04-12T04:19:59.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012b09a7b461a2f05-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2024-04-12T04:19:59.708 DEBUG:boto:path=/dkyges-63/obj68 2024-04-12T04:19:59.708 DEBUG:boto:auth_path=/dkyges-63/obj68 2024-04-12T04:19:59.708 DEBUG:boto:Method: GET 2024-04-12T04:19:59.708 DEBUG:boto:Path: /dkyges-63/obj68 2024-04-12T04:19:59.708 DEBUG:boto:Data: 2024-04-12T04:19:59.708 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.708 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.708 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.709 DEBUG:boto:Params: {} 2024-04-12T04:19:59.709 DEBUG:boto:Token: None 2024-04-12T04:19:59.709 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj68 2024-04-12T04:19:59.709 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UURyOSnnuvqT9lXnIBA0BIFLfYE= 2024-04-12T04:19:59.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UURyOSnnuvqT9lXnIBA0BIFLfYE=', 'Content-Length': '0'} 2024-04-12T04:19:59.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017dc03f17b9e126e-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.711 DEBUG:boto:path=/dkyges-63/obj68 2024-04-12T04:19:59.711 DEBUG:boto:auth_path=/dkyges-63/obj68 2024-04-12T04:19:59.711 DEBUG:boto:Method: GET 2024-04-12T04:19:59.711 DEBUG:boto:Path: /dkyges-63/obj68 2024-04-12T04:19:59.712 DEBUG:boto:Data: 2024-04-12T04:19:59.712 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.712 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.712 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.712 DEBUG:boto:Params: {} 2024-04-12T04:19:59.712 DEBUG:boto:Token: None 2024-04-12T04:19:59.712 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj68 2024-04-12T04:19:59.712 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UURyOSnnuvqT9lXnIBA0BIFLfYE= 2024-04-12T04:19:59.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UURyOSnnuvqT9lXnIBA0BIFLfYE=', 'Content-Length': '0'} 2024-04-12T04:19:59.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000932a109b10aa804c-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.754 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.755 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.755 DEBUG:boto:path=/dkyges-63/?prefix=obj68 2024-04-12T04:19:59.755 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj68 2024-04-12T04:19:59.755 DEBUG:boto:Method: GET 2024-04-12T04:19:59.755 DEBUG:boto:Path: /dkyges-63/?prefix=obj68 2024-04-12T04:19:59.755 DEBUG:boto:Data: 2024-04-12T04:19:59.755 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.755 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.755 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.755 DEBUG:boto:Params: {} 2024-04-12T04:19:59.756 DEBUG:boto:Token: None 2024-04-12T04:19:59.756 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.756 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecbe65addd153345-006618b66f-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.758 DEBUG:boto:b'dkyges-63obj681000falseobj682024-04-12T04:15:47.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.759 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.759 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.759 DEBUG:boto:path=/dkyges-63/?prefix=obj68 2024-04-12T04:19:59.759 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj68 2024-04-12T04:19:59.759 DEBUG:boto:Method: GET 2024-04-12T04:19:59.759 DEBUG:boto:Path: /dkyges-63/?prefix=obj68 2024-04-12T04:19:59.759 DEBUG:boto:Data: 2024-04-12T04:19:59.760 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.760 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.760 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.760 DEBUG:boto:Params: {} 2024-04-12T04:19:59.760 DEBUG:boto:Token: None 2024-04-12T04:19:59.760 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.760 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000877f3ddf477355fc-006618b66f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.765 DEBUG:boto:b'dkyges-63obj681000falseobj682024-04-12T04:15:47.869Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:19:59.765 DEBUG:boto:path=/dkyges-63/obj69 2024-04-12T04:19:59.766 DEBUG:boto:auth_path=/dkyges-63/obj69 2024-04-12T04:19:59.766 DEBUG:boto:path=/dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.766 DEBUG:boto:auth_path=/dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.766 DEBUG:boto:Method: GET 2024-04-12T04:19:59.766 DEBUG:boto:Path: /dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.766 DEBUG:boto:Data: 2024-04-12T04:19:59.766 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.766 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.766 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.766 DEBUG:boto:Params: {} 2024-04-12T04:19:59.766 DEBUG:boto:Token: None 2024-04-12T04:19:59.767 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.767 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TOBvn2nbNH5F3x953ngZJU7aoJo= 2024-04-12T04:19:59.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TOBvn2nbNH5F3x953ngZJU7aoJo=', 'Content-Length': '0'} 2024-04-12T04:19:59.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc0f1e5398e794f7-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.769 DEBUG:boto:path=/dkyges-63/obj69 2024-04-12T04:19:59.769 DEBUG:boto:auth_path=/dkyges-63/obj69 2024-04-12T04:19:59.769 DEBUG:boto:path=/dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.769 DEBUG:boto:auth_path=/dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.769 DEBUG:boto:Method: GET 2024-04-12T04:19:59.769 DEBUG:boto:Path: /dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.770 DEBUG:boto:Data: 2024-04-12T04:19:59.770 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.770 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.770 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.770 DEBUG:boto:Params: {} 2024-04-12T04:19:59.770 DEBUG:boto:Token: None 2024-04-12T04:19:59.770 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.770 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TOBvn2nbNH5F3x953ngZJU7aoJo= 2024-04-12T04:19:59.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TOBvn2nbNH5F3x953ngZJU7aoJo=', 'Content-Length': '0'} 2024-04-12T04:19:59.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082963b471fc4c903-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.773 DEBUG:boto:path=/dkyges-63/obj69 2024-04-12T04:19:59.773 DEBUG:boto:auth_path=/dkyges-63/obj69 2024-04-12T04:19:59.773 DEBUG:boto:path=/dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.773 DEBUG:boto:auth_path=/dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.773 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.773 DEBUG:boto:Path: /dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.773 DEBUG:boto:Data: 2024-04-12T04:19:59.773 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.773 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.773 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.774 DEBUG:boto:Params: {} 2024-04-12T04:19:59.774 DEBUG:boto:Token: None 2024-04-12T04:19:59.774 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.774 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GHfmO5HvDBIayUY3c9jSq/ybSKA= 2024-04-12T04:19:59.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GHfmO5HvDBIayUY3c9jSq/ybSKA=', 'Content-Length': '0'} 2024-04-12T04:19:59.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006e0be14fe3bf9c6-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.776 DEBUG:boto:path=/dkyges-63/obj69 2024-04-12T04:19:59.776 DEBUG:boto:auth_path=/dkyges-63/obj69 2024-04-12T04:19:59.776 DEBUG:boto:path=/dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.776 DEBUG:boto:auth_path=/dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.777 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.777 DEBUG:boto:Path: /dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.777 DEBUG:boto:Data: 2024-04-12T04:19:59.777 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.777 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.777 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.777 DEBUG:boto:Params: {} 2024-04-12T04:19:59.777 DEBUG:boto:Token: None 2024-04-12T04:19:59.777 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.777 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GHfmO5HvDBIayUY3c9jSq/ybSKA= 2024-04-12T04:19:59.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GHfmO5HvDBIayUY3c9jSq/ybSKA=', 'Content-Length': '0'} 2024-04-12T04:19:59.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5cbdb0d3e08b69a-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:19:59.780 DEBUG:boto:path=/dkyges-63/obj69 2024-04-12T04:19:59.780 DEBUG:boto:auth_path=/dkyges-63/obj69 2024-04-12T04:19:59.780 DEBUG:boto:path=/dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.780 DEBUG:boto:auth_path=/dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.780 DEBUG:boto:Method: GET 2024-04-12T04:19:59.780 DEBUG:boto:Path: /dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.780 DEBUG:boto:Data: 2024-04-12T04:19:59.780 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.780 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.780 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.780 DEBUG:boto:Params: {} 2024-04-12T04:19:59.781 DEBUG:boto:Token: None 2024-04-12T04:19:59.781 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.781 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TOBvn2nbNH5F3x953ngZJU7aoJo= 2024-04-12T04:19:59.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TOBvn2nbNH5F3x953ngZJU7aoJo=', 'Content-Length': '0'} 2024-04-12T04:19:59.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c59adb9b5d2d4e8-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.783 DEBUG:boto:path=/dkyges-63/obj69 2024-04-12T04:19:59.783 DEBUG:boto:auth_path=/dkyges-63/obj69 2024-04-12T04:19:59.783 DEBUG:boto:path=/dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.783 DEBUG:boto:auth_path=/dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.783 DEBUG:boto:Method: GET 2024-04-12T04:19:59.783 DEBUG:boto:Path: /dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.784 DEBUG:boto:Data: 2024-04-12T04:19:59.784 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.784 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.784 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.784 DEBUG:boto:Params: {} 2024-04-12T04:19:59.784 DEBUG:boto:Token: None 2024-04-12T04:19:59.784 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj69?versionId=null 2024-04-12T04:19:59.784 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TOBvn2nbNH5F3x953ngZJU7aoJo= 2024-04-12T04:19:59.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TOBvn2nbNH5F3x953ngZJU7aoJo=', 'Content-Length': '0'} 2024-04-12T04:19:59.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee8b6e2c4db2652f-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.826 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.827 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.827 DEBUG:boto:path=/dkyges-63/?prefix=obj69 2024-04-12T04:19:59.827 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj69 2024-04-12T04:19:59.827 DEBUG:boto:Method: GET 2024-04-12T04:19:59.827 DEBUG:boto:Path: /dkyges-63/?prefix=obj69 2024-04-12T04:19:59.827 DEBUG:boto:Data: 2024-04-12T04:19:59.827 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.827 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.827 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.827 DEBUG:boto:Params: {} 2024-04-12T04:19:59.827 DEBUG:boto:Token: None 2024-04-12T04:19:59.828 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.828 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.830 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de48ed39127c739e-006618b66f-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.831 DEBUG:boto:b'dkyges-63obj691000falseobj692024-04-12T04:15:48.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.831 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.831 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.831 DEBUG:boto:path=/dkyges-63/?prefix=obj69 2024-04-12T04:19:59.831 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj69 2024-04-12T04:19:59.832 DEBUG:boto:Method: GET 2024-04-12T04:19:59.832 DEBUG:boto:Path: /dkyges-63/?prefix=obj69 2024-04-12T04:19:59.832 DEBUG:boto:Data: 2024-04-12T04:19:59.832 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.832 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.832 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.832 DEBUG:boto:Params: {} 2024-04-12T04:19:59.832 DEBUG:boto:Token: None 2024-04-12T04:19:59.832 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.832 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f79388500f7bd8f-006618b66f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.837 DEBUG:boto:b'dkyges-63obj691000falseobj692024-04-12T04:15:48.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.838 DEBUG:boto:path=/dkyges-63/obj69 2024-04-12T04:19:59.838 DEBUG:boto:auth_path=/dkyges-63/obj69 2024-04-12T04:19:59.838 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.838 DEBUG:boto:Path: /dkyges-63/obj69 2024-04-12T04:19:59.838 DEBUG:boto:Data: 2024-04-12T04:19:59.838 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.838 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.838 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.838 DEBUG:boto:Params: {} 2024-04-12T04:19:59.838 DEBUG:boto:Token: None 2024-04-12T04:19:59.839 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj69 2024-04-12T04:19:59.839 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:filGalC5cjlG47jz4V12ELyeKfI= 2024-04-12T04:19:59.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:filGalC5cjlG47jz4V12ELyeKfI=', 'Content-Length': '0'} 2024-04-12T04:19:59.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a3b32383fd07eec-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.841 DEBUG:boto:path=/dkyges-63/obj69 2024-04-12T04:19:59.841 DEBUG:boto:auth_path=/dkyges-63/obj69 2024-04-12T04:19:59.841 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.841 DEBUG:boto:Path: /dkyges-63/obj69 2024-04-12T04:19:59.841 DEBUG:boto:Data: 2024-04-12T04:19:59.841 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.842 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.842 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.842 DEBUG:boto:Params: {} 2024-04-12T04:19:59.842 DEBUG:boto:Token: None 2024-04-12T04:19:59.842 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj69 2024-04-12T04:19:59.842 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:filGalC5cjlG47jz4V12ELyeKfI= 2024-04-12T04:19:59.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:filGalC5cjlG47jz4V12ELyeKfI=', 'Content-Length': '0'} 2024-04-12T04:19:59.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7c77f2ab5c193b9-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2024-04-12T04:19:59.844 DEBUG:boto:path=/dkyges-63/obj69 2024-04-12T04:19:59.845 DEBUG:boto:auth_path=/dkyges-63/obj69 2024-04-12T04:19:59.845 DEBUG:boto:Method: GET 2024-04-12T04:19:59.845 DEBUG:boto:Path: /dkyges-63/obj69 2024-04-12T04:19:59.845 DEBUG:boto:Data: 2024-04-12T04:19:59.845 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.845 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.845 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.845 DEBUG:boto:Params: {} 2024-04-12T04:19:59.845 DEBUG:boto:Token: None 2024-04-12T04:19:59.845 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj69 2024-04-12T04:19:59.845 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9d29lCNiFpZSUpqtlQs7RI9YPG8= 2024-04-12T04:19:59.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9d29lCNiFpZSUpqtlQs7RI9YPG8=', 'Content-Length': '0'} 2024-04-12T04:19:59.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd0cc207ccfd9121-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.890 DEBUG:boto:path=/dkyges-63/obj69 2024-04-12T04:19:59.890 DEBUG:boto:auth_path=/dkyges-63/obj69 2024-04-12T04:19:59.890 DEBUG:boto:Method: GET 2024-04-12T04:19:59.891 DEBUG:boto:Path: /dkyges-63/obj69 2024-04-12T04:19:59.891 DEBUG:boto:Data: 2024-04-12T04:19:59.891 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.891 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.891 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.891 DEBUG:boto:Params: {} 2024-04-12T04:19:59.891 DEBUG:boto:Token: None 2024-04-12T04:19:59.891 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj69 2024-04-12T04:19:59.891 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9d29lCNiFpZSUpqtlQs7RI9YPG8= 2024-04-12T04:19:59.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9d29lCNiFpZSUpqtlQs7RI9YPG8=', 'Content-Length': '0'} 2024-04-12T04:19:59.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089eb97a76cb6e5ee-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.893 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.893 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.894 DEBUG:boto:path=/dkyges-63/?prefix=obj69 2024-04-12T04:19:59.894 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj69 2024-04-12T04:19:59.894 DEBUG:boto:Method: GET 2024-04-12T04:19:59.894 DEBUG:boto:Path: /dkyges-63/?prefix=obj69 2024-04-12T04:19:59.894 DEBUG:boto:Data: 2024-04-12T04:19:59.894 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.894 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.894 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.894 DEBUG:boto:Params: {} 2024-04-12T04:19:59.894 DEBUG:boto:Token: None 2024-04-12T04:19:59.894 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.895 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe367e0cde44eda6-006618b66f-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.897 DEBUG:boto:b'dkyges-63obj691000falseobj692024-04-12T04:15:48.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.898 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.898 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.898 DEBUG:boto:path=/dkyges-63/?prefix=obj69 2024-04-12T04:19:59.898 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj69 2024-04-12T04:19:59.898 DEBUG:boto:Method: GET 2024-04-12T04:19:59.898 DEBUG:boto:Path: /dkyges-63/?prefix=obj69 2024-04-12T04:19:59.898 DEBUG:boto:Data: 2024-04-12T04:19:59.898 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.898 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.898 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.898 DEBUG:boto:Params: {} 2024-04-12T04:19:59.899 DEBUG:boto:Token: None 2024-04-12T04:19:59.899 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.899 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000270c35da89170c02-006618b66f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.903 DEBUG:boto:b'dkyges-63obj691000falseobj692024-04-12T04:15:48.502Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T04:19:59.904 DEBUG:boto:path=/dkyges-63/obj74 2024-04-12T04:19:59.904 DEBUG:boto:auth_path=/dkyges-63/obj74 2024-04-12T04:19:59.904 DEBUG:boto:path=/dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.904 DEBUG:boto:auth_path=/dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.904 DEBUG:boto:Method: GET 2024-04-12T04:19:59.904 DEBUG:boto:Path: /dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.904 DEBUG:boto:Data: 2024-04-12T04:19:59.904 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.904 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.904 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.904 DEBUG:boto:Params: {} 2024-04-12T04:19:59.905 DEBUG:boto:Token: None 2024-04-12T04:19:59.905 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.905 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3EXjeBdQEq9BMyEt/PkNoG1IcIM= 2024-04-12T04:19:59.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3EXjeBdQEq9BMyEt/PkNoG1IcIM=', 'Content-Length': '0'} 2024-04-12T04:19:59.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ede1d53ad3689a32-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.907 DEBUG:boto:path=/dkyges-63/obj74 2024-04-12T04:19:59.907 DEBUG:boto:auth_path=/dkyges-63/obj74 2024-04-12T04:19:59.907 DEBUG:boto:path=/dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.907 DEBUG:boto:auth_path=/dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.908 DEBUG:boto:Method: GET 2024-04-12T04:19:59.908 DEBUG:boto:Path: /dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.908 DEBUG:boto:Data: 2024-04-12T04:19:59.908 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.908 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.908 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.908 DEBUG:boto:Params: {} 2024-04-12T04:19:59.908 DEBUG:boto:Token: None 2024-04-12T04:19:59.908 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.908 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3EXjeBdQEq9BMyEt/PkNoG1IcIM= 2024-04-12T04:19:59.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3EXjeBdQEq9BMyEt/PkNoG1IcIM=', 'Content-Length': '0'} 2024-04-12T04:19:59.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000966a13f2c2e18312-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.950 DEBUG:boto:path=/dkyges-63/obj74 2024-04-12T04:19:59.950 DEBUG:boto:auth_path=/dkyges-63/obj74 2024-04-12T04:19:59.950 DEBUG:boto:path=/dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.950 DEBUG:boto:auth_path=/dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.951 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.951 DEBUG:boto:Path: /dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.951 DEBUG:boto:Data: 2024-04-12T04:19:59.951 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.951 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.951 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.951 DEBUG:boto:Params: {} 2024-04-12T04:19:59.951 DEBUG:boto:Token: None 2024-04-12T04:19:59.951 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.951 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8ctMTCqRj2d8IxG4RE7vFhOwiGY= 2024-04-12T04:19:59.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8ctMTCqRj2d8IxG4RE7vFhOwiGY=', 'Content-Length': '0'} 2024-04-12T04:19:59.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c4b45eed28c34d1-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.953 DEBUG:boto:path=/dkyges-63/obj74 2024-04-12T04:19:59.953 DEBUG:boto:auth_path=/dkyges-63/obj74 2024-04-12T04:19:59.953 DEBUG:boto:path=/dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.954 DEBUG:boto:auth_path=/dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.954 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.954 DEBUG:boto:Path: /dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.954 DEBUG:boto:Data: 2024-04-12T04:19:59.954 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.954 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.954 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.954 DEBUG:boto:Params: {} 2024-04-12T04:19:59.954 DEBUG:boto:Token: None 2024-04-12T04:19:59.954 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.954 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8ctMTCqRj2d8IxG4RE7vFhOwiGY= 2024-04-12T04:19:59.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8ctMTCqRj2d8IxG4RE7vFhOwiGY=', 'Content-Length': '0'} 2024-04-12T04:19:59.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3cb6f0ab2a3d1db-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T04:19:59.957 DEBUG:boto:path=/dkyges-63/obj74 2024-04-12T04:19:59.957 DEBUG:boto:auth_path=/dkyges-63/obj74 2024-04-12T04:19:59.957 DEBUG:boto:path=/dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.957 DEBUG:boto:auth_path=/dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.957 DEBUG:boto:Method: GET 2024-04-12T04:19:59.957 DEBUG:boto:Path: /dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.957 DEBUG:boto:Data: 2024-04-12T04:19:59.957 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.957 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.957 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.957 DEBUG:boto:Params: {} 2024-04-12T04:19:59.958 DEBUG:boto:Token: None 2024-04-12T04:19:59.958 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.958 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3EXjeBdQEq9BMyEt/PkNoG1IcIM= 2024-04-12T04:19:59.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3EXjeBdQEq9BMyEt/PkNoG1IcIM=', 'Content-Length': '0'} 2024-04-12T04:19:59.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096d13d18aac1da78-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.960 DEBUG:boto:path=/dkyges-63/obj74 2024-04-12T04:19:59.960 DEBUG:boto:auth_path=/dkyges-63/obj74 2024-04-12T04:19:59.960 DEBUG:boto:path=/dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.960 DEBUG:boto:auth_path=/dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.960 DEBUG:boto:Method: GET 2024-04-12T04:19:59.960 DEBUG:boto:Path: /dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.960 DEBUG:boto:Data: 2024-04-12T04:19:59.960 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.960 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.960 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.961 DEBUG:boto:Params: {} 2024-04-12T04:19:59.961 DEBUG:boto:Token: None 2024-04-12T04:19:59.961 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj74?versionId=null 2024-04-12T04:19:59.961 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3EXjeBdQEq9BMyEt/PkNoG1IcIM= 2024-04-12T04:19:59.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3EXjeBdQEq9BMyEt/PkNoG1IcIM=', 'Content-Length': '0'} 2024-04-12T04:19:59.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000571ed94078182e1a-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.963 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.963 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.963 DEBUG:boto:path=/dkyges-63/?prefix=obj74 2024-04-12T04:19:59.963 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj74 2024-04-12T04:19:59.963 DEBUG:boto:Method: GET 2024-04-12T04:19:59.963 DEBUG:boto:Path: /dkyges-63/?prefix=obj74 2024-04-12T04:19:59.964 DEBUG:boto:Data: 2024-04-12T04:19:59.964 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.964 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.964 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.964 DEBUG:boto:Params: {} 2024-04-12T04:19:59.964 DEBUG:boto:Token: None 2024-04-12T04:19:59.964 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.964 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af6906eef2eb0f33-006618b66f-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.967 DEBUG:boto:b'dkyges-63obj741000falseobj742024-04-12T04:15:49.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.967 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:19:59.967 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:19:59.967 DEBUG:boto:path=/dkyges-63/?prefix=obj74 2024-04-12T04:19:59.968 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj74 2024-04-12T04:19:59.968 DEBUG:boto:Method: GET 2024-04-12T04:19:59.968 DEBUG:boto:Path: /dkyges-63/?prefix=obj74 2024-04-12T04:19:59.968 DEBUG:boto:Data: 2024-04-12T04:19:59.968 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.968 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.968 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.968 DEBUG:boto:Params: {} 2024-04-12T04:19:59.968 DEBUG:boto:Token: None 2024-04-12T04:19:59.968 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/ 2024-04-12T04:19:59.968 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs= 2024-04-12T04:19:59.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Qc3MQnePOtINhHAVImzsShBzVPs=', 'Content-Length': '0'} 2024-04-12T04:19:59.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cc605ab4624d5a6-006618b66f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.973 DEBUG:boto:b'dkyges-63obj741000falseobj742024-04-12T04:15:49.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:19:59.974 DEBUG:boto:path=/dkyges-63/obj74 2024-04-12T04:19:59.974 DEBUG:boto:auth_path=/dkyges-63/obj74 2024-04-12T04:19:59.974 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.974 DEBUG:boto:Path: /dkyges-63/obj74 2024-04-12T04:19:59.974 DEBUG:boto:Data: 2024-04-12T04:19:59.974 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.974 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.974 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.974 DEBUG:boto:Params: {} 2024-04-12T04:19:59.975 DEBUG:boto:Token: None 2024-04-12T04:19:59.975 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj74 2024-04-12T04:19:59.975 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GFeZfuqmdaBohxfTsHsErQkeltA= 2024-04-12T04:19:59.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GFeZfuqmdaBohxfTsHsErQkeltA=', 'Content-Length': '0'} 2024-04-12T04:19:59.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039107f5f6ff36def-006618b66f-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.977 DEBUG:boto:path=/dkyges-63/obj74 2024-04-12T04:19:59.977 DEBUG:boto:auth_path=/dkyges-63/obj74 2024-04-12T04:19:59.977 DEBUG:boto:Method: HEAD 2024-04-12T04:19:59.977 DEBUG:boto:Path: /dkyges-63/obj74 2024-04-12T04:19:59.977 DEBUG:boto:Data: 2024-04-12T04:19:59.977 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.978 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.978 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.978 DEBUG:boto:Params: {} 2024-04-12T04:19:59.978 DEBUG:boto:Token: None 2024-04-12T04:19:59.978 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj74 2024-04-12T04:19:59.978 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GFeZfuqmdaBohxfTsHsErQkeltA= 2024-04-12T04:19:59.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GFeZfuqmdaBohxfTsHsErQkeltA=', 'Content-Length': '0'} 2024-04-12T04:19:59.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b42402da7fe22d9-006618b66f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:19:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2024-04-12T04:19:59.980 DEBUG:boto:path=/dkyges-63/obj74 2024-04-12T04:19:59.981 DEBUG:boto:auth_path=/dkyges-63/obj74 2024-04-12T04:19:59.981 DEBUG:boto:Method: GET 2024-04-12T04:19:59.981 DEBUG:boto:Path: /dkyges-63/obj74 2024-04-12T04:19:59.981 DEBUG:boto:Data: 2024-04-12T04:19:59.981 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.981 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:19:59.981 DEBUG:boto:Port: 8001 2024-04-12T04:19:59.981 DEBUG:boto:Params: {} 2024-04-12T04:19:59.981 DEBUG:boto:Token: None 2024-04-12T04:19:59.981 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj74 2024-04-12T04:19:59.981 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j55dyZSmp3PXXxub6ypBVWDK6+4= 2024-04-12T04:19:59.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j55dyZSmp3PXXxub6ypBVWDK6+4=', 'Content-Length': '0'} 2024-04-12T04:19:59.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b70c9e92decdd871-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:19:59.984 DEBUG:boto:path=/dkyges-63/obj74 2024-04-12T04:19:59.984 DEBUG:boto:auth_path=/dkyges-63/obj74 2024-04-12T04:19:59.984 DEBUG:boto:Method: GET 2024-04-12T04:19:59.984 DEBUG:boto:Path: /dkyges-63/obj74 2024-04-12T04:19:59.984 DEBUG:boto:Data: 2024-04-12T04:19:59.984 DEBUG:boto:Headers: {} 2024-04-12T04:19:59.984 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:19:59.984 DEBUG:boto:Port: 8000 2024-04-12T04:19:59.984 DEBUG:boto:Params: {} 2024-04-12T04:19:59.984 DEBUG:boto:Token: None 2024-04-12T04:19:59.985 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:19:59 GMT /dkyges-63/obj74 2024-04-12T04:19:59.985 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j55dyZSmp3PXXxub6ypBVWDK6+4= 2024-04-12T04:19:59.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:19:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j55dyZSmp3PXXxub6ypBVWDK6+4=', 'Content-Length': '0'} 2024-04-12T04:19:59.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb281dcf40dcebb5-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.030 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.031 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.031 DEBUG:boto:path=/dkyges-63/?prefix=obj74 2024-04-12T04:20:00.031 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj74 2024-04-12T04:20:00.031 DEBUG:boto:Method: GET 2024-04-12T04:20:00.031 DEBUG:boto:Path: /dkyges-63/?prefix=obj74 2024-04-12T04:20:00.031 DEBUG:boto:Data: 2024-04-12T04:20:00.031 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.031 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.031 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.031 DEBUG:boto:Params: {} 2024-04-12T04:20:00.032 DEBUG:boto:Token: None 2024-04-12T04:20:00.032 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.032 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b365d128f57abff-006618b670-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.035 DEBUG:boto:b'dkyges-63obj741000falseobj742024-04-12T04:15:49.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.035 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.035 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.035 DEBUG:boto:path=/dkyges-63/?prefix=obj74 2024-04-12T04:20:00.035 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj74 2024-04-12T04:20:00.036 DEBUG:boto:Method: GET 2024-04-12T04:20:00.036 DEBUG:boto:Path: /dkyges-63/?prefix=obj74 2024-04-12T04:20:00.036 DEBUG:boto:Data: 2024-04-12T04:20:00.036 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.036 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.036 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.036 DEBUG:boto:Params: {} 2024-04-12T04:20:00.036 DEBUG:boto:Token: None 2024-04-12T04:20:00.036 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.036 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b15cd1f1c45de0b-006618b670-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.041 DEBUG:boto:b'dkyges-63obj741000falseobj742024-04-12T04:15:49.607Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:20:00.042 DEBUG:boto:path=/dkyges-63/obj76 2024-04-12T04:20:00.042 DEBUG:boto:auth_path=/dkyges-63/obj76 2024-04-12T04:20:00.042 DEBUG:boto:path=/dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.042 DEBUG:boto:auth_path=/dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.042 DEBUG:boto:Method: GET 2024-04-12T04:20:00.042 DEBUG:boto:Path: /dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.042 DEBUG:boto:Data: 2024-04-12T04:20:00.042 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.042 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.042 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.042 DEBUG:boto:Params: {} 2024-04-12T04:20:00.043 DEBUG:boto:Token: None 2024-04-12T04:20:00.043 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.043 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eJMFs6VGGdWKYyLgM43rKwo2AzA= 2024-04-12T04:20:00.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eJMFs6VGGdWKYyLgM43rKwo2AzA=', 'Content-Length': '0'} 2024-04-12T04:20:00.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009991cc507fb4a9d5-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.086 DEBUG:boto:path=/dkyges-63/obj76 2024-04-12T04:20:00.086 DEBUG:boto:auth_path=/dkyges-63/obj76 2024-04-12T04:20:00.086 DEBUG:boto:path=/dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.087 DEBUG:boto:auth_path=/dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.087 DEBUG:boto:Method: GET 2024-04-12T04:20:00.087 DEBUG:boto:Path: /dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.087 DEBUG:boto:Data: 2024-04-12T04:20:00.087 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.087 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.087 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.087 DEBUG:boto:Params: {} 2024-04-12T04:20:00.087 DEBUG:boto:Token: None 2024-04-12T04:20:00.087 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.087 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eJMFs6VGGdWKYyLgM43rKwo2AzA= 2024-04-12T04:20:00.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eJMFs6VGGdWKYyLgM43rKwo2AzA=', 'Content-Length': '0'} 2024-04-12T04:20:00.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008b9690f665b9794-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.089 DEBUG:boto:path=/dkyges-63/obj76 2024-04-12T04:20:00.090 DEBUG:boto:auth_path=/dkyges-63/obj76 2024-04-12T04:20:00.090 DEBUG:boto:path=/dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.090 DEBUG:boto:auth_path=/dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.090 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.090 DEBUG:boto:Path: /dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.090 DEBUG:boto:Data: 2024-04-12T04:20:00.090 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.090 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.090 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.090 DEBUG:boto:Params: {} 2024-04-12T04:20:00.090 DEBUG:boto:Token: None 2024-04-12T04:20:00.091 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.091 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AzZD7bu0hfL1eEEroDbhrqthHz4= 2024-04-12T04:20:00.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AzZD7bu0hfL1eEEroDbhrqthHz4=', 'Content-Length': '0'} 2024-04-12T04:20:00.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ce68ae3d3b73133-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.092 DEBUG:boto:path=/dkyges-63/obj76 2024-04-12T04:20:00.093 DEBUG:boto:auth_path=/dkyges-63/obj76 2024-04-12T04:20:00.093 DEBUG:boto:path=/dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.093 DEBUG:boto:auth_path=/dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.093 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.093 DEBUG:boto:Path: /dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.093 DEBUG:boto:Data: 2024-04-12T04:20:00.093 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.093 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.093 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.093 DEBUG:boto:Params: {} 2024-04-12T04:20:00.093 DEBUG:boto:Token: None 2024-04-12T04:20:00.093 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.094 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AzZD7bu0hfL1eEEroDbhrqthHz4= 2024-04-12T04:20:00.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AzZD7bu0hfL1eEEroDbhrqthHz4=', 'Content-Length': '0'} 2024-04-12T04:20:00.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1391e831523318d-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:20:00.096 DEBUG:boto:path=/dkyges-63/obj76 2024-04-12T04:20:00.096 DEBUG:boto:auth_path=/dkyges-63/obj76 2024-04-12T04:20:00.096 DEBUG:boto:path=/dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.096 DEBUG:boto:auth_path=/dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.096 DEBUG:boto:Method: GET 2024-04-12T04:20:00.096 DEBUG:boto:Path: /dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.096 DEBUG:boto:Data: 2024-04-12T04:20:00.096 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.096 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.096 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.096 DEBUG:boto:Params: {} 2024-04-12T04:20:00.097 DEBUG:boto:Token: None 2024-04-12T04:20:00.097 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.097 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eJMFs6VGGdWKYyLgM43rKwo2AzA= 2024-04-12T04:20:00.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eJMFs6VGGdWKYyLgM43rKwo2AzA=', 'Content-Length': '0'} 2024-04-12T04:20:00.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df3342c4bfff9161-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.099 DEBUG:boto:path=/dkyges-63/obj76 2024-04-12T04:20:00.099 DEBUG:boto:auth_path=/dkyges-63/obj76 2024-04-12T04:20:00.099 DEBUG:boto:path=/dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.099 DEBUG:boto:auth_path=/dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.100 DEBUG:boto:Method: GET 2024-04-12T04:20:00.100 DEBUG:boto:Path: /dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.100 DEBUG:boto:Data: 2024-04-12T04:20:00.100 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.100 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.100 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.100 DEBUG:boto:Params: {} 2024-04-12T04:20:00.100 DEBUG:boto:Token: None 2024-04-12T04:20:00.100 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj76?versionId=null 2024-04-12T04:20:00.100 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eJMFs6VGGdWKYyLgM43rKwo2AzA= 2024-04-12T04:20:00.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eJMFs6VGGdWKYyLgM43rKwo2AzA=', 'Content-Length': '0'} 2024-04-12T04:20:00.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be633549ccf515de-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.103 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.103 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.103 DEBUG:boto:path=/dkyges-63/?prefix=obj76 2024-04-12T04:20:00.103 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj76 2024-04-12T04:20:00.103 DEBUG:boto:Method: GET 2024-04-12T04:20:00.103 DEBUG:boto:Path: /dkyges-63/?prefix=obj76 2024-04-12T04:20:00.103 DEBUG:boto:Data: 2024-04-12T04:20:00.103 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.103 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.103 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.104 DEBUG:boto:Params: {} 2024-04-12T04:20:00.104 DEBUG:boto:Token: None 2024-04-12T04:20:00.104 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.104 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000586481107b1a2188-006618b670-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.106 DEBUG:boto:b'dkyges-63obj761000falseobj762024-04-12T04:15:48.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.107 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.107 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.107 DEBUG:boto:path=/dkyges-63/?prefix=obj76 2024-04-12T04:20:00.107 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj76 2024-04-12T04:20:00.107 DEBUG:boto:Method: GET 2024-04-12T04:20:00.107 DEBUG:boto:Path: /dkyges-63/?prefix=obj76 2024-04-12T04:20:00.107 DEBUG:boto:Data: 2024-04-12T04:20:00.107 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.108 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.108 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.108 DEBUG:boto:Params: {} 2024-04-12T04:20:00.108 DEBUG:boto:Token: None 2024-04-12T04:20:00.108 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.108 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aff3e7c3ac1ccaae-006618b670-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.154 DEBUG:boto:b'dkyges-63obj761000falseobj762024-04-12T04:15:48.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.154 DEBUG:boto:path=/dkyges-63/obj76 2024-04-12T04:20:00.155 DEBUG:boto:auth_path=/dkyges-63/obj76 2024-04-12T04:20:00.155 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.155 DEBUG:boto:Path: /dkyges-63/obj76 2024-04-12T04:20:00.155 DEBUG:boto:Data: 2024-04-12T04:20:00.155 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.155 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.155 DEBUG:boto:Params: {} 2024-04-12T04:20:00.155 DEBUG:boto:Token: None 2024-04-12T04:20:00.156 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj76 2024-04-12T04:20:00.156 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ATzWGLtWFSHMj4NK5wAImE00tjw= 2024-04-12T04:20:00.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ATzWGLtWFSHMj4NK5wAImE00tjw=', 'Content-Length': '0'} 2024-04-12T04:20:00.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3f335b4d8cb9908-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.158 DEBUG:boto:path=/dkyges-63/obj76 2024-04-12T04:20:00.158 DEBUG:boto:auth_path=/dkyges-63/obj76 2024-04-12T04:20:00.158 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.158 DEBUG:boto:Path: /dkyges-63/obj76 2024-04-12T04:20:00.158 DEBUG:boto:Data: 2024-04-12T04:20:00.158 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.159 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.159 DEBUG:boto:Params: {} 2024-04-12T04:20:00.159 DEBUG:boto:Token: None 2024-04-12T04:20:00.159 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj76 2024-04-12T04:20:00.159 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ATzWGLtWFSHMj4NK5wAImE00tjw= 2024-04-12T04:20:00.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ATzWGLtWFSHMj4NK5wAImE00tjw=', 'Content-Length': '0'} 2024-04-12T04:20:00.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004983c114abe49f6d-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2024-04-12T04:20:00.161 DEBUG:boto:path=/dkyges-63/obj76 2024-04-12T04:20:00.161 DEBUG:boto:auth_path=/dkyges-63/obj76 2024-04-12T04:20:00.161 DEBUG:boto:Method: GET 2024-04-12T04:20:00.161 DEBUG:boto:Path: /dkyges-63/obj76 2024-04-12T04:20:00.161 DEBUG:boto:Data: 2024-04-12T04:20:00.161 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.162 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.162 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.162 DEBUG:boto:Params: {} 2024-04-12T04:20:00.162 DEBUG:boto:Token: None 2024-04-12T04:20:00.162 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj76 2024-04-12T04:20:00.162 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Jr7hTAWH5DKwHJri8hU5L/AEbaI= 2024-04-12T04:20:00.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Jr7hTAWH5DKwHJri8hU5L/AEbaI=', 'Content-Length': '0'} 2024-04-12T04:20:00.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058cb8dbd546052f4-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.164 DEBUG:boto:path=/dkyges-63/obj76 2024-04-12T04:20:00.164 DEBUG:boto:auth_path=/dkyges-63/obj76 2024-04-12T04:20:00.164 DEBUG:boto:Method: GET 2024-04-12T04:20:00.164 DEBUG:boto:Path: /dkyges-63/obj76 2024-04-12T04:20:00.164 DEBUG:boto:Data: 2024-04-12T04:20:00.164 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.164 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.164 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.165 DEBUG:boto:Params: {} 2024-04-12T04:20:00.165 DEBUG:boto:Token: None 2024-04-12T04:20:00.165 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj76 2024-04-12T04:20:00.165 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Jr7hTAWH5DKwHJri8hU5L/AEbaI= 2024-04-12T04:20:00.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Jr7hTAWH5DKwHJri8hU5L/AEbaI=', 'Content-Length': '0'} 2024-04-12T04:20:00.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd172dbd851f0704-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.167 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.167 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.167 DEBUG:boto:path=/dkyges-63/?prefix=obj76 2024-04-12T04:20:00.167 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj76 2024-04-12T04:20:00.168 DEBUG:boto:Method: GET 2024-04-12T04:20:00.168 DEBUG:boto:Path: /dkyges-63/?prefix=obj76 2024-04-12T04:20:00.168 DEBUG:boto:Data: 2024-04-12T04:20:00.168 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.168 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.168 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.168 DEBUG:boto:Params: {} 2024-04-12T04:20:00.168 DEBUG:boto:Token: None 2024-04-12T04:20:00.168 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.168 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011b7d7a0327b8812-006618b670-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.171 DEBUG:boto:b'dkyges-63obj761000falseobj762024-04-12T04:15:48.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.171 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.171 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.171 DEBUG:boto:path=/dkyges-63/?prefix=obj76 2024-04-12T04:20:00.171 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj76 2024-04-12T04:20:00.171 DEBUG:boto:Method: GET 2024-04-12T04:20:00.171 DEBUG:boto:Path: /dkyges-63/?prefix=obj76 2024-04-12T04:20:00.172 DEBUG:boto:Data: 2024-04-12T04:20:00.172 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.172 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.172 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.172 DEBUG:boto:Params: {} 2024-04-12T04:20:00.172 DEBUG:boto:Token: None 2024-04-12T04:20:00.172 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.172 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000658399bcafb7965b-006618b670-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.177 DEBUG:boto:b'dkyges-63obj761000falseobj762024-04-12T04:15:48.186Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:20:00.177 DEBUG:boto:path=/dkyges-63/obj85 2024-04-12T04:20:00.177 DEBUG:boto:auth_path=/dkyges-63/obj85 2024-04-12T04:20:00.177 DEBUG:boto:path=/dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.177 DEBUG:boto:auth_path=/dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.178 DEBUG:boto:Method: GET 2024-04-12T04:20:00.178 DEBUG:boto:Path: /dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.178 DEBUG:boto:Data: 2024-04-12T04:20:00.178 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.178 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.178 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.178 DEBUG:boto:Params: {} 2024-04-12T04:20:00.178 DEBUG:boto:Token: None 2024-04-12T04:20:00.178 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.179 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EmB+axGmUtq8P3cWUzLhhiT5NY4= 2024-04-12T04:20:00.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EmB+axGmUtq8P3cWUzLhhiT5NY4=', 'Content-Length': '0'} 2024-04-12T04:20:00.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000519e1dfd010e7c72-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.222 DEBUG:boto:path=/dkyges-63/obj85 2024-04-12T04:20:00.222 DEBUG:boto:auth_path=/dkyges-63/obj85 2024-04-12T04:20:00.222 DEBUG:boto:path=/dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.223 DEBUG:boto:auth_path=/dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.223 DEBUG:boto:Method: GET 2024-04-12T04:20:00.223 DEBUG:boto:Path: /dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.223 DEBUG:boto:Data: 2024-04-12T04:20:00.223 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.223 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.223 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.223 DEBUG:boto:Params: {} 2024-04-12T04:20:00.223 DEBUG:boto:Token: None 2024-04-12T04:20:00.223 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.223 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EmB+axGmUtq8P3cWUzLhhiT5NY4= 2024-04-12T04:20:00.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EmB+axGmUtq8P3cWUzLhhiT5NY4=', 'Content-Length': '0'} 2024-04-12T04:20:00.226 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a270c170a7972fd4-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.226 DEBUG:boto:path=/dkyges-63/obj85 2024-04-12T04:20:00.226 DEBUG:boto:auth_path=/dkyges-63/obj85 2024-04-12T04:20:00.226 DEBUG:boto:path=/dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.226 DEBUG:boto:auth_path=/dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.227 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.227 DEBUG:boto:Path: /dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.227 DEBUG:boto:Data: 2024-04-12T04:20:00.227 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.227 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.227 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.227 DEBUG:boto:Params: {} 2024-04-12T04:20:00.227 DEBUG:boto:Token: None 2024-04-12T04:20:00.227 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.227 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+CQClybEQ38sr0Qi8ynxovtVf6g= 2024-04-12T04:20:00.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+CQClybEQ38sr0Qi8ynxovtVf6g=', 'Content-Length': '0'} 2024-04-12T04:20:00.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078b11a8903c78aa4-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.229 DEBUG:boto:path=/dkyges-63/obj85 2024-04-12T04:20:00.230 DEBUG:boto:auth_path=/dkyges-63/obj85 2024-04-12T04:20:00.230 DEBUG:boto:path=/dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.230 DEBUG:boto:auth_path=/dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.230 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.230 DEBUG:boto:Path: /dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.230 DEBUG:boto:Data: 2024-04-12T04:20:00.230 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.230 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.230 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.230 DEBUG:boto:Params: {} 2024-04-12T04:20:00.231 DEBUG:boto:Token: None 2024-04-12T04:20:00.231 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.231 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+CQClybEQ38sr0Qi8ynxovtVf6g= 2024-04-12T04:20:00.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+CQClybEQ38sr0Qi8ynxovtVf6g=', 'Content-Length': '0'} 2024-04-12T04:20:00.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b756dde172a950ee-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:20:00.233 DEBUG:boto:path=/dkyges-63/obj85 2024-04-12T04:20:00.233 DEBUG:boto:auth_path=/dkyges-63/obj85 2024-04-12T04:20:00.233 DEBUG:boto:path=/dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.233 DEBUG:boto:auth_path=/dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.233 DEBUG:boto:Method: GET 2024-04-12T04:20:00.234 DEBUG:boto:Path: /dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.234 DEBUG:boto:Data: 2024-04-12T04:20:00.234 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.234 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.234 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.234 DEBUG:boto:Params: {} 2024-04-12T04:20:00.234 DEBUG:boto:Token: None 2024-04-12T04:20:00.234 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.234 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EmB+axGmUtq8P3cWUzLhhiT5NY4= 2024-04-12T04:20:00.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EmB+axGmUtq8P3cWUzLhhiT5NY4=', 'Content-Length': '0'} 2024-04-12T04:20:00.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7fdcb65ee74b6fd-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.236 DEBUG:boto:path=/dkyges-63/obj85 2024-04-12T04:20:00.236 DEBUG:boto:auth_path=/dkyges-63/obj85 2024-04-12T04:20:00.237 DEBUG:boto:path=/dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.237 DEBUG:boto:auth_path=/dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.237 DEBUG:boto:Method: GET 2024-04-12T04:20:00.237 DEBUG:boto:Path: /dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.237 DEBUG:boto:Data: 2024-04-12T04:20:00.237 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.237 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.237 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.237 DEBUG:boto:Params: {} 2024-04-12T04:20:00.237 DEBUG:boto:Token: None 2024-04-12T04:20:00.237 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj85?versionId=null 2024-04-12T04:20:00.238 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EmB+axGmUtq8P3cWUzLhhiT5NY4= 2024-04-12T04:20:00.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EmB+axGmUtq8P3cWUzLhhiT5NY4=', 'Content-Length': '0'} 2024-04-12T04:20:00.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f34de2932f7e691-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.240 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.240 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.240 DEBUG:boto:path=/dkyges-63/?prefix=obj85 2024-04-12T04:20:00.240 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj85 2024-04-12T04:20:00.240 DEBUG:boto:Method: GET 2024-04-12T04:20:00.240 DEBUG:boto:Path: /dkyges-63/?prefix=obj85 2024-04-12T04:20:00.240 DEBUG:boto:Data: 2024-04-12T04:20:00.240 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.240 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.240 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.240 DEBUG:boto:Params: {} 2024-04-12T04:20:00.241 DEBUG:boto:Token: None 2024-04-12T04:20:00.241 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.241 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.243 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e0e40af5b38d14c-006618b670-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.243 DEBUG:boto:b'dkyges-63obj851000falseobj852024-04-12T04:15:47.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.244 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.244 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.244 DEBUG:boto:path=/dkyges-63/?prefix=obj85 2024-04-12T04:20:00.244 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj85 2024-04-12T04:20:00.244 DEBUG:boto:Method: GET 2024-04-12T04:20:00.244 DEBUG:boto:Path: /dkyges-63/?prefix=obj85 2024-04-12T04:20:00.244 DEBUG:boto:Data: 2024-04-12T04:20:00.244 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.244 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.245 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.245 DEBUG:boto:Params: {} 2024-04-12T04:20:00.245 DEBUG:boto:Token: None 2024-04-12T04:20:00.245 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.245 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f96111924210ddc8-006618b670-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.290 DEBUG:boto:b'dkyges-63obj851000falseobj852024-04-12T04:15:47.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.291 DEBUG:boto:path=/dkyges-63/obj85 2024-04-12T04:20:00.291 DEBUG:boto:auth_path=/dkyges-63/obj85 2024-04-12T04:20:00.291 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.291 DEBUG:boto:Path: /dkyges-63/obj85 2024-04-12T04:20:00.291 DEBUG:boto:Data: 2024-04-12T04:20:00.291 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.291 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.291 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.291 DEBUG:boto:Params: {} 2024-04-12T04:20:00.291 DEBUG:boto:Token: None 2024-04-12T04:20:00.292 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj85 2024-04-12T04:20:00.292 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A9ACRGTq7aBYa4FZm/q0u0bfqfs= 2024-04-12T04:20:00.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A9ACRGTq7aBYa4FZm/q0u0bfqfs=', 'Content-Length': '0'} 2024-04-12T04:20:00.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c07336791630ffde-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.294 DEBUG:boto:path=/dkyges-63/obj85 2024-04-12T04:20:00.294 DEBUG:boto:auth_path=/dkyges-63/obj85 2024-04-12T04:20:00.294 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.294 DEBUG:boto:Path: /dkyges-63/obj85 2024-04-12T04:20:00.294 DEBUG:boto:Data: 2024-04-12T04:20:00.294 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.294 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.295 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.295 DEBUG:boto:Params: {} 2024-04-12T04:20:00.295 DEBUG:boto:Token: None 2024-04-12T04:20:00.295 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj85 2024-04-12T04:20:00.295 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A9ACRGTq7aBYa4FZm/q0u0bfqfs= 2024-04-12T04:20:00.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A9ACRGTq7aBYa4FZm/q0u0bfqfs=', 'Content-Length': '0'} 2024-04-12T04:20:00.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001756ad8119d783e0-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2024-04-12T04:20:00.297 DEBUG:boto:path=/dkyges-63/obj85 2024-04-12T04:20:00.297 DEBUG:boto:auth_path=/dkyges-63/obj85 2024-04-12T04:20:00.297 DEBUG:boto:Method: GET 2024-04-12T04:20:00.297 DEBUG:boto:Path: /dkyges-63/obj85 2024-04-12T04:20:00.298 DEBUG:boto:Data: 2024-04-12T04:20:00.298 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.298 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.298 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.298 DEBUG:boto:Params: {} 2024-04-12T04:20:00.298 DEBUG:boto:Token: None 2024-04-12T04:20:00.298 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj85 2024-04-12T04:20:00.298 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Saac4D0qNWhToIq1b4Q+iVQ6jBU= 2024-04-12T04:20:00.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Saac4D0qNWhToIq1b4Q+iVQ6jBU=', 'Content-Length': '0'} 2024-04-12T04:20:00.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eed99a8c651b04a-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.300 DEBUG:boto:path=/dkyges-63/obj85 2024-04-12T04:20:00.300 DEBUG:boto:auth_path=/dkyges-63/obj85 2024-04-12T04:20:00.301 DEBUG:boto:Method: GET 2024-04-12T04:20:00.301 DEBUG:boto:Path: /dkyges-63/obj85 2024-04-12T04:20:00.301 DEBUG:boto:Data: 2024-04-12T04:20:00.301 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.301 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.301 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.301 DEBUG:boto:Params: {} 2024-04-12T04:20:00.301 DEBUG:boto:Token: None 2024-04-12T04:20:00.301 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj85 2024-04-12T04:20:00.301 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Saac4D0qNWhToIq1b4Q+iVQ6jBU= 2024-04-12T04:20:00.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Saac4D0qNWhToIq1b4Q+iVQ6jBU=', 'Content-Length': '0'} 2024-04-12T04:20:00.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de449bf4dba6e84c-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.304 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.304 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.304 DEBUG:boto:path=/dkyges-63/?prefix=obj85 2024-04-12T04:20:00.304 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj85 2024-04-12T04:20:00.304 DEBUG:boto:Method: GET 2024-04-12T04:20:00.304 DEBUG:boto:Path: /dkyges-63/?prefix=obj85 2024-04-12T04:20:00.304 DEBUG:boto:Data: 2024-04-12T04:20:00.304 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.304 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.304 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.304 DEBUG:boto:Params: {} 2024-04-12T04:20:00.304 DEBUG:boto:Token: None 2024-04-12T04:20:00.305 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.305 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059eb54a7ea59d364-006618b670-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.307 DEBUG:boto:b'dkyges-63obj851000falseobj852024-04-12T04:15:47.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.308 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.308 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.308 DEBUG:boto:path=/dkyges-63/?prefix=obj85 2024-04-12T04:20:00.308 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj85 2024-04-12T04:20:00.308 DEBUG:boto:Method: GET 2024-04-12T04:20:00.308 DEBUG:boto:Path: /dkyges-63/?prefix=obj85 2024-04-12T04:20:00.308 DEBUG:boto:Data: 2024-04-12T04:20:00.308 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.308 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.308 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.308 DEBUG:boto:Params: {} 2024-04-12T04:20:00.309 DEBUG:boto:Token: None 2024-04-12T04:20:00.309 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.309 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb0eeee03690bb75-006618b670-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.313 DEBUG:boto:b'dkyges-63obj851000falseobj852024-04-12T04:15:47.881Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:20:00.314 DEBUG:boto:path=/dkyges-63/obj87 2024-04-12T04:20:00.314 DEBUG:boto:auth_path=/dkyges-63/obj87 2024-04-12T04:20:00.314 DEBUG:boto:path=/dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.314 DEBUG:boto:auth_path=/dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.314 DEBUG:boto:Method: GET 2024-04-12T04:20:00.314 DEBUG:boto:Path: /dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.314 DEBUG:boto:Data: 2024-04-12T04:20:00.314 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.315 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.315 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.315 DEBUG:boto:Params: {} 2024-04-12T04:20:00.315 DEBUG:boto:Token: None 2024-04-12T04:20:00.315 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.315 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QZ8glf4VBYumWx3hAeG8AJmo5VQ= 2024-04-12T04:20:00.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QZ8glf4VBYumWx3hAeG8AJmo5VQ=', 'Content-Length': '0'} 2024-04-12T04:20:00.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077f2518fabca06e7-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.317 DEBUG:boto:path=/dkyges-63/obj87 2024-04-12T04:20:00.318 DEBUG:boto:auth_path=/dkyges-63/obj87 2024-04-12T04:20:00.318 DEBUG:boto:path=/dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.318 DEBUG:boto:auth_path=/dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.318 DEBUG:boto:Method: GET 2024-04-12T04:20:00.318 DEBUG:boto:Path: /dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.318 DEBUG:boto:Data: 2024-04-12T04:20:00.318 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.318 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.318 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.318 DEBUG:boto:Params: {} 2024-04-12T04:20:00.319 DEBUG:boto:Token: None 2024-04-12T04:20:00.319 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.319 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QZ8glf4VBYumWx3hAeG8AJmo5VQ= 2024-04-12T04:20:00.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QZ8glf4VBYumWx3hAeG8AJmo5VQ=', 'Content-Length': '0'} 2024-04-12T04:20:00.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dba0ce24c60011d-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.362 DEBUG:boto:path=/dkyges-63/obj87 2024-04-12T04:20:00.362 DEBUG:boto:auth_path=/dkyges-63/obj87 2024-04-12T04:20:00.362 DEBUG:boto:path=/dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.363 DEBUG:boto:auth_path=/dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.363 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.363 DEBUG:boto:Path: /dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.363 DEBUG:boto:Data: 2024-04-12T04:20:00.363 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.363 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.363 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.363 DEBUG:boto:Params: {} 2024-04-12T04:20:00.363 DEBUG:boto:Token: None 2024-04-12T04:20:00.363 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.364 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oZIdtjkcslfK38RinEwt+5pv38A= 2024-04-12T04:20:00.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oZIdtjkcslfK38RinEwt+5pv38A=', 'Content-Length': '0'} 2024-04-12T04:20:00.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e6fb12953f3d024-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.366 DEBUG:boto:path=/dkyges-63/obj87 2024-04-12T04:20:00.366 DEBUG:boto:auth_path=/dkyges-63/obj87 2024-04-12T04:20:00.366 DEBUG:boto:path=/dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.366 DEBUG:boto:auth_path=/dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.366 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.366 DEBUG:boto:Path: /dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.366 DEBUG:boto:Data: 2024-04-12T04:20:00.366 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.366 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.366 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.366 DEBUG:boto:Params: {} 2024-04-12T04:20:00.367 DEBUG:boto:Token: None 2024-04-12T04:20:00.367 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.367 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oZIdtjkcslfK38RinEwt+5pv38A= 2024-04-12T04:20:00.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oZIdtjkcslfK38RinEwt+5pv38A=', 'Content-Length': '0'} 2024-04-12T04:20:00.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007e822e512f58ef4-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:20:00.369 DEBUG:boto:path=/dkyges-63/obj87 2024-04-12T04:20:00.369 DEBUG:boto:auth_path=/dkyges-63/obj87 2024-04-12T04:20:00.370 DEBUG:boto:path=/dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.370 DEBUG:boto:auth_path=/dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.370 DEBUG:boto:Method: GET 2024-04-12T04:20:00.370 DEBUG:boto:Path: /dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.370 DEBUG:boto:Data: 2024-04-12T04:20:00.370 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.370 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.370 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.370 DEBUG:boto:Params: {} 2024-04-12T04:20:00.370 DEBUG:boto:Token: None 2024-04-12T04:20:00.370 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.370 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QZ8glf4VBYumWx3hAeG8AJmo5VQ= 2024-04-12T04:20:00.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QZ8glf4VBYumWx3hAeG8AJmo5VQ=', 'Content-Length': '0'} 2024-04-12T04:20:00.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6a8619ad6a204e3-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.372 DEBUG:boto:path=/dkyges-63/obj87 2024-04-12T04:20:00.373 DEBUG:boto:auth_path=/dkyges-63/obj87 2024-04-12T04:20:00.373 DEBUG:boto:path=/dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.373 DEBUG:boto:auth_path=/dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.373 DEBUG:boto:Method: GET 2024-04-12T04:20:00.373 DEBUG:boto:Path: /dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.373 DEBUG:boto:Data: 2024-04-12T04:20:00.373 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.373 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.373 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.373 DEBUG:boto:Params: {} 2024-04-12T04:20:00.373 DEBUG:boto:Token: None 2024-04-12T04:20:00.374 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj87?versionId=null 2024-04-12T04:20:00.374 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QZ8glf4VBYumWx3hAeG8AJmo5VQ= 2024-04-12T04:20:00.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QZ8glf4VBYumWx3hAeG8AJmo5VQ=', 'Content-Length': '0'} 2024-04-12T04:20:00.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d02d43c356c60709-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.376 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.377 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.377 DEBUG:boto:path=/dkyges-63/?prefix=obj87 2024-04-12T04:20:00.377 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj87 2024-04-12T04:20:00.377 DEBUG:boto:Method: GET 2024-04-12T04:20:00.377 DEBUG:boto:Path: /dkyges-63/?prefix=obj87 2024-04-12T04:20:00.377 DEBUG:boto:Data: 2024-04-12T04:20:00.377 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.377 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.377 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.377 DEBUG:boto:Params: {} 2024-04-12T04:20:00.378 DEBUG:boto:Token: None 2024-04-12T04:20:00.378 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.378 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a97456f08420ee06-006618b670-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.381 DEBUG:boto:b'dkyges-63obj871000falseobj872024-04-12T04:15:48.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.382 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.382 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.382 DEBUG:boto:path=/dkyges-63/?prefix=obj87 2024-04-12T04:20:00.382 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj87 2024-04-12T04:20:00.382 DEBUG:boto:Method: GET 2024-04-12T04:20:00.382 DEBUG:boto:Path: /dkyges-63/?prefix=obj87 2024-04-12T04:20:00.382 DEBUG:boto:Data: 2024-04-12T04:20:00.382 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.382 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.382 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.382 DEBUG:boto:Params: {} 2024-04-12T04:20:00.383 DEBUG:boto:Token: None 2024-04-12T04:20:00.383 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.383 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003595935a07a46046-006618b670-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.387 DEBUG:boto:b'dkyges-63obj871000falseobj872024-04-12T04:15:48.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.388 DEBUG:boto:path=/dkyges-63/obj87 2024-04-12T04:20:00.388 DEBUG:boto:auth_path=/dkyges-63/obj87 2024-04-12T04:20:00.388 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.388 DEBUG:boto:Path: /dkyges-63/obj87 2024-04-12T04:20:00.388 DEBUG:boto:Data: 2024-04-12T04:20:00.388 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.389 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.389 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.389 DEBUG:boto:Params: {} 2024-04-12T04:20:00.389 DEBUG:boto:Token: None 2024-04-12T04:20:00.389 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj87 2024-04-12T04:20:00.389 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XcBvYbz5T8ZC+0u+CE6bLce8sXI= 2024-04-12T04:20:00.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XcBvYbz5T8ZC+0u+CE6bLce8sXI=', 'Content-Length': '0'} 2024-04-12T04:20:00.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5b063e8681ff453-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.391 DEBUG:boto:path=/dkyges-63/obj87 2024-04-12T04:20:00.391 DEBUG:boto:auth_path=/dkyges-63/obj87 2024-04-12T04:20:00.391 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.392 DEBUG:boto:Path: /dkyges-63/obj87 2024-04-12T04:20:00.392 DEBUG:boto:Data: 2024-04-12T04:20:00.392 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.392 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.392 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.392 DEBUG:boto:Params: {} 2024-04-12T04:20:00.392 DEBUG:boto:Token: None 2024-04-12T04:20:00.392 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj87 2024-04-12T04:20:00.392 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XcBvYbz5T8ZC+0u+CE6bLce8sXI= 2024-04-12T04:20:00.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XcBvYbz5T8ZC+0u+CE6bLce8sXI=', 'Content-Length': '0'} 2024-04-12T04:20:00.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000121dbcc4f431b42d-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2024-04-12T04:20:00.395 DEBUG:boto:path=/dkyges-63/obj87 2024-04-12T04:20:00.395 DEBUG:boto:auth_path=/dkyges-63/obj87 2024-04-12T04:20:00.395 DEBUG:boto:Method: GET 2024-04-12T04:20:00.395 DEBUG:boto:Path: /dkyges-63/obj87 2024-04-12T04:20:00.395 DEBUG:boto:Data: 2024-04-12T04:20:00.395 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.395 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.395 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.396 DEBUG:boto:Params: {} 2024-04-12T04:20:00.396 DEBUG:boto:Token: None 2024-04-12T04:20:00.396 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj87 2024-04-12T04:20:00.396 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:y/1jvKzhsL8fwXV0kU5yq9CtbzA= 2024-04-12T04:20:00.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y/1jvKzhsL8fwXV0kU5yq9CtbzA=', 'Content-Length': '0'} 2024-04-12T04:20:00.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6c7da0dba5d4cc9-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.438 DEBUG:boto:path=/dkyges-63/obj87 2024-04-12T04:20:00.439 DEBUG:boto:auth_path=/dkyges-63/obj87 2024-04-12T04:20:00.439 DEBUG:boto:Method: GET 2024-04-12T04:20:00.439 DEBUG:boto:Path: /dkyges-63/obj87 2024-04-12T04:20:00.439 DEBUG:boto:Data: 2024-04-12T04:20:00.439 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.439 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.439 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.439 DEBUG:boto:Params: {} 2024-04-12T04:20:00.439 DEBUG:boto:Token: None 2024-04-12T04:20:00.440 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj87 2024-04-12T04:20:00.440 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:y/1jvKzhsL8fwXV0kU5yq9CtbzA= 2024-04-12T04:20:00.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y/1jvKzhsL8fwXV0kU5yq9CtbzA=', 'Content-Length': '0'} 2024-04-12T04:20:00.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc21470ea8fea99f-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.442 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.442 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.442 DEBUG:boto:path=/dkyges-63/?prefix=obj87 2024-04-12T04:20:00.442 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj87 2024-04-12T04:20:00.442 DEBUG:boto:Method: GET 2024-04-12T04:20:00.443 DEBUG:boto:Path: /dkyges-63/?prefix=obj87 2024-04-12T04:20:00.443 DEBUG:boto:Data: 2024-04-12T04:20:00.443 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.443 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.443 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.443 DEBUG:boto:Params: {} 2024-04-12T04:20:00.443 DEBUG:boto:Token: None 2024-04-12T04:20:00.443 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.443 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bcf9e2487f124b2-006618b670-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.446 DEBUG:boto:b'dkyges-63obj871000falseobj872024-04-12T04:15:48.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.447 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.447 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.447 DEBUG:boto:path=/dkyges-63/?prefix=obj87 2024-04-12T04:20:00.447 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj87 2024-04-12T04:20:00.447 DEBUG:boto:Method: GET 2024-04-12T04:20:00.447 DEBUG:boto:Path: /dkyges-63/?prefix=obj87 2024-04-12T04:20:00.447 DEBUG:boto:Data: 2024-04-12T04:20:00.447 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.447 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.448 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.448 DEBUG:boto:Params: {} 2024-04-12T04:20:00.448 DEBUG:boto:Token: None 2024-04-12T04:20:00.448 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.448 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e649b0e046dbf57-006618b670-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.452 DEBUG:boto:b'dkyges-63obj871000falseobj872024-04-12T04:15:48.894Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:20:00.453 DEBUG:boto:path=/dkyges-63/obj92 2024-04-12T04:20:00.453 DEBUG:boto:auth_path=/dkyges-63/obj92 2024-04-12T04:20:00.453 DEBUG:boto:path=/dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.453 DEBUG:boto:auth_path=/dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.453 DEBUG:boto:Method: GET 2024-04-12T04:20:00.453 DEBUG:boto:Path: /dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.453 DEBUG:boto:Data: 2024-04-12T04:20:00.453 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.454 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.454 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.454 DEBUG:boto:Params: {} 2024-04-12T04:20:00.454 DEBUG:boto:Token: None 2024-04-12T04:20:00.454 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.454 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gLioATcMUKFBXaxeH247N0CM4hU= 2024-04-12T04:20:00.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gLioATcMUKFBXaxeH247N0CM4hU=', 'Content-Length': '0'} 2024-04-12T04:20:00.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098aaf7e71e49878f-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.456 DEBUG:boto:path=/dkyges-63/obj92 2024-04-12T04:20:00.456 DEBUG:boto:auth_path=/dkyges-63/obj92 2024-04-12T04:20:00.457 DEBUG:boto:path=/dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.457 DEBUG:boto:auth_path=/dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.457 DEBUG:boto:Method: GET 2024-04-12T04:20:00.457 DEBUG:boto:Path: /dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.457 DEBUG:boto:Data: 2024-04-12T04:20:00.457 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.457 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.457 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.457 DEBUG:boto:Params: {} 2024-04-12T04:20:00.457 DEBUG:boto:Token: None 2024-04-12T04:20:00.457 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.458 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gLioATcMUKFBXaxeH247N0CM4hU= 2024-04-12T04:20:00.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gLioATcMUKFBXaxeH247N0CM4hU=', 'Content-Length': '0'} 2024-04-12T04:20:00.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc6233ff750238e6-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.502 DEBUG:boto:path=/dkyges-63/obj92 2024-04-12T04:20:00.502 DEBUG:boto:auth_path=/dkyges-63/obj92 2024-04-12T04:20:00.503 DEBUG:boto:path=/dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.503 DEBUG:boto:auth_path=/dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.503 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.503 DEBUG:boto:Path: /dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.503 DEBUG:boto:Data: 2024-04-12T04:20:00.503 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.503 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.503 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.503 DEBUG:boto:Params: {} 2024-04-12T04:20:00.503 DEBUG:boto:Token: None 2024-04-12T04:20:00.504 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.504 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SibW3zVdHnnHzVqphP1SvA3g56c= 2024-04-12T04:20:00.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SibW3zVdHnnHzVqphP1SvA3g56c=', 'Content-Length': '0'} 2024-04-12T04:20:00.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fd54782173f37ff-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.506 DEBUG:boto:path=/dkyges-63/obj92 2024-04-12T04:20:00.506 DEBUG:boto:auth_path=/dkyges-63/obj92 2024-04-12T04:20:00.506 DEBUG:boto:path=/dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.506 DEBUG:boto:auth_path=/dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.506 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.506 DEBUG:boto:Path: /dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.507 DEBUG:boto:Data: 2024-04-12T04:20:00.507 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.507 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.507 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.507 DEBUG:boto:Params: {} 2024-04-12T04:20:00.507 DEBUG:boto:Token: None 2024-04-12T04:20:00.507 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.507 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SibW3zVdHnnHzVqphP1SvA3g56c= 2024-04-12T04:20:00.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SibW3zVdHnnHzVqphP1SvA3g56c=', 'Content-Length': '0'} 2024-04-12T04:20:00.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eebebf689f1a9b5-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:20:00.510 DEBUG:boto:path=/dkyges-63/obj92 2024-04-12T04:20:00.510 DEBUG:boto:auth_path=/dkyges-63/obj92 2024-04-12T04:20:00.510 DEBUG:boto:path=/dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.510 DEBUG:boto:auth_path=/dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.510 DEBUG:boto:Method: GET 2024-04-12T04:20:00.510 DEBUG:boto:Path: /dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.510 DEBUG:boto:Data: 2024-04-12T04:20:00.510 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.511 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.511 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.511 DEBUG:boto:Params: {} 2024-04-12T04:20:00.511 DEBUG:boto:Token: None 2024-04-12T04:20:00.511 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.511 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gLioATcMUKFBXaxeH247N0CM4hU= 2024-04-12T04:20:00.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gLioATcMUKFBXaxeH247N0CM4hU=', 'Content-Length': '0'} 2024-04-12T04:20:00.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0057acccf6f7a10-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.513 DEBUG:boto:path=/dkyges-63/obj92 2024-04-12T04:20:00.514 DEBUG:boto:auth_path=/dkyges-63/obj92 2024-04-12T04:20:00.514 DEBUG:boto:path=/dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.514 DEBUG:boto:auth_path=/dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.514 DEBUG:boto:Method: GET 2024-04-12T04:20:00.514 DEBUG:boto:Path: /dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.514 DEBUG:boto:Data: 2024-04-12T04:20:00.514 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.514 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.514 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.515 DEBUG:boto:Params: {} 2024-04-12T04:20:00.515 DEBUG:boto:Token: None 2024-04-12T04:20:00.515 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj92?versionId=null 2024-04-12T04:20:00.515 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gLioATcMUKFBXaxeH247N0CM4hU= 2024-04-12T04:20:00.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gLioATcMUKFBXaxeH247N0CM4hU=', 'Content-Length': '0'} 2024-04-12T04:20:00.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b97bfb4d0f71d6b4-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.517 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.518 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.518 DEBUG:boto:path=/dkyges-63/?prefix=obj92 2024-04-12T04:20:00.518 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj92 2024-04-12T04:20:00.518 DEBUG:boto:Method: GET 2024-04-12T04:20:00.518 DEBUG:boto:Path: /dkyges-63/?prefix=obj92 2024-04-12T04:20:00.518 DEBUG:boto:Data: 2024-04-12T04:20:00.518 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.518 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.519 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.519 DEBUG:boto:Params: {} 2024-04-12T04:20:00.519 DEBUG:boto:Token: None 2024-04-12T04:20:00.519 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.519 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a73a2be61b4a1b34-006618b670-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.522 DEBUG:boto:b'dkyges-63obj921000falseobj922024-04-12T04:15:48.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.522 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.522 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.522 DEBUG:boto:path=/dkyges-63/?prefix=obj92 2024-04-12T04:20:00.523 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj92 2024-04-12T04:20:00.523 DEBUG:boto:Method: GET 2024-04-12T04:20:00.523 DEBUG:boto:Path: /dkyges-63/?prefix=obj92 2024-04-12T04:20:00.523 DEBUG:boto:Data: 2024-04-12T04:20:00.523 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.523 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.523 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.523 DEBUG:boto:Params: {} 2024-04-12T04:20:00.524 DEBUG:boto:Token: None 2024-04-12T04:20:00.524 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.524 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.528 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7b34282d4de1314-006618b670-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.529 DEBUG:boto:b'dkyges-63obj921000falseobj922024-04-12T04:15:48.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.529 DEBUG:boto:path=/dkyges-63/obj92 2024-04-12T04:20:00.529 DEBUG:boto:auth_path=/dkyges-63/obj92 2024-04-12T04:20:00.529 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.529 DEBUG:boto:Path: /dkyges-63/obj92 2024-04-12T04:20:00.529 DEBUG:boto:Data: 2024-04-12T04:20:00.530 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.530 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.530 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.530 DEBUG:boto:Params: {} 2024-04-12T04:20:00.530 DEBUG:boto:Token: None 2024-04-12T04:20:00.530 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj92 2024-04-12T04:20:00.530 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EZCqQWeKpYcbfX52T+duyql7RrY= 2024-04-12T04:20:00.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EZCqQWeKpYcbfX52T+duyql7RrY=', 'Content-Length': '0'} 2024-04-12T04:20:00.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000078a6b4282e6521b-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.532 DEBUG:boto:path=/dkyges-63/obj92 2024-04-12T04:20:00.532 DEBUG:boto:auth_path=/dkyges-63/obj92 2024-04-12T04:20:00.533 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.533 DEBUG:boto:Path: /dkyges-63/obj92 2024-04-12T04:20:00.533 DEBUG:boto:Data: 2024-04-12T04:20:00.533 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.533 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.533 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.533 DEBUG:boto:Params: {} 2024-04-12T04:20:00.533 DEBUG:boto:Token: None 2024-04-12T04:20:00.534 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj92 2024-04-12T04:20:00.534 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:EZCqQWeKpYcbfX52T+duyql7RrY= 2024-04-12T04:20:00.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EZCqQWeKpYcbfX52T+duyql7RrY=', 'Content-Length': '0'} 2024-04-12T04:20:00.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f135882e1e406fc-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2024-04-12T04:20:00.536 DEBUG:boto:path=/dkyges-63/obj92 2024-04-12T04:20:00.536 DEBUG:boto:auth_path=/dkyges-63/obj92 2024-04-12T04:20:00.536 DEBUG:boto:Method: GET 2024-04-12T04:20:00.536 DEBUG:boto:Path: /dkyges-63/obj92 2024-04-12T04:20:00.536 DEBUG:boto:Data: 2024-04-12T04:20:00.537 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.537 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.537 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.537 DEBUG:boto:Params: {} 2024-04-12T04:20:00.537 DEBUG:boto:Token: None 2024-04-12T04:20:00.538 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj92 2024-04-12T04:20:00.538 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hX+pYBiJMqPVyiVe3v/uRr7+wyM= 2024-04-12T04:20:00.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hX+pYBiJMqPVyiVe3v/uRr7+wyM=', 'Content-Length': '0'} 2024-04-12T04:20:00.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088eaaeadab3b9ea2-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.540 DEBUG:boto:path=/dkyges-63/obj92 2024-04-12T04:20:00.540 DEBUG:boto:auth_path=/dkyges-63/obj92 2024-04-12T04:20:00.540 DEBUG:boto:Method: GET 2024-04-12T04:20:00.540 DEBUG:boto:Path: /dkyges-63/obj92 2024-04-12T04:20:00.541 DEBUG:boto:Data: 2024-04-12T04:20:00.541 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.541 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.541 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.541 DEBUG:boto:Params: {} 2024-04-12T04:20:00.541 DEBUG:boto:Token: None 2024-04-12T04:20:00.541 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj92 2024-04-12T04:20:00.541 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hX+pYBiJMqPVyiVe3v/uRr7+wyM= 2024-04-12T04:20:00.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hX+pYBiJMqPVyiVe3v/uRr7+wyM=', 'Content-Length': '0'} 2024-04-12T04:20:00.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023024ba3cb5f2ef7-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.544 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.544 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.544 DEBUG:boto:path=/dkyges-63/?prefix=obj92 2024-04-12T04:20:00.544 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj92 2024-04-12T04:20:00.544 DEBUG:boto:Method: GET 2024-04-12T04:20:00.544 DEBUG:boto:Path: /dkyges-63/?prefix=obj92 2024-04-12T04:20:00.544 DEBUG:boto:Data: 2024-04-12T04:20:00.545 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.545 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.545 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.545 DEBUG:boto:Params: {} 2024-04-12T04:20:00.545 DEBUG:boto:Token: None 2024-04-12T04:20:00.545 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.545 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.548 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f90167e9b2a4235-006618b670-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.548 DEBUG:boto:b'dkyges-63obj921000falseobj922024-04-12T04:15:48.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.548 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.548 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.548 DEBUG:boto:path=/dkyges-63/?prefix=obj92 2024-04-12T04:20:00.549 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj92 2024-04-12T04:20:00.549 DEBUG:boto:Method: GET 2024-04-12T04:20:00.549 DEBUG:boto:Path: /dkyges-63/?prefix=obj92 2024-04-12T04:20:00.549 DEBUG:boto:Data: 2024-04-12T04:20:00.549 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.549 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.549 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.549 DEBUG:boto:Params: {} 2024-04-12T04:20:00.549 DEBUG:boto:Token: None 2024-04-12T04:20:00.550 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.550 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064a85805bdb5f194-006618b670-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.594 DEBUG:boto:b'dkyges-63obj921000falseobj922024-04-12T04:15:48.555Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T04:20:00.595 DEBUG:boto:path=/dkyges-63/obj93 2024-04-12T04:20:00.595 DEBUG:boto:auth_path=/dkyges-63/obj93 2024-04-12T04:20:00.595 DEBUG:boto:path=/dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.595 DEBUG:boto:auth_path=/dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.595 DEBUG:boto:Method: GET 2024-04-12T04:20:00.595 DEBUG:boto:Path: /dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.595 DEBUG:boto:Data: 2024-04-12T04:20:00.595 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.595 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.596 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.596 DEBUG:boto:Params: {} 2024-04-12T04:20:00.596 DEBUG:boto:Token: None 2024-04-12T04:20:00.596 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.596 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pzOf1AVPO3+/ih+wz5EXgWNZf6w= 2024-04-12T04:20:00.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pzOf1AVPO3+/ih+wz5EXgWNZf6w=', 'Content-Length': '0'} 2024-04-12T04:20:00.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba9425653df88c69-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.598 DEBUG:boto:path=/dkyges-63/obj93 2024-04-12T04:20:00.598 DEBUG:boto:auth_path=/dkyges-63/obj93 2024-04-12T04:20:00.599 DEBUG:boto:path=/dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.599 DEBUG:boto:auth_path=/dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.599 DEBUG:boto:Method: GET 2024-04-12T04:20:00.599 DEBUG:boto:Path: /dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.599 DEBUG:boto:Data: 2024-04-12T04:20:00.599 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.599 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.599 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.599 DEBUG:boto:Params: {} 2024-04-12T04:20:00.599 DEBUG:boto:Token: None 2024-04-12T04:20:00.599 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.600 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pzOf1AVPO3+/ih+wz5EXgWNZf6w= 2024-04-12T04:20:00.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pzOf1AVPO3+/ih+wz5EXgWNZf6w=', 'Content-Length': '0'} 2024-04-12T04:20:00.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007736a2846af68a53-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.602 DEBUG:boto:path=/dkyges-63/obj93 2024-04-12T04:20:00.602 DEBUG:boto:auth_path=/dkyges-63/obj93 2024-04-12T04:20:00.602 DEBUG:boto:path=/dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.602 DEBUG:boto:auth_path=/dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.602 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.602 DEBUG:boto:Path: /dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.602 DEBUG:boto:Data: 2024-04-12T04:20:00.602 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.602 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.603 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.603 DEBUG:boto:Params: {} 2024-04-12T04:20:00.603 DEBUG:boto:Token: None 2024-04-12T04:20:00.603 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.603 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pOUn2Up/O6O7Aeyxv68pz0SMUnQ= 2024-04-12T04:20:00.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pOUn2Up/O6O7Aeyxv68pz0SMUnQ=', 'Content-Length': '0'} 2024-04-12T04:20:00.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008febd3514750293a-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.605 DEBUG:boto:path=/dkyges-63/obj93 2024-04-12T04:20:00.605 DEBUG:boto:auth_path=/dkyges-63/obj93 2024-04-12T04:20:00.605 DEBUG:boto:path=/dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.605 DEBUG:boto:auth_path=/dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.605 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.605 DEBUG:boto:Path: /dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.605 DEBUG:boto:Data: 2024-04-12T04:20:00.606 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.606 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.606 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.606 DEBUG:boto:Params: {} 2024-04-12T04:20:00.606 DEBUG:boto:Token: None 2024-04-12T04:20:00.606 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.606 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pOUn2Up/O6O7Aeyxv68pz0SMUnQ= 2024-04-12T04:20:00.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pOUn2Up/O6O7Aeyxv68pz0SMUnQ=', 'Content-Length': '0'} 2024-04-12T04:20:00.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca1c92060afea789-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T04:20:00.608 DEBUG:boto:path=/dkyges-63/obj93 2024-04-12T04:20:00.608 DEBUG:boto:auth_path=/dkyges-63/obj93 2024-04-12T04:20:00.609 DEBUG:boto:path=/dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.609 DEBUG:boto:auth_path=/dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.609 DEBUG:boto:Method: GET 2024-04-12T04:20:00.609 DEBUG:boto:Path: /dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.609 DEBUG:boto:Data: 2024-04-12T04:20:00.609 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.609 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.609 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.609 DEBUG:boto:Params: {} 2024-04-12T04:20:00.609 DEBUG:boto:Token: None 2024-04-12T04:20:00.609 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.610 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pzOf1AVPO3+/ih+wz5EXgWNZf6w= 2024-04-12T04:20:00.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pzOf1AVPO3+/ih+wz5EXgWNZf6w=', 'Content-Length': '0'} 2024-04-12T04:20:00.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011d0e52d8ba8db93-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.654 DEBUG:boto:path=/dkyges-63/obj93 2024-04-12T04:20:00.654 DEBUG:boto:auth_path=/dkyges-63/obj93 2024-04-12T04:20:00.655 DEBUG:boto:path=/dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.655 DEBUG:boto:auth_path=/dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.655 DEBUG:boto:Method: GET 2024-04-12T04:20:00.655 DEBUG:boto:Path: /dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.655 DEBUG:boto:Data: 2024-04-12T04:20:00.655 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.655 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.655 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.655 DEBUG:boto:Params: {} 2024-04-12T04:20:00.655 DEBUG:boto:Token: None 2024-04-12T04:20:00.656 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj93?versionId=null 2024-04-12T04:20:00.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:pzOf1AVPO3+/ih+wz5EXgWNZf6w= 2024-04-12T04:20:00.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pzOf1AVPO3+/ih+wz5EXgWNZf6w=', 'Content-Length': '0'} 2024-04-12T04:20:00.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008fd73c4c3814b10d-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.658 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.658 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.658 DEBUG:boto:path=/dkyges-63/?prefix=obj93 2024-04-12T04:20:00.658 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj93 2024-04-12T04:20:00.659 DEBUG:boto:Method: GET 2024-04-12T04:20:00.659 DEBUG:boto:Path: /dkyges-63/?prefix=obj93 2024-04-12T04:20:00.659 DEBUG:boto:Data: 2024-04-12T04:20:00.659 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.659 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.659 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.659 DEBUG:boto:Params: {} 2024-04-12T04:20:00.659 DEBUG:boto:Token: None 2024-04-12T04:20:00.659 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.659 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ab6719a74df32cc-006618b670-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.662 DEBUG:boto:b'dkyges-63obj931000falseobj932024-04-12T04:15:49.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.663 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.663 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.663 DEBUG:boto:path=/dkyges-63/?prefix=obj93 2024-04-12T04:20:00.663 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj93 2024-04-12T04:20:00.663 DEBUG:boto:Method: GET 2024-04-12T04:20:00.663 DEBUG:boto:Path: /dkyges-63/?prefix=obj93 2024-04-12T04:20:00.663 DEBUG:boto:Data: 2024-04-12T04:20:00.663 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.663 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.663 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.663 DEBUG:boto:Params: {} 2024-04-12T04:20:00.664 DEBUG:boto:Token: None 2024-04-12T04:20:00.664 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.664 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c2b3f48e5e45d4c-006618b670-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.669 DEBUG:boto:b'dkyges-63obj931000falseobj932024-04-12T04:15:49.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.669 DEBUG:boto:path=/dkyges-63/obj93 2024-04-12T04:20:00.669 DEBUG:boto:auth_path=/dkyges-63/obj93 2024-04-12T04:20:00.669 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.669 DEBUG:boto:Path: /dkyges-63/obj93 2024-04-12T04:20:00.669 DEBUG:boto:Data: 2024-04-12T04:20:00.669 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.669 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.669 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.670 DEBUG:boto:Params: {} 2024-04-12T04:20:00.670 DEBUG:boto:Token: None 2024-04-12T04:20:00.670 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj93 2024-04-12T04:20:00.670 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hKcAhmeK3B5kTTVeLNIAcsDauVw= 2024-04-12T04:20:00.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hKcAhmeK3B5kTTVeLNIAcsDauVw=', 'Content-Length': '0'} 2024-04-12T04:20:00.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa7598e81340be1a-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.672 DEBUG:boto:path=/dkyges-63/obj93 2024-04-12T04:20:00.672 DEBUG:boto:auth_path=/dkyges-63/obj93 2024-04-12T04:20:00.672 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.672 DEBUG:boto:Path: /dkyges-63/obj93 2024-04-12T04:20:00.672 DEBUG:boto:Data: 2024-04-12T04:20:00.673 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.673 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.673 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.673 DEBUG:boto:Params: {} 2024-04-12T04:20:00.673 DEBUG:boto:Token: None 2024-04-12T04:20:00.673 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj93 2024-04-12T04:20:00.673 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hKcAhmeK3B5kTTVeLNIAcsDauVw= 2024-04-12T04:20:00.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hKcAhmeK3B5kTTVeLNIAcsDauVw=', 'Content-Length': '0'} 2024-04-12T04:20:00.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034660dd2de93b109-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2024-04-12T04:20:00.675 DEBUG:boto:path=/dkyges-63/obj93 2024-04-12T04:20:00.676 DEBUG:boto:auth_path=/dkyges-63/obj93 2024-04-12T04:20:00.676 DEBUG:boto:Method: GET 2024-04-12T04:20:00.676 DEBUG:boto:Path: /dkyges-63/obj93 2024-04-12T04:20:00.676 DEBUG:boto:Data: 2024-04-12T04:20:00.676 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.676 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.676 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.676 DEBUG:boto:Params: {} 2024-04-12T04:20:00.676 DEBUG:boto:Token: None 2024-04-12T04:20:00.676 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj93 2024-04-12T04:20:00.677 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VHAdUimWSSFeSgbHFH87r8WWLvo= 2024-04-12T04:20:00.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VHAdUimWSSFeSgbHFH87r8WWLvo=', 'Content-Length': '0'} 2024-04-12T04:20:00.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005db044c24824ce6b-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.679 DEBUG:boto:path=/dkyges-63/obj93 2024-04-12T04:20:00.679 DEBUG:boto:auth_path=/dkyges-63/obj93 2024-04-12T04:20:00.679 DEBUG:boto:Method: GET 2024-04-12T04:20:00.680 DEBUG:boto:Path: /dkyges-63/obj93 2024-04-12T04:20:00.680 DEBUG:boto:Data: 2024-04-12T04:20:00.680 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.680 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.680 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.680 DEBUG:boto:Params: {} 2024-04-12T04:20:00.680 DEBUG:boto:Token: None 2024-04-12T04:20:00.680 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj93 2024-04-12T04:20:00.681 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VHAdUimWSSFeSgbHFH87r8WWLvo= 2024-04-12T04:20:00.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VHAdUimWSSFeSgbHFH87r8WWLvo=', 'Content-Length': '0'} 2024-04-12T04:20:00.682 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df595591469fa625-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.726 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.727 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.727 DEBUG:boto:path=/dkyges-63/?prefix=obj93 2024-04-12T04:20:00.727 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj93 2024-04-12T04:20:00.727 DEBUG:boto:Method: GET 2024-04-12T04:20:00.727 DEBUG:boto:Path: /dkyges-63/?prefix=obj93 2024-04-12T04:20:00.727 DEBUG:boto:Data: 2024-04-12T04:20:00.727 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.727 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.727 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.727 DEBUG:boto:Params: {} 2024-04-12T04:20:00.727 DEBUG:boto:Token: None 2024-04-12T04:20:00.728 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.728 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2b89ff6a33d2bdb-006618b670-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.731 DEBUG:boto:b'dkyges-63obj931000falseobj932024-04-12T04:15:49.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.731 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.731 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.731 DEBUG:boto:path=/dkyges-63/?prefix=obj93 2024-04-12T04:20:00.731 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj93 2024-04-12T04:20:00.732 DEBUG:boto:Method: GET 2024-04-12T04:20:00.732 DEBUG:boto:Path: /dkyges-63/?prefix=obj93 2024-04-12T04:20:00.732 DEBUG:boto:Data: 2024-04-12T04:20:00.732 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.732 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.732 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.732 DEBUG:boto:Params: {} 2024-04-12T04:20:00.732 DEBUG:boto:Token: None 2024-04-12T04:20:00.732 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.732 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a9f96c45c487f42-006618b670-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.738 DEBUG:boto:b'dkyges-63obj931000falseobj932024-04-12T04:15:49.252Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:20:00.739 DEBUG:boto:path=/dkyges-63/obj95 2024-04-12T04:20:00.739 DEBUG:boto:auth_path=/dkyges-63/obj95 2024-04-12T04:20:00.739 DEBUG:boto:path=/dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.739 DEBUG:boto:auth_path=/dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.740 DEBUG:boto:Method: GET 2024-04-12T04:20:00.740 DEBUG:boto:Path: /dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.740 DEBUG:boto:Data: 2024-04-12T04:20:00.740 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.740 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.740 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.740 DEBUG:boto:Params: {} 2024-04-12T04:20:00.741 DEBUG:boto:Token: None 2024-04-12T04:20:00.741 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.741 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lznRZUjy6jjgoKDbrq+ZdtrBG40= 2024-04-12T04:20:00.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lznRZUjy6jjgoKDbrq+ZdtrBG40=', 'Content-Length': '0'} 2024-04-12T04:20:00.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016208da20de32f2c-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.744 DEBUG:boto:path=/dkyges-63/obj95 2024-04-12T04:20:00.744 DEBUG:boto:auth_path=/dkyges-63/obj95 2024-04-12T04:20:00.745 DEBUG:boto:path=/dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.745 DEBUG:boto:auth_path=/dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.745 DEBUG:boto:Method: GET 2024-04-12T04:20:00.745 DEBUG:boto:Path: /dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.745 DEBUG:boto:Data: 2024-04-12T04:20:00.745 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.745 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.745 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.745 DEBUG:boto:Params: {} 2024-04-12T04:20:00.745 DEBUG:boto:Token: None 2024-04-12T04:20:00.746 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.746 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lznRZUjy6jjgoKDbrq+ZdtrBG40= 2024-04-12T04:20:00.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lznRZUjy6jjgoKDbrq+ZdtrBG40=', 'Content-Length': '0'} 2024-04-12T04:20:00.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000314f12c9b794b1-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.748 DEBUG:boto:path=/dkyges-63/obj95 2024-04-12T04:20:00.748 DEBUG:boto:auth_path=/dkyges-63/obj95 2024-04-12T04:20:00.748 DEBUG:boto:path=/dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.748 DEBUG:boto:auth_path=/dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.749 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.749 DEBUG:boto:Path: /dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.749 DEBUG:boto:Data: 2024-04-12T04:20:00.749 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.749 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.749 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.749 DEBUG:boto:Params: {} 2024-04-12T04:20:00.749 DEBUG:boto:Token: None 2024-04-12T04:20:00.749 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.749 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:npoQKd1q6s/bFqYfNINc+vsXBS8= 2024-04-12T04:20:00.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:npoQKd1q6s/bFqYfNINc+vsXBS8=', 'Content-Length': '0'} 2024-04-12T04:20:00.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cd32ec1c5f8d73c-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.751 DEBUG:boto:path=/dkyges-63/obj95 2024-04-12T04:20:00.751 DEBUG:boto:auth_path=/dkyges-63/obj95 2024-04-12T04:20:00.752 DEBUG:boto:path=/dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.752 DEBUG:boto:auth_path=/dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.752 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.752 DEBUG:boto:Path: /dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.752 DEBUG:boto:Data: 2024-04-12T04:20:00.752 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.752 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.752 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.752 DEBUG:boto:Params: {} 2024-04-12T04:20:00.752 DEBUG:boto:Token: None 2024-04-12T04:20:00.752 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.753 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:npoQKd1q6s/bFqYfNINc+vsXBS8= 2024-04-12T04:20:00.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:npoQKd1q6s/bFqYfNINc+vsXBS8=', 'Content-Length': '0'} 2024-04-12T04:20:00.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093d64ebe6640a1c0-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:20:00.755 DEBUG:boto:path=/dkyges-63/obj95 2024-04-12T04:20:00.755 DEBUG:boto:auth_path=/dkyges-63/obj95 2024-04-12T04:20:00.755 DEBUG:boto:path=/dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.755 DEBUG:boto:auth_path=/dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.755 DEBUG:boto:Method: GET 2024-04-12T04:20:00.755 DEBUG:boto:Path: /dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.755 DEBUG:boto:Data: 2024-04-12T04:20:00.755 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.755 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.756 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.756 DEBUG:boto:Params: {} 2024-04-12T04:20:00.756 DEBUG:boto:Token: None 2024-04-12T04:20:00.756 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.756 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lznRZUjy6jjgoKDbrq+ZdtrBG40= 2024-04-12T04:20:00.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lznRZUjy6jjgoKDbrq+ZdtrBG40=', 'Content-Length': '0'} 2024-04-12T04:20:00.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc2bb7124354310d-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.798 DEBUG:boto:path=/dkyges-63/obj95 2024-04-12T04:20:00.798 DEBUG:boto:auth_path=/dkyges-63/obj95 2024-04-12T04:20:00.798 DEBUG:boto:path=/dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.799 DEBUG:boto:auth_path=/dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.799 DEBUG:boto:Method: GET 2024-04-12T04:20:00.799 DEBUG:boto:Path: /dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.799 DEBUG:boto:Data: 2024-04-12T04:20:00.799 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.799 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.799 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.799 DEBUG:boto:Params: {} 2024-04-12T04:20:00.799 DEBUG:boto:Token: None 2024-04-12T04:20:00.799 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj95?versionId=null 2024-04-12T04:20:00.800 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lznRZUjy6jjgoKDbrq+ZdtrBG40= 2024-04-12T04:20:00.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lznRZUjy6jjgoKDbrq+ZdtrBG40=', 'Content-Length': '0'} 2024-04-12T04:20:00.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041007b7289508fde-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.802 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.802 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.802 DEBUG:boto:path=/dkyges-63/?prefix=obj95 2024-04-12T04:20:00.802 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj95 2024-04-12T04:20:00.802 DEBUG:boto:Method: GET 2024-04-12T04:20:00.802 DEBUG:boto:Path: /dkyges-63/?prefix=obj95 2024-04-12T04:20:00.802 DEBUG:boto:Data: 2024-04-12T04:20:00.803 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.803 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.803 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.803 DEBUG:boto:Params: {} 2024-04-12T04:20:00.803 DEBUG:boto:Token: None 2024-04-12T04:20:00.803 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.803 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac177b944309327d-006618b670-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.806 DEBUG:boto:b'dkyges-63obj951000falseobj952024-04-12T04:15:48.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.806 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.807 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.807 DEBUG:boto:path=/dkyges-63/?prefix=obj95 2024-04-12T04:20:00.807 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj95 2024-04-12T04:20:00.807 DEBUG:boto:Method: GET 2024-04-12T04:20:00.807 DEBUG:boto:Path: /dkyges-63/?prefix=obj95 2024-04-12T04:20:00.807 DEBUG:boto:Data: 2024-04-12T04:20:00.807 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.807 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.807 DEBUG:boto:Params: {} 2024-04-12T04:20:00.808 DEBUG:boto:Token: None 2024-04-12T04:20:00.808 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.808 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000552a04c01e94fd73-006618b670-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.812 DEBUG:boto:b'dkyges-63obj951000falseobj952024-04-12T04:15:48.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.813 DEBUG:boto:path=/dkyges-63/obj95 2024-04-12T04:20:00.813 DEBUG:boto:auth_path=/dkyges-63/obj95 2024-04-12T04:20:00.813 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.813 DEBUG:boto:Path: /dkyges-63/obj95 2024-04-12T04:20:00.813 DEBUG:boto:Data: 2024-04-12T04:20:00.813 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.813 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.813 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.813 DEBUG:boto:Params: {} 2024-04-12T04:20:00.814 DEBUG:boto:Token: None 2024-04-12T04:20:00.814 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj95 2024-04-12T04:20:00.814 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zfCnrIA0x65NqbsrLE0iKTS+tvI= 2024-04-12T04:20:00.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zfCnrIA0x65NqbsrLE0iKTS+tvI=', 'Content-Length': '0'} 2024-04-12T04:20:00.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000816b08df2e8c1f-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.816 DEBUG:boto:path=/dkyges-63/obj95 2024-04-12T04:20:00.816 DEBUG:boto:auth_path=/dkyges-63/obj95 2024-04-12T04:20:00.816 DEBUG:boto:Method: HEAD 2024-04-12T04:20:00.816 DEBUG:boto:Path: /dkyges-63/obj95 2024-04-12T04:20:00.816 DEBUG:boto:Data: 2024-04-12T04:20:00.816 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.816 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.816 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.817 DEBUG:boto:Params: {} 2024-04-12T04:20:00.817 DEBUG:boto:Token: None 2024-04-12T04:20:00.817 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj95 2024-04-12T04:20:00.817 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zfCnrIA0x65NqbsrLE0iKTS+tvI= 2024-04-12T04:20:00.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zfCnrIA0x65NqbsrLE0iKTS+tvI=', 'Content-Length': '0'} 2024-04-12T04:20:00.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba7e88a4a882bb2e-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2024-04-12T04:20:00.819 DEBUG:boto:path=/dkyges-63/obj95 2024-04-12T04:20:00.819 DEBUG:boto:auth_path=/dkyges-63/obj95 2024-04-12T04:20:00.819 DEBUG:boto:Method: GET 2024-04-12T04:20:00.819 DEBUG:boto:Path: /dkyges-63/obj95 2024-04-12T04:20:00.820 DEBUG:boto:Data: 2024-04-12T04:20:00.820 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.820 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.820 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.820 DEBUG:boto:Params: {} 2024-04-12T04:20:00.820 DEBUG:boto:Token: None 2024-04-12T04:20:00.820 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj95 2024-04-12T04:20:00.820 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GTVhnh6Aq6+DISNcNQHmoSSsm4Q= 2024-04-12T04:20:00.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GTVhnh6Aq6+DISNcNQHmoSSsm4Q=', 'Content-Length': '0'} 2024-04-12T04:20:00.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-63:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4620a944fc3d2a4-006618b670-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.822 DEBUG:boto:path=/dkyges-63/obj95 2024-04-12T04:20:00.822 DEBUG:boto:auth_path=/dkyges-63/obj95 2024-04-12T04:20:00.822 DEBUG:boto:Method: GET 2024-04-12T04:20:00.823 DEBUG:boto:Path: /dkyges-63/obj95 2024-04-12T04:20:00.823 DEBUG:boto:Data: 2024-04-12T04:20:00.823 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.823 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.823 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.823 DEBUG:boto:Params: {} 2024-04-12T04:20:00.823 DEBUG:boto:Token: None 2024-04-12T04:20:00.823 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/obj95 2024-04-12T04:20:00.823 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GTVhnh6Aq6+DISNcNQHmoSSsm4Q= 2024-04-12T04:20:00.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GTVhnh6Aq6+DISNcNQHmoSSsm4Q=', 'Content-Length': '0'} 2024-04-12T04:20:00.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:15:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da0099da52eab8c3-006618b670-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.866 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.867 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.867 DEBUG:boto:path=/dkyges-63/?prefix=obj95 2024-04-12T04:20:00.867 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj95 2024-04-12T04:20:00.867 DEBUG:boto:Method: GET 2024-04-12T04:20:00.867 DEBUG:boto:Path: /dkyges-63/?prefix=obj95 2024-04-12T04:20:00.867 DEBUG:boto:Data: 2024-04-12T04:20:00.867 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.867 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:00.867 DEBUG:boto:Port: 8001 2024-04-12T04:20:00.867 DEBUG:boto:Params: {} 2024-04-12T04:20:00.868 DEBUG:boto:Token: None 2024-04-12T04:20:00.868 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.868 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000840680bd7dc3fef-006618b670-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.871 DEBUG:boto:b'dkyges-63obj951000falseobj952024-04-12T04:15:48.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.872 DEBUG:boto:path=/dkyges-63/ 2024-04-12T04:20:00.872 DEBUG:boto:auth_path=/dkyges-63/ 2024-04-12T04:20:00.872 DEBUG:boto:path=/dkyges-63/?prefix=obj95 2024-04-12T04:20:00.872 DEBUG:boto:auth_path=/dkyges-63/?prefix=obj95 2024-04-12T04:20:00.872 DEBUG:boto:Method: GET 2024-04-12T04:20:00.872 DEBUG:boto:Path: /dkyges-63/?prefix=obj95 2024-04-12T04:20:00.872 DEBUG:boto:Data: 2024-04-12T04:20:00.872 DEBUG:boto:Headers: {} 2024-04-12T04:20:00.872 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:00.872 DEBUG:boto:Port: 8000 2024-04-12T04:20:00.872 DEBUG:boto:Params: {} 2024-04-12T04:20:00.873 DEBUG:boto:Token: None 2024-04-12T04:20:00.873 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:00 GMT /dkyges-63/ 2024-04-12T04:20:00.873 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU= 2024-04-12T04:20:00.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:00 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yUQMmTJp/1+lsOLzBT+caIth2dU=', 'Content-Length': '0'} 2024-04-12T04:20:00.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfa19ead18ed7868-006618b670-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:00 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:00.877 DEBUG:boto:b'dkyges-63obj951000falseobj952024-04-12T04:15:48.237Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:00.878 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-63 zones={a2, a1} 2024-04-12T04:20:00.878 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-12T04:20:00.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:00.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:00.879 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:00.996 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:20:00.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:00.996 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000034468", 2024-04-12T04:20:00.996 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:16:00.029445Z" 2024-04-12T04:20:00.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:00.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:00.997 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000023848", 2024-04-12T04:20:00.997 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:16:00.031207Z" 2024-04-12T04:20:00.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:00.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:00.997 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000040207", 2024-04-12T04:20:00.997 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:15:49.733854Z" 2024-04-12T04:20:00.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:00.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:00.998 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000034694", 2024-04-12T04:20:00.998 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:16:00.032722Z" 2024-04-12T04:20:00.998 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:00.998 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:20:00.999 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000034468', 1: '00000000000000000000:00000000000000023848', 2: '00000000000000000000:00000000000000040207', 3: '00000000000000000000:00000000000000034694'} 2024-04-12T04:20:00.999 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:20:00.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:01.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:01.000 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:01.057 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:01.057 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:01.111 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:20:01.112 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:20:01.112 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:20:01.112 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:20:01.112 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:20:01.112 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 10722328783376023132 2024-04-12T04:20:01.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.112 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:20:01.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.112 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:20:01.113 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:20:01.113 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:20:01.113 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000034468", 2024-04-12T04:20:01.113 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:20:01.113 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:20:01.113 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:20:01.113 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:16:00.029445Z" 2024-04-12T04:20:01.113 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.113 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:20:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:20:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:20:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000023848", 2024-04-12T04:20:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:20:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:20:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:20:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:16:00.031207Z" 2024-04-12T04:20:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.114 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:20:01.115 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:20:01.115 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:20:01.115 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000040207", 2024-04-12T04:20:01.115 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:20:01.115 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 54, 2024-04-12T04:20:01.115 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:20:01.115 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:15:49.733854Z" 2024-04-12T04:20:01.115 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.115 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:20:01.116 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:20:01.116 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:20:01.116 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000034694", 2024-04-12T04:20:01.116 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:20:01.116 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:20:01.116 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:20:01.116 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:16:00.032722Z" 2024-04-12T04:20:01.116 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.116 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.116 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:20:01.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.116 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:20:01.117 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 63, 2024-04-12T04:20:01.117 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 63 2024-04-12T04:20:01.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.117 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:20:01Z" 2024-04-12T04:20:01.117 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:20:01.118 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 10722328783376023132 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000034468", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:16:00.029445Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023848", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:16:00.031207Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040207", "next_step_marker": "", "total_entries": 54, "pos": 0, "timestamp": "2024-04-12T04:15:49.733854Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000034694", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:16:00.032722Z" } } ] }, "full_sync": { "total": 63, "complete": 63 }, "current_time": "2024-04-12T04:20:01Z" } 2024-04-12T04:20:01.118 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000034468', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:16:00.029445Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023848', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:16:00.031207Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040207', 'next_step_marker': '', 'total_entries': 54, 'pos': 0, 'timestamp': '2024-04-12T04:15:49.733854Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000034694', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:16:00.032722Z'}}] 2024-04-12T04:20:01.118 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000034468', 1: '00000000000000000000:00000000000000023848', 2: '00000000000000000000:00000000000000040207', 3: '00000000000000000000:00000000000000034694'} 2024-04-12T04:20:01.118 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000034468', 1: '00000000000000000000:00000000000000023848', 2: '00000000000000000000:00000000000000040207', 3: '00000000000000000000:00000000000000034694'} 2024-04-12T04:20:01.119 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:20:01.119 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-12T04:20:01.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:01.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:01.119 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:01.172 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:01.172 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:01.232 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:20:01.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.232 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000013090", 2024-04-12T04:20:01.232 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:17:39.988871Z" 2024-04-12T04:20:01.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.233 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006573", 2024-04-12T04:20:01.233 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:17:50.167613Z" 2024-04-12T04:20:01.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.233 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000018126", 2024-04-12T04:20:01.233 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:18:01.996492Z" 2024-04-12T04:20:01.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.233 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000016296", 2024-04-12T04:20:01.233 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:18:01.998514Z" 2024-04-12T04:20:01.234 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.234 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:20:01.235 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000013090', 1: '00000000000000000000:00000000000000006573', 2: '00000000000000000000:00000000000000018126', 3: '00000000000000000000:00000000000000016296'} 2024-04-12T04:20:01.235 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T04:20:01.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:01.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:01.235 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:01.293 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:01.294 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:01.347 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:20:01.347 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:20:01.347 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:20:01.348 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:20:01.348 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:20:01.348 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5944041172307441597 2024-04-12T04:20:01.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.348 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:20:01.348 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.348 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:20:01.348 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:20:01.348 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:20:01.348 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000013090", 2024-04-12T04:20:01.349 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:20:01.349 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:20:01.349 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:20:01.349 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:17:39.988871Z" 2024-04-12T04:20:01.349 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.349 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.349 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:20:01.349 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:20:01.349 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:20:01.349 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006573", 2024-04-12T04:20:01.350 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:20:01.350 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:20:01.350 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:20:01.350 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:17:50.167613Z" 2024-04-12T04:20:01.350 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.350 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.350 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.350 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:20:01.350 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:20:01.351 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:20:01.351 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000018126", 2024-04-12T04:20:01.351 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:20:01.351 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:20:01.351 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:20:01.351 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:18:01.996492Z" 2024-04-12T04:20:01.351 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.351 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.351 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.351 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:20:01.352 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:20:01.352 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:20:01.352 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000016296", 2024-04-12T04:20:01.352 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:20:01.352 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:20:01.352 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:20:01.352 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:18:01.998514Z" 2024-04-12T04:20:01.352 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.352 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.352 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:20:01.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.353 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:20:01.353 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T04:20:01.353 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T04:20:01.353 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.353 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:20:01Z" 2024-04-12T04:20:01.353 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:20:01.354 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5944041172307441597 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013090", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:39.988871Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006573", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:50.167613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018126", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:18:01.996492Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016296", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:18:01.998514Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T04:20:01Z" } 2024-04-12T04:20:01.354 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013090', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:39.988871Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006573', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:50.167613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018126', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:18:01.996492Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016296', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:18:01.998514Z'}}] 2024-04-12T04:20:01.354 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013090', 1: '00000000000000000000:00000000000000006573', 2: '00000000000000000000:00000000000000018126', 3: '00000000000000000000:00000000000000016296'} 2024-04-12T04:20:01.355 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013090', 1: '00000000000000000000:00000000000000006573', 2: '00000000000000000000:00000000000000018126', 3: '00000000000000000000:00000000000000016296'} 2024-04-12T04:20:01.355 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T04:20:01.355 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... ok 2024-04-12T04:20:01.357 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:01.357 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:01.357 DEBUG:boto:Method: PUT 2024-04-12T04:20:01.357 DEBUG:boto:Path: /dkyges-64/ 2024-04-12T04:20:01.357 DEBUG:boto:Data: 2024-04-12T04:20:01.358 DEBUG:boto:Headers: {} 2024-04-12T04:20:01.358 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:01.358 DEBUG:boto:Port: 8000 2024-04-12T04:20:01.358 DEBUG:boto:Params: {} 2024-04-12T04:20:01.358 DEBUG:boto:Token: None 2024-04-12T04:20:01.358 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:20:01 GMT /dkyges-64/ 2024-04-12T04:20:01.358 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R+a+qCq1D2Wek7NNRVtti3iaRzA= 2024-04-12T04:20:01.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R+a+qCq1D2Wek7NNRVtti3iaRzA=', 'Content-Length': '0'} 2024-04-12T04:20:01.376 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000537aa30b62e3600c-006618b671-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:01.376 DEBUG:rgw_multi.tests:created bucket=dkyges-64 2024-04-12T04:20:01.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:01.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:01.376 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:01.430 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:01.430 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:01.465 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:20:01.481 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:20:01.482 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.482 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:20:01.482 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:20:01.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.482 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.482 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:20:01.482 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:20:01.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.482 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.482 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:20:01.483 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:20:01.483 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.483 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.483 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895601.392109_700.1", 2024-04-12T04:20:01.483 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:20:01.392109Z" 2024-04-12T04:20:01.483 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.483 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:20:01.484 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895601.392109_700.1'} 2024-04-12T04:20:01.484 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:20:01.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:01.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:20:01.485 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:20:01.543 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:01.543 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:01.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:01.596+0000 7f34e5dd8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:20:01.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:01.597+0000 7f34e5dd8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:20:01.595 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:20:01.595 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:20:01.595 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:20:01.595 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:20:01.595 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:20:01.595 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:20:01.595 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:20:01.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.596 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:20:01.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.596 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:20:01.596 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:20:01.596 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:20:01.596 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:20:01.596 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:20:01.596 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:20:01.596 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:20:01.596 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:20:01.597 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:20:01.597 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.597 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.597 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:20:01.597 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:20:01.597 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:20:01.597 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:20:01.597 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:20:01.597 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:20:01.597 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:20:01.598 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:20:01.598 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:20:01.598 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.598 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:20:01.598 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:20:01.598 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:20:01.598 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:20:01.598 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:20:01.599 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:20:01.599 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:20:01.599 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:20:01.599 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:20:01.599 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:20:01.599 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:20:01.599 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:20:01.599 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:20:01.599 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895601.392109_700.1", 2024-04-12T04:20:01.600 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:20:01.600 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:20:01.600 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:20:01.600 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:20:01.392109Z", 2024-04-12T04:20:01.600 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:20:01.600 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.600 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:20:01.600 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:20:01.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.600 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:20:01.601 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:20:01.601 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:20:01.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:20:01.601 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:20:01Z" 2024-04-12T04:20:01.601 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:20:01.602 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895601.392109_700.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:20:01.392109Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:20:01Z" } 2024-04-12T04:20:01.602 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895601.392109_700.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:20:01.392109Z', 'realm_epoch': 2}}] 2024-04-12T04:20:01.603 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895601.392109_700.1'} 2024-04-12T04:20:01.603 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895601.392109_700.1'} 2024-04-12T04:20:01.603 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:20:01.603 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:01.603 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:01.603 DEBUG:boto:Method: HEAD 2024-04-12T04:20:01.603 DEBUG:boto:Path: /dkyges-64/ 2024-04-12T04:20:01.603 DEBUG:boto:Data: 2024-04-12T04:20:01.603 DEBUG:boto:Headers: {} 2024-04-12T04:20:01.604 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:01.604 DEBUG:boto:Port: 8000 2024-04-12T04:20:01.604 DEBUG:boto:Params: {} 2024-04-12T04:20:01.604 DEBUG:boto:Token: None 2024-04-12T04:20:01.604 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:01 GMT /dkyges-64/ 2024-04-12T04:20:01.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mdNsc/X2b3faC9glvP0S7flpplc= 2024-04-12T04:20:01.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mdNsc/X2b3faC9glvP0S7flpplc=', 'Content-Length': '0'} 2024-04-12T04:20:01.607 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e4c86916afcc1e1-006618b671-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:01.607 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:20:01.607 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:20:01.607 DEBUG:boto:Method: PUT 2024-04-12T04:20:01.607 DEBUG:boto:Path: /dkyges-64/dummy 2024-04-12T04:20:01.607 DEBUG:boto:Data: 2024-04-12T04:20:01.608 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:20:01.608 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:01.608 DEBUG:boto:Port: 8000 2024-04-12T04:20:01.608 DEBUG:boto:Params: {} 2024-04-12T04:20:01.608 DEBUG:boto:Token: None 2024-04-12T04:20:01.608 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:20:01 GMT /dkyges-64/dummy 2024-04-12T04:20:01.608 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:d6PvrAebSU8njBbzIl6cTCA7DYc= 2024-04-12T04:20:01.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:20:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d6PvrAebSU8njBbzIl6cTCA7DYc='} 2024-04-12T04:20:01.615 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000051387dde62be97b9-006618b671-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:01.615 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-64 2024-04-12T04:20:01.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:20:01.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:20:01.616 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:20:01.678 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:01.679 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:01.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:01.736+0000 7fb9efdf7780 1 waiting to reach incremental sync.. 2024-04-12T04:20:06.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:06.736+0000 7fb9efdf7780 1 waiting to reach incremental sync.. 2024-04-12T04:20:11.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:11.737+0000 7fb9efdf7780 1 waiting to reach incremental sync.. 2024-04-12T04:20:16.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:16.739+0000 7fb9efdf7780 1 bucket sync caught up with source: 2024-04-12T04:20:16.723 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:20:16.723 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:20:16.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:16.739+0000 7fb9efdf7780 0 bucket checkpoint complete 2024-04-12T04:20:16.742 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-64 2024-04-12T04:20:16.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:20:16.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:20:16.743 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:20:16.808 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:16.808 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:16.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:16.867+0000 7f2dc86fa780 1 waiting to reach incremental sync.. 2024-04-12T04:20:21.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:21.868+0000 7f2dc86fa780 1 waiting to reach incremental sync.. 2024-04-12T04:20:26.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:26.868+0000 7f2dc86fa780 1 waiting to reach incremental sync.. 2024-04-12T04:20:31.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:31.869+0000 7f2dc86fa780 1 waiting to reach incremental sync.. 2024-04-12T04:20:32.940 INFO:tasks.rgw.c1.client.0.smithi043.stdout:2024-04-12T04:20:32.956+0000 7f1e272bd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:20:36.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:36.872+0000 7f2dc86fa780 1 bucket sync caught up with source: 2024-04-12T04:20:36.855 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.3235.6, , , , , ] 2024-04-12T04:20:36.855 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.3235.6, , , , , ] 2024-04-12T04:20:36.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:36.872+0000 7f2dc86fa780 0 bucket checkpoint complete 2024-04-12T04:20:36.874 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-64 zones={a2, a1} 2024-04-12T04:20:36.874 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:36.875 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:36.875 DEBUG:boto:Method: HEAD 2024-04-12T04:20:36.875 DEBUG:boto:Path: /dkyges-64/ 2024-04-12T04:20:36.875 DEBUG:boto:Data: 2024-04-12T04:20:36.875 DEBUG:boto:Headers: {} 2024-04-12T04:20:36.875 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:36.875 DEBUG:boto:Port: 8001 2024-04-12T04:20:36.875 DEBUG:boto:Params: {} 2024-04-12T04:20:36.875 DEBUG:boto:Token: None 2024-04-12T04:20:36.876 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:36 GMT /dkyges-64/ 2024-04-12T04:20:36.876 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:abztmaIsnmdei0aq0v7SyfgMEP4= 2024-04-12T04:20:36.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:abztmaIsnmdei0aq0v7SyfgMEP4=', 'Content-Length': '0'} 2024-04-12T04:20:36.879 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000709264fff401edb6-006618b694-210b-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:20:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:36.879 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:36.879 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:36.879 DEBUG:boto:Method: HEAD 2024-04-12T04:20:36.879 DEBUG:boto:Path: /dkyges-64/ 2024-04-12T04:20:36.879 DEBUG:boto:Data: 2024-04-12T04:20:36.879 DEBUG:boto:Headers: {} 2024-04-12T04:20:36.879 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:36.879 DEBUG:boto:Port: 8000 2024-04-12T04:20:36.879 DEBUG:boto:Params: {} 2024-04-12T04:20:36.880 DEBUG:boto:Token: None 2024-04-12T04:20:36.880 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:36 GMT /dkyges-64/ 2024-04-12T04:20:36.880 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:abztmaIsnmdei0aq0v7SyfgMEP4= 2024-04-12T04:20:36.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:abztmaIsnmdei0aq0v7SyfgMEP4=', 'Content-Length': '0'} 2024-04-12T04:20:36.882 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000baf2459383a56415-006618b694-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:20:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:36.883 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:20:36.883 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:36.883 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:36.883 DEBUG:boto:path=/dkyges-64/?versions&max-keys=999 2024-04-12T04:20:36.883 DEBUG:boto:auth_path=/dkyges-64/?versions&max-keys=999 2024-04-12T04:20:36.883 DEBUG:boto:Method: GET 2024-04-12T04:20:36.883 DEBUG:boto:Path: /dkyges-64/?versions&max-keys=999 2024-04-12T04:20:36.883 DEBUG:boto:Data: 2024-04-12T04:20:36.883 DEBUG:boto:Headers: {} 2024-04-12T04:20:36.883 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:36.883 DEBUG:boto:Port: 8001 2024-04-12T04:20:36.883 DEBUG:boto:Params: {} 2024-04-12T04:20:36.884 DEBUG:boto:Token: None 2024-04-12T04:20:36.884 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:36 GMT /dkyges-64/?versions 2024-04-12T04:20:36.884 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qT/4jmpVD046syrZNdcXhhAoDb4= 2024-04-12T04:20:36.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qT/4jmpVD046syrZNdcXhhAoDb4=', 'Content-Length': '0'} 2024-04-12T04:20:36.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfd38ce7d444d8ba-006618b694-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:36.886 DEBUG:boto:b'dkyges-64999falsedummynulltrue2024-04-12T04:20:01.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:36.887 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T04:20:36.887 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:20:36.887 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:36.887 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:36.887 DEBUG:boto:path=/dkyges-64/?versions&max-keys=999 2024-04-12T04:20:36.887 DEBUG:boto:auth_path=/dkyges-64/?versions&max-keys=999 2024-04-12T04:20:36.887 DEBUG:boto:Method: GET 2024-04-12T04:20:36.887 DEBUG:boto:Path: /dkyges-64/?versions&max-keys=999 2024-04-12T04:20:36.888 DEBUG:boto:Data: 2024-04-12T04:20:36.888 DEBUG:boto:Headers: {} 2024-04-12T04:20:36.888 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:36.888 DEBUG:boto:Port: 8000 2024-04-12T04:20:36.888 DEBUG:boto:Params: {} 2024-04-12T04:20:36.888 DEBUG:boto:Token: None 2024-04-12T04:20:36.888 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:36 GMT /dkyges-64/?versions 2024-04-12T04:20:36.888 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qT/4jmpVD046syrZNdcXhhAoDb4= 2024-04-12T04:20:36.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qT/4jmpVD046syrZNdcXhhAoDb4=', 'Content-Length': '0'} 2024-04-12T04:20:36.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e43db99f8ffb665-006618b694-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:36.934 DEBUG:boto:b'dkyges-64999falsedummynulltrue2024-04-12T04:20:01.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:36.935 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T04:20:36.935 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:36.935 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:36.935 DEBUG:boto:path=/dkyges-64/?versions&max-keys=999 2024-04-12T04:20:36.935 DEBUG:boto:auth_path=/dkyges-64/?versions&max-keys=999 2024-04-12T04:20:36.935 DEBUG:boto:Method: GET 2024-04-12T04:20:36.935 DEBUG:boto:Path: /dkyges-64/?versions&max-keys=999 2024-04-12T04:20:36.935 DEBUG:boto:Data: 2024-04-12T04:20:36.935 DEBUG:boto:Headers: {} 2024-04-12T04:20:36.935 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:36.935 DEBUG:boto:Port: 8001 2024-04-12T04:20:36.936 DEBUG:boto:Params: {} 2024-04-12T04:20:36.936 DEBUG:boto:Token: None 2024-04-12T04:20:36.936 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:36 GMT /dkyges-64/?versions 2024-04-12T04:20:36.936 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qT/4jmpVD046syrZNdcXhhAoDb4= 2024-04-12T04:20:36.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qT/4jmpVD046syrZNdcXhhAoDb4=', 'Content-Length': '0'} 2024-04-12T04:20:36.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009528feab982dc49d-006618b694-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:36.939 DEBUG:boto:b'dkyges-64999falsedummynulltrue2024-04-12T04:20:01.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:36.939 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:36.939 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:36.939 DEBUG:boto:path=/dkyges-64/?versions&max-keys=999 2024-04-12T04:20:36.939 DEBUG:boto:auth_path=/dkyges-64/?versions&max-keys=999 2024-04-12T04:20:36.939 DEBUG:boto:Method: GET 2024-04-12T04:20:36.939 DEBUG:boto:Path: /dkyges-64/?versions&max-keys=999 2024-04-12T04:20:36.940 DEBUG:boto:Data: 2024-04-12T04:20:36.940 DEBUG:boto:Headers: {} 2024-04-12T04:20:36.940 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:36.940 DEBUG:boto:Port: 8000 2024-04-12T04:20:36.940 DEBUG:boto:Params: {} 2024-04-12T04:20:36.940 DEBUG:boto:Token: None 2024-04-12T04:20:36.940 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:36 GMT /dkyges-64/?versions 2024-04-12T04:20:36.940 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qT/4jmpVD046syrZNdcXhhAoDb4= 2024-04-12T04:20:36.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qT/4jmpVD046syrZNdcXhhAoDb4=', 'Content-Length': '0'} 2024-04-12T04:20:36.943 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000808193d3741c0358-006618b694-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:36.943 DEBUG:boto:b'dkyges-64999falsedummynulltrue2024-04-12T04:20:01.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:36.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T04:20:36.943 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:20:36.943 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:20:36.943 DEBUG:boto:path=/dkyges-64/dummy?versionId=null 2024-04-12T04:20:36.943 DEBUG:boto:auth_path=/dkyges-64/dummy?versionId=null 2024-04-12T04:20:36.944 DEBUG:boto:Method: GET 2024-04-12T04:20:36.944 DEBUG:boto:Path: /dkyges-64/dummy?versionId=null 2024-04-12T04:20:36.944 DEBUG:boto:Data: 2024-04-12T04:20:36.944 DEBUG:boto:Headers: {} 2024-04-12T04:20:36.944 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:36.944 DEBUG:boto:Port: 8001 2024-04-12T04:20:36.944 DEBUG:boto:Params: {} 2024-04-12T04:20:36.944 DEBUG:boto:Token: None 2024-04-12T04:20:36.944 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:36 GMT /dkyges-64/dummy?versionId=null 2024-04-12T04:20:36.944 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8k4PweZZz6fivHnbjE1s+t1O5xU= 2024-04-12T04:20:36.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8k4PweZZz6fivHnbjE1s+t1O5xU=', 'Content-Length': '0'} 2024-04-12T04:20:36.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005505e72318fd16ec-006618b694-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:36.990 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:20:36.991 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:20:36.991 DEBUG:boto:path=/dkyges-64/dummy?versionId=null 2024-04-12T04:20:36.991 DEBUG:boto:auth_path=/dkyges-64/dummy?versionId=null 2024-04-12T04:20:36.991 DEBUG:boto:Method: GET 2024-04-12T04:20:36.991 DEBUG:boto:Path: /dkyges-64/dummy?versionId=null 2024-04-12T04:20:36.991 DEBUG:boto:Data: 2024-04-12T04:20:36.991 DEBUG:boto:Headers: {} 2024-04-12T04:20:36.991 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:36.991 DEBUG:boto:Port: 8000 2024-04-12T04:20:36.991 DEBUG:boto:Params: {} 2024-04-12T04:20:36.991 DEBUG:boto:Token: None 2024-04-12T04:20:36.992 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:36 GMT /dkyges-64/dummy?versionId=null 2024-04-12T04:20:36.992 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8k4PweZZz6fivHnbjE1s+t1O5xU= 2024-04-12T04:20:36.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8k4PweZZz6fivHnbjE1s+t1O5xU=', 'Content-Length': '0'} 2024-04-12T04:20:36.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012569ddb9fd14868-006618b695-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:36.994 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:20:36.994 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:20:36.994 DEBUG:boto:path=/dkyges-64/dummy?versionId=null 2024-04-12T04:20:36.994 DEBUG:boto:auth_path=/dkyges-64/dummy?versionId=null 2024-04-12T04:20:36.994 DEBUG:boto:Method: HEAD 2024-04-12T04:20:36.994 DEBUG:boto:Path: /dkyges-64/dummy?versionId=null 2024-04-12T04:20:36.994 DEBUG:boto:Data: 2024-04-12T04:20:36.995 DEBUG:boto:Headers: {} 2024-04-12T04:20:36.995 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:36.995 DEBUG:boto:Port: 8001 2024-04-12T04:20:36.995 DEBUG:boto:Params: {} 2024-04-12T04:20:36.995 DEBUG:boto:Token: None 2024-04-12T04:20:36.995 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:36 GMT /dkyges-64/dummy?versionId=null 2024-04-12T04:20:36.995 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nTTnXUfQvkIhhmgskQD+8J+lmk8= 2024-04-12T04:20:36.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nTTnXUfQvkIhhmgskQD+8J+lmk8=', 'Content-Length': '0'} 2024-04-12T04:20:36.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e90f4f1233ed3b9-006618b695-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:36.998 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:20:36.998 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:20:36.998 DEBUG:boto:path=/dkyges-64/dummy?versionId=null 2024-04-12T04:20:36.998 DEBUG:boto:auth_path=/dkyges-64/dummy?versionId=null 2024-04-12T04:20:36.998 DEBUG:boto:Method: HEAD 2024-04-12T04:20:36.998 DEBUG:boto:Path: /dkyges-64/dummy?versionId=null 2024-04-12T04:20:36.998 DEBUG:boto:Data: 2024-04-12T04:20:36.998 DEBUG:boto:Headers: {} 2024-04-12T04:20:36.998 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:36.999 DEBUG:boto:Port: 8000 2024-04-12T04:20:36.999 DEBUG:boto:Params: {} 2024-04-12T04:20:36.999 DEBUG:boto:Token: None 2024-04-12T04:20:36.999 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:36 GMT /dkyges-64/dummy?versionId=null 2024-04-12T04:20:36.999 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nTTnXUfQvkIhhmgskQD+8J+lmk8= 2024-04-12T04:20:36.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nTTnXUfQvkIhhmgskQD+8J+lmk8=', 'Content-Length': '0'} 2024-04-12T04:20:37.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b8c2c45b12f0954-006618b695-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:37.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T04:20:37.001 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:20:37.001 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:20:37.001 DEBUG:boto:path=/dkyges-64/dummy?versionId=null 2024-04-12T04:20:37.002 DEBUG:boto:auth_path=/dkyges-64/dummy?versionId=null 2024-04-12T04:20:37.002 DEBUG:boto:Method: GET 2024-04-12T04:20:37.002 DEBUG:boto:Path: /dkyges-64/dummy?versionId=null 2024-04-12T04:20:37.002 DEBUG:boto:Data: 2024-04-12T04:20:37.002 DEBUG:boto:Headers: {} 2024-04-12T04:20:37.002 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:37.002 DEBUG:boto:Port: 8001 2024-04-12T04:20:37.002 DEBUG:boto:Params: {} 2024-04-12T04:20:37.002 DEBUG:boto:Token: None 2024-04-12T04:20:37.003 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:37 GMT /dkyges-64/dummy?versionId=null 2024-04-12T04:20:37.003 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ACmajfCDHS6PzQfvTDCzHSSUG04= 2024-04-12T04:20:37.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:37 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ACmajfCDHS6PzQfvTDCzHSSUG04=', 'Content-Length': '0'} 2024-04-12T04:20:37.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cd076b5c1023449-006618b695-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:37.005 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:20:37.005 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:20:37.005 DEBUG:boto:path=/dkyges-64/dummy?versionId=null 2024-04-12T04:20:37.005 DEBUG:boto:auth_path=/dkyges-64/dummy?versionId=null 2024-04-12T04:20:37.005 DEBUG:boto:Method: GET 2024-04-12T04:20:37.005 DEBUG:boto:Path: /dkyges-64/dummy?versionId=null 2024-04-12T04:20:37.005 DEBUG:boto:Data: 2024-04-12T04:20:37.006 DEBUG:boto:Headers: {} 2024-04-12T04:20:37.006 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:37.006 DEBUG:boto:Port: 8000 2024-04-12T04:20:37.006 DEBUG:boto:Params: {} 2024-04-12T04:20:37.006 DEBUG:boto:Token: None 2024-04-12T04:20:37.006 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:37 GMT /dkyges-64/dummy?versionId=null 2024-04-12T04:20:37.006 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ACmajfCDHS6PzQfvTDCzHSSUG04= 2024-04-12T04:20:37.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:37 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ACmajfCDHS6PzQfvTDCzHSSUG04=', 'Content-Length': '0'} 2024-04-12T04:20:37.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003ddf19730c86043-006618b695-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:37.009 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:37.009 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:37.009 DEBUG:boto:path=/dkyges-64/?prefix=dummy 2024-04-12T04:20:37.009 DEBUG:boto:auth_path=/dkyges-64/?prefix=dummy 2024-04-12T04:20:37.009 DEBUG:boto:Method: GET 2024-04-12T04:20:37.009 DEBUG:boto:Path: /dkyges-64/?prefix=dummy 2024-04-12T04:20:37.009 DEBUG:boto:Data: 2024-04-12T04:20:37.009 DEBUG:boto:Headers: {} 2024-04-12T04:20:37.009 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:37.009 DEBUG:boto:Port: 8001 2024-04-12T04:20:37.009 DEBUG:boto:Params: {} 2024-04-12T04:20:37.010 DEBUG:boto:Token: None 2024-04-12T04:20:37.010 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:37 GMT /dkyges-64/ 2024-04-12T04:20:37.010 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iMeNlB1snNVo9d1+K8aC3B9c8P8= 2024-04-12T04:20:37.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:37 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iMeNlB1snNVo9d1+K8aC3B9c8P8=', 'Content-Length': '0'} 2024-04-12T04:20:37.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a853537e4bd25644-006618b695-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:37.013 DEBUG:boto:b'dkyges-64dummy1000falsedummy2024-04-12T04:20:01.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:37.013 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:37.013 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:37.013 DEBUG:boto:path=/dkyges-64/?prefix=dummy 2024-04-12T04:20:37.013 DEBUG:boto:auth_path=/dkyges-64/?prefix=dummy 2024-04-12T04:20:37.014 DEBUG:boto:Method: GET 2024-04-12T04:20:37.014 DEBUG:boto:Path: /dkyges-64/?prefix=dummy 2024-04-12T04:20:37.014 DEBUG:boto:Data: 2024-04-12T04:20:37.014 DEBUG:boto:Headers: {} 2024-04-12T04:20:37.014 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:37.014 DEBUG:boto:Port: 8000 2024-04-12T04:20:37.014 DEBUG:boto:Params: {} 2024-04-12T04:20:37.014 DEBUG:boto:Token: None 2024-04-12T04:20:37.014 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:37 GMT /dkyges-64/ 2024-04-12T04:20:37.014 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iMeNlB1snNVo9d1+K8aC3B9c8P8= 2024-04-12T04:20:37.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:37 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iMeNlB1snNVo9d1+K8aC3B9c8P8=', 'Content-Length': '0'} 2024-04-12T04:20:37.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000122c878b0a3ba378-006618b695-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:37.017 DEBUG:boto:b'dkyges-64dummy1000falsedummy2024-04-12T04:20:01.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:37.017 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:20:37.018 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:20:37.018 DEBUG:boto:Method: HEAD 2024-04-12T04:20:37.018 DEBUG:boto:Path: /dkyges-64/dummy 2024-04-12T04:20:37.018 DEBUG:boto:Data: 2024-04-12T04:20:37.018 DEBUG:boto:Headers: {} 2024-04-12T04:20:37.018 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:37.018 DEBUG:boto:Port: 8001 2024-04-12T04:20:37.018 DEBUG:boto:Params: {} 2024-04-12T04:20:37.018 DEBUG:boto:Token: None 2024-04-12T04:20:37.019 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:37 GMT /dkyges-64/dummy 2024-04-12T04:20:37.019 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycFCkE4djMFbPQaZNd7ea4aogOc= 2024-04-12T04:20:37.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:37 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycFCkE4djMFbPQaZNd7ea4aogOc=', 'Content-Length': '0'} 2024-04-12T04:20:37.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f62315211a03da9-006618b695-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:37.021 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:20:37.021 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:20:37.021 DEBUG:boto:Method: HEAD 2024-04-12T04:20:37.021 DEBUG:boto:Path: /dkyges-64/dummy 2024-04-12T04:20:37.021 DEBUG:boto:Data: 2024-04-12T04:20:37.021 DEBUG:boto:Headers: {} 2024-04-12T04:20:37.021 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:37.021 DEBUG:boto:Port: 8000 2024-04-12T04:20:37.021 DEBUG:boto:Params: {} 2024-04-12T04:20:37.022 DEBUG:boto:Token: None 2024-04-12T04:20:37.022 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:37 GMT /dkyges-64/dummy 2024-04-12T04:20:37.022 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycFCkE4djMFbPQaZNd7ea4aogOc= 2024-04-12T04:20:37.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:37 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycFCkE4djMFbPQaZNd7ea4aogOc=', 'Content-Length': '0'} 2024-04-12T04:20:37.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062863ac02adee8cd-006618b695-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:37.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T04:20:37.024 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:20:37.024 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:20:37.024 DEBUG:boto:Method: GET 2024-04-12T04:20:37.024 DEBUG:boto:Path: /dkyges-64/dummy 2024-04-12T04:20:37.024 DEBUG:boto:Data: 2024-04-12T04:20:37.024 DEBUG:boto:Headers: {} 2024-04-12T04:20:37.024 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:37.025 DEBUG:boto:Port: 8001 2024-04-12T04:20:37.025 DEBUG:boto:Params: {} 2024-04-12T04:20:37.025 DEBUG:boto:Token: None 2024-04-12T04:20:37.025 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:37 GMT /dkyges-64/dummy 2024-04-12T04:20:37.025 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QvdGl72qgaxXFUC+IKAtj8JOxVM= 2024-04-12T04:20:37.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:37 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QvdGl72qgaxXFUC+IKAtj8JOxVM=', 'Content-Length': '0'} 2024-04-12T04:20:37.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003972e7f967ec78d8-006618b695-210b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:37.027 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:20:37.027 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:20:37.027 DEBUG:boto:Method: GET 2024-04-12T04:20:37.027 DEBUG:boto:Path: /dkyges-64/dummy 2024-04-12T04:20:37.027 DEBUG:boto:Data: 2024-04-12T04:20:37.027 DEBUG:boto:Headers: {} 2024-04-12T04:20:37.027 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:37.028 DEBUG:boto:Port: 8000 2024-04-12T04:20:37.028 DEBUG:boto:Params: {} 2024-04-12T04:20:37.028 DEBUG:boto:Token: None 2024-04-12T04:20:37.028 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:37 GMT /dkyges-64/dummy 2024-04-12T04:20:37.028 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QvdGl72qgaxXFUC+IKAtj8JOxVM= 2024-04-12T04:20:37.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:37 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QvdGl72qgaxXFUC+IKAtj8JOxVM=', 'Content-Length': '0'} 2024-04-12T04:20:37.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc0bee8df68973b7-006618b695-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:20:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:37.030 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:37.030 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:37.031 DEBUG:boto:path=/dkyges-64/?prefix=dummy 2024-04-12T04:20:37.031 DEBUG:boto:auth_path=/dkyges-64/?prefix=dummy 2024-04-12T04:20:37.031 DEBUG:boto:Method: GET 2024-04-12T04:20:37.031 DEBUG:boto:Path: /dkyges-64/?prefix=dummy 2024-04-12T04:20:37.031 DEBUG:boto:Data: 2024-04-12T04:20:37.031 DEBUG:boto:Headers: {} 2024-04-12T04:20:37.031 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:37.031 DEBUG:boto:Port: 8001 2024-04-12T04:20:37.031 DEBUG:boto:Params: {} 2024-04-12T04:20:37.031 DEBUG:boto:Token: None 2024-04-12T04:20:37.032 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:37 GMT /dkyges-64/ 2024-04-12T04:20:37.032 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iMeNlB1snNVo9d1+K8aC3B9c8P8= 2024-04-12T04:20:37.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:37 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iMeNlB1snNVo9d1+K8aC3B9c8P8=', 'Content-Length': '0'} 2024-04-12T04:20:37.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e4cc277e9b3ef01-006618b695-210b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:37.034 DEBUG:boto:b'dkyges-64dummy1000falsedummy2024-04-12T04:20:01.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:37.035 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:37.035 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:37.035 DEBUG:boto:path=/dkyges-64/?prefix=dummy 2024-04-12T04:20:37.035 DEBUG:boto:auth_path=/dkyges-64/?prefix=dummy 2024-04-12T04:20:37.035 DEBUG:boto:Method: GET 2024-04-12T04:20:37.035 DEBUG:boto:Path: /dkyges-64/?prefix=dummy 2024-04-12T04:20:37.035 DEBUG:boto:Data: 2024-04-12T04:20:37.035 DEBUG:boto:Headers: {} 2024-04-12T04:20:37.035 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:37.035 DEBUG:boto:Port: 8000 2024-04-12T04:20:37.035 DEBUG:boto:Params: {} 2024-04-12T04:20:37.036 DEBUG:boto:Token: None 2024-04-12T04:20:37.036 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:20:37 GMT /dkyges-64/ 2024-04-12T04:20:37.036 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:iMeNlB1snNVo9d1+K8aC3B9c8P8= 2024-04-12T04:20:37.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:37 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:iMeNlB1snNVo9d1+K8aC3B9c8P8=', 'Content-Length': '0'} 2024-04-12T04:20:37.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e413b20d01d1dad7-006618b695-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:20:37 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:37.039 DEBUG:boto:b'dkyges-64dummy1000falsedummy2024-04-12T04:20:01.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:20:37.039 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-64 zones={a2, a1} 2024-04-12T04:20:37.039 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T04:20:37.039 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:20:37.040 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T04:20:37.057+0000 7ff40d1e4700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 106973) UID: 0 2024-04-12T04:20:37.040 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T04:20:37.057+0000 7ff4123b97c0 -1 shutting down 2024-04-12T04:20:37.943 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T04:20:43.140 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T04:20:43.140 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:43.140 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:43.140 DEBUG:boto:Method: HEAD 2024-04-12T04:20:43.141 DEBUG:boto:Path: /dkyges-64/ 2024-04-12T04:20:43.141 DEBUG:boto:Data: 2024-04-12T04:20:43.141 DEBUG:boto:Headers: {} 2024-04-12T04:20:43.141 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:43.141 DEBUG:boto:Port: 8000 2024-04-12T04:20:43.141 DEBUG:boto:Params: {} 2024-04-12T04:20:43.141 DEBUG:boto:Token: None 2024-04-12T04:20:43.141 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:43 GMT /dkyges-64/ 2024-04-12T04:20:43.141 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1tmpIu9OHI6DmbsAeN9gHVU/KXE= 2024-04-12T04:20:43.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1tmpIu9OHI6DmbsAeN9gHVU/KXE=', 'Content-Length': '0'} 2024-04-12T04:20:43.144 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af152921702cc8f6-006618b69b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:43.145 DEBUG:boto:path=/dkyges-64/obj11 2024-04-12T04:20:43.145 DEBUG:boto:auth_path=/dkyges-64/obj11 2024-04-12T04:20:43.145 DEBUG:boto:Method: PUT 2024-04-12T04:20:43.145 DEBUG:boto:Path: /dkyges-64/obj11 2024-04-12T04:20:43.145 DEBUG:boto:Data: 2024-04-12T04:20:43.145 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:20:43.145 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:43.145 DEBUG:boto:Port: 8000 2024-04-12T04:20:43.145 DEBUG:boto:Params: {} 2024-04-12T04:20:43.146 DEBUG:boto:Token: None 2024-04-12T04:20:43.146 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:20:43 GMT /dkyges-64/obj11 2024-04-12T04:20:43.146 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NCk4DmZ0NlK8G9eCwKOFtJ7P9qA= 2024-04-12T04:20:43.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:20:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NCk4DmZ0NlK8G9eCwKOFtJ7P9qA='} 2024-04-12T04:20:43.153 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d20d79061cd675ae-006618b69b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:43.153 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:43.153 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:43.153 DEBUG:boto:Method: HEAD 2024-04-12T04:20:43.153 DEBUG:boto:Path: /dkyges-64/ 2024-04-12T04:20:43.153 DEBUG:boto:Data: 2024-04-12T04:20:43.153 DEBUG:boto:Headers: {} 2024-04-12T04:20:43.154 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:43.154 DEBUG:boto:Port: 8000 2024-04-12T04:20:43.154 DEBUG:boto:Params: {} 2024-04-12T04:20:43.154 DEBUG:boto:Token: None 2024-04-12T04:20:43.154 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:43 GMT /dkyges-64/ 2024-04-12T04:20:43.154 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1tmpIu9OHI6DmbsAeN9gHVU/KXE= 2024-04-12T04:20:43.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1tmpIu9OHI6DmbsAeN9gHVU/KXE=', 'Content-Length': '0'} 2024-04-12T04:20:43.157 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000531a0b5959577035-006618b69b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:43.157 DEBUG:boto:path=/dkyges-64/obj22 2024-04-12T04:20:43.157 DEBUG:boto:auth_path=/dkyges-64/obj22 2024-04-12T04:20:43.157 DEBUG:boto:Method: PUT 2024-04-12T04:20:43.157 DEBUG:boto:Path: /dkyges-64/obj22 2024-04-12T04:20:43.157 DEBUG:boto:Data: 2024-04-12T04:20:43.158 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:20:43.158 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:43.158 DEBUG:boto:Port: 8000 2024-04-12T04:20:43.158 DEBUG:boto:Params: {} 2024-04-12T04:20:43.158 DEBUG:boto:Token: None 2024-04-12T04:20:43.158 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:20:43 GMT /dkyges-64/obj22 2024-04-12T04:20:43.158 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2oIRqeWCI653pO7Wo9qQ2H0WYOY= 2024-04-12T04:20:43.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:20:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2oIRqeWCI653pO7Wo9qQ2H0WYOY='} 2024-04-12T04:20:43.203 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d04d2971b2a2ec6-006618b69b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:43.203 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:43.203 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:43.203 DEBUG:boto:Method: HEAD 2024-04-12T04:20:43.203 DEBUG:boto:Path: /dkyges-64/ 2024-04-12T04:20:43.203 DEBUG:boto:Data: 2024-04-12T04:20:43.203 DEBUG:boto:Headers: {} 2024-04-12T04:20:43.203 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:43.203 DEBUG:boto:Port: 8000 2024-04-12T04:20:43.203 DEBUG:boto:Params: {} 2024-04-12T04:20:43.203 DEBUG:boto:Token: None 2024-04-12T04:20:43.204 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:43 GMT /dkyges-64/ 2024-04-12T04:20:43.204 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1tmpIu9OHI6DmbsAeN9gHVU/KXE= 2024-04-12T04:20:43.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1tmpIu9OHI6DmbsAeN9gHVU/KXE=', 'Content-Length': '0'} 2024-04-12T04:20:43.206 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ad7d3a813b48085-006618b69b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:43.207 DEBUG:boto:path=/dkyges-64/obj33 2024-04-12T04:20:43.207 DEBUG:boto:auth_path=/dkyges-64/obj33 2024-04-12T04:20:43.207 DEBUG:boto:Method: PUT 2024-04-12T04:20:43.207 DEBUG:boto:Path: /dkyges-64/obj33 2024-04-12T04:20:43.207 DEBUG:boto:Data: 2024-04-12T04:20:43.207 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:20:43.207 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:43.207 DEBUG:boto:Port: 8000 2024-04-12T04:20:43.207 DEBUG:boto:Params: {} 2024-04-12T04:20:43.208 DEBUG:boto:Token: None 2024-04-12T04:20:43.208 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:20:43 GMT /dkyges-64/obj33 2024-04-12T04:20:43.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jXbWLHTFqb6z3ASbLFnkkUTKzY0= 2024-04-12T04:20:43.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:20:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jXbWLHTFqb6z3ASbLFnkkUTKzY0='} 2024-04-12T04:20:43.215 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004748a4edc76684dd-006618b69b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:43.215 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:43.215 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:43.215 DEBUG:boto:Method: HEAD 2024-04-12T04:20:43.215 DEBUG:boto:Path: /dkyges-64/ 2024-04-12T04:20:43.215 DEBUG:boto:Data: 2024-04-12T04:20:43.215 DEBUG:boto:Headers: {} 2024-04-12T04:20:43.215 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:43.216 DEBUG:boto:Port: 8000 2024-04-12T04:20:43.216 DEBUG:boto:Params: {} 2024-04-12T04:20:43.216 DEBUG:boto:Token: None 2024-04-12T04:20:43.216 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:43 GMT /dkyges-64/ 2024-04-12T04:20:43.216 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1tmpIu9OHI6DmbsAeN9gHVU/KXE= 2024-04-12T04:20:43.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1tmpIu9OHI6DmbsAeN9gHVU/KXE=', 'Content-Length': '0'} 2024-04-12T04:20:43.219 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b967e5554265d695-006618b69b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:43.219 DEBUG:boto:path=/dkyges-64/obj44 2024-04-12T04:20:43.219 DEBUG:boto:auth_path=/dkyges-64/obj44 2024-04-12T04:20:43.219 DEBUG:boto:Method: PUT 2024-04-12T04:20:43.219 DEBUG:boto:Path: /dkyges-64/obj44 2024-04-12T04:20:43.219 DEBUG:boto:Data: 2024-04-12T04:20:43.219 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:20:43.219 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:43.219 DEBUG:boto:Port: 8000 2024-04-12T04:20:43.220 DEBUG:boto:Params: {} 2024-04-12T04:20:43.220 DEBUG:boto:Token: None 2024-04-12T04:20:43.220 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:20:43 GMT /dkyges-64/obj44 2024-04-12T04:20:43.220 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:plFeV9qY9Si5uy68JWXp2tvx4rM= 2024-04-12T04:20:43.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:20:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:plFeV9qY9Si5uy68JWXp2tvx4rM='} 2024-04-12T04:20:43.262 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db0852db92d1b71b-006618b69b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:43.263 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:20:43.263 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:20:43.263 DEBUG:boto:Method: HEAD 2024-04-12T04:20:43.263 DEBUG:boto:Path: /dkyges-64/ 2024-04-12T04:20:43.263 DEBUG:boto:Data: 2024-04-12T04:20:43.263 DEBUG:boto:Headers: {} 2024-04-12T04:20:43.263 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:43.263 DEBUG:boto:Port: 8000 2024-04-12T04:20:43.263 DEBUG:boto:Params: {} 2024-04-12T04:20:43.263 DEBUG:boto:Token: None 2024-04-12T04:20:43.264 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:20:43 GMT /dkyges-64/ 2024-04-12T04:20:43.264 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1tmpIu9OHI6DmbsAeN9gHVU/KXE= 2024-04-12T04:20:43.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:20:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1tmpIu9OHI6DmbsAeN9gHVU/KXE=', 'Content-Length': '0'} 2024-04-12T04:20:43.266 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000068e1b681d1a6c6f-006618b69b-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:43.267 DEBUG:boto:path=/dkyges-64/obj55 2024-04-12T04:20:43.267 DEBUG:boto:auth_path=/dkyges-64/obj55 2024-04-12T04:20:43.267 DEBUG:boto:Method: PUT 2024-04-12T04:20:43.267 DEBUG:boto:Path: /dkyges-64/obj55 2024-04-12T04:20:43.267 DEBUG:boto:Data: 2024-04-12T04:20:43.267 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:20:43.267 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:20:43.267 DEBUG:boto:Port: 8000 2024-04-12T04:20:43.267 DEBUG:boto:Params: {} 2024-04-12T04:20:43.267 DEBUG:boto:Token: None 2024-04-12T04:20:43.268 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:20:43 GMT /dkyges-64/obj55 2024-04-12T04:20:43.268 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7N9eytatbFqtVFdmWm6Vl4S5SBI= 2024-04-12T04:20:43.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:20:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7N9eytatbFqtVFdmWm6Vl4S5SBI='} 2024-04-12T04:20:43.273 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000406f4ebb957945a-006618b69b-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:20:43.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:20:43.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:20:43.274 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:20:43.335 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:43.335 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:43.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:43.391+0000 7f82c4b9d780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-12T04:20:43.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:43.393+0000 7f82c4b9d780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-12T04:20:43.403 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T04:20:43.403 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T04:20:43.405 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T04:20:43.406 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:43.428 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:20:43.428 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T04:20:43.430 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T04:20:43.430 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi043.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T04:20:43.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T04:20:44.431 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T04:20:44.458 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:20:44.458 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T04:20:44.459 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-12T04:20:44.460 INFO:teuthology.orchestra.run.smithi043.stdout:anonymous 2024-04-12T04:20:44.460 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-64 2024-04-12T04:20:44.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:20:44.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:20:44.461 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:20:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:20:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:20:44.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:44.625+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:20:44.610 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:20:44.610 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:20:49.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:49.627+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:20:49.612 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:20:49.612 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:20:54.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:54.628+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:20:54.613 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:20:54.614 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:20:59.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:20:59.630+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:20:59.615 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:20:59.615 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:21:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:21:04.631+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:21:04.616 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:21:04.617 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:21:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:21:09.633+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:21:09.618 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:21:09.618 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:21:14.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:21:14.634+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:21:14.620 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:21:14.620 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:21:19.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:21:19.635+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:21:19.621 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:21:19.621 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:21:24.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:21:24.637+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:21:24.623 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:21:24.623 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:21:29.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:21:29.638+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:21:29.624 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:21:29.624 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:21:34.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:21:34.639+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:21:34.625 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:21:34.625 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:21:39.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:21:39.640+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:21:39.627 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:21:39.627 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:21:44.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:21:44.641+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:21:44.627 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:21:44.628 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:21:49.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:21:49.642+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:21:49.629 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:21:49.629 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:21:54.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:21:54.645+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:21:54.630 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:21:54.631 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:21:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:21:59.645+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:21:59.632 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:21:59.632 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:22:04.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:22:04.646+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:22:04.634 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:22:04.634 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:22:09.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:22:09.648+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:22:09.635 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , ] 2024-04-12T04:22:09.635 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:22:14.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:22:14.649+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:22:14.637 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , , 00000000001.70.6] 2024-04-12T04:22:14.637 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:22:19.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:22:19.650+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:22:19.638 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.104.6, 00000000001.70.6] 2024-04-12T04:22:19.638 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:22:24.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:22:24.652+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:22:24.640 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.104.6, 00000000001.70.6] 2024-04-12T04:22:24.640 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:22:29.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:22:29.653+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:22:29.641 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.104.6, 00000000001.70.6] 2024-04-12T04:22:29.641 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:22:34.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:22:34.654+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:22:34.642 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.104.6, 00000000001.70.6] 2024-04-12T04:22:34.642 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:22:39.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:22:39.656+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:22:39.644 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.104.6, 00000000001.70.6] 2024-04-12T04:22:39.644 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:22:44.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:22:44.657+0000 7fced9558780 1 waiting for incremental sync to catch up: 2024-04-12T04:22:44.645 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , , 00000000002.104.6, 00000000001.70.6] 2024-04-12T04:22:44.645 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:22:49.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:22:49.658+0000 7fced9558780 1 bucket sync caught up with source: 2024-04-12T04:22:49.666 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:22:49.666 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.100.6, , , 00000000001.74.6, 00000000002.104.6, 00000000002.72.6] 2024-04-12T04:22:49.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:22:49.658+0000 7fced9558780 0 bucket checkpoint complete 2024-04-12T04:22:49.667 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-64 2024-04-12T04:22:49.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:22:49.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:22:49.668 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:22:49.736 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:49.736 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:49.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:22:49.794+0000 7fc193947780 1 waiting for incremental sync to catch up: 2024-04-12T04:22:49.782 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.3235.6, , , , 00000000002.3239.6, 00000000001.888.6] 2024-04-12T04:22:49.782 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.3235.6, , , 00000000001.608.6, 00000000002.3239.6, 00000000002.890.6] 2024-04-12T04:22:54.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:22:54.795+0000 7fc193947780 1 bucket sync caught up with source: 2024-04-12T04:22:54.785 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , 00000000001.3235.6, , , 00000000001.608.6, 00000000002.3239.6, 00000000002.890.6] 2024-04-12T04:22:54.785 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , 00000000001.3235.6, , , 00000000001.608.6, 00000000002.3239.6, 00000000002.890.6] 2024-04-12T04:22:54.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:22:54.795+0000 7fc193947780 0 bucket checkpoint complete 2024-04-12T04:22:54.804 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-64 zones={a2, a1} 2024-04-12T04:22:54.804 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:54.804 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:54.805 DEBUG:boto:Method: HEAD 2024-04-12T04:22:54.805 DEBUG:boto:Path: /dkyges-64/ 2024-04-12T04:22:54.805 DEBUG:boto:Data: 2024-04-12T04:22:54.806 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.806 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:54.806 DEBUG:boto:Port: 8001 2024-04-12T04:22:54.806 DEBUG:boto:Params: {} 2024-04-12T04:22:54.807 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:22:54.807 DEBUG:boto:Token: None 2024-04-12T04:22:54.808 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/ 2024-04-12T04:22:54.808 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/raHuVo5Q24j/Z1t+Xw0s1J39dw= 2024-04-12T04:22:54.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/raHuVo5Q24j/Z1t+Xw0s1J39dw=', 'Content-Length': '0'} 2024-04-12T04:22:54.826 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f4cdfa3c2a9a11a-006618b71e-216b-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:54.827 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:54.827 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:54.827 DEBUG:boto:Method: HEAD 2024-04-12T04:22:54.827 DEBUG:boto:Path: /dkyges-64/ 2024-04-12T04:22:54.827 DEBUG:boto:Data: 2024-04-12T04:22:54.827 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.827 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:54.827 DEBUG:boto:Port: 8000 2024-04-12T04:22:54.827 DEBUG:boto:Params: {} 2024-04-12T04:22:54.828 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:22:54.828 DEBUG:boto:Token: None 2024-04-12T04:22:54.828 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/ 2024-04-12T04:22:54.828 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/raHuVo5Q24j/Z1t+Xw0s1J39dw= 2024-04-12T04:22:54.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/raHuVo5Q24j/Z1t+Xw0s1J39dw=', 'Content-Length': '0'} 2024-04-12T04:22:54.842 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009c44533f225bfab0-006618b71e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:54.843 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:22:54.843 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:54.843 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:54.844 DEBUG:boto:path=/dkyges-64/?versions&max-keys=999 2024-04-12T04:22:54.844 DEBUG:boto:auth_path=/dkyges-64/?versions&max-keys=999 2024-04-12T04:22:54.844 DEBUG:boto:Method: GET 2024-04-12T04:22:54.844 DEBUG:boto:Path: /dkyges-64/?versions&max-keys=999 2024-04-12T04:22:54.844 DEBUG:boto:Data: 2024-04-12T04:22:54.844 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.844 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:54.844 DEBUG:boto:Port: 8001 2024-04-12T04:22:54.844 DEBUG:boto:Params: {} 2024-04-12T04:22:54.844 DEBUG:boto:Token: None 2024-04-12T04:22:54.845 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/?versions 2024-04-12T04:22:54.845 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H4PEK6rHnyHXbUV2HqC8WKwRmVk= 2024-04-12T04:22:54.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H4PEK6rHnyHXbUV2HqC8WKwRmVk=', 'Content-Length': '0'} 2024-04-12T04:22:54.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e14ac560e2a1289e-006618b71e-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:54.849 DEBUG:boto:b'dkyges-64999falsedummynulltrue2024-04-12T04:20:01.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-12T04:20:43.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:20:43.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:20:43.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:20:43.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:20:43.286Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:54.850 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T04:22:54.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:22:54.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:22:54.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:22:54.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:22:54.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:22:54.851 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:22:54.851 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:54.851 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:54.851 DEBUG:boto:path=/dkyges-64/?versions&max-keys=999 2024-04-12T04:22:54.852 DEBUG:boto:auth_path=/dkyges-64/?versions&max-keys=999 2024-04-12T04:22:54.852 DEBUG:boto:Method: GET 2024-04-12T04:22:54.852 DEBUG:boto:Path: /dkyges-64/?versions&max-keys=999 2024-04-12T04:22:54.852 DEBUG:boto:Data: 2024-04-12T04:22:54.852 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.852 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:54.852 DEBUG:boto:Port: 8000 2024-04-12T04:22:54.852 DEBUG:boto:Params: {} 2024-04-12T04:22:54.852 DEBUG:boto:Token: None 2024-04-12T04:22:54.852 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/?versions 2024-04-12T04:22:54.853 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H4PEK6rHnyHXbUV2HqC8WKwRmVk= 2024-04-12T04:22:54.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H4PEK6rHnyHXbUV2HqC8WKwRmVk=', 'Content-Length': '0'} 2024-04-12T04:22:54.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a411857664732f7-006618b71e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:54.856 DEBUG:boto:b'dkyges-64999falsedummynulltrue2024-04-12T04:20:01.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-12T04:20:43.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:20:43.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:20:43.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:20:43.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:20:43.286Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:54.856 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2024-04-12T04:22:54.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:22:54.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:22:54.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:22:54.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:22:54.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:22:54.857 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:54.857 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:54.857 DEBUG:boto:path=/dkyges-64/?versions&max-keys=999 2024-04-12T04:22:54.857 DEBUG:boto:auth_path=/dkyges-64/?versions&max-keys=999 2024-04-12T04:22:54.857 DEBUG:boto:Method: GET 2024-04-12T04:22:54.857 DEBUG:boto:Path: /dkyges-64/?versions&max-keys=999 2024-04-12T04:22:54.857 DEBUG:boto:Data: 2024-04-12T04:22:54.857 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.858 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:54.858 DEBUG:boto:Port: 8001 2024-04-12T04:22:54.858 DEBUG:boto:Params: {} 2024-04-12T04:22:54.858 DEBUG:boto:Token: None 2024-04-12T04:22:54.858 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/?versions 2024-04-12T04:22:54.858 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H4PEK6rHnyHXbUV2HqC8WKwRmVk= 2024-04-12T04:22:54.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H4PEK6rHnyHXbUV2HqC8WKwRmVk=', 'Content-Length': '0'} 2024-04-12T04:22:54.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034b8d07b501c2f93-006618b71e-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:54.861 DEBUG:boto:b'dkyges-64999falsedummynulltrue2024-04-12T04:20:01.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-12T04:20:43.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:20:43.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:20:43.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:20:43.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:20:43.286Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:54.861 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:54.861 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:54.862 DEBUG:boto:path=/dkyges-64/?versions&max-keys=999 2024-04-12T04:22:54.862 DEBUG:boto:auth_path=/dkyges-64/?versions&max-keys=999 2024-04-12T04:22:54.862 DEBUG:boto:Method: GET 2024-04-12T04:22:54.862 DEBUG:boto:Path: /dkyges-64/?versions&max-keys=999 2024-04-12T04:22:54.862 DEBUG:boto:Data: 2024-04-12T04:22:54.862 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.862 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:54.862 DEBUG:boto:Port: 8000 2024-04-12T04:22:54.862 DEBUG:boto:Params: {} 2024-04-12T04:22:54.862 DEBUG:boto:Token: None 2024-04-12T04:22:54.863 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/?versions 2024-04-12T04:22:54.863 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:H4PEK6rHnyHXbUV2HqC8WKwRmVk= 2024-04-12T04:22:54.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H4PEK6rHnyHXbUV2HqC8WKwRmVk=', 'Content-Length': '0'} 2024-04-12T04:22:54.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc3af01111e62e6b-006618b71e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:54.867 DEBUG:boto:b'dkyges-64999falsedummynulltrue2024-04-12T04:20:01.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2024-04-12T04:20:43.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:20:43.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:20:43.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:20:43.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:20:43.286Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:54.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T04:22:54.868 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:22:54.868 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:22:54.868 DEBUG:boto:path=/dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.868 DEBUG:boto:auth_path=/dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.869 DEBUG:boto:Method: GET 2024-04-12T04:22:54.869 DEBUG:boto:Path: /dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.869 DEBUG:boto:Data: 2024-04-12T04:22:54.869 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.869 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:54.869 DEBUG:boto:Port: 8001 2024-04-12T04:22:54.869 DEBUG:boto:Params: {} 2024-04-12T04:22:54.869 DEBUG:boto:Token: None 2024-04-12T04:22:54.870 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R/HYYY9pgJBnzccd6mtFt00EBes= 2024-04-12T04:22:54.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R/HYYY9pgJBnzccd6mtFt00EBes=', 'Content-Length': '0'} 2024-04-12T04:22:54.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b856b8b029967ecb-006618b71e-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:54.873 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:22:54.873 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:22:54.873 DEBUG:boto:path=/dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.874 DEBUG:boto:auth_path=/dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.874 DEBUG:boto:Method: GET 2024-04-12T04:22:54.874 DEBUG:boto:Path: /dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.874 DEBUG:boto:Data: 2024-04-12T04:22:54.874 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.874 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:54.874 DEBUG:boto:Port: 8000 2024-04-12T04:22:54.874 DEBUG:boto:Params: {} 2024-04-12T04:22:54.874 DEBUG:boto:Token: None 2024-04-12T04:22:54.874 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.875 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R/HYYY9pgJBnzccd6mtFt00EBes= 2024-04-12T04:22:54.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R/HYYY9pgJBnzccd6mtFt00EBes=', 'Content-Length': '0'} 2024-04-12T04:22:54.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000607783390ab3c4ab-006618b71e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:54.878 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:22:54.878 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:22:54.878 DEBUG:boto:path=/dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.878 DEBUG:boto:auth_path=/dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.878 DEBUG:boto:Method: HEAD 2024-04-12T04:22:54.878 DEBUG:boto:Path: /dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.879 DEBUG:boto:Data: 2024-04-12T04:22:54.879 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.879 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:54.879 DEBUG:boto:Port: 8001 2024-04-12T04:22:54.879 DEBUG:boto:Params: {} 2024-04-12T04:22:54.879 DEBUG:boto:Token: None 2024-04-12T04:22:54.879 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.879 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JBMYq1D6ysSLROTEcm+tB9bnpVU= 2024-04-12T04:22:54.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JBMYq1D6ysSLROTEcm+tB9bnpVU=', 'Content-Length': '0'} 2024-04-12T04:22:54.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015f72663d2169d16-006618b71e-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:54.883 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:22:54.883 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:22:54.883 DEBUG:boto:path=/dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.883 DEBUG:boto:auth_path=/dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.883 DEBUG:boto:Method: HEAD 2024-04-12T04:22:54.883 DEBUG:boto:Path: /dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.883 DEBUG:boto:Data: 2024-04-12T04:22:54.883 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.883 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:54.883 DEBUG:boto:Port: 8000 2024-04-12T04:22:54.883 DEBUG:boto:Params: {} 2024-04-12T04:22:54.884 DEBUG:boto:Token: None 2024-04-12T04:22:54.884 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.884 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JBMYq1D6ysSLROTEcm+tB9bnpVU= 2024-04-12T04:22:54.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JBMYq1D6ysSLROTEcm+tB9bnpVU=', 'Content-Length': '0'} 2024-04-12T04:22:54.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a18b6960893b8a49-006618b71e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:54.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T04:22:54.886 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:22:54.886 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:22:54.886 DEBUG:boto:path=/dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.886 DEBUG:boto:auth_path=/dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.887 DEBUG:boto:Method: GET 2024-04-12T04:22:54.887 DEBUG:boto:Path: /dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.887 DEBUG:boto:Data: 2024-04-12T04:22:54.887 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.887 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:54.887 DEBUG:boto:Port: 8001 2024-04-12T04:22:54.887 DEBUG:boto:Params: {} 2024-04-12T04:22:54.887 DEBUG:boto:Token: None 2024-04-12T04:22:54.887 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.887 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R/HYYY9pgJBnzccd6mtFt00EBes= 2024-04-12T04:22:54.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R/HYYY9pgJBnzccd6mtFt00EBes=', 'Content-Length': '0'} 2024-04-12T04:22:54.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd9c7330de2e2ed8-006618b71e-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:54.890 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:22:54.890 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:22:54.890 DEBUG:boto:path=/dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.891 DEBUG:boto:auth_path=/dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.891 DEBUG:boto:Method: GET 2024-04-12T04:22:54.891 DEBUG:boto:Path: /dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.891 DEBUG:boto:Data: 2024-04-12T04:22:54.891 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.891 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:54.891 DEBUG:boto:Port: 8000 2024-04-12T04:22:54.891 DEBUG:boto:Params: {} 2024-04-12T04:22:54.891 DEBUG:boto:Token: None 2024-04-12T04:22:54.891 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/dummy?versionId=null 2024-04-12T04:22:54.892 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:R/HYYY9pgJBnzccd6mtFt00EBes= 2024-04-12T04:22:54.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R/HYYY9pgJBnzccd6mtFt00EBes=', 'Content-Length': '0'} 2024-04-12T04:22:54.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e767090a9d6cbc07-006618b71e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:54.895 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:54.896 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:54.896 DEBUG:boto:path=/dkyges-64/?prefix=dummy 2024-04-12T04:22:54.896 DEBUG:boto:auth_path=/dkyges-64/?prefix=dummy 2024-04-12T04:22:54.896 DEBUG:boto:Method: GET 2024-04-12T04:22:54.896 DEBUG:boto:Path: /dkyges-64/?prefix=dummy 2024-04-12T04:22:54.896 DEBUG:boto:Data: 2024-04-12T04:22:54.896 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.896 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:54.896 DEBUG:boto:Port: 8001 2024-04-12T04:22:54.896 DEBUG:boto:Params: {} 2024-04-12T04:22:54.897 DEBUG:boto:Token: None 2024-04-12T04:22:54.897 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/ 2024-04-12T04:22:54.897 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mgDNv5zsOhSPQiDwvt9RiHFUbWY= 2024-04-12T04:22:54.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mgDNv5zsOhSPQiDwvt9RiHFUbWY=', 'Content-Length': '0'} 2024-04-12T04:22:54.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fad8a000dc8296ba-006618b71e-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:54.900 DEBUG:boto:b'dkyges-64dummy1000falsedummy2024-04-12T04:20:01.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:54.901 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:54.901 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:54.901 DEBUG:boto:path=/dkyges-64/?prefix=dummy 2024-04-12T04:22:54.901 DEBUG:boto:auth_path=/dkyges-64/?prefix=dummy 2024-04-12T04:22:54.901 DEBUG:boto:Method: GET 2024-04-12T04:22:54.901 DEBUG:boto:Path: /dkyges-64/?prefix=dummy 2024-04-12T04:22:54.901 DEBUG:boto:Data: 2024-04-12T04:22:54.901 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.901 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:54.901 DEBUG:boto:Port: 8000 2024-04-12T04:22:54.901 DEBUG:boto:Params: {} 2024-04-12T04:22:54.902 DEBUG:boto:Token: None 2024-04-12T04:22:54.902 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/ 2024-04-12T04:22:54.902 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mgDNv5zsOhSPQiDwvt9RiHFUbWY= 2024-04-12T04:22:54.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mgDNv5zsOhSPQiDwvt9RiHFUbWY=', 'Content-Length': '0'} 2024-04-12T04:22:54.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8fa72b7e11cc03d-006618b71e-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:54.951 DEBUG:boto:b'dkyges-64dummy1000falsedummy2024-04-12T04:20:01.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:54.951 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:22:54.951 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:22:54.951 DEBUG:boto:Method: HEAD 2024-04-12T04:22:54.951 DEBUG:boto:Path: /dkyges-64/dummy 2024-04-12T04:22:54.952 DEBUG:boto:Data: 2024-04-12T04:22:54.952 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.952 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:54.952 DEBUG:boto:Port: 8001 2024-04-12T04:22:54.952 DEBUG:boto:Params: {} 2024-04-12T04:22:54.952 DEBUG:boto:Token: None 2024-04-12T04:22:54.952 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/dummy 2024-04-12T04:22:54.952 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QR4UZkI+lBFcF9Ehm8KOEG0s2h8= 2024-04-12T04:22:54.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QR4UZkI+lBFcF9Ehm8KOEG0s2h8=', 'Content-Length': '0'} 2024-04-12T04:22:54.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000068f90055c3278d2-006618b71e-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:54.955 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:22:54.955 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:22:54.955 DEBUG:boto:Method: HEAD 2024-04-12T04:22:54.955 DEBUG:boto:Path: /dkyges-64/dummy 2024-04-12T04:22:54.955 DEBUG:boto:Data: 2024-04-12T04:22:54.955 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.955 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:54.955 DEBUG:boto:Port: 8000 2024-04-12T04:22:54.955 DEBUG:boto:Params: {} 2024-04-12T04:22:54.955 DEBUG:boto:Token: None 2024-04-12T04:22:54.956 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/dummy 2024-04-12T04:22:54.956 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QR4UZkI+lBFcF9Ehm8KOEG0s2h8= 2024-04-12T04:22:54.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QR4UZkI+lBFcF9Ehm8KOEG0s2h8=', 'Content-Length': '0'} 2024-04-12T04:22:54.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a493703d052b8d5-006618b71e-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:54.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2024-04-12T04:22:54.958 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:22:54.958 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:22:54.958 DEBUG:boto:Method: GET 2024-04-12T04:22:54.958 DEBUG:boto:Path: /dkyges-64/dummy 2024-04-12T04:22:54.958 DEBUG:boto:Data: 2024-04-12T04:22:54.958 DEBUG:boto:Headers: {} 2024-04-12T04:22:54.959 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:54.959 DEBUG:boto:Port: 8001 2024-04-12T04:22:54.959 DEBUG:boto:Params: {} 2024-04-12T04:22:54.959 DEBUG:boto:Token: None 2024-04-12T04:22:54.959 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:54 GMT /dkyges-64/dummy 2024-04-12T04:22:54.959 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r1tMwJSwdEj4hBaLS9rq4wkxBDY= 2024-04-12T04:22:54.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r1tMwJSwdEj4hBaLS9rq4wkxBDY=', 'Content-Length': '0'} 2024-04-12T04:22:54.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006516cba7e2d258a-006618b71e-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.003 DEBUG:boto:path=/dkyges-64/dummy 2024-04-12T04:22:55.003 DEBUG:boto:auth_path=/dkyges-64/dummy 2024-04-12T04:22:55.004 DEBUG:boto:Method: GET 2024-04-12T04:22:55.004 DEBUG:boto:Path: /dkyges-64/dummy 2024-04-12T04:22:55.004 DEBUG:boto:Data: 2024-04-12T04:22:55.004 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.004 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.004 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.004 DEBUG:boto:Params: {} 2024-04-12T04:22:55.004 DEBUG:boto:Token: None 2024-04-12T04:22:55.004 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/dummy 2024-04-12T04:22:55.005 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XUWqasHTp8sG4zhJDLw6qO3s/y8= 2024-04-12T04:22:55.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XUWqasHTp8sG4zhJDLw6qO3s/y8=', 'Content-Length': '0'} 2024-04-12T04:22:55.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:01 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057ace575d16716a3-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.007 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.007 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.007 DEBUG:boto:path=/dkyges-64/?prefix=dummy 2024-04-12T04:22:55.007 DEBUG:boto:auth_path=/dkyges-64/?prefix=dummy 2024-04-12T04:22:55.007 DEBUG:boto:Method: GET 2024-04-12T04:22:55.008 DEBUG:boto:Path: /dkyges-64/?prefix=dummy 2024-04-12T04:22:55.008 DEBUG:boto:Data: 2024-04-12T04:22:55.008 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.008 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.008 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.008 DEBUG:boto:Params: {} 2024-04-12T04:22:55.008 DEBUG:boto:Token: None 2024-04-12T04:22:55.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.008 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008afeee03217eaffb-006618b71f-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.011 DEBUG:boto:b'dkyges-64dummy1000falsedummy2024-04-12T04:20:01.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.012 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.012 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.012 DEBUG:boto:path=/dkyges-64/?prefix=dummy 2024-04-12T04:22:55.012 DEBUG:boto:auth_path=/dkyges-64/?prefix=dummy 2024-04-12T04:22:55.012 DEBUG:boto:Method: GET 2024-04-12T04:22:55.012 DEBUG:boto:Path: /dkyges-64/?prefix=dummy 2024-04-12T04:22:55.012 DEBUG:boto:Data: 2024-04-12T04:22:55.012 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.013 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.013 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.013 DEBUG:boto:Params: {} 2024-04-12T04:22:55.013 DEBUG:boto:Token: None 2024-04-12T04:22:55.013 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.013 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e964869e67f5c6da-006618b71f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.016 DEBUG:boto:b'dkyges-64dummy1000falsedummy2024-04-12T04:20:01.629Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:22:55.017 DEBUG:boto:path=/dkyges-64/obj11 2024-04-12T04:22:55.017 DEBUG:boto:auth_path=/dkyges-64/obj11 2024-04-12T04:22:55.017 DEBUG:boto:path=/dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.018 DEBUG:boto:auth_path=/dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.018 DEBUG:boto:Method: GET 2024-04-12T04:22:55.018 DEBUG:boto:Path: /dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.018 DEBUG:boto:Data: 2024-04-12T04:22:55.018 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.019 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.019 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.019 DEBUG:boto:Params: {} 2024-04-12T04:22:55.019 DEBUG:boto:Token: None 2024-04-12T04:22:55.019 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.020 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I5FC8S4E39gBcUCqdZj3KPiRIDs= 2024-04-12T04:22:55.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I5FC8S4E39gBcUCqdZj3KPiRIDs=', 'Content-Length': '0'} 2024-04-12T04:22:55.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b24b7788a7fe2b07-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.023 DEBUG:boto:path=/dkyges-64/obj11 2024-04-12T04:22:55.023 DEBUG:boto:auth_path=/dkyges-64/obj11 2024-04-12T04:22:55.023 DEBUG:boto:path=/dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.024 DEBUG:boto:auth_path=/dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.024 DEBUG:boto:Method: GET 2024-04-12T04:22:55.024 DEBUG:boto:Path: /dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.024 DEBUG:boto:Data: 2024-04-12T04:22:55.024 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.025 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.025 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.025 DEBUG:boto:Params: {} 2024-04-12T04:22:55.025 DEBUG:boto:Token: None 2024-04-12T04:22:55.025 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.025 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I5FC8S4E39gBcUCqdZj3KPiRIDs= 2024-04-12T04:22:55.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I5FC8S4E39gBcUCqdZj3KPiRIDs=', 'Content-Length': '0'} 2024-04-12T04:22:55.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ddeae9f75ba687b-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.071 DEBUG:boto:path=/dkyges-64/obj11 2024-04-12T04:22:55.071 DEBUG:boto:auth_path=/dkyges-64/obj11 2024-04-12T04:22:55.071 DEBUG:boto:path=/dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.072 DEBUG:boto:auth_path=/dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.072 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.072 DEBUG:boto:Path: /dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.072 DEBUG:boto:Data: 2024-04-12T04:22:55.072 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.072 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.072 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.072 DEBUG:boto:Params: {} 2024-04-12T04:22:55.072 DEBUG:boto:Token: None 2024-04-12T04:22:55.073 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.073 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mXScDgAvyLr2CV8ovJ4b7MaAWFg= 2024-04-12T04:22:55.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mXScDgAvyLr2CV8ovJ4b7MaAWFg=', 'Content-Length': '0'} 2024-04-12T04:22:55.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063f8905a17b976cd-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.075 DEBUG:boto:path=/dkyges-64/obj11 2024-04-12T04:22:55.075 DEBUG:boto:auth_path=/dkyges-64/obj11 2024-04-12T04:22:55.075 DEBUG:boto:path=/dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.075 DEBUG:boto:auth_path=/dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.075 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.076 DEBUG:boto:Path: /dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.076 DEBUG:boto:Data: 2024-04-12T04:22:55.076 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.076 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.076 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.076 DEBUG:boto:Params: {} 2024-04-12T04:22:55.076 DEBUG:boto:Token: None 2024-04-12T04:22:55.076 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.076 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mXScDgAvyLr2CV8ovJ4b7MaAWFg= 2024-04-12T04:22:55.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mXScDgAvyLr2CV8ovJ4b7MaAWFg=', 'Content-Length': '0'} 2024-04-12T04:22:55.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055ab8ea98d9957f7-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:22:55.079 DEBUG:boto:path=/dkyges-64/obj11 2024-04-12T04:22:55.079 DEBUG:boto:auth_path=/dkyges-64/obj11 2024-04-12T04:22:55.079 DEBUG:boto:path=/dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.079 DEBUG:boto:auth_path=/dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.080 DEBUG:boto:Method: GET 2024-04-12T04:22:55.080 DEBUG:boto:Path: /dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.080 DEBUG:boto:Data: 2024-04-12T04:22:55.080 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.080 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.080 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.080 DEBUG:boto:Params: {} 2024-04-12T04:22:55.080 DEBUG:boto:Token: None 2024-04-12T04:22:55.081 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.081 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I5FC8S4E39gBcUCqdZj3KPiRIDs= 2024-04-12T04:22:55.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I5FC8S4E39gBcUCqdZj3KPiRIDs=', 'Content-Length': '0'} 2024-04-12T04:22:55.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014bbc80515a2b75c-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.130 DEBUG:boto:path=/dkyges-64/obj11 2024-04-12T04:22:55.131 DEBUG:boto:auth_path=/dkyges-64/obj11 2024-04-12T04:22:55.131 DEBUG:boto:path=/dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.131 DEBUG:boto:auth_path=/dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.131 DEBUG:boto:Method: GET 2024-04-12T04:22:55.131 DEBUG:boto:Path: /dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.131 DEBUG:boto:Data: 2024-04-12T04:22:55.131 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.131 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.131 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.132 DEBUG:boto:Params: {} 2024-04-12T04:22:55.132 DEBUG:boto:Token: None 2024-04-12T04:22:55.132 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj11?versionId=null 2024-04-12T04:22:55.132 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I5FC8S4E39gBcUCqdZj3KPiRIDs= 2024-04-12T04:22:55.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I5FC8S4E39gBcUCqdZj3KPiRIDs=', 'Content-Length': '0'} 2024-04-12T04:22:55.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005643ab4b90388567-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.134 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.135 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.135 DEBUG:boto:path=/dkyges-64/?prefix=obj11 2024-04-12T04:22:55.135 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj11 2024-04-12T04:22:55.135 DEBUG:boto:Method: GET 2024-04-12T04:22:55.135 DEBUG:boto:Path: /dkyges-64/?prefix=obj11 2024-04-12T04:22:55.135 DEBUG:boto:Data: 2024-04-12T04:22:55.135 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.135 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.135 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.135 DEBUG:boto:Params: {} 2024-04-12T04:22:55.136 DEBUG:boto:Token: None 2024-04-12T04:22:55.136 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.136 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.138 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a448fc8fd65925ae-006618b71f-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.139 DEBUG:boto:b'dkyges-64obj111000falseobj112024-04-12T04:20:43.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.139 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.139 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.139 DEBUG:boto:path=/dkyges-64/?prefix=obj11 2024-04-12T04:22:55.139 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj11 2024-04-12T04:22:55.140 DEBUG:boto:Method: GET 2024-04-12T04:22:55.140 DEBUG:boto:Path: /dkyges-64/?prefix=obj11 2024-04-12T04:22:55.140 DEBUG:boto:Data: 2024-04-12T04:22:55.140 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.140 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.140 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.140 DEBUG:boto:Params: {} 2024-04-12T04:22:55.140 DEBUG:boto:Token: None 2024-04-12T04:22:55.140 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.140 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011e7f963e1ac96ec-006618b71f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.144 DEBUG:boto:b'dkyges-64obj111000falseobj112024-04-12T04:20:43.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.144 DEBUG:boto:path=/dkyges-64/obj11 2024-04-12T04:22:55.144 DEBUG:boto:auth_path=/dkyges-64/obj11 2024-04-12T04:22:55.144 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.144 DEBUG:boto:Path: /dkyges-64/obj11 2024-04-12T04:22:55.144 DEBUG:boto:Data: 2024-04-12T04:22:55.144 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.144 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.145 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.145 DEBUG:boto:Params: {} 2024-04-12T04:22:55.145 DEBUG:boto:Token: None 2024-04-12T04:22:55.145 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj11 2024-04-12T04:22:55.145 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V71rei8zc0441mk74Yd2bEhp2M4= 2024-04-12T04:22:55.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V71rei8zc0441mk74Yd2bEhp2M4=', 'Content-Length': '0'} 2024-04-12T04:22:55.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7a1f918374cf841-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.147 DEBUG:boto:path=/dkyges-64/obj11 2024-04-12T04:22:55.147 DEBUG:boto:auth_path=/dkyges-64/obj11 2024-04-12T04:22:55.147 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.147 DEBUG:boto:Path: /dkyges-64/obj11 2024-04-12T04:22:55.147 DEBUG:boto:Data: 2024-04-12T04:22:55.147 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.148 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.148 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.148 DEBUG:boto:Params: {} 2024-04-12T04:22:55.148 DEBUG:boto:Token: None 2024-04-12T04:22:55.148 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj11 2024-04-12T04:22:55.148 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:V71rei8zc0441mk74Yd2bEhp2M4= 2024-04-12T04:22:55.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V71rei8zc0441mk74Yd2bEhp2M4=', 'Content-Length': '0'} 2024-04-12T04:22:55.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049783c0543793a66-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:22:55.150 DEBUG:boto:path=/dkyges-64/obj11 2024-04-12T04:22:55.151 DEBUG:boto:auth_path=/dkyges-64/obj11 2024-04-12T04:22:55.151 DEBUG:boto:Method: GET 2024-04-12T04:22:55.151 DEBUG:boto:Path: /dkyges-64/obj11 2024-04-12T04:22:55.151 DEBUG:boto:Data: 2024-04-12T04:22:55.151 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.151 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.151 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.151 DEBUG:boto:Params: {} 2024-04-12T04:22:55.151 DEBUG:boto:Token: None 2024-04-12T04:22:55.151 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj11 2024-04-12T04:22:55.152 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mmFG9QU8M0C2INk9RaD2dCVcyuY= 2024-04-12T04:22:55.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mmFG9QU8M0C2INk9RaD2dCVcyuY=', 'Content-Length': '0'} 2024-04-12T04:22:55.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f007e1ad6adfee8-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.198 DEBUG:boto:path=/dkyges-64/obj11 2024-04-12T04:22:55.199 DEBUG:boto:auth_path=/dkyges-64/obj11 2024-04-12T04:22:55.199 DEBUG:boto:Method: GET 2024-04-12T04:22:55.199 DEBUG:boto:Path: /dkyges-64/obj11 2024-04-12T04:22:55.199 DEBUG:boto:Data: 2024-04-12T04:22:55.199 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.199 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.199 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.199 DEBUG:boto:Params: {} 2024-04-12T04:22:55.199 DEBUG:boto:Token: None 2024-04-12T04:22:55.200 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj11 2024-04-12T04:22:55.200 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mmFG9QU8M0C2INk9RaD2dCVcyuY= 2024-04-12T04:22:55.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mmFG9QU8M0C2INk9RaD2dCVcyuY=', 'Content-Length': '0'} 2024-04-12T04:22:55.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1a38c4c2a14c914-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.202 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.202 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.202 DEBUG:boto:path=/dkyges-64/?prefix=obj11 2024-04-12T04:22:55.203 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj11 2024-04-12T04:22:55.203 DEBUG:boto:Method: GET 2024-04-12T04:22:55.203 DEBUG:boto:Path: /dkyges-64/?prefix=obj11 2024-04-12T04:22:55.203 DEBUG:boto:Data: 2024-04-12T04:22:55.203 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.203 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.203 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.203 DEBUG:boto:Params: {} 2024-04-12T04:22:55.203 DEBUG:boto:Token: None 2024-04-12T04:22:55.203 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.203 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.206 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e6f6a528f9df855-006618b71f-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.206 DEBUG:boto:b'dkyges-64obj111000falseobj112024-04-12T04:20:43.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.207 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.207 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.207 DEBUG:boto:path=/dkyges-64/?prefix=obj11 2024-04-12T04:22:55.207 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj11 2024-04-12T04:22:55.207 DEBUG:boto:Method: GET 2024-04-12T04:22:55.207 DEBUG:boto:Path: /dkyges-64/?prefix=obj11 2024-04-12T04:22:55.207 DEBUG:boto:Data: 2024-04-12T04:22:55.207 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.207 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.207 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.207 DEBUG:boto:Params: {} 2024-04-12T04:22:55.208 DEBUG:boto:Token: None 2024-04-12T04:22:55.208 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000457c7f5acb296d87-006618b71f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.212 DEBUG:boto:b'dkyges-64obj111000falseobj112024-04-12T04:20:43.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:22:55.212 DEBUG:boto:path=/dkyges-64/obj22 2024-04-12T04:22:55.212 DEBUG:boto:auth_path=/dkyges-64/obj22 2024-04-12T04:22:55.213 DEBUG:boto:path=/dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.213 DEBUG:boto:auth_path=/dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.213 DEBUG:boto:Method: GET 2024-04-12T04:22:55.213 DEBUG:boto:Path: /dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.213 DEBUG:boto:Data: 2024-04-12T04:22:55.213 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.213 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.213 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.213 DEBUG:boto:Params: {} 2024-04-12T04:22:55.213 DEBUG:boto:Token: None 2024-04-12T04:22:55.213 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.214 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S4a7WaTs6XcLCDDTG/246qDb770= 2024-04-12T04:22:55.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S4a7WaTs6XcLCDDTG/246qDb770=', 'Content-Length': '0'} 2024-04-12T04:22:55.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000acf8af3541b1ed-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.216 DEBUG:boto:path=/dkyges-64/obj22 2024-04-12T04:22:55.216 DEBUG:boto:auth_path=/dkyges-64/obj22 2024-04-12T04:22:55.216 DEBUG:boto:path=/dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.216 DEBUG:boto:auth_path=/dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.216 DEBUG:boto:Method: GET 2024-04-12T04:22:55.216 DEBUG:boto:Path: /dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.216 DEBUG:boto:Data: 2024-04-12T04:22:55.216 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.217 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.217 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.217 DEBUG:boto:Params: {} 2024-04-12T04:22:55.217 DEBUG:boto:Token: None 2024-04-12T04:22:55.217 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.217 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S4a7WaTs6XcLCDDTG/246qDb770= 2024-04-12T04:22:55.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S4a7WaTs6XcLCDDTG/246qDb770=', 'Content-Length': '0'} 2024-04-12T04:22:55.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000008cdd583e001e6-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.220 DEBUG:boto:path=/dkyges-64/obj22 2024-04-12T04:22:55.220 DEBUG:boto:auth_path=/dkyges-64/obj22 2024-04-12T04:22:55.220 DEBUG:boto:path=/dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.220 DEBUG:boto:auth_path=/dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.220 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.221 DEBUG:boto:Path: /dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.221 DEBUG:boto:Data: 2024-04-12T04:22:55.221 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.221 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.221 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.221 DEBUG:boto:Params: {} 2024-04-12T04:22:55.221 DEBUG:boto:Token: None 2024-04-12T04:22:55.221 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.221 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:92LtLtpsqvptP8v89yTprnyUin0= 2024-04-12T04:22:55.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:92LtLtpsqvptP8v89yTprnyUin0=', 'Content-Length': '0'} 2024-04-12T04:22:55.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e5ac7551a407b21-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.224 DEBUG:boto:path=/dkyges-64/obj22 2024-04-12T04:22:55.224 DEBUG:boto:auth_path=/dkyges-64/obj22 2024-04-12T04:22:55.224 DEBUG:boto:path=/dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.224 DEBUG:boto:auth_path=/dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.224 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.224 DEBUG:boto:Path: /dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.224 DEBUG:boto:Data: 2024-04-12T04:22:55.224 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.224 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.225 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.225 DEBUG:boto:Params: {} 2024-04-12T04:22:55.225 DEBUG:boto:Token: None 2024-04-12T04:22:55.225 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.225 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:92LtLtpsqvptP8v89yTprnyUin0= 2024-04-12T04:22:55.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:92LtLtpsqvptP8v89yTprnyUin0=', 'Content-Length': '0'} 2024-04-12T04:22:55.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050288e86eca9a0d7-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:22:55.228 DEBUG:boto:path=/dkyges-64/obj22 2024-04-12T04:22:55.228 DEBUG:boto:auth_path=/dkyges-64/obj22 2024-04-12T04:22:55.228 DEBUG:boto:path=/dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.228 DEBUG:boto:auth_path=/dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.228 DEBUG:boto:Method: GET 2024-04-12T04:22:55.228 DEBUG:boto:Path: /dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.228 DEBUG:boto:Data: 2024-04-12T04:22:55.229 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.229 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.229 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.229 DEBUG:boto:Params: {} 2024-04-12T04:22:55.229 DEBUG:boto:Token: None 2024-04-12T04:22:55.229 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.229 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S4a7WaTs6XcLCDDTG/246qDb770= 2024-04-12T04:22:55.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S4a7WaTs6XcLCDDTG/246qDb770=', 'Content-Length': '0'} 2024-04-12T04:22:55.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000303f981cfd39842b-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.274 DEBUG:boto:path=/dkyges-64/obj22 2024-04-12T04:22:55.275 DEBUG:boto:auth_path=/dkyges-64/obj22 2024-04-12T04:22:55.275 DEBUG:boto:path=/dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.275 DEBUG:boto:auth_path=/dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.275 DEBUG:boto:Method: GET 2024-04-12T04:22:55.275 DEBUG:boto:Path: /dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.275 DEBUG:boto:Data: 2024-04-12T04:22:55.275 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.275 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.275 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.275 DEBUG:boto:Params: {} 2024-04-12T04:22:55.276 DEBUG:boto:Token: None 2024-04-12T04:22:55.276 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj22?versionId=null 2024-04-12T04:22:55.276 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:S4a7WaTs6XcLCDDTG/246qDb770= 2024-04-12T04:22:55.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:S4a7WaTs6XcLCDDTG/246qDb770=', 'Content-Length': '0'} 2024-04-12T04:22:55.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d84d39d08aebd620-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.278 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.279 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.279 DEBUG:boto:path=/dkyges-64/?prefix=obj22 2024-04-12T04:22:55.279 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj22 2024-04-12T04:22:55.279 DEBUG:boto:Method: GET 2024-04-12T04:22:55.279 DEBUG:boto:Path: /dkyges-64/?prefix=obj22 2024-04-12T04:22:55.279 DEBUG:boto:Data: 2024-04-12T04:22:55.279 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.279 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.279 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.279 DEBUG:boto:Params: {} 2024-04-12T04:22:55.280 DEBUG:boto:Token: None 2024-04-12T04:22:55.280 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.280 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfc7cda1cdfebc72-006618b71f-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.282 DEBUG:boto:b'dkyges-64obj221000falseobj222024-04-12T04:20:43.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.283 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.283 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.283 DEBUG:boto:path=/dkyges-64/?prefix=obj22 2024-04-12T04:22:55.283 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj22 2024-04-12T04:22:55.283 DEBUG:boto:Method: GET 2024-04-12T04:22:55.283 DEBUG:boto:Path: /dkyges-64/?prefix=obj22 2024-04-12T04:22:55.284 DEBUG:boto:Data: 2024-04-12T04:22:55.284 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.284 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.284 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.284 DEBUG:boto:Params: {} 2024-04-12T04:22:55.284 DEBUG:boto:Token: None 2024-04-12T04:22:55.284 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.284 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab72106a6d61fb55-006618b71f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.330 DEBUG:boto:b'dkyges-64obj221000falseobj222024-04-12T04:20:43.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.331 DEBUG:boto:path=/dkyges-64/obj22 2024-04-12T04:22:55.331 DEBUG:boto:auth_path=/dkyges-64/obj22 2024-04-12T04:22:55.331 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.331 DEBUG:boto:Path: /dkyges-64/obj22 2024-04-12T04:22:55.331 DEBUG:boto:Data: 2024-04-12T04:22:55.331 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.331 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.332 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.332 DEBUG:boto:Params: {} 2024-04-12T04:22:55.332 DEBUG:boto:Token: None 2024-04-12T04:22:55.332 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj22 2024-04-12T04:22:55.332 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JWHT3U0VpbYgjG9wtdcBzD8L3jc= 2024-04-12T04:22:55.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JWHT3U0VpbYgjG9wtdcBzD8L3jc=', 'Content-Length': '0'} 2024-04-12T04:22:55.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d82658ea1a98dfd7-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.334 DEBUG:boto:path=/dkyges-64/obj22 2024-04-12T04:22:55.335 DEBUG:boto:auth_path=/dkyges-64/obj22 2024-04-12T04:22:55.335 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.335 DEBUG:boto:Path: /dkyges-64/obj22 2024-04-12T04:22:55.335 DEBUG:boto:Data: 2024-04-12T04:22:55.335 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.335 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.335 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.335 DEBUG:boto:Params: {} 2024-04-12T04:22:55.335 DEBUG:boto:Token: None 2024-04-12T04:22:55.335 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj22 2024-04-12T04:22:55.336 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JWHT3U0VpbYgjG9wtdcBzD8L3jc= 2024-04-12T04:22:55.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JWHT3U0VpbYgjG9wtdcBzD8L3jc=', 'Content-Length': '0'} 2024-04-12T04:22:55.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006551f34d39326b6d-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:22:55.338 DEBUG:boto:path=/dkyges-64/obj22 2024-04-12T04:22:55.338 DEBUG:boto:auth_path=/dkyges-64/obj22 2024-04-12T04:22:55.338 DEBUG:boto:Method: GET 2024-04-12T04:22:55.338 DEBUG:boto:Path: /dkyges-64/obj22 2024-04-12T04:22:55.338 DEBUG:boto:Data: 2024-04-12T04:22:55.338 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.339 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.339 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.339 DEBUG:boto:Params: {} 2024-04-12T04:22:55.339 DEBUG:boto:Token: None 2024-04-12T04:22:55.339 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj22 2024-04-12T04:22:55.339 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fNpKLfW8H7/Jbbaxp3nNRD4KipM= 2024-04-12T04:22:55.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fNpKLfW8H7/Jbbaxp3nNRD4KipM=', 'Content-Length': '0'} 2024-04-12T04:22:55.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e924b2332bd43c5-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.341 DEBUG:boto:path=/dkyges-64/obj22 2024-04-12T04:22:55.341 DEBUG:boto:auth_path=/dkyges-64/obj22 2024-04-12T04:22:55.341 DEBUG:boto:Method: GET 2024-04-12T04:22:55.342 DEBUG:boto:Path: /dkyges-64/obj22 2024-04-12T04:22:55.342 DEBUG:boto:Data: 2024-04-12T04:22:55.342 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.342 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.342 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.342 DEBUG:boto:Params: {} 2024-04-12T04:22:55.342 DEBUG:boto:Token: None 2024-04-12T04:22:55.342 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj22 2024-04-12T04:22:55.342 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fNpKLfW8H7/Jbbaxp3nNRD4KipM= 2024-04-12T04:22:55.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fNpKLfW8H7/Jbbaxp3nNRD4KipM=', 'Content-Length': '0'} 2024-04-12T04:22:55.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003d75037ca49dc21-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.345 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.345 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.345 DEBUG:boto:path=/dkyges-64/?prefix=obj22 2024-04-12T04:22:55.345 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj22 2024-04-12T04:22:55.345 DEBUG:boto:Method: GET 2024-04-12T04:22:55.345 DEBUG:boto:Path: /dkyges-64/?prefix=obj22 2024-04-12T04:22:55.345 DEBUG:boto:Data: 2024-04-12T04:22:55.345 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.345 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.345 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.345 DEBUG:boto:Params: {} 2024-04-12T04:22:55.346 DEBUG:boto:Token: None 2024-04-12T04:22:55.346 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.346 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee2b5c0788a4371a-006618b71f-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.348 DEBUG:boto:b'dkyges-64obj221000falseobj222024-04-12T04:20:43.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.349 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.349 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.349 DEBUG:boto:path=/dkyges-64/?prefix=obj22 2024-04-12T04:22:55.349 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj22 2024-04-12T04:22:55.349 DEBUG:boto:Method: GET 2024-04-12T04:22:55.349 DEBUG:boto:Path: /dkyges-64/?prefix=obj22 2024-04-12T04:22:55.349 DEBUG:boto:Data: 2024-04-12T04:22:55.349 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.350 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.350 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.350 DEBUG:boto:Params: {} 2024-04-12T04:22:55.350 DEBUG:boto:Token: None 2024-04-12T04:22:55.350 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.350 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fea29e4ed209d2b-006618b71f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.353 DEBUG:boto:b'dkyges-64obj221000falseobj222024-04-12T04:20:43.177Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:22:55.353 DEBUG:boto:path=/dkyges-64/obj33 2024-04-12T04:22:55.354 DEBUG:boto:auth_path=/dkyges-64/obj33 2024-04-12T04:22:55.354 DEBUG:boto:path=/dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.354 DEBUG:boto:auth_path=/dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.354 DEBUG:boto:Method: GET 2024-04-12T04:22:55.354 DEBUG:boto:Path: /dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.354 DEBUG:boto:Data: 2024-04-12T04:22:55.354 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.354 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.354 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.354 DEBUG:boto:Params: {} 2024-04-12T04:22:55.355 DEBUG:boto:Token: None 2024-04-12T04:22:55.355 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.355 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7dxoDDfFFVaFB3ozhe93cYdCyDw= 2024-04-12T04:22:55.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7dxoDDfFFVaFB3ozhe93cYdCyDw=', 'Content-Length': '0'} 2024-04-12T04:22:55.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cd54a147a9253d6-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.357 DEBUG:boto:path=/dkyges-64/obj33 2024-04-12T04:22:55.357 DEBUG:boto:auth_path=/dkyges-64/obj33 2024-04-12T04:22:55.357 DEBUG:boto:path=/dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.357 DEBUG:boto:auth_path=/dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.358 DEBUG:boto:Method: GET 2024-04-12T04:22:55.358 DEBUG:boto:Path: /dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.358 DEBUG:boto:Data: 2024-04-12T04:22:55.358 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.358 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.358 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.358 DEBUG:boto:Params: {} 2024-04-12T04:22:55.358 DEBUG:boto:Token: None 2024-04-12T04:22:55.358 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.358 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7dxoDDfFFVaFB3ozhe93cYdCyDw= 2024-04-12T04:22:55.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7dxoDDfFFVaFB3ozhe93cYdCyDw=', 'Content-Length': '0'} 2024-04-12T04:22:55.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000186b8c3497d5bf5f-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.361 DEBUG:boto:path=/dkyges-64/obj33 2024-04-12T04:22:55.361 DEBUG:boto:auth_path=/dkyges-64/obj33 2024-04-12T04:22:55.361 DEBUG:boto:path=/dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.361 DEBUG:boto:auth_path=/dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.361 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.361 DEBUG:boto:Path: /dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.361 DEBUG:boto:Data: 2024-04-12T04:22:55.361 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.361 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.362 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.362 DEBUG:boto:Params: {} 2024-04-12T04:22:55.362 DEBUG:boto:Token: None 2024-04-12T04:22:55.362 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.362 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P5N60COq9Z8CjgEJ2jqtAdyG1IE= 2024-04-12T04:22:55.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P5N60COq9Z8CjgEJ2jqtAdyG1IE=', 'Content-Length': '0'} 2024-04-12T04:22:55.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000308a408646fc2f77-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.364 DEBUG:boto:path=/dkyges-64/obj33 2024-04-12T04:22:55.365 DEBUG:boto:auth_path=/dkyges-64/obj33 2024-04-12T04:22:55.365 DEBUG:boto:path=/dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.365 DEBUG:boto:auth_path=/dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.365 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.365 DEBUG:boto:Path: /dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.365 DEBUG:boto:Data: 2024-04-12T04:22:55.365 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.365 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.365 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.365 DEBUG:boto:Params: {} 2024-04-12T04:22:55.365 DEBUG:boto:Token: None 2024-04-12T04:22:55.366 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.366 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:P5N60COq9Z8CjgEJ2jqtAdyG1IE= 2024-04-12T04:22:55.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P5N60COq9Z8CjgEJ2jqtAdyG1IE=', 'Content-Length': '0'} 2024-04-12T04:22:55.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027b68cd7e4d7ea78-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:22:55.368 DEBUG:boto:path=/dkyges-64/obj33 2024-04-12T04:22:55.368 DEBUG:boto:auth_path=/dkyges-64/obj33 2024-04-12T04:22:55.368 DEBUG:boto:path=/dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.368 DEBUG:boto:auth_path=/dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.368 DEBUG:boto:Method: GET 2024-04-12T04:22:55.368 DEBUG:boto:Path: /dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.368 DEBUG:boto:Data: 2024-04-12T04:22:55.369 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.369 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.369 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.369 DEBUG:boto:Params: {} 2024-04-12T04:22:55.369 DEBUG:boto:Token: None 2024-04-12T04:22:55.369 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.369 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7dxoDDfFFVaFB3ozhe93cYdCyDw= 2024-04-12T04:22:55.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7dxoDDfFFVaFB3ozhe93cYdCyDw=', 'Content-Length': '0'} 2024-04-12T04:22:55.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2e80481f1629847-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.371 DEBUG:boto:path=/dkyges-64/obj33 2024-04-12T04:22:55.372 DEBUG:boto:auth_path=/dkyges-64/obj33 2024-04-12T04:22:55.372 DEBUG:boto:path=/dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.372 DEBUG:boto:auth_path=/dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.372 DEBUG:boto:Method: GET 2024-04-12T04:22:55.372 DEBUG:boto:Path: /dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.372 DEBUG:boto:Data: 2024-04-12T04:22:55.372 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.372 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.372 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.372 DEBUG:boto:Params: {} 2024-04-12T04:22:55.372 DEBUG:boto:Token: None 2024-04-12T04:22:55.373 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj33?versionId=null 2024-04-12T04:22:55.373 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7dxoDDfFFVaFB3ozhe93cYdCyDw= 2024-04-12T04:22:55.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7dxoDDfFFVaFB3ozhe93cYdCyDw=', 'Content-Length': '0'} 2024-04-12T04:22:55.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c69bf2683622a645-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.375 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.375 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.375 DEBUG:boto:path=/dkyges-64/?prefix=obj33 2024-04-12T04:22:55.375 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj33 2024-04-12T04:22:55.375 DEBUG:boto:Method: GET 2024-04-12T04:22:55.375 DEBUG:boto:Path: /dkyges-64/?prefix=obj33 2024-04-12T04:22:55.376 DEBUG:boto:Data: 2024-04-12T04:22:55.376 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.376 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.376 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.376 DEBUG:boto:Params: {} 2024-04-12T04:22:55.376 DEBUG:boto:Token: None 2024-04-12T04:22:55.376 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.376 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.379 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b4f1b86bff622d0-006618b71f-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.379 DEBUG:boto:b'dkyges-64obj331000falseobj332024-04-12T04:20:43.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.379 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.380 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.380 DEBUG:boto:path=/dkyges-64/?prefix=obj33 2024-04-12T04:22:55.380 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj33 2024-04-12T04:22:55.380 DEBUG:boto:Method: GET 2024-04-12T04:22:55.380 DEBUG:boto:Path: /dkyges-64/?prefix=obj33 2024-04-12T04:22:55.380 DEBUG:boto:Data: 2024-04-12T04:22:55.380 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.380 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.380 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.380 DEBUG:boto:Params: {} 2024-04-12T04:22:55.380 DEBUG:boto:Token: None 2024-04-12T04:22:55.381 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.381 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.383 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093eb64ae3692c936-006618b71f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.383 DEBUG:boto:b'dkyges-64obj331000falseobj332024-04-12T04:20:43.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.384 DEBUG:boto:path=/dkyges-64/obj33 2024-04-12T04:22:55.384 DEBUG:boto:auth_path=/dkyges-64/obj33 2024-04-12T04:22:55.384 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.384 DEBUG:boto:Path: /dkyges-64/obj33 2024-04-12T04:22:55.384 DEBUG:boto:Data: 2024-04-12T04:22:55.384 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.384 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.384 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.385 DEBUG:boto:Params: {} 2024-04-12T04:22:55.385 DEBUG:boto:Token: None 2024-04-12T04:22:55.385 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj33 2024-04-12T04:22:55.385 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5joViPHYPw8fffHkbj2vPybdjk4= 2024-04-12T04:22:55.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5joViPHYPw8fffHkbj2vPybdjk4=', 'Content-Length': '0'} 2024-04-12T04:22:55.387 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea9953c7c63510f0-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.387 DEBUG:boto:path=/dkyges-64/obj33 2024-04-12T04:22:55.387 DEBUG:boto:auth_path=/dkyges-64/obj33 2024-04-12T04:22:55.388 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.388 DEBUG:boto:Path: /dkyges-64/obj33 2024-04-12T04:22:55.388 DEBUG:boto:Data: 2024-04-12T04:22:55.388 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.388 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.388 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.388 DEBUG:boto:Params: {} 2024-04-12T04:22:55.388 DEBUG:boto:Token: None 2024-04-12T04:22:55.388 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj33 2024-04-12T04:22:55.388 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5joViPHYPw8fffHkbj2vPybdjk4= 2024-04-12T04:22:55.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5joViPHYPw8fffHkbj2vPybdjk4=', 'Content-Length': '0'} 2024-04-12T04:22:55.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdba8c4acdb9be7b-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:22:55.391 DEBUG:boto:path=/dkyges-64/obj33 2024-04-12T04:22:55.391 DEBUG:boto:auth_path=/dkyges-64/obj33 2024-04-12T04:22:55.391 DEBUG:boto:Method: GET 2024-04-12T04:22:55.391 DEBUG:boto:Path: /dkyges-64/obj33 2024-04-12T04:22:55.391 DEBUG:boto:Data: 2024-04-12T04:22:55.391 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.391 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.391 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.392 DEBUG:boto:Params: {} 2024-04-12T04:22:55.392 DEBUG:boto:Token: None 2024-04-12T04:22:55.392 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj33 2024-04-12T04:22:55.392 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZHxdaINEvCsaUw3g7VGDocRkgzg= 2024-04-12T04:22:55.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZHxdaINEvCsaUw3g7VGDocRkgzg=', 'Content-Length': '0'} 2024-04-12T04:22:55.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a092d3a78cdc7d4-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.434 DEBUG:boto:path=/dkyges-64/obj33 2024-04-12T04:22:55.435 DEBUG:boto:auth_path=/dkyges-64/obj33 2024-04-12T04:22:55.435 DEBUG:boto:Method: GET 2024-04-12T04:22:55.435 DEBUG:boto:Path: /dkyges-64/obj33 2024-04-12T04:22:55.435 DEBUG:boto:Data: 2024-04-12T04:22:55.435 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.435 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.435 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.435 DEBUG:boto:Params: {} 2024-04-12T04:22:55.435 DEBUG:boto:Token: None 2024-04-12T04:22:55.436 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj33 2024-04-12T04:22:55.436 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZHxdaINEvCsaUw3g7VGDocRkgzg= 2024-04-12T04:22:55.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZHxdaINEvCsaUw3g7VGDocRkgzg=', 'Content-Length': '0'} 2024-04-12T04:22:55.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001dcdf6b002c5bf18-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.438 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.438 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.438 DEBUG:boto:path=/dkyges-64/?prefix=obj33 2024-04-12T04:22:55.438 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj33 2024-04-12T04:22:55.439 DEBUG:boto:Method: GET 2024-04-12T04:22:55.439 DEBUG:boto:Path: /dkyges-64/?prefix=obj33 2024-04-12T04:22:55.439 DEBUG:boto:Data: 2024-04-12T04:22:55.439 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.439 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.439 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.439 DEBUG:boto:Params: {} 2024-04-12T04:22:55.439 DEBUG:boto:Token: None 2024-04-12T04:22:55.439 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.439 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000972181ddd5268abc-006618b71f-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.442 DEBUG:boto:b'dkyges-64obj331000falseobj332024-04-12T04:20:43.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.443 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.443 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.443 DEBUG:boto:path=/dkyges-64/?prefix=obj33 2024-04-12T04:22:55.443 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj33 2024-04-12T04:22:55.443 DEBUG:boto:Method: GET 2024-04-12T04:22:55.443 DEBUG:boto:Path: /dkyges-64/?prefix=obj33 2024-04-12T04:22:55.443 DEBUG:boto:Data: 2024-04-12T04:22:55.443 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.443 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.444 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.444 DEBUG:boto:Params: {} 2024-04-12T04:22:55.444 DEBUG:boto:Token: None 2024-04-12T04:22:55.444 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.444 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8e726441b699cb0-006618b71f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.447 DEBUG:boto:b'dkyges-64obj331000falseobj332024-04-12T04:20:43.226Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:22:55.447 DEBUG:boto:path=/dkyges-64/obj44 2024-04-12T04:22:55.447 DEBUG:boto:auth_path=/dkyges-64/obj44 2024-04-12T04:22:55.447 DEBUG:boto:path=/dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.448 DEBUG:boto:auth_path=/dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.448 DEBUG:boto:Method: GET 2024-04-12T04:22:55.448 DEBUG:boto:Path: /dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.448 DEBUG:boto:Data: 2024-04-12T04:22:55.448 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.448 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.448 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.448 DEBUG:boto:Params: {} 2024-04-12T04:22:55.448 DEBUG:boto:Token: None 2024-04-12T04:22:55.448 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.449 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jMMCIWMbYQ+wt+qy+W9ixCZxVqg= 2024-04-12T04:22:55.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jMMCIWMbYQ+wt+qy+W9ixCZxVqg=', 'Content-Length': '0'} 2024-04-12T04:22:55.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fb49f32d7ade262-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.451 DEBUG:boto:path=/dkyges-64/obj44 2024-04-12T04:22:55.452 DEBUG:boto:auth_path=/dkyges-64/obj44 2024-04-12T04:22:55.452 DEBUG:boto:path=/dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.452 DEBUG:boto:auth_path=/dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.452 DEBUG:boto:Method: GET 2024-04-12T04:22:55.453 DEBUG:boto:Path: /dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.453 DEBUG:boto:Data: 2024-04-12T04:22:55.453 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.453 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.453 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.453 DEBUG:boto:Params: {} 2024-04-12T04:22:55.454 DEBUG:boto:Token: None 2024-04-12T04:22:55.454 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.454 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jMMCIWMbYQ+wt+qy+W9ixCZxVqg= 2024-04-12T04:22:55.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jMMCIWMbYQ+wt+qy+W9ixCZxVqg=', 'Content-Length': '0'} 2024-04-12T04:22:55.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd09c9d3671f04da-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.456 DEBUG:boto:path=/dkyges-64/obj44 2024-04-12T04:22:55.457 DEBUG:boto:auth_path=/dkyges-64/obj44 2024-04-12T04:22:55.457 DEBUG:boto:path=/dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.457 DEBUG:boto:auth_path=/dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.457 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.457 DEBUG:boto:Path: /dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.457 DEBUG:boto:Data: 2024-04-12T04:22:55.458 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.458 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.458 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.458 DEBUG:boto:Params: {} 2024-04-12T04:22:55.458 DEBUG:boto:Token: None 2024-04-12T04:22:55.458 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.458 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G07AzxyoBiylyexmtJbcHfcQ4r0= 2024-04-12T04:22:55.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G07AzxyoBiylyexmtJbcHfcQ4r0=', 'Content-Length': '0'} 2024-04-12T04:22:55.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce9e97ac617881ca-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.461 DEBUG:boto:path=/dkyges-64/obj44 2024-04-12T04:22:55.461 DEBUG:boto:auth_path=/dkyges-64/obj44 2024-04-12T04:22:55.461 DEBUG:boto:path=/dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.461 DEBUG:boto:auth_path=/dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.461 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.461 DEBUG:boto:Path: /dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.461 DEBUG:boto:Data: 2024-04-12T04:22:55.461 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.461 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.461 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.461 DEBUG:boto:Params: {} 2024-04-12T04:22:55.462 DEBUG:boto:Token: None 2024-04-12T04:22:55.462 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.462 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:G07AzxyoBiylyexmtJbcHfcQ4r0= 2024-04-12T04:22:55.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G07AzxyoBiylyexmtJbcHfcQ4r0=', 'Content-Length': '0'} 2024-04-12T04:22:55.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041e3531fdf406fc8-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:22:55.464 DEBUG:boto:path=/dkyges-64/obj44 2024-04-12T04:22:55.464 DEBUG:boto:auth_path=/dkyges-64/obj44 2024-04-12T04:22:55.464 DEBUG:boto:path=/dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.464 DEBUG:boto:auth_path=/dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.464 DEBUG:boto:Method: GET 2024-04-12T04:22:55.465 DEBUG:boto:Path: /dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.465 DEBUG:boto:Data: 2024-04-12T04:22:55.465 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.465 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.465 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.465 DEBUG:boto:Params: {} 2024-04-12T04:22:55.465 DEBUG:boto:Token: None 2024-04-12T04:22:55.465 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.465 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jMMCIWMbYQ+wt+qy+W9ixCZxVqg= 2024-04-12T04:22:55.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jMMCIWMbYQ+wt+qy+W9ixCZxVqg=', 'Content-Length': '0'} 2024-04-12T04:22:55.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c75a1719b9e016d7-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.510 DEBUG:boto:path=/dkyges-64/obj44 2024-04-12T04:22:55.511 DEBUG:boto:auth_path=/dkyges-64/obj44 2024-04-12T04:22:55.511 DEBUG:boto:path=/dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.511 DEBUG:boto:auth_path=/dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.511 DEBUG:boto:Method: GET 2024-04-12T04:22:55.511 DEBUG:boto:Path: /dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.511 DEBUG:boto:Data: 2024-04-12T04:22:55.511 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.511 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.511 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.511 DEBUG:boto:Params: {} 2024-04-12T04:22:55.512 DEBUG:boto:Token: None 2024-04-12T04:22:55.512 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj44?versionId=null 2024-04-12T04:22:55.512 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jMMCIWMbYQ+wt+qy+W9ixCZxVqg= 2024-04-12T04:22:55.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jMMCIWMbYQ+wt+qy+W9ixCZxVqg=', 'Content-Length': '0'} 2024-04-12T04:22:55.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2252a10634812b9-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.514 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.515 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.515 DEBUG:boto:path=/dkyges-64/?prefix=obj44 2024-04-12T04:22:55.515 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj44 2024-04-12T04:22:55.515 DEBUG:boto:Method: GET 2024-04-12T04:22:55.515 DEBUG:boto:Path: /dkyges-64/?prefix=obj44 2024-04-12T04:22:55.515 DEBUG:boto:Data: 2024-04-12T04:22:55.515 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.515 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.515 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.515 DEBUG:boto:Params: {} 2024-04-12T04:22:55.515 DEBUG:boto:Token: None 2024-04-12T04:22:55.516 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.516 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c32ff633f6dd1fbb-006618b71f-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.518 DEBUG:boto:b'dkyges-64obj441000falseobj442024-04-12T04:20:43.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.519 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.519 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.519 DEBUG:boto:path=/dkyges-64/?prefix=obj44 2024-04-12T04:22:55.519 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj44 2024-04-12T04:22:55.519 DEBUG:boto:Method: GET 2024-04-12T04:22:55.519 DEBUG:boto:Path: /dkyges-64/?prefix=obj44 2024-04-12T04:22:55.519 DEBUG:boto:Data: 2024-04-12T04:22:55.519 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.520 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.520 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.520 DEBUG:boto:Params: {} 2024-04-12T04:22:55.520 DEBUG:boto:Token: None 2024-04-12T04:22:55.520 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.520 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.523 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df37bd408bf86d3e-006618b71f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.523 DEBUG:boto:b'dkyges-64obj441000falseobj442024-04-12T04:20:43.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.524 DEBUG:boto:path=/dkyges-64/obj44 2024-04-12T04:22:55.524 DEBUG:boto:auth_path=/dkyges-64/obj44 2024-04-12T04:22:55.524 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.524 DEBUG:boto:Path: /dkyges-64/obj44 2024-04-12T04:22:55.524 DEBUG:boto:Data: 2024-04-12T04:22:55.524 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.524 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.524 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.525 DEBUG:boto:Params: {} 2024-04-12T04:22:55.525 DEBUG:boto:Token: None 2024-04-12T04:22:55.525 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj44 2024-04-12T04:22:55.525 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4SLFZ1nhMcTOTc2LATAZlcw6x8U= 2024-04-12T04:22:55.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4SLFZ1nhMcTOTc2LATAZlcw6x8U=', 'Content-Length': '0'} 2024-04-12T04:22:55.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000515dda2f8bc8ba49-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.527 DEBUG:boto:path=/dkyges-64/obj44 2024-04-12T04:22:55.528 DEBUG:boto:auth_path=/dkyges-64/obj44 2024-04-12T04:22:55.528 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.528 DEBUG:boto:Path: /dkyges-64/obj44 2024-04-12T04:22:55.528 DEBUG:boto:Data: 2024-04-12T04:22:55.528 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.529 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.529 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.529 DEBUG:boto:Params: {} 2024-04-12T04:22:55.529 DEBUG:boto:Token: None 2024-04-12T04:22:55.529 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj44 2024-04-12T04:22:55.530 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4SLFZ1nhMcTOTc2LATAZlcw6x8U= 2024-04-12T04:22:55.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4SLFZ1nhMcTOTc2LATAZlcw6x8U=', 'Content-Length': '0'} 2024-04-12T04:22:55.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8052639340ff79a-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:22:55.533 DEBUG:boto:path=/dkyges-64/obj44 2024-04-12T04:22:55.533 DEBUG:boto:auth_path=/dkyges-64/obj44 2024-04-12T04:22:55.533 DEBUG:boto:Method: GET 2024-04-12T04:22:55.533 DEBUG:boto:Path: /dkyges-64/obj44 2024-04-12T04:22:55.533 DEBUG:boto:Data: 2024-04-12T04:22:55.533 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.533 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.533 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.534 DEBUG:boto:Params: {} 2024-04-12T04:22:55.534 DEBUG:boto:Token: None 2024-04-12T04:22:55.534 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj44 2024-04-12T04:22:55.534 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BUINYikQLuwp9QNtyAaP+mEWHsY= 2024-04-12T04:22:55.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BUINYikQLuwp9QNtyAaP+mEWHsY=', 'Content-Length': '0'} 2024-04-12T04:22:55.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079d3c74f963710af-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.578 DEBUG:boto:path=/dkyges-64/obj44 2024-04-12T04:22:55.579 DEBUG:boto:auth_path=/dkyges-64/obj44 2024-04-12T04:22:55.579 DEBUG:boto:Method: GET 2024-04-12T04:22:55.579 DEBUG:boto:Path: /dkyges-64/obj44 2024-04-12T04:22:55.579 DEBUG:boto:Data: 2024-04-12T04:22:55.579 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.579 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.579 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.579 DEBUG:boto:Params: {} 2024-04-12T04:22:55.579 DEBUG:boto:Token: None 2024-04-12T04:22:55.579 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj44 2024-04-12T04:22:55.580 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BUINYikQLuwp9QNtyAaP+mEWHsY= 2024-04-12T04:22:55.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BUINYikQLuwp9QNtyAaP+mEWHsY=', 'Content-Length': '0'} 2024-04-12T04:22:55.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000692651ec810167f6-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.582 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.582 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.582 DEBUG:boto:path=/dkyges-64/?prefix=obj44 2024-04-12T04:22:55.582 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj44 2024-04-12T04:22:55.582 DEBUG:boto:Method: GET 2024-04-12T04:22:55.582 DEBUG:boto:Path: /dkyges-64/?prefix=obj44 2024-04-12T04:22:55.583 DEBUG:boto:Data: 2024-04-12T04:22:55.583 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.583 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.583 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.583 DEBUG:boto:Params: {} 2024-04-12T04:22:55.583 DEBUG:boto:Token: None 2024-04-12T04:22:55.583 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.583 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005faa3bb843c6158e-006618b71f-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.586 DEBUG:boto:b'dkyges-64obj441000falseobj442024-04-12T04:20:43.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.587 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.587 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.587 DEBUG:boto:path=/dkyges-64/?prefix=obj44 2024-04-12T04:22:55.587 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj44 2024-04-12T04:22:55.587 DEBUG:boto:Method: GET 2024-04-12T04:22:55.587 DEBUG:boto:Path: /dkyges-64/?prefix=obj44 2024-04-12T04:22:55.587 DEBUG:boto:Data: 2024-04-12T04:22:55.587 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.587 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.587 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.588 DEBUG:boto:Params: {} 2024-04-12T04:22:55.588 DEBUG:boto:Token: None 2024-04-12T04:22:55.588 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.588 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c830f1deea07038-006618b71f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.590 DEBUG:boto:b'dkyges-64obj441000falseobj442024-04-12T04:20:43.238Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:22:55.591 DEBUG:boto:path=/dkyges-64/obj55 2024-04-12T04:22:55.591 DEBUG:boto:auth_path=/dkyges-64/obj55 2024-04-12T04:22:55.591 DEBUG:boto:path=/dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.591 DEBUG:boto:auth_path=/dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.591 DEBUG:boto:Method: GET 2024-04-12T04:22:55.591 DEBUG:boto:Path: /dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.591 DEBUG:boto:Data: 2024-04-12T04:22:55.591 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.592 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.592 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.592 DEBUG:boto:Params: {} 2024-04-12T04:22:55.592 DEBUG:boto:Token: None 2024-04-12T04:22:55.592 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.592 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XSiBqHzUOsOu7a8UlvhvmPAwFLs= 2024-04-12T04:22:55.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XSiBqHzUOsOu7a8UlvhvmPAwFLs=', 'Content-Length': '0'} 2024-04-12T04:22:55.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b763c39f0ff3b242-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.595 DEBUG:boto:path=/dkyges-64/obj55 2024-04-12T04:22:55.595 DEBUG:boto:auth_path=/dkyges-64/obj55 2024-04-12T04:22:55.595 DEBUG:boto:path=/dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.595 DEBUG:boto:auth_path=/dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.595 DEBUG:boto:Method: GET 2024-04-12T04:22:55.595 DEBUG:boto:Path: /dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.595 DEBUG:boto:Data: 2024-04-12T04:22:55.595 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.595 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.595 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.596 DEBUG:boto:Params: {} 2024-04-12T04:22:55.596 DEBUG:boto:Token: None 2024-04-12T04:22:55.596 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.596 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XSiBqHzUOsOu7a8UlvhvmPAwFLs= 2024-04-12T04:22:55.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XSiBqHzUOsOu7a8UlvhvmPAwFLs=', 'Content-Length': '0'} 2024-04-12T04:22:55.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa9a39e45c387cbd-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.598 DEBUG:boto:path=/dkyges-64/obj55 2024-04-12T04:22:55.598 DEBUG:boto:auth_path=/dkyges-64/obj55 2024-04-12T04:22:55.599 DEBUG:boto:path=/dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.599 DEBUG:boto:auth_path=/dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.599 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.599 DEBUG:boto:Path: /dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.599 DEBUG:boto:Data: 2024-04-12T04:22:55.599 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.599 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.599 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.599 DEBUG:boto:Params: {} 2024-04-12T04:22:55.599 DEBUG:boto:Token: None 2024-04-12T04:22:55.599 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.600 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lnTYuJ+E/XWBv5kV0mD5ZPgGF7E= 2024-04-12T04:22:55.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lnTYuJ+E/XWBv5kV0mD5ZPgGF7E=', 'Content-Length': '0'} 2024-04-12T04:22:55.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068aff66d4df8b621-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.602 DEBUG:boto:path=/dkyges-64/obj55 2024-04-12T04:22:55.602 DEBUG:boto:auth_path=/dkyges-64/obj55 2024-04-12T04:22:55.602 DEBUG:boto:path=/dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.602 DEBUG:boto:auth_path=/dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.602 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.602 DEBUG:boto:Path: /dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.602 DEBUG:boto:Data: 2024-04-12T04:22:55.602 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.602 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.602 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.603 DEBUG:boto:Params: {} 2024-04-12T04:22:55.603 DEBUG:boto:Token: None 2024-04-12T04:22:55.603 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.603 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lnTYuJ+E/XWBv5kV0mD5ZPgGF7E= 2024-04-12T04:22:55.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lnTYuJ+E/XWBv5kV0mD5ZPgGF7E=', 'Content-Length': '0'} 2024-04-12T04:22:55.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000252620c83fcc7b36-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:22:55.605 DEBUG:boto:path=/dkyges-64/obj55 2024-04-12T04:22:55.605 DEBUG:boto:auth_path=/dkyges-64/obj55 2024-04-12T04:22:55.605 DEBUG:boto:path=/dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.605 DEBUG:boto:auth_path=/dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.605 DEBUG:boto:Method: GET 2024-04-12T04:22:55.606 DEBUG:boto:Path: /dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.606 DEBUG:boto:Data: 2024-04-12T04:22:55.606 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.606 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.606 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.606 DEBUG:boto:Params: {} 2024-04-12T04:22:55.606 DEBUG:boto:Token: None 2024-04-12T04:22:55.606 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.606 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XSiBqHzUOsOu7a8UlvhvmPAwFLs= 2024-04-12T04:22:55.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XSiBqHzUOsOu7a8UlvhvmPAwFLs=', 'Content-Length': '0'} 2024-04-12T04:22:55.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000774706bd52fcadde-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.650 DEBUG:boto:path=/dkyges-64/obj55 2024-04-12T04:22:55.651 DEBUG:boto:auth_path=/dkyges-64/obj55 2024-04-12T04:22:55.651 DEBUG:boto:path=/dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.651 DEBUG:boto:auth_path=/dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.651 DEBUG:boto:Method: GET 2024-04-12T04:22:55.651 DEBUG:boto:Path: /dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.651 DEBUG:boto:Data: 2024-04-12T04:22:55.651 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.651 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.651 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.652 DEBUG:boto:Params: {} 2024-04-12T04:22:55.652 DEBUG:boto:Token: None 2024-04-12T04:22:55.652 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj55?versionId=null 2024-04-12T04:22:55.652 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XSiBqHzUOsOu7a8UlvhvmPAwFLs= 2024-04-12T04:22:55.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XSiBqHzUOsOu7a8UlvhvmPAwFLs=', 'Content-Length': '0'} 2024-04-12T04:22:55.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cdb21135946e629-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.655 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.655 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.655 DEBUG:boto:path=/dkyges-64/?prefix=obj55 2024-04-12T04:22:55.655 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj55 2024-04-12T04:22:55.655 DEBUG:boto:Method: GET 2024-04-12T04:22:55.655 DEBUG:boto:Path: /dkyges-64/?prefix=obj55 2024-04-12T04:22:55.655 DEBUG:boto:Data: 2024-04-12T04:22:55.655 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.655 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.655 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.655 DEBUG:boto:Params: {} 2024-04-12T04:22:55.655 DEBUG:boto:Token: None 2024-04-12T04:22:55.656 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf59f23956f64296-006618b71f-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.658 DEBUG:boto:b'dkyges-64obj551000falseobj552024-04-12T04:20:43.286Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.659 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.659 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.659 DEBUG:boto:path=/dkyges-64/?prefix=obj55 2024-04-12T04:22:55.659 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj55 2024-04-12T04:22:55.659 DEBUG:boto:Method: GET 2024-04-12T04:22:55.659 DEBUG:boto:Path: /dkyges-64/?prefix=obj55 2024-04-12T04:22:55.659 DEBUG:boto:Data: 2024-04-12T04:22:55.659 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.659 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.660 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.660 DEBUG:boto:Params: {} 2024-04-12T04:22:55.660 DEBUG:boto:Token: None 2024-04-12T04:22:55.660 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.660 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a9921fb6ca9dee8-006618b71f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.706 DEBUG:boto:b'dkyges-64obj551000falseobj552024-04-12T04:20:43.286Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.707 DEBUG:boto:path=/dkyges-64/obj55 2024-04-12T04:22:55.707 DEBUG:boto:auth_path=/dkyges-64/obj55 2024-04-12T04:22:55.707 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.707 DEBUG:boto:Path: /dkyges-64/obj55 2024-04-12T04:22:55.707 DEBUG:boto:Data: 2024-04-12T04:22:55.707 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.707 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.707 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.708 DEBUG:boto:Params: {} 2024-04-12T04:22:55.708 DEBUG:boto:Token: None 2024-04-12T04:22:55.708 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj55 2024-04-12T04:22:55.708 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nku6b9QlVw1Okx7cfTcx+SiXA8E= 2024-04-12T04:22:55.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nku6b9QlVw1Okx7cfTcx+SiXA8E=', 'Content-Length': '0'} 2024-04-12T04:22:55.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ea1a49c71f80895-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.710 DEBUG:boto:path=/dkyges-64/obj55 2024-04-12T04:22:55.710 DEBUG:boto:auth_path=/dkyges-64/obj55 2024-04-12T04:22:55.711 DEBUG:boto:Method: HEAD 2024-04-12T04:22:55.711 DEBUG:boto:Path: /dkyges-64/obj55 2024-04-12T04:22:55.711 DEBUG:boto:Data: 2024-04-12T04:22:55.711 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.711 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.711 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.711 DEBUG:boto:Params: {} 2024-04-12T04:22:55.711 DEBUG:boto:Token: None 2024-04-12T04:22:55.711 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj55 2024-04-12T04:22:55.711 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nku6b9QlVw1Okx7cfTcx+SiXA8E= 2024-04-12T04:22:55.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nku6b9QlVw1Okx7cfTcx+SiXA8E=', 'Content-Length': '0'} 2024-04-12T04:22:55.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff06ffc7cced1f8a-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:22:55.714 DEBUG:boto:path=/dkyges-64/obj55 2024-04-12T04:22:55.714 DEBUG:boto:auth_path=/dkyges-64/obj55 2024-04-12T04:22:55.714 DEBUG:boto:Method: GET 2024-04-12T04:22:55.714 DEBUG:boto:Path: /dkyges-64/obj55 2024-04-12T04:22:55.714 DEBUG:boto:Data: 2024-04-12T04:22:55.714 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.714 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.714 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.715 DEBUG:boto:Params: {} 2024-04-12T04:22:55.715 DEBUG:boto:Token: None 2024-04-12T04:22:55.715 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj55 2024-04-12T04:22:55.715 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e0DOY6iZ4N89WOh/v2m6Iiu0wxQ= 2024-04-12T04:22:55.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e0DOY6iZ4N89WOh/v2m6Iiu0wxQ=', 'Content-Length': '0'} 2024-04-12T04:22:55.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-64:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aad01da25f5574df-006618b71f-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.717 DEBUG:boto:path=/dkyges-64/obj55 2024-04-12T04:22:55.717 DEBUG:boto:auth_path=/dkyges-64/obj55 2024-04-12T04:22:55.717 DEBUG:boto:Method: GET 2024-04-12T04:22:55.717 DEBUG:boto:Path: /dkyges-64/obj55 2024-04-12T04:22:55.717 DEBUG:boto:Data: 2024-04-12T04:22:55.717 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.718 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.718 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.718 DEBUG:boto:Params: {} 2024-04-12T04:22:55.718 DEBUG:boto:Token: None 2024-04-12T04:22:55.718 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/obj55 2024-04-12T04:22:55.718 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:e0DOY6iZ4N89WOh/v2m6Iiu0wxQ= 2024-04-12T04:22:55.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:e0DOY6iZ4N89WOh/v2m6Iiu0wxQ=', 'Content-Length': '0'} 2024-04-12T04:22:55.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:20:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cad7b315030b258-006618b71f-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.721 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.721 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.721 DEBUG:boto:path=/dkyges-64/?prefix=obj55 2024-04-12T04:22:55.721 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj55 2024-04-12T04:22:55.721 DEBUG:boto:Method: GET 2024-04-12T04:22:55.721 DEBUG:boto:Path: /dkyges-64/?prefix=obj55 2024-04-12T04:22:55.721 DEBUG:boto:Data: 2024-04-12T04:22:55.721 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.721 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:22:55.721 DEBUG:boto:Port: 8001 2024-04-12T04:22:55.721 DEBUG:boto:Params: {} 2024-04-12T04:22:55.722 DEBUG:boto:Token: None 2024-04-12T04:22:55.722 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.722 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.724 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4386030d1885bbf-006618b71f-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.724 DEBUG:boto:b'dkyges-64obj551000falseobj552024-04-12T04:20:43.286Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.725 DEBUG:boto:path=/dkyges-64/ 2024-04-12T04:22:55.725 DEBUG:boto:auth_path=/dkyges-64/ 2024-04-12T04:22:55.725 DEBUG:boto:path=/dkyges-64/?prefix=obj55 2024-04-12T04:22:55.725 DEBUG:boto:auth_path=/dkyges-64/?prefix=obj55 2024-04-12T04:22:55.725 DEBUG:boto:Method: GET 2024-04-12T04:22:55.725 DEBUG:boto:Path: /dkyges-64/?prefix=obj55 2024-04-12T04:22:55.725 DEBUG:boto:Data: 2024-04-12T04:22:55.725 DEBUG:boto:Headers: {} 2024-04-12T04:22:55.725 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:55.725 DEBUG:boto:Port: 8000 2024-04-12T04:22:55.725 DEBUG:boto:Params: {} 2024-04-12T04:22:55.726 DEBUG:boto:Token: None 2024-04-12T04:22:55.726 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:22:55 GMT /dkyges-64/ 2024-04-12T04:22:55.726 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0= 2024-04-12T04:22:55.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T5CZ5EeNSWbq7EFa+rmFyQq8PO0=', 'Content-Length': '0'} 2024-04-12T04:22:55.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c068281aeac070a-006618b71f-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:22:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:55.728 DEBUG:boto:b'dkyges-64obj551000falseobj552024-04-12T04:20:43.286Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:22:55.729 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-64 zones={a2, a1} 2024-04-12T04:22:55.729 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2024-04-12T04:22:55.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:55.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:55.729 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:55.788 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:55.788 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:55.848 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:22:55.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:55.848 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000034752", 2024-04-12T04:22:55.848 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:20:46.036264Z" 2024-04-12T04:22:55.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:55.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:55.849 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000023848", 2024-04-12T04:22:55.849 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:16:00.031207Z" 2024-04-12T04:22:55.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:55.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:55.849 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000040349", 2024-04-12T04:22:55.849 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:20:43.241899Z" 2024-04-12T04:22:55.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:55.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:55.849 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000035117", 2024-04-12T04:22:55.849 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:20:46.038121Z" 2024-04-12T04:22:55.849 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:55.850 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:22:55.851 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000034752', 1: '00000000000000000000:00000000000000023848', 2: '00000000000000000000:00000000000000040349', 3: '00000000000000000000:00000000000000035117'} 2024-04-12T04:22:55.851 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:22:55.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:55.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:55.852 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:55.909 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:55.909 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:55.961 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:22:55.962 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:22:55.962 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:22:55.962 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:22:55.962 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:22:55.962 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 15926411381064122149 2024-04-12T04:22:55.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:55.962 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:22:55.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:55.962 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:22:55.962 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:22:55.963 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:22:55.963 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000034752", 2024-04-12T04:22:55.963 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:22:55.963 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:22:55.963 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:22:55.963 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:20:46.036264Z" 2024-04-12T04:22:55.963 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:55.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:55.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:55.963 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:22:55.963 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:22:55.963 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:22:55.964 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000023848", 2024-04-12T04:22:55.964 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:22:55.964 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:22:55.964 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:22:55.964 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:16:00.031207Z" 2024-04-12T04:22:55.964 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:55.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:55.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:55.964 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:22:55.964 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:22:55.964 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:22:55.964 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000040349", 2024-04-12T04:22:55.965 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:22:55.965 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 55, 2024-04-12T04:22:55.965 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:22:55.965 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:20:43.241899Z" 2024-04-12T04:22:55.965 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:55.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:55.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:55.965 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:22:55.965 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:22:55.965 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:22:55.965 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000035117", 2024-04-12T04:22:55.965 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:22:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:22:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:22:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:20:46.038121Z" 2024-04-12T04:22:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:22:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:22:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 64, 2024-04-12T04:22:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 64 2024-04-12T04:22:55.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:55.967 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:22:55Z" 2024-04-12T04:22:55.967 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:22:55.967 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 15926411381064122149 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000034752", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:20:46.036264Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000023848", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:16:00.031207Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000040349", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2024-04-12T04:20:43.241899Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000035117", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:20:46.038121Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2024-04-12T04:22:55Z" } 2024-04-12T04:22:55.968 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000034752', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:20:46.036264Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000023848', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:16:00.031207Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000040349', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2024-04-12T04:20:43.241899Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000035117', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:20:46.038121Z'}}] 2024-04-12T04:22:55.968 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000034752', 1: '00000000000000000000:00000000000000023848', 2: '00000000000000000000:00000000000000040349', 3: '00000000000000000000:00000000000000035117'} 2024-04-12T04:22:55.968 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000034752', 1: '00000000000000000000:00000000000000023848', 2: '00000000000000000000:00000000000000040349', 3: '00000000000000000000:00000000000000035117'} 2024-04-12T04:22:55.968 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:22:55.968 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2024-04-12T04:22:55.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:55.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:55.969 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:56.084 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:22:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000013374", 2024-04-12T04:22:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:22:47.939638Z" 2024-04-12T04:22:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:56.085 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006573", 2024-04-12T04:22:56.085 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:17:50.167613Z" 2024-04-12T04:22:56.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:56.085 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000018267", 2024-04-12T04:22:56.085 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:22:47.982514Z" 2024-04-12T04:22:56.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:56.085 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000016719", 2024-04-12T04:22:56.085 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:22:33.575218Z" 2024-04-12T04:22:56.085 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:56.085 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:22:56.086 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000013374', 1: '00000000000000000000:00000000000000006573', 2: '00000000000000000000:00000000000000018267', 3: '00000000000000000000:00000000000000016719'} 2024-04-12T04:22:56.087 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T04:22:56.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:56.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:56.087 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:56.144 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:56.144 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:56.196 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:22:56.196 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:22:56.196 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:22:56.196 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:22:56.196 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:22:56.196 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5944041172307441597 2024-04-12T04:22:56.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:22:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:22:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:22:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:22:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000013374", 2024-04-12T04:22:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:22:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:22:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:22:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:22:47.939638Z" 2024-04-12T04:22:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:22:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:22:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:22:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000006573", 2024-04-12T04:22:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:22:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:22:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:22:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:17:50.167613Z" 2024-04-12T04:22:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:22:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:22:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:22:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000018267", 2024-04-12T04:22:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:22:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:22:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:22:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:22:47.982514Z" 2024-04-12T04:22:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:22:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:22:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:22:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000016719", 2024-04-12T04:22:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:22:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:22:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:22:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:22:33.575218Z" 2024-04-12T04:22:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:22:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:22:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T04:22:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T04:22:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:22:56Z" 2024-04-12T04:22:56.201 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:22:56.202 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5944041172307441597 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013374", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:22:47.939638Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006573", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:17:50.167613Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018267", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:22:47.982514Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016719", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:22:33.575218Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T04:22:56Z" } 2024-04-12T04:22:56.202 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013374', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:22:47.939638Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006573', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:17:50.167613Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018267', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:22:47.982514Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016719', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:22:33.575218Z'}}] 2024-04-12T04:22:56.202 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000013374', 1: '00000000000000000000:00000000000000006573', 2: '00000000000000000000:00000000000000018267', 3: '00000000000000000000:00000000000000016719'} 2024-04-12T04:22:56.203 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000013374', 1: '00000000000000000000:00000000000000006573', 2: '00000000000000000000:00000000000000018267', 3: '00000000000000000000:00000000000000016719'} 2024-04-12T04:22:56.203 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T04:22:56.203 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... ok 2024-04-12T04:22:56.204 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:22:56.204 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:22:56.205 DEBUG:boto:Method: PUT 2024-04-12T04:22:56.205 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:22:56.205 DEBUG:boto:Data: 2024-04-12T04:22:56.205 DEBUG:boto:Headers: {} 2024-04-12T04:22:56.205 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:22:56.205 DEBUG:boto:Port: 8000 2024-04-12T04:22:56.205 DEBUG:boto:Params: {} 2024-04-12T04:22:56.205 DEBUG:boto:Token: None 2024-04-12T04:22:56.205 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:22:56 GMT /dkyges-65/ 2024-04-12T04:22:56.206 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Px26zZ2Luw6BoCB4irdzVIYW+Us= 2024-04-12T04:22:56.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Px26zZ2Luw6BoCB4irdzVIYW+Us=', 'Content-Length': '0'} 2024-04-12T04:22:56.223 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000673d814c50f2f61b-006618b720-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:22:56.223 DEBUG:rgw_multi.tests:created bucket=dkyges-65 2024-04-12T04:22:56.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:56.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:56.224 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:56.314 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:22:56.330 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:22:56.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:56.330 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:22:56.330 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:22:56.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:56.330 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:22:56.330 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:22:56.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:56.331 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:22:56.331 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:22:56.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:56.331 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895776.234016_710.1", 2024-04-12T04:22:56.331 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:22:56.234016Z" 2024-04-12T04:22:56.331 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:56.331 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:22:56.332 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895776.234016_710.1'} 2024-04-12T04:22:56.332 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:22:56.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:56.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:22:56.333 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:22:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:22:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:22:56.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:22:56.434+0000 7fe7a5df1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:56.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:22:56.435+0000 7fe7a5df1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:22:56.439 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:22:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:22:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:22:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:22:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:22:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:22:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:22:56.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:22:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:22:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:22:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:22:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:22:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:22:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:22:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:22:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:22:56.440 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:22:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:22:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:22:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:22:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:22:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:22:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:22:56.441 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:22:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:22:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:22:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:22:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:22:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:22:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:22:56.442 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:22:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:22:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:22:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:22:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:22:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:22:56.443 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:22:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:22:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:22:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895776.227017_707.1", 2024-04-12T04:22:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:22:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:22:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:22:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:22:56.227017Z", 2024-04-12T04:22:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:22:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:56.444 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:22:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:22:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:22:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:22:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:22:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:22:56.445 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:22:56Z" 2024-04-12T04:22:56.445 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:22:56.447 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895776.227017_707.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:22:56.227017Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:22:56Z" } 2024-04-12T04:22:56.447 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895776.227017_707.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:22:56.227017Z', 'realm_epoch': 2}}] 2024-04-12T04:22:56.447 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895776.234016_710.1'} 2024-04-12T04:22:56.447 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895776.227017_707.1'} 2024-04-12T04:22:56.447 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712895776.234016_710.1 target=1_1712895776.227017_707.1 2024-04-12T04:23:01.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:01.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:23:01.449 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:23:01.508 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:01.508 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:01.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:23:01.555+0000 7f8b1c8cf780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:01.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:23:01.556+0000 7f8b1c8cf780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:23:01.560 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:23:01.560 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:23:01.560 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:23:01.560 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:23:01.560 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:23:01.560 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:23:01.561 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:23:01.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:23:01.561 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:23:01.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:23:01.561 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:23:01.561 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:23:01.561 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:23:01.561 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:23:01.561 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:23:01.561 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:23:01.562 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:23:01.562 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:23:01.562 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:23:01.562 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:23:01.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:23:01.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:23:01.562 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:23:01.562 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:23:01.562 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:23:01.562 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:23:01.562 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:23:01.563 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:23:01.563 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:23:01.563 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:23:01.563 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:23:01.563 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:23:01.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:23:01.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:23:01.563 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:23:01.563 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:23:01.563 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:23:01.563 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:23:01.564 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:23:01.564 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:23:01.564 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:23:01.564 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:23:01.564 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:23:01.564 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:23:01.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:23:01.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:23:01.564 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:23:01.564 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:23:01.564 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:23:01.565 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895776.234016_710.1", 2024-04-12T04:23:01.565 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:23:01.565 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:23:01.565 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:23:01.565 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:22:56.234016Z", 2024-04-12T04:23:01.565 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:23:01.565 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:23:01.565 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:23:01.565 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:23:01.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:23:01.565 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:23:01.565 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:23:01.566 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:23:01.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:23:01.566 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:23:01Z" 2024-04-12T04:23:01.566 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:23:01.567 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712895776.234016_710.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:22:56.234016Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:23:01Z" } 2024-04-12T04:23:01.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895776.234016_710.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:22:56.234016Z', 'realm_epoch': 2}}] 2024-04-12T04:23:01.567 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895776.234016_710.1'} 2024-04-12T04:23:01.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895776.234016_710.1'} 2024-04-12T04:23:01.567 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:23:01.568 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:01.568 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:01.568 DEBUG:boto:Method: HEAD 2024-04-12T04:23:01.568 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:01.568 DEBUG:boto:Data: 2024-04-12T04:23:01.568 DEBUG:boto:Headers: {} 2024-04-12T04:23:01.568 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.568 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.568 DEBUG:boto:Params: {} 2024-04-12T04:23:01.569 DEBUG:boto:Token: None 2024-04-12T04:23:01.569 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/ 2024-04-12T04:23:01.569 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:23:01.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:23:01.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ab1efb7e51c28df-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.572 DEBUG:boto:path=/dkyges-65/obj11 2024-04-12T04:23:01.572 DEBUG:boto:auth_path=/dkyges-65/obj11 2024-04-12T04:23:01.572 DEBUG:boto:Method: PUT 2024-04-12T04:23:01.572 DEBUG:boto:Path: /dkyges-65/obj11 2024-04-12T04:23:01.572 DEBUG:boto:Data: 2024-04-12T04:23:01.572 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:01.572 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.572 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.573 DEBUG:boto:Params: {} 2024-04-12T04:23:01.573 DEBUG:boto:Token: None 2024-04-12T04:23:01.573 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/obj11 2024-04-12T04:23:01.573 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TamJochJjeBZpFBjdgUM5o3meeA= 2024-04-12T04:23:01.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TamJochJjeBZpFBjdgUM5o3meeA='} 2024-04-12T04:23:01.580 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006de889b4c3b0e4a7-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.580 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:01.580 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:01.580 DEBUG:boto:Method: HEAD 2024-04-12T04:23:01.580 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:01.580 DEBUG:boto:Data: 2024-04-12T04:23:01.580 DEBUG:boto:Headers: {} 2024-04-12T04:23:01.581 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.581 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.581 DEBUG:boto:Params: {} 2024-04-12T04:23:01.581 DEBUG:boto:Token: None 2024-04-12T04:23:01.581 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/ 2024-04-12T04:23:01.581 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:23:01.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:23:01.583 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002683771de7f57379-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.583 DEBUG:boto:path=/dkyges-65/obj22 2024-04-12T04:23:01.583 DEBUG:boto:auth_path=/dkyges-65/obj22 2024-04-12T04:23:01.584 DEBUG:boto:Method: PUT 2024-04-12T04:23:01.584 DEBUG:boto:Path: /dkyges-65/obj22 2024-04-12T04:23:01.584 DEBUG:boto:Data: 2024-04-12T04:23:01.584 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:01.584 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.584 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.584 DEBUG:boto:Params: {} 2024-04-12T04:23:01.584 DEBUG:boto:Token: None 2024-04-12T04:23:01.585 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/obj22 2024-04-12T04:23:01.585 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:vW6XFzSHFFq/USuIX+tHHzoFR98= 2024-04-12T04:23:01.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vW6XFzSHFFq/USuIX+tHHzoFR98='} 2024-04-12T04:23:01.626 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f4265dda3527e4c-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.627 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:01.627 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:01.627 DEBUG:boto:Method: HEAD 2024-04-12T04:23:01.627 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:01.627 DEBUG:boto:Data: 2024-04-12T04:23:01.627 DEBUG:boto:Headers: {} 2024-04-12T04:23:01.627 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.627 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.627 DEBUG:boto:Params: {} 2024-04-12T04:23:01.627 DEBUG:boto:Token: None 2024-04-12T04:23:01.628 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/ 2024-04-12T04:23:01.628 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:23:01.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:23:01.630 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f96514ded0fc2c7-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.631 DEBUG:boto:path=/dkyges-65/obj33 2024-04-12T04:23:01.631 DEBUG:boto:auth_path=/dkyges-65/obj33 2024-04-12T04:23:01.631 DEBUG:boto:Method: PUT 2024-04-12T04:23:01.631 DEBUG:boto:Path: /dkyges-65/obj33 2024-04-12T04:23:01.631 DEBUG:boto:Data: 2024-04-12T04:23:01.631 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:01.631 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.631 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.631 DEBUG:boto:Params: {} 2024-04-12T04:23:01.632 DEBUG:boto:Token: None 2024-04-12T04:23:01.632 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/obj33 2024-04-12T04:23:01.632 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4o+RbMX2W659ut9PuKjdMIipQLk= 2024-04-12T04:23:01.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4o+RbMX2W659ut9PuKjdMIipQLk='} 2024-04-12T04:23:01.638 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe78d4186379afbd-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.638 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:01.638 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:01.639 DEBUG:boto:Method: HEAD 2024-04-12T04:23:01.639 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:01.639 DEBUG:boto:Data: 2024-04-12T04:23:01.639 DEBUG:boto:Headers: {} 2024-04-12T04:23:01.639 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.639 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.639 DEBUG:boto:Params: {} 2024-04-12T04:23:01.639 DEBUG:boto:Token: None 2024-04-12T04:23:01.639 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/ 2024-04-12T04:23:01.639 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:23:01.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:23:01.642 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7448aa70cc65272-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.643 DEBUG:boto:path=/dkyges-65/obj44 2024-04-12T04:23:01.643 DEBUG:boto:auth_path=/dkyges-65/obj44 2024-04-12T04:23:01.643 DEBUG:boto:Method: PUT 2024-04-12T04:23:01.643 DEBUG:boto:Path: /dkyges-65/obj44 2024-04-12T04:23:01.643 DEBUG:boto:Data: 2024-04-12T04:23:01.643 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:01.643 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.643 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.643 DEBUG:boto:Params: {} 2024-04-12T04:23:01.643 DEBUG:boto:Token: None 2024-04-12T04:23:01.644 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/obj44 2024-04-12T04:23:01.644 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lLt+sILykaAxamIhhmvp8+031OU= 2024-04-12T04:23:01.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lLt+sILykaAxamIhhmvp8+031OU='} 2024-04-12T04:23:01.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fea2da3370bf44b6-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.691 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:01.691 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:01.691 DEBUG:boto:Method: HEAD 2024-04-12T04:23:01.691 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:01.691 DEBUG:boto:Data: 2024-04-12T04:23:01.691 DEBUG:boto:Headers: {} 2024-04-12T04:23:01.691 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.691 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.691 DEBUG:boto:Params: {} 2024-04-12T04:23:01.692 DEBUG:boto:Token: None 2024-04-12T04:23:01.692 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/ 2024-04-12T04:23:01.692 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:23:01.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:23:01.694 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b2e220156bef33d-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.695 DEBUG:boto:path=/dkyges-65/obj55 2024-04-12T04:23:01.695 DEBUG:boto:auth_path=/dkyges-65/obj55 2024-04-12T04:23:01.695 DEBUG:boto:Method: PUT 2024-04-12T04:23:01.695 DEBUG:boto:Path: /dkyges-65/obj55 2024-04-12T04:23:01.695 DEBUG:boto:Data: 2024-04-12T04:23:01.695 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:01.695 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.695 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.695 DEBUG:boto:Params: {} 2024-04-12T04:23:01.696 DEBUG:boto:Token: None 2024-04-12T04:23:01.696 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/obj55 2024-04-12T04:23:01.696 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zOLyprXBA0N7JjjbVf6YjfIsFo4= 2024-04-12T04:23:01.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zOLyprXBA0N7JjjbVf6YjfIsFo4='} 2024-04-12T04:23:01.701 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001af9cf35a13ee4ee-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:23:01.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:23:01.701 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-65 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:23:01.764 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:01.764 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:01.818 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:23:01.818 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-65 2024-04-12T04:23:01.823 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 5 2024-04-12T04:23:01.873 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:01.873 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:01.873 DEBUG:boto:Method: HEAD 2024-04-12T04:23:01.874 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:01.874 DEBUG:boto:Data: 2024-04-12T04:23:01.874 DEBUG:boto:Headers: {} 2024-04-12T04:23:01.874 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.874 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.874 DEBUG:boto:Params: {} 2024-04-12T04:23:01.874 DEBUG:boto:Token: None 2024-04-12T04:23:01.874 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/ 2024-04-12T04:23:01.875 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:23:01.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:23:01.878 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd1f30e1e10b3aa1-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.878 DEBUG:boto:path=/dkyges-65/obj17 2024-04-12T04:23:01.878 DEBUG:boto:auth_path=/dkyges-65/obj17 2024-04-12T04:23:01.878 DEBUG:boto:Method: PUT 2024-04-12T04:23:01.879 DEBUG:boto:Path: /dkyges-65/obj17 2024-04-12T04:23:01.879 DEBUG:boto:Data: 2024-04-12T04:23:01.879 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:01.879 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.879 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.879 DEBUG:boto:Params: {} 2024-04-12T04:23:01.879 DEBUG:boto:Token: None 2024-04-12T04:23:01.879 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/obj17 2024-04-12T04:23:01.879 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:jeStFdceOfVVYnEPMraEvTI7idA= 2024-04-12T04:23:01.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jeStFdceOfVVYnEPMraEvTI7idA='} 2024-04-12T04:23:01.886 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000694b66c105622b09-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.886 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:01.886 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:01.886 DEBUG:boto:Method: HEAD 2024-04-12T04:23:01.886 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:01.886 DEBUG:boto:Data: 2024-04-12T04:23:01.886 DEBUG:boto:Headers: {} 2024-04-12T04:23:01.886 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.887 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.887 DEBUG:boto:Params: {} 2024-04-12T04:23:01.887 DEBUG:boto:Token: None 2024-04-12T04:23:01.887 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/ 2024-04-12T04:23:01.887 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:23:01.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:23:01.889 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcc6943a2f449dd4-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.890 DEBUG:boto:path=/dkyges-65/obj34 2024-04-12T04:23:01.890 DEBUG:boto:auth_path=/dkyges-65/obj34 2024-04-12T04:23:01.890 DEBUG:boto:Method: PUT 2024-04-12T04:23:01.890 DEBUG:boto:Path: /dkyges-65/obj34 2024-04-12T04:23:01.890 DEBUG:boto:Data: 2024-04-12T04:23:01.890 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:01.890 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.890 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.890 DEBUG:boto:Params: {} 2024-04-12T04:23:01.891 DEBUG:boto:Token: None 2024-04-12T04:23:01.891 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/obj34 2024-04-12T04:23:01.891 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:paXK8eL4odhwMkbzaVmG67LexuU= 2024-04-12T04:23:01.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:paXK8eL4odhwMkbzaVmG67LexuU='} 2024-04-12T04:23:01.935 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074df2edcfd806301-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.935 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:01.935 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:01.935 DEBUG:boto:Method: HEAD 2024-04-12T04:23:01.935 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:01.935 DEBUG:boto:Data: 2024-04-12T04:23:01.935 DEBUG:boto:Headers: {} 2024-04-12T04:23:01.935 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.936 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.936 DEBUG:boto:Params: {} 2024-04-12T04:23:01.936 DEBUG:boto:Token: None 2024-04-12T04:23:01.936 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/ 2024-04-12T04:23:01.936 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:23:01.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:23:01.939 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bdb4210e0307c32-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.939 DEBUG:boto:path=/dkyges-65/obj51 2024-04-12T04:23:01.939 DEBUG:boto:auth_path=/dkyges-65/obj51 2024-04-12T04:23:01.939 DEBUG:boto:Method: PUT 2024-04-12T04:23:01.939 DEBUG:boto:Path: /dkyges-65/obj51 2024-04-12T04:23:01.939 DEBUG:boto:Data: 2024-04-12T04:23:01.939 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:01.940 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.940 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.940 DEBUG:boto:Params: {} 2024-04-12T04:23:01.940 DEBUG:boto:Token: None 2024-04-12T04:23:01.940 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/obj51 2024-04-12T04:23:01.940 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a4wNOQzbyh9uH1WJ9NfH8grScmU= 2024-04-12T04:23:01.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a4wNOQzbyh9uH1WJ9NfH8grScmU='} 2024-04-12T04:23:01.947 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b02fe0beeddb761f-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.947 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:01.947 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:01.947 DEBUG:boto:Method: HEAD 2024-04-12T04:23:01.947 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:01.947 DEBUG:boto:Data: 2024-04-12T04:23:01.947 DEBUG:boto:Headers: {} 2024-04-12T04:23:01.947 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.948 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.948 DEBUG:boto:Params: {} 2024-04-12T04:23:01.948 DEBUG:boto:Token: None 2024-04-12T04:23:01.948 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/ 2024-04-12T04:23:01.948 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:23:01.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:23:01.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006cc60a27c872ddbb-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.951 DEBUG:boto:path=/dkyges-65/obj68 2024-04-12T04:23:01.951 DEBUG:boto:auth_path=/dkyges-65/obj68 2024-04-12T04:23:01.951 DEBUG:boto:Method: PUT 2024-04-12T04:23:01.951 DEBUG:boto:Path: /dkyges-65/obj68 2024-04-12T04:23:01.952 DEBUG:boto:Data: 2024-04-12T04:23:01.952 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:01.952 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.952 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.952 DEBUG:boto:Params: {} 2024-04-12T04:23:01.952 DEBUG:boto:Token: None 2024-04-12T04:23:01.952 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:01 GMT /dkyges-65/obj68 2024-04-12T04:23:01.952 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:otf2YyPKShxKETiJY+CAsw635bc= 2024-04-12T04:23:01.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:otf2YyPKShxKETiJY+CAsw635bc='} 2024-04-12T04:23:01.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000bd2f1835c90684a-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:01.999 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:01.999 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:01.999 DEBUG:boto:Method: HEAD 2024-04-12T04:23:01.999 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:01.999 DEBUG:boto:Data: 2024-04-12T04:23:01.999 DEBUG:boto:Headers: {} 2024-04-12T04:23:01.999 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:01.999 DEBUG:boto:Port: 8000 2024-04-12T04:23:01.999 DEBUG:boto:Params: {} 2024-04-12T04:23:02.000 DEBUG:boto:Token: None 2024-04-12T04:23:02.000 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/ 2024-04-12T04:23:02.000 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:23:02.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:23:02.003 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a598f786a4e4176d-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.003 DEBUG:boto:path=/dkyges-65/obj85 2024-04-12T04:23:02.003 DEBUG:boto:auth_path=/dkyges-65/obj85 2024-04-12T04:23:02.003 DEBUG:boto:Method: PUT 2024-04-12T04:23:02.003 DEBUG:boto:Path: /dkyges-65/obj85 2024-04-12T04:23:02.003 DEBUG:boto:Data: 2024-04-12T04:23:02.003 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:02.003 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.004 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.004 DEBUG:boto:Params: {} 2024-04-12T04:23:02.004 DEBUG:boto:Token: None 2024-04-12T04:23:02.004 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/obj85 2024-04-12T04:23:02.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3ATwZxRNfIsdtWFNMcVEuKfN5Ks= 2024-04-12T04:23:02.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3ATwZxRNfIsdtWFNMcVEuKfN5Ks='} 2024-04-12T04:23:02.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ca36133c1f8c897-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:23:02.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:23:02.011 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-65 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:23:02.073 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:02.073 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:02.127 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:23:02.127 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-65 2024-04-12T04:23:02.135 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 10 2024-04-12T04:23:02.191 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:02.191 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:02.191 DEBUG:boto:Method: HEAD 2024-04-12T04:23:02.192 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:02.192 DEBUG:boto:Data: 2024-04-12T04:23:02.192 DEBUG:boto:Headers: {} 2024-04-12T04:23:02.192 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.192 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.192 DEBUG:boto:Params: {} 2024-04-12T04:23:02.192 DEBUG:boto:Token: None 2024-04-12T04:23:02.192 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/ 2024-04-12T04:23:02.192 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:23:02.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:23:02.195 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c765b4102e8d99a4-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.196 DEBUG:boto:path=/dkyges-65/obj19 2024-04-12T04:23:02.196 DEBUG:boto:auth_path=/dkyges-65/obj19 2024-04-12T04:23:02.196 DEBUG:boto:Method: PUT 2024-04-12T04:23:02.196 DEBUG:boto:Path: /dkyges-65/obj19 2024-04-12T04:23:02.196 DEBUG:boto:Data: 2024-04-12T04:23:02.196 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:02.197 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.197 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.197 DEBUG:boto:Params: {} 2024-04-12T04:23:02.197 DEBUG:boto:Token: None 2024-04-12T04:23:02.197 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/obj19 2024-04-12T04:23:02.197 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nrRRiqSTb8PSSdORUl0k0/JjOpA= 2024-04-12T04:23:02.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nrRRiqSTb8PSSdORUl0k0/JjOpA='} 2024-04-12T04:23:02.242 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042d8d750c8ddb9ee-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.243 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:02.243 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:02.243 DEBUG:boto:Method: HEAD 2024-04-12T04:23:02.243 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:02.243 DEBUG:boto:Data: 2024-04-12T04:23:02.243 DEBUG:boto:Headers: {} 2024-04-12T04:23:02.244 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.244 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.244 DEBUG:boto:Params: {} 2024-04-12T04:23:02.244 DEBUG:boto:Token: None 2024-04-12T04:23:02.244 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/ 2024-04-12T04:23:02.244 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:23:02.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:23:02.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007b3f051cdfd759b-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.247 DEBUG:boto:path=/dkyges-65/obj38 2024-04-12T04:23:02.248 DEBUG:boto:auth_path=/dkyges-65/obj38 2024-04-12T04:23:02.248 DEBUG:boto:Method: PUT 2024-04-12T04:23:02.248 DEBUG:boto:Path: /dkyges-65/obj38 2024-04-12T04:23:02.248 DEBUG:boto:Data: 2024-04-12T04:23:02.248 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:02.248 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.249 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.249 DEBUG:boto:Params: {} 2024-04-12T04:23:02.249 DEBUG:boto:Token: None 2024-04-12T04:23:02.249 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/obj38 2024-04-12T04:23:02.249 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FVblpfpL8Ke6mzJfZ7lEPdENVSQ= 2024-04-12T04:23:02.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FVblpfpL8Ke6mzJfZ7lEPdENVSQ='} 2024-04-12T04:23:02.254 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ffad7725ac20bb61-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.254 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:02.254 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:02.254 DEBUG:boto:Method: HEAD 2024-04-12T04:23:02.254 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:02.254 DEBUG:boto:Data: 2024-04-12T04:23:02.254 DEBUG:boto:Headers: {} 2024-04-12T04:23:02.254 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.255 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.255 DEBUG:boto:Params: {} 2024-04-12T04:23:02.255 DEBUG:boto:Token: None 2024-04-12T04:23:02.255 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/ 2024-04-12T04:23:02.255 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:23:02.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:23:02.257 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5c5e41acef08872-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.258 DEBUG:boto:path=/dkyges-65/obj57 2024-04-12T04:23:02.258 DEBUG:boto:auth_path=/dkyges-65/obj57 2024-04-12T04:23:02.258 DEBUG:boto:Method: PUT 2024-04-12T04:23:02.258 DEBUG:boto:Path: /dkyges-65/obj57 2024-04-12T04:23:02.258 DEBUG:boto:Data: 2024-04-12T04:23:02.258 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:02.258 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.258 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.258 DEBUG:boto:Params: {} 2024-04-12T04:23:02.258 DEBUG:boto:Token: None 2024-04-12T04:23:02.259 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/obj57 2024-04-12T04:23:02.259 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:awSFzwG9MnVEuaTg21PVYVnzNP8= 2024-04-12T04:23:02.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:awSFzwG9MnVEuaTg21PVYVnzNP8='} 2024-04-12T04:23:02.306 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000038a8581b85c047dc-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.307 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:02.307 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:02.307 DEBUG:boto:Method: HEAD 2024-04-12T04:23:02.307 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:02.307 DEBUG:boto:Data: 2024-04-12T04:23:02.307 DEBUG:boto:Headers: {} 2024-04-12T04:23:02.307 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.307 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.307 DEBUG:boto:Params: {} 2024-04-12T04:23:02.307 DEBUG:boto:Token: None 2024-04-12T04:23:02.308 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/ 2024-04-12T04:23:02.308 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:23:02.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:23:02.311 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001585a2499aacdd78-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.311 DEBUG:boto:path=/dkyges-65/obj76 2024-04-12T04:23:02.311 DEBUG:boto:auth_path=/dkyges-65/obj76 2024-04-12T04:23:02.311 DEBUG:boto:Method: PUT 2024-04-12T04:23:02.311 DEBUG:boto:Path: /dkyges-65/obj76 2024-04-12T04:23:02.311 DEBUG:boto:Data: 2024-04-12T04:23:02.311 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:02.312 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.312 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.312 DEBUG:boto:Params: {} 2024-04-12T04:23:02.312 DEBUG:boto:Token: None 2024-04-12T04:23:02.312 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/obj76 2024-04-12T04:23:02.312 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7tNsvkv+ctes0wcKUDEjeTn8KXg= 2024-04-12T04:23:02.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7tNsvkv+ctes0wcKUDEjeTn8KXg='} 2024-04-12T04:23:02.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ef29de6b8f0522c-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.318 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:02.319 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:02.319 DEBUG:boto:Method: HEAD 2024-04-12T04:23:02.319 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:02.319 DEBUG:boto:Data: 2024-04-12T04:23:02.319 DEBUG:boto:Headers: {} 2024-04-12T04:23:02.319 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.319 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.319 DEBUG:boto:Params: {} 2024-04-12T04:23:02.319 DEBUG:boto:Token: None 2024-04-12T04:23:02.320 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/ 2024-04-12T04:23:02.320 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:23:02.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:23:02.323 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f670f7750b6077f-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.323 DEBUG:boto:path=/dkyges-65/obj95 2024-04-12T04:23:02.323 DEBUG:boto:auth_path=/dkyges-65/obj95 2024-04-12T04:23:02.323 DEBUG:boto:Method: PUT 2024-04-12T04:23:02.323 DEBUG:boto:Path: /dkyges-65/obj95 2024-04-12T04:23:02.323 DEBUG:boto:Data: 2024-04-12T04:23:02.323 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:02.324 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.324 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.324 DEBUG:boto:Params: {} 2024-04-12T04:23:02.324 DEBUG:boto:Token: None 2024-04-12T04:23:02.324 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/obj95 2024-04-12T04:23:02.324 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OO5qQ+z3WwxyJEpL8vhVdsZ01bI= 2024-04-12T04:23:02.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OO5qQ+z3WwxyJEpL8vhVdsZ01bI='} 2024-04-12T04:23:02.367 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065b58d984df16681-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:23:02.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:23:02.367 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-65 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:23:02.434 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:02.434 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:02.487 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:23:02.487 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-65 2024-04-12T04:23:02.495 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 15 2024-04-12T04:23:02.552 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:02.552 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:02.552 DEBUG:boto:Method: HEAD 2024-04-12T04:23:02.552 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:02.552 DEBUG:boto:Data: 2024-04-12T04:23:02.552 DEBUG:boto:Headers: {} 2024-04-12T04:23:02.552 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.552 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.552 DEBUG:boto:Params: {} 2024-04-12T04:23:02.553 DEBUG:boto:Token: None 2024-04-12T04:23:02.553 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/ 2024-04-12T04:23:02.553 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:23:02.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:23:02.556 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f4ad25ad546c7a5-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.557 DEBUG:boto:path=/dkyges-65/obj23 2024-04-12T04:23:02.557 DEBUG:boto:auth_path=/dkyges-65/obj23 2024-04-12T04:23:02.557 DEBUG:boto:Method: PUT 2024-04-12T04:23:02.557 DEBUG:boto:Path: /dkyges-65/obj23 2024-04-12T04:23:02.557 DEBUG:boto:Data: 2024-04-12T04:23:02.557 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:02.557 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.557 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.557 DEBUG:boto:Params: {} 2024-04-12T04:23:02.558 DEBUG:boto:Token: None 2024-04-12T04:23:02.558 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/obj23 2024-04-12T04:23:02.558 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0HbJcbbBig3Y8D4RGmDBGSYMIwM= 2024-04-12T04:23:02.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0HbJcbbBig3Y8D4RGmDBGSYMIwM='} 2024-04-12T04:23:02.564 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd9c799f217b3a69-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.564 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:02.564 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:02.564 DEBUG:boto:Method: HEAD 2024-04-12T04:23:02.564 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:02.564 DEBUG:boto:Data: 2024-04-12T04:23:02.564 DEBUG:boto:Headers: {} 2024-04-12T04:23:02.565 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.565 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.565 DEBUG:boto:Params: {} 2024-04-12T04:23:02.565 DEBUG:boto:Token: None 2024-04-12T04:23:02.565 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/ 2024-04-12T04:23:02.565 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:23:02.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:23:02.568 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000345dcb4da22625e9-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.568 DEBUG:boto:path=/dkyges-65/obj46 2024-04-12T04:23:02.569 DEBUG:boto:auth_path=/dkyges-65/obj46 2024-04-12T04:23:02.569 DEBUG:boto:Method: PUT 2024-04-12T04:23:02.569 DEBUG:boto:Path: /dkyges-65/obj46 2024-04-12T04:23:02.569 DEBUG:boto:Data: 2024-04-12T04:23:02.569 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:02.569 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.569 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.569 DEBUG:boto:Params: {} 2024-04-12T04:23:02.569 DEBUG:boto:Token: None 2024-04-12T04:23:02.570 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/obj46 2024-04-12T04:23:02.570 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:UN/y0hR19VOQlxqsyI7fLI91eww= 2024-04-12T04:23:02.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UN/y0hR19VOQlxqsyI7fLI91eww='} 2024-04-12T04:23:02.615 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000149a7f7383e148ab-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.615 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:02.615 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:02.615 DEBUG:boto:Method: HEAD 2024-04-12T04:23:02.615 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:02.615 DEBUG:boto:Data: 2024-04-12T04:23:02.615 DEBUG:boto:Headers: {} 2024-04-12T04:23:02.615 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.615 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.616 DEBUG:boto:Params: {} 2024-04-12T04:23:02.616 DEBUG:boto:Token: None 2024-04-12T04:23:02.616 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/ 2024-04-12T04:23:02.616 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:23:02.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:23:02.619 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eba0f01b77673e47-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.620 DEBUG:boto:path=/dkyges-65/obj69 2024-04-12T04:23:02.620 DEBUG:boto:auth_path=/dkyges-65/obj69 2024-04-12T04:23:02.620 DEBUG:boto:Method: PUT 2024-04-12T04:23:02.620 DEBUG:boto:Path: /dkyges-65/obj69 2024-04-12T04:23:02.620 DEBUG:boto:Data: 2024-04-12T04:23:02.620 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:02.621 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.621 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.621 DEBUG:boto:Params: {} 2024-04-12T04:23:02.621 DEBUG:boto:Token: None 2024-04-12T04:23:02.621 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/obj69 2024-04-12T04:23:02.621 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2Xp8ozrM2dR0je6C2GYmpnmRsHI= 2024-04-12T04:23:02.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2Xp8ozrM2dR0je6C2GYmpnmRsHI='} 2024-04-12T04:23:02.627 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000787e78e40c6d61be-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.628 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:02.628 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:02.628 DEBUG:boto:Method: HEAD 2024-04-12T04:23:02.628 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:02.628 DEBUG:boto:Data: 2024-04-12T04:23:02.628 DEBUG:boto:Headers: {} 2024-04-12T04:23:02.628 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.628 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.628 DEBUG:boto:Params: {} 2024-04-12T04:23:02.628 DEBUG:boto:Token: None 2024-04-12T04:23:02.629 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/ 2024-04-12T04:23:02.629 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:23:02.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:23:02.632 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000555dc94fab24931-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.632 DEBUG:boto:path=/dkyges-65/obj92 2024-04-12T04:23:02.632 DEBUG:boto:auth_path=/dkyges-65/obj92 2024-04-12T04:23:02.632 DEBUG:boto:Method: PUT 2024-04-12T04:23:02.632 DEBUG:boto:Path: /dkyges-65/obj92 2024-04-12T04:23:02.632 DEBUG:boto:Data: 2024-04-12T04:23:02.633 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:02.633 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.633 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.633 DEBUG:boto:Params: {} 2024-04-12T04:23:02.633 DEBUG:boto:Token: None 2024-04-12T04:23:02.633 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/obj92 2024-04-12T04:23:02.633 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RzjJ24UeNho3rjwkdckdiCqDkUU= 2024-04-12T04:23:02.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RzjJ24UeNho3rjwkdckdiCqDkUU='} 2024-04-12T04:23:02.683 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081ec7a2310dd87ed-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.683 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:02.683 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:02.683 DEBUG:boto:Method: HEAD 2024-04-12T04:23:02.683 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:02.683 DEBUG:boto:Data: 2024-04-12T04:23:02.683 DEBUG:boto:Headers: {} 2024-04-12T04:23:02.683 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.683 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.684 DEBUG:boto:Params: {} 2024-04-12T04:23:02.684 DEBUG:boto:Token: None 2024-04-12T04:23:02.684 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/ 2024-04-12T04:23:02.684 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:23:02.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:23:02.687 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006164995ba18fd317-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.689 DEBUG:boto:path=/dkyges-65/obj115 2024-04-12T04:23:02.689 DEBUG:boto:auth_path=/dkyges-65/obj115 2024-04-12T04:23:02.689 DEBUG:boto:Method: PUT 2024-04-12T04:23:02.689 DEBUG:boto:Path: /dkyges-65/obj115 2024-04-12T04:23:02.689 DEBUG:boto:Data: 2024-04-12T04:23:02.689 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:02.689 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.689 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.690 DEBUG:boto:Params: {} 2024-04-12T04:23:02.690 DEBUG:boto:Token: None 2024-04-12T04:23:02.690 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/obj115 2024-04-12T04:23:02.690 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OmxnuPrlYmx3ssIsTaxPHGsZahc= 2024-04-12T04:23:02.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OmxnuPrlYmx3ssIsTaxPHGsZahc='} 2024-04-12T04:23:02.697 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085d013cd2849f50e-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:23:02.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:23:02.697 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-65 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:23:02.760 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:02.760 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:02.816 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:23:02.816 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-65 2024-04-12T04:23:02.827 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 20 2024-04-12T04:23:02.888 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:02.889 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:02.889 DEBUG:boto:Method: HEAD 2024-04-12T04:23:02.889 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:02.889 DEBUG:boto:Data: 2024-04-12T04:23:02.889 DEBUG:boto:Headers: {} 2024-04-12T04:23:02.889 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.889 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.889 DEBUG:boto:Params: {} 2024-04-12T04:23:02.889 DEBUG:boto:Token: None 2024-04-12T04:23:02.890 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/ 2024-04-12T04:23:02.890 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:23:02.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:23:02.893 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a79648cc58c9b28-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.894 DEBUG:boto:path=/dkyges-65/obj29 2024-04-12T04:23:02.894 DEBUG:boto:auth_path=/dkyges-65/obj29 2024-04-12T04:23:02.894 DEBUG:boto:Method: PUT 2024-04-12T04:23:02.894 DEBUG:boto:Path: /dkyges-65/obj29 2024-04-12T04:23:02.894 DEBUG:boto:Data: 2024-04-12T04:23:02.894 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:02.894 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.894 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.895 DEBUG:boto:Params: {} 2024-04-12T04:23:02.895 DEBUG:boto:Token: None 2024-04-12T04:23:02.895 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/obj29 2024-04-12T04:23:02.895 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KYonadGBROkkQ2N2Z/e0YtrMEOQ= 2024-04-12T04:23:02.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KYonadGBROkkQ2N2Z/e0YtrMEOQ='} 2024-04-12T04:23:02.901 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b865916a46c8529e-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.901 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:02.901 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:02.901 DEBUG:boto:Method: HEAD 2024-04-12T04:23:02.902 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:02.902 DEBUG:boto:Data: 2024-04-12T04:23:02.902 DEBUG:boto:Headers: {} 2024-04-12T04:23:02.902 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.902 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.902 DEBUG:boto:Params: {} 2024-04-12T04:23:02.902 DEBUG:boto:Token: None 2024-04-12T04:23:02.902 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/ 2024-04-12T04:23:02.903 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:23:02.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:23:02.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036a5c661a06ff808-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.907 DEBUG:boto:path=/dkyges-65/obj58 2024-04-12T04:23:02.907 DEBUG:boto:auth_path=/dkyges-65/obj58 2024-04-12T04:23:02.907 DEBUG:boto:Method: PUT 2024-04-12T04:23:02.907 DEBUG:boto:Path: /dkyges-65/obj58 2024-04-12T04:23:02.907 DEBUG:boto:Data: 2024-04-12T04:23:02.907 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:02.907 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.908 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.908 DEBUG:boto:Params: {} 2024-04-12T04:23:02.908 DEBUG:boto:Token: None 2024-04-12T04:23:02.908 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/obj58 2024-04-12T04:23:02.908 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ETITXRId2uDkuG0Prz9yasDeIW0= 2024-04-12T04:23:02.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ETITXRId2uDkuG0Prz9yasDeIW0='} 2024-04-12T04:23:02.954 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2f7a570d541185e-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.955 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:02.955 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:02.955 DEBUG:boto:Method: HEAD 2024-04-12T04:23:02.955 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:02.955 DEBUG:boto:Data: 2024-04-12T04:23:02.955 DEBUG:boto:Headers: {} 2024-04-12T04:23:02.955 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.955 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.955 DEBUG:boto:Params: {} 2024-04-12T04:23:02.956 DEBUG:boto:Token: None 2024-04-12T04:23:02.956 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/ 2024-04-12T04:23:02.956 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:23:02.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:23:02.959 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000a55ddc4ed4f7e7-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.960 DEBUG:boto:path=/dkyges-65/obj87 2024-04-12T04:23:02.960 DEBUG:boto:auth_path=/dkyges-65/obj87 2024-04-12T04:23:02.960 DEBUG:boto:Method: PUT 2024-04-12T04:23:02.960 DEBUG:boto:Path: /dkyges-65/obj87 2024-04-12T04:23:02.960 DEBUG:boto:Data: 2024-04-12T04:23:02.960 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:02.960 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.960 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.960 DEBUG:boto:Params: {} 2024-04-12T04:23:02.961 DEBUG:boto:Token: None 2024-04-12T04:23:02.961 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/obj87 2024-04-12T04:23:02.961 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:XOO3f8ywOTv44/Ed8wzs4BX52sU= 2024-04-12T04:23:02.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XOO3f8ywOTv44/Ed8wzs4BX52sU='} 2024-04-12T04:23:02.967 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043de237aca5f3d1e-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.967 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:02.967 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:02.967 DEBUG:boto:Method: HEAD 2024-04-12T04:23:02.967 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:02.968 DEBUG:boto:Data: 2024-04-12T04:23:02.968 DEBUG:boto:Headers: {} 2024-04-12T04:23:02.968 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.968 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.968 DEBUG:boto:Params: {} 2024-04-12T04:23:02.968 DEBUG:boto:Token: None 2024-04-12T04:23:02.968 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/ 2024-04-12T04:23:02.968 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:23:02.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:23:02.972 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089051a60b7fb597b-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:02.972 DEBUG:boto:path=/dkyges-65/obj116 2024-04-12T04:23:02.973 DEBUG:boto:auth_path=/dkyges-65/obj116 2024-04-12T04:23:02.973 DEBUG:boto:Method: PUT 2024-04-12T04:23:02.973 DEBUG:boto:Path: /dkyges-65/obj116 2024-04-12T04:23:02.973 DEBUG:boto:Data: 2024-04-12T04:23:02.973 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:02.973 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:02.973 DEBUG:boto:Port: 8000 2024-04-12T04:23:02.973 DEBUG:boto:Params: {} 2024-04-12T04:23:02.973 DEBUG:boto:Token: None 2024-04-12T04:23:02.974 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:02 GMT /dkyges-65/obj116 2024-04-12T04:23:02.974 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+I6LUH7r9GY26hpHxgU3nE3ePEo= 2024-04-12T04:23:02.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+I6LUH7r9GY26hpHxgU3nE3ePEo='} 2024-04-12T04:23:03.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000eafc8e5b29dd38d-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.019 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:03.019 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:03.019 DEBUG:boto:Method: HEAD 2024-04-12T04:23:03.019 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:03.019 DEBUG:boto:Data: 2024-04-12T04:23:03.019 DEBUG:boto:Headers: {} 2024-04-12T04:23:03.019 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.019 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.019 DEBUG:boto:Params: {} 2024-04-12T04:23:03.020 DEBUG:boto:Token: None 2024-04-12T04:23:03.020 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/ 2024-04-12T04:23:03.020 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:23:03.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:23:03.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de8c80993a6f2da3-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.024 DEBUG:boto:path=/dkyges-65/obj145 2024-04-12T04:23:03.024 DEBUG:boto:auth_path=/dkyges-65/obj145 2024-04-12T04:23:03.024 DEBUG:boto:Method: PUT 2024-04-12T04:23:03.024 DEBUG:boto:Path: /dkyges-65/obj145 2024-04-12T04:23:03.024 DEBUG:boto:Data: 2024-04-12T04:23:03.024 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:03.024 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.024 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.024 DEBUG:boto:Params: {} 2024-04-12T04:23:03.024 DEBUG:boto:Token: None 2024-04-12T04:23:03.025 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/obj145 2024-04-12T04:23:03.025 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:931H1F/SoWE88Zx+Am5TK5cGHCs= 2024-04-12T04:23:03.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:931H1F/SoWE88Zx+Am5TK5cGHCs='} 2024-04-12T04:23:03.031 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000185f3d96c80c6d4c-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:23:03.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:23:03.032 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-65 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:23:03.095 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:03.096 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:03.153 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:23:03.153 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-65 2024-04-12T04:23:03.167 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 25 2024-04-12T04:23:03.244 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:03.244 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:03.244 DEBUG:boto:Method: HEAD 2024-04-12T04:23:03.244 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:03.244 DEBUG:boto:Data: 2024-04-12T04:23:03.245 DEBUG:boto:Headers: {} 2024-04-12T04:23:03.245 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.245 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.245 DEBUG:boto:Params: {} 2024-04-12T04:23:03.245 DEBUG:boto:Token: None 2024-04-12T04:23:03.245 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/ 2024-04-12T04:23:03.245 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:23:03.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:23:03.250 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000211fca70d7c8bb4a-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.251 DEBUG:boto:path=/dkyges-65/obj31 2024-04-12T04:23:03.251 DEBUG:boto:auth_path=/dkyges-65/obj31 2024-04-12T04:23:03.251 DEBUG:boto:Method: PUT 2024-04-12T04:23:03.251 DEBUG:boto:Path: /dkyges-65/obj31 2024-04-12T04:23:03.251 DEBUG:boto:Data: 2024-04-12T04:23:03.251 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:03.251 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.251 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.251 DEBUG:boto:Params: {} 2024-04-12T04:23:03.252 DEBUG:boto:Token: None 2024-04-12T04:23:03.252 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/obj31 2024-04-12T04:23:03.252 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NqXx1fCsyPL6EKAbKLwhzjvRmcY= 2024-04-12T04:23:03.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NqXx1fCsyPL6EKAbKLwhzjvRmcY='} 2024-04-12T04:23:03.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097bafad68a6aacae-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.299 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:03.299 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:03.299 DEBUG:boto:Method: HEAD 2024-04-12T04:23:03.299 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:03.299 DEBUG:boto:Data: 2024-04-12T04:23:03.299 DEBUG:boto:Headers: {} 2024-04-12T04:23:03.300 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.300 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.300 DEBUG:boto:Params: {} 2024-04-12T04:23:03.300 DEBUG:boto:Token: None 2024-04-12T04:23:03.300 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/ 2024-04-12T04:23:03.300 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:23:03.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:23:03.305 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006438f6101e62ffab-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.306 DEBUG:boto:path=/dkyges-65/obj62 2024-04-12T04:23:03.306 DEBUG:boto:auth_path=/dkyges-65/obj62 2024-04-12T04:23:03.306 DEBUG:boto:Method: PUT 2024-04-12T04:23:03.306 DEBUG:boto:Path: /dkyges-65/obj62 2024-04-12T04:23:03.306 DEBUG:boto:Data: 2024-04-12T04:23:03.306 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:03.306 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.307 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.307 DEBUG:boto:Params: {} 2024-04-12T04:23:03.307 DEBUG:boto:Token: None 2024-04-12T04:23:03.307 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/obj62 2024-04-12T04:23:03.307 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eIEeBn+DSXFNVvhOMwJmKxA3CAE= 2024-04-12T04:23:03.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eIEeBn+DSXFNVvhOMwJmKxA3CAE='} 2024-04-12T04:23:03.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e5ced51b44ea508-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.314 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:03.314 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:03.314 DEBUG:boto:Method: HEAD 2024-04-12T04:23:03.314 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:03.315 DEBUG:boto:Data: 2024-04-12T04:23:03.315 DEBUG:boto:Headers: {} 2024-04-12T04:23:03.315 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.315 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.315 DEBUG:boto:Params: {} 2024-04-12T04:23:03.315 DEBUG:boto:Token: None 2024-04-12T04:23:03.315 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/ 2024-04-12T04:23:03.315 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:23:03.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:23:03.319 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008673f053b1808817-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.319 DEBUG:boto:path=/dkyges-65/obj93 2024-04-12T04:23:03.319 DEBUG:boto:auth_path=/dkyges-65/obj93 2024-04-12T04:23:03.319 DEBUG:boto:Method: PUT 2024-04-12T04:23:03.319 DEBUG:boto:Path: /dkyges-65/obj93 2024-04-12T04:23:03.320 DEBUG:boto:Data: 2024-04-12T04:23:03.320 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:03.320 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.320 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.320 DEBUG:boto:Params: {} 2024-04-12T04:23:03.320 DEBUG:boto:Token: None 2024-04-12T04:23:03.320 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/obj93 2024-04-12T04:23:03.320 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eiAfpjr6RWJ2jozp2mP0zfoMDu8= 2024-04-12T04:23:03.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eiAfpjr6RWJ2jozp2mP0zfoMDu8='} 2024-04-12T04:23:03.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061ff966f9dde377e-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.363 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:03.363 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:03.363 DEBUG:boto:Method: HEAD 2024-04-12T04:23:03.363 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:03.363 DEBUG:boto:Data: 2024-04-12T04:23:03.363 DEBUG:boto:Headers: {} 2024-04-12T04:23:03.363 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.363 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.363 DEBUG:boto:Params: {} 2024-04-12T04:23:03.364 DEBUG:boto:Token: None 2024-04-12T04:23:03.364 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/ 2024-04-12T04:23:03.364 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:23:03.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:23:03.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7ee098ef9b6c0c8-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.368 DEBUG:boto:path=/dkyges-65/obj124 2024-04-12T04:23:03.369 DEBUG:boto:auth_path=/dkyges-65/obj124 2024-04-12T04:23:03.369 DEBUG:boto:Method: PUT 2024-04-12T04:23:03.369 DEBUG:boto:Path: /dkyges-65/obj124 2024-04-12T04:23:03.369 DEBUG:boto:Data: 2024-04-12T04:23:03.369 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:03.369 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.369 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.369 DEBUG:boto:Params: {} 2024-04-12T04:23:03.369 DEBUG:boto:Token: None 2024-04-12T04:23:03.369 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/obj124 2024-04-12T04:23:03.370 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nyAm+Sg30vDhYjAUlsgKpBQzWA4= 2024-04-12T04:23:03.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nyAm+Sg30vDhYjAUlsgKpBQzWA4='} 2024-04-12T04:23:03.377 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf526fb2fcffb42d-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.377 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:03.377 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:03.377 DEBUG:boto:Method: HEAD 2024-04-12T04:23:03.377 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:03.377 DEBUG:boto:Data: 2024-04-12T04:23:03.378 DEBUG:boto:Headers: {} 2024-04-12T04:23:03.378 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.378 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.378 DEBUG:boto:Params: {} 2024-04-12T04:23:03.378 DEBUG:boto:Token: None 2024-04-12T04:23:03.378 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/ 2024-04-12T04:23:03.378 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:23:03.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:23:03.381 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025bf09acab2cbb60-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.382 DEBUG:boto:path=/dkyges-65/obj155 2024-04-12T04:23:03.382 DEBUG:boto:auth_path=/dkyges-65/obj155 2024-04-12T04:23:03.382 DEBUG:boto:Method: PUT 2024-04-12T04:23:03.382 DEBUG:boto:Path: /dkyges-65/obj155 2024-04-12T04:23:03.382 DEBUG:boto:Data: 2024-04-12T04:23:03.382 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:03.382 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.382 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.382 DEBUG:boto:Params: {} 2024-04-12T04:23:03.383 DEBUG:boto:Token: None 2024-04-12T04:23:03.383 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/obj155 2024-04-12T04:23:03.383 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RRo1rvw5T9IgCifo1Df/ytTniIo= 2024-04-12T04:23:03.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RRo1rvw5T9IgCifo1Df/ytTniIo='} 2024-04-12T04:23:03.427 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ca3bccf30f43883-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:23:03.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:23:03.427 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-65 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:23:03.491 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:03.491 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:03.550 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:23:03.550 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-65 2024-04-12T04:23:03.564 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 30 2024-04-12T04:23:03.641 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:03.641 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:03.642 DEBUG:boto:Method: HEAD 2024-04-12T04:23:03.642 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:03.642 DEBUG:boto:Data: 2024-04-12T04:23:03.642 DEBUG:boto:Headers: {} 2024-04-12T04:23:03.642 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.642 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.642 DEBUG:boto:Params: {} 2024-04-12T04:23:03.642 DEBUG:boto:Token: None 2024-04-12T04:23:03.642 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/ 2024-04-12T04:23:03.643 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:23:03.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:23:03.647 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d2413d18292a0a8-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.647 DEBUG:boto:path=/dkyges-65/obj37 2024-04-12T04:23:03.648 DEBUG:boto:auth_path=/dkyges-65/obj37 2024-04-12T04:23:03.648 DEBUG:boto:Method: PUT 2024-04-12T04:23:03.648 DEBUG:boto:Path: /dkyges-65/obj37 2024-04-12T04:23:03.648 DEBUG:boto:Data: 2024-04-12T04:23:03.648 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:03.648 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.648 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.648 DEBUG:boto:Params: {} 2024-04-12T04:23:03.648 DEBUG:boto:Token: None 2024-04-12T04:23:03.649 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/obj37 2024-04-12T04:23:03.649 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cIOpdB4ukxj/JU7ubn898MEHmZU= 2024-04-12T04:23:03.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cIOpdB4ukxj/JU7ubn898MEHmZU='} 2024-04-12T04:23:03.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d5cee8cbaef31ac5-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.655 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:03.655 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:03.655 DEBUG:boto:Method: HEAD 2024-04-12T04:23:03.655 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:03.655 DEBUG:boto:Data: 2024-04-12T04:23:03.655 DEBUG:boto:Headers: {} 2024-04-12T04:23:03.655 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.655 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.656 DEBUG:boto:Params: {} 2024-04-12T04:23:03.656 DEBUG:boto:Token: None 2024-04-12T04:23:03.656 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/ 2024-04-12T04:23:03.656 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:23:03.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:23:03.660 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000285c4dc4667ab4ff-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.660 DEBUG:boto:path=/dkyges-65/obj74 2024-04-12T04:23:03.660 DEBUG:boto:auth_path=/dkyges-65/obj74 2024-04-12T04:23:03.661 DEBUG:boto:Method: PUT 2024-04-12T04:23:03.661 DEBUG:boto:Path: /dkyges-65/obj74 2024-04-12T04:23:03.661 DEBUG:boto:Data: 2024-04-12T04:23:03.661 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:03.661 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.661 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.661 DEBUG:boto:Params: {} 2024-04-12T04:23:03.661 DEBUG:boto:Token: None 2024-04-12T04:23:03.661 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/obj74 2024-04-12T04:23:03.662 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4/rDqyX1IYUDDrB6U4UuRjP9hc0= 2024-04-12T04:23:03.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4/rDqyX1IYUDDrB6U4UuRjP9hc0='} 2024-04-12T04:23:03.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe31ccb3f6291ee7-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.707 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:03.707 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:03.707 DEBUG:boto:Method: HEAD 2024-04-12T04:23:03.707 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:03.707 DEBUG:boto:Data: 2024-04-12T04:23:03.707 DEBUG:boto:Headers: {} 2024-04-12T04:23:03.708 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.708 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.708 DEBUG:boto:Params: {} 2024-04-12T04:23:03.708 DEBUG:boto:Token: None 2024-04-12T04:23:03.708 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/ 2024-04-12T04:23:03.708 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:23:03.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:23:03.713 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4a730ca264f3510-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.714 DEBUG:boto:path=/dkyges-65/obj111 2024-04-12T04:23:03.714 DEBUG:boto:auth_path=/dkyges-65/obj111 2024-04-12T04:23:03.714 DEBUG:boto:Method: PUT 2024-04-12T04:23:03.714 DEBUG:boto:Path: /dkyges-65/obj111 2024-04-12T04:23:03.714 DEBUG:boto:Data: 2024-04-12T04:23:03.714 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:03.714 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.714 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.714 DEBUG:boto:Params: {} 2024-04-12T04:23:03.714 DEBUG:boto:Token: None 2024-04-12T04:23:03.715 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/obj111 2024-04-12T04:23:03.715 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ElsVJ3u4wc9eO1BYeFw0kWTt8Sk= 2024-04-12T04:23:03.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ElsVJ3u4wc9eO1BYeFw0kWTt8Sk='} 2024-04-12T04:23:03.722 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000722996a2481c6fd1-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.722 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:03.722 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:03.722 DEBUG:boto:Method: HEAD 2024-04-12T04:23:03.722 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:03.722 DEBUG:boto:Data: 2024-04-12T04:23:03.722 DEBUG:boto:Headers: {} 2024-04-12T04:23:03.722 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.723 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.723 DEBUG:boto:Params: {} 2024-04-12T04:23:03.723 DEBUG:boto:Token: None 2024-04-12T04:23:03.723 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/ 2024-04-12T04:23:03.723 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:23:03.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:23:03.727 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b74ec14649c71e4-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.727 DEBUG:boto:path=/dkyges-65/obj148 2024-04-12T04:23:03.727 DEBUG:boto:auth_path=/dkyges-65/obj148 2024-04-12T04:23:03.728 DEBUG:boto:Method: PUT 2024-04-12T04:23:03.728 DEBUG:boto:Path: /dkyges-65/obj148 2024-04-12T04:23:03.728 DEBUG:boto:Data: 2024-04-12T04:23:03.728 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:03.728 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.728 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.728 DEBUG:boto:Params: {} 2024-04-12T04:23:03.728 DEBUG:boto:Token: None 2024-04-12T04:23:03.728 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/obj148 2024-04-12T04:23:03.728 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MrUw6dyUQFgLIolBi/bH7nX1awc= 2024-04-12T04:23:03.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MrUw6dyUQFgLIolBi/bH7nX1awc='} 2024-04-12T04:23:03.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033b7c61e7d8198a9-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.775 DEBUG:boto:path=/dkyges-65/ 2024-04-12T04:23:03.775 DEBUG:boto:auth_path=/dkyges-65/ 2024-04-12T04:23:03.775 DEBUG:boto:Method: HEAD 2024-04-12T04:23:03.775 DEBUG:boto:Path: /dkyges-65/ 2024-04-12T04:23:03.775 DEBUG:boto:Data: 2024-04-12T04:23:03.775 DEBUG:boto:Headers: {} 2024-04-12T04:23:03.775 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.775 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.776 DEBUG:boto:Params: {} 2024-04-12T04:23:03.776 DEBUG:boto:Token: None 2024-04-12T04:23:03.776 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/ 2024-04-12T04:23:03.776 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:23:03.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:23:03.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d696c3412d99b80-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.780 DEBUG:boto:path=/dkyges-65/obj185 2024-04-12T04:23:03.780 DEBUG:boto:auth_path=/dkyges-65/obj185 2024-04-12T04:23:03.781 DEBUG:boto:Method: PUT 2024-04-12T04:23:03.781 DEBUG:boto:Path: /dkyges-65/obj185 2024-04-12T04:23:03.781 DEBUG:boto:Data: 2024-04-12T04:23:03.781 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:23:03.781 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:23:03.781 DEBUG:boto:Port: 8000 2024-04-12T04:23:03.781 DEBUG:boto:Params: {} 2024-04-12T04:23:03.781 DEBUG:boto:Token: None 2024-04-12T04:23:03.781 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:23:03 GMT /dkyges-65/obj185 2024-04-12T04:23:03.781 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I/x7RqIA1dMEqLRcddsqisshSYc= 2024-04-12T04:23:03.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I/x7RqIA1dMEqLRcddsqisshSYc='} 2024-04-12T04:23:03.788 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c56e2c487296f33-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:23:03.788 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-65 2024-04-12T04:23:03.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-65', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:23:03.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-65', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:23:03.788 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-65 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:23:03.856 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:03.856 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:03.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:23:03.913+0000 7f83aaad6780 1 waiting to reach incremental sync.. 2024-04-12T04:23:08.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:23:08.915+0000 7f83aaad6780 1 waiting to reach incremental sync.. 2024-04-12T04:23:13.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:23:13.915+0000 7f83aaad6780 1 waiting to reach incremental sync.. 2024-04-12T04:23:18.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:23:18.919+0000 7f83aaad6780 1 bucket sync caught up with source: 2024-04-12T04:23:18.908 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , 00000000002.203.6, , , , , , 00000000002.100.6, , , , , 00000000002.182.6, , , , , , , , , , , , , , , 00000000001.119.6, , , , 00000000001.112.6] 2024-04-12T04:23:18.908 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , 00000000002.203.6, , , , , , 00000000002.100.6, , , , , 00000000002.182.6, , , , , , , , , , , , , , , 00000000001.119.6, , , , 00000000001.112.6] 2024-04-12T04:23:18.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:23:18.919+0000 7f83aaad6780 0 bucket checkpoint complete 2024-04-12T04:23:18.925 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-65 2024-04-12T04:23:18.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-65', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:23:18.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-65', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:23:18.926 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-65 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:23:18.991 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:23:18.992 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:23:19.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:23:19.044+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:23:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:23:24.044+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:23:29.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:23:29.045+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:23:34.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:23:34.045+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:23:39.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:23:39.045+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:23:44.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:23:44.046+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:23:49.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:23:49.047+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:23:54.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:23:54.047+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:23:59.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:23:59.048+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:24:04.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:24:04.049+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:24:09.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:24:09.049+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:24:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:24:14.050+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:24:19.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:24:19.051+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:24:24.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:24:24.052+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:24:29.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:24:29.052+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:24:34.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:24:34.052+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:24:39.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:24:39.053+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:24:44.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:24:44.054+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:24:49.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:24:49.054+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:24:54.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:24:54.055+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:24:59.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:24:59.056+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:25:04.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:25:04.056+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:25:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:25:09.057+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:25:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:25:14.057+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:25:19.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:25:19.058+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:25:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:25:24.059+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:25:29.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:25:29.059+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:25:33.981 INFO:tasks.rgw.c1.client.0.smithi043.stdout:2024-04-12T04:25:33.989+0000 7f1e272bd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:25:34.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:25:34.060+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:25:39.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:25:39.062+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:25:44.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:25:44.061+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:25:49.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:25:49.062+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:25:54.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:25:54.063+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:25:59.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:25:59.063+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:26:04.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:26:04.064+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:26:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:26:09.065+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:26:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:26:14.065+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:26:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:26:19.066+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:26:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:26:24.067+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:26:29.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:26:29.068+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:26:34.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:26:34.068+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:26:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:26:39.069+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:26:44.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:26:44.070+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:26:49.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:26:49.071+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:26:54.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:26:54.072+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:26:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:26:59.072+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:27:04.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:27:04.072+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:27:09.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:27:09.073+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:27:14.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:27:14.074+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:27:19.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:27:19.074+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:27:24.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:27:24.075+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:27:29.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:27:29.076+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:27:34.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:27:34.076+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:27:39.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:27:39.077+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:27:44.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:27:44.077+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:27:49.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:27:49.078+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:27:54.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:27:54.078+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:27:59.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:27:59.079+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:28:04.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:04.080+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:28:09.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:09.082+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:28:14.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:14.082+0000 7ff35ff2a780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-12T04:28:14.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:14.082+0000 7ff35ff2a780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:28:14.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:14.082+0000 7ff35ff2a780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:28:14.095 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-12T04:28:14.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2024-04-12T04:28:14.099 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:14.100 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:14.100 DEBUG:boto:Method: PUT 2024-04-12T04:28:14.100 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:14.100 DEBUG:boto:Data: 2024-04-12T04:28:14.100 DEBUG:boto:Headers: {} 2024-04-12T04:28:14.100 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:14.100 DEBUG:boto:Port: 8000 2024-04-12T04:28:14.100 DEBUG:boto:Params: {} 2024-04-12T04:28:14.101 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:28:14.101 DEBUG:boto:Token: None 2024-04-12T04:28:14.101 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:28:14 GMT /dkyges-66/ 2024-04-12T04:28:14.102 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uCHE4TL56iUfx6MKliM7UpBIrEY= 2024-04-12T04:28:14.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uCHE4TL56iUfx6MKliM7UpBIrEY=', 'Content-Length': '0'} 2024-04-12T04:28:14.152 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dc48d1bdf0644d5d-006618b85e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:14.153 DEBUG:rgw_multi.tests:created bucket=dkyges-66 2024-04-12T04:28:14.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:14.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:14.154 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:14.259 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:28:14.275 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:28:14.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:28:14.275 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:28:14.275 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:28:14.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:28:14.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:28:14.276 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:28:14.276 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:28:14.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:28:14.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:28:14.276 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:28:14.276 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:28:14.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:28:14.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:28:14.276 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896094.156795_758.1", 2024-04-12T04:28:14.276 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:28:14.156795Z" 2024-04-12T04:28:14.277 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:28:14.277 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:28:14.278 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896094.156795_758.1'} 2024-04-12T04:28:14.278 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:28:14.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:14.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:28:14.278 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:28:14.333 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:14.333 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:14.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:14.373+0000 7f8673989780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:14.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:14.374+0000 7f8673989780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:28:14.384 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:28:14.384 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:28:14.385 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:28:14.385 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:28:14.385 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:28:14.385 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:28:14.385 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:28:14.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:28:14.385 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:28:14.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:28:14.385 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:28:14.385 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:28:14.385 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:28:14.386 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:28:14.386 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:28:14.386 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:28:14.386 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:28:14.386 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:28:14.386 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:28:14.386 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:28:14.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:28:14.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:28:14.386 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:28:14.386 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:28:14.387 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:28:14.387 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:28:14.387 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:28:14.387 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:28:14.387 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:28:14.387 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:28:14.387 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:28:14.387 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:28:14.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:28:14.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:28:14.388 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:28:14.388 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:28:14.388 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:28:14.388 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:28:14.388 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:28:14.388 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:28:14.388 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:28:14.388 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:28:14.388 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:28:14.388 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:28:14.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:28:14.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:28:14.389 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:28:14.389 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:28:14.389 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:28:14.389 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896094.156795_758.1", 2024-04-12T04:28:14.389 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:28:14.389 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:28:14.389 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:28:14.389 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:14.156795Z", 2024-04-12T04:28:14.389 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:28:14.389 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:28:14.390 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:28:14.390 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:28:14.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:28:14.390 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:28:14.390 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:28:14.390 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:28:14.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:28:14.390 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:28:14Z" 2024-04-12T04:28:14.390 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:28:14.391 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896094.156795_758.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:14.156795Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:28:14Z" } 2024-04-12T04:28:14.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896094.156795_758.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:14.156795Z', 'realm_epoch': 2}}] 2024-04-12T04:28:14.392 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896094.156795_758.1'} 2024-04-12T04:28:14.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896094.156795_758.1'} 2024-04-12T04:28:14.392 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:28:14.392 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:14.392 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:14.392 DEBUG:boto:Method: HEAD 2024-04-12T04:28:14.392 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:14.392 DEBUG:boto:Data: 2024-04-12T04:28:14.392 DEBUG:boto:Headers: {} 2024-04-12T04:28:14.392 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:14.393 DEBUG:boto:Port: 8000 2024-04-12T04:28:14.393 DEBUG:boto:Params: {} 2024-04-12T04:28:14.393 DEBUG:boto:Token: None 2024-04-12T04:28:14.393 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:14 GMT /dkyges-66/ 2024-04-12T04:28:14.393 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM= 2024-04-12T04:28:14.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM=', 'Content-Length': '0'} 2024-04-12T04:28:14.396 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecdbb167e34d4527-006618b85e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:14.396 DEBUG:boto:path=/dkyges-66/obj11 2024-04-12T04:28:14.396 DEBUG:boto:auth_path=/dkyges-66/obj11 2024-04-12T04:28:14.396 DEBUG:boto:Method: PUT 2024-04-12T04:28:14.396 DEBUG:boto:Path: /dkyges-66/obj11 2024-04-12T04:28:14.397 DEBUG:boto:Data: 2024-04-12T04:28:14.397 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:14.397 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:14.397 DEBUG:boto:Port: 8000 2024-04-12T04:28:14.397 DEBUG:boto:Params: {} 2024-04-12T04:28:14.397 DEBUG:boto:Token: None 2024-04-12T04:28:14.397 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:14 GMT /dkyges-66/obj11 2024-04-12T04:28:14.397 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:FtQXxFKbmW9ETaMrdGaFpjmg4to= 2024-04-12T04:28:14.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FtQXxFKbmW9ETaMrdGaFpjmg4to='} 2024-04-12T04:28:14.404 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c798cd0c631c0856-006618b85e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:14.405 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:14.405 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:14.405 DEBUG:boto:Method: HEAD 2024-04-12T04:28:14.405 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:14.405 DEBUG:boto:Data: 2024-04-12T04:28:14.405 DEBUG:boto:Headers: {} 2024-04-12T04:28:14.405 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:14.405 DEBUG:boto:Port: 8000 2024-04-12T04:28:14.405 DEBUG:boto:Params: {} 2024-04-12T04:28:14.405 DEBUG:boto:Token: None 2024-04-12T04:28:14.406 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:14 GMT /dkyges-66/ 2024-04-12T04:28:14.406 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM= 2024-04-12T04:28:14.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM=', 'Content-Length': '0'} 2024-04-12T04:28:14.408 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea63812dabe16235-006618b85e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:14.408 DEBUG:boto:path=/dkyges-66/obj22 2024-04-12T04:28:14.409 DEBUG:boto:auth_path=/dkyges-66/obj22 2024-04-12T04:28:14.409 DEBUG:boto:Method: PUT 2024-04-12T04:28:14.409 DEBUG:boto:Path: /dkyges-66/obj22 2024-04-12T04:28:14.409 DEBUG:boto:Data: 2024-04-12T04:28:14.409 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:14.409 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:14.409 DEBUG:boto:Port: 8000 2024-04-12T04:28:14.409 DEBUG:boto:Params: {} 2024-04-12T04:28:14.409 DEBUG:boto:Token: None 2024-04-12T04:28:14.409 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:14 GMT /dkyges-66/obj22 2024-04-12T04:28:14.410 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OmBPDwlNo5wXcuv305+WkMDac3w= 2024-04-12T04:28:14.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OmBPDwlNo5wXcuv305+WkMDac3w='} 2024-04-12T04:28:14.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e76753a143ad4f08-006618b85e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:14.414 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:14.414 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:14.414 DEBUG:boto:Method: HEAD 2024-04-12T04:28:14.414 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:14.415 DEBUG:boto:Data: 2024-04-12T04:28:14.415 DEBUG:boto:Headers: {} 2024-04-12T04:28:14.415 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:14.415 DEBUG:boto:Port: 8000 2024-04-12T04:28:14.415 DEBUG:boto:Params: {} 2024-04-12T04:28:14.415 DEBUG:boto:Token: None 2024-04-12T04:28:14.415 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:14 GMT /dkyges-66/ 2024-04-12T04:28:14.415 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM= 2024-04-12T04:28:14.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM=', 'Content-Length': '0'} 2024-04-12T04:28:14.418 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a617b3cc23de1486-006618b85e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:14.418 DEBUG:boto:path=/dkyges-66/obj33 2024-04-12T04:28:14.418 DEBUG:boto:auth_path=/dkyges-66/obj33 2024-04-12T04:28:14.418 DEBUG:boto:Method: PUT 2024-04-12T04:28:14.418 DEBUG:boto:Path: /dkyges-66/obj33 2024-04-12T04:28:14.418 DEBUG:boto:Data: 2024-04-12T04:28:14.418 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:14.419 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:14.419 DEBUG:boto:Port: 8000 2024-04-12T04:28:14.419 DEBUG:boto:Params: {} 2024-04-12T04:28:14.419 DEBUG:boto:Token: None 2024-04-12T04:28:14.419 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:14 GMT /dkyges-66/obj33 2024-04-12T04:28:14.419 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:m5gbihvPkV+XMSCV+pTylSEFuAo= 2024-04-12T04:28:14.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m5gbihvPkV+XMSCV+pTylSEFuAo='} 2024-04-12T04:28:14.425 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3a0fd2dae1a2cae-006618b85e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:14.425 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:14.425 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:14.425 DEBUG:boto:Method: HEAD 2024-04-12T04:28:14.425 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:14.425 DEBUG:boto:Data: 2024-04-12T04:28:14.425 DEBUG:boto:Headers: {} 2024-04-12T04:28:14.425 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:14.426 DEBUG:boto:Port: 8000 2024-04-12T04:28:14.426 DEBUG:boto:Params: {} 2024-04-12T04:28:14.426 DEBUG:boto:Token: None 2024-04-12T04:28:14.426 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:14 GMT /dkyges-66/ 2024-04-12T04:28:14.426 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM= 2024-04-12T04:28:14.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM=', 'Content-Length': '0'} 2024-04-12T04:28:14.428 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d10d33712f271895-006618b85e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:14.429 DEBUG:boto:path=/dkyges-66/obj44 2024-04-12T04:28:14.429 DEBUG:boto:auth_path=/dkyges-66/obj44 2024-04-12T04:28:14.429 DEBUG:boto:Method: PUT 2024-04-12T04:28:14.429 DEBUG:boto:Path: /dkyges-66/obj44 2024-04-12T04:28:14.429 DEBUG:boto:Data: 2024-04-12T04:28:14.429 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:14.429 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:14.429 DEBUG:boto:Port: 8000 2024-04-12T04:28:14.429 DEBUG:boto:Params: {} 2024-04-12T04:28:14.429 DEBUG:boto:Token: None 2024-04-12T04:28:14.430 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:14 GMT /dkyges-66/obj44 2024-04-12T04:28:14.430 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:y5CNBw+Gx4BUkLe8Aprw/5NpopM= 2024-04-12T04:28:14.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y5CNBw+Gx4BUkLe8Aprw/5NpopM='} 2024-04-12T04:28:14.474 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d357be6dfdd2bf11-006618b85e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:14.475 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:14.475 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:14.475 DEBUG:boto:Method: HEAD 2024-04-12T04:28:14.475 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:14.475 DEBUG:boto:Data: 2024-04-12T04:28:14.475 DEBUG:boto:Headers: {} 2024-04-12T04:28:14.475 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:14.475 DEBUG:boto:Port: 8000 2024-04-12T04:28:14.475 DEBUG:boto:Params: {} 2024-04-12T04:28:14.475 DEBUG:boto:Token: None 2024-04-12T04:28:14.476 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:14 GMT /dkyges-66/ 2024-04-12T04:28:14.476 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM= 2024-04-12T04:28:14.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM=', 'Content-Length': '0'} 2024-04-12T04:28:14.478 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009845d9bec2a015f2-006618b85e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:14.478 DEBUG:boto:path=/dkyges-66/obj55 2024-04-12T04:28:14.479 DEBUG:boto:auth_path=/dkyges-66/obj55 2024-04-12T04:28:14.479 DEBUG:boto:Method: PUT 2024-04-12T04:28:14.479 DEBUG:boto:Path: /dkyges-66/obj55 2024-04-12T04:28:14.479 DEBUG:boto:Data: 2024-04-12T04:28:14.479 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:14.479 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:14.479 DEBUG:boto:Port: 8000 2024-04-12T04:28:14.479 DEBUG:boto:Params: {} 2024-04-12T04:28:14.479 DEBUG:boto:Token: None 2024-04-12T04:28:14.479 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:14 GMT /dkyges-66/obj55 2024-04-12T04:28:14.480 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zybU6MoaPHY3dZwh0lKx7UOtpuI= 2024-04-12T04:28:14.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zybU6MoaPHY3dZwh0lKx7UOtpuI='} 2024-04-12T04:28:14.484 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001273841af1fe5561-006618b85e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:14.484 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-66 2024-04-12T04:28:14.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:28:14.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:28:14.485 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-66 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:28:14.550 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:14.551 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:14.598+0000 7fc61d715780 1 waiting to reach incremental sync.. 2024-04-12T04:28:19.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:19.599+0000 7fc61d715780 1 bucket sync caught up with source: 2024-04-12T04:28:19.594 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000001.131.6, 00000000002.104.6, 00000000002.82.6] 2024-04-12T04:28:19.595 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000001.131.6, 00000000002.104.6, 00000000002.82.6] 2024-04-12T04:28:19.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:19.599+0000 7fc61d715780 0 bucket checkpoint complete 2024-04-12T04:28:19.614 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=dkyges-66 2024-04-12T04:28:19.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-66', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:28:19.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-66', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:28:19.614 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-66 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:28:19.680 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:19.680 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:19.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:19.728+0000 7f344e100780 1 waiting to reach incremental sync.. 2024-04-12T04:28:24.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:24.728+0000 7f344e100780 1 waiting to reach incremental sync.. 2024-04-12T04:28:29.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:29.729+0000 7f344e100780 1 waiting to reach incremental sync.. 2024-04-12T04:28:34.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:34.729+0000 7f344e100780 1 waiting to reach incremental sync.. 2024-04-12T04:28:39.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:39.731+0000 7f344e100780 1 bucket sync caught up with source: 2024-04-12T04:28:39.727 INFO:teuthology.orchestra.run.smithi043.stderr: local status: [, , , , , , , , 00000000001.1993.6, 00000000002.2069.6, 00000000002.1098.6] 2024-04-12T04:28:39.727 INFO:teuthology.orchestra.run.smithi043.stderr: remote markers: [, , , , , , , , 00000000001.1993.6, 00000000002.2069.6, 00000000002.1098.6] 2024-04-12T04:28:39.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:39.731+0000 7f344e100780 0 bucket checkpoint complete 2024-04-12T04:28:39.745 INFO:tasks.rgw_multi.zone_rados:comparing bucket=dkyges-66 zones={a2, a1} 2024-04-12T04:28:39.746 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:39.746 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:39.746 DEBUG:boto:Method: HEAD 2024-04-12T04:28:39.746 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:39.746 DEBUG:boto:Data: 2024-04-12T04:28:39.746 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.746 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:39.746 DEBUG:boto:Port: 8001 2024-04-12T04:28:39.746 DEBUG:boto:Params: {} 2024-04-12T04:28:39.747 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:28:39.747 DEBUG:boto:Token: None 2024-04-12T04:28:39.747 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/ 2024-04-12T04:28:39.747 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lhzWmSEIFCkgawr6QCKWJGgN768= 2024-04-12T04:28:39.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lhzWmSEIFCkgawr6QCKWJGgN768=', 'Content-Length': '0'} 2024-04-12T04:28:39.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000709e98e136150fcc-006618b877-216b-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.780 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:39.780 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:39.780 DEBUG:boto:Method: HEAD 2024-04-12T04:28:39.781 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:39.781 DEBUG:boto:Data: 2024-04-12T04:28:39.781 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.781 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:39.781 DEBUG:boto:Port: 8000 2024-04-12T04:28:39.781 DEBUG:boto:Params: {} 2024-04-12T04:28:39.781 DEBUG:boto:Token: None 2024-04-12T04:28:39.781 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/ 2024-04-12T04:28:39.781 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lhzWmSEIFCkgawr6QCKWJGgN768= 2024-04-12T04:28:39.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lhzWmSEIFCkgawr6QCKWJGgN768=', 'Content-Length': '0'} 2024-04-12T04:28:39.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3a15561db574e4e-006618b877-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.784 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-04-12T04:28:39.784 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:39.784 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:39.785 DEBUG:boto:path=/dkyges-66/?versions&max-keys=999 2024-04-12T04:28:39.785 DEBUG:boto:auth_path=/dkyges-66/?versions&max-keys=999 2024-04-12T04:28:39.785 DEBUG:boto:Method: GET 2024-04-12T04:28:39.785 DEBUG:boto:Path: /dkyges-66/?versions&max-keys=999 2024-04-12T04:28:39.785 DEBUG:boto:Data: 2024-04-12T04:28:39.785 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.785 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:39.785 DEBUG:boto:Port: 8001 2024-04-12T04:28:39.785 DEBUG:boto:Params: {} 2024-04-12T04:28:39.785 DEBUG:boto:Token: None 2024-04-12T04:28:39.785 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/?versions 2024-04-12T04:28:39.786 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wQWmXJpssIgVlJIu+qPIp7bBC+w= 2024-04-12T04:28:39.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wQWmXJpssIgVlJIu+qPIp7bBC+w=', 'Content-Length': '0'} 2024-04-12T04:28:39.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005086fda2dad11350-006618b877-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.789 DEBUG:boto:b'dkyges-66999falseobj11nulltrue2024-04-12T04:28:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:28:14.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:28:14.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:28:14.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:28:14.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:39.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:28:39.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:28:39.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:28:39.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:28:39.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:28:39.790 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-04-12T04:28:39.790 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:39.790 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:39.790 DEBUG:boto:path=/dkyges-66/?versions&max-keys=999 2024-04-12T04:28:39.791 DEBUG:boto:auth_path=/dkyges-66/?versions&max-keys=999 2024-04-12T04:28:39.791 DEBUG:boto:Method: GET 2024-04-12T04:28:39.791 DEBUG:boto:Path: /dkyges-66/?versions&max-keys=999 2024-04-12T04:28:39.791 DEBUG:boto:Data: 2024-04-12T04:28:39.791 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.791 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:39.791 DEBUG:boto:Port: 8000 2024-04-12T04:28:39.791 DEBUG:boto:Params: {} 2024-04-12T04:28:39.791 DEBUG:boto:Token: None 2024-04-12T04:28:39.791 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/?versions 2024-04-12T04:28:39.791 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wQWmXJpssIgVlJIu+qPIp7bBC+w= 2024-04-12T04:28:39.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wQWmXJpssIgVlJIu+qPIp7bBC+w=', 'Content-Length': '0'} 2024-04-12T04:28:39.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f740bd0d998b582-006618b877-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.794 DEBUG:boto:b'dkyges-66999falseobj11nulltrue2024-04-12T04:28:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:28:14.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:28:14.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:28:14.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:28:14.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:39.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2024-04-12T04:28:39.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2024-04-12T04:28:39.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2024-04-12T04:28:39.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2024-04-12T04:28:39.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2024-04-12T04:28:39.795 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:39.795 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:39.795 DEBUG:boto:path=/dkyges-66/?versions&max-keys=999 2024-04-12T04:28:39.796 DEBUG:boto:auth_path=/dkyges-66/?versions&max-keys=999 2024-04-12T04:28:39.796 DEBUG:boto:Method: GET 2024-04-12T04:28:39.796 DEBUG:boto:Path: /dkyges-66/?versions&max-keys=999 2024-04-12T04:28:39.796 DEBUG:boto:Data: 2024-04-12T04:28:39.796 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.796 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:39.796 DEBUG:boto:Port: 8001 2024-04-12T04:28:39.796 DEBUG:boto:Params: {} 2024-04-12T04:28:39.796 DEBUG:boto:Token: None 2024-04-12T04:28:39.796 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/?versions 2024-04-12T04:28:39.796 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wQWmXJpssIgVlJIu+qPIp7bBC+w= 2024-04-12T04:28:39.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wQWmXJpssIgVlJIu+qPIp7bBC+w=', 'Content-Length': '0'} 2024-04-12T04:28:39.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039f426f230821857-006618b877-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.799 DEBUG:boto:b'dkyges-66999falseobj11nulltrue2024-04-12T04:28:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:28:14.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:28:14.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:28:14.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:28:14.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:39.800 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:39.800 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:39.800 DEBUG:boto:path=/dkyges-66/?versions&max-keys=999 2024-04-12T04:28:39.800 DEBUG:boto:auth_path=/dkyges-66/?versions&max-keys=999 2024-04-12T04:28:39.800 DEBUG:boto:Method: GET 2024-04-12T04:28:39.800 DEBUG:boto:Path: /dkyges-66/?versions&max-keys=999 2024-04-12T04:28:39.800 DEBUG:boto:Data: 2024-04-12T04:28:39.800 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.800 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:39.800 DEBUG:boto:Port: 8000 2024-04-12T04:28:39.800 DEBUG:boto:Params: {} 2024-04-12T04:28:39.801 DEBUG:boto:Token: None 2024-04-12T04:28:39.801 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/?versions 2024-04-12T04:28:39.801 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:wQWmXJpssIgVlJIu+qPIp7bBC+w= 2024-04-12T04:28:39.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wQWmXJpssIgVlJIu+qPIp7bBC+w=', 'Content-Length': '0'} 2024-04-12T04:28:39.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2cfcce1e5ec5877-006618b877-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.803 DEBUG:boto:b'dkyges-66999falseobj11nulltrue2024-04-12T04:28:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:28:14.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:28:14.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:28:14.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:28:14.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:39.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:28:39.804 DEBUG:boto:path=/dkyges-66/obj11 2024-04-12T04:28:39.804 DEBUG:boto:auth_path=/dkyges-66/obj11 2024-04-12T04:28:39.804 DEBUG:boto:path=/dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.804 DEBUG:boto:auth_path=/dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.804 DEBUG:boto:Method: GET 2024-04-12T04:28:39.805 DEBUG:boto:Path: /dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.805 DEBUG:boto:Data: 2024-04-12T04:28:39.805 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.805 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:39.805 DEBUG:boto:Port: 8001 2024-04-12T04:28:39.805 DEBUG:boto:Params: {} 2024-04-12T04:28:39.805 DEBUG:boto:Token: None 2024-04-12T04:28:39.805 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.805 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NPg1Jgi2fRRc556r/yX2tkMSbPI= 2024-04-12T04:28:39.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NPg1Jgi2fRRc556r/yX2tkMSbPI=', 'Content-Length': '0'} 2024-04-12T04:28:39.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a30cc8e32c80ef0d-006618b877-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.809 DEBUG:boto:path=/dkyges-66/obj11 2024-04-12T04:28:39.810 DEBUG:boto:auth_path=/dkyges-66/obj11 2024-04-12T04:28:39.810 DEBUG:boto:path=/dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.810 DEBUG:boto:auth_path=/dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.810 DEBUG:boto:Method: GET 2024-04-12T04:28:39.810 DEBUG:boto:Path: /dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.810 DEBUG:boto:Data: 2024-04-12T04:28:39.810 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.810 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:39.810 DEBUG:boto:Port: 8000 2024-04-12T04:28:39.810 DEBUG:boto:Params: {} 2024-04-12T04:28:39.811 DEBUG:boto:Token: None 2024-04-12T04:28:39.811 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.811 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NPg1Jgi2fRRc556r/yX2tkMSbPI= 2024-04-12T04:28:39.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NPg1Jgi2fRRc556r/yX2tkMSbPI=', 'Content-Length': '0'} 2024-04-12T04:28:39.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009668e115a5817689-006618b877-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.855 DEBUG:boto:path=/dkyges-66/obj11 2024-04-12T04:28:39.855 DEBUG:boto:auth_path=/dkyges-66/obj11 2024-04-12T04:28:39.855 DEBUG:boto:path=/dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.855 DEBUG:boto:auth_path=/dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.855 DEBUG:boto:Method: HEAD 2024-04-12T04:28:39.855 DEBUG:boto:Path: /dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.855 DEBUG:boto:Data: 2024-04-12T04:28:39.855 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.855 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:39.855 DEBUG:boto:Port: 8001 2024-04-12T04:28:39.855 DEBUG:boto:Params: {} 2024-04-12T04:28:39.856 DEBUG:boto:Token: None 2024-04-12T04:28:39.856 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.856 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GHJhXGOiwjJb2c9/dNB9J/PL4mU= 2024-04-12T04:28:39.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GHJhXGOiwjJb2c9/dNB9J/PL4mU=', 'Content-Length': '0'} 2024-04-12T04:28:39.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee1c3786c495a6c0-006618b877-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.860 DEBUG:boto:path=/dkyges-66/obj11 2024-04-12T04:28:39.860 DEBUG:boto:auth_path=/dkyges-66/obj11 2024-04-12T04:28:39.861 DEBUG:boto:path=/dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.861 DEBUG:boto:auth_path=/dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.861 DEBUG:boto:Method: HEAD 2024-04-12T04:28:39.861 DEBUG:boto:Path: /dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.861 DEBUG:boto:Data: 2024-04-12T04:28:39.861 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.861 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:39.861 DEBUG:boto:Port: 8000 2024-04-12T04:28:39.861 DEBUG:boto:Params: {} 2024-04-12T04:28:39.861 DEBUG:boto:Token: None 2024-04-12T04:28:39.862 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.862 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GHJhXGOiwjJb2c9/dNB9J/PL4mU= 2024-04-12T04:28:39.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GHJhXGOiwjJb2c9/dNB9J/PL4mU=', 'Content-Length': '0'} 2024-04-12T04:28:39.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f14c34d93f165d04-006618b877-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:28:39.864 DEBUG:boto:path=/dkyges-66/obj11 2024-04-12T04:28:39.864 DEBUG:boto:auth_path=/dkyges-66/obj11 2024-04-12T04:28:39.864 DEBUG:boto:path=/dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.864 DEBUG:boto:auth_path=/dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.864 DEBUG:boto:Method: GET 2024-04-12T04:28:39.865 DEBUG:boto:Path: /dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.865 DEBUG:boto:Data: 2024-04-12T04:28:39.865 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.865 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:39.865 DEBUG:boto:Port: 8001 2024-04-12T04:28:39.865 DEBUG:boto:Params: {} 2024-04-12T04:28:39.865 DEBUG:boto:Token: None 2024-04-12T04:28:39.865 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.865 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NPg1Jgi2fRRc556r/yX2tkMSbPI= 2024-04-12T04:28:39.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NPg1Jgi2fRRc556r/yX2tkMSbPI=', 'Content-Length': '0'} 2024-04-12T04:28:39.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b40ae8a120346767-006618b877-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.867 DEBUG:boto:path=/dkyges-66/obj11 2024-04-12T04:28:39.867 DEBUG:boto:auth_path=/dkyges-66/obj11 2024-04-12T04:28:39.868 DEBUG:boto:path=/dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.868 DEBUG:boto:auth_path=/dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.868 DEBUG:boto:Method: GET 2024-04-12T04:28:39.868 DEBUG:boto:Path: /dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.868 DEBUG:boto:Data: 2024-04-12T04:28:39.868 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.868 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:39.868 DEBUG:boto:Port: 8000 2024-04-12T04:28:39.868 DEBUG:boto:Params: {} 2024-04-12T04:28:39.868 DEBUG:boto:Token: None 2024-04-12T04:28:39.868 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/obj11?versionId=null 2024-04-12T04:28:39.868 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NPg1Jgi2fRRc556r/yX2tkMSbPI= 2024-04-12T04:28:39.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NPg1Jgi2fRRc556r/yX2tkMSbPI=', 'Content-Length': '0'} 2024-04-12T04:28:39.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cca3867ad53a05f-006618b877-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.871 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:39.871 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:39.871 DEBUG:boto:path=/dkyges-66/?prefix=obj11 2024-04-12T04:28:39.871 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj11 2024-04-12T04:28:39.871 DEBUG:boto:Method: GET 2024-04-12T04:28:39.871 DEBUG:boto:Path: /dkyges-66/?prefix=obj11 2024-04-12T04:28:39.872 DEBUG:boto:Data: 2024-04-12T04:28:39.872 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.872 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:39.872 DEBUG:boto:Port: 8001 2024-04-12T04:28:39.872 DEBUG:boto:Params: {} 2024-04-12T04:28:39.872 DEBUG:boto:Token: None 2024-04-12T04:28:39.872 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/ 2024-04-12T04:28:39.872 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j/bmImaUFIGMWqpWdIw5Pbtgb/I= 2024-04-12T04:28:39.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j/bmImaUFIGMWqpWdIw5Pbtgb/I=', 'Content-Length': '0'} 2024-04-12T04:28:39.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aee07690e991397b-006618b877-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.918 DEBUG:boto:b'dkyges-66obj111000falseobj112024-04-12T04:28:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:39.919 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:39.919 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:39.919 DEBUG:boto:path=/dkyges-66/?prefix=obj11 2024-04-12T04:28:39.919 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj11 2024-04-12T04:28:39.919 DEBUG:boto:Method: GET 2024-04-12T04:28:39.919 DEBUG:boto:Path: /dkyges-66/?prefix=obj11 2024-04-12T04:28:39.920 DEBUG:boto:Data: 2024-04-12T04:28:39.920 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.920 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:39.920 DEBUG:boto:Port: 8000 2024-04-12T04:28:39.920 DEBUG:boto:Params: {} 2024-04-12T04:28:39.920 DEBUG:boto:Token: None 2024-04-12T04:28:39.920 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/ 2024-04-12T04:28:39.920 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j/bmImaUFIGMWqpWdIw5Pbtgb/I= 2024-04-12T04:28:39.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j/bmImaUFIGMWqpWdIw5Pbtgb/I=', 'Content-Length': '0'} 2024-04-12T04:28:39.923 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf5320157775abdd-006618b877-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.924 DEBUG:boto:b'dkyges-66obj111000falseobj112024-04-12T04:28:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:39.925 DEBUG:boto:path=/dkyges-66/obj11 2024-04-12T04:28:39.925 DEBUG:boto:auth_path=/dkyges-66/obj11 2024-04-12T04:28:39.925 DEBUG:boto:Method: HEAD 2024-04-12T04:28:39.925 DEBUG:boto:Path: /dkyges-66/obj11 2024-04-12T04:28:39.925 DEBUG:boto:Data: 2024-04-12T04:28:39.925 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.925 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:39.925 DEBUG:boto:Port: 8001 2024-04-12T04:28:39.925 DEBUG:boto:Params: {} 2024-04-12T04:28:39.925 DEBUG:boto:Token: None 2024-04-12T04:28:39.926 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/obj11 2024-04-12T04:28:39.926 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lmsIehaeuhHx8AcBEkpZDxDgOw0= 2024-04-12T04:28:39.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lmsIehaeuhHx8AcBEkpZDxDgOw0=', 'Content-Length': '0'} 2024-04-12T04:28:39.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f783d85a03d66809-006618b877-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.928 DEBUG:boto:path=/dkyges-66/obj11 2024-04-12T04:28:39.928 DEBUG:boto:auth_path=/dkyges-66/obj11 2024-04-12T04:28:39.928 DEBUG:boto:Method: HEAD 2024-04-12T04:28:39.928 DEBUG:boto:Path: /dkyges-66/obj11 2024-04-12T04:28:39.928 DEBUG:boto:Data: 2024-04-12T04:28:39.928 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.928 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:39.928 DEBUG:boto:Port: 8000 2024-04-12T04:28:39.929 DEBUG:boto:Params: {} 2024-04-12T04:28:39.929 DEBUG:boto:Token: None 2024-04-12T04:28:39.929 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/obj11 2024-04-12T04:28:39.929 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lmsIehaeuhHx8AcBEkpZDxDgOw0= 2024-04-12T04:28:39.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lmsIehaeuhHx8AcBEkpZDxDgOw0=', 'Content-Length': '0'} 2024-04-12T04:28:39.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007edfaca122005591-006618b877-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2024-04-12T04:28:39.931 DEBUG:boto:path=/dkyges-66/obj11 2024-04-12T04:28:39.931 DEBUG:boto:auth_path=/dkyges-66/obj11 2024-04-12T04:28:39.931 DEBUG:boto:Method: GET 2024-04-12T04:28:39.931 DEBUG:boto:Path: /dkyges-66/obj11 2024-04-12T04:28:39.932 DEBUG:boto:Data: 2024-04-12T04:28:39.932 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.932 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:39.932 DEBUG:boto:Port: 8001 2024-04-12T04:28:39.932 DEBUG:boto:Params: {} 2024-04-12T04:28:39.932 DEBUG:boto:Token: None 2024-04-12T04:28:39.932 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/obj11 2024-04-12T04:28:39.932 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HpKSrRafOHMh9/dME30iHQ+KdC8= 2024-04-12T04:28:39.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HpKSrRafOHMh9/dME30iHQ+KdC8=', 'Content-Length': '0'} 2024-04-12T04:28:39.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0a1838e35047369-006618b877-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.935 DEBUG:boto:path=/dkyges-66/obj11 2024-04-12T04:28:39.935 DEBUG:boto:auth_path=/dkyges-66/obj11 2024-04-12T04:28:39.935 DEBUG:boto:Method: GET 2024-04-12T04:28:39.935 DEBUG:boto:Path: /dkyges-66/obj11 2024-04-12T04:28:39.935 DEBUG:boto:Data: 2024-04-12T04:28:39.935 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.935 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:39.935 DEBUG:boto:Port: 8000 2024-04-12T04:28:39.935 DEBUG:boto:Params: {} 2024-04-12T04:28:39.937 DEBUG:boto:Token: None 2024-04-12T04:28:39.937 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/obj11 2024-04-12T04:28:39.937 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HpKSrRafOHMh9/dME30iHQ+KdC8= 2024-04-12T04:28:39.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HpKSrRafOHMh9/dME30iHQ+KdC8=', 'Content-Length': '0'} 2024-04-12T04:28:39.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001afd1ee753fbc2c-006618b877-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.983 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:39.984 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:39.984 DEBUG:boto:path=/dkyges-66/?prefix=obj11 2024-04-12T04:28:39.984 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj11 2024-04-12T04:28:39.984 DEBUG:boto:Method: GET 2024-04-12T04:28:39.984 DEBUG:boto:Path: /dkyges-66/?prefix=obj11 2024-04-12T04:28:39.984 DEBUG:boto:Data: 2024-04-12T04:28:39.984 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.984 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:39.984 DEBUG:boto:Port: 8001 2024-04-12T04:28:39.984 DEBUG:boto:Params: {} 2024-04-12T04:28:39.984 DEBUG:boto:Token: None 2024-04-12T04:28:39.985 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/ 2024-04-12T04:28:39.985 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j/bmImaUFIGMWqpWdIw5Pbtgb/I= 2024-04-12T04:28:39.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j/bmImaUFIGMWqpWdIw5Pbtgb/I=', 'Content-Length': '0'} 2024-04-12T04:28:39.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000312694bc734429b3-006618b877-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.988 DEBUG:boto:b'dkyges-66obj111000falseobj112024-04-12T04:28:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:39.988 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:39.988 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:39.989 DEBUG:boto:path=/dkyges-66/?prefix=obj11 2024-04-12T04:28:39.989 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj11 2024-04-12T04:28:39.989 DEBUG:boto:Method: GET 2024-04-12T04:28:39.989 DEBUG:boto:Path: /dkyges-66/?prefix=obj11 2024-04-12T04:28:39.989 DEBUG:boto:Data: 2024-04-12T04:28:39.989 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.989 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:39.989 DEBUG:boto:Port: 8000 2024-04-12T04:28:39.989 DEBUG:boto:Params: {} 2024-04-12T04:28:39.989 DEBUG:boto:Token: None 2024-04-12T04:28:39.989 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/ 2024-04-12T04:28:39.989 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:j/bmImaUFIGMWqpWdIw5Pbtgb/I= 2024-04-12T04:28:39.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j/bmImaUFIGMWqpWdIw5Pbtgb/I=', 'Content-Length': '0'} 2024-04-12T04:28:39.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017e5991ce5b8738b-006618b877-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.992 DEBUG:boto:b'dkyges-66obj111000falseobj112024-04-12T04:28:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:39.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:28:39.993 DEBUG:boto:path=/dkyges-66/obj22 2024-04-12T04:28:39.993 DEBUG:boto:auth_path=/dkyges-66/obj22 2024-04-12T04:28:39.993 DEBUG:boto:path=/dkyges-66/obj22?versionId=null 2024-04-12T04:28:39.993 DEBUG:boto:auth_path=/dkyges-66/obj22?versionId=null 2024-04-12T04:28:39.993 DEBUG:boto:Method: GET 2024-04-12T04:28:39.993 DEBUG:boto:Path: /dkyges-66/obj22?versionId=null 2024-04-12T04:28:39.993 DEBUG:boto:Data: 2024-04-12T04:28:39.993 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.993 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:39.993 DEBUG:boto:Port: 8001 2024-04-12T04:28:39.993 DEBUG:boto:Params: {} 2024-04-12T04:28:39.993 DEBUG:boto:Token: None 2024-04-12T04:28:39.994 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/obj22?versionId=null 2024-04-12T04:28:39.994 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZADtSajCSKrGf3YLLD9nlqhh8VI= 2024-04-12T04:28:39.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZADtSajCSKrGf3YLLD9nlqhh8VI=', 'Content-Length': '0'} 2024-04-12T04:28:39.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090a964117f2e417c-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.996 DEBUG:boto:path=/dkyges-66/obj22 2024-04-12T04:28:39.996 DEBUG:boto:auth_path=/dkyges-66/obj22 2024-04-12T04:28:39.996 DEBUG:boto:path=/dkyges-66/obj22?versionId=null 2024-04-12T04:28:39.996 DEBUG:boto:auth_path=/dkyges-66/obj22?versionId=null 2024-04-12T04:28:39.996 DEBUG:boto:Method: GET 2024-04-12T04:28:39.996 DEBUG:boto:Path: /dkyges-66/obj22?versionId=null 2024-04-12T04:28:39.996 DEBUG:boto:Data: 2024-04-12T04:28:39.997 DEBUG:boto:Headers: {} 2024-04-12T04:28:39.997 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:39.997 DEBUG:boto:Port: 8000 2024-04-12T04:28:39.997 DEBUG:boto:Params: {} 2024-04-12T04:28:39.997 DEBUG:boto:Token: None 2024-04-12T04:28:39.997 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:39 GMT /dkyges-66/obj22?versionId=null 2024-04-12T04:28:39.997 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZADtSajCSKrGf3YLLD9nlqhh8VI= 2024-04-12T04:28:39.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZADtSajCSKrGf3YLLD9nlqhh8VI=', 'Content-Length': '0'} 2024-04-12T04:28:39.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6ecee70de842c9c-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:39.999 DEBUG:boto:path=/dkyges-66/obj22 2024-04-12T04:28:40.000 DEBUG:boto:auth_path=/dkyges-66/obj22 2024-04-12T04:28:40.000 DEBUG:boto:path=/dkyges-66/obj22?versionId=null 2024-04-12T04:28:40.000 DEBUG:boto:auth_path=/dkyges-66/obj22?versionId=null 2024-04-12T04:28:40.000 DEBUG:boto:Method: HEAD 2024-04-12T04:28:40.000 DEBUG:boto:Path: /dkyges-66/obj22?versionId=null 2024-04-12T04:28:40.000 DEBUG:boto:Data: 2024-04-12T04:28:40.000 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.000 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.000 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.000 DEBUG:boto:Params: {} 2024-04-12T04:28:40.000 DEBUG:boto:Token: None 2024-04-12T04:28:40.001 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj22?versionId=null 2024-04-12T04:28:40.001 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0lwK25D/DYWVVUqM+wD88WNpcvI= 2024-04-12T04:28:40.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0lwK25D/DYWVVUqM+wD88WNpcvI=', 'Content-Length': '0'} 2024-04-12T04:28:40.002 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c8d62acba5e82f3-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.003 DEBUG:boto:path=/dkyges-66/obj22 2024-04-12T04:28:40.003 DEBUG:boto:auth_path=/dkyges-66/obj22 2024-04-12T04:28:40.003 DEBUG:boto:path=/dkyges-66/obj22?versionId=null 2024-04-12T04:28:40.003 DEBUG:boto:auth_path=/dkyges-66/obj22?versionId=null 2024-04-12T04:28:40.003 DEBUG:boto:Method: HEAD 2024-04-12T04:28:40.003 DEBUG:boto:Path: /dkyges-66/obj22?versionId=null 2024-04-12T04:28:40.003 DEBUG:boto:Data: 2024-04-12T04:28:40.003 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.003 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.003 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.004 DEBUG:boto:Params: {} 2024-04-12T04:28:40.004 DEBUG:boto:Token: None 2024-04-12T04:28:40.004 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj22?versionId=null 2024-04-12T04:28:40.004 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0lwK25D/DYWVVUqM+wD88WNpcvI= 2024-04-12T04:28:40.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0lwK25D/DYWVVUqM+wD88WNpcvI=', 'Content-Length': '0'} 2024-04-12T04:28:40.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fb138a9f93a135b-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:28:40.007 DEBUG:boto:path=/dkyges-66/obj22 2024-04-12T04:28:40.007 DEBUG:boto:auth_path=/dkyges-66/obj22 2024-04-12T04:28:40.007 DEBUG:boto:path=/dkyges-66/obj22?versionId=null 2024-04-12T04:28:40.007 DEBUG:boto:auth_path=/dkyges-66/obj22?versionId=null 2024-04-12T04:28:40.007 DEBUG:boto:Method: GET 2024-04-12T04:28:40.007 DEBUG:boto:Path: /dkyges-66/obj22?versionId=null 2024-04-12T04:28:40.007 DEBUG:boto:Data: 2024-04-12T04:28:40.007 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.007 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.007 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.007 DEBUG:boto:Params: {} 2024-04-12T04:28:40.008 DEBUG:boto:Token: None 2024-04-12T04:28:40.008 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj22?versionId=null 2024-04-12T04:28:40.008 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bvi0FEn8D7DcD3eD+Kec2BredTg= 2024-04-12T04:28:40.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bvi0FEn8D7DcD3eD+Kec2BredTg=', 'Content-Length': '0'} 2024-04-12T04:28:40.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc82e588f94e4327-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.050 DEBUG:boto:path=/dkyges-66/obj22 2024-04-12T04:28:40.051 DEBUG:boto:auth_path=/dkyges-66/obj22 2024-04-12T04:28:40.051 DEBUG:boto:path=/dkyges-66/obj22?versionId=null 2024-04-12T04:28:40.051 DEBUG:boto:auth_path=/dkyges-66/obj22?versionId=null 2024-04-12T04:28:40.051 DEBUG:boto:Method: GET 2024-04-12T04:28:40.051 DEBUG:boto:Path: /dkyges-66/obj22?versionId=null 2024-04-12T04:28:40.051 DEBUG:boto:Data: 2024-04-12T04:28:40.051 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.051 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.051 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.051 DEBUG:boto:Params: {} 2024-04-12T04:28:40.052 DEBUG:boto:Token: None 2024-04-12T04:28:40.052 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj22?versionId=null 2024-04-12T04:28:40.052 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bvi0FEn8D7DcD3eD+Kec2BredTg= 2024-04-12T04:28:40.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bvi0FEn8D7DcD3eD+Kec2BredTg=', 'Content-Length': '0'} 2024-04-12T04:28:40.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000668b817a8796ce56-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.055 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:40.055 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:40.055 DEBUG:boto:path=/dkyges-66/?prefix=obj22 2024-04-12T04:28:40.056 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj22 2024-04-12T04:28:40.056 DEBUG:boto:Method: GET 2024-04-12T04:28:40.056 DEBUG:boto:Path: /dkyges-66/?prefix=obj22 2024-04-12T04:28:40.056 DEBUG:boto:Data: 2024-04-12T04:28:40.056 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.056 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.056 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.056 DEBUG:boto:Params: {} 2024-04-12T04:28:40.057 DEBUG:boto:Token: None 2024-04-12T04:28:40.057 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/ 2024-04-12T04:28:40.057 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:28:40.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:28:40.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e33feb8c660675b3-006618b878-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.061 DEBUG:boto:b'dkyges-66obj221000falseobj222024-04-12T04:28:14.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:40.062 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:40.062 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:40.062 DEBUG:boto:path=/dkyges-66/?prefix=obj22 2024-04-12T04:28:40.062 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj22 2024-04-12T04:28:40.062 DEBUG:boto:Method: GET 2024-04-12T04:28:40.062 DEBUG:boto:Path: /dkyges-66/?prefix=obj22 2024-04-12T04:28:40.063 DEBUG:boto:Data: 2024-04-12T04:28:40.063 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.063 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.063 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.063 DEBUG:boto:Params: {} 2024-04-12T04:28:40.063 DEBUG:boto:Token: None 2024-04-12T04:28:40.064 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/ 2024-04-12T04:28:40.064 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:28:40.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:28:40.067 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e86016a6578566e-006618b878-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.069 DEBUG:boto:b'dkyges-66obj221000falseobj222024-04-12T04:28:14.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:40.069 DEBUG:boto:path=/dkyges-66/obj22 2024-04-12T04:28:40.069 DEBUG:boto:auth_path=/dkyges-66/obj22 2024-04-12T04:28:40.070 DEBUG:boto:Method: HEAD 2024-04-12T04:28:40.070 DEBUG:boto:Path: /dkyges-66/obj22 2024-04-12T04:28:40.070 DEBUG:boto:Data: 2024-04-12T04:28:40.070 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.070 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.070 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.070 DEBUG:boto:Params: {} 2024-04-12T04:28:40.071 DEBUG:boto:Token: None 2024-04-12T04:28:40.071 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj22 2024-04-12T04:28:40.071 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KiLEcrJ3OuTTA/5IMHY7sdYkTuo= 2024-04-12T04:28:40.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KiLEcrJ3OuTTA/5IMHY7sdYkTuo=', 'Content-Length': '0'} 2024-04-12T04:28:40.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021080a13297b47ab-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.073 DEBUG:boto:path=/dkyges-66/obj22 2024-04-12T04:28:40.074 DEBUG:boto:auth_path=/dkyges-66/obj22 2024-04-12T04:28:40.074 DEBUG:boto:Method: HEAD 2024-04-12T04:28:40.074 DEBUG:boto:Path: /dkyges-66/obj22 2024-04-12T04:28:40.074 DEBUG:boto:Data: 2024-04-12T04:28:40.074 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.074 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.074 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.074 DEBUG:boto:Params: {} 2024-04-12T04:28:40.075 DEBUG:boto:Token: None 2024-04-12T04:28:40.075 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj22 2024-04-12T04:28:40.075 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:KiLEcrJ3OuTTA/5IMHY7sdYkTuo= 2024-04-12T04:28:40.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KiLEcrJ3OuTTA/5IMHY7sdYkTuo=', 'Content-Length': '0'} 2024-04-12T04:28:40.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a3f623965982a78-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2024-04-12T04:28:40.078 DEBUG:boto:path=/dkyges-66/obj22 2024-04-12T04:28:40.078 DEBUG:boto:auth_path=/dkyges-66/obj22 2024-04-12T04:28:40.078 DEBUG:boto:Method: GET 2024-04-12T04:28:40.078 DEBUG:boto:Path: /dkyges-66/obj22 2024-04-12T04:28:40.078 DEBUG:boto:Data: 2024-04-12T04:28:40.078 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.078 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.079 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.079 DEBUG:boto:Params: {} 2024-04-12T04:28:40.079 DEBUG:boto:Token: None 2024-04-12T04:28:40.079 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj22 2024-04-12T04:28:40.079 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:asOF0smMwHECrbBoKHsVrcZwKlY= 2024-04-12T04:28:40.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:asOF0smMwHECrbBoKHsVrcZwKlY=', 'Content-Length': '0'} 2024-04-12T04:28:40.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6d237d5cb591c84-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.082 DEBUG:boto:path=/dkyges-66/obj22 2024-04-12T04:28:40.082 DEBUG:boto:auth_path=/dkyges-66/obj22 2024-04-12T04:28:40.082 DEBUG:boto:Method: GET 2024-04-12T04:28:40.083 DEBUG:boto:Path: /dkyges-66/obj22 2024-04-12T04:28:40.083 DEBUG:boto:Data: 2024-04-12T04:28:40.083 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.083 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.083 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.083 DEBUG:boto:Params: {} 2024-04-12T04:28:40.083 DEBUG:boto:Token: None 2024-04-12T04:28:40.084 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj22 2024-04-12T04:28:40.084 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:asOF0smMwHECrbBoKHsVrcZwKlY= 2024-04-12T04:28:40.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:asOF0smMwHECrbBoKHsVrcZwKlY=', 'Content-Length': '0'} 2024-04-12T04:28:40.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b422afbed1f8502c-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.087 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:40.087 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:40.087 DEBUG:boto:path=/dkyges-66/?prefix=obj22 2024-04-12T04:28:40.087 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj22 2024-04-12T04:28:40.088 DEBUG:boto:Method: GET 2024-04-12T04:28:40.088 DEBUG:boto:Path: /dkyges-66/?prefix=obj22 2024-04-12T04:28:40.088 DEBUG:boto:Data: 2024-04-12T04:28:40.088 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.088 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.089 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.089 DEBUG:boto:Params: {} 2024-04-12T04:28:40.089 DEBUG:boto:Token: None 2024-04-12T04:28:40.089 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/ 2024-04-12T04:28:40.090 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:28:40.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:28:40.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e25c9b0bd21f6e25-006618b878-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.094 DEBUG:boto:b'dkyges-66obj221000falseobj222024-04-12T04:28:14.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:40.095 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:40.095 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:40.095 DEBUG:boto:path=/dkyges-66/?prefix=obj22 2024-04-12T04:28:40.095 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj22 2024-04-12T04:28:40.095 DEBUG:boto:Method: GET 2024-04-12T04:28:40.095 DEBUG:boto:Path: /dkyges-66/?prefix=obj22 2024-04-12T04:28:40.095 DEBUG:boto:Data: 2024-04-12T04:28:40.096 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.096 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.096 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.096 DEBUG:boto:Params: {} 2024-04-12T04:28:40.096 DEBUG:boto:Token: None 2024-04-12T04:28:40.097 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/ 2024-04-12T04:28:40.097 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:28:40.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:28:40.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f0846d0b4a3da15-006618b878-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.142 DEBUG:boto:b'dkyges-66obj221000falseobj222024-04-12T04:28:14.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:40.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:28:40.143 DEBUG:boto:path=/dkyges-66/obj33 2024-04-12T04:28:40.143 DEBUG:boto:auth_path=/dkyges-66/obj33 2024-04-12T04:28:40.143 DEBUG:boto:path=/dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.143 DEBUG:boto:auth_path=/dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.143 DEBUG:boto:Method: GET 2024-04-12T04:28:40.143 DEBUG:boto:Path: /dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.144 DEBUG:boto:Data: 2024-04-12T04:28:40.144 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.144 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.144 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.144 DEBUG:boto:Params: {} 2024-04-12T04:28:40.144 DEBUG:boto:Token: None 2024-04-12T04:28:40.144 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.144 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:10mJOSet6hYVuVJZyhyRsszZJnU= 2024-04-12T04:28:40.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:10mJOSet6hYVuVJZyhyRsszZJnU=', 'Content-Length': '0'} 2024-04-12T04:28:40.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d07506caa450fc58-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.147 DEBUG:boto:path=/dkyges-66/obj33 2024-04-12T04:28:40.147 DEBUG:boto:auth_path=/dkyges-66/obj33 2024-04-12T04:28:40.147 DEBUG:boto:path=/dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.147 DEBUG:boto:auth_path=/dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.147 DEBUG:boto:Method: GET 2024-04-12T04:28:40.147 DEBUG:boto:Path: /dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.147 DEBUG:boto:Data: 2024-04-12T04:28:40.147 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.148 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.148 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.148 DEBUG:boto:Params: {} 2024-04-12T04:28:40.148 DEBUG:boto:Token: None 2024-04-12T04:28:40.148 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.148 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:10mJOSet6hYVuVJZyhyRsszZJnU= 2024-04-12T04:28:40.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:10mJOSet6hYVuVJZyhyRsszZJnU=', 'Content-Length': '0'} 2024-04-12T04:28:40.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0bc25221d66886c-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.150 DEBUG:boto:path=/dkyges-66/obj33 2024-04-12T04:28:40.151 DEBUG:boto:auth_path=/dkyges-66/obj33 2024-04-12T04:28:40.151 DEBUG:boto:path=/dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.151 DEBUG:boto:auth_path=/dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.151 DEBUG:boto:Method: HEAD 2024-04-12T04:28:40.151 DEBUG:boto:Path: /dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.151 DEBUG:boto:Data: 2024-04-12T04:28:40.151 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.151 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.151 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.151 DEBUG:boto:Params: {} 2024-04-12T04:28:40.152 DEBUG:boto:Token: None 2024-04-12T04:28:40.152 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.152 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l6nnomiSPRNdVMqaPZUP8zRYxyM= 2024-04-12T04:28:40.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l6nnomiSPRNdVMqaPZUP8zRYxyM=', 'Content-Length': '0'} 2024-04-12T04:28:40.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b88f456ea9f4a996-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.154 DEBUG:boto:path=/dkyges-66/obj33 2024-04-12T04:28:40.154 DEBUG:boto:auth_path=/dkyges-66/obj33 2024-04-12T04:28:40.154 DEBUG:boto:path=/dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.154 DEBUG:boto:auth_path=/dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.154 DEBUG:boto:Method: HEAD 2024-04-12T04:28:40.154 DEBUG:boto:Path: /dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.154 DEBUG:boto:Data: 2024-04-12T04:28:40.154 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.155 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.155 DEBUG:boto:Params: {} 2024-04-12T04:28:40.155 DEBUG:boto:Token: None 2024-04-12T04:28:40.155 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.155 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:l6nnomiSPRNdVMqaPZUP8zRYxyM= 2024-04-12T04:28:40.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l6nnomiSPRNdVMqaPZUP8zRYxyM=', 'Content-Length': '0'} 2024-04-12T04:28:40.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a87855ec46df063-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:28:40.157 DEBUG:boto:path=/dkyges-66/obj33 2024-04-12T04:28:40.157 DEBUG:boto:auth_path=/dkyges-66/obj33 2024-04-12T04:28:40.158 DEBUG:boto:path=/dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.158 DEBUG:boto:auth_path=/dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.158 DEBUG:boto:Method: GET 2024-04-12T04:28:40.158 DEBUG:boto:Path: /dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.158 DEBUG:boto:Data: 2024-04-12T04:28:40.158 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.158 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.158 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.158 DEBUG:boto:Params: {} 2024-04-12T04:28:40.159 DEBUG:boto:Token: None 2024-04-12T04:28:40.159 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.159 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:10mJOSet6hYVuVJZyhyRsszZJnU= 2024-04-12T04:28:40.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:10mJOSet6hYVuVJZyhyRsszZJnU=', 'Content-Length': '0'} 2024-04-12T04:28:40.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000464ce7672ca5e77a-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.203 DEBUG:boto:path=/dkyges-66/obj33 2024-04-12T04:28:40.203 DEBUG:boto:auth_path=/dkyges-66/obj33 2024-04-12T04:28:40.203 DEBUG:boto:path=/dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.203 DEBUG:boto:auth_path=/dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.203 DEBUG:boto:Method: GET 2024-04-12T04:28:40.203 DEBUG:boto:Path: /dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.203 DEBUG:boto:Data: 2024-04-12T04:28:40.203 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.203 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.203 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.203 DEBUG:boto:Params: {} 2024-04-12T04:28:40.204 DEBUG:boto:Token: None 2024-04-12T04:28:40.204 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj33?versionId=null 2024-04-12T04:28:40.204 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:10mJOSet6hYVuVJZyhyRsszZJnU= 2024-04-12T04:28:40.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:10mJOSet6hYVuVJZyhyRsszZJnU=', 'Content-Length': '0'} 2024-04-12T04:28:40.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dc3f94323356142-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.206 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:40.206 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:40.206 DEBUG:boto:path=/dkyges-66/?prefix=obj33 2024-04-12T04:28:40.207 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj33 2024-04-12T04:28:40.207 DEBUG:boto:Method: GET 2024-04-12T04:28:40.207 DEBUG:boto:Path: /dkyges-66/?prefix=obj33 2024-04-12T04:28:40.207 DEBUG:boto:Data: 2024-04-12T04:28:40.207 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.207 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.207 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.207 DEBUG:boto:Params: {} 2024-04-12T04:28:40.207 DEBUG:boto:Token: None 2024-04-12T04:28:40.207 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/ 2024-04-12T04:28:40.207 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:28:40.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:28:40.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072a11a1e92f0e623-006618b878-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.211 DEBUG:boto:b'dkyges-66obj331000falseobj332024-04-12T04:28:14.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:40.211 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:40.211 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:40.211 DEBUG:boto:path=/dkyges-66/?prefix=obj33 2024-04-12T04:28:40.211 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj33 2024-04-12T04:28:40.211 DEBUG:boto:Method: GET 2024-04-12T04:28:40.211 DEBUG:boto:Path: /dkyges-66/?prefix=obj33 2024-04-12T04:28:40.211 DEBUG:boto:Data: 2024-04-12T04:28:40.211 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.212 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.212 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.212 DEBUG:boto:Params: {} 2024-04-12T04:28:40.212 DEBUG:boto:Token: None 2024-04-12T04:28:40.212 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/ 2024-04-12T04:28:40.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:28:40.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:28:40.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012369c9db5ee325b-006618b878-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.258 DEBUG:boto:b'dkyges-66obj331000falseobj332024-04-12T04:28:14.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:40.259 DEBUG:boto:path=/dkyges-66/obj33 2024-04-12T04:28:40.259 DEBUG:boto:auth_path=/dkyges-66/obj33 2024-04-12T04:28:40.259 DEBUG:boto:Method: HEAD 2024-04-12T04:28:40.259 DEBUG:boto:Path: /dkyges-66/obj33 2024-04-12T04:28:40.259 DEBUG:boto:Data: 2024-04-12T04:28:40.259 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.259 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.260 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.260 DEBUG:boto:Params: {} 2024-04-12T04:28:40.260 DEBUG:boto:Token: None 2024-04-12T04:28:40.260 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj33 2024-04-12T04:28:40.260 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0PfHDXFOD02WCdsHrCwtodfbscM= 2024-04-12T04:28:40.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0PfHDXFOD02WCdsHrCwtodfbscM=', 'Content-Length': '0'} 2024-04-12T04:28:40.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b844bda67108399-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.262 DEBUG:boto:path=/dkyges-66/obj33 2024-04-12T04:28:40.262 DEBUG:boto:auth_path=/dkyges-66/obj33 2024-04-12T04:28:40.263 DEBUG:boto:Method: HEAD 2024-04-12T04:28:40.263 DEBUG:boto:Path: /dkyges-66/obj33 2024-04-12T04:28:40.263 DEBUG:boto:Data: 2024-04-12T04:28:40.263 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.263 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.263 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.263 DEBUG:boto:Params: {} 2024-04-12T04:28:40.263 DEBUG:boto:Token: None 2024-04-12T04:28:40.263 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj33 2024-04-12T04:28:40.263 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0PfHDXFOD02WCdsHrCwtodfbscM= 2024-04-12T04:28:40.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0PfHDXFOD02WCdsHrCwtodfbscM=', 'Content-Length': '0'} 2024-04-12T04:28:40.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008050243219b25b9c-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2024-04-12T04:28:40.266 DEBUG:boto:path=/dkyges-66/obj33 2024-04-12T04:28:40.266 DEBUG:boto:auth_path=/dkyges-66/obj33 2024-04-12T04:28:40.267 DEBUG:boto:Method: GET 2024-04-12T04:28:40.267 DEBUG:boto:Path: /dkyges-66/obj33 2024-04-12T04:28:40.267 DEBUG:boto:Data: 2024-04-12T04:28:40.267 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.267 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.267 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.267 DEBUG:boto:Params: {} 2024-04-12T04:28:40.267 DEBUG:boto:Token: None 2024-04-12T04:28:40.267 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj33 2024-04-12T04:28:40.267 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WJMQrV9PlNsdwAx4+sLm67kaGto= 2024-04-12T04:28:40.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WJMQrV9PlNsdwAx4+sLm67kaGto=', 'Content-Length': '0'} 2024-04-12T04:28:40.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042b58155a9a76e3c-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.270 DEBUG:boto:path=/dkyges-66/obj33 2024-04-12T04:28:40.270 DEBUG:boto:auth_path=/dkyges-66/obj33 2024-04-12T04:28:40.270 DEBUG:boto:Method: GET 2024-04-12T04:28:40.270 DEBUG:boto:Path: /dkyges-66/obj33 2024-04-12T04:28:40.270 DEBUG:boto:Data: 2024-04-12T04:28:40.270 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.271 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.271 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.271 DEBUG:boto:Params: {} 2024-04-12T04:28:40.271 DEBUG:boto:Token: None 2024-04-12T04:28:40.271 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj33 2024-04-12T04:28:40.271 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WJMQrV9PlNsdwAx4+sLm67kaGto= 2024-04-12T04:28:40.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WJMQrV9PlNsdwAx4+sLm67kaGto=', 'Content-Length': '0'} 2024-04-12T04:28:40.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012ae135d7a3ac656-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.273 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:40.273 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:40.273 DEBUG:boto:path=/dkyges-66/?prefix=obj33 2024-04-12T04:28:40.274 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj33 2024-04-12T04:28:40.274 DEBUG:boto:Method: GET 2024-04-12T04:28:40.274 DEBUG:boto:Path: /dkyges-66/?prefix=obj33 2024-04-12T04:28:40.274 DEBUG:boto:Data: 2024-04-12T04:28:40.274 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.274 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.274 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.274 DEBUG:boto:Params: {} 2024-04-12T04:28:40.274 DEBUG:boto:Token: None 2024-04-12T04:28:40.274 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/ 2024-04-12T04:28:40.275 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:28:40.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:28:40.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000383b63bca535489c-006618b878-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.278 DEBUG:boto:b'dkyges-66obj331000falseobj332024-04-12T04:28:14.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:40.278 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:40.278 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:40.278 DEBUG:boto:path=/dkyges-66/?prefix=obj33 2024-04-12T04:28:40.278 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj33 2024-04-12T04:28:40.278 DEBUG:boto:Method: GET 2024-04-12T04:28:40.279 DEBUG:boto:Path: /dkyges-66/?prefix=obj33 2024-04-12T04:28:40.279 DEBUG:boto:Data: 2024-04-12T04:28:40.279 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.279 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.279 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.279 DEBUG:boto:Params: {} 2024-04-12T04:28:40.279 DEBUG:boto:Token: None 2024-04-12T04:28:40.279 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/ 2024-04-12T04:28:40.279 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:28:40.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:28:40.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b72d08257e32546b-006618b878-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.282 DEBUG:boto:b'dkyges-66obj331000falseobj332024-04-12T04:28:14.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:40.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:28:40.283 DEBUG:boto:path=/dkyges-66/obj44 2024-04-12T04:28:40.283 DEBUG:boto:auth_path=/dkyges-66/obj44 2024-04-12T04:28:40.283 DEBUG:boto:path=/dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.283 DEBUG:boto:auth_path=/dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.283 DEBUG:boto:Method: GET 2024-04-12T04:28:40.283 DEBUG:boto:Path: /dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.283 DEBUG:boto:Data: 2024-04-12T04:28:40.283 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.283 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.283 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.284 DEBUG:boto:Params: {} 2024-04-12T04:28:40.284 DEBUG:boto:Token: None 2024-04-12T04:28:40.284 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.284 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:El/Zgf250YJcHiavzKELP5MnQpE= 2024-04-12T04:28:40.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:El/Zgf250YJcHiavzKELP5MnQpE=', 'Content-Length': '0'} 2024-04-12T04:28:40.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d09e929b72df59ed-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.286 DEBUG:boto:path=/dkyges-66/obj44 2024-04-12T04:28:40.287 DEBUG:boto:auth_path=/dkyges-66/obj44 2024-04-12T04:28:40.287 DEBUG:boto:path=/dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.287 DEBUG:boto:auth_path=/dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.287 DEBUG:boto:Method: GET 2024-04-12T04:28:40.287 DEBUG:boto:Path: /dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.287 DEBUG:boto:Data: 2024-04-12T04:28:40.287 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.287 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.287 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.287 DEBUG:boto:Params: {} 2024-04-12T04:28:40.287 DEBUG:boto:Token: None 2024-04-12T04:28:40.288 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.288 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:El/Zgf250YJcHiavzKELP5MnQpE= 2024-04-12T04:28:40.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:El/Zgf250YJcHiavzKELP5MnQpE=', 'Content-Length': '0'} 2024-04-12T04:28:40.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094ec1217d981e73c-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.290 DEBUG:boto:path=/dkyges-66/obj44 2024-04-12T04:28:40.290 DEBUG:boto:auth_path=/dkyges-66/obj44 2024-04-12T04:28:40.291 DEBUG:boto:path=/dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.291 DEBUG:boto:auth_path=/dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.291 DEBUG:boto:Method: HEAD 2024-04-12T04:28:40.291 DEBUG:boto:Path: /dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.291 DEBUG:boto:Data: 2024-04-12T04:28:40.291 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.291 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.291 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.291 DEBUG:boto:Params: {} 2024-04-12T04:28:40.292 DEBUG:boto:Token: None 2024-04-12T04:28:40.292 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.292 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2vWWfUtTjrM+2JFdYvxRJWmJZH8= 2024-04-12T04:28:40.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2vWWfUtTjrM+2JFdYvxRJWmJZH8=', 'Content-Length': '0'} 2024-04-12T04:28:40.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073da06ea8e91be5c-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.297 DEBUG:boto:path=/dkyges-66/obj44 2024-04-12T04:28:40.297 DEBUG:boto:auth_path=/dkyges-66/obj44 2024-04-12T04:28:40.297 DEBUG:boto:path=/dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.297 DEBUG:boto:auth_path=/dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.298 DEBUG:boto:Method: HEAD 2024-04-12T04:28:40.298 DEBUG:boto:Path: /dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.298 DEBUG:boto:Data: 2024-04-12T04:28:40.298 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.299 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.299 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.299 DEBUG:boto:Params: {} 2024-04-12T04:28:40.300 DEBUG:boto:Token: None 2024-04-12T04:28:40.300 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.300 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2vWWfUtTjrM+2JFdYvxRJWmJZH8= 2024-04-12T04:28:40.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2vWWfUtTjrM+2JFdYvxRJWmJZH8=', 'Content-Length': '0'} 2024-04-12T04:28:40.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e37e8ec985c6538-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:28:40.304 DEBUG:boto:path=/dkyges-66/obj44 2024-04-12T04:28:40.304 DEBUG:boto:auth_path=/dkyges-66/obj44 2024-04-12T04:28:40.304 DEBUG:boto:path=/dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.304 DEBUG:boto:auth_path=/dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.304 DEBUG:boto:Method: GET 2024-04-12T04:28:40.304 DEBUG:boto:Path: /dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.305 DEBUG:boto:Data: 2024-04-12T04:28:40.305 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.305 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.305 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.305 DEBUG:boto:Params: {} 2024-04-12T04:28:40.305 DEBUG:boto:Token: None 2024-04-12T04:28:40.306 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.306 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:El/Zgf250YJcHiavzKELP5MnQpE= 2024-04-12T04:28:40.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:El/Zgf250YJcHiavzKELP5MnQpE=', 'Content-Length': '0'} 2024-04-12T04:28:40.308 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d026a47571571076-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.350 DEBUG:boto:path=/dkyges-66/obj44 2024-04-12T04:28:40.351 DEBUG:boto:auth_path=/dkyges-66/obj44 2024-04-12T04:28:40.351 DEBUG:boto:path=/dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.351 DEBUG:boto:auth_path=/dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.351 DEBUG:boto:Method: GET 2024-04-12T04:28:40.351 DEBUG:boto:Path: /dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.351 DEBUG:boto:Data: 2024-04-12T04:28:40.351 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.351 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.351 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.351 DEBUG:boto:Params: {} 2024-04-12T04:28:40.352 DEBUG:boto:Token: None 2024-04-12T04:28:40.352 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj44?versionId=null 2024-04-12T04:28:40.352 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:El/Zgf250YJcHiavzKELP5MnQpE= 2024-04-12T04:28:40.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:El/Zgf250YJcHiavzKELP5MnQpE=', 'Content-Length': '0'} 2024-04-12T04:28:40.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e70100cf49b7e7eb-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.355 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:40.355 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:40.355 DEBUG:boto:path=/dkyges-66/?prefix=obj44 2024-04-12T04:28:40.355 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj44 2024-04-12T04:28:40.355 DEBUG:boto:Method: GET 2024-04-12T04:28:40.355 DEBUG:boto:Path: /dkyges-66/?prefix=obj44 2024-04-12T04:28:40.355 DEBUG:boto:Data: 2024-04-12T04:28:40.355 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.355 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.355 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.355 DEBUG:boto:Params: {} 2024-04-12T04:28:40.356 DEBUG:boto:Token: None 2024-04-12T04:28:40.356 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/ 2024-04-12T04:28:40.356 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:28:40.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:28:40.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e79b2e295b83f87-006618b878-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.359 DEBUG:boto:b'dkyges-66obj441000falseobj442024-04-12T04:28:14.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:40.359 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:40.359 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:40.359 DEBUG:boto:path=/dkyges-66/?prefix=obj44 2024-04-12T04:28:40.359 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj44 2024-04-12T04:28:40.359 DEBUG:boto:Method: GET 2024-04-12T04:28:40.359 DEBUG:boto:Path: /dkyges-66/?prefix=obj44 2024-04-12T04:28:40.360 DEBUG:boto:Data: 2024-04-12T04:28:40.360 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.360 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.360 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.360 DEBUG:boto:Params: {} 2024-04-12T04:28:40.360 DEBUG:boto:Token: None 2024-04-12T04:28:40.360 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/ 2024-04-12T04:28:40.361 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:28:40.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:28:40.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061eacbf2f43db846-006618b878-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.364 DEBUG:boto:b'dkyges-66obj441000falseobj442024-04-12T04:28:14.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:40.365 DEBUG:boto:path=/dkyges-66/obj44 2024-04-12T04:28:40.365 DEBUG:boto:auth_path=/dkyges-66/obj44 2024-04-12T04:28:40.367 DEBUG:boto:Method: HEAD 2024-04-12T04:28:40.367 DEBUG:boto:Path: /dkyges-66/obj44 2024-04-12T04:28:40.367 DEBUG:boto:Data: 2024-04-12T04:28:40.367 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.368 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.368 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.368 DEBUG:boto:Params: {} 2024-04-12T04:28:40.368 DEBUG:boto:Token: None 2024-04-12T04:28:40.368 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj44 2024-04-12T04:28:40.368 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xggdqUHo2tep8poDsEr85ffZ+Ts= 2024-04-12T04:28:40.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xggdqUHo2tep8poDsEr85ffZ+Ts=', 'Content-Length': '0'} 2024-04-12T04:28:40.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005059b1b51fded9a-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.371 DEBUG:boto:path=/dkyges-66/obj44 2024-04-12T04:28:40.371 DEBUG:boto:auth_path=/dkyges-66/obj44 2024-04-12T04:28:40.371 DEBUG:boto:Method: HEAD 2024-04-12T04:28:40.371 DEBUG:boto:Path: /dkyges-66/obj44 2024-04-12T04:28:40.371 DEBUG:boto:Data: 2024-04-12T04:28:40.371 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.371 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.371 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.371 DEBUG:boto:Params: {} 2024-04-12T04:28:40.371 DEBUG:boto:Token: None 2024-04-12T04:28:40.372 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj44 2024-04-12T04:28:40.372 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xggdqUHo2tep8poDsEr85ffZ+Ts= 2024-04-12T04:28:40.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xggdqUHo2tep8poDsEr85ffZ+Ts=', 'Content-Length': '0'} 2024-04-12T04:28:40.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e0c152c9e54e8f3-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2024-04-12T04:28:40.374 DEBUG:boto:path=/dkyges-66/obj44 2024-04-12T04:28:40.375 DEBUG:boto:auth_path=/dkyges-66/obj44 2024-04-12T04:28:40.375 DEBUG:boto:Method: GET 2024-04-12T04:28:40.375 DEBUG:boto:Path: /dkyges-66/obj44 2024-04-12T04:28:40.375 DEBUG:boto:Data: 2024-04-12T04:28:40.375 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.375 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.375 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.375 DEBUG:boto:Params: {} 2024-04-12T04:28:40.375 DEBUG:boto:Token: None 2024-04-12T04:28:40.375 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj44 2024-04-12T04:28:40.375 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2ky2KrW9h2Y1t/KNoBfthuPd3P4= 2024-04-12T04:28:40.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2ky2KrW9h2Y1t/KNoBfthuPd3P4=', 'Content-Length': '0'} 2024-04-12T04:28:40.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fc0124e76e4df00-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.418 DEBUG:boto:path=/dkyges-66/obj44 2024-04-12T04:28:40.418 DEBUG:boto:auth_path=/dkyges-66/obj44 2024-04-12T04:28:40.419 DEBUG:boto:Method: GET 2024-04-12T04:28:40.419 DEBUG:boto:Path: /dkyges-66/obj44 2024-04-12T04:28:40.419 DEBUG:boto:Data: 2024-04-12T04:28:40.419 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.419 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.419 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.419 DEBUG:boto:Params: {} 2024-04-12T04:28:40.419 DEBUG:boto:Token: None 2024-04-12T04:28:40.419 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj44 2024-04-12T04:28:40.420 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2ky2KrW9h2Y1t/KNoBfthuPd3P4= 2024-04-12T04:28:40.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2ky2KrW9h2Y1t/KNoBfthuPd3P4=', 'Content-Length': '0'} 2024-04-12T04:28:40.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bd9cb231fe957fc-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.422 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:40.422 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:40.422 DEBUG:boto:path=/dkyges-66/?prefix=obj44 2024-04-12T04:28:40.422 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj44 2024-04-12T04:28:40.422 DEBUG:boto:Method: GET 2024-04-12T04:28:40.422 DEBUG:boto:Path: /dkyges-66/?prefix=obj44 2024-04-12T04:28:40.423 DEBUG:boto:Data: 2024-04-12T04:28:40.423 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.423 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.423 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.423 DEBUG:boto:Params: {} 2024-04-12T04:28:40.423 DEBUG:boto:Token: None 2024-04-12T04:28:40.423 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/ 2024-04-12T04:28:40.423 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:28:40.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:28:40.426 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047ab5ce268121115-006618b878-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.426 DEBUG:boto:b'dkyges-66obj441000falseobj442024-04-12T04:28:14.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:40.427 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:40.427 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:40.427 DEBUG:boto:path=/dkyges-66/?prefix=obj44 2024-04-12T04:28:40.427 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj44 2024-04-12T04:28:40.427 DEBUG:boto:Method: GET 2024-04-12T04:28:40.428 DEBUG:boto:Path: /dkyges-66/?prefix=obj44 2024-04-12T04:28:40.428 DEBUG:boto:Data: 2024-04-12T04:28:40.428 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.428 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.428 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.428 DEBUG:boto:Params: {} 2024-04-12T04:28:40.428 DEBUG:boto:Token: None 2024-04-12T04:28:40.429 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/ 2024-04-12T04:28:40.429 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:28:40.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:28:40.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f0e133a4de931f9-006618b878-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.431 DEBUG:boto:b'dkyges-66obj441000falseobj442024-04-12T04:28:14.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:40.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:28:40.432 DEBUG:boto:path=/dkyges-66/obj55 2024-04-12T04:28:40.432 DEBUG:boto:auth_path=/dkyges-66/obj55 2024-04-12T04:28:40.432 DEBUG:boto:path=/dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.432 DEBUG:boto:auth_path=/dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.432 DEBUG:boto:Method: GET 2024-04-12T04:28:40.432 DEBUG:boto:Path: /dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.432 DEBUG:boto:Data: 2024-04-12T04:28:40.432 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.433 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.433 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.433 DEBUG:boto:Params: {} 2024-04-12T04:28:40.433 DEBUG:boto:Token: None 2024-04-12T04:28:40.433 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.433 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RwTUkth6PHgaf489t+GZ9O0gB0g= 2024-04-12T04:28:40.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RwTUkth6PHgaf489t+GZ9O0gB0g=', 'Content-Length': '0'} 2024-04-12T04:28:40.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080b45581fbe44b7d-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.435 DEBUG:boto:path=/dkyges-66/obj55 2024-04-12T04:28:40.435 DEBUG:boto:auth_path=/dkyges-66/obj55 2024-04-12T04:28:40.435 DEBUG:boto:path=/dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.435 DEBUG:boto:auth_path=/dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.435 DEBUG:boto:Method: GET 2024-04-12T04:28:40.436 DEBUG:boto:Path: /dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.436 DEBUG:boto:Data: 2024-04-12T04:28:40.436 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.436 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.436 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.436 DEBUG:boto:Params: {} 2024-04-12T04:28:40.436 DEBUG:boto:Token: None 2024-04-12T04:28:40.436 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.436 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RwTUkth6PHgaf489t+GZ9O0gB0g= 2024-04-12T04:28:40.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RwTUkth6PHgaf489t+GZ9O0gB0g=', 'Content-Length': '0'} 2024-04-12T04:28:40.438 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd0cee22e3b1ac6-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.482 DEBUG:boto:path=/dkyges-66/obj55 2024-04-12T04:28:40.483 DEBUG:boto:auth_path=/dkyges-66/obj55 2024-04-12T04:28:40.483 DEBUG:boto:path=/dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.483 DEBUG:boto:auth_path=/dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.483 DEBUG:boto:Method: HEAD 2024-04-12T04:28:40.483 DEBUG:boto:Path: /dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.483 DEBUG:boto:Data: 2024-04-12T04:28:40.483 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.483 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.483 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.483 DEBUG:boto:Params: {} 2024-04-12T04:28:40.484 DEBUG:boto:Token: None 2024-04-12T04:28:40.484 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.484 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HSSwnQtSwvwB2uDRir+6DZhXScM= 2024-04-12T04:28:40.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HSSwnQtSwvwB2uDRir+6DZhXScM=', 'Content-Length': '0'} 2024-04-12T04:28:40.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a16f15b68254c03-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.486 DEBUG:boto:path=/dkyges-66/obj55 2024-04-12T04:28:40.486 DEBUG:boto:auth_path=/dkyges-66/obj55 2024-04-12T04:28:40.486 DEBUG:boto:path=/dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.486 DEBUG:boto:auth_path=/dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.486 DEBUG:boto:Method: HEAD 2024-04-12T04:28:40.487 DEBUG:boto:Path: /dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.487 DEBUG:boto:Data: 2024-04-12T04:28:40.487 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.487 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.487 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.487 DEBUG:boto:Params: {} 2024-04-12T04:28:40.487 DEBUG:boto:Token: None 2024-04-12T04:28:40.487 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.487 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HSSwnQtSwvwB2uDRir+6DZhXScM= 2024-04-12T04:28:40.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HSSwnQtSwvwB2uDRir+6DZhXScM=', 'Content-Length': '0'} 2024-04-12T04:28:40.489 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acaaff3aea0bb65b-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:28:40.490 DEBUG:boto:path=/dkyges-66/obj55 2024-04-12T04:28:40.490 DEBUG:boto:auth_path=/dkyges-66/obj55 2024-04-12T04:28:40.490 DEBUG:boto:path=/dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.490 DEBUG:boto:auth_path=/dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.490 DEBUG:boto:Method: GET 2024-04-12T04:28:40.490 DEBUG:boto:Path: /dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.490 DEBUG:boto:Data: 2024-04-12T04:28:40.490 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.490 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.491 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.491 DEBUG:boto:Params: {} 2024-04-12T04:28:40.491 DEBUG:boto:Token: None 2024-04-12T04:28:40.491 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.491 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RwTUkth6PHgaf489t+GZ9O0gB0g= 2024-04-12T04:28:40.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RwTUkth6PHgaf489t+GZ9O0gB0g=', 'Content-Length': '0'} 2024-04-12T04:28:40.493 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e20f215e9676bad4-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.493 DEBUG:boto:path=/dkyges-66/obj55 2024-04-12T04:28:40.493 DEBUG:boto:auth_path=/dkyges-66/obj55 2024-04-12T04:28:40.493 DEBUG:boto:path=/dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.493 DEBUG:boto:auth_path=/dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.493 DEBUG:boto:Method: GET 2024-04-12T04:28:40.494 DEBUG:boto:Path: /dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.494 DEBUG:boto:Data: 2024-04-12T04:28:40.494 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.494 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.494 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.494 DEBUG:boto:Params: {} 2024-04-12T04:28:40.494 DEBUG:boto:Token: None 2024-04-12T04:28:40.494 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj55?versionId=null 2024-04-12T04:28:40.494 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RwTUkth6PHgaf489t+GZ9O0gB0g= 2024-04-12T04:28:40.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RwTUkth6PHgaf489t+GZ9O0gB0g=', 'Content-Length': '0'} 2024-04-12T04:28:40.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e20c6e94b431ae6d-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.497 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:40.497 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:40.497 DEBUG:boto:path=/dkyges-66/?prefix=obj55 2024-04-12T04:28:40.497 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj55 2024-04-12T04:28:40.497 DEBUG:boto:Method: GET 2024-04-12T04:28:40.497 DEBUG:boto:Path: /dkyges-66/?prefix=obj55 2024-04-12T04:28:40.497 DEBUG:boto:Data: 2024-04-12T04:28:40.497 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.497 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.497 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.497 DEBUG:boto:Params: {} 2024-04-12T04:28:40.498 DEBUG:boto:Token: None 2024-04-12T04:28:40.498 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/ 2024-04-12T04:28:40.498 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:28:40.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:28:40.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000493da45eb6ef3089-006618b878-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.501 DEBUG:boto:b'dkyges-66obj551000falseobj552024-04-12T04:28:14.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:40.501 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:40.501 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:40.501 DEBUG:boto:path=/dkyges-66/?prefix=obj55 2024-04-12T04:28:40.501 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj55 2024-04-12T04:28:40.501 DEBUG:boto:Method: GET 2024-04-12T04:28:40.502 DEBUG:boto:Path: /dkyges-66/?prefix=obj55 2024-04-12T04:28:40.502 DEBUG:boto:Data: 2024-04-12T04:28:40.502 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.502 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.502 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.502 DEBUG:boto:Params: {} 2024-04-12T04:28:40.502 DEBUG:boto:Token: None 2024-04-12T04:28:40.502 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/ 2024-04-12T04:28:40.502 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:28:40.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:28:40.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdc0008c39cb3000-006618b878-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.505 DEBUG:boto:b'dkyges-66obj551000falseobj552024-04-12T04:28:14.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:40.505 DEBUG:boto:path=/dkyges-66/obj55 2024-04-12T04:28:40.505 DEBUG:boto:auth_path=/dkyges-66/obj55 2024-04-12T04:28:40.505 DEBUG:boto:Method: HEAD 2024-04-12T04:28:40.506 DEBUG:boto:Path: /dkyges-66/obj55 2024-04-12T04:28:40.506 DEBUG:boto:Data: 2024-04-12T04:28:40.506 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.506 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.506 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.506 DEBUG:boto:Params: {} 2024-04-12T04:28:40.506 DEBUG:boto:Token: None 2024-04-12T04:28:40.506 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj55 2024-04-12T04:28:40.506 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D/HThyBtYsP/rYPDtmgoXbBSUL8= 2024-04-12T04:28:40.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D/HThyBtYsP/rYPDtmgoXbBSUL8=', 'Content-Length': '0'} 2024-04-12T04:28:40.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f500700e0b7daa5-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.508 DEBUG:boto:path=/dkyges-66/obj55 2024-04-12T04:28:40.508 DEBUG:boto:auth_path=/dkyges-66/obj55 2024-04-12T04:28:40.508 DEBUG:boto:Method: HEAD 2024-04-12T04:28:40.509 DEBUG:boto:Path: /dkyges-66/obj55 2024-04-12T04:28:40.509 DEBUG:boto:Data: 2024-04-12T04:28:40.509 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.509 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.509 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.509 DEBUG:boto:Params: {} 2024-04-12T04:28:40.509 DEBUG:boto:Token: None 2024-04-12T04:28:40.509 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj55 2024-04-12T04:28:40.509 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:D/HThyBtYsP/rYPDtmgoXbBSUL8= 2024-04-12T04:28:40.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D/HThyBtYsP/rYPDtmgoXbBSUL8=', 'Content-Length': '0'} 2024-04-12T04:28:40.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001524a1dfdb709ad-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2024-04-12T04:28:40.511 DEBUG:boto:path=/dkyges-66/obj55 2024-04-12T04:28:40.511 DEBUG:boto:auth_path=/dkyges-66/obj55 2024-04-12T04:28:40.512 DEBUG:boto:Method: GET 2024-04-12T04:28:40.512 DEBUG:boto:Path: /dkyges-66/obj55 2024-04-12T04:28:40.512 DEBUG:boto:Data: 2024-04-12T04:28:40.512 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.512 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.512 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.512 DEBUG:boto:Params: {} 2024-04-12T04:28:40.512 DEBUG:boto:Token: None 2024-04-12T04:28:40.512 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj55 2024-04-12T04:28:40.512 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oqQfCMCf0lXsCdx8btkcluiyk0g= 2024-04-12T04:28:40.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oqQfCMCf0lXsCdx8btkcluiyk0g=', 'Content-Length': '0'} 2024-04-12T04:28:40.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055bd55692b710b8a-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.514 DEBUG:boto:path=/dkyges-66/obj55 2024-04-12T04:28:40.514 DEBUG:boto:auth_path=/dkyges-66/obj55 2024-04-12T04:28:40.515 DEBUG:boto:Method: GET 2024-04-12T04:28:40.515 DEBUG:boto:Path: /dkyges-66/obj55 2024-04-12T04:28:40.515 DEBUG:boto:Data: 2024-04-12T04:28:40.515 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.515 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.515 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.515 DEBUG:boto:Params: {} 2024-04-12T04:28:40.515 DEBUG:boto:Token: None 2024-04-12T04:28:40.515 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/obj55 2024-04-12T04:28:40.515 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oqQfCMCf0lXsCdx8btkcluiyk0g= 2024-04-12T04:28:40.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oqQfCMCf0lXsCdx8btkcluiyk0g=', 'Content-Length': '0'} 2024-04-12T04:28:40.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089cd108dc4c18a4a-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.518 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:40.518 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:40.518 DEBUG:boto:path=/dkyges-66/?prefix=obj55 2024-04-12T04:28:40.518 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj55 2024-04-12T04:28:40.518 DEBUG:boto:Method: GET 2024-04-12T04:28:40.518 DEBUG:boto:Path: /dkyges-66/?prefix=obj55 2024-04-12T04:28:40.518 DEBUG:boto:Data: 2024-04-12T04:28:40.518 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.518 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:40.519 DEBUG:boto:Port: 8001 2024-04-12T04:28:40.519 DEBUG:boto:Params: {} 2024-04-12T04:28:40.519 DEBUG:boto:Token: None 2024-04-12T04:28:40.519 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/ 2024-04-12T04:28:40.519 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:28:40.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:28:40.521 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9d758464a6f183a-006618b878-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.521 DEBUG:boto:b'dkyges-66obj551000falseobj552024-04-12T04:28:14.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:40.522 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:40.522 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:40.522 DEBUG:boto:path=/dkyges-66/?prefix=obj55 2024-04-12T04:28:40.522 DEBUG:boto:auth_path=/dkyges-66/?prefix=obj55 2024-04-12T04:28:40.522 DEBUG:boto:Method: GET 2024-04-12T04:28:40.522 DEBUG:boto:Path: /dkyges-66/?prefix=obj55 2024-04-12T04:28:40.522 DEBUG:boto:Data: 2024-04-12T04:28:40.523 DEBUG:boto:Headers: {} 2024-04-12T04:28:40.523 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:40.523 DEBUG:boto:Port: 8000 2024-04-12T04:28:40.523 DEBUG:boto:Params: {} 2024-04-12T04:28:40.523 DEBUG:boto:Token: None 2024-04-12T04:28:40.523 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:28:40 GMT /dkyges-66/ 2024-04-12T04:28:40.523 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:28:40.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:28:40.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e371b326eda1b55-006618b878-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:40.526 DEBUG:boto:b'dkyges-66obj551000falseobj552024-04-12T04:28:14.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:28:40.526 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=dkyges-66 zones={a2, a1} 2024-04-12T04:28:40.527 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T04:28:40.527 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:28:40.528 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T04:28:40.533+0000 7f417edaf700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 108871) UID: 0 2024-04-12T04:28:40.528 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T04:28:40.533+0000 7f4183f847c0 -1 shutting down 2024-04-12T04:28:42.861 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T04:28:46.626 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T04:28:46.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:46.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:46.627 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-66 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:46.691 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:46.691 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:46.745 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:28:46.745 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-66 2024-04-12T04:28:46.750 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 5 2024-04-12T04:28:46.796 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:46.796 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:46.796 DEBUG:boto:Method: HEAD 2024-04-12T04:28:46.796 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:46.797 DEBUG:boto:Data: 2024-04-12T04:28:46.797 DEBUG:boto:Headers: {} 2024-04-12T04:28:46.797 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:46.797 DEBUG:boto:Port: 8000 2024-04-12T04:28:46.797 DEBUG:boto:Params: {} 2024-04-12T04:28:46.797 DEBUG:boto:Token: None 2024-04-12T04:28:46.797 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:46 GMT /dkyges-66/ 2024-04-12T04:28:46.797 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ= 2024-04-12T04:28:46.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ=', 'Content-Length': '0'} 2024-04-12T04:28:46.800 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f5d037c09add470-006618b87e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:46.800 DEBUG:boto:path=/dkyges-66/obj17 2024-04-12T04:28:46.800 DEBUG:boto:auth_path=/dkyges-66/obj17 2024-04-12T04:28:46.801 DEBUG:boto:Method: PUT 2024-04-12T04:28:46.801 DEBUG:boto:Path: /dkyges-66/obj17 2024-04-12T04:28:46.801 DEBUG:boto:Data: 2024-04-12T04:28:46.801 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:46.801 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:46.801 DEBUG:boto:Port: 8000 2024-04-12T04:28:46.801 DEBUG:boto:Params: {} 2024-04-12T04:28:46.801 DEBUG:boto:Token: None 2024-04-12T04:28:46.801 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:46 GMT /dkyges-66/obj17 2024-04-12T04:28:46.801 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SKtAJ5ZnZihAiQFKteBHsn0nu/M= 2024-04-12T04:28:46.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SKtAJ5ZnZihAiQFKteBHsn0nu/M='} 2024-04-12T04:28:46.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e846af5e502f8b8f-006618b87e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:46.847 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:46.847 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:46.847 DEBUG:boto:Method: HEAD 2024-04-12T04:28:46.847 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:46.847 DEBUG:boto:Data: 2024-04-12T04:28:46.847 DEBUG:boto:Headers: {} 2024-04-12T04:28:46.847 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:46.848 DEBUG:boto:Port: 8000 2024-04-12T04:28:46.848 DEBUG:boto:Params: {} 2024-04-12T04:28:46.848 DEBUG:boto:Token: None 2024-04-12T04:28:46.848 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:46 GMT /dkyges-66/ 2024-04-12T04:28:46.848 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ= 2024-04-12T04:28:46.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ=', 'Content-Length': '0'} 2024-04-12T04:28:46.851 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004fd685715a4cffcc-006618b87e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:46.851 DEBUG:boto:path=/dkyges-66/obj34 2024-04-12T04:28:46.851 DEBUG:boto:auth_path=/dkyges-66/obj34 2024-04-12T04:28:46.851 DEBUG:boto:Method: PUT 2024-04-12T04:28:46.851 DEBUG:boto:Path: /dkyges-66/obj34 2024-04-12T04:28:46.851 DEBUG:boto:Data: 2024-04-12T04:28:46.851 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:46.852 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:46.852 DEBUG:boto:Port: 8000 2024-04-12T04:28:46.852 DEBUG:boto:Params: {} 2024-04-12T04:28:46.852 DEBUG:boto:Token: None 2024-04-12T04:28:46.852 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:46 GMT /dkyges-66/obj34 2024-04-12T04:28:46.852 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0/yjrjp6b2LO5EeHXb7PAButPjU= 2024-04-12T04:28:46.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0/yjrjp6b2LO5EeHXb7PAButPjU='} 2024-04-12T04:28:46.859 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081b3e1ec8a9da472-006618b87e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:46.859 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:46.859 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:46.859 DEBUG:boto:Method: HEAD 2024-04-12T04:28:46.859 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:46.859 DEBUG:boto:Data: 2024-04-12T04:28:46.859 DEBUG:boto:Headers: {} 2024-04-12T04:28:46.859 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:46.860 DEBUG:boto:Port: 8000 2024-04-12T04:28:46.860 DEBUG:boto:Params: {} 2024-04-12T04:28:46.860 DEBUG:boto:Token: None 2024-04-12T04:28:46.860 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:46 GMT /dkyges-66/ 2024-04-12T04:28:46.860 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ= 2024-04-12T04:28:46.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ=', 'Content-Length': '0'} 2024-04-12T04:28:46.863 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6afccafe4925584-006618b87e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:46.863 DEBUG:boto:path=/dkyges-66/obj51 2024-04-12T04:28:46.864 DEBUG:boto:auth_path=/dkyges-66/obj51 2024-04-12T04:28:46.864 DEBUG:boto:Method: PUT 2024-04-12T04:28:46.864 DEBUG:boto:Path: /dkyges-66/obj51 2024-04-12T04:28:46.864 DEBUG:boto:Data: 2024-04-12T04:28:46.864 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:46.864 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:46.864 DEBUG:boto:Port: 8000 2024-04-12T04:28:46.864 DEBUG:boto:Params: {} 2024-04-12T04:28:46.864 DEBUG:boto:Token: None 2024-04-12T04:28:46.864 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:46 GMT /dkyges-66/obj51 2024-04-12T04:28:46.865 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L7/iXnba+btsa5mLQSLyUcqW3do= 2024-04-12T04:28:46.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L7/iXnba+btsa5mLQSLyUcqW3do='} 2024-04-12T04:28:46.910 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb2e51576f06aab1-006618b87e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:46.911 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:46.911 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:46.911 DEBUG:boto:Method: HEAD 2024-04-12T04:28:46.911 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:46.911 DEBUG:boto:Data: 2024-04-12T04:28:46.911 DEBUG:boto:Headers: {} 2024-04-12T04:28:46.911 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:46.911 DEBUG:boto:Port: 8000 2024-04-12T04:28:46.912 DEBUG:boto:Params: {} 2024-04-12T04:28:46.912 DEBUG:boto:Token: None 2024-04-12T04:28:46.912 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:46 GMT /dkyges-66/ 2024-04-12T04:28:46.912 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ= 2024-04-12T04:28:46.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ=', 'Content-Length': '0'} 2024-04-12T04:28:46.915 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b35cadd645c1c8ac-006618b87e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:46.915 DEBUG:boto:path=/dkyges-66/obj68 2024-04-12T04:28:46.915 DEBUG:boto:auth_path=/dkyges-66/obj68 2024-04-12T04:28:46.915 DEBUG:boto:Method: PUT 2024-04-12T04:28:46.915 DEBUG:boto:Path: /dkyges-66/obj68 2024-04-12T04:28:46.915 DEBUG:boto:Data: 2024-04-12T04:28:46.916 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:46.916 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:46.916 DEBUG:boto:Port: 8000 2024-04-12T04:28:46.916 DEBUG:boto:Params: {} 2024-04-12T04:28:46.916 DEBUG:boto:Token: None 2024-04-12T04:28:46.916 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:46 GMT /dkyges-66/obj68 2024-04-12T04:28:46.916 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Z7Wm/zgOpSJiQEdYe9ZACM8gEUU= 2024-04-12T04:28:46.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Z7Wm/zgOpSJiQEdYe9ZACM8gEUU='} 2024-04-12T04:28:46.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003895c5917f654de5-006618b87e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:46.923 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:46.923 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:46.923 DEBUG:boto:Method: HEAD 2024-04-12T04:28:46.923 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:46.923 DEBUG:boto:Data: 2024-04-12T04:28:46.923 DEBUG:boto:Headers: {} 2024-04-12T04:28:46.923 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:46.923 DEBUG:boto:Port: 8000 2024-04-12T04:28:46.923 DEBUG:boto:Params: {} 2024-04-12T04:28:46.923 DEBUG:boto:Token: None 2024-04-12T04:28:46.924 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:46 GMT /dkyges-66/ 2024-04-12T04:28:46.924 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ= 2024-04-12T04:28:46.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ=', 'Content-Length': '0'} 2024-04-12T04:28:46.926 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022e33e125e73ce48-006618b87e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:46.927 DEBUG:boto:path=/dkyges-66/obj85 2024-04-12T04:28:46.927 DEBUG:boto:auth_path=/dkyges-66/obj85 2024-04-12T04:28:46.927 DEBUG:boto:Method: PUT 2024-04-12T04:28:46.927 DEBUG:boto:Path: /dkyges-66/obj85 2024-04-12T04:28:46.927 DEBUG:boto:Data: 2024-04-12T04:28:46.927 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:46.927 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:46.927 DEBUG:boto:Port: 8000 2024-04-12T04:28:46.927 DEBUG:boto:Params: {} 2024-04-12T04:28:46.927 DEBUG:boto:Token: None 2024-04-12T04:28:46.928 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:46 GMT /dkyges-66/obj85 2024-04-12T04:28:46.928 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:obrIAv/z13GUuF+ekovHF9c+jEc= 2024-04-12T04:28:46.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obrIAv/z13GUuF+ekovHF9c+jEc='} 2024-04-12T04:28:46.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d8849f10f3bf1c7e-006618b87e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:46.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:46.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:46.971 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-66 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:47.034 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:47.034 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:47.087 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:28:47.087 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-66 2024-04-12T04:28:47.094 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 10 2024-04-12T04:28:47.148 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:47.148 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:47.149 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.149 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:47.149 DEBUG:boto:Data: 2024-04-12T04:28:47.149 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.149 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.149 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.149 DEBUG:boto:Params: {} 2024-04-12T04:28:47.149 DEBUG:boto:Token: None 2024-04-12T04:28:47.149 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/ 2024-04-12T04:28:47.150 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:28:47.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:28:47.153 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053a2390d30178123-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.153 DEBUG:boto:path=/dkyges-66/obj19 2024-04-12T04:28:47.153 DEBUG:boto:auth_path=/dkyges-66/obj19 2024-04-12T04:28:47.153 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.153 DEBUG:boto:Path: /dkyges-66/obj19 2024-04-12T04:28:47.154 DEBUG:boto:Data: 2024-04-12T04:28:47.154 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:47.154 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.154 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.154 DEBUG:boto:Params: {} 2024-04-12T04:28:47.154 DEBUG:boto:Token: None 2024-04-12T04:28:47.154 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/obj19 2024-04-12T04:28:47.154 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/FK+XKDWoS1AKNZkAwBC4ttdcmk= 2024-04-12T04:28:47.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/FK+XKDWoS1AKNZkAwBC4ttdcmk='} 2024-04-12T04:28:47.160 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000308b12dfb2dc8831-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.160 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:47.161 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:47.161 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.161 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:47.161 DEBUG:boto:Data: 2024-04-12T04:28:47.161 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.161 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.161 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.161 DEBUG:boto:Params: {} 2024-04-12T04:28:47.161 DEBUG:boto:Token: None 2024-04-12T04:28:47.161 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/ 2024-04-12T04:28:47.162 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:28:47.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:28:47.164 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1cd1311360cefa9-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.164 DEBUG:boto:path=/dkyges-66/obj38 2024-04-12T04:28:47.164 DEBUG:boto:auth_path=/dkyges-66/obj38 2024-04-12T04:28:47.164 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.164 DEBUG:boto:Path: /dkyges-66/obj38 2024-04-12T04:28:47.165 DEBUG:boto:Data: 2024-04-12T04:28:47.165 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:47.165 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.165 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.165 DEBUG:boto:Params: {} 2024-04-12T04:28:47.165 DEBUG:boto:Token: None 2024-04-12T04:28:47.165 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/obj38 2024-04-12T04:28:47.165 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kW+bD6r9YiGAEXind5oWxYxRgOI= 2024-04-12T04:28:47.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kW+bD6r9YiGAEXind5oWxYxRgOI='} 2024-04-12T04:28:47.169 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c44d06be28ce875-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.170 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:47.170 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:47.170 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.170 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:47.170 DEBUG:boto:Data: 2024-04-12T04:28:47.170 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.170 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.170 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.170 DEBUG:boto:Params: {} 2024-04-12T04:28:47.171 DEBUG:boto:Token: None 2024-04-12T04:28:47.171 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/ 2024-04-12T04:28:47.175 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:28:47.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:28:47.178 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbe4941d47e0b6f2-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.178 DEBUG:boto:path=/dkyges-66/obj57 2024-04-12T04:28:47.178 DEBUG:boto:auth_path=/dkyges-66/obj57 2024-04-12T04:28:47.179 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.179 DEBUG:boto:Path: /dkyges-66/obj57 2024-04-12T04:28:47.179 DEBUG:boto:Data: 2024-04-12T04:28:47.179 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:47.179 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.179 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.179 DEBUG:boto:Params: {} 2024-04-12T04:28:47.179 DEBUG:boto:Token: None 2024-04-12T04:28:47.179 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/obj57 2024-04-12T04:28:47.179 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OlzSMjML5CEUaq98j7yYLLdLSHw= 2024-04-12T04:28:47.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OlzSMjML5CEUaq98j7yYLLdLSHw='} 2024-04-12T04:28:47.223 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009965994354ce40e6-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.223 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:47.223 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:47.223 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.224 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:47.224 DEBUG:boto:Data: 2024-04-12T04:28:47.224 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.224 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.224 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.225 DEBUG:boto:Params: {} 2024-04-12T04:28:47.225 DEBUG:boto:Token: None 2024-04-12T04:28:47.225 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/ 2024-04-12T04:28:47.225 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:28:47.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:28:47.230 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d82b954aca01f6e7-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.231 DEBUG:boto:path=/dkyges-66/obj76 2024-04-12T04:28:47.231 DEBUG:boto:auth_path=/dkyges-66/obj76 2024-04-12T04:28:47.231 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.231 DEBUG:boto:Path: /dkyges-66/obj76 2024-04-12T04:28:47.231 DEBUG:boto:Data: 2024-04-12T04:28:47.232 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:47.232 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.232 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.232 DEBUG:boto:Params: {} 2024-04-12T04:28:47.232 DEBUG:boto:Token: None 2024-04-12T04:28:47.232 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/obj76 2024-04-12T04:28:47.233 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ouYDshmy3mzRvw+oq+cjKD59mmQ= 2024-04-12T04:28:47.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ouYDshmy3mzRvw+oq+cjKD59mmQ='} 2024-04-12T04:28:47.242 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de56a3651aaba064-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.242 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:47.242 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:47.243 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.243 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:47.243 DEBUG:boto:Data: 2024-04-12T04:28:47.243 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.243 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.243 DEBUG:boto:Params: {} 2024-04-12T04:28:47.243 DEBUG:boto:Token: None 2024-04-12T04:28:47.243 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/ 2024-04-12T04:28:47.244 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:28:47.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:28:47.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000edbe4f0d1b7d18e-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.247 DEBUG:boto:path=/dkyges-66/obj95 2024-04-12T04:28:47.247 DEBUG:boto:auth_path=/dkyges-66/obj95 2024-04-12T04:28:47.247 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.247 DEBUG:boto:Path: /dkyges-66/obj95 2024-04-12T04:28:47.247 DEBUG:boto:Data: 2024-04-12T04:28:47.247 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:47.247 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.247 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.248 DEBUG:boto:Params: {} 2024-04-12T04:28:47.248 DEBUG:boto:Token: None 2024-04-12T04:28:47.248 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/obj95 2024-04-12T04:28:47.248 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PSOHPD77X8QMoFWjp4km3bY7c7U= 2024-04-12T04:28:47.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PSOHPD77X8QMoFWjp4km3bY7c7U='} 2024-04-12T04:28:47.291 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0662e6e474eb992-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:47.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:47.291 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-66 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:47.354 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:47.355 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:47.410 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:28:47.410 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-66 2024-04-12T04:28:47.417 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 15 2024-04-12T04:28:47.474 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:47.474 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:47.474 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.474 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:47.475 DEBUG:boto:Data: 2024-04-12T04:28:47.475 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.475 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.475 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.475 DEBUG:boto:Params: {} 2024-04-12T04:28:47.475 DEBUG:boto:Token: None 2024-04-12T04:28:47.475 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/ 2024-04-12T04:28:47.475 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:28:47.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:28:47.479 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000403a76b37ba53137-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.479 DEBUG:boto:path=/dkyges-66/obj23 2024-04-12T04:28:47.479 DEBUG:boto:auth_path=/dkyges-66/obj23 2024-04-12T04:28:47.479 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.479 DEBUG:boto:Path: /dkyges-66/obj23 2024-04-12T04:28:47.479 DEBUG:boto:Data: 2024-04-12T04:28:47.479 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:47.480 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.480 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.480 DEBUG:boto:Params: {} 2024-04-12T04:28:47.480 DEBUG:boto:Token: None 2024-04-12T04:28:47.480 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/obj23 2024-04-12T04:28:47.480 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:L2mKbHbWALBkc4vo74Y3IO5rTT8= 2024-04-12T04:28:47.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L2mKbHbWALBkc4vo74Y3IO5rTT8='} 2024-04-12T04:28:47.486 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e245fbcfb819b2b6-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.486 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:47.486 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:47.487 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.487 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:47.487 DEBUG:boto:Data: 2024-04-12T04:28:47.487 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.487 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.487 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.487 DEBUG:boto:Params: {} 2024-04-12T04:28:47.487 DEBUG:boto:Token: None 2024-04-12T04:28:47.487 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/ 2024-04-12T04:28:47.487 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:28:47.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:28:47.490 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000284232918303bf26-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.490 DEBUG:boto:path=/dkyges-66/obj46 2024-04-12T04:28:47.491 DEBUG:boto:auth_path=/dkyges-66/obj46 2024-04-12T04:28:47.491 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.491 DEBUG:boto:Path: /dkyges-66/obj46 2024-04-12T04:28:47.491 DEBUG:boto:Data: 2024-04-12T04:28:47.491 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:47.491 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.491 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.491 DEBUG:boto:Params: {} 2024-04-12T04:28:47.491 DEBUG:boto:Token: None 2024-04-12T04:28:47.491 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/obj46 2024-04-12T04:28:47.492 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8A3XhaEOv2wy92LhUu268EUe1/I= 2024-04-12T04:28:47.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8A3XhaEOv2wy92LhUu268EUe1/I='} 2024-04-12T04:28:47.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a45087c134c1b685-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.535 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:47.535 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:47.535 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.535 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:47.535 DEBUG:boto:Data: 2024-04-12T04:28:47.535 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.535 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.535 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.535 DEBUG:boto:Params: {} 2024-04-12T04:28:47.536 DEBUG:boto:Token: None 2024-04-12T04:28:47.536 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/ 2024-04-12T04:28:47.536 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:28:47.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:28:47.539 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3d3c8f87b129ab0-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.539 DEBUG:boto:path=/dkyges-66/obj69 2024-04-12T04:28:47.539 DEBUG:boto:auth_path=/dkyges-66/obj69 2024-04-12T04:28:47.539 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.539 DEBUG:boto:Path: /dkyges-66/obj69 2024-04-12T04:28:47.539 DEBUG:boto:Data: 2024-04-12T04:28:47.540 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:47.540 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.540 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.540 DEBUG:boto:Params: {} 2024-04-12T04:28:47.540 DEBUG:boto:Token: None 2024-04-12T04:28:47.540 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/obj69 2024-04-12T04:28:47.540 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nqz+KM1CF+9JASGgCA8HH/rzFjc= 2024-04-12T04:28:47.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nqz+KM1CF+9JASGgCA8HH/rzFjc='} 2024-04-12T04:28:47.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e302247fb767a14-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.546 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:47.546 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:47.546 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.546 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:47.546 DEBUG:boto:Data: 2024-04-12T04:28:47.547 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.547 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.547 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.547 DEBUG:boto:Params: {} 2024-04-12T04:28:47.547 DEBUG:boto:Token: None 2024-04-12T04:28:47.547 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/ 2024-04-12T04:28:47.547 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:28:47.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:28:47.550 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdb2b83269ee7994-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.550 DEBUG:boto:path=/dkyges-66/obj92 2024-04-12T04:28:47.551 DEBUG:boto:auth_path=/dkyges-66/obj92 2024-04-12T04:28:47.551 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.551 DEBUG:boto:Path: /dkyges-66/obj92 2024-04-12T04:28:47.551 DEBUG:boto:Data: 2024-04-12T04:28:47.551 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:47.551 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.551 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.551 DEBUG:boto:Params: {} 2024-04-12T04:28:47.551 DEBUG:boto:Token: None 2024-04-12T04:28:47.551 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/obj92 2024-04-12T04:28:47.552 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:3RIhuPFHWs9MLOLczQnJk6NUQhU= 2024-04-12T04:28:47.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3RIhuPFHWs9MLOLczQnJk6NUQhU='} 2024-04-12T04:28:47.598 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082f2c07a12ac6320-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.599 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:47.599 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:47.599 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.599 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:47.599 DEBUG:boto:Data: 2024-04-12T04:28:47.599 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.599 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.599 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.599 DEBUG:boto:Params: {} 2024-04-12T04:28:47.599 DEBUG:boto:Token: None 2024-04-12T04:28:47.600 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/ 2024-04-12T04:28:47.600 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:28:47.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:28:47.603 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b78f7b697bc06cb-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.603 DEBUG:boto:path=/dkyges-66/obj115 2024-04-12T04:28:47.603 DEBUG:boto:auth_path=/dkyges-66/obj115 2024-04-12T04:28:47.603 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.603 DEBUG:boto:Path: /dkyges-66/obj115 2024-04-12T04:28:47.603 DEBUG:boto:Data: 2024-04-12T04:28:47.603 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:47.604 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.604 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.604 DEBUG:boto:Params: {} 2024-04-12T04:28:47.604 DEBUG:boto:Token: None 2024-04-12T04:28:47.604 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/obj115 2024-04-12T04:28:47.604 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QVTYj5NramhJKAo6INSTWD8Mzck= 2024-04-12T04:28:47.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QVTYj5NramhJKAo6INSTWD8Mzck='} 2024-04-12T04:28:47.611 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036c1b3bbf2f7e3ce-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:47.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:47.611 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-66 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:47.673 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:47.673 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:47.731 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:28:47.731 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-66 2024-04-12T04:28:47.741 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 20 2024-04-12T04:28:47.803 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:47.803 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:47.804 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.804 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:47.804 DEBUG:boto:Data: 2024-04-12T04:28:47.804 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.804 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.804 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.804 DEBUG:boto:Params: {} 2024-04-12T04:28:47.804 DEBUG:boto:Token: None 2024-04-12T04:28:47.804 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/ 2024-04-12T04:28:47.805 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:28:47.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:28:47.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b74d6013148a47b3-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.808 DEBUG:boto:path=/dkyges-66/obj29 2024-04-12T04:28:47.808 DEBUG:boto:auth_path=/dkyges-66/obj29 2024-04-12T04:28:47.809 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.809 DEBUG:boto:Path: /dkyges-66/obj29 2024-04-12T04:28:47.809 DEBUG:boto:Data: 2024-04-12T04:28:47.809 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:47.809 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.809 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.809 DEBUG:boto:Params: {} 2024-04-12T04:28:47.809 DEBUG:boto:Token: None 2024-04-12T04:28:47.809 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/obj29 2024-04-12T04:28:47.809 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8l+zBdZXfwL1ghoDxLGq+MuCAso= 2024-04-12T04:28:47.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8l+zBdZXfwL1ghoDxLGq+MuCAso='} 2024-04-12T04:28:47.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000620b9002c20a9508-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.855 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:47.855 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:47.855 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.855 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:47.855 DEBUG:boto:Data: 2024-04-12T04:28:47.855 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.855 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.855 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.855 DEBUG:boto:Params: {} 2024-04-12T04:28:47.856 DEBUG:boto:Token: None 2024-04-12T04:28:47.856 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/ 2024-04-12T04:28:47.856 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:28:47.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:28:47.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081a3ff616518c58e-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.860 DEBUG:boto:path=/dkyges-66/obj58 2024-04-12T04:28:47.860 DEBUG:boto:auth_path=/dkyges-66/obj58 2024-04-12T04:28:47.860 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.860 DEBUG:boto:Path: /dkyges-66/obj58 2024-04-12T04:28:47.860 DEBUG:boto:Data: 2024-04-12T04:28:47.860 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:47.860 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.861 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.861 DEBUG:boto:Params: {} 2024-04-12T04:28:47.861 DEBUG:boto:Token: None 2024-04-12T04:28:47.861 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/obj58 2024-04-12T04:28:47.861 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:MS6wYcuysTzUDTW1DtqZFcOBZBw= 2024-04-12T04:28:47.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MS6wYcuysTzUDTW1DtqZFcOBZBw='} 2024-04-12T04:28:47.867 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025b49b5ed1b6eec9-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.867 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:47.867 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:47.868 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.868 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:47.868 DEBUG:boto:Data: 2024-04-12T04:28:47.868 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.868 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.868 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.868 DEBUG:boto:Params: {} 2024-04-12T04:28:47.868 DEBUG:boto:Token: None 2024-04-12T04:28:47.868 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/ 2024-04-12T04:28:47.868 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:28:47.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:28:47.872 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b3f150687feb490-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.872 DEBUG:boto:path=/dkyges-66/obj87 2024-04-12T04:28:47.872 DEBUG:boto:auth_path=/dkyges-66/obj87 2024-04-12T04:28:47.872 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.872 DEBUG:boto:Path: /dkyges-66/obj87 2024-04-12T04:28:47.872 DEBUG:boto:Data: 2024-04-12T04:28:47.872 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:47.872 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.872 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.873 DEBUG:boto:Params: {} 2024-04-12T04:28:47.873 DEBUG:boto:Token: None 2024-04-12T04:28:47.873 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/obj87 2024-04-12T04:28:47.873 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:uQvG5M/I8HmenPalP2ngzF1y5Xc= 2024-04-12T04:28:47.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uQvG5M/I8HmenPalP2ngzF1y5Xc='} 2024-04-12T04:28:47.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002910a387e27e8abb-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.919 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:47.919 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:47.919 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.919 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:47.919 DEBUG:boto:Data: 2024-04-12T04:28:47.919 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.919 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.919 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.919 DEBUG:boto:Params: {} 2024-04-12T04:28:47.920 DEBUG:boto:Token: None 2024-04-12T04:28:47.920 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/ 2024-04-12T04:28:47.920 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:28:47.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:28:47.923 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000835bd753f7be8c90-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.924 DEBUG:boto:path=/dkyges-66/obj116 2024-04-12T04:28:47.924 DEBUG:boto:auth_path=/dkyges-66/obj116 2024-04-12T04:28:47.924 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.924 DEBUG:boto:Path: /dkyges-66/obj116 2024-04-12T04:28:47.924 DEBUG:boto:Data: 2024-04-12T04:28:47.924 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:47.924 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.924 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.925 DEBUG:boto:Params: {} 2024-04-12T04:28:47.925 DEBUG:boto:Token: None 2024-04-12T04:28:47.925 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/obj116 2024-04-12T04:28:47.925 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9KSXg9axMXezx4qc/FyjtyOkLG8= 2024-04-12T04:28:47.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9KSXg9axMXezx4qc/FyjtyOkLG8='} 2024-04-12T04:28:47.931 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f27e99c8e14e19f9-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.932 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:47.932 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:47.932 DEBUG:boto:Method: HEAD 2024-04-12T04:28:47.932 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:47.932 DEBUG:boto:Data: 2024-04-12T04:28:47.932 DEBUG:boto:Headers: {} 2024-04-12T04:28:47.932 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.932 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.932 DEBUG:boto:Params: {} 2024-04-12T04:28:47.932 DEBUG:boto:Token: None 2024-04-12T04:28:47.933 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/ 2024-04-12T04:28:47.933 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:28:47.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:28:47.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1fec857b42d6325-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.937 DEBUG:boto:path=/dkyges-66/obj145 2024-04-12T04:28:47.937 DEBUG:boto:auth_path=/dkyges-66/obj145 2024-04-12T04:28:47.937 DEBUG:boto:Method: PUT 2024-04-12T04:28:47.937 DEBUG:boto:Path: /dkyges-66/obj145 2024-04-12T04:28:47.937 DEBUG:boto:Data: 2024-04-12T04:28:47.937 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:47.937 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:47.937 DEBUG:boto:Port: 8000 2024-04-12T04:28:47.937 DEBUG:boto:Params: {} 2024-04-12T04:28:47.938 DEBUG:boto:Token: None 2024-04-12T04:28:47.938 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:47 GMT /dkyges-66/obj145 2024-04-12T04:28:47.938 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AbOQnfeBwbbtp7279XpArmPZbqE= 2024-04-12T04:28:47.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AbOQnfeBwbbtp7279XpArmPZbqE='} 2024-04-12T04:28:47.983 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001842ba1a2885b5b2-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:47.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:47.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:47.983 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-66 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:48.048 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:48.048 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:48.110 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:28:48.110 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-66 2024-04-12T04:28:48.123 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 25 2024-04-12T04:28:48.196 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:48.196 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:48.197 DEBUG:boto:Method: HEAD 2024-04-12T04:28:48.197 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:48.197 DEBUG:boto:Data: 2024-04-12T04:28:48.197 DEBUG:boto:Headers: {} 2024-04-12T04:28:48.197 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.197 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.197 DEBUG:boto:Params: {} 2024-04-12T04:28:48.197 DEBUG:boto:Token: None 2024-04-12T04:28:48.197 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/ 2024-04-12T04:28:48.198 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:28:48.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:28:48.202 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036678942352b6fd3-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.202 DEBUG:boto:path=/dkyges-66/obj31 2024-04-12T04:28:48.202 DEBUG:boto:auth_path=/dkyges-66/obj31 2024-04-12T04:28:48.203 DEBUG:boto:Method: PUT 2024-04-12T04:28:48.203 DEBUG:boto:Path: /dkyges-66/obj31 2024-04-12T04:28:48.203 DEBUG:boto:Data: 2024-04-12T04:28:48.203 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:48.203 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.203 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.203 DEBUG:boto:Params: {} 2024-04-12T04:28:48.203 DEBUG:boto:Token: None 2024-04-12T04:28:48.203 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/obj31 2024-04-12T04:28:48.203 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ipQsuEP/bWlb1sjdNeG2vFdrM0s= 2024-04-12T04:28:48.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ipQsuEP/bWlb1sjdNeG2vFdrM0s='} 2024-04-12T04:28:48.209 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008635abfa4286f978-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.209 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:48.209 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:48.209 DEBUG:boto:Method: HEAD 2024-04-12T04:28:48.210 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:48.210 DEBUG:boto:Data: 2024-04-12T04:28:48.210 DEBUG:boto:Headers: {} 2024-04-12T04:28:48.210 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.210 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.210 DEBUG:boto:Params: {} 2024-04-12T04:28:48.210 DEBUG:boto:Token: None 2024-04-12T04:28:48.210 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/ 2024-04-12T04:28:48.210 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:28:48.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:28:48.214 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ebae62fa5d2920d-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.214 DEBUG:boto:path=/dkyges-66/obj62 2024-04-12T04:28:48.214 DEBUG:boto:auth_path=/dkyges-66/obj62 2024-04-12T04:28:48.214 DEBUG:boto:Method: PUT 2024-04-12T04:28:48.214 DEBUG:boto:Path: /dkyges-66/obj62 2024-04-12T04:28:48.215 DEBUG:boto:Data: 2024-04-12T04:28:48.215 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:48.215 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.215 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.215 DEBUG:boto:Params: {} 2024-04-12T04:28:48.215 DEBUG:boto:Token: None 2024-04-12T04:28:48.215 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/obj62 2024-04-12T04:28:48.215 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:I6v5vRxgeyhihmTSCVaRvAha0+M= 2024-04-12T04:28:48.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I6v5vRxgeyhihmTSCVaRvAha0+M='} 2024-04-12T04:28:48.221 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087b5717e9388e0cf-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.221 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:48.221 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:48.221 DEBUG:boto:Method: HEAD 2024-04-12T04:28:48.221 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:48.222 DEBUG:boto:Data: 2024-04-12T04:28:48.222 DEBUG:boto:Headers: {} 2024-04-12T04:28:48.222 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.222 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.222 DEBUG:boto:Params: {} 2024-04-12T04:28:48.222 DEBUG:boto:Token: None 2024-04-12T04:28:48.222 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/ 2024-04-12T04:28:48.222 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:28:48.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:28:48.226 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003056335c9533e31d-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.226 DEBUG:boto:path=/dkyges-66/obj93 2024-04-12T04:28:48.226 DEBUG:boto:auth_path=/dkyges-66/obj93 2024-04-12T04:28:48.227 DEBUG:boto:Method: PUT 2024-04-12T04:28:48.227 DEBUG:boto:Path: /dkyges-66/obj93 2024-04-12T04:28:48.227 DEBUG:boto:Data: 2024-04-12T04:28:48.227 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:48.227 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.227 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.227 DEBUG:boto:Params: {} 2024-04-12T04:28:48.227 DEBUG:boto:Token: None 2024-04-12T04:28:48.227 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/obj93 2024-04-12T04:28:48.228 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ZEw4VmcR+hhePl5mlaHqLISjFpc= 2024-04-12T04:28:48.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZEw4VmcR+hhePl5mlaHqLISjFpc='} 2024-04-12T04:28:48.233 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c08ddf7a42615d43-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.233 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:48.233 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:48.233 DEBUG:boto:Method: HEAD 2024-04-12T04:28:48.234 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:48.234 DEBUG:boto:Data: 2024-04-12T04:28:48.234 DEBUG:boto:Headers: {} 2024-04-12T04:28:48.234 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.234 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.234 DEBUG:boto:Params: {} 2024-04-12T04:28:48.235 DEBUG:boto:Token: None 2024-04-12T04:28:48.235 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/ 2024-04-12T04:28:48.235 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:28:48.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:28:48.238 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b6274a7ac29c96e-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.239 DEBUG:boto:path=/dkyges-66/obj124 2024-04-12T04:28:48.239 DEBUG:boto:auth_path=/dkyges-66/obj124 2024-04-12T04:28:48.239 DEBUG:boto:Method: PUT 2024-04-12T04:28:48.239 DEBUG:boto:Path: /dkyges-66/obj124 2024-04-12T04:28:48.239 DEBUG:boto:Data: 2024-04-12T04:28:48.239 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:48.239 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.239 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.239 DEBUG:boto:Params: {} 2024-04-12T04:28:48.240 DEBUG:boto:Token: None 2024-04-12T04:28:48.240 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/obj124 2024-04-12T04:28:48.240 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fb4pCPKP6XlrhLqoLYXC11gYFkI= 2024-04-12T04:28:48.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fb4pCPKP6XlrhLqoLYXC11gYFkI='} 2024-04-12T04:28:48.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db06e91f594df2c2-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.283 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:48.283 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:48.283 DEBUG:boto:Method: HEAD 2024-04-12T04:28:48.283 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:48.283 DEBUG:boto:Data: 2024-04-12T04:28:48.283 DEBUG:boto:Headers: {} 2024-04-12T04:28:48.283 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.283 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.284 DEBUG:boto:Params: {} 2024-04-12T04:28:48.284 DEBUG:boto:Token: None 2024-04-12T04:28:48.284 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/ 2024-04-12T04:28:48.284 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:28:48.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:28:48.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db4c91fa12d8d358-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.288 DEBUG:boto:path=/dkyges-66/obj155 2024-04-12T04:28:48.288 DEBUG:boto:auth_path=/dkyges-66/obj155 2024-04-12T04:28:48.288 DEBUG:boto:Method: PUT 2024-04-12T04:28:48.289 DEBUG:boto:Path: /dkyges-66/obj155 2024-04-12T04:28:48.289 DEBUG:boto:Data: 2024-04-12T04:28:48.289 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:48.289 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.289 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.289 DEBUG:boto:Params: {} 2024-04-12T04:28:48.289 DEBUG:boto:Token: None 2024-04-12T04:28:48.289 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/obj155 2024-04-12T04:28:48.289 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lPAvbsITyOldjUHTWwyMJH5ZNIM= 2024-04-12T04:28:48.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lPAvbsITyOldjUHTWwyMJH5ZNIM='} 2024-04-12T04:28:48.296 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e230586a1f3b86c9-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:48.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:48.296 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-66 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:48.358 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:48.358 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:48.418 INFO:teuthology.orchestra.run.smithi043.stdout:tenant: 2024-04-12T04:28:48.419 INFO:teuthology.orchestra.run.smithi043.stdout:bucket name: dkyges-66 2024-04-12T04:28:48.433 INFO:teuthology.orchestra.run.smithi043.stdout:total entries: 30 2024-04-12T04:28:48.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T04:28:48.508 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:48.508 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:48.508 DEBUG:boto:Method: HEAD 2024-04-12T04:28:48.509 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:48.509 DEBUG:boto:Data: 2024-04-12T04:28:48.509 DEBUG:boto:Headers: {} 2024-04-12T04:28:48.509 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.509 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.509 DEBUG:boto:Params: {} 2024-04-12T04:28:48.509 DEBUG:boto:Token: None 2024-04-12T04:28:48.509 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/ 2024-04-12T04:28:48.509 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:28:48.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:28:48.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b74d20a674036cd6-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.514 DEBUG:boto:path=/dkyges-66/obj37 2024-04-12T04:28:48.514 DEBUG:boto:auth_path=/dkyges-66/obj37 2024-04-12T04:28:48.514 DEBUG:boto:Method: PUT 2024-04-12T04:28:48.514 DEBUG:boto:Path: /dkyges-66/obj37 2024-04-12T04:28:48.514 DEBUG:boto:Data: 2024-04-12T04:28:48.514 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:48.514 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.514 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.515 DEBUG:boto:Params: {} 2024-04-12T04:28:48.515 DEBUG:boto:Token: None 2024-04-12T04:28:48.515 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/obj37 2024-04-12T04:28:48.515 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:p9J7F5o4phmTxVuMJe74Vtcq0h0= 2024-04-12T04:28:48.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p9J7F5o4phmTxVuMJe74Vtcq0h0='} 2024-04-12T04:28:48.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c048e232e0f0837-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.559 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:48.559 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:48.559 DEBUG:boto:Method: HEAD 2024-04-12T04:28:48.559 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:48.559 DEBUG:boto:Data: 2024-04-12T04:28:48.559 DEBUG:boto:Headers: {} 2024-04-12T04:28:48.559 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.559 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.559 DEBUG:boto:Params: {} 2024-04-12T04:28:48.560 DEBUG:boto:Token: None 2024-04-12T04:28:48.560 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/ 2024-04-12T04:28:48.560 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:28:48.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:28:48.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000917e52ed32f5a049-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.564 DEBUG:boto:path=/dkyges-66/obj74 2024-04-12T04:28:48.564 DEBUG:boto:auth_path=/dkyges-66/obj74 2024-04-12T04:28:48.565 DEBUG:boto:Method: PUT 2024-04-12T04:28:48.565 DEBUG:boto:Path: /dkyges-66/obj74 2024-04-12T04:28:48.565 DEBUG:boto:Data: 2024-04-12T04:28:48.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:48.565 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.565 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.565 DEBUG:boto:Params: {} 2024-04-12T04:28:48.565 DEBUG:boto:Token: None 2024-04-12T04:28:48.565 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/obj74 2024-04-12T04:28:48.565 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0qYlQiatzitV9E6iwimEQi8v+AY= 2024-04-12T04:28:48.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0qYlQiatzitV9E6iwimEQi8v+AY='} 2024-04-12T04:28:48.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073b21b8415923f12-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.572 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:48.572 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:48.572 DEBUG:boto:Method: HEAD 2024-04-12T04:28:48.572 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:48.573 DEBUG:boto:Data: 2024-04-12T04:28:48.573 DEBUG:boto:Headers: {} 2024-04-12T04:28:48.573 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.573 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.573 DEBUG:boto:Params: {} 2024-04-12T04:28:48.573 DEBUG:boto:Token: None 2024-04-12T04:28:48.573 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/ 2024-04-12T04:28:48.573 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:28:48.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:28:48.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5ab893bcad32d79-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.577 DEBUG:boto:path=/dkyges-66/obj111 2024-04-12T04:28:48.577 DEBUG:boto:auth_path=/dkyges-66/obj111 2024-04-12T04:28:48.578 DEBUG:boto:Method: PUT 2024-04-12T04:28:48.578 DEBUG:boto:Path: /dkyges-66/obj111 2024-04-12T04:28:48.578 DEBUG:boto:Data: 2024-04-12T04:28:48.578 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:48.578 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.578 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.578 DEBUG:boto:Params: {} 2024-04-12T04:28:48.578 DEBUG:boto:Token: None 2024-04-12T04:28:48.578 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/obj111 2024-04-12T04:28:48.579 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q8wp7b4aNvLxHeAJiCYXTmjfl28= 2024-04-12T04:28:48.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q8wp7b4aNvLxHeAJiCYXTmjfl28='} 2024-04-12T04:28:48.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061ed2aaeab674ea1-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.623 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:48.623 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:48.623 DEBUG:boto:Method: HEAD 2024-04-12T04:28:48.623 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:48.623 DEBUG:boto:Data: 2024-04-12T04:28:48.623 DEBUG:boto:Headers: {} 2024-04-12T04:28:48.623 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.623 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.623 DEBUG:boto:Params: {} 2024-04-12T04:28:48.624 DEBUG:boto:Token: None 2024-04-12T04:28:48.624 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/ 2024-04-12T04:28:48.624 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:28:48.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:28:48.628 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bad8b643e7ec4880-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.628 DEBUG:boto:path=/dkyges-66/obj148 2024-04-12T04:28:48.628 DEBUG:boto:auth_path=/dkyges-66/obj148 2024-04-12T04:28:48.628 DEBUG:boto:Method: PUT 2024-04-12T04:28:48.629 DEBUG:boto:Path: /dkyges-66/obj148 2024-04-12T04:28:48.629 DEBUG:boto:Data: 2024-04-12T04:28:48.629 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:48.629 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.629 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.629 DEBUG:boto:Params: {} 2024-04-12T04:28:48.629 DEBUG:boto:Token: None 2024-04-12T04:28:48.629 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/obj148 2024-04-12T04:28:48.629 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6NPl2BEKmtMClzU3ZU2NDXlHSPw= 2024-04-12T04:28:48.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6NPl2BEKmtMClzU3ZU2NDXlHSPw='} 2024-04-12T04:28:48.636 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005aad8a2803ddcb2b-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.636 DEBUG:boto:path=/dkyges-66/ 2024-04-12T04:28:48.637 DEBUG:boto:auth_path=/dkyges-66/ 2024-04-12T04:28:48.637 DEBUG:boto:Method: HEAD 2024-04-12T04:28:48.637 DEBUG:boto:Path: /dkyges-66/ 2024-04-12T04:28:48.637 DEBUG:boto:Data: 2024-04-12T04:28:48.637 DEBUG:boto:Headers: {} 2024-04-12T04:28:48.637 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.637 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.637 DEBUG:boto:Params: {} 2024-04-12T04:28:48.637 DEBUG:boto:Token: None 2024-04-12T04:28:48.638 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/ 2024-04-12T04:28:48.638 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:28:48.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:28:48.642 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f3ff2a9e1a9d6e7-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.642 DEBUG:boto:path=/dkyges-66/obj185 2024-04-12T04:28:48.642 DEBUG:boto:auth_path=/dkyges-66/obj185 2024-04-12T04:28:48.642 DEBUG:boto:Method: PUT 2024-04-12T04:28:48.642 DEBUG:boto:Path: /dkyges-66/obj185 2024-04-12T04:28:48.643 DEBUG:boto:Data: 2024-04-12T04:28:48.643 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:28:48.643 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:28:48.643 DEBUG:boto:Port: 8000 2024-04-12T04:28:48.643 DEBUG:boto:Params: {} 2024-04-12T04:28:48.643 DEBUG:boto:Token: None 2024-04-12T04:28:48.643 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 12 Apr 2024 04:28:48 GMT /dkyges-66/obj185 2024-04-12T04:28:48.643 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:5B98DKq84jPGpJZKLj6QnAvC1io= 2024-04-12T04:28:48.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5B98DKq84jPGpJZKLj6QnAvC1io='} 2024-04-12T04:28:48.687 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f067ca2c1a78bd72-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:28:48.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:48.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:28:48.687 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:28:48.749 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:48.749 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:48.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:48.795+0000 7f6db40e9780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-12T04:28:48.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:48.796+0000 7f6db40e9780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-12T04:28:48.817 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-04-12T04:28:48.818 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T04:28:48.820 INFO:tasks.rgw.c1.client.1:Started 2024-04-12T04:28:48.820 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:48.843 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:28:48.843 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T04:28:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-12T04:28:48.845 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi043.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T04:28:49.846 DEBUG:teuthology.orchestra.run.smithi043:> curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T04:28:49.872 INFO:teuthology.orchestra.run.smithi043.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:28:49.872 INFO:teuthology.orchestra.run.smithi043.stderr: Dload Upload Total Spent Left Speed 2024-04-12T04:28:49.875 INFO:teuthology.orchestra.run.smithi043.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-12T04:28:49.875 INFO:teuthology.orchestra.run.smithi043.stdout:anonymous 2024-04-12T04:28:49.876 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=dkyges-66 2024-04-12T04:28:49.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:28:49.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:28:49.876 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-66 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:28:49.982 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:28:49.982 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:28:50.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:50.035+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:28:55.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:28:55.036+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:29:00.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:29:00.037+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:29:05.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:29:05.037+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:29:10.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:29:10.038+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:29:15.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:29:15.039+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:29:20.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:29:20.040+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:29:25.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:29:25.041+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:29:30.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:29:30.041+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:29:35.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:29:35.041+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:29:40.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:29:40.042+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:29:45.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:29:45.043+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:29:50.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:29:50.043+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:29:55.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:29:55.044+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:30:00.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:30:00.045+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:30:05.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:30:05.045+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:30:10.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:30:10.046+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:30:15.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:30:15.047+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:30:20.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:30:20.047+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:30:25.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:30:25.048+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:30:30.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:30:30.049+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:30:35.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:30:35.050+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:30:36.084 INFO:tasks.rgw.c1.client.0.smithi043.stdout:2024-04-12T04:30:36.088+0000 7f1e272bd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:30:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:30:40.051+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:30:45.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:30:45.051+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:30:50.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:30:50.051+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:30:55.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:30:55.052+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:31:00.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:31:00.053+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:31:05.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:31:05.054+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:31:10.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:31:10.054+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:31:15.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:31:15.055+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:31:20.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:31:20.056+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:31:25.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:31:25.056+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:31:30.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:31:30.057+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:31:35.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:31:35.058+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:31:40.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:31:40.058+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:31:45.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:31:45.059+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:31:50.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:31:50.060+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:31:55.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:31:55.061+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:32:00.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:32:00.061+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:32:05.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:32:05.062+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:32:10.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:32:10.062+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:32:15.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:32:15.063+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:32:20.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:32:20.064+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:32:25.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:32:25.065+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:32:30.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:32:30.065+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:32:35.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:32:35.066+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:32:40.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:32:40.067+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:32:45.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:32:45.067+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:32:50.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:32:50.068+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:32:55.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:32:55.068+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:33:00.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:00.068+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:33:05.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:05.069+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:33:10.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:10.071+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:33:15.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:15.070+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:33:20.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:20.071+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:33:25.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:25.072+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:33:30.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:30.072+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:33:35.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:35.073+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:33:40.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:40.073+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:33:45.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:45.074+0000 7f552acc1780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2024-04-12T04:33:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:45.074+0000 7f552acc1780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:33:45.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:45.074+0000 7f552acc1780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:33:45.093 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-12T04:33:45.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2024-04-12T04:33:45.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:45.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:45.098 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:45.157 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:45.158 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:45.199 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:33:45.215 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:33:45.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:45.215 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:33:45.215 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:33:45.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:45.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:45.216 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:33:45.216 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:33:45.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:45.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:45.216 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:33:45.216 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:33:45.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:45.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:45.216 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:33:45.216 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:28:48.449153Z" 2024-04-12T04:33:45.217 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:45.217 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:33:45.218 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:33:45.218 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:33:45.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:45.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:45.218 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:45.274 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:45.275 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:45.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:45.311+0000 7f075263f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:33:45.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:45.313+0000 7f075263f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:33:45.326 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:33:45.326 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:33:45.327 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:33:45.327 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:33:45.327 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:33:45.327 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:33:45.327 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:45.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:45.327 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:33:45.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:45.327 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:33:45.327 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:33:45.327 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:33:45.328 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:33:45.328 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:33:45.328 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:33:45.328 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:33:45.328 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:33:45.328 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:45.328 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:45.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:45.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:33:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:33:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:33:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:33:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:33:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:33:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:33:45.329 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:33:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:33:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:33:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:33:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:33:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:33:45.330 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:33:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:33:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:33:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:45.331 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:33:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:33:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:33:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:33:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:33:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:33:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:33:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:33:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:45.332 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:33:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:33:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:33:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:45.333 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:33:45Z" 2024-04-12T04:33:45.333 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:33:45.334 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:33:45Z" } 2024-04-12T04:33:45.335 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:33:45.335 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:33:45.335 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:33:45.335 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:33:45.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:45.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:45.336 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:45.389 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:45.389 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:45.443 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[]} 2024-04-12T04:33:45.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:45.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:45.443 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:45.498 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:45.498 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:45.555 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:33:45.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:45.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:45.555 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:45.611 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:45.611 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:45.665 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-12T04:33:45.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:45.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:45.666 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:45.719 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:45.719 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:45.773 INFO:teuthology.orchestra.run.smithi043.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2024-04-12T04:33:45.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:45.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:45.774 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:45.824 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:45.825 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:45.878 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-12T04:33:45.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:45.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:45.879 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:45.937 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:45.938 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:45.995 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:33:45.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:45.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:45.995 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:46.055 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:46.055 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:46.110 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:33:46.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:46.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:46.111 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:46.169 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:46.170 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:46.227 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:33:46.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:46.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:46.228 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:46.280 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:46.280 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:46.333 INFO:teuthology.orchestra.run.smithi043.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2024-04-12T04:33:46.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:46.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:46.334 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:46.389 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:46.389 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:46.452 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:33:46.452 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:33:46.452 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 13, 2024-04-12T04:33:46.452 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T04:33:46.452 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T04:33:46.452 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T04:33:46.452 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:33:46.452 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T04:33:46.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:46.453 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T04:33:46.453 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T04:33:46.453 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:33:46.453 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T04:33:46.453 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:33:46.453 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:33:46.453 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:33:46.453 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:33:46.453 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:33:46.453 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:33:46.453 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:33:46.454 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:46.454 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:33:46.454 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T04:33:46.454 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:33:46.454 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T04:33:46.454 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:33:46.454 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:33:46.454 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:33:46.454 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:33:46.454 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:33:46.454 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:33:46.455 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:33:46.455 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:33:46.455 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:33:46.455 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:33:46.455 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:33:46.455 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:33:46.455 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:46.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:46.455 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:33:46.455 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T04:33:46.455 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:33:46.456 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:33:46.456 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:33:46.456 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:33:46.456 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:33:46.456 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:33:46.456 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:33:46.456 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:33:46.456 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:33:46.456 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:33:46.456 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:33:46.456 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:33:46.456 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:33:46.457 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:33:46.457 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:46.457 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:46.457 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:33:46.457 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:33:46.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:46.457 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:33:46.457 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:33:46.457 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:33:46.457 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:33:46.457 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:46.458 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:46.458 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:33:46.458 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:33:46.458 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:33:46.458 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:33:46.458 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [ 2024-04-12T04:33:46.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:46.458 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-group", 2024-04-12T04:33:46.458 INFO:teuthology.orchestra.run.smithi043.stdout: "data_flow": { 2024-04-12T04:33:46.458 INFO:teuthology.orchestra.run.smithi043.stdout: "symmetrical": [ 2024-04-12T04:33:46.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:46.458 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-flow1", 2024-04-12T04:33:46.459 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:33:46.459 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:33:46.459 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:33:46.459 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:46.459 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:46.459 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:33:46.459 INFO:teuthology.orchestra.run.smithi043.stdout: "directional": [ 2024-04-12T04:33:46.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:46.459 INFO:teuthology.orchestra.run.smithi043.stdout: "source_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:33:46.459 INFO:teuthology.orchestra.run.smithi043.stdout: "dest_zone": "1ba046e0-d0ae-437d-b741-28b4299423c2" 2024-04-12T04:33:46.459 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:46.460 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:46.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.460 INFO:teuthology.orchestra.run.smithi043.stdout: "pipes": [ 2024-04-12T04:33:46.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:46.460 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-pipe", 2024-04-12T04:33:46.460 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:33:46.460 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "*", 2024-04-12T04:33:46.460 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:33:46.460 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:33:46.460 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:33:46.460 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:46.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.461 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": { 2024-04-12T04:33:46.461 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "*", 2024-04-12T04:33:46.461 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:33:46.461 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:33:46.461 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:33:46.461 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:46.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.461 INFO:teuthology.orchestra.run.smithi043.stdout: "params": { 2024-04-12T04:33:46.461 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:33:46.461 INFO:teuthology.orchestra.run.smithi043.stdout: "filter": { 2024-04-12T04:33:46.461 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [] 2024-04-12T04:33:46.461 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:46.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.462 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": {}, 2024-04-12T04:33:46.462 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-12T04:33:46.462 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "system", 2024-04-12T04:33:46.462 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "" 2024-04-12T04:33:46.462 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:46.462 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:46.462 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:33:46.462 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "enabled" 2024-04-12T04:33:46.462 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:46.462 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:46.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.463 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:33:46.463 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:33:46.463 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:46.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:46.463 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T04:33:46.463 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T04:33:46.463 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:33:46.463 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T04:33:46.463 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:33:46.463 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:33:46.464 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:33:46.464 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:33:46.464 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:33:46.464 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:33:46.464 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:33:46.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:46.464 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:33:46.464 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T04:33:46.464 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:33:46.464 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:33:46.464 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:33:46.465 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:33:46.465 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:33:46.465 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:33:46.465 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:33:46.465 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:33:46.465 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:33:46.465 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:33:46.465 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:33:46.465 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:33:46.465 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:33:46.465 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:33:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:33:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T04:33:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:33:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T04:33:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:33:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:33:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:33:46.466 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:33:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:33:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:33:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:33:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:33:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:33:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:33:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:33:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:33:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:33:46.467 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:33:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:33:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:33:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:33:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:33:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:33:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:33:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:33:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:33:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T04:33:46.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.469 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:33:46.469 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:33:46.469 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:46.469 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:46.469 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:33:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T04:33:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:33:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T04:33:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:33:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T04:33:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:46.473 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:33:46.474 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T04:33:46.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.474 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:46.474 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:33:46.474 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T04:33:46.474 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:46.474 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:46.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.474 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T04:33:46.474 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:33:46.475 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T04:33:46.475 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T04:33:46.475 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:33:46.475 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:33:46.475 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:33:46.475 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:33:46.475 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:33:46.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.475 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T04:33:46.475 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:33:46.476 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:33:46.476 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:33:46.476 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:33:46.476 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:33:46.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.476 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T04:33:46.476 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:33:46.476 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:33:46.476 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:33:46.476 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:33:46.476 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:33:46.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.477 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T04:33:46.477 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:33:46.477 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:33:46.477 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:33:46.477 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:33:46.477 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:33:46.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.477 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T04:33:46.477 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:33:46.477 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:33:46.478 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:33:46.478 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:33:46.478 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:33:46.478 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:46.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:46.478 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:33:46.478 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T04:33:46.478 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:46.478 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:33:46.479 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-67 2024-04-12T04:33:46.480 DEBUG:boto:path=/dkyges-67/ 2024-04-12T04:33:46.480 DEBUG:boto:auth_path=/dkyges-67/ 2024-04-12T04:33:46.480 DEBUG:boto:Method: PUT 2024-04-12T04:33:46.480 DEBUG:boto:Path: /dkyges-67/ 2024-04-12T04:33:46.480 DEBUG:boto:Data: 2024-04-12T04:33:46.480 DEBUG:boto:Headers: {} 2024-04-12T04:33:46.480 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:33:46.481 DEBUG:boto:Port: 8000 2024-04-12T04:33:46.481 DEBUG:boto:Params: {} 2024-04-12T04:33:46.481 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:33:46.481 DEBUG:boto:Token: None 2024-04-12T04:33:46.482 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:33:46 GMT /dkyges-67/ 2024-04-12T04:33:46.482 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DU/vSzxPXJFF8Ej5iuO/vuUWtAA= 2024-04-12T04:33:46.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:33:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DU/vSzxPXJFF8Ej5iuO/vuUWtAA=', 'Content-Length': '0'} 2024-04-12T04:33:46.719 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000edb371db0925c042-006618b9aa-22c1-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:33:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:33:46.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:46.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:46.720 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dkyges-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:46.800 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:46.800 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:47.006 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:33:47.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:47.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:47.007 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dkyges-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:47.097 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:47.097 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:47.235 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-12T04:33:47.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:47.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:47.236 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket dkyges-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:47.310 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:47.310 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:47.394 INFO:teuthology.orchestra.run.smithi043.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2024-04-12T04:33:47.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:47.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:47.395 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dkyges-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:47.459 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:47.459 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:47.524 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-12T04:33:47.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:47.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:47.525 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket dkyges-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:47.586 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:47.586 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:47.644 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:33:47.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:47.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:47.644 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket dkyges-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:47.706 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:47.707 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:47.766 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:33:47.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:47.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:47.767 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=dkyges-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:47.828 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:47.829 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:47.890 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:33:47.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:47.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:33:47.890 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket dkyges-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:33:47.946 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:47.947 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:48.003 INFO:teuthology.orchestra.run.smithi043.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2024-04-12T04:33:48.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:48.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:48.003 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:48.057 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:48.057 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:48.094 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:33:48.111 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:33:48.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:48.111 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:33:48.111 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:33:48.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:48.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:48.112 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:33:48.112 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:33:48.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:48.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:48.112 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:33:48.112 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:33:48.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:48.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:48.112 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896427.875736_849.1", 2024-04-12T04:33:48.112 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:33:47.875736Z" 2024-04-12T04:33:48.113 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:48.113 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:33:48.114 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:33:48.114 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:33:48.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:48.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:48.114 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:48.168 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:48.169 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:48.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:48.207+0000 7f9549a7f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:33:48.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:48.208+0000 7f9549a7f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:33:48.221 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:33:48.221 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:33:48.221 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:33:48.222 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:33:48.222 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:33:48.222 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:33:48.222 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:48.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:48.222 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:33:48.222 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:48.222 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:33:48.222 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:33:48.222 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:33:48.222 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:33:48.222 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:33:48.223 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:33:48.223 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:33:48.223 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:33:48.223 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:48.223 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:48.223 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:48.223 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:48.223 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:33:48.223 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:33:48.223 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:33:48.223 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:33:48.223 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:33:48.223 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:33:48.224 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:33:48.224 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:33:48.224 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:48.224 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:48.224 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:48.224 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:48.224 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:33:48.224 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:33:48.224 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:33:48.224 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:33:48.224 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:33:48.225 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:33:48.225 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:33:48.225 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:33:48.225 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:48.225 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:48.225 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:48.225 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:48.225 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:33:48.225 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:33:48.225 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:33:48.225 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:33:48.225 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:33:48.225 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:33:48.226 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:33:48.226 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:33:48.226 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:48.226 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:48.226 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:48.226 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:48.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:48.226 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:33:48.226 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:33:48.226 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:33:48.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:48.227 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:33:48Z" 2024-04-12T04:33:48.227 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:33:48.227 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:33:48Z" } 2024-04-12T04:33:48.228 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:33:48.228 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:33:48.228 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:33:48.228 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:33:53.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:53.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:53.229 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:53.291 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:53.291 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:53.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:53.330+0000 7fb920d91780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:33:53.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:53.331+0000 7fb920d91780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:33:53.345 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:33:53.345 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:33:53.346 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:33:53.346 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:33:53.346 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:33:53.346 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:33:53.346 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:53.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:53.346 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:33:53.346 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:53.346 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:33:53.346 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:33:53.347 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:33:53.347 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:33:53.347 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:33:53.347 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:33:53.347 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:33:53.347 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:33:53.347 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:53.347 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:53.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:53.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:53.348 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:33:53.348 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:33:53.348 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:33:53.348 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:33:53.348 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:33:53.348 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:33:53.348 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:33:53.348 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:33:53.348 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:53.348 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:53.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:53.349 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:53.349 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:33:53.349 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:33:53.349 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:33:53.349 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:33:53.349 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:33:53.349 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:33:53.349 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:33:53.349 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:33:53.349 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:53.349 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:53.350 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:53.350 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:53.350 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:33:53.350 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:33:53.350 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:33:53.350 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:33:53.350 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:33:53.350 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:33:53.350 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:33:53.350 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:33:53.350 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:53.351 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:53.351 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:53.351 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:53.351 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:53.351 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:33:53.351 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:33:53.351 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:33:53.351 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:53.351 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:33:53Z" 2024-04-12T04:33:53.351 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:33:53.352 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:33:53Z" } 2024-04-12T04:33:53.352 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:33:53.353 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:33:53.353 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:33:53.353 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:33:58.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:58.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:33:58.354 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:33:58.414 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:33:58.415 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:33:58.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:58.454+0000 7f20393de780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:33:58.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:33:58.456+0000 7f20393de780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:33:58.470 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:33:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:33:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:33:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:33:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:33:58.470 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:33:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:33:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:33:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:33:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:33:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:33:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:33:58.471 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:33:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:33:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:33:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:33:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:33:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:33:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:33:58.472 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:33:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:33:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:33:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:33:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:58.473 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:33:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:33:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:33:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:33:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:33:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:33:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:33:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:33:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:58.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:33:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:33:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:33:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:33:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:33:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:33:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:33:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:33:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:33:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:33:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:58.475 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:33:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:33:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:33:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:33:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:33:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:33:58.476 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:33:58Z" 2024-04-12T04:33:58.476 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:33:58.477 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:33:58Z" } 2024-04-12T04:33:58.477 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:33:58.477 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:33:58.478 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:33:58.478 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:34:03.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:03.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:03.479 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:03.540 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:03.540 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:03.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:03.581+0000 7f9e1b83b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:03.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:03.582+0000 7f9e1b83b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:03.595 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:34:03.596 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:34:03.596 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:34:03.596 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:34:03.596 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:34:03.596 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:34:03.596 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:03.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:03.596 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:34:03.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:03.596 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:34:03.597 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:03.597 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:03.597 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:34:03.597 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:03.597 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:34:03.597 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:03.597 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:34:03.597 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:03.597 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:03.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:03.597 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:03.598 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:34:03.598 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:03.598 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:03.598 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:34:03.598 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:03.598 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:03.598 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:03.598 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:34:03.598 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:03.598 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:03.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:03.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:03.599 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:34:03.599 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:03.599 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:03.599 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:34:03.599 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:03.599 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:03.599 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:03.599 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:34:03.599 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:03.599 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:03.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:03.600 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:03.600 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:34:03.600 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:03.600 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:03.600 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:34:03.600 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:03.600 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:03.600 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:03.600 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:34:03.600 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:03.601 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:03.601 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:03.601 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:34:03.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:03.601 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:34:03.601 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:34:03.601 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:34:03.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:03.601 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:34:03Z" 2024-04-12T04:34:03.601 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:34:03.602 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:34:03Z" } 2024-04-12T04:34:03.602 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:34:03.603 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:34:03.603 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:34:03.603 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:34:08.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:08.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:08.604 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:08.663 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:08.663 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:08.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:08.702+0000 7f30cf742780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:08.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:08.703+0000 7f30cf742780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:08.722 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:34:08.722 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:34:08.722 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:34:08.722 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:34:08.722 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:34:08.722 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:34:08.722 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:08.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:08.723 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:34:08.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:08.723 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:34:08.723 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:08.723 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:08.723 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:34:08.723 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:08.723 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:34:08.723 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:08.723 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:34:08.723 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:08.723 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:08.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:08.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:08.724 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:34:08.724 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:08.724 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:08.724 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:34:08.724 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:08.724 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:08.724 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:08.724 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:34:08.724 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:08.725 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:08.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:08.725 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:08.725 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:34:08.725 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:08.725 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:08.725 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:34:08.725 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:08.725 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:08.725 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:08.725 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:34:08.726 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:08.726 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:08.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:08.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:08.726 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:34:08.726 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:08.726 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:08.726 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:34:08.726 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:08.726 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:08.726 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:08.726 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:34:08.727 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:08.727 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:08.727 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:08.727 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:34:08.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:08.727 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:34:08.727 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:34:08.727 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:34:08.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:08.727 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:34:08Z" 2024-04-12T04:34:08.727 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:34:08.728 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:34:08Z" } 2024-04-12T04:34:08.728 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:34:08.728 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:34:08.729 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:34:08.729 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:34:13.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:13.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:13.730 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:13.790 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:13.790 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:13.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:13.829+0000 7f14a4220780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:13.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:13.830+0000 7f14a4220780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:13.844 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:34:13.844 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:34:13.844 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:34:13.844 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:34:13.844 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:34:13.845 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:34:13.845 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:13.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:13.845 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:34:13.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:13.845 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:34:13.845 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:13.845 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:13.845 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:34:13.845 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:13.845 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:34:13.846 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:13.846 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:34:13.846 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:13.846 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:13.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:13.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:13.846 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:34:13.846 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:13.846 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:13.846 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:34:13.846 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:13.847 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:13.847 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:13.847 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:34:13.847 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:13.847 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:13.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:13.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:13.847 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:34:13.847 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:13.847 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:13.847 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:34:13.848 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:13.848 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:13.848 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:13.848 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:34:13.848 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:13.848 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:13.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:13.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:13.848 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:34:13.848 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:13.849 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:13.849 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:34:13.849 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:13.849 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:13.849 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:13.849 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:34:13.849 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:13.849 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:13.849 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:13.849 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:34:13.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:13.849 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:34:13.850 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:34:13.850 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:34:13.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:13.850 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:34:13Z" 2024-04-12T04:34:13.850 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:34:13.851 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:34:13Z" } 2024-04-12T04:34:13.851 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:34:13.851 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:34:13.851 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:34:13.851 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:34:18.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:18.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:18.853 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:18.913 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:18.913 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:18.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:18.951+0000 7f28c6d24780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:18.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:18.952+0000 7f28c6d24780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:18.966 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:34:18.966 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:34:18.967 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:34:18.967 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:34:18.967 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:34:18.967 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:34:18.967 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:18.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:18.967 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:34:18.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:18.967 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:34:18.967 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:18.968 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:18.968 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:34:18.968 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:18.968 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:34:18.968 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:18.968 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:34:18.968 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:18.968 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:18.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:18.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:18.968 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:34:18.969 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:18.969 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:18.969 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:34:18.969 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:18.969 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:18.969 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:18.969 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:34:18.969 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:18.969 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:18.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:18.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:18.969 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:34:18.970 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:18.970 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:18.970 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:34:18.970 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:18.970 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:18.970 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:18.970 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:34:18.970 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:18.970 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:18.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:18.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:18.971 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:34:18.971 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:18.971 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:18.971 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:34:18.971 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:18.971 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:18.971 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:18.971 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:34:18.971 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:18.971 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:18.971 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:18.972 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:34:18.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:18.972 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:34:18.972 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:34:18.972 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:34:18.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:18.972 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:34:18Z" 2024-04-12T04:34:18.972 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:34:18.973 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:34:18Z" } 2024-04-12T04:34:18.973 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:34:18.973 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:34:18.974 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:34:18.974 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:34:23.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:23.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:23.975 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:24.073+0000 7fbdb8edb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:24.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:24.074+0000 7fbdb8edb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:24.089 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:34:24.089 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:34:24.089 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:34:24.089 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:34:24.089 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:34:24.089 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:34:24.089 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:24.089 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:24.089 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:34:24.090 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:24.090 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:34:24.090 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:24.090 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:24.090 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:34:24.090 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:24.090 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:34:24.090 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:24.090 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:34:24.090 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:24.091 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:24.091 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:24.091 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:24.091 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:34:24.091 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:24.091 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:24.091 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:34:24.091 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:24.091 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:24.091 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:24.091 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:34:24.092 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:24.092 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:24.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:24.092 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:24.092 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:34:24.092 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:24.092 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:24.092 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:34:24.092 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:24.092 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:24.092 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:24.092 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:34:24.093 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:24.093 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:24.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:24.093 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:24.093 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:34:24.093 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:24.093 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:24.093 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:34:24.093 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:24.093 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:24.093 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:24.094 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:34:24.094 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:24.094 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:24.094 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:24.094 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:34:24.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:24.094 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:34:24.094 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:34:24.094 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:34:24.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:24.094 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:34:24Z" 2024-04-12T04:34:24.095 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:34:24.095 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:34:24Z" } 2024-04-12T04:34:24.096 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:34:24.096 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:34:24.096 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:34:24.096 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:34:29.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:29.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:29.098 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:29.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:29.198+0000 7fa1335b0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:29.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:29.200+0000 7fa1335b0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:29.214 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:34:29.214 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:34:29.214 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:34:29.214 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:34:29.214 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:34:29.214 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:34:29.215 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:29.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:29.215 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:34:29.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:29.215 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:34:29.215 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:29.215 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:29.215 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:34:29.215 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:29.215 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:34:29.215 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:29.216 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:34:29.216 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:29.216 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:29.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:29.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:29.216 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:34:29.216 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:29.216 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:29.216 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:34:29.216 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:29.216 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:29.216 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:29.216 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:34:29.217 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:29.217 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:29.217 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:29.217 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:29.217 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:34:29.217 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:29.217 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:29.217 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:34:29.217 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:29.217 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:29.217 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:29.218 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:34:29.218 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:29.218 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:29.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:29.218 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:29.218 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:34:29.218 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:29.218 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:29.218 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:34:29.218 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:29.218 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:29.219 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:29.219 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:34:29.219 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:29.219 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:29.219 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:29.219 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:34:29.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:29.219 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:34:29.219 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:34:29.219 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:34:29.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:29.219 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:34:29Z" 2024-04-12T04:34:29.220 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:34:29.220 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:34:29Z" } 2024-04-12T04:34:29.221 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:34:29.221 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:34:29.221 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:34:29.221 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:34:34.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:34.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:34.222 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:34.282 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:34.282 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:34.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:34.321+0000 7f3627a92780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:34.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:34.322+0000 7f3627a92780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:34.336 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:34:34.336 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:34:34.337 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:34:34.337 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:34:34.337 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:34:34.337 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:34:34.337 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:34.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:34.337 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:34:34.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:34.337 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:34:34.337 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:34.337 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:34.338 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:34:34.338 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:34.338 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:34:34.338 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:34.338 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:34:34.338 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:34.338 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:34.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:34.338 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:34.338 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:34:34.338 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:34.339 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:34.339 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:34:34.339 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:34.339 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:34.339 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:34.339 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:34:34.339 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:34.339 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:34.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:34.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:34.339 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:34:34.340 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:34.340 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:34.340 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:34:34.340 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:34.340 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:34.340 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:34.340 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:34:34.340 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:34.340 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:34.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:34.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:34.341 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:34:34.341 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:34.341 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:34.341 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:34:34.341 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:34.341 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:34.341 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:34.341 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:34:34.341 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:34.341 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:34.341 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:34.342 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:34:34.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:34.342 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:34:34.342 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:34:34.342 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:34:34.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:34.342 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:34:34Z" 2024-04-12T04:34:34.342 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:34:34.343 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:34:34Z" } 2024-04-12T04:34:34.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:34:34.344 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:34:34.344 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:34:34.344 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:34:39.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:39.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:39.345 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:39.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:39.445+0000 7fede9559780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:39.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:39.447+0000 7fede9559780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:39.461 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:34:39.461 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:34:39.461 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:34:39.462 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:34:39.462 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:34:39.462 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:34:39.462 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:39.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:39.462 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:34:39.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:39.462 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:34:39.462 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:39.462 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:39.463 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:34:39.463 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:39.463 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:34:39.463 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:39.463 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:34:39.463 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:39.463 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:39.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:39.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:39.463 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:34:39.463 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:39.464 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:39.464 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:34:39.464 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:39.464 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:39.464 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:39.464 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:34:39.464 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:39.464 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:39.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:39.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:39.464 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:34:39.465 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:39.465 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:39.465 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:34:39.465 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:39.465 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:39.465 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:39.465 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:34:39.465 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:39.465 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:39.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:39.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:39.466 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:34:39.466 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:39.466 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:39.466 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:34:39.466 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:39.466 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:39.466 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:39.466 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:34:39.466 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:39.466 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:39.466 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:39.467 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:34:39.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:39.467 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:34:39.467 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:34:39.467 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:34:39.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:39.467 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:34:39Z" 2024-04-12T04:34:39.467 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:34:39.468 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:34:39Z" } 2024-04-12T04:34:39.468 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:34:39.468 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:34:39.468 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:34:39.469 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:34:44.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:44.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:44.470 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:44.529 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:44.529 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:44.567+0000 7f9da005e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:44.568+0000 7f9da005e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:44.583 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:34:44.583 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:34:44.583 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:34:44.583 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:34:44.583 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:34:44.583 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:34:44.583 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:44.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:44.584 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:34:44.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:44.584 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:34:44.584 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:44.584 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:44.584 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:34:44.584 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:44.584 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:34:44.584 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:44.585 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:34:44.585 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:44.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:44.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:44.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:44.585 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:34:44.585 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:44.585 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:44.585 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:34:44.585 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:44.585 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:44.586 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:44.586 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:34:44.586 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:44.586 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:44.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:44.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:44.586 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:34:44.586 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:44.586 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:44.586 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:34:44.587 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:44.587 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:44.587 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:44.587 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:34:44.587 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:44.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:44.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:44.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:44.588 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:34:44.588 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:44.588 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:44.588 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:34:44.588 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:44.588 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:44.588 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:44.589 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:34:44.589 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:44.589 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:44.589 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:44.589 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:34:44.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:44.589 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:34:44.589 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:34:44.589 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:34:44.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:44.589 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:34:44Z" 2024-04-12T04:34:44.589 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:34:44.590 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:34:44Z" } 2024-04-12T04:34:44.591 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:34:44.591 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:34:44.591 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:34:44.591 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:34:49.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:49.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:49.593 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:49.651 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:49.652 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:49.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:49.689+0000 7f81a0801780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:49.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:49.690+0000 7f81a0801780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:49.704 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:34:49.705 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:34:49.705 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:34:49.705 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:34:49.705 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:34:49.705 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:34:49.705 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:49.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:49.705 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:34:49.705 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:49.705 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:34:49.705 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:49.706 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:49.706 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:34:49.706 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:49.706 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:34:49.706 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:49.706 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:34:49.706 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:49.706 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:49.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:49.707 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:49.707 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:34:49.707 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:49.707 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:49.707 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:34:49.707 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:49.707 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:49.707 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:49.707 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:34:49.707 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:49.707 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:49.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:49.708 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:49.708 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:34:49.708 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:49.708 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:49.708 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:34:49.708 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:49.708 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:49.708 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:49.708 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:34:49.708 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:49.708 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:49.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:49.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:49.709 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:34:49.709 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:49.709 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:49.709 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:34:49.709 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:49.709 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:49.709 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:49.709 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:34:49.709 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:49.709 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:49.710 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:49.710 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:34:49.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:49.710 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:34:49.710 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:34:49.710 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:34:49.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:49.710 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:34:49Z" 2024-04-12T04:34:49.710 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:34:49.711 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:34:49Z" } 2024-04-12T04:34:49.711 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:34:49.712 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:34:49.712 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:34:49.712 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:34:54.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:54.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:54.713 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:54.775 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:54.775 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:54.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:54.814+0000 7fc62cd5b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:54.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:54.815+0000 7fc62cd5b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:54.830 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:34:54.830 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:34:54.830 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:34:54.830 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:34:54.830 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:34:54.830 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:34:54.831 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:54.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:54.831 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:34:54.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:54.831 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:34:54.831 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:54.831 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:54.831 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:34:54.831 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:54.831 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:34:54.831 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:54.832 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:34:54.832 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:54.832 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:54.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:54.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:54.832 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:34:54.832 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:54.832 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:54.832 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:34:54.832 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:54.832 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:54.832 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:54.833 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:34:54.833 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:54.833 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:54.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:54.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:54.833 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:34:54.833 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:54.833 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:54.833 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:34:54.833 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:54.833 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:54.833 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:54.834 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:34:54.834 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:54.834 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:54.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:54.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:54.834 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:34:54.834 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:54.834 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:54.834 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:34:54.834 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:54.834 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:54.834 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:54.835 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:34:54.835 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:54.835 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:54.835 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:54.835 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:34:54.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:54.835 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:34:54.835 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:34:54.835 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:34:54.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:54.835 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:34:54Z" 2024-04-12T04:34:54.835 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:34:54.836 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:34:54Z" } 2024-04-12T04:34:54.837 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:34:54.837 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:34:54.837 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:34:54.837 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:34:59.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:59.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:34:59.839 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:34:59.898 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:34:59.898 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:34:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:59.937+0000 7f2f822ee780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:34:59.938+0000 7f2f822ee780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:34:59.952 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:34:59.952 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:34:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:34:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:34:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:34:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:34:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:34:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:34:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:34:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:34:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:34:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:34:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:34:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:34:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:34:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:34:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:34:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:34:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:34:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:34:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:34:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:34:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:34:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:34:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:34:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:34:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:34:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:34:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:34:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:34:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:34:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:34:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:34:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:34:59Z" 2024-04-12T04:34:59.958 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:34:59.959 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:34:59Z" } 2024-04-12T04:34:59.959 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:34:59.959 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:34:59.959 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:34:59.959 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:35:04.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:04.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:04.961 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:05.020 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:05.020 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:05.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:05.059+0000 7ff487e36780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:05.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:05.060+0000 7ff487e36780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:05.075 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:35:05.075 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:35:05.075 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:35:05.075 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:35:05.075 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:35:05.075 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:35:05.075 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:05.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:05.076 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:35:05.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:05.076 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:35:05.076 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:05.076 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:05.076 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:35:05.076 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:05.076 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:35:05.076 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:05.076 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:35:05.076 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:05.076 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:05.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:05.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:05.077 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:35:05.077 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:05.077 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:05.077 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:35:05.077 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:05.077 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:05.077 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:05.077 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:35:05.077 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:05.077 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:05.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:05.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:05.078 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:35:05.078 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:05.078 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:05.078 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:35:05.078 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:05.078 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:05.078 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:05.078 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:35:05.078 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:05.079 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:05.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:05.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:05.079 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:35:05.079 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:05.079 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:05.079 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:35:05.079 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:05.079 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:05.079 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:05.079 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:35:05.079 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:05.080 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:05.080 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:05.080 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:35:05.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:05.080 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:35:05.080 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:35:05.080 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:35:05.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:05.080 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:35:05Z" 2024-04-12T04:35:05.080 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:35:05.081 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:05Z" } 2024-04-12T04:35:05.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:35:05.082 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:35:05.082 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:35:05.082 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:35:10.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:10.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:10.083 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:10.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:10.180+0000 7f2d24316780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:10.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:10.181+0000 7f2d24316780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:10.197 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:35:10.198 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:35:10.198 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:35:10.198 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:35:10.198 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:35:10.198 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:35:10.198 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:10.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:10.198 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:35:10.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:10.198 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:35:10.199 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:10.199 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:10.199 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:35:10.199 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:10.199 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:35:10.199 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:10.199 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:35:10.199 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:10.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:10.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:10.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:10.200 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:35:10.200 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:10.200 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:10.200 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:35:10.200 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:10.200 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:10.200 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:10.200 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:35:10.200 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:10.200 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:10.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:10.200 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:10.200 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:35:10.201 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:10.201 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:10.201 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:35:10.201 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:10.201 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:10.201 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:10.201 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:35:10.201 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:10.201 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:10.201 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:10.201 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:10.201 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:35:10.202 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:10.202 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:10.202 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:35:10.202 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:10.202 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:10.202 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:10.202 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:35:10.202 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:10.202 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:10.202 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:10.203 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:35:10.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:10.203 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:35:10.203 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:35:10.203 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:35:10.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:10.203 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:35:10Z" 2024-04-12T04:35:10.203 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:35:10.204 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:10Z" } 2024-04-12T04:35:10.204 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:35:10.204 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:35:10.205 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:35:10.205 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:35:15.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:15.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:15.206 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:15.266 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:15.266 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:15.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:15.309+0000 7f6b601cf780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:15.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:15.310+0000 7f6b601cf780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:15.325 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:35:15.325 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:35:15.325 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:35:15.325 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:35:15.325 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:35:15.325 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:35:15.325 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:15.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:15.326 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:35:15.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:15.326 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:35:15.326 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:15.326 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:15.326 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:35:15.326 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:15.326 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:35:15.326 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:15.327 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:35:15.327 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:15.327 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:15.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:15.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:15.327 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:35:15.327 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:15.327 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:15.327 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:35:15.327 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:15.327 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:15.327 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:15.328 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:35:15.328 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:15.328 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:15.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:15.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:15.328 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:35:15.328 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:15.328 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:15.328 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:35:15.328 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:15.328 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:15.328 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:15.329 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:35:15.329 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:15.329 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:15.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:15.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:15.329 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:35:15.329 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:15.329 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:15.329 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:35:15.329 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:15.329 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:15.329 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:15.330 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:35:15.330 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:15.330 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:15.330 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:15.330 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:35:15.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:15.330 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:35:15.330 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:35:15.330 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:35:15.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:15.331 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:35:15Z" 2024-04-12T04:35:15.331 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:35:15.331 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:15Z" } 2024-04-12T04:35:15.332 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:35:15.332 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:35:15.332 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:35:15.332 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:35:18.035 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T04:35:18.035+0000 7fd539413700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:35:20.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:20.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:20.334 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:20.393 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:20.393 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:20.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:20.430+0000 7f2bd7b4f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:20.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:20.431+0000 7f2bd7b4f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:20.445 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:35:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:35:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:35:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:35:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:35:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:35:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:35:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:20.446 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:35:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:35:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:35:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:35:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:20.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:35:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:35:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:35:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:20.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:35:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:35:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:35:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:20.449 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:35:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:35:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:35:20.450 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:35:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:35:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:35:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:35:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:20.451 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:35:20Z" 2024-04-12T04:35:20.451 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:35:20.452 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:20Z" } 2024-04-12T04:35:20.453 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:35:20.453 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:35:20.453 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:35:20.453 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:35:25.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:25.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:25.454 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:25.516 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:25.516 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:25.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:25.554+0000 7f17ff5bf780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:25.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:25.555+0000 7f17ff5bf780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:25.570 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:35:25.570 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:35:25.570 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:35:25.570 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:35:25.570 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:35:25.571 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:35:25.571 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:25.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:25.571 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:35:25.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:25.571 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:35:25.571 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:25.571 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:25.571 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:35:25.571 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:25.571 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:35:25.571 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:25.572 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:35:25.572 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:25.572 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:25.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:25.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:25.572 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:35:25.572 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:25.572 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:25.572 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:35:25.572 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:25.572 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:25.573 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:25.573 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:35:25.573 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:25.573 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:25.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:25.573 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:25.573 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:35:25.573 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:25.573 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:25.573 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:35:25.573 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:25.574 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:25.574 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:25.574 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:35:25.574 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:25.574 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:25.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:25.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:25.574 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:35:25.574 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:25.574 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:25.574 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896128.449153_805.1", 2024-04-12T04:35:25.575 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:25.575 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:25.575 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:25.575 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.449153Z", 2024-04-12T04:35:25.575 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:25.575 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:25.575 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:25.575 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:35:25.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:25.575 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:35:25.575 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:35:25.575 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:35:25.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:25.576 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:35:25Z" 2024-04-12T04:35:25.576 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:35:25.577 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896128.449153_805.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.449153Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:25Z" } 2024-04-12T04:35:25.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896128.449153_805.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.449153Z', 'realm_epoch': 2}}] 2024-04-12T04:35:25.577 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:35:25.577 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896128.449153_805.1'} 2024-04-12T04:35:25.577 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896128.449153_805.1 2024-04-12T04:35:30.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:30.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:30.578 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:30.639 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:30.639 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:30.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:30.677+0000 7f055962d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:30.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:30.678+0000 7f055962d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:30.693 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:35:30.694 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:35:30.694 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:35:30.694 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:35:30.694 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:35:30.694 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:35:30.694 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:30.694 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:30.694 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:35:30.694 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:30.694 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:35:30.695 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:30.695 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:30.695 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:35:30.695 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:30.695 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:35:30.695 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:30.695 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:35:30.695 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:30.695 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:30.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:30.695 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:30.696 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:35:30.696 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:30.696 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:30.696 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:35:30.696 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:30.696 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:30.696 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:30.696 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:35:30.696 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:30.696 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:30.696 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:30.697 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:30.697 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:35:30.697 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:30.697 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:30.697 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:35:30.697 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:30.697 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:30.697 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:30.697 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:35:30.697 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:30.697 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:30.697 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:30.698 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:30.698 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:35:30.698 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:30.698 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:30.698 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896426.696014_825.1", 2024-04-12T04:35:30.698 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:30.698 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:30.698 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:30.698 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:33:46.696014Z", 2024-04-12T04:35:30.698 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:30.698 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:30.699 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:30.699 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:35:30.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:30.699 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:35:30.699 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:35:30.699 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:35:30.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:30.699 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:35:30Z" 2024-04-12T04:35:30.699 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:35:30.700 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896426.696014_825.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:33:46.696014Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:30Z" } 2024-04-12T04:35:30.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896426.696014_825.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:33:46.696014Z', 'realm_epoch': 2}}] 2024-04-12T04:35:30.701 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:35:30.701 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896426.696014_825.1'} 2024-04-12T04:35:30.701 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896426.696014_825.1 2024-04-12T04:35:35.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:35.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:35.703 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:35.762 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:35.762 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:35.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:35.801+0000 7f5f6c715780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:35.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:35.802+0000 7f5f6c715780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:35.818 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:35:35.818 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:35:35.818 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:35:35.818 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:35:35.818 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:35:35.818 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:35:35.818 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:35.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:35.818 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:35:35.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:35.819 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:35:35.819 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:35.819 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:35.819 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:35:35.819 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:35.819 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:35:35.819 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:35.819 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:35:35.819 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:35.819 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:35.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:35.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:35.820 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:35:35.820 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:35.820 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:35.820 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:35:35.820 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:35.820 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:35.820 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:35.820 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:35:35.821 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:35.821 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:35.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:35.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:35.821 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:35:35.821 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:35.821 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:35.821 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:35:35.821 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:35.822 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:35.822 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:35.822 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:35:35.822 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:35.822 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:35.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:35.822 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:35.822 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:35:35.822 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:35.822 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:35.822 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896426.944411_833.1", 2024-04-12T04:35:35.823 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:35.823 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:35.823 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:35.823 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:33:46.944411Z", 2024-04-12T04:35:35.823 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:35.823 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:35.823 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:35.823 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:35:35.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:35.823 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:35:35.823 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:35:35.823 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:35:35.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:35.824 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:35:35Z" 2024-04-12T04:35:35.824 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:35:35.825 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896426.944411_833.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:33:46.944411Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:35Z" } 2024-04-12T04:35:35.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896426.944411_833.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:33:46.944411Z', 'realm_epoch': 2}}] 2024-04-12T04:35:35.825 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:35:35.825 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896426.944411_833.1'} 2024-04-12T04:35:35.825 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896427.875736_849.1 target=1_1712896426.944411_833.1 2024-04-12T04:35:40.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:40.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:40.826 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:40.886 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:40.887 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:40.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:40.928+0000 7f7e048d7780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:40.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:40.929+0000 7f7e048d7780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:40.944 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:35:40.944 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:35:40.944 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:35:40.944 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:35:40.944 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:35:40.944 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:35:40.944 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:40.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:40.944 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:35:40.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:40.945 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:35:40.945 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:40.945 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:40.945 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:35:40.945 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:40.945 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:35:40.945 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:40.945 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:35:40.945 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:40.945 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:40.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:40.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:40.946 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:35:40.946 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:40.946 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:40.946 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:35:40.946 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:40.946 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:40.946 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:40.946 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:35:40.946 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:40.946 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:40.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:40.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:40.947 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:35:40.947 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:40.947 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:40.947 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:35:40.947 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:40.947 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:40.947 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:40.947 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:35:40.947 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:40.947 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:40.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:40.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:40.948 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:35:40.948 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:40.948 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:40.948 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896427.875736_849.1", 2024-04-12T04:35:40.948 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:40.948 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:40.948 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:40.948 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:33:47.875736Z", 2024-04-12T04:35:40.948 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:40.948 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:40.949 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:40.949 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:35:40.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:40.949 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:35:40.949 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:35:40.949 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:35:40.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:40.949 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:35:40Z" 2024-04-12T04:35:40.949 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:35:40.950 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896427.875736_849.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:33:47.875736Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:40Z" } 2024-04-12T04:35:40.950 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896427.875736_849.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:33:47.875736Z', 'realm_epoch': 2}}] 2024-04-12T04:35:40.950 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:35:40.951 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:35:40.951 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:35:40.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:35:40.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:35:40.951 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-bucket --pipe-id sync-pipe --bucket=dkyges-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:35:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:41.068 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:35:41.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:35:41.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:35:41.068 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket dkyges-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:35:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:41.131 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:41.193 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:35:41.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:35:41.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:35:41.193 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket dkyges-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:35:41.256 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:41.256 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:41.320 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-12T04:35:41.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:35:41.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:35:41.321 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dkyges-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:35:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:41.436 INFO:teuthology.orchestra.run.smithi043.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:35:41.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:35:41.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dkyges-67', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:35:41.436 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dkyges-67 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:35:41.490 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:41.490 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:41.545 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[]} 2024-04-12T04:35:41.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:41.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:41.546 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:41.600 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:41.600 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:41.637 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:35:41.653 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:35:41.654 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:41.654 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:35:41.654 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:35:41.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:41.654 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:41.654 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:35:41.654 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:35:41.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:41.654 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:41.654 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:35:41.655 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:35:41.655 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:41.655 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:41.655 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:35:41.655 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:35:41.420306Z" 2024-04-12T04:35:41.655 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:41.655 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:35:41.656 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:35:41.656 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:35:41.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:41.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:41.657 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:41.710 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:41.710 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:41.747+0000 7fe9957fc780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:41.748+0000 7fe9957fc780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:41.762 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:35:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:35:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:35:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:35:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:35:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:35:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:35:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:35:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:35:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:35:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:35:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:35:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:35:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:35:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:35:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:35:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:35:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:35:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896427.875736_849.1", 2024-04-12T04:35:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:33:47.875736Z", 2024-04-12T04:35:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:35:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:35:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:35:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:35:41.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:41.768 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:35:41Z" 2024-04-12T04:35:41.768 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:35:41.769 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896427.875736_849.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:33:47.875736Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:41Z" } 2024-04-12T04:35:41.769 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896427.875736_849.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:33:47.875736Z', 'realm_epoch': 2}}] 2024-04-12T04:35:41.769 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:35:41.769 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:35:41.769 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896541.420306_857.1 target=1_1712896427.875736_849.1 2024-04-12T04:35:46.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:46.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:46.771 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:46.830 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:46.831 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:46.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:46.869+0000 7fa6eab8e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:46.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:46.870+0000 7fa6eab8e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:46.884 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:35:46.884 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:35:46.884 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:35:46.884 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:35:46.884 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:35:46.884 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:35:46.885 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:46.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:46.885 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:35:46.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:46.885 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:35:46.885 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:46.885 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:46.885 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:35:46.885 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:46.885 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:35:46.885 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:46.885 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:35:46.886 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:46.886 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:46.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:46.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:46.886 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:35:46.886 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:46.886 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:46.886 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:35:46.886 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:46.886 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:46.886 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:46.886 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:35:46.886 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:46.887 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:46.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:46.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:46.887 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:35:46.887 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:46.887 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:46.887 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:35:46.887 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:46.887 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:46.887 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:46.887 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:35:46.888 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:46.888 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:46.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:46.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:46.888 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:35:46.888 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:46.888 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:46.888 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896427.875736_849.1", 2024-04-12T04:35:46.888 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:46.888 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:46.888 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:46.888 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:33:47.875736Z", 2024-04-12T04:35:46.888 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:46.889 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:46.889 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:46.889 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:35:46.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:46.889 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:35:46.889 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:35:46.889 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:35:46.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:46.889 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:35:46Z" 2024-04-12T04:35:46.889 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:35:46.890 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896427.875736_849.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:33:47.875736Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:46Z" } 2024-04-12T04:35:46.890 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896427.875736_849.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:33:47.875736Z', 'realm_epoch': 2}}] 2024-04-12T04:35:46.891 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:35:46.891 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:35:46.891 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896541.420306_857.1 target=1_1712896427.875736_849.1 2024-04-12T04:35:47.842 INFO:tasks.rgw.c1.client.0.smithi043.stdout:2024-04-12T04:35:47.843+0000 7f1e272bd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:35:51.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:51.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:51.892 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:51.951 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:51.951 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:51.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:51.989+0000 7f2c682e8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:51.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:51.990+0000 7f2c682e8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:52.006 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:35:52.006 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:35:52.006 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:35:52.006 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:35:52.006 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:35:52.006 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:35:52.006 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:52.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:35:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:35:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:35:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:35:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:35:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:52.007 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:35:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:35:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:35:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:52.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:35:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:35:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:35:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:52.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:35:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896427.875736_849.1", 2024-04-12T04:35:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:33:47.875736Z", 2024-04-12T04:35:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:52.010 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:52.011 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:35:52.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:52.011 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:35:52.011 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:35:52.011 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:35:52.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:52.011 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:35:51Z" 2024-04-12T04:35:52.011 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:35:52.012 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896427.875736_849.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:33:47.875736Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:51Z" } 2024-04-12T04:35:52.012 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896427.875736_849.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:33:47.875736Z', 'realm_epoch': 2}}] 2024-04-12T04:35:52.012 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:35:52.013 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:35:52.013 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896541.420306_857.1 target=1_1712896427.875736_849.1 2024-04-12T04:35:57.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:57.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:35:57.014 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:35:57.072 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:35:57.072 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:35:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:57.110+0000 7f99bcda1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:57.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:35:57.111+0000 7f99bcda1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:35:57.127 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:35:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:35:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:35:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:35:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:35:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:35:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:35:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:35:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:35:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:35:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:35:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:35:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:35:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:35:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:35:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:35:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:35:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:35:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:35:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:35:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:35:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896427.875736_849.1", 2024-04-12T04:35:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:35:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:35:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:35:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:33:47.875736Z", 2024-04-12T04:35:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:35:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:35:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:35:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:35:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:35:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:35:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:35:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:35:57Z" 2024-04-12T04:35:57.133 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:35:57.133 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896427.875736_849.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:33:47.875736Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:35:57Z" } 2024-04-12T04:35:57.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896427.875736_849.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:33:47.875736Z', 'realm_epoch': 2}}] 2024-04-12T04:35:57.134 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:35:57.134 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896427.875736_849.1'} 2024-04-12T04:35:57.134 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896541.420306_857.1 target=1_1712896427.875736_849.1 2024-04-12T04:36:02.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:02.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:02.136 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:02.194 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:02.195 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:02.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:02.233+0000 7fdde2da8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:02.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:02.234+0000 7fdde2da8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:02.250 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:36:02.250 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:36:02.250 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:36:02.250 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:36:02.251 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:36:02.251 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:02.251 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:02.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.251 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:36:02.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:02.251 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:36:02.251 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:02.251 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:02.251 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:36:02.251 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:02.252 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:36:02.252 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:02.252 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:36:02.252 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:02.252 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:02.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:02.252 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:36:02.252 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:02.252 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:02.252 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:36:02.252 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:02.252 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:02.253 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:02.253 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:36:02.253 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:02.253 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:02.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:02.253 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:36:02.253 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:02.253 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:02.253 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:36:02.253 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:02.254 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:02.254 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:02.254 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:36:02.254 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:02.254 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:02.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:02.254 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:36:02.254 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:02.254 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:02.254 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:36:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:36:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:36:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:36:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:36:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.256 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:36:02Z" 2024-04-12T04:36:02.256 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:36:02.257 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:02Z" } 2024-04-12T04:36:02.257 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:36:02.257 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:36:02.257 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:36:02.257 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:36:02.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:02.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:02.258 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:02.314 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:02.315 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:02.370 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:36:02.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:02.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:02.371 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:02.432 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:02.432 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:02.486 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:36:02.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:02.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:02.486 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:02.544 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:02.545 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:02.602 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-12T04:36:02.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:02.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:02.602 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:02.656 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:02.656 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:02.713 INFO:teuthology.orchestra.run.smithi043.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:36:02.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:02.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:02.713 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:02.763 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:02.764 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:02.819 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[]} 2024-04-12T04:36:02.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:02.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:02.819 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:02.873 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:02.874 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:02.936 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:36:02.936 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:02.936 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 14, 2024-04-12T04:36:02.936 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T04:36:02.936 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T04:36:02.936 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T04:36:02.936 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:02.936 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T04:36:02.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:02.937 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T04:36:02.937 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T04:36:02.937 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:36:02.937 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T04:36:02.937 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:36:02.937 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:36:02.937 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:02.937 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:36:02.937 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:36:02.937 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:36:02.938 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:36:02.938 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:02.938 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:36:02.938 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T04:36:02.938 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:36:02.938 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T04:36:02.938 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:02.938 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:36:02.938 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:36:02.938 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:36:02.938 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:36:02.938 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:36:02.939 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:36:02.939 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:36:02.939 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:36:02.939 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:36:02.939 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:36:02.939 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:36:02.939 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:02.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:02.939 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:36:02.939 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T04:36:02.940 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:36:02.940 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:36:02.940 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:02.940 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:36:02.940 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:36:02.940 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:36:02.940 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:36:02.940 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:36:02.940 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:36:02.940 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:36:02.940 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:36:02.940 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:36:02.941 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:36:02.941 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:36:02.941 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:02.941 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:02.941 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:02.941 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:36:02.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:02.941 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:36:02.941 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:36:02.941 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:36:02.941 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:36:02.941 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:02.942 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:02.942 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:02.942 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:36:02.942 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:36:02.942 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:36:02.942 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T04:36:02.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.942 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:36:02.942 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:36:02.942 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:02.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:02.943 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T04:36:02.943 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T04:36:02.943 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:36:02.943 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T04:36:02.943 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:36:02.943 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:36:02.943 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:02.943 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:36:02.943 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:36:02.943 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:36:02.943 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:36:02.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:02.944 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:36:02.944 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T04:36:02.944 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:36:02.944 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:36:02.944 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:02.944 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:36:02.944 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:36:02.944 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:36:02.944 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:36:02.944 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:36:02.945 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:36:02.945 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:36:02.945 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:36:02.945 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:36:02.945 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:36:02.952 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:36:02.952 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:02.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:02.953 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:36:02.953 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T04:36:02.953 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:36:02.953 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T04:36:02.953 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:02.953 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:36:02.953 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:36:02.953 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:36:02.953 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:36:02.953 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:36:02.954 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:36:02.954 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:36:02.954 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:36:02.954 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:36:02.954 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:36:02.954 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:36:02.954 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:02.954 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:02.954 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:02.954 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:36:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:36:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:36:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:36:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:36:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:36:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:36:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:36:02.955 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T04:36:02.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.956 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:36:02.956 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:36:02.956 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:02.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:02.956 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:02.956 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T04:36:02.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:02.956 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:36:02.956 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T04:36:02.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:02.956 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:36:02.957 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T04:36:02.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:02.957 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:36:02.957 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T04:36:02.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:02.957 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:36:02.957 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T04:36:02.957 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:02.957 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:02.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.958 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T04:36:02.958 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:36:02.958 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T04:36:02.958 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T04:36:02.958 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:36:02.958 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:36:02.958 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:36:02.958 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:36:02.958 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:36:02.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.959 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T04:36:02.959 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:36:02.959 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:36:02.959 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:36:02.959 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:36:02.959 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:36:02.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.959 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T04:36:02.959 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:36:02.959 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:36:02.959 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:36:02.960 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:36:02.960 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:36:02.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.960 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T04:36:02.960 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:36:02.960 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:36:02.960 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:36:02.960 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:36:02.960 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:36:02.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.960 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T04:36:02.961 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:36:02.961 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:36:02.961 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:36:02.961 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:36:02.961 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:36:02.961 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:02.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:02.961 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:36:02.961 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T04:36:02.961 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:02.961 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:36:02.963 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ok 2024-04-12T04:36:02.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:02.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:02.965 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:03.018 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:03.018 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:03.056 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:36:03.073 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:36:03.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:03.073 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:36:03.073 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:36:03.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:03.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:03.073 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:35:41.420306Z" 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:03.074 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:36:03.075 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:36:03.076 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:36:03.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:03.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:03.076 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:03.139 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:03.140 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:03.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:03.206+0000 7f5a6709d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:03.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:03.212+0000 7f5a6709d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:03.252 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:36:03.252 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:36:03.252 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:36:03.252 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:36:03.252 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:36:03.252 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:03.252 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:03.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:03.252 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:36:03.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:03.253 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:36:03.253 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:03.253 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:03.253 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:36:03.253 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:03.253 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:36:03.253 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:03.253 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:36:03.253 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:03.253 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:03.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:03.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:03.254 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:36:03.254 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:03.254 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:03.254 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:36:03.254 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:03.254 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:03.254 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:03.254 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:36:03.254 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:03.254 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:03.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:03.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:03.255 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:36:03.255 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:03.255 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:03.255 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:36:03.255 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:03.255 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:03.255 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:03.255 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:36:03.255 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:03.255 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:03.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:03.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:03.256 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:36:03.256 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:03.256 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:03.256 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:36:03.256 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:03.256 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:03.256 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:03.256 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:36:03.256 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:03.256 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:03.256 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:03.256 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:03.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:03.257 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:36:03.257 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:36:03.257 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:36:03.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:03.257 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:36:03Z" 2024-04-12T04:36:03.257 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:36:03.258 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:03Z" } 2024-04-12T04:36:03.258 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:36:03.258 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:36:03.258 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:36:03.258 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:36:03.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:03.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:03.259 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:03.338 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:03.338 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:03.532 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[]} 2024-04-12T04:36:03.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:03.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:03.532 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:03.623 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:03.624 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:03.772 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:36:03.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:03.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:03.773 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:03.849 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:03.849 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:03.933 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:36:03.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:03.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:03.934 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:04.007 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:04.007 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:04.082 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:36:04.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:04.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:04.083 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:04.138 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:04.138 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:04.193 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:36:04.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:04.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:04.194 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:04.248 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:04.248 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:04.311 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:36:04.311 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:04.311 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 15, 2024-04-12T04:36:04.311 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T04:36:04.311 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T04:36:04.311 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T04:36:04.311 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:04.311 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T04:36:04.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.312 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T04:36:04.312 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T04:36:04.312 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:36:04.312 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T04:36:04.312 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:36:04.312 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:36:04.312 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:04.312 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:36:04.312 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:36:04.312 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:36:04.312 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:36:04.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.313 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:36:04.313 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T04:36:04.313 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:36:04.313 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T04:36:04.313 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:04.313 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:36:04.313 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:36:04.313 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:36:04.313 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:36:04.313 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:36:04.313 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:36:04.314 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:36:04.314 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:36:04.314 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:36:04.314 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:36:04.314 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:36:04.314 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:04.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.314 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.314 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:36:04.314 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T04:36:04.314 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:36:04.315 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:36:04.315 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:04.315 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:36:04.315 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:36:04.315 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:36:04.315 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:36:04.315 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:36:04.315 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:36:04.315 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:36:04.315 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:36:04.315 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:36:04.316 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:36:04.316 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:36:04.316 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:04.316 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:04.316 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:04.316 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:36:04.316 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.316 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:36:04.316 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:36:04.316 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:36:04.316 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:36:04.316 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:04.316 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:04.317 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:04.317 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:36:04.317 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:36:04.317 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:36:04.317 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [ 2024-04-12T04:36:04.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.317 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-group", 2024-04-12T04:36:04.317 INFO:teuthology.orchestra.run.smithi043.stdout: "data_flow": { 2024-04-12T04:36:04.317 INFO:teuthology.orchestra.run.smithi043.stdout: "symmetrical": [ 2024-04-12T04:36:04.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.317 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-flow1", 2024-04-12T04:36:04.318 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:36:04.318 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:36:04.318 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:36:04.318 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:04.318 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:04.318 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:04.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.318 INFO:teuthology.orchestra.run.smithi043.stdout: "pipes": [ 2024-04-12T04:36:04.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.318 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-pipe", 2024-04-12T04:36:04.318 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:36:04.318 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "*", 2024-04-12T04:36:04.319 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:36:04.319 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:36:04.319 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:36:04.319 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:04.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.319 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": { 2024-04-12T04:36:04.319 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "*", 2024-04-12T04:36:04.319 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:36:04.319 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:36:04.319 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:36:04.319 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:04.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.320 INFO:teuthology.orchestra.run.smithi043.stdout: "params": { 2024-04-12T04:36:04.320 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:36:04.320 INFO:teuthology.orchestra.run.smithi043.stdout: "filter": { 2024-04-12T04:36:04.320 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [] 2024-04-12T04:36:04.320 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:04.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.320 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": {}, 2024-04-12T04:36:04.320 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-12T04:36:04.320 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "system", 2024-04-12T04:36:04.320 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "" 2024-04-12T04:36:04.320 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:04.320 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:04.320 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:04.321 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "enabled" 2024-04-12T04:36:04.321 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:04.321 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:04.321 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.321 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:36:04.321 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:36:04.321 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:04.321 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.321 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.321 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T04:36:04.321 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T04:36:04.322 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:36:04.322 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T04:36:04.322 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:36:04.322 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:36:04.322 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:04.322 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:36:04.322 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:36:04.322 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:36:04.322 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:36:04.322 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.322 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:36:04.322 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T04:36:04.323 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:36:04.323 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:36:04.323 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:04.323 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:36:04.323 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:36:04.323 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:36:04.323 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:36:04.323 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:36:04.323 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:36:04.323 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:36:04.323 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:36:04.323 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:36:04.324 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:36:04.324 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:36:04.324 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:04.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.324 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:36:04.324 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T04:36:04.324 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:36:04.324 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T04:36:04.324 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:04.324 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:36:04.325 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:36:04.325 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:36:04.325 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:36:04.325 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:36:04.325 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:36:04.325 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:36:04.325 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:36:04.325 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:36:04.325 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:36:04.325 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:36:04.325 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:04.326 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:04.326 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:04.326 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:36:04.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.326 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:36:04.326 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:36:04.326 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:36:04.326 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:36:04.326 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:04.326 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:04.326 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:04.327 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:36:04.327 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:36:04.327 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:36:04.327 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T04:36:04.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.327 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:36:04.327 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:36:04.327 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:04.327 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:04.327 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:36:04.327 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T04:36:04.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.328 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:36:04.328 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T04:36:04.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.328 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:36:04.328 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T04:36:04.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.329 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:36:04.329 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T04:36:04.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.330 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:36:04.330 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T04:36:04.330 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:04.330 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:04.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.330 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T04:36:04.330 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:36:04.330 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T04:36:04.330 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T04:36:04.331 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:36:04.331 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:36:04.331 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:36:04.331 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:36:04.331 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:36:04.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.331 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T04:36:04.331 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:36:04.331 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:36:04.331 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:36:04.331 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:36:04.332 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:36:04.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.332 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T04:36:04.332 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:36:04.332 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:36:04.332 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:36:04.332 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:36:04.332 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:36:04.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.332 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T04:36:04.332 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:36:04.333 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:36:04.333 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:36:04.333 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:36:04.333 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:36:04.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.333 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T04:36:04.333 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:36:04.333 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:36:04.333 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:36:04.333 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:36:04.333 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:36:04.334 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:04.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.334 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:36:04.334 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T04:36:04.334 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:04.334 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:36:04.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:04.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:36:04.335 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:36:04.392 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:04.392 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:04.489 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:36:04.490 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-68 2024-04-12T04:36:04.490 DEBUG:boto:path=/dkyges-68/ 2024-04-12T04:36:04.490 DEBUG:boto:auth_path=/dkyges-68/ 2024-04-12T04:36:04.490 DEBUG:boto:Method: PUT 2024-04-12T04:36:04.490 DEBUG:boto:Path: /dkyges-68/ 2024-04-12T04:36:04.490 DEBUG:boto:Data: 2024-04-12T04:36:04.491 DEBUG:boto:Headers: {} 2024-04-12T04:36:04.491 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:36:04.491 DEBUG:boto:Port: 8000 2024-04-12T04:36:04.491 DEBUG:boto:Params: {} 2024-04-12T04:36:04.491 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:36:04.491 DEBUG:boto:Token: None 2024-04-12T04:36:04.492 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:36:04 GMT /dkyges-68/ 2024-04-12T04:36:04.492 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/1d88k0BUc14ho+SPWqe3iXMnt8= 2024-04-12T04:36:04.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/1d88k0BUc14ho+SPWqe3iXMnt8=', 'Content-Length': '0'} 2024-04-12T04:36:04.565 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004e3dccddbdb3e760-006618ba34-2493-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:36:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:04.565 DEBUG:boto:path=/dkyges-68/ 2024-04-12T04:36:04.565 DEBUG:boto:auth_path=/dkyges-68/ 2024-04-12T04:36:04.566 DEBUG:boto:Method: HEAD 2024-04-12T04:36:04.566 DEBUG:boto:Path: /dkyges-68/ 2024-04-12T04:36:04.566 DEBUG:boto:Data: 2024-04-12T04:36:04.566 DEBUG:boto:Headers: {} 2024-04-12T04:36:04.566 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:36:04.566 DEBUG:boto:Port: 8000 2024-04-12T04:36:04.566 DEBUG:boto:Params: {} 2024-04-12T04:36:04.566 DEBUG:boto:Token: None 2024-04-12T04:36:04.567 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:36:04 GMT /dkyges-68/ 2024-04-12T04:36:04.567 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Bkoxma7GWFdV5MMyExuob7DMvhU= 2024-04-12T04:36:04.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bkoxma7GWFdV5MMyExuob7DMvhU=', 'Content-Length': '0'} 2024-04-12T04:36:04.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005baa10fa85dc481-006618ba34-2493-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:36:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:04.571 DEBUG:boto:path=/dkyges-68/obj1 2024-04-12T04:36:04.571 DEBUG:boto:auth_path=/dkyges-68/obj1 2024-04-12T04:36:04.571 DEBUG:boto:Method: PUT 2024-04-12T04:36:04.572 DEBUG:boto:Path: /dkyges-68/obj1 2024-04-12T04:36:04.572 DEBUG:boto:Data: 2024-04-12T04:36:04.572 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:04.572 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:36:04.572 DEBUG:boto:Port: 8000 2024-04-12T04:36:04.572 DEBUG:boto:Params: {} 2024-04-12T04:36:04.572 DEBUG:boto:Token: None 2024-04-12T04:36:04.572 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:36:04 GMT /dkyges-68/obj1 2024-04-12T04:36:04.572 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BdyDqqaU8tHkWyGcwxs30REqqp8= 2024-04-12T04:36:04.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:36:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BdyDqqaU8tHkWyGcwxs30REqqp8='} 2024-04-12T04:36:04.612 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa8d4d433b65b326-006618ba34-2493-a1'), ('Date', 'Fri, 12 Apr 2024 04:36:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:04.612 INFO:rgw_multi.tests:create bucket zone=a2 name=dkyges-69 2024-04-12T04:36:04.612 DEBUG:boto:path=/dkyges-69/ 2024-04-12T04:36:04.613 DEBUG:boto:auth_path=/dkyges-69/ 2024-04-12T04:36:04.613 DEBUG:boto:Method: PUT 2024-04-12T04:36:04.613 DEBUG:boto:Path: /dkyges-69/ 2024-04-12T04:36:04.613 DEBUG:boto:Data: 2024-04-12T04:36:04.613 DEBUG:boto:Headers: {} 2024-04-12T04:36:04.613 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:36:04.613 DEBUG:boto:Port: 8001 2024-04-12T04:36:04.613 DEBUG:boto:Params: {} 2024-04-12T04:36:04.614 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:36:04.614 DEBUG:boto:Token: None 2024-04-12T04:36:04.614 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:36:04 GMT /dkyges-69/ 2024-04-12T04:36:04.614 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fo4qHf+nicY2irHLYJOi7HUTEgc= 2024-04-12T04:36:04.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fo4qHf+nicY2irHLYJOi7HUTEgc=', 'Content-Length': '0'} 2024-04-12T04:36:04.739 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e8646bf1904788d1-006618ba34-249c-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:36:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:04.739 DEBUG:boto:path=/dkyges-69/ 2024-04-12T04:36:04.739 DEBUG:boto:auth_path=/dkyges-69/ 2024-04-12T04:36:04.739 DEBUG:boto:Method: HEAD 2024-04-12T04:36:04.740 DEBUG:boto:Path: /dkyges-69/ 2024-04-12T04:36:04.740 DEBUG:boto:Data: 2024-04-12T04:36:04.740 DEBUG:boto:Headers: {} 2024-04-12T04:36:04.740 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:36:04.740 DEBUG:boto:Port: 8001 2024-04-12T04:36:04.740 DEBUG:boto:Params: {} 2024-04-12T04:36:04.740 DEBUG:boto:Token: None 2024-04-12T04:36:04.740 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:36:04 GMT /dkyges-69/ 2024-04-12T04:36:04.740 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ghamh1Zsfs/YEcGEOKtPlUbj2TM= 2024-04-12T04:36:04.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ghamh1Zsfs/YEcGEOKtPlUbj2TM=', 'Content-Length': '0'} 2024-04-12T04:36:04.744 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4e71db4ef05f240-006618ba34-249c-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:36:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:04.744 DEBUG:boto:path=/dkyges-69/obj2 2024-04-12T04:36:04.744 DEBUG:boto:auth_path=/dkyges-69/obj2 2024-04-12T04:36:04.745 DEBUG:boto:Method: PUT 2024-04-12T04:36:04.745 DEBUG:boto:Path: /dkyges-69/obj2 2024-04-12T04:36:04.745 DEBUG:boto:Data: 2024-04-12T04:36:04.745 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:36:04.745 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:36:04.745 DEBUG:boto:Port: 8001 2024-04-12T04:36:04.745 DEBUG:boto:Params: {} 2024-04-12T04:36:04.745 DEBUG:boto:Token: None 2024-04-12T04:36:04.746 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:36:04 GMT /dkyges-69/obj2 2024-04-12T04:36:04.746 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VlL5V7GqftCVqFWpubwHzcBzokk= 2024-04-12T04:36:04.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:36:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VlL5V7GqftCVqFWpubwHzcBzokk='} 2024-04-12T04:36:04.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000017e72080ae71c275-006618ba34-249c-a2'), ('Date', 'Fri, 12 Apr 2024 04:36:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:36:04.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:04.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:04.768 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:04.859 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:04.859 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:04.948 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:36:04.973 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:36:04.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.974 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:36:04.974 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:36:04.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.974 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:36:04.974 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:36:04.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.974 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:36:04.974 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:36:04.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:04.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:04.975 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896564.701296_877.1", 2024-04-12T04:36:04.975 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:36:04.701296Z" 2024-04-12T04:36:04.975 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:04.975 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:36:04.976 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:36:04.976 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:36:04.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:04.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:04.977 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:05.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:05.105+0000 7fecc3ade780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:05.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:05.107+0000 7fecc3ade780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:05.127 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:36:05.128 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:36:05.128 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:36:05.128 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:36:05.128 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:36:05.128 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:05.128 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:05.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:05.128 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:36:05.128 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:05.128 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:36:05.129 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:05.129 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:05.129 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:36:05.129 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:05.129 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:36:05.131 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:05.131 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:36:05.131 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:05.131 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:05.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:05.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:05.131 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:36:05.132 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:05.132 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:05.132 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:36:05.132 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:05.132 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:05.132 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:05.132 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:36:05.132 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:05.132 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:05.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:05.133 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:05.133 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:36:05.133 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:05.133 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:05.133 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:36:05.133 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:05.133 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:05.133 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:05.133 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:36:05.133 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:05.134 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:05.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:05.134 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:05.134 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:36:05.134 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:05.134 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:05.134 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:36:05.134 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:05.134 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:05.134 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:05.134 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:36:05.135 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:05.135 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:05.135 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:05.135 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:05.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:05.135 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:36:05.135 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:36:05.135 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:36:05.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:05.135 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:36:05Z" 2024-04-12T04:36:05.135 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:36:05.136 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:05Z" } 2024-04-12T04:36:05.137 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:36:05.137 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:36:05.137 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:36:05.137 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:36:10.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:10.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:10.139 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:10.201 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:10.201 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:10.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:10.240+0000 7f43683aa780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:10.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:10.241+0000 7f43683aa780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:10.257 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:36:10.257 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:36:10.257 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:36:10.257 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:36:10.257 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:36:10.257 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:10.257 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:10.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:10.258 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:36:10.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:10.258 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:36:10.258 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:10.258 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:10.258 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:36:10.258 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:10.258 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:36:10.259 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:10.259 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:36:10.259 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:10.259 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:10.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:10.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:10.259 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:36:10.259 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:10.259 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:10.259 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:36:10.260 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:10.260 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:10.260 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:10.260 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:36:10.260 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:10.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:10.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:10.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:10.260 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:36:10.260 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:10.261 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:10.261 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:36:10.261 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:10.261 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:10.261 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:10.261 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:36:10.261 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:10.261 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:10.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:36:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:36:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:36:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:36:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:36:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:36:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:36:10Z" 2024-04-12T04:36:10.264 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:36:10.265 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:10Z" } 2024-04-12T04:36:10.265 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:36:10.265 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:36:10.265 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:36:10.265 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:36:15.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:15.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:15.267 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:15.329 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:15.329 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:15.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:15.370+0000 7f6e55f65780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:15.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:15.372+0000 7f6e55f65780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:15.387 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:36:15.387 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:36:15.387 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:36:15.388 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:36:15.388 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:36:15.388 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:15.388 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:15.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:15.388 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:36:15.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:15.388 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:36:15.388 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:15.388 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:15.389 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:36:15.389 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:15.389 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:36:15.389 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:15.389 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:36:15.389 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:15.389 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:15.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:15.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:15.389 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:36:15.389 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:15.390 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:15.390 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:36:15.390 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:15.390 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:15.390 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:15.390 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:36:15.390 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:15.390 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:15.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:15.390 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:15.390 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:36:15.391 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:15.391 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:15.391 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:36:15.391 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:15.391 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:15.391 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:15.391 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:36:15.391 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:15.391 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:15.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:15.391 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:15.391 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:36:15.392 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:15.392 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:15.392 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:36:15.392 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:15.392 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:15.392 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:15.392 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:36:15.392 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:15.392 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:15.392 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:15.392 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:15.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:15.393 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:36:15.393 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:36:15.393 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:36:15.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:15.393 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:36:15Z" 2024-04-12T04:36:15.393 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:36:15.394 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:15Z" } 2024-04-12T04:36:15.394 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:36:15.394 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:36:15.394 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:36:15.394 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:36:20.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:20.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:20.396 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:20.458 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:20.458 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:20.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:20.498+0000 7f562d8ba780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:20.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:20.499+0000 7f562d8ba780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:20.515 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:36:20.515 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:36:20.515 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:36:20.515 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:36:20.515 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:36:20.516 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:20.516 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:20.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:20.516 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:36:20.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:20.516 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:36:20.516 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:20.516 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:20.516 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:36:20.516 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:20.516 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:36:20.517 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:20.517 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:36:20.517 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:20.517 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:20.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:20.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:20.517 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:36:20.517 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:20.517 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:20.517 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:36:20.517 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:20.518 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:20.518 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:20.518 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:36:20.518 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:20.518 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:20.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:20.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:20.518 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:36:20.518 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:20.518 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:20.519 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:36:20.519 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:20.519 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:20.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:20.519 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:36:20.519 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:20.519 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:20.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:20.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:20.519 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:36:20.519 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:20.520 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:20.520 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:36:20.520 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:20.520 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:20.520 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:20.520 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:36:20.520 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:20.520 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:20.520 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:20.520 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:20.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:20.521 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:36:20.521 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:36:20.521 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:36:20.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:20.521 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:36:20Z" 2024-04-12T04:36:20.521 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:36:20.522 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:20Z" } 2024-04-12T04:36:20.522 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:36:20.523 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:36:20.523 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:36:20.523 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:36:25.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:25.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:25.524 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:25.582 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:25.582 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:25.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:25.620+0000 7fa927749780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:25.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:25.621+0000 7fa927749780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:25.637 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:36:25.637 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:36:25.637 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:36:25.637 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:36:25.637 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:36:25.637 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:25.637 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:25.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:25.637 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:36:25.638 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:25.638 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:36:25.638 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:25.638 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:25.638 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:36:25.638 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:25.638 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:36:25.638 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:25.638 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:36:25.638 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:25.638 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:25.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:25.639 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:25.639 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:36:25.639 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:25.639 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:25.639 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:36:25.639 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:25.639 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:25.639 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:25.639 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:36:25.639 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:25.639 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:25.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:25.640 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:25.640 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:36:25.640 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:25.640 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:25.640 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:36:25.640 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:25.640 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:25.640 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:25.640 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:36:25.640 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:25.640 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:25.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:25.641 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:25.641 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:36:25.641 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:25.641 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:25.641 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:36:25.641 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:25.641 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:25.641 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:25.641 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:36:25.641 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:25.641 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:25.641 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:25.642 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:25.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:25.642 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:36:25.642 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:36:25.642 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:36:25.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:25.642 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:36:25Z" 2024-04-12T04:36:25.642 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:36:25.643 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:25Z" } 2024-04-12T04:36:25.644 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:36:25.644 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:36:25.644 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:36:25.644 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:36:30.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:30.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:30.646 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:30.705 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:30.705 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:30.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:30.743+0000 7f439783b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:30.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:30.744+0000 7f439783b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:30.760 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:36:30.760 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:36:30.760 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:36:30.760 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:36:30.760 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:36:30.760 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:30.760 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:30.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:30.761 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:36:30.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:30.761 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:36:30.761 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:30.761 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:30.761 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:36:30.761 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:30.761 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:36:30.761 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:30.762 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:36:30.762 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:30.762 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:30.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:30.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:30.762 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:36:30.762 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:30.762 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:30.762 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:36:30.762 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:30.762 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:30.763 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:30.763 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:36:30.763 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:30.763 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:30.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:30.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:30.763 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:36:30.763 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:30.763 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:30.763 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:36:30.763 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:30.764 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:30.764 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:30.764 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:36:30.764 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:30.764 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:30.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:30.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:30.764 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:36:30.764 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:30.764 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:30.764 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:36:30.765 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:30.765 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:30.765 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:30.765 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:36:30.765 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:30.765 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:30.765 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:30.765 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:30.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:30.765 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:36:30.765 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:36:30.765 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:36:30.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:30.766 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:36:30Z" 2024-04-12T04:36:30.766 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:36:30.767 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:30Z" } 2024-04-12T04:36:30.767 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:36:30.767 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:36:30.767 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:36:30.767 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:36:35.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:35.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:35.769 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:35.828 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:35.828 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:35.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:35.866+0000 7fee00c6e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:35.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:35.867+0000 7fee00c6e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:35.883 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:36:35.883 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:36:35.884 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:36:35.884 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:36:35.884 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:36:35.884 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:35.884 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:35.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:35.884 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:36:35.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:35.884 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:36:35.884 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:35.885 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:35.885 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:36:35.885 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:35.885 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:36:35.885 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:35.885 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:36:35.885 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:35.885 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:35.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:35.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:35.886 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:36:35.886 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:35.886 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:35.886 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:36:35.886 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:35.886 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:35.886 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:35.886 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:36:35.886 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:35.886 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:35.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:35.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:35.887 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:36:35.887 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:35.887 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:35.887 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:36:35.887 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:35.887 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:35.887 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:35.887 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:36:35.888 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:35.888 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:35.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:35.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:35.888 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:36:35.888 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:35.888 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:35.888 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:36:35.888 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:35.888 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:35.888 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:35.888 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:36:35.889 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:35.889 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:35.889 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:35.889 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:35.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:35.889 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:36:35.889 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:36:35.889 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:36:35.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:35.890 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:36:35Z" 2024-04-12T04:36:35.890 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:36:35.891 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:35Z" } 2024-04-12T04:36:35.891 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:36:35.891 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:36:35.891 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:36:35.891 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:36:40.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:40.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:40.893 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:40.954 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:40.954 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:40.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:40.997+0000 7f6a0e1f0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:40.998+0000 7f6a0e1f0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:41.014 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:36:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:36:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:36:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:36:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:36:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:36:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:36:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:36:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:36:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:36:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:36:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:36:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:36:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:36:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:36:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:36:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:36:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:36:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:36:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:36:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:36:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:36:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:36:41Z" 2024-04-12T04:36:41.020 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:36:41.021 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:41Z" } 2024-04-12T04:36:41.022 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:36:41.022 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:36:41.022 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:36:41.022 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:36:46.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:46.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:46.023 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:46.083 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:46.083 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:46.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:46.120+0000 7f30c3ce8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:46.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:46.121+0000 7f30c3ce8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:46.136 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:36:46.136 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:36:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:36:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:36:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:36:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:36:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:36:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:46.137 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:36:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:36:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:36:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:36:46.138 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:46.139 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:46.139 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:36:46.139 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:46.139 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:46.139 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:46.139 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:36:46.139 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:46.139 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:46.139 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:46.139 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:46.139 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:36:46.140 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:46.140 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:46.140 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:36:46.140 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:46.140 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:46.140 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:46.140 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:36:46.140 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:46.140 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:46.140 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:46.140 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:46.140 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:36:46.141 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:46.141 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:46.141 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:36:46.141 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:46.141 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:46.141 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:46.141 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:36:46.141 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:46.141 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:46.141 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:46.141 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:46.142 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:46.142 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:36:46.142 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:36:46.142 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:36:46.142 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:46.142 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:36:46Z" 2024-04-12T04:36:46.142 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:36:46.143 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:46Z" } 2024-04-12T04:36:46.143 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:36:46.143 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:36:46.144 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:36:46.144 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:36:51.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:51.145 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:51.206 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:51.206 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:51.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:51.244+0000 7f9b66776780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:51.245+0000 7f9b66776780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:51.261 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:36:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:36:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:36:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:36:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:36:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:36:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:36:51.262 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:36:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:36:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:36:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:51.263 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:36:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:36:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:36:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:51.264 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:36:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:36:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:36:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:51.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:36:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:36:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:36:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:51.266 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:36:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:36:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:36:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:51.267 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:36:51Z" 2024-04-12T04:36:51.267 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:36:51.268 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:51Z" } 2024-04-12T04:36:51.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:36:51.268 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:36:51.269 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:36:51.269 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:36:56.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:56.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:36:56.270 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:36:56.330 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:36:56.330 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:36:56.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:56.370+0000 7fb07e958780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:56.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:36:56.372+0000 7fb07e958780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:36:56.386 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:36:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:36:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:36:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:36:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:36:56.386 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:36:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:36:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:36:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:36:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:36:56.387 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:36:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:36:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:36:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:56.388 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:36:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:36:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:36:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:56.389 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:36:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:36:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:36:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:36:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:36:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:36:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:36:56.390 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:36:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:36:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:36:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:36:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:36:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:36:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:36:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:36:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:36:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:36:56.391 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:36:56Z" 2024-04-12T04:36:56.392 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:36:56.392 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:36:56Z" } 2024-04-12T04:36:56.392 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:36:56.393 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:36:56.393 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:36:56.393 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:37:01.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:01.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:01.395 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:01.454 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:01.455 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:01.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:01.494+0000 7fafe7ada780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:01.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:01.495+0000 7fafe7ada780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:01.510 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:01.511 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:37:01.511 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:37:01.511 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:37:01.511 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:37:01.511 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:37:01.511 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:01.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:01.511 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:37:01.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:01.511 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:37:01.511 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:01.512 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:01.512 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:37:01.512 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:01.512 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:37:01.512 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:01.512 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:37:01.512 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:01.512 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:01.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:01.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:01.512 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:37:01.513 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:01.513 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:01.513 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:37:01.513 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:01.513 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:01.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:01.513 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:37:01.513 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:01.513 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:01.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:01.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:01.513 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:37:01.514 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:01.514 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:01.514 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:37:01.514 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:01.514 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:01.514 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:01.514 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:37:01.514 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:01.514 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:01.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:01.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:01.515 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:37:01.515 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:01.515 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:01.515 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:37:01.515 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:01.515 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:01.515 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:01.515 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:37:01.515 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:01.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:01.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:01.516 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:01.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:01.516 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:37:01.516 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:37:01.516 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:37:01.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:01.516 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:37:01Z" 2024-04-12T04:37:01.516 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:01.517 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:37:01Z" } 2024-04-12T04:37:01.517 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:37:01.517 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:37:01.518 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:37:01.518 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:37:06.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:06.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:06.519 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:06.579 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:06.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:06.616+0000 7fbef1b32780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:06.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:06.617+0000 7fbef1b32780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:06.633 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:06.634 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:37:06.634 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:37:06.634 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:37:06.634 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:37:06.634 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:37:06.634 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:06.634 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:06.634 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:37:06.634 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:06.634 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:37:06.635 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:06.635 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:06.635 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:37:06.635 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:06.635 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:37:06.635 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:06.635 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:37:06.635 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:06.635 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:06.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:06.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:06.635 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:37:06.636 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:06.636 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:06.636 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:37:06.636 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:06.636 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:06.636 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:06.636 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:37:06.636 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:06.636 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:06.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:06.636 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:06.637 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:37:06.637 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:06.637 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:06.637 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:37:06.637 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:06.637 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:06.637 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:06.637 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:37:06.637 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:06.637 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:06.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:06.637 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:06.638 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:37:06.638 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:06.638 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:06.638 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:37:06.638 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:06.638 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:06.638 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:06.638 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:37:06.638 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:06.638 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:06.638 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:06.639 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:06.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:06.639 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:37:06.639 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:37:06.639 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:37:06.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:06.639 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:37:06Z" 2024-04-12T04:37:06.639 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:06.640 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:37:06Z" } 2024-04-12T04:37:06.640 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:37:06.640 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:37:06.640 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:37:06.640 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:37:11.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:11.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:11.642 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:11.703 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:11.703 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:11.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:11.741+0000 7fd14c528780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:11.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:11.742+0000 7fd14c528780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:11.758 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:11.758 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:37:11.758 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:37:11.758 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:37:11.758 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:37:11.758 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:37:11.759 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:11.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:11.759 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:37:11.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:11.759 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:37:11.759 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:11.759 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:11.759 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:37:11.759 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:11.759 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:37:11.759 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:11.760 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:37:11.760 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:11.760 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:11.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:11.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:11.760 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:37:11.760 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:11.760 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:11.760 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:37:11.760 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:11.760 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:11.760 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:11.761 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:37:11.761 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:11.761 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:11.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:11.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:11.761 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:37:11.761 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:11.761 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:11.761 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:37:11.761 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:11.761 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:11.761 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:11.762 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:37:11.762 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:11.762 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:11.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:11.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:11.762 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:37:11.762 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:11.762 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:11.762 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:37:11.762 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:11.762 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:11.763 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:11.763 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:37:11.763 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:11.763 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:11.763 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:11.763 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:11.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:11.763 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:37:11.763 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:37:11.763 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:37:11.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:11.764 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:37:11Z" 2024-04-12T04:37:11.764 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:11.765 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:37:11Z" } 2024-04-12T04:37:11.765 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:37:11.765 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:37:11.765 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:37:11.765 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:37:16.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:16.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:16.766 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:16.826 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:16.827 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:16.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:16.864+0000 7f108478c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:16.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:16.865+0000 7f108478c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:16.880 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:16.880 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:37:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:37:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:37:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:37:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:37:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:37:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:37:16.881 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:37:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:37:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:37:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:16.882 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:37:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:37:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:37:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:16.883 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:37:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:37:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:37:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:16.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:37:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:37:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:37:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:16.885 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:37:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:37:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:37:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:16.886 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:37:16Z" 2024-04-12T04:37:16.886 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:16.887 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:37:16Z" } 2024-04-12T04:37:16.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:37:16.887 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:37:16.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:37:16.888 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:37:21.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:21.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:21.889 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:21.949 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:21.950 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:21.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:21.987+0000 7f2968582780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:21.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:21.988+0000 7f2968582780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:22.004 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:22.004 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:37:22.004 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:37:22.004 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:37:22.004 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:37:22.004 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:37:22.004 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:22.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:22.005 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:37:22.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:22.005 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:37:22.005 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:22.005 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:22.005 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:37:22.005 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:22.005 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:37:22.005 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:22.005 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:37:22.006 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:22.006 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:22.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:22.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:22.006 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:37:22.006 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:22.006 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:22.006 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:37:22.006 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:22.006 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:22.006 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:22.006 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:37:22.007 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:22.007 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:22.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:22.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:22.007 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:37:22.007 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:22.007 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:22.007 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:37:22.007 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:22.007 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:22.007 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:22.007 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:37:22.008 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:22.008 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:22.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:22.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:22.008 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:37:22.008 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:22.008 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:22.008 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:37:22.008 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:22.008 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:22.008 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:22.009 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:37:22.009 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:22.009 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:22.009 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:22.009 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:22.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:22.009 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:37:22.009 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:37:22.009 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:37:22.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:22.009 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:37:21Z" 2024-04-12T04:37:22.009 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:22.010 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:37:21Z" } 2024-04-12T04:37:22.011 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:37:22.011 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:37:22.011 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:37:22.011 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:37:27.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:27.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:27.012 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:27.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:27.113+0000 7f987aa70780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:27.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:27.114+0000 7f987aa70780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:27.130 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:27.130 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:37:27.130 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:37:27.130 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:37:27.130 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:37:27.130 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:37:27.130 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:27.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:27.131 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:37:27.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:27.131 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:37:27.131 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:27.131 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:27.131 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:37:27.131 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:27.131 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:37:27.131 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:27.131 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:37:27.131 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:27.131 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:27.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:27.132 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:27.132 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:37:27.132 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:27.132 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:27.132 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:37:27.132 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:27.132 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:27.132 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:27.132 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:37:27.132 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:27.132 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:27.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:27.133 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:27.133 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:37:27.133 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:27.133 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:27.133 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:37:27.133 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:27.133 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:27.133 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:27.133 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:37:27.133 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:27.133 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:27.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:27.134 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:27.134 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:37:27.134 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:27.134 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:27.134 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:37:27.134 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:27.134 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:27.134 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:27.134 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:37:27.135 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:27.135 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:27.135 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:27.135 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:27.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:27.135 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:37:27.135 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:37:27.135 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:37:27.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:27.135 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:37:27Z" 2024-04-12T04:37:27.135 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:27.138 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:37:27Z" } 2024-04-12T04:37:27.138 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:37:27.139 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:37:27.139 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:37:27.139 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:37:32.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:32.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:32.140 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:32.199 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:32.236+0000 7fce41c7a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:32.237+0000 7fce41c7a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:32.253 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:37:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:37:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:37:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:37:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:37:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:37:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:37:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:37:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:37:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:37:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:37:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:37:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:37:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:37:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:37:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:37:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:37:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:37:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:37:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:37:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:37:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:37:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:37:32Z" 2024-04-12T04:37:32.258 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:32.259 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:37:32Z" } 2024-04-12T04:37:32.259 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:37:32.259 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:37:32.259 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:37:32.260 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:37:35.619 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T04:37:35.619+0000 7f23f2e6b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T04:37:37.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:37.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:37.261 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:37.321 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:37.321 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:37.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:37.359+0000 7f173fbd8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:37.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:37.360+0000 7f173fbd8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:37.379 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:37.379 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:37:37.380 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:37:37.380 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:37:37.380 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:37:37.380 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:37:37.380 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:37.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:37.380 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:37:37.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:37.380 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:37:37.380 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:37.380 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:37.380 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:37:37.381 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:37.381 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:37:37.381 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:37.381 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:37:37.381 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:37.381 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:37.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:37.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:37.381 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:37:37.381 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:37.382 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:37.382 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:37:37.382 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:37.382 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:37.382 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:37.382 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:37:37.382 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:37.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:37.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:37.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:37.382 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:37:37.382 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:37.383 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:37.383 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:37:37.383 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:37.383 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:37.383 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:37.383 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:37:37.383 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:37.383 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:37.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:37.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:37.383 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:37:37.383 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:37.384 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:37.384 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:37:37.384 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:37.384 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:37.384 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:37.384 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:37:37.384 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:37.384 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:37.384 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:37.384 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:37.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:37.384 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:37:37.384 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:37:37.385 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:37:37.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:37.385 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:37:37Z" 2024-04-12T04:37:37.385 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:37.385 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896541.420306_857.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:35:41.420306Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:37:37Z" } 2024-04-12T04:37:37.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:37:37.385 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:37:37.386 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:37:37.386 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:37:42.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:42.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:42.387 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:42.446 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:42.446 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:42.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:42.484+0000 7f2374316780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:42.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:42.485+0000 7f2374316780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:42.501 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:42.501 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:37:42.502 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:37:42.502 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:37:42.502 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:37:42.502 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:37:42.502 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:42.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.502 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:37:42.502 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:42.502 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:37:42.503 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:42.503 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:42.503 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:37:42.503 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:42.503 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:37:42.503 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:42.503 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:37:42.503 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:42.503 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:42.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.503 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:42.504 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:37:42.504 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:42.504 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:42.504 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:37:42.504 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:42.504 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:42.504 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:42.504 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:37:42.504 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:42.504 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:42.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.504 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:42.505 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:37:42.505 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:42.505 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:42.505 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:37:42.505 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:42.505 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:42.505 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:42.505 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:37:42.505 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:42.505 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:42.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.505 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:42.506 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:37:42.506 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:42.506 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:42.506 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896564.701296_877.1", 2024-04-12T04:37:42.506 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:42.506 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:42.506 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:42.506 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.701296Z", 2024-04-12T04:37:42.506 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:42.506 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:42.507 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:42.507 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:42.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.507 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:37:42.507 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:37:42.507 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:37:42.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.507 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:37:42Z" 2024-04-12T04:37:42.507 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:42.508 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896564.701296_877.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:36:04.701296Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:37:42Z" } 2024-04-12T04:37:42.508 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896564.701296_877.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.701296Z', 'realm_epoch': 2}}] 2024-04-12T04:37:42.509 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:37:42.509 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:37:42.509 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:37:42.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:42.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:42.510 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:42.563 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:42.564 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:42.624 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:37:42.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:42.625 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:37:42.625 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:37:42.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.625 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:42.625 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:37:42.625 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:37:42.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.625 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:42.625 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:37:42.625 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:37:42.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.626 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:42.626 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:37:42.626 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:37:42.626 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:42.626 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:37:42.627 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:37:42.627 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:37:42.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:42.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:42.627 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:42.684 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:42.684 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:42.739 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:42.739 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:37:42.739 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:37:42.739 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:37:42.739 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:37:42.739 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:37:42.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.739 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:37:42.739 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:42.739 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:37:42.740 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:42.740 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:37:42.740 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:37:42.740 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:42.740 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:42.740 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:42.740 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:37:42.740 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:42.740 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.740 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:42.740 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:37:42.740 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:42.741 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:37:42.741 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:37:42.741 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:42.741 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:42.741 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:42.741 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:37:42.741 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:42.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.741 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:42.741 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:37:42.741 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:42.741 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:37:42.742 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:37:42.742 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:42.742 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:37:42.742 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:42.742 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:37:42.742 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:42.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.742 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:42.742 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:37:42.742 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:42.742 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:37:42.743 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:37:42.743 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:42.743 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:37:42.743 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:42.743 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:37:42.743 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:42.743 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:42.743 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:42.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.743 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:37:42.743 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:37:42.744 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:37:42.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.744 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:37:42Z" 2024-04-12T04:37:42.744 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:42.745 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:37:42Z" } 2024-04-12T04:37:42.745 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:37:42.745 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:37:42.745 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:37:42.745 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:37:42.746 DEBUG:boto:path=/dkyges-68/ 2024-04-12T04:37:42.746 DEBUG:boto:auth_path=/dkyges-68/ 2024-04-12T04:37:42.746 DEBUG:boto:Method: HEAD 2024-04-12T04:37:42.746 DEBUG:boto:Path: /dkyges-68/ 2024-04-12T04:37:42.746 DEBUG:boto:Data: 2024-04-12T04:37:42.746 DEBUG:boto:Headers: {} 2024-04-12T04:37:42.746 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:37:42.746 DEBUG:boto:Port: 8001 2024-04-12T04:37:42.746 DEBUG:boto:Params: {} 2024-04-12T04:37:42.747 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:37:42.747 DEBUG:boto:Token: None 2024-04-12T04:37:42.748 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:37:42 GMT /dkyges-68/ 2024-04-12T04:37:42.748 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cEDarPlPzzScIa1rOivW3HkUcIA= 2024-04-12T04:37:42.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:37:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cEDarPlPzzScIa1rOivW3HkUcIA=', 'Content-Length': '0'} 2024-04-12T04:37:42.785 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009fbb8aaba42e891b-006618ba96-249c-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:37:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:37:42.785 DEBUG:boto:path=/dkyges-68/obj1 2024-04-12T04:37:42.785 DEBUG:boto:auth_path=/dkyges-68/obj1 2024-04-12T04:37:42.785 DEBUG:boto:Method: HEAD 2024-04-12T04:37:42.785 DEBUG:boto:Path: /dkyges-68/obj1 2024-04-12T04:37:42.785 DEBUG:boto:Data: 2024-04-12T04:37:42.786 DEBUG:boto:Headers: {} 2024-04-12T04:37:42.786 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:37:42.786 DEBUG:boto:Port: 8001 2024-04-12T04:37:42.786 DEBUG:boto:Params: {} 2024-04-12T04:37:42.786 DEBUG:boto:Token: None 2024-04-12T04:37:42.786 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:37:42 GMT /dkyges-68/obj1 2024-04-12T04:37:42.786 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oycoHSN4N7Pj1Ifbag8hKQHQVbY= 2024-04-12T04:37:42.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:37:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oycoHSN4N7Pj1Ifbag8hKQHQVbY=', 'Content-Length': '0'} 2024-04-12T04:37:42.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:36:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-68:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9370.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004bbe021a20013dcd-006618ba96-249c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:37:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:37:42.789 DEBUG:boto:path=/dkyges-68/obj1 2024-04-12T04:37:42.789 DEBUG:boto:auth_path=/dkyges-68/obj1 2024-04-12T04:37:42.789 DEBUG:boto:Method: GET 2024-04-12T04:37:42.789 DEBUG:boto:Path: /dkyges-68/obj1 2024-04-12T04:37:42.789 DEBUG:boto:Data: 2024-04-12T04:37:42.789 DEBUG:boto:Headers: {} 2024-04-12T04:37:42.789 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:37:42.789 DEBUG:boto:Port: 8001 2024-04-12T04:37:42.789 DEBUG:boto:Params: {} 2024-04-12T04:37:42.790 DEBUG:boto:Token: None 2024-04-12T04:37:42.790 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:37:42 GMT /dkyges-68/obj1 2024-04-12T04:37:42.790 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eJcZi6WU/X0Noh7c0dv/eS1spOQ= 2024-04-12T04:37:42.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:37:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eJcZi6WU/X0Noh7c0dv/eS1spOQ=', 'Content-Length': '0'} 2024-04-12T04:37:42.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:36:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-68:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9370.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca68280cce9d5937-006618ba96-249c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:37:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:37:42.792 DEBUG:boto:path=/dkyges-69/ 2024-04-12T04:37:42.792 DEBUG:boto:auth_path=/dkyges-69/ 2024-04-12T04:37:42.792 DEBUG:boto:Method: HEAD 2024-04-12T04:37:42.792 DEBUG:boto:Path: /dkyges-69/ 2024-04-12T04:37:42.792 DEBUG:boto:Data: 2024-04-12T04:37:42.792 DEBUG:boto:Headers: {} 2024-04-12T04:37:42.792 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:37:42.793 DEBUG:boto:Port: 8000 2024-04-12T04:37:42.793 DEBUG:boto:Params: {} 2024-04-12T04:37:42.793 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:37:42.793 DEBUG:boto:Token: None 2024-04-12T04:37:42.793 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:37:42 GMT /dkyges-69/ 2024-04-12T04:37:42.793 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:U3OROQ87fYNWR9YzwqhAJr9gjik= 2024-04-12T04:37:42.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:37:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3OROQ87fYNWR9YzwqhAJr9gjik=', 'Content-Length': '0'} 2024-04-12T04:37:42.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005139236537a2441a-006618ba96-2493-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:37:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:37:42.807 DEBUG:boto:path=/dkyges-69/obj2 2024-04-12T04:37:42.807 DEBUG:boto:auth_path=/dkyges-69/obj2 2024-04-12T04:37:42.807 DEBUG:boto:Method: HEAD 2024-04-12T04:37:42.807 DEBUG:boto:Path: /dkyges-69/obj2 2024-04-12T04:37:42.807 DEBUG:boto:Data: 2024-04-12T04:37:42.807 DEBUG:boto:Headers: {} 2024-04-12T04:37:42.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:37:42.808 DEBUG:boto:Port: 8000 2024-04-12T04:37:42.808 DEBUG:boto:Params: {} 2024-04-12T04:37:42.808 DEBUG:boto:Token: None 2024-04-12T04:37:42.808 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:37:42 GMT /dkyges-69/obj2 2024-04-12T04:37:42.808 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tBj91tpuQP+uMKUo4V+mlytIcgQ= 2024-04-12T04:37:42.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:37:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tBj91tpuQP+uMKUo4V+mlytIcgQ=', 'Content-Length': '0'} 2024-04-12T04:37:42.810 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000041bb12903a9457a4-006618ba96-2493-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:37:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:37:42.812 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... FAIL 2024-04-12T04:37:42.813 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2024-04-12T04:37:42.814 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2024-04-12T04:37:42.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:42.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:42.814 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:42.913 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:37:42.930 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:37:42.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:42.930 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:37:42.930 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:37:42.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:42.931 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:37:42.931 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:37:42.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:42.931 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:37:42.931 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:37:42.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:42.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:42.931 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896564.701296_877.1", 2024-04-12T04:37:42.931 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:36:04.701296Z" 2024-04-12T04:37:42.932 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:42.932 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:37:42.933 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:37:42.933 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:37:42.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:42.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:42.933 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:42.988 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:42.988 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:43.024+0000 7fc0068ef780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:43.025+0000 7fc0068ef780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:43.041 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:37:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:37:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:37:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:37:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:37:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:37:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:37:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:37:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:37:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:37:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:37:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:37:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:37:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:37:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:37:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:37:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:37:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896564.701296_877.1", 2024-04-12T04:37:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.701296Z", 2024-04-12T04:37:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:37:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:37:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:37:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:37:43Z" 2024-04-12T04:37:43.046 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:43.047 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896564.701296_877.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:36:04.701296Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:37:43Z" } 2024-04-12T04:37:43.047 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896564.701296_877.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.701296Z', 'realm_epoch': 2}}] 2024-04-12T04:37:43.047 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:37:43.048 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:37:43.048 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:37:43.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:43.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:43.048 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:37:43.098 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:43.098 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:43.151 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:37:43.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:43.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:43.152 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:37:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:43.260 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:37:43.261 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-70 2024-04-12T04:37:43.261 DEBUG:boto:path=/dkyges-70/ 2024-04-12T04:37:43.261 DEBUG:boto:auth_path=/dkyges-70/ 2024-04-12T04:37:43.261 DEBUG:boto:Method: PUT 2024-04-12T04:37:43.261 DEBUG:boto:Path: /dkyges-70/ 2024-04-12T04:37:43.261 DEBUG:boto:Data: 2024-04-12T04:37:43.261 DEBUG:boto:Headers: {} 2024-04-12T04:37:43.261 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:37:43.261 DEBUG:boto:Port: 8000 2024-04-12T04:37:43.262 DEBUG:boto:Params: {} 2024-04-12T04:37:43.262 DEBUG:boto:Token: None 2024-04-12T04:37:43.262 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:37:43 GMT /dkyges-70/ 2024-04-12T04:37:43.262 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:7T/KxO4SlFwYj5p6KJu/u8RnQ8E= 2024-04-12T04:37:43.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:37:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7T/KxO4SlFwYj5p6KJu/u8RnQ8E=', 'Content-Length': '0'} 2024-04-12T04:37:43.281 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fdc4e43a1bb7bed9-006618ba97-2493-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:37:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:37:43.281 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-71 2024-04-12T04:37:43.282 DEBUG:boto:path=/dkyges-71/ 2024-04-12T04:37:43.282 DEBUG:boto:auth_path=/dkyges-71/ 2024-04-12T04:37:43.282 DEBUG:boto:Method: PUT 2024-04-12T04:37:43.282 DEBUG:boto:Path: /dkyges-71/ 2024-04-12T04:37:43.282 DEBUG:boto:Data: 2024-04-12T04:37:43.282 DEBUG:boto:Headers: {} 2024-04-12T04:37:43.282 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:37:43.282 DEBUG:boto:Port: 8000 2024-04-12T04:37:43.282 DEBUG:boto:Params: {} 2024-04-12T04:37:43.283 DEBUG:boto:Token: None 2024-04-12T04:37:43.283 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:37:43 GMT /dkyges-71/ 2024-04-12T04:37:43.283 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ex0ja6wreKQWZDXl+HAJZOOTZfk= 2024-04-12T04:37:43.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:37:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ex0ja6wreKQWZDXl+HAJZOOTZfk=', 'Content-Length': '0'} 2024-04-12T04:37:43.302 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c6ea4302e00502be-006618ba97-2493-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:37:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:37:43.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:43.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:43.303 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:43.360 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:43.360 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:43.408 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:37:43.424 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:37:43.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:37:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:37:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:37:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:37:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:37:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:37:43.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:43.426 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:43.426 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:37:43.426 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:37:43.302022Z" 2024-04-12T04:37:43.426 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:43.426 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:37:43.427 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:37:43.427 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:37:43.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:43.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:43.428 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:43.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:43.516+0000 7f3f3d870780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:43.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:43.517+0000 7f3f3d870780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:43.532 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:43.532 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:37:43.532 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:37:43.532 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:37:43.532 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:37:43.533 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:37:43.533 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:43.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:43.533 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:37:43.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:43.533 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:37:43.533 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:43.533 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:43.533 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:37:43.533 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:43.533 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:37:43.534 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:43.534 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:37:43.534 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:43.534 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:43.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:43.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:43.534 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:37:43.534 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:43.534 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:43.534 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:37:43.534 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:43.535 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:43.535 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:43.535 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:37:43.535 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:43.535 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:43.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:43.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:43.535 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:37:43.535 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:43.535 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:43.535 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:37:43.535 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:43.536 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:43.536 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:43.536 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:37:43.536 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:43.536 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:43.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:43.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:43.536 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:37:43.536 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:43.536 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:43.536 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.273902_886.1", 2024-04-12T04:37:43.537 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:43.537 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:43.537 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:43.537 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.273902Z", 2024-04-12T04:37:43.537 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:43.537 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:43.537 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:43.537 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:43.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:43.537 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:37:43.537 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:37:43.537 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:37:43.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:43.538 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:37:43Z" 2024-04-12T04:37:43.538 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:43.539 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.273902_886.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.273902Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:37:43Z" } 2024-04-12T04:37:43.539 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.273902_886.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.273902Z', 'realm_epoch': 2}}] 2024-04-12T04:37:43.539 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:37:43.539 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.273902_886.1'} 2024-04-12T04:37:43.539 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896663.302022_893.1 target=1_1712896663.273902_886.1 2024-04-12T04:37:48.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:48.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:48.540 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:48.600 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:48.600 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:48.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:48.637+0000 7fdc775dc780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:48.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:48.639+0000 7fdc775dc780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:48.655 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:48.655 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:37:48.655 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:37:48.656 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:37:48.656 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:37:48.656 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:37:48.656 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:48.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:48.656 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:37:48.656 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:48.656 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:37:48.656 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:48.657 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:48.657 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:37:48.657 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:48.657 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:37:48.657 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:48.657 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:37:48.657 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:48.657 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:48.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:48.658 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:48.658 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:37:48.658 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:48.658 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:48.658 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:37:48.658 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:48.658 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:48.658 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:48.658 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:37:48.659 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:48.659 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:48.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:48.659 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:48.659 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:37:48.659 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:48.659 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:48.659 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:37:48.659 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:48.659 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:48.659 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:48.660 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:37:48.660 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:48.660 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:48.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:48.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:48.660 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:37:48.660 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:48.660 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:48.660 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:37:48.661 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:48.661 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:48.661 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:48.661 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:37:48.661 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:48.661 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:48.661 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:48.661 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:48.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:48.661 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:37:48.662 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:37:48.662 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:37:48.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:48.662 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:37:48Z" 2024-04-12T04:37:48.662 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:48.663 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:37:48Z" } 2024-04-12T04:37:48.663 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:37:48.663 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:37:48.664 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:37:48.664 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:37:48.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:48.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:48.664 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:37:48.721 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:48.721 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:48.778 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:37:48.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:48.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:48.779 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:37:48.838 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:48.838 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:48.895 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:37:48.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=dkyges-70', '--dest-bucket=dkyges-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:48.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=dkyges-70', '--dest-bucket=dkyges-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:48.896 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --source-bucket=dkyges-70 --dest-bucket=dkyges-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:37:48.956 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:48.956 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:49.012 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dkyges-70","zones":["a2","a1"]},"dest":{"bucket":"dkyges-70","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:37:49.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:49.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:49.013 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:37:49.067 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:49.067 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:49.122 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dkyges-70","zones":["a2","a1"]},"dest":{"bucket":"dkyges-70","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:37:49.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:49.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:49.123 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:37:49.173 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:49.173 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:49.230 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dkyges-70","zones":["a2","a1"]},"dest":{"bucket":"dkyges-70","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:37:49.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:49.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:49.231 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:37:49.290 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:49.290 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:49.357 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:49.357 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:37:49.357 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 16, 2024-04-12T04:37:49.357 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T04:37:49.357 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T04:37:49.357 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T04:37:49.357 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:37:49.357 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T04:37:49.357 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:49.358 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T04:37:49.358 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T04:37:49.358 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:37:49.358 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T04:37:49.358 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:37:49.358 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:37:49.358 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:37:49.358 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:37:49.358 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:37:49.358 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:37:49.358 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:37:49.359 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:49.359 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:37:49.359 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T04:37:49.359 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:37:49.359 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T04:37:49.359 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:37:49.359 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:37:49.359 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:37:49.359 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:37:49.359 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:37:49.359 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:37:49.360 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:37:49.360 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:37:49.360 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:37:49.360 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:37:49.360 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:37:49.360 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:37:49.360 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:49.360 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.360 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:49.360 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:37:49.360 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T04:37:49.361 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:37:49.361 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:37:49.361 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:37:49.361 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:37:49.361 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:37:49.361 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:37:49.361 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:37:49.361 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:37:49.361 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:37:49.361 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:37:49.361 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:37:49.362 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:37:49.362 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:37:49.362 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:37:49.362 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:49.362 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:49.362 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:37:49.362 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:37:49.362 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:49.362 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:37:49.363 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:37:49.363 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:37:49.363 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:37:49.363 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:49.363 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:49.363 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:37:49.363 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:37:49.363 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:37:49.363 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:37:49.364 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [ 2024-04-12T04:37:49.364 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:49.364 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-group", 2024-04-12T04:37:49.364 INFO:teuthology.orchestra.run.smithi043.stdout: "data_flow": { 2024-04-12T04:37:49.364 INFO:teuthology.orchestra.run.smithi043.stdout: "symmetrical": [ 2024-04-12T04:37:49.364 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:49.364 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-flow1", 2024-04-12T04:37:49.364 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:37:49.364 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:37:49.365 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:37:49.365 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:49.365 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:49.365 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:49.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.365 INFO:teuthology.orchestra.run.smithi043.stdout: "pipes": [ 2024-04-12T04:37:49.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:49.365 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-pipe", 2024-04-12T04:37:49.365 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:37:49.365 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-70", 2024-04-12T04:37:49.366 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:37:49.366 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:37:49.366 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:37:49.366 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:49.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.366 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": { 2024-04-12T04:37:49.366 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-70", 2024-04-12T04:37:49.366 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:37:49.366 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:37:49.366 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:37:49.366 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:49.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.367 INFO:teuthology.orchestra.run.smithi043.stdout: "params": { 2024-04-12T04:37:49.367 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:37:49.367 INFO:teuthology.orchestra.run.smithi043.stdout: "filter": { 2024-04-12T04:37:49.367 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [] 2024-04-12T04:37:49.367 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:49.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.367 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": {}, 2024-04-12T04:37:49.367 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-12T04:37:49.367 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "system", 2024-04-12T04:37:49.367 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "" 2024-04-12T04:37:49.367 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:49.368 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:49.368 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:37:49.368 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "enabled" 2024-04-12T04:37:49.368 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:49.368 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:49.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.368 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:37:49.368 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:37:49.368 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:49.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:49.369 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T04:37:49.369 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T04:37:49.369 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:37:49.369 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T04:37:49.369 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:37:49.369 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:37:49.369 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:37:49.369 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:37:49.369 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:37:49.369 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:37:49.370 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:37:49.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:49.370 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:37:49.370 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T04:37:49.370 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:37:49.370 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:37:49.370 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:37:49.370 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:37:49.370 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:37:49.370 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:37:49.370 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:37:49.371 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:37:49.371 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:37:49.371 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:37:49.371 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:37:49.371 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:37:49.371 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:37:49.371 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:37:49.371 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:49.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:49.371 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:37:49.372 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T04:37:49.372 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:37:49.372 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T04:37:49.372 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:37:49.372 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:37:49.372 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:37:49.372 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:37:49.372 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:37:49.372 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:37:49.372 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:37:49.372 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:37:49.372 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:37:49.373 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:37:49.373 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:37:49.373 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:37:49.373 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:49.373 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:49.373 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:37:49.373 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:37:49.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:49.373 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:37:49.373 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:37:49.373 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:37:49.374 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:37:49.374 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:49.374 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:49.374 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:37:49.374 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:37:49.374 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:37:49.374 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:37:49.374 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T04:37:49.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.374 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:37:49.375 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:37:49.375 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:49.375 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:49.375 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:37:49.375 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T04:37:49.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:49.375 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:37:49.375 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T04:37:49.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:49.375 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:37:49.377 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T04:37:49.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:49.377 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:37:49.377 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T04:37:49.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:49.377 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:37:49.377 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T04:37:49.377 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:49.377 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:49.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.378 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T04:37:49.378 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:37:49.378 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T04:37:49.378 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T04:37:49.378 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:37:49.378 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:37:49.378 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:37:49.378 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:37:49.379 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:37:49.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.379 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T04:37:49.379 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:37:49.379 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:37:49.379 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:37:49.379 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:37:49.379 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:37:49.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.379 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T04:37:49.380 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:37:49.380 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:37:49.380 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:37:49.380 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:37:49.380 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:37:49.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.380 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T04:37:49.380 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:37:49.380 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:37:49.380 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:37:49.380 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:37:49.381 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:37:49.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.381 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T04:37:49.381 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:37:49.381 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:37:49.381 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:37:49.381 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:37:49.381 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:37:49.381 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:49.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:49.382 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:37:49.382 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T04:37:49.382 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:49.382 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:49.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:49.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:37:49.383 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:37:49.434 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:49.434 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:49.507 INFO:teuthology.orchestra.run.smithi043.stdout:{"sources":[{"id":"sync-pipe","source":{"zone":"a2","bucket":"dkyges-70"},"dest":{"zone":"a1","bucket":"dkyges-70"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"a1","bucket":"dkyges-70"},"dest":{"zone":"a2","bucket":"dkyges-70"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2024-04-12T04:37:49.507 DEBUG:boto:path=/dkyges-70/ 2024-04-12T04:37:49.507 DEBUG:boto:auth_path=/dkyges-70/ 2024-04-12T04:37:49.507 DEBUG:boto:Method: HEAD 2024-04-12T04:37:49.507 DEBUG:boto:Path: /dkyges-70/ 2024-04-12T04:37:49.507 DEBUG:boto:Data: 2024-04-12T04:37:49.507 DEBUG:boto:Headers: {} 2024-04-12T04:37:49.508 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:37:49.508 DEBUG:boto:Port: 8000 2024-04-12T04:37:49.508 DEBUG:boto:Params: {} 2024-04-12T04:37:49.508 DEBUG:boto:Token: None 2024-04-12T04:37:49.508 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:37:49 GMT /dkyges-70/ 2024-04-12T04:37:49.508 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q9hLO0GWJNTOPLl7Ph8oi0PXGtk= 2024-04-12T04:37:49.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:37:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q9hLO0GWJNTOPLl7Ph8oi0PXGtk=', 'Content-Length': '0'} 2024-04-12T04:37:49.823 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c00e9446fa2cfdb-006618ba9d-25ce-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:37:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:37:49.824 DEBUG:boto:path=/dkyges-70/obj1 2024-04-12T04:37:49.824 DEBUG:boto:auth_path=/dkyges-70/obj1 2024-04-12T04:37:49.824 DEBUG:boto:Method: PUT 2024-04-12T04:37:49.824 DEBUG:boto:Path: /dkyges-70/obj1 2024-04-12T04:37:49.824 DEBUG:boto:Data: 2024-04-12T04:37:49.824 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:37:49.824 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:37:49.824 DEBUG:boto:Port: 8000 2024-04-12T04:37:49.825 DEBUG:boto:Params: {} 2024-04-12T04:37:49.825 DEBUG:boto:Token: None 2024-04-12T04:37:49.825 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:37:49 GMT /dkyges-70/obj1 2024-04-12T04:37:49.825 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:tsGY4/dADqieYajEOU8Qzw47QUA= 2024-04-12T04:37:49.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:37:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tsGY4/dADqieYajEOU8Qzw47QUA='} 2024-04-12T04:37:49.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008ad08b29d7811212-006618ba9d-25ce-a1'), ('Date', 'Fri, 12 Apr 2024 04:37:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:37:49.871 DEBUG:boto:path=/dkyges-70/ 2024-04-12T04:37:49.871 DEBUG:boto:auth_path=/dkyges-70/ 2024-04-12T04:37:49.871 DEBUG:boto:Method: HEAD 2024-04-12T04:37:49.871 DEBUG:boto:Path: /dkyges-70/ 2024-04-12T04:37:49.871 DEBUG:boto:Data: 2024-04-12T04:37:49.871 DEBUG:boto:Headers: {} 2024-04-12T04:37:49.871 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:37:49.871 DEBUG:boto:Port: 8000 2024-04-12T04:37:49.871 DEBUG:boto:Params: {} 2024-04-12T04:37:49.872 DEBUG:boto:Token: None 2024-04-12T04:37:49.872 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:37:49 GMT /dkyges-70/ 2024-04-12T04:37:49.872 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q9hLO0GWJNTOPLl7Ph8oi0PXGtk= 2024-04-12T04:37:49.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:37:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q9hLO0GWJNTOPLl7Ph8oi0PXGtk=', 'Content-Length': '0'} 2024-04-12T04:37:49.880 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6b809145ddb9a92-006618ba9d-25ce-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:37:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:37:49.880 DEBUG:boto:path=/dkyges-70/obj2 2024-04-12T04:37:49.880 DEBUG:boto:auth_path=/dkyges-70/obj2 2024-04-12T04:37:49.880 DEBUG:boto:Method: PUT 2024-04-12T04:37:49.880 DEBUG:boto:Path: /dkyges-70/obj2 2024-04-12T04:37:49.881 DEBUG:boto:Data: 2024-04-12T04:37:49.881 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:37:49.881 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:37:49.881 DEBUG:boto:Port: 8000 2024-04-12T04:37:49.881 DEBUG:boto:Params: {} 2024-04-12T04:37:49.881 DEBUG:boto:Token: None 2024-04-12T04:37:49.881 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:37:49 GMT /dkyges-70/obj2 2024-04-12T04:37:49.881 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8KOY48qHAR4vg4WIMfEh+ZJApXM= 2024-04-12T04:37:49.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:37:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8KOY48qHAR4vg4WIMfEh+ZJApXM='} 2024-04-12T04:37:49.899 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000098bd7f3b9ee4e38f-006618ba9d-25ce-a1'), ('Date', 'Fri, 12 Apr 2024 04:37:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:37:49.899 DEBUG:boto:path=/dkyges-70/ 2024-04-12T04:37:49.900 DEBUG:boto:auth_path=/dkyges-70/ 2024-04-12T04:37:49.900 DEBUG:boto:Method: HEAD 2024-04-12T04:37:49.900 DEBUG:boto:Path: /dkyges-70/ 2024-04-12T04:37:49.900 DEBUG:boto:Data: 2024-04-12T04:37:49.900 DEBUG:boto:Headers: {} 2024-04-12T04:37:49.900 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:37:49.900 DEBUG:boto:Port: 8000 2024-04-12T04:37:49.900 DEBUG:boto:Params: {} 2024-04-12T04:37:49.900 DEBUG:boto:Token: None 2024-04-12T04:37:49.900 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:37:49 GMT /dkyges-70/ 2024-04-12T04:37:49.901 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:q9hLO0GWJNTOPLl7Ph8oi0PXGtk= 2024-04-12T04:37:49.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:37:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q9hLO0GWJNTOPLl7Ph8oi0PXGtk=', 'Content-Length': '0'} 2024-04-12T04:37:49.907 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044e8242eb59f8e37-006618ba9d-25ce-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:37:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:37:49.907 DEBUG:boto:path=/dkyges-70/obj3 2024-04-12T04:37:49.907 DEBUG:boto:auth_path=/dkyges-70/obj3 2024-04-12T04:37:49.907 DEBUG:boto:Method: PUT 2024-04-12T04:37:49.907 DEBUG:boto:Path: /dkyges-70/obj3 2024-04-12T04:37:49.907 DEBUG:boto:Data: 2024-04-12T04:37:49.907 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:37:49.907 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:37:49.907 DEBUG:boto:Port: 8000 2024-04-12T04:37:49.908 DEBUG:boto:Params: {} 2024-04-12T04:37:49.908 DEBUG:boto:Token: None 2024-04-12T04:37:49.908 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:37:49 GMT /dkyges-70/obj3 2024-04-12T04:37:49.908 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JdwVkzYQa/j9HU9bwFW8RDrQx2w= 2024-04-12T04:37:49.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:37:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JdwVkzYQa/j9HU9bwFW8RDrQx2w='} 2024-04-12T04:37:49.950 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e487819fe683d326-006618ba9d-25ce-a1'), ('Date', 'Fri, 12 Apr 2024 04:37:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:37:49.951 DEBUG:boto:path=/dkyges-71/ 2024-04-12T04:37:49.951 DEBUG:boto:auth_path=/dkyges-71/ 2024-04-12T04:37:49.951 DEBUG:boto:Method: HEAD 2024-04-12T04:37:49.951 DEBUG:boto:Path: /dkyges-71/ 2024-04-12T04:37:49.951 DEBUG:boto:Data: 2024-04-12T04:37:49.951 DEBUG:boto:Headers: {} 2024-04-12T04:37:49.951 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:37:49.951 DEBUG:boto:Port: 8000 2024-04-12T04:37:49.951 DEBUG:boto:Params: {} 2024-04-12T04:37:49.951 DEBUG:boto:Token: None 2024-04-12T04:37:49.951 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:37:49 GMT /dkyges-71/ 2024-04-12T04:37:49.952 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:LZyzntVy0tv34a3orRdQ9zDnaL4= 2024-04-12T04:37:49.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:37:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LZyzntVy0tv34a3orRdQ9zDnaL4=', 'Content-Length': '0'} 2024-04-12T04:37:49.955 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecf9302c83b27792-006618ba9d-25ce-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:37:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:37:49.955 DEBUG:boto:path=/dkyges-71/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-12T04:37:49.955 DEBUG:boto:auth_path=/dkyges-71/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-12T04:37:49.955 DEBUG:boto:Method: PUT 2024-04-12T04:37:49.955 DEBUG:boto:Path: /dkyges-71/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-12T04:37:49.955 DEBUG:boto:Data: 2024-04-12T04:37:49.955 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:37:49.955 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:37:49.956 DEBUG:boto:Port: 8000 2024-04-12T04:37:49.956 DEBUG:boto:Params: {} 2024-04-12T04:37:49.956 DEBUG:boto:Token: None 2024-04-12T04:37:49.956 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:37:49 GMT /dkyges-71/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2024-04-12T04:37:49.956 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gycBlLTjfJHbSiXtMUtnaU3P2lw= 2024-04-12T04:37:49.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:37:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gycBlLTjfJHbSiXtMUtnaU3P2lw='} 2024-04-12T04:37:49.980 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfd0483f4730d6eb-006618ba9d-25ce-a1'), ('Date', 'Fri, 12 Apr 2024 04:37:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:37:49.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:49.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:49.980 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:50.066 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:50.067 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:50.152 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:37:50.189 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:37:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:37:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:37:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:50.189 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:37:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:37:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:37:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:37:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:37:50.190 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:37:43.302022Z" 2024-04-12T04:37:50.191 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:50.191 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:37:50.192 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:37:50.192 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:37:50.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:50.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:50.192 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:50.277 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:50.277 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:50.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:50.342+0000 7f4798988780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:50.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:37:50.343+0000 7f4798988780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:37:50.366 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:37:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:37:50.366 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:37:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:37:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:37:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.367 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:37:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:37:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:50.368 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:37:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:37:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:37:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:50.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:37:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:50.370 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:37:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:37:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:50.371 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:37:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:50.372 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:37:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:37:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:50.373 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:37:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:37:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:37:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:50.374 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:37:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:37:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:37:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:37:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:37:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.375 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:37:50Z" 2024-04-12T04:37:50.375 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:50.376 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:37:50Z" } 2024-04-12T04:37:50.377 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:37:50.377 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:37:50.377 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:37:50.377 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:37:50.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:50.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:50.377 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:50.441 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:50.442 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:50.519 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:37:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:37:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:37:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:37:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:37:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:50.520 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054070", 2024-04-12T04:37:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:37:49.925093Z" 2024-04-12T04:37:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:37:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:37:50.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:50.521 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:37:50.522 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:37:50.522 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:37:50.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:50.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:50.523 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:50.579 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:50.579 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:50.632 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:50.632 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:37:50.632 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:37:50.633 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:37:50.633 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:37:50.633 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:37:50.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.633 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:37:50.633 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:50.633 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:37:50.633 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:50.634 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:37:50.634 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:37:50.634 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:50.634 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:50.634 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:50.634 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:37:50.634 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:50.634 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.634 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:50.634 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:37:50.635 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:50.635 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:37:50.635 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:37:50.635 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:50.635 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:50.635 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:50.635 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:37:50.635 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:50.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:50.635 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:37:50.636 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:50.636 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:37:50.636 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:37:50.636 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:50.636 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:37:50.636 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:50.636 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:37:50.636 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:50.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.636 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:50.636 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:37:50.636 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:50.637 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:37:50.637 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:37:50.637 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:50.637 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:37:50.637 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:50.637 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:37:50.637 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:50.637 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:50.637 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:50.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.638 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:37:50.638 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:37:50.638 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:37:50.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:50.638 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:37:50Z" 2024-04-12T04:37:50.638 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:50.639 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:37:50Z" } 2024-04-12T04:37:50.639 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:37:50.639 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:37:50.639 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:37:50.640 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:37:55.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:55.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:37:55.641 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:37:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:37:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:37:55.758 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:37:55.759 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:37:55.759 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:37:55.759 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:37:55.759 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:37:55.759 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:37:55.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:55.760 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:37:55.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:55.760 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:37:55.761 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:55.761 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:37:55.761 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:37:55.761 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:55.761 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:55.762 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:55.762 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:37:55.762 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:55.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:55.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:55.763 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:37:55.763 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:55.763 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:37:55.763 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:37:55.763 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:55.764 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:37:55.764 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:55.764 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:37:55.764 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:55.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:55.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:55.765 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:37:55.765 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:55.765 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:37:55.766 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:37:55.766 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:55.766 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:37:55.766 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:55.766 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:37:55.767 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:55.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:55.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:37:55.767 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:37:55.767 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:37:55.768 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:37:55.768 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:37:55.768 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:37:55.768 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:37:55.768 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:37:55.769 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:37:55.769 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:55.769 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:37:55.769 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:37:55.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:55.770 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:37:55.770 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:37:55.770 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:37:55.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:37:55.771 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:37:55Z" 2024-04-12T04:37:55.771 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:37:55.772 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:37:55Z" } 2024-04-12T04:37:55.773 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:37:55.773 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:37:55.773 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:37:55.773 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:38:00.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:00.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:00.775 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:38:00.837 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:38:00.837 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:38:00.893 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:38:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:38:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:38:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:38:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:38:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:38:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:38:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:38:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:38:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:38:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:38:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:38:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:38:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:38:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:38:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:38:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:38:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:38:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:38:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:38:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:38:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:38:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:38:00.898 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:38:00.898 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:38:00.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:00.898 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:38:00Z" 2024-04-12T04:38:00.898 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:38:00.899 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:38:00Z" } 2024-04-12T04:38:00.899 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:38:00.899 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:38:00.899 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:38:00.899 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:38:05.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:05.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:05.900 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:38:05.962 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:38:05.963 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:38:06.019 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:38:06.019 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:38:06.019 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:38:06.020 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:38:06.020 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:38:06.020 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:38:06.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:06.020 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:38:06.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:06.020 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:38:06.020 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:06.020 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:06.020 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:38:06.020 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:06.021 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:06.021 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:06.021 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:38:06.021 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:06.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:06.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:06.021 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:38:06.021 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:06.021 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:06.021 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:38:06.021 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:06.022 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:06.022 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:06.022 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:38:06.022 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:06.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:06.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:06.022 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:38:06.022 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:06.022 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:06.022 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:38:06.023 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:06.023 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:38:06.023 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:06.023 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:38:06.023 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:06.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:06.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:06.023 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:38:06.023 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:06.023 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:06.023 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:38:06.024 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:06.024 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:38:06.024 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:06.024 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:38:06.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:06.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:06.024 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:38:06.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:06.024 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:38:06.024 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:38:06.024 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:38:06.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:06.025 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:38:06Z" 2024-04-12T04:38:06.025 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:38:06.026 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:38:06Z" } 2024-04-12T04:38:06.026 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:38:06.026 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:38:06.026 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:38:06.026 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:38:11.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:11.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:11.028 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:38:11.090 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:38:11.090 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:38:11.147 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:38:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:38:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:38:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:38:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:38:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:38:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:38:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:38:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:38:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:38:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:38:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:38:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:38:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:38:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:38:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:38:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:38:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:38:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:38:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:38:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:38:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:38:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:38:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:38:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:38:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:38:11Z" 2024-04-12T04:38:11.152 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:38:11.153 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:38:11Z" } 2024-04-12T04:38:11.153 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:38:11.154 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:38:11.154 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:38:11.154 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:38:16.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:16.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:16.156 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:38:16.218 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:38:16.218 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:38:16.274 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:38:16.274 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:38:16.274 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:38:16.274 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:38:16.274 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:38:16.274 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:38:16.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:16.275 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:38:16.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:16.275 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:38:16.275 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:16.275 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:16.275 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:38:16.275 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:16.275 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:16.276 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:16.276 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:38:16.276 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:16.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:16.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:16.276 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:38:16.276 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:16.276 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:16.276 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:38:16.277 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:16.277 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:16.277 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:16.277 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:38:16.277 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:16.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:16.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:16.277 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:38:16.277 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:16.277 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:16.277 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:38:16.278 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:16.278 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:38:16.278 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:16.278 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:38:16.278 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:16.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:16.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:16.278 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:38:16.278 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:16.278 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:16.279 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:38:16.279 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:16.279 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:38:16.279 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:16.279 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:38:16.279 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:16.279 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:16.279 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:38:16.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:16.279 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:38:16.279 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:38:16.280 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:38:16.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:16.280 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:38:16Z" 2024-04-12T04:38:16.280 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:38:16.281 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:38:16Z" } 2024-04-12T04:38:16.281 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:38:16.281 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:38:16.281 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:38:16.281 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:38:21.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:21.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:21.283 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:38:21.346 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:38:21.346 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:38:21.402 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:38:21.402 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:38:21.402 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:38:21.403 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:38:21.403 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:38:21.403 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:38:21.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:21.403 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:38:21.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:21.403 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:38:21.403 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:21.403 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:21.403 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:38:21.403 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:21.404 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:21.404 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:21.404 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:38:21.404 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:21.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:21.404 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:21.404 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:38:21.404 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:21.404 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:21.404 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:38:21.404 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:21.405 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:21.405 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:21.405 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:38:21.405 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:21.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:21.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:21.405 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:38:21.405 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:21.405 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:21.405 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:38:21.405 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:21.406 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:38:21.406 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:21.406 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:38:21.406 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:21.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:21.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:21.406 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:38:21.406 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:21.406 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:21.406 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:38:21.406 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:21.407 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:38:21.407 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:21.407 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:38:21.407 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:21.407 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:21.407 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:38:21.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:21.407 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:38:21.407 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:38:21.407 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:38:21.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:21.408 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:38:21Z" 2024-04-12T04:38:21.408 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:38:21.408 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:38:21Z" } 2024-04-12T04:38:21.409 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:38:21.409 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:38:21.409 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:38:21.409 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:38:26.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:26.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:26.411 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:38:26.473 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:38:26.473 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:38:26.529 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:38:26.529 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:38:26.529 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:38:26.530 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:38:26.530 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:38:26.530 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:38:26.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:26.530 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:38:26.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:26.530 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:38:26.530 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:26.530 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:26.530 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:38:26.530 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:26.531 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:26.531 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:26.531 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:38:26.531 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:26.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:26.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:26.531 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:38:26.531 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:26.531 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:26.531 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:38:26.532 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:26.532 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:26.532 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:26.532 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:38:26.532 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:26.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:26.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:26.532 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:38:26.532 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:26.532 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:26.532 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:38:26.532 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:38:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:38:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:38:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:38:26.533 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:26.534 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:38:26.534 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:26.534 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:38:26.534 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:26.534 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:26.534 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:38:26.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:26.534 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:38:26.534 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:38:26.534 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:38:26.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:26.535 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:38:26Z" 2024-04-12T04:38:26.535 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:38:26.536 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:38:26Z" } 2024-04-12T04:38:26.536 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:38:26.536 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:38:26.536 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:38:26.536 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:38:31.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:31.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:31.538 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:38:31.599 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:38:31.599 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:38:31.655 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:38:31.655 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:38:31.655 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:38:31.655 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:38:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:38:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:38:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:38:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:38:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:38:31.656 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:38:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:38:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:38:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:31.657 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:38:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:38:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:38:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:38:31.658 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:38:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:38:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:38:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:38:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:31.659 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:38:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:38:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:38:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:38:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:38:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:31.660 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:38:31Z" 2024-04-12T04:38:31.660 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:38:31.661 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:38:31Z" } 2024-04-12T04:38:31.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:38:31.661 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:38:31.661 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:38:31.661 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:38:36.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:36.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:36.663 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:38:36.726 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:38:36.726 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:38:36.783 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:38:36.783 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:38:36.783 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:38:36.783 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:38:36.783 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:38:36.783 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:38:36.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:36.783 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:38:36.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:36.783 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:38:36.784 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:36.784 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:36.784 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:38:36.784 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:36.784 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:36.784 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:36.784 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:38:36.784 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:36.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:36.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:36.784 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:38:36.785 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:36.785 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:36.785 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:38:36.785 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:36.785 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:36.785 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:36.785 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:38:36.785 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:36.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:36.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:36.785 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:38:36.786 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:36.786 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:36.786 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:38:36.786 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:36.786 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:38:36.786 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:36.786 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:38:36.786 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:36.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:36.786 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:36.786 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:38:36.787 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:36.787 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:36.787 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:38:36.787 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:36.787 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:38:36.787 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:36.787 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:38:36.787 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:36.787 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:36.787 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:38:36.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:36.787 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:38:36.787 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:38:36.788 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:38:36.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:36.788 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:38:36Z" 2024-04-12T04:38:36.788 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:38:36.789 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:38:36Z" } 2024-04-12T04:38:36.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:38:36.789 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:38:36.789 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:38:36.789 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:38:41.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:41.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:41.791 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:38:41.853 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:38:41.853 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:38:41.908 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:38:41.909 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:38:41.909 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:38:41.909 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:38:41.909 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:38:41.909 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:38:41.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:41.909 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:38:41.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:41.909 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:38:41.910 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:41.910 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:41.910 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:38:41.910 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:41.910 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:41.910 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:41.910 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:38:41.910 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:41.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:41.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:41.911 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:38:41.911 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:41.911 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:41.911 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:38:41.911 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:41.911 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:41.911 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:41.911 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:38:41.911 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:41.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:41.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:41.912 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:38:41.912 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:41.912 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:41.912 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:38:41.912 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:41.912 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:38:41.912 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:41.912 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:38:41.913 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:41.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:41.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:41.913 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:38:41.913 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:41.913 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:41.913 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:38:41.913 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:41.913 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:38:41.913 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:41.913 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:38:41.914 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:41.914 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:41.914 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:38:41.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:41.914 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:38:41.914 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:38:41.914 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:38:41.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:41.914 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:38:41Z" 2024-04-12T04:38:41.915 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:38:41.916 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:38:41Z" } 2024-04-12T04:38:41.916 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:38:41.916 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:38:41.916 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:38:41.917 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:38:46.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:46.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:46.918 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:38:46.981 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:38:46.981 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:38:47.036 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:38:47.036 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:38:47.037 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:38:47.037 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:38:47.037 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:38:47.037 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:38:47.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:47.037 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:38:47.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:47.037 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:38:47.037 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:47.037 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:47.037 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:38:47.038 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:47.038 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:47.038 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:47.038 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:38:47.038 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:47.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:47.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:47.038 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:38:47.038 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:47.038 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:47.039 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:38:47.039 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:47.039 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:47.039 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:47.039 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:38:47.039 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:47.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:47.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:47.039 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:38:47.039 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:47.039 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:47.040 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:38:47.040 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:47.040 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:38:47.040 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:47.040 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:38:47.040 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:47.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:47.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:47.040 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:38:47.040 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:47.041 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:47.041 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:38:47.041 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:47.041 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:38:47.041 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:47.041 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:38:47.041 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:47.041 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:47.041 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:38:47.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:47.041 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:38:47.042 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:38:47.042 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:38:47.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:47.042 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:38:47Z" 2024-04-12T04:38:47.042 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:38:47.043 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:38:47Z" } 2024-04-12T04:38:47.043 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:38:47.043 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:38:47.043 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:38:47.043 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:38:52.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:52.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:52.045 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:38:52.107 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:38:52.107 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:38:52.167 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:38:52.167 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:38:52.167 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:38:52.167 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:38:52.167 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:38:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:38:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:38:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:38:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:38:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:38:52.168 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:38:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:38:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:38:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:52.169 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:38:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:38:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:38:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:38:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:52.170 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:38:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:38:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:38:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:38:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:52.171 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:52.172 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:38:52.172 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:52.172 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:38:52.172 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:38:52.172 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:38:52.172 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:52.172 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:38:52Z" 2024-04-12T04:38:52.172 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:38:52.173 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:38:52Z" } 2024-04-12T04:38:52.173 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:38:52.173 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:38:52.173 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:38:52.173 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:38:57.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:57.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:38:57.176 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:38:57.238 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:38:57.238 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:38:57.297 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:38:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:38:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:38:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:38:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:38:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:38:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:38:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:38:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:38:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:38:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:38:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:38:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:38:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:38:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:38:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:38:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:38:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:38:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:38:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:38:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:38:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:38:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:38:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:38:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:38:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:38:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:38:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:38:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:38:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:38:57.302 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:38:57.303 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:38:57.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:38:57.303 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:38:57Z" 2024-04-12T04:38:57.303 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:38:57.304 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:38:57Z" } 2024-04-12T04:38:57.304 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:38:57.304 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:38:57.304 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:38:57.304 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:39:02.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:02.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:02.306 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:39:02.368 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:02.368 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:02.426 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:39:02.426 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:39:02.426 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:39:02.426 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:39:02.426 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:39:02.426 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:39:02.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:02.427 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:39:02.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:02.427 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:39:02.427 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:02.427 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:02.427 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:39:02.427 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:02.427 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:02.427 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:02.427 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:39:02.428 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:02.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:02.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:02.428 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:39:02.428 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:02.428 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:02.428 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:39:02.428 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:02.428 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:02.428 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:02.428 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:39:02.429 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:02.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:02.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:02.429 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:39:02.429 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:02.429 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:02.429 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:39:02.429 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:02.429 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:39:02.429 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:02.430 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:39:02.430 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:02.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:02.430 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:02.430 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:39:02.430 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:02.430 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:02.430 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:39:02.430 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:02.430 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:39:02.431 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:02.431 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:39:02.431 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:02.431 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:02.431 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:02.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:02.431 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:39:02.431 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:39:02.431 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:39:02.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:02.431 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:39:02Z" 2024-04-12T04:39:02.432 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:39:02.433 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:39:02Z" } 2024-04-12T04:39:02.433 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:39:02.433 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:39:02.433 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:39:02.433 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:39:07.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:07.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:07.435 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:39:07.496 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:07.496 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:07.552 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:39:07.552 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:39:07.552 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:39:07.552 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:39:07.552 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:39:07.552 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:39:07.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:07.552 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:39:07.553 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:07.553 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:39:07.553 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:07.553 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:07.553 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:39:07.553 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:07.553 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:07.553 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:07.553 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:39:07.554 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:07.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:07.554 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:07.554 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:39:07.554 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:07.554 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:07.554 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:39:07.554 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:07.554 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:07.555 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:07.555 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:39:07.555 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:07.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:07.555 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:07.555 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:39:07.555 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:07.555 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:07.555 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:39:07.555 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:07.556 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:39:07.556 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:07.556 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:39:07.556 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:07.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:07.556 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:07.556 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:39:07.556 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:07.556 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:07.556 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:39:07.556 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:07.557 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:39:07.557 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:07.557 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:39:07.557 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:07.557 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:07.557 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:07.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:07.557 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:39:07.557 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:39:07.557 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:39:07.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:07.558 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:39:07Z" 2024-04-12T04:39:07.558 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:39:07.559 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:39:07Z" } 2024-04-12T04:39:07.559 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:39:07.559 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:39:07.559 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:39:07.559 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:39:12.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:12.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:12.560 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:39:12.623 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:12.623 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:12.679 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:39:12.679 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:39:12.679 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:39:12.680 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:39:12.680 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:39:12.680 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:39:12.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:12.680 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:39:12.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:12.680 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:39:12.680 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:12.680 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:12.680 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:39:12.680 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:12.681 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:12.681 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:12.681 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:39:12.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:12.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:12.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:12.681 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:39:12.681 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:12.681 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:12.681 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:39:12.681 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:12.681 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:12.682 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:12.682 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:39:12.682 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:12.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:12.682 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:12.682 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:39:12.682 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:12.682 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:12.682 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:39:12.682 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:12.682 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:39:12.682 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:12.683 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:39:12.683 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:12.683 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:12.683 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:12.683 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:39:12.683 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:12.683 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:12.683 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:39:12.683 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:12.683 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:39:12.684 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:12.684 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:39:12.684 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:12.684 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:12.684 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:12.684 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:12.684 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:39:12.684 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:39:12.684 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:39:12.684 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:12.684 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:39:12Z" 2024-04-12T04:39:12.684 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:39:12.685 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:39:12Z" } 2024-04-12T04:39:12.685 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:39:12.686 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:39:12.686 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:39:12.686 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:39:17.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:17.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:17.687 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:39:17.751 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:17.751 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:17.807 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:39:17.807 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:39:17.807 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:39:17.808 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:39:17.808 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:39:17.808 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:39:17.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:17.808 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:39:17.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:17.808 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:39:17.808 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:17.808 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:17.808 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:39:17.809 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:17.809 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:17.809 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:17.809 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:39:17.809 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:17.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:17.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:17.809 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:39:17.809 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:17.809 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:17.809 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:39:17.810 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:17.810 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:17.810 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:17.810 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:39:17.810 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:17.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:17.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:17.810 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:39:17.810 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:17.810 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:17.810 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:39:17.811 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:17.811 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:39:17.811 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:17.811 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:39:17.811 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:17.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:17.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:17.811 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:39:17.811 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:17.811 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:17.811 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:39:17.812 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:17.812 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:39:17.812 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:17.812 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:39:17.812 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:17.812 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:17.812 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:17.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:17.812 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:39:17.812 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:39:17.812 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:39:17.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:17.813 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:39:17Z" 2024-04-12T04:39:17.813 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:39:17.814 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:39:17Z" } 2024-04-12T04:39:17.814 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:39:17.814 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:39:17.814 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:39:17.814 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:39:22.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:22.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:22.815 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:39:22.876 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:22.877 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:22.932 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:39:22.932 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:39:22.932 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:39:22.932 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:39:22.932 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:39:22.932 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:39:22.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:22.933 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:39:22.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:22.933 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:39:22.933 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:22.933 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:22.933 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:39:22.933 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:22.933 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:22.933 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:22.933 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:39:22.934 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:22.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:22.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:22.934 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:39:22.934 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:22.934 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:22.934 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:39:22.934 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:22.934 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:22.934 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:22.935 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:39:22.935 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:22.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:22.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:22.935 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:39:22.935 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:22.935 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:22.935 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:39:22.935 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:22.935 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:39:22.935 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:39:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:39:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:39:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:39:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:39:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:39:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:39:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:39:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:39:22Z" 2024-04-12T04:39:22.937 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:39:22.938 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:39:22Z" } 2024-04-12T04:39:22.938 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:39:22.939 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:39:22.939 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:39:22.939 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:39:27.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:27.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:27.940 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:39:28.002 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:28.002 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:28.057 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:39:28.057 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:39:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:39:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:39:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:39:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:39:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:39:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:28.058 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:39:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:39:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:39:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:28.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:39:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:39:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:39:28.060 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:28.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:28.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:28.061 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:39:28.061 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:28.061 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:28.061 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:39:28.061 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:28.061 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:39:28.061 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:28.062 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:39:28.062 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:28.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:28.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:28.062 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:39:28.062 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:28.062 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:28.062 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:39:28.062 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:28.063 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:39:28.063 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:28.063 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:39:28.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:28.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:28.063 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:28.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:28.063 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:39:28.063 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:39:28.063 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:39:28.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:28.064 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:39:28Z" 2024-04-12T04:39:28.064 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:39:28.065 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:39:28Z" } 2024-04-12T04:39:28.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:39:28.065 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:39:28.065 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:39:28.065 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:39:33.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:33.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:33.067 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:39:33.128 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:33.128 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:33.183 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:39:33.183 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:39:33.184 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:39:33.184 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:39:33.184 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:39:33.184 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:39:33.184 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:33.184 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:39:33.184 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:33.184 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:39:33.184 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:33.184 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:33.184 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:39:33.184 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:33.185 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:33.185 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:33.185 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:39:33.185 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:33.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:33.185 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:33.185 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:39:33.185 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:33.185 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:33.185 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:39:33.185 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:33.186 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:33.186 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:33.186 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:39:33.186 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:33.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:33.186 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:33.186 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:39:33.186 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:33.186 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:33.186 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:39:33.186 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:33.187 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:39:33.187 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:33.187 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:39:33.187 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:33.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:33.187 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:33.187 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:39:33.187 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:33.187 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:33.187 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:39:33.187 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:33.188 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:39:33.188 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:33.188 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:39:33.188 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:33.188 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:33.188 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:33.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:33.188 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:39:33.188 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:39:33.188 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:39:33.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:33.188 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:39:33Z" 2024-04-12T04:39:33.188 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:39:33.189 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:39:33Z" } 2024-04-12T04:39:33.189 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:39:33.190 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:39:33.190 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:39:33.190 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:39:38.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:38.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:38.191 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:39:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:38.308 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:39:38.308 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:39:38.308 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:39:38.308 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:39:38.309 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:39:38.309 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:39:38.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:38.309 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:39:38.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:38.309 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:39:38.309 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:38.309 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:38.309 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:39:38.309 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:38.309 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:38.310 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:38.310 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:39:38.310 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:38.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:38.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:38.310 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:39:38.310 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:38.310 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:38.310 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:39:38.310 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:38.311 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:38.311 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:38.311 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:39:38.311 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:38.311 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:38.311 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:38.311 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:39:38.311 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:38.311 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:38.311 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:39:38.311 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:38.312 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:39:38.312 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:38.312 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:39:38.312 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:38.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:38.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:38.312 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:39:38.312 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:38.312 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:38.312 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:39:38.312 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:38.312 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:39:38.313 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:38.313 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:39:38.313 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:38.313 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:38.313 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:38.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:38.313 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:39:38.313 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:39:38.313 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:39:38.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:38.313 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:39:38Z" 2024-04-12T04:39:38.313 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:39:38.314 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:39:38Z" } 2024-04-12T04:39:38.315 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:39:38.315 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:39:38.315 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:39:38.315 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:39:43.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:43.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:43.317 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:39:43.381 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:43.381 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:43.436 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:39:43.436 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:39:43.436 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:39:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:39:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:39:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:39:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:39:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:39:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:39:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:39:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:39:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:39:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:39:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:39:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:39:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:39:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:39:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:39:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:39:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:39:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:39:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:39:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:39:43.442 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:39:43.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:43.442 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:39:43Z" 2024-04-12T04:39:43.442 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:39:43.443 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:39:43Z" } 2024-04-12T04:39:43.443 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:39:43.443 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:39:43.443 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:39:43.443 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:39:48.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:48.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:48.445 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:39:48.511 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:48.512 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:48.567 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:39:48.568 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:39:48.568 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:39:48.568 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:39:48.568 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:39:48.568 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:39:48.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:48.568 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:39:48.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:48.568 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:39:48.568 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:48.568 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:48.568 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:39:48.569 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:48.569 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:48.569 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:48.569 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:39:48.569 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:48.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:48.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:48.569 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:39:48.569 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:48.569 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:48.569 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:39:48.569 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:48.569 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:48.570 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:48.570 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:39:48.570 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:48.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:48.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:48.570 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:39:48.570 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:48.570 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:48.570 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:39:48.570 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:48.570 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:39:48.571 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:48.571 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:39:48.571 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:48.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:48.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:48.571 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:39:48.571 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:48.571 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:48.571 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:39:48.571 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:48.571 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:39:48.571 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:48.572 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:39:48.572 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:48.572 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:48.572 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:48.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:48.572 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:39:48.572 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:39:48.572 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:39:48.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:48.572 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:39:48Z" 2024-04-12T04:39:48.572 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:39:48.573 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:48.591190Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000053790", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:36:04.610431Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046432", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:28:48.526647Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:39:48Z" } 2024-04-12T04:39:48.573 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:39:48.574 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:39:48.574 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:39:48.574 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047638 target=00000000000000000000:00000000000000047497, shard=2 master=00000000000000000000:00000000000000054070 target=00000000000000000000:00000000000000053790, shard=3 master=00000000000000000000:00000000000000046574 target=00000000000000000000:00000000000000046432 2024-04-12T04:39:53.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:53.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:53.575 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:39:53.639 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:53.639 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:53.696 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:39:53.696 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:39:53.696 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:39:53.696 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:39:53.696 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:39:53.696 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:39:53.696 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:53.696 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:39:53.697 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:53.697 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:39:53.697 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:53.697 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:53.697 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:39:53.697 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:53.697 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:53.697 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:53.697 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:39:53.697 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:53.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:53.698 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:53.698 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:39:53.698 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:53.698 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:53.698 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:39:53.698 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:53.698 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:53.698 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:53.698 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:39:53.699 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:53.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:53.699 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:53.699 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:39:53.699 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:53.699 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:53.699 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054070", 2024-04-12T04:39:53.699 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:53.699 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:39:53.699 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:53.699 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.925093Z" 2024-04-12T04:39:53.699 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:53.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:53.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:53.700 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:39:53.700 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:53.700 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:39:53.700 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:39:53.700 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:53.700 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:39:53.700 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:53.700 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:39:53.700 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:53.700 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:53.701 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:53.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:53.701 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:39:53.701 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:39:53.701 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:39:53.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:53.701 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:39:53Z" 2024-04-12T04:39:53.701 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:39:53.702 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054070", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:37:49.925093Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:39:53Z" } 2024-04-12T04:39:53.702 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054070', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.925093Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:39:53.702 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:39:53.703 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054070', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:39:53.703 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:39:53.703 DEBUG:boto:path=/dkyges-70/ 2024-04-12T04:39:53.703 DEBUG:boto:auth_path=/dkyges-70/ 2024-04-12T04:39:53.703 DEBUG:boto:Method: HEAD 2024-04-12T04:39:53.703 DEBUG:boto:Path: /dkyges-70/ 2024-04-12T04:39:53.703 DEBUG:boto:Data: 2024-04-12T04:39:53.703 DEBUG:boto:Headers: {} 2024-04-12T04:39:53.703 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:39:53.704 DEBUG:boto:Port: 8001 2024-04-12T04:39:53.704 DEBUG:boto:Params: {} 2024-04-12T04:39:53.704 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:39:53.704 DEBUG:boto:Token: None 2024-04-12T04:39:53.705 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:39:53 GMT /dkyges-70/ 2024-04-12T04:39:53.705 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BAcvh2gY5AVDCiWEHdBvI2TRAow= 2024-04-12T04:39:53.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:39:53 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BAcvh2gY5AVDCiWEHdBvI2TRAow=', 'Content-Length': '0'} 2024-04-12T04:39:53.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b229913ac56f045-006618bb19-25c8-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:39:53.720 DEBUG:boto:path=/dkyges-70/obj1 2024-04-12T04:39:53.720 DEBUG:boto:auth_path=/dkyges-70/obj1 2024-04-12T04:39:53.721 DEBUG:boto:Method: HEAD 2024-04-12T04:39:53.721 DEBUG:boto:Path: /dkyges-70/obj1 2024-04-12T04:39:53.721 DEBUG:boto:Data: 2024-04-12T04:39:53.721 DEBUG:boto:Headers: {} 2024-04-12T04:39:53.721 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:39:53.721 DEBUG:boto:Port: 8001 2024-04-12T04:39:53.721 DEBUG:boto:Params: {} 2024-04-12T04:39:53.721 DEBUG:boto:Token: None 2024-04-12T04:39:53.721 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:39:53 GMT /dkyges-70/obj1 2024-04-12T04:39:53.721 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:r0IKoFrysgB9TH3kEJeANdM87IU= 2024-04-12T04:39:53.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:39:53 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:r0IKoFrysgB9TH3kEJeANdM87IU=', 'Content-Length': '0'} 2024-04-12T04:39:53.724 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:37:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-70:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9370.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000568403f790322b61-006618bb19-25c8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:39:53.724 DEBUG:boto:path=/dkyges-70/obj1 2024-04-12T04:39:53.724 DEBUG:boto:auth_path=/dkyges-70/obj1 2024-04-12T04:39:53.724 DEBUG:boto:Method: GET 2024-04-12T04:39:53.724 DEBUG:boto:Path: /dkyges-70/obj1 2024-04-12T04:39:53.725 DEBUG:boto:Data: 2024-04-12T04:39:53.725 DEBUG:boto:Headers: {} 2024-04-12T04:39:53.725 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:39:53.725 DEBUG:boto:Port: 8001 2024-04-12T04:39:53.725 DEBUG:boto:Params: {} 2024-04-12T04:39:53.725 DEBUG:boto:Token: None 2024-04-12T04:39:53.725 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:39:53 GMT /dkyges-70/obj1 2024-04-12T04:39:53.725 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:HmmBRlnlJ6LnhWp7cx936z3TLc4= 2024-04-12T04:39:53.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:39:53 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HmmBRlnlJ6LnhWp7cx936z3TLc4=', 'Content-Length': '0'} 2024-04-12T04:39:53.727 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:37:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-70:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9370.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e99022aa456ac4d9-006618bb19-25c8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:39:53.727 DEBUG:boto:path=/dkyges-70/obj2 2024-04-12T04:39:53.728 DEBUG:boto:auth_path=/dkyges-70/obj2 2024-04-12T04:39:53.728 DEBUG:boto:Method: HEAD 2024-04-12T04:39:53.728 DEBUG:boto:Path: /dkyges-70/obj2 2024-04-12T04:39:53.728 DEBUG:boto:Data: 2024-04-12T04:39:53.728 DEBUG:boto:Headers: {} 2024-04-12T04:39:53.728 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:39:53.728 DEBUG:boto:Port: 8001 2024-04-12T04:39:53.728 DEBUG:boto:Params: {} 2024-04-12T04:39:53.728 DEBUG:boto:Token: None 2024-04-12T04:39:53.728 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:39:53 GMT /dkyges-70/obj2 2024-04-12T04:39:53.729 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BXjSAhMFdenIi31cbbKsl/iqjNQ= 2024-04-12T04:39:53.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:39:53 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BXjSAhMFdenIi31cbbKsl/iqjNQ=', 'Content-Length': '0'} 2024-04-12T04:39:53.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:37:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-70:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9370.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074ba31d585546830-006618bb19-25c8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:39:53.731 DEBUG:boto:path=/dkyges-70/obj2 2024-04-12T04:39:53.731 DEBUG:boto:auth_path=/dkyges-70/obj2 2024-04-12T04:39:53.731 DEBUG:boto:Method: GET 2024-04-12T04:39:53.731 DEBUG:boto:Path: /dkyges-70/obj2 2024-04-12T04:39:53.731 DEBUG:boto:Data: 2024-04-12T04:39:53.731 DEBUG:boto:Headers: {} 2024-04-12T04:39:53.731 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:39:53.731 DEBUG:boto:Port: 8001 2024-04-12T04:39:53.731 DEBUG:boto:Params: {} 2024-04-12T04:39:53.731 DEBUG:boto:Token: None 2024-04-12T04:39:53.732 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:39:53 GMT /dkyges-70/obj2 2024-04-12T04:39:53.732 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kVxWZYHeGUZTJeft12+EZgoZ8/U= 2024-04-12T04:39:53.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:39:53 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kVxWZYHeGUZTJeft12+EZgoZ8/U=', 'Content-Length': '0'} 2024-04-12T04:39:53.733 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:37:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-70:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9370.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c47833b74d38f1e9-006618bb19-25c8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:39:53.778 DEBUG:boto:path=/dkyges-70/obj3 2024-04-12T04:39:53.779 DEBUG:boto:auth_path=/dkyges-70/obj3 2024-04-12T04:39:53.779 DEBUG:boto:Method: HEAD 2024-04-12T04:39:53.779 DEBUG:boto:Path: /dkyges-70/obj3 2024-04-12T04:39:53.779 DEBUG:boto:Data: 2024-04-12T04:39:53.779 DEBUG:boto:Headers: {} 2024-04-12T04:39:53.779 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:39:53.779 DEBUG:boto:Port: 8001 2024-04-12T04:39:53.779 DEBUG:boto:Params: {} 2024-04-12T04:39:53.779 DEBUG:boto:Token: None 2024-04-12T04:39:53.779 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:39:53 GMT /dkyges-70/obj3 2024-04-12T04:39:53.780 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ts0wbbLbI0rObOhe3rWS1qB8QZ0= 2024-04-12T04:39:53.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:39:53 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ts0wbbLbI0rObOhe3rWS1qB8QZ0=', 'Content-Length': '0'} 2024-04-12T04:39:53.782 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:37:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-70:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9370.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030e461a4fffc5e24-006618bb19-25c8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:39:53.782 DEBUG:boto:path=/dkyges-70/obj3 2024-04-12T04:39:53.782 DEBUG:boto:auth_path=/dkyges-70/obj3 2024-04-12T04:39:53.782 DEBUG:boto:Method: GET 2024-04-12T04:39:53.782 DEBUG:boto:Path: /dkyges-70/obj3 2024-04-12T04:39:53.782 DEBUG:boto:Data: 2024-04-12T04:39:53.782 DEBUG:boto:Headers: {} 2024-04-12T04:39:53.782 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:39:53.783 DEBUG:boto:Port: 8001 2024-04-12T04:39:53.783 DEBUG:boto:Params: {} 2024-04-12T04:39:53.783 DEBUG:boto:Token: None 2024-04-12T04:39:53.783 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:39:53 GMT /dkyges-70/obj3 2024-04-12T04:39:53.783 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:spuEf8IrbLbqSj47NId0MCh/Kn8= 2024-04-12T04:39:53.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:39:53 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:spuEf8IrbLbqSj47NId0MCh/Kn8=', 'Content-Length': '0'} 2024-04-12T04:39:53.785 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:37:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-70:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9370.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d362221bd1d59968-006618bb19-25c8-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:39:53.785 DEBUG:boto:path=/dkyges-71/ 2024-04-12T04:39:53.785 DEBUG:boto:auth_path=/dkyges-71/ 2024-04-12T04:39:53.785 DEBUG:boto:Method: HEAD 2024-04-12T04:39:53.785 DEBUG:boto:Path: /dkyges-71/ 2024-04-12T04:39:53.786 DEBUG:boto:Data: 2024-04-12T04:39:53.786 DEBUG:boto:Headers: {} 2024-04-12T04:39:53.786 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:39:53.786 DEBUG:boto:Port: 8001 2024-04-12T04:39:53.786 DEBUG:boto:Params: {} 2024-04-12T04:39:53.786 DEBUG:boto:Token: None 2024-04-12T04:39:53.786 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:39:53 GMT /dkyges-71/ 2024-04-12T04:39:53.786 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:a041Bn7NPaxwIbtlqAGsGMQwqkU= 2024-04-12T04:39:53.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:39:53 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a041Bn7NPaxwIbtlqAGsGMQwqkU=', 'Content-Length': '0'} 2024-04-12T04:39:53.789 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d661395fb882da2-006618bb19-25c8-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:39:53.789 DEBUG:boto:path=/dkyges-71/obj1 2024-04-12T04:39:53.790 DEBUG:boto:auth_path=/dkyges-71/obj1 2024-04-12T04:39:53.790 DEBUG:boto:Method: HEAD 2024-04-12T04:39:53.790 DEBUG:boto:Path: /dkyges-71/obj1 2024-04-12T04:39:53.790 DEBUG:boto:Data: 2024-04-12T04:39:53.790 DEBUG:boto:Headers: {} 2024-04-12T04:39:53.790 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:39:53.790 DEBUG:boto:Port: 8001 2024-04-12T04:39:53.790 DEBUG:boto:Params: {} 2024-04-12T04:39:53.790 DEBUG:boto:Token: None 2024-04-12T04:39:53.790 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:39:53 GMT /dkyges-71/obj1 2024-04-12T04:39:53.791 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nTfBgc1tj8K2AlGBZsW1rDvPJDE= 2024-04-12T04:39:53.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:39:53 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nTfBgc1tj8K2AlGBZsW1rDvPJDE=', 'Content-Length': '0'} 2024-04-12T04:39:53.792 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000003479fa999682882-006618bb19-25c8-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:39:53.792 DEBUG:boto:path=/dkyges-71/obj2 2024-04-12T04:39:53.792 DEBUG:boto:auth_path=/dkyges-71/obj2 2024-04-12T04:39:53.792 DEBUG:boto:Method: HEAD 2024-04-12T04:39:53.792 DEBUG:boto:Path: /dkyges-71/obj2 2024-04-12T04:39:53.793 DEBUG:boto:Data: 2024-04-12T04:39:53.793 DEBUG:boto:Headers: {} 2024-04-12T04:39:53.793 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:39:53.793 DEBUG:boto:Port: 8001 2024-04-12T04:39:53.793 DEBUG:boto:Params: {} 2024-04-12T04:39:53.793 DEBUG:boto:Token: None 2024-04-12T04:39:53.793 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:39:53 GMT /dkyges-71/obj2 2024-04-12T04:39:53.793 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:bW0logSI3PDqmEObiLRV1bE8xdE= 2024-04-12T04:39:53.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:39:53 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bW0logSI3PDqmEObiLRV1bE8xdE=', 'Content-Length': '0'} 2024-04-12T04:39:53.795 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000681d110bf821c481-006618bb19-25c8-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:39:53.795 DEBUG:boto:path=/dkyges-71/obj3 2024-04-12T04:39:53.795 DEBUG:boto:auth_path=/dkyges-71/obj3 2024-04-12T04:39:53.795 DEBUG:boto:Method: HEAD 2024-04-12T04:39:53.795 DEBUG:boto:Path: /dkyges-71/obj3 2024-04-12T04:39:53.795 DEBUG:boto:Data: 2024-04-12T04:39:53.796 DEBUG:boto:Headers: {} 2024-04-12T04:39:53.796 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:39:53.796 DEBUG:boto:Port: 8001 2024-04-12T04:39:53.796 DEBUG:boto:Params: {} 2024-04-12T04:39:53.796 DEBUG:boto:Token: None 2024-04-12T04:39:53.796 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:39:53 GMT /dkyges-71/obj3 2024-04-12T04:39:53.796 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+fD/UsRCrXOhwGuAu9E6LTmA3Do= 2024-04-12T04:39:53.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:39:53 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+fD/UsRCrXOhwGuAu9E6LTmA3Do=', 'Content-Length': '0'} 2024-04-12T04:39:53.798 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000009219d0dd715a19ee-006618bb19-25c8-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:39:53 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:39:53.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:53.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:53.798 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:39:53.859 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:53.859 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:53.918 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:39:53.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:53.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:53.919 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:39:53.977 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:53.977 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:54.032 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:39:54.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:54.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:54.033 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:39:54.088 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:54.088 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:54.144 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:39:54.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:54.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:54.145 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:39:54.198 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:54.198 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:54.260 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:39:54.260 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:39:54.260 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 17, 2024-04-12T04:39:54.260 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T04:39:54.260 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T04:39:54.260 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T04:39:54.260 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:39:54.260 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T04:39:54.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:54.261 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T04:39:54.261 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T04:39:54.261 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:39:54.261 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T04:39:54.261 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:39:54.261 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:39:54.261 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:39:54.261 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:39:54.261 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:39:54.261 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:39:54.261 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:39:54.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:54.262 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:39:54.262 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T04:39:54.262 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:39:54.262 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T04:39:54.262 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:39:54.262 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:39:54.262 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:39:54.262 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:39:54.262 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:39:54.262 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:39:54.263 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:39:54.263 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:39:54.263 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:39:54.263 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:39:54.263 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:39:54.263 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:39:54.263 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:54.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:54.263 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:39:54.263 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T04:39:54.263 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:39:54.264 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:39:54.264 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:39:54.264 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:39:54.264 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:39:54.264 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:39:54.264 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:39:54.264 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:39:54.264 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:39:54.264 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:39:54.264 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:39:54.264 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:39:54.264 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:39:54.265 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:39:54.265 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:54.265 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:54.265 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:39:54.265 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:39:54.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:54.265 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:39:54.265 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:39:54.265 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:39:54.265 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:39:54.265 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:54.265 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:54.266 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:39:54.266 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:39:54.266 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:39:54.266 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:39:54.266 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [ 2024-04-12T04:39:54.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:54.266 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-group", 2024-04-12T04:39:54.266 INFO:teuthology.orchestra.run.smithi043.stdout: "data_flow": { 2024-04-12T04:39:54.266 INFO:teuthology.orchestra.run.smithi043.stdout: "symmetrical": [ 2024-04-12T04:39:54.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:54.266 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-flow1", 2024-04-12T04:39:54.267 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:39:54.267 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:39:54.267 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:39:54.267 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:54.267 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:54.267 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:54.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.267 INFO:teuthology.orchestra.run.smithi043.stdout: "pipes": [ 2024-04-12T04:39:54.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:54.267 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-pipe", 2024-04-12T04:39:54.267 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:39:54.267 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "*", 2024-04-12T04:39:54.267 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:39:54.268 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:39:54.268 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:39:54.268 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:54.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.268 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": { 2024-04-12T04:39:54.268 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "*", 2024-04-12T04:39:54.268 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:39:54.268 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:39:54.268 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:39:54.268 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:54.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.268 INFO:teuthology.orchestra.run.smithi043.stdout: "params": { 2024-04-12T04:39:54.269 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:39:54.269 INFO:teuthology.orchestra.run.smithi043.stdout: "filter": { 2024-04-12T04:39:54.269 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [] 2024-04-12T04:39:54.269 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:54.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.269 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": {}, 2024-04-12T04:39:54.269 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-12T04:39:54.269 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "system", 2024-04-12T04:39:54.269 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "" 2024-04-12T04:39:54.269 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:54.269 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:54.269 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:39:54.269 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "allowed" 2024-04-12T04:39:54.270 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:54.270 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:54.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.270 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:39:54.270 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:39:54.270 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:54.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:54.270 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T04:39:54.270 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T04:39:54.270 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:39:54.271 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T04:39:54.271 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:39:54.271 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:39:54.271 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:39:54.271 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:39:54.271 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:39:54.271 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:39:54.271 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:39:54.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:54.271 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:39:54.271 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T04:39:54.271 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:39:54.272 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:39:54.272 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:39:54.272 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:39:54.272 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:39:54.272 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:39:54.272 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:39:54.272 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:39:54.272 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:39:54.272 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:39:54.272 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:39:54.272 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:39:54.272 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:39:54.272 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:39:54.273 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:54.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:54.273 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:39:54.273 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T04:39:54.273 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:39:54.273 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T04:39:54.273 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:39:54.273 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:39:54.273 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:39:54.273 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:39:54.273 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:39:54.273 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:39:54.274 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:39:54.274 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:39:54.274 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:39:54.274 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:39:54.274 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:39:54.274 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:39:54.274 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:54.274 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:54.274 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:39:54.274 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:39:54.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:54.275 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:39:54.275 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:39:54.275 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:39:54.275 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:39:54.275 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:54.275 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:54.275 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:39:54.275 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:39:54.275 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:39:54.275 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:39:54.276 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T04:39:54.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.276 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:39:54.276 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:39:54.276 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:54.276 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:54.276 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:39:54.276 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T04:39:54.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:54.276 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:39:54.276 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T04:39:54.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:54.277 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:39:54.277 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T04:39:54.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:54.277 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:39:54.277 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T04:39:54.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:54.278 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:39:54.278 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T04:39:54.278 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:54.278 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:54.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.278 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T04:39:54.278 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:39:54.278 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T04:39:54.278 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T04:39:54.278 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:39:54.278 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:39:54.278 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:39:54.279 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:39:54.279 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:39:54.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.279 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T04:39:54.279 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:39:54.279 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:39:54.279 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:39:54.279 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:39:54.279 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:39:54.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.279 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T04:39:54.279 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:39:54.280 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:39:54.280 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:39:54.280 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:39:54.280 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:39:54.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.280 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T04:39:54.280 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:39:54.280 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:39:54.280 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:39:54.280 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:39:54.280 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:39:54.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.280 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T04:39:54.281 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:39:54.281 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:39:54.281 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:39:54.281 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:39:54.281 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:39:54.281 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:54.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:54.281 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:39:54.281 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T04:39:54.281 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:39:54.281 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:39:54.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:54.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:54.283 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:39:54.342 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:54.343 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:54.442 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:39:54.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dkyges-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:54.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dkyges-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:54.443 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dkyges-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:39:54.518 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:54.518 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:54.655 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:39:54.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dkyges-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:54.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dkyges-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:54.656 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket dkyges-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:39:54.744 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:54.744 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:54.874 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:39:54.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dkyges-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:54.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dkyges-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:54.875 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=dkyges-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:39:54.960 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:54.960 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:55.059 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-12T04:39:55.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dkyges-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:55.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dkyges-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:55.060 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dkyges-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:39:55.135 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:55.135 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:55.210 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:39:55.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dkyges-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:55.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dkyges-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:39:55.211 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dkyges-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:39:55.268 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:55.269 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:55.328 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:39:55.328 DEBUG:boto:path=/dkyges-70/ 2024-04-12T04:39:55.329 DEBUG:boto:auth_path=/dkyges-70/ 2024-04-12T04:39:55.329 DEBUG:boto:Method: HEAD 2024-04-12T04:39:55.329 DEBUG:boto:Path: /dkyges-70/ 2024-04-12T04:39:55.329 DEBUG:boto:Data: 2024-04-12T04:39:55.329 DEBUG:boto:Headers: {} 2024-04-12T04:39:55.329 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:39:55.329 DEBUG:boto:Port: 8000 2024-04-12T04:39:55.329 DEBUG:boto:Params: {} 2024-04-12T04:39:55.329 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:39:55.330 DEBUG:boto:Token: None 2024-04-12T04:39:55.330 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:39:55 GMT /dkyges-70/ 2024-04-12T04:39:55.330 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JSEzcAYMVaH3sCmIR/ne7g5Egxo= 2024-04-12T04:39:55.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:39:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JSEzcAYMVaH3sCmIR/ne7g5Egxo=', 'Content-Length': '0'} 2024-04-12T04:39:55.346 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002fe7a3ebe1d7daa7-006618bb1b-26d1-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:39:55.346 DEBUG:boto:path=/dkyges-70/obj3 2024-04-12T04:39:55.346 DEBUG:boto:auth_path=/dkyges-70/obj3 2024-04-12T04:39:55.347 DEBUG:boto:Method: PUT 2024-04-12T04:39:55.347 DEBUG:boto:Path: /dkyges-70/obj3 2024-04-12T04:39:55.347 DEBUG:boto:Data: 2024-04-12T04:39:55.347 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:39:55.347 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:39:55.347 DEBUG:boto:Port: 8000 2024-04-12T04:39:55.347 DEBUG:boto:Params: {} 2024-04-12T04:39:55.347 DEBUG:boto:Token: None 2024-04-12T04:39:55.347 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:39:55 GMT /dkyges-70/obj3 2024-04-12T04:39:55.347 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kbDGmg0ecE/0F2kq9b6so0G5J1o= 2024-04-12T04:39:55.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:39:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kbDGmg0ecE/0F2kq9b6so0G5J1o='} 2024-04-12T04:39:55.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015cd31664d981324-006618bb1b-26d1-a1'), ('Date', 'Fri, 12 Apr 2024 04:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:39:55.359 DEBUG:boto:path=/dkyges-71/ 2024-04-12T04:39:55.359 DEBUG:boto:auth_path=/dkyges-71/ 2024-04-12T04:39:55.359 DEBUG:boto:Method: HEAD 2024-04-12T04:39:55.359 DEBUG:boto:Path: /dkyges-71/ 2024-04-12T04:39:55.359 DEBUG:boto:Data: 2024-04-12T04:39:55.359 DEBUG:boto:Headers: {} 2024-04-12T04:39:55.359 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:39:55.360 DEBUG:boto:Port: 8000 2024-04-12T04:39:55.360 DEBUG:boto:Params: {} 2024-04-12T04:39:55.360 DEBUG:boto:Token: None 2024-04-12T04:39:55.360 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:39:55 GMT /dkyges-71/ 2024-04-12T04:39:55.360 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:roTtKoCfiId3JLf5KnClGoB+BP8= 2024-04-12T04:39:55.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:39:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:roTtKoCfiId3JLf5KnClGoB+BP8=', 'Content-Length': '0'} 2024-04-12T04:39:55.364 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035601ad51b837a9d-006618bb1b-26d1-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:39:55.364 DEBUG:boto:path=/dkyges-71/obj3 2024-04-12T04:39:55.364 DEBUG:boto:auth_path=/dkyges-71/obj3 2024-04-12T04:39:55.364 DEBUG:boto:Method: PUT 2024-04-12T04:39:55.364 DEBUG:boto:Path: /dkyges-71/obj3 2024-04-12T04:39:55.364 DEBUG:boto:Data: 2024-04-12T04:39:55.364 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:39:55.364 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:39:55.365 DEBUG:boto:Port: 8000 2024-04-12T04:39:55.365 DEBUG:boto:Params: {} 2024-04-12T04:39:55.365 DEBUG:boto:Token: None 2024-04-12T04:39:55.365 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:39:55 GMT /dkyges-71/obj3 2024-04-12T04:39:55.365 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BZGYzuHn0IVO49Z/WM+QeiQrc78= 2024-04-12T04:39:55.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:39:55 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BZGYzuHn0IVO49Z/WM+QeiQrc78='} 2024-04-12T04:39:55.415 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e90548bf0fcd7f0-006618bb1b-26d1-a1'), ('Date', 'Fri, 12 Apr 2024 04:39:55 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:39:55.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:55.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:55.415 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:39:55.473 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:55.473 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:39:55.529 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:39:55.529 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:55.529 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:39:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:39:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:39:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:39:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:39:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:39:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:55.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:55.531 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896795.189600_921.1", 2024-04-12T04:39:55.531 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:39:55.189600Z" 2024-04-12T04:39:55.531 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:55.531 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:39:55.532 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:39:55.532 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:39:55.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:55.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:39:55.532 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:39:55.588 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:39:55.588 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:39:55.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:39:55.624+0000 7f1e8822a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:39:55.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:39:55.625+0000 7f1e8822a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:39:55.642 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:39:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:39:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:39:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:39:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:39:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:39:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:39:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:55.642 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:39:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:39:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:39:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:39:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:39:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:39:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:39:55.643 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:39:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:39:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:39:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:39:55.644 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:39:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:39:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:39:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:39:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:55.645 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:39:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:39:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:39:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:39:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:39:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:39:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:39:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:39:55.646 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:39:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:39:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:39:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:39:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:39:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:39:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:39:55.647 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:39:55.648 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:39:55.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:39:55.648 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:39:55Z" 2024-04-12T04:39:55.648 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:39:55.649 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:39:55Z" } 2024-04-12T04:39:55.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:39:55.649 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:39:55.649 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:39:55.649 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:40:00.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:00.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:00.651 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:40:00.710 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:40:00.710 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:40:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:00.747+0000 7f221d298780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:00.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:00.748+0000 7f221d298780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:00.765 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:40:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:40:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:40:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:40:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:40:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:40:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:40:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:40:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:40:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:40:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:40:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:40:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:40:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:40:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:40:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:40:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:40:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:40:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:40:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:40:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:40:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:40:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:40:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:40:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:40:00Z" 2024-04-12T04:40:00.770 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:40:00.771 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:40:00Z" } 2024-04-12T04:40:00.771 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:40:00.771 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:40:00.772 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:40:00.772 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:40:05.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:05.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:05.773 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:40:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:40:05.832 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:40:05.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:05.868+0000 7fac9cdef780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:05.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:05.869+0000 7fac9cdef780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:05.886 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:40:05.886 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:40:05.886 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:40:05.886 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:40:05.886 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:40:05.887 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:40:05.887 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:05.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:05.887 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:40:05.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:05.887 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:40:05.887 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:05.887 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:05.887 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:40:05.887 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:05.887 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:40:05.888 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:05.888 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:40:05.888 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:05.888 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:05.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:05.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:05.888 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:40:05.888 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:05.888 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:05.888 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:40:05.888 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:05.889 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:05.889 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:05.889 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:40:05.889 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:05.889 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:05.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:05.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:05.889 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:40:05.889 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:05.889 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:05.889 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:40:05.889 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:05.890 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:05.890 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:05.890 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:40:05.890 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:05.890 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:05.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:05.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:05.890 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:40:05.890 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:05.890 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:05.890 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:40:05.891 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:05.891 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:05.891 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:05.891 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:40:05.891 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:05.891 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:05.891 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:05.891 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:40:05.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:05.891 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:40:05.891 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:40:05.891 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:40:05.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:05.892 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:40:05Z" 2024-04-12T04:40:05.892 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:40:05.893 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:40:05Z" } 2024-04-12T04:40:05.893 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:40:05.893 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:40:05.893 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:40:05.893 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:40:10.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:10.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:10.895 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:40:10.954 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:40:10.954 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:40:10.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:10.991+0000 7efbff1ea780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:10.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:10.992+0000 7efbff1ea780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:11.009 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:40:11.009 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:40:11.009 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:40:11.009 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:40:11.009 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:40:11.009 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:40:11.009 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:11.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:11.009 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:40:11.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:11.010 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:40:11.010 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:11.010 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:11.010 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:40:11.010 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:11.010 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:40:11.010 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:11.010 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:40:11.010 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:11.010 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:11.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:11.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:11.011 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:40:11.011 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:11.011 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:11.011 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:40:11.011 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:11.011 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:11.011 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:11.011 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:40:11.011 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:11.012 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:11.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:11.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:11.012 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:40:11.012 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:11.012 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:11.012 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:40:11.012 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:11.012 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:11.012 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:11.013 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:40:11.013 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:11.013 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:11.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:11.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:11.013 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:40:11.013 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:11.013 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:11.013 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:40:11.013 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:11.013 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:11.013 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:11.014 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:40:11.014 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:11.014 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:11.014 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:11.014 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:40:11.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:11.014 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:40:11.014 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:40:11.014 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:40:11.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:11.015 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:40:10Z" 2024-04-12T04:40:11.015 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:40:11.016 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:40:10Z" } 2024-04-12T04:40:11.016 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:40:11.016 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:40:11.016 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:40:11.016 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:40:16.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:16.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:16.018 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:40:16.077 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:40:16.077 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:40:16.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:16.114+0000 7f33939a1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:16.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:16.115+0000 7f33939a1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:16.132 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:40:16.132 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:40:16.132 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:40:16.132 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:40:16.133 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:40:16.133 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:40:16.133 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:16.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:16.133 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:40:16.133 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:16.133 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:40:16.133 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:16.133 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:16.133 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:40:16.133 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:16.133 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:40:16.134 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:16.134 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:40:16.134 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:16.134 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:16.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:16.134 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:16.134 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:40:16.134 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:16.134 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:16.134 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:40:16.135 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:16.135 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:16.135 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:16.135 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:40:16.135 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:16.135 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:16.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:16.135 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:16.135 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:40:16.135 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:16.135 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:16.136 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:40:16.136 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:16.136 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:16.136 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:16.136 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:40:16.136 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:16.136 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:16.136 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:16.136 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:16.136 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:40:16.136 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:16.136 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:16.137 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:40:16.137 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:16.137 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:16.137 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:16.137 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:40:16.137 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:16.137 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:16.137 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:16.137 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:40:16.137 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:16.137 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:40:16.138 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:40:16.138 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:40:16.138 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:16.138 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:40:16Z" 2024-04-12T04:40:16.138 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:40:16.139 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:40:16Z" } 2024-04-12T04:40:16.139 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:40:16.139 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:40:16.139 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:40:16.139 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:40:21.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:21.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:21.141 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:40:21.202 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:40:21.203 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:40:21.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:21.242+0000 7ff8c0879780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:21.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:21.244+0000 7ff8c0879780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:21.262 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:40:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:40:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:40:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:40:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:40:21.262 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:40:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:40:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:40:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:40:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:21.263 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:40:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:40:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:40:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:40:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:21.264 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:40:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:40:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:21.265 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:40:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:40:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:21.266 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:40:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:40:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:40:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:21.267 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:40:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:40:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:40:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:40:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:21.268 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:40:21Z" 2024-04-12T04:40:21.268 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:40:21.269 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:40:21Z" } 2024-04-12T04:40:21.269 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:40:21.269 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:40:21.270 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:40:21.270 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:40:26.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:26.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:26.271 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:40:26.330 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:40:26.330 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:40:26.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:26.367+0000 7f079adf1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:26.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:26.368+0000 7f079adf1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:26.385 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:40:26.385 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:40:26.385 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:40:26.385 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:40:26.385 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:40:26.385 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:40:26.385 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:26.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:26.386 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:40:26.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:26.386 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:40:26.386 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:26.386 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:26.386 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:40:26.386 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:26.386 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:40:26.386 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:26.386 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:40:26.386 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:26.387 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:26.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:26.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:26.387 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:40:26.387 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:26.387 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:26.387 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:40:26.387 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:26.387 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:26.387 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:26.387 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:40:26.387 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:26.388 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:26.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:26.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:26.388 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:40:26.388 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:26.388 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:26.388 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:40:26.388 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:26.388 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:26.388 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:26.388 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:40:26.388 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:26.388 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:26.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:26.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:26.389 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:40:26.389 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:26.389 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:26.389 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:40:26.389 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:26.389 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:26.389 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:26.389 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:40:26.389 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:26.389 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:26.390 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:26.390 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:40:26.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:26.390 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:40:26.390 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:40:26.390 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:40:26.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:26.390 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:40:26Z" 2024-04-12T04:40:26.390 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:40:26.391 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:40:26Z" } 2024-04-12T04:40:26.391 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:40:26.392 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:40:26.392 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:40:26.392 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:40:31.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:31.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:31.393 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:40:31.452 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:40:31.452 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:40:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:31.487+0000 7f2a35ba4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:31.489+0000 7f2a35ba4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:31.505 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:40:31.505 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:40:31.505 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:40:31.505 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:40:31.505 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:40:31.506 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:40:31.506 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:31.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:31.506 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:40:31.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:31.506 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:40:31.506 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:31.506 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:31.506 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:40:31.506 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:31.506 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:40:31.507 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:31.507 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:40:31.507 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:31.507 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:31.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:31.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:31.507 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:40:31.507 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:31.507 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:31.507 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:40:31.507 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:31.508 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:31.508 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:31.508 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:40:31.508 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:31.508 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:31.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:31.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:31.508 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:40:31.508 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:31.508 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:31.508 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:40:31.509 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:31.509 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:31.509 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:31.509 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:40:31.509 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:31.509 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:31.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:31.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:31.509 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:40:31.509 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:31.509 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:40:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:40:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:40:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:31.510 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:40:31.511 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:40:31.511 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:40:31.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:31.511 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:40:31Z" 2024-04-12T04:40:31.511 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:40:31.512 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:40:31Z" } 2024-04-12T04:40:31.512 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:40:31.512 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:40:31.512 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:40:31.512 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:40:36.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:36.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:36.514 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:40:36.572 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:40:36.573 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:40:36.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:36.609+0000 7fc5dc022780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:36.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:36.610+0000 7fc5dc022780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:36.627 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:40:36.627 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:40:36.627 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:40:36.627 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:40:36.627 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:40:36.627 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:40:36.627 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:36.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:36.628 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:40:36.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:36.628 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:40:36.628 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:36.628 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:36.628 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:40:36.628 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:36.629 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:40:36.629 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:36.629 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:40:36.629 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:36.629 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:36.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:36.629 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:36.629 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:40:36.629 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:36.629 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:36.629 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:40:36.629 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:36.630 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:36.630 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:36.630 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:40:36.630 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:36.630 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:36.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:36.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:36.630 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:40:36.630 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:36.631 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:36.631 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:40:36.631 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:36.631 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:36.631 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:36.631 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:40:36.631 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:36.631 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:36.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:36.631 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:36.631 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:40:36.632 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:36.632 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:36.632 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:40:36.632 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:36.632 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:36.632 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:36.632 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:40:36.632 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:36.632 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:36.632 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:36.632 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:40:36.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:36.633 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:40:36.633 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:40:36.633 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:40:36.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:36.633 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:40:36Z" 2024-04-12T04:40:36.633 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:40:36.634 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:40:36Z" } 2024-04-12T04:40:36.635 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:40:36.635 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:40:36.635 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:40:36.635 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:40:41.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:41.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:41.637 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:40:41.697 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:40:41.697 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:40:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:41.734+0000 7f9807342780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:41.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:41.735+0000 7f9807342780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:41.751 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:40:41.752 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:40:41.752 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:40:41.752 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:40:41.752 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:40:41.752 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:40:41.752 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:41.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:41.752 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:40:41.752 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:41.752 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:40:41.753 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:41.753 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:41.753 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:40:41.753 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:41.753 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:40:41.753 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:41.753 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:40:41.753 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:41.753 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:41.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:41.753 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:40:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:40:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:40:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:40:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:40:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:40:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:40:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:40:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:40:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:40:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:40:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:40:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:40:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:40:41Z" 2024-04-12T04:40:41.757 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:40:41.758 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:40:41Z" } 2024-04-12T04:40:41.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:40:41.759 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:40:41.759 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:40:41.759 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:40:46.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:46.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:46.760 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:40:46.819 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:40:46.820 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:40:46.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:46.856+0000 7f78903d3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:46.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:46.857+0000 7f78903d3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:46.874 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:40:46.874 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:40:46.874 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:40:46.874 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:40:46.875 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:40:46.875 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:40:46.875 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:46.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:46.875 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:40:46.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:46.875 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:40:46.875 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:46.875 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:46.875 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:40:46.875 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:46.875 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:40:46.876 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:46.876 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:40:46.876 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:46.876 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:46.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:46.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:46.876 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:40:46.876 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:46.876 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:46.876 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:40:46.876 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:46.876 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:46.877 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:46.877 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:40:46.877 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:46.877 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:46.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:46.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:46.877 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:40:46.877 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:46.877 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:46.877 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:40:46.877 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:46.877 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:46.878 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:46.878 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:40:46.878 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:46.878 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:46.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:46.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:46.878 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:40:46.878 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:46.878 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:46.878 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:40:46.878 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:46.879 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:46.879 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:46.879 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:40:46.879 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:46.879 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:46.879 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:46.879 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:40:46.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:46.879 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:40:46.879 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:40:46.879 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:40:46.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:46.880 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:40:46Z" 2024-04-12T04:40:46.880 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:40:46.882 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:40:46Z" } 2024-04-12T04:40:46.883 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:40:46.883 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:40:46.883 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:40:46.883 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:40:51.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:51.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:51.884 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:40:51.944 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:40:51.944 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:40:51.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:51.980+0000 7f4c9169d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:51.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:51.982+0000 7f4c9169d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:51.999 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:40:51.999 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:40:51.999 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:40:51.999 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:40:51.999 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:40:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:40:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:40:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:40:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:40:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:52.000 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:40:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:40:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:40:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:40:52.001 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:40:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:40:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:52.002 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:40:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:40:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:40:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:52.003 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:40:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:40:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:40:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:52.004 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:40:52.005 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:40:52.005 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:40:52.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:52.005 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:40:51Z" 2024-04-12T04:40:52.005 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:40:52.006 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:40:51Z" } 2024-04-12T04:40:52.006 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:40:52.006 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:40:52.006 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:40:52.006 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:40:57.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:57.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:40:57.008 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:40:57.068 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:40:57.068 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:40:57.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:57.105+0000 7f02e081a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:57.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:40:57.106+0000 7f02e081a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:40:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:40:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:40:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:40:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:40:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:40:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:40:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:40:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:40:57Z" 2024-04-12T04:40:57.133 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:40:57.134 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:40:57Z" } 2024-04-12T04:40:57.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:40:57.134 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:40:57.134 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:40:57.135 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:41:02.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:02.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:02.136 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:41:02.196 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:02.196 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:02.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:02.236+0000 7f6a62f01780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:02.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:02.237+0000 7f6a62f01780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:02.254 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:41:02.254 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:41:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:41:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:41:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:41:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:41:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:41:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:41:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:02.255 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:41:02.256 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:02.256 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:41:02.256 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:02.256 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:41:02.256 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:02.256 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:02.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:02.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:02.256 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:41:02.256 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:02.256 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:02.256 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:41:02.257 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:02.257 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:02.257 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:02.257 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:41:02.257 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:02.257 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:02.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:02.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:02.257 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:41:02.257 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:02.257 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:02.257 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:41:02.258 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:02.258 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:02.258 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:02.258 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:41:02.258 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:02.258 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:02.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:02.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:02.258 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:41:02.258 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:02.258 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:02.259 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:41:02.259 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:02.259 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:02.259 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:02.259 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:41:02.259 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:02.259 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:02.259 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:02.259 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:41:02.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:02.259 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:41:02.259 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:41:02.260 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:41:02.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:02.260 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:41:02Z" 2024-04-12T04:41:02.260 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:41:02.261 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:41:02Z" } 2024-04-12T04:41:02.261 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:41:02.261 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:41:02.261 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:41:02.261 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:41:07.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:07.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:07.263 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:41:07.325 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:07.325 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:07.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:07.361+0000 7fcd1d5e0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:07.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:07.362+0000 7fcd1d5e0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:07.379 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:41:07.380 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:41:07.380 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:41:07.380 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:41:07.380 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:41:07.380 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:41:07.380 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:07.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:07.380 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:41:07.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:07.381 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:41:07.381 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:07.381 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:07.381 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:41:07.381 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:07.381 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:41:07.381 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:07.382 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:41:07.382 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:07.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:07.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:07.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:07.383 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:41:07.383 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:07.383 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:07.383 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:41:07.383 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:07.383 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:07.384 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:07.384 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:41:07.384 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:07.384 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:07.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:07.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:07.384 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:41:07.385 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:07.385 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:07.385 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:41:07.385 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:07.385 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:07.385 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:07.385 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:41:07.385 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:07.385 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:07.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:07.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:07.386 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:41:07.386 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:07.386 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:07.386 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:41:07.387 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:07.387 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:07.387 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:07.387 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:41:07.387 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:07.387 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:07.387 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:07.388 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:41:07.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:07.388 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:41:07.388 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:41:07.388 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:41:07.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:07.388 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:41:07Z" 2024-04-12T04:41:07.388 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:41:07.389 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:41:07Z" } 2024-04-12T04:41:07.389 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:41:07.390 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:41:07.390 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:41:07.390 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:41:12.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:12.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:12.392 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:41:12.451 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:12.451 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:12.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:12.488+0000 7fe72082e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:12.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:12.489+0000 7fe72082e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:12.505 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:41:12.505 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:41:12.505 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:41:12.505 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:41:12.505 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:41:12.506 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:41:12.506 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:12.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:12.506 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:41:12.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:12.506 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:41:12.506 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:12.506 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:12.507 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:41:12.507 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:12.507 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:41:12.507 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:12.507 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:41:12.507 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:12.507 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:12.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:12.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:12.507 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:41:12.508 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:12.508 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:12.508 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:41:12.508 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:12.508 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:12.508 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:12.508 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:41:12.508 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:12.508 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:12.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:12.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:12.508 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:41:12.509 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:12.509 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:12.509 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:41:12.509 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:12.509 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:12.509 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:12.509 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:41:12.509 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:12.509 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:12.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:12.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:12.510 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:41:12.510 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:12.510 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:12.510 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:41:12.510 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:12.510 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:12.510 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:12.510 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:41:12.510 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:12.511 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:12.511 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:12.511 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:41:12.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:12.511 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:41:12.511 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:41:12.511 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:41:12.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:12.511 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:41:12Z" 2024-04-12T04:41:12.511 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:41:12.512 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:41:12Z" } 2024-04-12T04:41:12.513 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:41:12.513 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:41:12.513 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:41:12.513 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:41:17.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:17.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:17.514 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:41:17.573 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:17.574 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:17.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:17.609+0000 7f4b30816780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:17.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:17.610+0000 7f4b30816780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:17.627 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:41:17.627 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:41:17.627 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:41:17.627 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:41:17.627 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:41:17.627 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:41:17.627 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:17.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:17.628 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:41:17.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:17.628 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:41:17.628 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:17.628 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:17.628 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:41:17.628 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:17.628 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:41:17.628 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:17.628 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:41:17.628 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:17.629 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:17.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:17.629 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:17.629 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:41:17.629 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:17.629 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:17.629 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:41:17.629 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:17.629 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:17.629 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:17.629 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:41:17.630 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:17.630 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:17.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:17.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:17.630 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:41:17.630 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:17.630 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:17.630 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:41:17.630 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:17.630 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:17.630 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:17.631 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:41:17.631 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:17.631 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:17.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:17.631 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:17.631 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:41:17.631 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:17.631 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:17.631 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:41:17.631 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:17.631 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:17.631 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:17.632 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:41:17.632 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:17.632 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:17.632 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:17.632 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:41:17.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:17.632 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:41:17.632 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:41:17.632 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:41:17.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:17.632 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:41:17Z" 2024-04-12T04:41:17.632 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:41:17.633 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:41:17Z" } 2024-04-12T04:41:17.633 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:41:17.634 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:41:17.634 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:41:17.634 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:41:22.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:22.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:22.635 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:41:22.694 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:22.695 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:22.731+0000 7f4618632780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:22.732+0000 7f4618632780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:22.749 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:41:22.750 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:41:22.750 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:41:22.750 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:41:22.750 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:41:22.750 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:41:22.750 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:22.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:22.750 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:41:22.750 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:22.750 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:41:22.750 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:22.751 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:22.751 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:41:22.751 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:22.751 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:41:22.751 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:22.751 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:41:22.751 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:22.751 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:22.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:22.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:22.751 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:41:22.751 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:22.751 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:22.752 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:41:22.752 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:22.752 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:22.752 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:22.752 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:41:22.752 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:22.752 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:22.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:22.752 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:22.752 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:41:22.752 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:22.752 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:22.752 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:41:22.753 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:22.753 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:22.753 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:22.753 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:41:22.753 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:22.753 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:22.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:22.753 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:22.753 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:41:22.753 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:22.753 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:41:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:41:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:41:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:41:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:41:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:41:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:41:22Z" 2024-04-12T04:41:22.755 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:41:22.755 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:41:22Z" } 2024-04-12T04:41:22.756 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:41:22.756 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:41:22.756 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:41:22.756 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:41:27.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:27.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:27.758 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:41:27.819 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:27.819 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:27.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:27.858+0000 7f0c26dea780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:27.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:27.859+0000 7f0c26dea780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:27.877 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:41:27.877 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:41:27.877 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:41:27.877 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:41:27.877 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:41:27.877 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:41:27.877 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:27.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:27.877 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:41:27.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:27.878 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:41:27.878 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:27.878 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:27.878 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:41:27.878 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:27.878 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:41:27.878 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:27.878 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:41:27.878 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:27.878 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:27.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:27.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:27.879 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:41:27.879 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:27.879 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:27.879 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:41:27.879 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:27.879 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:27.879 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:27.879 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:41:27.879 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:27.879 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:27.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:27.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:27.880 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:41:27.880 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:27.880 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:27.880 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:41:27.880 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:27.880 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:27.880 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:27.880 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:41:27.880 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:27.880 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:27.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:27.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:27.881 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:41:27.881 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:27.881 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:27.881 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:41:27.881 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:27.881 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:27.881 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:27.881 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:41:27.881 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:27.881 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:27.881 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:27.882 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:41:27.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:27.882 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:41:27.882 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:41:27.882 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:41:27.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:27.882 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:41:27Z" 2024-04-12T04:41:27.882 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:41:27.883 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:41:27Z" } 2024-04-12T04:41:27.883 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:41:27.883 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:41:27.884 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:41:27.884 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:41:32.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:32.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:32.886 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:41:32.944 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:32.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:32.981+0000 7f34e6090780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:32.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:32.982+0000 7f34e6090780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:32.999 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:41:32.999 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:41:33.000 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:41:33.000 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:41:33.000 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:41:33.000 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:41:33.000 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:33.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:33.000 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:41:33.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:33.000 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:41:33.000 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:33.000 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:33.001 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:41:33.001 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:33.001 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:41:33.001 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:33.001 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:41:33.001 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:33.001 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:33.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:33.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:33.001 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:41:33.001 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:33.001 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:33.001 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:41:33.002 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:33.002 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:33.002 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:33.002 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:41:33.002 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:33.002 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:33.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:33.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:33.002 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:41:33.002 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:33.002 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:33.002 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:41:33.003 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:33.003 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:33.003 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:33.003 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:41:33.003 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:33.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:33.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:33.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:33.003 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:41:33.003 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:33.003 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:33.003 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:41:33.004 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:33.004 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:33.004 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:33.004 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:41:33.004 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:33.004 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:33.004 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:33.004 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:41:33.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:33.004 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:41:33.004 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:41:33.004 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:41:33.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:33.005 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:41:32Z" 2024-04-12T04:41:33.005 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:41:33.006 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:41:32Z" } 2024-04-12T04:41:33.006 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:41:33.006 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:41:33.006 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:41:33.007 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:41:38.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:38.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:38.008 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:41:38.068 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:38.068 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:38.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:38.105+0000 7f0e69506780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:38.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:38.106+0000 7f0e69506780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:38.124 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:41:38.124 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:41:38.124 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:41:38.124 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:41:38.124 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:41:38.124 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:41:38.124 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:38.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:38.125 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:41:38.125 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:38.125 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:41:38.125 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:38.125 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:38.125 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:41:38.125 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:38.125 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:41:38.125 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:38.126 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:41:38.126 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:38.126 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:38.126 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:38.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:38.126 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:41:38.126 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:38.126 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:38.126 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:41:38.127 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:38.127 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:38.127 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:38.127 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:41:38.127 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:38.127 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:38.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:38.127 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:38.128 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:41:38.128 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:38.128 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:38.128 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:41:38.128 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:38.128 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:38.128 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:38.128 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:41:38.128 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:38.129 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:38.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:38.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:38.129 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:41:38.129 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:38.129 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:38.129 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:41:38.129 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:38.129 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:38.129 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:38.129 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:41:38.130 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:38.130 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:38.130 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:38.130 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:41:38.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:38.130 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:41:38.130 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:41:38.130 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:41:38.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:38.130 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:41:38Z" 2024-04-12T04:41:38.131 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:41:38.131 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:41:38Z" } 2024-04-12T04:41:38.132 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:41:38.132 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:41:38.132 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:41:38.132 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:41:43.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:43.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:43.133 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:41:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:43.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:43.233+0000 7fadbb83b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:43.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:43.234+0000 7fadbb83b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:43.255 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:41:43.255 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:41:43.255 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:41:43.255 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:41:43.255 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:41:43.255 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:41:43.256 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:43.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:43.256 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:41:43.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:43.256 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:41:43.256 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:43.256 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:43.256 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:41:43.256 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:43.256 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:41:43.256 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:43.256 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:41:43.257 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:43.257 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:43.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:43.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:43.257 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:41:43.257 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:43.257 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:43.257 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:41:43.257 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:43.257 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:43.257 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:43.257 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:41:43.258 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:43.258 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:43.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:43.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:43.258 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:41:43.258 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:43.258 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:43.258 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:41:43.258 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:43.258 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:43.258 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:43.258 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:41:43.259 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:43.259 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:43.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:43.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:43.259 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:41:43.259 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:43.259 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:43.259 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:41:43.260 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:43.260 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:43.260 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:43.260 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:41:43.260 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:43.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:43.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:43.260 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:41:43.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:43.260 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:41:43.260 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:41:43.260 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:41:43.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:43.261 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:41:43Z" 2024-04-12T04:41:43.261 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:41:43.262 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:41:43Z" } 2024-04-12T04:41:43.262 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:41:43.262 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:41:43.263 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:41:43.263 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:41:48.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:48.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:48.264 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:41:48.324 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:48.324 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:48.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:48.360+0000 7f54c0b05780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:48.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:48.361+0000 7f54c0b05780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:48.379 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:41:48.379 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:41:48.379 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:41:48.379 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:41:48.379 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:41:48.379 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:41:48.379 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:48.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:48.380 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:41:48.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:48.380 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:41:48.380 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:48.380 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:48.380 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:41:48.380 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:48.380 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:41:48.380 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:48.380 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:41:48.380 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:48.381 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:48.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:48.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:48.381 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:41:48.381 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:48.381 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:48.381 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:41:48.381 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:48.381 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:48.382 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:48.382 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:41:48.382 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:48.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:48.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:48.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:48.382 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:41:48.382 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:48.382 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:48.382 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:41:48.382 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:48.383 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:48.383 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:48.383 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:41:48.383 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:48.383 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:48.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:48.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:48.383 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:41:48.383 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:48.383 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:48.384 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:41:48.384 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:48.384 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:48.384 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:48.384 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:41:48.384 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:48.384 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:48.384 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:48.384 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:41:48.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:48.384 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:41:48.384 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:41:48.385 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:41:48.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:48.385 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:41:48Z" 2024-04-12T04:41:48.385 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:41:48.386 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:41:48Z" } 2024-04-12T04:41:48.386 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:41:48.386 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:41:48.386 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:41:48.386 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:41:53.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:53.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:53.387 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:41:53.448 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:53.448 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:53.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:53.484+0000 7f4f5fef9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:53.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:53.485+0000 7f4f5fef9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:53.501 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:41:53.502 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:41:53.502 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:41:53.502 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:41:53.502 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:41:53.502 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:41:53.502 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:53.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:53.502 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:41:53.502 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:53.503 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:41:53.503 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:53.503 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:53.503 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:41:53.503 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:53.503 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:41:53.503 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:53.503 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:41:53.503 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:53.503 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:53.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:53.504 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:53.504 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:41:53.504 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:53.504 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:53.504 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:41:53.504 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:53.504 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:53.504 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:53.504 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:41:53.504 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:53.504 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:53.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:53.504 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:53.505 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:41:53.505 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:53.505 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:53.505 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:41:53.505 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:53.505 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:53.505 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:53.505 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:41:53.505 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:53.505 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:53.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:53.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:53.506 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:41:53.506 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:53.506 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:53.506 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896663.302022_893.1", 2024-04-12T04:41:53.506 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:53.506 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:53.506 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:53.506 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:43.302022Z", 2024-04-12T04:41:53.506 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:53.507 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:53.507 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:53.507 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:41:53.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:53.507 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:41:53.507 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:41:53.507 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:41:53.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:53.507 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:41:53Z" 2024-04-12T04:41:53.507 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:41:53.508 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896663.302022_893.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:43.302022Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:41:53Z" } 2024-04-12T04:41:53.508 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896663.302022_893.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:43.302022Z', 'realm_epoch': 2}}] 2024-04-12T04:41:53.509 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:41:53.509 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896663.302022_893.1'} 2024-04-12T04:41:53.509 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896795.189600_921.1 target=1_1712896663.302022_893.1 2024-04-12T04:41:55.930 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T04:41:55.928+0000 7fd539413700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:41:58.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:58.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:58.511 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:41:58.570 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:58.570 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:58.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:58.608+0000 7fcec05e2780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:58.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:58.610+0000 7fcec05e2780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:58.627 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:41:58.627 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:41:58.627 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:41:58.627 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:41:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:41:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:41:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:41:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:41:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:41:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:58.628 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:41:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:41:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:41:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:41:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:58.629 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:41:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:41:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:58.630 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:41:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:41:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:41:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:58.631 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896795.189600_921.1", 2024-04-12T04:41:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.189600Z", 2024-04-12T04:41:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:41:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:58.632 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:41:58.633 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:41:58.633 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:41:58.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:58.633 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:41:58Z" 2024-04-12T04:41:58.633 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:41:58.634 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896795.189600_921.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:39:55.189600Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:41:58Z" } 2024-04-12T04:41:58.634 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896795.189600_921.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.189600Z', 'realm_epoch': 2}}] 2024-04-12T04:41:58.634 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:41:58.634 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:41:58.634 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:41:58.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:58.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:58.635 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:41:58.688 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:58.689 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:58.748 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:41:58.748 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:58.748 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:41:58.748 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:41:58.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:58.748 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:58.748 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:41:58.749 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:41:58.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:58.749 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:58.749 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:41:58.749 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:41:58.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:58.749 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:58.749 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:41:58.749 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:41:58.749 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:58.749 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:41:58.750 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:41:58.751 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:41:58.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:58.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:58.751 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:41:58.807 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:58.807 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:58.860 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:41:58.861 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:41:58.861 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:41:58.861 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:41:58.861 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:41:58.861 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:41:58.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:58.861 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:41:58.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:58.861 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:41:58.861 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:58.862 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:41:58.862 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:41:58.862 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:58.862 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:58.862 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:58.862 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:41:58.862 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:58.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:58.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:58.862 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:41:58.863 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:58.863 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:41:58.863 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:41:58.863 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:58.863 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:58.863 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:58.863 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:41:58.863 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:58.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:58.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:58.863 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:41:58.864 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:58.864 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:41:58.864 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:41:58.864 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:58.864 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:41:58.864 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:58.864 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:41:58.864 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:58.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:58.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:58.864 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:41:58.864 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:58.865 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:41:58.865 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:41:58.865 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:58.865 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:41:58.865 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:58.865 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:41:58.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:58.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:58.865 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:41:58.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:58.865 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:41:58.866 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:41:58.866 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:41:58.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:58.866 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:41:58Z" 2024-04-12T04:41:58.866 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:41:58.867 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:41:58Z" } 2024-04-12T04:41:58.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:41:58.867 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:41:58.867 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:41:58.868 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:41:58.868 DEBUG:boto:path=/dkyges-70/ 2024-04-12T04:41:58.868 DEBUG:boto:auth_path=/dkyges-70/ 2024-04-12T04:41:58.868 DEBUG:boto:Method: HEAD 2024-04-12T04:41:58.868 DEBUG:boto:Path: /dkyges-70/ 2024-04-12T04:41:58.868 DEBUG:boto:Data: 2024-04-12T04:41:58.869 DEBUG:boto:Headers: {} 2024-04-12T04:41:58.869 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:41:58.869 DEBUG:boto:Port: 8001 2024-04-12T04:41:58.869 DEBUG:boto:Params: {} 2024-04-12T04:41:58.869 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:41:58.870 DEBUG:boto:Token: None 2024-04-12T04:41:58.870 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:41:58 GMT /dkyges-70/ 2024-04-12T04:41:58.870 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:c2zXWIC9sqfVOLy5GUZNOIobvRY= 2024-04-12T04:41:58.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:41:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:c2zXWIC9sqfVOLy5GUZNOIobvRY=', 'Content-Length': '0'} 2024-04-12T04:41:58.903 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000219d883fd44286a1-006618bb96-26dd-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:41:58.904 DEBUG:boto:path=/dkyges-70/obj3 2024-04-12T04:41:58.904 DEBUG:boto:auth_path=/dkyges-70/obj3 2024-04-12T04:41:58.904 DEBUG:boto:Method: HEAD 2024-04-12T04:41:58.904 DEBUG:boto:Path: /dkyges-70/obj3 2024-04-12T04:41:58.904 DEBUG:boto:Data: 2024-04-12T04:41:58.904 DEBUG:boto:Headers: {} 2024-04-12T04:41:58.904 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:41:58.904 DEBUG:boto:Port: 8001 2024-04-12T04:41:58.905 DEBUG:boto:Params: {} 2024-04-12T04:41:58.905 DEBUG:boto:Token: None 2024-04-12T04:41:58.905 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:41:58 GMT /dkyges-70/obj3 2024-04-12T04:41:58.905 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GeeJCblYIGwvoGUufYVEF4W4FTU= 2024-04-12T04:41:58.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:41:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GeeJCblYIGwvoGUufYVEF4W4FTU=', 'Content-Length': '0'} 2024-04-12T04:41:58.907 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:37:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-70:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9370.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cc193495e538ee4-006618bb96-26dd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:41:58.907 DEBUG:boto:path=/dkyges-70/obj3 2024-04-12T04:41:58.908 DEBUG:boto:auth_path=/dkyges-70/obj3 2024-04-12T04:41:58.908 DEBUG:boto:Method: GET 2024-04-12T04:41:58.908 DEBUG:boto:Path: /dkyges-70/obj3 2024-04-12T04:41:58.908 DEBUG:boto:Data: 2024-04-12T04:41:58.908 DEBUG:boto:Headers: {} 2024-04-12T04:41:58.908 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:41:58.908 DEBUG:boto:Port: 8001 2024-04-12T04:41:58.908 DEBUG:boto:Params: {} 2024-04-12T04:41:58.908 DEBUG:boto:Token: None 2024-04-12T04:41:58.908 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:41:58 GMT /dkyges-70/obj3 2024-04-12T04:41:58.909 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:h1l9t55Tl0+kqZUPEqGuvxH7ylU= 2024-04-12T04:41:58.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:41:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h1l9t55Tl0+kqZUPEqGuvxH7ylU=', 'Content-Length': '0'} 2024-04-12T04:41:58.910 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:37:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-70:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9370.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d2c861f3781432e-006618bb96-26dd-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:41:58.911 DEBUG:boto:path=/dkyges-71/ 2024-04-12T04:41:58.911 DEBUG:boto:auth_path=/dkyges-71/ 2024-04-12T04:41:58.911 DEBUG:boto:Method: HEAD 2024-04-12T04:41:58.911 DEBUG:boto:Path: /dkyges-71/ 2024-04-12T04:41:58.911 DEBUG:boto:Data: 2024-04-12T04:41:58.911 DEBUG:boto:Headers: {} 2024-04-12T04:41:58.911 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:41:58.911 DEBUG:boto:Port: 8001 2024-04-12T04:41:58.911 DEBUG:boto:Params: {} 2024-04-12T04:41:58.912 DEBUG:boto:Token: None 2024-04-12T04:41:58.912 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:41:58 GMT /dkyges-71/ 2024-04-12T04:41:58.912 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:53303ZebzDZq5XJO5d3dhe/Jo34= 2024-04-12T04:41:58.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:41:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:53303ZebzDZq5XJO5d3dhe/Jo34=', 'Content-Length': '0'} 2024-04-12T04:41:58.914 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d9d2f20fe8fefc0-006618bb96-26dd-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:41:58.914 DEBUG:boto:path=/dkyges-71/obj3 2024-04-12T04:41:58.915 DEBUG:boto:auth_path=/dkyges-71/obj3 2024-04-12T04:41:58.915 DEBUG:boto:Method: HEAD 2024-04-12T04:41:58.915 DEBUG:boto:Path: /dkyges-71/obj3 2024-04-12T04:41:58.915 DEBUG:boto:Data: 2024-04-12T04:41:58.915 DEBUG:boto:Headers: {} 2024-04-12T04:41:58.915 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:41:58.915 DEBUG:boto:Port: 8001 2024-04-12T04:41:58.915 DEBUG:boto:Params: {} 2024-04-12T04:41:58.915 DEBUG:boto:Token: None 2024-04-12T04:41:58.916 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:41:58 GMT /dkyges-71/obj3 2024-04-12T04:41:58.916 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xX+fM3Ram2nI4oqWqS6KoybUu88= 2024-04-12T04:41:58.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:41:58 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xX+fM3Ram2nI4oqWqS6KoybUu88=', 'Content-Length': '0'} 2024-04-12T04:41:58.917 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d14bd922500cd2a5-006618bb96-26dd-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:41:58 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:41:58.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dkyges-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:41:58.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dkyges-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:41:58.918 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dkyges-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:41:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:59.040 INFO:teuthology.orchestra.run.smithi043.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:41:59.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:41:59.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:41:59.041 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:41:59.094 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:59.094 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:59.152 INFO:teuthology.orchestra.run.smithi043.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:41:59.153 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... ok 2024-04-12T04:41:59.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:59.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:59.155 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:41:59.208 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:59.208 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:59.248 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:41:59.267 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:41:59.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:59.267 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:41:59.267 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:41:59.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:59.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:59.267 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:41:59.268 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:41:59.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:59.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:59.268 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:41:59.268 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:41:59.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:59.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:59.268 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896919.022472_926.1", 2024-04-12T04:41:59.268 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:41:59.022472Z" 2024-04-12T04:41:59.268 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:59.268 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:41:59.269 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896919.022472_926.1'} 2024-04-12T04:41:59.269 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:41:59.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:59.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:41:59.270 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:41:59.327 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:41:59.327 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:41:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:59.363+0000 7fd7cc603780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:41:59.365+0000 7fd7cc603780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:41:59.381 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:41:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:41:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:41:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:41:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:41:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:41:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:41:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:41:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:41:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:41:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:41:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:41:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:41:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:41:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:41:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:41:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:41:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:41:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:41:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:41:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:41:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896795.189600_921.1", 2024-04-12T04:41:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:41:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:41:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:41:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.189600Z", 2024-04-12T04:41:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:41:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:41:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:41:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:41:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:41:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:41:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:41:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:41:59Z" 2024-04-12T04:41:59.386 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:41:59.387 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896795.189600_921.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:39:55.189600Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:41:59Z" } 2024-04-12T04:41:59.387 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896795.189600_921.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.189600Z', 'realm_epoch': 2}}] 2024-04-12T04:41:59.388 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896919.022472_926.1'} 2024-04-12T04:41:59.388 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:41:59.388 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896919.022472_926.1 target=1_1712896795.189600_921.1 2024-04-12T04:42:04.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:04.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:04.389 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:04.448 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:04.449 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:04.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:42:04.487+0000 7f6c40cb5780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:42:04.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:42:04.488+0000 7f6c40cb5780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:42:04.505 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:42:04.505 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:42:04.505 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:42:04.505 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:42:04.505 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:42:04.505 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:42:04.505 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:04.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:04.506 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:42:04.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:04.506 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:42:04.506 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:04.506 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:04.506 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:42:04.506 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:04.506 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:42:04.506 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:04.506 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:42:04.507 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:04.507 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:04.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:04.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:04.507 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:42:04.507 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:04.507 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:04.507 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:42:04.507 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:04.507 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:04.507 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:04.508 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:42:04.508 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:04.508 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:04.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:04.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:04.508 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:42:04.508 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:04.508 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:04.508 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:42:04.508 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:04.508 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:04.508 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:04.509 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:42:04.509 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:04.509 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:04.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:04.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:04.509 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:42:04.509 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:04.509 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:04.509 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896795.189600_921.1", 2024-04-12T04:42:04.509 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:04.509 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:04.509 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:04.509 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.189600Z", 2024-04-12T04:42:04.510 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:04.510 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:04.510 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:04.510 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:04.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:04.510 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:42:04.510 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:42:04.510 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:42:04.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:04.510 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:42:04Z" 2024-04-12T04:42:04.510 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:42:04.511 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896795.189600_921.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:39:55.189600Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:42:04Z" } 2024-04-12T04:42:04.512 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896795.189600_921.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.189600Z', 'realm_epoch': 2}}] 2024-04-12T04:42:04.512 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896919.022472_926.1'} 2024-04-12T04:42:04.512 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:42:04.512 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896919.022472_926.1 target=1_1712896795.189600_921.1 2024-04-12T04:42:09.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:09.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:09.513 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:09.574 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:09.574 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:09.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:42:09.612+0000 7fa9f2782780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:42:09.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:42:09.613+0000 7fa9f2782780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:42:09.631 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:42:09.631 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:42:09.631 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:42:09.631 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:42:09.631 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:42:09.631 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:42:09.631 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:09.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:09.632 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:42:09.632 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:09.632 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:42:09.632 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:09.632 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:09.632 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:42:09.632 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:09.632 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:42:09.632 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:09.632 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:42:09.632 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:09.633 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:09.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:09.633 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:09.633 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:42:09.633 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:09.633 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:09.633 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:42:09.633 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:09.633 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:09.633 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:09.633 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:42:09.634 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:09.634 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:09.634 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:09.634 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:09.634 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:42:09.634 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:09.634 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:09.634 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:42:09.634 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:09.634 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:09.634 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:09.634 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:42:09.635 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:09.635 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:09.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:09.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:09.635 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:42:09.635 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:09.635 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:09.635 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896795.189600_921.1", 2024-04-12T04:42:09.635 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:09.635 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:09.635 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:09.636 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.189600Z", 2024-04-12T04:42:09.636 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:09.636 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:09.636 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:09.636 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:09.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:09.636 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:42:09.636 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:42:09.636 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:42:09.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:09.636 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:42:09Z" 2024-04-12T04:42:09.636 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:42:09.637 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896795.189600_921.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:39:55.189600Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:42:09Z" } 2024-04-12T04:42:09.637 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896795.189600_921.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.189600Z', 'realm_epoch': 2}}] 2024-04-12T04:42:09.638 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896919.022472_926.1'} 2024-04-12T04:42:09.638 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:42:09.638 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896919.022472_926.1 target=1_1712896795.189600_921.1 2024-04-12T04:42:14.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:14.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:14.639 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:14.699 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:14.699 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:14.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:42:14.737+0000 7f3b6f345780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:42:14.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:42:14.739+0000 7f3b6f345780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:42:14.756 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:42:14.757 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:42:14.757 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:42:14.757 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:42:14.757 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:42:14.757 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:42:14.758 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:14.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:14.758 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:42:14.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:14.758 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:42:14.758 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:14.758 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:14.758 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:42:14.758 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:14.758 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:42:14.758 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:14.759 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:42:14.759 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:14.759 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:14.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:14.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:14.759 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:42:14.759 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:14.760 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:14.760 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:42:14.760 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:14.760 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:14.760 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:14.761 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:42:14.761 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:14.761 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:14.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:14.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:14.762 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:42:14.762 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:14.762 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:14.762 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:42:14.762 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:14.762 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:14.762 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:14.762 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:42:14.763 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:14.763 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:14.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:14.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:14.763 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:42:14.763 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:14.763 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:14.763 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896795.189600_921.1", 2024-04-12T04:42:14.764 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:14.764 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:14.764 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:14.764 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.189600Z", 2024-04-12T04:42:14.764 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:14.765 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:14.765 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:14.765 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:14.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:14.765 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:42:14.765 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:42:14.766 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:42:14.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:14.766 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:42:14Z" 2024-04-12T04:42:14.766 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:42:14.767 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896795.189600_921.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:39:55.189600Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:42:14Z" } 2024-04-12T04:42:14.767 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896795.189600_921.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.189600Z', 'realm_epoch': 2}}] 2024-04-12T04:42:14.768 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896919.022472_926.1'} 2024-04-12T04:42:14.768 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896795.189600_921.1'} 2024-04-12T04:42:14.768 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712896919.022472_926.1 target=1_1712896795.189600_921.1 2024-04-12T04:42:19.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:19.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:19.769 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:19.829 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:19.829 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:19.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:42:19.864+0000 7f47dd7f1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:42:19.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:42:19.865+0000 7f47dd7f1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:42:19.883 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:42:19.883 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:42:19.883 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:42:19.883 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:42:19.883 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:42:19.883 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:42:19.884 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:19.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:19.884 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:42:19.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:19.884 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:42:19.884 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:19.884 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:19.884 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:42:19.884 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:19.884 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:42:19.885 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:19.885 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:42:19.885 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:19.885 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:19.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:19.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:19.885 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:42:19.885 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:19.885 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:19.885 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:42:19.886 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:19.886 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:19.886 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:19.886 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:42:19.886 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:19.886 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:19.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:19.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:19.886 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:42:19.886 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:19.887 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:19.887 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:42:19.887 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:19.887 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:19.887 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:19.887 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:42:19.887 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:19.887 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:19.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:19.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:19.887 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:42:19.888 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:19.888 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:19.888 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896919.022472_926.1", 2024-04-12T04:42:19.888 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:19.888 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:19.888 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:19.888 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:41:59.022472Z", 2024-04-12T04:42:19.888 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:19.888 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:19.888 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:19.888 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:19.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:19.888 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:42:19.889 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:42:19.889 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:42:19.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:19.889 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:42:19Z" 2024-04-12T04:42:19.889 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:42:19.890 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896919.022472_926.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:41:59.022472Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:42:19Z" } 2024-04-12T04:42:19.890 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896919.022472_926.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:41:59.022472Z', 'realm_epoch': 2}}] 2024-04-12T04:42:19.890 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896919.022472_926.1'} 2024-04-12T04:42:19.890 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896919.022472_926.1'} 2024-04-12T04:42:19.891 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:42:19.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:42:19.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:42:19.891 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:42:19.943 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:19.943 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:20.007 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[]} 2024-04-12T04:42:20.007 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-72 2024-04-12T04:42:20.007 DEBUG:boto:path=/dkyges-72/ 2024-04-12T04:42:20.007 DEBUG:boto:auth_path=/dkyges-72/ 2024-04-12T04:42:20.007 DEBUG:boto:Method: PUT 2024-04-12T04:42:20.008 DEBUG:boto:Path: /dkyges-72/ 2024-04-12T04:42:20.008 DEBUG:boto:Data: 2024-04-12T04:42:20.008 DEBUG:boto:Headers: {} 2024-04-12T04:42:20.008 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:42:20.008 DEBUG:boto:Port: 8000 2024-04-12T04:42:20.008 DEBUG:boto:Params: {} 2024-04-12T04:42:20.008 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:42:20.008 DEBUG:boto:Token: None 2024-04-12T04:42:20.009 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:42:20 GMT /dkyges-72/ 2024-04-12T04:42:20.009 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:rGDrJ+YQeT0wVZbG89q3+V+UXpQ= 2024-04-12T04:42:20.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:42:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rGDrJ+YQeT0wVZbG89q3+V+UXpQ=', 'Content-Length': '0'} 2024-04-12T04:42:20.041 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000819ec6313752ada9-006618bbac-26d1-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:42:20.041 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-73 2024-04-12T04:42:20.041 DEBUG:boto:path=/dkyges-73/ 2024-04-12T04:42:20.042 DEBUG:boto:auth_path=/dkyges-73/ 2024-04-12T04:42:20.042 DEBUG:boto:Method: PUT 2024-04-12T04:42:20.042 DEBUG:boto:Path: /dkyges-73/ 2024-04-12T04:42:20.042 DEBUG:boto:Data: 2024-04-12T04:42:20.042 DEBUG:boto:Headers: {} 2024-04-12T04:42:20.042 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:42:20.042 DEBUG:boto:Port: 8000 2024-04-12T04:42:20.042 DEBUG:boto:Params: {} 2024-04-12T04:42:20.042 DEBUG:boto:Token: None 2024-04-12T04:42:20.043 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:42:20 GMT /dkyges-73/ 2024-04-12T04:42:20.043 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VITZcrGE6GrZQodWK06lmBUVjmg= 2024-04-12T04:42:20.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:42:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VITZcrGE6GrZQodWK06lmBUVjmg=', 'Content-Length': '0'} 2024-04-12T04:42:20.062 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d821396b7a0009c1-006618bbac-26d1-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:42:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:42:20.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:20.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:20.062 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:20.126 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:20.126 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:20.164 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:42:20.180 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:42:20.180 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.180 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:42:20.180 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:42:20.180 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.181 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.181 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:42:20.181 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:42:20.181 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.181 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.181 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:42:20.181 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:42:20.181 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.181 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.181 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:42:20.182 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:42:20.059773Z" 2024-04-12T04:42:20.182 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.182 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:42:20.183 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:42:20.183 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:42:20.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:20.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:20.183 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:20.237 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:20.237 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:20.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:42:20.271+0000 7f70b959a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:42:20.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:42:20.273+0000 7f70b959a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:42:20.289 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:42:20.289 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:42:20.290 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:42:20.290 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:42:20.290 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:42:20.290 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:42:20.290 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:20.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.290 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:42:20.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.290 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:42:20.290 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:20.290 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:20.290 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:42:20.291 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:20.291 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:42:20.291 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:20.291 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:42:20.291 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:20.291 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.291 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:42:20.291 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:20.291 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:20.292 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:42:20.292 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:20.292 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:20.292 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:20.292 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:42:20.292 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:20.292 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.292 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:42:20.292 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:20.292 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:20.292 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:42:20.293 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:20.293 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:20.293 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:20.293 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:42:20.293 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:20.293 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.293 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:42:20.293 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:20.293 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:20.293 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:42:20.294 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:20.294 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:20.294 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:20.294 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:42:20.294 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:20.294 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.294 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.294 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:20.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.294 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:42:20.294 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:42:20.295 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:42:20.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.295 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:42:20Z" 2024-04-12T04:42:20.295 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:42:20.296 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:42:20Z" } 2024-04-12T04:42:20.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:42:20.296 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:42:20.296 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:42:20.296 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:42:20.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:42:20.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:42:20.297 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:42:20.350 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:20.351 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:20.405 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:42:20.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:42:20.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:42:20.406 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:42:20.465 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:20.465 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:20.521 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:42:20.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dkyges-72', '--dest-bucket=dkyges-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:42:20.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dkyges-72', '--dest-bucket=dkyges-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:42:20.522 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=dkyges-72 --dest-bucket=dkyges-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:42:20.582 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:20.583 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:20.638 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dkyges-72","zones":["a1"]},"dest":{"bucket":"dkyges-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:42:20.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:42:20.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:42:20.638 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:42:20.695 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:20.695 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:20.751 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dkyges-72","zones":["a1"]},"dest":{"bucket":"dkyges-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:42:20.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:42:20.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:42:20.752 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:42:20.806 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:20.806 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:20.867 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:42:20.867 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:42:20.867 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 18, 2024-04-12T04:42:20.867 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T04:42:20.867 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T04:42:20.867 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T04:42:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:42:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T04:42:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T04:42:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T04:42:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:42:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T04:42:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:42:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:42:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:42:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:42:20.869 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:42:20.869 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:42:20.869 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:42:20.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.869 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:42:20.869 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T04:42:20.869 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:42:20.869 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T04:42:20.869 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:42:20.869 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:42:20.869 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:42:20.870 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:42:20.870 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:42:20.870 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:42:20.870 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:42:20.870 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:42:20.870 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:42:20.870 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:42:20.870 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:42:20.870 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:42:20.870 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:20.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.871 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.871 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:42:20.871 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T04:42:20.871 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:42:20.871 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:42:20.871 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:42:20.871 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:42:20.871 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:42:20.871 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:42:20.871 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:42:20.871 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:42:20.872 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:42:20.872 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:42:20.872 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:42:20.872 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:42:20.872 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:42:20.872 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:42:20.872 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:20.872 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.872 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:42:20.872 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:42:20.872 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.872 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:42:20.873 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:42:20.873 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:42:20.873 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:42:20.873 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:20.873 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.873 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:42:20.873 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:42:20.873 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:42:20.873 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:42:20.873 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [ 2024-04-12T04:42:20.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.873 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-group", 2024-04-12T04:42:20.874 INFO:teuthology.orchestra.run.smithi043.stdout: "data_flow": { 2024-04-12T04:42:20.874 INFO:teuthology.orchestra.run.smithi043.stdout: "symmetrical": [ 2024-04-12T04:42:20.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.874 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-flow1", 2024-04-12T04:42:20.874 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:42:20.874 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:42:20.874 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:42:20.874 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:20.874 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.874 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:20.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.875 INFO:teuthology.orchestra.run.smithi043.stdout: "pipes": [ 2024-04-12T04:42:20.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.875 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-pipe", 2024-04-12T04:42:20.875 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:42:20.875 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-72", 2024-04-12T04:42:20.875 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:42:20.875 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:42:20.875 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:20.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.875 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": { 2024-04-12T04:42:20.875 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-73", 2024-04-12T04:42:20.876 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:42:20.876 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2" 2024-04-12T04:42:20.876 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:20.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.876 INFO:teuthology.orchestra.run.smithi043.stdout: "params": { 2024-04-12T04:42:20.876 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:42:20.876 INFO:teuthology.orchestra.run.smithi043.stdout: "filter": { 2024-04-12T04:42:20.876 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [] 2024-04-12T04:42:20.876 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.876 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": {}, 2024-04-12T04:42:20.877 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-12T04:42:20.877 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "system", 2024-04-12T04:42:20.877 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "" 2024-04-12T04:42:20.877 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.877 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.877 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:42:20.877 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "enabled" 2024-04-12T04:42:20.877 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.877 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:20.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.877 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:42:20.877 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:42:20.878 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:20.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.878 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T04:42:20.878 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T04:42:20.878 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:42:20.878 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T04:42:20.878 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:42:20.878 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:42:20.878 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:42:20.878 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:42:20.878 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:42:20.879 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:42:20.879 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:42:20.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.879 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:42:20.879 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T04:42:20.879 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:42:20.879 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:42:20.879 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:42:20.879 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:42:20.879 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:42:20.879 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:42:20.880 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:42:20.880 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:42:20.880 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:42:20.880 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:42:20.880 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:42:20.880 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:42:20.880 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:42:20.880 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:42:20.880 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:20.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.881 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:42:20.881 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T04:42:20.881 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:42:20.881 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T04:42:20.881 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:42:20.881 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:42:20.881 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:42:20.881 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:42:20.881 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:42:20.881 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:42:20.881 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:42:20.881 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:42:20.882 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:42:20.882 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:42:20.882 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:42:20.882 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:42:20.882 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:20.882 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.882 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:42:20.882 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:42:20.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.882 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:42:20.882 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:42:20.882 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:42:20.883 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:42:20.883 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:20.883 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.883 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:42:20.883 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:42:20.883 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:42:20.883 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:42:20.883 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T04:42:20.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.883 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:42:20.883 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:42:20.884 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:20.884 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.884 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:42:20.884 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T04:42:20.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.884 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:42:20.884 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T04:42:20.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.884 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:42:20.884 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T04:42:20.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.885 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:42:20.885 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T04:42:20.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:20.885 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:42:20.886 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T04:42:20.886 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.886 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:20.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.886 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T04:42:20.886 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:42:20.886 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T04:42:20.886 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T04:42:20.886 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:42:20.886 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:42:20.887 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:42:20.887 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:42:20.887 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:42:20.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.887 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T04:42:20.887 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:42:20.887 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:42:20.887 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:42:20.887 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:42:20.887 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:42:20.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.888 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T04:42:20.888 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:42:20.888 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:42:20.888 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:42:20.888 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:42:20.888 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:42:20.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.888 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T04:42:20.888 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:42:20.888 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:42:20.888 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:42:20.888 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:42:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:42:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T04:42:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:42:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:42:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:42:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:42:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:42:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:42:20.889 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T04:42:20.890 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:20.890 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:42:20.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:42:20.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:42:20.891 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:42:20.941 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:20.942 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:21.015 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dkyges-72","zones":["a1"]},"dest":{"bucket":"dkyges-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:42:21.016 DEBUG:boto:path=/dkyges-72/ 2024-04-12T04:42:21.016 DEBUG:boto:auth_path=/dkyges-72/ 2024-04-12T04:42:21.016 DEBUG:boto:Method: HEAD 2024-04-12T04:42:21.016 DEBUG:boto:Path: /dkyges-72/ 2024-04-12T04:42:21.016 DEBUG:boto:Data: 2024-04-12T04:42:21.016 DEBUG:boto:Headers: {} 2024-04-12T04:42:21.016 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:42:21.016 DEBUG:boto:Port: 8000 2024-04-12T04:42:21.016 DEBUG:boto:Params: {} 2024-04-12T04:42:21.017 DEBUG:boto:Token: None 2024-04-12T04:42:21.017 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:42:21 GMT /dkyges-72/ 2024-04-12T04:42:21.017 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OUiBFaLcqarvsdTtSafLqdmErvI= 2024-04-12T04:42:21.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:42:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OUiBFaLcqarvsdTtSafLqdmErvI=', 'Content-Length': '0'} 2024-04-12T04:42:21.325 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000322134d97ba54a16-006618bbad-2873-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:42:21.326 DEBUG:boto:path=/dkyges-72/obj1 2024-04-12T04:42:21.326 DEBUG:boto:auth_path=/dkyges-72/obj1 2024-04-12T04:42:21.326 DEBUG:boto:Method: PUT 2024-04-12T04:42:21.326 DEBUG:boto:Path: /dkyges-72/obj1 2024-04-12T04:42:21.326 DEBUG:boto:Data: 2024-04-12T04:42:21.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:42:21.327 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:42:21.327 DEBUG:boto:Port: 8000 2024-04-12T04:42:21.327 DEBUG:boto:Params: {} 2024-04-12T04:42:21.327 DEBUG:boto:Token: None 2024-04-12T04:42:21.327 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:42:21 GMT /dkyges-72/obj1 2024-04-12T04:42:21.327 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:/PXIClxixYVj8k7KXhpVmJHlw5I= 2024-04-12T04:42:21.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:42:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/PXIClxixYVj8k7KXhpVmJHlw5I='} 2024-04-12T04:42:21.357 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006bbaf202cde2f2f7-006618bbad-2873-a1'), ('Date', 'Fri, 12 Apr 2024 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:42:21.357 DEBUG:boto:path=/dkyges-72/ 2024-04-12T04:42:21.357 DEBUG:boto:auth_path=/dkyges-72/ 2024-04-12T04:42:21.357 DEBUG:boto:Method: HEAD 2024-04-12T04:42:21.357 DEBUG:boto:Path: /dkyges-72/ 2024-04-12T04:42:21.358 DEBUG:boto:Data: 2024-04-12T04:42:21.358 DEBUG:boto:Headers: {} 2024-04-12T04:42:21.358 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:42:21.358 DEBUG:boto:Port: 8000 2024-04-12T04:42:21.358 DEBUG:boto:Params: {} 2024-04-12T04:42:21.358 DEBUG:boto:Token: None 2024-04-12T04:42:21.358 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:42:21 GMT /dkyges-72/ 2024-04-12T04:42:21.358 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:OUiBFaLcqarvsdTtSafLqdmErvI= 2024-04-12T04:42:21.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:42:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OUiBFaLcqarvsdTtSafLqdmErvI=', 'Content-Length': '0'} 2024-04-12T04:42:21.369 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000013052be7a4f67c93-006618bbad-2873-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:42:21.369 DEBUG:boto:path=/dkyges-72/obj2 2024-04-12T04:42:21.369 DEBUG:boto:auth_path=/dkyges-72/obj2 2024-04-12T04:42:21.369 DEBUG:boto:Method: PUT 2024-04-12T04:42:21.369 DEBUG:boto:Path: /dkyges-72/obj2 2024-04-12T04:42:21.369 DEBUG:boto:Data: 2024-04-12T04:42:21.370 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:42:21.370 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:42:21.370 DEBUG:boto:Port: 8000 2024-04-12T04:42:21.370 DEBUG:boto:Params: {} 2024-04-12T04:42:21.370 DEBUG:boto:Token: None 2024-04-12T04:42:21.370 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:42:21 GMT /dkyges-72/obj2 2024-04-12T04:42:21.370 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AeDTCZ1k/0vxk5Zs3qvmle17Hzs= 2024-04-12T04:42:21.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:42:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AeDTCZ1k/0vxk5Zs3qvmle17Hzs='} 2024-04-12T04:42:21.414 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c8fd627bc2c79a3-006618bbad-2873-a1'), ('Date', 'Fri, 12 Apr 2024 04:42:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:42:21.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:21.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:21.415 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:21.502 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:21.502 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:21.618 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:42:21.669 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:42:21.669 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:21.669 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:42:21.669 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:42:21.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:21.670 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:21.670 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:42:21.670 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:42:21.670 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:21.670 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:21.670 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:42:21.670 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:42:21.670 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:21.670 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:21.670 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:42:21.670 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:42:20.059773Z" 2024-04-12T04:42:21.671 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:21.671 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:42:21.672 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:42:21.672 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:42:21.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:21.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:21.672 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:21.752 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:21.752 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:21.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:42:21.825+0000 7fe3f6802780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:42:21.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:42:21.828+0000 7fe3f6802780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:42:21.860 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:42:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:42:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:42:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:42:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:42:21.860 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:42:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:42:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:42:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:42:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:42:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:21.861 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:42:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:42:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:42:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:21.862 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:42:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:42:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:21.863 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:42:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:42:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:42:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:21.864 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:42:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:42:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:42:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:42:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:42:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:42:21.865 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:42:21.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:21.866 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:42:21Z" 2024-04-12T04:42:21.866 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:42:21.867 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:42:21Z" } 2024-04-12T04:42:21.867 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:42:21.867 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:42:21.867 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:42:21.867 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:42:21.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:21.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:21.868 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:21.939 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:21.939 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:22.011 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:42:22.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:22.011 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047778", 2024-04-12T04:42:22.012 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:42:21.389310Z" 2024-04-12T04:42:22.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:22.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:22.012 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:42:22.012 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:42:22.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:22.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:22.012 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:42:22.012 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:42:21.352708Z" 2024-04-12T04:42:22.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:22.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:22.013 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:42:22.013 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:42:22.013 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:22.013 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:42:22.014 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:22.014 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:42:22.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:22.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:22.014 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:22.075 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:22.076 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:22.130 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:42:22.130 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:42:22.130 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:42:22.130 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:42:22.130 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:42:22.130 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:42:22.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:22.130 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:42:22.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:22.131 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:42:22.131 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:22.131 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:22.131 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:42:22.131 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:22.131 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:22.131 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:22.131 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:42:22.131 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:22.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:22.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:22.131 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:42:22.132 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:22.132 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:22.132 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:42:22.132 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:22.132 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:22.132 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:22.132 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:42:22.132 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:22.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:22.132 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:22.132 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:42:22.132 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:22.133 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:22.133 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:42:22.133 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:22.133 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:42:22.133 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:22.133 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:42:22.133 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:22.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:22.133 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:22.133 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:42:22.133 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:22.133 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:22.133 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:42:22.134 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:22.134 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:42:22.134 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:22.134 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:42:22.134 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:22.134 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:22.134 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:22.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:22.134 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:42:22.134 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:42:22.134 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:42:22.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:22.135 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:42:22Z" 2024-04-12T04:42:22.135 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:42:22.135 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:42:22Z" } 2024-04-12T04:42:22.136 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:42:22.136 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:22.136 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:22.136 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:42:27.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:27.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:27.137 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:27.201 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:27.201 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:27.261 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:42:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:42:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:42:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:42:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:42:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:42:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:42:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:42:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:42:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:42:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:42:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:42:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:42:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:42:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:42:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:42:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:42:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:42:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:42:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:42:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:42:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:42:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:42:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:42:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:27.267 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:42:27Z" 2024-04-12T04:42:27.267 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:42:27.267 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:42:27Z" } 2024-04-12T04:42:27.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:42:27.268 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:27.268 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:27.268 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:42:32.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:32.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:32.269 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:32.332 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:32.332 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:32.388 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:42:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:42:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:42:32.388 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:42:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:42:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:42:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:42:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:42:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:42:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:32.389 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:42:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:42:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:42:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:32.390 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:42:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:42:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:42:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:32.391 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:42:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:42:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:42:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:42:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:32.392 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:42:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:42:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:42:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:42:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:42:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:32.393 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:42:32Z" 2024-04-12T04:42:32.394 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:42:32.395 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:42:32Z" } 2024-04-12T04:42:32.395 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:42:32.395 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:32.395 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:32.395 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:42:37.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:37.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:37.397 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:37.458 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:37.458 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:37.516 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:42:37.516 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:42:37.516 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:42:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:42:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:42:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:42:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:42:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:42:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:42:37.517 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:42:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:42:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:42:37.518 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:42:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:42:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:42:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:42:37.519 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:42:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:42:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:42:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:42:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:37.520 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:42:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:42:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:42:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:42:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:37.521 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:42:37Z" 2024-04-12T04:42:37.521 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:42:37.522 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:42:37Z" } 2024-04-12T04:42:37.523 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:42:37.523 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:37.523 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:37.523 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:42:42.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:42.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:42.525 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:42.586 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:42.586 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:42.642 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:42:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:42:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:42:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:42:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:42:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:42:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:42.642 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:42:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:42:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:42:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:42:42.643 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:42:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:42:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:42:42.644 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:42:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:42:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:42:42.645 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:42:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:42:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:42:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:42.646 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:42:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:42:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:42:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:42:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:42:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:42.647 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:42:42Z" 2024-04-12T04:42:42.647 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:42:42.648 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:42:42Z" } 2024-04-12T04:42:42.649 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:42:42.649 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:42.649 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:42.649 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:42:47.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:47.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:47.651 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:47.712 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:47.712 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:48.774 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:42:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:42:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:42:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:42:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:42:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:42:48.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:42:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:42:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:42:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:42:48.775 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:42:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:42:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:42:48.776 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:42:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:42:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:42:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:42:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:48.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:42:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:42:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:42:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:42:48.778 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:42:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:42:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:42:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:48.779 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:42:48Z" 2024-04-12T04:42:48.779 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:42:48.780 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:42:48Z" } 2024-04-12T04:42:48.780 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:42:48.781 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:48.781 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:48.781 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:42:53.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:53.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:53.782 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:53.842 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:53.845 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:53.897 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:42:53.898 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:42:53.898 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:42:53.898 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:42:53.898 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:42:53.898 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:42:53.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:53.898 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:42:53.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:53.898 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:42:53.899 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:53.899 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:53.899 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:42:53.899 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:53.899 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:53.899 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:53.899 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:42:53.899 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:53.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:53.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:53.899 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:42:53.900 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:53.900 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:53.900 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:42:53.900 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:53.900 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:53.900 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:53.900 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:42:53.900 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:53.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:53.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:53.900 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:42:53.901 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:53.901 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:53.901 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:42:53.901 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:53.901 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:42:53.901 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:53.901 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:42:53.901 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:53.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:53.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:53.901 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:42:53.901 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:53.902 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:53.902 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:42:53.902 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:53.902 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:42:53.902 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:53.902 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:42:53.902 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:53.902 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:53.902 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:53.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:53.902 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:42:53.903 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:42:53.903 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:42:53.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:53.903 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:42:53Z" 2024-04-12T04:42:53.903 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:42:53.904 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:42:53Z" } 2024-04-12T04:42:53.904 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:42:53.904 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:53.904 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:53.904 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:42:58.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:58.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:42:58.906 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:42:58.967 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:42:58.967 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:42:59.023 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:42:59.023 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:42:59.023 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:42:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:42:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:42:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:42:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:42:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:42:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:42:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:42:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:42:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:42:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:42:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:42:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:42:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:42:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:42:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:42:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:42:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:42:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:42:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:42:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:42:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:42:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:42:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:42:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:42:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:42:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:42:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:42:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:42:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:42:59.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:42:59.029 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:42:59Z" 2024-04-12T04:42:59.029 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:42:59.030 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:42:59Z" } 2024-04-12T04:42:59.030 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:42:59.030 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:59.030 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:42:59.030 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:43:04.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:04.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:04.032 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:43:04.094 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:43:04.095 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:43:04.150 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:43:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:43:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:43:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:43:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:43:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:43:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:43:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:43:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:43:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:43:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:43:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:43:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:43:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:43:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:43:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:43:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:43:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:43:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:43:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:43:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:43:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:43:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:43:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:43:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:43:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:43:04Z" 2024-04-12T04:43:04.155 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:43:04.156 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:43:04Z" } 2024-04-12T04:43:04.156 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:43:04.157 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:04.157 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:04.157 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:43:09.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:09.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:09.158 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:43:09.222 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:43:09.222 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:43:09.282 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:43:09.283 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:43:09.283 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:43:09.283 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:43:09.283 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:43:09.283 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:43:09.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:09.283 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:43:09.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:09.283 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:43:09.283 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:09.284 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:09.284 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:43:09.284 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:09.284 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:09.284 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:09.284 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:43:09.284 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:09.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:09.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:09.284 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:43:09.284 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:09.285 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:09.285 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:43:09.285 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:09.285 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:09.285 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:09.285 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:43:09.285 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:09.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:09.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:09.285 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:43:09.285 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:09.286 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:09.286 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:43:09.286 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:09.286 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:43:09.286 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:09.286 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:43:09.286 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:09.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:09.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:09.286 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:43:09.287 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:09.287 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:09.287 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:43:09.287 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:09.287 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:43:09.287 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:09.287 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:43:09.287 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:09.287 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:09.287 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:43:09.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:09.288 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:43:09.288 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:43:09.288 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:43:09.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:09.288 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:43:09Z" 2024-04-12T04:43:09.288 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:43:09.289 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:43:09Z" } 2024-04-12T04:43:09.289 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:43:09.289 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:09.289 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:09.290 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:43:14.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:14.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:14.291 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:43:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:43:14.352 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:43:14.408 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:43:14.408 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:43:14.408 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:43:14.408 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:43:14.408 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:43:14.408 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:43:14.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:14.409 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:43:14.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:14.409 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:43:14.409 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:14.409 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:14.409 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:43:14.409 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:14.409 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:14.409 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:14.409 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:43:14.410 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:14.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:14.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:14.410 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:43:14.410 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:14.410 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:14.410 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:43:14.410 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:14.410 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:14.410 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:14.411 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:43:14.411 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:14.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:14.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:14.411 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:43:14.411 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:14.411 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:14.411 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:43:14.411 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:14.411 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:43:14.411 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:14.412 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:43:14.412 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:14.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:14.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:14.412 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:43:14.412 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:14.412 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:14.412 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:43:14.412 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:14.412 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:43:14.412 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:14.413 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:43:14.413 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:14.413 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:14.413 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:43:14.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:14.413 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:43:14.413 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:43:14.413 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:43:14.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:14.413 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:43:14Z" 2024-04-12T04:43:14.413 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:43:14.415 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:43:14Z" } 2024-04-12T04:43:14.415 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:43:14.415 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:14.415 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:14.415 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:43:19.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:19.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:19.417 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:43:19.478 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:43:19.479 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:43:19.533 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:43:19.533 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:43:19.533 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:43:19.533 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:43:19.534 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:43:19.534 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:43:19.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:19.534 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:43:19.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:19.534 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:43:19.534 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:19.534 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:19.534 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:43:19.534 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:19.534 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:19.535 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:19.535 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:43:19.535 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:19.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:19.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:19.535 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:43:19.535 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:19.535 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:19.535 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:43:19.535 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:19.536 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:19.536 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:19.536 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:43:19.536 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:19.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:19.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:19.536 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:43:19.536 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:19.537 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:19.537 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:43:19.537 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:19.537 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:43:19.537 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:19.537 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:43:19.537 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:19.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:19.537 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:19.537 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:43:19.537 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:19.538 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:19.538 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:43:19.538 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:19.538 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:43:19.538 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:19.538 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:43:19.538 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:19.538 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:19.538 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:43:19.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:19.538 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:43:19.539 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:43:19.539 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:43:19.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:19.539 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:43:19Z" 2024-04-12T04:43:19.539 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:43:19.540 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:43:19Z" } 2024-04-12T04:43:19.540 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:43:19.540 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:19.540 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:19.540 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:43:24.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:24.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:24.542 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:43:24.603 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:43:24.604 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:43:24.661 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:43:24.661 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:43:24.661 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:43:24.661 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:43:24.662 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:43:24.662 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:43:24.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:24.662 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:43:24.662 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:24.662 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:43:24.662 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:24.662 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:24.662 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:43:24.662 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:24.662 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:24.663 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:24.663 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:43:24.663 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:24.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:24.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:24.663 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:43:24.663 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:24.663 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:24.663 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:43:24.663 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:24.663 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:24.663 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:24.664 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:43:24.664 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:24.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:24.664 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:24.664 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:43:24.664 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:24.664 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:24.664 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:43:24.664 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:24.664 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:43:24.664 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:24.664 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:43:24.665 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:24.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:24.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:24.665 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:43:24.665 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:24.665 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:24.665 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:43:24.665 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:24.665 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:43:24.665 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:24.665 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:43:24.665 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:24.666 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:24.666 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:43:24.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:24.666 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:43:24.666 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:43:24.666 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:43:24.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:24.666 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:43:24Z" 2024-04-12T04:43:24.666 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:43:24.667 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:43:24Z" } 2024-04-12T04:43:24.667 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:43:24.668 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:24.668 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:24.668 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:43:29.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:29.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:29.669 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:43:29.733 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:43:29.733 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:43:29.789 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:43:29.789 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:43:29.789 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:43:29.789 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:43:29.790 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:43:29.790 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:43:29.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:29.790 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:43:29.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:29.790 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:43:29.790 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:29.790 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:29.790 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:43:29.790 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:29.791 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:29.791 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:29.791 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:43:29.791 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:29.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:29.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:29.791 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:43:29.791 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:29.791 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:29.791 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:43:29.791 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:29.792 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:29.792 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:29.792 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:43:29.792 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:29.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:29.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:29.792 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:43:29.792 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:29.792 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:29.792 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:43:29.792 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:29.793 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:43:29.793 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:29.793 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:43:29.793 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:29.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:29.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:29.793 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:43:29.793 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:29.793 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:29.793 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:43:29.794 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:29.794 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:43:29.794 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:29.794 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:43:29.794 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:29.794 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:29.794 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:43:29.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:29.794 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:43:29.794 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:43:29.794 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:43:29.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:29.795 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:43:29Z" 2024-04-12T04:43:29.795 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:43:29.796 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:43:29Z" } 2024-04-12T04:43:29.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:43:29.796 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:29.796 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:29.796 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:43:34.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:34.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:34.798 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:43:34.860 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:43:34.860 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:43:34.918 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:43:34.918 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:43:34.918 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:43:34.918 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:43:34.919 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:43:34.919 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:43:34.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:34.919 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:43:34.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:34.920 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:43:34.920 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:34.920 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:34.920 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:43:34.920 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:34.920 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:34.921 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:34.921 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:43:34.921 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:34.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:34.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:34.921 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:43:34.921 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:34.921 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:34.921 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:43:34.922 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:34.922 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:34.922 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:34.922 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:43:34.922 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:34.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:34.922 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:34.922 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:43:34.923 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:34.923 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:34.923 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:43:34.923 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:34.923 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:43:34.923 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:34.923 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:43:34.923 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:34.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:34.923 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:34.923 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:43:34.924 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:34.924 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:34.924 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:43:34.924 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:34.924 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:43:34.924 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:34.924 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:43:34.924 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:34.924 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:34.924 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:43:34.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:34.925 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:43:34.925 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:43:34.925 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:43:34.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:34.925 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:43:34Z" 2024-04-12T04:43:34.925 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:43:34.926 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:43:34Z" } 2024-04-12T04:43:34.926 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:43:34.927 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:34.927 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:34.927 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:43:39.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:39.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:39.928 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:43:39.989 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:43:39.989 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:43:40.046 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:43:40.047 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:43:40.047 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:43:40.047 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:43:40.047 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:43:40.047 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:43:40.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:40.047 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:43:40.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:40.047 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:43:40.047 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:40.048 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:40.048 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:43:40.048 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:40.048 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:40.048 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:40.048 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:43:40.048 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:40.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:40.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:40.048 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:43:40.048 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:40.049 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:40.049 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:43:40.049 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:40.049 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:40.049 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:40.049 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:43:40.049 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:40.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:40.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:40.049 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:43:40.049 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:40.049 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:40.050 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:43:40.050 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:40.050 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:43:40.050 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:40.050 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:43:40.050 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:40.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:40.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:40.050 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:43:40.050 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:40.050 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:40.051 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:43:40.051 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:40.051 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:43:40.051 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:40.051 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:43:40.051 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:40.051 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:40.051 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:43:40.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:40.051 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:43:40.051 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:43:40.051 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:43:40.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:40.052 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:43:40Z" 2024-04-12T04:43:40.052 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:43:40.053 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:43:40Z" } 2024-04-12T04:43:40.053 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:43:40.053 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:40.053 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:40.053 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:43:45.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:45.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:45.055 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:43:45.117 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:43:45.118 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:43:45.172 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:43:45.172 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:43:45.172 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:43:45.172 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:43:45.172 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:43:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:43:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:43:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:43:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:43:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:43:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:43:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:43:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:43:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:43:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:43:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:43:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:43:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:43:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:43:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:43:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:43:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:43:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:43:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:43:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:43:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:43:45Z" 2024-04-12T04:43:45.178 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:43:45.179 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:43:45Z" } 2024-04-12T04:43:45.179 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:43:45.179 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:45.179 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:45.179 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:43:50.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:50.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:50.181 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:43:50.246 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:43:50.246 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:43:50.301 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:43:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:43:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:43:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:43:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:43:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:43:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:43:50.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:43:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:43:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:43:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:50.302 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:43:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:43:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:43:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:50.303 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:43:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:43:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:43:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:43:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:50.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:43:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:43:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:43:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:43:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:50.305 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:43:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:43:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:43:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:43:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:50.306 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:43:50Z" 2024-04-12T04:43:50.306 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:43:50.307 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:43:50Z" } 2024-04-12T04:43:50.307 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:43:50.307 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:50.308 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:50.308 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:43:55.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:55.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:43:55.309 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:43:55.373 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:43:55.373 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:43:55.430 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:43:55.430 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:43:55.430 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:43:55.431 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:43:55.431 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:43:55.431 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:43:55.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:55.431 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:43:55.431 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:55.431 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:43:55.431 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:55.431 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:55.431 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:43:55.431 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:55.431 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:55.432 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:55.432 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:43:55.432 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:55.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:55.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:55.432 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:43:55.432 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:55.432 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:55.432 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:43:55.432 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:55.432 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:43:55.433 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:55.433 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:43:55.433 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:55.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:55.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:55.433 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:43:55.433 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:55.433 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:55.433 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:43:55.433 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:55.433 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:43:55.434 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:55.434 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:43:55.434 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:55.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:55.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:43:55.434 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:43:55.434 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:43:55.434 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:43:55.434 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:43:55.434 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:43:55.434 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:43:55.435 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:43:55.435 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:43:55.435 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:55.435 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:43:55.435 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:43:55.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:55.435 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:43:55.435 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:43:55.435 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:43:55.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:43:55.435 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:43:55Z" 2024-04-12T04:43:55.435 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:43:55.436 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:43:55Z" } 2024-04-12T04:43:55.436 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:43:55.437 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:55.437 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:43:55.437 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:44:00.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:00.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:00.439 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:44:00.501 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:00.501 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:00.557 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:44:00.557 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:44:00.558 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:44:00.558 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:44:00.558 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:44:00.558 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:44:00.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:00.558 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:44:00.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:00.558 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:44:00.558 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:00.558 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:00.559 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:44:00.559 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:00.559 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:00.559 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:00.559 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:44:00.559 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:00.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:00.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:00.559 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:44:00.559 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:00.559 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:00.560 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:44:00.560 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:00.560 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:00.560 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:00.560 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:44:00.560 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:00.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:00.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:00.560 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:44:00.560 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:00.561 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:00.561 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:44:00.561 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:00.561 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:44:00.561 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:00.561 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:44:00.561 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:00.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:00.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:00.561 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:44:00.562 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:00.562 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:00.562 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:44:00.562 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:00.562 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:44:00.562 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:00.562 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:44:00.562 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:00.562 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:00.562 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:00.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:00.563 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:44:00.563 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:44:00.563 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:44:00.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:00.563 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:44:00Z" 2024-04-12T04:44:00.563 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:44:00.564 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:44:00Z" } 2024-04-12T04:44:00.564 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:44:00.564 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:44:00.564 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:44:00.565 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:44:05.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:05.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:05.566 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:44:05.629 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:05.629 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:05.687 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:44:05.688 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:44:05.688 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:44:05.688 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:44:05.688 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:44:05.688 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:44:05.688 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:05.688 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:44:05.688 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:05.688 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:44:05.688 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:05.688 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:05.689 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:44:05.689 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:05.689 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:05.689 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:05.689 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:44:05.689 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:05.689 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:05.689 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:05.689 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:44:05.689 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:05.689 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:05.690 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:44:05.690 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:05.690 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:05.690 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:05.690 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:44:05.690 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:05.690 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:05.690 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:05.690 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:44:05.690 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:05.690 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:05.691 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:44:05.691 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:05.691 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:44:05.691 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:05.691 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:44:05.691 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:05.691 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:05.691 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:05.691 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:44:05.691 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:05.691 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:05.691 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:44:05.692 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:05.692 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:44:05.692 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:05.692 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:44:05.692 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:05.692 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:05.692 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:05.692 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:05.692 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:44:05.692 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:44:05.692 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:44:05.692 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:05.692 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:44:05Z" 2024-04-12T04:44:05.693 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:44:05.693 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:44:05Z" } 2024-04-12T04:44:05.694 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:44:05.694 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:44:05.694 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:44:05.694 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:44:10.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:10.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:10.695 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:44:10.757 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:10.757 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:10.819 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:44:10.819 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:44:10.819 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:44:10.819 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:44:10.819 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:44:10.820 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:44:10.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:10.820 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:44:10.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:10.820 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:44:10.820 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:10.820 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:10.820 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:44:10.820 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:10.820 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:10.820 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:10.820 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:44:10.821 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:10.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:10.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:10.821 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:44:10.821 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:10.821 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:10.821 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:44:10.821 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:10.821 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:10.821 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:10.821 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:44:10.822 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:10.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:10.822 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:10.822 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:44:10.822 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:10.822 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:10.822 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:44:10.822 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:10.822 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:44:10.822 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:10.822 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:44:10.822 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:10.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:10.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:10.823 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:44:10.823 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:10.823 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:10.823 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:44:10.823 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:10.823 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:44:10.823 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:10.823 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:44:10.823 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:10.823 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:10.823 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:10.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:10.824 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:44:10.824 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:44:10.824 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:44:10.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:10.824 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:44:10Z" 2024-04-12T04:44:10.824 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:44:10.825 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:44:10Z" } 2024-04-12T04:44:10.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:44:10.825 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:44:10.825 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:44:10.825 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:44:15.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:15.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:15.827 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:44:15.889 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:15.889 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:15.944 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:44:15.945 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:44:15.945 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:44:15.945 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:44:15.945 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:44:15.945 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:44:15.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:15.945 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:44:15.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:15.945 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:44:15.946 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:15.946 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:15.946 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:44:15.946 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:15.946 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:15.946 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:15.946 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:44:15.946 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:15.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:15.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:15.946 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:44:15.947 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:15.947 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:15.947 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:44:15.950 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:15.950 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:15.950 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:15.950 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:44:15.950 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:15.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:15.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:15.950 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:44:15.951 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:15.951 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:15.951 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:44:15.951 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:15.951 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:44:15.951 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:15.951 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:44:15.951 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:15.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:15.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:15.951 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:44:15.952 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:15.952 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:15.952 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:44:15.952 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:15.952 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:44:15.952 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:15.952 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:44:15.952 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:15.952 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:15.952 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:15.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:15.952 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:44:15.953 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:44:15.953 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:44:15.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:15.953 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:44:15Z" 2024-04-12T04:44:15.953 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:44:15.954 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:44:15Z" } 2024-04-12T04:44:15.954 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:44:15.954 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:44:15.954 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:44:15.955 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:44:20.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:20.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:20.957 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:44:21.018 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:21.018 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:21.074 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:44:21.074 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:44:21.075 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:44:21.075 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:44:21.075 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:44:21.075 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:44:21.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:21.075 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:44:21.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:21.075 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:44:21.075 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:21.075 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:21.076 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047638", 2024-04-12T04:44:21.076 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:21.076 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:21.076 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:21.076 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.898541Z" 2024-04-12T04:44:21.076 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:21.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:21.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:21.076 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:44:21.076 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:21.076 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:21.076 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:44:21.077 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:21.077 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:21.077 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:21.077 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:44:21.077 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:21.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:21.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:21.077 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:44:21.077 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:21.077 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:21.077 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054350", 2024-04-12T04:44:21.078 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:21.078 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:44:21.078 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:21.078 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:39:55.372367Z" 2024-04-12T04:44:21.078 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:21.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:21.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:21.078 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:44:21.078 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:21.078 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:21.078 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:44:21.078 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:21.079 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:44:21.079 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:21.079 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:44:21.079 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:21.079 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:21.079 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:21.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:21.079 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:44:21.079 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:44:21.080 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:44:21.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:21.080 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:44:21Z" 2024-04-12T04:44:21.080 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:44:21.081 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047638", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:37:49.898541Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054350", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:39:55.372367Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:44:21Z" } 2024-04-12T04:44:21.081 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047638', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.898541Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054350', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:39:55.372367Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:44:21.081 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:44:21.082 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047638', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054350', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:44:21.082 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047778 target=00000000000000000000:00000000000000047638, shard=2 master=00000000000000000000:00000000000000054490 target=00000000000000000000:00000000000000054350 2024-04-12T04:44:22.189 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T04:44:22.187+0000 7fd539413700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:44:26.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:26.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:26.083 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:44:26.145 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:26.145 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:26.204 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:44:26.204 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:44:26.204 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:44:26.205 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:44:26.205 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:44:26.205 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:44:26.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.205 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:44:26.205 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.205 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:44:26.205 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:26.205 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:26.205 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047778", 2024-04-12T04:44:26.205 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:26.206 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:26.206 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:26.206 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:21.389310Z" 2024-04-12T04:44:26.206 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.206 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.206 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.206 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:44:26.206 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:26.206 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:26.206 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:44:26.206 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:26.207 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:26.207 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:26.207 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:44:26.207 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.207 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.207 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:44:26.207 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:26.207 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:26.207 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:44:26.208 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:26.208 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:44:26.208 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:26.208 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:21.352708Z" 2024-04-12T04:44:26.208 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.208 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.208 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.208 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:44:26.208 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:26.208 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:44:26.208 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:44:26.208 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:26.208 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:44:26.209 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:26.209 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:44:26.209 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.209 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.209 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:26.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.209 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:44:26.209 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:44:26.209 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:44:26.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.209 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:44:26Z" 2024-04-12T04:44:26.210 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:44:26.210 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047778", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:21.389310Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:42:21.352708Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:44:26Z" } 2024-04-12T04:44:26.211 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047778', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:21.389310Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:42:21.352708Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:44:26.211 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:44:26.211 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:44:26.211 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:44:26.211 DEBUG:boto:path=/dkyges-72/ 2024-04-12T04:44:26.211 DEBUG:boto:auth_path=/dkyges-72/ 2024-04-12T04:44:26.212 DEBUG:boto:Method: HEAD 2024-04-12T04:44:26.212 DEBUG:boto:Path: /dkyges-72/ 2024-04-12T04:44:26.212 DEBUG:boto:Data: 2024-04-12T04:44:26.212 DEBUG:boto:Headers: {} 2024-04-12T04:44:26.212 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:44:26.212 DEBUG:boto:Port: 8001 2024-04-12T04:44:26.212 DEBUG:boto:Params: {} 2024-04-12T04:44:26.213 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:44:26.213 DEBUG:boto:Token: None 2024-04-12T04:44:26.213 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:44:26 GMT /dkyges-72/ 2024-04-12T04:44:26.214 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:nJvdC7Z1HDS3xied3A92ASsK3sQ= 2024-04-12T04:44:26.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:44:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nJvdC7Z1HDS3xied3A92ASsK3sQ=', 'Content-Length': '0'} 2024-04-12T04:44:26.229 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054d5ef5a7f53dd46-006618bc2a-286d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:44:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:44:26.230 DEBUG:boto:path=/dkyges-72/obj1 2024-04-12T04:44:26.230 DEBUG:boto:auth_path=/dkyges-72/obj1 2024-04-12T04:44:26.230 DEBUG:boto:Method: HEAD 2024-04-12T04:44:26.230 DEBUG:boto:Path: /dkyges-72/obj1 2024-04-12T04:44:26.230 DEBUG:boto:Data: 2024-04-12T04:44:26.230 DEBUG:boto:Headers: {} 2024-04-12T04:44:26.230 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:44:26.230 DEBUG:boto:Port: 8001 2024-04-12T04:44:26.230 DEBUG:boto:Params: {} 2024-04-12T04:44:26.231 DEBUG:boto:Token: None 2024-04-12T04:44:26.231 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:44:26 GMT /dkyges-72/obj1 2024-04-12T04:44:26.231 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:u3I3vrZD4pq0Ivpb8+owSHjZ3Lg= 2024-04-12T04:44:26.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:44:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:u3I3vrZD4pq0Ivpb8+owSHjZ3Lg=', 'Content-Length': '0'} 2024-04-12T04:44:26.232 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000001b44b9640cbdec1-006618bc2a-286d-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:44:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:44:26.233 DEBUG:boto:path=/dkyges-72/obj2 2024-04-12T04:44:26.233 DEBUG:boto:auth_path=/dkyges-72/obj2 2024-04-12T04:44:26.233 DEBUG:boto:Method: HEAD 2024-04-12T04:44:26.233 DEBUG:boto:Path: /dkyges-72/obj2 2024-04-12T04:44:26.233 DEBUG:boto:Data: 2024-04-12T04:44:26.233 DEBUG:boto:Headers: {} 2024-04-12T04:44:26.233 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:44:26.233 DEBUG:boto:Port: 8001 2024-04-12T04:44:26.233 DEBUG:boto:Params: {} 2024-04-12T04:44:26.234 DEBUG:boto:Token: None 2024-04-12T04:44:26.234 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:44:26 GMT /dkyges-72/obj2 2024-04-12T04:44:26.234 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:yluInBWRMc15J3LlKWW56iPwE/E= 2024-04-12T04:44:26.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:44:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:yluInBWRMc15J3LlKWW56iPwE/E=', 'Content-Length': '0'} 2024-04-12T04:44:26.235 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d7e77770db0ac79d-006618bc2a-286d-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:44:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:44:26.235 DEBUG:boto:path=/dkyges-73/ 2024-04-12T04:44:26.235 DEBUG:boto:auth_path=/dkyges-73/ 2024-04-12T04:44:26.236 DEBUG:boto:Method: HEAD 2024-04-12T04:44:26.236 DEBUG:boto:Path: /dkyges-73/ 2024-04-12T04:44:26.236 DEBUG:boto:Data: 2024-04-12T04:44:26.236 DEBUG:boto:Headers: {} 2024-04-12T04:44:26.236 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:44:26.236 DEBUG:boto:Port: 8001 2024-04-12T04:44:26.236 DEBUG:boto:Params: {} 2024-04-12T04:44:26.236 DEBUG:boto:Token: None 2024-04-12T04:44:26.236 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:44:26 GMT /dkyges-73/ 2024-04-12T04:44:26.236 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ROxcmkOBGrEBS8jbXcoDuc7CLwA= 2024-04-12T04:44:26.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:44:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ROxcmkOBGrEBS8jbXcoDuc7CLwA=', 'Content-Length': '0'} 2024-04-12T04:44:26.239 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038a0cd90a7ad5c4c-006618bc2a-286d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:44:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:44:26.239 DEBUG:boto:path=/dkyges-73/obj1 2024-04-12T04:44:26.239 DEBUG:boto:auth_path=/dkyges-73/obj1 2024-04-12T04:44:26.239 DEBUG:boto:Method: HEAD 2024-04-12T04:44:26.240 DEBUG:boto:Path: /dkyges-73/obj1 2024-04-12T04:44:26.240 DEBUG:boto:Data: 2024-04-12T04:44:26.240 DEBUG:boto:Headers: {} 2024-04-12T04:44:26.240 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:44:26.240 DEBUG:boto:Port: 8001 2024-04-12T04:44:26.240 DEBUG:boto:Params: {} 2024-04-12T04:44:26.240 DEBUG:boto:Token: None 2024-04-12T04:44:26.240 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:44:26 GMT /dkyges-73/obj1 2024-04-12T04:44:26.240 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:lhDb4AfzXKBF6ue0pafo5Km9b/k= 2024-04-12T04:44:26.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:44:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lhDb4AfzXKBF6ue0pafo5Km9b/k=', 'Content-Length': '0'} 2024-04-12T04:44:26.242 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:42:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-72:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9939.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030bdb02e27fd2c96-006618bc2a-286d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:44:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:44:26.243 DEBUG:boto:path=/dkyges-73/obj1 2024-04-12T04:44:26.243 DEBUG:boto:auth_path=/dkyges-73/obj1 2024-04-12T04:44:26.243 DEBUG:boto:Method: GET 2024-04-12T04:44:26.243 DEBUG:boto:Path: /dkyges-73/obj1 2024-04-12T04:44:26.243 DEBUG:boto:Data: 2024-04-12T04:44:26.243 DEBUG:boto:Headers: {} 2024-04-12T04:44:26.243 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:44:26.243 DEBUG:boto:Port: 8001 2024-04-12T04:44:26.243 DEBUG:boto:Params: {} 2024-04-12T04:44:26.243 DEBUG:boto:Token: None 2024-04-12T04:44:26.244 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:44:26 GMT /dkyges-73/obj1 2024-04-12T04:44:26.244 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PlgkZS3j5M93bcoqIJvrfebr5Fc= 2024-04-12T04:44:26.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:44:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PlgkZS3j5M93bcoqIJvrfebr5Fc=', 'Content-Length': '0'} 2024-04-12T04:44:26.245 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:42:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-72:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9939.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006025e34a2e443c41-006618bc2a-286d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:44:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:44:26.286 DEBUG:boto:path=/dkyges-73/obj2 2024-04-12T04:44:26.287 DEBUG:boto:auth_path=/dkyges-73/obj2 2024-04-12T04:44:26.287 DEBUG:boto:Method: HEAD 2024-04-12T04:44:26.288 DEBUG:boto:Path: /dkyges-73/obj2 2024-04-12T04:44:26.288 DEBUG:boto:Data: 2024-04-12T04:44:26.288 DEBUG:boto:Headers: {} 2024-04-12T04:44:26.288 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:44:26.288 DEBUG:boto:Port: 8001 2024-04-12T04:44:26.288 DEBUG:boto:Params: {} 2024-04-12T04:44:26.288 DEBUG:boto:Token: None 2024-04-12T04:44:26.288 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:44:26 GMT /dkyges-73/obj2 2024-04-12T04:44:26.288 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Q+n8kSELfXjFXC4KMxdfuW/ldzU= 2024-04-12T04:44:26.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:44:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Q+n8kSELfXjFXC4KMxdfuW/ldzU=', 'Content-Length': '0'} 2024-04-12T04:44:26.290 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:42:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-72:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9939.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000175cb8b2b351565b-006618bc2a-286d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:44:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:44:26.291 DEBUG:boto:path=/dkyges-73/obj2 2024-04-12T04:44:26.291 DEBUG:boto:auth_path=/dkyges-73/obj2 2024-04-12T04:44:26.291 DEBUG:boto:Method: GET 2024-04-12T04:44:26.291 DEBUG:boto:Path: /dkyges-73/obj2 2024-04-12T04:44:26.291 DEBUG:boto:Data: 2024-04-12T04:44:26.291 DEBUG:boto:Headers: {} 2024-04-12T04:44:26.291 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:44:26.291 DEBUG:boto:Port: 8001 2024-04-12T04:44:26.291 DEBUG:boto:Params: {} 2024-04-12T04:44:26.291 DEBUG:boto:Token: None 2024-04-12T04:44:26.292 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:44:26 GMT /dkyges-73/obj2 2024-04-12T04:44:26.292 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:DBACc6mp+0sSO0pE2E26AHQrwiU= 2024-04-12T04:44:26.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:44:26 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DBACc6mp+0sSO0pE2E26AHQrwiU=', 'Content-Length': '0'} 2024-04-12T04:44:26.294 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:42:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-72:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9939.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bfa2f8a34a5a5b85-006618bc2a-286d-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:44:26 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:44:26.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:26.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:26.295 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:44:26.354 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:26.355 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:26.412 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:44:26.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:26.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:26.413 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:44:26.471 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:26.471 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:26.528 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:44:26.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:26.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:26.529 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:44:26.583 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:26.583 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:26.640 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:44:26.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:26.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:26.641 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:44:26.693 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:26.694 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:26.755 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:44:26.755 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:44:26.755 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 19, 2024-04-12T04:44:26.755 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T04:44:26.755 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T04:44:26.755 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T04:44:26.755 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:44:26.755 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T04:44:26.755 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.756 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T04:44:26.756 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T04:44:26.756 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:44:26.756 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T04:44:26.756 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:44:26.756 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:44:26.756 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:44:26.756 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:44:26.756 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:44:26.756 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:44:26.756 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:44:26.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.757 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:44:26.757 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T04:44:26.757 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:44:26.757 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T04:44:26.757 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:44:26.757 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:44:26.757 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:44:26.757 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:44:26.757 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:44:26.757 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:44:26.758 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:44:26.758 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:44:26.758 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:44:26.758 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:44:26.758 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:44:26.758 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:44:26.758 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:26.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.758 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:44:26.758 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T04:44:26.759 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:44:26.759 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:44:26.759 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:44:26.759 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:44:26.759 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:44:26.759 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:44:26.759 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:44:26.759 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:44:26.759 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:44:26.759 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:44:26.760 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:44:26.760 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:44:26.760 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:44:26.760 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:44:26.760 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:26.760 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.760 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:44:26.760 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:44:26.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.761 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:44:26.761 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:44:26.761 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:44:26.761 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:44:26.761 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:26.761 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.761 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:44:26.761 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:44:26.761 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:44:26.761 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:44:26.761 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [ 2024-04-12T04:44:26.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.762 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-group", 2024-04-12T04:44:26.762 INFO:teuthology.orchestra.run.smithi043.stdout: "data_flow": { 2024-04-12T04:44:26.762 INFO:teuthology.orchestra.run.smithi043.stdout: "symmetrical": [ 2024-04-12T04:44:26.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.762 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-flow1", 2024-04-12T04:44:26.762 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:44:26.762 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:44:26.762 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:44:26.762 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:26.763 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.763 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:26.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.763 INFO:teuthology.orchestra.run.smithi043.stdout: "pipes": [ 2024-04-12T04:44:26.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.763 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-pipe", 2024-04-12T04:44:26.763 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:44:26.763 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "*", 2024-04-12T04:44:26.763 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:44:26.763 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:44:26.763 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:44:26.763 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:26.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.764 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": { 2024-04-12T04:44:26.764 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "*", 2024-04-12T04:44:26.764 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:44:26.764 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:44:26.764 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:44:26.764 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:26.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.764 INFO:teuthology.orchestra.run.smithi043.stdout: "params": { 2024-04-12T04:44:26.764 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:44:26.764 INFO:teuthology.orchestra.run.smithi043.stdout: "filter": { 2024-04-12T04:44:26.765 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [] 2024-04-12T04:44:26.765 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.765 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": {}, 2024-04-12T04:44:26.765 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-12T04:44:26.765 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "system", 2024-04-12T04:44:26.765 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "" 2024-04-12T04:44:26.765 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.765 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.765 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:44:26.765 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "allowed" 2024-04-12T04:44:26.766 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.766 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:26.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.766 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:44:26.766 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:44:26.766 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:26.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.766 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T04:44:26.766 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T04:44:26.766 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:44:26.766 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T04:44:26.767 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:44:26.767 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:44:26.767 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:44:26.767 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:44:26.767 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:44:26.767 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:44:26.767 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:44:26.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.767 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:44:26.767 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T04:44:26.768 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:44:26.768 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:44:26.768 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:44:26.769 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:44:26.769 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:44:26.769 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:44:26.769 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:44:26.769 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:44:26.769 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:44:26.769 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:44:26.769 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:44:26.769 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:44:26.770 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:44:26.770 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:44:26.770 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:26.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.770 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:44:26.770 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T04:44:26.771 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:44:26.771 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T04:44:26.771 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:44:26.771 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:44:26.771 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:44:26.771 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:44:26.771 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:44:26.771 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:44:26.771 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:44:26.772 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:44:26.772 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:44:26.772 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:44:26.772 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:44:26.772 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:44:26.772 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:26.772 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.772 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:44:26.772 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:44:26.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.773 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:44:26.773 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:44:26.773 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:44:26.773 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:44:26.773 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:26.773 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.773 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:44:26.773 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:44:26.773 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:44:26.774 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:44:26.774 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T04:44:26.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.774 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:44:26.774 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:44:26.774 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:26.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.774 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:44:26.774 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T04:44:26.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.775 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:44:26.775 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T04:44:26.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.775 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:44:26.775 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T04:44:26.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.776 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:44:26.777 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T04:44:26.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:26.777 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:44:26.777 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T04:44:26.777 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.777 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:26.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.777 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T04:44:26.777 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:44:26.778 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T04:44:26.778 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T04:44:26.778 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:44:26.778 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:44:26.778 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:44:26.778 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:44:26.778 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:44:26.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.778 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T04:44:26.779 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:44:26.779 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:44:26.779 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:44:26.779 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:44:26.779 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:44:26.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.779 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T04:44:26.779 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:44:26.779 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:44:26.779 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:44:26.780 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:44:26.780 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:44:26.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.780 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T04:44:26.780 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:44:26.780 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:44:26.780 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:44:26.780 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:44:26.780 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:44:26.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.781 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T04:44:26.781 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:44:26.781 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:44:26.781 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:44:26.781 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:44:26.781 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:44:26.781 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:26.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:26.781 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:44:26.781 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T04:44:26.782 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:26.782 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:44:26.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:26.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:26.783 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:44:26.841 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:26.841 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:26.952 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:44:26.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:26.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:26.952 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dkyges-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:44:27.036 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:27.036 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:27.166 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:44:27.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:27.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:27.167 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket dkyges-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:44:27.263 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:27.263 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:27.396 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:44:27.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dkyges-72', '--source-bucket=*', '--dest-bucket=dkyges-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:27.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dkyges-72', '--source-bucket=*', '--dest-bucket=dkyges-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:27.397 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipeA --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=dkyges-72 '--source-bucket=*' --dest-bucket=dkyges-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:44:27.486 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:27.487 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:27.586 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"dkyges-73","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-12T04:44:27.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:27.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:27.587 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dkyges-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:44:27.660 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:27.660 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:27.736 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"dkyges-73","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:44:27.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:27.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:44:27.737 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dkyges-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:44:27.791 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:27.792 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:27.848 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"dkyges-73","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:44:27.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:27.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:27.849 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:44:27.883 INFO:tasks.rgw.c1.client.0.smithi043.stdout:2024-04-12T04:44:27.883+0000 7f1e272bd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:44:27.907 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:27.907 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:27.943 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:44:27.960 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:44:27.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:27.960 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:44:27.960 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:44:27.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:27.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:27.961 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:44:27.961 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:44:27.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:27.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:27.961 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:44:27.961 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:44:27.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:27.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:27.961 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897067.715675_960.1", 2024-04-12T04:44:27.962 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:44:27.715675Z" 2024-04-12T04:44:27.962 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:27.962 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:44:27.963 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:44:27.963 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:44:27.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:27.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:27.963 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:44:28.059 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:28.060 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:28.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:44:28.102+0000 7f061471e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:44:28.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:44:28.104+0000 7f061471e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:44:28.120 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:44:28.120 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:44:28.120 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:44:28.120 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:44:28.120 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:44:28.120 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:44:28.120 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:28.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:28.120 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:44:28.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:28.121 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:44:28.121 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:28.121 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:28.121 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:44:28.121 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:28.121 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:44:28.121 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:28.121 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:44:28.121 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:28.121 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:28.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:28.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:28.121 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:44:28.122 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:28.122 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:28.122 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:44:28.122 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:28.122 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:28.122 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:28.122 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:44:28.122 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:28.122 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:28.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:28.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:28.122 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:44:28.123 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:28.123 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:28.123 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:44:28.123 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:28.123 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:28.123 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:28.123 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:44:28.123 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:28.123 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:28.123 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:28.123 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:28.123 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:44:28.123 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:28.124 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:28.124 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:44:28.124 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:28.124 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:28.124 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:28.124 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:44:28.124 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:28.124 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:28.124 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:28.124 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:28.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:28.124 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:44:28.124 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:44:28.125 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:44:28.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:28.125 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:44:28Z" 2024-04-12T04:44:28.125 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:44:28.126 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:44:28Z" } 2024-04-12T04:44:28.126 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:44:28.126 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:44:28.126 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:44:28.126 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:44:33.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:33.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:33.128 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:44:33.188 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:33.188 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:33.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:44:33.225+0000 7f33b7832780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:44:33.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:44:33.227+0000 7f33b7832780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:44:33.245 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:44:33.245 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:44:33.245 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:44:33.246 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:44:33.246 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:44:33.246 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:44:33.246 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:33.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:33.246 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:44:33.246 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:33.246 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:44:33.246 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:33.246 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:33.246 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:44:33.247 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:33.247 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:44:33.247 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:33.247 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:44:33.247 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:33.247 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:33.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:33.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:33.247 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:44:33.247 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:33.247 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:33.247 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:44:33.248 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:33.248 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:33.248 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:33.248 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:44:33.248 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:33.248 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:33.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:33.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:33.248 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:44:33.248 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:33.248 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:33.249 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:44:33.249 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:33.249 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:33.249 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:33.249 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:44:33.249 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:33.249 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:33.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:33.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:33.249 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:44:33.249 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:33.250 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:33.250 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:44:33.250 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:33.250 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:33.250 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:33.250 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:44:33.250 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:33.250 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:33.250 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:33.251 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:33.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:33.251 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:44:33.251 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:44:33.251 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:44:33.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:33.251 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:44:33Z" 2024-04-12T04:44:33.251 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:44:33.252 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:44:33Z" } 2024-04-12T04:44:33.252 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:44:33.252 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:44:33.253 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:44:33.253 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:44:38.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:38.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:38.254 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:44:38.316 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:38.316 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:38.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:44:38.351+0000 7fb148baa780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:44:38.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:44:38.352+0000 7fb148baa780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:44:38.369 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:44:38.369 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:44:38.369 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:44:38.370 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:44:38.370 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:44:38.370 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:44:38.370 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:38.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:38.370 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:44:38.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:38.370 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:44:38.370 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:38.370 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:38.370 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:44:38.371 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:38.371 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:44:38.371 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:38.371 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:44:38.371 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:38.371 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:38.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:38.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:38.371 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:44:38.371 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:38.371 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:38.371 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:44:38.372 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:38.372 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:38.372 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:38.372 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:44:38.372 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:38.372 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:38.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:38.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:38.372 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:44:38.372 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:38.372 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:38.373 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:44:38.373 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:38.373 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:38.373 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:38.373 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:44:38.373 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:38.373 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:38.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:38.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:38.373 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:44:38.373 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:38.374 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:38.374 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:44:38.374 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:38.374 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:38.374 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:38.374 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:44:38.374 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:38.374 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:38.374 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:38.374 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:38.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:38.374 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:44:38.375 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:44:38.375 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:44:38.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:38.375 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:44:38Z" 2024-04-12T04:44:38.375 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:44:38.376 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:44:38Z" } 2024-04-12T04:44:38.376 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:44:38.376 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:44:38.376 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:44:38.376 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:44:43.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:43.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:43.378 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:44:43.439 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:43.439 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:43.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:44:43.475+0000 7fa092825780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:44:43.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:44:43.476+0000 7fa092825780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:44:43.494 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:44:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:44:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:44:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:44:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:44:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:44:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:44:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:44:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:44:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:44:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:44:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:44:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:44:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:44:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:44:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:44:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:44:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:44:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:44:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:44:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:44:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:44:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:44:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:43.500 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:44:43Z" 2024-04-12T04:44:43.500 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:44:43.501 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:44:43Z" } 2024-04-12T04:44:43.501 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:44:43.501 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:44:43.501 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:44:43.501 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:44:48.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:48.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:48.502 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:44:48.562 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:48.562 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:48.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:44:48.597+0000 7f68fee06780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:44:48.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:44:48.598+0000 7f68fee06780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:44:48.616 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:44:48.616 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:44:48.616 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:44:48.616 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:44:48.616 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:44:48.617 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:44:48.617 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:48.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:48.617 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:44:48.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:48.617 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:44:48.617 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:48.617 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:48.617 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:44:48.617 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:48.617 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:44:48.618 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:48.618 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:44:48.618 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:48.618 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:48.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:48.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:48.618 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:44:48.618 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:48.618 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:48.618 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:44:48.618 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:48.619 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:48.619 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:48.619 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:44:48.619 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:48.619 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:48.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:48.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:48.619 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:44:48.619 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:48.619 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:48.619 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:44:48.619 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:48.620 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:48.620 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:48.620 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:44:48.620 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:48.620 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:48.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:48.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:48.620 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:44:48.620 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:48.620 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:48.620 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:44:48.621 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:48.621 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:48.621 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:48.621 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:44:48.621 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:48.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:48.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:48.621 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:48.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:48.621 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:44:48.622 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:44:48.622 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:44:48.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:48.622 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:44:48Z" 2024-04-12T04:44:48.622 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:44:48.623 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:44:48Z" } 2024-04-12T04:44:48.623 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:44:48.623 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:44:48.623 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:44:48.624 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:44:53.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:53.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:53.625 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:44:53.685 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:53.685 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:53.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:44:53.723+0000 7f6dcefab780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:44:53.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:44:53.724+0000 7f6dcefab780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:44:53.742 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:44:53.742 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:44:53.742 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:44:53.742 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:44:53.742 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:44:53.742 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:44:53.742 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:53.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:53.743 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:44:53.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:53.743 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:44:53.743 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:53.743 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:53.743 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:44:53.743 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:53.743 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:44:53.743 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:53.743 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:44:53.743 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:53.743 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:53.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:53.744 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:53.744 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:44:53.744 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:53.744 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:53.744 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:44:53.744 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:53.744 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:53.744 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:53.744 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:44:53.744 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:53.744 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:53.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:53.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:53.745 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:44:53.745 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:53.745 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:53.745 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:44:53.745 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:53.745 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:53.745 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:53.745 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:44:53.745 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:53.745 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:53.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:53.746 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:53.746 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:44:53.746 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:53.746 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:53.746 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:44:53.746 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:53.746 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:53.746 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:53.746 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:44:53.746 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:53.746 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:53.747 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:53.747 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:53.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:53.747 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:44:53.747 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:44:53.747 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:44:53.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:53.747 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:44:53Z" 2024-04-12T04:44:53.747 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:44:53.748 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:44:53Z" } 2024-04-12T04:44:53.748 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:44:53.748 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:44:53.749 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:44:53.749 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:44:58.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:58.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:44:58.751 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:44:58.811 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:44:58.811 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:44:58.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:44:58.847+0000 7f59c8daa780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:44:58.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:44:58.848+0000 7f59c8daa780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:44:58.865 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:44:58.866 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:44:58.866 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:44:58.866 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:44:58.866 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:44:58.866 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:44:58.866 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:58.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:58.866 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:44:58.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:58.866 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:44:58.867 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:58.867 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:58.867 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:44:58.867 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:58.867 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:44:58.867 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:58.867 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:44:58.867 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:58.867 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:58.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:58.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:58.868 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:44:58.868 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:58.868 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:58.868 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:44:58.868 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:58.868 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:58.868 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:58.868 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:44:58.868 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:58.868 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:58.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:58.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:58.869 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:44:58.869 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:58.869 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:58.869 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:44:58.869 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:58.869 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:58.869 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:58.869 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:44:58.869 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:58.870 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:58.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:58.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:44:58.870 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:44:58.870 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:44:58.870 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:44:58.870 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:44:58.870 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:44:58.870 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:44:58.870 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:44:58.870 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:44:58.871 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:44:58.871 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:58.871 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:44:58.871 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:44:58.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:58.871 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:44:58.871 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:44:58.871 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:44:58.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:44:58.871 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:44:58Z" 2024-04-12T04:44:58.871 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:44:58.872 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:44:58Z" } 2024-04-12T04:44:58.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:44:58.873 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:44:58.873 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:44:58.873 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:45:03.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:03.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:03.874 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:45:03.935 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:45:03.935 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:45:03.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:03.970+0000 7f471b511780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:03.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:03.971+0000 7f471b511780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:03.989 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:45:03.990 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:45:03.990 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:45:03.990 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:45:03.990 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:45:03.990 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:45:03.990 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:03.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:03.990 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:45:03.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:03.991 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:45:03.991 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:03.991 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:03.991 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:45:03.991 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:03.991 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:45:03.991 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:03.991 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:45:03.991 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:03.991 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:03.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:03.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:03.992 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:45:03.992 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:03.992 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:03.992 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:45:03.992 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:03.992 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:03.992 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:03.992 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:45:03.992 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:03.992 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:03.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:03.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:03.993 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:45:03.993 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:03.993 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:03.993 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:45:03.993 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:03.993 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:03.993 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:03.993 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:45:03.993 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:03.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:03.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:03.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:03.994 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:45:03.994 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:03.994 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:03.994 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:45:03.994 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:03.994 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:03.994 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:03.994 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:45:03.994 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:03.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:03.995 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:03.995 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:45:03.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:03.995 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:45:03.995 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:45:03.995 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:45:03.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:03.995 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:45:03Z" 2024-04-12T04:45:03.995 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:45:03.996 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:45:03Z" } 2024-04-12T04:45:03.996 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:45:03.997 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:45:03.997 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:45:03.997 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:45:08.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:08.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:08.999 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:45:09.058 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:45:09.058 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:45:09.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:09.094+0000 7ff26f485780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:09.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:09.095+0000 7ff26f485780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:09.113 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:45:09.113 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:45:09.114 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:45:09.114 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:45:09.114 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:45:09.114 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:45:09.114 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:09.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:09.114 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:45:09.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:09.115 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:45:09.115 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:09.115 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:09.115 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:45:09.115 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:09.115 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:45:09.115 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:09.115 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:45:09.115 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:09.115 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:09.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:09.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:09.116 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:45:09.116 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:09.116 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:09.116 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:45:09.116 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:09.116 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:09.116 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:09.116 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:45:09.116 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:09.116 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:09.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:09.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:09.116 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:45:09.117 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:09.117 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:09.117 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:45:09.117 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:09.117 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:09.117 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:09.117 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:45:09.117 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:09.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:09.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:09.117 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:09.117 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:45:09.118 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:09.118 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:09.118 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:45:09.118 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:09.118 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:09.118 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:09.118 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:45:09.118 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:09.118 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:09.118 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:09.118 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:45:09.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:09.119 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:45:09.119 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:45:09.119 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:45:09.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:09.119 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:45:09Z" 2024-04-12T04:45:09.119 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:45:09.120 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:45:09Z" } 2024-04-12T04:45:09.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:45:09.120 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:45:09.120 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:45:09.120 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:45:14.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:14.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:14.122 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:45:14.181 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:45:14.181 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:45:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:14.219+0000 7f5b56161780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:14.220+0000 7f5b56161780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:14.238 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:45:14.238 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:45:14.238 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:45:14.238 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:45:14.238 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:45:14.238 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:45:14.238 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:14.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:45:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:45:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:45:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:45:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:45:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:45:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:45:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:45:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:45:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:45:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:45:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:45:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:45:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:45:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:45:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:45:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:45:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:45:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:45:14Z" 2024-04-12T04:45:14.244 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:45:14.245 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:45:14Z" } 2024-04-12T04:45:14.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:45:14.245 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:45:14.245 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:45:14.245 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:45:19.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:19.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:19.250 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:45:19.315 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:45:19.315 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:45:19.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:19.350+0000 7f95153fa780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:19.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:19.351+0000 7f95153fa780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:19.369 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:45:19.369 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:45:19.369 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:45:19.370 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:45:19.370 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:45:19.370 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:45:19.370 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:19.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:19.370 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:45:19.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:19.370 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:45:19.370 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:19.370 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:19.371 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:45:19.371 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:19.371 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:45:19.371 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:19.371 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:45:19.371 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:19.371 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:19.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:19.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:19.371 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:45:19.371 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:19.371 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:19.371 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:45:19.372 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:19.372 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:19.372 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:19.372 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:45:19.372 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:19.372 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:19.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:19.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:19.372 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:45:19.372 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:19.372 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:19.372 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:45:19.373 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:19.373 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:19.373 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:19.373 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:45:19.373 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:19.373 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:19.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:19.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:19.373 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:45:19.373 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:19.373 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:19.373 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:45:19.373 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:19.374 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:19.374 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:19.374 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:45:19.374 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:19.374 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:19.374 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:19.374 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:45:19.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:19.374 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:45:19.374 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:45:19.374 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:45:19.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:19.375 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:45:19Z" 2024-04-12T04:45:19.375 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:45:19.376 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:45:19Z" } 2024-04-12T04:45:19.376 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:45:19.376 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:45:19.376 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:45:19.376 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:45:24.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:24.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:24.378 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:45:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:45:24.439 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:45:24.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:24.474+0000 7f73e2ee0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:24.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:24.475+0000 7f73e2ee0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:24.492 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:45:24.493 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:45:24.493 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:45:24.493 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:45:24.493 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:45:24.493 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:45:24.493 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:24.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:24.493 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:45:24.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:24.493 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:45:24.494 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:24.494 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:24.494 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:45:24.494 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:24.494 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:45:24.494 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:24.494 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:45:24.494 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:24.494 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:24.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:24.495 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:24.495 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:45:24.495 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:24.495 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:24.495 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:45:24.495 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:24.495 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:24.495 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:24.495 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:45:24.495 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:24.496 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:24.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:24.496 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:24.496 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:45:24.496 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:24.496 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:24.496 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:45:24.496 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:24.496 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:24.496 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:24.496 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:45:24.497 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:24.497 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:24.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:24.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:24.497 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:45:24.497 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:24.497 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:24.497 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:45:24.497 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:24.497 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:24.497 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:24.497 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:45:24.498 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:24.498 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:24.498 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:24.498 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:45:24.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:24.498 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:45:24.498 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:45:24.498 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:45:24.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:24.498 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:45:24Z" 2024-04-12T04:45:24.499 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:45:24.500 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:45:24Z" } 2024-04-12T04:45:24.500 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:45:24.500 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:45:24.500 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:45:24.500 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:45:29.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:29.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:29.502 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:45:29.562 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:45:29.562 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:45:29.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:29.599+0000 7fe217749780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:29.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:29.600+0000 7fe217749780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:29.619 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:45:29.619 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:45:29.619 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:45:29.619 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:45:29.620 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:45:29.620 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:45:29.620 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:29.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:29.620 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:45:29.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:29.620 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:45:29.620 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:29.620 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:29.620 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:45:29.620 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:29.621 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:45:29.621 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:29.621 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:45:29.621 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:29.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:29.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:29.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:29.621 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:45:29.621 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:29.621 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:29.622 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:45:29.622 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:29.622 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:29.622 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:29.622 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:45:29.622 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:29.622 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:29.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:29.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:29.623 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:45:29.623 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:29.623 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:29.623 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:45:29.623 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:29.623 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:29.623 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:29.624 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:45:29.624 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:29.624 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:29.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:29.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:29.624 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:45:29.624 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:29.624 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:29.624 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:45:29.624 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:29.624 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:29.624 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:29.625 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:45:29.625 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:29.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:29.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:29.625 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:45:29.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:29.625 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:45:29.625 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:45:29.625 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:45:29.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:29.625 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:45:29Z" 2024-04-12T04:45:29.625 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:45:29.626 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:45:29Z" } 2024-04-12T04:45:29.627 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:45:29.627 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:45:29.627 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:45:29.627 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:45:34.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:34.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:34.628 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:45:34.688 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:45:34.688 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:45:34.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:34.725+0000 7f4e330ca780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:34.726+0000 7f4e330ca780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:34.744 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:45:34.744 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:45:34.744 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:45:34.744 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:45:34.744 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:45:34.745 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:45:34.745 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:34.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:34.745 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:45:34.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:34.745 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:45:34.745 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:34.745 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:34.745 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:45:34.745 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:34.745 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:45:34.745 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:34.746 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:45:34.746 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:34.746 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:34.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:34.746 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:34.746 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:45:34.746 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:34.746 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:34.746 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:45:34.746 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:34.746 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:34.747 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:34.747 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:45:34.747 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:34.747 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:34.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:34.747 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:34.747 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:45:34.747 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:34.747 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:34.747 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:45:34.747 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:34.748 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:34.748 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:34.748 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:45:34.748 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:34.748 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:34.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:34.748 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:34.748 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:45:34.748 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:34.748 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:34.748 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:45:34.749 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:34.749 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:34.749 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:34.749 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:45:34.749 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:34.749 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:34.749 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:34.749 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:45:34.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:34.749 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:45:34.749 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:45:34.750 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:45:34.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:34.750 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:45:34Z" 2024-04-12T04:45:34.750 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:45:34.751 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:45:34Z" } 2024-04-12T04:45:34.751 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:45:34.751 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:45:34.752 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:45:34.752 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:45:39.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:39.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:39.753 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:45:39.811 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:45:39.811 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:45:39.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:39.847+0000 7f2e0adfe780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:39.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:39.848+0000 7f2e0adfe780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:39.866 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:45:39.866 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:45:39.866 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:45:39.867 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:45:39.867 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:45:39.867 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:45:39.867 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:39.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:39.867 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:45:39.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:39.868 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:45:39.868 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:39.868 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:39.868 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:45:39.868 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:39.868 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:45:39.868 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:39.868 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:45:39.868 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:39.868 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:39.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:39.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:39.869 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:45:39.869 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:39.869 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:39.869 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:45:39.869 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:39.869 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:39.869 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:39.870 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:45:39.870 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:39.870 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:39.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:39.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:39.870 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:45:39.871 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:39.871 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:39.871 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:45:39.871 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:39.871 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:39.871 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:39.871 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:45:39.871 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:39.871 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:39.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:39.871 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:39.871 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:45:39.872 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:39.872 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:39.872 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:45:39.872 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:39.872 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:39.872 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:39.872 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:45:39.872 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:39.872 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:39.872 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:39.872 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:45:39.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:39.873 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:45:39.873 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:45:39.873 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:45:39.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:39.873 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:45:39Z" 2024-04-12T04:45:39.873 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:45:39.874 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:45:39Z" } 2024-04-12T04:45:39.874 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:45:39.874 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:45:39.874 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:45:39.875 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:45:44.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:44.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:44.876 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:45:44.936 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:45:44.936 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:45:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:44.972+0000 7f5130f20780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:44.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:44.973+0000 7f5130f20780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:44.991 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:45:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:45:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:45:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:45:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:45:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:45:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:45:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:45:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:45:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:45:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:45:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:45:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:45:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:45:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:45:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:45:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:45:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:45:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:45:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:45:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:45:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:45:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:45:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:45:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:45:44Z" 2024-04-12T04:45:44.997 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:45:44.998 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:45:44Z" } 2024-04-12T04:45:44.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:45:44.998 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:45:44.998 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:45:44.998 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:45:49.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:49.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:50.000 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:45:50.058 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:45:50.059 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:45:50.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:50.094+0000 7f3a5b615780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:50.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:50.095+0000 7f3a5b615780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:50.113 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:45:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:45:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:45:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:45:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:45:50.113 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:45:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:45:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:45:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:45:50.114 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:45:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:45:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:45:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:50.115 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:45:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:45:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:45:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:50.116 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:45:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:45:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:45:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:50.117 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:45:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:45:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:45:50.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:45:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:45:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:45:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:50.119 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:45:50Z" 2024-04-12T04:45:50.119 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:45:50.120 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:45:50Z" } 2024-04-12T04:45:50.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:45:50.120 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:45:50.120 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:45:50.121 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:45:55.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:55.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:45:55.122 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:45:55.182 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:45:55.182 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:45:55.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:55.219+0000 7faa2ebb8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:55.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:45:55.220+0000 7faa2ebb8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:45:55.238 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:45:55.238 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:45:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:45:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:45:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:45:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:45:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:45:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:45:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:55.239 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:45:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:45:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:45:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:45:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:55.240 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:45:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:45:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:45:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:55.241 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:45:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:45:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:55.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:45:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:45:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:45:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:45:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:45:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:45:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:45:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:45:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:45:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:45:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:45:55.243 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:45:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:45:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:45:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:45:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:45:55.244 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:45:55Z" 2024-04-12T04:45:55.244 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:45:55.245 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:45:55Z" } 2024-04-12T04:45:55.245 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:45:55.245 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:45:55.245 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:45:55.246 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:46:00.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:00.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:00.247 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:00.308 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:00.308 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:00.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:00.343+0000 7f93f91fe780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:00.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:00.344+0000 7f93f91fe780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:00.362 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:46:00.363 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:46:00.363 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:46:00.363 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:46:00.363 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:46:00.363 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:46:00.363 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:00.363 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:00.363 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:46:00.363 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:00.363 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:46:00.364 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:00.364 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:00.364 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:46:00.364 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:00.364 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:46:00.364 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:00.364 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:46:00.364 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:00.364 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:00.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:00.364 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:00.365 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:46:00.365 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:00.365 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:00.365 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:46:00.365 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:00.365 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:00.365 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:00.365 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:46:00.365 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:00.365 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:00.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:00.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:00.366 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:46:00.366 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:00.366 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:00.366 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:46:00.366 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:00.366 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:00.366 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:00.366 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:46:00.367 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:00.367 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:00.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:00.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:00.367 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:46:00.367 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:00.367 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:00.367 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:46:00.367 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:00.368 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:00.368 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:00.368 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:46:00.368 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:00.368 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:00.368 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:00.368 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:46:00.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:00.368 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:46:00.369 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:46:00.369 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:46:00.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:00.369 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:46:00Z" 2024-04-12T04:46:00.369 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:46:00.370 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:46:00Z" } 2024-04-12T04:46:00.370 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:46:00.370 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:46:00.370 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:46:00.370 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:46:05.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:05.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:05.372 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:05.434 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:05.434 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:05.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:05.469+0000 7fd717bff780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:05.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:05.471+0000 7fd717bff780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:05.488 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:46:05.489 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:46:05.489 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:46:05.489 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:46:05.489 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:46:05.489 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:46:05.489 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:05.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:05.489 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:46:05.489 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:05.489 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:46:05.490 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:05.490 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:05.490 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:46:05.490 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:05.490 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:46:05.490 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:05.490 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:46:05.490 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:05.490 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:05.490 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:05.491 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:05.491 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:46:05.491 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:05.491 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:05.491 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:46:05.491 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:05.491 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:05.491 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:05.491 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:46:05.491 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:05.491 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:05.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:05.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:05.492 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:46:05.492 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:05.492 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:05.492 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:46:05.492 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:05.492 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:05.492 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:05.492 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:46:05.492 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:05.492 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:05.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:05.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:05.493 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:46:05.493 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:05.493 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:05.493 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:46:05.493 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:05.493 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:05.493 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:05.493 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:46:05.493 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:05.494 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:05.494 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:05.494 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:46:05.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:05.494 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:46:05.494 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:46:05.494 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:46:05.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:05.494 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:46:05Z" 2024-04-12T04:46:05.494 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:46:05.495 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:46:05Z" } 2024-04-12T04:46:05.495 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:46:05.496 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:46:05.496 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:46:05.496 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:46:10.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:10.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:10.497 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:10.557 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:10.557 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:10.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:10.592+0000 7f0dc28a2780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:10.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:10.593+0000 7f0dc28a2780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:10.611 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:46:10.611 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:46:10.611 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:46:10.612 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:46:10.612 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:46:10.612 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:46:10.612 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:10.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:10.612 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:46:10.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:10.612 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:46:10.612 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:10.612 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:10.612 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:46:10.613 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:10.613 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:46:10.613 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:10.613 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:46:10.613 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:10.613 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:10.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:10.613 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:10.613 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:46:10.613 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:10.613 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:10.614 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:46:10.614 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:10.614 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:10.614 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:10.614 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:46:10.614 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:10.614 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:10.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:10.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:10.614 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:46:10.615 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:10.615 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:10.615 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:46:10.615 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:10.615 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:10.615 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:10.615 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:46:10.615 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:10.615 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:10.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:10.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:10.616 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:46:10.616 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:10.616 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:10.616 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:46:10.616 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:10.616 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:10.616 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:10.616 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:46:10.616 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:10.616 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:10.617 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:10.617 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:46:10.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:10.617 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:46:10.617 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:46:10.617 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:46:10.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:10.617 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:46:10Z" 2024-04-12T04:46:10.617 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:46:10.618 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:46:10Z" } 2024-04-12T04:46:10.618 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:46:10.619 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:46:10.619 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:46:10.619 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:46:15.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:15.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:15.620 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:15.680 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:15.680 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:15.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:15.719+0000 7f63b5c2d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:15.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:15.720+0000 7f63b5c2d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:15.738 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:46:15.738 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:46:15.739 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:46:15.739 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:46:15.739 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:46:15.739 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:46:15.739 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:15.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:15.739 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:46:15.739 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:15.739 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:46:15.739 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:15.739 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:15.740 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:46:15.740 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:15.740 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:46:15.740 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:15.740 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:46:15.740 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:15.740 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:15.740 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:15.740 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:15.740 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:46:15.740 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:15.741 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:15.741 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:46:15.741 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:15.741 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:15.741 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:15.741 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:46:15.741 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:15.741 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:15.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:15.741 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:15.742 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:46:15.742 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:15.742 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:15.742 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:46:15.742 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:15.742 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:15.742 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:15.742 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:46:15.742 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:15.743 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:15.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:15.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:15.743 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:46:15.743 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:15.743 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:15.743 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:46:15.743 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:15.743 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:15.743 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:15.743 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:46:15.744 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:15.744 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:15.744 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:15.744 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:46:15.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:15.744 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:46:15.744 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:46:15.744 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:46:15.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:15.744 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:46:15Z" 2024-04-12T04:46:15.744 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:46:15.745 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:46:15Z" } 2024-04-12T04:46:15.746 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:46:15.746 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:46:15.746 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:46:15.746 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:46:20.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:20.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:20.748 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:20.807 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:20.807 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:20.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:20.843+0000 7f91eb244780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:20.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:20.844+0000 7f91eb244780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:20.862 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:46:20.862 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:46:20.862 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:46:20.862 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:46:20.862 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:46:20.863 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:46:20.863 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:20.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:20.863 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:46:20.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:20.863 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:46:20.863 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:20.863 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:20.863 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:46:20.863 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:20.863 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:46:20.864 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:20.864 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:46:20.864 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:20.864 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:20.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:20.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:20.864 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:46:20.864 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:20.864 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:20.864 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:46:20.865 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:20.865 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:20.865 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:20.865 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:46:20.865 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:20.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:20.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:20.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:20.865 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:46:20.865 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:20.865 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:20.866 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:46:20.866 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:20.866 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:20.866 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:20.866 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:46:20.866 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:20.866 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:20.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:20.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:20.866 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:46:20.867 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:20.867 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:20.867 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:46:20.867 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:20.867 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:20.867 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:20.867 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:46:20.867 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:20.867 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:20.867 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:20.867 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:46:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:46:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:46:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:46:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:20.868 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:46:20Z" 2024-04-12T04:46:20.868 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:46:20.869 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:46:20Z" } 2024-04-12T04:46:20.869 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:46:20.869 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:46:20.869 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:46:20.870 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:46:25.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:25.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:25.871 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:25.931 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:25.931 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:25.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:25.968+0000 7fd1b7306780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:25.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:25.969+0000 7fd1b7306780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:25.988 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:46:25.988 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:46:25.988 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:46:25.988 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:46:25.988 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:46:25.988 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:46:25.989 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:25.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:25.989 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:46:25.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:25.989 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:46:25.989 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:25.989 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:25.989 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:46:25.989 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:25.990 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:46:25.990 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:25.990 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:46:25.990 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:25.990 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:25.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:25.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:25.990 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:46:25.991 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:25.991 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:25.991 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:46:25.991 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:25.991 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:25.991 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:25.991 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:46:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:46:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:46:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:25.992 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:46:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:46:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712896940.059773_934.1", 2024-04-12T04:46:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:25.993 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:20.059773Z", 2024-04-12T04:46:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:46:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:46:25.994 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:46:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:46:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:25.995 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:46:25Z" 2024-04-12T04:46:25.995 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:46:25.996 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712896940.059773_934.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:20.059773Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:46:25Z" } 2024-04-12T04:46:25.997 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896940.059773_934.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:20.059773Z', 'realm_epoch': 2}}] 2024-04-12T04:46:25.997 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:46:25.997 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896940.059773_934.1'} 2024-04-12T04:46:25.997 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897067.715675_960.1 target=1_1712896940.059773_934.1 2024-04-12T04:46:28.095 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T04:46:28.093+0000 7fd539413700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:46:30.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:30.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:30.998 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:31.057 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:31.057 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:31.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:31.093+0000 7f782c676780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:31.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:31.095+0000 7f782c676780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:31.113 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:46:31.113 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:46:31.113 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:46:31.113 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:46:31.113 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:46:31.113 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:46:31.113 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:31.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.113 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:46:31.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.114 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:46:31.114 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:31.114 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:31.114 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:46:31.114 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:31.114 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:46:31.114 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:31.114 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:46:31.114 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:31.115 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.115 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:46:31.115 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:31.115 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:31.115 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:46:31.115 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:31.115 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:31.115 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:31.115 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:46:31.115 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:31.116 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.116 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:46:31.116 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:31.116 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:31.116 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:46:31.116 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:31.116 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:31.116 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:31.116 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:46:31.117 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:31.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.117 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.117 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:46:31.117 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:31.117 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:31.117 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897067.715675_960.1", 2024-04-12T04:46:31.117 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:31.118 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:31.118 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:31.118 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:44:27.715675Z", 2024-04-12T04:46:31.118 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:31.118 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.118 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.118 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:46:31.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.119 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:46:31.119 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:46:31.119 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:46:31.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.119 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:46:31Z" 2024-04-12T04:46:31.119 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:46:31.120 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897067.715675_960.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:44:27.715675Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:46:31Z" } 2024-04-12T04:46:31.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897067.715675_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:44:27.715675Z', 'realm_epoch': 2}}] 2024-04-12T04:46:31.120 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:46:31.120 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:46:31.121 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:46:31.121 DEBUG:boto:path=/dkyges-72/ 2024-04-12T04:46:31.121 DEBUG:boto:auth_path=/dkyges-72/ 2024-04-12T04:46:31.121 DEBUG:boto:Method: HEAD 2024-04-12T04:46:31.121 DEBUG:boto:Path: /dkyges-72/ 2024-04-12T04:46:31.121 DEBUG:boto:Data: 2024-04-12T04:46:31.121 DEBUG:boto:Headers: {} 2024-04-12T04:46:31.122 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:46:31.122 DEBUG:boto:Port: 8000 2024-04-12T04:46:31.122 DEBUG:boto:Params: {} 2024-04-12T04:46:31.122 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:46:31.122 DEBUG:boto:Token: None 2024-04-12T04:46:31.123 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:46:31 GMT /dkyges-72/ 2024-04-12T04:46:31.123 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qfU+cXq05vwVs1KssfhEm3vJ8L4= 2024-04-12T04:46:31.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:46:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qfU+cXq05vwVs1KssfhEm3vJ8L4=', 'Content-Length': '0'} 2024-04-12T04:46:31.157 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003843fca98df28281-006618bca7-2977-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:46:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:46:31.158 DEBUG:boto:path=/dkyges-72/obj5 2024-04-12T04:46:31.158 DEBUG:boto:auth_path=/dkyges-72/obj5 2024-04-12T04:46:31.158 DEBUG:boto:Method: PUT 2024-04-12T04:46:31.158 DEBUG:boto:Path: /dkyges-72/obj5 2024-04-12T04:46:31.158 DEBUG:boto:Data: 2024-04-12T04:46:31.158 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:46:31.158 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:46:31.158 DEBUG:boto:Port: 8000 2024-04-12T04:46:31.159 DEBUG:boto:Params: {} 2024-04-12T04:46:31.159 DEBUG:boto:Token: None 2024-04-12T04:46:31.159 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:46:31 GMT /dkyges-72/obj5 2024-04-12T04:46:31.159 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BlDYMiBWe3FzKEuIl7SItJld4Nc= 2024-04-12T04:46:31.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:46:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BlDYMiBWe3FzKEuIl7SItJld4Nc='} 2024-04-12T04:46:31.167 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000032bd5b75108a0629-006618bca7-2977-a1'), ('Date', 'Fri, 12 Apr 2024 04:46:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:46:31.168 DEBUG:boto:path=/dkyges-72/ 2024-04-12T04:46:31.168 DEBUG:boto:auth_path=/dkyges-72/ 2024-04-12T04:46:31.168 DEBUG:boto:Method: HEAD 2024-04-12T04:46:31.168 DEBUG:boto:Path: /dkyges-72/ 2024-04-12T04:46:31.168 DEBUG:boto:Data: 2024-04-12T04:46:31.168 DEBUG:boto:Headers: {} 2024-04-12T04:46:31.168 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:46:31.168 DEBUG:boto:Port: 8000 2024-04-12T04:46:31.168 DEBUG:boto:Params: {} 2024-04-12T04:46:31.168 DEBUG:boto:Token: None 2024-04-12T04:46:31.169 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:46:31 GMT /dkyges-72/ 2024-04-12T04:46:31.169 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qfU+cXq05vwVs1KssfhEm3vJ8L4= 2024-04-12T04:46:31.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:46:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qfU+cXq05vwVs1KssfhEm3vJ8L4=', 'Content-Length': '0'} 2024-04-12T04:46:31.171 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095e824c81fef5f1e-006618bca7-2977-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:46:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:46:31.172 DEBUG:boto:path=/dkyges-72/obj6 2024-04-12T04:46:31.172 DEBUG:boto:auth_path=/dkyges-72/obj6 2024-04-12T04:46:31.172 DEBUG:boto:Method: PUT 2024-04-12T04:46:31.172 DEBUG:boto:Path: /dkyges-72/obj6 2024-04-12T04:46:31.172 DEBUG:boto:Data: 2024-04-12T04:46:31.172 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:46:31.172 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:46:31.172 DEBUG:boto:Port: 8000 2024-04-12T04:46:31.172 DEBUG:boto:Params: {} 2024-04-12T04:46:31.172 DEBUG:boto:Token: None 2024-04-12T04:46:31.173 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:46:31 GMT /dkyges-72/obj6 2024-04-12T04:46:31.173 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Il2nxilT+SNe1KaYBUEatc9qFC8= 2024-04-12T04:46:31.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:46:31 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Il2nxilT+SNe1KaYBUEatc9qFC8='} 2024-04-12T04:46:31.214 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e4b848986a75f07-006618bca7-2977-a1'), ('Date', 'Fri, 12 Apr 2024 04:46:31 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:46:31.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:31.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:31.215 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:31.274 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:31.274 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:46:31.335 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:46:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:46:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:46:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:46:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:46:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:46:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:46:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897067.715675_960.1", 2024-04-12T04:46:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:44:27.715675Z" 2024-04-12T04:46:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.337 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:46:31.338 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:46:31.338 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:46:31.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:31.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:31.339 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:31.395 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:31.396 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:31.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:31.429+0000 7f4d12314780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:31.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:31.430+0000 7f4d12314780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:31.447 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:46:31.448 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:46:31.448 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:46:31.448 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:46:31.448 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:46:31.448 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:46:31.448 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:31.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.448 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:46:31.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.448 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:46:31.449 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:31.449 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:31.449 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:46:31.449 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:31.449 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:46:31.449 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:31.449 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:46:31.449 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:31.449 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.449 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:46:31.450 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:31.450 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:31.450 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:46:31.450 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:31.450 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:31.450 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:31.450 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:46:31.450 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:31.450 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.451 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:46:31.451 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:31.451 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:31.451 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:46:31.451 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:31.451 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:31.451 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:31.451 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:46:31.451 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:31.451 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.452 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:46:31.452 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:31.452 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:31.452 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897067.715675_960.1", 2024-04-12T04:46:31.452 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:31.452 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:31.452 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:31.452 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:44:27.715675Z", 2024-04-12T04:46:31.452 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:31.452 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.453 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.453 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:46:31.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.453 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:46:31.453 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:46:31.453 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:46:31.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.453 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:46:31Z" 2024-04-12T04:46:31.453 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:46:31.454 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897067.715675_960.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:44:27.715675Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:46:31Z" } 2024-04-12T04:46:31.454 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897067.715675_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:44:27.715675Z', 'realm_epoch': 2}}] 2024-04-12T04:46:31.455 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:46:31.455 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:46:31.455 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:46:31.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:31.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:31.455 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:31.568 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:46:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:46:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:46:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:46:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:46:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:46:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:46:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:46:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:46:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.570 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:46:31.572 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:46:31.572 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:46:31.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:31.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:31.572 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:31.629 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:31.630 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:31.685 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:46:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:46:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:46:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:46:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:46:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:46:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:46:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:46:31.686 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047778", 2024-04-12T04:46:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:21.389310Z" 2024-04-12T04:46:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:46:31.687 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:46:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:46:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.688 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:46:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:46:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:46:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:21.352708Z" 2024-04-12T04:46:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:46:31.689 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:46:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:46:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:46:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:46:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.690 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:46:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:46:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:46:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:31.691 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:46:31Z" 2024-04-12T04:46:31.691 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:46:31.692 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047778", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:21.389310Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:42:21.352708Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:46:31Z" } 2024-04-12T04:46:31.692 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047778', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:21.389310Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:42:21.352708Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:46:31.692 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:46:31.692 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:46:31.692 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047918 target=00000000000000000000:00000000000000047778, shard=2 master=00000000000000000000:00000000000000054630 target=00000000000000000000:00000000000000054490 2024-04-12T04:46:36.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:36.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:36.694 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:36.755 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:36.756 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:36.811 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:46:36.811 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:46:36.811 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:46:36.811 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:46:36.812 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:46:36.812 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:46:36.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:36.812 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:46:36.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:36.812 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:46:36.812 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:36.812 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:36.812 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047778", 2024-04-12T04:46:36.812 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:36.813 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:36.813 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:36.813 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:21.389310Z" 2024-04-12T04:46:36.813 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:36.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:36.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:36.813 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:46:36.813 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:36.813 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:36.813 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:46:36.813 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:36.813 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:36.814 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:36.814 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:46:36.814 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:36.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:36.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:36.814 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:46:36.814 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:36.814 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:36.814 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:46:36.815 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:36.815 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:46:36.815 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:36.815 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:21.352708Z" 2024-04-12T04:46:36.815 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:36.816 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:36.816 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:36.816 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:46:36.816 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:36.816 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:36.816 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:46:36.816 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:36.816 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:46:36.816 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:36.816 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:46:36.816 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:36.816 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:36.817 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:46:36.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:36.817 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:46:36.817 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:46:36.817 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:46:36.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:36.817 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:46:36Z" 2024-04-12T04:46:36.817 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:46:36.818 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047778", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:21.389310Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:42:21.352708Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:46:36Z" } 2024-04-12T04:46:36.818 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047778', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:21.389310Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:42:21.352708Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:46:36.818 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:46:36.819 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:46:36.819 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047918 target=00000000000000000000:00000000000000047778, shard=2 master=00000000000000000000:00000000000000054630 target=00000000000000000000:00000000000000054490 2024-04-12T04:46:41.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:41.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:41.820 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:41.882 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:41.882 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:41.943 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:46:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:46:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:46:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:46:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:46:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:46:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:46:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:46:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047778", 2024-04-12T04:46:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:21.389310Z" 2024-04-12T04:46:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:46:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:46:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:46:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:46:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:46:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:46:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:21.352708Z" 2024-04-12T04:46:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:46:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:46:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:46:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:46:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:46:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:46:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:46:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:46:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:41.949 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:46:41Z" 2024-04-12T04:46:41.949 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:46:41.949 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047778", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:21.389310Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:42:21.352708Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:46:41Z" } 2024-04-12T04:46:41.949 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047778', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:21.389310Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:42:21.352708Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:46:41.949 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:46:41.949 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:46:41.949 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047918 target=00000000000000000000:00000000000000047778, shard=2 master=00000000000000000000:00000000000000054630 target=00000000000000000000:00000000000000054490 2024-04-12T04:46:46.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:46.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:46.951 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:47.013 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:47.013 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:47.070 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:46:47.070 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:46:47.071 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:46:47.071 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:46:47.071 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:46:47.071 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:46:47.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:47.071 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:46:47.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:47.071 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:46:47.071 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:47.071 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:47.071 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047778", 2024-04-12T04:46:47.072 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:47.072 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:47.072 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:47.072 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:21.389310Z" 2024-04-12T04:46:47.072 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:47.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:47.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:47.072 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:46:47.072 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:47.072 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:47.072 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:46:47.073 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:47.073 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:47.073 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:47.073 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:46:47.073 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:47.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:47.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:47.073 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:46:47.073 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:47.073 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:47.073 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054490", 2024-04-12T04:46:47.074 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:47.074 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:46:47.074 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:47.074 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:42:21.352708Z" 2024-04-12T04:46:47.074 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:47.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:47.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:47.074 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:46:47.074 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:47.074 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:47.074 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:46:47.075 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:47.075 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:46:47.075 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:47.075 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:46:47.075 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:47.075 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:47.075 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:46:47.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:47.075 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:46:47.075 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:46:47.075 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:46:47.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:47.076 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:46:47Z" 2024-04-12T04:46:47.076 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:46:47.076 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047778", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:42:21.389310Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054490", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:42:21.352708Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:46:47Z" } 2024-04-12T04:46:47.076 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047778', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:42:21.389310Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054490', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:42:21.352708Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:46:47.076 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:46:47.076 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047778', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054490', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:46:47.076 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000047918 target=00000000000000000000:00000000000000047778, shard=2 master=00000000000000000000:00000000000000054630 target=00000000000000000000:00000000000000054490 2024-04-12T04:46:52.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:52.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:52.077 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:52.139 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:52.139 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:52.194 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:46:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:46:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:46:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:46:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:46:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:46:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:46:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:52.195 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:46:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:46:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:46:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:52.196 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:46:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:46:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:46:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:52.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:46:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:46:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:46:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:46:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:52.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:46:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:46:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046574", 2024-04-12T04:46:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:46:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:37:49.979474Z" 2024-04-12T04:46:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:52.199 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:46:52.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:52.200 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:46:52.200 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:46:52.200 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:46:52.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:52.200 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:46:52Z" 2024-04-12T04:46:52.200 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:46:52.201 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032624", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:28:50.052149Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046574", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:37:49.979474Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:46:52Z" } 2024-04-12T04:46:52.201 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046574', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:37:49.979474Z'}}] 2024-04-12T04:46:52.201 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:46:52.201 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046574'} 2024-04-12T04:46:52.201 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:46:52.202 DEBUG:boto:path=/dkyges-72/ 2024-04-12T04:46:52.202 DEBUG:boto:auth_path=/dkyges-72/ 2024-04-12T04:46:52.202 DEBUG:boto:Method: HEAD 2024-04-12T04:46:52.202 DEBUG:boto:Path: /dkyges-72/ 2024-04-12T04:46:52.202 DEBUG:boto:Data: 2024-04-12T04:46:52.202 DEBUG:boto:Headers: {} 2024-04-12T04:46:52.202 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:46:52.202 DEBUG:boto:Port: 8001 2024-04-12T04:46:52.202 DEBUG:boto:Params: {} 2024-04-12T04:46:52.203 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:46:52.203 DEBUG:boto:Token: None 2024-04-12T04:46:52.203 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:46:52 GMT /dkyges-72/ 2024-04-12T04:46:52.203 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4XzbXCYwK1536H5C7x1plVX6gI0= 2024-04-12T04:46:52.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:46:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4XzbXCYwK1536H5C7x1plVX6gI0=', 'Content-Length': '0'} 2024-04-12T04:46:52.235 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac6e3b18c01d5ca3-006618bcbc-2983-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:46:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:46:52.235 DEBUG:boto:path=/dkyges-72/obj5 2024-04-12T04:46:52.235 DEBUG:boto:auth_path=/dkyges-72/obj5 2024-04-12T04:46:52.236 DEBUG:boto:Method: HEAD 2024-04-12T04:46:52.236 DEBUG:boto:Path: /dkyges-72/obj5 2024-04-12T04:46:52.236 DEBUG:boto:Data: 2024-04-12T04:46:52.236 DEBUG:boto:Headers: {} 2024-04-12T04:46:52.236 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:46:52.236 DEBUG:boto:Port: 8001 2024-04-12T04:46:52.236 DEBUG:boto:Params: {} 2024-04-12T04:46:52.236 DEBUG:boto:Token: None 2024-04-12T04:46:52.236 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:46:52 GMT /dkyges-72/obj5 2024-04-12T04:46:52.236 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8AAeKI+7trvjlqxbR9pusZf8HFY= 2024-04-12T04:46:52.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:46:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8AAeKI+7trvjlqxbR9pusZf8HFY=', 'Content-Length': '0'} 2024-04-12T04:46:52.238 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f099015257eb6246-006618bcbc-2983-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:46:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:46:52.238 DEBUG:boto:path=/dkyges-72/obj6 2024-04-12T04:46:52.238 DEBUG:boto:auth_path=/dkyges-72/obj6 2024-04-12T04:46:52.238 DEBUG:boto:Method: HEAD 2024-04-12T04:46:52.238 DEBUG:boto:Path: /dkyges-72/obj6 2024-04-12T04:46:52.239 DEBUG:boto:Data: 2024-04-12T04:46:52.239 DEBUG:boto:Headers: {} 2024-04-12T04:46:52.239 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:46:52.239 DEBUG:boto:Port: 8001 2024-04-12T04:46:52.239 DEBUG:boto:Params: {} 2024-04-12T04:46:52.239 DEBUG:boto:Token: None 2024-04-12T04:46:52.239 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:46:52 GMT /dkyges-72/obj6 2024-04-12T04:46:52.239 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ex/fmfU/STtQxOdO8Gv6BqI13iI= 2024-04-12T04:46:52.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:46:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ex/fmfU/STtQxOdO8Gv6BqI13iI=', 'Content-Length': '0'} 2024-04-12T04:46:52.241 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e6937fa320f70cc3-006618bcbc-2983-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:46:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:46:52.241 DEBUG:boto:path=/dkyges-73/ 2024-04-12T04:46:52.241 DEBUG:boto:auth_path=/dkyges-73/ 2024-04-12T04:46:52.241 DEBUG:boto:Method: HEAD 2024-04-12T04:46:52.241 DEBUG:boto:Path: /dkyges-73/ 2024-04-12T04:46:52.241 DEBUG:boto:Data: 2024-04-12T04:46:52.241 DEBUG:boto:Headers: {} 2024-04-12T04:46:52.242 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:46:52.242 DEBUG:boto:Port: 8001 2024-04-12T04:46:52.242 DEBUG:boto:Params: {} 2024-04-12T04:46:52.242 DEBUG:boto:Token: None 2024-04-12T04:46:52.242 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:46:52 GMT /dkyges-73/ 2024-04-12T04:46:52.242 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:4B7w/x5l6zS/9K97Q7nZmBaTiZ4= 2024-04-12T04:46:52.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:46:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4B7w/x5l6zS/9K97Q7nZmBaTiZ4=', 'Content-Length': '0'} 2024-04-12T04:46:52.244 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b33f6925c3259c09-006618bcbc-2983-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:46:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:46:52.245 DEBUG:boto:path=/dkyges-73/obj5 2024-04-12T04:46:52.245 DEBUG:boto:auth_path=/dkyges-73/obj5 2024-04-12T04:46:52.245 DEBUG:boto:Method: HEAD 2024-04-12T04:46:52.245 DEBUG:boto:Path: /dkyges-73/obj5 2024-04-12T04:46:52.245 DEBUG:boto:Data: 2024-04-12T04:46:52.245 DEBUG:boto:Headers: {} 2024-04-12T04:46:52.245 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:46:52.245 DEBUG:boto:Port: 8001 2024-04-12T04:46:52.245 DEBUG:boto:Params: {} 2024-04-12T04:46:52.245 DEBUG:boto:Token: None 2024-04-12T04:46:52.246 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:46:52 GMT /dkyges-73/obj5 2024-04-12T04:46:52.246 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hYtchyd4DoMmauYENHHb06RKn5I= 2024-04-12T04:46:52.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:46:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hYtchyd4DoMmauYENHHb06RKn5I=', 'Content-Length': '0'} 2024-04-12T04:46:52.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:46:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-72:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9939.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090a172e9ec7b5589-006618bcbc-2983-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:46:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:46:52.248 DEBUG:boto:path=/dkyges-73/obj5 2024-04-12T04:46:52.249 DEBUG:boto:auth_path=/dkyges-73/obj5 2024-04-12T04:46:52.249 DEBUG:boto:Method: GET 2024-04-12T04:46:52.249 DEBUG:boto:Path: /dkyges-73/obj5 2024-04-12T04:46:52.249 DEBUG:boto:Data: 2024-04-12T04:46:52.249 DEBUG:boto:Headers: {} 2024-04-12T04:46:52.249 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:46:52.249 DEBUG:boto:Port: 8001 2024-04-12T04:46:52.249 DEBUG:boto:Params: {} 2024-04-12T04:46:52.249 DEBUG:boto:Token: None 2024-04-12T04:46:52.249 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:46:52 GMT /dkyges-73/obj5 2024-04-12T04:46:52.249 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2F74zDW5B0FaTNp759OWJqnC17Q= 2024-04-12T04:46:52.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:46:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2F74zDW5B0FaTNp759OWJqnC17Q=', 'Content-Length': '0'} 2024-04-12T04:46:52.252 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:46:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-72:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9939.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aca8dc9ed1b7e141-006618bcbc-2983-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:46:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:46:52.252 DEBUG:boto:path=/dkyges-73/obj6 2024-04-12T04:46:52.252 DEBUG:boto:auth_path=/dkyges-73/obj6 2024-04-12T04:46:52.252 DEBUG:boto:Method: HEAD 2024-04-12T04:46:52.252 DEBUG:boto:Path: /dkyges-73/obj6 2024-04-12T04:46:52.252 DEBUG:boto:Data: 2024-04-12T04:46:52.252 DEBUG:boto:Headers: {} 2024-04-12T04:46:52.252 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:46:52.253 DEBUG:boto:Port: 8001 2024-04-12T04:46:52.253 DEBUG:boto:Params: {} 2024-04-12T04:46:52.253 DEBUG:boto:Token: None 2024-04-12T04:46:52.253 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:46:52 GMT /dkyges-73/obj6 2024-04-12T04:46:52.253 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zldOEhltD1AjKHzq1U5V1e2jjUs= 2024-04-12T04:46:52.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:46:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zldOEhltD1AjKHzq1U5V1e2jjUs=', 'Content-Length': '0'} 2024-04-12T04:46:52.255 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:46:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-72:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9939.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e008372bf69eed01-006618bcbc-2983-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:46:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:46:52.255 DEBUG:boto:path=/dkyges-73/obj6 2024-04-12T04:46:52.255 DEBUG:boto:auth_path=/dkyges-73/obj6 2024-04-12T04:46:52.255 DEBUG:boto:Method: GET 2024-04-12T04:46:52.255 DEBUG:boto:Path: /dkyges-73/obj6 2024-04-12T04:46:52.255 DEBUG:boto:Data: 2024-04-12T04:46:52.255 DEBUG:boto:Headers: {} 2024-04-12T04:46:52.256 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:46:52.256 DEBUG:boto:Port: 8001 2024-04-12T04:46:52.256 DEBUG:boto:Params: {} 2024-04-12T04:46:52.256 DEBUG:boto:Token: None 2024-04-12T04:46:52.256 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:46:52 GMT /dkyges-73/obj6 2024-04-12T04:46:52.256 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:n31CHhzrt5hMbYUWXXQZIYG2OkA= 2024-04-12T04:46:52.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:46:52 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:n31CHhzrt5hMbYUWXXQZIYG2OkA=', 'Content-Length': '0'} 2024-04-12T04:46:52.258 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:46:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-72:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9939.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001243aaa767e35f4d-006618bcbc-2983-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:46:52 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:46:52.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:46:52.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:46:52.299 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dkyges-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:46:52.357 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:52.357 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:52.423 INFO:teuthology.orchestra.run.smithi043.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:46:52.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:52.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:52.424 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:52.477 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:52.477 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:52.518 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:46:52.534 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:46:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:52.534 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:46:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:46:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:46:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:46:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:46:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:46:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:52.535 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897212.400118_964.1", 2024-04-12T04:46:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:46:52.400118Z" 2024-04-12T04:46:52.536 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:52.536 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:46:52.537 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897212.400118_964.1'} 2024-04-12T04:46:52.537 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:46:52.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:52.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:52.537 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:52.592 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:52.592 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:52.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:52.626+0000 7fa0348cd780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:52.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:52.627+0000 7fa0348cd780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:52.645 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:46:52.645 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:46:52.645 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:46:52.645 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:46:52.645 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:46:52.645 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:46:52.645 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:52.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:46:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:46:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:46:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:46:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:52.646 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:46:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:46:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:46:52.647 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:46:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:46:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:52.648 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:46:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:46:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:52.649 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:46:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897067.715675_960.1", 2024-04-12T04:46:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:44:27.715675Z", 2024-04-12T04:46:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:52.650 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:46:52.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:52.651 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:46:52.651 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:46:52.651 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:46:52.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:52.651 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:46:52Z" 2024-04-12T04:46:52.651 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:46:52.652 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897067.715675_960.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:44:27.715675Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:46:52Z" } 2024-04-12T04:46:52.652 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897067.715675_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:44:27.715675Z', 'realm_epoch': 2}}] 2024-04-12T04:46:52.652 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897212.400118_964.1'} 2024-04-12T04:46:52.653 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:46:52.653 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897212.400118_964.1 target=1_1712897067.715675_960.1 2024-04-12T04:46:57.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:57.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:46:57.654 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:46:57.713 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:46:57.713 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:46:57.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:57.751+0000 7f3b33348780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:57.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:46:57.752+0000 7f3b33348780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:46:57.769 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:46:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:46:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:46:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:46:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:46:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:46:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:46:57.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:46:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:46:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:46:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:46:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:57.771 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:46:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:46:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:46:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:57.772 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:46:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:46:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:46:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:57.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:46:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:46:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:46:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:46:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897067.715675_960.1", 2024-04-12T04:46:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:46:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:46:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:46:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:44:27.715675Z", 2024-04-12T04:46:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:46:57.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:46:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:46:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:46:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:46:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:46:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:46:57.775 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:46:57Z" 2024-04-12T04:46:57.775 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:46:57.776 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897067.715675_960.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:44:27.715675Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:46:57Z" } 2024-04-12T04:46:57.776 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897067.715675_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:44:27.715675Z', 'realm_epoch': 2}}] 2024-04-12T04:46:57.776 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897212.400118_964.1'} 2024-04-12T04:46:57.777 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:46:57.777 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897212.400118_964.1 target=1_1712897067.715675_960.1 2024-04-12T04:47:02.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:02.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:02.778 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:02.837 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:02.838 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:02.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:02.871+0000 7f5c4ffb3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:02.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:02.872+0000 7f5c4ffb3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:02.890 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:02.891 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:02.891 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:02.891 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:02.891 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:02.891 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:02.891 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:02.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:02.891 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:02.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:02.891 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:02.891 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:02.892 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:02.892 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:02.892 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:02.892 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:47:02.892 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:02.892 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:47:02.892 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:02.892 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:02.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:02.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:02.893 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:02.893 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:02.893 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:02.893 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:02.893 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:02.893 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:02.893 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:02.893 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:47:02.893 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:02.893 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:02.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:02.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:02.894 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:02.894 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:02.894 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:02.894 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:02.894 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:02.894 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:02.894 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:02.894 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:47:02.895 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:02.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:02.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:02.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:02.895 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:02.895 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:02.895 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:02.895 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897067.715675_960.1", 2024-04-12T04:47:02.895 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:02.895 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:02.895 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:02.896 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:44:27.715675Z", 2024-04-12T04:47:02.896 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:02.896 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:02.896 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:02.896 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:02.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:02.896 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:02.896 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:47:02.896 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:47:02.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:02.897 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:02Z" 2024-04-12T04:47:02.897 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:02.897 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897067.715675_960.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:44:27.715675Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:47:02Z" } 2024-04-12T04:47:02.898 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897067.715675_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:44:27.715675Z', 'realm_epoch': 2}}] 2024-04-12T04:47:02.898 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897212.400118_964.1'} 2024-04-12T04:47:02.898 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:47:02.898 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897212.400118_964.1 target=1_1712897067.715675_960.1 2024-04-12T04:47:07.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:07.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:07.900 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:07.959 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:07.959 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:07.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:07.995+0000 7f1611e42780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:07.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:07.996+0000 7f1611e42780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:08.014 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:08.014 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:08.014 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:08.015 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:08.015 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:08.015 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:08.015 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:08.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:08.015 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:08.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:08.015 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:08.015 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:08.015 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:08.015 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:47:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:47:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:47:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:47:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897067.715675_960.1", 2024-04-12T04:47:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:44:27.715675Z", 2024-04-12T04:47:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:47:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:47:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:07Z" 2024-04-12T04:47:08.020 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:08.021 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897067.715675_960.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:44:27.715675Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:47:07Z" } 2024-04-12T04:47:08.021 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897067.715675_960.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:44:27.715675Z', 'realm_epoch': 2}}] 2024-04-12T04:47:08.021 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897212.400118_964.1'} 2024-04-12T04:47:08.021 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897067.715675_960.1'} 2024-04-12T04:47:08.022 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897212.400118_964.1 target=1_1712897067.715675_960.1 2024-04-12T04:47:13.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:13.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:13.023 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:13.086 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:13.086 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:13.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:13.118+0000 7fe60f392780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:13.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:13.119+0000 7fe60f392780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:13.137 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:13.138 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:13.138 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:13.138 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:13.138 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:13.138 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:13.138 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:13.138 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:13.138 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:13.138 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:13.139 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:13.139 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:13.139 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:13.139 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:13.139 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:13.139 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:47:13.139 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:13.139 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:47:13.139 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:13.139 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:13.139 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:13.140 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:13.140 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:13.140 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:13.140 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:13.140 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:13.140 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:13.140 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:13.140 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:13.140 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:47:13.141 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:13.141 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:13.141 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:13.141 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:13.141 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:13.141 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:13.141 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:13.141 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:13.141 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:13.141 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:13.141 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:13.141 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:47:13.142 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:13.142 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:13.142 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:13.142 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:13.142 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:13.142 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:13.142 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:13.142 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897212.400118_964.1", 2024-04-12T04:47:13.142 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:13.142 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:13.142 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:13.143 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:52.400118Z", 2024-04-12T04:47:13.143 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:13.143 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:13.143 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:13.143 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:13.143 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:13.143 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:13.143 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:47:13.143 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:47:13.143 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:13.143 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:13Z" 2024-04-12T04:47:13.143 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:13.144 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897212.400118_964.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:52.400118Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:47:13Z" } 2024-04-12T04:47:13.145 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897212.400118_964.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:52.400118Z', 'realm_epoch': 2}}] 2024-04-12T04:47:13.145 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897212.400118_964.1'} 2024-04-12T04:47:13.145 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897212.400118_964.1'} 2024-04-12T04:47:13.145 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:47:13.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:13.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:13.145 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dkyges-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:47:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:13.198 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:13.256 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[]} 2024-04-12T04:47:13.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:13.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:13.257 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dkyges-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:47:13.316 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:13.316 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:13.377 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:47:13.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:13.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:13.378 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket dkyges-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:47:13.441 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:13.441 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:13.502 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:47:13.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dkyges-72', '--source-bucket=dkyges-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:13.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=dkyges-72', '--source-bucket=dkyges-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:13.503 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=dkyges-72 --source-bucket=dkyges-73 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:47:13.564 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:13.564 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:13.629 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dkyges-73","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-12T04:47:13.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:13.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:13.630 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dkyges-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:47:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:13.747 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dkyges-73","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:47:13.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:13.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:13.747 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket dkyges-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:47:13.800 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:13.800 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:13.856 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"dkyges-73","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:47:13.856 DEBUG:boto:path=/dkyges-72/ 2024-04-12T04:47:13.856 DEBUG:boto:auth_path=/dkyges-72/ 2024-04-12T04:47:13.857 DEBUG:boto:Method: HEAD 2024-04-12T04:47:13.857 DEBUG:boto:Path: /dkyges-72/ 2024-04-12T04:47:13.857 DEBUG:boto:Data: 2024-04-12T04:47:13.857 DEBUG:boto:Headers: {} 2024-04-12T04:47:13.857 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:47:13.857 DEBUG:boto:Port: 8001 2024-04-12T04:47:13.857 DEBUG:boto:Params: {} 2024-04-12T04:47:13.858 DEBUG:boto:Token: None 2024-04-12T04:47:13.858 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:47:13 GMT /dkyges-72/ 2024-04-12T04:47:13.858 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BGaktzl2nD3JoYS5AzJJ33Wh1xE= 2024-04-12T04:47:13.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BGaktzl2nD3JoYS5AzJJ33Wh1xE=', 'Content-Length': '0'} 2024-04-12T04:47:13.861 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4c486f3d1e1abbc-006618bcd1-2983-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:47:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:13.861 DEBUG:boto:path=/dkyges-72/obj7 2024-04-12T04:47:13.861 DEBUG:boto:auth_path=/dkyges-72/obj7 2024-04-12T04:47:13.862 DEBUG:boto:Method: PUT 2024-04-12T04:47:13.862 DEBUG:boto:Path: /dkyges-72/obj7 2024-04-12T04:47:13.862 DEBUG:boto:Data: 2024-04-12T04:47:13.862 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:47:13.862 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:47:13.862 DEBUG:boto:Port: 8001 2024-04-12T04:47:13.862 DEBUG:boto:Params: {} 2024-04-12T04:47:13.862 DEBUG:boto:Token: None 2024-04-12T04:47:13.862 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:47:13 GMT /dkyges-72/obj7 2024-04-12T04:47:13.863 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:VgusIvh+2G+WjUWYYemVOU92Pe4= 2024-04-12T04:47:13.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:47:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VgusIvh+2G+WjUWYYemVOU92Pe4='} 2024-04-12T04:47:13.870 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5bde20eb394d38a-006618bcd1-2983-a2'), ('Date', 'Fri, 12 Apr 2024 04:47:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:13.870 DEBUG:boto:path=/dkyges-72/ 2024-04-12T04:47:13.870 DEBUG:boto:auth_path=/dkyges-72/ 2024-04-12T04:47:13.870 DEBUG:boto:Method: HEAD 2024-04-12T04:47:13.871 DEBUG:boto:Path: /dkyges-72/ 2024-04-12T04:47:13.871 DEBUG:boto:Data: 2024-04-12T04:47:13.871 DEBUG:boto:Headers: {} 2024-04-12T04:47:13.871 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:47:13.871 DEBUG:boto:Port: 8001 2024-04-12T04:47:13.871 DEBUG:boto:Params: {} 2024-04-12T04:47:13.871 DEBUG:boto:Token: None 2024-04-12T04:47:13.871 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:47:13 GMT /dkyges-72/ 2024-04-12T04:47:13.871 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BGaktzl2nD3JoYS5AzJJ33Wh1xE= 2024-04-12T04:47:13.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BGaktzl2nD3JoYS5AzJJ33Wh1xE=', 'Content-Length': '0'} 2024-04-12T04:47:13.874 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086d6f02a770981a3-006618bcd1-2983-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:47:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:13.874 DEBUG:boto:path=/dkyges-72/obj8 2024-04-12T04:47:13.874 DEBUG:boto:auth_path=/dkyges-72/obj8 2024-04-12T04:47:13.875 DEBUG:boto:Method: PUT 2024-04-12T04:47:13.875 DEBUG:boto:Path: /dkyges-72/obj8 2024-04-12T04:47:13.875 DEBUG:boto:Data: 2024-04-12T04:47:13.875 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:47:13.875 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:47:13.875 DEBUG:boto:Port: 8001 2024-04-12T04:47:13.875 DEBUG:boto:Params: {} 2024-04-12T04:47:13.875 DEBUG:boto:Token: None 2024-04-12T04:47:13.875 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:47:13 GMT /dkyges-72/obj8 2024-04-12T04:47:13.876 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:0KYXi152Vreo5E67D35VebsTixc= 2024-04-12T04:47:13.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:47:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0KYXi152Vreo5E67D35VebsTixc='} 2024-04-12T04:47:13.881 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c6339d03b462f9a-006618bcd1-2983-a2'), ('Date', 'Fri, 12 Apr 2024 04:47:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:13.882 DEBUG:boto:path=/dkyges-73/ 2024-04-12T04:47:13.882 DEBUG:boto:auth_path=/dkyges-73/ 2024-04-12T04:47:13.882 DEBUG:boto:Method: HEAD 2024-04-12T04:47:13.882 DEBUG:boto:Path: /dkyges-73/ 2024-04-12T04:47:13.882 DEBUG:boto:Data: 2024-04-12T04:47:13.882 DEBUG:boto:Headers: {} 2024-04-12T04:47:13.882 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:47:13.882 DEBUG:boto:Port: 8001 2024-04-12T04:47:13.882 DEBUG:boto:Params: {} 2024-04-12T04:47:13.883 DEBUG:boto:Token: None 2024-04-12T04:47:13.883 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:47:13 GMT /dkyges-73/ 2024-04-12T04:47:13.883 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+Hy5IIpZfUCJgIozGVCffHbdDoQ= 2024-04-12T04:47:13.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+Hy5IIpZfUCJgIozGVCffHbdDoQ=', 'Content-Length': '0'} 2024-04-12T04:47:13.885 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000584a0ebdeaf4de14-006618bcd1-2983-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:47:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:13.885 DEBUG:boto:path=/dkyges-73/obj9 2024-04-12T04:47:13.885 DEBUG:boto:auth_path=/dkyges-73/obj9 2024-04-12T04:47:13.885 DEBUG:boto:Method: PUT 2024-04-12T04:47:13.886 DEBUG:boto:Path: /dkyges-73/obj9 2024-04-12T04:47:13.886 DEBUG:boto:Data: 2024-04-12T04:47:13.886 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:47:13.886 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:47:13.886 DEBUG:boto:Port: 8001 2024-04-12T04:47:13.886 DEBUG:boto:Params: {} 2024-04-12T04:47:13.886 DEBUG:boto:Token: None 2024-04-12T04:47:13.886 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:47:13 GMT /dkyges-73/obj9 2024-04-12T04:47:13.886 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:b0FUFLgvlcsOmLEfASF/Wa8OZpc= 2024-04-12T04:47:13.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:47:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:b0FUFLgvlcsOmLEfASF/Wa8OZpc='} 2024-04-12T04:47:13.893 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c829f54d54aaad0f-006618bcd1-2983-a2'), ('Date', 'Fri, 12 Apr 2024 04:47:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:13.893 DEBUG:boto:path=/dkyges-73/ 2024-04-12T04:47:13.893 DEBUG:boto:auth_path=/dkyges-73/ 2024-04-12T04:47:13.893 DEBUG:boto:Method: HEAD 2024-04-12T04:47:13.893 DEBUG:boto:Path: /dkyges-73/ 2024-04-12T04:47:13.893 DEBUG:boto:Data: 2024-04-12T04:47:13.894 DEBUG:boto:Headers: {} 2024-04-12T04:47:13.894 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:47:13.894 DEBUG:boto:Port: 8001 2024-04-12T04:47:13.894 DEBUG:boto:Params: {} 2024-04-12T04:47:13.894 DEBUG:boto:Token: None 2024-04-12T04:47:13.895 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:47:13 GMT /dkyges-73/ 2024-04-12T04:47:13.895 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:+Hy5IIpZfUCJgIozGVCffHbdDoQ= 2024-04-12T04:47:13.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+Hy5IIpZfUCJgIozGVCffHbdDoQ=', 'Content-Length': '0'} 2024-04-12T04:47:13.897 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057eb9a4afa105b75-006618bcd1-2983-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:47:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:13.897 DEBUG:boto:path=/dkyges-73/obj10 2024-04-12T04:47:13.897 DEBUG:boto:auth_path=/dkyges-73/obj10 2024-04-12T04:47:13.897 DEBUG:boto:Method: PUT 2024-04-12T04:47:13.897 DEBUG:boto:Path: /dkyges-73/obj10 2024-04-12T04:47:13.898 DEBUG:boto:Data: 2024-04-12T04:47:13.898 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:47:13.898 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:47:13.898 DEBUG:boto:Port: 8001 2024-04-12T04:47:13.898 DEBUG:boto:Params: {} 2024-04-12T04:47:13.898 DEBUG:boto:Token: None 2024-04-12T04:47:13.898 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:47:13 GMT /dkyges-73/obj10 2024-04-12T04:47:13.898 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JXYR1VjnwZpPnnUp1sSke0EuIts= 2024-04-12T04:47:13.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:47:13 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JXYR1VjnwZpPnnUp1sSke0EuIts='} 2024-04-12T04:47:13.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d129b0f34b9d27ec-006618bcd1-2983-a2'), ('Date', 'Fri, 12 Apr 2024 04:47:13 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:13.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:13.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:13.943 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:13.999 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:14.000 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:14.040 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:47:14.058 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:47:14.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:14.058 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:14.059 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:47:14.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:14.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:14.059 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:14.059 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:47:14.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:14.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:14.059 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:14.059 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:47:14.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:14.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:14.061 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897233.727966_972.1", 2024-04-12T04:47:14.061 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:47:13.727966Z" 2024-04-12T04:47:14.061 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:14.061 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:47:14.062 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897233.727966_972.1'} 2024-04-12T04:47:14.062 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:47:14.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:14.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:14.062 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:14.118 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:14.118 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:14.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:14.151+0000 7f8e9e1ec780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:14.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:14.152+0000 7f8e9e1ec780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:14.169 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:14.169 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:14.169 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:14.170 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:14.170 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:14.170 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:14.170 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:14.170 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:14.170 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:14.170 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:14.170 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:14.170 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:14.170 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:14.170 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:14.171 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:14.171 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:47:14.171 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:14.171 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:47:14.171 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:14.171 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:14.171 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:14.171 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:14.171 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:14.171 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:47:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:14.172 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:47:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:14.173 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:14.174 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:14.174 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897212.400118_964.1", 2024-04-12T04:47:14.174 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:14.174 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:14.174 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:14.174 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:52.400118Z", 2024-04-12T04:47:14.174 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:14.174 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:14.174 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:14.174 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:14.174 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:14.175 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:14.175 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:47:14.175 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:47:14.175 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:14.175 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:14Z" 2024-04-12T04:47:14.175 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:14.176 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897212.400118_964.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:52.400118Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:47:14Z" } 2024-04-12T04:47:14.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897212.400118_964.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:52.400118Z', 'realm_epoch': 2}}] 2024-04-12T04:47:14.176 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897233.727966_972.1'} 2024-04-12T04:47:14.176 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897212.400118_964.1'} 2024-04-12T04:47:14.176 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897233.727966_972.1 target=1_1712897212.400118_964.1 2024-04-12T04:47:19.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:19.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:19.178 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:19.238 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:19.238 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:19.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:19.273+0000 7f95febd3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:19.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:19.276+0000 7f95febd3780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:19.296 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:19.297 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:19.297 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:19.297 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:19.297 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:19.297 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:19.297 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:19.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:19.297 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:19.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:19.297 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:19.297 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:19.297 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:19.298 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:19.298 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:19.298 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:47:19.298 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:19.298 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:47:19.298 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:19.298 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:19.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:19.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:19.298 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:19.298 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:19.299 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:19.299 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:19.299 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:19.299 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:19.299 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:19.299 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:47:19.299 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:19.299 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:19.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:19.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:19.299 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:19.300 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:19.300 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:19.300 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:19.300 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:19.300 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:19.300 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:19.300 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:47:19.300 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:19.300 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:19.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:19.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:19.300 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:19.301 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:19.301 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:19.301 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897212.400118_964.1", 2024-04-12T04:47:19.301 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:19.301 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:19.301 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:19.301 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:52.400118Z", 2024-04-12T04:47:19.301 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:19.301 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:19.301 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:19.301 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:19.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:19.302 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:19.302 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:47:19.302 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:47:19.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:19.302 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:19Z" 2024-04-12T04:47:19.302 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:19.303 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897212.400118_964.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:52.400118Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:47:19Z" } 2024-04-12T04:47:19.303 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897212.400118_964.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:52.400118Z', 'realm_epoch': 2}}] 2024-04-12T04:47:19.303 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897233.727966_972.1'} 2024-04-12T04:47:19.304 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897212.400118_964.1'} 2024-04-12T04:47:19.304 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897233.727966_972.1 target=1_1712897212.400118_964.1 2024-04-12T04:47:24.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:24.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:24.305 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:24.365 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:24.366 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:24.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:24.400+0000 7f511768d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:24.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:24.401+0000 7f511768d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:24.419 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:24.419 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:24.419 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:24.420 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:24.420 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:24.420 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:24.420 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:24.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:24.420 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:24.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:24.420 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:24.420 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:24.420 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:24.421 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:24.421 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:24.421 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:47:24.421 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:24.421 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:47:24.421 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:24.421 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:24.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:24.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:24.421 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:24.422 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:24.422 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:24.422 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:24.422 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:24.422 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:24.422 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:24.422 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:47:24.422 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:24.423 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:24.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:24.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:24.423 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:24.423 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:24.423 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:24.423 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:24.423 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:24.423 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:24.423 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:24.423 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:47:24.424 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:24.424 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:24.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:24.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:24.424 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:24.424 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:24.424 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:24.424 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897212.400118_964.1", 2024-04-12T04:47:24.424 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:24.424 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:24.424 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:24.425 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:52.400118Z", 2024-04-12T04:47:24.425 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:24.425 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:24.425 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:24.425 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:24.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:24.425 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:24.425 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:47:24.425 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:47:24.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:24.425 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:24Z" 2024-04-12T04:47:24.426 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:24.427 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897212.400118_964.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:52.400118Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:47:24Z" } 2024-04-12T04:47:24.427 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897212.400118_964.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:52.400118Z', 'realm_epoch': 2}}] 2024-04-12T04:47:24.427 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897233.727966_972.1'} 2024-04-12T04:47:24.427 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897212.400118_964.1'} 2024-04-12T04:47:24.427 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897233.727966_972.1 target=1_1712897212.400118_964.1 2024-04-12T04:47:29.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:29.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:29.429 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:29.489 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:29.490 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:29.526+0000 7f5554278780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:29.527+0000 7f5554278780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:29.545 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:29.545 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:47:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:47:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:47:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:47:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897233.727966_972.1", 2024-04-12T04:47:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:13.727966Z", 2024-04-12T04:47:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:47:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:47:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:29Z" 2024-04-12T04:47:29.551 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:29.552 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897233.727966_972.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:13.727966Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:47:29Z" } 2024-04-12T04:47:29.553 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897233.727966_972.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:13.727966Z', 'realm_epoch': 2}}] 2024-04-12T04:47:29.553 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897233.727966_972.1'} 2024-04-12T04:47:29.553 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897233.727966_972.1'} 2024-04-12T04:47:29.553 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:47:29.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:29.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:29.553 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:29.606 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:29.606 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:29.666 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:47:29.666 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:29.666 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000015493", 2024-04-12T04:47:29.666 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:46:48.335126Z" 2024-04-12T04:47:29.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:29.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:29.667 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000007418", 2024-04-12T04:47:29.667 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:47:13.902919Z" 2024-04-12T04:47:29.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:29.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:29.667 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000020236", 2024-04-12T04:47:29.667 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:47:13.868291Z" 2024-04-12T04:47:29.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:29.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:29.667 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000018834", 2024-04-12T04:47:29.667 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:47:13.891241Z" 2024-04-12T04:47:29.668 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:29.668 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:47:29.669 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000015493', 1: '00000000000000000000:00000000000000007418', 2: '00000000000000000000:00000000000000020236', 3: '00000000000000000000:00000000000000018834'} 2024-04-12T04:47:29.669 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T04:47:29.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:29.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:29.669 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:29.726 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:29.726 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:29.780 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:29.780 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:29.780 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:29.780 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:29.780 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:29.780 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5944041172307441597 2024-04-12T04:47:29.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:29.780 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:29.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:29.781 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:29.781 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:29.781 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:47:29.781 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000015493", 2024-04-12T04:47:29.781 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:29.781 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:29.781 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:29.781 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:48.335126Z" 2024-04-12T04:47:29.781 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:29.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:29.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:29.782 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:29.782 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:29.782 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:47:29.782 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000007418", 2024-04-12T04:47:29.782 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:29.782 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:29.782 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:29.782 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:13.902919Z" 2024-04-12T04:47:29.782 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:29.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:29.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:29.783 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:29.783 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:29.783 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:47:29.783 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000020236", 2024-04-12T04:47:29.783 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:29.783 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:29.783 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:29.783 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:13.868291Z" 2024-04-12T04:47:29.783 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:29.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:29.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:29.783 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:29.784 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:29.784 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:47:29.784 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000018834", 2024-04-12T04:47:29.784 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:29.784 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:29.784 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:29.784 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:13.891241Z" 2024-04-12T04:47:29.784 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:29.784 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:29.784 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:29.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:29.785 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:29.785 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 0, 2024-04-12T04:47:29.785 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 0 2024-04-12T04:47:29.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:29.785 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:29Z" 2024-04-12T04:47:29.785 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:29.786 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5944041172307441597 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015493", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:48.335126Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007418", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:13.902919Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000020236", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:13.868291Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018834", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:13.891241Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2024-04-12T04:47:29Z" } 2024-04-12T04:47:29.786 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015493', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:48.335126Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007418', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:13.902919Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000020236', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:13.868291Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018834', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:13.891241Z'}}] 2024-04-12T04:47:29.787 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015493', 1: '00000000000000000000:00000000000000007418', 2: '00000000000000000000:00000000000000020236', 3: '00000000000000000000:00000000000000018834'} 2024-04-12T04:47:29.787 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015493', 1: '00000000000000000000:00000000000000007418', 2: '00000000000000000000:00000000000000020236', 3: '00000000000000000000:00000000000000018834'} 2024-04-12T04:47:29.787 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2024-04-12T04:47:29.787 DEBUG:boto:path=/dkyges-72/ 2024-04-12T04:47:29.787 DEBUG:boto:auth_path=/dkyges-72/ 2024-04-12T04:47:29.787 DEBUG:boto:Method: HEAD 2024-04-12T04:47:29.787 DEBUG:boto:Path: /dkyges-72/ 2024-04-12T04:47:29.787 DEBUG:boto:Data: 2024-04-12T04:47:29.787 DEBUG:boto:Headers: {} 2024-04-12T04:47:29.788 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:29.788 DEBUG:boto:Port: 8000 2024-04-12T04:47:29.788 DEBUG:boto:Params: {} 2024-04-12T04:47:29.788 DEBUG:boto:Token: None 2024-04-12T04:47:29.788 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:47:29 GMT /dkyges-72/ 2024-04-12T04:47:29.788 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1PUHJlTw9BuQ3WjMqIZedU8g45g= 2024-04-12T04:47:29.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1PUHJlTw9BuQ3WjMqIZedU8g45g=', 'Content-Length': '0'} 2024-04-12T04:47:29.791 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068481af90aa6ea2f-006618bce1-2977-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:47:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:29.791 DEBUG:boto:path=/dkyges-72/obj7 2024-04-12T04:47:29.791 DEBUG:boto:auth_path=/dkyges-72/obj7 2024-04-12T04:47:29.791 DEBUG:boto:Method: HEAD 2024-04-12T04:47:29.792 DEBUG:boto:Path: /dkyges-72/obj7 2024-04-12T04:47:29.792 DEBUG:boto:Data: 2024-04-12T04:47:29.792 DEBUG:boto:Headers: {} 2024-04-12T04:47:29.792 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:29.792 DEBUG:boto:Port: 8000 2024-04-12T04:47:29.792 DEBUG:boto:Params: {} 2024-04-12T04:47:29.792 DEBUG:boto:Token: None 2024-04-12T04:47:29.792 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:47:29 GMT /dkyges-72/obj7 2024-04-12T04:47:29.792 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hYh8ANNML4ox88PWR3w/ZlepEso= 2024-04-12T04:47:29.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hYh8ANNML4ox88PWR3w/ZlepEso=', 'Content-Length': '0'} 2024-04-12T04:47:29.794 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000003ba8fa1bb6d56096-006618bce1-2977-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:47:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:29.794 DEBUG:boto:path=/dkyges-72/obj8 2024-04-12T04:47:29.794 DEBUG:boto:auth_path=/dkyges-72/obj8 2024-04-12T04:47:29.794 DEBUG:boto:Method: HEAD 2024-04-12T04:47:29.795 DEBUG:boto:Path: /dkyges-72/obj8 2024-04-12T04:47:29.795 DEBUG:boto:Data: 2024-04-12T04:47:29.795 DEBUG:boto:Headers: {} 2024-04-12T04:47:29.795 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:29.795 DEBUG:boto:Port: 8000 2024-04-12T04:47:29.795 DEBUG:boto:Params: {} 2024-04-12T04:47:29.795 DEBUG:boto:Token: None 2024-04-12T04:47:29.795 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:47:29 GMT /dkyges-72/obj8 2024-04-12T04:47:29.795 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hpWzXWuoDsxRpCOmG5jXy7FkFTw= 2024-04-12T04:47:29.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hpWzXWuoDsxRpCOmG5jXy7FkFTw=', 'Content-Length': '0'} 2024-04-12T04:47:29.797 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e98eca9b94f7e527-006618bce1-2977-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:47:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:29.797 DEBUG:boto:path=/dkyges-72/obj9 2024-04-12T04:47:29.797 DEBUG:boto:auth_path=/dkyges-72/obj9 2024-04-12T04:47:29.797 DEBUG:boto:Method: HEAD 2024-04-12T04:47:29.797 DEBUG:boto:Path: /dkyges-72/obj9 2024-04-12T04:47:29.797 DEBUG:boto:Data: 2024-04-12T04:47:29.797 DEBUG:boto:Headers: {} 2024-04-12T04:47:29.798 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:29.798 DEBUG:boto:Port: 8000 2024-04-12T04:47:29.798 DEBUG:boto:Params: {} 2024-04-12T04:47:29.798 DEBUG:boto:Token: None 2024-04-12T04:47:29.798 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:47:29 GMT /dkyges-72/obj9 2024-04-12T04:47:29.798 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NdDMj2yiZS8HFIr0Qke/FRs2bcA= 2024-04-12T04:47:29.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NdDMj2yiZS8HFIr0Qke/FRs2bcA=', 'Content-Length': '0'} 2024-04-12T04:47:29.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:47:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-73:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9939.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003595d7f92ad5d4e2-006618bce1-2977-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:47:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:29.800 DEBUG:boto:path=/dkyges-72/obj9 2024-04-12T04:47:29.800 DEBUG:boto:auth_path=/dkyges-72/obj9 2024-04-12T04:47:29.801 DEBUG:boto:Method: GET 2024-04-12T04:47:29.801 DEBUG:boto:Path: /dkyges-72/obj9 2024-04-12T04:47:29.801 DEBUG:boto:Data: 2024-04-12T04:47:29.801 DEBUG:boto:Headers: {} 2024-04-12T04:47:29.801 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:29.801 DEBUG:boto:Port: 8000 2024-04-12T04:47:29.801 DEBUG:boto:Params: {} 2024-04-12T04:47:29.801 DEBUG:boto:Token: None 2024-04-12T04:47:29.801 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:47:29 GMT /dkyges-72/obj9 2024-04-12T04:47:29.801 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:M5WSF+0qRV6kObT4IbzbDFsEr4E= 2024-04-12T04:47:29.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:M5WSF+0qRV6kObT4IbzbDFsEr4E=', 'Content-Length': '0'} 2024-04-12T04:47:29.803 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:47:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-73:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9939.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c15b8b55bccf57b-006618bce1-2977-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:47:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:29.804 DEBUG:boto:path=/dkyges-72/obj10 2024-04-12T04:47:29.804 DEBUG:boto:auth_path=/dkyges-72/obj10 2024-04-12T04:47:29.804 DEBUG:boto:Method: HEAD 2024-04-12T04:47:29.804 DEBUG:boto:Path: /dkyges-72/obj10 2024-04-12T04:47:29.804 DEBUG:boto:Data: 2024-04-12T04:47:29.804 DEBUG:boto:Headers: {} 2024-04-12T04:47:29.804 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:29.804 DEBUG:boto:Port: 8000 2024-04-12T04:47:29.804 DEBUG:boto:Params: {} 2024-04-12T04:47:29.804 DEBUG:boto:Token: None 2024-04-12T04:47:29.805 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:47:29 GMT /dkyges-72/obj10 2024-04-12T04:47:29.805 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RVPEyyW9aL/JLa9Wi0tK0rLQQvY= 2024-04-12T04:47:29.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RVPEyyW9aL/JLa9Wi0tK0rLQQvY=', 'Content-Length': '0'} 2024-04-12T04:47:29.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:47:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-73:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9939.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1e6400bbf5871bb-006618bce1-2977-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:47:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:29.807 DEBUG:boto:path=/dkyges-72/obj10 2024-04-12T04:47:29.807 DEBUG:boto:auth_path=/dkyges-72/obj10 2024-04-12T04:47:29.807 DEBUG:boto:Method: GET 2024-04-12T04:47:29.807 DEBUG:boto:Path: /dkyges-72/obj10 2024-04-12T04:47:29.807 DEBUG:boto:Data: 2024-04-12T04:47:29.807 DEBUG:boto:Headers: {} 2024-04-12T04:47:29.807 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:29.807 DEBUG:boto:Port: 8000 2024-04-12T04:47:29.808 DEBUG:boto:Params: {} 2024-04-12T04:47:29.808 DEBUG:boto:Token: None 2024-04-12T04:47:29.808 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:47:29 GMT /dkyges-72/obj10 2024-04-12T04:47:29.808 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:unsFCMbJHGGD1pCXLc42rUydkBM= 2024-04-12T04:47:29.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:29 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:unsFCMbJHGGD1pCXLc42rUydkBM=', 'Content-Length': '0'} 2024-04-12T04:47:29.810 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:47:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '1ba046e0-d0ae-437d-b741-28b4299423c2:dkyges-73:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9939.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000389940ba486ce2c3-006618bce1-2977-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:47:29 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:29.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:29.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dkyges-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:29.851 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dkyges-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:47:29.908 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:29.908 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:29.975 INFO:teuthology.orchestra.run.smithi043.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:47:29.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:29.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:29.975 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:47:30.028 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:30.028 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:30.085 INFO:teuthology.orchestra.run.smithi043.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:47:30.085 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... ok 2024-04-12T04:47:30.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:30.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:30.088 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:30.178 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:47:30.196 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:47:30.196 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:30.196 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:30.196 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:47:30.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:30.196 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:30.196 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:30.196 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:47:30.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:30.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:30.197 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:30.197 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:47:30.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:30.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:30.197 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897249.951095_974.1", 2024-04-12T04:47:30.197 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:47:29.951095Z" 2024-04-12T04:47:30.197 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:30.197 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:47:30.198 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897249.951095_974.1'} 2024-04-12T04:47:30.198 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:47:30.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:30.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:30.199 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:30.254 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:30.254 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:30.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:30.286+0000 7f2d96ea5780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:30.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:30.288+0000 7f2d96ea5780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:30.305 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:30.305 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:30.305 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:30.306 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:30.306 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:30.306 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:30.306 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:30.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:30.306 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:30.306 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:30.306 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:30.306 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:30.307 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:30.307 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:30.307 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:30.307 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:47:30.307 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:30.307 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:47:30.307 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:30.307 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:30.307 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:30.307 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:30.308 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:30.308 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:30.308 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:30.308 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:30.308 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:30.308 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:30.308 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:30.308 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:47:30.308 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:30.308 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:30.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:30.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:30.309 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:30.309 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:30.309 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:30.309 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:30.309 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:30.309 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:30.309 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:30.309 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:47:30.310 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:30.310 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:30.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:30.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:30.310 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:30.310 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:30.310 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:30.310 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897233.727966_972.1", 2024-04-12T04:47:30.310 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:30.310 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:30.311 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:30.311 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:13.727966Z", 2024-04-12T04:47:30.311 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:30.311 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:30.311 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:30.311 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:30.311 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:30.311 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:30.311 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:47:30.311 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:47:30.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:30.312 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:30Z" 2024-04-12T04:47:30.312 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:30.313 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897233.727966_972.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:13.727966Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:47:30Z" } 2024-04-12T04:47:30.313 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897233.727966_972.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:13.727966Z', 'realm_epoch': 2}}] 2024-04-12T04:47:30.313 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897249.951095_974.1'} 2024-04-12T04:47:30.313 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897233.727966_972.1'} 2024-04-12T04:47:30.313 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897249.951095_974.1 target=1_1712897233.727966_972.1 2024-04-12T04:47:35.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:35.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:35.314 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:35.375 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:35.375 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:35.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:35.412+0000 7fc9c30a9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:35.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:35.413+0000 7fc9c30a9780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:35.431 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:35.431 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:35.431 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:35.431 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:35.431 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:35.432 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:35.432 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:35.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:35.432 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:35.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:35.432 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:35.432 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:35.432 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:35.432 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:35.432 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:35.433 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:47:35.433 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:35.433 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:47:35.433 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:35.433 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:35.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:35.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:35.433 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:35.433 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:35.433 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:35.433 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:35.433 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:35.434 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:35.434 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:35.434 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:47:35.434 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:35.434 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:35.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:35.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:35.434 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:35.434 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:35.434 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:35.434 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:35.435 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:35.435 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:35.435 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:35.435 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:47:35.435 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:35.435 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:35.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:35.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:35.435 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:35.435 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:35.435 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:35.435 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897233.727966_972.1", 2024-04-12T04:47:35.436 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:35.436 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:35.436 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:35.436 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:13.727966Z", 2024-04-12T04:47:35.436 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:35.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:35.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:35.436 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:35.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:35.436 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:35.436 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:47:35.437 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:47:35.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:35.437 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:35Z" 2024-04-12T04:47:35.437 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:35.438 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897233.727966_972.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:13.727966Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:47:35Z" } 2024-04-12T04:47:35.438 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897233.727966_972.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:13.727966Z', 'realm_epoch': 2}}] 2024-04-12T04:47:35.438 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897249.951095_974.1'} 2024-04-12T04:47:35.438 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897233.727966_972.1'} 2024-04-12T04:47:35.438 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897249.951095_974.1 target=1_1712897233.727966_972.1 2024-04-12T04:47:40.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:40.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:40.440 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:40.500 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:40.501 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:40.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:40.537+0000 7f8d7808a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:40.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:40.538+0000 7f8d7808a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:40.557 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:47:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:47:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:47:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:47:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897233.727966_972.1", 2024-04-12T04:47:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:13.727966Z", 2024-04-12T04:47:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:47:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:47:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:40Z" 2024-04-12T04:47:40.563 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:40.563 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897233.727966_972.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:13.727966Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:47:40Z" } 2024-04-12T04:47:40.564 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897233.727966_972.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:13.727966Z', 'realm_epoch': 2}}] 2024-04-12T04:47:40.564 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897249.951095_974.1'} 2024-04-12T04:47:40.564 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897233.727966_972.1'} 2024-04-12T04:47:40.564 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897249.951095_974.1 target=1_1712897233.727966_972.1 2024-04-12T04:47:45.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:45.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:45.565 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:45.625 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:45.625 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:45.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:45.660+0000 7fc4b10b4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:45.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:45.662+0000 7fc4b10b4780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:45.680 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:45.680 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:45.680 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:45.680 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:45.681 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:45.681 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:45.681 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:45.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:45.681 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:45.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:45.681 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:45.681 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:45.681 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:45.681 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:45.681 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:45.682 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:47:45.682 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:45.682 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:47:45.682 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:45.682 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:45.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:45.682 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:45.682 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:45.682 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:45.682 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:45.683 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:45.683 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:45.683 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:45.683 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:45.683 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:47:45.683 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:45.683 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:45.683 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:45.683 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:45.683 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:45.683 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:45.684 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:45.684 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:45.684 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:45.684 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:45.684 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:45.684 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:47:45.684 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:45.684 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:45.684 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:45.684 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:45.684 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:45.685 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:45.685 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:45.685 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897233.727966_972.1", 2024-04-12T04:47:45.685 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:45.685 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:45.685 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:45.685 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:13.727966Z", 2024-04-12T04:47:45.685 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:45.685 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:45.685 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:45.685 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:45.686 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:45.686 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:45.686 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:47:45.686 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:47:45.686 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:45.686 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:45Z" 2024-04-12T04:47:45.686 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:45.687 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897233.727966_972.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:13.727966Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:47:45Z" } 2024-04-12T04:47:45.687 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897233.727966_972.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:13.727966Z', 'realm_epoch': 2}}] 2024-04-12T04:47:45.687 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897249.951095_974.1'} 2024-04-12T04:47:45.687 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897233.727966_972.1'} 2024-04-12T04:47:45.688 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897249.951095_974.1 target=1_1712897233.727966_972.1 2024-04-12T04:47:50.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:50.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:50.689 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:50.750 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:50.750 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:50.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:50.784+0000 7f731820b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:50.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:50.785+0000 7f731820b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:50.803 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:50.803 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:50.803 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:50.803 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:50.804 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:50.804 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:50.804 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:50.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:50.804 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:50.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:50.804 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:50.804 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:50.804 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:50.804 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:50.805 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:50.805 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:47:50.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:50.805 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:47:50.805 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:50.805 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:50.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:50.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:50.805 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:50.805 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:50.806 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:50.806 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:50.806 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:50.806 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:50.806 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:50.806 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:47:50.806 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:50.806 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:50.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:50.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:50.807 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:50.807 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:50.807 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:50.807 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:50.807 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:50.807 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:50.807 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:50.807 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:47:50.807 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:50.808 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:50.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:50.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:50.808 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:50.808 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:50.808 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:50.808 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897249.951095_974.1", 2024-04-12T04:47:50.808 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:50.808 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:50.808 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:50.809 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:29.951095Z", 2024-04-12T04:47:50.809 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:50.809 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:50.809 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:50.809 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:50.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:50.809 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:50.810 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:47:50.810 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:47:50.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:50.810 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:50Z" 2024-04-12T04:47:50.810 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:50.811 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897249.951095_974.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:29.951095Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:47:50Z" } 2024-04-12T04:47:50.811 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897249.951095_974.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:29.951095Z', 'realm_epoch': 2}}] 2024-04-12T04:47:50.812 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897249.951095_974.1'} 2024-04-12T04:47:50.812 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897249.951095_974.1'} 2024-04-12T04:47:50.812 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:47:50.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:50.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:50.813 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:47:50.863 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:50.864 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:50.917 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[]} 2024-04-12T04:47:50.917 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-74 2024-04-12T04:47:50.917 DEBUG:boto:path=/dkyges-74/ 2024-04-12T04:47:50.917 DEBUG:boto:auth_path=/dkyges-74/ 2024-04-12T04:47:50.917 DEBUG:boto:Method: PUT 2024-04-12T04:47:50.917 DEBUG:boto:Path: /dkyges-74/ 2024-04-12T04:47:50.918 DEBUG:boto:Data: 2024-04-12T04:47:50.918 DEBUG:boto:Headers: {} 2024-04-12T04:47:50.918 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:50.918 DEBUG:boto:Port: 8000 2024-04-12T04:47:50.918 DEBUG:boto:Params: {} 2024-04-12T04:47:50.918 DEBUG:boto:Token: None 2024-04-12T04:47:50.918 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:47:50 GMT /dkyges-74/ 2024-04-12T04:47:50.919 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:BuqpQeAJqVDJmuCR6kxMH8d1Ick= 2024-04-12T04:47:50.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BuqpQeAJqVDJmuCR6kxMH8d1Ick=', 'Content-Length': '0'} 2024-04-12T04:47:50.938 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f8bfd40710f6b04f-006618bcf6-2977-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:47:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:50.938 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-75 2024-04-12T04:47:50.938 DEBUG:boto:path=/dkyges-75/ 2024-04-12T04:47:50.938 DEBUG:boto:auth_path=/dkyges-75/ 2024-04-12T04:47:50.938 DEBUG:boto:Method: PUT 2024-04-12T04:47:50.938 DEBUG:boto:Path: /dkyges-75/ 2024-04-12T04:47:50.939 DEBUG:boto:Data: 2024-04-12T04:47:50.939 DEBUG:boto:Headers: {} 2024-04-12T04:47:50.939 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:50.939 DEBUG:boto:Port: 8000 2024-04-12T04:47:50.939 DEBUG:boto:Params: {} 2024-04-12T04:47:50.939 DEBUG:boto:Token: None 2024-04-12T04:47:50.939 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:47:50 GMT /dkyges-75/ 2024-04-12T04:47:50.939 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AxqjBiloykDmk3iLuNULH6fqpyM= 2024-04-12T04:47:50.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:50 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AxqjBiloykDmk3iLuNULH6fqpyM=', 'Content-Length': '0'} 2024-04-12T04:47:50.958 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005eb3e3e06d9d2821-006618bcf6-2977-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:47:50 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:50.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:50.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:50.959 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:51.018 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:51.018 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:51.061 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:47:51.078 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:47:51.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:51.078 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:51.078 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:47:51.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:51.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897270.955650_982.1", 2024-04-12T04:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:47:50.955650Z" 2024-04-12T04:47:51.079 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:51.080 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:47:51.081 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897270.955650_982.1'} 2024-04-12T04:47:51.081 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:47:51.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:51.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:51.082 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:51.136 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:51.136 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:51.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:51.170+0000 7f0453735780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:51.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:51.171+0000 7f0453735780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:51.190 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:51.190 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:51.190 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:51.190 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:51.190 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:51.190 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:51.190 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:51.191 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:51.192 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:51.193 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:51.194 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897270.927053_975.1", 2024-04-12T04:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:50.927053Z", 2024-04-12T04:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:51.195 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:51.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:51.196 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:51.196 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:47:51.196 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:47:51.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:51.196 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:51Z" 2024-04-12T04:47:51.196 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:51.197 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897270.927053_975.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:50.927053Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:47:51Z" } 2024-04-12T04:47:51.197 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897270.927053_975.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:50.927053Z', 'realm_epoch': 2}}] 2024-04-12T04:47:51.198 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897270.955650_982.1'} 2024-04-12T04:47:51.198 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897270.927053_975.1'} 2024-04-12T04:47:51.198 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897270.955650_982.1 target=1_1712897270.927053_975.1 2024-04-12T04:47:56.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:56.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:56.199 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:56.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:56.295+0000 7f7927932780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:56.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:56.297+0000 7f7927932780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:56.315 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:56.315 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:56.316 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:56.316 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:56.316 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:56.316 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:56.316 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:56.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:56.317 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:56.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:56.317 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:56.317 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:56.317 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:56.317 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:56.317 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:56.318 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:47:56.318 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:56.318 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:47:56.318 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:56.318 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:56.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:56.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:56.318 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:56.318 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:56.318 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:56.318 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:56.319 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:56.319 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:56.319 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:56.319 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:47:56.319 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:56.319 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:56.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:56.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:56.319 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:56.319 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:56.320 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:56.320 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:56.320 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:56.320 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:56.320 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:56.320 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:47:56.320 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:56.320 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:56.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:56.320 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:56.320 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:56.320 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:56.321 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:56.321 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897270.955650_982.1", 2024-04-12T04:47:56.321 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:56.321 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:56.321 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:56.321 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:50.955650Z", 2024-04-12T04:47:56.321 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:56.321 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:56.321 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:56.321 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:56.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:56.322 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:56.322 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:47:56.322 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:47:56.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:56.322 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:56Z" 2024-04-12T04:47:56.322 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:56.323 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897270.955650_982.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:50.955650Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:47:56Z" } 2024-04-12T04:47:56.323 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897270.955650_982.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:50.955650Z', 'realm_epoch': 2}}] 2024-04-12T04:47:56.323 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897270.955650_982.1'} 2024-04-12T04:47:56.323 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897270.955650_982.1'} 2024-04-12T04:47:56.324 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:47:56.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:56.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:56.324 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:47:56.379 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:56.379 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:56.434 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:47:56.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:56.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:56.435 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:47:56.494 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:56.494 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:56.550 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:47:56.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dkyges-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dkyges-74', '--dest-bucket=dkyges-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:56.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dkyges-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dkyges-74', '--dest-bucket=dkyges-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:56.551 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dkyges-74 --source-zones=a1 --dest-zones=a2 --source-bucket=dkyges-74 --dest-bucket=dkyges-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:47:56.610 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:56.610 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:56.665 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-dkyges-74","source":{"bucket":"dkyges-74","zones":["a1"]},"dest":{"bucket":"dkyges-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:47:56.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dkyges-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dkyges-75', '--dest-bucket=dkyges-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:56.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dkyges-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dkyges-75', '--dest-bucket=dkyges-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:56.666 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dkyges-75 --source-zones=a1 --dest-zones=a2 --source-bucket=dkyges-75 --dest-bucket=dkyges-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:47:56.725 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:56.725 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:56.781 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-dkyges-74","source":{"bucket":"dkyges-74","zones":["a1"]},"dest":{"bucket":"dkyges-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dkyges-75","source":{"bucket":"dkyges-75","zones":["a1"]},"dest":{"bucket":"dkyges-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:47:56.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:56.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:56.782 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:47:56.837 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:56.837 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:56.892 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-dkyges-74","source":{"bucket":"dkyges-74","zones":["a1"]},"dest":{"bucket":"dkyges-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dkyges-75","source":{"bucket":"dkyges-75","zones":["a1"]},"dest":{"bucket":"dkyges-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:47:56.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:56.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:56.892 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:47:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:57.007 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:57.007 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:57.007 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 20, 2024-04-12T04:47:57.007 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T04:47:57.007 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T04:47:57.007 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T04:47:57.007 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:57.008 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T04:47:57.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.008 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T04:47:57.008 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T04:47:57.008 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:47:57.008 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T04:47:57.008 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:47:57.008 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:47:57.008 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:47:57.008 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:47:57.009 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:47:57.009 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:47:57.009 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:47:57.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.009 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:47:57.009 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T04:47:57.009 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:47:57.009 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T04:47:57.009 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:47:57.010 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:47:57.010 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:47:57.010 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:47:57.010 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:47:57.010 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:47:57.010 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:47:57.010 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:47:57.010 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:47:57.010 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:47:57.010 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:47:57.010 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:47:57.011 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:57.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.011 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:47:57.011 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T04:47:57.011 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:47:57.011 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:47:57.011 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:47:57.011 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:47:57.011 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:47:57.011 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:47:57.012 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:47:57.012 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:47:57.012 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:47:57.012 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:47:57.012 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:47:57.012 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:47:57.012 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:47:57.012 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:47:57.013 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:57.013 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:57.013 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:47:57.013 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:47:57.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.013 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:47:57.013 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:47:57.013 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:47:57.013 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:47:57.013 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:57.013 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:57.014 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:47:57.014 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:47:57.014 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:47:57.014 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:47:57.014 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [ 2024-04-12T04:47:57.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.014 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-group", 2024-04-12T04:47:57.014 INFO:teuthology.orchestra.run.smithi043.stdout: "data_flow": { 2024-04-12T04:47:57.015 INFO:teuthology.orchestra.run.smithi043.stdout: "directional": [ 2024-04-12T04:47:57.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.015 INFO:teuthology.orchestra.run.smithi043.stdout: "source_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:47:57.015 INFO:teuthology.orchestra.run.smithi043.stdout: "dest_zone": "1ba046e0-d0ae-437d-b741-28b4299423c2" 2024-04-12T04:47:57.015 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:57.015 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:57.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.015 INFO:teuthology.orchestra.run.smithi043.stdout: "pipes": [ 2024-04-12T04:47:57.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-pipe-dkyges-74", 2024-04-12T04:47:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:47:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-74", 2024-04-12T04:47:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:47:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:47:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": { 2024-04-12T04:47:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-75", 2024-04-12T04:47:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:47:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2" 2024-04-12T04:47:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "params": { 2024-04-12T04:47:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:47:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "filter": { 2024-04-12T04:47:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [] 2024-04-12T04:47:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": {}, 2024-04-12T04:47:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-12T04:47:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "system", 2024-04-12T04:47:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "" 2024-04-12T04:47:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-pipe-dkyges-75", 2024-04-12T04:47:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:47:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-75", 2024-04-12T04:47:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:47:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:47:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": { 2024-04-12T04:47:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-75", 2024-04-12T04:47:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:47:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2" 2024-04-12T04:47:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "params": { 2024-04-12T04:47:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:47:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "filter": { 2024-04-12T04:47:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [] 2024-04-12T04:47:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": {}, 2024-04-12T04:47:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-12T04:47:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "system", 2024-04-12T04:47:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "" 2024-04-12T04:47:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:47:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "enabled" 2024-04-12T04:47:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:47:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:47:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T04:47:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T04:47:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:47:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T04:47:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:47:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:47:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:47:57.024 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:47:57.024 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:47:57.024 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:47:57.024 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:47:57.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.024 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:47:57.024 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T04:47:57.024 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:47:57.024 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:47:57.024 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:47:57.025 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:47:57.025 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:47:57.025 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:47:57.025 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:47:57.025 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:47:57.025 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:47:57.025 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:47:57.025 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:47:57.025 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:47:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:47:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:47:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:47:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T04:47:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:47:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T04:47:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:47:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:47:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:47:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:47:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:47:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:47:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:47:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:47:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:47:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:47:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:47:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:47:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:47:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:47:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:47:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:47:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:47:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:47:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:47:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:47:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:47:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:47:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T04:47:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:47:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:47:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:47:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T04:47:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:47:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T04:47:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:47:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T04:47:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:47:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T04:47:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:47:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T04:47:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T04:47:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:47:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T04:47:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T04:47:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:47:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:47:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:47:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:47:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:47:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T04:47:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:47:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:47:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:47:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:47:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:47:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T04:47:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:47:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:47:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:47:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:47:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:47:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T04:47:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:47:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:47:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:47:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:47:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:47:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T04:47:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:47:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:47:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:47:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:47:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:47:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:47:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T04:47:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:57.037 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:57.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:57.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:47:57.038 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:47:57.091 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:57.162 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-dkyges-74","source":{"bucket":"dkyges-74","zones":["a1"]},"dest":{"bucket":"dkyges-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dkyges-75","source":{"bucket":"dkyges-75","zones":["a1"]},"dest":{"bucket":"dkyges-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:47:57.163 DEBUG:boto:path=/dkyges-74/ 2024-04-12T04:47:57.163 DEBUG:boto:auth_path=/dkyges-74/ 2024-04-12T04:47:57.163 DEBUG:boto:Method: HEAD 2024-04-12T04:47:57.163 DEBUG:boto:Path: /dkyges-74/ 2024-04-12T04:47:57.163 DEBUG:boto:Data: 2024-04-12T04:47:57.163 DEBUG:boto:Headers: {} 2024-04-12T04:47:57.164 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:57.164 DEBUG:boto:Port: 8000 2024-04-12T04:47:57.164 DEBUG:boto:Params: {} 2024-04-12T04:47:57.164 DEBUG:boto:Token: None 2024-04-12T04:47:57.164 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:47:57 GMT /dkyges-74/ 2024-04-12T04:47:57.164 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mzy+HFW003qIFC4Oe+3nQV3PWwE= 2024-04-12T04:47:57.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mzy+HFW003qIFC4Oe+3nQV3PWwE=', 'Content-Length': '0'} 2024-04-12T04:47:57.438 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d5e24b72a02f081-006618bcfd-2bf8-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:47:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:57.439 DEBUG:boto:path=/dkyges-74/obj1 2024-04-12T04:47:57.439 DEBUG:boto:auth_path=/dkyges-74/obj1 2024-04-12T04:47:57.439 DEBUG:boto:Method: PUT 2024-04-12T04:47:57.439 DEBUG:boto:Path: /dkyges-74/obj1 2024-04-12T04:47:57.439 DEBUG:boto:Data: 2024-04-12T04:47:57.440 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:47:57.440 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:57.440 DEBUG:boto:Port: 8000 2024-04-12T04:47:57.440 DEBUG:boto:Params: {} 2024-04-12T04:47:57.440 DEBUG:boto:Token: None 2024-04-12T04:47:57.440 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:47:57 GMT /dkyges-74/obj1 2024-04-12T04:47:57.440 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:t/7f9s7N9j4k2TlrhubvffD2wYk= 2024-04-12T04:47:57.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:47:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:t/7f9s7N9j4k2TlrhubvffD2wYk='} 2024-04-12T04:47:57.485 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000053772ffb1e02abc3-006618bcfd-2bf8-a1'), ('Date', 'Fri, 12 Apr 2024 04:47:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:57.485 DEBUG:boto:path=/dkyges-74/ 2024-04-12T04:47:57.485 DEBUG:boto:auth_path=/dkyges-74/ 2024-04-12T04:47:57.485 DEBUG:boto:Method: HEAD 2024-04-12T04:47:57.485 DEBUG:boto:Path: /dkyges-74/ 2024-04-12T04:47:57.485 DEBUG:boto:Data: 2024-04-12T04:47:57.486 DEBUG:boto:Headers: {} 2024-04-12T04:47:57.486 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:57.486 DEBUG:boto:Port: 8000 2024-04-12T04:47:57.486 DEBUG:boto:Params: {} 2024-04-12T04:47:57.486 DEBUG:boto:Token: None 2024-04-12T04:47:57.486 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:47:57 GMT /dkyges-74/ 2024-04-12T04:47:57.486 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mzy+HFW003qIFC4Oe+3nQV3PWwE= 2024-04-12T04:47:57.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mzy+HFW003qIFC4Oe+3nQV3PWwE=', 'Content-Length': '0'} 2024-04-12T04:47:57.491 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000016be41426e643db3-006618bcfd-2bf8-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:47:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:57.492 DEBUG:boto:path=/dkyges-74/obj2 2024-04-12T04:47:57.492 DEBUG:boto:auth_path=/dkyges-74/obj2 2024-04-12T04:47:57.492 DEBUG:boto:Method: PUT 2024-04-12T04:47:57.492 DEBUG:boto:Path: /dkyges-74/obj2 2024-04-12T04:47:57.492 DEBUG:boto:Data: 2024-04-12T04:47:57.492 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:47:57.492 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:57.492 DEBUG:boto:Port: 8000 2024-04-12T04:47:57.493 DEBUG:boto:Params: {} 2024-04-12T04:47:57.493 DEBUG:boto:Token: None 2024-04-12T04:47:57.493 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:47:57 GMT /dkyges-74/obj2 2024-04-12T04:47:57.493 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xjTodDdl7FebSKWgK/PIdOgpRuQ= 2024-04-12T04:47:57.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:47:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xjTodDdl7FebSKWgK/PIdOgpRuQ='} 2024-04-12T04:47:57.538 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000168d775668f61b49-006618bcfd-2bf8-a1'), ('Date', 'Fri, 12 Apr 2024 04:47:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:57.539 DEBUG:boto:path=/dkyges-75/ 2024-04-12T04:47:57.539 DEBUG:boto:auth_path=/dkyges-75/ 2024-04-12T04:47:57.539 DEBUG:boto:Method: HEAD 2024-04-12T04:47:57.539 DEBUG:boto:Path: /dkyges-75/ 2024-04-12T04:47:57.539 DEBUG:boto:Data: 2024-04-12T04:47:57.539 DEBUG:boto:Headers: {} 2024-04-12T04:47:57.539 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:57.539 DEBUG:boto:Port: 8000 2024-04-12T04:47:57.539 DEBUG:boto:Params: {} 2024-04-12T04:47:57.540 DEBUG:boto:Token: None 2024-04-12T04:47:57.540 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:47:57 GMT /dkyges-75/ 2024-04-12T04:47:57.540 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cOC6X5potRXKIRSGZPPa5jr7+Ys= 2024-04-12T04:47:57.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cOC6X5potRXKIRSGZPPa5jr7+Ys=', 'Content-Length': '0'} 2024-04-12T04:47:57.549 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f521bf2c2098944b-006618bcfd-2bf8-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:47:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:57.549 DEBUG:boto:path=/dkyges-75/obj3 2024-04-12T04:47:57.549 DEBUG:boto:auth_path=/dkyges-75/obj3 2024-04-12T04:47:57.549 DEBUG:boto:Method: PUT 2024-04-12T04:47:57.549 DEBUG:boto:Path: /dkyges-75/obj3 2024-04-12T04:47:57.549 DEBUG:boto:Data: 2024-04-12T04:47:57.549 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:47:57.550 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:57.550 DEBUG:boto:Port: 8000 2024-04-12T04:47:57.550 DEBUG:boto:Params: {} 2024-04-12T04:47:57.550 DEBUG:boto:Token: None 2024-04-12T04:47:57.550 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:47:57 GMT /dkyges-75/obj3 2024-04-12T04:47:57.550 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:kFJ+OxkN0sjGHaFQeUCDjR8PmCg= 2024-04-12T04:47:57.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:47:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kFJ+OxkN0sjGHaFQeUCDjR8PmCg='} 2024-04-12T04:47:57.569 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058bb33a3718af95f-006618bcfd-2bf8-a1'), ('Date', 'Fri, 12 Apr 2024 04:47:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:57.569 DEBUG:boto:path=/dkyges-75/ 2024-04-12T04:47:57.569 DEBUG:boto:auth_path=/dkyges-75/ 2024-04-12T04:47:57.569 DEBUG:boto:Method: HEAD 2024-04-12T04:47:57.570 DEBUG:boto:Path: /dkyges-75/ 2024-04-12T04:47:57.570 DEBUG:boto:Data: 2024-04-12T04:47:57.570 DEBUG:boto:Headers: {} 2024-04-12T04:47:57.570 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:57.570 DEBUG:boto:Port: 8000 2024-04-12T04:47:57.570 DEBUG:boto:Params: {} 2024-04-12T04:47:57.570 DEBUG:boto:Token: None 2024-04-12T04:47:57.570 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:47:57 GMT /dkyges-75/ 2024-04-12T04:47:57.570 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cOC6X5potRXKIRSGZPPa5jr7+Ys= 2024-04-12T04:47:57.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:47:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cOC6X5potRXKIRSGZPPa5jr7+Ys=', 'Content-Length': '0'} 2024-04-12T04:47:57.576 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007430535e50a6cb5-006618bcfd-2bf8-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:47:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:57.576 DEBUG:boto:path=/dkyges-75/obj4 2024-04-12T04:47:57.577 DEBUG:boto:auth_path=/dkyges-75/obj4 2024-04-12T04:47:57.577 DEBUG:boto:Method: PUT 2024-04-12T04:47:57.577 DEBUG:boto:Path: /dkyges-75/obj4 2024-04-12T04:47:57.577 DEBUG:boto:Data: 2024-04-12T04:47:57.577 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:47:57.577 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:47:57.577 DEBUG:boto:Port: 8000 2024-04-12T04:47:57.577 DEBUG:boto:Params: {} 2024-04-12T04:47:57.577 DEBUG:boto:Token: None 2024-04-12T04:47:57.578 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:47:57 GMT /dkyges-75/obj4 2024-04-12T04:47:57.578 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Oga+DyArsMQveZ8INUqR+PQBo0k= 2024-04-12T04:47:57.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:47:57 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Oga+DyArsMQveZ8INUqR+PQBo0k='} 2024-04-12T04:47:57.622 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075f22b9604501c95-006618bcfd-2bf8-a1'), ('Date', 'Fri, 12 Apr 2024 04:47:57 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:47:57.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:57.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:57.623 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:57.712 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:57.712 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:57.820 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:47:57.850 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:47:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:47:57.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:47:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:57.851 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:47:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897270.955650_982.1", 2024-04-12T04:47:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:47:50.955650Z" 2024-04-12T04:47:57.852 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:57.852 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:47:57.853 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897270.955650_982.1'} 2024-04-12T04:47:57.853 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:47:57.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:57.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:57.854 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:57.934 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:57.934 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:57.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:57.992+0000 7f37569de780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:58.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:47:57.994+0000 7f37569de780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:47:58.099 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:58.099 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:58.099 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:58.099 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:58.099 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:58.099 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:47:58.099 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:58.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:47:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:47:58.100 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:47:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:47:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:58.101 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:47:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:47:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:58.102 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:47:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:47:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897270.955650_982.1", 2024-04-12T04:47:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:58.103 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:50.955650Z", 2024-04-12T04:47:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:47:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:47:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:47:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:58.104 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:57Z" 2024-04-12T04:47:58.105 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:58.105 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897270.955650_982.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:50.955650Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:47:57Z" } 2024-04-12T04:47:58.105 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897270.955650_982.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:50.955650Z', 'realm_epoch': 2}}] 2024-04-12T04:47:58.105 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897270.955650_982.1'} 2024-04-12T04:47:58.105 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897270.955650_982.1'} 2024-04-12T04:47:58.105 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:47:58.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:58.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:58.106 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:58.166 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:58.167 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:58.229 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:47:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:47:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:47:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:47:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:47:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:58.230 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:47:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:47:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:47:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:47:58.231 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:58.231 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:47:58.232 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:47:58.232 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:47:58.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:58.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:47:58.233 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:47:58.292 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:47:58.293 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:47:58.346 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:47:58.346 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:47:58.346 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:47:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:47:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:47:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:47:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:47:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:47:58.347 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:47:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:47:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:47:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:47:58.348 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:47:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:47:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:47:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:47:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:58.349 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:47:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:47:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:47:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:47:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:47:58.350 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:47:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:47:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:47:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:47:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:47:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:47:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:47:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:47:58.351 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:47:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:47:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:47:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:47:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:47:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:47:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:47:58.352 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:47:58Z" 2024-04-12T04:47:58.352 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:47:58.353 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:47:58Z" } 2024-04-12T04:47:58.354 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:47:58.355 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:47:58.355 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:47:58.355 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:48:03.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:03.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:03.357 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:48:03.418 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:48:03.419 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:48:03.477 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:48:03.477 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:48:03.477 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:48:03.477 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:48:03.477 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:48:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:48:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:48:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:48:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:48:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:03.478 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:48:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:48:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:48:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:03.479 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:48:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:48:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:48:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:48:03.480 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:48:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:48:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:48:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:48:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:48:03.481 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:48:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:48:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:48:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:48:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:03.482 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:48:03Z" 2024-04-12T04:48:03.482 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:48:03.483 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:48:03Z" } 2024-04-12T04:48:03.483 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:48:03.484 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:03.484 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:03.484 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:48:08.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:08.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:08.487 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:48:08.550 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:48:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:48:08.606 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:48:08.606 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:48:08.606 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:48:08.606 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:48:08.606 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:48:08.606 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:48:08.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:08.607 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:48:08.607 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:08.607 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:48:08.607 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:08.607 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:08.607 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:48:08.607 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:08.607 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:08.607 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:08.607 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:48:08.608 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:08.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:08.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:08.608 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:48:08.608 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:08.608 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:08.608 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:48:08.608 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:08.608 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:08.608 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:08.608 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:48:08.609 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:08.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:08.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:08.609 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:48:08.609 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:08.609 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:08.609 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:48:08.609 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:08.609 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:48:08.609 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:08.609 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:48:08.609 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:08.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:08.610 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:08.610 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:48:08.610 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:08.610 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:08.610 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:48:08.610 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:08.610 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:48:08.610 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:08.610 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:48:08.611 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:08.611 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:08.611 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:48:08.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:08.611 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:48:08.611 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:48:08.611 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:48:08.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:08.611 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:48:08Z" 2024-04-12T04:48:08.611 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:48:08.612 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:48:08Z" } 2024-04-12T04:48:08.612 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:48:08.613 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:08.613 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:08.613 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:48:13.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:13.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:13.614 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:48:13.676 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:48:13.676 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:48:13.733 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:48:13.733 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:48:13.733 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:48:13.734 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:48:13.734 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:48:13.734 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:48:13.734 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:13.734 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:48:13.734 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:13.734 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:48:13.734 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:13.734 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:13.734 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:48:13.734 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:13.735 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:13.735 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:13.735 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:48:13.735 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:13.735 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:13.735 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:13.735 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:48:13.735 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:13.735 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:13.735 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:48:13.735 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:13.735 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:13.735 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:13.736 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:48:13.736 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:13.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:13.736 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:13.736 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:48:13.736 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:13.736 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:13.736 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:48:13.736 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:13.736 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:48:13.736 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:13.736 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:48:13.737 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:13.737 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:13.737 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:13.737 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:48:13.737 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:13.737 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:13.737 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:48:13.737 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:13.737 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:48:13.737 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:13.738 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:48:13.738 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:13.738 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:13.738 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:48:13.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:13.738 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:48:13.738 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:48:13.738 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:48:13.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:13.738 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:48:13Z" 2024-04-12T04:48:13.738 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:48:13.739 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:48:13Z" } 2024-04-12T04:48:13.739 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:48:13.740 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:13.740 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:13.740 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:48:18.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:18.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:18.741 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:48:18.802 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:48:18.802 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:48:18.860 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:48:18.861 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:48:18.861 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:48:18.861 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:48:18.861 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:48:18.861 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:48:18.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:18.862 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:48:18.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:18.862 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:48:18.862 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:18.862 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:18.862 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:48:18.863 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:18.863 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:18.863 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:18.863 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:48:18.863 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:18.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:18.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:18.863 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:48:18.863 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:18.863 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:18.864 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:48:18.864 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:18.864 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:18.864 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:18.864 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:48:18.864 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:18.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:18.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:18.864 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:48:18.864 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:18.864 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:18.865 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:48:18.865 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:18.865 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:48:18.865 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:18.865 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:48:18.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:18.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:18.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:18.865 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:48:18.865 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:18.865 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:48:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:48:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:48:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:48:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:48:18.866 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:48:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:48:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:18.867 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:48:18Z" 2024-04-12T04:48:18.867 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:48:18.868 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:48:18Z" } 2024-04-12T04:48:18.868 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:48:18.868 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:18.868 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:18.868 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:48:23.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:23.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:23.870 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:48:23.932 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:48:23.932 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:48:23.989 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:48:23.989 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:48:23.990 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:48:23.990 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:48:23.990 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:48:23.990 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:48:23.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:23.990 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:48:23.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:23.991 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:48:23.991 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:23.991 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:23.991 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:48:23.991 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:23.991 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:23.991 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:23.991 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:48:23.991 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:23.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:23.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:23.992 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:48:23.992 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:23.992 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:23.992 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:48:23.992 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:23.992 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:23.992 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:23.992 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:48:23.992 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:23.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:23.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:23.993 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:48:23.993 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:23.993 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:23.993 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:48:23.993 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:23.993 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:48:23.993 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:23.993 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:48:23.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:23.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:23.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:23.994 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:48:23.994 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:23.994 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:23.994 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:48:23.994 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:23.994 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:48:23.994 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:23.994 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:48:23.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:23.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:23.995 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:48:23.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:23.995 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:48:23.996 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:48:23.996 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:48:23.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:23.996 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:48:23Z" 2024-04-12T04:48:23.996 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:48:23.997 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:48:23Z" } 2024-04-12T04:48:23.997 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:48:23.997 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:23.997 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:23.998 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:48:28.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:28.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:28.999 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:48:29.060 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:48:29.060 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:48:29.116 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:48:29.116 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:48:29.116 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:48:29.116 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:48:29.116 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:48:29.117 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:48:29.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:29.117 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:48:29.117 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:29.117 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:48:29.117 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:29.117 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:29.117 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:48:29.117 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:29.117 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:29.117 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:29.118 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:48:29.118 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:29.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:29.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:29.118 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:48:29.118 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:29.118 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:29.118 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:48:29.118 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:29.118 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:29.118 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:29.119 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:48:29.119 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:29.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:29.119 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:29.119 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:48:29.119 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:29.119 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:29.119 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:48:29.119 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:29.119 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:48:29.119 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:29.120 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:48:29.120 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:29.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:29.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:29.120 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:48:29.120 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:29.120 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:29.120 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:48:29.120 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:29.120 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:48:29.120 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:29.120 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:48:29.121 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:29.121 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:29.121 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:48:29.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:29.121 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:48:29.121 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:48:29.121 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:48:29.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:29.121 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:48:29Z" 2024-04-12T04:48:29.121 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:48:29.122 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:48:29Z" } 2024-04-12T04:48:29.122 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:48:29.123 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:29.123 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:29.123 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:48:34.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:34.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:34.124 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:48:34.186 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:48:34.186 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:48:34.241 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:48:34.241 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:48:34.241 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:48:34.242 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:48:34.242 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:48:34.242 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:48:34.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:34.242 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:48:34.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:34.243 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:48:34.243 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:34.243 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:34.243 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:48:34.244 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:34.244 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:34.244 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:34.244 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:48:34.244 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:34.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:34.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:34.245 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:48:34.245 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:34.245 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:34.245 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:48:34.245 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:34.246 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:34.246 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:34.246 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:48:34.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:34.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:34.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:34.247 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:48:34.247 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:34.247 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:34.247 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:48:34.248 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:34.248 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:48:34.248 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:34.248 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:48:34.248 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:34.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:34.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:34.249 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:48:34.249 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:34.249 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:34.250 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:48:34.250 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:34.250 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:48:34.251 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:34.251 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:48:34.252 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:34.252 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:34.252 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:48:34.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:34.252 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:48:34.252 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:48:34.252 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:48:34.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:34.253 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:48:34Z" 2024-04-12T04:48:34.254 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:48:34.255 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:48:34Z" } 2024-04-12T04:48:34.255 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:48:34.255 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:34.255 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:34.255 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:48:39.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:39.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:39.257 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:48:39.323 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:48:39.323 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:48:39.379 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:48:39.380 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:48:39.380 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:48:39.380 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:48:39.380 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:48:39.380 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:48:39.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:39.380 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:48:39.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:39.381 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:48:39.381 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:39.381 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:39.381 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:48:39.381 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:39.381 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:39.381 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:48:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:48:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:48:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:48:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:48:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:48:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:48:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:48:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:48:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:48:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:48:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:48:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:48:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:48:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:48:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:48:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:48:39Z" 2024-04-12T04:48:39.386 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:48:39.387 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:48:39Z" } 2024-04-12T04:48:39.387 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:48:39.387 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:39.387 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:39.387 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:48:44.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:44.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:44.390 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:48:44.451 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:48:44.451 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:48:44.506 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:48:44.506 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:48:44.506 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:48:44.506 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:48:44.506 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:48:44.507 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:48:44.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:44.507 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:48:44.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:44.507 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:48:44.507 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:44.507 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:44.507 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:48:44.507 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:44.508 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:44.508 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:44.508 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:48:44.508 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:44.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:44.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:44.508 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:48:44.508 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:44.508 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:44.509 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:48:44.509 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:44.509 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:44.509 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:44.509 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:48:44.509 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:44.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:44.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:44.509 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:48:44.509 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:44.509 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:44.510 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:48:44.510 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:44.510 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:48:44.510 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:44.510 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:48:44.510 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:44.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:44.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:44.510 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:48:44.510 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:44.510 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:44.510 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:48:44.511 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:44.511 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:48:44.511 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:44.511 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:48:44.511 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:44.511 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:44.511 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:48:44.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:44.511 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:48:44.511 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:48:44.511 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:48:44.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:44.512 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:48:44Z" 2024-04-12T04:48:44.512 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:48:44.514 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:48:44Z" } 2024-04-12T04:48:44.514 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:48:44.515 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:44.515 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:44.515 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:48:49.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:49.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:49.517 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:48:49.579 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:48:49.579 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:48:49.637 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:48:49.637 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:48:49.637 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:48:49.637 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:48:49.637 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:48:49.638 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:48:49.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:49.638 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:48:49.638 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:49.638 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:48:49.638 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:49.638 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:49.638 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:48:49.638 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:49.638 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:49.639 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:49.639 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:48:49.639 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:49.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:49.639 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:49.639 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:48:49.639 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:49.639 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:49.639 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:48:49.639 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:49.640 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:49.640 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:49.640 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:48:49.640 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:49.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:49.640 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:49.640 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:48:49.640 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:49.640 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:49.640 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:48:49.640 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:49.641 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:48:49.641 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:49.641 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:48:49.641 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:49.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:49.641 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:49.641 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:48:49.641 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:49.641 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:49.641 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:48:49.641 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:49.642 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:48:49.642 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:49.642 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:48:49.642 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:49.642 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:49.642 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:48:49.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:49.642 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:48:49.642 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:48:49.642 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:48:49.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:49.643 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:48:49Z" 2024-04-12T04:48:49.643 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:48:49.644 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:48:49Z" } 2024-04-12T04:48:49.644 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:48:49.644 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:49.644 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:49.644 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:48:54.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:54.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:54.646 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:48:54.709 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:48:54.710 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:48:54.768 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:48:54.769 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:48:54.769 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:48:54.769 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:48:54.769 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:48:54.769 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:48:54.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:54.769 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:48:54.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:54.769 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:48:54.770 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:54.770 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:54.770 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:48:54.770 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:54.770 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:54.770 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:54.770 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:48:54.770 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:54.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:54.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:54.771 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:48:54.771 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:54.771 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:54.771 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:48:54.771 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:54.771 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:54.771 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:54.771 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:48:54.771 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:54.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:54.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:54.772 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:48:54.772 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:54.772 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:54.772 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:48:54.772 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:54.772 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:48:54.772 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:54.772 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:48:54.773 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:54.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:54.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:54.773 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:48:54.773 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:54.773 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:54.773 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:48:54.773 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:54.773 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:48:54.773 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:54.774 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:48:54.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:54.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:54.774 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:48:54.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:54.774 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:48:54.774 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:48:54.774 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:48:54.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:54.774 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:48:54Z" 2024-04-12T04:48:54.775 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:48:54.776 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:48:54Z" } 2024-04-12T04:48:54.776 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:48:54.776 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:54.776 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:54.776 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:48:59.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:59.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:48:59.778 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:48:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:48:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:48:59.899 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:48:59.899 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:48:59.899 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:48:59.899 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:48:59.899 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:48:59.899 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:48:59.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:59.899 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:48:59.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:59.899 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:48:59.900 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:59.900 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:59.900 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:48:59.900 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:59.900 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:59.900 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:59.900 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:48:59.900 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:59.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:59.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:59.900 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:48:59.901 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:59.901 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:59.901 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:48:59.901 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:59.901 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:48:59.901 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:59.901 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:48:59.901 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:59.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:59.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:59.901 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:48:59.901 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:59.901 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:59.902 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:48:59.902 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:59.902 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:48:59.902 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:59.902 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:48:59.902 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:59.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:59.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:48:59.902 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:48:59.902 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:48:59.903 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:48:59.903 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:48:59.903 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:48:59.903 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:48:59.903 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:48:59.903 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:48:59.903 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:59.903 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:48:59.903 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:48:59.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:59.903 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:48:59.903 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:48:59.904 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:48:59.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:48:59.904 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:48:59Z" 2024-04-12T04:48:59.904 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:48:59.905 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:48:59Z" } 2024-04-12T04:48:59.905 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:48:59.905 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:59.905 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:48:59.905 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:49:04.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:04.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:04.906 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:49:04.968 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:04.969 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:05.024 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:49:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:49:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:49:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:49:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:49:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:49:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:49:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:49:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:49:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:49:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:49:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:49:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:49:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:49:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:49:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:49:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:49:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:49:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:49:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:49:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:49:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:49:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:49:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:49:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:49:05Z" 2024-04-12T04:49:05.029 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:49:05.030 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:49:05Z" } 2024-04-12T04:49:05.033 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:49:05.033 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:49:05.034 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:49:05.034 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:49:10.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:10.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:10.035 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:49:10.097 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:10.098 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:10.156 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:49:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:49:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:49:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:49:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:49:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:49:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:49:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:49:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:49:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:49:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:49:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:49:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:49:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:49:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:49:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:49:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:49:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:49:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:49:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:49:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:49:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:49:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:49:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:49:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:49:10Z" 2024-04-12T04:49:10.161 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:49:10.162 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:49:10Z" } 2024-04-12T04:49:10.162 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:49:10.162 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:49:10.162 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:49:10.163 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:49:15.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:15.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:15.164 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:49:15.227 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:15.227 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:15.282 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:49:15.282 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:49:15.282 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:49:15.283 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:49:15.283 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:49:15.283 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:49:15.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:15.283 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:49:15.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:15.283 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:49:15.283 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:15.283 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:15.283 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:49:15.283 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:15.284 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:15.284 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:15.284 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:49:15.284 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:15.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:15.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:15.284 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:49:15.284 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:15.284 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:15.284 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:49:15.285 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:15.285 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:15.285 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:15.285 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:49:15.285 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:15.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:15.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:15.285 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:49:15.285 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:15.285 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:15.285 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:49:15.286 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:15.286 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:49:15.286 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:15.286 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:49:15.286 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:15.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:15.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:15.286 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:49:15.286 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:15.286 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:15.286 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:49:15.287 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:15.287 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:49:15.287 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:15.287 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:49:15.287 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:15.287 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:15.287 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:15.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:15.287 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:49:15.287 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:49:15.288 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:49:15.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:15.288 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:49:15Z" 2024-04-12T04:49:15.288 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:49:15.289 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:49:15Z" } 2024-04-12T04:49:15.289 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:49:15.289 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:49:15.289 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:49:15.289 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:49:20.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:20.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:20.291 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:49:20.354 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:20.354 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:20.408 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:49:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:49:20.408 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:49:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:49:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:49:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:49:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:49:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:49:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:20.409 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:49:20.410 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:20.410 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:20.410 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:20.410 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:49:20.410 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:20.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:20.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:20.410 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:49:20.410 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:49:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:49:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:20.411 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:49:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:49:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:49:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:49:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:20.412 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:49:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:49:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:49:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:49:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:20.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:20.414 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:49:20.414 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:49:20.414 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:49:20.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:20.414 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:49:20Z" 2024-04-12T04:49:20.414 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:49:20.415 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:49:20Z" } 2024-04-12T04:49:20.415 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:49:20.415 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:49:20.415 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:49:20.416 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:49:25.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:25.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:25.418 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:49:25.479 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:25.479 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:25.534 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:49:25.535 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:49:25.535 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:49:25.535 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:49:25.535 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:49:25.535 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:49:25.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:25.535 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:49:25.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:25.535 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:49:25.536 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:25.536 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:25.536 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000047918", 2024-04-12T04:49:25.536 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:25.536 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:25.536 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:25.536 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.178517Z" 2024-04-12T04:49:25.536 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:25.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:25.537 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:25.537 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:49:25.537 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:25.537 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:25.537 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032765", 2024-04-12T04:49:25.537 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:25.537 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:25.537 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:25.537 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:27.990677Z" 2024-04-12T04:49:25.537 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:25.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:25.538 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:25.538 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:49:25.538 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:25.538 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:25.538 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054630", 2024-04-12T04:49:25.538 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:25.538 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:49:25.538 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:25.538 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:46:31.165194Z" 2024-04-12T04:49:25.538 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:25.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:25.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:25.539 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:49:25.539 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:25.539 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:25.539 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:49:25.539 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:25.539 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:49:25.539 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:25.539 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:49:25.539 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:25.540 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:25.540 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:25.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:25.540 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:49:25.540 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:49:25.540 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:49:25.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:25.540 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:49:25Z" 2024-04-12T04:49:25.540 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:49:25.541 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000047918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:46:31.178517Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032765", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:27.990677Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054630", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:46:31.165194Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:49:25Z" } 2024-04-12T04:49:25.541 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.178517Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032765', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:27.990677Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054630', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:46:31.165194Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:49:25.542 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:49:25.542 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000047918', 1: '00000000000000000000:00000000000000032765', 2: '00000000000000000000:00000000000000054630', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:49:25.542 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048058 target=00000000000000000000:00000000000000047918, shard=1 master=00000000000000000000:00000000000000032905 target=00000000000000000000:00000000000000032765, shard=2 master=00000000000000000000:00000000000000054911 target=00000000000000000000:00000000000000054630 2024-04-12T04:49:28.475 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T04:49:28.474+0000 7f23e5650700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:49:30.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:30.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:30.543 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:49:30.606 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:30.606 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:30.663 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:49:30.663 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:49:30.664 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:49:30.664 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:49:30.664 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:49:30.664 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:49:30.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:30.664 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:49:30.664 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:30.664 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:49:30.664 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:30.664 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:30.664 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:49:30.665 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:30.665 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:30.665 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:30.665 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:49:30.665 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:30.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:30.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:30.665 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:49:30.665 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:30.665 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:30.665 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:49:30.665 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:30.666 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:30.666 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:30.666 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:49:30.666 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:30.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:30.666 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:30.666 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:49:30.666 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:30.666 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:30.666 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:49:30.666 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:30.667 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:49:30.667 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:30.667 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:49:30.667 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:30.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:30.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:30.667 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:49:30.667 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:30.667 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:49:30.667 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:49:30.667 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:30.667 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:49:30.668 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:30.668 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:49:30.668 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:30.668 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:30.668 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:30.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:30.668 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:49:30.668 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:49:30.668 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:49:30.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:30.668 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:49:30Z" 2024-04-12T04:49:30.668 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:49:30.669 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:49:30Z" } 2024-04-12T04:49:30.670 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:49:30.670 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:49:30.670 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:49:30.670 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:49:30.670 DEBUG:boto:path=/dkyges-74/ 2024-04-12T04:49:30.670 DEBUG:boto:auth_path=/dkyges-74/ 2024-04-12T04:49:30.670 DEBUG:boto:Method: HEAD 2024-04-12T04:49:30.671 DEBUG:boto:Path: /dkyges-74/ 2024-04-12T04:49:30.671 DEBUG:boto:Data: 2024-04-12T04:49:30.671 DEBUG:boto:Headers: {} 2024-04-12T04:49:30.671 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:49:30.671 DEBUG:boto:Port: 8001 2024-04-12T04:49:30.671 DEBUG:boto:Params: {} 2024-04-12T04:49:30.671 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:49:30.672 DEBUG:boto:Token: None 2024-04-12T04:49:30.672 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:49:30 GMT /dkyges-74/ 2024-04-12T04:49:30.672 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hbZk2R3jSxz2OeJpdYSG/rc/5zE= 2024-04-12T04:49:30.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:49:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hbZk2R3jSxz2OeJpdYSG/rc/5zE=', 'Content-Length': '0'} 2024-04-12T04:49:30.687 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca9db5ba45981615-006618bd5a-2bef-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:49:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:49:30.687 DEBUG:boto:path=/dkyges-74/obj1 2024-04-12T04:49:30.687 DEBUG:boto:auth_path=/dkyges-74/obj1 2024-04-12T04:49:30.687 DEBUG:boto:Method: HEAD 2024-04-12T04:49:30.688 DEBUG:boto:Path: /dkyges-74/obj1 2024-04-12T04:49:30.688 DEBUG:boto:Data: 2024-04-12T04:49:30.688 DEBUG:boto:Headers: {} 2024-04-12T04:49:30.688 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:49:30.688 DEBUG:boto:Port: 8001 2024-04-12T04:49:30.688 DEBUG:boto:Params: {} 2024-04-12T04:49:30.688 DEBUG:boto:Token: None 2024-04-12T04:49:30.688 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:49:30 GMT /dkyges-74/obj1 2024-04-12T04:49:30.688 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:609CTu43c26R5zRrFjq85sLOMDQ= 2024-04-12T04:49:30.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:49:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:609CTu43c26R5zRrFjq85sLOMDQ=', 'Content-Length': '0'} 2024-04-12T04:49:30.690 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f84fbc91148e85c0-006618bd5a-2bef-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:49:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:49:30.690 DEBUG:boto:path=/dkyges-74/obj2 2024-04-12T04:49:30.690 DEBUG:boto:auth_path=/dkyges-74/obj2 2024-04-12T04:49:30.690 DEBUG:boto:Method: HEAD 2024-04-12T04:49:30.690 DEBUG:boto:Path: /dkyges-74/obj2 2024-04-12T04:49:30.690 DEBUG:boto:Data: 2024-04-12T04:49:30.690 DEBUG:boto:Headers: {} 2024-04-12T04:49:30.690 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:49:30.691 DEBUG:boto:Port: 8001 2024-04-12T04:49:30.691 DEBUG:boto:Params: {} 2024-04-12T04:49:30.691 DEBUG:boto:Token: None 2024-04-12T04:49:30.691 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:49:30 GMT /dkyges-74/obj2 2024-04-12T04:49:30.691 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1cSrcVMZq5W30pz3DfWt0jRZAp0= 2024-04-12T04:49:30.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:49:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1cSrcVMZq5W30pz3DfWt0jRZAp0=', 'Content-Length': '0'} 2024-04-12T04:49:30.692 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f9b43faa6ab2790d-006618bd5a-2bef-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:49:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:49:30.692 DEBUG:boto:path=/dkyges-74/obj3 2024-04-12T04:49:30.693 DEBUG:boto:auth_path=/dkyges-74/obj3 2024-04-12T04:49:30.693 DEBUG:boto:Method: HEAD 2024-04-12T04:49:30.693 DEBUG:boto:Path: /dkyges-74/obj3 2024-04-12T04:49:30.693 DEBUG:boto:Data: 2024-04-12T04:49:30.693 DEBUG:boto:Headers: {} 2024-04-12T04:49:30.693 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:49:30.693 DEBUG:boto:Port: 8001 2024-04-12T04:49:30.693 DEBUG:boto:Params: {} 2024-04-12T04:49:30.693 DEBUG:boto:Token: None 2024-04-12T04:49:30.693 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:49:30 GMT /dkyges-74/obj3 2024-04-12T04:49:30.694 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zq8+ZTLK/TC6SlR0t7W/hiD5gBc= 2024-04-12T04:49:30.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:49:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zq8+ZTLK/TC6SlR0t7W/hiD5gBc=', 'Content-Length': '0'} 2024-04-12T04:49:30.695 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000637e98ff8ae4a6a7-006618bd5a-2bef-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:49:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:49:30.695 DEBUG:boto:path=/dkyges-74/obj4 2024-04-12T04:49:30.695 DEBUG:boto:auth_path=/dkyges-74/obj4 2024-04-12T04:49:30.696 DEBUG:boto:Method: HEAD 2024-04-12T04:49:30.696 DEBUG:boto:Path: /dkyges-74/obj4 2024-04-12T04:49:30.696 DEBUG:boto:Data: 2024-04-12T04:49:30.696 DEBUG:boto:Headers: {} 2024-04-12T04:49:30.696 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:49:30.696 DEBUG:boto:Port: 8001 2024-04-12T04:49:30.696 DEBUG:boto:Params: {} 2024-04-12T04:49:30.696 DEBUG:boto:Token: None 2024-04-12T04:49:30.696 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:49:30 GMT /dkyges-74/obj4 2024-04-12T04:49:30.696 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:8XV+jUMiumQkoL+8tlnHSc/BL38= 2024-04-12T04:49:30.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:49:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8XV+jUMiumQkoL+8tlnHSc/BL38=', 'Content-Length': '0'} 2024-04-12T04:49:30.698 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000cb516a1d6ed92611-006618bd5a-2bef-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:49:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:49:30.698 DEBUG:boto:path=/dkyges-75/ 2024-04-12T04:49:30.698 DEBUG:boto:auth_path=/dkyges-75/ 2024-04-12T04:49:30.698 DEBUG:boto:Method: HEAD 2024-04-12T04:49:30.699 DEBUG:boto:Path: /dkyges-75/ 2024-04-12T04:49:30.699 DEBUG:boto:Data: 2024-04-12T04:49:30.699 DEBUG:boto:Headers: {} 2024-04-12T04:49:30.699 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:49:30.699 DEBUG:boto:Port: 8001 2024-04-12T04:49:30.699 DEBUG:boto:Params: {} 2024-04-12T04:49:30.699 DEBUG:boto:Token: None 2024-04-12T04:49:30.699 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:49:30 GMT /dkyges-75/ 2024-04-12T04:49:30.699 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1JE8C7Oq1pc6uoXwrv82E6OMCjo= 2024-04-12T04:49:30.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:49:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1JE8C7Oq1pc6uoXwrv82E6OMCjo=', 'Content-Length': '0'} 2024-04-12T04:49:30.702 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a0605197a5bdefd-006618bd5a-2bef-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:49:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:49:30.702 DEBUG:boto:path=/dkyges-75/obj1 2024-04-12T04:49:30.702 DEBUG:boto:auth_path=/dkyges-75/obj1 2024-04-12T04:49:30.703 DEBUG:boto:Method: HEAD 2024-04-12T04:49:30.703 DEBUG:boto:Path: /dkyges-75/obj1 2024-04-12T04:49:30.703 DEBUG:boto:Data: 2024-04-12T04:49:30.703 DEBUG:boto:Headers: {} 2024-04-12T04:49:30.703 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:49:30.703 DEBUG:boto:Port: 8001 2024-04-12T04:49:30.703 DEBUG:boto:Params: {} 2024-04-12T04:49:30.703 DEBUG:boto:Token: None 2024-04-12T04:49:30.703 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:49:30 GMT /dkyges-75/obj1 2024-04-12T04:49:30.704 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ycTt3zd6+3nmVswF61EarYYm744= 2024-04-12T04:49:30.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:49:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ycTt3zd6+3nmVswF61EarYYm744=', 'Content-Length': '0'} 2024-04-12T04:49:30.706 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-74:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.10617.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008cf9c7f8918a537a-006618bd5a-2bef-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:49:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:49:30.706 DEBUG:boto:path=/dkyges-75/obj1 2024-04-12T04:49:30.706 DEBUG:boto:auth_path=/dkyges-75/obj1 2024-04-12T04:49:30.706 DEBUG:boto:Method: GET 2024-04-12T04:49:30.706 DEBUG:boto:Path: /dkyges-75/obj1 2024-04-12T04:49:30.707 DEBUG:boto:Data: 2024-04-12T04:49:30.707 DEBUG:boto:Headers: {} 2024-04-12T04:49:30.707 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:49:30.707 DEBUG:boto:Port: 8001 2024-04-12T04:49:30.707 DEBUG:boto:Params: {} 2024-04-12T04:49:30.707 DEBUG:boto:Token: None 2024-04-12T04:49:30.707 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:49:30 GMT /dkyges-75/obj1 2024-04-12T04:49:30.707 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:782lwqZ4YjQK+Xz1SvKyb8VhE0A= 2024-04-12T04:49:30.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:49:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:782lwqZ4YjQK+Xz1SvKyb8VhE0A=', 'Content-Length': '0'} 2024-04-12T04:49:30.710 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-74:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.10617.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa0cc641b1278f0c-006618bd5a-2bef-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:49:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:49:30.710 DEBUG:boto:path=/dkyges-75/obj2 2024-04-12T04:49:30.711 DEBUG:boto:auth_path=/dkyges-75/obj2 2024-04-12T04:49:30.711 DEBUG:boto:Method: HEAD 2024-04-12T04:49:30.711 DEBUG:boto:Path: /dkyges-75/obj2 2024-04-12T04:49:30.711 DEBUG:boto:Data: 2024-04-12T04:49:30.711 DEBUG:boto:Headers: {} 2024-04-12T04:49:30.711 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:49:30.711 DEBUG:boto:Port: 8001 2024-04-12T04:49:30.711 DEBUG:boto:Params: {} 2024-04-12T04:49:30.711 DEBUG:boto:Token: None 2024-04-12T04:49:30.711 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:49:30 GMT /dkyges-75/obj2 2024-04-12T04:49:30.712 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TaAo1KgqPsMevZPPcj+6/Bn61Tg= 2024-04-12T04:49:30.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:49:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TaAo1KgqPsMevZPPcj+6/Bn61Tg=', 'Content-Length': '0'} 2024-04-12T04:49:30.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-74:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.10617.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2210291c1deeca4-006618bd5a-2bef-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:49:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:49:30.713 DEBUG:boto:path=/dkyges-75/obj2 2024-04-12T04:49:30.714 DEBUG:boto:auth_path=/dkyges-75/obj2 2024-04-12T04:49:30.714 DEBUG:boto:Method: GET 2024-04-12T04:49:30.714 DEBUG:boto:Path: /dkyges-75/obj2 2024-04-12T04:49:30.714 DEBUG:boto:Data: 2024-04-12T04:49:30.714 DEBUG:boto:Headers: {} 2024-04-12T04:49:30.714 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:49:30.714 DEBUG:boto:Port: 8001 2024-04-12T04:49:30.714 DEBUG:boto:Params: {} 2024-04-12T04:49:30.714 DEBUG:boto:Token: None 2024-04-12T04:49:30.714 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:49:30 GMT /dkyges-75/obj2 2024-04-12T04:49:30.715 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:9g2H4Ylj2jcCoFDbYAGnYscFNoM= 2024-04-12T04:49:30.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:49:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9g2H4Ylj2jcCoFDbYAGnYscFNoM=', 'Content-Length': '0'} 2024-04-12T04:49:30.716 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-74:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.10617.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a55c41d1825bf1dd-006618bd5a-2bef-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:49:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:49:30.758 DEBUG:boto:path=/dkyges-75/obj3 2024-04-12T04:49:30.759 DEBUG:boto:auth_path=/dkyges-75/obj3 2024-04-12T04:49:30.759 DEBUG:boto:Method: HEAD 2024-04-12T04:49:30.759 DEBUG:boto:Path: /dkyges-75/obj3 2024-04-12T04:49:30.759 DEBUG:boto:Data: 2024-04-12T04:49:30.759 DEBUG:boto:Headers: {} 2024-04-12T04:49:30.759 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:49:30.759 DEBUG:boto:Port: 8001 2024-04-12T04:49:30.759 DEBUG:boto:Params: {} 2024-04-12T04:49:30.759 DEBUG:boto:Token: None 2024-04-12T04:49:30.760 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:49:30 GMT /dkyges-75/obj3 2024-04-12T04:49:30.760 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6O9PkLw5FF1m0VeV9UPNr/c5IlQ= 2024-04-12T04:49:30.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:49:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6O9PkLw5FF1m0VeV9UPNr/c5IlQ=', 'Content-Length': '0'} 2024-04-12T04:49:30.762 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-75:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.10617.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea4d4439c442e796-006618bd5a-2bef-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:49:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:49:30.762 DEBUG:boto:path=/dkyges-75/obj3 2024-04-12T04:49:30.762 DEBUG:boto:auth_path=/dkyges-75/obj3 2024-04-12T04:49:30.762 DEBUG:boto:Method: GET 2024-04-12T04:49:30.762 DEBUG:boto:Path: /dkyges-75/obj3 2024-04-12T04:49:30.763 DEBUG:boto:Data: 2024-04-12T04:49:30.763 DEBUG:boto:Headers: {} 2024-04-12T04:49:30.763 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:49:30.763 DEBUG:boto:Port: 8001 2024-04-12T04:49:30.763 DEBUG:boto:Params: {} 2024-04-12T04:49:30.763 DEBUG:boto:Token: None 2024-04-12T04:49:30.763 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:49:30 GMT /dkyges-75/obj3 2024-04-12T04:49:30.763 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Nyq66EfG4x1bL9RpgWfQJBeS4aA= 2024-04-12T04:49:30.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:49:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Nyq66EfG4x1bL9RpgWfQJBeS4aA=', 'Content-Length': '0'} 2024-04-12T04:49:30.765 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-75:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.10617.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020284a2869988bd7-006618bd5a-2bef-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:49:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:49:30.766 DEBUG:boto:path=/dkyges-75/obj4 2024-04-12T04:49:30.766 DEBUG:boto:auth_path=/dkyges-75/obj4 2024-04-12T04:49:30.766 DEBUG:boto:Method: HEAD 2024-04-12T04:49:30.766 DEBUG:boto:Path: /dkyges-75/obj4 2024-04-12T04:49:30.766 DEBUG:boto:Data: 2024-04-12T04:49:30.766 DEBUG:boto:Headers: {} 2024-04-12T04:49:30.766 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:49:30.766 DEBUG:boto:Port: 8001 2024-04-12T04:49:30.766 DEBUG:boto:Params: {} 2024-04-12T04:49:30.767 DEBUG:boto:Token: None 2024-04-12T04:49:30.767 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:49:30 GMT /dkyges-75/obj4 2024-04-12T04:49:30.767 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:v745qGYDByOPARMNkdYrGh+DlIY= 2024-04-12T04:49:30.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:49:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:v745qGYDByOPARMNkdYrGh+DlIY=', 'Content-Length': '0'} 2024-04-12T04:49:30.768 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-75:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.10617.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002537d6eed520d5e9-006618bd5a-2bef-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:49:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:49:30.769 DEBUG:boto:path=/dkyges-75/obj4 2024-04-12T04:49:30.769 DEBUG:boto:auth_path=/dkyges-75/obj4 2024-04-12T04:49:30.770 DEBUG:boto:Method: GET 2024-04-12T04:49:30.770 DEBUG:boto:Path: /dkyges-75/obj4 2024-04-12T04:49:30.770 DEBUG:boto:Data: 2024-04-12T04:49:30.770 DEBUG:boto:Headers: {} 2024-04-12T04:49:30.770 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:49:30.770 DEBUG:boto:Port: 8001 2024-04-12T04:49:30.770 DEBUG:boto:Params: {} 2024-04-12T04:49:30.770 DEBUG:boto:Token: None 2024-04-12T04:49:30.770 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:49:30 GMT /dkyges-75/obj4 2024-04-12T04:49:30.771 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gQHJNGah1ORd3K1BwAlA6mTesxY= 2024-04-12T04:49:30.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:49:30 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gQHJNGah1ORd3K1BwAlA6mTesxY=', 'Content-Length': '0'} 2024-04-12T04:49:30.773 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:47:57 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-75:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.10617.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a54bb8d3077a1cbb-006618bd5a-2bef-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:49:30 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:49:30.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dkyges-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:30.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dkyges-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:30.773 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-dkyges-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:49:30.834 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:30.834 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:30.894 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-dkyges-75","source":{"bucket":"dkyges-75","zones":["a1"]},"dest":{"bucket":"dkyges-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:49:30.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dkyges-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:30.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dkyges-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:30.894 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-dkyges-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:49:30.950 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:30.951 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:31.007 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:49:31.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:31.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:31.007 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:49:31.067 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:31.068 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:31.122 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2024-04-12T04:49:31.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:31.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:31.123 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:49:31.181 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:31.182 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:31.239 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2024-04-12T04:49:31.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:31.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:31.239 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:49:31.298 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:31.298 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:31.359 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:49:31.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:31.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:31.360 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:49:31.414 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:31.414 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:31.472 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:49:31.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:31.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:31.473 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:49:31.528 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:31.528 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:31.592 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:49:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:49:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 21, 2024-04-12T04:49:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T04:49:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T04:49:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T04:49:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:49:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T04:49:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T04:49:31.593 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T04:49:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:49:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T04:49:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:49:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:49:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:49:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:49:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:49:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:49:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:49:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:31.594 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:49:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T04:49:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:49:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T04:49:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:49:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:49:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:49:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:49:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:49:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:49:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:49:31.595 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:49:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:49:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:49:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:49:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:49:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:49:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T04:49:31.596 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:49:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:49:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:49:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:49:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:49:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:49:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:49:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:49:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:49:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:49:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:49:31.597 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:49:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:49:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:49:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:49:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:49:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:49:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:49:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:49:31.598 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:49:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:49:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:49:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:49:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:49:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [ 2024-04-12T04:49:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-group", 2024-04-12T04:49:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "data_flow": { 2024-04-12T04:49:31.599 INFO:teuthology.orchestra.run.smithi043.stdout: "symmetrical": [ 2024-04-12T04:49:31.600 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:31.600 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-flow1", 2024-04-12T04:49:31.600 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:49:31.600 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:49:31.600 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:49:31.600 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:31.600 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:31.600 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:31.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.600 INFO:teuthology.orchestra.run.smithi043.stdout: "pipes": [ 2024-04-12T04:49:31.600 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:31.601 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-pipe", 2024-04-12T04:49:31.601 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:49:31.601 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "*", 2024-04-12T04:49:31.601 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:49:31.601 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:49:31.601 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:49:31.601 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:31.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.601 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": { 2024-04-12T04:49:31.601 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "*", 2024-04-12T04:49:31.601 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:49:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:49:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:49:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "params": { 2024-04-12T04:49:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:49:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "filter": { 2024-04-12T04:49:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [] 2024-04-12T04:49:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.602 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": {}, 2024-04-12T04:49:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-12T04:49:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "system", 2024-04-12T04:49:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "" 2024-04-12T04:49:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:49:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "allowed" 2024-04-12T04:49:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.603 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:49:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:49:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T04:49:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T04:49:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:49:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T04:49:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:49:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:49:31.604 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:49:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:49:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:49:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:49:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:49:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:49:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T04:49:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:49:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:49:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:49:31.605 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:49:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:49:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:49:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:49:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:49:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:49:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:49:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:49:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:49:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:49:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:49:31.606 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:49:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T04:49:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:49:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T04:49:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:49:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:49:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:49:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:49:31.607 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:49:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:49:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:49:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:49:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:49:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:49:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:49:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:49:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:49:31.608 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:49:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:49:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:49:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:49:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:49:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:49:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:49:31.609 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:49:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:49:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T04:49:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:49:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:49:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:49:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T04:49:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:31.610 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:49:31.611 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T04:49:31.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:31.611 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:49:31.611 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T04:49:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:49:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T04:49:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:49:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T04:49:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T04:49:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:49:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T04:49:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T04:49:31.613 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:49:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:49:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:49:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:49:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:49:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T04:49:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:49:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:49:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:49:31.614 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:49:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:49:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T04:49:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:49:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:49:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:49:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:49:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:49:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T04:49:31.615 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:49:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:49:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:49:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:49:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:49:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T04:49:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:49:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:49:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:49:31.616 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:49:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:49:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:49:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T04:49:31.617 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:31.617 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:49:31.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:31.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:31.618 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:49:31.678 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:31.678 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:31.770 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:49:31.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dkyges-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:31.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dkyges-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:31.771 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dkyges-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:49:31.847 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:31.848 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:32.064 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:49:32.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dkyges-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:32.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dkyges-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:32.064 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket dkyges-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:49:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:32.164 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:32.281 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:49:32.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dkyges-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=dkyges-74', '--source-bucket=dkyges-74', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:32.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dkyges-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=dkyges-74', '--source-bucket=dkyges-74', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:32.281 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dkyges-74 --source-zones=a1 --dest-zones=a2 --bucket=dkyges-74 --source-bucket=dkyges-74 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:49:32.362 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:32.362 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:32.454 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dkyges-74","source":{"bucket":"dkyges-74","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-12T04:49:32.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dkyges-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=dkyges-74', '--source-bucket=dkyges-75', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:32.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dkyges-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=dkyges-74', '--source-bucket=dkyges-75', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:32.455 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dkyges-75 --source-zones=a1 --dest-zones=a2 --bucket=dkyges-74 --source-bucket=dkyges-75 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:49:32.534 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:32.534 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:32.608 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dkyges-74","source":{"bucket":"dkyges-74","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dkyges-75","source":{"bucket":"dkyges-75","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-12T04:49:32.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dkyges-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:32.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dkyges-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:32.609 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dkyges-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:49:32.665 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:32.665 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:32.727 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dkyges-74","source":{"bucket":"dkyges-74","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dkyges-75","source":{"bucket":"dkyges-75","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:49:32.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:32.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:49:32.728 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:49:32.779 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:32.779 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:32.833 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:49:32.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:32.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:32.834 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:49:32.892 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:32.892 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:32.931 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:49:32.947 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:49:32.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:32.948 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:49:32.948 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:49:32.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:32.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:32.948 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:49:32.948 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:49:32.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:32.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:32.948 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:49:32.949 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:49:32.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:32.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:32.949 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897372.707923_1010.1", 2024-04-12T04:49:32.949 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:49:32.707923Z" 2024-04-12T04:49:32.949 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:32.949 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:49:32.950 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897372.707923_1010.1'} 2024-04-12T04:49:32.950 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:49:32.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:32.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:32.951 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:49:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:33.005 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:33.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:49:33.037+0000 7faa895af780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:49:33.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:49:33.038+0000 7faa895af780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:49:33.058 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:49:33.058 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:49:33.058 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:49:33.059 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:49:33.059 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:49:33.059 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:49:33.059 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:33.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:33.059 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:49:33.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:33.059 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:49:33.059 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:33.059 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:33.059 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:49:33.059 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:33.060 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:49:33.060 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:33.060 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:49:33.060 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:33.060 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:33.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:33.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:33.061 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:49:33.061 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:33.061 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:33.061 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:49:33.061 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:33.061 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:33.061 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:33.061 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:49:33.061 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:33.062 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:33.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:33.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:33.062 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:49:33.062 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:33.062 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:33.062 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:49:33.062 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:33.062 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:33.062 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:33.062 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:49:33.063 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:33.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:33.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:33.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:33.063 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:49:33.063 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:33.063 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:33.063 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897270.955650_982.1", 2024-04-12T04:49:33.063 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:33.063 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:33.063 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:33.064 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:50.955650Z", 2024-04-12T04:49:33.064 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:33.064 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:33.064 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:33.064 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:33.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:33.064 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:49:33.064 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:49:33.064 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:49:33.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:33.064 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:49:33Z" 2024-04-12T04:49:33.064 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:49:33.065 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897270.955650_982.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:50.955650Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:49:33Z" } 2024-04-12T04:49:33.065 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897270.955650_982.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:50.955650Z', 'realm_epoch': 2}}] 2024-04-12T04:49:33.065 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897372.707923_1010.1'} 2024-04-12T04:49:33.065 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897270.955650_982.1'} 2024-04-12T04:49:33.065 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897372.707923_1010.1 target=1_1712897270.955650_982.1 2024-04-12T04:49:38.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:38.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:38.067 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:49:38.127 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:38.127 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:38.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:49:38.162+0000 7f32376af780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:49:38.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:49:38.163+0000 7f32376af780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:49:38.182 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:49:38.183 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:49:38.183 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:49:38.183 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:49:38.183 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:49:38.183 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:49:38.183 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:38.183 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:38.183 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:49:38.183 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:38.184 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:49:38.184 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:38.184 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:38.184 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:49:38.184 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:38.184 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:49:38.184 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:38.184 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:49:38.184 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:38.184 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:38.184 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:38.185 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:38.185 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:49:38.185 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:38.185 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:38.185 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:49:38.185 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:38.185 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:38.185 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:38.185 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:49:38.185 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:38.186 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:38.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:38.186 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:38.186 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:49:38.186 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:38.186 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:38.186 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:49:38.186 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:38.187 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:38.187 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:38.187 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:49:38.187 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:38.187 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:38.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:38.187 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:38.187 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:49:38.187 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:38.187 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:38.187 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897270.955650_982.1", 2024-04-12T04:49:38.187 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:38.188 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:38.188 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:38.188 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:50.955650Z", 2024-04-12T04:49:38.188 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:38.188 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:38.188 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:38.188 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:38.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:38.189 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:49:38.189 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:49:38.189 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:49:38.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:38.189 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:49:38Z" 2024-04-12T04:49:38.189 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:49:38.190 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897270.955650_982.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:50.955650Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:49:38Z" } 2024-04-12T04:49:38.190 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897270.955650_982.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:50.955650Z', 'realm_epoch': 2}}] 2024-04-12T04:49:38.190 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897372.707923_1010.1'} 2024-04-12T04:49:38.191 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897270.955650_982.1'} 2024-04-12T04:49:38.191 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897372.707923_1010.1 target=1_1712897270.955650_982.1 2024-04-12T04:49:43.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:43.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:43.192 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:49:43.253 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:43.253 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:43.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:49:43.292+0000 7f429eb7f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:49:43.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:49:43.293+0000 7f429eb7f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:49:43.311 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:49:43.311 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:49:43.311 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:49:43.311 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:49:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:49:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:49:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:49:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:49:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:49:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:49:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:49:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:49:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:49:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:49:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:49:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:43.315 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:43.315 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:49:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:49:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:49:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897270.955650_982.1", 2024-04-12T04:49:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:50.955650Z", 2024-04-12T04:49:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:49:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:49:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:49:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:49:43Z" 2024-04-12T04:49:43.318 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:49:43.319 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897270.955650_982.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:50.955650Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:49:43Z" } 2024-04-12T04:49:43.320 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897270.955650_982.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:50.955650Z', 'realm_epoch': 2}}] 2024-04-12T04:49:43.320 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897372.707923_1010.1'} 2024-04-12T04:49:43.320 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897270.955650_982.1'} 2024-04-12T04:49:43.320 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897372.707923_1010.1 target=1_1712897270.955650_982.1 2024-04-12T04:49:48.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:48.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:48.322 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:49:48.381 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:48.381 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:48.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:49:48.415+0000 7f4a35820780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:49:48.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:49:48.416+0000 7f4a35820780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:49:48.434 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:49:48.435 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:49:48.435 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:49:48.435 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:49:48.435 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:49:48.435 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:49:48.435 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:48.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:48.435 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:49:48.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:48.436 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:49:48.436 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:48.436 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:48.436 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:49:48.436 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:48.436 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:49:48.436 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:48.436 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:49:48.436 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:48.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:48.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:48.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:48.437 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:49:48.437 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:48.437 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:48.437 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:49:48.437 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:48.437 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:48.437 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:48.437 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:49:48.437 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:48.437 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:48.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:48.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:48.438 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:49:48.438 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:48.438 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:48.438 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:49:48.438 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:48.438 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:48.438 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:48.438 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:49:48.438 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:48.439 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:48.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:48.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:48.439 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:49:48.439 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:48.439 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:48.439 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897270.955650_982.1", 2024-04-12T04:49:48.439 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:48.439 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:48.439 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:48.439 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:50.955650Z", 2024-04-12T04:49:48.439 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:48.439 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:48.440 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:48.440 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:48.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:48.440 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:49:48.440 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:49:48.440 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:49:48.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:48.440 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:49:48Z" 2024-04-12T04:49:48.440 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:49:48.441 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897270.955650_982.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:50.955650Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:49:48Z" } 2024-04-12T04:49:48.442 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897270.955650_982.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:50.955650Z', 'realm_epoch': 2}}] 2024-04-12T04:49:48.442 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897372.707923_1010.1'} 2024-04-12T04:49:48.442 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897270.955650_982.1'} 2024-04-12T04:49:48.442 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897372.707923_1010.1 target=1_1712897270.955650_982.1 2024-04-12T04:49:53.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:53.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:53.444 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:49:53.503 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:53.503 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:53.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:49:53.537+0000 7efdb2a27780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:49:53.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:49:53.538+0000 7efdb2a27780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:49:53.557 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:49:53.557 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:49:53.557 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:49:53.557 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:49:53.557 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:49:53.557 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:49:53.557 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:53.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:53.557 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:49:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:49:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:49:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:49:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:49:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:53.558 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:49:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:49:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:53.559 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:49:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:49:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:49:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:53.560 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:49:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:49:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897270.955650_982.1", 2024-04-12T04:49:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:53.561 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:50.955650Z", 2024-04-12T04:49:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:49:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:49:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:49:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:53.562 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:49:53Z" 2024-04-12T04:49:53.563 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:49:53.563 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897270.955650_982.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:50.955650Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:49:53Z" } 2024-04-12T04:49:53.564 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897270.955650_982.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:50.955650Z', 'realm_epoch': 2}}] 2024-04-12T04:49:53.564 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897372.707923_1010.1'} 2024-04-12T04:49:53.564 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897270.955650_982.1'} 2024-04-12T04:49:53.564 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897372.707923_1010.1 target=1_1712897270.955650_982.1 2024-04-12T04:49:58.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:58.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:49:58.566 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:49:58.624 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:49:58.625 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:49:58.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:49:58.658+0000 7f8f34e36780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:49:58.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:49:58.659+0000 7f8f34e36780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:49:58.678 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:49:58.678 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:49:58.678 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:49:58.678 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:49:58.678 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:49:58.678 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:49:58.678 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:58.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:49:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:49:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:49:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:49:58.679 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:49:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:49:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:49:58.680 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:49:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:49:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:58.681 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:49:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:49:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:58.682 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:49:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:49:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:49:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:49:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897270.955650_982.1", 2024-04-12T04:49:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:49:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:49:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:49:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:50.955650Z", 2024-04-12T04:49:58.683 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:49:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:49:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:49:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:49:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:49:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:49:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:49:58.684 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:49:58Z" 2024-04-12T04:49:58.685 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:49:58.686 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897270.955650_982.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:50.955650Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:49:58Z" } 2024-04-12T04:49:58.686 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897270.955650_982.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:50.955650Z', 'realm_epoch': 2}}] 2024-04-12T04:49:58.686 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897372.707923_1010.1'} 2024-04-12T04:49:58.686 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897270.955650_982.1'} 2024-04-12T04:49:58.687 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897372.707923_1010.1 target=1_1712897270.955650_982.1 2024-04-12T04:50:03.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:03.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:03.688 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:50:03.749 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:50:03.750 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:50:03.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:50:03.784+0000 7f77161fe780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:50:03.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:50:03.785+0000 7f77161fe780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:50:03.803 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:50:03.803 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:50:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:50:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:50:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:50:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:50:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:50:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:50:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:50:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:50:03.804 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:50:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:50:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:50:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:50:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:50:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:03.805 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:50:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:50:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:50:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:50:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:50:03.806 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:03.807 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:50:03.807 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:50:03.807 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:03.807 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:03.807 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:03.807 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:50:03.807 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:50:03.807 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:03.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:03.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:03.807 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:50:03.808 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:03.808 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:50:03.808 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897372.707923_1010.1", 2024-04-12T04:50:03.808 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:03.808 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:03.808 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:03.808 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:49:32.707923Z", 2024-04-12T04:50:03.808 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:50:03.808 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:03.808 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:03.808 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:50:03.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:03.809 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:50:03.809 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:50:03.809 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:50:03.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:03.809 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:50:03Z" 2024-04-12T04:50:03.809 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:50:03.810 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897372.707923_1010.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:49:32.707923Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:50:03Z" } 2024-04-12T04:50:03.810 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897372.707923_1010.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:49:32.707923Z', 'realm_epoch': 2}}] 2024-04-12T04:50:03.811 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897372.707923_1010.1'} 2024-04-12T04:50:03.811 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897372.707923_1010.1'} 2024-04-12T04:50:03.811 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:50:03.811 DEBUG:boto:path=/dkyges-74/ 2024-04-12T04:50:03.811 DEBUG:boto:auth_path=/dkyges-74/ 2024-04-12T04:50:03.811 DEBUG:boto:Method: HEAD 2024-04-12T04:50:03.811 DEBUG:boto:Path: /dkyges-74/ 2024-04-12T04:50:03.811 DEBUG:boto:Data: 2024-04-12T04:50:03.812 DEBUG:boto:Headers: {} 2024-04-12T04:50:03.812 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:50:03.812 DEBUG:boto:Port: 8000 2024-04-12T04:50:03.812 DEBUG:boto:Params: {} 2024-04-12T04:50:03.812 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:50:03.812 DEBUG:boto:Token: None 2024-04-12T04:50:03.813 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:50:03 GMT /dkyges-74/ 2024-04-12T04:50:03.813 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cN3//TKI5kBNdPo4Ma6bFOU4VWM= 2024-04-12T04:50:03.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:50:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cN3//TKI5kBNdPo4Ma6bFOU4VWM=', 'Content-Length': '0'} 2024-04-12T04:50:03.828 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090007422db91da5e-006618bd7b-2cea-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:50:03.829 DEBUG:boto:path=/dkyges-74/obj5 2024-04-12T04:50:03.829 DEBUG:boto:auth_path=/dkyges-74/obj5 2024-04-12T04:50:03.829 DEBUG:boto:Method: PUT 2024-04-12T04:50:03.829 DEBUG:boto:Path: /dkyges-74/obj5 2024-04-12T04:50:03.829 DEBUG:boto:Data: 2024-04-12T04:50:03.829 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:50:03.829 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:50:03.829 DEBUG:boto:Port: 8000 2024-04-12T04:50:03.830 DEBUG:boto:Params: {} 2024-04-12T04:50:03.830 DEBUG:boto:Token: None 2024-04-12T04:50:03.830 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:50:03 GMT /dkyges-74/obj5 2024-04-12T04:50:03.830 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:SaB6NsnYne6cHnTZpxtKqGeGdtU= 2024-04-12T04:50:03.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:50:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SaB6NsnYne6cHnTZpxtKqGeGdtU='} 2024-04-12T04:50:03.840 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a2d4c10ff06757a-006618bd7b-2cea-a1'), ('Date', 'Fri, 12 Apr 2024 04:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:50:03.841 DEBUG:boto:path=/dkyges-74/ 2024-04-12T04:50:03.841 DEBUG:boto:auth_path=/dkyges-74/ 2024-04-12T04:50:03.841 DEBUG:boto:Method: HEAD 2024-04-12T04:50:03.841 DEBUG:boto:Path: /dkyges-74/ 2024-04-12T04:50:03.841 DEBUG:boto:Data: 2024-04-12T04:50:03.841 DEBUG:boto:Headers: {} 2024-04-12T04:50:03.841 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:50:03.841 DEBUG:boto:Port: 8000 2024-04-12T04:50:03.841 DEBUG:boto:Params: {} 2024-04-12T04:50:03.842 DEBUG:boto:Token: None 2024-04-12T04:50:03.842 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:50:03 GMT /dkyges-74/ 2024-04-12T04:50:03.842 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:cN3//TKI5kBNdPo4Ma6bFOU4VWM= 2024-04-12T04:50:03.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:50:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cN3//TKI5kBNdPo4Ma6bFOU4VWM=', 'Content-Length': '0'} 2024-04-12T04:50:03.845 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007f48e8991a1b93ad-006618bd7b-2cea-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:50:03.845 DEBUG:boto:path=/dkyges-74/obj6 2024-04-12T04:50:03.845 DEBUG:boto:auth_path=/dkyges-74/obj6 2024-04-12T04:50:03.845 DEBUG:boto:Method: PUT 2024-04-12T04:50:03.845 DEBUG:boto:Path: /dkyges-74/obj6 2024-04-12T04:50:03.845 DEBUG:boto:Data: 2024-04-12T04:50:03.845 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:50:03.846 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:50:03.846 DEBUG:boto:Port: 8000 2024-04-12T04:50:03.846 DEBUG:boto:Params: {} 2024-04-12T04:50:03.846 DEBUG:boto:Token: None 2024-04-12T04:50:03.846 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:50:03 GMT /dkyges-74/obj6 2024-04-12T04:50:03.846 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ajJPXgc8n818mL50i4VWl9L6NqI= 2024-04-12T04:50:03.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:50:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ajJPXgc8n818mL50i4VWl9L6NqI='} 2024-04-12T04:50:03.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000018b95439fbac61c6-006618bd7b-2cea-a1'), ('Date', 'Fri, 12 Apr 2024 04:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:50:03.855 DEBUG:boto:path=/dkyges-75/ 2024-04-12T04:50:03.855 DEBUG:boto:auth_path=/dkyges-75/ 2024-04-12T04:50:03.855 DEBUG:boto:Method: HEAD 2024-04-12T04:50:03.855 DEBUG:boto:Path: /dkyges-75/ 2024-04-12T04:50:03.855 DEBUG:boto:Data: 2024-04-12T04:50:03.855 DEBUG:boto:Headers: {} 2024-04-12T04:50:03.856 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:50:03.856 DEBUG:boto:Port: 8000 2024-04-12T04:50:03.856 DEBUG:boto:Params: {} 2024-04-12T04:50:03.856 DEBUG:boto:Token: None 2024-04-12T04:50:03.856 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:50:03 GMT /dkyges-75/ 2024-04-12T04:50:03.856 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mxR4b/FsbtJlDRNVvygDXOkvBZQ= 2024-04-12T04:50:03.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:50:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mxR4b/FsbtJlDRNVvygDXOkvBZQ=', 'Content-Length': '0'} 2024-04-12T04:50:03.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007042d8872821676a-006618bd7b-2cea-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:50:03.859 DEBUG:boto:path=/dkyges-75/obj7 2024-04-12T04:50:03.859 DEBUG:boto:auth_path=/dkyges-75/obj7 2024-04-12T04:50:03.859 DEBUG:boto:Method: PUT 2024-04-12T04:50:03.859 DEBUG:boto:Path: /dkyges-75/obj7 2024-04-12T04:50:03.859 DEBUG:boto:Data: 2024-04-12T04:50:03.859 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:50:03.859 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:50:03.859 DEBUG:boto:Port: 8000 2024-04-12T04:50:03.859 DEBUG:boto:Params: {} 2024-04-12T04:50:03.860 DEBUG:boto:Token: None 2024-04-12T04:50:03.860 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:50:03 GMT /dkyges-75/obj7 2024-04-12T04:50:03.860 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:CTPCGa9UHoVruBhMkdkm/XBIm9A= 2024-04-12T04:50:03.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:50:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CTPCGa9UHoVruBhMkdkm/XBIm9A='} 2024-04-12T04:50:03.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e7327725a79b18a-006618bd7b-2cea-a1'), ('Date', 'Fri, 12 Apr 2024 04:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:50:03.903 DEBUG:boto:path=/dkyges-75/ 2024-04-12T04:50:03.903 DEBUG:boto:auth_path=/dkyges-75/ 2024-04-12T04:50:03.903 DEBUG:boto:Method: HEAD 2024-04-12T04:50:03.903 DEBUG:boto:Path: /dkyges-75/ 2024-04-12T04:50:03.903 DEBUG:boto:Data: 2024-04-12T04:50:03.903 DEBUG:boto:Headers: {} 2024-04-12T04:50:03.903 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:50:03.903 DEBUG:boto:Port: 8000 2024-04-12T04:50:03.903 DEBUG:boto:Params: {} 2024-04-12T04:50:03.904 DEBUG:boto:Token: None 2024-04-12T04:50:03.904 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:50:03 GMT /dkyges-75/ 2024-04-12T04:50:03.904 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:mxR4b/FsbtJlDRNVvygDXOkvBZQ= 2024-04-12T04:50:03.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:50:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mxR4b/FsbtJlDRNVvygDXOkvBZQ=', 'Content-Length': '0'} 2024-04-12T04:50:03.906 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e407229f90cb5ca3-006618bd7b-2cea-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:50:03.907 DEBUG:boto:path=/dkyges-75/obj8 2024-04-12T04:50:03.907 DEBUG:boto:auth_path=/dkyges-75/obj8 2024-04-12T04:50:03.907 DEBUG:boto:Method: PUT 2024-04-12T04:50:03.907 DEBUG:boto:Path: /dkyges-75/obj8 2024-04-12T04:50:03.907 DEBUG:boto:Data: 2024-04-12T04:50:03.907 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:50:03.907 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:50:03.907 DEBUG:boto:Port: 8000 2024-04-12T04:50:03.907 DEBUG:boto:Params: {} 2024-04-12T04:50:03.908 DEBUG:boto:Token: None 2024-04-12T04:50:03.908 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:50:03 GMT /dkyges-75/obj8 2024-04-12T04:50:03.908 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:JE+uQ2BWYmnB06MAe1PpFGAgRlU= 2024-04-12T04:50:03.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:50:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JE+uQ2BWYmnB06MAe1PpFGAgRlU='} 2024-04-12T04:50:03.919 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cea73561cca9f8ca-006618bd7b-2cea-a1'), ('Date', 'Fri, 12 Apr 2024 04:50:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:50:03.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:03.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:03.919 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:50:03.976 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:50:03.976 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:50:04.015 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:50:04.037 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:50:04.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:04.037 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:50:04.037 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:50:04.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:04.037 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:50:04.037 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:50:04.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:04.038 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:50:04.038 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:50:04.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:04.038 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897372.707923_1010.1", 2024-04-12T04:50:04.038 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:49:32.707923Z" 2024-04-12T04:50:04.038 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:04.039 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:50:04.040 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897372.707923_1010.1'} 2024-04-12T04:50:04.040 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:50:04.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:04.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:04.040 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:50:04.096 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:50:04.096 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:50:04.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:50:04.129+0000 7f7a485cb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:50:04.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:50:04.130+0000 7f7a485cb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:50:04.148 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:50:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:50:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:50:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:50:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:50:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:50:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:50:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:50:04.149 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:50:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:50:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:50:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:50:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:04.150 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:50:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:50:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:50:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:50:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:50:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:04.151 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:50:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:50:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:50:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:50:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:50:04.152 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:50:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:50:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:50:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:50:04.153 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897372.707923_1010.1", 2024-04-12T04:50:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:49:32.707923Z", 2024-04-12T04:50:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:50:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:50:04.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:50:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:50:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:50:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.155 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:50:04Z" 2024-04-12T04:50:04.155 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:50:04.156 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897372.707923_1010.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:49:32.707923Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:50:04Z" } 2024-04-12T04:50:04.156 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897372.707923_1010.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:49:32.707923Z', 'realm_epoch': 2}}] 2024-04-12T04:50:04.156 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897372.707923_1010.1'} 2024-04-12T04:50:04.156 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897372.707923_1010.1'} 2024-04-12T04:50:04.156 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:50:04.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:04.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:04.157 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:50:04.211 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:50:04.211 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:50:04.274 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:50:04.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:04.274 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:50:04.275 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:50:04.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:04.275 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:50:04.275 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:50:04.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:04.275 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:50:04.275 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:50:04.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:04.276 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:50:04.276 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:50:04.276 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:04.276 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:50:04.276 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:04.276 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:50:04.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:04.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:04.277 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:50:04.334 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:50:04.334 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:50:04.387 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:50:04.387 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:50:04.387 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:50:04.387 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:50:04.388 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:50:04.388 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:50:04.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.388 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:50:04.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:04.388 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:50:04.388 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:04.388 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:04.388 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:50:04.388 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:04.388 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:04.389 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:04.389 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:50:04.389 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:04.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:04.389 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:50:04.389 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:04.389 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:04.389 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:50:04.389 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:04.389 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:04.390 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:04.390 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:50:04.390 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:04.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.390 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:04.390 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:50:04.390 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:04.390 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:04.390 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:50:04.390 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:04.390 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:50:04.391 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:04.391 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:50:04.391 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:04.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.391 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:04.391 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:50:04.391 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:04.391 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:04.391 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:50:04.391 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:04.391 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:50:04.392 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:04.392 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:50:04.392 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:04.392 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:04.392 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:50:04.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.392 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:50:04.392 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:50:04.392 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:50:04.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:04.392 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:50:04Z" 2024-04-12T04:50:04.392 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:50:04.393 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:50:04Z" } 2024-04-12T04:50:04.394 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:50:04.394 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:04.394 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:04.394 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:50:09.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:09.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:09.395 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:50:09.458 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:50:09.458 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:50:09.513 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:50:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:50:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:50:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:50:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:50:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:50:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:50:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:50:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:50:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:50:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:50:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:50:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:50:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:50:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:50:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:50:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:50:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:50:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:50:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:50:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:50:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:50:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:50:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:50:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:50:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:50:09Z" 2024-04-12T04:50:09.519 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:50:09.520 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:50:09Z" } 2024-04-12T04:50:09.520 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:50:09.520 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:09.520 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:09.520 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:50:14.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:14.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:14.522 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:50:14.584 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:50:14.584 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:50:14.641 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:50:14.641 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:50:14.641 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:50:14.641 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:50:14.641 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:50:14.641 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:50:14.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:14.641 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:50:14.641 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:14.641 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:50:14.642 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:14.642 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:14.642 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:50:14.642 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:14.642 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:14.642 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:14.642 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:50:14.642 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:14.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:14.642 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:14.642 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:50:14.643 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:14.643 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:14.643 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:50:14.643 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:14.643 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:14.643 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:14.643 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:50:14.643 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:14.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:14.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:14.643 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:50:14.644 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:14.644 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:14.644 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:50:14.644 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:14.644 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:50:14.644 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:14.644 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:50:14.644 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:14.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:14.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:14.644 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:50:14.645 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:14.645 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:14.645 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:50:14.645 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:14.645 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:50:14.645 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:14.645 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:50:14.645 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:14.645 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:14.645 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:50:14.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:14.646 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:50:14.646 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:50:14.646 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:50:14.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:14.646 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:50:14Z" 2024-04-12T04:50:14.646 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:50:14.647 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:50:14Z" } 2024-04-12T04:50:14.647 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:50:14.647 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:14.647 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:14.648 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:50:19.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:19.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:19.649 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:50:19.712 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:50:19.712 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:50:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:50:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:50:19.770 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:19.771 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:50:19.771 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:19.771 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:50:19.771 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:19.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:19.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:19.771 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:50:19.771 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:19.771 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:19.771 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:50:19.771 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:19.772 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:50:19.772 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:19.772 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:50:19.772 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:19.772 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:19.772 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:50:19.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:19.772 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:50:19.772 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:50:19.772 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:50:19.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:19.772 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:50:19Z" 2024-04-12T04:50:19.773 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:50:19.774 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:50:19Z" } 2024-04-12T04:50:19.774 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:50:19.774 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:19.774 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:19.774 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:50:24.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:24.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:24.775 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:50:24.837 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:50:24.837 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:50:24.892 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:50:24.892 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:50:24.892 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:50:24.892 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:50:24.892 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:50:24.892 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:50:24.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:24.893 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:50:24.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:24.893 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:50:24.893 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:24.893 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:24.893 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:50:24.893 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:24.893 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:24.893 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:24.894 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:50:24.894 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:24.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:24.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:24.894 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:50:24.894 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:24.894 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:24.894 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:50:24.895 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:24.895 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:24.895 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:24.895 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:50:24.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:24.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:24.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:24.895 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:50:24.895 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:24.895 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:24.896 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:50:24.896 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:24.896 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:50:24.896 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:24.896 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:50:24.896 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:24.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:24.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:24.896 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:50:24.896 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:24.897 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:24.897 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:50:24.897 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:24.897 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:50:24.897 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:24.897 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:50:24.897 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:24.897 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:24.897 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:50:24.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:24.898 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:50:24.898 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:50:24.898 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:50:24.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:24.898 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:50:24Z" 2024-04-12T04:50:24.898 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:50:24.899 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:50:24Z" } 2024-04-12T04:50:24.900 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:50:24.900 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:24.900 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:24.900 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:50:29.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:29.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:29.902 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:50:29.964 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:50:29.964 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:50:30.022 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:50:30.022 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:50:30.022 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:50:30.022 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:50:30.022 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:50:30.022 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:50:30.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:30.023 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:50:30.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:30.023 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:50:30.023 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:30.023 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:30.023 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:50:30.023 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:30.023 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:30.023 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:30.023 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:50:30.023 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:30.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:30.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:30.024 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:50:30.024 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:30.024 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:30.024 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:50:30.024 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:30.024 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:30.024 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:30.024 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:50:30.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:30.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:30.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:30.025 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:50:30.025 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:30.025 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:30.025 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:50:30.025 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:30.025 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:50:30.025 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:30.025 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:50:30.025 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:30.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:30.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:30.026 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:50:30.026 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:30.026 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:30.026 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:50:30.026 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:30.026 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:50:30.026 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:30.026 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:50:30.026 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:30.027 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:30.027 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:50:30.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:30.027 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:50:30.027 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:50:30.027 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:50:30.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:30.027 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:50:30Z" 2024-04-12T04:50:30.027 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:50:30.028 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:50:30Z" } 2024-04-12T04:50:30.028 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:50:30.029 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:30.029 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:30.029 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:50:35.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:35.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:35.031 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:50:35.093 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:50:35.093 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:50:35.149 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:50:35.149 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:50:35.149 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:50:35.149 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:50:35.149 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:50:35.149 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:50:35.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:35.150 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:50:35.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:35.150 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:50:35.150 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:35.150 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:35.150 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:50:35.150 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:35.150 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:35.150 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:35.151 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:50:35.151 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:35.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:35.151 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:35.151 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:50:35.151 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:35.151 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:35.151 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:50:35.151 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:35.151 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:35.151 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:35.151 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:50:35.152 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:35.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:35.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:35.152 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:50:35.152 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:35.152 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:35.152 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:50:35.152 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:35.152 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:50:35.152 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:35.152 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:50:35.153 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:35.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:35.153 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:35.153 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:50:35.153 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:35.153 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:35.153 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:50:35.153 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:35.153 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:50:35.153 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:35.153 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:50:35.154 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:35.154 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:35.154 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:50:35.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:35.154 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:50:35.154 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:50:35.154 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:50:35.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:35.154 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:50:35Z" 2024-04-12T04:50:35.154 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:50:35.155 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:50:35Z" } 2024-04-12T04:50:35.156 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:50:35.156 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:35.156 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:35.156 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:50:40.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:40.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:40.158 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:50:40.220 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:50:40.220 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:50:40.275 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:50:40.276 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:50:40.276 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:50:40.276 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:50:40.276 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:50:40.276 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:50:40.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:40.276 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:50:40.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:40.276 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:50:40.277 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:40.277 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:40.277 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:50:40.277 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:40.277 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:40.277 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:40.277 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:50:40.277 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:40.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:40.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:40.277 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:50:40.278 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:40.278 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:40.278 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:50:40.278 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:40.278 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:40.278 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:40.278 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:50:40.278 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:40.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:40.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:40.279 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:50:40.279 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:40.279 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:40.279 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:50:40.279 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:40.279 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:50:40.279 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:40.279 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:50:40.279 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:40.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:40.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:40.280 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:50:40.280 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:40.280 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:40.280 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:50:40.280 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:40.280 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:50:40.280 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:40.280 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:50:40.280 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:40.280 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:40.280 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:50:40.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:40.281 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:50:40.281 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:50:40.281 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:50:40.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:40.281 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:50:40Z" 2024-04-12T04:50:40.281 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:50:40.282 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:50:40Z" } 2024-04-12T04:50:40.282 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:50:40.283 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:40.283 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:40.283 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:50:45.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:45.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:45.284 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:50:45.349 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:50:45.350 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:50:45.406 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:50:45.407 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:50:45.407 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:50:45.407 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:50:45.407 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:50:45.407 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:50:45.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:45.407 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:50:45.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:45.407 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:50:45.407 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:45.407 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:45.408 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:50:45.408 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:45.408 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:45.408 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:45.408 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:50:45.408 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:45.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:45.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:45.408 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:50:45.408 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:45.408 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:45.409 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:50:45.409 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:45.409 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:45.409 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:45.409 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:50:45.409 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:45.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:45.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:45.409 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:50:45.409 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:45.409 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:45.410 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:50:45.410 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:45.410 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:50:45.410 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:45.410 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:50:45.410 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:45.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:45.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:45.410 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:50:45.410 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:45.410 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:45.410 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:50:45.411 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:45.411 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:50:45.411 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:45.411 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:50:45.411 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:45.411 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:45.411 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:50:45.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:45.411 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:50:45.411 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:50:45.411 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:50:45.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:45.412 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:50:45Z" 2024-04-12T04:50:45.412 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:50:45.413 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:50:45Z" } 2024-04-12T04:50:45.413 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:50:45.413 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:45.413 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:45.413 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:50:50.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:50.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:50.415 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:50:50.477 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:50:50.477 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:50:50.533 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:50:50.533 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:50:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:50:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:50:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:50:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:50:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:50:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:50:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:50.534 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:50:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:50:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:50:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:50.535 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:50:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:50:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:50:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:50:50.536 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:50:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:50:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:50:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:50:50.537 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:50:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:50:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:50:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:50:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:50:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:50:50.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:50.539 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:50:50Z" 2024-04-12T04:50:50.539 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:50:50.539 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:50:50Z" } 2024-04-12T04:50:50.540 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:50:50.540 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:50.540 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:50.540 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:50:55.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:55.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:50:55.541 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:50:55.602 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:50:55.603 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:50:55.659 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:50:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:50:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:50:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:50:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:50:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:50:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:50:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:50:55.660 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:50:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:50:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:50:55.661 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:50:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:50:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:50:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:50:55.662 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:50:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:50:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:50:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:50:55.663 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:50:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:50:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:50:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:50:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:50:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:50:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:50:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:50:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:50:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:50:55.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:50:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:50:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:50:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:50:55.665 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:50:55Z" 2024-04-12T04:50:55.665 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:50:55.666 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:50:55Z" } 2024-04-12T04:50:55.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:50:55.666 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:55.667 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:50:55.667 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:51:00.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:00.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:00.668 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:00.786 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:51:00.786 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:51:00.786 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:51:00.786 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:51:00.786 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:51:00.786 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:51:00.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:00.787 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:51:00.787 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:00.787 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:51:00.787 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:00.787 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:00.787 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:51:00.787 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:00.787 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:00.787 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:00.787 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:51:00.788 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:00.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:00.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:00.788 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:51:00.788 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:00.788 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:00.788 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:51:00.788 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:00.788 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:00.788 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:00.789 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:51:00.789 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:00.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:00.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:00.789 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:51:00.789 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:00.789 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:00.789 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:51:00.789 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:00.789 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:51:00.789 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:00.789 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:51:00.790 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:00.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:00.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:00.790 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:51:00.790 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:00.790 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:00.790 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:51:00.791 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:00.791 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:51:00.791 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:00.791 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:51:00.791 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:00.791 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:00.791 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:00.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:00.791 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:51:00.791 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:51:00.791 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:51:00.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:00.792 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:51:00Z" 2024-04-12T04:51:00.792 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:51:00.793 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:51:00Z" } 2024-04-12T04:51:00.793 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:51:00.793 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:00.793 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:00.793 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:51:05.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:05.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:05.798 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:05.861 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:05.862 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:05.918 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:51:05.918 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:51:05.918 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:51:05.918 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:51:05.918 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:51:05.918 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:51:05.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:05.918 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:51:05.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:05.919 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:51:05.919 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:05.919 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:05.919 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:51:05.919 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:05.919 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:05.919 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:05.919 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:51:05.919 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:05.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:05.920 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:05.920 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:51:05.920 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:05.920 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:05.920 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:51:05.920 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:05.920 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:05.920 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:05.920 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:51:05.920 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:05.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:05.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:05.921 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:51:05.921 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:05.921 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:05.921 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:51:05.921 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:05.921 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:51:05.921 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:05.921 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:51:05.921 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:05.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:05.922 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:05.922 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:51:05.922 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:05.922 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:05.922 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:51:05.922 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:05.922 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:51:05.922 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:05.922 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:51:05.922 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:05.923 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:05.923 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:05.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:05.923 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:51:05.923 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:51:05.923 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:51:05.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:05.923 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:51:05Z" 2024-04-12T04:51:05.923 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:51:05.924 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:51:05Z" } 2024-04-12T04:51:05.924 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:51:05.925 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:05.925 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:05.925 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:51:10.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:10.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:10.926 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:10.987 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:10.987 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:11.045 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:51:11.045 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:51:11.046 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:51:11.046 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:51:11.046 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:51:11.046 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:51:11.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:11.046 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:51:11.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:11.046 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:51:11.046 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:11.046 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:11.047 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:51:11.047 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:11.047 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:11.047 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:11.047 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:51:11.047 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:11.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:11.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:11.047 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:51:11.047 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:11.047 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:11.048 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:51:11.048 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:11.048 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:11.048 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:11.048 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:51:11.048 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:11.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:11.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:11.048 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:51:11.048 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:11.048 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:11.048 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:51:11.049 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:11.049 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:51:11.049 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:11.049 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:51:11.049 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:11.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:11.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:11.050 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:51:11.050 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:11.050 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:11.050 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:51:11.050 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:11.050 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:51:11.050 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:11.051 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:51:11.051 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:11.051 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:11.051 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:11.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:11.051 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:51:11.051 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:51:11.051 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:51:11.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:11.051 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:51:11Z" 2024-04-12T04:51:11.051 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:51:11.052 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:51:11Z" } 2024-04-12T04:51:11.053 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:51:11.053 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:11.053 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:11.053 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:51:16.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:16.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:16.055 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:16.117 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:16.117 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:16.175 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:51:16.175 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:51:16.175 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:51:16.175 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:51:16.175 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:51:16.176 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:51:16.176 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:16.176 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:51:16.176 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:16.176 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:51:16.176 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:16.176 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:16.176 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:51:16.176 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:16.176 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:16.177 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:16.177 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:51:16.177 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:16.177 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:16.177 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:16.177 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:51:16.177 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:16.177 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:16.177 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:51:16.177 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:16.177 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:16.178 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:16.178 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:51:16.178 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:16.178 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:16.178 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:16.178 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:51:16.178 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:16.178 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:16.178 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:51:16.178 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:16.179 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:51:16.179 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:16.179 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:51:16.179 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:16.179 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:16.179 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:16.179 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:51:16.179 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:16.179 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:16.179 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:51:16.179 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:16.179 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:51:16.180 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:16.180 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:51:16.180 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:16.180 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:16.180 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:16.180 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:16.180 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:51:16.180 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:51:16.180 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:51:16.180 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:16.180 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:51:16Z" 2024-04-12T04:51:16.181 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:51:16.181 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:51:16Z" } 2024-04-12T04:51:16.182 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:51:16.182 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:16.182 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:16.182 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:51:21.184 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:21.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:21.184 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:21.247 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:21.248 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:21.305 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:51:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:51:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:51:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:51:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:51:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:51:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:51:21.305 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:51:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:51:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:51:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:21.306 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:51:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:51:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:51:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:21.307 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:51:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:51:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:51:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:51:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:21.308 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:51:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:51:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:51:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:21.309 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:51:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:51:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:51:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:51:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:21.310 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:51:21Z" 2024-04-12T04:51:21.310 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:51:21.311 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:51:21Z" } 2024-04-12T04:51:21.311 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:51:21.312 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:21.312 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:21.312 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:51:26.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:26.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:26.313 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:26.376 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:26.376 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:26.433 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:51:26.433 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:51:26.433 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:51:26.434 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:51:26.434 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:51:26.434 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:51:26.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:26.434 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:51:26.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:26.434 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:51:26.434 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:26.434 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:26.434 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:51:26.434 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:26.434 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:26.435 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:26.435 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:51:26.435 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:26.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:26.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:26.435 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:51:26.435 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:26.435 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:26.435 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:51:26.435 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:26.435 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:26.436 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:26.436 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:51:26.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:26.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:26.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:26.436 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:51:26.436 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:26.436 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:26.436 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:51:26.436 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:26.436 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:51:26.437 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:26.437 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:51:26.437 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:26.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:26.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:26.437 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:51:26.437 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:26.437 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:26.437 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:51:26.437 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:26.438 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:51:26.438 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:26.438 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:51:26.438 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:26.438 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:26.438 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:26.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:26.438 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:51:26.438 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:51:26.438 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:51:26.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:26.439 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:51:26Z" 2024-04-12T04:51:26.439 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:51:26.440 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:51:26Z" } 2024-04-12T04:51:26.440 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:51:26.440 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:26.440 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:26.440 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:51:31.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:31.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:31.442 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:31.566 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:51:31.566 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:51:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:51:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:51:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:51:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:51:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:51:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:51:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:31.567 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048058", 2024-04-12T04:51:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.519752Z" 2024-04-12T04:51:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:51:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:31.568 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000032905", 2024-04-12T04:51:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.598184Z" 2024-04-12T04:51:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:51:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000054911", 2024-04-12T04:51:31.569 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:51:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:57.564882Z" 2024-04-12T04:51:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:51:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:31.570 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:51:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:51:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:51:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:51:31.571 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:51:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:51:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:31.572 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:51:31Z" 2024-04-12T04:51:31.572 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:51:31.573 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048058", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.519752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032905", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:47:57.598184Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000054911", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:47:57.564882Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:51:31Z" } 2024-04-12T04:51:31.573 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048058', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.519752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032905', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.598184Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000054911', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:47:57.564882Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:51:31.573 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:31.573 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048058', 1: '00000000000000000000:00000000000000032905', 2: '00000000000000000000:00000000000000054911', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:31.573 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048199 target=00000000000000000000:00000000000000048058, shard=1 master=00000000000000000000:00000000000000033046 target=00000000000000000000:00000000000000032905, shard=2 master=00000000000000000000:00000000000000055193 target=00000000000000000000:00000000000000054911 2024-04-12T04:51:33.071 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T04:51:33.068+0000 7fd539413700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:51:36.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:36.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:36.575 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:36.637 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:36.638 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:36.693 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:51:36.693 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:51:36.694 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:51:36.694 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:51:36.694 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:51:36.694 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:51:36.694 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:36.694 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:51:36.694 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:36.694 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:51:36.694 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:36.694 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:36.694 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:51:36.695 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:36.695 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:36.695 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:36.695 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:51:36.695 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:36.697 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:36.697 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:36.697 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:51:36.698 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:36.698 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:36.698 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:51:36.698 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:36.698 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:36.698 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:36.698 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:51:36.698 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:36.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:36.698 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:36.698 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:51:36.699 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:36.699 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:36.699 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:51:36.699 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:36.699 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:51:36.699 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:36.699 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:51:36.699 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:36.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:36.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:36.700 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:51:36.700 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:36.700 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:36.700 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:51:36.700 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:36.700 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:51:36.700 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:36.700 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:51:36.700 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:36.701 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:36.701 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:36.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:36.701 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:51:36.701 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:51:36.701 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:51:36.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:36.701 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:51:36Z" 2024-04-12T04:51:36.701 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:51:36.703 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:51:36Z" } 2024-04-12T04:51:36.703 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:51:36.703 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:36.703 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:36.703 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:51:36.704 DEBUG:boto:path=/dkyges-75/ 2024-04-12T04:51:36.704 DEBUG:boto:auth_path=/dkyges-75/ 2024-04-12T04:51:36.704 DEBUG:boto:Method: HEAD 2024-04-12T04:51:36.704 DEBUG:boto:Path: /dkyges-75/ 2024-04-12T04:51:36.704 DEBUG:boto:Data: 2024-04-12T04:51:36.704 DEBUG:boto:Headers: {} 2024-04-12T04:51:36.704 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:51:36.705 DEBUG:boto:Port: 8001 2024-04-12T04:51:36.705 DEBUG:boto:Params: {} 2024-04-12T04:51:36.705 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:51:36.705 DEBUG:boto:Token: None 2024-04-12T04:51:36.706 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:51:36 GMT /dkyges-75/ 2024-04-12T04:51:36.706 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:TmPzosGTFoKztdEfiyrILeHPEbA= 2024-04-12T04:51:36.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TmPzosGTFoKztdEfiyrILeHPEbA=', 'Content-Length': '0'} 2024-04-12T04:51:36.721 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006a2f78e4fc98180-006618bdd8-2ce2-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:51:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:36.721 DEBUG:boto:path=/dkyges-75/obj5 2024-04-12T04:51:36.721 DEBUG:boto:auth_path=/dkyges-75/obj5 2024-04-12T04:51:36.721 DEBUG:boto:Method: HEAD 2024-04-12T04:51:36.722 DEBUG:boto:Path: /dkyges-75/obj5 2024-04-12T04:51:36.722 DEBUG:boto:Data: 2024-04-12T04:51:36.722 DEBUG:boto:Headers: {} 2024-04-12T04:51:36.722 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:51:36.722 DEBUG:boto:Port: 8001 2024-04-12T04:51:36.722 DEBUG:boto:Params: {} 2024-04-12T04:51:36.722 DEBUG:boto:Token: None 2024-04-12T04:51:36.722 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:51:36 GMT /dkyges-75/obj5 2024-04-12T04:51:36.722 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:eifNZwBZSyNGwW1ervndm1KNF3k= 2024-04-12T04:51:36.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eifNZwBZSyNGwW1ervndm1KNF3k=', 'Content-Length': '0'} 2024-04-12T04:51:36.724 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007a5093a396d95f1d-006618bdd8-2ce2-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:51:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:36.724 DEBUG:boto:path=/dkyges-75/obj6 2024-04-12T04:51:36.724 DEBUG:boto:auth_path=/dkyges-75/obj6 2024-04-12T04:51:36.724 DEBUG:boto:Method: HEAD 2024-04-12T04:51:36.724 DEBUG:boto:Path: /dkyges-75/obj6 2024-04-12T04:51:36.725 DEBUG:boto:Data: 2024-04-12T04:51:36.725 DEBUG:boto:Headers: {} 2024-04-12T04:51:36.725 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:51:36.725 DEBUG:boto:Port: 8001 2024-04-12T04:51:36.725 DEBUG:boto:Params: {} 2024-04-12T04:51:36.725 DEBUG:boto:Token: None 2024-04-12T04:51:36.725 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:51:36 GMT /dkyges-75/obj6 2024-04-12T04:51:36.725 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:AEb/5RGFjVRDKudFRe446vqg89M= 2024-04-12T04:51:36.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AEb/5RGFjVRDKudFRe446vqg89M=', 'Content-Length': '0'} 2024-04-12T04:51:36.727 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000058b4eedcba898478-006618bdd8-2ce2-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:51:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:36.727 DEBUG:boto:path=/dkyges-75/obj7 2024-04-12T04:51:36.727 DEBUG:boto:auth_path=/dkyges-75/obj7 2024-04-12T04:51:36.727 DEBUG:boto:Method: HEAD 2024-04-12T04:51:36.727 DEBUG:boto:Path: /dkyges-75/obj7 2024-04-12T04:51:36.727 DEBUG:boto:Data: 2024-04-12T04:51:36.727 DEBUG:boto:Headers: {} 2024-04-12T04:51:36.728 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:51:36.728 DEBUG:boto:Port: 8001 2024-04-12T04:51:36.728 DEBUG:boto:Params: {} 2024-04-12T04:51:36.728 DEBUG:boto:Token: None 2024-04-12T04:51:36.728 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:51:36 GMT /dkyges-75/obj7 2024-04-12T04:51:36.728 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:hBtjLX+ZZuRilHdj/aMBa8hdBkU= 2024-04-12T04:51:36.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:hBtjLX+ZZuRilHdj/aMBa8hdBkU=', 'Content-Length': '0'} 2024-04-12T04:51:36.730 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000003a1d6bb65aa04529-006618bdd8-2ce2-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:51:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:36.730 DEBUG:boto:path=/dkyges-75/obj8 2024-04-12T04:51:36.730 DEBUG:boto:auth_path=/dkyges-75/obj8 2024-04-12T04:51:36.730 DEBUG:boto:Method: HEAD 2024-04-12T04:51:36.730 DEBUG:boto:Path: /dkyges-75/obj8 2024-04-12T04:51:36.730 DEBUG:boto:Data: 2024-04-12T04:51:36.730 DEBUG:boto:Headers: {} 2024-04-12T04:51:36.730 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:51:36.731 DEBUG:boto:Port: 8001 2024-04-12T04:51:36.731 DEBUG:boto:Params: {} 2024-04-12T04:51:36.731 DEBUG:boto:Token: None 2024-04-12T04:51:36.731 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:51:36 GMT /dkyges-75/obj8 2024-04-12T04:51:36.731 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Ag5mjrAR53La+TEI16y3tH1ma3I= 2024-04-12T04:51:36.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Ag5mjrAR53La+TEI16y3tH1ma3I=', 'Content-Length': '0'} 2024-04-12T04:51:36.732 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000048935468dec97264-006618bdd8-2ce2-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:51:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:36.733 DEBUG:boto:path=/dkyges-74/ 2024-04-12T04:51:36.733 DEBUG:boto:auth_path=/dkyges-74/ 2024-04-12T04:51:36.733 DEBUG:boto:Method: HEAD 2024-04-12T04:51:36.733 DEBUG:boto:Path: /dkyges-74/ 2024-04-12T04:51:36.733 DEBUG:boto:Data: 2024-04-12T04:51:36.733 DEBUG:boto:Headers: {} 2024-04-12T04:51:36.733 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:51:36.733 DEBUG:boto:Port: 8001 2024-04-12T04:51:36.733 DEBUG:boto:Params: {} 2024-04-12T04:51:36.733 DEBUG:boto:Token: None 2024-04-12T04:51:36.734 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:51:36 GMT /dkyges-74/ 2024-04-12T04:51:36.734 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:2HQqoQqd2zt6+M5fBVICyi7rPms= 2024-04-12T04:51:36.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2HQqoQqd2zt6+M5fBVICyi7rPms=', 'Content-Length': '0'} 2024-04-12T04:51:36.736 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000095f2de6ea00c509-006618bdd8-2ce2-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:51:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:36.736 DEBUG:boto:path=/dkyges-74/obj7 2024-04-12T04:51:36.737 DEBUG:boto:auth_path=/dkyges-74/obj7 2024-04-12T04:51:36.737 DEBUG:boto:Method: HEAD 2024-04-12T04:51:36.737 DEBUG:boto:Path: /dkyges-74/obj7 2024-04-12T04:51:36.737 DEBUG:boto:Data: 2024-04-12T04:51:36.737 DEBUG:boto:Headers: {} 2024-04-12T04:51:36.737 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:51:36.737 DEBUG:boto:Port: 8001 2024-04-12T04:51:36.737 DEBUG:boto:Params: {} 2024-04-12T04:51:36.737 DEBUG:boto:Token: None 2024-04-12T04:51:36.737 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:51:36 GMT /dkyges-74/obj7 2024-04-12T04:51:36.738 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Aq/BRmA1fVrXjef7B6W82v6lLxM= 2024-04-12T04:51:36.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Aq/BRmA1fVrXjef7B6W82v6lLxM=', 'Content-Length': '0'} 2024-04-12T04:51:36.740 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:50:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-75:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.10617.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043577069b5cdc6aa-006618bdd8-2ce2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:51:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:36.741 DEBUG:boto:path=/dkyges-74/obj7 2024-04-12T04:51:36.741 DEBUG:boto:auth_path=/dkyges-74/obj7 2024-04-12T04:51:36.741 DEBUG:boto:Method: GET 2024-04-12T04:51:36.741 DEBUG:boto:Path: /dkyges-74/obj7 2024-04-12T04:51:36.741 DEBUG:boto:Data: 2024-04-12T04:51:36.741 DEBUG:boto:Headers: {} 2024-04-12T04:51:36.741 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:51:36.741 DEBUG:boto:Port: 8001 2024-04-12T04:51:36.741 DEBUG:boto:Params: {} 2024-04-12T04:51:36.742 DEBUG:boto:Token: None 2024-04-12T04:51:36.742 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:51:36 GMT /dkyges-74/obj7 2024-04-12T04:51:36.742 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RedCwSXS07L69s3i1w6qveyBYVQ= 2024-04-12T04:51:36.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RedCwSXS07L69s3i1w6qveyBYVQ=', 'Content-Length': '0'} 2024-04-12T04:51:36.744 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:50:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-75:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.10617.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000068cff8f638858cdc-006618bdd8-2ce2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:51:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:36.744 DEBUG:boto:path=/dkyges-74/obj8 2024-04-12T04:51:36.744 DEBUG:boto:auth_path=/dkyges-74/obj8 2024-04-12T04:51:36.744 DEBUG:boto:Method: HEAD 2024-04-12T04:51:36.744 DEBUG:boto:Path: /dkyges-74/obj8 2024-04-12T04:51:36.744 DEBUG:boto:Data: 2024-04-12T04:51:36.744 DEBUG:boto:Headers: {} 2024-04-12T04:51:36.745 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:51:36.745 DEBUG:boto:Port: 8001 2024-04-12T04:51:36.745 DEBUG:boto:Params: {} 2024-04-12T04:51:36.745 DEBUG:boto:Token: None 2024-04-12T04:51:36.745 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:51:36 GMT /dkyges-74/obj8 2024-04-12T04:51:36.745 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GhhqSjgWGgMU1gRLooRL7nL8nuI= 2024-04-12T04:51:36.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GhhqSjgWGgMU1gRLooRL7nL8nuI=', 'Content-Length': '0'} 2024-04-12T04:51:36.747 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:50:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-75:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.10617.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003ac625ffe2b9bb95-006618bdd8-2ce2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:51:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:36.747 DEBUG:boto:path=/dkyges-74/obj8 2024-04-12T04:51:36.747 DEBUG:boto:auth_path=/dkyges-74/obj8 2024-04-12T04:51:36.747 DEBUG:boto:Method: GET 2024-04-12T04:51:36.747 DEBUG:boto:Path: /dkyges-74/obj8 2024-04-12T04:51:36.747 DEBUG:boto:Data: 2024-04-12T04:51:36.747 DEBUG:boto:Headers: {} 2024-04-12T04:51:36.747 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:51:36.747 DEBUG:boto:Port: 8001 2024-04-12T04:51:36.748 DEBUG:boto:Params: {} 2024-04-12T04:51:36.748 DEBUG:boto:Token: None 2024-04-12T04:51:36.748 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:51:36 GMT /dkyges-74/obj8 2024-04-12T04:51:36.748 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:By2pOzUDqH5+Tm+Br3YBF7bR8qo= 2024-04-12T04:51:36.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:By2pOzUDqH5+Tm+Br3YBF7bR8qo=', 'Content-Length': '0'} 2024-04-12T04:51:36.750 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:50:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-75:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.10617.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f24673349554f552-006618bdd8-2ce2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:51:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:36.790 DEBUG:boto:path=/dkyges-74/obj7 2024-04-12T04:51:36.791 DEBUG:boto:auth_path=/dkyges-74/obj7 2024-04-12T04:51:36.791 DEBUG:boto:Method: HEAD 2024-04-12T04:51:36.791 DEBUG:boto:Path: /dkyges-74/obj7 2024-04-12T04:51:36.791 DEBUG:boto:Data: 2024-04-12T04:51:36.791 DEBUG:boto:Headers: {} 2024-04-12T04:51:36.791 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:51:36.791 DEBUG:boto:Port: 8001 2024-04-12T04:51:36.791 DEBUG:boto:Params: {} 2024-04-12T04:51:36.791 DEBUG:boto:Token: None 2024-04-12T04:51:36.792 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:51:36 GMT /dkyges-74/obj7 2024-04-12T04:51:36.792 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Aq/BRmA1fVrXjef7B6W82v6lLxM= 2024-04-12T04:51:36.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Aq/BRmA1fVrXjef7B6W82v6lLxM=', 'Content-Length': '0'} 2024-04-12T04:51:36.793 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:50:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-75:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.10617.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005405c49bbd4805e0-006618bdd8-2ce2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:51:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:36.794 DEBUG:boto:path=/dkyges-74/obj7 2024-04-12T04:51:36.794 DEBUG:boto:auth_path=/dkyges-74/obj7 2024-04-12T04:51:36.794 DEBUG:boto:Method: GET 2024-04-12T04:51:36.794 DEBUG:boto:Path: /dkyges-74/obj7 2024-04-12T04:51:36.794 DEBUG:boto:Data: 2024-04-12T04:51:36.794 DEBUG:boto:Headers: {} 2024-04-12T04:51:36.794 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:51:36.795 DEBUG:boto:Port: 8001 2024-04-12T04:51:36.795 DEBUG:boto:Params: {} 2024-04-12T04:51:36.795 DEBUG:boto:Token: None 2024-04-12T04:51:36.795 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:51:36 GMT /dkyges-74/obj7 2024-04-12T04:51:36.795 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:RedCwSXS07L69s3i1w6qveyBYVQ= 2024-04-12T04:51:36.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RedCwSXS07L69s3i1w6qveyBYVQ=', 'Content-Length': '0'} 2024-04-12T04:51:36.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:50:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-75:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.10617.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008dfad879674b1ccc-006618bdd8-2ce2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:51:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:36.797 DEBUG:boto:path=/dkyges-74/obj8 2024-04-12T04:51:36.797 DEBUG:boto:auth_path=/dkyges-74/obj8 2024-04-12T04:51:36.797 DEBUG:boto:Method: HEAD 2024-04-12T04:51:36.798 DEBUG:boto:Path: /dkyges-74/obj8 2024-04-12T04:51:36.798 DEBUG:boto:Data: 2024-04-12T04:51:36.798 DEBUG:boto:Headers: {} 2024-04-12T04:51:36.798 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:51:36.798 DEBUG:boto:Port: 8001 2024-04-12T04:51:36.798 DEBUG:boto:Params: {} 2024-04-12T04:51:36.798 DEBUG:boto:Token: None 2024-04-12T04:51:36.798 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:51:36 GMT /dkyges-74/obj8 2024-04-12T04:51:36.798 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:GhhqSjgWGgMU1gRLooRL7nL8nuI= 2024-04-12T04:51:36.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GhhqSjgWGgMU1gRLooRL7nL8nuI=', 'Content-Length': '0'} 2024-04-12T04:51:36.800 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:50:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-75:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.10617.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b39c8f9808bf0f5c-006618bdd8-2ce2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:51:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:36.800 DEBUG:boto:path=/dkyges-74/obj8 2024-04-12T04:51:36.801 DEBUG:boto:auth_path=/dkyges-74/obj8 2024-04-12T04:51:36.801 DEBUG:boto:Method: GET 2024-04-12T04:51:36.801 DEBUG:boto:Path: /dkyges-74/obj8 2024-04-12T04:51:36.801 DEBUG:boto:Data: 2024-04-12T04:51:36.801 DEBUG:boto:Headers: {} 2024-04-12T04:51:36.801 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:51:36.801 DEBUG:boto:Port: 8001 2024-04-12T04:51:36.801 DEBUG:boto:Params: {} 2024-04-12T04:51:36.801 DEBUG:boto:Token: None 2024-04-12T04:51:36.801 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:51:36 GMT /dkyges-74/obj8 2024-04-12T04:51:36.802 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:By2pOzUDqH5+Tm+Br3YBF7bR8qo= 2024-04-12T04:51:36.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:36 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:By2pOzUDqH5+Tm+Br3YBF7bR8qo=', 'Content-Length': '0'} 2024-04-12T04:51:36.803 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:50:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-75:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.10617.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000941575c04977ee95-006618bdd8-2ce2-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:51:36 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:36.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dkyges-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:36.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dkyges-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:36.804 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dkyges-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:51:36.863 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:36.863 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:36.934 INFO:teuthology.orchestra.run.smithi043.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:51:36.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:36.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:36.935 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:51:36.988 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:36.988 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:37.046 INFO:teuthology.orchestra.run.smithi043.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:51:37.046 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... ok 2024-04-12T04:51:37.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:37.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:37.048 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:37.101 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:37.102 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:51:37.154 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:51:37.154 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:37.154 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:51:37.154 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:51:37.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:37.154 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:37.155 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:51:37.155 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:51:37.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:37.155 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:37.155 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:51:37.155 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:51:37.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:37.155 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:37.155 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897496.907543_1012.1", 2024-04-12T04:51:37.155 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:51:36.907543Z" 2024-04-12T04:51:37.155 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:37.156 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:51:37.156 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897496.907543_1012.1'} 2024-04-12T04:51:37.157 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:51:37.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:37.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:37.157 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:37.214 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:37.214 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:37.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:51:37.246+0000 7f637007b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:51:37.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:51:37.247+0000 7f637007b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:51:37.266 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:51:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:51:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:51:37.266 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:51:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:51:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:51:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:51:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:51:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:51:37.267 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:51:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:51:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:51:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:37.268 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:51:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:51:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:51:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:37.269 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:51:37.270 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:37.271 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:51:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:51:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:37.272 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897372.707923_1010.1", 2024-04-12T04:51:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:49:32.707923Z", 2024-04-12T04:51:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:51:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:51:37.273 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:51:37.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:37.274 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:51:37Z" 2024-04-12T04:51:37.274 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:51:37.275 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897372.707923_1010.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:49:32.707923Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:51:37Z" } 2024-04-12T04:51:37.275 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897372.707923_1010.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:49:32.707923Z', 'realm_epoch': 2}}] 2024-04-12T04:51:37.275 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897496.907543_1012.1'} 2024-04-12T04:51:37.275 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897372.707923_1010.1'} 2024-04-12T04:51:37.275 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897496.907543_1012.1 target=1_1712897372.707923_1010.1 2024-04-12T04:51:42.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:42.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:42.277 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:42.337 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:42.337 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:42.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:51:42.374+0000 7fd47f070780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:51:42.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:51:42.377+0000 7fd47f070780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:51:42.396 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:51:42.396 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:51:42.397 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:51:42.397 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:51:42.397 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:51:42.397 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:51:42.397 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:42.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:42.397 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:51:42.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:42.397 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:51:42.397 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:42.397 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:42.398 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:51:42.398 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:42.398 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:51:42.398 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:42.398 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:51:42.398 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:42.398 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:42.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:42.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:42.398 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:51:42.399 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:42.399 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:42.399 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:51:42.399 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:42.399 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:42.399 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:42.399 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:51:42.399 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:42.400 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:42.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:42.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:42.400 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:51:42.400 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:42.400 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:42.400 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:51:42.400 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:42.400 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:42.400 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:42.400 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:51:42.401 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:42.401 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:42.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:42.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:42.401 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:51:42.401 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:42.401 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:42.401 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897372.707923_1010.1", 2024-04-12T04:51:42.401 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:42.401 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:42.401 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:42.402 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:49:32.707923Z", 2024-04-12T04:51:42.402 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:42.402 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:42.402 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:42.402 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:42.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:42.402 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:51:42.402 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:51:42.402 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:51:42.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:42.402 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:51:42Z" 2024-04-12T04:51:42.403 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:51:42.403 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897372.707923_1010.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:49:32.707923Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:51:42Z" } 2024-04-12T04:51:42.404 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897372.707923_1010.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:49:32.707923Z', 'realm_epoch': 2}}] 2024-04-12T04:51:42.404 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897496.907543_1012.1'} 2024-04-12T04:51:42.404 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897372.707923_1010.1'} 2024-04-12T04:51:42.404 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897496.907543_1012.1 target=1_1712897372.707923_1010.1 2024-04-12T04:51:47.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:47.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:47.406 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:47.466 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:47.466 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:47.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:51:47.499+0000 7f615b803780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:51:47.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:51:47.500+0000 7f615b803780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:51:47.519 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:51:47.519 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:51:47.519 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:51:47.519 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:51:47.519 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:51:47.519 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:51:47.519 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:47.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:47.520 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:51:47.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:47.520 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:51:47.520 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:47.520 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:47.520 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:51:47.520 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:47.520 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:51:47.520 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:47.520 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:51:47.520 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:47.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:47.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:47.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:47.521 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:51:47.521 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:47.521 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:47.521 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:51:47.521 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:47.521 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:47.521 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:47.522 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:51:47.522 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:47.522 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:47.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:47.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:47.522 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:51:47.522 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:47.522 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:47.522 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:51:47.522 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:47.523 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:47.523 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:47.523 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:51:47.523 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:47.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:47.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:47.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:47.523 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:51:47.523 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:47.523 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:47.523 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897496.907543_1012.1", 2024-04-12T04:51:47.524 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:47.524 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:47.524 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:47.524 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:36.907543Z", 2024-04-12T04:51:47.524 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:47.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:47.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:47.524 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:47.524 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:47.524 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:51:47.524 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:51:47.524 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:51:47.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:47.525 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:51:47Z" 2024-04-12T04:51:47.525 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:51:47.526 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897496.907543_1012.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:36.907543Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:51:47Z" } 2024-04-12T04:51:47.526 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897496.907543_1012.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:36.907543Z', 'realm_epoch': 2}}] 2024-04-12T04:51:47.526 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897496.907543_1012.1'} 2024-04-12T04:51:47.526 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897496.907543_1012.1'} 2024-04-12T04:51:47.526 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:51:47.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:47.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:47.527 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:51:47.577 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:47.578 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:47.631 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[]} 2024-04-12T04:51:47.632 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-76 2024-04-12T04:51:47.632 DEBUG:boto:path=/dkyges-76/ 2024-04-12T04:51:47.632 DEBUG:boto:auth_path=/dkyges-76/ 2024-04-12T04:51:47.632 DEBUG:boto:Method: PUT 2024-04-12T04:51:47.633 DEBUG:boto:Path: /dkyges-76/ 2024-04-12T04:51:47.633 DEBUG:boto:Data: 2024-04-12T04:51:47.633 DEBUG:boto:Headers: {} 2024-04-12T04:51:47.633 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:51:47.633 DEBUG:boto:Port: 8000 2024-04-12T04:51:47.633 DEBUG:boto:Params: {} 2024-04-12T04:51:47.633 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:51:47.634 DEBUG:boto:Token: None 2024-04-12T04:51:47.634 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:51:47 GMT /dkyges-76/ 2024-04-12T04:51:47.634 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:PUzUkBUYj4GwpIS9A/NSEwZNgI0= 2024-04-12T04:51:47.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PUzUkBUYj4GwpIS9A/NSEwZNgI0=', 'Content-Length': '0'} 2024-04-12T04:51:47.664 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002802e0ca09dd04ba-006618bde3-2cea-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:51:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:47.664 INFO:rgw_multi.tests:create bucket zone=a1 name=dkyges-77 2024-04-12T04:51:47.664 DEBUG:boto:path=/dkyges-77/ 2024-04-12T04:51:47.664 DEBUG:boto:auth_path=/dkyges-77/ 2024-04-12T04:51:47.664 DEBUG:boto:Method: PUT 2024-04-12T04:51:47.664 DEBUG:boto:Path: /dkyges-77/ 2024-04-12T04:51:47.664 DEBUG:boto:Data: 2024-04-12T04:51:47.665 DEBUG:boto:Headers: {} 2024-04-12T04:51:47.665 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:51:47.665 DEBUG:boto:Port: 8000 2024-04-12T04:51:47.665 DEBUG:boto:Params: {} 2024-04-12T04:51:47.665 DEBUG:boto:Token: None 2024-04-12T04:51:47.665 DEBUG:boto:StringToSign: PUT Fri, 12 Apr 2024 04:51:47 GMT /dkyges-77/ 2024-04-12T04:51:47.665 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:fCYRZ82Ww8aP9s137LnkDlRXTZs= 2024-04-12T04:51:47.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fCYRZ82Ww8aP9s137LnkDlRXTZs=', 'Content-Length': '0'} 2024-04-12T04:51:47.684 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fe58bc9c6e2e71c4-006618bde3-2cea-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:51:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:47.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:47.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:47.684 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:47.745 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:47.745 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:47.789 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:51:47.806 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:51:47.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:47.806 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:51:47.806 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:51:47.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:47.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:47.806 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:51:47.806 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:51:47.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:47.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:47.807 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:51:47.807 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:51:47.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:47.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:47.807 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:51:47.807 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:51:47.680289Z" 2024-04-12T04:51:47.807 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:47.807 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:51:47.808 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:51:47.809 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:51:47.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:47.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:47.809 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:47.863 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:47.863 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:47.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:51:47.895+0000 7f424a65a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:51:47.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:51:47.896+0000 7f424a65a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:51:47.914 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:51:47.914 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:51:47.914 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:51:47.914 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:51:47.915 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:51:47.915 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:51:47.915 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:47.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:47.915 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:51:47.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:47.915 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:51:47.915 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:47.915 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:47.916 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:51:47.916 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:47.916 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:51:47.916 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:47.916 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:51:47.916 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:47.916 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:47.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:47.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:47.916 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:51:47.917 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:47.917 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:47.917 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:51:47.917 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:47.917 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:47.917 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:47.917 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:51:47.917 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:47.917 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:47.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:47.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:47.918 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:51:47.918 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:47.918 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:47.918 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:51:47.918 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:47.919 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:47.919 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:47.919 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:51:47.919 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:47.919 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:47.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:47.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:47.919 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:51:47.919 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:47.920 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:47.920 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:51:47.920 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:47.920 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:47.920 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:47.920 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:51:47.920 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:47.921 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:47.921 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:47.921 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:47.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:47.921 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:51:47.921 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:51:47.922 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:51:47.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:47.922 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:51:47Z" 2024-04-12T04:51:47.922 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:51:47.923 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:51:47Z" } 2024-04-12T04:51:47.923 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:51:47.924 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:51:47.924 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:51:47.924 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:51:47.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:47.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:47.924 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:51:47.979 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:47.979 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:48.034 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:51:48.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:48.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:48.035 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:51:48.093 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:48.093 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:48.149 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:51:48.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dkyges-76', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dkyges-76', '--dest-bucket=dkyges-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:48.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dkyges-76', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dkyges-76', '--dest-bucket=dkyges-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:48.149 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dkyges-76 --source-zones=a1 --dest-zones=a2 --source-bucket=dkyges-76 --dest-bucket=dkyges-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:51:48.209 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:48.209 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:48.265 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dkyges-76","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:51:48.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dkyges-77', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dkyges-76', '--dest-bucket=dkyges-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:48.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-dkyges-77', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dkyges-76', '--dest-bucket=dkyges-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:48.266 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-dkyges-77 --source-zones=a1 --dest-zones=a2 --source-bucket=dkyges-76 --dest-bucket=dkyges-77 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:51:48.326 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:48.327 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:48.382 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dkyges-76","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dkyges-77","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:51:48.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dkyges-76', '--dest-bucket=dkyges-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:48.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=dkyges-76', '--dest-bucket=dkyges-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:48.383 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=dkyges-76 --dest-bucket=dkyges-77 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:51:48.443 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:48.444 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:48.500 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dkyges-76","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dkyges-77","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:51:48.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:48.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:48.500 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:51:48.555 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:48.555 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:48.612 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dkyges-76","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dkyges-77","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:51:48.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:48.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:48.613 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:51:48.666 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:48.666 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:48.728 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:51:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:51:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 22, 2024-04-12T04:51:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T04:51:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T04:51:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T04:51:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:51:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T04:51:48.728 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T04:51:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T04:51:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:51:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T04:51:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:51:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:51:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:51:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:51:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:51:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:51:48.729 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:51:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:51:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T04:51:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:51:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T04:51:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:51:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:51:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:51:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:51:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:51:48.730 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:51:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:51:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:51:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:51:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:51:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:51:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:51:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.731 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:51:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T04:51:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:51:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:51:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:51:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:51:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:51:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:51:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:51:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:51:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:51:48.732 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:51:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:51:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:51:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:51:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:51:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:51:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:51:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:51:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:51:48.733 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:51:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:51:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:51:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:51:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:51:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:51:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [ 2024-04-12T04:51:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-group", 2024-04-12T04:51:48.734 INFO:teuthology.orchestra.run.smithi043.stdout: "data_flow": { 2024-04-12T04:51:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: "symmetrical": [ 2024-04-12T04:51:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-flow1", 2024-04-12T04:51:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:51:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:51:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:51:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.735 INFO:teuthology.orchestra.run.smithi043.stdout: "pipes": [ 2024-04-12T04:51:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-pipe-dkyges-76", 2024-04-12T04:51:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:51:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-76", 2024-04-12T04:51:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:51:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:51:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": { 2024-04-12T04:51:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-76", 2024-04-12T04:51:48.736 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:51:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2" 2024-04-12T04:51:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "params": { 2024-04-12T04:51:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:51:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "filter": { 2024-04-12T04:51:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [] 2024-04-12T04:51:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": {}, 2024-04-12T04:51:48.737 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-12T04:51:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "system", 2024-04-12T04:51:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "" 2024-04-12T04:51:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-pipe-dkyges-77", 2024-04-12T04:51:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:51:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-76", 2024-04-12T04:51:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:51:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:51:48.738 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": { 2024-04-12T04:51:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-77", 2024-04-12T04:51:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:51:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2" 2024-04-12T04:51:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "params": { 2024-04-12T04:51:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:51:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "filter": { 2024-04-12T04:51:48.739 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [] 2024-04-12T04:51:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": {}, 2024-04-12T04:51:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-12T04:51:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "system", 2024-04-12T04:51:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "" 2024-04-12T04:51:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-pipe", 2024-04-12T04:51:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:51:48.740 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-76", 2024-04-12T04:51:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:51:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:51:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": { 2024-04-12T04:51:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-77", 2024-04-12T04:51:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:51:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2" 2024-04-12T04:51:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.741 INFO:teuthology.orchestra.run.smithi043.stdout: "params": { 2024-04-12T04:51:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:51:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "filter": { 2024-04-12T04:51:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [] 2024-04-12T04:51:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": {}, 2024-04-12T04:51:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-12T04:51:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "system", 2024-04-12T04:51:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "" 2024-04-12T04:51:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:48.742 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:51:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "enabled" 2024-04-12T04:51:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:51:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:51:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.743 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T04:51:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T04:51:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:51:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T04:51:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:51:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:51:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:51:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:51:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:51:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:51:48.744 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:51:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:51:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T04:51:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:51:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:51:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:51:48.745 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:51:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:51:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:51:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:51:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:51:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:51:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:51:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:51:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:51:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:51:48.746 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:51:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:51:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T04:51:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:51:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T04:51:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:51:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:51:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:51:48.747 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:51:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:51:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:51:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:51:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:51:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:51:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:51:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:51:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:51:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:48.748 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:51:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:51:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:51:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:51:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:51:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:51:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:51:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:51:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:51:48.749 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:51:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T04:51:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:51:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:51:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:51:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T04:51:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:51:48.750 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T04:51:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:51:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T04:51:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:51:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T04:51:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:51:48.751 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T04:51:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T04:51:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:51:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T04:51:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T04:51:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:51:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:51:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:51:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:51:48.752 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:51:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T04:51:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:51:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:51:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:51:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:51:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:51:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T04:51:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:51:48.753 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:51:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:51:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:51:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:51:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T04:51:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:51:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:51:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:51:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:51:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:51:48.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T04:51:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:51:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:51:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:51:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:51:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:51:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:51:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T04:51:48.755 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:48.755 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:51:48.757 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:48.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:51:48.757 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:51:48.808 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:48.808 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:48.865 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dkyges-76","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dkyges-77","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:51:48.865 DEBUG:boto:path=/dkyges-76/ 2024-04-12T04:51:48.865 DEBUG:boto:auth_path=/dkyges-76/ 2024-04-12T04:51:48.865 DEBUG:boto:Method: HEAD 2024-04-12T04:51:48.865 DEBUG:boto:Path: /dkyges-76/ 2024-04-12T04:51:48.866 DEBUG:boto:Data: 2024-04-12T04:51:48.866 DEBUG:boto:Headers: {} 2024-04-12T04:51:48.866 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:51:48.866 DEBUG:boto:Port: 8000 2024-04-12T04:51:48.866 DEBUG:boto:Params: {} 2024-04-12T04:51:48.866 DEBUG:boto:Token: None 2024-04-12T04:51:48.866 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:51:48 GMT /dkyges-76/ 2024-04-12T04:51:48.866 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:J8hDtIZR1ImmnNY29zp+kwRMQ1c= 2024-04-12T04:51:48.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:J8hDtIZR1ImmnNY29zp+kwRMQ1c=', 'Content-Length': '0'} 2024-04-12T04:51:49.170 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5ddc4102d723812-006618bde5-2e77-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:51:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:49.171 DEBUG:boto:path=/dkyges-76/obj1 2024-04-12T04:51:49.171 DEBUG:boto:auth_path=/dkyges-76/obj1 2024-04-12T04:51:49.171 DEBUG:boto:Method: PUT 2024-04-12T04:51:49.172 DEBUG:boto:Path: /dkyges-76/obj1 2024-04-12T04:51:49.172 DEBUG:boto:Data: 2024-04-12T04:51:49.172 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:51:49.172 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:51:49.172 DEBUG:boto:Port: 8000 2024-04-12T04:51:49.172 DEBUG:boto:Params: {} 2024-04-12T04:51:49.173 DEBUG:boto:Token: None 2024-04-12T04:51:49.173 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:51:49 GMT /dkyges-76/obj1 2024-04-12T04:51:49.173 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:WeB8YWbWoRrx3NipVycvJjmfcuE= 2024-04-12T04:51:49.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:51:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WeB8YWbWoRrx3NipVycvJjmfcuE='} 2024-04-12T04:51:49.201 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000044c7f9c0369cbea5-006618bde5-2e77-a1'), ('Date', 'Fri, 12 Apr 2024 04:51:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:49.201 DEBUG:boto:path=/dkyges-76/ 2024-04-12T04:51:49.201 DEBUG:boto:auth_path=/dkyges-76/ 2024-04-12T04:51:49.201 DEBUG:boto:Method: HEAD 2024-04-12T04:51:49.201 DEBUG:boto:Path: /dkyges-76/ 2024-04-12T04:51:49.202 DEBUG:boto:Data: 2024-04-12T04:51:49.202 DEBUG:boto:Headers: {} 2024-04-12T04:51:49.202 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:51:49.202 DEBUG:boto:Port: 8000 2024-04-12T04:51:49.202 DEBUG:boto:Params: {} 2024-04-12T04:51:49.202 DEBUG:boto:Token: None 2024-04-12T04:51:49.202 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:51:49 GMT /dkyges-76/ 2024-04-12T04:51:49.202 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:aIYpsCVvsNfOx8luFv5ulrM2RSU= 2024-04-12T04:51:49.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:51:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aIYpsCVvsNfOx8luFv5ulrM2RSU=', 'Content-Length': '0'} 2024-04-12T04:51:49.208 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007cbbf1143ab2f59d-006618bde5-2e77-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:51:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:49.209 DEBUG:boto:path=/dkyges-76/obj2 2024-04-12T04:51:49.209 DEBUG:boto:auth_path=/dkyges-76/obj2 2024-04-12T04:51:49.209 DEBUG:boto:Method: PUT 2024-04-12T04:51:49.209 DEBUG:boto:Path: /dkyges-76/obj2 2024-04-12T04:51:49.209 DEBUG:boto:Data: 2024-04-12T04:51:49.209 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:51:49.209 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:51:49.209 DEBUG:boto:Port: 8000 2024-04-12T04:51:49.210 DEBUG:boto:Params: {} 2024-04-12T04:51:49.210 DEBUG:boto:Token: None 2024-04-12T04:51:49.210 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:51:49 GMT /dkyges-76/obj2 2024-04-12T04:51:49.210 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6P09mBVuUg2wqyP2he6EAQaOx5E= 2024-04-12T04:51:49.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:51:49 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6P09mBVuUg2wqyP2he6EAQaOx5E='} 2024-04-12T04:51:49.255 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003a5507e25fa03cf-006618bde5-2e77-a1'), ('Date', 'Fri, 12 Apr 2024 04:51:49 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:51:49.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:49.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:49.255 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:49.353 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:49.353 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:49.472 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:51:49.533 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:51:49.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:49.533 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:51:49.533 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:51:49.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:49.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:49.533 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:51:49.534 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:51:49.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:49.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:49.534 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:51:49.534 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:51:49.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:49.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:49.534 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:51:49.534 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:51:47.680289Z" 2024-04-12T04:51:49.534 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:49.535 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:51:49.535 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:51:49.536 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:51:49.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:49.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:49.536 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:49.618 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:49.618 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:49.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:51:49.688+0000 7fab06ddd780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:51:49.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:51:49.689+0000 7fab06ddd780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:51:49.722 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:51:49.723 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:51:49.723 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:51:49.723 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:51:49.723 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:51:49.723 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:51:49.723 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:49.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:49.723 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:51:49.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:49.723 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:51:49.724 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:49.724 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:49.724 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:51:49.724 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:49.724 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:51:49.724 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:49.724 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:51:49.724 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:49.724 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:49.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:49.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:49.725 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:51:49.725 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:49.725 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:49.725 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:51:49.725 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:49.725 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:49.725 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:49.725 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:51:49.725 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:49.725 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:49.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:49.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:49.726 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:51:49.726 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:49.726 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:49.726 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:51:49.726 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:49.726 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:49.726 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:49.726 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:51:49.726 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:49.727 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:49.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:49.727 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:49.727 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:51:49.727 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:49.727 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:51:49.727 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:51:49.727 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:49.728 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:49.728 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:49.728 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:51:49.728 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:51:49.728 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:49.728 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:49.728 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:49.728 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:49.728 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:51:49.728 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:51:49.728 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:51:49.729 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:49.729 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:51:49Z" 2024-04-12T04:51:49.729 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:51:49.730 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:51:49Z" } 2024-04-12T04:51:49.730 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:51:49.730 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:51:49.730 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:51:49.730 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:51:49.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:49.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:49.731 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:49.803 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:49.804 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:49.893 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:51:49.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:49.893 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048340", 2024-04-12T04:51:49.893 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:51:49.229855Z" 2024-04-12T04:51:49.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:49.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:49.894 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:51:49.894 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:51:49.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:49.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:49.894 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055334", 2024-04-12T04:51:49.894 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:51:49.195479Z" 2024-04-12T04:51:49.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:49.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:49.895 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:51:49.895 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:51:49.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:49.895 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:51:49.896 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:49.896 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:51:49.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:49.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:49.897 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:49.963 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:49.963 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:50.016 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:51:50.016 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:51:50.017 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:51:50.017 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:51:50.017 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:51:50.017 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:51:50.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:50.018 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:51:50.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:50.018 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:51:50.018 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:50.019 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:50.019 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:51:50.019 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:50.019 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:50.019 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:50.019 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:51:50.019 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:50.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:50.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:50.020 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:51:50.020 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:50.020 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:50.021 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:51:50.021 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:50.021 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:50.021 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:50.021 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:51:50.022 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:50.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:50.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:50.022 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:51:50.022 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:50.022 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:50.022 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:51:50.023 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:50.023 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:51:50.023 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:50.023 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:51:50.023 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:50.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:50.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:50.023 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:51:50.023 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:50.024 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:50.024 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:51:50.024 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:50.024 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:51:50.024 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:50.024 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:51:50.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:50.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:50.024 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:50.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:50.025 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:51:50.025 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:51:50.025 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:51:50.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:50.025 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:51:49Z" 2024-04-12T04:51:50.025 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:51:50.026 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:51:49Z" } 2024-04-12T04:51:50.027 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:51:50.027 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:50.027 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:50.028 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:51:55.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:55.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:51:55.029 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:51:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:51:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:51:55.147 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:51:55.147 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:51:55.147 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:51:55.148 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:51:55.148 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:51:55.148 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:51:55.148 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:55.148 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:51:55.148 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:55.148 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:51:55.148 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:55.148 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:55.148 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:51:55.148 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:55.148 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:55.149 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:55.149 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:51:55.149 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:55.149 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:55.149 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:55.149 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:51:55.149 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:55.149 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:55.149 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:51:55.149 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:55.149 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:51:55.150 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:55.150 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:51:55.150 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:55.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:55.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:55.150 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:51:55.150 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:55.150 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:55.150 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:51:55.150 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:55.150 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:51:55.150 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:55.151 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:51:55.151 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:55.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:55.151 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:51:55.151 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:51:55.151 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:51:55.151 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:51:55.151 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:51:55.151 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:51:55.151 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:51:55.151 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:51:55.152 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:51:55.152 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:55.152 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:51:55.152 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:51:55.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:55.152 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:51:55.152 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:51:55.152 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:51:55.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:51:55.152 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:51:55Z" 2024-04-12T04:51:55.152 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:51:55.153 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:51:55Z" } 2024-04-12T04:51:55.153 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:51:55.154 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:55.154 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:51:55.154 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:52:00.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:00.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:00.155 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:52:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:52:00.226 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:52:00.273 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:52:00.273 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:52:00.273 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:52:00.274 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:52:00.274 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:52:00.274 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:52:00.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:00.274 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:52:00.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:00.274 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:52:00.274 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:00.274 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:00.274 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:52:00.275 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:00.275 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:00.275 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:00.275 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:52:00.275 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:00.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:00.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:00.275 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:52:00.275 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:00.275 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:00.275 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:52:00.276 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:00.276 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:00.276 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:00.276 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:52:00.276 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:00.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:00.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:00.276 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:52:00.276 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:00.276 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:00.276 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:52:00.277 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:00.277 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:52:00.277 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:00.277 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:52:00.277 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:00.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:00.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:00.277 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:52:00.278 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:00.278 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:00.278 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:52:00.278 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:00.278 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:52:00.278 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:00.278 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:52:00.278 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:00.278 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:00.278 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:52:00.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:00.279 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:52:00.279 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:52:00.279 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:52:00.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:00.279 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:52:00Z" 2024-04-12T04:52:00.279 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:52:00.280 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:52:00Z" } 2024-04-12T04:52:00.280 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:52:00.280 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:00.281 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:00.281 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:52:05.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:05.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:05.282 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:52:05.347 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:52:05.347 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:52:05.406 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:52:05.406 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:52:05.406 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:52:05.406 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:52:05.406 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:52:05.407 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:52:05.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:05.407 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:52:05.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:05.407 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:52:05.407 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:05.407 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:05.407 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:52:05.407 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:05.407 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:05.407 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:05.408 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:52:05.408 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:05.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:05.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:05.408 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:52:05.408 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:05.408 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:05.408 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:52:05.408 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:05.409 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:05.409 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:05.409 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:52:05.409 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:05.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:05.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:05.409 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:52:05.409 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:05.409 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:05.409 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:52:05.409 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:05.410 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:52:05.410 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:05.410 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:52:05.410 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:05.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:05.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:05.410 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:52:05.410 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:05.410 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:05.410 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:52:05.410 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:05.410 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:52:05.411 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:05.411 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:52:05.411 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:05.411 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:05.411 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:52:05.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:05.411 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:52:05.411 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:52:05.411 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:52:05.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:05.411 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:52:05Z" 2024-04-12T04:52:05.412 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:52:05.413 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:52:05Z" } 2024-04-12T04:52:05.413 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:52:05.413 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:05.413 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:05.413 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:52:10.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:10.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:10.414 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:52:10.476 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:52:10.476 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:52:10.531 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:52:10.532 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:52:10.532 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:52:10.532 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:52:10.532 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:52:10.532 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:52:10.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:10.532 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:52:10.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:10.533 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:52:10.533 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:10.533 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:10.533 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:52:10.533 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:10.533 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:10.533 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:10.534 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:52:10.534 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:10.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:10.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:10.534 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:52:10.534 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:10.534 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:10.534 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:52:10.534 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:52:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:52:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:10.535 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:52:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:52:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:52:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:52:10.536 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:10.537 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:10.537 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:52:10.537 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:10.537 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:52:10.537 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:10.537 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:52:10.537 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:10.537 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:10.537 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:52:10.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:10.538 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:52:10.538 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:52:10.538 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:52:10.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:10.538 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:52:10Z" 2024-04-12T04:52:10.538 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:52:10.539 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:52:10Z" } 2024-04-12T04:52:10.540 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:52:10.540 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:10.541 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:10.541 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:52:15.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:15.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:15.542 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:52:15.604 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:52:15.604 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:52:15.659 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:52:15.660 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:52:15.660 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:52:15.660 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:52:15.660 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:52:15.660 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:52:15.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:15.660 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:52:15.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:15.661 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:52:15.661 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:15.661 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:15.661 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:52:15.661 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:15.661 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:15.661 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:15.661 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:52:15.661 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:15.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:15.661 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:15.662 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:52:15.662 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:15.662 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:15.662 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:52:15.662 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:15.662 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:15.662 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:15.662 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:52:15.662 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:15.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:15.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:15.663 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:52:15.663 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:15.663 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:15.663 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:52:15.663 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:15.663 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:52:15.663 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:15.663 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:52:15.663 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:15.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:15.664 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:15.664 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:52:15.664 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:15.664 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:15.664 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:52:15.664 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:15.664 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:52:15.664 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:15.664 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:52:15.664 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:15.664 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:15.665 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:52:15.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:15.665 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:52:15.665 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:52:15.665 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:52:15.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:15.665 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:52:15Z" 2024-04-12T04:52:15.665 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:52:15.666 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:52:15Z" } 2024-04-12T04:52:15.666 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:52:15.667 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:15.667 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:15.667 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:52:20.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:20.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:20.669 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:52:20.731 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:52:20.731 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:52:20.789 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:52:20.789 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:52:20.789 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:52:20.789 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:52:20.789 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:52:20.789 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:52:20.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:20.789 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:52:20.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:20.790 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:52:20.790 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:20.790 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:20.790 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:52:20.790 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:20.790 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:20.790 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:20.790 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:52:20.790 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:20.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:20.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:20.791 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:52:20.791 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:20.791 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:20.791 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:52:20.791 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:20.791 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:20.791 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:20.791 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:52:20.791 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:20.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:20.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:20.792 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:52:20.792 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:20.792 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:20.792 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:52:20.792 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:20.792 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:52:20.792 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:20.792 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:52:20.793 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:20.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:20.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:20.793 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:52:20.793 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:20.793 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:20.793 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:52:20.793 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:20.793 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:52:20.793 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:20.793 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:52:20.793 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:20.794 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:20.794 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:52:20.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:20.794 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:52:20.794 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:52:20.794 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:52:20.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:20.794 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:52:20Z" 2024-04-12T04:52:20.794 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:52:20.795 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:52:20Z" } 2024-04-12T04:52:20.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:52:20.796 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:20.796 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:20.796 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:52:25.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:25.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:25.798 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:52:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:52:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:52:25.915 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:52:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:52:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:52:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:52:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:52:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:52:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:25.915 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:52:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:52:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:52:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:52:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:25.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:52:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:52:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:52:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:25.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:52:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:52:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:52:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:52:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:25.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:52:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:52:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:52:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:52:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:25.919 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:52:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:52:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:52:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:52:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:25.920 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:52:25Z" 2024-04-12T04:52:25.920 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:52:25.921 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:52:25Z" } 2024-04-12T04:52:25.921 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:52:25.921 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:25.922 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:25.922 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:52:30.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:30.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:30.923 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:52:30.983 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:52:30.984 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:52:31.041 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:52:31.042 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:52:31.042 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:52:31.042 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:52:31.042 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:52:31.042 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:52:31.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:31.042 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:52:31.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:31.043 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:52:31.043 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:31.043 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:31.043 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:52:31.043 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:31.043 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:31.043 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:31.043 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:52:31.043 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:31.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:31.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:31.043 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:52:31.044 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:31.044 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:31.044 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:52:31.044 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:31.044 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:31.044 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:31.044 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:52:31.044 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:31.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:31.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:31.044 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:52:31.044 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:31.045 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:31.045 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:52:31.045 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:31.045 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:52:31.045 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:31.045 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:52:31.045 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:31.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:31.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:31.045 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:52:31.045 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:31.045 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:31.045 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:52:31.046 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:31.046 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:52:31.046 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:31.046 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:52:31.046 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:31.046 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:31.046 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:52:31.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:31.046 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:52:31.046 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:52:31.046 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:52:31.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:31.047 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:52:31Z" 2024-04-12T04:52:31.047 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:52:31.048 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:52:31Z" } 2024-04-12T04:52:31.048 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:52:31.048 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:31.048 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:31.048 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:52:36.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:36.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:36.050 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:52:36.112 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:52:36.113 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:52:36.169 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:52:36.169 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:52:36.169 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:52:36.169 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:52:36.169 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:52:36.169 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:52:36.169 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:36.170 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:52:36.170 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:36.170 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:52:36.170 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:36.170 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:36.170 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:52:36.170 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:36.170 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:36.171 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:36.171 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:52:36.171 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:36.171 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:36.171 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:36.171 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:52:36.171 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:36.171 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:36.171 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:52:36.171 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:36.172 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:36.172 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:36.172 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:52:36.172 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:36.172 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:36.172 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:36.172 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:52:36.172 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:36.172 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:36.172 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:52:36.173 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:36.173 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:52:36.173 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:36.173 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:52:36.173 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:36.173 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:36.173 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:36.173 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:52:36.173 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:36.173 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:36.174 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:52:36.174 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:36.174 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:52:36.174 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:36.174 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:52:36.174 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:36.174 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:36.174 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:52:36.174 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:36.175 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:52:36.175 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:52:36.175 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:52:36.175 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:36.175 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:52:36Z" 2024-04-12T04:52:36.175 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:52:36.176 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:52:36Z" } 2024-04-12T04:52:36.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:52:36.176 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:36.177 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:36.177 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:52:41.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:41.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:41.178 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:52:41.240 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:52:41.240 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:52:41.299 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:52:41.299 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:52:41.299 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:52:41.299 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:52:41.299 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:52:41.299 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:52:41.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:41.299 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:52:41.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:41.300 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:52:41.300 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:41.300 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:41.300 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:52:41.300 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:41.300 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:41.300 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:41.300 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:52:41.300 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:41.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:41.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:41.301 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:52:41.301 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:41.301 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:41.301 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:52:41.301 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:41.301 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:41.301 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:41.301 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:52:41.301 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:41.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:41.302 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:41.302 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:52:41.302 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:41.302 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:41.302 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:52:41.302 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:41.302 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:52:41.302 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:41.302 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:52:41.302 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:41.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:41.303 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:41.303 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:52:41.303 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:41.303 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:41.303 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:52:41.303 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:41.303 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:52:41.303 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:41.303 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:52:41.303 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:41.304 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:41.304 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:52:41.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:41.304 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:52:41.304 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:52:41.304 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:52:41.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:41.304 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:52:41Z" 2024-04-12T04:52:41.304 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:52:41.305 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:52:41Z" } 2024-04-12T04:52:41.305 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:52:41.306 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:41.306 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:41.306 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:52:46.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:46.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:46.307 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:52:46.367 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:52:46.368 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:52:46.424 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:52:46.424 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:52:46.424 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:52:46.424 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:52:46.424 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:52:46.424 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:52:46.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:46.424 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:52:46.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:46.425 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:52:46.425 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:46.425 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:46.425 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:52:46.425 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:46.425 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:46.425 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:46.425 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:52:46.425 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:46.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:46.426 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:46.426 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:52:46.426 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:46.426 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:46.426 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:52:46.426 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:46.426 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:46.426 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:46.426 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:52:46.426 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:46.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:46.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:46.427 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:52:46.427 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:46.427 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:46.427 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:52:46.427 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:46.427 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:52:46.427 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:46.427 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:52:46.428 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:46.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:46.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:46.428 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:52:46.428 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:46.428 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:46.428 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:52:46.428 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:46.428 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:52:46.428 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:46.428 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:52:46.428 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:46.429 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:46.429 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:52:46.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:46.429 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:52:46.429 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:52:46.429 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:52:46.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:46.429 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:52:46Z" 2024-04-12T04:52:46.429 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:52:46.430 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:52:46Z" } 2024-04-12T04:52:46.430 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:52:46.431 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:46.431 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:46.431 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:52:51.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:51.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:51.432 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:52:51.493 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:52:51.493 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:52:51.554 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:52:51.554 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:52:51.554 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:52:51.554 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:52:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:52:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:52:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:52:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:52:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:51.555 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:52:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:52:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:52:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:51.556 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:52:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:52:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:52:51.557 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:52:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:52:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:52:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:51.558 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:52:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:52:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:52:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:52:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:51.559 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:52:51.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:51.560 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:52:51.560 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:52:51.560 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:52:51.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:51.560 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:52:51Z" 2024-04-12T04:52:51.560 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:52:51.561 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:52:51Z" } 2024-04-12T04:52:51.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:52:51.562 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:51.562 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:51.562 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:52:56.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:56.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:52:56.564 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:52:56.626 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:52:56.627 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:52:56.683 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:52:56.683 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:52:56.684 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:52:56.684 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:52:56.684 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:52:56.684 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:52:56.684 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:56.684 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:52:56.684 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:56.684 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:52:56.684 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:56.684 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:56.684 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:52:56.685 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:56.685 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:56.685 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:56.685 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:52:56.685 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:56.685 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:56.685 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:56.685 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:52:56.685 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:56.685 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:56.685 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:52:56.686 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:56.686 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:52:56.686 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:56.686 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:52:56.686 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:56.686 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:56.686 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:56.686 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:52:56.686 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:56.686 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:56.686 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:52:56.687 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:56.687 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:52:56.687 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:56.687 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:52:56.687 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:56.687 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:56.687 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:52:56.687 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:52:56.687 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:52:56.687 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:52:56.687 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:52:56.687 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:52:56.688 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:52:56.688 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:52:56.688 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:52:56.688 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:56.688 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:52:56.688 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:52:56.688 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:56.688 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:52:56.688 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:52:56.688 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:52:56.689 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:52:56.689 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:52:56Z" 2024-04-12T04:52:56.689 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:52:56.690 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:52:56Z" } 2024-04-12T04:52:56.690 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:52:56.690 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:56.690 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:52:56.690 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:53:01.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:01.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:01.692 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:53:01.754 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:01.754 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:01.811 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:53:01.812 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:53:01.812 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:53:01.812 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:53:01.812 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:53:01.812 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:53:01.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:01.812 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:53:01.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:01.812 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:53:01.813 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:01.813 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:01.813 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:53:01.813 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:01.813 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:01.813 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:01.813 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:53:01.813 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:01.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:01.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:01.814 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:53:01.814 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:01.814 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:01.814 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:53:01.814 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:01.814 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:01.814 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:01.814 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:53:01.814 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:01.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:01.815 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:01.815 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:53:01.815 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:01.817 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:01.817 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:53:01.817 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:01.817 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:53:01.818 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:01.818 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:53:01.818 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:01.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:01.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:01.818 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:53:01.818 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:01.818 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:01.818 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:53:01.818 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:01.818 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:53:01.819 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:01.819 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:53:01.819 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:01.819 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:01.819 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:01.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:01.819 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:53:01.819 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:53:01.819 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:53:01.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:01.819 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:53:01Z" 2024-04-12T04:53:01.820 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:53:01.821 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:53:01Z" } 2024-04-12T04:53:01.821 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:53:01.821 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:01.821 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:01.821 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:53:06.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:06.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:06.823 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:53:06.884 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:06.885 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:06.941 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:53:06.941 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:53:06.941 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:53:06.941 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:53:06.941 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:53:06.941 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:53:06.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:06.942 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:53:06.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:06.942 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:53:06.942 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:06.942 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:06.942 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:53:06.943 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:06.943 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:06.943 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:06.943 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:53:06.943 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:06.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:06.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:06.943 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:53:06.943 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:06.943 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:06.943 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:53:06.944 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:06.944 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:06.944 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:06.944 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:53:06.944 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:06.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:06.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:06.944 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:53:06.944 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:06.944 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:06.944 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:53:06.945 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:06.945 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:53:06.945 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:06.945 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:53:06.945 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:06.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:06.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:06.945 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:53:06.945 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:06.945 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:06.945 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:53:06.945 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:06.946 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:53:06.946 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:06.946 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:53:06.946 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:06.946 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:06.946 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:06.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:06.946 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:53:06.946 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:53:06.946 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:53:06.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:06.947 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:53:06Z" 2024-04-12T04:53:06.947 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:53:06.948 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:53:06Z" } 2024-04-12T04:53:06.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:53:06.948 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:06.948 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:06.948 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:53:11.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:11.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:11.950 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:53:12.012 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:12.012 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:12.068 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:53:12.068 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:53:12.068 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:53:12.068 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:53:12.068 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:53:12.068 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:53:12.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:12.069 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:53:12.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:12.069 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:53:12.069 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:12.069 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:12.069 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:53:12.069 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:12.069 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:12.069 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:12.069 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:53:12.070 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:12.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:12.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:12.070 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:53:12.070 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:12.070 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:12.070 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:53:12.070 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:12.070 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:12.070 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:12.070 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:53:12.071 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:12.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:12.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:12.071 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:53:12.071 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:12.071 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:12.071 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:53:12.071 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:12.071 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:53:12.071 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:12.071 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:53:12.071 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:12.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:12.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:12.072 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:53:12.072 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:12.072 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:12.072 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:53:12.072 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:12.072 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:53:12.072 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:12.072 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:53:12.072 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:12.073 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:12.073 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:12.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:12.073 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:53:12.073 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:53:12.073 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:53:12.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:12.073 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:53:12Z" 2024-04-12T04:53:12.073 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:53:12.074 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:53:12Z" } 2024-04-12T04:53:12.074 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:53:12.075 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:12.075 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:12.075 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:53:17.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:17.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:17.076 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:53:17.138 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:17.138 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:17.193 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:53:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:53:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:53:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:53:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:53:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:53:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:53:17.194 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:53:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:53:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:53:17.195 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:53:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:53:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:17.196 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:53:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:53:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:53:17.197 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:53:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:53:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:53:17.198 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:17.199 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:17.199 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:53:17.199 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:17.199 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:53:17.199 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:17.199 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:53:17.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:17.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:17.199 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:17.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:17.200 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:53:17.200 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:53:17.200 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:53:17.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:17.200 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:53:17Z" 2024-04-12T04:53:17.200 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:53:17.203 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:53:17Z" } 2024-04-12T04:53:17.203 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:53:17.203 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:17.203 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:17.204 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:53:22.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:22.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:22.205 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:53:22.267 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:22.268 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:22.324 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:53:22.324 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:53:22.325 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:53:22.325 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:53:22.325 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:53:22.325 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:53:22.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:22.325 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:53:22.325 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:22.325 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:53:22.325 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:22.325 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:22.325 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:53:22.325 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:22.326 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:22.326 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:22.326 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:53:22.326 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:22.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:22.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:22.326 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:53:22.326 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:22.326 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:22.326 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:53:22.326 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:22.327 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:22.327 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:22.327 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:53:22.327 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:22.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:22.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:22.327 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:53:22.327 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:22.327 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:22.327 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:53:22.328 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:22.328 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:53:22.328 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:22.328 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:53:22.328 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:22.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:22.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:22.328 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:53:22.328 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:22.328 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:22.328 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:53:22.328 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:22.329 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:53:22.329 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:22.329 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:53:22.329 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:22.329 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:22.329 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:22.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:22.329 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:53:22.329 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:53:22.329 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:53:22.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:22.330 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:53:22Z" 2024-04-12T04:53:22.330 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:53:22.331 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:53:22Z" } 2024-04-12T04:53:22.331 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:53:22.331 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:22.331 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:22.331 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:53:27.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:27.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:27.332 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:53:27.395 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:27.395 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:28.456 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:53:28.456 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:53:28.456 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:53:28.456 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:53:28.456 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:53:28.457 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:53:28.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:28.457 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:53:28.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:28.457 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:53:28.457 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:28.457 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:28.457 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:53:28.457 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:28.457 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:28.458 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:28.458 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:53:28.458 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:28.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:28.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:28.458 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:53:28.458 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:28.458 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:28.458 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:53:28.458 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:28.459 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:28.459 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:28.459 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:53:28.459 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:28.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:28.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:28.459 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:53:28.459 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:28.459 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:28.459 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:53:28.459 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:28.460 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:53:28.460 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:28.460 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:53:28.460 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:28.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:28.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:28.460 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:53:28.460 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:28.460 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:28.460 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:53:28.460 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:28.461 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:53:28.461 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:28.461 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:53:28.461 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:28.461 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:28.461 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:28.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:28.461 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:53:28.461 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:53:28.461 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:53:28.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:28.461 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:53:28Z" 2024-04-12T04:53:28.462 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:53:28.463 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:53:28Z" } 2024-04-12T04:53:28.463 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:53:28.463 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:28.463 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:28.463 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:53:33.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:33.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:33.465 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:53:33.527 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:33.528 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:33.582 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:53:33.582 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:53:33.582 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:53:33.583 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:53:33.583 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:53:33.583 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:53:33.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:33.583 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:53:33.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:33.583 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:53:33.583 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:33.583 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:33.583 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:53:33.584 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:33.584 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:33.584 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:33.584 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:53:33.584 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:33.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:33.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:33.584 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:53:33.584 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:33.584 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:33.584 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:53:33.584 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:33.585 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:33.585 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:33.585 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:53:33.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:33.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:33.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:33.585 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:53:33.585 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:33.585 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:33.585 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:53:33.585 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:33.586 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:53:33.586 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:33.586 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:53:33.586 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:33.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:33.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:33.586 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:53:33.586 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:33.586 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:33.586 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:53:33.586 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:33.587 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:53:33.587 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:33.587 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:53:33.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:33.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:33.587 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:33.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:33.587 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:53:33.587 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:53:33.587 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:53:33.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:33.587 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:53:33Z" 2024-04-12T04:53:33.588 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:53:33.588 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:53:33Z" } 2024-04-12T04:53:33.589 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:53:33.589 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:33.589 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:33.589 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:53:38.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:38.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:38.590 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:53:38.652 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:38.652 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:38.709 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:53:38.710 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:53:38.710 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:53:38.710 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:53:38.710 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:53:38.710 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:53:38.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:38.710 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:53:38.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:38.711 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:53:38.711 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:38.711 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:38.711 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:53:38.711 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:38.711 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:38.711 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:38.711 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:53:38.711 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:38.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:38.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:38.711 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:53:38.712 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:38.712 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:38.712 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:53:38.712 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:38.712 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:38.712 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:38.712 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:53:38.712 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:38.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:38.713 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:38.713 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:53:38.713 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:38.713 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:38.713 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:53:38.713 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:38.713 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:53:38.713 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:38.713 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:53:38.713 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:38.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:38.714 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:38.714 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:53:38.714 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:38.714 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:38.714 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:53:38.714 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:38.714 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:53:38.714 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:38.714 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:53:38.714 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:38.714 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:38.715 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:38.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:38.715 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:53:38.715 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:53:38.715 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:53:38.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:38.715 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:53:38Z" 2024-04-12T04:53:38.715 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:53:38.716 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:53:38Z" } 2024-04-12T04:53:38.716 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:53:38.716 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:38.717 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:38.717 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:53:43.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:43.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:43.718 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:53:43.780 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:43.780 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:43.836 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:53:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:53:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:53:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:53:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:53:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:53:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:53:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:53:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:53:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:53:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:53:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:53:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:53:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:53:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:53:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:53:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:53:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:53:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:53:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:53:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:53:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:53:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:53:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:53:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:53:43Z" 2024-04-12T04:53:43.841 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:53:43.843 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:53:43Z" } 2024-04-12T04:53:43.843 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:53:43.843 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:43.843 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:43.843 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:53:48.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:48.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:48.845 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:53:48.907 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:48.907 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:48.963 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:53:48.964 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:53:48.964 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:53:48.964 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:53:48.964 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:53:48.964 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:53:48.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:48.965 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:53:48.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:48.965 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:53:48.965 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:48.965 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:48.965 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048199", 2024-04-12T04:53:48.965 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:48.966 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:48.966 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:48.966 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.852893Z" 2024-04-12T04:53:48.966 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:48.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:48.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:48.966 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:53:48.966 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:48.966 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:48.966 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:53:48.966 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:48.967 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:48.967 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:48.967 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:53:48.967 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:48.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:48.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:48.967 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:53:48.967 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:48.967 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:48.967 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055193", 2024-04-12T04:53:48.967 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:48.967 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:53:48.968 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:48.968 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.864914Z" 2024-04-12T04:53:48.968 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:48.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:48.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:48.968 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:53:48.968 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:48.968 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:48.969 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:53:48.969 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:48.969 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:53:48.969 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:48.969 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:53:48.969 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:48.969 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:48.969 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:48.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:48.969 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:53:48.969 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:53:48.970 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:53:48.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:48.970 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:53:48Z" 2024-04-12T04:53:48.970 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:53:48.971 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048199", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.852893Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055193", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:50:03.864914Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:53:48Z" } 2024-04-12T04:53:48.971 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048199', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.852893Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055193', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.864914Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:53:48.971 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:48.971 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048199', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055193', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:48.971 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000048340 target=00000000000000000000:00000000000000048199, shard=2 master=00000000000000000000:00000000000000055334 target=00000000000000000000:00000000000000055193 2024-04-12T04:53:50.207 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T04:53:50.204+0000 7f23e5650700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:53:53.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:53.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:53.973 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:53:54.040 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:54.040 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:54.094 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:53:54.095 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:53:54.095 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:53:54.095 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:53:54.095 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:53:54.095 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:53:54.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.096 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:53:54.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.096 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:53:54.096 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:54.097 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:54.097 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048340", 2024-04-12T04:53:54.097 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:54.097 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:54.097 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:54.098 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:49.229855Z" 2024-04-12T04:53:54.098 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.098 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:53:54.099 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:54.099 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:54.099 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:53:54.099 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:54.099 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:54.100 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:54.100 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:53:54.100 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.100 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:53:54.100 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:54.101 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:54.101 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055334", 2024-04-12T04:53:54.101 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:54.101 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:53:54.101 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:54.101 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:49.195479Z" 2024-04-12T04:53:54.102 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.102 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.102 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:53:54.102 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:54.103 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:53:54.103 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:53:54.103 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:54.103 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:53:54.104 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:54.104 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:53:54.104 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.104 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.104 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.105 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:53:54.105 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:53:54.105 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:53:54.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.105 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:53:54Z" 2024-04-12T04:53:54.105 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:53:54.106 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048340", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:49.229855Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055334", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:51:49.195479Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:53:54Z" } 2024-04-12T04:53:54.106 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048340', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:49.229855Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055334', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:51:49.195479Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:53:54.107 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:54.107 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:53:54.107 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:53:54.107 DEBUG:boto:path=/dkyges-76/ 2024-04-12T04:53:54.107 DEBUG:boto:auth_path=/dkyges-76/ 2024-04-12T04:53:54.108 DEBUG:boto:Method: HEAD 2024-04-12T04:53:54.108 DEBUG:boto:Path: /dkyges-76/ 2024-04-12T04:53:54.108 DEBUG:boto:Data: 2024-04-12T04:53:54.108 DEBUG:boto:Headers: {} 2024-04-12T04:53:54.108 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:53:54.108 DEBUG:boto:Port: 8001 2024-04-12T04:53:54.108 DEBUG:boto:Params: {} 2024-04-12T04:53:54.108 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:53:54.109 DEBUG:boto:Token: None 2024-04-12T04:53:54.109 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:53:54 GMT /dkyges-76/ 2024-04-12T04:53:54.109 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:f5T3djRKvlDaHNZgpJUcQyHotto= 2024-04-12T04:53:54.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:53:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:f5T3djRKvlDaHNZgpJUcQyHotto=', 'Content-Length': '0'} 2024-04-12T04:53:54.143 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006eddca200a8c8736-006618be62-2e73-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:53:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:53:54.143 DEBUG:boto:path=/dkyges-76/obj1 2024-04-12T04:53:54.144 DEBUG:boto:auth_path=/dkyges-76/obj1 2024-04-12T04:53:54.144 DEBUG:boto:Method: HEAD 2024-04-12T04:53:54.145 DEBUG:boto:Path: /dkyges-76/obj1 2024-04-12T04:53:54.145 DEBUG:boto:Data: 2024-04-12T04:53:54.145 DEBUG:boto:Headers: {} 2024-04-12T04:53:54.145 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:53:54.145 DEBUG:boto:Port: 8001 2024-04-12T04:53:54.145 DEBUG:boto:Params: {} 2024-04-12T04:53:54.145 DEBUG:boto:Token: None 2024-04-12T04:53:54.145 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:53:54 GMT /dkyges-76/obj1 2024-04-12T04:53:54.145 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:ui6VS4vBCJStN+7zmrMK7TppMSo= 2024-04-12T04:53:54.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:53:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ui6VS4vBCJStN+7zmrMK7TppMSo=', 'Content-Length': '0'} 2024-04-12T04:53:54.149 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:51:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-76:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.11504.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079e71c3d004325ea-006618be62-2e73-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:53:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:53:54.149 DEBUG:boto:path=/dkyges-76/obj1 2024-04-12T04:53:54.149 DEBUG:boto:auth_path=/dkyges-76/obj1 2024-04-12T04:53:54.149 DEBUG:boto:Method: GET 2024-04-12T04:53:54.149 DEBUG:boto:Path: /dkyges-76/obj1 2024-04-12T04:53:54.149 DEBUG:boto:Data: 2024-04-12T04:53:54.150 DEBUG:boto:Headers: {} 2024-04-12T04:53:54.150 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:53:54.150 DEBUG:boto:Port: 8001 2024-04-12T04:53:54.150 DEBUG:boto:Params: {} 2024-04-12T04:53:54.150 DEBUG:boto:Token: None 2024-04-12T04:53:54.150 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:53:54 GMT /dkyges-76/obj1 2024-04-12T04:53:54.150 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:1wj7ASxz4AmW6UIHk57lmTnIIEY= 2024-04-12T04:53:54.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:53:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1wj7ASxz4AmW6UIHk57lmTnIIEY=', 'Content-Length': '0'} 2024-04-12T04:53:54.154 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:51:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-76:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.11504.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cef72ebcd0619e6a-006618be62-2e73-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:53:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:53:54.154 DEBUG:boto:path=/dkyges-76/obj2 2024-04-12T04:53:54.154 DEBUG:boto:auth_path=/dkyges-76/obj2 2024-04-12T04:53:54.154 DEBUG:boto:Method: HEAD 2024-04-12T04:53:54.154 DEBUG:boto:Path: /dkyges-76/obj2 2024-04-12T04:53:54.155 DEBUG:boto:Data: 2024-04-12T04:53:54.155 DEBUG:boto:Headers: {} 2024-04-12T04:53:54.155 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:53:54.155 DEBUG:boto:Port: 8001 2024-04-12T04:53:54.155 DEBUG:boto:Params: {} 2024-04-12T04:53:54.155 DEBUG:boto:Token: None 2024-04-12T04:53:54.155 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:53:54 GMT /dkyges-76/obj2 2024-04-12T04:53:54.155 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:13H+HJrnTmlQW5g6lQBvkEYYT0Q= 2024-04-12T04:53:54.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:53:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:13H+HJrnTmlQW5g6lQBvkEYYT0Q=', 'Content-Length': '0'} 2024-04-12T04:53:54.158 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:51:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-76:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.11504.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000045597bb3562ba878-006618be62-2e73-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:53:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:53:54.158 DEBUG:boto:path=/dkyges-76/obj2 2024-04-12T04:53:54.158 DEBUG:boto:auth_path=/dkyges-76/obj2 2024-04-12T04:53:54.158 DEBUG:boto:Method: GET 2024-04-12T04:53:54.158 DEBUG:boto:Path: /dkyges-76/obj2 2024-04-12T04:53:54.158 DEBUG:boto:Data: 2024-04-12T04:53:54.158 DEBUG:boto:Headers: {} 2024-04-12T04:53:54.159 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:53:54.159 DEBUG:boto:Port: 8001 2024-04-12T04:53:54.159 DEBUG:boto:Params: {} 2024-04-12T04:53:54.159 DEBUG:boto:Token: None 2024-04-12T04:53:54.159 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:53:54 GMT /dkyges-76/obj2 2024-04-12T04:53:54.159 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:T+gF8Qus2NVztYzcrxVm5vpF85U= 2024-04-12T04:53:54.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:53:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:T+gF8Qus2NVztYzcrxVm5vpF85U=', 'Content-Length': '0'} 2024-04-12T04:53:54.162 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:51:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-76:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.11504.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000712e2c81d8211fea-006618be62-2e73-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:53:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:53:54.206 DEBUG:boto:path=/dkyges-77/ 2024-04-12T04:53:54.207 DEBUG:boto:auth_path=/dkyges-77/ 2024-04-12T04:53:54.207 DEBUG:boto:Method: HEAD 2024-04-12T04:53:54.207 DEBUG:boto:Path: /dkyges-77/ 2024-04-12T04:53:54.207 DEBUG:boto:Data: 2024-04-12T04:53:54.207 DEBUG:boto:Headers: {} 2024-04-12T04:53:54.207 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:53:54.207 DEBUG:boto:Port: 8001 2024-04-12T04:53:54.207 DEBUG:boto:Params: {} 2024-04-12T04:53:54.207 DEBUG:boto:Token: None 2024-04-12T04:53:54.207 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:53:54 GMT /dkyges-77/ 2024-04-12T04:53:54.208 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:QrF+gbKW9oQGPgpDfHI26UTLFnk= 2024-04-12T04:53:54.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:53:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QrF+gbKW9oQGPgpDfHI26UTLFnk=', 'Content-Length': '0'} 2024-04-12T04:53:54.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067cc7b68a443d86b-006618be62-2e73-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:53:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:53:54.211 DEBUG:boto:path=/dkyges-77/obj1 2024-04-12T04:53:54.211 DEBUG:boto:auth_path=/dkyges-77/obj1 2024-04-12T04:53:54.211 DEBUG:boto:Method: HEAD 2024-04-12T04:53:54.212 DEBUG:boto:Path: /dkyges-77/obj1 2024-04-12T04:53:54.212 DEBUG:boto:Data: 2024-04-12T04:53:54.212 DEBUG:boto:Headers: {} 2024-04-12T04:53:54.212 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:53:54.212 DEBUG:boto:Port: 8001 2024-04-12T04:53:54.212 DEBUG:boto:Params: {} 2024-04-12T04:53:54.212 DEBUG:boto:Token: None 2024-04-12T04:53:54.212 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:53:54 GMT /dkyges-77/obj1 2024-04-12T04:53:54.212 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:NU4HGO7cEzibIr9AguhX1CJY5jA= 2024-04-12T04:53:54.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:53:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NU4HGO7cEzibIr9AguhX1CJY5jA=', 'Content-Length': '0'} 2024-04-12T04:53:54.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:51:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-76:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.11504.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad4ff632841bc10f-006618be62-2e73-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:53:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:53:54.215 DEBUG:boto:path=/dkyges-77/obj1 2024-04-12T04:53:54.215 DEBUG:boto:auth_path=/dkyges-77/obj1 2024-04-12T04:53:54.215 DEBUG:boto:Method: GET 2024-04-12T04:53:54.215 DEBUG:boto:Path: /dkyges-77/obj1 2024-04-12T04:53:54.215 DEBUG:boto:Data: 2024-04-12T04:53:54.215 DEBUG:boto:Headers: {} 2024-04-12T04:53:54.215 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:53:54.215 DEBUG:boto:Port: 8001 2024-04-12T04:53:54.215 DEBUG:boto:Params: {} 2024-04-12T04:53:54.215 DEBUG:boto:Token: None 2024-04-12T04:53:54.215 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:53:54 GMT /dkyges-77/obj1 2024-04-12T04:53:54.216 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:df1ax7F2hcwhvECyW3fb1OdAf7E= 2024-04-12T04:53:54.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:53:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:df1ax7F2hcwhvECyW3fb1OdAf7E=', 'Content-Length': '0'} 2024-04-12T04:53:54.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:51:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-76:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.11504.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e7cb88a427c52b4d-006618be62-2e73-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:53:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:53:54.219 DEBUG:boto:path=/dkyges-77/obj2 2024-04-12T04:53:54.220 DEBUG:boto:auth_path=/dkyges-77/obj2 2024-04-12T04:53:54.220 DEBUG:boto:Method: HEAD 2024-04-12T04:53:54.220 DEBUG:boto:Path: /dkyges-77/obj2 2024-04-12T04:53:54.220 DEBUG:boto:Data: 2024-04-12T04:53:54.220 DEBUG:boto:Headers: {} 2024-04-12T04:53:54.220 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:53:54.220 DEBUG:boto:Port: 8001 2024-04-12T04:53:54.220 DEBUG:boto:Params: {} 2024-04-12T04:53:54.220 DEBUG:boto:Token: None 2024-04-12T04:53:54.220 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:53:54 GMT /dkyges-77/obj2 2024-04-12T04:53:54.221 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:6YfnIFPLO/zUy6FUreEFaylO0yc= 2024-04-12T04:53:54.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:53:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6YfnIFPLO/zUy6FUreEFaylO0yc=', 'Content-Length': '0'} 2024-04-12T04:53:54.223 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:51:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-76:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.11504.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000071b60ed9883b5440-006618be62-2e73-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:53:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:53:54.223 DEBUG:boto:path=/dkyges-77/obj2 2024-04-12T04:53:54.223 DEBUG:boto:auth_path=/dkyges-77/obj2 2024-04-12T04:53:54.223 DEBUG:boto:Method: GET 2024-04-12T04:53:54.223 DEBUG:boto:Path: /dkyges-77/obj2 2024-04-12T04:53:54.224 DEBUG:boto:Data: 2024-04-12T04:53:54.224 DEBUG:boto:Headers: {} 2024-04-12T04:53:54.224 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:53:54.224 DEBUG:boto:Port: 8001 2024-04-12T04:53:54.224 DEBUG:boto:Params: {} 2024-04-12T04:53:54.224 DEBUG:boto:Token: None 2024-04-12T04:53:54.224 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:53:54 GMT /dkyges-77/obj2 2024-04-12T04:53:54.224 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:qIe9UtcObrRr48968ERD5soQ7DI= 2024-04-12T04:53:54.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:53:54 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qIe9UtcObrRr48968ERD5soQ7DI=', 'Content-Length': '0'} 2024-04-12T04:53:54.226 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:51:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-76:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.11504.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c42eff50c25731c-006618be62-2e73-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:53:54 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:53:54.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:54.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:54.267 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:53:54.329 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:54.329 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:54.389 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dkyges-76","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dkyges-77","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:53:54.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:54.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:54.390 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe '--source-zones=*' '--dest-zones=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:53:54.451 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:54.451 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:54.487 INFO:teuthology.orchestra.run.smithi043.stderr:WARNING: cannot find source zone id for name=* 2024-04-12T04:53:54.487 INFO:teuthology.orchestra.run.smithi043.stderr:WARNING: cannot find source zone id for name=* 2024-04-12T04:53:54.507 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dkyges-76","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dkyges-77","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:53:54.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:54.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:54.508 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:53:54.562 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:54.563 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:54.619 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dkyges-76","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dkyges-77","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:53:54.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:54.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:54.620 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:53:54.672 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:54.673 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:54.735 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:53:54.735 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:53:54.735 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 23, 2024-04-12T04:53:54.735 INFO:teuthology.orchestra.run.smithi043.stdout: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T04:53:54.735 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": [], 2024-04-12T04:53:54.735 INFO:teuthology.orchestra.run.smithi043.stdout: "period_map": { 2024-04-12T04:53:54.735 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:53:54.735 INFO:teuthology.orchestra.run.smithi043.stdout: "zonegroups": [ 2024-04-12T04:53:54.735 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.735 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "a", 2024-04-12T04:53:54.736 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-12T04:53:54.736 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:53:54.736 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": true, 2024-04-12T04:53:54.736 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:53:54.736 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:53:54.736 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:53:54.736 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:53:54.736 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:53:54.736 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:53:54.736 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:53:54.736 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.737 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:53:54.737 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a2", 2024-04-12T04:53:54.737 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:53:54.737 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T04:53:54.737 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:53:54.737 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:53:54.737 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:53:54.737 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:53:54.737 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:53:54.737 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:53:54.737 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:53:54.738 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:53:54.738 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:53:54.738 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:53:54.738 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:53:54.738 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:53:54.738 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.738 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.738 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:53:54.738 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a1", 2024-04-12T04:53:54.738 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:53:54.738 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:53:54.739 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:53:54.739 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:53:54.739 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:53:54.739 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:53:54.739 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:53:54.739 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:53:54.739 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:53:54.739 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:53:54.739 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:53:54.739 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:53:54.739 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:53:54.740 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:53:54.740 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.740 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.740 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:53:54.740 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:53:54.740 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.740 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:53:54.740 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:53:54.740 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:53:54.740 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:53:54.740 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.741 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.741 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:53:54.741 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:53:54.741 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:53:54.741 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:53:54.741 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [ 2024-04-12T04:53:54.741 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.741 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-group", 2024-04-12T04:53:54.741 INFO:teuthology.orchestra.run.smithi043.stdout: "data_flow": { 2024-04-12T04:53:54.741 INFO:teuthology.orchestra.run.smithi043.stdout: "symmetrical": [ 2024-04-12T04:53:54.741 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.741 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-flow1", 2024-04-12T04:53:54.742 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:53:54.742 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:53:54.742 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:53:54.742 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.742 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.742 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.742 INFO:teuthology.orchestra.run.smithi043.stdout: "pipes": [ 2024-04-12T04:53:54.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.743 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-pipe-dkyges-76", 2024-04-12T04:53:54.743 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:53:54.743 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-76", 2024-04-12T04:53:54.743 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:53:54.743 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:53:54.743 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.743 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": { 2024-04-12T04:53:54.743 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-76", 2024-04-12T04:53:54.743 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:53:54.743 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2" 2024-04-12T04:53:54.744 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.744 INFO:teuthology.orchestra.run.smithi043.stdout: "params": { 2024-04-12T04:53:54.744 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:53:54.744 INFO:teuthology.orchestra.run.smithi043.stdout: "filter": { 2024-04-12T04:53:54.744 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [] 2024-04-12T04:53:54.744 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.745 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": {}, 2024-04-12T04:53:54.745 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-12T04:53:54.745 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "system", 2024-04-12T04:53:54.745 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "" 2024-04-12T04:53:54.745 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.745 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-pipe-dkyges-77", 2024-04-12T04:53:54.745 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:53:54.746 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-76", 2024-04-12T04:53:54.746 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:53:54.746 INFO:teuthology.orchestra.run.smithi043.stdout: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:53:54.746 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.746 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": { 2024-04-12T04:53:54.746 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "dkyges-77", 2024-04-12T04:53:54.746 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:53:54.746 INFO:teuthology.orchestra.run.smithi043.stdout: "1ba046e0-d0ae-437d-b741-28b4299423c2" 2024-04-12T04:53:54.746 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.747 INFO:teuthology.orchestra.run.smithi043.stdout: "params": { 2024-04-12T04:53:54.747 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:53:54.747 INFO:teuthology.orchestra.run.smithi043.stdout: "filter": { 2024-04-12T04:53:54.747 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [] 2024-04-12T04:53:54.747 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.747 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": {}, 2024-04-12T04:53:54.747 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-12T04:53:54.747 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "system", 2024-04-12T04:53:54.747 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "" 2024-04-12T04:53:54.747 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.748 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.748 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "sync-pipe", 2024-04-12T04:53:54.748 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:53:54.748 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "*", 2024-04-12T04:53:54.748 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:53:54.748 INFO:teuthology.orchestra.run.smithi043.stdout: "*" 2024-04-12T04:53:54.748 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.748 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": { 2024-04-12T04:53:54.748 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket": "*", 2024-04-12T04:53:54.748 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:53:54.748 INFO:teuthology.orchestra.run.smithi043.stdout: "*" 2024-04-12T04:53:54.749 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.749 INFO:teuthology.orchestra.run.smithi043.stdout: "params": { 2024-04-12T04:53:54.749 INFO:teuthology.orchestra.run.smithi043.stdout: "source": { 2024-04-12T04:53:54.749 INFO:teuthology.orchestra.run.smithi043.stdout: "filter": { 2024-04-12T04:53:54.749 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [] 2024-04-12T04:53:54.749 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.749 INFO:teuthology.orchestra.run.smithi043.stdout: "dest": {}, 2024-04-12T04:53:54.749 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-12T04:53:54.749 INFO:teuthology.orchestra.run.smithi043.stdout: "mode": "system", 2024-04-12T04:53:54.749 INFO:teuthology.orchestra.run.smithi043.stdout: "user": "" 2024-04-12T04:53:54.750 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.750 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.750 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:53:54.750 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "allowed" 2024-04-12T04:53:54.750 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.750 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.750 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:53:54.750 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:53:54.750 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.751 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "b", 2024-04-12T04:53:54.751 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-12T04:53:54.751 INFO:teuthology.orchestra.run.smithi043.stdout: "api_name": "", 2024-04-12T04:53:54.751 INFO:teuthology.orchestra.run.smithi043.stdout: "is_master": false, 2024-04-12T04:53:54.751 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:53:54.751 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:53:54.751 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:53:54.751 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames": [], 2024-04-12T04:53:54.751 INFO:teuthology.orchestra.run.smithi043.stdout: "hostnames_s3website": [], 2024-04-12T04:53:54.751 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:53:54.752 INFO:teuthology.orchestra.run.smithi043.stdout: "zones": [ 2024-04-12T04:53:54.752 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.752 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:53:54.753 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b1", 2024-04-12T04:53:54.753 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:53:54.753 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:53:54.753 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:53:54.753 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:53:54.753 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:53:54.753 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:53:54.753 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:53:54.754 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:53:54.754 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:53:54.754 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:53:54.754 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:53:54.754 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:53:54.754 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:53:54.754 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:53:54.754 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.754 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.755 INFO:teuthology.orchestra.run.smithi043.stdout: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:53:54.755 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b2", 2024-04-12T04:53:54.755 INFO:teuthology.orchestra.run.smithi043.stdout: "endpoints": [ 2024-04-12T04:53:54.755 INFO:teuthology.orchestra.run.smithi043.stdout: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T04:53:54.755 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:53:54.755 INFO:teuthology.orchestra.run.smithi043.stdout: "log_meta": false, 2024-04-12T04:53:54.755 INFO:teuthology.orchestra.run.smithi043.stdout: "log_data": true, 2024-04-12T04:53:54.755 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_index_max_shards": 11, 2024-04-12T04:53:54.755 INFO:teuthology.orchestra.run.smithi043.stdout: "read_only": false, 2024-04-12T04:53:54.755 INFO:teuthology.orchestra.run.smithi043.stdout: "tier_type": "", 2024-04-12T04:53:54.755 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from_all": true, 2024-04-12T04:53:54.755 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_from": [], 2024-04-12T04:53:54.756 INFO:teuthology.orchestra.run.smithi043.stdout: "redirect_zone": "", 2024-04-12T04:53:54.756 INFO:teuthology.orchestra.run.smithi043.stdout: "supported_features": [ 2024-04-12T04:53:54.756 INFO:teuthology.orchestra.run.smithi043.stdout: "compress-encrypted", 2024-04-12T04:53:54.756 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:53:54.756 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.756 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.756 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:53:54.756 INFO:teuthology.orchestra.run.smithi043.stdout: "placement_targets": [ 2024-04-12T04:53:54.756 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.756 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "default-placement", 2024-04-12T04:53:54.756 INFO:teuthology.orchestra.run.smithi043.stdout: "tags": [], 2024-04-12T04:53:54.756 INFO:teuthology.orchestra.run.smithi043.stdout: "storage_classes": [ 2024-04-12T04:53:54.757 INFO:teuthology.orchestra.run.smithi043.stdout: "STANDARD" 2024-04-12T04:53:54.757 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.757 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.757 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:53:54.757 INFO:teuthology.orchestra.run.smithi043.stdout: "default_placement": "default-placement", 2024-04-12T04:53:54.757 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:53:54.757 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_policy": { 2024-04-12T04:53:54.757 INFO:teuthology.orchestra.run.smithi043.stdout: "groups": [] 2024-04-12T04:53:54.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.757 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled_features": [ 2024-04-12T04:53:54.757 INFO:teuthology.orchestra.run.smithi043.stdout: "resharding" 2024-04-12T04:53:54.758 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.758 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.758 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-12T04:53:54.758 INFO:teuthology.orchestra.run.smithi043.stdout: "short_zone_ids": [ 2024-04-12T04:53:54.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.758 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:53:54.758 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 755891344 2024-04-12T04:53:54.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.759 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:53:54.759 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 4153048242 2024-04-12T04:53:54.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.759 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:53:54.759 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 2639555996 2024-04-12T04:53:54.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:54.759 INFO:teuthology.orchestra.run.smithi043.stdout: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:53:54.759 INFO:teuthology.orchestra.run.smithi043.stdout: "val": 400045239 2024-04-12T04:53:54.759 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.760 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:54.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.760 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zonegroup": "a", 2024-04-12T04:53:54.760 INFO:teuthology.orchestra.run.smithi043.stdout: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:53:54.760 INFO:teuthology.orchestra.run.smithi043.stdout: "period_config": { 2024-04-12T04:53:54.760 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_quota": { 2024-04-12T04:53:54.760 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:53:54.760 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:53:54.760 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:53:54.760 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:53:54.760 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:53:54.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.761 INFO:teuthology.orchestra.run.smithi043.stdout: "user_quota": { 2024-04-12T04:53:54.761 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false, 2024-04-12T04:53:54.761 INFO:teuthology.orchestra.run.smithi043.stdout: "check_on_raw": false, 2024-04-12T04:53:54.761 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size": -1, 2024-04-12T04:53:54.761 INFO:teuthology.orchestra.run.smithi043.stdout: "max_size_kb": 0, 2024-04-12T04:53:54.761 INFO:teuthology.orchestra.run.smithi043.stdout: "max_objects": -1 2024-04-12T04:53:54.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.761 INFO:teuthology.orchestra.run.smithi043.stdout: "user_ratelimit": { 2024-04-12T04:53:54.761 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:53:54.761 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:53:54.761 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:53:54.762 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:53:54.762 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:53:54.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.762 INFO:teuthology.orchestra.run.smithi043.stdout: "bucket_ratelimit": { 2024-04-12T04:53:54.762 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:53:54.762 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:53:54.762 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:53:54.762 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:53:54.762 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:53:54.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.762 INFO:teuthology.orchestra.run.smithi043.stdout: "anonymous_ratelimit": { 2024-04-12T04:53:54.763 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_ops": 0, 2024-04-12T04:53:54.763 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_ops": 0, 2024-04-12T04:53:54.763 INFO:teuthology.orchestra.run.smithi043.stdout: "max_read_bytes": 0, 2024-04-12T04:53:54.763 INFO:teuthology.orchestra.run.smithi043.stdout: "max_write_bytes": 0, 2024-04-12T04:53:54.763 INFO:teuthology.orchestra.run.smithi043.stdout: "enabled": false 2024-04-12T04:53:54.763 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:54.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:54.763 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:53:54.763 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_name": "test-realm", 2024-04-12T04:53:54.763 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:53:54.763 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:53:54.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:54.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:54.764 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:53:54.817 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:54.818 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:54.905 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dkyges-76","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dkyges-77","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:53:54.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dkyges-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:54.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'dkyges-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:54.907 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket dkyges-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:53:54.980 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:54.980 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:55.050 INFO:tasks.rgw.c1.client.0.smithi043.stdout:2024-04-12T04:53:55.047+0000 7f1e272bd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:53:55.215 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:53:55.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dkyges-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:55.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'dkyges-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:55.216 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket dkyges-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:53:55.323 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:55.323 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:55.464 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:53:55.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dkyges-76', '--source-zones=a1', '--dest-zones=a2', '--bucket=dkyges-76', '--source-bucket=*', '--dest-bucket=dkyges-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:55.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dkyges-76', '--source-zones=a1', '--dest-zones=a2', '--bucket=dkyges-76', '--source-bucket=*', '--dest-bucket=dkyges-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:55.465 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dkyges-76 --source-zones=a1 --dest-zones=a2 --bucket=dkyges-76 '--source-bucket=*' --dest-bucket=dkyges-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:53:55.545 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:55.545 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:55.645 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dkyges-76","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"dkyges-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-12T04:53:55.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dkyges-77', '--source-zones=a1', '--dest-zones=a2', '--bucket=dkyges-76', '--source-bucket=*', '--dest-bucket=dkyges-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:55.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-dkyges-77', '--source-zones=a1', '--dest-zones=a2', '--bucket=dkyges-76', '--source-bucket=*', '--dest-bucket=dkyges-77', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:55.646 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-dkyges-77 --source-zones=a1 --dest-zones=a2 --bucket=dkyges-76 '--source-bucket=*' --dest-bucket=dkyges-77 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:53:55.724 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:55.725 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:55.798 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dkyges-76","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"dkyges-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dkyges-77","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"dkyges-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2024-04-12T04:53:55.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dkyges-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:55.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'dkyges-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:55.799 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket dkyges-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:55.917 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dkyges-76","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"dkyges-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-dkyges-77","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"dkyges-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2024-04-12T04:53:55.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:55.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:53:55.917 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:53:55.969 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:55.969 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:56.025 INFO:teuthology.orchestra.run.smithi043.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-dkyges-76","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-76","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-dkyges-77","source":{"bucket":"dkyges-76","zones":["a1"]},"dest":{"bucket":"dkyges-77","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:53:56.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:56.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:56.026 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:53:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:56.117 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:53:56.133 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:53:56.134 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:56.134 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:53:56.134 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:53:56.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:56.134 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:56.134 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:53:56.134 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:53:56.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:56.134 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:56.134 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:53:56.135 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:53:56.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:56.135 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:56.135 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897635.897290_1048.1", 2024-04-12T04:53:56.135 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:53:55.897290Z" 2024-04-12T04:53:56.135 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:56.135 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:53:56.136 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:53:56.136 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:53:56.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:56.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:53:56.137 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:53:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:53:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:53:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:53:56.224+0000 7fa0c1346780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:53:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:53:56.225+0000 7fa0c1346780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:53:56Z" 2024-04-12T04:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:53:56.250 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:53:56Z" } 2024-04-12T04:53:56.251 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:53:56.251 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:53:56.251 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:53:56.251 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:54:01.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:01.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:01.253 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:54:01.314 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:54:01.314 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:54:01.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:01.351+0000 7fa9cf636780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:01.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:01.353+0000 7fa9cf636780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:01.372 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:54:01.372 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:54:01.372 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:54:01.372 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:54:01.372 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:54:01.372 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:54:01.372 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:01.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:01.373 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:54:01.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:01.373 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:54:01.373 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:01.373 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:01.373 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:54:01.373 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:01.373 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:54:01.373 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:01.373 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:54:01.374 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:01.374 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:01.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:01.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:01.374 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:54:01.374 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:01.374 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:01.374 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:54:01.374 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:01.374 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:01.375 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:01.375 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:54:01.375 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:01.375 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:01.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:01.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:01.375 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:54:01.375 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:01.375 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:01.375 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:54:01.375 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:01.375 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:54:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:54:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:54:01.376 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:54:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:54:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:54:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:54:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:54:01.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:01.378 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:54:01Z" 2024-04-12T04:54:01.378 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:54:01.379 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:54:01Z" } 2024-04-12T04:54:01.379 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:54:01.379 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:54:01.379 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:54:01.379 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:54:06.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:06.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:06.381 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:54:06.441 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:54:06.442 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:54:06.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:06.477+0000 7fb348d2f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:06.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:06.478+0000 7fb348d2f780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:06.497 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:54:06.497 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:54:06.497 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:54:06.498 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:54:06.498 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:54:06.498 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:54:06.498 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:06.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:06.498 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:54:06.498 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:06.498 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:54:06.498 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:06.498 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:06.498 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:54:06.499 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:06.499 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:54:06.499 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:06.499 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:54:06.499 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:06.499 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:06.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:06.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:06.499 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:54:06.499 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:06.499 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:06.499 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:54:06.500 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:06.500 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:06.500 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:06.500 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:54:06.500 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:06.500 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:06.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:06.500 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:06.500 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:54:06.500 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:06.500 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:06.501 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:54:06.501 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:06.501 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:06.501 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:06.501 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:54:06.501 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:06.501 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:06.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:06.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:06.501 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:54:06.501 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:06.501 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:06.502 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:54:06.502 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:06.502 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:06.502 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:06.502 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:54:06.502 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:06.502 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:06.502 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:06.502 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:54:06.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:06.502 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:54:06.503 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:54:06.503 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:54:06.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:06.503 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:54:06Z" 2024-04-12T04:54:06.503 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:54:06.504 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:54:06Z" } 2024-04-12T04:54:06.504 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:54:06.504 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:54:06.504 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:54:06.504 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:54:11.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:11.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:11.506 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:54:11.566 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:54:11.567 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:54:11.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:11.602+0000 7f9b60a9a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:11.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:11.603+0000 7f9b60a9a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:11.623 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:54:11.623 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:54:11.623 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:54:11.623 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:54:11.623 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:54:11.623 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:54:11.623 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:11.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:11.624 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:54:11.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:11.624 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:54:11.624 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:11.624 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:11.624 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:54:11.624 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:11.624 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:54:11.624 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:11.624 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:54:11.624 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:11.624 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:11.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:11.625 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:11.625 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:54:11.625 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:11.625 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:11.625 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:54:11.625 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:11.625 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:11.625 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:11.625 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:54:11.625 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:11.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:11.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:11.626 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:11.626 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:54:11.626 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:11.626 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:11.626 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:54:11.626 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:11.626 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:11.626 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:11.626 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:54:11.626 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:11.627 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:11.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:11.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:11.627 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:54:11.627 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:11.627 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:11.627 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:54:11.627 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:11.627 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:11.627 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:11.627 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:54:11.628 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:11.628 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:11.628 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:11.628 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:54:11.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:11.628 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:54:11.628 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:54:11.628 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:54:11.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:11.628 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:54:11Z" 2024-04-12T04:54:11.629 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:54:11.629 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:54:11Z" } 2024-04-12T04:54:11.630 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:54:11.630 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:54:11.630 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:54:11.630 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:54:16.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:16.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:16.632 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:54:16.692 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:54:16.692 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:54:16.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:16.727+0000 7f30be1dc780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:16.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:16.728+0000 7f30be1dc780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:16.746 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:54:16.747 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:54:16.747 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:54:16.747 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:54:16.747 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:54:16.747 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:54:16.747 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:16.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:16.747 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:54:16.747 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:16.747 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:54:16.747 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:16.747 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:16.748 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:54:16.748 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:16.748 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:54:16.748 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:16.748 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:54:16.748 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:16.748 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:16.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:16.748 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:16.748 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:54:16.748 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:16.748 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:16.749 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:54:16.749 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:16.749 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:16.749 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:16.749 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:54:16.749 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:16.749 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:16.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:16.749 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:16.749 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:54:16.749 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:16.749 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:16.750 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:54:16.750 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:16.750 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:16.750 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:16.750 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:54:16.750 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:16.750 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:16.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:16.750 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:16.750 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:54:16.750 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:16.750 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:16.751 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:54:16.751 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:16.751 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:16.751 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:16.751 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:54:16.751 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:16.751 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:16.751 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:16.751 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:54:16.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:16.751 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:54:16.751 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:54:16.752 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:54:16.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:16.752 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:54:16Z" 2024-04-12T04:54:16.752 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:54:16.753 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:54:16Z" } 2024-04-12T04:54:16.753 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:54:16.753 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:54:16.753 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:54:16.753 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:54:21.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:21.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:21.755 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:54:21.815 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:54:21.815 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:54:21.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:21.851+0000 7f10393e2780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:21.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:21.852+0000 7f10393e2780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:21.872 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:54:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:54:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:54:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:54:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:54:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:54:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:21.872 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:54:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:54:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:54:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:54:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:54:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:21.873 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:54:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:54:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:54:21.874 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:54:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:54:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:21.875 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:54:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:54:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:54:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:21.876 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:54:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:54:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:54:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:54:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:54:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:54:21Z" 2024-04-12T04:54:21.878 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:54:21.879 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:54:21Z" } 2024-04-12T04:54:21.879 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:54:21.879 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:54:21.879 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:54:21.879 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:54:26.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:26.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:26.881 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:54:26.940 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:54:26.940 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:54:26.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:26.976+0000 7fb3d55a8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:26.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:26.977+0000 7fb3d55a8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:26.995 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:54:26.995 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:54:26.995 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:54:26.995 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:54:26.995 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:54:26.995 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:54:26.995 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:26.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:26.996 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:54:26.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:26.996 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:54:26.996 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:26.996 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:26.996 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:54:26.996 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:26.996 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:54:26.996 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:26.996 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:54:26.996 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:26.996 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:54:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:54:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:54:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:54:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:54:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:54:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:54:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:54:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:54:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:54:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:54:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:54:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:54:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:54:26Z" 2024-04-12T04:54:27.000 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:54:27.001 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:54:26Z" } 2024-04-12T04:54:27.001 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:54:27.002 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:54:27.002 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:54:27.002 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:54:32.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:32.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:32.005 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:54:32.064 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:54:32.065 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:54:32.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:32.099+0000 7feecc4fc780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:32.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:32.100+0000 7feecc4fc780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:32.118 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:54:32.119 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:54:32.119 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:54:32.119 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:54:32.119 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:54:32.119 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:54:32.119 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:32.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:32.119 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:54:32.119 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:32.119 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:54:32.120 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:32.120 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:32.120 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:54:32.120 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:32.120 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:54:32.120 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:32.120 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:54:32.120 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:32.120 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:32.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:32.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:32.121 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:54:32.121 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:32.121 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:32.121 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:54:32.121 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:32.121 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:32.121 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:32.121 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:54:32.121 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:32.121 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:32.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:32.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:32.122 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:54:32.122 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:32.122 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:32.122 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:54:32.122 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:32.122 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:32.122 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:32.122 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:54:32.122 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:32.122 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:32.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:32.123 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:32.123 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:54:32.123 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:32.123 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:32.123 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:54:32.123 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:32.123 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:32.123 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:32.123 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:54:32.123 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:32.123 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:32.124 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:32.124 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:54:32.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:32.124 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:54:32.124 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:54:32.124 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:54:32.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:32.124 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:54:32Z" 2024-04-12T04:54:32.124 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:54:32.125 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:54:32Z" } 2024-04-12T04:54:32.125 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:54:32.126 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:54:32.126 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:54:32.126 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:54:37.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:37.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:37.130 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:54:37.190 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:54:37.190 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:54:37.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:37.226+0000 7f389913e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:37.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:37.228+0000 7f389913e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:37.248 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:54:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:54:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:54:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:54:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:54:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:54:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:54:37.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:54:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:54:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:54:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:54:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:37.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:54:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:54:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:54:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:37.250 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:54:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:54:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:54:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:37.251 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:54:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:54:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:54:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:37.252 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:54:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:54:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:54:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:54:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:37.253 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:54:37Z" 2024-04-12T04:54:37.253 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:54:37.254 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:54:37Z" } 2024-04-12T04:54:37.254 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:54:37.255 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:54:37.255 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:54:37.255 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:54:42.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:42.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:42.256 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:54:42.315 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:54:42.315 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:54:42.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:42.349+0000 7f66724ee780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:42.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:42.350+0000 7f66724ee780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:42.369 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:54:42.369 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:54:42.369 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:54:42.369 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:54:42.370 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:54:42.370 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:54:42.370 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:42.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:42.370 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:54:42.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:42.370 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:54:42.370 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:42.371 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:42.371 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:54:42.371 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:42.371 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:54:42.371 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:42.371 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:54:42.371 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:42.371 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:42.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:42.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:42.372 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:54:42.372 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:42.372 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:42.372 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:54:42.372 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:42.372 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:42.372 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:42.372 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:54:42.373 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:42.373 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:42.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:42.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:42.373 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:54:42.373 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:42.373 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:42.373 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:54:42.373 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:42.373 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:42.374 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:42.374 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:54:42.374 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:42.374 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:42.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:42.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:42.374 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:54:42.374 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:42.374 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:42.375 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:54:42.375 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:42.375 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:42.375 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:42.375 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:54:42.375 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:42.375 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:42.375 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:42.375 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:54:42.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:42.376 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:54:42.376 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:54:42.376 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:54:42.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:42.376 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:54:42Z" 2024-04-12T04:54:42.376 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:54:42.377 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:54:42Z" } 2024-04-12T04:54:42.377 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:54:42.378 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:54:42.378 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:54:42.378 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:54:47.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:47.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:47.380 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:54:47.440 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:54:47.440 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:54:47.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:47.475+0000 7f50a7455780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:47.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:47.477+0000 7f50a7455780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:47.496 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:54:47.497 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:54:47.497 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:54:47.497 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:54:47.497 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:54:47.497 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:54:47.497 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:47.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:47.497 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:54:47.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:47.497 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:54:47.498 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:47.498 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:47.498 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:54:47.498 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:47.498 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:54:47.498 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:47.498 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:54:47.498 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:47.498 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:47.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:47.498 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:47.499 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:54:47.499 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:47.499 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:47.499 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:54:47.499 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:47.499 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:47.499 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:47.499 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:54:47.499 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:47.500 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:47.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:47.500 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:47.500 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:54:47.500 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:47.500 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:47.500 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:54:47.500 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:47.500 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:47.500 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:47.500 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:54:47.500 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:47.501 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:47.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:47.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:47.501 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:54:47.501 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:47.501 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:47.501 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:54:47.501 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:47.501 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:47.501 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:47.501 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:54:47.501 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:47.502 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:47.502 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:47.502 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:54:47.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:47.502 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:54:47.502 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:54:47.502 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:54:47.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:47.502 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:54:47Z" 2024-04-12T04:54:47.502 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:54:47.503 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:54:47Z" } 2024-04-12T04:54:47.504 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:54:47.504 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:54:47.504 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:54:47.504 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:54:52.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:52.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:52.505 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:54:52.565 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:54:52.565 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:54:52.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:52.600+0000 7f5ffc845780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:52.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:52.601+0000 7f5ffc845780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:52.620 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:54:52.620 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:54:52.620 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:54:52.620 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:54:52.620 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:54:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:54:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:54:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:54:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:54:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:52.621 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:54:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:54:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:54:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:54:52.622 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:54:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:54:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:52.623 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:54:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:54:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:54:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:54:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:54:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:54:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:54:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:54:52.626 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:54:52.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:52.626 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:54:52Z" 2024-04-12T04:54:52.626 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:54:52.627 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:54:52Z" } 2024-04-12T04:54:52.627 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:54:52.627 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:54:52.627 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:54:52.627 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:54:57.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:57.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:54:57.629 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:54:57.688 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:54:57.688 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:54:57.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:57.722+0000 7ff599c8b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:57.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:54:57.723+0000 7ff599c8b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:54:57.742 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:54:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:54:57.742 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:54:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:54:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:54:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:54:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:54:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:54:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:57.743 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:54:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:54:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:54:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:54:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:57.744 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:54:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:54:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:54:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:57.745 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:54:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:54:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:54:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:54:57.746 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:54:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:54:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:54:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:54:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:54:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:54:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:54:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:54:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:54:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:54:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:57.747 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:54:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:54:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:54:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:54:57.748 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:54:57Z" 2024-04-12T04:54:57.748 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:54:57.749 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:54:57Z" } 2024-04-12T04:54:57.749 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:54:57.749 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:54:57.749 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:54:57.749 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:55:02.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:02.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:02.751 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:55:02.811 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:02.811 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:02.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:55:02.846+0000 7f4995f8c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:55:02.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:55:02.847+0000 7f4995f8c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:55:02.866 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:55:02.866 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:55:02.866 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:55:02.866 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:55:02.866 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:55:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:55:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:55:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:55:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:55:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:02.867 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:55:02.868 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:02.868 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:55:02.868 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:02.868 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:02.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:02.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:02.868 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:55:02.868 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:02.868 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:02.868 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:55:02.869 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:02.869 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:02.869 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:02.869 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:55:02.869 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:02.869 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:02.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:02.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:02.869 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:55:02.869 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:02.869 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:02.869 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:55:02.870 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:02.870 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:02.870 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:02.870 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:55:02.870 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:02.870 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:02.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:02.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:02.870 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:55:02.870 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:02.871 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:02.871 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:55:02.871 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:02.871 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:02.871 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:02.871 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:55:02.871 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:02.871 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:02.871 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:02.871 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:55:02.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:02.871 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:55:02.872 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:55:02.872 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:55:02.872 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:02.872 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:55:02Z" 2024-04-12T04:55:02.872 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:55:02.873 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:55:02Z" } 2024-04-12T04:55:02.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:55:02.873 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:55:02.873 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:55:02.873 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:55:07.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:07.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:07.875 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:55:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:07.935 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:07.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:55:07.969+0000 7f3381e86780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:55:07.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:55:07.970+0000 7f3381e86780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:55:07.989 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:55:07.989 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:55:07.989 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:55:07.989 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:55:07.989 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:55:07.989 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:55:07.989 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:07.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:07.990 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:55:07.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:07.990 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:55:07.990 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:07.990 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:07.990 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:55:07.990 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:07.990 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:55:07.990 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:07.991 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:55:07.991 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:07.991 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:07.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:07.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:07.991 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:55:07.991 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:07.991 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:07.991 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:55:07.991 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:07.992 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:07.992 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:07.992 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:55:07.992 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:07.992 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:07.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:07.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:07.992 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:55:07.992 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:07.992 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:07.992 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:55:07.993 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:07.993 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:07.993 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:07.993 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:55:07.993 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:07.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:07.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:07.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:07.993 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:55:07.993 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:07.993 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:07.994 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:55:07.994 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:07.994 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:07.994 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:07.994 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:55:07.994 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:07.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:07.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:07.994 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:55:07.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:07.995 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:55:07.995 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:55:07.995 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:55:07.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:07.995 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:55:07Z" 2024-04-12T04:55:07.995 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:55:07.996 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:55:07Z" } 2024-04-12T04:55:07.996 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:55:07.996 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:55:07.996 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:55:07.997 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:55:12.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:12.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:12.998 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:55:13.058 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:13.058 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:13.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:55:13.093+0000 7f1f558c6780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:55:13.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:55:13.094+0000 7f1f558c6780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:55:13.113 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:55:13.113 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:55:13.113 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:55:13.113 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:55:13.113 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:55:13.113 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:55:13.113 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:13.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:13.114 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:55:13.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:13.114 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:55:13.114 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:13.114 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:13.114 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:55:13.114 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:13.114 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:55:13.114 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:13.115 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:55:13.115 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:13.115 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:13.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:13.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:13.115 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:55:13.115 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:13.115 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:13.115 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:55:13.115 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:13.115 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:13.116 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:13.116 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:55:13.116 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:13.116 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:13.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:13.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:13.116 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:55:13.116 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:13.116 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:13.116 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:55:13.116 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:13.117 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:13.117 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:13.117 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:55:13.117 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:13.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:13.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:13.117 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:13.117 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:55:13.117 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:13.117 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:13.117 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:55:13.118 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:13.118 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:13.118 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:13.118 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:55:13.118 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:13.118 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:13.118 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:13.118 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:55:13.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:13.118 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:55:13.119 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:55:13.119 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:55:13.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:13.119 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:55:13Z" 2024-04-12T04:55:13.119 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:55:13.120 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:55:13Z" } 2024-04-12T04:55:13.120 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:55:13.120 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:55:13.120 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:55:13.120 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:55:18.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:18.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:18.122 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:55:18.182 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:18.182 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:18.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:55:18.215+0000 7f36cade8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:55:18.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:55:18.216+0000 7f36cade8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:55:18.235 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:55:18.235 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:55:18.235 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:55:18.235 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:55:18.235 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:55:18.235 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:55:18.235 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:18.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:18.236 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:55:18.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:18.236 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:55:18.236 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:18.236 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:18.236 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:55:18.236 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:18.236 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:55:18.236 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:18.236 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:55:18.236 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:18.237 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:18.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:18.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:18.237 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:55:18.237 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:18.237 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:18.237 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:55:18.237 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:18.237 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:18.237 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:18.237 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:55:18.237 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:18.238 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:18.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:18.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:18.238 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:55:18.238 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:18.238 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:18.238 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:55:18.238 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:18.238 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:18.238 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:18.239 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:55:18.239 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:18.239 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:18.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:18.239 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:18.239 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:55:18.239 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:18.239 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:18.239 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:55:18.239 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:18.239 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:18.239 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:18.240 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:55:18.240 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:18.240 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:18.240 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:18.240 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:55:18.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:18.240 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:55:18.240 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:55:18.240 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:55:18.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:18.240 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:55:18Z" 2024-04-12T04:55:18.240 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:55:18.241 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:55:18Z" } 2024-04-12T04:55:18.241 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:55:18.242 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:55:18.242 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:55:18.242 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:55:23.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:23.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:23.243 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:55:23.303 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:23.303 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:55:23.341+0000 7f61a6cbb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:55:23.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:55:23.345+0000 7f61a6cbb780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:55:23.364 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:55:23.365 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:55:23.365 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:55:23.365 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:55:23.365 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:55:23.365 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:55:23.365 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:23.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:23.365 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:55:23.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:23.365 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:55:23.366 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:23.366 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:23.366 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:55:23.366 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:23.366 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:55:23.366 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:23.366 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:55:23.366 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:23.366 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:23.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:23.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:23.366 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:55:23.367 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:23.367 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:23.367 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:55:23.367 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:23.367 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:23.367 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:23.367 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:55:23.367 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:23.367 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:23.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:23.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:23.367 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:55:23.368 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:23.368 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:23.368 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:55:23.368 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:23.368 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:23.368 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:23.368 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:55:23.368 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:23.368 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:23.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:23.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:23.368 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:55:23.368 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:23.369 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:23.369 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897507.680289_1020.1", 2024-04-12T04:55:23.369 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:23.369 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:23.369 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:23.369 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:47.680289Z", 2024-04-12T04:55:23.369 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:23.369 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:23.369 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:23.369 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:55:23.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:23.369 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:55:23.369 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:55:23.370 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:55:23.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:23.370 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:55:23Z" 2024-04-12T04:55:23.370 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:55:23.371 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897507.680289_1020.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:47.680289Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:55:23Z" } 2024-04-12T04:55:23.371 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897507.680289_1020.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:47.680289Z', 'realm_epoch': 2}}] 2024-04-12T04:55:23.371 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:55:23.371 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897507.680289_1020.1'} 2024-04-12T04:55:23.371 WARNING:rgw_multi.tests:zone a2 behind master: shard=3 master=1_1712897635.897290_1048.1 target=1_1712897507.680289_1020.1 2024-04-12T04:55:28.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:28.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:28.373 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:55:28.433 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:28.434 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:28.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:55:28.469+0000 7f3bfa1b0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:55:28.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:55:28.470+0000 7f3bfa1b0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:55:28.489 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:55:28.489 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:55:28.489 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:55:28.489 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:55:28.489 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:55:28.489 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:55:28.489 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:28.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.490 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:55:28.490 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:28.490 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:55:28.490 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:28.490 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:28.490 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:55:28.490 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:28.490 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:55:28.490 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:28.490 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:55:28.490 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:28.491 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:28.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.491 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:28.491 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:55:28.491 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:28.491 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:28.491 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:55:28.491 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:28.491 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:28.491 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:28.491 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:55:28.492 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:28.492 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:28.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:28.492 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:55:28.492 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:28.492 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:28.492 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:55:28.492 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:28.492 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:28.492 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:28.492 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:55:28.493 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:28.493 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:28.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:28.493 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:55:28.493 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:28.493 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:28.493 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897635.897290_1048.1", 2024-04-12T04:55:28.493 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:28.493 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:28.493 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:28.493 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:53:55.897290Z", 2024-04-12T04:55:28.494 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:28.494 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:28.494 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:28.494 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:55:28.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.494 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:55:28.494 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:55:28.494 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:55:28.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.494 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:55:28Z" 2024-04-12T04:55:28.494 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:55:28.495 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897635.897290_1048.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:53:55.897290Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:55:28Z" } 2024-04-12T04:55:28.496 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897635.897290_1048.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:53:55.897290Z', 'realm_epoch': 2}}] 2024-04-12T04:55:28.496 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:55:28.496 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:55:28.496 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:55:28.496 DEBUG:boto:path=/dkyges-76/ 2024-04-12T04:55:28.496 DEBUG:boto:auth_path=/dkyges-76/ 2024-04-12T04:55:28.497 DEBUG:boto:Method: HEAD 2024-04-12T04:55:28.497 DEBUG:boto:Path: /dkyges-76/ 2024-04-12T04:55:28.497 DEBUG:boto:Data: 2024-04-12T04:55:28.497 DEBUG:boto:Headers: {} 2024-04-12T04:55:28.497 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:55:28.497 DEBUG:boto:Port: 8000 2024-04-12T04:55:28.497 DEBUG:boto:Params: {} 2024-04-12T04:55:28.498 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:55:28.498 DEBUG:boto:Token: None 2024-04-12T04:55:28.498 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:55:28 GMT /dkyges-76/ 2024-04-12T04:55:28.498 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gB+ce+UJJQ2IKFIqQglYa7QsjqM= 2024-04-12T04:55:28.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:55:28 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gB+ce+UJJQ2IKFIqQglYa7QsjqM=', 'Content-Length': '0'} 2024-04-12T04:55:28.513 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ccce888ae3bbefb-006618bec0-2f80-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:55:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:55:28.514 DEBUG:boto:path=/dkyges-76/obj3 2024-04-12T04:55:28.514 DEBUG:boto:auth_path=/dkyges-76/obj3 2024-04-12T04:55:28.514 DEBUG:boto:Method: PUT 2024-04-12T04:55:28.514 DEBUG:boto:Path: /dkyges-76/obj3 2024-04-12T04:55:28.514 DEBUG:boto:Data: 2024-04-12T04:55:28.514 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:55:28.515 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:55:28.515 DEBUG:boto:Port: 8000 2024-04-12T04:55:28.515 DEBUG:boto:Params: {} 2024-04-12T04:55:28.515 DEBUG:boto:Token: None 2024-04-12T04:55:28.515 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:55:28 GMT /dkyges-76/obj3 2024-04-12T04:55:28.515 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Mn6CzsvdW4cj0NS+0jxK2KBEBoo= 2024-04-12T04:55:28.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:55:28 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Mn6CzsvdW4cj0NS+0jxK2KBEBoo='} 2024-04-12T04:55:28.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8519f1f0c07a636-006618bec0-2f80-a1'), ('Date', 'Fri, 12 Apr 2024 04:55:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:55:28.527 DEBUG:boto:path=/dkyges-76/ 2024-04-12T04:55:28.527 DEBUG:boto:auth_path=/dkyges-76/ 2024-04-12T04:55:28.527 DEBUG:boto:Method: HEAD 2024-04-12T04:55:28.527 DEBUG:boto:Path: /dkyges-76/ 2024-04-12T04:55:28.527 DEBUG:boto:Data: 2024-04-12T04:55:28.527 DEBUG:boto:Headers: {} 2024-04-12T04:55:28.527 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:55:28.528 DEBUG:boto:Port: 8000 2024-04-12T04:55:28.528 DEBUG:boto:Params: {} 2024-04-12T04:55:28.528 DEBUG:boto:Token: None 2024-04-12T04:55:28.528 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:55:28 GMT /dkyges-76/ 2024-04-12T04:55:28.528 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:gB+ce+UJJQ2IKFIqQglYa7QsjqM= 2024-04-12T04:55:28.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:55:28 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gB+ce+UJJQ2IKFIqQglYa7QsjqM=', 'Content-Length': '0'} 2024-04-12T04:55:28.530 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000852fc20df01d9634-006618bec0-2f80-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:55:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:55:28.531 DEBUG:boto:path=/dkyges-76/obj4 2024-04-12T04:55:28.531 DEBUG:boto:auth_path=/dkyges-76/obj4 2024-04-12T04:55:28.531 DEBUG:boto:Method: PUT 2024-04-12T04:55:28.531 DEBUG:boto:Path: /dkyges-76/obj4 2024-04-12T04:55:28.531 DEBUG:boto:Data: 2024-04-12T04:55:28.531 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:55:28.531 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:55:28.531 DEBUG:boto:Port: 8000 2024-04-12T04:55:28.532 DEBUG:boto:Params: {} 2024-04-12T04:55:28.532 DEBUG:boto:Token: None 2024-04-12T04:55:28.532 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 12 Apr 2024 04:55:28 GMT /dkyges-76/obj4 2024-04-12T04:55:28.532 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:oKvi88FfNz+PZ1h4OQQpAoS/58Y= 2024-04-12T04:55:28.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:55:28 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oKvi88FfNz+PZ1h4OQQpAoS/58Y='} 2024-04-12T04:55:28.574 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035a5c146d3d1edd9-006618bec0-2f80-a1'), ('Date', 'Fri, 12 Apr 2024 04:55:28 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:55:28.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:28.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:28.575 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:55:28.631 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:28.632 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:28.669 INFO:teuthology.orchestra.run.smithi043.stderr:No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:55:28.686 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:55:28.686 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:28.686 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:55:28.686 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:55:28.686 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.687 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:28.687 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:55:28.687 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:55:28.687 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.687 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:28.687 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:55:28.687 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:55:28.687 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.687 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:28.687 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897635.897290_1048.1", 2024-04-12T04:55:28.687 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:53:55.897290Z" 2024-04-12T04:55:28.688 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:28.688 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:55:28.689 DEBUG:rgw_multi.tests:master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:55:28.689 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-04-12T04:55:28.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:28.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:28.689 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:55:28.745 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:28.745 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:55:28.778+0000 7f001f269780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:55:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-12T04:55:28.779+0000 7f001f269780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:55:28.798 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:55:28.798 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:55:28.798 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:55:28.798 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:55:28.798 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:55:28.798 INFO:teuthology.orchestra.run.smithi043.stdout: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:55:28.798 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:28.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.799 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:55:28.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:28.799 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:55:28.799 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:28.799 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:28.799 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:55:28.799 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:28.799 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 1, 2024-04-12T04:55:28.799 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:28.799 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:55:28.799 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:28.800 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:28.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:28.800 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:55:28.800 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:28.800 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:28.800 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:55:28.800 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:28.800 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:28.800 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:28.800 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:55:28.800 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:28.801 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:28.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:28.801 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:55:28.801 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:28.801 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:28.801 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:55:28.801 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:28.801 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:28.801 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:28.801 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:55:28.801 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:28.802 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:28.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:28.802 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:55:28.802 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:28.802 INFO:teuthology.orchestra.run.smithi043.stdout: "state": 1, 2024-04-12T04:55:28.802 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "1_1712897635.897290_1048.1", 2024-04-12T04:55:28.802 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:28.802 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:28.802 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:28.802 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:53:55.897290Z", 2024-04-12T04:55:28.803 INFO:teuthology.orchestra.run.smithi043.stdout: "realm_epoch": 2 2024-04-12T04:55:28.803 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:28.803 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:28.803 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:55:28.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.803 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:55:28.803 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 1, 2024-04-12T04:55:28.803 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 1 2024-04-12T04:55:28.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.803 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:55:28Z" 2024-04-12T04:55:28.803 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:55:28.804 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1712891567.956902_97.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2024-04-12T03:12:47.956902Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1712895069.167783_650.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.167783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "1_1712895069.196453_290.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:11:09.196453Z", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "1_1712897635.897290_1048.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:53:55.897290Z", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2024-04-12T04:55:28Z" } 2024-04-12T04:55:28.804 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712897635.897290_1048.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:53:55.897290Z', 'realm_epoch': 2}}] 2024-04-12T04:55:28.805 DEBUG:rgw_multi.tests:log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:55:28.805 DEBUG:rgw_multi.tests:sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712897635.897290_1048.1'} 2024-04-12T04:55:28.805 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-04-12T04:55:28.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:28.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:28.805 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:55:28.858 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:28.858 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:28.918 INFO:teuthology.orchestra.run.smithi043.stdout:[ 2024-04-12T04:55:28.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:28.918 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048340", 2024-04-12T04:55:28.919 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:51:49.229855Z" 2024-04-12T04:55:28.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:28.919 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:55:28.919 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:55:28.539577Z" 2024-04-12T04:55:28.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:28.919 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055475", 2024-04-12T04:55:28.919 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:55:28.524261Z" 2024-04-12T04:55:28.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:28.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:28.919 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:55:28.920 INFO:teuthology.orchestra.run.smithi043.stdout: "last_update": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:55:28.920 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:28.920 INFO:teuthology.orchestra.run.smithi043.stdout:] 2024-04-12T04:55:28.921 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000055475', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:55:28.921 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:55:28.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:28.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:28.921 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:55:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:28.978 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:29.031 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:55:29.031 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:55:29.031 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:55:29.031 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:55:29.031 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:55:29.031 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:55:29.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:29.032 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:55:29.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:29.032 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:55:29.032 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:29.032 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:29.032 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048340", 2024-04-12T04:55:29.032 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:29.032 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:29.032 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:29.033 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:49.229855Z" 2024-04-12T04:55:29.033 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:29.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:29.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:29.033 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:55:29.033 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:29.033 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:29.033 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:55:29.033 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:29.033 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:29.034 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:29.034 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:55:29.034 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:29.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:29.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:29.034 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:55:29.034 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:29.034 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:29.035 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055334", 2024-04-12T04:55:29.035 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:29.035 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:55:29.035 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:29.035 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:49.195479Z" 2024-04-12T04:55:29.035 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:29.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:29.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:29.035 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:55:29.035 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:29.036 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:29.036 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:55:29.036 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:29.036 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:55:29.036 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:29.036 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:55:29.036 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:29.036 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:29.036 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:55:29.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:29.037 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:55:29.037 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:55:29.037 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:55:29.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:29.037 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:55:29Z" 2024-04-12T04:55:29.037 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:55:29.038 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048340", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:49.229855Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055334", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:51:49.195479Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:55:29Z" } 2024-04-12T04:55:29.038 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048340', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:49.229855Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055334', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:51:49.195479Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:55:29.039 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000055475', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:55:29.039 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:55:29.039 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000055475 target=00000000000000000000:00000000000000055334 2024-04-12T04:55:34.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:34.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:34.040 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:55:34.102 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:34.102 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:34.159 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:55:34.159 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:55:34.160 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:55:34.160 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:55:34.160 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:55:34.160 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:55:34.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:34.160 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:55:34.160 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:34.160 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:55:34.160 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:34.160 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:34.160 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048340", 2024-04-12T04:55:34.160 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:34.161 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:34.161 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:34.161 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:49.229855Z" 2024-04-12T04:55:34.161 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:34.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:34.161 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:34.161 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:55:34.161 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:34.161 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:34.161 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:55:34.162 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:34.162 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:34.162 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:34.162 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:55:34.162 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:34.162 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:34.162 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:34.162 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:55:34.162 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:34.162 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:34.163 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055334", 2024-04-12T04:55:34.163 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:34.163 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:55:34.163 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:34.163 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:49.195479Z" 2024-04-12T04:55:34.163 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:34.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:34.163 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:34.163 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:55:34.163 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:34.163 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:34.164 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:55:34.164 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:34.164 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:55:34.164 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:34.164 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:55:34.164 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:34.164 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:34.164 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:55:34.164 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:34.164 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:55:34.164 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:55:34.164 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:55:34.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:34.165 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:55:34Z" 2024-04-12T04:55:34.165 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:55:34.166 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048340", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:49.229855Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055334", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:51:49.195479Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:55:34Z" } 2024-04-12T04:55:34.166 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048340', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:49.229855Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055334', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:51:49.195479Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:55:34.166 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000055475', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:55:34.166 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:55:34.166 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000055475 target=00000000000000000000:00000000000000055334 2024-04-12T04:55:39.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:39.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:39.168 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:55:39.230 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:39.230 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:39.289 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:55:39.289 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:55:39.289 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:55:39.289 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:55:39.289 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:55:39.289 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:55:39.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:39.289 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:55:39.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:39.290 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:55:39.290 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:39.290 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:39.290 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048340", 2024-04-12T04:55:39.290 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:39.290 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:39.290 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:39.290 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:49.229855Z" 2024-04-12T04:55:39.290 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:39.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:39.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:39.291 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:55:39.291 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:39.291 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:39.291 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:55:39.291 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:39.291 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:39.291 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:39.291 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:55:39.291 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:39.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:39.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:39.292 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:55:39.292 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:39.292 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:39.292 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055334", 2024-04-12T04:55:39.292 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:39.292 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:55:39.292 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:39.292 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:49.195479Z" 2024-04-12T04:55:39.292 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:39.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:39.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:39.293 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:55:39.293 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:39.293 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:39.293 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:55:39.293 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:39.293 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:55:39.293 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:39.293 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:55:39.293 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:39.293 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:39.293 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:55:39.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:39.294 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:55:39.294 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:55:39.294 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:55:39.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:39.294 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:55:39Z" 2024-04-12T04:55:39.294 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:55:39.295 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048340", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:49.229855Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055334", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:51:49.195479Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:55:39Z" } 2024-04-12T04:55:39.295 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048340', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:49.229855Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055334', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:51:49.195479Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:55:39.295 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000055475', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:55:39.295 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:55:39.295 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000055475 target=00000000000000000000:00000000000000055334 2024-04-12T04:55:44.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:44.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:44.296 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:55:44.360 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:44.360 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:44.416 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:55:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:55:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:55:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:55:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:55:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:55:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:55:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:55:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048340", 2024-04-12T04:55:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:49.229855Z" 2024-04-12T04:55:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:55:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:55:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:55:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:55:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055334", 2024-04-12T04:55:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:55:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:49.195479Z" 2024-04-12T04:55:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:55:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:55:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:55:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:55:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:55:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:55:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:55:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:55:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:55:44Z" 2024-04-12T04:55:44.422 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:55:44.422 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048340", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:49.229855Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055334", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:51:49.195479Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:55:44Z" } 2024-04-12T04:55:44.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048340', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:49.229855Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055334', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:51:49.195479Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:55:44.423 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000055475', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:55:44.423 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:55:44.423 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000055475 target=00000000000000000000:00000000000000055334 2024-04-12T04:55:49.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:49.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:49.425 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:55:49.487 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:49.487 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:49.545 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:55:49.546 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:55:49.546 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:55:49.546 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:55:49.546 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:55:49.546 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:55:49.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:49.546 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:55:49.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:49.546 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:55:49.546 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:49.547 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:49.547 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048340", 2024-04-12T04:55:49.547 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:49.547 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:49.547 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:49.547 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:49.229855Z" 2024-04-12T04:55:49.547 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:49.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:49.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:49.547 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:55:49.548 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:49.548 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:49.548 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:55:49.548 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:49.548 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:49.548 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:49.548 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:55:49.548 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:49.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:49.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:49.549 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:55:49.549 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:49.549 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:49.549 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055334", 2024-04-12T04:55:49.549 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:49.549 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:55:49.549 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:49.549 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:49.195479Z" 2024-04-12T04:55:49.549 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:49.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:49.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:49.550 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:55:49.550 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:49.550 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:49.550 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:55:49.550 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:49.550 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:55:49.550 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:49.550 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:55:49.550 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:49.550 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:49.550 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:55:49.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:49.551 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:55:49.551 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:55:49.551 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:55:49.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:49.551 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:55:49Z" 2024-04-12T04:55:49.551 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:55:49.552 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048340", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:49.229855Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055334", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:51:49.195479Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:55:49Z" } 2024-04-12T04:55:49.552 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048340', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:49.229855Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055334', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:51:49.195479Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:55:49.552 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000055475', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:55:49.552 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:55:49.553 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000055475 target=00000000000000000000:00000000000000055334 2024-04-12T04:55:54.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:54.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:54.554 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:55:54.618 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:54.618 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:54.676 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:55:54.676 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:55:54.676 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:55:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:55:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:55:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:55:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:55:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:55:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:54.677 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048340", 2024-04-12T04:55:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:49.229855Z" 2024-04-12T04:55:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:55:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:54.678 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033046", 2024-04-12T04:55:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:50:03.916286Z" 2024-04-12T04:55:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:55:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:54.679 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:54.680 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055334", 2024-04-12T04:55:54.680 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:54.680 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:55:54.680 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:54.680 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:49.195479Z" 2024-04-12T04:55:54.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:54.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:54.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:54.680 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:55:54.680 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:54.681 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:54.681 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:55:54.681 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:54.681 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:55:54.681 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:54.681 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:55:54.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:54.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:54.681 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:55:54.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:54.681 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:55:54.681 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:55:54.681 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:55:54.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:54.682 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:55:54Z" 2024-04-12T04:55:54.682 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:55:54.683 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048340", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:49.229855Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033046", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:50:03.916286Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055334", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:51:49.195479Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:55:54Z" } 2024-04-12T04:55:54.683 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048340', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:49.229855Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033046', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:50:03.916286Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055334', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:51:49.195479Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:55:54.683 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000055475', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:55:54.683 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033046', 2: '00000000000000000000:00000000000000055334', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:55:54.683 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=1 master=00000000000000000000:00000000000000033187 target=00000000000000000000:00000000000000033046, shard=2 master=00000000000000000000:00000000000000055475 target=00000000000000000000:00000000000000055334 2024-04-12T04:55:59.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:59.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:55:59.685 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:55:59.746 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:59.747 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:59.802 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-12T04:55:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_status": { 2024-04-12T04:55:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "info": { 2024-04-12T04:55:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "sync", 2024-04-12T04:55:59.802 INFO:teuthology.orchestra.run.smithi043.stdout: "num_shards": 4, 2024-04-12T04:55:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "instance_id": 5236430379833617501 2024-04-12T04:55:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "markers": [ 2024-04-12T04:55:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 0, 2024-04-12T04:55:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000048340", 2024-04-12T04:55:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:59.803 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:51:49.229855Z" 2024-04-12T04:55:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 1, 2024-04-12T04:55:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000033187", 2024-04-12T04:55:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 0, 2024-04-12T04:55:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:55:28.539577Z" 2024-04-12T04:55:59.804 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 2, 2024-04-12T04:55:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000055475", 2024-04-12T04:55:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 57, 2024-04-12T04:55:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:55:28.524261Z" 2024-04-12T04:55:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:59.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-12T04:55:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: "key": 3, 2024-04-12T04:55:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: "val": { 2024-04-12T04:55:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: "status": "incremental-sync", 2024-04-12T04:55:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: "marker": "00000000000000000000:00000000000000046714", 2024-04-12T04:55:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: "next_step_marker": "", 2024-04-12T04:55:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: "total_entries": 9, 2024-04-12T04:55:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: "pos": 0, 2024-04-12T04:55:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: "timestamp": "2024-04-12T04:47:28.039725Z" 2024-04-12T04:55:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-12T04:55:59.806 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-12T04:55:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: "full_sync": { 2024-04-12T04:55:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: "total": 66, 2024-04-12T04:55:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: "complete": 66 2024-04-12T04:55:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-12T04:55:59.807 INFO:teuthology.orchestra.run.smithi043.stdout: "current_time": "2024-04-12T04:55:59Z" 2024-04-12T04:55:59.807 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-12T04:55:59.808 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5236430379833617501 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000048340", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:51:49.229855Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033187", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2024-04-12T04:55:28.539577Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000055475", "next_step_marker": "", "total_entries": 57, "pos": 0, "timestamp": "2024-04-12T04:55:28.524261Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046714", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2024-04-12T04:47:28.039725Z" } } ] }, "full_sync": { "total": 66, "complete": 66 }, "current_time": "2024-04-12T04:55:59Z" } 2024-04-12T04:55:59.808 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000048340', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:51:49.229855Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033187', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:55:28.539577Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000055475', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:55:28.524261Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046714', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:47:28.039725Z'}}] 2024-04-12T04:55:59.808 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000055475', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:55:59.808 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000048340', 1: '00000000000000000000:00000000000000033187', 2: '00000000000000000000:00000000000000055475', 3: '00000000000000000000:00000000000000046714'} 2024-04-12T04:55:59.808 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:55:59.809 DEBUG:boto:path=/dkyges-76/ 2024-04-12T04:55:59.809 DEBUG:boto:auth_path=/dkyges-76/ 2024-04-12T04:55:59.809 DEBUG:boto:Method: HEAD 2024-04-12T04:55:59.809 DEBUG:boto:Path: /dkyges-76/ 2024-04-12T04:55:59.809 DEBUG:boto:Data: 2024-04-12T04:55:59.809 DEBUG:boto:Headers: {} 2024-04-12T04:55:59.809 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:55:59.809 DEBUG:boto:Port: 8001 2024-04-12T04:55:59.809 DEBUG:boto:Params: {} 2024-04-12T04:55:59.810 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:55:59.810 DEBUG:boto:Token: None 2024-04-12T04:55:59.810 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:55:59 GMT /dkyges-76/ 2024-04-12T04:55:59.810 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:p8f27v525QRjhwMdOLsEWw3+htA= 2024-04-12T04:55:59.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:55:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p8f27v525QRjhwMdOLsEWw3+htA=', 'Content-Length': '0'} 2024-04-12T04:55:59.826 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dba66474a3845a9d-006618bedf-2f82-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:55:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:55:59.826 DEBUG:boto:path=/dkyges-76/obj3 2024-04-12T04:55:59.826 DEBUG:boto:auth_path=/dkyges-76/obj3 2024-04-12T04:55:59.826 DEBUG:boto:Method: HEAD 2024-04-12T04:55:59.826 DEBUG:boto:Path: /dkyges-76/obj3 2024-04-12T04:55:59.826 DEBUG:boto:Data: 2024-04-12T04:55:59.826 DEBUG:boto:Headers: {} 2024-04-12T04:55:59.826 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:55:59.827 DEBUG:boto:Port: 8001 2024-04-12T04:55:59.827 DEBUG:boto:Params: {} 2024-04-12T04:55:59.827 DEBUG:boto:Token: None 2024-04-12T04:55:59.827 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:55:59 GMT /dkyges-76/obj3 2024-04-12T04:55:59.827 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:A0HTDlEgMlVFHU5wjTgIA4yLirM= 2024-04-12T04:55:59.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:55:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A0HTDlEgMlVFHU5wjTgIA4yLirM=', 'Content-Length': '0'} 2024-04-12T04:55:59.829 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:55:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-76:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.11504.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bc570784fa0ecf0-006618bedf-2f82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:55:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:55:59.830 DEBUG:boto:path=/dkyges-76/obj3 2024-04-12T04:55:59.830 DEBUG:boto:auth_path=/dkyges-76/obj3 2024-04-12T04:55:59.830 DEBUG:boto:Method: GET 2024-04-12T04:55:59.830 DEBUG:boto:Path: /dkyges-76/obj3 2024-04-12T04:55:59.830 DEBUG:boto:Data: 2024-04-12T04:55:59.830 DEBUG:boto:Headers: {} 2024-04-12T04:55:59.830 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:55:59.830 DEBUG:boto:Port: 8001 2024-04-12T04:55:59.830 DEBUG:boto:Params: {} 2024-04-12T04:55:59.831 DEBUG:boto:Token: None 2024-04-12T04:55:59.831 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:55:59 GMT /dkyges-76/obj3 2024-04-12T04:55:59.831 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:52VF2SaBpct62zfDWHgaFrJ422k= 2024-04-12T04:55:59.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:55:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:52VF2SaBpct62zfDWHgaFrJ422k=', 'Content-Length': '0'} 2024-04-12T04:55:59.833 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:55:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-76:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.11504.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0f9f0480e381620-006618bedf-2f82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:55:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:55:59.833 DEBUG:boto:path=/dkyges-76/obj4 2024-04-12T04:55:59.833 DEBUG:boto:auth_path=/dkyges-76/obj4 2024-04-12T04:55:59.833 DEBUG:boto:Method: HEAD 2024-04-12T04:55:59.833 DEBUG:boto:Path: /dkyges-76/obj4 2024-04-12T04:55:59.834 DEBUG:boto:Data: 2024-04-12T04:55:59.834 DEBUG:boto:Headers: {} 2024-04-12T04:55:59.834 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:55:59.834 DEBUG:boto:Port: 8001 2024-04-12T04:55:59.834 DEBUG:boto:Params: {} 2024-04-12T04:55:59.834 DEBUG:boto:Token: None 2024-04-12T04:55:59.834 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:55:59 GMT /dkyges-76/obj4 2024-04-12T04:55:59.834 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:xa/vzBgJdowg1QTCoAfwQ8MueGA= 2024-04-12T04:55:59.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:55:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xa/vzBgJdowg1QTCoAfwQ8MueGA=', 'Content-Length': '0'} 2024-04-12T04:55:59.836 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:55:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-76:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.11504.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000daced14b14e59341-006618bedf-2f82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:55:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:55:59.836 DEBUG:boto:path=/dkyges-76/obj4 2024-04-12T04:55:59.836 DEBUG:boto:auth_path=/dkyges-76/obj4 2024-04-12T04:55:59.836 DEBUG:boto:Method: GET 2024-04-12T04:55:59.836 DEBUG:boto:Path: /dkyges-76/obj4 2024-04-12T04:55:59.837 DEBUG:boto:Data: 2024-04-12T04:55:59.837 DEBUG:boto:Headers: {} 2024-04-12T04:55:59.837 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:55:59.837 DEBUG:boto:Port: 8001 2024-04-12T04:55:59.837 DEBUG:boto:Params: {} 2024-04-12T04:55:59.837 DEBUG:boto:Token: None 2024-04-12T04:55:59.837 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:55:59 GMT /dkyges-76/obj4 2024-04-12T04:55:59.837 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zh5MVwo/O4WWOl+Wgjj2V26A+Dk= 2024-04-12T04:55:59.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:55:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zh5MVwo/O4WWOl+Wgjj2V26A+Dk=', 'Content-Length': '0'} 2024-04-12T04:55:59.839 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:55:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-76:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.11504.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004a6dd7e3b93c6d7-006618bedf-2f82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:55:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:55:59.839 DEBUG:boto:path=/dkyges-77/ 2024-04-12T04:55:59.839 DEBUG:boto:auth_path=/dkyges-77/ 2024-04-12T04:55:59.840 DEBUG:boto:Method: HEAD 2024-04-12T04:55:59.840 DEBUG:boto:Path: /dkyges-77/ 2024-04-12T04:55:59.840 DEBUG:boto:Data: 2024-04-12T04:55:59.840 DEBUG:boto:Headers: {} 2024-04-12T04:55:59.840 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:55:59.840 DEBUG:boto:Port: 8001 2024-04-12T04:55:59.840 DEBUG:boto:Params: {} 2024-04-12T04:55:59.840 DEBUG:boto:Token: None 2024-04-12T04:55:59.840 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:55:59 GMT /dkyges-77/ 2024-04-12T04:55:59.840 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:Zduz/eIbZi/Nn9LDFmZ16WyIzBU= 2024-04-12T04:55:59.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:55:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zduz/eIbZi/Nn9LDFmZ16WyIzBU=', 'Content-Length': '0'} 2024-04-12T04:55:59.842 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec5d107bf259fc6d-006618bedf-2f82-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:55:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:55:59.843 DEBUG:boto:path=/dkyges-77/obj3 2024-04-12T04:55:59.843 DEBUG:boto:auth_path=/dkyges-77/obj3 2024-04-12T04:55:59.843 DEBUG:boto:Method: HEAD 2024-04-12T04:55:59.843 DEBUG:boto:Path: /dkyges-77/obj3 2024-04-12T04:55:59.843 DEBUG:boto:Data: 2024-04-12T04:55:59.843 DEBUG:boto:Headers: {} 2024-04-12T04:55:59.843 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:55:59.843 DEBUG:boto:Port: 8001 2024-04-12T04:55:59.843 DEBUG:boto:Params: {} 2024-04-12T04:55:59.843 DEBUG:boto:Token: None 2024-04-12T04:55:59.844 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:55:59 GMT /dkyges-77/obj3 2024-04-12T04:55:59.844 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:zHvf2hzRiURZFeEm5WdJc05vuEw= 2024-04-12T04:55:59.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:55:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zHvf2hzRiURZFeEm5WdJc05vuEw=', 'Content-Length': '0'} 2024-04-12T04:55:59.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:55:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-76:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.11504.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c39be1144f3fa4e9-006618bedf-2f82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:55:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:55:59.846 DEBUG:boto:path=/dkyges-77/obj3 2024-04-12T04:55:59.846 DEBUG:boto:auth_path=/dkyges-77/obj3 2024-04-12T04:55:59.846 DEBUG:boto:Method: GET 2024-04-12T04:55:59.846 DEBUG:boto:Path: /dkyges-77/obj3 2024-04-12T04:55:59.846 DEBUG:boto:Data: 2024-04-12T04:55:59.846 DEBUG:boto:Headers: {} 2024-04-12T04:55:59.846 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:55:59.846 DEBUG:boto:Port: 8001 2024-04-12T04:55:59.846 DEBUG:boto:Params: {} 2024-04-12T04:55:59.846 DEBUG:boto:Token: None 2024-04-12T04:55:59.847 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:55:59 GMT /dkyges-77/obj3 2024-04-12T04:55:59.847 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:i95dtOF7rcfAg1SU/gvuYtOMe48= 2024-04-12T04:55:59.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:55:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i95dtOF7rcfAg1SU/gvuYtOMe48=', 'Content-Length': '0'} 2024-04-12T04:55:59.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:55:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-76:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.11504.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006a8355da7ee75628-006618bedf-2f82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:55:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:55:59.849 DEBUG:boto:path=/dkyges-77/obj4 2024-04-12T04:55:59.849 DEBUG:boto:auth_path=/dkyges-77/obj4 2024-04-12T04:55:59.849 DEBUG:boto:Method: HEAD 2024-04-12T04:55:59.849 DEBUG:boto:Path: /dkyges-77/obj4 2024-04-12T04:55:59.849 DEBUG:boto:Data: 2024-04-12T04:55:59.850 DEBUG:boto:Headers: {} 2024-04-12T04:55:59.850 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:55:59.850 DEBUG:boto:Port: 8001 2024-04-12T04:55:59.850 DEBUG:boto:Params: {} 2024-04-12T04:55:59.850 DEBUG:boto:Token: None 2024-04-12T04:55:59.850 DEBUG:boto:StringToSign: HEAD Fri, 12 Apr 2024 04:55:59 GMT /dkyges-77/obj4 2024-04-12T04:55:59.850 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:k63z8epuMoRVN8Ve/J+Wqo8xgfw= 2024-04-12T04:55:59.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:55:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:k63z8epuMoRVN8Ve/J+Wqo8xgfw=', 'Content-Length': '0'} 2024-04-12T04:55:59.852 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:55:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-76:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.11504.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000182798f3beb76cf5-006618bedf-2f82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:55:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:55:59.852 DEBUG:boto:path=/dkyges-77/obj4 2024-04-12T04:55:59.852 DEBUG:boto:auth_path=/dkyges-77/obj4 2024-04-12T04:55:59.852 DEBUG:boto:Method: GET 2024-04-12T04:55:59.852 DEBUG:boto:Path: /dkyges-77/obj4 2024-04-12T04:55:59.852 DEBUG:boto:Data: 2024-04-12T04:55:59.852 DEBUG:boto:Headers: {} 2024-04-12T04:55:59.852 DEBUG:boto:Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:55:59.853 DEBUG:boto:Port: 8001 2024-04-12T04:55:59.853 DEBUG:boto:Params: {} 2024-04-12T04:55:59.853 DEBUG:boto:Token: None 2024-04-12T04:55:59.853 DEBUG:boto:StringToSign: GET Fri, 12 Apr 2024 04:55:59 GMT /dkyges-77/obj4 2024-04-12T04:55:59.853 DEBUG:boto:Signature: AWS KUXF4P9H15V1YOGZBQF6:x515lekiOElJn6htgEhVQZn6ph0= 2024-04-12T04:55:59.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:55:59 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:x515lekiOElJn6htgEhVQZn6ph0=', 'Content-Length': '0'} 2024-04-12T04:55:59.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:55:28 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-76:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.11504.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009ffa035f2e0147be-006618bedf-2f82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:55:59 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:55:59.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dkyges-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:55:59.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'dkyges-76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:55:59.855 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket dkyges-76 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:55:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:55:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:55:59.982 INFO:teuthology.orchestra.run.smithi043.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:55:59.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:55:59.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:55:59.983 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.095 INFO:teuthology.orchestra.run.smithi043.stdout:{"sync_policy":{"groups":[]}} 2024-04-12T04:56:00.096 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... ok 2024-04-12T04:56:00.097 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.097 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-12T04:56:00.097 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_object_delete 2024-04-12T04:56:00.097 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-12T04:56:00.098 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-12T04:56:00.098 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-12T04:56:00.098 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-12T04:56:00.098 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 687, in test_multi_object_delete 2024-04-12T04:56:00.098 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket.name) 2024-04-12T04:56:00.099 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-12T04:56:00.099 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-12T04:56:00.099 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-12T04:56:00.099 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-12T04:56:00.099 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-12T04:56:00.099 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-12T04:56:00.099 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=dkyges-11 2024-04-12T04:56:00.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.100 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.100 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.100 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.100 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:42 GMT 2024-04-12T04:56:00.100 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.100 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ObmF7ZPgDmoetIO297QL1jpvMGY= 2024-04-12T04:56:00.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ObmF7ZPgDmoetIO297QL1jpvMGY=', 'Content-Length': '0'} 2024-04-12T04:56:00.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx0000037444dabed7e33b4-006618a84e-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.101 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=dkyges-12 2024-04-12T04:56:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.101 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.101 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.101 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.102 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:42 GMT 2024-04-12T04:56:00.102 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.102 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gdpmZWGj/iRGRCWfBNByD18q3ww= 2024-04-12T04:56:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gdpmZWGj/iRGRCWfBNByD18q3ww=', 'Content-Length': '0'} 2024-04-12T04:56:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000006f2ed29c19506f61-006618a84e-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.103 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.103 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.103 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.103 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:42 GMT 2024-04-12T04:56:00.103 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.103 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:xDPVQOae4G9S5DTOgeK1bAfY0Cw= 2024-04-12T04:56:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xDPVQOae4G9S5DTOgeK1bAfY0Cw=', 'Content-Length': '0'} 2024-04-12T04:56:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ec05f64bc6c5988e-006618a84e-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj1 2024-04-12T04:56:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj1 2024-04-12T04:56:00.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj1 2024-04-12T04:56:00.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.104 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.104 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.104 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.104 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:42 GMT 2024-04-12T04:56:00.104 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj1 2024-04-12T04:56:00.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.104 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NCnRWL4u2UCsgzQqX8WBdhcewUM= 2024-04-12T04:56:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NCnRWL4u2UCsgzQqX8WBdhcewUM='} 2024-04-12T04:56:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000227e1415fcb9b709-006618a84e-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.105 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.105 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.106 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.106 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.106 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.106 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa95f3608ae5778f-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj2 2024-04-12T04:56:00.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj2 2024-04-12T04:56:00.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj2 2024-04-12T04:56:00.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.107 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.107 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.107 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.107 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.107 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj2 2024-04-12T04:56:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.107 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:5wWSt4aobjCTIpOdfL2ewfI+GNQ= 2024-04-12T04:56:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5wWSt4aobjCTIpOdfL2ewfI+GNQ='} 2024-04-12T04:56:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f7662a96cb605e3-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.108 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.108 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.108 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.109 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.109 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.109 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064713742f802f1a3-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj3 2024-04-12T04:56:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj3 2024-04-12T04:56:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj3 2024-04-12T04:56:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.110 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.110 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.110 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.110 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.110 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj3 2024-04-12T04:56:00.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.110 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:aCaet8mYtOEx4xZydzW3tMEMYXU= 2024-04-12T04:56:00.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aCaet8mYtOEx4xZydzW3tMEMYXU='} 2024-04-12T04:56:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dea8973b22229e94-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.111 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.111 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.112 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.112 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.112 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.112 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000752eb90b48342044-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj4 2024-04-12T04:56:00.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj4 2024-04-12T04:56:00.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj4 2024-04-12T04:56:00.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.113 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.113 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.113 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.113 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.113 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj4 2024-04-12T04:56:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.113 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:EghNKGR95MNwc8rkY4UFybIDHKU= 2024-04-12T04:56:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EghNKGR95MNwc8rkY4UFybIDHKU='} 2024-04-12T04:56:00.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000011d31e232f72f30c-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.114 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.114 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.114 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.114 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.115 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.115 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000056c48cbfa8db62e-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj5 2024-04-12T04:56:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj5 2024-04-12T04:56:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj5 2024-04-12T04:56:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.116 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.116 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.116 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.116 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.116 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj5 2024-04-12T04:56:00.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.116 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:5BanJtk7qwcsmxUbMKwX5971dzI= 2024-04-12T04:56:00.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5BanJtk7qwcsmxUbMKwX5971dzI='} 2024-04-12T04:56:00.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009029d4f494130b97-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.117 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.117 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.117 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.117 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.118 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.118 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009acc5ac4d1877645-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj6 2024-04-12T04:56:00.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj6 2024-04-12T04:56:00.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj6 2024-04-12T04:56:00.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.119 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.119 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.119 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.119 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.119 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj6 2024-04-12T04:56:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.119 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:3yHkarfswFEa+MY3vTQQOqF8+N0= 2024-04-12T04:56:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3yHkarfswFEa+MY3vTQQOqF8+N0='} 2024-04-12T04:56:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6a84693e8c5f1ec-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.120 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.120 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.120 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.121 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.121 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.121 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c5c01e5e792f5e0-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj7 2024-04-12T04:56:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj7 2024-04-12T04:56:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj7 2024-04-12T04:56:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.122 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.122 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.122 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.122 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.122 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj7 2024-04-12T04:56:00.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.122 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:izQ8461tBXgVuqf+R1s9BhJvVLI= 2024-04-12T04:56:00.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:izQ8461tBXgVuqf+R1s9BhJvVLI='} 2024-04-12T04:56:00.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0dd79bad552519a-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.123 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.123 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.123 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.123 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.123 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.123 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fdbcaf106623e2ac-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj8 2024-04-12T04:56:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj8 2024-04-12T04:56:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj8 2024-04-12T04:56:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.124 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.125 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.125 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.125 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.125 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj8 2024-04-12T04:56:00.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.125 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:nwHhiouhgQ7KT2tf2eBV9s6f/ZE= 2024-04-12T04:56:00.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nwHhiouhgQ7KT2tf2eBV9s6f/ZE='} 2024-04-12T04:56:00.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025f3b0a3abd563fc-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.126 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.126 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.126 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.126 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.126 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.126 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a87b5e9bd515a574-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj9 2024-04-12T04:56:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj9 2024-04-12T04:56:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj9 2024-04-12T04:56:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.127 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.127 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.127 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.128 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.128 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj9 2024-04-12T04:56:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.128 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6JRxZxW671VDGxHyUgR6vaH9Cl0= 2024-04-12T04:56:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6JRxZxW671VDGxHyUgR6vaH9Cl0='} 2024-04-12T04:56:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000977d100f4960f9e9-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.129 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.129 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.129 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.129 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.129 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.129 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000062d75b68c60b23d-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj10 2024-04-12T04:56:00.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj10 2024-04-12T04:56:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj10 2024-04-12T04:56:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.130 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.130 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.130 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.130 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.131 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj10 2024-04-12T04:56:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.131 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:lfQacQMHZyZHwX0hJRtcpqpC/zc= 2024-04-12T04:56:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lfQacQMHZyZHwX0hJRtcpqpC/zc='} 2024-04-12T04:56:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000703ef13c27a52ab8-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.132 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.132 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.132 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.132 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.132 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.132 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009275fc2cb6280c2-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj11 2024-04-12T04:56:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj11 2024-04-12T04:56:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj11 2024-04-12T04:56:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.133 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.133 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.133 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.133 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.133 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj11 2024-04-12T04:56:00.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.133 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:2vpBmV/6oENe2ntHFjPL6j9zjGU= 2024-04-12T04:56:00.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2vpBmV/6oENe2ntHFjPL6j9zjGU='} 2024-04-12T04:56:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021886eb00214b491-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.135 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.135 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.135 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.135 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.135 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.135 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f9e26e2919dc8b3-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj12 2024-04-12T04:56:00.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj12 2024-04-12T04:56:00.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj12 2024-04-12T04:56:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.136 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.136 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.136 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.136 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.136 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj12 2024-04-12T04:56:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.137 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Zp0A8sBQDknwnsdSBFNtG7SEHcY= 2024-04-12T04:56:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Zp0A8sBQDknwnsdSBFNtG7SEHcY='} 2024-04-12T04:56:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000710ca795276b6c45-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.137 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.138 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.138 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.138 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.138 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.138 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003997855eaf0ce8d5-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj13 2024-04-12T04:56:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj13 2024-04-12T04:56:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj13 2024-04-12T04:56:00.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.139 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.139 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.139 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.139 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.139 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj13 2024-04-12T04:56:00.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.139 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:h+Mf1NUu3Mvmk9Ob/6L25rJR/AY= 2024-04-12T04:56:00.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:h+Mf1NUu3Mvmk9Ob/6L25rJR/AY='} 2024-04-12T04:56:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d02300039323317e-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.140 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.140 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.141 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.141 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.141 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.141 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ffc4e535b038bc19-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj14 2024-04-12T04:56:00.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj14 2024-04-12T04:56:00.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj14 2024-04-12T04:56:00.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.142 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.142 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.142 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.142 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.142 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj14 2024-04-12T04:56:00.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.142 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:UfqeKfP7M6dFNgnLpkUCan/NWgE= 2024-04-12T04:56:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UfqeKfP7M6dFNgnLpkUCan/NWgE='} 2024-04-12T04:56:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1d47e5ec41a3a1f-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.144 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.144 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.144 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.144 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.144 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.144 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d4c5e7ecc70167a-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj15 2024-04-12T04:56:00.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj15 2024-04-12T04:56:00.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj15 2024-04-12T04:56:00.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.145 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.146 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.146 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.146 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.146 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj15 2024-04-12T04:56:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.146 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Fpoo9Bkx2X+xOxOhAkRNlnIybjE= 2024-04-12T04:56:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Fpoo9Bkx2X+xOxOhAkRNlnIybjE='} 2024-04-12T04:56:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac8078199a9a8200-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.147 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.147 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.147 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.147 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.147 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.147 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0606f16d4bcaa59-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj16 2024-04-12T04:56:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj16 2024-04-12T04:56:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj16 2024-04-12T04:56:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.148 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.149 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.149 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.149 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.149 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj16 2024-04-12T04:56:00.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.149 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:VwLT5RYP7vXy6BgSzNofPRCKIIE= 2024-04-12T04:56:00.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VwLT5RYP7vXy6BgSzNofPRCKIIE='} 2024-04-12T04:56:00.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d16b1b72581bc09-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.150 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.150 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.150 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.150 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.150 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.150 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072f78267e348a83b-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj17 2024-04-12T04:56:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj17 2024-04-12T04:56:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj17 2024-04-12T04:56:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.151 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.151 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.151 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.151 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.151 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj17 2024-04-12T04:56:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.152 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:2mAebRhafTBwALFD6dVkP87KLbE= 2024-04-12T04:56:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2mAebRhafTBwALFD6dVkP87KLbE='} 2024-04-12T04:56:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000055e1bb19d5ffc234-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.153 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.153 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.153 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.153 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.153 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.153 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009de2b078a3d75b78-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj18 2024-04-12T04:56:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj18 2024-04-12T04:56:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj18 2024-04-12T04:56:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.154 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.154 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.154 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.154 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.154 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj18 2024-04-12T04:56:00.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.154 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:mEOJWmMGNFl8hueiAts2j4KqPP0= 2024-04-12T04:56:00.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mEOJWmMGNFl8hueiAts2j4KqPP0='} 2024-04-12T04:56:00.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002905748ea6394a10-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.155 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.155 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.155 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.155 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.155 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.156 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098aab3097af80562-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj19 2024-04-12T04:56:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj19 2024-04-12T04:56:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj19 2024-04-12T04:56:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.156 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.157 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.157 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.157 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.157 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj19 2024-04-12T04:56:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.157 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:itEUQwg9AGclj4ZM86Cnx9n/KcM= 2024-04-12T04:56:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:itEUQwg9AGclj4ZM86Cnx9n/KcM='} 2024-04-12T04:56:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002208e94b65f576e2-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.158 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.158 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.158 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.158 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.158 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.158 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a49de8d063ca722a-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj20 2024-04-12T04:56:00.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj20 2024-04-12T04:56:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj20 2024-04-12T04:56:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.159 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.159 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.159 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.159 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.159 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj20 2024-04-12T04:56:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.160 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:lqmGbMyLkdEPNlvUWyk5tp8uAlE= 2024-04-12T04:56:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lqmGbMyLkdEPNlvUWyk5tp8uAlE='} 2024-04-12T04:56:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000653e7fe23f834d6b-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.161 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.161 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.161 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.161 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.161 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.161 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008cbb4af60581e901-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj21 2024-04-12T04:56:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj21 2024-04-12T04:56:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj21 2024-04-12T04:56:00.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.162 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.162 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.162 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.162 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.162 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj21 2024-04-12T04:56:00.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.162 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Lg1QYV5C6h3fgbQgnk5AfvaO4rI= 2024-04-12T04:56:00.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Lg1QYV5C6h3fgbQgnk5AfvaO4rI='} 2024-04-12T04:56:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000033cc4df884f8c13-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.163 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.163 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.163 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.163 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.164 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.164 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000265957bfcdbb067c-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj22 2024-04-12T04:56:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj22 2024-04-12T04:56:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj22 2024-04-12T04:56:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.165 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.165 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.165 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.165 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.165 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj22 2024-04-12T04:56:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.165 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:KIlhalNP/COI2aOP0Bj7FP8ANrQ= 2024-04-12T04:56:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KIlhalNP/COI2aOP0Bj7FP8ANrQ='} 2024-04-12T04:56:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000848f1d218d48e3b2-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.166 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.166 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.166 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.166 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.166 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.166 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d84d77eaaac5594a-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj23 2024-04-12T04:56:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj23 2024-04-12T04:56:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj23 2024-04-12T04:56:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.167 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.167 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.167 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.167 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.168 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj23 2024-04-12T04:56:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.168 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:8+2nrC/QdIGYVH638acuYD2yXRw= 2024-04-12T04:56:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8+2nrC/QdIGYVH638acuYD2yXRw='} 2024-04-12T04:56:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b27d0c0f955dac3-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.169 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.169 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.169 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.169 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.169 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.169 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dbe3add891c9bdc4-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj24 2024-04-12T04:56:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj24 2024-04-12T04:56:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj24 2024-04-12T04:56:00.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.170 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.170 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.170 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.170 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.170 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj24 2024-04-12T04:56:00.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.170 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/rZ1RRgcQKTeTFAxmebBcfa8yuI= 2024-04-12T04:56:00.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/rZ1RRgcQKTeTFAxmebBcfa8yuI='} 2024-04-12T04:56:00.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c21d1a73011187d-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.171 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.171 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.171 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.171 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.172 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.172 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037ba770606cf3631-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj25 2024-04-12T04:56:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj25 2024-04-12T04:56:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj25 2024-04-12T04:56:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.173 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.173 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.173 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.173 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.173 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj25 2024-04-12T04:56:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.173 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:FVmMBXm78/nKdEUB0qy/SgsO2dQ= 2024-04-12T04:56:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FVmMBXm78/nKdEUB0qy/SgsO2dQ='} 2024-04-12T04:56:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c19bf748c6a2eb89-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.174 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.174 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.174 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.174 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.174 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.174 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000395c3958c601dcdc-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj26 2024-04-12T04:56:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj26 2024-04-12T04:56:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj26 2024-04-12T04:56:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.175 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.175 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.175 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.175 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.176 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj26 2024-04-12T04:56:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.176 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LddhUHoTPD15FCxkZKUwFFO8mvI= 2024-04-12T04:56:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LddhUHoTPD15FCxkZKUwFFO8mvI='} 2024-04-12T04:56:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8d1824a223f9a65-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.177 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.177 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.177 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.177 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.177 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.177 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b4047d7155801d26-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj27 2024-04-12T04:56:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj27 2024-04-12T04:56:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj27 2024-04-12T04:56:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.178 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.178 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.178 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.178 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.178 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj27 2024-04-12T04:56:00.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.178 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Kn/A4cGFUShfI/byazE38CGkfP8= 2024-04-12T04:56:00.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Kn/A4cGFUShfI/byazE38CGkfP8='} 2024-04-12T04:56:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7d29e7469fb3b83-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.179 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.179 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.179 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.180 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.180 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.180 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c45823e45bdb65ca-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj28 2024-04-12T04:56:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj28 2024-04-12T04:56:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj28 2024-04-12T04:56:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.181 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.181 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.181 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.181 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.181 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj28 2024-04-12T04:56:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.181 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:HtoY+JuD4fn9Aw3nSTAp6+RkXgc= 2024-04-12T04:56:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HtoY+JuD4fn9Aw3nSTAp6+RkXgc='} 2024-04-12T04:56:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e73871e8b8a5d42-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.182 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.182 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.182 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.182 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.182 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.182 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000836ac96ba1f28dea-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj29 2024-04-12T04:56:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj29 2024-04-12T04:56:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj29 2024-04-12T04:56:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.183 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.184 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.184 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.184 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.184 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj29 2024-04-12T04:56:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.184 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:rqSn+4Is4buxj3kH4po7XGRyKyI= 2024-04-12T04:56:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rqSn+4Is4buxj3kH4po7XGRyKyI='} 2024-04-12T04:56:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000118132f2ac6b2533-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.185 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.185 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.185 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.185 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.185 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.185 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cc37dcbe9e1eac87-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj30 2024-04-12T04:56:00.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj30 2024-04-12T04:56:00.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj30 2024-04-12T04:56:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.186 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.186 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.186 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.186 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.186 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj30 2024-04-12T04:56:00.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.187 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:QCzWw5MMkh3texDZPmiwKuVVZJ0= 2024-04-12T04:56:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QCzWw5MMkh3texDZPmiwKuVVZJ0='} 2024-04-12T04:56:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab59fa2a13088b17-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.187 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.188 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.188 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.188 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.188 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.188 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e6bdf5c1e7c5e19-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj31 2024-04-12T04:56:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj31 2024-04-12T04:56:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj31 2024-04-12T04:56:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.189 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.189 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.189 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.189 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.189 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj31 2024-04-12T04:56:00.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.189 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:97wxdRiwnUvfrpGZmT1EOthGQr8= 2024-04-12T04:56:00.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:97wxdRiwnUvfrpGZmT1EOthGQr8='} 2024-04-12T04:56:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d55ff1b6466fad9c-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.191 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.191 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.191 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.191 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.191 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.191 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1bde25b6bbbb8fd-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj32 2024-04-12T04:56:00.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj32 2024-04-12T04:56:00.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj32 2024-04-12T04:56:00.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.192 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.192 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.192 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.192 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.192 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj32 2024-04-12T04:56:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.192 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:0F1fOyCgvoaGiRQ6YfyFp+K4U+o= 2024-04-12T04:56:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0F1fOyCgvoaGiRQ6YfyFp+K4U+o='} 2024-04-12T04:56:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000028082c7bc120ba8e-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.193 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.193 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.193 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.193 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.193 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.194 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8= 2024-04-12T04:56:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/5Ae74/S3Vf/Up1UCRAxM3u/Oi8=', 'Content-Length': '0'} 2024-04-12T04:56:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081893cc72bc7b6ef-006618a84f-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj33 2024-04-12T04:56:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj33 2024-04-12T04:56:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj33 2024-04-12T04:56:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.195 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.195 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.195 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.195 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:43 GMT 2024-04-12T04:56:00.195 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj33 2024-04-12T04:56:00.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.195 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:8TJ91LlBh65Qm2SmEgrCwv6+1Yw= 2024-04-12T04:56:00.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8TJ91LlBh65Qm2SmEgrCwv6+1Yw='} 2024-04-12T04:56:00.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002deb7ff6c88f14cc-006618a84f-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.196 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.196 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.196 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.196 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.196 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.196 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T04:56:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T04:56:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006fcaeec328d88847-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj34 2024-04-12T04:56:00.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj34 2024-04-12T04:56:00.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj34 2024-04-12T04:56:00.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.197 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.197 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.197 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.197 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.197 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj34 2024-04-12T04:56:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.198 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:0VwER8EiO++IWqvaIFyg+0DATlE= 2024-04-12T04:56:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0VwER8EiO++IWqvaIFyg+0DATlE='} 2024-04-12T04:56:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000067730255c1721f9-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.199 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.199 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.199 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.199 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.199 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.199 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T04:56:00.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T04:56:00.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000047e5992745c221b-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj35 2024-04-12T04:56:00.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj35 2024-04-12T04:56:00.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj35 2024-04-12T04:56:00.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.200 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.200 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.200 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.200 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.200 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj35 2024-04-12T04:56:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.200 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:as9REBOJ+xJ/y+/lnXeyhXPRdk4= 2024-04-12T04:56:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:as9REBOJ+xJ/y+/lnXeyhXPRdk4='} 2024-04-12T04:56:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000013948c89acdb51fa-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.201 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.201 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.201 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.201 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.201 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.202 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T04:56:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T04:56:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000342297c6c5ab6d4c-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj36 2024-04-12T04:56:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj36 2024-04-12T04:56:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj36 2024-04-12T04:56:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.202 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.203 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.203 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.203 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.203 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj36 2024-04-12T04:56:00.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.203 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:U5YMNLJZkCeuYjWFVu1vZyvCJYo= 2024-04-12T04:56:00.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U5YMNLJZkCeuYjWFVu1vZyvCJYo='} 2024-04-12T04:56:00.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc24c685343b87dc-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.204 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.204 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.204 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.204 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.204 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.204 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T04:56:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T04:56:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c10e0715fbcac66-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj37 2024-04-12T04:56:00.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj37 2024-04-12T04:56:00.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj37 2024-04-12T04:56:00.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.205 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.205 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.205 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.205 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.205 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj37 2024-04-12T04:56:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.206 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:36dCdMGvj3WxNR2dgyq3h8R81hg= 2024-04-12T04:56:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:36dCdMGvj3WxNR2dgyq3h8R81hg='} 2024-04-12T04:56:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000056bd06e54d2cce68-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.207 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.207 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.207 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.207 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.207 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.207 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T04:56:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T04:56:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003501098c5a3061c7-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj38 2024-04-12T04:56:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj38 2024-04-12T04:56:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj38 2024-04-12T04:56:00.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.208 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.208 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.208 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.208 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.208 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj38 2024-04-12T04:56:00.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.208 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:4DpKWPaZBPKcgJ9SJLxR+XS1XPM= 2024-04-12T04:56:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4DpKWPaZBPKcgJ9SJLxR+XS1XPM='} 2024-04-12T04:56:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000516c30a9662e279c-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.210 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.210 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.210 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.210 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.210 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.210 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T04:56:00.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T04:56:00.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dec81a2fcb4196e1-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj39 2024-04-12T04:56:00.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj39 2024-04-12T04:56:00.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj39 2024-04-12T04:56:00.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.211 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.211 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.211 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.211 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.211 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj39 2024-04-12T04:56:00.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.211 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:utygtF/ZtbScdedqS6VAc8M1Hgc= 2024-04-12T04:56:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:utygtF/ZtbScdedqS6VAc8M1Hgc='} 2024-04-12T04:56:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7507505292773e8-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.212 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.213 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.213 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.213 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.213 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.213 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T04:56:00.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T04:56:00.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041c407d352ac860b-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj40 2024-04-12T04:56:00.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj40 2024-04-12T04:56:00.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj40 2024-04-12T04:56:00.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.214 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.214 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.214 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.214 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.214 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj40 2024-04-12T04:56:00.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.214 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:UUggWyiojPRvvfSUSi1/u/Gu0m0= 2024-04-12T04:56:00.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UUggWyiojPRvvfSUSi1/u/Gu0m0='} 2024-04-12T04:56:00.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000031503d375085ce6c-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.215 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.216 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.216 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.216 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.216 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.216 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T04:56:00.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T04:56:00.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8eba9346bb78122-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj41 2024-04-12T04:56:00.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj41 2024-04-12T04:56:00.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj41 2024-04-12T04:56:00.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.217 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.217 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.217 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.217 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.217 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj41 2024-04-12T04:56:00.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.217 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:y2uarvryy8SHjdJtlDpwzRGkx8g= 2024-04-12T04:56:00.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y2uarvryy8SHjdJtlDpwzRGkx8g='} 2024-04-12T04:56:00.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c27ab103000f0bd7-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.218 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.218 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.218 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.219 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.219 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.219 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T04:56:00.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T04:56:00.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005240687dbc180963-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj42 2024-04-12T04:56:00.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj42 2024-04-12T04:56:00.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj42 2024-04-12T04:56:00.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.220 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.220 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.220 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.220 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.220 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj42 2024-04-12T04:56:00.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.220 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:l/YoM2VQHWcxIQLRZ4i99q+ju2s= 2024-04-12T04:56:00.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l/YoM2VQHWcxIQLRZ4i99q+ju2s='} 2024-04-12T04:56:00.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a7a8e45de3d5c3b-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.221 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.221 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.221 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.221 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.222 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.222 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T04:56:00.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T04:56:00.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a61f10d5ea9b1772-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj43 2024-04-12T04:56:00.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj43 2024-04-12T04:56:00.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj43 2024-04-12T04:56:00.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.223 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.223 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.223 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.223 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.223 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj43 2024-04-12T04:56:00.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.223 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6lIxmEh3YhFhAlmV9vYBx3mQgDk= 2024-04-12T04:56:00.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6lIxmEh3YhFhAlmV9vYBx3mQgDk='} 2024-04-12T04:56:00.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a79a620272d662c-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.224 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.224 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.225 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.225 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.225 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.225 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T04:56:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T04:56:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ea0164165abad26-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj44 2024-04-12T04:56:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj44 2024-04-12T04:56:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj44 2024-04-12T04:56:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.226 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.226 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.226 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.226 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.226 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj44 2024-04-12T04:56:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.226 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:DGr4zG6HCxc995i0C6j3e77l5BM= 2024-04-12T04:56:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:DGr4zG6HCxc995i0C6j3e77l5BM='} 2024-04-12T04:56:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f6af4d5f83480b46-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.227 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.227 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.227 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.227 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.227 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.228 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T04:56:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T04:56:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000118d8aff010afe5d-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj45 2024-04-12T04:56:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj45 2024-04-12T04:56:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj45 2024-04-12T04:56:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.229 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.229 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.229 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.229 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.229 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj45 2024-04-12T04:56:00.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.229 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:YOIT4pZKtkIvNLGAxvv9kvnrXwU= 2024-04-12T04:56:00.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YOIT4pZKtkIvNLGAxvv9kvnrXwU='} 2024-04-12T04:56:00.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000099aad73257837543-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.230 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.230 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.230 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.230 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.230 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.230 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T04:56:00.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T04:56:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f63b15f269533e4d-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj46 2024-04-12T04:56:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj46 2024-04-12T04:56:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj46 2024-04-12T04:56:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.231 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.231 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.231 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.231 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.232 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj46 2024-04-12T04:56:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.232 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:kpv+KU73udWLB5pj8OLkclxixFQ= 2024-04-12T04:56:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kpv+KU73udWLB5pj8OLkclxixFQ='} 2024-04-12T04:56:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e59f2bcb536f4758-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.233 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.233 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.233 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.233 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.233 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.233 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T04:56:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T04:56:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d839c2f40f78218-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj47 2024-04-12T04:56:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj47 2024-04-12T04:56:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj47 2024-04-12T04:56:00.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.234 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.234 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.234 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.234 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.234 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj47 2024-04-12T04:56:00.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.234 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:uVzhM9ouvg+OBo74hifbl3tXEUI= 2024-04-12T04:56:00.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uVzhM9ouvg+OBo74hifbl3tXEUI='} 2024-04-12T04:56:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7f985fa5db3ba67-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.235 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.235 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.235 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.236 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.236 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.236 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T04:56:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T04:56:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a74bc216203e2f8-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj48 2024-04-12T04:56:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj48 2024-04-12T04:56:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj48 2024-04-12T04:56:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.237 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.237 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.237 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.237 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.237 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj48 2024-04-12T04:56:00.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.237 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:VKlsaUBc5SvMyT7KJfE5NZRowG4= 2024-04-12T04:56:00.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VKlsaUBc5SvMyT7KJfE5NZRowG4='} 2024-04-12T04:56:00.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d92a3ac7625eb87-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/ 2024-04-12T04:56:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/ 2024-04-12T04:56:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/ 2024-04-12T04:56:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.238 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.238 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.238 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.239 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.239 INFO:tasks.rgw_multisite_tests:/dkyges-11/ 2024-04-12T04:56:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.239 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug= 2024-04-12T04:56:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NFg4Gvu8aVW3WV1f3N1zXd31mug=', 'Content-Length': '0'} 2024-04-12T04:56:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000080cb53e446b63b8e-006618a850-110f-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-11/obj49 2024-04-12T04:56:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-11/obj49 2024-04-12T04:56:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-11/obj49 2024-04-12T04:56:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.240 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.240 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.240 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.240 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.240 INFO:tasks.rgw_multisite_tests:/dkyges-11/obj49 2024-04-12T04:56:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.240 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ZZLdNKc1LHvY51slZRdSY6ElTRU= 2024-04-12T04:56:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZZLdNKc1LHvY51slZRdSY6ElTRU='} 2024-04-12T04:56:00.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007b5953beddaeb8fa-006618a850-110f-a1'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.241 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.242 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.242 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.242 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.242 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.242 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003b32606363e38747-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj1 2024-04-12T04:56:00.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj1 2024-04-12T04:56:00.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj1 2024-04-12T04:56:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.243 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.243 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.243 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.243 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.243 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj1 2024-04-12T04:56:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.244 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NqXG9TTTiseiusyRhR1rwfOFDXY= 2024-04-12T04:56:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NqXG9TTTiseiusyRhR1rwfOFDXY='} 2024-04-12T04:56:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009a04ea9e897d56d7-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.245 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.245 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.245 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.245 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.245 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.245 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cad09d2ee0bfdd1c-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj2 2024-04-12T04:56:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj2 2024-04-12T04:56:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj2 2024-04-12T04:56:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.246 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.247 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.247 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.247 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.247 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj2 2024-04-12T04:56:00.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.247 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:kn4mUiNbzJluDTec/6GIqFaT+J0= 2024-04-12T04:56:00.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kn4mUiNbzJluDTec/6GIqFaT+J0='} 2024-04-12T04:56:00.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004578f5dc7ac83518-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.248 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.248 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.248 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.248 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.248 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.248 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2f3ae30ac969463-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj3 2024-04-12T04:56:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj3 2024-04-12T04:56:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj3 2024-04-12T04:56:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.249 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.250 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.250 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.250 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.250 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj3 2024-04-12T04:56:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.250 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:RKwbP1p6J+FnENsO8d6uDV1wdSs= 2024-04-12T04:56:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RKwbP1p6J+FnENsO8d6uDV1wdSs='} 2024-04-12T04:56:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2c5f486f95c8896-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.251 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.251 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.251 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.251 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.251 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.251 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000437b0a6cf7e81652-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj4 2024-04-12T04:56:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj4 2024-04-12T04:56:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj4 2024-04-12T04:56:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.252 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.252 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.253 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.253 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.253 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj4 2024-04-12T04:56:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.253 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Y7ya2D8d41f3AgpUy4bE0/SZERc= 2024-04-12T04:56:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y7ya2D8d41f3AgpUy4bE0/SZERc='} 2024-04-12T04:56:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3f98b3278607e04-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.254 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.254 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.254 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.254 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.254 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.254 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe779de6c464b0ed-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj5 2024-04-12T04:56:00.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj5 2024-04-12T04:56:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj5 2024-04-12T04:56:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.255 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.255 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.255 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.255 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.255 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj5 2024-04-12T04:56:00.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.255 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:z/6jw5JCXFD/ab8nk7vl/YPw5dE= 2024-04-12T04:56:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z/6jw5JCXFD/ab8nk7vl/YPw5dE='} 2024-04-12T04:56:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006120456b16db7dcc-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.256 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.256 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.257 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.257 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.257 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.257 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e04a1fd143b8dd9c-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj6 2024-04-12T04:56:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj6 2024-04-12T04:56:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj6 2024-04-12T04:56:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.258 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.258 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.258 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.258 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.258 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj6 2024-04-12T04:56:00.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.258 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:qpXvnTY42Jhx+1LAv2Zv6hyS8Y4= 2024-04-12T04:56:00.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qpXvnTY42Jhx+1LAv2Zv6hyS8Y4='} 2024-04-12T04:56:00.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ff1220d0020d0c2-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.259 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.259 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.259 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.259 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.259 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.260 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed0d3ba74b5d788f-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj7 2024-04-12T04:56:00.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj7 2024-04-12T04:56:00.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj7 2024-04-12T04:56:00.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.261 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.261 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.261 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.261 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.261 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj7 2024-04-12T04:56:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.261 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:7nwG1CznDOPhxLJNwoa3Si3jxIY= 2024-04-12T04:56:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7nwG1CznDOPhxLJNwoa3Si3jxIY='} 2024-04-12T04:56:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000686fa4dbd1a3ad50-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.262 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.262 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.262 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.262 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.262 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.262 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007db1c156cb533152-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj8 2024-04-12T04:56:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj8 2024-04-12T04:56:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj8 2024-04-12T04:56:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.263 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.263 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.263 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.263 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.263 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj8 2024-04-12T04:56:00.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.264 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:j86hssB4cD5ntc1FlvFrovbizog= 2024-04-12T04:56:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j86hssB4cD5ntc1FlvFrovbizog='} 2024-04-12T04:56:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f4b02bc46156016c-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.264 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.264 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.265 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.265 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.265 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.265 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a99c514204e3b6d-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj9 2024-04-12T04:56:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj9 2024-04-12T04:56:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj9 2024-04-12T04:56:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.266 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.266 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.266 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.266 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.266 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj9 2024-04-12T04:56:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.266 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:CsXKGfbrEIDSQUYZesNFbEOg2LY= 2024-04-12T04:56:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CsXKGfbrEIDSQUYZesNFbEOg2LY='} 2024-04-12T04:56:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6739e934794ce02-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.267 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.267 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.267 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.267 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.267 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.267 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053bc65069c28e3dd-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj10 2024-04-12T04:56:00.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj10 2024-04-12T04:56:00.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj10 2024-04-12T04:56:00.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.268 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.268 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.269 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.269 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.269 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj10 2024-04-12T04:56:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.269 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Z3k7JLDyyosv5ON58EeA0n9ElK0= 2024-04-12T04:56:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Z3k7JLDyyosv5ON58EeA0n9ElK0='} 2024-04-12T04:56:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007093bc2223fb838b-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.270 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.270 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.270 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.270 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.270 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.270 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b34b205ac548b067-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj11 2024-04-12T04:56:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj11 2024-04-12T04:56:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj11 2024-04-12T04:56:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.271 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.271 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.272 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.272 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.272 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj11 2024-04-12T04:56:00.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.272 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:twBAwyacwShaY98wwdpx5PZfRfo= 2024-04-12T04:56:00.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:twBAwyacwShaY98wwdpx5PZfRfo='} 2024-04-12T04:56:00.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020e1619756171fab-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.273 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.273 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.273 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.273 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.273 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.273 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000060095ff90ed3d3ba-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj12 2024-04-12T04:56:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj12 2024-04-12T04:56:00.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj12 2024-04-12T04:56:00.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.274 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.274 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.274 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.274 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.274 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj12 2024-04-12T04:56:00.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.275 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:8YgdE6UHNUyd0w6UYsG12U7e5S4= 2024-04-12T04:56:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8YgdE6UHNUyd0w6UYsG12U7e5S4='} 2024-04-12T04:56:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000895d92c39728cf74-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.275 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.276 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.276 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.276 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.276 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.276 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e77a58ee8506603-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj13 2024-04-12T04:56:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj13 2024-04-12T04:56:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj13 2024-04-12T04:56:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.277 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.277 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.277 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.277 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.277 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj13 2024-04-12T04:56:00.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.277 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/mgLGh0Wy/jMG/vJWrLAzqC220A= 2024-04-12T04:56:00.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/mgLGh0Wy/jMG/vJWrLAzqC220A='} 2024-04-12T04:56:00.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1e3b2ce105470ea-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.278 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.278 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.278 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.278 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.278 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.279 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000473e0f92d9e9c784-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj14 2024-04-12T04:56:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj14 2024-04-12T04:56:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj14 2024-04-12T04:56:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.280 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.280 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.280 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.280 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.280 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj14 2024-04-12T04:56:00.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.280 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:KqN+ImU72TahQ+RMjSf0cRBIdI4= 2024-04-12T04:56:00.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KqN+ImU72TahQ+RMjSf0cRBIdI4='} 2024-04-12T04:56:00.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a9b98cce82b31a79-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.281 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.281 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.281 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.281 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.281 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.281 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b99d142388b8c857-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj15 2024-04-12T04:56:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj15 2024-04-12T04:56:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj15 2024-04-12T04:56:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.282 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.282 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.282 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.283 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.283 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj15 2024-04-12T04:56:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.283 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:tz+wUWYrGaMtZkBuVCZ3ktd9KE8= 2024-04-12T04:56:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tz+wUWYrGaMtZkBuVCZ3ktd9KE8='} 2024-04-12T04:56:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000022668c866b8dd750-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.284 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.284 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.284 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.284 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.284 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.284 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c87af6cfdfdb2641-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj16 2024-04-12T04:56:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj16 2024-04-12T04:56:00.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj16 2024-04-12T04:56:00.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.285 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.285 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.285 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.285 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.285 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj16 2024-04-12T04:56:00.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.286 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:SNlfw4b4yFKqO5Oj6qebvFxGGqw= 2024-04-12T04:56:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SNlfw4b4yFKqO5Oj6qebvFxGGqw='} 2024-04-12T04:56:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000323ff521c0edf8ab-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.287 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.287 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.287 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.287 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.287 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.287 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003fd6e4006acaee6c-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj17 2024-04-12T04:56:00.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj17 2024-04-12T04:56:00.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj17 2024-04-12T04:56:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.288 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.288 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.288 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.288 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.288 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj17 2024-04-12T04:56:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.288 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:xsXRAXWkQjQnaYjuDeXvBJYEZ5Q= 2024-04-12T04:56:00.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xsXRAXWkQjQnaYjuDeXvBJYEZ5Q='} 2024-04-12T04:56:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000462883b2831bb7a8-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.290 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.290 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.290 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.290 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.290 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.290 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g= 2024-04-12T04:56:00.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dB7gRSxwhV1DlHsBB9+t4dWdB+g=', 'Content-Length': '0'} 2024-04-12T04:56:00.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c314fc2da11f863e-006618a850-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:44 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj18 2024-04-12T04:56:00.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj18 2024-04-12T04:56:00.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj18 2024-04-12T04:56:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.291 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.291 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.291 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.291 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:44 GMT 2024-04-12T04:56:00.291 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj18 2024-04-12T04:56:00.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.291 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:uzncsSU6PMmtVmUlengPSxOT2Ck= 2024-04-12T04:56:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:44 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uzncsSU6PMmtVmUlengPSxOT2Ck='} 2024-04-12T04:56:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c98697e620a5f9af-006618a850-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.293 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.293 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.293 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.293 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.293 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.293 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c267a2513a07b64f-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj19 2024-04-12T04:56:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj19 2024-04-12T04:56:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj19 2024-04-12T04:56:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.294 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.294 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.294 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.294 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.294 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj19 2024-04-12T04:56:00.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.294 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:nvyB353nwTv+w3JVHf4Fca5/iSo= 2024-04-12T04:56:00.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nvyB353nwTv+w3JVHf4Fca5/iSo='} 2024-04-12T04:56:00.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e5ef649277adc7d6-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.295 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.295 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.295 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.296 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.296 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.296 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba893e8c26b0c6a4-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj20 2024-04-12T04:56:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj20 2024-04-12T04:56:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj20 2024-04-12T04:56:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.297 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.297 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.297 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.297 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.297 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj20 2024-04-12T04:56:00.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.297 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Y9zQJCZujqg13DTvCeVlvAUUiUI= 2024-04-12T04:56:00.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Y9zQJCZujqg13DTvCeVlvAUUiUI='} 2024-04-12T04:56:00.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002c46c72f846392f1-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.298 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.298 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.298 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.298 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.298 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.298 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6deff5a7924ef05-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj21 2024-04-12T04:56:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj21 2024-04-12T04:56:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj21 2024-04-12T04:56:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.299 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.299 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.299 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.300 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.300 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj21 2024-04-12T04:56:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.300 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:c5hKndp2fes7ygUdNPvc7rC8Ioo= 2024-04-12T04:56:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:c5hKndp2fes7ygUdNPvc7rC8Ioo='} 2024-04-12T04:56:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da969076bb76181a-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.301 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.301 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.301 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.301 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.301 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.301 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000845e91f8a80aaea6-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj22 2024-04-12T04:56:00.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj22 2024-04-12T04:56:00.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj22 2024-04-12T04:56:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.302 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.302 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.302 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.302 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.302 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj22 2024-04-12T04:56:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.302 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:9BSXkbaWjGT+2ydVfatrHb8YNEE= 2024-04-12T04:56:00.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9BSXkbaWjGT+2ydVfatrHb8YNEE='} 2024-04-12T04:56:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e18f00a960b751d-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.303 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.303 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.303 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.304 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.304 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.304 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b1daf2617354b84e-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj23 2024-04-12T04:56:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj23 2024-04-12T04:56:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj23 2024-04-12T04:56:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.305 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.305 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.305 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.305 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.305 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj23 2024-04-12T04:56:00.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.305 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:i8rIss1OnXwdsIcFdSSmScFEvJ4= 2024-04-12T04:56:00.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:i8rIss1OnXwdsIcFdSSmScFEvJ4='} 2024-04-12T04:56:00.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d54f5c174537b5b-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.306 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.306 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.306 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.306 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.306 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.306 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c4782f0fcad5e2fe-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj24 2024-04-12T04:56:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj24 2024-04-12T04:56:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj24 2024-04-12T04:56:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.307 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.307 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.307 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.307 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.308 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj24 2024-04-12T04:56:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.308 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:O20hFh9zkb+YVzxYSyfKh1FITOU= 2024-04-12T04:56:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:O20hFh9zkb+YVzxYSyfKh1FITOU='} 2024-04-12T04:56:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bab2753849949886-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.309 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.309 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.309 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.309 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.309 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.309 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007bad5f2177cd6b49-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj25 2024-04-12T04:56:00.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj25 2024-04-12T04:56:00.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj25 2024-04-12T04:56:00.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.310 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.310 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.311 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.311 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.311 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj25 2024-04-12T04:56:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.311 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ap37GXl6irP3oZ8/uj4QGBWJsaw= 2024-04-12T04:56:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ap37GXl6irP3oZ8/uj4QGBWJsaw='} 2024-04-12T04:56:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f2162cf1d36c9e68-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.312 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.312 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.312 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.312 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.312 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.312 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058f70e06a39904e0-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj26 2024-04-12T04:56:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj26 2024-04-12T04:56:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj26 2024-04-12T04:56:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.313 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.314 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.314 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.314 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.314 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj26 2024-04-12T04:56:00.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.314 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:AgK9gY1AkqZ8bZBpcQcWGIErqJY= 2024-04-12T04:56:00.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AgK9gY1AkqZ8bZBpcQcWGIErqJY='} 2024-04-12T04:56:00.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b15d61180dc3995-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.315 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.315 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.315 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.315 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.315 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.315 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001430c251c9887810-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj27 2024-04-12T04:56:00.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj27 2024-04-12T04:56:00.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj27 2024-04-12T04:56:00.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.316 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.316 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.316 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.316 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.317 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj27 2024-04-12T04:56:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.317 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ltN3GCI8OHX43+aZAUIbRvqnwQE= 2024-04-12T04:56:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ltN3GCI8OHX43+aZAUIbRvqnwQE='} 2024-04-12T04:56:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008b85fecde6216c34-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.318 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.318 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.318 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.318 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.318 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.318 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047832f644d2d2eda-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj28 2024-04-12T04:56:00.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj28 2024-04-12T04:56:00.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj28 2024-04-12T04:56:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.319 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.319 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.319 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.319 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.319 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj28 2024-04-12T04:56:00.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.319 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:owqDEIp4Gv7htdP8uv0DeGNZQ4s= 2024-04-12T04:56:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:owqDEIp4Gv7htdP8uv0DeGNZQ4s='} 2024-04-12T04:56:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008bb88dce53ec040d-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.320 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.320 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.321 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.321 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.321 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.321 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b09757187d55447-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj29 2024-04-12T04:56:00.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj29 2024-04-12T04:56:00.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj29 2024-04-12T04:56:00.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.322 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.322 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.322 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.322 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.322 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj29 2024-04-12T04:56:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.322 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:aXSlDvYwgMWShTLniheZXWJU3r0= 2024-04-12T04:56:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:aXSlDvYwgMWShTLniheZXWJU3r0='} 2024-04-12T04:56:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085a5b7beeef02ed7-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.323 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.323 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.323 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.323 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.323 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.324 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000343dbe6432472f6f-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj30 2024-04-12T04:56:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj30 2024-04-12T04:56:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj30 2024-04-12T04:56:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.325 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.325 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.325 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.325 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.325 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj30 2024-04-12T04:56:00.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.325 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:OHd7dc54/OpdVlDxXZgg1L/1l6w= 2024-04-12T04:56:00.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OHd7dc54/OpdVlDxXZgg1L/1l6w='} 2024-04-12T04:56:00.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000071519a8983acc4b9-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.326 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.326 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.326 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.326 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.327 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.327 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005d61a8c3f79981a-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj31 2024-04-12T04:56:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj31 2024-04-12T04:56:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj31 2024-04-12T04:56:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.328 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.328 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.328 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.328 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.328 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj31 2024-04-12T04:56:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.328 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:p+T322FY82AvXYzG69wv8pWEssk= 2024-04-12T04:56:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p+T322FY82AvXYzG69wv8pWEssk='} 2024-04-12T04:56:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003d9d98c774bd02aa-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.329 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.329 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.329 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.329 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.329 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.329 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '186'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc8f1b603c642097-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj32 2024-04-12T04:56:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj32 2024-04-12T04:56:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj32 2024-04-12T04:56:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.330 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.330 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.330 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.331 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.331 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj32 2024-04-12T04:56:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.331 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:cqfKYNRSOoW4Egop18iOBHAy4wg= 2024-04-12T04:56:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cqfKYNRSOoW4Egop18iOBHAy4wg='} 2024-04-12T04:56:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000023d6ebb2d6bd2127-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.332 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.332 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.332 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.332 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.332 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.332 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '192'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078d95d7829ab7289-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj33 2024-04-12T04:56:00.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj33 2024-04-12T04:56:00.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj33 2024-04-12T04:56:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.333 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.333 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.333 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.333 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.333 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj33 2024-04-12T04:56:00.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.334 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:exZaFqkuNjsJbbBJNhLIFohEI9g= 2024-04-12T04:56:00.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:exZaFqkuNjsJbbBJNhLIFohEI9g='} 2024-04-12T04:56:00.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094d4a6a3d6c07b9c-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.335 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.335 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.335 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.335 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.335 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.335 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '198'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ace9e053a98444d-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj34 2024-04-12T04:56:00.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj34 2024-04-12T04:56:00.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj34 2024-04-12T04:56:00.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.336 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.336 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.336 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.336 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.336 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj34 2024-04-12T04:56:00.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.337 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:nzinLDtf9DfVqesRnRPTeYRw7SQ= 2024-04-12T04:56:00.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nzinLDtf9DfVqesRnRPTeYRw7SQ='} 2024-04-12T04:56:00.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f7530560f9da504-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.338 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.338 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.338 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.338 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.338 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.338 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '204'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b24cc1c8d15ca32e-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj35 2024-04-12T04:56:00.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj35 2024-04-12T04:56:00.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj35 2024-04-12T04:56:00.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.339 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.339 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.339 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.339 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.340 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj35 2024-04-12T04:56:00.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.340 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Is/FZorg5Ljp3ebk8dDMUL72Edc= 2024-04-12T04:56:00.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Is/FZorg5Ljp3ebk8dDMUL72Edc='} 2024-04-12T04:56:00.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b64089e235398b1b-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.341 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.341 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.341 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.341 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.341 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.341 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '210'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1a29ffd70e46fb5-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj36 2024-04-12T04:56:00.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj36 2024-04-12T04:56:00.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj36 2024-04-12T04:56:00.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.342 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.342 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.342 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.342 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.343 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj36 2024-04-12T04:56:00.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.343 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:pTFA+vF1Ux3LJeKx91c/OT8qaQ8= 2024-04-12T04:56:00.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pTFA+vF1Ux3LJeKx91c/OT8qaQ8='} 2024-04-12T04:56:00.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bce603ed4e827a1a-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.344 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.344 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.344 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.344 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.344 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.344 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '216'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000058f7feb79f092b0b-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj37 2024-04-12T04:56:00.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj37 2024-04-12T04:56:00.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj37 2024-04-12T04:56:00.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.345 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.345 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.345 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.345 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.345 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj37 2024-04-12T04:56:00.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.346 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:9aSzN3R3R4uXNL8YM7aMk2Fq/D0= 2024-04-12T04:56:00.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9aSzN3R3R4uXNL8YM7aMk2Fq/D0='} 2024-04-12T04:56:00.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df166aa675117cf2-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.347 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.347 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.347 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.347 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.347 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.347 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '222'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e67ddfdb89f3c682-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj38 2024-04-12T04:56:00.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj38 2024-04-12T04:56:00.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj38 2024-04-12T04:56:00.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.348 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.348 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.348 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.348 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.348 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj38 2024-04-12T04:56:00.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.349 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:1oo2dpv3oIfZ2B11/cpe8FtEk/0= 2024-04-12T04:56:00.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1oo2dpv3oIfZ2B11/cpe8FtEk/0='} 2024-04-12T04:56:00.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000360effeecd4e02d5-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.350 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.350 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.350 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.350 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.350 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.350 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '228'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd4e6dc6fadbf39a-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj39 2024-04-12T04:56:00.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj39 2024-04-12T04:56:00.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj39 2024-04-12T04:56:00.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.351 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.351 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.351 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.351 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.351 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj39 2024-04-12T04:56:00.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.352 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Sf2hawxvEl7wZV00DFaLMrV2ecs= 2024-04-12T04:56:00.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Sf2hawxvEl7wZV00DFaLMrV2ecs='} 2024-04-12T04:56:00.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000024945f0cff87c993-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.352 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.353 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.353 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.353 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.353 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.353 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '234'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cef590edfc291392-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj40 2024-04-12T04:56:00.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj40 2024-04-12T04:56:00.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj40 2024-04-12T04:56:00.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.354 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.354 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.354 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.354 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.354 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj40 2024-04-12T04:56:00.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.354 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:50nY986B8eWq42wyHAztEc6TjEU= 2024-04-12T04:56:00.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:50nY986B8eWq42wyHAztEc6TjEU='} 2024-04-12T04:56:00.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000029c8ad4244ab893b-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.355 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.355 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.355 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.356 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.356 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.356 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '240'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006c8c18b311247453-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj41 2024-04-12T04:56:00.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj41 2024-04-12T04:56:00.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj41 2024-04-12T04:56:00.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.357 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.357 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.357 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.357 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.357 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj41 2024-04-12T04:56:00.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.357 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:vbwJ8fz9vwHCVMo952IYVtHS4vs= 2024-04-12T04:56:00.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vbwJ8fz9vwHCVMo952IYVtHS4vs='} 2024-04-12T04:56:00.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca5df6d6947c7d91-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.358 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.359 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.359 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.359 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.359 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.359 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '246'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009e083510618935e9-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj42 2024-04-12T04:56:00.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj42 2024-04-12T04:56:00.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj42 2024-04-12T04:56:00.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.360 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.360 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.360 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.360 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.360 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj42 2024-04-12T04:56:00.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.360 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:nrrWfhYFr+zPkceLsIYn7y2EpMc= 2024-04-12T04:56:00.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nrrWfhYFr+zPkceLsIYn7y2EpMc='} 2024-04-12T04:56:00.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c0ef858f56981eb9-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.361 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.361 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.362 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.362 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.362 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.362 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '252'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007408ba6201e7c597-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj43 2024-04-12T04:56:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj43 2024-04-12T04:56:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj43 2024-04-12T04:56:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.363 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.363 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.363 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.363 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.363 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj43 2024-04-12T04:56:00.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.363 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sy9tvttK0/k0mansIzzUj7AvYbE= 2024-04-12T04:56:00.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sy9tvttK0/k0mansIzzUj7AvYbE='} 2024-04-12T04:56:00.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000049ccb211144e0eb-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.364 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.364 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.364 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.364 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.364 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.364 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '258'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2c1b7294cbcf9a3-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj44 2024-04-12T04:56:00.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj44 2024-04-12T04:56:00.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj44 2024-04-12T04:56:00.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.365 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.366 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.366 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.366 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.366 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj44 2024-04-12T04:56:00.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.366 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:JIxp7a9T4lMF86xr9LTqnaISSbI= 2024-04-12T04:56:00.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JIxp7a9T4lMF86xr9LTqnaISSbI='} 2024-04-12T04:56:00.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000467cc0e936c84d3f-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.367 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.367 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.367 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.367 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.367 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.367 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '264'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f070d5cd3e097d05-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj45 2024-04-12T04:56:00.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj45 2024-04-12T04:56:00.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj45 2024-04-12T04:56:00.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.368 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.368 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.369 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.369 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.369 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj45 2024-04-12T04:56:00.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.369 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ucJWZFAXb4fhtRRfzbKvdmlgSTU= 2024-04-12T04:56:00.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ucJWZFAXb4fhtRRfzbKvdmlgSTU='} 2024-04-12T04:56:00.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ab46265467e70ecc-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.370 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.370 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.370 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.370 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.370 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.370 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '270'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000241f540bf91117d1-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj46 2024-04-12T04:56:00.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj46 2024-04-12T04:56:00.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj46 2024-04-12T04:56:00.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.371 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.371 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.371 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.371 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.372 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj46 2024-04-12T04:56:00.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.372 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:fv38QD/jF7ZcuFs3zAO5WyoMpOs= 2024-04-12T04:56:00.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fv38QD/jF7ZcuFs3zAO5WyoMpOs='} 2024-04-12T04:56:00.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d762c43accfa5779-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.373 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.373 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.373 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.373 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.373 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.373 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '276'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ff32f460d7edb6e-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj47 2024-04-12T04:56:00.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj47 2024-04-12T04:56:00.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj47 2024-04-12T04:56:00.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.374 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.374 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.374 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.374 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.374 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj47 2024-04-12T04:56:00.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.374 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:z6AH6QjWWAzOTkbFHcAc+PlNsUY= 2024-04-12T04:56:00.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:z6AH6QjWWAzOTkbFHcAc+PlNsUY='} 2024-04-12T04:56:00.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042034defc5d76009-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.377 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.377 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.377 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.377 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.377 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.377 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '282'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd9264e3f5f549ab-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj48 2024-04-12T04:56:00.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj48 2024-04-12T04:56:00.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj48 2024-04-12T04:56:00.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.378 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.379 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.379 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.379 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.379 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj48 2024-04-12T04:56:00.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.379 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:BuyfECLpa96bhAWIf0/KqCE6tN8= 2024-04-12T04:56:00.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BuyfECLpa96bhAWIf0/KqCE6tN8='} 2024-04-12T04:56:00.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000213dab6a2a8b7008-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/ 2024-04-12T04:56:00.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/ 2024-04-12T04:56:00.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/ 2024-04-12T04:56:00.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.380 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.380 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.380 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.380 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.380 INFO:tasks.rgw_multisite_tests:/dkyges-12/ 2024-04-12T04:56:00.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.380 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU= 2024-04-12T04:56:00.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:d1z6G/MT20cj/qUiG7QCAveu0TU=', 'Content-Length': '0'} 2024-04-12T04:56:00.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '288'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a73cb569e864675f-006618a851-1113-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-12/obj49 2024-04-12T04:56:00.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-12/obj49 2024-04-12T04:56:00.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-12/obj49 2024-04-12T04:56:00.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:00.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.381 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.381 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:00.381 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.381 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:19:45 GMT 2024-04-12T04:56:00.381 INFO:tasks.rgw_multisite_tests:/dkyges-12/obj49 2024-04-12T04:56:00.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.381 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ST+8b+8ZknZL/7ij++g87iDUV9k= 2024-04-12T04:56:00.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:19:45 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ST+8b+8ZknZL/7ij++g87iDUV9k='} 2024-04-12T04:56:00.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c6469e73f3828566-006618a851-1113-a2'), ('Date', 'Fri, 12 Apr 2024 03:19:45 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.382 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:56:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: [ 2024-04-12T04:56:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:56:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "", 2024-04-12T04:56:00.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "0.000000" 2024-04-12T04:56:00.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:56:00.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T04:56:00.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891982.967859_56.1", 2024-04-12T04:56:00.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T03:19:42.967859Z" 2024-04-12T04:56:00.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712891982.967859_56.1'} 2024-04-12T04:56:00.383 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-12T04:56:00.383 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:19:46.118+0000 7f050ea7e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:19:46.119+0000 7f050ea7e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:00.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "", 2024-04-12T04:56:00.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:00.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:56:00.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:00.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:56:00.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:56:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891982.967859_56.1", 2024-04-12T04:56:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:00.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:19:42.967859Z", 2024-04-12T04:56:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T03:19:46Z" 2024-04-12T04:56:00.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.389 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:00.389 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:00.389 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:00.389 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:00.389 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:00.389 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:00.389 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.389 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.389 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:00.389 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.389 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:00.389 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.389 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.390 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:00.390 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.390 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:00.390 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.390 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:00.390 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.390 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.390 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.390 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.390 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:00.390 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.390 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.391 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:56:00.391 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.391 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:00.391 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.391 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:56:00.391 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.391 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.391 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.391 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.391 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:00.391 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.391 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.391 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:56:00.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.392 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:00.392 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.392 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:56:00.392 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.392 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.392 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.392 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.392 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:00.392 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.392 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.392 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891982.967859_56.1", 2024-04-12T04:56:00.392 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.392 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:00.393 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.393 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:19:42.967859Z", 2024-04-12T04:56:00.393 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.393 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.393 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.393 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:00.393 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.393 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:00.393 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:00.393 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:00.393 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.393 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:19:46Z" 2024-04-12T04:56:00.393 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:00.393 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712891982.967859_56.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:19:42.967859Z', 'realm_epoch': 2}}] 2024-04-12T04:56:00.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712891982.967859_56.1'} 2024-04-12T04:56:00.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712891982.967859_56.1'} 2024-04-12T04:56:00.394 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-12T04:56:00.394 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:00.394 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:00.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:56:00.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:19:46.241+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:19:51.242+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:19:56.242+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:20:01.243+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:20:06.244+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:20:11.245+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:20:16.246+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:20:21.245+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:20:26.246+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:20:31.247+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:20:36.247+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:20:41.248+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:20:46.249+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:20:51.250+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:20:56.250+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:21:01.251+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:21:06.251+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:21:11.252+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:21:16.253+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:21:21.254+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:21:26.254+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:21:31.255+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:21:36.255+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:21:41.256+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:21:46.257+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:21:51.257+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:21:56.258+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:22:01.259+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:22:06.259+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:22:11.260+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:22:16.260+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:22:21.261+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:22:26.262+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:22:31.262+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:22:36.263+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:22:41.263+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:22:46.265+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:22:51.265+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:22:56.266+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:23:01.266+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:23:06.267+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:23:11.268+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:23:16.269+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:23:21.269+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:23:26.270+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:23:31.271+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:23:36.271+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:23:41.272+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:23:46.273+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:23:51.273+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:23:56.274+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:24:01.276+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:24:06.275+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:24:11.276+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:24:16.277+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:24:21.277+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:24:26.278+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:24:31.279+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:24:36.279+0000 7f1275b0f780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:24:41.280+0000 7f1275b0f780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-12T04:56:00.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:24:41.280+0000 7f1275b0f780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:56:00.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:24:41.280+0000 7f1275b0f780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:56:00.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-12T04:56:00.399 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-12T04:56:00.399 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.399 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-12T04:56:00.399 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2024-04-12T04:56:00.399 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-12T04:56:00.400 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-12T04:56:00.401 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-12T04:56:00.401 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-12T04:56:00.401 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1451, in test_bucket_reshard_index_log_trim 2024-04-12T04:56:00.401 INFO:tasks.rgw_multisite_tests: assert(len(json_obj_5['layout']['logs']) == 1) 2024-04-12T04:56:00.401 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-12T04:56:00.401 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-12T04:56:00.401 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=dkyges-57 2024-04-12T04:56:00.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/ 2024-04-12T04:56:00.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.402 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.402 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.402 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.402 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:21 GMT 2024-04-12T04:56:00.402 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.402 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:V4exqninCVTDFoP74elA6MPv0bo= 2024-04-12T04:56:00.402 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V4exqninCVTDFoP74elA6MPv0bo=', 'Content-Length': '0'} 2024-04-12T04:56:00.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000006e0df52cf4809f23-006618af7d-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/ 2024-04-12T04:56:00.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.403 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.403 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.403 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.403 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.404 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:21 GMT 2024-04-12T04:56:00.404 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.404 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:BTTHEelLQseAUKlARavSSnUsiWw= 2024-04-12T04:56:00.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BTTHEelLQseAUKlARavSSnUsiWw=', 'Content-Length': '0'} 2024-04-12T04:56:00.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000caf9acd36f9f9053-006618af7d-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a 2024-04-12T04:56:00.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:00.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.404 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.405 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.405 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:00.405 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.405 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:21 GMT 2024-04-12T04:56:00.405 INFO:tasks.rgw_multisite_tests:/dkyges-57/a 2024-04-12T04:56:00.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.405 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:CMrvJ8HERNzTGgXT+a5FBetggow= 2024-04-12T04:56:00.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:CMrvJ8HERNzTGgXT+a5FBetggow='} 2024-04-12T04:56:00.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000079c7ff4f8b7740a3-006618af7d-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.405 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/ 2024-04-12T04:56:00.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.406 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.406 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.406 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.406 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.407 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:21 GMT 2024-04-12T04:56:00.407 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.407 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:BTTHEelLQseAUKlARavSSnUsiWw= 2024-04-12T04:56:00.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BTTHEelLQseAUKlARavSSnUsiWw=', 'Content-Length': '0'} 2024-04-12T04:56:00.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001aac41545b397cef-006618af7d-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b 2024-04-12T04:56:00.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:00.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.407 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.408 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.408 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:00.408 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.408 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:21 GMT 2024-04-12T04:56:00.408 INFO:tasks.rgw_multisite_tests:/dkyges-57/b 2024-04-12T04:56:00.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.408 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:qhWXqgWtUAZ1z/C2LfJarUTJNqk= 2024-04-12T04:56:00.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qhWXqgWtUAZ1z/C2LfJarUTJNqk='} 2024-04-12T04:56:00.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005bcd98c13e9a9132-006618af7d-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.408 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/ 2024-04-12T04:56:00.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.409 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.409 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.409 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.409 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:21 GMT 2024-04-12T04:56:00.409 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.409 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.410 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:BTTHEelLQseAUKlARavSSnUsiWw= 2024-04-12T04:56:00.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BTTHEelLQseAUKlARavSSnUsiWw=', 'Content-Length': '0'} 2024-04-12T04:56:00.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007050fcce68c87aa4-006618af7d-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c 2024-04-12T04:56:00.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:00.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.410 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.411 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.411 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:00.411 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.411 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:21 GMT 2024-04-12T04:56:00.411 INFO:tasks.rgw_multisite_tests:/dkyges-57/c 2024-04-12T04:56:00.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.411 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:cW+tN0DQ1DZ5ZOpesFa5x3Vk8PQ= 2024-04-12T04:56:00.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cW+tN0DQ1DZ5ZOpesFa5x3Vk8PQ='} 2024-04-12T04:56:00.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002736c0cbda7c88d7-006618af7d-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.411 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/ 2024-04-12T04:56:00.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.412 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.412 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.412 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.412 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.412 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:21 GMT 2024-04-12T04:56:00.412 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.413 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:BTTHEelLQseAUKlARavSSnUsiWw= 2024-04-12T04:56:00.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BTTHEelLQseAUKlARavSSnUsiWw=', 'Content-Length': '0'} 2024-04-12T04:56:00.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000902eb3b89379bc44-006618af7d-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d 2024-04-12T04:56:00.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:00.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.413 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.414 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.414 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:00.414 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.414 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:21 GMT 2024-04-12T04:56:00.414 INFO:tasks.rgw_multisite_tests:/dkyges-57/d 2024-04-12T04:56:00.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.414 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:rcgy2JrsO1NqV23yPl3X/MAI7Ro= 2024-04-12T04:56:00.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:21 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rcgy2JrsO1NqV23yPl3X/MAI7Ro='} 2024-04-12T04:56:00.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005cee5cffd67f47eb-006618af7d-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.414 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.415 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:00.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:56:00.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: [ 2024-04-12T04:56:00.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:00.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:56:00.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "", 2024-04-12T04:56:00.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "0.000000" 2024-04-12T04:56:00.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:56:00.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T03:18:45.655973Z" 2024-04-12T04:56:00.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712893821.196084_497.1", 2024-04-12T04:56:00.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T03:50:21.196084Z" 2024-04-12T04:56:00.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893821.196084_497.1'} 2024-04-12T04:56:00.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-12T04:56:00.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:50:21.531+0000 7fb2a6c59780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:50:21.532+0000 7fb2a6c59780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:00.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:00.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:00.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:00.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:00.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "", 2024-04-12T04:56:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:56:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:56:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:56:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:00.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712893821.189168_494.1", 2024-04-12T04:56:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:50:21.189168Z", 2024-04-12T04:56:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:00.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:00.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T03:50:21Z" 2024-04-12T04:56:00.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.422 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:00.422 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:00.422 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:00.422 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:00.422 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:00.422 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:00.422 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.422 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.422 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:00.422 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.423 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:00.423 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.423 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.423 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:00.423 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.423 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:00.423 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.423 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:00.423 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.423 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.423 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.423 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.423 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:00.423 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.423 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.423 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:56:00.424 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.424 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:00.424 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.424 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:56:00.424 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.424 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.424 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.424 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.424 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:00.424 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.424 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.424 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:56:00.424 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.424 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:00.425 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.425 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:56:00.425 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.425 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.425 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.425 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.425 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:00.425 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.425 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.425 INFO:tasks.rgw_multisite_tests: "marker": "1_1712893821.189168_494.1", 2024-04-12T04:56:00.425 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.425 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:00.425 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.425 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:50:21.189168Z", 2024-04-12T04:56:00.425 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.425 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.426 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.426 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:00.426 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.426 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:00.426 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:00.426 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:00.426 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.426 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:50:21Z" 2024-04-12T04:56:00.426 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:00.426 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893821.189168_494.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:50:21.189168Z', 'realm_epoch': 2}}] 2024-04-12T04:56:00.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893821.196084_497.1'} 2024-04-12T04:56:00.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893821.189168_494.1'} 2024-04-12T04:56:00.426 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712893821.196084_497.1 target=1_1712893821.189168_494.1 2024-04-12T04:56:00.426 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.427 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:50:26.652+0000 7fafc273d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:50:26.653+0000 7fafc273d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.427 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.428 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:00.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "", 2024-04-12T04:56:00.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:00.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "0.000000", 2024-04-12T04:56:00.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.429 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:56:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:56:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712893821.196084_497.1", 2024-04-12T04:56:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:50:21.196084Z", 2024-04-12T04:56:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:00.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:00.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:00.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T03:50:26Z" 2024-04-12T04:56:00.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:00.432 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:00.432 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:00.432 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:00.432 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:00.432 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:00.432 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.432 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.433 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:00.433 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.433 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:00.433 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.433 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.433 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:00.433 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.433 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:00.433 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.433 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:00.433 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.433 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.433 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.434 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.434 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:00.434 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.434 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.434 INFO:tasks.rgw_multisite_tests: "marker": "", 2024-04-12T04:56:00.434 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.434 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:00.434 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.434 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2024-04-12T04:56:00.434 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.434 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.434 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.435 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.435 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:00.435 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.435 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.435 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891925.655973_112.1", 2024-04-12T04:56:00.435 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.435 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:00.435 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.435 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:18:45.655973Z", 2024-04-12T04:56:00.435 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.435 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.435 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.436 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.436 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:00.436 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.436 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.436 INFO:tasks.rgw_multisite_tests: "marker": "1_1712893821.196084_497.1", 2024-04-12T04:56:00.436 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.436 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:00.436 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.436 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:50:21.196084Z", 2024-04-12T04:56:00.436 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.436 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.436 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.437 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:00.437 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.437 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:00.437 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:00.437 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:00.437 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.437 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T03:50:26Z" 2024-04-12T04:56:00.437 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:00.437 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712891925.655973_112.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:18:45.655973Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712893821.196084_497.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T03:50:21.196084Z', 'realm_epoch': 2}}] 2024-04-12T04:56:00.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893821.196084_497.1'} 2024-04-12T04:56:00.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '', 2: '1_1712891925.655973_112.1', 3: '1_1712893821.196084_497.1'} 2024-04-12T04:56:00.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-12T04:56:00.438 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dkyges-57 2024-04-12T04:56:00.438 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:00.438 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:00.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:56:00.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:50:26.780+0000 7f3b9a929780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:50:31.781+0000 7f3b9a929780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:50:36.783+0000 7f3b9a929780 1 bucket sync caught up with source: 2024-04-12T04:56:00.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [, 00000000001.53.6, , , , 00000000001.19.6, , , 00000000001.93.6, , 00000000001.36.6] 2024-04-12T04:56:00.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [, 00000000001.53.6, , , , 00000000001.19.6, , , 00000000001.93.6, , 00000000001.36.6] 2024-04-12T04:56:00.438 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:50:36.783+0000 7f3b9a929780 0 bucket checkpoint complete 2024-04-12T04:56:00.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=dkyges-57 2024-04-12T04:56:00.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:00.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:00.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:56:00.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:50:36.909+0000 7f4a53157780 1 waiting to reach incremental sync.. 2024-04-12T04:56:00.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:50:41.911+0000 7f4a53157780 1 bucket sync caught up with source: 2024-04-12T04:56:00.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [, 00000000001.2906.6, , , , 00000000001.1066.6, , , 00000000001.1602.6, , 00000000001.3698.6] 2024-04-12T04:56:00.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [, 00000000001.2906.6, , , , 00000000001.1066.6, , , 00000000001.1602.6, , 00000000001.3698.6] 2024-04-12T04:56:00.439 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:50:41.911+0000 7f4a53157780 0 bucket checkpoint complete 2024-04-12T04:56:00.439 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=dkyges-57 zones={a2, a1} 2024-04-12T04:56:00.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.439 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/ 2024-04-12T04:56:00.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.440 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.440 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.440 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.440 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.440 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:41 GMT 2024-04-12T04:56:00.440 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.441 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gft4wKK0TzuNf39+w4XqPcng0w4= 2024-04-12T04:56:00.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gft4wKK0TzuNf39+w4XqPcng0w4=', 'Content-Length': '0'} 2024-04-12T04:56:00.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cd5e56c8a3b988f-006618af91-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/ 2024-04-12T04:56:00.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.441 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.442 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.442 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.442 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.442 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:41 GMT 2024-04-12T04:56:00.442 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.442 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gft4wKK0TzuNf39+w4XqPcng0w4= 2024-04-12T04:56:00.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gft4wKK0TzuNf39+w4XqPcng0w4=', 'Content-Length': '0'} 2024-04-12T04:56:00.442 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ad85d690a57eb672-006618af91-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.442 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-12T04:56:00.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.444 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.444 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.444 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.444 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:41 GMT 2024-04-12T04:56:00.444 INFO:tasks.rgw_multisite_tests:/dkyges-57/?versions 2024-04-12T04:56:00.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.444 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:fuHj8srEFFZQOgpqfd8gzn6gSUE= 2024-04-12T04:56:00.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fuHj8srEFFZQOgpqfd8gzn6gSUE=', 'Content-Length': '0'} 2024-04-12T04:56:00.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040a869e55ee0141c-006618af91-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.444 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-12T04:56:00.445 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-12T04:56:00.445 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-12T04:56:00.445 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-12T04:56:00.445 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-12T04:56:00.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.446 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.446 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.446 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.446 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:41 GMT 2024-04-12T04:56:00.446 INFO:tasks.rgw_multisite_tests:/dkyges-57/?versions 2024-04-12T04:56:00.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.446 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:fuHj8srEFFZQOgpqfd8gzn6gSUE= 2024-04-12T04:56:00.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fuHj8srEFFZQOgpqfd8gzn6gSUE=', 'Content-Length': '0'} 2024-04-12T04:56:00.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2fa104ad2a22d84-006618af91-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.447 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-12T04:56:00.447 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-12T04:56:00.447 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-12T04:56:00.447 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-12T04:56:00.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.448 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.448 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.448 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.448 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:41 GMT 2024-04-12T04:56:00.448 INFO:tasks.rgw_multisite_tests:/dkyges-57/?versions 2024-04-12T04:56:00.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.448 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:fuHj8srEFFZQOgpqfd8gzn6gSUE= 2024-04-12T04:56:00.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fuHj8srEFFZQOgpqfd8gzn6gSUE=', 'Content-Length': '0'} 2024-04-12T04:56:00.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cc759735466550e-006618af91-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.450 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.450 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.450 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.450 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:41 GMT 2024-04-12T04:56:00.450 INFO:tasks.rgw_multisite_tests:/dkyges-57/?versions 2024-04-12T04:56:00.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.450 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:fuHj8srEFFZQOgpqfd8gzn6gSUE= 2024-04-12T04:56:00.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fuHj8srEFFZQOgpqfd8gzn6gSUE=', 'Content-Length': '0'} 2024-04-12T04:56:00.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a2f0ca1c510f9bf-006618af91-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.450 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:56:00.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.451 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.451 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.451 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.452 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:41 GMT 2024-04-12T04:56:00.452 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.452 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:rejUZ20WKjx52D0KUZBOjLXK8yo= 2024-04-12T04:56:00.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rejUZ20WKjx52D0KUZBOjLXK8yo=', 'Content-Length': '0'} 2024-04-12T04:56:00.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004db7b54ff8e3af70-006618af91-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:41 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.453 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.453 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.453 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.453 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:41 GMT 2024-04-12T04:56:00.453 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.453 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:rejUZ20WKjx52D0KUZBOjLXK8yo= 2024-04-12T04:56:00.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rejUZ20WKjx52D0KUZBOjLXK8yo=', 'Content-Length': '0'} 2024-04-12T04:56:00.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6e8a942ade1a34c-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.454 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.454 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.454 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.455 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:41 GMT 2024-04-12T04:56:00.455 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.455 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:RNJbsCkns/BFabmw64vwQWeUz6g= 2024-04-12T04:56:00.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:41 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RNJbsCkns/BFabmw64vwQWeUz6g=', 'Content-Length': '0'} 2024-04-12T04:56:00.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9272988af9639c6-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.456 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.456 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.456 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.456 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.456 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.456 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:C6vKTTFB62DMlSSVr+wffUWb9x0= 2024-04-12T04:56:00.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C6vKTTFB62DMlSSVr+wffUWb9x0=', 'Content-Length': '0'} 2024-04-12T04:56:00.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf25ccde18c5e513-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.456 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:56:00.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.457 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.457 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.457 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.458 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.458 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.458 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ccEwEMCzbuhjwijKm7iMAkyrqMo= 2024-04-12T04:56:00.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ccEwEMCzbuhjwijKm7iMAkyrqMo=', 'Content-Length': '0'} 2024-04-12T04:56:00.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003273fd2ff01b0e0-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.459 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.459 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.459 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.459 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.459 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.459 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ccEwEMCzbuhjwijKm7iMAkyrqMo= 2024-04-12T04:56:00.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ccEwEMCzbuhjwijKm7iMAkyrqMo=', 'Content-Length': '0'} 2024-04-12T04:56:00.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcd18d4a8fe3abac-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=a 2024-04-12T04:56:00.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.460 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.460 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.461 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.461 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.461 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.461 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.461 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T04:56:00.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T04:56:00.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e61c689b0dc10a5-006618af92-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=a 2024-04-12T04:56:00.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.462 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.462 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.462 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.462 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.462 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.462 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T04:56:00.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T04:56:00.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f91c8f671fa73c14-006618af92-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a 2024-04-12T04:56:00.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.464 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.464 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.464 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.464 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.464 INFO:tasks.rgw_multisite_tests:/dkyges-57/a 2024-04-12T04:56:00.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.464 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gzUgsw3BZqyPgZ0tp30snRglmuQ= 2024-04-12T04:56:00.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gzUgsw3BZqyPgZ0tp30snRglmuQ=', 'Content-Length': '0'} 2024-04-12T04:56:00.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2165499c7c1b451-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a 2024-04-12T04:56:00.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.465 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.465 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.465 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.465 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.465 INFO:tasks.rgw_multisite_tests:/dkyges-57/a 2024-04-12T04:56:00.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.465 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gzUgsw3BZqyPgZ0tp30snRglmuQ= 2024-04-12T04:56:00.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gzUgsw3BZqyPgZ0tp30snRglmuQ=', 'Content-Length': '0'} 2024-04-12T04:56:00.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d32978534292aa1-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.466 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:56:00.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a 2024-04-12T04:56:00.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.467 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.467 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.467 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.467 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.467 INFO:tasks.rgw_multisite_tests:/dkyges-57/a 2024-04-12T04:56:00.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.467 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:WsrnbViAVR9ZDdPYi8y7F3B8mnY= 2024-04-12T04:56:00.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WsrnbViAVR9ZDdPYi8y7F3B8mnY=', 'Content-Length': '0'} 2024-04-12T04:56:00.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cd37ca8cded458a-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a 2024-04-12T04:56:00.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.468 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.468 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.468 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.468 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.469 INFO:tasks.rgw_multisite_tests:/dkyges-57/a 2024-04-12T04:56:00.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.469 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:WsrnbViAVR9ZDdPYi8y7F3B8mnY= 2024-04-12T04:56:00.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WsrnbViAVR9ZDdPYi8y7F3B8mnY=', 'Content-Length': '0'} 2024-04-12T04:56:00.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dd0457df1c559b7-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=a 2024-04-12T04:56:00.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.470 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.470 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.470 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.470 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.470 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.470 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T04:56:00.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T04:56:00.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000299c43c1fdd88a0e-006618af92-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=a 2024-04-12T04:56:00.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.472 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.472 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.472 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.472 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.472 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.472 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T04:56:00.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T04:56:00.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d74f5c995956be36-006618af92-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.472 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:56:00.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.473 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.473 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.474 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.474 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.474 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.474 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:1qDgvbqtWc+aikdQtfdLDXgcV4Y= 2024-04-12T04:56:00.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1qDgvbqtWc+aikdQtfdLDXgcV4Y=', 'Content-Length': '0'} 2024-04-12T04:56:00.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000514b272bf4c630b3-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.475 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.475 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.475 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.475 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.475 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.475 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:1qDgvbqtWc+aikdQtfdLDXgcV4Y= 2024-04-12T04:56:00.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1qDgvbqtWc+aikdQtfdLDXgcV4Y=', 'Content-Length': '0'} 2024-04-12T04:56:00.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a18e767299ef1f7-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.476 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.477 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.477 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.477 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.477 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.477 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:a+ww9uLwQ0NYwYvaZDXx3GDLhkU= 2024-04-12T04:56:00.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a+ww9uLwQ0NYwYvaZDXx3GDLhkU=', 'Content-Length': '0'} 2024-04-12T04:56:00.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cb781eeb4893b31-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.478 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.478 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.478 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.478 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.478 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.479 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:a+ww9uLwQ0NYwYvaZDXx3GDLhkU= 2024-04-12T04:56:00.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a+ww9uLwQ0NYwYvaZDXx3GDLhkU=', 'Content-Length': '0'} 2024-04-12T04:56:00.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f1d434fd7e8bcdf-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.479 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:56:00.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.480 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.480 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.480 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.480 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.480 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.480 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:1qDgvbqtWc+aikdQtfdLDXgcV4Y= 2024-04-12T04:56:00.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1qDgvbqtWc+aikdQtfdLDXgcV4Y=', 'Content-Length': '0'} 2024-04-12T04:56:00.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000706d911190e99f1f-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.481 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.481 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.481 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.481 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.482 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.482 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:1qDgvbqtWc+aikdQtfdLDXgcV4Y= 2024-04-12T04:56:00.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1qDgvbqtWc+aikdQtfdLDXgcV4Y=', 'Content-Length': '0'} 2024-04-12T04:56:00.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075861a13181d0348-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=b 2024-04-12T04:56:00.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.483 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.483 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.483 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.483 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.483 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.483 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T04:56:00.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T04:56:00.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce89b457ab1a4c22-006618af92-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=b 2024-04-12T04:56:00.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.484 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.484 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.485 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.485 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.485 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.485 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T04:56:00.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T04:56:00.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045d2aeb0fe4c6061-006618af92-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b 2024-04-12T04:56:00.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.486 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.486 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.486 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.486 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.486 INFO:tasks.rgw_multisite_tests:/dkyges-57/b 2024-04-12T04:56:00.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.487 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:9By2kDgLYqGGul2UUYTS18X0mOg= 2024-04-12T04:56:00.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9By2kDgLYqGGul2UUYTS18X0mOg=', 'Content-Length': '0'} 2024-04-12T04:56:00.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd9b2ce456d74af0-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b 2024-04-12T04:56:00.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.488 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.488 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.488 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.488 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.488 INFO:tasks.rgw_multisite_tests:/dkyges-57/b 2024-04-12T04:56:00.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.488 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:9By2kDgLYqGGul2UUYTS18X0mOg= 2024-04-12T04:56:00.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9By2kDgLYqGGul2UUYTS18X0mOg=', 'Content-Length': '0'} 2024-04-12T04:56:00.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8846eeedb0fccaa-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.488 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:56:00.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b 2024-04-12T04:56:00.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.489 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.489 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.489 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.489 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.489 INFO:tasks.rgw_multisite_tests:/dkyges-57/b 2024-04-12T04:56:00.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.490 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:V3TKYnu9QCCYSnJg1D2MaPMfM6I= 2024-04-12T04:56:00.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V3TKYnu9QCCYSnJg1D2MaPMfM6I=', 'Content-Length': '0'} 2024-04-12T04:56:00.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4e030f701e059b6-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b 2024-04-12T04:56:00.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.491 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.491 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.491 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.491 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.491 INFO:tasks.rgw_multisite_tests:/dkyges-57/b 2024-04-12T04:56:00.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.491 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:V3TKYnu9QCCYSnJg1D2MaPMfM6I= 2024-04-12T04:56:00.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:V3TKYnu9QCCYSnJg1D2MaPMfM6I=', 'Content-Length': '0'} 2024-04-12T04:56:00.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022f2e067630a39ea-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=b 2024-04-12T04:56:00.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.492 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.492 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.492 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.492 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.492 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.492 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T04:56:00.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T04:56:00.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012a23cee6813fd34-006618af92-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=b 2024-04-12T04:56:00.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.494 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.494 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.494 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.494 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.494 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.494 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T04:56:00.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T04:56:00.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053f16f6bcfcfd5eb-006618af92-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.494 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:56:00.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.495 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.495 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.495 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.495 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.495 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.496 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:WKT0NTfM4GA8D5rQ5eDOV58cC80= 2024-04-12T04:56:00.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WKT0NTfM4GA8D5rQ5eDOV58cC80=', 'Content-Length': '0'} 2024-04-12T04:56:00.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe543b589ebb89bd-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.497 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.497 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.497 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.497 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.497 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.497 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:WKT0NTfM4GA8D5rQ5eDOV58cC80= 2024-04-12T04:56:00.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WKT0NTfM4GA8D5rQ5eDOV58cC80=', 'Content-Length': '0'} 2024-04-12T04:56:00.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c59471b213261a6-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.498 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.498 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.498 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.498 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.498 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.498 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:fTtQG5fbfKK+LScz8vwcHDP9zfg= 2024-04-12T04:56:00.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fTtQG5fbfKK+LScz8vwcHDP9zfg=', 'Content-Length': '0'} 2024-04-12T04:56:00.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a030922b28df9dc-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.499 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.500 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.500 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.500 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.500 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.500 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:fTtQG5fbfKK+LScz8vwcHDP9zfg= 2024-04-12T04:56:00.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fTtQG5fbfKK+LScz8vwcHDP9zfg=', 'Content-Length': '0'} 2024-04-12T04:56:00.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d6c204c95c97f8f-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.500 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:56:00.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.501 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.501 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.501 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.501 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.501 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.501 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:WKT0NTfM4GA8D5rQ5eDOV58cC80= 2024-04-12T04:56:00.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WKT0NTfM4GA8D5rQ5eDOV58cC80=', 'Content-Length': '0'} 2024-04-12T04:56:00.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0aca77c4b9f8fd7-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.502 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.503 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.503 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.503 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.503 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.503 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:WKT0NTfM4GA8D5rQ5eDOV58cC80= 2024-04-12T04:56:00.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WKT0NTfM4GA8D5rQ5eDOV58cC80=', 'Content-Length': '0'} 2024-04-12T04:56:00.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044fed36d57db31df-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=c 2024-04-12T04:56:00.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.504 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.504 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.504 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.504 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.504 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.504 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T04:56:00.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T04:56:00.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001e948f16dc2ac45-006618af92-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=c 2024-04-12T04:56:00.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.506 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.506 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.506 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.506 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.506 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.506 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T04:56:00.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T04:56:00.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000faa80cd1f326fdf2-006618af92-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c 2024-04-12T04:56:00.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.507 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.507 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.507 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.507 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.507 INFO:tasks.rgw_multisite_tests:/dkyges-57/c 2024-04-12T04:56:00.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.508 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:14IQT2FGso7SmQ4LvrfyuYS4j9o= 2024-04-12T04:56:00.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:14IQT2FGso7SmQ4LvrfyuYS4j9o=', 'Content-Length': '0'} 2024-04-12T04:56:00.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2e07e0ef568d198-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c 2024-04-12T04:56:00.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.508 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.509 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.509 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.509 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.509 INFO:tasks.rgw_multisite_tests:/dkyges-57/c 2024-04-12T04:56:00.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.509 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:14IQT2FGso7SmQ4LvrfyuYS4j9o= 2024-04-12T04:56:00.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:14IQT2FGso7SmQ4LvrfyuYS4j9o=', 'Content-Length': '0'} 2024-04-12T04:56:00.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc2ef24f8b941b68-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.509 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:56:00.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c 2024-04-12T04:56:00.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.510 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.510 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.510 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.510 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.510 INFO:tasks.rgw_multisite_tests:/dkyges-57/c 2024-04-12T04:56:00.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.511 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:MeUKTRJCsgdmxFOQ7vGdBIogcgY= 2024-04-12T04:56:00.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MeUKTRJCsgdmxFOQ7vGdBIogcgY=', 'Content-Length': '0'} 2024-04-12T04:56:00.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e2c1f5b6ccd959b-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c 2024-04-12T04:56:00.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.511 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.512 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.512 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.512 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.512 INFO:tasks.rgw_multisite_tests:/dkyges-57/c 2024-04-12T04:56:00.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.512 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:MeUKTRJCsgdmxFOQ7vGdBIogcgY= 2024-04-12T04:56:00.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MeUKTRJCsgdmxFOQ7vGdBIogcgY=', 'Content-Length': '0'} 2024-04-12T04:56:00.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c6602d1c8f0609c-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=c 2024-04-12T04:56:00.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.513 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.513 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.513 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.513 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.513 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.514 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T04:56:00.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T04:56:00.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006389a8a23717532b-006618af92-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=c 2024-04-12T04:56:00.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.515 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.515 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.515 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.515 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.515 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.515 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T04:56:00.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T04:56:00.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fd151424f18f432-006618af92-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.516 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:56:00.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.517 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.517 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.517 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.517 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.517 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.517 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Bw+//k5ZFvE2VOA7dl2hMrI6tUI= 2024-04-12T04:56:00.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bw+//k5ZFvE2VOA7dl2hMrI6tUI=', 'Content-Length': '0'} 2024-04-12T04:56:00.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000949716fc44723e7-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.518 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.518 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.518 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.518 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.519 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.519 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Bw+//k5ZFvE2VOA7dl2hMrI6tUI= 2024-04-12T04:56:00.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bw+//k5ZFvE2VOA7dl2hMrI6tUI=', 'Content-Length': '0'} 2024-04-12T04:56:00.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000670599aa25421b0e-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.520 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.520 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.520 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.520 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.520 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.520 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:wV2XdrkxHBbG6qYqw5fmfrJVF34= 2024-04-12T04:56:00.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wV2XdrkxHBbG6qYqw5fmfrJVF34=', 'Content-Length': '0'} 2024-04-12T04:56:00.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000715d70b315afe944-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.522 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.522 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.522 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.522 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.522 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.522 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:wV2XdrkxHBbG6qYqw5fmfrJVF34= 2024-04-12T04:56:00.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wV2XdrkxHBbG6qYqw5fmfrJVF34=', 'Content-Length': '0'} 2024-04-12T04:56:00.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9eff826cb5d204f-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.522 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:56:00.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.523 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.523 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.523 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.523 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.524 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.524 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Bw+//k5ZFvE2VOA7dl2hMrI6tUI= 2024-04-12T04:56:00.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bw+//k5ZFvE2VOA7dl2hMrI6tUI=', 'Content-Length': '0'} 2024-04-12T04:56:00.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a07da31eaecc7420-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.525 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.525 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.525 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.525 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.525 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.525 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Bw+//k5ZFvE2VOA7dl2hMrI6tUI= 2024-04-12T04:56:00.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bw+//k5ZFvE2VOA7dl2hMrI6tUI=', 'Content-Length': '0'} 2024-04-12T04:56:00.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e7dad277fef946e-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=d 2024-04-12T04:56:00.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.526 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.526 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.526 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.526 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.526 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.527 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T04:56:00.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T04:56:00.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bab24032ae63746f-006618af92-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=d 2024-04-12T04:56:00.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.528 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.528 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.528 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.528 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.528 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.528 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T04:56:00.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T04:56:00.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000595b399102d5392a-006618af92-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d 2024-04-12T04:56:00.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.529 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.529 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.529 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.529 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.529 INFO:tasks.rgw_multisite_tests:/dkyges-57/d 2024-04-12T04:56:00.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.530 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:D+NJpaaAT2F/zDlF1w2a5rTp0gg= 2024-04-12T04:56:00.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D+NJpaaAT2F/zDlF1w2a5rTp0gg=', 'Content-Length': '0'} 2024-04-12T04:56:00.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42fe89299def7dc-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d 2024-04-12T04:56:00.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.531 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.531 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.531 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.531 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.531 INFO:tasks.rgw_multisite_tests:/dkyges-57/d 2024-04-12T04:56:00.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.531 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:D+NJpaaAT2F/zDlF1w2a5rTp0gg= 2024-04-12T04:56:00.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D+NJpaaAT2F/zDlF1w2a5rTp0gg=', 'Content-Length': '0'} 2024-04-12T04:56:00.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000601d2a19b4fae2b3-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.531 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:56:00.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d 2024-04-12T04:56:00.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.532 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.532 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.532 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.532 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.532 INFO:tasks.rgw_multisite_tests:/dkyges-57/d 2024-04-12T04:56:00.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.533 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:JwYfbopNZSwXMd/9zNxL9zc0xWY= 2024-04-12T04:56:00.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JwYfbopNZSwXMd/9zNxL9zc0xWY=', 'Content-Length': '0'} 2024-04-12T04:56:00.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9cdbe80fdf7e010-006618af92-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d 2024-04-12T04:56:00.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.534 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.534 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.534 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.534 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.534 INFO:tasks.rgw_multisite_tests:/dkyges-57/d 2024-04-12T04:56:00.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.534 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:JwYfbopNZSwXMd/9zNxL9zc0xWY= 2024-04-12T04:56:00.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JwYfbopNZSwXMd/9zNxL9zc0xWY=', 'Content-Length': '0'} 2024-04-12T04:56:00.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fb5b2bba4fe4db5-006618af92-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=d 2024-04-12T04:56:00.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.535 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.535 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.535 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.535 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.535 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.536 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T04:56:00.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T04:56:00.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050f80e44d702c898-006618af92-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=d 2024-04-12T04:56:00.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.537 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.537 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.537 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.537 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:42 GMT 2024-04-12T04:56:00.537 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.537 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw= 2024-04-12T04:56:00.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sc2v+B89vPMkGNNkAPdsdBMPwJw=', 'Content-Length': '0'} 2024-04-12T04:56:00.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a68a8a5808d9bd74-006618af92-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:50:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.538 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=dkyges-57 zones={a2, a1} 2024-04-12T04:56:00.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.538 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dkyges-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:00.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "resharding": "None", 2024-04-12T04:56:00.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_index": { 2024-04-12T04:56:00.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 0, 2024-04-12T04:56:00.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "type": "Normal", 2024-04-12T04:56:00.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "normal": { 2024-04-12T04:56:00.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 11, 2024-04-12T04:56:00.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:56:00.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "logs": [ 2024-04-12T04:56:00.539 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 0, 2024-04-12T04:56:00.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "type": "InIndex", 2024-04-12T04:56:00.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "in_index": { 2024-04-12T04:56:00.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 0, 2024-04-12T04:56:00.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 11, 2024-04-12T04:56:00.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:56:00.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.540 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.541 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dkyges-57', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.541 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dkyges-57', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dkyges-57 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:00.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: [ 2024-04-12T04:56:00.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "op_id": "1#00000000001.53.6", 2024-04-12T04:56:00.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "op_tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.8775263116062048419", 2024-04-12T04:56:00.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "op": "write", 2024-04-12T04:56:00.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "object": "a", 2024-04-12T04:56:00.541 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "instance": "", 2024-04-12T04:56:00.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": "complete", 2024-04-12T04:56:00.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "index_ver": 1, 2024-04-12T04:56:00.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:50:21.205175995Z", 2024-04-12T04:56:00.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "ver": { 2024-04-12T04:56:00.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pool": 9, 2024-04-12T04:56:00.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "epoch": 4 2024-04-12T04:56:00.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "bilog_flags": 0, 2024-04-12T04:56:00.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "versioned": false, 2024-04-12T04:56:00.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "owner": "", 2024-04-12T04:56:00.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "owner_display_name": "", 2024-04-12T04:56:00.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "zones_trace": [ 2024-04-12T04:56:00.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.542 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "entry": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20" 2024-04-12T04:56:00.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "op_id": "5#00000000001.19.6", 2024-04-12T04:56:00.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "op_tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.6696391949831587819", 2024-04-12T04:56:00.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "op": "write", 2024-04-12T04:56:00.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "object": "d", 2024-04-12T04:56:00.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "instance": "", 2024-04-12T04:56:00.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": "complete", 2024-04-12T04:56:00.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "index_ver": 1, 2024-04-12T04:56:00.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:50:21.275981344Z", 2024-04-12T04:56:00.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "ver": { 2024-04-12T04:56:00.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pool": 9, 2024-04-12T04:56:00.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "epoch": 8 2024-04-12T04:56:00.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "bilog_flags": 0, 2024-04-12T04:56:00.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "versioned": false, 2024-04-12T04:56:00.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "owner": "", 2024-04-12T04:56:00.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "owner_display_name": "", 2024-04-12T04:56:00.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "zones_trace": [ 2024-04-12T04:56:00.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "entry": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20" 2024-04-12T04:56:00.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "op_id": "8#00000000001.93.6", 2024-04-12T04:56:00.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "op_tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.2825657797998053591", 2024-04-12T04:56:00.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "op": "write", 2024-04-12T04:56:00.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "object": "c", 2024-04-12T04:56:00.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "instance": "", 2024-04-12T04:56:00.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": "complete", 2024-04-12T04:56:00.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "index_ver": 1, 2024-04-12T04:56:00.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:50:21.264841306Z", 2024-04-12T04:56:00.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "ver": { 2024-04-12T04:56:00.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pool": 9, 2024-04-12T04:56:00.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "epoch": 18 2024-04-12T04:56:00.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "bilog_flags": 0, 2024-04-12T04:56:00.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "versioned": false, 2024-04-12T04:56:00.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "owner": "", 2024-04-12T04:56:00.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "owner_display_name": "", 2024-04-12T04:56:00.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "zones_trace": [ 2024-04-12T04:56:00.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "entry": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20" 2024-04-12T04:56:00.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "op_id": "10#00000000001.36.6", 2024-04-12T04:56:00.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "op_tag": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5845.6615111383423553842", 2024-04-12T04:56:00.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "op": "write", 2024-04-12T04:56:00.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "object": "b", 2024-04-12T04:56:00.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "instance": "", 2024-04-12T04:56:00.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": "complete", 2024-04-12T04:56:00.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "index_ver": 1, 2024-04-12T04:56:00.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:50:21.217597097Z", 2024-04-12T04:56:00.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "ver": { 2024-04-12T04:56:00.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pool": 9, 2024-04-12T04:56:00.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "epoch": 2 2024-04-12T04:56:00.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "bilog_flags": 0, 2024-04-12T04:56:00.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "versioned": false, 2024-04-12T04:56:00.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "owner": "", 2024-04-12T04:56:00.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "owner_display_name": "", 2024-04-12T04:56:00.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "zones_trace": [ 2024-04-12T04:56:00.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "entry": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20" 2024-04-12T04:56:00.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.548 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dkyges-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.548 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dkyges-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dkyges-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:00.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: tenant: 2024-04-12T04:56:00.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: bucket name: dkyges-57 2024-04-12T04:56:00.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: total entries: 4 2024-04-12T04:56:00.548 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.548 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dkyges-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:00.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "resharding": "None", 2024-04-12T04:56:00.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_index": { 2024-04-12T04:56:00.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 1, 2024-04-12T04:56:00.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "type": "Normal", 2024-04-12T04:56:00.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "normal": { 2024-04-12T04:56:00.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 3, 2024-04-12T04:56:00.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:56:00.549 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "logs": [ 2024-04-12T04:56:00.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 0, 2024-04-12T04:56:00.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "type": "InIndex", 2024-04-12T04:56:00.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "in_index": { 2024-04-12T04:56:00.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 0, 2024-04-12T04:56:00.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 11, 2024-04-12T04:56:00.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:56:00.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.550 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 1, 2024-04-12T04:56:00.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "type": "InIndex", 2024-04-12T04:56:00.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "in_index": { 2024-04-12T04:56:00.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 1, 2024-04-12T04:56:00.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 3, 2024-04-12T04:56:00.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:56:00.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.551 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.552 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'dkyges-57', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.552 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'dkyges-57', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket dkyges-57 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:00.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.552 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: [] 2024-04-12T04:56:00.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/ 2024-04-12T04:56:00.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.553 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.553 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.553 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.553 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:43 GMT 2024-04-12T04:56:00.553 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.553 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:uzcilWOkK6VgOd7p8ce8aLkwfRU= 2024-04-12T04:56:00.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uzcilWOkK6VgOd7p8ce8aLkwfRU=', 'Content-Length': '0'} 2024-04-12T04:56:00.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000074c539e8ada6b85f-006618af93-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e 2024-04-12T04:56:00.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:00.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.554 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.554 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:00.554 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.555 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:43 GMT 2024-04-12T04:56:00.555 INFO:tasks.rgw_multisite_tests:/dkyges-57/e 2024-04-12T04:56:00.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.555 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:YJHjnE2QCuKTeFrHao5whOc/bqI= 2024-04-12T04:56:00.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YJHjnE2QCuKTeFrHao5whOc/bqI='} 2024-04-12T04:56:00.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000084764007ccf4ac14-006618af93-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/ 2024-04-12T04:56:00.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.556 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.556 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.556 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.556 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:43 GMT 2024-04-12T04:56:00.556 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.556 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:uzcilWOkK6VgOd7p8ce8aLkwfRU= 2024-04-12T04:56:00.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uzcilWOkK6VgOd7p8ce8aLkwfRU=', 'Content-Length': '0'} 2024-04-12T04:56:00.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012ada3b8c0f95349-006618af93-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f 2024-04-12T04:56:00.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:00.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.557 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.557 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:00.557 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.557 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:43 GMT 2024-04-12T04:56:00.557 INFO:tasks.rgw_multisite_tests:/dkyges-57/f 2024-04-12T04:56:00.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.558 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:EyNxIdRQgOp2bMBpQeUVIYwLKjE= 2024-04-12T04:56:00.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EyNxIdRQgOp2bMBpQeUVIYwLKjE='} 2024-04-12T04:56:00.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ddc531ce0394441-006618af93-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/ 2024-04-12T04:56:00.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.559 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.559 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.559 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.559 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:43 GMT 2024-04-12T04:56:00.559 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.559 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:uzcilWOkK6VgOd7p8ce8aLkwfRU= 2024-04-12T04:56:00.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uzcilWOkK6VgOd7p8ce8aLkwfRU=', 'Content-Length': '0'} 2024-04-12T04:56:00.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bff2da88c20ba29-006618af93-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g 2024-04-12T04:56:00.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:00.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.560 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.560 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:00.560 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.560 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:43 GMT 2024-04-12T04:56:00.560 INFO:tasks.rgw_multisite_tests:/dkyges-57/g 2024-04-12T04:56:00.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.560 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:BfCoICVOZIUzRB8uc4bNRxEZ2II= 2024-04-12T04:56:00.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BfCoICVOZIUzRB8uc4bNRxEZ2II='} 2024-04-12T04:56:00.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000201b931f64420830-006618af93-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/ 2024-04-12T04:56:00.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.561 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.561 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.562 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.562 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:43 GMT 2024-04-12T04:56:00.562 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.562 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:uzcilWOkK6VgOd7p8ce8aLkwfRU= 2024-04-12T04:56:00.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:50:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uzcilWOkK6VgOd7p8ce8aLkwfRU=', 'Content-Length': '0'} 2024-04-12T04:56:00.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6d6e168af785a07-006618af93-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h 2024-04-12T04:56:00.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:00.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.563 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.563 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:00.563 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.563 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:50:43 GMT 2024-04-12T04:56:00.563 INFO:tasks.rgw_multisite_tests:/dkyges-57/h 2024-04-12T04:56:00.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.563 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:C04i4i/4Bm8dexUQuPfRTpwB41o= 2024-04-12T04:56:00.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 03:50:43 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:C04i4i/4Bm8dexUQuPfRTpwB41o='} 2024-04-12T04:56:00.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0bbf016c232f4b7-006618af93-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dkyges-57 2024-04-12T04:56:00.563 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:00.564 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:00.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:56:00.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:50:43.246+0000 7f8986a60780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T04:56:00.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:50:48.247+0000 7f8986a60780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T04:56:00.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:50:53.247+0000 7f8986a60780 1 waiting to reach latest gen 1, on 0.. 2024-04-12T04:56:00.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:50:58.249+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T04:56:00.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [, , ] 2024-04-12T04:56:00.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T04:56:00.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:51:03.250+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T04:56:00.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [, , ] 2024-04-12T04:56:00.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T04:56:00.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:51:08.251+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T04:56:00.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [, , ] 2024-04-12T04:56:00.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T04:56:00.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:51:13.252+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T04:56:00.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [, , ] 2024-04-12T04:56:00.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T04:56:00.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:51:18.252+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T04:56:00.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [00000000003.39.6, , ] 2024-04-12T04:56:00.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T04:56:00.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:51:23.253+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T04:56:00.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [00000000003.39.6, , ] 2024-04-12T04:56:00.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T04:56:00.565 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi043.stdout: INFO: 2024-04-12T03:51:27.130+0000 7f1e272bd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:56:00.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:51:28.255+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T04:56:00.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [00000000003.39.6, , ] 2024-04-12T04:56:00.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T04:56:00.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:51:33.255+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T04:56:00.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [00000000003.39.6, , ] 2024-04-12T04:56:00.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T04:56:00.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:51:38.256+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T04:56:00.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [00000000003.39.6, 00000000002.54.6, ] 2024-04-12T04:56:00.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T04:56:00.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:51:43.257+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T04:56:00.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [00000000003.39.6, 00000000002.54.6, ] 2024-04-12T04:56:00.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T04:56:00.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:51:48.258+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T04:56:00.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [00000000003.39.6, 00000000002.54.6, ] 2024-04-12T04:56:00.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T04:56:00.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:51:53.258+0000 7f8986a60780 1 waiting for incremental sync to catch up: 2024-04-12T04:56:00.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [00000000003.39.6, 00000000002.54.6, ] 2024-04-12T04:56:00.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T04:56:00.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:51:58.260+0000 7f8986a60780 1 bucket sync caught up with source: 2024-04-12T04:56:00.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T04:56:00.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [00000000003.39.6, 00000000002.54.6, 00000000002.22.6] 2024-04-12T04:56:00.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:51:58.260+0000 7f8986a60780 0 bucket checkpoint complete 2024-04-12T04:56:00.567 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=dkyges-57 2024-04-12T04:56:00.567 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:00.567 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:00.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:56:00.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:51:58.390+0000 7f934ab19780 1 waiting for incremental sync to catch up: 2024-04-12T04:56:00.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [, 00000000002.2910.6, , , 00000000001.3705.6, 00000000001.1066.6, , , 00000000001.1602.6, , 00000000002.3706.6] 2024-04-12T04:56:00.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [, 00000000002.2910.6, , , 00000000001.3705.6, 00000000001.1066.6, , , 00000000002.1606.6, , 00000000002.3706.6] 2024-04-12T04:56:00.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:52:03.391+0000 7f934ab19780 1 waiting for incremental sync to catch up: 2024-04-12T04:56:00.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [, 00000000002.2910.6, , , 00000000001.3705.6, 00000000001.1066.6, , , 00000000001.1602.6, , 00000000002.3706.6] 2024-04-12T04:56:00.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [, 00000000002.2910.6, , , 00000000001.3705.6, 00000000001.1066.6, , , 00000000002.1606.6, , 00000000002.3706.6] 2024-04-12T04:56:00.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:52:08.392+0000 7f934ab19780 1 bucket sync caught up with source: 2024-04-12T04:56:00.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [, 00000000002.2910.6, , , 00000000001.3705.6, 00000000001.1066.6, , , 00000000002.1606.6, , 00000000002.3706.6] 2024-04-12T04:56:00.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [, 00000000002.2910.6, , , 00000000001.3705.6, 00000000001.1066.6, , , 00000000002.1606.6, , 00000000002.3706.6] 2024-04-12T04:56:00.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:52:08.392+0000 7f934ab19780 0 bucket checkpoint complete 2024-04-12T04:56:00.568 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=dkyges-57 zones={a2, a1} 2024-04-12T04:56:00.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/ 2024-04-12T04:56:00.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:56:00.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.569 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.569 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.569 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.569 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.569 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.569 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:7Xz8aTP9keMruBOZgpVAmE7K0uY= 2024-04-12T04:56:00.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7Xz8aTP9keMruBOZgpVAmE7K0uY=', 'Content-Length': '0'} 2024-04-12T04:56:00.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a44947787cc40809-006618afe8-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/ 2024-04-12T04:56:00.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:56:00.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.570 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.570 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.571 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.571 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.571 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.571 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:7Xz8aTP9keMruBOZgpVAmE7K0uY= 2024-04-12T04:56:00.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7Xz8aTP9keMruBOZgpVAmE7K0uY=', 'Content-Length': '0'} 2024-04-12T04:56:00.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000018dd848c8adc9144-006618afe8-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.571 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-12T04:56:00.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.572 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.572 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.572 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.572 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.572 INFO:tasks.rgw_multisite_tests:/dkyges-57/?versions 2024-04-12T04:56:00.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.572 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dMgTrz9yOLU+gZfkQZLyMnk0OUk= 2024-04-12T04:56:00.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMgTrz9yOLU+gZfkQZLyMnk0OUk=', 'Content-Length': '0'} 2024-04-12T04:56:00.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060ab74bcae4ddf63-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.573 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-12T04:56:00.573 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-12T04:56:00.573 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-12T04:56:00.573 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-12T04:56:00.573 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2024-04-12T04:56:00.573 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2024-04-12T04:56:00.573 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2024-04-12T04:56:00.573 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2024-04-12T04:56:00.573 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-12T04:56:00.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.573 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.574 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.574 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.574 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.574 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.574 INFO:tasks.rgw_multisite_tests:/dkyges-57/?versions 2024-04-12T04:56:00.574 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.575 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dMgTrz9yOLU+gZfkQZLyMnk0OUk= 2024-04-12T04:56:00.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMgTrz9yOLU+gZfkQZLyMnk0OUk=', 'Content-Length': '0'} 2024-04-12T04:56:00.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001291255b2412b1ff-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.575 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-12T04:56:00.575 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-12T04:56:00.575 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-12T04:56:00.575 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-12T04:56:00.575 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2024-04-12T04:56:00.575 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2024-04-12T04:56:00.575 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2024-04-12T04:56:00.575 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2024-04-12T04:56:00.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.575 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.576 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.576 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.576 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.576 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.577 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.577 INFO:tasks.rgw_multisite_tests:/dkyges-57/?versions 2024-04-12T04:56:00.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.577 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dMgTrz9yOLU+gZfkQZLyMnk0OUk= 2024-04-12T04:56:00.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMgTrz9yOLU+gZfkQZLyMnk0OUk=', 'Content-Length': '0'} 2024-04-12T04:56:00.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ec5a79609e5dcd0-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.577 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.578 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.578 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.578 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.578 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.578 INFO:tasks.rgw_multisite_tests:/dkyges-57/?versions 2024-04-12T04:56:00.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.578 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dMgTrz9yOLU+gZfkQZLyMnk0OUk= 2024-04-12T04:56:00.578 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dMgTrz9yOLU+gZfkQZLyMnk0OUk=', 'Content-Length': '0'} 2024-04-12T04:56:00.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fe6b25105f004e7-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.579 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:56:00.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.579 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.580 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.580 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.580 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.580 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.580 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.580 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:8Ghuk8ws0gq7lXtyobRkBov+YRA= 2024-04-12T04:56:00.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8Ghuk8ws0gq7lXtyobRkBov+YRA=', 'Content-Length': '0'} 2024-04-12T04:56:00.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb12d855b82957f0-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.580 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.581 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.581 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.582 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.582 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.582 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.582 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:8Ghuk8ws0gq7lXtyobRkBov+YRA= 2024-04-12T04:56:00.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8Ghuk8ws0gq7lXtyobRkBov+YRA=', 'Content-Length': '0'} 2024-04-12T04:56:00.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001964f53600aaa44-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.583 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.583 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.583 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.583 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.583 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.583 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:87zSg4f79b89o7axtVlYTIH2sXA= 2024-04-12T04:56:00.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:87zSg4f79b89o7axtVlYTIH2sXA=', 'Content-Length': '0'} 2024-04-12T04:56:00.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012172f7597fd2cab-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.584 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.584 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.585 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.585 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.585 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.585 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.585 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:87zSg4f79b89o7axtVlYTIH2sXA= 2024-04-12T04:56:00.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:87zSg4f79b89o7axtVlYTIH2sXA=', 'Content-Length': '0'} 2024-04-12T04:56:00.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9b3133a309cf805-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.585 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:56:00.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.585 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.586 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.586 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.586 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.586 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.586 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.586 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.586 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:8Ghuk8ws0gq7lXtyobRkBov+YRA= 2024-04-12T04:56:00.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8Ghuk8ws0gq7lXtyobRkBov+YRA=', 'Content-Length': '0'} 2024-04-12T04:56:00.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044353fcc770b632c-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.587 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.588 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.588 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.588 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.588 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.588 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.588 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:8Ghuk8ws0gq7lXtyobRkBov+YRA= 2024-04-12T04:56:00.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8Ghuk8ws0gq7lXtyobRkBov+YRA=', 'Content-Length': '0'} 2024-04-12T04:56:00.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003384be0b519f89f7-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.588 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=a 2024-04-12T04:56:00.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.589 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.589 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.590 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.590 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.590 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.590 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.590 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T04:56:00.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T04:56:00.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca9a6bad15c00750-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.590 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=a 2024-04-12T04:56:00.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.591 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.591 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.591 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.591 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.592 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.592 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.592 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T04:56:00.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T04:56:00.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a56ddc6909b18bb-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a 2024-04-12T04:56:00.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.592 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.593 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.593 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.593 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.593 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.593 INFO:tasks.rgw_multisite_tests:/dkyges-57/a 2024-04-12T04:56:00.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.593 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:99v6ThYq2z7oU/3h2XwoHG67rks= 2024-04-12T04:56:00.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:99v6ThYq2z7oU/3h2XwoHG67rks=', 'Content-Length': '0'} 2024-04-12T04:56:00.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006931108083a23ac9-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.593 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a 2024-04-12T04:56:00.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.594 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.594 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.594 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.594 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.594 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.595 INFO:tasks.rgw_multisite_tests:/dkyges-57/a 2024-04-12T04:56:00.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.595 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:99v6ThYq2z7oU/3h2XwoHG67rks= 2024-04-12T04:56:00.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:99v6ThYq2z7oU/3h2XwoHG67rks=', 'Content-Length': '0'} 2024-04-12T04:56:00.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9a7892825a0cef4-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.595 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:56:00.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a 2024-04-12T04:56:00.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.595 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.596 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.596 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.596 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.596 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.596 INFO:tasks.rgw_multisite_tests:/dkyges-57/a 2024-04-12T04:56:00.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.596 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:WHiCwMAZkwJ+CERBe8yXWZLC0Ww= 2024-04-12T04:56:00.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WHiCwMAZkwJ+CERBe8yXWZLC0Ww=', 'Content-Length': '0'} 2024-04-12T04:56:00.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000833158ddacbf5aa3-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a 2024-04-12T04:56:00.596 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.597 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.597 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.597 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.597 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.597 INFO:tasks.rgw_multisite_tests:/dkyges-57/a 2024-04-12T04:56:00.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.597 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:WHiCwMAZkwJ+CERBe8yXWZLC0Ww= 2024-04-12T04:56:00.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WHiCwMAZkwJ+CERBe8yXWZLC0Ww=', 'Content-Length': '0'} 2024-04-12T04:56:00.597 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c15a6a0d286aa593-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=a 2024-04-12T04:56:00.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.599 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.599 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.599 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.599 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.599 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.599 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T04:56:00.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T04:56:00.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000265f84edf8a364c-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=a 2024-04-12T04:56:00.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.600 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.600 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.600 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.600 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.600 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.600 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T04:56:00.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T04:56:00.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccb8e1cf49019550-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.601 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:56:00.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.602 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.602 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.602 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.602 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.602 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.602 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:+AE+mjoWDAKGQwxjN5WWXcxRZVY= 2024-04-12T04:56:00.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+AE+mjoWDAKGQwxjN5WWXcxRZVY=', 'Content-Length': '0'} 2024-04-12T04:56:00.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e419abd9127beac-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.603 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.603 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.603 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.603 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.603 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.603 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:+AE+mjoWDAKGQwxjN5WWXcxRZVY= 2024-04-12T04:56:00.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+AE+mjoWDAKGQwxjN5WWXcxRZVY=', 'Content-Length': '0'} 2024-04-12T04:56:00.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049bc91ea4f2e3dde-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.605 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.605 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.605 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.605 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.605 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.605 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:7IHUpHLBpsZOev5a4f7xPAnI6os= 2024-04-12T04:56:00.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7IHUpHLBpsZOev5a4f7xPAnI6os=', 'Content-Length': '0'} 2024-04-12T04:56:00.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035f2266281a43a9a-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.606 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.606 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.606 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.606 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.606 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.607 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:7IHUpHLBpsZOev5a4f7xPAnI6os= 2024-04-12T04:56:00.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7IHUpHLBpsZOev5a4f7xPAnI6os=', 'Content-Length': '0'} 2024-04-12T04:56:00.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce7fe3e02296e30d-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.607 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:56:00.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.608 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.608 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.608 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.608 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.608 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.608 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:+AE+mjoWDAKGQwxjN5WWXcxRZVY= 2024-04-12T04:56:00.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+AE+mjoWDAKGQwxjN5WWXcxRZVY=', 'Content-Length': '0'} 2024-04-12T04:56:00.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000281902a8e397ec47-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.609 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.609 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.609 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.609 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.609 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.610 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:+AE+mjoWDAKGQwxjN5WWXcxRZVY= 2024-04-12T04:56:00.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+AE+mjoWDAKGQwxjN5WWXcxRZVY=', 'Content-Length': '0'} 2024-04-12T04:56:00.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000005d90497be1521f-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=b 2024-04-12T04:56:00.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.611 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.611 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.611 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.611 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.611 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.611 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T04:56:00.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T04:56:00.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a92b7fec290b78ee-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=b 2024-04-12T04:56:00.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.612 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.612 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.613 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.613 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.613 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.613 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T04:56:00.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T04:56:00.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbd7c38755703d3f-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b 2024-04-12T04:56:00.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.614 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.614 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.614 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.614 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.614 INFO:tasks.rgw_multisite_tests:/dkyges-57/b 2024-04-12T04:56:00.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.614 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:H1nDxpbtrHt7+X3fDHfoEVMhjO0= 2024-04-12T04:56:00.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H1nDxpbtrHt7+X3fDHfoEVMhjO0=', 'Content-Length': '0'} 2024-04-12T04:56:00.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022f9020408034853-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b 2024-04-12T04:56:00.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.615 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.615 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.616 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.616 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.616 INFO:tasks.rgw_multisite_tests:/dkyges-57/b 2024-04-12T04:56:00.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.616 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:H1nDxpbtrHt7+X3fDHfoEVMhjO0= 2024-04-12T04:56:00.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H1nDxpbtrHt7+X3fDHfoEVMhjO0=', 'Content-Length': '0'} 2024-04-12T04:56:00.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009877591b251b3c62-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.616 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:56:00.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b 2024-04-12T04:56:00.616 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.617 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.617 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.617 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.617 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.617 INFO:tasks.rgw_multisite_tests:/dkyges-57/b 2024-04-12T04:56:00.617 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.617 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Wb+rG0+bvBQMSnWMPg8Yb7W1R4E= 2024-04-12T04:56:00.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wb+rG0+bvBQMSnWMPg8Yb7W1R4E=', 'Content-Length': '0'} 2024-04-12T04:56:00.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ba327a89a756043-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b 2024-04-12T04:56:00.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.619 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.619 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.619 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.619 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.619 INFO:tasks.rgw_multisite_tests:/dkyges-57/b 2024-04-12T04:56:00.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.619 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Wb+rG0+bvBQMSnWMPg8Yb7W1R4E= 2024-04-12T04:56:00.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Wb+rG0+bvBQMSnWMPg8Yb7W1R4E=', 'Content-Length': '0'} 2024-04-12T04:56:00.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005ef5cf0b3c036c1-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=b 2024-04-12T04:56:00.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.620 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.620 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.620 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.620 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.621 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.621 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T04:56:00.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T04:56:00.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae22acc41470fc76-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=b 2024-04-12T04:56:00.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.622 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.622 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.622 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.622 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.622 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.622 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T04:56:00.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T04:56:00.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063ce2c125a27a1d1-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.623 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:56:00.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.624 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.624 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.624 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.624 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.624 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.624 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:BzzoauNgcb8nZCKDjviLRVrZYZ8= 2024-04-12T04:56:00.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BzzoauNgcb8nZCKDjviLRVrZYZ8=', 'Content-Length': '0'} 2024-04-12T04:56:00.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055134680f4a7c226-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.625 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.625 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.625 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.625 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.625 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.625 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:BzzoauNgcb8nZCKDjviLRVrZYZ8= 2024-04-12T04:56:00.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BzzoauNgcb8nZCKDjviLRVrZYZ8=', 'Content-Length': '0'} 2024-04-12T04:56:00.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ef7509e21bedb24-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.627 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.627 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.627 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.627 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.627 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.627 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:thbk7vN6RGbBd4cphI8UmkI3Oc0= 2024-04-12T04:56:00.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:thbk7vN6RGbBd4cphI8UmkI3Oc0=', 'Content-Length': '0'} 2024-04-12T04:56:00.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c1e4ae42114f20b-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.628 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.628 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.628 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.628 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.628 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.628 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:thbk7vN6RGbBd4cphI8UmkI3Oc0= 2024-04-12T04:56:00.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:thbk7vN6RGbBd4cphI8UmkI3Oc0=', 'Content-Length': '0'} 2024-04-12T04:56:00.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7280cb750180a8b-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.629 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:56:00.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.630 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.630 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.630 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.630 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.630 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.630 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:BzzoauNgcb8nZCKDjviLRVrZYZ8= 2024-04-12T04:56:00.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BzzoauNgcb8nZCKDjviLRVrZYZ8=', 'Content-Length': '0'} 2024-04-12T04:56:00.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0b024b1d5b4c5f1-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.632 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.632 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.632 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.632 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.632 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.632 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:BzzoauNgcb8nZCKDjviLRVrZYZ8= 2024-04-12T04:56:00.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BzzoauNgcb8nZCKDjviLRVrZYZ8=', 'Content-Length': '0'} 2024-04-12T04:56:00.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000180b9a68e35eb7aa-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=c 2024-04-12T04:56:00.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.633 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.633 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.633 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.633 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.633 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.633 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T04:56:00.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T04:56:00.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd97ee6a0cf5681-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=c 2024-04-12T04:56:00.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.634 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.635 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.635 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.635 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.635 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.635 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T04:56:00.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T04:56:00.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f1882cec80680da-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c 2024-04-12T04:56:00.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.636 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.636 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.636 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.636 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.636 INFO:tasks.rgw_multisite_tests:/dkyges-57/c 2024-04-12T04:56:00.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.636 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:H1HrQB6nCHDY57qlHoXHbJCVImc= 2024-04-12T04:56:00.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H1HrQB6nCHDY57qlHoXHbJCVImc=', 'Content-Length': '0'} 2024-04-12T04:56:00.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbcbc2888d74e2c6-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c 2024-04-12T04:56:00.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.637 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.638 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.638 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.638 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.638 INFO:tasks.rgw_multisite_tests:/dkyges-57/c 2024-04-12T04:56:00.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.638 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:H1HrQB6nCHDY57qlHoXHbJCVImc= 2024-04-12T04:56:00.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:H1HrQB6nCHDY57qlHoXHbJCVImc=', 'Content-Length': '0'} 2024-04-12T04:56:00.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e8c9cb6e8d8ab89-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.638 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:56:00.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c 2024-04-12T04:56:00.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.639 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.639 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.639 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.639 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.639 INFO:tasks.rgw_multisite_tests:/dkyges-57/c 2024-04-12T04:56:00.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.639 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:HWtB/xNvrzzQ5BqpmjVtjhOvWSc= 2024-04-12T04:56:00.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HWtB/xNvrzzQ5BqpmjVtjhOvWSc=', 'Content-Length': '0'} 2024-04-12T04:56:00.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004abf7e0d18dc6a06-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c 2024-04-12T04:56:00.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.640 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.640 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.641 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.641 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.641 INFO:tasks.rgw_multisite_tests:/dkyges-57/c 2024-04-12T04:56:00.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.641 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:HWtB/xNvrzzQ5BqpmjVtjhOvWSc= 2024-04-12T04:56:00.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HWtB/xNvrzzQ5BqpmjVtjhOvWSc=', 'Content-Length': '0'} 2024-04-12T04:56:00.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c5867710e7d6a72-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=c 2024-04-12T04:56:00.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.642 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.642 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.642 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.642 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.642 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.642 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T04:56:00.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T04:56:00.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000523b9b8745d2a69d-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=c 2024-04-12T04:56:00.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.644 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.644 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.644 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.644 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.644 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.644 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T04:56:00.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T04:56:00.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8e201b3783612ce-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.644 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:56:00.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.645 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.645 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.645 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.645 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.645 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.646 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:SaSRyJLItx11IubJLbqp44WAzyA= 2024-04-12T04:56:00.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SaSRyJLItx11IubJLbqp44WAzyA=', 'Content-Length': '0'} 2024-04-12T04:56:00.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d02e1e54c6dda999-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.647 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.647 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.647 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.647 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.647 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.647 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:SaSRyJLItx11IubJLbqp44WAzyA= 2024-04-12T04:56:00.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SaSRyJLItx11IubJLbqp44WAzyA=', 'Content-Length': '0'} 2024-04-12T04:56:00.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f1736bb9db9d0f3-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.648 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.648 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.648 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.649 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.649 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.649 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:1IpYRKFvBinT9/G2wOCxfTxBpzA= 2024-04-12T04:56:00.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1IpYRKFvBinT9/G2wOCxfTxBpzA=', 'Content-Length': '0'} 2024-04-12T04:56:00.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061d969b489f19849-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.650 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.650 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.650 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.650 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.650 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.650 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:1IpYRKFvBinT9/G2wOCxfTxBpzA= 2024-04-12T04:56:00.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1IpYRKFvBinT9/G2wOCxfTxBpzA=', 'Content-Length': '0'} 2024-04-12T04:56:00.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a250e2b64f7adb9-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.651 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:56:00.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.652 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.652 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.652 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.652 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.652 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.652 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:SaSRyJLItx11IubJLbqp44WAzyA= 2024-04-12T04:56:00.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SaSRyJLItx11IubJLbqp44WAzyA=', 'Content-Length': '0'} 2024-04-12T04:56:00.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c3673b5e4f5eff9-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.653 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.653 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.653 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.654 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.654 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.654 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:SaSRyJLItx11IubJLbqp44WAzyA= 2024-04-12T04:56:00.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SaSRyJLItx11IubJLbqp44WAzyA=', 'Content-Length': '0'} 2024-04-12T04:56:00.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004cdc843b0f6b496-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=d 2024-04-12T04:56:00.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.655 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.655 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.655 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.655 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.655 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.655 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T04:56:00.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T04:56:00.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7fce050f5e60a82-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=d 2024-04-12T04:56:00.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.657 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.657 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.657 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.657 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.657 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.657 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T04:56:00.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T04:56:00.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a0edeb884461d64-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d 2024-04-12T04:56:00.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.658 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.658 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.659 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.659 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.659 INFO:tasks.rgw_multisite_tests:/dkyges-57/d 2024-04-12T04:56:00.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.659 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ibIj1tvrWZYFFILmUa70Qj4kvrs= 2024-04-12T04:56:00.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ibIj1tvrWZYFFILmUa70Qj4kvrs=', 'Content-Length': '0'} 2024-04-12T04:56:00.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029eae1fbcf89b37d-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d 2024-04-12T04:56:00.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.660 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.660 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.660 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.660 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.660 INFO:tasks.rgw_multisite_tests:/dkyges-57/d 2024-04-12T04:56:00.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.660 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ibIj1tvrWZYFFILmUa70Qj4kvrs= 2024-04-12T04:56:00.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ibIj1tvrWZYFFILmUa70Qj4kvrs=', 'Content-Length': '0'} 2024-04-12T04:56:00.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072fcf3c72ddf0ab6-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.661 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:56:00.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d 2024-04-12T04:56:00.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.661 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.661 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.661 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.662 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.662 INFO:tasks.rgw_multisite_tests:/dkyges-57/d 2024-04-12T04:56:00.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.662 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:16DzJ6BEZR4Tp1UwNXII03TRNa0= 2024-04-12T04:56:00.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:16DzJ6BEZR4Tp1UwNXII03TRNa0=', 'Content-Length': '0'} 2024-04-12T04:56:00.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb0872f5e347ea25-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d 2024-04-12T04:56:00.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.663 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.663 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.663 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.663 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.663 INFO:tasks.rgw_multisite_tests:/dkyges-57/d 2024-04-12T04:56:00.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.663 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:16DzJ6BEZR4Tp1UwNXII03TRNa0= 2024-04-12T04:56:00.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:16DzJ6BEZR4Tp1UwNXII03TRNa0=', 'Content-Length': '0'} 2024-04-12T04:56:00.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000460a2c1df385691c-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=d 2024-04-12T04:56:00.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.664 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.664 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.664 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.664 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.664 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.664 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.664 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T04:56:00.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T04:56:00.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052a571110cafbd48-006618afe8-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=d 2024-04-12T04:56:00.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.666 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.666 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.666 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.666 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.666 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.666 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew= 2024-04-12T04:56:00.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jR4MuZp/PMKcOAx43uS531RsCew=', 'Content-Length': '0'} 2024-04-12T04:56:00.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065981da435252ddb-006618afe8-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.666 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-12T04:56:00.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e?versionId=null 2024-04-12T04:56:00.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.667 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.667 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.667 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.667 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.667 INFO:tasks.rgw_multisite_tests:/dkyges-57/e?versionId=null 2024-04-12T04:56:00.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.668 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:q89BiJwl2t9qUozc6JIu60aKJnU= 2024-04-12T04:56:00.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q89BiJwl2t9qUozc6JIu60aKJnU=', 'Content-Length': '0'} 2024-04-12T04:56:00.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089d335046b30f9b7-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e?versionId=null 2024-04-12T04:56:00.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.669 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.669 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.669 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.669 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.669 INFO:tasks.rgw_multisite_tests:/dkyges-57/e?versionId=null 2024-04-12T04:56:00.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.669 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:q89BiJwl2t9qUozc6JIu60aKJnU= 2024-04-12T04:56:00.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q89BiJwl2t9qUozc6JIu60aKJnU=', 'Content-Length': '0'} 2024-04-12T04:56:00.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063df1acc0935d1a7-006618afe8-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:08 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e?versionId=null 2024-04-12T04:56:00.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.670 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.670 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.670 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.670 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.671 INFO:tasks.rgw_multisite_tests:/dkyges-57/e?versionId=null 2024-04-12T04:56:00.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.671 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:pbGDEzqbsNxBbVjNC0Q9TyuZwMI= 2024-04-12T04:56:00.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pbGDEzqbsNxBbVjNC0Q9TyuZwMI=', 'Content-Length': '0'} 2024-04-12T04:56:00.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de0036acc80dc936-006618afe8-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e?versionId=null 2024-04-12T04:56:00.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.672 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.672 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.672 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.672 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.672 INFO:tasks.rgw_multisite_tests:/dkyges-57/e?versionId=null 2024-04-12T04:56:00.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.672 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:pbGDEzqbsNxBbVjNC0Q9TyuZwMI= 2024-04-12T04:56:00.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:pbGDEzqbsNxBbVjNC0Q9TyuZwMI=', 'Content-Length': '0'} 2024-04-12T04:56:00.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fccdf9147a0352df-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.673 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-12T04:56:00.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e?versionId=null 2024-04-12T04:56:00.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.674 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.674 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.674 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.674 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:08 GMT 2024-04-12T04:56:00.674 INFO:tasks.rgw_multisite_tests:/dkyges-57/e?versionId=null 2024-04-12T04:56:00.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.674 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:q89BiJwl2t9qUozc6JIu60aKJnU= 2024-04-12T04:56:00.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:08 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q89BiJwl2t9qUozc6JIu60aKJnU=', 'Content-Length': '0'} 2024-04-12T04:56:00.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f15598d78ff83e79-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e?versionId=null 2024-04-12T04:56:00.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.675 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.675 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.675 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.675 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.675 INFO:tasks.rgw_multisite_tests:/dkyges-57/e?versionId=null 2024-04-12T04:56:00.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.676 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Aw/Z4FqqhzVaZjuUJ3E+xwoIKnw= 2024-04-12T04:56:00.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Aw/Z4FqqhzVaZjuUJ3E+xwoIKnw=', 'Content-Length': '0'} 2024-04-12T04:56:00.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073731d79475d7049-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=e 2024-04-12T04:56:00.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=e 2024-04-12T04:56:00.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=e 2024-04-12T04:56:00.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.677 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.677 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.677 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.677 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.677 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.677 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T04:56:00.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T04:56:00.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000158c628c9102b9dd-006618afe9-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57e1000falsee2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=e 2024-04-12T04:56:00.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=e 2024-04-12T04:56:00.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=e 2024-04-12T04:56:00.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.678 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.678 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.678 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.678 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.679 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.679 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T04:56:00.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T04:56:00.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000579c726f61c98331-006618afe9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57e1000falsee2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e 2024-04-12T04:56:00.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.680 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.680 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.680 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.680 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.680 INFO:tasks.rgw_multisite_tests:/dkyges-57/e 2024-04-12T04:56:00.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.680 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:9QgWUojv7EAgIbj+/3mq4NkwK/A= 2024-04-12T04:56:00.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9QgWUojv7EAgIbj+/3mq4NkwK/A=', 'Content-Length': '0'} 2024-04-12T04:56:00.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dffab6bd09ff4183-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e 2024-04-12T04:56:00.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.681 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.681 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.681 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.682 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.682 INFO:tasks.rgw_multisite_tests:/dkyges-57/e 2024-04-12T04:56:00.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.682 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:9QgWUojv7EAgIbj+/3mq4NkwK/A= 2024-04-12T04:56:00.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9QgWUojv7EAgIbj+/3mq4NkwK/A=', 'Content-Length': '0'} 2024-04-12T04:56:00.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c993c93ae4ca45c-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.682 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-12T04:56:00.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e 2024-04-12T04:56:00.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.683 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.683 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.683 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.683 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.683 INFO:tasks.rgw_multisite_tests:/dkyges-57/e 2024-04-12T04:56:00.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.683 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:EPkGI6dpfggOwomWYLsVNfcMHwQ= 2024-04-12T04:56:00.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EPkGI6dpfggOwomWYLsVNfcMHwQ=', 'Content-Length': '0'} 2024-04-12T04:56:00.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0d0069e2f8819ef-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e 2024-04-12T04:56:00.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.684 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.684 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.684 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.684 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.684 INFO:tasks.rgw_multisite_tests:/dkyges-57/e 2024-04-12T04:56:00.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.685 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:EPkGI6dpfggOwomWYLsVNfcMHwQ= 2024-04-12T04:56:00.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:EPkGI6dpfggOwomWYLsVNfcMHwQ=', 'Content-Length': '0'} 2024-04-12T04:56:00.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b7af509dcf15dea-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=e 2024-04-12T04:56:00.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=e 2024-04-12T04:56:00.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=e 2024-04-12T04:56:00.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.686 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.686 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.686 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.686 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.686 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.686 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T04:56:00.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T04:56:00.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068e9796ccca152b7-006618afe9-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57e1000falsee2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=e 2024-04-12T04:56:00.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=e 2024-04-12T04:56:00.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=e 2024-04-12T04:56:00.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.687 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.687 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.687 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.687 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.687 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.687 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T04:56:00.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T04:56:00.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b73e4d3959898845-006618afe9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57e1000falsee2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.688 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-12T04:56:00.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f?versionId=null 2024-04-12T04:56:00.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.689 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.689 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.689 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.689 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.689 INFO:tasks.rgw_multisite_tests:/dkyges-57/f?versionId=null 2024-04-12T04:56:00.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.689 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:njH1nDiczU9QU/NbNXKIXA9xlXc= 2024-04-12T04:56:00.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:njH1nDiczU9QU/NbNXKIXA9xlXc=', 'Content-Length': '0'} 2024-04-12T04:56:00.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd4e9cfd7311e925-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f?versionId=null 2024-04-12T04:56:00.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.690 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.690 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.690 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.690 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.691 INFO:tasks.rgw_multisite_tests:/dkyges-57/f?versionId=null 2024-04-12T04:56:00.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.691 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:njH1nDiczU9QU/NbNXKIXA9xlXc= 2024-04-12T04:56:00.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:njH1nDiczU9QU/NbNXKIXA9xlXc=', 'Content-Length': '0'} 2024-04-12T04:56:00.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000239c989bed5c5745-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f?versionId=null 2024-04-12T04:56:00.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.692 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.692 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.692 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.692 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.692 INFO:tasks.rgw_multisite_tests:/dkyges-57/f?versionId=null 2024-04-12T04:56:00.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.692 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:fzppvKa60NzV9RgL3+6WfpTy4ZU= 2024-04-12T04:56:00.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fzppvKa60NzV9RgL3+6WfpTy4ZU=', 'Content-Length': '0'} 2024-04-12T04:56:00.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f31257f8a65562cb-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f?versionId=null 2024-04-12T04:56:00.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.693 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.694 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.694 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.694 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.694 INFO:tasks.rgw_multisite_tests:/dkyges-57/f?versionId=null 2024-04-12T04:56:00.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.694 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:fzppvKa60NzV9RgL3+6WfpTy4ZU= 2024-04-12T04:56:00.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fzppvKa60NzV9RgL3+6WfpTy4ZU=', 'Content-Length': '0'} 2024-04-12T04:56:00.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c5e6c7ac4cf0dc6-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.694 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-12T04:56:00.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f?versionId=null 2024-04-12T04:56:00.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.695 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.695 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.695 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.695 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.695 INFO:tasks.rgw_multisite_tests:/dkyges-57/f?versionId=null 2024-04-12T04:56:00.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.696 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:njH1nDiczU9QU/NbNXKIXA9xlXc= 2024-04-12T04:56:00.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:njH1nDiczU9QU/NbNXKIXA9xlXc=', 'Content-Length': '0'} 2024-04-12T04:56:00.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004388f09198b14a6d-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f?versionId=null 2024-04-12T04:56:00.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.697 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.697 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.697 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.697 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.697 INFO:tasks.rgw_multisite_tests:/dkyges-57/f?versionId=null 2024-04-12T04:56:00.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.697 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:njH1nDiczU9QU/NbNXKIXA9xlXc= 2024-04-12T04:56:00.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:njH1nDiczU9QU/NbNXKIXA9xlXc=', 'Content-Length': '0'} 2024-04-12T04:56:00.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009af2af26f9ca00c0-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=f 2024-04-12T04:56:00.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=f 2024-04-12T04:56:00.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=f 2024-04-12T04:56:00.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.698 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.698 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.699 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.699 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.699 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.699 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T04:56:00.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T04:56:00.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eb971fd06f06f6a-006618afe9-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57f1000falsef2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=f 2024-04-12T04:56:00.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=f 2024-04-12T04:56:00.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=f 2024-04-12T04:56:00.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.700 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.700 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.700 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.700 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.700 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.701 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T04:56:00.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T04:56:00.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065ddd1ef748dee76-006618afe9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57f1000falsef2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f 2024-04-12T04:56:00.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.702 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.702 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.702 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.702 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.702 INFO:tasks.rgw_multisite_tests:/dkyges-57/f 2024-04-12T04:56:00.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.702 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:bckDdjNiGB0KGSyb9N62Se/noHY= 2024-04-12T04:56:00.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bckDdjNiGB0KGSyb9N62Se/noHY=', 'Content-Length': '0'} 2024-04-12T04:56:00.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000136bc644439cb2df-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f 2024-04-12T04:56:00.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.703 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.703 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.703 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.703 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.703 INFO:tasks.rgw_multisite_tests:/dkyges-57/f 2024-04-12T04:56:00.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.704 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:bckDdjNiGB0KGSyb9N62Se/noHY= 2024-04-12T04:56:00.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bckDdjNiGB0KGSyb9N62Se/noHY=', 'Content-Length': '0'} 2024-04-12T04:56:00.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bb995881b2b880d-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.704 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-12T04:56:00.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f 2024-04-12T04:56:00.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.705 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.705 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.705 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.705 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.705 INFO:tasks.rgw_multisite_tests:/dkyges-57/f 2024-04-12T04:56:00.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.705 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ZSXCjSvc/OjwM2iq0xYTDRZk4OM= 2024-04-12T04:56:00.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZSXCjSvc/OjwM2iq0xYTDRZk4OM=', 'Content-Length': '0'} 2024-04-12T04:56:00.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd3857cc8739d5e4-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f 2024-04-12T04:56:00.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.706 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.706 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.706 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.706 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.706 INFO:tasks.rgw_multisite_tests:/dkyges-57/f 2024-04-12T04:56:00.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.706 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ZSXCjSvc/OjwM2iq0xYTDRZk4OM= 2024-04-12T04:56:00.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZSXCjSvc/OjwM2iq0xYTDRZk4OM=', 'Content-Length': '0'} 2024-04-12T04:56:00.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d68ea6530471840-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=f 2024-04-12T04:56:00.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=f 2024-04-12T04:56:00.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=f 2024-04-12T04:56:00.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.708 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.708 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.708 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.708 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.708 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.708 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T04:56:00.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T04:56:00.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7a0024e1571f1cc-006618afe9-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57f1000falsef2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=f 2024-04-12T04:56:00.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=f 2024-04-12T04:56:00.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=f 2024-04-12T04:56:00.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.709 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.709 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.709 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.709 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.709 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.710 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T04:56:00.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T04:56:00.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db15527bbd62bd0e-006618afe9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57f1000falsef2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.710 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-12T04:56:00.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g?versionId=null 2024-04-12T04:56:00.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.711 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.711 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.711 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.711 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.711 INFO:tasks.rgw_multisite_tests:/dkyges-57/g?versionId=null 2024-04-12T04:56:00.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.711 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:bh2BoZgaF1X9j9/w2zWsQ0Qpi5g= 2024-04-12T04:56:00.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bh2BoZgaF1X9j9/w2zWsQ0Qpi5g=', 'Content-Length': '0'} 2024-04-12T04:56:00.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049ed5d2cd54c06c9-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g?versionId=null 2024-04-12T04:56:00.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.713 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.713 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.713 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.713 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.713 INFO:tasks.rgw_multisite_tests:/dkyges-57/g?versionId=null 2024-04-12T04:56:00.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.713 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:bh2BoZgaF1X9j9/w2zWsQ0Qpi5g= 2024-04-12T04:56:00.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bh2BoZgaF1X9j9/w2zWsQ0Qpi5g=', 'Content-Length': '0'} 2024-04-12T04:56:00.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097faad90354984b4-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g?versionId=null 2024-04-12T04:56:00.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.714 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.714 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.714 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.714 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.714 INFO:tasks.rgw_multisite_tests:/dkyges-57/g?versionId=null 2024-04-12T04:56:00.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.715 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gLo7WklETjIPHN+HYXO9FAprDEo= 2024-04-12T04:56:00.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gLo7WklETjIPHN+HYXO9FAprDEo=', 'Content-Length': '0'} 2024-04-12T04:56:00.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cc02bfd260fceeb-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g?versionId=null 2024-04-12T04:56:00.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.716 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.716 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.716 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.716 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.716 INFO:tasks.rgw_multisite_tests:/dkyges-57/g?versionId=null 2024-04-12T04:56:00.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.716 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gLo7WklETjIPHN+HYXO9FAprDEo= 2024-04-12T04:56:00.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gLo7WklETjIPHN+HYXO9FAprDEo=', 'Content-Length': '0'} 2024-04-12T04:56:00.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b98da4d31e62774-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.716 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-12T04:56:00.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g?versionId=null 2024-04-12T04:56:00.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.717 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.717 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.717 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.717 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.717 INFO:tasks.rgw_multisite_tests:/dkyges-57/g?versionId=null 2024-04-12T04:56:00.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.718 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:bh2BoZgaF1X9j9/w2zWsQ0Qpi5g= 2024-04-12T04:56:00.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bh2BoZgaF1X9j9/w2zWsQ0Qpi5g=', 'Content-Length': '0'} 2024-04-12T04:56:00.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3c6b1a5a45ad686-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g?versionId=null 2024-04-12T04:56:00.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.719 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.719 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.719 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.719 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.719 INFO:tasks.rgw_multisite_tests:/dkyges-57/g?versionId=null 2024-04-12T04:56:00.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.719 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:bh2BoZgaF1X9j9/w2zWsQ0Qpi5g= 2024-04-12T04:56:00.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bh2BoZgaF1X9j9/w2zWsQ0Qpi5g=', 'Content-Length': '0'} 2024-04-12T04:56:00.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023416118a596420a-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=g 2024-04-12T04:56:00.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=g 2024-04-12T04:56:00.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=g 2024-04-12T04:56:00.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.720 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.721 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.721 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.721 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.721 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.721 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T04:56:00.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T04:56:00.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002051cc43332e060-006618afe9-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57g1000falseg2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=g 2024-04-12T04:56:00.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=g 2024-04-12T04:56:00.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=g 2024-04-12T04:56:00.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.722 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.722 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.722 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.722 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.722 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.723 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T04:56:00.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T04:56:00.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006eb319b125ecb9f4-006618afe9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57g1000falseg2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g 2024-04-12T04:56:00.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.724 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.724 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.724 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.724 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.724 INFO:tasks.rgw_multisite_tests:/dkyges-57/g 2024-04-12T04:56:00.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.724 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:46J5ArzLuULtB68uTm6zt/9YPOA= 2024-04-12T04:56:00.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:46J5ArzLuULtB68uTm6zt/9YPOA=', 'Content-Length': '0'} 2024-04-12T04:56:00.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062e0e10ba82b9d7d-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g 2024-04-12T04:56:00.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.725 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.726 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.726 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.726 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.726 INFO:tasks.rgw_multisite_tests:/dkyges-57/g 2024-04-12T04:56:00.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.726 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:46J5ArzLuULtB68uTm6zt/9YPOA= 2024-04-12T04:56:00.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:46J5ArzLuULtB68uTm6zt/9YPOA=', 'Content-Length': '0'} 2024-04-12T04:56:00.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a80255d1c299bc94-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.726 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-12T04:56:00.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g 2024-04-12T04:56:00.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.727 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.727 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.727 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.727 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.727 INFO:tasks.rgw_multisite_tests:/dkyges-57/g 2024-04-12T04:56:00.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.727 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sh8mSx1bcnkXXdsmf8OX7g689NM= 2024-04-12T04:56:00.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sh8mSx1bcnkXXdsmf8OX7g689NM=', 'Content-Length': '0'} 2024-04-12T04:56:00.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f72e877e42e0330-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g 2024-04-12T04:56:00.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.728 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.729 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.729 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.729 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.729 INFO:tasks.rgw_multisite_tests:/dkyges-57/g 2024-04-12T04:56:00.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.729 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:sh8mSx1bcnkXXdsmf8OX7g689NM= 2024-04-12T04:56:00.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:sh8mSx1bcnkXXdsmf8OX7g689NM=', 'Content-Length': '0'} 2024-04-12T04:56:00.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ab3dbe8e1867b73-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=g 2024-04-12T04:56:00.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=g 2024-04-12T04:56:00.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=g 2024-04-12T04:56:00.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.730 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.730 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.730 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.730 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.730 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.731 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T04:56:00.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T04:56:00.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000407c66824c6d7d8-006618afe9-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57g1000falseg2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=g 2024-04-12T04:56:00.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=g 2024-04-12T04:56:00.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=g 2024-04-12T04:56:00.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.732 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.732 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.732 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.732 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.732 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.732 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T04:56:00.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T04:56:00.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002191faa56775886c-006618afe9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57g1000falseg2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.733 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-12T04:56:00.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h?versionId=null 2024-04-12T04:56:00.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.734 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.734 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.734 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.734 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.734 INFO:tasks.rgw_multisite_tests:/dkyges-57/h?versionId=null 2024-04-12T04:56:00.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.734 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:JS3xVsvJtGhF8Ws0XtUa7GPAMPk= 2024-04-12T04:56:00.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JS3xVsvJtGhF8Ws0XtUa7GPAMPk=', 'Content-Length': '0'} 2024-04-12T04:56:00.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035f395213db9fad2-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h?versionId=null 2024-04-12T04:56:00.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.735 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.735 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.735 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.735 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.735 INFO:tasks.rgw_multisite_tests:/dkyges-57/h?versionId=null 2024-04-12T04:56:00.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.736 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:JS3xVsvJtGhF8Ws0XtUa7GPAMPk= 2024-04-12T04:56:00.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JS3xVsvJtGhF8Ws0XtUa7GPAMPk=', 'Content-Length': '0'} 2024-04-12T04:56:00.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004139ac85ddf57896-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h?versionId=null 2024-04-12T04:56:00.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.737 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.737 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.737 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.737 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.737 INFO:tasks.rgw_multisite_tests:/dkyges-57/h?versionId=null 2024-04-12T04:56:00.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.737 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:G8C9tnD8MgJOjrt3A5jPFeG+7rM= 2024-04-12T04:56:00.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G8C9tnD8MgJOjrt3A5jPFeG+7rM=', 'Content-Length': '0'} 2024-04-12T04:56:00.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e239d8cdfeadcfbb-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h?versionId=null 2024-04-12T04:56:00.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.738 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.738 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.738 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.738 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.738 INFO:tasks.rgw_multisite_tests:/dkyges-57/h?versionId=null 2024-04-12T04:56:00.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.739 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:G8C9tnD8MgJOjrt3A5jPFeG+7rM= 2024-04-12T04:56:00.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:G8C9tnD8MgJOjrt3A5jPFeG+7rM=', 'Content-Length': '0'} 2024-04-12T04:56:00.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027b9aea8800d6240-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.739 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-12T04:56:00.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h?versionId=null 2024-04-12T04:56:00.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.740 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.740 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.740 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.740 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.740 INFO:tasks.rgw_multisite_tests:/dkyges-57/h?versionId=null 2024-04-12T04:56:00.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.740 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:JS3xVsvJtGhF8Ws0XtUa7GPAMPk= 2024-04-12T04:56:00.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JS3xVsvJtGhF8Ws0XtUa7GPAMPk=', 'Content-Length': '0'} 2024-04-12T04:56:00.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd6f8887951e82f0-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h?versionId=null 2024-04-12T04:56:00.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.741 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.742 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.742 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.742 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.742 INFO:tasks.rgw_multisite_tests:/dkyges-57/h?versionId=null 2024-04-12T04:56:00.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.742 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:JS3xVsvJtGhF8Ws0XtUa7GPAMPk= 2024-04-12T04:56:00.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:JS3xVsvJtGhF8Ws0XtUa7GPAMPk=', 'Content-Length': '0'} 2024-04-12T04:56:00.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054c37af7086f4df0-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=h 2024-04-12T04:56:00.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=h 2024-04-12T04:56:00.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=h 2024-04-12T04:56:00.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.743 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.743 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.743 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.743 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.743 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.743 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T04:56:00.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T04:56:00.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004300c004407ad87a-006618afe9-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57h1000falseh2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=h 2024-04-12T04:56:00.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=h 2024-04-12T04:56:00.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=h 2024-04-12T04:56:00.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.745 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.745 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.745 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.745 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.745 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.745 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T04:56:00.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T04:56:00.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003db73fb21c9183b3-006618afe9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57h1000falseh2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h 2024-04-12T04:56:00.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.746 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.746 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.747 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.747 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.747 INFO:tasks.rgw_multisite_tests:/dkyges-57/h 2024-04-12T04:56:00.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.747 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:zRpm0g/Dbe3JbrzoKhmChhv2GFs= 2024-04-12T04:56:00.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zRpm0g/Dbe3JbrzoKhmChhv2GFs=', 'Content-Length': '0'} 2024-04-12T04:56:00.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4bdbaa0555be112-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h 2024-04-12T04:56:00.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.748 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.748 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.748 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.748 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.748 INFO:tasks.rgw_multisite_tests:/dkyges-57/h 2024-04-12T04:56:00.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.748 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:zRpm0g/Dbe3JbrzoKhmChhv2GFs= 2024-04-12T04:56:00.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zRpm0g/Dbe3JbrzoKhmChhv2GFs=', 'Content-Length': '0'} 2024-04-12T04:56:00.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000330e44a40f407b12-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.749 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-12T04:56:00.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h 2024-04-12T04:56:00.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.749 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.749 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.749 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.750 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.750 INFO:tasks.rgw_multisite_tests:/dkyges-57/h 2024-04-12T04:56:00.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.750 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:goIijJL+Jcrj/hMqRqfmNdfVup0= 2024-04-12T04:56:00.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:goIijJL+Jcrj/hMqRqfmNdfVup0=', 'Content-Length': '0'} 2024-04-12T04:56:00.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001755bd0e9e876b46-006618afe9-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h 2024-04-12T04:56:00.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.751 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.751 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.751 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.751 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.751 INFO:tasks.rgw_multisite_tests:/dkyges-57/h 2024-04-12T04:56:00.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.751 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:goIijJL+Jcrj/hMqRqfmNdfVup0= 2024-04-12T04:56:00.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:goIijJL+Jcrj/hMqRqfmNdfVup0=', 'Content-Length': '0'} 2024-04-12T04:56:00.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d388f9793d44b317-006618afe9-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=h 2024-04-12T04:56:00.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=h 2024-04-12T04:56:00.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=h 2024-04-12T04:56:00.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.752 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.752 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.752 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.752 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.753 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.753 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T04:56:00.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T04:56:00.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf4ee1602a2458d1-006618afe9-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57h1000falseh2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=h 2024-04-12T04:56:00.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=h 2024-04-12T04:56:00.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=h 2024-04-12T04:56:00.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.754 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.754 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.754 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.754 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:09 GMT 2024-04-12T04:56:00.754 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.754 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M= 2024-04-12T04:56:00.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:09 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:gyQYpdljZkdh+xv30OTwF2OYd8M=', 'Content-Length': '0'} 2024-04-12T04:56:00.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb490b5303eb8c49-006618afe9-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:09 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57h1000falseh2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.755 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=dkyges-57 zones={a2, a1} 2024-04-12T04:56:00.755 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'dkyges-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.755 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'dkyges-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket dkyges-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:00.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: tenant: 2024-04-12T04:56:00.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: bucket name: dkyges-57 2024-04-12T04:56:00.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: total entries: 8 2024-04-12T04:56:00.755 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.755 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dkyges-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:00.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "resharding": "None", 2024-04-12T04:56:00.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_index": { 2024-04-12T04:56:00.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 2, 2024-04-12T04:56:00.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "type": "Normal", 2024-04-12T04:56:00.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "normal": { 2024-04-12T04:56:00.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 3, 2024-04-12T04:56:00.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:56:00.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "logs": [ 2024-04-12T04:56:00.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 0, 2024-04-12T04:56:00.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "type": "InIndex", 2024-04-12T04:56:00.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "in_index": { 2024-04-12T04:56:00.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 0, 2024-04-12T04:56:00.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 11, 2024-04-12T04:56:00.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:56:00.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 1, 2024-04-12T04:56:00.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "type": "InIndex", 2024-04-12T04:56:00.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "in_index": { 2024-04-12T04:56:00.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 1, 2024-04-12T04:56:00.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 3, 2024-04-12T04:56:00.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:56:00.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 2, 2024-04-12T04:56:00.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "type": "InIndex", 2024-04-12T04:56:00.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "in_index": { 2024-04-12T04:56:00.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 2, 2024-04-12T04:56:00.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 3, 2024-04-12T04:56:00.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:56:00.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.759 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dkyges-57 2024-04-12T04:56:00.760 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:00.760 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:00.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:56:00.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:52:09.767+0000 7f4422409780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T04:56:00.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:52:14.767+0000 7f4422409780 1 waiting to reach latest gen 2, on 1.. 2024-04-12T04:56:00.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:52:19.768+0000 7f4422409780 1 bucket sync caught up with empty source 2024-04-12T04:56:00.760 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:52:19.768+0000 7f4422409780 0 bucket checkpoint complete 2024-04-12T04:56:00.760 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=dkyges-57 2024-04-12T04:56:00.761 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:00.761 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:00.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:56:00.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:52:19.895+0000 7ff2727f9780 1 bucket sync caught up with source: 2024-04-12T04:56:00.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [, 00000000002.2910.6, , , 00000000001.3705.6, 00000000001.1066.6, , , 00000000002.1606.6, , 00000000002.3706.6] 2024-04-12T04:56:00.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [, 00000000002.2910.6, , , 00000000001.3705.6, 00000000001.1066.6, , , 00000000002.1606.6, , 00000000002.3706.6] 2024-04-12T04:56:00.761 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T03:52:19.895+0000 7ff2727f9780 0 bucket checkpoint complete 2024-04-12T04:56:00.761 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=dkyges-57 zones={a2, a1} 2024-04-12T04:56:00.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/ 2024-04-12T04:56:00.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.762 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.762 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.762 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.762 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:19 GMT 2024-04-12T04:56:00.763 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.763 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:qEnChDTCyFDPGew7fIvwxBdd6uk= 2024-04-12T04:56:00.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qEnChDTCyFDPGew7fIvwxBdd6uk=', 'Content-Length': '0'} 2024-04-12T04:56:00.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000403e8f629577ee0-006618aff3-16de-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/ 2024-04-12T04:56:00.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.764 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.764 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.764 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.764 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:19 GMT 2024-04-12T04:56:00.764 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.764 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:qEnChDTCyFDPGew7fIvwxBdd6uk= 2024-04-12T04:56:00.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qEnChDTCyFDPGew7fIvwxBdd6uk=', 'Content-Length': '0'} 2024-04-12T04:56:00.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dffdeeb8b09322b9-006618aff3-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.764 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-12T04:56:00.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.765 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.765 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.765 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.765 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:19 GMT 2024-04-12T04:56:00.765 INFO:tasks.rgw_multisite_tests:/dkyges-57/?versions 2024-04-12T04:56:00.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.766 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:mMI7AYg6pH3d6cAPfLYn8jGhs2o= 2024-04-12T04:56:00.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mMI7AYg6pH3d6cAPfLYn8jGhs2o=', 'Content-Length': '0'} 2024-04-12T04:56:00.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf0aaa5d4e467c63-006618aff3-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.766 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-12T04:56:00.766 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-12T04:56:00.766 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-12T04:56:00.766 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-12T04:56:00.766 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2024-04-12T04:56:00.766 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2024-04-12T04:56:00.766 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2024-04-12T04:56:00.766 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2024-04-12T04:56:00.767 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-12T04:56:00.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.767 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.768 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.768 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.768 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:19 GMT 2024-04-12T04:56:00.768 INFO:tasks.rgw_multisite_tests:/dkyges-57/?versions 2024-04-12T04:56:00.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.768 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:mMI7AYg6pH3d6cAPfLYn8jGhs2o= 2024-04-12T04:56:00.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mMI7AYg6pH3d6cAPfLYn8jGhs2o=', 'Content-Length': '0'} 2024-04-12T04:56:00.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008036eeadac0615b5-006618aff3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.768 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2024-04-12T04:56:00.768 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2024-04-12T04:56:00.768 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2024-04-12T04:56:00.768 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2024-04-12T04:56:00.768 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2024-04-12T04:56:00.769 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2024-04-12T04:56:00.769 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2024-04-12T04:56:00.769 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2024-04-12T04:56:00.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.770 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.770 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.770 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.770 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:19 GMT 2024-04-12T04:56:00.770 INFO:tasks.rgw_multisite_tests:/dkyges-57/?versions 2024-04-12T04:56:00.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.770 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:mMI7AYg6pH3d6cAPfLYn8jGhs2o= 2024-04-12T04:56:00.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mMI7AYg6pH3d6cAPfLYn8jGhs2o=', 'Content-Length': '0'} 2024-04-12T04:56:00.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d128b873cc2c1490-006618aff3-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?versions&max-keys=999 2024-04-12T04:56:00.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.771 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.772 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.772 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.772 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:19 GMT 2024-04-12T04:56:00.772 INFO:tasks.rgw_multisite_tests:/dkyges-57/?versions 2024-04-12T04:56:00.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.772 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:mMI7AYg6pH3d6cAPfLYn8jGhs2o= 2024-04-12T04:56:00.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:mMI7AYg6pH3d6cAPfLYn8jGhs2o=', 'Content-Length': '0'} 2024-04-12T04:56:00.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b8838d04a738c12-006618aff3-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57999falseanulltrue2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.772 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:56:00.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.773 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.773 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.773 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.773 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:19 GMT 2024-04-12T04:56:00.773 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.774 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Xkz/s9cbmNYHIDg02kUK1WFOG5I= 2024-04-12T04:56:00.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Xkz/s9cbmNYHIDg02kUK1WFOG5I=', 'Content-Length': '0'} 2024-04-12T04:56:00.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0f9279a074bf2d2-006618aff3-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.775 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.775 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.775 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.775 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:19 GMT 2024-04-12T04:56:00.775 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.775 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Xkz/s9cbmNYHIDg02kUK1WFOG5I= 2024-04-12T04:56:00.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Xkz/s9cbmNYHIDg02kUK1WFOG5I=', 'Content-Length': '0'} 2024-04-12T04:56:00.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0dc7d0413b88578-006618aff3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.776 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.776 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.776 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.776 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:19 GMT 2024-04-12T04:56:00.776 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.776 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:nKDihOxaoLKZF6Z+AacKBolUg3M= 2024-04-12T04:56:00.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nKDihOxaoLKZF6Z+AacKBolUg3M=', 'Content-Length': '0'} 2024-04-12T04:56:00.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095e02ae288c0b23f-006618aff3-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.778 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.778 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.778 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.778 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:19 GMT 2024-04-12T04:56:00.778 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.778 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:nKDihOxaoLKZF6Z+AacKBolUg3M= 2024-04-12T04:56:00.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nKDihOxaoLKZF6Z+AacKBolUg3M=', 'Content-Length': '0'} 2024-04-12T04:56:00.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000920ce556d11b03a-006618aff3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.778 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:56:00.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.779 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.779 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.779 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.779 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:19 GMT 2024-04-12T04:56:00.779 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.780 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Xkz/s9cbmNYHIDg02kUK1WFOG5I= 2024-04-12T04:56:00.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Xkz/s9cbmNYHIDg02kUK1WFOG5I=', 'Content-Length': '0'} 2024-04-12T04:56:00.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028821163447f5c76-006618aff3-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a?versionId=null 2024-04-12T04:56:00.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a?versionId=null 2024-04-12T04:56:00.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.781 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.781 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.781 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.781 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:19 GMT 2024-04-12T04:56:00.781 INFO:tasks.rgw_multisite_tests:/dkyges-57/a?versionId=null 2024-04-12T04:56:00.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.781 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Xkz/s9cbmNYHIDg02kUK1WFOG5I= 2024-04-12T04:56:00.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Xkz/s9cbmNYHIDg02kUK1WFOG5I=', 'Content-Length': '0'} 2024-04-12T04:56:00.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000babaa56afc9c5ea7-006618aff3-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:19 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=a 2024-04-12T04:56:00.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.782 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.782 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.782 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.782 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:19 GMT 2024-04-12T04:56:00.782 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.783 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:KOkL/T/PQrrhxIRPfS5JnavPqEc= 2024-04-12T04:56:00.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KOkL/T/PQrrhxIRPfS5JnavPqEc=', 'Content-Length': '0'} 2024-04-12T04:56:00.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1ff87e4966aef80-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=a 2024-04-12T04:56:00.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.785 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.785 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.785 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.785 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:19 GMT 2024-04-12T04:56:00.786 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.786 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:KOkL/T/PQrrhxIRPfS5JnavPqEc= 2024-04-12T04:56:00.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:19 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KOkL/T/PQrrhxIRPfS5JnavPqEc=', 'Content-Length': '0'} 2024-04-12T04:56:00.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc54c86dd3160ec2-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a 2024-04-12T04:56:00.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.787 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.787 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.787 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.787 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.787 INFO:tasks.rgw_multisite_tests:/dkyges-57/a 2024-04-12T04:56:00.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.787 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:KkpdsuQx4uMpmRRp7sMKiiYLzgg= 2024-04-12T04:56:00.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KkpdsuQx4uMpmRRp7sMKiiYLzgg=', 'Content-Length': '0'} 2024-04-12T04:56:00.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e493c780b1d2293d-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a 2024-04-12T04:56:00.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.788 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.788 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.788 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.789 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.789 INFO:tasks.rgw_multisite_tests:/dkyges-57/a 2024-04-12T04:56:00.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.789 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:KkpdsuQx4uMpmRRp7sMKiiYLzgg= 2024-04-12T04:56:00.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KkpdsuQx4uMpmRRp7sMKiiYLzgg=', 'Content-Length': '0'} 2024-04-12T04:56:00.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078c30aa1b6d3bcf1-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.789 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2024-04-12T04:56:00.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a 2024-04-12T04:56:00.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.790 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.790 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.790 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.790 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.790 INFO:tasks.rgw_multisite_tests:/dkyges-57/a 2024-04-12T04:56:00.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.790 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:1GCpXGiP0W7VTDTktHVbK93Nuyw= 2024-04-12T04:56:00.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1GCpXGiP0W7VTDTktHVbK93Nuyw=', 'Content-Length': '0'} 2024-04-12T04:56:00.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000268f0183531d23b6-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/a 2024-04-12T04:56:00.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/a 2024-04-12T04:56:00.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/a 2024-04-12T04:56:00.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.791 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.791 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.791 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.791 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.791 INFO:tasks.rgw_multisite_tests:/dkyges-57/a 2024-04-12T04:56:00.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.792 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:1GCpXGiP0W7VTDTktHVbK93Nuyw= 2024-04-12T04:56:00.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:1GCpXGiP0W7VTDTktHVbK93Nuyw=', 'Content-Length': '0'} 2024-04-12T04:56:00.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000972c19ba43ec4604-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=a 2024-04-12T04:56:00.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.793 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.793 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.793 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.793 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.793 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.793 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2adf8ef6cafec52-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=a 2024-04-12T04:56:00.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=a 2024-04-12T04:56:00.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.794 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.794 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.794 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.794 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.794 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.795 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbcf86377bae0060-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57a1000falsea2024-04-12T03:50:21.205Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:56:00.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.796 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.796 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.796 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.796 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.796 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.796 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dbnoRv6wiB0iGnTS2bdeY9W4QcU= 2024-04-12T04:56:00.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dbnoRv6wiB0iGnTS2bdeY9W4QcU=', 'Content-Length': '0'} 2024-04-12T04:56:00.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014a3213af12922df-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.797 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.797 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.798 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.798 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.798 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.798 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dbnoRv6wiB0iGnTS2bdeY9W4QcU= 2024-04-12T04:56:00.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dbnoRv6wiB0iGnTS2bdeY9W4QcU=', 'Content-Length': '0'} 2024-04-12T04:56:00.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a4815f27c12d2cc-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.799 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.799 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.799 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.799 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.799 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.800 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:5TZQNWKJuZQooCkBzhIJgJLZsnE= 2024-04-12T04:56:00.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5TZQNWKJuZQooCkBzhIJgJLZsnE=', 'Content-Length': '0'} 2024-04-12T04:56:00.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b60a3943dac86d-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.801 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.801 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.801 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.801 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.801 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.801 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:5TZQNWKJuZQooCkBzhIJgJLZsnE= 2024-04-12T04:56:00.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5TZQNWKJuZQooCkBzhIJgJLZsnE=', 'Content-Length': '0'} 2024-04-12T04:56:00.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ee86ee63ff0116c-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.801 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:56:00.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.802 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.802 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.802 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.802 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.802 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.803 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dbnoRv6wiB0iGnTS2bdeY9W4QcU= 2024-04-12T04:56:00.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dbnoRv6wiB0iGnTS2bdeY9W4QcU=', 'Content-Length': '0'} 2024-04-12T04:56:00.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbaf22a75f5e889f-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b?versionId=null 2024-04-12T04:56:00.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b?versionId=null 2024-04-12T04:56:00.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.804 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.804 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.804 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.804 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.804 INFO:tasks.rgw_multisite_tests:/dkyges-57/b?versionId=null 2024-04-12T04:56:00.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.804 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:dbnoRv6wiB0iGnTS2bdeY9W4QcU= 2024-04-12T04:56:00.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:dbnoRv6wiB0iGnTS2bdeY9W4QcU=', 'Content-Length': '0'} 2024-04-12T04:56:00.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ea5707ea463f335-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=b 2024-04-12T04:56:00.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.805 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.805 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.805 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.805 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.805 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.806 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d96a5e4bc7039578-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=b 2024-04-12T04:56:00.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.807 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.807 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.807 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.807 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.807 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.807 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa28ea791fd26db4-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b 2024-04-12T04:56:00.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.808 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.808 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.808 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.808 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.808 INFO:tasks.rgw_multisite_tests:/dkyges-57/b 2024-04-12T04:56:00.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.809 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LJxDmkET03W9TRCVEi2ze3LOgbk= 2024-04-12T04:56:00.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LJxDmkET03W9TRCVEi2ze3LOgbk=', 'Content-Length': '0'} 2024-04-12T04:56:00.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072db870c89922d2b-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b 2024-04-12T04:56:00.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.810 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.810 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.810 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.810 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.810 INFO:tasks.rgw_multisite_tests:/dkyges-57/b 2024-04-12T04:56:00.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.810 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LJxDmkET03W9TRCVEi2ze3LOgbk= 2024-04-12T04:56:00.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LJxDmkET03W9TRCVEi2ze3LOgbk=', 'Content-Length': '0'} 2024-04-12T04:56:00.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7e9e7039ad6ff78-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.810 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2024-04-12T04:56:00.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b 2024-04-12T04:56:00.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.811 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.811 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.811 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.811 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.811 INFO:tasks.rgw_multisite_tests:/dkyges-57/b 2024-04-12T04:56:00.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.812 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:SB4StwBBRjiM63/gZSQUxhhrZtA= 2024-04-12T04:56:00.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SB4StwBBRjiM63/gZSQUxhhrZtA=', 'Content-Length': '0'} 2024-04-12T04:56:00.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001122e461cef74fb-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/b 2024-04-12T04:56:00.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/b 2024-04-12T04:56:00.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/b 2024-04-12T04:56:00.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.813 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.813 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.813 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.813 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.813 INFO:tasks.rgw_multisite_tests:/dkyges-57/b 2024-04-12T04:56:00.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.813 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:SB4StwBBRjiM63/gZSQUxhhrZtA= 2024-04-12T04:56:00.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SB4StwBBRjiM63/gZSQUxhhrZtA=', 'Content-Length': '0'} 2024-04-12T04:56:00.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7a485b7c5ff9f57-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=b 2024-04-12T04:56:00.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.814 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.814 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.815 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.815 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.815 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.815 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ae0830d27f72e37-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=b 2024-04-12T04:56:00.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=b 2024-04-12T04:56:00.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.816 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.816 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.816 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.816 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.816 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.817 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af3e174d6de225d8-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57b1000falseb2024-04-12T03:50:21.217Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.817 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:56:00.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.818 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.818 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.818 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.818 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.818 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.819 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:9MwANBy93fyvFAelbvyRqYAf+0k= 2024-04-12T04:56:00.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9MwANBy93fyvFAelbvyRqYAf+0k=', 'Content-Length': '0'} 2024-04-12T04:56:00.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000964e4b58833276cc-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.820 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.820 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.820 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.820 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.820 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.820 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:9MwANBy93fyvFAelbvyRqYAf+0k= 2024-04-12T04:56:00.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9MwANBy93fyvFAelbvyRqYAf+0k=', 'Content-Length': '0'} 2024-04-12T04:56:00.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e014e1aef5893fec-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.821 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.822 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.822 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.822 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.822 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.822 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:+mz3jgglbOhRJ1ITvrxJz1kvv7Q= 2024-04-12T04:56:00.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+mz3jgglbOhRJ1ITvrxJz1kvv7Q=', 'Content-Length': '0'} 2024-04-12T04:56:00.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040a688aa6813c80c-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.823 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.823 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.823 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.823 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.823 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.824 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:+mz3jgglbOhRJ1ITvrxJz1kvv7Q= 2024-04-12T04:56:00.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+mz3jgglbOhRJ1ITvrxJz1kvv7Q=', 'Content-Length': '0'} 2024-04-12T04:56:00.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cec4d70d7ef576fb-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.824 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:56:00.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.825 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.825 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.825 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.825 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.825 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.825 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:9MwANBy93fyvFAelbvyRqYAf+0k= 2024-04-12T04:56:00.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9MwANBy93fyvFAelbvyRqYAf+0k=', 'Content-Length': '0'} 2024-04-12T04:56:00.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5e8f73f4bdfbea3-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c?versionId=null 2024-04-12T04:56:00.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c?versionId=null 2024-04-12T04:56:00.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.827 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.827 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.827 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.827 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.827 INFO:tasks.rgw_multisite_tests:/dkyges-57/c?versionId=null 2024-04-12T04:56:00.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.827 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:9MwANBy93fyvFAelbvyRqYAf+0k= 2024-04-12T04:56:00.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9MwANBy93fyvFAelbvyRqYAf+0k=', 'Content-Length': '0'} 2024-04-12T04:56:00.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000587cce71f1b96450-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=c 2024-04-12T04:56:00.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.828 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.828 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.828 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.828 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.828 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.828 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1ba4ae1b42dcd2c-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=c 2024-04-12T04:56:00.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.830 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.830 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.830 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.830 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.830 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.830 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094769600990c0178-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c 2024-04-12T04:56:00.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.831 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.831 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.831 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.831 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.831 INFO:tasks.rgw_multisite_tests:/dkyges-57/c 2024-04-12T04:56:00.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.832 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:uC/J7TmpxTM65GlEIGxoyQewVLU= 2024-04-12T04:56:00.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uC/J7TmpxTM65GlEIGxoyQewVLU=', 'Content-Length': '0'} 2024-04-12T04:56:00.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015857ebf0872b7ef-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c 2024-04-12T04:56:00.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.833 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.833 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.833 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.833 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.833 INFO:tasks.rgw_multisite_tests:/dkyges-57/c 2024-04-12T04:56:00.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.833 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:uC/J7TmpxTM65GlEIGxoyQewVLU= 2024-04-12T04:56:00.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uC/J7TmpxTM65GlEIGxoyQewVLU=', 'Content-Length': '0'} 2024-04-12T04:56:00.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008201f86a96b29f2e-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.833 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2024-04-12T04:56:00.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c 2024-04-12T04:56:00.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.834 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.834 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.834 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.834 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.834 INFO:tasks.rgw_multisite_tests:/dkyges-57/c 2024-04-12T04:56:00.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.834 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:54Dh3xCO0LjtH2VC8DZRgrFpows= 2024-04-12T04:56:00.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:54Dh3xCO0LjtH2VC8DZRgrFpows=', 'Content-Length': '0'} 2024-04-12T04:56:00.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000800dfb692bb3b63e-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/c 2024-04-12T04:56:00.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/c 2024-04-12T04:56:00.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/c 2024-04-12T04:56:00.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.835 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.835 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.836 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.836 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.836 INFO:tasks.rgw_multisite_tests:/dkyges-57/c 2024-04-12T04:56:00.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.836 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:54Dh3xCO0LjtH2VC8DZRgrFpows= 2024-04-12T04:56:00.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:54Dh3xCO0LjtH2VC8DZRgrFpows=', 'Content-Length': '0'} 2024-04-12T04:56:00.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000732e81fa17f63ea7-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=c 2024-04-12T04:56:00.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.837 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.837 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.837 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.837 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.837 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.837 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051718274a092ad4f-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=c 2024-04-12T04:56:00.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=c 2024-04-12T04:56:00.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.839 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.839 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.839 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.839 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.839 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.839 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e0d7bf5965724c6-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57c1000falsec2024-04-12T03:50:21.264Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.839 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:56:00.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.840 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.840 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.840 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.840 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.841 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.841 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:R9h5vNJKcDAP88Pus1arlv7Tx/g= 2024-04-12T04:56:00.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R9h5vNJKcDAP88Pus1arlv7Tx/g=', 'Content-Length': '0'} 2024-04-12T04:56:00.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a5fbaa29c2db8d2-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.842 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.842 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.842 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.842 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.842 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.842 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:R9h5vNJKcDAP88Pus1arlv7Tx/g= 2024-04-12T04:56:00.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R9h5vNJKcDAP88Pus1arlv7Tx/g=', 'Content-Length': '0'} 2024-04-12T04:56:00.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002202747e1652cc9a-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.843 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.843 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.843 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.843 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.844 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.844 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:p79gbhR3DNZxkdlWLzZCikgqqAU= 2024-04-12T04:56:00.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p79gbhR3DNZxkdlWLzZCikgqqAU=', 'Content-Length': '0'} 2024-04-12T04:56:00.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047d01d24c1136a4e-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.845 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.845 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.845 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.845 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.845 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.845 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:p79gbhR3DNZxkdlWLzZCikgqqAU= 2024-04-12T04:56:00.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p79gbhR3DNZxkdlWLzZCikgqqAU=', 'Content-Length': '0'} 2024-04-12T04:56:00.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d73b747923cd44e2-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.845 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:56:00.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.846 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.846 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.847 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.847 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.847 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.847 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:R9h5vNJKcDAP88Pus1arlv7Tx/g= 2024-04-12T04:56:00.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R9h5vNJKcDAP88Pus1arlv7Tx/g=', 'Content-Length': '0'} 2024-04-12T04:56:00.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d82a83223d67fdb1-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d?versionId=null 2024-04-12T04:56:00.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d?versionId=null 2024-04-12T04:56:00.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.848 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.848 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.848 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.848 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.848 INFO:tasks.rgw_multisite_tests:/dkyges-57/d?versionId=null 2024-04-12T04:56:00.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.848 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:R9h5vNJKcDAP88Pus1arlv7Tx/g= 2024-04-12T04:56:00.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R9h5vNJKcDAP88Pus1arlv7Tx/g=', 'Content-Length': '0'} 2024-04-12T04:56:00.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083c3e8a5d6b04309-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=d 2024-04-12T04:56:00.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.849 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.850 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.850 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.850 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.850 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.850 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004585ecff20fe4c9e-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=d 2024-04-12T04:56:00.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.851 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.851 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.851 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.852 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.852 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.852 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f2949e7e2c1d925-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d 2024-04-12T04:56:00.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.853 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.853 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.853 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.853 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.853 INFO:tasks.rgw_multisite_tests:/dkyges-57/d 2024-04-12T04:56:00.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.853 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:R3/mK+ze1RRTpuWqKD6GTCHw4fs= 2024-04-12T04:56:00.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R3/mK+ze1RRTpuWqKD6GTCHw4fs=', 'Content-Length': '0'} 2024-04-12T04:56:00.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e07a33f2ebebd3fc-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d 2024-04-12T04:56:00.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.854 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.855 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.855 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.855 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.855 INFO:tasks.rgw_multisite_tests:/dkyges-57/d 2024-04-12T04:56:00.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.855 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:R3/mK+ze1RRTpuWqKD6GTCHw4fs= 2024-04-12T04:56:00.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:R3/mK+ze1RRTpuWqKD6GTCHw4fs=', 'Content-Length': '0'} 2024-04-12T04:56:00.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e359ef97bd9d83a1-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.855 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2024-04-12T04:56:00.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d 2024-04-12T04:56:00.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.856 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.856 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.856 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.856 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.856 INFO:tasks.rgw_multisite_tests:/dkyges-57/d 2024-04-12T04:56:00.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.856 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:5wFkjAGKT6oKfyZ7TO1AHMuO8oY= 2024-04-12T04:56:00.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5wFkjAGKT6oKfyZ7TO1AHMuO8oY=', 'Content-Length': '0'} 2024-04-12T04:56:00.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cfb426af39f59e6-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/d 2024-04-12T04:56:00.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/d 2024-04-12T04:56:00.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/d 2024-04-12T04:56:00.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.858 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.858 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.858 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.858 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.858 INFO:tasks.rgw_multisite_tests:/dkyges-57/d 2024-04-12T04:56:00.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.858 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:5wFkjAGKT6oKfyZ7TO1AHMuO8oY= 2024-04-12T04:56:00.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5wFkjAGKT6oKfyZ7TO1AHMuO8oY=', 'Content-Length': '0'} 2024-04-12T04:56:00.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:21 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afb951a8205d53ff-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=d 2024-04-12T04:56:00.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.859 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.859 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.859 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.859 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.860 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.860 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002d123bdd5be91d0-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=d 2024-04-12T04:56:00.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=d 2024-04-12T04:56:00.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.861 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.861 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.861 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.861 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.861 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.861 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008414cce052dc2bc5-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57d1000falsed2024-04-12T03:50:21.275Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.862 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-12T04:56:00.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e?versionId=null 2024-04-12T04:56:00.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.863 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.863 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.863 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.863 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.863 INFO:tasks.rgw_multisite_tests:/dkyges-57/e?versionId=null 2024-04-12T04:56:00.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.863 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:P0HM4KjwepbpwfhFXwKXaBt7csE= 2024-04-12T04:56:00.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P0HM4KjwepbpwfhFXwKXaBt7csE=', 'Content-Length': '0'} 2024-04-12T04:56:00.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099e7c98e1926981a-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e?versionId=null 2024-04-12T04:56:00.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.865 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.865 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.865 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.865 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.865 INFO:tasks.rgw_multisite_tests:/dkyges-57/e?versionId=null 2024-04-12T04:56:00.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.865 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:P0HM4KjwepbpwfhFXwKXaBt7csE= 2024-04-12T04:56:00.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P0HM4KjwepbpwfhFXwKXaBt7csE=', 'Content-Length': '0'} 2024-04-12T04:56:00.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f77123766bdfc0e-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e?versionId=null 2024-04-12T04:56:00.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.866 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.866 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.867 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.867 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.867 INFO:tasks.rgw_multisite_tests:/dkyges-57/e?versionId=null 2024-04-12T04:56:00.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.867 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:0SPdO0BdTA2jYDRRj0VJntevMcM= 2024-04-12T04:56:00.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0SPdO0BdTA2jYDRRj0VJntevMcM=', 'Content-Length': '0'} 2024-04-12T04:56:00.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9621a9ba9fd04f9-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e?versionId=null 2024-04-12T04:56:00.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.868 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.868 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.868 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.868 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.868 INFO:tasks.rgw_multisite_tests:/dkyges-57/e?versionId=null 2024-04-12T04:56:00.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.868 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:0SPdO0BdTA2jYDRRj0VJntevMcM= 2024-04-12T04:56:00.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0SPdO0BdTA2jYDRRj0VJntevMcM=', 'Content-Length': '0'} 2024-04-12T04:56:00.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f9a79b9aef3cd05-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.869 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-12T04:56:00.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e?versionId=null 2024-04-12T04:56:00.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.870 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.870 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.870 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.870 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.870 INFO:tasks.rgw_multisite_tests:/dkyges-57/e?versionId=null 2024-04-12T04:56:00.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.870 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:P0HM4KjwepbpwfhFXwKXaBt7csE= 2024-04-12T04:56:00.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P0HM4KjwepbpwfhFXwKXaBt7csE=', 'Content-Length': '0'} 2024-04-12T04:56:00.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab6c08cbd5bdabfe-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e?versionId=null 2024-04-12T04:56:00.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e?versionId=null 2024-04-12T04:56:00.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.872 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.872 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.872 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.872 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.872 INFO:tasks.rgw_multisite_tests:/dkyges-57/e?versionId=null 2024-04-12T04:56:00.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.872 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:P0HM4KjwepbpwfhFXwKXaBt7csE= 2024-04-12T04:56:00.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:P0HM4KjwepbpwfhFXwKXaBt7csE=', 'Content-Length': '0'} 2024-04-12T04:56:00.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb89cc569969183d-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=e 2024-04-12T04:56:00.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=e 2024-04-12T04:56:00.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=e 2024-04-12T04:56:00.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.873 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.873 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.873 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.873 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.873 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.873 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c897638073b0e2e3-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57e1000falsee2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=e 2024-04-12T04:56:00.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=e 2024-04-12T04:56:00.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=e 2024-04-12T04:56:00.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.875 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.875 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.875 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.875 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.875 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.875 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d500c466f6e0cc8-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57e1000falsee2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e 2024-04-12T04:56:00.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.876 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.876 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.876 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.876 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.876 INFO:tasks.rgw_multisite_tests:/dkyges-57/e 2024-04-12T04:56:00.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.877 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:XUEEoyfysn9nVc7O0aw2RV6yeNk= 2024-04-12T04:56:00.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XUEEoyfysn9nVc7O0aw2RV6yeNk=', 'Content-Length': '0'} 2024-04-12T04:56:00.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049db07666e1ae97b-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e 2024-04-12T04:56:00.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.878 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.878 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.878 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.878 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.878 INFO:tasks.rgw_multisite_tests:/dkyges-57/e 2024-04-12T04:56:00.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.878 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:XUEEoyfysn9nVc7O0aw2RV6yeNk= 2024-04-12T04:56:00.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XUEEoyfysn9nVc7O0aw2RV6yeNk=', 'Content-Length': '0'} 2024-04-12T04:56:00.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff6ee3db14ab0818-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.879 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2024-04-12T04:56:00.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e 2024-04-12T04:56:00.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.879 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.880 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.880 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.880 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.880 INFO:tasks.rgw_multisite_tests:/dkyges-57/e 2024-04-12T04:56:00.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.880 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Tj6X6BmSSLuiutQkvc73/5RjDus= 2024-04-12T04:56:00.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Tj6X6BmSSLuiutQkvc73/5RjDus=', 'Content-Length': '0'} 2024-04-12T04:56:00.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000164d31d7f2d621b-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/e 2024-04-12T04:56:00.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/e 2024-04-12T04:56:00.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/e 2024-04-12T04:56:00.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.881 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.881 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.881 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.881 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.881 INFO:tasks.rgw_multisite_tests:/dkyges-57/e 2024-04-12T04:56:00.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.881 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Tj6X6BmSSLuiutQkvc73/5RjDus= 2024-04-12T04:56:00.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Tj6X6BmSSLuiutQkvc73/5RjDus=', 'Content-Length': '0'} 2024-04-12T04:56:00.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031215b224547b52a-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=e 2024-04-12T04:56:00.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=e 2024-04-12T04:56:00.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=e 2024-04-12T04:56:00.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.882 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.883 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.883 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.883 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.883 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.883 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3d23c6bcfb424fb-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57e1000falsee2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=e 2024-04-12T04:56:00.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=e 2024-04-12T04:56:00.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=e 2024-04-12T04:56:00.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.884 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.884 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.884 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.884 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.884 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.884 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca1e0704ca48b49b-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57e1000falsee2024-04-12T03:50:43.025Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.885 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-12T04:56:00.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f?versionId=null 2024-04-12T04:56:00.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.886 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.886 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.886 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.886 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.886 INFO:tasks.rgw_multisite_tests:/dkyges-57/f?versionId=null 2024-04-12T04:56:00.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.886 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:rj0UNc3Bo+rAI7dQnnpoDLAS/gU= 2024-04-12T04:56:00.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rj0UNc3Bo+rAI7dQnnpoDLAS/gU=', 'Content-Length': '0'} 2024-04-12T04:56:00.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b42d664d507c00e8-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f?versionId=null 2024-04-12T04:56:00.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.887 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.887 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.887 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.887 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.887 INFO:tasks.rgw_multisite_tests:/dkyges-57/f?versionId=null 2024-04-12T04:56:00.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.888 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:rj0UNc3Bo+rAI7dQnnpoDLAS/gU= 2024-04-12T04:56:00.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rj0UNc3Bo+rAI7dQnnpoDLAS/gU=', 'Content-Length': '0'} 2024-04-12T04:56:00.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047555ea3ce1c5586-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f?versionId=null 2024-04-12T04:56:00.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.889 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.889 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.889 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.889 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.889 INFO:tasks.rgw_multisite_tests:/dkyges-57/f?versionId=null 2024-04-12T04:56:00.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.889 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:qrG2eOKeUozlt4BmAbAJXlS4Gw0= 2024-04-12T04:56:00.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qrG2eOKeUozlt4BmAbAJXlS4Gw0=', 'Content-Length': '0'} 2024-04-12T04:56:00.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f08c9eb47de0068d-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f?versionId=null 2024-04-12T04:56:00.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.890 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.891 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.891 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.891 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.891 INFO:tasks.rgw_multisite_tests:/dkyges-57/f?versionId=null 2024-04-12T04:56:00.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.891 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:qrG2eOKeUozlt4BmAbAJXlS4Gw0= 2024-04-12T04:56:00.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qrG2eOKeUozlt4BmAbAJXlS4Gw0=', 'Content-Length': '0'} 2024-04-12T04:56:00.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb6520690ad47a78-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.891 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-12T04:56:00.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.891 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f?versionId=null 2024-04-12T04:56:00.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.892 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.892 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.892 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.892 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.892 INFO:tasks.rgw_multisite_tests:/dkyges-57/f?versionId=null 2024-04-12T04:56:00.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.892 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:rj0UNc3Bo+rAI7dQnnpoDLAS/gU= 2024-04-12T04:56:00.892 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rj0UNc3Bo+rAI7dQnnpoDLAS/gU=', 'Content-Length': '0'} 2024-04-12T04:56:00.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010f4eb00bb0d82af-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f?versionId=null 2024-04-12T04:56:00.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f?versionId=null 2024-04-12T04:56:00.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.893 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.893 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.894 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.894 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.894 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.894 INFO:tasks.rgw_multisite_tests:/dkyges-57/f?versionId=null 2024-04-12T04:56:00.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.894 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:rj0UNc3Bo+rAI7dQnnpoDLAS/gU= 2024-04-12T04:56:00.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:rj0UNc3Bo+rAI7dQnnpoDLAS/gU=', 'Content-Length': '0'} 2024-04-12T04:56:00.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9550af25732b103-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=f 2024-04-12T04:56:00.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=f 2024-04-12T04:56:00.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=f 2024-04-12T04:56:00.894 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.895 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.895 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.895 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.895 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.895 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.895 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.895 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c4f3b64da4c8af1-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57f1000falsef2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=f 2024-04-12T04:56:00.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=f 2024-04-12T04:56:00.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=f 2024-04-12T04:56:00.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.896 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.896 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.897 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.897 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.897 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.897 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.897 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b714183cb439019d-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57f1000falsef2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f 2024-04-12T04:56:00.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.897 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.898 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.898 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.898 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.898 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.898 INFO:tasks.rgw_multisite_tests:/dkyges-57/f 2024-04-12T04:56:00.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.898 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:qVjB51adJCZd5r4CjJh3EHhrKFA= 2024-04-12T04:56:00.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qVjB51adJCZd5r4CjJh3EHhrKFA=', 'Content-Length': '0'} 2024-04-12T04:56:00.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cb0e49fb0200345-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.898 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f 2024-04-12T04:56:00.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.899 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.899 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.899 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.899 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.899 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.900 INFO:tasks.rgw_multisite_tests:/dkyges-57/f 2024-04-12T04:56:00.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.900 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:qVjB51adJCZd5r4CjJh3EHhrKFA= 2024-04-12T04:56:00.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:qVjB51adJCZd5r4CjJh3EHhrKFA=', 'Content-Length': '0'} 2024-04-12T04:56:00.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7da9933819973c1-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.900 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2024-04-12T04:56:00.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f 2024-04-12T04:56:00.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.900 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.901 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.901 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.901 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.901 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.901 INFO:tasks.rgw_multisite_tests:/dkyges-57/f 2024-04-12T04:56:00.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.901 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:OD5noZlc58kXco8sAsy75Alyosk= 2024-04-12T04:56:00.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OD5noZlc58kXco8sAsy75Alyosk=', 'Content-Length': '0'} 2024-04-12T04:56:00.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000787ebcd22f441030-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/f 2024-04-12T04:56:00.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/f 2024-04-12T04:56:00.901 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/f 2024-04-12T04:56:00.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.902 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.902 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.902 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.902 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.902 INFO:tasks.rgw_multisite_tests:/dkyges-57/f 2024-04-12T04:56:00.902 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.902 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:OD5noZlc58kXco8sAsy75Alyosk= 2024-04-12T04:56:00.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OD5noZlc58kXco8sAsy75Alyosk=', 'Content-Length': '0'} 2024-04-12T04:56:00.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000823788938ef3463f-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=f 2024-04-12T04:56:00.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=f 2024-04-12T04:56:00.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=f 2024-04-12T04:56:00.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.903 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.904 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.904 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.904 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.904 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.904 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.904 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c82de1d157f670d-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.904 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57f1000falsef2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=f 2024-04-12T04:56:00.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=f 2024-04-12T04:56:00.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=f 2024-04-12T04:56:00.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.905 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.906 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.906 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.906 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.906 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.906 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.906 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d1e1f94b4df741f-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57f1000falsef2024-04-12T03:50:43.073Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.906 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-12T04:56:00.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.906 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g?versionId=null 2024-04-12T04:56:00.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.907 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.907 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.907 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.908 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.908 INFO:tasks.rgw_multisite_tests:/dkyges-57/g?versionId=null 2024-04-12T04:56:00.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.908 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:A8NTHgT3wEjIMhpj6w7Wic4AuiQ= 2024-04-12T04:56:00.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A8NTHgT3wEjIMhpj6w7Wic4AuiQ=', 'Content-Length': '0'} 2024-04-12T04:56:00.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f9c0b44fce0f99c-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g?versionId=null 2024-04-12T04:56:00.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.909 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.909 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.909 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.909 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.909 INFO:tasks.rgw_multisite_tests:/dkyges-57/g?versionId=null 2024-04-12T04:56:00.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.909 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:A8NTHgT3wEjIMhpj6w7Wic4AuiQ= 2024-04-12T04:56:00.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A8NTHgT3wEjIMhpj6w7Wic4AuiQ=', 'Content-Length': '0'} 2024-04-12T04:56:00.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a2fd0ac299c0ae1-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g?versionId=null 2024-04-12T04:56:00.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.911 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.911 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.911 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.911 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.911 INFO:tasks.rgw_multisite_tests:/dkyges-57/g?versionId=null 2024-04-12T04:56:00.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.911 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:8R3JCDN1Cvg2U7toLTSJYVQBVoY= 2024-04-12T04:56:00.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8R3JCDN1Cvg2U7toLTSJYVQBVoY=', 'Content-Length': '0'} 2024-04-12T04:56:00.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f422c3e5dc4f1f0a-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g?versionId=null 2024-04-12T04:56:00.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.912 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.912 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.912 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.912 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.912 INFO:tasks.rgw_multisite_tests:/dkyges-57/g?versionId=null 2024-04-12T04:56:00.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.913 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:8R3JCDN1Cvg2U7toLTSJYVQBVoY= 2024-04-12T04:56:00.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8R3JCDN1Cvg2U7toLTSJYVQBVoY=', 'Content-Length': '0'} 2024-04-12T04:56:00.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000751fca791fd0813-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.913 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-12T04:56:00.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g?versionId=null 2024-04-12T04:56:00.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.914 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.914 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.914 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.914 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.914 INFO:tasks.rgw_multisite_tests:/dkyges-57/g?versionId=null 2024-04-12T04:56:00.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.914 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:A8NTHgT3wEjIMhpj6w7Wic4AuiQ= 2024-04-12T04:56:00.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A8NTHgT3wEjIMhpj6w7Wic4AuiQ=', 'Content-Length': '0'} 2024-04-12T04:56:00.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000035f341e3130afc7-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g?versionId=null 2024-04-12T04:56:00.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g?versionId=null 2024-04-12T04:56:00.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.916 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.916 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.916 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.916 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.916 INFO:tasks.rgw_multisite_tests:/dkyges-57/g?versionId=null 2024-04-12T04:56:00.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.916 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:A8NTHgT3wEjIMhpj6w7Wic4AuiQ= 2024-04-12T04:56:00.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:A8NTHgT3wEjIMhpj6w7Wic4AuiQ=', 'Content-Length': '0'} 2024-04-12T04:56:00.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b4967c153515641-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=g 2024-04-12T04:56:00.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=g 2024-04-12T04:56:00.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=g 2024-04-12T04:56:00.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.917 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.917 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.917 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.917 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.917 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.918 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3d297905e71d8f9-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57g1000falseg2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=g 2024-04-12T04:56:00.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=g 2024-04-12T04:56:00.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=g 2024-04-12T04:56:00.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.919 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.919 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.919 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.919 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.919 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.919 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb71dc1c928f96e8-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57g1000falseg2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g 2024-04-12T04:56:00.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.920 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.921 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.921 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.921 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.921 INFO:tasks.rgw_multisite_tests:/dkyges-57/g 2024-04-12T04:56:00.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.921 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ruwT++u4nyrdQZmPxKSZqgx7Wzw= 2024-04-12T04:56:00.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ruwT++u4nyrdQZmPxKSZqgx7Wzw=', 'Content-Length': '0'} 2024-04-12T04:56:00.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044eea00608dea109-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g 2024-04-12T04:56:00.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.922 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.922 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.922 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.922 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.922 INFO:tasks.rgw_multisite_tests:/dkyges-57/g 2024-04-12T04:56:00.922 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.922 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ruwT++u4nyrdQZmPxKSZqgx7Wzw= 2024-04-12T04:56:00.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ruwT++u4nyrdQZmPxKSZqgx7Wzw=', 'Content-Length': '0'} 2024-04-12T04:56:00.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcb82dcb0caf9091-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.923 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2024-04-12T04:56:00.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g 2024-04-12T04:56:00.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.923 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.923 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.923 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.924 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.924 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.924 INFO:tasks.rgw_multisite_tests:/dkyges-57/g 2024-04-12T04:56:00.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.924 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:KNOSbjHn2f+DAQOKw1ABhGp3hdU= 2024-04-12T04:56:00.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KNOSbjHn2f+DAQOKw1ABhGp3hdU=', 'Content-Length': '0'} 2024-04-12T04:56:00.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d92d4e66763a8c62-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/g 2024-04-12T04:56:00.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/g 2024-04-12T04:56:00.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/g 2024-04-12T04:56:00.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.924 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.925 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.925 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.925 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.925 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.925 INFO:tasks.rgw_multisite_tests:/dkyges-57/g 2024-04-12T04:56:00.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.925 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:KNOSbjHn2f+DAQOKw1ABhGp3hdU= 2024-04-12T04:56:00.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KNOSbjHn2f+DAQOKw1ABhGp3hdU=', 'Content-Length': '0'} 2024-04-12T04:56:00.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078a81ffce8f97a23-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.925 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=g 2024-04-12T04:56:00.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=g 2024-04-12T04:56:00.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=g 2024-04-12T04:56:00.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.926 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.926 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.926 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.926 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.926 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.927 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038eeb3ff9b1c840c-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57g1000falseg2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=g 2024-04-12T04:56:00.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=g 2024-04-12T04:56:00.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=g 2024-04-12T04:56:00.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.928 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.928 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.928 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.928 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.928 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.928 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007dfaee78529261cc-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57g1000falseg2024-04-12T03:50:43.085Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.929 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-12T04:56:00.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h?versionId=null 2024-04-12T04:56:00.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.929 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.930 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.930 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.930 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.930 INFO:tasks.rgw_multisite_tests:/dkyges-57/h?versionId=null 2024-04-12T04:56:00.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.930 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:YxpXfG7sECYtQ4gY/nju4HSR+rM= 2024-04-12T04:56:00.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YxpXfG7sECYtQ4gY/nju4HSR+rM=', 'Content-Length': '0'} 2024-04-12T04:56:00.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049196797a15bc9de-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h?versionId=null 2024-04-12T04:56:00.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.931 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.931 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.931 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.931 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.931 INFO:tasks.rgw_multisite_tests:/dkyges-57/h?versionId=null 2024-04-12T04:56:00.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.931 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:YxpXfG7sECYtQ4gY/nju4HSR+rM= 2024-04-12T04:56:00.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YxpXfG7sECYtQ4gY/nju4HSR+rM=', 'Content-Length': '0'} 2024-04-12T04:56:00.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000589faee354a9a20-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h?versionId=null 2024-04-12T04:56:00.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.933 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.933 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.933 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.933 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.933 INFO:tasks.rgw_multisite_tests:/dkyges-57/h?versionId=null 2024-04-12T04:56:00.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.933 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:36/V0M7cNOGzenjsTkBdPPUTCBY= 2024-04-12T04:56:00.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:36/V0M7cNOGzenjsTkBdPPUTCBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006695d56a5386a01b-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h?versionId=null 2024-04-12T04:56:00.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.934 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.934 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.934 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.934 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.934 INFO:tasks.rgw_multisite_tests:/dkyges-57/h?versionId=null 2024-04-12T04:56:00.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.935 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:36/V0M7cNOGzenjsTkBdPPUTCBY= 2024-04-12T04:56:00.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:36/V0M7cNOGzenjsTkBdPPUTCBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000818e3db3bcb300d1-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.935 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-12T04:56:00.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h?versionId=null 2024-04-12T04:56:00.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.936 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.936 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.936 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.936 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.936 INFO:tasks.rgw_multisite_tests:/dkyges-57/h?versionId=null 2024-04-12T04:56:00.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.936 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:YxpXfG7sECYtQ4gY/nju4HSR+rM= 2024-04-12T04:56:00.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YxpXfG7sECYtQ4gY/nju4HSR+rM=', 'Content-Length': '0'} 2024-04-12T04:56:00.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008530c12f72f57a64-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h?versionId=null 2024-04-12T04:56:00.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h?versionId=null 2024-04-12T04:56:00.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.937 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.937 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.937 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.938 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.938 INFO:tasks.rgw_multisite_tests:/dkyges-57/h?versionId=null 2024-04-12T04:56:00.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.938 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:YxpXfG7sECYtQ4gY/nju4HSR+rM= 2024-04-12T04:56:00.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:YxpXfG7sECYtQ4gY/nju4HSR+rM=', 'Content-Length': '0'} 2024-04-12T04:56:00.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e45b08a697c16fff-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=h 2024-04-12T04:56:00.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=h 2024-04-12T04:56:00.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=h 2024-04-12T04:56:00.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.939 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.939 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.939 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.939 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.939 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.939 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000641b05bb43c007fa-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57h1000falseh2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=h 2024-04-12T04:56:00.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=h 2024-04-12T04:56:00.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=h 2024-04-12T04:56:00.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.941 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.941 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.941 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.941 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.941 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.941 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caf0374995bdf865-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57h1000falseh2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h 2024-04-12T04:56:00.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.942 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.942 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.942 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.942 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.942 INFO:tasks.rgw_multisite_tests:/dkyges-57/h 2024-04-12T04:56:00.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.942 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:2WUhWd1Og+bZSh2IYdGPBM0qAN4= 2024-04-12T04:56:00.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2WUhWd1Og+bZSh2IYdGPBM0qAN4=', 'Content-Length': '0'} 2024-04-12T04:56:00.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff3150f800b42795-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h 2024-04-12T04:56:00.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.943 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.943 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.943 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.944 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.944 INFO:tasks.rgw_multisite_tests:/dkyges-57/h 2024-04-12T04:56:00.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.944 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:2WUhWd1Og+bZSh2IYdGPBM0qAN4= 2024-04-12T04:56:00.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2WUhWd1Og+bZSh2IYdGPBM0qAN4=', 'Content-Length': '0'} 2024-04-12T04:56:00.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000032be85ce298c523-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.944 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2024-04-12T04:56:00.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h 2024-04-12T04:56:00.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.945 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.946 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.946 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.946 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.946 INFO:tasks.rgw_multisite_tests:/dkyges-57/h 2024-04-12T04:56:00.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.946 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:HEzcfchZ59vlqVhpnHzVcY2+Avc= 2024-04-12T04:56:00.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HEzcfchZ59vlqVhpnHzVcY2+Avc=', 'Content-Length': '0'} 2024-04-12T04:56:00.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-57:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaff7f812c1da178-006618aff4-16de-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/h 2024-04-12T04:56:00.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/h 2024-04-12T04:56:00.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/h 2024-04-12T04:56:00.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.947 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.947 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.948 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.948 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.948 INFO:tasks.rgw_multisite_tests:/dkyges-57/h 2024-04-12T04:56:00.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.948 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:HEzcfchZ59vlqVhpnHzVcY2+Avc= 2024-04-12T04:56:00.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HEzcfchZ59vlqVhpnHzVcY2+Avc=', 'Content-Length': '0'} 2024-04-12T04:56:00.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 03:50:43 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af59f1f6b6d3d5ce-006618aff4-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=h 2024-04-12T04:56:00.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=h 2024-04-12T04:56:00.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=h 2024-04-12T04:56:00.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:00.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:00.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.949 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.949 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.949 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.949 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.949 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.949 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b6f8c8a632bda1a-006618aff4-16de-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57h1000falseh2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/ 2024-04-12T04:56:00.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/ 2024-04-12T04:56:00.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-57/?prefix=h 2024-04-12T04:56:00.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-57/?prefix=h 2024-04-12T04:56:00.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:00.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-57/?prefix=h 2024-04-12T04:56:00.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.951 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:00.951 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.951 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.951 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 03:52:20 GMT 2024-04-12T04:56:00.951 INFO:tasks.rgw_multisite_tests:/dkyges-57/ 2024-04-12T04:56:00.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.951 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY= 2024-04-12T04:56:00.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 03:52:20 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:LslcaoKmFGNCot3sZIbGLSQILBY=', 'Content-Length': '0'} 2024-04-12T04:56:00.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8d3616b77f6e5ef-006618aff4-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 03:52:20 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-57h1000falseh2024-04-12T03:50:43.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:00.951 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=dkyges-57 zones={a2, a1} 2024-04-12T04:56:00.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.951 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:00.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.952 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T03:52:20.932+0000 7f3330984700 -1 req 1858292394514936859 0.000999987s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-32:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.32,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-32:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.32,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T04:56:00.952 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T03:52:20.933+0000 7f3391245700 -1 req 12650732993877336180 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-24:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.24,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-24:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.24,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T04:56:00.952 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T03:52:20.984+0000 7f337520d700 -1 req 789324565558273563 0.000999987s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-25:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.25,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-25:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.25,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T04:56:00.952 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.952 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dkyges-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:00.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "resharding": "None", 2024-04-12T04:56:00.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_index": { 2024-04-12T04:56:00.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 2, 2024-04-12T04:56:00.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "type": "Normal", 2024-04-12T04:56:00.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "normal": { 2024-04-12T04:56:00.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 3, 2024-04-12T04:56:00.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:56:00.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "logs": [ 2024-04-12T04:56:00.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 1, 2024-04-12T04:56:00.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "type": "InIndex", 2024-04-12T04:56:00.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "in_index": { 2024-04-12T04:56:00.953 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 1, 2024-04-12T04:56:00.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 3, 2024-04-12T04:56:00.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:56:00.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 2, 2024-04-12T04:56:00.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "type": "InIndex", 2024-04-12T04:56:00.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "in_index": { 2024-04-12T04:56:00.954 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 2, 2024-04-12T04:56:00.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 3, 2024-04-12T04:56:00.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:56:00.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.955 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.955 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:00.955 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.956 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.956 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T03:52:21.306+0000 7f3337992700 -1 req 8728977924978192002 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-2:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.2,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-2:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.2,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T04:56:00.956 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T03:52:21.306+0000 7f33bb299700 -1 req 637262317642509595 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-21:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.21,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-21:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.21,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T04:56:00.956 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T03:52:21.307+0000 7f338ea40700 -1 req 17551905316112605668 0.000999987s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-30:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.30,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-30:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.30,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T04:56:00.956 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T03:52:21.310+0000 7f32fd11d700 -1 req 12505786100015442675 0.000999987s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-27:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.27,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-27:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.27,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T04:56:00.956 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T03:52:21.310+0000 7f3330183700 -1 req 15303215955899797734 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-1:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.1,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-1:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.1,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T04:56:00.956 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T03:52:21.313+0000 7f33b3289700 -1 req 186565308761057666 0.000999987s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-43:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.6,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-43:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.6,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T04:56:00.956 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T03:52:21.354+0000 7f33449ac700 -1 req 15481524183401078451 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-4:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.4,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-4:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.4,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T04:56:00.956 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T03:52:21.355+0000 7f3312948700 -1 req 4994374400251301308 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-38:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.1,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-38:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.1,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T04:56:00.956 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T03:52:21.396+0000 7f337ba1a700 -1 req 13232022127411770878 0.000999987s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-5:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.5,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-5:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.5,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T04:56:00.956 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T03:52:21.396+0000 7f333418b700 -1 req 16761627938963292835 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-45:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.8,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-45:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.8,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T04:56:00.956 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T03:52:21.402+0000 7f330d93e700 -1 req 3233970052148653791 0.000999987s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=dkyges-28:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.28,z=3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61,az=0},d={b=dkyges-28:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.4369.28,z=1ba046e0-d0ae-437d-b741-28b4299423c2,az=0}} returned ret=-2 2024-04-12T04:56:00.956 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.957 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'dkyges-57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:00.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket dkyges-57 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:00.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "resharding": "None", 2024-04-12T04:56:00.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_index": { 2024-04-12T04:56:00.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 2, 2024-04-12T04:56:00.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "type": "Normal", 2024-04-12T04:56:00.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "normal": { 2024-04-12T04:56:00.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 3, 2024-04-12T04:56:00.957 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:56:00.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "logs": [ 2024-04-12T04:56:00.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 1, 2024-04-12T04:56:00.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "type": "InIndex", 2024-04-12T04:56:00.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "in_index": { 2024-04-12T04:56:00.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 1, 2024-04-12T04:56:00.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 3, 2024-04-12T04:56:00.958 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:56:00.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 2, 2024-04-12T04:56:00.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "type": "InIndex", 2024-04-12T04:56:00.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "in_index": { 2024-04-12T04:56:00.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "gen": 2, 2024-04-12T04:56:00.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "layout": { 2024-04-12T04:56:00.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 3, 2024-04-12T04:56:00.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hash_type": "Mod" 2024-04-12T04:56:00.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.960 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-12T04:56:00.960 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.961 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-12T04:56:00.961 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-04-12T04:56:00.961 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-12T04:56:00.961 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-12T04:56:00.961 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-12T04:56:00.961 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-12T04:56:00.961 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1875, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2024-04-12T04:56:00.961 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-12T04:56:00.962 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-12T04:56:00.962 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-12T04:56:00.962 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-12T04:56:00.962 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-12T04:56:00.962 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-12T04:56:00.962 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-12T04:56:00.962 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-12T04:56:00.962 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-12T04:56:00.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:00.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:00.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:00.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.963 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.963 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.963 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.963 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:22:56 GMT 2024-04-12T04:56:00.963 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:00.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.963 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Px26zZ2Luw6BoCB4irdzVIYW+Us= 2024-04-12T04:56:00.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:22:56 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Px26zZ2Luw6BoCB4irdzVIYW+Us=', 'Content-Length': '0'} 2024-04-12T04:56:00.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000673d814c50f2f61b-006618b720-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:22:56 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.964 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=dkyges-65 2024-04-12T04:56:00.964 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.964 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:00.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:56:00.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: [ 2024-04-12T04:56:00.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:00.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:56:00.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:00.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:56:00.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:00.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:56:00.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895776.234016_710.1", 2024-04-12T04:56:00.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T04:22:56.234016Z" 2024-04-12T04:56:00.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895776.234016_710.1'} 2024-04-12T04:56:00.966 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-12T04:56:00.966 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.966 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:00.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:22:56.434+0000 7fe7a5df1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:00.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:22:56.435+0000 7fe7a5df1780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:00.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:00.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:00.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:00.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:00.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:00.966 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:00.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:00.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:00.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:00.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:00.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.967 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:00.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:00.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:00.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:00.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:00.968 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:00.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:00.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:00.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:00.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.969 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895776.227017_707.1", 2024-04-12T04:56:00.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:00.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:22:56.227017Z", 2024-04-12T04:56:00.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:00.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:00.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:00.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.970 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:22:56Z" 2024-04-12T04:56:00.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:00.971 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:00.971 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:00.971 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:00.971 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:00.971 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:00.971 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.971 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.971 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:00.971 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.971 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:00.971 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.972 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.972 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:00.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.972 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:00.972 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.972 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:00.972 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.972 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.972 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.972 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.972 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:00.972 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.972 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.972 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:00.972 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.973 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:00.973 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.973 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:00.973 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.973 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.973 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.973 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.973 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:00.973 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.973 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.973 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:00.973 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.973 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:00.973 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.973 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:00.974 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.974 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.974 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.974 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.974 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:00.974 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.974 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.974 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895776.227017_707.1", 2024-04-12T04:56:00.974 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.974 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:00.974 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.974 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:22:56.227017Z", 2024-04-12T04:56:00.974 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.975 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.975 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.975 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:00.975 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.975 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:00.975 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:00.975 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:00.975 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.975 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:22:56Z" 2024-04-12T04:56:00.975 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:00.975 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895776.227017_707.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:22:56.227017Z', 'realm_epoch': 2}}] 2024-04-12T04:56:00.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895776.234016_710.1'} 2024-04-12T04:56:00.976 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895776.227017_707.1'} 2024-04-12T04:56:00.976 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712895776.234016_710.1 target=1_1712895776.227017_707.1 2024-04-12T04:56:00.976 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.976 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:00.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:00.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:00.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:00.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:23:01.555+0000 7f8b1c8cf780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:00.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:23:01.556+0000 7f8b1c8cf780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:00.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:00.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:00.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:00.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:00.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:00.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:00.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:00.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:00.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:00.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:00.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:00.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:00.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:00.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:00.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:00.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:00.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:00.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:00.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:00.979 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:00.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:00.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:00.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895776.234016_710.1", 2024-04-12T04:56:00.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:00.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:00.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:00.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:22:56.234016Z", 2024-04-12T04:56:00.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:00.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:00.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:00.980 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:00.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:00.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:00.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:23:01Z" 2024-04-12T04:56:00.981 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:00.981 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:00.981 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:00.981 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:00.981 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:00.981 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:00.981 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:00.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.981 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.981 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:00.982 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.982 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:00.982 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.982 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.982 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:00.982 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.982 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:00.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.982 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:00.982 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.982 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.982 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.982 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.982 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:00.982 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.983 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.983 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:00.983 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.983 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:00.983 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.983 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:00.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.983 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.983 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.983 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.983 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:00.983 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.983 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.983 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:00.984 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.984 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:00.984 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.984 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:00.984 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.984 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.984 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.984 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:00.984 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:00.984 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:00.984 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:00.984 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895776.234016_710.1", 2024-04-12T04:56:00.984 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:00.984 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:00.984 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:00.985 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:22:56.234016Z", 2024-04-12T04:56:00.985 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:00.985 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.985 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:00.985 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:00.985 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.985 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:00.985 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:00.985 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:00.985 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:00.985 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:23:01Z" 2024-04-12T04:56:00.985 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:00.985 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712895776.234016_710.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:22:56.234016Z', 'realm_epoch': 2}}] 2024-04-12T04:56:00.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895776.234016_710.1'} 2024-04-12T04:56:00.985 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712895776.234016_710.1'} 2024-04-12T04:56:00.986 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-12T04:56:00.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:00.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:00.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:00.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.986 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.987 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.987 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.987 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:00.987 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:00.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.987 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:56:00.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:56:00.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ab1efb7e51c28df-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj11 2024-04-12T04:56:00.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj11 2024-04-12T04:56:00.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj11 2024-04-12T04:56:00.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:00.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.988 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.988 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:00.988 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.988 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:00.989 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj11 2024-04-12T04:56:00.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.989 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:TamJochJjeBZpFBjdgUM5o3meeA= 2024-04-12T04:56:00.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:TamJochJjeBZpFBjdgUM5o3meeA='} 2024-04-12T04:56:00.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006de889b4c3b0e4a7-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:00.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:00.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:00.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.990 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.990 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.990 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.990 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:00.990 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:00.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.990 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:56:00.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:56:00.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002683771de7f57379-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj22 2024-04-12T04:56:00.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj22 2024-04-12T04:56:00.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj22 2024-04-12T04:56:00.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:00.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.991 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.991 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:00.991 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.991 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:00.992 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj22 2024-04-12T04:56:00.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.992 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:vW6XFzSHFFq/USuIX+tHHzoFR98= 2024-04-12T04:56:00.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:vW6XFzSHFFq/USuIX+tHHzoFR98='} 2024-04-12T04:56:00.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002f4265dda3527e4c-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:00.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:00.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:00.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.993 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.993 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.993 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.993 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:00.993 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:00.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.993 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:56:00.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:56:00.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004f96514ded0fc2c7-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj33 2024-04-12T04:56:00.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj33 2024-04-12T04:56:00.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj33 2024-04-12T04:56:00.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:00.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.994 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.994 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:00.994 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.994 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:00.994 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj33 2024-04-12T04:56:00.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.994 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:4o+RbMX2W659ut9PuKjdMIipQLk= 2024-04-12T04:56:00.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4o+RbMX2W659ut9PuKjdMIipQLk='} 2024-04-12T04:56:00.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe78d4186379afbd-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:00.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:00.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:00.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.995 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.995 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.995 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.995 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:00.995 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:00.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.996 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:56:00.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:56:00.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7448aa70cc65272-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj44 2024-04-12T04:56:00.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj44 2024-04-12T04:56:00.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj44 2024-04-12T04:56:00.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:00.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.996 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.997 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:00.997 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.997 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:00.997 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj44 2024-04-12T04:56:00.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.997 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:lLt+sILykaAxamIhhmvp8+031OU= 2024-04-12T04:56:00.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lLt+sILykaAxamIhhmvp8+031OU='} 2024-04-12T04:56:00.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fea2da3370bf44b6-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:00.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:00.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:00.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:00.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:00.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.998 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:00.998 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.998 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:00.998 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:00.998 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:00.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.998 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:56:00.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:56:00.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b2e220156bef33d-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:00.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj55 2024-04-12T04:56:00.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj55 2024-04-12T04:56:00.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:00.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj55 2024-04-12T04:56:00.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:00.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:00.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:00.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:00.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:00.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:00.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:00.999 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:00.999 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:00.999 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:00.999 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:00.999 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj55 2024-04-12T04:56:00.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:00.999 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:zOLyprXBA0N7JjjbVf6YjfIsFo4= 2024-04-12T04:56:01.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zOLyprXBA0N7JjjbVf6YjfIsFo4='} 2024-04-12T04:56:01.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001af9cf35a13ee4ee-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.000 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.000 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-65 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: tenant: 2024-04-12T04:56:01.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: bucket name: dkyges-65 2024-04-12T04:56:01.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: total entries: 5 2024-04-12T04:56:01.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.001 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.001 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.001 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.001 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:01.001 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.001 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:56:01.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:56:01.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dd1f30e1e10b3aa1-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj17 2024-04-12T04:56:01.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj17 2024-04-12T04:56:01.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj17 2024-04-12T04:56:01.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.002 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.002 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.003 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.003 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:01.003 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj17 2024-04-12T04:56:01.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.003 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:jeStFdceOfVVYnEPMraEvTI7idA= 2024-04-12T04:56:01.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:jeStFdceOfVVYnEPMraEvTI7idA='} 2024-04-12T04:56:01.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000694b66c105622b09-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.004 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.004 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.004 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.004 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:01.004 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.004 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:56:01.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:56:01.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcc6943a2f449dd4-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj34 2024-04-12T04:56:01.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj34 2024-04-12T04:56:01.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj34 2024-04-12T04:56:01.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.005 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.005 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.006 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.006 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:01.006 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj34 2024-04-12T04:56:01.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.006 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:paXK8eL4odhwMkbzaVmG67LexuU= 2024-04-12T04:56:01.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:paXK8eL4odhwMkbzaVmG67LexuU='} 2024-04-12T04:56:01.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074df2edcfd806301-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.007 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.007 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.007 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.007 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:01.007 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.008 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:56:01.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:56:01.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002bdb4210e0307c32-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj51 2024-04-12T04:56:01.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj51 2024-04-12T04:56:01.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj51 2024-04-12T04:56:01.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.009 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.009 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.009 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.009 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:01.009 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj51 2024-04-12T04:56:01.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.009 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:a4wNOQzbyh9uH1WJ9NfH8grScmU= 2024-04-12T04:56:01.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:a4wNOQzbyh9uH1WJ9NfH8grScmU='} 2024-04-12T04:56:01.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b02fe0beeddb761f-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.010 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.010 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.010 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.010 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:01.010 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.011 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0= 2024-04-12T04:56:01.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cK8E9xM1ZTmScz1HbdlM78S7eK0=', 'Content-Length': '0'} 2024-04-12T04:56:01.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006cc60a27c872ddbb-006618b725-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj68 2024-04-12T04:56:01.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj68 2024-04-12T04:56:01.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj68 2024-04-12T04:56:01.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.012 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.012 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.012 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.012 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:01 GMT 2024-04-12T04:56:01.012 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj68 2024-04-12T04:56:01.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.012 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:otf2YyPKShxKETiJY+CAsw635bc= 2024-04-12T04:56:01.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:01 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:otf2YyPKShxKETiJY+CAsw635bc='} 2024-04-12T04:56:01.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000bd2f1835c90684a-006618b725-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:01 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.013 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.013 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.013 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.013 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.014 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.014 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:56:01.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:56:01.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a598f786a4e4176d-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj85 2024-04-12T04:56:01.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj85 2024-04-12T04:56:01.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj85 2024-04-12T04:56:01.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.015 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.015 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.015 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.015 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.015 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj85 2024-04-12T04:56:01.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.015 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:3ATwZxRNfIsdtWFNMcVEuKfN5Ks= 2024-04-12T04:56:01.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3ATwZxRNfIsdtWFNMcVEuKfN5Ks='} 2024-04-12T04:56:01.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ca36133c1f8c897-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.015 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.016 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-65 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: tenant: 2024-04-12T04:56:01.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: bucket name: dkyges-65 2024-04-12T04:56:01.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: total entries: 10 2024-04-12T04:56:01.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.017 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.017 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.017 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.017 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.017 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.017 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:56:01.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:56:01.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c765b4102e8d99a4-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj19 2024-04-12T04:56:01.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj19 2024-04-12T04:56:01.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj19 2024-04-12T04:56:01.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.018 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.018 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.018 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.018 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.018 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj19 2024-04-12T04:56:01.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.018 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:nrRRiqSTb8PSSdORUl0k0/JjOpA= 2024-04-12T04:56:01.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nrRRiqSTb8PSSdORUl0k0/JjOpA='} 2024-04-12T04:56:01.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042d8d750c8ddb9ee-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.019 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.020 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.020 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.020 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.020 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.020 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:56:01.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:56:01.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007b3f051cdfd759b-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj38 2024-04-12T04:56:01.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj38 2024-04-12T04:56:01.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj38 2024-04-12T04:56:01.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.021 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.021 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.021 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.021 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.021 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj38 2024-04-12T04:56:01.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.021 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:FVblpfpL8Ke6mzJfZ7lEPdENVSQ= 2024-04-12T04:56:01.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FVblpfpL8Ke6mzJfZ7lEPdENVSQ='} 2024-04-12T04:56:01.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ffad7725ac20bb61-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.022 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.022 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.022 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.022 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.023 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.023 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:56:01.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:56:01.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5c5e41acef08872-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj57 2024-04-12T04:56:01.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj57 2024-04-12T04:56:01.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj57 2024-04-12T04:56:01.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.024 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.024 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.024 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.024 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.024 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj57 2024-04-12T04:56:01.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.024 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:awSFzwG9MnVEuaTg21PVYVnzNP8= 2024-04-12T04:56:01.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:awSFzwG9MnVEuaTg21PVYVnzNP8='} 2024-04-12T04:56:01.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000038a8581b85c047dc-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.025 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.025 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.025 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.025 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.025 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.025 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:56:01.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:56:01.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001585a2499aacdd78-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj76 2024-04-12T04:56:01.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj76 2024-04-12T04:56:01.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj76 2024-04-12T04:56:01.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.026 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.026 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.026 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.026 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.026 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj76 2024-04-12T04:56:01.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.027 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:7tNsvkv+ctes0wcKUDEjeTn8KXg= 2024-04-12T04:56:01.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:7tNsvkv+ctes0wcKUDEjeTn8KXg='} 2024-04-12T04:56:01.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ef29de6b8f0522c-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.028 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.028 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.028 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.028 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.028 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.028 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:56:01.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:56:01.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f670f7750b6077f-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj95 2024-04-12T04:56:01.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj95 2024-04-12T04:56:01.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj95 2024-04-12T04:56:01.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.029 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.029 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.029 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.029 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.029 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj95 2024-04-12T04:56:01.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.029 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:OO5qQ+z3WwxyJEpL8vhVdsZ01bI= 2024-04-12T04:56:01.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OO5qQ+z3WwxyJEpL8vhVdsZ01bI='} 2024-04-12T04:56:01.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000065b58d984df16681-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.030 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.030 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-65 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: tenant: 2024-04-12T04:56:01.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: bucket name: dkyges-65 2024-04-12T04:56:01.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: total entries: 15 2024-04-12T04:56:01.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.031 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.031 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.031 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.031 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.031 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.031 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:56:01.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:56:01.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005f4ad25ad546c7a5-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj23 2024-04-12T04:56:01.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj23 2024-04-12T04:56:01.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj23 2024-04-12T04:56:01.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.032 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.032 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.032 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.033 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.033 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj23 2024-04-12T04:56:01.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.033 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:0HbJcbbBig3Y8D4RGmDBGSYMIwM= 2024-04-12T04:56:01.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0HbJcbbBig3Y8D4RGmDBGSYMIwM='} 2024-04-12T04:56:01.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dd9c799f217b3a69-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.034 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.034 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.034 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.034 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.034 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.034 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:56:01.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:56:01.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000345dcb4da22625e9-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj46 2024-04-12T04:56:01.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj46 2024-04-12T04:56:01.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj46 2024-04-12T04:56:01.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.035 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.035 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.035 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.035 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.036 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj46 2024-04-12T04:56:01.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.036 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:UN/y0hR19VOQlxqsyI7fLI91eww= 2024-04-12T04:56:01.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:UN/y0hR19VOQlxqsyI7fLI91eww='} 2024-04-12T04:56:01.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000149a7f7383e148ab-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.037 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.037 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.037 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.037 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.037 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.037 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:56:01.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:56:01.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eba0f01b77673e47-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj69 2024-04-12T04:56:01.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj69 2024-04-12T04:56:01.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj69 2024-04-12T04:56:01.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.038 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.038 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.038 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.038 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.038 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj69 2024-04-12T04:56:01.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.039 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:2Xp8ozrM2dR0je6C2GYmpnmRsHI= 2024-04-12T04:56:01.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2Xp8ozrM2dR0je6C2GYmpnmRsHI='} 2024-04-12T04:56:01.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000787e78e40c6d61be-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.040 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.040 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.040 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.040 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.040 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.040 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:56:01.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:56:01.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000555dc94fab24931-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj92 2024-04-12T04:56:01.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj92 2024-04-12T04:56:01.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj92 2024-04-12T04:56:01.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.041 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.041 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.041 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.041 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.041 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj92 2024-04-12T04:56:01.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.041 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:RzjJ24UeNho3rjwkdckdiCqDkUU= 2024-04-12T04:56:01.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RzjJ24UeNho3rjwkdckdiCqDkUU='} 2024-04-12T04:56:01.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081ec7a2310dd87ed-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.042 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.042 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.042 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.042 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.042 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.042 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:56:01.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:56:01.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006164995ba18fd317-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj115 2024-04-12T04:56:01.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj115 2024-04-12T04:56:01.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj115 2024-04-12T04:56:01.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.043 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.043 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.043 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.044 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.044 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj115 2024-04-12T04:56:01.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.044 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:OmxnuPrlYmx3ssIsTaxPHGsZahc= 2024-04-12T04:56:01.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OmxnuPrlYmx3ssIsTaxPHGsZahc='} 2024-04-12T04:56:01.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085d013cd2849f50e-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.044 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.044 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-65 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: tenant: 2024-04-12T04:56:01.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: bucket name: dkyges-65 2024-04-12T04:56:01.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: total entries: 20 2024-04-12T04:56:01.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.045 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.045 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.045 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.045 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.045 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.046 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:56:01.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:56:01.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a79648cc58c9b28-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj29 2024-04-12T04:56:01.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj29 2024-04-12T04:56:01.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj29 2024-04-12T04:56:01.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.046 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.047 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.047 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.047 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.047 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj29 2024-04-12T04:56:01.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.047 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:KYonadGBROkkQ2N2Z/e0YtrMEOQ= 2024-04-12T04:56:01.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KYonadGBROkkQ2N2Z/e0YtrMEOQ='} 2024-04-12T04:56:01.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b865916a46c8529e-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.048 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.048 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.048 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.048 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.048 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.048 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:56:01.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:56:01.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036a5c661a06ff808-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj58 2024-04-12T04:56:01.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj58 2024-04-12T04:56:01.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj58 2024-04-12T04:56:01.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.049 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.049 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.049 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.049 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.049 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj58 2024-04-12T04:56:01.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.050 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ETITXRId2uDkuG0Prz9yasDeIW0= 2024-04-12T04:56:01.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ETITXRId2uDkuG0Prz9yasDeIW0='} 2024-04-12T04:56:01.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2f7a570d541185e-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.051 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.051 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.051 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.051 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.051 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.051 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:56:01.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:56:01.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000a55ddc4ed4f7e7-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj87 2024-04-12T04:56:01.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj87 2024-04-12T04:56:01.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj87 2024-04-12T04:56:01.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.052 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.052 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.052 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.052 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.052 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj87 2024-04-12T04:56:01.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.053 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:XOO3f8ywOTv44/Ed8wzs4BX52sU= 2024-04-12T04:56:01.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:XOO3f8ywOTv44/Ed8wzs4BX52sU='} 2024-04-12T04:56:01.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043de237aca5f3d1e-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.054 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.054 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.054 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.054 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.054 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.054 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg= 2024-04-12T04:56:01.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PkRKNbmf92UlO3ppJhFt0IEyhpg=', 'Content-Length': '0'} 2024-04-12T04:56:01.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089051a60b7fb597b-006618b726-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj116 2024-04-12T04:56:01.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj116 2024-04-12T04:56:01.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj116 2024-04-12T04:56:01.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.055 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.055 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.055 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.055 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:02 GMT 2024-04-12T04:56:01.056 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj116 2024-04-12T04:56:01.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.056 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:+I6LUH7r9GY26hpHxgU3nE3ePEo= 2024-04-12T04:56:01.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:02 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+I6LUH7r9GY26hpHxgU3nE3ePEo='} 2024-04-12T04:56:01.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000eafc8e5b29dd38d-006618b726-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:02 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.057 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.057 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.057 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.057 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.057 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.057 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:56:01.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:56:01.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000de8c80993a6f2da3-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj145 2024-04-12T04:56:01.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj145 2024-04-12T04:56:01.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj145 2024-04-12T04:56:01.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.058 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.058 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.058 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.058 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.058 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj145 2024-04-12T04:56:01.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.058 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:931H1F/SoWE88Zx+Am5TK5cGHCs= 2024-04-12T04:56:01.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:931H1F/SoWE88Zx+Am5TK5cGHCs='} 2024-04-12T04:56:01.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000185f3d96c80c6d4c-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.059 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.059 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-65 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: tenant: 2024-04-12T04:56:01.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: bucket name: dkyges-65 2024-04-12T04:56:01.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: total entries: 25 2024-04-12T04:56:01.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.060 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.060 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.060 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.060 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.060 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.060 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:56:01.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:56:01.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000211fca70d7c8bb4a-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj31 2024-04-12T04:56:01.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj31 2024-04-12T04:56:01.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj31 2024-04-12T04:56:01.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.061 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.061 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.062 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.062 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.062 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj31 2024-04-12T04:56:01.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.062 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NqXx1fCsyPL6EKAbKLwhzjvRmcY= 2024-04-12T04:56:01.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NqXx1fCsyPL6EKAbKLwhzjvRmcY='} 2024-04-12T04:56:01.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097bafad68a6aacae-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.063 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.063 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.063 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.063 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.063 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.063 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:56:01.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:56:01.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006438f6101e62ffab-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj62 2024-04-12T04:56:01.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj62 2024-04-12T04:56:01.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj62 2024-04-12T04:56:01.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.064 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.064 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.064 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.064 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.064 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj62 2024-04-12T04:56:01.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.065 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:eIEeBn+DSXFNVvhOMwJmKxA3CAE= 2024-04-12T04:56:01.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eIEeBn+DSXFNVvhOMwJmKxA3CAE='} 2024-04-12T04:56:01.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e5ced51b44ea508-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.066 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.066 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.066 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.066 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.066 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.066 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:56:01.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:56:01.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008673f053b1808817-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj93 2024-04-12T04:56:01.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj93 2024-04-12T04:56:01.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj93 2024-04-12T04:56:01.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.067 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.067 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.067 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.067 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.067 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj93 2024-04-12T04:56:01.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.067 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:eiAfpjr6RWJ2jozp2mP0zfoMDu8= 2024-04-12T04:56:01.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eiAfpjr6RWJ2jozp2mP0zfoMDu8='} 2024-04-12T04:56:01.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061ff966f9dde377e-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.068 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.068 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.068 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.068 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.068 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.069 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:56:01.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:56:01.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7ee098ef9b6c0c8-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj124 2024-04-12T04:56:01.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj124 2024-04-12T04:56:01.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj124 2024-04-12T04:56:01.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.070 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.070 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.070 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.070 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.070 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj124 2024-04-12T04:56:01.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.070 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:nyAm+Sg30vDhYjAUlsgKpBQzWA4= 2024-04-12T04:56:01.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nyAm+Sg30vDhYjAUlsgKpBQzWA4='} 2024-04-12T04:56:01.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bf526fb2fcffb42d-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.071 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.071 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.071 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.072 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.072 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.072 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:56:01.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:56:01.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025bf09acab2cbb60-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj155 2024-04-12T04:56:01.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj155 2024-04-12T04:56:01.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj155 2024-04-12T04:56:01.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.073 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.073 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.073 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.073 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.073 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj155 2024-04-12T04:56:01.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.073 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:RRo1rvw5T9IgCifo1Df/ytTniIo= 2024-04-12T04:56:01.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RRo1rvw5T9IgCifo1Df/ytTniIo='} 2024-04-12T04:56:01.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006ca3bccf30f43883-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.073 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.074 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-65', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-65 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: tenant: 2024-04-12T04:56:01.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: bucket name: dkyges-65 2024-04-12T04:56:01.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: total entries: 30 2024-04-12T04:56:01.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.075 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.075 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.075 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.075 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.075 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.075 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:56:01.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:56:01.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d2413d18292a0a8-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj37 2024-04-12T04:56:01.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj37 2024-04-12T04:56:01.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj37 2024-04-12T04:56:01.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.076 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.076 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.076 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.076 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.076 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj37 2024-04-12T04:56:01.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.077 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:cIOpdB4ukxj/JU7ubn898MEHmZU= 2024-04-12T04:56:01.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cIOpdB4ukxj/JU7ubn898MEHmZU='} 2024-04-12T04:56:01.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d5cee8cbaef31ac5-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.078 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.078 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.078 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.078 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.078 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.078 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:56:01.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:56:01.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000285c4dc4667ab4ff-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj74 2024-04-12T04:56:01.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj74 2024-04-12T04:56:01.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj74 2024-04-12T04:56:01.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.079 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.079 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.079 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.079 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.079 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj74 2024-04-12T04:56:01.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.079 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:4/rDqyX1IYUDDrB6U4UuRjP9hc0= 2024-04-12T04:56:01.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:4/rDqyX1IYUDDrB6U4UuRjP9hc0='} 2024-04-12T04:56:01.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe31ccb3f6291ee7-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.080 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.080 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.080 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.081 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.081 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.081 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:56:01.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:56:01.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4a730ca264f3510-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj111 2024-04-12T04:56:01.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj111 2024-04-12T04:56:01.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj111 2024-04-12T04:56:01.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.082 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.082 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.082 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.082 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.082 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj111 2024-04-12T04:56:01.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.082 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ElsVJ3u4wc9eO1BYeFw0kWTt8Sk= 2024-04-12T04:56:01.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ElsVJ3u4wc9eO1BYeFw0kWTt8Sk='} 2024-04-12T04:56:01.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000722996a2481c6fd1-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.083 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.083 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.083 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.083 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.083 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.083 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:56:01.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:56:01.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b74ec14649c71e4-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj148 2024-04-12T04:56:01.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj148 2024-04-12T04:56:01.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj148 2024-04-12T04:56:01.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.084 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.084 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.084 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.085 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.085 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj148 2024-04-12T04:56:01.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.085 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:MrUw6dyUQFgLIolBi/bH7nX1awc= 2024-04-12T04:56:01.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MrUw6dyUQFgLIolBi/bH7nX1awc='} 2024-04-12T04:56:01.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033b7c61e7d8198a9-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/ 2024-04-12T04:56:01.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/ 2024-04-12T04:56:01.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/ 2024-04-12T04:56:01.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.086 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.086 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.086 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.086 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.086 INFO:tasks.rgw_multisite_tests:/dkyges-65/ 2024-04-12T04:56:01.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.086 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY= 2024-04-12T04:56:01.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tiQCsLhhTnC3ngHAzFTv0kjCGBY=', 'Content-Length': '0'} 2024-04-12T04:56:01.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006d696c3412d99b80-006618b727-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-65/obj185 2024-04-12T04:56:01.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-65/obj185 2024-04-12T04:56:01.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-65/obj185 2024-04-12T04:56:01.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.087 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.087 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.087 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.087 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:23:03 GMT 2024-04-12T04:56:01.087 INFO:tasks.rgw_multisite_tests:/dkyges-65/obj185 2024-04-12T04:56:01.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.088 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:I/x7RqIA1dMEqLRcddsqisshSYc= 2024-04-12T04:56:01.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:23:03 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I/x7RqIA1dMEqLRcddsqisshSYc='} 2024-04-12T04:56:01.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c56e2c487296f33-006618b727-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:23:03 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.088 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dkyges-65 2024-04-12T04:56:01.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-65', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:01.088 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-65', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:01.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-65 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:56:01.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:23:03.913+0000 7f83aaad6780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:23:08.915+0000 7f83aaad6780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:23:13.915+0000 7f83aaad6780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:23:18.919+0000 7f83aaad6780 1 bucket sync caught up with source: 2024-04-12T04:56:01.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [, , , , , , 00000000002.203.6, , , , , , 00000000002.100.6, , , , , 00000000002.182.6, , , , , , , , , , , , , , , 00000000001.119.6, , , , 00000000001.112.6] 2024-04-12T04:56:01.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [, , , , , , 00000000002.203.6, , , , , , 00000000002.100.6, , , , , 00000000002.182.6, , , , , , , , , , , , , , , 00000000001.119.6, , , , 00000000001.112.6] 2024-04-12T04:56:01.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:23:18.919+0000 7f83aaad6780 0 bucket checkpoint complete 2024-04-12T04:56:01.089 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=dkyges-65 2024-04-12T04:56:01.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-65', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:01.089 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-65', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:01.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-65 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:56:01.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:23:19.044+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:23:24.044+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:23:29.045+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:23:34.045+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:23:39.045+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:23:44.046+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:23:49.047+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:23:54.047+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:23:59.048+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:24:04.049+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:24:09.049+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:24:14.050+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:24:19.051+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:24:24.052+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:24:29.052+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:24:34.052+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:24:39.053+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:24:44.054+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:24:49.054+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:24:54.055+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:24:59.056+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:25:04.056+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:25:09.057+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:25:14.057+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:25:19.058+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:25:24.059+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:25:29.059+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.091 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi043.stdout: INFO: 2024-04-12T04:25:33.989+0000 7f1e272bd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:56:01.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:25:34.060+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:25:39.062+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:25:44.061+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:25:49.062+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:25:54.063+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:25:59.063+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:26:04.064+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:26:09.065+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:26:14.065+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:26:19.066+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:26:24.067+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:26:29.068+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:26:34.068+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:26:39.069+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:26:44.070+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:26:49.071+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:26:54.072+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:26:59.072+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:27:04.072+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:27:09.073+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:27:14.074+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:27:19.074+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:27:24.075+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:27:29.076+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:27:34.076+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:27:39.077+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:27:44.077+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:27:49.078+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:27:54.078+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:27:59.079+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:04.080+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:09.082+0000 7ff35ff2a780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:14.082+0000 7ff35ff2a780 -1 bucket checkpoint timed out waiting to reach incremental sync 2024-04-12T04:56:01.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:14.082+0000 7ff35ff2a780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:56:01.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:14.082+0000 7ff35ff2a780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:56:01.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-12T04:56:01.094 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-12T04:56:01.094 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.094 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-12T04:56:01.094 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-04-12T04:56:01.094 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-12T04:56:01.095 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-12T04:56:01.095 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-12T04:56:01.095 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-12T04:56:01.095 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 1937, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2024-04-12T04:56:01.095 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-04-12T04:56:01.095 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2024-04-12T04:56:01.096 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2024-04-12T04:56:01.096 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2024-04-12T04:56:01.096 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2024-04-12T04:56:01.096 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite.py", line 197, in admin 2024-04-12T04:56:01.096 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-04-12T04:56:01.096 INFO:tasks.rgw_multisite_tests:AssertionError: 2024-04-12T04:56:01.096 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-12T04:56:01.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:56:01.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.097 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.097 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.097 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.097 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:14 GMT 2024-04-12T04:56:01.097 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.097 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:uCHE4TL56iUfx6MKliM7UpBIrEY= 2024-04-12T04:56:01.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uCHE4TL56iUfx6MKliM7UpBIrEY=', 'Content-Length': '0'} 2024-04-12T04:56:01.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000dc48d1bdf0644d5d-006618b85e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=dkyges-66 2024-04-12T04:56:01.098 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.098 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:56:01.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: [ 2024-04-12T04:56:01.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.098 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:56:01.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:56:01.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:56:01.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896094.156795_758.1", 2024-04-12T04:56:01.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T04:28:14.156795Z" 2024-04-12T04:56:01.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.099 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896094.156795_758.1'} 2024-04-12T04:56:01.100 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-12T04:56:01.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.100 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:14.373+0000 7f8673989780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:14.374+0000 7f8673989780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.100 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.101 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.102 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.103 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896094.156795_758.1", 2024-04-12T04:56:01.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:28:14.156795Z", 2024-04-12T04:56:01.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.104 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:28:14Z" 2024-04-12T04:56:01.105 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.105 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.105 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.105 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.105 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.105 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.105 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.105 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.105 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.105 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.105 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.105 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.105 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.105 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.105 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.105 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.106 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.106 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.106 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.106 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.106 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.106 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.106 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.106 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.106 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.106 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.106 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.106 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.106 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.107 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.107 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.107 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.107 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.107 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.107 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.107 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.107 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.107 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.107 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.107 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.107 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.107 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.107 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.107 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.107 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.108 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.108 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.108 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.108 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.108 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.108 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896094.156795_758.1", 2024-04-12T04:56:01.108 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.108 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.108 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.108 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:28:14.156795Z", 2024-04-12T04:56:01.108 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.108 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.108 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.108 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.108 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.109 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.109 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.109 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.109 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.109 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:28:14Z" 2024-04-12T04:56:01.109 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.109 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896094.156795_758.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:14.156795Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896094.156795_758.1'} 2024-04-12T04:56:01.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896094.156795_758.1'} 2024-04-12T04:56:01.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-12T04:56:01.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.110 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.110 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.110 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.110 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:14 GMT 2024-04-12T04:56:01.110 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.110 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM= 2024-04-12T04:56:01.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM=', 'Content-Length': '0'} 2024-04-12T04:56:01.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecdbb167e34d4527-006618b85e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11 2024-04-12T04:56:01.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11 2024-04-12T04:56:01.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj11 2024-04-12T04:56:01.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.112 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.112 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.112 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.112 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:14 GMT 2024-04-12T04:56:01.112 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj11 2024-04-12T04:56:01.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.112 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:FtQXxFKbmW9ETaMrdGaFpjmg4to= 2024-04-12T04:56:01.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:FtQXxFKbmW9ETaMrdGaFpjmg4to='} 2024-04-12T04:56:01.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c798cd0c631c0856-006618b85e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.113 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.113 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.113 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.113 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:14 GMT 2024-04-12T04:56:01.113 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.113 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM= 2024-04-12T04:56:01.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM=', 'Content-Length': '0'} 2024-04-12T04:56:01.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea63812dabe16235-006618b85e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22 2024-04-12T04:56:01.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22 2024-04-12T04:56:01.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj22 2024-04-12T04:56:01.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.114 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.115 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.115 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.115 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:14 GMT 2024-04-12T04:56:01.115 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj22 2024-04-12T04:56:01.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.115 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:OmBPDwlNo5wXcuv305+WkMDac3w= 2024-04-12T04:56:01.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OmBPDwlNo5wXcuv305+WkMDac3w='} 2024-04-12T04:56:01.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e76753a143ad4f08-006618b85e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.116 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.116 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.116 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.116 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:14 GMT 2024-04-12T04:56:01.116 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.116 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM= 2024-04-12T04:56:01.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM=', 'Content-Length': '0'} 2024-04-12T04:56:01.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a617b3cc23de1486-006618b85e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33 2024-04-12T04:56:01.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33 2024-04-12T04:56:01.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj33 2024-04-12T04:56:01.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.117 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.117 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.118 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.118 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:14 GMT 2024-04-12T04:56:01.118 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj33 2024-04-12T04:56:01.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.118 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:m5gbihvPkV+XMSCV+pTylSEFuAo= 2024-04-12T04:56:01.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:m5gbihvPkV+XMSCV+pTylSEFuAo='} 2024-04-12T04:56:01.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a3a0fd2dae1a2cae-006618b85e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.119 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.119 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.119 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.119 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:14 GMT 2024-04-12T04:56:01.119 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.119 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM= 2024-04-12T04:56:01.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM=', 'Content-Length': '0'} 2024-04-12T04:56:01.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d10d33712f271895-006618b85e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44 2024-04-12T04:56:01.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44 2024-04-12T04:56:01.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj44 2024-04-12T04:56:01.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.120 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.120 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.121 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.121 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:14 GMT 2024-04-12T04:56:01.121 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj44 2024-04-12T04:56:01.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.121 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:y5CNBw+Gx4BUkLe8Aprw/5NpopM= 2024-04-12T04:56:01.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:y5CNBw+Gx4BUkLe8Aprw/5NpopM='} 2024-04-12T04:56:01.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d357be6dfdd2bf11-006618b85e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.122 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.122 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.122 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.122 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:14 GMT 2024-04-12T04:56:01.122 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.122 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM= 2024-04-12T04:56:01.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9SPEixSGpSNmBlFQRkJxdB/ibzM=', 'Content-Length': '0'} 2024-04-12T04:56:01.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009845d9bec2a015f2-006618b85e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55 2024-04-12T04:56:01.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55 2024-04-12T04:56:01.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj55 2024-04-12T04:56:01.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.123 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.123 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.123 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.123 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:14 GMT 2024-04-12T04:56:01.123 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj55 2024-04-12T04:56:01.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.123 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:zybU6MoaPHY3dZwh0lKx7UOtpuI= 2024-04-12T04:56:01.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:14 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:zybU6MoaPHY3dZwh0lKx7UOtpuI='} 2024-04-12T04:56:01.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001273841af1fe5561-006618b85e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dkyges-66 2024-04-12T04:56:01.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:01.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:01.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-66 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:56:01.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:14.598+0000 7fc61d715780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:19.599+0000 7fc61d715780 1 bucket sync caught up with source: 2024-04-12T04:56:01.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [, , , , , , , , 00000000001.131.6, 00000000002.104.6, 00000000002.82.6] 2024-04-12T04:56:01.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [, , , , , , , , 00000000001.131.6, 00000000002.104.6, 00000000002.82.6] 2024-04-12T04:56:01.124 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:19.599+0000 7fc61d715780 0 bucket checkpoint complete 2024-04-12T04:56:01.124 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=dkyges-66 2024-04-12T04:56:01.124 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-66', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:01.125 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-66', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:01.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-66 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:56:01.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:19.728+0000 7f344e100780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:24.728+0000 7f344e100780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:29.729+0000 7f344e100780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:34.729+0000 7f344e100780 1 waiting to reach incremental sync.. 2024-04-12T04:56:01.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:39.731+0000 7f344e100780 1 bucket sync caught up with source: 2024-04-12T04:56:01.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: local status: [, , , , , , , , 00000000001.1993.6, 00000000002.2069.6, 00000000002.1098.6] 2024-04-12T04:56:01.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: remote markers: [, , , , , , , , 00000000001.1993.6, 00000000002.2069.6, 00000000002.1098.6] 2024-04-12T04:56:01.125 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:39.731+0000 7f344e100780 0 bucket checkpoint complete 2024-04-12T04:56:01.125 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=dkyges-66 zones={a2, a1} 2024-04-12T04:56:01.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:56:01.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.126 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.126 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.126 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.126 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.126 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.127 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:lhzWmSEIFCkgawr6QCKWJGgN768= 2024-04-12T04:56:01.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lhzWmSEIFCkgawr6QCKWJGgN768=', 'Content-Length': '0'} 2024-04-12T04:56:01.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000709e98e136150fcc-006618b877-216b-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.128 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.128 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.128 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.128 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.128 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.128 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:lhzWmSEIFCkgawr6QCKWJGgN768= 2024-04-12T04:56:01.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lhzWmSEIFCkgawr6QCKWJGgN768=', 'Content-Length': '0'} 2024-04-12T04:56:01.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e3a15561db574e4e-006618b877-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.128 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2024-04-12T04:56:01.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?versions&max-keys=999 2024-04-12T04:56:01.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?versions&max-keys=999 2024-04-12T04:56:01.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?versions&max-keys=999 2024-04-12T04:56:01.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.129 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.129 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.129 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.129 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.130 INFO:tasks.rgw_multisite_tests:/dkyges-66/?versions 2024-04-12T04:56:01.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.130 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:wQWmXJpssIgVlJIu+qPIp7bBC+w= 2024-04-12T04:56:01.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wQWmXJpssIgVlJIu+qPIp7bBC+w=', 'Content-Length': '0'} 2024-04-12T04:56:01.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005086fda2dad11350-006618b877-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66999falseobj11nulltrue2024-04-12T04:28:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:28:14.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:28:14.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:28:14.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:28:14.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.130 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2024-04-12T04:56:01.130 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2024-04-12T04:56:01.130 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2024-04-12T04:56:01.130 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2024-04-12T04:56:01.130 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2024-04-12T04:56:01.130 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2024-04-12T04:56:01.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?versions&max-keys=999 2024-04-12T04:56:01.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?versions&max-keys=999 2024-04-12T04:56:01.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?versions&max-keys=999 2024-04-12T04:56:01.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.131 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.131 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.131 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.131 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.132 INFO:tasks.rgw_multisite_tests:/dkyges-66/?versions 2024-04-12T04:56:01.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.132 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:wQWmXJpssIgVlJIu+qPIp7bBC+w= 2024-04-12T04:56:01.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wQWmXJpssIgVlJIu+qPIp7bBC+w=', 'Content-Length': '0'} 2024-04-12T04:56:01.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f740bd0d998b582-006618b877-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66999falseobj11nulltrue2024-04-12T04:28:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:28:14.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:28:14.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:28:14.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:28:14.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.132 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2024-04-12T04:56:01.132 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2024-04-12T04:56:01.132 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2024-04-12T04:56:01.132 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2024-04-12T04:56:01.132 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2024-04-12T04:56:01.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?versions&max-keys=999 2024-04-12T04:56:01.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?versions&max-keys=999 2024-04-12T04:56:01.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?versions&max-keys=999 2024-04-12T04:56:01.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.133 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.133 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.133 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.133 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.133 INFO:tasks.rgw_multisite_tests:/dkyges-66/?versions 2024-04-12T04:56:01.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.134 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:wQWmXJpssIgVlJIu+qPIp7bBC+w= 2024-04-12T04:56:01.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wQWmXJpssIgVlJIu+qPIp7bBC+w=', 'Content-Length': '0'} 2024-04-12T04:56:01.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039f426f230821857-006618b877-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66999falseobj11nulltrue2024-04-12T04:28:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:28:14.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:28:14.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:28:14.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:28:14.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?versions&max-keys=999 2024-04-12T04:56:01.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?versions&max-keys=999 2024-04-12T04:56:01.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?versions&max-keys=999 2024-04-12T04:56:01.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.135 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.135 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.135 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.135 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.135 INFO:tasks.rgw_multisite_tests:/dkyges-66/?versions 2024-04-12T04:56:01.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.135 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:wQWmXJpssIgVlJIu+qPIp7bBC+w= 2024-04-12T04:56:01.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:wQWmXJpssIgVlJIu+qPIp7bBC+w=', 'Content-Length': '0'} 2024-04-12T04:56:01.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2cfcce1e5ec5877-006618b877-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66999falseobj11nulltrue2024-04-12T04:28:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2024-04-12T04:28:14.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2024-04-12T04:28:14.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2024-04-12T04:28:14.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2024-04-12T04:28:14.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.136 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-12T04:56:01.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11 2024-04-12T04:56:01.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11 2024-04-12T04:56:01.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.137 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.137 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.137 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.137 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.137 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.137 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NPg1Jgi2fRRc556r/yX2tkMSbPI= 2024-04-12T04:56:01.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NPg1Jgi2fRRc556r/yX2tkMSbPI=', 'Content-Length': '0'} 2024-04-12T04:56:01.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a30cc8e32c80ef0d-006618b877-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11 2024-04-12T04:56:01.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11 2024-04-12T04:56:01.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.138 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.138 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.138 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.138 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.139 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.139 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NPg1Jgi2fRRc556r/yX2tkMSbPI= 2024-04-12T04:56:01.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NPg1Jgi2fRRc556r/yX2tkMSbPI=', 'Content-Length': '0'} 2024-04-12T04:56:01.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009668e115a5817689-006618b877-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11 2024-04-12T04:56:01.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11 2024-04-12T04:56:01.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.140 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.140 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.140 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.140 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.140 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.140 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:GHJhXGOiwjJb2c9/dNB9J/PL4mU= 2024-04-12T04:56:01.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GHJhXGOiwjJb2c9/dNB9J/PL4mU=', 'Content-Length': '0'} 2024-04-12T04:56:01.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee1c3786c495a6c0-006618b877-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11 2024-04-12T04:56:01.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11 2024-04-12T04:56:01.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.141 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.142 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.142 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.142 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.142 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.142 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:GHJhXGOiwjJb2c9/dNB9J/PL4mU= 2024-04-12T04:56:01.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GHJhXGOiwjJb2c9/dNB9J/PL4mU=', 'Content-Length': '0'} 2024-04-12T04:56:01.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f14c34d93f165d04-006618b877-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.142 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-12T04:56:01.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11 2024-04-12T04:56:01.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11 2024-04-12T04:56:01.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.143 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.143 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.143 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.143 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.143 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.144 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NPg1Jgi2fRRc556r/yX2tkMSbPI= 2024-04-12T04:56:01.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NPg1Jgi2fRRc556r/yX2tkMSbPI=', 'Content-Length': '0'} 2024-04-12T04:56:01.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b40ae8a120346767-006618b877-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11 2024-04-12T04:56:01.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11 2024-04-12T04:56:01.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.145 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.145 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.145 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.145 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.145 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj11?versionId=null 2024-04-12T04:56:01.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.145 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:NPg1Jgi2fRRc556r/yX2tkMSbPI= 2024-04-12T04:56:01.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:NPg1Jgi2fRRc556r/yX2tkMSbPI=', 'Content-Length': '0'} 2024-04-12T04:56:01.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cca3867ad53a05f-006618b877-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj11 2024-04-12T04:56:01.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj11 2024-04-12T04:56:01.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj11 2024-04-12T04:56:01.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.146 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.147 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.147 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.147 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.147 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.147 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:j/bmImaUFIGMWqpWdIw5Pbtgb/I= 2024-04-12T04:56:01.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j/bmImaUFIGMWqpWdIw5Pbtgb/I=', 'Content-Length': '0'} 2024-04-12T04:56:01.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aee07690e991397b-006618b877-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj111000falseobj112024-04-12T04:28:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj11 2024-04-12T04:56:01.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj11 2024-04-12T04:56:01.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj11 2024-04-12T04:56:01.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.148 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.148 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.148 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.148 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.148 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.149 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:j/bmImaUFIGMWqpWdIw5Pbtgb/I= 2024-04-12T04:56:01.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j/bmImaUFIGMWqpWdIw5Pbtgb/I=', 'Content-Length': '0'} 2024-04-12T04:56:01.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf5320157775abdd-006618b877-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj111000falseobj112024-04-12T04:28:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11 2024-04-12T04:56:01.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11 2024-04-12T04:56:01.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj11 2024-04-12T04:56:01.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.150 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.150 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.150 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.150 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.150 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj11 2024-04-12T04:56:01.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.150 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:lmsIehaeuhHx8AcBEkpZDxDgOw0= 2024-04-12T04:56:01.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lmsIehaeuhHx8AcBEkpZDxDgOw0=', 'Content-Length': '0'} 2024-04-12T04:56:01.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f783d85a03d66809-006618b877-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11 2024-04-12T04:56:01.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11 2024-04-12T04:56:01.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj11 2024-04-12T04:56:01.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.151 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.151 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.151 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.151 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.151 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj11 2024-04-12T04:56:01.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.152 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:lmsIehaeuhHx8AcBEkpZDxDgOw0= 2024-04-12T04:56:01.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lmsIehaeuhHx8AcBEkpZDxDgOw0=', 'Content-Length': '0'} 2024-04-12T04:56:01.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007edfaca122005591-006618b877-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.152 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2024-04-12T04:56:01.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11 2024-04-12T04:56:01.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11 2024-04-12T04:56:01.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj11 2024-04-12T04:56:01.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.153 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.153 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.153 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.153 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.153 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj11 2024-04-12T04:56:01.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.153 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:HpKSrRafOHMh9/dME30iHQ+KdC8= 2024-04-12T04:56:01.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HpKSrRafOHMh9/dME30iHQ+KdC8=', 'Content-Length': '0'} 2024-04-12T04:56:01.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0a1838e35047369-006618b877-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj11 2024-04-12T04:56:01.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj11 2024-04-12T04:56:01.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj11 2024-04-12T04:56:01.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.154 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.154 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.154 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.154 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.154 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj11 2024-04-12T04:56:01.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.154 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:HpKSrRafOHMh9/dME30iHQ+KdC8= 2024-04-12T04:56:01.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HpKSrRafOHMh9/dME30iHQ+KdC8=', 'Content-Length': '0'} 2024-04-12T04:56:01.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001afd1ee753fbc2c-006618b877-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj11 2024-04-12T04:56:01.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj11 2024-04-12T04:56:01.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj11 2024-04-12T04:56:01.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.155 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.156 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.156 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.156 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.156 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.156 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:j/bmImaUFIGMWqpWdIw5Pbtgb/I= 2024-04-12T04:56:01.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j/bmImaUFIGMWqpWdIw5Pbtgb/I=', 'Content-Length': '0'} 2024-04-12T04:56:01.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000312694bc734429b3-006618b877-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj111000falseobj112024-04-12T04:28:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj11 2024-04-12T04:56:01.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj11 2024-04-12T04:56:01.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj11 2024-04-12T04:56:01.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.157 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.157 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.157 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.157 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.157 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.157 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:j/bmImaUFIGMWqpWdIw5Pbtgb/I= 2024-04-12T04:56:01.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:j/bmImaUFIGMWqpWdIw5Pbtgb/I=', 'Content-Length': '0'} 2024-04-12T04:56:01.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017e5991ce5b8738b-006618b877-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:39 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj111000falseobj112024-04-12T04:28:14.406Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.158 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-12T04:56:01.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22 2024-04-12T04:56:01.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22 2024-04-12T04:56:01.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.159 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.159 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.159 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.159 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.159 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.159 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ZADtSajCSKrGf3YLLD9nlqhh8VI= 2024-04-12T04:56:01.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZADtSajCSKrGf3YLLD9nlqhh8VI=', 'Content-Length': '0'} 2024-04-12T04:56:01.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090a964117f2e417c-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22 2024-04-12T04:56:01.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22 2024-04-12T04:56:01.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.160 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.160 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.160 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.160 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:39 GMT 2024-04-12T04:56:01.160 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.160 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ZADtSajCSKrGf3YLLD9nlqhh8VI= 2024-04-12T04:56:01.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:39 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZADtSajCSKrGf3YLLD9nlqhh8VI=', 'Content-Length': '0'} 2024-04-12T04:56:01.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6ecee70de842c9c-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22 2024-04-12T04:56:01.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22 2024-04-12T04:56:01.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.162 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.162 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.162 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.162 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.162 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.162 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:0lwK25D/DYWVVUqM+wD88WNpcvI= 2024-04-12T04:56:01.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0lwK25D/DYWVVUqM+wD88WNpcvI=', 'Content-Length': '0'} 2024-04-12T04:56:01.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c8d62acba5e82f3-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22 2024-04-12T04:56:01.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22 2024-04-12T04:56:01.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.163 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.163 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.163 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.163 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.163 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.163 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:0lwK25D/DYWVVUqM+wD88WNpcvI= 2024-04-12T04:56:01.163 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0lwK25D/DYWVVUqM+wD88WNpcvI=', 'Content-Length': '0'} 2024-04-12T04:56:01.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fb138a9f93a135b-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.164 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-12T04:56:01.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22 2024-04-12T04:56:01.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22 2024-04-12T04:56:01.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.164 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.164 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.165 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.165 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.165 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.165 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.165 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:bvi0FEn8D7DcD3eD+Kec2BredTg= 2024-04-12T04:56:01.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bvi0FEn8D7DcD3eD+Kec2BredTg=', 'Content-Length': '0'} 2024-04-12T04:56:01.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc82e588f94e4327-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22 2024-04-12T04:56:01.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22 2024-04-12T04:56:01.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.165 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.166 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.166 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.166 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.166 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.166 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj22?versionId=null 2024-04-12T04:56:01.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.166 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:bvi0FEn8D7DcD3eD+Kec2BredTg= 2024-04-12T04:56:01.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:bvi0FEn8D7DcD3eD+Kec2BredTg=', 'Content-Length': '0'} 2024-04-12T04:56:01.166 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000668b817a8796ce56-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj22 2024-04-12T04:56:01.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj22 2024-04-12T04:56:01.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj22 2024-04-12T04:56:01.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.167 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.167 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.167 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.168 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.168 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.168 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.168 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:56:01.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:56:01.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e33feb8c660675b3-006618b878-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj221000falseobj222024-04-12T04:28:14.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj22 2024-04-12T04:56:01.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj22 2024-04-12T04:56:01.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj22 2024-04-12T04:56:01.168 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.169 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.169 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.169 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.169 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.169 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.169 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:56:01.169 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:56:01.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e86016a6578566e-006618b878-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj221000falseobj222024-04-12T04:28:14.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22 2024-04-12T04:56:01.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22 2024-04-12T04:56:01.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj22 2024-04-12T04:56:01.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.170 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.171 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.171 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.171 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.171 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj22 2024-04-12T04:56:01.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.171 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:KiLEcrJ3OuTTA/5IMHY7sdYkTuo= 2024-04-12T04:56:01.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KiLEcrJ3OuTTA/5IMHY7sdYkTuo=', 'Content-Length': '0'} 2024-04-12T04:56:01.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021080a13297b47ab-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22 2024-04-12T04:56:01.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22 2024-04-12T04:56:01.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj22 2024-04-12T04:56:01.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.172 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.172 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.172 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.172 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.172 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj22 2024-04-12T04:56:01.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.172 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:KiLEcrJ3OuTTA/5IMHY7sdYkTuo= 2024-04-12T04:56:01.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:KiLEcrJ3OuTTA/5IMHY7sdYkTuo=', 'Content-Length': '0'} 2024-04-12T04:56:01.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a3f623965982a78-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.172 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2024-04-12T04:56:01.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22 2024-04-12T04:56:01.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22 2024-04-12T04:56:01.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj22 2024-04-12T04:56:01.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.173 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.173 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.173 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.173 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.173 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj22 2024-04-12T04:56:01.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.174 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:asOF0smMwHECrbBoKHsVrcZwKlY= 2024-04-12T04:56:01.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:asOF0smMwHECrbBoKHsVrcZwKlY=', 'Content-Length': '0'} 2024-04-12T04:56:01.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6d237d5cb591c84-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj22 2024-04-12T04:56:01.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj22 2024-04-12T04:56:01.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj22 2024-04-12T04:56:01.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.174 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.175 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.175 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.175 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.175 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj22 2024-04-12T04:56:01.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.175 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:asOF0smMwHECrbBoKHsVrcZwKlY= 2024-04-12T04:56:01.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:asOF0smMwHECrbBoKHsVrcZwKlY=', 'Content-Length': '0'} 2024-04-12T04:56:01.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b422afbed1f8502c-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj22 2024-04-12T04:56:01.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj22 2024-04-12T04:56:01.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj22 2024-04-12T04:56:01.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.176 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.176 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.176 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.176 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.176 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.176 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:56:01.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:56:01.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e25c9b0bd21f6e25-006618b878-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj221000falseobj222024-04-12T04:28:14.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj22 2024-04-12T04:56:01.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj22 2024-04-12T04:56:01.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj22 2024-04-12T04:56:01.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.178 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.178 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.178 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.178 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.178 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.178 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:56:01.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:56:01.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f0846d0b4a3da15-006618b878-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj221000falseobj222024-04-12T04:28:14.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.178 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-12T04:56:01.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33 2024-04-12T04:56:01.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33 2024-04-12T04:56:01.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.179 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.179 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.179 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.179 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.179 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.180 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:10mJOSet6hYVuVJZyhyRsszZJnU= 2024-04-12T04:56:01.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:10mJOSet6hYVuVJZyhyRsszZJnU=', 'Content-Length': '0'} 2024-04-12T04:56:01.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d07506caa450fc58-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33 2024-04-12T04:56:01.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33 2024-04-12T04:56:01.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.181 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.181 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.181 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.181 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.181 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.181 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:10mJOSet6hYVuVJZyhyRsszZJnU= 2024-04-12T04:56:01.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:10mJOSet6hYVuVJZyhyRsszZJnU=', 'Content-Length': '0'} 2024-04-12T04:56:01.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0bc25221d66886c-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33 2024-04-12T04:56:01.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33 2024-04-12T04:56:01.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.182 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.182 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.182 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.182 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.182 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.182 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:l6nnomiSPRNdVMqaPZUP8zRYxyM= 2024-04-12T04:56:01.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l6nnomiSPRNdVMqaPZUP8zRYxyM=', 'Content-Length': '0'} 2024-04-12T04:56:01.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b88f456ea9f4a996-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33 2024-04-12T04:56:01.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33 2024-04-12T04:56:01.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.183 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.184 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.184 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.184 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.184 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.184 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:l6nnomiSPRNdVMqaPZUP8zRYxyM= 2024-04-12T04:56:01.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:l6nnomiSPRNdVMqaPZUP8zRYxyM=', 'Content-Length': '0'} 2024-04-12T04:56:01.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a87855ec46df063-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.184 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-12T04:56:01.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33 2024-04-12T04:56:01.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33 2024-04-12T04:56:01.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.185 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.185 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.185 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.185 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.185 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.185 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:10mJOSet6hYVuVJZyhyRsszZJnU= 2024-04-12T04:56:01.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:10mJOSet6hYVuVJZyhyRsszZJnU=', 'Content-Length': '0'} 2024-04-12T04:56:01.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000464ce7672ca5e77a-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33 2024-04-12T04:56:01.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33 2024-04-12T04:56:01.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.187 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.187 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.187 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.187 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.187 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj33?versionId=null 2024-04-12T04:56:01.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.187 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:10mJOSet6hYVuVJZyhyRsszZJnU= 2024-04-12T04:56:01.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:10mJOSet6hYVuVJZyhyRsszZJnU=', 'Content-Length': '0'} 2024-04-12T04:56:01.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007dc3f94323356142-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj33 2024-04-12T04:56:01.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj33 2024-04-12T04:56:01.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj33 2024-04-12T04:56:01.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.188 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.188 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.188 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.189 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.189 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.189 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:56:01.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:56:01.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072a11a1e92f0e623-006618b878-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj331000falseobj332024-04-12T04:28:14.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj33 2024-04-12T04:56:01.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj33 2024-04-12T04:56:01.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj33 2024-04-12T04:56:01.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.190 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.190 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.190 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.190 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.190 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.191 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:56:01.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:56:01.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012369c9db5ee325b-006618b878-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj331000falseobj332024-04-12T04:28:14.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33 2024-04-12T04:56:01.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33 2024-04-12T04:56:01.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj33 2024-04-12T04:56:01.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.192 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.192 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.192 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.192 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.192 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj33 2024-04-12T04:56:01.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.192 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:0PfHDXFOD02WCdsHrCwtodfbscM= 2024-04-12T04:56:01.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0PfHDXFOD02WCdsHrCwtodfbscM=', 'Content-Length': '0'} 2024-04-12T04:56:01.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b844bda67108399-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33 2024-04-12T04:56:01.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33 2024-04-12T04:56:01.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj33 2024-04-12T04:56:01.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.193 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.193 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.193 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.194 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.194 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj33 2024-04-12T04:56:01.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.194 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:0PfHDXFOD02WCdsHrCwtodfbscM= 2024-04-12T04:56:01.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0PfHDXFOD02WCdsHrCwtodfbscM=', 'Content-Length': '0'} 2024-04-12T04:56:01.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008050243219b25b9c-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.194 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2024-04-12T04:56:01.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33 2024-04-12T04:56:01.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33 2024-04-12T04:56:01.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj33 2024-04-12T04:56:01.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.195 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.195 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.195 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.195 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.195 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj33 2024-04-12T04:56:01.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.195 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:WJMQrV9PlNsdwAx4+sLm67kaGto= 2024-04-12T04:56:01.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WJMQrV9PlNsdwAx4+sLm67kaGto=', 'Content-Length': '0'} 2024-04-12T04:56:01.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042b58155a9a76e3c-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj33 2024-04-12T04:56:01.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj33 2024-04-12T04:56:01.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj33 2024-04-12T04:56:01.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.196 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.196 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.196 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.197 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.197 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj33 2024-04-12T04:56:01.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.197 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:WJMQrV9PlNsdwAx4+sLm67kaGto= 2024-04-12T04:56:01.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:WJMQrV9PlNsdwAx4+sLm67kaGto=', 'Content-Length': '0'} 2024-04-12T04:56:01.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012ae135d7a3ac656-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj33 2024-04-12T04:56:01.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj33 2024-04-12T04:56:01.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj33 2024-04-12T04:56:01.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.198 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.198 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.198 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.198 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.198 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.198 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:56:01.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:56:01.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000383b63bca535489c-006618b878-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj331000falseobj332024-04-12T04:28:14.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj33 2024-04-12T04:56:01.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj33 2024-04-12T04:56:01.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj33 2024-04-12T04:56:01.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.200 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.200 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.200 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.200 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.200 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.200 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:56:01.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:56:01.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b72d08257e32546b-006618b878-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj331000falseobj332024-04-12T04:28:14.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.200 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-12T04:56:01.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44 2024-04-12T04:56:01.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44 2024-04-12T04:56:01.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.201 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.201 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.201 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.201 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.201 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.202 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:El/Zgf250YJcHiavzKELP5MnQpE= 2024-04-12T04:56:01.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:El/Zgf250YJcHiavzKELP5MnQpE=', 'Content-Length': '0'} 2024-04-12T04:56:01.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d09e929b72df59ed-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44 2024-04-12T04:56:01.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44 2024-04-12T04:56:01.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.203 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.203 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.203 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.203 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.203 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.203 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:El/Zgf250YJcHiavzKELP5MnQpE= 2024-04-12T04:56:01.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:El/Zgf250YJcHiavzKELP5MnQpE=', 'Content-Length': '0'} 2024-04-12T04:56:01.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094ec1217d981e73c-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44 2024-04-12T04:56:01.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44 2024-04-12T04:56:01.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.204 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.204 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.204 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.205 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.205 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.205 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:2vWWfUtTjrM+2JFdYvxRJWmJZH8= 2024-04-12T04:56:01.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2vWWfUtTjrM+2JFdYvxRJWmJZH8=', 'Content-Length': '0'} 2024-04-12T04:56:01.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073da06ea8e91be5c-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44 2024-04-12T04:56:01.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44 2024-04-12T04:56:01.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.206 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.206 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.206 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.206 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.206 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.206 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:2vWWfUtTjrM+2JFdYvxRJWmJZH8= 2024-04-12T04:56:01.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2vWWfUtTjrM+2JFdYvxRJWmJZH8=', 'Content-Length': '0'} 2024-04-12T04:56:01.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e37e8ec985c6538-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.206 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-12T04:56:01.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44 2024-04-12T04:56:01.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44 2024-04-12T04:56:01.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.207 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.207 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.207 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.208 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.208 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.208 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:El/Zgf250YJcHiavzKELP5MnQpE= 2024-04-12T04:56:01.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:El/Zgf250YJcHiavzKELP5MnQpE=', 'Content-Length': '0'} 2024-04-12T04:56:01.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d026a47571571076-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44 2024-04-12T04:56:01.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44 2024-04-12T04:56:01.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.209 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.209 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.209 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.209 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.209 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj44?versionId=null 2024-04-12T04:56:01.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.209 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:El/Zgf250YJcHiavzKELP5MnQpE= 2024-04-12T04:56:01.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:El/Zgf250YJcHiavzKELP5MnQpE=', 'Content-Length': '0'} 2024-04-12T04:56:01.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e70100cf49b7e7eb-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj44 2024-04-12T04:56:01.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj44 2024-04-12T04:56:01.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj44 2024-04-12T04:56:01.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.210 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.210 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.210 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.210 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.211 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.211 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:56:01.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:56:01.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e79b2e295b83f87-006618b878-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj441000falseobj442024-04-12T04:28:14.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj44 2024-04-12T04:56:01.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj44 2024-04-12T04:56:01.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj44 2024-04-12T04:56:01.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.212 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.212 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.212 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.212 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.212 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.212 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:56:01.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:56:01.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061eacbf2f43db846-006618b878-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj441000falseobj442024-04-12T04:28:14.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44 2024-04-12T04:56:01.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44 2024-04-12T04:56:01.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj44 2024-04-12T04:56:01.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.213 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.213 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.213 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.213 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.214 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj44 2024-04-12T04:56:01.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.214 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:xggdqUHo2tep8poDsEr85ffZ+Ts= 2024-04-12T04:56:01.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xggdqUHo2tep8poDsEr85ffZ+Ts=', 'Content-Length': '0'} 2024-04-12T04:56:01.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005059b1b51fded9a-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44 2024-04-12T04:56:01.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44 2024-04-12T04:56:01.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj44 2024-04-12T04:56:01.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.215 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.215 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.215 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.215 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.215 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj44 2024-04-12T04:56:01.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.215 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:xggdqUHo2tep8poDsEr85ffZ+Ts= 2024-04-12T04:56:01.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:xggdqUHo2tep8poDsEr85ffZ+Ts=', 'Content-Length': '0'} 2024-04-12T04:56:01.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e0c152c9e54e8f3-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.215 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2024-04-12T04:56:01.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44 2024-04-12T04:56:01.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44 2024-04-12T04:56:01.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj44 2024-04-12T04:56:01.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.216 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.216 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.216 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.216 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.216 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj44 2024-04-12T04:56:01.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.217 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:2ky2KrW9h2Y1t/KNoBfthuPd3P4= 2024-04-12T04:56:01.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2ky2KrW9h2Y1t/KNoBfthuPd3P4=', 'Content-Length': '0'} 2024-04-12T04:56:01.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fc0124e76e4df00-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj44 2024-04-12T04:56:01.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj44 2024-04-12T04:56:01.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj44 2024-04-12T04:56:01.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.218 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.218 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.218 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.218 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.218 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj44 2024-04-12T04:56:01.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.218 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:2ky2KrW9h2Y1t/KNoBfthuPd3P4= 2024-04-12T04:56:01.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:2ky2KrW9h2Y1t/KNoBfthuPd3P4=', 'Content-Length': '0'} 2024-04-12T04:56:01.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bd9cb231fe957fc-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj44 2024-04-12T04:56:01.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj44 2024-04-12T04:56:01.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj44 2024-04-12T04:56:01.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.219 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.219 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.219 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.220 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.220 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.220 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:56:01.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:56:01.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047ab5ce268121115-006618b878-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj441000falseobj442024-04-12T04:28:14.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj44 2024-04-12T04:56:01.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj44 2024-04-12T04:56:01.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj44 2024-04-12T04:56:01.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.221 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.221 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.221 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.221 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.221 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.221 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:56:01.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:56:01.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f0e133a4de931f9-006618b878-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj441000falseobj442024-04-12T04:28:14.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.222 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2024-04-12T04:56:01.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55 2024-04-12T04:56:01.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55 2024-04-12T04:56:01.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.223 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.223 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.223 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.223 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.223 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.223 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:RwTUkth6PHgaf489t+GZ9O0gB0g= 2024-04-12T04:56:01.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RwTUkth6PHgaf489t+GZ9O0gB0g=', 'Content-Length': '0'} 2024-04-12T04:56:01.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080b45581fbe44b7d-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55 2024-04-12T04:56:01.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55 2024-04-12T04:56:01.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.224 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.224 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.224 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.224 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.224 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.224 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:RwTUkth6PHgaf489t+GZ9O0gB0g= 2024-04-12T04:56:01.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RwTUkth6PHgaf489t+GZ9O0gB0g=', 'Content-Length': '0'} 2024-04-12T04:56:01.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd0cee22e3b1ac6-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55 2024-04-12T04:56:01.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55 2024-04-12T04:56:01.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.226 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.226 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.226 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.226 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.226 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.226 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:HSSwnQtSwvwB2uDRir+6DZhXScM= 2024-04-12T04:56:01.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HSSwnQtSwvwB2uDRir+6DZhXScM=', 'Content-Length': '0'} 2024-04-12T04:56:01.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a16f15b68254c03-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55 2024-04-12T04:56:01.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55 2024-04-12T04:56:01.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.227 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.227 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.227 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.227 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.227 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.227 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:HSSwnQtSwvwB2uDRir+6DZhXScM= 2024-04-12T04:56:01.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:HSSwnQtSwvwB2uDRir+6DZhXScM=', 'Content-Length': '0'} 2024-04-12T04:56:01.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acaaff3aea0bb65b-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.228 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2024-04-12T04:56:01.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55 2024-04-12T04:56:01.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55 2024-04-12T04:56:01.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.229 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.229 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.229 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.229 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.229 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.229 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:RwTUkth6PHgaf489t+GZ9O0gB0g= 2024-04-12T04:56:01.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RwTUkth6PHgaf489t+GZ9O0gB0g=', 'Content-Length': '0'} 2024-04-12T04:56:01.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e20f215e9676bad4-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55 2024-04-12T04:56:01.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55 2024-04-12T04:56:01.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.230 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.230 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.230 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.230 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.230 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj55?versionId=null 2024-04-12T04:56:01.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.230 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:RwTUkth6PHgaf489t+GZ9O0gB0g= 2024-04-12T04:56:01.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:RwTUkth6PHgaf489t+GZ9O0gB0g=', 'Content-Length': '0'} 2024-04-12T04:56:01.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e20c6e94b431ae6d-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj55 2024-04-12T04:56:01.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj55 2024-04-12T04:56:01.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj55 2024-04-12T04:56:01.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.231 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.231 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.232 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.232 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.232 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.232 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:56:01.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:56:01.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000493da45eb6ef3089-006618b878-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj551000falseobj552024-04-12T04:28:14.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj55 2024-04-12T04:56:01.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj55 2024-04-12T04:56:01.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj55 2024-04-12T04:56:01.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.233 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.233 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.233 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.233 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.233 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.233 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:56:01.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:56:01.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdc0008c39cb3000-006618b878-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj551000falseobj552024-04-12T04:28:14.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55 2024-04-12T04:56:01.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55 2024-04-12T04:56:01.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj55 2024-04-12T04:56:01.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.235 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.235 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.235 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.235 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.235 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj55 2024-04-12T04:56:01.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.235 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:D/HThyBtYsP/rYPDtmgoXbBSUL8= 2024-04-12T04:56:01.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D/HThyBtYsP/rYPDtmgoXbBSUL8=', 'Content-Length': '0'} 2024-04-12T04:56:01.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f500700e0b7daa5-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55 2024-04-12T04:56:01.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55 2024-04-12T04:56:01.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj55 2024-04-12T04:56:01.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.236 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.236 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.236 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.236 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.236 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj55 2024-04-12T04:56:01.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.237 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:D/HThyBtYsP/rYPDtmgoXbBSUL8= 2024-04-12T04:56:01.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:D/HThyBtYsP/rYPDtmgoXbBSUL8=', 'Content-Length': '0'} 2024-04-12T04:56:01.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001524a1dfdb709ad-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.237 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2024-04-12T04:56:01.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55 2024-04-12T04:56:01.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55 2024-04-12T04:56:01.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj55 2024-04-12T04:56:01.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.238 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.238 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.238 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.238 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.238 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj55 2024-04-12T04:56:01.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.238 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:oqQfCMCf0lXsCdx8btkcluiyk0g= 2024-04-12T04:56:01.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oqQfCMCf0lXsCdx8btkcluiyk0g=', 'Content-Length': '0'} 2024-04-12T04:56:01.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-66:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.5851.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055bd55692b710b8a-006618b878-216b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj55 2024-04-12T04:56:01.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj55 2024-04-12T04:56:01.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj55 2024-04-12T04:56:01.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.239 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.239 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.239 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.239 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.239 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj55 2024-04-12T04:56:01.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.240 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:oqQfCMCf0lXsCdx8btkcluiyk0g= 2024-04-12T04:56:01.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oqQfCMCf0lXsCdx8btkcluiyk0g=', 'Content-Length': '0'} 2024-04-12T04:56:01.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:28:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089cd108dc4c18a4a-006618b878-16d5-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj55 2024-04-12T04:56:01.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj55 2024-04-12T04:56:01.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj55 2024-04-12T04:56:01.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.241 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.241 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.241 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.241 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.241 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.241 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:56:01.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:56:01.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9d758464a6f183a-006618b878-216b-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj551000falseobj552024-04-12T04:28:14.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/?prefix=obj55 2024-04-12T04:56:01.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/?prefix=obj55 2024-04-12T04:56:01.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/?prefix=obj55 2024-04-12T04:56:01.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.242 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.242 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.243 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.243 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:40 GMT 2024-04-12T04:56:01.243 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.243 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA= 2024-04-12T04:56:01.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:40 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6cFVTwAEFyOgpNqogU720kiW8xA=', 'Content-Length': '0'} 2024-04-12T04:56:01.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e371b326eda1b55-006618b878-16d5-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:28:40 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'dkyges-66obj551000falseobj552024-04-12T04:28:14.488Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2024-04-12T04:56:01.243 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=dkyges-66 zones={a2, a1} 2024-04-12T04:56:01.243 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2024-04-12T04:56:01.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2024-04-12T04:56:01.243 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T04:28:40.533+0000 7f417edaf700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 108871) UID: 0 2024-04-12T04:56:01.243 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T04:28:40.533+0000 7f4183f847c0 -1 shutting down 2024-04-12T04:56:01.244 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T04:56:01.244 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2024-04-12T04:56:01.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.244 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-66 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: tenant: 2024-04-12T04:56:01.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: bucket name: dkyges-66 2024-04-12T04:56:01.244 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: total entries: 5 2024-04-12T04:56:01.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.245 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.245 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.245 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.245 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:46 GMT 2024-04-12T04:56:01.246 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.246 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ= 2024-04-12T04:56:01.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ=', 'Content-Length': '0'} 2024-04-12T04:56:01.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f5d037c09add470-006618b87e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj17 2024-04-12T04:56:01.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj17 2024-04-12T04:56:01.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj17 2024-04-12T04:56:01.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.247 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.247 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.247 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.247 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:46 GMT 2024-04-12T04:56:01.247 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj17 2024-04-12T04:56:01.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.247 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:SKtAJ5ZnZihAiQFKteBHsn0nu/M= 2024-04-12T04:56:01.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:SKtAJ5ZnZihAiQFKteBHsn0nu/M='} 2024-04-12T04:56:01.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e846af5e502f8b8f-006618b87e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.248 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.248 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.248 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.248 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:46 GMT 2024-04-12T04:56:01.248 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.249 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ= 2024-04-12T04:56:01.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ=', 'Content-Length': '0'} 2024-04-12T04:56:01.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004fd685715a4cffcc-006618b87e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj34 2024-04-12T04:56:01.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj34 2024-04-12T04:56:01.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj34 2024-04-12T04:56:01.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.250 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.250 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.250 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.250 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:46 GMT 2024-04-12T04:56:01.250 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj34 2024-04-12T04:56:01.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.250 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:0/yjrjp6b2LO5EeHXb7PAButPjU= 2024-04-12T04:56:01.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0/yjrjp6b2LO5EeHXb7PAButPjU='} 2024-04-12T04:56:01.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000081b3e1ec8a9da472-006618b87e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.251 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.251 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.251 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.251 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:46 GMT 2024-04-12T04:56:01.251 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.251 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ= 2024-04-12T04:56:01.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ=', 'Content-Length': '0'} 2024-04-12T04:56:01.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6afccafe4925584-006618b87e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj51 2024-04-12T04:56:01.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj51 2024-04-12T04:56:01.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj51 2024-04-12T04:56:01.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.252 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.252 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.253 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.253 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:46 GMT 2024-04-12T04:56:01.253 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj51 2024-04-12T04:56:01.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.253 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:L7/iXnba+btsa5mLQSLyUcqW3do= 2024-04-12T04:56:01.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L7/iXnba+btsa5mLQSLyUcqW3do='} 2024-04-12T04:56:01.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb2e51576f06aab1-006618b87e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.254 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.254 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.254 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.254 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:46 GMT 2024-04-12T04:56:01.254 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.254 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ= 2024-04-12T04:56:01.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ=', 'Content-Length': '0'} 2024-04-12T04:56:01.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b35cadd645c1c8ac-006618b87e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj68 2024-04-12T04:56:01.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj68 2024-04-12T04:56:01.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj68 2024-04-12T04:56:01.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.255 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.255 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.255 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.255 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:46 GMT 2024-04-12T04:56:01.255 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj68 2024-04-12T04:56:01.255 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.255 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Z7Wm/zgOpSJiQEdYe9ZACM8gEUU= 2024-04-12T04:56:01.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Z7Wm/zgOpSJiQEdYe9ZACM8gEUU='} 2024-04-12T04:56:01.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003895c5917f654de5-006618b87e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.256 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.256 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.256 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.257 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.257 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:46 GMT 2024-04-12T04:56:01.257 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.257 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ= 2024-04-12T04:56:01.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:+yUMcAwMjysNJdBb4r65QnmeoKQ=', 'Content-Length': '0'} 2024-04-12T04:56:01.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000022e33e125e73ce48-006618b87e-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj85 2024-04-12T04:56:01.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj85 2024-04-12T04:56:01.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj85 2024-04-12T04:56:01.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.258 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.258 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.258 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.258 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:46 GMT 2024-04-12T04:56:01.258 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj85 2024-04-12T04:56:01.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.258 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:obrIAv/z13GUuF+ekovHF9c+jEc= 2024-04-12T04:56:01.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:46 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:obrIAv/z13GUuF+ekovHF9c+jEc='} 2024-04-12T04:56:01.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d8849f10f3bf1c7e-006618b87e-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:46 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.259 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-66 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: tenant: 2024-04-12T04:56:01.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: bucket name: dkyges-66 2024-04-12T04:56:01.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: total entries: 10 2024-04-12T04:56:01.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.260 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.260 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.260 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.260 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.260 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.261 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:56:01.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:56:01.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053a2390d30178123-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj19 2024-04-12T04:56:01.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj19 2024-04-12T04:56:01.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj19 2024-04-12T04:56:01.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.261 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.261 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.262 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.262 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.262 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj19 2024-04-12T04:56:01.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.262 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/FK+XKDWoS1AKNZkAwBC4ttdcmk= 2024-04-12T04:56:01.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/FK+XKDWoS1AKNZkAwBC4ttdcmk='} 2024-04-12T04:56:01.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000308b12dfb2dc8831-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.263 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.263 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.263 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.263 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.263 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.263 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:56:01.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:56:01.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1cd1311360cefa9-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj38 2024-04-12T04:56:01.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj38 2024-04-12T04:56:01.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj38 2024-04-12T04:56:01.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.264 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.264 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.264 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.264 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.264 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj38 2024-04-12T04:56:01.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.265 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:kW+bD6r9YiGAEXind5oWxYxRgOI= 2024-04-12T04:56:01.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:kW+bD6r9YiGAEXind5oWxYxRgOI='} 2024-04-12T04:56:01.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c44d06be28ce875-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.266 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.266 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.266 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.266 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.266 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.266 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:56:01.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:56:01.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bbe4941d47e0b6f2-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj57 2024-04-12T04:56:01.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj57 2024-04-12T04:56:01.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj57 2024-04-12T04:56:01.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.267 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.267 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.267 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.267 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.267 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj57 2024-04-12T04:56:01.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.267 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:OlzSMjML5CEUaq98j7yYLLdLSHw= 2024-04-12T04:56:01.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:OlzSMjML5CEUaq98j7yYLLdLSHw='} 2024-04-12T04:56:01.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009965994354ce40e6-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.268 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.268 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.268 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.268 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.269 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.269 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:56:01.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:56:01.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d82b954aca01f6e7-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj76 2024-04-12T04:56:01.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj76 2024-04-12T04:56:01.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj76 2024-04-12T04:56:01.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.270 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.270 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.270 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.270 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.270 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj76 2024-04-12T04:56:01.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.270 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ouYDshmy3mzRvw+oq+cjKD59mmQ= 2024-04-12T04:56:01.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ouYDshmy3mzRvw+oq+cjKD59mmQ='} 2024-04-12T04:56:01.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de56a3651aaba064-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.271 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.271 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.271 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.271 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.271 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.272 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:56:01.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:56:01.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000edbe4f0d1b7d18e-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj95 2024-04-12T04:56:01.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj95 2024-04-12T04:56:01.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj95 2024-04-12T04:56:01.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.273 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.273 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.273 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.273 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.273 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj95 2024-04-12T04:56:01.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.273 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:PSOHPD77X8QMoFWjp4km3bY7c7U= 2024-04-12T04:56:01.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:PSOHPD77X8QMoFWjp4km3bY7c7U='} 2024-04-12T04:56:01.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0662e6e474eb992-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.273 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-66 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: tenant: 2024-04-12T04:56:01.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: bucket name: dkyges-66 2024-04-12T04:56:01.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: total entries: 15 2024-04-12T04:56:01.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.275 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.275 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.275 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.275 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.275 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.275 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:56:01.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:56:01.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000403a76b37ba53137-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj23 2024-04-12T04:56:01.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj23 2024-04-12T04:56:01.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj23 2024-04-12T04:56:01.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.276 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.276 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.276 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.276 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.276 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj23 2024-04-12T04:56:01.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.276 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:L2mKbHbWALBkc4vo74Y3IO5rTT8= 2024-04-12T04:56:01.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:L2mKbHbWALBkc4vo74Y3IO5rTT8='} 2024-04-12T04:56:01.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e245fbcfb819b2b6-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.277 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.277 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.278 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.278 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.278 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.278 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:56:01.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:56:01.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000284232918303bf26-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj46 2024-04-12T04:56:01.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj46 2024-04-12T04:56:01.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj46 2024-04-12T04:56:01.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.279 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.279 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.279 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.279 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.279 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj46 2024-04-12T04:56:01.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.279 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:8A3XhaEOv2wy92LhUu268EUe1/I= 2024-04-12T04:56:01.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8A3XhaEOv2wy92LhUu268EUe1/I='} 2024-04-12T04:56:01.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a45087c134c1b685-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.280 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.281 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.281 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.281 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.281 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.281 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:56:01.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:56:01.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3d3c8f87b129ab0-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj69 2024-04-12T04:56:01.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj69 2024-04-12T04:56:01.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj69 2024-04-12T04:56:01.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.282 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.282 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.282 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.282 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.282 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj69 2024-04-12T04:56:01.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.283 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:nqz+KM1CF+9JASGgCA8HH/rzFjc= 2024-04-12T04:56:01.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:nqz+KM1CF+9JASGgCA8HH/rzFjc='} 2024-04-12T04:56:01.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e302247fb767a14-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.284 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.284 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.284 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.284 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.284 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.284 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:56:01.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:56:01.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cdb2b83269ee7994-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj92 2024-04-12T04:56:01.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj92 2024-04-12T04:56:01.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj92 2024-04-12T04:56:01.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.285 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.285 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.285 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.285 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.285 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj92 2024-04-12T04:56:01.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.285 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:3RIhuPFHWs9MLOLczQnJk6NUQhU= 2024-04-12T04:56:01.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:3RIhuPFHWs9MLOLczQnJk6NUQhU='} 2024-04-12T04:56:01.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082f2c07a12ac6320-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.286 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.286 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.287 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.287 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.287 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.287 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:56:01.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:56:01.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b78f7b697bc06cb-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj115 2024-04-12T04:56:01.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj115 2024-04-12T04:56:01.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj115 2024-04-12T04:56:01.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.288 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.288 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.288 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.288 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.288 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj115 2024-04-12T04:56:01.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.288 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:QVTYj5NramhJKAo6INSTWD8Mzck= 2024-04-12T04:56:01.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:QVTYj5NramhJKAo6INSTWD8Mzck='} 2024-04-12T04:56:01.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036c1b3bbf2f7e3ce-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.289 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.289 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-66 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: tenant: 2024-04-12T04:56:01.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: bucket name: dkyges-66 2024-04-12T04:56:01.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: total entries: 20 2024-04-12T04:56:01.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.290 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.290 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.290 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.290 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.290 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.291 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:56:01.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:56:01.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b74d6013148a47b3-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj29 2024-04-12T04:56:01.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj29 2024-04-12T04:56:01.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj29 2024-04-12T04:56:01.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.292 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.292 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.292 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.292 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.292 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj29 2024-04-12T04:56:01.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.292 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:8l+zBdZXfwL1ghoDxLGq+MuCAso= 2024-04-12T04:56:01.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:8l+zBdZXfwL1ghoDxLGq+MuCAso='} 2024-04-12T04:56:01.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000620b9002c20a9508-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.293 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.293 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.293 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.293 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.293 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.294 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:56:01.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:56:01.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000081a3ff616518c58e-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj58 2024-04-12T04:56:01.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj58 2024-04-12T04:56:01.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj58 2024-04-12T04:56:01.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.295 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.295 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.295 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.295 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.295 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj58 2024-04-12T04:56:01.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.295 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:MS6wYcuysTzUDTW1DtqZFcOBZBw= 2024-04-12T04:56:01.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:MS6wYcuysTzUDTW1DtqZFcOBZBw='} 2024-04-12T04:56:01.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000025b49b5ed1b6eec9-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.296 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.296 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.296 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.296 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.296 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.297 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:56:01.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:56:01.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b3f150687feb490-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj87 2024-04-12T04:56:01.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj87 2024-04-12T04:56:01.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj87 2024-04-12T04:56:01.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.298 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.298 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.298 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.298 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.298 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj87 2024-04-12T04:56:01.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.298 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:uQvG5M/I8HmenPalP2ngzF1y5Xc= 2024-04-12T04:56:01.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:uQvG5M/I8HmenPalP2ngzF1y5Xc='} 2024-04-12T04:56:01.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002910a387e27e8abb-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.299 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.299 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.299 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.300 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.300 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.300 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:56:01.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:56:01.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000835bd753f7be8c90-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj116 2024-04-12T04:56:01.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj116 2024-04-12T04:56:01.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj116 2024-04-12T04:56:01.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.301 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.301 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.301 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.301 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.301 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj116 2024-04-12T04:56:01.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.301 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:9KSXg9axMXezx4qc/FyjtyOkLG8= 2024-04-12T04:56:01.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:9KSXg9axMXezx4qc/FyjtyOkLG8='} 2024-04-12T04:56:01.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f27e99c8e14e19f9-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.302 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.302 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.302 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.302 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.303 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.303 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q= 2024-04-12T04:56:01.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:my81/NXL/8p00TkQl+FjvrqNG+Q=', 'Content-Length': '0'} 2024-04-12T04:56:01.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a1fec857b42d6325-006618b87f-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj145 2024-04-12T04:56:01.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj145 2024-04-12T04:56:01.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj145 2024-04-12T04:56:01.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.304 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.304 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.304 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.304 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:47 GMT 2024-04-12T04:56:01.304 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj145 2024-04-12T04:56:01.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.304 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:AbOQnfeBwbbtp7279XpArmPZbqE= 2024-04-12T04:56:01.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:47 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:AbOQnfeBwbbtp7279XpArmPZbqE='} 2024-04-12T04:56:01.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001842ba1a2885b5b2-006618b87f-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:47 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-66 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.304 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: tenant: 2024-04-12T04:56:01.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: bucket name: dkyges-66 2024-04-12T04:56:01.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: total entries: 25 2024-04-12T04:56:01.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.305 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.306 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.306 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.306 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.306 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.306 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:56:01.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:56:01.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036678942352b6fd3-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj31 2024-04-12T04:56:01.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj31 2024-04-12T04:56:01.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj31 2024-04-12T04:56:01.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.307 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.307 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.307 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.307 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.307 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj31 2024-04-12T04:56:01.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.307 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ipQsuEP/bWlb1sjdNeG2vFdrM0s= 2024-04-12T04:56:01.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ipQsuEP/bWlb1sjdNeG2vFdrM0s='} 2024-04-12T04:56:01.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008635abfa4286f978-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.309 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.309 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.309 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.309 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.309 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.309 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:56:01.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:56:01.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006ebae62fa5d2920d-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj62 2024-04-12T04:56:01.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj62 2024-04-12T04:56:01.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj62 2024-04-12T04:56:01.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.310 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.310 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.310 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.310 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.310 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj62 2024-04-12T04:56:01.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.311 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:I6v5vRxgeyhihmTSCVaRvAha0+M= 2024-04-12T04:56:01.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:I6v5vRxgeyhihmTSCVaRvAha0+M='} 2024-04-12T04:56:01.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000087b5717e9388e0cf-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.311 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.312 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.312 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.312 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.312 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.312 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:56:01.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:56:01.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003056335c9533e31d-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj93 2024-04-12T04:56:01.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj93 2024-04-12T04:56:01.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj93 2024-04-12T04:56:01.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.313 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.313 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.313 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.313 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.313 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj93 2024-04-12T04:56:01.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.313 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ZEw4VmcR+hhePl5mlaHqLISjFpc= 2024-04-12T04:56:01.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ZEw4VmcR+hhePl5mlaHqLISjFpc='} 2024-04-12T04:56:01.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c08ddf7a42615d43-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.314 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.314 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.314 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.314 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.315 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.315 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:56:01.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:56:01.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b6274a7ac29c96e-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj124 2024-04-12T04:56:01.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj124 2024-04-12T04:56:01.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj124 2024-04-12T04:56:01.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.316 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.316 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.316 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.316 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.316 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj124 2024-04-12T04:56:01.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.316 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:fb4pCPKP6XlrhLqoLYXC11gYFkI= 2024-04-12T04:56:01.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fb4pCPKP6XlrhLqoLYXC11gYFkI='} 2024-04-12T04:56:01.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db06e91f594df2c2-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.317 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.317 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.317 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.317 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.317 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.317 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:56:01.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:56:01.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db4c91fa12d8d358-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj155 2024-04-12T04:56:01.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj155 2024-04-12T04:56:01.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj155 2024-04-12T04:56:01.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.318 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.318 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.319 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.319 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.319 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj155 2024-04-12T04:56:01.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.319 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:lPAvbsITyOldjUHTWwyMJH5ZNIM= 2024-04-12T04:56:01.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:lPAvbsITyOldjUHTWwyMJH5ZNIM='} 2024-04-12T04:56:01.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e230586a1f3b86c9-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.319 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.319 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'dkyges-66', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket dkyges-66 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: tenant: 2024-04-12T04:56:01.319 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: bucket name: dkyges-66 2024-04-12T04:56:01.320 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: total entries: 30 2024-04-12T04:56:01.320 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2024-04-12T04:56:01.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.320 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.321 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.321 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.321 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.321 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.321 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:56:01.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:56:01.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b74d20a674036cd6-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj37 2024-04-12T04:56:01.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj37 2024-04-12T04:56:01.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj37 2024-04-12T04:56:01.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.322 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.322 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.322 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.322 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.322 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj37 2024-04-12T04:56:01.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.322 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:p9J7F5o4phmTxVuMJe74Vtcq0h0= 2024-04-12T04:56:01.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:p9J7F5o4phmTxVuMJe74Vtcq0h0='} 2024-04-12T04:56:01.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c048e232e0f0837-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.323 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.323 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.323 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.323 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.323 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.324 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:56:01.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:56:01.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000917e52ed32f5a049-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj74 2024-04-12T04:56:01.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj74 2024-04-12T04:56:01.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj74 2024-04-12T04:56:01.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.324 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.325 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.325 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.325 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.325 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj74 2024-04-12T04:56:01.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.325 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:0qYlQiatzitV9E6iwimEQi8v+AY= 2024-04-12T04:56:01.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:0qYlQiatzitV9E6iwimEQi8v+AY='} 2024-04-12T04:56:01.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073b21b8415923f12-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.326 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.326 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.326 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.326 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.326 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.326 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:56:01.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:56:01.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5ab893bcad32d79-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj111 2024-04-12T04:56:01.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj111 2024-04-12T04:56:01.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj111 2024-04-12T04:56:01.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.327 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.327 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.327 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.328 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.328 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj111 2024-04-12T04:56:01.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.328 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:q8wp7b4aNvLxHeAJiCYXTmjfl28= 2024-04-12T04:56:01.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:q8wp7b4aNvLxHeAJiCYXTmjfl28='} 2024-04-12T04:56:01.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000061ed2aaeab674ea1-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.329 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.329 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.329 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.329 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.329 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.329 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:56:01.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:56:01.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bad8b643e7ec4880-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj148 2024-04-12T04:56:01.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj148 2024-04-12T04:56:01.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj148 2024-04-12T04:56:01.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.330 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.330 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.330 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.330 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.330 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj148 2024-04-12T04:56:01.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.330 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:6NPl2BEKmtMClzU3ZU2NDXlHSPw= 2024-04-12T04:56:01.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:6NPl2BEKmtMClzU3ZU2NDXlHSPw='} 2024-04-12T04:56:01.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005aad8a2803ddcb2b-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/ 2024-04-12T04:56:01.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/ 2024-04-12T04:56:01.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/ 2024-04-12T04:56:01.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.331 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.331 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.332 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.332 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.332 INFO:tasks.rgw_multisite_tests:/dkyges-66/ 2024-04-12T04:56:01.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.332 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0= 2024-04-12T04:56:01.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:GBGYQZljLdiqkRuYH5yDj4HZSK0=', 'Content-Length': '0'} 2024-04-12T04:56:01.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f3ff2a9e1a9d6e7-006618b880-16d5-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-66/obj185 2024-04-12T04:56:01.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-66/obj185 2024-04-12T04:56:01.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-66/obj185 2024-04-12T04:56:01.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2024-04-12T04:56:01.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.333 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.333 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2024-04-12T04:56:01.333 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.333 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:28:48 GMT 2024-04-12T04:56:01.333 INFO:tasks.rgw_multisite_tests:/dkyges-66/obj185 2024-04-12T04:56:01.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.333 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:5B98DKq84jPGpJZKLj6QnAvC1io= 2024-04-12T04:56:01.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:28:48 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:5B98DKq84jPGpJZKLj6QnAvC1io='} 2024-04-12T04:56:01.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f067ca2c1a78bd72-006618b880-16d5-a1'), ('Date', 'Fri, 12 Apr 2024 04:28:48 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.333 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.333 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.333 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:48.795+0000 7f6db40e9780 5 RGW-SYNC:data: waiting on data sync status lease 2024-04-12T04:56:01.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:48.796+0000 7f6db40e9780 5 RGW-SYNC:data: acquired data sync status lease 2024-04-12T04:56:01.334 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2024-04-12T04:56:01.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-04-12T04:56:01.334 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2024-04-12T04:56:01.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:56:01.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-12T04:56:01.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2024-04-12T04:56:01.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi043.front.sepia.ceph.com port 8001: Connection refused 2024-04-12T04:56:01.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > curl http://smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.334 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-12T04:56:01.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: Dload Upload Total Spent Left Speed 2024-04-12T04:56:01.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-04-12T04:56:01.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: anonymous 2024-04-12T04:56:01.335 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=dkyges-66 2024-04-12T04:56:01.335 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:01.335 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'dkyges-66', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-04-12T04:56:01.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket dkyges-66 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-04-12T04:56:01.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:50.035+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:28:55.036+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:29:00.037+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:29:05.037+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.335 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:29:10.038+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:29:15.039+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:29:20.040+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:29:25.041+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:29:30.041+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:29:35.041+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:29:40.042+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:29:45.043+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:29:50.043+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:29:55.044+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:30:00.045+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:30:05.045+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:30:10.046+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:30:15.047+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.336 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:30:20.047+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:30:25.048+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:30:30.049+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:30:35.050+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.337 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi043.stdout: INFO: 2024-04-12T04:30:36.088+0000 7f1e272bd700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2024-04-12T04:56:01.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:30:40.051+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:30:45.051+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:30:50.051+0000 7f552acc1780 1 waiting to reach latest gen 6, on 0.. 2024-04-12T04:56:01.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:30:55.052+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:31:00.053+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:31:05.054+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:31:10.054+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:31:15.055+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:31:20.056+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:31:25.056+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.337 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:31:30.057+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:31:35.058+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:31:40.058+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:31:45.059+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:31:50.060+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:31:55.061+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:32:00.061+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:32:05.062+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:32:10.062+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:32:15.063+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:32:20.064+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:32:25.065+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:32:30.065+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:32:35.066+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.338 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:32:40.067+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:32:45.067+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:32:50.068+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:32:55.068+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:33:00.068+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:33:05.069+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:33:10.071+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:33:15.070+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:33:20.071+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:33:25.072+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:33:30.072+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:33:35.073+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.339 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:33:40.073+0000 7f552acc1780 1 waiting to reach latest gen 6, on 3.. 2024-04-12T04:56:01.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:33:45.074+0000 7f552acc1780 -1 bucket checkpoint timed out waiting to reach latest generation 6 2024-04-12T04:56:01.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:33:45.074+0000 7f552acc1780 0 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:56:01.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:33:45.074+0000 7f552acc1780 -1 bucket sync checkpoint failed: (110) Connection timed out 2024-04-12T04:56:01.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2024-04-12T04:56:01.340 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-12T04:56:01.340 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.340 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-04-12T04:56:01.340 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_all: 2024-04-12T04:56:01.340 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-12T04:56:01.341 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-04-12T04:56:01.341 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2024-04-12T04:56:01.341 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-04-12T04:56:01.341 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 2213, in test_sync_flow_symmetrical_zonegroup_all 2024-04-12T04:56:01.341 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objnames[1], content) 2024-04-12T04:56:01.341 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/../src/test/rgw/rgw_multi/tests.py", line 2074, in check_object_exists 2024-04-12T04:56:01.341 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2024-04-12T04:56:01.341 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2024-04-12T04:56:01.342 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2024-04-12T04:56:01.342 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.342 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:56:01.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: [ 2024-04-12T04:56:01.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:56:01.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:56:01.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:56:01.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T04:35:41.420306Z" 2024-04-12T04:56:01.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.344 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-12T04:56:01.344 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.344 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:03.206+0000 7f5a6709d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:03.212+0000 7f5a6709d780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:36:03Z" 2024-04-12T04:56:01.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.349 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.349 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.349 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.349 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.349 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.349 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.349 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.349 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.349 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.349 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.350 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.350 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.350 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.350 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.350 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.350 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.350 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.350 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.350 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.350 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.350 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.350 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.351 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.351 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.351 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.351 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.351 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.351 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.351 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.351 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.351 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.351 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.351 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.351 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.351 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.351 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.352 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.352 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.352 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.352 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.352 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.352 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.352 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.352 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.352 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.352 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.352 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.352 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.352 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.352 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.352 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.353 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.353 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.353 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.353 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.353 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.353 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.353 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.353 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.353 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.353 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.353 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.353 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:36:03Z" 2024-04-12T04:56:01.353 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.353 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.353 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.354 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-12T04:56:01.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: {"groups":[]} 2024-04-12T04:56:01.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.354 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.354 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2024-04-12T04:56:01.355 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.355 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2024-04-12T04:56:01.355 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.355 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.355 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2024-04-12T04:56:01.356 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.356 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:56:01.356 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.356 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.356 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "epoch": 15, 2024-04-12T04:56:01.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "predecessor_uuid": "ce930f76-1fa9-4790-8e77-21dfefa411cf", 2024-04-12T04:56:01.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": [], 2024-04-12T04:56:01.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period_map": { 2024-04-12T04:56:01.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "id": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "zonegroups": [ 2024-04-12T04:56:01.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "id": "a", 2024-04-12T04:56:01.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "name": "a", 2024-04-12T04:56:01.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "api_name": "", 2024-04-12T04:56:01.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "is_master": true, 2024-04-12T04:56:01.357 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "endpoints": [ 2024-04-12T04:56:01.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:56:01.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ], 2024-04-12T04:56:01.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hostnames": [], 2024-04-12T04:56:01.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hostnames_s3website": [], 2024-04-12T04:56:01.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:56:01.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "zones": [ 2024-04-12T04:56:01.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "id": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:56:01.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "name": "a2", 2024-04-12T04:56:01.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "endpoints": [ 2024-04-12T04:56:01.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "http://smithi043.front.sepia.ceph.com:8001" 2024-04-12T04:56:01.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ], 2024-04-12T04:56:01.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "log_meta": false, 2024-04-12T04:56:01.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "log_data": true, 2024-04-12T04:56:01.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-12T04:56:01.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "read_only": false, 2024-04-12T04:56:01.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "tier_type": "", 2024-04-12T04:56:01.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_from_all": true, 2024-04-12T04:56:01.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_from": [], 2024-04-12T04:56:01.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "redirect_zone": "", 2024-04-12T04:56:01.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "supported_features": [ 2024-04-12T04:56:01.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "compress-encrypted", 2024-04-12T04:56:01.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "resharding" 2024-04-12T04:56:01.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "id": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:56:01.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "name": "a1", 2024-04-12T04:56:01.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "endpoints": [ 2024-04-12T04:56:01.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "http://smithi043.front.sepia.ceph.com:8000" 2024-04-12T04:56:01.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ], 2024-04-12T04:56:01.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "log_meta": false, 2024-04-12T04:56:01.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "log_data": true, 2024-04-12T04:56:01.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-12T04:56:01.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "read_only": false, 2024-04-12T04:56:01.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "tier_type": "", 2024-04-12T04:56:01.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_from_all": true, 2024-04-12T04:56:01.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_from": [], 2024-04-12T04:56:01.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "redirect_zone": "", 2024-04-12T04:56:01.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "supported_features": [ 2024-04-12T04:56:01.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "compress-encrypted", 2024-04-12T04:56:01.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "resharding" 2024-04-12T04:56:01.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ], 2024-04-12T04:56:01.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "placement_targets": [ 2024-04-12T04:56:01.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "name": "default-placement", 2024-04-12T04:56:01.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "tags": [], 2024-04-12T04:56:01.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "storage_classes": [ 2024-04-12T04:56:01.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "STANDARD" 2024-04-12T04:56:01.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ], 2024-04-12T04:56:01.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "default_placement": "default-placement", 2024-04-12T04:56:01.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:56:01.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_policy": { 2024-04-12T04:56:01.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "groups": [ 2024-04-12T04:56:01.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "id": "sync-group", 2024-04-12T04:56:01.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "data_flow": { 2024-04-12T04:56:01.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "symmetrical": [ 2024-04-12T04:56:01.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "id": "sync-flow1", 2024-04-12T04:56:01.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "zones": [ 2024-04-12T04:56:01.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:56:01.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:56:01.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pipes": [ 2024-04-12T04:56:01.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "id": "sync-pipe", 2024-04-12T04:56:01.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "source": { 2024-04-12T04:56:01.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "bucket": "*", 2024-04-12T04:56:01.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "zones": [ 2024-04-12T04:56:01.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:56:01.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:56:01.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "dest": { 2024-04-12T04:56:01.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "bucket": "*", 2024-04-12T04:56:01.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "zones": [ 2024-04-12T04:56:01.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:56:01.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61" 2024-04-12T04:56:01.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "params": { 2024-04-12T04:56:01.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "source": { 2024-04-12T04:56:01.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "filter": { 2024-04-12T04:56:01.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "tags": [] 2024-04-12T04:56:01.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "dest": {}, 2024-04-12T04:56:01.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "priority": 0, 2024-04-12T04:56:01.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "mode": "system", 2024-04-12T04:56:01.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "user": "" 2024-04-12T04:56:01.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ], 2024-04-12T04:56:01.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "enabled" 2024-04-12T04:56:01.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "enabled_features": [ 2024-04-12T04:56:01.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "resharding" 2024-04-12T04:56:01.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "id": "b", 2024-04-12T04:56:01.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "name": "b", 2024-04-12T04:56:01.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "api_name": "", 2024-04-12T04:56:01.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "is_master": false, 2024-04-12T04:56:01.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "endpoints": [ 2024-04-12T04:56:01.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:56:01.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ], 2024-04-12T04:56:01.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hostnames": [], 2024-04-12T04:56:01.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "hostnames_s3website": [], 2024-04-12T04:56:01.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "master_zone": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:56:01.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "zones": [ 2024-04-12T04:56:01.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "id": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:56:01.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "name": "b1", 2024-04-12T04:56:01.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "endpoints": [ 2024-04-12T04:56:01.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "http://smithi149.front.sepia.ceph.com:8000" 2024-04-12T04:56:01.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ], 2024-04-12T04:56:01.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "log_meta": false, 2024-04-12T04:56:01.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "log_data": true, 2024-04-12T04:56:01.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-12T04:56:01.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "read_only": false, 2024-04-12T04:56:01.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "tier_type": "", 2024-04-12T04:56:01.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_from_all": true, 2024-04-12T04:56:01.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_from": [], 2024-04-12T04:56:01.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "redirect_zone": "", 2024-04-12T04:56:01.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "supported_features": [ 2024-04-12T04:56:01.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "compress-encrypted", 2024-04-12T04:56:01.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "resharding" 2024-04-12T04:56:01.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "id": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:56:01.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "name": "b2", 2024-04-12T04:56:01.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "endpoints": [ 2024-04-12T04:56:01.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "http://smithi149.front.sepia.ceph.com:8001" 2024-04-12T04:56:01.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ], 2024-04-12T04:56:01.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "log_meta": false, 2024-04-12T04:56:01.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "log_data": true, 2024-04-12T04:56:01.368 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "bucket_index_max_shards": 11, 2024-04-12T04:56:01.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "read_only": false, 2024-04-12T04:56:01.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "tier_type": "", 2024-04-12T04:56:01.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_from_all": true, 2024-04-12T04:56:01.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_from": [], 2024-04-12T04:56:01.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "redirect_zone": "", 2024-04-12T04:56:01.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "supported_features": [ 2024-04-12T04:56:01.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "compress-encrypted", 2024-04-12T04:56:01.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "resharding" 2024-04-12T04:56:01.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ], 2024-04-12T04:56:01.369 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "placement_targets": [ 2024-04-12T04:56:01.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "name": "default-placement", 2024-04-12T04:56:01.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "tags": [], 2024-04-12T04:56:01.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "storage_classes": [ 2024-04-12T04:56:01.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "STANDARD" 2024-04-12T04:56:01.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ], 2024-04-12T04:56:01.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "default_placement": "default-placement", 2024-04-12T04:56:01.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:56:01.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_policy": { 2024-04-12T04:56:01.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "groups": [] 2024-04-12T04:56:01.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "enabled_features": [ 2024-04-12T04:56:01.370 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "resharding" 2024-04-12T04:56:01.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ], 2024-04-12T04:56:01.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "short_zone_ids": [ 2024-04-12T04:56:01.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": "1873e84b-44dc-48e3-babd-40afdce2764a", 2024-04-12T04:56:01.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": 755891344 2024-04-12T04:56:01.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": "1ba046e0-d0ae-437d-b741-28b4299423c2", 2024-04-12T04:56:01.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": 4153048242 2024-04-12T04:56:01.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.371 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": "334587ae-93af-45b4-96b1-ade7b1773c00", 2024-04-12T04:56:01.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": 2639555996 2024-04-12T04:56:01.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:56:01.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": 400045239 2024-04-12T04:56:01.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "master_zonegroup": "a", 2024-04-12T04:56:01.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "master_zone": "3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61", 2024-04-12T04:56:01.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period_config": { 2024-04-12T04:56:01.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "bucket_quota": { 2024-04-12T04:56:01.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "enabled": false, 2024-04-12T04:56:01.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "check_on_raw": false, 2024-04-12T04:56:01.372 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_size": -1, 2024-04-12T04:56:01.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_size_kb": 0, 2024-04-12T04:56:01.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_objects": -1 2024-04-12T04:56:01.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "user_quota": { 2024-04-12T04:56:01.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "enabled": false, 2024-04-12T04:56:01.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "check_on_raw": false, 2024-04-12T04:56:01.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_size": -1, 2024-04-12T04:56:01.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_size_kb": 0, 2024-04-12T04:56:01.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_objects": -1 2024-04-12T04:56:01.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "user_ratelimit": { 2024-04-12T04:56:01.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_read_ops": 0, 2024-04-12T04:56:01.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_write_ops": 0, 2024-04-12T04:56:01.373 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_read_bytes": 0, 2024-04-12T04:56:01.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_write_bytes": 0, 2024-04-12T04:56:01.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "enabled": false 2024-04-12T04:56:01.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "bucket_ratelimit": { 2024-04-12T04:56:01.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_read_ops": 0, 2024-04-12T04:56:01.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_write_ops": 0, 2024-04-12T04:56:01.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_read_bytes": 0, 2024-04-12T04:56:01.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_write_bytes": 0, 2024-04-12T04:56:01.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "enabled": false 2024-04-12T04:56:01.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "anonymous_ratelimit": { 2024-04-12T04:56:01.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_read_ops": 0, 2024-04-12T04:56:01.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_write_ops": 0, 2024-04-12T04:56:01.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_read_bytes": 0, 2024-04-12T04:56:01.374 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "max_write_bytes": 0, 2024-04-12T04:56:01.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "enabled": false 2024-04-12T04:56:01.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_id": "a55da930-a452-44e6-b1f3-2c34e822e09c", 2024-04-12T04:56:01.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_name": "test-realm", 2024-04-12T04:56:01.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-04-12T04:56:01.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2024-04-12T04:56:01.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2024-04-12T04:56:01.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=dkyges-68 2024-04-12T04:56:01.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-68/ 2024-04-12T04:56:01.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-68/ 2024-04-12T04:56:01.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-68/ 2024-04-12T04:56:01.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:56:01.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.376 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.377 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.377 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.377 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:36:04 GMT 2024-04-12T04:56:01.377 INFO:tasks.rgw_multisite_tests:/dkyges-68/ 2024-04-12T04:56:01.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.377 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:/1d88k0BUc14ho+SPWqe3iXMnt8= 2024-04-12T04:56:01.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:/1d88k0BUc14ho+SPWqe3iXMnt8=', 'Content-Length': '0'} 2024-04-12T04:56:01.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx000004e3dccddbdb3e760-006618ba34-2493-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:36:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-68/ 2024-04-12T04:56:01.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-68/ 2024-04-12T04:56:01.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-68/ 2024-04-12T04:56:01.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.378 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.378 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.378 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.378 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:36:04 GMT 2024-04-12T04:56:01.378 INFO:tasks.rgw_multisite_tests:/dkyges-68/ 2024-04-12T04:56:01.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.378 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:Bkoxma7GWFdV5MMyExuob7DMvhU= 2024-04-12T04:56:01.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:Bkoxma7GWFdV5MMyExuob7DMvhU=', 'Content-Length': '0'} 2024-04-12T04:56:01.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005baa10fa85dc481-006618ba34-2493-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:36:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-68/obj1 2024-04-12T04:56:01.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-68/obj1 2024-04-12T04:56:01.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-68/obj1 2024-04-12T04:56:01.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:01.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.379 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.379 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:01.380 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.380 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:36:04 GMT 2024-04-12T04:56:01.380 INFO:tasks.rgw_multisite_tests:/dkyges-68/obj1 2024-04-12T04:56:01.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.380 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:BdyDqqaU8tHkWyGcwxs30REqqp8= 2024-04-12T04:56:01.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:36:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:BdyDqqaU8tHkWyGcwxs30REqqp8='} 2024-04-12T04:56:01.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa8d4d433b65b326-006618ba34-2493-a1'), ('Date', 'Fri, 12 Apr 2024 04:36:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.380 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=dkyges-69 2024-04-12T04:56:01.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-69/ 2024-04-12T04:56:01.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-69/ 2024-04-12T04:56:01.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-69/ 2024-04-12T04:56:01.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:56:01.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.381 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.381 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.381 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.381 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:36:04 GMT 2024-04-12T04:56:01.381 INFO:tasks.rgw_multisite_tests:/dkyges-69/ 2024-04-12T04:56:01.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.381 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:fo4qHf+nicY2irHLYJOi7HUTEgc= 2024-04-12T04:56:01.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:fo4qHf+nicY2irHLYJOi7HUTEgc=', 'Content-Length': '0'} 2024-04-12T04:56:01.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000e8646bf1904788d1-006618ba34-249c-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:36:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-69/ 2024-04-12T04:56:01.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-69/ 2024-04-12T04:56:01.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-69/ 2024-04-12T04:56:01.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.382 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.382 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.382 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.383 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:36:04 GMT 2024-04-12T04:56:01.383 INFO:tasks.rgw_multisite_tests:/dkyges-69/ 2024-04-12T04:56:01.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.383 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:ghamh1Zsfs/YEcGEOKtPlUbj2TM= 2024-04-12T04:56:01.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:36:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:ghamh1Zsfs/YEcGEOKtPlUbj2TM=', 'Content-Length': '0'} 2024-04-12T04:56:01.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e4e71db4ef05f240-006618ba34-249c-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:36:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-69/obj2 2024-04-12T04:56:01.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-69/obj2 2024-04-12T04:56:01.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2024-04-12T04:56:01.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-69/obj2 2024-04-12T04:56:01.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2024-04-12T04:56:01.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.384 INFO:tasks.rgw_multisite_tests:PUT 2024-04-12T04:56:01.384 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2024-04-12T04:56:01.384 INFO:tasks.rgw_multisite_tests:application/octet-stream 2024-04-12T04:56:01.384 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:36:04 GMT 2024-04-12T04:56:01.384 INFO:tasks.rgw_multisite_tests:/dkyges-69/obj2 2024-04-12T04:56:01.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.384 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:VlL5V7GqftCVqFWpubwHzcBzokk= 2024-04-12T04:56:01.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 12 Apr 2024 04:36:04 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:VlL5V7GqftCVqFWpubwHzcBzokk='} 2024-04-12T04:56:01.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000017e72080ae71c275-006618ba34-249c-a2'), ('Date', 'Fri, 12 Apr 2024 04:36:04 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.384 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.385 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: No --period given, using current period=b69a664f-970b-4d72-be0d-0071b991ee5c 2024-04-12T04:56:01.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: [ 2024-04-12T04:56:01.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T03:12:47.956902Z" 2024-04-12T04:56:01.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T04:11:09.167783Z" 2024-04-12T04:56:01.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.385 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T04:11:09.196453Z" 2024-04-12T04:56:01.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896564.701296_877.1", 2024-04-12T04:56:01.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T04:36:04.701296Z" 2024-04-12T04:56:01.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.386 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2024-04-12T04:56:01.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.386 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.386 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:05.105+0000 7fecc3ade780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:05.107+0000 7fecc3ade780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.387 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.388 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.389 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.390 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:36:05Z" 2024-04-12T04:56:01.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.392 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.392 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.392 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.392 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.392 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.392 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.392 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.392 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.392 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.392 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.392 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.392 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.392 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.393 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.393 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.393 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.393 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.393 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.393 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.393 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.393 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.393 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.393 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.393 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.393 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.393 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.393 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.394 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.394 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.394 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.394 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.394 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.394 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.394 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.394 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.394 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.394 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.394 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.394 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.394 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.395 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.395 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.395 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.395 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.395 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.395 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.395 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.395 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.395 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.397 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.397 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.397 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.397 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.397 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.397 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.397 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.397 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.397 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.397 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.397 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.398 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.398 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.398 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.398 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:36:05Z" 2024-04-12T04:56:01.398 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.398 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.398 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.398 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:10.240+0000 7f43683aa780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:10.241+0000 7f43683aa780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.400 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:36:10Z" 2024-04-12T04:56:01.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.404 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.404 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.404 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.404 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.404 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.404 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.404 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.404 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.404 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.405 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.405 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.405 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.405 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.405 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.405 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.405 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.405 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.405 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.405 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.405 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.405 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.406 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.406 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.406 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.406 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.406 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.406 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.406 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.406 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.406 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.406 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.406 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.406 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.406 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.406 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.407 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.407 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.407 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.407 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.407 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.407 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.407 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.407 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.407 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.407 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.407 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.407 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.407 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.407 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.407 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.407 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.408 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.408 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.408 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.408 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.408 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.408 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.408 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.408 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.408 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.408 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.408 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.408 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:36:10Z" 2024-04-12T04:56:01.409 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.409 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:15.370+0000 7f6e55f65780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:15.372+0000 7f6e55f65780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:36:15Z" 2024-04-12T04:56:01.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.414 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.414 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.415 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.415 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.415 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.415 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.415 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.415 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.415 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.415 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.415 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.415 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.415 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.415 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.416 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.416 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.416 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.416 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.416 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.416 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.416 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.416 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.416 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.416 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.416 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.416 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.416 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.417 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.417 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.417 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.417 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.417 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.417 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.417 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.417 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.417 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.417 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.417 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.417 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.417 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.417 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.418 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.418 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.418 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.418 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.418 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.418 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.418 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.418 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.418 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.418 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.418 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.418 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.418 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.419 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.419 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.419 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.419 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.419 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.419 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.419 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.419 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.419 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.419 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:36:15Z" 2024-04-12T04:56:01.419 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.419 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.419 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.420 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.420 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:20.498+0000 7f562d8ba780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:20.499+0000 7f562d8ba780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:36:20Z" 2024-04-12T04:56:01.425 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.425 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.425 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.425 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.425 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.425 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.425 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.425 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.425 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.425 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.425 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.425 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.425 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.426 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.426 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.426 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.426 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.426 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.426 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.426 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.426 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.426 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.426 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.426 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.426 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.426 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.426 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.427 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.427 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.427 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.427 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.427 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.427 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.427 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.427 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.427 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.427 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.427 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.427 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.427 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.427 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.428 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.428 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.428 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.428 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.428 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.428 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.428 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.428 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.428 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.428 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.428 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.428 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.428 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.428 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.428 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.428 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.429 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.429 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.429 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.429 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.429 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.429 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:36:20Z" 2024-04-12T04:56:01.429 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.429 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.429 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.429 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.429 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:25.620+0000 7fa927749780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:25.621+0000 7fa927749780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.430 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:36:25Z" 2024-04-12T04:56:01.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.434 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.434 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.434 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.434 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.435 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.435 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.435 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.435 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.435 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.435 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.435 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.435 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.435 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.435 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.435 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.435 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.435 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.436 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.436 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.436 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.436 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.436 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.436 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.436 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.436 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.436 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.436 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.436 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.436 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.436 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.437 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.437 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.437 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.437 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.437 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.437 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.437 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.437 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.437 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.437 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.437 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.437 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.437 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.437 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.437 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.438 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.438 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.438 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.438 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.438 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.438 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.438 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.438 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.438 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.438 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.438 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.438 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.438 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.439 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.439 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.439 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.439 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.439 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.439 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:36:25Z" 2024-04-12T04:56:01.439 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.439 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.439 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.440 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:30.743+0000 7f439783b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:30.744+0000 7f439783b780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.440 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.441 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.442 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.443 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:36:30Z" 2024-04-12T04:56:01.444 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.444 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.444 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.444 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.445 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.445 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.445 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.445 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.445 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.445 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.445 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.445 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.445 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.445 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.445 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.445 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.445 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.445 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.445 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.445 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.446 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.446 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.446 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.446 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.446 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.446 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.446 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.446 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.446 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.446 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.446 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.446 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.446 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.446 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.447 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.447 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.447 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.447 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.447 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.447 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.447 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.447 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.447 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.447 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.447 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.448 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.448 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.448 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.448 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.448 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.448 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.448 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.448 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.448 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.448 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.449 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.449 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.449 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.449 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.449 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.449 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.449 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.449 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.449 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.449 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:36:30Z" 2024-04-12T04:56:01.450 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.450 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.450 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.450 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.450 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.450 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:35.866+0000 7fee00c6e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:35.867+0000 7fee00c6e780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.451 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.452 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.453 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.454 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:36:35Z" 2024-04-12T04:56:01.455 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.455 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.455 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.455 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.455 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.455 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.455 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.456 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.456 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.456 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.456 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.456 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.456 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.456 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.456 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.456 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.456 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.456 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.456 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.456 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.456 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.456 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.457 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.457 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.457 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.457 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.457 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.457 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.457 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.457 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.457 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.457 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.457 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.457 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.457 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.457 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.457 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.458 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.458 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.458 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.458 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.458 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.458 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.458 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.458 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.458 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.458 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.458 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.458 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.458 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.458 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.458 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.459 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.459 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.459 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.459 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.459 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.459 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.459 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.459 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.459 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.459 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.459 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.459 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.459 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:36:35Z" 2024-04-12T04:56:01.459 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.459 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.460 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.460 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.460 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:40.997+0000 7f6a0e1f0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:40.998+0000 7f6a0e1f0780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.460 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.461 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.464 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:36:41Z" 2024-04-12T04:56:01.465 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.465 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.465 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.465 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.465 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.465 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.465 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.465 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.465 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.465 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.466 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.466 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.466 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.466 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.466 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.466 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.466 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.466 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.466 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.466 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.466 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.466 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.467 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.467 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.467 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.467 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.467 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.467 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.467 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.467 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.467 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.467 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.467 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.467 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.467 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.468 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.468 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.468 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.468 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.468 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.468 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.468 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.468 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.468 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.468 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.468 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.468 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.469 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.469 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.469 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.469 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.469 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.469 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.469 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.469 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.469 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.469 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.469 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.469 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.469 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.469 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.470 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.470 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.470 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.470 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:36:41Z" 2024-04-12T04:56:01.470 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.470 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.470 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.470 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.470 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:46.120+0000 7f30c3ce8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:46.121+0000 7f30c3ce8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.471 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.472 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.473 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.474 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.475 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:36:46Z" 2024-04-12T04:56:01.476 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.476 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.476 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.476 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.476 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.476 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.476 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.476 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.476 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.476 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.476 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.476 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.476 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.476 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.476 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.476 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.477 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.477 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.477 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.477 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.477 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.477 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.477 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.477 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.477 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.477 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.477 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.477 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.477 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.478 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.478 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.478 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.478 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.478 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.478 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.478 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.478 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.478 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.478 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.478 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.478 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.479 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.479 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.479 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.479 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.479 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.479 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.479 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.479 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.479 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.479 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.479 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.479 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.479 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.480 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.480 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.480 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.480 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.480 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.480 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.480 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.480 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.480 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.480 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.480 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:36:46Z" 2024-04-12T04:56:01.480 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.480 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.481 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.481 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.481 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:51.244+0000 7f9b66776780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:51.245+0000 7f9b66776780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.481 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.484 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.485 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:36:51Z" 2024-04-12T04:56:01.486 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.486 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.486 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.486 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.486 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.486 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.486 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.486 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.486 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.487 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.487 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.487 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.487 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.487 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.487 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.487 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.487 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.487 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.487 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.487 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.487 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.487 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.487 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.487 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.488 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.488 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.488 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.488 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.488 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.488 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.488 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.488 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.488 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.488 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.488 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.488 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.488 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.488 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.488 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.488 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.489 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.489 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.489 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.489 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.489 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.489 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.489 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.489 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.489 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.489 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.489 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.489 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.489 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.490 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.490 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.490 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.490 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.490 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.490 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.490 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.490 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.490 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.490 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.490 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.490 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:36:51Z" 2024-04-12T04:56:01.490 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.490 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.491 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.491 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:56.370+0000 7fb07e958780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:36:56.372+0000 7fb07e958780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.491 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.492 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.493 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.494 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.495 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:36:56Z" 2024-04-12T04:56:01.496 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.496 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.496 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.497 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.497 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.497 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.497 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.497 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.497 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.497 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.497 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.497 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.497 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.497 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.497 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.497 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.497 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.497 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.498 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.498 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.498 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.498 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.498 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.498 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.498 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.498 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.498 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.498 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.498 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.498 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.498 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.498 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.499 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.499 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.499 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.499 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.499 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.499 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.499 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.499 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.499 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.499 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.499 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.499 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.499 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.499 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.499 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.500 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.500 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.500 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.500 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.500 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.500 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.500 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.500 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.500 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.500 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.500 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.500 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.500 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.500 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.500 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.501 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.501 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.501 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:36:56Z" 2024-04-12T04:56:01.501 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.501 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.501 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.501 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:01.494+0000 7fafe7ada780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:01.495+0000 7fafe7ada780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.502 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.503 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.504 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.505 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:37:01Z" 2024-04-12T04:56:01.506 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.506 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.506 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.507 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.507 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.507 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.507 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.507 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.507 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.507 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.507 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.507 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.507 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.507 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.507 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.507 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.507 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.507 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.508 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.508 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.508 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.508 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.508 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.508 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.508 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.508 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.508 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.508 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.508 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.508 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.508 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.509 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.509 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.509 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.509 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.509 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.509 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.509 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.509 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.509 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.509 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.509 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.509 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.509 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.509 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.510 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.510 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.510 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.510 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.510 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.510 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.510 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.510 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.510 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.510 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.510 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.510 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.511 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.511 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.511 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.511 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.511 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.511 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.511 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.511 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:37:01Z" 2024-04-12T04:56:01.511 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.511 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.511 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.512 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.512 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.512 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:06.616+0000 7fbef1b32780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:06.617+0000 7fbef1b32780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.512 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.513 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.514 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.515 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.516 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:37:06Z" 2024-04-12T04:56:01.517 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.517 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.517 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.517 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.517 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.517 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.517 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.517 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.518 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.518 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.518 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.518 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.518 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.518 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.518 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.518 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.518 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.518 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.518 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.518 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.518 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.519 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.519 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.519 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.519 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.519 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.519 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.519 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.519 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.519 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.519 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.519 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.519 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.520 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.520 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.520 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.520 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.520 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.520 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.520 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.520 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.520 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.520 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.520 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.520 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.520 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.521 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.521 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.521 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.521 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.521 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.521 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.521 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.521 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.521 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.521 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.521 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.522 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.522 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.522 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.522 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.522 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.522 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.522 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.522 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:37:06Z" 2024-04-12T04:56:01.522 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.522 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.522 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.523 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.523 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:11.741+0000 7fd14c528780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:11.742+0000 7fd14c528780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.523 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.524 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.525 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.526 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.527 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:37:11Z" 2024-04-12T04:56:01.528 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.528 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.528 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.528 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.528 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.528 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.528 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.528 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.528 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.528 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.528 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.528 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.528 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.529 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.529 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.529 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.529 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.529 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.529 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.529 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.529 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.529 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.529 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.529 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.529 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.529 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.529 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.529 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.530 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.530 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.530 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.530 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.530 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.530 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.530 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.530 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.530 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.530 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.530 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.530 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.530 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.531 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.531 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.531 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.531 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.531 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.531 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.531 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.531 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.531 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.531 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.531 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.531 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.531 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.532 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.532 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.532 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.532 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.532 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.532 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.532 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.532 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.532 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.532 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.532 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:37:11Z" 2024-04-12T04:56:01.532 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.532 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.533 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.533 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.533 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:16.864+0000 7f108478c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:16.865+0000 7f108478c780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.533 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.534 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.535 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:37:16Z" 2024-04-12T04:56:01.538 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.538 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.538 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.538 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.538 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.538 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.538 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.538 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.538 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.539 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.539 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.539 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.539 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.539 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.539 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.539 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.539 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.539 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.539 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.539 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.539 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.539 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.539 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.540 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.540 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.540 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.540 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.540 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.540 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.540 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.540 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.540 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.540 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.540 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.540 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.540 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.540 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.540 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.541 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.541 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.541 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.541 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.541 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.541 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.541 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.541 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.541 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.541 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.541 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.541 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.541 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.541 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.541 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.542 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.542 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.542 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.542 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.542 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.542 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.542 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.542 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.542 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.542 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.542 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.542 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:37:16Z" 2024-04-12T04:56:01.542 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.542 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.543 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.543 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:21.987+0000 7f2968582780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:21.988+0000 7f2968582780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.543 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.544 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.545 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.546 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.547 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:37:21Z" 2024-04-12T04:56:01.548 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.548 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.548 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.548 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.548 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.548 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.548 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.548 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.549 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.549 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.549 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.549 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.549 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.549 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.549 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.549 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.549 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.549 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.549 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.549 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.549 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.549 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.549 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.550 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.550 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.550 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.550 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.550 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.550 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.550 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.550 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.550 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.550 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.550 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.550 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.551 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.551 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.551 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.551 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.551 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.551 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.551 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.551 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.551 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.551 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.551 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.551 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.551 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.551 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.552 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.552 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.552 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.552 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.552 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.552 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.552 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.552 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.552 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.552 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.552 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.552 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.552 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.552 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.552 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.553 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:37:21Z" 2024-04-12T04:56:01.553 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.553 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.553 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.553 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.553 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:27.113+0000 7f987aa70780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:27.114+0000 7f987aa70780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.556 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.557 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:37:27Z" 2024-04-12T04:56:01.558 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.558 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.558 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.559 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.559 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.559 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.559 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.559 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.559 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.559 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.559 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.559 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.559 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.559 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.559 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.559 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.559 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.559 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.560 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.560 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.560 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.560 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.560 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.560 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.560 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.560 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.560 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.560 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.560 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.560 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.560 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.560 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.561 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.561 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.561 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.561 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.561 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.561 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.561 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.561 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.561 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.561 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.561 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.561 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.561 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.561 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.562 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.562 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.562 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.562 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.562 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.562 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.562 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.562 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.562 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.562 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.562 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.562 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.562 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.562 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.562 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.563 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.563 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.563 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.563 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:37:27Z" 2024-04-12T04:56:01.563 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.563 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.563 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.563 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.563 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.563 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:32.236+0000 7fce41c7a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:32.237+0000 7fce41c7a780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.564 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.565 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.566 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.567 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.568 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:37:32Z" 2024-04-12T04:56:01.569 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.569 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.569 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.569 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.569 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.569 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.569 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.569 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.570 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.570 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.570 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.570 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.570 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.570 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.570 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.570 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.570 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.570 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.570 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.570 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.570 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.571 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.571 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.571 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.571 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.571 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.571 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.571 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.571 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.571 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.571 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.571 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.571 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.571 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.571 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.572 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.572 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.572 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.572 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.572 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.572 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.572 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.572 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.572 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.572 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.572 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.572 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.572 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.573 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.573 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.573 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.573 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.573 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.573 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.573 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.573 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.573 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.573 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.573 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.573 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.574 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.574 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.574 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.574 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.574 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:37:32Z" 2024-04-12T04:56:01.574 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.574 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.574 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.574 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi043.stdout: INFO: 2024-04-12T04:37:35.619+0000 7f23f2e6b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2024-04-12T04:56:01.574 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.575 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:37.359+0000 7f173fbd8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:37.360+0000 7f173fbd8780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:37:37Z" 2024-04-12T04:56:01.580 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.580 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.580 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.580 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.580 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.580 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.580 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.580 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.580 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.580 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.580 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.580 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.581 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.581 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.581 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.581 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.581 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.581 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.581 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.581 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.581 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.581 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.581 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.581 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.581 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.582 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.582 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.582 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.582 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.582 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.582 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.582 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.582 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.582 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.582 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.582 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.582 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.582 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.582 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.583 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.583 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.583 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.583 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.583 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.583 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.583 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.583 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.583 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.583 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.583 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.583 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896541.420306_857.1", 2024-04-12T04:56:01.583 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.583 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.583 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.584 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:35:41.420306Z", 2024-04-12T04:56:01.584 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.584 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.584 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.584 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.584 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.584 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.584 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.584 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.584 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.584 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:37:37Z" 2024-04-12T04:56:01.584 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.584 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.584 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896541.420306_857.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:35:41.420306Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896541.420306_857.1'} 2024-04-12T04:56:01.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=3 master=1_1712896564.701296_877.1 target=1_1712896541.420306_857.1 2024-04-12T04:56:01.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:42.484+0000 7f2374316780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: 2024-04-12T04:37:42.485+0000 7f2374316780 20 RGW-SYNC:meta: read sync status 2024-04-12T04:56:01.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 1, 2024-04-12T04:56:01.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "state": 1, 2024-04-12T04:56:01.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "1_1712896564.701296_877.1", 2024-04-12T04:56:01.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:36:04.701296Z", 2024-04-12T04:56:01.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "realm_epoch": 2 2024-04-12T04:56:01.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 1, 2024-04-12T04:56:01.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 1 2024-04-12T04:56:01.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:37:42Z" 2024-04-12T04:56:01.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.590 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2024-04-12T04:56:01.590 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.590 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.590 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.590 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.590 INFO:tasks.rgw_multisite_tests: "period": "b69a664f-970b-4d72-be0d-0071b991ee5c", 2024-04-12T04:56:01.590 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.590 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.590 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.590 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.590 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.591 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.591 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.591 INFO:tasks.rgw_multisite_tests: "marker": "1_1712891567.956902_97.1", 2024-04-12T04:56:01.591 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.591 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2024-04-12T04:56:01.591 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.591 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T03:12:47.956902Z", 2024-04-12T04:56:01.591 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.591 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.591 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.591 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.591 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.591 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.592 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.592 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.167783_650.1", 2024-04-12T04:56:01.592 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.592 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.592 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.592 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.167783Z", 2024-04-12T04:56:01.592 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.592 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.592 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.592 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.592 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.592 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.592 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.592 INFO:tasks.rgw_multisite_tests: "marker": "1_1712895069.196453_290.1", 2024-04-12T04:56:01.592 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.593 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.593 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.593 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:11:09.196453Z", 2024-04-12T04:56:01.593 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.593 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.593 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.593 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.593 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.593 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.593 INFO:tasks.rgw_multisite_tests: "state": 1, 2024-04-12T04:56:01.593 INFO:tasks.rgw_multisite_tests: "marker": "1_1712896564.701296_877.1", 2024-04-12T04:56:01.593 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.593 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.593 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.594 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:36:04.701296Z", 2024-04-12T04:56:01.594 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2024-04-12T04:56:01.594 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.594 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.594 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.594 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.594 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.594 INFO:tasks.rgw_multisite_tests: "total": 1, 2024-04-12T04:56:01.594 INFO:tasks.rgw_multisite_tests: "complete": 1 2024-04-12T04:56:01.594 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.594 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:37:42Z" 2024-04-12T04:56:01.594 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.594 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1712891567.956902_97.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2024-04-12T03:12:47.956902Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1712895069.167783_650.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.167783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '1_1712895069.196453_290.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:11:09.196453Z', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '1_1712896564.701296_877.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.701296Z', 'realm_epoch': 2}}] 2024-04-12T04:56:01.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1712891567.956902_97.1', 1: '1_1712895069.167783_650.1', 2: '1_1712895069.196453_290.1', 3: '1_1712896564.701296_877.1'} 2024-04-12T04:56:01.595 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2024-04-12T04:56:01.595 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.595 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: [ 2024-04-12T04:56:01.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:56:01.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:56:01.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.595 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:56:01.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:56:01.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:56:01.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:56:01.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:56:01.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "last_update": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:56:01.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.596 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:56:01.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:56:01.596 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.597 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-04-12T04:56:01.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-04-12T04:56:01.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setuser ceph since I am not root 2024-04-12T04:56:01.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stderr: INFO: ignoring --setgroup ceph since I am not root 2024-04-12T04:56:01.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "sync_status": { 2024-04-12T04:56:01.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "info": { 2024-04-12T04:56:01.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "sync", 2024-04-12T04:56:01.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "num_shards": 4, 2024-04-12T04:56:01.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "instance_id": 5236430379833617501 2024-04-12T04:56:01.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "markers": [ 2024-04-12T04:56:01.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 0, 2024-04-12T04:56:01.597 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "incremental-sync", 2024-04-12T04:56:01.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:56:01.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:56:01.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 1, 2024-04-12T04:56:01.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "incremental-sync", 2024-04-12T04:56:01.598 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:56:01.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.599 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 0, 2024-04-12T04:56:01.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:56:01.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 2, 2024-04-12T04:56:01.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "incremental-sync", 2024-04-12T04:56:01.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:56:01.601 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 57, 2024-04-12T04:56:01.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:56:01.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: { 2024-04-12T04:56:01.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "key": 3, 2024-04-12T04:56:01.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "val": { 2024-04-12T04:56:01.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "status": "incremental-sync", 2024-04-12T04:56:01.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:56:01.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "next_step_marker": "", 2024-04-12T04:56:01.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total_entries": 9, 2024-04-12T04:56:01.602 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "pos": 0, 2024-04-12T04:56:01.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:56:01.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: ] 2024-04-12T04:56:01.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "full_sync": { 2024-04-12T04:56:01.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "total": 66, 2024-04-12T04:56:01.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "complete": 66 2024-04-12T04:56:01.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: }, 2024-04-12T04:56:01.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: "current_time": "2024-04-12T04:37:42Z" 2024-04-12T04:56:01.603 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi043.stdout: INFO: } 2024-04-12T04:56:01.603 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2024-04-12T04:56:01.603 INFO:tasks.rgw_multisite_tests: "sync_status": { 2024-04-12T04:56:01.603 INFO:tasks.rgw_multisite_tests: "info": { 2024-04-12T04:56:01.603 INFO:tasks.rgw_multisite_tests: "status": "sync", 2024-04-12T04:56:01.604 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2024-04-12T04:56:01.604 INFO:tasks.rgw_multisite_tests: "instance_id": 5236430379833617501 2024-04-12T04:56:01.604 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.604 INFO:tasks.rgw_multisite_tests: "markers": [ 2024-04-12T04:56:01.604 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.604 INFO:tasks.rgw_multisite_tests: "key": 0, 2024-04-12T04:56:01.604 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.604 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-12T04:56:01.604 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000047497", 2024-04-12T04:56:01.604 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.604 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.604 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.604 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:28:48.591190Z" 2024-04-12T04:56:01.604 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.604 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.604 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.605 INFO:tasks.rgw_multisite_tests: "key": 1, 2024-04-12T04:56:01.605 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.605 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-12T04:56:01.605 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000032624", 2024-04-12T04:56:01.605 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.605 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2024-04-12T04:56:01.605 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.605 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:28:50.052149Z" 2024-04-12T04:56:01.605 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.605 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.605 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.605 INFO:tasks.rgw_multisite_tests: "key": 2, 2024-04-12T04:56:01.605 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.605 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-12T04:56:01.605 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000053790", 2024-04-12T04:56:01.606 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.606 INFO:tasks.rgw_multisite_tests: "total_entries": 57, 2024-04-12T04:56:01.606 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.606 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:36:04.610431Z" 2024-04-12T04:56:01.606 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.606 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.606 INFO:tasks.rgw_multisite_tests: { 2024-04-12T04:56:01.606 INFO:tasks.rgw_multisite_tests: "key": 3, 2024-04-12T04:56:01.606 INFO:tasks.rgw_multisite_tests: "val": { 2024-04-12T04:56:01.606 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2024-04-12T04:56:01.606 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000046432", 2024-04-12T04:56:01.606 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2024-04-12T04:56:01.606 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2024-04-12T04:56:01.606 INFO:tasks.rgw_multisite_tests: "pos": 0, 2024-04-12T04:56:01.606 INFO:tasks.rgw_multisite_tests: "timestamp": "2024-04-12T04:28:48.526647Z" 2024-04-12T04:56:01.607 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.607 INFO:tasks.rgw_multisite_tests: } 2024-04-12T04:56:01.607 INFO:tasks.rgw_multisite_tests: ] 2024-04-12T04:56:01.607 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.607 INFO:tasks.rgw_multisite_tests: "full_sync": { 2024-04-12T04:56:01.607 INFO:tasks.rgw_multisite_tests: "total": 66, 2024-04-12T04:56:01.607 INFO:tasks.rgw_multisite_tests: "complete": 66 2024-04-12T04:56:01.607 INFO:tasks.rgw_multisite_tests: }, 2024-04-12T04:56:01.607 INFO:tasks.rgw_multisite_tests: "current_time": "2024-04-12T04:37:42Z" 2024-04-12T04:56:01.607 INFO:tasks.rgw_multisite_tests:} 2024-04-12T04:56:01.607 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000047497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.591190Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032624', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2024-04-12T04:28:50.052149Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000053790', 'next_step_marker': '', 'total_entries': 57, 'pos': 0, 'timestamp': '2024-04-12T04:36:04.610431Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046432', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2024-04-12T04:28:48.526647Z'}}] 2024-04-12T04:56:01.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:56:01.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000047497', 1: '00000000000000000000:00000000000000032624', 2: '00000000000000000000:00000000000000053790', 3: '00000000000000000000:00000000000000046432'} 2024-04-12T04:56:01.607 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2024-04-12T04:56:01.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-68/ 2024-04-12T04:56:01.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-68/ 2024-04-12T04:56:01.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-68/ 2024-04-12T04:56:01.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2024-04-12T04:56:01.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.608 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.608 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.608 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.609 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:37:42 GMT 2024-04-12T04:56:01.609 INFO:tasks.rgw_multisite_tests:/dkyges-68/ 2024-04-12T04:56:01.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.609 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:cEDarPlPzzScIa1rOivW3HkUcIA= 2024-04-12T04:56:01.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:37:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:cEDarPlPzzScIa1rOivW3HkUcIA=', 'Content-Length': '0'} 2024-04-12T04:56:01.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009fbb8aaba42e891b-006618ba96-249c-a2'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:37:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-68/obj1 2024-04-12T04:56:01.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-68/obj1 2024-04-12T04:56:01.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-68/obj1 2024-04-12T04:56:01.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.610 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.610 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.610 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.610 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:37:42 GMT 2024-04-12T04:56:01.610 INFO:tasks.rgw_multisite_tests:/dkyges-68/obj1 2024-04-12T04:56:01.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.610 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:oycoHSN4N7Pj1Ifbag8hKQHQVbY= 2024-04-12T04:56:01.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:37:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:oycoHSN4N7Pj1Ifbag8hKQHQVbY=', 'Content-Length': '0'} 2024-04-12T04:56:01.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:36:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-68:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9370.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004bbe021a20013dcd-006618ba96-249c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:37:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-68/obj1 2024-04-12T04:56:01.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-68/obj1 2024-04-12T04:56:01.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2024-04-12T04:56:01.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-68/obj1 2024-04-12T04:56:01.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8001 2024-04-12T04:56:01.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2024-04-12T04:56:01.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.611 INFO:tasks.rgw_multisite_tests:GET 2024-04-12T04:56:01.611 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.611 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.611 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:37:42 GMT 2024-04-12T04:56:01.611 INFO:tasks.rgw_multisite_tests:/dkyges-68/obj1 2024-04-12T04:56:01.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.611 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:eJcZi6WU/X0Noh7c0dv/eS1spOQ= 2024-04-12T04:56:01.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:37:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:eJcZi6WU/X0Noh7c0dv/eS1spOQ=', 'Content-Length': '0'} 2024-04-12T04:56:01.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 12 Apr 2024 04:36:04 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61:dkyges-68:3b5d1d0a-3ee2-45bb-83d2-cfbf93535a61.9370.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca68280cce9d5937-006618ba96-249c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 12 Apr 2024 04:37:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-69/ 2024-04-12T04:56:01.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-69/ 2024-04-12T04:56:01.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-69/ 2024-04-12T04:56:01.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2024-04-12T04:56:01.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.612 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.613 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.613 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.613 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:37:42 GMT 2024-04-12T04:56:01.613 INFO:tasks.rgw_multisite_tests:/dkyges-69/ 2024-04-12T04:56:01.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.613 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:U3OROQ87fYNWR9YzwqhAJr9gjik= 2024-04-12T04:56:01.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:37:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:U3OROQ87fYNWR9YzwqhAJr9gjik=', 'Content-Length': '0'} 2024-04-12T04:56:01.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005139236537a2441a-006618ba96-2493-a1'), ('Content-Length', '0'), ('Date', 'Fri, 12 Apr 2024 04:37:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/dkyges-69/obj2 2024-04-12T04:56:01.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/dkyges-69/obj2 2024-04-12T04:56:01.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2024-04-12T04:56:01.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /dkyges-69/obj2 2024-04-12T04:56:01.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2024-04-12T04:56:01.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2024-04-12T04:56:01.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi043.front.sepia.ceph.com:8000 2024-04-12T04:56:01.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2024-04-12T04:56:01.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2024-04-12T04:56:01.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2024-04-12T04:56:01.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2024-04-12T04:56:01.614 INFO:tasks.rgw_multisite_tests:HEAD 2024-04-12T04:56:01.614 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.614 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.614 INFO:tasks.rgw_multisite_tests:Fri, 12 Apr 2024 04:37:42 GMT 2024-04-12T04:56:01.614 INFO:tasks.rgw_multisite_tests:/dkyges-69/obj2 2024-04-12T04:56:01.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2024-04-12T04:56:01.614 INFO:tasks.rgw_multisite_tests:AWS KUXF4P9H15V1YOGZBQF6:tBj91tpuQP+uMKUo4V+mlytIcgQ= 2024-04-12T04:56:01.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-166-generic', 'Date': 'Fri, 12 Apr 2024 04:37:42 GMT', 'Authorization': 'AWS KUXF4P9H15V1YOGZBQF6:tBj91tpuQP+uMKUo4V+mlytIcgQ=', 'Content-Length': '0'} 2024-04-12T04:56:01.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000041bb12903a9457a4-006618ba96-2493-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 12 Apr 2024 04:37:42 GMT'), ('Connection', 'Keep-Alive')] 2024-04-12T04:56:01.615 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2024-04-12T04:56:01.615 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.615 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-04-12T04:56:01.615 INFO:tasks.rgw_multisite_tests:Ran 48 tests in 6073.547s 2024-04-12T04:56:01.615 INFO:tasks.rgw_multisite_tests: 2024-04-12T04:56:01.615 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=4, failures=5) 2024-04-12T04:56:01.617 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-04-12T04:56:01.830 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=305af32a31a6404ca16fea19c3f63294 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_d540ebaca6b131a1dd560e7f69e024b133bbaa42/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-04-12T04:56:01.831 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2024-04-12T04:56:01.841 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2024-04-12T04:56:01.880 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-04-12T04:56:01.913 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-04-12T04:56:01.913 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:56:01.930 INFO:tasks.rgw.c1.client.0.smithi043.stdout:2024-04-12T04:56:01.911+0000 7f1e45afa700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm (PID: 47709) UID: 0 2024-04-12T04:56:01.930 INFO:tasks.rgw.c1.client.0.smithi043.stdout:2024-04-12T04:56:01.911+0000 7f1e4accf7c0 -1 shutting down 2024-04-12T04:56:07.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-04-12T04:56:08.014 INFO:tasks.rgw.c1.client.0:Stopped 2024-04-12T04:56:08.015 DEBUG:teuthology.orchestra.run.smithi043:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2024-04-12T04:56:08.037 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f /etc/ceph/vault-root-token 2024-04-12T04:56:08.127 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-04-12T04:56:08.127 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:56:08.127 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T04:56:08.125+0000 7f2402e8b700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 112560) UID: 0 2024-04-12T04:56:08.128 INFO:tasks.rgw.c1.client.1.smithi043.stdout:2024-04-12T04:56:08.125+0000 7f24080607c0 -1 shutting down 2024-04-12T04:56:13.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-04-12T04:56:13.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-04-12T04:56:14.228 INFO:tasks.rgw.c1.client.1:Stopped 2024-04-12T04:56:14.228 DEBUG:teuthology.orchestra.run.smithi043:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2024-04-12T04:56:14.252 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f /etc/ceph/vault-root-token 2024-04-12T04:56:14.327 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-04-12T04:56:14.327 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:56:14.328 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T04:56:14.326+0000 7fd557869700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm (PID: 47890) UID: 0 2024-04-12T04:56:14.328 INFO:tasks.rgw.c2.client.0.smithi149.stdout:2024-04-12T04:56:14.326+0000 7fd55ca3e7c0 -1 shutting down 2024-04-12T04:56:18.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-04-12T04:56:19.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-04-12T04:56:20.429 INFO:tasks.rgw.c2.client.0:Stopped 2024-04-12T04:56:20.430 DEBUG:teuthology.orchestra.run.smithi149:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2024-04-12T04:56:20.648 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f /etc/ceph/vault-root-token 2024-04-12T04:56:20.772 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2024-04-12T04:56:20.772 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:56:20.773 INFO:tasks.rgw.c2.client.1.smithi149.stdout:2024-04-12T04:56:20.771+0000 7f0f7de55700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm (PID: 49166) UID: 0 2024-04-12T04:56:20.773 INFO:tasks.rgw.c2.client.1.smithi149.stdout:2024-04-12T04:56:20.771+0000 7f0f8302a7c0 -1 shutting down 2024-04-12T04:56:25.154 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2024-04-12T04:56:25.154 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-04-12T04:56:26.874 INFO:tasks.rgw.c2.client.1:Stopped 2024-04-12T04:56:26.874 DEBUG:teuthology.orchestra.run.smithi149:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2024-04-12T04:56:26.895 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f /etc/ceph/vault-root-token 2024-04-12T04:56:26.968 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-12T04:56:26.978 DEBUG:teuthology.orchestra.run.smithi149:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2024-04-12T04:56:27.685 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-12T04:56:27.685 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-12T04:56:27.685 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2024-04-12T04:56:27.685 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:56:27.755 INFO:tasks.ceph.c2.osd.0:Stopped 2024-04-12T04:56:27.756 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2024-04-12T04:56:27.756 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:56:27.851 INFO:tasks.ceph.c2.osd.1:Stopped 2024-04-12T04:56:27.852 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2024-04-12T04:56:27.852 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:56:27.919 INFO:tasks.ceph.c2.osd.2:Stopped 2024-04-12T04:56:27.919 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-12T04:56:27.919 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2024-04-12T04:56:27.919 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:56:27.945 INFO:tasks.ceph.c2.mgr.x:Stopped 2024-04-12T04:56:27.945 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-12T04:56:27.945 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2024-04-12T04:56:27.945 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:56:28.005 INFO:tasks.ceph.c2.mon.a:Stopped 2024-04-12T04:56:28.005 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-12T04:56:28.006 DEBUG:teuthology.orchestra.run.smithi149:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-04-12T04:56:28.125 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi149.front.sepia.ceph.com 2024-04-12T04:56:28.125 DEBUG:teuthology.orchestra.run.smithi149:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2024-04-12T04:56:28.626 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi149.front.sepia.ceph.com 2024-04-12T04:56:28.627 DEBUG:teuthology.orchestra.run.smithi149:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2024-04-12T04:56:28.776 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi149.front.sepia.ceph.com 2024-04-12T04:56:28.776 DEBUG:teuthology.orchestra.run.smithi149:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2024-04-12T04:56:28.877 INFO:tasks.ceph:Archiving mon data... 2024-04-12T04:56:28.879 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596/data/c2.mon.a.tgz 2024-04-12T04:56:28.879 DEBUG:teuthology.orchestra.run.smithi149:> mktemp 2024-04-12T04:56:28.896 INFO:teuthology.orchestra.run.smithi149.stdout:/tmp/tmp.OjTru2x9Vd 2024-04-12T04:56:28.897 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.OjTru2x9Vd 2024-04-12T04:56:29.129 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0666 /tmp/tmp.OjTru2x9Vd 2024-04-12T04:56:29.255 DEBUG:teuthology.orchestra.remote:smithi149:/tmp/tmp.OjTru2x9Vd is 459KB 2024-04-12T04:56:29.329 DEBUG:teuthology.orchestra.run.smithi149:> rm -fr /tmp/tmp.OjTru2x9Vd 2024-04-12T04:56:29.346 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-12T04:56:29.347 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-12T04:56:29.350 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-12T04:56:29.413 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-12T04:56:29.425 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2024-04-12T04:56:29.818 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-12T04:56:29.818 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-12T04:56:29.819 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2024-04-12T04:56:29.819 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:56:29.875 INFO:tasks.ceph.c1.osd.0:Stopped 2024-04-12T04:56:29.876 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2024-04-12T04:56:29.876 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:56:29.944 INFO:tasks.ceph.c1.osd.1:Stopped 2024-04-12T04:56:29.944 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2024-04-12T04:56:29.944 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:56:30.004 INFO:tasks.ceph.c1.osd.2:Stopped 2024-04-12T04:56:30.004 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-12T04:56:30.005 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2024-04-12T04:56:30.005 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:56:30.032 INFO:tasks.ceph.c1.mgr.x:Stopped 2024-04-12T04:56:30.032 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-12T04:56:30.032 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2024-04-12T04:56:30.033 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T04:56:30.095 INFO:tasks.ceph.c1.mon.a:Stopped 2024-04-12T04:56:30.095 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-12T04:56:30.096 DEBUG:teuthology.orchestra.run.smithi043:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-04-12T04:56:30.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2024-04-12T04:56:30.171 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi043.front.sepia.ceph.com 2024-04-12T04:56:30.171 DEBUG:teuthology.orchestra.run.smithi043:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2024-04-12T04:56:30.541 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi043.front.sepia.ceph.com 2024-04-12T04:56:30.541 DEBUG:teuthology.orchestra.run.smithi043:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2024-04-12T04:56:30.630 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi043.front.sepia.ceph.com 2024-04-12T04:56:30.631 DEBUG:teuthology.orchestra.run.smithi043:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2024-04-12T04:56:30.714 INFO:tasks.ceph:Archiving mon data... 2024-04-12T04:56:30.714 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596/data/c1.mon.a.tgz 2024-04-12T04:56:30.714 DEBUG:teuthology.orchestra.run.smithi043:> mktemp 2024-04-12T04:56:30.735 INFO:teuthology.orchestra.run.smithi043.stdout:/tmp/tmp.OqiIRrXE5y 2024-04-12T04:56:30.735 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.OqiIRrXE5y 2024-04-12T04:56:30.973 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0666 /tmp/tmp.OqiIRrXE5y 2024-04-12T04:56:31.069 DEBUG:teuthology.orchestra.remote:smithi043:/tmp/tmp.OqiIRrXE5y is 501KB 2024-04-12T04:56:31.143 DEBUG:teuthology.orchestra.run.smithi043:> rm -fr /tmp/tmp.OqiIRrXE5y 2024-04-12T04:56:31.163 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-12T04:56:31.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-12T04:56:31.206 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-12T04:56:31.237 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-12T04:56:31.237 DEBUG:teuthology.orchestra.run.smithi043:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-12T04:56:31.279 DEBUG:teuthology.orchestra.run.smithi149:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-12T04:56:31.307 INFO:teuthology.orchestra.run.smithi149.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-12T04:56:31.309 INFO:teuthology.orchestra.run.smithi043.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-12T04:56:31.316 INFO:tasks.ceph:Archiving crash dumps... 2024-04-12T04:56:31.318 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596/remote/smithi043/crash 2024-04-12T04:56:31.319 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-12T04:56:31.387 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596/remote/smithi149/crash 2024-04-12T04:56:31.388 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-12T04:56:31.414 INFO:tasks.ceph:Compressing logs... 2024-04-12T04:56:31.414 DEBUG:teuthology.orchestra.run.smithi043:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T04:56:31.431 DEBUG:teuthology.orchestra.run.smithi149:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T05:07:55.803 INFO:tasks.ceph:Archiving logs... 2024-04-12T05:07:55.804 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596/remote/smithi043/log 2024-04-12T05:07:55.805 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-12T05:08:17.664 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596/remote/smithi149/log 2024-04-12T05:08:17.665 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-12T05:09:40.212 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-12T05:09:40.223 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-12T05:09:40.223 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-12T05:09:40.227 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-12T05:09:40.371 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-12T05:09:40.380 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-12T05:09:40.381 DEBUG:teuthology.orchestra.run.smithi043:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T05:09:40.383 DEBUG:teuthology.orchestra.run.smithi149:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T05:09:40.401 INFO:teuthology.orchestra.run.smithi043.stderr:bash: ntpq: command not found 2024-04-12T05:09:40.402 INFO:teuthology.orchestra.run.smithi149.stderr:bash: ntpq: command not found 2024-04-12T05:09:40.406 INFO:teuthology.orchestra.run.smithi043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T05:09:40.406 INFO:teuthology.orchestra.run.smithi043.stdout:=============================================================================== 2024-04-12T05:09:40.407 INFO:teuthology.orchestra.run.smithi043.stdout:^* hv01.front.sepia.ceph.com 3 8 377 33 +1213us[+1194us] +/- 52ms 2024-04-12T05:09:40.407 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv02.front.sepia.ceph.com 3 9 377 36 -847us[ -866us] +/- 60ms 2024-04-12T05:09:40.407 INFO:teuthology.orchestra.run.smithi043.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 93 -820us[ -838us] +/- 68ms 2024-04-12T05:09:40.407 INFO:teuthology.orchestra.run.smithi043.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T05:09:40.439 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T05:09:40.439 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2024-04-12T05:09:40.439 INFO:teuthology.orchestra.run.smithi149.stdout:^* hv01.front.sepia.ceph.com 3 6 377 38 +1146us[+1100us] +/- 52ms 2024-04-12T05:09:40.439 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv02.front.sepia.ceph.com 3 10 377 1028 -502us[ -777us] +/- 72ms 2024-04-12T05:09:40.439 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 50 -831us[ -877us] +/- 68ms 2024-04-12T05:09:40.440 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T05:09:40.441 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-12T05:09:40.451 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-12T05:09:40.451 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-12T05:09:40.473 DEBUG:teuthology.orchestra.run.smithi043:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-12T05:09:40.556 DEBUG:teuthology.orchestra.run.smithi149:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-12T05:09:40.695 DEBUG:teuthology.orchestra.run.smithi043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T05:09:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T05:09:40.734 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T05:09:40.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T05:09:40.767 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-12T05:09:40.777 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-12T05:09:40.807 INFO:teuthology.task.internal:Duration was 8152.184280 seconds 2024-04-12T05:09:40.807 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-12T05:09:40.841 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-12T05:09:40.842 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-12T05:09:40.845 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-12T05:09:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T05:09:40.909 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T05:09:41.359 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-12T05:09:41.360 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi043.front.sepia.ceph.com 2024-04-12T05:09:41.360 DEBUG:teuthology.orchestra.run.smithi043:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-12T05:09:41.388 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi149.front.sepia.ceph.com 2024-04-12T05:09:41.388 DEBUG:teuthology.orchestra.run.smithi149:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-12T05:09:41.414 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-12T05:09:41.414 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T05:09:41.430 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T05:09:41.502 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-12T05:09:41.503 DEBUG:teuthology.orchestra.run.smithi043:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-12T05:09:41.505 DEBUG:teuthology.orchestra.run.smithi149:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-12T05:09:41.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-12T05:09:41.600 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-12T05:09:41.600 DEBUG:teuthology.orchestra.run.smithi043:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-12T05:09:41.631 DEBUG:teuthology.orchestra.run.smithi149:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-12T05:09:41.684 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-12T05:09:41.697 DEBUG:teuthology.orchestra.run.smithi043:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-12T05:09:41.699 DEBUG:teuthology.orchestra.run.smithi149:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-12T05:09:41.729 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = core 2024-04-12T05:09:41.744 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = core 2024-04-12T05:09:41.759 DEBUG:teuthology.orchestra.run.smithi043:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-12T05:09:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T05:09:41.815 DEBUG:teuthology.orchestra.run.smithi149:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-12T05:09:41.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T05:09:41.832 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-12T05:09:41.843 INFO:teuthology.task.internal:Transferring archived files... 2024-04-12T05:09:41.844 DEBUG:teuthology.misc:Transferring archived files from smithi043:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596/remote/smithi043 2024-04-12T05:09:41.844 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-12T05:09:41.902 DEBUG:teuthology.misc:Transferring archived files from smithi149:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596/remote/smithi149 2024-04-12T05:09:41.903 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-12T05:09:41.941 INFO:teuthology.task.internal:Removing archive directory... 2024-04-12T05:09:41.941 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-12T05:09:41.944 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-12T05:09:41.992 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-12T05:09:42.004 INFO:teuthology.task.internal:Not uploading archives. 2024-04-12T05:09:42.005 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-12T05:09:42.024 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-12T05:09:42.025 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-12T05:09:42.027 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-12T05:09:42.046 INFO:teuthology.orchestra.run.smithi043.stdout: 262175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 12 05:09 /home/ubuntu/cephtest 2024-04-12T05:09:42.049 INFO:teuthology.orchestra.run.smithi149.stdout: 262179 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 12 05:09 /home/ubuntu/cephtest 2024-04-12T05:09:42.050 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-12T05:09:42.061 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-12T05:09:42.125 INFO:teuthology.nuke:Checking targets against current locks 2024-04-12T05:09:42.162 DEBUG:teuthology.nuke:shortname: smithi043 2024-04-12T05:09:42.162 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T05:09:42.206 DEBUG:teuthology.nuke:shortname: smithi149 2024-04-12T05:09:42.207 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T05:09:42.225 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 02:48:34.056169', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM360nfrc8+vzqLBmt/bUgq/Qji/8Ta1ZqA1yis6gch2B7vP9ozECqHJvYXqFyNKiYsii47X9221GsM42/v4qhM='} 2024-04-12T05:09:42.263 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-11_20:30:42-rgw-reef-release-distro-default-smithi/7652596', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 02:48:34.054259', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN4+Al/t3FeS8VxGyA79xzsyQJKFWB6bq35ATLnGIa+oeWgp3QpA7yvw/te4TGZwz36fc5R+dr7d8WywxqnZzLc='} 2024-04-12T05:09:42.327 INFO:teuthology.orchestra.console.smithi043:Power off 2024-04-12T05:09:42.328 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-12T05:09:42.364 INFO:teuthology.orchestra.console.smithi149:Power off 2024-04-12T05:09:42.364 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-12T05:09:42.379 DEBUG:teuthology.orchestra.console.smithi043:power off output: Chassis Power Control: Down/Off 2024-04-12T05:09:42.379 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T05:09:42.416 DEBUG:teuthology.orchestra.console.smithi149:power off output: Chassis Power Control: Down/Off 2024-04-12T05:09:42.417 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T05:09:42.432 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is on 2024-04-12T05:09:42.472 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2024-04-12T05:09:46.434 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T05:09:46.473 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T05:09:46.547 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is on 2024-04-12T05:09:46.584 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2024-04-12T05:09:50.548 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T05:09:50.586 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T05:09:50.660 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is off 2024-04-12T05:09:50.698 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is off 2024-04-12T05:09:50.761 INFO:teuthology.orchestra.console.smithi043:Power off completed 2024-04-12T05:09:50.800 INFO:teuthology.orchestra.console.smithi149:Power off completed 2024-04-12T05:09:50.904 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} duration: 8152.184280157089 failure_reason: rgw multisite test failures flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=305af32a31a6404ca16fea19c3f63294 status: fail success: false 2024-04-12T05:09:50.904 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-12T05:09:50.995 INFO:teuthology.run:FAIL